Crawling is the process by which automated programs, called crawlers or bots, follow links and sitemaps across the web to discover pages that exist — the first of three stages a page has to pass through before it can rank. This article is part of the What Is SEO? guide and goes deeper into crawling specifically than the overview of how search engines work, covering the technical details that actually determine whether a page gets found at all.
The Basics, Quickly
Google’s crawler is called Googlebot — and it’s actually a family of specialized crawlers rather than one bot. It discovers pages two ways: by following links from pages it already knows about, and by reading URLs listed in an XML sitemap submitted for the site. Once it fetches a page, it reads the content, follows any links on that page too, and the process continues outward. For the full three-stage picture — crawling, indexing, and ranking — see the dedicated guide. For more on Googlebot’s own variants and how to verify a real one, see What Is Googlebot? What follows here is what actually controls and limits that first stage.
Crawl Budget: The Limit Most Beginners Don’t Know Exists
Crawl budget is the number of pages a crawler will actually visit on a site within a given period, and it’s the product of two separate factors: crawl rate limit and crawl demand. Crawl rate limit is a technical ceiling — how many requests Googlebot can send to a server without overloading it, based on how quickly the server responds. Crawl demand is about interest — how many pages Google actually wants to explore, based on the site’s popularity, how frequently content changes, and what’s listed in the sitemap. The effective crawl budget is whichever of the two is lower.
This matters most for large sites. A ten-page brochure site will almost always get fully crawled regardless of budget. A site with tens of thousands of pages — especially one that generates URLs dynamically — can genuinely run out of crawl budget before reaching everything worth indexing.
Crawl Traps: Where Budget Gets Wasted
A crawl trap is a section of a site that generates an effectively unlimited number of URLs, which can consume crawl budget without ever leading to genuinely new content. Common causes: calendar widgets that generate a new URL for every future month indefinitely, faceted navigation where combining filters (size, color, price, brand) multiplies into thousands of URL variants, session IDs appended to URLs that make every visit technically unique, and sort/pagination parameters that produce a separate URL for the same underlying content sorted differently.
Redirect chains waste budget too, in a smaller but still real way — each hop from one redirect to the next consumes a separate crawl request before the crawler ever reaches the final destination.
How robots.txt Actually Works
robots.txt is a plain text file placed at a site’s root that tells crawlers which parts of the site they’re allowed to access, using a small set of supported directives — primarily user-agent, allow, disallow, and sitemap. It only governs the exact protocol, host, and port it’s hosted on — a robots.txt file on the main domain doesn’t automatically apply to a subdomain or a different protocol.
The most common and consequential mistake with robots.txt: a blanket 'Disallow: /' rule added during development, meant to keep a staging site out of search results, that accidentally makes it to the live site and blocks the entire domain from being crawled.
The misconception worth clearing up directly: blocking a URL in robots.txt does not guarantee it stays out of the index. If a blocked page is linked to from elsewhere, Google can still index it — just without being able to read or show its content, since it was never allowed to crawl it. Disallowing a page in robots.txt and adding a noindex tag to that same page is actually a contradiction: noindex only works if the page can be crawled in the first place, so blocking it via robots.txt prevents Google from ever seeing the noindex instruction.
XML Sitemaps: The Other Half of the Picture
An XML sitemap is a direct, submitted list of a site’s URLs that helps crawlers discover pages faster than they would by following links alone. Sitemaps and robots.txt work as complementary tools rather than overlapping ones: a sitemap should only include pages meant to be crawled and indexed, and any page disallowed in robots.txt shouldn’t also appear in the sitemap — including both sends a contradictory signal about the same URL.
Not Just Search Engines Anymore: AI Crawlers
A newer wrinkle as of 2026: AI-focused bots now crawl the web too, sometimes to train models and sometimes to retrieve content in response to a specific user question, which is a different purpose than a traditional search engine’s crawler. Some sites now manage these separately using an emerging file called llms.txt, distinct from robots.txt, specifically to signal how AI systems should treat their content. This space is still developing and isn’t governed by the same long-established standards as traditional search crawling.
Checking Your Own Site’s Crawl Activity
Google Search Console’s Crawl Stats report shows how often Googlebot is actually visiting a site, which file types it’s requesting, and any crawl errors it’s hitting — the most direct way to see whether crawl budget is actually a problem worth addressing rather than a theoretical concern.
Common Crawlability Mistakes
- Leaving a development-only
'Disallow: /'rule live after launch, blocking the entire site - Letting faceted navigation or calendar pages generate endless URL variants that eat crawl budget without adding real content
- Contradicting a noindex tag with a robots.txt block, which prevents Google from ever reading the noindex instruction
- Long redirect chains that waste crawl requests before reaching the actual destination
- A sitemap that includes disallowed or non-indexable URLs, sending mixed signals about the same pages
Frequently Asked Questions
Does every website need to worry about crawl budget?
Not really. Crawl budget becomes a genuine concern mainly for large sites with tens of thousands of pages or more, or sites that generate URLs dynamically. Smaller sites are typically crawled fully without any special management needed.
If a page is blocked in robots.txt, is it definitely not in Google’s index?
Not necessarily — a blocked page can still appear in search results if it’s linked to from elsewhere, just without a content snippet, since Google was never allowed to read the page itself.
How often does Google recrawl a site?
It varies enormously by site. Frequently updated, high-authority sites can be crawled multiple times a day; smaller or rarely updated sites may be crawled far less often.
The Bottom Line
Crawling is the entry point to everything else in SEO — a page that isn’t crawled can’t be indexed or ranked, no matter how good it is. For most small sites, the basics (letting crawlers in, submitting a clean sitemap) are enough. For larger or more complex sites, crawl budget, crawl traps, and correctly configured robots.txt rules become real, measurable factors worth actively managing rather than background details.
