Crawl Budget: What Google Actually Rations, and How to Tell Whether You Have a Problem

Crawl budget is the most over-applied concept in technical SEO. It gets invoked to explain rankings, indexation, traffic drops and slow publishing, on sites where Google has explicitly said it does not apply. Meanwhile the sites that genuinely have a crawl budget problem often diagnose it as something else, because the symptom shows up in a report that does not have the words “crawl budget” anywhere on it.

Google publishes enough detail to settle almost all of this. This guide works from that documentation, and from the two Search Console reports that will tell you which situation you are in.

Crawl budget is the set of URLs on a site that Google both can and wants to crawl in a given period. It is the product of two independent limits — a crawl capacity limit derived from how well your server handles Googlebot, and crawl demand derived from how much Google judges your URLs worth revisiting — and only a specific class of large or fast-changing site needs to manage it.

Key takeaways

• Google’s definition has two halves, and most advice covers one. Taking capacity and demand together, Google defines crawl budget as the set of URLs that Google can and wants to crawl — and notes that even when capacity is not reached, low demand means less crawling.
• Google tells most sites not to bother. Its guide opens by saying that if your site does not have many rapidly changing pages, or pages seem to be crawled the same day that they are published, you don’t need to read this guide.
• There are three qualifying site types, and one of them is a symptom. The third is sites with a large portion of their total URLs classified as “Discovered - currently not indexed”.
• You cannot ask for more. Search Console’s own guidance is blunt: “You can’t tell Google to increase your crawl rate.” You can only make the site cheaper to crawl or the inventory worth crawling.
• A budget is per hostname. Google’s crawling infrastructure defines a site as a unique hostname, so www.example.com and shop.example.com are budgeted separately.

Do you actually have a crawl budget problem?

Start here, because for most readers the answer is no, and acting on the rest of this guide would be work with no return.

Google names three site types the crawl budget guide is written for:

• Large sites — 1 million+ unique pages — with content that changes moderately often, about once a week.
• Medium or larger sites — 10,000+ unique pages — with very rapidly changing content, daily.
• Sites with a large portion of their total URLs classified by Search Console as Discovered – currently not indexed.

Search Console’s Crawl Stats report draws a similar line from the other direction: it is aimed at advanced users, and “if you have a site with fewer than a thousand pages, you should not need to use this report or worry about this level of crawling detail.”

The third bullet is the interesting one, because it converts a vague worry into an observable measurement. If a meaningful share of your known URLs sits in Discovered – currently not indexed — Google found them and has not fetched them — you have the symptom, whatever your page count. What that status means and how to diagnose it is covered in discovered – currently not indexed.

And the simplest disqualifying test is Google’s own: publish something and see when it gets crawled. If new pages are fetched the same day, the budget is not your constraint, and whatever problem you are chasing is a different one.

The two halves: capacity and demand

Crawl budget is not a single dial. It is the smaller of two independent constraints, and the fix differs completely depending on which one is binding.

Crawl capacity limit

Google calculates a crawl capacity limit — also called hostload — to avoid overwhelming your server. It caps the total time your server spends holding connections open for Google, factoring in both the number of parallel connections and how long each lasts.

Two things about it are worth internalising. First, every site starts with the same default, conservative crawl capacity limit, which Google’s systems raise over time if there is demand to crawl more and the site stays healthy. Second, it is a feedback loop driven by your server’s measured behaviour:

“If the site responds consistently and its response times (including latency and Time-to-First Byte) remain stable or improve, the limit goes up… If the site slows down (latency increases or response times become longer), or responds with server errors (5xx HTTP status codes) or rate-limiting signals (such as HTTP 429), the limit goes down and Google crawls less.”

That is the whole mechanism. Latency and errors are the inputs; crawl rate is the output. Search Console’s troubleshooting section says the same thing from the symptom side — if your site responds slowly, or your error rate increases, Googlebot will throttle back its requests to avoid overloading your server.

There is a trap attached. If you deliberately return 503 or 429 to shed load during a spike, Google’s guidance is to keep it brief: “Be sure not to return 503 or 429 for more than two or three days, though, or it can signal Google to crawl your site less frequently in the long term.” A temporary defence held too long becomes a permanent penalty on your crawl rate.

Crawl demand

Capacity says how much Google can crawl. Demand says how much it wants to. For Googlebot, demand varies with a site’s size, update frequency, page quality and relevance compared with other sites, and Google names three factors you can influence:

Perceived inventory — and Google flags this one as decisive:

“Without guidance from you, Google tries to crawl all or most of the URLs that it knows about on your site. If many of these URLs are duplicates, or you don’t want them crawled for some other reason (removed, unimportant, and so on), this wastes a lot of Google crawling time on your site. This is the factor that you can positively control the most.”

Popularity — URLs more popular on the internet are crawled more often to keep them fresh.

Staleness — Google’s systems want to recrawl frequently enough to catch changes.

Site-wide events change demand too: Google notes that a site move may trigger an increase in crawl demand in order to reprocess content under the new URLs.

The practical consequence of having two halves is that a fast server does not guarantee a well-crawled site. Google’s summary is explicit: “Even if the crawl capacity limit isn’t reached, if crawl demand is low, Google will crawl your site less.” Buying more origin capacity to fix a demand problem is a common and expensive mistake.

Reading the Crawl Stats report properly

Search Console → Settings → Crawl stats. It is available only for root-level properties — a Domain property, or a URL-prefix property at the root — which is the first reason many teams have never opened it.

Four things in it repay attention.

Host status. Three states, covering the last 90 days: no significant availability issues, issues but not recently, or an issue in the last week. It is assessed across robots.txt fetching, DNS resolution and server connectivity, and Google draws a dotted threshold line on each chart — cross it and the category counts as an issue. Anything other than green here means capacity is your binding constraint, and nothing you do to inventory will help until it is fixed.

Crawl responses. The distribution of what Google actually received, by response type, as a percentage of all crawl responses. Google’s expectation is plain: “In normal circumstances, the vast majority of responses should be 200 responses.” Note that the data is based on total number of requests, not by URL — a URL that returned 500 once and 200 once shows as 50% each.

Crawl purpose. Two values, and this is the most underused number in the report. Discovery means the URL was never crawled before; Refresh is a recrawl of a known page. The ratio tells you where your budget is going. A site publishing heavily that shows almost no discovery crawls has a discovery problem; a site with a stable page count showing mostly discovery crawls is generating URLs it does not know about.

File type and Googlebot type. Google’s own prompt here is worth copying into an audit: “Is Google requesting many small images that should be blocked? Is Google requesting resources that are hosted on another, less responsive site?” The Page resource load agent covers secondary fetches for images and CSS needed to render a page. On a JavaScript-heavy site that share can be substantial, and it comes out of the same capacity as your HTML.

One caveat before you reconcile the report against your logs. Google flags a known issue that the report captures most crawl requests but not necessarily all. There is also a specific case where the report reads higher than your logs: when robots.txt has been unavailable for too long, Google counts crawls it might have made without actually making them.

The robots.txt dependency almost nobody accounts for

This is documented in the Crawl Stats help page rather than the crawl budget guide, which is why it rarely appears in crawl budget advice — and it is the single fastest way to lose your entire crawl rate.

Your site does not need a robots.txt file, but it must return an acceptable response when asked for one. Google counts HTTP 200 with a file (valid, invalid or empty) as successful, and also counts 403, 404 or 410 as successful — those simply mean there is no file. What is not successful is 429 or 5XX, because that reads as a connection problem rather than an answer.

What happens next is a documented escalation:

Time since last successful robots.txt fetch	Google’s behaviour
Under 24 hours	Uses the cached robots.txt and crawls normally
First 12 hours after failure	Stops crawling the site, keeps requesting robots.txt
12 hours to 30 days	Uses the last successfully fetched file, keeps retrying
After 30 days, homepage available	Acts as if there is no robots.txt and crawls without restraint
After 30 days, homepage unavailable	Stops crawling the site

A WAF or rate limiter that starts returning 429 for /robots.txt will halt crawling within hours, and nothing in the Page Indexing report will say so. Check the robots.txt fetching chart in Host status before you check anything else. The file’s other failure modes — including the noindex directive that does not work there — are in robots.txt for developers.

What actually moves the number

Ordered by typical effect on a site that genuinely qualifies.

Cut perceived inventory

The highest-leverage work, because it is the factor Google says you control most. In order:

Consolidate duplicates so crawling focuses on unique content rather than unique URLs. Faceted navigation, sort parameters, session IDs, tracking parameters and paginated variants are the usual generators.

Block genuinely unimportant URLs in robots.txt. Google’s own examples are infinite-scrolling pages that duplicate information on linked pages, and differently sorted versions of the same page. Blocking prevents crawling and, in Google’s words, “significantly decreases the chance the URLs will be processed by other Google systems.”

Return 404 or 410 for permanently removed pages — and understand the asymmetry Google spells out. It “won’t forget a URL that it knows about, but a 404 status code is a strong signal not to crawl that URL again.” Whereas blocked URLs will stay part of your crawl queue much longer, and get recrawled when the block is removed. Deleting a page properly retires it; hiding it behind robots.txt does not.

Eliminate soft 404s. Google is direct: “soft 404 pages will continue to be crawled, and waste your budget.” Any framework that serves an application shell with a 200 for unmatched URLs manufactures these at scale — the mechanism is covered in Vue SEO and single-page application SEO.

Make each crawl cheaper

Improve response times. This raises the capacity limit directly, because the limit is computed from measured latency and error rate. Front-end work matters too, since page resources are fetched from the same budget — Core Web Vitals for developers covers the measurement.

Support conditional requests. Google asks for 304 (Not Modified) responses: if a page has not changed since the last crawl, a 304 tells Google to reuse its cached version, saving bandwidth on both sides. On a large catalogue where most pages are unchanged between crawls, this is the least glamorous and most effective item on the list.

Shorten redirect chains. Google’s guidance — “avoid long redirect chains, which have a negative effect on crawling” — has a precise cost attached in the Crawl Stats documentation: each request in the redirect chain is counted as a separate request. A three-hop chain spends three requests to deliver one page.

Give the crawler better signals

Keep sitemaps current, including <lastmod> for updated content. Google reads sitemaps regularly, and a sitemap where every URL claims today’s date teaches Google to discount the field.

Link the pages you care about from pages that get crawled often. Internal links are how demand gets distributed inside a site; a page reachable only from the sitemap is competing for attention with nothing to recommend it. Finding those systematically is what an internal linking audit does.

The two things Google says increase budget

Google’s own summary of how to get more is short. Add server resources, if your constraint is capacity — the URL Inspection tool reporting Hostload exceeded is the specific signal. Or improve content quality for the product you are targeting, since Google allocates crawling resources by factoring in popularity, overall user value, content uniqueness and serving capacity.

Note what is absent: there is no submission, no setting, no request form. As Search Console puts it, you cannot tell Google to increase your crawl rate.

Diagnosing a crawl rate that dropped

A sudden decline over a week or two is a distinct problem from a chronically low rate, and Google lists the likely causes. Work them in this order, because the first is the fastest to check and the most common:

1. A new or overly broad robots.txt rule. Confirm you are not blocking CSS or JavaScript that Google needs to understand the content.
2. Slower responses. Googlebot throttles back to avoid overloading the server. The Average response time chart will show it.
3. A higher error rate. Same throttling mechanism, different trigger.
4. Lower demand. Google’s phrasing: if a site has information that changes less frequently, or is not very high quality, “we might not crawl it as frequently.”

The first three are visible in Crawl Stats within minutes. The fourth is not a technical finding and cannot be fixed technically.

For anything that needs URL-level attribution rather than aggregates — which templates absorb the crawl, whether Googlebot is spending its requests on parameter URLs, how the ratio of 200s to redirects varies by section — server logs are the only source with the resolution to answer it. The method is in log file analysis for SEO.

What crawl budget is not

It is not a ranking factor. Crawling is a prerequisite for indexing, and indexing is a prerequisite for ranking. Being crawled more does not make a page rank better.

It is not why one page is not indexed. A single URL missing from the index is almost never a budget question. Check its status in the Page Indexing report first: if it says Crawled – currently not indexed, Google read the page and declined it, which is a content verdict with a content fix.

It is not the same as index bloat, though they share causes. Both get worse when a site generates URLs it does not need, but bloat is about what ends up in the index and budget is about what gets fetched. The decision framework for which URLs should exist at all is in canonical vs noindex.

It is not a fixed quota you can see. There is no crawl budget figure in Search Console. Total crawl requests is an observation of what happened, not a statement of what was allowed.

Where this fits

Crawl budget is the system beneath two Search Console statuses: discovered – currently not indexed, which is what a capacity or demand shortfall looks like in the report, and crawled – currently not indexed, which is not a budget problem at all. The controls sit in robots.txt for developers and canonical vs noindex; the measurement sits in log file analysis for SEO. Crawl and indexation together make up one phase of the 12-phase SEO and GEO audit.

If your URL inventory has outgrown what Google is willing to fetch and you would rather have that quantified than argued about, GEO and technical SEO consulting is where I do that work.

FAQ

What is crawl budget in SEO?

Crawl budget is how much of your site Google can and will crawl in a given period. Google defines it as the combination of two elements: a crawl capacity limit, calculated so that crawling does not overwhelm your server, and crawl demand, which reflects how much Google wants to crawl your URLs based on inventory, popularity and staleness. Taken together, Google describes crawl budget as the set of URLs that Google can and wants to crawl.

Does my site have a crawl budget problem?

Probably not. Google’s guide is written for sites with over a million pages changing weekly, sites over ten thousand pages changing daily, and sites with a large share of URLs in Discovered – currently not indexed. Its own disqualifying test is simpler: if your pages seem to be crawled the same day that they are published, you do not need the guide. Search Console adds that sites under a thousand pages should not need the Crawl Stats report at all.

How do I increase my crawl budget?

Google lists two levers. Add server resources if capacity is the constraint — the signal is Hostload exceeded in the URL Inspection tool. And improve content quality for the Google product you are targeting, since crawling resources are allocated by factoring in popularity, user value, content uniqueness and serving capacity. Beyond that, the work is subtractive: remove the duplicate, removed and unimportant URLs that Google is spending the budget on. There is no way to request a higher crawl rate — Search Console states plainly that “you can’t tell Google to increase your crawl rate.”

Is crawl budget a ranking factor?

No. Crawling is what allows a page to be indexed, and indexing is what allows it to rank, but crawl frequency itself is not a quality signal. A page crawled daily does not outrank an equivalent page crawled monthly. The reason crawl budget matters on large sites is coverage and freshness — whether your pages get into the index at all, and how quickly changes are picked up.

Do subdomains share a crawl budget?

No. Google’s crawling infrastructure defines a site as a unique hostname, so https://www.example.com/ and https://code.example.com/ are treated as separate sites with separate crawl budgets. That has an architectural consequence: moving a large, low-value section onto its own subdomain isolates its crawl demand from the main site’s.

Does blocking URLs in robots.txt save crawl budget?

Yes for the blocked URLs, but with a caveat Google states explicitly. Blocking prevents crawling and reduces the chance the URLs are processed by other Google systems. But blocked URLs “will stay part of your crawl queue much longer, and will be recrawled when the block is removed” — whereas a 404 is “a strong signal not to crawl that URL again.” For pages that are permanently gone, delete them and return 404 or 410. Reserve robots.txt for URLs that exist and should keep existing but do not need crawling.

Why did my crawl rate suddenly drop?

Google names four causes: a new or overly broad robots.txt rule, slower server responses, a higher server error rate, or reduced demand because the content changes rarely or is judged low quality. Check the Crawl Stats report’s Host status and Average response time first — the first three are visible there immediately. Also verify robots.txt is returning a successful response: if it starts returning 429 or 5XX, Google stops crawling within twelve hours while it keeps retrying.