Crawl budget
Crawl budget is the set of URLs on a site that Googlebot can and wants to crawl. Google defines it as the combination of crawl capacity limit, meaning how much crawling the server can handle without slowing down, and crawl demand, meaning how much Google wants to crawl the site's URLs.
Does crawl budget matter for every site?
No. When Google first explained the term on 16 January 2017, it said crawl budget is not something most publishers have to worry about, and that a site with fewer than a few thousand URLs will usually be crawled efficiently.
Its current guide is aimed at sites with more than 1 million unique pages that change about once a week, sites with more than 10,000 pages that change daily, and sites where Search Console reports many URLs as discovered but not indexed. Google calls these rough estimates.
What wastes crawl budget?
Crawl demand depends partly on the URLs Google knows about, which it calls perceived inventory. Duplicate URLs, endless filter and parameter combinations, soft 404 pages and long redirect chains all add URLs that Googlebot spends time on. Crawl capacity falls when the server slows down or returns 5xx errors or 429 responses.
How to manage it
Google advises against using noindex to save crawl budget, because Googlebot still requests the page before dropping it.
- Consolidate duplicate content and block unimportant URLs in robots.txt
- Return 404 or 410 for pages removed permanently, and fix soft 404s
- Keep XML sitemaps current, with lastmod dates for updated pages
- Avoid long redirect chains
- Speed up server responses and support 304 Not Modified responses
Example
A fashion retailer with 8,000 products lets every combination of size, colour, brand and price filters create its own URL. Googlebot finds hundreds of thousands of near-duplicate pages and new products wait to be crawled. Blocking the filter parameters in robots.txt and linking to products from category pages points crawling back at the URLs that sell.
Related terms
Further reading
Our Organic Search service puts this into practice for brands in the UK and US.
