Crawl budget

Crawl budget is the amount of crawling a crawler is willing to spend on a given site over a period of time, determined jointly by how much the server can handle and whether the content is worth fetching again.

Google’s documentation splits it into two parts:

• Crawl capacity limit. Depends on how fast the site responds and how often it errors. When a site slows down or returns frequent errors, crawling automatically backs off.
• Crawl demand. Depends on how popular and how stale a URL is. Pages that never change get crawled less and less over time.

What wastes crawl budget

• Endless parameter combinations and faceted filter pages.
• Many addresses serving duplicate content (see canonical).
• Soft 404s — empty results pages that return HTTP 200 instead of 404.
• Multi-step redirect chains.
• A site structure diluted by large numbers of low-value pages.

When a crawler spends its allowance on these addresses, your real content pages get refreshed more slowly.

For most sites, it isn’t the bottleneck

Google’s documentation states plainly that sites with a few thousand pages generally don’t need to optimize for crawl budget. It tends to become a real issue only on very large sites, or on sites where automatically generated URLs have gotten out of hand — large e-commerce catalogs, sites with unchecked filter combinations, and the like.

For small and mid-sized sites, getting the basics right covers most of the benefit:

• fast responses;
• correct status codes (a missing page returns 404, a moved page returns a single 301);
• correct canonical links;
• an accurate sitemap.

How to check whether you have a problem

In Google Search Console, the Crawl Stats report shows how many requests Googlebot makes per day, average response time and the breakdown of response codes. Server logs give the same picture for every crawler. Warning signs include crawlers spending most of their requests on parameter URLs, a high share of redirects or errors, and important new pages taking a long time to be fetched for the first time.

What about AI crawlers?

How AI companies’ crawlers allocate their crawling has not been made public. The only way to see what they actually do is to look at your server logs and observe how often each one visits, and which pages.

What you can be sure of is the same set of engineering fundamentals:

• respond quickly;
• return correct status codes;
• put the main content in the HTML itself rather than behind JavaScript (see SSR vs. CSR);
• make sure robots.txt isn’t blocking crawlers you want by mistake.

Where it fits in AI visibility work

Crawl budget belongs to the technical layer of an AI visibility audit. On most company sites, the useful question is less “are we out of crawl budget?” and more “can a crawler that reaches our important pages actually read them?” If the answer is no, no amount of content work will make those pages a citation source. For how this connects to the bigger picture, see generative engine optimization (GEO) and SEO.