If Googlebot never crawls a page, it can’t index it. If it’s never indexed, it can’t rank. Crawl budget and indexing sit at the very start of that chain, which is why problems here quietly cap everything else a site does for SEO.

Crawl budget is the number of pages on a site that Googlebot is willing and able to crawl within a given period, based on the site’s server capacity and how valuable Google judges its content to be.

What crawl budget actually is

Crawl budget comes from two separate factors working together: the crawl rate limit, which is how many requests your server can handle without straining, and crawl demand, which is how interested Google actually is in your URLs based on their popularity and freshness. A site with a fast, stable server but low-value or rarely updated content still won’t get crawled heavily, because demand is the limiting factor rather than capacity.

Most sites never need to think about this directly. This is part of the technical SEO foundation that becomes an active concern mainly at scale — Google’s own documentation frames crawl budget optimization as most relevant for very large sites whose content changes moderately, plus smaller sites in the ten-thousand-page range whose content changes daily, and any site accumulating a large number of pages stuck in “discovered but not indexed” status. Below that scale, Google generally has more than enough capacity to crawl everything that matters.

Googlebot crawling

Crawling is the discovery step — Googlebot follows links and reads sitemaps to find URLs, then requests them to see what’s there. Nothing downstream happens without it. Two things commonly waste crawl activity on the pages that don’t need it:

• Duplicate URLs. Faceted navigation, sorting parameters, and session IDs can generate thousands of URLs that all serve the same or near-identical content. Googlebot crawls each one separately even when the content is identical, spending requests it could have used elsewhere.
• Redirect chains and soft 404s. Every hop in a redirect chain is a separate crawl request, and a soft 404 (a page that returns a 200 status but shows no real content) forces Googlebot to keep revisiting a page that’s effectively dead.

Internal linking also shapes crawl demand directly — pages that get linked to frequently, especially from high-authority pages, signal to Google that they’re a priority, which increases how often they get revisited.

robots.txt SEO

A robots.txt file controls crawling, not indexing — and mixing those up is the single most common mistake in this area. Disallowing a URL in robots.txt stops Googlebot from requesting the page at all, but if that page is already linked to from elsewhere, Google can still list the URL in search results using just the anchor text and surrounding context it’s already seen, without ever crawling the page’s actual content.

This matters directly for a common misconfiguration: pairing a robots.txt disallow with a noindex meta tag on the same page. Since Googlebot can’t crawl a disallowed page, it never sees the noindex instruction sitting inside it, so the page can still appear indexed against the site owner’s intent. The two directives need to be used separately depending on the goal:

• Use robots.txt disallow when a URL shouldn’t be crawled at all — internal search results pages, admin paths, or infinite filter combinations.
• Use noindex when a page should still be crawled and its links followed, but shouldn’t appear in search results itself.

noindex vs nofollow

These two directives control different things entirely: noindex governs whether a page can appear in search results, while nofollow governs whether link authority passes through a specific link. Confusing them is easy because both are commonly set in the same meta tag, but they answer different questions.

Google’s own documentation confirms that noindex can be combined with a nofollow hint in the same tag, written as <meta name="robots" content="noindex, nofollow" />, when a page should be excluded from results and its outbound links shouldn’t pass authority. More often, though, the useful pairing is noindex, follow — used on something like a paginated archive page you don’t want ranking on its own, but that links out to articles you do want indexed. noindex, follow tells Google to skip the page itself while still crawling and crediting what it links to.

Nofollow alone is not a substitute for noindex. Googlebot can still discover and index a nofollowed link’s destination through other paths, so relying on nofollow to keep a page out of the index doesn’t reliably work.

Index coverage report

The Page Indexing (Index Coverage) report in Search Console is the fastest way to see whether crawl and indexing issues are actually happening on a live site, rather than guessing from the outside. Rather than working through every possible status label, start with these three checks:

• Compare “All known pages” against “All submitted pages.” A large gap usually means Google is discovering URLs through links that aren’t in your sitemap, which is worth investigating on its own.
• Look at “Discovered — currently not indexed.” This means Google found the URL but hasn’t crawled it yet, often because of low perceived priority or limited crawl budget — a strong signal to check internal linking to that page.
• Check “Duplicate, submitted URL not selected as canonical.” This means Google indexed a different version of the page than the one submitted, usually harmless for rankings but worth confirming the canonical points where intended.

Run this check on a schedule rather than only after a traffic drop — indexing issues tend to accumulate quietly as a site grows and gets restructured.

Crawl errors

Crawl errors show up as specific status codes in Search Console’s Crawl Stats and Coverage reports, each pointing to a different fix:

• Server errors (5xx) mean the server failed to respond to Googlebot’s request. Persistent 5xx errors on important pages can get them dropped from crawling consideration until the underlying server issue is resolved.
• Redirect errors happen when Googlebot hits a redirect loop or a chain that’s too long to follow. Point redirects directly at their final destination rather than through intermediate hops.
• 404s on internally linked pages waste crawl activity on dead ends. A 404 on a page that used to exist isn’t automatically a problem, but one reached through your own internal links is worth fixing at the link level, not just the page level.

Fix these in order of how many pages and how much internal link equity they affect, not in the order Search Console happens to list them.

FAQ

What is crawl budget in SEO?

Crawl budget is the number of pages Googlebot is willing and able to crawl on a site within a given time period, determined by the site’s server capacity (crawl rate) and how much Google wants to crawl the site’s content (crawl demand). It mainly becomes a practical concern on large or fast-changing sites.

Does robots.txt affect SEO rankings?

Robots.txt controls crawling, not ranking directly, but blocking crawl access to important pages can prevent Google from understanding their content, which affects how well they can rank. It doesn’t reliably prevent a page from appearing in search results if the page is linked to from elsewhere.

What’s the difference between noindex and nofollow?

Noindex controls whether a specific page can appear in search results at all. Nofollow controls whether authority passes through a specific link. They can be combined in the same meta tag but serve entirely separate functions.

How do I check crawl errors in Google Search Console?

Open the Crawl Stats report under Settings to see request volume, response codes, and average response time, and check the Page Indexing report under the Index section for URL-level status details, including server errors, redirect issues, and duplicate content flags.

Does my small website need crawl budget optimization?

Generally no. Google’s own guidance treats crawl budget as most relevant for sites with roughly ten thousand or more URLs, or sites that publish new content daily. Smaller, stable sites are usually crawled fully without any active management needed.

Where to start

Open Search Console’s Page Indexing report and check the “Discovered — currently not indexed” count first. If that number is meaningfully large, the fix usually isn’t a crawl-budget overhaul — it’s stronger internal links pointing at whatever’s sitting in that bucket.