robots.txt for Developers: Crawl Control, the noindex Trap, and AI Crawlers

Every site I audit has a robots.txt, and a surprising number of them are quietly doing the opposite of what the person who wrote them intended — usually because robots.txt gets treated as an indexing control when it is really a crawling control. Those are not the same thing, and the gap between them is where pages leak into Google’s index that were meant to be private, and where “just block it in robots.txt” fails to remove anything at all. This guide is a spoke under the SEO for engineers series: the file is four directives deep, but the behaviour behind them decides what every crawler — Googlebot and the AI bots now sitting alongside it — is allowed to fetch.

robots.txt is a plain-text file at the root of your domain that tells compliant crawlers which URL paths they may request. It governs crawling — whether a bot fetches a URL — not indexing, and not removal. A URL you disallow can still be indexed, and a page already in the index cannot be removed by blocking it.

Key takeaways

• robots.txt controls crawling, not indexing. Google is explicit that it “is not a mechanism for keeping a web page out of Google” — a disallowed URL can still be indexed if other pages link to it.
• Disallow is not noindex, and it cannot remove an indexed page. To keep a page out of results you need a noindex directive the crawler can actually reach — which means not blocking it in robots.txt.
• Precedence has two rules. Google matches the most specific rule by path length, and breaks ties in favour of the least restrictive rule — so a matching Allow beats an equally specific Disallow.
• AI crawlers are per-vendor, per-purpose switches. GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended are separate tokens for training, search, and answer generation — robots.txt is where you decide what to expose to each.
• Test against ground truth, not assumptions. Fetch the file as a specific bot and confirm each URL’s status in Search Console’s URL Inspection before you trust a rule.

What robots.txt actually controls

Start with the mental model, because almost every robots.txt mistake is a category error. The file is a set of instructions about fetching. When a compliant crawler wants a URL, it checks whether a rule disallows that path; if so, it doesn’t request the page. That is the entire mechanism. It says nothing about whether the URL can appear in search results, whether it gets a meta description, or whether an already-known page stays or goes.

This is the distinction Google states plainly: a robots.txt file “is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google.” Read that sentence twice, because it inverts how most teams use the file. Blocking a URL in robots.txt stops Google from reading it — it does not stop Google from listing it.

The consequence is the one that catches people out: “A page that’s disallowed in robots.txt can still be indexed if linked to from other sites.” When Googlebot finds a disallowed URL through an external link, it can’t fetch the content, but it can still index the URL itself — which is why you occasionally see a result with no description and the note “No information is available for this page.” That is a disallowed page that got indexed anyway, from the outside in.

So the correct use of robots.txt is narrow and it is about crawl economy: keep bots out of infinite faceted-search spaces, staging paths, internal search results, and low-value parameter URLs so they spend their crawl budget on pages that matter. It is not the tool for privacy, and it is not the tool for removal.

The syntax that matters

The file lives at https://yourdomain.com/robots.txt — one per origin, at the root, nowhere else. A crawler groups the rules by User-agent and applies the group that matches its token. Four directives carry almost all the weight:

User-agent: *
Disallow: /admin/
Disallow: /*?sessionid=
Allow: /admin/public-report.pdf

Sitemap: https://example.com/sitemap.xml

• User-agent names the crawler a group applies to. * is the fallback for any bot without its own group. A crawler obeys the single most specific group that matches its token, not all matching groups combined.
• Disallow blocks a path prefix. Disallow: /admin/ blocks everything under /admin/. An empty Disallow: blocks nothing (it is how you say “allow everything”).
• Allow carves an exception out of a broader Disallow, so you can block a directory but expose one file inside it.
• Sitemap points crawlers at your sitemap. Per Google’s specification, this field “isn’t tied to any specific user agent and may be followed by all crawlers” — it sits outside the user-agent groups and you can list several. Always include it; it is free crawl guidance.

Wildcards help with parameter noise: * matches any sequence of characters and $ anchors the end of a URL, so Disallow: /*.pdf$ blocks PDF URLs specifically.

Precedence is where intuition fails. When an Allow and a Disallow both match a URL, which wins? Google’s rule is two-stage: “crawlers use the most specific rule based on the length of the rule path,” and “in case of conflicting rules, including those with wildcards, Google uses the least restrictive rule.” So the longer, more specific path wins first; when two rules are equally specific, the Allow wins. In practice that means this pair exposes the quarterly folder while blocking the rest of /reports/:

User-agent: *
Disallow: /reports/
Allow: /reports/quarterly/

Two operational notes that save incidents. First, do not disallow the CSS and JavaScript that render your pages — blocking those resources stops Googlebot from rendering the page in its second pass, which is a self-inflicted JavaScript SEO failure I see after well-meaning “tidy up robots.txt” tickets. Second, a robots.txt that returns a 5xx error can cause Google to stop crawling the whole site, so treat the file’s availability as a production concern, not an afterthought.

The trap: Disallow is not noindex

▤ Screenshot slot Diagram of the crawl-versus-index split: a URL blocked by robots.txt that Google still keeps indexed, with crawl status and index status shown as two separate facts Drop the image at /assets/screenshots/robots-txt-for-developers/url-inspection-crawl-vs-index.png

This is the mistake that costs the most, so it gets its own section. If you want a page out of Google’s index, robots.txt is the wrong tool — and worse, using it actively prevents the right tool from working.

The mechanism is simple once you see it. To drop a page from the index you serve a noindex directive, either as a meta tag or an HTTP header:

<!-- In the page <head> -->
<meta name="robots" content="noindex">

# As a response header — also works for non-HTML files like PDFs
X-Robots-Tag: noindex

But a crawler can only obey a noindex it is allowed to read. Google’s guidance is unambiguous: “For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler.” Block the URL in robots.txt and Googlebot never fetches the page, never sees the noindex, and — if the URL is already indexed — leaves it in the index. You have locked the door on the very instruction meant to remove it.

So the correct removal recipe is the opposite of the instinct:

1. Keep the URL crawlable — do not disallow it in robots.txt.
2. Add noindex via meta tag or X-Robots-Tag header so the crawler sees it on the next fetch.
3. Wait for a recrawl, or request one, so Google drops the URL from the index.
4. Only then, if you also want to save crawl budget, disallow the path — after the noindex has done its work.

For genuinely sensitive content, neither directive is a security control: both robots.txt and noindex are requests that compliant bots honour and hostile ones ignore. Anything that must not be public belongs behind authentication, full stop.

Controlling AI crawlers

Here is where robots.txt earns fresh relevance in 2026. The same file that steers Googlebot is now the primary control surface for the AI crawlers behind ChatGPT, Claude, Perplexity, and Gemini — and each vendor ships multiple tokens split by purpose, so this is a set of deliberate choices, not one on/off switch. This is a direct lever on your generative engine optimization posture: you decide, per bot, whether your content feeds model training, appears in AI search answers, or both.

I verified every token below against the vendor’s own live documentation, because these strings are exact and case-sensitive — a typo means the rule silently does nothing.

Read the purpose column carefully, because it is what makes a nuanced policy possible. OpenAI’s own docs draw the line: GPTBot “is used to crawl content that may be used in training our generative AI foundation models,” while OAI-SearchBot “is used to surface websites in search results in ChatGPT’s search features.” Those are separate permissions. So the common developer intent — keep me out of training data, but keep me visible in AI answers — is expressible:

# Opt out of AI training…
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# …but stay visible in AI search and answers
User-agent: OAI-SearchBot
Disallow:

User-agent: Claude-SearchBot
Disallow:

User-agent: PerplexityBot
Disallow:

Three caveats keep this honest. First, Google-Extended is purely about generative use — Google states it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal,” so blocking it costs you nothing in classic Search. Second, the user-initiated agents behave differently from the automated crawlers: OpenAI notes that because ChatGPT-User actions are triggered by a person, “robots.txt rules may not apply,” and Perplexity says the same of Perplexity-User. A Disallow will not reliably stop a bot fetching a page because a user asked it to. Third, robots.txt compliance is voluntary; these named vendors document that they honour it, but blocking a token only works for crawlers that choose to obey — it is a policy signal, not enforcement.

Testing your robots.txt

A robots.txt rule is one string away from doing nothing, so I never trust one by reading it — I test it two ways.

Fetch the file as the bot in question. The fastest check is to request the raw file the way a specific crawler would, and read exactly what it receives:

curl -sA "GPTBot" https://example.com/robots.txt

That confirms the file is reachable, returns 200, and contains the group you expect. If the group you wrote for GPTBot isn’t in that output — or the file 404s — the rule you think you have doesn’t exist.

Confirm a URL’s real status in Search Console. The ground truth for Googlebot specifically is the URL Inspection tool: enter a URL and it reports whether the page is Blocked by robots.txt, crawlable, and indexed. This closes the loop that trips people up — a page can be disallowed (blocked from crawling) yet still indexed, and URL Inspection shows you both states separately rather than letting you assume one implies the other. Pair it with Search Console’s robots.txt report to see the exact file Google last fetched and any parse errors, so a stray character or bad encoding doesn’t silently void a rule.

Fold both checks into a recurring pass rather than a one-off — a robots.txt regression from a deploy is exactly the kind of silent failure the 12-phase SEO & GEO audit is built to catch before it costs you crawl coverage.

Where robots.txt fits

robots.txt is a small file with an outsized ability to help or harm, and the whole of using it well comes down to respecting one boundary: it controls crawling, and crawling is upstream of — but distinct from — indexing and AI exposure. Use it to steer crawl budget and to set your per-bot AI policy. Reach for noindex when you need a page gone from results, and remember the two cannot be stacked: a page you want de-indexed must stay crawlable long enough for the crawler to read the noindex.

That boundary is the same discipline the rest of the SEO for engineers series runs on — treating each of these as an engineering control with a specific, testable behaviour rather than a folk remedy you apply and hope. Get the crawl layer right, and everything downstream, from Google’s index to the answer engines quoting you, is working from the pages you actually meant to expose.

FAQ

Does robots.txt prevent a page from being indexed?

No. robots.txt controls crawling, not indexing. Google states directly that it “is not a mechanism for keeping a web page out of Google,” and a URL you disallow can still be indexed if other sites link to it — appearing as a bare URL with no description. To keep a page out of the index, use a noindex meta tag or X-Robots-Tag header, and make sure the page is not also blocked in robots.txt, or the crawler will never see the noindex.

Can I use robots.txt to remove a page that’s already in Google?

No. Disallowing an already-indexed URL prevents Google from re-crawling it, which means it can’t see any removal signal and the URL often stays in the index. The correct sequence is to keep the URL crawlable, add a noindex directive so Google reads it on the next fetch, wait for the recrawl to drop it, and only then disallow the path if you also want to save crawl budget. For urgent cases, use the Removals tool in Search Console alongside the noindex.

What’s the difference between Disallow and noindex?

Disallow in robots.txt tells a crawler not to fetch a URL. noindex tells a crawler not to list a URL in search results. They operate at different stages and don’t substitute for each other: Disallow can leave a page indexed-but-uncrawled, while noindex requires the page to be crawlable so the directive can be read. If your goal is “not in search results,” noindex on a crawlable page is the tool; if your goal is “don’t waste crawl budget here,” Disallow is.

How do I block AI crawlers like GPTBot and ClaudeBot with robots.txt?

Add a User-agent group for each token and Disallow: /. The main tokens are GPTBot (OpenAI training), ClaudeBot (Anthropic training), PerplexityBot (Perplexity search), and Google-Extended (Gemini training/grounding). Because these are split by purpose, you can block training crawlers while allowing search crawlers such as OAI-SearchBot and Claude-SearchBot to keep you visible in AI answers. Note that user-initiated agents like ChatGPT-User may ignore robots.txt, and compliance overall is voluntary — it is a policy signal, not enforcement.

Does blocking Google-Extended hurt my Google rankings?

No. Google documents that Google-Extended governs only whether your content may be used to train Gemini models and for grounding in AI applications, and that it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.” Blocking it opts you out of that generative use without any effect on classic Search crawling, indexing, or ranking.

Where does the Sitemap directive go in robots.txt?

Anywhere in the file, outside the User-agent groups. Per Google’s specification the Sitemap field “isn’t tied to any specific user agent and may be followed by all crawlers,” so it applies globally. Use an absolute URL, and you can list multiple Sitemap lines if you split your sitemap into several files.