Before fetching any URL, a well-behaved crawler asks the host's robots.txt a yes/no question: may this user-agent fetch this path? The answer comes from a short sequence of rules most people never fully learn:
User-agent line names the crawler. If none matches, fall back to the * group. If a named group exists, the * group is ignored for that bot — the groups don't merge.Disallow: /admin blocks /admin, /admin/login, and /admin-2024 alike.That third step is the one that surprises people: Disallow: /search with Allow: /search/about lets /search/about through, because its rule is longer. Wildcards extend this — /*.pdf$ blocks every URL ending in .pdf (the * matches anything, $ pins the end), but not /report.pdfx.
The fastest path is a validator that runs the same matching logic crawlers do. Google Search Console has a robots report with a URL checker, but it only knows Googlebot. For everything else — Bing, the AI crawlers, or a file you haven't deployed yet — an in-browser validator like our robots.txt validator parses the file locally, tests any URL for any user-agent, names the exact rule and line that decided the outcome, and flags structural mistakes (rules before the first User-agent, non-absolute sitemap URLs, duplicate groups). Since it runs in your browser, you can test drafts before they ship.
Manually, the same steps work on paper for small files: pick the group, list matching rules, longest path wins. The manual route fails precisely where it matters, though — files with 50+ rules, wildcards, and multiple groups — which is how accidental blocks survive for months.
Each major AI company documents its own crawler tokens, and each token needs its own group in your file:
| Token | Operator | Purpose |
|---|---|---|
GPTBot | OpenAI | Model training |
OAI-SearchBot | OpenAI | ChatGPT search |
ClaudeBot | Anthropic | Model training |
Claude-SearchBot | Anthropic | Claude search |
Google-Extended | Gemini training & grounding | |
PerplexityBot | Perplexity | Answer engine |
Applebot-Extended | Apple | Apple Intelligence training |
CCBot | Common Crawl | Open training dataset |
Two consequences worth internalizing. Blocking a training crawler doesn't block the same company's search crawler — GPTBot and OAI-SearchBot are independent tokens, and blocking Google-Extended leaves Google Search untouched. And robots.txt compliance is voluntary; a crawler that ignores the protocol won't be stopped by any rule you write. The validator's dropdown includes the common AI tokens so you can test each group you write.
Google re-fetches robots.txt about once a day and uses its cached copy until the next successful fetch, so a fixed rule usually takes effect within 24 hours. Bing follows a similar rhythm. The asymmetry is recovery: if a bad rule blocked pages long enough for them to drop out of the index, re-inclusion requires re-crawling and re-processing, which ramps over days to weeks after the rule is fixed. That's the argument for validating before deploying — the cost of a mistake is paid at recovery speed, not at fix speed.
Disallow: /admin also blocks /admin-guide. Add a trailing slash (/admin/) when you mean the directory, or an Allow: exception for the escapes.* group for that bot. Write User-agent: GPTBot with one permissive line and GPTBot inherits nothing from * — it gets exactly what its own group says.robots.txt controls crawling; noindex controls indexing. They fail in opposite directions when confused. A robots-blocked page can still appear in results as a bare URL if enough links point at it, because Google can index what it can't crawl. And a page you want deindexed must be crawlable and carry the noindex tag — block it in robots.txt and Google never sees the tag. Rule of thumb: robots.txt for crawl budget and private folders, noindex for pages that exist publicly but shouldn't list in search.
Paste the file, pick a URL and a crawler — Googlebot, GPTBot, or any token — and see the exact rule that decides.
Open the Robots.txt Validator →Matching is mechanical: one group per bot, longest matching path wins, wildcards do what they look like they do. Validate before deploying, remember the ~24-hour cache, and keep an up-to-date generator on hand when starting fresh rather than editing a file you don't fully trust. For the neighboring plumbing — making sure pages carry the right tags once crawlers arrive — the meta tag generator and schema generator cover the next steps.
Google re-fetches robots.txt roughly once every 24 hours and caches that copy until the next fetch, so most changes land within a day. Bing runs its own similar refresh cycle. Recovery is slower than the change: pages blocked long enough to fall out of the index need to be re-crawled and re-processed over the following days to weeks.
Only the crawlers that honor it, and each under its own token: GPTBot and OAI-SearchBot for OpenAI, ClaudeBot and Claude-SearchBot for Anthropic, PerplexityBot, Google-Extended for Google's AI training, Applebot-Extended, and CCBot. Add a group per token with the rules you want. Blocking training crawlers doesn't block search crawlers, and compliance is voluntary on the crawler's side.
Yes — as a bare URL. Google can index a page it has never crawled if other pages link to it strongly enough; without crawling it has no title or snippet, so the listing shows the URL alone. To remove a page from results, let it be crawled and use a noindex tag or header instead — a robots.txt block actually prevents Google from ever seeing the noindex.