Parse the file, test any URL, see the rule and line that decided it — in your browser
| User-agent token | Operated by | Used for | If you block it |
|---|---|---|---|
GPTBot | OpenAI | Training data for models | ChatGPT search still crawls separately |
OAI-SearchBot | OpenAI | ChatGPT search results | Pages stop appearing in ChatGPT search |
ChatGPT-User | OpenAI | Fetches triggered by user actions | Only user-initiated fetches stop |
ClaudeBot | Anthropic | Training data for Claude | Claude search crawls separately |
Claude-SearchBot | Anthropic | Claude search results | Pages stop appearing in Claude answers |
Google-Extended | Gemini training & grounding | Google Search is unaffected | |
PerplexityBot | Perplexity | Perplexity search & answers | Pages drop from Perplexity results |
Applebot-Extended | Apple | Apple Intelligence training | Siri/Spotlight search unaffected |
CCBot | Common Crawl | Open dataset many models train on | Pages leave future crawls |
Meta-ExternalAgent | Meta | Meta AI agent fetching | Meta AI can't fetch your pages |
Amazonbot | Amazon | Alexa & AI assistants | Amazon AI features lose access |
Bytespider | ByteDance | Training data (TikTok family) | Blocks well-behaved fetches only |
Each token is a separate robots.txt group: a rule under User-agent: GPTBot doesn't affect OAI-SearchBot, and blocking Google-Extended never touches Google Search's Googlebot. Check the operator's crawler documentation before relying on a block for anything legally sensitive — enforcement is voluntary and varies.
| Directive | Googlebot | Bingbot | Notes |
|---|---|---|---|
User-agent / Disallow / Allow | Yes | Yes | The core of RFC 9309 — universally supported |
* and $ wildcards in paths | Yes | Yes | Longest matching path wins; $ anchors to the end |
Sitemap: (absolute URL, anywhere in file) | Yes | Yes | Multiple lines allowed, each must be absolute |
Crawl-delay: seconds | Ignored | Yes | Yandex also honors it; use Search Console rate settings for Google |
Rules before the first User-agent | No effect | No effect | A record starts at User-agent — stray rules apply to nothing |
| File over 500 KiB | Truncated | Varies | Google treats over-limit files as fully allowing everything |
| HTTP 410/401/403 on robots.txt | Full block | Varies | 404/5xx errors read as "allow all" instead |
This page parses your robots.txt with robots-parser 3.0.1, a spec-compliant implementation of the Robots Exclusion Protocol (RFC 9309) that supports the * and $ wildcards Google and Bing use. Parsing happens in your browser; the file you paste never goes anywhere.
A crawler reads only the group whose User-agent matches its own token — or, if none matches its full name, the * group. Within that group, the single most specific matching rule decides the outcome: an Allow: /search/about beats Disallow: /search for /search/about, because its path is longer. A dedicated group for a bot replaces the * group for that bot entirely; it doesn't merge with it. Those three facts explain almost every "why is this URL blocked" mystery.
yoursite.com/robots.txt) and set the site URL so relative paths resolve.The sample loaded above is 11 lines. Check /admin/settings as Googlebot: line 2's Disallow: /admin matches by prefix, so it's blocked. Check /docs/report.pdf: line 5's Disallow: /*.pdf$ matches because the * swallows "/docs/report" and the $ pins ".pdf" to the end — but /docs/report.pdfx escapes, since its tail isn't .pdf. And /search/about beats line 3's block via line 4's longer Allow path, the specificity rule in action. Switch the user-agent to GPTBot and everything flips: line 9's Disallow: / blocks the whole site, and the * group's rules stop mattering because GPTBot has its own group. Crawl-delay 10 applies to * only — the GPTBot group declares none.
Paste your robots.txt into the validator, enter the URL and the user-agent you care about (Googlebot, GPTBot, or * for the general rules), and read the verdict. The tool reports allowed or blocked plus the exact line that decided it, with wildcards (* and $) resolved the way real crawlers resolve them. Manually, you'd find the most specific matching rule for that user-agent; the validator automates exactly that, including the rule that a group for a specific bot replaces the * group entirely.
Yes. AI companies run their own crawlers that honor robots.txt under their own user-agent tokens: GPTBot and OAI-SearchBot for OpenAI, ClaudeBot and Claude-SearchBot for Anthropic, PerplexityBot for Perplexity, Google-Extended for Google's AI training (separate from Googlebot), Applebot-Extended for Apple, and CCBot for Common Crawl. Blocking a training crawler does not block search crawlers, and vice versa: Google-Extended can be blocked while Google Search keeps indexing you. Each token needs its own rule, or a Disallow: / under User-agent: GPTBot style group.
Google typically re-fetches robots.txt about once every 24 hours and caches it for the lifetime of that fetch, so a change usually lands within a day. Bing behaves similarly with its own refresh cycle. If you fix a rule that was accidentally blocking pages, expect re-crawling to ramp over the following days, not instantly — and pages that were blocked long enough can drop out of the index and need time to return.
robots.txt blocks crawling; the noindex meta tag blocks indexing. A page Disallowed in robots.txt can still appear in results if other pages link to it (Google can index without crawling, showing a bare URL), while a crawled page with a noindex tag is removed from results but its content is still readable by the crawler. Blocking crawling also blocks the crawler from ever seeing a noindex tag, which is why robots.txt is the wrong tool for deindexing existing pages.
The four classics: rules placed before the first User-agent line apply to nothing; a more specific Allow rule wins over a shorter Disallow (longest path wins); a group written for a specific bot replaces the * group for that bot rather than adding to it; and Google ignores Crawl-delay entirely. The validator flags all four patterns — check the warnings list before assuming a crawler is misbehaving.
At the root of the host, on port 80 or 443, and only one per host: https://example.com/robots.txt. Subdomains each get their own file. Google stops reading after 500 KiB and treats an over-limit file as if it were fully open, so oversized files silently allow everything. HTTP 401-403 responses are treated as a full disallow for Google; 404 and 5xx errors are treated as allowing everything.