Generate crawler directives for Googlebot, Bingbot, GPTBot & more
Presets:
robots.txt Preview
Advertisement
Robots.txt Best Practices for SEO
Robots.txt is a text file placed in your website's root directory that tells search engine crawlers which pages they can and cannot access. It's one of the most fundamental SEO tools — misconfiguring it can block Google from indexing your entire site.
When to Use Robots.txt
Block duplicate content: Prevent indexing of search results, filtered pages, and tagged archives
Save crawl budget: Block low-value pages so Googlebot focuses on important content
Staging environments: Disallow all bots on dev/staging subdomains
AI training crawlers: Control whether GPTBot, ClaudeBot, and others can scrape your content for model training
Resource management: Block crawlers from accessing large media files or API endpoints
Common Mistakes to Avoid
Don't block CSS/JS files — Google needs them to render your page properly
Robots.txt doesn't prevent indexing — use noindex meta tags for that
Always include your sitemap URL to help crawlers discover new pages
Test with Google Search Console's Robots Testing Tool before deploying
Be specific with paths — /admin also matches /administrator
AI Crawler Decisions
With the rise of AI training crawlers, many sites are choosing to block them. Consider the tradeoffs: blocking GPTBot prevents your content from being used in ChatGPT training, but it doesn't prevent your competitors from benefiting. Some publishers allow certain AI crawlers (like PerplexityBot for citation links) while blocking others.