What robots.txt does and does not do
Robots.txt is the crawler-enforcement file you serve at yourdomain.com/robots.txt to tell well-behaved bots which paths they may fetch. It is permission, not appearance. It controls whether a bot retrieves a URL at all. It does not control how a URL renders in a SERP once retrieved - for that, you need structured data in the page head.
A clean read on the four limits worth naming up front:
- Not a Google ranking factor. Robots.txt does not move you up or down in search results on its own. It only changes what Googlebot reads.
- Will not de-index already-indexed pages without noindex. If a URL is already in Google's index and you Disallow it, Google keeps the URL in the index (often with a "no information available" snippet) because it can no longer crawl the page to see a noindex tag.
- Will not stop bots that ignore the standard. The Robots Exclusion Protocol is voluntary. Scrapers, content-theft bots, and some AI training crawlers ignore it entirely; for those, you need IP-level blocking via your CDN or WAF.
- Will not influence how the page renders in SERPs. Rich results, breadcrumbs, FAQ accordions in the result snippet - those come from JSON-LD schema, not robots.txt.
The 2026 AI-bot map: retrieval vs training
The 2026 AI-crawler landscape splits into two functional buckets that classic robots.txt advice does not distinguish. Training bots ingest your content into model weights. Retrieval bots fetch live when a user asks a question and may cite your URL in the response. Blocking training does not stop retrieval; blocking retrieval kills the AI-citation channel entirely.
| User-agent | Operator | Purpose | Default |
|---|---|---|---|
| GPTBot | OpenAI | Training (GPT models) | Allow for reach |
| OAI-SearchBot | OpenAI | Retrieval for ChatGPT Search citations | Allow |
| ChatGPT-User | OpenAI | On-demand fetch from a user prompt | Allow |
| ClaudeBot | Anthropic | Anthropic web fetch (retrieval and training) | Allow |
| PerplexityBot | Perplexity | Retrieval for Perplexity citations | Allow |
| Google-Extended | Opt-out for Gemini and Vertex AI training | Allow for reach | |
| CCBot | Common Crawl | Open corpus (used widely as LLM training data) | Allow |
| Applebot-Extended | Apple | Apple Intelligence and on-device AI | Allow |
| Bytespider | ByteDance | ByteDance and TikTok ingestion | Often blocked |
| Amazonbot | Amazon | Powers Alexa and Amazon AI features | Allow |
| Meta-ExternalAgent | Meta | Meta AI training crawler | Allow |
The strategic trade-off: blocking training bots protects your content from being baked into model weights, but the model can still cite you via retrieval. Blocking retrieval bots is the move that severs the AI-citation channel. If you allow AI crawlers, pair this with an llms.txt file to guide them to your best content - robots.txt is the permission layer, llms.txt is the curation layer. The broader citation-side playbook for ChatGPT, including the corpus-vs-search distinction, is documented here.
Common robots.txt mistakes that de-index sites
Most robots.txt accidents are simple. A single character on a single line can pull an entire site out of Google's index by morning. These are the patterns that cause real damage:
- Disallow: / on production. The classic post-launch oops. A staging robots.txt that blocks everything ships to prod with the deploy. Within 48 hours, Googlebot reads it and starts deprioritizing your URLs.
- Blocking /wp-admin/admin-ajax.php. Some WordPress security plugins write this rule. It breaks JavaScript on the front-end, which can hurt Core Web Vitals and mobile-friendly scoring.
- Disallow on a path you also try to noindex via meta tag. Google cannot see the noindex tag if it cannot crawl the page. Use one or the other - not both - per URL.
- Blocking JS or CSS folders. Modern Googlebot renders pages with full JavaScript. Blocking script or stylesheet directories kills mobile-friendly evaluation.
- Trusting User-agent: * to cover named bots. Bots that target their own user-agent (GPTBot, ClaudeBot, AhrefsBot) read their own block first and ignore the wildcard rule. To block them, write rules with their specific UA.
How to deploy this robots.txt
Five steps. Under five minutes once you have the file.
- Copy or download the file using the buttons above the output panel.
- Place it at your site root. Next.js: public/robots.txt or build dynamically with app/robots.ts. WordPress: Yoast or RankMath robots.txt editor, or FTP to document root. Vercel and Cloudflare Pages: /public folder. nginx: alias inside the server block. Apache: DocumentRoot.
- Deploy. Ship the change. The file becomes reachable at https://yourdomain.com/robots.txt.
- Verify with curl. Run curl -I https://yourdomain.com/robots.txt and confirm a 200 response with Content-Type text/plain.
- Test in Search Console. Open the robots.txt report in Google Search Console to confirm Googlebot's read of the file matches what you intended.
Where this fits in the AI-search file stack
Robots.txt is the enforcement layer. llms.txt is the curation layer above it - the file that tells LLMs which of your canonical URLs to prioritize once they have permission to fetch.JSON-LD schema is the entity-resolution layer that decides whether you get cited once a bot reaches you. All three are necessary; none of them alone is sufficient. The deeper read on file-stack interaction lives in the llms.txt complete guide. And the heaviest AI-citation lever, despite what the file-format discourse suggests, is still off-site: the link-building side of crawl velocity earns the inbound signals that pull crawlers in faster. The full free SEO tools index lives at /tools.