AI companies run different bots for different jobs, and robots.txt lets you treat each one separately. Training crawlers collect content that may be used to train models. Search crawlers index pages so an AI search product can cite them. User fetchers retrieve a page because a person asked the assistant to look at it. Blocking a training crawler doesn't remove you from AI search, but blocking a search crawler usually does.
Many sites got this wrong in 2024 and 2025 by pasting a long "block all AI" list into robots.txt. The result was no AI training, which they wanted, and no AI citations, which they didn't.
The main bots, by job
From each company's own documentation:
OpenAI (bots page)
- GPTBot: training.
- OAI-SearchBot: ChatGPT search. Blocking it removes you from ChatGPT search answers.
- ChatGPT-User: user-initiated fetches. OpenAI says robots.txt rules may not apply.
Anthropic (crawler article)
- ClaudeBot: training.
- Claude-SearchBot: indexing for search results in Claude.
- Claude-User: fetches pages when a Claude user's question needs web access.
- Anthropic says its bots respect robots.txt and supports
Crawl-delayfor ClaudeBot.
Perplexity (crawler docs)
- PerplexityBot: surfaces and links sites in Perplexity search. Perplexity says it isn't used to train foundation models.
- Perplexity-User: user-requested fetches. Perplexity says it generally ignores robots.txt because a person asked for the page.
Google (common crawlers)
- Googlebot: Search, including AI Overviews and AI Mode.
- Google-Extended: a robots.txt token (not a separate bot) that controls Gemini training and grounding in Gemini Apps and Vertex AI. Google says it doesn't affect Search inclusion or ranking.
Apple documents Applebot-Extended in the same way: a token that controls whether content crawled by Applebot can be used to train Apple's models, separate from appearing in Apple's search features.
Three sensible policies
1. Visible everywhere, no training
For most businesses that want AI visibility but don't want to donate content to training:
User-agent: GPTBotthenDisallow: /User-agent: ClaudeBotthenDisallow: /User-agent: Google-ExtendedthenDisallow: /User-agent: Applebot-ExtendedthenDisallow: /User-agent: *thenAllow: /
Each pair goes on its own two lines in the file, with a blank line between groups.
OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot fall under the final rule and stay allowed.
2. Fully open
If you sell services and want the widest possible exposure, including in what models learn about your category, allow everything. A professional services firm usually benefits from models knowing it exists.
3. Protect a section
Keep the site open but block a members area or a pricing configurator with path rules for every bot. Remember robots.txt is a request, not access control. Anything truly private needs a login.
Things robots.txt can't do
- It can't stop user fetchers that don't honour it. OpenAI and Perplexity both say their user-initiated agents may ignore it.
- It can't remove content a model has already learned. Training opt-outs apply going forward.
- It can't override your CDN. If Cloudflare or a security plugin blocks bots at the edge, robots.txt is never read. Since July 2025 Cloudflare has offered new domains a default AI block, so check your dashboard.
How to check it's working
- Open
yoursite.com/robots.txtin a browser and read it line by line. WordPress sites often generate it dynamically, and plugins can add rules you didn't write. - Search your server logs for each user agent and confirm the status codes are 200, not 403. Our guide to reading server logs for AI crawlers walks through it.
- Verify bots against the IP lists each company publishes. User agents are easy to fake.
- Allow about a day for changes to register. OpenAI and Perplexity both mention a window of roughly 24 hours.
A note on WordPress and plugins
On WordPress, robots.txt is often virtual: there's no file on disk, and WordPress builds it on request. SEO plugins add their own rules, and some security plugins add bot rules at the server level that never appear in robots.txt at all. If what you see at /robots.txt doesn't match what you expect, check your SEO plugin settings first, then your security plugin, then your host's control panel.
Our default recommendation
Allow search crawlers, decide on training bots as a business choice, and audit the CDN. It takes about an hour, and it's the first item on our GEO service checklist because nothing else matters if the bots can't get in.



