Free pdf guide
robots.txt for AI Crawlers Cheat Sheet
Search bots and training bots are different, and blocking the wrong one removes you from AI answers. This cheat sheet lists the main AI crawlers, what each one does and what blocking it means, with copy-ready policies.
What’s inside
In this guide.
- A table of 12 crawlers and tokens from Google, OpenAI, Anthropic, Perplexity, Microsoft and Apple
- Three copy-ready policies
- A short test list to confirm your rules work
How to use it
Getting started.
- Open your current robots.txt.
- Choose policy A or B.
- Check your CDN or firewall bot settings match.
- Confirm search bots get 200 responses in your logs.
Related reading
Go deeper.
Want it done for you?
Related services.
We use this guide on client projects. If you would rather hand the work over, see GEO: generative engine optimisation and SEO services, or send us a brief.
More free resources
You might also need.
PDF guide, 47 KB
Answer-First Content Guide
Seven rules, before-and-after examples and a one-page content brief template.
Excel workbook, 19 KB
Technical SEO Audit Workbook
68 checks across 9 areas, scored by impact and effort so you know what to fix first.
Excel workbook, 22 KB
Local SEO Citation Tracker (South Africa)
Keep your name, address and phone consistent across Google, Bing, Apple and South African directories.
Next step
Have a project in mind?
Send a short brief. We reply within one business day with questions and a suggested first step.