Free pdf guide

robots.txt for AI Crawlers Cheat Sheet

Search bots and training bots are different, and blocking the wrong one removes you from AI answers. This cheat sheet lists the main AI crawlers, what each one does and what blocking it means, with copy-ready policies.

What’s inside

In this guide.

  • A table of 12 crawlers and tokens from Google, OpenAI, Anthropic, Perplexity, Microsoft and Apple
  • Three copy-ready policies
  • A short test list to confirm your rules work

How to use it

Getting started.

  1. Open your current robots.txt.
  2. Choose policy A or B.
  3. Check your CDN or firewall bot settings match.
  4. Confirm search bots get 200 responses in your logs.

Want it done for you?

Related services.

We use this guide on client projects. If you would rather hand the work over, see GEO: generative engine optimisation and SEO services, or send us a brief.

More free resources

You might also need.

PDF guide, 47 KB

Answer-First Content Guide

Seven rules, before-and-after examples and a one-page content brief template.

Excel workbook, 19 KB

Technical SEO Audit Workbook

68 checks across 9 areas, scored by impact and effort so you know what to fix first.

Excel workbook, 22 KB

Local SEO Citation Tracker (South Africa)

Keep your name, address and phone consistent across Google, Bing, Apple and South African directories.

Next step

Have a project in mind?

Send a short brief. We reply within one business day with questions and a suggested first step.