Your server logs are the only place that shows exactly which AI crawlers visit your site, which pages they request and whether your server let them in. Filter the logs by user agent for bots like OAI-SearchBot, GPTBot, ClaudeBot, Claude-SearchBot and PerplexityBot, verify the requests against each company's published IP ranges, then check status codes and request volume. Blocked search bots and wasted crawling on junk URLs are the two problems you'll find most often.
Where to find the logs
- cPanel or similar hosting: usually under "Raw Access" or "Access Logs".
- Managed WordPress hosts: many offer log downloads in the dashboard or on request.
- VPS or cloud servers: typically
/var/log/nginx/access.logor/var/log/apache2/access.log. - Cloudflare and other CDNs: the CDN sees requests your server never does. If it blocks a bot, your server logs won't show it, so check the CDN's security events too.
Keep at least 30 days. AI crawler activity is bursty, and a week can be misleading.
The user agents to look for
From each company's documentation:
- OpenAI (bots page): GPTBot (training), OAI-SearchBot (ChatGPT search), ChatGPT-User (user-initiated fetches), OAI-AdsBot (checks pages submitted as ads).
- Anthropic (crawler article): ClaudeBot (training), Claude-SearchBot (search), Claude-User (user-initiated).
- Perplexity (crawler docs): PerplexityBot (search), Perplexity-User (user-initiated).
- Google: Googlebot covers Search including AI Overviews. Google-Extended is a robots.txt token, not a separate crawler, so you won't see it in logs.
- Microsoft: Bingbot.
A quick command-line pass
On a Linux server, these give a fast overview:
- Count requests per AI bot:
grep -Eo "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User" access.log | sort | uniq -c - See status codes for one bot:
grep "OAI-SearchBot" access.log | awk '{print $9}' | sort | uniq -c - Top pages requested by a bot:
grep "Claude-SearchBot" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
Log formats vary, so the field numbers in awk may need adjusting. Spreadsheet users can import the log as space-separated text and filter the user agent column.
Verify before you trust
User agent strings are trivial to fake. Scrapers often pretend to be GPTBot or Googlebot. Each company publishes IP lists:
- OpenAI: separate JSON files for GPTBot, OAI-SearchBot and ChatGPT-User, linked from its bots page.
- Anthropic: a list at
claude.com/crawling/bots.json. - Perplexity: JSON lists for each agent, linked from its crawler docs.
- Google and Bing: reverse DNS verification, documented by each.
Treat unverified requests as unknown bots. If they're heavy, rate-limit them.
What to look for
Blocked search bots
403 or 429 responses to OAI-SearchBot, Claude-SearchBot or PerplexityBot mean you're probably invisible in that product. Usually the cause is a security plugin or CDN rule, not robots.txt.
Crawling that goes nowhere
Bots requesting thousands of filter URLs, search result pages, calendar pages or old parameters waste their attention. Fix with canonical tags, cleaner internal links and robots.txt rules for genuinely useless paths.
Missing important pages
If your main service pages never appear in AI bot requests, check internal linking and your XML sitemap. Bots find pages the same way people do: through links.
Server strain
If a training crawler is hitting you hard, Anthropic supports Crawl-delay for ClaudeBot, and you can disallow training bots without affecting search bots. Our robots.txt guide shows how.
Connect logs to results
Logs tell you about access and attention, not outcomes. Pair them with:
- Search Console's generative AI reports for Google AI impressions.
- Bing Webmaster Tools' AI Performance report for Copilot citations.
- GA4 referrals from chatgpt.com, perplexity.ai and claude.ai.
A page that search bots fetch often but that never earns citations usually has a content problem, not a technical one. Our post on measuring AI search visibility puts all of these together.
Make it routine
Check logs monthly, or set up an alert for 4xx responses to known search bots. It takes minutes once the queries are saved, and it catches the silent blocks that otherwise go unnoticed for months. Log review is part of every technical audit in our GEO service.



