Archive
Tag: ai crawlers
-

How ChatGPT Search Picks the Sources It Cites
ChatGPT search can only cite pages it can find and fetch. This post covers the crawler it relies on, how it chooses sources and…
-

robots.txt for AI Crawlers: Training Bots, Search Bots and User Fetchers
Blocking every AI bot can quietly remove you from AI search. How to tell training crawlers from search crawlers, and a robots.txt that matches…
-

How to Get Your Site Cited by Perplexity
Perplexity runs its own index through PerplexityBot and cites sources inline. Allow the bot, publish specific answers and keep pages fresh.
-

Reading Your Server Logs for AI Crawler Activity
Server logs show which AI bots visit, what they request and whether they get blocked. How to find them, verify them and act on…