What this checks
- What robots.txt says to each AI search crawler (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Claude-User)
- Which AI training crawlers are opted out (GPTBot, ClaudeBot, Google-Extended and others)
- Whether the server or firewall turns AI crawlers away even when robots.txt allows them
- Whether the site has an llms.txt file
Free, no signup. We read public DNS records and public pages only, and never change anything.
Questions
What's the difference between AI search crawlers and AI training crawlers?
Search crawlers (like OAI-SearchBot or PerplexityBot) fetch pages so an assistant can quote and link them in answers. Training crawlers (like GPTBot or ClaudeBot) collect text to train future models. You can block training and still allow search, and many sites do.
If I block GPTBot, will my site disappear from ChatGPT?
No. GPTBot is OpenAI's training crawler. ChatGPT's search uses OAI-SearchBot, and pages a user asks about are fetched as ChatGPT-User. Blocking GPTBot only opts you out of training.
My robots.txt allows AI crawlers. Why does it say they're blocked?
Some hosts and CDNs (Cloudflare has a one-click setting) block AI crawlers at the firewall, before robots.txt is ever read. We request your homepage identifying as those crawlers and compare the answer with an ordinary request.
Does robots.txt control Google's AI Overviews?
Not separately. AI Overviews use the normal Googlebot index, so they follow the same rules as Google Search. Google-Extended only controls Gemini training.
Check everything at once
The full SiteScore report runs this and the other free checks, plus a real browser test of speed, accessibility, mobile, SEO and AI readiness, and shows what to fix first. About 30 seconds.
Scan a whole site