Citare Tools · Free
AI Robots.txt Checker
Paste a URL — we fetch robots.txt, parse every User-agent group, and report allow/block across 25+ AI crawlers (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended, …).
Free. No signup. Cached 24h.
Frequently asked
How do I check if my robots.txt blocks GPTBot, ClaudeBot, and PerplexityBot?
Paste your URL into the form above. Citare's AI Robots.txt Checker fetches your site's robots.txt, parses every User-agent group, and reports the allow/block status for 25+ AI crawlers — including GPTBot (OpenAI training), ChatGPT-User and OAI-SearchBot (ChatGPT live grounding), ClaudeBot (Anthropic), PerplexityBot, Perplexity-User, Google-Extended (Gemini training opt-in), GoogleOther (Google AI fetchers), and Bingbot (used by ChatGPT and Copilot grounding). For each blocked bot, the tool surfaces the exact line in your robots.txt that caused the block.
How do I unblock GPTBot, ClaudeBot, or PerplexityBot in robots.txt?
Add an explicit User-agent group for each AI bot you want to allow, with an Allow: / rule. The most common cause of accidental AI-bot blocking is a wildcard User-agent: * group with restrictive Disallow lines that catches every crawler without its own explicit group. When the Citare tool finds a block, copy paste-ready Allow lines for every blocked bot into your robots.txt and re-run the check to verify.
Should I block GPTBot, or allow it?
For most sites that benefit from organic discovery, allow GPTBot. Blocking GPTBot opts your content out of OpenAI's training corpus but does not improve ChatGPT-citation outcomes — ChatGPT's live grounding uses different bots (OAI-SearchBot, ChatGPT-User, Bingbot). Blocking is the right choice only when content is paywalled, copyrighted in a way that disallows AI training, or genuinely sensitive (legal, medical, internal documentation). For brand visibility in AI search, the rule of thumb is: block GPTBot only if you have made a deliberate training-opt-out decision; otherwise allow.
Will blocking GPTBot stop ChatGPT from citing my site?
No. Blocking GPTBot stops OpenAI from training on your content but does not stop ChatGPT from citing your site through grounded search. ChatGPT's live grounding fetches via OAI-SearchBot, ChatGPT-User, or Bingbot — those are separate user-agents with their own robots.txt rules. To remain visible in ChatGPT's grounded answers, allow OAI-SearchBot, ChatGPT-User, and Bingbot. To opt out of training only, block GPTBot but keep the others allowed. The same training-vs-grounding split applies to Anthropic (ClaudeBot trains; live Claude grounding has no separate published bot) and Google (Google-Extended controls Gemini training; live Gemini grounding uses Googlebot rules and JavaScript-rendered HTML access).
How is Citare's AI Robots.txt Checker different from Google's robots.txt tester?
Google Search Console's robots.txt tester only validates rules against Googlebot. Citare's tool validates against 25+ AI crawler user-agents in one pass — GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, and 17+ others. Each AI search platform fetches via its own user-agent string, so a robots.txt that's clean for Googlebot can still block ChatGPT, Claude, or Perplexity entirely. The Citare tool catches that gap.