Is your site blocking AI crawlers?
GPTBot, ClaudeBot, PerplexityBot, and Google-Extended each read your robots.txt on their own terms. Paste a URL and we’ll check which crawlers can actually reach it.
Free · no sign-up · reads your live robots.txt and page headers
What this checks — and why it matters
Every AI crawler reads your robots.txt against its own user-agent name. A blanket rule aimed at one bot doesn’t automatically cover the others — it’s common for a site to allow Googlebot, forget GPTBot, and quietly block ClaudeBot with an old rule nobody remembers writing.
This checker reads your live robots.txt and evaluates it for Googlebot, Bingbot, GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, CCBot, Applebot-Extended, and Bytespider — the standard longest-match rule resolution, per crawler. It also checks meta robots and the X-Robots-Tag header for noindex/noai directives, and runs a render check: AI crawlers don’t execute JavaScript, so a page that looks complete in a browser can be nearly empty to a bot.
Percy runs checks like this on every page, weekly, and tells you what to change. Connect your site.
Frequently asked questions
What is an AI crawler?
An AI crawler is a bot that fetches web pages to train or ground an AI model's answers — GPTBot and OAI-SearchBot (OpenAI), ClaudeBot and anthropic-ai (Anthropic), PerplexityBot (Perplexity), Google-Extended (Google's AI features), and others. They're separate from the search crawlers (Googlebot, Bingbot) that build the classic search index, and a site can allow one while blocking another.
How do I know if GPTBot or ClaudeBot is blocked on my site?
Check your robots.txt for a Disallow rule under that bot's User-agent group (or the wildcard * group, which applies to any bot without its own group). This tool reads your live robots.txt and evaluates it for each major crawler, so you don't have to parse the rules by hand.
Why would a page look fine in a browser but be invisible to AI crawlers?
AI crawlers generally don't execute JavaScript. If a page's content is added client-side after the initial HTML loads, a person sees it fine but a crawler sees an empty shell. This tool includes a render check that flags exactly that gap.