Which AI crawlers can read your firm?
Paste your URL and see, bot by bot, what your robots.txt allows or blocks — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and more. If you've blocked the bots that cite, an engine can't recommend a firm it isn't allowed to read.
Remember: robots.txt is a request, not a lock. A blocked bot is asked not to crawl; a well-behaved one complies, but nothing forces it to. Blocking is a legitimate choice — just know that you can't be cited by an engine you've blocked.
Want the full picture — schema, readability, sitemap, and more? Run the free AEO checker
What are GPTBot, ClaudeBot, and PerplexityBot?
They’re the crawlers the big AI engines send to read the web. GPTBot feeds OpenAI, ClaudeBot feeds Anthropic’s Claude, PerplexityBot feeds Perplexity, and Google-Extended governs whether Google uses your pages for Gemini and AI Overviews. Each is named in your robots.txt by its user-agent, so you allow or block them one line at a time.
Should a law firm block AI crawlers?
It’s a legitimate choice, and some firms make it on principle. But be clear about the trade: an engine can’t cite or recommend a firm whose pages it isn’t allowed to read. If you want to show up in ChatGPT or an AI Overview, the crawlers behind them need in. Blocking training bots like CCBot is a separate decision from blocking the ones that cite.
Does robots.txt actually stop AI crawlers?
Not by force. robots.txt is a request, not a lock — it asks well-behaved crawlers not to fetch certain paths, and the major ones honor it. But nothing technically prevents a crawler from ignoring it. If you need to truly block a bot, that’s server- or firewall-level work; robots.txt is a polite sign, not a gate.
How often does the AI bot list change?
Often enough to matter. Engines add, rename, and split their crawlers — OpenAI alone runs GPTBot, OAI-SearchBot, and ChatGPT-User for different jobs. This tool’s list is dated so you know how current it is; if a bot you care about isn’t here, check the engine’s own docs for its latest user-agent.