AI-crawler robots.txt generator.
Pick a stance for the crawlers that matter and get a valid robots.txt — every line explained. The answer-engine crawlers are the ones that can cite your firm in an AI answer; the training-only crawlers just collect text to train models. Blocking the second group does nothing about the first.
Save this as robots.txt at your site root, so it loads at yourfirm.com/robots.txt. It replaces your current robots.txt — if you already have one with other rules, merge these lines in rather than overwriting. Want to see what your live robots.txt allows today? Run the AI crawler access checker.
What does this robots.txt generator do?
You pick two things — whether to let AI answer engines read your site, and whether to let training-only crawlers collect your text — and it writes a valid robots.txt with every line explained. It runs entirely in your browser, and the file it makes is plain text you paste at your site root. No account, no scan, no cost.
Should a law firm block AI crawlers?
Usually not the ones that matter. The answer-engine crawlers — GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended — are how an AI answer can cite your firm, and a crawler you’ve blocked can’t cite you. Blocking those to "protect" your content also hides you from the tools your clients now ask. The one defensible block is the training-only crawlers, and that’s a values choice, not a visibility one.
What’s the difference between answer-engine and training crawlers?
Answer-engine crawlers fetch your pages to answer a live question — that’s the moment a firm can get named. Training crawlers (CCBot, Bytespider) collect text in bulk to train models, with no live answer and no citation. A lot of "block AI" robots.txt templates only cover the training crawlers, so they feel protective while doing nothing about the crawlers that actually cite. This tool keeps the two decisions separate on purpose.
Where does robots.txt go, and will this replace mine?
It lives at your site root, so it loads at yourfirm.com/robots.txt. A site has exactly one, so this file replaces your current one. If you already have robots.txt with other rules — a sitemap line, disallowed admin paths — merge these crawler groups into it rather than pasting over the whole thing. On WordPress an SEO plugin or your host’s file manager can edit it; some hosted builders manage robots.txt for you and won’t let you set it by hand.
Does letting these crawlers in get my firm cited?
No — it removes a blocker, it doesn’t manufacture a citation. Allowing the answer-engine crawlers is necessary (they can’t cite a site they can’t read) but not sufficient; whether an engine actually names you depends on authority signals no robots.txt can create. This tool makes sure you’re not accidentally invisible. The rest is the work.