Check which AI crawlers your site blocks
Most sites have never decided what AI crawlers may do. Their robots.txt decided for them, usually years ago, and usually by accident.
Paste a robots.txt
This runs in your browser; nothing is sent anywhere. It applies the rule crawlers use: the group with the longest matching user-agent name wins, and the wildcard group applies only when no named group matches. It is the same logic as the Toolbox AI crawlers tool, which fetches the file from a live site for you.
To check a live site instead, give a URL to Toolbox. It also reports whether you publish an llms.txt and drafts one. Ten URL checks a day are free with no signup.
How to read the result
Blocked means the agent's effective group contains Disallow: / with no Allow: / beside it. Partial means some paths are off limits, which is what a wildcard group with a handful of Disallow lines produces. Allowed means no rule stops it.
The agents are not interchangeable. Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent) collect material to build models. Search agents such as OAI-SearchBot index pages so a product can cite them. On-demand fetchers (ChatGPT-User, Claude-User, Perplexity-User) load a page because a person asked an assistant about it. Blocking the first group and allowing the others is the combination most publishers say they want: out of the training set, still citable.
What 122 well-known sites do
On October 19, 2026 I resolved the 19 agents above against the robots.txt of 122 sites, using the same checker. The full write-up is in the note and every per-site result is in the raw data file.
- 35 of the 122 (29%) block at least one AI crawler by name; 23 (19%) block none.
- ClaudeBot is blocked by 28 sites, GPTBot by 23, and Bytespider by 30, the most of any agent.
- Six news publishers block ClaudeBot and allow GPTBot. One site, Khan Academy, does the reverse.
- 47 sites publish a real llms.txt, almost all of them developer tools; no news site does.
One snapshot, one evening, a list I chose for recognisability. It is a description of those 122 sites and not a sample of the web.
What to do with it
Decide the policy first, then write the file. If you want to stay out of training but keep appearing in AI answers, name the training agents with Disallow: / and leave the search and on-demand agents alone. If you are happy to be used everywhere, you already are; the useful step is an llms.txt that points agents at your best pages. If a block matters legally or commercially, do not rely on robots.txt alone, since it binds only crawlers that choose to obey it.
The same file feeds other checks. Your sitemap, your security headers and your page metadata are all one request away in Toolbox, and the EAA guide covers a different kind of site audit if you sell into Europe.
Questions people ask
How do I block GPTBot but keep appearing in ChatGPT search?
OpenAI documents separate agents: GPTBot for training, OAI-SearchBot for search and ChatGPT-User for fetches a person triggers. A robots.txt can disallow one and allow another. In our survey of 122 sites, nine did exactly this with GPTBot and OAI-SearchBot.
Does robots.txt stop AI crawlers?
It is a request. Well-behaved crawlers honour it and others do not. Server logs and a firewall rule are the only enforcement.
Does a wildcard Disallow block AI crawlers too?
Yes, for any crawler that has no group of its own. A group that names a specific agent takes precedence over the wildcard group for that agent.