Which well-known sites block AI crawlers, from 122 robots.txt files

October 19, 2026

On the evening of October 19, 2026 I fetched robots.txt from 122 well-known sites and resolved 19 AI crawler user agents against each one the way the crawlers themselves do: the longest matching user-agent group wins, otherwise the wildcard group applies. The checker is the AI crawlers tool in Toolbox, which I built, so read the method before the numbers.

Method

The 122 sites span developer tools, SaaS, social networks, news, documentation, education and retail. "Blocked" means the agent's effective group contains Disallow: / with no Allow carve-out. "Partial" means some paths are disallowed, usually because the agent inherits a wildcard group with a few lines in it. For llms.txt I fetched the file and counted it only when the body was plain text, because 15 sites return their HTML 404 page with a 200 status.

What came back

• 113 of 122 have a robots.txt at all. The nine without one include Stack Overflow, LinkedIn, npm, Next.js, Tailwind, Rust, Udemy, DoorDash and Autodesk.
• 35 sites (29%) block at least one AI crawler by name. 23 (19%) block none, with every agent allowed everywhere: Cloudflare, Supabase, Netlify, Resend, PlanetScale, React, Vue, Svelte, Astro, freeCodeCamp, and Anthropic's own site.
• 17 sites block nearly every agent on the list: Reddit, X, Facebook, Instagram, Pinterest, TikTok, Quora, Amazon, Yelp, and on the news side the New York Times, BBC, Reuters, CNN, Bloomberg, the Wall Street Journal, The Verge and Ars Technica.
• Six sites block ClaudeBot and allow GPTBot: the Washington Post, The Guardian, The Verge, the Wall Street Journal, Wired and Ars Technica. Exactly one does the reverse, Khan Academy.
• Nine sites block GPTBot (training) but allow OAI-SearchBot (search): Figma, Medium, Reuters, TechCrunch, Khan Academy, Airbnb, Tripadvisor, eBay and Canva.
• On-demand fetchers are blocked far less than training bots: ChatGPT-User by 15 sites, Claude-User and Perplexity-User by 18 each.
• 47 of 122 publish a real llms.txt. Almost all of them sell to developers: GitHub, Stripe, Cloudflare, Vercel, Supabase, Shopify, Notion, Slack, Salesforce. Not one news site does.

Most blocked agents

The headers, since the same fetch grades them

The security headers tool graded 121 of the same sites from one GET each. Twelve earned an A, 42 a B, 31 a C, 26 a D and 10 an F. 84 sites (69%) send no Content-Security-Policy at all, 85 send no Referrer-Policy, and 71 reveal a server or framework version in a Server, X-Powered-By or Via header. The F grades include SendGrid, MongoDB, the Wall Street Journal, rust-lang.org, Khan Academy, Duolingo, Amazon, Etsy, Yelp and Unity. A grade from one response is a blunt instrument; the absence of a CSP is not.

Caveats

robots.txt is a request, not enforcement, and plenty of crawlers ignore it; server logs are the only ground truth. This is one snapshot from one evening. Wildcard groups with a handful of Disallow lines show up as partial rather than blocked, so the numbers above are the clear-cut cases only. The site list is mine, chosen for recognisability, and is not a sample of the web.

Raw data

Every per-site result, including the partial lists, the llms.txt status and the headers grade, is at /data/ai-crawlers-2026-10-19.json. If a number looks wrong for a site you know, the file shows exactly what the checker saw.

Run it on your own site

The checks are free with daily caps and need no signup: the AI crawlers, security headers and page meta tools at toolbox.allthepossibles.com. The same tools run as an MCP server, so an agent can run them before a deploy.