We audit a lot of B2B sites, and one of the most common findings has nothing to do with content quality: the robots.txt file is quietly blocking the exact AI crawlers the company is paying us to get cited by. This usually isn't a decision anyone made on purpose — it's a leftover rule from 2023 or 2024, when "block all AI bots" felt like the safe default. It's no longer safe. It's actively working against GEO.
In Q1 2026, GPTBot was the single most-blocked AI crawler across scanned sites, appearing in roughly 5.5% of DISALLOW rules — ahead of CCBot, ClaudeBot, Google-Extended, and Bytespider. That's notable because GPTBot is also the crawler responsible for ChatGPT's real-time web retrieval, not just training data collection.
| Crawler | Q1 2026 DISALLOW share | What blocking it actually affects |
|---|---|---|
| GPTBot | ~5.5% | ChatGPT training and real-time retrieval for citations |
| CCBot | ~5.1% | Common Crawl dataset used to train many models |
| ClaudeBot | ~4.9% | Claude training and web-search tool retrieval |
| Google-Extended | ~4.4% | Gemini training and Google AI features (not classic Search) |
| Bytespider | ~4.2% | ByteDance/TikTok AI training crawler |
By May 2026, the pattern had started to shift — at least for GPTBot. Its ALLOW share climbed to roughly 5.8%, overtaking its own DISALLOW share for the first time since tracking began, alongside rising ALLOW rates for PerplexityBot (~5.5%), ChatGPT-User (~5.0%), OAI-SearchBot (~4.4%), Google-Extended (~4.8%), and ClaudeBot (~4.6%). The web, in aggregate, is leaning toward allowing AI crawlers rather than blocking them by default — but a meaningful minority of sites, especially in B2B, haven't caught up.
"Block training, allow search" is now the visible industry consensus: distinguish between crawlers that scrape content for future model training and crawlers that fetch a page in real time to answer a specific user's question right now.
This isn't a static picture. In early July 2026, Cloudflare announced that starting September 15, 2026, its default settings will block "mixed-use" AI crawlers — bots that both train models and serve real-time answers under the same user-agent — from any page that carries ads, unless the crawler operator agrees to separate its search and training traffic.
"Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge," said Matthew Prince, Cloudflare's co-founder and CEO, announcing the change. "We hope that our proposed default changes encourage mixed-use crawlers to separate out search from agent use and training."
For B2B sites on Cloudflare, this makes the training-versus-search distinction a practical, not just theoretical, exercise: crawlers that don't clearly identify which purpose they're serving risk losing access to ad-supported pages by default later this year. Confirming whether your AI search crawlers are cleanly separated from training crawlers in your own configuration is worth doing well before that deadline.
Not all AI bots do the same job, and treating them identically in robots.txt is where most accidental blocking happens. Training crawlers like the original GPTBot crawl mode and CCBot scrape content to improve future model versions — blocking these has no effect on today's citation rates. Search crawlers, by contrast, fetch a page live to answer a specific query: OAI-SearchBot, ChatGPT-User, and PerplexityBot all fall into this category, working functionally similar to how Googlebot serves classic search results.
Block a training crawler, and nothing changes about whether your brand shows up in tomorrow's ChatGPT answer. Block a search crawler, and you can be functionally invisible in that platform's real-time citations — even if your content is excellent and your schema markup is flawless.
That gap between "block at least one bot" and "block Google-Extended specifically" shows publishers are already making the training-versus-search distinction Google itself recommends. B2B sites, which typically set their robots.txt once and rarely revisit it, are behind that curve.
The fix here is genuinely fast — usually a same-day engineering task once you know what to look for.
User-agent: * Disallow: / rule that predates any specific AI bot carve-outs — this silently blocks everything, including AI search crawlers, unless overridden by a more specific allow ruleAllowing a crawler in robots.txt doesn't guarantee it can read your content — if your site relies on client-side JavaScript rendering for body copy, headers, or navigation, some AI crawlers won't execute that JavaScript and will see an empty shell regardless of what robots.txt says. Verify with a raw HTML fetch (no JS execution) that your actual page content is present in the initial server response.
Every other GEO tactic — schema markup, statistics-rich content, off-site authority building — assumes the crawler can reach the page in the first place. A misconfigured robots.txt is a single point of failure that silently undoes all of it. It's also the fastest fix available in any AI visibility audit: no content rewrite, no new pages, just a file update and a redeploy.
As of Q1 2026, GPTBot was the most-blocked AI crawler, appearing in roughly 5.5% of DISALLOW rules across scanned sites, ahead of CCBot, ClaudeBot, Google-Extended, and Bytespider. By May 2026 that pattern had begun reversing for GPTBot specifically, with its ALLOW share overtaking its DISALLOW share for the first time in tracking.
Yes. An estimated 41% of B2B sites still block at least one major AI crawler, and much of this traces back to blanket "block all AI bots" robots.txt rules added during 2023-2024, when AI training scraping was the dominant concern. Many of those same rules also block the AI search crawlers that a GEO strategy depends on.
Training crawlers (like the original GPTBot and CCBot) scrape content to train future models. Search crawlers (like OAI-SearchBot, PerplexityBot, and ChatGPT-User) fetch pages in real time to answer a specific user query, similar to how Googlebot works. Blocking a training crawler doesn't affect AI search citation; blocking a search crawler can remove a site from that platform's answers entirely. The current industry consensus, "block training, allow search," recommends treating these categories differently rather than blocking all AI bots as one group.
For AI search visibility, allow the retrieval-time crawlers: GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, and Google-Extended. Companies with sensitive competitive data, unpublished pricing, or content licensing concerns may still choose to block training-only crawlers like CCBot and Bytespider — that's a legitimate business decision distinct from AI search visibility, and blocking those does not reduce citation rates in ChatGPT, Perplexity, or Google AI Overviews.
Our AI Visibility Check includes a full crawler-access audit — we'll tell you in minutes if your site is invisible to the AI platforms your buyers are asking.
Book a Free AI Visibility CheckFree — No Commitment
We test 15 of your buyers' top queries across ChatGPT, Perplexity, and Google AI Overviews — and show you exactly where you appear, where competitors are being cited, and what's driving the gap.
Check My AI VisibilityTakes 2 minutes to request. We'll follow up within one business day.