Blocking GPTBot in your robots.txt stops OpenAI from crawling your site for training. A worried question follows: does it also stop ChatGPT from citing you? We cross-referenced the AI-crawler rules of 1,058 of the web's most-cited and most-visible domains against how often each one actually gets cited in ChatGPT and Perplexity answers.
Published July 6, 2026 · cloro citation corpus × robots.txt crawl · 1,058 domains, 88% fetched
0.003 vs 0.417
The typical (median) GPTBot-blocking domain earns almost no ChatGPT citations per Google-ranking it holds — 0.003 vs 0.417 for domains that let GPTBot in. Blocking the crawler tracks with near-invisibility in the answer layer.
It's engine-specific
Blocking a provider's own crawler depresses citations in that provider's engine specifically: PerplexityBot-blockers get a median 0 Perplexity propensity vs 1.167 for others — while blocking non-OpenAI bots barely dents ChatGPT. That specificity is what a coincidence wouldn't produce.
Training blocked ≫ retrieval blocked
Sites block AI training crawlers far more than the bots that fetch pages to answer a live query: CCBot 17.8%, ClaudeBot 15.4%, GPTBot 13.9% — but OAI-SearchBot just 3.4% and ChatGPT-User 6.2%. Publishers are drawing a line between "don't train on me" and "do cite me."
Share of the 1,058 most-cited and most-visible domains whose robots.txt disallows each crawler at the site root.
The headline block-rate says nothing about outcomes on its own — cited domains might just be prominent ones. So we measure citation propensity: citations per Google-organic appearance, which normalizes for how visible a domain already is. Below, the median propensity for domains that block vs allow each crawler.
| Crawler blocked | ChatGPT · blockers | ChatGPT · others | Perplexity · blockers | Perplexity · others |
|---|---|---|---|---|
| GPTBot | 0.003 | 0.417 | 0.566 | 1.143 |
| OAI-SearchBot | 0 | 0.394 | 0.039 | 1.083 |
| ClaudeBot | 0.22 | 0.4 | 0.279 | 1.182 |
| PerplexityBot | 0.259 | 0.333 | 0 | 1.167 |
| Google-Extended | 0.308 | 0.348 | 0.273 | 1.162 |
Read across a row: OpenAI's crawlers (GPTBot, OAI-SearchBot) collapse ChatGPT propensity to near zero when blocked, and PerplexityBot collapses Perplexity propensity to zero. Blocking a rival provider's bot moves the other engine far less — the effect follows the crawler's owner.
If being named in AI answers matters to you, a blanket "block every AI bot" robots.txt is working against you. The data can't prove causation — domains that block crawlers may differ in other ways — but the engine-specific pattern is exactly what a real access effect looks like and exactly what a spurious one wouldn't: blocking OpenAI's crawlers tracks with vanishing ChatGPT citations, blocking Perplexity's tracks with vanishing Perplexity citations.
The practical split is between the training crawlers and theretrieval ones. Publishers are already making it — blocking GPTBot and CCBot at 3–5× the rate they block OAI-SearchBot. If you want to keep your content out of model training but still be citable, that distinction is the lever: allow the live-retrieval and user-agent bots even where you disallow the training crawlers.
We took the 1,058 most-prominent domains in cloro's citation and organic corpus (those most cited by ChatGPT/Perplexity plus those ranking most often in Google's top 10) and fetched each one's robots.txt — 88% resolved. A domain "blocks" a crawler when its robots.txt disallows the site root for that user-agent's group (or for * when the bot has no group of its own). Citation propensity is citations per Google-organic appearance over a two-day citation window, which controls for baseline prominence; domains with no organic appearances are excluded from the propensity split. Figures are aggregate rates, not per-site disclosures, and are directional — the corpus reflects cloro's monitored prompt mix rather than the whole web.
cloro. (July 6, 2026). Do Sites That Block GPTBot Get Cited Less by ChatGPT?. cloro Research. https://cloro.dev/research/ai-crawler-blocks/
More studies from cloro's monitoring corpus are in the research index.
cloro tracks which sources ChatGPT, Perplexity, and Google's AI Overview cite — live, for any query. Start with 500 free credits.
