Blocking GPTBot in your robots.txt stops OpenAI from crawling your site for training. Does it also stop ChatGPT from citing you? We cross-referenced the AI-crawler rules of 1,058 of the web's most-cited and most-visible domains against how often each one actually gets cited in ChatGPT and Perplexity answers.
Published July 6, 2026 · cloro citation corpus × robots.txt crawl · 1,058 domains, 88% fetched
0.003 vs 0.417
The typical (median) GPTBot-blocking domain earns almost no ChatGPT citations per Google-ranking it holds, 0.003 against 0.417 for domains that let GPTBot in. Blocking the crawler tracks with near-invisibility in the answer layer.
It's engine-specific
Blocking a provider's own crawler depresses citations in that provider's engine specifically: PerplexityBot-blockers get a median 0 Perplexity propensity vs 1.167 for others, while blocking non-OpenAI bots barely dents ChatGPT. A coincidence would be unlikely to line up that neatly with who owns which crawler.
Training blocked ≫ retrieval blocked
Sites block AI training crawlers far more than the bots that fetch pages to answer a live query: CCBot 17.8%, ClaudeBot 15.4%, GPTBot 13.9%, but OAI-SearchBot just 3.4% and ChatGPT-User 6.2%. Publishers are drawing a line between "don't train on me" and "do cite me."
Share of the 1,058 most-cited and most-visible domains whose robots.txt disallows each crawler at the site root. The training crawlers sit at the top of the bars, the ones that fetch a page to answer a live question at the bottom.
Most publishers block training and leave retrieval open. Check which of these seven your disallow list covers: the bottom two decide whether an engine can read you when it answers.
The headline block-rate says nothing about outcomes on its own, since cited domains might just be prominent ones. So we measure citation propensity: citations per Google-organic appearance, which normalizes for how visible a domain already is. Below, the median propensity for domains that block vs allow each crawler.
| Crawler blocked | ChatGPT · blockers | ChatGPT · others | Perplexity · blockers | Perplexity · others |
|---|---|---|---|---|
| GPTBot | 0.003 | 0.417 | 0.566 | 1.143 |
| OAI-SearchBot | 0 | 0.394 | 0.039 | 1.083 |
| ClaudeBot | 0.22 | 0.4 | 0.279 | 1.182 |
| PerplexityBot | 0.259 | 0.333 | 0 | 1.167 |
| Google-Extended | 0.308 | 0.348 | 0.273 | 1.162 |
Read across a row: OpenAI's crawlers (GPTBot, OAI-SearchBot) collapse ChatGPT propensity to near zero when blocked, and PerplexityBot collapses Perplexity propensity to zero. Blocking a rival provider's bot moves the other engine far less, so the drop stays with the provider whose crawler was shut out. The bot you disallow predicts the engine you stop appearing in, so audit robots.txt per crawler, not as one policy.
If being named in AI answers matters to you, a blanket "block every AI bot" robots.txt is working against you. The data can't prove causation, and domains that block crawlers may differ in other ways. But the engine-specific pattern is what a real access effect would look like: blocking OpenAI's crawlers tracks with vanishing ChatGPT citations, blocking Perplexity's tracks with vanishing Perplexity citations.
The practical split is between the training crawlers and the retrieval ones. Publishers are already making it, blocking GPTBot and CCBot at 3–5× the rate they block OAI-SearchBot. If you want to keep your content out of model training but still be citable, that is where to act: allow the live-retrieval and user-agent bots even where you disallow the training crawlers.
We took the 1,058 most-prominent domains in cloro's citation and organic corpus (those most cited by ChatGPT/Perplexity plus those ranking most often in Google's top 10) and fetched each one's robots.txt. 88% resolved. A domain "blocks" a crawler when its robots.txt disallows the site root for that user-agent's group (or for * when the bot has no group of its own). Citation propensity is citations per Google-organic appearance over a two-day citation window, which controls for baseline prominence; domains with no organic appearances are excluded from the propensity split. Figures are aggregate rates, not per-site disclosures, and are directional: the corpus reflects cloro's monitored prompt mix rather than the whole web.
cloro. (July 6, 2026). Do Sites That Block GPTBot Get Cited Less by ChatGPT?. cloro Research. https://cloro.dev/research/ai-crawler-blocks/
More studies from cloro's monitoring corpus are in the research index.
cloro tracks which sources ChatGPT, Perplexity, and Google's AI Overview cite — live, for any query. Start with 500 free credits a month.
