AI Crawlers and robots.txt: Is Your Site Blocking ChatGPT?

Is your site blocking ChatGPT without you knowing?

It can happen: one line copied into robots.txt or a default CDN setting is enough to stop ChatGPT, Claude or Perplexity from reading your pages, and therefore from citing them. The site works, Google indexes it, and nothing warns you.

The classic scenario. In 2024, you read that you should “block AI” to protect your content. You pasted a list from a forum into your robots.txt. Two years later, a prospect asks ChatGPT for a provider in your city: your competitors show up, with a link to their sites. You never do. The copied list included OAI-SearchBot, the very crawler that powers ChatGPT’s search.

This guide sorts the crawlers based on each provider’s documentation, then shows how to allow citation without allowing training.

Which crawlers read your site for AI assistants?

There are two families: citation crawlers, which fetch your pages to answer a question and cite them, and training crawlers, which collect content to train future models. Blocking the first makes you invisible in answers; blocking the second changes nothing there, according to the providers.

These 12 crawlers, classified this way, are the ones Tonaily’s free audit and MCP server check. Others exist.

What does each crawler do, according to its provider?

OpenAI, Anthropic and Perplexity each separate a crawler that indexes for their search from a crawler that opens a page at a user’s request; OpenAI and Anthropic add a training crawler. Google, Apple, Meta and Common Crawl appear here for training only. Each provider documents this on a dedicated page, the only reliable source.

OpenAI: OAI-SearchBot, ChatGPT-User, GPTBot

OpenAI draws a clear line between search and training, and writes that “Each setting is independent of the others” (Overview of OpenAI Crawlers).

In other words: without OAI-SearchBot, your pages no longer show up in ChatGPT search answers, at best as a plain navigational link. GPTBot, on the other hand, collects content that may be used to train models; blocking it signals that you refuse that use. ChatGPT-User visits a page when a user asks for it in a conversation; OpenAI warns that for these visits “robots.txt rules may not apply”, and that this crawler does not determine whether you appear in search. After a robots.txt change, OpenAI says it takes about 24 hours for its systems to adjust.

Anthropic: Claude-SearchBot, Claude-User, ClaudeBot

Anthropic describes three crawlers and what blocking each one costs you (Anthropic help page). Blocking Claude-SearchBot prevents indexing for search and may reduce your visibility in results. Blocking Claude-User prevents Claude from retrieving your page when a user asks a question. Blocking ClaudeBot signals that your future content should be excluded from training data. Anthropic states that its crawlers respect robots.txt.

Perplexity: PerplexityBot, Perplexity-User

PerplexityBot is designed to “surface and link websites in search results on Perplexity” and is not used to train models (Perplexity Crawlers). Perplexity recommends allowing it. Perplexity-User supports user actions and, according to the same page, “generally ignores robots.txt rules”. Allow up to 24 hours for a change to take effect.

Google, Apple, Meta, Common Crawl: the training crawlers

Google-Extended is not a crawler: it is a robots.txt token, with no user agent of its own, that controls whether content already crawled can be used to train Gemini models and for grounding in the Gemini apps and Vertex AI (Google’s common crawlers). Apple works the same way: Applebot-Extended “does not crawl webpages”, it controls whether what Applebot collected can be used for training, and pages that opt out stay in Apple’s search results (About Applebot).

Meta-ExternalAgent crawls the web for “training foundation AI models or improving products by indexing content directly” (Meta Web Crawlers): a mixed use, which Tonaily classifies as training. CCBot is the crawler of Common Crawl, a foundation that publishes an open archive of the web; its FAQ says it reads robots.txt first (Common Crawl FAQ), and a User-agent: CCBot followed by Disallow: / is enough to keep it out (CCBot). This archive is used to train models: in the GPT-3 paper, a filtered version of Common Crawl makes up 60% of the training mix (Brown et al., 2020).

How do you allow citation and refuse training?

Write one group of rules per family: a group that allows the six citation crawlers, a group that disallows the six training crawlers, then your generic group for everyone else. Here is a commented file to adapt.

# Citation crawlers: they read your pages to cite them
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
# Copy your exclusions here: this group replaces the * group
Disallow: /admin/
Disallow: /cart/

# Training crawlers: they collect content to train models
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
Disallow: /

# Everyone else, including Googlebot and Bingbot
User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://www.yoursite.com/sitemap.xml

The surprising line is the repeated Disallow: /admin/. The robots.txt standard, RFC 9309, says a crawler follows the group that names it and only reads the * group if no group names it. As soon as you write a group for OAI-SearchBot, your general exclusions no longer apply to it: you have to copy them over. The reverse is a trap too: a User-agent: GPTBot group with Disallow: / protects nothing but GPTBot.

Three parsing rules, again from RFC 9309: the crawler name is matched case-insensitively; multiple groups naming the same crawler are combined; between two rules matching a path, the longest one wins, and on a tie Allow is preferred. The same RFC points out that these rules “are not a form of access authorization”: a crawler can ignore them, which user-triggered visits sometimes do.

Blocking training is a defensible publisher choice, not a mistake. Just know what it implies: a future model will be less likely to have learned your brand and to cite it from memory, without a web search. That is an inference, not a documented fact; the guide GEO: get cited by ChatGPT, Gemini and Grok explains the difference between answers from memory and answers with search.

Does your robots.txt block a citation crawler?

Paste it below: the checker applies RFC 9309 rules to the 12 crawlers for your homepage and separates the two families. The prefilled example is a typical case: an “anti-AI” list copied as is.

Result on the example: five crawlers disallowed, four of them for training (a deliberate choice) and OAI-SearchBot, a citation crawler. This site can no longer be shown in ChatGPT search answers, even though Claude and Perplexity can read it. Remove the User-agent: OAI-SearchBot line and run it again: all six citation crawlers get through.

Always paste the file served live (yoursite.com/robots.txt), not the one in your CMS: a CDN or a plugin can add lines to it. And a perfect robots.txt says nothing about network-level blocks.

Is Cloudflare blocking AI crawlers behind your back?

Maybe: since 2025, Cloudflare has offered to block AI crawlers as soon as a domain is onboarded, and since September 15, 2026, it recommends that new ad-supported sites block agents like ChatGPT-User on their pages with ads. These blocks happen at the firewall, so they apply even if your robots.txt allows everything.

The timeline, according to Cloudflare:

1. July 1, 2025. Cloudflare announces it is “changing the default to block AI crawlers unless they pay creators for their content” (Content Independence Day). In practice, every new domain is asked at signup whether it allows AI crawlers (press release), with the option to block them everywhere or only on hostnames that show ads (Control content use for AI training).
2. July 1, 2026. Crawlers are sorted into three categories: Search, Agent and Training. The Agent category explicitly includes “chat fetch bots (e.g., ChatGPT-User)” (Your site, your rules).
3. September 15, 2026. A new setting, “Disallow AI Training”, publishes the training opt-out in robots.txt through Bot Preference Sync, which replaces managed robots.txt. When a new domain is onboarded, Cloudflare now suggests a recommended configuration depending on whether the site runs on ads: without ads, everything is allowed; with ads, Search stays allowed, Training is set to “Disallow AI Training” and Agent is blocked on pages with ads. Settings on existing domains are migrated automatically (Have it both ways), and the old “Block AI bots” toggle is deprecated in favor of the three categories (Block AI bots).

Two traps follow. First, an ad-supported site that accepts the recommended configuration gets ChatGPT-User blocked on its pages with ads: ChatGPT can no longer open them when a user asks. Second, and this is the costliest: Cloudflare writes that “Block and Block on pages with ads now apply to mixed-use crawlers, including Applebot, Bingbot, and Googlebot”. Choosing “Block” (or “Block on pages with ads”) for the Training category can therefore block Googlebot, and your Google rankings with it. If you want to refuse training without touching search, “Disallow AI Training” is the option to choose.

To check, Cloudflare documents the steps in AI Crawl Control, available on the Free plan too:

1. Open the Cloudflare dashboard, select the domain, then AI Crawl Control.
2. In the crawler list, find OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User.
3. In the Actions column, set them to Allow if they are blocked. A block there takes the form of a WAF rule on your zone.
4. Review the Search, Agent and Training settings in your security settings, and open your live robots.txt to see the lines Cloudflare publishes there.

AI Crawl Control also counts each crawler’s robots.txt violations. Hosting firewalls and WordPress security plugins keep their own lists of blocked bots: check them the same way.

Does blocking Google-Extended remove you from AI Overviews?

No. Google-Extended controls Gemini training and grounding in the Gemini apps and Vertex AI; it does not touch Google Search, AI Overviews or AI Mode.

AI Overviews and AI Mode rely on Googlebot’s index. Google writes that to appear as a link there, a page must be indexed and eligible to be shown with a snippet, with no special file or markup (AI features and your website). To limit what gets used, Google points to the nosnippet, data-nosnippet, max-snippet and noindex controls.

New in 2026: a Search generative AI setting in Search Console lets you exclude a site from AI Overviews, AI Mode and Discover’s generative features. Announced on June 3, 2026 and first tested in the United Kingdom, it has been open to all sites since August 31, 2026 (Google). You will find it under Settings > Search generative AI. Inclusion is the default; exclusion usually takes a few days, is not used as a ranking signal for the rest of Search, and “doesn’t affect AI training”: for training, it is still Google-Extended (Search Console Help). If you want to be cited, make sure nobody switched it to “Exclude”.

How do you check that the fix works?

Check the three layers in order (live robots.txt, CDN or firewall, then the assistants’ answers), and leave at least 24 hours between the fix and the check.

flowchart TD
  A[Site missing from AI answers] --> B{robots.txt blocks a crawler?}
  B -- Yes --> C[Fix robots.txt]
  B -- No --> D{Blocked by CDN or firewall?}
  D -- Yes --> E[Unblock in the CDN]
  D -- No --> F{Excluded in Search Console?}
  F -- Yes --> G[Switch back to Include]
  F -- No --> H["Access OK: check the content"]
  C --> I[Wait 24 h, then retest]
  E --> I
  G --> I

If everything is open and you are still not cited, the cause is no longer technical: the guide Why does ChatGPT recommend your competitor and not you? walks through the five other likely causes.

In your server logs, the user agent is easy to spoof: OpenAI, Anthropic and Perplexity publish their crawlers’ IP addresses (openai.com/searchbot.json, claude.com/crawling/bots.json, www.perplexity.com/perplexitybot.json) so you can verify that a visit is genuine.

Once access is sorted, you still need to measure whether assistants actually cite you. The free test asks one question to ChatGPT, Claude, Gemini and Perplexity, no account needed; for tracking over time, first pick the right questions with the guide Choose which questions to track in AI assistants.