AI Crawler Traffic
- AI crawler traffic has three main documented purposes: training, search indexing and user-triggered fetching.
- Only user-triggered fetches trace a person asking a question — report them separately.
- Read them against LLM referral sessions; a fetch is neither a user nor a citation.
Consensus definition
OpenAI and Anthropic each document separate bots for three purposes: training, search indexing and user-triggered fetching12; Perplexity documents bots for the last two and states that neither is used to train foundation models3. Training crawlers (GPTBot, ClaudeBot) collect content for model training12. Search indexers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) index sites for the providers' AI search answers132. User-triggered fetchers (ChatGPT-User, Perplexity-User, Claude-User) retrieve a page when a person asks the assistant a question, so that the answer can draw on it132; Perplexity adds that the answer includes a link to the page3. Google lists user-triggered fetchers as a category of its own, next to common crawlers such as Googlebot and special-case crawlers such as AdsBot4. Because a person requested the fetch, Google's and Perplexity's user-triggered fetchers generally ignore robots.txt, and OpenAI writes that robots.txt rules 'may not apply' to ChatGPT-User531; Anthropic, by contrast, describes blocking Claude-User as a way to stop such fetches2. Not every AI control is visible in a log: Google-Extended, which governs use of crawled content for Gemini training and grounding, is a robots.txt token without a user-agent string of its own; Google crawls with its existing user agents6.
rhinegold operator refinement
Rhinegold's refinement: for ChatGPT, Perplexity and Claude, a user-triggered fetch is the server-side trace of a demand event that no other first-party report records. Someone asked, and the assistant chose to open this page; one question can leave several traces, and an answer from the provider's own index leaves none. When one of these assistants opens a page to answer someone, no search query reaches the site owner, and a tag-based analytics session typically starts only if the user later clicks through to the page in a browser. The request in the server log is then the only first-party trace; Microsoft and Google report citations or impressions in AI answers instead78. It is the site-side trace of live retrieval in grounding: the moment the model's retrieval step reaches your server. Rhinegold's recommended practice is therefore to split AI crawler traffic by the providers' documented purpose and to report only the user-triggered stream next to LLM referral traffic — fetches per week, per page and per provider. Training crawls stay in infrastructure reporting; their volume says nothing about whether anyone is asking. Our working hypothesis, not yet tested against outcomes, is that rising user-triggered fetches on a page precede referrals and enquiries, because the fetch happens at question time and the click, if it comes at all, comes later.
Operational use
Classify log lines by user agent into training, search indexing and user-triggered fetches, using each provider's published bot documentation. Report the user-triggered stream per page and per week, and set it against referral sessions from the same providers: the ratio is a rough indicator of how often answer-time retrieval of a page is followed by a visit. Pages with many fetches and few referrals were retrieved often but sent few visits — a page-level view of the zero-click pattern, although the log does not show whether they were cited. Check which status codes user-triggered fetchers receive: an assistant requesting addresses on your domain that do not exist is a candidate signal for phantom URLs and yields a list of redirect targets. Several fetches of related pages within seconds may come from query fan-out behind a single question, which is why fetch counts are not user counts.
Measurement boundary
The log shows that a page was fetched — not the question, not the answer, and not whether the page was cited. One question can trigger several fetches, and an answer built from a provider's own index may trigger none, so fetches are neither users nor citations. User-agent strings are declared by the bot itself and can be spoofed5, so the label in the log is the bot's own claim. Behaviour also varies by bot: a large-scale empirical study of web logs found that bots comply less with stricter robots.txt directives and that AI search crawlers rarely check robots.txt at all9. Purposes that share a user agent cannot be separated: Google-Extended has no user-agent string of its own6. Sites behind a CDN or bot-management layer see only the requests that layer passes through. We know of no public study that tests user-triggered fetches as a predictor of demand.
Distinct from
From LLM referral traffic: referrals are visits by people who clicked a link in an answer; AI crawler traffic is requests by provider bots, and its user-triggered part happens whether or not anyone clicks. From retrieval-augmented generation and grounding: those are the model-side mechanisms that decide to retrieve; a user-triggered fetch is the part of that retrieval that reaches your server. From search indexing: an indexer such as OAI-SearchBot or PerplexityBot prepares a site for future answers, while a user-triggered fetch serves one answer now13. From zero-click search: zero-click describes the visit that does not happen; user-triggered fetches are one of the few first-party records of the answer that replaced it.
Empirical anchor
Common mistakes
- Reporting AI crawler traffic as one number — on Cloudflare's network, training crawling far outweighed user-triggered fetching in 202510, so a total there mostly measured training activity, not demand; on a single site the mix varies, so measure the split in your own log.
- Blocking every AI bot in robots.txt to stop training and thereby also blocking the search indexer that decides whether the site can appear in AI search answers — OpenAI and Anthropic document separate tokens for each purpose12.
- Assuming robots.txt controls user-triggered fetchers — Google and Perplexity state that these fetches generally ignore it, and OpenAI that its rules may not apply531.
- Reading fetch counts as user counts or as citations — a fetch shows that a page was retrieved, not how many people asked or whether the page was quoted.
- Taking the user-agent label at face value — the string is self-declared, and robots.txt compliance varies by bot category9.
Where consensus is missing
Providers use similar purpose categories but no shared taxonomy. There is no agreed method for linking a fetch to the question that caused it, and no published benchmark of fetch-to-referral ratios at site level.
Sources & deeper reading
- 1OpenAI — "Overview of OpenAI Crawlers" (OAI-SearchBot, GPTBot, ChatGPT-User; robots.txt rules 'may not apply' to user-initiated fetches)
- 3Perplexity — crawler documentation (PerplexityBot for search results, Perplexity-User for user-triggered fetches; neither used for foundation-model training)
- 4Google Search Central — "Overview of Google crawlers and fetchers (user agents)" (common crawlers, special-case crawlers, user-triggered fetchers)
- 6Google Search Central — "Google's common crawlers" (Google-Extended: robots.txt token without its own user-agent string; last updated 15 October 2026)
- 5Google Search Central — "Google's user-triggered fetchers" (generally ignore robots.txt because a user requested the fetch; last updated 19 August 2026)
- 2Anthropic Support — "Does Anthropic crawl data from the web, and how can site owners block the crawler?" (ClaudeBot, Claude-User, Claude-SearchBot)
- 7Bing Webmaster Blog — "Introducing AI Performance in Bing Webmaster Tools Public Preview" (10 February 2026; cited pages and a sample of grounding queries for Copilot and Bing AI answers)
- 8Google Search Console Help — "Generative AI performance report" (impressions = how often links to a site were shown in a generative AI feature)
- 10Cloudflare Blog — João Tomé, "The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals" (29 August 2025; crawl purpose shares for July 2025)
- 11Cloudflare Blog — Cloudflare Radar 2025 Year in Review, AI crawler section (15 December 2025; user-action growth and crawl-to-refer ratios)