API vs UI Measurement (LLM Visibility)
Consensus definition
That API and UI answers diverge is well documented and not disputed. A comparison published in August 2026 ran 555 prompts, 47 times each, through both the ChatGPT interface and the API on the same underlying model — more than 52,000 chats — and found the average brand shifting visibility by 41% between the two paths, fewer search fan-outs through the API (up to 7, versus up to 20 in the UI), a substantially different source mix — with direct consequences for any Citation Rate read on top of it — and answers around 31% longer via the API1. Perplexity states the same fact about its own stack: API and UI share the search system but differ in configuration, so outputs may differ2. The disputed question is what follows for measurement: the study's author concludes that tools using the API are "lying to you" and that only the web interface may be relied on1.
rhinegold operator caution
Rhinegold measures through provider APIs with the retrieval layer switched on, and holds that the divergence finding is right while the exclusivity conclusion is wrong. No access path observes 'the truth'. OpenAI's own documentation describes how ChatGPT personalises answers per account from saved memories and recent conversations3 — so a UI observation reflects one account state among millions, and an API call reflects one pinned configuration. The study's own design concedes that single answers are unstable: it needed 47 runs per prompt, consistent with peer-reviewed work showing single-run LLM measurements are statistically invalid4. Nor is the API the "bare-bones" text endpoint the argument assumes: current access paths expose the retrieval layer explicitly — OpenAI's web search tool returns answers with sourced citations5, Anthropic's equivalent does the same and accepts a user location for partial geo-emulation6, and Perplexity's API is search-grounded by design2. For tracking over time, a documented, replayable configuration is what a measurement instrument requires: Share of Voice, Mention Rate and trend deltas compare brands under identical conditions, so a constant access-path offset subtracts out — 'different from the UI' means calibrated differently, not wrong. And the consumer surface moves: ChatGPT's retrieval behaviour changed twice within three weeks in August 2026, unannounced — Reddit citations collapsing 95% while site-restricted queries jumped from under 0.11% to 26.2%7. An unpinned instrument inherits every such swing as apparent market movement. UI observation is the complementary lens, not the enemy: it is the closest available proxy for what a signed-in user in a specific locale is shown, which makes it the right check for localisation and the right answer to 'what does the interface actually say about us today?'. The two paths answer different questions; a serious programme states which one each number came from — see Platform Divergence for the same discipline across providers.
| Dimension | API-based measurement | UI-observed measurement |
|---|---|---|
| Access path | Documented endpoint with explicit parameters | Observation of the consumer product as served |
| Measurement object | A pinned, versionable configuration | An account-shaped surface: memory, instructions and experiments vary per account |
| Retrieval layer | Explicit and citable — web search tools with sourced citations | Present, but its configuration cannot be pinned from outside |
| Reproducibility | Same request replayable; configuration diffs diagnosable | Depends on session and account state |
| Localisation | Partial emulation via location parameters | Native — the locale-specific answer a user is actually shown |
| Strength | Relative metrics under identical conditions: share of voice, trends, cross-brand deltas | Proximity to the lived user experience; local answers; face-validity checks |
| Limitation | Does not show any individual user's on-screen answer | Observes one account state; cross-account variance is uncharacterised |
| Coverage economics | Uniform instrumentation across providers | Per-interface effort; breadth is costlier |
Operational use
Use this comparison when choosing or defending a tracking setup, and when a vendor or a study argues that one access path reveals 'the truth' about LLM visibility. The operational rule: fix the instrument — including the discovery-prompt set that runs through it — measure relatively, use UI observation as the complementary check for localisation and user-facing plausibility, and read every visibility number together with the access path, provider, configuration and date that produced it. A number without its access path is not comparable to anything — which applies to UI-observed and API-based figures alike.
Measurement boundary
API-based measurement does not reproduce any individual user's on-screen experience, and does not claim to — memory-driven personalisation is invisible to it by design. Conversely, a UI observation covers exactly one account state, and how much answers vary across accounts has not been characterised by published research — so generalising from either path has open error bars. Absolute visibility values are not portable across access paths in either direction; only within-instrument comparisons are defensible. The August 2026 comparison itself is a single-author test on a single provider, published without raw data and without disclosure of the UI account state (memory on or off, personalisation, location) — its headline percentages should be read as one measurement of configuration divergence, not as a calibration constant.
Distinct from
From Platform Divergence: that entry covers divergence across providers (ChatGPT vs Gemini vs Perplexity); this one covers divergence across access paths within a single provider. The measurement discipline is the same — treat divergence as signal about configurations, not as proof that one configuration is real and the other fake. From LLM Brand Tracking: the access-path decision documented here is an input to tracking design, not a tracking metric itself. From Query Fan-Out: fan-out behaviour is one of the dimensions on which the paths diverge — the same provider fans out up to 20 times in the UI and at most 7 through the API1 — which makes the access path a mandatory disclosure for any fan-out study.
Observed pattern in practice
Common mistakes
- Treating either access path as ground truth — each observes one configuration of the model, and personalisation means there is no single 'what users see'.
- Concluding that a stable instrument 'lies' because its absolute values differ from another configuration — relative metrics absorb a constant offset; absolute claims survive neither path.
- Generalising a single-provider API-vs-UI test to all providers and all tools.
- Reading a methodology verdict from a tracking vendor without checking whether the verdict matches the vendor's own product.
- Publishing visibility numbers without stating the access path, configuration and date that produced them.
- Comparing visibility figures across access paths as if they shared a scale.
Where consensus is missing
No published study quantifies how much UI answers vary across accounts — the personalisation variance that would bound what 'what users see' even means. The August 2026 comparison has not been independently replicated, published no raw data, and covers one provider; whether the divergence pattern holds for Gemini, Claude or Perplexity interfaces is open. And the complementary design this entry argues for — pinned API tracking plus targeted UI observation for localisation and face-validity — has no published benchmark yet. There is no standard for disclosing access-path configuration in visibility studies; there should be.
Sources & deeper reading
- 1Malte Landwehr — "API vs UI - How to track LLM visibility?" (LinkedIn, 2026-08-29) — 555 prompts × 47 runs, UI vs API, 52K+ chats; the comparison under review
- 8Search Engine Journal — author profile Malte Landwehr: "CPO & CMO at Peec AI, a leading AI search analytics solution"
- 9Discovered Labs — Peec AI review: "The platform uses UI scraping technology to simulate real user interactions with AI tools"
- 7Metehan Yeşilyurt — "Reddit lost most of its visibility in ChatGPT, and the site parameter filled the gap" (LinkedIn, 2026-08-21) — two unannounced retrieval changes within three weeks; Reddit citations −95%, site-restricted queries 0.11% → 26.2%
- 5OpenAI — Web search tool, Responses API (official documentation): "Web search allows models to access up-to-date information from the internet and provide answers with sourced citations"
- 6Anthropic — Web search tool (official documentation): API answers with citations for sources drawn from live search results; user_location parameter for localised results
- 2Perplexity — Sonar API (official announcement): search-grounded API; same search system as the UI with differences in configuration
- 3OpenAI — Memory FAQ (official help center): ChatGPT references saved memories and recent conversations to personalise responses per account