Measurement Consistency
Consensus definition
Measurement consistency is the property of a measurement series in which every observed change can be attributed to the thing measured, because the method did not change in between. For AI-visibility metrics — mention rate, citation rate, share of voice, shortlist position — the method has at least five moving parts: the counting unit (every occurrence of a brand name, or one brand per answer), the entity map (which strings count as the brand, which as competitors), the denominator (all answers, or only answers of one type), the provider weights (how four assistants are blended into one number), and the prompt set (which questions are asked, with or without brand names). None of these is given by the data; each is a decision. A series is consistent when the decisions are written down, frozen, and every later change is dated and marked in the series as a break.
rhinegold operator refinement
Rhinegold's reframe: a visibility dashboard is only as good as its least stable definition. Most practitioners worry about model updates and prompt variance — real, but visible. The larger errors we have seen were invisible: the method moved and nobody noticed, because every individual number was computed correctly. In our own measurement practice for B2B clients, method changes have moved headline metrics by factors of three to eight and have reversed the sign of a trend. Measurement consistency is therefore not a hygiene topic; it decides whether a quarter's "progress" existed.
Operational use
Before trusting any trend: ask for the counting unit, the entity map, the denominator, the weights, and the prompt set at both ends of the series. If any of them changed, the two ends are two series. Keep a written methodology register with a date for every change; show breaks as vertical lines in charts; re-measure the earlier wave with the new method before comparing. Treat a tool switch, a provider switch and a parser update as method changes, because they are. In an AI visibility audit, the register is the first document to ask for.
Measurement boundary
Consistency makes a series interpretable; it does not make it right. A consistently applied bad definition yields a consistently misleading number. And the metric cannot be frozen entirely: providers change retrieval and models without notice, so part of every series is exogenous. The honest position is a frozen method on a moving platform, with the platform changes logged as well.
Distinct from
From Platform Divergence: divergence is providers disagreeing with each other; inconsistency is the measurement disagreeing with itself. From Prompt Coverage: coverage asks whether the right questions are in the set; consistency asks whether the set stayed the same. From AI Answer Sentiment: the published SISTRIX test is the external version of this entry — the same brand question, asked directly, scattered by 17 points on a 200-point scale and by more than 40 when reworded; measured indirectly from real answers, two brands that direct questioning placed about 30 points apart were 118 points apart1.
Obstacles & resolutions
Observed pattern in practice
Common mistakes
- Comparing two waves measured with different tools, parsers or prompt sets as one series.
- Reporting a share without its numerator and denominator in absolute terms.
- Changing provider weights because the result "looked wrong".
- Blending aided and unaided prompts into one visibility figure.
- Treating consistency as achieved once the method is documented — it has to be re-checked at every change of tool or provider.
Where consensus is missing
There is no industry standard for any of the five decisions, and tools rarely disclose theirs. Switching between two visibility tools therefore means starting a new series. Whether and how series from different tools can be bridged is an open question; Rhinegold's position is that they cannot — only re-measured. See also LLM Brand Tracking on what a tracking set-up has to hold constant.