A KPI that's showing up on more and more agency dashboards this year is "brand mention percentage in AI answers", the share of AI-generated responses that mention your brand when asked a relevant question. It's an intuitive number, treated the way a search-ranking position used to be treated. We wanted to know how much of the movement in that number, week to week, is signal versus noise, so we ran a variance decomposition across 12,933 AI-answer responses.
The answer is that most of it is noise. Resampling the identical question against the identical model, nothing about the brand or the prompt changed, accounted for 34.8% of the variance we measured. The actual brand-mention signal we were trying to track accounted for 0.7%. Put plainly: if you ran the same query twice in a row and got different answers, and you usually would, most of what moved between those two answers had nothing to do with your brand at all.
Share of the variance in brand-mention percentage
That's not a small caveat on an otherwise-useful metric. It's most of the variance in the number. A dashboard tracking this week over week, without accounting for resampling noise, is mostly reporting the AI model's own inconsistency back to you dressed up as a brand trend. A real change in your visibility can sit inside a swing that size and be indistinguishable from nothing having happened at all.
This doesn't mean the underlying question, does an AI model mention your brand when it's relevant, is a bad one to ask. It means asking it once and charting the answer isn't measuring what people think it's measuring. Getting a usable number out of it means sampling enough times per period to separate the signal from the noise the model itself introduces, which is a different and considerably less convenient exercise than a weekly dashboard refresh.