Why one AI answer tells you almost nothing about your visibility

Short answer: AI assistants give a different answer almost every time you ask, so a visibility number is only meaningful if it comes from many answers and tells you how uncertain it is. A rate of "30%" from 40 answers could really be anywhere from about 18% to 45%. Before you act on a number, check the sample size and the margin of error.

Why do AI answers vary so much?

Ask ChatGPT "What's the best running app?" five times and you will probably get five different lists. There are three reasons:

• Sampling. Language models choose each word from a set of likely options, with some randomness. Two runs of the same prompt can go different ways early and end up naming different brands.
• Changing search results. Assistants that search the web get different results from day to day, and sometimes from minute to minute.
• Phrasing. "Best running app", "which running app should I use" and "good app for tracking runs" are the same question to a person and three different prompts to a model.

So a single answer is like a single survey response. It is real, but it doesn't tell you what most answers look like.

What is a margin of error, in plain terms?

Suppose your brand was mentioned in 12 out of 40 answers. The observed rate is 30%. But if you collected another 40 answers next week, with nothing changed, you might see 9 or 15 mentions just by chance.

A 95% interval gives the range of true rates that are consistent with what you observed. For 12 mentions in 40 answers, the interval (using the Wilson method, which behaves well for small samples and for rates near 0% or 100%) is roughly 18% to 45%.

That is a wide range. It means "you are mentioned in a meaningful minority of answers", not "you are at 30%".

How many answers do you need?

The interval narrows as the sample grows. Here is the approximate half-width of a 95% interval for a rate near 50%, which is the worst case:

Two practical rules follow:

1. Under about 30 answers, don't report a percentage. Say "insufficient data" instead. Notable does this automatically, and also when the interval is wider than ±15 points.
2. To see a change of a few points, you need hundreds of answers per comparison. Smaller changes than that are usually noise.

How do you tell a real change from noise?

You moved from 30% to 45% this month. Is that real?

Look at the two intervals:

• Last month: 12 of 40 answers, interval about 18% to 45%.
• This month: 18 of 40 answers, interval about 31% to 60%.

The intervals overlap a lot, so the data can't rule out "nothing changed". Collect more answers before you celebrate, or before you cut what you were doing.

Compare that with 120 of 400 (interval about 26% to 35%) moving to 180 of 400 (about 40% to 50%). Those intervals don't overlap, which is good evidence of a real change.

Non-overlapping intervals are a conservative check. Intervals that overlap slightly can still hide a real difference. But it is a sensible rule for anyone without a statistics background.

Why is "one prompt, once a day" not enough?

Many AI visibility tools run each prompt once per assistant per day and report the share of days you appeared. Over a week, that is seven answers per prompt per assistant, which is far too few for a stable rate.

Better practice:

• Several phrasings per question. Two or three ways a buyer would actually ask it.
• Pooling over a window. A rolling 28 days gives far more answers than a single day.
• Separate engines, then pooled. ChatGPT and Perplexity behave differently. Look at each, then at the combined number with its own interval.
• Failed answers excluded, not counted as zero. An answer that failed to collect says nothing about your brand.

We go through the individual metrics in how to measure AI search visibility.

Frequently asked questions

What is a good AI visibility score?

It depends on the category and on how competitive the questions are. A more useful comparison is your rate against your direct competitors on the same questions, measured the same way, with intervals that don't overlap.

Why does my AI visibility tool show a different number to what I see in ChatGPT?

Because what you see is one answer, shaped by your account, location, chat history and the moment you asked. A tool's number is an average over many answers. Both can be correct.

Is a 95% interval the same as being 95% sure?

Roughly, for everyday use: if you repeated the whole measurement many times, about 95% of the intervals calculated this way would contain the true rate. It doesn't mean every value in the range is equally likely.

Should I ignore AI visibility numbers with small samples?

Don't act on the exact percentage. A small sample can still tell you something coarse, such as "never mentioned in 25 answers", which is worth investigating.