How Many Times Should You Check Your AI Visibility Before the Number Means Anything?

Somebody asks ChatGPT to recommend a coach in their city, sees a competitor's name, and concludes something about their business. Then they ask again an hour later and get a different list.

Both answers are real. Neither is a measurement yet. The question underneath is how many looks it takes before a number is telling you about your business rather than about the machine.

The answer is smaller than people expect for the first reading, and larger than they expect for the second.

How many checks before the number means anything?

About 25 different buyer questions, asked once each, is enough for a first reading. Below ten, a single different answer swings the result far enough to mislead you. Above 25, you are mostly buying the ability to detect smaller changes later.

That is a sample-size answer, and it depends on one thing being true: the 25 have to be 25 different questions, not one question asked 25 times. Those measure different things, and only one of them is about you.

It also depends on what you are asking the number to do. "Do I appear at all" needs far fewer observations than "did last quarter's work move anything", because the first is a question about a level and the second is a question about a difference. Differences are always harder to see.

What one run can already settle

More than people assume, when the result is lopsided. If your name does not appear in any of 25 answers, you have learned something on the first run, and a second run is unlikely to rescue it.

The reason is that the failure modes are not symmetric. There are many ways to be absent from an answer and only a few ways to be present in one. A business that assistants can find easily tends to show up somewhere across 25 different phrasings, even on a bad day. Scoring zero across all of them is a pattern, not a coincidence.

The middle is where single runs stop being useful. Named in 6 of 25 on Tuesday and 9 of 25 on Thursday is, in most cases, the same situation described twice. Treating the gap as progress or decline is the most common misreading of these reports, and it is covered along with the others in how to read an AI visibility report.

Why the variation exists at all is worth knowing before you size anything. OpenAI's own explanation of what ChatGPT is describes a system that generates a response rather than retrieving a stored one. Perplexity's help center describes answers built from sources fetched at the moment you ask, and Google says much the same about its AI features. Two runs an hour apart can differ without anything about your business having changed. The longer version of that mechanism is in why the same question gives different answers.

The arithmetic of one different answer

The size of the run sets how much a single answer can move the score. That is arithmetic, not statistics, and it is the most useful thing to have in your head when you read one of these numbers.

HOW FAR ONE DIFFERENT ANSWER MOVES THE SCORESOURCE: ARITHMETIC

• 5 questions 20 points

  One different answer and the score has moved a fifth of its range.

• 25 questions 4 points

  The size of the free check. One answer is a nudge, not a swing.

• 125 checks 0.8 points

  The size of the paid audit, spread across five tools.

Each track is the full range of the score. The lit block is the share of that range that moves when exactly one answer comes back differently. A small run is a coarse instrument, and no amount of care in reading it makes it finer.

AnswerHalo answerhalo.com

CLICK TO ENLARGE

Five questions is a coarse instrument. One answer flipping takes the score from 40 to 60, which reads as a transformation and is nothing of the kind. This is why a quick test on a handful of prompts produces such confident and such wrong conclusions.

Twenty-five questions puts one answer at 4 points. That is fine enough that the number carries information and coarse enough that you will not chase single-mention wobbles. It is the size the free check runs for exactly this reason.

A hundred and twenty-five checks puts one answer at 0.8 points, and buys something else besides precision: spread across five tools. Two businesses with the same total can have completely different shapes underneath, and only the wider run shows you whether you are losing everywhere or losing in one place.

Spread the checks across questions, not repeats

Twenty-five different buyer questions beats one question asked 25 times, every time. The repeats measure the tool's variation. The spread measures your visibility, which is the thing you were trying to find out.

Build the list from how buyers actually talk. A mix of broad questions ("best strength coach for runners"), local ones ("strength coach in Denver"), situation ones ("coach for someone coming back from a knee injury"), and vetting ones ("what should I ask a coach before signing up"). Spread across those four shapes matters more than the exact count, because your business can be invisible in one shape and fine in another.

Then hold the method still. Same wording every time, a fresh session for each question, logged out where you can be, and the same tools in the same order. You are the easiest variable to remove and the one most likely to flatter you, because your own account history makes you look more findable than a stranger would find you.

Record the businesses named, not just whether you were. The rival list is stable in a way the score is not: the same three or four names tend to recur across phrasings and across runs, and that recurrence is a stronger signal than any single total.

What it takes to prove something changed

Two runs of the same 25 questions, run the same way, far enough apart for the work to have landed. That is the real reason to care about sample size, because detecting a difference is much harder than detecting a level.

A working rule for a 25-question run, offered as a rule of thumb rather than a statistical test: a move of one or two mentions is noise, a move of four or more is worth investigating, and anything between the two calls for a repeat before you act. The same rule scaled to 125 checks moves those boundaries proportionally.

Leave enough time between runs. Pages have to be found and read before they can be used, and that takes longer than publishing feels like it should. Re-running weekly mostly samples the tools' own variation and produces a chart that goes up and down for no reason, which is a good way to spend a quarter reacting to nothing. The cadence question has its own post: how often you should check.

Here is the admission this post owes you. For most small coaching and consulting businesses, the first run comes back at or near zero, and every argument above about sampling is beside the point. Sample size matters when there is a number to be precise about. When the answer is "you do not appear anywhere", one run has already told you what you needed, and the useful next step is the rival list rather than a second measurement.

If you have not taken the first reading yet, the free check is 25 questions put to ChatGPT for your niche at $0, and it emails a plain-English readout: your score, who got named in your place, and the one fix to start with. It runs as soon as you ask for it, and the report lands within the hour between 7am and 5pm Central, or by 10am the next morning outside those hours.