Key Takeaways
- Record the baseline before the page exists, not after. Once the page is live you can never recover what the answers looked like without it, and every later comparison becomes a guess.
- Ask each question three times, not once. The same question asked twice gives different answers. One run is an anecdote, and three runs tell you how much this particular question naturally wanders.
- Measure the band before you measure the change. Find out how much a question moves when you change nothing. Anything inside that range afterward is noise wearing the costume of a result.
- Record who else was named, not just whether you were. A named-or-not answer throws away most of the signal. The list of other businesses is what tells you whether anything actually shifted.
- Wait a season before you judge, and accept a null result. A page needs time to be read and re-read. A month later with nothing changed is real information about the page, not a reason to rewrite it that week.
Somebody publishes a good new page on a Tuesday, checks ChatGPT on the Thursday, sees their name where it was not before, and concludes the page worked. It probably did not. Two days is not long enough, and the same question asked twice on the same afternoon often gives different names.
The reverse happens just as often. A page goes up, the answer looks identical a week later, and the page gets rewritten out of impatience before anything had a chance to read it.
Both mistakes have the same root. Nobody wrote down what the answers looked like beforehand, and nobody knows how much these answers move on their own.
How do I measure whether a new page changed AI answers?
Ask a fixed list of questions, in fixed wording, several times each, before the page exists. Wait. Ask the same list the same way weeks later. Compare whether you were named and which other businesses were, and ignore anything that falls within the range the question already wandered across.
That is the entire method. It is unglamorous and it is a spreadsheet, and it is the only version of this that produces an answer you can trust.
The part people skip is the several times each. It feels like padding. It is actually the measurement, because without it you have no way to distinguish a change you caused from the ordinary movement that was going to happen anyway.
Why the baseline has to exist before the page does
Because it cannot be reconstructed. Once the page is live, the answers you would have gotten without it are gone permanently, and every comparison afterward is against a memory.
Write down ten to twenty-five questions in your buyers words, run each one three times in fresh logged out sessions, and record what came back. That is an hour of work and it is the only hour in this process that has a deadline attached, because the deadline is the moment you publish.
Use your buyers phrasing, not your own. A question worded the way you talk about your work will not resemble the question a buyer types, and the two return different answers. Which questions are worth including is in baselining your competitors in AI answers.
Do it by hand the first time even if you intend to automate it later. What the answers actually say is far more useful than a score, and the first round is where you find out that three of your questions were badly worded.
Telling a real change from ordinary variation
Every question has a range it wanders across when nothing has changed. Some questions are stable and some are wildly unstable, and you cannot tell which is which without asking repeatedly.
Best consultant for X in Denver
Inside the band. Not a result.
Who helps with [your narrow specialty]
Clear of the band. A real move.
What does this kind of work cost
Just clear. Re-run before believing it.
Is hiring someone worth it for this
No movement at all.
Alternatives to [the big competitor]
Below the band. Worth investigating.
Two of the five moved in a way worth believing. One moved backward and is worth a look. One never left its own noise. One did not move at all.
Illustrative rather than measured: the tracks show how the method reads, not results from any client. Every question, band, and verdict shown here is also named in the section above or in the table below. The band is not a margin of error you assume. It is something you find out, by asking the same question repeatedly before you change anything.
Row four is the one worth sitting with. No movement is not a failed measurement. It is a clean result that says this question did not care about your page, which is useful before you write a second one like it.
What to write down every time
Record more than whether you were named. The binary is the least informative thing in the answer, and the list of other businesses is where the actual movement shows up first.
| What you measure | What it tells you | How noisy | How to record it |
|---|---|---|---|
| Were you named at all | Whether you are in consideration | Very noisy on one run | Yes or no, per run, three runs per question |
| How many of the questions named you | Breadth of coverage across the niche | Moderate | A count out of the fixed question list |
| Who else was named | Whether the whole answer shifted or just you | Low, and the most informative | The full list of names, in order, per run |
| What the answer said you do | Whether your description is landing correctly | Low | Paste the sentence about you, verbatim |
| Which sources the answer pointed at | Which pages are doing the work | Varies by assistant | Copy every link shown, where links are shown |
The third row is the one most people leave out and the one that pays. If your name did not appear but the other three names changed completely, the answer moved and you nearly made it. If the same three names came back verbatim, nothing moved and the page is not being read yet.
How long to wait, and what nothing means
Four to eight weeks before the first re-check, and do not peek in between in a way that changes your mind. Assistants that fetch live pages can pick something up within days. Anything answering from stored knowledge can lag a great deal longer, as Google's account of how AI features in Search work and the Princeton generative engine optimization study both imply in different ways: what gets used depends on what gets fetched and how quotable it is once fetched.
If nothing changed, check the boring explanations before the interesting ones. Is the page reachable and linked from somewhere. Does it state plainly who it is for. Does it contain a specific claim rather than a description of your philosophy. Most null results are one of those three.
And accept that some pages simply do not move anything. That is a real finding about that page and that question, and it is worth more than a month of rewriting on a hunch. The broader signals worth watching over time are in how to tell if AI visibility is improving, and the no-software version of the whole routine is in tracking AI visibility without buying software.
One honest limit on all of this. Even done carefully, this is a small sample of a system that varies, so treat a single round as evidence rather than proof. Three rounds over three months tell you something real. One round tells you what happened on one morning.
If you want a baseline recorded properly without doing it by hand, the free check asks 25 real buyer questions from your niche through ChatGPT at $0 and emails a plain-English readout within the hour. It runs as soon as you ask for it.
Questions we hear the most
How do I measure whether a new page changed anything in AI answers?
Record the same questions, in the same words, before the page goes live and again weeks later, asking each one at least three times per round. Compare whether you were named and who else was, and treat small movements as noise.
Why do I get different answers when I ask the same question twice?
Assistants do not produce one fixed answer per question. Wording, session history, and the sources fetched that moment all shift the result, which is why a single run cannot tell you whether anything changed.
How many times should I ask each question?
At least three per round, in fresh logged out sessions. Three runs show you how much that question naturally wanders, which is the only way to know whether a later difference means anything.
How long should I wait before re-checking after publishing a page?
Four to eight weeks is a reasonable first re-check. Systems that fetch live pages can pick things up sooner, and anything answering from stored knowledge can lag far longer than that.
What should I record each time I run a question?
The exact question wording, the date, the assistant used, whether you were named, every other business named, and any source the answer pointed at. The other names carry most of the signal.
What does it mean if nothing changed at all?
Usually that the page has not been read yet, or that it did not add anything the answer needed. Both are findings. Check first whether the page is reachable and clearly written before assuming it failed.
Can I measure this without buying a monitoring tool?
Yes. A spreadsheet and a fixed list of questions covers it for a single business, and doing it by hand teaches you what the answers actually say, which a summary number does not.
Can the AnswerHalo free check give me a baseline?
Yes, that is close to what it is for. It asks ChatGPT 25 real buyer questions from your niche and emails your score, who got named in your place, and the one fix to start with. It costs $0 and it runs as soon as you ask for it.
Your buyers are already asking. Find out what AI tells them.
The free check asks ChatGPT 25 real buyer questions about your niche and sends you the report within the hour, at $0. If it shows you are already getting named everywhere, we will say so in plain words.
Get my free check →