We Asked the Same Question Six Times. It Named Six Different Businesses.

The third number above is ours, and it is the reason this article exists. We were building a check that asks an assistant the questions a customer would ask, and we ran one of them six times in a row against the same business. We got six different shortlists. The business appeared in one of them.

Had we run it once and stopped, we would have had a five in six chance of telling that owner they were invisible, and a one in six chance of telling them they were fine. Neither sentence would have been true.

Why the answer moves

Two independent sources of variation stack on top of each other, and it helps to keep them apart.

The first is the model. A language model does not look an answer up, it generates one word at a time by sampling from a probability distribution. That sampling is deliberate: it is what stops the output reading like a form letter. It also means two identical prompts can diverge at the first sentence and never come back together. Ask for the best three bakeries and the model commits to a first name early, then builds a list that stays consistent with the choice it already made.

The second is retrieval. When an assistant answers a local question well, it has run a live search first and is summarising what came back. That search is subject to everything an ordinary search is subject to: freshness, ranking shifts, whatever a publisher pushed this morning. The pages returned at nine are not reliably the pages returned at five.

You are not reading a record of what the assistant thinks about you. You are taking one sample from a process that will answer differently tomorrow.

The gap this sits inside

The variance would be a curiosity if the stakes were small. They are not, and the reason is the distance between how many people now ask and how few businesses ever get named.

Being named is rare. That is what makes a single no so easy to misread: the base rate is already low, so a negative answer is the expected outcome whether or not anything is wrong with your listing. A single negative confirms almost nothing. The signal is in the rate.

The free version of this, in ten minutes

You do not need a product for this, and you should not take our word for any of it. Here is the whole method.

1. 01

   Write three questions a customer would actually type

   Not your business name. The job someone is trying to get done: best pizza near Verdun, quiet cafe to work from in Mile End, where can I get a last-minute birthday cake. Use the words and the language your customers use, not the ones on your signage.

2. 02

   Ask each one five times, in a fresh chat every time

   The fresh conversation matters. Inside a single thread the assistant is anchored by what it just said, so five asks in a row give you one answer wearing five hats.

3. 03

   Count, do not read

   For each question, write down how many times out of five you were named. Three numbers out of five. That is the entire measurement, and it takes about ten minutes.

4. 04

   Write down who else got named

   This is the half people skip and the half that pays. The names that keep coming back are your real competitive set in the eyes of the thing your customers are asking, and it is not always the set you assumed.

5. 05

   Repeat monthly, same questions, same wording

   Changing the wording resets the baseline. The number is only useful against the same number a month earlier.

If you would rather not run it by hand, our free AI visibility check asks three questions of an assistant with live search and gives you the same two numbers, plus the names it reached for instead. No account, and it shows nothing at all rather than guessing when the daily allowance runs out.

What actually moves the number

Once you have a rate, the obvious question is what raises it. The honest answer is that nobody optimises an assistant directly, because there is nothing there to optimise. It reads. What you can change is what there is to read about you.

That means a page of your own that is current, because a business website is the single largest bucket of sources behind local AI answers and a directory listing does not sit in that bucket. It means reviews that are recent rather than merely numerous, because freshness is what a retrieval step rewards. And it means the details agreeing with themselves everywhere they appear, because a model reconciling three different sets of opening hours will often just pick a business whose hours are not in dispute.

None of that is new advice. What is new is that you can now measure whether it worked, in the place your customers are actually asking.

The mistake worth avoiding

The tempting move, once you see a bad number, is to ask the assistant why. It will happily tell you. It will explain that your reviews are too sparse, or your page too thin, or your category too crowded, in fluent and confident prose.

Treat that explanation as fiction. The model has no access to its own retrieval step and no memory of why it picked the names it picked. Asked to justify a choice, it produces a plausible rationalisation, which is a different object from a reason. The list of names is evidence. The account of how the list came to be is not.

The names are data. The reasons the assistant gives for them are written after the fact, and they are worth nothing.

Questions people actually ask

Related: why a Google profile is not a website in citation terms, and why recent reviews outweigh total review count.