Can You Trust AI Visibility Data?

AI visibility numbers are samples, not facts, and you can trust them when they rest on enough runs, a monthly trend, app answers rather than API ones, and a tool that shows the raw answer behind every number.

So yes, you can track brand mentions in AI search. But the doubt about the numbers is fair. One reading from one tool on one day can be far from the truth. This guide is part of our guide to measuring AI visibility.

Can AI visibility be measured at all, or is it guesswork?

It can be measured. A tool asks AI engines a fixed set of prompts and records each answer. Then it counts the share of answers that name your brand. Every recorded answer is real: an engine wrote it, for that prompt, in that country, on that day. The number is a share of answers to your prompts, so the prompts decide what it means. That is why choosing the prompts to track comes first.

So the counting is not a black box, as long as you can read the answers behind it. The limits are elsewhere. The answers come from the prompts you chose, not from a buyer's own chat, which no tool can see. And the engines answer differently each time. So any number is a sample of what an engine says, never the full picture.

That puts AI visibility between performance marketing and brand marketing. Do not expect an exact value per point, as you would from an ad campaign. But do not give up on measuring it either.

Why is a single AI visibility number not a fact?

Because the engines rarely give the same answer twice. We looked at data from the Zadoosh data platform (April to October 2026, B2B SaaS brands). When ChatGPT answered the same prompt twice on the same day, the two answers shared only about 3 in 10 of the sites they cited (31%). When one answer named a brand, the rerun named it again only half the time (50%).

Most prompts behave like this. In the same data, a brand was named in some answers to a prompt and not in others in about 3 in 4 months (75%). For those prompts, one run came within 10 points of the brand's share for the month only about a third of the time (32%). About half the time, it was off by more than 20 points.

Waiting a day does not fix it. A ChatGPT rerun the next day shared about a quarter of the cited sites (26%). A week later, it shared about a fifth (21%). Other engines are no steadier. In Gemini (June to October 2026, B2B SaaS brands), two answers on the same day shared about a quarter of their cited sites (25%). So a daily chart mostly shows the engine changing its answer, not your visibility changing.

A real person adds more variation. ChatGPT can remember details from past chats and use them in later answers. So a check from your own logged-in account carries context about you that a buyer's first question does not. The most common mistake we see is one check, often logged in, treated as the truth. Why answers vary, and how to check from a clean session, is in does ChatGPT give everyone the same answer.

How many runs before a number means something?

Enough runs that the noise is smaller than the change you care about. In the same data (April to October 2026), one run was off by about 19 points on average. Three runs were off by 13, five by 11 and ten by 9. With ten runs, the share came within 10 points of the month's figure about 6 times in 10 (64%). It came within 20 points nearly 9 times in 10 (87%).

Each extra run helps, but some error always stays. For a first snapshot, such as an AI visibility audit, get 4 to 5 answers per prompt, engine and country. For ongoing tracking, get about 10 or more: one run a day per prompt, read as a monthly trend. On a small budget, track fewer engines but keep the prompts and the daily runs.

Why read the direction over a month, not the daily number?

Because the share itself moves over time. In the same data (April to October 2026), we compared the first four weeks of each prompt's history with the last four, at least two months apart. The brand's share moved by a median of 15 points. About 4 in 10 prompts moved by more than 20 points (38%). For comparison, noise alone moved it by a median of 5 points.

So a baseline is a starting point, not a fixed truth. The engines change how they find sources, and the real answer changes too. Collect data as often as you can, ideally daily. But set strategy from the trend over a month or more, never from one week. A rise of 3 to 5 points in a month is a good sign.

Before you take credit for a rise, check your competitors. If they all rose too, the engine probably changed. The best proof is a page you made being retrieved, then cited, and then your mentions going up. How to check each of those steps is in did your AEO/GEO change really work?

Why do different AI visibility tools show different numbers?

Because each tool runs a different test. We hear this often, and one marketer on Reddit tried 20 tools and got a different score from every one. Two tools can both record answers correctly and still disagree. They ask different questions, in different places, and count the answers differently.

So two numbers far apart do not mean one tool is lying. Usually the tools measured different things. You cannot compare a score from one tool with a score from another. Only each tool's own trend means something.

What should you do when two tools disagree?

Three steps settle most disagreements:

1. Compare what each tool does differently. Use the rows of the table above: prompts, engines, country, runs per prompt and how each counts a mention. The gap usually has a plain cause. For example, one tool includes branded prompts, or counts each country as a separate prompt.
2. Pick one tool and read its trend. A method that stays the same for months is worth more than a number that looks more correct. Switching tools resets your history.
3. Spot-check a few prompts by hand. The steps are below. Keep the tool whose recorded answers match what you see.

What can you trust in an AI visibility tool?

Four things hold up well, if the tool shows its work:

• The recorded answers. What an engine said, for a prompt, in a country, on a date, is a fact. Everything else is built from these.
• The trend from one method. Keep the same prompts, engines, country and schedule for a month or more. The trend then shows whether you are gaining or losing ground, even when each daily figure is noisy.
• The gap to competitors, measured the same way. If a rival is named far more often than you on the same prompts, that gap is real, even when neither exact percentage is. A good score is relative, never a universal number.
• The sources the engines cite. The pages an engine cites for your prompts show why a brand is or is not in the answer. When you drop out, first check whether the source that used to name you dropped out too.

Which AI visibility metrics should you stop trusting?

The more a number squeezes into one figure, the more it hides. Do not take these at face value:

• Position, such as "rank 3". AI answers have no fixed list of positions. Where a brand is named moves from answer to answer. In our data (April to October 2026), the first brand named changed in about 4 in 10 same-day ChatGPT reruns. Track average position only once your visibility is decent, and read it as a rough signal.
• Share of voice against a hand-picked list. It depends on the competitor set. Against brands you picked, it can look strong while the brands AI really names are missing. Measure it against the brands that appear in the answers.
• A citation count of every source. Some tools count every page cited in an answer, from a listicle to a Reddit thread. Many of those pages do not name you at all, and they change more between answers than brand mentions do. The citation count that matters is your own pages cited. Ask which one a tool shows.
• One blended score. A single number across engines, countries and metrics hides which part moved. Read each engine on its own before you read the total.

Two more traps come from how a tool handles its own data:

• History that changes. Some tools reprocess old answers, for example with a new rule for counting mentions. Then past numbers move and your trend line breaks. That is not fraud, but you need to know. Ask whether history can change, and keep an export of every number you report.
• Zeros from failed runs. If an engine fails or an answer cannot be read, that day was not measured. It was not 0%. A tool that shows a failed day as zero shows drops that never happened. Gaps should show as gaps.

How do you check whether an AI visibility tool's numbers are accurate?

Check two things. Does the tool show its evidence? And does its evidence match what you see yourself? This takes about an hour. Do it before you report any number to a client or a CMO.

1. Open the raw answers. Pick one number and click through to the answers behind it: the full text, the cited links, the engine, the country and the date. If a tool shows a score but not the answers, you cannot check the score. Use only tools that show the raw answer behind every number.

2. Spot-check a few prompts by hand. Choose a few prompts that matter to you. Run each one a few times, logged out, in a private window, from the same country as the tool. Then compare with what the tool recorded on the same days. Do not expect identical answers, since answers change from run to run. Check whether your runs fit what the tool recorded. If the tool says you are named in most answers and you never see your brand, find out why.

3. Read the method. A tool you can trust answers these plainly:

• How many times is each prompt run, and how often?
• Which country and language does each answer come from?
• Can you see and edit the prompts?
• Is each engine shown on its own?
• How is a mention counted: exact name, variants, and what about a name that is also a common word?
• Are branded prompts kept out of the visibility share? They show how AI describes you. That is a positioning check, not a visibility measure.
• How does a failed run show up?
• Are answers taken from the app buyers use, or from the API? App answers (the chat in a browser) are closer to what buyers see. Prefer a tool that captures the app, or check a sample yourself.

When you report a number, report its sample too: how many prompts, which engines and country, how many runs per prompt, and which dates. A visibility percentage on its own means little. Compare it with "buyer prompts, ChatGPT, US, one run a day for a month, up from last month, still behind the top competitor". A client or a CFO can act on that.

Checks by hand tell you whether the tool is right. They do not measure your visibility. Once the tool passes, let it do the repeated runs. By hand, you will never collect enough answers to read a trend.

Is AI visibility a vanity metric?

It becomes one when it measures the wrong thing. Three versions are common. The prompts are questions no buyer asks, such as definitions or prompts that name your brand. The mention describes you wrongly. Or one snapshot is reported as a result. A mention with the wrong words does not drive demand, because the visitors it sends expect something else.

It is a real metric when three things hold. The prompts are the questions buyers ask while looking for a solution. The number is compared with your top competitors on the same prompts. And you read it as a trend over about three months. There is no universal good percentage, only a share that rises against the brands you compete with.

Do not expect it to match your lead count either. A tool samples your prompts. It cannot see the chats buyers actually had. So the tool's numbers and the meetings that came from AI answers measure different things. To link AI answers to leads, ask every lead how they heard about you. Report visibility next to that as the early signal.

Frequently asked questions

Is it possible to track brand mentions in AI search? Yes. You run a fixed set of buyer prompts on each AI engine at regular intervals, record every answer, and count the share of answers that name your brand. Run daily and read as a monthly trend, that share is a usable number. A single check is not.

Are AI visibility tools scams, or are their numbers made up? A good tool's numbers are counts of real answers it recorded. You can verify them if it shows the raw answer behind each number. What a number cannot do is stand for every answer an engine gives, because answers change from run to run. Be wary of a tool that hides its answers.

Why did my historical AI visibility data change? Usually because the tool reprocessed past answers, for example after changing how it counts mentions. Ask the vendor whether history can change, and keep exports of the numbers you have reported.

Should a failed data source count as a zero AI visibility score? No. If an engine failed or an answer could not be read, that period was not measured. Counting it as zero creates a drop that never happened. It should show as a gap.

Can I compare visibility scores from two different tools? No. Tools differ in prompts, engines, countries, runs per prompt and how they count a mention, so their scores measure different things. Pick one tool, read its trend, and spot-check a few prompts by hand.