AI Visibility Tracking Verification: How to Test Tracker Accuracy

What Visibility Scores Actually Measure

A visibility score is a ratio. The tracker sends a set of prompts to one or more AI products, reads the answers, and counts how often your brand or domain shows up. The count is divided by the number of prompts or answers, and the result is a percentage, an index or a rank against competitors. Each vendor picks its own numerator, its own denominator and its own weighting, so a 32 in one tool has no relation to a 32 in another.

AEO (answer engine optimization) measurement tools all follow this pattern. Whether a vendor sells itself as an AEO tracker, a GEO tracker or an AI visibility platform, the output is still a ratio built from sampled answers, and the same verification questions apply.

The numerator is the first thing to pin down. Some tools count a mention of the brand name in the answer text. Others count a URL from your domain in the source list. Others add the two together, or weight a first-position mention more heavily than a later one. A page that is cited but never named, and a brand that is named but never cited, can produce the same score. They tell you different things about your site.

The denominator matters just as much. The prompt set is chosen by whoever configured the tracker, and it usually reflects what that person expected to rank for. Change ten prompts out of fifty and the score moves without anything changing on your site or in the model.

Language models also do not return the same answer twice. A score built from one run per prompt is a single draw from a distribution. If the tracker reports one number with no range and no sample count, you cannot tell a real shift from noise.

The most important gap is origin. A model can mention your brand because it learned about you during training, or because it ran a search and pulled your page into the answer. Both outcomes look identical in the response. The first reflects what the model absorbed months ago. The second depends on your current content, your crawl access and your ranking in whatever search layer the product uses. A score that blends them hides which one you are looking at, and the fix for each is different.

Scores also say nothing about answer quality. A mention can be accurate, outdated, or attached to the wrong product, and most scores count all three the same way.

So before you put a visibility number in a report, you need answers to a few questions:

• What counts as a hit?
• How many prompts and how many repeats stand behind the figure?
• Was web access on or off when the answer was generated?
• Can you read the stored answer text?

The next sections break the score into its component signals and give you tests you can run yourself.

Three Signals: Training Data, Live Search, Citations

A single visibility score can bundle three different signals, and each one responds to different work on your site.

Training data (parametric memory). The model answers from what it absorbed during training and does no web lookup. If it names your brand, it is recalling it. Anything you publish today cannot change that answer until the provider trains a newer model, and you have no say in when that happens. A tracker that queries a model with search switched off measures this signal.

Live search (retrieval). The model runs a search, reads pages and writes the answer from what it found. Your content, bot access to your pages and your position in the underlying search index all feed into the result. Only this signal can move within days or weeks of a site change. A tracker that queries with search on measures retrieval, but the result still carries memory in it.

Citations. These are the links displayed next to an answer. A citation shows which URL the product displayed. It does not prove the claim came from that page. Some products attach sources to a finished answer, and the behavior differs between products and changes between versions. Check what a given product does before you rely on it.

The overlap causes most of the confusion. With search on, a brand mention can come from memory, from retrieval, or from both, and most dashboards still report one number.

You can separate the signals with a short manual routine. Pick one prompt and one model, then run it twice.

1. Run it with web search off. If your brand appears, you are looking at memory.
2. Run it with web search on. If the brand appears only now, retrieval produced the mention.
3. Open every cited URL. Find the sentence in the answer, then find the supporting text on the page. Note whether the page loads, whether it is the page you expected, and whether it backs the claim.
4. Log three values per run: brand mentioned, brand cited, cited page supports the claim.

The combinations are informative. A mention with no citation points to memory or to retrieval the product did not display. A citation with no mention means the product showed your page while the answer ignored your brand. A mention, a citation and a supporting page together are the strongest case you can make to a client.

Run each prompt several times before you trust a result. Model output varies between runs, so one sample can show a brand that appears in a minority of runs, or hide one that appears in most.

Before you read any score, ask which of the three signals it reflects. If the vendor cannot answer, run the routine above on five of your own prompts this week.

Six Practical Tests for Tracker Accuracy

Run these tests on your own tracker before any number goes into a client report. Each takes under an hour with a small prompt set, and ten prompts is enough to start.

1. Toggle web search. Take a prompt where your brand should appear and repeat the off/on routine from the previous section in the same model. If the brand shows up in both runs, the model is recalling it from training data. If it appears only with search on, you are looking at live retrieval. Then ask the vendor which mode the tracker runs. A vendor who cannot answer is selling you a score you cannot interpret.

2. Audit the sources. Open every citation URL the tracker reports for a sample of prompts. Check that the page loads without a 404 or a surprise redirect, that it relates to the question, and that the cited claim exists on the page. Then compare the URL list against the answer text you collected yourself. A citation that was attached after the answer was written does not prove the answer came from that page.

3. Repeat the prompt. Run one prompt five or more times, in fresh sessions and at different times of day, and note how often the brand appears. If it shows up in three of five runs, a single-run tracker will report either 100% or 0% depending on which run it happened to sample. Ask how many samples the tool takes per prompt and whether it reports a range or a single value.

4. Vary location and language. Run the same prompt from two countries and in two languages, then compare against what the tracker reports. Find out whether the tool queries a model API or drives the consumer interface. The two can return different answers, because consumer products add their own system prompts, search layers and personalization. Test the version your clients’ customers actually use.

5. Add control queries. Include a brand that certainly appears in the answers, such as a market leader in your category, and a brand you invented that cannot exist. The leader should score high. The invented brand should score zero. If the fake name registers any visibility, the tool is counting noise, probably through loose string matching or fuzzy entity resolution. Keep both controls in every future run as a standing sanity check.

6. Cross-check server logs. Live retrieval has to fetch pages, and fetches leave traces. Run a prompt that should trigger search, then look in your access logs for AI-related user agents within a few minutes. OpenAI documents separate user agents for search, training and user-triggered requests, so read which one means what before you interpret a hit. Google lists its own crawlers in its crawler documentation. A tracker claiming that an engine retrieved your page while your logs show no matching request deserves a direct question to the vendor. Keep in mind that a cached copy can serve an answer without a fresh fetch, so absence in the logs is a lead, not a verdict.

Write down the date, model, mode and prompt set for every test. The exercise only pays off if you can run it again next quarter and see whether the tracker’s behavior changed after a model update.

If you want a small first experiment, take ten prompts, run tests 1, 3 and 5 on them, and log the results in a spreadsheet. That alone will show you whether the tracker’s numbers are stable enough to put in front of a client.

Questions to Ask Before Choosing a Measurement Tool

The tests above tell you how a tracker behaves. Before you buy one, put these questions to the vendor in writing. Vague or missing answers tell you how much the score can carry in a client report.

Which model and version does it query? “ChatGPT” is not an answer. Ask for the exact model name and version, and ask how the vendor handles version changes. A model update can move scores overnight with no change on your site. If the dashboard doesn’t log the version per run, you can’t tell a real shift from a model swap.

Is web search on or off? This is the question from the live-versus-training split. Ask whether the tracker runs prompts with search enabled, disabled, or both, and whether the setting is recorded for each result. A tracker that mixes both modes into one number is blending two different signals.

How many repetitions per prompt? Ask how many runs feed each data point and whether you get a range or a single value. A vendor who reports a lone percentage without a sample size has not told you its precision.

Does it store the full answer text? Stored responses let you audit any score yourself. Without them you are trusting a number you can’t trace back. Ask how long the text is kept and whether you can export it.

How does it extract citations? Ask whether the tool reads the sources the product returns or infers them from the answer text. Also ask whether it checks that the cited URLs resolve. A tool that equates a citation with the origin of the answer overstates its case.

Does it query through an API or the consumer interface? Output from the two can differ, and your client sees the consumer product. Ask which one the tracker uses, and which location and language settings it applies.

Is the methodology written down? Look for a public page that covers the points above. If the vendor can only explain it on a sales call, you can’t cite it in your own report.

A quick checklist to carry into the trial:

• Model name and version logged per run
• Search mode recorded per result
• Run count and variance shown
• Raw answers exportable
• Citation extraction described and URLs checked
• Query surface (API or interface), location and language stated
• Methodology published

Then run the control test from the previous section during the trial. If the invented brand scores any visibility, the tool is measuring noise, whatever the vendor’s documentation says.

How to Report Visibility Data to Clients

Once you have picked a tracker and tested it, the next step is the report. Report AI visibility as a range with a direction, never as a single number. A tracker that shows “42% visibility” has sampled a non-deterministic system a limited number of times. Write “between 30% and 50% across 10 prompts, 3 runs each, trending up since March.” The range tells the client how much weight the number can carry. The direction is the part they can act on.

State the method next to every figure. A client should be able to read the footnote and know which model was queried, whether web search was on, how many repeats ran, and on what date. If your tracker does not disclose those details, say so in the report. A number without its method cannot be defended in a meeting, and you will be the one defending it.

Keep the three signals in separate columns. Mentions with web search off come from training data, and a site change will not move them for months. Mentions that appear only with web search on come from live retrieval, and those respond to content and crawl access. Citations are the third column, and a linked source does not prove the claim came from that page. Merge the columns and a client will expect last week’s content update to show up in a number it cannot influence.

Add a section called “What we cannot measure.” Include at least these items:

• Answers given to real users in their own locations, languages and personalized sessions. Trackers query through an API or automated interface, and what a person sees can differ.
• Models that run with web search switched on by the user or by the product, when the tracker does not say which mode it used.
• Traffic that never reaches your site. An AI answer can mention a brand and send no click, so analytics will not show it.
• Cause and effect. A rise after a content change is a correlation, and one quarter of data rarely separates it from model updates.

Put the control query results in the appendix: the certain brand and the invented brand, run through the same prompts. If the invented brand scores above zero, tell the client the tool produces noise and explain how you adjusted for it.

Finish the report with decisions rather than scores. Name two or three actions the client can take, such as fixing pages that crawlers cannot read, and the signal you will check afterward. Clients keep reading a report that tells them what to do next. They stop trusting one that only gives them a chart.

Site Readiness: What Trackers Can and Cannot See

Seodisias is a desktop crawler with an AI Ready check, and the two stop at the edge of your site. The crawler fetches your pages the way a bot would, renders JavaScript, and reports what it finds: status codes, redirect chains, canonical tags, robots.txt rules, structured data, and whether the content shows up in the rendered HTML. The AI Ready check looks at the same site for signals that matter to AI crawlers, such as whether robots.txt blocks them and whether the main content is readable without a script running.

That covers your side of the connection. A tracker measures the other side, which is what a model says when someone asks it a question. Those are two separate questions, and a green AI Ready result does not tell you whether ChatGPT, Gemini, or Perplexity will mention your brand.

Seodisias does not track brand mentions, run prompts against AI engines, or score citations. It has no visibility metric, and nothing in it can confirm or refute a number a tracker gives you. If a vendor claims a crawler can validate their scores, that claim is wrong.

What the crawler does give you is a clean baseline for the tests above. Before you blame a tracker for a low score, rule out your own site. A page that returns a 403 to bots, hides its text behind client-side rendering, or sits behind a redirect chain will be missed by live retrieval no matter how good the tracker is. Crawl first. If the crawl shows a blocked or empty page, the tracker’s low number is probably correct, and the fix is on your server, not in the report.

The reverse also holds. If the crawl is clean and the tracker still shows you absent from live answers, the site is no longer the first suspect. Now the tracker’s method deserves the scrutiny: which model, whether web search was on, how many repetitions, and whether the cited URLs open. You can run the web-on versus web-off comparison and the log check from the earlier tests against that question.

Seodisias runs locally on Windows, macOS, and Linux. It is free, has no URL limit, and needs no sign-up, so the crawl data stays on your machine. That matters when the audit covers a client’s staging site. Run the crawl, keep the export next to your tracker data, and note the date. When a client asks why the score is low, you can show which part of the problem was the site and which part was never measurable.

Download Seodisias, crawl your site, and run the AI Ready check before you test your next tracker.