Preserve AI Visibility Comparisons When Models Change

Freeze the comparison before switching

Treat a known version migration as a change to the measuring setup. Save the old configuration, the intended new configuration and the migration time with an explicit UTC offset. The event should be supported by an actual configuration change or documented provider notice. A different answer alone does not establish a version change.

The OpenAI prompt engineering guide says model output is non-deterministic and that snapshots within a model family can produce different results. It recommends pinning applications to specific snapshots and using tests and evaluation suites when changing model versions. These recommendations support version control and change assessment. They do not guarantee identical answers.

Collect an overlap from both versions

Choose a small batch from the original question set before reading the new results. Include questions that matter to the existing measurement; keep their wording intact rather than rewriting them for the new model.

1. Freeze the exact question text, accompanying instructions and comparison rules. A qualifying brand mention or citation must mean the same thing on both sides.
2. Use the same search-tool configuration for both versions. Collect within nearby observation windows and save each actual collection time with its UTC offset; proximity reduces one source of variation without eliminating it.
3. Preserve a pair of complete raw answers for every question, together with the source links and passages used to support each judgment. A summary alone cannot show what changed.
4. Mark any pair that did not keep the agreed configuration. Do not present it as evidence from the controlled overlap.

Keep the shared setup alongside the existing measurement environment record. The extra deliverable here is the overlap between model versions, rather than another general environment checklist.

Record the version you can actually observe

For each side, preserve the requested model identifier and any returned or displayed identifier that the collection surface actually exposes. Keep their literal values. If no returned identifier is available, write “unavailable”; do not fill the gap with a guessed snapshot. Also preserve the evidence that identifies the migration event.

An unchanged alias is insufficient evidence of a fixed underlying version. For example, OpenAI's chat-latest documentation says its underlying snapshot will be regularly updated. This describes the alias's behavior, but does not establish when an unseen change occurred in your own test.

Neither a stable alias nor a changed answer reveals an inaccessible backend version. Describe what the interface or official notice establishes, and leave the rest unobserved.

Read the bridge by question

Use one row per original question. Each side needs its own raw answer, judgment and supporting passage or source evidence. State exactly what is the same or different under the frozen rule. Two matching mention decisions can still contain different wording or citations.

The following rows illustrate possible comparisons, not collected results.

The bridge table proposed here is a measurement method, not an official GEO scoring rule. General evaluation principles in OpenAI's evaluation best practices include defining success criteria and metrics, running and comparing evaluations, and evaluating changes.

The retrieved sources may also change during the overlap. If source URLs or supporting material differ, record that difference alongside the answers. The pair shows an observed result under two configurations; it does not isolate the model as the sole cause. Keep incomplete pairs visible so a reader can see which questions lack a comparison.

Keep the curve break and start a new baseline

Leave the historical series as originally measured. Label the migration point and begin a separate segment for the new version using its own baseline. State which questions and completed pairs support the overlap. A bridge sample selected for diagnosis should not silently become the denominator for the whole visibility series.

Do not subtract the bridge difference from past results or add it to future results to manufacture one continuous curve. The overlap describes differences for those questions and that observation window. It supplies context for interpreting the break, without establishing a universal conversion factor. Continue the ordinary measurement schedule under the AI visibility monitoring method after the new segment is established.

When the old version is already unavailable

Record “bridge unavailable” with the reason and the migration time. Retain the old raw answers, but identify them as historical observations rather than a fresh parallel collection. Start the new baseline independently and disclose that the comparison spans different versions and observation windows.

Reports may describe what was observed before and after the switch, with that limitation attached. They should leave the cause unresolved where the evidence cannot separate model, retrieval and content changes. No bridge or new baseline guarantees future inclusion, citations or rankings.

Related reading

• Record AI Visibility Environments and Repeat Tests
• AI Visibility Evidence: Attempt Selection and Correction History
• How to Measure AI Visibility: Metrics and a Reproducible Method
• Our agency reports AI visibility but nothing changes — how do we audit that?
• SaaS · B2B GEO: Getting AI to Recall You When Buyers Ask