Should a Follow-Up AI Answer Share a Fresh-Session Baseline?

Three conversations that change the question

The same short prompt can ask for very different things depending on what came before it. The examples below use anonymous placeholders and invented conversations. They are illustrations, not test results.

In the first case, a later mention of [Brand A] belongs to a guided comparison. The second case tests whether recommendations respect existing constraints. A fresh session containing only its current prompt would leave those constraints unstated. The third case tests the usefulness of a continuation: removing the preceding answer also removes the claim the reader wants to examine.

Save the full transcript behind each example in a real study. A summary such as “discussed hosting earlier” can hide wording that shaped the answer.

Keep enough evidence to reconstruct the turn

OpenAI's conversation-state guide describes supplying previous messages to retain context across requests. That API example supports keeping the history; it does not prescribe the reporting groups used here.

Capture the conversation before deciding which group its answer belongs to. A practical record has the following fields:

1. session_id: an identifier connecting every saved turn in the tested conversation. If the interface exposes no identifier, assign a local one and label it as local.
2. turn_number: the current user-and-assistant exchange, with a stated counting convention so different reviewers count the same way.
3. preceding_transcript: every earlier prompt and answer in order, preserved verbatim. An empty value means no earlier turns in the tested session.
4. current_prompt: the exact text submitted for the answer being scored. Keep this separate from the preceding transcript.
5. answer_text and visible_citations: the answer as shown, plus citation labels and destination URLs that are visible. Record “none visible” when appropriate.
6. test_group and context_inputs: fresh session or follow-up, together with any brand, product requirement, or preceding claim supplied by earlier turns.

These fields explain what the answer was responding to. A visible citation is evidence of what the interface displayed; it does not, by itself, explain why a brand appeared. Keep the answer screenshot alongside the transcript so a reviewer can inspect both the wording and citation placement.

Choose a comparison that fits the reader task

There are two useful comparison designs, each answering a different question.

1. Keep fresh-session discovery separate. Submit the intended question in a session with no earlier turns in that test. If the question is meant to measure brand-free discovery, exclude brand names from the prompt too. Report mentions against that question set. This design is useful for asking what a reader encounters before supplying a shortlist, but it tells you little about the quality of later evaluation.
2. Build a matched follow-up set. Fix the preceding prompts and answers, then test the same current question with that context supplied. Keep the complete preceding text attached to each sample. This design is useful for comparing how answers handle a reader's established requirements, but its mentions remain conditional on the supplied context. If you cannot reproduce the same preceding text, record the difference and avoid presenting the samples as matched.

Do not delete the transcript and relabel a follow-up as a fresh-session answer. For repeated multi-turn tests, give each replay its own session identifier while retaining a shared context-set identifier. That lets reviewers distinguish a new run from a new reader task.

For metrics and a repeatable measurement process, see measuring AI visibility. If the broader test setup also needs documentation, keep that work in a separate AI visibility measurement environment record. The grouping here concerns the conversation that the scored answer continues.

Make the reporting boundary visible

A report should show the question count and result count within each group. “Mentioned in a follow-up after the reader supplied [Brand A]” is a different finding from “mentioned in a brand-free fresh session.” Neither needs to be discarded; each needs a label that preserves the task it measures.

When a report contains both, show separate denominators and explain the inherited inputs beside the follow-up results. If an overall figure is requested, disclose which groups it combines and keep the individual figures available. Do not present that combined figure as a fresh-session discovery rate. These are also useful questions when auditing an agency's AI visibility report.

Where the preceding transcript is missing, leave the sample unclassified until the evidence is recovered. A clean answer screenshot cannot establish that the conversation started with the visible prompt. The practical decision is whether the record supports the claimed reader task, not whether the final answer looks comparable.

Related reading

• Record AI Visibility Environments and Repeat Tests
• How to Measure AI Visibility: Metrics and a Reproducible Method
• Our agency reports AI visibility but nothing changes — how do we audit that?
• AI Visibility Evidence: Attempt Selection and Correction History
• Preserve AI Visibility Comparisons When Models Change
• SaaS · B2B GEO: Getting AI to Recall You When Buyers Ask