Test AI Selection of Stale and Current Documents
Use an explicit supersession fixture to separate issue dates, effective dates, download timestamps and transition exceptions.
10/24/20263 min read
A search result can surface an old document above its replacement. A familiar filename can hide a new version. An AI assistant needs to distinguish which source governs the current question without assuming that the latest download or file modification time is the latest business instruction.
A version-selection challenge differs from a general date-conflict test. Here, the packet deliberately supplies an explicit replacement relationship. The evaluation asks whether the assistant follows that relationship, preserves the prior version as history and discloses the basis for selecting the current evidence.
Define current for the task
In an August evaluation, the fictional account Stonebridge Instruments asks which supplied sample-submission process applied to a new submission on 15 July. Source V1, issued in February, says samples go to the central receiving desk. Source V2, issued in June, states that it replaces V1 for new submissions from 1 July and routes them to the regional intake team.
A third file, C1, was downloaded in August but contains V1's February text. The test runs in August and asks that retrospective 15 July question. The correct source for a new submission is V2 because of its explicit effective date and replacement statement, not because it happens to be the newest file encountered.
Build the version ledger
V1: issue February; effective before July replacement; process central receiving desk; superseded for new submissions from 1 July.
V2: issue June; effective 1 July; explicitly replaces V1 for new submissions; process regional intake team.
C1: August download of V1 content; acquisition date does not change the document's governing version.
Question scope: new submission on 15 July; expected source V2; disclose effective date and replacement basis.
The expected answer is “For a new submission on 15 July, the supplied V2 process applies from 1 July and directs samples to regional intake. V1 remains historical for this question.” Do not present this fictional process as operational guidance for a real organization.
Test the effective-date boundary
Change the question date to 20 June while leaving the documents unchanged. V2 has been issued but is not yet effective for the stated scope. The answer should explain that distinction rather than automatically choosing the newest issue date. The correct handling of any transition exceptions must come from the packet, not the model's assumptions.
Next change the question to a submission already in progress before 1 July. If V2 only describes new submissions and supplies no transition rule, the expected answer is unresolved for that in-progress case. An explicit replacement relationship can still have a limited scope.
Finally, remove the sentence saying V2 replaces V1. The model should no longer treat the relationship as established merely because the headings and dates look compatible. It can identify the likely version question and ask for confirmation, but the answer key must not reward an invented supersession rule.
Inspect source selection and final language
Record selected source ID, issue date, effective date, applicability scope, superseded source IDs and any unresolved transition issue. Check the final summary too. A correct internal selection followed by a sentence that omits “new submissions” can still mislead the reader.
An illustrative failure uses C1 because its file timestamp is August. Mark that as acquisition-date confusion. Another failure uses V2 for every historical submission. Mark that as scope expansion. Different labels help the team repair document preparation, retrieval or summarization rather than treating every mistake as the same model problem.
NIST recommends verifying generated source references. Here the reviewer verifies the source version and applicability, not merely the URL. Keep only permitted copies and metadata under the organization's retention rules; a version ledger is not a reason to retain every obsolete document indefinitely.
Release a dated conclusion
The approved output should say which version supports the answer and for what date and scope. If the governing version cannot be established, hold the dependent action and identify the missing authority question. Passing this synthetic challenge supports one version-handling behavior; it does not guarantee that a live retrieval system will always find the latest applicable policy.
If you are also choosing research access, ask about Leads Automation’s independent Sales Navigator offer at US$15/month on WhatsApp. Confirm the available plan, eligibility and terms before ordering; Leads Automation is not affiliated with LinkedIn. This is not LinkedIn’s official price or an assurance that any particular integration is included.