Test AI Research for Input-Order Sensitivity
Compare reversed evidence packets using proposition-level invariants, not identical prose, to catch unstable account conclusions.
10/24/20263 min read
An account conclusion should not flip simply because the same evidence appears in a different order. Yet a summary may emphasize the first passage, the last passage or whichever note sits nearest the question. An order-sensitivity test looks for those changes while keeping the evidence itself constant.
This is a controlled comparison, not proof that any particular model has a positional bias. The test asks whether the workflow preserves the same decision-relevant facts across equivalent packet arrangements. Natural wording can vary; evidence attribution and supported conclusions should remain stable.
Choose a task with an explicit answer boundary
The fictional account Rivermill Controls is evaluating a reporting connector. Its synthetic packet contains P1: the connector exports daily totals; P2: hourly export support is not documented; P3: a buyer asked whether hourly data is available; and P4: a general industry note says hourly reporting can be useful. The task is to summarize established capability and the unresolved buyer question.
The required result is daily totals supported, hourly support unverified and buyer interest expressed as a question. The industry note does not establish the connector's capability or the buyer's purchase decision. Write this answer boundary before changing packet order.
Create paired input manifests
Use sequence A as P1, P2, P3, P4 and sequence B as P4, P3, P2, P1. Keep source text, identifiers, prompt, output format and configured model settings unchanged. Store a manifest showing the exact passage order and input version for each run.
Add a third sequence that interleaves the account and industry material without changing either. If the workflow uses retrieval, distinguish a pure ordering test from a retrieval test. Different retrieved passages would change the evidence set and make the comparison harder to interpret.
The proposed worksheet has these columns: pair ID, sequence, daily support preserved, hourly unknown preserved, buyer question preserved, industry attribution preserved, unsupported addition, reviewer decision and observed wording difference. This gives the reviewer specific invariants instead of asking whether two paragraphs are “similar.”
A hypothetical comparison
Imagine output A says, “Daily totals are available; hourly support is not established, and the buyer has asked about it.” Output B says, “Rivermill is pursuing hourly reporting through the connector.” These are invented outputs used to illustrate scoring, not measured model behavior.
A passes the stated facts. B fails because it promotes a question into a project intention and leaves the capability gap unclear. The fact that both mention hourly reporting does not make them equivalent. Record which proposition changed, then inspect whether the input pipeline actually supplied identical passages.
Another pair may differ only in sentence order: one begins with the buyer's question and another with daily export support. If both preserve the same meaning and evidence, that is acceptable variation. Do not demand identical prose unless the application genuinely requires a fixed rendering.
Repeat without hiding variability
If repeated outputs vary, retain the individual results rather than choosing the best example. State how many runs and sequence variants were reviewed. A small pack cannot establish a stable failure rate, but it can expose a defect worth investigating. Keep human reviewer disagreements visible too.
Try a variant with one clearly irrelevant passage removed. If the unsupported conclusion disappears, that helps locate the contamination route, though it does not prove a complete causal explanation. Avoid changing the prompt at the same time; otherwise, the source of improvement becomes unclear.
NIST AI RMF 1.0 frames measurement within intended use. This proposed comparison tests one input-stability property for account summaries. It should be combined with evidence and authorization checks before any downstream use.
Decide what to change
A consequential order-dependent conclusion should block automatic use of that summary field. Possible repairs include clearer source labeling, extraction before synthesis or a narrower output contract. Retest with fresh cases as well as the original pair. Do not assume that sorting every document by date solves the problem, especially when date order does not establish authority. The intended result is stable meaning under equivalent evidence, not merely a repeatable-looking paragraph.
If you are also choosing research access, ask about Leads Automation’s independent Sales Navigator offer at US$15/month on WhatsApp. Confirm the available plan, eligibility and terms before ordering; Leads Automation is not affiliated with LinkedIn. This is not LinkedIn’s official price or an assurance that any particular integration is included.