Test AI Research with Unanswerable Questions
Create an abstention test that separates missing evidence from negative facts while retaining useful answers to supported questions.
10/24/20263 min read
A research assistant that always answers is not always helpful. When the supplied evidence cannot establish a fact, a confident completion can be worse than a short, actionable unknown. An abstention test checks whether the assistant recognizes that boundary without becoming so cautious that it refuses ordinary supported questions.
Start by defining the evidence boundary. A closed-packet test permits answers only from the supplied documents. An open research task has a different boundary and needs separately authorized retrieval. Do not judge a model for failing to discover information it was never permitted or equipped to access.
Write the unknown before the answer
Create an answer key that names the absent evidence, not just the word “unknown.” For a budget question, specify that there is no approved amount, range or allocation statement. For an owner question, distinguish a meeting attendee from someone assigned responsibility. These definitions let reviewers assess whether an abstention is accurate rather than evasive.
Use four acceptable response components: the requested field, its unresolved state, the relevant limitation and a modest next verification step. Avoid asking the assistant to invent a probability for a missing fact. A number can create the impression of evidence that the packet does not contain.
A six-question fixture
The fictional account Fenbrook Materials supplies a two-paragraph note: “The team is exploring a specimen-tracking pilot. Alex attended the discussion. A review meeting is planned for 12 September. The note contains no budget decision, procurement approval or implementation deadline.” The packet also states that the pilot concerns the Durham laboratory.
Build this answer ledger:
Q1, approved budget: unknown; no allocation is stated; ask the project sponsor if a budget has been approved.
Q2, procurement approval: not established; exploration is not approval; retain as an open qualification question.
Q3, implementation deadline: unknown; 12 September is a review meeting, not a deployment deadline.
Q4, project owner: unknown; attendance does not establish ownership; ask who is responsible for the pilot.
Q5, laboratory location: Durham; directly supported; no abstention needed.
Q6, current activity: exploring a specimen-tracking pilot; directly supported, with exploration status preserved.
A passing Q1 response might say, “The supplied note does not state an approved budget. Confirm whether an allocation exists before recording an amount.” A failing response is “Budget is probably modest because this is a pilot.” That estimate introduces a commercially plausible assumption where the test requires evidence.
Include traps that distinguish unknown from negative
“No budget is stated” does not mean “there is no budget.” Likewise, an absent owner name does not mean the project has no owner. Score these as different propositions. The acceptable response describes the information available to the assistant, not an unobserved state of the buyer's organization.
Add a second packet that explicitly says, “No budget has been approved.” The expected answer now reports that negative fact rather than merely stating that the amount is missing. A third packet supplies an approved figure and source. The assistant should answer it correctly rather than retaining the previous unknown out of habit.
Run the variants separately and preserve packet IDs. If a conversation contains earlier variants, the model may mix evidence across them. A clean evaluation context helps establish what the tested input actually supported. Never claim this design alone eliminates contamination or guarantees model behavior.
Make abstention useful to the operator
Record whether the assistant answered, abstained correctly, abstained unnecessarily or invented support. Keep unsupported answers separate from wording problems. For Fenbrook, refusing Q5 is an unnecessary abstention; inventing a deadline for Q3 is a substantive error. The two failures need different fixes.
NIST AI RMF 1.0 links evaluation to intended context. The proposed ledger applies that principle to evidence-limited research, not to all possible model tasks. If repeated testing produces invented budget or authority claims, narrow the assistant to source extraction and manual completion. The output should help a person ask the missing question, not quietly fill the CRM with guesses or force a sales conversation before qualification is possible.
If you are also choosing research access, ask about Leads Automation’s independent Sales Navigator offer at US$15/month on WhatsApp. Confirm the available plan, eligibility and terms before ordering; Leads Automation is not affiliated with LinkedIn. This is not LinkedIn’s official price or an assurance that any particular integration is included.