Evaluate AI Summary Completeness Without Length Bias
Build a must-preserve fact card that catches omissions, distortions and invented next steps in both short and long summaries.
10/24/20263 min read
Long summaries can appear thorough because they contain many sentences. Short summaries can appear efficient because they are easy to scan. Neither property establishes completeness. Evaluate whether the output preserves the facts a reader needs for the intended decision, without rewarding repetition or unnecessary detail.
A must-preserve checklist makes that judgment concrete. It defines the essential propositions before the model writes, then checks both omissions and unsupported additions. The goal is not to force every source detail into the output. It is to preserve the decision boundary in the smallest useful form.
Define the decision before the checklist
For a fictional account, Copperline Diagnostics, the reader needs to decide whether to prepare a technical review agenda. The supplied note says that the buyer requested an agenda, the session is not booked, a sample-file check must happen first, the technical reviewer is unnamed, and the seller agreed to provide the agenda by Thursday. A lengthy history of the company is not relevant to that decision.
Write five required propositions and one prohibited inference. Keep them atomic enough that a reviewer can tell whether each survived. Do not put “everything important” into a single criterion, because reviewers will fill that phrase with their own assumptions.
The completeness card
F1: agenda requested by buyer; preserve the request rather than presenting it as the seller's unsolicited plan.
F2: session not booked; preserve the negative status.
F3: sample-file check precedes the session decision; preserve the dependency.
F4: technical reviewer unknown; preserve the missing owner.
F5: seller's agenda deadline Thursday; preserve actor, action and time.
X1: no approval to book the session or treat the technical review as completed.
A concise passing summary could say: “The buyer requested an agenda, which the seller agreed to provide by Thursday. No session is booked; the sample-file check comes first, and the technical reviewer is still unnamed.” This invented example preserves the five required propositions without adding a commitment.
An apparently efficient alternative, “Send Thursday's agenda and book the technical review,” fails F2, F3 and F4 and introduces X1. A long alternative with several paragraphs of company background can fail the same checks. Word count is a constraint to manage, not the measure of substantive coverage.
Score coverage without averaging away harm
Count preserved required propositions for a descriptive coverage measure, but keep critical omissions and prohibited additions as separate gates. If four of five facts survive and the missing fact is “do not schedule yet,” a favorable percentage does not make the summary usable.
Record each fact as preserved, distorted, omitted or unclear. Distorted means the concept appears with changed meaning, such as turning an unknown reviewer into the engineering director. Unclear means the wording could reasonably be read in incompatible ways. Those categories reveal different repair needs.
For the fictional card, mark F2 and F3 critical because they govern whether scheduling is appropriate. Mark F1, F4 and F5 required because they affect the handoff. This weighting is a proposed task-specific choice, not a universal hierarchy of sales facts.
Test length constraints deliberately
Ask for the same content at two reasonable length limits and inspect what disappears. If the shorter limit makes the required meaning impossible to express clearly, change the format or limit rather than accepting an omission. A compact bullet list may preserve dependencies better than a slogan-like sentence.
Add irrelevant background to one input variant. Required coverage should stay stable; the output should not become longer merely because more text was supplied. Remove F5 from another variant. The summary must then stop inventing Thursday as a deadline, even if that date appeared in familiar examples.
NIST recommends checking generated claims against their sources. Apply that check to missing as well as added meaning. The final artifact is a completeness card, reviewed outputs and a clear decision about the usable format. Release a summary only when its necessary facts survive, not when it wins a contest for length or polished prose.
If you are also choosing research access, ask about Leads Automation’s independent Sales Navigator offer at US$15/month on WhatsApp. Confirm the available plan, eligibility and terms before ordering; Leads Automation is not affiliated with LinkedIn. This is not LinkedIn’s official price or an assurance that any particular integration is included.