Verify Performance Claims Against Their Workload

“Ninety percent accurate” is not a complete performance claim. Accurate at what, on which inputs, under which checking rule, and with how much human intervention? Until those questions are answered, the number cannot tell a buying team whether the result is relevant to its workload.

Do not begin by arguing with the percentage. Reconstruct the calculation and the work behind it. A claim can be mathematically correct yet answer the wrong business question. Your review should preserve that distinction instead of labeling every narrow result deceptive or accepting it as a general promise.

Recover the missing context

Request the exact claim wording and the evidence supplied for it. Identify the task, the test-set construction, the denominator, the acceptance rule, the evaluation date and any excluded cases. Ask whether results were checked independently and whether corrections occurred before measurement.

For speed claims, identify the clock. Generation time, operator time and elapsed time to an approved deliverable are different measurements. A fast first draft may still require substantial review. For coverage claims, distinguish attempted records from completed records and completed records from records that met the intended quality standard.

NIST AI RMF 1.0, MEASURE 2.3 emphasizes evaluating performance under conditions similar to deployment. The claim-context check below is an original illustration of why that context matters.

Work through the fictional claim

An invented vendor advertises “90% accuracy on account extraction.” Its fictional evidence pack contains twenty synthetic account notes, each with five required fields, producing one hundred checked fields. Ninety fields match the agreed answers. The arithmetic supports ninety correct fields out of one hundred checked fields.

However, the buyer needs complete records suitable for review without reconstructing missing facts. The ten incorrect fields occur in ten different records. Only ten of the twenty records have all five fields correct. Thus the same test supports 90% field accuracy and 50% completely correct records under the defined rules. Neither figure can replace the other.

The completed context checklist reads:

• Claim unit: Individual extracted fields, not complete account records. Evidence: fictional scoring sheet SC-20, dated 18 November 2026.

• Denominator: One hundred required fields across twenty attempted records. No failed attempts were omitted from this invented example.

• Acceptance rule: Exact agreement for identifiers and dates; a documented equivalence rule for names. Unknown values were expected where the source lacked an answer. Guessed values counted as incorrect.

• Workload: Short English plain-text notes in one layout. No scanned pages, contradictory sources, multilingual notes or parent-subsidiary ambiguity.

• Human intervention: The stated score is before corrections. A reviewer subsequently repaired the ten affected records; that work is not evidence that the original records were correct.

• Verification limits: The evaluator has the supplied scoring sheet but has not independently rerun the test or checked a representative sample from its own workload.

• Buying implication: The claim supports a narrow description of the fictional test. It does not establish suitability for unattended record creation or predict performance on the buyer's different inputs.

Check what the headline encourages you to infer

A fair summary would retain the field-level denominator and the simple workload. A misleading summary would say “nine in ten accounts are ready to use.” That inference is directly contradicted by the fictional record-level distribution.

The reverse distortion is possible too. If ten incorrect fields were concentrated in two records, eighteen records would be fully correct. The field score would remain the same while the operational consequence changed. This is why an aggregate percentage alone cannot describe the pattern of failure.

Ask which errors matter most. An incorrect punctuation mark and a wrong company identifier may each count as one field error, but they do not necessarily carry the same consequence. Keep the raw measure and the business-impact review separate rather than quietly changing the scoring rule after seeing results.

Decide whether more evidence is worth collecting

A narrow benchmark can be enough to justify a synthetic demonstration, especially if its limitations are transparent. It is not automatically enough for a purchase or live deployment. Define the missing evidence that would change the next decision and ask for an appropriately scoped evaluation.

If the vendor cannot explain the denominator or provide a clear acceptance rule, mark the claim unverifiable for your decision. You need not prove the headline false to decline to rely on it. A useful review ends with a statement of what the evidence supports, what it leaves unknown and whether that gap matters to the proposed workload.

Continue with Prepare a Sales Demo Around One Confirmed Workflow or Build a Small Reference Library for Evidence-Based Selling.