Write Pilot Acceptance Criteria Before Testing

A pilot produces faster drafts but more unsupported claims. One stakeholder calls it a success because drafting time fell; another calls it a failure because review became harder. The disagreement began before the pilot: nobody agreed what an acceptable result would mean.

Write the criterion sheet before running the work. It should define the behavior, measurement method, decision owner, and unresolved thresholds. A threshold left open is a decision still to make, not permission to choose the most flattering value afterward.

Separate safeguards from preferences

Some requirements should block the workflow regardless of convenience. An unauthorized external action or invented buyer commitment cannot be offset by attractive formatting. Other requirements concern tolerable effort, presentation, or usability and need a buyer-defined tradeoff.

For an AI evaluation, NIST's AI RMF 1.0 connects measurement to deployment context. The criterion sheet here is an original operating aid; it is not a universal pass mark or an official compliance checklist.

Name the unit being measured. A generated draft, a reviewed brief, and a seller-ready handoff are different outputs. If the pilot promises a ready brief, measurement must include the checking and correction needed to reach that state.

Complete the sheet before the first run

The following example is fictional. Cedar Review is an invented team evaluating an internal brief assistant. The pilot uses eight synthetic evidence packets, one reviewer role, a fixed configuration, and no external sending or production writes.

• AC-1, evidence integrity: every factual statement in the brief must be supported by the supplied packet or explicitly marked unresolved. Measurement: reviewer checks each statement against its source ID. Owner: research lead. Gate: any unsupported consequential claim blocks that case and prevents rollout pending correction.

• AC-2, permission boundary: the assistant may produce drafts only. Measurement: compare execution records and authorized test destinations for each case. Owner: trial administrator. Gate: any unexplained state-changing action stops the pilot.

• AC-3, completeness: each brief must contain the requested decision, relevant evidence, material unknowns, and next internal question. Measurement: four-field checklist. Owner: account-workflow lead. Gate: missing material information requires revision before acceptance.

• AC-4, reviewer effort: active preparation, review, and correction minutes per accepted brief. Measurement: task ledger using the same start and finish boundaries for all cases. Owner: operations manager. Threshold: unresolved; baseline exercise required before any efficiency claim.

• AC-5, readability: reviewer can locate the unresolved decision without opening another document. Measurement: observed task and explanation. Owner: intended user representative. Threshold: agreed qualitative judgment with recorded reasons, not an invented numerical score.

The sheet's version is CR-P1, approved for the bounded functional evaluation only. Its decision note explicitly excludes a claim of cost or time improvement until AC-4 has an agreed comparison. That makes the unresolved threshold visible before anyone sees an attractive result.

Work through a mixed outcome

In the illustrative result, seven briefs preserve the supplied evidence. One converts a proposed meeting into an agreed commitment. All eight remain within the draft-only boundary. Reviewers find the layout readable, but effort varies substantially across cases.

The decision is revise and retest the commitment failure. The team cannot declare the pilot successful because most cases looked good, and it cannot claim efficiency while AC-4 remains unresolved. It can record the narrower positive observation that no state-changing action was observed within the tested boundary.

Keep each result tied to its criterion. A single overall score would hide the reason deployment is blocked and make it difficult to assign the corrective work.

Change a criterion only for a stated reason

A legitimate scope change may reveal that an original criterion tests the wrong workflow. For example, users may need a conflict report rather than a finished brief. Record a new scope, new expected output, and new criterion version. Preserve the earlier result instead of retroactively judging it against the easier task.

Do not treat an unresolved threshold as a failure of every useful pilot. Exploration can establish a baseline or expose unknown requirements. Label the work exploratory and avoid a deployment decision that depends on an unagreed standard.

The pilot scope and cost guide addresses the broader engagement. This sheet governs the verdict: what passes, who decides, what remains unresolved, and which evidence is still required before the next commitment.