Document Pilot Limitations Before Sharing Results

“The pilot worked” is not a result boundary. It is an invitation for the next reader to imagine a broader test than the team actually ran. An internal slide can quietly transform a small, supervised exercise into a belief that the workflow is ready for every account, language and operating condition.

Put a result boundary beside the headline before circulating it. The boundary should travel with copied slides and forwarded summaries, not remain buried in the test folder. It explains the population observed, the conditions excluded and the decision the evidence can reasonably support.

Write the boundary before writing the recommendation

Separate the unit of testing from the unit of intended deployment. Twelve notes are not twelve customers. Two reviewers are not two independent organizations. Five working days do not show behavior across a quarter of changing inputs. These distinctions determine what a result means, even when every reported number is arithmetically correct.

Also distinguish assistance from autonomy. A reviewer who repaired each draft may have demonstrated a workable supervised process. That does not establish that the unreviewed output is suitable for direct use. List the intervention rather than treating it as invisible background work.

NIST AI RMF 1.0, MEASURE 2.5 calls for documenting limits on generalization beyond the evaluated conditions. The statement below is an illustrative way to make those limits visible.

A finished statement for an internal pilot report

Fictional evaluation: From 2 to 6 November 2026, two operations reviewers assessed draft summaries produced from twelve synthetic English account notes. The example and every result below are invented for explanation; no product was tested.

Observed result: Ten of twelve drafts preserved the required source references and uncertainty labels on the first review. Two failed: one omitted an unresolved ownership question; one turned a proposed meeting into an agreed meeting. Both were corrected manually before being accepted into the test review sheet. All twelve final summaries therefore passed only after the stated human review and corrections.

Workload boundary: Notes contained between four and eight factual statements, used one consistent input format and contained no personal contact details. The test set included two deliberately contradictory notes and one missing-source note. It did not include scanned documents, multilingual material, long meeting transcripts or notes requiring specialist interpretation.

Environment boundary: Reviewers copied approved outputs into a disposable sheet. No live CRM connection, scheduled automation, customer communication or permission-revocation behavior was evaluated. The same fictional feature configuration was used throughout; behavior after model or configuration changes remains unknown.

Supported decision: Continue designing a supervised note-review workflow and investigate the two error types. The exercise does not support unattended CRM updates, a production accuracy claim, or a financial return estimate. The next evaluation should deliberately challenge uncertainty preservation before expanding the input range.

Accountable owner: Operations lead Ivo owns the boundary statement and the test record. Any summary quoting the result must retain the distinction between first-review drafts and corrected final summaries.

Make the denominator survive editing

Suppose a presentation editor wants the headline “Twelve successful summaries.” That wording loses the two first-review failures and the manual work required. A more informative headline is “Ten of twelve drafts met the selected checks initially; two required correction.” It is longer because it carries the actual result.

Do not substitute a percentage unless it helps the decision and the small sample remains conspicuous. A precise-looking rate can encourage an unsupported forecast about a much larger workload. Keep the count, the relevant checks and the sample construction available together.

Nor should excluded cases disappear because they were inconvenient. If three planned cases could not be run, report them separately from failures and completed tests. “Not evaluated” is neither a pass nor a demonstrated defect.

When a narrow pilot is enough

A limited test can still settle a limited question. If the team only needed to learn whether reviewers could identify source references in the proposed layout, a small synthetic exercise may be useful. The mistake is not smallness; it is changing the decision after seeing encouraging results without changing the evidence.

Conversely, if the next decision involves live customer records or unsupervised effects, the example's evidence is insufficient. Keep that progression paused until the relevant approval and evaluation conditions are addressed.

Before sharing, ask a colleague who did not attend the pilot to explain what was demonstrated and what remains unknown. If their interpretation is broader than yours, revise the boundary. A result is ready to circulate when its limitations remain understandable without the original presenter in the room.

Continue with Prepare a Sales Demo Around One Confirmed Workflow or Build a Small Reference Library for Evidence-Based Selling.