Report Small-Sample Results Without False Confidence

Four successful cases out of six can teach a team something important. They cannot, by themselves, establish that a workflow will succeed two-thirds of the time across future customers. The arithmetic may be correct while the generalization is unsupported.

Report a small pilot as a defined collection of observations. Explain how the cases were chosen, which conditions were tested and what each result means. The most valuable finding may be one repeatable defect, not the percentage that makes the summary slide look strongest.

Start with selection, not precision

State whether the cases were randomly selected, chosen for convenience or deliberately assembled to test difficult conditions. A challenge set is useful for finding weaknesses but should not be presented as a representative estimate of ordinary work. Conversely, a set of unusually clean cases can hide problems common in deployment.

Statistics Canada's Statistics Canada: Non-probability sampling discusses the representativeness difficulties of non-random samples. For an operational pilot, the practical implication is to keep conclusions tied to the cases and conditions you actually examined.

Do not assume that adding decimal places creates reliability. “66.6667% success” describes the same four successes as “four of six.” In a six-case set, changing one classification moves the percentage by about 16.7 percentage points. Show the counts and the case-level evidence first.

A completed fictional pilot memo

This illustrative memo concerns an invented account-note review workflow. No product was tested, and the outcomes are fictional.

• Purpose: identify whether the workflow preserves source references and unresolved questions in a reviewer-ready note.

• Selection: six synthetic cases chosen by the workflow owner; two straightforward records, two with conflicting dates and two with ambiguous entity relationships.

• Conditions: one product configuration, one reviewer role and one language; no external messages or live customer records.

• Outcomes: four cases met the stated acceptance checks; two did not.

• Failure A: a conflicting date became a single confident date without the unresolved qualifier.

• Failure B: two related entities were summarized as one organization.

• Interpretation: the examples reveal two material failure patterns requiring correction. They do not estimate the prevalence of those failures in ordinary account work.

• Decision: do not release the affected workflow for the intended use; correct the behavior and test fresh cases alongside the original failures.

This memo supports a concrete decision without announcing a market benchmark. It also avoids the opposite mistake of saying a small sample teaches nothing. One demonstrated loss of an essential qualifier can be sufficient to block a particular use under the team's predeclared criteria.

Keep uncertainty types separate

Sampling uncertainty concerns how a sample relates to a larger population under an appropriate design. Measurement uncertainty concerns whether the outcomes were classified correctly. Scope uncertainty concerns whether the test conditions resemble the intended work. A confidence interval cannot automatically repair a biased case selection or an inconsistent pass definition.

NIST's NIST Engineering Statistics Handbook: Confidence intervals discusses interval methods for proportions and limitations of common approximations with small samples. If a formal estimate is necessary, use a suitable design and qualified statistical review. Do not attach a confidence claim to a convenience pilot simply because a calculator can produce one.

For the fictional memo, descriptive reporting is enough to decide that the two failures need attention. The team does not need to infer a population defect rate before correcting a reproducible issue.

Do not turn retesting into a flattering average

After a fix, keep the first run and retest distinguishable. If all six original cases now pass, that shows those specific cases behaved differently under the revised configuration. It does not prove that the workflow generalizes, especially when the fix was developed with those exact examples visible.

Add previously unseen cases that test the same underlying distinctions and record the new version. Explain any change in case difficulty, reviewer or acceptance criteria. Combining every run into one percentage can conceal both the original defect and the narrowness of the repair.

End the report with what is established, what remains untested and the next bounded decision. A useful small-sample result is neither a sales promise nor an apology for limited data. It is an honest account of evidence that helps the team choose what to inspect, change or stop next.

Continue with Measure Sales Navigator Prospecting with a Small, Honest Funnel or Compare Two Target Segments with a Fair Sales Experiment.