Keep Quality Samples Separate from Staff Rankings

A quality reviewer inspects the hardest identity cases from one researcher and a random handful of ordinary records from another. The first researcher has more defects in the sample. A leaderboard built from those counts would compare different selection methods, not establish who performs better.

Sampling serves a question. A batch-release check, a targeted investigation, and an individual development review need different evidence and safeguards. Do not repurpose one into another without examining whether the sample can support the new conclusion.

Put the purpose on the review sheet

Before selecting records, state the population, selection method, intended decision, and prohibited interpretations. A diagnostic sample may deliberately concentrate on a suspected transformation error. That is useful for finding the mechanism, but its defect fraction is not automatically a fair estimate for every researcher or the whole database.

The NIST Engineering Statistics Handbook: What is Acceptance Sampling? distinguishes deciding whether a lot is acceptable from estimating its quality. Its manufacturing context is different from account research, but the central lesson is relevant: a sampling plan must match the decision it is intended to support.

If formal statistical assurance is required, use a qualified design rather than attaching confidence language to an arbitrary handful of records. This article's examples are operational illustrations, not statistical acceptance plans.

A fictional sample with an unfair comparison

An invented team reviews twenty records. Ten come from researcher A because they passed through a new entity-matching transformation. Ten come from researcher B's routine location checks. The first group contains four identity defects; the second contains one missing source date.

The proper finding is that the inspected transformed group needs investigation and the missing-date issue needs correction. The sample does not establish that A is four times worse than B. The task types, selection reasons, and possible sources of error differ.

Further review shows that three of A's defects were introduced after the researcher submitted correct records. One was present in the original work. A report that attributes all four to the person whose name appears on the current record would confuse authorship, transformation, and ownership.

Use a sampling-purpose record

For the fictional transformation review, the record says:

• Purpose: identify whether the new matching step creates wrong-entity relationships.

• Population: records processed by transformation version M4 during the stated period.

• Selection: deliberately chosen challenge cases plus a separately labelled random diagnostic subset.

• Unit: processed account record, with source and output versions retained.

• Decision: pause or repair the affected transformation and check the scoped records.

• Not supported: individual productivity ranking or a universal estimate of researcher accuracy.

Keep findings from the challenge subset separate from the random subset. Combining them into one percentage would obscure why the difficult records were selected.

Provide useful feedback without misusing the sample

A sampled defect can still justify correcting that record and discussing the observed decision. The reviewer can say the source qualifier was lost here and show the evidence. That is narrower than concluding that the researcher generally ignores qualifiers.

If a development concern remains, agree a separate review with representative task context, clear criteria, comparable evidence, and an opportunity to explain or correct the record. Account for specification quality, tool behavior, assigned difficulty, and support work. Do not infer private traits or motivations from an error count.

Make the purpose clear to the people whose work is reviewed. An improvement exercise loses trust when its notes later appear in an unexplained ranking that participants were told was not the goal.

Report the process decision first

The fictional team's release note identifies the affected transformation, inspected scope, defects, repair owner, and verification plan. It includes the single source-date correction without turning the report into a contest between A and B.

After repair, inspect new outputs and appropriate unchanged cases. Rechecking only the original failures can show that those rows were fixed while missing a new defect elsewhere.

Quality sampling is valuable when it reveals what should change in the work. Keeping its purpose explicit protects that value and prevents a narrow diagnostic result from becoming a broader judgment the evidence was never designed to support.

Continue with Plan a First Sales Hire's Research Training or Create a Useful Sales Research Service-Level Agreement.