Measure Research Review Disagreement

Two reviewers can use the same lead-confidence rubric and reach different conclusions. Until you inspect those differences, a score that looks objective may be hiding separate assumptions about source age, entity scope, and what counts as direct evidence.

A double-review exercise makes that variation visible. It measures agreement on the selected records, not whether the underlying facts are certainly true. Both reviewers can agree on an unsupported interpretation.

Freeze what each person is reviewing

Choose one classification task, such as whether a company's operating responsibility is supported, unresolved, or contradicted by the supplied evidence. Do not combine commercial fit, contact permission, writing style, and evidence confidence into one agreement number.

Give both reviewers the same record version, source excerpts or permitted source references, intended use, and rubric version. Ask them to work independently before discussing results. If one reviewer has additional account context, record that difference rather than pretending the inputs were identical.

W3C's W3C: Data Quality Vocabulary allows quality information to include assessments and annotations from different perspectives. That supports keeping the reasoning visible. It does not establish a universal lead-scoring rubric or make a majority judgment correct.

Work through a fictional twelve-record exercise

In this invented example, reviewer A and reviewer B independently classify twelve company records. Their joint result is:

• Both supported: five records.

• Both unresolved: two records.

• Both contradicted: one record.

• A supported, B unresolved: two records.

• A unresolved, B contradicted: one record.

• A contradicted, B supported: one record.

Eight of twelve judgments match, so observed exact agreement is 66.7 percent. Four differ. Report the counts alongside the percentage because twelve selected cases are a small diagnostic exercise, not a representative estimate of every future review.

The last disagreement is especially important: supported versus contradicted may lead to opposite handling. It deserves closer examination than a difference in how much explanation each reviewer wrote. Do not flatten every disagreement into the same operational consequence.

Keep a disagreement-resolution log

Use a separate entry for each disputed record. Include the two original classifications, each reason, the relevant source passage, the cause of disagreement, and the agreed next action. Preserve the initial judgments so later calibration does not erase evidence that the rubric was unclear.

For fictional record Q08, A marks responsibility supported because a company page names a regional operations lead. B marks it unresolved because the project concerns a different region. The resolution is not to split the difference. The claim's scope is narrowed and the project responsibility remains unresolved.

For Q10, A reads an announcement as a completed transfer while B notices that the transaction is proposed. The source supports a plan, not the completed ownership change. The correction updates the record and adds a rubric example about proposed versus effective events.

Q12 remains disputed because two current sources conflict. The reviewers agree on an unresolved handling state, not on which source is factually right. Escalation is a valid resolution of the workflow even when factual uncertainty remains.

Separate missing context from inconsistent judgment

Classify disagreement causes before adding more training. Different source versions call for input control. An ambiguous term calls for a clearer definition. A missed qualifier calls for focused review practice. A genuine unresolved fact needs more evidence or a controlled hold.

Averaging numeric confidence labels would solve none of those problems. Likewise, replacing both original labels with the senior reviewer's answer would improve apparent agreement only by concealing disagreement.

If you need a formal reliability statistic, involve someone able to select and interpret an appropriate method for the label structure and sampling design. Do not attach a sophisticated coefficient to a convenience sample and describe the result as proof of objectivity.

Test the clarification on new records

After the exercise, publish the changed examples and give reviewers a fresh set of similar cases. Repeating the same twelve immediately may measure memory of the discussion rather than consistent application.

Compare where disagreements moved, not just whether the headline percentage rose. A process that now reliably flags scope conflicts is more useful even if difficult new cases still produce uncertainty. The goal is a shared, explainable decision standard that preserves unknowns, not agreement achieved by making every answer look certain.

Continue with Measure Sales Navigator Prospecting with a Small, Honest Funnel or Compare Two Target Segments with a Fair Sales Experiment.