Compare Researcher Judgments Without Rewarding List Size

Two researchers follow the same account brief. One returns twelve businesses and the other returns twenty. Those totals reveal almost nothing about whose interpretation is more useful. One person may accept distributors, the other only manufacturers. One may regard a regional sales office as an operating location. Resolve those differences before rewarding speed or list size.

A blind inclusion review lets the team examine judgments without seeing who supplied them. Its purpose is to improve the brief and expose uncertain evidence, not to declare a winner from a small exercise.

Freeze the decision the reviewers are making

Write a single inclusion question. For example: does this account operate a food-production site in the specified service region and appear relevant to equipment-cleaning services? State the required evidence and explicit exclusions. Do not ask reviewers to infer budget, an active purchase, or an individual's authority.

Separate company facts from search settings. LinkedIn's filter definitions distinguish a person's geography from company headquarters. If the brief actually concerns a production site, neither field alone settles the inclusion question. This is a research gap, not necessarily a researcher's mistake.

Choose an approved set of company-level evidence packets. Remove researcher names, original recommendations, and persuasive comments, while retaining the facts and source dates needed for a decision. Give each packet a neutral code. Each reviewer records include, exclude, or unresolved, plus the decisive reason before discussing it.

Work through the disagreements

Here is a fully illustrative exercise using fictional companies. Both reviewers receive the same six packets. These outcomes are invented to demonstrate the method, not a performance benchmark.

Packet and evidence: A1: Mallow Foods lists an operating bakery in the target region.

• Reviewer A: Include

• Reviewer B: Include

• Adjudicated result: Include. Relevant operation evidenced.

Packet and evidence: A2: Birch Ingredients sells imported ingredients; no production is described.

• Reviewer A: Include

• Reviewer B: Exclude

• Adjudicated result: Exclude under the agreed production-site criterion.

Packet and evidence: A3: Linden Kitchens announces a plant opening next year.

• Reviewer A: Include

• Reviewer B: Unresolved

• Adjudicated result: Unresolved. Future-site work needs a separate brief.

Packet and evidence: A4: Hazel Snacks has local headquarters but an identified plant outside the region.

• Reviewer A: Include

• Reviewer B: Exclude

• Adjudicated result: Exclude from this regional service exercise.

Packet and evidence: A5: Oak Bakery has a local plant, but its operating status is unclear.

• Reviewer A: Unresolved

• Reviewer B: Unresolved

• Adjudicated result: Unresolved pending a current operating source.

Packet and evidence: A6: Reed Design supplies packaging artwork, not food production.

• Reviewer A: Exclude

• Reviewer B: Exclude

• Adjudicated result: Exclude. Business-model mismatch.

The reviewers disagree on three of six packets. That observation describes this exercise only. It is not an estimate of error across their normal work, and agreement on A5 does not mean the underlying fact is known. Both reviewers correctly preserved uncertainty.

Assign causes before assigning corrections

A2 reveals a definition problem: the word food was doing more work than the requirement to operate a production site. A3 reveals a timing boundary. A4 reveals confusion between headquarters and the place where service would be delivered. Writing three precise corrections is more useful than instructing one reviewer to be stricter.

For each disagreement, ask whether the cause is an unclear rule, different evidence, a stale source, or an unsupported interpretation. A reviewer should be able to change their decision without being treated as unreliable. The goal is a reason someone else can inspect.

Keep the adjudicator's reasoning beside the packet. If the team cannot agree because the commercial owner has not decided whether planned sites belong, escalate that policy question. Do not resolve it by majority vote among researchers who lack the authority to redefine the market.

Retest the interpretation on unfamiliar accounts

Update the brief with a compact contrast: operating local factory included; distributor excluded; future factory unresolved until a separate project scope exists. Then review fresh examples. Repeating only the same six packets tests memory as much as understanding.

Keep difficulty visible. A researcher assigned sparse company websites should not be compared directly with someone reviewing well-documented manufacturers. Review time, unresolved cases, and the adequacy of the brief help explain the work. Raw inclusion totals do not.

If a batch needs wider quality evaluation, use a separate sampling review with an appropriate design. This calibration exercise answers a narrower question: are people applying the same decision rule? Finish when the remaining disagreements have explicit owners and the revised rule works on fresh cases. Only then compare throughput under similar research conditions.