Separate Product Bugs from Configuration Mistakes

A missing output can indicate a product defect, an incorrect setting, an unavailable feature or an expectation the documentation never promised. Calling every failure a bug makes support conversations noisy. Calling every failure user error can hide genuine defects. During evaluation, the better first move is to construct a reproducible account of the mismatch.

The evidence packet should let a second authorized person understand what was attempted without receiving real customer data or guessing the original setup. It is a diagnostic record, not an accusation and not a demand that support accept your preferred explanation.

Use four provisional categories

• Configuration mismatch: The selected settings or mapping do not match the intended workflow.

• Unsupported expectation: The operation, plan, input type or environment is outside the documented scope.

• Possible product defect: A documented behavior appears to fail under the stated supported conditions.

• Unresolved: Evidence is insufficient or contradictory, so classification remains open.

These are working labels, not verdicts. A configuration mistake and a confusing product interface can coexist. A missing permission may explain an error while an inadequate error message still deserves a separate improvement request.

The fictional support packet

Imagine an evaluator testing an invented note-to-review-sheet feature. All details and outcomes here are illustrative. The test uses synthetic business names and a disposable destination; no software was actually run.

Case EVAL-32, opened 12 November 2026: The expected operation is to place an approved account note into the “Account name” and “Research summary” fields. The fictional vendor's mapping guide MG-4, revision 3, says destination columns are selected explicitly. The evaluator initially expected the feature to infer them from headings.

Environment: Evaluation workspace W-TEST, feature build displayed as 4.2, browser version recorded in the private packet, test operator with the documented write permission. Subscription eligibility was confirmed for this exact operation in the fictional evaluation agreement. No assumptions are made about another plan or a similarly named feature.

Input and steps: Synthetic record SYN-8 has account name “Pine Test Works,” summary “Ownership remains unconfirmed,” and approval state “Approved.” Open mapping settings, select the test destination, submit once, then inspect the destination and operation log. Expected output is one row with both values in the intended columns.

Initial observation: One row was created, but the account name appeared in the summary column and the summary column mapping was empty. Redacted configuration capture C-1 shows “Account name” mapped to “Research summary.” The recorded behavior matches that mistaken mapping.

Controlled correction: The authorized evaluator corrected only the two field mappings. A new synthetic record SYN-9 then appeared in the intended columns. Classification: configuration mismatch supported by the before-and-after mapping evidence. Separate usability note: the evaluator misunderstood the mapping screen; this does not establish a functional defect.

Unresolved companion case: SYN-10 used the corrected mapping but returned an error after submission. Destination outcome was not yet established. Classification: unresolved. Next action: inspect the authorized test destination and request support review of operation reference OP-10 before considering a retry. Do not assume the error means no row was written.

Change one relevant condition at a time

Changing the mapping, user role, browser and input simultaneously may make the next attempt succeed without explaining why. Narrow the test to the smallest representative input and vary the suspected cause deliberately. Record each change and its purpose. Preserve enough of the original conditions to compare the observations.

For generated output, identical prompts may not always yield identical wording. Specify the requirement that matters, such as retaining an unknown-status label, rather than requiring an exact sentence unless exactness is genuinely part of the contract. Record repeated observations honestly, including intermittent behavior.

NIST AI RMF 1.0, MEASURE 2.1 includes documenting test sets, metrics and evaluation tools. The packet here extends that documentation discipline to a practical support handoff; it is not an official NIST form.

Stop where diagnosis would exceed permission

Do not escalate your own privileges, disable security checks or use production records simply to make a test pass. If the investigation needs administrator action, give the authorized owner the narrow request and explain why it matters. Remove tokens, confidential payloads and unrelated records from support material.

Finally, preserve the provisional classification when reporting to the buying team. “Corrected a field-mapping mistake; a separate uncertain write remains under review” is more useful than either “the product is broken” or “everything works now.” The evaluation advances when the cause and remaining uncertainty become clearer, not when the error count looks better.

Continue with Prepare a Sales Demo Around One Confirmed Workflow or Build a Small Reference Library for Evidence-Based Selling.