Log AI Evaluation Versions Without Overclaiming Reproduction
Record prompts, inputs, settings, retrieval and rubrics so evaluations are traceable while unexposed versions and output variation remain explicit.
10/24/20263 min read
When an AI evaluation changes, the team needs to know what changed with it. A prompt name and a model label are rarely enough to reconstruct the conditions. Input preparation, retrieved sources, settings and scoring rules can all affect the result.
An evaluation provenance log records those conditions without promising that rerunning them will reproduce identical output. The useful goal is explainability of the experiment: what was supplied, what configuration was observable and how the result was judged. Do not fill unobservable provider details with guesses.
Separate identities from descriptions
Give the evaluation run, prompt, test pack and rubric stable identifiers. Store the actual approved versions or controlled references, not only a friendly name such as “best summary prompt.” A descriptive label can be reused accidentally while the content changes.
Record the model identifier exactly as exposed by the service, together with the date and any available version information. If the provider does not expose a fixed underlying snapshot, say so. A dated record of the observed label is more honest than claiming control over an implementation the team cannot inspect.
A filled provenance record
For fictional evaluation RUN-27, the task is extracting unresolved maintenance requirements. The proposed record contains:
Prompt: PR-8 revision 3; approved text reference; changed instruction requires explicit unresolved status.
Inputs: PACK-12 revision 2; cases C01–C18; synthetic material; preparation rule PREP-4.
Model: service-reported identifier copied at run time; underlying snapshot availability recorded as known or not exposed.
Configuration: exposed generation settings; tool availability; retrieval enabled or disabled; no credentials or secret values.
Retrieval: permitted source identifiers and versions actually supplied; failed or omitted sources recorded separately.
Evaluation: RUBRIC-5 revision 1; reviewer IDs; adjudication reference; run date; raw output references under approved retention.
Environment: nonproduction evaluation; consequential tools unavailable; application build reference if relevant.
These fields are a usable proposed log, not a record of an evaluation that was performed. Replace every placeholder with observed information during an authorized run, or explicitly mark it unknown.
Preserve the comparison boundary
Suppose RUN-28 uses the same prompt but a revised document converter. The comparison should not be described as a pure model change. If the converter now retains footnotes that were previously missing, improved summaries may result from better input rather than different model behavior.
Similarly, a changed rubric can alter pass counts without changing any output. Keep original judgments and revised judgments distinguishable. When reviewers learn new information, record whether that information was available to the model and whether the evaluation question itself changed.
Use a change note that identifies one or several known differences. “Multiple components changed” is an acceptable conclusion when precise attribution is not possible. It is better than choosing the most marketable explanation for an improvement.
Do not overpromise reproducibility
Even with a careful record, repeated outputs may differ. Hosted services, retrieval contents and unexposed implementation details can limit exact reconstruction. Describe what the log supports: repeating a documented procedure under observed settings, comparing outputs and investigating discrepancies.
NIST recommends documenting model details, assumptions and limitations. The proposed log applies that principle to a small evaluation process. It is not a claim that a particular service exposes all relevant internals or offers a deterministic mode.
Avoid retaining full contact payloads simply because more logging seems safer. Use permitted source references, minimal excerpts and controlled access. Exclude passwords, session tokens and authentication headers. A reproducibility project must not create a new uncontrolled archive of sensitive material.
Test whether the log is useful
Ask an authorized colleague to identify the exact prompt, input pack, rubric and known configuration differences for RUN-27 and RUN-28. If they cannot, improve the references. If they can but cannot obtain identical output, record that limitation rather than declaring the log defective. The deliverable is a traceable evaluation history with honest uncertainty, not an assurance that every generated sentence can be recreated forever.
If you are also choosing research access, ask about Leads Automation’s independent Sales Navigator offer at US$15/month on WhatsApp. Confirm the available plan, eligibility and terms before ordering; Leads Automation is not affiliated with LinkedIn. This is not LinkedIn’s official price or an assurance that any particular integration is included.