Decide what your dataset can represent

A teammate adds twenty versions of “When do you open?” to the evaluation set. The score rises. The application has not changed, but the report now looks more reassuring. What changed was the balance of evidence. Before improving a system against a dataset, decide what one row means, how it was selected, and which claims the collection can support.

In this module you will inspect a versioned case register, distinguish development examples from held-out review, identify related examples that should stay together, and resolve a label disagreement without rewriting history. The runnable example uses only original synthetic messages and invented reviewer judgments.

Give each row a job

A sampling unit is the thing selected for evaluation. It might be one request, one conversation, or one complete task. These are different units. Splitting a five-turn conversation into five unrelated requests can lose the context that determines the correct behavior. Counting all five as independent users also overstates diversity.

Our unit is a single support message with its explicitly supplied context. The register in project/fixtures/operations.json, under dataset, contains eight units. Each has an ID, a split, a related-example group, a slice, the input, two synthetic reviewer votes, an adjudicated label, and a rationale. Its version is juniper-review-v2; its change note explains what was added. The original twelve-case dataset remains available unchanged for module 1.

A useful dataset card states the task, unit, collection method, permitted use, coverage, known exclusions, label process, version, and owner. For this register, collection method is “handwritten synthetic,” permitted use is local teaching, and exclusions include real traffic, multilingual coverage, multi-turn context, actual private information, and empirical traffic weights. Eight rows cannot fill those gaps by being carefully formatted.

Separate learning from checking

Development examples help you inspect failures and change the system. A held-out collection is reserved for a later check after those choices are made. If you repeatedly optimize against a held-out collection, it becomes development information in practice, whatever its filename says.

We assign four cases to development and four to holdout. Here every file is visible because the purpose is teaching. These are demonstrations of a split, not a genuinely secret test or an independent real benchmark after you have studied them. A production evaluation plan would also specify who can see labels and how often a final review set can be used.

The scikit-learn common pitfalls guide explains how using test information during model choices can produce optimistic estimates. We apply that principle to prompts and evaluation examples, without installing scikit-learn or fitting a model.

Keep related examples together

Suppose development contains “Approve my refund,” while holdout contains “Please approve my refund now.” Different IDs do not establish independence. Both may be paraphrases of the same source request. A group connects related examples so the split check can catch them.

The supplied register uses different groups across its two splits. Audit it from the project directory:

python3 -B opslab.py dataset

Expect four development records, four holdout records, no cross-split groups, no unresolved labels, and PASS. This means the supplied register meets this mechanical audit. It does not prove that its groups are correct: somebody still has to identify related examples honestly.

Now rehearse a leak in memory:

import copy
import opslab
register = copy.deepcopy(opslab.load()["dataset"])
register["records"][-1]["group"] = "refund"
print(opslab.dataset_audit(register))

Run that snippet with python3 -B - from project, or in a Python session there. The audit reports refund as a cross-split group and blocks. It does not delete the example or reassign it automatically. A reviewer should decide whether to regroup related cases, recollect examples, or redesign the split, then create a new version with a reason.

A label is a reviewed interpretation

Case d2 says “Can I move it?” Reviewer A chose clarify; reviewer B chose answer. The disagreement is useful: B assumed a booking context that was absent from the unit. Adjudication chose clarify, and the rationale preserves the interpretation. A majority vote would not explain the mistake, and two agreeing reviewers could still share the same mistaken assumption.

The audit requires two distinct reviewer IDs and an explicit adjudicated label. It flags unresolved labels and a consensus that has been changed without a new review record. Those are workflow checks. It does not authenticate the reviewers, determine semantic truth, or confirm that a domain expert reviewed anything. All votes in this package were invented for the exercise.

Real label repair should preserve the original version, record the correction and reason, and rerun relevant comparisons against the new version. Comparing a baseline scored on old labels with a candidate scored on new labels confounds the system change with the answer-key change.

Practice

1. Write a six-line dataset card for this register. Include one claim it supports and one important claim it cannot support.
2. Reproduce the group leak above. Explain why unique IDs did not prevent it and name a plausible additional relationship the mechanical check could miss.
3. In a copied register, set d2's adjudicated value to None. Predict and run the audit. Describe the review record you would require before resolving it.
4. Propose one multi-turn synthetic case. State why it needs a different unit definition rather than silently adding it to a single-message score.

Do not collect real messages as part of this exercise. Your exit artifact is a dataset card plus a proposed version change, not a larger number of unreviewed rows. Explained solutions.