Write an Eval That Can Say No

Most evals work like scoreboards: when the number goes up, the change ships. This tutorial builds a different kind: an eval that is a decision, written before the change it judges, that can refuse a model even when its headline score improves.

The result that makes the point: the candidate model scores 7 percentage points higher than the one it replaces on held-out data, and the eval rejects it anyway. By the end you will know why that is the right call, and how to build evals that can make it.

An average score can improve while a behaviour your users depend on gets worse. A support assistant answers more questions correctly and starts inventing a refund policy. A classifier gets more reviews right and starts misreading "slow start, but the last hour is terrific". Your users feel the second thing, not the average. An offline gate makes that trade visible: it combines an overall requirement with specific behaviours the new version must keep.

Who this is for

Developers who already have a working model or AI application and need to decide whether a proposed change is worth releasing. You need basic Python and a terminal. You do not need a GPU, or any experience training language models.

When to run one: before you change a model, a prompt, a retrieval setup or a tool workflow. Say what the decision is before you judge it. A quality upgrade might need a meaningful accuracy gain. A cost reduction might only need no material loss at a lower price. Either way, decide what improvement matters and which failures are unacceptable before you look at the candidate.

Time: the scripts run in under a minute on a laptop CPU, plus a one-off dataset download. Reading it, building the files and understanding the results takes 45 to 60 minutes.

Where this fits

This is where the gate sits in a release:

build the candidate → run your normal software checks → run the offline gate → read the failures → shadow run or canary → promote → monitor

Passing the gate means eligible for the next step, not permission to ship. It judges the behaviour of the whole thing that changed, not just the model weights: for an application, the candidate is the model plus its prompt, retrieval settings and tools.

The offline gate is the cheapest place to catch a bad change, and the other steps in that line are evals too; they are described after step 7.

What you are building

A gate for one decision: should this new model replace the one we run today?

• The task: sentiment on movie review sentences, using the SST-2 dataset. It is small, public and familiar, so the method stays in focus.
• Three models. An incumbent (what we run today), a candidate (the proposed change) and a deliberately broken model, a known-bad change the eval must reject.
• Two kinds of test data. Twenty hand-written cases for the behaviour we care about most, plus a fixed random sample of 300 held-out sentences.
• One rule, written first. The pass conditions go in a file before any model is trained.

With the pinned package versions and dataset commit, the results should reproduce exactly: the sampling and shuffles are seeded, and logistic regression's default solver is deterministic. We got identical numbers on macOS and on GitHub's Linux runners, under Python 3.11 and 3.12. Different package versions may shift a decimal.

When you are done, the folder looks like this:

evals-tutorial/
├── evals/
│   ├── gate.toml        # you write (step 1)
│   ├── cases.jsonl      # you write (step 2)
│   ├── heldout.jsonl    # freeze.py writes (step 3)
│   └── FROZEN.sha256    # freeze.py writes (step 3)
├── freeze.py            # step 3
├── train.py             # step 4, writes the three .joblib model files
├── run_eval.py          # step 5
├── test_gate.py         # step 7
└── bootstrap.py         # optional, "Is the gain real?"

The complete project, with a lockfile and a CI job that reruns it, is in the companion repository. It does not include evals/heldout.jsonl: freeze.py rebuilds it (see step 3).

Setup

The commands are for macOS and Linux. With uv:

mkdir evals-tutorial && cd evals-tutorial
uv init --python 3.12
uv add datasets==5.0.1 scikit-learn==1.9.1 joblib==1.6.0
mkdir evals

With pip instead (Python 3.11 or newer, because the scorer uses tomllib):

mkdir evals-tutorial && cd evals-tutorial
python3 -m venv .venv && source .venv/bin/activate
pip install datasets==5.0.1 scikit-learn==1.9.1 joblib==1.6.0
mkdir evals

If you use pip, drop the uv run prefix from every command below. On Windows the activate command differs, and an editor that saves your pasted files with CRLF line endings will change the frozen hash in step 3 (the held-out data check still works; see there).

These are the exact versions we measured with, on Python 3.12.13. Newer versions will probably work but may shift a decimal. The dataset is pinned to one commit on the Hub in the scripts, for the same reason.

Step 1: Write the decision before the models

The most important file in this tutorial is written before anything is trained. It says what "better" has to mean for the candidate to replace the incumbent. Save this as evals/gate.toml:

# evals/gate.toml
# Written BEFORE any model was trained or scored. Changing these after seeing
# results turns the eval into a description of what you got.
[gate]
# The candidate replaces the incumbent only if all of these hold:
min_heldout_gain = 0.02      # beats the incumbent on the held-out sample by >= 2 percentage points
max_case_regressions = 0     # breaks no authored case the incumbent got right
min_case_accuracy = 0.70     # and clears an absolute floor on the authored cases

Three conditions, each for a different failure:

1. Held-out gain. The candidate has to be measurably better on data neither model trained on, not merely equal.
2. No regressions on authored cases. A model can raise the average by breaking something the old model got right. This line makes that trade visible, and refuses it unless you change the rule on purpose.
3. An absolute floor. Beating a weak incumbent is not enough on its own.

These numbers are illustrative. Yours should come from a requirement, not from this tutorial:

• Why 2 points? Stand-in for "the smallest gain worth the cost of switching". If a switch costs a migration and a week of watching, that bar is higher than if it costs a config flag.
• Why zero regressions? It suits a small set of critical, unambiguous behaviours like these twenty. Applied to thousands of noisy cases, zero would block almost every worthwhile change; there you would allow a small number, or a rate.
• Why 70%? The level below which the model is not usable for this job, whatever the incumbent does.

A different decision has a different rule. A cheaper model does not have to beat the incumbent; it has to not lose much. Sketched as a policy, that gate might read:

# policy sketch for "can we move to the cheaper model?"
# not usable with this tutorial's scorer, which only knows a minimum gain
[gate]
max_heldout_loss = 0.01      # no more than 1 percentage point worse on held-out data
max_case_regressions = 0     # and still nothing critical broken

Why write it first? Once you have seen results, every threshold you pick describes what you got. Written in advance, it is a prediction your model can fail. Researchers call this preregistration. The cheaper-model sketch above is a non-inferiority test: it asks whether the candidate is no worse than the incumbent by more than a set margin, rather than whether it is better.

Step 2: Write the cases you actually care about

Random samples measure the average. They rarely contain enough of the sentences that break models: negation, contrast, faint praise. So we write those by hand, one line each, with a note on what each case is for. Save this as evals/cases.jsonl:

{"id": "neg-01", "text": "not a bad movie at all , honestly", "label": 1, "why": "negated negative"}
{"id": "neg-02", "text": "i did not enjoy a single minute of it", "label": 0, "why": "negated positive"}
{"id": "neg-03", "text": "it is not as funny as it thinks it is", "label": 0, "why": "negated comparison"}
{"id": "neg-04", "text": "never boring , never predictable", "label": 1, "why": "stacked negation"}
{"id": "con-01", "text": "the cast is wonderful but the script is a mess", "label": 0, "why": "contrast, second clause wins"}
{"id": "con-02", "text": "slow start , but the last hour is terrific", "label": 1, "why": "contrast, second clause wins"}
{"id": "con-03", "text": "beautiful to look at , painful to sit through", "label": 0, "why": "contrast"}
{"id": "con-04", "text": "a clumsy premise that somehow becomes moving", "label": 1, "why": "contrast"}
{"id": "pos-01", "text": "a sharp , funny and generous film", "label": 1, "why": "plain positive"}
{"id": "pos-02", "text": "the best thing i have seen all year", "label": 1, "why": "plain positive"}
{"id": "pos-03", "text": "warm , smart and beautifully acted", "label": 1, "why": "plain positive"}
{"id": "pos-04", "text": "it earns every one of its tears", "label": 1, "why": "positive without sentiment words"}
{"id": "neg-05", "text": "a tedious , joyless slog", "label": 0, "why": "plain negative"}
{"id": "neg-06", "text": "the worst kind of lazy sequel", "label": 0, "why": "plain negative"}
{"id": "neg-07", "text": "i checked my watch more than the plot deserved", "label": 0, "why": "negative without sentiment words"}
{"id": "neg-08", "text": "flat , loud and instantly forgettable", "label": 0, "why": "plain negative"}
{"id": "mix-01", "text": "too long , yet i would watch it again", "label": 1, "why": "mixed, verdict positive"}
{"id": "mix-02", "text": "great soundtrack , nothing else", "label": 0, "why": "faint praise"}
{"id": "mix-03", "text": "fine , i guess , if you have nothing better to do", "label": 0, "why": "faint praise"}
{"id": "mix-04", "text": "rough edges and all , it works", "label": 1, "why": "concession then verdict"}

The why field pays off later: when a case fails, it tells you which behaviour broke, which a moved number never will. Twenty cases is small on purpose: every one is readable, and every miss gets read.

The lowercase text and the spaces before commas are deliberate: they match how SST-2's sentences are written, so the models see the same kind of input they were trained on.

A note on the labels. "Contrast, second clause wins" is how we labelled these particular sentences, not a law of sentiment. Mixed sentences need a labelling policy: here, the label is the reviewer's overall verdict. Write your policy down, because two people labelling the same mixed sentence will otherwise disagree, and your eval will measure that disagreement instead of the model.

Step 3: Freeze the eval

The held-out sample is 300 of the validation split's 872 rows: enough to measure a few points of difference, few enough to read the misses. Pick it and copy those rows into a file of your own, then hash the files the gate reads: the cases, the held-out rows and the rule. Copy the rows, not their positions: if you only stored which rows to use, a change to the dataset upstream could silently change what you score while your hash stayed the same. Save this as freeze.py:

# freeze.py
"""Freeze the eval: copy the held-out rows out of the dataset and hash every
file the gate reads. Run once, before training anything. run_eval.py refuses
to score if any frozen file changed afterwards.
"""
import hashlib, json, random
from pathlib import Path

REVISION = "8d51e7e4887a4caaa95b3fbebbf53c0490b58bbb"  # exact stanfordnlp/sst2 commit
SEED, N_HELDOUT = 1234, 300
FROZEN = ("evals/cases.jsonl", "evals/heldout.jsonl", "evals/gate.toml")


def frozen_digest(names=FROZEN):
    """SHA-256 over each file's name, byte length and bytes. Framing each file
    means bytes cannot move from one file into the next without changing it."""
    h = hashlib.sha256()
    for name in names:
        data = Path(name).read_bytes()
        h.update(name.encode() + b"\0" + len(data).to_bytes(8, "big") + data)
    return h.hexdigest()


def main():
    from datasets import load_dataset
    val = load_dataset("stanfordnlp/sst2", split="validation", revision=REVISION)
    idx = sorted(random.Random(SEED).sample(range(len(val)), N_HELDOUT))
    rows = [json.dumps({"text": r["sentence"], "label": r["label"]}) + "\n" for r in val.select(idx)]
    Path("evals/heldout.jsonl").write_bytes("".join(rows).encode("utf-8"))  # the rows themselves, LF on every OS

    digest = frozen_digest()
    Path("evals/FROZEN.sha256").write_text(digest + "\n")
    rows_hash = hashlib.sha256(Path("evals/heldout.jsonl").read_bytes()).hexdigest()
    print(f"held-out rows {rows_hash[:16]} ({N_HELDOUT} of {len(val)} validation rows)")
    print(f"frozen        {digest[:16]} (cases + held-out rows + gate)")


if __name__ == "__main__":
    main()

$ uv run python freeze.py
held-out rows a3a3519a046a328d (300 of 872 validation rows)
frozen        25c4473f259533b2 (cases + held-out rows + gate)

From here on, the scorer refuses to run if any of those bytes change.

Check that you got our data. Compare the first line, the held-out rows hash. If yours starts a3a3519a046a328d, your rows match ours byte for byte. Use that line, not the frozen hash, for this check: the frozen hash also covers the two files you pasted by hand, and a missing final newline or different line endings would change it even though your data is right. The pinned dataset commit and seed mean anyone can rebuild the rows, so you can share the recipe instead of the data, which matters here: SST-2's license is listed as unknown, and the companion repository commits FROZEN.sha256, not the rows.

What the hash covers, and what it does not. It detects changes to the authored cases, the held-out rows and the rule. Each file's name and length are hashed along with its bytes, so text cannot move from one file into another without changing it. It does not cover the model files, the scoring code or your installed packages, and it cannot prove when the rule was written. For that wider record, use version control and save run metadata (commit, package versions, model file hashes) with each result.

Re-freezing is allowed, and it creates a new policy. Improving the eval is often necessary. But if you change the rule after a failure and the candidate then passes, that pass is under a new rule, not a confirmation of the old decision. Keep the old result, write down what requirement changed and why, then re-freeze.

Step 4: Train the three models

In real work the candidate is your change: a new prompt, a fine-tune, a different base model. Here all three come from the same recipe, so the example stays small. Save this as train.py:

# train.py
"""Train three models on SST-2's train split (the eval uses the validation split).

incumbent: what we run today   - single words, 2,000 examples
candidate: the proposed change - words and word pairs, 20,000 examples
broken:    the candidate recipe trained on SHUFFLED labels - a known-bad change
           the eval must reject, or the eval can't detect anything
"""
import random
import joblib
from datasets import load_dataset
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

REVISION = "8d51e7e4887a4caaa95b3fbebbf53c0490b58bbb"  # same pinned commit as freeze.py
train = load_dataset("stanfordnlp/sst2", split="train", revision=REVISION).shuffle(seed=7)

def fit(n, ngrams, shuffle_labels=False):
    rows = train.select(range(n))
    x, y = rows["sentence"], list(rows["label"])
    if shuffle_labels:
        random.Random(7).shuffle(y)
    model = make_pipeline(TfidfVectorizer(ngram_range=ngrams, min_df=2),
                          LogisticRegression(max_iter=1000))
    return model.fit(x, y)

joblib.dump(fit(2_000, (1, 1)), "incumbent.joblib")
joblib.dump(fit(20_000, (1, 2)), "candidate.joblib")
joblib.dump(fit(20_000, (1, 2), shuffle_labels=True), "broken.joblib")
print("trained: incumbent, candidate, broken")

The candidate should be better in theory: ten times the data, and word pairs, so it can see "not bad" as a phrase rather than two separate words. The broken model uses the candidate's exact recipe on shuffled labels, so any skill it shows is luck.

$ uv run python train.py
trained: incumbent, candidate, broken

Training all three took about 3 seconds on our machine.

Step 5: Score against the gate

Save this as run_eval.py. The gate itself is one small function, decide(), so you can test it on its own (step 7):

# run_eval.py
"""Score each model on the frozen eval and apply the pre-declared gate.

Usage: python run_eval.py candidate      (or: broken)
Exit code 0 = eligible for the next release step, 1 = rejected,
2 = eval inputs were altered, 3 = the eval could not run (e.g. a model file
is missing). A crash must never look like a verdict.
"""
import json, math, sys, tomllib
from fractions import Fraction
from pathlib import Path


def need(share, total):
    """Smallest whole count that meets `share` of `total`, in exact arithmetic.

    Floats are not safe here: 210/300 - 204/300 == 0.019999999999999907, which
    would reject an exactly-2-point gain. Fraction("0.02") * 300 is exactly 6.
    """
    return math.ceil(Fraction(str(share)) * total)


def decide(gate, inc_held, new_held, inc_cases, new_cases):
    """The whole gate as one pure function over per-row right/wrong lists."""
    # An empty or mismatched input must be an error, never a verdict: zip() would
    # silently truncate, and zero rows would pass every condition.
    if not (len(inc_held) == len(new_held) > 0 and len(inc_cases) == len(new_cases) > 0):
        raise ValueError("the gate needs non-empty, equal-length results for both models")
    n_held, n_cases = len(new_held), len(new_cases)
    gain = sum(new_held) - sum(inc_held)
    regressions = [i for i, (a, b) in enumerate(zip(inc_cases, new_cases)) if a and not b]
    return {
        f"held-out gain >= {gate['min_heldout_gain']} (need +{need(gate['min_heldout_gain'], n_held)} of {n_held}, got {gain:+d})":
            gain >= need(gate["min_heldout_gain"], n_held),
        f"authored-case regressions <= {gate['max_case_regressions']}":
            len(regressions) <= gate["max_case_regressions"],
        f"authored accuracy >= {gate['min_case_accuracy']} (need {need(gate['min_case_accuracy'], n_cases)} of {n_cases}, got {sum(new_cases)})":
            sum(new_cases) >= need(gate["min_case_accuracy"], n_cases),
    }, regressions


def main():
    # Everything that can fail lives inside main(), so any crash becomes exit 3.
    import joblib  # third-party: a missing install is a crash, not a verdict
    from freeze import frozen_digest  # step 3's file; missing, it is a crash (3), not a verdict
    digest = frozen_digest()
    if digest != Path("evals/FROZEN.sha256").read_text().strip():
        print("REFUSED: eval inputs changed after freezing. Re-freeze deliberately, and say why.", file=sys.stderr)
        return 2

    gate = tomllib.loads(Path("evals/gate.toml").read_text())["gate"]
    cases = [json.loads(l) for l in Path("evals/cases.jsonl").read_text().splitlines()]
    held = [json.loads(l) for l in Path("evals/heldout.jsonl").read_text().splitlines()]  # the frozen bytes

    def right(m, rows):
        preds = m.predict([r["text"] for r in rows])
        if len(preds) != len(rows):
            raise ValueError(f"got {len(preds)} predictions for {len(rows)} rows")
        return [int(p) == r["label"] for p, r in zip(preds, rows)]

    def score(name):
        m = joblib.load(f"{name}.joblib")
        return right(m, held), right(m, cases)

    inc_held, inc_cases = score("incumbent")
    name = sys.argv[1] if len(sys.argv) > 1 else "candidate"
    new_held, new_cases = score(name)

    n_held, n_cases = len(held), len(cases)
    print(f"{'':12}{f'held-out ({n_held})':>16}{f'authored ({n_cases})':>16}")
    print(f"{'incumbent':12}{sum(inc_held) / n_held:>16.3f}{sum(inc_cases):>13}/{n_cases}")
    print(f"{name:12}{sum(new_held) / n_held:>16.3f}{sum(new_cases):>13}/{n_cases}")

    print(f"\nMisses for {name} on authored cases (read these, don't just count them):")
    for c, ok in zip(cases, new_cases):
        if not ok:
            print(f"  {c['id']:7} [{c['why']}] {c['text']!r} (expected {'pos' if c['label'] else 'neg'})")

    checks, regressions = decide(gate, inc_held, new_held, inc_cases, new_cases)
    print(f"\nGate (frozen inputs {digest[:16]}):")
    for label, passed in checks.items():
        extra = f" {[cases[i]['id'] for i in regressions]}" if "regressions" in label and regressions else ""
        print(f"  {'PASS' if passed else 'FAIL'}  {label}{extra}")
    verdict = all(checks.values())
    print(f"\nVERDICT: {name} {'is eligible for the next release step' if verdict else 'must NOT replace the incumbent'}")
    return 0 if verdict else 1


if __name__ == "__main__":
    try:
        code = main()
    except Exception as e:  # a crash is not a verdict: missing model, bad file, anything
        print(f"ERROR: the eval could not run ({type(e).__name__}: {e}). No verdict.", file=sys.stderr)
        code = 3
    sys.exit(code)

Two details matter more than they look.

Counts, not floating-point averages. An exact 2-point gain on 300 rows is 210 correct against 204. In floating point, 210/300 - 204/300 is 0.019999999999999907, which fails >= 0.02. A gate whose job is precise pass/fail rules would reject a candidate that met its rule exactly. So need() converts each threshold into a whole number of rows, using exact fractions, and the gate compares counts.

The exit code is the verdict: 0 means eligible for the next release step, 1 means rejected, 2 means the eval inputs were altered, and 3 means the eval could not run at all. Keep that last one separate. If a missing model file crashed the script with Python's default exit code 1, a broken CI job would look exactly like a rejected model.

Step 6: Read the result

$ uv run python run_eval.py candidate
              held-out (300)   authored (20)
incumbent              0.710           14/20
candidate              0.780           14/20

Misses for candidate on authored cases (read these, don't just count them):
  neg-01  [negated negative] 'not a bad movie at all , honestly' (expected pos)
  neg-04  [stacked negation] 'never boring , never predictable' (expected pos)
  con-01  [contrast, second clause wins] 'the cast is wonderful but the script is a mess' (expected neg)
  con-02  [contrast, second clause wins] 'slow start , but the last hour is terrific' (expected pos)
  con-04  [contrast] 'a clumsy premise that somehow becomes moving' (expected pos)
  mix-01  [mixed, verdict positive] 'too long , yet i would watch it again' (expected pos)

Gate (frozen inputs 25c4473f259533b2):
  PASS  held-out gain >= 0.02 (need +6 of 300, got +21)
  FAIL  authored-case regressions <= 0 ['con-01', 'con-02', 'con-04']
  PASS  authored accuracy >= 0.7 (need 14 of 20, got 14)

VERDICT: candidate must NOT replace the incumbent
$ echo $?
1

The candidate is better on held-out data, 0.780 against 0.710, which is exactly the kind of number that usually ships. The gate still says no.

Read the regressions: con-01, con-02 and con-04. All three are contrast sentences, where the verdict sits in one clause and the other clause pulls the opposite way ("slow start, but the last hour is terrific"). The incumbent got them right and the candidate gets them wrong. This is the lesson of the whole tutorial: a total hides which cases changed. Here both models score 14 out of 20 while failing different sentences, the most vivid version of it: the candidate fixed some cases and broke others, and the total alone shows nothing.

Be careful how far you take it, because two of those three regressions are close calls. Here is each model's probability that the sentence is positive:

Only con-04 is a clear flip. On con-01 and con-02 the incumbent was right by a hair, and a slightly different training run could move either one. A zero-regression rule treats a 0.507 exactly like a 0.99. Declaring the rule in advance makes it binding, not right. If near-ties like these bother you, the fix is a margin written into the next gate, for example counting a case as a regression only if the incumbent was right with a probability above 0.6.

How typical is this result? After writing the tutorial, we ran a sensitivity check, which is not part of the gate: we retrained the incumbent and candidate with the same recipe on training shuffle seeds 0 to 19 and applied the same gate to each pair. Across the 20 runs:

• the candidate cleared the held-out gain bar in all 20
• it regressed on at least one authored case in 16, almost always a contrast sentence (con-04 in 12 runs, con-02 in 9)
• it passed the whole gate in 4
• the incumbent fell below the 70% floor in 13
• both models scored 14/20 in only 2

This measures variation between training runs of this recipe, not behaviour on real traffic. The companion repository's sensitivity.py reproduces every number above in under a minute.

What it means for seed 7: the pattern of a higher average hiding contrast regressions recurred in 16 of 20 training shuffles. But seed 7's matching totals are a vivid illustration, not a typical run.

On provenance: the cases and all three thresholds were written before any model was trained, and seed 7 was the first seed we tried. That is from our session log, which you cannot see, and no public commit shows the order, because the companion repository was published after the fact. Nothing in the hash can prove it either. For your own gates, commit gate.toml and the cases before the training code, and the history becomes the evidence.

There is a second, free lesson in the misses. Step 4 said word pairs let the candidate see "not bad" as a phrase, yet neg-01 got worse (0.216 to 0.111 probability of positive). The sentence says "not a bad movie", so the pair "not bad" never appears, and the phrase feature cannot fire. A feature you designed for a behaviour only helps if your cases contain that behaviour in the form the feature can see.

Three failed examples say these contrast sentences regressed. They do not prove that contrast handling got worse across real traffic; for that, you would write more contrast cases.

Whether the trade is acceptable is a product decision, not a statistics one. If contrast sentences matter to your users, the gate just stopped a regression that raised your metric. If they don't, change the rule on purpose, as a new policy (step 3). What the eval must not do is let the trade through unnoticed.

Step 7: Prove the gate works

A gate you have never seen reject anything might not be able to. A gate you have never seen pass anything might reject everything. Check both.

First, a known-bad change:

$ uv run python run_eval.py broken
              held-out (300)   authored (20)
incumbent              0.710           14/20
broken                 0.513           10/20

Misses for broken on authored cases (read these, don't just count them):
  neg-03  [negated comparison] 'it is not as funny as it thinks it is' (expected neg)
  con-01  [contrast, second clause wins] 'the cast is wonderful but the script is a mess' (expected neg)
  con-03  [contrast] 'beautiful to look at , painful to sit through' (expected neg)
  pos-01  [plain positive] 'a sharp , funny and generous film' (expected pos)
  pos-02  [plain positive] 'the best thing i have seen all year' (expected pos)
  neg-06  [plain negative] 'the worst kind of lazy sequel' (expected neg)
  neg-08  [plain negative] 'flat , loud and instantly forgettable' (expected neg)
  mix-02  [faint praise] 'great soundtrack , nothing else' (expected neg)
  mix-03  [faint praise] 'fine , i guess , if you have nothing better to do' (expected neg)
  mix-04  [concession then verdict] 'rough edges and all , it works' (expected pos)

Gate (frozen inputs 25c4473f259533b2):
  FAIL  held-out gain >= 0.02 (need +6 of 300, got -59)
  FAIL  authored-case regressions <= 0 ['neg-03', 'con-01', 'pos-01', 'pos-02', 'neg-06', 'neg-08', 'mix-02', 'mix-03', 'mix-04']
  FAIL  authored accuracy >= 0.7 (need 14 of 20, got 10)

VERDICT: broken must NOT replace the incumbent
$ echo $?
1

It fails all three conditions. The floor alone would have caught this one: 10 of 20 is well under 70%. Floors are not useless; they catch changes that are simply bad. What a floor cannot catch is the step 6 case, where the candidate cleared the floor with the same 14/20 as the incumbent while breaking different sentences. That takes the comparison.

Second, the gate's logic. The broken model shows the gate can say no, but a gate that always says no would pass that demonstration too. So test decide() directly with made-up results: one that must pass, then one that violates each condition on its own. Save this as test_gate.py:

# test_gate.py
"""Tests of the gate's LOGIC with synthetic results. These say nothing about
whether any real model is good; they prove the gate can pass, and that each
condition can fail on its own."""
import unittest
from run_eval import decide

GATE = {"min_heldout_gain": 0.02, "max_case_regressions": 0, "min_case_accuracy": 0.70}
INC_HELD = [True] * 204 + [False] * 96            # incumbent: 204/300 held-out
INC_CASES = [True] * 14 + [False] * 6              # incumbent: 14/20 authored


class GateLogic(unittest.TestCase):
    def verdict(self, new_held, new_cases):
        checks, _ = decide(GATE, INC_HELD, new_held, INC_CASES, new_cases)
        return list(checks.values())

    def test_can_pass(self):
        self.assertEqual(self.verdict([True] * 214 + [False] * 86, [True] * 16 + [False] * 4), [True] * 3)

    def test_exact_two_point_gain_passes(self):
        # 210/300 - 204/300 is 0.0199999... in floating point; exact counting must still pass it.
        self.assertTrue(self.verdict([True] * 210 + [False] * 90, [True] * 16 + [False] * 4)[0])

    def test_small_gain_fails_alone(self):
        self.assertEqual(self.verdict([True] * 209 + [False] * 91, [True] * 16 + [False] * 4), [False, True, True])

    def test_one_regression_fails_alone(self):
        new_cases = [True] * 13 + [False] + [True] * 6    # breaks case 13, fixes the other six
        self.assertEqual(self.verdict([True] * 214 + [False] * 86, new_cases), [True, False, True])

    def test_floor_fails_alone(self):
        weak_inc = [True] * 10 + [False] * 10              # a weak incumbent at 10/20
        checks, _ = decide(GATE, INC_HELD, [True] * 214 + [False] * 86, weak_inc, [True] * 13 + [False] * 7)
        self.assertEqual(list(checks.values()), [True, True, False])

    def test_malformed_input_is_an_error_not_a_verdict(self):
        for inc_held, new_held, inc_cases, new_cases in [
            ([], [], [], []),                                  # nothing scored
            (INC_HELD, INC_HELD, INC_CASES, [True] * 5),       # candidate scored on 5 of 20 cases
            (INC_HELD, INC_HELD[:299], INC_CASES, INC_CASES),  # one held-out row missing
        ]:
            with self.assertRaises(ValueError):
                decide(GATE, inc_held, new_held, inc_cases, new_cases)


if __name__ == "__main__":
    unittest.main()

$ uv run python -m unittest test_gate
......
----------------------------------------------------------------------
Ran 6 tests in 0.000s

OK

These are tests of the gate, not evidence about any model. They prove it can pass, that an exact 2-point gain counts, that each condition fails independently, and that empty or mismatched results raise an error instead of producing a verdict. Without that last check, decide() would pass all three conditions on zero rows, and zip() would quietly score a candidate on 5 of 20 cases as if that were all of them. There is no need to weaken the real thresholds to manufacture a passing candidate.

Third, the freeze. Back up the rule, change it, and the scorer refuses:

$ cp evals/gate.toml gate.toml.bak
$ echo "# tamper" >> evals/gate.toml
$ uv run python run_eval.py candidate
REFUSED: eval inputs changed after freezing. Re-freeze deliberately, and say why.
$ echo $?
2
$ mv gate.toml.bak evals/gate.toml

The last line restores the original, and the scorer runs again.

Is the gain real?

Optional, and worth doing once. The 7-point gain is a fact about these 300 rows: on them, the candidate got 21 more right. The question a confidence interval answers is different: how much might that gain move on other sentences drawn the same way?

A paired bootstrap estimates it. Resample the 300 rows with replacement many times, and each time score both models on the same resampled rows. Save this as bootstrap.py:

# bootstrap.py
"""Optional: how uncertain is the held-out gain? Paired bootstrap over the 300 rows.

Both models are scored on the SAME resampled rows each round (that is what
"paired" means), so the interval is for the difference, not two separate scores.
"""
import json
from pathlib import Path
import joblib
import numpy as np

held = [json.loads(l) for l in Path("evals/heldout.jsonl").read_text().splitlines()]
texts, labels = [r["text"] for r in held], np.array([r["label"] for r in held])
right = {n: joblib.load(f"{n}.joblib").predict(texts) == labels for n in ("incumbent", "candidate")}
diff = right["candidate"].astype(int) - right["incumbent"].astype(int)   # +1, 0 or -1 per row

rng = np.random.default_rng(0)
rows = rng.integers(0, len(diff), size=(20_000, len(diff)))              # 20,000 resamples of row indices
gains = diff[rows].mean(axis=1)
low, high = np.percentile(gains, [2.5, 97.5])
print(f"observed gain on these 300 rows: {diff.mean():+.3f}")
print(f"rows only the candidate got right: {(diff == 1).sum()}, only the incumbent: {(diff == -1).sum()}")
print(f"95% paired bootstrap interval: [{low:+.3f}, {high:+.3f}]")

$ uv run python bootstrap.py
observed gain on these 300 rows: +0.070
rows only the candidate got right: 37, only the incumbent: 16
95% paired bootstrap interval: [+0.023, +0.117]

How to read it, and how not to:

• The interval is about the wider population, not these rows. The 0.070 on these 300 is already known exactly. The interval says the gain on similar SST-2 sentences is plausibly between about 2 and 12 points, assuming the sample is representative. It says nothing about your users' sentences.
• Above zero is not the same as "at least 2 points". An interval that clears zero supports some improvement. If your rule is a minimum useful gain, check that the interval clears that. Here the lower end, +2.3, happens to clear 2, only just.
• More rows narrow it. If an interval is too wide to decide with, the fix is usually more data. Simply raising the bar does not help: the right bar comes from the smallest gain that matters and how much risk of a wrong decision you can accept.

The gate above decides on the observed gain; this analysis does not change its verdict. Requiring the interval to clear a bound would be a different policy, and you would write it into gate.toml before scoring, like any other rule.

SciPy can do the same calculation with scipy.stats.bootstrap and its paired option.

The other evals around the gate

An eval is any repeatable measurement of whether a model does what you need. The gate you built is one layer:

• Offline gate (this tutorial). Runs before a change ships, on a fixed set of cases, and answers yes or no. Cheap, fast and repeatable. It can only judge the situations you thought to include.
• Regression suite. The same idea, run on every change to catch old bugs coming back. Your gate's cases grow into one over time.
• Shadow run. The candidate runs on real traffic next to the current model, and you compare the answers without showing users the new one. Catches what your cases missed.
• Canary. The candidate serves a small slice of real users while you watch for problems, with a quick way back. Unlike a shadow run, some users do see its answers.
• Online monitoring. Watches the model in production: error reports, user corrections, drift in what people ask. Tells you when the world changed under you.
• Human review. People read samples, especially where the models disagree. It is how you check the rubric or model grader you use for open-ended output, and how you settle ambiguous cases.

If you have none of these today, start with the offline gate.

What this eval cannot tell you

Say these limits out loud. They are part of the result.

• Twenty authored cases is a smoke test, not a survey. It catches the failure kinds you thought to write down, and nothing else.
• The held-out sample comes from the same distribution as the training data. It says nothing about sentences your users write that look nothing like movie reviews.
• A pass is not proof the model is good. It means the model met the conditions you declared in advance. If those conditions are wrong, a pass is wrong too.
• Offline is not live. A shadow run or canary on real traffic is the next step for a change that matters.
• Public cases are not a promotion set. The next paragraph explains the difference.

Three kinds of data, three jobs. Our tutorial uses public data for everything, because the method is the point. Real projects separate three roles:

1. Training data, for fitting the model.
2. Regression and development cases, like our twenty. These should be visible: developers read them, fix failures against them, and they keep old bugs from coming back. That value survives being read.
3. A promotion set, for the final go/no-go. This one is kept away from the people and jobs that build models, because once a team tunes against a set, a good score on it stops being independent evidence of how the model does on new inputs. Store it where only the gate can read it, and commit its hash rather than its contents.

Even a hidden set wears out: every time you run candidates against it and act on the result, a little information leaks back. Refresh it from time to time. Scikit-learn's guide to common pitfalls covers how leakage happens.

Taking it back to your application

The demo scores a classifier, where every answer is simply right or wrong. Most applications need a few translations.

Name the decision first. "Should we switch to the new model?", "Is this prompt change safe?" and "Can we move to the cheaper model?" are different evals with different cases and different rules. One gate per decision.

Get cases from where your system actually fails. Support tickets, user corrections, bug reports and incidents are better sources than your imagination. Start with 20 to 50 cases you would be embarrassed to get wrong, give each one a reason like our why field, and add a case every time something slips through in production.

Grade each case with a check that measures its requirement. This is the part the demo skips, because a label either matches or it doesn't. No kind of check is trustworthy in general; each is good for one kind of requirement:

• Structural checks (valid JSON, a required field present, the right category, a tool called with the right arguments) are reliable for structure. Valid JSON can still contain the wrong answer.
• Rule checks (must mention the refund policy, must not include a phone number, under 200 words) catch specific, spelled-out requirements.
• Model-graded checks ("LLM as judge": another model scores the answer against a written rubric) handle open-ended quality, but the grader is a model too, with known biases. Check its scores against human-graded examples before you trust it, and treat that check as an eval of its own.

Two worked translations:

• A retrieval-based support assistant. The candidate is the prompt, the model, the retrieval settings and the knowledge base together; change any one and you run the gate. Grade two things per case: did it retrieve the right source passage, and is the answer correct and supported by that passage? A case might be "customer asks for a refund after 45 days", with a rule check that the answer states the 30-day policy and a check that the refund-policy document was retrieved.
• An agent. Grade the tool choices, the arguments, and whether the task actually got done. "Book a table for four on Friday" passes only if a reservation actually exists afterwards, confirmed for Friday and four people, and nothing else was changed. A correct-looking tool call that failed is still a failure: check the outcome, not the attempt.

Expect the same input to give different outputs. Generative models vary between runs, so one sampled answer per case is a weak basis for "zero regressions". Run each case several times, for both the incumbent and the candidate under the same protocol, and define a regression relative to the incumbent: for example, a case the incumbent passed in all five runs and the candidate failed in any. How many failures you tolerate depends on the requirement. For a critical behaviour, one observed violation can be enough to block promotion. Five runs is a starting point, not strong evidence of reliability.

Put it where the decision happens. If model changes arrive as pull requests, it's a required CI check. If you switch models through a config flag, run it before the flag flips. If a vendor updates the model under you, run it on a schedule and alert when a formerly passing gate fails.

In CI, the job has to produce or fetch both models first (download the incumbent you deploy, build the candidate from the change under review) and treat exit 3 as a broken job to fix, never as a verdict. When a candidate ships, it becomes the incumbent the next change must beat. Keep the gate, the public cases and the freeze hash in version control next to the code they judge, so a re-freeze shows up in review as a diff with a reason attached; commit held-out rows only if their license allows it.

Keep it cheap enough to run every time. A gate that takes an hour gets skipped. Keep a small fast gate for every change and a larger suite for releases.

Where to go from here

• Design better cases: Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. The closest relative of this tutorial: it moves from individual cases like ours to systematic coverage of capabilities such as negation.
• See the whole evaluation process: Hugging Face's Evaluation Guidebook. The natural next stop when this small gate grows into an evaluation programme.
• Quantify uncertainty: SciPy's bootstrap. Reproduce our interval with it as an exercise.
• Evaluate open-ended answers: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Explains the biases of model graders (position, answer length, favouring their own outputs), and why you check them against human grades.
• Scale evaluation runs: Lighteval, for running broader evaluations of models on the Hub. The evaluate library is still useful as a metric library, though its own docs point to Lighteval as the more actively maintained option.

Your turn

Pick one change you plan to make to a real application: a new model, a prompt edit, a cheaper provider.

1. Write its decision in one sentence, and its rule in a gate.toml, before you run anything.
2. Write ten cases from real failures, each with a why.
3. Decide how each case is graded.
4. Run the current version and the candidate on identical inputs.
5. Find one case where they disagree, explain it, and write down what evidence would change your release decision.

If you can do step 5, you have understood the tutorial.

The checklist

1. Name the decision the eval makes, in one sentence.
2. Write the pass conditions before training or scoring anything, and know which requirement each one comes from.
3. Hand-write cases for the behaviour you care about, each with a reason.
4. Freeze the inputs the gate reads, and refuse to score if they change.
5. Compare against what you run today, on identical cases, in exact counts.
6. Read the misses, not just the totals.
7. Show that a known-bad change fails, and that the gate can pass.
8. Write down what a pass does not prove.