Build patches

Evaluation methodology

How baseline and patched runs are compared, the thresholds that set a starting evidence state, and the limits.

A patch is worth installing only if it fixes the failure on cases it never trained on and doesn't break other things. This page summarizes the protocol the evaluation harness runs. The full write-up, with the split table and the grader, is on the methodology page.

One harness, four arms

The base checkpoint is loaded once at its pinned revision, in float32, and every arm uses the same tokenizer, chat template, system prompt and decoding settings.

Arm	What runs
Base	The unmodified model on the target suite and on the regression suite.
Prompt-only control	The base model with three worked examples from the train split added to the system prompt. If a cheaper prompt change matches the patch, the report shows it.
Patched	The adapter loaded with PEFT, on both suites.
Rollback check	The same loaded model with the adapter disabled. Its outputs are compared character for character with the base arm, and the report records whether they were identical.

Decoding is greedy with a 128-token output limit. The report pins the grader source digests, the library versions, the device and the dtype.

Splits and sealed held-out cases

Splits are made by template family, never by shuffling rows, so a held-out case never shares a sentence shape with anything the adapter trained on. For the first-party payment task the split is also made by payee pool.

Held-out is the only split a published version may cite. Its cases are not published. The report publishes the suite's sha256, the aggregate counts and per-field error counts, and reviewers with access can reproduce the run. The packager refuses to build a version from anything other than a held-out evaluation.
Validation is for development checks. Its cases can appear on a patch page as examples.
Train is used for gradient updates and the prompt-only examples.

Suites are built deterministically, and each report records the sha256 of the exact suite it ran, so a run on a different suite can't be passed off as the same one.

Every case counts

Every generated case is eligible. Empty, unparseable and truncated replies count as failures. Nothing is excluded afterwards, so the denominator on a patch page is always the full suite size. Scores are shown as correct over eligible with a Wilson 95% interval, and changes are reported in absolute percentage points.

Regression suite

general-regression-v1 has 60 cases across six families of unrelated behaviour, graded by exact match. A regression is introduced when the base passed a case and the patched model failed it. One family, contact details into JSON with a different schema, is there to catch a patch that forces every JSON request into its own format.

Thresholds and the starting state

Thresholds are declared before the held-out run and written into each version's manifest. The defaults:

at least +10 percentage points of gain on the held-out suite, and
no more than 2 regression cases introduced.

Result	Starting state
No gain, or the regression score drops by more than the allowance	`DEGRADED`
Both thresholds met	`IMPROVED_ON_SCOPE`
Anything in between	`EXPERIMENTAL`

Only an operator who reproduces the held-out result and checks rights and regressions can move a version to VERIFIED_FOR_SCOPE. The evidence states are defined on the methodology page.

Limits

First-party task data is synthetic, generated from templates Retoque wrote. Real inputs are messier.
One small base model, Qwen2.5-1.5B-Instruct. Results don't transfer to larger models by assumption.
Training and evaluation run on Apple Silicon through PyTorch MPS. CUDA and CPU numerics can differ slightly, and that hasn't been measured.
Only greedy decoding is evaluated. There are no sampled-decoding or repeated-seed variance figures yet.
vLLM and other runtimes are untested for published patches.