Proving a model swap is safe.

Where it came from

In a real automation, a document-extraction step ran on a metered prompt builder. I replaced it with a general agent runtime that costs less per run. The new step produced the same output on the test I ran, which is not the same as knowing it would on every document. This repository is the version of that check I would want next time: repeatable, and strict where it should be.

The documents here are synthetic, and the candidate outputs are hand-written fixtures standing in for model output. No live model is called.

Method

1. Fix the expected output first. 12 golden records define what correct means before any candidate is judged.
2. Validate against a schema. A small validator returns structured errors with paths and codes. The same schema is the contract for the golden set and the candidates.
3. Normalize, then compare. 4,5 out of 5, 14 Mar 2025 and N/A are cosmetic. A rating of 9.0 on a 10 scale where the golden says 4.5 on a 5 scale is not, and normalization must not hide it.
4. Classify every mismatch. Missing, extra (a hallucinated field), wrong type, wrong value, or format only.
5. Gate per field class. Not just an overall number.
6. Reason about cost separately. A parametric model with user-supplied prices compares per-call and flat-capacity pricing and finds the break-even volume. The numbers in the docs are labelled illustrative placeholders.

Result

Real output of python3 -m extraction_demo summary:

The gap between field accuracy and strict accuracy for candidate_a (0.992 against 0.856) is the point of normalizing: most of its differences are formatting, and strict comparison would have rejected a good candidate. candidate_b carries the failures I expect from a swap: a dropped participant count, a rating on the wrong scale, a hallucinated field, a wrong year, and two reviewers merged into one string.

Why a gate per class

The rule-based baseline scores 0.977 overall and still fails the gate, because it is wrong on a numeric field and an identity field. An overall number averages a wrong recommendation into 131 right cells. In a pipeline that feeds decisions, one wrong rating or date matters more than a slightly paraphrased finding, so the gate has a strict floor for identity, categorical and numeric fields and a looser one for narrative text.

Limits