Evaluating agents before they ship.

Why build it

The agent I run for a team had no automated evaluation when I first rolled it out. I found a real metric error only because the agent happened to read a glossary and told me (the story is here). Nothing I had built was designed to catch it. This repository is what I built so the next one is caught by a test.

The diff that matters

25 synthetic cases for a fictional inventory assistant, graded on two recorded runs. The transcripts are hand-written to show a realistic failure and fix; they are not output from a real model.

$ python3 -m evalkit diff --cases evals/cases/inventory_assistant.jsonl \
        --before evals/transcripts/v1.jsonl --after evals/transcripts/v2.jsonl
v1 -> v2
  pass rate 72% -> 92%
  regressed: tr4
  fixed: md1, tr3, ap1, fm3, sr2, nm1
  still_failing: nm4

Case ids are a category plus a number: md metric definition, tr tool routing, ap approval, fm format, sr source routing, nm numeric.

The pass rate rose 20 points and a regression sits inside it. An instruction meant to make the agent check the data first fixed six cases, and made it query the inventory for an out-of-scope question (tr4). If I had looked only at the pass rate I would have shipped it. --fail-on-regression turns that into a hard gate.

The same run also shows the failure that started this: md1 is the case where a quantity already net of reserved stock gets reduced a second time.

Design

• Deterministic graders. Contains, regex, number within a tolerance, tool called (with an argument subset), tool not called, tool order, length, JSON fields. A model-based judge adds its own variance to the thing you are measuring, so I use one only for what these cannot express.
• A missing answer is a failure, not a skip. A harness that quietly skips cases reports a pass rate that means nothing.
• Grade behavior, not only text. Right tool, right arguments, right order, and not called when it should not be: an out-of-scope question must not trigger a data query, and a write must not happen before approval.
• Every incident becomes a case before it becomes a rule. The case proves the fix and keeps proving it.

One test failed while I wrote the suite and found a real bug: 21.5 - 21.4 is 0.10000000000000142 in floating point, so a tolerance of exactly 0.1 wrongly failed. I fixed the grader, not the test.

Monitoring

A request log with one field people forget: which instruction version answered. Then a weekly report and an alert rule that needs both an absolute and a relative rise.

week       events  corrected  tool errors  instructions
2026-W12       42        10%           2%  v13
2026-W13       46        28%          22%  v14
2026-W14       45         7%           2%  v15

Alerts:
  2026-W13: correction rate 28% vs 16% before (instructions v14)
  2026-W13: tool error rate 22% vs 4% before (instructions v14)

The log is synthetic, generated with a fixed seed to show what the report looks like. It says nothing about any real agent. I also found the alert is noisy at about 45 events a week: several random seeds raised an alert in a week where nothing had changed. The docs say so, because the honest guidance is "an alert is a prompt to read the corrected conversations", not a verdict.

The learning loop

The same repository documents how my personal agent learns: every correction is triaged against a quality bar (general, specific, with a concrete example, not already covered), and only qualifying ones become rules. About 130 rules have accumulated in roughly three months of daily use. You can try the triage yourself.

Limits