Lab
What did this change fix, and what did it break?
The real output of my evaluation harness, on 25 synthetic cases for a fictional inventory assistant, graded on two recorded runs. Start on the diff: the pass rate goes up and one case quietly goes down.
- Loading…
regressed passed in v1, fails in v2
fixed failed in v1, passes in v2
still failing fails in both
How to read it
md1is the failure that started this: a quantity already net of reserved stock is reduced a second time, so the agent answers 70 instead of 100. Run v2 fixes it.tr4is the regression. An instruction meant to make the agent check data first made it query the inventory for a question about return policy, which has nothing to do with inventory. Open it in the diff view to see both answers.nm4fails in both runs. The tool returns stock one page at a time, so the agent ranks the first page and answers as if it had seen everything, without saying there is more. The fix is a rule to read to the last page or to say the list is partial. It is a known gap, not a surprise.
What is real and what is not
The grading is real: the harness in agent-playbook produced these results from these answers. The answers themselves are hand-written to illustrate a realistic failure and fix. They are not output from a model.