Lab
Which corrections become rules?
My own agent has about 130 rules, and most of what I tell it never becomes one. Pick a correction and see how it is judged. These are real kinds of corrections, generalized; the quality bar is the one my agent actually uses.
The quality bar
A correction becomes a rule only if it is all four of these:
- General: it would help in a future session, not just this one.
- Specific: concrete enough to act on.
- Has a concrete example: a right-versus-wrong case. A rule without one gets paraphrased differently on every turn.
- Not already covered: a duplicate adds noise and a chance to contradict.
Then the agent writes the rule without being asked, and tells me in one line what it captured so I can veto it. One-offs are done and forgotten.
Why be this strict
A rule file that records everything is one nobody, including the agent, can follow. The bar keeps it short enough to keep following, and the evaluation set is what tells me when a new rule has quietly broken an old behavior.
I apply the bar by judgment, with the agent's help. It is not automated, and I have no measurement of how often I judge it wrong. The corrections here are generalized from real ones and simplified to fit on a page.