skill-autobench — write the eval from lived usage
Convention: see conventions/brain-first.md — mining starts in the brain. Search the conversation archive before touching raw transcript files, and never declare "no history" without having queried the brain first.
Convention: see conventions/model-routing.md — mining and synthesis run on the cheap tier by default. The full multi-model judging pass is an explicit opt-in (see Contract).
The self-improving loop has three legs: an eval, a variant generator
(SkillOpt), and a replay + judge harness (gbrain eval cross-modal). The
generator and the judge ship with gbrain. The persistently missing leg is the
eval author — someone has to WRITE the eval, and a spec-derived benchmark
only tests what the skill promised, not what users actually asked for or what
actually went wrong. This skill writes the eval from reality instead of
imagination.
Pipeline
1. MINE — extract real invocation windows
Substrates, in priority order:
- Brain conversation archive — pages under
conversations/, populated by the conversation-…
