Turns the selector's raw confidence into a score that means what it says.
- lib/agent/categorize/calibration.ts: the engine. Isotonic regression
(pool-adjacent-violators, distribution-free + monotonic) over
(confidence, was_correct) samples → a calibrator; plus reliabilityByBucket,
ECE, and bandFor(). bandFor NEVER returns 'auto' without a fitted calibrator
(no silent booking on an unproven score) and never auto-books above an amount
cap. 12 engine tests (overconfidence pulled down, underconfidence lifted,
monotonicity, ECE, band gating).
- Measurement loop: migration categorize_calibration_samples (append-only,
company-scoped RLS, confidence CHECK [0,1]) + POST /api/agent/categorize/
outcome logging one sample (proposed vs actually booked) fire-and-forget from
QuickReviewDialog on a successful book (sandbox skipped). AiCategorizeProposal
surfaces the proposal metadata via onProposal.
- scripts/fit-categorize-calibration.ts (read-only): prints the reliability
diagram + ECE + fitted calibrator once data has accumulated.
Fitting needs a few hundred real outcomes, so nothing calibrates today — the
loop starts collecting, and "säker" stays uncalibrated (no auto-book) until the
data proves it. 131 unit tests green; RLS covered by a pg-real test.
Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Jakob Wennberg
704bf93e08