A backtest against real bookings (scripts/backtest-categorize.ts, read-only)
showed the selector reporting 0.95 on pure category guesses, so "säker" was a
lie: high-confidence picks were only ~52% accurate.
Confidence is now driven by DETERMINISTIC BACKING — the confidence of a
candidate that independently points at the chosen account — not the model's
verbalized confidence (which the backtest showed is ~always "high"):
- a BACKED pick takes the candidate's confidence, reduced only when the model
itself is unsure;
- an UNBACKED pick (a category guess no candidate agreed with) is capped at 0.7,
below the säker band (0.8) — a guess is never "säker", however sure the model
claims to be.
Re-running the backtest: säker (conf ≥0.8) accuracy 52% → 73%, and it now fires
only on template-backed picks. Still not auto-book-grade (want ~95%), so
auto-book stays off until isotonic calibration on real approvals — but the band
is now honest, which is what makes the whole UX trustworthy.
Also adds the read-only backtest harness so we can re-measure after any change.
37 categorize tests green; lint + guards clean.
Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Jakob Wennberg
d3409183c0