A backtest against real bookings (scripts/backtest-categorize.ts, read-only)
showed the selector reporting 0.95 on pure category guesses, so "säker" was a
lie: high-confidence picks were only ~52% accurate.
Confidence is now driven by DETERMINISTIC BACKING — the confidence of a
candidate that independently points at the chosen account — not the model's
verbalized confidence (which the backtest showed is ~always "high"):
- a BACKED pick takes the candidate's confidence, reduced only when the model
itself is unsure;
- an UNBACKED pick (a category guess no candidate agreed with) is capped at 0.7,
below the säker band (0.8) — a guess is never "säker", however sure the model
claims to be.
Re-running the backtest: säker (conf ≥0.8) accuracy 52% → 73%, and it now fires
only on template-backed picks. Still not auto-book-grade (want ~95%), so
auto-book stays off until isotonic calibration on real approvals — but the band
is now honest, which is what makes the whole UX trustworthy.
Also adds the read-only backtest harness so we can re-measure after any change.
37 categorize tests green; lint + guards clean.
Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The core of the "optimal" RIP-4 categorizer, built to the researched 2026
architecture (retrieve → SELECT → escalate). Given a transaction, its underlag,
and the deterministic candidate accounts the engine already retrieved, the
model reasons and then CHOOSES from a closed set:
- a retrieved candidate account (the known path), or
- a standard business category → deterministic BAS account (the novel path,
a first-time vendor with no candidate), or
- needs_review (routed to a human, never auto-applied).
Because it picks from a closed enum, the model can't invent an account; the
account + VAT resolution stays deterministic and validated (the model chooses,
code resolves the numbers). It runs on any backend via getAiService()
.generateStructured — Bedrock or a local model.
Founder chose the optimal path (the model selects on every transaction, LLM
calls are fine), so confidence uses self-consistency: N samples (default 3),
majority vote, agreement fraction, combined with the model's stated confidence
and floored by the winning candidate's deterministic confidence — never the
model's verbalized confidence alone (systematically overconfident). reasoning
precedes choice in the schema (reason-before-choice); an unknown/hallucinated
choice degrades to needs_review.
13 unit tests (candidate/category/needs_review resolution, reverse-charge gating,
self-consistency majority + agreement + candidate floor, prompt/schema shape).
Not yet wired: Tier 1 candidate gathering + a route + the ApprovalCard UI.
Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
