fix(categorize): make confidence honest — backing-driven, not the model's word (#1791)

A backtest against real bookings (scripts/backtest-categorize.ts, read-only)
showed the selector reporting 0.95 on pure category guesses, so "säker" was a
lie: high-confidence picks were only ~52% accurate.

Confidence is now driven by DETERMINISTIC BACKING — the confidence of a
candidate that independently points at the chosen account — not the model's
verbalized confidence (which the backtest showed is ~always "high"):
- a BACKED pick takes the candidate's confidence, reduced only when the model
  itself is unsure;
- an UNBACKED pick (a category guess no candidate agreed with) is capped at 0.7,
  below the säker band (0.8) — a guess is never "säker", however sure the model
  claims to be.

Re-running the backtest: säker (conf ≥0.8) accuracy 52% → 73%, and it now fires
only on template-backed picks. Still not auto-book-grade (want ~95%), so
auto-book stays off until isotonic calibration on real approvals — but the band
is now honest, which is what makes the whole UX trustworthy.

Also adds the read-only backtest harness so we can re-measure after any change.
37 categorize tests green; lint + guards clean.

Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

DECISIONS.md

@@ -1149,3 +1149,4 @@ One line per decision: `[YYYY-MM-DD] <decision>: <why>`. Appended by agents and
 [2026-08-21] RIP-4 cascade step 3 (UI): the AI booking proposal is surfaced INSIDE the existing QuickReviewDialog rather than a new inline-row card, so it reuses that dialog's proven, deterministic, balanced commit path (POST /api/transactions/[id]/categorize) instead of a parallel one. components/transactions/AiCategorizeProposal.tsx fetches POST /api/agent/categorize on dialog open (keyed on tx.id so it remounts per transaction), pre-fills accountOverride + vatTreatment via handleAccountChange (class-2 VAT clearing preserved), and shows the confidence band (säker/trolig/välj konto) + "Varför" + the candidate alternatives (click to re-apply). Falls back SILENTLY to the deterministic defaults on error, and shows a soft note on 503 (ai_unconfigured) — the dialog always works without AI. NO silent auto-posting (founder call, avoids the storno-on-undo mess): "säker" = pre-filled, one-tap Bokför via the dialog's existing button; true hands-off auto-book waits for calibration. i18n: strings inline Swedish for now (assistant surface), lift to messages/{sv,en}.json before final merge. Confidence bands (0.8/0.5) are placeholders until calibration. Needs founder visual sign-off before merge ([[project_nav_ia_redesign]]).
 [2026-08-21] RIP-4 step 4 = calibration. lib/agent/categorize/calibration.ts is the engine: isotonic regression via pool-adjacent-violators (distribution-free, monotonic) over (confidence, was_correct) samples → a calibrator that turns raw selector confidence into a probability that actually means what it says; plus reliabilityByBucket/ECE and bandFor(). bandFor NEVER returns 'auto' without a fitted calibrator (no silent booking on an unproven score) and never auto-books above an amount cap (default 2000 kr) — so "säker" stays honest until proven. Measurement loop: migration 20260821100000 categorize_calibration_samples (append-only, company-scoped RLS, confidence CHECK [0,1]); POST /api/agent/categorize/outcome logs one sample (proposed vs booked account → was_correct) fire-and-forget from QuickReviewDialog on a successful book (sandbox skipped to keep the corpus clean); AiCategorizeProposal surfaces the proposal metadata via onProposal. scripts/fit-categorize-calibration.ts (READ-ONLY) prints the reliability diagram + ECE + fitted calibrator once data exists — run it in a few weeks, then store the calibrator/thresholds where bandFor reads them and only THEN consider enabling auto-book. Fitting needs >=200 real samples so nothing calibrates today; the loop just starts collecting. Migration applies on merge (auto-apply-on-merge active) — not applied manually.
 [2026-08-21] The categorize selector now reads the underlag, not just the bank line — the highest-leverage quality lever for the cold-start majority (prod: 365 companies with 32.7k unbooked tx, median 0 templates, so the LLM carries them). lib/agent/categorize/underlag.ts gathers the matched receipt/invoice text (receipts.matched_transaction_id + invoice_inbox_items.matched_transaction_id + the transaction's own document_attachments), rendered as bounded Swedish text (supplier, date, total, moms, line items). Core reads these tables directly via supabase (table names, not @/extensions imports — the tables live in the shared DB). POST /api/agent/categorize gathers it server-side when the caller didn't pass `underlag`, so the model sees the actual supplier + line items. Best-effort ('' on any failure); server-side only, no client change (so no conflict with the calibration PR #1784 which also touches the dialog). Prod read (project pwxtzglxptnnvjrpixpg) also confirmed: 3342 active counterparty templates / 26k occurrences → established users get strong instant candidates; NO backfill needed (templates already reflect historical bookings).
+[2026-08-21] Confidence honesty fix, driven by a real backtest (scripts/backtest-categorize.ts, read-only: runs the real cascade on already-booked prod transactions and scores the model's pick vs the human's actual account). Backtest exposed the selector reporting 0.95 on pure category guesses → "säker" was a lie (high-conf picks only 52% accurate). Fix: confidence is now driven by DETERMINISTIC BACKING (the confidence of a candidate that independently points at the chosen account), not the model's verbalized confidence (which the backtest showed is ~always "high"). A backed pick takes the candidate confidence, reduced only when the model is unsure (BACKED_MODEL_FACTOR); an UNBACKED pick (category guess no candidate agreed with) is capped at 0.7 — below the säker band (0.8) — so a guess is never "säker". Re-backtest: säker accuracy 52% → 73%, and far fewer picks claim säker (only template-backed ones). Still not auto-book-grade (~73%, want ~95%); auto-book stays off until isotonic calibration on real approvals. Backtest caveats: exact-account match is strict (penalizes reasonable-but-different picks + companies' idiosyncratic charts), sample is established users (cold-start majority has no ground truth yet), backtest ran samples=1 (no self-consistency). Some confident-wrong cases are POISONED templates (a past mis-booking → wrong candidate the model correctly follows), a data-quality issue not fixable in the confidence math.

lib/agent/categorize/__tests__/select-account.test.ts

@@ -121,23 +121,37 @@ describe('selectAccount', () => {
       expect(generateStructured).toHaveBeenCalledTimes(1)
     })
 
-    it('lower agreement lowers the combined confidence', async () => {
+    it('lower agreement lowers an unbacked (category-guess) confidence', async () => {
       generateStructured
         .mockResolvedValueOnce(pick('cat:expense_office', { confidence: 'high' }))
         .mockResolvedValueOnce(pick('cat:expense_office', { confidence: 'high' }))
         .mockResolvedValueOnce(pick('cat:expense_travel', { confidence: 'high' }))
       const split = await selectAccount(input({ candidates: [], samples: 3 }))
-      // 2/3 agreement × 0.95 model weight
-      expect(split.confidence).toBeCloseTo(0.63, 1)
+      // No candidate backs the pick → unbacked: 2/3 agreement × 0.7 (high) ≈ 0.47.
+      expect(split.confidence).toBeCloseTo(0.47, 1)
       expect(split.agreement).toBe(0.67)
     })
 
-    it('a high-confidence candidate floors the score at its deterministic confidence × agreement', async () => {
-      // model says "low", but the chosen candidate is a 0.9 counterparty template
+    it('an unbacked category guess never reaches the säker band, however sure the model is', async () => {
+      generateStructured.mockResolvedValue(pick('cat:expense_software', { confidence: 'high' }))
+      const res = await selectAccount(input({ candidates: [], samples: 1 }))
+      // high model conf, full agreement, but no deterministic backing → capped at 0.7 (< 0.8).
+      expect(res.confidence).toBeLessThan(0.8)
+      expect(res.confidence).toBeCloseTo(0.7, 2)
+    })
+
+    it('a backed pick takes the candidate confidence, only reduced when the model is unsure', async () => {
+      // model says "low", but the chosen candidate is a 0.9 counterparty template.
       generateStructured.mockResolvedValue(pick('cand:0', { confidence: 'low' }))
       const res = await selectAccount(input({ samples: 1 }))
-      // low weight 0.5 vs candidate 0.9 × agreement 1 → floored to 0.9
-      expect(res.confidence).toBe(0.9)
+      // backing 0.9 × agreement 1 × low-factor 0.75 = 0.675.
+      expect(res.confidence).toBeCloseTo(0.68, 2)
+    })
+
+    it('a backed pick the model is sure about keeps the full candidate confidence', async () => {
+      generateStructured.mockResolvedValue(pick('cand:0', { confidence: 'high' }))
+      const res = await selectAccount(input({ samples: 1 }))
+      expect(res.confidence).toBe(0.9) // 0.9 × 1 × 1
     })
   })
 

lib/agent/categorize/select-account.ts

@@ -256,10 +256,22 @@ function parsePick(value: unknown, validIds: Set<string>): RawPick {
   }
 }
 
-const MODEL_CONF_WEIGHT: Record<'high' | 'medium' | 'low', number> = {
-  high: 0.95,
-  medium: 0.75,
-  low: 0.5,
+// A backtest against real bookings showed the model reports "high" almost
+// always, so its verbalized confidence can't drive the score. Instead:
+// - a pick BACKED by a deterministic candidate takes that candidate's
+//   confidence (the retrieval signal), only pulled down when the model itself
+//   is unsure;
+// - an UNBACKED pick (a pure category guess no candidate agreed with) is capped
+//   below the "säker" threshold, however sure the model claims to be.
+const BACKED_MODEL_FACTOR: Record<'high' | 'medium' | 'low', number> = {
+  high: 1,
+  medium: 0.9,
+  low: 0.75,
+}
+const UNBACKED_CONFIDENCE: Record<'high' | 'medium' | 'low', number> = {
+  high: 0.7, // stays under the säker band (0.8): a guess is never "säker"
+  medium: 0.5,
+  low: 0.3,
 }
 
 /**
@@ -313,13 +325,11 @@ export async function selectAccount(input: SelectAccountInput): Promise<AccountS
   let category: TransactionCategory | null = null
   let vatTreatment: VatTreatment | null = null
   let fromCandidate = false
-  let candidateConfidence = 0
 
   if (choice.kind === 'candidate') {
     const cand = input.candidates.find((c) => c.account === choice.account)
     account = choice.account
     fromCandidate = true
-    candidateConfidence = cand?.confidence ?? 0
     vatTreatment = cand?.vatTreatment ?? null
   } else if (choice.kind === 'category') {
     category = choice.category
@@ -332,12 +342,25 @@ export async function selectAccount(input: SelectAccountInput): Promise<AccountS
   const reverseCharge = winningSample.reverseCharge && choice.kind !== 'needs_review'
   if (reverseCharge && input.vatRegistered) vatTreatment = 'reverse_charge'
 
-  // Combined confidence: model conf × agreement, floored by the deterministic
-  // candidate signal when a known candidate won. needs_review is always 0.
+  // Confidence is driven by DETERMINISTIC BACKING, not the model's self-report.
+  // "backing" = the confidence of a candidate that independently points at the
+  // chosen account (a candidate pick backs itself; a category pick is backed
+  // only if a candidate resolved to the same account). needs_review is 0.
+  const backing =
+    choice.kind === 'needs_review'
+      ? 0
+      : (input.candidates.find((c) => c.account === account)?.confidence ?? 0)
+
   let confidence = 0
   if (choice.kind !== 'needs_review') {
-    confidence = MODEL_CONF_WEIGHT[winningSample.confidence] * agreement
-    if (fromCandidate) confidence = Math.max(confidence, candidateConfidence * agreement)
+    confidence =
+      backing > 0
+        ? // Memory backs the pick → the retrieval signal, tempered by self-
+          // consistency; the model's own confidence only pulls it down when low.
+          agreement * backing * BACKED_MODEL_FACTOR[winningSample.confidence]
+        : // Pure model guess → capped below "säker"; verbalized confidence is
+          // not trusted to lift an unbacked pick past a suggestion.
+          agreement * UNBACKED_CONFIDENCE[winningSample.confidence]
     confidence = roundOre(Math.min(1, confidence))
   }
 

scripts/backtest-categorize.ts

@@ -0,0 +1,160 @@
+/**
+ * Backtest the auto-booking cascade against REAL, already-booked transactions.
+ *
+ * READ-ONLY. For each recent booked expense transaction it: reconstructs the
+ * candidate slate + underlag from prod, runs the real selector (against the
+ * configured AI backend), and compares the model's proposed account to the
+ * account the human actually booked (the expense debit line). Prints per-row
+ * detail + an aggregate: overall accuracy, and — the honest signal — accuracy
+ * on the cases where the top deterministic candidate was NOT the answer, i.e.
+ * where the model had to add value.
+ *
+ *   cp ~/erp-base/.env.local .   # prod DB + Bedrock, read-only
+ *   npx tsx scripts/backtest-categorize.ts [N]
+ *   rm .env.local
+ *
+ * Leakage caveat: a known vendor's counterparty template may already reflect
+ * the very booking under test, inflating the "deterministic nailed it" segment.
+ * The "model had to decide" segment below is the leakage-free measure.
+ */
+import { config } from 'dotenv'
+config({ path: '.env.local' })
+
+const N = Number(process.argv[2] ?? 50)
+const CONCURRENCY = 4
+
+async function main() {
+  const { createClient } = await import('@supabase/supabase-js')
+  // Import after dotenv so lib/ai resolves the provider/model from .env.local.
+  const { gatherCandidates } = await import('../lib/agent/categorize/candidates')
+  const { gatherUnderlag } = await import('../lib/agent/categorize/underlag')
+  const { selectAccount } = await import('../lib/agent/categorize/select-account')
+
+  const url = process.env.NEXT_PUBLIC_SUPABASE_URL!
+  const key = process.env.SUPABASE_SERVICE_ROLE_KEY!
+  const supabase = createClient(url, key)
+
+  // Recent booked expense transactions with a counterparty.
+  const { data: txs, error } = await supabase
+    .from('transactions')
+    .select('id, company_id, merchant_name, description, original_description, amount, date, currency, document_id, journal_entry_id')
+    .not('journal_entry_id', 'is', null)
+    .lt('amount', 0)
+    .eq('is_business', true)
+    .not('merchant_name', 'is', null)
+    .order('created_at', { ascending: false })
+    .limit(N)
+  if (error) throw error
+  const rows = txs ?? []
+  console.log(`\nBacktesting ${rows.length} booked transactions on ${process.env.BEDROCK_MODEL_ID ?? process.env.AI_MODEL ?? 'the configured model'}…\n`)
+
+  // Ground-truth debit account per journal entry (expense line, not cash/VAT).
+  const jeIds = rows.map((r) => r.journal_entry_id).filter(Boolean) as string[]
+  const truth = new Map<string, string>()
+  for (let i = 0; i < jeIds.length; i += 100) {
+    const { data: lines } = await supabase
+      .from('journal_entry_lines')
+      .select('journal_entry_id, account_number, debit_amount')
+      .in('journal_entry_id', jeIds.slice(i, i + 100))
+    for (const l of (lines ?? []) as { journal_entry_id: string; account_number: string; debit_amount: number | null }[]) {
+      const acct = l.account_number ?? ''
+      if (!(Number(l.debit_amount) > 0)) continue
+      if (acct.startsWith('19') || acct.startsWith('26') || acct.startsWith('264')) continue // cash + VAT
+      const cur = truth.get(l.journal_entry_id)
+      if (!cur) truth.set(l.journal_entry_id, acct) // first expense debit line
+    }
+  }
+
+  const companyCtx = new Map<string, { entityType: string; vatRegistered: boolean }>()
+  async function ctxFor(companyId: string) {
+    const hit = companyCtx.get(companyId)
+    if (hit) return hit
+    const [{ data: c }, { data: s }] = await Promise.all([
+      supabase.from('companies').select('entity_type').eq('id', companyId).maybeSingle(),
+      supabase.from('company_settings').select('vat_registered').eq('company_id', companyId).maybeSingle(),
+    ])
+    const ctx = { entityType: (c?.entity_type as string) ?? 'enskild_firma', vatRegistered: !!s?.vat_registered }
+    companyCtx.set(companyId, ctx)
+    return ctx
+  }
+
+  interface Result {
+    merchant: string
+    truth: string | null
+    proposed: string | null
+    conf: number
+    fromCandidate: boolean
+    topCandidate: string | null
+    hadUnderlag: boolean
+    correct: boolean | null
+  }
+  const results: Result[] = []
+
+  async function run(r: (typeof rows)[number]) {
+    const gt = r.journal_entry_id ? truth.get(r.journal_entry_id) ?? null : null
+    if (!gt) return
+    const ctx = await ctxFor(r.company_id)
+    const [candidates, underlag] = await Promise.all([
+      gatherCandidates(supabase as never, r.company_id, r as never),
+      gatherUnderlag(supabase as never, r.company_id, r.id, r.document_id),
+    ])
+    const sel = await selectAccount({
+      transaction: {
+        merchantName: r.merchant_name,
+        description: r.description,
+        amount: r.amount,
+        date: r.date,
+        currency: r.currency,
+      },
+      underlag,
+      candidates,
+      entityType: ctx.entityType as never,
+      vatRegistered: ctx.vatRegistered,
+      samples: 1,
+    })
+    results.push({
+      merchant: (r.merchant_name ?? '').slice(0, 22),
+      truth: gt,
+      proposed: sel.account,
+      conf: sel.confidence,
+      fromCandidate: sel.fromCandidate,
+      topCandidate: candidates[0]?.account ?? null,
+      hadUnderlag: underlag.length > 0,
+      correct: sel.account ? sel.account === gt : null,
+    })
+  }
+
+  for (let i = 0; i < rows.length; i += CONCURRENCY) {
+    await Promise.all(rows.slice(i, i + CONCURRENCY).map((r) => run(r).catch((e) => console.error('row failed', e?.message))))
+    process.stdout.write('.')
+  }
+  console.log('\n')
+
+  // Per-row.
+  for (const r of results) {
+    const mark = r.correct === null ? '·' : r.correct ? '✓' : '✗'
+    console.log(
+      `${mark} ${r.merchant.padEnd(22)} truth=${(r.truth ?? '—').padEnd(6)} pick=${(r.proposed ?? 'review').padEnd(6)} ` +
+        `conf=${r.conf.toFixed(2)} ${r.fromCandidate ? 'cand' : 'cat '} ${r.hadUnderlag ? 'underlag' : '        '} topcand=${r.topCandidate ?? '—'}`,
+    )
+  }
+
+  const scored = results.filter((r) => r.correct !== null)
+  const acc = (xs: Result[]) => (xs.length ? (xs.filter((r) => r.correct).length / xs.length) : 0)
+  const detWrong = scored.filter((r) => r.topCandidate !== r.truth) // deterministic top candidate was NOT the answer
+  const withU = scored.filter((r) => r.hadUnderlag)
+
+  console.log('\n──────── summary ────────')
+  console.log(`scored:                 ${scored.length} / ${results.length} (rest = needs_review)`)
+  console.log(`overall accuracy:       ${(acc(scored) * 100).toFixed(1)}%`)
+  console.log(`  model-decided (top candidate ≠ truth): ${(acc(detWrong) * 100).toFixed(1)}%  (n=${detWrong.length})  ← leakage-free`)
+  console.log(`  with underlag:        ${(acc(withU) * 100).toFixed(1)}%  (n=${withU.length})`)
+  console.log(`needs_review rate:      ${(((results.length - scored.length) / Math.max(1, results.length)) * 100).toFixed(1)}%`)
+  console.log(`reliability (conf ≥0.8): ${(acc(scored.filter((r) => r.conf >= 0.8)) * 100).toFixed(1)}%  (n=${scored.filter((r) => r.conf >= 0.8).length})`)
+  console.log(`reliability (conf <0.5): ${(acc(scored.filter((r) => r.conf < 0.5)) * 100).toFixed(1)}%  (n=${scored.filter((r) => r.conf < 0.5).length})`)
+}
+
+main().catch((e) => {
+  console.error(e)
+  process.exit(1)
+})