diff --git a/DECISIONS.md b/DECISIONS.md index c3b40135..e8484667 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -1149,3 +1149,4 @@ One line per decision: `[YYYY-MM-DD] : `. Appended by agents and [2026-08-21] RIP-4 cascade step 3 (UI): the AI booking proposal is surfaced INSIDE the existing QuickReviewDialog rather than a new inline-row card, so it reuses that dialog's proven, deterministic, balanced commit path (POST /api/transactions/[id]/categorize) instead of a parallel one. components/transactions/AiCategorizeProposal.tsx fetches POST /api/agent/categorize on dialog open (keyed on tx.id so it remounts per transaction), pre-fills accountOverride + vatTreatment via handleAccountChange (class-2 VAT clearing preserved), and shows the confidence band (säker/trolig/välj konto) + "Varför" + the candidate alternatives (click to re-apply). Falls back SILENTLY to the deterministic defaults on error, and shows a soft note on 503 (ai_unconfigured) — the dialog always works without AI. NO silent auto-posting (founder call, avoids the storno-on-undo mess): "säker" = pre-filled, one-tap Bokför via the dialog's existing button; true hands-off auto-book waits for calibration. i18n: strings inline Swedish for now (assistant surface), lift to messages/{sv,en}.json before final merge. Confidence bands (0.8/0.5) are placeholders until calibration. Needs founder visual sign-off before merge ([[project_nav_ia_redesign]]). [2026-08-21] RIP-4 step 4 = calibration. lib/agent/categorize/calibration.ts is the engine: isotonic regression via pool-adjacent-violators (distribution-free, monotonic) over (confidence, was_correct) samples → a calibrator that turns raw selector confidence into a probability that actually means what it says; plus reliabilityByBucket/ECE and bandFor(). bandFor NEVER returns 'auto' without a fitted calibrator (no silent booking on an unproven score) and never auto-books above an amount cap (default 2000 kr) — so "säker" stays honest until proven. Measurement loop: migration 20260821100000 categorize_calibration_samples (append-only, company-scoped RLS, confidence CHECK [0,1]); POST /api/agent/categorize/outcome logs one sample (proposed vs booked account → was_correct) fire-and-forget from QuickReviewDialog on a successful book (sandbox skipped to keep the corpus clean); AiCategorizeProposal surfaces the proposal metadata via onProposal. scripts/fit-categorize-calibration.ts (READ-ONLY) prints the reliability diagram + ECE + fitted calibrator once data exists — run it in a few weeks, then store the calibrator/thresholds where bandFor reads them and only THEN consider enabling auto-book. Fitting needs >=200 real samples so nothing calibrates today; the loop just starts collecting. Migration applies on merge (auto-apply-on-merge active) — not applied manually. [2026-08-21] The categorize selector now reads the underlag, not just the bank line — the highest-leverage quality lever for the cold-start majority (prod: 365 companies with 32.7k unbooked tx, median 0 templates, so the LLM carries them). lib/agent/categorize/underlag.ts gathers the matched receipt/invoice text (receipts.matched_transaction_id + invoice_inbox_items.matched_transaction_id + the transaction's own document_attachments), rendered as bounded Swedish text (supplier, date, total, moms, line items). Core reads these tables directly via supabase (table names, not @/extensions imports — the tables live in the shared DB). POST /api/agent/categorize gathers it server-side when the caller didn't pass `underlag`, so the model sees the actual supplier + line items. Best-effort ('' on any failure); server-side only, no client change (so no conflict with the calibration PR #1784 which also touches the dialog). Prod read (project pwxtzglxptnnvjrpixpg) also confirmed: 3342 active counterparty templates / 26k occurrences → established users get strong instant candidates; NO backfill needed (templates already reflect historical bookings). +[2026-08-21] Confidence honesty fix, driven by a real backtest (scripts/backtest-categorize.ts, read-only: runs the real cascade on already-booked prod transactions and scores the model's pick vs the human's actual account). Backtest exposed the selector reporting 0.95 on pure category guesses → "säker" was a lie (high-conf picks only 52% accurate). Fix: confidence is now driven by DETERMINISTIC BACKING (the confidence of a candidate that independently points at the chosen account), not the model's verbalized confidence (which the backtest showed is ~always "high"). A backed pick takes the candidate confidence, reduced only when the model is unsure (BACKED_MODEL_FACTOR); an UNBACKED pick (category guess no candidate agreed with) is capped at 0.7 — below the säker band (0.8) — so a guess is never "säker". Re-backtest: säker accuracy 52% → 73%, and far fewer picks claim säker (only template-backed ones). Still not auto-book-grade (~73%, want ~95%); auto-book stays off until isotonic calibration on real approvals. Backtest caveats: exact-account match is strict (penalizes reasonable-but-different picks + companies' idiosyncratic charts), sample is established users (cold-start majority has no ground truth yet), backtest ran samples=1 (no self-consistency). Some confident-wrong cases are POISONED templates (a past mis-booking → wrong candidate the model correctly follows), a data-quality issue not fixable in the confidence math. diff --git a/lib/agent/categorize/__tests__/select-account.test.ts b/lib/agent/categorize/__tests__/select-account.test.ts index 9bce8789..c339f6b3 100644 --- a/lib/agent/categorize/__tests__/select-account.test.ts +++ b/lib/agent/categorize/__tests__/select-account.test.ts @@ -121,23 +121,37 @@ describe('selectAccount', () => { expect(generateStructured).toHaveBeenCalledTimes(1) }) - it('lower agreement lowers the combined confidence', async () => { + it('lower agreement lowers an unbacked (category-guess) confidence', async () => { generateStructured .mockResolvedValueOnce(pick('cat:expense_office', { confidence: 'high' })) .mockResolvedValueOnce(pick('cat:expense_office', { confidence: 'high' })) .mockResolvedValueOnce(pick('cat:expense_travel', { confidence: 'high' })) const split = await selectAccount(input({ candidates: [], samples: 3 })) - // 2/3 agreement × 0.95 model weight - expect(split.confidence).toBeCloseTo(0.63, 1) + // No candidate backs the pick → unbacked: 2/3 agreement × 0.7 (high) ≈ 0.47. + expect(split.confidence).toBeCloseTo(0.47, 1) expect(split.agreement).toBe(0.67) }) - it('a high-confidence candidate floors the score at its deterministic confidence × agreement', async () => { - // model says "low", but the chosen candidate is a 0.9 counterparty template + it('an unbacked category guess never reaches the säker band, however sure the model is', async () => { + generateStructured.mockResolvedValue(pick('cat:expense_software', { confidence: 'high' })) + const res = await selectAccount(input({ candidates: [], samples: 1 })) + // high model conf, full agreement, but no deterministic backing → capped at 0.7 (< 0.8). + expect(res.confidence).toBeLessThan(0.8) + expect(res.confidence).toBeCloseTo(0.7, 2) + }) + + it('a backed pick takes the candidate confidence, only reduced when the model is unsure', async () => { + // model says "low", but the chosen candidate is a 0.9 counterparty template. generateStructured.mockResolvedValue(pick('cand:0', { confidence: 'low' })) const res = await selectAccount(input({ samples: 1 })) - // low weight 0.5 vs candidate 0.9 × agreement 1 → floored to 0.9 - expect(res.confidence).toBe(0.9) + // backing 0.9 × agreement 1 × low-factor 0.75 = 0.675. + expect(res.confidence).toBeCloseTo(0.68, 2) + }) + + it('a backed pick the model is sure about keeps the full candidate confidence', async () => { + generateStructured.mockResolvedValue(pick('cand:0', { confidence: 'high' })) + const res = await selectAccount(input({ samples: 1 })) + expect(res.confidence).toBe(0.9) // 0.9 × 1 × 1 }) }) diff --git a/lib/agent/categorize/select-account.ts b/lib/agent/categorize/select-account.ts index cbe85b05..a2fdf462 100644 --- a/lib/agent/categorize/select-account.ts +++ b/lib/agent/categorize/select-account.ts @@ -256,10 +256,22 @@ function parsePick(value: unknown, validIds: Set): RawPick { } } -const MODEL_CONF_WEIGHT: Record<'high' | 'medium' | 'low', number> = { - high: 0.95, - medium: 0.75, - low: 0.5, +// A backtest against real bookings showed the model reports "high" almost +// always, so its verbalized confidence can't drive the score. Instead: +// - a pick BACKED by a deterministic candidate takes that candidate's +// confidence (the retrieval signal), only pulled down when the model itself +// is unsure; +// - an UNBACKED pick (a pure category guess no candidate agreed with) is capped +// below the "säker" threshold, however sure the model claims to be. +const BACKED_MODEL_FACTOR: Record<'high' | 'medium' | 'low', number> = { + high: 1, + medium: 0.9, + low: 0.75, +} +const UNBACKED_CONFIDENCE: Record<'high' | 'medium' | 'low', number> = { + high: 0.7, // stays under the säker band (0.8): a guess is never "säker" + medium: 0.5, + low: 0.3, } /** @@ -313,13 +325,11 @@ export async function selectAccount(input: SelectAccountInput): Promise c.account === choice.account) account = choice.account fromCandidate = true - candidateConfidence = cand?.confidence ?? 0 vatTreatment = cand?.vatTreatment ?? null } else if (choice.kind === 'category') { category = choice.category @@ -332,12 +342,25 @@ export async function selectAccount(input: SelectAccountInput): Promise c.account === account)?.confidence ?? 0) + let confidence = 0 if (choice.kind !== 'needs_review') { - confidence = MODEL_CONF_WEIGHT[winningSample.confidence] * agreement - if (fromCandidate) confidence = Math.max(confidence, candidateConfidence * agreement) + confidence = + backing > 0 + ? // Memory backs the pick → the retrieval signal, tempered by self- + // consistency; the model's own confidence only pulls it down when low. + agreement * backing * BACKED_MODEL_FACTOR[winningSample.confidence] + : // Pure model guess → capped below "säker"; verbalized confidence is + // not trusted to lift an unbacked pick past a suggestion. + agreement * UNBACKED_CONFIDENCE[winningSample.confidence] confidence = roundOre(Math.min(1, confidence)) } diff --git a/scripts/backtest-categorize.ts b/scripts/backtest-categorize.ts new file mode 100644 index 00000000..4af9a1ad --- /dev/null +++ b/scripts/backtest-categorize.ts @@ -0,0 +1,160 @@ +/** + * Backtest the auto-booking cascade against REAL, already-booked transactions. + * + * READ-ONLY. For each recent booked expense transaction it: reconstructs the + * candidate slate + underlag from prod, runs the real selector (against the + * configured AI backend), and compares the model's proposed account to the + * account the human actually booked (the expense debit line). Prints per-row + * detail + an aggregate: overall accuracy, and — the honest signal — accuracy + * on the cases where the top deterministic candidate was NOT the answer, i.e. + * where the model had to add value. + * + * cp ~/erp-base/.env.local . # prod DB + Bedrock, read-only + * npx tsx scripts/backtest-categorize.ts [N] + * rm .env.local + * + * Leakage caveat: a known vendor's counterparty template may already reflect + * the very booking under test, inflating the "deterministic nailed it" segment. + * The "model had to decide" segment below is the leakage-free measure. + */ +import { config } from 'dotenv' +config({ path: '.env.local' }) + +const N = Number(process.argv[2] ?? 50) +const CONCURRENCY = 4 + +async function main() { + const { createClient } = await import('@supabase/supabase-js') + // Import after dotenv so lib/ai resolves the provider/model from .env.local. + const { gatherCandidates } = await import('../lib/agent/categorize/candidates') + const { gatherUnderlag } = await import('../lib/agent/categorize/underlag') + const { selectAccount } = await import('../lib/agent/categorize/select-account') + + const url = process.env.NEXT_PUBLIC_SUPABASE_URL! + const key = process.env.SUPABASE_SERVICE_ROLE_KEY! + const supabase = createClient(url, key) + + // Recent booked expense transactions with a counterparty. + const { data: txs, error } = await supabase + .from('transactions') + .select('id, company_id, merchant_name, description, original_description, amount, date, currency, document_id, journal_entry_id') + .not('journal_entry_id', 'is', null) + .lt('amount', 0) + .eq('is_business', true) + .not('merchant_name', 'is', null) + .order('created_at', { ascending: false }) + .limit(N) + if (error) throw error + const rows = txs ?? [] + console.log(`\nBacktesting ${rows.length} booked transactions on ${process.env.BEDROCK_MODEL_ID ?? process.env.AI_MODEL ?? 'the configured model'}…\n`) + + // Ground-truth debit account per journal entry (expense line, not cash/VAT). + const jeIds = rows.map((r) => r.journal_entry_id).filter(Boolean) as string[] + const truth = new Map() + for (let i = 0; i < jeIds.length; i += 100) { + const { data: lines } = await supabase + .from('journal_entry_lines') + .select('journal_entry_id, account_number, debit_amount') + .in('journal_entry_id', jeIds.slice(i, i + 100)) + for (const l of (lines ?? []) as { journal_entry_id: string; account_number: string; debit_amount: number | null }[]) { + const acct = l.account_number ?? '' + if (!(Number(l.debit_amount) > 0)) continue + if (acct.startsWith('19') || acct.startsWith('26') || acct.startsWith('264')) continue // cash + VAT + const cur = truth.get(l.journal_entry_id) + if (!cur) truth.set(l.journal_entry_id, acct) // first expense debit line + } + } + + const companyCtx = new Map() + async function ctxFor(companyId: string) { + const hit = companyCtx.get(companyId) + if (hit) return hit + const [{ data: c }, { data: s }] = await Promise.all([ + supabase.from('companies').select('entity_type').eq('id', companyId).maybeSingle(), + supabase.from('company_settings').select('vat_registered').eq('company_id', companyId).maybeSingle(), + ]) + const ctx = { entityType: (c?.entity_type as string) ?? 'enskild_firma', vatRegistered: !!s?.vat_registered } + companyCtx.set(companyId, ctx) + return ctx + } + + interface Result { + merchant: string + truth: string | null + proposed: string | null + conf: number + fromCandidate: boolean + topCandidate: string | null + hadUnderlag: boolean + correct: boolean | null + } + const results: Result[] = [] + + async function run(r: (typeof rows)[number]) { + const gt = r.journal_entry_id ? truth.get(r.journal_entry_id) ?? null : null + if (!gt) return + const ctx = await ctxFor(r.company_id) + const [candidates, underlag] = await Promise.all([ + gatherCandidates(supabase as never, r.company_id, r as never), + gatherUnderlag(supabase as never, r.company_id, r.id, r.document_id), + ]) + const sel = await selectAccount({ + transaction: { + merchantName: r.merchant_name, + description: r.description, + amount: r.amount, + date: r.date, + currency: r.currency, + }, + underlag, + candidates, + entityType: ctx.entityType as never, + vatRegistered: ctx.vatRegistered, + samples: 1, + }) + results.push({ + merchant: (r.merchant_name ?? '').slice(0, 22), + truth: gt, + proposed: sel.account, + conf: sel.confidence, + fromCandidate: sel.fromCandidate, + topCandidate: candidates[0]?.account ?? null, + hadUnderlag: underlag.length > 0, + correct: sel.account ? sel.account === gt : null, + }) + } + + for (let i = 0; i < rows.length; i += CONCURRENCY) { + await Promise.all(rows.slice(i, i + CONCURRENCY).map((r) => run(r).catch((e) => console.error('row failed', e?.message)))) + process.stdout.write('.') + } + console.log('\n') + + // Per-row. + for (const r of results) { + const mark = r.correct === null ? '·' : r.correct ? '✓' : '✗' + console.log( + `${mark} ${r.merchant.padEnd(22)} truth=${(r.truth ?? '—').padEnd(6)} pick=${(r.proposed ?? 'review').padEnd(6)} ` + + `conf=${r.conf.toFixed(2)} ${r.fromCandidate ? 'cand' : 'cat '} ${r.hadUnderlag ? 'underlag' : ' '} topcand=${r.topCandidate ?? '—'}`, + ) + } + + const scored = results.filter((r) => r.correct !== null) + const acc = (xs: Result[]) => (xs.length ? (xs.filter((r) => r.correct).length / xs.length) : 0) + const detWrong = scored.filter((r) => r.topCandidate !== r.truth) // deterministic top candidate was NOT the answer + const withU = scored.filter((r) => r.hadUnderlag) + + console.log('\n──────── summary ────────') + console.log(`scored: ${scored.length} / ${results.length} (rest = needs_review)`) + console.log(`overall accuracy: ${(acc(scored) * 100).toFixed(1)}%`) + console.log(` model-decided (top candidate ≠ truth): ${(acc(detWrong) * 100).toFixed(1)}% (n=${detWrong.length}) ← leakage-free`) + console.log(` with underlag: ${(acc(withU) * 100).toFixed(1)}% (n=${withU.length})`) + console.log(`needs_review rate: ${(((results.length - scored.length) / Math.max(1, results.length)) * 100).toFixed(1)}%`) + console.log(`reliability (conf ≥0.8): ${(acc(scored.filter((r) => r.conf >= 0.8)) * 100).toFixed(1)}% (n=${scored.filter((r) => r.conf >= 0.8).length})`) + console.log(`reliability (conf <0.5): ${(acc(scored.filter((r) => r.conf < 0.5)) * 100).toFixed(1)}% (n=${scored.filter((r) => r.conf < 0.5).length})`) +} + +main().catch((e) => { + console.error(e) + process.exit(1) +})