feat(inbox): match non-invoice documents via prominent amounts (#2048)
* feat(inbox): match non-invoice documents via prominent amounts Bankintyg, bank agreements and other documentKind "other" PDFs carry no invoice-style total, so extraction correctly left totals.total null and the document became structurally unmatchable: findUnderlagCandidates hard-drops items without a comparable amount and the picker lost the 40% amount signal. - extraction: new prominentAmounts[] field (amount + document's own label), populated only when totals.total is null; account/org/phone/reference numbers and zero amounts excluded. totals.total semantics untouched. - matching: bestProminentAmountVariance() tries each printed amount and feeds calculateMatchConfidence at reduced weight (0.3 vs 0.4) in both the agent candidate scorer and TransactionMatchPicker. - UI: inbox rail shows the detected amounts read-only for such documents, list falls back to a single distinct prominent amount, and extraction no longer reads as "found nothing". Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hqm9QgdyNAFaWiz6Ww7pgb * fix(inbox): discount prominent-amount fallback instead of reweighting it Skeptic pass refutations on the first commit: normalized weighting made a reduced amount weight self-defeating. Date + exact fallback amount with no merchant scored (0.25+0.3)/0.55 = 1.0 ("100% sakerhet" on a wrong same-day transaction), and a DISAGREEING fallback amount scored above a disagreeing invoice total (0.67 vs 0.60) because shrinking the weight also shrank the penalty. - score fallbacks at full amount weight, then multiply by a flat FALLBACK_CONFIDENCE_FACTOR (0.85): agreement caps below certainty, disagreement stays at least as damning as for a real total. - agent candidate surface additionally requires the document date within DATE_TOLERANCE_DAYS, so an avtal listing 349 kr no longer matches every future 349 kr charge from the same counterparty. - bestProminentAmountVariance returns which amount matched + its document label, and the match reason names it ("Exakt belopp i dokumentet: 2 500 SEK (Engangspris)"): no more bare "Exakt belopp" reaching the agent while total_amount is null. - prompt: prominentAmounts restricted to non-invoice documentKinds, and never a parking spot for an unreadable invoice total. - fix the stale "deliberately the same list" comment on EXTRACTED_FIELD_ACCESSORS. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hqm9QgdyNAFaWiz6Ww7pgb * fix(receipt-hunt): never propose on the prominent-amounts fallback Second skeptic pass: the nightly hunt is a third consumer of scoreUnderlagCandidates and inherited the fallback unaware. A bankintyg whose printed "Insatt belopp" equals a same-day outflow scores 0.85, which clears CERTAIN_CONFIDENCE (0.8) and skips LLM adjudication, on a pairing wrong by construction (the hunt scans outflows only; "Insatt belopp" labels an inflow), with document_amount null in the approval preview. UnderlagCandidate now carries amountSource ('total' | 'prominent') and selectProposals drops fallback-scored candidates. Non-invoice documents stay reachable through the manual picker and the agent candidate surface, both of which have a human reading the amounts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hqm9QgdyNAFaWiz6Ww7pgb * fix(inbox): round fallback confidence via roundOre, not the naive pattern The two confidence discounts (and their test) tripped the naive-ore-round antipattern ratchet (625 vs baseline 622); use the sanctioned helper. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hqm9QgdyNAFaWiz6Ww7pgb --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Mattsson
co-authored by
Claude Fable 5
Claude Fable 5
parent
84ba8323b4
commit
516e8b62ff
@@ -1,12 +1,15 @@
|
||||
import { describe, it, expect } from 'vitest'
|
||||
import {
|
||||
FALLBACK_CONFIDENCE_FACTOR,
|
||||
levenshteinDistance,
|
||||
normalizeMerchantName,
|
||||
normalizeForMatch,
|
||||
calculateMerchantSimilarity,
|
||||
calculateMatchConfidence,
|
||||
amountVarianceForMatch,
|
||||
bestProminentAmountVariance,
|
||||
} from '../core-receipt-matcher'
|
||||
import { roundOre } from '@/lib/money'
|
||||
|
||||
describe('levenshteinDistance', () => {
|
||||
it('returns 0 for identical strings', () => {
|
||||
@@ -163,6 +166,56 @@ describe('amountVarianceForMatch', () => {
|
||||
})
|
||||
})
|
||||
|
||||
describe('bestProminentAmountVariance', () => {
|
||||
it('picks the closest of several printed amounts and names it', () => {
|
||||
// An agreement listing both a monthly price and a one-off price: the
|
||||
// one-off 2500 matches the -2500 AVGIFT charge exactly.
|
||||
const best = bestProminentAmountVariance(
|
||||
[
|
||||
{ amount: 49, label: 'Månadspris' },
|
||||
{ amount: 2500, label: 'Engångspris' },
|
||||
],
|
||||
'SEK',
|
||||
-2500,
|
||||
'SEK',
|
||||
-2500,
|
||||
)
|
||||
expect(best).toEqual({ variance: 0, amount: 2500, label: 'Engångspris' })
|
||||
})
|
||||
|
||||
it('returns null when nothing is comparable', () => {
|
||||
expect(bestProminentAmountVariance([], 'SEK', -2500, 'SEK', -2500)).toBeNull()
|
||||
// Cross-currency without a rate stays incomparable, like a total would.
|
||||
expect(
|
||||
bestProminentAmountVariance([{ amount: 2500, label: null }], 'EUR', -2500, 'SEK', -2500),
|
||||
).toBeNull()
|
||||
// Zero amounts carry no signal (amountVarianceForMatch drops them).
|
||||
expect(
|
||||
bestProminentAmountVariance([{ amount: 0, label: null }], 'SEK', -2500, 'SEK', -2500),
|
||||
).toBeNull()
|
||||
})
|
||||
|
||||
it('the discount factor keeps fallback agreement below certainty', () => {
|
||||
// Exact date + exact amount + no merchant normalises to 1.0; a fallback
|
||||
// match must not present that as certainty (this exact geometry scored
|
||||
// "100% säkerhet" on a wrong same-day transaction before the factor).
|
||||
const { confidence } = calculateMatchConfidence(0, 0, 0)
|
||||
const discounted = roundOre(confidence * FALLBACK_CONFIDENCE_FACTOR)
|
||||
expect(confidence).toBe(1)
|
||||
expect(discounted).toBeLessThan(1)
|
||||
})
|
||||
|
||||
it('a disagreeing fallback amount scores no better than a disagreeing total', () => {
|
||||
// Renormalized weights made a wrong fallback amount OUTSCORE a wrong
|
||||
// invoice total (0.67 vs 0.60 with exact date + merchant); the factor
|
||||
// approach scores both at full weight and then discounts the fallback.
|
||||
const asTotal = calculateMatchConfidence(0, 1.4, 0.9).confidence
|
||||
const asFallback = roundOre(asTotal * FALLBACK_CONFIDENCE_FACTOR)
|
||||
expect(asFallback).toBeLessThanOrEqual(asTotal)
|
||||
expect(asFallback).toBeLessThan(0.6)
|
||||
})
|
||||
})
|
||||
|
||||
describe('normalizeForMatch', () => {
|
||||
it('leaves the frozen key normalizer alone', () => {
|
||||
// normalizeMerchantName feeds a PERSISTED unique key with a SQL mirror.
|
||||
|
||||
@@ -37,6 +37,24 @@ export const AMOUNT_TOLERANCE_PERCENT = 0.05
|
||||
export const CONVERTED_AMOUNT_TOLERANCE_PERCENT = 0.09
|
||||
export const MIN_MATCH_CONFIDENCE = 0.4
|
||||
|
||||
/**
|
||||
* Flat discount on a confidence scored from a prominentAmounts fallback
|
||||
* (bankintyg, avtal, contracts: no invoice-style total). Such an amount is one
|
||||
* of possibly several figures printed on the document rather than "what the
|
||||
* buyer pays", so an agreement is real evidence but must stay weaker than a
|
||||
* total agreeing.
|
||||
*
|
||||
* A discount FACTOR, deliberately not a reduced amount weight inside
|
||||
* calculateMatchConfidence: the confidence is normalised over the included
|
||||
* weights, so shrinking the amount weight both let a date+amount-only fallback
|
||||
* reach 1.0 ((0.25+0.3)/0.55) and, when the amount DISAGREED, shrank the
|
||||
* penalty so a wrong fallback amount outscored a wrong invoice total
|
||||
* (0.67 vs 0.60). Scoring at full weight and discounting the result keeps
|
||||
* agreement capped below certainty and disagreement at least as damning as it
|
||||
* is for a real total.
|
||||
*/
|
||||
export const FALLBACK_CONFIDENCE_FACTOR = 0.85
|
||||
|
||||
/**
|
||||
* Normalize a merchant name for comparison.
|
||||
* Removes special characters, Swedish company suffixes, and extra whitespace.
|
||||
@@ -263,6 +281,55 @@ export function amountVarianceForMatch(
|
||||
return null
|
||||
}
|
||||
|
||||
export interface ProminentAmountMatch {
|
||||
variance: number
|
||||
/** The printed amount that produced the variance. */
|
||||
amount: number
|
||||
/** The document's own label for it ("Insatt belopp", "Engångspris"). */
|
||||
label: string | null
|
||||
}
|
||||
|
||||
/**
|
||||
* Fallback amount variance for documents with no invoice-style total but one
|
||||
* or more prominent amounts (bankintyg "Insatt belopp", an agreement's
|
||||
* "Engångspris", ...). Tries each amount against the transaction and returns
|
||||
* the smallest variance, or null when none is comparable.
|
||||
*
|
||||
* Same-currency only by construction: prominent amounts never carry a resolved
|
||||
* SEK value, so a cross-currency pair stays incomparable (receiptSek = null in
|
||||
* amountVarianceForMatch) exactly like a cross-currency total without a rate.
|
||||
* Returns the closest amount with its variance and the document's own label.
|
||||
*
|
||||
* Callers must multiply the resulting confidence by
|
||||
* FALLBACK_CONFIDENCE_FACTOR (see its comment for why a factor, not a weight),
|
||||
* and should surface WHICH amount matched: a bare "Exakt belopp" with no
|
||||
* number attached is certainty the reader cannot check.
|
||||
*/
|
||||
export function bestProminentAmountVariance(
|
||||
amounts: readonly { amount: number; label: string | null }[],
|
||||
receiptCurrency: string,
|
||||
txAmount: number,
|
||||
txCurrency: string,
|
||||
txSek: number,
|
||||
): ProminentAmountMatch | null {
|
||||
let best: ProminentAmountMatch | null = null
|
||||
for (const candidate of amounts) {
|
||||
if (!Number.isFinite(candidate.amount)) continue
|
||||
const variance = amountVarianceForMatch(
|
||||
candidate.amount,
|
||||
receiptCurrency,
|
||||
null,
|
||||
txAmount,
|
||||
txCurrency,
|
||||
txSek,
|
||||
)
|
||||
if (variance != null && (best == null || variance < best.variance)) {
|
||||
best = { variance, amount: candidate.amount, label: candidate.label }
|
||||
}
|
||||
}
|
||||
return best
|
||||
}
|
||||
|
||||
/**
|
||||
* Calculate a weighted match confidence score from date, amount, and merchant signals.
|
||||
* Weights: amount 40%, merchant 35%, date 25%.
|
||||
|
||||
Reference in New Issue
Block a user