Add/issue in absurdum (#739)
* feat(assets): allow editing fixed asset fields before depreciation The fixed asset register only offered a "Dispose" action, so correcting a mis-entered acquisition date/cost/category meant running the disposal flow — which posts a real divestment voucher plus a Ch. 8a VAT adjustment. Disproportionate and wrong for a data-entry fix. Add an Edit action that allows correcting those fields directly, gated for correctness: - service: extend updateAsset() with category/acquisition_date/ acquisition_cost; block the change once the asset is disposed or has posted depreciation (AssetCorrectionBlockedError) where it would desync posted vouchers from the register; realign the BAS triple on category change. Name, useful life, and method stay editable. - api: extend the PATCH schema; annotate GET /api/assets with has_posted_depreciation so the UI can lock basis fields proactively. - ui: EditAssetDialog + pencil action; disables date/cost/category when depreciation has been booked, with an inline explanation. - errors: register ASSET_CORRECTION_BLOCKED (409). - tests: unit tests for the guard; pg test for pre-disposal editability. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(assets): also block basis edits when depreciation was hand-posted The correction guard only consulted depreciation_schedules, so an avskrivning booked as a manual journal entry (no schedule row) slipped through and a basis correction was wrongly allowed. Add a ledger scan: any posted credit to the asset's ackumulerade- avskrivningar account (12x9) counts as depreciation. Entries that depreciation_schedules attributes to a *different* asset are excluded, so a sibling's engine avskrivning on a shared 12x9 account doesn't produce a false block. What remains is depreciation tied to this asset (engine or manual); a basis correction is blocked there and must go through storno. Adds two unit tests: blocks on a hand-posted credit, allows when the only 12x9 credit belongs to a sibling's engine entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(invoices): allow negative unit prices for discount lines The invoice creation form rejected negative unit prices via a frontend superRefine check, blocking valid discount lines (e.g. "Rabatt -100"). The unit_price error was never rendered inline, so submission failed silently. The backend schema already allows negative unit prices (see CreateInvoiceItemSchema test), so the form was simply out of sync. Remove the non-negative constraint; empty/NaN prices are still rejected by the base z.number() type. Drop the now-unused validation_price_positive translation key from both locale files. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(invoices): allow editing draft invoices Drafts could be saved but not edited — the only way to change a draft's lines, customer, dates or amounts was to delete and recreate it. Add a "Redigera" action on draft invoices that opens the invoice editor pre-filled with the draft and saves changes in place. A verifikat is only created when an invoice is sent (or paid, under kontantmetoden), so every status=draft invoice is uncommitted and safe to edit; sent/paid invoices stay immutable and still require a credit note. - Extract buildInvoiceWriteData() with the shared validation + computation (VAT rules, ROT/RUT, accruals, totals, currency, item rows); POST now uses it too, behaviour unchanged. - Add UpdateInvoiceSchema and PATCH /api/invoices/[id], guarded to drafts (status=draft, no journal entry, not self-billed); number and status are preserved and no invoice.created is emitted. - Extract the invoice creator into a shared InvoiceEditor with create / edit modes; /invoices/new is now a thin wrapper and /invoices/[id]/edit is the new edit page. - Add a "Redigera" button on draft invoice detail pages + sv/en strings. - Tests for the builder, UpdateInvoiceSchema and the PATCH route. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(reports): make Huvudbok findable via account/saldo search terms Searching the command palette for natural phrases like 'saldo per konto', 'kontoutdrag', 'kontoanalys' or 'transaktioner per konto' returned nothing, so users couldn't find the general ledger. Enrich the Huvudbok entry's keywords with those synonyms, and let Saldobalans and Balansrapport match 'saldo per konto' too since they are genuinely per-account balance views. Companion change — the clearer Huvudbok report description ('Saldo och alla transaktioner per konto') — already landed in d5f474cb. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(settings): let users edit their personal name Add an editable Namn field to /settings/account that updates profiles.full_name and best-effort syncs auth user_metadata. Previously the personal name was only ever set from BankID's legal name at signup with no way to correct it, so users whose tilltalsnamn isn't their first given name were greeted by the wrong name (and email/password users had no name at all). New POST /api/user/profile route (requireAuth, RLS-scoped update) mirrors /api/user/locale. sv/en strings added. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(invoices): per-invoice öresavrundning override Add a display-only öresavrundning flag per invoice that wins over the company-wide setting. Resolution order in getDisplayTotal: per-invoice override -> company setting -> default-on. The stored total and the booked verifikat keep the exact öre; only the rendered total changes. Supplier invoices gain the same flag but resolve a null to off (they never had rounding historically), exposed via a toggle on the new-invoice form and a rounding row on the detail page. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(transactions): warn on possible duplicate before booking Before committing a transaction (via book or categorize), detect an already-booked sibling with the same date and amount and return a 409 TRANSACTION_BOOK_POSSIBLE_DUPLICATE instead of silently double-booking. The user can override with force=true, which must be bound to the reviewed sibling via expected_duplicate_transaction_id; the candidate is re-detected server-side, so a stale or guessed id is rejected with TRANSACTION_BOOK_FORCE_CANDIDATE_MISMATCH. Detection is fail-open on the non-force path and fail-closed under force. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(transactions): shadow-mode scope-drift dedup counter in bank ingest Count rows that an enforcing same-feed scope-drift rule WOULD treat as re-imports (the IBAN-drift re-imports the external_id check misses) and surface it as IngestResult.shadow_scope_drift_candidates. Nothing is blocked yet -- the counter only measures how often the rule would fire so it can be validated against real data before enforcement. Also gitignore scripts/delete-duplicate-transactions.ts: a destructive, hand-run cleanup tool kept out of the repo so it can't run in CI/cron or be mistaken for a supported feature. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(bokslut): base bolagsskatt on post-disposition result Bokslutsdispositioner are booked as source_type='year_end', which the income statement excludes, so net_result alone overstates resultat före skatt and the booked tax ignored the periodiseringsfond avsättning (too-high tax, ÅR/INK2 mismatch). calculateBolagsskatt now accepts resultBeforeTaxOverride. The preview builder mirrors each proposal's P&L effect (+återföring, -avsättning, -SLP) onto the pre-disposition result; the commit path sums the already-posted dispositions via the new sumPostedYearEndDispositions (class 88 + 7533) since bolagsskatt is committed last. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(settings): fiscal years manager Add a FiscalYearsManager to the bookkeeping settings that lists fiscal periods with their status (closed > locked > open) and creates the next year via CreatePeriodDialog, seeded to chain forward from the latest period end. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(api): return 400 when locking a period with unbooked transactions lockPeriod() refuses to lock a period that still has uncategorized business transactions. Detect that message in the lock route and surface it as a clear PERIOD_HAS_UNBOOKED_TRANSACTIONS (400) instead of a generic 500. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(invoices): implement isEditableInvoiceDraft utility and apply it across invoice edit routes feat(transactions): log duplicate dismissal events in behandlingshistorik test(invoices): add tests for isEditableInvoiceDraft function test(transactions): enhance tests to verify behandlingshistorik logging refactor(bokslut): update tax calculation test descriptions for clarity --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Mattsson
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
d95a0b6105
commit
8322830f46
@@ -0,0 +1,128 @@
|
||||
/**
|
||||
* Tests for the booking-time duplicate guard.
|
||||
*
|
||||
* Detection queries `transactions` for same-date already-booked siblings, then
|
||||
* filters by öre + cash-account compatibility in JS, then resolves the voucher
|
||||
* label from `journal_entries`. The mock returns the rows each query yields.
|
||||
*/
|
||||
import { describe, it, expect } from 'vitest'
|
||||
import { detectBookedDuplicateTransaction } from '../booking-duplicate-detection'
|
||||
|
||||
type TxRow = {
|
||||
id: string
|
||||
date: string
|
||||
amount: number | string
|
||||
description: string | null
|
||||
cash_account_id: string | null
|
||||
journal_entry_id: string
|
||||
}
|
||||
type JeRow = { voucher_series: string | null; voucher_number: number | null; entry_date: string | null }
|
||||
|
||||
function txChain(data: TxRow[]) {
|
||||
const c: Record<string, unknown> = {}
|
||||
c.select = () => c
|
||||
c.eq = () => c
|
||||
c.not = () => c
|
||||
c.neq = () => c
|
||||
c.limit = () => Promise.resolve({ data, error: null })
|
||||
return c
|
||||
}
|
||||
function jeChain(data: JeRow | null) {
|
||||
const c: Record<string, unknown> = {}
|
||||
c.select = () => c
|
||||
c.eq = () => c
|
||||
c.maybeSingle = () => Promise.resolve({ data, error: null })
|
||||
return c
|
||||
}
|
||||
function makeSupabase(txData: TxRow[], jeData: JeRow | null = { voucher_series: 'A', voucher_number: 142, entry_date: '2025-12-19' }) {
|
||||
return {
|
||||
from: (table: string) => (table === 'transactions' ? txChain(txData) : jeChain(jeData)),
|
||||
} as never
|
||||
}
|
||||
|
||||
const COMPANY = 'co-1'
|
||||
const sibling = (over: Partial<TxRow> = {}): TxRow => ({
|
||||
id: 'sib-1',
|
||||
date: '2025-12-19',
|
||||
amount: -1616,
|
||||
description: 'TELENOR SVERIGE AB',
|
||||
cash_account_id: null,
|
||||
journal_entry_id: 'je-1',
|
||||
...over,
|
||||
})
|
||||
|
||||
describe('detectBookedDuplicateTransaction', () => {
|
||||
it('returns null when no same-date booked sibling exists', async () => {
|
||||
const supabase = makeSupabase([])
|
||||
const result = await detectBookedDuplicateTransaction(supabase, COMPANY, {
|
||||
id: 'self', date: '2025-12-19', amount: -1616, cash_account_id: null,
|
||||
})
|
||||
expect(result).toBeNull()
|
||||
})
|
||||
|
||||
it('flags a same date+amount+account booked sibling with its voucher label', async () => {
|
||||
const supabase = makeSupabase([sibling()])
|
||||
const result = await detectBookedDuplicateTransaction(supabase, COMPANY, {
|
||||
id: 'self', date: '2025-12-19', amount: -1616, cash_account_id: null,
|
||||
})
|
||||
expect(result).toEqual({
|
||||
transaction_id: 'sib-1',
|
||||
journal_entry_id: 'je-1',
|
||||
voucher_label: 'A142',
|
||||
entry_date: '2025-12-19',
|
||||
description: 'TELENOR SVERIGE AB',
|
||||
amount: -1616,
|
||||
})
|
||||
})
|
||||
|
||||
it('does NOT flag a sibling on a different known cash account', async () => {
|
||||
const supabase = makeSupabase([sibling({ cash_account_id: 'acct-A' })])
|
||||
const result = await detectBookedDuplicateTransaction(supabase, COMPANY, {
|
||||
id: 'self', date: '2025-12-19', amount: -1616, cash_account_id: 'acct-B',
|
||||
})
|
||||
expect(result).toBeNull()
|
||||
})
|
||||
|
||||
it('flags when accounts are compatible via a null on either side', async () => {
|
||||
const supabase = makeSupabase([sibling({ cash_account_id: 'acct-A' })])
|
||||
const result = await detectBookedDuplicateTransaction(supabase, COMPANY, {
|
||||
id: 'self', date: '2025-12-19', amount: -1616, cash_account_id: null,
|
||||
})
|
||||
expect(result?.transaction_id).toBe('sib-1')
|
||||
})
|
||||
|
||||
it('does NOT flag a sibling with a different amount', async () => {
|
||||
const supabase = makeSupabase([sibling({ amount: -1000 })])
|
||||
const result = await detectBookedDuplicateTransaction(supabase, COMPANY, {
|
||||
id: 'self', date: '2025-12-19', amount: -1616, cash_account_id: null,
|
||||
})
|
||||
expect(result).toBeNull()
|
||||
})
|
||||
|
||||
it('matches a numeric-string amount from PostgREST against a JS number (öre)', async () => {
|
||||
const supabase = makeSupabase([sibling({ amount: '-1616.00' })])
|
||||
const result = await detectBookedDuplicateTransaction(supabase, COMPANY, {
|
||||
id: 'self', date: '2025-12-19', amount: -1616, cash_account_id: null,
|
||||
})
|
||||
expect(result?.transaction_id).toBe('sib-1')
|
||||
})
|
||||
|
||||
it('returns null for a zero-amount target without querying', async () => {
|
||||
const supabase = makeSupabase([sibling({ amount: 0 })])
|
||||
const result = await detectBookedDuplicateTransaction(supabase, COMPANY, {
|
||||
id: 'self', date: '2025-12-19', amount: 0, cash_account_id: null,
|
||||
})
|
||||
expect(result).toBeNull()
|
||||
})
|
||||
|
||||
it('picks the lowest-id sibling deterministically (stable under force re-detect)', async () => {
|
||||
const supabase = makeSupabase([
|
||||
sibling({ id: 'sib-9' }),
|
||||
sibling({ id: 'sib-2' }),
|
||||
])
|
||||
const result = await detectBookedDuplicateTransaction(supabase, COMPANY, {
|
||||
id: 'self', date: '2025-12-19', amount: -1616, cash_account_id: null,
|
||||
})
|
||||
expect(result?.transaction_id).toBe('sib-2')
|
||||
})
|
||||
})
|
||||
@@ -356,17 +356,19 @@ describe('ingestTransactions', () => {
|
||||
})
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// 2c. No false positive: same date+amount but different description does
|
||||
// NOT trigger content dedup — guards against the historical concern
|
||||
// about unrelated transfers colliding on (date, amount) alone.
|
||||
// 2c. No false positive within ONE channel: same date+amount but a different
|
||||
// description does NOT dedupe when the stored row is from the SAME feed
|
||||
// (a re-import that legitimately holds two distinct same-(date,amount)
|
||||
// transactions). Only a cross-channel mirror (2c-bis) drops the
|
||||
// description requirement.
|
||||
// -----------------------------------------------------------------------
|
||||
it('does not dedupe when date+amount match but description differs', async () => {
|
||||
it('does not dedupe a same-channel row when the description differs', async () => {
|
||||
const { supabase, enqueue } = createQueueMockSupabase()
|
||||
const raw = makeRaw({
|
||||
date: '2024-06-15',
|
||||
amount: -250.0,
|
||||
description: 'Coop Stockholm',
|
||||
external_id: 'lunar_csvhash456',
|
||||
external_id: 'csv_lunar_456',
|
||||
import_source: 'csv_lunar',
|
||||
})
|
||||
const inserted = makeTransaction({
|
||||
@@ -377,9 +379,10 @@ describe('ingestTransactions', () => {
|
||||
|
||||
// Booked transaction map query — none
|
||||
enqueue({ data: [], error: null })
|
||||
// Unbooked bank-synced transaction map query — a PSD2 row with same date/amount but DIFFERENT description
|
||||
// Unbooked row from the SAME feed (csv_lunar), same date/amount, DIFFERENT
|
||||
// description → not a cross-channel mirror → must NOT dedupe.
|
||||
enqueue({
|
||||
data: [{ date: '2024-06-15', amount: -250.0, description: 'ICA Maxi Solna' }],
|
||||
data: [{ date: '2024-06-15', amount: -250.0, original_description: 'ICA Maxi Solna', description: 'ICA Maxi Solna', import_source: 'csv_lunar' }],
|
||||
error: null,
|
||||
})
|
||||
// Supplier invoices fetch
|
||||
@@ -398,6 +401,250 @@ describe('ingestTransactions', () => {
|
||||
expect(result.transaction_ids).toEqual(['tx-no-collision'])
|
||||
})
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// 2c-bis. Cross-channel mirror: the SAME bank account imported via two feeds
|
||||
// (Nordea CSV payee text vs PSD2 OCR/message) — same date+amount, one row
|
||||
// per channel, descriptions that do NOT bridge — IS deduped on
|
||||
// (date, öre). This is the AXMD/Axel case: a CSV import landing on top of
|
||||
// existing Enable Banking rows whose descriptions share no text.
|
||||
// -----------------------------------------------------------------------
|
||||
it('dedupes a cross-channel mirror (CSV vs PSD2) even when descriptions do not bridge', async () => {
|
||||
const { supabase, enqueue } = createQueueMockSupabase()
|
||||
const raw = makeRaw({
|
||||
date: '2025-10-23',
|
||||
amount: -941,
|
||||
description: 'Fortnox Finans AB', // Nordea CSV payee
|
||||
external_id: 'nordea_business_abc123',
|
||||
import_source: 'csv_nordea_business',
|
||||
})
|
||||
|
||||
// Booked transaction map query — none
|
||||
enqueue({ data: [], error: null })
|
||||
// Stored unbooked PSD2 row: same date+amount, DIFFERENT text (the OCR), from
|
||||
// a DIFFERENT feed (enable_banking).
|
||||
enqueue({
|
||||
data: [{ date: '2025-10-23', amount: -941, original_description: '506401841738056', description: '506401841738056', import_source: 'enable_banking' }],
|
||||
error: null,
|
||||
})
|
||||
// Supplier invoices fetch
|
||||
enqueue({ data: [], error: null })
|
||||
// Batch external_id dedup query — different namespace, no match
|
||||
enqueue({ data: [], error: null })
|
||||
// No insert — deduped by the cross-channel mirror.
|
||||
|
||||
const result = await ingestTransactions(supabase as never, COMPANY_ID, USER_ID, [raw])
|
||||
|
||||
expect(result.duplicates).toBe(1)
|
||||
expect(result.imported).toBe(0)
|
||||
})
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// 2c-ter. Cross-channel but NOT a mirror (counts differ) → ambiguous, so the
|
||||
// description requirement stands and nothing is dropped. Two incoming CSV
|
||||
// rows + one stored PSD2 row with no bridging text → both incoming kept.
|
||||
// Guards the rare case where the two feeds disagree on how many
|
||||
// transactions a (date, öre) bucket holds: prefer a visible (deletable)
|
||||
// duplicate over silently collapsing a genuinely-new row.
|
||||
// -----------------------------------------------------------------------
|
||||
it('does not text-independently dedupe an asymmetric cross-channel bucket', async () => {
|
||||
const { supabase, enqueue } = createQueueMockSupabase()
|
||||
const rows = [
|
||||
makeRaw({ date: '2025-10-23', amount: -500, description: 'Betalning A', external_id: 'nordea_business_a', import_source: 'csv_nordea_business' }),
|
||||
makeRaw({ date: '2025-10-23', amount: -500, description: 'Betalning B', external_id: 'nordea_business_b', import_source: 'csv_nordea_business' }),
|
||||
]
|
||||
|
||||
// Booked transaction map query — none
|
||||
enqueue({ data: [], error: null })
|
||||
// Only ONE stored PSD2 row (different text) → incoming 2 vs cross 1 = asymmetric.
|
||||
enqueue({
|
||||
data: [{ date: '2025-10-23', amount: -500, original_description: 'A107 RAMBER', description: 'A107 RAMBER', import_source: 'enable_banking' }],
|
||||
error: null,
|
||||
})
|
||||
// Supplier invoices fetch
|
||||
enqueue({ data: [], error: null })
|
||||
// Batch external_id dedup query — no match
|
||||
enqueue({ data: [], error: null })
|
||||
enqueue({ data: makeTransaction({ id: 'tx-a', amount: -500 }), error: null })
|
||||
enqueue({ data: makeTransaction({ id: 'tx-b', amount: -500 }), error: null })
|
||||
|
||||
mockEvaluateMappingRules.mockResolvedValue(makeMappingResult({ confidence: 0.5 }))
|
||||
|
||||
const result = await ingestTransactions(supabase as never, COMPANY_ID, USER_ID, rows)
|
||||
|
||||
expect(result.imported).toBe(2)
|
||||
expect(result.duplicates).toBe(0)
|
||||
})
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// 2c-quater. The cross-channel mirror is still subject to the account guard:
|
||||
// a mirror match on a DIFFERENT known cash account is rejected, so a
|
||||
// multi-account company never collapses a transaction across accounts.
|
||||
// -----------------------------------------------------------------------
|
||||
it('respects the cash-account guard on the cross-channel mirror path', async () => {
|
||||
const { supabase, enqueue } = createQueueMockSupabase()
|
||||
const raw = makeRaw({
|
||||
date: '2025-10-23',
|
||||
amount: -941,
|
||||
description: 'Fortnox Finans AB',
|
||||
external_id: 'nordea_business_xyz',
|
||||
import_source: 'csv_nordea_business',
|
||||
})
|
||||
const inserted = makeTransaction({ id: 'tx-acctB', amount: -941 })
|
||||
|
||||
// Booked transaction map query — none
|
||||
enqueue({ data: [], error: null })
|
||||
// Stored cross-feed twin, but it settled on a DIFFERENT account (A).
|
||||
enqueue({
|
||||
data: [{ date: '2025-10-23', amount: -941, original_description: '506401841738056', description: '506401841738056', import_source: 'enable_banking', cash_account_id: 'acct-A' }],
|
||||
error: null,
|
||||
})
|
||||
// Supplier invoices fetch
|
||||
enqueue({ data: [], error: null })
|
||||
// Batch external_id dedup query — no match
|
||||
enqueue({ data: [], error: null })
|
||||
// cash_accounts lookup → batch settled on account B
|
||||
enqueue({ data: { id: 'acct-B' }, error: null })
|
||||
// Insert — different account, not a duplicate
|
||||
enqueue({ data: inserted, error: null })
|
||||
|
||||
mockEvaluateMappingRules.mockResolvedValue(makeMappingResult({ confidence: 0.5 }))
|
||||
|
||||
const result = await ingestTransactions(supabase as never, COMPANY_ID, USER_ID, [raw], {
|
||||
settlementAccount: '1931',
|
||||
})
|
||||
|
||||
expect(result.imported).toBe(1)
|
||||
expect(result.duplicates).toBe(0)
|
||||
})
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// 2c-shadow. Same-feed scope-drift (Hole A) — SHADOW MODE. Enable Banking
|
||||
// returns the same account under a drifted IBAN, so the IBAN-embedded
|
||||
// external_id is new (Layer-1 misses) and, because both rows are the SAME
|
||||
// feed, the cross-channel mirror does not fire. The shadow detector
|
||||
// MEASURES how often an enforcing rule would treat this as a re-import —
|
||||
// it logs/counts but NEVER changes what is inserted. These tests pin both
|
||||
// that it detects the real case and, crucially, that it never flags a
|
||||
// genuine row (the only failure mode that would matter).
|
||||
// -----------------------------------------------------------------------
|
||||
it('shadow-flags a same-feed scope-drift re-import but still imports it (no behavior change)', async () => {
|
||||
const { supabase, enqueue } = createQueueMockSupabase()
|
||||
// Same account re-fetched under a drifted IBAN → new external_id, and a
|
||||
// description that shares no prefix with the stored row (so the text bridge
|
||||
// cannot catch it either — this is purely the scope-drift signal).
|
||||
const raw = makeRaw({
|
||||
date: '2024-06-15',
|
||||
amount: -250,
|
||||
description: 'TELENOR SVERIGE',
|
||||
external_id: 'eb_SE_NEW_2024-06-15_-25000_0',
|
||||
import_source: 'enable_banking',
|
||||
bank_connection_id: 'conn-1',
|
||||
})
|
||||
const inserted = makeTransaction({ id: 'tx-new', external_id: raw.external_id })
|
||||
|
||||
enqueue({ data: [], error: null }) // booked map — none
|
||||
// Unbooked map — the stored twin from the SAME feed under the OLD id scope.
|
||||
enqueue({
|
||||
data: [{
|
||||
date: '2024-06-15', amount: -250,
|
||||
original_description: 'LAN AXMD 19', description: 'LAN AXMD 19',
|
||||
import_source: 'enable_banking', bank_connection_id: 'conn-1',
|
||||
cash_account_id: 'ca-1930', external_id: 'eb_SE_OLD_2024-06-15_-25000_0',
|
||||
}],
|
||||
error: null,
|
||||
})
|
||||
enqueue({ data: [], error: null }) // supplier invoices — none
|
||||
enqueue({ data: [], error: null }) // external_id dedup — OLD id not among incoming NEW ids
|
||||
enqueue({ data: { id: 'ca-1930' }, error: null }) // cash_accounts — same account as the stored row
|
||||
enqueue({ data: inserted, error: null }) // insert — STILL imported (shadow only logs)
|
||||
mockEvaluateMappingRules.mockResolvedValue(makeMappingResult({ confidence: 0.5 }))
|
||||
|
||||
const result = await ingestTransactions(supabase as never, COMPANY_ID, USER_ID, [raw], {
|
||||
settlementAccount: '1930',
|
||||
})
|
||||
|
||||
// Detected, but NOT acted on: imports exactly as before.
|
||||
expect(result.imported).toBe(1)
|
||||
expect(result.duplicates).toBe(0)
|
||||
expect(result.shadow_scope_drift_candidates).toBe(1)
|
||||
})
|
||||
|
||||
it('does not shadow-flag an asymmetric same-feed bucket (counts differ)', async () => {
|
||||
const { supabase, enqueue } = createQueueMockSupabase()
|
||||
// TWO incoming rows share (date, amount) but only ONE stored twin exists →
|
||||
// the channels disagree on how many transactions the bucket holds, so the
|
||||
// signal is ambiguous and we stay silent.
|
||||
const rows = [
|
||||
makeRaw({ date: '2024-06-15', amount: -250, description: 'BETALNING A', external_id: 'eb_NEW_a', import_source: 'enable_banking' }),
|
||||
makeRaw({ date: '2024-06-15', amount: -250, description: 'BETALNING B', external_id: 'eb_NEW_b', import_source: 'enable_banking' }),
|
||||
]
|
||||
enqueue({ data: [], error: null }) // booked
|
||||
enqueue({
|
||||
data: [{ date: '2024-06-15', amount: -250, original_description: 'OCR 9988', description: 'OCR 9988', import_source: 'enable_banking', cash_account_id: null, external_id: 'eb_OLD_x' }],
|
||||
error: null,
|
||||
}) // unbooked — ONE stored twin
|
||||
enqueue({ data: [], error: null }) // supplier
|
||||
enqueue({ data: [], error: null }) // external_id dedup
|
||||
enqueue({ data: makeTransaction({ id: 'tx-a' }), error: null }) // insert a
|
||||
enqueue({ data: makeTransaction({ id: 'tx-b' }), error: null }) // insert b
|
||||
mockEvaluateMappingRules.mockResolvedValue(makeMappingResult({ confidence: 0.5 }))
|
||||
|
||||
const result = await ingestTransactions(supabase as never, COMPANY_ID, USER_ID, rows)
|
||||
|
||||
expect(result.imported).toBe(2)
|
||||
expect(result.duplicates).toBe(0)
|
||||
expect(result.shadow_scope_drift_candidates).toBe(0)
|
||||
})
|
||||
|
||||
it('does not shadow-flag when the stored twin is on a different known cash account', async () => {
|
||||
const { supabase, enqueue } = createQueueMockSupabase()
|
||||
const raw = makeRaw({ date: '2024-06-15', amount: -250, description: 'TELENOR', external_id: 'eb_NEW_acctB', import_source: 'enable_banking' })
|
||||
const inserted = makeTransaction({ id: 'tx-b', external_id: raw.external_id })
|
||||
enqueue({ data: [], error: null }) // booked
|
||||
enqueue({
|
||||
data: [{ date: '2024-06-15', amount: -250, original_description: 'OCR', description: 'OCR', import_source: 'enable_banking', cash_account_id: 'acct-A', external_id: 'eb_OLD_acctA' }],
|
||||
error: null,
|
||||
}) // unbooked twin on account A
|
||||
enqueue({ data: [], error: null }) // supplier
|
||||
enqueue({ data: [], error: null }) // external_id dedup
|
||||
enqueue({ data: { id: 'acct-B' }, error: null }) // cash_accounts → batch settled on account B
|
||||
enqueue({ data: inserted, error: null }) // insert
|
||||
mockEvaluateMappingRules.mockResolvedValue(makeMappingResult({ confidence: 0.5 }))
|
||||
|
||||
const result = await ingestTransactions(supabase as never, COMPANY_ID, USER_ID, [raw], { settlementAccount: '1931' })
|
||||
|
||||
expect(result.imported).toBe(1)
|
||||
expect(result.shadow_scope_drift_candidates).toBe(0)
|
||||
})
|
||||
|
||||
it('does not shadow-flag a genuinely new row when the stored sibling id re-arrives (not drift)', async () => {
|
||||
const { supabase, enqueue } = createQueueMockSupabase()
|
||||
// The stored row's id IS present in this batch (normal re-sync, no drift) →
|
||||
// Layer-1 dedupes it. The SECOND incoming row is a genuinely new same-day /
|
||||
// same-amount transaction with a non-bridging description: it must import
|
||||
// AND must NOT be shadow-flagged, because the stored sibling is not
|
||||
// "orphaned" by a drifted id. This is the data-loss guard.
|
||||
const rows = [
|
||||
makeRaw({ date: '2024-06-15', amount: -250, description: 'COFFEE STARBUCKS', external_id: 'eb_X_0', import_source: 'enable_banking' }),
|
||||
makeRaw({ date: '2024-06-15', amount: -250, description: 'LUNCH RESTAURANG', external_id: 'eb_X_1', import_source: 'enable_banking' }),
|
||||
]
|
||||
enqueue({ data: [], error: null }) // booked
|
||||
enqueue({
|
||||
data: [{ date: '2024-06-15', amount: -250, original_description: 'COFFEE STARBUCKS', description: 'COFFEE STARBUCKS', import_source: 'enable_banking', cash_account_id: null, external_id: 'eb_X_0' }],
|
||||
error: null,
|
||||
}) // stored = the _0 sibling
|
||||
enqueue({ data: [], error: null }) // supplier
|
||||
enqueue({ data: [{ external_id: 'eb_X_0' }], error: null }) // external_id dedup → eb_X_0 matches stored
|
||||
enqueue({ data: makeTransaction({ id: 'tx-x1' }), error: null }) // insert eb_X_1 only
|
||||
mockEvaluateMappingRules.mockResolvedValue(makeMappingResult({ confidence: 0.5 }))
|
||||
|
||||
const result = await ingestTransactions(supabase as never, COMPANY_ID, USER_ID, rows)
|
||||
|
||||
expect(result.duplicates).toBe(1) // eb_X_0 deduped by Layer-1
|
||||
expect(result.imported).toBe(1) // eb_X_1 (genuine new) imported
|
||||
expect(result.shadow_scope_drift_candidates).toBe(0) // and NOT shadow-flagged
|
||||
})
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// 2d. Description drift: PSD2 enrichment is prefix-preserving, so an
|
||||
// enriched re-import ("TIC" → "TIC BG … via internet") still bridges
|
||||
@@ -957,6 +1204,7 @@ describe('ingestTransactions', () => {
|
||||
auto_matched_invoices: 0,
|
||||
errors: 0,
|
||||
transaction_ids: [],
|
||||
shadow_scope_drift_candidates: 0,
|
||||
})
|
||||
})
|
||||
|
||||
|
||||
@@ -0,0 +1,134 @@
|
||||
/**
|
||||
* Booking-time duplicate guard for bank transactions.
|
||||
*
|
||||
* Why this exists
|
||||
* ---------------
|
||||
* A bank account's transactions can land in the `transactions` table twice — a
|
||||
* CSV import on top of a PSD2 sync, or a re-sync whose external_id drifted (see
|
||||
* the import dedup in lib/transactions/ingest.ts). Import-time dedup is
|
||||
* best-effort and can miss. The cosmetic cost of a missed duplicate is a second
|
||||
* row in the "Att bokföra" list. The REAL cost is booking BOTH copies: that
|
||||
* creates two verifikationer for one affärshändelse, double-counts the
|
||||
* cost/income, and is felaktig bokföring under BFL (the second verifikat has no
|
||||
* underlying event). Rättelse would then require storno, not deletion.
|
||||
*
|
||||
* This guard runs at booking time. Before a transaction becomes a verifikat it
|
||||
* looks for ANOTHER transaction in the same company that is already booked and
|
||||
* shares this one's (date, amount, cash account). If found, the caller surfaces
|
||||
* it as a WARNING — never a hard block, because genuinely repeated
|
||||
* same-(date,amount) payments do occur (e.g. several identical Swish transfers
|
||||
* in one day). The user confirms with force=true after reviewing the candidate.
|
||||
*
|
||||
* Mirrors the invoice-side `detectDuplicatePaymentVoucher`
|
||||
* (lib/invoices/duplicate-payment-detection.ts), but keyed on an already-booked
|
||||
* sibling TRANSACTION rather than a manually-posted journal entry.
|
||||
*/
|
||||
import type { SupabaseClient } from '@supabase/supabase-js'
|
||||
|
||||
/** Integer öre — representation-agnostic amount key (mirrors the ingest dedup). */
|
||||
function toOre(amount: number | string): number {
|
||||
return Math.round(Number(amount) * 100)
|
||||
}
|
||||
|
||||
/** An already-booked transaction that looks like the same real movement. */
|
||||
export interface BookedDuplicateCandidate {
|
||||
/** The sibling transaction that is already booked. */
|
||||
transaction_id: string
|
||||
/** Its verifikat. */
|
||||
journal_entry_id: string
|
||||
/** Human label, e.g. "A142" (voucher_series + voucher_number). */
|
||||
voucher_label: string
|
||||
entry_date: string
|
||||
description: string | null
|
||||
amount: number
|
||||
}
|
||||
|
||||
/** Minimal shape of the transaction about to be booked. */
|
||||
export interface BookingTarget {
|
||||
id: string
|
||||
date: string
|
||||
amount: number | string
|
||||
cash_account_id?: string | null
|
||||
}
|
||||
|
||||
/**
|
||||
* Find an already-booked sibling transaction sharing (date, amount, account).
|
||||
* Returns the single best candidate, or null.
|
||||
*
|
||||
* Account guard mirrors the import dedup bridge: when BOTH sides know their
|
||||
* cash_account_id they must match; a null on either side is treated as
|
||||
* compatible (single-account companies and un-backfilled rows behave as before).
|
||||
*
|
||||
* Fail-open: a query error returns null rather than throwing — a detection
|
||||
* failure must never block a legitimate booking. The pick is deterministic
|
||||
* (lowest id) so a re-detection under force=true returns the same candidate the
|
||||
* user reviewed.
|
||||
*/
|
||||
export async function detectBookedDuplicateTransaction(
|
||||
supabase: SupabaseClient,
|
||||
companyId: string,
|
||||
target: BookingTarget,
|
||||
): Promise<BookedDuplicateCandidate | null> {
|
||||
const targetOre = toOre(target.amount)
|
||||
if (targetOre === 0 || Number.isNaN(targetOre)) return null
|
||||
|
||||
// Same company, same date, already booked, not the target row itself. The
|
||||
// amount and account match is applied in JS so a numeric-string amount from
|
||||
// PostgREST ("-1616.00") collapses to the same öre as the number (-1616).
|
||||
const { data, error } = await supabase
|
||||
.from('transactions')
|
||||
.select('id, date, amount, description, cash_account_id, journal_entry_id')
|
||||
.eq('company_id', companyId)
|
||||
.eq('date', target.date)
|
||||
.not('journal_entry_id', 'is', null)
|
||||
.neq('id', target.id)
|
||||
.limit(100)
|
||||
|
||||
if (error || !data || data.length === 0) return null
|
||||
|
||||
type Row = {
|
||||
id: string
|
||||
date: string
|
||||
amount: number | string
|
||||
description: string | null
|
||||
cash_account_id: string | null
|
||||
journal_entry_id: string
|
||||
}
|
||||
const targetAccount = target.cash_account_id ?? null
|
||||
const matches = (data as unknown as Row[]).filter((r) => {
|
||||
if (toOre(r.amount) !== targetOre) return false
|
||||
// Account guard: both-known must match; a null on either side is compatible.
|
||||
if (targetAccount !== null && r.cash_account_id !== null && r.cash_account_id !== targetAccount) {
|
||||
return false
|
||||
}
|
||||
return r.journal_entry_id != null
|
||||
})
|
||||
if (matches.length === 0) return null
|
||||
|
||||
matches.sort((a, b) => a.id.localeCompare(b.id))
|
||||
const best = matches[0]
|
||||
|
||||
// Resolve the voucher label for the warning (best-effort — a missing label
|
||||
// still yields a usable candidate the UI can render by date/amount).
|
||||
let voucherLabel = ''
|
||||
let entryDate = best.date
|
||||
const { data: je } = await supabase
|
||||
.from('journal_entries')
|
||||
.select('voucher_series, voucher_number, entry_date')
|
||||
.eq('id', best.journal_entry_id)
|
||||
.maybeSingle()
|
||||
if (je) {
|
||||
const j = je as { voucher_series: string | null; voucher_number: number | null; entry_date: string | null }
|
||||
voucherLabel = `${j.voucher_series ?? 'A'}${j.voucher_number ?? ''}`
|
||||
entryDate = j.entry_date ?? best.date
|
||||
}
|
||||
|
||||
return {
|
||||
transaction_id: best.id,
|
||||
journal_entry_id: best.journal_entry_id,
|
||||
voucher_label: voucherLabel,
|
||||
entry_date: entryDate,
|
||||
description: best.description,
|
||||
amount: Math.round(Number(best.amount) * 100) / 100,
|
||||
}
|
||||
}
|
||||
+245
-47
@@ -8,6 +8,8 @@ import { fetchExchangeRate } from '@/lib/currency/riksbanken'
|
||||
import { logMatchEvent } from '@/lib/invoices/match-log'
|
||||
import { fetchAllRows } from '@/lib/supabase/fetch-all'
|
||||
import { contentBucketKey, descriptionsBridge, normalizeImportedDescription } from '@/lib/transactions/external-id'
|
||||
import { isImportedTransaction } from '@/lib/transactions/origin'
|
||||
import { createLogger } from '@/lib/logger'
|
||||
import type { Transaction, RawTransaction, IngestResult, IngestOptions, SupplierInvoice, Currency, ExchangeRate } from '@/types'
|
||||
|
||||
// Re-export types for backward compatibility
|
||||
@@ -15,11 +17,26 @@ export type { RawTransaction, IngestResult } from '@/types'
|
||||
|
||||
/**
|
||||
* One existing row in a content-dedup bucket: its normalized/lowercased
|
||||
* description plus the cash account it settled on (null for legacy rows that
|
||||
* predate the cash_account_id backfill). `cashAccountId` is the cross-account
|
||||
* guard — see `consumeBridgingTwin`.
|
||||
* description, the cash account it settled on (null for legacy rows that
|
||||
* predate the cash_account_id backfill), the import channel it came from, and
|
||||
* whether that channel is an external feed (vs a hand-entered row). `source` +
|
||||
* `isImportFeed` drive the cross-channel mirror bridge (see
|
||||
* `consumeBridgingTwin`); `cashAccountId` is the cross-account guard.
|
||||
*/
|
||||
type BucketEntry = { desc: string; cashAccountId: string | null }
|
||||
type BucketEntry = {
|
||||
desc: string
|
||||
cashAccountId: string | null
|
||||
source: string | null
|
||||
isImportFeed: boolean
|
||||
/**
|
||||
* The stored row's `external_id`. Used ONLY by the shadow-mode same-feed
|
||||
* scope-drift instrumentation (see ingestTransactions): a stored row is a
|
||||
* "drift candidate" when its id is NOT among the incoming batch's ids, which
|
||||
* is what distinguishes an IBAN-scope re-import from a normal sibling whose id
|
||||
* Layer-1 already reconciles. Null for rows predating the column.
|
||||
*/
|
||||
externalId: string | null
|
||||
}
|
||||
|
||||
/**
|
||||
* Content-dedup bucket: a `{date}|{öre}` key mapped to the multiset of existing
|
||||
@@ -34,13 +51,16 @@ interface ExistingTransactionMaps {
|
||||
/** Booked transactions (any source) — consumed by any incoming raw transaction. */
|
||||
booked: DescBucket
|
||||
/**
|
||||
* Unbooked enable_banking transactions — consumed by any incoming raw
|
||||
* transaction regardless of source. Catches two cases: PSD2 reconnect
|
||||
* duplicates (external_id regenerated, same tx already pending) AND
|
||||
* CSV imports overlapping an active PSD2 sync (same Lunar/etc tx arriving
|
||||
* twice, once via PSD2 and once via file upload).
|
||||
* Unbooked rows from ANY external import feed (Enable Banking PSD2 sync,
|
||||
* bank-file CSV/CAMT import) — consumed by any incoming raw transaction
|
||||
* regardless of source. Catches the cross-channel re-import: the same bank
|
||||
* account pulled once via PSD2 and once via a CSV/CAMT file upload (in either
|
||||
* order), plus PSD2 reconnect duplicates whose external_id regenerated.
|
||||
* Hand-entered rows (import_source manual/mcp/null) are deliberately
|
||||
* excluded — only real feeds mirror one another, and a manual row must never
|
||||
* be silently consumed by an import.
|
||||
*/
|
||||
unbookedEnableBanking: DescBucket
|
||||
unbookedImported: DescBucket
|
||||
}
|
||||
|
||||
/** Push a row into its (date, öre) bucket, normalizing the description. */
|
||||
@@ -50,9 +70,18 @@ function addToBucket(
|
||||
amount: number | string,
|
||||
description: string,
|
||||
cashAccountId: string | null,
|
||||
source: string | null,
|
||||
isImportFeed: boolean,
|
||||
externalId: string | null,
|
||||
): void {
|
||||
const key = contentBucketKey(date, amount)
|
||||
const entry: BucketEntry = { desc: description.toLowerCase().trim(), cashAccountId }
|
||||
const entry: BucketEntry = {
|
||||
desc: description.toLowerCase().trim(),
|
||||
cashAccountId,
|
||||
source,
|
||||
isImportFeed,
|
||||
externalId,
|
||||
}
|
||||
const entries = bucket.get(key)
|
||||
if (entries) entries.push(entry)
|
||||
else bucket.set(key, [entry])
|
||||
@@ -64,8 +93,8 @@ async function buildExistingTransactionMaps(
|
||||
rawTransactions: RawTransaction[]
|
||||
): Promise<ExistingTransactionMaps> {
|
||||
const booked: DescBucket = new Map()
|
||||
const unbookedEnableBanking: DescBucket = new Map()
|
||||
if (rawTransactions.length === 0) return { booked, unbookedEnableBanking }
|
||||
const unbookedImported: DescBucket = new Map()
|
||||
if (rawTransactions.length === 0) return { booked, unbookedImported }
|
||||
|
||||
const dates = rawTransactions.map((t) => t.date).sort()
|
||||
const dateFrom = dates[0]
|
||||
@@ -74,7 +103,7 @@ async function buildExistingTransactionMaps(
|
||||
try {
|
||||
const { data: bookedRows } = await supabase
|
||||
.from('transactions')
|
||||
.select('date, amount, original_description, description, cash_account_id')
|
||||
.select('date, amount, original_description, description, cash_account_id, import_source, bank_connection_id, external_id')
|
||||
.eq('company_id', companyId)
|
||||
.not('journal_entry_id', 'is', null)
|
||||
.gte('date', dateFrom)
|
||||
@@ -92,6 +121,9 @@ async function buildExistingTransactionMaps(
|
||||
tx.amount,
|
||||
normalizeImportedDescription(tx.original_description ?? tx.description),
|
||||
tx.cash_account_id ?? null,
|
||||
tx.import_source ?? null,
|
||||
isImportedTransaction({ import_source: tx.import_source, bank_connection_id: tx.bank_connection_id }),
|
||||
tx.external_id ?? null,
|
||||
)
|
||||
}
|
||||
}
|
||||
@@ -100,25 +132,35 @@ async function buildExistingTransactionMaps(
|
||||
}
|
||||
|
||||
try {
|
||||
const { data: unbookedBank } = await supabase
|
||||
// ALL unbooked import-feed rows — not just enable_banking. An unbooked CSV
|
||||
// row must dedup an incoming PSD2 sync of the same account, and an unbooked
|
||||
// PSD2 row must dedup an incoming CSV import. Feeds always set a non-null
|
||||
// import_source outside the user-created allowlist (manual/mcp); null /
|
||||
// manual / mcp are hand-entered and intentionally excluded.
|
||||
const { data: unbookedRows } = await supabase
|
||||
.from('transactions')
|
||||
.select('date, amount, original_description, description, cash_account_id')
|
||||
.select('date, amount, original_description, description, cash_account_id, import_source, bank_connection_id, external_id')
|
||||
.eq('company_id', companyId)
|
||||
.is('journal_entry_id', null)
|
||||
.eq('import_source', 'enable_banking')
|
||||
.not('import_source', 'is', null)
|
||||
.neq('import_source', 'manual')
|
||||
.neq('import_source', 'mcp')
|
||||
.gte('date', dateFrom)
|
||||
.lte('date', dateTo)
|
||||
|
||||
if (unbookedBank) {
|
||||
for (const tx of unbookedBank) {
|
||||
if (unbookedRows) {
|
||||
for (const tx of unbookedRows) {
|
||||
// See booked-map note: dedup on the immutable bank original so a
|
||||
// user title edit cannot reopen the duplicate-import window.
|
||||
addToBucket(
|
||||
unbookedEnableBanking,
|
||||
unbookedImported,
|
||||
tx.date,
|
||||
tx.amount,
|
||||
normalizeImportedDescription(tx.original_description ?? tx.description),
|
||||
tx.cash_account_id ?? null,
|
||||
tx.import_source ?? null,
|
||||
isImportedTransaction({ import_source: tx.import_source, bank_connection_id: tx.bank_connection_id }),
|
||||
tx.external_id ?? null,
|
||||
)
|
||||
}
|
||||
}
|
||||
@@ -126,7 +168,7 @@ async function buildExistingTransactionMaps(
|
||||
// Non-critical — reconnect dedup will be skipped
|
||||
}
|
||||
|
||||
return { booked, unbookedEnableBanking }
|
||||
return { booked, unbookedImported }
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -163,15 +205,60 @@ export async function ingestTransactions(
|
||||
auto_matched_invoices: 0,
|
||||
errors: 0,
|
||||
transaction_ids: [],
|
||||
shadow_scope_drift_candidates: 0,
|
||||
}
|
||||
|
||||
// Pre-fetch existing transactions for content-based dedup
|
||||
// (date+amount+description prefix). Booked rows catch cross-source
|
||||
// duplicates after they've been booked; unbooked enable_banking rows
|
||||
// catch the more common case where a PSD2 row is still pending in the
|
||||
// inbox when the user re-imports the same period via CSV.
|
||||
const log = createLogger('transactions.ingest', { companyId })
|
||||
// SHADOW-ONLY instrumentation for the same-feed scope-drift bridge (Hole A:
|
||||
// Enable Banking returns the same account under a drifted IBAN, the
|
||||
// IBAN-embedded external_id changes, Layer-1 dedup misses the re-import, and
|
||||
// because both rows are the SAME feed the cross-channel mirror does not fire).
|
||||
// When on, we LOG which rows an enforcing rule WOULD treat as re-imports and
|
||||
// count them — but never change what gets inserted. Default on; set
|
||||
// DEDUP_SCOPE_DRIFT_MODE=off to silence. There is deliberately NO 'enforce'
|
||||
// branch yet: we validate on real fleet data first (see the plan).
|
||||
const scopeDriftShadow = process.env.DEDUP_SCOPE_DRIFT_MODE !== 'off'
|
||||
|
||||
// Pre-fetch existing transactions for content-based dedup (date+amount+
|
||||
// description prefix, plus the cross-channel mirror below). Booked rows catch
|
||||
// cross-source duplicates after they've been booked; unbooked import-feed rows
|
||||
// catch the common case where a PSD2 row is still unbooked when the user
|
||||
// re-imports the same period via CSV — or the reverse, a CSV import that
|
||||
// predates the first PSD2 sync of the same account.
|
||||
const existingMaps = await buildExistingTransactionMaps(supabase, companyId, rawTransactions)
|
||||
|
||||
// Every row in one ingest call shares an import_source — EB sync passes
|
||||
// 'enable_banking', bank-file import passes 'csv_<format>'/'camt053' — so the
|
||||
// first row's source identifies this batch's channel. We use it to find
|
||||
// "cross-channel mirror" buckets: a (date, öre) bucket where the number of
|
||||
// incoming rows EQUALS the number of stored rows from a DIFFERENT feed. That
|
||||
// equality is the signal that the same set of real transactions is arriving
|
||||
// once per channel (e.g. Nordea's CSV export and its PSD2 feed), where the
|
||||
// per-row description is known-unreliable — CSV shows the payee, PSD2 the
|
||||
// OCR/message, or vice versa. Only in those buckets do we dedup on
|
||||
// (date, öre, account) without a description match (see consumeBridgingTwin).
|
||||
// An asymmetric bucket keeps the description requirement, so a genuinely-new
|
||||
// row is never collapsed into a different one.
|
||||
const batchSource = rawTransactions[0]?.import_source ?? null
|
||||
const batchIsImportFeed = isImportedTransaction({ import_source: batchSource })
|
||||
const incomingByBucket = new Map<string, number>()
|
||||
const crossSourceStoredByBucket = new Map<string, number>()
|
||||
if (batchIsImportFeed) {
|
||||
for (const raw of rawTransactions) {
|
||||
const k = contentBucketKey(raw.date, raw.amount)
|
||||
incomingByBucket.set(k, (incomingByBucket.get(k) ?? 0) + 1)
|
||||
}
|
||||
for (const bucket of [existingMaps.booked, existingMaps.unbookedImported]) {
|
||||
for (const [k, entries] of bucket) {
|
||||
for (const entry of entries) {
|
||||
if (entry.isImportFeed && entry.source !== batchSource) {
|
||||
crossSourceStoredByBucket.set(k, (crossSourceStoredByBucket.get(k) ?? 0) + 1)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// When rawInsertOnly is set (viewer imports), skip pre-fetching supplier
|
||||
// invoices and exchange rates — they are not used.
|
||||
let unpaidSupplierInvoices: SupplierInvoice[] = []
|
||||
@@ -266,6 +353,50 @@ export async function ingestTransactions(
|
||||
cashAccountId = (ca?.id as string | undefined) ?? null
|
||||
}
|
||||
|
||||
// ── Shadow-mode same-feed scope-drift precompute (measure only) ──────────
|
||||
// Two per-(date, öre) bucket counts that, when EQUAL and non-zero, mark a
|
||||
// bucket as a probable scope-drift mirror:
|
||||
// - unmatchedIncomingByBucket: incoming rows whose external_id is NOT
|
||||
// already stored (i.e. Layer-1 will not reconcile them — the ones that
|
||||
// would otherwise insert as fresh rows).
|
||||
// - driftCandidateStoredByBucket: stored rows from THIS SAME feed whose id
|
||||
// the incoming batch does NOT carry (so they are "orphaned" by a drifted
|
||||
// id), restricted to account-compatible rows. Account compatibility uses
|
||||
// the batch settlement account (cash_account_id), which is keyed on the
|
||||
// provider's STABLE account uid — not the drifting IBAN that broke the
|
||||
// external_id (see lib/cash-accounts/service.ts upsertFromPsd2). So a
|
||||
// genuinely different account on the same company is never a candidate.
|
||||
// Equality is the safety signal (same as the cross-channel mirror): it means
|
||||
// the same set of transactions re-arrived once, under new ids. An asymmetric
|
||||
// bucket is left alone. Counts are pre-loop snapshots; the gate is evaluated
|
||||
// per incoming row inside the loop.
|
||||
const incomingIdSet = new Set(externalIds)
|
||||
const unmatchedIncomingByBucket = new Map<string, number>()
|
||||
const driftCandidateStoredByBucket = new Map<string, number>()
|
||||
if (batchIsImportFeed && scopeDriftShadow) {
|
||||
for (const raw of rawTransactions) {
|
||||
if (!existingExternalIds.has(raw.external_id)) {
|
||||
const k = contentBucketKey(raw.date, raw.amount)
|
||||
unmatchedIncomingByBucket.set(k, (unmatchedIncomingByBucket.get(k) ?? 0) + 1)
|
||||
}
|
||||
}
|
||||
for (const bucket of [existingMaps.booked, existingMaps.unbookedImported]) {
|
||||
for (const [k, entries] of bucket) {
|
||||
for (const entry of entries) {
|
||||
const sameFeed = entry.isImportFeed && entry.source === batchSource
|
||||
const accountCompatible =
|
||||
cashAccountId === null ||
|
||||
entry.cashAccountId === null ||
|
||||
entry.cashAccountId === cashAccountId
|
||||
const idOrphaned = entry.externalId !== null && !incomingIdSet.has(entry.externalId)
|
||||
if (sameFeed && accountCompatible && idOrphaned) {
|
||||
driftCandidateStoredByBucket.set(k, (driftCandidateStoredByBucket.get(k) ?? 0) + 1)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Track already-matched invoice IDs within this ingestion batch
|
||||
// to prevent suggesting the same invoice for multiple transactions
|
||||
const matchedInvoiceIds = new Set<string>()
|
||||
@@ -287,60 +418,127 @@ export async function ingestTransactions(
|
||||
}
|
||||
|
||||
// 1b/1c. Content-dedup bridge: skip if an existing booked row (any source)
|
||||
// OR an unbooked enable_banking row shares this (date, öre) bucket and a
|
||||
// *bridging* description (prefix-containment, see descriptionsBridge). This
|
||||
// is the net that catches re-imports the external_id check misses — chiefly
|
||||
// old-format ids re-synced after the id scheme changed, and PSD2 description
|
||||
// enrichment between syncs ("TIC" → "TIC BG … via internet"). Booked first,
|
||||
// then unbooked, preserving the historical 1b-before-1c order.
|
||||
// OR an unbooked import-feed row shares this (date, öre) bucket and EITHER
|
||||
// (a) a *bridging* description (prefix-containment, see descriptionsBridge),
|
||||
// OR (b) the bucket is a cross-channel mirror (crossSourceMirror below).
|
||||
// (a) catches re-imports the external_id check misses — old-format ids
|
||||
// re-synced after the id scheme changed, and PSD2 description enrichment
|
||||
// between syncs ("TIC" → "TIC BG … via internet"). (b) catches the same
|
||||
// bank account imported via two channels whose descriptions don't bridge at
|
||||
// all (Nordea CSV payee "TELENOR"/"Nordea" vs PSD2 OCR/message), which (a)
|
||||
// alone cannot. Booked first, then unbooked.
|
||||
//
|
||||
// Consumed with COUNTING semantics: each match splices one stored entry out
|
||||
// of its bucket, so N stored twins dedup exactly N incoming and two
|
||||
// genuinely-distinct same-(date,amount) transactions are kept apart. We
|
||||
// consume the LONGEST bridging stored description first so a more-specific
|
||||
// twin is matched before a generic one, leaving generic entries for shorter
|
||||
// incoming rows.
|
||||
// genuinely-distinct same-(date,amount) transactions are kept apart. The
|
||||
// text bridge is tried first (LONGEST bridging description wins, so a
|
||||
// more-specific twin is matched before a generic one); the cross-channel
|
||||
// mirror is the text-independent fallback.
|
||||
//
|
||||
// Account guard: when BOTH the incoming batch and a stored entry have a known
|
||||
// cash_account_id, they must match — so a transaction on one bank account
|
||||
// never deduplicates a genuinely-different one on another account of the same
|
||||
// company (the content bucket is company-wide; only external_id embeds the
|
||||
// account). A null on either side falls back to bridge-allowed, leaving
|
||||
// single-account and legacy (un-backfilled) rows exactly as before.
|
||||
// single-account and legacy (un-backfilled) rows exactly as before. The guard
|
||||
// applies to BOTH the text and the cross-channel-mirror path.
|
||||
//
|
||||
// Residual trade-off: within one account, a genuinely-new row whose
|
||||
// description is a prefix-extension of an existing same-(date,öre) row can be
|
||||
// mis-deduped; it is rare, bounded to the ~90-day PSD2 window where old-format
|
||||
// ids still overlap, and the frozen external_id (Layer 1) is the exact dedup
|
||||
// going forward — accepted to stop the re-import flood (the inverse, a visible
|
||||
// duplicate, was the reported pain).
|
||||
// crossSourceMirror: this (date, öre) bucket holds the same number of
|
||||
// incoming rows as stored rows from a different feed → the same real
|
||||
// transactions arriving once per channel. Only then is the description
|
||||
// requirement dropped; an asymmetric bucket keeps it, so when the channels
|
||||
// disagree on how many transactions a bucket holds we keep a visible
|
||||
// (deletable) duplicate rather than risk collapsing a genuinely-new row.
|
||||
const bucketKey = contentBucketKey(raw.date, raw.amount)
|
||||
const crossSourceMirror =
|
||||
batchIsImportFeed &&
|
||||
(crossSourceStoredByBucket.get(bucketKey) ?? 0) > 0 &&
|
||||
incomingByBucket.get(bucketKey) === crossSourceStoredByBucket.get(bucketKey)
|
||||
const consumeBridgingTwin = (bucket: DescBucket): boolean => {
|
||||
const entries = bucket.get(bucketKey)
|
||||
if (!entries || entries.length === 0) return false
|
||||
let bestIdx = -1
|
||||
let bestLen = -1
|
||||
let crossIdx = -1
|
||||
for (let i = 0; i < entries.length; i++) {
|
||||
const entry = entries[i]
|
||||
const sameAccount =
|
||||
cashAccountId === null || entry.cashAccountId === null || entry.cashAccountId === cashAccountId
|
||||
if (sameAccount && descriptionsBridge(description, entry.desc) && entry.desc.length > bestLen) {
|
||||
if (!sameAccount) continue
|
||||
if (descriptionsBridge(description, entry.desc) && entry.desc.length > bestLen) {
|
||||
bestIdx = i
|
||||
bestLen = entry.desc.length
|
||||
}
|
||||
// Text-independent fallback: in a cross-channel mirror bucket a stored
|
||||
// entry from a different feed is the same transaction even when the
|
||||
// descriptions don't bridge. Remember the first eligible one.
|
||||
if (crossIdx === -1 && crossSourceMirror && entry.isImportFeed && entry.source !== batchSource) {
|
||||
crossIdx = i
|
||||
}
|
||||
}
|
||||
if (bestIdx === -1) return false
|
||||
entries.splice(bestIdx, 1)
|
||||
const idx = bestIdx !== -1 ? bestIdx : crossIdx
|
||||
if (idx === -1) return false
|
||||
entries.splice(idx, 1)
|
||||
return true
|
||||
}
|
||||
if (
|
||||
consumeBridgingTwin(existingMaps.booked) ||
|
||||
consumeBridgingTwin(existingMaps.unbookedEnableBanking)
|
||||
consumeBridgingTwin(existingMaps.unbookedImported)
|
||||
) {
|
||||
result.duplicates++
|
||||
continue
|
||||
}
|
||||
|
||||
// SHADOW-ONLY: this row survived Layer-1 and Layer-2, so today it WILL
|
||||
// insert. If its bucket is a symmetric same-feed scope-drift mirror (equal
|
||||
// non-zero counts of unreconciled incoming rows and account-compatible
|
||||
// same-feed drift candidates), an enforcing rule WOULD treat it as a
|
||||
// re-import. We only record it — full content on both sides so every
|
||||
// decision can be human-verified against real fleet data before any
|
||||
// enforcement is switched on — then fall through and insert exactly as
|
||||
// before. This block has NO effect on result.imported/duplicates.
|
||||
if (scopeDriftShadow && batchIsImportFeed) {
|
||||
const driftCount = driftCandidateStoredByBucket.get(bucketKey) ?? 0
|
||||
const unmatchedCount = unmatchedIncomingByBucket.get(bucketKey) ?? 0
|
||||
if (driftCount > 0 && unmatchedCount === driftCount) {
|
||||
let matched: BucketEntry | undefined
|
||||
for (const bucket of [existingMaps.booked, existingMaps.unbookedImported]) {
|
||||
const entries = bucket.get(bucketKey)
|
||||
if (!entries) continue
|
||||
matched = entries.find(
|
||||
(e) =>
|
||||
e.isImportFeed &&
|
||||
e.source === batchSource &&
|
||||
e.externalId !== null &&
|
||||
!incomingIdSet.has(e.externalId) &&
|
||||
(cashAccountId === null ||
|
||||
e.cashAccountId === null ||
|
||||
e.cashAccountId === cashAccountId)
|
||||
)
|
||||
if (matched) break
|
||||
}
|
||||
if (matched) {
|
||||
result.shadow_scope_drift_candidates =
|
||||
(result.shadow_scope_drift_candidates ?? 0) + 1
|
||||
log.info('import dedup shadow: same-feed scope-drift candidate', {
|
||||
decision: 'same-feed-scope-drift',
|
||||
mode: 'shadow',
|
||||
bucket: bucketKey,
|
||||
unmatchedIncoming: unmatchedCount,
|
||||
driftCandidates: driftCount,
|
||||
incomingExternalId: raw.external_id,
|
||||
incomingDescription: description,
|
||||
incomingAmount: raw.amount,
|
||||
incomingSource: raw.import_source ?? null,
|
||||
cashAccountId,
|
||||
matchedStoredExternalId: matched.externalId,
|
||||
matchedStoredDescription: matched.desc,
|
||||
matchedStoredCashAccountId: matched.cashAccountId,
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 2. Insert new transaction (with SEK conversion for foreign currencies)
|
||||
const rateInfo = raw.currency && raw.currency !== 'SEK'
|
||||
? exchangeRatesByDate.get(`${raw.currency}|${raw.date}`)
|
||||
|
||||
Reference in New Issue
Block a user