331ae11867
* fix(transactions): make content-dedup bridge resilient to PSD2 description drift
Enable Banking re-syncs were re-importing every overlapping transaction as a
duplicate. Two changes in the June 1 deploy combined to defeat both dedup layers
at once: the external_id format changed (old rows' stored ids no longer match the
new scheme, so the exact-match layer misses) AND PSD2 descriptions were enriched
("TIC" -> "TIC BG 0000005786439 Bg-bet. via internet"), so the content-dedup
bridge — which compared a fixed 24-char description prefix for equality — also
missed. Result: a full re-import (observed: 53 of 54 "new" rows were dupes).
The external_id format has changed several times historically and 7,120 of 9,393
old rows have no reconstructable canonical id, so a backfill is not viable; the
content bridge is the mechanism meant to survive id-scheme changes. Harden it:
- Split the bridge into bucketing (date, öre) and matching (description), and
match by prefix-containment instead of fixed-prefix equality. PSD2 enrichment
is prefix-preserving, so the enriched re-import bridges its stored original,
while genuinely-distinct same-(date,amount) rows (distinct descriptions) are
kept apart. Consumed with counting semantics + longest-match, so N stored twins
dedup exactly N incoming.
- Replace contentDedupKey with contentBucketKey + descriptionsBridge; update the
live pipeline (ingest.ts) and the v1 dry-run preview to the same logic.
- Freeze the external_id format with a regression test + a header warning: any
future format change must ship a coordinated backfill.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test/refactor: address PR review — sanitize fixtures, mirror preview counting
- Replace real customer names ("Carl Bennet AB", "Brorsan AB") and prod-derived
reference strings in tests with clearly fictional stand-ins (compliance A.8.33).
- v1 dry-run preview: use the same longest-match + counting/consume semantics as
the live pipeline so a batch of N copies against M booked twins previews M skips,
not N (greptile P2). Update stale pitfall docs: content dedup is now
date+amount+description (prefix-containment), and the preview is booked-only so
its skip count is a lower bound on the live skip count.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(transactions): scope content-dedup bridge by cash account
The content-dedup bridge buckets by (date, öre) company-wide, with no account
scope — while bank reconciliation IS account-scoped (cash_account_id). For a
company with multiple bank accounts, a transaction on account A could therefore
deduplicate a genuinely-different transaction on account B that shares the same
date, amount, and a prefix-bridging description (round-number fees/transfers are
the realistic trigger), dropping a real row before it reaches reconciliation.
Layer 1 (external_id) is already account-safe because the account IBAN is
embedded in the id; only the fuzzy content bridge was account-blind.
Add an account guard: store cash_account_id alongside each bucket entry and only
bridge when BOTH the incoming batch and the stored entry have a known
cash_account_id that matches. A null on either side falls back to bridge-allowed,
so single-account companies and legacy (un-backfilled) rows are unchanged — and
CSV-vs-PSD2 dedup for the same account still works. Affects the 11 multi-account
companies; everyone else is behaviourally identical.
Note: external_id format heterogeneity (old entry_reference / old date+amount /
new öre+index schemes) is harmless downstream — nothing parses the id; it is an
opaque exact-match dedup key and a display string. Reconciliation, invoice/
supplier/payment matching, and reporting all key off real transaction columns,
never external_id.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* harden(transactions): blank description never wildcards a described row
PR review (OWASP V8.2.1 + Swedish compliance) flagged that descriptionsBridge
returned true whenever either side was empty, so a blank stored/incoming title
could wildcard-match any same-(date,öre) transaction and silently consume a real
one. Every live caller normalizes blanks to FALLBACK_DESCRIPTION upstream, so the
branch was unreachable in production — but make the function safe in isolation:
a blank now bridges only another blank (date+öre identity), never a described row.
No live behaviour change; removes the footgun.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
178 lines
7.5 KiB
TypeScript
178 lines
7.5 KiB
TypeScript
import { describe, it, expect } from 'vitest'
|
|
import {
|
|
amountToOre,
|
|
buildStableExternalIds,
|
|
contentBucketKey,
|
|
descriptionsBridge,
|
|
normalizeImportedDescription,
|
|
FALLBACK_DESCRIPTION,
|
|
} from '../external-id'
|
|
|
|
describe('amountToOre', () => {
|
|
it('normalizes a JS number to integer öre', () => {
|
|
expect(amountToOre(1234.5)).toBe(123450)
|
|
expect(amountToOre(-250)).toBe(-25000)
|
|
expect(amountToOre(0)).toBe(0)
|
|
})
|
|
|
|
it('normalizes a numeric string (PostgREST representation) to the same öre', () => {
|
|
// The core fix: a DB-fetched numeric string and a raw JS number for the
|
|
// same amount must collapse to the same integer.
|
|
expect(amountToOre('1234.50')).toBe(123450)
|
|
expect(amountToOre('1234.5')).toBe(amountToOre(1234.5))
|
|
expect(amountToOre('-250.00')).toBe(amountToOre(-250))
|
|
expect(amountToOre('100')).toBe(10000)
|
|
})
|
|
|
|
it('rounds sub-öre noise deterministically (never toFixed)', () => {
|
|
expect(amountToOre(0.1 + 0.2)).toBe(30) // 0.30000000000000004 → 30
|
|
expect(amountToOre(19.995)).toBe(2000)
|
|
})
|
|
})
|
|
|
|
describe('buildStableExternalIds', () => {
|
|
it('derives the id from account + date + öre, not from any bank id', () => {
|
|
const ids = buildStableExternalIds('eb', 'SE123', [{ date: '2024-06-15', amount: -500 }])
|
|
expect(ids).toEqual(['eb_SE123_2024-06-15_-50000_0'])
|
|
})
|
|
|
|
it('disambiguates genuinely identical transactions with an occurrence index', () => {
|
|
const ids = buildStableExternalIds('eb', 'acc', [
|
|
{ date: '2024-06-15', amount: -250 },
|
|
{ date: '2024-06-15', amount: -250 },
|
|
{ date: '2024-06-15', amount: -250 },
|
|
])
|
|
expect(ids).toEqual([
|
|
'eb_acc_2024-06-15_-25000_0',
|
|
'eb_acc_2024-06-15_-25000_1',
|
|
'eb_acc_2024-06-15_-25000_2',
|
|
])
|
|
})
|
|
|
|
it('produces the SAME set of ids regardless of provider ordering (re-sync dedupe)', () => {
|
|
const a = buildStableExternalIds('eb', 'acc', [
|
|
{ date: '2024-06-15', amount: -250 },
|
|
{ date: '2024-06-16', amount: -100 },
|
|
{ date: '2024-06-15', amount: -250 },
|
|
])
|
|
// Same transactions, different order on a later sync.
|
|
const b = buildStableExternalIds('eb', 'acc', [
|
|
{ date: '2024-06-15', amount: -250 },
|
|
{ date: '2024-06-15', amount: -250 },
|
|
{ date: '2024-06-16', amount: -100 },
|
|
])
|
|
expect(new Set(a)).toEqual(new Set(b))
|
|
})
|
|
|
|
it('treats string and number amounts as the same id (provider type drift)', () => {
|
|
const num = buildStableExternalIds('eb', 'acc', [{ date: '2024-06-15', amount: 1234.5 }])
|
|
const str = buildStableExternalIds('eb', 'acc', [{ date: '2024-06-15', amount: '1234.50' }])
|
|
expect(num).toEqual(str)
|
|
})
|
|
|
|
it('keeps distinct amounts and dates on separate occurrence counters', () => {
|
|
const ids = buildStableExternalIds('eb', 'acc', [
|
|
{ date: '2024-06-15', amount: -250 },
|
|
{ date: '2024-06-15', amount: -100 },
|
|
{ date: '2024-06-16', amount: -250 },
|
|
{ date: '2024-06-15', amount: -250 },
|
|
])
|
|
expect(ids).toEqual([
|
|
'eb_acc_2024-06-15_-25000_0',
|
|
'eb_acc_2024-06-15_-10000_0',
|
|
'eb_acc_2024-06-16_-25000_0',
|
|
'eb_acc_2024-06-15_-25000_1',
|
|
])
|
|
})
|
|
|
|
it('returns an empty array for an empty batch', () => {
|
|
expect(buildStableExternalIds('eb', 'acc', [])).toEqual([])
|
|
})
|
|
|
|
// FORMAT-FREEZE guard. `external_id` is a STORED key: changing the template
|
|
// string orphans every prior row (its stored id stops matching the new scheme)
|
|
// and re-imports the lot on the next sync — the June 2026 fleet-wide incident.
|
|
// If you must change the format, you MUST ship a coordinated backfill of
|
|
// existing rows; updating this assertion without one is the bug.
|
|
it('FORMAT IS FROZEN — changing this template silently orphans every prior external_id', () => {
|
|
expect(
|
|
buildStableExternalIds('eb', 'SE0000000000000000000000', [
|
|
{ date: '2026-04-07', amount: -11231 },
|
|
]),
|
|
).toEqual(['eb_SE0000000000000000000000_2026-04-07_-1123100_0'])
|
|
})
|
|
})
|
|
|
|
describe('contentBucketKey', () => {
|
|
it('keys off (date, öre) only — no description', () => {
|
|
expect(contentBucketKey('2024-06-15', -250)).toBe('2024-06-15|-25000')
|
|
})
|
|
|
|
it('matches a JS number against a PostgREST numeric string for the same amount', () => {
|
|
// A DB-fetched numeric string and a raw JS number for the same amount must
|
|
// land in the SAME bucket, otherwise the bridge silently misses.
|
|
expect(contentBucketKey('2024-06-15', -250)).toBe(contentBucketKey('2024-06-15', '-250.00'))
|
|
})
|
|
|
|
it('separates distinct amounts and dates into distinct buckets', () => {
|
|
expect(contentBucketKey('2024-06-15', -250)).not.toBe(contentBucketKey('2024-06-15', -100))
|
|
expect(contentBucketKey('2024-06-15', -250)).not.toBe(contentBucketKey('2024-06-16', -250))
|
|
})
|
|
})
|
|
|
|
describe('descriptionsBridge', () => {
|
|
it('bridges prefix-preserving PSD2 enrichment (the June 2026 drift)', () => {
|
|
// The same transaction whose title grew between syncs must still bridge.
|
|
// (Synthetic stand-ins for the real prefix-preserving enrichment pattern.)
|
|
expect(descriptionsBridge('KAFFE', 'KAFFE BG 0000000000 Bg-bet. via internet')).toBe(true)
|
|
expect(descriptionsBridge('UTBETALNING Insättning', 'UTBETALNING')).toBe(true)
|
|
expect(descriptionsBridge('REF 000000 Europabetalning', 'REF 000000')).toBe(true)
|
|
})
|
|
|
|
it('is case- and whitespace-insensitive', () => {
|
|
expect(descriptionsBridge(' Mataffär Solna ', 'mataffär solna')).toBe(true)
|
|
})
|
|
|
|
it('does NOT bridge genuinely distinct descriptions sharing a date+amount', () => {
|
|
// Distinct reference codes on same-day same-amount rows (e.g. verification
|
|
// micro-deposits) must NOT collapse — each is a real transaction.
|
|
expect(descriptionsBridge('REF-AAAA1111', 'REF-BBBB2222')).toBe(false)
|
|
// Same common stem, diverging tails — still distinct.
|
|
expect(descriptionsBridge('PMT.Ref AAA', 'PMT.Ref BBB')).toBe(false)
|
|
expect(descriptionsBridge('Coffee', 'Lunch')).toBe(false)
|
|
})
|
|
|
|
it('does not let a blank description wildcard-match a described row', () => {
|
|
// A blank carries no signal: it must not consume a described same-(date,öre)
|
|
// row. Live callers normalize blanks to FALLBACK_DESCRIPTION, so this is
|
|
// defense-in-depth. Only two blanks bridge each other (date+öre identity).
|
|
expect(descriptionsBridge('', 'anything')).toBe(false)
|
|
expect(descriptionsBridge('anything', null)).toBe(false)
|
|
expect(descriptionsBridge(undefined, '')).toBe(true)
|
|
})
|
|
})
|
|
|
|
describe('normalizeImportedDescription', () => {
|
|
it('maps empty / whitespace-only titles to the Swedish neutral', () => {
|
|
expect(normalizeImportedDescription('')).toBe(FALLBACK_DESCRIPTION)
|
|
expect(normalizeImportedDescription(' ')).toBe(FALLBACK_DESCRIPTION)
|
|
expect(normalizeImportedDescription(null)).toBe(FALLBACK_DESCRIPTION)
|
|
expect(normalizeImportedDescription(undefined)).toBe(FALLBACK_DESCRIPTION)
|
|
})
|
|
|
|
it('maps the legacy English "Unknown" sentinel to the Swedish neutral (case-insensitive)', () => {
|
|
expect(normalizeImportedDescription('Unknown')).toBe(FALLBACK_DESCRIPTION)
|
|
expect(normalizeImportedDescription('unknown')).toBe(FALLBACK_DESCRIPTION)
|
|
expect(normalizeImportedDescription(' UNKNOWN ')).toBe(FALLBACK_DESCRIPTION)
|
|
})
|
|
|
|
it('preserves a real title and trims surrounding whitespace', () => {
|
|
expect(normalizeImportedDescription('ICA Maxi Solna')).toBe('ICA Maxi Solna')
|
|
expect(normalizeImportedDescription(' Lön juni ')).toBe('Lön juni')
|
|
})
|
|
|
|
it('does NOT clobber a real title that merely contains the word "unknown"', () => {
|
|
expect(normalizeImportedDescription('Unknown Pizza AB')).toBe('Unknown Pizza AB')
|
|
})
|
|
})
|