Files
accounted/tests/pg/transactions-transaction-method.pg.test.ts
Jakob Wennberg 70845edf69 feat(transactions): structured transaction_method instead of channel-in-the-name (#1459)
* feat(transactions): structured transaction_method instead of channel-in-the-name

Swedish bank feeds embed the payment channel in the description string
("Vercel Jul Överföring via internet", "ANTHROPIC* ... Kortköp/uttag"):
the PSD2 remittance array is joined into one string and the ISO 20022
type codes were dropped at insert. This promotes the channel to data:

- transactions.transaction_method (text + CHECK closed vocabulary: card,
  transfer, bankgiro, plusgiro, swish, autogiro, e_invoice, international,
  deposit, withdrawal, salary, fee, interest, adjustment) plus verbatim
  bank_transaction_code / proprietary_bank_transaction_code evidence
  columns (data_quality_master Appendix B "Layer-A capture").
- classifyTransactionMethod() in lib/transactions/transaction-method.ts:
  explicit source method (Stripe txn.type) > trailing Swedish channel
  phrase > ISO 20022 family/subfamily > proprietary-code keywords > MCC.
  It also splits the clean display title off the description.
- Ingest stores the clean title as description and the full bank string
  as original_description; dedup is untouched (external_id is date+öre,
  the content bridge reads original_description and is prefix-based, and
  a trailing strip leaves a prefix). Enable Banking passes the codes
  through; the Stripe feed sets methods from its balance-txn types.
- Backfill migration classifies existing rows from the description text
  (+ MCC and Stripe prefixes) and strips unedited titles; user-edited
  titles are never rewritten.
- mapping-engine also matches original_description so user rules written
  against the full bank text keep firing.
- UI: the inbox row shows the clean name; clicking it now folds out
  "Betalsätt: Kortköp" etc. (sv/en), making every classified row
  expandable.

A card purchase implies a physical receipt, a Bankgiro/e-invoice payment
implies a supplier invoice: downstream automations can now branch on the
rail instead of regexing display strings.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(transactions): anchor counterparty-template identity on original_description

Audit follow-up to the phrase-strip change: counterparty template lookup
AND learning derived their key from merchant_name || description. With
the working title now stripped ("SPOTIFY AB Kortköp" -> "SPOTIFY AB"),
templates learned from the full bank string would only re-match via the
occurrence-gated single-token tier, and single-token counterparties with
fewer than 3 bookings would silently stop matching.

Both sides now read merchant_name || original_description || description:
the immutable bank original is identical across eras (and across user
renames), so every stored key and alias keeps matching exactly. Same
anchoring rationale as buildMerchantHistory in category-suggestions.

Existing tests that relied on the fixture's default original_description
now state it explicitly; two new regression tests pin the era stability
(lookup via alias on the full string, learning key derivation).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(transactions): review follow-ups on method classification

- methodFromCodes: two-pass subfamily-then-family scan so a SALA/XBCT
  refinement on the proprietary code beats a bare family match on the
  ISO code, matching the documented precedence; pinned by a test.
- mapping-engine: regression tests for merchant/description patterns
  that only match original_description, including the invalid-regex
  substring fallback and the no-match default.
- Stripe: regression test for the SDK-unmodeled 'tax' balance-txn type
  mapping to 'fee'.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(transactions): scope method classification to feed rows + adjective guard

Multi-bank risk hardening before the backfill ships:

- Feed-row scope: classification and title stripping now require a real
  import feed (import_source present, not manual/mcp), both at the
  ingest boundary (USER_CREATED_IMPORT_SOURCES, now exported) and in
  every backfill statement. User-authored titles like "Egen insättning"
  on manual/MCP rows are never classified and never rewritten.
- Adjective guard (TS + SQL): a strip that would leave the title ending
  in a possessive/scope adjective (egen/eget/privat/intern/extern ...)
  is skipped, so "Egen insättning" stays whole even on bank-feed rows;
  the method column still classifies (deposit).
- Unknown bank phrasings remain untouched by construction: an unmatched
  phrase means no method and no rewrite, so the worst case for any bank
  whose vocabulary we have not seen is the status quo.

Pinned by new unit + pg-real cases (user-created exclusion for
NULL/manual/mcp, adjective guard, feed defaults in the pg fixture).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore(migrations): re-timestamp transaction_method migrations after rebase

Main gained migrations dated 20260729-20260730 (already applied to prod)
while this branch carried 20260728 versions, which would have applied
out-of-order on merge. The files have never reached prod, so renaming to
current timestamps is safe and removes any dependence on the integration's
out-of-order handling. All code/doc references updated; the pg test reads
the backfill by its new filename.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore(migrations): bump transaction_method versions past prod's max

Main's newest applied migration is 20260730090000 (future-leaning
timestamp), so the previous 202607300731xx rename still sorted before
prod's tail and risked a silent skip on merge-time apply. Versions are
now 20260730100000/20260730100100, strictly after everything applied to
prod. References updated; full migration stream replays clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(transactions): final review round: keyboard guard + bank_connection_id feed marker

- TransactionInboxCard: row-level Enter/Space handling now ignores events
  bubbling from nested controls, so keyboard activation of Bokför / the
  overflow menu is no longer cancelled by the (now much more common)
  expandable row.
- Feed predicate parity with isImportedTransaction(): a live
  bank_connection_id marks a feed row even when import_source is unset
  (the oldest PSD2 rows predate that column), in both the ingest
  classifier and every backfill statement: those legacy rows now get
  classified instead of being skipped as user-created.
- pg fixture typing uses the TransactionMethod union.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore(migrations): re-timestamp transaction_method migrations past prod's 20260807 tail

Prod max applied is 20260807170000 (verified by name via list_migrations);
the 20260730-stamped pair would sort before it. References in code,
tests, and DECISIONS.md updated to the new versions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(migrations): enforce, not assume, original_description preservation in the title strip

The strip UPDATE now fills a NULL original_description from the
pre-strip description in the same statement. Prod has zero such rows
(0/25,566 feed-scope rows, verified read-only), and 20260605120000's
backfill plus ingest make the NULL case unreachable on any DB that
replayed history, but the migration should not depend on that history
to avoid losing the only copy of a bank string.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: record the compliance-review triage of the backfill's booked-row title strip

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Jakob Wennberg <jakob.wennberg@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com>
2026-08-08 11:58:51 +02:00

312 lines
11 KiB
TypeScript

import { readFileSync } from 'node:fs'
import { join } from 'node:path'
import { describe, expect, it } from 'vitest'
import type { TransactionMethod } from '@/types'
import { seedCompany, insertTransaction } from '@/tests/pg/fixtures'
import { getPool } from '@/tests/pg/setup'
/**
* pg-real coverage for transactions.transaction_method
* (20260808090000_transactions_transaction_method.sql + the paired backfill
* 20260808090100_..._backfill.sql).
*
* Locks in:
* - The closed-vocabulary CHECK constraint.
* - The backfill's trailing-phrase classification (real production strings),
* its precedence (word-boundary: "Löneinsättning" is salary, never
* deposit), the MCC and Stripe-prefix fallbacks, and the NULL result for
* unclassifiable rows.
* - FEED-ROW SCOPE: user-created rows (import_source NULL/manual/mcp) are
* never classified and never rewritten ("Egen insättning" stays intact).
* - Title stripping: unedited titles lose the trailing channel phrase but
* are never emptied; user-edited titles are untouched even when the
* original carries a phrase (classification still fires off the original);
* the adjective guard keeps "Egen insättning" whole even on feed rows.
* - Idempotence: a second run changes nothing.
*/
// Run the real backfill migration SQL so the test exercises exactly what
// ships, not a re-implementation.
const BACKFILL_SQL = readFileSync(
join(
process.cwd(),
'supabase/migrations/20260808090100_transactions_transaction_method_backfill.sql',
),
'utf8',
)
async function runBackfill(): Promise<void> {
await getPool().query(BACKFILL_SQL)
}
async function insertFeedRow(params: {
companyId: string
userId: string
description: string
originalDescription?: string | null
titleEditedAt?: string | null
/** Defaults to a bank feed; pass null/'manual'/'mcp' for user-created rows. */
importSource?: string | null
mccCode?: number | null
}): Promise<string> {
const id = await insertTransaction({
companyId: params.companyId,
userId: params.userId,
description: params.description,
})
await getPool().query(
`UPDATE public.transactions
SET original_description = $2,
title_edited_at = $3::timestamptz,
import_source = $4,
mcc_code = $5
WHERE id = $1`,
[
id,
params.originalDescription === undefined ? params.description : params.originalDescription,
params.titleEditedAt ?? null,
params.importSource === undefined ? 'enable_banking' : params.importSource,
params.mccCode ?? null,
],
)
return id
}
async function getRow(
txId: string,
): Promise<{ description: string; transaction_method: TransactionMethod | null }> {
const { rows } = await getPool().query(
`SELECT description, transaction_method FROM public.transactions WHERE id = $1`,
[txId],
)
return rows[0]
}
describe('transactions.transaction_method: CHECK constraint', () => {
it('accepts vocabulary values and NULL, rejects anything else', async () => {
const { userId, companyId } = await seedCompany()
const tx = await insertTransaction({ companyId, userId })
await getPool().query(
`UPDATE public.transactions SET transaction_method = 'card' WHERE id = $1`,
[tx],
)
await getPool().query(
`UPDATE public.transactions SET transaction_method = NULL WHERE id = $1`,
[tx],
)
await expect(
getPool().query(
`UPDATE public.transactions SET transaction_method = 'bankid' WHERE id = $1`,
[tx],
),
).rejects.toMatchObject({ code: '23514' })
})
})
describe('transactions.transaction_method: backfill classification + title strip', () => {
it('classifies the real production strings and strips the trailing phrase', async () => {
const { userId, companyId } = await seedCompany()
const cases: Array<{
description: string
method: TransactionMethod
stripped: string
}> = [
{
description: 'Vercel Jul Överföring via internet',
method: 'transfer',
stripped: 'Vercel Jul',
},
{
description: 'ANTHROPIC* CLAUDE SUB SAN FRANCISCO Kortköp/uttag',
method: 'card',
stripped: 'ANTHROPIC* CLAUDE SUB SAN FRANCISCO',
},
{
description: 'Inbetalning skat BG 0000050501055 Bg-bet. via internet',
method: 'bankgiro',
stripped: 'Inbetalning skat BG 0000050501055',
},
{ description: '1260624917587 Europabetalning', method: 'international', stripped: '1260624917587' },
{ description: '1260624917587 Pris betalning', method: 'fee', stripped: '1260624917587' },
{ description: 'SWED2607270AUEOU Insättning', method: 'deposit', stripped: 'SWED2607270AUEOU' },
{ description: 'Lön Juli Emil Överföring via internet', method: 'transfer', stripped: 'Lön Juli Emil' },
{ description: 'ACME AB Löneinsättning', method: 'salary', stripped: 'ACME AB' },
{ description: 'Swish till Erik Andersson', method: 'swish', stripped: 'Swish till Erik Andersson' },
]
const ids = []
for (const c of cases) {
ids.push(await insertFeedRow({ companyId, userId, description: c.description }))
}
await runBackfill()
for (let i = 0; i < cases.length; i++) {
const row = await getRow(ids[i])
expect(row.transaction_method).toBe(cases[i].method)
expect(row.description).toBe(cases[i].stripped)
}
})
it('never touches user-created rows (import_source NULL, manual, mcp)', async () => {
const { userId, companyId } = await seedCompany()
const cases = [null, 'manual', 'mcp']
const ids: string[] = []
for (const src of cases) {
// Titles that WOULD classify+strip if they came from a bank feed.
ids.push(
await insertFeedRow({ companyId, userId, description: 'Egen insättning', importSource: src }),
)
}
await runBackfill()
for (const id of ids) {
const row = await getRow(id)
expect(row.transaction_method).toBeNull()
expect(row.description).toBe('Egen insättning')
}
})
it('adjective guard: feed rows classify but keep "Egen insättning"-style titles', async () => {
const { userId, companyId } = await seedCompany()
const deposit = await insertFeedRow({ companyId, userId, description: 'Egen insättning' })
const withdrawal = await insertFeedRow({ companyId, userId, description: 'Eget uttag' })
const transfer = await insertFeedRow({ companyId, userId, description: 'Intern överföring' })
await runBackfill()
expect(await getRow(deposit)).toEqual({
description: 'Egen insättning',
transaction_method: 'deposit',
})
expect(await getRow(withdrawal)).toEqual({
description: 'Eget uttag',
transaction_method: 'withdrawal',
})
expect(await getRow(transfer)).toEqual({
description: 'Intern överföring',
transaction_method: 'transfer',
})
})
it('never empties a title that IS the phrase', async () => {
const { userId, companyId } = await seedCompany()
const tx = await insertFeedRow({ companyId, userId, description: 'Insättning' })
await runBackfill()
const row = await getRow(tx)
expect(row.transaction_method).toBe('deposit')
expect(row.description).toBe('Insättning')
})
it('classifies an edited row from the original but leaves the edited title alone', async () => {
const { userId, companyId } = await seedCompany()
const tx = await insertFeedRow({
companyId,
userId,
description: 'Serverhyra juli Överföring via internet',
originalDescription: 'Hetzner GmbH Överföring via internet',
titleEditedAt: '2026-07-01T10:00:00Z',
})
await runBackfill()
const row = await getRow(tx)
expect(row.transaction_method).toBe('transfer')
// User-edited titles are never rewritten, even when they end in a phrase.
expect(row.description).toBe('Serverhyra juli Överföring via internet')
})
it('falls back to the legacy description when original_description is NULL', async () => {
const { userId, companyId } = await seedCompany()
const tx = await insertFeedRow({
companyId,
userId,
description: 'TELIA AB Autogiro',
originalDescription: null,
})
await runBackfill()
const row = await getRow(tx)
expect(row.transaction_method).toBe('autogiro')
expect(row.description).toBe('TELIA AB')
// The strip must preserve the only copy of the full bank string: a legacy
// NULL original_description is filled from the pre-strip description in
// the same statement, never lost.
const { rows } = await getPool().query(
`SELECT original_description FROM public.transactions WHERE id = $1`,
[tx],
)
expect(rows[0].original_description).toBe('TELIA AB Autogiro')
})
it('uses MCC presence as the card-rail fallback (6011 = withdrawal)', async () => {
const { userId, companyId } = await seedCompany()
const card = await insertFeedRow({
companyId,
userId,
description: 'COOP KONSUM STOCKHOLM',
mccCode: 5411,
})
const atm = await insertFeedRow({
companyId,
userId,
description: 'BANKOMAT VASAGATAN',
mccCode: 6011,
})
await runBackfill()
expect((await getRow(card)).transaction_method).toBe('card')
expect((await getRow(atm)).transaction_method).toBe('withdrawal')
// No phrase → the title is untouched.
expect((await getRow(card)).description).toBe('COOP KONSUM STOCKHOLM')
})
it('classifies Stripe feed rows from their deterministic prefixes', async () => {
const { userId, companyId } = await seedCompany()
const mk = (description: string) =>
insertFeedRow({ companyId, userId, description, importSource: 'stripe' })
const fee = await mk('Stripe-avgift (Stripe-betalning Carl)')
const usage = await mk('Stripe: Billing - Usage Fee (2026-07-26)')
const charge = await mk('Stripe-betalning Fredrik Schöön')
const payout = await mk('Stripe-utbetalning po_1')
await runBackfill()
expect((await getRow(fee)).transaction_method).toBe('fee')
expect((await getRow(usage)).transaction_method).toBe('fee')
expect((await getRow(charge)).transaction_method).toBe('card')
expect((await getRow(payout)).transaction_method).toBe('transfer')
})
it('leaves unclassifiable rows NULL and untouched, and is idempotent', async () => {
const { userId, companyId } = await seedCompany()
const plain = await insertFeedRow({ companyId, userId, description: 'Okänd transaktion' })
const phrased = await insertFeedRow({
companyId,
userId,
description: 'Vercel Jul Överföring via internet',
})
// Pre-classified rows must not be reclassified.
const preset = await insertFeedRow({ companyId, userId, description: 'X Kortköp' })
await getPool().query(
`UPDATE public.transactions SET transaction_method = 'swish' WHERE id = $1`,
[preset],
)
await runBackfill()
const first = [await getRow(plain), await getRow(phrased), await getRow(preset)]
await runBackfill()
const second = [await getRow(plain), await getRow(phrased), await getRow(preset)]
expect(first[0]).toEqual({ description: 'Okänd transaktion', transaction_method: null })
expect(first[1]).toEqual({ description: 'Vercel Jul', transaction_method: 'transfer' })
expect(first[2].transaction_method).toBe('swish')
expect(second).toEqual(first)
})
})