Mcp/template data feedback (#617)
* fix(booking-templates): scope template list to the active company GET /api/settings/booking-templates relied solely on the btl_select RLS policy, which is membership-wide (user_company_ids) and returns templates from every company the user belongs to. A user who owns multiple companies saw all their templates merged regardless of which company was active. Narrow the list in the API layer (mirroring counterparty-templates) to system + the active company + the active company's team. RLS stays the security backstop; this fixes the cross-company merge within a single user's own view (it was never a cross-tenant data leak). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(import): show proper message for duplicate bank file upload The bank file import page mis-parsed the structured error envelope ({ error: { code, message, details } }), so a BANK_FILE_DUPLICATE (409) fell through to the generic "Kunde inte läsa filen" fallback. The upload step also hardcoded that same string as the error heading, so duplicates were doubly misreported as parse failures. - Parse the structured envelope by error.code; surface error.message for all codes instead of rendering the error object. - Add a dedicated BANK_FILE_DUPLICATE message using the importedAt / importedCount details the route already returns. - Add an optional errorTitle prop to BankFileUploadStep (defaults to the previous text) and pass "Filen är redan importerad" for dupes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(tests): add comprehensive tests for recordateEntry, inbox-linking, and external-id handling - Implemented unit tests for recordateEntry in the bookkeeping module to validate various scenarios including date changes, non-posted entries, and fiscal period restrictions. - Created tests for inbox-linking status in pending operations to ensure correct handling of invoice inbox items and supplier invoices, addressing historical bugs related to status updates. - Added tests for external-id utilities to ensure consistent handling of monetary amounts and deduplication keys across different transaction sources. - Introduced new functions in external-id.ts for stable external ID generation and normalization of imported descriptions, enhancing transaction deduplication reliability. feat(migrations): add new database migrations for transaction handling - Created migration to exclude storno and correction vouchers from unmatched GL lines, ensuring accurate reconciliation. - Added a migration to preserve original bank transaction descriptions in a new immutable column, allowing for user edits while maintaining audit trails and deduplication integrity. * feat(migrations): add function to exclude storno/correction vouchers from unmatched GL lines * feat(transactions): enhance transaction handling with improved description normalization and preloaded original entries --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Mattsson
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
2c59c3633f
commit
c6c86cded4
@@ -0,0 +1,144 @@
|
||||
import { describe, it, expect } from 'vitest'
|
||||
import {
|
||||
amountToOre,
|
||||
buildStableExternalIds,
|
||||
contentDedupKey,
|
||||
normalizeImportedDescription,
|
||||
FALLBACK_DESCRIPTION,
|
||||
} from '../external-id'
|
||||
|
||||
describe('amountToOre', () => {
|
||||
it('normalizes a JS number to integer öre', () => {
|
||||
expect(amountToOre(1234.5)).toBe(123450)
|
||||
expect(amountToOre(-250)).toBe(-25000)
|
||||
expect(amountToOre(0)).toBe(0)
|
||||
})
|
||||
|
||||
it('normalizes a numeric string (PostgREST representation) to the same öre', () => {
|
||||
// The core fix: a DB-fetched numeric string and a raw JS number for the
|
||||
// same amount must collapse to the same integer.
|
||||
expect(amountToOre('1234.50')).toBe(123450)
|
||||
expect(amountToOre('1234.5')).toBe(amountToOre(1234.5))
|
||||
expect(amountToOre('-250.00')).toBe(amountToOre(-250))
|
||||
expect(amountToOre('100')).toBe(10000)
|
||||
})
|
||||
|
||||
it('rounds sub-öre noise deterministically (never toFixed)', () => {
|
||||
expect(amountToOre(0.1 + 0.2)).toBe(30) // 0.30000000000000004 → 30
|
||||
expect(amountToOre(19.995)).toBe(2000)
|
||||
})
|
||||
})
|
||||
|
||||
describe('buildStableExternalIds', () => {
|
||||
it('derives the id from account + date + öre, not from any bank id', () => {
|
||||
const ids = buildStableExternalIds('eb', 'SE123', [{ date: '2024-06-15', amount: -500 }])
|
||||
expect(ids).toEqual(['eb_SE123_2024-06-15_-50000_0'])
|
||||
})
|
||||
|
||||
it('disambiguates genuinely identical transactions with an occurrence index', () => {
|
||||
const ids = buildStableExternalIds('eb', 'acc', [
|
||||
{ date: '2024-06-15', amount: -250 },
|
||||
{ date: '2024-06-15', amount: -250 },
|
||||
{ date: '2024-06-15', amount: -250 },
|
||||
])
|
||||
expect(ids).toEqual([
|
||||
'eb_acc_2024-06-15_-25000_0',
|
||||
'eb_acc_2024-06-15_-25000_1',
|
||||
'eb_acc_2024-06-15_-25000_2',
|
||||
])
|
||||
})
|
||||
|
||||
it('produces the SAME set of ids regardless of provider ordering (re-sync dedupe)', () => {
|
||||
const a = buildStableExternalIds('eb', 'acc', [
|
||||
{ date: '2024-06-15', amount: -250 },
|
||||
{ date: '2024-06-16', amount: -100 },
|
||||
{ date: '2024-06-15', amount: -250 },
|
||||
])
|
||||
// Same transactions, different order on a later sync.
|
||||
const b = buildStableExternalIds('eb', 'acc', [
|
||||
{ date: '2024-06-15', amount: -250 },
|
||||
{ date: '2024-06-15', amount: -250 },
|
||||
{ date: '2024-06-16', amount: -100 },
|
||||
])
|
||||
expect(new Set(a)).toEqual(new Set(b))
|
||||
})
|
||||
|
||||
it('treats string and number amounts as the same id (provider type drift)', () => {
|
||||
const num = buildStableExternalIds('eb', 'acc', [{ date: '2024-06-15', amount: 1234.5 }])
|
||||
const str = buildStableExternalIds('eb', 'acc', [{ date: '2024-06-15', amount: '1234.50' }])
|
||||
expect(num).toEqual(str)
|
||||
})
|
||||
|
||||
it('keeps distinct amounts and dates on separate occurrence counters', () => {
|
||||
const ids = buildStableExternalIds('eb', 'acc', [
|
||||
{ date: '2024-06-15', amount: -250 },
|
||||
{ date: '2024-06-15', amount: -100 },
|
||||
{ date: '2024-06-16', amount: -250 },
|
||||
{ date: '2024-06-15', amount: -250 },
|
||||
])
|
||||
expect(ids).toEqual([
|
||||
'eb_acc_2024-06-15_-25000_0',
|
||||
'eb_acc_2024-06-15_-10000_0',
|
||||
'eb_acc_2024-06-16_-25000_0',
|
||||
'eb_acc_2024-06-15_-25000_1',
|
||||
])
|
||||
})
|
||||
|
||||
it('returns an empty array for an empty batch', () => {
|
||||
expect(buildStableExternalIds('eb', 'acc', [])).toEqual([])
|
||||
})
|
||||
})
|
||||
|
||||
describe('contentDedupKey', () => {
|
||||
it('matches a JS number against a PostgREST numeric string for the same amount', () => {
|
||||
// The core dedup-bridge fix: an incoming raw number and a DB-fetched string
|
||||
// for the same amount + date + description must produce the SAME key.
|
||||
const incoming = contentDedupKey('2024-06-15', -250, 'ICA Maxi Solna')
|
||||
const stored = contentDedupKey('2024-06-15', '-250.00', 'ICA Maxi Solna')
|
||||
expect(incoming).toBe(stored)
|
||||
})
|
||||
|
||||
it('normalizes description (lowercase, trim, 24-char prefix)', () => {
|
||||
expect(contentDedupKey('2024-06-15', -100, ' ICA Maxi Solna '))
|
||||
.toBe(contentDedupKey('2024-06-15', -100, 'ica maxi solna'))
|
||||
// Differs only past the 24-char prefix → same key.
|
||||
expect(contentDedupKey('2024-06-15', -100, 'Betalning till leverantör AAA'))
|
||||
.toBe(contentDedupKey('2024-06-15', -100, 'Betalning till leverantör BBB'))
|
||||
})
|
||||
|
||||
it('keeps distinct transactions apart when description differs in the prefix', () => {
|
||||
expect(contentDedupKey('2024-06-15', -250, 'ICA Maxi'))
|
||||
.not.toBe(contentDedupKey('2024-06-15', -250, 'Coop Stockholm'))
|
||||
})
|
||||
|
||||
it('treats a null/undefined description as an empty prefix', () => {
|
||||
expect(contentDedupKey('2024-06-15', -100, null))
|
||||
.toBe(contentDedupKey('2024-06-15', -100, undefined))
|
||||
expect(contentDedupKey('2024-06-15', -100, null))
|
||||
.toBe(contentDedupKey('2024-06-15', -100, ''))
|
||||
})
|
||||
})
|
||||
|
||||
describe('normalizeImportedDescription', () => {
|
||||
it('maps empty / whitespace-only titles to the Swedish neutral', () => {
|
||||
expect(normalizeImportedDescription('')).toBe(FALLBACK_DESCRIPTION)
|
||||
expect(normalizeImportedDescription(' ')).toBe(FALLBACK_DESCRIPTION)
|
||||
expect(normalizeImportedDescription(null)).toBe(FALLBACK_DESCRIPTION)
|
||||
expect(normalizeImportedDescription(undefined)).toBe(FALLBACK_DESCRIPTION)
|
||||
})
|
||||
|
||||
it('maps the legacy English "Unknown" sentinel to the Swedish neutral (case-insensitive)', () => {
|
||||
expect(normalizeImportedDescription('Unknown')).toBe(FALLBACK_DESCRIPTION)
|
||||
expect(normalizeImportedDescription('unknown')).toBe(FALLBACK_DESCRIPTION)
|
||||
expect(normalizeImportedDescription(' UNKNOWN ')).toBe(FALLBACK_DESCRIPTION)
|
||||
})
|
||||
|
||||
it('preserves a real title and trims surrounding whitespace', () => {
|
||||
expect(normalizeImportedDescription('ICA Maxi Solna')).toBe('ICA Maxi Solna')
|
||||
expect(normalizeImportedDescription(' Lön juni ')).toBe('Lön juni')
|
||||
})
|
||||
|
||||
it('does NOT clobber a real title that merely contains the word "unknown"', () => {
|
||||
expect(normalizeImportedDescription('Unknown Pizza AB')).toBe('Unknown Pizza AB')
|
||||
})
|
||||
})
|
||||
@@ -211,6 +211,85 @@ describe('ingestTransactions', () => {
|
||||
expect(result.transaction_ids).toEqual([])
|
||||
})
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// 2b-edit. Edit-safety regression: a user-edited stored title must NOT
|
||||
// reopen the duplicate-import window. The content bridge keys off the
|
||||
// immutable original_description, so a re-import whose bank text still
|
||||
// matches the original is deduped even though the stored (editable)
|
||||
// description was changed.
|
||||
// -----------------------------------------------------------------------
|
||||
it('dedupes against the original bank description even after the stored title was edited', async () => {
|
||||
const { supabase, enqueue } = createQueueMockSupabase()
|
||||
const raw = makeRaw({
|
||||
date: '2024-06-15',
|
||||
amount: -250.0,
|
||||
description: 'ICA Maxi Solna', // original bank text, re-imported via CSV
|
||||
external_id: 'lunar_csvhash999', // different external_id → primary dedup misses
|
||||
import_source: 'csv_lunar',
|
||||
})
|
||||
|
||||
// Booked transaction map query — none
|
||||
enqueue({ data: [], error: null })
|
||||
// Unbooked bank-synced row whose TITLE was edited by the user, but whose
|
||||
// original_description still holds the bank's verbatim text.
|
||||
enqueue({
|
||||
data: [
|
||||
{
|
||||
date: '2024-06-15',
|
||||
amount: -250.0,
|
||||
original_description: 'ICA Maxi Solna',
|
||||
description: 'Mataffär (egen rubrik)',
|
||||
},
|
||||
],
|
||||
error: null,
|
||||
})
|
||||
// Supplier invoices fetch
|
||||
enqueue({ data: [], error: null })
|
||||
// Batch external_id dedup query — external_id differs, so no match
|
||||
enqueue({ data: [], error: null })
|
||||
|
||||
const result = await ingestTransactions(supabase as never, COMPANY_ID, USER_ID, [raw])
|
||||
|
||||
expect(result.duplicates).toBe(1)
|
||||
expect(result.imported).toBe(0)
|
||||
})
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// 2b-unknown. Legacy 'Unknown'/empty rows must still dedup: the stored-side
|
||||
// content key is normalized the same way as the incoming side, so an
|
||||
// existing row whose original_description is the legacy 'Unknown'
|
||||
// sentinel matches an incoming 'Unknown' re-import (both → 'Okänd
|
||||
// transaktion'). Without symmetric normalization this row would
|
||||
// re-import as a duplicate.
|
||||
// -----------------------------------------------------------------------
|
||||
it('dedupes legacy "Unknown" rows by normalizing both the stored and incoming keys', async () => {
|
||||
const { supabase, enqueue } = createQueueMockSupabase()
|
||||
const raw = makeRaw({
|
||||
date: '2024-06-15',
|
||||
amount: -250.0,
|
||||
description: 'Unknown', // legacy English sentinel re-imported via CSV
|
||||
external_id: 'lunar_csvhashU',
|
||||
import_source: 'csv_lunar',
|
||||
})
|
||||
|
||||
// Booked transaction map query — none
|
||||
enqueue({ data: [], error: null })
|
||||
// Unbooked bank-synced row whose original_description is the legacy sentinel.
|
||||
enqueue({
|
||||
data: [{ date: '2024-06-15', amount: -250.0, original_description: 'Unknown', description: 'Unknown' }],
|
||||
error: null,
|
||||
})
|
||||
// Supplier invoices fetch
|
||||
enqueue({ data: [], error: null })
|
||||
// Batch external_id dedup query — external_id differs, so no match
|
||||
enqueue({ data: [], error: null })
|
||||
|
||||
const result = await ingestTransactions(supabase as never, COMPANY_ID, USER_ID, [raw])
|
||||
|
||||
expect(result.duplicates).toBe(1)
|
||||
expect(result.imported).toBe(0)
|
||||
})
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// 2c. No false positive: same date+amount but different description does
|
||||
// NOT trigger content dedup — guards against the historical concern
|
||||
@@ -791,6 +870,72 @@ describe('ingestTransactions', () => {
|
||||
expect(result.duplicates).toBe(0)
|
||||
})
|
||||
|
||||
it('bridges the external_id scheme change: a booked OLD-scheme eb_ row is caught by content dedup on re-sync', async () => {
|
||||
// Transition scenario: an enable_banking row was imported+booked under the
|
||||
// OLD unstable scheme (eb_{iban}_{txid}). After deploy, the re-sync derives
|
||||
// a NEW content-based external_id that will NOT match by external_id, so
|
||||
// layer-1 misses. Layer 1b (booked content dedup) MUST catch it, otherwise
|
||||
// the user sees the exact duplicate the fix is meant to prevent.
|
||||
const { supabase, enqueue } = createQueueMockSupabase()
|
||||
const raw = makeRaw({
|
||||
external_id: 'eb_SE123_2024-06-15_-25000_0', // new scheme
|
||||
date: '2024-06-15',
|
||||
amount: -250,
|
||||
description: 'ICA Maxi Solna',
|
||||
import_source: 'enable_banking',
|
||||
})
|
||||
|
||||
// Booked map: the SAME transaction still carries its OLD-scheme external_id
|
||||
// in the DB; dedup matches on content, not on external_id.
|
||||
enqueue({
|
||||
data: [{ date: '2024-06-15', amount: -250, description: 'ICA Maxi Solna' }],
|
||||
error: null,
|
||||
})
|
||||
// Unbooked enable_banking map — none
|
||||
enqueue({ data: [], error: null })
|
||||
// Supplier invoices fetch
|
||||
enqueue({ data: [], error: null })
|
||||
// Batch external_id dedup query — DB still holds eb_SE123_{old_txid}, so the
|
||||
// new external_id finds NO match here.
|
||||
enqueue({ data: [], error: null })
|
||||
|
||||
const result = await ingestTransactions(supabase as never, COMPANY_ID, USER_ID, [raw])
|
||||
|
||||
expect(result.duplicates).toBe(1)
|
||||
expect(result.imported).toBe(0)
|
||||
})
|
||||
|
||||
it('dedupes against a booked row whose amount is a numeric string (PostgREST), not a number', async () => {
|
||||
// Regression: PostgREST can serialize a `numeric` column as a string
|
||||
// ("-250.00") while the incoming raw amount is a JS number (-250). Before
|
||||
// the öre-normalized dedup key these never compared equal, so content dedup
|
||||
// silently missed and the row was re-imported as a duplicate.
|
||||
const { supabase, enqueue } = createQueueMockSupabase()
|
||||
const raw = makeRaw({
|
||||
external_id: 'eb_acc_2024-06-15_-25000_0',
|
||||
date: '2024-06-15',
|
||||
amount: -250,
|
||||
description: 'ICA Maxi Solna',
|
||||
})
|
||||
|
||||
// Booked map: same transaction, amount as a STRING with trailing zeros.
|
||||
enqueue({
|
||||
data: [{ date: '2024-06-15', amount: '-250.00', description: 'ICA Maxi Solna' }],
|
||||
error: null,
|
||||
})
|
||||
// Unbooked bank-synced transaction map query
|
||||
enqueue({ data: [], error: null })
|
||||
// Supplier invoices fetch
|
||||
enqueue({ data: [], error: null })
|
||||
// Batch external_id dedup query (no match by external_id — the id scheme changed)
|
||||
enqueue({ data: [], error: null })
|
||||
|
||||
const result = await ingestTransactions(supabase as never, COMPANY_ID, USER_ID, [raw])
|
||||
|
||||
expect(result.duplicates).toBe(1)
|
||||
expect(result.imported).toBe(0)
|
||||
})
|
||||
|
||||
it('handles multiple booked transactions with same date+amount correctly', async () => {
|
||||
const { supabase, enqueue } = createQueueMockSupabase()
|
||||
|
||||
|
||||
Reference in New Issue
Block a user