Bug/year end numbers (#744)
* fix(bookkeeping): allow creating a fiscal year that fills an interior gap Fiscal-period creation only allowed chaining a new räkenskapsår before the earliest or after the latest existing period, so a company with a gap between years (e.g. 2024 + 2026 from an SIE import, missing 2025) could not create the missing year — it failed with "New period must chain before the earliest or after the latest existing period". Generalise forward chaining onto the new period's immediate predecessor, which covers both appending a new latest year and filling an interior gap. The "prior year must be locked" guard now applies only to true appends, not gap fills (a backfill, like backward chaining). previous_period_id is set to the predecessor and the successor is relinked so the BFNAR 2013:2 continuity chain stays intact. The create dialog suggests the missing year (capped so it never overlaps the next period), the settings page seeds the dialog at the earliest gap, and the default suggested name is now "Räkenskapsår <year>". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(bookkeeping): omföra föregående års resultat (2099 → 2098) at year-end Year-end closing posts the result to 2099 "Årets resultat" and the opening balance carried it forward on 2099 every year, so 2099 accumulated across years and the prior result never moved off "Årets resultat". executeYearEndClosing now posts a separate "Omföring av föregående års resultat" verifikat (Dr 2099 / Cr 2098 for a profit, reversed for a loss) into the new period after the continuity check passes, so 2099 starts each year at zero. Kept as a standalone entry rather than folded into the opening balance so the IB stays a faithful mirror of the prior UB and IB/UB continuity still holds. Aktiebolag only; idempotent; no-op when 2099 is flat. The 2098 → 2091/2898 disposition (bolagsstämma decision) is intentionally left to a separate step. - new source_type 'result_appropriation' (migration + type + Zod enum) - generateResultAppropriation helper (planner + poster) wired as step 11 - ResultStep surfaces the omföring voucher - unit tests + pg-real invariant - scripts/repair-result-appropriation.ts: retroactive catch-up (dry-run default) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(transactions): shadow-detect date-drift duplicate bank transactions The content-dedup bridge buckets on exact (date, ore), so the same transaction re-imported with a booking date that drifted a day lands in a different bucket and slips past every dedup layer. Add a measure-only ("shadow") detector that flags would-be +/-1-day duplicates and counts them, without changing what is inserted - so the gap can be validated on real data before any enforcement, mirroring the scope-drift shadow. - shiftIsoDate(): pure, deterministic adjacent-date helper - ingest: DEDUP_DATE_DRIFT_MODE flag (default on), pre-loop bucket snapshot, per-row gate with desc-bridge + cross-channel-symmetry signals; logs shadow_date_drift_candidates, never alters inserts - fail-safe date guard so the measurement can never abort an import - regression tests for both signals, account/window/distinct guards, no-double-count, and the malformed-date fail-safe Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(bookkeeping): anonymize a customer reference in fiscal-period tests Remove a real customer name ("AXMD AB") from regression-test comments; no logic change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(workflows): enhance Docker image scanning and caching mechanisms * fix(bookkeeping): enhance year-end result appropriation handling and error reporting --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Mattsson
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
5c40fa9aeb
commit
2a8bf9b42e
@@ -5,6 +5,7 @@ import {
|
||||
contentBucketKey,
|
||||
descriptionsBridge,
|
||||
normalizeImportedDescription,
|
||||
shiftIsoDate,
|
||||
FALLBACK_DESCRIPTION,
|
||||
} from '../external-id'
|
||||
|
||||
@@ -175,3 +176,24 @@ describe('normalizeImportedDescription', () => {
|
||||
expect(normalizeImportedDescription('Unknown Pizza AB')).toBe('Unknown Pizza AB')
|
||||
})
|
||||
})
|
||||
|
||||
describe('shiftIsoDate', () => {
|
||||
it('shifts a date forward and backward by whole days', () => {
|
||||
expect(shiftIsoDate('2024-06-15', 1)).toBe('2024-06-16')
|
||||
expect(shiftIsoDate('2024-06-15', -1)).toBe('2024-06-14')
|
||||
expect(shiftIsoDate('2024-06-15', 0)).toBe('2024-06-15')
|
||||
})
|
||||
|
||||
it('crosses month and year boundaries', () => {
|
||||
expect(shiftIsoDate('2024-06-30', 1)).toBe('2024-07-01')
|
||||
expect(shiftIsoDate('2024-07-01', -1)).toBe('2024-06-30')
|
||||
expect(shiftIsoDate('2025-12-31', 1)).toBe('2026-01-01')
|
||||
expect(shiftIsoDate('2026-01-01', -1)).toBe('2025-12-31')
|
||||
})
|
||||
|
||||
it('handles the leap day deterministically (no wall-clock dependency)', () => {
|
||||
expect(shiftIsoDate('2024-02-28', 1)).toBe('2024-02-29') // 2024 is a leap year
|
||||
expect(shiftIsoDate('2024-03-01', -1)).toBe('2024-02-29')
|
||||
expect(shiftIsoDate('2025-02-28', 1)).toBe('2025-03-01') // 2025 is not
|
||||
})
|
||||
})
|
||||
|
||||
@@ -405,7 +405,7 @@ describe('ingestTransactions', () => {
|
||||
// 2c-bis. Cross-channel mirror: the SAME bank account imported via two feeds
|
||||
// (Nordea CSV payee text vs PSD2 OCR/message) — same date+amount, one row
|
||||
// per channel, descriptions that do NOT bridge — IS deduped on
|
||||
// (date, öre). This is the AXMD/Axel case: a CSV import landing on top of
|
||||
// (date, öre). The real-world trigger: a CSV import landing on top of
|
||||
// existing Enable Banking rows whose descriptions share no text.
|
||||
// -----------------------------------------------------------------------
|
||||
it('dedupes a cross-channel mirror (CSV vs PSD2) even when descriptions do not bridge', async () => {
|
||||
@@ -547,7 +547,7 @@ describe('ingestTransactions', () => {
|
||||
enqueue({
|
||||
data: [{
|
||||
date: '2024-06-15', amount: -250,
|
||||
original_description: 'LAN AXMD 19', description: 'LAN AXMD 19',
|
||||
original_description: 'LOAN PAYMENT 19', description: 'LOAN PAYMENT 19',
|
||||
import_source: 'enable_banking', bank_connection_id: 'conn-1',
|
||||
cash_account_id: 'ca-1930', external_id: 'eb_SE_OLD_2024-06-15_-25000_0',
|
||||
}],
|
||||
@@ -825,6 +825,236 @@ describe('ingestTransactions', () => {
|
||||
expect(result.imported).toBe(0)
|
||||
})
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// 2h-shadow. Date-drift (the residual gap behind the reported bank↔bank dupes).
|
||||
// Every dedup layer buckets on EXACT (date, öre), so a twin whose booking
|
||||
// date drifted a day is invisible to all of them. The date-drift shadow
|
||||
// MEASURES how often a ±1-day rule would fire — it logs/counts but NEVER
|
||||
// changes what is inserted. These pin both that it detects the real cases
|
||||
// and, crucially, that it never flags a genuine row.
|
||||
// -----------------------------------------------------------------------
|
||||
it('shadow-flags an EB↔EB twin one day apart with a bridging description, but still imports it', async () => {
|
||||
const { supabase, enqueue } = createQueueMockSupabase()
|
||||
// Same hotel expense, booking date drifted 15→16 (a real date-drift case).
|
||||
const raw = makeRaw({
|
||||
date: '2024-06-16',
|
||||
amount: -1500,
|
||||
description: 'Hotel expense',
|
||||
external_id: 'eb_SE_2024-06-16_-150000_0',
|
||||
import_source: 'enable_banking',
|
||||
})
|
||||
const inserted = makeTransaction({ id: 'tx-drift', external_id: raw.external_id })
|
||||
|
||||
enqueue({ data: [], error: null }) // booked map — none
|
||||
// Unbooked EB twin one day earlier — same amount/desc/account, OLD-scheme id.
|
||||
enqueue({
|
||||
data: [{
|
||||
date: '2024-06-15', amount: -1500,
|
||||
original_description: 'Hotel expense', description: 'Hotel expense',
|
||||
import_source: 'enable_banking', bank_connection_id: 'conn-1',
|
||||
cash_account_id: 'ca-1930', external_id: 'eb_SE_2024-06-15_-150000_0',
|
||||
}],
|
||||
error: null,
|
||||
})
|
||||
enqueue({ data: [], error: null }) // supplier invoices
|
||||
enqueue({ data: [], error: null }) // external_id dedup — different date bucket, no match
|
||||
enqueue({ data: { id: 'ca-1930' }, error: null }) // cash_accounts — same account
|
||||
enqueue({ data: inserted, error: null }) // insert — STILL imported (shadow only logs)
|
||||
mockEvaluateMappingRules.mockResolvedValue(makeMappingResult({ confidence: 0.5 }))
|
||||
|
||||
const result = await ingestTransactions(supabase as never, COMPANY_ID, USER_ID, [raw], {
|
||||
settlementAccount: '1930',
|
||||
})
|
||||
|
||||
// Detected, but NOT acted on: imports exactly as before.
|
||||
expect(result.imported).toBe(1)
|
||||
expect(result.duplicates).toBe(0)
|
||||
expect(result.shadow_date_drift_candidates).toBe(1)
|
||||
// A different (adjacent) bucket → this is date-drift, not same-bucket scope-drift.
|
||||
expect(result.shadow_scope_drift_candidates).toBe(0)
|
||||
})
|
||||
|
||||
it('shadow-flags a CSV↔EB twin one day apart via cross-channel symmetry when descriptions do not bridge', async () => {
|
||||
const { supabase, enqueue } = createQueueMockSupabase()
|
||||
// Nordea CSV row (payee-only desc) and its PSD2 twin booked a day later
|
||||
// (OCR/message desc) — descriptions share no prefix, so only the
|
||||
// cross-channel mirror DISPLACED by a day can catch it (a real date-drift case).
|
||||
const raw = makeRaw({
|
||||
date: '2024-06-15',
|
||||
amount: -2500,
|
||||
description: 'Nordea',
|
||||
external_id: 'nordea_business_csvhash',
|
||||
import_source: 'csv_nordea_business',
|
||||
})
|
||||
const inserted = makeTransaction({ id: 'tx-cross', external_id: raw.external_id })
|
||||
|
||||
enqueue({ data: [], error: null }) // booked map — none
|
||||
enqueue({
|
||||
data: [{
|
||||
date: '2024-06-16', amount: -2500,
|
||||
original_description: 'Reimbursement', description: 'Reimbursement',
|
||||
import_source: 'enable_banking', bank_connection_id: 'conn-1',
|
||||
cash_account_id: null, external_id: 'eb_SE_2024-06-16_-250000_0',
|
||||
}],
|
||||
error: null,
|
||||
})
|
||||
enqueue({ data: [], error: null }) // supplier invoices
|
||||
enqueue({ data: [], error: null }) // external_id dedup — no match
|
||||
enqueue({ data: inserted, error: null }) // insert — STILL imported
|
||||
mockEvaluateMappingRules.mockResolvedValue(makeMappingResult({ confidence: 0.5 }))
|
||||
|
||||
const result = await ingestTransactions(supabase as never, COMPANY_ID, USER_ID, [raw])
|
||||
|
||||
expect(result.imported).toBe(1)
|
||||
expect(result.duplicates).toBe(0)
|
||||
expect(result.shadow_date_drift_candidates).toBe(1)
|
||||
})
|
||||
|
||||
it('does not shadow-flag a date-drift twin on a different known cash account', async () => {
|
||||
const { supabase, enqueue } = createQueueMockSupabase()
|
||||
const raw = makeRaw({
|
||||
date: '2024-06-16', amount: -250, description: 'Hotel expense',
|
||||
external_id: 'eb_acctB_2024-06-16_-25000_0', import_source: 'enable_banking',
|
||||
})
|
||||
const inserted = makeTransaction({ id: 'tx-b', external_id: raw.external_id })
|
||||
|
||||
enqueue({ data: [], error: null }) // booked
|
||||
enqueue({
|
||||
data: [{
|
||||
date: '2024-06-15', amount: -250,
|
||||
original_description: 'Hotel expense', description: 'Hotel expense',
|
||||
import_source: 'enable_banking', cash_account_id: 'acct-A',
|
||||
external_id: 'eb_acctA_2024-06-15_-25000_0',
|
||||
}],
|
||||
error: null,
|
||||
}) // bridging twin one day earlier, but on account A
|
||||
enqueue({ data: [], error: null }) // supplier
|
||||
enqueue({ data: [], error: null }) // external_id dedup
|
||||
enqueue({ data: { id: 'acct-B' }, error: null }) // cash_accounts → batch on account B
|
||||
enqueue({ data: inserted, error: null }) // insert
|
||||
mockEvaluateMappingRules.mockResolvedValue(makeMappingResult({ confidence: 0.5 }))
|
||||
|
||||
const result = await ingestTransactions(supabase as never, COMPANY_ID, USER_ID, [raw], {
|
||||
settlementAccount: '1931',
|
||||
})
|
||||
|
||||
expect(result.imported).toBe(1)
|
||||
expect(result.shadow_date_drift_candidates).toBe(0)
|
||||
})
|
||||
|
||||
it('does not shadow-flag a twin two days away (outside the ±1-day window)', async () => {
|
||||
const { supabase, enqueue } = createQueueMockSupabase()
|
||||
const raw = makeRaw({
|
||||
date: '2024-06-17', amount: -250, description: 'Hotel expense',
|
||||
external_id: 'eb_2024-06-17_-25000_0', import_source: 'enable_banking',
|
||||
})
|
||||
const inserted = makeTransaction({ id: 'tx-far', external_id: raw.external_id })
|
||||
|
||||
enqueue({ data: [], error: null }) // booked
|
||||
enqueue({
|
||||
data: [{
|
||||
date: '2024-06-15', amount: -250,
|
||||
original_description: 'Hotel expense', description: 'Hotel expense',
|
||||
import_source: 'enable_banking', cash_account_id: null,
|
||||
external_id: 'eb_2024-06-15_-25000_0',
|
||||
}],
|
||||
error: null,
|
||||
}) // bridging twin TWO days earlier
|
||||
enqueue({ data: [], error: null }) // supplier
|
||||
enqueue({ data: [], error: null }) // external_id dedup
|
||||
enqueue({ data: inserted, error: null }) // insert
|
||||
mockEvaluateMappingRules.mockResolvedValue(makeMappingResult({ confidence: 0.5 }))
|
||||
|
||||
const result = await ingestTransactions(supabase as never, COMPANY_ID, USER_ID, [raw])
|
||||
|
||||
expect(result.imported).toBe(1)
|
||||
expect(result.shadow_date_drift_candidates).toBe(0)
|
||||
})
|
||||
|
||||
it('does not shadow-flag two genuinely-distinct same-amount rows a day apart (non-bridging, same feed)', async () => {
|
||||
const { supabase, enqueue } = createQueueMockSupabase()
|
||||
const raw = makeRaw({
|
||||
date: '2024-06-16', amount: -250, description: 'LUNCH RESTAURANG',
|
||||
external_id: 'eb_2024-06-16_-25000_0', import_source: 'enable_banking',
|
||||
})
|
||||
const inserted = makeTransaction({ id: 'tx-lunch', external_id: raw.external_id })
|
||||
|
||||
enqueue({ data: [], error: null }) // booked
|
||||
enqueue({
|
||||
data: [{
|
||||
date: '2024-06-15', amount: -250,
|
||||
original_description: 'COFFEE STARBUCKS', description: 'COFFEE STARBUCKS',
|
||||
import_source: 'enable_banking', cash_account_id: null,
|
||||
external_id: 'eb_2024-06-15_-25000_0',
|
||||
}],
|
||||
error: null,
|
||||
}) // distinct same-amount neighbour, same feed, non-bridging desc
|
||||
enqueue({ data: [], error: null }) // supplier
|
||||
enqueue({ data: [], error: null }) // external_id dedup
|
||||
enqueue({ data: inserted, error: null }) // insert
|
||||
mockEvaluateMappingRules.mockResolvedValue(makeMappingResult({ confidence: 0.5 }))
|
||||
|
||||
const result = await ingestTransactions(supabase as never, COMPANY_ID, USER_ID, [raw])
|
||||
|
||||
expect(result.imported).toBe(1)
|
||||
expect(result.shadow_date_drift_candidates).toBe(0)
|
||||
})
|
||||
|
||||
it('does not double-count: an exact-date Layer-2 dedupe is not also a date-drift candidate', async () => {
|
||||
const { supabase, enqueue } = createQueueMockSupabase()
|
||||
const raw = makeRaw({
|
||||
date: '2024-06-15', amount: -250, description: 'KAFFE',
|
||||
external_id: 'eb_new_2024-06-15_-25000_0', import_source: 'enable_banking',
|
||||
})
|
||||
|
||||
enqueue({ data: [], error: null }) // booked
|
||||
// Two stored twins: one EXACT-date (Layer-2 dedupes it) and one a day later.
|
||||
// The row is consumed by Layer-2 and never reaches the date-drift gate.
|
||||
enqueue({
|
||||
data: [
|
||||
{ date: '2024-06-15', amount: -250, original_description: 'KAFFE', description: 'KAFFE',
|
||||
import_source: 'enable_banking', cash_account_id: null, external_id: 'eb_old_0615' },
|
||||
{ date: '2024-06-16', amount: -250, original_description: 'KAFFE', description: 'KAFFE',
|
||||
import_source: 'enable_banking', cash_account_id: null, external_id: 'eb_old_0616' },
|
||||
],
|
||||
error: null,
|
||||
})
|
||||
enqueue({ data: [], error: null }) // supplier
|
||||
enqueue({ data: [], error: null }) // external_id dedup — no match
|
||||
// No insert — deduped by Layer-2.
|
||||
mockEvaluateMappingRules.mockResolvedValue(makeMappingResult({ confidence: 0.5 }))
|
||||
|
||||
const result = await ingestTransactions(supabase as never, COMPANY_ID, USER_ID, [raw])
|
||||
|
||||
expect(result.duplicates).toBe(1)
|
||||
expect(result.imported).toBe(0)
|
||||
expect(result.shadow_date_drift_candidates).toBe(0)
|
||||
})
|
||||
|
||||
it('never lets the date-drift measurement break an import (malformed date is fail-safe)', async () => {
|
||||
const { supabase, enqueue } = createQueueMockSupabase()
|
||||
// A malformed date would make shiftIsoDate throw; the guard must skip
|
||||
// detection so the row imports exactly as before — measurement can never
|
||||
// abort a sync. (Without the guard this test throws instead of asserting.)
|
||||
const raw = makeRaw({
|
||||
date: 'not-a-date', amount: -250, description: 'Hotel expense',
|
||||
external_id: 'eb_bad_date_0', import_source: 'enable_banking',
|
||||
})
|
||||
const inserted = makeTransaction({ id: 'tx-baddate', external_id: raw.external_id })
|
||||
|
||||
enqueue({ data: [], error: null }) // booked
|
||||
enqueue({ data: [], error: null }) // unbooked
|
||||
enqueue({ data: [], error: null }) // supplier
|
||||
enqueue({ data: [], error: null }) // external_id dedup
|
||||
enqueue({ data: inserted, error: null }) // insert — still happens
|
||||
mockEvaluateMappingRules.mockResolvedValue(makeMappingResult({ confidence: 0.5 }))
|
||||
|
||||
const result = await ingestTransactions(supabase as never, COMPANY_ID, USER_ID, [raw])
|
||||
|
||||
expect(result.imported).toBe(1)
|
||||
expect(result.shadow_date_drift_candidates).toBe(0)
|
||||
})
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// 3. Counts errors when insert fails
|
||||
// -----------------------------------------------------------------------
|
||||
@@ -1205,6 +1435,7 @@ describe('ingestTransactions', () => {
|
||||
errors: 0,
|
||||
transaction_ids: [],
|
||||
shadow_scope_drift_candidates: 0,
|
||||
shadow_date_drift_candidates: 0,
|
||||
})
|
||||
})
|
||||
|
||||
|
||||
@@ -164,3 +164,22 @@ export function descriptionsBridge(
|
||||
if (x === '' || y === '') return x === y
|
||||
return x.startsWith(y) || y.startsWith(x)
|
||||
}
|
||||
|
||||
/**
|
||||
* Shift an ISO `YYYY-MM-DD` date by a whole number of days, returning a new
|
||||
* `YYYY-MM-DD` string. Deterministic and INPUT-ONLY — it does the arithmetic
|
||||
* with `Date.UTC` on the parsed components and `new Date(ms)`, never the wall
|
||||
* clock (`Date.now()` / argless `new Date()`), so it is safe in dedup code that
|
||||
* must not depend on the current time. Correctly crosses month, year and
|
||||
* leap-day boundaries via UTC epoch math.
|
||||
*
|
||||
* Used to enumerate the adjacent date buckets the date-drift dedup shadow
|
||||
* inspects: a booking date that drifts a day between syncs lands its twin in
|
||||
* `contentBucketKey(shiftIsoDate(date, ±1), amount)`, which the exact-date
|
||||
* content bridge cannot see.
|
||||
*/
|
||||
export function shiftIsoDate(date: string, deltaDays: number): string {
|
||||
const [y, m, d] = date.split('-').map(Number)
|
||||
const ms = Date.UTC(y, m - 1, d) + deltaDays * 86_400_000
|
||||
return new Date(ms).toISOString().slice(0, 10)
|
||||
}
|
||||
|
||||
@@ -7,7 +7,7 @@ import { findSupplierInvoiceMatch } from '@/lib/invoices/supplier-invoice-matchi
|
||||
import { fetchExchangeRate } from '@/lib/currency/riksbanken'
|
||||
import { logMatchEvent } from '@/lib/invoices/match-log'
|
||||
import { fetchAllRows } from '@/lib/supabase/fetch-all'
|
||||
import { contentBucketKey, descriptionsBridge, normalizeImportedDescription } from '@/lib/transactions/external-id'
|
||||
import { contentBucketKey, descriptionsBridge, normalizeImportedDescription, shiftIsoDate } from '@/lib/transactions/external-id'
|
||||
import { isImportedTransaction } from '@/lib/transactions/origin'
|
||||
import { createLogger } from '@/lib/logger'
|
||||
import type { Transaction, RawTransaction, IngestResult, IngestOptions, SupplierInvoice, Currency, ExchangeRate } from '@/types'
|
||||
@@ -206,6 +206,7 @@ export async function ingestTransactions(
|
||||
errors: 0,
|
||||
transaction_ids: [],
|
||||
shadow_scope_drift_candidates: 0,
|
||||
shadow_date_drift_candidates: 0,
|
||||
}
|
||||
|
||||
const log = createLogger('transactions.ingest', { companyId })
|
||||
@@ -218,6 +219,15 @@ export async function ingestTransactions(
|
||||
// DEDUP_SCOPE_DRIFT_MODE=off to silence. There is deliberately NO 'enforce'
|
||||
// branch yet: we validate on real fleet data first (see the plan).
|
||||
const scopeDriftShadow = process.env.DEDUP_SCOPE_DRIFT_MODE !== 'off'
|
||||
// SHADOW-ONLY instrumentation for the date-drift bridge: the content bridge
|
||||
// buckets on EXACT (date, öre), so a booking date that drifts a day between
|
||||
// syncs lands its twin in an ADJACENT bucket and every dedup layer misses it
|
||||
// (this produced the observed EB↔EB and CSV↔EB 1-day-apart duplicates). When
|
||||
// on, we LOG + COUNT which surviving rows a ±1-day-tolerant rule WOULD treat
|
||||
// as re-imports, but never change what is inserted. Default on; set
|
||||
// DEDUP_DATE_DRIFT_MODE=off to silence. No 'enforce' branch — same as
|
||||
// scope-drift, we validate on real fleet data first.
|
||||
const dateDriftShadow = process.env.DEDUP_DATE_DRIFT_MODE !== 'off'
|
||||
|
||||
// Pre-fetch existing transactions for content-based dedup (date+amount+
|
||||
// description prefix, plus the cross-channel mirror below). Booked rows catch
|
||||
@@ -397,6 +407,26 @@ export async function ingestTransactions(
|
||||
}
|
||||
}
|
||||
|
||||
// ── Shadow-mode date-drift precompute (measure only) ─────────────────────
|
||||
// The content bridge matches only the EXACT (date, öre) bucket, so a twin
|
||||
// whose booking date drifted a day is invisible to it. Snapshot the stored
|
||||
// buckets BEFORE the dedup loop — a COPY of each bucket's entries, so Layer-2's
|
||||
// splices don't mutate what the shadow reads — so each surviving row can look
|
||||
// one day to either side for an account-compatible twin without disturbing
|
||||
// real dedup. Window is ±1 day (the only gap observed); a named constant so
|
||||
// widening to ±2 is one line if fleet data shows it.
|
||||
const DATE_DRIFT_WINDOW_DAYS = 1
|
||||
const storedByBucketForDrift = new Map<string, BucketEntry[]>()
|
||||
if (batchIsImportFeed && dateDriftShadow) {
|
||||
for (const bucket of [existingMaps.booked, existingMaps.unbookedImported]) {
|
||||
for (const [k, entries] of bucket) {
|
||||
const snapshot = storedByBucketForDrift.get(k)
|
||||
if (snapshot) snapshot.push(...entries)
|
||||
else storedByBucketForDrift.set(k, [...entries])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Track already-matched invoice IDs within this ingestion batch
|
||||
// to prevent suggesting the same invoice for multiple transactions
|
||||
const matchedInvoiceIds = new Set<string>()
|
||||
@@ -539,6 +569,74 @@ export async function ingestTransactions(
|
||||
}
|
||||
}
|
||||
|
||||
// SHADOW-ONLY: date-drift. This row survived Layer-1 + Layer-2 and WILL
|
||||
// insert. The content bridge only matched its EXACT (date, öre) bucket, so a
|
||||
// twin whose booking date drifted a day is invisible to it. Look ±1 day for
|
||||
// an account-compatible stored twin that EITHER bridges by description
|
||||
// (same/enriched title — the EB↔EB hotel/fee case) OR is a cross-feed
|
||||
// count-symmetric mirror displaced by a day (the CSV↔EB case where the
|
||||
// descriptions don't bridge). Record it for fleet validation, then insert
|
||||
// unchanged — this block never affects result.imported/duplicates.
|
||||
//
|
||||
// Fail-safe date guard: measurement must NEVER abort a real import. raw.date
|
||||
// is always ISO in practice, but a malformed value would make shiftIsoDate
|
||||
// throw (new Date(NaN).toISOString()), so we skip detection rather than risk
|
||||
// it. Any /^\d{4}-\d{2}-\d{2}$/ value is safe — Date.UTC normalizes
|
||||
// out-of-range parts to a finite epoch, never NaN.
|
||||
if (dateDriftShadow && batchIsImportFeed && /^\d{4}-\d{2}-\d{2}$/.test(raw.date)) {
|
||||
let driftMatch: { entry: BucketEntry; gap: number; signal: 'desc' | 'cross-channel' } | undefined
|
||||
const incomingHere = incomingByBucket.get(bucketKey) ?? 0
|
||||
for (let delta = 1; delta <= DATE_DRIFT_WINDOW_DAYS && !driftMatch; delta++) {
|
||||
for (const sign of [-1, 1] as const) {
|
||||
const adjKey = contentBucketKey(shiftIsoDate(raw.date, sign * delta), raw.amount)
|
||||
const entries = storedByBucketForDrift.get(adjKey)
|
||||
if (!entries) continue
|
||||
// Cross-feed count-symmetry across the drift: equal counts of incoming
|
||||
// rows in THIS bucket and account-compatible cross-feed rows one day
|
||||
// over — the cross-channel mirror, displaced by a date drift.
|
||||
const adjCrossFeed = entries.filter(
|
||||
(e) =>
|
||||
e.isImportFeed &&
|
||||
e.source !== batchSource &&
|
||||
(cashAccountId === null || e.cashAccountId === null || e.cashAccountId === cashAccountId),
|
||||
).length
|
||||
const mirrorSymmetric = adjCrossFeed > 0 && incomingHere === adjCrossFeed
|
||||
for (const entry of entries) {
|
||||
const sameAccount =
|
||||
cashAccountId === null || entry.cashAccountId === null || entry.cashAccountId === cashAccountId
|
||||
if (!sameAccount) continue
|
||||
if (descriptionsBridge(description, entry.desc)) {
|
||||
driftMatch = { entry, gap: sign * delta, signal: 'desc' }
|
||||
break
|
||||
}
|
||||
if (mirrorSymmetric && entry.isImportFeed && entry.source !== batchSource) {
|
||||
driftMatch = { entry, gap: sign * delta, signal: 'cross-channel' }
|
||||
break
|
||||
}
|
||||
}
|
||||
if (driftMatch) break
|
||||
}
|
||||
}
|
||||
if (driftMatch) {
|
||||
result.shadow_date_drift_candidates = (result.shadow_date_drift_candidates ?? 0) + 1
|
||||
log.info('import dedup shadow: date-drift candidate', {
|
||||
decision: 'date-drift',
|
||||
mode: 'shadow',
|
||||
signal: driftMatch.signal,
|
||||
dayGap: driftMatch.gap,
|
||||
bucket: bucketKey,
|
||||
incomingExternalId: raw.external_id,
|
||||
incomingDescription: description,
|
||||
incomingAmount: raw.amount,
|
||||
incomingSource: raw.import_source ?? null,
|
||||
cashAccountId,
|
||||
matchedStoredExternalId: driftMatch.entry.externalId,
|
||||
matchedStoredDescription: driftMatch.entry.desc,
|
||||
matchedStoredCashAccountId: driftMatch.entry.cashAccountId,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// 2. Insert new transaction (with SEK conversion for foreign currencies)
|
||||
const rateInfo = raw.currency && raw.currency !== 'SEK'
|
||||
? exchangeRatesByDate.get(`${raw.currency}|${raw.date}`)
|
||||
|
||||
Reference in New Issue
Block a user