fix(arcim): allow Fortnox re-sync to replace prior SIE import per fiscal year (#512)

* fix(arcim): allow Fortnox re-sync to replace prior SIE import per fiscal year A user reported a sync failure when retrying Fortnox after adding more verifications: Import failed: Failed to create pending import record: duplicate key value violates unique constraint "sie_imports_company_id_file_hash_active_idx" Root cause: Fortnox embeds the export-time #GEN date in every SIE export, so the file hash always differs between syncs. The wizard's hash-based duplicate detection treated each sync as a brand new file, but the engine still rejected the insert because the per-period import slot was held by the prior 'completed' row. This change reframes Fortnox re-sync as a replace operation rather than a fresh import: - New executeSIEImport option `onExistingPeriod: 'block' | 'replace'`. Manual SIE upload at /api/import/sie keeps default 'block' (current behavior, no regression). The Fortnox /import-sie endpoint passes 'replace', which runs replaceSIEImport on any overlapping completed import before insert. Imported journal entries from the prior import are cancelled per BFL 5 kap 5§; user-created entries (manual, transaction, invoice) are untouched. - /sie-data switches from hash-based to period-based duplicate detection and returns previousImport metadata per fiscal year. - Wizard drops the alreadyImported skip filter, surfaces an amber callout in the confirm dialog listing fiscal years that will be replaced, and shows "ersatte N tidigare importerade verifikationer" per year. - createPendingImportRecord translates 23505 partial-index violations to a clear Swedish recovery message instead of leaking the raw constraint name. - cleanupStaleImportRecords drops the 1-hour age gate and also cleans status='mapped' orphans. SIE imports are single-flight per company so the gate just made legitimate retries fail. - After replace, the fiscal_periods row's opening_balances_set and opening_balance_entry_id are cleared (only when they pointed at the cancelled prior IB entry), so the new IB import isn't skipped. Schema-drift migration captures the partial unique index sie_imports_company_id_file_hash_active_idx that already exists in production (added out-of-band) and drops the now-superseded plain sie_imports_company_id_file_hash_key constraint. Both statements are idempotent — verified no-op against production. Tests: new pg-real test covers the partial index admit-replaced semantics, source_type='import'-only cancellation in replace_sie_import, and the post-replace insert path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(arcim): address PR review — restore 5-min cleanup gate, fix test source_type Greptile P2: `cleanupStaleImportRecords` was deleting `pending` rows unconditionally, which could wipe a concurrent in-flight import in another tab/session. Restored a 5-minute age gate (long enough for any normal interactive import, short enough that legitimate retries after a crash still succeed). Also dropped `mapped` from the cleanup — it is defined in SIEImportStatus but no code path writes it, so including it was both unnecessary and added the concurrent-session risk Greptile flagged. pg-real test: insertPostedEntry used `source_type='transaction'` which is not a valid value per the journal_entries_source_type_check constraint (migration 20260516060000). Switched to `'bank_transaction'` — the actual source_type emitted when a user categorizes a bank transaction in gnubok. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 16:25:43 +02:00
parent f98ffee145
commit 831920fede
6 changed files with 527 additions and 96 deletions
@@ -0,0 +1,212 @@
+import { randomUUID } from 'node:crypto'
+import { describe, expect, it } from 'vitest'
+import { getPool } from '@/tests/pg/setup'
+import { seedCompany } from '@/tests/pg/fixtures'
+
+// Covers the Fortnox re-sync flow:
+//  1. The partial unique index `sie_imports_company_id_file_hash_active_idx`
+//     (added in migration 20260517150000) blocks duplicate (company_id,
+//     file_hash) rows for active statuses but allows them once a prior row
+//     is marked 'replaced' or 'failed'.
+//  2. The replace_sie_import RPC cancels journal entries with
+//     source_type='import' while leaving user-created entries
+//     (source_type='manual', 'bank_transaction', etc.) intact.
+
+async function insertSIEImport(params: {
+  companyId: string
+  userId: string
+  fileHash: string
+  status: 'pending' | 'mapped' | 'completed' | 'failed' | 'replaced'
+  fiscalPeriodId?: string
+  openingBalanceEntryId?: string
+  fiscalYearStart?: string
+  fiscalYearEnd?: string
+}): Promise<string> {
+  const id = randomUUID()
+  await getPool().query(
+    `INSERT INTO public.sie_imports
+       (id, user_id, company_id, filename, file_hash, sie_type,
+        fiscal_year_start, fiscal_year_end, accounts_count, transactions_count,
+        status, fiscal_period_id, opening_balance_entry_id, imported_at)
+     VALUES ($1, $2, $3, 'fortnox-export.se', $4, 4,
+             $5, $6, 0, 0,
+             $7, $8, $9, $10)`,
+    [
+      id,
+      params.userId,
+      params.companyId,
+      params.fileHash,
+      params.fiscalYearStart ?? '2026-01-01',
+      params.fiscalYearEnd ?? '2026-12-31',
+      params.status,
+      params.fiscalPeriodId ?? null,
+      params.openingBalanceEntryId ?? null,
+      params.status === 'completed' ? new Date().toISOString() : null,
+    ],
+  )
+  return id
+}
+
+async function insertPostedEntry(params: {
+  userId: string
+  companyId: string
+  fiscalPeriodId: string
+  sourceType: 'import' | 'manual' | 'bank_transaction'
+  voucherNumber: number
+  entryDate?: string
+}): Promise<string> {
+  const id = randomUUID()
+  await getPool().query(
+    `INSERT INTO public.journal_entries
+       (id, user_id, company_id, fiscal_period_id, voucher_number, voucher_series,
+        entry_date, description, source_type, status)
+     VALUES ($1, $2, $3, $4, $5, 'A', $6, 'Test entry', $7, 'posted')`,
+    [
+      id,
+      params.userId,
+      params.companyId,
+      params.fiscalPeriodId,
+      params.voucherNumber,
+      params.entryDate ?? '2026-06-01',
+      params.sourceType,
+    ],
+  )
+  await getPool().query(
+    `INSERT INTO public.journal_entry_lines
+       (journal_entry_id, account_number, debit_amount, credit_amount)
+     VALUES ($1, '1930', 100, 0),
+            ($1, '3001', 0, 100)`,
+    [id],
+  )
+  return id
+}
+
+describe('sie_imports: partial unique index + replace flow', () => {
+  it('blocks a second active row with the same (company_id, file_hash)', async () => {
+    const { companyId, userId, fiscalPeriodId } = await seedCompany()
+    const hash = `hash-${randomUUID()}`
+
+    await insertSIEImport({
+      companyId,
+      userId,
+      fileHash: hash,
+      status: 'completed',
+      fiscalPeriodId,
+    })
+
+    await expect(
+      insertSIEImport({
+        companyId,
+        userId,
+        fileHash: hash,
+        status: 'pending',
+        fiscalPeriodId,
+      }),
+    ).rejects.toThrow(/sie_imports_company_id_file_hash_active_idx/)
+  })
+
+  it('allows a new pending row with the same hash once the prior row is replaced', async () => {
+    const { companyId, userId, fiscalPeriodId } = await seedCompany()
+    const hash = `hash-${randomUUID()}`
+
+    const priorId = await insertSIEImport({
+      companyId,
+      userId,
+      fileHash: hash,
+      status: 'completed',
+      fiscalPeriodId,
+    })
+
+    // Mark the prior row as replaced (simulating what replace_sie_import does)
+    await getPool().query(
+      `UPDATE public.sie_imports SET status = 'replaced', replaced_at = now() WHERE id = $1`,
+      [priorId],
+    )
+
+    // A new pending row with the same hash now succeeds
+    const newId = await insertSIEImport({
+      companyId,
+      userId,
+      fileHash: hash,
+      status: 'pending',
+      fiscalPeriodId,
+    })
+    expect(newId).toBeTruthy()
+  })
+
+  it('replace_sie_import cancels source_type=import entries and leaves manual/bank_transaction entries posted', async () => {
+    const { companyId, userId, fiscalPeriodId } = await seedCompany()
+
+    const obEntry = await insertPostedEntry({
+      userId,
+      companyId,
+      fiscalPeriodId,
+      sourceType: 'import',
+      voucherNumber: 1,
+    })
+    const importEntry1 = await insertPostedEntry({
+      userId,
+      companyId,
+      fiscalPeriodId,
+      sourceType: 'import',
+      voucherNumber: 2,
+    })
+    const importEntry2 = await insertPostedEntry({
+      userId,
+      companyId,
+      fiscalPeriodId,
+      sourceType: 'import',
+      voucherNumber: 3,
+    })
+    const manualEntry = await insertPostedEntry({
+      userId,
+      companyId,
+      fiscalPeriodId,
+      sourceType: 'manual',
+      voucherNumber: 4,
+    })
+    const txnEntry = await insertPostedEntry({
+      userId,
+      companyId,
+      fiscalPeriodId,
+      sourceType: 'bank_transaction',
+      voucherNumber: 5,
+    })
+
+    const importId = await insertSIEImport({
+      companyId,
+      userId,
+      fileHash: `hash-${randomUUID()}`,
+      status: 'completed',
+      fiscalPeriodId,
+      openingBalanceEntryId: obEntry,
+    })
+
+    const { rows } = await getPool().query<{ replace_sie_import: number }>(
+      `SELECT public.replace_sie_import($1::uuid, $2::uuid) AS replace_sie_import`,
+      [companyId, importId],
+    )
+    const cancelled = rows[0]!.replace_sie_import
+
+    // OB entry + 2 import entries = 3 cancelled. Manual & transaction stay posted.
+    expect(cancelled).toBe(3)
+
+    const statuses = await getPool().query<{ id: string; status: string }>(
+      `SELECT id, status FROM public.journal_entries WHERE id = ANY($1)`,
+      [[obEntry, importEntry1, importEntry2, manualEntry, txnEntry]],
+    )
+    const statusById = Object.fromEntries(statuses.rows.map(r => [r.id, r.status]))
+    expect(statusById[obEntry]).toBe('cancelled')
+    expect(statusById[importEntry1]).toBe('cancelled')
+    expect(statusById[importEntry2]).toBe('cancelled')
+    expect(statusById[manualEntry]).toBe('posted')
+    expect(statusById[txnEntry]).toBe('posted')
+
+    const importRow = await getPool().query<{ status: string; replaced_at: string | null }>(
+      `SELECT status, replaced_at FROM public.sie_imports WHERE id = $1`,
+      [importId],
+    )
+    expect(importRow.rows[0]!.status).toBe('replaced')
+    expect(importRow.rows[0]!.replaced_at).not.toBeNull()
+  })
+})
@@ -191,23 +191,41 @@ export async function replaceSIEImport(
 }

 /**
- * Clean up stale pending/failed import records for a given file hash.
- * Prevents UNIQUE constraint conflicts when re-importing after a failure.
+ * Clean up orphan in-flight import records for a given file hash.
+ *
+ * Targets rows in status='pending' — left behind when a prior import
+ * crashed (or short-circuited at checkDuplicatePeriodImport) before
+ * reaching finalizeImportRecord. They hold the slot in the partial
+ * unique index `sie_imports_company_id_file_hash_active_idx`, so a
+ * retry would fail with a constraint violation.
+ *
+ * Five-minute age gate protects an in-flight import in another tab/
+ * session: createPendingImportRecord → ... → finalizeImportRecord can
+ * take tens of seconds for large SIE files. Without the gate, a
+ * concurrent retry of the same file would delete the live pending row
+ * mid-flight and the original session's finalize would silently no-op.
+ * Five minutes is long enough for any normal interactive import yet
+ * short enough that legitimate retries after a crash succeed.
+ *
+ * The 'mapped' status is defined in the type but never written by any
+ * code path, so we don't include it. 'failed' and 'replaced' rows are
+ * allowed by the partial index (excluded from its predicate), so they
+ * stay in place for the audit trail.
 */
 async function cleanupStaleImportRecords(
  supabase: SupabaseClient,
  companyId: string,
  fileHash: string
 ): Promise<void> {
-  const oneHourAgo = new Date(Date.now() - 60 * 60 * 1000).toISOString()
+  const fiveMinutesAgo = new Date(Date.now() - 5 * 60 * 1000).toISOString()

  await supabase
    .from('sie_imports')
    .delete()
    .eq('company_id', companyId)
    .eq('file_hash', fileHash)
-    .in('status', ['pending', 'failed'])
-    .lt('created_at', oneHourAgo)
+    .eq('status', 'pending')
+    .lt('created_at', fiveMinutesAgo)
 }

 /**
@@ -1377,7 +1395,22 @@ async function createPendingImportRecord(
    .single()

  if (error || !data) {
-    throw new Error(`Failed to create pending import record: ${error?.message}`)
+    // PG error 23505 (unique_violation) on the partial index means another
+    // active row exists for the same (company_id, file_hash). Surface the
+    // recovery path in Swedish instead of leaking the raw Postgres message.
+    const pgCode = (error as { code?: string } | null | undefined)?.code
+    const pgMessage = error?.message ?? ''
+    const hitsActiveIdx =
+      pgCode === '23505' &&
+      pgMessage.includes('sie_imports_company_id_file_hash_active_idx')
+
+    if (hitsActiveIdx) {
+      throw new Error(
+        'En tidigare SIE-import för samma fil finns redan i gnubok. Öppna importhistoriken och välj "Ersätt import" på den befintliga raden, eller använd Fortnox-synkningen för att hämta uppdaterad data automatiskt.'
+      )
+    }
+
+    throw new Error(`Failed to create pending import record: ${pgMessage}`)
  }

  return data.id
@@ -1490,6 +1523,19 @@ export async function loadMappings(supabase: SupabaseClient, companyId: string):

 /**
 * Execute the full SIE import
+ *
+ * `onExistingPeriod` controls how a prior completed import that overlaps
+ * the new SIE's fiscal year is handled:
+ *   - 'block' (default): refuse with a Swedish error. Used by the manual
+ *     upload route in app/api/import/sie. Preserves prior behavior.
+ *   - 'replace': automatically call replaceSIEImport on the prior row
+ *     (marks it 'replaced', cancels its imported journal entries) and
+ *     proceed. Used by the Fortnox re-sync flow so the user can pull
+ *     updated data from Fortnox without manual cleanup.
+ *
+ * Replace only cancels journal entries with source_type='import' — entries
+ * the user created natively in gnubok (categorized transactions, invoices,
+ * etc.) are left alone. See the replace_sie_import RPC.
 */
 export async function executeSIEImport(
  supabase: SupabaseClient,
@@ -1504,6 +1550,7 @@ export async function executeSIEImport(
    importOpeningBalances: boolean
    importTransactions: boolean
    voucherSeries?: string
+    onExistingPeriod?: 'block' | 'replace'
  }
 ): Promise<ImportResult> {
  const result: ImportResult = {
@@ -1515,8 +1562,11 @@ export async function executeSIEImport(
    journalEntryIds: [],
    errors: [],
    warnings: [],
+    replacedPriorImport: null,
  }

+  const onExistingPeriod = options.onExistingPeriod ?? 'block'
+
  try {
    // Validate all accounts are mapped
    const unmapped = mappings.filter((m) => !m.targetAccount)
@@ -1527,13 +1577,66 @@ export async function executeSIEImport(
      return result
    }

-    // Check for duplicate import (only completed imports count as duplicates)
-    const duplicate = await checkDuplicateImport(supabase, companyId, options.fileContent)
-    if (duplicate) {
-      result.errors.push(
-        `This file has already been imported on ${duplicate.imported_at ? new Date(duplicate.imported_at).toLocaleDateString('sv-SE') : 'okänt datum'}`
-      )
-      return result
+    // Replace mode: if a prior completed import overlaps the new SIE's fiscal
+    // year, mark it 'replaced' (and cancel its imported entries) before we
+    // try to insert. Done before checkDuplicateImport / checkDuplicatePeriodImport
+    // since both of those would otherwise reject the replace flow.
+    if (onExistingPeriod === 'replace') {
+      const fyStart = parsed.stats.fiscalYearStart
+      const fyEnd = parsed.stats.fiscalYearEnd
+      if (fyStart && fyEnd) {
+        const priorPeriodImport = await checkDuplicatePeriodImport(
+          supabase, companyId, fyStart, fyEnd
+        )
+        if (priorPeriodImport) {
+          const replaceResult = await replaceSIEImport(
+            supabase, companyId, priorPeriodImport.id
+          )
+          if (!replaceResult.success) {
+            result.errors.push(
+              replaceResult.error ?? 'Kunde inte ersätta tidigare SIE-import'
+            )
+            return result
+          }
+          result.replacedPriorImport = {
+            importId: priorPeriodImport.id,
+            cancelledEntries: replaceResult.cancelledEntries,
+          }
+
+          // The replace_sie_import RPC cancelled the prior import's opening
+          // balance entry, but the fiscal_periods row still flags
+          // opening_balances_set=true and points opening_balance_entry_id at
+          // the now-cancelled row. Without clearing those, the IB import
+          // below would skip ("Ingående balanser finns redan...") and the
+          // new IB would be lost. Only reset when the cleared entry was the
+          // prior import's IB — if the period's IB came from somewhere else
+          // (manual entry, year-end carryover), we must not touch it.
+          if (priorPeriodImport.fiscal_period_id && priorPeriodImport.opening_balance_entry_id) {
+            await supabase
+              .from('fiscal_periods')
+              .update({
+                opening_balances_set: false,
+                opening_balance_entry_id: null,
+              })
+              .eq('id', priorPeriodImport.fiscal_period_id)
+              .eq('company_id', companyId)
+              .eq('opening_balance_entry_id', priorPeriodImport.opening_balance_entry_id)
+          }
+        }
+      }
+    }
+
+    // Block mode (default): the hash and period checks reject duplicates with
+    // graceful Swedish errors. Skipped in replace mode because we've already
+    // resolved any prior import above.
+    if (onExistingPeriod === 'block') {
+      const duplicate = await checkDuplicateImport(supabase, companyId, options.fileContent)
+      if (duplicate) {
+        result.errors.push(
+          `This file has already been imported on ${duplicate.imported_at ? new Date(duplicate.imported_at).toLocaleDateString('sv-SE') : 'okänt datum'}`
+        )
+        return result
+      }
    }

    // Create pending import record early — ensures tracking even if later steps fail
@@ -1628,15 +1731,19 @@ export async function executeSIEImport(
      return result
    }

-    // Safety net: reject if a completed import already exists for this period
-    const periodDuplicate = await checkDuplicatePeriodImport(
-      supabase, companyId, fiscalYearStart, fiscalYearEnd
-    )
-    if (periodDuplicate) {
-      result.errors.push(
-        `En SIE-import för ett överlappande räkenskapsår (${periodDuplicate.fiscal_year_start} – ${periodDuplicate.fiscal_year_end}) finns redan`
+    // Safety net: reject if a completed import already exists for this period.
+    // Skipped in replace mode — any overlapping prior import was already
+    // marked 'replaced' at the top of executeSIEImport.
+    if (onExistingPeriod === 'block') {
+      const periodDuplicate = await checkDuplicatePeriodImport(
+        supabase, companyId, fiscalYearStart, fiscalYearEnd
      )
-      return result
+      if (periodDuplicate) {
+        result.errors.push(
+          `En SIE-import för ett överlappande räkenskapsår (${periodDuplicate.fiscal_year_start} – ${periodDuplicate.fiscal_year_end}) finns redan`
+        )
+        return result
+      }
    }

    if (options.createFiscalPeriod) {
@@ -296,6 +296,11 @@ export interface ImportResult {

  // Structured details for UI (populated alongside warnings for backwards compat)
  details?: ImportResultDetails
+
+  // If this import replaced a prior completed import for the same fiscal year
+  // (Fortnox re-sync flow), the prior import's id and the count of journal
+  // entries that were cancelled as a result.
+  replacedPriorImport?: { importId: string; cancelledEntries: number } | null
 }

 /**