fix(import): chunk SIE account creation and close the import row on every exit (#2451)

* fix(import): chunk SIE account creation and close the import row on every exit

A full-BAS Bokio SIE file creates 1 200+ chart_of_accounts rows in one
INSERT. PostgREST runs it under the authenticated role's 8 s
statement_timeout, and with four row-level triggers plus the RLS WITH
CHECK that single statement measured 6.5 s to 8.2 s on prod: it was
cancelled for one company and passed for the next (2026-09-09).

- syncMappedAccounts inserts in chunks of 100 rows (INSERT_CHUNK_SIZE),
  so every statement stays an order of magnitude inside the limit. A
  failed chunk leaves the earlier ones committed; the next attempt reads
  the chart again and inserts only what is still missing.
- executeSIEImport closes its pending sie_imports row in a finally
  block. Every early `return result` after createPendingImportRecord
  (account sync failure, missing fiscal year, overlapping import,
  vouchers outside the year) used to leave the row 'pending'. That row
  holds the (company_id, file_hash) slot in the partial unique index,
  so a retry inside the five-minute cleanup gate failed on the index
  instead of on the real error.
- The slot-held message no longer names "gnubok", an "Ersätt import"
  button the import history has never had, or Fortnox; it says the
  same file is being imported or was interrupted moments ago and to
  retry in a few minutes. The thrown-error prefix is Swedish
  ("Importen misslyckades:") and the two undo hints name
  accounted_undo_sie_import.

Tests: chunk sizes and first-failing-chunk behaviour in
account-sync.test.ts; failed-row finalize on two early exits and the
new slot message in sie-import.account-names.test.ts.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBTRraVXkCnx93Pa9wxJ6G

* fix(import): write account chunks as ignore-duplicates upserts with an exact count

Review follow-up on #2451. A plain INSERT per chunk still had the old
race: a concurrent import (or the replace flow) creating one account
between our read and write raised a duplicate-key error that the code
swallowed as success, while PostgREST had rolled back the whole chunk,
so every other account in it was silently missing.

Each chunk is now an upsert with onConflict (company_id, account_number)
and ignoreDuplicates, selecting the landed rows: the race becomes a
skipped row, `created` counts exactly what was written (also across a
mid-loop failure, which the compliance review flagged), and the
"duplicate" string special-case is gone.

Tests: conflict-skipped row not counted; a race yields no error; a
failing later chunk reports the rows the earlier chunks committed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBTRraVXkCnx93Pa9wxJ6G

---------

Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Jakob Wennberg
2026-09-09 12:24:29 +02:00
committed by GitHub
co-authored by Claude Fable 5.1 Jakob Wennberg
parent fc2d78a7c4
commit bb28968151
5 changed files with 288 additions and 23 deletions
+2
View File
@@ -1684,6 +1684,7 @@ One line per decision: `[YYYY-MM-DD] <decision>: <why>`. Appended by agents and
[2026-09-08] Issue #2224 follow-up from the correctness skeptic: the quote decision (open/declined) is now locked in the database while a live kundorder exists (migration 20260908165100 extends invoices_quote_decision_guard), reversing the earlier call to leave it open; a declined offert behind a confirmed, invoiced order was a contradictory agreement trail and the dashboard hid the re-accept button, so the quote was stuck. The three source and decision guards run as SECURITY DEFINER: a SELECT FOR UPDATE under RLS admits only the caller's active company, so a multi-company member writing for another company through raw PostgREST got no row, no lock and no guard. Both landed as a second migration rather than an edit of 20260908165000, which was already applied to staging under that version.
[2026-09-08] Draft invoice PDF marks a draft with one diagonal, faint word (UTKAST / DRAFT) across every page instead of a banner in the top margin (#2437): a banner reads as UI chrome on a document, a watermark reads as a stamp and leaves the preview pixel-identical to the final print. The long legal sentence (saknar löpnummer, ML 17 kap 24 §) is dropped on purpose: the word alone says the document is not a valid invoice, and the download dialog (#2399) already explains why before the file exists. Rotation and opacity sit on a padded wrapper View so the word turns about its own centre. Skeptic refutation accepted: the first cut (#6b7280 at 0.14, about 92% brightness) would drop out of a monochrome print or greyscale scan, and a numbered draft otherwise prints title, number and OCR like a real faktura; now #4b5563 at 0.3 (about 79% brightness), with a test pinning the composited grey between 70% and 85%. A 1-bit scan can still threshold the word away; a second explicit line on numbered drafts was left out because the request was the word alone, and that residual is Emil's call. Second refutation accepted: the overlay is emitted as the LAST child of the Page, because react-pdf paints in document order and `fixed` does not hoist, so an overlay emitted first was painted under the opaque payment and customer boxes and the word vanished on the page that carries totals and OCR; a test now inflates the PDF content streams and asserts the glyph run comes after the last rectangle fill on every page. BETALD and MAKULERAD banners are left as they are.
[2026-09-08] Negative journal-line amounts: fixed the sign at three levels (producers flip the SIDE via lib/bookkeeping/line-side.ts, the engine refuses negative amounts before any write, and a NOT VALID CHECK on journal_entry_lines) instead of only patching the supplier-invoice generator or hiding negative items in the form. Why: the invariant lived nowhere (no Zod rule, no engine check, no constraint), so MCP, templates and any future producer could repeat it; negative items themselves are valid input (rabatt, öresavrundning), so rejecting them at input would break real invoices. reverseEntry now swaps on the net so legacy negative lines storno cleanly before the data repair runs.
[2026-09-09] SIE import account creation (syncMappedAccounts) inserts in chunks of 100 rows instead of one statement per chart, and executeSIEImport closes its pending sie_imports row in a finally block on every exit: a full-BAS Bokio import (1 242 accounts) hit the authenticated role's 8 s statement_timeout on prod (8.2 s cancelled for one company, 6.5 s passed for the next), and the early `return result` after the failed account sync left the row 'pending', so the user's retry 40 s later was refused by the file-hash unique index with a message that named "gnubok" and an "Ersätt import" button that does not exist. Chunking was chosen over moving the insert into an RPC with SET statement_timeout (the import_sie_journal_entries precedent): the insert is idempotent per row, a partial success is safe to retry, and no migration or pg-real ratchet is needed; the per-row trigger cost (audit log, writer-role guard, RLS WITH CHECK) is untouched and stays a separate concern. The slot-held message is static (no lookup of the holder) so it stays deterministic in the queued-mock tests.
[2026-09-09] Onboarding search-as-you-type (#2448) reads SCB's företagsregister per debounced keystroke and runs TIC once, on the pick, instead of TIC per keystroke: the 3000/month Lens budget is why #2421 fired on Enter only, and SCB is free; the pick still needs TIC because SCB knows no F-skatt, VAT or fiscal year. No rate limit on either route, founder's call (2026-09-09): the debounce, the 3-char minimum and the abort of superseded requests are the only throttles. Sole traders are included in the onboarding search (SCB legal form 10) but stay excluded from the parties picker; the row names the form, never the personnummer.
[2026-09-09] Bank picker: an UNCHECKED account holds its bokföringskonto only as a soft claim (yields to a checked account that wants it), instead of either a hard claim (status quo: 400 "Flera bankkonton kan inte bokföras på samma konto", the support dead end where 1930 could never move from the wrong bank account to the right one because the picker hides the ledger dropdown for unchecked rows and disconnect + reconnect re-claims the same rows by IBAN) or a full release on every save (fewer states, but it demotes every unchecked row to a manual ghost holding its ledger and loses the "re-check lands back on the same account" prefill). Contested rows are demoted to manual in ONE update before the mirror, and upsertFromPsd2 then promotes the manual holder in place, which keeps row ids, transactions.cash_account_id links and the is_primary flag on the 1930 row; the same release pass also makes two checked accounts swapping ledgers work (previously both upserts tripped the unique constraint and were swallowed per-account). Another connection's UNCHECKED row yields the same way (cash_accounts.enabled = false), matching how session sharing already counts only enabled rows as claims; a synced-elsewhere row stays a 400. No new client/UI: the picker needed no change once the server stopped counting unchecked rows.
@@ -1692,3 +1693,4 @@ One line per decision: `[YYYY-MM-DD] <decision>: <why>`. Appended by agents and
[2026-09-08] Zettle orders cron pages candidates (range) and caps entitled syncs at 50 instead of limit(50) before hasCapability: entitlement skips must not consume the batch or advance last_order_synced_at (purchase recovery cursor). Declined a separate cron_checked_at column for now; revisit if scanned non-entitled volume becomes a time-budget problem.
[2026-09-09] PR #2416 (Zettle, community-authored) adopted on the contributor's branch instead of re-implemented: kept the Shopify-shaped feed-only design, added migration 20260909100400 for the sites that enumerate platforms/connection tables (platform CHECKs on webshop_orders and webshop_store_settings, writer-role gate trigger, migration-reset snapshot and lock via the 20260826150000 wrapper pattern rather than re-issuing the 400-line reset body), renamed the four PR migrations past prod's 20260908143051, froze the validated connect origin on zettle_connections.return_origin so white-label users return to their brand domain (Zettle has one registered callback URL), and derive per-rate VAT net from Zettle's own product rows (tax / rate drifted from what was charged) with a one-öre-per-row line-sum tolerance instead of 0.005 kr (silently dropped multi-row 12%/6% underlag). Per-purchase rows kept for v1; daily kassarapport aggregation and Finance API fees/payouts are the follow-up.
[2026-09-09] Zettle v1 imports split-tender, gift-card (sale or tender) and tip-carrying purchases as is_paid = false rows titled 'bokför manuellt' and skips their refunds, instead of booking them through the one-account / revenue-per-rate model: the skeptic showed a card+cash split would put the whole gross on 1686, a card+invoice split would count as paid, a 0 % gift-card row would land on 3004 / ruta 42 (it is a 2421 liability), and tips would book as momsfri sale. Proper support needs per-payment amounts and a voucher liability on webshop_orders (follow-up). Sync runs claim the connection (sync_lock_until) before refreshing the rotating token: a concurrent cron + manual sync otherwise reuses a refresh token, Zettle answers 400, and the connection flips to revoked. The cron snapshots its candidate list before syncing because each sync moves the row to the tail of the last_order_synced_at ordering, so live offset paging re-fetched synced rows and skipped unseen ones.
[2026-09-09] SIE account creation writes each chunk as an ignore-duplicates upsert (ON CONFLICT (company_id, account_number) DO NOTHING, returning the landed rows) instead of a plain INSERT whose duplicate-key error was swallowed: under PostgREST one request is one transaction, so a single concurrent duplicate rolled back the whole statement while the caller counted it as created and moved on with accounts missing. The conflict clause makes the race a skipped row, `created` counts exactly what landed, and the "duplicate" string special-case is gone. Raised by CodeRabbit and the compliance review on PR #2451 (count accuracy under BFNAR 2013:2 p. 9.16); the audit_log trigger stays the per-row record.