fix(import): let provider re-sync re-import an earlier fiscal year after data deletion (#1763)

* fix(import): let provider re-sync re-import an earlier fiscal year after data deletion

After partially deleting imported data, a provider re-sync could not bring
back the previous fiscal year: the sie_imports 'completed' watermark
survives data deletion, the replace path aborted the whole year when the
prior import row could not be resolved, and prior-import detection picked
an arbitrary row when several overlapped the same year.

- findOverlappingPeriodImports returns ALL overlapping completed rows,
  newest first; checkDuplicatePeriodImport now picks deterministically.
- executeSIEImport replace mode resolves every overlapping row. A row that
  is gone or no longer 'completed' (replaceSIEImport codes not_found /
  not_completed) is a stale watermark: skip it with a warning and import
  the year fresh instead of stranding the user. Locked/closed periods and
  RPC failures still abort the year.
- The arcim-migration wizard names the fiscal year in every per-file
  import failure and shows the newest prior import in the options step.

Fixes #1667

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Emil <emilmattsson14@gmail.com>

* fix(import): fail closed when the replace pre-check query errors

replaceSIEImport's pre-check discarded the .single() error, so a
transient query failure (statement timeout, network error, 5xx via
PostgREST) was indistinguishable from a genuinely absent row and got
classified not_found. The replace loop in executeSIEImport then treated
it as a stale watermark and imported the fiscal year fresh while the
prior completed import's verifikationer were still in the ledger, with
duplicate checks skipped in replace mode: silent duplicate
verifikationer for a whole year (BFL 4:1 risk).

Only PGRST116 (zero rows from .single()) now classifies as not_found;
any other pre-check error returns rpc_error, which aborts the year in
the replace loop. Tests cover both classifications plus the
executeSIEImport-level abort.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Emil <emilmattsson14@gmail.com>

* fix(import): fail closed on overlap lookup, verify zero surviving entries before a stale-watermark skip

Review findings on #1763 (CodeRabbit + Swedish compliance review):
- findOverlappingPeriodImports now uses fetchAllRows: query errors throw
  instead of returning [] (which let replace mode import fresh over rows it
  never resolved), pagination passes the PostgREST row cap, id tiebreak
  keeps the order total.
- A stale-watermark skip (not_found/not_completed) is only trusted after a
  positive check that zero posted import entries survive in the fiscal
  year: replace_sie_import deletes by fiscal period, so entries can outlive
  their sie_imports row. Survivors or a failed check abort the year.
- Contract comment tying the stale-race regex to the RPC's RAISE wording.
- Suite-level beforeEach clears mocks and the event bus (repo convention).

Signed-off-by: Emil <emilmattsson14@gmail.com>

---------

Signed-off-by: Emil <emilmattsson14@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Mattsson
2026-08-20 21:11:15 +02:00
committed by GitHub
co-authored by Claude Fable 5
parent 3f6f1ab06e
commit febb4cc0c2
5 changed files with 689 additions and 47 deletions
+1
View File
@@ -1127,3 +1127,4 @@ One line per decision: `[YYYY-MM-DD] <decision>: <why>`. Appended by agents and
[2026-08-20] AI_API_KEY made optional for the OpenAI-compatible backend: a base URL alone now counts as configured (hasAiCredentials / resolveAiProvider), so a local model server (llama.cpp/Ollama/LM Studio/vLLM), which usually has no auth, works with just AI_BASE_URL + AI_MODEL. The openai-compatible service only sends an Authorization: Bearer when AI_API_KEY is set, so a keyless local server is never handed an empty bearer. Hosted providers that require a key still set AI_API_KEY. Bedrock/Anthropic credential logic unchanged.
[2026-08-20] poppler-utils is the one system package added to the self-host runner image (Sovereign plan WS1 PR2): pdftoppm renders the first pages of a PDF for AI backends without native PDF input (an OpenAI-compatible Swedish endpoint), measured at ~4 MB plus shared libs on node:22-alpine (pdftoppm 25.12), written to /tmp which docker-compose.yml already mounts as tmpfs under the read-only root. Hosted never calls it (Bedrock reads PDFs natively) and the cron image is untouched. pdfjs-dist + @napi-rs/canvas were rejected earlier (two npm deps, memory spikes, dead weight on hosted). scripts/smoke-ai-provider.ts is the backend-agnostic "is AI wired up" check; verified live against hosted Bedrock and against a local OpenAI-compatible mock (the mock received Bearer auth, per-tier model ids and one image_url part per rasterized page).
[2026-08-20] RIP-3 chat cutover is scoped to general.help only: the free-form Q&A /chat panel now runs on a page-scoped single-call console (AskConsole → POST /api/agent/ask, persist:true), so it works on any configured backend incl. a local OpenAI-compatible model. The tool-loop intents (transaction.categorization, invoice.draft, supplier_invoice.review) and the docked AgentSheet still use AgentChat + run-turn.ts because they stage operations and need the tool loop, so run-turn.ts is NOT deleted here (the plan gates its deletion on "once nothing calls them"; RIP-4 migrates the rest). Persistence is an opt-in branch on the existing /api/agent/ask route rather than a new endpoint, so page-scoped one-off asks (a report page) stay stateless; the console writes both turns to agent_conversations/agent_messages as canonical Anthropic text blocks so the /chat sidebar and resume keep working across old streaming threads and new single-call ones.
[2026-08-19] Provider re-sync replace mode resolves EVERY overlapping completed sie_imports row and treats one it cannot resolve (not_found/not_completed) as a stale watermark to skip, importing the year fresh, but still aborts on a locked or closed period: an unresolvable row has nothing left to delete, while importing over entries that could not be deleted would duplicate verifikationer.