fix(migration): Visma pagination + chunk-insert resilience (the '300 misslyckades' case) (#1455)
* fix(providers): paginate Visma eAccounting with $page/$pagesize eAccounting silently ignores OData $top/$skip, so every request returned page 1 and getPaginated appended the first page TotalNumberOfPages times: customers were imported in triplicate and invoice chunks hit unique violations. Also stop on an empty page so a stale Meta can never loop or duplicate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(migration): survive bad rows in entity imports instead of failing whole chunks One PostgREST insert per 500-row chunk is all-or-nothing, so a single duplicate reported every row as failed ('300 misslyckades') with no cause shown. Now: dedupe repeats within the fetched data (paging faults, source duplicates), fall back to per-row inserts when a chunk is rejected, store empty invoice numbers as NULL instead of colliding '', surface the first DB error in the result UI, and mark all-failed steps with an error icon. Sales invoices also carry remaining_amount so open invoices no longer land as settled. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(migration): never per-row retry after a successful bulk insert with short read-back A succeeded statement whose .select() returns fewer rows than sent means the rows ARE in the table; retrying them one by one would duplicate every unreturned row. Pair what came back and report the tail instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(migration): count stub-insert casualties as failed and sample enrichment errors Review follow-ups: invoices dropped because their customer/supplier stub insert errored are DB failures, not matching misses; classifying them as noMatch rendered a green result row with the database error hidden. Enrichment failures now also feed errorSample. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
Jakob Wennberg
parent
cf373e9dd1
commit
a49d75db77
@@ -88,8 +88,12 @@ export class VismaClient {
|
||||
const page = options?.page ?? 1;
|
||||
|
||||
const params = new URLSearchParams();
|
||||
params.set('$top', String(pageSize));
|
||||
params.set('$skip', String((page - 1) * pageSize));
|
||||
// eAccounting paginates with $page/$pagesize (default 50, max 1000).
|
||||
// OData-style $top/$skip are silently IGNORED by the API: every request
|
||||
// returns page 1, so a multi-page fetch yields the first page N times
|
||||
// and never reaches the rest of the collection.
|
||||
params.set('$page', String(page));
|
||||
params.set('$pagesize', String(pageSize));
|
||||
|
||||
if (options?.modifiedSince && options?.modifiedField) {
|
||||
params.set(
|
||||
@@ -135,6 +139,11 @@ export class VismaClient {
|
||||
modifiedField: options?.modifiedField,
|
||||
});
|
||||
|
||||
// An empty page means the collection is exhausted regardless of what
|
||||
// Meta claims; trusting a stale/buggy TotalNumberOfPages here would
|
||||
// re-append duplicate rows or loop for nothing.
|
||||
if (result.items.length === 0) break;
|
||||
|
||||
allItems.push(...result.items);
|
||||
totalPages = result.totalPages;
|
||||
page++;
|
||||
|
||||
Reference in New Issue
Block a user