fix(migration): complete the rows of migrated sales invoices the hydration budget did not reach (#2291)
* fix(migration): complete the rows of migrated sales invoices the hydration budget did not reach The migration maps sales invoices from the provider's list payload and hydrates the detail form (rows, net, VAT) inside a fixed 90 s budget, open invoices first. Fortnox, Briox and Björn Lundén ship no rows in a list response, so every invoice the budget did not reach was imported as a header with a total and no invoice_items, and nothing ever came back for it: the wizard never showed the hydration report, so the user found out on the invoice page. Measured on prod today: Profilio 384 of 384 (migrated before hydration existed), Loftux 311 of 672, Damac 182 of 542, Clearstoq 1 125 of 1 125. - lib/providers: hydrateSalesInvoices() hydrates a caller-chosen subset of an already-listed register, so a follow-up can spend its budget on the invoices still incomplete on our side instead of re-walking the register open-first and never reaching the rest. - arcim-migration: completeMigratedInvoiceLines() starts from OUR row-less non-draft invoices, joins them to the provider register on number + date (unique on both sides), hydrates only that subset and writes each invoice's rows once the detail total matches the stored total to the öre. The header VAT split is rewritten only when the stored one holds no evidence (null rate, or a non-zero rate label beside 0 kr VAT and subtotal = total). Never the total, status, payments or a journal entry. - Hourly cron (/api/extensions/arcim-migration/complete-invoice-lines/cron, vercel.json + Docker crontabs) drives the pass over consents accepted in the last 60 days, newest first, with a per-company share of the run. - The wizard's result screen now shows "x av y fakturor hämtade med rader" and that the rest are fetched in the background within the hour. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DG5aYcshzKJ1EA7PPhGtVf * fix(migration): write the header VAT fill as a literal, raise the schema-guard ceiling for the row inserts The phantom-column scanner resolves only object-literal payloads. The header update is now a literal (so its six columns are checked); the two invoice_items inserts are runtime row arrays from mapSalesInvoiceLine, the same shape the orchestrator already inserts, so the ceiling moves 399 to 401 with the reason recorded beside the earlier ones. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DG5aYcshzKJ1EA7PPhGtVf * fix(migration): gate the completion cron on token freshness, not consent age, and visit every usable consent Two review findings held. Prod holds 57 accepted consents from the last 60 days, so a fixed page of the newest 25 would leave older companies with row-less invoices waiting behind companies that are already done: the cap is gone (a company with nothing left costs one query and no provider call). And the consent's created_at said nothing about whether its credentials still work: Fortnox refresh tokens live 45 days and rotate on every refresh, so eligibility is now read off the token row (access token expired within the last 45 days, or no expiry at all), which also stops a dead consent from being retried every hour. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DG5aYcshzKJ1EA7PPhGtVf --------- Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Jakob Wennberg
co-authored by
Claude Fable 5.1
Jakob Wennberg
Claude Fable 5.1
Jakob Wennberg
parent
0bd3c27fba
commit
8e1f9d5201
@@ -0,0 +1,370 @@
|
||||
import { describe, it, expect, beforeEach, vi, type Mock } from 'vitest'
|
||||
import type { SupabaseClient } from '@supabase/supabase-js'
|
||||
import type { SalesInvoiceDto } from '@/lib/providers/dto'
|
||||
|
||||
/**
|
||||
* The follow-up pass for migrated sales invoices imported without rows
|
||||
* (Fortnox, Briox and Björn Lundén ship none in a list payload, and the
|
||||
* migration's detail hydration is budget-bounded). It must start from OUR
|
||||
* row-less invoices, join strictly, hydrate only that subset, write rows only
|
||||
* when the provider's total agrees with the stored one, and leave anything it
|
||||
* could not reach for the next run rather than guessing.
|
||||
*/
|
||||
|
||||
vi.mock('@/lib/providers/resolve-consent', () => ({
|
||||
resolveConsent: vi.fn().mockResolvedValue({
|
||||
consent: { provider: 'fortnox' },
|
||||
accessToken: 'tok',
|
||||
providerCompanyId: undefined,
|
||||
}),
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/providers/provider-data-fetcher', () => ({
|
||||
fetchSalesInvoicesDirect: vi.fn(),
|
||||
hydrateSalesInvoices: vi.fn(),
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/supabase/fetch-all', () => ({ fetchAllRows: vi.fn() }))
|
||||
|
||||
import { resolveConsent } from '@/lib/providers/resolve-consent'
|
||||
import { fetchAllRows } from '@/lib/supabase/fetch-all'
|
||||
import { fetchSalesInvoicesDirect, hydrateSalesInvoices } from '@/lib/providers/provider-data-fetcher'
|
||||
import { completeMigratedInvoiceLines } from '../complete-invoice-lines'
|
||||
|
||||
const mResolve = resolveConsent as Mock
|
||||
const mFetchAll = fetchAllRows as Mock
|
||||
const mList = fetchSalesInvoicesDirect as Mock
|
||||
const mHydrate = hydrateSalesInvoices as Mock
|
||||
|
||||
const HYDRATION = { needed: 0, hydrated: 0, failed: 0, skippedForBudget: 0 }
|
||||
|
||||
const amount = (value: number, currencyCode = 'SEK') => ({ value, currencyCode })
|
||||
|
||||
/** A hydrated Fortnox-shaped invoice: 1 000 kr net, 25 % VAT, 1 250 kr total. */
|
||||
function providerInvoice(overrides: Partial<SalesInvoiceDto> = {}): SalesInvoiceDto {
|
||||
return {
|
||||
id: '1001',
|
||||
invoiceNumber: '1001',
|
||||
issueDate: '2026-03-14',
|
||||
currencyCode: 'SEK',
|
||||
status: 'paid',
|
||||
supplier: { name: 'Profilio Sweden AB', identifications: [] },
|
||||
customer: { name: 'Kund AB', identifications: [] },
|
||||
lines: [
|
||||
{
|
||||
id: '1',
|
||||
description: 'Konsulttid',
|
||||
quantity: 10,
|
||||
unitCode: 'h',
|
||||
unitPrice: amount(100),
|
||||
lineExtensionAmount: amount(1000),
|
||||
taxPercent: 25,
|
||||
},
|
||||
],
|
||||
taxTotal: { taxAmount: amount(250) },
|
||||
legalMonetaryTotal: {
|
||||
lineExtensionAmount: amount(1000),
|
||||
taxInclusiveAmount: amount(1250),
|
||||
payableAmount: amount(1250),
|
||||
},
|
||||
paymentStatus: { paid: true, balance: amount(0) },
|
||||
...overrides,
|
||||
}
|
||||
}
|
||||
|
||||
/** A stored row as the pre-#1745 import left it: 25 % label, 0 kr VAT, no rows. */
|
||||
function storedRow(overrides: Record<string, unknown> = {}) {
|
||||
return {
|
||||
id: 'inv-1',
|
||||
user_id: 'user-1',
|
||||
customer_id: 'cust-1',
|
||||
invoice_number: '1001',
|
||||
invoice_date: '2026-03-14',
|
||||
total: 1250,
|
||||
subtotal: 1250,
|
||||
vat_amount: 0,
|
||||
vat_rate: 25,
|
||||
currency: 'SEK',
|
||||
exchange_rate: null,
|
||||
invoice_items: [],
|
||||
...overrides,
|
||||
}
|
||||
}
|
||||
|
||||
/** Every hydrated as given, in order, nothing left unhydrated. */
|
||||
function hydratedAll(invoices: SalesInvoiceDto[], unhydratedIds: string[] = []) {
|
||||
return {
|
||||
invoices,
|
||||
hydration: { ...HYDRATION, needed: invoices.length, hydrated: invoices.length - unhydratedIds.length },
|
||||
unhydratedIds: new Set(unhydratedIds),
|
||||
}
|
||||
}
|
||||
|
||||
interface Call { table: string; method: string; args: unknown[] }
|
||||
|
||||
/**
|
||||
* Thenable query-builder stand-in. Records every call; resolves with what
|
||||
* `respond` returns for the table and the methods used on the chain.
|
||||
*/
|
||||
function makeSupabase(respond: (table: string, methods: string[], calls: Call[]) => unknown) {
|
||||
const calls: Call[] = []
|
||||
const from = vi.fn((table: string) => {
|
||||
const chain: Call[] = []
|
||||
const builder: Record<string, unknown> = {}
|
||||
for (const method of ['select', 'in', 'insert', 'update', 'eq', 'neq', 'order', 'range']) {
|
||||
builder[method] = (...args: unknown[]) => {
|
||||
const call = { table, method, args }
|
||||
chain.push(call)
|
||||
calls.push(call)
|
||||
return builder
|
||||
}
|
||||
}
|
||||
builder.then = (resolve: (v: unknown) => void, reject: (e: unknown) => void) =>
|
||||
Promise.resolve()
|
||||
.then(() => respond(table, chain.map((c) => c.method), chain))
|
||||
.then(resolve, reject)
|
||||
return builder
|
||||
})
|
||||
return { supabase: { from } as unknown as SupabaseClient, calls }
|
||||
}
|
||||
|
||||
const ok = { data: [], error: null }
|
||||
|
||||
function insertedRows(calls: Call[]): Record<string, unknown>[] {
|
||||
return calls
|
||||
.filter((c) => c.table === 'invoice_items' && c.method === 'insert')
|
||||
.flatMap((c) => c.args[0] as Record<string, unknown>[])
|
||||
}
|
||||
|
||||
function headerUpdates(calls: Call[]): Record<string, unknown>[] {
|
||||
return calls
|
||||
.filter((c) => c.table === 'invoices' && c.method === 'update')
|
||||
.map((c) => c.args[0] as Record<string, unknown>)
|
||||
}
|
||||
|
||||
describe('completeMigratedInvoiceLines', () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
mResolve.mockResolvedValue({ consent: { provider: 'fortnox' }, accessToken: 'tok', providerCompanyId: undefined })
|
||||
})
|
||||
|
||||
it('writes the rows and fills a header that held no VAT evidence', async () => {
|
||||
mFetchAll.mockResolvedValue([storedRow()])
|
||||
const dto = providerInvoice()
|
||||
mList.mockResolvedValue([dto])
|
||||
mHydrate.mockResolvedValue(hydratedAll([dto]))
|
||||
const { supabase, calls } = makeSupabase(() => ok)
|
||||
|
||||
const result = await completeMigratedInvoiceLines({ supabase, companyId: 'co-1', consentId: 'c-1' })
|
||||
|
||||
expect(result).toMatchObject({
|
||||
candidates: 1, providerInvoices: 1, matched: 1, unmatched: 0,
|
||||
completed: 1, headersUpdated: 1, remaining: 0, totalMismatch: 0, notHydrated: 0, failed: 0,
|
||||
})
|
||||
// Only the matched subset is hydrated, so the budget is never spent on
|
||||
// invoices already complete on our side.
|
||||
expect(mHydrate).toHaveBeenCalledTimes(1)
|
||||
expect(mHydrate.mock.calls[0][3]).toEqual([dto])
|
||||
|
||||
const rows = insertedRows(calls)
|
||||
expect(rows).toHaveLength(1)
|
||||
expect(rows[0]).toMatchObject({
|
||||
invoice_id: 'inv-1',
|
||||
sort_order: 1,
|
||||
description: 'Konsulttid',
|
||||
quantity: 10,
|
||||
unit: 'h',
|
||||
unit_price: 100,
|
||||
line_total: 1000,
|
||||
vat_rate: 25,
|
||||
vat_amount: 250,
|
||||
})
|
||||
|
||||
const headers = headerUpdates(calls)
|
||||
expect(headers).toHaveLength(1)
|
||||
expect(headers[0]).toEqual({
|
||||
subtotal: 1000,
|
||||
subtotal_sek: 1000,
|
||||
vat_amount: 250,
|
||||
vat_amount_sek: 250,
|
||||
vat_rate: 25,
|
||||
vat_treatment: 'standard_25',
|
||||
})
|
||||
// Scoped to the company as well as the id: defense in depth on a
|
||||
// service-role client.
|
||||
const update = calls.find((c) => c.table === 'invoices' && c.method === 'update')!
|
||||
const scope = calls.filter((c) => c.table === 'invoices' && c.method === 'eq' && calls.indexOf(c) > calls.indexOf(update))
|
||||
expect(scope.map((c) => c.args)).toEqual([['id', 'inv-1'], ['company_id', 'co-1']])
|
||||
// Never the total, status or payments.
|
||||
expect(Object.keys(headers[0])).not.toContain('total')
|
||||
expect(Object.keys(headers[0])).not.toContain('status')
|
||||
})
|
||||
|
||||
it('writes the rows but leaves a header whose split is consistent (momsfri)', async () => {
|
||||
mFetchAll.mockResolvedValue([storedRow({ vat_rate: 0, vat_amount: 0, subtotal: 1250 })])
|
||||
const dto = providerInvoice()
|
||||
mList.mockResolvedValue([dto])
|
||||
mHydrate.mockResolvedValue(hydratedAll([dto]))
|
||||
const { supabase, calls } = makeSupabase(() => ok)
|
||||
|
||||
const result = await completeMigratedInvoiceLines({ supabase, companyId: 'co-1', consentId: 'c-1' })
|
||||
|
||||
expect(result).toMatchObject({ completed: 1, headersUpdated: 0 })
|
||||
expect(insertedRows(calls)).toHaveLength(1)
|
||||
expect(headerUpdates(calls)).toHaveLength(0)
|
||||
})
|
||||
|
||||
it('fills a header whose rate is null (the post-#1745 "source did not say")', async () => {
|
||||
mFetchAll.mockResolvedValue([storedRow({ vat_rate: null, vat_amount: 0, subtotal: 1250 })])
|
||||
const dto = providerInvoice()
|
||||
mList.mockResolvedValue([dto])
|
||||
mHydrate.mockResolvedValue(hydratedAll([dto]))
|
||||
const { supabase, calls } = makeSupabase(() => ok)
|
||||
|
||||
const result = await completeMigratedInvoiceLines({ supabase, companyId: 'co-1', consentId: 'c-1' })
|
||||
|
||||
expect(result).toMatchObject({ completed: 1, headersUpdated: 1 })
|
||||
expect(headerUpdates(calls)[0]).toMatchObject({ vat_rate: 25, vat_amount: 250, subtotal: 1000 })
|
||||
})
|
||||
|
||||
it('derives the SEK twins from the rate the row already carries', async () => {
|
||||
mFetchAll.mockResolvedValue([storedRow({ currency: 'EUR', exchange_rate: 11.2, total: 1250 })])
|
||||
const dto = providerInvoice({ currencyCode: 'EUR' })
|
||||
mList.mockResolvedValue([dto])
|
||||
mHydrate.mockResolvedValue(hydratedAll([dto]))
|
||||
const { supabase, calls } = makeSupabase(() => ok)
|
||||
|
||||
await completeMigratedInvoiceLines({ supabase, companyId: 'co-1', consentId: 'c-1' })
|
||||
|
||||
expect(headerUpdates(calls)[0]).toMatchObject({ subtotal: 1000, subtotal_sek: 11200, vat_amount: 250, vat_amount_sek: 2800 })
|
||||
})
|
||||
|
||||
it('leaves an invoice untouched when the provider total differs from the stored one', async () => {
|
||||
mFetchAll.mockResolvedValue([storedRow({ total: 1300, subtotal: 1300 })])
|
||||
const dto = providerInvoice()
|
||||
mList.mockResolvedValue([dto])
|
||||
mHydrate.mockResolvedValue(hydratedAll([dto]))
|
||||
const { supabase, calls } = makeSupabase(() => ok)
|
||||
|
||||
const result = await completeMigratedInvoiceLines({ supabase, companyId: 'co-1', consentId: 'c-1' })
|
||||
|
||||
expect(result).toMatchObject({ matched: 1, completed: 0, totalMismatch: 1, remaining: 1 })
|
||||
expect(insertedRows(calls)).toHaveLength(0)
|
||||
expect(headerUpdates(calls)).toHaveLength(0)
|
||||
})
|
||||
|
||||
it('reverses the rows of a kreditfaktura the way the migration does', async () => {
|
||||
mFetchAll.mockResolvedValue([storedRow({ total: -1250, subtotal: -1250 })])
|
||||
const dto = providerInvoice({ invoiceTypeCode: '381', status: 'credited' })
|
||||
mList.mockResolvedValue([dto])
|
||||
mHydrate.mockResolvedValue(hydratedAll([dto]))
|
||||
const { supabase, calls } = makeSupabase(() => ok)
|
||||
|
||||
const result = await completeMigratedInvoiceLines({ supabase, companyId: 'co-1', consentId: 'c-1' })
|
||||
|
||||
expect(result).toMatchObject({ completed: 1, totalMismatch: 0 })
|
||||
expect(insertedRows(calls)[0]).toMatchObject({ quantity: -10, unit_price: 100, line_total: -1000, vat_amount: -250 })
|
||||
expect(headerUpdates(calls)[0]).toMatchObject({ subtotal: -1000, vat_amount: -250 })
|
||||
})
|
||||
|
||||
it('leaves invoices the budget did not reach for the next run', async () => {
|
||||
mFetchAll.mockResolvedValue([storedRow(), storedRow({ id: 'inv-2', invoice_number: '1002' })])
|
||||
const first = providerInvoice()
|
||||
const second = providerInvoice({ id: '1002', invoiceNumber: '1002', lines: [] })
|
||||
mList.mockResolvedValue([first, second])
|
||||
// The second came back in list form only (no rows): the budget ran out.
|
||||
mHydrate.mockResolvedValue(hydratedAll([first, second], ['1002']))
|
||||
const { supabase, calls } = makeSupabase(() => ok)
|
||||
|
||||
const result = await completeMigratedInvoiceLines({ supabase, companyId: 'co-1', consentId: 'c-1' })
|
||||
|
||||
expect(result).toMatchObject({ candidates: 2, matched: 2, completed: 1, notHydrated: 1, remaining: 1 })
|
||||
expect(insertedRows(calls).map((r) => r.invoice_id)).toEqual(['inv-1'])
|
||||
})
|
||||
|
||||
it('does not join an ambiguous key, and does not hydrate when nothing joined', async () => {
|
||||
// Two stored rows with the same number and date: a wrong join would put
|
||||
// one invoice's rows under the other.
|
||||
mFetchAll.mockResolvedValue([storedRow(), storedRow({ id: 'inv-dup' })])
|
||||
mList.mockResolvedValue([providerInvoice()])
|
||||
const { supabase, calls } = makeSupabase(() => ok)
|
||||
|
||||
const result = await completeMigratedInvoiceLines({ supabase, companyId: 'co-1', consentId: 'c-1' })
|
||||
|
||||
expect(result).toMatchObject({ candidates: 2, matched: 0, unmatched: 2, completed: 0, remaining: 2 })
|
||||
expect(mHydrate).not.toHaveBeenCalled()
|
||||
expect(insertedRows(calls)).toHaveLength(0)
|
||||
})
|
||||
|
||||
it('costs one query and no provider call when the company has nothing to complete', async () => {
|
||||
mFetchAll.mockResolvedValue([storedRow({ invoice_items: [{ id: 'item-1' }] })])
|
||||
const { supabase, calls } = makeSupabase(() => ok)
|
||||
|
||||
const result = await completeMigratedInvoiceLines({ supabase, companyId: 'co-1', consentId: 'c-1' })
|
||||
|
||||
expect(result).toMatchObject({ candidates: 0, completed: 0, remaining: 0 })
|
||||
expect(mResolve).not.toHaveBeenCalled()
|
||||
expect(mList).not.toHaveBeenCalled()
|
||||
expect(calls).toHaveLength(0)
|
||||
})
|
||||
|
||||
it('skips an invoice that gained rows since the candidates were loaded', async () => {
|
||||
mFetchAll.mockResolvedValue([storedRow()])
|
||||
const dto = providerInvoice()
|
||||
mList.mockResolvedValue([dto])
|
||||
mHydrate.mockResolvedValue(hydratedAll([dto]))
|
||||
const { supabase, calls } = makeSupabase((table, methods) =>
|
||||
table === 'invoice_items' && methods.includes('in') ? { data: [{ invoice_id: 'inv-1' }], error: null } : ok,
|
||||
)
|
||||
|
||||
const result = await completeMigratedInvoiceLines({ supabase, companyId: 'co-1', consentId: 'c-1' })
|
||||
|
||||
expect(result).toMatchObject({ completed: 0, failed: 0, remaining: 1 })
|
||||
expect(insertedRows(calls)).toHaveLength(0)
|
||||
expect(headerUpdates(calls)).toHaveLength(0)
|
||||
})
|
||||
|
||||
it('retries per invoice when the batch insert fails, and counts the offender', async () => {
|
||||
mFetchAll.mockResolvedValue([storedRow(), storedRow({ id: 'inv-2', invoice_number: '1002' })])
|
||||
const first = providerInvoice()
|
||||
const second = providerInvoice({ id: '1002', invoiceNumber: '1002' })
|
||||
mList.mockResolvedValue([first, second])
|
||||
mHydrate.mockResolvedValue(hydratedAll([first, second]))
|
||||
const { supabase, calls } = makeSupabase((table, methods, chain) => {
|
||||
if (table !== 'invoice_items' || !methods.includes('insert')) return ok
|
||||
const rows = chain[0].args[0] as { invoice_id: string }[]
|
||||
if (rows.length > 1) return { data: null, error: { message: 'batch rejected' } }
|
||||
return rows[0].invoice_id === 'inv-2' ? { data: null, error: { message: 'check violation' } } : ok
|
||||
})
|
||||
|
||||
const result = await completeMigratedInvoiceLines({ supabase, companyId: 'co-1', consentId: 'c-1' })
|
||||
|
||||
expect(result).toMatchObject({ completed: 1, failed: 1, headersUpdated: 1, remaining: 1 })
|
||||
expect(headerUpdates(calls)).toHaveLength(1)
|
||||
})
|
||||
|
||||
it('dry run: reports the plan and writes nothing', async () => {
|
||||
mFetchAll.mockResolvedValue([storedRow()])
|
||||
const dto = providerInvoice()
|
||||
mList.mockResolvedValue([dto])
|
||||
mHydrate.mockResolvedValue(hydratedAll([dto]))
|
||||
const { supabase, calls } = makeSupabase(() => ok)
|
||||
|
||||
const result = await completeMigratedInvoiceLines({ supabase, companyId: 'co-1', consentId: 'c-1', dryRun: true })
|
||||
|
||||
expect(result).toMatchObject({ dryRun: true, completed: 1, headersUpdated: 1, remaining: 0 })
|
||||
expect(calls).toHaveLength(0)
|
||||
})
|
||||
|
||||
it('passes the budget through to the hydration', async () => {
|
||||
mFetchAll.mockResolvedValue([storedRow()])
|
||||
const dto = providerInvoice()
|
||||
mList.mockResolvedValue([dto])
|
||||
mHydrate.mockResolvedValue(hydratedAll([dto]))
|
||||
const { supabase } = makeSupabase(() => ok)
|
||||
|
||||
await completeMigratedInvoiceLines({ supabase, companyId: 'co-1', consentId: 'c-1', budgetMs: 45_000 })
|
||||
|
||||
expect(mHydrate.mock.calls[0][4]).toBe(45_000)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,409 @@
|
||||
/**
|
||||
* Finish a migrated sales-invoice register: fetch the rows (and the VAT split)
|
||||
* for the invoices that were imported without them.
|
||||
*
|
||||
* The migration maps invoices from the provider's LIST payload and hydrates
|
||||
* the detail form inside a fixed budget (see lib/providers/provider-data-
|
||||
* fetcher.ts). Fortnox, Briox and Björn Lundén ship no rows in a list
|
||||
* response, so every invoice the budget did not reach lands as a header with
|
||||
* a total and no invoice_items behind it: Profilio (384 of 384, migrated
|
||||
* before hydration existed), Loftux (311 of 672), Damac (182 of 542),
|
||||
* Clearstoq (1 125 of 1 125). Nothing ever came back for them, and the wizard
|
||||
* had not said so.
|
||||
*
|
||||
* This pass is the follow-up. It is re-runnable and makes progress on every
|
||||
* run: it starts from OUR side (the non-draft invoices in this company that
|
||||
* have no rows), joins them to the provider's register on invoice number AND
|
||||
* date (unique on both sides, the same strictness as the registration-voucher
|
||||
* relink), hydrates only those, and writes each invoice's rows once the
|
||||
* provider's detail total agrees with the stored total to the öre. A run that
|
||||
* runs out of budget leaves the rest for the next one; nothing is guessed and
|
||||
* nothing is inserted for an invoice the provider does not know.
|
||||
*
|
||||
* What it writes, and only this:
|
||||
* - invoice_items for an invoice that has none;
|
||||
* - the header VAT split (subtotal, vat_amount, vat_rate, vat_treatment and
|
||||
* their SEK twins) when the stored split is the "unknown" shape the old
|
||||
* import left behind (no rate, or 0 kr VAT beside subtotal = total).
|
||||
* Never the total, the status, the customer, payments or any journal entry:
|
||||
* momsdeklaration and every report read the ledger, not these columns, so
|
||||
* filling them changes what the invoice page shows and nothing that was
|
||||
* filed (see the 2026-08-22 verification in DECISIONS.md).
|
||||
*/
|
||||
|
||||
import type { SupabaseClient } from '@supabase/supabase-js'
|
||||
import { ISO_DATE_RE } from '@/lib/invariants'
|
||||
import { createLogger } from '@/lib/logger'
|
||||
import { equalOre, roundOre } from '@/lib/money'
|
||||
import { chunk } from '@/lib/utils'
|
||||
import type { SalesInvoiceDto } from '@/lib/providers/dto'
|
||||
import type { ProviderName } from '@/lib/providers/types'
|
||||
import { resolveConsent } from '@/lib/providers/resolve-consent'
|
||||
import {
|
||||
fetchSalesInvoicesDirect,
|
||||
hydrateSalesInvoices,
|
||||
type HydrationReport,
|
||||
} from '@/lib/providers/provider-data-fetcher'
|
||||
import { fetchAllRows } from '@/lib/supabase/fetch-all'
|
||||
import { mapSalesInvoice } from './entity-mapper'
|
||||
|
||||
const log = createLogger('extensions/arcim-migration/complete-invoice-lines')
|
||||
|
||||
export interface CompleteInvoiceLinesOptions {
|
||||
supabase: SupabaseClient
|
||||
companyId: string
|
||||
consentId: string
|
||||
/** Report the plan without writing. */
|
||||
dryRun?: boolean
|
||||
/** Wall-clock ceiling for the provider detail fetches, in ms. */
|
||||
budgetMs?: number
|
||||
}
|
||||
|
||||
export interface CompleteInvoiceLinesResult {
|
||||
/** Non-draft invoices in this company that carry no rows. */
|
||||
candidates: number
|
||||
/** Sales invoices the provider's register lists. */
|
||||
providerInvoices: number
|
||||
/** Candidates that joined to exactly one provider invoice (number + date). */
|
||||
matched: number
|
||||
/** Candidates with no unique provider counterpart: native rows, ambiguous keys, or gone at the provider. */
|
||||
unmatched: number
|
||||
/** Invoices whose rows were written this run (or would be, on a dry run). */
|
||||
completed: number
|
||||
/** Of `completed`, those whose header VAT split was rewritten from the detail form. */
|
||||
headersUpdated: number
|
||||
/** Matched and hydrated, but the provider's total differs from the stored one; left untouched. */
|
||||
totalMismatch: number
|
||||
/** Matched and hydrated, but the detail form itself carries no rows. */
|
||||
noLinesAtProvider: number
|
||||
/** Matched but not hydrated this run (budget, auth, or a failed fetch); the next run retries them. */
|
||||
notHydrated: number
|
||||
/** Rows written but the header left alone because the detail form established no VAT. */
|
||||
vatUnresolved: number
|
||||
/** Invoices whose write failed at the database. */
|
||||
failed: number
|
||||
/** Candidates still without rows after this run: `candidates - completed`. */
|
||||
remaining: number
|
||||
hydration: HydrationReport
|
||||
dryRun: boolean
|
||||
}
|
||||
|
||||
interface CandidateRow {
|
||||
id: string
|
||||
user_id: string
|
||||
customer_id: string
|
||||
invoice_number: string | null
|
||||
invoice_date: string
|
||||
total: number
|
||||
subtotal: number | null
|
||||
vat_amount: number | null
|
||||
vat_rate: number | null
|
||||
currency: string | null
|
||||
exchange_rate: number | null
|
||||
invoice_items: { id: string }[] | null
|
||||
}
|
||||
|
||||
/** Invoices per statement. Small enough that a chunk's rows stay one request. */
|
||||
const WRITE_CHUNK_SIZE = 100
|
||||
|
||||
/**
|
||||
* "number::YYYY-MM-DD", the same key the registration-voucher relink joins
|
||||
* on. A date that does not start like an ISO date joins nothing.
|
||||
*/
|
||||
function joinKey(number: string | null | undefined, date: string | null | undefined): string | null {
|
||||
if (!number || !date) return null
|
||||
const day = date.slice(0, 10)
|
||||
return ISO_DATE_RE.test(day) ? `${number}::${day}` : null
|
||||
}
|
||||
|
||||
/** A map that forgets keys seen more than once, so those are never joined on. */
|
||||
function uniqueByKey<T>(rows: readonly T[], keyOf: (row: T) => string | null): Map<string, T> {
|
||||
const out = new Map<string, T>()
|
||||
const dupes = new Set<string>()
|
||||
for (const row of rows) {
|
||||
const key = keyOf(row)
|
||||
if (!key) continue
|
||||
if (out.has(key) || dupes.has(key)) {
|
||||
out.delete(key)
|
||||
dupes.add(key)
|
||||
continue
|
||||
}
|
||||
out.set(key, row)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* Is the stored VAT split the shape the old import left behind?
|
||||
*
|
||||
* A null rate is the post-#1745 "the source did not say". A non-zero rate
|
||||
* label beside 0 kr of VAT and subtotal = total is what the pre-#1745 mapper
|
||||
* wrote for every Fortnox invoice (25 % on top of nothing): a contradiction,
|
||||
* not a fact. Both mean the columns hold no evidence, so the detail form may
|
||||
* overwrite them. A split that is consistent with itself, including a
|
||||
* genuinely momsfri one (rate 0, 0 kr VAT, subtotal = total), is evidence
|
||||
* and is left as it is; the rows are still written.
|
||||
*/
|
||||
function headerHoldsNoVatEvidence(row: CandidateRow): boolean {
|
||||
if (row.vat_rate === null) return true
|
||||
if (row.vat_rate === 0) return false
|
||||
const subtotal = row.subtotal ?? row.total
|
||||
const vat = row.vat_amount ?? 0
|
||||
return equalOre(vat, 0) && equalOre(subtotal, row.total)
|
||||
}
|
||||
|
||||
/** SEK twin of an amount on this row, or null when the row has no rate. */
|
||||
function toRowSek(amount: number, row: CandidateRow): number | null {
|
||||
if (!row.currency || row.currency.toUpperCase() === 'SEK') return roundOre(amount)
|
||||
return row.exchange_rate != null ? roundOre(amount * row.exchange_rate) : null
|
||||
}
|
||||
|
||||
async function loadCandidates(supabase: SupabaseClient, companyId: string): Promise<CandidateRow[]> {
|
||||
const rows = await fetchAllRows<CandidateRow>(({ from, to }) =>
|
||||
supabase
|
||||
.from('invoices')
|
||||
.select(
|
||||
'id, user_id, customer_id, invoice_number, invoice_date, total, subtotal, vat_amount, vat_rate, currency, exchange_rate, invoice_items(id)',
|
||||
)
|
||||
.eq('company_id', companyId)
|
||||
.eq('document_type', 'invoice')
|
||||
// A draft without rows is a draft someone is still writing, not an
|
||||
// import shortfall (a migrated row is a draft only when the source had
|
||||
// no voucher for it anyway).
|
||||
.neq('status', 'draft')
|
||||
.order('id', { ascending: true })
|
||||
.range(from, to),
|
||||
)
|
||||
return rows.filter((row) => (row.invoice_items?.length ?? 0) === 0)
|
||||
}
|
||||
|
||||
/** Ids among `ids` that gained rows since the candidates were loaded. */
|
||||
async function alreadyFilled(supabase: SupabaseClient, ids: string[]): Promise<Set<string>> {
|
||||
const { data, error } = await supabase
|
||||
.from('invoice_items')
|
||||
.select('invoice_id')
|
||||
.in('invoice_id', ids)
|
||||
if (error) throw new Error(`invoice_items lookup failed: ${error.message}`)
|
||||
return new Set(((data ?? []) as { invoice_id: string }[]).map((r) => r.invoice_id))
|
||||
}
|
||||
|
||||
/** The header VAT split the detail form established, ready to write. */
|
||||
interface HeaderFill {
|
||||
subtotal: number
|
||||
subtotalSek: number | null
|
||||
vatAmount: number
|
||||
vatAmountSek: number | null
|
||||
vatRate: number | null
|
||||
vatTreatment: string
|
||||
}
|
||||
|
||||
interface PlannedWrite {
|
||||
row: CandidateRow
|
||||
items: Record<string, unknown>[]
|
||||
header: HeaderFill | null
|
||||
}
|
||||
|
||||
export async function completeMigratedInvoiceLines(
|
||||
options: CompleteInvoiceLinesOptions,
|
||||
): Promise<CompleteInvoiceLinesResult> {
|
||||
const { supabase, companyId, consentId, dryRun = false, budgetMs } = options
|
||||
|
||||
const result: CompleteInvoiceLinesResult = {
|
||||
candidates: 0,
|
||||
providerInvoices: 0,
|
||||
matched: 0,
|
||||
unmatched: 0,
|
||||
completed: 0,
|
||||
headersUpdated: 0,
|
||||
totalMismatch: 0,
|
||||
noLinesAtProvider: 0,
|
||||
notHydrated: 0,
|
||||
vatUnresolved: 0,
|
||||
failed: 0,
|
||||
remaining: 0,
|
||||
hydration: { needed: 0, hydrated: 0, failed: 0, skippedForBudget: 0 },
|
||||
dryRun,
|
||||
}
|
||||
|
||||
// Our side first, and before the consent is touched: a company with nothing
|
||||
// to complete costs one query and no token refresh at the provider.
|
||||
const candidates = await loadCandidates(supabase, companyId)
|
||||
result.candidates = candidates.length
|
||||
if (candidates.length === 0) return result
|
||||
|
||||
const resolved = await resolveConsent(companyId, consentId)
|
||||
const provider = resolved.consent.provider as ProviderName
|
||||
|
||||
const listed = await fetchSalesInvoicesDirect(provider, resolved.accessToken, resolved.providerCompanyId)
|
||||
result.providerInvoices = listed.length
|
||||
|
||||
const providerByKey = uniqueByKey(listed, (dto) => joinKey(dto.invoiceNumber, dto.issueDate))
|
||||
const oursByKey = uniqueByKey(candidates, (row) => joinKey(row.invoice_number, row.invoice_date))
|
||||
|
||||
const pairs: { row: CandidateRow; dto: SalesInvoiceDto }[] = []
|
||||
for (const [key, row] of oursByKey) {
|
||||
const dto = providerByKey.get(key)
|
||||
if (dto) pairs.push({ row, dto })
|
||||
}
|
||||
result.matched = pairs.length
|
||||
result.unmatched = candidates.length - pairs.length
|
||||
|
||||
if (pairs.length === 0) {
|
||||
result.remaining = candidates.length
|
||||
return result
|
||||
}
|
||||
|
||||
// Only the matched subset is hydrated, so every run spends its budget on
|
||||
// invoices that are still incomplete here, never on ones already done.
|
||||
const hydrated = await hydrateSalesInvoices(
|
||||
provider,
|
||||
resolved.accessToken,
|
||||
resolved.providerCompanyId,
|
||||
pairs.map((pair) => pair.dto),
|
||||
budgetMs,
|
||||
)
|
||||
result.hydration = hydrated.hydration
|
||||
|
||||
const planned: PlannedWrite[] = []
|
||||
for (let i = 0; i < pairs.length; i++) {
|
||||
const { row } = pairs[i]
|
||||
const dto = hydrated.invoices[i]
|
||||
|
||||
if (hydrated.unhydratedIds.has(dto.id)) {
|
||||
result.notHydrated++
|
||||
continue
|
||||
}
|
||||
if (dto.lines.length === 0) {
|
||||
result.noLinesAtProvider++
|
||||
continue
|
||||
}
|
||||
|
||||
// The same mapper the migration used, so a row written here is
|
||||
// indistinguishable from one written by a fully hydrated import. No FX
|
||||
// index: the SEK twins are derived from the rate the row already carries.
|
||||
const mapped = mapSalesInvoice(dto, row.user_id, companyId, row.customer_id)
|
||||
const mappedTotal = mapped.invoice.total as number
|
||||
if (!equalOre(mappedTotal, row.total)) {
|
||||
result.totalMismatch++
|
||||
log.warn('detail total differs from the stored total; invoice left untouched', {
|
||||
companyId, invoiceId: row.id, invoiceNumber: row.invoice_number, stored: row.total, provider: mappedTotal,
|
||||
})
|
||||
continue
|
||||
}
|
||||
|
||||
let header: HeaderFill | null = null
|
||||
if (mapped.vatUnresolved) {
|
||||
result.vatUnresolved++
|
||||
} else if (headerHoldsNoVatEvidence(row)) {
|
||||
const subtotal = mapped.invoice.subtotal as number
|
||||
const vatAmount = mapped.invoice.vat_amount as number
|
||||
header = {
|
||||
subtotal,
|
||||
subtotalSek: toRowSek(subtotal, row),
|
||||
vatAmount,
|
||||
vatAmountSek: toRowSek(vatAmount, row),
|
||||
vatRate: mapped.invoice.vat_rate as number | null,
|
||||
vatTreatment: mapped.invoice.vat_treatment as string,
|
||||
}
|
||||
}
|
||||
|
||||
planned.push({
|
||||
row,
|
||||
items: mapped.items.map((item) => ({ ...item, invoice_id: row.id })),
|
||||
header,
|
||||
})
|
||||
}
|
||||
|
||||
if (dryRun) {
|
||||
result.completed = planned.length
|
||||
result.headersUpdated = planned.filter((p) => p.header).length
|
||||
result.remaining = candidates.length - result.completed
|
||||
return result
|
||||
}
|
||||
|
||||
for (const batch of chunk(planned, WRITE_CHUNK_SIZE)) {
|
||||
// A concurrent run (the wizard and the cron, or two crons overlapping)
|
||||
// may have filled some of these since the candidates were loaded. Rows
|
||||
// are appended, never replaced, so a second write would double them.
|
||||
const filled = await alreadyFilled(supabase, batch.map((p) => p.row.id))
|
||||
const todo = batch.filter((p) => !filled.has(p.row.id))
|
||||
|
||||
const written = await insertRows(supabase, todo)
|
||||
for (const plan of todo) {
|
||||
if (!written.has(plan.row.id)) {
|
||||
result.failed++
|
||||
continue
|
||||
}
|
||||
result.completed++
|
||||
if (!plan.header) continue
|
||||
// Written as a literal so the schema guard checks these columns.
|
||||
const { error } = await supabase
|
||||
.from('invoices')
|
||||
.update({
|
||||
subtotal: plan.header.subtotal,
|
||||
subtotal_sek: plan.header.subtotalSek,
|
||||
vat_amount: plan.header.vatAmount,
|
||||
vat_amount_sek: plan.header.vatAmountSek,
|
||||
vat_rate: plan.header.vatRate,
|
||||
vat_treatment: plan.header.vatTreatment,
|
||||
})
|
||||
.eq('id', plan.row.id)
|
||||
.eq('company_id', companyId)
|
||||
if (error) {
|
||||
// The rows landed; only the header split is still the old shape. The
|
||||
// next run will not revisit this invoice (it now has rows), so say so.
|
||||
log.error('header VAT update failed after the rows were written', {
|
||||
companyId, invoiceId: plan.row.id, reason: error.message,
|
||||
})
|
||||
continue
|
||||
}
|
||||
result.headersUpdated++
|
||||
}
|
||||
}
|
||||
|
||||
result.remaining = candidates.length - result.completed
|
||||
log.info('migrated invoice rows completed', {
|
||||
companyId,
|
||||
candidates: result.candidates,
|
||||
matched: result.matched,
|
||||
completed: result.completed,
|
||||
headersUpdated: result.headersUpdated,
|
||||
notHydrated: result.notHydrated,
|
||||
totalMismatch: result.totalMismatch,
|
||||
failed: result.failed,
|
||||
remaining: result.remaining,
|
||||
hydration: result.hydration,
|
||||
})
|
||||
return result
|
||||
}
|
||||
|
||||
/**
|
||||
* Insert every plan's rows: one statement for the batch, and on failure one
|
||||
* statement per invoice so a single bad row rejects its own invoice, not the
|
||||
* hundred beside it. An invoice's rows never split across statements: it
|
||||
* either has all of them or none.
|
||||
*/
|
||||
async function insertRows(supabase: SupabaseClient, plans: PlannedWrite[]): Promise<Set<string>> {
|
||||
const written = new Set<string>()
|
||||
if (plans.length === 0) return written
|
||||
|
||||
const bulk = await supabase.from('invoice_items').insert(plans.flatMap((p) => p.items))
|
||||
if (!bulk.error) {
|
||||
for (const plan of plans) written.add(plan.row.id)
|
||||
return written
|
||||
}
|
||||
|
||||
log.warn('bulk invoice_items insert failed; retrying per invoice', { reason: bulk.error.message })
|
||||
for (const plan of plans) {
|
||||
const { error } = await supabase.from('invoice_items').insert(plan.items)
|
||||
if (error) {
|
||||
log.error('invoice_items insert failed', {
|
||||
invoiceId: plan.row.id, invoiceNumber: plan.row.invoice_number, reason: error.message,
|
||||
})
|
||||
continue
|
||||
}
|
||||
written.add(plan.row.id)
|
||||
}
|
||||
return written
|
||||
}
|
||||
Reference in New Issue
Block a user