fix(migration): complete the rows of migrated sales invoices the hydration budget did not reach (#2291)
* fix(migration): complete the rows of migrated sales invoices the hydration budget did not reach The migration maps sales invoices from the provider's list payload and hydrates the detail form (rows, net, VAT) inside a fixed 90 s budget, open invoices first. Fortnox, Briox and Björn Lundén ship no rows in a list response, so every invoice the budget did not reach was imported as a header with a total and no invoice_items, and nothing ever came back for it: the wizard never showed the hydration report, so the user found out on the invoice page. Measured on prod today: Profilio 384 of 384 (migrated before hydration existed), Loftux 311 of 672, Damac 182 of 542, Clearstoq 1 125 of 1 125. - lib/providers: hydrateSalesInvoices() hydrates a caller-chosen subset of an already-listed register, so a follow-up can spend its budget on the invoices still incomplete on our side instead of re-walking the register open-first and never reaching the rest. - arcim-migration: completeMigratedInvoiceLines() starts from OUR row-less non-draft invoices, joins them to the provider register on number + date (unique on both sides), hydrates only that subset and writes each invoice's rows once the detail total matches the stored total to the öre. The header VAT split is rewritten only when the stored one holds no evidence (null rate, or a non-zero rate label beside 0 kr VAT and subtotal = total). Never the total, status, payments or a journal entry. - Hourly cron (/api/extensions/arcim-migration/complete-invoice-lines/cron, vercel.json + Docker crontabs) drives the pass over consents accepted in the last 60 days, newest first, with a per-company share of the run. - The wizard's result screen now shows "x av y fakturor hämtade med rader" and that the rest are fetched in the background within the hour. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DG5aYcshzKJ1EA7PPhGtVf * fix(migration): write the header VAT fill as a literal, raise the schema-guard ceiling for the row inserts The phantom-column scanner resolves only object-literal payloads. The header update is now a literal (so its six columns are checked); the two invoice_items inserts are runtime row arrays from mapSalesInvoiceLine, the same shape the orchestrator already inserts, so the ceiling moves 399 to 401 with the reason recorded beside the earlier ones. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DG5aYcshzKJ1EA7PPhGtVf * fix(migration): gate the completion cron on token freshness, not consent age, and visit every usable consent Two review findings held. Prod holds 57 accepted consents from the last 60 days, so a fixed page of the newest 25 would leave older companies with row-less invoices waiting behind companies that are already done: the cap is gone (a company with nothing left costs one query and no provider call). And the consent's created_at said nothing about whether its credentials still work: Fortnox refresh tokens live 45 days and rotate on every refresh, so eligibility is now read off the token row (access token expired within the last 45 days, or no expiry at all), which also stops a dead consent from being retried every hour. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DG5aYcshzKJ1EA7PPhGtVf --------- Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Jakob Wennberg
co-authored by
Claude Fable 5.1
Jakob Wennberg
Claude Fable 5.1
Jakob Wennberg
parent
0bd3c27fba
commit
8e1f9d5201
+219
@@ -0,0 +1,219 @@
|
||||
import { describe, it, expect, vi, beforeEach } from 'vitest'
|
||||
import { NextResponse } from 'next/server'
|
||||
|
||||
/**
|
||||
* The hourly pass that fills in the rows the migration's bounded hydration
|
||||
* did not reach. The route is thin: refuse without the cron secret, refuse
|
||||
* when the extension is off, then hand each recent accepted consent to the
|
||||
* pass with its share of the run and add the counts up.
|
||||
*/
|
||||
|
||||
vi.mock('@/lib/extensions/loader', () => ({ loadExtensions: vi.fn() }))
|
||||
vi.mock('@/lib/extensions/registry', () => ({ extensionRegistry: { get: vi.fn() } }))
|
||||
vi.mock('@/lib/auth/cron', () => ({ verifyCronSecret: vi.fn().mockReturnValue(null) }))
|
||||
|
||||
const h = vi.hoisted(() => ({
|
||||
consents: { data: [] as unknown[] | null, error: null as { message: string } | null },
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/auth/api-keys', () => ({
|
||||
createServiceClientNoCookies: vi.fn(() => ({
|
||||
from: vi.fn(() => {
|
||||
const builder: Record<string, unknown> = {}
|
||||
for (const method of ['select', 'eq', 'not', 'order']) {
|
||||
builder[method] = vi.fn(() => builder)
|
||||
}
|
||||
builder.limit = vi.fn(() => Promise.resolve(h.consents))
|
||||
return builder
|
||||
}),
|
||||
})),
|
||||
}))
|
||||
|
||||
vi.mock('@/extensions/general/arcim-migration/lib/complete-invoice-lines', () => ({
|
||||
completeMigratedInvoiceLines: vi.fn(),
|
||||
}))
|
||||
|
||||
import { GET, maxDuration, consentIsUsable } from '../route'
|
||||
import { extensionRegistry } from '@/lib/extensions/registry'
|
||||
import { verifyCronSecret } from '@/lib/auth/cron'
|
||||
import { completeMigratedInvoiceLines } from '@/extensions/general/arcim-migration/lib/complete-invoice-lines'
|
||||
|
||||
const mockRegistryGet = vi.mocked(extensionRegistry.get)
|
||||
const mockVerifyCronSecret = vi.mocked(verifyCronSecret)
|
||||
const mockComplete = vi.mocked(completeMigratedInvoiceLines)
|
||||
|
||||
const EMPTY = {
|
||||
candidates: 0, providerInvoices: 0, matched: 0, unmatched: 0, completed: 0, headersUpdated: 0,
|
||||
totalMismatch: 0, noLinesAtProvider: 0, notHydrated: 0, vatUnresolved: 0, failed: 0, remaining: 0,
|
||||
hydration: { needed: 0, hydrated: 0, failed: 0, skippedForBudget: 0 }, dryRun: false,
|
||||
}
|
||||
|
||||
const DAY_MS = 24 * 60 * 60 * 1000
|
||||
|
||||
/** A consent whose access token expired `daysAgo` days ago (null: never expires). */
|
||||
function consent(id: string, companyId: string, daysAgo: number | null, provider = 'fortnox') {
|
||||
return {
|
||||
id,
|
||||
company_id: companyId,
|
||||
provider,
|
||||
created_at: '2026-08-13T22:58:24Z',
|
||||
provider_consent_tokens: {
|
||||
token_expires_at: daysAgo === null ? null : new Date(Date.now() - daysAgo * DAY_MS).toISOString(),
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
function makeRequest() {
|
||||
return new Request('http://localhost/api/extensions/arcim-migration/complete-invoice-lines/cron', {
|
||||
headers: { authorization: 'Bearer synthetic-cron-secret' },
|
||||
})
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
mockVerifyCronSecret.mockReturnValue(null)
|
||||
mockRegistryGet.mockReturnValue({ id: 'arcim-migration' } as never)
|
||||
h.consents = { data: [], error: null }
|
||||
})
|
||||
|
||||
describe('GET /api/extensions/arcim-migration/complete-invoice-lines/cron', () => {
|
||||
it('reserves the full function window: hydration is rate-limited at the provider', () => {
|
||||
expect(maxDuration).toBe(300)
|
||||
})
|
||||
|
||||
it('returns 401 when the cron secret is rejected', async () => {
|
||||
mockVerifyCronSecret.mockReturnValue(NextResponse.json({ error: 'Unauthorized' }, { status: 401 }))
|
||||
|
||||
const response = await GET(makeRequest())
|
||||
|
||||
expect(response.status).toBe(401)
|
||||
expect(mockComplete).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('returns 503 EXTENSION_DISABLED when the extension is not in the registry', async () => {
|
||||
mockRegistryGet.mockReturnValue(undefined)
|
||||
|
||||
const response = await GET(makeRequest())
|
||||
const body = await response.json()
|
||||
|
||||
expect(response.status).toBe(503)
|
||||
expect(body.code).toBe('EXTENSION_DISABLED')
|
||||
expect(mockComplete).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('runs the pass once per usable consent and adds the counts up', async () => {
|
||||
h.consents = {
|
||||
data: [
|
||||
consent('c-new', 'co-1', 0),
|
||||
consent('c-old', 'co-2', 22),
|
||||
consent('c-done', 'co-3', 4, 'visma'),
|
||||
],
|
||||
error: null,
|
||||
}
|
||||
mockComplete
|
||||
.mockResolvedValueOnce({ ...EMPTY, candidates: 311, matched: 311, completed: 300, headersUpdated: 300, notHydrated: 11, remaining: 11 })
|
||||
.mockResolvedValueOnce({ ...EMPTY, candidates: 384, matched: 384, completed: 384, headersUpdated: 358 })
|
||||
.mockResolvedValueOnce({ ...EMPTY })
|
||||
|
||||
const response = await GET(makeRequest())
|
||||
const body = await response.json()
|
||||
|
||||
expect(response.status).toBe(200)
|
||||
expect(mockComplete).toHaveBeenCalledTimes(3)
|
||||
expect(mockComplete.mock.calls[0][0]).toMatchObject({ companyId: 'co-1', consentId: 'c-new' })
|
||||
expect(mockComplete.mock.calls[0][0].budgetMs).toBeGreaterThan(0)
|
||||
expect(mockComplete.mock.calls[0][0].budgetMs).toBeLessThanOrEqual(120_000)
|
||||
// The company with nothing to complete is not counted as worked on.
|
||||
expect(body.data).toMatchObject({
|
||||
consents: 3,
|
||||
consentsStale: 0,
|
||||
consentsFailed: 0,
|
||||
companies: 2,
|
||||
candidates: 695,
|
||||
completed: 684,
|
||||
headersUpdated: 658,
|
||||
remaining: 11,
|
||||
notHydrated: 11,
|
||||
skippedForBudget: 0,
|
||||
})
|
||||
})
|
||||
|
||||
it('isolates a failing consent: the others still run and the failure is counted', async () => {
|
||||
h.consents = {
|
||||
data: [consent('c-revoked', 'co-1', 1), consent('c-live', 'co-2', 2)],
|
||||
error: null,
|
||||
}
|
||||
mockComplete
|
||||
.mockRejectedValueOnce(new Error('Token refresh failed for fortnox; the connection must be re-authorized'))
|
||||
.mockResolvedValueOnce({ ...EMPTY, candidates: 5, matched: 5, completed: 5, headersUpdated: 5 })
|
||||
|
||||
const response = await GET(makeRequest())
|
||||
const body = await response.json()
|
||||
|
||||
expect(response.status).toBe(200)
|
||||
expect(mockComplete).toHaveBeenCalledTimes(2)
|
||||
expect(body.data).toMatchObject({ consents: 2, consentsFailed: 1, companies: 1, completed: 5 })
|
||||
})
|
||||
|
||||
it('skips consents whose credentials can no longer be refreshed, by token state not consent age', async () => {
|
||||
// Fortnox refresh tokens live 45 days and rotate on every refresh: a pair
|
||||
// whose access token expired 46 days ago is dead however young the consent
|
||||
// row is. A pair refreshed yesterday on a consent from months ago is live.
|
||||
// A token without an expiry (Bokio) never goes stale.
|
||||
h.consents = {
|
||||
data: [
|
||||
{ ...consent('c-dead', 'co-1', 46), created_at: '2026-09-01T00:00:00Z' },
|
||||
{ ...consent('c-live', 'co-2', 1), created_at: '2026-04-01T00:00:00Z' },
|
||||
consent('c-bokio', 'co-3', null, 'bokio'),
|
||||
{ ...consent('c-no-token', 'co-4', 1), provider_consent_tokens: null },
|
||||
],
|
||||
error: null,
|
||||
}
|
||||
mockComplete.mockResolvedValue({ ...EMPTY })
|
||||
|
||||
const response = await GET(makeRequest())
|
||||
const body = await response.json()
|
||||
|
||||
expect(response.status).toBe(200)
|
||||
expect(mockComplete.mock.calls.map((c) => c[0].consentId)).toEqual(['c-live', 'c-bokio'])
|
||||
expect(body.data).toMatchObject({ consents: 2, consentsStale: 2 })
|
||||
})
|
||||
|
||||
it('does not page: every usable consent is visited, not only the newest few', async () => {
|
||||
// 57 consents were accepted in the last 60 days on prod (2026-09-05); a
|
||||
// fixed page of the newest ones would leave older companies with row-less
|
||||
// invoices waiting forever behind companies that are already done.
|
||||
h.consents = {
|
||||
data: Array.from({ length: 80 }, (_, i) => consent(`c-${i}`, `co-${i}`, 1)),
|
||||
error: null,
|
||||
}
|
||||
mockComplete.mockResolvedValue({ ...EMPTY })
|
||||
|
||||
const response = await GET(makeRequest())
|
||||
const body = await response.json()
|
||||
|
||||
expect(response.status).toBe(200)
|
||||
expect(mockComplete).toHaveBeenCalledTimes(80)
|
||||
expect(body.data).toMatchObject({ consents: 80, skippedForBudget: 0 })
|
||||
})
|
||||
|
||||
it('consentIsUsable reads the token row, tolerating either embed cardinality', () => {
|
||||
const now = Date.now()
|
||||
const array = { ...consent('c', 'co', 1), provider_consent_tokens: [{ token_expires_at: new Date(now - DAY_MS).toISOString() }] }
|
||||
const stale = { ...consent('c', 'co', 1), provider_consent_tokens: [{ token_expires_at: new Date(now - 50 * DAY_MS).toISOString() }] }
|
||||
const garbage = { ...consent('c', 'co', 1), provider_consent_tokens: { token_expires_at: 'not a date' } }
|
||||
expect(consentIsUsable(array, now)).toBe(true)
|
||||
expect(consentIsUsable(stale, now)).toBe(false)
|
||||
expect(consentIsUsable(garbage, now)).toBe(false)
|
||||
expect(consentIsUsable({ ...consent('c', 'co', 1), provider_consent_tokens: [] }, now)).toBe(false)
|
||||
})
|
||||
|
||||
it('surfaces a failed consent lookup instead of reporting an empty run', async () => {
|
||||
h.consents = { data: null, error: { message: 'relation does not exist' } }
|
||||
|
||||
const response = await GET(makeRequest())
|
||||
|
||||
expect(response.status).toBeGreaterThanOrEqual(500)
|
||||
expect(mockComplete).not.toHaveBeenCalled()
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,172 @@
|
||||
import { NextResponse } from 'next/server'
|
||||
import { loadExtensions } from '@/lib/extensions/loader'
|
||||
import { extensionRegistry } from '@/lib/extensions/registry'
|
||||
import { withCronContext } from '@/lib/api/with-cron-context'
|
||||
import { createServiceClientNoCookies } from '@/lib/auth/api-keys'
|
||||
import {
|
||||
completeMigratedInvoiceLines,
|
||||
type CompleteInvoiceLinesResult,
|
||||
} from '@/extensions/general/arcim-migration/lib/complete-invoice-lines'
|
||||
|
||||
/**
|
||||
* GET /api/extensions/arcim-migration/complete-invoice-lines/cron: fetch the
|
||||
* rows (and the VAT split) for migrated sales invoices that were imported
|
||||
* without them.
|
||||
*
|
||||
* The migration hydrates the provider's detail form inside a fixed budget
|
||||
* and reports the shortfall; this is what picks the shortfall up. Every run
|
||||
* walks the accepted consents whose credentials can still be used, newest
|
||||
* first, and for each company completes as many of its row-less invoices as
|
||||
* its share of the run allows. A company with nothing left costs one query
|
||||
* and no provider call (the pass checks our side before it touches the
|
||||
* consent), so walking every live consent is cheap and no company waits
|
||||
* behind a fixed page of newer ones. Scheduled hourly in vercel.json (and
|
||||
* the Docker crontabs); a company the size of Clearstoq (1 125 invoices) is
|
||||
* done after two or three runs.
|
||||
*
|
||||
* "Can still be used" is read off the token row, not the consent's age:
|
||||
* Fortnox issues a new refresh token on every refresh and each one lives 45
|
||||
* days, so a pair whose access token expired more than 45 days ago has not
|
||||
* been refreshed since and its refresh token is dead. Trying such a consent
|
||||
* every hour would only log the same PROVIDER_AUTH_EXPIRED; when the company
|
||||
* reconnects through the wizard, the token row is renewed and the next run
|
||||
* finds it here. A token without an expiry (Bokio's private tokens) is
|
||||
* always eligible.
|
||||
*/
|
||||
|
||||
export const maxDuration = 300
|
||||
|
||||
/** Leave the function a margin for the DB writes after the last fetch. */
|
||||
const RUN_BUDGET_MS = 240_000
|
||||
/** One company's share of provider detail fetches per run. */
|
||||
const PER_COMPANY_BUDGET_MS = 120_000
|
||||
/** Below this the remaining companies wait for the next run. */
|
||||
const MIN_COMPANY_BUDGET_MS = 20_000
|
||||
/**
|
||||
* A token pair not refreshed for this long cannot be refreshed any more
|
||||
* (Fortnox: refresh tokens live 45 days and rotate on every refresh).
|
||||
*/
|
||||
const TOKEN_STALE_DAYS = 45
|
||||
/** Hard safety on the consent scan; prod holds ~120 accepted consents in total. */
|
||||
const MAX_CONSENTS_SCANNED = 500
|
||||
|
||||
interface ConsentRow {
|
||||
id: string
|
||||
company_id: string
|
||||
provider: string | null
|
||||
created_at: string
|
||||
provider_consent_tokens: { token_expires_at: string | null } | { token_expires_at: string | null }[] | null
|
||||
}
|
||||
|
||||
/** The consent's token row, whichever cardinality PostgREST rendered it with. */
|
||||
function tokenOf(consent: ConsentRow): { token_expires_at: string | null } | null {
|
||||
const tokens = consent.provider_consent_tokens
|
||||
if (!tokens) return null
|
||||
return Array.isArray(tokens) ? (tokens[0] ?? null) : tokens
|
||||
}
|
||||
|
||||
/** Does this consent still hold credentials a run can use? */
|
||||
export function consentIsUsable(consent: ConsentRow, now: number): boolean {
|
||||
const token = tokenOf(consent)
|
||||
if (!token) return false
|
||||
if (token.token_expires_at === null) return true
|
||||
const expiredAt = Date.parse(token.token_expires_at)
|
||||
if (Number.isNaN(expiredAt)) return false
|
||||
return now - expiredAt <= TOKEN_STALE_DAYS * 24 * 60 * 60 * 1000
|
||||
}
|
||||
|
||||
export const GET = withCronContext('cron.arcim_migration_complete_invoice_lines', async (_request, ctx) => {
|
||||
loadExtensions()
|
||||
|
||||
// Physical routes under app/api/extensions/<id>/ compile into every build;
|
||||
// the registry (generated from extensions.config.json) is what switches an
|
||||
// extension on. A scheduled-but-disabled cron must fail visibly.
|
||||
if (!extensionRegistry.get('arcim-migration')) {
|
||||
ctx.log.warn('arcim-migration extension is not enabled; cron refused')
|
||||
return NextResponse.json(
|
||||
{ error: 'Migration extension is not enabled', code: 'EXTENSION_DISABLED' },
|
||||
{ status: 503 },
|
||||
)
|
||||
}
|
||||
|
||||
const supabase = createServiceClientNoCookies()
|
||||
|
||||
const { data, error } = await supabase
|
||||
.from('provider_consents')
|
||||
.select('id, company_id, provider, created_at, provider_consent_tokens(token_expires_at)')
|
||||
.eq('status', 1)
|
||||
.not('provider', 'is', null)
|
||||
.order('created_at', { ascending: false })
|
||||
.limit(MAX_CONSENTS_SCANNED)
|
||||
|
||||
if (error) {
|
||||
throw new Error(`provider_consents lookup failed: ${error.message}`)
|
||||
}
|
||||
|
||||
const now = Date.now()
|
||||
const scanned = (data ?? []) as ConsentRow[]
|
||||
const consents = scanned.filter((consent) => consentIsUsable(consent, now))
|
||||
const deadline = now + RUN_BUDGET_MS
|
||||
let skippedForBudget = 0
|
||||
const totals = {
|
||||
companies: 0,
|
||||
candidates: 0,
|
||||
completed: 0,
|
||||
headersUpdated: 0,
|
||||
remaining: 0,
|
||||
notHydrated: 0,
|
||||
totalMismatch: 0,
|
||||
failed: 0,
|
||||
}
|
||||
|
||||
const summary = await ctx.forEach('consent', consents, async (consent, itemCtx) => {
|
||||
const budgetMs = Math.min(PER_COMPANY_BUDGET_MS, deadline - Date.now())
|
||||
if (budgetMs < MIN_COMPANY_BUDGET_MS) {
|
||||
skippedForBudget++
|
||||
return
|
||||
}
|
||||
|
||||
const result: CompleteInvoiceLinesResult = await completeMigratedInvoiceLines({
|
||||
supabase,
|
||||
companyId: consent.company_id,
|
||||
consentId: consent.id,
|
||||
budgetMs,
|
||||
})
|
||||
|
||||
if (result.candidates > 0) {
|
||||
totals.companies++
|
||||
totals.candidates += result.candidates
|
||||
totals.completed += result.completed
|
||||
totals.headersUpdated += result.headersUpdated
|
||||
totals.remaining += result.remaining
|
||||
totals.notHydrated += result.notHydrated
|
||||
totals.totalMismatch += result.totalMismatch
|
||||
totals.failed += result.failed
|
||||
itemCtx.log.info('migrated invoice rows completed for company', {
|
||||
companyId: consent.company_id,
|
||||
provider: consent.provider,
|
||||
candidates: result.candidates,
|
||||
matched: result.matched,
|
||||
completed: result.completed,
|
||||
remaining: result.remaining,
|
||||
notHydrated: result.notHydrated,
|
||||
totalMismatch: result.totalMismatch,
|
||||
hydration: result.hydration,
|
||||
})
|
||||
}
|
||||
})
|
||||
|
||||
ctx.log.info('complete-invoice-lines run finished', {
|
||||
...totals, skippedForBudget, consents: summary.total, consentsStale: scanned.length - consents.length,
|
||||
})
|
||||
|
||||
return NextResponse.json({
|
||||
data: {
|
||||
consents: summary.total,
|
||||
consentsStale: scanned.length - consents.length,
|
||||
consentsFailed: summary.failed,
|
||||
skippedForBudget,
|
||||
...totals,
|
||||
},
|
||||
})
|
||||
})
|
||||
Reference in New Issue
Block a user