admin/accounted
3
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(import): migration preview and theater read all fiscal years, not just the newest (#1614)

Browse Source 
The /sie-data route parsed only the newest fiscal year's SIE file for the
import preview and the returned SIEData.parsed. Mid-year provider exports
have few or zero vouchers in the newest year, so the first real Fortnox
migration (3 fiscal years, 4153 vouchers) previewed "0 verifikationer"
and drew an almost-empty migration theater while the import itself
landed all 4153 vouchers from the older files.

- New mergeParsedSIEFiles (lib/import/sie-merge.ts): pure, browser-clean
  whole-dataset merge (accounts union first-wins, vouchers concatenated,
  fiscal years union oldest-first re-indexed newest=0, balances and
  issues concatenated, dimensions deduped), with unit tests.
- /sie-data parses each file exactly once, builds the preview from the
  merged parse and returns parsed: merged; response shape unchanged.
  Validation stays newest-file-only so no previously accepted dataset
  is newly rejected.
- /preview drops latestOnly and computes sieStats from the merged parse:
  the connect step's "Hittade X konton och Y verifikationer" line
  renders from THESE stats, so this is where the founder-visible count
  was lying.
- The migration theater spreads its account waves across ~10s and
  births an additional wave on each real step label during the SIE
  phase (progress <= 55), through a shared rate-limited gate, so the
  canvas keeps performing over a multi-minute run. Narration labels and
  progress remain the wizard's real values; reduced motion unchanged.

Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Jakob Wennberg
2026-08-14 12:17:11 +02:00
committed by GitHub
co-authored by Jakob Wennberg Claude Fable 5
parent ea85f4c084
commit b556475b01
5 changed files with 418 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files
extensions/general/arcim-migration/index.ts
+44 -25
View File
@@ -28,6 +28,7 @@ import { reconcileSupplierInvoiceVouchers } from '@/lib/invoices/bulk-reconcile-
import type { ArcimProvider } from './types'
import { ARCIM_PROVIDERS } from './types'
import { parseSIEFile, validateSIEFile } from '@/lib/import/sie-parser'
import { mergeParsedSIEFiles } from '@/lib/import/sie-merge'
import { scanSieForCp1252Artifacts, formatSieArtifactWarning } from '@/lib/import/sie-artifact-scan'
import { suggestMappings, getMappingStats, isSystemAccount } from '@/lib/import/account-mapper'
import { loadMappings, generateImportPreview, executeSIEImport } from '@/lib/import/sie-import'
@@ -703,19 +704,24 @@ export const arcimMigrationExtension: Extension = {
if (providerSupportsSie(provider)) {
try {
log.info(`Fetching SIE export from ${provider} for consent ${consentId}...`)
// Fetch SIE type 4 for the most recent allowed year to get stats
// Fetch SIE type 4 for EVERY allowed year, not latestOnly: the
// stats below render as "Hittade X konton och Y verifikationer"
// on the connect step, and a mid-year export has few or zero
// vouchers in the newest year (the One.com migration previewed
// "0 verifikationer" and then imported 4153). Costs one SIE
// export per extra year, the same work /sie-data repeats right
// after: honest numbers are worth it.
const { files, availableYears } = await fetchProviderSieFiles(
provider,
resolved.accessToken,
resolved.providerCompanyId,
{ latestOnly: true },
)
if (files.length > 0) {
const parsed = parseSIEFile(files[files.length - 1].rawContent)
const merged = mergeParsedSIEFiles(files.map((f) => parseSIEFile(f.rawContent)))
sieAvailable = true
sieStats = {
accountCount: parsed.accounts.length,
transactionCount: parsed.vouchers.length,
accountCount: merged.accounts.length,
transactionCount: merged.vouchers.length,
fiscalYears: availableYears,
}
}
@@ -817,14 +823,26 @@ export const arcimMigrationExtension: Extension = {
})
}
// Parse most recent file for preview/validation
const sieFile = sieFiles[sieFiles.length - 1]
const parsed = parseSIEFile(sieFile.rawContent)
const validation = validateSIEFile(parsed)
// Parse every file ONCE: validation, account collection, the
// preview and the returned `parsed` all read from these parses.
const parsedFiles = sieFiles.map((file) => ({
file,
parsed: parseSIEFile(file.rawContent),
}))
// Validate the most recent file's parse: unchanged behavior, so no
// previously accepted dataset is newly rejected. Older years get no
// gate here (they never had one); their problems still surface
// per-file at import time. Validating the merged parse instead
// would be wrong: validateSIEFile assumes single-file invariants
// (balance yearIndexes relative to ONE current year) that the
// merged view deliberately does not preserve.
const newestFile = parsedFiles[parsedFiles.length - 1]
const validation = validateSIEFile(newestFile.parsed)
if (!validation.valid) {
log.warn(
`arcim sie-data validation failed for ${provider} fiscal year ${sieFile.fiscalYear}: ` +
`arcim sie-data validation failed for ${provider} fiscal year ${newestFile.file.fiscalYear}: ` +
`${validation.errors.length} error(s): ${validation.errors.slice(0, 3).join(' | ')}`,
)
return NextResponse.json({
@@ -834,17 +852,16 @@ export const arcimMigrationExtension: Extension = {
}, { status: 400 })
}
// Collect ALL unique accounts across ALL fiscal year files
const allAccountsMap = new Map<string, { number: string; name: string }>()
for (const file of sieFiles) {
const fileParsed = parseSIEFile(file.rawContent)
for (const acc of fileParsed.accounts) {
if (!allAccountsMap.has(acc.number)) {
allAccountsMap.set(acc.number, { number: acc.number, name: acc.name })
}
}
}
const allAccounts = [...allAccountsMap.values()]
// Whole-dataset view across ALL fiscal years. Mid-year provider
// exports have few or zero vouchers in the newest file, so the
// preview counts and the theater model must never be built from
// that file alone ("Hittade 740 konton och 0 verifikationer" while
// 4153 vouchers were about to be imported).
const merged = mergeParsedSIEFiles(parsedFiles.map((p) => p.parsed))
// All unique accounts across all files (first file's name wins,
// same as the merge's account union).
const allAccounts = merged.accounts
.filter(a => !isSystemAccount(a.number))
.map(a => ({ number: a.number, name: a.name }))
@@ -872,7 +889,7 @@ export const arcimMigrationExtension: Extension = {
log.info(`Account mapping: ${allAccounts.length} unique accounts across ${sieFiles.length} files, ${mappingStats.unmapped} unmapped`)
const preview = generateImportPreview(parsed, mappings)
const preview = generateImportPreview(merged, mappings)
// Detect prior imports by *fiscal period overlap*, not file hash.
// Providers embed the export-time #GEN date in every SIE export so
@@ -887,8 +904,7 @@ export const arcimMigrationExtension: Extension = {
fiscalYearEnd: string | null
} | null
}[] = []
for (const file of sieFiles) {
const fileParsed = parseSIEFile(file.rawContent)
for (const { file, parsed: fileParsed } of parsedFiles) {
const fyStart = fileParsed.stats.fiscalYearStart
const fyEnd = fileParsed.stats.fiscalYearEnd
@@ -927,7 +943,10 @@ export const arcimMigrationExtension: Extension = {
const replacedFileCount = fileStatuses.filter(f => f.previousImport).length
return NextResponse.json({
parsed,
// The merged whole-dataset parse: SIEData.parsed feeds the
// migration theater (buildTheaterModel), which must see every
// fiscal year, not just the newest file.
parsed: merged,
mappings,
mappingStats,
preview,