fix(import): guard against CP437-as-CP1252 mojibake entering via pre-decoded SIE text (#1569)

* refactor(arcim-migration): remove the dead gateway SIE export path

fetchSIEExport and SIEExportFile have had zero callers since the direct
provider clients replaced the Arcim Sync gateway (#181, #718). The path
returned SIE as a pre-decoded string, and the gateway's decode of CP437
bytes as windows-1252 is what wrote the 2026-03-17 mojibake into posted
entries. Deleting it makes the string-typed SIE fetch impossible to
re-wire; a comment marks the grave. The consent lifecycle and entity
accessors stay untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(import): warn when SIE text carries CP437-as-CP1252 mojibake

The 2026-03-17 migration wrote mojibake ("L"neutbetalning"-style C1
specials) into posted entries because the retired gateway handed the
/import-sie handler an already-decoded string: byte-level encoding
detection never saw it, and nothing downstream checked. The live bug is
gone; this is the tripwire so the signature can never land silently
again.

- lib/import/sie-artifact-scan.ts: pure scanner over parsed SIE account
  names and voucher/line descriptions, reusing hasCp1252Artifact from
  charset-repair; flags at >= 2 hits so a lone legitimate curly quote or
  apostrophe cannot false-positive a whole file.
- arcim-migration /import-sie: warn-never-block; the Swedish warning
  rides on result.warnings, which the workspace UI already renders, plus
  a server-side log.warn.
- wizard parse route: same scan, surfaced through the existing
  parse-issue warnings card in the preview, pointing at the first
  affected line.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(bookkeeping): pin the reported gateway mojibake strings

Adds the four strings reported from the affected company's journal as
reverse_cp437 cases (all reverse losslessly) plus a false-positive
guard: space-padded typography must never route into the CP437
reversal.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Jakob Wennberg
2026-08-13 15:14:23 +02:00
committed by GitHub
co-authored by Claude Fable 5 Jakob Wennberg
parent c4adc8eb7d
commit 78a581bca1
8 changed files with 618 additions and 21 deletions
@@ -69,6 +69,27 @@ describe('reverseCp437Mojibake (CP437 read as CP1252)', () => {
expect(reverseCp437Mojibake('Kassa')).toBeNull()
expect(reverseCp437Mojibake('Företagskonto')).toBeNull() // ö (0xF6→÷) not a CP437 letter byte
})
it('recovers the 2026-03-17 gateway-migration strings (reported prod fixtures)', () => {
// The retired Arcim Sync gateway decoded Bokio's CP437 SIE bytes as
// windows-1252 before JSON-encoding them: ö 0x94 → U+201D ”, ä 0x84 →
// U+201E „, Ä 0x8E → U+017D Ž. These four strings are the ones reported
// from the affected company's journal; all reverse losslessly.
expect(reverseCp437Mojibake('Ink”p tj„nster inom EU')).toBe('Inköp tjänster inom EU')
expect(reverseCp437Mojibake('L”neutbetalning')).toBe('Löneutbetalning')
expect(reverseCp437Mojibake('BANKTJŽNSTER')).toBe('BANKTJÄNSTER')
expect(reverseCp437Mojibake('UTLŽGG')).toBe('UTLÄGG')
})
it('never routes legitimate space-padded typography into the CP437 reversal (false-positive guard)', () => {
// A curly quote or ellipsis that is NOT letter-adjacent is punctuation,
// not a mangled diacritic: the artifact detector must stay quiet so
// resolveCorrectName leaves the text untouched.
expect(hasCp1252Artifact('Betalning enligt avtal ” 2024')).toBe(false)
expect(resolveCorrectName('Betalning enligt avtal ” 2024', [])).toBeNull()
expect(hasCp1252Artifact('Avvaktar underlag …')).toBe(false)
expect(resolveCorrectName('Avvaktar underlag …', [])).toBeNull()
})
})
describe('resolveCorrectName: CP437 branch', () => {