fix(import): guard against CP437-as-CP1252 mojibake entering via pre-decoded SIE text (#1569)
* refactor(arcim-migration): remove the dead gateway SIE export path fetchSIEExport and SIEExportFile have had zero callers since the direct provider clients replaced the Arcim Sync gateway (#181, #718). The path returned SIE as a pre-decoded string, and the gateway's decode of CP437 bytes as windows-1252 is what wrote the 2026-03-17 mojibake into posted entries. Deleting it makes the string-typed SIE fetch impossible to re-wire; a comment marks the grave. The consent lifecycle and entity accessors stay untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(import): warn when SIE text carries CP437-as-CP1252 mojibake The 2026-03-17 migration wrote mojibake ("L"neutbetalning"-style C1 specials) into posted entries because the retired gateway handed the /import-sie handler an already-decoded string: byte-level encoding detection never saw it, and nothing downstream checked. The live bug is gone; this is the tripwire so the signature can never land silently again. - lib/import/sie-artifact-scan.ts: pure scanner over parsed SIE account names and voucher/line descriptions, reusing hasCp1252Artifact from charset-repair; flags at >= 2 hits so a lone legitimate curly quote or apostrophe cannot false-positive a whole file. - arcim-migration /import-sie: warn-never-block; the Swedish warning rides on result.warnings, which the workspace UI already renders, plus a server-side log.warn. - wizard parse route: same scan, surfaced through the existing parse-issue warnings card in the preview, pointing at the first affected line. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(bookkeeping): pin the reported gateway mojibake strings Adds the four strings reported from the affected company's journal as reverse_cp437 cases (all reverse losslessly) plus a false-positive guard: space-padded typography must never route into the CP437 reversal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
Jakob Wennberg
parent
c4adc8eb7d
commit
78a581bca1
@@ -69,6 +69,27 @@ describe('reverseCp437Mojibake (CP437 read as CP1252)', () => {
|
||||
expect(reverseCp437Mojibake('Kassa')).toBeNull()
|
||||
expect(reverseCp437Mojibake('Företagskonto')).toBeNull() // ö (0xF6→÷) not a CP437 letter byte
|
||||
})
|
||||
|
||||
it('recovers the 2026-03-17 gateway-migration strings (reported prod fixtures)', () => {
|
||||
// The retired Arcim Sync gateway decoded Bokio's CP437 SIE bytes as
|
||||
// windows-1252 before JSON-encoding them: ö 0x94 → U+201D ”, ä 0x84 →
|
||||
// U+201E „, Ä 0x8E → U+017D Ž. These four strings are the ones reported
|
||||
// from the affected company's journal; all reverse losslessly.
|
||||
expect(reverseCp437Mojibake('Ink”p tj„nster inom EU')).toBe('Inköp tjänster inom EU')
|
||||
expect(reverseCp437Mojibake('L”neutbetalning')).toBe('Löneutbetalning')
|
||||
expect(reverseCp437Mojibake('BANKTJŽNSTER')).toBe('BANKTJÄNSTER')
|
||||
expect(reverseCp437Mojibake('UTLŽGG')).toBe('UTLÄGG')
|
||||
})
|
||||
|
||||
it('never routes legitimate space-padded typography into the CP437 reversal (false-positive guard)', () => {
|
||||
// A curly quote or ellipsis that is NOT letter-adjacent is punctuation,
|
||||
// not a mangled diacritic: the artifact detector must stay quiet so
|
||||
// resolveCorrectName leaves the text untouched.
|
||||
expect(hasCp1252Artifact('Betalning enligt avtal ” 2024')).toBe(false)
|
||||
expect(resolveCorrectName('Betalning enligt avtal ” 2024', [])).toBeNull()
|
||||
expect(hasCp1252Artifact('Avvaktar underlag …')).toBe(false)
|
||||
expect(resolveCorrectName('Avvaktar underlag …', [])).toBeNull()
|
||||
})
|
||||
})
|
||||
|
||||
describe('resolveCorrectName: CP437 branch', () => {
|
||||
|
||||
Reference in New Issue
Block a user