78a581bca1
* refactor(arcim-migration): remove the dead gateway SIE export path fetchSIEExport and SIEExportFile have had zero callers since the direct provider clients replaced the Arcim Sync gateway (#181, #718). The path returned SIE as a pre-decoded string, and the gateway's decode of CP437 bytes as windows-1252 is what wrote the 2026-03-17 mojibake into posted entries. Deleting it makes the string-typed SIE fetch impossible to re-wire; a comment marks the grave. The consent lifecycle and entity accessors stay untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(import): warn when SIE text carries CP437-as-CP1252 mojibake The 2026-03-17 migration wrote mojibake ("L"neutbetalning"-style C1 specials) into posted entries because the retired gateway handed the /import-sie handler an already-decoded string: byte-level encoding detection never saw it, and nothing downstream checked. The live bug is gone; this is the tripwire so the signature can never land silently again. - lib/import/sie-artifact-scan.ts: pure scanner over parsed SIE account names and voucher/line descriptions, reusing hasCp1252Artifact from charset-repair; flags at >= 2 hits so a lone legitimate curly quote or apostrophe cannot false-positive a whole file. - arcim-migration /import-sie: warn-never-block; the Swedish warning rides on result.warnings, which the workspace UI already renders, plus a server-side log.warn. - wizard parse route: same scan, surfaced through the existing parse-issue warnings card in the preview, pointing at the first affected line. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(bookkeeping): pin the reported gateway mojibake strings Adds the four strings reported from the affected company's journal as reverse_cp437 cases (all reverse losslessly) plus a false-positive guard: space-padded typography must never route into the CP437 reversal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
93 lines
3.6 KiB
TypeScript
93 lines
3.6 KiB
TypeScript
/**
|
|
* CP1252-artifact tripwire for SIE imports.
|
|
*
|
|
* Every in-repo SIE byte path decodes correctly (detectEncoding/decodeBuffer in
|
|
* sie-parser.ts), but some import surfaces receive an ALREADY-DECODED string
|
|
* (the provider-migration /import-sie handler takes rawContent as JSON). If an
|
|
* upstream system decoded CP437 bytes as windows-1252 before handing us the
|
|
* string, the damage is already baked in: CP437 diacritics become C1 specials
|
|
* (o-umlaut 0x94 -> U+201D, a-umlaut 0x84 -> U+201E, A-umlaut 0x8E -> U+017D),
|
|
* producing text like "Ink[U+201D]p tj[U+201E]nster" and "BANKTJ[U+017D]NSTER".
|
|
* That exact corruption shipped through the retired Arcim Sync gateway on
|
|
* 2026-03-17 and sat undetected in posted entries.
|
|
*
|
|
* This scanner is the tripwire: it flags parsed SIE text carrying that
|
|
* signature so the import can WARN (never block) and the user can abort or
|
|
* review. Detection reuses hasCp1252Artifact (lib/bookkeeping/charset-repair),
|
|
* whose letter-adjacency heuristic ignores legitimate typography such as a
|
|
* space-padded en dash. A minimum of two flagged fields is required before the
|
|
* file as a whole is flagged, so a single legitimate curly quote in one
|
|
* description cannot trigger the warning.
|
|
*/
|
|
|
|
import { hasCp1252Artifact } from '@/lib/bookkeeping/charset-repair'
|
|
import type { SIEAccount, SIEVoucher } from './types'
|
|
|
|
/** Minimum number of artifact-carrying text fields before the file is flagged. */
|
|
export const SIE_ARTIFACT_THRESHOLD = 2
|
|
|
|
/** Cap on distinct example strings carried in the result (for logs/messages). */
|
|
const MAX_SAMPLES = 3
|
|
|
|
export interface SieArtifactScanResult {
|
|
/** True when artifactCount >= SIE_ARTIFACT_THRESHOLD. */
|
|
flagged: boolean
|
|
/** Number of text fields (account names, voucher/line descriptions) with artifacts. */
|
|
artifactCount: number
|
|
/** Up to MAX_SAMPLES distinct flagged strings, in encounter order. */
|
|
samples: string[]
|
|
}
|
|
|
|
/**
|
|
* Scan parsed SIE content (account names, voucher descriptions, transaction
|
|
* line descriptions) for the CP437-decoded-as-CP1252 mojibake signature.
|
|
* Pure function: no I/O, no side effects.
|
|
*/
|
|
export function scanSieForCp1252Artifacts(parsed: {
|
|
accounts: SIEAccount[]
|
|
vouchers: SIEVoucher[]
|
|
}): SieArtifactScanResult {
|
|
let artifactCount = 0
|
|
const samples: string[] = []
|
|
|
|
const check = (text: string | undefined): void => {
|
|
if (!text || !hasCp1252Artifact(text)) return
|
|
artifactCount++
|
|
if (samples.length < MAX_SAMPLES && !samples.includes(text)) {
|
|
samples.push(text)
|
|
}
|
|
}
|
|
|
|
for (const account of parsed.accounts) {
|
|
check(account.name)
|
|
}
|
|
for (const voucher of parsed.vouchers) {
|
|
check(voucher.description)
|
|
for (const line of voucher.lines) {
|
|
check(line.description)
|
|
}
|
|
}
|
|
|
|
return {
|
|
flagged: artifactCount >= SIE_ARTIFACT_THRESHOLD,
|
|
artifactCount,
|
|
samples,
|
|
}
|
|
}
|
|
|
|
/**
|
|
* User-facing Swedish warning for a flagged scan. SIE import warnings are a
|
|
* Swedish-only surface (SIE is Swedish by spec); this follows the existing
|
|
* warnings-array convention in sie-parser/sie-import rather than i18n keys.
|
|
* The C1 special characters in the message are the mojibake signature itself,
|
|
* quoted literally so the user can recognize them in their data.
|
|
*/
|
|
export function formatSieArtifactWarning(scan: SieArtifactScanResult): string {
|
|
const example = scan.samples[0] ? ` (till exempel "${scan.samples[0]}")` : ''
|
|
return (
|
|
`Filens text verkar vara felaktigt teckenkodad: ${scan.artifactCount} textfält innehåller ` +
|
|
`tecken som ”, „ eller Ž i stället för å, ä eller ö${example}. ` +
|
|
`Importen blockeras inte, men granska kontonamn och verifikationstexter.`
|
|
)
|
|
}
|