fix(import): SEB CSV imports survive BOMs and bad format choices (#1565)

* fix(import): handle BOMs at the byte level in decodeFileContent

Inspect leading bytes before decoding: EF BB BF strips the UTF-8 BOM and
decodes the remainder (falling back to Windows-1252 for the remainder only,
so the fallback can no longer produce a literal mojibake prefix), and
FF FE / FE FF decode as UTF-16LE/BE. stripBOM additionally strips a literal
mojibake BOM prefix for string paths pre-decoded elsewhere.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(import): make an explicit SEB choice at least as good as auto-detect

Three changes for the SEB bank CSV report:

- parseBankFile: when an explicit format parses 0 transactions, fall back
  to auto-detection; a different format that parses rows is returned with a
  prepended info issue naming both formats. A working explicit parse is
  never overridden, and explicit generic_csv (the manual mapping escape
  hatch) is exempt.
- SEB profile: sniff the header delimiter (';' vs ',') and split with the
  quote-aware parseCSVLine; accept a bare Datum date column as a lowest
  priority tier in parse only, never in detect. Its user-reachable issue
  strings are now Swedish.
- Import page: when a parse yields 0 transactions, show the parser's real
  issues instead of only the generic no-transactions hint.

The v1 agent route now decodes through the shared decodeFileContent and
stamps external ids, import_source, and the stored file format from the
format the parse result actually carries, so fallback imports dedup
identically to auto-detected ones.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(api-v1): the bank import route also decodes UTF-16

CodeRabbit on #1565: decodeFileContent gained UTF-16LE/BE BOM support
but the route overview and the registered endpoint description still
listed only UTF-8 / Windows-1252. Skill regenerated (apiskill:generate).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Jakob Wennberg
2026-08-13 15:22:19 +02:00
committed by GitHub
co-authored by Claude Fable 5 Jakob Wennberg
parent 0d3ba5268d
commit 9c891ee72d
10 changed files with 322 additions and 34 deletions
+37 -4
View File
@@ -7,12 +7,38 @@
*/
/**
* Decode file content, handling both UTF-8 and Windows-1252 encodings.
* Decode file content, handling BOMs plus UTF-8 and Windows-1252 encodings.
*
* Strategy: Try UTF-8 first. If the result contains replacement characters
* (U+FFFD) or garbled Swedish chars, fall back to Windows-1252.
* Strategy: inspect the leading BYTES first.
* - FF FE / FE FF: UTF-16LE/BE (e.g. Excel "Unicode text" re-saves). Decode
* as such; TextDecoder consumes the BOM itself.
* - EF BB BF (UTF-8 BOM): strip the three bytes and decode the REMAINDER.
* If the remainder still contains U+FFFD (invalid UTF-8 somewhere in the
* file), decode the remainder as Windows-1252. The BOM bytes are never
* re-included, so the fallback can no longer produce a literal mojibake
* "" prefix that breaks exact-match header detection.
* - No BOM: try UTF-8; fall back to Windows-1252 on replacement characters
* or garbled Swedish chars (unchanged behavior).
*/
export function decodeFileContent(buffer: ArrayBuffer): string {
const bytes = new Uint8Array(buffer)
if (bytes.length >= 2 && bytes[0] === 0xff && bytes[1] === 0xfe) {
return new TextDecoder('utf-16le', { fatal: false }).decode(buffer)
}
if (bytes.length >= 2 && bytes[0] === 0xfe && bytes[1] === 0xff) {
return new TextDecoder('utf-16be', { fatal: false }).decode(buffer)
}
if (bytes.length >= 3 && bytes[0] === 0xef && bytes[1] === 0xbb && bytes[2] === 0xbf) {
const rest = bytes.subarray(3)
const utf8Rest = new TextDecoder('utf-8', { fatal: false }).decode(rest)
if (!utf8Rest.includes('�')) {
return utf8Rest
}
return new TextDecoder('windows-1252', { fatal: false }).decode(rest)
}
const utf8Decoder = new TextDecoder('utf-8', { fatal: false })
const utf8Result = utf8Decoder.decode(buffer)
@@ -72,12 +98,19 @@ export function normalizeLineEndings(content: string): string {
}
/**
* Strip BOM (Byte Order Mark) from start of content
* Strip BOM (Byte Order Mark) from start of content.
*
* Also strips the literal mojibake form "" (U+00EF U+00BB U+00BF): a
* UTF-8 BOM whose bytes were decoded as Windows-1252/Latin-1 upstream.
* Covers string-entry paths that were pre-decoded outside decodeFileContent.
*/
export function stripBOM(content: string): string {
if (content.charCodeAt(0) === 0xfeff) {
return content.slice(1)
}
if (content.startsWith('')) {
return content.slice(3)
}
return content
}