fix(import): SEB CSV imports survive BOMs and bad format choices (#1565)
* fix(import): handle BOMs at the byte level in decodeFileContent Inspect leading bytes before decoding: EF BB BF strips the UTF-8 BOM and decodes the remainder (falling back to Windows-1252 for the remainder only, so the fallback can no longer produce a literal mojibake prefix), and FF FE / FE FF decode as UTF-16LE/BE. stripBOM additionally strips a literal mojibake BOM prefix for string paths pre-decoded elsewhere. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(import): make an explicit SEB choice at least as good as auto-detect Three changes for the SEB bank CSV report: - parseBankFile: when an explicit format parses 0 transactions, fall back to auto-detection; a different format that parses rows is returned with a prepended info issue naming both formats. A working explicit parse is never overridden, and explicit generic_csv (the manual mapping escape hatch) is exempt. - SEB profile: sniff the header delimiter (';' vs ',') and split with the quote-aware parseCSVLine; accept a bare Datum date column as a lowest priority tier in parse only, never in detect. Its user-reachable issue strings are now Swedish. - Import page: when a parse yields 0 transactions, show the parser's real issues instead of only the generic no-transactions hint. The v1 agent route now decodes through the shared decodeFileContent and stamps external ids, import_source, and the stored file format from the format the parse result actually carries, so fallback imports dedup identically to auto-detected ones. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(api-v1): the bank import route also decodes UTF-16 CodeRabbit on #1565: decodeFileContent gained UTF-16LE/BE BOM support but the route overview and the registered endpoint description still listed only UTF-8 / Windows-1252. Skill regenerated (apiskill:generate). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
Jakob Wennberg
parent
0d3ba5268d
commit
9c891ee72d
@@ -7,12 +7,38 @@
|
||||
*/
|
||||
|
||||
/**
|
||||
* Decode file content, handling both UTF-8 and Windows-1252 encodings.
|
||||
* Decode file content, handling BOMs plus UTF-8 and Windows-1252 encodings.
|
||||
*
|
||||
* Strategy: Try UTF-8 first. If the result contains replacement characters
|
||||
* (U+FFFD) or garbled Swedish chars, fall back to Windows-1252.
|
||||
* Strategy: inspect the leading BYTES first.
|
||||
* - FF FE / FE FF: UTF-16LE/BE (e.g. Excel "Unicode text" re-saves). Decode
|
||||
* as such; TextDecoder consumes the BOM itself.
|
||||
* - EF BB BF (UTF-8 BOM): strip the three bytes and decode the REMAINDER.
|
||||
* If the remainder still contains U+FFFD (invalid UTF-8 somewhere in the
|
||||
* file), decode the remainder as Windows-1252. The BOM bytes are never
|
||||
* re-included, so the fallback can no longer produce a literal mojibake
|
||||
* "" prefix that breaks exact-match header detection.
|
||||
* - No BOM: try UTF-8; fall back to Windows-1252 on replacement characters
|
||||
* or garbled Swedish chars (unchanged behavior).
|
||||
*/
|
||||
export function decodeFileContent(buffer: ArrayBuffer): string {
|
||||
const bytes = new Uint8Array(buffer)
|
||||
|
||||
if (bytes.length >= 2 && bytes[0] === 0xff && bytes[1] === 0xfe) {
|
||||
return new TextDecoder('utf-16le', { fatal: false }).decode(buffer)
|
||||
}
|
||||
if (bytes.length >= 2 && bytes[0] === 0xfe && bytes[1] === 0xff) {
|
||||
return new TextDecoder('utf-16be', { fatal: false }).decode(buffer)
|
||||
}
|
||||
|
||||
if (bytes.length >= 3 && bytes[0] === 0xef && bytes[1] === 0xbb && bytes[2] === 0xbf) {
|
||||
const rest = bytes.subarray(3)
|
||||
const utf8Rest = new TextDecoder('utf-8', { fatal: false }).decode(rest)
|
||||
if (!utf8Rest.includes('�')) {
|
||||
return utf8Rest
|
||||
}
|
||||
return new TextDecoder('windows-1252', { fatal: false }).decode(rest)
|
||||
}
|
||||
|
||||
const utf8Decoder = new TextDecoder('utf-8', { fatal: false })
|
||||
const utf8Result = utf8Decoder.decode(buffer)
|
||||
|
||||
@@ -72,12 +98,19 @@ export function normalizeLineEndings(content: string): string {
|
||||
}
|
||||
|
||||
/**
|
||||
* Strip BOM (Byte Order Mark) from start of content
|
||||
* Strip BOM (Byte Order Mark) from start of content.
|
||||
*
|
||||
* Also strips the literal mojibake form "" (U+00EF U+00BB U+00BF): a
|
||||
* UTF-8 BOM whose bytes were decoded as Windows-1252/Latin-1 upstream.
|
||||
* Covers string-entry paths that were pre-decoded outside decodeFileContent.
|
||||
*/
|
||||
export function stripBOM(content: string): string {
|
||||
if (content.charCodeAt(0) === 0xfeff) {
|
||||
return content.slice(1)
|
||||
}
|
||||
if (content.startsWith('')) {
|
||||
return content.slice(3)
|
||||
}
|
||||
return content
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user