accounted/.claude/skills/swedish-sie-import-export/references/encoding.md

SIE4 Character Encoding Reference

The spec vs. reality

The SIE4 spec mandates IBM PC Codepage 437 (declared as #FORMAT PC8). Modern cloud accounting software exports UTF-8. Many programs write #FORMAT PC8 regardless of actual encoding. This mismatch is the single largest source of SIE interoperability failures.

Byte values for Swedish characters

Char	CP437 (hex)	Latin-1 (hex)	UTF-8 (hex)
å	0x86	0xE5	0xC3 0xA5
ä	0x84	0xE4	0xC3 0xA4
ö	0x94	0xF6	0xC3 0xB6
Å	0x8F	0xC5	0xC3 0x85
Ä	0x8E	0xC4	0xC3 0x84
Ö	0x99	0xD6	0xC3 0x96
é	0x82	0xE9	0xC3 0xA9

Mojibake diagnosis patterns

UTF-8 bytes misread as Latin-1/CP1252

• å → Ã¥, ä → ä, ö → ö
• Å → Ã…, Ä → Ä, Ö → Ö
• Telltale: The Ã prefix always indicates UTF-8 content decoded as single-byte
• Fix: Re-read the file as UTF-8

CP437 bytes misread as Latin-1/CP1252

• å → †, ä → „, ö → "
• Å → control char (0x8F), Ä → control char (0x8E), Ö → ™
• Telltale: Typographic characters like †„"™ in account names
• Fix: Re-read the file as CP437

Latin-1 bytes misread as CP437

• å → σ, ä → Σ, ö → ÷
• Telltale: Greek letters or math symbols in Swedish text
• Fix: Re-read the file as Latin-1 (ISO 8859-1)

Double-encoding (UTF-8 encoded twice)

• å → ã¥ or similar multi-byte garbage
• Telltale: 3+ byte sequences where 2 were expected
• Fix: Decode as UTF-8, then decode the result as UTF-8 again (strip one layer)

Encoding detection algorithm

Apply in this order, stop at first match:

1. UTF-8 BOM check: Bytes 0xEF 0xBB 0xBF at file start → UTF-8
2. Strict UTF-8 validation: Attempt full decode. If valid AND multi-byte sequences found → UTF-8. Pure ASCII files are ambiguous (proceed to next step).
3. CP437 byte scan: Look for bytes in range 0x80-0x9F in text contexts (account names, descriptions). If bytes 0x84, 0x86, 0x8E, 0x8F, 0x94, 0x99 appear → CP437
4. Latin-1 byte scan: Look for bytes in range 0xC0-0xFF in text contexts. If bytes 0xC4, 0xC5, 0xD6, 0xE4, 0xE5, 0xF6 appear → Latin-1
5. #PROGRAM heuristic: Extract program name from the file (readable in any encoding since it's typically ASCII):
   • Cloud software → assume UTF-8
   • Desktop software → assume CP437
6. Fallback: Latin-1 (every byte 0x00-0xFF is valid, never fails)

Per-software encoding behaviors

Software	Platform	Typical encoding	Notes
Fortnox	Cloud	UTF-8	May declare #FORMAT PC8. Largest market share.
Visma Administration	Desktop	CP437 or Win-1252	Legacy Windows app. Requires non-UTF-8 locale.
Visma eEkonomi	Cloud	UTF-8 or Latin-1	Known garbage character issues in exports.
Visma Compact	Desktop	CP437	Standards-compliant. Referenced in SIE spec examples.
BL Administration (Björn Lundén)	Desktop	CP437	Standards-compliant.
Lundify (BL cloud)	Cloud	UTF-8	Modern cloud version of BL.
SpeedLedger	Cloud	UTF-8 (likely)	Web-based Unicode environment.
PE Accounting	Cloud	UTF-8 (likely)	Web-based.
Bokio	Cloud	UTF-8	Parser-level normalization documented.
Dooer	Cloud	UTF-8	Parser-level normalization documented.
Hogia	Desktop/Cloud	CP437 or UTF-8	Depends on product version.

#KSUMMA and encoding interaction

The CRC-32 checksum (#KSUMMA) is defined to operate on CP437 byte values. When a file is actually encoded in UTF-8:

1. Swedish characters have different byte representations (2 bytes in UTF-8 vs 1 byte in CP437)
2. The checksum computed on UTF-8 bytes will NOT match the expected CP437-based value
3. Recommendation: When non-CP437 encoding is detected, skip #KSUMMA validation entirely
4. When generating SIE files, either: (a) write CP437 with correct KSUMMA, or (b) write UTF-8 and omit KSUMMA

Normalization strategy for import

Best practice for robust SIE import:

1. Read raw bytes from file
2. Detect encoding using the algorithm above
3. Decode to Unicode/UTF-8 internal representation
4. Validate #KSUMMA only if encoding is CP437
5. Parse all records from the Unicode text
6. Store internally as UTF-8
7. When exporting: write CP437 if standards compliance is critical (auditors, Skatteverket), otherwise UTF-8 with a note that KSUMMA is omitted