Jakob WennbergandClaude Opus 4.6
7a18d89c70
* feat: add INK2 declaration improvements, invoice delivery date, and Swedish compliance skills Expand INK2 engine with full INK2S/INK2R support and improved SRU generation. Add delivery_date field to invoices and corresponding PDF/migration support. Add Claude skills for Swedish asset accounting, invoice compliance, SIE import/export, SRU filing, and tax planning. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address PR review — map BAS 4500–4899, strip CRLF in SRU, document P3 - Map BAS accounts 4500–4599 (legoarbeten), 4700–4899 (diverse varuinköpskostnader) to SRU 7512 so they are not silently dropped from INK2R declarations - Strip \r\n in sanitizeString to prevent CRLF injection in SRU fields - Document P3 period suffix limitation for brutet räkenskapsår Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: correct BAS 4500-4599, 4700-4899 mapping from 7512 to 7511 Per the official BAS-to-SRU mapping, these account ranges are cost of goods (legoarbeten, inkurans, svinn) and belong under 7511 (Råvaror och förnödenheter), not 7512 (Handelsvaror). 7512 remains 4600-4699. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
91 lines
4.4 KiB
Markdown
91 lines
4.4 KiB
Markdown
# SIE4 Character Encoding Reference
|
||
|
||
## The spec vs. reality
|
||
|
||
The SIE4 spec mandates IBM PC Codepage 437 (declared as `#FORMAT PC8`). Modern cloud accounting software exports UTF-8. Many programs write `#FORMAT PC8` regardless of actual encoding. This mismatch is the single largest source of SIE interoperability failures.
|
||
|
||
## Byte values for Swedish characters
|
||
|
||
| Char | CP437 (hex) | Latin-1 (hex) | UTF-8 (hex) |
|
||
|------|-------------|---------------|-------------|
|
||
| å | 0x86 | 0xE5 | 0xC3 0xA5 |
|
||
| ä | 0x84 | 0xE4 | 0xC3 0xA4 |
|
||
| ö | 0x94 | 0xF6 | 0xC3 0xB6 |
|
||
| Å | 0x8F | 0xC5 | 0xC3 0x85 |
|
||
| Ä | 0x8E | 0xC4 | 0xC3 0x84 |
|
||
| Ö | 0x99 | 0xD6 | 0xC3 0x96 |
|
||
| é | 0x82 | 0xE9 | 0xC3 0xA9 |
|
||
|
||
## Mojibake diagnosis patterns
|
||
|
||
### UTF-8 bytes misread as Latin-1/CP1252
|
||
- å → `Ã¥`, ä → `ä`, ö → `ö`
|
||
- Å → `Ã…`, Ä → `Ä`, Ö → `Ö`
|
||
- **Telltale**: The `Ã` prefix always indicates UTF-8 content decoded as single-byte
|
||
- **Fix**: Re-read the file as UTF-8
|
||
|
||
### CP437 bytes misread as Latin-1/CP1252
|
||
- å → `†`, ä → `„`, ö → `"`
|
||
- Å → control char (0x8F), Ä → control char (0x8E), Ö → `™`
|
||
- **Telltale**: Typographic characters like `†„"™` in account names
|
||
- **Fix**: Re-read the file as CP437
|
||
|
||
### Latin-1 bytes misread as CP437
|
||
- å → `σ`, ä → `Σ`, ö → `÷`
|
||
- **Telltale**: Greek letters or math symbols in Swedish text
|
||
- **Fix**: Re-read the file as Latin-1 (ISO 8859-1)
|
||
|
||
### Double-encoding (UTF-8 encoded twice)
|
||
- å → `ã¥` or similar multi-byte garbage
|
||
- **Telltale**: 3+ byte sequences where 2 were expected
|
||
- **Fix**: Decode as UTF-8, then decode the result as UTF-8 again (strip one layer)
|
||
|
||
## Encoding detection algorithm
|
||
|
||
Apply in this order, stop at first match:
|
||
|
||
1. **UTF-8 BOM check**: Bytes 0xEF 0xBB 0xBF at file start → UTF-8
|
||
2. **Strict UTF-8 validation**: Attempt full decode. If valid AND multi-byte sequences found → UTF-8. Pure ASCII files are ambiguous (proceed to next step).
|
||
3. **CP437 byte scan**: Look for bytes in range 0x80-0x9F in text contexts (account names, descriptions). If bytes 0x84, 0x86, 0x8E, 0x8F, 0x94, 0x99 appear → CP437
|
||
4. **Latin-1 byte scan**: Look for bytes in range 0xC0-0xFF in text contexts. If bytes 0xC4, 0xC5, 0xD6, 0xE4, 0xE5, 0xF6 appear → Latin-1
|
||
5. **#PROGRAM heuristic**: Extract program name from the file (readable in any encoding since it's typically ASCII):
|
||
- Cloud software → assume UTF-8
|
||
- Desktop software → assume CP437
|
||
6. **Fallback**: Latin-1 (every byte 0x00-0xFF is valid, never fails)
|
||
|
||
## Per-software encoding behaviors
|
||
|
||
| Software | Platform | Typical encoding | Notes |
|
||
|----------|----------|-----------------|-------|
|
||
| Fortnox | Cloud | UTF-8 | May declare `#FORMAT PC8`. Largest market share. |
|
||
| Visma Administration | Desktop | CP437 or Win-1252 | Legacy Windows app. Requires non-UTF-8 locale. |
|
||
| Visma eEkonomi | Cloud | UTF-8 or Latin-1 | Known garbage character issues in exports. |
|
||
| Visma Compact | Desktop | CP437 | Standards-compliant. Referenced in SIE spec examples. |
|
||
| BL Administration (Björn Lundén) | Desktop | CP437 | Standards-compliant. |
|
||
| Lundify (BL cloud) | Cloud | UTF-8 | Modern cloud version of BL. |
|
||
| SpeedLedger | Cloud | UTF-8 (likely) | Web-based Unicode environment. |
|
||
| PE Accounting | Cloud | UTF-8 (likely) | Web-based. |
|
||
| Bokio | Cloud | UTF-8 | Parser-level normalization documented. |
|
||
| Dooer | Cloud | UTF-8 | Parser-level normalization documented. |
|
||
| Hogia | Desktop/Cloud | CP437 or UTF-8 | Depends on product version. |
|
||
|
||
## #KSUMMA and encoding interaction
|
||
|
||
The CRC-32 checksum (#KSUMMA) is defined to operate on CP437 byte values. When a file is actually encoded in UTF-8:
|
||
|
||
1. Swedish characters have different byte representations (2 bytes in UTF-8 vs 1 byte in CP437)
|
||
2. The checksum computed on UTF-8 bytes will NOT match the expected CP437-based value
|
||
3. **Recommendation**: When non-CP437 encoding is detected, skip #KSUMMA validation entirely
|
||
4. When generating SIE files, either: (a) write CP437 with correct KSUMMA, or (b) write UTF-8 and omit KSUMMA
|
||
|
||
## Normalization strategy for import
|
||
|
||
Best practice for robust SIE import:
|
||
|
||
1. Read raw bytes from file
|
||
2. Detect encoding using the algorithm above
|
||
3. Decode to Unicode/UTF-8 internal representation
|
||
4. Validate #KSUMMA only if encoding is CP437
|
||
5. Parse all records from the Unicode text
|
||
6. Store internally as UTF-8
|
||
7. When exporting: write CP437 if standards compliance is critical (auditors, Skatteverket), otherwise UTF-8 with a note that KSUMMA is omitted |