fix(bookkeeping): comprehensive chart_of_accounts charset repair (#736)
* fix(bookkeeping): comprehensive chart_of_accounts charset repair The 20260625120000 backfill (PR #734) only covered the 26 short-name seed accounts. Investigation found the corruption was far broader — ~4,500 rows across 858 companies — in four signatures, and verified the root cause is already closed (prod's seed_chart_of_accounts() carries correct diacritics; the corruption was prod-migration-drift, the seed fix reached prod ~2026-06-12, no companies corrupted since). Adds a tested, reusable repair core + a guarded script: - lib/bookkeeping/charset-repair.ts — pure, unit-tested resolvers: * stripped diacritics ("Utgaende moms forsaljning...") → restore from a de-accent-equal clean sibling. DIRECTIONAL guard (only acts on a fully de-accented input) so a correct name is never stripped down; unique-match only, so user-renamed accounts are never clobbered. * double-encoded UTF-8-as-CP1252 ("Företagskonto") → lossless CP1252-aware byte reversal (recovers custom names too). * CP437-as-CP1252 ("F”rmedlad", "™vriga", "V„rdef”r„ndring") → lossless CP437 letter reversal. * lost-byte U+FFFD ("p� bilar") → fill via single-char-wildcard match to a unique clean sibling (the byte is gone, so only a confident sibling wins). isClean() rejects mojibake AND mid-word CP1252 artifacts, but treats a space-padded en-dash ("Kundfordringar – delad faktura") as legitimate. - scripts/repair-chart-of-accounts-charset.ts — dry-run by default, --execute to apply; idempotent; refuses any non-prod project. Sources canonical names from the table's own clean sibling rows + BAS_REFERENCE. Applied to production (UPDATE-only, account_name is display-only): 4,499 rows across 858 companies repaired, 0 double-encoded remaining, 0 errors. 247 rows left untouched and reported — custom account names with lost bytes and no canonical (unrecoverable from the data; need the source SIE file or manual fix). 21 unit tests cover every transform with real prod fixtures, plus the two dry-run bugs caught before any write (correct→stripped direction; matching a CP437-mojibake sibling). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(scripts): avoid supabase-js generic mismatch in charset repair fetch next build's tsc rejected fetchAll(supabase: ReturnType<typeof createClient>) — the default-generic SupabaseClient type doesn't unify with the inferred createClient() return. Make fetchAll a closure over the inferred client. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(scripts): add TOCTOU guard to charset repair updates Per PR review: only write when the row still holds the exact corrupted value read (.eq account_name), so a concurrent rename is skipped, not clobbered, and the script is strictly idempotent. Track skipped count. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(charset-repair): build combining-marks regex from ASCII string Per PR review: the deaccent regex literal embedded raw U+0300–U+036F combining marks (invisible, encoding-fragile). Build it via RegExp('[\\u0300-\\u036f]') so the source is plain ASCII. Behavior-identical; 21 tests still green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jakob Wennberg
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
e5f1d7a916
commit
d95a0b6105
@@ -0,0 +1,194 @@
|
||||
/**
|
||||
* Comprehensive charset repair for chart_of_accounts account names.
|
||||
*
|
||||
* Background: account names were corrupted for ~888 companies in the seeding /
|
||||
* import window 2026-03-30 → 2026-06-12, in three signatures (see
|
||||
* lib/bookkeeping/charset-repair.ts): stripped diacritics, lost-byte U+FFFD, and
|
||||
* double-encoded UTF-8. The root causes are CLOSED — the runtime
|
||||
* seed_chart_of_accounts() function on prod now carries correct diacritics (the
|
||||
* fix migration 20260516130000 was applied to prod ~2026-06-12; classic
|
||||
* prod-migration-drift, so companies created during the drift window were
|
||||
* seeded by the old stripped function). This script repairs the legacy rows.
|
||||
*
|
||||
* Strategy (per row): resolveCorrectName() against CANDIDATE correct names for
|
||||
* the same account number — the clean sibling rows already in the table plus the
|
||||
* BAS reference name. Double-encoded reverses losslessly (recovers customs too);
|
||||
* stripped/lost-byte require an unambiguous clean sibling, else the row is left
|
||||
* untouched and reported. Never clobbers user-renamed accounts.
|
||||
*
|
||||
* Usage: npx tsx scripts/repair-chart-of-accounts-charset.ts [--execute]
|
||||
* Without --execute it prints the plan only (dry run). Idempotent: clean rows
|
||||
* resolve to no-op, so re-running is safe.
|
||||
*/
|
||||
import fs from 'fs'
|
||||
import path from 'path'
|
||||
import { createClient } from '@supabase/supabase-js'
|
||||
import { getBASReference } from '@/lib/bookkeeping/bas-reference'
|
||||
import {
|
||||
resolveCorrectName,
|
||||
hasLostByte,
|
||||
hasMojibakeSignature,
|
||||
hasCp1252Artifact,
|
||||
isClean,
|
||||
type RepairResult,
|
||||
} from '@/lib/bookkeeping/charset-repair'
|
||||
|
||||
const isCorrupt = (s: string) =>
|
||||
hasLostByte(s) || hasMojibakeSignature(s) || hasCp1252Artifact(s)
|
||||
|
||||
function loadEnv(): { url: string; key: string } {
|
||||
const envPath = path.resolve(process.cwd(), '.env.local')
|
||||
const vars: Record<string, string> = {}
|
||||
for (const line of fs.readFileSync(envPath, 'utf8').split('\n')) {
|
||||
const m = line.match(/^([A-Z0-9_]+)=(.*)$/)
|
||||
if (m) vars[m[1]] = m[2].trim()
|
||||
}
|
||||
const url = vars.NEXT_PUBLIC_SUPABASE_URL
|
||||
const key = vars.SUPABASE_SERVICE_ROLE_KEY
|
||||
if (!url || !key) throw new Error('Missing Supabase env in .env.local')
|
||||
if (!url.includes('pwxtzglxptnnvjrpixpg')) {
|
||||
throw new Error(`Refusing to run against unexpected project: ${url}`)
|
||||
}
|
||||
return { url, key }
|
||||
}
|
||||
|
||||
const EXECUTE = process.argv.includes('--execute')
|
||||
|
||||
interface Row {
|
||||
id: string
|
||||
company_id: string
|
||||
account_number: string
|
||||
account_name: string
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const { url, key } = loadEnv()
|
||||
const supabase = createClient(url, key, { auth: { persistSession: false } })
|
||||
|
||||
console.log(`=== chart_of_accounts charset repair — ${EXECUTE ? 'EXECUTE' : 'DRY RUN'} ===\n`)
|
||||
|
||||
// Closure over the inferred client so we don't annotate (and mismatch) the
|
||||
// supabase-js generic parameters.
|
||||
const fetchAll = async (): Promise<Row[]> => {
|
||||
const out: Row[] = []
|
||||
const PAGE = 1000
|
||||
for (let from = 0; ; from += PAGE) {
|
||||
const { data, error } = await supabase
|
||||
.from('chart_of_accounts')
|
||||
.select('id, company_id, account_number, account_name')
|
||||
.order('id', { ascending: true })
|
||||
.range(from, from + PAGE - 1)
|
||||
if (error) throw new Error(`fetch chart_of_accounts failed: ${error.message}`)
|
||||
const batch = (data ?? []) as Row[]
|
||||
out.push(...batch)
|
||||
if (batch.length < PAGE) break
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
const rows = await fetchAll()
|
||||
console.log(`Scanned ${rows.length} chart_of_accounts rows across ${new Set(rows.map((r) => r.company_id)).size} companies.\n`)
|
||||
|
||||
// Candidate correct names per account number = clean sibling names in the
|
||||
// table + the BAS reference name. (Clean = no lost-byte, no mojibake.)
|
||||
const candidates = new Map<string, Set<string>>()
|
||||
for (const r of rows) {
|
||||
if (isClean(r.account_name)) {
|
||||
const set = candidates.get(r.account_number) ?? new Set<string>()
|
||||
set.add(r.account_name)
|
||||
candidates.set(r.account_number, set)
|
||||
}
|
||||
}
|
||||
for (const r of rows) {
|
||||
const bas = getBASReference(r.account_number)?.account_name
|
||||
if (bas) {
|
||||
const set = candidates.get(r.account_number) ?? new Set<string>()
|
||||
set.add(bas)
|
||||
candidates.set(r.account_number, set)
|
||||
}
|
||||
}
|
||||
|
||||
const fixes: Array<Row & { result: RepairResult }> = []
|
||||
const unresolved: Row[] = []
|
||||
for (const r of rows) {
|
||||
const cand = [...(candidates.get(r.account_number) ?? new Set<string>())]
|
||||
const result = resolveCorrectName(r.account_name, cand)
|
||||
if (result && result.corrected !== r.account_name) {
|
||||
fixes.push({ ...r, result })
|
||||
} else if (!result && isCorrupt(r.account_name)) {
|
||||
unresolved.push(r) // carries a corruption signature but no confident fix
|
||||
}
|
||||
}
|
||||
|
||||
// ── Report ──────────────────────────────────────────────────────
|
||||
const byMethod = (m: RepairResult['method']) => fixes.filter((f) => f.result.method === m)
|
||||
const sample = (arr: Array<Row & { result: RepairResult }>) =>
|
||||
arr.slice(0, 8).map((f) => ` ${f.account_number} "${f.account_name}" → "${f.result.corrected}"`)
|
||||
|
||||
for (const m of ['sibling_stripped', 'reverse', 'reverse_cp437', 'sibling_lostbyte'] as const) {
|
||||
const g = byMethod(m)
|
||||
console.log(`[${m}] ${g.length} rows across ${new Set(g.map((f) => f.company_id)).size} companies`)
|
||||
if (g.length) console.log(sample(g).join('\n'))
|
||||
console.log()
|
||||
}
|
||||
console.log(`Total fixes: ${fixes.length} rows across ${new Set(fixes.map((f) => f.company_id)).size} companies.`)
|
||||
|
||||
if (unresolved.length) {
|
||||
console.log(`\n[unresolved] ${unresolved.length} rows carry a corruption signature but have no unambiguous clean sibling — left untouched:`)
|
||||
console.log(
|
||||
[...new Map(unresolved.map((u) => [`${u.account_number}|${u.account_name}`, u])).values()]
|
||||
.slice(0, 20)
|
||||
.map((u) => ` ${u.account_number} "${u.account_name}"`)
|
||||
.join('\n'),
|
||||
)
|
||||
}
|
||||
|
||||
if (!EXECUTE) {
|
||||
console.log('\nDRY RUN — no rows changed. Re-run with --execute to apply.')
|
||||
return
|
||||
}
|
||||
|
||||
// ── Apply (chunked) ─────────────────────────────────────────────
|
||||
let applied = 0
|
||||
let skipped = 0
|
||||
let errors = 0
|
||||
const CHUNK = 25
|
||||
for (let i = 0; i < fixes.length; i += CHUNK) {
|
||||
const chunk = fixes.slice(i, i + CHUNK)
|
||||
const results = await Promise.all(
|
||||
chunk.map((f) =>
|
||||
supabase
|
||||
.from('chart_of_accounts')
|
||||
.update({ account_name: f.result.corrected })
|
||||
.eq('id', f.id)
|
||||
.eq('company_id', f.company_id)
|
||||
// TOCTOU guard: only write if the row still holds the exact corrupted
|
||||
// value we read. A concurrent rename → 0 rows changed (skipped, not
|
||||
// clobbered). Also makes the script strictly idempotent.
|
||||
.eq('account_name', f.account_name)
|
||||
.select('id')
|
||||
.then(({ data, error }) =>
|
||||
error
|
||||
? { ok: false as const, msg: error.message }
|
||||
: { ok: true as const, changed: (data?.length ?? 0) > 0 }),
|
||||
),
|
||||
)
|
||||
for (const r of results) {
|
||||
if (!r.ok) {
|
||||
errors++
|
||||
console.error(` update failed: ${r.msg}`)
|
||||
} else if (r.changed) {
|
||||
applied++
|
||||
} else {
|
||||
skipped++ // row changed under us since the read — left as-is
|
||||
}
|
||||
}
|
||||
if ((i / CHUNK) % 10 === 0) console.log(` …${applied}/${fixes.length} applied`)
|
||||
}
|
||||
console.log(`\nDone. Applied ${applied}, skipped ${skipped}, errors ${errors}.`)
|
||||
}
|
||||
|
||||
main().catch((err) => {
|
||||
console.error('\nREPAIR FAILED:', err)
|
||||
process.exit(1)
|
||||
})
|
||||
Reference in New Issue
Block a user