feat: enable banking hardening, arcim inference, SIE fixes, onboarding (#32)
* feat: import system improvements, INK2 fix, and Swedish text corrections - SIE parser: Windows-1252 and CP437 encoding detection and decoding - Bank file parser: add Nordea Business (Företag) CSV format - Bank file parser: improve format detection for SEB, Länsförsäkringar, generic CSV - INK2 engine: calculate årets resultat (7222) from income statement for open fiscal years - Dashboard: parallel Supabase queries, simplified dashboard page - Fix Swedish characters (å, ä, ö) in BAS data descriptions, validation messages, AI consent disclosures - Import wizard UI improvements across all steps - Migration: add 'bas_range' match type to sie_account_mappings constraint - Extensive new tests for SIE parser encoding and bank file parser Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: arcim migration wizard UX fixes, Sentry setup, and extension scaffolding Arcim migration wizard improvements: - Progress bar now excludes non-interactive steps (migrating/result) - Fix OAuth text to match target="_blank" behavior (new tab, not redirect) - Display month names instead of "Månad X" in preview - Fix Swedish typo "förifylla" in no-company-info message - Replace native checkboxes with shadcn Switch in options step - Add ConfirmationDialog before starting migration - Show progress percentage during migration - Add "Nästa steg" guidance and navigation links in result step - Add "Försök igen" button in error state (returns to options) - Add Bokio company ID help text (GUID from URL) - Add Fortnox integration add-on hint on connection failure Also includes: SIE import system improvements, INK2 fixes, Swedish text corrections, Sentry error tracking setup, and arcim-migration extension scaffolding. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: address PR review feedback - Fix OAuth error recovery blank page (restore provider from URL params) - Pass real userId to MigrationWizard instead of empty string - Remove ~50 debug console.log statements from sie-import.ts - Fix comment referencing account 3740 → 3741 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: comprehensive UI design audit and normalization Dashboard audit: - Fix muted-foreground contrast (4.31:1 → 5.08:1) for WCAG AA - Add prefers-reduced-motion media query for all animations - Replace border-l-2 accent anti-pattern with subtle full-border colors - Add aria-expanded to toggle buttons, role="status" to live counters - Fix touch targets on deadline buttons (28px → 36px) - Vary section spacing for rhythm (mb-12/mb-10/mb-8) - Remove unused imports and dead code Transactions audit + hardening: - Add pagination (200 per page) with "Ladda fler" button - Replace height animation with transform-only exit animation - Show batch progress in floating action bar during processing - Fix batch bar mobile overlap (bottom-20 on mobile) - Replace clickable badges with proper button elements - Add safe area padding to fullscreen swipe view - Add response.ok check to suggestion fetch - Add truncation to invoice number buttons Invoicing audit: - Remove border-l-4 accent pattern from invoice cards - Replace string concatenation with cn() utility Systemic sweep (34 files): - All page headings: font-bold → font-display font-medium (Fraunces) - All stat numbers: font-bold → font-display font-medium tabular-nums - All hard-coded blue/amber/emerald colors → design tokens - Remove all dark mode overrides (tokens handle automatically) - Tint pure white card background to 99% Design context added to CLAUDE.md with brand personality, aesthetic direction, and 5 design principles. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: bookkeeping flow audit — design system, accessibility, UX - Replace raw <select> with shadcn Select component (JournalEntryForm) - Add confirmation dialog for account deletion (ChartOfAccountsManager) - Remove console.error from production code (JournalEntryList, JournalEntryForm) - Fix contradictory h-7/min-h-[44px] button sizing → h-10 (ChartOfAccountsManager) - Increase BAS catalog "Lägg till" touch target h-7 → h-9 - Improve loading state with spinner (JournalEntryList) - Improve empty state with icon, description, and guidance (JournalEntryList) - Add response.ok check on journal entry fetch - Add aria-expanded to entry expand buttons - Add tabular-nums to desktop debit/credit columns Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: onboarding and empty state improvements Onboarding: - Replace font-serif with font-display (Fraunces) for brand consistency - Remove console.error calls from production code Empty states: - Fix broken /transactions/new link in EmptyTransactions (route doesn't exist) - Add actionHref fallback to EmptyCustomers when no onAction prop provided - Improve EmptyTransactions description copy Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: clarify Swedish UX copy — terminology, errors, descriptions Terminology consistency: - "Försenad" → "Förfallen" for overdue invoices (customers/[id]) - "bokföringsorder" → actionable description in bookkeeping page - "verifikation har bifogats" → "underlag har bifogats" in doc warning - "Fortsätt ändå" → "Bokför utan underlag" (specific action) Error messages — replace generic "Fel" + "Något gick fel" with specific: - "Något gick fel vid bokföring" → "Transaktionen kunde inte bokföras" - "Något gick fel vid matchning" → "Transaktionen kunde inte matchas" - "Kunde inte hämta X" → "Kunde inte ladda X" + recovery hint - Add "Försök igen" guidance to all error toasts Page descriptions — replace redundant with actionable: - Invoices: "Skapa och hantera" → "Skicka, följ betalningar, skapa kreditnotor" - Bookkeeping: list of features → actionable description Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: design critique — dashboard affordance, reports description Dashboard: - Add ChevronRight indicator to clickable summary cards (Att få betalt, Koppla bank) to distinguish from static cards - Add cursor-pointer to linked cards Reports: - Replace feature list description with actionable guidance "Huvudbok, grundbok..." → "Generera skattedeklarationer..." Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: replace generic "Fel" error toasts with specific messages Deadlines: 5 generic "Fel" → specific per-action titles (create, toggle, edit, delete, load) Expenses detail: 5 generic "Fel" → specific per-action titles (load, approve, pay, credit, delete) Expenses new: 3 generic "Fel" → instructional validation messages (supplier name, supplier selection, invoice number) Customers: 1 generic "Fel" → specific load error with recovery hint All error toasts now follow pattern: title = what failed, description = how to recover Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: replace all remaining generic "Fel" error toasts (37 instances) Systematic sweep across 12 dashboard pages replacing generic title: 'Fel' with context-specific error titles: - Load errors: "Kunde inte ladda [resurs]" - Action errors: "[Åtgärd] misslyckades" - Validation: "[Fält] saknas" Every error toast now tells the user what failed without needing to read the description. Recovery hints added where missing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: import flow — normalize stat typography, remove console.warn - Replace font-bold with font-display font-medium on 13 stat numbers across SIEPreviewStep, BankFilePreviewStep, BankFileConfirmStep, ImportResultStep (missed by systemic sweep since these are in components/import/, not app/(dashboard)/) - Add tabular-nums to stat numbers displaying counts/currency - Remove console.warn in ArcimMigrationWorkspace Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: final cleanup — console statements, remaining font-bold stats Remove production console statements: - Step1EntityType: remove debug console.warn (dead code after onNext) - TransactionBookingDialog: remove console.error on doc link failure - JournalEntryAttachments: remove 3 console.error calls Normalize remaining font-bold stat displays: - SwipeCategorizationView: 3 instances (completion, amount displays) - NEDeclarationView: yearly result heading + value Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address Greptile review feedback loadMoreTransactions: add inbox item enrichment matching fetchTransactions - Paginated transactions now fetch invoice_inbox_items in parallel - Fixes missing document indicator, template suggestions, and inbox match card for transactions loaded via "Ladda fler" fetchAllPages: add maxPages guard (default 500) to prevent infinite loop - If Arcim gateway returns hasMore:true indefinitely, the loop now exits after 500 pages instead of running forever Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: minimize CLAUDE.md — remove derivable content, fix stale data Remove ~230 lines (51% reduction) of content that duplicates what's already in the source code (directory tree, function tables, type definitions, migration lists). Update migration count (63→65), add missing test helpers, fix cron job list. Keep all high-value sections: accounting guard rails, BAS accounts, VAT rutor, design context. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: enable banking hardening, arcim entity inference, SIE import fixes, and onboarding improvements - Enable Banking: OAuth CSRF state tokens, JWT caching, retry with timeouts, raw PSD2 response archival (BFL 7 kap), expired/error connection UI, consent expiry notifications, pagination safety limits - Arcim migration: Smarter entity type inference from org numbers, VAT prefixes, company name suffixes (GmbH, Ltd, etc.), and country codes - SIE import: Parser and import fixes with new migration - BAS accounts: Added vehicle accounts (1241, 1242, 1249, 1259) - Dashboard: New SIE import and stale uncategorized transaction queries - Onboarding: Enhanced NewUserChecklist - Period service: Improvements with updated tests - Transaction ingest: Updated logic and tests Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address Greptile review — credit note type, EU country codes, notification thresholds, migration timestamps - Fix dead ternary: credit notes now correctly stored as 'credit_note' instead of 'invoice' - Add 'GR' (Greece ISO 3166-1) to EU_COUNTRIES alongside 'EL' (VAT prefix) - Fix consent notification condition: fire at exactly 7 days or ≤3 days, not every day in 7-day window - Deduplicate migration timestamps: rename SIE migration to 20260316120100 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Jakob Wennberg
co-authored by
Claude Opus 4.6
Claude Opus 4.6
parent
56fb117d16
commit
98cd253bce
@@ -65,7 +65,7 @@ describe('BAS_REFERENCE data integrity', () => {
|
||||
})
|
||||
|
||||
describe('Non-standard accounts removed', () => {
|
||||
const nonStandard = ['1249', '1259', '1400', '1580', '3109', '4100', '4990', '7834', '7835', '7910', '8710']
|
||||
const nonStandard = ['1400', '1580', '3109', '4100', '4990', '7834', '7835', '7910', '8710']
|
||||
|
||||
for (const num of nonStandard) {
|
||||
it(`${num} is not in the catalog`, () => {
|
||||
|
||||
@@ -695,27 +695,104 @@ export const CLASS_1_ACCOUNTS: BASReferenceAccount[] = [
|
||||
k2_excluded: false,
|
||||
},
|
||||
{
|
||||
account_number: '1250',
|
||||
account_name: '(Fritt konto för Inventarier, verktyg och installationer)',
|
||||
account_number: '1241',
|
||||
account_name: 'Personbilar',
|
||||
account_class: 1,
|
||||
account_group: '12',
|
||||
account_type: 'asset',
|
||||
normal_balance: 'debit',
|
||||
description: '(Fritt konto för Inventarier, verktyg och installationer)',
|
||||
description: 'Personbilar',
|
||||
sru_code: '7202',
|
||||
k2_excluded: false,
|
||||
},
|
||||
{
|
||||
account_number: '1260',
|
||||
account_name: '(Fritt konto för Inventarier, verktyg och installationer)',
|
||||
account_number: '1242',
|
||||
account_name: 'Lastbilar och andra transportmedel',
|
||||
account_class: 1,
|
||||
account_group: '12',
|
||||
account_type: 'asset',
|
||||
normal_balance: 'debit',
|
||||
description: '(Fritt konto för Inventarier, verktyg och installationer)',
|
||||
description: 'Lastbilar och andra transportmedel',
|
||||
sru_code: '7202',
|
||||
k2_excluded: false,
|
||||
},
|
||||
{
|
||||
account_number: '1249',
|
||||
account_name: 'Ack. avskrivningar på bilar och andra transportmedel',
|
||||
account_class: 1,
|
||||
account_group: '12',
|
||||
account_type: 'asset',
|
||||
normal_balance: 'credit',
|
||||
description: 'Ackumulerade avskrivningar på bilar och andra transportmedel (kontra-tillgång)',
|
||||
sru_code: '7202',
|
||||
k2_excluded: false,
|
||||
},
|
||||
{
|
||||
account_number: '1250',
|
||||
account_name: 'Inventarier och verktyg',
|
||||
account_class: 1,
|
||||
account_group: '12',
|
||||
account_type: 'asset',
|
||||
normal_balance: 'debit',
|
||||
description: 'Inventarier och verktyg',
|
||||
sru_code: '7215',
|
||||
k2_excluded: false,
|
||||
},
|
||||
{
|
||||
account_number: '1251',
|
||||
account_name: 'Inventarier',
|
||||
account_class: 1,
|
||||
account_group: '12',
|
||||
account_type: 'asset',
|
||||
normal_balance: 'debit',
|
||||
description: 'Inventarier',
|
||||
sru_code: '7215',
|
||||
k2_excluded: false,
|
||||
},
|
||||
{
|
||||
account_number: '1259',
|
||||
account_name: 'Ack. avskrivningar på inventarier och verktyg',
|
||||
account_class: 1,
|
||||
account_group: '12',
|
||||
account_type: 'asset',
|
||||
normal_balance: 'credit',
|
||||
description: 'Ackumulerade avskrivningar på inventarier och verktyg (kontra-tillgång)',
|
||||
sru_code: '7215',
|
||||
k2_excluded: false,
|
||||
},
|
||||
{
|
||||
account_number: '1260',
|
||||
account_name: 'Datorer',
|
||||
account_class: 1,
|
||||
account_group: '12',
|
||||
account_type: 'asset',
|
||||
normal_balance: 'debit',
|
||||
description: 'Datorer',
|
||||
sru_code: '7215',
|
||||
k2_excluded: false,
|
||||
},
|
||||
{
|
||||
account_number: '1261',
|
||||
account_name: 'Datorer, bärbara',
|
||||
account_class: 1,
|
||||
account_group: '12',
|
||||
account_type: 'asset',
|
||||
normal_balance: 'debit',
|
||||
description: 'Datorer, bärbara',
|
||||
sru_code: '7215',
|
||||
k2_excluded: false,
|
||||
},
|
||||
{
|
||||
account_number: '1269',
|
||||
account_name: 'Ack. avskrivningar på datorer',
|
||||
account_class: 1,
|
||||
account_group: '12',
|
||||
account_type: 'asset',
|
||||
normal_balance: 'credit',
|
||||
description: 'Ackumulerade avskrivningar på datorer (kontra-tillgång)',
|
||||
sru_code: '7215',
|
||||
k2_excluded: false,
|
||||
},
|
||||
{
|
||||
account_number: '1280',
|
||||
account_name: 'Pågående nyanläggningar och förskott för maskiner respektive inventarier',
|
||||
|
||||
@@ -44,8 +44,9 @@ describe('lockPeriod', () => {
|
||||
const lockedPeriod = { ...period, locked_at: '2024-12-31T23:59:59Z' }
|
||||
|
||||
results = [
|
||||
{ data: period, error: null }, // fetch
|
||||
{ data: lockedPeriod, error: null }, // update
|
||||
{ data: period, error: null }, // fetch
|
||||
{ count: 0, data: null, error: null }, // uncategorized tx count check
|
||||
{ data: lockedPeriod, error: null }, // update
|
||||
]
|
||||
|
||||
const handler = vi.fn()
|
||||
|
||||
@@ -33,6 +33,22 @@ export async function lockPeriod(
|
||||
throw new Error('Period is already locked')
|
||||
}
|
||||
|
||||
// Check for uncategorized business transactions in this period
|
||||
const { count: unbookedCount } = await supabase
|
||||
.from('transactions')
|
||||
.select('id', { count: 'exact', head: true })
|
||||
.eq('user_id', userId)
|
||||
.is('journal_entry_id', null)
|
||||
.eq('is_business', true)
|
||||
.gte('date', period.period_start)
|
||||
.lte('date', period.period_end)
|
||||
|
||||
if (unbookedCount && unbookedCount > 0) {
|
||||
throw new Error(
|
||||
`Kan inte låsa period: ${unbookedCount} affärstransaktion(er) saknar bokföring. Bokför alla transaktioner innan perioden låses.`
|
||||
)
|
||||
}
|
||||
|
||||
const { data: updated, error: updateError } = await supabase
|
||||
.from('fiscal_periods')
|
||||
.update({ locked_at: new Date().toISOString() })
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
export interface ConsentExpiryEmailData {
|
||||
bankName: string
|
||||
daysUntilExpiry: number
|
||||
renewalUrl: string
|
||||
companyName: string
|
||||
isExpired: boolean
|
||||
}
|
||||
|
||||
/**
|
||||
* Generate HTML email for consent expiry notification
|
||||
*/
|
||||
export function generateConsentExpiryEmailHtml(data: ConsentExpiryEmailData): string {
|
||||
const { bankName, daysUntilExpiry, renewalUrl, companyName, isExpired } = data
|
||||
const headerColor = isExpired ? '#dc2626' : '#ea580c'
|
||||
const title = isExpired
|
||||
? 'Banksynkronisering har stoppats'
|
||||
: `Samtycket för ${bankName} löper ut snart`
|
||||
|
||||
return `
|
||||
<!DOCTYPE html>
|
||||
<html lang="sv">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>${title}</title>
|
||||
</head>
|
||||
<body style="margin: 0; padding: 0; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, sans-serif; line-height: 1.6; color: #333; background-color: #f9fafb;">
|
||||
<div style="max-width: 600px; margin: 0 auto; padding: 40px 20px;">
|
||||
<div style="background: white; border-radius: 12px; padding: 40px; box-shadow: 0 1px 3px rgba(0,0,0,0.1);">
|
||||
<div style="text-align: center; margin-bottom: 30px;">
|
||||
<div style="display: inline-block; background: ${headerColor}15; color: ${headerColor}; padding: 8px 16px; border-radius: 20px; font-size: 12px; font-weight: 600; text-transform: uppercase; letter-spacing: 0.5px;">
|
||||
${isExpired ? 'Åtgärd krävs' : 'Påminnelse'}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<h1 style="margin: 0 0 20px 0; font-size: 22px; font-weight: 600; color: #111; text-align: center;">
|
||||
${title}
|
||||
</h1>
|
||||
|
||||
<div style="margin-bottom: 30px;">
|
||||
${isExpired ? `
|
||||
<p style="margin: 0 0 15px 0; color: #dc2626; font-weight: 500;">
|
||||
PSD2-samtycket för ${bankName} har löpt ut. Automatisk transaktionssynkronisering är stoppad.
|
||||
</p>
|
||||
<p style="margin: 0 0 15px 0;">
|
||||
Förnya anslutningen för att återuppta synkroniseringen.
|
||||
</p>
|
||||
` : `
|
||||
<p style="margin: 0 0 15px 0;">
|
||||
Samtycket för ${bankName} löper ut om ${daysUntilExpiry} ${daysUntilExpiry === 1 ? 'dag' : 'dagar'}.
|
||||
</p>
|
||||
<p style="margin: 0 0 15px 0;">
|
||||
Förnya anslutningen innan samtycket löper ut för att undvika avbrott i transaktionssynkroniseringen.
|
||||
</p>
|
||||
`}
|
||||
</div>
|
||||
|
||||
<div style="text-align: center; margin-bottom: 30px;">
|
||||
<a href="${renewalUrl}" style="display: inline-block; background: ${headerColor}; color: white; padding: 14px 28px; border-radius: 8px; text-decoration: none; font-weight: 500; font-size: 14px;">
|
||||
${isExpired ? 'Förnya anslutning' : 'Hantera bankanslutningar'}
|
||||
</a>
|
||||
</div>
|
||||
|
||||
<div style="padding-top: 20px; border-top: 1px solid #e5e7eb;">
|
||||
<p style="margin: 0; color: #666; font-size: 14px;">
|
||||
Med vänliga hälsningar,<br>
|
||||
<strong>${companyName || 'gnubok'}</strong>
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
`
|
||||
}
|
||||
|
||||
/**
|
||||
* Generate plain text email for consent expiry notification
|
||||
*/
|
||||
export function generateConsentExpiryEmailText(data: ConsentExpiryEmailData): string {
|
||||
const { bankName, daysUntilExpiry, renewalUrl, companyName, isExpired } = data
|
||||
|
||||
let text = ''
|
||||
|
||||
if (isExpired) {
|
||||
text += `BANKSYNKRONISERING HAR STOPPATS\n`
|
||||
text += `=`.repeat(40) + `\n\n`
|
||||
text += `PSD2-samtycket för ${bankName} har löpt ut.\n`
|
||||
text += `Automatisk transaktionssynkronisering är stoppad.\n\n`
|
||||
text += `Förnya anslutningen för att återuppta synkroniseringen.\n\n`
|
||||
} else {
|
||||
text += `SAMTYCKET FÖR ${bankName.toUpperCase()} LÖPER UT SNART\n`
|
||||
text += `=`.repeat(40) + `\n\n`
|
||||
text += `Samtycket för ${bankName} löper ut om ${daysUntilExpiry} ${daysUntilExpiry === 1 ? 'dag' : 'dagar'}.\n\n`
|
||||
text += `Förnya anslutningen innan samtycket löper ut för att undvika avbrott.\n\n`
|
||||
}
|
||||
|
||||
text += `Hantera bankanslutningar: ${renewalUrl}\n\n`
|
||||
text += `Med vänliga hälsningar,\n`
|
||||
text += `${companyName || 'gnubok'}\n`
|
||||
|
||||
return text
|
||||
}
|
||||
|
||||
/**
|
||||
* Generate email subject for consent expiry notification
|
||||
*/
|
||||
export function generateConsentExpiryEmailSubject(data: ConsentExpiryEmailData): string {
|
||||
if (data.isExpired) {
|
||||
return `Banksynkronisering stoppad — ${data.bankName}`
|
||||
}
|
||||
return `Banksamtycke löper ut om ${data.daysUntilExpiry} ${data.daysUntilExpiry === 1 ? 'dag' : 'dagar'} — ${data.bankName}`
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
import {
|
||||
Camera,
|
||||
Sparkles,
|
||||
Wand,
|
||||
MessageSquare,
|
||||
Bell,
|
||||
Inbox,
|
||||
@@ -33,7 +33,7 @@ import {
|
||||
|
||||
const ICON_MAP: Record<string, LucideIcon> = {
|
||||
Camera,
|
||||
Sparkles,
|
||||
Wand,
|
||||
MessageSquare,
|
||||
Bell,
|
||||
Inbox,
|
||||
|
||||
+306
-102
@@ -94,11 +94,32 @@ export async function checkDuplicateImport(
|
||||
.select('*')
|
||||
.eq('user_id', userId)
|
||||
.eq('file_hash', fileHash)
|
||||
.eq('status', 'completed')
|
||||
.single()
|
||||
|
||||
return data as SIEImport | null
|
||||
}
|
||||
|
||||
/**
|
||||
* Clean up stale pending/failed import records for a given file hash.
|
||||
* Prevents UNIQUE constraint conflicts when re-importing after a failure.
|
||||
*/
|
||||
async function cleanupStaleImportRecords(
|
||||
supabase: SupabaseClient,
|
||||
userId: string,
|
||||
fileHash: string
|
||||
): Promise<void> {
|
||||
const oneHourAgo = new Date(Date.now() - 60 * 60 * 1000).toISOString()
|
||||
|
||||
await supabase
|
||||
.from('sie_imports')
|
||||
.delete()
|
||||
.eq('user_id', userId)
|
||||
.eq('file_hash', fileHash)
|
||||
.in('status', ['pending', 'failed'])
|
||||
.lt('created_at', oneHourAgo)
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a fiscal period if one doesn't exist for the date range
|
||||
*/
|
||||
@@ -342,6 +363,8 @@ async function importVouchers(
|
||||
originalLineCount?: number
|
||||
}[]
|
||||
voucherNumberMapping: Array<{ sourceId: string; targetNumber: number }>
|
||||
retriedBatches: number
|
||||
failedBatches: number
|
||||
}> {
|
||||
const results = {
|
||||
created: 0,
|
||||
@@ -366,6 +389,8 @@ async function importVouchers(
|
||||
originalLineCount?: number
|
||||
}[],
|
||||
voucherNumberMapping: [] as Array<{ sourceId: string; targetNumber: number }>,
|
||||
retriedBatches: 0,
|
||||
failedBatches: 0,
|
||||
}
|
||||
|
||||
// Pre-filter and prepare all valid vouchers
|
||||
@@ -507,17 +532,9 @@ async function importVouchers(
|
||||
})
|
||||
}
|
||||
|
||||
// Compute per-account net movements from vouchers that will be imported.
|
||||
// Used for UB/RES reconciliation to generate migration adjustment entries.
|
||||
for (const v of preparedVouchers) {
|
||||
for (const l of v.lines) {
|
||||
const net = l.debit_amount - l.credit_amount
|
||||
results.movementsByAccount.set(
|
||||
l.account_number,
|
||||
(results.movementsByAccount.get(l.account_number) || 0) + net
|
||||
)
|
||||
}
|
||||
}
|
||||
// NOTE: Per-account net movements are tracked inside the batch loop below,
|
||||
// so that only SUCCESSFULLY inserted vouchers are counted. This ensures
|
||||
// the migration adjustment entry correctly compensates for failed batches.
|
||||
|
||||
if (preparedVouchers.length === 0) {
|
||||
return results
|
||||
@@ -552,11 +569,17 @@ async function importVouchers(
|
||||
|
||||
const currentVoucherNumber = (startNumber as number) || 1
|
||||
|
||||
// Batch insert journal entries (in chunks of 100)
|
||||
// Batch insert journal entries (in chunks of 100) with retry logic.
|
||||
// Retries handle transient errors (Supabase rate limits, Cloudflare 500s).
|
||||
const BATCH_SIZE = 100
|
||||
const MAX_RETRIES = 3
|
||||
const INTER_BATCH_DELAY_MS = 50 // Prevent rate limiting under sustained load
|
||||
let retriedBatches = 0
|
||||
let failedBatches = 0
|
||||
|
||||
for (let batchStart = 0; batchStart < preparedVouchers.length; batchStart += BATCH_SIZE) {
|
||||
const batch = preparedVouchers.slice(batchStart, batchStart + BATCH_SIZE)
|
||||
const batchNumber = Math.floor(batchStart / BATCH_SIZE) + 1
|
||||
|
||||
// Prepare journal entry headers
|
||||
const entryInserts = batch.map((v, i) => ({
|
||||
@@ -571,14 +594,37 @@ async function importVouchers(
|
||||
committed_at: new Date().toISOString(),
|
||||
}))
|
||||
|
||||
// Insert headers
|
||||
const { data: entries, error: entryError } = await supabase
|
||||
.from('journal_entries')
|
||||
.insert(entryInserts)
|
||||
.select('id')
|
||||
// Insert headers with retry
|
||||
let entries: { id: string }[] | null = null
|
||||
let lastEntryError: string | null = null
|
||||
|
||||
if (entryError || !entries) {
|
||||
results.errors.push(`Batch ${Math.floor(batchStart / BATCH_SIZE) + 1}: ${entryError?.message || 'Failed to insert entries'}`)
|
||||
for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {
|
||||
if (attempt > 0) {
|
||||
retriedBatches++
|
||||
const backoffMs = Math.pow(2, attempt - 1) * 1000 // 1s, 2s, 4s
|
||||
console.log(`[sie-import] Retrying batch ${batchNumber} (attempt ${attempt + 1}/${MAX_RETRIES + 1}) after ${backoffMs}ms`)
|
||||
await new Promise(resolve => setTimeout(resolve, backoffMs))
|
||||
}
|
||||
|
||||
const { data, error: entryError } = await supabase
|
||||
.from('journal_entries')
|
||||
.insert(entryInserts)
|
||||
.select('id')
|
||||
|
||||
if (!entryError && data) {
|
||||
entries = data
|
||||
lastEntryError = null
|
||||
break
|
||||
}
|
||||
|
||||
lastEntryError = entryError?.message || 'Failed to insert entries'
|
||||
}
|
||||
|
||||
if (!entries) {
|
||||
failedBatches++
|
||||
results.errors.push(
|
||||
`Batch ${batchNumber} misslyckades efter ${MAX_RETRIES + 1} försök: ${lastEntryError}`
|
||||
)
|
||||
continue
|
||||
}
|
||||
|
||||
@@ -613,7 +659,6 @@ async function importVouchers(
|
||||
})
|
||||
})
|
||||
|
||||
// Fix 7: Capture voucher number mapping (source → target)
|
||||
results.voucherNumberMapping.push({
|
||||
sourceId: voucher.sourceId,
|
||||
targetNumber: assignedNumber,
|
||||
@@ -623,15 +668,68 @@ async function importVouchers(
|
||||
results.created++
|
||||
}
|
||||
|
||||
// Insert all lines for this batch
|
||||
// Insert all lines with retry
|
||||
if (allLines.length > 0) {
|
||||
const { error: linesError } = await supabase
|
||||
.from('journal_entry_lines')
|
||||
.insert(allLines)
|
||||
let linesInserted = false
|
||||
let lastLinesError: string | null = null
|
||||
|
||||
if (linesError) {
|
||||
results.errors.push(`Batch ${Math.floor(batchStart / BATCH_SIZE) + 1} lines: ${linesError.message}`)
|
||||
for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {
|
||||
if (attempt > 0) {
|
||||
retriedBatches++
|
||||
const backoffMs = Math.pow(2, attempt - 1) * 1000
|
||||
console.log(`[sie-import] Retrying batch ${batchNumber} lines (attempt ${attempt + 1}/${MAX_RETRIES + 1}) after ${backoffMs}ms`)
|
||||
await new Promise(resolve => setTimeout(resolve, backoffMs))
|
||||
}
|
||||
|
||||
const { error: linesError } = await supabase
|
||||
.from('journal_entry_lines')
|
||||
.insert(allLines)
|
||||
|
||||
if (!linesError) {
|
||||
linesInserted = true
|
||||
break
|
||||
}
|
||||
|
||||
lastLinesError = linesError.message
|
||||
}
|
||||
|
||||
if (linesInserted) {
|
||||
// Track movements ONLY for successfully inserted vouchers.
|
||||
// This ensures the migration adjustment correctly compensates for
|
||||
// any batches that failed completely.
|
||||
for (let i = 0; i < batch.length; i++) {
|
||||
const voucher = batch[i]
|
||||
for (const line of voucher.lines) {
|
||||
const net = line.debit_amount - line.credit_amount
|
||||
results.movementsByAccount.set(
|
||||
line.account_number,
|
||||
(results.movementsByAccount.get(line.account_number) || 0) + net
|
||||
)
|
||||
}
|
||||
}
|
||||
} else {
|
||||
failedBatches++
|
||||
results.errors.push(
|
||||
`Batch ${batchNumber} rader misslyckades efter ${MAX_RETRIES + 1} försök: ${lastLinesError}`
|
||||
)
|
||||
}
|
||||
} else {
|
||||
// No lines to insert — still count movements for vouchers with entries
|
||||
for (let i = 0; i < batch.length; i++) {
|
||||
const voucher = batch[i]
|
||||
for (const line of voucher.lines) {
|
||||
const net = line.debit_amount - line.credit_amount
|
||||
results.movementsByAccount.set(
|
||||
line.account_number,
|
||||
(results.movementsByAccount.get(line.account_number) || 0) + net
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Small delay between batches to prevent Supabase/Cloudflare rate limiting
|
||||
if (batchStart + BATCH_SIZE < preparedVouchers.length) {
|
||||
await new Promise(resolve => setTimeout(resolve, INTER_BATCH_DELAY_MS))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -648,6 +746,10 @@ async function importVouchers(
|
||||
})
|
||||
}
|
||||
|
||||
// Propagate batch retry stats
|
||||
results.retriedBatches = retriedBatches
|
||||
results.failedBatches = failedBatches
|
||||
|
||||
return results
|
||||
}
|
||||
|
||||
@@ -882,19 +984,21 @@ async function ensureAccountExists(
|
||||
}
|
||||
|
||||
/**
|
||||
* Record the import in the database and archive the SIE file to Supabase Storage.
|
||||
* Phase 1: Create a pending import record early, before any journal entries.
|
||||
* This ensures the import is tracked even if later steps fail.
|
||||
*/
|
||||
async function recordImport(
|
||||
async function createPendingImportRecord(
|
||||
supabase: SupabaseClient,
|
||||
userId: string,
|
||||
parsed: ParsedSIEFile,
|
||||
fileContent: string,
|
||||
filename: string,
|
||||
result: ImportResult,
|
||||
documentation?: MigrationDocumentation
|
||||
filename: string
|
||||
): Promise<string> {
|
||||
const fileHash = await calculateFileHash(fileContent)
|
||||
|
||||
// Clean up any stale pending/failed records for this hash to avoid UNIQUE conflicts
|
||||
await cleanupStaleImportRecords(supabase, userId, fileHash)
|
||||
|
||||
const { data, error } = await supabase
|
||||
.from('sie_imports')
|
||||
.insert({
|
||||
@@ -911,40 +1015,65 @@ async function recordImport(
|
||||
? formatDate(parsed.stats.fiscalYearEnd)
|
||||
: null,
|
||||
accounts_count: parsed.stats.totalAccounts,
|
||||
transactions_count: result.journalEntriesCreated,
|
||||
status: result.success ? 'completed' : 'failed',
|
||||
error_message: result.errors.length > 0 ? result.errors.join('; ') : null,
|
||||
fiscal_period_id: result.fiscalPeriodId,
|
||||
opening_balance_entry_id: result.openingBalanceEntryId,
|
||||
imported_at: new Date().toISOString(),
|
||||
migration_documentation: documentation ?? null,
|
||||
transactions_count: 0,
|
||||
status: 'pending',
|
||||
imported_at: null,
|
||||
})
|
||||
.select('id')
|
||||
.single()
|
||||
|
||||
if (error || !data) {
|
||||
throw new Error(`Failed to record import: ${error?.message}`)
|
||||
}
|
||||
|
||||
// Archive the SIE file to Supabase Storage (BFL 7 kap 1-2§ retention)
|
||||
const storagePath = `${userId}/${data.id}.se`
|
||||
const fileBlob = new Blob([fileContent], { type: 'text/plain; charset=cp437' })
|
||||
const { error: uploadError } = await supabase.storage
|
||||
.from('sie-files')
|
||||
.upload(storagePath, fileBlob, { upsert: false })
|
||||
|
||||
if (uploadError) {
|
||||
console.error(`[sie-import] Failed to archive SIE file: ${uploadError.message}`)
|
||||
} else {
|
||||
await supabase
|
||||
.from('sie_imports')
|
||||
.update({ file_storage_path: storagePath })
|
||||
.eq('id', data.id)
|
||||
throw new Error(`Failed to create pending import record: ${error?.message}`)
|
||||
}
|
||||
|
||||
return data.id
|
||||
}
|
||||
|
||||
/**
|
||||
* Phase 2: Finalize the import record with results and archive the SIE file.
|
||||
*/
|
||||
async function finalizeImportRecord(
|
||||
supabase: SupabaseClient,
|
||||
importId: string,
|
||||
userId: string,
|
||||
result: ImportResult,
|
||||
fileContent: string,
|
||||
documentation?: MigrationDocumentation
|
||||
): Promise<void> {
|
||||
const status = result.success ? 'completed' : 'failed'
|
||||
|
||||
await supabase
|
||||
.from('sie_imports')
|
||||
.update({
|
||||
status,
|
||||
imported_at: result.success ? new Date().toISOString() : null,
|
||||
transactions_count: result.journalEntriesCreated,
|
||||
error_message: result.errors.length > 0 ? result.errors.join('; ') : null,
|
||||
fiscal_period_id: result.fiscalPeriodId,
|
||||
opening_balance_entry_id: result.openingBalanceEntryId,
|
||||
migration_documentation: documentation ?? null,
|
||||
})
|
||||
.eq('id', importId)
|
||||
|
||||
// Archive the SIE file to Supabase Storage (BFL 7 kap 1-2§ retention)
|
||||
if (result.success) {
|
||||
const storagePath = `${userId}/${importId}.se`
|
||||
const fileBlob = new Blob([fileContent], { type: 'text/plain; charset=cp437' })
|
||||
const { error: uploadError } = await supabase.storage
|
||||
.from('sie-files')
|
||||
.upload(storagePath, fileBlob, { upsert: false })
|
||||
|
||||
if (uploadError) {
|
||||
console.error(`[sie-import] Failed to archive SIE file: ${uploadError.message}`)
|
||||
} else {
|
||||
await supabase
|
||||
.from('sie_imports')
|
||||
.update({ file_storage_path: storagePath })
|
||||
.eq('id', importId)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Save account mappings to the database for future use
|
||||
*/
|
||||
@@ -1043,15 +1172,24 @@ export async function executeSIEImport(
|
||||
return result
|
||||
}
|
||||
|
||||
// Check for duplicate import
|
||||
// Check for duplicate import (only completed imports count as duplicates)
|
||||
const duplicate = await checkDuplicateImport(supabase, userId, options.fileContent)
|
||||
if (duplicate) {
|
||||
result.errors.push(
|
||||
`This file has already been imported on ${new Date(duplicate.imported_at!).toLocaleDateString('sv-SE')}`
|
||||
`This file has already been imported on ${duplicate.imported_at ? new Date(duplicate.imported_at).toLocaleDateString('sv-SE') : 'okänt datum'}`
|
||||
)
|
||||
return result
|
||||
}
|
||||
|
||||
// Create pending import record early — ensures tracking even if later steps fail
|
||||
result.importId = await createPendingImportRecord(
|
||||
supabase,
|
||||
userId,
|
||||
parsed,
|
||||
options.fileContent,
|
||||
options.filename
|
||||
)
|
||||
|
||||
// Build account mapping lookup
|
||||
const accountMap = mappingsToMap(mappings)
|
||||
|
||||
@@ -1108,8 +1246,10 @@ export async function executeSIEImport(
|
||||
|
||||
// Track documentation data across import phases
|
||||
let ibRoundingAdjustment = 0
|
||||
let ibExplanation: 'unallocated_result' | 'excluded_accounts' | 'rounding' | null = null
|
||||
let migrationAdjustmentInfo = { created: false, deltaAccounts: 0, entryId: null as string | null }
|
||||
let voucherNumberMapping: Array<{ sourceId: string; targetNumber: number }> = []
|
||||
let voucherRetryStats = { retriedBatches: 0, failedBatches: 0 }
|
||||
let voucherStats = {
|
||||
total: parsed.vouchers.length,
|
||||
imported: 0,
|
||||
@@ -1131,48 +1271,62 @@ export async function executeSIEImport(
|
||||
// explicit documentation. We never reject based on IB imbalance — the
|
||||
// original goal was to stop SILENT equity alteration, not prevent it.
|
||||
if (options.importOpeningBalances && parsed.openingBalances.length > 0 && result.fiscalPeriodId) {
|
||||
const ibValidation = validateIBBalance(parsed, accountMap)
|
||||
// Check if opening balances already exist for this period
|
||||
const { data: period } = await supabase
|
||||
.from('fiscal_periods')
|
||||
.select('opening_balances_set, opening_balance_entry_id')
|
||||
.eq('id', result.fiscalPeriodId)
|
||||
.single()
|
||||
|
||||
if (ibValidation.lines.length > 0) {
|
||||
const absAdj = Math.abs(ibValidation.roundingAdjustment)
|
||||
if (period?.opening_balances_set || period?.opening_balance_entry_id) {
|
||||
result.warnings.push('Ingående balanser finns redan för denna period — hoppar över IB-import')
|
||||
} else {
|
||||
const ibValidation = validateIBBalance(parsed, accountMap)
|
||||
|
||||
if (absAdj > 0.01) {
|
||||
ibRoundingAdjustment = ibValidation.roundingAdjustment
|
||||
if (ibValidation.lines.length > 0) {
|
||||
const absAdj = Math.abs(ibValidation.roundingAdjustment)
|
||||
|
||||
// Produce a descriptive warning explaining the source of the imbalance
|
||||
if (Math.abs(ibValidation.excludedAccountsTotal) > 0.01 && ibValidation.fileImbalance <= 1.00) {
|
||||
// File-level IB is balanced — imbalance is entirely from excluded system accounts
|
||||
result.warnings.push(
|
||||
`Exkluderade systemkonton har IB-saldon på totalt ${ibValidation.excludedAccountsTotal} SEK. ` +
|
||||
`Differensen (${ibValidation.roundingAdjustment} SEK) bokförs på konto 2099.`
|
||||
)
|
||||
} else if (ibValidation.fileImbalance > 1.00) {
|
||||
// File-level IB doesn't balance — likely unallocated årets resultat from previous year
|
||||
result.warnings.push(
|
||||
`Ingående balanser obalanserade med ${ibValidation.roundingAdjustment} SEK ` +
|
||||
`(troligen ej allokerat årets resultat från föregående räkenskapsår). ` +
|
||||
`Differensen bokförs på konto 2099 (Årets resultat).`
|
||||
)
|
||||
} else {
|
||||
// Small rounding
|
||||
result.warnings.push(
|
||||
`Avrundningsdifferens vid SIE-import: ${ibValidation.roundingAdjustment} SEK bokförd på konto 2099`
|
||||
)
|
||||
if (absAdj > 0.01) {
|
||||
ibRoundingAdjustment = ibValidation.roundingAdjustment
|
||||
|
||||
// Produce a descriptive warning explaining the source of the imbalance
|
||||
if (Math.abs(ibValidation.excludedAccountsTotal) > 0.01 && ibValidation.fileImbalance <= 1.00) {
|
||||
// File-level IB is balanced — imbalance is entirely from excluded system accounts
|
||||
ibExplanation = 'excluded_accounts'
|
||||
result.warnings.push(
|
||||
`Exkluderade systemkonton har IB-saldon på totalt ${ibValidation.excludedAccountsTotal} SEK. ` +
|
||||
`Differensen (${ibValidation.roundingAdjustment} SEK) bokförs på konto 2099.`
|
||||
)
|
||||
} else if (ibValidation.fileImbalance > 1.00) {
|
||||
// File-level IB doesn't balance — likely unallocated årets resultat from previous year
|
||||
ibExplanation = 'unallocated_result'
|
||||
result.warnings.push(
|
||||
`Ingående balanser obalanserade med ${ibValidation.roundingAdjustment} SEK ` +
|
||||
`(troligen ej allokerat årets resultat från föregående räkenskapsår). ` +
|
||||
`Differensen bokförs på konto 2099 (Årets resultat).`
|
||||
)
|
||||
} else {
|
||||
// Small rounding
|
||||
ibExplanation = 'rounding'
|
||||
result.warnings.push(
|
||||
`Avrundningsdifferens vid SIE-import: ${ibValidation.roundingAdjustment} SEK bokförd på konto 2099`
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
result.openingBalanceEntryId = await createOpeningBalanceEntry(
|
||||
supabase,
|
||||
userId,
|
||||
result.fiscalPeriodId,
|
||||
parsed,
|
||||
accountMap,
|
||||
ibRoundingAdjustment
|
||||
)
|
||||
result.openingBalanceEntryId = await createOpeningBalanceEntry(
|
||||
supabase,
|
||||
userId,
|
||||
result.fiscalPeriodId,
|
||||
parsed,
|
||||
accountMap,
|
||||
ibRoundingAdjustment
|
||||
)
|
||||
|
||||
if (result.openingBalanceEntryId) {
|
||||
result.journalEntriesCreated++
|
||||
result.journalEntryIds.push(result.openingBalanceEntryId)
|
||||
if (result.openingBalanceEntryId) {
|
||||
result.journalEntriesCreated++
|
||||
result.journalEntryIds.push(result.openingBalanceEntryId)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1216,6 +1370,10 @@ export async function executeSIEImport(
|
||||
result.journalEntryIds.push(...voucherResults.ids)
|
||||
result.errors.push(...voucherResults.errors)
|
||||
voucherNumberMapping = voucherResults.voucherNumberMapping
|
||||
voucherRetryStats = {
|
||||
retriedBatches: voucherResults.retriedBatches,
|
||||
failedBatches: voucherResults.failedBatches,
|
||||
}
|
||||
|
||||
// Update stats for documentation
|
||||
voucherStats = {
|
||||
@@ -1287,10 +1445,15 @@ export async function executeSIEImport(
|
||||
}
|
||||
}
|
||||
|
||||
// Save account mappings for future use
|
||||
await saveMappings(supabase, userId, mappings)
|
||||
// Save account mappings for future use (non-fatal — import data is already committed)
|
||||
try {
|
||||
await saveMappings(supabase, userId, mappings)
|
||||
} catch (mappingError) {
|
||||
console.error('[sie-import] Failed to save mappings (non-fatal):', mappingError)
|
||||
result.warnings.push('Kunde inte spara kontomappningar — påverkar inte importerade data')
|
||||
}
|
||||
|
||||
// Fix 6: Generate systemdokumentation (MigrationDocumentation)
|
||||
// Generate systemdokumentation (MigrationDocumentation)
|
||||
const mappingStats = getMappingStats(mappings)
|
||||
const documentation: MigrationDocumentation = {
|
||||
sourceSystem: parsed.header.program,
|
||||
@@ -1323,17 +1486,43 @@ export async function executeSIEImport(
|
||||
voucherNumberMapping,
|
||||
}
|
||||
|
||||
// Set success before recording so recordImport() sees correct status
|
||||
// Populate structured details for the UI
|
||||
const totalSkippedForDetails = voucherStats.skippedUnbalanced + voucherStats.skippedUnmapped +
|
||||
voucherStats.skippedSingleLine + voucherStats.skippedEmpty
|
||||
result.details = {
|
||||
fiscalYear: fiscalYearStart && fiscalYearEnd
|
||||
? { start: formatDate(fiscalYearStart), end: formatDate(fiscalYearEnd) }
|
||||
: undefined,
|
||||
skippedVouchers: totalSkippedForDetails > 0 ? {
|
||||
unbalanced: voucherStats.skippedUnbalanced,
|
||||
unmapped: voucherStats.skippedUnmapped,
|
||||
singleLine: voucherStats.skippedSingleLine,
|
||||
empty: voucherStats.skippedEmpty,
|
||||
total: totalSkippedForDetails,
|
||||
} : undefined,
|
||||
openingBalance: ibRoundingAdjustment !== 0 ? {
|
||||
imbalance: ibRoundingAdjustment,
|
||||
explanation: ibExplanation,
|
||||
bookedToAccount: '2099',
|
||||
} : undefined,
|
||||
migrationAdjustment: migrationAdjustmentInfo.created ? {
|
||||
created: true,
|
||||
accountsAdjusted: migrationAdjustmentInfo.deltaAccounts,
|
||||
} : undefined,
|
||||
retriedBatches: voucherRetryStats.retriedBatches,
|
||||
failedBatches: voucherRetryStats.failedBatches,
|
||||
}
|
||||
|
||||
// Set success before finalizing
|
||||
result.success = result.errors.length === 0
|
||||
|
||||
// Record the import with documentation
|
||||
result.importId = await recordImport(
|
||||
// Finalize the import record with results and documentation
|
||||
await finalizeImportRecord(
|
||||
supabase,
|
||||
result.importId,
|
||||
userId,
|
||||
parsed,
|
||||
options.fileContent,
|
||||
options.filename,
|
||||
result,
|
||||
options.fileContent,
|
||||
documentation
|
||||
)
|
||||
|
||||
@@ -1348,6 +1537,21 @@ export async function executeSIEImport(
|
||||
result.errors.push(
|
||||
`Import failed: ${error instanceof Error ? error.message : 'Unknown error'}`
|
||||
)
|
||||
|
||||
// Mark the pending import as failed if we created one
|
||||
if (result.importId) {
|
||||
try {
|
||||
await finalizeImportRecord(
|
||||
supabase,
|
||||
result.importId,
|
||||
userId,
|
||||
result,
|
||||
options.fileContent
|
||||
)
|
||||
} catch (finalizeError) {
|
||||
console.error('[sie-import] Failed to finalize import record on error:', finalizeError)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return result
|
||||
|
||||
@@ -140,15 +140,6 @@ export function detectEncoding(buffer: ArrayBuffer): SIEEncoding {
|
||||
}
|
||||
}
|
||||
|
||||
// CP437 Swedish chars live in 0x80-0x9F
|
||||
if (byte >= 0x80 && byte <= 0x9f && CP437_MAP[byte]) {
|
||||
cp437Count++
|
||||
}
|
||||
|
||||
// Windows-1252 Swedish chars live in 0xC0-0xFF
|
||||
if (WIN1252_SWEDISH_BYTES.has(byte)) {
|
||||
win1252Count++
|
||||
}
|
||||
}
|
||||
|
||||
if (utf8Count > cp437Count && utf8Count > win1252Count) return 'utf8'
|
||||
|
||||
@@ -191,6 +191,7 @@ export interface SIEImport {
|
||||
opening_balance_entry_id: string | null
|
||||
imported_at: string | null
|
||||
migration_documentation: MigrationDocumentation | null
|
||||
file_storage_path: string | null
|
||||
created_at: string
|
||||
updated_at: string
|
||||
}
|
||||
@@ -233,6 +234,47 @@ export interface ImportOptions {
|
||||
voucherSeries?: string
|
||||
}
|
||||
|
||||
/**
|
||||
* Structured import details for UI display.
|
||||
* Provides machine-readable data so the UI can render proper explanations
|
||||
* instead of parsing warning strings.
|
||||
*/
|
||||
export interface ImportResultDetails {
|
||||
/** Fiscal year this import covers */
|
||||
fiscalYear?: { start: string; end: string }
|
||||
|
||||
/** Breakdown of skipped vouchers by reason */
|
||||
skippedVouchers?: {
|
||||
unbalanced: number
|
||||
unmapped: number
|
||||
singleLine: number
|
||||
empty: number
|
||||
total: number
|
||||
}
|
||||
|
||||
/** Opening balance imbalance info */
|
||||
openingBalance?: {
|
||||
/** SEK amount of the imbalance (0 if balanced) */
|
||||
imbalance: number
|
||||
/** Why the imbalance exists */
|
||||
explanation: 'unallocated_result' | 'excluded_accounts' | 'rounding' | null
|
||||
/** Account the difference was booked to */
|
||||
bookedToAccount: string | null
|
||||
}
|
||||
|
||||
/** Migration adjustment entry info */
|
||||
migrationAdjustment?: {
|
||||
created: boolean
|
||||
accountsAdjusted: number
|
||||
}
|
||||
|
||||
/** Number of batches that needed retries (0 = clean run) */
|
||||
retriedBatches: number
|
||||
|
||||
/** Number of batches that still failed after all retries */
|
||||
failedBatches: number
|
||||
}
|
||||
|
||||
/**
|
||||
* Result of executing an import
|
||||
*/
|
||||
@@ -249,6 +291,9 @@ export interface ImportResult {
|
||||
// Issues
|
||||
errors: string[]
|
||||
warnings: string[]
|
||||
|
||||
// Structured details for UI (populated alongside warnings for backwards compat)
|
||||
details?: ImportResultDetails
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -112,6 +112,12 @@ function makeMappingResult(overrides: Record<string, unknown> = {}) {
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Tests
|
||||
//
|
||||
// Queue order after batch dedup refactor:
|
||||
// 1. Booked transaction map query
|
||||
// 2. Supplier invoices fetch
|
||||
// 3. Batch external_id dedup query (returns matching external_ids)
|
||||
// 4. Per-transaction: insert, updates, etc.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('ingestTransactions', () => {
|
||||
@@ -134,8 +140,8 @@ describe('ingestTransactions', () => {
|
||||
enqueue({ data: [], error: null })
|
||||
// Supplier invoices fetch (no unpaid invoices)
|
||||
enqueue({ data: [], error: null })
|
||||
// Dedup check returns null (no existing row)
|
||||
enqueue({ data: null, error: null })
|
||||
// Batch external_id dedup query (no matches)
|
||||
enqueue({ data: [], error: null })
|
||||
// Insert returns the new transaction
|
||||
enqueue({ data: inserted, error: null })
|
||||
// evaluateMappingRules will be called but we want low confidence
|
||||
@@ -160,8 +166,8 @@ describe('ingestTransactions', () => {
|
||||
enqueue({ data: [], error: null })
|
||||
// Supplier invoices fetch
|
||||
enqueue({ data: [], error: null })
|
||||
// Dedup check returns an existing record
|
||||
enqueue({ data: { id: 'existing-tx-1' }, error: null })
|
||||
// Batch external_id dedup query — returns matching external_id
|
||||
enqueue({ data: [{ external_id: raw.external_id }], error: null })
|
||||
|
||||
const result = await ingestTransactions(supabase as never, USER_ID, [raw])
|
||||
|
||||
@@ -181,8 +187,8 @@ describe('ingestTransactions', () => {
|
||||
enqueue({ data: [], error: null })
|
||||
// Supplier invoices fetch
|
||||
enqueue({ data: [], error: null })
|
||||
// Dedup check: no duplicate
|
||||
enqueue({ data: null, error: null })
|
||||
// Batch external_id dedup query (no matches)
|
||||
enqueue({ data: [], error: null })
|
||||
// Insert fails
|
||||
enqueue({ data: null, error: { message: 'DB constraint violation' } })
|
||||
|
||||
@@ -209,8 +215,8 @@ describe('ingestTransactions', () => {
|
||||
enqueue({ data: [], error: null })
|
||||
// Supplier invoices fetch
|
||||
enqueue({ data: [], error: null })
|
||||
// Dedup: no duplicate
|
||||
enqueue({ data: null, error: null })
|
||||
// Batch external_id dedup query (no matches)
|
||||
enqueue({ data: [], error: null })
|
||||
// Insert returns the new transaction
|
||||
enqueue({ data: inserted, error: null })
|
||||
// Invoice match update (supabase.from('transactions').update(...))
|
||||
@@ -252,8 +258,8 @@ describe('ingestTransactions', () => {
|
||||
enqueue({ data: [], error: null })
|
||||
// Supplier invoices fetch
|
||||
enqueue({ data: [], error: null })
|
||||
// Dedup: no duplicate
|
||||
enqueue({ data: null, error: null })
|
||||
// Batch external_id dedup query (no matches)
|
||||
enqueue({ data: [], error: null })
|
||||
// Insert
|
||||
enqueue({ data: inserted, error: null })
|
||||
|
||||
@@ -283,8 +289,8 @@ describe('ingestTransactions', () => {
|
||||
enqueue({ data: [], error: null })
|
||||
// Supplier invoices fetch
|
||||
enqueue({ data: [], error: null })
|
||||
// Dedup: no duplicate
|
||||
enqueue({ data: null, error: null })
|
||||
// Batch external_id dedup query (no matches)
|
||||
enqueue({ data: [], error: null })
|
||||
// Insert
|
||||
enqueue({ data: inserted, error: null })
|
||||
// Update after journal entry creation
|
||||
@@ -322,8 +328,8 @@ describe('ingestTransactions', () => {
|
||||
enqueue({ data: [], error: null })
|
||||
// Supplier invoices fetch
|
||||
enqueue({ data: [], error: null })
|
||||
// Dedup: no duplicate
|
||||
enqueue({ data: null, error: null })
|
||||
// Batch external_id dedup query (no matches)
|
||||
enqueue({ data: [], error: null })
|
||||
// Insert
|
||||
enqueue({ data: inserted, error: null })
|
||||
|
||||
@@ -353,8 +359,8 @@ describe('ingestTransactions', () => {
|
||||
enqueue({ data: [], error: null })
|
||||
// Supplier invoices fetch
|
||||
enqueue({ data: [], error: null })
|
||||
// Dedup: no duplicate
|
||||
enqueue({ data: null, error: null })
|
||||
// Batch external_id dedup query (no matches)
|
||||
enqueue({ data: [], error: null })
|
||||
// Insert
|
||||
enqueue({ data: inserted, error: null })
|
||||
|
||||
@@ -383,11 +389,11 @@ describe('ingestTransactions', () => {
|
||||
enqueue({ data: [], error: null })
|
||||
// Supplier invoices fetch
|
||||
enqueue({ data: [], error: null })
|
||||
// Transaction 1: dedup (no match), insert OK
|
||||
enqueue({ data: null, error: null })
|
||||
// Batch external_id dedup query (no matches)
|
||||
enqueue({ data: [], error: null })
|
||||
// Transaction 1: insert OK
|
||||
enqueue({ data: inserted1, error: null })
|
||||
// Transaction 2: dedup (no match), insert OK
|
||||
enqueue({ data: null, error: null })
|
||||
// Transaction 2: insert OK
|
||||
enqueue({ data: inserted2, error: null })
|
||||
|
||||
mockEvaluateMappingRules.mockResolvedValue(makeMappingResult({ confidence: 0.5 }))
|
||||
@@ -422,19 +428,16 @@ describe('ingestTransactions', () => {
|
||||
enqueue({ data: [], error: null })
|
||||
// Supplier invoices fetch
|
||||
enqueue({ data: [], error: null })
|
||||
// Transaction rawNew: dedup (no match), insert OK
|
||||
enqueue({ data: null, error: null })
|
||||
// Batch external_id dedup query — ext-dup already exists
|
||||
enqueue({ data: [{ external_id: 'ext-dup' }], error: null })
|
||||
// Transaction rawNew: insert OK
|
||||
enqueue({ data: insertedNew, error: null })
|
||||
// Invoice match update for income transaction
|
||||
enqueue({ data: null, error: null })
|
||||
// Auto-categorization update
|
||||
enqueue({ data: null, error: null })
|
||||
|
||||
// Transaction rawDup: dedup returns existing record
|
||||
enqueue({ data: { id: 'existing-dup' }, error: null })
|
||||
|
||||
// Transaction rawErr: dedup (no match), insert fails
|
||||
enqueue({ data: null, error: null })
|
||||
// rawDup: skipped (in Set) — no queue entry needed
|
||||
// Transaction rawErr: insert fails
|
||||
enqueue({ data: null, error: { message: 'Insert failed' } })
|
||||
|
||||
// Income transaction gets an invoice match
|
||||
@@ -496,7 +499,8 @@ describe('ingestTransactions', () => {
|
||||
enqueue({ data: [], error: null })
|
||||
// Supplier invoices fetch
|
||||
enqueue({ data: [], error: null })
|
||||
enqueue({ data: null, error: null })
|
||||
// Batch external_id dedup query (no matches)
|
||||
enqueue({ data: [], error: null })
|
||||
enqueue({ data: inserted, error: null })
|
||||
|
||||
mockGetBestInvoiceMatch.mockRejectedValue(new Error('Network error'))
|
||||
@@ -522,7 +526,8 @@ describe('ingestTransactions', () => {
|
||||
enqueue({ data: [], error: null })
|
||||
// Supplier invoices fetch
|
||||
enqueue({ data: [], error: null })
|
||||
enqueue({ data: null, error: null })
|
||||
// Batch external_id dedup query (no matches)
|
||||
enqueue({ data: [], error: null })
|
||||
enqueue({ data: inserted, error: null })
|
||||
|
||||
mockEvaluateMappingRules.mockRejectedValue(new Error('Mapping error'))
|
||||
@@ -574,8 +579,8 @@ describe('ingestTransactions', () => {
|
||||
enqueue({ data: [], error: null })
|
||||
// Supplier invoices fetch
|
||||
enqueue({ data: [], error: null })
|
||||
// Dedup: no duplicate
|
||||
enqueue({ data: null, error: null })
|
||||
// Batch external_id dedup query (no matches)
|
||||
enqueue({ data: [], error: null })
|
||||
// Insert
|
||||
enqueue({ data: inserted, error: null })
|
||||
// Reconciliation update
|
||||
@@ -618,8 +623,8 @@ describe('ingestTransactions', () => {
|
||||
enqueue({ data: [], error: null })
|
||||
// Supplier invoices fetch
|
||||
enqueue({ data: [], error: null })
|
||||
// Dedup: no duplicate
|
||||
enqueue({ data: null, error: null })
|
||||
// Batch external_id dedup query (no matches)
|
||||
enqueue({ data: [], error: null })
|
||||
// Insert
|
||||
enqueue({ data: inserted, error: null })
|
||||
|
||||
@@ -647,8 +652,8 @@ describe('ingestTransactions', () => {
|
||||
enqueue({ data: [], error: null })
|
||||
// Supplier invoices fetch
|
||||
enqueue({ data: [], error: null })
|
||||
// Dedup: no duplicate
|
||||
enqueue({ data: null, error: null })
|
||||
// Batch external_id dedup query (no matches)
|
||||
enqueue({ data: [], error: null })
|
||||
// Insert
|
||||
enqueue({ data: inserted, error: null })
|
||||
|
||||
@@ -679,8 +684,8 @@ describe('ingestTransactions', () => {
|
||||
})
|
||||
// Supplier invoices fetch
|
||||
enqueue({ data: [], error: null })
|
||||
// external_id dedup: no match (different source)
|
||||
enqueue({ data: null, error: null })
|
||||
// Batch external_id dedup query (no match by external_id)
|
||||
enqueue({ data: [], error: null })
|
||||
|
||||
const result = await ingestTransactions(supabase as never, USER_ID, [raw])
|
||||
|
||||
@@ -704,8 +709,8 @@ describe('ingestTransactions', () => {
|
||||
})
|
||||
// Supplier invoices fetch
|
||||
enqueue({ data: [], error: null })
|
||||
// external_id dedup: no match
|
||||
enqueue({ data: null, error: null })
|
||||
// Batch external_id dedup query (no match)
|
||||
enqueue({ data: [], error: null })
|
||||
// Insert
|
||||
enqueue({ data: inserted, error: null })
|
||||
|
||||
@@ -738,13 +743,12 @@ describe('ingestTransactions', () => {
|
||||
})
|
||||
// Supplier invoices fetch
|
||||
enqueue({ data: [], error: null })
|
||||
// Batch external_id dedup query (no matches for any)
|
||||
enqueue({ data: [], error: null })
|
||||
|
||||
// raw1: external_id dedup (no match) -> content dedup matches (bookedCount=2 -> 1)
|
||||
enqueue({ data: null, error: null })
|
||||
// raw2: external_id dedup (no match) -> content dedup matches (bookedCount=1 -> 0)
|
||||
enqueue({ data: null, error: null })
|
||||
// raw3: external_id dedup (no match) -> content dedup exhausted -> insert
|
||||
enqueue({ data: null, error: null })
|
||||
// raw1: not in external_id set → content dedup matches (bookedCount=2 -> 1)
|
||||
// raw2: not in external_id set → content dedup matches (bookedCount=1 -> 0)
|
||||
// raw3: not in external_id set → content dedup exhausted → insert
|
||||
enqueue({ data: inserted, error: null })
|
||||
|
||||
mockEvaluateMappingRules.mockResolvedValue(makeMappingResult({ confidence: 0.5 }))
|
||||
@@ -764,8 +768,8 @@ describe('ingestTransactions', () => {
|
||||
enqueue({ error: { message: 'Query failed' } })
|
||||
// Supplier invoices fetch
|
||||
enqueue({ data: [], error: null })
|
||||
// external_id dedup: no match
|
||||
enqueue({ data: null, error: null })
|
||||
// Batch external_id dedup query (no matches)
|
||||
enqueue({ data: [], error: null })
|
||||
// Insert
|
||||
enqueue({ data: inserted, error: null })
|
||||
|
||||
|
||||
@@ -124,16 +124,22 @@ export async function ingestTransactions(
|
||||
// Non-critical — amount_sek fields will stay null
|
||||
}
|
||||
|
||||
for (const raw of rawTransactions) {
|
||||
// 1. Check for duplicates via external_id
|
||||
const { data: existing } = await supabase
|
||||
// Pre-fetch existing external_ids in batches for dedup (avoids N+1 queries)
|
||||
const existingExternalIds = new Set<string>()
|
||||
const externalIds = rawTransactions.map(t => t.external_id)
|
||||
for (let i = 0; i < externalIds.length; i += 500) {
|
||||
const chunk = externalIds.slice(i, i + 500)
|
||||
const { data } = await supabase
|
||||
.from('transactions')
|
||||
.select('id')
|
||||
.select('external_id')
|
||||
.eq('user_id', userId)
|
||||
.eq('external_id', raw.external_id)
|
||||
.single()
|
||||
.in('external_id', chunk)
|
||||
data?.forEach(r => existingExternalIds.add(r.external_id))
|
||||
}
|
||||
|
||||
if (existing) {
|
||||
for (const raw of rawTransactions) {
|
||||
// 1. Check for duplicates via external_id (batch pre-fetched)
|
||||
if (existingExternalIds.has(raw.external_id)) {
|
||||
result.duplicates++
|
||||
continue
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user