From a4ceaafa4f39ef8a78f0927aeb98ff698fd2df65 Mon Sep 17 00:00:00 2001 From: Jakob Wennberg Date: Fri, 28 Aug 2026 17:45:10 +0200 Subject: [PATCH] feat(inbox): per-item underlag anchoring status and a daily reconcile cron for stranded underlag (#1548) (#2012) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(invoice-inbox): per-item underlag status and daily reconcile of stranded booked items (#1548) The inbox derives "booked" from the matched transaction's verifikat, but that says nothing about whether THIS item's document reached it: a link that failed at propagation time, or a document anchored to another verifikat, read as booked while the verifikat sat without its underlag (BFL 5 kap 6-7 §). GET /items and /items/:id now also emit underlag_status (anchored | unlinked | anchored_elsewhere) from one batched document_attachments read; the workspace keeps divergent items in "Att göra", drops the booking bridge for them (the book routes 409 on a booked transaction) and shows one explanatory line with a link to the verifikat. The backfill script's loop moves into lib/transactions/ inbox-underlag-reconcile.ts and runs daily from a new extension-owned cron (vercel.json plus the generated Docker crontabs): transient link failures heal without an ad-hoc script run, permanent conflicts are counted in one summary, and each repaired transaction leaves an InboxUnderlagReconciled row in behandlingshistorik. That event type is registered by migration 20260828154800: processing_history.event_type has an FK to processing_event_types, and the script's previous InboxUnderlagBackfilled type was never registered, so its appends had always failed silently. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_015nAd8XJ2RPCmG2eKoLBdna * fix(invoice-inbox): address review findings on the underlag reconcile (#1548) Findings 1, 3, 6 (scan cap starves the tail): the reconcile no longer caps the read. The matched-unconsumed candidate set holds permanent residents (samlingsverifikat siblings, anchored-elsewhere items) that never leave it, so a uuid-ordered read cap would revisit the same 1000 rows every night and never reach a stranded item sorting past the cut. The scan now pages through every candidate (four columns per row) and maxItems bounds the WORK: at most that many unlinked (or unreadable) items are propagated per run; already-anchored, anchored-elsewhere and locked items are counted from the pre-state without a propagation or budget. Items past the budget are counted as deferred and truncated is logged at warn level. Findings 2, 5 (false "linked automatically" promise for locked periods): resolveUnderlagAnchoring reads the fiscal period lock state of the verifikat for every unlinked item and reports unlinked_locked when is_closed or locked_at is set, the same pair enforce_period_lock_documents checks. The reconciler counts it separately (unlinkedLocked), never propagates it and never warns "still unlinked after re-run"; the rail shows a message that says the period must be unlocked first. Findings 4, 7 (absent anchoring read as booked): the list and detail enrichment emit underlag_status 'unknown' when the helper could not read the document row, and the workspace treats any status but 'anchored' as divergent (stays in Att göra, no booking bridge, own message). classify() counts a repair only when the pre-state was explicitly unlinked, so an unreadable before-read never earns an InboxUnderlagReconciled event. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_015nAd8XJ2RPCmG2eKoLBdna * fix(invoice-inbox): address round-2 review findings (#1548) 1. [minor] Round-1 fix dropped propagation for transactions whose inbox items already read anchored, so the pinned-document leg (transactions.document_id) was never repaired and settled items never received their created_journal_entry_id stamp, staying in the scan and inflating alreadyAnchored every night. reconcileCompany now propagates every stranded transaction that has an unlinked (budgeted) item or an anchored / document-less item, outside the maxItems budget: the helper is idempotent and the stamp shrinks its own population. Locked-only and anchored-elsewhere-only transactions stay skipped. Counting and the behandlingshistorik trail are unchanged (anchored items keep their pre-state verdict, no event). Tests updated and a new case pins the anchored-item plus document-less-item transaction: propagated, no after-read, no history. DECISIONS line amended. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_015nAd8XJ2RPCmG2eKoLBdna --------- Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com> Co-authored-by: Claude Fable 5 --- DECISIONS.md | 2 + .../cron/__tests__/route.test.ts | 105 +++++ .../underlag-reconcile/cron/route.ts | 55 +++ .../general/InvoiceInboxWorkspace.tsx | 92 +++- docker/crontab.hosted | 1 + docker/crontab.self-hosted | 1 + .../__tests__/items-enrichment.test.ts | 223 +++++++++ extensions/general/invoice-inbox/index.ts | 85 +++- .../inbox-underlag-reconcile.test.ts | 441 ++++++++++++++++++ .../__tests__/inbox-underlag.test.ts | 104 +++++ lib/transactions/inbox-underlag-reconcile.ts | 418 +++++++++++++++++ lib/transactions/inbox-underlag.ts | 130 ++++++ messages/en.json | 5 + messages/sv.json | 5 + scripts/backfill-inbox-booked-underlag.ts | 111 +---- ...r_inbox_underlag_reconciled_event_type.sql | 22 + ...-underlag-reconciled-event-type.pg.test.ts | 39 ++ vercel.json | 4 + 18 files changed, 1727 insertions(+), 116 deletions(-) create mode 100644 app/api/extensions/invoice-inbox/underlag-reconcile/cron/__tests__/route.test.ts create mode 100644 app/api/extensions/invoice-inbox/underlag-reconcile/cron/route.ts create mode 100644 extensions/general/invoice-inbox/__tests__/items-enrichment.test.ts create mode 100644 lib/transactions/__tests__/inbox-underlag-reconcile.test.ts create mode 100644 lib/transactions/inbox-underlag-reconcile.ts create mode 100644 supabase/migrations/20260828154800_register_inbox_underlag_reconciled_event_type.sql create mode 100644 tests/pg/inbox-underlag-reconciled-event-type.pg.test.ts diff --git a/DECISIONS.md b/DECISIONS.md index dc999844..a74157e7 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -1323,6 +1323,8 @@ One line per decision: `[YYYY-MM-DD] : `. Appended by agents and [2026-08-28] AR-PDF minus fix uses ASCII hyphen formatting, not font embedding: registering a Unicode TTF for react-pdf would change the whole document's typography and bundle size to fix one glyph; formatPdfKronor keeps built-in Helvetica and sidesteps WinAnsi's missing U+2212. [2026-08-28] Same-bank warning limited to observed one-session banks (SEB only): prod shows Handelsbanken tolerates 4 concurrent sessions, and the generic warning made a user abandon a legitimate renewal. Planned sync-death visibility work was dropped: already shipped via #1271 (health probe), #1727 (stale state), #1969 (cron unstarve). [2026-08-28] Same-bank warning revised to three tiers after skeptic refutation: hard warn SEB, silent/calm only for verified multi-session banks (Handelsbanken, 4 distinct session_ids observed), legacy hedged warning for unknown banks (fail closed), shared-session siblings exempt (fan-out carries them). +[2026-08-28] Inbox underlag divergence (#1548) is a separate underlag_status field, not a nulled matched_transaction_journal_entry_id: the book-direct and bulk-book routes 409 on an already-booked transaction, so hiding the verifikat would make the rail re-offer a booking that always fails; the UI keeps divergent items in Att göra with an explanation and a link instead. Anchored-elsewhere conflicts are counted and logged by the daily reconcile cron, never auto-resolved (moving a document between verifikat is a human decision; never-steal is the 2026-08-13 invariant), and not escalated to processing_history from the inline booking path (no dedupe key; it would fire on every booking). The reconcile's 'InboxUnderlagReconciled' event needed a processing_event_types row (FK): the script's old 'InboxUnderlagBackfilled' type was never registered, so its appends had always failed silently. +[2026-08-28] Inbox underlag reconcile (#1548) bounds link work, not the read: the candidate set (matched, unconsumed) holds permanent residents (samlingsverifikat siblings, anchored-elsewhere items) that never leave it, so a read cap ordered by uuid would revisit the same window nightly and starve the tail. The full scan is four columns per row; maxItems now caps unlinked items linked per run, the rest are counted as deferred; transactions whose items already read anchored are still propagated outside that budget, because only the propagation anchors the transaction's pinned document and stamps settled items out of the scan (idempotent, self-shrinking). A verifikat in a locked/closed period is its own status (unlinked_locked): the period-lock trigger rejects the link every time, so it is neither retried nor promised to the user as automatic. An unreadable document row is reported as 'unknown' on the wire and kept out of the booked bucket, matching the helper's absence-is-never-anchored contract. [2026-08-28] data_analysis_opt_in (#1346) defaults false with no grandfathering and gates only bookkeeping-content analysis (categorize_calibration_samples via POST /api/agent/categorize/outcome, the calibration-fit script, and the backtest script, whose wider inputs (transaction descriptions, merchant names, matched underlag re-run through the model) are named in the consent copy as "evaluation runs" after review found the outcomes-only wording false for that path; the toggle is admin/owner only to match the company_settings RLS update policy), not PostHog product analytics or MCP reliability telemetry: those carry no bookkeeping content and stay under Art. 6.1f per ropa.yaml, so widening the flag is a founder call; consequence: the calibration corpus collects nothing until a company opts in, and the flag is UI-only (absent from v1 REST / MCP settings pick lists) because consent is a human action. [2026-08-28] Bulk revenue rate-0 slot (#1912): item 1 (require a positive momsfri/export/EU classification) stays deferred because most real momsfri/export accounts are unconfigured today and refusing them needs a configure path first (the issue's own trade-off); resolveRevenueVatBox makes it a one-line switch later. The order-context check refuses only the export/EU-account direction (ruta 36 vs SE/EU country, ruta 40 vs SE, ruta 35/38/39 vs SE/non-EU), because the domestic-account + foreign-country direction is already advisory in the dialog (zero_rate_foreign) and hard-blocking it would regress every untemplated sweep; ruta 40 is 'omsatta utom landet', so an EU country does not contradict it. EXPORT_BOXES from moms-box-mapping was not reused because it merges EU and export boxes. [2026-08-28] Verifikat page promotes "Stryk rader i verifikatet" to a visible outline button only when the period-status endpoint answers open (anything else or unknown keeps it in the ⋯ menu only), and the ⋯ item stays: inline rättelse is the normal path in an open period (#1554), the promoted button must never invite an action the RPC will refuse, and the menu remains the complete action list per the 2026-08-20 detail-page grammar. Struck-line actor labels are resolved server-side via resolveUserLabelsFromProfiles (profiles RLS is self-only) rather than a new column on the log. diff --git a/app/api/extensions/invoice-inbox/underlag-reconcile/cron/__tests__/route.test.ts b/app/api/extensions/invoice-inbox/underlag-reconcile/cron/__tests__/route.test.ts new file mode 100644 index 00000000..ceed7576 --- /dev/null +++ b/app/api/extensions/invoice-inbox/underlag-reconcile/cron/__tests__/route.test.ts @@ -0,0 +1,105 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest' +import { NextResponse } from 'next/server' + +vi.mock('@/lib/extensions/loader', () => ({ + loadExtensions: vi.fn(), +})) + +vi.mock('@/lib/extensions/registry', () => ({ + extensionRegistry: { + get: vi.fn(), + }, +})) + +vi.mock('@/lib/auth/api-keys', () => ({ + createServiceClientNoCookies: vi.fn().mockReturnValue({}), +})) + +vi.mock('@/lib/transactions/inbox-underlag-reconcile', () => ({ + reconcileStrandedInboxUnderlag: vi.fn(), +})) + +vi.mock('@/lib/auth/cron', () => ({ + verifyCronSecret: vi.fn().mockReturnValue(null), +})) + +import { GET } from '../route' +import { extensionRegistry } from '@/lib/extensions/registry' +import { loadExtensions } from '@/lib/extensions/loader' +import { reconcileStrandedInboxUnderlag } from '@/lib/transactions/inbox-underlag-reconcile' +import { verifyCronSecret } from '@/lib/auth/cron' + +const mockRegistryGet = vi.mocked(extensionRegistry.get) +const mockVerifyCronSecret = vi.mocked(verifyCronSecret) +const mockReconcile = vi.mocked(reconcileStrandedInboxUnderlag) + +function makeRequest() { + return new Request('http://localhost/api/extensions/invoice-inbox/underlag-reconcile/cron', { + headers: { authorization: 'Bearer synthetic-cron-secret' }, + }) +} + +const SUMMARY = { + execute: true, + scanned: 4, + truncated: false, + strandedOnBooked: 3, + repaired: 2, + alreadyAnchored: 0, + stillUnlinked: 0, + unlinkedLocked: 0, + deferred: 0, + anchoredElsewhere: 1, + companiesTouched: 1, + historyAppended: 2, + failures: 0, +} + +beforeEach(() => { + vi.clearAllMocks() + mockVerifyCronSecret.mockReturnValue(null) +}) + +describe('GET /api/extensions/invoice-inbox/underlag-reconcile/cron', () => { + it('returns 401 when the cron secret is rejected', async () => { + mockVerifyCronSecret.mockReturnValue( + NextResponse.json({ error: 'Unauthorized' }, { status: 401 }), + ) + + const response = await GET(makeRequest()) + + expect(response.status).toBe(401) + expect(mockReconcile).not.toHaveBeenCalled() + }) + + it('returns 503 EXTENSION_DISABLED when the extension is not in the registry', async () => { + // Physical extension routes deploy in every build; the registry, generated + // from extensions.config.json, is what turns them on. Disabled must mean + // no reconciling AND a visible failure if the cron is scheduled anyway. + mockRegistryGet.mockReturnValue(undefined) + + const response = await GET(makeRequest()) + const body = await response.json() + + expect(response.status).toBe(503) + expect(body.code).toBe('EXTENSION_DISABLED') + expect(mockReconcile).not.toHaveBeenCalled() + }) + + it('runs the reconciliation in execute mode and returns its summary when enabled', async () => { + mockRegistryGet.mockReturnValue({ id: 'invoice-inbox' } as never) + mockReconcile.mockResolvedValue(SUMMARY) + + const response = await GET(makeRequest()) + const body = await response.json() + + expect(loadExtensions).toHaveBeenCalled() + expect(mockRegistryGet).toHaveBeenCalledWith('invoice-inbox') + expect(mockReconcile).toHaveBeenCalledWith( + expect.anything(), + expect.objectContaining({ execute: true, actorId: 'cron.invoice_inbox_underlag_reconcile' }), + ) + expect(response.status).toBe(200) + expect(body.data).toEqual(SUMMARY) + }) +}) diff --git a/app/api/extensions/invoice-inbox/underlag-reconcile/cron/route.ts b/app/api/extensions/invoice-inbox/underlag-reconcile/cron/route.ts new file mode 100644 index 00000000..fa42356a --- /dev/null +++ b/app/api/extensions/invoice-inbox/underlag-reconcile/cron/route.ts @@ -0,0 +1,55 @@ +import { NextResponse } from 'next/server' +import { loadExtensions } from '@/lib/extensions/loader' +import { extensionRegistry } from '@/lib/extensions/registry' +import { withCronContext } from '@/lib/api/with-cron-context' +import { createServiceClientNoCookies } from '@/lib/auth/api-keys' +import { reconcileStrandedInboxUnderlag } from '@/lib/transactions/inbox-underlag-reconcile' + +/** + * GET /api/extensions/invoice-inbox/underlag-reconcile/cron: daily + * reconciliation of inbox items stranded on already-booked transactions + * (#1548). Re-runs the underlag propagation for matched items whose + * transaction is booked but whose stamp never landed, so a transient link + * failure heals without an ad-hoc script run, and counts the permanent + * conflicts (document anchored to another verifikat) so they are visible + * in one summary instead of scattered warn lines. Scheduled daily in + * vercel.json (and the generated Docker crontabs). + * + * Idempotent and safe to overlap with a slow previous run: the propagation + * skips documents that already reference the verifikat and CASes the stamp + * on its null predicate. + */ + +// One bounded scan (1000 items) plus a handful of batched lookups per +// company, and a document link per stranded item. Same budget as the +// WhatsApp sweep so a large backlog on first run cannot time out midway. +export const maxDuration = 300 + +export const GET = withCronContext('cron.invoice_inbox_underlag_reconcile', async (_request, ctx) => { + // Load the registry so it reflects extensions.config.json. + loadExtensions() + + // Physical routes under app/api/extensions// compile into EVERY build, + // including the core-with-zero-extensions one: the registry (generated from + // extensions.config.json) is what actually switches an extension on. A + // scheduled-but-disabled cron must fail visibly (503) instead of quietly + // doing the work anyway. + if (!extensionRegistry.get('invoice-inbox')) { + ctx.log.warn('invoice-inbox extension is not enabled; cron refused') + return NextResponse.json( + { error: 'Invoice inbox extension is not enabled', code: 'EXTENSION_DISABLED' }, + { status: 503 }, + ) + } + + const supabase = createServiceClientNoCookies() + const summary = await reconcileStrandedInboxUnderlag(supabase, { + execute: true, + log: ctx.log, + actorId: 'cron.invoice_inbox_underlag_reconcile', + }) + + ctx.log.info('invoice inbox underlag reconcile complete', { ...summary }) + + return NextResponse.json({ data: summary }) +}) diff --git a/components/extensions/general/InvoiceInboxWorkspace.tsx b/components/extensions/general/InvoiceInboxWorkspace.tsx index bb578f22..b53f57d4 100644 --- a/components/extensions/general/InvoiceInboxWorkspace.tsx +++ b/components/extensions/general/InvoiceInboxWorkspace.tsx @@ -142,6 +142,11 @@ function reportUploadFailure(report: { // ── Types ──────────────────────────────────────────────────── +// Mirrors the extension's UnderlagStatus: the anchoring verdict, or 'unknown' +// when the server could not read the document row. Anything but 'anchored' +// keeps the item out of the booked bucket. +type UnderlagStatus = 'anchored' | 'unlinked' | 'unlinked_locked' | 'anchored_elsewhere' | 'unknown' + interface InboxItem { id: string // 'processing' is the staged-upload in-flight state: the row exists (the @@ -170,6 +175,16 @@ interface InboxItem { // samlingsverifikat only one of N items can carry the stamp; this field is // what lets the rest read as booked. Absent on client-side placeholders. matched_transaction_journal_entry_id?: string | null + // Whether THIS item's underlag reached that verifikat (#1548). The + // transaction being booked is a fact about the transaction, not about the + // item's document: one whose link failed ('unlinked', transient: the daily + // reconcile retries it), whose verifikat sits in a locked period + // ('unlinked_locked', unlock first), that sits on another verifikat + // ('anchored_elsewhere', a human decision), or that could not be read + // ('unknown') keeps the item in "Att göra". null when nothing was derived + // (no booked matched transaction, or the item is stamped). Absent on + // client-side placeholders. + underlag_status?: UnderlagStatus | null error_message: string | null // True when AI extraction was skipped: either because the upload caller // passed skip_extraction=true (MCP/agent path) or because the server's @@ -298,9 +313,31 @@ function countExtractedFields(data: InvoiceExtractionResult | null): number { // else needs a first action. type InboxStatus = 'needs_action' | 'processing' | 'linked' | 'booked' | 'error' +// A matched transaction that is booked while this item's own underlag is not +// on its verifikat: not "booked" for the inbox, and not bookable either (the +// book routes 409 on an already-booked transaction). The rail explains it +// instead of offering a bridge that can only fail. +function isUnderlagDivergent(item: InboxItem): boolean { + return ( + !!item.matched_transaction_journal_entry_id && + !!item.underlag_status && + item.underlag_status !== 'anchored' + ) +} + +// One explanatory line per non-anchored status (#1548). 'unlinked' is the +// only one the daily reconcile can heal on its own; the others say what +// stands in the way instead of promising an automatic link. +const UNDERLAG_STATUS_MESSAGE_KEY: Record, string> = { + unlinked: 'underlag_unlinked', + unlinked_locked: 'underlag_unlinked_locked', + anchored_elsewhere: 'underlag_anchored_elsewhere', + unknown: 'underlag_unknown', +} + function deriveInboxStatus(item: InboxItem): InboxStatus { if (item.created_supplier_invoice_id || item.created_journal_entry_id) return 'booked' - if (item.matched_transaction_journal_entry_id) return 'booked' + if (item.matched_transaction_journal_entry_id && !isUnderlagDivergent(item)) return 'booked' // Staged upload mid-extraction. Outranks 'linked': a transaction-anchored // upload is matched from birth, but offering the booking bridge before the // fields exist would book from empty data. Transient (seconds): stays in @@ -2701,14 +2738,22 @@ function FieldsRail({ }, [item.id]) const isProcessed = !!item.created_supplier_invoice_id + const underlagDivergent = isUnderlagDivergent(item) // The verifikat this item resolved into: its own stamp, or the entry that - // anchors its matched (and already booked) transaction. See InboxItem. + // anchors its matched (and already booked) transaction, unless this item's + // own underlag never reached it. See InboxItem. const bookedEntryId = - item.created_journal_entry_id ?? item.matched_transaction_journal_entry_id ?? null + item.created_journal_entry_id ?? + (underlagDivergent ? null : item.matched_transaction_journal_entry_id) ?? + null const isBookedDirectly = !isProcessed && !!bookedEntryId // "Resolved" now means a journal entry exists: matched_transaction_id alone // is not resolved, it's the prerequisite for booking against that tx. const isLinkedToTransaction = !isProcessed && !isBookedDirectly && !!item.matched_transaction_id + // The booking bridge (proposal, book/ask actions) only while the + // transaction is unbooked: a divergent item's transaction already has a + // verifikat, so booking would 409. It gets an explanation and a link. + const showBookingBridge = isLinkedToTransaction && !underlagDivergent const isResolved = isProcessed || isBookedDirectly // Staged upload mid-extraction: a real row whose deferred AI extraction has // not landed yet. Same disabled treatment as the optimistic placeholder @@ -2991,7 +3036,7 @@ function FieldsRail({ scrolling past nine values to reach the one thing to approve. Suppressed while extraction is in flight: a proposal computed from empty fields would be an invitation to book nothing. */} - {isLinkedToTransaction && !inFlight && ( + {showBookingBridge && !inFlight && ( )} @@ -3089,10 +3134,29 @@ function FieldsRail({ ) : isLinkedToTransaction && item.matched_transaction_id ? ( <> + {/* The transaction is booked but this item's underlag is not on + its verifikat (#1548): say so, link the verifikat, and keep + "Avbryt matchning" as the way out. No booking bridge: the + book routes 409 on an already-booked transaction. */} + {underlagDivergent && item.matched_transaction_journal_entry_id && ( + + {t( + UNDERLAG_STATUS_MESSAGE_KEY[ + (item.underlag_status ?? 'unknown') as Exclude + ], + )} + + )} {/* Matched-to-tx state: show the bridge to booking. The user picks one of two actions: book themselves with the deterministic dialog, or hand off to the assistant. */} - {onAskAssistant && ( + {showBookingBridge && onAskAssistant && ( + {showBookingBridge && ( + + )}