diff --git a/DECISIONS.md b/DECISIONS.md index fde84df4..449b8fa8 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -876,3 +876,4 @@ One line per decision: `[YYYY-MM-DD] : `. Appended by agents and [2026-08-11] Anthropic, Vercel and Supabase removed from the portal directory: all three email their invoices to European customers, so listing them told the user to go and log in for a document already in their inbox. The directory's bar is "does not send the invoice", not "also has a portal", and the poll it was seeded from asked which portals people log into, which people answered with where an invoice can ALSO be found. The same objection may reach further down the list; an entry is a claim that the invoice cannot be had any other way and is worth checking per vendor. [2026-08-11] Portal URLs are swept by scripts/check-portal-urls.mts rather than trusted: the directory shipped with 18 hand-written paths, none opened, the file said so and shipped anyway, and a founder then hit a 404 on Google Workspace (/ac/billing/history). A sweep found GitHub's /settings/billing 404 too. Rule now is the shallowest URL that certainly resolves: landing one click short of the invoice costs little, landing on an error page spends the trust the feature runs on. Google, OpenAI and Hetzner refuse automated requests, so they cannot be swept and are kept shallow deliberately; only a genuine 404 fails the script, since failing on an unreachable host would train people to ignore it. Trygg Hansa removed: neither candidate URL could be reached at all. [2026-08-11] Credit-note deduction fields (deduction_total, per-item deduction_amount) stay POSITIVE magnitudes, unlike every other amount on a credit note: both columns carry CHECK (>= 0) in the DB, and negating them made every ROT/RUT credit fail at insert (prod support case 2026-08-11). Verified inert: the reversing verifikat recomputes the ROT/RUT split from quantity/unit_price (generateRotRutLines), the PDF hides the deduction section for credit notes, getAmountToPay skips deductions when credited_invoice_id is set, and ROT payout candidates require status='paid', which invoices_credit_note_not_paid makes impossible for credit notes. Any future reader summing these fields across invoice + credit note must special-case credit notes. +[2026-08-12] Content dedupe on document ingest is an opt-in uploadDocument flag wired into the intake funnel (uploadAndExtract + mail-hunt ingest), NOT a unique index on (company_id, sha256_hash): archival callers (sent invoices, filings, bank exports) legitimately store repeating bytes and a blanket constraint would break them; the SELECT-then-insert race is accepted exactly as in the WhatsApp precedent. On a hit the funnel adopts the existing inbox item (callers always get a real inbox_item_id) or files an item against the existing document; the mail hunt skips outright since a second item would only duplicate work in Underlag. diff --git a/extensions/general/invoice-inbox/lib/upload-and-extract.ts b/extensions/general/invoice-inbox/lib/upload-and-extract.ts index 1f200e56..e23fc104 100644 --- a/extensions/general/invoice-inbox/lib/upload-and-extract.ts +++ b/extensions/general/invoice-inbox/lib/upload-and-extract.ts @@ -209,8 +209,75 @@ export async function uploadAndExtract( type: file.type, }, { upload_source: source === 'email' ? 'email' : source === 'whatsapp' ? 'whatsapp' : 'file_upload', + // Every inbox channel dedupes on content: the same receipt forwarded to + // two inboxes, re-hunted by a sweep, or uploaded twice must not become a + // second archived document. + dedupeByContent: true, }) + if (doc.deduplicated) { + // The company already archived this exact content. If an inbox item + // exists for it, adopt that item so the caller always gets a real + // inbox_item_id; only when the document entered outside the inbox do we + // fall through and file an item for the EXISTING document (no copy). + const { data: existingItem, error: itemLookupError } = await supabase + .from('invoice_inbox_items') + .select('id, status, extracted_data, matched_supplier_id, matched_transaction_id') + .eq('company_id', companyId) + .eq('document_id', doc.id) + .order('created_at', { ascending: true }) + .limit(1) + if (itemLookupError) { + // Fail closed: falling through would file a second item for a document + // that may already carry one. The delivering channel retries. + throw new Error(`Duplicate-item lookup failed: ${itemLookupError.message}`) + } + const adopted = (existingItem as Array<{ + id: string + status: string + extracted_data: unknown + matched_supplier_id: string | null + matched_transaction_id: string | null + }> | null)?.[0] + if (adopted) { + try { + await appendProcessingHistory({ + companyId, + correlationId, + aggregateType: 'Document', + aggregateId: doc.id, + eventType: 'DocumentDuplicateSkipped', + payload: { + channel: source, + document_id: doc.id, + inbox_item_id: adopted.id, + reason: 'duplicate_content', + }, + actor: opts.actorId + ? { type: 'system', id: opts.actorId } + : source === 'email' + ? { type: 'system', id: 'resend-inbound' } + : { type: 'user', id: userId }, + occurredAt: new Date(), + }) + } catch (err) { + console.error('[invoice-inbox] Failed to append DocumentDuplicateSkipped:', err) + } + return { + document_id: doc.id, + inbox_item_id: adopted.id, + status: adopted.status, + extracted_data: adopted.extracted_data, + matched_supplier_id: adopted.matched_supplier_id, + matched_transaction_id: adopted.matched_transaction_id, + extraction_skipped: true, + skip_reason: 'duplicate_content' as const, + page_count: null, + duplicate: true as const, + } + } + } + try { await appendProcessingHistory({ companyId, diff --git a/lib/core/documents/__tests__/document-service.test.ts b/lib/core/documents/__tests__/document-service.test.ts index fae68099..f0726a06 100644 --- a/lib/core/documents/__tests__/document-service.test.ts +++ b/lib/core/documents/__tests__/document-service.test.ts @@ -307,6 +307,72 @@ describe('uploadDocument', () => { ) }) + it('returns the existing document instead of storing a copy when dedupeByContent hits', async () => { + const existing = makeDocumentAttachment({ id: 'doc-orig', sha256_hash: 'same' }) + results = [ + { data: [existing], error: null }, // dedupe lookup + ] + + const handler = vi.fn() + eventBus.on('document.uploaded', handler) + + const upload = vi.fn().mockResolvedValue({ data: {}, error: null }) + const supabase = makeClient({ upload }) + const result = await uploadDocument(supabase as never, 'user-1', 'company-1', { + name: 'kvitto.pdf', + buffer: pdfBuffer(), + type: 'application/pdf', + }, { dedupeByContent: true }) + + expect(result.id).toBe('doc-orig') + expect(result.deduplicated).toBe(true) + // Nothing reaches storage and no document.uploaded fires: the archive + // already holds this content, so re-extraction must not run either. + expect(upload).not.toHaveBeenCalled() + expect(handler).not.toHaveBeenCalled() + }) + + it('rejects when the dedupe lookup itself fails, touching nothing', async () => { + // Fail closed: treating a broken lookup as "no match" would archive the + // duplicate the flag exists to prevent, silently, on transient DB errors. + results = [{ data: null, error: { message: 'connection reset' } }] + + const handler = vi.fn() + eventBus.on('document.uploaded', handler) + + const upload = vi.fn().mockResolvedValue({ data: {}, error: null }) + const supabase = makeClient({ upload }) + + await expect( + uploadDocument(supabase as never, 'user-1', 'company-1', { + name: 'kvitto.pdf', + buffer: pdfBuffer(), + type: 'application/pdf', + }, { dedupeByContent: true }), + ).rejects.toThrow(/dedupe lookup failed/i) + expect(upload).not.toHaveBeenCalled() + expect(handler).not.toHaveBeenCalled() + }) + + it('stores normally when dedupeByContent finds no match', async () => { + results = [ + { data: [], error: null }, // dedupe lookup: miss + { data: makeDocumentAttachment({ id: 'doc-new' }), error: null }, // insert + ] + + const upload = vi.fn().mockResolvedValue({ data: {}, error: null }) + const supabase = makeClient({ upload }) + const result = await uploadDocument(supabase as never, 'user-1', 'company-1', { + name: 'kvitto.pdf', + buffer: pdfBuffer(), + type: 'application/pdf', + }, { dedupeByContent: true }) + + expect(result.id).toBe('doc-new') + expect(result.deduplicated).toBeUndefined() + expect(upload).toHaveBeenCalledOnce() + }) + it('writes to the company-scoped key, not the legacy uploader-scoped key', async () => { results = [{ data: makeDocumentAttachment({ id: 'doc-1' }), error: null }] diff --git a/lib/core/documents/document-service.ts b/lib/core/documents/document-service.ts index 45591e85..ca6c8f7d 100644 --- a/lib/core/documents/document-service.ts +++ b/lib/core/documents/document-service.ts @@ -597,8 +597,18 @@ export async function uploadDocument( upload_source?: DocumentUploadSource journal_entry_id?: string journal_entry_line_id?: string + /** + * Content dedupe for intake channels: before storing, look for a + * current-version document in the same company with the same SHA-256 and + * return it (marked `deduplicated`) instead of archiving a copy. Opt-in, + * because archival callers (sent invoices, filings, bank exports) must + * store what they produced even when the bytes repeat. SELECT-then-insert + * leaves a small concurrent-upload race, accepted exactly as in the + * WhatsApp intake precedent: the loser stores a copy, nothing corrupts. + */ + dedupeByContent?: boolean } = {} -): Promise { +): Promise { await ensureDocumentsBucket() // Reject corrupt uploads at the boundary: see validateDocumentMagicBytes. @@ -610,6 +620,28 @@ export async function uploadDocument( // Compute SHA-256 hash const sha256Hash = await computeSHA256(file.buffer) + if (metadata.dedupeByContent) { + // Oldest current-version match wins so repeated deliveries keep + // converging on the same archived original. Pre-dedupe data can hold + // several identical documents, hence limit(1) rather than maybeSingle. + const { data: existing, error: dedupeError } = await supabase + .from('document_attachments') + .select('*') + .eq('company_id', companyId) + .eq('sha256_hash', sha256Hash) + .eq('is_current_version', true) + .order('created_at', { ascending: true }) + .limit(1) + if (dedupeError) { + // Fail closed: treating a broken lookup as "no match" would archive + // the duplicate this flag exists to prevent, silently, on every + // transient DB error. Intake callers (webhooks, sweeps) retry. + throw new Error(`Content dedupe lookup failed: ${dedupeError.message}`) + } + const hit = (existing as DocumentAttachment[] | null)?.[0] + if (hit) return { ...hit, deduplicated: true } + } + // Company-scoped storage key: the tenant id must be IN the key so the // storage RLS policy can revoke access when a membership is removed. const storagePath = buildDocumentStoragePath(companyId, userId, file.name) diff --git a/lib/receipt-hunt/__tests__/ingest.test.ts b/lib/receipt-hunt/__tests__/ingest.test.ts index 1baf9670..e999ea7c 100644 --- a/lib/receipt-hunt/__tests__/ingest.test.ts +++ b/lib/receipt-hunt/__tests__/ingest.test.ts @@ -11,6 +11,11 @@ vi.mock('@/lib/core/documents/document-service', () => ({ uploadDocument: (...args: unknown[]) => mockUploadDocument(...args), })) +const mockAppendHistory = vi.fn() +vi.mock('@/lib/processing-history/append', () => ({ + appendProcessingHistory: (...args: unknown[]) => mockAppendHistory(...args), +})) + const mockFetchAttachment = vi.fn() vi.mock('@/lib/mail-search/service', () => ({ getMailSearchService: () => ({ @@ -38,7 +43,12 @@ function candidate(overrides: Partial = {}): MailCandidate { } /** Table-dispatching Supabase stand-in with a settable existing-row answer. */ -function mockSupabase(existing: { id: string } | null, insertResult: { data?: unknown; error?: unknown } = {}) { +function mockSupabase( + existing: { id: string } | null, + insertResult: { data?: unknown; error?: unknown } = {}, + laterInboxAnswers: Array<{ id: string } | null> = [], +) { + const inboxQueue: Array<{ id: string } | null> = [existing, ...laterInboxAnswers] const inserted: Array> = [] const client = { from(table: string) { @@ -48,7 +58,7 @@ function mockSupabase(existing: { id: string } | null, insertResult: { data?: un Promise.resolve( table === 'document_attachments' ? { data: { extracted_data: { total_amount: 425 } }, error: null } - : { data: existing, error: null }, + : { data: inboxQueue.length ? inboxQueue.shift() ?? null : null, error: null }, ), ) chain.insert = vi.fn((row: Record) => { @@ -119,6 +129,66 @@ describe('ingestMailCandidate', () => { await expect(ingestMailCandidate(client, 'co-1', 'user-1', candidate())).resolves.toBeNull() }) + it('skips filing when the archived duplicate already has an inbox item', async () => { + // The provenance key catches a re-hunted message; this catches the same + // receipt arriving through ANOTHER inbox: the receipt is already in the + // Underlag flow, so no second item is filed. + mockUploadDocument.mockResolvedValue({ id: 'doc-orig', deduplicated: true }) + const { client, inserted } = mockSupabase(null, {}, [{ id: 'item-existing' }]) + const result = await ingestMailCandidate(client, 'co-1', 'user-1', candidate()) + expect(result).toBeNull() + expect(inserted).toHaveLength(0) + // Locks the flag itself: without dedupeByContent the mock still answers + // deduplicated, but production would silently archive copies again. + expect(mockUploadDocument).toHaveBeenCalledWith( + expect.anything(), + 'user-1', + 'co-1', + expect.anything(), + { upload_source: 'mail_hunt', dedupeByContent: true }, + ) + // The skip is behandlingshistorik, not just an app log; and the payload + // stays pseudonymous (never a mailbox address). + expect(mockAppendHistory).toHaveBeenCalledTimes(1) + const event = mockAppendHistory.mock.calls[0]![0] as { + eventType: string + aggregateId: string + payload: Record + } + expect(event.eventType).toBe('DocumentDuplicateSkipped') + expect(event.aggregateId).toBe('doc-orig') + expect(event.payload).toMatchObject({ + channel: 'mail_hunt', + document_id: 'doc-orig', + inbox_item_id: 'item-existing', + reason: 'duplicate_content', + }) + expect(JSON.stringify(event.payload)).not.toContain('@') + }) + + it('still skips the duplicate when the history append fails', async () => { + // The audit write is best-effort by design: a history outage must not + // turn a correct skip into a duplicate filing. + mockUploadDocument.mockResolvedValue({ id: 'doc-orig', deduplicated: true }) + mockAppendHistory.mockRejectedValueOnce(new Error('history down')) + const { client, inserted } = mockSupabase(null, {}, [{ id: 'item-existing' }]) + const result = await ingestMailCandidate(client, 'co-1', 'user-1', candidate()) + expect(result).toBeNull() + expect(inserted).toHaveLength(0) + }) + + it('files an item against the existing document when the duplicate never passed the inbox', async () => { + // A content match against a document with no inbox item (a manually + // attached copy) must not swallow the receipt: the affärshändelse still + // needs routing to matching (BFL 5 kap), just without a second archive copy. + mockUploadDocument.mockResolvedValue({ id: 'doc-orig', deduplicated: true }) + const { client, inserted } = mockSupabase(null, {}, [null]) + const result = await ingestMailCandidate(client, 'co-1', 'user-1', candidate()) + expect(result).toMatchObject({ documentId: 'doc-orig', inboxItemId: 'item-1' }) + expect(inserted).toHaveLength(1) + expect(inserted[0].document_id).toBe('doc-orig') + }) + it('ignores a body-only receipt, which has nothing to download', async () => { const { client } = mockSupabase(null) const result = await ingestMailCandidate( diff --git a/lib/receipt-hunt/ingest.ts b/lib/receipt-hunt/ingest.ts index 92197998..1b0b7b5f 100644 --- a/lib/receipt-hunt/ingest.ts +++ b/lib/receipt-hunt/ingest.ts @@ -11,6 +11,7 @@ */ import type { SupabaseClient } from '@supabase/supabase-js' import { uploadDocument } from '@/lib/core/documents/document-service' +import { appendProcessingHistory } from '@/lib/processing-history/append' import { getMailSearchService, type MailCandidate } from '@/lib/mail-search/service' import { createLogger } from '@/lib/logger' @@ -164,9 +165,65 @@ export async function ingestMailCandidate( ) as ArrayBuffer, type: sniffMimeType(fetched.bytes, fetched.mimeType, fileName), }, - { upload_source: 'mail_hunt' }, + { upload_source: 'mail_hunt', dedupeByContent: true }, ) + // Content already archived for this company: the provenance key above + // only catches the SAME message re-hunted, while this catches the same + // receipt arriving through another inbox or channel (forwards are + // common). Skip ONLY when an inbox item already carries the document: + // then the receipt is in the Underlag flow (or handled). When the match + // is a document that never passed the inbox (a manually attached copy, + // an archival file), fall through and file an item for the EXISTING + // document: silently dropping the receipt could leave a real + // affärshändelse without underlag routing (BFL 5 kap). + if (document.deduplicated) { + const { data: dupItem, error: dupErr } = await supabase + .from('invoice_inbox_items') + .select('id') + .eq('company_id', companyId) + .eq('document_id', document.id) + .limit(1) + .maybeSingle() + // Fail closed: a broken lookup must not file a second item. + if (dupErr) throw new Error(dupErr.message) + if (dupItem) { + // Behandlingshistorik, not just an app log: the dedupe decision is + // part of the auditable trail (BFNAR 2013:2 kap 8). + try { + await appendProcessingHistory({ + companyId, + correlationId: crypto.randomUUID(), + aggregateType: 'Document', + aggregateId: document.id, + eventType: 'DocumentDuplicateSkipped', + // No mailbox address here: the processing-history payload + // contract is pseudonymous IDs only (never emails). Which + // mailbox first delivered the receipt is on the existing + // item's channel_context. + payload: { + channel: 'mail_hunt', + document_id: document.id, + inbox_item_id: (dupItem as { id: string }).id, + mail_message_id: candidate.messageId, + reason: 'duplicate_content', + }, + actor: { type: 'system', id: 'receipt-hunt' }, + occurredAt: new Date(), + }) + } catch (histErr) { + log.warn('could not append DocumentDuplicateSkipped', { + error: histErr instanceof Error ? histErr.message : String(histErr), + }) + } + log.info('skipped duplicate attachment content', { + messageId: candidate.messageId, + documentId: document.id, + }) + continue + } + } + // uploadDocument emits document.uploaded and awaits its handlers, so the // extraction extension has already read the amount, date and vendor out // of this file by the time we get here. Copying it onto the inbox item is