fix(invoices): ROT/RUT credit notes; verifikat amount sort, HTML underlag, source chip (#1523)

* feat(invoice-inbox): store HTML mails as underlag, expandable field editor

Body-only mails and .html attachments (including forwarded .eml bodies) no
longer dead-end as "Fel vid bearbetning": the mail body is wrapped into a
self-contained text/html document, stored through the normal upload/extract
pipeline, and extracted via a new HTML-to-text Bedrock path, so the mail
itself can serve as bookable underlag. Empty mails keep the error row,
unsupported types are still rejected, and webhook retries dedupe on
resend_email_id.

Mail HTML is attacker-controlled, so rendering is fully sandboxed: iframe
sandbox in the workspace preview and a CSP sandbox header on
/api/documents/:id/inline for text/html. The type is accepted only from the
email pipeline (EMAIL_ALLOWED_MIME_TYPES), never from manual upload.

The "Extraherade falt" rail gains an expand button opening a centered
dialog with the same autosaving field editor at a readable size (two
columns), which also gives every failed or skipped extraction a manual
fallback.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(bookkeeping): sortable verifikat list headers with amount sort

- clickable sort toggles on the verifikat list headers (asc -> desc -> default)
- total_amount computed column + sort_by total/description on the list route
- failed list loads render an error card with retry, never the empty-ledger state

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(import): decode bank CSV as Windows-1252 fallback in column mapping

The client read the uploaded file with file.text(), which is UTF-8-only,
so Windows-1252 exports (e.g. Handelsbanken) rendered and re-parsed with
U+FFFD in place of Swedish characters. Decode from bytes with the shared
decodeFileContent() helper, matching what the server parse route does.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(bookkeeping): stackable sort keys on verifikat list headers

- shift-click adds a column as secondary/tertiary sort key (max 3), plain
  click keeps the single-key tri-state cycle
- sort_by accepts a comma-separated priority list; single tokens stay valid
- voucher tiebreak follows the last key's direction (#972 parity)
- priority numbers on stacked headers; hint text in the filter dialog

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(invoices): keep ROT/RUT deduction fields positive on credit notes

Crediting an invoice with a ROT/RUT deduction failed 100% of the time:
the credit-note path negated deduction_total (and per-item
deduction_amount) like the other amounts, but both columns carry
CHECK (>= 0), so Postgres rejected the insert and the user only saw
'Kunde inte skapa kreditfaktura'.

Store the deduction fields as positive magnitudes, matching the
convention everywhere else. The stored sign is inert on credit notes:
the reversing verifikat recomputes the ROT/RUT split from the items,
and the PDF and amount-to-pay logic skip deductions on credit notes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(transactions): share the source chip across inbox and history modes

Move SourceFilter to transaction-types.ts (widened with 'bank:other' and
'acct:<id>'), render the one toolbar ContextPicker in both view modes,
and drop the narrower duplicate chip inside TransactionHistoryList. The
history list now applies the acct:/bank:other narrowing itself and hides
skattekonto rows under any bank-side selection.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore(deps): bump js-yaml to 4.3.1

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(schema): recognize PostgREST computed columns in the migration parser

The verifikat amount sort orders by total_amount, a PostgREST computed
column (a function on the journal_entries row type, migration
20260811100000). The schema guard only modeled real columns, so
no-phantom-columns flagged the order as a phantom.

Teach the parser that a function whose only argument is a table's row
type joins that table's column set, with DROP FUNCTION retraction when
the signature names the row type.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: resolve PR #1523 review findings

- journal-entries route: append the globally unique id tiebreak to every
  direct-query sort; voucher series+number repeat across fiscal years, so
  the all-years scope could duplicate or drop rows at page boundaries.
  Existing order assertions updated, new all-years tiebreak test.
- documents inline route: CSP source policy on HTML previews; sandbox
  alone still loads remote resources, letting a tracking pixel notify the
  sender on open. New route test asserts the full header.
- JournalEntryList: catch rejected list requests so loading cannot stick
  forever, and gate every post-await state write behind a request
  generation so a slow earlier request cannot overwrite the current sort.
- TransactionHistoryList: pagination follows the selected source scope
  (reachable with zero matches on the current page, hidden for the
  skattekonto scope it cannot affect).
- transactions page: bank:other picker availability derives from history
  rows too, not only the pending inbox dataset.
- DECISIONS.md: mark the superseded single-sort decision; record the
  credit-note deduction positive-magnitude invariant and its verified
  reader inventory (Swedish review flag).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: guard metadata refetches behind the list request generation

fetchAttachmentCounts and fetchRattelseFlags write state after their own
awaits; a stale list request's late completion could overwrite attachment
counts and rattelse flags for rows a newer request just rendered, showing
false missing-underlag warnings. Both helpers now take the caller's
generation guard and discard stale completions, including the
attachment-counts loaded flag.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Mattsson
2026-08-11 23:12:18 +02:00
committed by GitHub
co-authored by Claude Fable 5
parent f2d9e98af3
commit 9dbaebcc50
29 changed files with 1551 additions and 157 deletions
@@ -0,0 +1,100 @@
import { describe, it, expect } from 'vitest'
import {
ensureHtmlDocument,
buildEmailBodyHtmlDocument,
EMAIL_ALLOWED_MIME_TYPES,
UPLOAD_ALLOWED_MIME_TYPES,
} from '@/extensions/general/invoice-inbox/lib/upload-and-extract'
import { htmlToText } from '@/extensions/general/invoice-inbox/lib/extract-invoice-fields'
import { validateDocumentMagicBytes } from '@/lib/core/documents/document-service'
const decode = (buf: ArrayBuffer) => new TextDecoder().decode(new Uint8Array(buf))
describe('EMAIL_ALLOWED_MIME_TYPES', () => {
it('is the upload set plus text/html, and only reachable from the email pipeline', () => {
for (const type of UPLOAD_ALLOWED_MIME_TYPES) {
expect(EMAIL_ALLOWED_MIME_TYPES.has(type)).toBe(true)
}
expect(EMAIL_ALLOWED_MIME_TYPES.has('text/html')).toBe(true)
// The manual upload surface must stay strict.
expect(UPLOAD_ALLOWED_MIME_TYPES.has('text/html')).toBe(false)
})
})
describe('ensureHtmlDocument', () => {
it('wraps a fragment-shaped mail body into a full document', () => {
const out = decode(ensureHtmlDocument('<div>Faktura 123</div>'))
expect(out.toLowerCase().startsWith('<!doctype html')).toBe(true)
expect(out).toContain('<meta charset="utf-8">')
expect(out).toContain('<div>Faktura 123</div>')
})
it('passes a full document through unchanged', () => {
const full = '<!DOCTYPE html>\n<html><body>Hej</body></html>'
expect(decode(ensureHtmlDocument(full))).toBe(full)
const rooted = '<html><body>Hej</body></html>'
expect(decode(ensureHtmlDocument(rooted))).toBe(rooted)
})
it('recognises a full document behind leading whitespace', () => {
const full = '\n <!doctype html><html></html>'
expect(decode(ensureHtmlDocument(full))).toBe(full)
})
it('produces bytes that pass the document-service magic check', () => {
expect(
validateDocumentMagicBytes(ensureHtmlDocument('<td>999 kr</td>'), 'text/html'),
).toBeNull()
})
})
describe('buildEmailBodyHtmlDocument', () => {
it('prefers the HTML part over the text part', () => {
const out = decode(buildEmailBodyHtmlDocument('<p>Belopp: 100 kr</p>', 'plain fallback')!)
expect(out).toContain('<p>Belopp: 100 kr</p>')
expect(out).not.toContain('plain fallback')
})
it('escapes a plain-text-only body into a <pre>', () => {
const out = decode(
buildEmailBodyHtmlDocument(null, 'Total <script>alert(1)</script> & 100 kr')!,
)
expect(out).toContain('<pre>')
expect(out).toContain('Total &lt;script&gt;alert(1)&lt;/script&gt; &amp; 100 kr')
expect(out).not.toContain('<script>')
})
it('returns null when the mail has no body worth storing', () => {
expect(buildEmailBodyHtmlDocument(null, null)).toBeNull()
expect(buildEmailBodyHtmlDocument(' ', ' \n ')).toBeNull()
})
})
describe('htmlToText', () => {
it('drops script/style/head blocks and comments', () => {
const text = htmlToText(
'<html><head><title>x</title><style>.a{color:red}</style></head>' +
'<body><script>evil()</script><!-- hidden --><p>Faktura 42</p></body></html>',
)
expect(text).toBe('Faktura 42')
})
it('keeps block boundaries as newlines so labels and amounts stay separated', () => {
const text = htmlToText('<table><tr><td>Att betala</td></tr><tr><td>1 234,56 kr</td></tr></table>')
expect(text).toContain('Att betala')
expect(text).toContain('1 234,56 kr')
expect(text.split('\n').length).toBeGreaterThan(1)
})
it('decodes entities, ampersand strictly last', () => {
expect(htmlToText('Moms &amp; frakt: 25&nbsp;kr')).toBe('Moms & frakt: 25 kr')
// Double-decoding would turn this into a real "<" instead of the literal.
expect(htmlToText('visar &amp;lt; som text')).toBe('visar &lt; som text')
expect(htmlToText('&#8364;100 och &#x27;citat&#x27;')).toBe("€100 och 'citat'")
})
it('caps the output length', () => {
const long = `<p>${'a'.repeat(80_000)}</p>`
expect(htmlToText(long).length).toBeLessThanOrEqual(50_000)
})
})
@@ -19,6 +19,16 @@ vi.mock('@supabase/supabase-js', () => ({
createClient: vi.fn(),
}))
// uploadAndExtract does real storage + Bedrock work; these tests only assert
// what the webhook hands it. Constants and the pure HTML helpers stay real so
// MIME gating and body-document building run genuine code.
vi.mock('@/extensions/general/invoice-inbox/lib/upload-and-extract', async () => {
const actual = await vi.importActual<typeof import('@/extensions/general/invoice-inbox/lib/upload-and-extract')>(
'@/extensions/general/invoice-inbox/lib/upload-and-extract'
)
return { ...actual, uploadAndExtract: vi.fn() }
})
// applyDomainStatusFromWebhook confirms the receiving capability with Resend
// before flipping a row to verified: keep that lookup off the network.
const { domainsMock } = vi.hoisted(() => ({
@@ -39,6 +49,7 @@ vi.mock('@/lib/rate-limits/inbox', () => ({
}))
import { verifyInboundWebhook, fetchReceivingEmail, fetchInboundAttachment } from '@/extensions/general/invoice-inbox/lib/resend-inbound'
import { uploadAndExtract } from '@/extensions/general/invoice-inbox/lib/upload-and-extract'
import { createClient } from '@supabase/supabase-js'
function findRoute(method: string, path: string) {
@@ -136,8 +147,9 @@ describe('POST /inbound', () => {
const { supabase, enqueue } = createQueuedMockSupabase()
enqueue({ data: [{ company_id: 'company-9', domain: 'hansbolag.example' }] }) // verified domain
enqueue({ data: { created_by: 'user-owner-9' } }) // company owner
enqueue({ data: null }) // no-attachments error-row insert
enqueue({ data: null }) // body-document dedupe check finds nothing
vi.mocked(createClient).mockReturnValue(supabase as never)
vi.mocked(uploadAndExtract).mockResolvedValue({ inbox_item_id: 'item-9' } as never)
vi.mocked(fetchReceivingEmail).mockResolvedValue({
object: 'email',
id: 'em_123',
@@ -160,7 +172,10 @@ describe('POST /inbound', () => {
const res = await webhookRoute.handler(request)
const body = await res.json()
expect(res.status).toBe(200)
expect(body.data.reason).toBe('no_attachments')
// A body-only mail now becomes a text/html document for company-9:
// proves the custom-domain routing reached the processing stage.
expect(body.data.reason).toBe('email_body')
expect(vi.mocked(uploadAndExtract).mock.calls[0][2]).toBe('company-9')
})
it('does not route mail for an unverified custom domain', async () => {
@@ -190,8 +205,9 @@ describe('POST /inbound', () => {
// would resolve to null (500). A 200 proves the shared path won.
enqueue({ data: { id: 'inbox-1', company_id: 'company-1', status: 'active' } })
enqueue({ data: { created_by: 'user-owner-1' } })
enqueue({ data: null }) // no-attachments error-row insert
enqueue({ data: null }) // body-document dedupe check finds nothing
vi.mocked(createClient).mockReturnValue(supabase as never)
vi.mocked(uploadAndExtract).mockResolvedValue({ inbox_item_id: 'item-1' } as never)
vi.mocked(fetchReceivingEmail).mockResolvedValue({
object: 'email',
id: 'em_123',
@@ -214,7 +230,7 @@ describe('POST /inbound', () => {
const res = await webhookRoute.handler(request)
const body = await res.json()
expect(res.status).toBe(200)
expect(body.data.reason).toBe('no_attachments')
expect(body.data.reason).toBe('email_body')
})
it('applies domain.updated events to custom-domain rows', async () => {
@@ -322,14 +338,61 @@ describe('POST /inbound', () => {
expect(res.status).toBe(500)
})
it('logs an error inbox item when email has no attachments', async () => {
it('stores the mail body as a text/html document when the mail has no attachments', async () => {
vi.mocked(verifyInboundWebhook).mockReturnValue(
mockReceivedEvent({ attachments: [] }) as never
)
const { supabase, enqueue } = createQueuedMockSupabase()
enqueue({ data: { id: 'inbox-1', company_id: 'company-1', status: 'active' } })
enqueue({ data: { created_by: 'user-owner-1' } })
enqueue({ data: null }) // insert succeeds
enqueue({ data: null }) // body-document dedupe check finds nothing
vi.mocked(createClient).mockReturnValue(supabase as never)
vi.mocked(uploadAndExtract).mockResolvedValue({ inbox_item_id: 'item-body-1' } as never)
vi.mocked(fetchReceivingEmail).mockResolvedValue({
object: 'email',
id: 'em_123',
to: ['acme-ab-x7f2@arcim.io'],
from: 'billing@supplier.com',
created_at: '2026-04-20T10:00:00Z',
subject: 'Kvitto på ditt köp',
bcc: null,
cc: null,
reply_to: null,
html: '<div>Att betala: <b>1 234,56 kr</b></div>',
text: 'Att betala: 1 234,56 kr',
headers: {},
message_id: '<msg@x>',
raw: null,
attachments: [],
} as never)
const request = createMockRequest('/inbound', { method: 'POST', body: {} })
const res = await webhookRoute.handler(request)
const body = await res.json()
expect(res.status).toBe(200)
expect(body.data.reason).toBe('email_body')
expect(body.data.processed).toBe(1)
expect(body.data.inbox_item_id).toBe('item-body-1')
expect(fetchInboundAttachment).not.toHaveBeenCalled()
const [, , companyId, file, source] = vi.mocked(uploadAndExtract).mock.calls[0]
expect(companyId).toBe('company-1')
expect(source).toBe('email')
expect(file.type).toBe('text/html')
expect(file.name).toMatch(/^mail-.*\.html$/)
const stored = new TextDecoder().decode(new Uint8Array(file.buffer))
expect(stored.toLowerCase().startsWith('<!doctype html')).toBe(true)
expect(stored).toContain('<div>Att betala: <b>1 234,56 kr</b></div>')
})
it('keeps the error row for a no-attachment mail with an empty body', async () => {
vi.mocked(verifyInboundWebhook).mockReturnValue(
mockReceivedEvent({ attachments: [] }) as never
)
const { supabase, enqueue } = createQueuedMockSupabase()
enqueue({ data: { id: 'inbox-1', company_id: 'company-1', status: 'active' } })
enqueue({ data: { created_by: 'user-owner-1' } })
enqueue({ data: null }) // error-row insert
vi.mocked(createClient).mockReturnValue(supabase as never)
vi.mocked(fetchReceivingEmail).mockResolvedValue({
object: 'email',
@@ -337,12 +400,12 @@ describe('POST /inbound', () => {
to: ['acme-ab-x7f2@arcim.io'],
from: 'billing@supplier.com',
created_at: '2026-04-20T10:00:00Z',
subject: 'No attachments here',
subject: 'Tomt mejl',
bcc: null,
cc: null,
reply_to: null,
html: null,
text: 'Body only',
text: ' ',
headers: {},
message_id: '<msg@x>',
raw: null,
@@ -354,6 +417,167 @@ describe('POST /inbound', () => {
const body = await res.json()
expect(res.status).toBe(200)
expect(body.data.reason).toBe('no_attachments')
expect(fetchInboundAttachment).not.toHaveBeenCalled()
expect(uploadAndExtract).not.toHaveBeenCalled()
})
it('does not duplicate the body document when Resend retries the webhook', async () => {
vi.mocked(verifyInboundWebhook).mockReturnValue(
mockReceivedEvent({ attachments: [] }) as never
)
const { supabase, enqueue } = createQueuedMockSupabase()
enqueue({ data: { id: 'inbox-1', company_id: 'company-1', status: 'active' } })
enqueue({ data: { created_by: 'user-owner-1' } })
enqueue({ data: { id: 'existing-body-item' } }) // dedupe check finds the first delivery's row
vi.mocked(createClient).mockReturnValue(supabase as never)
vi.mocked(fetchReceivingEmail).mockResolvedValue({
object: 'email',
id: 'em_123',
to: ['acme-ab-x7f2@arcim.io'],
from: 'billing@supplier.com',
created_at: '2026-04-20T10:00:00Z',
subject: 'Kvitto',
bcc: null,
cc: null,
reply_to: null,
html: '<div>Kvitto</div>',
text: null,
headers: {},
message_id: '<msg@x>',
raw: null,
attachments: [],
} as never)
const request = createMockRequest('/inbound', { method: 'POST', body: {} })
const res = await webhookRoute.handler(request)
const body = await res.json()
expect(res.status).toBe(200)
expect(body.data.reason).toBe('email_body_duplicate')
expect(body.data.inbox_item_id).toBe('existing-body-item')
expect(uploadAndExtract).not.toHaveBeenCalled()
})
it('falls back to the error row when the body document upload fails', async () => {
vi.mocked(verifyInboundWebhook).mockReturnValue(
mockReceivedEvent({ attachments: [] }) as never
)
const { supabase, enqueue } = createQueuedMockSupabase()
enqueue({ data: { id: 'inbox-1', company_id: 'company-1', status: 'active' } })
enqueue({ data: { created_by: 'user-owner-1' } })
enqueue({ data: null }) // dedupe check finds nothing
enqueue({ data: null }) // fallback error-row insert
vi.mocked(createClient).mockReturnValue(supabase as never)
vi.mocked(uploadAndExtract).mockRejectedValue(new Error('storage down'))
vi.mocked(fetchReceivingEmail).mockResolvedValue({
object: 'email',
id: 'em_123',
to: ['acme-ab-x7f2@arcim.io'],
from: 'billing@supplier.com',
created_at: '2026-04-20T10:00:00Z',
subject: 'Kvitto',
bcc: null,
cc: null,
reply_to: null,
html: '<div>Kvitto</div>',
text: null,
headers: {},
message_id: '<msg@x>',
raw: null,
attachments: [],
} as never)
const request = createMockRequest('/inbound', { method: 'POST', body: {} })
const res = await webhookRoute.handler(request)
const body = await res.json()
expect(res.status).toBe(200)
expect(body.data.reason).toBe('no_attachments')
})
it('accepts a text/html attachment and wraps it into a full document', async () => {
vi.mocked(verifyInboundWebhook).mockReturnValue(mockReceivedEvent() as never)
const { supabase, enqueue } = createQueuedMockSupabase()
enqueue({ data: { id: 'inbox-1', company_id: 'company-1', status: 'active' } })
enqueue({ data: { created_by: 'user-owner-1' } })
enqueue({ data: null }) // per-attachment dup check finds nothing
vi.mocked(createClient).mockReturnValue(supabase as never)
vi.mocked(uploadAndExtract).mockResolvedValue({ inbox_item_id: 'item-html-1' } as never)
vi.mocked(fetchReceivingEmail).mockResolvedValue({
object: 'email',
id: 'em_123',
to: ['acme-ab-x7f2@arcim.io'],
from: 'billing@supplier.com',
created_at: '2026-04-20T10:00:00Z',
subject: 'Faktura',
bcc: null,
cc: null,
reply_to: null,
html: null,
text: 'Se bifogad faktura',
headers: {},
message_id: '<msg@x>',
raw: null,
attachments: [
{ id: 'att_html', filename: 'faktura.html', size: 100, content_type: 'text/html', content_id: 'cid', content_disposition: 'attachment' },
],
} as never)
vi.mocked(fetchInboundAttachment).mockResolvedValue({
id: 'att_html',
filename: 'faktura.html',
contentType: 'text/html',
buffer: new TextEncoder().encode('<div>Faktura 123: 500 kr</div>').buffer as ArrayBuffer,
})
const request = createMockRequest('/inbound', { method: 'POST', body: {} })
const res = await webhookRoute.handler(request)
const body = await res.json()
expect(res.status).toBe(200)
expect(body.data.results[0].inbox_item_id).toBe('item-html-1')
const [, , , file] = vi.mocked(uploadAndExtract).mock.calls[0]
expect(file.type).toBe('text/html')
const stored = new TextDecoder().decode(new Uint8Array(file.buffer))
expect(stored.toLowerCase().startsWith('<!doctype html')).toBe(true)
expect(stored).toContain('<div>Faktura 123: 500 kr</div>')
})
it('still rejects attachment types outside the email allowlist', async () => {
vi.mocked(verifyInboundWebhook).mockReturnValue(mockReceivedEvent() as never)
const { supabase, enqueue } = createQueuedMockSupabase()
enqueue({ data: { id: 'inbox-1', company_id: 'company-1', status: 'active' } })
enqueue({ data: { created_by: 'user-owner-1' } })
enqueue({ data: null }) // per-attachment dup check finds nothing
enqueue({ data: null }) // rejection-row insert
vi.mocked(createClient).mockReturnValue(supabase as never)
vi.mocked(fetchReceivingEmail).mockResolvedValue({
object: 'email',
id: 'em_123',
to: ['acme-ab-x7f2@arcim.io'],
from: 'billing@supplier.com',
created_at: '2026-04-20T10:00:00Z',
subject: 'Zip',
bcc: null,
cc: null,
reply_to: null,
html: null,
text: 'Body',
headers: {},
message_id: '<msg@x>',
raw: null,
attachments: [
{ id: 'att_zip', filename: 'faktura.zip', size: 100, content_type: 'application/zip', content_id: 'cid', content_disposition: 'attachment' },
],
} as never)
vi.mocked(fetchInboundAttachment).mockResolvedValue({
id: 'att_zip',
filename: 'faktura.zip',
contentType: 'application/zip',
buffer: new ArrayBuffer(8),
})
const request = createMockRequest('/inbound', { method: 'POST', body: {} })
const res = await webhookRoute.handler(request)
const body = await res.json()
expect(res.status).toBe(200)
expect(body.data.results[0].error).toBe('Unsupported type application/zip')
expect(uploadAndExtract).not.toHaveBeenCalled()
})
})
+99 -6
View File
@@ -15,6 +15,9 @@ import {
MAX_FILE_SIZE,
MAX_PAGES_FOR_AUTO_EXTRACT,
UPLOAD_ALLOWED_MIME_TYPES,
EMAIL_ALLOWED_MIME_TYPES,
ensureHtmlDocument,
buildEmailBodyHtmlDocument,
} from './lib/upload-and-extract'
import {
verifyInboundWebhook,
@@ -1411,6 +1414,55 @@ export const invoiceInboxExtension: Extension = {
}
if (attachments.length === 0) {
// Body-only mail: for many suppliers the HTML body IS the invoice
// (SaaS receipts, e-mail invoices), and often the only underlag the
// user has. Store the body as a text/html document and run the
// normal extract pipeline instead of dead-ending in an error row.
// Mails with an empty body keep the old error row.
const bodyDoc = buildEmailBodyHtmlDocument(fullEmail.html ?? null, bodyText)
if (bodyDoc && bodyDoc.byteLength <= MAX_FILE_SIZE) {
// Resend retries the webhook on failure: a retry after success
// must not duplicate the body document. Body items carry the
// email_id with a NULL attachment id.
const { data: existingBody } = await serviceSupabase
.from('invoice_inbox_items')
.select('id')
.eq('resend_email_id', email_id)
.is('resend_attachment_id', null)
.maybeSingle()
if (existingBody) {
return NextResponse.json(
{ data: { processed: 0, reason: 'email_body_duplicate', inbox_item_id: existingBody.id } }
)
}
try {
const result = await uploadAndExtract(
serviceSupabase,
userId,
companyId,
{
name: `mail-${sanitiseFilename(subject, 'meddelande')}.html`,
buffer: bodyDoc,
type: 'text/html',
},
'email',
{
from,
subject,
receivedAt: created_at,
messageId: message_id,
bodyText,
resendEmailId: email_id,
}
)
return NextResponse.json(
{ data: { processed: 1, reason: 'email_body', inbox_item_id: result.inbox_item_id } }
)
} catch (err) {
// Fall through to the error row so the mail never vanishes.
console.error('[invoice-inbox/inbound] Email-body document failed:', err)
}
}
await serviceSupabase.from('invoice_inbox_items').insert({
company_id: companyId,
user_id: userId,
@@ -1488,13 +1540,44 @@ export const invoiceInboxExtension: Extension = {
if (download.contentType === 'message/rfc822') {
const parsed = await simpleParser(Buffer.from(download.buffer))
const innerAttachments = parsed.attachments || []
const innerFrom = parsed.from?.text || from
const innerSubject = parsed.subject || subject
if (innerAttachments.length === 0) {
// Gmail "Forward as attachment" of a body-only HTML invoice:
// the forwarded mail's body is the underlag. Same treatment
// as a direct body-only mail; empty bodies keep the rejection.
const innerBodyDoc = buildEmailBodyHtmlDocument(
typeof parsed.html === 'string' ? parsed.html : null,
parsed.text ?? null
)
if (innerBodyDoc && innerBodyDoc.byteLength <= MAX_FILE_SIZE) {
const innerBodyResult = await uploadAndExtract(
serviceSupabase,
userId,
companyId,
{
name: `mail-${sanitiseFilename(innerSubject, 'meddelande')}.html`,
buffer: innerBodyDoc,
type: 'text/html',
},
'email',
{
from: innerFrom,
subject: innerSubject,
receivedAt: created_at,
messageId: message_id,
bodyText,
resendEmailId: email_id,
resendAttachmentId: att.id,
}
)
results.push({ attachment_id: att.id, inbox_item_id: innerBodyResult.inbox_item_id })
continue
}
await logRejection(att.id, download.filename, download.contentType, 'Det vidarebefordrade meddelandet innehöll inga bilagor')
results.push({ attachment_id: att.id, error: 'eml_no_inner_attachments' })
continue
}
const innerFrom = parsed.from?.text || from
const innerSubject = parsed.subject || subject
for (let i = 0; i < innerAttachments.length; i++) {
const inner = innerAttachments[i]
const innerType = sanitiseMime(inner.contentType)
@@ -1502,7 +1585,7 @@ export const invoiceInboxExtension: Extension = {
const innerBuffer = inner.content
if (!innerBuffer) continue
const innerId = `${att.id}#${i}`
if (!UPLOAD_ALLOWED_MIME_TYPES.has(innerType)) {
if (!EMAIL_ALLOWED_MIME_TYPES.has(innerType)) {
await logRejection(innerId, innerName, innerType, `Avvisad bilaga från vidarebefordrat mejl: filtypen ${innerType} stöds inte`)
results.push({ attachment_id: innerId, error: `Unsupported type ${innerType}` })
continue
@@ -1512,7 +1595,10 @@ export const invoiceInboxExtension: Extension = {
results.push({ attachment_id: innerId, error: 'Inner attachment too large' })
continue
}
const innerArrayBuffer = new Uint8Array(innerBuffer).buffer
const innerArrayBuffer =
innerType === 'text/html'
? ensureHtmlDocument(innerBuffer.toString('utf8'))
: new Uint8Array(innerBuffer).buffer
const innerResult = await uploadAndExtract(
serviceSupabase,
userId,
@@ -1534,7 +1620,7 @@ export const invoiceInboxExtension: Extension = {
continue
}
if (!UPLOAD_ALLOWED_MIME_TYPES.has(download.contentType)) {
if (!EMAIL_ALLOWED_MIME_TYPES.has(download.contentType)) {
await logRejection(att.id, download.filename, download.contentType, `Avvisad: filtypen ${download.contentType} stöds inte`)
results.push({ attachment_id: att.id, error: `Unsupported type ${download.contentType}` })
continue
@@ -1545,11 +1631,18 @@ export const invoiceInboxExtension: Extension = {
continue
}
// Attached .html invoices are often fragments; wrap them into a
// self-contained document so the archive holds a renderable file.
const attachmentBuffer =
download.contentType === 'text/html'
? ensureHtmlDocument(Buffer.from(download.buffer).toString('utf8'))
: download.buffer
const result = await uploadAndExtract(
serviceSupabase,
userId,
companyId,
{ name: download.filename, buffer: download.buffer, type: download.contentType },
{ name: download.filename, buffer: attachmentBuffer, type: download.contentType },
'email',
{
from,
@@ -34,12 +34,15 @@ const MAX_TOKENS = (() => {
// Bedrock supports these document/image media types directly. HEIC/HEIF
// are not on the list, so we skip AI for those: the inbox row still
// lands and the user can edit fields manually or replace the file.
// text/html (mail-body invoices from the inbound pipeline) is not sent as
// a document block: it is converted to plain text via htmlToText() first.
const SUPPORTED_MEDIA_TYPES = new Set([
'application/pdf',
'image/jpeg',
'image/png',
'image/webp',
'image/gif',
'text/html',
])
export interface ExtractionInput {
@@ -378,7 +381,62 @@ async function normalizeImageForExtraction(
}
}
// Cap the text handed to the model. HTML mails can carry hundreds of KB of
// framework markup; the invoice fields sit in the first fraction of the
// visible text, and MAX_TOKENS bounds the output side anyway.
const MAX_EXTRACTION_TEXT_LENGTH = 50_000
/**
* Best-effort HTML-to-text for mail-body invoices. Not a sanitiser (the
* output is only ever sent to the model as plain text, never rendered):
* drops script/style/head blocks and comments, keeps block boundaries as
* newlines so amounts and labels stay line-separated, decodes the entities
* that occur in practice, and collapses whitespace.
*/
export function htmlToText(html: string): string {
const withoutBlocks = html
.replace(/<(script|style|head|title)\b[^>]*>[\s\S]*?<\/\1\s*>/gi, ' ')
.replace(/<!--[\s\S]*?-->/g, ' ')
const withBreaks = withoutBlocks
.replace(/<\/(p|div|tr|li|h[1-6]|table|thead|tbody|section|article|blockquote|pre)\s*>/gi, '\n')
.replace(/<(br|hr)\b[^>]*\/?>/gi, '\n')
const stripped = withBreaks.replace(/<[^>]+>/g, ' ')
const decoded = stripped
.replace(/&nbsp;/gi, ' ')
.replace(/&lt;/gi, '<')
.replace(/&gt;/gi, '>')
.replace(/&quot;/gi, '"')
.replace(/&apos;/gi, "'")
.replace(/&#(\d+);/g, (_, n: string) => {
const code = Number(n)
return code > 31 && code <= 0x10ffff ? String.fromCodePoint(code) : ' '
})
.replace(/&#x([0-9a-f]+);/gi, (_, n: string) => {
const code = parseInt(n, 16)
return code > 31 && code <= 0x10ffff ? String.fromCodePoint(code) : ' '
})
// &amp; strictly last: decoding it earlier would double-decode
// "&amp;lt;" into "<" instead of the literal "&lt;".
.replace(/&amp;/gi, '&')
return decoded
.replace(/[ \t]+/g, ' ')
.replace(/\s*\n\s*/g, '\n')
.replace(/\n{3,}/g, '\n\n')
.trim()
.slice(0, MAX_EXTRACTION_TEXT_LENGTH)
}
function buildContent(input: ExtractionInput) {
if (input.mimeType === 'text/html') {
const text = htmlToText(input.buffer.toString('utf8'))
return [
{
type: 'text' as const,
text: `The document is an HTML email invoice, converted to plain text:\n\n${text}`,
},
{ type: 'text' as const, text: 'Extract the fields per the schema. JSON only.' },
]
}
const base64 = input.buffer.toString('base64')
if (input.mimeType === 'application/pdf') {
return [
@@ -99,6 +99,58 @@ export const UPLOAD_ALLOWED_MIME_TYPES = new Set([
'image/webp',
])
// The email pipeline additionally accepts HTML: many suppliers send the
// invoice as the mail body or an attached .html file, and for those the mail
// IS the underlag (BFL: the received form must be preserved). Deliberately
// NOT added to UPLOAD_ALLOWED_MIME_TYPES: HTML only enters through the
// inbound webhook, where the preview surfaces render it fully sandboxed.
export const EMAIL_ALLOWED_MIME_TYPES = new Set([...UPLOAD_ALLOWED_MIME_TYPES, 'text/html'])
function escapeHtml(raw: string): string {
return raw
.replace(/&/g, '&amp;')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;')
.replace(/"/g, '&quot;')
.replace(/'/g, '&#39;')
}
/**
* Wrap HTML in a minimal document shell unless it already is a full document.
* Mail bodies are usually fragments (a bare <div>/<table>); the document
* archive should hold self-contained files, and the document-service
* magic-byte check requires HTML content to start with a doctype/root element.
*/
export function ensureHtmlDocument(html: string): ArrayBuffer {
const head = html.replace(/^[\s]+/, '').slice(0, 256).toLowerCase()
const isFullDocument =
head.startsWith('<!doctype html') || head.startsWith('<html') || head.startsWith('<?xml')
const full = isFullDocument
? html
: `<!doctype html>\n<html>\n<head><meta charset="utf-8"></head>\n<body>\n${html}\n</body>\n</html>\n`
const bytes = new TextEncoder().encode(full)
// Copy into a fresh ArrayBuffer: Uint8Array.buffer is ArrayBufferLike
// (possibly SharedArrayBuffer-backed) and may span more than the view.
const out = new ArrayBuffer(bytes.byteLength)
new Uint8Array(out).set(bytes)
return out
}
/**
* Build a text/html document from a mail's body parts, preferring the HTML
* part. A plain-text-only mail is escaped into a <pre> so whitespace, amounts
* and OCR numbers survive verbatim. Returns null when the mail has no body
* worth storing.
*/
export function buildEmailBodyHtmlDocument(
html: string | null,
text: string | null
): ArrayBuffer | null {
if (html?.trim()) return ensureHtmlDocument(html)
if (text?.trim()) return ensureHtmlDocument(`<pre>${escapeHtml(text)}</pre>`)
return null
}
export interface EmailMeta {
from?: string | null
subject?: string | null