feat(ai): job-shaped AI service with OpenAI-compatible backend, extraction-first; stop extracting every inbox document twice (#1740)
* feat(ai): job-shaped AI service with OpenAI-compatible backend, extraction-first; stop extracting every inbox document twice Sovereign plan WS1 PR1 (#1406 Tier 2, extraction-first, aligned with the AI surface audit). lib/ai grows a job-shaped service (generateText / generateStructured / extractFromDocument; no streaming members yet, see plan rule R3): - services/anthropic-family delegates to the existing createAiClient() and sends the exact request literals the inbox extractor sent before (request-shape tests deep-equal them), so hosted Bedrock stays byte-identical. - services/openai-compatible talks to any chat-completions endpoint (BYO Swedish provider) via Vercel AI SDK 6.x, exact-pinned and guarded: images as parts, PDFs rasterized with poppler (AI_PDF_MODE) or sent natively, AI_VISION / AI_STRICT_JSON declared, honest skips (ai_no_vision, pdf_rasterizer_missing) instead of fake failures. - config.ts: AI_PROVIDER/AI_BASE_URL/AI_API_KEY/AI_MODEL and per-tier AI_*_MODEL with the legacy BEDROCK_* names kept as the same overrides; getAiStatus() is the single source of truth for "is AI wired up". - provider.ts: openai-compatible in the auto-detect chain (after Bedrock and the direct API); createAiClient() refuses it loudly. Document extraction moves onto the service and gets the audit's fixes: - Inbox documents were extracted TWICE (pipeline A ran inside uploadDocument() before the inbox row existed, so its dedupe branch never fired; 3 707 + 1 666 calls / 30 d). The inbox now declares extractionOwner on the upload, the extension yields, and the inbox mirrors its single outcome onto document_attachments from every writer (sync, deferred, attach, retry, MCP). - Every "no extraction will ever happen" outcome is stamped (skipped:no_ai_entitlement / ai_unconfigured / system_generated / ...); the status route maps the quiet ones to 'disabled' on the first poll instead of a 30 s client timeout. Prod showed 309 of the 327 never-extracted uploads were the paywall working silently. - Self-generated documents (our own invoice PDFs, payout files) are no longer OCR'd. - Agent invoke answers 503 ai_unconfigured when the deployment has no assistant backend, distinct from the paywall. Guard: new direct-ai-client antipattern check (shrink-only allowlist of the pre-abstraction SDK callers) plus exact pins for @anthropic-ai/sdk, ai and @ai-sdk/openai-compatible. Verified: 15 958 unit tests green, guards, lint ratchet, typecheck, and a live smoke against hosted Bedrock through the new service (ping, streamed tool turn, thinking+cache, PDF extraction). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(ai): make AI_API_KEY optional for OpenAI-compatible endpoints (keyless local model servers) A local model server (llama.cpp's server, Ollama /v1, LM Studio, vLLM) usually has no auth. Before, the OpenAI-compatible backend required both AI_BASE_URL and AI_API_KEY to count as configured, so running Accounted on a local model meant setting a meaningless placeholder key. - resolveAiProvider / hasAiCredentials: a base URL alone is now enough. - services/openai-compatible: only send Authorization: Bearer when AI_API_KEY is set, so a keyless server is never handed an empty bearer; a hosted provider that needs a key still sets it. - Docs (SELF-HOSTING Option 3: local-model example, key marked optional), DECISIONS. Verified: with no AI_API_KEY, just AI_BASE_URL + AI_MODEL, getAiStatus() reports configured=true / provider=openai-compatible (live). lib/ai suite 71 green; tsc, guards, lint clean. Bedrock/Anthropic logic unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
Jakob Wennberg
parent
fbf47649f2
commit
c7a75d069d
@@ -0,0 +1,158 @@
|
||||
import { describe, it, expect, vi, beforeEach } from 'vitest'
|
||||
import { createQueuedMockSupabase } from '@/tests/helpers'
|
||||
|
||||
const { supabase, enqueue, reset, findCalls } = createQueuedMockSupabase()
|
||||
|
||||
vi.mock('@/lib/supabase/server', () => ({
|
||||
createServiceClient: () => supabase,
|
||||
}))
|
||||
|
||||
const extractMock = vi.fn()
|
||||
vi.mock('@/extensions/general/invoice-inbox/lib/extract-invoice-fields', () => ({
|
||||
extractInvoiceFields: (...args: unknown[]) => extractMock(...args),
|
||||
}))
|
||||
|
||||
const hasCapabilityMock = vi.fn()
|
||||
vi.mock('@/lib/entitlements/has-capability', () => ({
|
||||
hasCapability: (...args: unknown[]) => hasCapabilityMock(...args),
|
||||
}))
|
||||
|
||||
const aiStatusMock = vi.fn()
|
||||
vi.mock('@/lib/ai', () => ({
|
||||
getAiStatus: () => aiStatusMock(),
|
||||
}))
|
||||
|
||||
import { documentExtractionExtension } from '../index'
|
||||
|
||||
const handler = documentExtractionExtension.eventHandlers![0].handler
|
||||
|
||||
function doc(overrides: Record<string, unknown> = {}) {
|
||||
return {
|
||||
id: 'doc-1',
|
||||
company_id: 'company-1',
|
||||
file_name: 'kvitto.pdf',
|
||||
mime_type: 'application/pdf',
|
||||
storage_path: 'company-1/user-1/kvitto.pdf',
|
||||
upload_source: 'file_upload',
|
||||
...overrides,
|
||||
}
|
||||
}
|
||||
|
||||
function payload(overrides: Record<string, unknown> = {}, document = doc()) {
|
||||
return { document, userId: 'user-1', companyId: 'company-1', ...overrides }
|
||||
}
|
||||
|
||||
/** extraction_model of the LAST document_attachments update, or undefined. */
|
||||
function lastStamp(): string | undefined {
|
||||
const updates = findCalls('document_attachments', 'update')
|
||||
const last = updates[updates.length - 1]?.[0] as { extraction_model?: string } | undefined
|
||||
return last?.extraction_model
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
reset()
|
||||
aiStatusMock.mockReturnValue({ configured: true, assistantAvailable: true })
|
||||
hasCapabilityMock.mockResolvedValue(true)
|
||||
extractMock.mockResolvedValue({
|
||||
data: { supplier: { name: 'Elgiganten' } },
|
||||
rawText: '{"supplier":{"name":"Elgiganten"}}',
|
||||
model: 'eu.anthropic.claude-sonnet-5',
|
||||
})
|
||||
})
|
||||
|
||||
describe('document-extraction handler', () => {
|
||||
// THE dedupe: inbox-owned documents are extracted (and mirrored) by the
|
||||
// inbox itself. The handler used to race it and pay a second model call.
|
||||
it('yields entirely when the inbox owns extraction', async () => {
|
||||
await handler(payload({ extractionOwner: 'invoice-inbox' }))
|
||||
expect(supabase.from).not.toHaveBeenCalled()
|
||||
expect(extractMock).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('stamps unsupported types from the payload without reading the row', async () => {
|
||||
enqueue({ data: null }) // the stamp update
|
||||
await handler(payload({}, doc({ mime_type: 'application/json' })))
|
||||
expect(findCalls('document_attachments', 'select')).toHaveLength(0)
|
||||
expect(lastStamp()).toBe('skipped:unsupported_mime')
|
||||
expect(extractMock).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
// Our own invoice PDFs, payout files, filings: nothing to read, paid calls
|
||||
// to waste (on hosted and on a BYO-key self-host).
|
||||
it('stamps system-generated documents instead of extracting them', async () => {
|
||||
enqueue({ data: null })
|
||||
await handler(payload({}, doc({ upload_source: 'system' })))
|
||||
expect(lastStamp()).toBe('skipped:system_generated')
|
||||
expect(extractMock).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('does nothing for a row that was already attempted', async () => {
|
||||
enqueue({ data: { id: 'doc-1', mime_type: 'application/pdf', storage_path: 'p', extracted_at: '2026-08-20T00:00:00Z' } })
|
||||
await handler(payload())
|
||||
expect(findCalls('document_attachments', 'update')).toHaveLength(0)
|
||||
expect(extractMock).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
// Self-host without an AI key: stamp so the status route answers
|
||||
// 'disabled' on the first poll instead of after a 30 s timeout.
|
||||
it('stamps ai_unconfigured when the deployment has no AI', async () => {
|
||||
aiStatusMock.mockReturnValue({ configured: false, assistantAvailable: false })
|
||||
enqueue({ data: { id: 'doc-1', mime_type: 'application/pdf', storage_path: 'p', extracted_at: null } })
|
||||
enqueue({ data: null })
|
||||
await handler(payload())
|
||||
expect(lastStamp()).toBe('skipped:ai_unconfigured')
|
||||
expect(hasCapabilityMock).not.toHaveBeenCalled()
|
||||
expect(extractMock).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
// The paywall, made visible: 309 of the 327 never-extracted uploads in a
|
||||
// 30-day prod window belonged to companies without the ai capability.
|
||||
it('stamps no_ai_entitlement for companies without the ai capability', async () => {
|
||||
hasCapabilityMock.mockResolvedValue(false)
|
||||
enqueue({ data: { id: 'doc-1', mime_type: 'application/pdf', storage_path: 'p', extracted_at: null } })
|
||||
enqueue({ data: null })
|
||||
await handler(payload())
|
||||
expect(lastStamp()).toBe('skipped:no_ai_entitlement')
|
||||
expect(extractMock).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('stamps a storage download failure', async () => {
|
||||
enqueue({ data: { id: 'doc-1', mime_type: 'application/pdf', storage_path: 'p', extracted_at: null } })
|
||||
enqueue({ data: null })
|
||||
supabase.storage.from.mockReturnValueOnce({
|
||||
download: vi.fn().mockResolvedValue({ data: null, error: { message: 'boom' } }),
|
||||
})
|
||||
await handler(payload())
|
||||
expect(lastStamp()).toBe('failed:storage_download')
|
||||
expect(extractMock).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('persists the result with the model that answered', async () => {
|
||||
enqueue({ data: { id: 'doc-1', mime_type: 'application/pdf', storage_path: 'p', extracted_at: null } })
|
||||
enqueue({ data: null })
|
||||
await handler(payload())
|
||||
expect(extractMock).toHaveBeenCalledWith(expect.objectContaining({ mimeType: 'application/pdf', fileName: 'kvitto.pdf' }))
|
||||
const updates = findCalls('document_attachments', 'update')
|
||||
expect(updates[updates.length - 1][0]).toMatchObject({
|
||||
extracted_data: { supplier: { name: 'Elgiganten' } },
|
||||
extraction_model: 'eu.anthropic.claude-sonnet-5',
|
||||
})
|
||||
})
|
||||
|
||||
it('stamps the skip reason the extractor reports (no vision, rasterizer missing, ...)', async () => {
|
||||
extractMock.mockResolvedValue({ data: {}, rawText: null, skipped: 'pdf_rasterizer_missing' })
|
||||
enqueue({ data: { id: 'doc-1', mime_type: 'application/pdf', storage_path: 'p', extracted_at: null } })
|
||||
enqueue({ data: null })
|
||||
await handler(payload())
|
||||
expect(lastStamp()).toBe('skipped:pdf_rasterizer_missing')
|
||||
})
|
||||
|
||||
it('stamps failed:no_raw_text when the model call produced nothing parseable', async () => {
|
||||
extractMock.mockResolvedValue({ data: {}, rawText: null })
|
||||
enqueue({ data: { id: 'doc-1', mime_type: 'application/pdf', storage_path: 'p', extracted_at: null } })
|
||||
enqueue({ data: null })
|
||||
await handler(payload())
|
||||
expect(lastStamp()).toBe('failed:no_raw_text')
|
||||
})
|
||||
})
|
||||
@@ -1,7 +1,7 @@
|
||||
import type { Extension } from '@/lib/extensions/types'
|
||||
import type { SupabaseClient } from '@supabase/supabase-js'
|
||||
import { extractInvoiceFields } from '@/extensions/general/invoice-inbox/lib/extract-invoice-fields'
|
||||
import { toProviderModelId } from '@/lib/ai/provider'
|
||||
import { getAiStatus } from '@/lib/ai'
|
||||
import { hasCapability } from '@/lib/entitlements/has-capability'
|
||||
import { CAPABILITY } from '@/lib/entitlements/keys'
|
||||
import { createLogger } from '@/lib/logger'
|
||||
@@ -10,8 +10,9 @@ import type { DocumentAttachment } from '@/types'
|
||||
|
||||
const log = createLogger('document-extraction')
|
||||
|
||||
// Mime types we know Claude can read directly via Bedrock. Anything else
|
||||
// (HEIC, ZIP, TXT, …) is skipped: extracted_at still gets stamped so the
|
||||
// Mime types the extraction can read (a vision model reads PDFs natively on
|
||||
// Claude; OpenAI-compatible backends rasterize them). Anything else (HEIC,
|
||||
// ZIP, TXT, bank JSON, …) is skipped: extracted_at still gets stamped so the
|
||||
// row is marked as "attempted, not eligible".
|
||||
const SUPPORTED_MIME_TYPES = new Set([
|
||||
'application/pdf',
|
||||
@@ -23,43 +24,65 @@ const SUPPORTED_MIME_TYPES = new Set([
|
||||
|
||||
// AI-extraction extension: paid AI tier only.
|
||||
//
|
||||
// Subscribes to the existing document.uploaded event bus topic and runs
|
||||
// Sonnet 4.6 (via Bedrock, reusing invoice-inbox's extractInvoiceFields) on
|
||||
// every uploaded receipt or invoice. Writes the result to
|
||||
// Subscribes to the existing document.uploaded event bus topic and runs the
|
||||
// configured extraction model (reusing invoice-inbox's extractInvoiceFields)
|
||||
// on every uploaded receipt or invoice. Writes the result to
|
||||
// document_attachments.extracted_data so the agent intent capture can use
|
||||
// it without re-asking the user.
|
||||
//
|
||||
// Idempotency: skips when extracted_at is already set on the row. Also
|
||||
// dedupes against invoice_inbox_items.extracted_data: when the inbox
|
||||
// extension already extracted the same file, we copy its result instead
|
||||
// of paying for a second Sonnet call.
|
||||
// Idempotency: skips when extracted_at is already set on the row.
|
||||
//
|
||||
// Ownership: documents that enter through the invoice inbox (web upload,
|
||||
// email, WhatsApp, MCP upload) are extracted by the inbox itself, which
|
||||
// mirrors its result onto document_attachments when done. The inbox marks
|
||||
// the upload with `extractionOwner: 'invoice-inbox'` on the event payload
|
||||
// and this handler yields. It used to race instead: the inbox row did not
|
||||
// exist yet when this handler ran inside uploadDocument(), so every inbox
|
||||
// document was extracted twice (two paid model calls per receipt).
|
||||
//
|
||||
// Every outcome that means "no extraction will ever happen here" is stamped
|
||||
// as `skipped:<reason>` so the polling UI stops immediately instead of
|
||||
// timing out: paywall (`no_ai_entitlement`), no AI configured on this
|
||||
// deployment (`ai_unconfigured`), self-generated documents (our own invoice
|
||||
// PDFs, payout files: `system_generated`), unsupported types.
|
||||
//
|
||||
// Free tier: disable this extension in extensions.config.json. Uploads
|
||||
// still work; the agent intent will see null extracted_data and either
|
||||
// ask the user or call gnubok_get_document_content at chat-time.
|
||||
//
|
||||
// See dev_docs/specialized-agent-plan.md (§ paid/free tier note): to be
|
||||
// authored.
|
||||
export const documentExtractionExtension: Extension = {
|
||||
id: 'document-extraction',
|
||||
name: 'AI document extraction',
|
||||
version: '1.0.0',
|
||||
version: '1.1.0',
|
||||
|
||||
eventHandlers: [
|
||||
{
|
||||
eventType: 'document.uploaded',
|
||||
handler: async (payload) => {
|
||||
const { document, companyId } = payload as {
|
||||
const { document, companyId, extractionOwner } = payload as {
|
||||
document: DocumentAttachment
|
||||
userId: string
|
||||
companyId: string
|
||||
extractionOwner?: 'invoice-inbox'
|
||||
}
|
||||
if (extractionOwner === 'invoice-inbox') return
|
||||
await extractAndPersist(document, companyId)
|
||||
},
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
async function stamp(
|
||||
supabase: SupabaseClient,
|
||||
documentId: string,
|
||||
extractionModel: string
|
||||
): Promise<void> {
|
||||
const { error } = await supabase
|
||||
.from('document_attachments')
|
||||
.update({ extracted_at: new Date().toISOString(), extraction_model: extractionModel })
|
||||
.eq('id', documentId)
|
||||
if (error) log.warn('stamp failed', { doc: documentId, extractionModel, err: error.message })
|
||||
}
|
||||
|
||||
async function extractAndPersist(
|
||||
document: DocumentAttachment,
|
||||
companyId: string,
|
||||
@@ -69,9 +92,22 @@ async function extractAndPersist(
|
||||
// events have no user context.
|
||||
const supabase: SupabaseClient = createServiceClient()
|
||||
|
||||
// Cheap gates first, straight from the event payload: no DB round trip for
|
||||
// the thousands of bank-statement JSON documents a sync produces.
|
||||
const mimeType = (document.mime_type as string | null) ?? null
|
||||
if (!mimeType || !SUPPORTED_MIME_TYPES.has(mimeType)) {
|
||||
await stamp(supabase, document.id, 'skipped:unsupported_mime')
|
||||
return
|
||||
}
|
||||
// Documents the system produced itself (our own invoice PDFs, payment
|
||||
// files, filings) carry nothing to extract; reading them back with a paid
|
||||
// model is waste, on hosted and doubly so on a BYO-key self-host.
|
||||
if (document.upload_source === 'system') {
|
||||
await stamp(supabase, document.id, 'skipped:system_generated')
|
||||
return
|
||||
}
|
||||
|
||||
// Idempotency guard: never re-extract a row that already has extracted_at.
|
||||
// Note: the column may be null OR the row may not yet have the new
|
||||
// schema (legacy supabase types). Fail closed on missing schema.
|
||||
const { data: existing, error: existingErr } = await supabase
|
||||
.from('document_attachments')
|
||||
.select('id, mime_type, storage_path, extracted_at')
|
||||
@@ -88,104 +124,70 @@ async function extractAndPersist(
|
||||
return
|
||||
}
|
||||
|
||||
// Dedup against inbox: if invoice-inbox already extracted this exact file
|
||||
// (same document_id), copy its result to avoid a second AI call. If the
|
||||
// inbox row marked the upload as skip_extraction=true, the inbox row's
|
||||
// extracted_data is an empty skeleton: we must stamp the doc with a
|
||||
// 'skipped:*' model so the client-side useDocumentExtraction hook reports
|
||||
// 'unsupported' rather than 'succeeded' (otherwise the UI would claim AI
|
||||
// finished reading a doc it never opened).
|
||||
const { data: inboxRow } = await supabase
|
||||
.from('invoice_inbox_items')
|
||||
.select('extracted_data, extraction_skipped')
|
||||
.eq('document_id', document.id)
|
||||
.maybeSingle()
|
||||
|
||||
if (inboxRow?.extraction_skipped) {
|
||||
await supabase
|
||||
.from('document_attachments')
|
||||
.update({
|
||||
extracted_at: new Date().toISOString(),
|
||||
extraction_model: 'skipped:invoice_inbox_gate',
|
||||
})
|
||||
.eq('id', document.id)
|
||||
// No AI configured on this deployment: stamp and stop. This is the
|
||||
// self-host "key not set yet" state; the status route reports it as
|
||||
// disabled on the first poll instead of after a 30 s timeout.
|
||||
if (!getAiStatus().configured) {
|
||||
await stamp(supabase, document.id, 'skipped:ai_unconfigured')
|
||||
return
|
||||
}
|
||||
|
||||
// Paywall: the free/manual tier never triggers paid extraction. Stamp it so
|
||||
// the row reads "attempted, not entitled" rather than NULL forever (which
|
||||
// the polling UI could only interpret by timing out).
|
||||
if (!(await hasCapability(supabase, companyId, CAPABILITY.ai))) {
|
||||
log.info('extraction skipped, ai capability not entitled', { doc: document.id, companyId })
|
||||
await stamp(supabase, document.id, 'skipped:no_ai_entitlement')
|
||||
return
|
||||
}
|
||||
|
||||
// Download the file from Supabase Storage. The bucket is private: the
|
||||
// service-role client can read any path.
|
||||
const storagePath = existing.storage_path as string | null
|
||||
if (!storagePath) {
|
||||
log.warn('document has no storage_path, skipping', { doc: document.id })
|
||||
await stamp(supabase, document.id, 'failed:no_storage_path')
|
||||
return
|
||||
}
|
||||
|
||||
const { data: blob, error: dlErr } = await supabase.storage
|
||||
.from('documents')
|
||||
.download(storagePath)
|
||||
if (dlErr || !blob) {
|
||||
log.warn('storage download failed', { doc: document.id, err: dlErr?.message })
|
||||
await stamp(supabase, document.id, 'failed:storage_download')
|
||||
return
|
||||
}
|
||||
const buffer = Buffer.from(await blob.arrayBuffer())
|
||||
|
||||
let extractedData: Record<string, unknown> | null = null
|
||||
let model: string = 'copied-from-invoice-inbox'
|
||||
|
||||
if (inboxRow?.extracted_data) {
|
||||
extractedData = inboxRow.extracted_data as Record<string, unknown>
|
||||
} else {
|
||||
const mimeType = existing.mime_type as string | null
|
||||
if (!mimeType || !SUPPORTED_MIME_TYPES.has(mimeType)) {
|
||||
// Stamp the attempt so we don't keep retrying unsupported types.
|
||||
await supabase
|
||||
.from('document_attachments')
|
||||
.update({ extracted_at: new Date().toISOString(), extraction_model: 'skipped:unsupported_mime' })
|
||||
.eq('id', document.id)
|
||||
let model: string
|
||||
try {
|
||||
const { data, rawText, model: usedModel, skipped } = await extractInvoiceFields({
|
||||
buffer,
|
||||
mimeType,
|
||||
fileName: (document.file_name as string) || 'document',
|
||||
})
|
||||
// extractInvoiceFields returns an "empty" result on failure rather than
|
||||
// throwing. `skipped` means no model call was made (and why); a null
|
||||
// rawText with no skip means the call failed or the JSON parse did.
|
||||
if (skipped) {
|
||||
await stamp(supabase, document.id, `skipped:${skipped}`)
|
||||
return
|
||||
}
|
||||
|
||||
// Download the file from Supabase Storage. The bucket is private: the
|
||||
// service-role client can read any path.
|
||||
const storagePath = existing.storage_path as string | null
|
||||
if (!storagePath) {
|
||||
log.warn('document has no storage_path, skipping', { doc: document.id })
|
||||
return
|
||||
}
|
||||
|
||||
const { data: blob, error: dlErr } = await supabase.storage
|
||||
.from('documents')
|
||||
.download(storagePath)
|
||||
if (dlErr || !blob) {
|
||||
log.warn('storage download failed', { doc: document.id, err: dlErr?.message })
|
||||
return
|
||||
}
|
||||
const buffer = Buffer.from(await blob.arrayBuffer())
|
||||
|
||||
if (!(await hasCapability(supabase, companyId, CAPABILITY.ai))) {
|
||||
log.info('extraction skipped, ai capability not entitled', { doc: document.id, companyId })
|
||||
return
|
||||
}
|
||||
|
||||
try {
|
||||
const { data, rawText } = await extractInvoiceFields({
|
||||
buffer,
|
||||
mimeType,
|
||||
fileName: (document.file_name as string) || 'document',
|
||||
})
|
||||
// extractInvoiceFields returns an "empty" result on failure rather
|
||||
// than throwing: distinguish by checking rawText. When rawText is
|
||||
// null, the call was skipped (creds missing, unsupported type) or
|
||||
// the JSON parse failed.
|
||||
if (!rawText) {
|
||||
await supabase
|
||||
.from('document_attachments')
|
||||
.update({
|
||||
extracted_at: new Date().toISOString(),
|
||||
extraction_model: 'failed:no_raw_text',
|
||||
})
|
||||
.eq('id', document.id)
|
||||
return
|
||||
}
|
||||
extractedData = data as unknown as Record<string, unknown>
|
||||
model = toProviderModelId(process.env.BEDROCK_MODEL_ID || 'claude-sonnet-5')
|
||||
} catch (err) {
|
||||
log.warn('extraction threw', {
|
||||
doc: document.id,
|
||||
err: err instanceof Error ? err.message : String(err),
|
||||
})
|
||||
await supabase
|
||||
.from('document_attachments')
|
||||
.update({
|
||||
extracted_at: new Date().toISOString(),
|
||||
extraction_model: 'failed:exception',
|
||||
})
|
||||
.eq('id', document.id)
|
||||
if (!rawText) {
|
||||
await stamp(supabase, document.id, 'failed:no_raw_text')
|
||||
return
|
||||
}
|
||||
extractedData = data as unknown as Record<string, unknown>
|
||||
model = usedModel ?? 'unknown'
|
||||
} catch (err) {
|
||||
log.warn('extraction threw', {
|
||||
doc: document.id,
|
||||
err: err instanceof Error ? err.message : String(err),
|
||||
})
|
||||
await stamp(supabase, document.id, 'failed:exception')
|
||||
return
|
||||
}
|
||||
|
||||
const { error: updateErr } = await supabase
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
import { describe, it, expect, vi, beforeEach } from 'vitest'
|
||||
import { createQueuedMockSupabase } from '@/tests/helpers'
|
||||
|
||||
const { supabase, enqueue, reset, findCalls } = createQueuedMockSupabase()
|
||||
vi.mock('@/lib/auth/api-keys', () => ({
|
||||
createServiceClientNoCookies: () => supabase,
|
||||
}))
|
||||
|
||||
import { mirrorExtractionToDocument } from '../lib/mirror-extraction'
|
||||
|
||||
function lastUpdate() {
|
||||
const updates = findCalls('document_attachments', 'update')
|
||||
return updates[updates.length - 1]?.[0] as { extracted_data: unknown; extraction_model: string; extracted_at: string }
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
reset()
|
||||
enqueue({ data: null })
|
||||
})
|
||||
|
||||
describe('mirrorExtractionToDocument', () => {
|
||||
it('writes a successful extraction with the model that answered', async () => {
|
||||
await mirrorExtractionToDocument('doc-1', {
|
||||
data: { supplier: { name: 'x' } } as never,
|
||||
rawText: '{"supplier":{"name":"x"}}',
|
||||
model: 'google/gemma-4-31B-it',
|
||||
})
|
||||
const u = lastUpdate()
|
||||
expect(u.extracted_data).toEqual({ supplier: { name: 'x' } })
|
||||
expect(u.extraction_model).toBe('google/gemma-4-31B-it')
|
||||
expect(u.extracted_at).toBeTruthy()
|
||||
expect(findCalls('document_attachments', 'eq')[0]).toEqual(['id', 'doc-1'])
|
||||
})
|
||||
|
||||
it('writes skipped:<reason> with no data for inbox and AI skip reasons alike', async () => {
|
||||
await mirrorExtractionToDocument('doc-1', { data: {} as never, rawText: null, skipped: 'no_ai_entitlement' })
|
||||
expect(lastUpdate()).toMatchObject({ extracted_data: null, extraction_model: 'skipped:no_ai_entitlement' })
|
||||
await mirrorExtractionToDocument('doc-1', { data: {} as never, rawText: null, skipped: 'ai_no_vision' })
|
||||
expect(lastUpdate()).toMatchObject({ extracted_data: null, extraction_model: 'skipped:ai_no_vision' })
|
||||
})
|
||||
|
||||
it('writes failed:no_raw_text when the call yielded nothing parseable', async () => {
|
||||
await mirrorExtractionToDocument('doc-1', { data: {} as never, rawText: null })
|
||||
expect(lastUpdate()).toMatchObject({ extracted_data: null, extraction_model: 'failed:no_raw_text' })
|
||||
})
|
||||
|
||||
it('never throws: a failed update is logged and swallowed', async () => {
|
||||
reset()
|
||||
enqueue({ error: { message: 'rls says no' } })
|
||||
await expect(mirrorExtractionToDocument('doc-1', { data: {} as never, rawText: null })).resolves.toBeUndefined()
|
||||
})
|
||||
})
|
||||
@@ -24,6 +24,21 @@ vi.mock('@/lib/core/documents/document-service', () => ({
|
||||
uploadDocument: vi.fn().mockResolvedValue({ id: 'doc-1' }),
|
||||
}))
|
||||
|
||||
// The staged (deferred) upload path only exists when AI is configured on this
|
||||
// deployment: an unconfigured one skips synchronously (ai_unconfigured). These
|
||||
// tests simulate a configured deployment; the model call itself is mocked.
|
||||
vi.mock('@/lib/ai', () => ({
|
||||
getAiStatus: () => ({
|
||||
provider: 'bedrock',
|
||||
configured: true,
|
||||
reason: 'ok',
|
||||
capabilities: { pdfNative: true, imageInput: true, toolUse: true, forcedToolChoice: true, strictJsonSchema: false },
|
||||
models: { assistant: 'm', heavy: 'm', extraction: 'm' },
|
||||
pdfMode: 'native',
|
||||
assistantAvailable: true,
|
||||
}),
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/rate-limits/inbox', () => ({
|
||||
checkInboxUploadRateLimit: vi.fn().mockResolvedValue({ ok: true }),
|
||||
}))
|
||||
|
||||
@@ -21,6 +21,21 @@ vi.mock('@/lib/core/documents/document-service', () => ({
|
||||
uploadDocument: vi.fn().mockResolvedValue({ id: 'doc-1' }),
|
||||
}))
|
||||
|
||||
// The staged (deferred) upload path only exists when AI is configured on this
|
||||
// deployment: an unconfigured one skips synchronously (ai_unconfigured). These
|
||||
// tests simulate a configured deployment; the model call itself is mocked.
|
||||
vi.mock('@/lib/ai', () => ({
|
||||
getAiStatus: () => ({
|
||||
provider: 'bedrock',
|
||||
configured: true,
|
||||
reason: 'ok',
|
||||
capabilities: { pdfNative: true, imageInput: true, toolUse: true, forcedToolChoice: true, strictJsonSchema: false },
|
||||
models: { assistant: 'm', heavy: 'm', extraction: 'm' },
|
||||
pdfMode: 'native',
|
||||
assistantAvailable: true,
|
||||
}),
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/rate-limits/inbox', () => ({
|
||||
checkInboxUploadRateLimit: vi.fn().mockResolvedValue({ ok: true }),
|
||||
}))
|
||||
|
||||
@@ -6,6 +6,7 @@ import { uploadDocument } from '@/lib/core/documents/document-service'
|
||||
import { createServiceClient } from '@/lib/supabase/server'
|
||||
import { matchSupplierId } from '@/lib/suppliers/match-supplier'
|
||||
import { extractInvoiceFields, ExtractionSchema, emptyResult } from './lib/extract-invoice-fields'
|
||||
import { mirrorExtractionToDocument } from './lib/mirror-extraction'
|
||||
import {
|
||||
uploadAndExtract,
|
||||
sanitiseFilename,
|
||||
@@ -697,6 +698,9 @@ export const invoiceInboxExtension: Extension = {
|
||||
type: file.type,
|
||||
}, {
|
||||
upload_source: 'file_upload',
|
||||
// This route extracts below and mirrors the outcome onto the
|
||||
// document row; the document-extraction extension must not race it.
|
||||
extractionOwner: 'invoice-inbox',
|
||||
})
|
||||
|
||||
// Same page handling as /upload (issue #553): long PDFs extract
|
||||
@@ -726,16 +730,23 @@ export const invoiceInboxExtension: Extension = {
|
||||
: null
|
||||
const skipExtraction = skipReason !== null
|
||||
|
||||
const { data: extracted } = skipExtraction
|
||||
? { data: emptyResult() }
|
||||
const extraction = skipExtraction
|
||||
? { data: emptyResult(), rawText: null, model: null, skipped: null }
|
||||
: await extractInvoiceFields({
|
||||
buffer: Buffer.from(slicedBuffer ?? buffer),
|
||||
mimeType: file.type,
|
||||
fileName: file.name,
|
||||
})
|
||||
const { data: extracted } = extraction
|
||||
if (!skipExtraction && slicedBuffer != null && pageCount != null) {
|
||||
extracted.pages = { total: pageCount, analyzed: MAX_PAGES_FOR_AUTO_EXTRACT }
|
||||
}
|
||||
await mirrorExtractionToDocument(doc.id, {
|
||||
data: extracted,
|
||||
rawText: extraction.rawText,
|
||||
model: extraction.model ?? null,
|
||||
skipped: skipReason ?? extraction.skipped ?? null,
|
||||
})
|
||||
|
||||
const { error: linkError } = await ctx.supabase
|
||||
.from('invoice_inbox_items')
|
||||
@@ -1088,11 +1099,18 @@ export const invoiceInboxExtension: Extension = {
|
||||
|
||||
try {
|
||||
const buffer = Buffer.from(await blob.arrayBuffer())
|
||||
const { data: extracted } = await extractInvoiceFields({
|
||||
const extraction = await extractInvoiceFields({
|
||||
buffer,
|
||||
mimeType: doc.mime_type,
|
||||
fileName: doc.file_name,
|
||||
})
|
||||
const { data: extracted } = extraction
|
||||
await mirrorExtractionToDocument(item.document_id, {
|
||||
data: extracted,
|
||||
rawText: extraction.rawText,
|
||||
model: extraction.model ?? null,
|
||||
skipped: extraction.skipped ?? null,
|
||||
})
|
||||
|
||||
// Re-running the extraction has to re-run the match too, or the one
|
||||
// affordance the user reaches for when an item failed to auto-link
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
// AI-driven invoice/receipt field extraction.
|
||||
//
|
||||
// Sends the uploaded document directly to Claude Sonnet 4.6 via AWS
|
||||
// Bedrock and asks for a structured InvoiceExtractionResult JSON. Sonnet
|
||||
// reads PDFs, images, and scans natively, which the previous regex
|
||||
// extractor couldn't: that's why English receipts (Anthropic, AWS,
|
||||
// Stripe, …) and image-only PDFs came back empty.
|
||||
// Sends the uploaded document to the configured AI backend through the
|
||||
// job-shaped service in lib/ai (Claude on Bedrock or the direct API on
|
||||
// hosted; any OpenAI-compatible endpoint, e.g. a Swedish inference provider,
|
||||
// on a sovereign self-host) and asks for a structured InvoiceExtractionResult
|
||||
// JSON. Vision models read PDFs, images and scans, which the previous regex
|
||||
// extractor couldn't: that's why English receipts (Anthropic, AWS, Stripe, …)
|
||||
// and image-only PDFs came back empty.
|
||||
//
|
||||
// The AI output is validated against a Zod schema; anything that doesn't
|
||||
// parse falls back to an empty result so the inbox row still lands and
|
||||
@@ -13,30 +15,25 @@
|
||||
import { createHash } from 'node:crypto'
|
||||
import { z } from 'zod'
|
||||
import type { InvoiceExtractionResult } from '@/types'
|
||||
import { createAiClient, hasAiCredentials, toProviderModelId } from '@/lib/ai/provider'
|
||||
import { getAiService, readAiConfig, extractJsonObject } from '@/lib/ai'
|
||||
import type { AiDocumentInput, AiImageMediaType, ExtractionSkipReason } from '@/lib/ai'
|
||||
import { createLogger } from '@/lib/logger'
|
||||
|
||||
// Re-exported for callers and tests that import it from here.
|
||||
export { extractJsonObject }
|
||||
|
||||
const log = createLogger('invoice-inbox-extract')
|
||||
|
||||
// Both overridable via env vars so ops can swap models / raise token caps
|
||||
// without a code deploy. The model id is written bare and adapted to whichever
|
||||
// backend is configured (Bedrock in eu-north-1 on hosted, the direct Anthropic
|
||||
// API on self-hosted: see lib/ai/provider.ts). 8192 tokens is enough headroom
|
||||
// for invoices with 20+ line items.
|
||||
const MODEL = toProviderModelId(process.env.BEDROCK_MODEL_ID || 'claude-sonnet-5')
|
||||
const MAX_TOKENS = (() => {
|
||||
const parsed = Number(process.env.BEDROCK_MAX_TOKENS)
|
||||
// Use the env value only if it's a positive number: `||` would also
|
||||
// fall back on a deliberate `0`, masking what is really an invalid
|
||||
// configuration rather than the intent to disable.
|
||||
return Number.isFinite(parsed) && parsed > 0 ? parsed : 8192
|
||||
})()
|
||||
// Output cap: AI_EXTRACTION_MAX_TOKENS (legacy BEDROCK_MAX_TOKENS) or 8192,
|
||||
// enough headroom for invoices with 20+ line items. The model id per tier is
|
||||
// resolved by lib/ai/config.ts (AI_EXTRACTION_MODEL, legacy BEDROCK_MODEL_ID,
|
||||
// AI_MODEL, then the Claude default on the Anthropic family).
|
||||
|
||||
// Bedrock supports these document/image media types directly. HEIC/HEIF
|
||||
// are not on the list, so we skip AI for those: the inbox row still
|
||||
// lands and the user can edit fields manually or replace the file.
|
||||
// text/html (mail-body invoices from the inbound pipeline) is not sent as
|
||||
// a document block: it is converted to plain text via htmlToText() first.
|
||||
// Media types the extraction accepts. HEIC/HEIF are not on the list, so we
|
||||
// skip AI for those: the inbox row still lands and the user can edit fields
|
||||
// manually or replace the file. text/html (mail-body invoices from the
|
||||
// inbound pipeline) is not sent as a document block: it is converted to plain
|
||||
// text via htmlToText() first, which also makes it work on text-only models.
|
||||
const SUPPORTED_MEDIA_TYPES = new Set([
|
||||
'application/pdf',
|
||||
'image/jpeg',
|
||||
@@ -52,10 +49,16 @@ export interface ExtractionInput {
|
||||
fileName: string
|
||||
}
|
||||
|
||||
export type ExtractionSkipped = ExtractionSkipReason | 'unsupported_media'
|
||||
|
||||
export interface ExtractionOutput {
|
||||
data: InvoiceExtractionResult
|
||||
/** The raw JSON string returned by the model, or null on failure. */
|
||||
/** The raw JSON string returned by the model, or null on failure/skip. */
|
||||
rawText: string | null
|
||||
/** Provider-form id of the model that answered, when a call was made. */
|
||||
model?: string | null
|
||||
/** Set when no model call was made at all (and why). */
|
||||
skipped?: ExtractionSkipped | null
|
||||
}
|
||||
|
||||
// Classification fields are nullable AND .catch(null): a hallucinated enum
|
||||
@@ -244,61 +247,6 @@ Rules:
|
||||
- lineItems: include every line. Empty array is fine if the document has no itemised lines.
|
||||
- vatBreakdown: include one entry per distinct VAT rate. Empty array is fine.`
|
||||
|
||||
// Sonnet 5 intermittently wraps its answer in markdown fences (```json ... ```)
|
||||
// or adds prose around it, despite the JSON-only instruction in the system
|
||||
// prompt. Scan for balanced top-level '{'..'}' candidates (string- and
|
||||
// escape-aware, so braces inside JSON string values don't end a candidate
|
||||
// early) and return the first one JSON.parse accepts; prose braces around the
|
||||
// object form unparseable candidates and are skipped. Returns the input
|
||||
// unchanged when no candidate parses, so the existing parse-failure path
|
||||
// handles prose-only refusals. Zod validation downstream still rejects
|
||||
// well-formed-but-wrong JSON.
|
||||
// Bounds for the candidate scan below. Real model output is already capped
|
||||
// by MAX_TOKENS (roughly 33 KB of text at 8192 tokens), so genuine responses
|
||||
// never come near these; they exist so pathological or adversarially
|
||||
// brace-laden text cannot make the scan quadratic (compliance review
|
||||
// A.8.28). Oversized or exhausted inputs fall through to the raw text and
|
||||
// land in the existing empty-result path.
|
||||
const MAX_SCAN_INPUT_LENGTH = 256 * 1024
|
||||
const MAX_CANDIDATE_ATTEMPTS = 50
|
||||
|
||||
export function extractJsonObject(raw: string): string {
|
||||
if (raw.length > MAX_SCAN_INPUT_LENGTH) return raw
|
||||
let attempts = 0
|
||||
let start = raw.indexOf('{')
|
||||
while (start !== -1 && attempts < MAX_CANDIDATE_ATTEMPTS) {
|
||||
attempts++
|
||||
let depth = 0
|
||||
let inString = false
|
||||
let escaped = false
|
||||
for (let i = start; i < raw.length; i++) {
|
||||
const ch = raw[i]
|
||||
if (inString) {
|
||||
if (escaped) escaped = false
|
||||
else if (ch === '\\') escaped = true
|
||||
else if (ch === '"') inString = false
|
||||
} else if (ch === '"') {
|
||||
inString = true
|
||||
} else if (ch === '{') {
|
||||
depth++
|
||||
} else if (ch === '}') {
|
||||
depth--
|
||||
if (depth === 0) {
|
||||
const candidate = raw.slice(start, i + 1)
|
||||
try {
|
||||
JSON.parse(candidate)
|
||||
return candidate
|
||||
} catch {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
start = raw.indexOf('{', start + 1)
|
||||
}
|
||||
return raw
|
||||
}
|
||||
|
||||
export function emptyResult(): InvoiceExtractionResult {
|
||||
return {
|
||||
documentKind: null,
|
||||
@@ -427,44 +375,137 @@ export function htmlToText(html: string): string {
|
||||
.slice(0, MAX_EXTRACTION_TEXT_LENGTH)
|
||||
}
|
||||
|
||||
function buildContent(input: ExtractionInput) {
|
||||
const EXTRACTION_INSTRUCTION = 'Extract the fields per the schema. JSON only.'
|
||||
|
||||
/** Map an upload to the service's document input. HTML becomes plain text. */
|
||||
function toDocumentInput(input: ExtractionInput): AiDocumentInput {
|
||||
if (input.mimeType === 'text/html') {
|
||||
const text = htmlToText(input.buffer.toString('utf8'))
|
||||
return [
|
||||
{
|
||||
type: 'text' as const,
|
||||
text: `The document is an HTML email invoice, converted to plain text:\n\n${text}`,
|
||||
},
|
||||
{ type: 'text' as const, text: 'Extract the fields per the schema. JSON only.' },
|
||||
]
|
||||
return {
|
||||
kind: 'text',
|
||||
text: `The document is an HTML email invoice, converted to plain text:\n\n${text}`,
|
||||
}
|
||||
}
|
||||
const base64 = input.buffer.toString('base64')
|
||||
if (input.mimeType === 'application/pdf') {
|
||||
return [
|
||||
{
|
||||
type: 'document' as const,
|
||||
source: { type: 'base64' as const, media_type: 'application/pdf' as const, data: base64 },
|
||||
},
|
||||
{ type: 'text' as const, text: 'Extract the fields per the schema. JSON only.' },
|
||||
]
|
||||
return { kind: 'pdf', data: input.buffer, fileName: input.fileName }
|
||||
}
|
||||
return [
|
||||
{
|
||||
type: 'image' as const,
|
||||
source: {
|
||||
type: 'base64' as const,
|
||||
media_type: input.mimeType as 'image/jpeg' | 'image/png' | 'image/webp' | 'image/gif',
|
||||
data: base64,
|
||||
return { kind: 'image', data: input.buffer, mediaType: input.mimeType as AiImageMediaType }
|
||||
}
|
||||
|
||||
// Hand-maintained JSON-schema mirror of ExtractionSchema, used ONLY when the
|
||||
// operator opts into strict JSON mode on an OpenAI-compatible endpoint
|
||||
// (AI_STRICT_JSON=true). Deliberately not auto-converted from the Zod schema:
|
||||
// .catch()/.transform() have no JSON-schema equivalent and the conversion
|
||||
// would drift silently. Permissive on purpose (types + nullability only):
|
||||
// Zod stays the validator, this just keeps the model inside the shape.
|
||||
const nullable = (type: string) => ({ type: [type, 'null'] })
|
||||
const EXTRACTION_JSON_SCHEMA: Record<string, unknown> = {
|
||||
type: 'object',
|
||||
additionalProperties: false,
|
||||
properties: {
|
||||
documentKind: nullable('string'),
|
||||
merchantCategory: nullable('string'),
|
||||
legibility: nullable('string'),
|
||||
purchaseTime: nullable('string'),
|
||||
payment: {
|
||||
type: ['object', 'null'],
|
||||
additionalProperties: false,
|
||||
properties: { method: nullable('string'), cardLast4: nullable('string') },
|
||||
required: ['method', 'cardLast4'],
|
||||
},
|
||||
supplier: {
|
||||
type: 'object',
|
||||
additionalProperties: false,
|
||||
properties: {
|
||||
name: nullable('string'),
|
||||
orgNumber: nullable('string'),
|
||||
vatNumber: nullable('string'),
|
||||
address: nullable('string'),
|
||||
bankgiro: nullable('string'),
|
||||
plusgiro: nullable('string'),
|
||||
},
|
||||
required: ['name', 'orgNumber', 'vatNumber', 'address', 'bankgiro', 'plusgiro'],
|
||||
},
|
||||
invoice: {
|
||||
type: 'object',
|
||||
additionalProperties: false,
|
||||
properties: {
|
||||
invoiceNumber: nullable('string'),
|
||||
invoiceDate: nullable('string'),
|
||||
dueDate: nullable('string'),
|
||||
paymentReference: nullable('string'),
|
||||
currency: { type: 'string' },
|
||||
servicePeriodStart: nullable('string'),
|
||||
servicePeriodEnd: nullable('string'),
|
||||
},
|
||||
required: [
|
||||
'invoiceNumber',
|
||||
'invoiceDate',
|
||||
'dueDate',
|
||||
'paymentReference',
|
||||
'currency',
|
||||
'servicePeriodStart',
|
||||
'servicePeriodEnd',
|
||||
],
|
||||
},
|
||||
lineItems: {
|
||||
type: 'array',
|
||||
items: {
|
||||
type: 'object',
|
||||
additionalProperties: false,
|
||||
properties: {
|
||||
description: { type: 'string' },
|
||||
quantity: { type: 'number' },
|
||||
unitPrice: nullable('number'),
|
||||
lineTotal: { type: 'number' },
|
||||
vatRate: nullable('number'),
|
||||
accountSuggestion: { type: 'null' },
|
||||
},
|
||||
required: ['description', 'quantity', 'unitPrice', 'lineTotal', 'vatRate', 'accountSuggestion'],
|
||||
},
|
||||
},
|
||||
{ type: 'text' as const, text: 'Extract the fields per the schema. JSON only.' },
|
||||
]
|
||||
totals: {
|
||||
type: 'object',
|
||||
additionalProperties: false,
|
||||
properties: {
|
||||
subtotal: nullable('number'),
|
||||
vatAmount: nullable('number'),
|
||||
total: nullable('number'),
|
||||
roundingAmount: nullable('number'),
|
||||
},
|
||||
required: ['subtotal', 'vatAmount', 'total', 'roundingAmount'],
|
||||
},
|
||||
vatBreakdown: {
|
||||
type: 'array',
|
||||
items: {
|
||||
type: 'object',
|
||||
additionalProperties: false,
|
||||
properties: { rate: { type: 'number' }, base: { type: 'number' }, amount: { type: 'number' } },
|
||||
required: ['rate', 'base', 'amount'],
|
||||
},
|
||||
},
|
||||
},
|
||||
required: [
|
||||
'documentKind',
|
||||
'merchantCategory',
|
||||
'legibility',
|
||||
'purchaseTime',
|
||||
'payment',
|
||||
'supplier',
|
||||
'invoice',
|
||||
'lineItems',
|
||||
'totals',
|
||||
'vatBreakdown',
|
||||
],
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract invoice fields by sending the document directly to Claude
|
||||
* Sonnet 4.6 via AWS Bedrock. Never throws on extraction failure:
|
||||
* always returns an InvoiceExtractionResult. Empty fields are null.
|
||||
* Extract invoice fields by sending the document to the configured AI
|
||||
* backend. Never throws on extraction failure: always returns an
|
||||
* InvoiceExtractionResult. Empty fields are null. `skipped` is set when no
|
||||
* model call was made (unsupported file type, AI not configured, no vision
|
||||
* on this backend, PDF rasterizer missing), so callers can tell "nothing to
|
||||
* read" from "read and found nothing".
|
||||
*/
|
||||
export async function extractInvoiceFields(
|
||||
rawInput: ExtractionInput
|
||||
@@ -474,63 +515,48 @@ export async function extractInvoiceFields(
|
||||
const input = await normalizeImageForExtraction(rawInput)
|
||||
|
||||
if (!SUPPORTED_MEDIA_TYPES.has(input.mimeType)) {
|
||||
return { data: emptyResult(), rawText: null }
|
||||
return { data: emptyResult(), rawText: null, skipped: 'unsupported_media' }
|
||||
}
|
||||
|
||||
if (!hasAiCredentials()) {
|
||||
log.warn('AI credentials missing: returning empty extraction', {
|
||||
file_name_hash: createHash('sha256').update(input.fileName).digest('hex').slice(0, 12),
|
||||
})
|
||||
return { data: emptyResult(), rawText: null }
|
||||
}
|
||||
|
||||
const client = createAiClient()
|
||||
const fileNameHash = createHash('sha256').update(input.fileName).digest('hex').slice(0, 12)
|
||||
const service = getAiService()
|
||||
|
||||
let rawText: string | null = null
|
||||
let model: string | null = null
|
||||
try {
|
||||
// SYSTEM_PROMPT is byte-stable per deploy and ~3.5 KB: marking it as
|
||||
// ephemeral lets Bedrock reuse the prompt-cache on rapid sequential
|
||||
// extractions (e.g. a user uploading a stack of receipts within minutes).
|
||||
// Bedrock supports `{ type: 'ephemeral' }` with the default short TTL;
|
||||
// the 1h TTL from the agent-native API plan (item 10) requires the direct
|
||||
// Anthropic API rather than Bedrock and is out of scope here.
|
||||
const resp = await client.messages.create({
|
||||
model: MODEL,
|
||||
max_tokens: MAX_TOKENS,
|
||||
system: [{ type: 'text', text: SYSTEM_PROMPT, cache_control: { type: 'ephemeral' } }],
|
||||
messages: [{ role: 'user', content: buildContent(input) }],
|
||||
const result = await service.extractFromDocument({
|
||||
document: toDocumentInput(input),
|
||||
system: SYSTEM_PROMPT,
|
||||
instruction: EXTRACTION_INSTRUCTION,
|
||||
maxTokens: readAiConfig().extractionMaxTokens,
|
||||
jsonSchema: EXTRACTION_JSON_SCHEMA,
|
||||
})
|
||||
|
||||
rawText = resp.content
|
||||
.flatMap((b) => (b.type === 'text' ? [b.text] : []))
|
||||
.join('')
|
||||
.trim()
|
||||
if (!result.ok) {
|
||||
// Not a failure: the deployment cannot read this document at all.
|
||||
// `ai_unconfigured` is the self-host "no key yet" case the 30 s
|
||||
// upload hang used to hide; the others are honest capability gaps.
|
||||
log.warn('AI extraction skipped', { file_name_hash: fileNameHash, reason: result.skipped })
|
||||
return { data: emptyResult(), rawText: null, skipped: result.skipped }
|
||||
}
|
||||
rawText = result.text
|
||||
model = result.model
|
||||
|
||||
// Observability for the prompt-cache hit ratio. The agent-native plan
|
||||
// targets cache_read_input_tokens / total_input_tokens ≥ 0.85 in steady
|
||||
// state; logging here makes that measurable without a separate dashboard.
|
||||
const usage = resp.usage as
|
||||
| {
|
||||
input_tokens?: number
|
||||
output_tokens?: number
|
||||
cache_creation_input_tokens?: number
|
||||
cache_read_input_tokens?: number
|
||||
}
|
||||
| undefined
|
||||
if (usage) {
|
||||
// Raw fileName can constitute personal data (e.g. "faktura_Sven_Andersson.pdf").
|
||||
// Log a short hash so the operator can correlate without exposing PII
|
||||
// to the log destination (GDPR Art. 5(1)(f)).
|
||||
const fileNameHash = createHash('sha256').update(input.fileName).digest('hex').slice(0, 12)
|
||||
log.info('ai_extraction_usage', {
|
||||
file_name_hash: fileNameHash,
|
||||
mime_type: input.mimeType,
|
||||
input_tokens: usage.input_tokens ?? null,
|
||||
output_tokens: usage.output_tokens ?? null,
|
||||
cache_creation_input_tokens: usage.cache_creation_input_tokens ?? null,
|
||||
cache_read_input_tokens: usage.cache_read_input_tokens ?? null,
|
||||
})
|
||||
}
|
||||
// Raw fileName can constitute personal data (e.g. "faktura_Sven_Andersson.pdf").
|
||||
// Log a short hash so the operator can correlate without exposing PII
|
||||
// to the log destination (GDPR Art. 5(1)(f)).
|
||||
log.info('ai_extraction_usage', {
|
||||
file_name_hash: fileNameHash,
|
||||
mime_type: input.mimeType,
|
||||
model,
|
||||
input_tokens: result.usage.inputTokens,
|
||||
output_tokens: result.usage.outputTokens,
|
||||
cache_creation_input_tokens: result.usage.cacheCreationInputTokens,
|
||||
cache_read_input_tokens: result.usage.cacheReadInputTokens,
|
||||
...(result.pagesRasterized ? { pages_rasterized: result.pagesRasterized } : {}),
|
||||
})
|
||||
|
||||
const parsed = JSON.parse(extractJsonObject(rawText))
|
||||
const validated = ExtractionSchema.parse(parsed)
|
||||
@@ -540,14 +566,15 @@ export async function extractInvoiceFields(
|
||||
// .transform, so no post-validation coercion is needed.
|
||||
data: { ...validated, confidence: 1 },
|
||||
rawText,
|
||||
model,
|
||||
}
|
||||
} catch (err) {
|
||||
log.warn('AI extraction failed', {
|
||||
file_name_hash: createHash('sha256').update(input.fileName).digest('hex').slice(0, 12),
|
||||
file_name_hash: fileNameHash,
|
||||
mimeType: input.mimeType,
|
||||
error: err instanceof Error ? err.message : String(err),
|
||||
hasRawText: rawText != null,
|
||||
})
|
||||
return { data: emptyResult(), rawText }
|
||||
return { data: emptyResult(), rawText, model }
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
import { createServiceClientNoCookies } from '@/lib/auth/api-keys'
|
||||
import { createLogger } from '@/lib/logger'
|
||||
import type { InvoiceExtractionResult } from '@/types'
|
||||
|
||||
const log = createLogger('invoice-inbox-mirror')
|
||||
|
||||
export interface ExtractionOutcome {
|
||||
/** The parsed result (may be the empty skeleton). Ignored unless `rawText` is set and nothing was skipped. */
|
||||
data: InvoiceExtractionResult | null
|
||||
/** Raw model output; null when the call failed or was skipped. */
|
||||
rawText: string | null
|
||||
/** Provider-form model id that answered, when a call was made. */
|
||||
model?: string | null
|
||||
/** Why no model call was made: inbox skip reasons and lib/ai skip reasons alike. */
|
||||
skipped?: string | null
|
||||
}
|
||||
|
||||
/**
|
||||
* Mirror an inbox extraction outcome onto document_attachments
|
||||
* (extracted_data / extracted_at / extraction_model).
|
||||
*
|
||||
* The invoice inbox owns extraction for the documents it ingests, and the
|
||||
* document-extraction extension yields to it (see extractionOwner on the
|
||||
* document.uploaded event). Everything that read the document row before
|
||||
* (the extraction-status poll behind the upload UI, the agent intents'
|
||||
* "what do we already know" reads) keeps working because the inbox writes
|
||||
* the same columns the extension would have, with the one model call it
|
||||
* actually made.
|
||||
*
|
||||
* Service-role client, same as the extension: this runs from routes, the
|
||||
* deferred worker and the MCP server alike, and the write is a system
|
||||
* side-effect rather than a user action. Never throws: a failed mirror
|
||||
* leaves the document row unstamped, which the UI already tolerates.
|
||||
*/
|
||||
export async function mirrorExtractionToDocument(
|
||||
documentId: string,
|
||||
outcome: ExtractionOutcome
|
||||
): Promise<void> {
|
||||
try {
|
||||
const succeeded = !outcome.skipped && outcome.rawText != null && outcome.data != null
|
||||
const extractionModel = outcome.skipped
|
||||
? `skipped:${outcome.skipped}`
|
||||
: succeeded
|
||||
? outcome.model || 'invoice-inbox'
|
||||
: 'failed:no_raw_text'
|
||||
const supabase = createServiceClientNoCookies()
|
||||
const { error } = await supabase
|
||||
.from('document_attachments')
|
||||
.update({
|
||||
extracted_data: succeeded ? (outcome.data as unknown as Record<string, unknown>) : null,
|
||||
extracted_at: new Date().toISOString(),
|
||||
extraction_model: extractionModel,
|
||||
})
|
||||
.eq('id', documentId)
|
||||
if (error) {
|
||||
log.warn('mirror failed', { doc: documentId, extractionModel, err: error.message })
|
||||
}
|
||||
} catch (err) {
|
||||
log.warn('mirror threw', {
|
||||
doc: documentId,
|
||||
err: err instanceof Error ? err.message : String(err),
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,8 @@
|
||||
import { after } from 'next/server'
|
||||
import { uploadDocument } from '@/lib/core/documents/document-service'
|
||||
import { extractInvoiceFields, emptyResult } from './extract-invoice-fields'
|
||||
import { mirrorExtractionToDocument } from './mirror-extraction'
|
||||
import { getAiStatus } from '@/lib/ai'
|
||||
import { hasCapability } from '@/lib/entitlements/has-capability'
|
||||
import { CAPABILITY } from '@/lib/entitlements/keys'
|
||||
import { appendProcessingHistory } from '@/lib/processing-history/append'
|
||||
@@ -10,6 +12,16 @@ import type { InvoiceExtractionResult } from '@/types'
|
||||
import { PDFDocument } from 'pdf-lib'
|
||||
import path from 'node:path'
|
||||
|
||||
// Verdicts decidable before touching the file. `ai_unconfigured` is the
|
||||
// deployment-level "no AI backend" state (self-host without a key), distinct
|
||||
// from the per-company paywall (`no_ai_entitlement`).
|
||||
type SyncSkipReason =
|
||||
| 'no_ai_entitlement'
|
||||
| 'client_opt_out'
|
||||
| 'sandbox'
|
||||
| 'ai_unconfigured'
|
||||
| null
|
||||
|
||||
/**
|
||||
* Defensive filename sanitisation for content arriving from .eml inner
|
||||
* attachments and rejected-attachment metadata. The document-service already
|
||||
@@ -231,6 +243,9 @@ export async function uploadAndExtract(
|
||||
// two inboxes, re-hunted by a sweep, or uploaded twice must not become a
|
||||
// second archived document.
|
||||
dedupeByContent: true,
|
||||
// This function extracts (sync or deferred) and mirrors the outcome onto
|
||||
// the document row; the document-extraction extension must not race it.
|
||||
extractionOwner: 'invoice-inbox',
|
||||
})
|
||||
|
||||
if (doc.deduplicated) {
|
||||
@@ -342,14 +357,20 @@ export async function uploadAndExtract(
|
||||
// decidable without touching the PDF; too_many_pages is not (it only fires
|
||||
// when the slice fallback fails) and is resolved below, on whichever path
|
||||
// (sync or deferred) actually attempts the slice.
|
||||
const syncSkipReason: 'no_ai_entitlement' | 'client_opt_out' | 'sandbox' | null =
|
||||
// `ai_unconfigured` (no AI credentials/model on this deployment, the
|
||||
// self-host "key not set yet" state) is decided up front as well: a call
|
||||
// that can never succeed must not take the deferred path and leave the row
|
||||
// in 'processing' until the sweep.
|
||||
const syncSkipReason: SyncSkipReason =
|
||||
!hasAiEntitlement
|
||||
? 'no_ai_entitlement'
|
||||
: sandbox
|
||||
? 'sandbox'
|
||||
: opts.skipExtraction
|
||||
? 'client_opt_out'
|
||||
: null
|
||||
: !getAiStatus().configured
|
||||
? 'ai_unconfigured'
|
||||
: null
|
||||
|
||||
if (opts.deferExtraction && syncSkipReason === null) {
|
||||
// Staged path: extraction WILL call Bedrock, so create the row now and
|
||||
@@ -424,7 +445,7 @@ export async function uploadAndExtract(
|
||||
// page-count. Opt-out outranks the page gate (an opted-out caller never
|
||||
// extracts regardless of length), and too_many_pages only fires when the
|
||||
// slice fallback also failed (encrypted/malformed PDF).
|
||||
const skipReason: 'no_ai_entitlement' | 'too_many_pages' | 'client_opt_out' | 'sandbox' | null =
|
||||
const skipReason: SyncSkipReason | 'too_many_pages' =
|
||||
syncSkipReason ??
|
||||
(gatedByPageCount && slicedBuffer == null ? 'too_many_pages' : null)
|
||||
const skipExtraction = skipReason !== null
|
||||
@@ -434,13 +455,14 @@ export async function uploadAndExtract(
|
||||
// fields via /items/:id/extracted-data before converting to a supplier
|
||||
// invoice. extracted_data is never null in the DB; an empty skeleton
|
||||
// keeps downstream readers (UI, MCP) happy.
|
||||
const { data: extracted, rawText } = skipExtraction
|
||||
? { data: emptyResult(), rawText: null }
|
||||
const extraction = skipExtraction
|
||||
? { data: emptyResult(), rawText: null, model: null, skipped: null }
|
||||
: await extractInvoiceFields({
|
||||
buffer: Buffer.from(slicedBuffer ?? file.buffer),
|
||||
mimeType: file.type,
|
||||
fileName: file.name,
|
||||
})
|
||||
const { data: extracted, rawText } = extraction
|
||||
if (!skipExtraction && slicedBuffer != null && pageCount != null) {
|
||||
extracted.pages = { total: pageCount, analyzed: MAX_PAGES_FOR_AUTO_EXTRACT }
|
||||
}
|
||||
@@ -484,6 +506,15 @@ export async function uploadAndExtract(
|
||||
|
||||
if (inboxError) throw new Error(`Failed to create inbox item: ${inboxError.message}`)
|
||||
|
||||
// The extension yielded to us (extractionOwner); put the outcome where it
|
||||
// would have written it, so the document row tells the same story.
|
||||
await mirrorExtractionToDocument(doc.id, {
|
||||
data: extracted,
|
||||
rawText,
|
||||
model: extraction.model ?? null,
|
||||
skipped: skipReason ?? extraction.skipped ?? null,
|
||||
})
|
||||
|
||||
try {
|
||||
await appendProcessingHistory({
|
||||
companyId,
|
||||
@@ -552,8 +583,9 @@ function scheduleDeferredExtraction(job: DeferredExtractionJob): void {
|
||||
// an unsliceable long PDF becomes the too_many_pages skip.
|
||||
let extracted: InvoiceExtractionResult = emptyResult()
|
||||
let rawText: string | null = null
|
||||
let model: string | null = null
|
||||
let extractionSkipped = false
|
||||
let skipReason: 'too_many_pages' | null = null
|
||||
let skipReason: string | null = null
|
||||
try {
|
||||
const slicedBuffer = job.gatedByPageCount
|
||||
? await slicePdfForExtraction(job.file.buffer, MAX_PAGES_FOR_AUTO_EXTRACT)
|
||||
@@ -569,6 +601,14 @@ function scheduleDeferredExtraction(job: DeferredExtractionJob): void {
|
||||
})
|
||||
extracted = result.data
|
||||
rawText = result.rawText
|
||||
model = result.model ?? null
|
||||
if (result.skipped) {
|
||||
// No model call could be made (AI unconfigured, no vision on this
|
||||
// backend, rasterizer missing). Same UI affordance as the other
|
||||
// skips: the row lands with the empty skeleton and the hint.
|
||||
extractionSkipped = true
|
||||
skipReason = result.skipped
|
||||
}
|
||||
if (slicedBuffer != null && job.pageCount != null) {
|
||||
extracted.pages = { total: job.pageCount, analyzed: MAX_PAGES_FOR_AUTO_EXTRACT }
|
||||
}
|
||||
@@ -608,6 +648,13 @@ function scheduleDeferredExtraction(job: DeferredExtractionJob): void {
|
||||
}
|
||||
if (!Array.isArray(claimed) || claimed.length === 0) return
|
||||
|
||||
await mirrorExtractionToDocument(job.documentId, {
|
||||
data: extracted,
|
||||
rawText,
|
||||
model,
|
||||
skipped: skipReason,
|
||||
})
|
||||
|
||||
try {
|
||||
await appendProcessingHistory({
|
||||
companyId: job.companyId,
|
||||
@@ -646,6 +693,7 @@ function scheduleDeferredExtraction(job: DeferredExtractionJob): void {
|
||||
})
|
||||
.eq('id', job.itemId)
|
||||
.eq('status', 'processing')
|
||||
await mirrorExtractionToDocument(job.documentId, { data: null, rawText: null })
|
||||
} catch {
|
||||
// The sweep cron is the recovery of last resort.
|
||||
}
|
||||
|
||||
@@ -129,6 +129,10 @@ describe('MCP model-free document upload tools', () => {
|
||||
uploadId,
|
||||
'invoice.pdf',
|
||||
'application/pdf',
|
||||
undefined,
|
||||
// The inbox item created right after owns extraction; the
|
||||
// document-extraction extension must yield on the uploaded event.
|
||||
{ extractionOwner: 'invoice-inbox' },
|
||||
)
|
||||
expect(inboxInsert.insert).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ id: uploadId, document_id: uploadId }),
|
||||
|
||||
@@ -210,6 +210,7 @@ import {
|
||||
MAX_DOCUMENT_SIZE,
|
||||
} from '@/lib/core/documents/document-service'
|
||||
import { extractInvoiceFields, ExtractionSchema as InvoiceExtractionSchema, AgentExtractionSchema } from '@/extensions/general/invoice-inbox/lib/extract-invoice-fields'
|
||||
import { mirrorExtractionToDocument } from '@/extensions/general/invoice-inbox/lib/mirror-extraction'
|
||||
// Skatteverket filing tools (PR5). Cross-extension lib import, same sanctioned
|
||||
// pattern as invoice-inbox above: the CI guard only checks lib/, app/api/,
|
||||
// components/. The two submit tools stage ops whose commit dispatches back into
|
||||
@@ -497,7 +498,17 @@ async function createDocumentInboxItem(
|
||||
if (existing) return existing
|
||||
}
|
||||
|
||||
const { data: extracted } = await extractInvoiceFields({ buffer, mimeType, fileName })
|
||||
const extraction = await extractInvoiceFields({ buffer, mimeType, fileName })
|
||||
const { data: extracted } = extraction
|
||||
// uploadDocument()/completePendingDocumentUpload() were told this inbox
|
||||
// item owns extraction, so the document-extraction extension yielded;
|
||||
// mirror the outcome onto the document row in its place.
|
||||
await mirrorExtractionToDocument(documentId, {
|
||||
data: extracted,
|
||||
rawText: extraction.rawText,
|
||||
model: extraction.model ?? null,
|
||||
skipped: extraction.skipped ?? null,
|
||||
})
|
||||
|
||||
const matchedSupplierId = await matchSupplierId(supabase, companyId, extracted.supplier)
|
||||
|
||||
@@ -9777,6 +9788,8 @@ export const tools: McpTool[] = [
|
||||
uploadId,
|
||||
fileName,
|
||||
mimeType,
|
||||
undefined,
|
||||
{ extractionOwner: 'invoice-inbox' },
|
||||
)
|
||||
return createDocumentInboxItem(
|
||||
supabase,
|
||||
@@ -9837,7 +9850,7 @@ export const tools: McpTool[] = [
|
||||
name: fileName,
|
||||
buffer: buffer.buffer.slice(buffer.byteOffset, buffer.byteOffset + buffer.byteLength),
|
||||
type: mimeType,
|
||||
}, { upload_source: 'api' })
|
||||
}, { upload_source: 'api', extractionOwner: 'invoice-inbox' })
|
||||
return createDocumentInboxItem(
|
||||
supabase,
|
||||
companyId,
|
||||
|
||||
Reference in New Issue
Block a user