From 524d9978f127fe3e20aa69e880c0dce1a874b68b Mon Sep 17 00:00:00 2001 From: Jakob Wennberg Date: Fri, 21 Aug 2026 15:28:37 +0200 Subject: [PATCH] fix(migration): resumable underlag import without inline extraction + same-origin MCP storage URLs (#1783) * fix(migration): resumable underlag import without inline extraction, same-origin MCP storage URLs The Fortnox underlag import ran every file's AI extraction inline inside one request and hit the hosted 300 s function limit after ~17 of 113 files (twice on 2026-08-21); the UI showed the generic "underlagen kunde inte importeras" although the files it did reach were linked. The import now works in time-budgeted slices with a stable cursor (the UI loops until the server reports the end and shows "x av y") and opts out of extraction (extractionOwner 'none', stamped skipped:opted_out): every file is linked to its posted verifikat on arrival, so the booking is already known. MCP signed Storage URLs (upload_url, signed_url, download_url) are served through a same-origin proxy, /api/storage/[...path], because Claude Desktop's sandbox only reaches the MCP host and blocked the PUT to .supabase.co. The signed token stays the only credential; the proxy forwards only signed documents-bucket paths to our own Storage host and is a no-op rewrite when NEXT_PUBLIC_APP_URL is unset. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_013YoZ8iboyTj221axW6Gdtm * fix(mcp): keep the storage-proxy note out of the size-capped tool descriptions The per-tool 280-char cap and the tools/list payload ceiling both tripped on the two sentences added to gnubok_create_document_upload and gnubok_get_document_content; the why now lives in a code comment. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_013YoZ8iboyTj221axW6Gdtm * fix(review): id cursor, stall = error, capped upload body, encoded dot segments Review follow-ups on #1783: - the import cursor is the last handled provider attachment id, not an index, so a file Fortnox adds or removes mid-sweep shifts nothing - a partial answer whose cursor does not advance (or the round guard) is reported as ARCIM_DOCUMENT_IMPORT_STALLED instead of "complete"; the slices already landed stay reported and the retry button resumes - the storage proxy reads the PUT body as a capped stream instead of buffering an unbounded payload before measuring it - object paths are rejected when any segment decodes to "." or ".." (or holds a separator), and the URL fetch() would actually request is re-checked against the allowlist after normalisation - download_url description no longer claims a direct Storage URL Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_013YoZ8iboyTj221axW6Gdtm --------- Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com> Co-authored-by: Claude Fable 5 --- DECISIONS.md | 2 + app/api/storage/[...path]/route.ts | 161 +++++++++++++++ app/api/storage/__tests__/route.test.ts | 162 +++++++++++++++ .../general/ArcimMigrationWorkspace.tsx | 24 ++- .../arcim-document-import-flow.test.ts | 133 ++++++++++++ .../general/arcim-document-import-flow.ts | 123 +++++++++++- .../__tests__/import-documents-route.test.ts | 39 ++++ .../__tests__/import-documents.test.ts | 125 ++++++++++++ extensions/general/arcim-migration/index.ts | 20 +- .../arcim-migration/lib/import-documents.ts | 78 ++++++- .../__tests__/handler.test.ts | 7 + .../general/document-extraction/index.ts | 11 +- .../__tests__/document-upload-tools.test.ts | 32 ++- extensions/general/mcp-server/server.ts | 11 +- .../documents/__tests__/storage-proxy.test.ts | 190 ++++++++++++++++++ lib/core/documents/document-service.ts | 8 +- lib/core/documents/storage-proxy.ts | 177 ++++++++++++++++ lib/errors/structured-errors.ts | 27 +++ lib/events/types.ts | 14 +- messages/en.json | 1 + messages/sv.json | 1 + 21 files changed, 1326 insertions(+), 20 deletions(-) create mode 100644 app/api/storage/[...path]/route.ts create mode 100644 app/api/storage/__tests__/route.test.ts create mode 100644 lib/core/documents/__tests__/storage-proxy.test.ts create mode 100644 lib/core/documents/storage-proxy.ts diff --git a/DECISIONS.md b/DECISIONS.md index 94cdb9c1..e317c5aa 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -1144,4 +1144,6 @@ One line per decision: `[YYYY-MM-DD] : `. Appended by agents and [2026-08-21] SCHABLONINTAKT_RATE_BY_CLOSING_YEAR backfilled 2020-2024 (SLR 30 Nov per Riksgalden: -0.09/-0.10/0.23 floored to 0.5 %, 1.94 %, 2.62 %) and the rate now resolves lazily (resolveSchablonintaktRate: 0 when no 212X account carried an opening balance): the table only covered 2025/2026 and the builder consulted it unconditionally, so every AB closing a pre-2025 year got a generic 500 at bokslut step 3 (126 open FY2024 periods on prod, incl. a byra trial). 2019 and earlier stay unmapped on purpose: the 100 %-of-SLR rule keys on beskattningsar STARTING 2019-01-01+ (prop. 2017/18:245), so a 2019 closing can be a brutet ar under the old 72 % factor. Unmapped-year-with-fonder now raises SCHABLONINTAKT_RATE_NOT_CONFIGURED (typed, 500 so runtime-error clustering still flags the missed December update) instead of INTERNAL_ERROR. [2026-08-21] RIP-4 "optimal" auto-booking cascade, Tier 2 = the provider-agnostic account SELECTOR (lib/agent/categorize/select-account.ts), built per the 2026 research (artifact dc0c2760): the model does NOT free-form a categorizer; it CHOOSES from a closed set — the deterministic candidate accounts (Tier 1) + the 19 standard business categories (each maps deterministically to a BAS account via getDefaultAccountForCategory) + "needs_review". So the model can't invent an account, account/VAT stays deterministic and validated, and it runs on any provider (Bedrock or a local model) via getAiService().generateStructured. Founder chose the optimal path (model selects on EVERY transaction, LLM calls are fine), so confidence uses SELF-CONSISTENCY (default 3 samples, majority vote, agreement fraction) combined with the model's stated confidence and floored by the winning candidate's deterministic confidence — never the model's verbalized confidence alone (research: systematically overconfident). reasoning field precedes choice in the schema (reason-before-choice). needs_review is never auto-applied. Calibration of the combined score → the auto-book/suggest/review gate is a later tier. Not yet wired: Tier 1 candidate gathering (counterparty templates + getSuggestedCategories) + a route + the ApprovalCard UI (next PRs). [2026-08-21] RIP-4 cascade Tier 1 (candidate gathering) + the proposal route. lib/agent/categorize/candidates.ts assembles the deterministic candidate slate for a transaction exactly like the gnubok_suggest_categories MCP tool (mapping_rules + per-merchant history via buildMerchantHistory/getSuggestedCategories + the learned counterparty template), NO model call, deduped by account (highest confidence wins) and capped. Suggestions carry no VAT so it derives the category default (getDefaultVatTreatmentForCategory); the counterparty template carries its own. POST /api/agent/categorize runs Tier 1 → Tier 2 selectAccount and returns the proposal + the candidate slate; it NEVER posts (the caller renders an approval card). Gated on getAiStatus().configured (any provider incl. local), same gates as /api/agent/ask (auth via requireAuth, rate, membership, sandbox, capability). Next: wire the transaction row to this route + the ApprovalCard (UI, visual sign-off), then calibration + the auto-book gate. +[2026-08-21] Provider underlag import (arcim-migration /import-documents) is now time-budgeted + resumable (200 s slice, stable provider-id order, `cursor`/`nextCursor`/`partial`, UI loops with "x av y") and opts out of AI extraction (`extractionOwner: 'none'`, stamped `skipped:opted_out`): Fabian's 113-file Fortnox import ran every file's Sonnet extraction inline inside one request, hit the hosted 300 s function limit after ~17 files twice (09:02 and 10:49 UTC) and the UI showed the generic "underlagen kunde inte importeras". The files are linked to posted verifikat on arrival, so the extraction bought nothing and cost a model call per file; chunking rather than raising maxDuration because a 2,000-file archive would still not fit. Opt-out is per-call (not "skip whenever journal_entry_id is set") to keep the diff scoped. +[2026-08-21] MCP signed Storage URLs (gnubok_create_document_upload upload_url, gnubok_get_document_content signed_url, audit-package download_url) are served through a same-origin proxy, /api/storage/[...path] → .supabase.co/storage/v1/object/{sign,upload/sign}/documents/..., instead of a Next rewrite: Claude Desktop's sandbox only reaches the MCP host (app.accounted.se) and blocked the PUT to supabase.co (Fabian, 2026-08-21). A route handler reads NEXT_PUBLIC_SUPABASE_URL at runtime (rewrites bake at build, which breaks the Docker image), is deliberately NOT withRouteContext (the signed token is the only credential, validated by Storage per object path; the proxy forwards only signed documents-bucket paths to our own host) and is a no-op rewrite when NEXT_PUBLIC_APP_URL is unset so a self-host never gets a localhost link. [2026-08-21] RIP-4 cascade step 3 (UI): the AI booking proposal is surfaced INSIDE the existing QuickReviewDialog rather than a new inline-row card, so it reuses that dialog's proven, deterministic, balanced commit path (POST /api/transactions/[id]/categorize) instead of a parallel one. components/transactions/AiCategorizeProposal.tsx fetches POST /api/agent/categorize on dialog open (keyed on tx.id so it remounts per transaction), pre-fills accountOverride + vatTreatment via handleAccountChange (class-2 VAT clearing preserved), and shows the confidence band (säker/trolig/välj konto) + "Varför" + the candidate alternatives (click to re-apply). Falls back SILENTLY to the deterministic defaults on error, and shows a soft note on 503 (ai_unconfigured) — the dialog always works without AI. NO silent auto-posting (founder call, avoids the storno-on-undo mess): "säker" = pre-filled, one-tap Bokför via the dialog's existing button; true hands-off auto-book waits for calibration. i18n: strings inline Swedish for now (assistant surface), lift to messages/{sv,en}.json before final merge. Confidence bands (0.8/0.5) are placeholders until calibration. Needs founder visual sign-off before merge ([[project_nav_ia_redesign]]). diff --git a/app/api/storage/[...path]/route.ts b/app/api/storage/[...path]/route.ts new file mode 100644 index 00000000..c5fe4602 --- /dev/null +++ b/app/api/storage/[...path]/route.ts @@ -0,0 +1,161 @@ +import { createLogger } from '@/lib/logger' +import { errorResponseFromCode } from '@/lib/errors/get-structured-error' +import { + STORAGE_PROXY_ROUTE, + readBodyWithCap, + resolveUpstreamStorageUrl, +} from '@/lib/core/documents/storage-proxy' + +/** + * /api/storage/[...path]: same-origin proxy for signed Supabase Storage URLs. + * + * Why: agent sandboxes (Claude Desktop's code execution among them) only let + * traffic out to the host the MCP server runs on. Signed Storage URLs point + * at .supabase.co, so the MCP model-free upload (PUT bytes to + * upload_url) and signed document downloads were blocked inside such + * sandboxes. The MCP tools now hand out URLs under this route instead (see + * lib/core/documents/storage-proxy.ts), and this handler forwards them to + * Storage unchanged. + * + * Auth: deliberately NOT withRouteContext. The caller is a sandbox with no + * session, and the signed token in the query string is the credential: + * Storage validates it against the exact object path on every request, the + * same way it would on the public Storage host. The proxy only forwards + * signed object paths on the documents bucket, only to our own Storage + * origin, with a token present; it never mints, reads or relays any key. + * + * GET/HEAD /api/storage/sign/documents/?token=... download + * PUT /api/storage/upload/sign/documents/?token=... upload + */ + +export const dynamic = 'force-dynamic' +export const maxDuration = 60 + +const log = createLogger('api/storage-proxy') + +/** Above every caller's own cap (MCP upload 10 MB, document max 20 MB). */ +const MAX_UPLOAD_BYTES = 50 * 1024 * 1024 + +const REQUEST_HEADERS_FORWARDED = [ + 'content-type', + 'cache-control', + 'x-upsert', + 'range', + 'if-none-match', + 'if-modified-since', +] as const + +const RESPONSE_HEADERS_FORWARDED = [ + 'content-type', + 'content-length', + 'content-disposition', + 'content-range', + 'accept-ranges', + 'cache-control', + 'etag', + 'last-modified', +] as const + +// The sandbox fetch is not a browser, but a browser-based MCP client would +// need these and they cost nothing: the token still gates every request. +const CORS_HEADERS: Record = { + 'Access-Control-Allow-Origin': '*', + 'Access-Control-Allow-Methods': 'GET, HEAD, PUT, OPTIONS', + 'Access-Control-Allow-Headers': 'Content-Type, Cache-Control, x-upsert, Range', + 'Access-Control-Max-Age': '600', +} + +function withCors(response: Response): Response { + for (const [key, value] of Object.entries(CORS_HEADERS)) { + response.headers.set(key, value) + } + return response +} + +function objectPathOf(request: Request): string { + const { pathname } = new URL(request.url) + const prefix = `${STORAGE_PROXY_ROUTE}/` + return pathname.startsWith(prefix) ? pathname.slice(prefix.length) : '' +} + +function rejected(reason: 'unsupported_path' | 'missing_token' | 'storage_unconfigured') { + const code = + reason === 'unsupported_path' + ? 'STORAGE_PROXY_UNSUPPORTED_PATH' + : reason === 'missing_token' + ? 'STORAGE_PROXY_TOKEN_REQUIRED' + : 'STORAGE_PROXY_UNCONFIGURED' + return withCors(errorResponseFromCode(code, log)) +} + +async function proxy(request: Request, method: 'GET' | 'HEAD' | 'PUT'): Promise { + const url = new URL(request.url) + const resolved = resolveUpstreamStorageUrl(objectPathOf(request), url.searchParams) + if (!resolved.ok) return rejected(resolved.reason) + + const headers = new Headers() + for (const name of REQUEST_HEADERS_FORWARDED) { + const value = request.headers.get(name) + if (value) headers.set(name, value) + } + + let body: ArrayBuffer | undefined + if (method === 'PUT') { + const declared = Number(request.headers.get('content-length') ?? '0') + if (Number.isFinite(declared) && declared > MAX_UPLOAD_BYTES) { + return withCors(errorResponseFromCode('STORAGE_PROXY_BODY_TOO_LARGE', log)) + } + // Stream with a cap: the declared length is advisory (absent or wrong), + // and the token is only validated upstream, so never buffer an unbounded + // body before measuring it. + const read = await readBodyWithCap(request.body, MAX_UPLOAD_BYTES) + if (read === null) { + return withCors(errorResponseFromCode('STORAGE_PROXY_BODY_TOO_LARGE', log)) + } + body = read + if (!headers.has('content-type')) headers.set('content-type', 'application/octet-stream') + } + + let upstream: Response + try { + upstream = await fetch(resolved.url, { + method, + headers, + ...(body ? { body } : {}), + redirect: 'manual', + cache: 'no-store', + }) + } catch (error) { + log.error('storage proxy upstream request failed', error as Error, { method }) + return withCors(errorResponseFromCode('STORAGE_PROXY_UPSTREAM_UNAVAILABLE', log)) + } + + const responseHeaders = new Headers(CORS_HEADERS) + for (const name of RESPONSE_HEADERS_FORWARDED) { + const value = upstream.headers.get(name) + if (value) responseHeaders.set(name, value) + } + // Never let a served document be sniffed into something executable. + responseHeaders.set('X-Content-Type-Options', 'nosniff') + + return new Response(method === 'HEAD' ? null : upstream.body, { + status: upstream.status, + headers: responseHeaders, + }) +} + +export async function GET(request: Request) { + return proxy(request, 'GET') +} + +export async function HEAD(request: Request) { + return proxy(request, 'HEAD') +} + +export async function PUT(request: Request) { + return proxy(request, 'PUT') +} + +export async function OPTIONS() { + return new Response(null, { status: 204, headers: CORS_HEADERS }) +} diff --git a/app/api/storage/__tests__/route.test.ts b/app/api/storage/__tests__/route.test.ts new file mode 100644 index 00000000..3092b93a --- /dev/null +++ b/app/api/storage/__tests__/route.test.ts @@ -0,0 +1,162 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { GET, HEAD, OPTIONS, PUT } from '../[...path]/route' + +const SUPABASE = 'https://pwxtzglxptnnvjrpixpg.supabase.co' + +const fetchMock = vi.fn() + +beforeEach(() => { + vi.stubEnv('NEXT_PUBLIC_SUPABASE_URL', SUPABASE) + vi.stubEnv('NEXT_PUBLIC_APP_URL', 'https://app.accounted.se') + vi.stubGlobal('fetch', fetchMock) + fetchMock.mockReset() +}) + +afterEach(() => { + vi.unstubAllEnvs() + vi.unstubAllGlobals() +}) + +function upstreamResponse(body: string | null, init: ResponseInit = {}) { + return new Response(body, { + status: 200, + headers: { 'content-type': 'application/pdf', 'content-length': String(body?.length ?? 0) }, + ...init, + }) +} + +describe('/api/storage/[...path] same-origin Storage proxy', () => { + it('forwards a signed upload PUT (bytes, content-type, token) to our Storage host', async () => { + fetchMock.mockResolvedValue(new Response(JSON.stringify({ Key: 'documents/x' }), { status: 200, headers: { 'content-type': 'application/json' } })) + const bytes = new TextEncoder().encode('%PDF-1.4 hello') + const request = new Request( + 'https://app.accounted.se/api/storage/upload/sign/documents/co-1/user-1/pending/up-1/faktura%20maj.pdf?token=eyJ.sig', + { method: 'PUT', headers: { 'content-type': 'application/pdf', 'x-upsert': 'false' }, body: bytes }, + ) + + const response = await PUT(request) + + expect(response.status).toBe(200) + expect(response.headers.get('access-control-allow-origin')).toBe('*') + expect(fetchMock).toHaveBeenCalledTimes(1) + const [url, init] = fetchMock.mock.calls[0] as [string, RequestInit] + expect(url).toBe( + `${SUPABASE}/storage/v1/object/upload/sign/documents/co-1/user-1/pending/up-1/faktura%20maj.pdf?token=eyJ.sig`, + ) + expect(init.method).toBe('PUT') + const headers = init.headers as Headers + expect(headers.get('content-type')).toBe('application/pdf') + expect(headers.get('x-upsert')).toBe('false') + expect(new TextDecoder().decode(init.body as ArrayBuffer)).toBe('%PDF-1.4 hello') + }) + + it('streams a signed download GET through and keeps the document headers', async () => { + fetchMock.mockResolvedValue( + upstreamResponse('%PDF-1.4 bytes', { + headers: { + 'content-type': 'application/pdf', + 'content-disposition': 'attachment; filename="kvitto.pdf"', + 'set-cookie': 'leak=1', + }, + }), + ) + const request = new Request( + 'https://app.accounted.se/api/storage/sign/documents/co-1/user-1/kvitto.pdf?token=eyJ.sig', + ) + + const response = await GET(request) + + expect(response.status).toBe(200) + expect(await response.text()).toBe('%PDF-1.4 bytes') + expect(response.headers.get('content-type')).toBe('application/pdf') + expect(response.headers.get('content-disposition')).toBe('attachment; filename="kvitto.pdf"') + expect(response.headers.get('x-content-type-options')).toBe('nosniff') + expect(response.headers.get('set-cookie')).toBeNull() + const [url, init] = fetchMock.mock.calls[0] as [string, RequestInit] + expect(url).toBe(`${SUPABASE}/storage/v1/object/sign/documents/co-1/user-1/kvitto.pdf?token=eyJ.sig`) + expect(init.method).toBe('GET') + }) + + it('answers HEAD without a body and relays the upstream status', async () => { + fetchMock.mockResolvedValue(new Response(null, { status: 404, headers: { 'content-type': 'application/json' } })) + + const response = await HEAD( + new Request('https://app.accounted.se/api/storage/sign/documents/co-1/missing.pdf?token=t', { method: 'HEAD' }), + ) + + expect(response.status).toBe(404) + expect(response.body).toBeNull() + }) + + it('refuses paths outside the signed documents-bucket allowlist without touching Storage', async () => { + const response = await GET( + new Request('https://app.accounted.se/api/storage/public/documents/a.pdf?token=t'), + ) + + expect(response.status).toBe(404) + const body = (await response.json()) as { error: { code: string } } + expect(body.error.code).toBe('STORAGE_PROXY_UNSUPPORTED_PATH') + expect(fetchMock).not.toHaveBeenCalled() + }) + + it('refuses a link without its signed token', async () => { + const response = await GET( + new Request('https://app.accounted.se/api/storage/sign/documents/co-1/a.pdf'), + ) + + expect(response.status).toBe(400) + const body = (await response.json()) as { error: { code: string } } + expect(body.error.code).toBe('STORAGE_PROXY_TOKEN_REQUIRED') + expect(fetchMock).not.toHaveBeenCalled() + }) + + it('rejects an oversized upload before forwarding it', async () => { + const request = new Request( + 'https://app.accounted.se/api/storage/upload/sign/documents/co-1/big.pdf?token=t', + { method: 'PUT', headers: { 'content-length': String(51 * 1024 * 1024) }, body: 'x' }, + ) + + const response = await PUT(request) + + expect(response.status).toBe(413) + expect(fetchMock).not.toHaveBeenCalled() + }) + + it('rejects an oversized upload that lies about (or omits) its content-length, without buffering it all', async () => { + let pulled = 0 + const endless = new ReadableStream({ + pull(controller) { + pulled++ + controller.enqueue(new Uint8Array(1024 * 1024)) + }, + }) + const request = new Request( + 'https://app.accounted.se/api/storage/upload/sign/documents/co-1/big.pdf?token=t', + { method: 'PUT', body: endless, duplex: 'half' } as RequestInit, + ) + + const response = await PUT(request) + + expect(response.status).toBe(413) + expect(fetchMock).not.toHaveBeenCalled() + expect(pulled).toBeLessThan(60) + }) + + it('reports Storage being unreachable as 502 instead of crashing', async () => { + fetchMock.mockRejectedValue(new Error('ECONNRESET')) + + const response = await GET( + new Request('https://app.accounted.se/api/storage/sign/documents/co-1/a.pdf?token=t'), + ) + + expect(response.status).toBe(502) + const body = (await response.json()) as { error: { code: string } } + expect(body.error.code).toBe('STORAGE_PROXY_UPSTREAM_UNAVAILABLE') + }) + + it('answers CORS preflight', async () => { + const response = await OPTIONS() + expect(response.status).toBe(204) + expect(response.headers.get('access-control-allow-methods')).toContain('PUT') + }) +}) diff --git a/components/extensions/general/ArcimMigrationWorkspace.tsx b/components/extensions/general/ArcimMigrationWorkspace.tsx index 42ddc71c..452d2a68 100644 --- a/components/extensions/general/ArcimMigrationWorkspace.tsx +++ b/components/extensions/general/ArcimMigrationWorkspace.tsx @@ -37,6 +37,7 @@ import { parseArcimDocumentOAuthResume, PROVIDER_DOCUMENT_SCOPES_UNAVAILABLE, requestArcimDocumentImport, + runArcimDocumentImportToCompletion, resolveArcimDocumentFollowUpProvider, watchArcimOAuthPopup, type ArcimDocumentImportProblem, @@ -1477,10 +1478,26 @@ function DocumentImportFollowUp({ ? t('ext_arcim_documents_importing') : t('ext_arcim_documents_reconnecting') + // Running totals arrive after each server slice of a real import; the + // dry-run result that sits in state while the first slice runs is not + // progress, so it stays silent. + const progress = + state.phase === 'importing' && state.result && !state.result.dryRun && state.result.total > 0 + ? state.result + : null + return (
{title} {label} + {progress && ( +

+ {t('ext_arcim_documents_import_progress', { + done: progress.scanned, + total: progress.total, + })} +

+ )}
) } @@ -2316,7 +2333,12 @@ export default function ArcimMigrationWorkspace({ const runDocumentImport = useCallback(async (currentConsentId: string) => { dispatchDocumentImport({ type: 'import-started' }) try { - const result = await requestArcimDocumentImport(currentConsentId, false) + // One route call per time-budgeted slice; the helper loops until the + // server reports the end and feeds running totals back for the UI. + const result = await runArcimDocumentImportToCompletion(currentConsentId, { + onProgress: (progress) => + dispatchDocumentImport({ type: 'import-progress', result: progress }), + }) dispatchDocumentImport({ type: 'import-succeeded', result }) } catch (documentError) { dispatchDocumentImport({ diff --git a/components/extensions/general/__tests__/arcim-document-import-flow.test.ts b/components/extensions/general/__tests__/arcim-document-import-flow.test.ts index 24ed65d9..93bc504b 100644 --- a/components/extensions/general/__tests__/arcim-document-import-flow.test.ts +++ b/components/extensions/general/__tests__/arcim-document-import-flow.test.ts @@ -1,5 +1,6 @@ import { describe, expect, it, vi } from 'vitest' import { + ARCIM_DOCUMENT_IMPORT_STALLED, ARCIM_DOCUMENT_OAUTH_RESUME_KEY, INITIAL_ARCIM_DOCUMENT_IMPORT_STATE, PROVIDER_DOCUMENT_SCOPES_REQUIRED, @@ -8,8 +9,10 @@ import { arcimDocumentImportReducer, documentOAuthProblemFromReason, parseArcimDocumentOAuthResume, + mergeArcimDocumentImportResults, requestArcimDocumentImport, resolveArcimDocumentFollowUpProvider, + runArcimDocumentImportToCompletion, watchArcimOAuthPopup, type ArcimDocumentImportResult, } from '../arcim-document-import-flow' @@ -26,10 +29,20 @@ function result( failed: 0, dryRun: true, unmatchedSamples: [], + total: 7, + partial: false, + nextCursor: null, ...overrides, } } +function jsonResponse(body: unknown, status = 200): Response { + return new Response(JSON.stringify(body), { + status, + headers: { 'Content-Type': 'application/json' }, + }) +} + describe('Fortnox document follow-up state', () => { it('offers the prompt after a successful Fortnox migration using the honest found count', () => { const discovering = arcimDocumentImportReducer( @@ -298,3 +311,123 @@ describe('document scope OAuth recovery', () => { vi.useRealTimers() }) }) + +describe('resumable import (one server slice per call)', () => { + it('accepts an older server answer without resume fields as one complete slice', async () => { + const { total: _t, partial: _p, nextCursor: _c, ...legacy } = result({ scanned: 9 }) + const fetcher = vi.fn().mockResolvedValue(jsonResponse({ success: true, result: legacy })) + + const normalized = await requestArcimDocumentImport('consent-1', true, fetcher) + + expect(normalized).toMatchObject({ total: 9, partial: false, nextCursor: null }) + expect(JSON.parse((fetcher.mock.calls[0][1] as RequestInit).body as string)).toEqual({ + consentId: 'consent-1', + dryRun: true, + }) + }) + + it('sends the cursor only when resuming', async () => { + const fetcher = vi.fn().mockResolvedValue(jsonResponse({ success: true, result: result() })) + + await requestArcimDocumentImport('consent-1', false, fetcher, 'file-42') + + expect(JSON.parse((fetcher.mock.calls[0][1] as RequestInit).body as string)).toEqual({ + consentId: 'consent-1', + dryRun: false, + cursor: 'file-42', + }) + }) + + it('sums every slice and keeps the latest total', () => { + const merged = mergeArcimDocumentImportResults( + result({ scanned: 17, linked: 15, skipped: 1, unmatched: 1, failed: 0, total: 113, unmatchedSamples: [{ uploadId: 'u1', voucher: 'A1', date: '2026-01-01' }] }), + result({ scanned: 20, linked: 18, skipped: 0, unmatched: 1, failed: 1, total: 113, partial: true, nextCursor: 'file-37', unmatchedSamples: [{ uploadId: 'u2', voucher: 'A2', date: '2026-01-02' }] }), + ) + + expect(merged).toMatchObject({ + scanned: 37, + linked: 33, + skipped: 1, + unmatched: 2, + failed: 1, + total: 113, + partial: true, + nextCursor: 'file-37', + }) + expect(merged.unmatchedSamples).toHaveLength(2) + }) + + it('loops until the server reports the end, passing the cursor back and reporting running totals', async () => { + const fetcher = vi + .fn() + .mockResolvedValueOnce( + jsonResponse({ success: true, result: result({ dryRun: false, scanned: 17, linked: 17, total: 40, partial: true, nextCursor: 'file-17' }) }), + ) + .mockResolvedValueOnce( + jsonResponse({ success: true, result: result({ dryRun: false, scanned: 17, linked: 16, skipped: 1, total: 40, partial: true, nextCursor: 'file-34' }) }), + ) + .mockResolvedValueOnce( + jsonResponse({ success: true, result: result({ dryRun: false, scanned: 6, linked: 6, total: 40 }) }), + ) + const onProgress = vi.fn() + + const final = await runArcimDocumentImportToCompletion('consent-1', { fetcher, onProgress }) + + expect(fetcher).toHaveBeenCalledTimes(3) + const bodies = fetcher.mock.calls.map((call) => JSON.parse((call[1] as RequestInit).body as string)) + expect(bodies).toEqual([ + { consentId: 'consent-1', dryRun: false }, + { consentId: 'consent-1', dryRun: false, cursor: 'file-17' }, + { consentId: 'consent-1', dryRun: false, cursor: 'file-34' }, + ]) + expect(onProgress).toHaveBeenCalledTimes(2) + expect(onProgress.mock.calls[1][0]).toMatchObject({ scanned: 34, linked: 33, skipped: 1, total: 40 }) + expect(final).toMatchObject({ scanned: 40, linked: 39, skipped: 1, total: 40, partial: false, nextCursor: null }) + }) + + it('fails (does not report completion) when a server hands back a cursor that does not advance', async () => { + const fetcher = vi + .fn() + .mockResolvedValueOnce(jsonResponse({ success: true, result: result({ dryRun: false, scanned: 5, total: 10, partial: true, nextCursor: 'file-5' }) })) + .mockResolvedValueOnce(jsonResponse({ success: true, result: result({ dryRun: false, scanned: 0, total: 10, partial: true, nextCursor: 'file-5' }) })) + const onProgress = vi.fn() + + await expect( + runArcimDocumentImportToCompletion('consent-1', { fetcher, onProgress }), + ).rejects.toMatchObject({ problem: { code: ARCIM_DOCUMENT_IMPORT_STALLED } }) + + expect(fetcher).toHaveBeenCalledTimes(2) + // The slice that did land was reported, so the UI keeps honest totals. + expect(onProgress).toHaveBeenCalledTimes(1) + expect(onProgress.mock.calls[0][0]).toMatchObject({ scanned: 5, total: 10 }) + }) + + it('fails when a partial answer carries no cursor at all', async () => { + const fetcher = vi.fn().mockResolvedValue( + jsonResponse({ success: true, result: result({ dryRun: false, scanned: 5, total: 10, partial: true, nextCursor: null }) }), + ) + + await expect(runArcimDocumentImportToCompletion('consent-1', { fetcher })).rejects.toMatchObject({ + problem: { code: ARCIM_DOCUMENT_IMPORT_STALLED }, + }) + expect(fetcher).toHaveBeenCalledTimes(1) + }) + + it('keeps the import phase while recording running totals, then completes with the honest total', () => { + const importing = arcimDocumentImportReducer( + { phase: 'offered', found: 40, result: result({ dryRun: true, total: 40 }), problem: null }, + { type: 'import-started' }, + ) + const progressed = arcimDocumentImportReducer(importing, { + type: 'import-progress', + result: result({ dryRun: false, scanned: 17, linked: 17, total: 40, partial: true, nextCursor: 'file-17' }), + }) + expect(progressed).toMatchObject({ phase: 'importing', result: { scanned: 17, total: 40 } }) + + const complete = arcimDocumentImportReducer(progressed, { + type: 'import-succeeded', + result: result({ dryRun: false, scanned: 40, linked: 39, skipped: 1, total: 40 }), + }) + expect(complete).toMatchObject({ phase: 'complete', found: 40, result: { linked: 39, skipped: 1 } }) + }) +}) diff --git a/components/extensions/general/arcim-document-import-flow.ts b/components/extensions/general/arcim-document-import-flow.ts index 2e83e31a..03bd89fe 100644 --- a/components/extensions/general/arcim-document-import-flow.ts +++ b/components/extensions/general/arcim-document-import-flow.ts @@ -63,6 +63,42 @@ export interface ArcimDocumentImportResult { failed: number dryRun: boolean unmatchedSamples: { uploadId: string; voucher: string; date: string }[] + /** Attachments in the provider's whole list (not just this call). */ + total: number + /** The server stopped at its time budget; continue after nextCursor. */ + partial: boolean + /** Provider id of the last handled attachment, or null when complete. */ + nextCursor: string | null +} + +const MAX_UNMATCHED_SAMPLES = 20 + +/** + * Fold one server call into the running totals of a multi-call import. The + * route processes a slice per call (hosted function limit), so the numbers + * the user sees must be the sum of every slice, not the last one. + */ +export function mergeArcimDocumentImportResults( + accumulated: ArcimDocumentImportResult | null, + next: ArcimDocumentImportResult, +): ArcimDocumentImportResult { + if (!accumulated) return next + return { + provider: next.provider, + dryRun: next.dryRun, + scanned: accumulated.scanned + next.scanned, + linked: accumulated.linked + next.linked, + skipped: accumulated.skipped + next.skipped, + unmatched: accumulated.unmatched + next.unmatched, + failed: accumulated.failed + next.failed, + unmatchedSamples: [...accumulated.unmatchedSamples, ...next.unmatchedSamples].slice( + 0, + MAX_UNMATCHED_SAMPLES, + ), + total: next.total, + partial: next.partial, + nextCursor: next.nextCursor, + } } export interface ArcimDocumentImportProblem { @@ -133,6 +169,7 @@ export type ArcimDocumentImportAction = | { type: 'discovery-failed'; problem: ArcimDocumentImportProblem } | { type: 'dismissed' } | { type: 'import-started' } + | { type: 'import-progress'; result: ArcimDocumentImportResult } | { type: 'import-succeeded'; result: ArcimDocumentImportResult } | { type: 'import-failed'; problem: ArcimDocumentImportProblem } | { type: 'reconnect-started' } @@ -195,10 +232,14 @@ export function arcimDocumentImportReducer( return { ...state, phase: 'dismissed', problem: null } case 'import-started': return { ...state, phase: 'importing', problem: null } + case 'import-progress': + // Running totals after each server slice; the phase stays 'importing' + // so the UI keeps its spinner and shows "x av y". + return { ...state, phase: 'importing', result: action.result, problem: null } case 'import-succeeded': return { phase: 'complete', - found: state.found || action.result.scanned, + found: state.found || action.result.total || action.result.scanned, result: action.result, problem: null, } @@ -242,7 +283,13 @@ function problemFromPayload(payload: unknown): ArcimDocumentImportProblem { } } -function isDocumentImportResult(value: unknown): value is ArcimDocumentImportResult { +type WireDocumentImportResult = Omit< + ArcimDocumentImportResult, + 'total' | 'partial' | 'nextCursor' +> & + Partial> + +function isDocumentImportResult(value: unknown): value is WireDocumentImportResult { if (!value || typeof value !== 'object') return false const result = value as Partial return ( @@ -257,16 +304,37 @@ function isDocumentImportResult(value: unknown): value is ArcimDocumentImportRes ) } -/** Call the existing POST route for both discovery and the actual import. */ +/** Older servers answer without the resume fields: a single complete slice. */ +function normalizeDocumentImportResult( + result: WireDocumentImportResult, +): ArcimDocumentImportResult { + return { + ...result, + total: typeof result.total === 'number' ? result.total : result.scanned, + partial: result.partial === true, + nextCursor: + typeof result.nextCursor === 'string' && result.nextCursor.length > 0 && result.partial === true + ? result.nextCursor + : null, + } +} + +/** + * Call the existing POST route for both discovery and the actual import. + * `cursor` resumes a partial import; omit it for discovery and the first slice. + */ export async function requestArcimDocumentImport( consentId: string, dryRun: boolean, fetcher: typeof fetch = fetch, + cursor?: string, ): Promise { const response = await fetcher(ARCIM_DOCUMENT_IMPORT_ENDPOINT, { method: 'POST', headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ consentId, dryRun }), + body: JSON.stringify( + cursor === undefined ? { consentId, dryRun } : { consentId, dryRun, cursor }, + ), }) const payload = await response.json().catch(() => null) @@ -283,5 +351,50 @@ export async function requestArcimDocumentImport( }) } - return result + return normalizeDocumentImportResult(result) +} + +/** Far above any real list: a guard against a server that never completes. */ +const MAX_IMPORT_ROUNDS = 500 + +export const ARCIM_DOCUMENT_IMPORT_STALLED = 'ARCIM_DOCUMENT_IMPORT_STALLED' + +/** + * Run the real import to completion: the route works through one time-budgeted + * slice per call and hands back a cursor, so this loops until the server says + * it reached the end, reporting running totals after every slice. A thrown + * request error aborts the loop; the slices already archived stay linked and + * the next attempt skips them by hash. A server that keeps answering "partial" + * without advancing (or past the round guard) is reported as a failure, never + * as a completed import: the retry button resumes safely. + */ +export async function runArcimDocumentImportToCompletion( + consentId: string, + options: { + fetcher?: typeof fetch + onProgress?: (accumulated: ArcimDocumentImportResult) => void + } = {}, +): Promise { + const fetcher = options.fetcher ?? fetch + let accumulated: ArcimDocumentImportResult | null = null + let cursor: string | undefined + + for (let round = 0; round < MAX_IMPORT_ROUNDS; round++) { + const slice = await requestArcimDocumentImport(consentId, false, fetcher, cursor) + accumulated = mergeArcimDocumentImportResults(accumulated, slice) + if (!slice.partial) { + return { ...accumulated, partial: false, nextCursor: null } + } + if (slice.nextCursor === null || (cursor !== undefined && slice.nextCursor <= cursor)) { + break + } + cursor = slice.nextCursor + options.onProgress?.(accumulated) + } + + throw new ArcimDocumentImportRequestError({ + code: ARCIM_DOCUMENT_IMPORT_STALLED, + requestId: null, + reconnectRequired: false, + }) } diff --git a/extensions/general/arcim-migration/__tests__/import-documents-route.test.ts b/extensions/general/arcim-migration/__tests__/import-documents-route.test.ts index 8b691c29..280f1dfb 100644 --- a/extensions/general/arcim-migration/__tests__/import-documents-route.test.ts +++ b/extensions/general/arcim-migration/__tests__/import-documents-route.test.ts @@ -152,6 +152,45 @@ describe('POST /import-documents', () => { ) }) + it('passes a non-empty string cursor through and restarts from the top for anything else', async () => { + ;(importProviderDocuments as Mock).mockResolvedValue({ + provider: 'fortnox', + scanned: 1, + linked: 1, + skipped: 0, + unmatched: 0, + failed: 0, + dryRun: false, + unmatchedSamples: [], + total: 113, + partial: true, + nextCursor: 'file-18', + }) + + const withCursor = createMockRequest( + 'http://localhost/api/extensions/ext/arcim-migration/import-documents', + { method: 'POST', body: { consentId: 'consent-1', dryRun: false, cursor: 'file-17' } }, + ) + const { status, body } = await parseJsonResponse<{ + result: { partial: boolean; nextCursor: string | null } + }>(await handler(withCursor, buildContext())) + + expect(status).toBe(200) + expect(body.result).toMatchObject({ partial: true, nextCursor: 'file-18' }) + expect(importProviderDocuments).toHaveBeenLastCalledWith( + expect.objectContaining({ consentId: 'consent-1', dryRun: false, cursor: 'file-17' }), + ) + + const garbage = createMockRequest( + 'http://localhost/api/extensions/ext/arcim-migration/import-documents', + { method: 'POST', body: { consentId: 'consent-1', cursor: 17 } }, + ) + await handler(garbage, buildContext()) + expect(importProviderDocuments).toHaveBeenLastCalledWith( + expect.objectContaining({ cursor: null }), + ) + }) + it('asks the user to reconnect only once the connect request carries the scopes', async () => { fortnoxOAuth.documentScopesApproved = true ;(importProviderDocuments as Mock).mockRejectedValue( diff --git a/extensions/general/arcim-migration/__tests__/import-documents.test.ts b/extensions/general/arcim-migration/__tests__/import-documents.test.ts index d46389f2..5ec978fe 100644 --- a/extensions/general/arcim-migration/__tests__/import-documents.test.ts +++ b/extensions/general/arcim-migration/__tests__/import-documents.test.ts @@ -177,11 +177,16 @@ describe('importProviderDocuments', () => { expect(userId).toBe(USER) expect(companyId).toBe(COMPANY) expect(file).toMatchObject({ name: 'Kvitto.pdf', type: 'application/pdf' }) + // extractionOwner 'none': the file arrives linked to its posted + // verifikat, so no paid model pass (it was the inline extraction that + // blew the hosted function budget on a 113-file import). expect(metadata).toEqual({ upload_source: 'api', journal_entry_id: 'je-1', idempotency_key: 'je-1', + extractionOwner: 'none', }) + expect(result).toMatchObject({ total: 1, partial: false, nextCursor: null }) }) it('skips a receipt already archived on the same verifikat (sha256 + journal entry idempotency)', async () => { @@ -268,6 +273,7 @@ describe('importProviderDocuments', () => { upload_source: 'api', journal_entry_id: 'je-1', idempotency_key: 'je-1', + extractionOwner: 'none', }) }) @@ -648,4 +654,123 @@ describe('importProviderDocuments', () => { const [, , , file] = mockUpload.mock.calls[0] expect(file).toMatchObject({ name: 'Kvitto.jpg', type: 'image/jpeg' }) }) + + describe('time budget and cursor (resumable sweep)', () => { + const THREE_CONNECTIONS: FortnoxFileConnection[] = [ + { ...FORTNOX_CONNECTION, fileId: 'file-c', name: 'c.pdf' }, + { ...FORTNOX_CONNECTION, fileId: 'file-a', name: 'a.pdf' }, + { ...FORTNOX_CONNECTION, fileId: 'file-b', name: 'b.pdf' }, + ] + + it('stops after the attachment in flight once the budget is spent and hands back a cursor', async () => { + const supabase = wireFortnox({ connections: THREE_CONNECTIONS }) + // Each download "costs" 100 ms of wall clock; the budget allows one. + let clock = 0 + mockDownloadFortnoxArchiveFile.mockImplementation(async () => { + clock += 100 + return { bytes: bytesOf('FORTNOX-PDF-' + clock), contentType: 'application/pdf' } + }) + + const first = await importProviderDocuments({ + supabase, + companyId: COMPANY, + userId: USER, + consentId: 'c1', + timeBudgetMs: 50, + now: () => clock, + }) + + expect(first).toMatchObject({ + total: 3, + scanned: 1, + linked: 1, + partial: true, + nextCursor: 'file-a', + }) + // Sorted by provider id, so the first slice is file-a, not file-c. + expect(mockUpload.mock.calls[0][3]).toMatchObject({ name: 'a.pdf' }) + + const second = await importProviderDocuments({ + supabase, + companyId: COMPANY, + userId: USER, + consentId: 'c1', + cursor: first.nextCursor, + timeBudgetMs: 1_000_000, + now: () => clock, + }) + + expect(second).toMatchObject({ total: 3, scanned: 2, linked: 2, partial: false, nextCursor: null }) + expect(mockUpload.mock.calls.map((call) => (call[3] as { name: string }).name)).toEqual([ + 'a.pdf', + 'b.pdf', + 'c.pdf', + ]) + }) + + it('always processes at least one attachment per call even when the budget is already spent', async () => { + const supabase = wireFortnox({ connections: THREE_CONNECTIONS }) + + const result = await importProviderDocuments({ + supabase, + companyId: COMPANY, + userId: USER, + consentId: 'c1', + timeBudgetMs: 0, + now: () => 0, + }) + + expect(result).toMatchObject({ scanned: 1, linked: 1, partial: true, nextCursor: 'file-a' }) + }) + + it('treats a cursor past the last id as a complete, empty slice', async () => { + const supabase = wireFortnox({ connections: THREE_CONNECTIONS }) + + const result = await importProviderDocuments({ + supabase, + companyId: COMPANY, + userId: USER, + consentId: 'c1', + cursor: 'file-z', + }) + + expect(result).toMatchObject({ total: 3, scanned: 0, linked: 0, partial: false, nextCursor: null }) + expect(mockUpload).not.toHaveBeenCalled() + }) + + it('resumes after the last handled id even when the provider list changed in between', async () => { + // Slice 1 handled file-a. Before slice 2, Fortnox gained a file sorting + // BEFORE the resume point (file-0) and lost file-b. An index cursor + // would now skip one; an id cursor continues with everything > file-a. + const supabase = wireFortnox({ + connections: [ + { ...FORTNOX_CONNECTION, fileId: 'file-0', name: '0.pdf' }, + { ...FORTNOX_CONNECTION, fileId: 'file-a', name: 'a.pdf' }, + { ...FORTNOX_CONNECTION, fileId: 'file-c', name: 'c.pdf' }, + { ...FORTNOX_CONNECTION, fileId: 'file-d', name: 'd.pdf' }, + ], + }) + // Distinct bytes per file: the default mock returns the same bytes for + // every download, and identical content on the same verifikat is + // (correctly) deduped, which is not what this test is about. + mockDownloadFortnoxArchiveFile.mockImplementation(async (_client, _token, id) => ({ + bytes: bytesOf('PDF-' + id), + contentType: 'application/pdf', + })) + + const result = await importProviderDocuments({ + supabase, + companyId: COMPANY, + userId: USER, + consentId: 'c1', + cursor: 'file-a', + }) + + expect(result).toMatchObject({ total: 4, scanned: 2, linked: 2, partial: false }) + expect(mockUpload.mock.calls.map((call) => (call[3] as { name: string }).name)).toEqual([ + 'c.pdf', + 'd.pdf', + ]) + }) + }) }) diff --git a/extensions/general/arcim-migration/index.ts b/extensions/general/arcim-migration/index.ts index 5352a8fe..12f7fa07 100644 --- a/extensions/general/arcim-migration/index.ts +++ b/extensions/general/arcim-migration/index.ts @@ -1363,10 +1363,23 @@ export const arcimMigrationExtension: Extension = { let consentId: string | undefined let dryRun = false + let cursor: string | null = null try { - const body = (await request.json()) as { consentId?: string; dryRun?: boolean } + const body = (await request.json()) as { + consentId?: string + dryRun?: boolean + cursor?: string + } consentId = body?.consentId dryRun = body?.dryRun === true + // Resume point from a previous partial call (the last handled + // provider attachment id, see import-documents.ts); anything else + // restarts from the top, which is always safe: already-archived + // receipts are skipped by hash. + cursor = + typeof body?.cursor === 'string' && body.cursor.length > 0 && body.cursor.length <= 256 + ? body.cursor + : null } catch { // empty/invalid body: consentId check below rejects it } @@ -1382,15 +1395,20 @@ export const arcimMigrationExtension: Extension = { userId: user.id, consentId, dryRun, + cursor, }) log.info('arcim import-documents completed', { companyId, dryRun, + cursor, + total: result.total, scanned: result.scanned, linked: result.linked, skipped: result.skipped, unmatched: result.unmatched, failed: result.failed, + partial: result.partial, + nextCursor: result.nextCursor, }) return NextResponse.json({ success: true, dryRun, result }) } catch (error) { diff --git a/extensions/general/arcim-migration/lib/import-documents.ts b/extensions/general/arcim-migration/lib/import-documents.ts index cda82482..05846556 100644 --- a/extensions/general/arcim-migration/lib/import-documents.ts +++ b/extensions/general/arcim-migration/lib/import-documents.ts @@ -24,6 +24,22 @@ * critical path: provider document APIs are rate-limited and a full sweep can * issue hundreds of download calls. Fortnox requires archive and connectfile * scopes; existing consents must reconnect before this import can run. + * + * Resumable: one call works through the provider's attachment list (sorted + * by provider id so the order is stable between calls) until `timeBudgetMs` + * is spent, then returns `partial: true` with `nextCursor` = the id of the + * last attachment it handled. The caller loops, passing the cursor back, + * until `partial` is false. The cursor is an id, not an index, so a file the + * provider adds or removes mid-sweep shifts nothing: the next slice simply + * continues after the last handled id in sorted order. A 113-file Fortnox + * import used to run every file inline inside one request and hit the + * hosted 300 s function limit after ~17 files (2026-08-21); the UI then + * showed a generic failure even though the files it did reach were linked. + * + * No AI extraction: every file here is linked to a posted verifikat on + * arrival, so the booking is already known. The upload opts out + * (`extractionOwner: 'none'`); it was the per-file model pass that made the + * sweep blow its budget in the first place. */ import type { SupabaseClient } from '@supabase/supabase-js' @@ -78,8 +94,20 @@ export interface ImportDocumentsOptions { consentId: string /** Resolve + report what would be attached without downloading or writing. */ dryRun?: boolean + /** Resume after this provider attachment id (sorted order); omit to start from the top. */ + cursor?: string | null + /** + * Wall-clock budget for this call. Once spent, the sweep stops after the + * attachment in flight and reports `nextCursor`. Default leaves headroom + * under the hosted 300 s function limit for listing + the final response. + */ + timeBudgetMs?: number + /** Clock, injectable for tests. */ + now?: () => number } +export const DEFAULT_IMPORT_DOCUMENTS_TIME_BUDGET_MS = 200_000 + export interface ImportDocumentsResult { provider: string /** Provider attachments linked to a voucher that were considered. */ @@ -95,6 +123,12 @@ export interface ImportDocumentsResult { dryRun: boolean /** A few unmatched voucher labels, to aid diagnosis without dumping all. */ unmatchedSamples: { uploadId: string; voucher: string; date: string }[] + /** Provider attachments linked to a voucher in total (all calls). */ + total: number + /** True when the time budget ran out before the end of the list. */ + partial: boolean + /** Last handled attachment id; pass back as `cursor` to continue. Null when complete. */ + nextCursor: string | null } interface ProviderAttachment { @@ -232,7 +266,17 @@ function fortnoxSource( export async function importProviderDocuments( opts: ImportDocumentsOptions, ): Promise { - const { supabase, companyId, userId, consentId, dryRun = false } = opts + const { + supabase, + companyId, + userId, + consentId, + dryRun = false, + cursor = null, + timeBudgetMs = DEFAULT_IMPORT_DOCUMENTS_TIME_BUDGET_MS, + now = Date.now, + } = opts + const startedAt = now() let resolved = await resolveConsent(companyId, consentId) const provider = resolved.consent.provider as string @@ -246,6 +290,9 @@ export async function importProviderDocuments( failed: 0, dryRun, unmatchedSamples: [], + total: 0, + partial: false, + nextCursor: null, } // Unsupported providers are a no-op rather than an error @@ -263,7 +310,7 @@ export async function importProviderDocuments( : fortnoxSource(fortnoxClient as FortnoxClient, resolved.accessToken) // ── Bulk reads (one round of paged requests each, no per-item N+1) ── - const [attachments, periods, vouchers, existingAttachments] = await Promise.all([ + const [listedAttachments, periods, vouchers, existingAttachments] = await Promise.all([ source().list(), fetchFiscalPeriods(supabase, companyId), fetchSourceRefVouchers(supabase, companyId), @@ -280,6 +327,15 @@ export async function importProviderDocuments( ), ]) + // Stable order across calls: a provider may page its listing differently + // from one call to the next, and the cursor is "continue after this id". + const attachments = [...listedAttachments].sort((a, b) => + a.id < b.id ? -1 : a.id > b.id ? 1 : 0, + ) + result.total = attachments.length + const firstAfterCursor = cursor ? attachments.findIndex((a) => a.id > cursor) : 0 + const startIndex = firstAfterCursor === -1 ? attachments.length : firstAfterCursor + // Index gnubok verifikat by (period, series, number) for in-memory resolution. const voucherIndex = buildVoucherIndex(vouchers) @@ -302,7 +358,16 @@ export async function importProviderDocuments( let refreshedAfterUnauthorized = false - for (const attachment of attachments) { + for (let index = startIndex; index < attachments.length; index++) { + // Always make progress: at least one attachment per call, then stop at + // the budget so the route answers well inside the function limit and the + // caller resumes after the last handled id. + if (index > startIndex && now() - startedAt >= timeBudgetMs) { + result.partial = true + result.nextCursor = attachments[index - 1].id + break + } + const attachment = attachments[index] result.scanned++ const ref = attachment.ref @@ -363,6 +428,8 @@ export async function importProviderDocuments( upload_source: 'api', journal_entry_id: journalEntryId, idempotency_key: journalEntryId, + // Already booked: the verifikat is the booking. No model pass. + extractionOwner: 'none', }, ) @@ -405,11 +472,16 @@ export async function importProviderDocuments( log.info('document import complete', { companyId, dryRun, + cursor, + total: result.total, scanned: result.scanned, linked: result.linked, skipped: result.skipped, unmatched: result.unmatched, failed: result.failed, + partial: result.partial, + nextCursor: result.nextCursor, + elapsedMs: now() - startedAt, }) return result diff --git a/extensions/general/document-extraction/__tests__/handler.test.ts b/extensions/general/document-extraction/__tests__/handler.test.ts index dc048737..102eb7ae 100644 --- a/extensions/general/document-extraction/__tests__/handler.test.ts +++ b/extensions/general/document-extraction/__tests__/handler.test.ts @@ -64,6 +64,13 @@ beforeEach(() => { describe('document-extraction handler', () => { // THE dedupe: inbox-owned documents are extracted (and mirrored) by the // inbox itself. The handler used to race it and pay a second model call. + it('stamps and skips when the uploader opted out (already-booked provider underlag)', async () => { + await handler(payload({ extractionOwner: 'none' })) + + expect(extractMock).not.toHaveBeenCalled() + expect(lastStamp()).toBe('skipped:opted_out') + }) + it('yields entirely when the inbox owns extraction', async () => { await handler(payload({ extractionOwner: 'invoice-inbox' })) expect(supabase.from).not.toHaveBeenCalled() diff --git a/extensions/general/document-extraction/index.ts b/extensions/general/document-extraction/index.ts index 39524791..1c70a279 100644 --- a/extensions/general/document-extraction/index.ts +++ b/extensions/general/document-extraction/index.ts @@ -7,6 +7,7 @@ import { CAPABILITY } from '@/lib/entitlements/keys' import { createLogger } from '@/lib/logger' import { createServiceClient } from '@/lib/supabase/server' import type { DocumentAttachment } from '@/types' +import type { DocumentExtractionOwner } from '@/lib/events/types' const log = createLogger('document-extraction') @@ -62,9 +63,17 @@ export const documentExtractionExtension: Extension = { document: DocumentAttachment userId: string companyId: string - extractionOwner?: 'invoice-inbox' + extractionOwner?: DocumentExtractionOwner } if (extractionOwner === 'invoice-inbox') return + if (extractionOwner === 'none') { + // The uploader already holds the booking (provider underlag import + // links each file to its posted verifikat): a model pass would cost + // a paid call per file and, run inline, blew the 300 s route budget + // on a 113-file Fortnox import (2026-08-21). Stamp so nothing polls. + await stamp(createServiceClient(), document.id, 'skipped:opted_out') + return + } await extractAndPersist(document, companyId) }, }, diff --git a/extensions/general/mcp-server/__tests__/document-upload-tools.test.ts b/extensions/general/mcp-server/__tests__/document-upload-tools.test.ts index 16deb612..2a08bd4b 100644 --- a/extensions/general/mcp-server/__tests__/document-upload-tools.test.ts +++ b/extensions/general/mcp-server/__tests__/document-upload-tools.test.ts @@ -1,4 +1,4 @@ -import { beforeEach, describe, expect, it, vi } from 'vitest' +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import { makeDocumentAttachment } from '@/tests/helpers' import { TOOL_SCOPE_MAP } from '@/lib/auth/api-keys' import { MCP_TOOL_CAPABILITY_MAP } from '@/lib/entitlements/keys' @@ -51,6 +51,10 @@ function makeQueryBuilder(result: { data: unknown; error: unknown }) { } describe('MCP model-free document upload tools', () => { + afterEach(() => { + vi.unstubAllEnvs() + }) + beforeEach(() => { vi.clearAllMocks() mocks.createPendingDocumentUpload.mockResolvedValue({ @@ -76,6 +80,32 @@ describe('MCP model-free document upload tools', () => { }) }) + // Claude Desktop's sandbox only reaches the MCP host: a signed URL on + // .supabase.co was refused there (2026-08-21), so the tool hands + // out the same-origin /api/storage proxy URL instead. + it('serves the upload URL from the app origin when the signed URL points at our Storage host', async () => { + vi.stubEnv('NEXT_PUBLIC_SUPABASE_URL', 'https://pwxtzglxptnnvjrpixpg.supabase.co') + vi.stubEnv('NEXT_PUBLIC_APP_URL', 'https://app.accounted.se') + mocks.createPendingDocumentUpload.mockResolvedValue({ + uploadId, + signedUrl: + 'https://pwxtzglxptnnvjrpixpg.supabase.co/storage/v1/object/upload/sign/documents/co/user/pending/up/invoice.pdf?token=signed', + expiresAt: '2026-08-03T12:00:00.000Z', + }) + + const result = await findTool('gnubok_create_document_upload').execute( + { file_name: 'invoice.pdf' }, + companyId, + userId, + {} as never, + ) + + expect(result).toMatchObject({ + upload_url: + 'https://app.accounted.se/api/storage/upload/sign/documents/co/user/pending/up/invoice.pdf?token=signed', + }) + }) + it('returns an unauthenticated PUT URL without accepting file bytes', async () => { const tool = findTool('gnubok_create_document_upload') const result = await tool.execute( diff --git a/extensions/general/mcp-server/server.ts b/extensions/general/mcp-server/server.ts index 6aa5aac1..255edd35 100644 --- a/extensions/general/mcp-server/server.ts +++ b/extensions/general/mcp-server/server.ts @@ -209,6 +209,7 @@ import { uploadDocument, MAX_DOCUMENT_SIZE, } from '@/lib/core/documents/document-service' +import { toSameOriginStorageUrl } from '@/lib/core/documents/storage-proxy' import { extractInvoiceFields, ExtractionSchema as InvoiceExtractionSchema, AgentExtractionSchema } from '@/extensions/general/invoice-inbox/lib/extract-invoice-fields' import { mirrorExtractionToDocument } from '@/extensions/general/invoice-inbox/lib/mirror-extraction' // Skatteverket filing tools (PR5). Cross-extension lib import, same sanctioned @@ -9715,9 +9716,11 @@ export const tools: McpTool[] = [ uploadId, fileName, ) + // Served from the app origin: agent sandboxes (Claude Desktop) only + // reach the MCP host, not .supabase.co. See storage-proxy.ts. return { upload_id: reservation.uploadId, - upload_url: reservation.signedUrl, + upload_url: toSameOriginStorageUrl(reservation.signedUrl), expires_at: reservation.expiresAt, } }, @@ -10664,7 +10667,7 @@ export const tools: McpTool[] = [ file_name: doc.file_name, mime_type: doc.mime_type, size_bytes: doc.file_size_bytes, - signed_url: signed.signedUrl, + signed_url: toSameOriginStorageUrl(signed.signedUrl), expires_at: expiresAt, } }, @@ -13574,7 +13577,7 @@ export const tools: McpTool[] = [ type: 'object', additionalProperties: false, properties: { - download_url: { type: ['string', 'null'], description: 'Signed Supabase Storage URL, valid 1 hour; null when estimate_only=true. Restricted-egress proxies may 403; needs a network with Supabase egress.' }, + download_url: { type: ['string', 'null'], description: 'Signed download URL, valid 1 hour, on the app origin (direct Storage URL only when NEXT_PUBLIC_APP_URL is unset); null when estimate_only=true.' }, storage_path: { type: ['string', 'null'] }, file_name: { type: 'string' }, size_bytes: { type: 'number' }, @@ -13679,7 +13682,7 @@ export const tools: McpTool[] = [ const expiresAt = new Date(Date.now() + SIGNED_URL_TTL_SECONDS * 1000).toISOString() return { - download_url: signed.signedUrl, + download_url: toSameOriginStorageUrl(signed.signedUrl), storage_path: storagePath, file_name: fileName, size_bytes: zipBuffer.byteLength, diff --git a/lib/core/documents/__tests__/storage-proxy.test.ts b/lib/core/documents/__tests__/storage-proxy.test.ts new file mode 100644 index 00000000..365a5bd0 --- /dev/null +++ b/lib/core/documents/__tests__/storage-proxy.test.ts @@ -0,0 +1,190 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { + readBodyWithCap, + resolveUpstreamStorageUrl, + toSameOriginStorageUrl, +} from '../storage-proxy' + +const SUPABASE = 'https://pwxtzglxptnnvjrpixpg.supabase.co' +const APP = 'https://app.accounted.se' +const UPLOAD_URL = `${SUPABASE}/storage/v1/object/upload/sign/documents/co-1/user-1/pending/up-1/faktura%20maj.pdf?token=eyJ.sig` +const DOWNLOAD_URL = `${SUPABASE}/storage/v1/object/sign/documents/co-1/user-1/kvitto.pdf?token=eyJ.sig&download=` + +beforeEach(() => { + vi.stubEnv('NEXT_PUBLIC_SUPABASE_URL', SUPABASE) + vi.stubEnv('NEXT_PUBLIC_APP_URL', APP) +}) + +afterEach(() => { + vi.unstubAllEnvs() +}) + +describe('toSameOriginStorageUrl', () => { + it('moves a signed upload URL onto the app origin, keeping path encoding and the token', () => { + expect(toSameOriginStorageUrl(UPLOAD_URL)).toBe( + `${APP}/api/storage/upload/sign/documents/co-1/user-1/pending/up-1/faktura%20maj.pdf?token=eyJ.sig`, + ) + }) + + it('moves a signed download URL onto the app origin with every query parameter', () => { + expect(toSameOriginStorageUrl(DOWNLOAD_URL)).toBe( + `${APP}/api/storage/sign/documents/co-1/user-1/kvitto.pdf?token=eyJ.sig&download=`, + ) + }) + + it('tolerates a trailing slash on the app URL', () => { + expect(toSameOriginStorageUrl(DOWNLOAD_URL, `${APP}/`)).toMatch( + new RegExp(`^${APP}/api/storage/sign/`), + ) + }) + + it('leaves the URL alone when the app has no public URL (self-host without NEXT_PUBLIC_APP_URL)', () => { + vi.stubEnv('NEXT_PUBLIC_APP_URL', '') + expect(toSameOriginStorageUrl(UPLOAD_URL)).toBe(UPLOAD_URL) + expect(toSameOriginStorageUrl(UPLOAD_URL, undefined)).toBe(UPLOAD_URL) + }) + + it('leaves URLs on other hosts, other buckets, unsigned paths and token-less links alone', () => { + expect(toSameOriginStorageUrl('https://storage.example/upload?token=signed')).toBe( + 'https://storage.example/upload?token=signed', + ) + const otherBucket = `${SUPABASE}/storage/v1/object/sign/avatars/a.png?token=t` + expect(toSameOriginStorageUrl(otherBucket)).toBe(otherBucket) + const publicObject = `${SUPABASE}/storage/v1/object/public/documents/a.pdf` + expect(toSameOriginStorageUrl(publicObject)).toBe(publicObject) + const noToken = `${SUPABASE}/storage/v1/object/sign/documents/a.pdf` + expect(toSameOriginStorageUrl(noToken)).toBe(noToken) + expect(toSameOriginStorageUrl('not a url')).toBe('not a url') + }) +}) + +describe('resolveUpstreamStorageUrl', () => { + it('rebuilds the upstream signed URL for upload and download paths', () => { + expect( + resolveUpstreamStorageUrl( + 'upload/sign/documents/co-1/user-1/pending/up-1/faktura%20maj.pdf', + new URLSearchParams('token=eyJ.sig'), + ), + ).toEqual({ + ok: true, + url: `${SUPABASE}/storage/v1/object/upload/sign/documents/co-1/user-1/pending/up-1/faktura%20maj.pdf?token=eyJ.sig`, + }) + expect( + resolveUpstreamStorageUrl( + 'sign/documents/co-1/user-1/kvitto.pdf', + new URLSearchParams('token=eyJ.sig&download='), + ), + ).toEqual({ + ok: true, + url: `${SUPABASE}/storage/v1/object/sign/documents/co-1/user-1/kvitto.pdf?token=eyJ.sig&download=`, + }) + }) + + it('fails closed on anything that is not a signed documents-bucket object path', () => { + const token = new URLSearchParams('token=t') + expect(resolveUpstreamStorageUrl('public/documents/a.pdf', token)).toEqual({ + ok: false, + reason: 'unsupported_path', + }) + expect(resolveUpstreamStorageUrl('sign/avatars/a.png', token)).toEqual({ + ok: false, + reason: 'unsupported_path', + }) + expect(resolveUpstreamStorageUrl('sign/documents/', token)).toEqual({ + ok: false, + reason: 'unsupported_path', + }) + expect(resolveUpstreamStorageUrl('sign/documents/../bucket/x', token)).toEqual({ + ok: false, + reason: 'unsupported_path', + }) + expect(resolveUpstreamStorageUrl('list/documents', token)).toEqual({ + ok: false, + reason: 'unsupported_path', + }) + }) + + it('rejects dot segments in every spelling the URL parser would normalise away', () => { + const token = new URLSearchParams('token=t') + for (const evil of [ + 'sign/documents/%2e%2e/avatars/x', + 'sign/documents/.%2e/avatars/x', + 'sign/documents/%2e./avatars/x', + 'sign/documents/%2E%2E/avatars/x', + 'sign/documents/a/../../avatars/x', + 'sign/documents/./x', + 'sign/documents/a//x', + 'sign/documents/a%5c..%5cavatars/x', + 'sign/documents/a\\..\\avatars/x', + 'sign/documents/a%2f..%2favatars/x', + 'sign/documents/%zz/x', + 'upload/sign/documents/%2e%2e/avatars/x', + ]) { + expect(resolveUpstreamStorageUrl(evil, token), evil).toEqual({ + ok: false, + reason: 'unsupported_path', + }) + } + }) + + it('still accepts real keys with spaces, non-ASCII and dots inside names', () => { + const token = new URLSearchParams('token=t') + const resolved = resolveUpstreamStorageUrl( + 'sign/documents/co-1/user-1/faktura%20maj%20%C3%A5r.2026.pdf', + token, + ) + expect(resolved.ok).toBe(true) + }) + + it('requires the signed token', () => { + expect(resolveUpstreamStorageUrl('sign/documents/a.pdf', new URLSearchParams())).toEqual({ + ok: false, + reason: 'missing_token', + }) + }) + + it('reports an unconfigured Storage host instead of guessing', () => { + vi.stubEnv('NEXT_PUBLIC_SUPABASE_URL', '') + expect( + resolveUpstreamStorageUrl('sign/documents/a.pdf', new URLSearchParams('token=t')), + ).toEqual({ ok: false, reason: 'storage_unconfigured' }) + }) +}) + +describe('readBodyWithCap', () => { + function stream(chunks: Uint8Array[]): ReadableStream { + return new ReadableStream({ + start(controller) { + for (const chunk of chunks) controller.enqueue(chunk) + controller.close() + }, + }) + } + + it('concatenates a body under the cap', async () => { + const out = await readBodyWithCap( + stream([new TextEncoder().encode('%PDF'), new TextEncoder().encode('-1.4')]), + 100, + ) + expect(out && new TextDecoder().decode(out)).toBe('%PDF-1.4') + }) + + it('returns null as soon as the running total passes the cap, without draining the rest', async () => { + let pulled = 0 + const endless = new ReadableStream({ + pull(controller) { + pulled++ + controller.enqueue(new Uint8Array(1024)) + }, + }) + + const out = await readBodyWithCap(endless, 4096) + + expect(out).toBeNull() + expect(pulled).toBeLessThan(10) + }) + + it('treats a missing body as empty', async () => { + expect((await readBodyWithCap(null, 10))?.byteLength).toBe(0) + }) +}) diff --git a/lib/core/documents/document-service.ts b/lib/core/documents/document-service.ts index 241c6352..7e00338c 100644 --- a/lib/core/documents/document-service.ts +++ b/lib/core/documents/document-service.ts @@ -1,6 +1,7 @@ import type { SupabaseClient } from '@supabase/supabase-js' import { createServiceClientNoCookies } from '@/lib/auth/api-keys' import { eventBus } from '@/lib/events' +import type { DocumentExtractionOwner } from '@/lib/events/types' import type { DocumentAttachment, DocumentUploadSource } from '@/types' /** @@ -504,7 +505,7 @@ export async function completePendingDocumentUpload( fileName: string, mimeType: string, now: number = Date.now(), - options: { extractionOwner?: 'invoice-inbox' } = {} + options: { extractionOwner?: DocumentExtractionOwner } = {} ): Promise { const serviceClient = createServiceClientNoCookies() const storage = serviceClient.storage.from(DOCUMENTS_BUCKET) @@ -656,9 +657,10 @@ export async function uploadDocument( * Who runs AI extraction on this document. The invoice inbox extracts * the documents it ingests itself (and mirrors the result onto the * document row), so it declares ownership here and the - * document-extraction extension yields. Default: the extension extracts. + * document-extraction extension yields. 'none' opts out entirely (the + * caller already knows the booking). Default: the extension extracts. */ - extractionOwner?: 'invoice-inbox' + extractionOwner?: DocumentExtractionOwner } = {} ): Promise { await ensureDocumentsBucket() diff --git a/lib/core/documents/storage-proxy.ts b/lib/core/documents/storage-proxy.ts new file mode 100644 index 00000000..97427d06 --- /dev/null +++ b/lib/core/documents/storage-proxy.ts @@ -0,0 +1,177 @@ +/** + * Same-origin proxy for Supabase Storage signed URLs. + * + * Agent sandboxes (Claude Desktop's code execution, some MCP clients) only + * allow network egress to the host the MCP server lives on. Our signed + * Storage URLs point at .supabase.co, so a model-free upload or a + * document download from such a sandbox was refused before it left the box + * (user report 2026-08-21). These helpers rewrite a signed URL onto this + * app's own origin (`/api/storage/...`) and resolve it back to the upstream + * Storage URL inside the proxy route. + * + * The signed token travels unchanged and is the only credential on both + * sides: the proxy grants nothing the public Storage host did not already + * grant, and it refuses every path that is not a signed object path on the + * documents bucket. Without NEXT_PUBLIC_APP_URL (a self-host that never set + * it) the rewrite is a no-op rather than a broken localhost link. + */ + +export const STORAGE_PROXY_ROUTE = '/api/storage' + +/** Upstream prefix under the Storage API that every signed object URL shares. */ +const UPSTREAM_OBJECT_PREFIX = '/storage/v1/object/' + +/** + * Only token-authenticated object paths on the documents bucket: signed + * downloads (`sign/documents/...`) and signed uploads + * (`upload/sign/documents/...`). Anything else (public objects, bucket + * admin, other buckets) is not this proxy's business. + */ +const ALLOWED_OBJECT_PATH_RE = /^(sign|upload\/sign)\/documents\/[^/]+/ + +export type StorageProxyResolution = + | { ok: true; url: string } + | { ok: false; reason: 'unsupported_path' | 'missing_token' | 'storage_unconfigured' } + +/** + * True when every segment of the (still percent-encoded) object path is a + * plain name: no empty segment, no `.`/`..` in raw or percent-encoded form + * (`%2e%2e`, `.%2e`, `%2e.`: the WHATWG URL parser normalises those away, + * which would let `sign/documents/%2e%2e/other/x` leave the documents + * bucket), no backslash (a path separator for https URLs), no malformed + * escapes. + */ +function hasOnlyPlainSegments(objectPath: string): boolean { + for (const segment of objectPath.split('/')) { + if (segment === '') return false + let decoded: string + try { + decoded = decodeURIComponent(segment) + } catch { + return false + } + if (decoded === '.' || decoded === '..') return false + if (decoded.includes('\\') || decoded.includes('/')) return false + } + return true +} + +function trimTrailingSlash(value: string): string { + return value.replace(/\/+$/, '') +} + +function upstreamStorageOrigin(): string | null { + const raw = process.env.NEXT_PUBLIC_SUPABASE_URL?.trim() + if (!raw) return null + try { + return new URL(raw).origin + } catch { + return null + } +} + +/** + * Rewrite a Supabase signed Storage URL onto this app's origin. Returns the + * input unchanged when it is not a signed documents-bucket URL on our Storage + * host, or when the app has no public URL configured. + */ +export function toSameOriginStorageUrl( + signedUrl: string, + appBaseUrl: string | undefined = process.env.NEXT_PUBLIC_APP_URL, +): string { + const base = appBaseUrl?.trim() + const upstream = upstreamStorageOrigin() + if (!base || !upstream) return signedUrl + + let url: URL + try { + url = new URL(signedUrl) + } catch { + return signedUrl + } + if (url.origin !== upstream) return signedUrl + if (!url.pathname.startsWith(UPSTREAM_OBJECT_PREFIX)) return signedUrl + + // Keep the pathname exactly as Storage encoded it: object keys may hold + // spaces and non-ASCII, and the token was signed over the real key. + const objectPath = url.pathname.slice(UPSTREAM_OBJECT_PREFIX.length) + if (!ALLOWED_OBJECT_PATH_RE.test(objectPath)) return signedUrl + if (!url.searchParams.has('token')) return signedUrl + + return `${trimTrailingSlash(base)}${STORAGE_PROXY_ROUTE}/${objectPath}${url.search}` +} + +/** + * Resolve the proxied path (everything after `/api/storage/`, still + * percent-encoded as received) plus its query back to the upstream Storage + * URL. Fails closed on anything outside the signed documents-bucket paths. + */ +export function resolveUpstreamStorageUrl( + objectPath: string, + search: URLSearchParams, +): StorageProxyResolution { + const upstream = upstreamStorageOrigin() + if (!upstream) return { ok: false, reason: 'storage_unconfigured' } + if (!ALLOWED_OBJECT_PATH_RE.test(objectPath) || !hasOnlyPlainSegments(objectPath)) { + return { ok: false, reason: 'unsupported_path' } + } + if (!search.get('token')) return { ok: false, reason: 'missing_token' } + + const query = search.toString() + const url = `${upstream}${UPSTREAM_OBJECT_PREFIX}${objectPath}${query ? `?${query}` : ''}` + // Belt and braces: whatever fetch() will actually request, after URL + // normalisation, must still sit inside the allowlist. + let normalisedPath: string + try { + normalisedPath = new URL(url).pathname + } catch { + return { ok: false, reason: 'unsupported_path' } + } + if ( + !normalisedPath.startsWith(UPSTREAM_OBJECT_PREFIX) || + !ALLOWED_OBJECT_PATH_RE.test(normalisedPath.slice(UPSTREAM_OBJECT_PREFIX.length)) + ) { + return { ok: false, reason: 'unsupported_path' } + } + + return { ok: true, url } +} + +/** + * Read a request body into memory, aborting as soon as it exceeds `maxBytes` + * instead of buffering an unbounded payload first and measuring afterwards + * (the proxy is unauthenticated until Storage checks the token, so a + * self-host without a platform body limit must not be forced to hold an + * arbitrary upload in RAM). Returns null when the cap is exceeded. + */ +export async function readBodyWithCap( + body: ReadableStream | null, + maxBytes: number, +): Promise { + if (!body) return new ArrayBuffer(0) + const reader = body.getReader() + const chunks: Uint8Array[] = [] + let total = 0 + try { + for (;;) { + const { done, value } = await reader.read() + if (done) break + if (!value) continue + total += value.byteLength + if (total > maxBytes) { + await reader.cancel() + return null + } + chunks.push(value) + } + } finally { + reader.releaseLock() + } + const out = new Uint8Array(total) + let offset = 0 + for (const chunk of chunks) { + out.set(chunk, offset) + offset += chunk.byteLength + } + return out.buffer +} diff --git a/lib/errors/structured-errors.ts b/lib/errors/structured-errors.ts index ac20b8fc..2e1f3419 100644 --- a/lib/errors/structured-errors.ts +++ b/lib/errors/structured-errors.ts @@ -2076,6 +2076,33 @@ const PROVIDER_MIGRATION: Record = { message_sv: 'Kunde inte importera underlag från leverantören.', message_en: 'Failed to import documents from provider.', }, + // Same-origin storage proxy (/api/storage): signed Storage URLs served + // from the app's own host for agent sandboxes that only reach the MCP host. + STORAGE_PROXY_UNSUPPORTED_PATH: { + httpStatus: 404, + message_sv: 'Sökvägen stöds inte av lagringsproxyn.', + message_en: 'The storage proxy does not serve this path.', + }, + STORAGE_PROXY_TOKEN_REQUIRED: { + httpStatus: 400, + message_sv: 'Länken saknar sin signerade token.', + message_en: 'The link is missing its signed token.', + }, + STORAGE_PROXY_BODY_TOO_LARGE: { + httpStatus: 413, + message_sv: 'Filen är för stor för att laddas upp via länken.', + message_en: 'The file is too large to upload through this link.', + }, + STORAGE_PROXY_UNCONFIGURED: { + httpStatus: 503, + message_sv: 'Lagringen är inte konfigurerad på den här servern.', + message_en: 'Storage is not configured on this server.', + }, + STORAGE_PROXY_UPSTREAM_UNAVAILABLE: { + httpStatus: 502, + message_sv: 'Lagringen svarade inte.', + message_en: 'Storage did not respond.', + }, PROVIDER_DOCUMENT_SCOPES_REQUIRED: { httpStatus: 403, message_sv: diff --git a/lib/events/types.ts b/lib/events/types.ts index 4428197b..8610b417 100644 --- a/lib/events/types.ts +++ b/lib/events/types.ts @@ -18,6 +18,14 @@ import type { // Core Event Types: discriminated union of all system events // ============================================================ +/** + * Who runs AI extraction on an uploaded document. + * - 'invoice-inbox': the inbox extracts and mirrors the result itself. + * - 'none': nobody should; the caller already holds the booking. + * Unset: the document-extraction extension extracts (default). + */ +export type DocumentExtractionOwner = 'invoice-inbox' | 'none' + export type CoreEvent = // Bookkeeping | { type: 'journal_entry.drafted'; payload: { entry: JournalEntry; userId: string; companyId: string } } @@ -29,7 +37,11 @@ export type CoreEvent = // extractionOwner: set by the invoice inbox on documents it extracts itself, // so the document-extraction extension yields instead of racing it (the // inbox row does not exist yet when this event fires inside uploadDocument). - | { type: 'document.uploaded'; payload: { document: DocumentAttachment; userId: string; companyId: string; extractionOwner?: 'invoice-inbox' } } + // 'none' is an explicit opt-out: the uploader already knows the booking + // (provider underlag import links each file to its posted verifikat), so + // running a paid model over it would buy nothing. The extension stamps the + // row as skipped instead of extracting. + | { type: 'document.uploaded'; payload: { document: DocumentAttachment; userId: string; companyId: string; extractionOwner?: DocumentExtractionOwner } } | { type: 'document.accessed'; payload: { document: { id: string; file_name: string }; userId: string; companyId: string } } | { type: 'document.deleted'; payload: { document: { id: string; file_name: string }; userId: string; companyId: string } } // Invoicing diff --git a/messages/en.json b/messages/en.json index 2144dcef..42fda610 100644 --- a/messages/en.json +++ b/messages/en.json @@ -5463,6 +5463,7 @@ "ext_arcim_documents_import_action": "Also import documents from Fortnox", "ext_arcim_documents_not_now": "Not now", "ext_arcim_documents_importing": "Importing and linking the documents to the vouchers...", + "ext_arcim_documents_import_progress": "{done} of {total} documents processed. The import continues automatically.", "ext_arcim_documents_reconnecting": "Waiting for Fortnox to reconnect...", "ext_arcim_documents_discovery_error": "The migration is complete, but we could not check the documents in Fortnox. Try again without rerunning the migration.", "ext_arcim_documents_import_error": "The migration is still complete, but the documents could not be imported. Try again; documents already imported will be skipped.", diff --git a/messages/sv.json b/messages/sv.json index f7df1a3e..766ef58c 100644 --- a/messages/sv.json +++ b/messages/sv.json @@ -5463,6 +5463,7 @@ "ext_arcim_documents_import_action": "Importera även underlag från Fortnox", "ext_arcim_documents_not_now": "Inte nu", "ext_arcim_documents_importing": "Importerar och kopplar underlagen till verifikaten...", + "ext_arcim_documents_import_progress": "{done} av {total} underlag behandlade. Importen fortsätter automatiskt.", "ext_arcim_documents_reconnecting": "Väntar på att Fortnox ska kopplas om...", "ext_arcim_documents_discovery_error": "Migreringen är klar, men vi kunde inte kontrollera underlagen i Fortnox. Försök igen utan att köra om migreringen.", "ext_arcim_documents_import_error": "Migreringen är fortfarande klar, men underlagen kunde inte importeras. Försök igen; redan importerade underlag hoppas över.",