fix(migration): resumable underlag import without inline extraction + same-origin MCP storage URLs (#1783)
* fix(migration): resumable underlag import without inline extraction, same-origin MCP storage URLs The Fortnox underlag import ran every file's AI extraction inline inside one request and hit the hosted 300 s function limit after ~17 of 113 files (twice on 2026-08-21); the UI showed the generic "underlagen kunde inte importeras" although the files it did reach were linked. The import now works in time-budgeted slices with a stable cursor (the UI loops until the server reports the end and shows "x av y") and opts out of extraction (extractionOwner 'none', stamped skipped:opted_out): every file is linked to its posted verifikat on arrival, so the booking is already known. MCP signed Storage URLs (upload_url, signed_url, download_url) are served through a same-origin proxy, /api/storage/[...path], because Claude Desktop's sandbox only reaches the MCP host and blocked the PUT to <project>.supabase.co. The signed token stays the only credential; the proxy forwards only signed documents-bucket paths to our own Storage host and is a no-op rewrite when NEXT_PUBLIC_APP_URL is unset. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013YoZ8iboyTj221axW6Gdtm * fix(mcp): keep the storage-proxy note out of the size-capped tool descriptions The per-tool 280-char cap and the tools/list payload ceiling both tripped on the two sentences added to gnubok_create_document_upload and gnubok_get_document_content; the why now lives in a code comment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013YoZ8iboyTj221axW6Gdtm * fix(review): id cursor, stall = error, capped upload body, encoded dot segments Review follow-ups on #1783: - the import cursor is the last handled provider attachment id, not an index, so a file Fortnox adds or removes mid-sweep shifts nothing - a partial answer whose cursor does not advance (or the round guard) is reported as ARCIM_DOCUMENT_IMPORT_STALLED instead of "complete"; the slices already landed stay reported and the retry button resumes - the storage proxy reads the PUT body as a capped stream instead of buffering an unbounded payload before measuring it - object paths are rejected when any segment decodes to "." or ".." (or holds a separator), and the URL fetch() would actually request is re-checked against the allowlist after normalisation - download_url description no longer claims a direct Storage URL Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013YoZ8iboyTj221axW6Gdtm --------- Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
Jakob Wennberg
parent
0531576807
commit
524d9978f1
@@ -1144,4 +1144,6 @@ One line per decision: `[YYYY-MM-DD] <decision>: <why>`. Appended by agents and
|
||||
[2026-08-21] SCHABLONINTAKT_RATE_BY_CLOSING_YEAR backfilled 2020-2024 (SLR 30 Nov per Riksgalden: -0.09/-0.10/0.23 floored to 0.5 %, 1.94 %, 2.62 %) and the rate now resolves lazily (resolveSchablonintaktRate: 0 when no 212X account carried an opening balance): the table only covered 2025/2026 and the builder consulted it unconditionally, so every AB closing a pre-2025 year got a generic 500 at bokslut step 3 (126 open FY2024 periods on prod, incl. a byra trial). 2019 and earlier stay unmapped on purpose: the 100 %-of-SLR rule keys on beskattningsar STARTING 2019-01-01+ (prop. 2017/18:245), so a 2019 closing can be a brutet ar under the old 72 % factor. Unmapped-year-with-fonder now raises SCHABLONINTAKT_RATE_NOT_CONFIGURED (typed, 500 so runtime-error clustering still flags the missed December update) instead of INTERNAL_ERROR.
|
||||
[2026-08-21] RIP-4 "optimal" auto-booking cascade, Tier 2 = the provider-agnostic account SELECTOR (lib/agent/categorize/select-account.ts), built per the 2026 research (artifact dc0c2760): the model does NOT free-form a categorizer; it CHOOSES from a closed set — the deterministic candidate accounts (Tier 1) + the 19 standard business categories (each maps deterministically to a BAS account via getDefaultAccountForCategory) + "needs_review". So the model can't invent an account, account/VAT stays deterministic and validated, and it runs on any provider (Bedrock or a local model) via getAiService().generateStructured. Founder chose the optimal path (model selects on EVERY transaction, LLM calls are fine), so confidence uses SELF-CONSISTENCY (default 3 samples, majority vote, agreement fraction) combined with the model's stated confidence and floored by the winning candidate's deterministic confidence — never the model's verbalized confidence alone (research: systematically overconfident). reasoning field precedes choice in the schema (reason-before-choice). needs_review is never auto-applied. Calibration of the combined score → the auto-book/suggest/review gate is a later tier. Not yet wired: Tier 1 candidate gathering (counterparty templates + getSuggestedCategories) + a route + the ApprovalCard UI (next PRs).
|
||||
[2026-08-21] RIP-4 cascade Tier 1 (candidate gathering) + the proposal route. lib/agent/categorize/candidates.ts assembles the deterministic candidate slate for a transaction exactly like the gnubok_suggest_categories MCP tool (mapping_rules + per-merchant history via buildMerchantHistory/getSuggestedCategories + the learned counterparty template), NO model call, deduped by account (highest confidence wins) and capped. Suggestions carry no VAT so it derives the category default (getDefaultVatTreatmentForCategory); the counterparty template carries its own. POST /api/agent/categorize runs Tier 1 → Tier 2 selectAccount and returns the proposal + the candidate slate; it NEVER posts (the caller renders an approval card). Gated on getAiStatus().configured (any provider incl. local), same gates as /api/agent/ask (auth via requireAuth, rate, membership, sandbox, capability). Next: wire the transaction row to this route + the ApprovalCard (UI, visual sign-off), then calibration + the auto-book gate.
|
||||
[2026-08-21] Provider underlag import (arcim-migration /import-documents) is now time-budgeted + resumable (200 s slice, stable provider-id order, `cursor`/`nextCursor`/`partial`, UI loops with "x av y") and opts out of AI extraction (`extractionOwner: 'none'`, stamped `skipped:opted_out`): Fabian's 113-file Fortnox import ran every file's Sonnet extraction inline inside one request, hit the hosted 300 s function limit after ~17 files twice (09:02 and 10:49 UTC) and the UI showed the generic "underlagen kunde inte importeras". The files are linked to posted verifikat on arrival, so the extraction bought nothing and cost a model call per file; chunking rather than raising maxDuration because a 2,000-file archive would still not fit. Opt-out is per-call (not "skip whenever journal_entry_id is set") to keep the diff scoped.
|
||||
[2026-08-21] MCP signed Storage URLs (gnubok_create_document_upload upload_url, gnubok_get_document_content signed_url, audit-package download_url) are served through a same-origin proxy, /api/storage/[...path] → <project>.supabase.co/storage/v1/object/{sign,upload/sign}/documents/..., instead of a Next rewrite: Claude Desktop's sandbox only reaches the MCP host (app.accounted.se) and blocked the PUT to supabase.co (Fabian, 2026-08-21). A route handler reads NEXT_PUBLIC_SUPABASE_URL at runtime (rewrites bake at build, which breaks the Docker image), is deliberately NOT withRouteContext (the signed token is the only credential, validated by Storage per object path; the proxy forwards only signed documents-bucket paths to our own host) and is a no-op rewrite when NEXT_PUBLIC_APP_URL is unset so a self-host never gets a localhost link.
|
||||
[2026-08-21] RIP-4 cascade step 3 (UI): the AI booking proposal is surfaced INSIDE the existing QuickReviewDialog rather than a new inline-row card, so it reuses that dialog's proven, deterministic, balanced commit path (POST /api/transactions/[id]/categorize) instead of a parallel one. components/transactions/AiCategorizeProposal.tsx fetches POST /api/agent/categorize on dialog open (keyed on tx.id so it remounts per transaction), pre-fills accountOverride + vatTreatment via handleAccountChange (class-2 VAT clearing preserved), and shows the confidence band (säker/trolig/välj konto) + "Varför" + the candidate alternatives (click to re-apply). Falls back SILENTLY to the deterministic defaults on error, and shows a soft note on 503 (ai_unconfigured) — the dialog always works without AI. NO silent auto-posting (founder call, avoids the storno-on-undo mess): "säker" = pre-filled, one-tap Bokför via the dialog's existing button; true hands-off auto-book waits for calibration. i18n: strings inline Swedish for now (assistant surface), lift to messages/{sv,en}.json before final merge. Confidence bands (0.8/0.5) are placeholders until calibration. Needs founder visual sign-off before merge ([[project_nav_ia_redesign]]).
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
import { createLogger } from '@/lib/logger'
|
||||
import { errorResponseFromCode } from '@/lib/errors/get-structured-error'
|
||||
import {
|
||||
STORAGE_PROXY_ROUTE,
|
||||
readBodyWithCap,
|
||||
resolveUpstreamStorageUrl,
|
||||
} from '@/lib/core/documents/storage-proxy'
|
||||
|
||||
/**
|
||||
* /api/storage/[...path]: same-origin proxy for signed Supabase Storage URLs.
|
||||
*
|
||||
* Why: agent sandboxes (Claude Desktop's code execution among them) only let
|
||||
* traffic out to the host the MCP server runs on. Signed Storage URLs point
|
||||
* at <project>.supabase.co, so the MCP model-free upload (PUT bytes to
|
||||
* upload_url) and signed document downloads were blocked inside such
|
||||
* sandboxes. The MCP tools now hand out URLs under this route instead (see
|
||||
* lib/core/documents/storage-proxy.ts), and this handler forwards them to
|
||||
* Storage unchanged.
|
||||
*
|
||||
* Auth: deliberately NOT withRouteContext. The caller is a sandbox with no
|
||||
* session, and the signed token in the query string is the credential:
|
||||
* Storage validates it against the exact object path on every request, the
|
||||
* same way it would on the public Storage host. The proxy only forwards
|
||||
* signed object paths on the documents bucket, only to our own Storage
|
||||
* origin, with a token present; it never mints, reads or relays any key.
|
||||
*
|
||||
* GET/HEAD /api/storage/sign/documents/<key>?token=... download
|
||||
* PUT /api/storage/upload/sign/documents/<key>?token=... upload
|
||||
*/
|
||||
|
||||
export const dynamic = 'force-dynamic'
|
||||
export const maxDuration = 60
|
||||
|
||||
const log = createLogger('api/storage-proxy')
|
||||
|
||||
/** Above every caller's own cap (MCP upload 10 MB, document max 20 MB). */
|
||||
const MAX_UPLOAD_BYTES = 50 * 1024 * 1024
|
||||
|
||||
const REQUEST_HEADERS_FORWARDED = [
|
||||
'content-type',
|
||||
'cache-control',
|
||||
'x-upsert',
|
||||
'range',
|
||||
'if-none-match',
|
||||
'if-modified-since',
|
||||
] as const
|
||||
|
||||
const RESPONSE_HEADERS_FORWARDED = [
|
||||
'content-type',
|
||||
'content-length',
|
||||
'content-disposition',
|
||||
'content-range',
|
||||
'accept-ranges',
|
||||
'cache-control',
|
||||
'etag',
|
||||
'last-modified',
|
||||
] as const
|
||||
|
||||
// The sandbox fetch is not a browser, but a browser-based MCP client would
|
||||
// need these and they cost nothing: the token still gates every request.
|
||||
const CORS_HEADERS: Record<string, string> = {
|
||||
'Access-Control-Allow-Origin': '*',
|
||||
'Access-Control-Allow-Methods': 'GET, HEAD, PUT, OPTIONS',
|
||||
'Access-Control-Allow-Headers': 'Content-Type, Cache-Control, x-upsert, Range',
|
||||
'Access-Control-Max-Age': '600',
|
||||
}
|
||||
|
||||
function withCors(response: Response): Response {
|
||||
for (const [key, value] of Object.entries(CORS_HEADERS)) {
|
||||
response.headers.set(key, value)
|
||||
}
|
||||
return response
|
||||
}
|
||||
|
||||
function objectPathOf(request: Request): string {
|
||||
const { pathname } = new URL(request.url)
|
||||
const prefix = `${STORAGE_PROXY_ROUTE}/`
|
||||
return pathname.startsWith(prefix) ? pathname.slice(prefix.length) : ''
|
||||
}
|
||||
|
||||
function rejected(reason: 'unsupported_path' | 'missing_token' | 'storage_unconfigured') {
|
||||
const code =
|
||||
reason === 'unsupported_path'
|
||||
? 'STORAGE_PROXY_UNSUPPORTED_PATH'
|
||||
: reason === 'missing_token'
|
||||
? 'STORAGE_PROXY_TOKEN_REQUIRED'
|
||||
: 'STORAGE_PROXY_UNCONFIGURED'
|
||||
return withCors(errorResponseFromCode(code, log))
|
||||
}
|
||||
|
||||
async function proxy(request: Request, method: 'GET' | 'HEAD' | 'PUT'): Promise<Response> {
|
||||
const url = new URL(request.url)
|
||||
const resolved = resolveUpstreamStorageUrl(objectPathOf(request), url.searchParams)
|
||||
if (!resolved.ok) return rejected(resolved.reason)
|
||||
|
||||
const headers = new Headers()
|
||||
for (const name of REQUEST_HEADERS_FORWARDED) {
|
||||
const value = request.headers.get(name)
|
||||
if (value) headers.set(name, value)
|
||||
}
|
||||
|
||||
let body: ArrayBuffer | undefined
|
||||
if (method === 'PUT') {
|
||||
const declared = Number(request.headers.get('content-length') ?? '0')
|
||||
if (Number.isFinite(declared) && declared > MAX_UPLOAD_BYTES) {
|
||||
return withCors(errorResponseFromCode('STORAGE_PROXY_BODY_TOO_LARGE', log))
|
||||
}
|
||||
// Stream with a cap: the declared length is advisory (absent or wrong),
|
||||
// and the token is only validated upstream, so never buffer an unbounded
|
||||
// body before measuring it.
|
||||
const read = await readBodyWithCap(request.body, MAX_UPLOAD_BYTES)
|
||||
if (read === null) {
|
||||
return withCors(errorResponseFromCode('STORAGE_PROXY_BODY_TOO_LARGE', log))
|
||||
}
|
||||
body = read
|
||||
if (!headers.has('content-type')) headers.set('content-type', 'application/octet-stream')
|
||||
}
|
||||
|
||||
let upstream: Response
|
||||
try {
|
||||
upstream = await fetch(resolved.url, {
|
||||
method,
|
||||
headers,
|
||||
...(body ? { body } : {}),
|
||||
redirect: 'manual',
|
||||
cache: 'no-store',
|
||||
})
|
||||
} catch (error) {
|
||||
log.error('storage proxy upstream request failed', error as Error, { method })
|
||||
return withCors(errorResponseFromCode('STORAGE_PROXY_UPSTREAM_UNAVAILABLE', log))
|
||||
}
|
||||
|
||||
const responseHeaders = new Headers(CORS_HEADERS)
|
||||
for (const name of RESPONSE_HEADERS_FORWARDED) {
|
||||
const value = upstream.headers.get(name)
|
||||
if (value) responseHeaders.set(name, value)
|
||||
}
|
||||
// Never let a served document be sniffed into something executable.
|
||||
responseHeaders.set('X-Content-Type-Options', 'nosniff')
|
||||
|
||||
return new Response(method === 'HEAD' ? null : upstream.body, {
|
||||
status: upstream.status,
|
||||
headers: responseHeaders,
|
||||
})
|
||||
}
|
||||
|
||||
export async function GET(request: Request) {
|
||||
return proxy(request, 'GET')
|
||||
}
|
||||
|
||||
export async function HEAD(request: Request) {
|
||||
return proxy(request, 'HEAD')
|
||||
}
|
||||
|
||||
export async function PUT(request: Request) {
|
||||
return proxy(request, 'PUT')
|
||||
}
|
||||
|
||||
export async function OPTIONS() {
|
||||
return new Response(null, { status: 204, headers: CORS_HEADERS })
|
||||
}
|
||||
@@ -0,0 +1,162 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import { GET, HEAD, OPTIONS, PUT } from '../[...path]/route'
|
||||
|
||||
const SUPABASE = 'https://pwxtzglxptnnvjrpixpg.supabase.co'
|
||||
|
||||
const fetchMock = vi.fn()
|
||||
|
||||
beforeEach(() => {
|
||||
vi.stubEnv('NEXT_PUBLIC_SUPABASE_URL', SUPABASE)
|
||||
vi.stubEnv('NEXT_PUBLIC_APP_URL', 'https://app.accounted.se')
|
||||
vi.stubGlobal('fetch', fetchMock)
|
||||
fetchMock.mockReset()
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
vi.unstubAllEnvs()
|
||||
vi.unstubAllGlobals()
|
||||
})
|
||||
|
||||
function upstreamResponse(body: string | null, init: ResponseInit = {}) {
|
||||
return new Response(body, {
|
||||
status: 200,
|
||||
headers: { 'content-type': 'application/pdf', 'content-length': String(body?.length ?? 0) },
|
||||
...init,
|
||||
})
|
||||
}
|
||||
|
||||
describe('/api/storage/[...path] same-origin Storage proxy', () => {
|
||||
it('forwards a signed upload PUT (bytes, content-type, token) to our Storage host', async () => {
|
||||
fetchMock.mockResolvedValue(new Response(JSON.stringify({ Key: 'documents/x' }), { status: 200, headers: { 'content-type': 'application/json' } }))
|
||||
const bytes = new TextEncoder().encode('%PDF-1.4 hello')
|
||||
const request = new Request(
|
||||
'https://app.accounted.se/api/storage/upload/sign/documents/co-1/user-1/pending/up-1/faktura%20maj.pdf?token=eyJ.sig',
|
||||
{ method: 'PUT', headers: { 'content-type': 'application/pdf', 'x-upsert': 'false' }, body: bytes },
|
||||
)
|
||||
|
||||
const response = await PUT(request)
|
||||
|
||||
expect(response.status).toBe(200)
|
||||
expect(response.headers.get('access-control-allow-origin')).toBe('*')
|
||||
expect(fetchMock).toHaveBeenCalledTimes(1)
|
||||
const [url, init] = fetchMock.mock.calls[0] as [string, RequestInit]
|
||||
expect(url).toBe(
|
||||
`${SUPABASE}/storage/v1/object/upload/sign/documents/co-1/user-1/pending/up-1/faktura%20maj.pdf?token=eyJ.sig`,
|
||||
)
|
||||
expect(init.method).toBe('PUT')
|
||||
const headers = init.headers as Headers
|
||||
expect(headers.get('content-type')).toBe('application/pdf')
|
||||
expect(headers.get('x-upsert')).toBe('false')
|
||||
expect(new TextDecoder().decode(init.body as ArrayBuffer)).toBe('%PDF-1.4 hello')
|
||||
})
|
||||
|
||||
it('streams a signed download GET through and keeps the document headers', async () => {
|
||||
fetchMock.mockResolvedValue(
|
||||
upstreamResponse('%PDF-1.4 bytes', {
|
||||
headers: {
|
||||
'content-type': 'application/pdf',
|
||||
'content-disposition': 'attachment; filename="kvitto.pdf"',
|
||||
'set-cookie': 'leak=1',
|
||||
},
|
||||
}),
|
||||
)
|
||||
const request = new Request(
|
||||
'https://app.accounted.se/api/storage/sign/documents/co-1/user-1/kvitto.pdf?token=eyJ.sig',
|
||||
)
|
||||
|
||||
const response = await GET(request)
|
||||
|
||||
expect(response.status).toBe(200)
|
||||
expect(await response.text()).toBe('%PDF-1.4 bytes')
|
||||
expect(response.headers.get('content-type')).toBe('application/pdf')
|
||||
expect(response.headers.get('content-disposition')).toBe('attachment; filename="kvitto.pdf"')
|
||||
expect(response.headers.get('x-content-type-options')).toBe('nosniff')
|
||||
expect(response.headers.get('set-cookie')).toBeNull()
|
||||
const [url, init] = fetchMock.mock.calls[0] as [string, RequestInit]
|
||||
expect(url).toBe(`${SUPABASE}/storage/v1/object/sign/documents/co-1/user-1/kvitto.pdf?token=eyJ.sig`)
|
||||
expect(init.method).toBe('GET')
|
||||
})
|
||||
|
||||
it('answers HEAD without a body and relays the upstream status', async () => {
|
||||
fetchMock.mockResolvedValue(new Response(null, { status: 404, headers: { 'content-type': 'application/json' } }))
|
||||
|
||||
const response = await HEAD(
|
||||
new Request('https://app.accounted.se/api/storage/sign/documents/co-1/missing.pdf?token=t', { method: 'HEAD' }),
|
||||
)
|
||||
|
||||
expect(response.status).toBe(404)
|
||||
expect(response.body).toBeNull()
|
||||
})
|
||||
|
||||
it('refuses paths outside the signed documents-bucket allowlist without touching Storage', async () => {
|
||||
const response = await GET(
|
||||
new Request('https://app.accounted.se/api/storage/public/documents/a.pdf?token=t'),
|
||||
)
|
||||
|
||||
expect(response.status).toBe(404)
|
||||
const body = (await response.json()) as { error: { code: string } }
|
||||
expect(body.error.code).toBe('STORAGE_PROXY_UNSUPPORTED_PATH')
|
||||
expect(fetchMock).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('refuses a link without its signed token', async () => {
|
||||
const response = await GET(
|
||||
new Request('https://app.accounted.se/api/storage/sign/documents/co-1/a.pdf'),
|
||||
)
|
||||
|
||||
expect(response.status).toBe(400)
|
||||
const body = (await response.json()) as { error: { code: string } }
|
||||
expect(body.error.code).toBe('STORAGE_PROXY_TOKEN_REQUIRED')
|
||||
expect(fetchMock).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('rejects an oversized upload before forwarding it', async () => {
|
||||
const request = new Request(
|
||||
'https://app.accounted.se/api/storage/upload/sign/documents/co-1/big.pdf?token=t',
|
||||
{ method: 'PUT', headers: { 'content-length': String(51 * 1024 * 1024) }, body: 'x' },
|
||||
)
|
||||
|
||||
const response = await PUT(request)
|
||||
|
||||
expect(response.status).toBe(413)
|
||||
expect(fetchMock).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('rejects an oversized upload that lies about (or omits) its content-length, without buffering it all', async () => {
|
||||
let pulled = 0
|
||||
const endless = new ReadableStream<Uint8Array>({
|
||||
pull(controller) {
|
||||
pulled++
|
||||
controller.enqueue(new Uint8Array(1024 * 1024))
|
||||
},
|
||||
})
|
||||
const request = new Request(
|
||||
'https://app.accounted.se/api/storage/upload/sign/documents/co-1/big.pdf?token=t',
|
||||
{ method: 'PUT', body: endless, duplex: 'half' } as RequestInit,
|
||||
)
|
||||
|
||||
const response = await PUT(request)
|
||||
|
||||
expect(response.status).toBe(413)
|
||||
expect(fetchMock).not.toHaveBeenCalled()
|
||||
expect(pulled).toBeLessThan(60)
|
||||
})
|
||||
|
||||
it('reports Storage being unreachable as 502 instead of crashing', async () => {
|
||||
fetchMock.mockRejectedValue(new Error('ECONNRESET'))
|
||||
|
||||
const response = await GET(
|
||||
new Request('https://app.accounted.se/api/storage/sign/documents/co-1/a.pdf?token=t'),
|
||||
)
|
||||
|
||||
expect(response.status).toBe(502)
|
||||
const body = (await response.json()) as { error: { code: string } }
|
||||
expect(body.error.code).toBe('STORAGE_PROXY_UPSTREAM_UNAVAILABLE')
|
||||
})
|
||||
|
||||
it('answers CORS preflight', async () => {
|
||||
const response = await OPTIONS()
|
||||
expect(response.status).toBe(204)
|
||||
expect(response.headers.get('access-control-allow-methods')).toContain('PUT')
|
||||
})
|
||||
})
|
||||
@@ -37,6 +37,7 @@ import {
|
||||
parseArcimDocumentOAuthResume,
|
||||
PROVIDER_DOCUMENT_SCOPES_UNAVAILABLE,
|
||||
requestArcimDocumentImport,
|
||||
runArcimDocumentImportToCompletion,
|
||||
resolveArcimDocumentFollowUpProvider,
|
||||
watchArcimOAuthPopup,
|
||||
type ArcimDocumentImportProblem,
|
||||
@@ -1477,10 +1478,26 @@ function DocumentImportFollowUp({
|
||||
? t('ext_arcim_documents_importing')
|
||||
: t('ext_arcim_documents_reconnecting')
|
||||
|
||||
// Running totals arrive after each server slice of a real import; the
|
||||
// dry-run result that sits in state while the first slice runs is not
|
||||
// progress, so it stays silent.
|
||||
const progress =
|
||||
state.phase === 'importing' && state.result && !state.result.dryRun && state.result.total > 0
|
||||
? state.result
|
||||
: null
|
||||
|
||||
return (
|
||||
<section className="space-y-3" aria-live="polite">
|
||||
{title}
|
||||
<SpinnerLine>{label}</SpinnerLine>
|
||||
{progress && (
|
||||
<p className="text-sm tabular-nums text-muted-foreground">
|
||||
{t('ext_arcim_documents_import_progress', {
|
||||
done: progress.scanned,
|
||||
total: progress.total,
|
||||
})}
|
||||
</p>
|
||||
)}
|
||||
</section>
|
||||
)
|
||||
}
|
||||
@@ -2316,7 +2333,12 @@ export default function ArcimMigrationWorkspace({
|
||||
const runDocumentImport = useCallback(async (currentConsentId: string) => {
|
||||
dispatchDocumentImport({ type: 'import-started' })
|
||||
try {
|
||||
const result = await requestArcimDocumentImport(currentConsentId, false)
|
||||
// One route call per time-budgeted slice; the helper loops until the
|
||||
// server reports the end and feeds running totals back for the UI.
|
||||
const result = await runArcimDocumentImportToCompletion(currentConsentId, {
|
||||
onProgress: (progress) =>
|
||||
dispatchDocumentImport({ type: 'import-progress', result: progress }),
|
||||
})
|
||||
dispatchDocumentImport({ type: 'import-succeeded', result })
|
||||
} catch (documentError) {
|
||||
dispatchDocumentImport({
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { describe, expect, it, vi } from 'vitest'
|
||||
import {
|
||||
ARCIM_DOCUMENT_IMPORT_STALLED,
|
||||
ARCIM_DOCUMENT_OAUTH_RESUME_KEY,
|
||||
INITIAL_ARCIM_DOCUMENT_IMPORT_STATE,
|
||||
PROVIDER_DOCUMENT_SCOPES_REQUIRED,
|
||||
@@ -8,8 +9,10 @@ import {
|
||||
arcimDocumentImportReducer,
|
||||
documentOAuthProblemFromReason,
|
||||
parseArcimDocumentOAuthResume,
|
||||
mergeArcimDocumentImportResults,
|
||||
requestArcimDocumentImport,
|
||||
resolveArcimDocumentFollowUpProvider,
|
||||
runArcimDocumentImportToCompletion,
|
||||
watchArcimOAuthPopup,
|
||||
type ArcimDocumentImportResult,
|
||||
} from '../arcim-document-import-flow'
|
||||
@@ -26,10 +29,20 @@ function result(
|
||||
failed: 0,
|
||||
dryRun: true,
|
||||
unmatchedSamples: [],
|
||||
total: 7,
|
||||
partial: false,
|
||||
nextCursor: null,
|
||||
...overrides,
|
||||
}
|
||||
}
|
||||
|
||||
function jsonResponse(body: unknown, status = 200): Response {
|
||||
return new Response(JSON.stringify(body), {
|
||||
status,
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
})
|
||||
}
|
||||
|
||||
describe('Fortnox document follow-up state', () => {
|
||||
it('offers the prompt after a successful Fortnox migration using the honest found count', () => {
|
||||
const discovering = arcimDocumentImportReducer(
|
||||
@@ -298,3 +311,123 @@ describe('document scope OAuth recovery', () => {
|
||||
vi.useRealTimers()
|
||||
})
|
||||
})
|
||||
|
||||
describe('resumable import (one server slice per call)', () => {
|
||||
it('accepts an older server answer without resume fields as one complete slice', async () => {
|
||||
const { total: _t, partial: _p, nextCursor: _c, ...legacy } = result({ scanned: 9 })
|
||||
const fetcher = vi.fn().mockResolvedValue(jsonResponse({ success: true, result: legacy }))
|
||||
|
||||
const normalized = await requestArcimDocumentImport('consent-1', true, fetcher)
|
||||
|
||||
expect(normalized).toMatchObject({ total: 9, partial: false, nextCursor: null })
|
||||
expect(JSON.parse((fetcher.mock.calls[0][1] as RequestInit).body as string)).toEqual({
|
||||
consentId: 'consent-1',
|
||||
dryRun: true,
|
||||
})
|
||||
})
|
||||
|
||||
it('sends the cursor only when resuming', async () => {
|
||||
const fetcher = vi.fn().mockResolvedValue(jsonResponse({ success: true, result: result() }))
|
||||
|
||||
await requestArcimDocumentImport('consent-1', false, fetcher, 'file-42')
|
||||
|
||||
expect(JSON.parse((fetcher.mock.calls[0][1] as RequestInit).body as string)).toEqual({
|
||||
consentId: 'consent-1',
|
||||
dryRun: false,
|
||||
cursor: 'file-42',
|
||||
})
|
||||
})
|
||||
|
||||
it('sums every slice and keeps the latest total', () => {
|
||||
const merged = mergeArcimDocumentImportResults(
|
||||
result({ scanned: 17, linked: 15, skipped: 1, unmatched: 1, failed: 0, total: 113, unmatchedSamples: [{ uploadId: 'u1', voucher: 'A1', date: '2026-01-01' }] }),
|
||||
result({ scanned: 20, linked: 18, skipped: 0, unmatched: 1, failed: 1, total: 113, partial: true, nextCursor: 'file-37', unmatchedSamples: [{ uploadId: 'u2', voucher: 'A2', date: '2026-01-02' }] }),
|
||||
)
|
||||
|
||||
expect(merged).toMatchObject({
|
||||
scanned: 37,
|
||||
linked: 33,
|
||||
skipped: 1,
|
||||
unmatched: 2,
|
||||
failed: 1,
|
||||
total: 113,
|
||||
partial: true,
|
||||
nextCursor: 'file-37',
|
||||
})
|
||||
expect(merged.unmatchedSamples).toHaveLength(2)
|
||||
})
|
||||
|
||||
it('loops until the server reports the end, passing the cursor back and reporting running totals', async () => {
|
||||
const fetcher = vi
|
||||
.fn()
|
||||
.mockResolvedValueOnce(
|
||||
jsonResponse({ success: true, result: result({ dryRun: false, scanned: 17, linked: 17, total: 40, partial: true, nextCursor: 'file-17' }) }),
|
||||
)
|
||||
.mockResolvedValueOnce(
|
||||
jsonResponse({ success: true, result: result({ dryRun: false, scanned: 17, linked: 16, skipped: 1, total: 40, partial: true, nextCursor: 'file-34' }) }),
|
||||
)
|
||||
.mockResolvedValueOnce(
|
||||
jsonResponse({ success: true, result: result({ dryRun: false, scanned: 6, linked: 6, total: 40 }) }),
|
||||
)
|
||||
const onProgress = vi.fn()
|
||||
|
||||
const final = await runArcimDocumentImportToCompletion('consent-1', { fetcher, onProgress })
|
||||
|
||||
expect(fetcher).toHaveBeenCalledTimes(3)
|
||||
const bodies = fetcher.mock.calls.map((call) => JSON.parse((call[1] as RequestInit).body as string))
|
||||
expect(bodies).toEqual([
|
||||
{ consentId: 'consent-1', dryRun: false },
|
||||
{ consentId: 'consent-1', dryRun: false, cursor: 'file-17' },
|
||||
{ consentId: 'consent-1', dryRun: false, cursor: 'file-34' },
|
||||
])
|
||||
expect(onProgress).toHaveBeenCalledTimes(2)
|
||||
expect(onProgress.mock.calls[1][0]).toMatchObject({ scanned: 34, linked: 33, skipped: 1, total: 40 })
|
||||
expect(final).toMatchObject({ scanned: 40, linked: 39, skipped: 1, total: 40, partial: false, nextCursor: null })
|
||||
})
|
||||
|
||||
it('fails (does not report completion) when a server hands back a cursor that does not advance', async () => {
|
||||
const fetcher = vi
|
||||
.fn()
|
||||
.mockResolvedValueOnce(jsonResponse({ success: true, result: result({ dryRun: false, scanned: 5, total: 10, partial: true, nextCursor: 'file-5' }) }))
|
||||
.mockResolvedValueOnce(jsonResponse({ success: true, result: result({ dryRun: false, scanned: 0, total: 10, partial: true, nextCursor: 'file-5' }) }))
|
||||
const onProgress = vi.fn()
|
||||
|
||||
await expect(
|
||||
runArcimDocumentImportToCompletion('consent-1', { fetcher, onProgress }),
|
||||
).rejects.toMatchObject({ problem: { code: ARCIM_DOCUMENT_IMPORT_STALLED } })
|
||||
|
||||
expect(fetcher).toHaveBeenCalledTimes(2)
|
||||
// The slice that did land was reported, so the UI keeps honest totals.
|
||||
expect(onProgress).toHaveBeenCalledTimes(1)
|
||||
expect(onProgress.mock.calls[0][0]).toMatchObject({ scanned: 5, total: 10 })
|
||||
})
|
||||
|
||||
it('fails when a partial answer carries no cursor at all', async () => {
|
||||
const fetcher = vi.fn().mockResolvedValue(
|
||||
jsonResponse({ success: true, result: result({ dryRun: false, scanned: 5, total: 10, partial: true, nextCursor: null }) }),
|
||||
)
|
||||
|
||||
await expect(runArcimDocumentImportToCompletion('consent-1', { fetcher })).rejects.toMatchObject({
|
||||
problem: { code: ARCIM_DOCUMENT_IMPORT_STALLED },
|
||||
})
|
||||
expect(fetcher).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
|
||||
it('keeps the import phase while recording running totals, then completes with the honest total', () => {
|
||||
const importing = arcimDocumentImportReducer(
|
||||
{ phase: 'offered', found: 40, result: result({ dryRun: true, total: 40 }), problem: null },
|
||||
{ type: 'import-started' },
|
||||
)
|
||||
const progressed = arcimDocumentImportReducer(importing, {
|
||||
type: 'import-progress',
|
||||
result: result({ dryRun: false, scanned: 17, linked: 17, total: 40, partial: true, nextCursor: 'file-17' }),
|
||||
})
|
||||
expect(progressed).toMatchObject({ phase: 'importing', result: { scanned: 17, total: 40 } })
|
||||
|
||||
const complete = arcimDocumentImportReducer(progressed, {
|
||||
type: 'import-succeeded',
|
||||
result: result({ dryRun: false, scanned: 40, linked: 39, skipped: 1, total: 40 }),
|
||||
})
|
||||
expect(complete).toMatchObject({ phase: 'complete', found: 40, result: { linked: 39, skipped: 1 } })
|
||||
})
|
||||
})
|
||||
|
||||
@@ -63,6 +63,42 @@ export interface ArcimDocumentImportResult {
|
||||
failed: number
|
||||
dryRun: boolean
|
||||
unmatchedSamples: { uploadId: string; voucher: string; date: string }[]
|
||||
/** Attachments in the provider's whole list (not just this call). */
|
||||
total: number
|
||||
/** The server stopped at its time budget; continue after nextCursor. */
|
||||
partial: boolean
|
||||
/** Provider id of the last handled attachment, or null when complete. */
|
||||
nextCursor: string | null
|
||||
}
|
||||
|
||||
const MAX_UNMATCHED_SAMPLES = 20
|
||||
|
||||
/**
|
||||
* Fold one server call into the running totals of a multi-call import. The
|
||||
* route processes a slice per call (hosted function limit), so the numbers
|
||||
* the user sees must be the sum of every slice, not the last one.
|
||||
*/
|
||||
export function mergeArcimDocumentImportResults(
|
||||
accumulated: ArcimDocumentImportResult | null,
|
||||
next: ArcimDocumentImportResult,
|
||||
): ArcimDocumentImportResult {
|
||||
if (!accumulated) return next
|
||||
return {
|
||||
provider: next.provider,
|
||||
dryRun: next.dryRun,
|
||||
scanned: accumulated.scanned + next.scanned,
|
||||
linked: accumulated.linked + next.linked,
|
||||
skipped: accumulated.skipped + next.skipped,
|
||||
unmatched: accumulated.unmatched + next.unmatched,
|
||||
failed: accumulated.failed + next.failed,
|
||||
unmatchedSamples: [...accumulated.unmatchedSamples, ...next.unmatchedSamples].slice(
|
||||
0,
|
||||
MAX_UNMATCHED_SAMPLES,
|
||||
),
|
||||
total: next.total,
|
||||
partial: next.partial,
|
||||
nextCursor: next.nextCursor,
|
||||
}
|
||||
}
|
||||
|
||||
export interface ArcimDocumentImportProblem {
|
||||
@@ -133,6 +169,7 @@ export type ArcimDocumentImportAction =
|
||||
| { type: 'discovery-failed'; problem: ArcimDocumentImportProblem }
|
||||
| { type: 'dismissed' }
|
||||
| { type: 'import-started' }
|
||||
| { type: 'import-progress'; result: ArcimDocumentImportResult }
|
||||
| { type: 'import-succeeded'; result: ArcimDocumentImportResult }
|
||||
| { type: 'import-failed'; problem: ArcimDocumentImportProblem }
|
||||
| { type: 'reconnect-started' }
|
||||
@@ -195,10 +232,14 @@ export function arcimDocumentImportReducer(
|
||||
return { ...state, phase: 'dismissed', problem: null }
|
||||
case 'import-started':
|
||||
return { ...state, phase: 'importing', problem: null }
|
||||
case 'import-progress':
|
||||
// Running totals after each server slice; the phase stays 'importing'
|
||||
// so the UI keeps its spinner and shows "x av y".
|
||||
return { ...state, phase: 'importing', result: action.result, problem: null }
|
||||
case 'import-succeeded':
|
||||
return {
|
||||
phase: 'complete',
|
||||
found: state.found || action.result.scanned,
|
||||
found: state.found || action.result.total || action.result.scanned,
|
||||
result: action.result,
|
||||
problem: null,
|
||||
}
|
||||
@@ -242,7 +283,13 @@ function problemFromPayload(payload: unknown): ArcimDocumentImportProblem {
|
||||
}
|
||||
}
|
||||
|
||||
function isDocumentImportResult(value: unknown): value is ArcimDocumentImportResult {
|
||||
type WireDocumentImportResult = Omit<
|
||||
ArcimDocumentImportResult,
|
||||
'total' | 'partial' | 'nextCursor'
|
||||
> &
|
||||
Partial<Pick<ArcimDocumentImportResult, 'total' | 'partial' | 'nextCursor'>>
|
||||
|
||||
function isDocumentImportResult(value: unknown): value is WireDocumentImportResult {
|
||||
if (!value || typeof value !== 'object') return false
|
||||
const result = value as Partial<ArcimDocumentImportResult>
|
||||
return (
|
||||
@@ -257,16 +304,37 @@ function isDocumentImportResult(value: unknown): value is ArcimDocumentImportRes
|
||||
)
|
||||
}
|
||||
|
||||
/** Call the existing POST route for both discovery and the actual import. */
|
||||
/** Older servers answer without the resume fields: a single complete slice. */
|
||||
function normalizeDocumentImportResult(
|
||||
result: WireDocumentImportResult,
|
||||
): ArcimDocumentImportResult {
|
||||
return {
|
||||
...result,
|
||||
total: typeof result.total === 'number' ? result.total : result.scanned,
|
||||
partial: result.partial === true,
|
||||
nextCursor:
|
||||
typeof result.nextCursor === 'string' && result.nextCursor.length > 0 && result.partial === true
|
||||
? result.nextCursor
|
||||
: null,
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Call the existing POST route for both discovery and the actual import.
|
||||
* `cursor` resumes a partial import; omit it for discovery and the first slice.
|
||||
*/
|
||||
export async function requestArcimDocumentImport(
|
||||
consentId: string,
|
||||
dryRun: boolean,
|
||||
fetcher: typeof fetch = fetch,
|
||||
cursor?: string,
|
||||
): Promise<ArcimDocumentImportResult> {
|
||||
const response = await fetcher(ARCIM_DOCUMENT_IMPORT_ENDPOINT, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ consentId, dryRun }),
|
||||
body: JSON.stringify(
|
||||
cursor === undefined ? { consentId, dryRun } : { consentId, dryRun, cursor },
|
||||
),
|
||||
})
|
||||
const payload = await response.json().catch(() => null)
|
||||
|
||||
@@ -283,5 +351,50 @@ export async function requestArcimDocumentImport(
|
||||
})
|
||||
}
|
||||
|
||||
return result
|
||||
return normalizeDocumentImportResult(result)
|
||||
}
|
||||
|
||||
/** Far above any real list: a guard against a server that never completes. */
|
||||
const MAX_IMPORT_ROUNDS = 500
|
||||
|
||||
export const ARCIM_DOCUMENT_IMPORT_STALLED = 'ARCIM_DOCUMENT_IMPORT_STALLED'
|
||||
|
||||
/**
|
||||
* Run the real import to completion: the route works through one time-budgeted
|
||||
* slice per call and hands back a cursor, so this loops until the server says
|
||||
* it reached the end, reporting running totals after every slice. A thrown
|
||||
* request error aborts the loop; the slices already archived stay linked and
|
||||
* the next attempt skips them by hash. A server that keeps answering "partial"
|
||||
* without advancing (or past the round guard) is reported as a failure, never
|
||||
* as a completed import: the retry button resumes safely.
|
||||
*/
|
||||
export async function runArcimDocumentImportToCompletion(
|
||||
consentId: string,
|
||||
options: {
|
||||
fetcher?: typeof fetch
|
||||
onProgress?: (accumulated: ArcimDocumentImportResult) => void
|
||||
} = {},
|
||||
): Promise<ArcimDocumentImportResult> {
|
||||
const fetcher = options.fetcher ?? fetch
|
||||
let accumulated: ArcimDocumentImportResult | null = null
|
||||
let cursor: string | undefined
|
||||
|
||||
for (let round = 0; round < MAX_IMPORT_ROUNDS; round++) {
|
||||
const slice = await requestArcimDocumentImport(consentId, false, fetcher, cursor)
|
||||
accumulated = mergeArcimDocumentImportResults(accumulated, slice)
|
||||
if (!slice.partial) {
|
||||
return { ...accumulated, partial: false, nextCursor: null }
|
||||
}
|
||||
if (slice.nextCursor === null || (cursor !== undefined && slice.nextCursor <= cursor)) {
|
||||
break
|
||||
}
|
||||
cursor = slice.nextCursor
|
||||
options.onProgress?.(accumulated)
|
||||
}
|
||||
|
||||
throw new ArcimDocumentImportRequestError({
|
||||
code: ARCIM_DOCUMENT_IMPORT_STALLED,
|
||||
requestId: null,
|
||||
reconnectRequired: false,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -152,6 +152,45 @@ describe('POST /import-documents', () => {
|
||||
)
|
||||
})
|
||||
|
||||
it('passes a non-empty string cursor through and restarts from the top for anything else', async () => {
|
||||
;(importProviderDocuments as Mock).mockResolvedValue({
|
||||
provider: 'fortnox',
|
||||
scanned: 1,
|
||||
linked: 1,
|
||||
skipped: 0,
|
||||
unmatched: 0,
|
||||
failed: 0,
|
||||
dryRun: false,
|
||||
unmatchedSamples: [],
|
||||
total: 113,
|
||||
partial: true,
|
||||
nextCursor: 'file-18',
|
||||
})
|
||||
|
||||
const withCursor = createMockRequest(
|
||||
'http://localhost/api/extensions/ext/arcim-migration/import-documents',
|
||||
{ method: 'POST', body: { consentId: 'consent-1', dryRun: false, cursor: 'file-17' } },
|
||||
)
|
||||
const { status, body } = await parseJsonResponse<{
|
||||
result: { partial: boolean; nextCursor: string | null }
|
||||
}>(await handler(withCursor, buildContext()))
|
||||
|
||||
expect(status).toBe(200)
|
||||
expect(body.result).toMatchObject({ partial: true, nextCursor: 'file-18' })
|
||||
expect(importProviderDocuments).toHaveBeenLastCalledWith(
|
||||
expect.objectContaining({ consentId: 'consent-1', dryRun: false, cursor: 'file-17' }),
|
||||
)
|
||||
|
||||
const garbage = createMockRequest(
|
||||
'http://localhost/api/extensions/ext/arcim-migration/import-documents',
|
||||
{ method: 'POST', body: { consentId: 'consent-1', cursor: 17 } },
|
||||
)
|
||||
await handler(garbage, buildContext())
|
||||
expect(importProviderDocuments).toHaveBeenLastCalledWith(
|
||||
expect.objectContaining({ cursor: null }),
|
||||
)
|
||||
})
|
||||
|
||||
it('asks the user to reconnect only once the connect request carries the scopes', async () => {
|
||||
fortnoxOAuth.documentScopesApproved = true
|
||||
;(importProviderDocuments as Mock).mockRejectedValue(
|
||||
|
||||
@@ -177,11 +177,16 @@ describe('importProviderDocuments', () => {
|
||||
expect(userId).toBe(USER)
|
||||
expect(companyId).toBe(COMPANY)
|
||||
expect(file).toMatchObject({ name: 'Kvitto.pdf', type: 'application/pdf' })
|
||||
// extractionOwner 'none': the file arrives linked to its posted
|
||||
// verifikat, so no paid model pass (it was the inline extraction that
|
||||
// blew the hosted function budget on a 113-file import).
|
||||
expect(metadata).toEqual({
|
||||
upload_source: 'api',
|
||||
journal_entry_id: 'je-1',
|
||||
idempotency_key: 'je-1',
|
||||
extractionOwner: 'none',
|
||||
})
|
||||
expect(result).toMatchObject({ total: 1, partial: false, nextCursor: null })
|
||||
})
|
||||
|
||||
it('skips a receipt already archived on the same verifikat (sha256 + journal entry idempotency)', async () => {
|
||||
@@ -268,6 +273,7 @@ describe('importProviderDocuments', () => {
|
||||
upload_source: 'api',
|
||||
journal_entry_id: 'je-1',
|
||||
idempotency_key: 'je-1',
|
||||
extractionOwner: 'none',
|
||||
})
|
||||
})
|
||||
|
||||
@@ -648,4 +654,123 @@ describe('importProviderDocuments', () => {
|
||||
const [, , , file] = mockUpload.mock.calls[0]
|
||||
expect(file).toMatchObject({ name: 'Kvitto.jpg', type: 'image/jpeg' })
|
||||
})
|
||||
|
||||
describe('time budget and cursor (resumable sweep)', () => {
|
||||
const THREE_CONNECTIONS: FortnoxFileConnection[] = [
|
||||
{ ...FORTNOX_CONNECTION, fileId: 'file-c', name: 'c.pdf' },
|
||||
{ ...FORTNOX_CONNECTION, fileId: 'file-a', name: 'a.pdf' },
|
||||
{ ...FORTNOX_CONNECTION, fileId: 'file-b', name: 'b.pdf' },
|
||||
]
|
||||
|
||||
it('stops after the attachment in flight once the budget is spent and hands back a cursor', async () => {
|
||||
const supabase = wireFortnox({ connections: THREE_CONNECTIONS })
|
||||
// Each download "costs" 100 ms of wall clock; the budget allows one.
|
||||
let clock = 0
|
||||
mockDownloadFortnoxArchiveFile.mockImplementation(async () => {
|
||||
clock += 100
|
||||
return { bytes: bytesOf('FORTNOX-PDF-' + clock), contentType: 'application/pdf' }
|
||||
})
|
||||
|
||||
const first = await importProviderDocuments({
|
||||
supabase,
|
||||
companyId: COMPANY,
|
||||
userId: USER,
|
||||
consentId: 'c1',
|
||||
timeBudgetMs: 50,
|
||||
now: () => clock,
|
||||
})
|
||||
|
||||
expect(first).toMatchObject({
|
||||
total: 3,
|
||||
scanned: 1,
|
||||
linked: 1,
|
||||
partial: true,
|
||||
nextCursor: 'file-a',
|
||||
})
|
||||
// Sorted by provider id, so the first slice is file-a, not file-c.
|
||||
expect(mockUpload.mock.calls[0][3]).toMatchObject({ name: 'a.pdf' })
|
||||
|
||||
const second = await importProviderDocuments({
|
||||
supabase,
|
||||
companyId: COMPANY,
|
||||
userId: USER,
|
||||
consentId: 'c1',
|
||||
cursor: first.nextCursor,
|
||||
timeBudgetMs: 1_000_000,
|
||||
now: () => clock,
|
||||
})
|
||||
|
||||
expect(second).toMatchObject({ total: 3, scanned: 2, linked: 2, partial: false, nextCursor: null })
|
||||
expect(mockUpload.mock.calls.map((call) => (call[3] as { name: string }).name)).toEqual([
|
||||
'a.pdf',
|
||||
'b.pdf',
|
||||
'c.pdf',
|
||||
])
|
||||
})
|
||||
|
||||
it('always processes at least one attachment per call even when the budget is already spent', async () => {
|
||||
const supabase = wireFortnox({ connections: THREE_CONNECTIONS })
|
||||
|
||||
const result = await importProviderDocuments({
|
||||
supabase,
|
||||
companyId: COMPANY,
|
||||
userId: USER,
|
||||
consentId: 'c1',
|
||||
timeBudgetMs: 0,
|
||||
now: () => 0,
|
||||
})
|
||||
|
||||
expect(result).toMatchObject({ scanned: 1, linked: 1, partial: true, nextCursor: 'file-a' })
|
||||
})
|
||||
|
||||
it('treats a cursor past the last id as a complete, empty slice', async () => {
|
||||
const supabase = wireFortnox({ connections: THREE_CONNECTIONS })
|
||||
|
||||
const result = await importProviderDocuments({
|
||||
supabase,
|
||||
companyId: COMPANY,
|
||||
userId: USER,
|
||||
consentId: 'c1',
|
||||
cursor: 'file-z',
|
||||
})
|
||||
|
||||
expect(result).toMatchObject({ total: 3, scanned: 0, linked: 0, partial: false, nextCursor: null })
|
||||
expect(mockUpload).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('resumes after the last handled id even when the provider list changed in between', async () => {
|
||||
// Slice 1 handled file-a. Before slice 2, Fortnox gained a file sorting
|
||||
// BEFORE the resume point (file-0) and lost file-b. An index cursor
|
||||
// would now skip one; an id cursor continues with everything > file-a.
|
||||
const supabase = wireFortnox({
|
||||
connections: [
|
||||
{ ...FORTNOX_CONNECTION, fileId: 'file-0', name: '0.pdf' },
|
||||
{ ...FORTNOX_CONNECTION, fileId: 'file-a', name: 'a.pdf' },
|
||||
{ ...FORTNOX_CONNECTION, fileId: 'file-c', name: 'c.pdf' },
|
||||
{ ...FORTNOX_CONNECTION, fileId: 'file-d', name: 'd.pdf' },
|
||||
],
|
||||
})
|
||||
// Distinct bytes per file: the default mock returns the same bytes for
|
||||
// every download, and identical content on the same verifikat is
|
||||
// (correctly) deduped, which is not what this test is about.
|
||||
mockDownloadFortnoxArchiveFile.mockImplementation(async (_client, _token, id) => ({
|
||||
bytes: bytesOf('PDF-' + id),
|
||||
contentType: 'application/pdf',
|
||||
}))
|
||||
|
||||
const result = await importProviderDocuments({
|
||||
supabase,
|
||||
companyId: COMPANY,
|
||||
userId: USER,
|
||||
consentId: 'c1',
|
||||
cursor: 'file-a',
|
||||
})
|
||||
|
||||
expect(result).toMatchObject({ total: 4, scanned: 2, linked: 2, partial: false })
|
||||
expect(mockUpload.mock.calls.map((call) => (call[3] as { name: string }).name)).toEqual([
|
||||
'c.pdf',
|
||||
'd.pdf',
|
||||
])
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
@@ -1363,10 +1363,23 @@ export const arcimMigrationExtension: Extension = {
|
||||
|
||||
let consentId: string | undefined
|
||||
let dryRun = false
|
||||
let cursor: string | null = null
|
||||
try {
|
||||
const body = (await request.json()) as { consentId?: string; dryRun?: boolean }
|
||||
const body = (await request.json()) as {
|
||||
consentId?: string
|
||||
dryRun?: boolean
|
||||
cursor?: string
|
||||
}
|
||||
consentId = body?.consentId
|
||||
dryRun = body?.dryRun === true
|
||||
// Resume point from a previous partial call (the last handled
|
||||
// provider attachment id, see import-documents.ts); anything else
|
||||
// restarts from the top, which is always safe: already-archived
|
||||
// receipts are skipped by hash.
|
||||
cursor =
|
||||
typeof body?.cursor === 'string' && body.cursor.length > 0 && body.cursor.length <= 256
|
||||
? body.cursor
|
||||
: null
|
||||
} catch {
|
||||
// empty/invalid body: consentId check below rejects it
|
||||
}
|
||||
@@ -1382,15 +1395,20 @@ export const arcimMigrationExtension: Extension = {
|
||||
userId: user.id,
|
||||
consentId,
|
||||
dryRun,
|
||||
cursor,
|
||||
})
|
||||
log.info('arcim import-documents completed', {
|
||||
companyId,
|
||||
dryRun,
|
||||
cursor,
|
||||
total: result.total,
|
||||
scanned: result.scanned,
|
||||
linked: result.linked,
|
||||
skipped: result.skipped,
|
||||
unmatched: result.unmatched,
|
||||
failed: result.failed,
|
||||
partial: result.partial,
|
||||
nextCursor: result.nextCursor,
|
||||
})
|
||||
return NextResponse.json({ success: true, dryRun, result })
|
||||
} catch (error) {
|
||||
|
||||
@@ -24,6 +24,22 @@
|
||||
* critical path: provider document APIs are rate-limited and a full sweep can
|
||||
* issue hundreds of download calls. Fortnox requires archive and connectfile
|
||||
* scopes; existing consents must reconnect before this import can run.
|
||||
*
|
||||
* Resumable: one call works through the provider's attachment list (sorted
|
||||
* by provider id so the order is stable between calls) until `timeBudgetMs`
|
||||
* is spent, then returns `partial: true` with `nextCursor` = the id of the
|
||||
* last attachment it handled. The caller loops, passing the cursor back,
|
||||
* until `partial` is false. The cursor is an id, not an index, so a file the
|
||||
* provider adds or removes mid-sweep shifts nothing: the next slice simply
|
||||
* continues after the last handled id in sorted order. A 113-file Fortnox
|
||||
* import used to run every file inline inside one request and hit the
|
||||
* hosted 300 s function limit after ~17 files (2026-08-21); the UI then
|
||||
* showed a generic failure even though the files it did reach were linked.
|
||||
*
|
||||
* No AI extraction: every file here is linked to a posted verifikat on
|
||||
* arrival, so the booking is already known. The upload opts out
|
||||
* (`extractionOwner: 'none'`); it was the per-file model pass that made the
|
||||
* sweep blow its budget in the first place.
|
||||
*/
|
||||
|
||||
import type { SupabaseClient } from '@supabase/supabase-js'
|
||||
@@ -78,8 +94,20 @@ export interface ImportDocumentsOptions {
|
||||
consentId: string
|
||||
/** Resolve + report what would be attached without downloading or writing. */
|
||||
dryRun?: boolean
|
||||
/** Resume after this provider attachment id (sorted order); omit to start from the top. */
|
||||
cursor?: string | null
|
||||
/**
|
||||
* Wall-clock budget for this call. Once spent, the sweep stops after the
|
||||
* attachment in flight and reports `nextCursor`. Default leaves headroom
|
||||
* under the hosted 300 s function limit for listing + the final response.
|
||||
*/
|
||||
timeBudgetMs?: number
|
||||
/** Clock, injectable for tests. */
|
||||
now?: () => number
|
||||
}
|
||||
|
||||
export const DEFAULT_IMPORT_DOCUMENTS_TIME_BUDGET_MS = 200_000
|
||||
|
||||
export interface ImportDocumentsResult {
|
||||
provider: string
|
||||
/** Provider attachments linked to a voucher that were considered. */
|
||||
@@ -95,6 +123,12 @@ export interface ImportDocumentsResult {
|
||||
dryRun: boolean
|
||||
/** A few unmatched voucher labels, to aid diagnosis without dumping all. */
|
||||
unmatchedSamples: { uploadId: string; voucher: string; date: string }[]
|
||||
/** Provider attachments linked to a voucher in total (all calls). */
|
||||
total: number
|
||||
/** True when the time budget ran out before the end of the list. */
|
||||
partial: boolean
|
||||
/** Last handled attachment id; pass back as `cursor` to continue. Null when complete. */
|
||||
nextCursor: string | null
|
||||
}
|
||||
|
||||
interface ProviderAttachment {
|
||||
@@ -232,7 +266,17 @@ function fortnoxSource(
|
||||
export async function importProviderDocuments(
|
||||
opts: ImportDocumentsOptions,
|
||||
): Promise<ImportDocumentsResult> {
|
||||
const { supabase, companyId, userId, consentId, dryRun = false } = opts
|
||||
const {
|
||||
supabase,
|
||||
companyId,
|
||||
userId,
|
||||
consentId,
|
||||
dryRun = false,
|
||||
cursor = null,
|
||||
timeBudgetMs = DEFAULT_IMPORT_DOCUMENTS_TIME_BUDGET_MS,
|
||||
now = Date.now,
|
||||
} = opts
|
||||
const startedAt = now()
|
||||
|
||||
let resolved = await resolveConsent(companyId, consentId)
|
||||
const provider = resolved.consent.provider as string
|
||||
@@ -246,6 +290,9 @@ export async function importProviderDocuments(
|
||||
failed: 0,
|
||||
dryRun,
|
||||
unmatchedSamples: [],
|
||||
total: 0,
|
||||
partial: false,
|
||||
nextCursor: null,
|
||||
}
|
||||
|
||||
// Unsupported providers are a no-op rather than an error
|
||||
@@ -263,7 +310,7 @@ export async function importProviderDocuments(
|
||||
: fortnoxSource(fortnoxClient as FortnoxClient, resolved.accessToken)
|
||||
|
||||
// ── Bulk reads (one round of paged requests each, no per-item N+1) ──
|
||||
const [attachments, periods, vouchers, existingAttachments] = await Promise.all([
|
||||
const [listedAttachments, periods, vouchers, existingAttachments] = await Promise.all([
|
||||
source().list(),
|
||||
fetchFiscalPeriods(supabase, companyId),
|
||||
fetchSourceRefVouchers(supabase, companyId),
|
||||
@@ -280,6 +327,15 @@ export async function importProviderDocuments(
|
||||
),
|
||||
])
|
||||
|
||||
// Stable order across calls: a provider may page its listing differently
|
||||
// from one call to the next, and the cursor is "continue after this id".
|
||||
const attachments = [...listedAttachments].sort((a, b) =>
|
||||
a.id < b.id ? -1 : a.id > b.id ? 1 : 0,
|
||||
)
|
||||
result.total = attachments.length
|
||||
const firstAfterCursor = cursor ? attachments.findIndex((a) => a.id > cursor) : 0
|
||||
const startIndex = firstAfterCursor === -1 ? attachments.length : firstAfterCursor
|
||||
|
||||
// Index gnubok verifikat by (period, series, number) for in-memory resolution.
|
||||
const voucherIndex = buildVoucherIndex(vouchers)
|
||||
|
||||
@@ -302,7 +358,16 @@ export async function importProviderDocuments(
|
||||
|
||||
let refreshedAfterUnauthorized = false
|
||||
|
||||
for (const attachment of attachments) {
|
||||
for (let index = startIndex; index < attachments.length; index++) {
|
||||
// Always make progress: at least one attachment per call, then stop at
|
||||
// the budget so the route answers well inside the function limit and the
|
||||
// caller resumes after the last handled id.
|
||||
if (index > startIndex && now() - startedAt >= timeBudgetMs) {
|
||||
result.partial = true
|
||||
result.nextCursor = attachments[index - 1].id
|
||||
break
|
||||
}
|
||||
const attachment = attachments[index]
|
||||
result.scanned++
|
||||
const ref = attachment.ref
|
||||
|
||||
@@ -363,6 +428,8 @@ export async function importProviderDocuments(
|
||||
upload_source: 'api',
|
||||
journal_entry_id: journalEntryId,
|
||||
idempotency_key: journalEntryId,
|
||||
// Already booked: the verifikat is the booking. No model pass.
|
||||
extractionOwner: 'none',
|
||||
},
|
||||
)
|
||||
|
||||
@@ -405,11 +472,16 @@ export async function importProviderDocuments(
|
||||
log.info('document import complete', {
|
||||
companyId,
|
||||
dryRun,
|
||||
cursor,
|
||||
total: result.total,
|
||||
scanned: result.scanned,
|
||||
linked: result.linked,
|
||||
skipped: result.skipped,
|
||||
unmatched: result.unmatched,
|
||||
failed: result.failed,
|
||||
partial: result.partial,
|
||||
nextCursor: result.nextCursor,
|
||||
elapsedMs: now() - startedAt,
|
||||
})
|
||||
|
||||
return result
|
||||
|
||||
@@ -64,6 +64,13 @@ beforeEach(() => {
|
||||
describe('document-extraction handler', () => {
|
||||
// THE dedupe: inbox-owned documents are extracted (and mirrored) by the
|
||||
// inbox itself. The handler used to race it and pay a second model call.
|
||||
it('stamps and skips when the uploader opted out (already-booked provider underlag)', async () => {
|
||||
await handler(payload({ extractionOwner: 'none' }))
|
||||
|
||||
expect(extractMock).not.toHaveBeenCalled()
|
||||
expect(lastStamp()).toBe('skipped:opted_out')
|
||||
})
|
||||
|
||||
it('yields entirely when the inbox owns extraction', async () => {
|
||||
await handler(payload({ extractionOwner: 'invoice-inbox' }))
|
||||
expect(supabase.from).not.toHaveBeenCalled()
|
||||
|
||||
@@ -7,6 +7,7 @@ import { CAPABILITY } from '@/lib/entitlements/keys'
|
||||
import { createLogger } from '@/lib/logger'
|
||||
import { createServiceClient } from '@/lib/supabase/server'
|
||||
import type { DocumentAttachment } from '@/types'
|
||||
import type { DocumentExtractionOwner } from '@/lib/events/types'
|
||||
|
||||
const log = createLogger('document-extraction')
|
||||
|
||||
@@ -62,9 +63,17 @@ export const documentExtractionExtension: Extension = {
|
||||
document: DocumentAttachment
|
||||
userId: string
|
||||
companyId: string
|
||||
extractionOwner?: 'invoice-inbox'
|
||||
extractionOwner?: DocumentExtractionOwner
|
||||
}
|
||||
if (extractionOwner === 'invoice-inbox') return
|
||||
if (extractionOwner === 'none') {
|
||||
// The uploader already holds the booking (provider underlag import
|
||||
// links each file to its posted verifikat): a model pass would cost
|
||||
// a paid call per file and, run inline, blew the 300 s route budget
|
||||
// on a 113-file Fortnox import (2026-08-21). Stamp so nothing polls.
|
||||
await stamp(createServiceClient(), document.id, 'skipped:opted_out')
|
||||
return
|
||||
}
|
||||
await extractAndPersist(document, companyId)
|
||||
},
|
||||
},
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import { makeDocumentAttachment } from '@/tests/helpers'
|
||||
import { TOOL_SCOPE_MAP } from '@/lib/auth/api-keys'
|
||||
import { MCP_TOOL_CAPABILITY_MAP } from '@/lib/entitlements/keys'
|
||||
@@ -51,6 +51,10 @@ function makeQueryBuilder(result: { data: unknown; error: unknown }) {
|
||||
}
|
||||
|
||||
describe('MCP model-free document upload tools', () => {
|
||||
afterEach(() => {
|
||||
vi.unstubAllEnvs()
|
||||
})
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
mocks.createPendingDocumentUpload.mockResolvedValue({
|
||||
@@ -76,6 +80,32 @@ describe('MCP model-free document upload tools', () => {
|
||||
})
|
||||
})
|
||||
|
||||
// Claude Desktop's sandbox only reaches the MCP host: a signed URL on
|
||||
// <project>.supabase.co was refused there (2026-08-21), so the tool hands
|
||||
// out the same-origin /api/storage proxy URL instead.
|
||||
it('serves the upload URL from the app origin when the signed URL points at our Storage host', async () => {
|
||||
vi.stubEnv('NEXT_PUBLIC_SUPABASE_URL', 'https://pwxtzglxptnnvjrpixpg.supabase.co')
|
||||
vi.stubEnv('NEXT_PUBLIC_APP_URL', 'https://app.accounted.se')
|
||||
mocks.createPendingDocumentUpload.mockResolvedValue({
|
||||
uploadId,
|
||||
signedUrl:
|
||||
'https://pwxtzglxptnnvjrpixpg.supabase.co/storage/v1/object/upload/sign/documents/co/user/pending/up/invoice.pdf?token=signed',
|
||||
expiresAt: '2026-08-03T12:00:00.000Z',
|
||||
})
|
||||
|
||||
const result = await findTool('gnubok_create_document_upload').execute(
|
||||
{ file_name: 'invoice.pdf' },
|
||||
companyId,
|
||||
userId,
|
||||
{} as never,
|
||||
)
|
||||
|
||||
expect(result).toMatchObject({
|
||||
upload_url:
|
||||
'https://app.accounted.se/api/storage/upload/sign/documents/co/user/pending/up/invoice.pdf?token=signed',
|
||||
})
|
||||
})
|
||||
|
||||
it('returns an unauthenticated PUT URL without accepting file bytes', async () => {
|
||||
const tool = findTool('gnubok_create_document_upload')
|
||||
const result = await tool.execute(
|
||||
|
||||
@@ -209,6 +209,7 @@ import {
|
||||
uploadDocument,
|
||||
MAX_DOCUMENT_SIZE,
|
||||
} from '@/lib/core/documents/document-service'
|
||||
import { toSameOriginStorageUrl } from '@/lib/core/documents/storage-proxy'
|
||||
import { extractInvoiceFields, ExtractionSchema as InvoiceExtractionSchema, AgentExtractionSchema } from '@/extensions/general/invoice-inbox/lib/extract-invoice-fields'
|
||||
import { mirrorExtractionToDocument } from '@/extensions/general/invoice-inbox/lib/mirror-extraction'
|
||||
// Skatteverket filing tools (PR5). Cross-extension lib import, same sanctioned
|
||||
@@ -9715,9 +9716,11 @@ export const tools: McpTool[] = [
|
||||
uploadId,
|
||||
fileName,
|
||||
)
|
||||
// Served from the app origin: agent sandboxes (Claude Desktop) only
|
||||
// reach the MCP host, not <project>.supabase.co. See storage-proxy.ts.
|
||||
return {
|
||||
upload_id: reservation.uploadId,
|
||||
upload_url: reservation.signedUrl,
|
||||
upload_url: toSameOriginStorageUrl(reservation.signedUrl),
|
||||
expires_at: reservation.expiresAt,
|
||||
}
|
||||
},
|
||||
@@ -10664,7 +10667,7 @@ export const tools: McpTool[] = [
|
||||
file_name: doc.file_name,
|
||||
mime_type: doc.mime_type,
|
||||
size_bytes: doc.file_size_bytes,
|
||||
signed_url: signed.signedUrl,
|
||||
signed_url: toSameOriginStorageUrl(signed.signedUrl),
|
||||
expires_at: expiresAt,
|
||||
}
|
||||
},
|
||||
@@ -13574,7 +13577,7 @@ export const tools: McpTool[] = [
|
||||
type: 'object',
|
||||
additionalProperties: false,
|
||||
properties: {
|
||||
download_url: { type: ['string', 'null'], description: 'Signed Supabase Storage URL, valid 1 hour; null when estimate_only=true. Restricted-egress proxies may 403; needs a network with Supabase egress.' },
|
||||
download_url: { type: ['string', 'null'], description: 'Signed download URL, valid 1 hour, on the app origin (direct Storage URL only when NEXT_PUBLIC_APP_URL is unset); null when estimate_only=true.' },
|
||||
storage_path: { type: ['string', 'null'] },
|
||||
file_name: { type: 'string' },
|
||||
size_bytes: { type: 'number' },
|
||||
@@ -13679,7 +13682,7 @@ export const tools: McpTool[] = [
|
||||
const expiresAt = new Date(Date.now() + SIGNED_URL_TTL_SECONDS * 1000).toISOString()
|
||||
|
||||
return {
|
||||
download_url: signed.signedUrl,
|
||||
download_url: toSameOriginStorageUrl(signed.signedUrl),
|
||||
storage_path: storagePath,
|
||||
file_name: fileName,
|
||||
size_bytes: zipBuffer.byteLength,
|
||||
|
||||
@@ -0,0 +1,190 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import {
|
||||
readBodyWithCap,
|
||||
resolveUpstreamStorageUrl,
|
||||
toSameOriginStorageUrl,
|
||||
} from '../storage-proxy'
|
||||
|
||||
const SUPABASE = 'https://pwxtzglxptnnvjrpixpg.supabase.co'
|
||||
const APP = 'https://app.accounted.se'
|
||||
const UPLOAD_URL = `${SUPABASE}/storage/v1/object/upload/sign/documents/co-1/user-1/pending/up-1/faktura%20maj.pdf?token=eyJ.sig`
|
||||
const DOWNLOAD_URL = `${SUPABASE}/storage/v1/object/sign/documents/co-1/user-1/kvitto.pdf?token=eyJ.sig&download=`
|
||||
|
||||
beforeEach(() => {
|
||||
vi.stubEnv('NEXT_PUBLIC_SUPABASE_URL', SUPABASE)
|
||||
vi.stubEnv('NEXT_PUBLIC_APP_URL', APP)
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
vi.unstubAllEnvs()
|
||||
})
|
||||
|
||||
describe('toSameOriginStorageUrl', () => {
|
||||
it('moves a signed upload URL onto the app origin, keeping path encoding and the token', () => {
|
||||
expect(toSameOriginStorageUrl(UPLOAD_URL)).toBe(
|
||||
`${APP}/api/storage/upload/sign/documents/co-1/user-1/pending/up-1/faktura%20maj.pdf?token=eyJ.sig`,
|
||||
)
|
||||
})
|
||||
|
||||
it('moves a signed download URL onto the app origin with every query parameter', () => {
|
||||
expect(toSameOriginStorageUrl(DOWNLOAD_URL)).toBe(
|
||||
`${APP}/api/storage/sign/documents/co-1/user-1/kvitto.pdf?token=eyJ.sig&download=`,
|
||||
)
|
||||
})
|
||||
|
||||
it('tolerates a trailing slash on the app URL', () => {
|
||||
expect(toSameOriginStorageUrl(DOWNLOAD_URL, `${APP}/`)).toMatch(
|
||||
new RegExp(`^${APP}/api/storage/sign/`),
|
||||
)
|
||||
})
|
||||
|
||||
it('leaves the URL alone when the app has no public URL (self-host without NEXT_PUBLIC_APP_URL)', () => {
|
||||
vi.stubEnv('NEXT_PUBLIC_APP_URL', '')
|
||||
expect(toSameOriginStorageUrl(UPLOAD_URL)).toBe(UPLOAD_URL)
|
||||
expect(toSameOriginStorageUrl(UPLOAD_URL, undefined)).toBe(UPLOAD_URL)
|
||||
})
|
||||
|
||||
it('leaves URLs on other hosts, other buckets, unsigned paths and token-less links alone', () => {
|
||||
expect(toSameOriginStorageUrl('https://storage.example/upload?token=signed')).toBe(
|
||||
'https://storage.example/upload?token=signed',
|
||||
)
|
||||
const otherBucket = `${SUPABASE}/storage/v1/object/sign/avatars/a.png?token=t`
|
||||
expect(toSameOriginStorageUrl(otherBucket)).toBe(otherBucket)
|
||||
const publicObject = `${SUPABASE}/storage/v1/object/public/documents/a.pdf`
|
||||
expect(toSameOriginStorageUrl(publicObject)).toBe(publicObject)
|
||||
const noToken = `${SUPABASE}/storage/v1/object/sign/documents/a.pdf`
|
||||
expect(toSameOriginStorageUrl(noToken)).toBe(noToken)
|
||||
expect(toSameOriginStorageUrl('not a url')).toBe('not a url')
|
||||
})
|
||||
})
|
||||
|
||||
describe('resolveUpstreamStorageUrl', () => {
|
||||
it('rebuilds the upstream signed URL for upload and download paths', () => {
|
||||
expect(
|
||||
resolveUpstreamStorageUrl(
|
||||
'upload/sign/documents/co-1/user-1/pending/up-1/faktura%20maj.pdf',
|
||||
new URLSearchParams('token=eyJ.sig'),
|
||||
),
|
||||
).toEqual({
|
||||
ok: true,
|
||||
url: `${SUPABASE}/storage/v1/object/upload/sign/documents/co-1/user-1/pending/up-1/faktura%20maj.pdf?token=eyJ.sig`,
|
||||
})
|
||||
expect(
|
||||
resolveUpstreamStorageUrl(
|
||||
'sign/documents/co-1/user-1/kvitto.pdf',
|
||||
new URLSearchParams('token=eyJ.sig&download='),
|
||||
),
|
||||
).toEqual({
|
||||
ok: true,
|
||||
url: `${SUPABASE}/storage/v1/object/sign/documents/co-1/user-1/kvitto.pdf?token=eyJ.sig&download=`,
|
||||
})
|
||||
})
|
||||
|
||||
it('fails closed on anything that is not a signed documents-bucket object path', () => {
|
||||
const token = new URLSearchParams('token=t')
|
||||
expect(resolveUpstreamStorageUrl('public/documents/a.pdf', token)).toEqual({
|
||||
ok: false,
|
||||
reason: 'unsupported_path',
|
||||
})
|
||||
expect(resolveUpstreamStorageUrl('sign/avatars/a.png', token)).toEqual({
|
||||
ok: false,
|
||||
reason: 'unsupported_path',
|
||||
})
|
||||
expect(resolveUpstreamStorageUrl('sign/documents/', token)).toEqual({
|
||||
ok: false,
|
||||
reason: 'unsupported_path',
|
||||
})
|
||||
expect(resolveUpstreamStorageUrl('sign/documents/../bucket/x', token)).toEqual({
|
||||
ok: false,
|
||||
reason: 'unsupported_path',
|
||||
})
|
||||
expect(resolveUpstreamStorageUrl('list/documents', token)).toEqual({
|
||||
ok: false,
|
||||
reason: 'unsupported_path',
|
||||
})
|
||||
})
|
||||
|
||||
it('rejects dot segments in every spelling the URL parser would normalise away', () => {
|
||||
const token = new URLSearchParams('token=t')
|
||||
for (const evil of [
|
||||
'sign/documents/%2e%2e/avatars/x',
|
||||
'sign/documents/.%2e/avatars/x',
|
||||
'sign/documents/%2e./avatars/x',
|
||||
'sign/documents/%2E%2E/avatars/x',
|
||||
'sign/documents/a/../../avatars/x',
|
||||
'sign/documents/./x',
|
||||
'sign/documents/a//x',
|
||||
'sign/documents/a%5c..%5cavatars/x',
|
||||
'sign/documents/a\\..\\avatars/x',
|
||||
'sign/documents/a%2f..%2favatars/x',
|
||||
'sign/documents/%zz/x',
|
||||
'upload/sign/documents/%2e%2e/avatars/x',
|
||||
]) {
|
||||
expect(resolveUpstreamStorageUrl(evil, token), evil).toEqual({
|
||||
ok: false,
|
||||
reason: 'unsupported_path',
|
||||
})
|
||||
}
|
||||
})
|
||||
|
||||
it('still accepts real keys with spaces, non-ASCII and dots inside names', () => {
|
||||
const token = new URLSearchParams('token=t')
|
||||
const resolved = resolveUpstreamStorageUrl(
|
||||
'sign/documents/co-1/user-1/faktura%20maj%20%C3%A5r.2026.pdf',
|
||||
token,
|
||||
)
|
||||
expect(resolved.ok).toBe(true)
|
||||
})
|
||||
|
||||
it('requires the signed token', () => {
|
||||
expect(resolveUpstreamStorageUrl('sign/documents/a.pdf', new URLSearchParams())).toEqual({
|
||||
ok: false,
|
||||
reason: 'missing_token',
|
||||
})
|
||||
})
|
||||
|
||||
it('reports an unconfigured Storage host instead of guessing', () => {
|
||||
vi.stubEnv('NEXT_PUBLIC_SUPABASE_URL', '')
|
||||
expect(
|
||||
resolveUpstreamStorageUrl('sign/documents/a.pdf', new URLSearchParams('token=t')),
|
||||
).toEqual({ ok: false, reason: 'storage_unconfigured' })
|
||||
})
|
||||
})
|
||||
|
||||
describe('readBodyWithCap', () => {
|
||||
function stream(chunks: Uint8Array[]): ReadableStream<Uint8Array> {
|
||||
return new ReadableStream({
|
||||
start(controller) {
|
||||
for (const chunk of chunks) controller.enqueue(chunk)
|
||||
controller.close()
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
it('concatenates a body under the cap', async () => {
|
||||
const out = await readBodyWithCap(
|
||||
stream([new TextEncoder().encode('%PDF'), new TextEncoder().encode('-1.4')]),
|
||||
100,
|
||||
)
|
||||
expect(out && new TextDecoder().decode(out)).toBe('%PDF-1.4')
|
||||
})
|
||||
|
||||
it('returns null as soon as the running total passes the cap, without draining the rest', async () => {
|
||||
let pulled = 0
|
||||
const endless = new ReadableStream<Uint8Array>({
|
||||
pull(controller) {
|
||||
pulled++
|
||||
controller.enqueue(new Uint8Array(1024))
|
||||
},
|
||||
})
|
||||
|
||||
const out = await readBodyWithCap(endless, 4096)
|
||||
|
||||
expect(out).toBeNull()
|
||||
expect(pulled).toBeLessThan(10)
|
||||
})
|
||||
|
||||
it('treats a missing body as empty', async () => {
|
||||
expect((await readBodyWithCap(null, 10))?.byteLength).toBe(0)
|
||||
})
|
||||
})
|
||||
@@ -1,6 +1,7 @@
|
||||
import type { SupabaseClient } from '@supabase/supabase-js'
|
||||
import { createServiceClientNoCookies } from '@/lib/auth/api-keys'
|
||||
import { eventBus } from '@/lib/events'
|
||||
import type { DocumentExtractionOwner } from '@/lib/events/types'
|
||||
import type { DocumentAttachment, DocumentUploadSource } from '@/types'
|
||||
|
||||
/**
|
||||
@@ -504,7 +505,7 @@ export async function completePendingDocumentUpload(
|
||||
fileName: string,
|
||||
mimeType: string,
|
||||
now: number = Date.now(),
|
||||
options: { extractionOwner?: 'invoice-inbox' } = {}
|
||||
options: { extractionOwner?: DocumentExtractionOwner } = {}
|
||||
): Promise<CompletedPendingDocumentUpload> {
|
||||
const serviceClient = createServiceClientNoCookies()
|
||||
const storage = serviceClient.storage.from(DOCUMENTS_BUCKET)
|
||||
@@ -656,9 +657,10 @@ export async function uploadDocument(
|
||||
* Who runs AI extraction on this document. The invoice inbox extracts
|
||||
* the documents it ingests itself (and mirrors the result onto the
|
||||
* document row), so it declares ownership here and the
|
||||
* document-extraction extension yields. Default: the extension extracts.
|
||||
* document-extraction extension yields. 'none' opts out entirely (the
|
||||
* caller already knows the booking). Default: the extension extracts.
|
||||
*/
|
||||
extractionOwner?: 'invoice-inbox'
|
||||
extractionOwner?: DocumentExtractionOwner
|
||||
} = {}
|
||||
): Promise<DocumentAttachment & { deduplicated?: boolean }> {
|
||||
await ensureDocumentsBucket()
|
||||
|
||||
@@ -0,0 +1,177 @@
|
||||
/**
|
||||
* Same-origin proxy for Supabase Storage signed URLs.
|
||||
*
|
||||
* Agent sandboxes (Claude Desktop's code execution, some MCP clients) only
|
||||
* allow network egress to the host the MCP server lives on. Our signed
|
||||
* Storage URLs point at <project>.supabase.co, so a model-free upload or a
|
||||
* document download from such a sandbox was refused before it left the box
|
||||
* (user report 2026-08-21). These helpers rewrite a signed URL onto this
|
||||
* app's own origin (`/api/storage/...`) and resolve it back to the upstream
|
||||
* Storage URL inside the proxy route.
|
||||
*
|
||||
* The signed token travels unchanged and is the only credential on both
|
||||
* sides: the proxy grants nothing the public Storage host did not already
|
||||
* grant, and it refuses every path that is not a signed object path on the
|
||||
* documents bucket. Without NEXT_PUBLIC_APP_URL (a self-host that never set
|
||||
* it) the rewrite is a no-op rather than a broken localhost link.
|
||||
*/
|
||||
|
||||
export const STORAGE_PROXY_ROUTE = '/api/storage'
|
||||
|
||||
/** Upstream prefix under the Storage API that every signed object URL shares. */
|
||||
const UPSTREAM_OBJECT_PREFIX = '/storage/v1/object/'
|
||||
|
||||
/**
|
||||
* Only token-authenticated object paths on the documents bucket: signed
|
||||
* downloads (`sign/documents/...`) and signed uploads
|
||||
* (`upload/sign/documents/...`). Anything else (public objects, bucket
|
||||
* admin, other buckets) is not this proxy's business.
|
||||
*/
|
||||
const ALLOWED_OBJECT_PATH_RE = /^(sign|upload\/sign)\/documents\/[^/]+/
|
||||
|
||||
export type StorageProxyResolution =
|
||||
| { ok: true; url: string }
|
||||
| { ok: false; reason: 'unsupported_path' | 'missing_token' | 'storage_unconfigured' }
|
||||
|
||||
/**
|
||||
* True when every segment of the (still percent-encoded) object path is a
|
||||
* plain name: no empty segment, no `.`/`..` in raw or percent-encoded form
|
||||
* (`%2e%2e`, `.%2e`, `%2e.`: the WHATWG URL parser normalises those away,
|
||||
* which would let `sign/documents/%2e%2e/other/x` leave the documents
|
||||
* bucket), no backslash (a path separator for https URLs), no malformed
|
||||
* escapes.
|
||||
*/
|
||||
function hasOnlyPlainSegments(objectPath: string): boolean {
|
||||
for (const segment of objectPath.split('/')) {
|
||||
if (segment === '') return false
|
||||
let decoded: string
|
||||
try {
|
||||
decoded = decodeURIComponent(segment)
|
||||
} catch {
|
||||
return false
|
||||
}
|
||||
if (decoded === '.' || decoded === '..') return false
|
||||
if (decoded.includes('\\') || decoded.includes('/')) return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
function trimTrailingSlash(value: string): string {
|
||||
return value.replace(/\/+$/, '')
|
||||
}
|
||||
|
||||
function upstreamStorageOrigin(): string | null {
|
||||
const raw = process.env.NEXT_PUBLIC_SUPABASE_URL?.trim()
|
||||
if (!raw) return null
|
||||
try {
|
||||
return new URL(raw).origin
|
||||
} catch {
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Rewrite a Supabase signed Storage URL onto this app's origin. Returns the
|
||||
* input unchanged when it is not a signed documents-bucket URL on our Storage
|
||||
* host, or when the app has no public URL configured.
|
||||
*/
|
||||
export function toSameOriginStorageUrl(
|
||||
signedUrl: string,
|
||||
appBaseUrl: string | undefined = process.env.NEXT_PUBLIC_APP_URL,
|
||||
): string {
|
||||
const base = appBaseUrl?.trim()
|
||||
const upstream = upstreamStorageOrigin()
|
||||
if (!base || !upstream) return signedUrl
|
||||
|
||||
let url: URL
|
||||
try {
|
||||
url = new URL(signedUrl)
|
||||
} catch {
|
||||
return signedUrl
|
||||
}
|
||||
if (url.origin !== upstream) return signedUrl
|
||||
if (!url.pathname.startsWith(UPSTREAM_OBJECT_PREFIX)) return signedUrl
|
||||
|
||||
// Keep the pathname exactly as Storage encoded it: object keys may hold
|
||||
// spaces and non-ASCII, and the token was signed over the real key.
|
||||
const objectPath = url.pathname.slice(UPSTREAM_OBJECT_PREFIX.length)
|
||||
if (!ALLOWED_OBJECT_PATH_RE.test(objectPath)) return signedUrl
|
||||
if (!url.searchParams.has('token')) return signedUrl
|
||||
|
||||
return `${trimTrailingSlash(base)}${STORAGE_PROXY_ROUTE}/${objectPath}${url.search}`
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the proxied path (everything after `/api/storage/`, still
|
||||
* percent-encoded as received) plus its query back to the upstream Storage
|
||||
* URL. Fails closed on anything outside the signed documents-bucket paths.
|
||||
*/
|
||||
export function resolveUpstreamStorageUrl(
|
||||
objectPath: string,
|
||||
search: URLSearchParams,
|
||||
): StorageProxyResolution {
|
||||
const upstream = upstreamStorageOrigin()
|
||||
if (!upstream) return { ok: false, reason: 'storage_unconfigured' }
|
||||
if (!ALLOWED_OBJECT_PATH_RE.test(objectPath) || !hasOnlyPlainSegments(objectPath)) {
|
||||
return { ok: false, reason: 'unsupported_path' }
|
||||
}
|
||||
if (!search.get('token')) return { ok: false, reason: 'missing_token' }
|
||||
|
||||
const query = search.toString()
|
||||
const url = `${upstream}${UPSTREAM_OBJECT_PREFIX}${objectPath}${query ? `?${query}` : ''}`
|
||||
// Belt and braces: whatever fetch() will actually request, after URL
|
||||
// normalisation, must still sit inside the allowlist.
|
||||
let normalisedPath: string
|
||||
try {
|
||||
normalisedPath = new URL(url).pathname
|
||||
} catch {
|
||||
return { ok: false, reason: 'unsupported_path' }
|
||||
}
|
||||
if (
|
||||
!normalisedPath.startsWith(UPSTREAM_OBJECT_PREFIX) ||
|
||||
!ALLOWED_OBJECT_PATH_RE.test(normalisedPath.slice(UPSTREAM_OBJECT_PREFIX.length))
|
||||
) {
|
||||
return { ok: false, reason: 'unsupported_path' }
|
||||
}
|
||||
|
||||
return { ok: true, url }
|
||||
}
|
||||
|
||||
/**
|
||||
* Read a request body into memory, aborting as soon as it exceeds `maxBytes`
|
||||
* instead of buffering an unbounded payload first and measuring afterwards
|
||||
* (the proxy is unauthenticated until Storage checks the token, so a
|
||||
* self-host without a platform body limit must not be forced to hold an
|
||||
* arbitrary upload in RAM). Returns null when the cap is exceeded.
|
||||
*/
|
||||
export async function readBodyWithCap(
|
||||
body: ReadableStream<Uint8Array> | null,
|
||||
maxBytes: number,
|
||||
): Promise<ArrayBuffer | null> {
|
||||
if (!body) return new ArrayBuffer(0)
|
||||
const reader = body.getReader()
|
||||
const chunks: Uint8Array[] = []
|
||||
let total = 0
|
||||
try {
|
||||
for (;;) {
|
||||
const { done, value } = await reader.read()
|
||||
if (done) break
|
||||
if (!value) continue
|
||||
total += value.byteLength
|
||||
if (total > maxBytes) {
|
||||
await reader.cancel()
|
||||
return null
|
||||
}
|
||||
chunks.push(value)
|
||||
}
|
||||
} finally {
|
||||
reader.releaseLock()
|
||||
}
|
||||
const out = new Uint8Array(total)
|
||||
let offset = 0
|
||||
for (const chunk of chunks) {
|
||||
out.set(chunk, offset)
|
||||
offset += chunk.byteLength
|
||||
}
|
||||
return out.buffer
|
||||
}
|
||||
@@ -2076,6 +2076,33 @@ const PROVIDER_MIGRATION: Record<string, StructuredErrorEntry> = {
|
||||
message_sv: 'Kunde inte importera underlag från leverantören.',
|
||||
message_en: 'Failed to import documents from provider.',
|
||||
},
|
||||
// Same-origin storage proxy (/api/storage): signed Storage URLs served
|
||||
// from the app's own host for agent sandboxes that only reach the MCP host.
|
||||
STORAGE_PROXY_UNSUPPORTED_PATH: {
|
||||
httpStatus: 404,
|
||||
message_sv: 'Sökvägen stöds inte av lagringsproxyn.',
|
||||
message_en: 'The storage proxy does not serve this path.',
|
||||
},
|
||||
STORAGE_PROXY_TOKEN_REQUIRED: {
|
||||
httpStatus: 400,
|
||||
message_sv: 'Länken saknar sin signerade token.',
|
||||
message_en: 'The link is missing its signed token.',
|
||||
},
|
||||
STORAGE_PROXY_BODY_TOO_LARGE: {
|
||||
httpStatus: 413,
|
||||
message_sv: 'Filen är för stor för att laddas upp via länken.',
|
||||
message_en: 'The file is too large to upload through this link.',
|
||||
},
|
||||
STORAGE_PROXY_UNCONFIGURED: {
|
||||
httpStatus: 503,
|
||||
message_sv: 'Lagringen är inte konfigurerad på den här servern.',
|
||||
message_en: 'Storage is not configured on this server.',
|
||||
},
|
||||
STORAGE_PROXY_UPSTREAM_UNAVAILABLE: {
|
||||
httpStatus: 502,
|
||||
message_sv: 'Lagringen svarade inte.',
|
||||
message_en: 'Storage did not respond.',
|
||||
},
|
||||
PROVIDER_DOCUMENT_SCOPES_REQUIRED: {
|
||||
httpStatus: 403,
|
||||
message_sv:
|
||||
|
||||
+13
-1
@@ -18,6 +18,14 @@ import type {
|
||||
// Core Event Types: discriminated union of all system events
|
||||
// ============================================================
|
||||
|
||||
/**
|
||||
* Who runs AI extraction on an uploaded document.
|
||||
* - 'invoice-inbox': the inbox extracts and mirrors the result itself.
|
||||
* - 'none': nobody should; the caller already holds the booking.
|
||||
* Unset: the document-extraction extension extracts (default).
|
||||
*/
|
||||
export type DocumentExtractionOwner = 'invoice-inbox' | 'none'
|
||||
|
||||
export type CoreEvent =
|
||||
// Bookkeeping
|
||||
| { type: 'journal_entry.drafted'; payload: { entry: JournalEntry; userId: string; companyId: string } }
|
||||
@@ -29,7 +37,11 @@ export type CoreEvent =
|
||||
// extractionOwner: set by the invoice inbox on documents it extracts itself,
|
||||
// so the document-extraction extension yields instead of racing it (the
|
||||
// inbox row does not exist yet when this event fires inside uploadDocument).
|
||||
| { type: 'document.uploaded'; payload: { document: DocumentAttachment; userId: string; companyId: string; extractionOwner?: 'invoice-inbox' } }
|
||||
// 'none' is an explicit opt-out: the uploader already knows the booking
|
||||
// (provider underlag import links each file to its posted verifikat), so
|
||||
// running a paid model over it would buy nothing. The extension stamps the
|
||||
// row as skipped instead of extracting.
|
||||
| { type: 'document.uploaded'; payload: { document: DocumentAttachment; userId: string; companyId: string; extractionOwner?: DocumentExtractionOwner } }
|
||||
| { type: 'document.accessed'; payload: { document: { id: string; file_name: string }; userId: string; companyId: string } }
|
||||
| { type: 'document.deleted'; payload: { document: { id: string; file_name: string }; userId: string; companyId: string } }
|
||||
// Invoicing
|
||||
|
||||
@@ -5463,6 +5463,7 @@
|
||||
"ext_arcim_documents_import_action": "Also import documents from Fortnox",
|
||||
"ext_arcim_documents_not_now": "Not now",
|
||||
"ext_arcim_documents_importing": "Importing and linking the documents to the vouchers...",
|
||||
"ext_arcim_documents_import_progress": "{done} of {total} documents processed. The import continues automatically.",
|
||||
"ext_arcim_documents_reconnecting": "Waiting for Fortnox to reconnect...",
|
||||
"ext_arcim_documents_discovery_error": "The migration is complete, but we could not check the documents in Fortnox. Try again without rerunning the migration.",
|
||||
"ext_arcim_documents_import_error": "The migration is still complete, but the documents could not be imported. Try again; documents already imported will be skipped.",
|
||||
|
||||
@@ -5463,6 +5463,7 @@
|
||||
"ext_arcim_documents_import_action": "Importera även underlag från Fortnox",
|
||||
"ext_arcim_documents_not_now": "Inte nu",
|
||||
"ext_arcim_documents_importing": "Importerar och kopplar underlagen till verifikaten...",
|
||||
"ext_arcim_documents_import_progress": "{done} av {total} underlag behandlade. Importen fortsätter automatiskt.",
|
||||
"ext_arcim_documents_reconnecting": "Väntar på att Fortnox ska kopplas om...",
|
||||
"ext_arcim_documents_discovery_error": "Migreringen är klar, men vi kunde inte kontrollera underlagen i Fortnox. Försök igen utan att köra om migreringen.",
|
||||
"ext_arcim_documents_import_error": "Migreringen är fortfarande klar, men underlagen kunde inte importeras. Försök igen; redan importerade underlag hoppas över.",
|
||||
|
||||
Reference in New Issue
Block a user