fix(migration): resumable underlag import without inline extraction + same-origin MCP storage URLs (#1783)
* fix(migration): resumable underlag import without inline extraction, same-origin MCP storage URLs The Fortnox underlag import ran every file's AI extraction inline inside one request and hit the hosted 300 s function limit after ~17 of 113 files (twice on 2026-08-21); the UI showed the generic "underlagen kunde inte importeras" although the files it did reach were linked. The import now works in time-budgeted slices with a stable cursor (the UI loops until the server reports the end and shows "x av y") and opts out of extraction (extractionOwner 'none', stamped skipped:opted_out): every file is linked to its posted verifikat on arrival, so the booking is already known. MCP signed Storage URLs (upload_url, signed_url, download_url) are served through a same-origin proxy, /api/storage/[...path], because Claude Desktop's sandbox only reaches the MCP host and blocked the PUT to <project>.supabase.co. The signed token stays the only credential; the proxy forwards only signed documents-bucket paths to our own Storage host and is a no-op rewrite when NEXT_PUBLIC_APP_URL is unset. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013YoZ8iboyTj221axW6Gdtm * fix(mcp): keep the storage-proxy note out of the size-capped tool descriptions The per-tool 280-char cap and the tools/list payload ceiling both tripped on the two sentences added to gnubok_create_document_upload and gnubok_get_document_content; the why now lives in a code comment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013YoZ8iboyTj221axW6Gdtm * fix(review): id cursor, stall = error, capped upload body, encoded dot segments Review follow-ups on #1783: - the import cursor is the last handled provider attachment id, not an index, so a file Fortnox adds or removes mid-sweep shifts nothing - a partial answer whose cursor does not advance (or the round guard) is reported as ARCIM_DOCUMENT_IMPORT_STALLED instead of "complete"; the slices already landed stay reported and the retry button resumes - the storage proxy reads the PUT body as a capped stream instead of buffering an unbounded payload before measuring it - object paths are rejected when any segment decodes to "." or ".." (or holds a separator), and the URL fetch() would actually request is re-checked against the allowlist after normalisation - download_url description no longer claims a direct Storage URL Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013YoZ8iboyTj221axW6Gdtm --------- Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Jakob Wennberg
co-authored by
Claude Fable 5
Jakob Wennberg
Claude Fable 5
Jakob Wennberg
parent
0531576807
commit
524d9978f1
@@ -0,0 +1,161 @@
|
||||
import { createLogger } from '@/lib/logger'
|
||||
import { errorResponseFromCode } from '@/lib/errors/get-structured-error'
|
||||
import {
|
||||
STORAGE_PROXY_ROUTE,
|
||||
readBodyWithCap,
|
||||
resolveUpstreamStorageUrl,
|
||||
} from '@/lib/core/documents/storage-proxy'
|
||||
|
||||
/**
|
||||
* /api/storage/[...path]: same-origin proxy for signed Supabase Storage URLs.
|
||||
*
|
||||
* Why: agent sandboxes (Claude Desktop's code execution among them) only let
|
||||
* traffic out to the host the MCP server runs on. Signed Storage URLs point
|
||||
* at <project>.supabase.co, so the MCP model-free upload (PUT bytes to
|
||||
* upload_url) and signed document downloads were blocked inside such
|
||||
* sandboxes. The MCP tools now hand out URLs under this route instead (see
|
||||
* lib/core/documents/storage-proxy.ts), and this handler forwards them to
|
||||
* Storage unchanged.
|
||||
*
|
||||
* Auth: deliberately NOT withRouteContext. The caller is a sandbox with no
|
||||
* session, and the signed token in the query string is the credential:
|
||||
* Storage validates it against the exact object path on every request, the
|
||||
* same way it would on the public Storage host. The proxy only forwards
|
||||
* signed object paths on the documents bucket, only to our own Storage
|
||||
* origin, with a token present; it never mints, reads or relays any key.
|
||||
*
|
||||
* GET/HEAD /api/storage/sign/documents/<key>?token=... download
|
||||
* PUT /api/storage/upload/sign/documents/<key>?token=... upload
|
||||
*/
|
||||
|
||||
export const dynamic = 'force-dynamic'
|
||||
export const maxDuration = 60
|
||||
|
||||
const log = createLogger('api/storage-proxy')
|
||||
|
||||
/** Above every caller's own cap (MCP upload 10 MB, document max 20 MB). */
|
||||
const MAX_UPLOAD_BYTES = 50 * 1024 * 1024
|
||||
|
||||
const REQUEST_HEADERS_FORWARDED = [
|
||||
'content-type',
|
||||
'cache-control',
|
||||
'x-upsert',
|
||||
'range',
|
||||
'if-none-match',
|
||||
'if-modified-since',
|
||||
] as const
|
||||
|
||||
const RESPONSE_HEADERS_FORWARDED = [
|
||||
'content-type',
|
||||
'content-length',
|
||||
'content-disposition',
|
||||
'content-range',
|
||||
'accept-ranges',
|
||||
'cache-control',
|
||||
'etag',
|
||||
'last-modified',
|
||||
] as const
|
||||
|
||||
// The sandbox fetch is not a browser, but a browser-based MCP client would
|
||||
// need these and they cost nothing: the token still gates every request.
|
||||
const CORS_HEADERS: Record<string, string> = {
|
||||
'Access-Control-Allow-Origin': '*',
|
||||
'Access-Control-Allow-Methods': 'GET, HEAD, PUT, OPTIONS',
|
||||
'Access-Control-Allow-Headers': 'Content-Type, Cache-Control, x-upsert, Range',
|
||||
'Access-Control-Max-Age': '600',
|
||||
}
|
||||
|
||||
function withCors(response: Response): Response {
|
||||
for (const [key, value] of Object.entries(CORS_HEADERS)) {
|
||||
response.headers.set(key, value)
|
||||
}
|
||||
return response
|
||||
}
|
||||
|
||||
function objectPathOf(request: Request): string {
|
||||
const { pathname } = new URL(request.url)
|
||||
const prefix = `${STORAGE_PROXY_ROUTE}/`
|
||||
return pathname.startsWith(prefix) ? pathname.slice(prefix.length) : ''
|
||||
}
|
||||
|
||||
function rejected(reason: 'unsupported_path' | 'missing_token' | 'storage_unconfigured') {
|
||||
const code =
|
||||
reason === 'unsupported_path'
|
||||
? 'STORAGE_PROXY_UNSUPPORTED_PATH'
|
||||
: reason === 'missing_token'
|
||||
? 'STORAGE_PROXY_TOKEN_REQUIRED'
|
||||
: 'STORAGE_PROXY_UNCONFIGURED'
|
||||
return withCors(errorResponseFromCode(code, log))
|
||||
}
|
||||
|
||||
async function proxy(request: Request, method: 'GET' | 'HEAD' | 'PUT'): Promise<Response> {
|
||||
const url = new URL(request.url)
|
||||
const resolved = resolveUpstreamStorageUrl(objectPathOf(request), url.searchParams)
|
||||
if (!resolved.ok) return rejected(resolved.reason)
|
||||
|
||||
const headers = new Headers()
|
||||
for (const name of REQUEST_HEADERS_FORWARDED) {
|
||||
const value = request.headers.get(name)
|
||||
if (value) headers.set(name, value)
|
||||
}
|
||||
|
||||
let body: ArrayBuffer | undefined
|
||||
if (method === 'PUT') {
|
||||
const declared = Number(request.headers.get('content-length') ?? '0')
|
||||
if (Number.isFinite(declared) && declared > MAX_UPLOAD_BYTES) {
|
||||
return withCors(errorResponseFromCode('STORAGE_PROXY_BODY_TOO_LARGE', log))
|
||||
}
|
||||
// Stream with a cap: the declared length is advisory (absent or wrong),
|
||||
// and the token is only validated upstream, so never buffer an unbounded
|
||||
// body before measuring it.
|
||||
const read = await readBodyWithCap(request.body, MAX_UPLOAD_BYTES)
|
||||
if (read === null) {
|
||||
return withCors(errorResponseFromCode('STORAGE_PROXY_BODY_TOO_LARGE', log))
|
||||
}
|
||||
body = read
|
||||
if (!headers.has('content-type')) headers.set('content-type', 'application/octet-stream')
|
||||
}
|
||||
|
||||
let upstream: Response
|
||||
try {
|
||||
upstream = await fetch(resolved.url, {
|
||||
method,
|
||||
headers,
|
||||
...(body ? { body } : {}),
|
||||
redirect: 'manual',
|
||||
cache: 'no-store',
|
||||
})
|
||||
} catch (error) {
|
||||
log.error('storage proxy upstream request failed', error as Error, { method })
|
||||
return withCors(errorResponseFromCode('STORAGE_PROXY_UPSTREAM_UNAVAILABLE', log))
|
||||
}
|
||||
|
||||
const responseHeaders = new Headers(CORS_HEADERS)
|
||||
for (const name of RESPONSE_HEADERS_FORWARDED) {
|
||||
const value = upstream.headers.get(name)
|
||||
if (value) responseHeaders.set(name, value)
|
||||
}
|
||||
// Never let a served document be sniffed into something executable.
|
||||
responseHeaders.set('X-Content-Type-Options', 'nosniff')
|
||||
|
||||
return new Response(method === 'HEAD' ? null : upstream.body, {
|
||||
status: upstream.status,
|
||||
headers: responseHeaders,
|
||||
})
|
||||
}
|
||||
|
||||
export async function GET(request: Request) {
|
||||
return proxy(request, 'GET')
|
||||
}
|
||||
|
||||
export async function HEAD(request: Request) {
|
||||
return proxy(request, 'HEAD')
|
||||
}
|
||||
|
||||
export async function PUT(request: Request) {
|
||||
return proxy(request, 'PUT')
|
||||
}
|
||||
|
||||
export async function OPTIONS() {
|
||||
return new Response(null, { status: 204, headers: CORS_HEADERS })
|
||||
}
|
||||
@@ -0,0 +1,162 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import { GET, HEAD, OPTIONS, PUT } from '../[...path]/route'
|
||||
|
||||
const SUPABASE = 'https://pwxtzglxptnnvjrpixpg.supabase.co'
|
||||
|
||||
const fetchMock = vi.fn()
|
||||
|
||||
beforeEach(() => {
|
||||
vi.stubEnv('NEXT_PUBLIC_SUPABASE_URL', SUPABASE)
|
||||
vi.stubEnv('NEXT_PUBLIC_APP_URL', 'https://app.accounted.se')
|
||||
vi.stubGlobal('fetch', fetchMock)
|
||||
fetchMock.mockReset()
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
vi.unstubAllEnvs()
|
||||
vi.unstubAllGlobals()
|
||||
})
|
||||
|
||||
function upstreamResponse(body: string | null, init: ResponseInit = {}) {
|
||||
return new Response(body, {
|
||||
status: 200,
|
||||
headers: { 'content-type': 'application/pdf', 'content-length': String(body?.length ?? 0) },
|
||||
...init,
|
||||
})
|
||||
}
|
||||
|
||||
describe('/api/storage/[...path] same-origin Storage proxy', () => {
|
||||
it('forwards a signed upload PUT (bytes, content-type, token) to our Storage host', async () => {
|
||||
fetchMock.mockResolvedValue(new Response(JSON.stringify({ Key: 'documents/x' }), { status: 200, headers: { 'content-type': 'application/json' } }))
|
||||
const bytes = new TextEncoder().encode('%PDF-1.4 hello')
|
||||
const request = new Request(
|
||||
'https://app.accounted.se/api/storage/upload/sign/documents/co-1/user-1/pending/up-1/faktura%20maj.pdf?token=eyJ.sig',
|
||||
{ method: 'PUT', headers: { 'content-type': 'application/pdf', 'x-upsert': 'false' }, body: bytes },
|
||||
)
|
||||
|
||||
const response = await PUT(request)
|
||||
|
||||
expect(response.status).toBe(200)
|
||||
expect(response.headers.get('access-control-allow-origin')).toBe('*')
|
||||
expect(fetchMock).toHaveBeenCalledTimes(1)
|
||||
const [url, init] = fetchMock.mock.calls[0] as [string, RequestInit]
|
||||
expect(url).toBe(
|
||||
`${SUPABASE}/storage/v1/object/upload/sign/documents/co-1/user-1/pending/up-1/faktura%20maj.pdf?token=eyJ.sig`,
|
||||
)
|
||||
expect(init.method).toBe('PUT')
|
||||
const headers = init.headers as Headers
|
||||
expect(headers.get('content-type')).toBe('application/pdf')
|
||||
expect(headers.get('x-upsert')).toBe('false')
|
||||
expect(new TextDecoder().decode(init.body as ArrayBuffer)).toBe('%PDF-1.4 hello')
|
||||
})
|
||||
|
||||
it('streams a signed download GET through and keeps the document headers', async () => {
|
||||
fetchMock.mockResolvedValue(
|
||||
upstreamResponse('%PDF-1.4 bytes', {
|
||||
headers: {
|
||||
'content-type': 'application/pdf',
|
||||
'content-disposition': 'attachment; filename="kvitto.pdf"',
|
||||
'set-cookie': 'leak=1',
|
||||
},
|
||||
}),
|
||||
)
|
||||
const request = new Request(
|
||||
'https://app.accounted.se/api/storage/sign/documents/co-1/user-1/kvitto.pdf?token=eyJ.sig',
|
||||
)
|
||||
|
||||
const response = await GET(request)
|
||||
|
||||
expect(response.status).toBe(200)
|
||||
expect(await response.text()).toBe('%PDF-1.4 bytes')
|
||||
expect(response.headers.get('content-type')).toBe('application/pdf')
|
||||
expect(response.headers.get('content-disposition')).toBe('attachment; filename="kvitto.pdf"')
|
||||
expect(response.headers.get('x-content-type-options')).toBe('nosniff')
|
||||
expect(response.headers.get('set-cookie')).toBeNull()
|
||||
const [url, init] = fetchMock.mock.calls[0] as [string, RequestInit]
|
||||
expect(url).toBe(`${SUPABASE}/storage/v1/object/sign/documents/co-1/user-1/kvitto.pdf?token=eyJ.sig`)
|
||||
expect(init.method).toBe('GET')
|
||||
})
|
||||
|
||||
it('answers HEAD without a body and relays the upstream status', async () => {
|
||||
fetchMock.mockResolvedValue(new Response(null, { status: 404, headers: { 'content-type': 'application/json' } }))
|
||||
|
||||
const response = await HEAD(
|
||||
new Request('https://app.accounted.se/api/storage/sign/documents/co-1/missing.pdf?token=t', { method: 'HEAD' }),
|
||||
)
|
||||
|
||||
expect(response.status).toBe(404)
|
||||
expect(response.body).toBeNull()
|
||||
})
|
||||
|
||||
it('refuses paths outside the signed documents-bucket allowlist without touching Storage', async () => {
|
||||
const response = await GET(
|
||||
new Request('https://app.accounted.se/api/storage/public/documents/a.pdf?token=t'),
|
||||
)
|
||||
|
||||
expect(response.status).toBe(404)
|
||||
const body = (await response.json()) as { error: { code: string } }
|
||||
expect(body.error.code).toBe('STORAGE_PROXY_UNSUPPORTED_PATH')
|
||||
expect(fetchMock).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('refuses a link without its signed token', async () => {
|
||||
const response = await GET(
|
||||
new Request('https://app.accounted.se/api/storage/sign/documents/co-1/a.pdf'),
|
||||
)
|
||||
|
||||
expect(response.status).toBe(400)
|
||||
const body = (await response.json()) as { error: { code: string } }
|
||||
expect(body.error.code).toBe('STORAGE_PROXY_TOKEN_REQUIRED')
|
||||
expect(fetchMock).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('rejects an oversized upload before forwarding it', async () => {
|
||||
const request = new Request(
|
||||
'https://app.accounted.se/api/storage/upload/sign/documents/co-1/big.pdf?token=t',
|
||||
{ method: 'PUT', headers: { 'content-length': String(51 * 1024 * 1024) }, body: 'x' },
|
||||
)
|
||||
|
||||
const response = await PUT(request)
|
||||
|
||||
expect(response.status).toBe(413)
|
||||
expect(fetchMock).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('rejects an oversized upload that lies about (or omits) its content-length, without buffering it all', async () => {
|
||||
let pulled = 0
|
||||
const endless = new ReadableStream<Uint8Array>({
|
||||
pull(controller) {
|
||||
pulled++
|
||||
controller.enqueue(new Uint8Array(1024 * 1024))
|
||||
},
|
||||
})
|
||||
const request = new Request(
|
||||
'https://app.accounted.se/api/storage/upload/sign/documents/co-1/big.pdf?token=t',
|
||||
{ method: 'PUT', body: endless, duplex: 'half' } as RequestInit,
|
||||
)
|
||||
|
||||
const response = await PUT(request)
|
||||
|
||||
expect(response.status).toBe(413)
|
||||
expect(fetchMock).not.toHaveBeenCalled()
|
||||
expect(pulled).toBeLessThan(60)
|
||||
})
|
||||
|
||||
it('reports Storage being unreachable as 502 instead of crashing', async () => {
|
||||
fetchMock.mockRejectedValue(new Error('ECONNRESET'))
|
||||
|
||||
const response = await GET(
|
||||
new Request('https://app.accounted.se/api/storage/sign/documents/co-1/a.pdf?token=t'),
|
||||
)
|
||||
|
||||
expect(response.status).toBe(502)
|
||||
const body = (await response.json()) as { error: { code: string } }
|
||||
expect(body.error.code).toBe('STORAGE_PROXY_UPSTREAM_UNAVAILABLE')
|
||||
})
|
||||
|
||||
it('answers CORS preflight', async () => {
|
||||
const response = await OPTIONS()
|
||||
expect(response.status).toBe(204)
|
||||
expect(response.headers.get('access-control-allow-methods')).toContain('PUT')
|
||||
})
|
||||
})
|
||||
Reference in New Issue
Block a user