fix(documents): record archive integrity checks in their own ledger so the nightly control advances again (#2108)
The 03:00 WORM verification cron stamped last_integrity_check_at on document_attachments. enforce_period_lock_documents() fires on any UPDATE of a row whose journal entry sits in a closed or locked period, without checking whether the entry link actually changed, so a read-only integrity stamp was rejected. The queue orders last_integrity_check_at ASC NULLS FIRST, so the rejected rows re-sorted to the head every night and the batch became permanently 200/200 blocked. Both call sites discarded the update error, so nothing logged and nothing alerted. Prod state: 34 557 current-version documents, 24 083 never checked, last successful stamp 2026-08-31 03:00, nightly successes already decayed to single digits. Migration 017's enforcement triggers are legally required and never-touch, so this does not narrow the trigger. The verification outcome moves to its own document_integrity_checks table and the cron stops writing document_attachments altogether, which takes the trigger off the write path. The legacy column stays in place. Failures are now counted, logged and reported in the route's summary: the silence is why this went unnoticed for weeks. Claude-Session: https://claude.ai/code/session_016ifKg6Ec67A39oxfGPU1yc Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1427,6 +1427,7 @@ One line per decision: `[YYYY-MM-DD] <decision>: <why>`. Appended by agents and
|
||||
[2026-09-01] multi_user skeptic fixes: Stripe cancel EXPIRES the multi_user stripe grant instead of deleting it (grace anchor; other grants still deleted per freeze-and-retain); app-side state checks go RPC-first via SECURITY DEFINER company_multi_user_state (capability_grants RLS hides team rows from non-team users, byrå clients would misread as frozen); byra-kind teams get a standing team-scoped multi_user grant via backfill + teams trigger (WL-10 assumption made real; partner billing is out-of-band); PGRST202 on resolution fails OPEN (pre-migration DB has no multi_user rows: gated fallback would freeze all non-owners); /api/v1 got the same dormancy gate as MCP. RLS-level enforcement and the mid-session API fallback write-back window stay v2 follow-ups (documented, same class as pre-existing stale-preference fallback).
|
||||
[2026-09-01] Declined CodeRabbit's UpgradeNote suggestion (PR #1758 follow-up) to append the self-host connector sentence to children instead of replacing them: every caller's children is hosted subscription copy ("... kräver ett abonnemang"), so appending would show subscription wording on a self-host, the exact thing the branch exists to avoid; the "CSV/SIE import stays free" text it cited is a code comment in BankSyncNowButton, not children. Replace-on-self-host stays; a dedicated selfHosted children prop can come when a caller actually needs per-panel reassurance there.
|
||||
[2026-09-01] getConnectorConfig() rebuilds baseUrl as origin + path (userinfo/query/fragment stripped, warn-logged without the raw value): /api/connector/status echoes baseUrl to the operator and the sync/proxy URLs get paths appended, so nothing secret-shaped pasted into GNUBOK_CONNECT_URL may survive; the stripped parts were never meaningful in a base URL. The status route is also Cache-Control: no-store (key prefix + wiring layout out of shared browser caches).
|
||||
[2026-09-01] WORM integrity check moved to a new document_integrity_checks table instead of narrowing enforce_period_lock_documents(). CLAUDE.md declares migration 017's enforcement triggers legally required and never-touch, and the nightly stamp does not need to write document_attachments at all. Trades one extra table for leaving the BFL enforcement surface untouched.
|
||||
[2026-09-01] processing_history PII: the sender address and mail subject are dropped outright from RateLimitedDropped and AttachmentsTruncated rather than hashed, and the strip ships in the same commit as, and ahead of, the migration registering those event types. Registering first would start persisting PII into an append-only table the archive's erasure path excludes.
|
||||
[2026-09-01] White-label backend guard inverted from an allowlist of protected hosts to an assertion that any customer-facing production host is served by the production Supabase project. This reverses the earlier explicit-allowlist decision: that model failed open for improveone.accounted.se, which was serving a byra login page wired to the staging project with no alert, because it was never added to the list.
|
||||
[2026-09-01] Anon-callable SECURITY DEFINER writes: the guard shape `IF auth.uid() IS NOT NULL AND NOT EXISTS (membership)` is unsafe on its own. The anon JWT carries no `sub` claim, so auth.uid() is NULL for role anon too and the guard short-circuits into the trusted branch. It is defense in depth behind a REVOKE FROM PUBLIC, anon, never a substitute for one. Every new SECURITY DEFINER function ships with that REVOKE; tests/pg/definer-function-grants.pg.test.ts enforces it from a sweep rather than a hand list, because hand-listing is exactly how three guarded numbering RPCs were wrongly declared safe.
|
||||
|
||||
@@ -1,12 +1,17 @@
|
||||
/**
|
||||
* Tests for the nightly document integrity-verify cron.
|
||||
*
|
||||
* Covers the two production defects fixed in this route:
|
||||
* - the run must fit its budget (maxDuration 300 + batch default 200), and
|
||||
* Covers the production defects fixed in this route:
|
||||
* - the run must fit its budget (maxDuration 300 + batch default 200),
|
||||
* - a document whose storage object cannot be downloaded must surface as an
|
||||
* audit incident (INTEGRITY_FAILURE / DOCUMENT_OBJECT_MISSING) AND get its
|
||||
* last_integrity_check_at stamped so it stops head-blocking the
|
||||
* nulls-first queue every night.
|
||||
* audit incident (INTEGRITY_FAILURE / DOCUMENT_OBJECT_MISSING) AND get a
|
||||
* ledger row so it stops head-blocking the queue,
|
||||
* - the queue and the stamp both live in document_integrity_checks: the route
|
||||
* must never write to document_attachments, whose UPDATE trigger
|
||||
* (enforce_period_lock_documents, migration 017) rejects every document
|
||||
* linked to a closed/locked period and wedged the cron at 200/200 rejected,
|
||||
* - and a rejected ledger or audit write must be counted, logged and reported
|
||||
* rather than discarded, which is why the stall went unnoticed for weeks.
|
||||
*/
|
||||
import { describe, it, expect, vi, beforeEach } from 'vitest'
|
||||
import { createHash } from 'node:crypto'
|
||||
@@ -24,39 +29,36 @@ interface MockDoc {
|
||||
storage_path: string
|
||||
sha256_hash: string
|
||||
file_name: string
|
||||
last_checked_at: string | null
|
||||
}
|
||||
|
||||
const state = {
|
||||
documents: [] as MockDoc[],
|
||||
fetchError: null as { message: string } | null,
|
||||
downloadResults: new Map<string, { data: unknown; error: { message: string } | null }>(),
|
||||
updates: [] as Array<{ values: Record<string, unknown>; id: string }>,
|
||||
rpcCalls: [] as Array<{ fn: string; args: Record<string, unknown> }>,
|
||||
ledgerInserts: [] as Array<Record<string, unknown>>,
|
||||
ledgerInsertError: null as { message: string } | null,
|
||||
auditInserts: [] as Array<Record<string, unknown>>,
|
||||
auditInsertError: null as { message: string } | null,
|
||||
limitCalls: [] as number[],
|
||||
}
|
||||
|
||||
function makeMockClient() {
|
||||
return {
|
||||
rpc: (fn: string, args: Record<string, unknown>) => {
|
||||
state.rpcCalls.push({ fn, args })
|
||||
return Promise.resolve({
|
||||
data: state.fetchError ? null : state.documents,
|
||||
error: state.fetchError,
|
||||
})
|
||||
},
|
||||
from: (table: string) => {
|
||||
if (table === 'document_attachments') {
|
||||
if (table === 'document_integrity_checks') {
|
||||
return {
|
||||
select: () => ({
|
||||
eq: () => ({
|
||||
order: () => ({
|
||||
limit: (n: number) => {
|
||||
state.limitCalls.push(n)
|
||||
return Promise.resolve({ data: state.documents, error: state.fetchError })
|
||||
},
|
||||
}),
|
||||
}),
|
||||
}),
|
||||
update: (values: Record<string, unknown>) => ({
|
||||
eq: (_column: string, id: string) => {
|
||||
state.updates.push({ values, id })
|
||||
return Promise.resolve({ error: null })
|
||||
},
|
||||
}),
|
||||
insert: (row: Record<string, unknown>) => {
|
||||
state.ledgerInserts.push(row)
|
||||
return Promise.resolve({ error: state.ledgerInsertError })
|
||||
},
|
||||
}
|
||||
}
|
||||
if (table === 'audit_log') {
|
||||
@@ -67,6 +69,12 @@ function makeMockClient() {
|
||||
},
|
||||
}
|
||||
}
|
||||
if (table === 'document_attachments') {
|
||||
// The whole point of the fix: a write here runs through
|
||||
// enforce_period_lock_documents() and is rejected for every document
|
||||
// linked to a closed/locked period.
|
||||
throw new Error('the verify cron must never write to document_attachments')
|
||||
}
|
||||
throw new Error(`unexpected table: ${table}`)
|
||||
},
|
||||
storage: {
|
||||
@@ -101,6 +109,7 @@ function makeDoc(overrides: Partial<MockDoc> = {}): MockDoc {
|
||||
storage_path: 'user-1/company-1/inbox/file.pdf',
|
||||
sha256_hash: 'deadbeef',
|
||||
file_name: 'file.pdf',
|
||||
last_checked_at: null,
|
||||
...overrides,
|
||||
}
|
||||
}
|
||||
@@ -124,10 +133,11 @@ beforeEach(() => {
|
||||
state.documents = []
|
||||
state.fetchError = null
|
||||
state.downloadResults.clear()
|
||||
state.updates = []
|
||||
state.rpcCalls = []
|
||||
state.ledgerInserts = []
|
||||
state.ledgerInsertError = null
|
||||
state.auditInserts = []
|
||||
state.auditInsertError = null
|
||||
state.limitCalls = []
|
||||
})
|
||||
|
||||
describe('GET /api/documents/verify/cron', () => {
|
||||
@@ -139,23 +149,28 @@ describe('GET /api/documents/verify/cron', () => {
|
||||
const response = await GET(cronRequest())
|
||||
|
||||
expect(response.status).toBe(401)
|
||||
expect(state.limitCalls).toHaveLength(0)
|
||||
expect(state.rpcCalls).toHaveLength(0)
|
||||
})
|
||||
|
||||
it('declares a 300s function budget', () => {
|
||||
expect(maxDuration).toBe(300)
|
||||
})
|
||||
|
||||
it('requests a batch of 200 by default and honors the env override', async () => {
|
||||
it('draws the batch from the ledger queue, 200 by default, env override honored', async () => {
|
||||
await GET(cronRequest())
|
||||
expect(state.limitCalls).toEqual([200])
|
||||
expect(state.rpcCalls).toEqual([
|
||||
{ fn: 'next_documents_for_integrity_check', args: { p_limit: 200 } },
|
||||
])
|
||||
|
||||
process.env.DOCUMENT_VERIFY_BATCH_SIZE = '50'
|
||||
await GET(cronRequest())
|
||||
expect(state.limitCalls).toEqual([200, 50])
|
||||
expect(state.rpcCalls[1]).toEqual({
|
||||
fn: 'next_documents_for_integrity_check',
|
||||
args: { p_limit: 50 },
|
||||
})
|
||||
})
|
||||
|
||||
it('stamps last_integrity_check_at on a successful verification', async () => {
|
||||
it('appends a passed ledger row on a successful verification', async () => {
|
||||
const doc = makeDoc()
|
||||
const hash = registerObject(doc.storage_path, '%PDF-1.4 demo content')
|
||||
state.documents = [{ ...doc, sha256_hash: hash }]
|
||||
@@ -168,15 +183,24 @@ describe('GET /api/documents/verify/cron', () => {
|
||||
verified: 1,
|
||||
failures: 0,
|
||||
missingObjects: 0,
|
||||
writeFailures: 0,
|
||||
errors: 0,
|
||||
})
|
||||
expect(state.updates).toHaveLength(1)
|
||||
expect(state.updates[0].id).toBe(doc.id)
|
||||
expect(state.updates[0].values.last_integrity_check_at).toEqual(expect.any(String))
|
||||
expect(state.ledgerInserts).toHaveLength(1)
|
||||
expect(state.ledgerInserts[0]).toMatchObject({
|
||||
document_id: doc.id,
|
||||
company_id: doc.company_id,
|
||||
result: 'passed',
|
||||
expected_sha256: hash,
|
||||
computed_sha256: hash,
|
||||
storage_path: doc.storage_path,
|
||||
detail: null,
|
||||
})
|
||||
expect(state.ledgerInserts[0].checked_at).toEqual(expect.any(String))
|
||||
expect(state.auditInserts).toHaveLength(0)
|
||||
})
|
||||
|
||||
it('writes an INTEGRITY_FAILURE audit row and still stamps on hash mismatch', async () => {
|
||||
it('writes an INTEGRITY_FAILURE audit row and a hash_mismatch ledger row on mismatch', async () => {
|
||||
const doc = makeDoc({ sha256_hash: 'not-the-real-hash' })
|
||||
registerObject(doc.storage_path, 'tampered content')
|
||||
state.documents = [doc]
|
||||
@@ -186,10 +210,16 @@ describe('GET /api/documents/verify/cron', () => {
|
||||
|
||||
expect(json.failures).toBe(1)
|
||||
expect(json.missingObjects).toBe(0)
|
||||
expect(state.updates).toHaveLength(1)
|
||||
expect(json.writeFailures).toBe(0)
|
||||
expect(state.auditInserts).toHaveLength(1)
|
||||
expect(state.auditInserts[0].action).toBe('INTEGRITY_FAILURE')
|
||||
expect(String(state.auditInserts[0].description)).not.toContain('DOCUMENT_OBJECT_MISSING')
|
||||
expect(state.ledgerInserts).toHaveLength(1)
|
||||
expect(state.ledgerInserts[0]).toMatchObject({
|
||||
document_id: doc.id,
|
||||
result: 'hash_mismatch',
|
||||
expected_sha256: 'not-the-real-hash',
|
||||
})
|
||||
})
|
||||
|
||||
it('verifies via the company-scoped fallback key when a concurrent backfill re-homed the object', async () => {
|
||||
@@ -208,17 +238,18 @@ describe('GET /api/documents/verify/cron', () => {
|
||||
const json = await response.json()
|
||||
|
||||
expect(state.auditInserts).toHaveLength(0)
|
||||
expect(state.updates.map((u) => u.id)).toEqual(['doc-repointed'])
|
||||
expect(state.ledgerInserts.map((row) => row.document_id)).toEqual(['doc-repointed'])
|
||||
expect(json).toEqual({
|
||||
processed: 1,
|
||||
verified: 1,
|
||||
failures: 0,
|
||||
missingObjects: 0,
|
||||
writeFailures: 0,
|
||||
errors: 0,
|
||||
})
|
||||
})
|
||||
|
||||
it('surfaces a missing storage object as an audit incident AND stamps the check', async () => {
|
||||
it('surfaces a missing storage object as an audit incident AND an object_missing ledger row', async () => {
|
||||
const missing = makeDoc({ id: 'doc-missing', storage_path: 'user-1/company-1/gone.pdf' })
|
||||
const healthy = makeDoc({ id: 'doc-healthy', storage_path: 'user-1/company-1/ok.pdf' })
|
||||
const healthyHash = registerObject(healthy.storage_path, 'healthy content')
|
||||
@@ -235,20 +266,28 @@ describe('GET /api/documents/verify/cron', () => {
|
||||
expect(String(audit.description)).toContain('DOCUMENT_OBJECT_MISSING')
|
||||
expect(audit.new_state).toMatchObject({ reason: 'DOCUMENT_OBJECT_MISSING' })
|
||||
|
||||
// Both documents are stamped: the failing one must stop head-blocking
|
||||
// the nulls-first queue, and the healthy one was verified.
|
||||
expect(state.updates.map((u) => u.id).sort()).toEqual(['doc-healthy', 'doc-missing'])
|
||||
// Both documents get a ledger row: the failing one must stop head-blocking
|
||||
// the queue, and the healthy one was verified.
|
||||
expect(state.ledgerInserts.map((row) => row.document_id).sort()).toEqual([
|
||||
'doc-healthy',
|
||||
'doc-missing',
|
||||
])
|
||||
expect(state.ledgerInserts.find((row) => row.document_id === 'doc-missing')).toMatchObject({
|
||||
result: 'object_missing',
|
||||
computed_sha256: null,
|
||||
})
|
||||
|
||||
expect(json).toEqual({
|
||||
processed: 2,
|
||||
verified: 1,
|
||||
failures: 0,
|
||||
missingObjects: 1,
|
||||
writeFailures: 0,
|
||||
errors: 1,
|
||||
})
|
||||
})
|
||||
|
||||
it('does not stamp a missing object when the audit insert fails, so it retries next run', async () => {
|
||||
it('does not write a ledger row when the audit insert fails, so it retries next run', async () => {
|
||||
const missing = makeDoc({ id: 'doc-missing', storage_path: 'user-1/company-1/gone.pdf' })
|
||||
state.documents = [missing]
|
||||
state.auditInsertError = { message: 'insert blocked' }
|
||||
@@ -257,12 +296,49 @@ describe('GET /api/documents/verify/cron', () => {
|
||||
const json = await response.json()
|
||||
|
||||
expect(state.auditInserts).toHaveLength(1)
|
||||
expect(state.updates).toHaveLength(0)
|
||||
expect(state.ledgerInserts).toHaveLength(0)
|
||||
expect(json.missingObjects).toBe(0)
|
||||
expect(json.writeFailures).toBe(1)
|
||||
expect(json.errors).toBe(1)
|
||||
})
|
||||
|
||||
it('returns an error envelope when the document fetch fails', async () => {
|
||||
it('reports a rejected ledger write instead of swallowing it', async () => {
|
||||
const doc = makeDoc()
|
||||
const hash = registerObject(doc.storage_path, 'healthy content')
|
||||
state.documents = [{ ...doc, sha256_hash: hash }]
|
||||
state.ledgerInsertError = { message: 'permission denied for table document_integrity_checks' }
|
||||
|
||||
const response = await GET(cronRequest())
|
||||
const json = await response.json()
|
||||
|
||||
expect(state.ledgerInserts).toHaveLength(1)
|
||||
expect(json).toEqual({
|
||||
processed: 1,
|
||||
verified: 0,
|
||||
failures: 0,
|
||||
missingObjects: 0,
|
||||
writeFailures: 1,
|
||||
errors: 1,
|
||||
})
|
||||
})
|
||||
|
||||
it('counts a rejected audit write on a hash mismatch as a write failure', async () => {
|
||||
const doc = makeDoc({ sha256_hash: 'not-the-real-hash' })
|
||||
registerObject(doc.storage_path, 'tampered content')
|
||||
state.documents = [doc]
|
||||
state.auditInsertError = { message: 'insert blocked' }
|
||||
|
||||
const response = await GET(cronRequest())
|
||||
const json = await response.json()
|
||||
|
||||
expect(state.auditInserts).toHaveLength(1)
|
||||
expect(state.ledgerInserts).toHaveLength(0)
|
||||
expect(json.failures).toBe(0)
|
||||
expect(json.writeFailures).toBe(1)
|
||||
expect(json.errors).toBe(1)
|
||||
})
|
||||
|
||||
it('returns an error envelope when the queue fetch fails', async () => {
|
||||
state.fetchError = { message: 'db down' }
|
||||
|
||||
const response = await GET(cronRequest())
|
||||
|
||||
@@ -7,21 +7,71 @@ import { errorResponse, errorResponseFromCode } from '@/lib/errors/get-structure
|
||||
/**
|
||||
* GET /api/documents/verify/cron: nightly 03:00 UTC (schedule in vercel.json).
|
||||
* Spot-checks WORM archive integrity by recomputing SHA-256 for the next
|
||||
* batch of documents and writing INTEGRITY_FAILURE rows to the audit log
|
||||
* for any mismatches. Documents whose storage object cannot be downloaded
|
||||
* get an INTEGRITY_FAILURE row marked DOCUMENT_OBJECT_MISSING and are still
|
||||
* stamped as checked so they stop head-blocking the nulls-first queue.
|
||||
* batch of documents and appending the outcome to document_integrity_checks.
|
||||
* Hash mismatches and unreadable storage objects additionally get an
|
||||
* INTEGRITY_FAILURE row in the audit log, which is the durable incident
|
||||
* surface.
|
||||
*
|
||||
* Both the queue and the stamp live in document_integrity_checks and NOT in
|
||||
* document_attachments.last_integrity_check_at (now legacy): any UPDATE of a
|
||||
* document linked to an entry in a closed/locked period is rejected by
|
||||
* enforce_period_lock_documents() (migration 017, legally required and never
|
||||
* touched), which had this cron wedged at 200 of 200 rejected per night with
|
||||
* both write errors discarded. See
|
||||
* supabase/migrations/20260901130000_document_integrity_checks.sql.
|
||||
*/
|
||||
|
||||
// Vercel function budget; verification is sequential, see batch size below.
|
||||
export const maxDuration = 300
|
||||
|
||||
// Measured ~0.8s per document (download + hash + stamp), so 200 documents
|
||||
// finish in ~160s with headroom inside the 300s budget. The previous default
|
||||
// of 500 hit the platform timeout around item ~250 every night, so the tail
|
||||
// of the queue was never reached.
|
||||
// Measured ~0.8s per document (download + hash + ledger append), so 200
|
||||
// documents finish in ~160s with headroom inside the 300s budget. The previous
|
||||
// default of 500 hit the platform timeout around item ~250 every night, so the
|
||||
// tail of the queue was never reached.
|
||||
const DEFAULT_VERIFY_BATCH_SIZE = 200
|
||||
|
||||
type ServiceClient = ReturnType<typeof createServiceRoleClient>
|
||||
|
||||
type IntegrityResult = 'passed' | 'hash_mismatch' | 'object_missing'
|
||||
|
||||
/** One row of public.next_documents_for_integrity_check(). */
|
||||
interface QueuedDocument {
|
||||
id: string
|
||||
user_id: string
|
||||
company_id: string
|
||||
storage_path: string
|
||||
sha256_hash: string
|
||||
file_name: string
|
||||
last_checked_at: string | null
|
||||
}
|
||||
|
||||
/**
|
||||
* Append the outcome of one check to the integrity ledger. The write error is
|
||||
* returned, never swallowed: a lost ledger row leaves the document at the head
|
||||
* of the queue, and discarding exactly this error is why the control stayed
|
||||
* dead for weeks without anyone noticing.
|
||||
*/
|
||||
async function recordCheck(
|
||||
supabase: ServiceClient,
|
||||
doc: QueuedDocument,
|
||||
result: IntegrityResult,
|
||||
computedHash: string | null,
|
||||
detail: string | null
|
||||
): Promise<{ error: { message: string } | null }> {
|
||||
const { error } = await supabase.from('document_integrity_checks').insert({
|
||||
company_id: doc.company_id,
|
||||
document_id: doc.id,
|
||||
checked_at: new Date().toISOString(),
|
||||
expected_sha256: doc.sha256_hash,
|
||||
computed_sha256: computedHash,
|
||||
storage_path: doc.storage_path,
|
||||
result,
|
||||
detail,
|
||||
})
|
||||
|
||||
return { error }
|
||||
}
|
||||
|
||||
export const GET = withCronContext('cron.documents_verify', async (_request, ctx) => {
|
||||
const supabaseUrl = process.env.NEXT_PUBLIC_SUPABASE_URL
|
||||
const supabaseServiceKey = process.env.SUPABASE_SERVICE_ROLE_KEY
|
||||
@@ -35,25 +85,33 @@ export const GET = withCronContext('cron.documents_verify', async (_request, ctx
|
||||
|
||||
const supabase = createServiceRoleClient(supabaseUrl, supabaseServiceKey)
|
||||
|
||||
const { data: documents, error: fetchError } = await supabase
|
||||
.from('document_attachments')
|
||||
.select('id, user_id, company_id, storage_path, sha256_hash, file_name')
|
||||
.eq('is_current_version', true)
|
||||
.order('last_integrity_check_at', { ascending: true, nullsFirst: true })
|
||||
.limit(parseInt(process.env.DOCUMENT_VERIFY_BATCH_SIZE || '', 10) || DEFAULT_VERIFY_BATCH_SIZE)
|
||||
const batchSize =
|
||||
parseInt(process.env.DOCUMENT_VERIFY_BATCH_SIZE || '', 10) || DEFAULT_VERIFY_BATCH_SIZE
|
||||
|
||||
// Least-recently-checked first, never-checked ahead of those, tie-broken on
|
||||
// created_at so the drain is a deterministic FIFO rather than heap order.
|
||||
const { data, error: fetchError } = await supabase.rpc('next_documents_for_integrity_check', {
|
||||
p_limit: batchSize,
|
||||
})
|
||||
|
||||
if (fetchError) {
|
||||
ctx.log.error('failed to fetch documents for verify', fetchError)
|
||||
return errorResponse(fetchError, ctx.log, { requestId: ctx.requestId })
|
||||
}
|
||||
|
||||
if (!documents || documents.length === 0) {
|
||||
const documents = (data ?? []) as QueuedDocument[]
|
||||
|
||||
if (documents.length === 0) {
|
||||
return NextResponse.json({ message: 'No documents to verify', processed: 0 })
|
||||
}
|
||||
|
||||
let verified = 0
|
||||
let failures = 0
|
||||
let missingObjects = 0
|
||||
// Ledger and audit writes the database rejected. Counted, logged and shipped
|
||||
// in the response so a broken write path shows up on the first run instead
|
||||
// of weeks later.
|
||||
let writeFailures = 0
|
||||
|
||||
const summary = await ctx.forEach('document', documents, async (doc, itemCtx) => {
|
||||
// Dual-layout download: the batch is snapshotted up front, and a
|
||||
@@ -89,18 +147,31 @@ export const GET = withCronContext('cron.documents_verify', async (_request, ctx
|
||||
})
|
||||
|
||||
if (auditError) {
|
||||
// Leave last_integrity_check_at untouched so the document is retried
|
||||
// (and the incident write re-attempted) on the next run.
|
||||
// Append no ledger row, so the document keeps its place at the head of
|
||||
// the queue and the incident write is re-attempted on the next run.
|
||||
writeFailures++
|
||||
itemCtx.log.error('audit insert failed for missing object', new Error(auditError.message), {
|
||||
documentId: doc.id,
|
||||
})
|
||||
throw new Error(`audit insert failed for missing object: ${auditError.message}`)
|
||||
}
|
||||
|
||||
// Stamp the check so the row stops sorting to the head of the
|
||||
// nulls-first queue every night; the audit row above is the durable
|
||||
// incident surface.
|
||||
await supabase
|
||||
.from('document_attachments')
|
||||
.update({ last_integrity_check_at: new Date().toISOString() })
|
||||
.eq('id', doc.id)
|
||||
const { error: ledgerError } = await recordCheck(
|
||||
supabase,
|
||||
doc,
|
||||
'object_missing',
|
||||
null,
|
||||
`DOCUMENT_OBJECT_MISSING: ${reason}`
|
||||
)
|
||||
|
||||
if (ledgerError) {
|
||||
writeFailures++
|
||||
itemCtx.log.error('integrity ledger write failed', new Error(ledgerError.message), {
|
||||
documentId: doc.id,
|
||||
result: 'object_missing',
|
||||
})
|
||||
throw new Error(`integrity ledger write failed: ${ledgerError.message}`)
|
||||
}
|
||||
|
||||
missingObjects++
|
||||
itemCtx.log.error('document object missing', new Error(reason), {
|
||||
@@ -118,13 +189,11 @@ export const GET = withCronContext('cron.documents_verify', async (_request, ctx
|
||||
|
||||
const isValid = computedHash === doc.sha256_hash
|
||||
|
||||
await supabase
|
||||
.from('document_attachments')
|
||||
.update({ last_integrity_check_at: new Date().toISOString() })
|
||||
.eq('id', doc.id)
|
||||
|
||||
if (!isValid) {
|
||||
await supabase.from('audit_log').insert({
|
||||
// Audit row before ledger row: if the ledger row landed first and the
|
||||
// audit write then failed, the document would leave the queue with the
|
||||
// incident lost. This order re-checks it tomorrow instead.
|
||||
const { error: auditError } = await supabase.from('audit_log').insert({
|
||||
user_id: doc.user_id,
|
||||
company_id: doc.company_id,
|
||||
action: 'INTEGRITY_FAILURE',
|
||||
@@ -135,6 +204,33 @@ export const GET = withCronContext('cron.documents_verify', async (_request, ctx
|
||||
new_state: { computed_hash: computedHash },
|
||||
})
|
||||
|
||||
if (auditError) {
|
||||
writeFailures++
|
||||
itemCtx.log.error('audit insert failed for hash mismatch', new Error(auditError.message), {
|
||||
documentId: doc.id,
|
||||
})
|
||||
throw new Error(`audit insert failed for hash mismatch: ${auditError.message}`)
|
||||
}
|
||||
}
|
||||
|
||||
const { error: ledgerError } = await recordCheck(
|
||||
supabase,
|
||||
doc,
|
||||
isValid ? 'passed' : 'hash_mismatch',
|
||||
computedHash,
|
||||
isValid ? null : `stored hash ${doc.sha256_hash}, computed hash ${computedHash}`
|
||||
)
|
||||
|
||||
if (ledgerError) {
|
||||
writeFailures++
|
||||
itemCtx.log.error('integrity ledger write failed', new Error(ledgerError.message), {
|
||||
documentId: doc.id,
|
||||
result: isValid ? 'passed' : 'hash_mismatch',
|
||||
})
|
||||
throw new Error(`integrity ledger write failed: ${ledgerError.message}`)
|
||||
}
|
||||
|
||||
if (!isValid) {
|
||||
itemCtx.log.error('integrity failure', new Error('hash_mismatch'), {
|
||||
documentId: doc.id,
|
||||
fileName: doc.file_name,
|
||||
@@ -152,14 +248,25 @@ export const GET = withCronContext('cron.documents_verify', async (_request, ctx
|
||||
verified,
|
||||
failures,
|
||||
missingObjects,
|
||||
writeFailures,
|
||||
downloadErrors: summary.failed,
|
||||
})
|
||||
|
||||
if (writeFailures > 0) {
|
||||
// Loud on its own line: a rejected ledger write means the queue does not
|
||||
// advance, which is exactly the silent stall this cron just came out of.
|
||||
ctx.log.error('integrity ledger writes rejected', new Error('integrity_write_failed'), {
|
||||
processed: summary.total,
|
||||
writeFailures,
|
||||
})
|
||||
}
|
||||
|
||||
return NextResponse.json({
|
||||
processed: summary.total,
|
||||
verified,
|
||||
failures,
|
||||
missingObjects,
|
||||
writeFailures,
|
||||
errors: summary.failed,
|
||||
})
|
||||
})
|
||||
|
||||
@@ -834,6 +834,17 @@ export async function uploadDocument(
|
||||
|
||||
if (error) {
|
||||
if (reservedDocumentId && error.code === '23505') {
|
||||
// The column list mirrors the DocumentAttachment interface one for one,
|
||||
// so the row this returns is the stored row and nothing else.
|
||||
// last_integrity_check_at stays in it even though it is now legacy
|
||||
// (migration 20260901130000 moved the verification stamp to
|
||||
// document_integrity_checks, and nothing writes this column any more):
|
||||
// this branch re-reads a row a concurrent request inserted seconds ago,
|
||||
// where the column is NULL by construction, and no caller interprets the
|
||||
// value. Dropping it would leave the returned object short of a key the
|
||||
// type declares; joining the new ledger for it would fetch a check that
|
||||
// cannot exist yet. Whoever wants "when was this document last verified"
|
||||
// reads document_integrity_checks, never this field.
|
||||
const { data: concurrent, error: concurrentError } = await supabase
|
||||
.from('document_attachments')
|
||||
.select('id, user_id, company_id, storage_path, file_name, file_size_bytes, mime_type, sha256_hash, version, original_id, superseded_by_id, is_current_version, uploaded_by, upload_source, digitization_date, journal_entry_id, journal_entry_line_id, prev_version_hash, last_integrity_check_at, created_at, updated_at')
|
||||
|
||||
@@ -1179,6 +1179,22 @@ export const ARCHIVE_EXCLUDED_TABLES: Record<string, string> = {
|
||||
company_subscriptions: 'billing state',
|
||||
deadlines: 'regenerable operational calendar state',
|
||||
dimension_retag_log: 'operation log',
|
||||
// Verification metadata ABOUT räkenskapsinformation, not räkenskapsinformation
|
||||
// itself: one row per nightly SHA-256 recompute of an archived document
|
||||
// (migration 20260901130000). The documents ship under dokument/ with their
|
||||
// upload-time hash in dokument/manifest.json, so a recipient can re-verify
|
||||
// every file from the archive alone, without our check log. The checks that
|
||||
// do carry legal weight are the failures, and those are already written to
|
||||
// audit_log as INTEGRITY_FAILURE and exported in
|
||||
// revision/behandlingshistorik.json; a passing check is evidence that our
|
||||
// cron ran, which belongs to the platform and not to the company's books.
|
||||
// Erasure needs nothing either: the ledger holds no personal data of its own
|
||||
// (company_id, document_id, hashes, storage key), it inherits
|
||||
// document_attachments' posture on the user id embedded in a legacy storage
|
||||
// key, and its rows go with the ON DELETE CASCADE from companies and
|
||||
// document_attachments when the underlying data is legally removed.
|
||||
document_integrity_checks:
|
||||
'WORM verification log (SHA-256 recompute outcomes); failures reach the archive via audit_log in revision/behandlingshistorik.json',
|
||||
event_log: '30-day TTL event bus log',
|
||||
extension_data: 'extension runtime state (includes this backup\'s own state)',
|
||||
graph_counterparties: 'derived AI context graph, regenerable',
|
||||
|
||||
@@ -0,0 +1,236 @@
|
||||
-- Document integrity ledger: move the WORM verification stamp OFF
|
||||
-- document_attachments.
|
||||
--
|
||||
-- The nightly cron (app/api/documents/verify/cron/route.ts, "0 3 * * *" in
|
||||
-- vercel.json) recorded every check by UPDATEing
|
||||
-- document_attachments.last_integrity_check_at. That UPDATE runs through
|
||||
-- enforce_period_lock_documents() (migration 20240101000017), which is
|
||||
-- BEFORE INSERT OR UPDATE FOR EACH ROW with no WHEN clause and no OLD/NEW
|
||||
-- comparison: it raises for ANY update of a document whose journal entry sits
|
||||
-- in a closed or locked fiscal period, even when the entry link is untouched.
|
||||
-- Because the queue ordered by last_integrity_check_at ASC NULLS FIRST, every
|
||||
-- rejected row sorted straight back to the head of the queue the next night.
|
||||
-- Measured on prod 2026-09-01: the whole batch was 200 of 200 rejected, 24 095
|
||||
-- of 34 569 current-version documents had never been verified, and the newest
|
||||
-- successful stamp was 2026-08-31 03:00. Both call sites discarded the update
|
||||
-- error, so nothing logged and the control had been dead for weeks.
|
||||
--
|
||||
-- Migration 017's enforcement triggers are legally required and are never
|
||||
-- touched (CLAUDE.md), so the fix is to stop writing to document_attachments
|
||||
-- at all: the outcome of each verification becomes a row in its own
|
||||
-- append-only ledger and the trigger leaves the write path entirely.
|
||||
--
|
||||
-- document_attachments.last_integrity_check_at is deliberately LEFT IN PLACE
|
||||
-- and is now legacy: lib/core/documents/document-service.ts still selects it
|
||||
-- and historical rows carry real values. Nothing writes it any more. Its index
|
||||
-- does go (section 6): the column keeps its history, the index served only the
|
||||
-- query this migration retires.
|
||||
--
|
||||
-- The new table is classified in lib/reports/full-archive-export.ts as
|
||||
-- deliberately outside the säkerhetsbackup. It is verification metadata ABOUT
|
||||
-- räkenskapsinformation, not räkenskapsinformation itself, and the checks that
|
||||
-- carry legal weight (the failures) already reach the archive through
|
||||
-- audit_log as INTEGRITY_FAILURE.
|
||||
|
||||
-- =============================================================
|
||||
-- 1. The ledger
|
||||
-- =============================================================
|
||||
|
||||
CREATE TABLE public.document_integrity_checks (
|
||||
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
|
||||
company_id UUID NOT NULL REFERENCES public.companies(id) ON DELETE CASCADE,
|
||||
document_id UUID NOT NULL REFERENCES public.document_attachments(id) ON DELETE CASCADE,
|
||||
checked_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
-- What the check compared. expected_sha256 is the hash recorded on the
|
||||
-- document at upload; computed_sha256 is what re-hashing the stored object
|
||||
-- produced, and is NULL when the object could not be downloaded at all.
|
||||
expected_sha256 TEXT NOT NULL,
|
||||
computed_sha256 TEXT,
|
||||
-- The storage key that was actually read. Kept on the row because the
|
||||
-- cron falls back to the company-scoped layout when the stored pointer is
|
||||
-- stale, so "which object did we hash" is not derivable afterwards.
|
||||
storage_path TEXT NOT NULL,
|
||||
result TEXT NOT NULL CHECK (result IN ('passed', 'hash_mismatch', 'object_missing')),
|
||||
-- Human-readable detail for the failing results (download error, the two
|
||||
-- hashes). NULL on a pass.
|
||||
detail TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
-- No updated_at: append-only table.
|
||||
);
|
||||
|
||||
-- =============================================================
|
||||
-- 2. Row-level security
|
||||
-- =============================================================
|
||||
--
|
||||
-- Same shape as processing_history (20260418130000) and audit_log: a
|
||||
-- company-scoped SELECT policy through user_company_ids(), and no
|
||||
-- INSERT/UPDATE/DELETE policies at all, so only the service-role cron can
|
||||
-- append. Service-role-only-with-no-policies (the connector_* ledgers) was the
|
||||
-- other candidate and was rejected: this ledger is the evidence a company
|
||||
-- shows that its verifikat archive is actually being verified, so the tenant
|
||||
-- must be able to read its own rows. Nobody, tenant included, may write them.
|
||||
|
||||
ALTER TABLE public.document_integrity_checks ENABLE ROW LEVEL SECURITY;
|
||||
|
||||
CREATE POLICY "document_integrity_checks_select" ON public.document_integrity_checks
|
||||
FOR SELECT USING (company_id IN (SELECT public.user_company_ids()));
|
||||
|
||||
-- Defense in depth on top of RLS, the same lockdown exchange_rates got in
|
||||
-- 20260710100000, but stated explicitly instead of leaning on Supabase's
|
||||
-- default privileges (which hand anon and authenticated full DML on every new
|
||||
-- public table): a company member reads, the cron appends, nobody else has the
|
||||
-- privilege at all. service_role gets no UPDATE or DELETE either, since the
|
||||
-- ledger is append-only; the ON DELETE CASCADE above still fires, because a
|
||||
-- referential action runs with the constraint owner's rights, not the
|
||||
-- caller's.
|
||||
REVOKE ALL ON public.document_integrity_checks FROM anon, authenticated, service_role;
|
||||
GRANT SELECT ON public.document_integrity_checks TO authenticated;
|
||||
GRANT SELECT, INSERT ON public.document_integrity_checks TO service_role;
|
||||
|
||||
-- =============================================================
|
||||
-- 3. Indexes
|
||||
-- =============================================================
|
||||
|
||||
-- The cron's queue. next_documents_for_integrity_check() orders by the most
|
||||
-- recent check per document, and that timestamp lives in THIS table, so no
|
||||
-- index on document_attachments can drive the ordering. The plan is instead a
|
||||
-- scan of the current-version documents, one index probe per document into the
|
||||
-- index below, and a top-N heapsort of p_limit rows: measured against prod on
|
||||
-- 2026-09-01, ~34.6k rows scanned in 42 ms end to end. The
|
||||
-- (document_id, checked_at DESC) column order is exactly the lateral's
|
||||
-- "WHERE document_id = ? ORDER BY checked_at DESC LIMIT 1", so each probe is
|
||||
-- an index-only scan of one entry.
|
||||
CREATE INDEX idx_document_integrity_checks_document
|
||||
ON public.document_integrity_checks (document_id, checked_at DESC);
|
||||
|
||||
-- Tenant-facing read: "show me this company's integrity checks, newest first".
|
||||
-- Matches the SELECT policy's company_id filter.
|
||||
CREATE INDEX idx_document_integrity_checks_company
|
||||
ON public.document_integrity_checks (company_id, checked_at DESC);
|
||||
|
||||
-- =============================================================
|
||||
-- 4. Immutability
|
||||
-- =============================================================
|
||||
-- Own one-line function rather than the shared audit_log_immutable(): reusing
|
||||
-- it works, but it raises "Audit log entries cannot be modified or deleted"
|
||||
-- from a table that is not the audit log, and that message is already
|
||||
-- pattern-matched as an audit-log signal elsewhere in the app
|
||||
-- (app/api/transactions/[id]/route.ts). A per-ledger function is the house
|
||||
-- pattern for exactly this reason: skatteverket_api_audit_log (20260517135000)
|
||||
-- and company_migration_resets (20260818084050) each carry their own. It also
|
||||
-- decouples this table from a shared function that keeps being amended for
|
||||
-- sandbox teardown.
|
||||
--
|
||||
-- UPDATE only, not DELETE: the FKs above cascade when a company or a document
|
||||
-- is legally deleted after its retention window, and a BEFORE DELETE trigger
|
||||
-- would block that cascade.
|
||||
|
||||
-- SECURITY INVOKER (the default), like audit_log_immutable(): the body only
|
||||
-- raises, so it reads and writes nothing that definer rights could reach.
|
||||
CREATE OR REPLACE FUNCTION public.document_integrity_check_immutable()
|
||||
RETURNS trigger
|
||||
LANGUAGE plpgsql
|
||||
SET search_path = public
|
||||
AS $$
|
||||
BEGIN
|
||||
RAISE EXCEPTION 'Document integrity check entries cannot be modified or deleted';
|
||||
END;
|
||||
$$;
|
||||
|
||||
CREATE TRIGGER document_integrity_checks_no_update
|
||||
BEFORE UPDATE ON public.document_integrity_checks
|
||||
FOR EACH ROW EXECUTE FUNCTION public.document_integrity_check_immutable();
|
||||
|
||||
-- =============================================================
|
||||
-- 5. Queue for the nightly cron
|
||||
-- =============================================================
|
||||
--
|
||||
-- Least-recently-verified first, never-verified before that. The tie-break on
|
||||
-- created_at is free: the ordering key is a column of the joined table, so the
|
||||
-- sort cannot be index-driven either way, and adding the second key costs
|
||||
-- nothing while turning an arbitrary heap-order queue into a deterministic
|
||||
-- FIFO drain. Heap order is precisely what let the same 200 rows occupy the
|
||||
-- head of the old queue every single night.
|
||||
|
||||
CREATE OR REPLACE FUNCTION public.next_documents_for_integrity_check(
|
||||
p_limit integer DEFAULT 200
|
||||
)
|
||||
RETURNS TABLE (
|
||||
id uuid,
|
||||
user_id uuid,
|
||||
company_id uuid,
|
||||
storage_path text,
|
||||
sha256_hash text,
|
||||
file_name text,
|
||||
last_checked_at timestamptz
|
||||
)
|
||||
LANGUAGE sql
|
||||
STABLE
|
||||
SECURITY INVOKER
|
||||
SET search_path = public, pg_temp
|
||||
AS $$
|
||||
SELECT
|
||||
d.id,
|
||||
d.user_id,
|
||||
d.company_id,
|
||||
d.storage_path,
|
||||
d.sha256_hash,
|
||||
d.file_name,
|
||||
last_check.checked_at
|
||||
FROM public.document_attachments d
|
||||
LEFT JOIN LATERAL (
|
||||
SELECT c.checked_at
|
||||
FROM public.document_integrity_checks c
|
||||
WHERE c.document_id = d.id
|
||||
ORDER BY c.checked_at DESC
|
||||
LIMIT 1
|
||||
) last_check ON true
|
||||
WHERE d.is_current_version = true
|
||||
ORDER BY last_check.checked_at ASC NULLS FIRST, d.created_at ASC
|
||||
LIMIT LEAST(GREATEST(COALESCE(p_limit, 200), 1), 1000);
|
||||
$$;
|
||||
|
||||
-- The cron is the only caller and runs as the service role, which bypasses
|
||||
-- RLS; nothing else has any business enumerating every tenant's documents.
|
||||
REVOKE ALL ON FUNCTION public.next_documents_for_integrity_check(integer) FROM PUBLIC, anon, authenticated;
|
||||
GRANT EXECUTE ON FUNCTION public.next_documents_for_integrity_check(integer) TO service_role;
|
||||
|
||||
COMMENT ON TABLE public.document_integrity_checks IS
|
||||
'Append-only outcome of each WORM archive integrity check (SHA-256 recompute) on document_attachments. Written only by the nightly cron under the service role; readable by the owning company. Replaces document_attachments.last_integrity_check_at, whose UPDATE was rejected by enforce_period_lock_documents() for every document linked to a closed/locked period.';
|
||||
|
||||
COMMENT ON COLUMN public.document_attachments.last_integrity_check_at IS
|
||||
'LEGACY. Superseded by public.document_integrity_checks (20260901130000). Historical values only: nothing writes this column any more, because any UPDATE of a document linked to an entry in a closed/locked period is rejected by enforce_period_lock_documents().';
|
||||
|
||||
-- =============================================================
|
||||
-- 6. Retire the index that served the old queue
|
||||
-- =============================================================
|
||||
--
|
||||
-- idx_document_attachments_integrity_check (20260330120000) is
|
||||
-- (last_integrity_check_at ASC NULLS FIRST) WHERE is_current_version = true.
|
||||
-- It existed for exactly one query, the old cron's ORDER BY, and that query no
|
||||
-- longer exists: the new queue orders by a column in ANOTHER table, so no index
|
||||
-- on document_attachments can drive it. Prod on 2026-09-01 shows the shape
|
||||
-- precisely: 152 scans since the index was created in March (one per nightly
|
||||
-- run) against 3.8 M on idx_document_attachments_journal_entry_id, and 616 kB
|
||||
-- kept up to date on every insert and update of the busiest document table for
|
||||
-- reads that are now zero. It does not survive as a filter for the new queue
|
||||
-- either: EXPLAIN on prod for the new outer scan (is_current_version = true
|
||||
-- over 34.6k rows) picks a Seq Scan, not this partial index.
|
||||
--
|
||||
-- Dropped rather than kept "just in case": the column it indexes is frozen, so
|
||||
-- the index can never become useful again without a new migration that also
|
||||
-- resurrects the writer. Nothing else in the repo names it (checked), so the
|
||||
-- only cost is a moment's ACCESS EXCLUSIVE lock on a 34.6k-row table. Plain
|
||||
-- DROP INDEX, not CONCURRENTLY: migrations run inside a transaction, which
|
||||
-- CONCURRENTLY forbids, and dropping needs the lock rather than a rebuild.
|
||||
-- Between this migration and the deploy that follows it, the old cron code
|
||||
-- degrades to a sequential scan of 34.6k rows once a night, which is
|
||||
-- irrelevant, and it was rejecting 200 of 200 writes anyway.
|
||||
|
||||
DROP INDEX IF EXISTS public.idx_document_attachments_integrity_check;
|
||||
|
||||
-- =============================================================
|
||||
-- 7. Schema reload
|
||||
-- =============================================================
|
||||
|
||||
NOTIFY pgrst, 'reload schema';
|
||||
@@ -0,0 +1,413 @@
|
||||
import { randomUUID } from 'node:crypto'
|
||||
import { beforeAll, describe, expect, it } from 'vitest'
|
||||
import { getPool, withUserContext } from './setup'
|
||||
import { insertAuthUser, insertPostedJournalEntry, seedCompany } from './fixtures'
|
||||
|
||||
/**
|
||||
* pg-real for migration 20260901130000: the WORM integrity ledger that
|
||||
* replaces document_attachments.last_integrity_check_at.
|
||||
*
|
||||
* The bug being closed: enforce_period_lock_documents() (migration 017) is
|
||||
* BEFORE INSERT OR UPDATE FOR EACH ROW with no OLD/NEW comparison, so the
|
||||
* nightly cron's stamp UPDATE was rejected for every document linked to an
|
||||
* entry in a closed or locked period. Because the queue sorted on that same
|
||||
* column NULLS FIRST, the rejected rows returned to the head of the queue
|
||||
* every night and the batch sat at 200 of 200 rejected. Migration 017 is
|
||||
* legally required and stays untouched, so the check moved to its own table:
|
||||
* the assertions below pin BOTH halves, that the old UPDATE is still blocked
|
||||
* and that the same document can now be integrity-checked.
|
||||
*/
|
||||
|
||||
function makeHash(): string {
|
||||
return randomUUID().replace(/-/g, '').padEnd(64, '0')
|
||||
}
|
||||
|
||||
async function attachDocument(params: {
|
||||
userId: string
|
||||
companyId: string
|
||||
journalEntryId: string | null
|
||||
fileName?: string
|
||||
}): Promise<string> {
|
||||
const id = randomUUID()
|
||||
await getPool().query(
|
||||
`INSERT INTO public.document_attachments
|
||||
(id, user_id, company_id, journal_entry_id, file_name, mime_type,
|
||||
file_size_bytes, storage_path, sha256_hash, upload_source)
|
||||
VALUES ($1, $2, $3, $4, $5, 'application/pdf', 1024, $6, $7, 'file_upload')`,
|
||||
[
|
||||
id,
|
||||
params.userId,
|
||||
params.companyId,
|
||||
params.journalEntryId,
|
||||
params.fileName ?? 'underlag.pdf',
|
||||
`documents/${params.companyId}/${id}.pdf`,
|
||||
makeHash(),
|
||||
],
|
||||
)
|
||||
return id
|
||||
}
|
||||
|
||||
async function recordCheck(params: {
|
||||
companyId: string
|
||||
documentId: string
|
||||
checkedAt?: string
|
||||
result?: 'passed' | 'hash_mismatch' | 'object_missing'
|
||||
}): Promise<string> {
|
||||
const { rows } = await getPool().query<{ id: string }>(
|
||||
`INSERT INTO public.document_integrity_checks
|
||||
(company_id, document_id, checked_at, expected_sha256, computed_sha256,
|
||||
storage_path, result)
|
||||
VALUES ($1, $2, COALESCE($3::timestamptz, now()), $4, $4, 'documents/x.pdf', $5)
|
||||
RETURNING id`,
|
||||
[
|
||||
params.companyId,
|
||||
params.documentId,
|
||||
params.checkedAt ?? null,
|
||||
makeHash(),
|
||||
params.result ?? 'passed',
|
||||
],
|
||||
)
|
||||
return rows[0].id
|
||||
}
|
||||
|
||||
/** Company + posted entry + anchored document, with the period then sealed. */
|
||||
async function seedSealedDocument(seal: 'closed' | 'locked'): Promise<{
|
||||
userId: string
|
||||
companyId: string
|
||||
documentId: string
|
||||
}> {
|
||||
// Seed open: the period-lock triggers block inserting a posted entry and an
|
||||
// anchored document into an already-sealed period.
|
||||
const s = await seedCompany()
|
||||
const entryId = await insertPostedJournalEntry({
|
||||
userId: s.userId,
|
||||
companyId: s.companyId,
|
||||
fiscalPeriodId: s.fiscalPeriodId,
|
||||
voucherNumber: 1,
|
||||
})
|
||||
const documentId = await attachDocument({
|
||||
userId: s.userId,
|
||||
companyId: s.companyId,
|
||||
journalEntryId: entryId,
|
||||
})
|
||||
|
||||
await getPool().query(
|
||||
seal === 'closed'
|
||||
? `UPDATE public.fiscal_periods SET is_closed = true, closed_at = now() WHERE id = $1`
|
||||
: `UPDATE public.fiscal_periods SET locked_at = now() WHERE id = $1`,
|
||||
[s.fiscalPeriodId],
|
||||
)
|
||||
|
||||
return { userId: s.userId, companyId: s.companyId, documentId }
|
||||
}
|
||||
|
||||
describe('document_integrity_checks: the locked-period write path', () => {
|
||||
it('still refuses the legacy stamp on a document in a CLOSED period', async () => {
|
||||
const { documentId } = await seedSealedDocument('closed')
|
||||
|
||||
await expect(
|
||||
getPool().query(
|
||||
`UPDATE public.document_attachments SET last_integrity_check_at = now() WHERE id = $1`,
|
||||
[documentId],
|
||||
),
|
||||
).rejects.toThrow(/locked\/closed fiscal period/)
|
||||
})
|
||||
|
||||
it('still refuses the legacy stamp on a document in a LOCKED period', async () => {
|
||||
const { documentId } = await seedSealedDocument('locked')
|
||||
|
||||
await expect(
|
||||
getPool().query(
|
||||
`UPDATE public.document_attachments SET last_integrity_check_at = now() WHERE id = $1`,
|
||||
[documentId],
|
||||
),
|
||||
).rejects.toThrow(/locked\/closed fiscal period/)
|
||||
})
|
||||
|
||||
it('lets the same closed-period document be integrity-checked', async () => {
|
||||
const { companyId, documentId } = await seedSealedDocument('closed')
|
||||
|
||||
await recordCheck({ companyId, documentId })
|
||||
|
||||
const { rows } = await getPool().query<{ result: string; document_id: string }>(
|
||||
`SELECT result, document_id FROM public.document_integrity_checks WHERE document_id = $1`,
|
||||
[documentId],
|
||||
)
|
||||
expect(rows).toEqual([{ result: 'passed', document_id: documentId }])
|
||||
})
|
||||
|
||||
it('lets the same locked-period document be integrity-checked', async () => {
|
||||
const { companyId, documentId } = await seedSealedDocument('locked')
|
||||
|
||||
await recordCheck({ companyId, documentId, result: 'hash_mismatch' })
|
||||
|
||||
const { rows } = await getPool().query<{ result: string }>(
|
||||
`SELECT result FROM public.document_integrity_checks WHERE document_id = $1`,
|
||||
[documentId],
|
||||
)
|
||||
expect(rows.map((r) => r.result)).toEqual(['hash_mismatch'])
|
||||
})
|
||||
|
||||
it('keeps the sealed document in the queue until a check row exists', async () => {
|
||||
const { documentId } = await seedSealedDocument('closed')
|
||||
|
||||
const queued = async () => {
|
||||
const { rows } = await getPool().query<{ id: string }>(
|
||||
`SELECT id FROM public.next_documents_for_integrity_check(1000) WHERE id = $1`,
|
||||
[documentId],
|
||||
)
|
||||
return rows.length
|
||||
}
|
||||
|
||||
expect(await queued()).toBe(1)
|
||||
|
||||
const { rows: doc } = await getPool().query<{ company_id: string }>(
|
||||
`SELECT company_id FROM public.document_attachments WHERE id = $1`,
|
||||
[documentId],
|
||||
)
|
||||
await recordCheck({ companyId: doc[0].company_id, documentId })
|
||||
|
||||
// Still in the table (the queue is least-recently-checked, not
|
||||
// check-once), but it now carries a checked_at instead of NULL.
|
||||
const { rows } = await getPool().query<{ last_checked_at: string | null }>(
|
||||
`SELECT last_checked_at FROM public.next_documents_for_integrity_check(1000) WHERE id = $1`,
|
||||
[documentId],
|
||||
)
|
||||
expect(rows[0]?.last_checked_at ?? null).not.toBeNull()
|
||||
})
|
||||
})
|
||||
|
||||
describe('next_documents_for_integrity_check ordering', () => {
|
||||
it('returns never-checked documents before the least recently checked', async () => {
|
||||
const s = await seedCompany()
|
||||
const never = await attachDocument({
|
||||
userId: s.userId,
|
||||
companyId: s.companyId,
|
||||
journalEntryId: null,
|
||||
fileName: 'never.pdf',
|
||||
})
|
||||
const stale = await attachDocument({
|
||||
userId: s.userId,
|
||||
companyId: s.companyId,
|
||||
journalEntryId: null,
|
||||
fileName: 'stale.pdf',
|
||||
})
|
||||
const fresh = await attachDocument({
|
||||
userId: s.userId,
|
||||
companyId: s.companyId,
|
||||
journalEntryId: null,
|
||||
fileName: 'fresh.pdf',
|
||||
})
|
||||
await recordCheck({ companyId: s.companyId, documentId: stale, checkedAt: '2026-01-01' })
|
||||
await recordCheck({ companyId: s.companyId, documentId: fresh, checkedAt: '2026-08-31' })
|
||||
|
||||
const { rows } = await getPool().query<{ id: string }>(
|
||||
`SELECT id FROM public.next_documents_for_integrity_check(1000)`,
|
||||
)
|
||||
const mine = rows.map((r) => r.id).filter((id) => [never, stale, fresh].includes(id))
|
||||
expect(mine).toEqual([never, stale, fresh])
|
||||
})
|
||||
|
||||
it('reads only the newest check per document', async () => {
|
||||
const s = await seedCompany()
|
||||
const documentId = await attachDocument({
|
||||
userId: s.userId,
|
||||
companyId: s.companyId,
|
||||
journalEntryId: null,
|
||||
})
|
||||
await recordCheck({ companyId: s.companyId, documentId, checkedAt: '2026-01-01' })
|
||||
await recordCheck({ companyId: s.companyId, documentId, checkedAt: '2026-08-31' })
|
||||
|
||||
const { rows } = await getPool().query<{ last_checked_at: Date }>(
|
||||
`SELECT last_checked_at FROM public.next_documents_for_integrity_check(1000) WHERE id = $1`,
|
||||
[documentId],
|
||||
)
|
||||
expect(new Date(rows[0].last_checked_at).toISOString().slice(0, 10)).toBe('2026-08-31')
|
||||
})
|
||||
|
||||
it('honours p_limit and clamps it to a sane range', async () => {
|
||||
const one = await getPool().query(`SELECT * FROM public.next_documents_for_integrity_check(1)`)
|
||||
expect(one.rows.length).toBeLessThanOrEqual(1)
|
||||
|
||||
// 0 and negatives clamp up to 1 rather than returning nothing, and the
|
||||
// upper clamp keeps a runaway env override from scanning the world.
|
||||
const zero = await getPool().query(`SELECT * FROM public.next_documents_for_integrity_check(0)`)
|
||||
expect(zero.rows.length).toBeLessThanOrEqual(1)
|
||||
const huge = await getPool().query(
|
||||
`SELECT * FROM public.next_documents_for_integrity_check(999999)`,
|
||||
)
|
||||
expect(huge.rows.length).toBeLessThanOrEqual(1000)
|
||||
})
|
||||
|
||||
it('is executable by service_role only', async () => {
|
||||
const { rows } = await getPool().query<{ role: string; ok: boolean }>(
|
||||
`SELECT r.role,
|
||||
has_function_privilege(r.role, 'public.next_documents_for_integrity_check(integer)', 'execute') AS ok
|
||||
FROM (VALUES ('anon'), ('authenticated'), ('service_role')) AS r(role)`,
|
||||
)
|
||||
expect(Object.fromEntries(rows.map((r) => [r.role, r.ok]))).toEqual({
|
||||
anon: false,
|
||||
authenticated: false,
|
||||
service_role: true,
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
describe('document_integrity_checks: RLS and grants', () => {
|
||||
let userId: string
|
||||
let companyId: string
|
||||
let documentId: string
|
||||
|
||||
beforeAll(async () => {
|
||||
const s = await seedCompany()
|
||||
userId = s.userId
|
||||
companyId = s.companyId
|
||||
documentId = await attachDocument({
|
||||
userId,
|
||||
companyId,
|
||||
journalEntryId: null,
|
||||
fileName: 'rls.pdf',
|
||||
})
|
||||
await recordCheck({ companyId, documentId })
|
||||
})
|
||||
|
||||
it('lets a company member read its own ledger rows', async () => {
|
||||
const rows = await withUserContext(userId, async (client) => {
|
||||
const res = await client.query<{ document_id: string }>(
|
||||
`SELECT document_id FROM public.document_integrity_checks WHERE company_id = $1`,
|
||||
[companyId],
|
||||
)
|
||||
return res.rows
|
||||
})
|
||||
expect(rows).toEqual([{ document_id: documentId }])
|
||||
})
|
||||
|
||||
it('hides them from a user outside the company', async () => {
|
||||
const outsiderId = await insertAuthUser()
|
||||
const rows = await withUserContext(outsiderId, async (client) => {
|
||||
const res = await client.query(
|
||||
`SELECT id FROM public.document_integrity_checks WHERE company_id = $1`,
|
||||
[companyId],
|
||||
)
|
||||
return res.rows
|
||||
})
|
||||
expect(rows).toHaveLength(0)
|
||||
})
|
||||
|
||||
it('has no INSERT/UPDATE/DELETE policy: only the service role appends', async () => {
|
||||
const { rows } = await getPool().query<{ polcmd: string }>(
|
||||
`SELECT polcmd FROM pg_policy
|
||||
WHERE polrelid = 'public.document_integrity_checks'::regclass`,
|
||||
)
|
||||
// 'r' = SELECT. Anything else would be a write policy.
|
||||
expect(rows.map((r) => r.polcmd)).toEqual(['r'])
|
||||
|
||||
const rls = await getPool().query<{ relrowsecurity: boolean }>(
|
||||
`SELECT relrowsecurity FROM pg_class WHERE oid = 'public.document_integrity_checks'::regclass`,
|
||||
)
|
||||
expect(rls.rows[0].relrowsecurity).toBe(true)
|
||||
})
|
||||
|
||||
it('grants only what each role needs: member reads, cron appends, nobody updates', async () => {
|
||||
const { rows } = await getPool().query<{
|
||||
role: string
|
||||
can_select: boolean
|
||||
can_insert: boolean
|
||||
can_update: boolean
|
||||
can_delete: boolean
|
||||
}>(
|
||||
`SELECT r.role,
|
||||
has_table_privilege(r.role, 'public.document_integrity_checks', 'SELECT') AS can_select,
|
||||
has_table_privilege(r.role, 'public.document_integrity_checks', 'INSERT') AS can_insert,
|
||||
has_table_privilege(r.role, 'public.document_integrity_checks', 'UPDATE') AS can_update,
|
||||
has_table_privilege(r.role, 'public.document_integrity_checks', 'DELETE') AS can_delete
|
||||
FROM (VALUES ('anon'), ('authenticated'), ('service_role')) AS r(role)`,
|
||||
)
|
||||
const byRole = Object.fromEntries(
|
||||
rows.map((r) => [
|
||||
r.role,
|
||||
{
|
||||
select: r.can_select,
|
||||
insert: r.can_insert,
|
||||
update: r.can_update,
|
||||
delete: r.can_delete,
|
||||
},
|
||||
]),
|
||||
)
|
||||
expect(byRole).toEqual({
|
||||
anon: { select: false, insert: false, update: false, delete: false },
|
||||
authenticated: { select: true, insert: false, update: false, delete: false },
|
||||
// Append-only: even the cron's role cannot rewrite or erase a check.
|
||||
service_role: { select: true, insert: true, update: false, delete: false },
|
||||
})
|
||||
})
|
||||
|
||||
it('rejects an authenticated INSERT even inside the user own company', async () => {
|
||||
await withUserContext(userId, async (client) => {
|
||||
await expect(
|
||||
client.query(
|
||||
`INSERT INTO public.document_integrity_checks
|
||||
(company_id, document_id, expected_sha256, storage_path, result)
|
||||
VALUES ($1, $2, $3, 'documents/x.pdf', 'passed')`,
|
||||
[companyId, documentId, makeHash()],
|
||||
),
|
||||
).rejects.toThrow(/permission denied|row-level security/i)
|
||||
})
|
||||
})
|
||||
|
||||
it('is append-only: an UPDATE raises in this table own voice', async () => {
|
||||
// Not the shared audit_log_immutable() message. "Audit log entries cannot
|
||||
// be modified" from a table that is not the audit log misleads whoever
|
||||
// reads the log, and app/api/transactions/[id]/route.ts pattern-matches
|
||||
// that exact string as an audit-log signal.
|
||||
await expect(
|
||||
getPool().query(
|
||||
`UPDATE public.document_integrity_checks SET result = 'passed' WHERE document_id = $1`,
|
||||
[documentId],
|
||||
),
|
||||
).rejects.toThrow(/Document integrity check entries cannot be modified or deleted/i)
|
||||
})
|
||||
|
||||
it('rejects an unknown result value', async () => {
|
||||
await expect(
|
||||
getPool().query(
|
||||
`INSERT INTO public.document_integrity_checks
|
||||
(company_id, document_id, expected_sha256, storage_path, result)
|
||||
VALUES ($1, $2, $3, 'documents/x.pdf', 'probably_fine')`,
|
||||
[companyId, documentId, makeHash()],
|
||||
),
|
||||
).rejects.toThrow(/document_integrity_checks_result_check|violates check constraint/)
|
||||
})
|
||||
|
||||
it('carries the indexes the queue and the tenant read need', async () => {
|
||||
const { rows } = await getPool().query<{ indexname: string }>(
|
||||
`SELECT indexname FROM pg_indexes
|
||||
WHERE schemaname = 'public' AND tablename = 'document_integrity_checks'
|
||||
ORDER BY indexname`,
|
||||
)
|
||||
expect(rows.map((r) => r.indexname)).toContain('idx_document_integrity_checks_document')
|
||||
expect(rows.map((r) => r.indexname)).toContain('idx_document_integrity_checks_company')
|
||||
})
|
||||
|
||||
it('retires the index that served the old queue', async () => {
|
||||
// idx_document_attachments_integrity_check (20260330120000) indexed
|
||||
// last_integrity_check_at for the cron's old ORDER BY. That query is gone
|
||||
// and the column is frozen, so the index was pure write cost on the
|
||||
// busiest document table. If it comes back, so has a writer to the legacy
|
||||
// column, which is the thing migration 017 rejects.
|
||||
const { rows } = await getPool().query<{ indexname: string }>(
|
||||
`SELECT indexname FROM pg_indexes
|
||||
WHERE schemaname = 'public' AND tablename = 'document_attachments'`,
|
||||
)
|
||||
expect(rows.map((r) => r.indexname)).not.toContain('idx_document_attachments_integrity_check')
|
||||
})
|
||||
|
||||
it('keeps the legacy column itself: historical values are evidence', async () => {
|
||||
const { rows } = await getPool().query<{ column_name: string }>(
|
||||
`SELECT column_name FROM information_schema.columns
|
||||
WHERE table_schema = 'public' AND table_name = 'document_attachments'
|
||||
AND column_name = 'last_integrity_check_at'`,
|
||||
)
|
||||
expect(rows).toHaveLength(1)
|
||||
})
|
||||
})
|
||||
Reference in New Issue
Block a user