fix(processing-history): register the missing event types, and strip the PII two of them carry (#2111)
* fix(processing-history): register the missing event types, and strip the PII two of them carry Ten event types are emitted by code but absent from processing_event_types, so every append fails the foreign key. Appends are best-effort try/catch, so no user request fails, but the internal audit trail is empty for ten kinds of legally motivated act, including the BFL 5 kap 5 § rattelse record when a user swaps a transaction's underlag (TransactionDocumentReplaced) and the SOC 2 revocation record (OAuthClientRevoked). Order matters and is deliberate. Two invoice-inbox events, RateLimitedDropped and AttachmentsTruncated, put the raw sender address and mail subject in their payload. Registering those types first would start persisting that PII into an append-only table whose UPDATE is trigger-blocked and which the archive's erasure path excludes. The strip therefore ships in this same commit, ahead of the migration. Only the invoice-inbox emitter was edited. whatsapp-inbox shares the RateLimitedDropped type name with a payload that carries no phone number. Closes the class rather than the two logged instances: a TypeScript union makes an unregistered literal a compile error, and the pg test asserts the database catalog is a superset of the code's list, generated from the union. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ifKg6Ec67A39oxfGPU1yc * fix(processing-history): strip the inbound-mail PII in the database, not by deploy ordering Review finding (superagent-security, P2): shipping the emitter fix and the catalog migration in one commit is not the same as one instant. Migrations apply on merge while the replacement build takes minutes, so an old instance can still write a sender address and mail subject in that window, and such a row is permanent: processing_history takes no UPDATE and no DELETE, and the archive export excludes it from the erasure path. Adds a BEFORE INSERT trigger stripping `from` and `subject` from the RateLimitedDropped and AttachmentsTruncated payloads, and keeps it afterwards so the invariant belongs to the table rather than to one emitter's good behaviour. The jsonb object check is load bearing: `payload - 'key'` raises on a jsonb array and payload's shape is not constrained. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ifKg6Ec67A39oxfGPU1yc --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jakob Wennberg
@@ -1427,6 +1427,7 @@ One line per decision: `[YYYY-MM-DD] <decision>: <why>`. Appended by agents and
|
||||
[2026-09-01] multi_user skeptic fixes: Stripe cancel EXPIRES the multi_user stripe grant instead of deleting it (grace anchor; other grants still deleted per freeze-and-retain); app-side state checks go RPC-first via SECURITY DEFINER company_multi_user_state (capability_grants RLS hides team rows from non-team users, byrå clients would misread as frozen); byra-kind teams get a standing team-scoped multi_user grant via backfill + teams trigger (WL-10 assumption made real; partner billing is out-of-band); PGRST202 on resolution fails OPEN (pre-migration DB has no multi_user rows: gated fallback would freeze all non-owners); /api/v1 got the same dormancy gate as MCP. RLS-level enforcement and the mid-session API fallback write-back window stay v2 follow-ups (documented, same class as pre-existing stale-preference fallback).
|
||||
[2026-09-01] Declined CodeRabbit's UpgradeNote suggestion (PR #1758 follow-up) to append the self-host connector sentence to children instead of replacing them: every caller's children is hosted subscription copy ("... kräver ett abonnemang"), so appending would show subscription wording on a self-host, the exact thing the branch exists to avoid; the "CSV/SIE import stays free" text it cited is a code comment in BankSyncNowButton, not children. Replace-on-self-host stays; a dedicated selfHosted children prop can come when a caller actually needs per-panel reassurance there.
|
||||
[2026-09-01] getConnectorConfig() rebuilds baseUrl as origin + path (userinfo/query/fragment stripped, warn-logged without the raw value): /api/connector/status echoes baseUrl to the operator and the sync/proxy URLs get paths appended, so nothing secret-shaped pasted into GNUBOK_CONNECT_URL may survive; the stripped parts were never meaningful in a base URL. The status route is also Cache-Control: no-store (key prefix + wiring layout out of shared browser caches).
|
||||
[2026-09-01] processing_history PII: the sender address and mail subject are dropped outright from RateLimitedDropped and AttachmentsTruncated rather than hashed, and the strip ships in the same commit as, and ahead of, the migration registering those event types. Registering first would start persisting PII into an append-only table the archive's erasure path excludes.
|
||||
[2026-09-01] White-label backend guard inverted from an allowlist of protected hosts to an assertion that any customer-facing production host is served by the production Supabase project. This reverses the earlier explicit-allowlist decision: that model failed open for improveone.accounted.se, which was serving a byra login page wired to the staging project with no alert, because it was never added to the list.
|
||||
[2026-09-01] Anon-callable SECURITY DEFINER writes: the guard shape `IF auth.uid() IS NOT NULL AND NOT EXISTS (membership)` is unsafe on its own. The anon JWT carries no `sub` claim, so auth.uid() is NULL for role anon too and the guard short-circuits into the trusted branch. It is defense in depth behind a REVOKE FROM PUBLIC, anon, never a substitute for one. Every new SECURITY DEFINER function ships with that REVOKE; tests/pg/definer-function-grants.pg.test.ts enforces it from a sweep rather than a hand list, because hand-listing is exactly how three guarded numbering RPCs were wrongly declared safe.
|
||||
[2026-09-01] public.create_invoice_with_items(jsonb,jsonb) is revoked, not dropped. It is prod-only, uncalled and already non-functional, so removing it is cleanup rather than security, and the revoke closes the hole in full. An irreversible schema deletion against production belongs in its own reviewable migration.
|
||||
|
||||
@@ -0,0 +1,208 @@
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'
|
||||
import { invoiceInboxExtension } from '@/extensions/general/invoice-inbox'
|
||||
import { createQueuedMockSupabase, createMockRequest } from '@/tests/helpers'
|
||||
|
||||
/**
|
||||
* PII boundary for the two inbound-mail behandlingshistorik events.
|
||||
*
|
||||
* RateLimitedDropped and AttachmentsTruncated used to carry the raw sender
|
||||
* address and the mail subject. Both were harmless only by accident: the types
|
||||
* were missing from processing_event_types, so the FK rejected every insert.
|
||||
* Registering them (migration 20260901110000) switches the writes on, and
|
||||
* processing_history is append-only, UPDATE-blocked by trigger, and excluded
|
||||
* from the archive's erasure path, so the payloads must stay pseudonymous:
|
||||
* lib/processing-history/append.ts only pattern-matches personnummer/orgnr
|
||||
* shapes and would happily persist an email address.
|
||||
*/
|
||||
|
||||
vi.mock('@/extensions/general/invoice-inbox/lib/resend-inbound', async () => {
|
||||
const actual = await vi.importActual<typeof import('@/extensions/general/invoice-inbox/lib/resend-inbound')>(
|
||||
'@/extensions/general/invoice-inbox/lib/resend-inbound'
|
||||
)
|
||||
return {
|
||||
...actual,
|
||||
verifyInboundWebhook: vi.fn(),
|
||||
fetchReceivingEmail: vi.fn(),
|
||||
fetchInboundAttachment: vi.fn(),
|
||||
}
|
||||
})
|
||||
|
||||
vi.mock('@supabase/supabase-js', () => ({
|
||||
createClient: vi.fn(),
|
||||
}))
|
||||
|
||||
vi.mock('@/extensions/general/invoice-inbox/lib/upload-and-extract', async () => {
|
||||
const actual = await vi.importActual<typeof import('@/extensions/general/invoice-inbox/lib/upload-and-extract')>(
|
||||
'@/extensions/general/invoice-inbox/lib/upload-and-extract'
|
||||
)
|
||||
return { ...actual, uploadAndExtract: vi.fn() }
|
||||
})
|
||||
|
||||
vi.mock('@/lib/rate-limits/inbox', () => ({
|
||||
checkInboxUploadRateLimit: vi.fn(),
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/processing-history/append', () => ({
|
||||
appendProcessingHistory: vi.fn().mockResolvedValue('event-id'),
|
||||
}))
|
||||
|
||||
import { verifyInboundWebhook, fetchReceivingEmail, fetchInboundAttachment } from '@/extensions/general/invoice-inbox/lib/resend-inbound'
|
||||
import { checkInboxUploadRateLimit } from '@/lib/rate-limits/inbox'
|
||||
import { appendProcessingHistory } from '@/lib/processing-history/append'
|
||||
import { createClient } from '@supabase/supabase-js'
|
||||
|
||||
const webhookRoute = invoiceInboxExtension.apiRoutes!.find(
|
||||
(r) => r.method === 'POST' && r.path === '/inbound'
|
||||
)!
|
||||
|
||||
const SENDER = 'anna.andersson@leverantoren.se'
|
||||
const SUBJECT = 'Faktura till Anna Andersson'
|
||||
|
||||
function makeAttachment(i: number) {
|
||||
return {
|
||||
id: `att_${i}`,
|
||||
filename: `invoice-${i}.pdf`,
|
||||
size: 1000,
|
||||
content_type: 'application/pdf',
|
||||
content_id: `cid${i}`,
|
||||
content_disposition: 'attachment',
|
||||
}
|
||||
}
|
||||
|
||||
function mockReceivedEvent(attachmentCount: number) {
|
||||
return {
|
||||
type: 'email.received' as const,
|
||||
created_at: '2026-09-01T10:00:00Z',
|
||||
data: {
|
||||
email_id: 'aa1c1c6e-7f2d-4a1e-9f0a-6d5b7c8e9f01',
|
||||
created_at: '2026-09-01T10:00:00Z',
|
||||
from: SENDER,
|
||||
to: ['acme-ab-x7f2@arcim.io'],
|
||||
cc: [],
|
||||
bcc: [],
|
||||
subject: SUBJECT,
|
||||
message_id: '<msg-id@leverantoren.se>',
|
||||
attachments: Array.from({ length: attachmentCount }, (_, i) => makeAttachment(i)),
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
function mockFullEmail(attachmentCount: number) {
|
||||
return {
|
||||
object: 'email',
|
||||
id: 'aa1c1c6e-7f2d-4a1e-9f0a-6d5b7c8e9f01',
|
||||
to: ['acme-ab-x7f2@arcim.io'],
|
||||
from: SENDER,
|
||||
created_at: '2026-09-01T10:00:00Z',
|
||||
subject: SUBJECT,
|
||||
bcc: null,
|
||||
cc: null,
|
||||
reply_to: null,
|
||||
html: null,
|
||||
text: 'Faktura bifogad',
|
||||
headers: {},
|
||||
message_id: '<msg-id@leverantoren.se>',
|
||||
raw: null,
|
||||
attachments: Array.from({ length: attachmentCount }, (_, i) => makeAttachment(i)),
|
||||
}
|
||||
}
|
||||
|
||||
/** The payload of the single history event of `eventType`, or undefined. */
|
||||
function historyPayload(eventType: string): Record<string, unknown> | undefined {
|
||||
const call = vi
|
||||
.mocked(appendProcessingHistory)
|
||||
.mock.calls.find(([input]) => input.eventType === eventType)
|
||||
return call?.[0].payload
|
||||
}
|
||||
|
||||
describe('POST /inbound behandlingshistorik payloads', () => {
|
||||
const originalEnv = { ...process.env }
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
process.env.RESEND_INBOUND_DOMAIN = 'arcim.io'
|
||||
process.env.NEXT_PUBLIC_SUPABASE_URL = 'http://localhost'
|
||||
process.env.SUPABASE_SERVICE_ROLE_KEY = 'test-service-key'
|
||||
vi.mocked(appendProcessingHistory).mockResolvedValue('event-id')
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
process.env = { ...originalEnv }
|
||||
})
|
||||
|
||||
it('records RateLimitedDropped without the sender address or the subject', async () => {
|
||||
vi.mocked(verifyInboundWebhook).mockReturnValue(mockReceivedEvent(3) as never)
|
||||
vi.mocked(fetchReceivingEmail).mockResolvedValue(mockFullEmail(3) as never)
|
||||
vi.mocked(checkInboxUploadRateLimit).mockResolvedValue({
|
||||
ok: false,
|
||||
scope: 'day',
|
||||
retryAfterSec: 3600,
|
||||
} as never)
|
||||
|
||||
const { supabase, enqueue } = createQueuedMockSupabase()
|
||||
enqueue({ data: { id: 'inbox-1', company_id: 'company-1', status: 'active' } })
|
||||
enqueue({ data: { created_by: 'user-owner-1' } })
|
||||
vi.mocked(createClient).mockReturnValue(supabase as never)
|
||||
|
||||
const res = await webhookRoute.handler(
|
||||
createMockRequest('/inbound', { method: 'POST', body: {} })
|
||||
)
|
||||
const body = await res.json()
|
||||
expect(body.data.reason).toBe('rate_limited')
|
||||
|
||||
const payload = historyPayload('RateLimitedDropped')
|
||||
expect(payload).toEqual({ scope: 'day', retry_after_sec: 3600, attachment_count: 3 })
|
||||
expect(JSON.stringify(payload)).not.toContain(SENDER)
|
||||
expect(JSON.stringify(payload)).not.toContain(SUBJECT)
|
||||
})
|
||||
|
||||
it('records AttachmentsTruncated without the sender address or the subject', async () => {
|
||||
// 21 attachments: one over the 20-per-email cap.
|
||||
vi.mocked(verifyInboundWebhook).mockReturnValue(mockReceivedEvent(21) as never)
|
||||
vi.mocked(fetchReceivingEmail).mockResolvedValue(mockFullEmail(21) as never)
|
||||
vi.mocked(checkInboxUploadRateLimit).mockResolvedValue({ ok: true } as never)
|
||||
// Each kept attachment dead-ends in the per-attachment catch, which is
|
||||
// enough: the truncation event is appended before the loop starts.
|
||||
vi.mocked(fetchInboundAttachment).mockRejectedValue(new Error('download disabled in test'))
|
||||
|
||||
const { supabase, enqueue } = createQueuedMockSupabase()
|
||||
enqueue({ data: { id: 'inbox-1', company_id: 'company-1', status: 'active' } })
|
||||
enqueue({ data: { created_by: 'user-owner-1' } })
|
||||
vi.mocked(createClient).mockReturnValue(supabase as never)
|
||||
|
||||
const res = await webhookRoute.handler(
|
||||
createMockRequest('/inbound', { method: 'POST', body: {} })
|
||||
)
|
||||
expect(res.status).toBe(200)
|
||||
|
||||
const payload = historyPayload('AttachmentsTruncated')
|
||||
expect(payload).toEqual({ total: 21, processed: 20, dropped: 1 })
|
||||
expect(JSON.stringify(payload)).not.toContain(SENDER)
|
||||
expect(JSON.stringify(payload)).not.toContain(SUBJECT)
|
||||
})
|
||||
|
||||
it('keeps the mail traceable through the Resend email id', async () => {
|
||||
vi.mocked(verifyInboundWebhook).mockReturnValue(mockReceivedEvent(1) as never)
|
||||
vi.mocked(fetchReceivingEmail).mockResolvedValue(mockFullEmail(1) as never)
|
||||
vi.mocked(checkInboxUploadRateLimit).mockResolvedValue({
|
||||
ok: false,
|
||||
scope: 'minute',
|
||||
retryAfterSec: 60,
|
||||
} as never)
|
||||
|
||||
const { supabase, enqueue } = createQueuedMockSupabase()
|
||||
enqueue({ data: { id: 'inbox-1', company_id: 'company-1', status: 'active' } })
|
||||
enqueue({ data: { created_by: 'user-owner-1' } })
|
||||
vi.mocked(createClient).mockReturnValue(supabase as never)
|
||||
|
||||
await webhookRoute.handler(createMockRequest('/inbound', { method: 'POST', body: {} }))
|
||||
|
||||
// Dropping `from` / `subject` costs nothing for triage: the correlation id
|
||||
// is the Resend email id, and invoice_inbox_items holds the rest.
|
||||
const [input] = vi
|
||||
.mocked(appendProcessingHistory)
|
||||
.mock.calls.find(([i]) => i.eventType === 'RateLimitedDropped')!
|
||||
expect(input.correlationId).toBe('aa1c1c6e-7f2d-4a1e-9f0a-6d5b7c8e9f01')
|
||||
expect(input.aggregateId).toBe('aa1c1c6e-7f2d-4a1e-9f0a-6d5b7c8e9f01')
|
||||
})
|
||||
})
|
||||
@@ -1856,12 +1856,15 @@ export const invoiceInboxExtension: Extension = {
|
||||
aggregateType: 'System',
|
||||
aggregateId: email_id,
|
||||
eventType: 'RateLimitedDropped',
|
||||
// No `from` / `subject`: processing_history is append-only
|
||||
// (UPDATE is trigger-blocked) and outside the archive's erasure
|
||||
// path, so the sender address and the free-text subject may not
|
||||
// land here. correlationId is the Resend email_id, which reaches
|
||||
// both through invoice_inbox_items.
|
||||
payload: {
|
||||
scope: limit.scope,
|
||||
retry_after_sec: limit.retryAfterSec,
|
||||
attachment_count: rawAttachments.length,
|
||||
from,
|
||||
subject,
|
||||
},
|
||||
actor: { type: 'system', id: 'resend-inbound' },
|
||||
occurredAt: new Date(),
|
||||
@@ -1886,12 +1889,11 @@ export const invoiceInboxExtension: Extension = {
|
||||
aggregateType: 'System',
|
||||
aggregateId: email_id,
|
||||
eventType: 'AttachmentsTruncated',
|
||||
// Counts only, for the same reason as RateLimitedDropped above.
|
||||
payload: {
|
||||
total: totalAttachments,
|
||||
processed: attachments.length,
|
||||
dropped: truncatedCount,
|
||||
from,
|
||||
subject,
|
||||
},
|
||||
actor: { type: 'system', id: 'resend-inbound' },
|
||||
occurredAt: new Date(),
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
import { describe, it, expect, vi, beforeEach } from 'vitest'
|
||||
import {
|
||||
PROCESSING_EVENT_TYPES,
|
||||
appendProcessingHistoryWithClient,
|
||||
type ProcessingHistoryEventType,
|
||||
} from '@/lib/processing-history/append'
|
||||
|
||||
vi.mock('@/lib/supabase/server', () => ({
|
||||
createClient: vi.fn(),
|
||||
createServiceClient: vi.fn(),
|
||||
}))
|
||||
|
||||
/** Minimal service-role stand-in: records the inserted row, resolves clean. */
|
||||
function createInsertSpy(error: { message: string } | null = null) {
|
||||
const insert = vi.fn().mockResolvedValue({ error })
|
||||
const from = vi.fn().mockReturnValue({ insert })
|
||||
return { supabase: { from } as never, from, insert }
|
||||
}
|
||||
|
||||
const baseInput = {
|
||||
companyId: '11111111-1111-4111-8111-111111111111',
|
||||
correlationId: '22222222-2222-4222-8222-222222222222',
|
||||
aggregateType: 'System' as const,
|
||||
aggregateId: '22222222-2222-4222-8222-222222222222',
|
||||
eventType: 'OAuthClientRevoked' as const,
|
||||
payload: { client_id: '33333333-3333-4333-8333-333333333333' },
|
||||
actor: { type: 'user' as const, id: '44444444-4444-4444-8444-444444444444' },
|
||||
occurredAt: new Date('2026-09-01T10:00:00Z'),
|
||||
}
|
||||
|
||||
describe('PROCESSING_EVENT_TYPES', () => {
|
||||
it('is sorted and free of duplicates', () => {
|
||||
// The list is read row-by-row against the migration that registers the same
|
||||
// strings; keeping it sorted and unique is what makes that diff readable.
|
||||
const sorted = [...PROCESSING_EVENT_TYPES].sort()
|
||||
expect([...PROCESSING_EVENT_TYPES]).toEqual(sorted)
|
||||
expect(new Set(PROCESSING_EVENT_TYPES).size).toBe(PROCESSING_EVENT_TYPES.length)
|
||||
})
|
||||
|
||||
it('rejects an event type no migration registers', () => {
|
||||
// The real guard is the compiler: processing_history.event_type has an FK
|
||||
// to processing_event_types and every append is best-effort try/catch, so
|
||||
// an unregistered literal is an audit record silently lost at runtime.
|
||||
// If this @ts-expect-error ever goes unused, the union stopped enforcing.
|
||||
const registered: ProcessingHistoryEventType = 'TransactionDocumentReplaced'
|
||||
// @ts-expect-error not a member of PROCESSING_EVENT_TYPES
|
||||
const unregistered: ProcessingHistoryEventType = 'SomethingNobodyRegistered'
|
||||
|
||||
expect(registered).toBe('TransactionDocumentReplaced')
|
||||
expect(unregistered).toBe('SomethingNobodyRegistered')
|
||||
})
|
||||
})
|
||||
|
||||
describe('appendProcessingHistoryWithClient', () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
})
|
||||
|
||||
it('writes the event row and returns the generated event id', async () => {
|
||||
const { supabase, from, insert } = createInsertSpy()
|
||||
|
||||
const eventId = await appendProcessingHistoryWithClient(supabase, baseInput)
|
||||
|
||||
expect(from).toHaveBeenCalledWith('processing_history')
|
||||
expect(insert).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
event_id: eventId,
|
||||
company_id: baseInput.companyId,
|
||||
correlation_id: baseInput.correlationId,
|
||||
causation_id: null,
|
||||
aggregate_type: 'System',
|
||||
aggregate_id: baseInput.aggregateId,
|
||||
event_type: 'OAuthClientRevoked',
|
||||
payload: baseInput.payload,
|
||||
payload_schema_version: 1,
|
||||
occurred_at: '2026-09-01T10:00:00.000Z',
|
||||
})
|
||||
)
|
||||
})
|
||||
|
||||
it('throws with the event type when the insert fails', async () => {
|
||||
// The FK violation on an unregistered event type arrives here. Every call
|
||||
// site swallows it, so the message is the only trace: it must name the type.
|
||||
const { supabase } = createInsertSpy({
|
||||
message: 'insert or update on table "processing_history" violates foreign key constraint',
|
||||
})
|
||||
|
||||
await expect(appendProcessingHistoryWithClient(supabase, baseInput)).rejects.toThrow(
|
||||
/OAuthClientRevoked/
|
||||
)
|
||||
})
|
||||
|
||||
it('rejects a payload carrying a personnummer', async () => {
|
||||
const { supabase, insert } = createInsertSpy()
|
||||
|
||||
await expect(
|
||||
appendProcessingHistoryWithClient(supabase, {
|
||||
...baseInput,
|
||||
payload: { note: '900101-1234' },
|
||||
})
|
||||
).rejects.toThrow()
|
||||
expect(insert).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('accepts a payload of UUIDs, counts and booleans', async () => {
|
||||
const { supabase, insert } = createInsertSpy()
|
||||
|
||||
await appendProcessingHistoryWithClient(supabase, {
|
||||
...baseInput,
|
||||
eventType: 'AttachmentsTruncated',
|
||||
payload: { total: 24, processed: 20, dropped: 4 },
|
||||
})
|
||||
|
||||
expect(insert).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
event_type: 'AttachmentsTruncated',
|
||||
payload: { total: 24, processed: 20, dropped: 4 },
|
||||
})
|
||||
)
|
||||
})
|
||||
})
|
||||
@@ -29,6 +29,43 @@ import { z } from 'zod'
|
||||
*/
|
||||
type SupabaseClientLike = Pick<SupabaseClient, 'from'>
|
||||
|
||||
// ── Event type catalog ──────────────────────────────────────────
|
||||
// Every event type the code emits, and the contract with the
|
||||
// processing_event_types reference table: processing_history.event_type has an
|
||||
// FK to it, and every append call site is best-effort try/catch, so a type
|
||||
// that is missing from the table fails the insert silently and the act leaves
|
||||
// no durable record at all. Ten types drifted out of the table exactly that
|
||||
// way before this list existed.
|
||||
//
|
||||
// Adding an entry here therefore REQUIRES a migration registering the same
|
||||
// string in public.processing_event_types, in the same change. The union
|
||||
// makes an unregistered literal a compile error;
|
||||
// tests/pg/processing-event-types.pg.test.ts makes an unregistered string a
|
||||
// test failure. Keep it sorted.
|
||||
|
||||
export const PROCESSING_EVENT_TYPES = [
|
||||
'AttachmentsTruncated',
|
||||
'BankTransactionDuplicateDismissed',
|
||||
'ChannelQuestionAnswered',
|
||||
'ChannelQuestionAsked',
|
||||
'ChannelQuestionExpired',
|
||||
'DocumentDuplicateSkipped',
|
||||
'DocumentExtractionAttempted',
|
||||
'DocumentExtractionOverridden',
|
||||
'DocumentExtractionRetried',
|
||||
'DocumentIngested',
|
||||
'InboxUnderlagReconciled',
|
||||
'InvoiceDuplicatePaymentDismissed',
|
||||
'InvoiceJournalEntrySkipped',
|
||||
'OAuthClientRevoked',
|
||||
'PendingOperationApproved',
|
||||
'PendingOperationRejected',
|
||||
'RateLimitedDropped',
|
||||
'TransactionDocumentReplaced',
|
||||
] as const
|
||||
|
||||
export type ProcessingHistoryEventType = (typeof PROCESSING_EVENT_TYPES)[number]
|
||||
|
||||
// ── PII validator ───────────────────────────────────────────────
|
||||
// Rejects payloads containing Swedish personal identity numbers.
|
||||
// Personnummer: YYMMDD-NNNN or YYMMDDNNNN (6+4 digits)
|
||||
@@ -88,7 +125,7 @@ export interface AppendEventInput {
|
||||
causationId?: string
|
||||
aggregateType: ProcessingHistoryAggregateType
|
||||
aggregateId: string
|
||||
eventType: string
|
||||
eventType: ProcessingHistoryEventType
|
||||
payload: Record<string, unknown>
|
||||
payloadSchemaVersion?: number
|
||||
actor: ProcessingHistoryActor
|
||||
|
||||
@@ -0,0 +1,110 @@
|
||||
-- Register every behandlingshistorik event type the code emits but the
|
||||
-- catalog is missing (10 of the 18 emitted types).
|
||||
--
|
||||
-- processing_history.event_type has an FK to processing_event_types, so an
|
||||
-- unregistered type fails the insert with 23503, and every appendProcessingHistory
|
||||
-- call site is best-effort try/catch: the failure is swallowed and the act
|
||||
-- leaves NO durable record at all. The catalog has been drifting behind the
|
||||
-- code since the v0.2 seed, and each previous repair (20260626120000,
|
||||
-- 20260721103000, 20260813033506, 20260828154800) registered only the one type
|
||||
-- that happened to surface in the production logs, which is why ten were still
|
||||
-- unwritable. This one backfills the whole emitted set, and
|
||||
-- lib/processing-history/append.ts now carries the same list as a TypeScript
|
||||
-- union so a new literal is a compile error until a migration registers it
|
||||
-- (tests/pg/processing-event-types.pg.test.ts asserts the two stay in sync).
|
||||
--
|
||||
-- What was being lost, per BFNAR 2013:2 kap 9 p. 9.16 (behandlingshistorik:
|
||||
-- the record of how the bookkeeping material was processed):
|
||||
-- TransactionDocumentReplaced the BFL 5 kap 5 § rättelse record when a
|
||||
-- transaction's underlag is swapped
|
||||
-- OAuthClientRevoked revocation evidence for a connected client
|
||||
-- PendingOperationRejected the MCP agent rejection trail (its sibling
|
||||
-- PendingOperationApproved was registered in
|
||||
-- 20260721103000; this one was left behind)
|
||||
-- DocumentExtractionOverridden provenance for an agent-supplied field
|
||||
-- DocumentExtractionRetried re-extraction of an already-ingested doc
|
||||
-- DocumentDuplicateSkipped an ingest that adopted an existing item
|
||||
-- InvoiceDuplicatePaymentDismissed a dismissed double-payment warning
|
||||
-- InvoiceJournalEntrySkipped a commit that wrote no verifikat
|
||||
-- RateLimitedDropped inbound mail/WhatsApp dropped on the cap
|
||||
-- AttachmentsTruncated inbound mail truncated on the 20-file cap
|
||||
--
|
||||
-- The events lost so far are unrecoverable: there is no source to reconstruct
|
||||
-- an append that failed months ago, so there is no backfill of rows to write,
|
||||
-- only of catalog entries.
|
||||
--
|
||||
-- Catalog rows only: every emitter's aggregate_type ('BankTransaction',
|
||||
-- 'Document', 'System') is already permitted by the aggregate_type CHECK, so
|
||||
-- no constraint change is needed.
|
||||
--
|
||||
-- RateLimitedDropped has TWO emitters and this row switches on both: the
|
||||
-- inbound-mail one in extensions/general/invoice-inbox/index.ts (payload
|
||||
-- stripped in this commit, see below) and the WhatsApp intake in
|
||||
-- extensions/general/whatsapp-inbox/lib/process-inbound.ts, whose payload is
|
||||
-- counts plus the inbox row id and was left as it is.
|
||||
--
|
||||
-- Ordering note: this migration must not reach production ahead of the deploy
|
||||
-- that strips `from` and `subject` from the RateLimitedDropped and
|
||||
-- AttachmentsTruncated payloads (extensions/general/invoice-inbox/index.ts).
|
||||
-- Registering those two types is what switches their inserts on, and the old
|
||||
-- payloads carried the sender address and the mail subject into an append-only
|
||||
-- table whose UPDATE is trigger-blocked. Both changes ship in this one commit.
|
||||
--
|
||||
-- One commit is NOT the same as one instant. Migrations apply on merge, while
|
||||
-- the replacement build takes minutes, so there is a window in which an old
|
||||
-- instance is still serving against a database that has just registered these
|
||||
-- two types. A row written in that window is permanent: processing_history
|
||||
-- takes no UPDATE and no DELETE, and lib/reports/full-archive-export.ts
|
||||
-- excludes it from the erasure path. So the strip is enforced in the database
|
||||
-- as well, below, and kept afterwards: the invariant belongs to the table, not
|
||||
-- to one emitter's good behaviour.
|
||||
|
||||
INSERT INTO public.processing_event_types (event_type) VALUES
|
||||
('AttachmentsTruncated'),
|
||||
('DocumentDuplicateSkipped'),
|
||||
('DocumentExtractionOverridden'),
|
||||
('DocumentExtractionRetried'),
|
||||
('InvoiceDuplicatePaymentDismissed'),
|
||||
('InvoiceJournalEntrySkipped'),
|
||||
('OAuthClientRevoked'),
|
||||
('PendingOperationRejected'),
|
||||
('RateLimitedDropped'),
|
||||
('TransactionDocumentReplaced')
|
||||
ON CONFLICT (event_type) DO NOTHING;
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- Database-side PII strip for the two inbound-mail event types.
|
||||
--
|
||||
-- `payload - 'key'` removes a key from a jsonb object and raises on a jsonb
|
||||
-- array, so the object check is load bearing: payload is NOT NULL but its
|
||||
-- shape is not constrained.
|
||||
-- ---------------------------------------------------------------------------
|
||||
|
||||
CREATE OR REPLACE FUNCTION public.strip_inbound_mail_pii_from_processing_history()
|
||||
RETURNS trigger
|
||||
LANGUAGE plpgsql
|
||||
SET search_path = pg_catalog, public
|
||||
AS $$
|
||||
BEGIN
|
||||
IF NEW.event_type IN ('RateLimitedDropped', 'AttachmentsTruncated')
|
||||
AND jsonb_typeof(NEW.payload) = 'object'
|
||||
THEN
|
||||
NEW.payload := NEW.payload - 'from' - 'subject';
|
||||
END IF;
|
||||
RETURN NEW;
|
||||
END;
|
||||
$$;
|
||||
|
||||
DROP TRIGGER IF EXISTS processing_history_strip_inbound_mail_pii ON public.processing_history;
|
||||
|
||||
CREATE TRIGGER processing_history_strip_inbound_mail_pii
|
||||
BEFORE INSERT ON public.processing_history
|
||||
FOR EACH ROW
|
||||
EXECUTE FUNCTION public.strip_inbound_mail_pii_from_processing_history();
|
||||
|
||||
-- The sweep in 20260901100000 revokes PUBLIC/anon on definer writers; this one
|
||||
-- is a trigger function and is never called directly, so it needs no grant.
|
||||
REVOKE ALL ON FUNCTION public.strip_inbound_mail_pii_from_processing_history()
|
||||
FROM PUBLIC, anon, authenticated;
|
||||
|
||||
NOTIFY pgrst, 'reload schema';
|
||||
@@ -0,0 +1,136 @@
|
||||
import { describe, it, expect, beforeAll } from 'vitest'
|
||||
import { randomUUID } from 'node:crypto'
|
||||
import { getClient, getPool } from '@/tests/pg/setup'
|
||||
import { seedCompany } from '@/tests/pg/fixtures'
|
||||
import { PROCESSING_EVENT_TYPES } from '@/lib/processing-history/append'
|
||||
|
||||
/**
|
||||
* Anti-drift guard for behandlingshistorik event types.
|
||||
*
|
||||
* processing_history.event_type has an FK to processing_event_types, and every
|
||||
* appendProcessingHistory call site is best-effort try/catch by design (the
|
||||
* user's operation must not fail because its audit append did). An event type
|
||||
* that is missing from the catalog therefore fails the insert silently, and the
|
||||
* act it records leaves no durable trace at all.
|
||||
*
|
||||
* That drifted four times before this test existed: each production
|
||||
* investigation registered only the one type that had surfaced in the logs
|
||||
* (20260626120000, 20260721103000, 20260813033506, 20260828154800), and ten
|
||||
* emitted types were still unregistered afterwards. The list in
|
||||
* lib/processing-history/append.ts is now the code's half of the contract, and
|
||||
* this test is the DB's half: adding a literal there without a migration fails
|
||||
* here, naming exactly what is missing.
|
||||
*/
|
||||
describe('processing_event_types catalog', () => {
|
||||
let catalog: Set<string>
|
||||
|
||||
beforeAll(async () => {
|
||||
const { rows } = await getPool().query<{ event_type: string }>(
|
||||
'SELECT event_type FROM public.processing_event_types',
|
||||
)
|
||||
catalog = new Set(rows.map((r) => r.event_type))
|
||||
})
|
||||
|
||||
it('registers every event type the code can emit', () => {
|
||||
const missing = PROCESSING_EVENT_TYPES.filter((t) => !catalog.has(t)).sort()
|
||||
|
||||
// Superset, never equality: the v0.2 seed deliberately holds aspirational
|
||||
// types with no emitter today (DocumentClassified, the Match* stream, the
|
||||
// Period* stream, ...). An unused catalog row is harmless; an unregistered
|
||||
// emitted type is a silently lost audit record.
|
||||
expect(missing).toEqual([])
|
||||
})
|
||||
|
||||
it('accepts a row for every registered type through the event_type FK', async () => {
|
||||
const { companyId } = await seedCompany()
|
||||
const client = await getClient()
|
||||
try {
|
||||
await client.query('BEGIN')
|
||||
for (const eventType of PROCESSING_EVENT_TYPES) {
|
||||
const aggregateId = randomUUID()
|
||||
const { rows } = await client.query<{ event_type: string }>(
|
||||
`INSERT INTO public.processing_history
|
||||
(company_id, correlation_id, aggregate_type, aggregate_id, event_type,
|
||||
payload, actor, occurred_at)
|
||||
VALUES ($1, $2, 'System', $2, $3, '{}'::jsonb,
|
||||
'{"type":"system","id":"processing-event-types-test"}', now())
|
||||
RETURNING event_type`,
|
||||
[companyId, aggregateId, eventType],
|
||||
)
|
||||
expect(rows).toEqual([{ event_type: eventType }])
|
||||
}
|
||||
} finally {
|
||||
// The catalog is what is under test; the sample rows are not worth
|
||||
// keeping, and rolling back leaves the shared database as it was.
|
||||
await client.query('ROLLBACK').catch(() => {})
|
||||
client.release()
|
||||
}
|
||||
})
|
||||
|
||||
// The application no longer sends `from` or `subject` for these two types,
|
||||
// but migrations land minutes before the replacement build is live, so an
|
||||
// old instance can still write them in that window. processing_history takes
|
||||
// no UPDATE and no DELETE and is excluded from the archive erasure path, so
|
||||
// such a row would be permanent. The strip is enforced in the database.
|
||||
it.each(['RateLimitedDropped', 'AttachmentsTruncated'])(
|
||||
'strips sender address and subject from a %s payload on insert',
|
||||
async (eventType) => {
|
||||
const { companyId } = await seedCompany()
|
||||
const client = await getClient()
|
||||
try {
|
||||
await client.query('BEGIN')
|
||||
const aggregateId = randomUUID()
|
||||
const { rows } = await client.query<{ payload: Record<string, unknown> }>(
|
||||
`INSERT INTO public.processing_history
|
||||
(company_id, correlation_id, aggregate_type, aggregate_id, event_type,
|
||||
payload, actor, occurred_at)
|
||||
VALUES ($1, $2, 'System', $2, $3,
|
||||
$4::jsonb,
|
||||
'{"type":"system","id":"processing-event-types-test"}', now())
|
||||
RETURNING payload`,
|
||||
[
|
||||
companyId,
|
||||
aggregateId,
|
||||
eventType,
|
||||
JSON.stringify({
|
||||
from: 'avsandare@example.com',
|
||||
subject: 'Faktura 12345',
|
||||
reason: 'rate_limited',
|
||||
count: 3,
|
||||
}),
|
||||
],
|
||||
)
|
||||
expect(rows[0].payload).not.toHaveProperty('from')
|
||||
expect(rows[0].payload).not.toHaveProperty('subject')
|
||||
// Everything that is not PII survives: this is a strip, not a wipe.
|
||||
expect(rows[0].payload).toMatchObject({ reason: 'rate_limited', count: 3 })
|
||||
} finally {
|
||||
await client.query('ROLLBACK').catch(() => {})
|
||||
client.release()
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
it('leaves payloads for other event types untouched', async () => {
|
||||
const { companyId } = await seedCompany()
|
||||
const client = await getClient()
|
||||
try {
|
||||
await client.query('BEGIN')
|
||||
const aggregateId = randomUUID()
|
||||
const { rows } = await client.query<{ payload: Record<string, unknown> }>(
|
||||
`INSERT INTO public.processing_history
|
||||
(company_id, correlation_id, aggregate_type, aggregate_id, event_type,
|
||||
payload, actor, occurred_at)
|
||||
VALUES ($1, $2, 'System', $2, 'DocumentDuplicateSkipped',
|
||||
$3::jsonb,
|
||||
'{"type":"system","id":"processing-event-types-test"}', now())
|
||||
RETURNING payload`,
|
||||
[companyId, aggregateId, JSON.stringify({ from: 'keep-me', subject: 'keep-me-too' })],
|
||||
)
|
||||
expect(rows[0].payload).toMatchObject({ from: 'keep-me', subject: 'keep-me-too' })
|
||||
} finally {
|
||||
await client.query('ROLLBACK').catch(() => {})
|
||||
client.release()
|
||||
}
|
||||
})
|
||||
})
|
||||
Reference in New Issue
Block a user