fix(processing-history): register the missing event types, and strip the PII two of them carry (#2111)

* fix(processing-history): register the missing event types, and strip the PII two of them carry Ten event types are emitted by code but absent from processing_event_types, so every append fails the foreign key. Appends are best-effort try/catch, so no user request fails, but the internal audit trail is empty for ten kinds of legally motivated act, including the BFL 5 kap 5 § rattelse record when a user swaps a transaction's underlag (TransactionDocumentReplaced) and the SOC 2 revocation record (OAuthClientRevoked). Order matters and is deliberate. Two invoice-inbox events, RateLimitedDropped and AttachmentsTruncated, put the raw sender address and mail subject in their payload. Registering those types first would start persisting that PII into an append-only table whose UPDATE is trigger-blocked and which the archive's erasure path excludes. The strip therefore ships in this same commit, ahead of the migration. Only the invoice-inbox emitter was edited. whatsapp-inbox shares the RateLimitedDropped type name with a payload that carries no phone number. Closes the class rather than the two logged instances: a TypeScript union makes an unregistered literal a compile error, and the pg test asserts the database catalog is a superset of the code's list, generated from the union. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ifKg6Ec67A39oxfGPU1yc * fix(processing-history): strip the inbound-mail PII in the database, not by deploy ordering Review finding (superagent-security, P2): shipping the emitter fix and the catalog migration in one commit is not the same as one instant. Migrations apply on merge while the replacement build takes minutes, so an old instance can still write a sender address and mail subject in that window, and such a row is permanent: processing_history takes no UPDATE and no DELETE, and the archive export excludes it from the erasure path. Adds a BEFORE INSERT trigger stripping `from` and `subject` from the RateLimitedDropped and AttachmentsTruncated payloads, and keeps it afterwards so the invariant belongs to the table rather than to one emitter's good behaviour. The jsonb object check is load bearing: `payload - 'key'` raises on a jsonb array and payload's shape is not constrained. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ifKg6Ec67A39oxfGPU1yc --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-01 14:14:09 +02:00
parent 21e6e2d314
commit 088754d61a
7 changed files with 620 additions and 5 deletions
@@ -0,0 +1,110 @@
+-- Register every behandlingshistorik event type the code emits but the
+-- catalog is missing (10 of the 18 emitted types).
+--
+-- processing_history.event_type has an FK to processing_event_types, so an
+-- unregistered type fails the insert with 23503, and every appendProcessingHistory
+-- call site is best-effort try/catch: the failure is swallowed and the act
+-- leaves NO durable record at all. The catalog has been drifting behind the
+-- code since the v0.2 seed, and each previous repair (20260626120000,
+-- 20260721103000, 20260813033506, 20260828154800) registered only the one type
+-- that happened to surface in the production logs, which is why ten were still
+-- unwritable. This one backfills the whole emitted set, and
+-- lib/processing-history/append.ts now carries the same list as a TypeScript
+-- union so a new literal is a compile error until a migration registers it
+-- (tests/pg/processing-event-types.pg.test.ts asserts the two stay in sync).
+--
+-- What was being lost, per BFNAR 2013:2 kap 9 p. 9.16 (behandlingshistorik:
+-- the record of how the bookkeeping material was processed):
+--   TransactionDocumentReplaced      the BFL 5 kap 5 § rättelse record when a
+--                                    transaction's underlag is swapped
+--   OAuthClientRevoked               revocation evidence for a connected client
+--   PendingOperationRejected         the MCP agent rejection trail (its sibling
+--                                    PendingOperationApproved was registered in
+--                                    20260721103000; this one was left behind)
+--   DocumentExtractionOverridden     provenance for an agent-supplied field
+--   DocumentExtractionRetried        re-extraction of an already-ingested doc
+--   DocumentDuplicateSkipped         an ingest that adopted an existing item
+--   InvoiceDuplicatePaymentDismissed a dismissed double-payment warning
+--   InvoiceJournalEntrySkipped       a commit that wrote no verifikat
+--   RateLimitedDropped               inbound mail/WhatsApp dropped on the cap
+--   AttachmentsTruncated             inbound mail truncated on the 20-file cap
+--
+-- The events lost so far are unrecoverable: there is no source to reconstruct
+-- an append that failed months ago, so there is no backfill of rows to write,
+-- only of catalog entries.
+--
+-- Catalog rows only: every emitter's aggregate_type ('BankTransaction',
+-- 'Document', 'System') is already permitted by the aggregate_type CHECK, so
+-- no constraint change is needed.
+--
+-- RateLimitedDropped has TWO emitters and this row switches on both: the
+-- inbound-mail one in extensions/general/invoice-inbox/index.ts (payload
+-- stripped in this commit, see below) and the WhatsApp intake in
+-- extensions/general/whatsapp-inbox/lib/process-inbound.ts, whose payload is
+-- counts plus the inbox row id and was left as it is.
+--
+-- Ordering note: this migration must not reach production ahead of the deploy
+-- that strips `from` and `subject` from the RateLimitedDropped and
+-- AttachmentsTruncated payloads (extensions/general/invoice-inbox/index.ts).
+-- Registering those two types is what switches their inserts on, and the old
+-- payloads carried the sender address and the mail subject into an append-only
+-- table whose UPDATE is trigger-blocked. Both changes ship in this one commit.
+--
+-- One commit is NOT the same as one instant. Migrations apply on merge, while
+-- the replacement build takes minutes, so there is a window in which an old
+-- instance is still serving against a database that has just registered these
+-- two types. A row written in that window is permanent: processing_history
+-- takes no UPDATE and no DELETE, and lib/reports/full-archive-export.ts
+-- excludes it from the erasure path. So the strip is enforced in the database
+-- as well, below, and kept afterwards: the invariant belongs to the table, not
+-- to one emitter's good behaviour.
+
+INSERT INTO public.processing_event_types (event_type) VALUES
+  ('AttachmentsTruncated'),
+  ('DocumentDuplicateSkipped'),
+  ('DocumentExtractionOverridden'),
+  ('DocumentExtractionRetried'),
+  ('InvoiceDuplicatePaymentDismissed'),
+  ('InvoiceJournalEntrySkipped'),
+  ('OAuthClientRevoked'),
+  ('PendingOperationRejected'),
+  ('RateLimitedDropped'),
+  ('TransactionDocumentReplaced')
+ON CONFLICT (event_type) DO NOTHING;
+
+-- ---------------------------------------------------------------------------
+-- Database-side PII strip for the two inbound-mail event types.
+--
+-- `payload - 'key'` removes a key from a jsonb object and raises on a jsonb
+-- array, so the object check is load bearing: payload is NOT NULL but its
+-- shape is not constrained.
+-- ---------------------------------------------------------------------------
+
+CREATE OR REPLACE FUNCTION public.strip_inbound_mail_pii_from_processing_history()
+RETURNS trigger
+LANGUAGE plpgsql
+SET search_path = pg_catalog, public
+AS $$
+BEGIN
+  IF NEW.event_type IN ('RateLimitedDropped', 'AttachmentsTruncated')
+     AND jsonb_typeof(NEW.payload) = 'object'
+  THEN
+    NEW.payload := NEW.payload - 'from' - 'subject';
+  END IF;
+  RETURN NEW;
+END;
+$$;
+
+DROP TRIGGER IF EXISTS processing_history_strip_inbound_mail_pii ON public.processing_history;
+
+CREATE TRIGGER processing_history_strip_inbound_mail_pii
+  BEFORE INSERT ON public.processing_history
+  FOR EACH ROW
+  EXECUTE FUNCTION public.strip_inbound_mail_pii_from_processing_history();
+
+-- The sweep in 20260901100000 revokes PUBLIC/anon on definer writers; this one
+-- is a trigger function and is never called directly, so it needs no grant.
+REVOKE ALL ON FUNCTION public.strip_inbound_mail_pii_from_processing_history()
+  FROM PUBLIC, anon, authenticated;
+
+NOTIFY pgrst, 'reload schema';