Files
accounted/lib/processing-history/append.ts
T
Mattsson 88a5d78594 fix(inbox): trace every received mail and file multi-recipient mail once per inbox (#2181) (#2244)
* fix(inbox): trace every received mail and file multi-recipient mail once per inbox (#2181)

A mail sent to both the +lev and +ver address of one inbox was read as
its first recipient only, and an attachment whose processing threw left
no row at all: the webhook answered 200, Resend never retried, and the
document was gone with nothing for the user to find. Prod showed both
shapes for the reporter (a +lev mail Resend accepted with zero inbox
rows, and the second PDF of the +ver mail missing).

- The webhook now reads every shared-domain recipient, groups them per
  inbox, files once per inbox with a company-scoped dedupe key, and
  resolves contradicting tags (+lev and +ver on one mail) to no hint so
  extraction classifies.
- The per-attachment catch writes an error row instead of only a
  console line.
- One InboundMailReceived behandlingshistorik event per mail and inbox
  records recipients, tags, hint, conflict and the outcome per
  attachment (filed, duplicate, rejected, failed). No sender or
  subject, matching the existing PII rule.
- GET /inbound-history?days=30 serves those events, company-scoped, and
  the inbox workspace shows them under Källor as "Inkomna mejl", each
  filed row a click away.
- The list says how many rows the type filter is hiding, with a click
  back to all types.
- Migration 20260903190000 registers the event type and replaces the
  (email, attachment) unique index with (company, email, attachment).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CoG2CXf8B33Q5wp8gk4kW4

* fix(inbox): keep addresses and sender-typed tags out of the mail record, and let redelivery heal a transient failure

Skeptic pass on #2244, two refutations:

- The InboundMailReceived payload carried the recipient addresses and every
  plus-tag verbatim. An enskild firma's inbox local part is the owner's
  name, the tag is whatever the sender typed, and processing_history is
  append-only and outside the erasure path; a numeric tag also tripped the
  PII validator so the record was silently dropped. The event now carries
  inbox_id, the documented tags (+lev/+ver), an unknown-tag count and the
  outcome codes. The history route resolves inbox_id to the company's own
  address at read time. The DB strip trigger from 20260901110000 covers the
  new type (and is recreated, since staging skipped that file).
- The catch-path error row made a Resend redelivery report "duplicate", so
  a transient download or storage failure that used to self-heal on retry
  became permanent. The row is marked transient and a redelivery replaces
  it; rejections (bad type, too large) stay duplicates.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CoG2CXf8B33Q5wp8gk4kW4

* fix(inbox): cap inbound fan-out, flag a truncated mail history, and name a replaced transient row

Review pass on #2244: Superagent (bound the number of inboxes one mail can
fan out to: five), CodeRabbit (the history route now returns has_more past
200 rows and the panel says so instead of "every mail"), and the Swedish
accounting review (a redelivery that replaces a transient error row names
the replaced row on the InboundMailReceived record, so the replacement
leaves a trace). The migration comment states why the index swap is not
CONCURRENTLY: Supabase branching applies migrations in a transaction.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(inbox): resolve every addressed inbox and record the ones past the fan-out cap

CodeRabbit and the Swedish accounting review on #2244: slicing recipient
groups before the lookup let five unknown local parts starve a real inbox
and left companies past the cap with no trace. Every addressed inbox is
now resolved (one cheap lookup each), the first five are processed, and
the rest get their own InboundMailReceived record with outcome
fan_out_capped, shown in the panel as "not processed".

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* chore(inbox): move the inbound-mail migration past the parties versions merged tonight

Main moved party_decision_undo to 20260904000100 and added
20260904000200 (#2257, #2258). A version below prod's head is skipped by
Supabase branching, so 20260903190000 becomes 20260904001000 unchanged.
Staging re-tracked under the new version.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-03 20:47:24 +02:00

247 lines
8.9 KiB
TypeScript

/**
* Processing history (behandlingshistorik): append helper.
*
* Uses a service-role client internally (no INSERT RLS policy on
* processing_history: matching the event_log pattern). Company scoping
* is enforced by companyId in the event payload, not by RLS.
* Throws on failure.
*
* PII BOUNDARY: payload MUST contain pseudonymous IDs only (user UUIDs,
* company UUIDs, counterparty IDs). Never names, emails, personnummer,
* addresses, or phone numbers. These live in their source tables (profiles,
* customers, suppliers) and are referenced by ID. GDPR erasure pseudonymizes
* the source tables; processing_history events become undecipherable by
* reference, which is the required behavior per v0.2 §10.
*/
import type {
ProcessingHistoryAggregateType,
ProcessingHistoryActor,
} from '@/types'
import type { SupabaseClient } from '@supabase/supabase-js'
import { createServiceClient } from '@/lib/supabase/server'
import { z } from 'zod'
/**
* Any service-role client with the query surface the append needs. Structural
* so both the Next-bound createServiceClient() and a script's own
* createClient(url, serviceRoleKey) satisfy it.
*/
type SupabaseClientLike = Pick<SupabaseClient, 'from'>
// ── Event type catalog ──────────────────────────────────────────
// Every event type the code emits, and the contract with the
// processing_event_types reference table: processing_history.event_type has an
// FK to it, and every append call site is best-effort try/catch, so a type
// that is missing from the table fails the insert silently and the act leaves
// no durable record at all. Ten types drifted out of the table exactly that
// way before this list existed.
//
// Adding an entry here therefore REQUIRES a migration registering the same
// string in public.processing_event_types, in the same change. The union
// makes an unregistered literal a compile error;
// tests/pg/processing-event-types.pg.test.ts makes an unregistered string a
// test failure. Keep it sorted.
export const PROCESSING_EVENT_TYPES = [
'AttachmentsTruncated',
'BankTransactionDuplicateDismissed',
'ChannelQuestionAnswered',
'ChannelQuestionAsked',
'ChannelQuestionExpired',
'DocumentDuplicateSkipped',
'DocumentExtractionAttempted',
'DocumentExtractionOverridden',
'DocumentExtractionRetried',
'DocumentIngested',
'InboundMailReceived',
'InboxUnderlagReconciled',
'InvoiceDuplicatePaymentDismissed',
'InvoiceJournalEntrySkipped',
'InvoicePaymentRowBackfilled',
'OAuthClientRevoked',
'PendingOperationApproved',
'PendingOperationRejected',
'RateLimitedDropped',
'TransactionDocumentReplaced',
] as const
export type ProcessingHistoryEventType = (typeof PROCESSING_EVENT_TYPES)[number]
// ── PII validator ───────────────────────────────────────────────
// Rejects payloads containing Swedish personal identity numbers.
// Personnummer: YYMMDD-NNNN or YYMMDDNNNN (6+4 digits)
// Samordningsnummer: Same format but day +60
// Organisationsnummer: NNNNNN-NNNN (10 digits, but we catch the pattern)
// Word boundaries prevent false positives on Bankgiro (123456-7890) and
// invoice references like 202312-1234 that share the digit shape but aren't PII.
const PII_PATTERNS = [
/\b\d{6}-?\d{4}\b/, // personnummer, samordningsnummer
/\b\d{8}-?\d{4}\b/, // 12-digit variant (YYYYMMDD-NNNN) or orgnr
]
// UUIDs (RFC 4122, 8-4-4-4-12 hex layout) frequently contain all-digit segments
// that incorrectly match the 8+4 personnummer pattern: e.g. `57484518-3409-...`.
// Strip UUID-shaped substrings before PII matching so legitimate identifiers
// aren't rejected. Personnummer always sit outside the UUID shape, so this keeps
// the original safety intent intact.
const UUID_PATTERN = /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/gi
function stringContainsPii(value: string): boolean {
const stripped = value.replace(UUID_PATTERN, '')
return PII_PATTERNS.some(pattern => pattern.test(stripped))
}
function containsPii(value: unknown): boolean {
if (typeof value === 'string') {
return stringContainsPii(value)
}
if (Array.isArray(value)) {
return value.some(containsPii)
}
if (value !== null && typeof value === 'object') {
return Object.values(value).some(containsPii)
}
return false
}
const piiSafePayload = z.record(z.string(), z.unknown()).refine(
(payload) => !containsPii(payload),
{ message: 'Payload contains PII (personnummer/samordningsnummer/orgnr pattern). Use pseudonymous IDs only.' }
)
function assertActorPiiSafe(actor: ProcessingHistoryActor): void {
if (actor.label && stringContainsPii(actor.label)) {
throw new Error(
'actor.label contains PII (personnummer/samordningsnummer/orgnr pattern). Use a pseudonymous descriptor only.'
)
}
}
// ── Input type ──────────────────────────────────────────────────
export interface AppendEventInput {
companyId: string
correlationId: string
causationId?: string
aggregateType: ProcessingHistoryAggregateType
aggregateId: string
eventType: ProcessingHistoryEventType
payload: Record<string, unknown>
payloadSchemaVersion?: number
actor: ProcessingHistoryActor
rubricVersion?: string
occurredAt: Date // mandatory: no default. Caller must set explicitly.
}
// ── Append functions ────────────────────────────────────────────
/**
* Append a single event to processing_history.
*
* Uses a service-role client internally (bypasses RLS) since processing_history
* has no INSERT policy: matching the event_log pattern. Company scoping is
* enforced by the companyId in the event payload, not by RLS.
*
* Returns the generated event_id (pre-generated client-side for causation chaining).
*/
export async function appendProcessingHistory(
input: AppendEventInput
): Promise<string> {
return appendProcessingHistoryWithClient(createServiceClient(), input)
}
/**
* Same append, on a caller-supplied service-role client. For standalone
* scripts (e.g. scripts/backfill-inbox-booked-underlag.ts) that cannot build
* the Next-bound service client but must still write behandlingshistorik
* through the one shared row shape and PII validation (BFNAR 2013:2 p. 9.16:
* the change log has to reconcile across writers, so scripts never hand-roll
* the insert).
*/
export async function appendProcessingHistoryWithClient(
supabase: SupabaseClientLike,
input: AppendEventInput
): Promise<string> {
// Validate payload + actor.label contain no PII
piiSafePayload.parse(input.payload)
assertActorPiiSafe(input.actor)
const eventId = crypto.randomUUID()
const { error } = await supabase
.from('processing_history')
.insert({
event_id: eventId,
company_id: input.companyId,
correlation_id: input.correlationId,
causation_id: input.causationId ?? null,
aggregate_type: input.aggregateType,
aggregate_id: input.aggregateId,
event_type: input.eventType,
payload: input.payload,
payload_schema_version: input.payloadSchemaVersion ?? 1,
actor: input.actor,
rubric_version: input.rubricVersion ?? null,
occurred_at: input.occurredAt.toISOString(),
})
if (error) {
throw new Error(
`Failed to append processing_history event ${input.eventType}: ${error.message}`
)
}
return eventId
}
/**
* Append multiple events atomically (single INSERT).
* Used for batch operations (e.g., migration commits, multi-event command handlers).
*
* Returns array of generated event_ids in input order.
*/
export async function appendProcessingHistoryBatch(
inputs: AppendEventInput[]
): Promise<string[]> {
if (inputs.length === 0) return []
const eventIds = inputs.map(() => crypto.randomUUID())
// Validate all payloads + actor labels before any DB write
for (const input of inputs) {
piiSafePayload.parse(input.payload)
assertActorPiiSafe(input.actor)
}
const rows = inputs.map((input, i) => ({
event_id: eventIds[i],
company_id: input.companyId,
correlation_id: input.correlationId,
causation_id: input.causationId ?? null,
aggregate_type: input.aggregateType,
aggregate_id: input.aggregateId,
event_type: input.eventType,
payload: input.payload,
payload_schema_version: input.payloadSchemaVersion ?? 1,
actor: input.actor,
rubric_version: input.rubricVersion ?? null,
occurred_at: input.occurredAt.toISOString(),
}))
const supabase = createServiceClient()
const { error } = await supabase
.from('processing_history')
.insert(rows)
if (error) {
throw new Error(
`Failed to append processing_history batch (${inputs.length} events): ${error.message}`
)
}
return eventIds
}