feat(whatsapp-inbox): GDPR retention cron and RoPA entry (#1341)
PR5b, the final code piece of the WhatsApp intake track. A daily cron (04:15) enforces the channel's retention table; the receipt itself stays 7-year WORM under BFL and is never touched. Retention actions (lib/retention.ts, each isolated and idempotent): - whatsapp_messages transcripts past 90 days: body_text + raw_payload cleared in id batches under a wall-clock budget; the row skeleton (wamid, direction, timestamps, status, inbox_item_id) survives for audit. Only rows still carrying content match. - Rows with phone_link_id IS NULL (unknown senders, orphans) past 30 days: deleted. - Link codes expired more than 24h ago: deleted, used or not. - Sender rate counters idle 2+ days: deleted (minute/day window keys are dead weight after that). - Links revoked 90+ days ago: phone_enc crypto-shredded to '' (column is NOT NULL), one-shot via neq guard; phone_hash and phone_masked kept for uniqueness history and audit display. Route mirrors the sweep cron exactly: withCronContext + registry gate (503 EXTENSION_DISABLED when the extension is off). vercel.json gets the schedule and both Docker crontabs are regenerated. Compliance: new whatsapp.receipt_intake activity in .compliance/ropa.yaml covering purpose, Art 6(1)(b)/(c)/(f) bases with the Art 14(5)(b) note for third-party attendee names, Meta Platforms Ireland as processor (Cloud API, EU SCC addendum, Local Storage region DE), the differentiated retention table, and security measures. Co-authored-by: Jakob Wennberg <jakob.wennberg@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Jakob Wennberg
Claude Fable 5
parent
629069e281
commit
bf5ca2c615
@@ -662,3 +662,93 @@ processing_activities:
|
|||||||
- support_ticket_identity_hmac_signed # POSTHOG_SECRET_API_KEY, serverside
|
- support_ticket_identity_hmac_signed # POSTHOG_SECRET_API_KEY, serverside
|
||||||
- support_message_body_never_in_event_properties # breadcrumb bär endast subject
|
- support_message_body_never_in_event_properties # breadcrumb bär endast subject
|
||||||
- email_remains_the_delivery_guarantee # ärendet är ett komplement, ej leveransväg
|
- email_remains_the_delivery_guarantee # ärendet är ett komplement, ej leveransväg
|
||||||
|
|
||||||
|
- id: whatsapp.receipt_intake
|
||||||
|
name: Kvittoinlämning och klargörande dialog via WhatsApp
|
||||||
|
purpose: >-
|
||||||
|
Ta emot kvitton som foto eller PDF skickade till Accounteds delade
|
||||||
|
WhatsApp-nummer och landa dem i dokumentinkorgen (Underlag) som
|
||||||
|
räkenskapsinformation, samt föra en kort klargörande dialog i samma
|
||||||
|
chatt: bekräfta mottagning, fråga vilket företag underlaget tillhör och
|
||||||
|
samla in de representationsuppgifter Skatteverket kräver (deltagarnas
|
||||||
|
namn och företag samt syfte). Avsändare är enbart befintliga
|
||||||
|
Accounted-användare som själva knutit sitt nummer med en engångskod;
|
||||||
|
okända nummer får ett enda begränsat standardsvar och behandlas aldrig
|
||||||
|
vidare (ingen AI, ingen mediahämtning, inget innehåll sparas).
|
||||||
|
Deltagarnamn i representationssvar är tredjepartsuppgifter som samlas
|
||||||
|
in från användaren, inte från de registrerade själva; individuell
|
||||||
|
information till varje deltagare vore oproportionerlig i förhållande
|
||||||
|
till den lagstadgade dokumentationsplikten och undantaget i art.
|
||||||
|
14.5 b tillämpas (uppgifterna framgår av verifikationen, som bevaras
|
||||||
|
enligt BFL).
|
||||||
|
# art 6.1 b: tjänsten (mottagning, dialog, inkorgsleverans).
|
||||||
|
# art 6.1 c: BFL 7 kap: kvittot bevaras 7 år som räkenskapsinformation.
|
||||||
|
# art 6.1 f: tredjeparts deltagarnamn i representationssvar (med
|
||||||
|
# art 14.5 b-undantaget dokumenterat i purpose ovan).
|
||||||
|
lawful_basis: art_6_1_b_c_and_f
|
||||||
|
special_category_basis: null
|
||||||
|
controller: gnubok-tenant
|
||||||
|
processor: supabase_and_meta
|
||||||
|
data_subjects:
|
||||||
|
- business_owner
|
||||||
|
- company_member
|
||||||
|
- representation_attendee # tredjepart: namn + företag i representationssvar
|
||||||
|
data_categories:
|
||||||
|
- user.contact.phone_number # HMAC-hash med peppar + AES-256-GCM + mask; aldrig i klartext i DB
|
||||||
|
- user.communication_content # chattext och rå webhook-payload, 90 dagars retention
|
||||||
|
- user.document # kvittofiler, 7 år WORM enligt BFL 7 kap
|
||||||
|
- user.name # deltagarnamn i representationssvar (tredjepart)
|
||||||
|
- user.activity_timestamp
|
||||||
|
recipients:
|
||||||
|
- name: Meta Platforms Ireland Ltd (WhatsApp Business Platform, Cloud API)
|
||||||
|
country: IE
|
||||||
|
role: processor
|
||||||
|
- name: Supabase
|
||||||
|
country: EU
|
||||||
|
role: processor
|
||||||
|
# AI-tolkningen av kvittobilder och fritextsvar sker via Amazon Bedrock
|
||||||
|
# och täcks av den befintliga aktiviteten ai.inference ovan.
|
||||||
|
international_transfers:
|
||||||
|
applicable: true
|
||||||
|
mechanism: scc_2021_c2p
|
||||||
|
note: >-
|
||||||
|
Meta Platforms Ireland behandlar meddelandena under WhatsApp Business
|
||||||
|
dataskyddsvillkor med EU SCC-addendum (Module 2) för vidareöverföring
|
||||||
|
till Meta Platforms Inc. (US). Cloud API Local Storage är aktiverat
|
||||||
|
med region DE, så meddelandeinnehåll lagras i vila inom EU hos Meta.
|
||||||
|
retention:
|
||||||
|
# Differentierad per datakategori; verkställs dagligen av
|
||||||
|
# /api/extensions/whatsapp-inbox/retention/cron:
|
||||||
|
# chattranskript (body_text, raw_payload) 90 dagar, sedan rensas
|
||||||
|
# innehållet men radskelettet (wamid, riktning, tidsstämplar,
|
||||||
|
# status, inbox-koppling) bevaras som behandlingsspår
|
||||||
|
# okända avsändares rader 30 dagar, sedan radering
|
||||||
|
# engångskoder 24 h efter utgång, radering
|
||||||
|
# telefonkoppling (phone_enc) tills användaren återkallar,
|
||||||
|
# därefter kryptoshredning 90 dagar efter revokering (phone_hash och
|
||||||
|
# phone_masked bevaras för unikhetshistorik respektive visning)
|
||||||
|
# kvittofilen 7 år, WORM (BFL 7 kap),
|
||||||
|
# hanteras av dokumentarkivet, ej av denna cron
|
||||||
|
duration: differentiated_per_data_category
|
||||||
|
basis: gdpr_storage_limitation_and_bfl_7_kap
|
||||||
|
stored_in:
|
||||||
|
- whatsapp_messages
|
||||||
|
- whatsapp_phone_links
|
||||||
|
- whatsapp_link_codes
|
||||||
|
- whatsapp_sender_rate_counters
|
||||||
|
- whatsapp_conversations
|
||||||
|
- invoice_inbox_items.channel_context
|
||||||
|
- document_attachments # kvittofilen, WORM
|
||||||
|
security_measures:
|
||||||
|
- phone_number_hmac_peppered_hash_lookup_only
|
||||||
|
- phone_number_aes_256_gcm_at_rest
|
||||||
|
- phone_number_masked_in_ui
|
||||||
|
- dedicated_whatsapp_key_material_not_bankid_keys
|
||||||
|
- webhook_signature_verified_over_raw_body
|
||||||
|
- service_role_only_tables_rls_no_policies
|
||||||
|
- unknown_senders_no_content_persistence_no_llm
|
||||||
|
- pre_binding_sender_rate_limits
|
||||||
|
- daily_retention_cron_with_crypto_shred
|
||||||
|
- llm_reads_answers_as_untrusted_data_no_tools
|
||||||
|
- receipt_files_worm_protected_bfl
|
||||||
|
- opt_out_stop_keyword_honored
|
||||||
|
|||||||
@@ -784,3 +784,4 @@ One line per decision: `[YYYY-MM-DD] <decision>: <why>`. Appended by agents and
|
|||||||
[2026-08-02] whatsapp-inbox PR4 re-send (M9): a sharper re-sent file creates a NEW inbox item and marks the old one channel_context.quality.superseded=true instead of swapping the old item's document. Replacing in place has no supported path (attach-document 409s when document_id is set) and would orphan the original WORM document against the anchored-doc invariant (see 2026-07-27 floating-underlag work).
|
[2026-08-02] whatsapp-inbox PR4 re-send (M9): a sharper re-sent file creates a NEW inbox item and marks the old one channel_context.quality.superseded=true instead of swapping the old item's document. Replacing in place has no supported path (attach-document 409s when document_id is set) and would orphan the original WORM document against the anchored-doc invariant (see 2026-07-27 floating-underlag work).
|
||||||
[2026-08-02] whatsapp-inbox PR4 schema: ONE new migration (20260802210000) adds whatsapp_messages.acked_at. Burst-ack membership (which ingested rows the single combined M4/M5 covered) must be derivable relationally across webhook invocations; deriving it from "last outbound ack timestamp" breaks the moment one outbound insert fails, and conversation-context accumulation races.
|
[2026-08-02] whatsapp-inbox PR4 schema: ONE new migration (20260802210000) adds whatsapp_messages.acked_at. Burst-ack membership (which ingested rows the single combined M4/M5 covered) must be derivable relationally across webhook invocations; deriving it from "last outbound ack timestamp" breaks the moment one outbound insert fails, and conversation-context accumulation races.
|
||||||
[2026-08-02] whatsapp-inbox M6 body omits merchant/amount (spec showed them): for multi-company senders the item is created only after the company answer, so nothing is extracted when the question is asked. Body degrades to "Vilket företag gäller kvittot/kvittona?".
|
[2026-08-02] whatsapp-inbox M6 body omits merchant/amount (spec showed them): for multi-company senders the item is created only after the company answer, so nothing is extracted when the question is asked. Body degrades to "Vilket företag gäller kvittot/kvittona?".
|
||||||
|
[2026-08-02] whatsapp-inbox PR5b retention: revoked links crypto-shred by setting phone_enc='' (column is NOT NULL; empty string is the cleared marker, guarded by neq for one-shot idempotency) while phone_hash and phone_masked survive on purpose (re-link uniqueness history + audit display). Scoped to whatsapp_* tables only: inbox_rate_counters (the per-company sibling) has NO cleanup anywhere and stays that way here; adding one is a separate change, noted in the PR body.
|
||||||
|
|||||||
@@ -0,0 +1,97 @@
|
|||||||
|
import { describe, it, expect, vi, beforeEach } from 'vitest'
|
||||||
|
import { NextResponse } from 'next/server'
|
||||||
|
|
||||||
|
vi.mock('@/lib/extensions/loader', () => ({
|
||||||
|
loadExtensions: vi.fn(),
|
||||||
|
}))
|
||||||
|
|
||||||
|
vi.mock('@/lib/extensions/registry', () => ({
|
||||||
|
extensionRegistry: {
|
||||||
|
get: vi.fn(),
|
||||||
|
},
|
||||||
|
}))
|
||||||
|
|
||||||
|
vi.mock('@/lib/auth/api-keys', () => ({
|
||||||
|
createServiceClientNoCookies: vi.fn().mockReturnValue({}),
|
||||||
|
}))
|
||||||
|
|
||||||
|
vi.mock('@/extensions/general/whatsapp-inbox/lib/retention', () => ({
|
||||||
|
runRetention: vi.fn(),
|
||||||
|
}))
|
||||||
|
|
||||||
|
vi.mock('@/lib/auth/cron', () => ({
|
||||||
|
verifyCronSecret: vi.fn().mockReturnValue(null),
|
||||||
|
}))
|
||||||
|
|
||||||
|
import { GET } from '../route'
|
||||||
|
import { extensionRegistry } from '@/lib/extensions/registry'
|
||||||
|
import { loadExtensions } from '@/lib/extensions/loader'
|
||||||
|
import { runRetention } from '@/extensions/general/whatsapp-inbox/lib/retention'
|
||||||
|
import { verifyCronSecret } from '@/lib/auth/cron'
|
||||||
|
|
||||||
|
const mockRegistryGet = vi.mocked(extensionRegistry.get)
|
||||||
|
const mockVerifyCronSecret = vi.mocked(verifyCronSecret)
|
||||||
|
const mockRunRetention = vi.mocked(runRetention)
|
||||||
|
|
||||||
|
function makeRequest() {
|
||||||
|
return new Request('http://localhost/api/extensions/whatsapp-inbox/retention/cron', {
|
||||||
|
headers: { authorization: 'Bearer synthetic-cron-secret' },
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
beforeEach(() => {
|
||||||
|
vi.clearAllMocks()
|
||||||
|
mockVerifyCronSecret.mockReturnValue(null)
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('GET /api/extensions/whatsapp-inbox/retention/cron', () => {
|
||||||
|
it('returns 401 when the cron secret is rejected', async () => {
|
||||||
|
mockVerifyCronSecret.mockReturnValue(
|
||||||
|
NextResponse.json({ error: 'Unauthorized' }, { status: 401 }),
|
||||||
|
)
|
||||||
|
|
||||||
|
const response = await GET(makeRequest())
|
||||||
|
|
||||||
|
expect(response.status).toBe(401)
|
||||||
|
expect(mockRunRetention).not.toHaveBeenCalled()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('returns 503 EXTENSION_DISABLED when the extension is not in the registry', async () => {
|
||||||
|
// Physical extension routes deploy in every build; the registry, generated
|
||||||
|
// from extensions.config.json, is what turns them on. Disabled must mean
|
||||||
|
// no purging AND a visible failure if the cron is scheduled anyway.
|
||||||
|
mockRegistryGet.mockReturnValue(undefined)
|
||||||
|
|
||||||
|
const response = await GET(makeRequest())
|
||||||
|
const body = await response.json()
|
||||||
|
|
||||||
|
expect(response.status).toBe(503)
|
||||||
|
expect(body.code).toBe('EXTENSION_DISABLED')
|
||||||
|
expect(mockRunRetention).not.toHaveBeenCalled()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('runs the retention pass and returns its summary when enabled', async () => {
|
||||||
|
mockRegistryGet.mockReturnValue({ id: 'whatsapp-inbox' } as never)
|
||||||
|
mockRunRetention.mockResolvedValue({
|
||||||
|
purgedTranscripts: 12,
|
||||||
|
deletedUnknownSenderMessages: 4,
|
||||||
|
deletedLinkCodes: 2,
|
||||||
|
deletedRateCounters: 9,
|
||||||
|
shreddedRevokedLinks: 1,
|
||||||
|
})
|
||||||
|
|
||||||
|
const response = await GET(makeRequest())
|
||||||
|
const body = await response.json()
|
||||||
|
|
||||||
|
expect(loadExtensions).toHaveBeenCalled()
|
||||||
|
expect(mockRegistryGet).toHaveBeenCalledWith('whatsapp-inbox')
|
||||||
|
expect(response.status).toBe(200)
|
||||||
|
expect(body.data).toEqual({
|
||||||
|
purgedTranscripts: 12,
|
||||||
|
deletedUnknownSenderMessages: 4,
|
||||||
|
deletedLinkCodes: 2,
|
||||||
|
deletedRateCounters: 9,
|
||||||
|
shreddedRevokedLinks: 1,
|
||||||
|
})
|
||||||
|
})
|
||||||
|
})
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
import { NextResponse } from 'next/server'
|
||||||
|
import { loadExtensions } from '@/lib/extensions/loader'
|
||||||
|
import { extensionRegistry } from '@/lib/extensions/registry'
|
||||||
|
import { withCronContext } from '@/lib/api/with-cron-context'
|
||||||
|
import { createServiceClientNoCookies } from '@/lib/auth/api-keys'
|
||||||
|
import { runRetention } from '@/extensions/general/whatsapp-inbox/lib/retention'
|
||||||
|
|
||||||
|
/**
|
||||||
|
* GET /api/extensions/whatsapp-inbox/retention/cron: daily GDPR purge for
|
||||||
|
* the WhatsApp channel. Enforces the retention table in .compliance/ropa.yaml
|
||||||
|
* (whatsapp.receipt_intake): 90-day chat transcripts, 30-day unknown-sender
|
||||||
|
* rows, expired link codes, dead rate counters, and the 90-day crypto-shred
|
||||||
|
* of revoked phone bindings. Receipts themselves are 7-year WORM under BFL
|
||||||
|
* and are never touched here. Scheduled in vercel.json (and the generated
|
||||||
|
* Docker crontabs).
|
||||||
|
*
|
||||||
|
* Overlap with a slow previous run is safe: every predicate only matches
|
||||||
|
* rows still carrying the data, so a second pass is a no-op.
|
||||||
|
*/
|
||||||
|
|
||||||
|
// The transcript purge loops over a possibly large first-run backlog.
|
||||||
|
export const maxDuration = 300
|
||||||
|
|
||||||
|
export const GET = withCronContext('cron.whatsapp_retention', async (_request, ctx) => {
|
||||||
|
// Load the registry so it reflects extensions.config.json.
|
||||||
|
loadExtensions()
|
||||||
|
|
||||||
|
// Physical routes under app/api/extensions/<id>/ compile into EVERY build,
|
||||||
|
// including the core-with-zero-extensions one: the registry (generated from
|
||||||
|
// extensions.config.json) is what actually switches an extension on. Mirror
|
||||||
|
// the ext/[...path] dispatcher: a disabled extension must not expose a live
|
||||||
|
// surface, and a scheduled-but-disabled cron must fail visibly (503)
|
||||||
|
// instead of quietly doing the work anyway.
|
||||||
|
if (!extensionRegistry.get('whatsapp-inbox')) {
|
||||||
|
ctx.log.warn('whatsapp-inbox extension is not enabled; cron refused')
|
||||||
|
return NextResponse.json(
|
||||||
|
{ error: 'WhatsApp inbox extension is not enabled', code: 'EXTENSION_DISABLED' },
|
||||||
|
{ status: 503 },
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
const supabase = createServiceClientNoCookies()
|
||||||
|
const summary = await runRetention(supabase)
|
||||||
|
|
||||||
|
ctx.log.info('whatsapp retention complete', { ...summary })
|
||||||
|
|
||||||
|
return NextResponse.json({ data: summary })
|
||||||
|
})
|
||||||
@@ -40,4 +40,5 @@
|
|||||||
30 */2 * * * curl -sf -H "Authorization: Bearer ${CRON_SECRET}" ${APP_URL}/api/extensions/skatteverket/vat/kvittenser/cron
|
30 */2 * * * curl -sf -H "Authorization: Bearer ${CRON_SECRET}" ${APP_URL}/api/extensions/skatteverket/vat/kvittenser/cron
|
||||||
* * * * * curl -sf -H "Authorization: Bearer ${CRON_SECRET}" ${APP_URL}/api/webhooks/dispatch/cron
|
* * * * * curl -sf -H "Authorization: Bearer ${CRON_SECRET}" ${APP_URL}/api/webhooks/dispatch/cron
|
||||||
* * * * * curl -sf -H "Authorization: Bearer ${CRON_SECRET}" ${APP_URL}/api/extensions/whatsapp-inbox/sweep/cron
|
* * * * * curl -sf -H "Authorization: Bearer ${CRON_SECRET}" ${APP_URL}/api/extensions/whatsapp-inbox/sweep/cron
|
||||||
|
15 4 * * * curl -sf -H "Authorization: Bearer ${CRON_SECRET}" ${APP_URL}/api/extensions/whatsapp-inbox/retention/cron
|
||||||
15 5 * * * curl -sf -H "Authorization: Bearer ${CRON_SECRET}" ${APP_URL}/api/bookkeeping/accruals/post-due/cron
|
15 5 * * * curl -sf -H "Authorization: Bearer ${CRON_SECRET}" ${APP_URL}/api/bookkeeping/accruals/post-due/cron
|
||||||
|
|||||||
@@ -40,4 +40,5 @@
|
|||||||
30 */2 * * * curl -sf -H "Authorization: Bearer ${CRON_SECRET}" ${APP_URL}/api/extensions/skatteverket/vat/kvittenser/cron
|
30 */2 * * * curl -sf -H "Authorization: Bearer ${CRON_SECRET}" ${APP_URL}/api/extensions/skatteverket/vat/kvittenser/cron
|
||||||
* * * * * curl -sf -H "Authorization: Bearer ${CRON_SECRET}" ${APP_URL}/api/webhooks/dispatch/cron
|
* * * * * curl -sf -H "Authorization: Bearer ${CRON_SECRET}" ${APP_URL}/api/webhooks/dispatch/cron
|
||||||
* * * * * curl -sf -H "Authorization: Bearer ${CRON_SECRET}" ${APP_URL}/api/extensions/whatsapp-inbox/sweep/cron
|
* * * * * curl -sf -H "Authorization: Bearer ${CRON_SECRET}" ${APP_URL}/api/extensions/whatsapp-inbox/sweep/cron
|
||||||
|
15 4 * * * curl -sf -H "Authorization: Bearer ${CRON_SECRET}" ${APP_URL}/api/extensions/whatsapp-inbox/retention/cron
|
||||||
15 5 * * * curl -sf -H "Authorization: Bearer ${CRON_SECRET}" ${APP_URL}/api/bookkeeping/accruals/post-due/cron
|
15 5 * * * curl -sf -H "Authorization: Bearer ${CRON_SECRET}" ${APP_URL}/api/bookkeeping/accruals/post-due/cron
|
||||||
|
|||||||
@@ -0,0 +1,209 @@
|
|||||||
|
import { describe, it, expect, vi, beforeEach } from 'vitest'
|
||||||
|
import { createQueuedMockSupabase } from '@/tests/helpers'
|
||||||
|
import type { SupabaseClient } from '@supabase/supabase-js'
|
||||||
|
import {
|
||||||
|
BATCH,
|
||||||
|
LINK_CODE_GRACE_HOURS,
|
||||||
|
RATE_COUNTER_RETENTION_DAYS,
|
||||||
|
REVOKED_LINK_SHRED_DAYS,
|
||||||
|
TRANSCRIPT_RETENTION_DAYS,
|
||||||
|
UNKNOWN_SENDER_RETENTION_DAYS,
|
||||||
|
runRetention,
|
||||||
|
} from '@/extensions/general/whatsapp-inbox/lib/retention'
|
||||||
|
|
||||||
|
const DAY_MS = 24 * 60 * 60 * 1000
|
||||||
|
const HOUR_MS = 60 * 60 * 1000
|
||||||
|
|
||||||
|
function daysAgo(days: number): string {
|
||||||
|
return new Date(Date.now() - days * DAY_MS).toISOString()
|
||||||
|
}
|
||||||
|
|
||||||
|
function hoursAgo(hours: number): string {
|
||||||
|
return new Date(Date.now() - hours * HOUR_MS).toISOString()
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The recorded cutoff must sit within a minute of `now - expectedMs`. */
|
||||||
|
function expectCutoffAt(cutoffIso: unknown, expectedMs: number) {
|
||||||
|
expect(typeof cutoffIso).toBe('string')
|
||||||
|
const drift = Math.abs(Date.now() - expectedMs - new Date(cutoffIso as string).getTime())
|
||||||
|
expect(drift).toBeLessThan(60 * 1000)
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Enqueue one full no-op pass in execution order:
|
||||||
|
* unknown-sender delete, transcript select (empty ends the loop),
|
||||||
|
* link codes, rate counters, revoked-link shred.
|
||||||
|
*/
|
||||||
|
function enqueueEmptyRun(
|
||||||
|
enqueue: (r: { data?: unknown; error?: unknown; count?: number | null }) => void,
|
||||||
|
overrides: Partial<
|
||||||
|
Record<'unknown' | 'select' | 'codes' | 'counters' | 'shred', {
|
||||||
|
data?: unknown
|
||||||
|
error?: unknown
|
||||||
|
count?: number | null
|
||||||
|
}>
|
||||||
|
> = {},
|
||||||
|
) {
|
||||||
|
enqueue(overrides.unknown ?? { data: null, count: 0 })
|
||||||
|
enqueue(overrides.select ?? { data: [] })
|
||||||
|
enqueue(overrides.codes ?? { data: null, count: 0 })
|
||||||
|
enqueue(overrides.counters ?? { data: null, count: 0 })
|
||||||
|
enqueue(overrides.shred ?? { data: null, count: 0 })
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('runRetention', () => {
|
||||||
|
beforeEach(() => {
|
||||||
|
vi.clearAllMocks()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('purges transcript content at the 90-day boundary: 89-day row untouched, 91-day row purged', async () => {
|
||||||
|
const { supabase, enqueue, findCalls, calls } = createQueuedMockSupabase()
|
||||||
|
enqueue({ data: null, count: 0 }) // unknown-sender delete
|
||||||
|
// The DB filter (created_at < cutoff) returns only the 91-day row.
|
||||||
|
enqueue({ data: [{ id: 'm-91d' }] }) // transcript select, batch 1
|
||||||
|
enqueue({ data: null }) // transcript update, batch 1 (partial batch ends loop)
|
||||||
|
enqueue({ data: null, count: 0 }) // link codes
|
||||||
|
enqueue({ data: null, count: 0 }) // counters
|
||||||
|
enqueue({ data: null, count: 0 }) // shred
|
||||||
|
|
||||||
|
const summary = await runRetention(supabase as unknown as SupabaseClient)
|
||||||
|
|
||||||
|
// Second lt() on whatsapp_messages belongs to the transcript select
|
||||||
|
// (the first is the unknown-sender delete's 30-day cutoff).
|
||||||
|
const ltCalls = findCalls('whatsapp_messages', 'lt')
|
||||||
|
const [, cutoff] = ltCalls[1] as [string, string]
|
||||||
|
expect(ltCalls[1][0]).toBe('created_at')
|
||||||
|
expectCutoffAt(cutoff, TRANSCRIPT_RETENTION_DAYS * DAY_MS)
|
||||||
|
// Predicate boundary: an 89-day row does NOT satisfy created_at < cutoff,
|
||||||
|
// a 91-day row does.
|
||||||
|
expect(daysAgo(89) < cutoff).toBe(false)
|
||||||
|
expect(daysAgo(91) < cutoff).toBe(true)
|
||||||
|
|
||||||
|
// Idempotency: only rows still carrying content are selected.
|
||||||
|
const orCall = calls.find((c) => c.table === 'whatsapp_messages' && c.method === 'or')
|
||||||
|
expect(orCall?.args[0]).toBe('body_text.not.is.null,raw_payload.not.is.null')
|
||||||
|
|
||||||
|
// The purge NULLs content only; the row skeleton survives.
|
||||||
|
const patch = findCalls('whatsapp_messages', 'update')[0][0] as Record<string, unknown>
|
||||||
|
expect(patch).toEqual({ body_text: null, raw_payload: null })
|
||||||
|
const inCall = calls.find((c) => c.table === 'whatsapp_messages' && c.method === 'in')
|
||||||
|
expect(inCall?.args).toEqual(['id', ['m-91d']])
|
||||||
|
|
||||||
|
expect(summary.purgedTranscripts).toBe(1)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('loops the transcript purge in batches until a partial batch', async () => {
|
||||||
|
const { supabase, enqueue, findCalls } = createQueuedMockSupabase()
|
||||||
|
const fullBatch = Array.from({ length: BATCH }, (_, i) => ({ id: `m-${i}` }))
|
||||||
|
enqueue({ data: null, count: 0 }) // unknown-sender delete
|
||||||
|
enqueue({ data: fullBatch }) // select, batch 1 (full -> loop again)
|
||||||
|
enqueue({ data: null }) // update, batch 1
|
||||||
|
enqueue({ data: [{ id: 'm-last' }] }) // select, batch 2 (partial -> stop)
|
||||||
|
enqueue({ data: null }) // update, batch 2
|
||||||
|
enqueue({ data: null, count: 0 }) // link codes
|
||||||
|
enqueue({ data: null, count: 0 }) // counters
|
||||||
|
enqueue({ data: null, count: 0 }) // shred
|
||||||
|
|
||||||
|
const summary = await runRetention(supabase as unknown as SupabaseClient)
|
||||||
|
|
||||||
|
expect(findCalls('whatsapp_messages', 'update')).toHaveLength(2)
|
||||||
|
expect(summary.purgedTranscripts).toBe(BATCH + 1)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('deletes unknown-sender rows at the 30-day boundary, never linked rows', async () => {
|
||||||
|
const { supabase, enqueue, findCalls, calls } = createQueuedMockSupabase()
|
||||||
|
enqueueEmptyRun(enqueue, { unknown: { data: null, count: 3 } })
|
||||||
|
|
||||||
|
const summary = await runRetention(supabase as unknown as SupabaseClient)
|
||||||
|
|
||||||
|
const isCall = calls.find((c) => c.table === 'whatsapp_messages' && c.method === 'is')
|
||||||
|
expect(isCall?.args).toEqual(['phone_link_id', null])
|
||||||
|
|
||||||
|
const [column, cutoff] = findCalls('whatsapp_messages', 'lt')[0] as [string, string]
|
||||||
|
expect(column).toBe('created_at')
|
||||||
|
expectCutoffAt(cutoff, UNKNOWN_SENDER_RETENTION_DAYS * DAY_MS)
|
||||||
|
expect(daysAgo(29) < cutoff).toBe(false)
|
||||||
|
expect(daysAgo(31) < cutoff).toBe(true)
|
||||||
|
|
||||||
|
expect(summary.deletedUnknownSenderMessages).toBe(3)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('deletes link codes 24h after expiry, used or not', async () => {
|
||||||
|
const { supabase, enqueue, findCalls } = createQueuedMockSupabase()
|
||||||
|
enqueueEmptyRun(enqueue, { codes: { data: null, count: 2 } })
|
||||||
|
|
||||||
|
const summary = await runRetention(supabase as unknown as SupabaseClient)
|
||||||
|
|
||||||
|
expect(findCalls('whatsapp_link_codes', 'delete')).toHaveLength(1)
|
||||||
|
const [column, cutoff] = findCalls('whatsapp_link_codes', 'lt')[0] as [string, string]
|
||||||
|
expect(column).toBe('expires_at')
|
||||||
|
expectCutoffAt(cutoff, LINK_CODE_GRACE_HOURS * HOUR_MS)
|
||||||
|
// No used_at filter: used and unused codes go alike.
|
||||||
|
expect(hoursAgo(23) < cutoff).toBe(false)
|
||||||
|
expect(hoursAgo(25) < cutoff).toBe(true)
|
||||||
|
|
||||||
|
expect(summary.deletedLinkCodes).toBe(2)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('deletes rate counters idle for more than 2 days', async () => {
|
||||||
|
const { supabase, enqueue, findCalls } = createQueuedMockSupabase()
|
||||||
|
enqueueEmptyRun(enqueue, { counters: { data: null, count: 5 } })
|
||||||
|
|
||||||
|
const summary = await runRetention(supabase as unknown as SupabaseClient)
|
||||||
|
|
||||||
|
expect(findCalls('whatsapp_sender_rate_counters', 'delete')).toHaveLength(1)
|
||||||
|
const [column, cutoff] = findCalls('whatsapp_sender_rate_counters', 'lt')[0] as [
|
||||||
|
string,
|
||||||
|
string,
|
||||||
|
]
|
||||||
|
expect(column).toBe('updated_at')
|
||||||
|
expectCutoffAt(cutoff, RATE_COUNTER_RETENTION_DAYS * DAY_MS)
|
||||||
|
|
||||||
|
expect(summary.deletedRateCounters).toBe(5)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('crypto-shreds revoked links only after 90 days, only once, keeping hash and mask', async () => {
|
||||||
|
const { supabase, enqueue, findCalls, calls } = createQueuedMockSupabase()
|
||||||
|
enqueueEmptyRun(enqueue, { shred: { data: null, count: 1 } })
|
||||||
|
|
||||||
|
const summary = await runRetention(supabase as unknown as SupabaseClient)
|
||||||
|
|
||||||
|
const [patch, options] = findCalls('whatsapp_phone_links', 'update')[0] as [
|
||||||
|
Record<string, unknown>,
|
||||||
|
Record<string, unknown>,
|
||||||
|
]
|
||||||
|
// phone_enc is NOT NULL in the schema: '' is the cleared marker. The
|
||||||
|
// patch must never touch phone_hash (uniqueness history) or phone_masked
|
||||||
|
// (audit display).
|
||||||
|
expect(patch).toEqual({ phone_enc: '' })
|
||||||
|
expect(options).toEqual({ count: 'exact' })
|
||||||
|
|
||||||
|
const [column, cutoff] = findCalls('whatsapp_phone_links', 'lt')[0] as [string, string]
|
||||||
|
expect(column).toBe('revoked_at')
|
||||||
|
expectCutoffAt(cutoff, REVOKED_LINK_SHRED_DAYS * DAY_MS)
|
||||||
|
// Boundary: revoked 89 days ago stays readable, 91 days ago is shredded.
|
||||||
|
expect(daysAgo(89) < cutoff).toBe(false)
|
||||||
|
expect(daysAgo(91) < cutoff).toBe(true)
|
||||||
|
// Only once: already-cleared rows are excluded by the neq guard, so a
|
||||||
|
// re-run never re-touches (or re-counts) them.
|
||||||
|
const neqCall = calls.find((c) => c.table === 'whatsapp_phone_links' && c.method === 'neq')
|
||||||
|
expect(neqCall?.args).toEqual(['phone_enc', ''])
|
||||||
|
|
||||||
|
expect(summary.shreddedRevokedLinks).toBe(1)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('isolates failures: one failing action never blocks the others', async () => {
|
||||||
|
const { supabase, enqueue } = createQueuedMockSupabase()
|
||||||
|
enqueueEmptyRun(enqueue, {
|
||||||
|
unknown: { error: { message: 'boom' } },
|
||||||
|
codes: { data: null, count: 4 },
|
||||||
|
shred: { data: null, count: 2 },
|
||||||
|
})
|
||||||
|
|
||||||
|
const summary = await runRetention(supabase as unknown as SupabaseClient)
|
||||||
|
|
||||||
|
expect(summary.deletedUnknownSenderMessages).toBe(0)
|
||||||
|
expect(summary.deletedLinkCodes).toBe(4)
|
||||||
|
expect(summary.shreddedRevokedLinks).toBe(2)
|
||||||
|
})
|
||||||
|
})
|
||||||
@@ -0,0 +1,168 @@
|
|||||||
|
/**
|
||||||
|
* Daily GDPR retention purge for the WhatsApp channel.
|
||||||
|
*
|
||||||
|
* The receipt itself is 7-year WORM räkenskapsinformation (BFL 7 kap) and is
|
||||||
|
* untouched here; everything conversational around it is short-lived. Each
|
||||||
|
* pass enforces the retention table documented in .compliance/ropa.yaml
|
||||||
|
* (whatsapp.receipt_intake):
|
||||||
|
*
|
||||||
|
* 1. Chat transcripts (body_text + raw_payload on whatsapp_messages) are
|
||||||
|
* purged after 90 days. The row skeleton survives for audit: wamid,
|
||||||
|
* direction, timestamps, processing_status and inbox_item_id keep the
|
||||||
|
* "a message existed and produced this Underlag row" trail without the
|
||||||
|
* content.
|
||||||
|
* 2. Rows with no phone_link_id (unknown senders, plus rows orphaned by a
|
||||||
|
* link deletion) are DELETED after 30 days: pre-binding traffic has no
|
||||||
|
* contractual basis to keep, only the abuse-throttle window.
|
||||||
|
* 3. Link codes are deleted 24h after expiry, used or not. The 10-minute
|
||||||
|
* TTL plus single-use flag already made them dead; this removes the
|
||||||
|
* hashes entirely.
|
||||||
|
* 4. Sender rate counters older than 2 days are deleted: window keys are
|
||||||
|
* minute/day, so anything older can never be read again.
|
||||||
|
* 5. Revoked phone links are crypto-shredded 90 days after revocation:
|
||||||
|
* phone_enc is cleared to '' (the column is NOT NULL, so empty string is
|
||||||
|
* the cleared marker). phone_hash stays (uniqueness history: the same
|
||||||
|
* phone re-linking later must still be resolvable) and phone_masked
|
||||||
|
* stays (display in any audit surface).
|
||||||
|
*
|
||||||
|
* Every action is idempotent (predicates only match rows still carrying the
|
||||||
|
* data) and isolated in its own try/catch: one failing table never blocks
|
||||||
|
* the purge of another. Volume-unbounded actions loop in id batches under a
|
||||||
|
* shared wall-clock budget, mirroring the sweep's stance that a partial pass
|
||||||
|
* is a latency regression, never a correctness problem: tomorrow's run picks
|
||||||
|
* up the remainder.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import type { SupabaseClient } from '@supabase/supabase-js'
|
||||||
|
import { createLogger } from '@/lib/logger'
|
||||||
|
|
||||||
|
const log = createLogger('whatsapp-inbox/retention')
|
||||||
|
|
||||||
|
const DAY_MS = 24 * 60 * 60 * 1000
|
||||||
|
const HOUR_MS = 60 * 60 * 1000
|
||||||
|
|
||||||
|
export const TRANSCRIPT_RETENTION_DAYS = 90
|
||||||
|
export const UNKNOWN_SENDER_RETENTION_DAYS = 30
|
||||||
|
export const LINK_CODE_GRACE_HOURS = 24
|
||||||
|
export const RATE_COUNTER_RETENTION_DAYS = 2
|
||||||
|
export const REVOKED_LINK_SHRED_DAYS = 90
|
||||||
|
|
||||||
|
export const BATCH = 200
|
||||||
|
/** maxDuration on the cron route is 300s; leave headroom for the response. */
|
||||||
|
const TIME_BUDGET_MS = 4 * 60 * 1000
|
||||||
|
|
||||||
|
export interface RetentionSummary {
|
||||||
|
purgedTranscripts: number
|
||||||
|
deletedUnknownSenderMessages: number
|
||||||
|
deletedLinkCodes: number
|
||||||
|
deletedRateCounters: number
|
||||||
|
shreddedRevokedLinks: number
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Run one retention pass. Never throws. */
|
||||||
|
export async function runRetention(supabase: SupabaseClient): Promise<RetentionSummary> {
|
||||||
|
const summary: RetentionSummary = {
|
||||||
|
purgedTranscripts: 0,
|
||||||
|
deletedUnknownSenderMessages: 0,
|
||||||
|
deletedLinkCodes: 0,
|
||||||
|
deletedRateCounters: 0,
|
||||||
|
shreddedRevokedLinks: 0,
|
||||||
|
}
|
||||||
|
const startedAt = Date.now()
|
||||||
|
const budgetLeft = () => Date.now() - startedAt < TIME_BUDGET_MS
|
||||||
|
|
||||||
|
// ── 1. Unknown-sender rows past 30 days: DELETE ─────────────
|
||||||
|
// Runs before the transcript purge so a row that is both link-less and
|
||||||
|
// past 90 days is deleted outright instead of being content-purged first.
|
||||||
|
// Also catches outbound greeting rows (persistOutbound stores them with
|
||||||
|
// phone_link_id null) and rows orphaned by ON DELETE SET NULL.
|
||||||
|
try {
|
||||||
|
const cutoff = new Date(startedAt - UNKNOWN_SENDER_RETENTION_DAYS * DAY_MS).toISOString()
|
||||||
|
const { count, error } = await supabase
|
||||||
|
.from('whatsapp_messages')
|
||||||
|
.delete({ count: 'exact' })
|
||||||
|
.is('phone_link_id', null)
|
||||||
|
.lt('created_at', cutoff)
|
||||||
|
if (error) throw error
|
||||||
|
summary.deletedUnknownSenderMessages = count ?? 0
|
||||||
|
} catch (err) {
|
||||||
|
log.error('retention: unknown-sender delete failed', err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── 2. Transcripts past 90 days: purge content, keep skeleton ──
|
||||||
|
// Batched: the first pass over a backlog can touch many rows, and an
|
||||||
|
// unbounded UPDATE risks the statement timeout. Only rows still carrying
|
||||||
|
// content match, so re-runs never churn already-purged rows.
|
||||||
|
try {
|
||||||
|
const cutoff = new Date(startedAt - TRANSCRIPT_RETENTION_DAYS * DAY_MS).toISOString()
|
||||||
|
while (budgetLeft()) {
|
||||||
|
const { data, error } = await supabase
|
||||||
|
.from('whatsapp_messages')
|
||||||
|
.select('id')
|
||||||
|
.lt('created_at', cutoff)
|
||||||
|
.or('body_text.not.is.null,raw_payload.not.is.null')
|
||||||
|
.order('created_at', { ascending: true })
|
||||||
|
.limit(BATCH)
|
||||||
|
if (error) throw error
|
||||||
|
const ids = ((data ?? []) as { id: string }[]).map((row) => row.id)
|
||||||
|
if (ids.length === 0) break
|
||||||
|
const { error: updateError } = await supabase
|
||||||
|
.from('whatsapp_messages')
|
||||||
|
.update({ body_text: null, raw_payload: null })
|
||||||
|
.in('id', ids)
|
||||||
|
if (updateError) throw updateError
|
||||||
|
summary.purgedTranscripts += ids.length
|
||||||
|
if (ids.length < BATCH) break
|
||||||
|
}
|
||||||
|
} catch (err) {
|
||||||
|
log.error('retention: transcript purge failed', err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── 3. Link codes expired more than 24h ago: DELETE ─────────
|
||||||
|
try {
|
||||||
|
const cutoff = new Date(startedAt - LINK_CODE_GRACE_HOURS * HOUR_MS).toISOString()
|
||||||
|
const { count, error } = await supabase
|
||||||
|
.from('whatsapp_link_codes')
|
||||||
|
.delete({ count: 'exact' })
|
||||||
|
.lt('expires_at', cutoff)
|
||||||
|
if (error) throw error
|
||||||
|
summary.deletedLinkCodes = count ?? 0
|
||||||
|
} catch (err) {
|
||||||
|
log.error('retention: link-code cleanup failed', err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── 4. Rate counters idle for 2+ days: DELETE ───────────────
|
||||||
|
// Window keys are minute/day; a counter not touched for 2 days can never
|
||||||
|
// be incremented or read again. (inbox_rate_counters, the per-company
|
||||||
|
// sibling, has no equivalent cleanup anywhere yet; ours starts clean.)
|
||||||
|
try {
|
||||||
|
const cutoff = new Date(startedAt - RATE_COUNTER_RETENTION_DAYS * DAY_MS).toISOString()
|
||||||
|
const { count, error } = await supabase
|
||||||
|
.from('whatsapp_sender_rate_counters')
|
||||||
|
.delete({ count: 'exact' })
|
||||||
|
.lt('updated_at', cutoff)
|
||||||
|
if (error) throw error
|
||||||
|
summary.deletedRateCounters = count ?? 0
|
||||||
|
} catch (err) {
|
||||||
|
log.error('retention: rate-counter cleanup failed', err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── 5. Links revoked 90+ days ago: crypto-shred phone_enc ───
|
||||||
|
// phone_enc is NOT NULL, so '' is the cleared marker; the neq guard makes
|
||||||
|
// the shred one-shot. phone_hash and phone_masked survive on purpose (see
|
||||||
|
// module docblock).
|
||||||
|
try {
|
||||||
|
const cutoff = new Date(startedAt - REVOKED_LINK_SHRED_DAYS * DAY_MS).toISOString()
|
||||||
|
const { count, error } = await supabase
|
||||||
|
.from('whatsapp_phone_links')
|
||||||
|
.update({ phone_enc: '' }, { count: 'exact' })
|
||||||
|
.lt('revoked_at', cutoff)
|
||||||
|
.neq('phone_enc', '')
|
||||||
|
if (error) throw error
|
||||||
|
summary.shreddedRevokedLinks = count ?? 0
|
||||||
|
} catch (err) {
|
||||||
|
log.error('retention: revoked-link shred failed', err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return summary
|
||||||
|
}
|
||||||
@@ -69,6 +69,10 @@
|
|||||||
"path": "/api/extensions/whatsapp-inbox/sweep/cron",
|
"path": "/api/extensions/whatsapp-inbox/sweep/cron",
|
||||||
"schedule": "* * * * *"
|
"schedule": "* * * * *"
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"path": "/api/extensions/whatsapp-inbox/retention/cron",
|
||||||
|
"schedule": "15 4 * * *"
|
||||||
|
},
|
||||||
{
|
{
|
||||||
"path": "/api/bookkeeping/accruals/post-due/cron",
|
"path": "/api/bookkeeping/accruals/post-due/cron",
|
||||||
"schedule": "15 5 * * *"
|
"schedule": "15 5 * * *"
|
||||||
|
|||||||
Reference in New Issue
Block a user