1a5d205bd6
* fix(webhooks): derive the stuck-in_flight window from the cycle bound and charge the stall an attempt recoverStuckInFlight re-armed any in_flight row older than 2x REQUEST_TIMEOUT_MS (20 s), but a cron cycle claims 50 rows and attempts them serially, stamping updated_at once at claim time. From row 3 onward every row was past the threshold before its own attempt started, so each cycle recovered and re-claimed the rows the previous cycle was still working through: duplicate POSTs of the same X-Gnubok-Delivery, and a terminal status decided by a race whose loser was swallowed by enforce_webhook_delivery_immutability as a log.warn. Both halves of #1257 are fixed: 1. The window is derived, not guessed. The attempt loop is now bounded by an explicit CYCLE_BUDGET_MS (120 s) instead of relying on the platform to kill it, and the sweep window is that bound plus one receiver timeout plus slack (160 s), floored at the cron's own batch size so the 5-row emit kick cannot re-arm rows the 50-row cron still owns. Each row is also re-stamped immediately before its own attempt, so a row's in_flight age measures the attempt rather than the claim. The same write doubles as an ownership check: a zero-row result means another cycle took the row, and the POST is dropped instead of duplicated. 2. The sweep charges an attempt, so MAX_ATTEMPTS is a real cap again. The predicate moves into a SECURITY DEFINER RPC because PostgREST can express neither `attempts = attempts + 1` nor the conditional flip at the cap, and a read-then-write loop would reopen a TOCTOU against the immutability trigger. A row recovered past the cap lands on exactly the terminal state the normal retry path produces: status 'dead', attempts = MAX_ATTEMPTS, error prefixed 'attempts_exhausted'. The trigger is neither weakened nor bypassed: the outer UPDATE keeps status = 'in_flight' in its own WHERE, so a row that raced to a terminal status fails re-evaluation under READ COMMITTED and is skipped rather than aborting the statement. Rows the cycle claimed but will not reach are handed back as re-claimable instead of being stranded in in_flight, without charging an attempt they never made. Adds the partial index the sweep needs (idx_webhook_deliveries_due is partial on pending/failed and structurally excludes in_flight). No retention or pruning cron: webhook_deliveries still has no cleanup path, which is a separate decision and stays a follow-up. Fixes #1257 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(webhooks): back the cycle budget with maxDuration and give the stall the normal retry backoff Review follow-up on the #1257 fix. Two of the findings were blocking and compound each other: the fix made stranding likely and destructive at the same time. 1. The 160 s sweep window was derived from CYCLE_BUDGET_MS, but nothing granted a dispatch cycle 120 s: the cron route declared no maxDuration. If the platform killed the invocation before the budget check fired, releaseUnattempted never ran and the claimed-but- unattempted rows stayed in in_flight carrying their claim-time updated_at, which is exactly the invariant the window depends on. The route now declares maxDuration = 300, the way the stripe transactions and documents verify crons pair a budget with one, and a route test asserts both the literal and its relation to CYCLE_BUDGET_MS. The kick path can never be given a maxDuration (after() runs inside an arbitrary route), so dispatch-kick.ts now states why it does not need one: KICK_BATCH_SIZE x REQUEST_TIMEOUT_MS is 50 s, so the dispatcher's budget check never fires there. 2. The sweep charged an attempt but re-armed at p_now, i.e. no backoff, while the normal failure path waits RETRY_BACKOFF_SECONDS. A row that kept getting stranded (deploy, instance recycle, any cycle that outlives its invocation) was re-claimable on the next per-minute tick and could burn all 8 attempts in roughly 20 minutes, landing in the terminal, immutable 'dead' state without its receiver ever being contacted. Pre-fix that loop was infinite but harmless, so this was a net-new way to lose a delivery. recover_stuck_webhook_deliveries now takes p_backoff int[] (RETRY_BACKOFF_SECONDS, still single-sourced in TS) and sets next_attempt_at with the same clamped index lookup markFailedForRetry uses, so a stall costs an attempt AND the same wait a 500 costs. A non-positive or empty schedule is rejected rather than silently degrading to p_now. The migration has not been applied to any deployed environment, so it is amended in place rather than superseded; it drops the old 3-argument signature so no ambiguous overload can survive in a dev or CI database. Also from the review: - stuckInFlightAfterMs(batchSize) was dead code whose Math.min clamp made every input return 120_000, so the documented DEFAULT_BATCH_SIZE floor never fired and the test that pinned it (stuckInFlightAfterMs(5) === stuckInFlightAfterMs(50)) was a tautology. It is now the plain constant STUCK_IN_FLIGHT_AFTER_MS with a comment that credits the budget, and the test drives the window through dispatchDueDeliveries at batch sizes 5, 50 and 500, which fails if the window ever becomes batch-derived again. - The sweep's outcome reaches the operator: recovered / recoveredDead are on DispatchSummary and in the cron's structured log, so a tick that takes deliveries terminal is visible without grepping helper-level warn lines. - releaseUnattempted no longer writes 'failed' onto a never-attempted row. claim_due_webhook_deliveries does not return the pre-claim status, but it does return attempts, and every path that writes 'failed' also writes attempts >= 1, so attempts = 0 identifies a row that was 'pending' and it is restored as such. webhook_deliveries is customer-visible behandlingshistorik; a delivery that was claimed and handed back without a single POST must not read as a failure there. The two deferred hygiene items (no retention path for webhook_deliveries, and the sweep still being an unbounded tenant-global UPDATE) are reported as a comment on #1257 and noted in the migration. Fixes #1257 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
136 lines
6.2 KiB
TypeScript
136 lines
6.2 KiB
TypeScript
/**
|
|
* Emit-triggered webhook dispatch kick.
|
|
*
|
|
* Without this, the floor on webhook delivery latency is the per-minute cron
|
|
* at /api/webhooks/dispatch/cron: a delivery enqueued one second after a tick
|
|
* waits the remaining 59 before anyone looks at it. External consumers had no
|
|
* way around that except polling /api/events, which the 100 rpm per-key limit
|
|
* makes expensive and which still cannot beat the tick interval (issue #1201).
|
|
*
|
|
* So: once fanOutToWebhooks has inserted delivery rows, schedule one dispatch
|
|
* cycle immediately. The cron is unchanged and remains the retry and sweep
|
|
* path; this only moves the FIRST attempt forward.
|
|
*
|
|
* Three properties matter and are all load-bearing:
|
|
*
|
|
* 1. NEVER awaited by the emitter. eventBus.emit() is awaited at ~99 call
|
|
* sites, including journal_entry.committed, and dispatchDueDeliveries POSTs
|
|
* to receivers with a 10 s timeout each. Awaiting it here would put a
|
|
* stranger's slow HTTP endpoint on the critical path of committing a
|
|
* verifikat. kickWebhookDispatch() is synchronous and returns immediately.
|
|
*
|
|
* 2. Coalesced per instance. A bulk operation emits once per row, and each
|
|
* emission would otherwise schedule its own cycle: 100 bulk-booked
|
|
* transactions would mean 100 claim round trips. While one kick is
|
|
* outstanding, further kicks are dropped; the rows they enqueued are picked
|
|
* up by that kick, the next one, or the cron.
|
|
*
|
|
* 3. Never throws. It runs inside an event-bus subscriber, where an
|
|
* unhandled rejection would surface as a failed emit on a route that has
|
|
* already done its real work.
|
|
*
|
|
* claim_due_webhook_deliveries claims with FOR UPDATE SKIP LOCKED and flips
|
|
* rows to in_flight in the same statement, so a kick and the cron never claim
|
|
* the same row at the same moment. That is a claim-time guarantee only, and it
|
|
* is worth being precise about what it does NOT buy: the RPC autocommits, so
|
|
* its locks are gone before any POST is issued, and from then on ownership is
|
|
* just status='in_flight'. Delivery therefore stays at-least-once, exactly as
|
|
* the public docs promise ("the same delivery id may arrive more than once ...
|
|
* idempotency is on you").
|
|
*
|
|
* The kick does not widen that. Since #1257 the stuck sweep's window is a
|
|
* constant derived from the dispatcher's cycle budget rather than from the
|
|
* caller's batch size, so it is identical for both callers and always wider
|
|
* than any live cycle: neither the cron nor this 5-row kick can re-arm a row
|
|
* the other still owns. What remains is the genuine crash case, where a
|
|
* hard-killed invocation's rows are recovered while its POSTs may still be in
|
|
* flight; the ownership check in touchInFlight drops the loser's POST before
|
|
* it is issued.
|
|
*
|
|
* One asymmetry is worth stating plainly: the cron route backs the dispatcher's
|
|
* CYCLE_BUDGET_MS with `export const maxDuration = 300`, but this path cannot.
|
|
* after() runs inside whatever arbitrary API route emitted the event, and that
|
|
* route's maxDuration is not ours to set. It does not need to be: a kick claims
|
|
* at most KICK_BATCH_SIZE (5) rows and each attempt is capped at
|
|
* REQUEST_TIMEOUT_MS (10 s), so a kick cycle cannot exceed ~50 s of attempt
|
|
* time and the dispatcher's 110 s budget check never fires here. The bound on
|
|
* the kick is its batch size, not a clock.
|
|
*/
|
|
|
|
import { after } from 'next/server'
|
|
import { dispatchDueDeliveries } from './dispatcher'
|
|
import { createServiceClientNoCookies } from '@/lib/auth/api-keys'
|
|
import { createLogger } from '@/lib/logger'
|
|
|
|
const log = createLogger('webhooks/dispatch-kick')
|
|
|
|
/**
|
|
* Deliveries claimed per emit-triggered cycle. Deliberately far below the
|
|
* cron's 50: this work runs after a user-facing request on the same function
|
|
* instance, and each delivery can burn up to REQUEST_TIMEOUT_MS. Five covers
|
|
* the realistic fanout (one or two receivers for the event that just fired)
|
|
* without letting a backlog turn one request's tail into a minute of work.
|
|
* Anything beyond it is the cron's job, which is what the cron is for.
|
|
*/
|
|
export const KICK_BATCH_SIZE = 5
|
|
|
|
/**
|
|
* True while a kick is scheduled but has not started running. Module scope,
|
|
* so it is per function instance rather than global: two concurrent instances
|
|
* each get one in-flight kick, which is fine (SKIP LOCKED makes the claims
|
|
* disjoint).
|
|
*/
|
|
let kickPending = false
|
|
|
|
/** Test seam: swap the dispatch implementation without a live Supabase. */
|
|
type DispatchFn = typeof dispatchDueDeliveries
|
|
|
|
/**
|
|
* Schedule one webhook dispatch cycle to run after the current response.
|
|
* Returns immediately; the caller must not await the delivery work.
|
|
*/
|
|
export function kickWebhookDispatch(dispatchImpl?: DispatchFn): void {
|
|
if (kickPending) return
|
|
kickPending = true
|
|
|
|
const dispatch = dispatchImpl ?? dispatchDueDeliveries
|
|
|
|
const run = async (): Promise<void> => {
|
|
// Cleared first, not last: a kick scheduled while this cycle is running
|
|
// targets rows this cycle may already have passed, so it deserves its own
|
|
// slot. Clearing here also means a cycle that dies mid-flight cannot
|
|
// wedge the flag on for the life of the instance.
|
|
kickPending = false
|
|
try {
|
|
await dispatch({
|
|
supabase: createServiceClientNoCookies(),
|
|
batchSize: KICK_BATCH_SIZE,
|
|
})
|
|
} catch (err) {
|
|
// The cron will retry every one of these rows. A failed kick is a
|
|
// latency regression, never a lost delivery, so warn rather than error.
|
|
log.warn('emit-triggered webhook dispatch failed; cron will retry', {
|
|
error: err instanceof Error ? err.message : String(err),
|
|
})
|
|
}
|
|
}
|
|
|
|
try {
|
|
// Keeps the serverless instance alive past the response so the POSTs
|
|
// actually complete. Same pattern as the enable-banking callback.
|
|
after(() => run())
|
|
} catch {
|
|
// Outside a request scope (unit tests, scripts, a plain node server):
|
|
// run it as a floating promise instead. Deferred rather than called
|
|
// inline, so this path coalesces a synchronous burst exactly like the
|
|
// after() path does: calling run() here would clear kickPending before
|
|
// the next kick in the same tick ever sees it. run() never rejects.
|
|
queueMicrotask(() => void run())
|
|
}
|
|
}
|
|
|
|
/** Test-only: reset the coalescing flag between cases. */
|
|
export function resetKickStateForTests(): void {
|
|
kickPending = false
|
|
}
|