* fix(webhooks): derive the stuck-in_flight window from the cycle bound and charge the stall an attempt
recoverStuckInFlight re-armed any in_flight row older than 2x
REQUEST_TIMEOUT_MS (20 s), but a cron cycle claims 50 rows and attempts
them serially, stamping updated_at once at claim time. From row 3 onward
every row was past the threshold before its own attempt started, so each
cycle recovered and re-claimed the rows the previous cycle was still
working through: duplicate POSTs of the same X-Gnubok-Delivery, and a
terminal status decided by a race whose loser was swallowed by
enforce_webhook_delivery_immutability as a log.warn.
Both halves of #1257 are fixed:
1. The window is derived, not guessed. The attempt loop is now bounded by
an explicit CYCLE_BUDGET_MS (120 s) instead of relying on the platform
to kill it, and the sweep window is that bound plus one receiver
timeout plus slack (160 s), floored at the cron's own batch size so
the 5-row emit kick cannot re-arm rows the 50-row cron still owns.
Each row is also re-stamped immediately before its own attempt, so a
row's in_flight age measures the attempt rather than the claim. The
same write doubles as an ownership check: a zero-row result means
another cycle took the row, and the POST is dropped instead of
duplicated.
2. The sweep charges an attempt, so MAX_ATTEMPTS is a real cap again.
The predicate moves into a SECURITY DEFINER RPC because PostgREST can
express neither `attempts = attempts + 1` nor the conditional flip at
the cap, and a read-then-write loop would reopen a TOCTOU against the
immutability trigger. A row recovered past the cap lands on exactly
the terminal state the normal retry path produces: status 'dead',
attempts = MAX_ATTEMPTS, error prefixed 'attempts_exhausted'. The
trigger is neither weakened nor bypassed: the outer UPDATE keeps
status = 'in_flight' in its own WHERE, so a row that raced to a
terminal status fails re-evaluation under READ COMMITTED and is
skipped rather than aborting the statement.
Rows the cycle claimed but will not reach are handed back as re-claimable
instead of being stranded in in_flight, without charging an attempt they
never made. Adds the partial index the sweep needs (idx_webhook_deliveries_due
is partial on pending/failed and structurally excludes in_flight).
No retention or pruning cron: webhook_deliveries still has no cleanup
path, which is a separate decision and stays a follow-up.
Fixes#1257
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(webhooks): back the cycle budget with maxDuration and give the stall the normal retry backoff
Review follow-up on the #1257 fix. Two of the findings were blocking and
compound each other: the fix made stranding likely and destructive at the
same time.
1. The 160 s sweep window was derived from CYCLE_BUDGET_MS, but nothing
granted a dispatch cycle 120 s: the cron route declared no
maxDuration. If the platform killed the invocation before the budget
check fired, releaseUnattempted never ran and the claimed-but-
unattempted rows stayed in in_flight carrying their claim-time
updated_at, which is exactly the invariant the window depends on.
The route now declares maxDuration = 300, the way the stripe
transactions and documents verify crons pair a budget with one, and a
route test asserts both the literal and its relation to
CYCLE_BUDGET_MS. The kick path can never be given a maxDuration
(after() runs inside an arbitrary route), so dispatch-kick.ts now
states why it does not need one: KICK_BATCH_SIZE x REQUEST_TIMEOUT_MS
is 50 s, so the dispatcher's budget check never fires there.
2. The sweep charged an attempt but re-armed at p_now, i.e. no backoff,
while the normal failure path waits RETRY_BACKOFF_SECONDS. A row that
kept getting stranded (deploy, instance recycle, any cycle that
outlives its invocation) was re-claimable on the next per-minute tick
and could burn all 8 attempts in roughly 20 minutes, landing in the
terminal, immutable 'dead' state without its receiver ever being
contacted. Pre-fix that loop was infinite but harmless, so this was a
net-new way to lose a delivery. recover_stuck_webhook_deliveries now
takes p_backoff int[] (RETRY_BACKOFF_SECONDS, still single-sourced in
TS) and sets next_attempt_at with the same clamped index lookup
markFailedForRetry uses, so a stall costs an attempt AND the same wait
a 500 costs. A non-positive or empty schedule is rejected rather than
silently degrading to p_now. The migration has not been applied to any
deployed environment, so it is amended in place rather than superseded;
it drops the old 3-argument signature so no ambiguous overload can
survive in a dev or CI database.
Also from the review:
- stuckInFlightAfterMs(batchSize) was dead code whose Math.min clamp made
every input return 120_000, so the documented DEFAULT_BATCH_SIZE floor
never fired and the test that pinned it (stuckInFlightAfterMs(5) ===
stuckInFlightAfterMs(50)) was a tautology. It is now the plain constant
STUCK_IN_FLIGHT_AFTER_MS with a comment that credits the budget, and
the test drives the window through dispatchDueDeliveries at batch sizes
5, 50 and 500, which fails if the window ever becomes batch-derived
again.
- The sweep's outcome reaches the operator: recovered / recoveredDead are
on DispatchSummary and in the cron's structured log, so a tick that
takes deliveries terminal is visible without grepping helper-level warn
lines.
- releaseUnattempted no longer writes 'failed' onto a never-attempted
row. claim_due_webhook_deliveries does not return the pre-claim status,
but it does return attempts, and every path that writes 'failed' also
writes attempts >= 1, so attempts = 0 identifies a row that was
'pending' and it is restored as such. webhook_deliveries is
customer-visible behandlingshistorik; a delivery that was claimed and
handed back without a single POST must not read as a failure there.
The two deferred hygiene items (no retention path for webhook_deliveries,
and the sweep still being an unbounded tenant-global UPDATE) are reported
as a comment on #1257 and noted in the migration.
Fixes#1257
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Jakob Wennberg
1a5d205bd6