Files
accounted/lib/webhooks/__tests__/dispatch-kick.test.ts
T
Jakob Wennberg 1a5d205bd6 fix(webhooks): derive the stuck-in_flight window from the cycle bound and charge the stall an attempt (#1311)
* fix(webhooks): derive the stuck-in_flight window from the cycle bound and charge the stall an attempt

recoverStuckInFlight re-armed any in_flight row older than 2x
REQUEST_TIMEOUT_MS (20 s), but a cron cycle claims 50 rows and attempts
them serially, stamping updated_at once at claim time. From row 3 onward
every row was past the threshold before its own attempt started, so each
cycle recovered and re-claimed the rows the previous cycle was still
working through: duplicate POSTs of the same X-Gnubok-Delivery, and a
terminal status decided by a race whose loser was swallowed by
enforce_webhook_delivery_immutability as a log.warn.

Both halves of #1257 are fixed:

1. The window is derived, not guessed. The attempt loop is now bounded by
   an explicit CYCLE_BUDGET_MS (120 s) instead of relying on the platform
   to kill it, and the sweep window is that bound plus one receiver
   timeout plus slack (160 s), floored at the cron's own batch size so
   the 5-row emit kick cannot re-arm rows the 50-row cron still owns.
   Each row is also re-stamped immediately before its own attempt, so a
   row's in_flight age measures the attempt rather than the claim. The
   same write doubles as an ownership check: a zero-row result means
   another cycle took the row, and the POST is dropped instead of
   duplicated.

2. The sweep charges an attempt, so MAX_ATTEMPTS is a real cap again.
   The predicate moves into a SECURITY DEFINER RPC because PostgREST can
   express neither `attempts = attempts + 1` nor the conditional flip at
   the cap, and a read-then-write loop would reopen a TOCTOU against the
   immutability trigger. A row recovered past the cap lands on exactly
   the terminal state the normal retry path produces: status 'dead',
   attempts = MAX_ATTEMPTS, error prefixed 'attempts_exhausted'. The
   trigger is neither weakened nor bypassed: the outer UPDATE keeps
   status = 'in_flight' in its own WHERE, so a row that raced to a
   terminal status fails re-evaluation under READ COMMITTED and is
   skipped rather than aborting the statement.

Rows the cycle claimed but will not reach are handed back as re-claimable
instead of being stranded in in_flight, without charging an attempt they
never made. Adds the partial index the sweep needs (idx_webhook_deliveries_due
is partial on pending/failed and structurally excludes in_flight).

No retention or pruning cron: webhook_deliveries still has no cleanup
path, which is a separate decision and stays a follow-up.

Fixes #1257

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(webhooks): back the cycle budget with maxDuration and give the stall the normal retry backoff

Review follow-up on the #1257 fix. Two of the findings were blocking and
compound each other: the fix made stranding likely and destructive at the
same time.

1. The 160 s sweep window was derived from CYCLE_BUDGET_MS, but nothing
   granted a dispatch cycle 120 s: the cron route declared no
   maxDuration. If the platform killed the invocation before the budget
   check fired, releaseUnattempted never ran and the claimed-but-
   unattempted rows stayed in in_flight carrying their claim-time
   updated_at, which is exactly the invariant the window depends on.
   The route now declares maxDuration = 300, the way the stripe
   transactions and documents verify crons pair a budget with one, and a
   route test asserts both the literal and its relation to
   CYCLE_BUDGET_MS. The kick path can never be given a maxDuration
   (after() runs inside an arbitrary route), so dispatch-kick.ts now
   states why it does not need one: KICK_BATCH_SIZE x REQUEST_TIMEOUT_MS
   is 50 s, so the dispatcher's budget check never fires there.

2. The sweep charged an attempt but re-armed at p_now, i.e. no backoff,
   while the normal failure path waits RETRY_BACKOFF_SECONDS. A row that
   kept getting stranded (deploy, instance recycle, any cycle that
   outlives its invocation) was re-claimable on the next per-minute tick
   and could burn all 8 attempts in roughly 20 minutes, landing in the
   terminal, immutable 'dead' state without its receiver ever being
   contacted. Pre-fix that loop was infinite but harmless, so this was a
   net-new way to lose a delivery. recover_stuck_webhook_deliveries now
   takes p_backoff int[] (RETRY_BACKOFF_SECONDS, still single-sourced in
   TS) and sets next_attempt_at with the same clamped index lookup
   markFailedForRetry uses, so a stall costs an attempt AND the same wait
   a 500 costs. A non-positive or empty schedule is rejected rather than
   silently degrading to p_now. The migration has not been applied to any
   deployed environment, so it is amended in place rather than superseded;
   it drops the old 3-argument signature so no ambiguous overload can
   survive in a dev or CI database.

Also from the review:

- stuckInFlightAfterMs(batchSize) was dead code whose Math.min clamp made
  every input return 120_000, so the documented DEFAULT_BATCH_SIZE floor
  never fired and the test that pinned it (stuckInFlightAfterMs(5) ===
  stuckInFlightAfterMs(50)) was a tautology. It is now the plain constant
  STUCK_IN_FLIGHT_AFTER_MS with a comment that credits the budget, and
  the test drives the window through dispatchDueDeliveries at batch sizes
  5, 50 and 500, which fails if the window ever becomes batch-derived
  again.
- The sweep's outcome reaches the operator: recovered / recoveredDead are
  on DispatchSummary and in the cron's structured log, so a tick that
  takes deliveries terminal is visible without grepping helper-level warn
  lines.
- releaseUnattempted no longer writes 'failed' onto a never-attempted
  row. claim_due_webhook_deliveries does not return the pre-claim status,
  but it does return attempts, and every path that writes 'failed' also
  writes attempts >= 1, so attempts = 0 identifies a row that was
  'pending' and it is restored as such. webhook_deliveries is
  customer-visible behandlingshistorik; a delivery that was claimed and
  handed back without a single POST must not read as a failure there.

The two deferred hygiene items (no retention path for webhook_deliveries,
and the sweep still being an unbounded tenant-global UPDATE) are reported
as a comment on #1257 and noted in the migration.

Fixes #1257

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 18:51:41 +02:00

127 lines
4.0 KiB
TypeScript

import { describe, it, expect, vi, beforeEach } from 'vitest'
vi.mock('@/lib/auth/api-keys', () => ({
createServiceClientNoCookies: vi.fn(() => ({ __client: true })),
}))
// `after` is a no-op outside a request scope in these tests; the module falls
// back to a floating promise, which is the path a plain node server takes too.
vi.mock('next/server', () => ({
after: () => {
throw new Error('after() called outside a request scope')
},
}))
import { kickWebhookDispatch, resetKickStateForTests, KICK_BATCH_SIZE } from '../dispatch-kick'
/**
* A zero DispatchSummary. Spread rather than repeated so adding a counter to
* the summary does not mean re-editing every case in this file; these tests
* are about scheduling and coalescing, never about the counts.
*/
const EMPTY_SUMMARY = {
picked: 0,
delivered: 0,
failed: 0,
dead: 0,
skipped: 0,
released: 0,
recovered: 0,
recoveredDead: 0,
}
/** Resolves after the microtask queue drains, so floating work has run. */
const flush = () => new Promise((resolve) => setTimeout(resolve, 0))
beforeEach(() => {
vi.clearAllMocks()
resetKickStateForTests()
})
describe('kickWebhookDispatch', () => {
it('runs one dispatch cycle with the small kick batch size', async () => {
const dispatch = vi.fn().mockResolvedValue({ ...EMPTY_SUMMARY, picked: 1, delivered: 1 })
kickWebhookDispatch(dispatch)
await flush()
expect(dispatch).toHaveBeenCalledTimes(1)
expect(dispatch.mock.calls[0][0]).toMatchObject({ batchSize: KICK_BATCH_SIZE })
})
it('claims far fewer rows than the cron, so a backlog cannot own a request tail', () => {
// Each delivery can burn the full 10 s receiver timeout; the cron's 50 is
// fine on a dedicated invocation, not on the tail of a user request.
expect(KICK_BATCH_SIZE).toBeLessThan(50)
})
it('returns synchronously: the emitter never waits on receiver HTTP', async () => {
let settled = false
const dispatch = vi.fn(
() =>
new Promise((resolve) => {
setTimeout(() => {
settled = true
resolve({ ...EMPTY_SUMMARY })
}, 20)
}),
)
const returned = kickWebhookDispatch(dispatch as never)
// eventBus.emit awaits its subscribers, so a kick that blocked here would
// put a stranger's slow endpoint on the critical path of committing a
// verifikat.
expect(returned).toBeUndefined()
expect(settled).toBe(false)
await new Promise((resolve) => setTimeout(resolve, 40))
expect(settled).toBe(true)
})
it('coalesces a burst of kicks into a single cycle', async () => {
// A bulk booking emits once per row. Without coalescing, 100 rows would
// mean 100 claim round trips against the same handful of due deliveries.
const dispatch = vi.fn().mockResolvedValue({ ...EMPTY_SUMMARY })
for (let i = 0; i < 100; i++) kickWebhookDispatch(dispatch)
await flush()
expect(dispatch).toHaveBeenCalledTimes(1)
})
it('accepts a new kick once the previous cycle has started', async () => {
const dispatch = vi.fn().mockResolvedValue({ ...EMPTY_SUMMARY })
kickWebhookDispatch(dispatch)
await flush()
kickWebhookDispatch(dispatch)
await flush()
expect(dispatch).toHaveBeenCalledTimes(2)
})
it('swallows a dispatch failure: a failed kick is latency, not a lost delivery', async () => {
const dispatch = vi.fn().mockRejectedValue(new Error('receiver unreachable'))
expect(() => kickWebhookDispatch(dispatch)).not.toThrow()
await flush()
expect(dispatch).toHaveBeenCalledTimes(1)
})
it('does not wedge the coalescing flag when a cycle fails', async () => {
// The cron still covers every row, but a stuck flag would silently
// disable the fast path for the life of the function instance.
const failing = vi.fn().mockRejectedValue(new Error('boom'))
kickWebhookDispatch(failing)
await flush()
const ok = vi.fn().mockResolvedValue({ ...EMPTY_SUMMARY })
kickWebhookDispatch(ok)
await flush()
expect(ok).toHaveBeenCalledTimes(1)
})
})