fix(pending-ops): recovery sweep for operations stuck in committing (#843) (#1108)

The commit dispatcher claims an op with an atomic pending -> committing
CAS; if the process dies after side-effects post but before the terminal
committed write (or that write fails, the PR #841 log line), the row sat
in status='committing' forever: the expire cron only sweeps 'pending'.

Add lib/pending-operations/recover-stuck-committing.ts, invoked from the
existing daily expire cron (no new vercel.json entry):

- Only rows whose updated_at (the claim timestamp: the CAS bumps it via
  the update_updated_at_column trigger) is older than 15 minutes, well
  past the 300s Vercel function ceiling, so in-flight executors are
  never raced.
- Positive evidence that side-effects posted finalizes the row to
  committed with result_data.recovered=true. Evidence exists only where
  params identify a target with an unambiguous posted state:
  categorize_transaction (is_transaction_booked RPC, skipped for
  allow_duplicate), link_transaction_journal_entry (exact tx+entry
  link), match_transaction_invoice (invoice_payments pair row).
- No evidence: terminal rejected with an explanatory result_data,
  never back to pending (re-execution could duplicate side-effects
  that posted without a trace). Reason 'stuck_committing' is distinct
  from 'expired' so the UI badge never claims these rows.
- Every terminal write is CAS-guarded on status='committing'; probe
  errors skip the row for the next run.
- One structured 'pending_op_recovery' warn per row (count by outcome);
  runbook comment added next to the #841 finalize-failure log line.

Tests: unit coverage for the decision logic and cron wiring (401, sweep
invoked, failure isolation), plus a pg-real test proving row selection,
the trustworthy updated_at anchor, committing -> terminal transitions
through the real immutability/input-frozen triggers, and the
is_transaction_booked evidence substrate.

Fixes #843

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

lib/pending-operations/commit.ts

@@ -4708,11 +4708,17 @@ async function commitPendingOperationInner(
 
   if (finalizeError) {
     // The executor's side-effects already committed (and are immutable); only
-    // the terminal status write failed. Without surfacing this, the row would
-    // sit in 'committing' indefinitely: the expire sweep only targets
-    // 'pending' ops, so nothing would ever reconcile it. Log loudly with the
-    // ids needed to finalize manually; the response still reports success
-    // because the actual work is done.
+    // the terminal status write failed. Log loudly with the ids needed to
+    // finalize manually; the response still reports success because the
+    // actual work is done.
+    //
+    // Runbook (#843): rows left in 'committing' by this failure are picked up
+    // by the daily recovery sweep in
+    // lib/pending-operations/recover-stuck-committing.ts (runs from the
+    // expire cron, threshold 15 min). Grep logs for 'pending_op_recovery' to
+    // see per-row outcomes: 'committed' when posted side-effects were
+    // verified, 'rejected' when no trace was detectable (nothing re-executes
+    // either way).
     log.error('failed to finalize pending_operation to committed (left in committing)', finalizeError, {
       pendingOperationId: pendingOp.id,
       operationType: pendingOp.operation_type,