Files
accounted/lib/cron/cron-interval.ts
T
Claude Code c3a2ace846
masterplan-lock / check (pull_request) Successful in 8s
security / scan (pull_request) Failing after 2m43s
feat(cron): missad-tick-detektion — heartbeat per schema, 503 när ett jobb slutar komma
Den 2026-11-15 tappade accounted-cron 1 497 av 10 455 körningar på ett dygn
(14,3 %) när pids_limit 30 fick fork/exec /bin/sh att misslyckas. supercronic
hoppade tyst över tickarna. Containern var Up hela tiden, appen svarade 200 på
allt, och enda spåret var cron-containerns egen logg. Orsaken är fixad i #43;
det här gör att ett tappat jobb SYNS nästa gång.

Tre luckor stängs:
1. Ingen körningshistorik — with-cron-context loggade men skrev inget
   beständigt, så en körning som aldrig startade lämnade inget spår.
2. Cron-tjänstens healthcheck var `ps ax | grep supercronic`, som bara blir röd
   när processen dör, inte när scheman slutar köra.
3. Ingen hade kodifierat vad "för sen" betyder per schema.

Implementering:
- lib/cron/cron-interval.ts: tolkare för 5-fälters cron (det enda vercel.json
  använder) som räknar ut VÄRSTA gapet i sekunder över ett 12-årsfönster
  (behövs för 29 februari), inte medelgapet. Okänd syntax kastar hellre än
  gissar.
- scripts/generate-crontabs.ts skriver nu även lib/cron/schedules.generated.ts
  (path + schedule + maxIntervalSeconds) från samma vercel.json-array som
  crontaben, så en schemalagd path som kontrollen inte känner till inte kan
  uppstå. Drift-testet jämför filen byte för byte.
- supabase/migrations/20261115190000_cron_heartbeats.sql: en rad per path
  (declared_at, last_started_at, last_completed_at, last_status,
  last_http_status, last_duration_ms). RLS på, noll policies: instansglobal
  schemaläggning är inte tenantdata.
- lib/cron/heartbeat-evaluation.ts: ren utvärdering — ok / pending (aldrig körd
  men deklarerad inom sin egen kadens) / stale. Tröskel 2 x kadens + 120 s.
  Mätt från last_started_at, inte completion: en hängd körning avslutas aldrig
  och supercronic skjuter då upp varje senare tick, så ett completion-baserat
  mått hade sett levande ut.
- lib/cron/heartbeat-store.ts: Supabase-IO. Skrivningar är best effort (kastar
  aldrig — en mätningsskrivning får inte fälla jobbet den mäter); läsningar
  kastar så endpointen kan säga "kan inte verifiera" istället för att ljuga
  grönt. No-op utan Supabase-env.
- app/api/cron/health/route.ts: 200 när alla scheman ligger inom sina fönster,
  503 vid stale eller oläsbart lager. Deklarerar okända paths först (nytt jobb
  får respit istället för omedelbart rött). Autentiseras med befintlig
  cron-hemlighet, report:false så en felkopplad probe inte spammar auth-larm.
- lib/api/with-cron-context.ts: skriver heartbeat per körning (path ur
  request-URL:en) med status och duration. 401 räknas inte som körning.
- docker-compose.yml: cron-healthcheck byter från supercronic-grep till att
  anropa /api/cron/health med `$${CRON_SECRET}`/`$${APP_URL}` (dubbelt $ krävs,
  annars interpolerar compose på värden), start_period 120 s.

Migrationen är redan applicerad på accounted-sb-db (additiv, 0 rader) så
endpointen fungerar direkt vid deploy. Ordning vid utrullning: deploya appen
först, recreata cron-containern sedan — den gamla imagen saknar routen, och
healthchecken failar då korrekt stängt (verifierat i containern: curl -f ger
404/rc=22 mot icke-deployad route).

Verifiering: 99 nya tester (kadens 60/120/300/3600/7200/86400/604800 s,
29 februari, cron-regeln OR; utvärdering ok/pending/stale/hängd körning;
endpoint 200/503/401/unverifiable; wiring inklusive 401 utan heartbeat och
kastat heartbeat-fel som inte fäller körningen; drift av genererad fil).
Hela sviten 1687 filer / 22 400 tester grön; check:types 533/533; check:lint
0 fel; check:guards passed.

Revert: git revert <sha> (kod + migration + compose i en commit). Migrationen
kan lämnas kvar om bara koden rullas tillbaka.
2026-11-15 19:56:45 +01:00

209 lines
7.8 KiB
TypeScript

/**
* Cron expression → the longest gap between two scheduled runs.
*
* Why this exists: the 2026-11-15 accounted incident. `accounted-cron` lost
* 1 497 of 10 455 runs in a day (14,3 %) because `pids_limit: 30` made
* `fork/exec /bin/sh` fail under the per-minute schedules, and supercronic
* simply skipped those ticks. Nothing in the product noticed: the container
* stayed "Up", every endpoint still answered, and the only trace was a line in
* the cron container's own log. A schedule whose runs stop arriving is the
* worst failure mode in this product — nothing looks broken, the work just
* never happens — so `cron_heartbeats` records every run and
* app/api/cron/health turns "no run arrived within the expected window" into a
* 503 the container healthcheck can fail on.
*
* That check needs to know, per path, how long a gap is normal. That is this
* module: a small, dependency-free interpreter for the 5-field cron syntax
* `vercel.json` is restricted to, plus a brute-force scan of a 12-year window
* to get the WORST-CASE gap rather than an average. Worst case is the correct
* threshold input: a job that runs at 00:00 daily must be allowed 24 h, not
* the 12 h its mean gap would suggest.
*
* The scan is exact for every expression the product uses, including the
* calendar-sensitive ones (`0 0 2 1 *` once a year, `0 0 29 2 *` only in leap
* years). It is not a general cron library: no seconds field, no `@daily`
* aliases, no `L`/`W`/`#` extensions. Anything outside the supported subset
* throws instead of guessing, and the generator that consumes it runs in CI, so
* an unsupported expression fails the build rather than silently producing a
* wrong alert threshold.
*/
const MINUTE_MS = 60_000
const HOUR_MS = 60 * MINUTE_MS
const DAY_MS = 24 * HOUR_MS
/**
* Scan window: 2001-01-01 + 12 years. Twelve years is what it takes to see
* three occurrences of 29 February (2004, 2008, 2012), which is the rarest
* yearly form a cron expression can express.
*/
const SCAN_START_MS = Date.UTC(2001, 0, 1)
const SCAN_DAYS = 12 * 366
/**
* Upper bound on matched timestamps per expression. A per-minute job matches
* 1 440 times a day; 20 000 is two weeks of it, far more than needed to see the
* 60-second gap, and it keeps the generator fast when the whole vercel.json is
* scanned.
*/
const MAX_MATCHES = 20_000
export interface CronSpec {
readonly minutes: readonly number[]
readonly hours: readonly number[]
readonly daysOfMonth: readonly number[]
readonly months: readonly number[]
readonly daysOfWeek: readonly number[]
/**
* Cron's day rule: when BOTH day-of-month and day-of-week are restricted the
* expression matches if EITHER does; when only one is restricted only that
* one applies. A step over the whole range (star followed by a step) counts
* as unrestricted, as in every cron implementation, because the field still
* starts with a star.
*/
readonly dayOfMonthRestricted: boolean
readonly dayOfWeekRestricted: boolean
}
function parseField(
field: string,
min: number,
max: number,
name: string,
): { values: number[]; restricted: boolean } {
const trimmed = field.trim()
if (trimmed.length === 0) throw new Error(`cron: empty ${name} field`)
const seen = new Set<number>()
for (const part of trimmed.split(',')) {
const [rangePart, stepPart, ...rest] = part.split('/')
if (rest.length > 0) throw new Error(`cron: bad step in ${name} field: ${part}`)
const step = stepPart === undefined ? 1 : Number(stepPart)
if (!Number.isInteger(step) || step < 1) {
throw new Error(`cron: bad step in ${name} field: ${part}`)
}
let lo: number
let hi: number
if (rangePart === '*') {
lo = min
hi = max
} else if (rangePart.includes('-')) {
const [a, b, ...tail] = rangePart.split('-')
if (tail.length > 0) throw new Error(`cron: bad range in ${name} field: ${part}`)
lo = Number(a)
hi = Number(b)
} else {
lo = Number(rangePart)
// A bare value with a step (`5/10`) means "from 5 to max, every 10th".
hi = stepPart === undefined ? lo : max
}
if (!Number.isInteger(lo) || !Number.isInteger(hi)) {
throw new Error(`cron: non-numeric value in ${name} field: ${part}`)
}
if (lo < min || hi > max || lo > hi) {
throw new Error(`cron: ${name} field out of range ${min}-${max}: ${part}`)
}
for (let v = lo; v <= hi; v += step) seen.add(v)
}
return { values: [...seen].sort((a, b) => a - b), restricted: !trimmed.startsWith('*') }
}
/** Parse a standard 5-field cron expression. Throws on anything unsupported. */
export function parseCronExpression(expression: string): CronSpec {
const fields = expression.trim().split(/\s+/)
if (fields.length !== 5) {
throw new Error(`cron: expected 5 fields, got ${fields.length}: ${JSON.stringify(expression)}`)
}
const [minute, hour, dayOfMonth, month, dayOfWeek] = fields
const minutes = parseField(minute, 0, 59, 'minute')
const hours = parseField(hour, 0, 23, 'hour')
const dom = parseField(dayOfMonth, 1, 31, 'day-of-month')
const months = parseField(month, 1, 12, 'month')
// Cron allows both 0 and 7 for Sunday; normalise 7 to 0.
const dow = parseField(dayOfWeek, 0, 7, 'day-of-week')
return {
minutes: minutes.values,
hours: hours.values,
daysOfMonth: dom.values,
months: months.values,
daysOfWeek: [...new Set(dow.values.map((d) => (d === 7 ? 0 : d)))].sort((a, b) => a - b),
dayOfMonthRestricted: dom.restricted,
dayOfWeekRestricted: dow.restricted,
}
}
function matchesDay(spec: CronSpec, dayStartMs: number): boolean {
const date = new Date(dayStartMs)
if (!spec.months.includes(date.getUTCMonth() + 1)) return false
const domMatch = spec.daysOfMonth.includes(date.getUTCDate())
const dowMatch = spec.daysOfWeek.includes(date.getUTCDay())
if (spec.dayOfMonthRestricted && spec.dayOfWeekRestricted) return domMatch || dowMatch
if (spec.dayOfMonthRestricted) return domMatch
if (spec.dayOfWeekRestricted) return dowMatch
return true
}
/**
* The longest interval, in seconds, between two consecutive runs of
* `expression` over a 12-year window.
*
* Throws when the expression never matches twice in the window — that is not a
* schedule, and silently returning 0 would make the health check permanently
* red or permanently green depending on which way the caller rounds.
*/
export function maxIntervalSeconds(expression: string): number {
const spec = parseCronExpression(expression)
// Every minute: the answer is exactly one minute and the scan is pointless.
if (spec.minutes.length * spec.hours.length === 24 * 60) return 60
let previous: number | null = null
let longest = 0
let matches = 0
for (let day = 0; day < SCAN_DAYS && matches < MAX_MATCHES; day++) {
const dayStart = SCAN_START_MS + day * DAY_MS
if (!matchesDay(spec, dayStart)) continue
for (const hour of spec.hours) {
for (const minute of spec.minutes) {
const at = dayStart + hour * HOUR_MS + minute * MINUTE_MS
if (previous !== null) {
const gap = (at - previous) / 1000
if (gap > longest) longest = gap
}
previous = at
if (++matches >= MAX_MATCHES) break
}
if (matches >= MAX_MATCHES) break
}
}
if (matches < 2) {
throw new Error(`cron: ${JSON.stringify(expression)} never matches twice in the scan window`)
}
return longest
}
/**
* Human-readable worst-case gap, for the generated file and for operators
* reading a 503 body. Deliberately coarse: it is a threshold input, not a
* display string that needs to be exact.
*/
export function describeGap(seconds: number): string {
if (seconds < 120) return `${seconds} s`
if (seconds < 2 * 3600) return `${Math.round(seconds / 60)} min`
if (seconds < 2 * 86_400) return `${Math.round(seconds / 3600)} h`
return `${Math.round(seconds / 86_400)} dygn`
}