Den 2026-11-15 tappade accounted-cron 1 497 av 10 455 körningar på ett dygn (14,3 %) när pids_limit 30 fick fork/exec /bin/sh att misslyckas. supercronic hoppade tyst över tickarna. Containern var Up hela tiden, appen svarade 200 på allt, och enda spåret var cron-containerns egen logg. Orsaken är fixad i #43; det här gör att ett tappat jobb SYNS nästa gång. Tre luckor stängs: 1. Ingen körningshistorik — with-cron-context loggade men skrev inget beständigt, så en körning som aldrig startade lämnade inget spår. 2. Cron-tjänstens healthcheck var `ps ax | grep supercronic`, som bara blir röd när processen dör, inte när scheman slutar köra. 3. Ingen hade kodifierat vad "för sen" betyder per schema. Implementering: - lib/cron/cron-interval.ts: tolkare för 5-fälters cron (det enda vercel.json använder) som räknar ut VÄRSTA gapet i sekunder över ett 12-årsfönster (behövs för 29 februari), inte medelgapet. Okänd syntax kastar hellre än gissar. - scripts/generate-crontabs.ts skriver nu även lib/cron/schedules.generated.ts (path + schedule + maxIntervalSeconds) från samma vercel.json-array som crontaben, så en schemalagd path som kontrollen inte känner till inte kan uppstå. Drift-testet jämför filen byte för byte. - supabase/migrations/20261115190000_cron_heartbeats.sql: en rad per path (declared_at, last_started_at, last_completed_at, last_status, last_http_status, last_duration_ms). RLS på, noll policies: instansglobal schemaläggning är inte tenantdata. - lib/cron/heartbeat-evaluation.ts: ren utvärdering — ok / pending (aldrig körd men deklarerad inom sin egen kadens) / stale. Tröskel 2 x kadens + 120 s. Mätt från last_started_at, inte completion: en hängd körning avslutas aldrig och supercronic skjuter då upp varje senare tick, så ett completion-baserat mått hade sett levande ut. - lib/cron/heartbeat-store.ts: Supabase-IO. Skrivningar är best effort (kastar aldrig — en mätningsskrivning får inte fälla jobbet den mäter); läsningar kastar så endpointen kan säga "kan inte verifiera" istället för att ljuga grönt. No-op utan Supabase-env. - app/api/cron/health/route.ts: 200 när alla scheman ligger inom sina fönster, 503 vid stale eller oläsbart lager. Deklarerar okända paths först (nytt jobb får respit istället för omedelbart rött). Autentiseras med befintlig cron-hemlighet, report:false så en felkopplad probe inte spammar auth-larm. - lib/api/with-cron-context.ts: skriver heartbeat per körning (path ur request-URL:en) med status och duration. 401 räknas inte som körning. - docker-compose.yml: cron-healthcheck byter från supercronic-grep till att anropa /api/cron/health med `$${CRON_SECRET}`/`$${APP_URL}` (dubbelt $ krävs, annars interpolerar compose på värden), start_period 120 s. Migrationen är redan applicerad på accounted-sb-db (additiv, 0 rader) så endpointen fungerar direkt vid deploy. Ordning vid utrullning: deploya appen först, recreata cron-containern sedan — den gamla imagen saknar routen, och healthchecken failar då korrekt stängt (verifierat i containern: curl -f ger 404/rc=22 mot icke-deployad route). Verifiering: 99 nya tester (kadens 60/120/300/3600/7200/86400/604800 s, 29 februari, cron-regeln OR; utvärdering ok/pending/stale/hängd körning; endpoint 200/503/401/unverifiable; wiring inklusive 401 utan heartbeat och kastat heartbeat-fel som inte fäller körningen; drift av genererad fil). Hela sviten 1687 filer / 22 400 tester grön; check:types 533/533; check:lint 0 fel; check:guards passed. Revert: git revert <sha> (kod + migration + compose i en commit). Migrationen kan lämnas kvar om bara koden rullas tillbaka.
209 lines
7.8 KiB
TypeScript
209 lines
7.8 KiB
TypeScript
/**
|
|
* Cron expression → the longest gap between two scheduled runs.
|
|
*
|
|
* Why this exists: the 2026-11-15 accounted incident. `accounted-cron` lost
|
|
* 1 497 of 10 455 runs in a day (14,3 %) because `pids_limit: 30` made
|
|
* `fork/exec /bin/sh` fail under the per-minute schedules, and supercronic
|
|
* simply skipped those ticks. Nothing in the product noticed: the container
|
|
* stayed "Up", every endpoint still answered, and the only trace was a line in
|
|
* the cron container's own log. A schedule whose runs stop arriving is the
|
|
* worst failure mode in this product — nothing looks broken, the work just
|
|
* never happens — so `cron_heartbeats` records every run and
|
|
* app/api/cron/health turns "no run arrived within the expected window" into a
|
|
* 503 the container healthcheck can fail on.
|
|
*
|
|
* That check needs to know, per path, how long a gap is normal. That is this
|
|
* module: a small, dependency-free interpreter for the 5-field cron syntax
|
|
* `vercel.json` is restricted to, plus a brute-force scan of a 12-year window
|
|
* to get the WORST-CASE gap rather than an average. Worst case is the correct
|
|
* threshold input: a job that runs at 00:00 daily must be allowed 24 h, not
|
|
* the 12 h its mean gap would suggest.
|
|
*
|
|
* The scan is exact for every expression the product uses, including the
|
|
* calendar-sensitive ones (`0 0 2 1 *` once a year, `0 0 29 2 *` only in leap
|
|
* years). It is not a general cron library: no seconds field, no `@daily`
|
|
* aliases, no `L`/`W`/`#` extensions. Anything outside the supported subset
|
|
* throws instead of guessing, and the generator that consumes it runs in CI, so
|
|
* an unsupported expression fails the build rather than silently producing a
|
|
* wrong alert threshold.
|
|
*/
|
|
|
|
const MINUTE_MS = 60_000
|
|
const HOUR_MS = 60 * MINUTE_MS
|
|
const DAY_MS = 24 * HOUR_MS
|
|
|
|
/**
|
|
* Scan window: 2001-01-01 + 12 years. Twelve years is what it takes to see
|
|
* three occurrences of 29 February (2004, 2008, 2012), which is the rarest
|
|
* yearly form a cron expression can express.
|
|
*/
|
|
const SCAN_START_MS = Date.UTC(2001, 0, 1)
|
|
const SCAN_DAYS = 12 * 366
|
|
|
|
/**
|
|
* Upper bound on matched timestamps per expression. A per-minute job matches
|
|
* 1 440 times a day; 20 000 is two weeks of it, far more than needed to see the
|
|
* 60-second gap, and it keeps the generator fast when the whole vercel.json is
|
|
* scanned.
|
|
*/
|
|
const MAX_MATCHES = 20_000
|
|
|
|
export interface CronSpec {
|
|
readonly minutes: readonly number[]
|
|
readonly hours: readonly number[]
|
|
readonly daysOfMonth: readonly number[]
|
|
readonly months: readonly number[]
|
|
readonly daysOfWeek: readonly number[]
|
|
/**
|
|
* Cron's day rule: when BOTH day-of-month and day-of-week are restricted the
|
|
* expression matches if EITHER does; when only one is restricted only that
|
|
* one applies. A step over the whole range (star followed by a step) counts
|
|
* as unrestricted, as in every cron implementation, because the field still
|
|
* starts with a star.
|
|
*/
|
|
readonly dayOfMonthRestricted: boolean
|
|
readonly dayOfWeekRestricted: boolean
|
|
}
|
|
|
|
function parseField(
|
|
field: string,
|
|
min: number,
|
|
max: number,
|
|
name: string,
|
|
): { values: number[]; restricted: boolean } {
|
|
const trimmed = field.trim()
|
|
if (trimmed.length === 0) throw new Error(`cron: empty ${name} field`)
|
|
|
|
const seen = new Set<number>()
|
|
for (const part of trimmed.split(',')) {
|
|
const [rangePart, stepPart, ...rest] = part.split('/')
|
|
if (rest.length > 0) throw new Error(`cron: bad step in ${name} field: ${part}`)
|
|
|
|
const step = stepPart === undefined ? 1 : Number(stepPart)
|
|
if (!Number.isInteger(step) || step < 1) {
|
|
throw new Error(`cron: bad step in ${name} field: ${part}`)
|
|
}
|
|
|
|
let lo: number
|
|
let hi: number
|
|
if (rangePart === '*') {
|
|
lo = min
|
|
hi = max
|
|
} else if (rangePart.includes('-')) {
|
|
const [a, b, ...tail] = rangePart.split('-')
|
|
if (tail.length > 0) throw new Error(`cron: bad range in ${name} field: ${part}`)
|
|
lo = Number(a)
|
|
hi = Number(b)
|
|
} else {
|
|
lo = Number(rangePart)
|
|
// A bare value with a step (`5/10`) means "from 5 to max, every 10th".
|
|
hi = stepPart === undefined ? lo : max
|
|
}
|
|
|
|
if (!Number.isInteger(lo) || !Number.isInteger(hi)) {
|
|
throw new Error(`cron: non-numeric value in ${name} field: ${part}`)
|
|
}
|
|
if (lo < min || hi > max || lo > hi) {
|
|
throw new Error(`cron: ${name} field out of range ${min}-${max}: ${part}`)
|
|
}
|
|
|
|
for (let v = lo; v <= hi; v += step) seen.add(v)
|
|
}
|
|
|
|
return { values: [...seen].sort((a, b) => a - b), restricted: !trimmed.startsWith('*') }
|
|
}
|
|
|
|
/** Parse a standard 5-field cron expression. Throws on anything unsupported. */
|
|
export function parseCronExpression(expression: string): CronSpec {
|
|
const fields = expression.trim().split(/\s+/)
|
|
if (fields.length !== 5) {
|
|
throw new Error(`cron: expected 5 fields, got ${fields.length}: ${JSON.stringify(expression)}`)
|
|
}
|
|
|
|
const [minute, hour, dayOfMonth, month, dayOfWeek] = fields
|
|
const minutes = parseField(minute, 0, 59, 'minute')
|
|
const hours = parseField(hour, 0, 23, 'hour')
|
|
const dom = parseField(dayOfMonth, 1, 31, 'day-of-month')
|
|
const months = parseField(month, 1, 12, 'month')
|
|
// Cron allows both 0 and 7 for Sunday; normalise 7 to 0.
|
|
const dow = parseField(dayOfWeek, 0, 7, 'day-of-week')
|
|
|
|
return {
|
|
minutes: minutes.values,
|
|
hours: hours.values,
|
|
daysOfMonth: dom.values,
|
|
months: months.values,
|
|
daysOfWeek: [...new Set(dow.values.map((d) => (d === 7 ? 0 : d)))].sort((a, b) => a - b),
|
|
dayOfMonthRestricted: dom.restricted,
|
|
dayOfWeekRestricted: dow.restricted,
|
|
}
|
|
}
|
|
|
|
function matchesDay(spec: CronSpec, dayStartMs: number): boolean {
|
|
const date = new Date(dayStartMs)
|
|
if (!spec.months.includes(date.getUTCMonth() + 1)) return false
|
|
|
|
const domMatch = spec.daysOfMonth.includes(date.getUTCDate())
|
|
const dowMatch = spec.daysOfWeek.includes(date.getUTCDay())
|
|
|
|
if (spec.dayOfMonthRestricted && spec.dayOfWeekRestricted) return domMatch || dowMatch
|
|
if (spec.dayOfMonthRestricted) return domMatch
|
|
if (spec.dayOfWeekRestricted) return dowMatch
|
|
return true
|
|
}
|
|
|
|
/**
|
|
* The longest interval, in seconds, between two consecutive runs of
|
|
* `expression` over a 12-year window.
|
|
*
|
|
* Throws when the expression never matches twice in the window — that is not a
|
|
* schedule, and silently returning 0 would make the health check permanently
|
|
* red or permanently green depending on which way the caller rounds.
|
|
*/
|
|
export function maxIntervalSeconds(expression: string): number {
|
|
const spec = parseCronExpression(expression)
|
|
|
|
// Every minute: the answer is exactly one minute and the scan is pointless.
|
|
if (spec.minutes.length * spec.hours.length === 24 * 60) return 60
|
|
|
|
let previous: number | null = null
|
|
let longest = 0
|
|
let matches = 0
|
|
|
|
for (let day = 0; day < SCAN_DAYS && matches < MAX_MATCHES; day++) {
|
|
const dayStart = SCAN_START_MS + day * DAY_MS
|
|
if (!matchesDay(spec, dayStart)) continue
|
|
|
|
for (const hour of spec.hours) {
|
|
for (const minute of spec.minutes) {
|
|
const at = dayStart + hour * HOUR_MS + minute * MINUTE_MS
|
|
if (previous !== null) {
|
|
const gap = (at - previous) / 1000
|
|
if (gap > longest) longest = gap
|
|
}
|
|
previous = at
|
|
if (++matches >= MAX_MATCHES) break
|
|
}
|
|
if (matches >= MAX_MATCHES) break
|
|
}
|
|
}
|
|
|
|
if (matches < 2) {
|
|
throw new Error(`cron: ${JSON.stringify(expression)} never matches twice in the scan window`)
|
|
}
|
|
|
|
return longest
|
|
}
|
|
|
|
/**
|
|
* Human-readable worst-case gap, for the generated file and for operators
|
|
* reading a 503 body. Deliberately coarse: it is a threshold input, not a
|
|
* display string that needs to be exact.
|
|
*/
|
|
export function describeGap(seconds: number): string {
|
|
if (seconds < 120) return `${seconds} s`
|
|
if (seconds < 2 * 3600) return `${Math.round(seconds / 60)} min`
|
|
if (seconds < 2 * 86_400) return `${Math.round(seconds / 3600)} h`
|
|
return `${Math.round(seconds / 86_400)} dygn`
|
|
}
|