Files
0pt/lib/health-checks.ts
T
Claude 0062edbec3
CI (SIAX Cloud) / integration (push) Skipped
masterplan-lock / check (push) Successful in 6s
CI (SIAX Cloud) / security (push) Successful in 15s
masterplan-lock / check (pull_request) Successful in 4s
Security Scan (SIAX Cloud) / security (push) Successful in 27s
CI (SIAX Cloud) / security (pull_request) Successful in 23s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 24s
CI (SIAX Cloud) / quality (push) Successful in 45s
CI (SIAX Cloud) / integration (pull_request) Successful in 31s
CI (SIAX Cloud) / quality (pull_request) Successful in 40s
fix(readiness): /api/ready svarade 200 med en död motor i produktion
Mätt live 2026-09-25 mot produktionsdeployen (0pt.app / api.0pt.app,
container shxd6lepxuxfsjsp0ath3z1g-172514162615 på server6, uppe 17,9 h,
Docker-status "healthy" hela tiden):

    GET /api/health -> HTTP/1.1 200 OK
      {"ok":false,"service":"0pt-web","db":"ok","queue":"not_configured",
       "engine":"error","stripe":"not_configured",
       "detail":{"engine":"This operation was aborted"}}

    GET /api/ready  -> HTTP/1.1 200 OK
      {"ready":true,"service":"0pt-web","db":"ok","engine":"error",...}

ENGINE_URL pekade på http://0pt-engine:8000. Mätt inifrån containern:
`getent hosts 0pt-engine` ger ingen post, och det finns ingen sådan
container på någon av server5-11. Ett direktanrop mot motorn misslyckas
efter ~5 s (DNS). Datamotorn — 0pt:s hela produkt — var alltså BORTA, och
samtliga signaler ovanför den var gröna: Docker healthy, /api/health 200,
/api/ready 200.

`ok:false` i en 200-kropp är ingen signal. Ingenting i kedjan läser
kroppar; allt läser statuskoder och exitkoder.

isReady skilde tidigare bara på "db ok". Den skillnad som saknades är inte
konfigurerad-vs-ej-konfigurerad utan FRÅNVARANDE-vs-TRASIG:

  * not_configured / not_integrated -> fortsatt ready. 0pt är byggt för att
    kunna köra utan motor, Stripe och kö (se .env.example).
  * error -> INTE ready. Statusen "error" betyder att någon medvetet
    konfigurerade beroendet och att en riktig round trip mot det just
    misslyckades. Det är inte längre en valfri integration.

Ändrar ingenting i driften i dag: container-HEALTHCHECK pekar på
/api/health (liveness) och Dockerfilen säger uttryckligen att en
readiness-probe inte ska peka dit. Den här ändringen gör bara att den
ärliga ytan faktiskt blir ärlig, så en yttre övervakning kan använda den.

Regressionstest tillagt; verifierat att det FALLER mot den gamla isReady
(1 failed | 18 passed) och passerar mot den nya (24 passed över
health-checks/ready/health).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-25 13:34:42 +02:00

196 lines
7.9 KiB
TypeScript

// Dependency health probes shared by GET /api/health (liveness) and
// GET /api/ready (readiness). See docs/architecture/ARCHITECTURE.md
// "Hälsokontroll" and BACKLOG P0-3 for the finding this fixes: the old
// /api/health handler only ever checked *presence* of an env var and
// hardcoded `ok: true` regardless of whether anything was actually
// reachable -- "an uptime monitor watching HTTP 200 would show green for a
// dying instance". Each check below returns one of four honest states:
//
// "ok" -- configured AND a live round trip succeeded.
// "error" -- configured but the round trip failed (bad creds,
// host unreachable, timeout, ...). detail carries the
// underlying error message.
// "not_configured" -- no env var set. Not a failure by itself (0pt is
// designed to build/boot with everything unset).
// "not_integrated" -- env var IS set, but no client code in this repo
// actually uses it (currently only REDIS_URL/queue --
// see .env.example). Reporting "ok" here would be a
// fabricated check; reporting "not_configured" would
// hide that a value was provided. This is its own
// honest state rather than either of those two.
import { dbUrl, getSql } from "./db";
import { getStripe } from "./stripe";
import { hasEngine } from "./engine";
export type DepStatus = "ok" | "error" | "not_configured" | "not_integrated";
export interface DepResult {
status: DepStatus;
detail?: string;
latencyMs?: number;
}
export const DEFAULT_TIMEOUT_MS = 2000;
function errMessage(err: unknown): string {
return err instanceof Error ? err.message : String(err);
}
function withTimeout<T>(promise: Promise<T>, ms: number, label: string): Promise<T> {
return new Promise<T>((resolve, reject) => {
const timer = setTimeout(() => reject(new Error(`${label} timed out after ${ms}ms`)), ms);
promise.then(
(v) => {
clearTimeout(timer);
resolve(v);
},
(e) => {
clearTimeout(timer);
reject(e);
},
);
});
}
// Minimal shape of the one call site needs from the postgres client --
// matches `postgres`'s tagged-template callable, but kept narrow so tests
// can inject a fake without constructing a real `postgres.Sql`.
export type SqlPing = (strings: TemplateStringsArray, ...values: unknown[]) => Promise<unknown>;
/**
* Postgres reachability. `select 1` is the standard zero-cost liveness query
* -- it never touches application tables, so it can't fail on schema drift.
*/
export async function checkDatabase(
sqlClient: SqlPing | null = getSql() as unknown as SqlPing | null,
timeoutMs: number = DEFAULT_TIMEOUT_MS,
): Promise<DepResult> {
if (!dbUrl() || !sqlClient) return { status: "not_configured" };
const start = Date.now();
try {
await withTimeout(sqlClient`select 1`, timeoutMs, "database ping");
return { status: "ok", latencyMs: Date.now() - start };
} catch (err) {
return { status: "error", detail: errMessage(err), latencyMs: Date.now() - start };
}
}
/**
* FastAPI data engine (services/api) reachability via its own `GET /health`
* (services/api/orchestrator/main.py). Uses a real HTTP round trip, not just
* `hasEngine()` presence -- ENGINE_URL pointing at a dead host used to read
* as healthy.
*/
export async function checkEngine(
fetchImpl: typeof fetch = fetch,
timeoutMs: number = DEFAULT_TIMEOUT_MS,
): Promise<DepResult> {
if (!hasEngine()) return { status: "not_configured" };
const base = process.env.ENGINE_URL as string;
const start = Date.now();
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), timeoutMs);
try {
const res = await fetchImpl(`${base}/health`, {
signal: controller.signal,
cache: "no-store",
headers: { "X-Internal-Token": process.env.INTERNAL_TOKEN ?? "" },
});
if (!res.ok) {
return { status: "error", detail: `HTTP ${res.status}`, latencyMs: Date.now() - start };
}
return { status: "ok", latencyMs: Date.now() - start };
} catch (err) {
return { status: "error", detail: errMessage(err), latencyMs: Date.now() - start };
} finally {
clearTimeout(timer);
}
}
// Minimal shape of the one Stripe call this needs -- matches
// `Stripe#balance.retrieve`, kept narrow so tests can inject a fake instead
// of constructing a real Stripe client.
export interface StripeBalanceClient {
balance: { retrieve(): Promise<unknown> };
}
/**
* Stripe reachability. `balance.retrieve()` is Stripe's documented
* lightweight "is this key valid and can I reach the API" call -- it makes
* one real API round trip (catching e.g. a revoked/placeholder key) without
* mutating anything or touching a specific resource.
*/
export async function checkStripe(
client: StripeBalanceClient | null = getStripe() as unknown as StripeBalanceClient | null,
timeoutMs: number = DEFAULT_TIMEOUT_MS,
): Promise<DepResult> {
if (!client) return { status: "not_configured" };
const start = Date.now();
try {
await withTimeout(client.balance.retrieve(), timeoutMs, "stripe balance.retrieve");
return { status: "ok", latencyMs: Date.now() - start };
} catch (err) {
return { status: "error", detail: errMessage(err), latencyMs: Date.now() - start };
}
}
/**
* Redis/queue: presence-only, honestly. No queue client library is a
* dependency of this package (grep-verified against package.json) and no
* code path in this repo reads REDIS_URL to open a connection -- so there is
* no real check to run. Reporting "ok" would fabricate a probe that doesn't
* exist; this reports "not_integrated" instead of silently upgrading to a
* pretend pass.
*/
export function checkQueue(): DepResult {
return { status: process.env.REDIS_URL ? "not_integrated" : "not_configured" };
}
export interface DependencyResults {
db: DepResult;
engine: DepResult;
stripe: DepResult;
queue: DepResult;
}
export async function checkAllDependencies(): Promise<DependencyResults> {
const [db, engine, stripe] = await Promise.all([checkDatabase(), checkEngine(), checkStripe()]);
return { db, engine, stripe, queue: checkQueue() };
}
/** True when any dependency that IS configured/integrated is broken. */
export function hasDegradedDependency(deps: DependencyResults): boolean {
return Object.values(deps).some((d) => d.status === "error");
}
/**
* Readiness (unlike liveness) treats "not configured" for the deps this app
* cannot serve real traffic without as not-ready too. `db` is the only
* dependency that is load-bearing merely by being absent -- engine/stripe/
* queue are optional integrations the app is designed to run WITHOUT (see
* .env.example), so "not_configured" for those stays ready.
*
* But "optional to run without" is not the same as "optional to run BROKEN".
* A dependency in the "error" state was deliberately configured by an
* operator AND then failed a live round trip -- somebody asked for it and it
* is not answering. Readiness fails closed on that too.
*
* Measured 2026-09-25 on the production deployment (0pt.app / api.0pt.app),
* which is exactly the hole this closes:
*
* GET /api/health -> HTTP 200 {"ok":false, ..., "engine":"error",
* "detail":{"engine":"This operation was aborted"}}
* GET /api/ready -> HTTP 200 {"ready":true, "db":"ok", "engine":"error"}
*
* ENGINE_URL pointed at `http://0pt-engine:8000`, a hostname with no DNS
* record anywhere on the fleet (`getent hosts 0pt-engine` -> nothing; no such
* container on server5-11). The data engine -- 0pt's whole product -- had
* been absent for the container's entire 17.9 h uptime, and every signal
* above it was green. `ok:false` in a 200 body is not a signal; nothing reads
* bodies.
*/
export function isReady(deps: DependencyResults): boolean {
if (deps.db.status !== "ok") return false;
return !Object.values(deps).some((d) => d.status === "error");
}