Files
0pt/src/analytics/cost/query-budget.ts
T
Claude Opus 5 c4862fb7a7
CI (SIAX Cloud) / quality (push) Successful in 48s
CI (SIAX Cloud) / integration (push) Successful in 37s
CI (SIAX Cloud) / security (push) Successful in 17s
Security Scan (SIAX Cloud) / security (push) Successful in 17s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m16s
CI (SIAX Cloud) / integration (pull_request) Successful in 55s
CI (SIAX Cloud) / security (pull_request) Successful in 35s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 29s
docs(analytics): quote the numbers the COMMITTED evidence actually contains
The live suites regenerate their evidence on every run, so the BI-P021
overshoot figures moved when they were last re-run — and the doc and module
comment still quoted the earlier ones (130,883,409 rows / 130x / 92ms) while
the committed file said 153,318,696 / 153x / 55ms. Prose that disagrees with
the evidence it cites is worse than prose with no number in it.

Both now match the committed file, and both say what the earlier version
implied but did not state: the multiple is MACHINE-DEPENDENT and moves between
runs on the same box. That is the actual finding. The bound is "roughly one
poll interval of engine throughput", not a number, and quoting any specific
multiple as if it were a property of the system is the mistake this correction
exists to prevent.

The unwatched-baseline figure is corrected the same way: 4,227,318,261 rows
read before cancellation in the current BI-P019 record, not the 6.5 billion of
the earlier run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 03:18:12 +02:00

416 lines
16 KiB
TypeScript

/**
* src/analytics/cost/query-budget.ts
* ===================================
* BI-P021 — QUERY COST / BUDGET.
*
* ── SAY THE HONEST THING FIRST ────────────────────────────────────────────
* A budget checked once before execution and never again bounds NOTHING. The
* estimate is a planner's guess; a query that scans ten times its estimate
* runs to completion regardless, and the budget discovers this after the money
* is spent. Every "cost control" that consists only of a pre-flight check is
* an accounting feature wearing a safety feature's name.
*
* ClickHouse's own `max_rows_to_read` (BI-P012) DOES stop a query mid-flight —
* but it bounds ONE query. It cannot express "this tenant may scan 10 billion
* rows per hour across all its queries", because it has no idea what the
* tenant's other queries did. That window is what a budget is for, and it is
* exactly the thing a per-query setting cannot enforce.
*
* So this module has three parts, and only the second one is a control:
*
* 1. ADMISSION — estimate vs. remaining window. Rejects the obviously
* unaffordable before it starts. Cheap, and insufficient.
* 2. IN-FLIGHT — polls the engine's live `read_rows`/`read_bytes` counters
* for the running query and CANCELS it (via BI-P019, with
* verification) the moment window spend would be exceeded.
* THIS is what makes the budget real.
* 3. ACCOUNTING — settles the actual cost into a ledger the control plane
* reads: CL0UD quotas, AUD0 usage events, B00K accounting.
*
* If part 2 is disabled (`enforceInFlight: false`), `describeEnforcement()`
* says so in plain words, and the admission-only posture is reported as
* "advisory". A control that does not control should say which one it is.
*
* ── THE CAVEAT THAT THE LIVE RUN MEASURED ────────────────────────────────
* In-flight enforcement is granular to the poll interval, so a query can
* OVERSHOOT its budget between polls — and on a columnar engine the overshoot
* is large. Measured against ClickHouse 24.8.14.39 with a 1,000,000-row window
* and a 100ms poll: the query was cancelled at 153,318,696 rows read, 55ms in.
* A 153x overshoot. See evidence/bi-p019-p021/cancellation-and-budget.json,
* which is regenerated by the live suite — the figure is machine-dependent and
* moves between runs (an earlier run on the same box recorded 131M / 130x).
* Do not quote a specific multiple; quote the SHAPE.
*
* That shape is still a bound — the same query, unwatched, read 4,227,318,261
* rows before it was stopped in the BI-P019 case — but it is a bound of the
* form "within roughly one poll interval of engine throughput", not "at
* exactly the budget". Anyone sizing a budget must size it for that, and
* anyone quoting this as a hard cap is wrong. Shortening the interval narrows
* the overshoot and costs a `system.processes` read per poll per running
* query.
*/
import { SiaxError } from '../../platform/errors';
import {
cancelAndVerify,
type CancellationOutcome,
type EngineProcessControl,
} from '../execution/cancellation';
/** The cost axes BI-P021 names. */
export interface QueryCost {
rows_scanned: number;
bytes_scanned: number;
execution_ms: number;
engine: string;
/** Bytes read from object storage, when the engine distinguishes it. */
storage_read_bytes?: number;
}
/**
* Weights turning raw counters into the single `cost_units` figure BI-P018
* records and B00K bills.
*
* The defaults are deliberately dominated by BYTES, not rows: on a columnar
* engine a billion-row scan of one narrow column is cheap and a
* hundred-million-row scan of a wide one is not, so a rows-weighted unit would
* bill the wrong query.
*/
export interface CostModel {
unitsPerBillionRows: number;
unitsPerGigabyteScanned: number;
unitsPerSecond: number;
unitsPerGigabyteStorageRead: number;
}
export const DEFAULT_COST_MODEL: CostModel = {
unitsPerBillionRows: 1,
unitsPerGigabyteScanned: 10,
unitsPerSecond: 2,
unitsPerGigabyteStorageRead: 15,
};
export function costUnits(cost: QueryCost, model: CostModel = DEFAULT_COST_MODEL): number {
const units =
(cost.rows_scanned / 1e9) * model.unitsPerBillionRows +
(cost.bytes_scanned / 2 ** 30) * model.unitsPerGigabyteScanned +
(cost.execution_ms / 1000) * model.unitsPerSecond +
((cost.storage_read_bytes ?? 0) / 2 ** 30) * model.unitsPerGigabyteStorageRead;
// Rounded to six places so the same cost always serialises identically;
// float noise in a billing figure produces support tickets.
return Math.round(units * 1e6) / 1e6;
}
/** A tenant's allowance over a rolling window. Issued by CL0UD entitlements. */
export interface QueryBudget {
windowMs: number;
maxRowsScanned: number;
maxBytesScanned: number;
maxCostUnits: number;
/** What to do when the window is exhausted. */
onExhausted: 'reject' | 'throttle';
/** For `throttle`: queries are delayed by this much instead of refused. */
throttleDelayMs?: number;
}
export const DEFAULT_QUERY_BUDGET: QueryBudget = {
windowMs: 60 * 60 * 1000,
maxRowsScanned: 50_000_000_000,
maxBytesScanned: 2 * 2 ** 40,
maxCostUnits: 10_000,
onExhausted: 'reject',
};
interface LedgerEntry {
at: number;
rows: number;
bytes: number;
units: number;
}
/** Where settled cost goes. AUD0/B00K/CL0UD sit behind this. */
export interface UsageLedgerSink {
record(entry: {
tenant_id: string;
project_id?: string;
principal_id: string;
query_id: string;
cost: QueryCost;
cost_units: number;
/** True when the query was stopped by the budget rather than by the user. */
terminated_by_budget: boolean;
at: string;
}): Promise<void> | void;
}
export interface WindowSpend {
rows_scanned: number;
bytes_scanned: number;
cost_units: number;
queries: number;
window_started_at: string;
}
/**
* In-memory rolling window per tenant.
*
* Explicitly NOT the durable ledger — this is one process's view, and a
* multi-replica deployment needs the shared counter (Valkey or the CL0UD quota
* service) behind `SharedWindowStore` for the budget to bind across replicas.
* Stated rather than implied: with N replicas and no shared store, a tenant
* can spend N times its budget. `describeEnforcement()` reports this.
*/
export interface SharedWindowStore {
/** Atomically add to the tenant's window and return the new totals. */
add(
tenantId: string,
entry: { rows: number; bytes: number; units: number; windowMs: number },
): Promise<WindowSpend>;
get(tenantId: string): Promise<WindowSpend>;
}
export class InMemoryWindowStore implements SharedWindowStore {
private readonly entries = new Map<string, LedgerEntry[]>();
constructor(private readonly now: () => number = Date.now) {}
private prune(tenantId: string, windowMs: number): LedgerEntry[] {
const cutoff = this.now() - windowMs;
const kept = (this.entries.get(tenantId) ?? []).filter((e) => e.at >= cutoff);
this.entries.set(tenantId, kept);
return kept;
}
async add(
tenantId: string,
entry: { rows: number; bytes: number; units: number; windowMs: number },
): Promise<WindowSpend> {
const kept = this.prune(tenantId, entry.windowMs);
kept.push({ at: this.now(), rows: entry.rows, bytes: entry.bytes, units: entry.units });
this.entries.set(tenantId, kept);
return this.totals(kept, entry.windowMs);
}
async get(tenantId: string, windowMs = DEFAULT_QUERY_BUDGET.windowMs): Promise<WindowSpend> {
return this.totals(this.prune(tenantId, windowMs), windowMs);
}
private totals(entries: LedgerEntry[], windowMs: number): WindowSpend {
return {
rows_scanned: entries.reduce((a, e) => a + e.rows, 0),
bytes_scanned: entries.reduce((a, e) => a + e.bytes, 0),
cost_units: entries.reduce((a, e) => a + e.units, 0),
queries: entries.length,
window_started_at: new Date(this.now() - windowMs).toISOString(),
};
}
}
export interface BudgetEnforcerOptions {
budget: QueryBudget;
store: SharedWindowStore;
ledger: UsageLedgerSink;
control?: EngineProcessControl;
costModel?: CostModel;
/** The control. Off means admission-only, and `describeEnforcement` says so. */
enforceInFlight?: boolean;
pollIntervalMs?: number;
/** True when `store` is shared across replicas. Affects the honest description. */
storeIsShared?: boolean;
now?: () => number;
sleep?: (ms: number) => Promise<void>;
}
export interface AdmissionDecision {
admitted: boolean;
reason?: string;
/** For `throttle`: how long the caller must wait first. */
delay_ms?: number;
remaining: { rows: number; bytes: number; units: number };
}
export interface InFlightOutcome {
/** True when the enforcer stopped the query. */
terminated: boolean;
reason?: 'rows' | 'bytes' | 'units';
/** Present when it terminated: the verified cancellation. */
cancellation?: CancellationOutcome;
observed: { rows_read: number; bytes_read: number; elapsed_ms: number };
polls: number;
}
export class QueryBudgetEnforcer {
private readonly opts: Required<
Pick<BudgetEnforcerOptions, 'enforceInFlight' | 'pollIntervalMs' | 'storeIsShared'>
> &
BudgetEnforcerOptions;
private readonly now: () => number;
private readonly sleep: (ms: number) => Promise<void>;
constructor(options: BudgetEnforcerOptions) {
this.opts = {
enforceInFlight: options.enforceInFlight ?? true,
pollIntervalMs: options.pollIntervalMs ?? 200,
storeIsShared: options.storeIsShared ?? false,
...options,
};
this.now = options.now ?? Date.now;
this.sleep = options.sleep ?? ((ms) => new Promise((r) => setTimeout(r, ms)));
if (this.opts.enforceInFlight && !this.opts.control) {
throw new SiaxError(
'INVALID_REQUEST',
'0pt.analytics.cost',
'in-flight budget enforcement requires an EngineProcessControl; ' +
'construct with enforceInFlight:false to run admission-only and be honest about it',
);
}
}
/**
* Exactly what this enforcer does and does not bound, in words an operator
* can paste into an incident review. Returned by the analytics health
* endpoint so the posture is visible without reading this file.
*/
describeEnforcement(): {
admission: string;
in_flight: string;
across_replicas: string;
binding: boolean;
} {
return {
admission: 'estimate is compared to the remaining window before the query starts',
in_flight: this.opts.enforceInFlight
? `live engine counters are polled every ${this.opts.pollIntervalMs}ms and the query is cancelled ` +
'(with verification) when window spend would be exceeded'
: 'NOT ENFORCED — this budget is advisory. A query that exceeds its estimate runs to completion.',
across_replicas: this.opts.storeIsShared
? 'window spend is held in a shared store, so the budget binds across replicas'
: 'window spend is per-process — with N replicas a tenant can spend up to N times this budget',
binding: this.opts.enforceInFlight && this.opts.storeIsShared,
};
}
async admit(
tenantId: string,
estimate: { rows_to_read: number; bytes_to_read: number },
): Promise<AdmissionDecision> {
const spend = await this.opts.store.get(tenantId);
const b = this.opts.budget;
const remaining = {
rows: Math.max(0, b.maxRowsScanned - spend.rows_scanned),
bytes: Math.max(0, b.maxBytesScanned - spend.bytes_scanned),
units: Math.max(0, b.maxCostUnits - spend.cost_units),
};
const over =
estimate.rows_to_read > remaining.rows
? 'rows'
: estimate.bytes_to_read > remaining.bytes
? 'bytes'
: undefined;
if (!over) return { admitted: true, remaining };
if (b.onExhausted === 'throttle') {
return {
admitted: true,
delay_ms: b.throttleDelayMs ?? 1000,
reason: `tenant window ${over} budget exhausted; throttled`,
remaining,
};
}
return { admitted: false, reason: `tenant window ${over} budget exhausted`, remaining };
}
/**
* Watch a RUNNING query and stop it if it breaches the window.
*
* Runs concurrently with the query itself; the caller awaits both. Returns
* as soon as the query leaves the engine's process list, so a well-behaved
* query costs one extra poll.
*/
async enforceInFlight(tenantId: string, queryId: string, signal?: AbortSignal): Promise<InFlightOutcome> {
const observed = { rows_read: 0, bytes_read: 0, elapsed_ms: 0 };
if (!this.opts.enforceInFlight || !this.opts.control?.progress) {
return { terminated: false, observed, polls: 0 };
}
const control = this.opts.control;
const spendBefore = await this.opts.store.get(tenantId);
const b = this.opts.budget;
const remainingRows = Math.max(0, b.maxRowsScanned - spendBefore.rows_scanned);
const remainingBytes = Math.max(0, b.maxBytesScanned - spendBefore.bytes_scanned);
const remainingUnits = Math.max(0, b.maxCostUnits - spendBefore.cost_units);
let polls = 0;
for (;;) {
if (signal?.aborted) return { terminated: false, observed, polls };
let progress;
try {
progress = await control.progress?.(queryId);
} catch {
progress = undefined;
}
polls += 1;
if (!progress) {
// Gone from the process list: finished, or never started. Either way
// there is nothing left to bound.
return { terminated: false, observed, polls };
}
observed.rows_read = progress.rows_read;
observed.bytes_read = progress.bytes_read;
observed.elapsed_ms = progress.elapsed_ms;
const liveUnits = costUnits(
{
rows_scanned: progress.rows_read,
bytes_scanned: progress.bytes_read,
execution_ms: progress.elapsed_ms,
engine: control.engine,
},
this.opts.costModel,
);
const breach =
progress.rows_read > remainingRows
? 'rows'
: progress.bytes_read > remainingBytes
? 'bytes'
: liveUnits > remainingUnits
? 'units'
: undefined;
if (breach) {
const cancellation = await cancelAndVerify(control, queryId, { timeoutMs: 5_000 });
return { terminated: true, reason: breach, cancellation, observed, polls };
}
await this.sleep(this.opts.pollIntervalMs);
}
}
/** Settle the actual cost. Called for EVERY outcome, including failures. */
async settle(args: {
tenant_id: string;
project_id?: string;
principal_id: string;
query_id: string;
cost: QueryCost;
terminated_by_budget?: boolean;
}): Promise<{ cost_units: number; window: WindowSpend }> {
const units = costUnits(args.cost, this.opts.costModel);
const window = await this.opts.store.add(args.tenant_id, {
rows: args.cost.rows_scanned,
bytes: args.cost.bytes_scanned,
units,
windowMs: this.opts.budget.windowMs,
});
await this.opts.ledger.record({
tenant_id: args.tenant_id,
project_id: args.project_id,
principal_id: args.principal_id,
query_id: args.query_id,
cost: args.cost,
cost_units: units,
// A query the budget killed still COST what it burned before it died.
// Not charging for it is how a tenant discovers that the cheapest query
// is one that gets cancelled.
terminated_by_budget: args.terminated_by_budget ?? false,
at: new Date(this.now()).toISOString(),
});
return { cost_units: units, window };
}
}