feat(analytics): BI-P018/BI-P019/BI-P021 — execution model, verified cancellation, a budget that stops queries
CI (SIAX Cloud) / integration (push) Successful in 43s
CI (SIAX Cloud) / security (push) Successful in 21s
CI (SIAX Cloud) / quality (push) Successful in 1m5s
Security Scan (SIAX Cloud) / security (push) Successful in 23s
CI (SIAX Cloud) / security (pull_request) Successful in 24s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m0s
CI (SIAX Cloud) / integration (pull_request) Successful in 42s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 38s
CI (SIAX Cloud) / integration (push) Successful in 43s
CI (SIAX Cloud) / security (push) Successful in 21s
CI (SIAX Cloud) / quality (push) Successful in 1m5s
Security Scan (SIAX Cloud) / security (push) Successful in 23s
CI (SIAX Cloud) / security (pull_request) Successful in 24s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m0s
CI (SIAX Cloud) / integration (pull_request) Successful in 42s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 38s
BI-P018 — every field the prompt names, plus the one it names by its absence.
The record carries `compiled_query_hash` and never the statement, and that is
ENFORCED: `assertNoRawSql` runs before any record is emitted and refuses one
whose `error` field looks like SQL. The realistic leak is ClickHouse's own
error text, which echoes the query — the test uses exactly that string. The
recorder writes at EVERY transition, because a recorder that only writes at
the end loses precisely the executions worth investigating. Illegal state
transitions throw rather than overwrite a terminal row.
The one place raw SQL is retained stays BI-P017's analyst statement history,
where a privileged human ran unreviewed SQL and "who ran what" must survive.
Different record, different retention posture, stated in both files.
BI-P019 — "stops" is VERIFIED, not requested. Aborting the client stops US
waiting, not the server; KILL QUERY without SYNC tells you nothing; even SYNC
can return while the query unwinds. So cancelAndVerify aborts, kills, then
POLLS the process list until the query is GONE or the deadline passes — and
if the deadline passes it returns `verified_stopped: false` rather than a
cheerful `true`. The return type is an outcome object, not a boolean, so a
caller cannot mistake "we asked" for "it stopped". An unreachable engine is
read as "still running" (fail closed), so the kill is still attempted at the
moment an operator most needs it to be.
Proven live against ClickHouse 24.8.14.39, with the counterfactual first:
* an identical bomb left alone IS still running after 2s — so "gone after
we cancelled" is not just "the query was short";
* the cancelled query had read 6,559,476,156 rows, and was verified gone
57ms after the request;
* system.query_log records ExceptionWhileProcessing with code 394,
QUERY_WAS_CANCELLED — ClickHouse's verdict, not ours.
BI-P021 — the honest version. A budget checked once before execution bounds
nothing, and the module says so in its first paragraph. ClickHouse's
max_rows_to_read bounds ONE query; it cannot express "this tenant may scan X
per hour", which is what a budget is for. So enforcement polls the engine's
live read_rows/read_bytes counters and CANCELS the running query, through
BI-P019 with verification, when window spend would be exceeded.
Measured live: a 1,000,000-row window cancelled the query at 130,883,409 rows,
92ms in, verified stopped, code 394. THE OVERSHOOT IS DOCUMENTED, in the module
and in the evidence: enforcement is granular to the poll interval, so this is
a bound of the form "within roughly one poll interval of engine throughput",
not a hard cap. Quoting it as a hard cap would be wrong.
Two more things stated rather than implied:
* `enforceInFlight: false` is reported by describeEnforcement() as
"NOT ENFORCED — this budget is advisory", and the constructor REFUSES to
accept enforceInFlight without a process control;
* an unshared window store is reported as non-binding across replicas
("with N replicas a tenant can spend up to N times this budget").
* a query the budget killed is still charged for the work it burned. Not
charging is how a tenant learns the cheapest query is a cancelled one.
Evidence: evidence/bi-p019-p021/cancellation-and-budget.json
268 unit tests, 7 live tests. typecheck + eslint clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,411 @@
|
||||
/**
|
||||
* src/analytics/cost/query-budget.ts
|
||||
* ===================================
|
||||
* BI-P021 — QUERY COST / BUDGET.
|
||||
*
|
||||
* ── SAY THE HONEST THING FIRST ────────────────────────────────────────────
|
||||
* A budget checked once before execution and never again bounds NOTHING. The
|
||||
* estimate is a planner's guess; a query that scans ten times its estimate
|
||||
* runs to completion regardless, and the budget discovers this after the money
|
||||
* is spent. Every "cost control" that consists only of a pre-flight check is
|
||||
* an accounting feature wearing a safety feature's name.
|
||||
*
|
||||
* ClickHouse's own `max_rows_to_read` (BI-P012) DOES stop a query mid-flight —
|
||||
* but it bounds ONE query. It cannot express "this tenant may scan 10 billion
|
||||
* rows per hour across all its queries", because it has no idea what the
|
||||
* tenant's other queries did. That window is what a budget is for, and it is
|
||||
* exactly the thing a per-query setting cannot enforce.
|
||||
*
|
||||
* So this module has three parts, and only the second one is a control:
|
||||
*
|
||||
* 1. ADMISSION — estimate vs. remaining window. Rejects the obviously
|
||||
* unaffordable before it starts. Cheap, and insufficient.
|
||||
* 2. IN-FLIGHT — polls the engine's live `read_rows`/`read_bytes` counters
|
||||
* for the running query and CANCELS it (via BI-P019, with
|
||||
* verification) the moment window spend would be exceeded.
|
||||
* THIS is what makes the budget real.
|
||||
* 3. ACCOUNTING — settles the actual cost into a ledger the control plane
|
||||
* reads: CL0UD quotas, AUD0 usage events, B00K accounting.
|
||||
*
|
||||
* If part 2 is disabled (`enforceInFlight: false`), `describeEnforcement()`
|
||||
* says so in plain words, and the admission-only posture is reported as
|
||||
* "advisory". A control that does not control should say which one it is.
|
||||
*
|
||||
* ── THE CAVEAT THAT THE LIVE RUN MEASURED ────────────────────────────────
|
||||
* In-flight enforcement is granular to the poll interval, so a query can
|
||||
* OVERSHOOT its budget between polls — and on a columnar engine the overshoot
|
||||
* is large. Measured against ClickHouse 24.8.14.39 with a 1,000,000-row window
|
||||
* and a 100ms poll: the query was cancelled at 130,883,409 rows read, 92ms in.
|
||||
* A 130x overshoot. See evidence/bi-p019-p021/cancellation-and-budget.json.
|
||||
*
|
||||
* That is still a bound — the same query, unwatched, read 6,559,476,156 rows
|
||||
* in one second — but it is a bound of the form "within roughly one poll
|
||||
* interval of engine throughput", not "at exactly the budget". Anyone sizing a
|
||||
* budget must size it for that, and anyone quoting this as a hard cap is
|
||||
* wrong. Shortening the interval narrows the overshoot and costs a
|
||||
* `system.processes` read per poll per running query.
|
||||
*/
|
||||
import { SiaxError } from '../../platform/errors';
|
||||
import {
|
||||
cancelAndVerify,
|
||||
type CancellationOutcome,
|
||||
type EngineProcessControl,
|
||||
} from '../execution/cancellation';
|
||||
|
||||
/** The cost axes BI-P021 names. */
|
||||
export interface QueryCost {
|
||||
rows_scanned: number;
|
||||
bytes_scanned: number;
|
||||
execution_ms: number;
|
||||
engine: string;
|
||||
/** Bytes read from object storage, when the engine distinguishes it. */
|
||||
storage_read_bytes?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Weights turning raw counters into the single `cost_units` figure BI-P018
|
||||
* records and B00K bills.
|
||||
*
|
||||
* The defaults are deliberately dominated by BYTES, not rows: on a columnar
|
||||
* engine a billion-row scan of one narrow column is cheap and a
|
||||
* hundred-million-row scan of a wide one is not, so a rows-weighted unit would
|
||||
* bill the wrong query.
|
||||
*/
|
||||
export interface CostModel {
|
||||
unitsPerBillionRows: number;
|
||||
unitsPerGigabyteScanned: number;
|
||||
unitsPerSecond: number;
|
||||
unitsPerGigabyteStorageRead: number;
|
||||
}
|
||||
|
||||
export const DEFAULT_COST_MODEL: CostModel = {
|
||||
unitsPerBillionRows: 1,
|
||||
unitsPerGigabyteScanned: 10,
|
||||
unitsPerSecond: 2,
|
||||
unitsPerGigabyteStorageRead: 15,
|
||||
};
|
||||
|
||||
export function costUnits(cost: QueryCost, model: CostModel = DEFAULT_COST_MODEL): number {
|
||||
const units =
|
||||
(cost.rows_scanned / 1e9) * model.unitsPerBillionRows +
|
||||
(cost.bytes_scanned / 2 ** 30) * model.unitsPerGigabyteScanned +
|
||||
(cost.execution_ms / 1000) * model.unitsPerSecond +
|
||||
((cost.storage_read_bytes ?? 0) / 2 ** 30) * model.unitsPerGigabyteStorageRead;
|
||||
// Rounded to six places so the same cost always serialises identically;
|
||||
// float noise in a billing figure produces support tickets.
|
||||
return Math.round(units * 1e6) / 1e6;
|
||||
}
|
||||
|
||||
/** A tenant's allowance over a rolling window. Issued by CL0UD entitlements. */
|
||||
export interface QueryBudget {
|
||||
windowMs: number;
|
||||
maxRowsScanned: number;
|
||||
maxBytesScanned: number;
|
||||
maxCostUnits: number;
|
||||
/** What to do when the window is exhausted. */
|
||||
onExhausted: 'reject' | 'throttle';
|
||||
/** For `throttle`: queries are delayed by this much instead of refused. */
|
||||
throttleDelayMs?: number;
|
||||
}
|
||||
|
||||
export const DEFAULT_QUERY_BUDGET: QueryBudget = {
|
||||
windowMs: 60 * 60 * 1000,
|
||||
maxRowsScanned: 50_000_000_000,
|
||||
maxBytesScanned: 2 * 2 ** 40,
|
||||
maxCostUnits: 10_000,
|
||||
onExhausted: 'reject',
|
||||
};
|
||||
|
||||
interface LedgerEntry {
|
||||
at: number;
|
||||
rows: number;
|
||||
bytes: number;
|
||||
units: number;
|
||||
}
|
||||
|
||||
/** Where settled cost goes. AUD0/B00K/CL0UD sit behind this. */
|
||||
export interface UsageLedgerSink {
|
||||
record(entry: {
|
||||
tenant_id: string;
|
||||
project_id?: string;
|
||||
principal_id: string;
|
||||
query_id: string;
|
||||
cost: QueryCost;
|
||||
cost_units: number;
|
||||
/** True when the query was stopped by the budget rather than by the user. */
|
||||
terminated_by_budget: boolean;
|
||||
at: string;
|
||||
}): Promise<void> | void;
|
||||
}
|
||||
|
||||
export interface WindowSpend {
|
||||
rows_scanned: number;
|
||||
bytes_scanned: number;
|
||||
cost_units: number;
|
||||
queries: number;
|
||||
window_started_at: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* In-memory rolling window per tenant.
|
||||
*
|
||||
* Explicitly NOT the durable ledger — this is one process's view, and a
|
||||
* multi-replica deployment needs the shared counter (Valkey or the CL0UD quota
|
||||
* service) behind `SharedWindowStore` for the budget to bind across replicas.
|
||||
* Stated rather than implied: with N replicas and no shared store, a tenant
|
||||
* can spend N times its budget. `describeEnforcement()` reports this.
|
||||
*/
|
||||
export interface SharedWindowStore {
|
||||
/** Atomically add to the tenant's window and return the new totals. */
|
||||
add(
|
||||
tenantId: string,
|
||||
entry: { rows: number; bytes: number; units: number; windowMs: number },
|
||||
): Promise<WindowSpend>;
|
||||
get(tenantId: string): Promise<WindowSpend>;
|
||||
}
|
||||
|
||||
export class InMemoryWindowStore implements SharedWindowStore {
|
||||
private readonly entries = new Map<string, LedgerEntry[]>();
|
||||
constructor(private readonly now: () => number = Date.now) {}
|
||||
|
||||
private prune(tenantId: string, windowMs: number): LedgerEntry[] {
|
||||
const cutoff = this.now() - windowMs;
|
||||
const kept = (this.entries.get(tenantId) ?? []).filter((e) => e.at >= cutoff);
|
||||
this.entries.set(tenantId, kept);
|
||||
return kept;
|
||||
}
|
||||
|
||||
async add(
|
||||
tenantId: string,
|
||||
entry: { rows: number; bytes: number; units: number; windowMs: number },
|
||||
): Promise<WindowSpend> {
|
||||
const kept = this.prune(tenantId, entry.windowMs);
|
||||
kept.push({ at: this.now(), rows: entry.rows, bytes: entry.bytes, units: entry.units });
|
||||
this.entries.set(tenantId, kept);
|
||||
return this.totals(kept, entry.windowMs);
|
||||
}
|
||||
|
||||
async get(tenantId: string, windowMs = DEFAULT_QUERY_BUDGET.windowMs): Promise<WindowSpend> {
|
||||
return this.totals(this.prune(tenantId, windowMs), windowMs);
|
||||
}
|
||||
|
||||
private totals(entries: LedgerEntry[], windowMs: number): WindowSpend {
|
||||
return {
|
||||
rows_scanned: entries.reduce((a, e) => a + e.rows, 0),
|
||||
bytes_scanned: entries.reduce((a, e) => a + e.bytes, 0),
|
||||
cost_units: entries.reduce((a, e) => a + e.units, 0),
|
||||
queries: entries.length,
|
||||
window_started_at: new Date(this.now() - windowMs).toISOString(),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
export interface BudgetEnforcerOptions {
|
||||
budget: QueryBudget;
|
||||
store: SharedWindowStore;
|
||||
ledger: UsageLedgerSink;
|
||||
control?: EngineProcessControl;
|
||||
costModel?: CostModel;
|
||||
/** The control. Off means admission-only, and `describeEnforcement` says so. */
|
||||
enforceInFlight?: boolean;
|
||||
pollIntervalMs?: number;
|
||||
/** True when `store` is shared across replicas. Affects the honest description. */
|
||||
storeIsShared?: boolean;
|
||||
now?: () => number;
|
||||
sleep?: (ms: number) => Promise<void>;
|
||||
}
|
||||
|
||||
export interface AdmissionDecision {
|
||||
admitted: boolean;
|
||||
reason?: string;
|
||||
/** For `throttle`: how long the caller must wait first. */
|
||||
delay_ms?: number;
|
||||
remaining: { rows: number; bytes: number; units: number };
|
||||
}
|
||||
|
||||
export interface InFlightOutcome {
|
||||
/** True when the enforcer stopped the query. */
|
||||
terminated: boolean;
|
||||
reason?: 'rows' | 'bytes' | 'units';
|
||||
/** Present when it terminated: the verified cancellation. */
|
||||
cancellation?: CancellationOutcome;
|
||||
observed: { rows_read: number; bytes_read: number; elapsed_ms: number };
|
||||
polls: number;
|
||||
}
|
||||
|
||||
export class QueryBudgetEnforcer {
|
||||
private readonly opts: Required<
|
||||
Pick<BudgetEnforcerOptions, 'enforceInFlight' | 'pollIntervalMs' | 'storeIsShared'>
|
||||
> &
|
||||
BudgetEnforcerOptions;
|
||||
private readonly now: () => number;
|
||||
private readonly sleep: (ms: number) => Promise<void>;
|
||||
|
||||
constructor(options: BudgetEnforcerOptions) {
|
||||
this.opts = {
|
||||
enforceInFlight: options.enforceInFlight ?? true,
|
||||
pollIntervalMs: options.pollIntervalMs ?? 200,
|
||||
storeIsShared: options.storeIsShared ?? false,
|
||||
...options,
|
||||
};
|
||||
this.now = options.now ?? Date.now;
|
||||
this.sleep = options.sleep ?? ((ms) => new Promise((r) => setTimeout(r, ms)));
|
||||
if (this.opts.enforceInFlight && !this.opts.control) {
|
||||
throw new SiaxError(
|
||||
'INVALID_REQUEST',
|
||||
'0pt.analytics.cost',
|
||||
'in-flight budget enforcement requires an EngineProcessControl; ' +
|
||||
'construct with enforceInFlight:false to run admission-only and be honest about it',
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Exactly what this enforcer does and does not bound, in words an operator
|
||||
* can paste into an incident review. Returned by the analytics health
|
||||
* endpoint so the posture is visible without reading this file.
|
||||
*/
|
||||
describeEnforcement(): {
|
||||
admission: string;
|
||||
in_flight: string;
|
||||
across_replicas: string;
|
||||
binding: boolean;
|
||||
} {
|
||||
return {
|
||||
admission: 'estimate is compared to the remaining window before the query starts',
|
||||
in_flight: this.opts.enforceInFlight
|
||||
? `live engine counters are polled every ${this.opts.pollIntervalMs}ms and the query is cancelled ` +
|
||||
'(with verification) when window spend would be exceeded'
|
||||
: 'NOT ENFORCED — this budget is advisory. A query that exceeds its estimate runs to completion.',
|
||||
across_replicas: this.opts.storeIsShared
|
||||
? 'window spend is held in a shared store, so the budget binds across replicas'
|
||||
: 'window spend is per-process — with N replicas a tenant can spend up to N times this budget',
|
||||
binding: this.opts.enforceInFlight && this.opts.storeIsShared,
|
||||
};
|
||||
}
|
||||
|
||||
async admit(
|
||||
tenantId: string,
|
||||
estimate: { rows_to_read: number; bytes_to_read: number },
|
||||
): Promise<AdmissionDecision> {
|
||||
const spend = await this.opts.store.get(tenantId);
|
||||
const b = this.opts.budget;
|
||||
const remaining = {
|
||||
rows: Math.max(0, b.maxRowsScanned - spend.rows_scanned),
|
||||
bytes: Math.max(0, b.maxBytesScanned - spend.bytes_scanned),
|
||||
units: Math.max(0, b.maxCostUnits - spend.cost_units),
|
||||
};
|
||||
const over =
|
||||
estimate.rows_to_read > remaining.rows
|
||||
? 'rows'
|
||||
: estimate.bytes_to_read > remaining.bytes
|
||||
? 'bytes'
|
||||
: undefined;
|
||||
if (!over) return { admitted: true, remaining };
|
||||
if (b.onExhausted === 'throttle') {
|
||||
return {
|
||||
admitted: true,
|
||||
delay_ms: b.throttleDelayMs ?? 1000,
|
||||
reason: `tenant window ${over} budget exhausted; throttled`,
|
||||
remaining,
|
||||
};
|
||||
}
|
||||
return { admitted: false, reason: `tenant window ${over} budget exhausted`, remaining };
|
||||
}
|
||||
|
||||
/**
|
||||
* Watch a RUNNING query and stop it if it breaches the window.
|
||||
*
|
||||
* Runs concurrently with the query itself; the caller awaits both. Returns
|
||||
* as soon as the query leaves the engine's process list, so a well-behaved
|
||||
* query costs one extra poll.
|
||||
*/
|
||||
async enforceInFlight(tenantId: string, queryId: string, signal?: AbortSignal): Promise<InFlightOutcome> {
|
||||
const observed = { rows_read: 0, bytes_read: 0, elapsed_ms: 0 };
|
||||
if (!this.opts.enforceInFlight || !this.opts.control?.progress) {
|
||||
return { terminated: false, observed, polls: 0 };
|
||||
}
|
||||
const control = this.opts.control;
|
||||
const spendBefore = await this.opts.store.get(tenantId);
|
||||
const b = this.opts.budget;
|
||||
const remainingRows = Math.max(0, b.maxRowsScanned - spendBefore.rows_scanned);
|
||||
const remainingBytes = Math.max(0, b.maxBytesScanned - spendBefore.bytes_scanned);
|
||||
const remainingUnits = Math.max(0, b.maxCostUnits - spendBefore.cost_units);
|
||||
|
||||
let polls = 0;
|
||||
for (;;) {
|
||||
if (signal?.aborted) return { terminated: false, observed, polls };
|
||||
let progress;
|
||||
try {
|
||||
progress = await control.progress?.(queryId);
|
||||
} catch {
|
||||
progress = undefined;
|
||||
}
|
||||
polls += 1;
|
||||
if (!progress) {
|
||||
// Gone from the process list: finished, or never started. Either way
|
||||
// there is nothing left to bound.
|
||||
return { terminated: false, observed, polls };
|
||||
}
|
||||
observed.rows_read = progress.rows_read;
|
||||
observed.bytes_read = progress.bytes_read;
|
||||
observed.elapsed_ms = progress.elapsed_ms;
|
||||
|
||||
const liveUnits = costUnits(
|
||||
{
|
||||
rows_scanned: progress.rows_read,
|
||||
bytes_scanned: progress.bytes_read,
|
||||
execution_ms: progress.elapsed_ms,
|
||||
engine: control.engine,
|
||||
},
|
||||
this.opts.costModel,
|
||||
);
|
||||
const breach =
|
||||
progress.rows_read > remainingRows
|
||||
? 'rows'
|
||||
: progress.bytes_read > remainingBytes
|
||||
? 'bytes'
|
||||
: liveUnits > remainingUnits
|
||||
? 'units'
|
||||
: undefined;
|
||||
|
||||
if (breach) {
|
||||
const cancellation = await cancelAndVerify(control, queryId, { timeoutMs: 5_000 });
|
||||
return { terminated: true, reason: breach, cancellation, observed, polls };
|
||||
}
|
||||
await this.sleep(this.opts.pollIntervalMs);
|
||||
}
|
||||
}
|
||||
|
||||
/** Settle the actual cost. Called for EVERY outcome, including failures. */
|
||||
async settle(args: {
|
||||
tenant_id: string;
|
||||
project_id?: string;
|
||||
principal_id: string;
|
||||
query_id: string;
|
||||
cost: QueryCost;
|
||||
terminated_by_budget?: boolean;
|
||||
}): Promise<{ cost_units: number; window: WindowSpend }> {
|
||||
const units = costUnits(args.cost, this.opts.costModel);
|
||||
const window = await this.opts.store.add(args.tenant_id, {
|
||||
rows: args.cost.rows_scanned,
|
||||
bytes: args.cost.bytes_scanned,
|
||||
units,
|
||||
windowMs: this.opts.budget.windowMs,
|
||||
});
|
||||
await this.opts.ledger.record({
|
||||
tenant_id: args.tenant_id,
|
||||
project_id: args.project_id,
|
||||
principal_id: args.principal_id,
|
||||
query_id: args.query_id,
|
||||
cost: args.cost,
|
||||
cost_units: units,
|
||||
// A query the budget killed still COST what it burned before it died.
|
||||
// Not charging for it is how a tenant discovers that the cheapest query
|
||||
// is one that gets cancelled.
|
||||
terminated_by_budget: args.terminated_by_budget ?? false,
|
||||
at: new Date(this.now()).toISOString(),
|
||||
});
|
||||
return { cost_units: units, window };
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,183 @@
|
||||
/**
|
||||
* src/analytics/execution/cancellation.ts
|
||||
* ========================================
|
||||
* BI-P019 — QUERY CANCELLATION.
|
||||
*
|
||||
* "Integrate engine cancellation. DoD: User cancellation stops expensive
|
||||
* query within defined timeout."
|
||||
*
|
||||
* THE THING THAT MAKES THIS REAL: "stops" is VERIFIED, not requested.
|
||||
*
|
||||
* The obvious implementation aborts the HTTP request and issues a KILL. Both
|
||||
* of those are requests. The client abort stops US waiting; it does not stop
|
||||
* the server, which keeps burning CPU on a query nobody will read. `KILL QUERY`
|
||||
* without `SYNC` returns immediately and tells you nothing. Even `KILL QUERY
|
||||
* SYNC` can return while the query is still winding down.
|
||||
*
|
||||
* So `cancelAndVerify` does three things and reports on all three:
|
||||
* 1. abort the client side;
|
||||
* 2. issue the engine's kill;
|
||||
* 3. POLL the engine's process list until the query is GONE, or the deadline
|
||||
* passes — and if the deadline passes, it says so with
|
||||
* `verified_stopped: false` rather than returning a cheerful `true`.
|
||||
*
|
||||
* A caller that treats `cancelQuery()` returning `true` as "the query stopped"
|
||||
* is the failure this module exists to prevent, which is why the return type
|
||||
* is an outcome object and not a boolean.
|
||||
*/
|
||||
import { SiaxError } from '../../platform/errors';
|
||||
|
||||
/** Whatever can list and kill running queries. ClickHouse and Trino both can. */
|
||||
export interface EngineProcessControl {
|
||||
readonly engine: string;
|
||||
/** True when a query with this id is still running on the engine. */
|
||||
isRunning(queryId: string): Promise<boolean>;
|
||||
/** Ask the engine to stop it. Returning true means "accepted", not "stopped". */
|
||||
requestKill(queryId: string): Promise<boolean>;
|
||||
/** Live progress for the running query, when the engine exposes it (BI-P021). */
|
||||
progress?(queryId: string): Promise<QueryProgress | undefined>;
|
||||
}
|
||||
|
||||
export interface QueryProgress {
|
||||
rows_read: number;
|
||||
bytes_read: number;
|
||||
elapsed_ms: number;
|
||||
}
|
||||
|
||||
export interface CancellationOutcome {
|
||||
query_id: string;
|
||||
requested_at: string;
|
||||
/** The engine accepted the kill request. */
|
||||
kill_accepted: boolean;
|
||||
/**
|
||||
* The query was OBSERVED to be gone from the engine's process list.
|
||||
* `false` means it may still be running — the honest answer.
|
||||
*/
|
||||
verified_stopped: boolean;
|
||||
/** Milliseconds from the cancel request to the observation. */
|
||||
elapsed_ms: number;
|
||||
/** The deadline this cancellation was held to. */
|
||||
timeout_ms: number;
|
||||
/** Present when the query had already finished before the request landed. */
|
||||
already_finished: boolean;
|
||||
}
|
||||
|
||||
export interface CancellationOptions {
|
||||
/** The defined timeout BI-P019's DoD refers to. */
|
||||
timeoutMs?: number;
|
||||
pollIntervalMs?: number;
|
||||
/** Aborts the client-side wait. */
|
||||
abort?: () => void;
|
||||
now?: () => number;
|
||||
sleep?: (ms: number) => Promise<void>;
|
||||
}
|
||||
|
||||
export const DEFAULT_CANCELLATION_TIMEOUT_MS = 5_000;
|
||||
const DEFAULT_POLL_INTERVAL_MS = 100;
|
||||
|
||||
const defaultSleep = (ms: number): Promise<void> => new Promise((r) => setTimeout(r, ms));
|
||||
|
||||
export async function cancelAndVerify(
|
||||
control: EngineProcessControl,
|
||||
queryId: string,
|
||||
opts: CancellationOptions = {},
|
||||
): Promise<CancellationOutcome> {
|
||||
if (!queryId?.trim()) {
|
||||
throw new SiaxError('INVALID_REQUEST', '0pt.analytics', 'cancellation requires a query id');
|
||||
}
|
||||
const timeoutMs = opts.timeoutMs ?? DEFAULT_CANCELLATION_TIMEOUT_MS;
|
||||
const pollIntervalMs = opts.pollIntervalMs ?? DEFAULT_POLL_INTERVAL_MS;
|
||||
const now = opts.now ?? Date.now;
|
||||
const sleep = opts.sleep ?? defaultSleep;
|
||||
const startedMs = now();
|
||||
const requestedAt = new Date(startedMs).toISOString();
|
||||
|
||||
// Was it even running? Cancelling a finished query is a no-op, and reporting
|
||||
// it as a successful stop would overstate what happened.
|
||||
//
|
||||
// FAIL CLOSED when the engine cannot be asked: an unreachable engine means
|
||||
// "unknown", and the safe reading of unknown is "still running" — so the
|
||||
// kill is still issued and the verification loop still has to observe the
|
||||
// query gone. Letting this throw would abandon the cancellation entirely at
|
||||
// the exact moment the operator most needs it to be attempted.
|
||||
let runningBefore: boolean;
|
||||
try {
|
||||
runningBefore = await control.isRunning(queryId);
|
||||
} catch {
|
||||
runningBefore = true;
|
||||
}
|
||||
|
||||
// 1. client side: stop waiting.
|
||||
opts.abort?.();
|
||||
|
||||
// 2. engine side: ask it to stop.
|
||||
let killAccepted = false;
|
||||
try {
|
||||
killAccepted = await control.requestKill(queryId);
|
||||
} catch {
|
||||
killAccepted = false;
|
||||
}
|
||||
|
||||
if (!runningBefore) {
|
||||
return {
|
||||
query_id: queryId,
|
||||
requested_at: requestedAt,
|
||||
kill_accepted: killAccepted,
|
||||
verified_stopped: true,
|
||||
elapsed_ms: now() - startedMs,
|
||||
timeout_ms: timeoutMs,
|
||||
already_finished: true,
|
||||
};
|
||||
}
|
||||
|
||||
// 3. VERIFY. This loop is the whole point of the module.
|
||||
let verified = false;
|
||||
for (;;) {
|
||||
let stillRunning: boolean;
|
||||
try {
|
||||
stillRunning = await control.isRunning(queryId);
|
||||
} catch {
|
||||
// An engine we cannot ask is an engine we cannot vouch for. Keep trying
|
||||
// until the deadline, then report unverified.
|
||||
stillRunning = true;
|
||||
}
|
||||
if (!stillRunning) {
|
||||
verified = true;
|
||||
break;
|
||||
}
|
||||
if (now() - startedMs >= timeoutMs) break;
|
||||
await sleep(pollIntervalMs);
|
||||
}
|
||||
|
||||
return {
|
||||
query_id: queryId,
|
||||
requested_at: requestedAt,
|
||||
kill_accepted: killAccepted,
|
||||
verified_stopped: verified,
|
||||
elapsed_ms: now() - startedMs,
|
||||
timeout_ms: timeoutMs,
|
||||
already_finished: false,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The same call, for a caller that wants a failure rather than a report.
|
||||
* Throws `RETRYABLE_FAILURE` when the query outlived the deadline — because
|
||||
* "we asked it to stop and it did not" is an operational condition someone
|
||||
* has to see, not a value to ignore.
|
||||
*/
|
||||
export async function cancelOrThrow(
|
||||
control: EngineProcessControl,
|
||||
queryId: string,
|
||||
opts: CancellationOptions = {},
|
||||
): Promise<CancellationOutcome> {
|
||||
const outcome = await cancelAndVerify(control, queryId, opts);
|
||||
if (!outcome.verified_stopped) {
|
||||
throw new SiaxError(
|
||||
'RETRYABLE_FAILURE',
|
||||
`0pt.analytics.${control.engine}`,
|
||||
`query was not observed to stop within ${outcome.timeout_ms}ms of cancellation`,
|
||||
);
|
||||
}
|
||||
return outcome;
|
||||
}
|
||||
@@ -0,0 +1,138 @@
|
||||
/**
|
||||
* src/analytics/execution/clickhouse-process-control.ts
|
||||
* ======================================================
|
||||
* BI-P019/BI-P021 — the ClickHouse implementation of `EngineProcessControl`.
|
||||
*
|
||||
* Three statements, each chosen for a measured reason:
|
||||
*
|
||||
* isRunning `SELECT count() FROM system.processes WHERE query_id = …`
|
||||
* — the authority on whether the server is still working. Note
|
||||
* that `KILL QUERY SYNC` returning is NOT this: a killed query
|
||||
* can still appear here briefly while it unwinds, which is
|
||||
* exactly why cancellation polls instead of trusting the kill.
|
||||
*
|
||||
* requestKill `KILL QUERY WHERE query_id = … ASYNC` — ASYNC, not SYNC,
|
||||
* because the verification loop is what establishes the stop and
|
||||
* a SYNC kill would block this call for up to the query's
|
||||
* remaining lifetime while providing no stronger guarantee.
|
||||
*
|
||||
* progress `read_rows`, `read_bytes` and `elapsed` from the same table —
|
||||
* live counters, which is what makes BI-P021's in-flight budget
|
||||
* enforcement possible at all.
|
||||
*
|
||||
* KILL and the system-table reads are ADMINISTRATIVE. They run on the engine's
|
||||
* own credential, never on the tenant's restricted user (which has no access
|
||||
* to system.processes — measured: ACCESS_DENIED, code 497). The tenant-scoping
|
||||
* check therefore happens HERE, on the query id, before any statement is sent.
|
||||
*/
|
||||
import { SiaxError } from '../../platform/errors';
|
||||
import type { EngineProcessControl, QueryProgress } from './cancellation';
|
||||
|
||||
export interface ClickHouseProcessControlOptions {
|
||||
baseUrl: string;
|
||||
/** "user:password" for the ADMIN credential. From a secret ref. */
|
||||
basicAuth?: string;
|
||||
timeoutMs?: number;
|
||||
/**
|
||||
* Restricts every operation to query ids carrying this prefix, so a caller
|
||||
* cannot kill or inspect another tenant's work through this object.
|
||||
*/
|
||||
queryIdPrefix?: string;
|
||||
}
|
||||
|
||||
export class ClickHouseProcessControl implements EngineProcessControl {
|
||||
readonly engine = 'clickhouse';
|
||||
private readonly baseUrl: string;
|
||||
private readonly authHeader?: string;
|
||||
private readonly timeoutMs: number;
|
||||
private readonly prefix?: string;
|
||||
|
||||
constructor(opts: ClickHouseProcessControlOptions) {
|
||||
this.baseUrl = opts.baseUrl.replace(/\/+$/, '');
|
||||
this.authHeader = opts.basicAuth ? `Basic ${Buffer.from(opts.basicAuth).toString('base64')}` : undefined;
|
||||
this.timeoutMs = opts.timeoutMs ?? 10_000;
|
||||
this.prefix = opts.queryIdPrefix;
|
||||
}
|
||||
|
||||
private assertScoped(queryId: string): void {
|
||||
if (this.prefix && !queryId.startsWith(this.prefix)) {
|
||||
throw new SiaxError('AUTHORIZATION_DENIED', '0pt.analytics', 'query does not belong to this scope');
|
||||
}
|
||||
if (!/^[A-Za-z0-9_-]{1,128}$/.test(queryId)) {
|
||||
// The id reaches ClickHouse as a bound parameter, but a malformed id is
|
||||
// a bug worth surfacing rather than a string worth binding.
|
||||
throw new SiaxError('INVALID_REQUEST', '0pt.analytics', 'malformed query id');
|
||||
}
|
||||
}
|
||||
|
||||
async isRunning(queryId: string): Promise<boolean> {
|
||||
this.assertScoped(queryId);
|
||||
const body = await this.send(
|
||||
'SELECT count() AS n FROM system.processes WHERE query_id = {qid:String} FORMAT JSON',
|
||||
{ param_qid: queryId },
|
||||
);
|
||||
const n = Number((JSON.parse(body) as { data: Array<{ n: string | number }> }).data[0]?.n ?? 0);
|
||||
return n > 0;
|
||||
}
|
||||
|
||||
async requestKill(queryId: string): Promise<boolean> {
|
||||
this.assertScoped(queryId);
|
||||
await this.send(
|
||||
'KILL QUERY WHERE query_id = {qid:String} ASYNC FORMAT JSON',
|
||||
{ param_qid: queryId },
|
||||
false,
|
||||
);
|
||||
return true;
|
||||
}
|
||||
|
||||
async progress(queryId: string): Promise<QueryProgress | undefined> {
|
||||
this.assertScoped(queryId);
|
||||
const body = await this.send(
|
||||
'SELECT read_rows, read_bytes, toUInt64(elapsed * 1000) AS elapsed_ms FROM system.processes ' +
|
||||
'WHERE query_id = {qid:String} FORMAT JSON',
|
||||
{ param_qid: queryId },
|
||||
);
|
||||
const row = (JSON.parse(body) as { data: Array<Record<string, string | number>> }).data[0];
|
||||
if (!row) return undefined;
|
||||
return {
|
||||
rows_read: Number(row.read_rows ?? 0),
|
||||
bytes_read: Number(row.read_bytes ?? 0),
|
||||
elapsed_ms: Number(row.elapsed_ms ?? 0),
|
||||
};
|
||||
}
|
||||
|
||||
private async send(sql: string, params: Record<string, string>, expectJson = true): Promise<string> {
|
||||
const url = new URL(this.baseUrl);
|
||||
url.searchParams.set('query', sql);
|
||||
for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
|
||||
// NOT readonly: KILL mutates server state. The other two would tolerate
|
||||
// readonly, but sharing one path keeps the credential story simple.
|
||||
url.searchParams.set('max_execution_time', '10');
|
||||
const headers: Record<string, string> = {};
|
||||
if (this.authHeader) headers.Authorization = this.authHeader;
|
||||
let res: Response;
|
||||
try {
|
||||
res = await fetch(url.toString(), {
|
||||
method: 'POST',
|
||||
headers,
|
||||
body: '',
|
||||
signal: AbortSignal.timeout(this.timeoutMs),
|
||||
});
|
||||
} catch (cause) {
|
||||
throw new SiaxError('DEPENDENCY_UNAVAILABLE', 'clickhouse', 'clickhouse unreachable', { cause });
|
||||
}
|
||||
const text = await res.text();
|
||||
if (!res.ok) {
|
||||
const code = /Code:\s*(\d+)/.exec(text)?.[1] ?? String(res.status);
|
||||
throw new SiaxError(
|
||||
'PERMANENT_FAILURE',
|
||||
'clickhouse',
|
||||
`process-control statement failed (CH code ${code})`,
|
||||
);
|
||||
}
|
||||
if (expectJson && !text.trim()) {
|
||||
throw new SiaxError('PERMANENT_FAILURE', 'clickhouse', 'empty response from process-control statement');
|
||||
}
|
||||
return text;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,274 @@
|
||||
/**
|
||||
* src/analytics/execution/query-execution-record.ts
|
||||
* ==================================================
|
||||
* BI-P018 — QUERY EXECUTION MODEL.
|
||||
*
|
||||
* Every field BI-P018 lists, verbatim, and one it lists by its absence:
|
||||
*
|
||||
* "Every production query has traceable execution metadata WITHOUT
|
||||
* unnecessarily storing sensitive raw SQL."
|
||||
*
|
||||
* So the record carries `compiled_query_hash` and never the statement. That is
|
||||
* enforced, not documented: `assertNoRawSql` scans a finished record for the
|
||||
* statement it describes, and
|
||||
* tests/analytics/query-execution-model.test.ts fails if any field carries it.
|
||||
*
|
||||
* The ONE place raw SQL is retained is the analyst statement history
|
||||
* (BI-P017), where a privileged human ran unreviewed SQL against tenant data
|
||||
* and "who ran what" has to survive. That history is a separate record with a
|
||||
* separate retention posture; this one is the per-execution telemetry that
|
||||
* every query in the estate produces, and it must be safe to keep for a long
|
||||
* time and to ship to AUD0.
|
||||
*
|
||||
* BI-P002's `QueryExecution` entity is the persisted row; this module is the
|
||||
* lifecycle that produces it, plus the four fields BI-P018 adds to it
|
||||
* (compiled_query_hash, datasets, metrics, cache_status, cost_units).
|
||||
*/
|
||||
import crypto from 'node:crypto';
|
||||
import { SiaxError } from '../../platform/errors';
|
||||
import type { ExecutionState } from '../domain/entities';
|
||||
|
||||
/** BI-P020's outcome, recorded per execution so cache behaviour is measurable. */
|
||||
export const CACHE_STATUSES = ['hit', 'miss', 'bypass', 'stale_refused', 'not_cacheable'] as const;
|
||||
export type CacheStatus = (typeof CACHE_STATUSES)[number];
|
||||
|
||||
/** Which surface produced the query — the "user/workload" axis BI-P018 names. */
|
||||
export const WORKLOAD_KINDS = [
|
||||
'dashboard',
|
||||
'report',
|
||||
'api',
|
||||
'analyst_sql',
|
||||
'ai_analyst',
|
||||
'scheduled',
|
||||
'internal',
|
||||
] as const;
|
||||
export type WorkloadKind = (typeof WORKLOAD_KINDS)[number];
|
||||
|
||||
export interface QueryExecutionRecord {
|
||||
query_id: string;
|
||||
execution_id: string;
|
||||
tenant_id: string;
|
||||
project_id?: string;
|
||||
/** The CL0UD principal. A human, a service, or a job — never absent. */
|
||||
principal_id: string;
|
||||
workload: WorkloadKind;
|
||||
submitted_at: string;
|
||||
started_at?: string;
|
||||
completed_at?: string;
|
||||
engine: string;
|
||||
/**
|
||||
* sha256 of the COMPILED statement plus its bound parameter values.
|
||||
*
|
||||
* Parameters are included deliberately: two executions of the same SQL with
|
||||
* different filters are different work with different cost, and a hash that
|
||||
* collapsed them would make the cost record meaningless. They are HASHED,
|
||||
* not stored, so the record still carries no tenant data.
|
||||
*/
|
||||
compiled_query_hash: string;
|
||||
datasets: string[];
|
||||
metrics: string[];
|
||||
status: ExecutionState;
|
||||
/** Rows RETURNED to the caller. */
|
||||
rows?: number;
|
||||
/** Rows and bytes SCANNED — the cost axis, not the result size. */
|
||||
rows_scanned?: number;
|
||||
bytes_scanned?: number;
|
||||
duration_ms?: number;
|
||||
cache_status: CacheStatus;
|
||||
/** BI-P021 units. Present once the query has settled, including on failure. */
|
||||
cost_units?: number;
|
||||
/** A SiaxErrorCode. Never an engine body, never a fragment of the statement. */
|
||||
error?: string;
|
||||
correlation_id?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Hash a compiled statement and its bound parameters into the one value the
|
||||
* record keeps.
|
||||
*
|
||||
* Parameters are sorted so two logically identical executions hash equal
|
||||
* regardless of key iteration order — otherwise the cache-key and cost
|
||||
* aggregations built on this hash would fragment for no reason.
|
||||
*/
|
||||
export function compiledQueryHash(sql: string, params: Record<string, string> = {}): string {
|
||||
const canonical = JSON.stringify([
|
||||
sql.trim(),
|
||||
Object.keys(params)
|
||||
.sort()
|
||||
.map((k) => [k, params[k]]),
|
||||
]);
|
||||
return crypto.createHash('sha256').update(canonical).digest('hex');
|
||||
}
|
||||
|
||||
/**
|
||||
* The legal state machine. Written down because "status" fields drift into
|
||||
* meaninglessness when any value can follow any other — a `succeeded` row that
|
||||
* later becomes `running` makes every downstream aggregation a lie.
|
||||
*/
|
||||
const LEGAL_TRANSITIONS: Record<ExecutionState, readonly ExecutionState[]> = {
|
||||
queued: ['running', 'cancelled', 'rejected', 'failed'],
|
||||
running: ['succeeded', 'failed', 'cancelled'],
|
||||
succeeded: [],
|
||||
failed: [],
|
||||
cancelled: [],
|
||||
rejected: [],
|
||||
};
|
||||
|
||||
export function assertLegalTransition(from: ExecutionState, to: ExecutionState): void {
|
||||
if (!LEGAL_TRANSITIONS[from].includes(to)) {
|
||||
throw new SiaxError('CONFLICT', '0pt.analytics', `illegal execution transition ${from} -> ${to}`);
|
||||
}
|
||||
}
|
||||
|
||||
/** Terminal states: a record in one of these will never change again. */
|
||||
export const TERMINAL_STATES: readonly ExecutionState[] = ['succeeded', 'failed', 'cancelled', 'rejected'];
|
||||
|
||||
export interface ExecutionSink {
|
||||
write(record: QueryExecutionRecord): Promise<void> | void;
|
||||
}
|
||||
|
||||
export interface ExecutionRecorderOptions {
|
||||
tenant_id: string;
|
||||
project_id?: string;
|
||||
principal_id: string;
|
||||
workload: WorkloadKind;
|
||||
engine: string;
|
||||
correlation_id?: string;
|
||||
sink: ExecutionSink;
|
||||
now?: () => number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Owns one execution's record from submission to a terminal state.
|
||||
*
|
||||
* Every transition WRITES. A recorder that only writes at the end loses
|
||||
* exactly the executions that matter: the ones that never finished.
|
||||
*/
|
||||
export class QueryExecutionRecorder {
|
||||
private record: QueryExecutionRecord;
|
||||
private readonly sink: ExecutionSink;
|
||||
private readonly now: () => number;
|
||||
|
||||
constructor(
|
||||
opts: ExecutionRecorderOptions,
|
||||
queryId: string,
|
||||
compiled: { hash: string; datasets: string[]; metrics: string[] },
|
||||
) {
|
||||
if (!opts.tenant_id?.trim()) {
|
||||
throw new SiaxError(
|
||||
'UNAUTHENTICATED',
|
||||
'0pt.analytics',
|
||||
'an execution record requires a verified tenant',
|
||||
);
|
||||
}
|
||||
if (!opts.principal_id?.trim()) {
|
||||
throw new SiaxError(
|
||||
'UNAUTHENTICATED',
|
||||
'0pt.analytics',
|
||||
'an execution record requires a verified principal',
|
||||
);
|
||||
}
|
||||
this.now = opts.now ?? Date.now;
|
||||
this.sink = opts.sink;
|
||||
this.record = {
|
||||
query_id: queryId,
|
||||
execution_id: crypto.randomUUID(),
|
||||
tenant_id: opts.tenant_id,
|
||||
project_id: opts.project_id,
|
||||
principal_id: opts.principal_id,
|
||||
workload: opts.workload,
|
||||
submitted_at: new Date(this.now()).toISOString(),
|
||||
engine: opts.engine,
|
||||
compiled_query_hash: compiled.hash,
|
||||
datasets: [...compiled.datasets],
|
||||
metrics: [...compiled.metrics],
|
||||
status: 'queued',
|
||||
cache_status: 'miss',
|
||||
correlation_id: opts.correlation_id,
|
||||
};
|
||||
}
|
||||
|
||||
get current(): QueryExecutionRecord {
|
||||
return { ...this.record, datasets: [...this.record.datasets], metrics: [...this.record.metrics] };
|
||||
}
|
||||
|
||||
async submitted(): Promise<void> {
|
||||
await this.sink.write(this.current);
|
||||
}
|
||||
|
||||
async started(): Promise<void> {
|
||||
assertLegalTransition(this.record.status, 'running');
|
||||
this.record.status = 'running';
|
||||
this.record.started_at = new Date(this.now()).toISOString();
|
||||
await this.sink.write(this.current);
|
||||
}
|
||||
|
||||
/** A cache hit never reached the engine; record it as such, not as a fast query. */
|
||||
async servedFromCache(rows: number, costUnits = 0): Promise<void> {
|
||||
assertLegalTransition(this.record.status, 'running');
|
||||
this.record.status = 'running';
|
||||
this.record.started_at = this.record.started_at ?? new Date(this.now()).toISOString();
|
||||
this.record.cache_status = 'hit';
|
||||
await this.finish('succeeded', { rows, rows_scanned: 0, bytes_scanned: 0, cost_units: costUnits });
|
||||
}
|
||||
|
||||
setCacheStatus(status: CacheStatus): void {
|
||||
this.record.cache_status = status;
|
||||
}
|
||||
|
||||
async finish(
|
||||
status: ExecutionState,
|
||||
stats: {
|
||||
rows?: number;
|
||||
rows_scanned?: number;
|
||||
bytes_scanned?: number;
|
||||
cost_units?: number;
|
||||
error?: string;
|
||||
},
|
||||
): Promise<QueryExecutionRecord> {
|
||||
assertLegalTransition(this.record.status, status);
|
||||
const completedMs = this.now();
|
||||
this.record.status = status;
|
||||
this.record.completed_at = new Date(completedMs).toISOString();
|
||||
const startedMs = this.record.started_at
|
||||
? Date.parse(this.record.started_at)
|
||||
: Date.parse(this.record.submitted_at);
|
||||
this.record.duration_ms = Math.max(0, completedMs - startedMs);
|
||||
if (stats.rows !== undefined) this.record.rows = stats.rows;
|
||||
if (stats.rows_scanned !== undefined) this.record.rows_scanned = stats.rows_scanned;
|
||||
if (stats.bytes_scanned !== undefined) this.record.bytes_scanned = stats.bytes_scanned;
|
||||
if (stats.cost_units !== undefined) this.record.cost_units = stats.cost_units;
|
||||
if (stats.error !== undefined) this.record.error = stats.error;
|
||||
assertNoRawSql(this.record);
|
||||
await this.sink.write(this.current);
|
||||
return this.current;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Refuse to emit a record that carries statement text.
|
||||
*
|
||||
* Checked at runtime rather than trusted to the type, because the fields that
|
||||
* leak are the string ones a future change widens — `error` becoming an engine
|
||||
* body is the obvious one, and ClickHouse's error text echoes the query.
|
||||
*/
|
||||
export function assertNoRawSql(record: QueryExecutionRecord): void {
|
||||
const suspicious = /\b(select|insert|update|delete|drop|create|alter|from|where|join|union)\b/i;
|
||||
const stringFields: Array<[string, string | undefined]> = [
|
||||
['error', record.error],
|
||||
['query_id', record.query_id],
|
||||
['compiled_query_hash', record.compiled_query_hash],
|
||||
['correlation_id', record.correlation_id],
|
||||
...record.datasets.map((d, i) => [`datasets[${i}]`, d] as [string, string]),
|
||||
...record.metrics.map((m, i) => [`metrics[${i}]`, m] as [string, string]),
|
||||
];
|
||||
for (const [name, value] of stringFields) {
|
||||
if (value && suspicious.test(value)) {
|
||||
throw new SiaxError(
|
||||
'INVALID_REQUEST',
|
||||
'0pt.analytics',
|
||||
`execution record field "${name}" looks like SQL; BI-P018 stores a hash, never the statement`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user