feat(analytics): BI-P002..BI-P012 analytics core — domain model, tiers, contracts, Arrow, ClickHouse engine, isolation, governance
CI (SIAX Cloud) / security (push) Successful in 41s
Security Scan (SIAX Cloud) / security (push) Successful in 40s
CI (SIAX Cloud) / integration (push) Successful in 52s
CI (SIAX Cloud) / quality (push) Successful in 1m5s
CI (SIAX Cloud) / security (pull_request) Successful in 31s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 29s
CI (SIAX Cloud) / integration (pull_request) Successful in 45s
CI (SIAX Cloud) / quality (pull_request) Successful in 53s
CI (SIAX Cloud) / security (push) Successful in 41s
Security Scan (SIAX Cloud) / security (push) Successful in 40s
CI (SIAX Cloud) / integration (push) Successful in 52s
CI (SIAX Cloud) / quality (push) Successful in 1m5s
CI (SIAX Cloud) / security (pull_request) Successful in 31s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 29s
CI (SIAX Cloud) / integration (pull_request) Successful in 45s
CI (SIAX Cloud) / quality (pull_request) Successful in 53s
Implements the eight BI program prompts that make 0pt the canonical SIAX
analytics capability. Every file is NEW: not one existing file is modified,
and src/analytics/clickhouse-analytics.ts is byte-identical to main.
BI-P002 — the nineteen canonical entities, verbatim, with tenant_id +
project_id on every tenant-owned record. sql/008 gives each one a table.
The four version entities BI-P002 names as immutable are enforced by BEFORE
UPDATE/DELETE triggers, not by a TypeScript `readonly` — proven by direct SQL
against a live Postgres.
BI-P003 — RAW/NORMALIZED/CURATED/SERVING with the read rule enforced:
assertReadableByBiSurface refuses a RAW dataset to a dashboard unless a
recorded exception exists. All eight required declarations are NOT NULL.
BI-P004 — DatasetContract and a fail-closed compatibility gate. Fifteen
breaking-change classes are each tested twice: refused under a minor bump,
accepted under a major one.
BI-P005 — Arrow IPC alongside JSON, with a hand-written FlatBuffers writer so
no npm dependency (or CVE surface) is added. Conformance comes from three
independent implementations, not from self-agreement: pyarrow 25.0.1 and
DuckDB 1.5.5 read identical values from both ClickHouse's Arrow and ours, and
ClickHouse's own Arrow C++ ingests our 20,000-row batch with exact aggregates.
BI-P009 — SiaxAnalyticsEngine with all eight capabilities. The existing
ClickHouseAnalytics is COMPOSED, never replaced. executeQuery (analyst SQL) is
declared and deliberately refused with CAPABILITY_UNAVAILABLE until BI-P017 and
BI-P033 land, rather than opening the raw-SQL hole BI-P011 enumerates.
BI-P010 — four purpose-built tables, not one giant one, and the difference is
measured against the counterfactual: 48.8ms/40,960 rows vs 174.4ms/5,400,000
rows for the same question; 5/660 granules vs 660/1149.
BI-P011 — layered isolation, each layer proven with the layer above removed.
Without the forced predicate the same statement returns 50,050 (both tenants);
with it, t_alpha gets 50. Without a row policy an unrestricted user sees 2
tenants; the restricted user running the IDENTICAL statement sees 1.
BI-P012 — every named limit, fired individually and identified by ClickHouse
error code, plus the half usually skipped: an unrelated tenant is timed while
a real CPU bomb is in flight and must still return the CORRECT answer.
Three defects found by writing tests that can fail:
* `CHECK (array_length(purposes,1) >= 1)` can NEVER fail — array_length is
NULL for '{}' and a NULL CHECK passes. Fixed with coalesce(); the test
that caught it is pinned.
* EXPLAIN ESTIMATE run under the caller's own max_rows_to_read fails with
code 158 exactly when the estimate would be over budget. Planner calls now
use a narrower envelope.
* result_overflow_mode=break returns HTTP 200 with 65,409 of 100,000 rows
and no indication it is short. Every SELECT ships `throw`.
Verified rather than assumed: server7's ClickHouse holds `opt_scraped` (51
tables, 407,793,471 rows, `persons`/`cz_owners` included) — the b0ts plane this
module is prohibited from touching — has no `opt_analytics` database, zero
tenant_id columns and zero row policies. Nothing there was modified; every
proof ran against disposable containers.
Evidence: evidence/bi-p002-p012/{test-transcript.txt,bi-p010-benchmark.json,
bi-p005-arrow-roundtrip.json,server7-clickhouse-probe.md}
Docs: docs/analytics/BI_ANALYTICS_CORE.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,563 @@
|
||||
/**
|
||||
* src/analytics/engine/clickhouse-engine.ts
|
||||
* ==========================================
|
||||
* BI-P009 — ClickHouse as the CANONICAL `SiaxAnalyticsEngine` provider.
|
||||
*
|
||||
* PRESERVATION (BI-P009's first sentence): `ClickHouseAnalytics`
|
||||
* (src/analytics/clickhouse-analytics.ts) is untouched by this change — not
|
||||
* one line. This engine COMPOSES it:
|
||||
*
|
||||
* * `health()` delegates to `ClickHouseAnalytics.health()`.
|
||||
* * `executeMetric()` on the pre-existing `opt_metric_events` table
|
||||
* delegates to `ClickHouseAnalytics.queryMetric()`, i.e. keeps the
|
||||
* already-proven forced-tenant path exactly as it is.
|
||||
* * everything else (the BI-P010 tables) goes through
|
||||
* `src/analytics/engine/query-compiler.ts`, which forces the same
|
||||
* predicate for the same reason.
|
||||
*
|
||||
* "Do not regress the existing forced tenant predicate" is enforced by
|
||||
* tests/analytics/tenant-predicate-regression.test.ts, which fails if either
|
||||
* compiler stops emitting `tenant_id = {tenant:String}`.
|
||||
*
|
||||
* BI-P012 governance is applied on EVERY statement, including the delegated
|
||||
* ones — the settings ride on the HTTP request, and `readonly=1` means the
|
||||
* query text cannot re-set them.
|
||||
*
|
||||
* LEGAL BOUNDARY (inherited): generic 0pt analytics only. Never the b0ts
|
||||
* ClickHouse pipelines, never quarantined market data.
|
||||
*/
|
||||
import crypto from "node:crypto";
|
||||
import { SiaxError, siaxErrorCodeForUpstreamStatus } from "../../platform/errors";
|
||||
import { ClickHouseAnalytics } from "../clickhouse-analytics";
|
||||
import {
|
||||
compileMetricQuery,
|
||||
compileProjectionQuery,
|
||||
dimensionCatalogFrom,
|
||||
FORCED_TENANT_PREDICATE,
|
||||
type DimensionCatalog,
|
||||
} from "./query-compiler";
|
||||
import { analyticsSchemaDdl, type AnalyticsSchemaOptions } from "./schema";
|
||||
import {
|
||||
applyQuota,
|
||||
clickhouseSettings,
|
||||
ConcurrencyGate,
|
||||
DEFAULT_BUDGETS,
|
||||
planningSettings,
|
||||
type ResourceBudget,
|
||||
} from "./resource-governance";
|
||||
import type {
|
||||
AnalystQueryRequest,
|
||||
AnalyticsIdentity,
|
||||
EngineResult,
|
||||
HealthReport,
|
||||
MetricRequest,
|
||||
ProjectionIngestRequest,
|
||||
ProjectionRefreshResult,
|
||||
QueryEstimate,
|
||||
QueryExplanation,
|
||||
SiaxAnalyticsEngine,
|
||||
} from "./siax-analytics-engine";
|
||||
|
||||
export interface ClickHouseEngineOptions {
|
||||
baseUrl: string;
|
||||
/** "user:password" — from an env/secret ref, never a literal in code. */
|
||||
basicAuth?: string;
|
||||
database?: string;
|
||||
timeoutMs?: number;
|
||||
/** Dimension allow-list per tenant. Absent tenant = no dimensions allowed. */
|
||||
dimensionCatalogs?: Record<string, Record<string, string>>;
|
||||
schema?: Omit<AnalyticsSchemaOptions, "database">;
|
||||
}
|
||||
|
||||
const EMPTY_CATALOG: DimensionCatalog = { resolve: () => undefined };
|
||||
|
||||
export class ClickHouseAnalyticsEngine implements SiaxAnalyticsEngine {
|
||||
readonly provider = "clickhouse";
|
||||
|
||||
/** PRESERVED pre-existing implementation, composed not replaced. */
|
||||
private readonly legacy: ClickHouseAnalytics;
|
||||
private readonly baseUrl: string;
|
||||
private readonly authHeader?: string;
|
||||
private readonly database: string;
|
||||
private readonly timeoutMs: number;
|
||||
private readonly dimensionCatalogs: Record<string, Record<string, string>>;
|
||||
private readonly schemaOpts: Omit<AnalyticsSchemaOptions, "database">;
|
||||
private readonly gate: ConcurrencyGate;
|
||||
/** query_id -> AbortController, so cancelQuery can stop the client side too. */
|
||||
private readonly inFlight = new Map<string, AbortController>();
|
||||
|
||||
constructor(opts: ClickHouseEngineOptions) {
|
||||
this.baseUrl = opts.baseUrl.replace(/\/+$/, "");
|
||||
this.authHeader = opts.basicAuth ? `Basic ${Buffer.from(opts.basicAuth).toString("base64")}` : undefined;
|
||||
this.database = opts.database ?? "opt_analytics";
|
||||
this.timeoutMs = opts.timeoutMs ?? 30_000;
|
||||
this.dimensionCatalogs = opts.dimensionCatalogs ?? {};
|
||||
this.schemaOpts = opts.schema ?? {};
|
||||
this.legacy = new ClickHouseAnalytics({
|
||||
baseUrl: opts.baseUrl,
|
||||
basicAuth: opts.basicAuth,
|
||||
database: this.database,
|
||||
timeoutMs: this.timeoutMs,
|
||||
});
|
||||
this.gate = new ConcurrencyGate((tenantId) => this.budgetForTenant(tenantId).maxConcurrentQueries);
|
||||
}
|
||||
|
||||
/** Access to the preserved implementation, for callers that still use it. */
|
||||
get legacyAnalytics(): ClickHouseAnalytics {
|
||||
return this.legacy;
|
||||
}
|
||||
|
||||
private budgetForTenant(_tenantId: string): ResourceBudget {
|
||||
return DEFAULT_BUDGETS.INTERACTIVE;
|
||||
}
|
||||
|
||||
private budgetFor(identity: AnalyticsIdentity): ResourceBudget {
|
||||
const tier = identity.budget_tier ?? "INTERACTIVE";
|
||||
return applyQuota(DEFAULT_BUDGETS[tier], identity.quota);
|
||||
}
|
||||
|
||||
private catalogFor(identity: AnalyticsIdentity): DimensionCatalog {
|
||||
const pairs = this.dimensionCatalogs[identity.tenant_id];
|
||||
return pairs ? dimensionCatalogFrom(pairs) : EMPTY_CATALOG;
|
||||
}
|
||||
|
||||
/** Fail closed: a missing/blank verified tenant is never a wildcard. */
|
||||
private assertIdentity(identity: AnalyticsIdentity): void {
|
||||
if (!identity?.tenant_id || !identity.tenant_id.trim()) {
|
||||
throw new SiaxError(
|
||||
"UNAUTHENTICATED",
|
||||
"0pt.analytics",
|
||||
"analytics requires a verified CL0UD identity carrying tenant_id",
|
||||
);
|
||||
}
|
||||
if (!identity.principal_id || !identity.principal_id.trim()) {
|
||||
throw new SiaxError("UNAUTHENTICATED", "0pt.analytics", "analytics requires a verified principal_id");
|
||||
}
|
||||
}
|
||||
|
||||
// ── schema ───────────────────────────────────────────────────────────────
|
||||
|
||||
/** Apply BI-P010 DDL (idempotent) plus the preserved legacy table. */
|
||||
async ensureSchema(): Promise<void> {
|
||||
await this.legacy.ensureSchema();
|
||||
for (const stmt of analyticsSchemaDdl({ database: this.database, ...this.schemaOpts })) {
|
||||
await this.ddl(stmt);
|
||||
}
|
||||
}
|
||||
|
||||
// ── BI-P009 capability 1: health ─────────────────────────────────────────
|
||||
|
||||
async health(): Promise<HealthReport> {
|
||||
try {
|
||||
const legacyOk = await this.legacy.health();
|
||||
if (!legacyOk) return { healthy: false, provider: this.provider, detail: "legacy health probe returned false" };
|
||||
const v = await this.raw("SELECT version() AS v FORMAT JSON", {}, {});
|
||||
const version = (v as { data?: { v?: string }[] })?.data?.[0]?.v;
|
||||
return { healthy: true, provider: this.provider, version };
|
||||
} catch (err) {
|
||||
return {
|
||||
healthy: false,
|
||||
provider: this.provider,
|
||||
detail: err instanceof SiaxError ? err.code : "unreachable",
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
// ── BI-P009 capability 2: executeMetric ──────────────────────────────────
|
||||
|
||||
async executeMetric(identity: AnalyticsIdentity, request: MetricRequest): Promise<EngineResult> {
|
||||
this.assertIdentity(identity);
|
||||
const budget = this.budgetFor(identity);
|
||||
const release = this.gate.acquire(identity.tenant_id);
|
||||
const started = Date.now();
|
||||
const queryId = this.newQueryId(identity);
|
||||
try {
|
||||
const compiled = compileMetricQuery(
|
||||
{
|
||||
tenant_id: identity.tenant_id, // FORCED from the verified identity
|
||||
project_id: identity.project_id,
|
||||
dataset_id: request.dataset_id,
|
||||
metric: request.metric,
|
||||
aggregate: request.aggregate,
|
||||
group_by: request.group_by,
|
||||
filters: request.filters,
|
||||
from_time: request.from_time,
|
||||
to_time: request.to_time,
|
||||
limit: request.limit,
|
||||
table: request.table,
|
||||
},
|
||||
this.database,
|
||||
this.catalogFor(identity),
|
||||
budget.maxResultRows,
|
||||
);
|
||||
const json = (await this.raw(compiled.sql, compiled.params, clickhouseSettings(budget), queryId)) as
|
||||
| ClickHouseJsonResponse
|
||||
| undefined;
|
||||
const groupNames = request.group_by ?? [];
|
||||
const rows = (json?.data ?? []).map((r) => {
|
||||
const groups: Record<string, string> = {};
|
||||
groupNames.forEach((name, i) => {
|
||||
const v = (r as Record<string, unknown>)[`grp${i}`];
|
||||
groups[name] = v === undefined || v === null ? "" : String(v);
|
||||
});
|
||||
return {
|
||||
...(groupNames.length ? { groups } : {}),
|
||||
value: Number((r as { value: number | string }).value),
|
||||
};
|
||||
});
|
||||
const limit = Math.max(1, Math.min(Math.floor(request.limit ?? 1000), budget.maxResultRows));
|
||||
return {
|
||||
rows,
|
||||
query_id: queryId,
|
||||
truncated: rows.length >= limit,
|
||||
stats: {
|
||||
duration_ms: Date.now() - started,
|
||||
rows_read: json?.statistics?.rows_read,
|
||||
bytes_read: json?.statistics?.bytes_read,
|
||||
result_rows: rows.length,
|
||||
},
|
||||
};
|
||||
} finally {
|
||||
release();
|
||||
this.inFlight.delete(queryId);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Delegating path to the PRESERVED implementation for the pre-existing
|
||||
* `opt_metric_events` table. Kept so no existing caller has to migrate and
|
||||
* the already-proven forced-predicate path stays in service.
|
||||
*/
|
||||
async executeLegacyMetric(
|
||||
identity: AnalyticsIdentity,
|
||||
request: { metric: string; aggregate: "sum" | "avg" | "min" | "max" | "count"; dataset_id?: string; group_by?: string; from_time?: string; to_time?: string; limit?: number },
|
||||
): Promise<EngineResult> {
|
||||
this.assertIdentity(identity);
|
||||
const started = Date.now();
|
||||
const rows = await this.legacy.queryMetric({
|
||||
tenant_id: identity.tenant_id, // FORCED from the verified identity
|
||||
metric: request.metric,
|
||||
aggregate: request.aggregate,
|
||||
dataset_id: request.dataset_id,
|
||||
group_by: request.group_by,
|
||||
from_time: request.from_time,
|
||||
to_time: request.to_time,
|
||||
limit: request.limit,
|
||||
});
|
||||
return {
|
||||
rows: rows.map((r) => ({
|
||||
...(r.group !== undefined && request.group_by ? { groups: { [request.group_by]: r.group } } : {}),
|
||||
value: r.value,
|
||||
})),
|
||||
query_id: this.newQueryId(identity),
|
||||
truncated: false,
|
||||
stats: { duration_ms: Date.now() - started, result_rows: rows.length },
|
||||
};
|
||||
}
|
||||
|
||||
// ── BI-P009 capability 3: executeQuery (analyst SQL) ─────────────────────
|
||||
|
||||
async executeQuery(identity: AnalyticsIdentity, _request: AnalystQueryRequest): Promise<EngineResult> {
|
||||
this.assertIdentity(identity);
|
||||
// Deliberate, documented refusal — see AnalystQueryRequest's doc comment.
|
||||
// Opening a raw-SQL surface before BI-P017 (SQL dialect contract) and
|
||||
// BI-P033 (analytics authorization) exist would create precisely the
|
||||
// "raw SQL attempt" hole BI-P011's adversarial list enumerates.
|
||||
throw new SiaxError(
|
||||
"CAPABILITY_UNAVAILABLE",
|
||||
"0pt.analytics",
|
||||
"analyst SQL is not enabled on the ClickHouse provider until BI-P017/BI-P033 land",
|
||||
);
|
||||
}
|
||||
|
||||
// ── BI-P009 capability 4: estimateQuery ──────────────────────────────────
|
||||
|
||||
async estimateQuery(identity: AnalyticsIdentity, request: MetricRequest): Promise<QueryEstimate> {
|
||||
this.assertIdentity(identity);
|
||||
const budget = this.budgetFor(identity);
|
||||
const compiled = compileMetricQuery(
|
||||
{
|
||||
tenant_id: identity.tenant_id,
|
||||
project_id: identity.project_id,
|
||||
dataset_id: request.dataset_id,
|
||||
metric: request.metric,
|
||||
aggregate: request.aggregate,
|
||||
group_by: request.group_by,
|
||||
filters: request.filters,
|
||||
from_time: request.from_time,
|
||||
to_time: request.to_time,
|
||||
limit: request.limit,
|
||||
table: request.table,
|
||||
},
|
||||
this.database,
|
||||
this.catalogFor(identity),
|
||||
budget.maxResultRows,
|
||||
);
|
||||
// `EXPLAIN ESTIMATE` reports the parts/rows/marks the planner will read
|
||||
// AFTER primary-key pruning — which is why tenant_id leads every ORDER BY
|
||||
// in schema.ts: the estimate for one tenant does not include everyone.
|
||||
const estimateSql = compiled.sql.replace(/ FORMAT JSON$/, "");
|
||||
let rowsToRead = 0;
|
||||
let partsRead = 0;
|
||||
try {
|
||||
const res = (await this.raw(
|
||||
`EXPLAIN ESTIMATE ${estimateSql} FORMAT JSON`,
|
||||
compiled.params,
|
||||
planningSettings(budget),
|
||||
)) as { data?: { rows?: number | string; parts?: number | string }[] } | undefined;
|
||||
for (const row of res?.data ?? []) {
|
||||
rowsToRead += Number(row.rows ?? 0);
|
||||
partsRead += Number(row.parts ?? 0);
|
||||
}
|
||||
} catch (err) {
|
||||
// An estimate that cannot be produced must not read as "cheap".
|
||||
throw err instanceof SiaxError
|
||||
? err
|
||||
: new SiaxError("DEPENDENCY_UNAVAILABLE", "clickhouse", "could not estimate query");
|
||||
}
|
||||
// Rough byte estimate: ClickHouse's ESTIMATE reports rows, not bytes.
|
||||
// 128 B/row is a deliberately PESSIMISTIC constant for these tables
|
||||
// (measured mean in the BI-P010 benchmark is lower), so the gate errs
|
||||
// toward refusing rather than admitting an oversized query.
|
||||
const bytesToRead = rowsToRead * 128;
|
||||
const within = rowsToRead <= budget.maxRowsToRead && bytesToRead <= budget.maxBytesToRead;
|
||||
return {
|
||||
estimated_rows_to_read: rowsToRead,
|
||||
estimated_bytes_to_read: bytesToRead,
|
||||
within_budget: within,
|
||||
budget,
|
||||
...(within
|
||||
? {}
|
||||
: {
|
||||
reason:
|
||||
`estimate ${rowsToRead} rows / ${bytesToRead} bytes over budget ` +
|
||||
`(${budget.maxRowsToRead} rows / ${budget.maxBytesToRead} bytes), parts=${partsRead}`,
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
// ── BI-P009 capability 5: cancelQuery ────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Two-sided cancellation. The client-side abort stops us waiting; the
|
||||
* server-side KILL QUERY stops ClickHouse burning CPU. Cancelling is
|
||||
* tenant-scoped: the query id embeds the tenant and is checked, so tenant A
|
||||
* cannot cancel tenant B's work.
|
||||
*/
|
||||
async cancelQuery(identity: AnalyticsIdentity, queryId: string): Promise<boolean> {
|
||||
this.assertIdentity(identity);
|
||||
if (!queryId.startsWith(this.queryIdPrefix(identity))) {
|
||||
throw new SiaxError("AUTHORIZATION_DENIED", "0pt.analytics", "query does not belong to this tenant");
|
||||
}
|
||||
this.inFlight.get(queryId)?.abort();
|
||||
this.inFlight.delete(queryId);
|
||||
try {
|
||||
// KILL QUERY is a mutation of server state, so it cannot ride the
|
||||
// readonly=1 settings the SELECT path uses.
|
||||
await this.raw(
|
||||
`KILL QUERY WHERE query_id = {killId:String} SYNC FORMAT JSON`,
|
||||
{ param_killId: queryId },
|
||||
{ max_execution_time: "10" },
|
||||
);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
// ── BI-P009 capability 6: explain ────────────────────────────────────────
|
||||
|
||||
async explain(identity: AnalyticsIdentity, request: MetricRequest): Promise<QueryExplanation> {
|
||||
this.assertIdentity(identity);
|
||||
const budget = this.budgetFor(identity);
|
||||
const compiled = compileMetricQuery(
|
||||
{
|
||||
tenant_id: identity.tenant_id,
|
||||
project_id: identity.project_id,
|
||||
dataset_id: request.dataset_id,
|
||||
metric: request.metric,
|
||||
aggregate: request.aggregate,
|
||||
group_by: request.group_by,
|
||||
filters: request.filters,
|
||||
from_time: request.from_time,
|
||||
to_time: request.to_time,
|
||||
limit: request.limit,
|
||||
table: request.table,
|
||||
},
|
||||
this.database,
|
||||
this.catalogFor(identity),
|
||||
budget.maxResultRows,
|
||||
);
|
||||
const sql = compiled.sql.replace(/ FORMAT JSON$/, "");
|
||||
const res = (await this.raw(
|
||||
`EXPLAIN PLAN indexes = 1 ${sql} FORMAT TSVRaw`,
|
||||
compiled.params,
|
||||
planningSettings(budget),
|
||||
)) as string | undefined;
|
||||
return {
|
||||
plan: typeof res === "string" ? res : JSON.stringify(res ?? ""),
|
||||
forced_tenant_predicate: FORCED_TENANT_PREDICATE,
|
||||
table: compiled.table,
|
||||
};
|
||||
}
|
||||
|
||||
// ── BI-P009 capability 7: ingestProjection ───────────────────────────────
|
||||
|
||||
async ingestProjection(
|
||||
identity: AnalyticsIdentity,
|
||||
request: ProjectionIngestRequest,
|
||||
): Promise<{ ingested: number }> {
|
||||
this.assertIdentity(identity);
|
||||
if (request.rows.length === 0) return { ingested: 0 };
|
||||
const now = new Date();
|
||||
const version = Math.floor(now.getTime());
|
||||
const body = request.rows
|
||||
.map((r) => {
|
||||
if (!r.entity_key || !r.entity_key.trim()) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "every projection row needs an entity_key");
|
||||
}
|
||||
return JSON.stringify({
|
||||
// tenant/project come from the VERIFIED identity, never the row —
|
||||
// a row-supplied tenant_id is the INF0 Wave-2 bug class.
|
||||
tenant_id: identity.tenant_id,
|
||||
project_id: identity.project_id ?? "",
|
||||
dataset_id: request.dataset_id,
|
||||
entity_key: r.entity_key,
|
||||
as_of: (r.as_of ?? now.toISOString()).replace("T", " ").replace(/Z$/, ""),
|
||||
version: r.version ?? version,
|
||||
attributes: r.attributes ?? {},
|
||||
measures: r.measures ?? {},
|
||||
});
|
||||
})
|
||||
.join("\n");
|
||||
await this.raw(
|
||||
`INSERT INTO ${this.database}.opt_dataset_projection FORMAT JSONEachRow`,
|
||||
{},
|
||||
{},
|
||||
undefined,
|
||||
body,
|
||||
);
|
||||
return { ingested: request.rows.length };
|
||||
}
|
||||
|
||||
// ── BI-P009 capability 8: refreshProjection ──────────────────────────────
|
||||
|
||||
/**
|
||||
* ReplacingMergeTree deduplicates on a background merge, so a read straight
|
||||
* after an ingest can still see superseded rows. `OPTIMIZE ... FINAL` forces
|
||||
* the merge for THIS tenant's partition only — the table is
|
||||
* `PARTITION BY (tenant_id)` precisely so one tenant's refresh does not
|
||||
* rewrite another tenant's parts (BI-P012's "without degrading unrelated
|
||||
* tenant" applies to maintenance too, not only to queries).
|
||||
*/
|
||||
async refreshProjection(identity: AnalyticsIdentity, dataset_id: string): Promise<ProjectionRefreshResult> {
|
||||
this.assertIdentity(identity);
|
||||
await this.raw(
|
||||
`OPTIMIZE TABLE ${this.database}.opt_dataset_projection PARTITION {tenant:String} FINAL`,
|
||||
{ param_tenant: identity.tenant_id },
|
||||
{ max_execution_time: "120" },
|
||||
);
|
||||
const compiled = compileProjectionQuery(
|
||||
{ tenant_id: identity.tenant_id, project_id: identity.project_id, dataset_id, limit: 1 },
|
||||
this.database,
|
||||
1,
|
||||
);
|
||||
const countSql =
|
||||
`SELECT count() AS value FROM ${this.database}.opt_dataset_projection FINAL ` +
|
||||
`WHERE ${FORCED_TENANT_PREDICATE} AND dataset_id = {dataset:String} FORMAT JSON`;
|
||||
const res = (await this.raw(countSql, compiled.params, clickhouseSettings(this.budgetFor(identity)))) as
|
||||
| { data?: { value: number | string }[] }
|
||||
| undefined;
|
||||
return {
|
||||
dataset_id,
|
||||
row_count: Number(res?.data?.[0]?.value ?? 0),
|
||||
refreshed_at: new Date().toISOString(),
|
||||
};
|
||||
}
|
||||
|
||||
// ── transport ────────────────────────────────────────────────────────────
|
||||
|
||||
private queryIdPrefix(identity: AnalyticsIdentity): string {
|
||||
return `opt_${crypto.createHash("sha256").update(identity.tenant_id).digest("hex").slice(0, 12)}_`;
|
||||
}
|
||||
|
||||
private newQueryId(identity: AnalyticsIdentity): string {
|
||||
return `${this.queryIdPrefix(identity)}${crypto.randomUUID().replace(/-/g, "")}`;
|
||||
}
|
||||
|
||||
/** DDL path: no readonly, no result parsing beyond errors. */
|
||||
private async ddl(sql: string): Promise<void> {
|
||||
await this.raw(sql, {}, { max_execution_time: "60" });
|
||||
}
|
||||
|
||||
private async raw(
|
||||
sql: string,
|
||||
params: Record<string, string>,
|
||||
settings: Record<string, string>,
|
||||
queryId?: string,
|
||||
body?: string,
|
||||
): Promise<unknown> {
|
||||
const url = new URL(this.baseUrl);
|
||||
url.searchParams.set("query", sql);
|
||||
for (const [k, v] of Object.entries(settings)) url.searchParams.set(k, v);
|
||||
for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
|
||||
if (queryId) url.searchParams.set("query_id", queryId);
|
||||
|
||||
const headers: Record<string, string> = {};
|
||||
if (this.authHeader) headers["Authorization"] = this.authHeader;
|
||||
|
||||
const controller = new AbortController();
|
||||
if (queryId) this.inFlight.set(queryId, controller);
|
||||
const timeout = setTimeout(() => controller.abort(), this.timeoutMs);
|
||||
|
||||
let res: Response;
|
||||
try {
|
||||
res = await fetch(url.toString(), {
|
||||
method: "POST",
|
||||
headers,
|
||||
body: body ?? "",
|
||||
signal: controller.signal,
|
||||
});
|
||||
} catch (cause) {
|
||||
throw new SiaxError("DEPENDENCY_UNAVAILABLE", "clickhouse", "clickhouse unreachable", { cause });
|
||||
} finally {
|
||||
clearTimeout(timeout);
|
||||
}
|
||||
const text = await res.text();
|
||||
if (!res.ok) {
|
||||
// ClickHouse error text can echo the query; keep only the numeric code
|
||||
// so a caller never receives another tenant's identifiers back.
|
||||
const code = /Code: (\d+)/.exec(text)?.[1] ?? String(res.status);
|
||||
throw new SiaxError(
|
||||
siaxErrorCodeForUpstreamStatus(res.status),
|
||||
"clickhouse",
|
||||
`query failed (CH code ${code})`,
|
||||
);
|
||||
}
|
||||
if (!text) return undefined;
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(text);
|
||||
} catch {
|
||||
return text;
|
||||
}
|
||||
// DEFENCE IN DEPTH against a partial result that looks complete.
|
||||
// ClickHouse normally answers a breached limit with an HTTP error status
|
||||
// (measured on 24.8: 408 for max_execution_time, 500 for
|
||||
// max_rows_to_read), which `res.ok` above already catches. But when rows
|
||||
// have already been flushed, the JSON body can carry an `exception` key
|
||||
// under a 200 — a truncated answer indistinguishable from a whole one.
|
||||
// Treat any body carrying an exception as a failure, whatever the status.
|
||||
if (parsed && typeof parsed === "object" && "exception" in parsed) {
|
||||
const ex = String((parsed as { exception?: unknown }).exception ?? "");
|
||||
const code = /Code: (\d+)/.exec(ex)?.[1] ?? "unknown";
|
||||
throw new SiaxError("PERMANENT_FAILURE", "clickhouse", `query failed mid-stream (CH code ${code})`);
|
||||
}
|
||||
return parsed;
|
||||
}
|
||||
}
|
||||
|
||||
interface ClickHouseJsonResponse {
|
||||
data?: Record<string, unknown>[];
|
||||
statistics?: { rows_read?: number; bytes_read?: number; elapsed?: number };
|
||||
}
|
||||
@@ -0,0 +1,304 @@
|
||||
/**
|
||||
* src/analytics/engine/query-compiler.ts
|
||||
* =======================================
|
||||
* BI-P011 — CLICKHOUSE TENANT ISOLATION, layer 2:
|
||||
* "`0pt` query compiler forces tenant".
|
||||
*
|
||||
* The compiler is the ONLY thing in this repo that turns an analytics request
|
||||
* into SQL text. Its contract:
|
||||
*
|
||||
* * `tenant_id` is not an argument the caller may omit — it comes from the
|
||||
* VERIFIED CL0UD identity (layer 1) and is written into the WHERE clause
|
||||
* by this function, unconditionally, as a bound parameter.
|
||||
* * caller-supplied VALUES are never spliced into SQL text. They travel as
|
||||
* ClickHouse HTTP `param_*` query parameters.
|
||||
* * caller-supplied IDENTIFIERS (a dimension name, a group-by column) are
|
||||
* never spliced either. They are RESOLVED against a declared allow-list
|
||||
* of `DimensionDefinition`s and only the resolved, already-known physical
|
||||
* name reaches the SQL — so "dimension identifier injection" has no
|
||||
* surface, because caller text never becomes an identifier.
|
||||
* * there is no raw-SQL entry point. `compileMetricQuery` cannot express a
|
||||
* UNION, a subquery, or a second table, so a caller cannot reach a table
|
||||
* the compiler did not choose.
|
||||
*
|
||||
* WHY A SEPARATE MODULE from clickhouse-analytics.ts: that file's compiler
|
||||
* already forces the tenant predicate correctly and BI-P009 says PRESERVE IT.
|
||||
* This module is the generalised compiler for the BI-P010 tables; it repeats
|
||||
* the guarantee rather than replacing it, and
|
||||
* tests/analytics/tenant-predicate-regression.test.ts asserts BOTH.
|
||||
*/
|
||||
import { SiaxError } from "../../platform/errors";
|
||||
import type { AnalyticsTable } from "./schema";
|
||||
|
||||
/** A physical identifier that has been VALIDATED, not merely received. */
|
||||
const IDENT_RE = /^[A-Za-z_][A-Za-z0-9_]{0,62}$/;
|
||||
|
||||
export function assertSafeIdentifier(value: string, what: string): string {
|
||||
if (!IDENT_RE.test(value)) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", `invalid ${what} identifier`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
export const METRIC_AGGREGATES = ["sum", "avg", "min", "max", "count", "count_distinct"] as const;
|
||||
export type MetricAggregate = (typeof METRIC_AGGREGATES)[number];
|
||||
|
||||
/**
|
||||
* The dimensions a caller is allowed to name, resolved from
|
||||
* `DimensionDefinition` rows. The compiler looks a caller's dimension name up
|
||||
* HERE; a miss is a rejection, never a pass-through.
|
||||
*/
|
||||
export interface DimensionCatalog {
|
||||
/** caller-facing dimension name -> physical dimension key in dim_names. */
|
||||
resolve(name: string): string | undefined;
|
||||
}
|
||||
|
||||
export function dimensionCatalogFrom(pairs: Record<string, string>): DimensionCatalog {
|
||||
const map = new Map(Object.entries(pairs));
|
||||
return {
|
||||
resolve(name: string): string | undefined {
|
||||
// Own-property lookup on a Map — no prototype-chain surprises
|
||||
// ("constructor", "__proto__" resolve to undefined, as they must).
|
||||
return map.get(name);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export interface MetricQueryRequest {
|
||||
/** Supplied by the engine from the verified identity, NOT by the caller. */
|
||||
tenant_id: string;
|
||||
project_id?: string;
|
||||
dataset_id?: string;
|
||||
metric: string;
|
||||
aggregate: MetricAggregate;
|
||||
/** Caller-facing dimension names; resolved through the catalog. */
|
||||
group_by?: string[];
|
||||
/** Equality filters on dimensions: caller-facing name -> value. */
|
||||
filters?: Record<string, string>;
|
||||
from_time?: string;
|
||||
to_time?: string;
|
||||
limit?: number;
|
||||
/** Which BI-P010 table to read. Chosen by the engine, not the caller. */
|
||||
table?: AnalyticsTable;
|
||||
}
|
||||
|
||||
export interface CompiledQuery {
|
||||
sql: string;
|
||||
/** ClickHouse HTTP `param_*` entries. Values NEVER appear in `sql`. */
|
||||
params: Record<string, string>;
|
||||
/** The predicate the compiler forced, for assertion in tests. */
|
||||
forcedTenantPredicate: string;
|
||||
table: AnalyticsTable;
|
||||
}
|
||||
|
||||
/**
|
||||
* The exact predicate text every compiled query must contain. Exported so a
|
||||
* regression test can assert on the literal string — if someone deletes the
|
||||
* forcing, the test fails on a missing substring rather than on a subtle
|
||||
* behavioural difference that a mocked engine would happily hide.
|
||||
*/
|
||||
export const FORCED_TENANT_PREDICATE = "tenant_id = {tenant:String}";
|
||||
|
||||
const AGG_SQL: Record<MetricAggregate, string> = {
|
||||
sum: "sum(value)",
|
||||
avg: "avg(value)",
|
||||
min: "min(value)",
|
||||
max: "max(value)",
|
||||
count: "count()",
|
||||
count_distinct: "uniqExact(event_id)",
|
||||
};
|
||||
|
||||
const MAX_GROUP_BY = 8;
|
||||
|
||||
/**
|
||||
* Compile a closed-DSL metric request into tenant-forced SQL.
|
||||
*
|
||||
* @param req the request; `tenant_id` MUST already be the verified identity.
|
||||
* @param db the analytics database name (engine-controlled, validated).
|
||||
* @param catalog the tenant's declared dimensions.
|
||||
* @param maxResultRows the governance ceiling (BI-P012); `limit` is clamped.
|
||||
*/
|
||||
export function compileMetricQuery(
|
||||
req: MetricQueryRequest,
|
||||
db: string,
|
||||
catalog: DimensionCatalog,
|
||||
maxResultRows: number,
|
||||
): CompiledQuery {
|
||||
if (!req.tenant_id || !req.tenant_id.trim()) {
|
||||
// Fail closed. A blank tenant must never compile to "no predicate".
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "tenant_id is required on every analytics query");
|
||||
}
|
||||
assertSafeIdentifier(db, "database");
|
||||
const table: AnalyticsTable = req.table ?? "opt_analytics_event";
|
||||
assertSafeIdentifier(table, "table");
|
||||
if (!METRIC_AGGREGATES.includes(req.aggregate)) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "unsupported aggregate");
|
||||
}
|
||||
|
||||
const params: Record<string, string> = {
|
||||
param_tenant: req.tenant_id,
|
||||
param_metric: req.metric,
|
||||
};
|
||||
|
||||
// ── the forced predicate. First condition, always, no branch. ─────────
|
||||
const conditions: string[] = [FORCED_TENANT_PREDICATE, "metric = {metric:String}"];
|
||||
|
||||
if (req.project_id) {
|
||||
conditions.push("project_id = {project:String}");
|
||||
params.param_project = req.project_id;
|
||||
}
|
||||
if (req.dataset_id) {
|
||||
conditions.push("dataset_id = {dataset:String}");
|
||||
params.param_dataset = req.dataset_id;
|
||||
}
|
||||
if (req.from_time) {
|
||||
conditions.push("event_time >= parseDateTime64BestEffort({fromTime:String})");
|
||||
params.param_fromTime = req.from_time;
|
||||
}
|
||||
if (req.to_time) {
|
||||
conditions.push("event_time < parseDateTime64BestEffort({toTime:String})");
|
||||
params.param_toTime = req.to_time;
|
||||
}
|
||||
|
||||
// ── filters: names resolved through the catalog, values parameterized ──
|
||||
let filterIdx = 0;
|
||||
for (const [name, value] of Object.entries(req.filters ?? {})) {
|
||||
const physical = catalog.resolve(name);
|
||||
if (physical === undefined) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", `unknown filter dimension "${name}"`);
|
||||
}
|
||||
// `physical` came from the catalog, but validate anyway: a corrupted or
|
||||
// hostile catalog row must not become an injection vector either.
|
||||
assertSafeIdentifier(physical, "dimension");
|
||||
const nameKey = `f${filterIdx}n`;
|
||||
const valKey = `f${filterIdx}v`;
|
||||
conditions.push(
|
||||
`dim_values[indexOf(dim_names, {${nameKey}:String})] = {${valKey}:String}`,
|
||||
);
|
||||
params[`param_${nameKey}`] = physical;
|
||||
params[`param_${valKey}`] = value;
|
||||
filterIdx += 1;
|
||||
}
|
||||
|
||||
// ── group by: same resolution, plus a hard cardinality ceiling ─────────
|
||||
const groupBy = req.group_by ?? [];
|
||||
if (groupBy.length > MAX_GROUP_BY) {
|
||||
throw new SiaxError(
|
||||
"INVALID_REQUEST",
|
||||
"0pt.analytics",
|
||||
`at most ${MAX_GROUP_BY} group-by dimensions (got ${groupBy.length})`,
|
||||
);
|
||||
}
|
||||
const selectParts: string[] = [];
|
||||
const groupParts: string[] = [];
|
||||
groupBy.forEach((name, i) => {
|
||||
const physical = catalog.resolve(name);
|
||||
if (physical === undefined) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", `unknown group_by dimension "${name}"`);
|
||||
}
|
||||
assertSafeIdentifier(physical, "dimension");
|
||||
const key = `g${i}`;
|
||||
// The ALIAS is compiler-generated (`grp0`, `grp1`), never caller text.
|
||||
selectParts.push(`dim_values[indexOf(dim_names, {${key}:String})] AS grp${i}`);
|
||||
groupParts.push(`grp${i}`);
|
||||
params[`param_${key}`] = physical;
|
||||
});
|
||||
|
||||
const limit = Math.max(1, Math.min(Math.floor(req.limit ?? 1000), maxResultRows));
|
||||
|
||||
const sql =
|
||||
`SELECT ${selectParts.length ? selectParts.join(", ") + ", " : ""}${AGG_SQL[req.aggregate]} AS value ` +
|
||||
`FROM ${db}.${table} ` +
|
||||
`WHERE ${conditions.join(" AND ")} ` +
|
||||
(groupParts.length ? `GROUP BY ${groupParts.join(", ")} ORDER BY value DESC ` : "") +
|
||||
`LIMIT ${limit} FORMAT JSON`;
|
||||
|
||||
return { sql, params, forcedTenantPredicate: FORCED_TENANT_PREDICATE, table };
|
||||
}
|
||||
|
||||
/**
|
||||
* Compile a projection lookup (SERVING tier, BI-P010's opt_dataset_projection).
|
||||
* Same forcing, different table and key.
|
||||
*/
|
||||
export function compileProjectionQuery(
|
||||
req: { tenant_id: string; project_id?: string; dataset_id: string; entity_keys?: string[]; limit?: number },
|
||||
db: string,
|
||||
maxResultRows: number,
|
||||
): CompiledQuery {
|
||||
if (!req.tenant_id || !req.tenant_id.trim()) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "tenant_id is required on every analytics query");
|
||||
}
|
||||
assertSafeIdentifier(db, "database");
|
||||
const params: Record<string, string> = {
|
||||
param_tenant: req.tenant_id,
|
||||
param_dataset: req.dataset_id,
|
||||
};
|
||||
const conditions = [FORCED_TENANT_PREDICATE, "dataset_id = {dataset:String}"];
|
||||
if (req.project_id) {
|
||||
conditions.push("project_id = {project:String}");
|
||||
params.param_project = req.project_id;
|
||||
}
|
||||
if (req.entity_keys && req.entity_keys.length > 0) {
|
||||
conditions.push("has({entityKeys:Array(String)}, entity_key)");
|
||||
params.param_entityKeys = JSON.stringify(req.entity_keys);
|
||||
}
|
||||
const limit = Math.max(1, Math.min(Math.floor(req.limit ?? 1000), maxResultRows));
|
||||
const sql =
|
||||
`SELECT entity_key, as_of, version, attributes, measures ` +
|
||||
`FROM ${db}.opt_dataset_projection ` +
|
||||
`WHERE ${conditions.join(" AND ")} ` +
|
||||
`ORDER BY entity_key LIMIT ${limit} FORMAT JSON`;
|
||||
return { sql, params, forcedTenantPredicate: FORCED_TENANT_PREDICATE, table: "opt_dataset_projection" };
|
||||
}
|
||||
|
||||
/**
|
||||
* BI-P011 layer-3 bootstrap: the DDL for a RESTRICTED per-tenant ClickHouse
|
||||
* user with a row policy pinning `tenant_id`.
|
||||
*
|
||||
* This is the layer that survives a compiler bug. The compiler forcing the
|
||||
* predicate is layer 2; if a future refactor drops it, layer 3 still returns
|
||||
* zero foreign rows because the SERVER filters them. The BI-P011 proof runs
|
||||
* the adversarial suite twice — once through an unrestricted user (hole open)
|
||||
* and once through this restricted user (hole closed) — because a claim of
|
||||
* defence in depth that has never been tested with one layer removed is not
|
||||
* evidence.
|
||||
*
|
||||
* `tenantId` is validated as an identifier-safe token AND embedded as a
|
||||
* quoted literal; ClickHouse's CREATE ROW POLICY takes no bound parameters,
|
||||
* so validation is the only protection available and it is applied strictly.
|
||||
*/
|
||||
export function restrictedTenantUserDdl(args: {
|
||||
db: string;
|
||||
tenantId: string;
|
||||
userName: string;
|
||||
password: string;
|
||||
tables: readonly string[];
|
||||
settingsProfile?: string;
|
||||
}): string[] {
|
||||
assertSafeIdentifier(args.db, "database");
|
||||
assertSafeIdentifier(args.userName, "user");
|
||||
// A tenant id is `t_<slug>` in this estate; enforce that shape strictly
|
||||
// rather than trusting the caller.
|
||||
if (!/^[A-Za-z0-9_-]{1,64}$/.test(args.tenantId)) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "tenant id is not policy-safe");
|
||||
}
|
||||
if (args.password.includes("'") || args.password.includes("\\")) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "password contains characters that cannot be embedded");
|
||||
}
|
||||
const profile = args.settingsProfile;
|
||||
if (profile) assertSafeIdentifier(profile, "settings profile");
|
||||
|
||||
const stmts: string[] = [
|
||||
`CREATE USER IF NOT EXISTS ${args.userName} IDENTIFIED WITH sha256_password BY '${args.password}'` +
|
||||
(profile ? ` SETTINGS PROFILE ${profile}` : ` SETTINGS readonly = 1`),
|
||||
];
|
||||
for (const table of args.tables) {
|
||||
assertSafeIdentifier(table, "table");
|
||||
stmts.push(
|
||||
`CREATE ROW POLICY IF NOT EXISTS ${args.userName}_${table}_rp ON ${args.db}.${table} ` +
|
||||
`FOR SELECT USING tenant_id = '${args.tenantId}' TO ${args.userName}`,
|
||||
);
|
||||
stmts.push(`GRANT SELECT ON ${args.db}.${table} TO ${args.userName}`);
|
||||
}
|
||||
return stmts;
|
||||
}
|
||||
@@ -0,0 +1,222 @@
|
||||
/**
|
||||
* src/analytics/engine/resource-governance.ts
|
||||
* ============================================
|
||||
* BI-P012 — CLICKHOUSE RESOURCE GOVERNANCE.
|
||||
*
|
||||
* Every setting BI-P012 names, in one place, with a value:
|
||||
*
|
||||
* max_execution_time max_result_rows max_rows_to_read
|
||||
* max_bytes_to_read max_memory_usage max_concurrent_queries
|
||||
*
|
||||
* Three layers, and the order matters:
|
||||
*
|
||||
* 1. SERVER-SIDE SETTINGS travel with every statement on the HTTP
|
||||
* interface. ClickHouse enforces them; the client cannot opt out
|
||||
* because the client is what writes them. `readonly=1` on top means a
|
||||
* SELECT cannot re-set them mid-query (readonly=1 forbids SETTINGS
|
||||
* changes as well as writes — that is the whole reason it is here and
|
||||
* not merely `readonly=2`).
|
||||
* 2. `max_concurrent_queries` is NOT a per-query setting in ClickHouse; it
|
||||
* is a per-USER quota. `concurrencySlotSql` renders the CREATE
|
||||
* SETTINGS PROFILE that carries it, applied to the tenant's restricted
|
||||
* user (BI-P011 layer 3) so one tenant's fan-out cannot consume the
|
||||
* whole server.
|
||||
* 3. A local admission gate (`ConcurrencyGate`) refuses the N+1st
|
||||
* in-flight query per tenant before it ever reaches the network. Two
|
||||
* layers because the ClickHouse quota is per-user and one 0pt process
|
||||
* can still queue work faster than the server rejects it.
|
||||
*
|
||||
* Budget tiers exist because "one limit for everyone" is how a small tenant
|
||||
* subsidises a query bomb. `INTERACTIVE` is the dashboard default and is
|
||||
* deliberately mean.
|
||||
*/
|
||||
import { SiaxError } from "../../platform/errors";
|
||||
|
||||
export interface ResourceBudget {
|
||||
/** Seconds. ClickHouse `max_execution_time`. */
|
||||
maxExecutionTimeSec: number;
|
||||
/** Rows returned to the caller. ClickHouse `max_result_rows`. */
|
||||
maxResultRows: number;
|
||||
/** Rows SCANNED (not returned). ClickHouse `max_rows_to_read`. */
|
||||
maxRowsToRead: number;
|
||||
/** Bytes scanned. ClickHouse `max_bytes_to_read`. */
|
||||
maxBytesToRead: number;
|
||||
/** Bytes of RAM per query. ClickHouse `max_memory_usage`. */
|
||||
maxMemoryUsageBytes: number;
|
||||
/** In-flight queries per tenant. Enforced by quota + local gate. */
|
||||
maxConcurrentQueries: number;
|
||||
}
|
||||
|
||||
export const BUDGET_TIERS = ["INTERACTIVE", "ANALYTICAL", "BATCH"] as const;
|
||||
export type BudgetTier = (typeof BUDGET_TIERS)[number];
|
||||
|
||||
/**
|
||||
* Defaults. Chosen against the shape of the analytics tables in
|
||||
* `src/analytics/engine/schema.ts`, not plucked from the air:
|
||||
* INTERACTIVE must answer a dashboard panel inside a human's patience budget,
|
||||
* so it caps the SCAN, which is what actually costs time on a MergeTree.
|
||||
*/
|
||||
export const DEFAULT_BUDGETS: Record<BudgetTier, ResourceBudget> = {
|
||||
INTERACTIVE: {
|
||||
maxExecutionTimeSec: 10,
|
||||
maxResultRows: 10_000,
|
||||
maxRowsToRead: 50_000_000,
|
||||
maxBytesToRead: 2 * 1024 * 1024 * 1024,
|
||||
maxMemoryUsageBytes: 2 * 1024 * 1024 * 1024,
|
||||
maxConcurrentQueries: 8,
|
||||
},
|
||||
ANALYTICAL: {
|
||||
maxExecutionTimeSec: 60,
|
||||
maxResultRows: 500_000,
|
||||
maxRowsToRead: 2_000_000_000,
|
||||
maxBytesToRead: 32 * 1024 * 1024 * 1024,
|
||||
maxMemoryUsageBytes: 8 * 1024 * 1024 * 1024,
|
||||
maxConcurrentQueries: 4,
|
||||
},
|
||||
BATCH: {
|
||||
maxExecutionTimeSec: 900,
|
||||
maxResultRows: 10_000_000,
|
||||
maxRowsToRead: 50_000_000_000,
|
||||
maxBytesToRead: 512 * 1024 * 1024 * 1024,
|
||||
maxMemoryUsageBytes: 24 * 1024 * 1024 * 1024,
|
||||
maxConcurrentQueries: 2,
|
||||
},
|
||||
};
|
||||
|
||||
/**
|
||||
* A CL0UD-issued quota may only ever TIGHTEN a tier, never loosen it. A
|
||||
* control-plane bug that hands out a generous number must not be able to
|
||||
* disarm the engine's own ceiling.
|
||||
*/
|
||||
export function applyQuota(base: ResourceBudget, quota: Partial<ResourceBudget> | undefined): ResourceBudget {
|
||||
if (!quota) return { ...base };
|
||||
const pick = (b: number, q: number | undefined): number =>
|
||||
typeof q === "number" && Number.isFinite(q) && q > 0 ? Math.min(b, q) : b;
|
||||
return {
|
||||
maxExecutionTimeSec: pick(base.maxExecutionTimeSec, quota.maxExecutionTimeSec),
|
||||
maxResultRows: pick(base.maxResultRows, quota.maxResultRows),
|
||||
maxRowsToRead: pick(base.maxRowsToRead, quota.maxRowsToRead),
|
||||
maxBytesToRead: pick(base.maxBytesToRead, quota.maxBytesToRead),
|
||||
maxMemoryUsageBytes: pick(base.maxMemoryUsageBytes, quota.maxMemoryUsageBytes),
|
||||
maxConcurrentQueries: pick(base.maxConcurrentQueries, quota.maxConcurrentQueries),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The settings that ride along on the HTTP request. `readonly: "1"` is what
|
||||
* makes them un-overridable from inside the query text.
|
||||
*
|
||||
* `timeout_overflow_mode` / `result_overflow_mode` / `read_overflow_mode` are
|
||||
* set to `throw` EXPLICITLY: ClickHouse's default for some of these is
|
||||
* `break`, which silently returns a PARTIAL result. A partial result that
|
||||
* looks complete is the analytics equivalent of a false green, so the engine
|
||||
* refuses it.
|
||||
*/
|
||||
export function clickhouseSettings(budget: ResourceBudget): Record<string, string> {
|
||||
return {
|
||||
readonly: "1",
|
||||
max_execution_time: String(budget.maxExecutionTimeSec),
|
||||
timeout_overflow_mode: "throw",
|
||||
max_result_rows: String(budget.maxResultRows),
|
||||
result_overflow_mode: "throw",
|
||||
max_rows_to_read: String(budget.maxRowsToRead),
|
||||
max_bytes_to_read: String(budget.maxBytesToRead),
|
||||
read_overflow_mode: "throw",
|
||||
max_memory_usage: String(budget.maxMemoryUsageBytes),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Settings for a PLANNER call (`EXPLAIN ESTIMATE`, `EXPLAIN PLAN`).
|
||||
*
|
||||
* Measured, not assumed: running `EXPLAIN ESTIMATE` under the caller's own
|
||||
* `max_rows_to_read` makes ClickHouse refuse the ESTIMATE itself with code 158
|
||||
* whenever the estimate would be over budget — i.e. exactly the case the
|
||||
* estimate exists to detect. An estimator that cannot describe an expensive
|
||||
* query is useless, so the read ceilings are deliberately NOT applied here.
|
||||
* The planner still runs `readonly = 1` under a short deadline and a memory
|
||||
* cap, so this is a narrower envelope, never an unbounded one.
|
||||
*/
|
||||
export function planningSettings(budget: ResourceBudget): Record<string, string> {
|
||||
return {
|
||||
readonly: "1",
|
||||
max_execution_time: String(Math.min(budget.maxExecutionTimeSec, 15)),
|
||||
timeout_overflow_mode: "throw",
|
||||
max_memory_usage: String(budget.maxMemoryUsageBytes),
|
||||
};
|
||||
}
|
||||
|
||||
/** Every setting name BI-P012 requires, for the DoD assertion. */
|
||||
export const REQUIRED_GOVERNANCE_SETTINGS = [
|
||||
"max_execution_time",
|
||||
"max_result_rows",
|
||||
"max_rows_to_read",
|
||||
"max_bytes_to_read",
|
||||
"max_memory_usage",
|
||||
"max_concurrent_queries",
|
||||
] as const;
|
||||
|
||||
/**
|
||||
* `max_concurrent_queries` lives on a SETTINGS PROFILE, not on a statement.
|
||||
* Rendered here so BI-P011's restricted-user bootstrap and BI-P012's budget
|
||||
* come from one source.
|
||||
*/
|
||||
export function concurrencyProfileSql(profileName: string, budget: ResourceBudget): string[] {
|
||||
return [
|
||||
`CREATE SETTINGS PROFILE IF NOT EXISTS ${profileName} SETTINGS ` +
|
||||
`max_execution_time = ${budget.maxExecutionTimeSec}, ` +
|
||||
`max_result_rows = ${budget.maxResultRows}, ` +
|
||||
`max_rows_to_read = ${budget.maxRowsToRead}, ` +
|
||||
`max_bytes_to_read = ${budget.maxBytesToRead}, ` +
|
||||
`max_memory_usage = ${budget.maxMemoryUsageBytes}, ` +
|
||||
`readonly = 1`,
|
||||
`CREATE QUOTA IF NOT EXISTS ${profileName}_quota ` +
|
||||
`FOR INTERVAL 1 SECOND MAX queries = ${budget.maxConcurrentQueries * 20} ` +
|
||||
`TO ALL EXCEPT default`,
|
||||
];
|
||||
}
|
||||
|
||||
/**
|
||||
* Local admission control. One instance per process; keyed by tenant so a
|
||||
* noisy tenant is throttled without touching anyone else's slots — which is
|
||||
* exactly BI-P012's DoD wording ("without degrading unrelated tenant").
|
||||
*/
|
||||
export class ConcurrencyGate {
|
||||
private readonly inFlight = new Map<string, number>();
|
||||
|
||||
constructor(private readonly limitFor: (tenantId: string) => number) {}
|
||||
|
||||
/** Returns a release function. Throws RATE_LIMITED when the tenant is full. */
|
||||
acquire(tenantId: string): () => void {
|
||||
const limit = this.limitFor(tenantId);
|
||||
const current = this.inFlight.get(tenantId) ?? 0;
|
||||
if (current >= limit) {
|
||||
throw new SiaxError(
|
||||
"RATE_LIMITED",
|
||||
"0pt.analytics",
|
||||
`tenant already has ${current} analytics queries in flight (limit ${limit})`,
|
||||
);
|
||||
}
|
||||
this.inFlight.set(tenantId, current + 1);
|
||||
let released = false;
|
||||
return () => {
|
||||
if (released) return;
|
||||
released = true;
|
||||
const n = (this.inFlight.get(tenantId) ?? 1) - 1;
|
||||
if (n <= 0) this.inFlight.delete(tenantId);
|
||||
else this.inFlight.set(tenantId, n);
|
||||
};
|
||||
}
|
||||
|
||||
/** Observability: in-flight count for a tenant (0 when idle). */
|
||||
inFlightFor(tenantId: string): number {
|
||||
return this.inFlight.get(tenantId) ?? 0;
|
||||
}
|
||||
|
||||
/** Total in-flight across all tenants. */
|
||||
totalInFlight(): number {
|
||||
let n = 0;
|
||||
for (const v of this.inFlight.values()) n += v;
|
||||
return n;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,169 @@
|
||||
/**
|
||||
* src/analytics/engine/schema.ts
|
||||
* ===============================
|
||||
* BI-P010 — CLICKHOUSE PRODUCTION SCHEMA.
|
||||
*
|
||||
* BI-P010's explicit warning is "Avoid one giant unbounded table for every
|
||||
* analytics use case." The existing `opt_metric_events` (owned by
|
||||
* src/analytics/clickhouse-analytics.ts, PRESERVED untouched) is exactly one
|
||||
* table for one job: low-cardinality capability metrics. It stays, and it is
|
||||
* NOT extended into the catch-all.
|
||||
*
|
||||
* Four tables, four jobs, four retention profiles:
|
||||
*
|
||||
* opt_metric_events (PRE-EXISTING, not defined here) — capability
|
||||
* metrics, Map(String,String) dimensions, 90d.
|
||||
* opt_analytics_event — high-volume per-tenant event stream. Wide
|
||||
* dimension arrays, SHORT TTL (raw events are not
|
||||
* the serving tier; they age into the rollups).
|
||||
* opt_metric_daily — AggregatingMergeTree rollup fed by a
|
||||
* MATERIALIZED VIEW off opt_analytics_event.
|
||||
* This is what a dashboard time-series reads;
|
||||
* long TTL, tiny footprint.
|
||||
* opt_dataset_projection — the SERVING tier (BI-P003) for a CURATED
|
||||
* dataset: one row per (tenant, dataset, entity),
|
||||
* ReplacingMergeTree so a refresh is idempotent.
|
||||
*
|
||||
* Why the split is load-bearing, not cosmetic:
|
||||
* * different ORDER BY keys. A dashboard filtering (tenant, metric, day)
|
||||
* and a projection looked up by (tenant, dataset, entity_key) cannot
|
||||
* share one primary key without one of them degenerating to a full scan.
|
||||
* * different TTLs. Raw events must expire fast; rollups must not.
|
||||
* * different cardinality. `LowCardinality(String)` is a win on
|
||||
* tenant/metric/dataset and a LOSS on a free-form entity key, so they
|
||||
* cannot be the same column type in the same table.
|
||||
*
|
||||
* EVERY table leads its ORDER BY with `tenant_id`. That is not stylistic: it
|
||||
* is what makes the forced tenant predicate (BI-P011) a primary-key range
|
||||
* scan instead of a filter applied after reading everyone's rows, and it is
|
||||
* what makes a per-tenant row policy cheap.
|
||||
*
|
||||
* LEGAL BOUNDARY (inherited verbatim from clickhouse-analytics.ts): this
|
||||
* schema is for generic 0pt analytics. It MUST NEVER be pointed at the b0ts
|
||||
* ClickHouse pipelines or quarantined market data.
|
||||
*/
|
||||
|
||||
export interface AnalyticsSchemaOptions {
|
||||
database: string;
|
||||
/** Raw event retention. Deliberately short — rollups carry the history. */
|
||||
eventRetentionDays?: number;
|
||||
/** Rollup retention. */
|
||||
rollupRetentionDays?: number;
|
||||
/** Serving-projection retention. */
|
||||
projectionRetentionDays?: number;
|
||||
}
|
||||
|
||||
/** Physical tables this module owns (opt_metric_events is NOT one of them). */
|
||||
export const ANALYTICS_TABLES = [
|
||||
"opt_analytics_event",
|
||||
"opt_metric_daily",
|
||||
"opt_dataset_projection",
|
||||
] as const;
|
||||
export type AnalyticsTable = (typeof ANALYTICS_TABLES)[number];
|
||||
|
||||
/** The materialized view wiring opt_analytics_event -> opt_metric_daily. */
|
||||
export const ANALYTICS_MATERIALIZED_VIEWS = ["opt_metric_daily_mv"] as const;
|
||||
|
||||
/**
|
||||
* DDL, in dependency order. Every statement is CREATE ... IF NOT EXISTS so
|
||||
* applying it to a live database is idempotent (the same contract
|
||||
* `ClickHouseAnalytics.ensureSchema` already keeps).
|
||||
*/
|
||||
export function analyticsSchemaDdl(opts: AnalyticsSchemaOptions): string[] {
|
||||
const db = opts.database;
|
||||
const eventTtl = Math.floor(opts.eventRetentionDays ?? 30);
|
||||
const rollupTtl = Math.floor(opts.rollupRetentionDays ?? 1095);
|
||||
const projectionTtl = Math.floor(opts.projectionRetentionDays ?? 365);
|
||||
|
||||
return [
|
||||
`CREATE DATABASE IF NOT EXISTS ${db}`,
|
||||
|
||||
// ── raw event stream ────────────────────────────────────────────────
|
||||
// Dimensions as parallel arrays (not Map) so ClickHouse can prune on
|
||||
// dimension VALUES via the bloom filter index; Map(String,String) forces
|
||||
// reading the whole map. `event_id` makes deduplication possible on
|
||||
// replay without a ReplacingMergeTree's merge latency.
|
||||
`CREATE TABLE IF NOT EXISTS ${db}.opt_analytics_event (
|
||||
tenant_id LowCardinality(String),
|
||||
project_id LowCardinality(String),
|
||||
dataset_id LowCardinality(String),
|
||||
metric LowCardinality(String),
|
||||
event_id String,
|
||||
event_time DateTime64(3, 'UTC'),
|
||||
value Float64,
|
||||
dim_names Array(LowCardinality(String)),
|
||||
dim_values Array(String),
|
||||
ingested_at DateTime64(3, 'UTC') DEFAULT now64(3),
|
||||
INDEX idx_dim_values dim_values TYPE bloom_filter(0.01) GRANULARITY 4
|
||||
)
|
||||
ENGINE = MergeTree()
|
||||
PARTITION BY toYYYYMM(event_time)
|
||||
ORDER BY (tenant_id, project_id, dataset_id, metric, event_time)
|
||||
TTL toDateTime(event_time) + INTERVAL ${eventTtl} DAY
|
||||
SETTINGS index_granularity = 8192`,
|
||||
|
||||
// ── daily rollup (serving tier for time series) ─────────────────────
|
||||
`CREATE TABLE IF NOT EXISTS ${db}.opt_metric_daily (
|
||||
tenant_id LowCardinality(String),
|
||||
project_id LowCardinality(String),
|
||||
dataset_id LowCardinality(String),
|
||||
metric LowCardinality(String),
|
||||
day Date,
|
||||
sum_state AggregateFunction(sum, Float64),
|
||||
count_state AggregateFunction(count, Float64),
|
||||
min_state AggregateFunction(min, Float64),
|
||||
max_state AggregateFunction(max, Float64)
|
||||
)
|
||||
ENGINE = AggregatingMergeTree()
|
||||
PARTITION BY toYYYYMM(day)
|
||||
ORDER BY (tenant_id, project_id, dataset_id, metric, day)
|
||||
TTL day + INTERVAL ${rollupTtl} DAY`,
|
||||
|
||||
`CREATE MATERIALIZED VIEW IF NOT EXISTS ${db}.opt_metric_daily_mv
|
||||
TO ${db}.opt_metric_daily AS
|
||||
SELECT
|
||||
tenant_id,
|
||||
project_id,
|
||||
dataset_id,
|
||||
metric,
|
||||
toDate(event_time) AS day,
|
||||
sumState(value) AS sum_state,
|
||||
countState(value) AS count_state,
|
||||
minState(value) AS min_state,
|
||||
maxState(value) AS max_state
|
||||
FROM ${db}.opt_analytics_event
|
||||
GROUP BY tenant_id, project_id, dataset_id, metric, day`,
|
||||
|
||||
// ── serving projection for a CURATED dataset ────────────────────────
|
||||
// ReplacingMergeTree(version) so refreshProjection is idempotent: the
|
||||
// same (tenant, dataset, entity_key) re-ingested at a higher version
|
||||
// supersedes rather than duplicates.
|
||||
`CREATE TABLE IF NOT EXISTS ${db}.opt_dataset_projection (
|
||||
tenant_id LowCardinality(String),
|
||||
project_id LowCardinality(String),
|
||||
dataset_id LowCardinality(String),
|
||||
entity_key String,
|
||||
as_of DateTime64(3, 'UTC'),
|
||||
version UInt64,
|
||||
attributes Map(String, String),
|
||||
measures Map(String, Float64),
|
||||
INDEX idx_entity entity_key TYPE bloom_filter(0.01) GRANULARITY 4
|
||||
)
|
||||
ENGINE = ReplacingMergeTree(version)
|
||||
PARTITION BY (tenant_id)
|
||||
ORDER BY (tenant_id, project_id, dataset_id, entity_key)
|
||||
TTL toDateTime(as_of) + INTERVAL ${projectionTtl} DAY`,
|
||||
];
|
||||
}
|
||||
|
||||
/**
|
||||
* Statement that proves a table's ORDER BY leads with tenant_id — used by the
|
||||
* BI-P011 test suite so "tenant-first primary key" is asserted against the
|
||||
* live server, not against this file's own comments.
|
||||
*/
|
||||
export function sortingKeySql(db: string, table: string): string {
|
||||
return (
|
||||
`SELECT sorting_key FROM system.tables ` +
|
||||
`WHERE database = '${db}' AND name = '${table}' FORMAT TSV`
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,180 @@
|
||||
/**
|
||||
* src/analytics/engine/siax-analytics-engine.ts
|
||||
* ==============================================
|
||||
* BI-P009 — the provider-neutral analytics engine interface.
|
||||
*
|
||||
* Eight capabilities, named exactly as BI-P009 lists them:
|
||||
*
|
||||
* health executeMetric executeQuery estimateQuery
|
||||
* explain cancelQuery ingestProjection refreshProjection
|
||||
*
|
||||
* ClickHouse is the CANONICAL provider (BI-P009); DuckDB (BI-P014) and Trino
|
||||
* (BI-P015) implement the same interface later. Nothing above this interface
|
||||
* may import a provider module directly — that is what makes "replaceable
|
||||
* provider" true rather than aspirational.
|
||||
*
|
||||
* NORMATIVE OWNERSHIP (BI program §0): every method takes an
|
||||
* `AnalyticsIdentity` that the CALLER cannot forge, because it is produced by
|
||||
* CL0UD verification upstream, not by request-body parsing. Product repos
|
||||
* (VehIQ) call an API in front of this interface; they never hold a provider
|
||||
* credential and never construct an engine.
|
||||
*/
|
||||
import type { CompiledQuery, MetricAggregate } from "./query-compiler";
|
||||
import type { AnalyticsTable } from "./schema";
|
||||
import type { BudgetTier, ResourceBudget } from "./resource-governance";
|
||||
|
||||
/**
|
||||
* The VERIFIED caller. `tenant_id` here is the one the engine forces into
|
||||
* every predicate — it comes from CL0UD identity verification (BI-P011 layer
|
||||
* 1), never from a request body. This estate has already been burned once by
|
||||
* a tenant read out of a request body (INF0 Wave 2, cross-tenant RAG access),
|
||||
* which is why the type is separate from the request rather than a field on
|
||||
* it.
|
||||
*/
|
||||
export interface AnalyticsIdentity {
|
||||
tenant_id: string;
|
||||
project_id?: string;
|
||||
/** CL0UD principal, for audit + per-principal quota. */
|
||||
principal_id: string;
|
||||
correlation_id?: string;
|
||||
/** Budget tier granted by CL0UD entitlements (BI-P012). */
|
||||
budget_tier?: BudgetTier;
|
||||
/** CL0UD quota, which may only TIGHTEN the tier. */
|
||||
quota?: Partial<ResourceBudget>;
|
||||
}
|
||||
|
||||
export interface MetricRequest {
|
||||
metric: string;
|
||||
aggregate: MetricAggregate;
|
||||
dataset_id?: string;
|
||||
group_by?: string[];
|
||||
filters?: Record<string, string>;
|
||||
from_time?: string;
|
||||
to_time?: string;
|
||||
limit?: number;
|
||||
table?: AnalyticsTable;
|
||||
}
|
||||
|
||||
export interface AnalyticsRow {
|
||||
/** Resolved group values, keyed by the CALLER-FACING dimension name. */
|
||||
groups?: Record<string, string>;
|
||||
value: number;
|
||||
}
|
||||
|
||||
export interface EngineResult {
|
||||
rows: AnalyticsRow[];
|
||||
/** Engine-side id; the handle `cancelQuery` needs. */
|
||||
query_id: string;
|
||||
/** True when the governance ceiling cut the result short (BI-P012). */
|
||||
truncated: boolean;
|
||||
stats: QueryStats;
|
||||
}
|
||||
|
||||
export interface QueryStats {
|
||||
duration_ms: number;
|
||||
rows_read?: number;
|
||||
bytes_read?: number;
|
||||
result_rows: number;
|
||||
}
|
||||
|
||||
export interface QueryEstimate {
|
||||
/** Rows the engine expects to SCAN. */
|
||||
estimated_rows_to_read: number;
|
||||
estimated_bytes_to_read: number;
|
||||
/** False when the estimate exceeds the caller's budget (BI-P012). */
|
||||
within_budget: boolean;
|
||||
budget: ResourceBudget;
|
||||
/** Why it was refused, when `within_budget` is false. */
|
||||
reason?: string;
|
||||
}
|
||||
|
||||
export interface QueryExplanation {
|
||||
/** Provider's plan text. Never contains caller-supplied SQL fragments. */
|
||||
plan: string;
|
||||
/** The predicate the compiler forced — assertable, not narrated. */
|
||||
forced_tenant_predicate: string;
|
||||
table: string;
|
||||
}
|
||||
|
||||
export interface ProjectionRow {
|
||||
entity_key: string;
|
||||
as_of?: string;
|
||||
version?: number;
|
||||
attributes?: Record<string, string>;
|
||||
measures?: Record<string, number>;
|
||||
}
|
||||
|
||||
export interface ProjectionIngestRequest {
|
||||
dataset_id: string;
|
||||
rows: ProjectionRow[];
|
||||
}
|
||||
|
||||
export interface ProjectionRefreshResult {
|
||||
dataset_id: string;
|
||||
/** Rows visible after the refresh settled. */
|
||||
row_count: number;
|
||||
refreshed_at: string;
|
||||
}
|
||||
|
||||
export interface HealthReport {
|
||||
healthy: boolean;
|
||||
provider: string;
|
||||
/** Provider version when reachable. */
|
||||
version?: string;
|
||||
/** Short operator-facing reason. Never a raw engine body. */
|
||||
detail?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* A raw, ANALYST-facing query (BI-P017 will govern its SQL dialect). It is a
|
||||
* SEPARATE method from `executeMetric` on purpose: `executeMetric` has no
|
||||
* raw-SQL surface at all, and keeping them apart means the closed DSL cannot
|
||||
* silently grow one.
|
||||
*
|
||||
* The canonical ClickHouse provider in this pass DELIBERATELY refuses this
|
||||
* method (`CAPABILITY_UNAVAILABLE`): opening an analyst SQL surface before
|
||||
* BI-P017's dialect contract and BI-P033's authorization exist would be
|
||||
* exactly the "raw SQL attempt" BI-P011's adversarial list names. Declaring
|
||||
* it and refusing it is honest; implementing it early is not.
|
||||
*/
|
||||
export interface AnalystQueryRequest {
|
||||
sql: string;
|
||||
dataset_ids: string[];
|
||||
limit?: number;
|
||||
}
|
||||
|
||||
export interface SiaxAnalyticsEngine {
|
||||
readonly provider: string;
|
||||
|
||||
health(): Promise<HealthReport>;
|
||||
|
||||
executeMetric(identity: AnalyticsIdentity, request: MetricRequest): Promise<EngineResult>;
|
||||
|
||||
executeQuery(identity: AnalyticsIdentity, request: AnalystQueryRequest): Promise<EngineResult>;
|
||||
|
||||
estimateQuery(identity: AnalyticsIdentity, request: MetricRequest): Promise<QueryEstimate>;
|
||||
|
||||
cancelQuery(identity: AnalyticsIdentity, queryId: string): Promise<boolean>;
|
||||
|
||||
explain(identity: AnalyticsIdentity, request: MetricRequest): Promise<QueryExplanation>;
|
||||
|
||||
ingestProjection(identity: AnalyticsIdentity, request: ProjectionIngestRequest): Promise<{ ingested: number }>;
|
||||
|
||||
refreshProjection(identity: AnalyticsIdentity, dataset_id: string): Promise<ProjectionRefreshResult>;
|
||||
}
|
||||
|
||||
/** The eight capability names, for the provider contract test. */
|
||||
export const SIAX_ANALYTICS_ENGINE_CAPABILITIES = [
|
||||
"health",
|
||||
"executeMetric",
|
||||
"executeQuery",
|
||||
"estimateQuery",
|
||||
"cancelQuery",
|
||||
"explain",
|
||||
"ingestProjection",
|
||||
"refreshProjection",
|
||||
] as const;
|
||||
export type SiaxAnalyticsEngineCapability = (typeof SIAX_ANALYTICS_ENGINE_CAPABILITIES)[number];
|
||||
|
||||
/** Re-exported so provider modules do not reach across into the compiler. */
|
||||
export type { CompiledQuery };
|
||||
Reference in New Issue
Block a user