feat(analytics): BI-P002..BI-P012 analytics core — domain model, tiers, contracts, Arrow, ClickHouse engine, isolation, governance
CI (SIAX Cloud) / security (push) Successful in 41s
Security Scan (SIAX Cloud) / security (push) Successful in 40s
CI (SIAX Cloud) / integration (push) Successful in 52s
CI (SIAX Cloud) / quality (push) Successful in 1m5s
CI (SIAX Cloud) / security (pull_request) Successful in 31s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 29s
CI (SIAX Cloud) / integration (pull_request) Successful in 45s
CI (SIAX Cloud) / quality (pull_request) Successful in 53s

Implements the eight BI program prompts that make 0pt the canonical SIAX
analytics capability. Every file is NEW: not one existing file is modified,
and src/analytics/clickhouse-analytics.ts is byte-identical to main.

BI-P002 — the nineteen canonical entities, verbatim, with tenant_id +
project_id on every tenant-owned record. sql/008 gives each one a table.
The four version entities BI-P002 names as immutable are enforced by BEFORE
UPDATE/DELETE triggers, not by a TypeScript `readonly` — proven by direct SQL
against a live Postgres.

BI-P003 — RAW/NORMALIZED/CURATED/SERVING with the read rule enforced:
assertReadableByBiSurface refuses a RAW dataset to a dashboard unless a
recorded exception exists. All eight required declarations are NOT NULL.

BI-P004 — DatasetContract and a fail-closed compatibility gate. Fifteen
breaking-change classes are each tested twice: refused under a minor bump,
accepted under a major one.

BI-P005 — Arrow IPC alongside JSON, with a hand-written FlatBuffers writer so
no npm dependency (or CVE surface) is added. Conformance comes from three
independent implementations, not from self-agreement: pyarrow 25.0.1 and
DuckDB 1.5.5 read identical values from both ClickHouse's Arrow and ours, and
ClickHouse's own Arrow C++ ingests our 20,000-row batch with exact aggregates.

BI-P009 — SiaxAnalyticsEngine with all eight capabilities. The existing
ClickHouseAnalytics is COMPOSED, never replaced. executeQuery (analyst SQL) is
declared and deliberately refused with CAPABILITY_UNAVAILABLE until BI-P017 and
BI-P033 land, rather than opening the raw-SQL hole BI-P011 enumerates.

BI-P010 — four purpose-built tables, not one giant one, and the difference is
measured against the counterfactual: 48.8ms/40,960 rows vs 174.4ms/5,400,000
rows for the same question; 5/660 granules vs 660/1149.

BI-P011 — layered isolation, each layer proven with the layer above removed.
Without the forced predicate the same statement returns 50,050 (both tenants);
with it, t_alpha gets 50. Without a row policy an unrestricted user sees 2
tenants; the restricted user running the IDENTICAL statement sees 1.

BI-P012 — every named limit, fired individually and identified by ClickHouse
error code, plus the half usually skipped: an unrelated tenant is timed while
a real CPU bomb is in flight and must still return the CORRECT answer.

Three defects found by writing tests that can fail:
  * `CHECK (array_length(purposes,1) >= 1)` can NEVER fail — array_length is
    NULL for '{}' and a NULL CHECK passes. Fixed with coalesce(); the test
    that caught it is pinned.
  * EXPLAIN ESTIMATE run under the caller's own max_rows_to_read fails with
    code 158 exactly when the estimate would be over budget. Planner calls now
    use a narrower envelope.
  * result_overflow_mode=break returns HTTP 200 with 65,409 of 100,000 rows
    and no indication it is short. Every SELECT ships `throw`.

Verified rather than assumed: server7's ClickHouse holds `opt_scraped` (51
tables, 407,793,471 rows, `persons`/`cz_owners` included) — the b0ts plane this
module is prohibited from touching — has no `opt_analytics` database, zero
tenant_id columns and zero row policies. Nothing there was modified; every
proof ran against disposable containers.

Evidence: evidence/bi-p002-p012/{test-transcript.txt,bi-p010-benchmark.json,
bi-p005-arrow-roundtrip.json,server7-clickhouse-probe.md}
Docs: docs/analytics/BI_ANALYTICS_CORE.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
2026-09-06 23:01:27 +02:00
parent 8d0a288668
commit 84bb9faa4f
32 changed files with 7844 additions and 0 deletions
+563
View File
@@ -0,0 +1,563 @@
/**
* src/analytics/engine/clickhouse-engine.ts
* ==========================================
* BI-P009 — ClickHouse as the CANONICAL `SiaxAnalyticsEngine` provider.
*
* PRESERVATION (BI-P009's first sentence): `ClickHouseAnalytics`
* (src/analytics/clickhouse-analytics.ts) is untouched by this change — not
* one line. This engine COMPOSES it:
*
* * `health()` delegates to `ClickHouseAnalytics.health()`.
* * `executeMetric()` on the pre-existing `opt_metric_events` table
* delegates to `ClickHouseAnalytics.queryMetric()`, i.e. keeps the
* already-proven forced-tenant path exactly as it is.
* * everything else (the BI-P010 tables) goes through
* `src/analytics/engine/query-compiler.ts`, which forces the same
* predicate for the same reason.
*
* "Do not regress the existing forced tenant predicate" is enforced by
* tests/analytics/tenant-predicate-regression.test.ts, which fails if either
* compiler stops emitting `tenant_id = {tenant:String}`.
*
* BI-P012 governance is applied on EVERY statement, including the delegated
* ones — the settings ride on the HTTP request, and `readonly=1` means the
* query text cannot re-set them.
*
* LEGAL BOUNDARY (inherited): generic 0pt analytics only. Never the b0ts
* ClickHouse pipelines, never quarantined market data.
*/
import crypto from "node:crypto";
import { SiaxError, siaxErrorCodeForUpstreamStatus } from "../../platform/errors";
import { ClickHouseAnalytics } from "../clickhouse-analytics";
import {
compileMetricQuery,
compileProjectionQuery,
dimensionCatalogFrom,
FORCED_TENANT_PREDICATE,
type DimensionCatalog,
} from "./query-compiler";
import { analyticsSchemaDdl, type AnalyticsSchemaOptions } from "./schema";
import {
applyQuota,
clickhouseSettings,
ConcurrencyGate,
DEFAULT_BUDGETS,
planningSettings,
type ResourceBudget,
} from "./resource-governance";
import type {
AnalystQueryRequest,
AnalyticsIdentity,
EngineResult,
HealthReport,
MetricRequest,
ProjectionIngestRequest,
ProjectionRefreshResult,
QueryEstimate,
QueryExplanation,
SiaxAnalyticsEngine,
} from "./siax-analytics-engine";
export interface ClickHouseEngineOptions {
baseUrl: string;
/** "user:password" — from an env/secret ref, never a literal in code. */
basicAuth?: string;
database?: string;
timeoutMs?: number;
/** Dimension allow-list per tenant. Absent tenant = no dimensions allowed. */
dimensionCatalogs?: Record<string, Record<string, string>>;
schema?: Omit<AnalyticsSchemaOptions, "database">;
}
const EMPTY_CATALOG: DimensionCatalog = { resolve: () => undefined };
export class ClickHouseAnalyticsEngine implements SiaxAnalyticsEngine {
readonly provider = "clickhouse";
/** PRESERVED pre-existing implementation, composed not replaced. */
private readonly legacy: ClickHouseAnalytics;
private readonly baseUrl: string;
private readonly authHeader?: string;
private readonly database: string;
private readonly timeoutMs: number;
private readonly dimensionCatalogs: Record<string, Record<string, string>>;
private readonly schemaOpts: Omit<AnalyticsSchemaOptions, "database">;
private readonly gate: ConcurrencyGate;
/** query_id -> AbortController, so cancelQuery can stop the client side too. */
private readonly inFlight = new Map<string, AbortController>();
constructor(opts: ClickHouseEngineOptions) {
this.baseUrl = opts.baseUrl.replace(/\/+$/, "");
this.authHeader = opts.basicAuth ? `Basic ${Buffer.from(opts.basicAuth).toString("base64")}` : undefined;
this.database = opts.database ?? "opt_analytics";
this.timeoutMs = opts.timeoutMs ?? 30_000;
this.dimensionCatalogs = opts.dimensionCatalogs ?? {};
this.schemaOpts = opts.schema ?? {};
this.legacy = new ClickHouseAnalytics({
baseUrl: opts.baseUrl,
basicAuth: opts.basicAuth,
database: this.database,
timeoutMs: this.timeoutMs,
});
this.gate = new ConcurrencyGate((tenantId) => this.budgetForTenant(tenantId).maxConcurrentQueries);
}
/** Access to the preserved implementation, for callers that still use it. */
get legacyAnalytics(): ClickHouseAnalytics {
return this.legacy;
}
private budgetForTenant(_tenantId: string): ResourceBudget {
return DEFAULT_BUDGETS.INTERACTIVE;
}
private budgetFor(identity: AnalyticsIdentity): ResourceBudget {
const tier = identity.budget_tier ?? "INTERACTIVE";
return applyQuota(DEFAULT_BUDGETS[tier], identity.quota);
}
private catalogFor(identity: AnalyticsIdentity): DimensionCatalog {
const pairs = this.dimensionCatalogs[identity.tenant_id];
return pairs ? dimensionCatalogFrom(pairs) : EMPTY_CATALOG;
}
/** Fail closed: a missing/blank verified tenant is never a wildcard. */
private assertIdentity(identity: AnalyticsIdentity): void {
if (!identity?.tenant_id || !identity.tenant_id.trim()) {
throw new SiaxError(
"UNAUTHENTICATED",
"0pt.analytics",
"analytics requires a verified CL0UD identity carrying tenant_id",
);
}
if (!identity.principal_id || !identity.principal_id.trim()) {
throw new SiaxError("UNAUTHENTICATED", "0pt.analytics", "analytics requires a verified principal_id");
}
}
// ── schema ───────────────────────────────────────────────────────────────
/** Apply BI-P010 DDL (idempotent) plus the preserved legacy table. */
async ensureSchema(): Promise<void> {
await this.legacy.ensureSchema();
for (const stmt of analyticsSchemaDdl({ database: this.database, ...this.schemaOpts })) {
await this.ddl(stmt);
}
}
// ── BI-P009 capability 1: health ─────────────────────────────────────────
async health(): Promise<HealthReport> {
try {
const legacyOk = await this.legacy.health();
if (!legacyOk) return { healthy: false, provider: this.provider, detail: "legacy health probe returned false" };
const v = await this.raw("SELECT version() AS v FORMAT JSON", {}, {});
const version = (v as { data?: { v?: string }[] })?.data?.[0]?.v;
return { healthy: true, provider: this.provider, version };
} catch (err) {
return {
healthy: false,
provider: this.provider,
detail: err instanceof SiaxError ? err.code : "unreachable",
};
}
}
// ── BI-P009 capability 2: executeMetric ──────────────────────────────────
async executeMetric(identity: AnalyticsIdentity, request: MetricRequest): Promise<EngineResult> {
this.assertIdentity(identity);
const budget = this.budgetFor(identity);
const release = this.gate.acquire(identity.tenant_id);
const started = Date.now();
const queryId = this.newQueryId(identity);
try {
const compiled = compileMetricQuery(
{
tenant_id: identity.tenant_id, // FORCED from the verified identity
project_id: identity.project_id,
dataset_id: request.dataset_id,
metric: request.metric,
aggregate: request.aggregate,
group_by: request.group_by,
filters: request.filters,
from_time: request.from_time,
to_time: request.to_time,
limit: request.limit,
table: request.table,
},
this.database,
this.catalogFor(identity),
budget.maxResultRows,
);
const json = (await this.raw(compiled.sql, compiled.params, clickhouseSettings(budget), queryId)) as
| ClickHouseJsonResponse
| undefined;
const groupNames = request.group_by ?? [];
const rows = (json?.data ?? []).map((r) => {
const groups: Record<string, string> = {};
groupNames.forEach((name, i) => {
const v = (r as Record<string, unknown>)[`grp${i}`];
groups[name] = v === undefined || v === null ? "" : String(v);
});
return {
...(groupNames.length ? { groups } : {}),
value: Number((r as { value: number | string }).value),
};
});
const limit = Math.max(1, Math.min(Math.floor(request.limit ?? 1000), budget.maxResultRows));
return {
rows,
query_id: queryId,
truncated: rows.length >= limit,
stats: {
duration_ms: Date.now() - started,
rows_read: json?.statistics?.rows_read,
bytes_read: json?.statistics?.bytes_read,
result_rows: rows.length,
},
};
} finally {
release();
this.inFlight.delete(queryId);
}
}
/**
* Delegating path to the PRESERVED implementation for the pre-existing
* `opt_metric_events` table. Kept so no existing caller has to migrate and
* the already-proven forced-predicate path stays in service.
*/
async executeLegacyMetric(
identity: AnalyticsIdentity,
request: { metric: string; aggregate: "sum" | "avg" | "min" | "max" | "count"; dataset_id?: string; group_by?: string; from_time?: string; to_time?: string; limit?: number },
): Promise<EngineResult> {
this.assertIdentity(identity);
const started = Date.now();
const rows = await this.legacy.queryMetric({
tenant_id: identity.tenant_id, // FORCED from the verified identity
metric: request.metric,
aggregate: request.aggregate,
dataset_id: request.dataset_id,
group_by: request.group_by,
from_time: request.from_time,
to_time: request.to_time,
limit: request.limit,
});
return {
rows: rows.map((r) => ({
...(r.group !== undefined && request.group_by ? { groups: { [request.group_by]: r.group } } : {}),
value: r.value,
})),
query_id: this.newQueryId(identity),
truncated: false,
stats: { duration_ms: Date.now() - started, result_rows: rows.length },
};
}
// ── BI-P009 capability 3: executeQuery (analyst SQL) ─────────────────────
async executeQuery(identity: AnalyticsIdentity, _request: AnalystQueryRequest): Promise<EngineResult> {
this.assertIdentity(identity);
// Deliberate, documented refusal — see AnalystQueryRequest's doc comment.
// Opening a raw-SQL surface before BI-P017 (SQL dialect contract) and
// BI-P033 (analytics authorization) exist would create precisely the
// "raw SQL attempt" hole BI-P011's adversarial list enumerates.
throw new SiaxError(
"CAPABILITY_UNAVAILABLE",
"0pt.analytics",
"analyst SQL is not enabled on the ClickHouse provider until BI-P017/BI-P033 land",
);
}
// ── BI-P009 capability 4: estimateQuery ──────────────────────────────────
async estimateQuery(identity: AnalyticsIdentity, request: MetricRequest): Promise<QueryEstimate> {
this.assertIdentity(identity);
const budget = this.budgetFor(identity);
const compiled = compileMetricQuery(
{
tenant_id: identity.tenant_id,
project_id: identity.project_id,
dataset_id: request.dataset_id,
metric: request.metric,
aggregate: request.aggregate,
group_by: request.group_by,
filters: request.filters,
from_time: request.from_time,
to_time: request.to_time,
limit: request.limit,
table: request.table,
},
this.database,
this.catalogFor(identity),
budget.maxResultRows,
);
// `EXPLAIN ESTIMATE` reports the parts/rows/marks the planner will read
// AFTER primary-key pruning — which is why tenant_id leads every ORDER BY
// in schema.ts: the estimate for one tenant does not include everyone.
const estimateSql = compiled.sql.replace(/ FORMAT JSON$/, "");
let rowsToRead = 0;
let partsRead = 0;
try {
const res = (await this.raw(
`EXPLAIN ESTIMATE ${estimateSql} FORMAT JSON`,
compiled.params,
planningSettings(budget),
)) as { data?: { rows?: number | string; parts?: number | string }[] } | undefined;
for (const row of res?.data ?? []) {
rowsToRead += Number(row.rows ?? 0);
partsRead += Number(row.parts ?? 0);
}
} catch (err) {
// An estimate that cannot be produced must not read as "cheap".
throw err instanceof SiaxError
? err
: new SiaxError("DEPENDENCY_UNAVAILABLE", "clickhouse", "could not estimate query");
}
// Rough byte estimate: ClickHouse's ESTIMATE reports rows, not bytes.
// 128 B/row is a deliberately PESSIMISTIC constant for these tables
// (measured mean in the BI-P010 benchmark is lower), so the gate errs
// toward refusing rather than admitting an oversized query.
const bytesToRead = rowsToRead * 128;
const within = rowsToRead <= budget.maxRowsToRead && bytesToRead <= budget.maxBytesToRead;
return {
estimated_rows_to_read: rowsToRead,
estimated_bytes_to_read: bytesToRead,
within_budget: within,
budget,
...(within
? {}
: {
reason:
`estimate ${rowsToRead} rows / ${bytesToRead} bytes over budget ` +
`(${budget.maxRowsToRead} rows / ${budget.maxBytesToRead} bytes), parts=${partsRead}`,
}),
};
}
// ── BI-P009 capability 5: cancelQuery ────────────────────────────────────
/**
* Two-sided cancellation. The client-side abort stops us waiting; the
* server-side KILL QUERY stops ClickHouse burning CPU. Cancelling is
* tenant-scoped: the query id embeds the tenant and is checked, so tenant A
* cannot cancel tenant B's work.
*/
async cancelQuery(identity: AnalyticsIdentity, queryId: string): Promise<boolean> {
this.assertIdentity(identity);
if (!queryId.startsWith(this.queryIdPrefix(identity))) {
throw new SiaxError("AUTHORIZATION_DENIED", "0pt.analytics", "query does not belong to this tenant");
}
this.inFlight.get(queryId)?.abort();
this.inFlight.delete(queryId);
try {
// KILL QUERY is a mutation of server state, so it cannot ride the
// readonly=1 settings the SELECT path uses.
await this.raw(
`KILL QUERY WHERE query_id = {killId:String} SYNC FORMAT JSON`,
{ param_killId: queryId },
{ max_execution_time: "10" },
);
return true;
} catch {
return false;
}
}
// ── BI-P009 capability 6: explain ────────────────────────────────────────
async explain(identity: AnalyticsIdentity, request: MetricRequest): Promise<QueryExplanation> {
this.assertIdentity(identity);
const budget = this.budgetFor(identity);
const compiled = compileMetricQuery(
{
tenant_id: identity.tenant_id,
project_id: identity.project_id,
dataset_id: request.dataset_id,
metric: request.metric,
aggregate: request.aggregate,
group_by: request.group_by,
filters: request.filters,
from_time: request.from_time,
to_time: request.to_time,
limit: request.limit,
table: request.table,
},
this.database,
this.catalogFor(identity),
budget.maxResultRows,
);
const sql = compiled.sql.replace(/ FORMAT JSON$/, "");
const res = (await this.raw(
`EXPLAIN PLAN indexes = 1 ${sql} FORMAT TSVRaw`,
compiled.params,
planningSettings(budget),
)) as string | undefined;
return {
plan: typeof res === "string" ? res : JSON.stringify(res ?? ""),
forced_tenant_predicate: FORCED_TENANT_PREDICATE,
table: compiled.table,
};
}
// ── BI-P009 capability 7: ingestProjection ───────────────────────────────
async ingestProjection(
identity: AnalyticsIdentity,
request: ProjectionIngestRequest,
): Promise<{ ingested: number }> {
this.assertIdentity(identity);
if (request.rows.length === 0) return { ingested: 0 };
const now = new Date();
const version = Math.floor(now.getTime());
const body = request.rows
.map((r) => {
if (!r.entity_key || !r.entity_key.trim()) {
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "every projection row needs an entity_key");
}
return JSON.stringify({
// tenant/project come from the VERIFIED identity, never the row —
// a row-supplied tenant_id is the INF0 Wave-2 bug class.
tenant_id: identity.tenant_id,
project_id: identity.project_id ?? "",
dataset_id: request.dataset_id,
entity_key: r.entity_key,
as_of: (r.as_of ?? now.toISOString()).replace("T", " ").replace(/Z$/, ""),
version: r.version ?? version,
attributes: r.attributes ?? {},
measures: r.measures ?? {},
});
})
.join("\n");
await this.raw(
`INSERT INTO ${this.database}.opt_dataset_projection FORMAT JSONEachRow`,
{},
{},
undefined,
body,
);
return { ingested: request.rows.length };
}
// ── BI-P009 capability 8: refreshProjection ──────────────────────────────
/**
* ReplacingMergeTree deduplicates on a background merge, so a read straight
* after an ingest can still see superseded rows. `OPTIMIZE ... FINAL` forces
* the merge for THIS tenant's partition only — the table is
* `PARTITION BY (tenant_id)` precisely so one tenant's refresh does not
* rewrite another tenant's parts (BI-P012's "without degrading unrelated
* tenant" applies to maintenance too, not only to queries).
*/
async refreshProjection(identity: AnalyticsIdentity, dataset_id: string): Promise<ProjectionRefreshResult> {
this.assertIdentity(identity);
await this.raw(
`OPTIMIZE TABLE ${this.database}.opt_dataset_projection PARTITION {tenant:String} FINAL`,
{ param_tenant: identity.tenant_id },
{ max_execution_time: "120" },
);
const compiled = compileProjectionQuery(
{ tenant_id: identity.tenant_id, project_id: identity.project_id, dataset_id, limit: 1 },
this.database,
1,
);
const countSql =
`SELECT count() AS value FROM ${this.database}.opt_dataset_projection FINAL ` +
`WHERE ${FORCED_TENANT_PREDICATE} AND dataset_id = {dataset:String} FORMAT JSON`;
const res = (await this.raw(countSql, compiled.params, clickhouseSettings(this.budgetFor(identity)))) as
| { data?: { value: number | string }[] }
| undefined;
return {
dataset_id,
row_count: Number(res?.data?.[0]?.value ?? 0),
refreshed_at: new Date().toISOString(),
};
}
// ── transport ────────────────────────────────────────────────────────────
private queryIdPrefix(identity: AnalyticsIdentity): string {
return `opt_${crypto.createHash("sha256").update(identity.tenant_id).digest("hex").slice(0, 12)}_`;
}
private newQueryId(identity: AnalyticsIdentity): string {
return `${this.queryIdPrefix(identity)}${crypto.randomUUID().replace(/-/g, "")}`;
}
/** DDL path: no readonly, no result parsing beyond errors. */
private async ddl(sql: string): Promise<void> {
await this.raw(sql, {}, { max_execution_time: "60" });
}
private async raw(
sql: string,
params: Record<string, string>,
settings: Record<string, string>,
queryId?: string,
body?: string,
): Promise<unknown> {
const url = new URL(this.baseUrl);
url.searchParams.set("query", sql);
for (const [k, v] of Object.entries(settings)) url.searchParams.set(k, v);
for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
if (queryId) url.searchParams.set("query_id", queryId);
const headers: Record<string, string> = {};
if (this.authHeader) headers["Authorization"] = this.authHeader;
const controller = new AbortController();
if (queryId) this.inFlight.set(queryId, controller);
const timeout = setTimeout(() => controller.abort(), this.timeoutMs);
let res: Response;
try {
res = await fetch(url.toString(), {
method: "POST",
headers,
body: body ?? "",
signal: controller.signal,
});
} catch (cause) {
throw new SiaxError("DEPENDENCY_UNAVAILABLE", "clickhouse", "clickhouse unreachable", { cause });
} finally {
clearTimeout(timeout);
}
const text = await res.text();
if (!res.ok) {
// ClickHouse error text can echo the query; keep only the numeric code
// so a caller never receives another tenant's identifiers back.
const code = /Code: (\d+)/.exec(text)?.[1] ?? String(res.status);
throw new SiaxError(
siaxErrorCodeForUpstreamStatus(res.status),
"clickhouse",
`query failed (CH code ${code})`,
);
}
if (!text) return undefined;
let parsed: unknown;
try {
parsed = JSON.parse(text);
} catch {
return text;
}
// DEFENCE IN DEPTH against a partial result that looks complete.
// ClickHouse normally answers a breached limit with an HTTP error status
// (measured on 24.8: 408 for max_execution_time, 500 for
// max_rows_to_read), which `res.ok` above already catches. But when rows
// have already been flushed, the JSON body can carry an `exception` key
// under a 200 — a truncated answer indistinguishable from a whole one.
// Treat any body carrying an exception as a failure, whatever the status.
if (parsed && typeof parsed === "object" && "exception" in parsed) {
const ex = String((parsed as { exception?: unknown }).exception ?? "");
const code = /Code: (\d+)/.exec(ex)?.[1] ?? "unknown";
throw new SiaxError("PERMANENT_FAILURE", "clickhouse", `query failed mid-stream (CH code ${code})`);
}
return parsed;
}
}
interface ClickHouseJsonResponse {
data?: Record<string, unknown>[];
statistics?: { rows_read?: number; bytes_read?: number; elapsed?: number };
}
+304
View File
@@ -0,0 +1,304 @@
/**
* src/analytics/engine/query-compiler.ts
* =======================================
* BI-P011 — CLICKHOUSE TENANT ISOLATION, layer 2:
* "`0pt` query compiler forces tenant".
*
* The compiler is the ONLY thing in this repo that turns an analytics request
* into SQL text. Its contract:
*
* * `tenant_id` is not an argument the caller may omit — it comes from the
* VERIFIED CL0UD identity (layer 1) and is written into the WHERE clause
* by this function, unconditionally, as a bound parameter.
* * caller-supplied VALUES are never spliced into SQL text. They travel as
* ClickHouse HTTP `param_*` query parameters.
* * caller-supplied IDENTIFIERS (a dimension name, a group-by column) are
* never spliced either. They are RESOLVED against a declared allow-list
* of `DimensionDefinition`s and only the resolved, already-known physical
* name reaches the SQL — so "dimension identifier injection" has no
* surface, because caller text never becomes an identifier.
* * there is no raw-SQL entry point. `compileMetricQuery` cannot express a
* UNION, a subquery, or a second table, so a caller cannot reach a table
* the compiler did not choose.
*
* WHY A SEPARATE MODULE from clickhouse-analytics.ts: that file's compiler
* already forces the tenant predicate correctly and BI-P009 says PRESERVE IT.
* This module is the generalised compiler for the BI-P010 tables; it repeats
* the guarantee rather than replacing it, and
* tests/analytics/tenant-predicate-regression.test.ts asserts BOTH.
*/
import { SiaxError } from "../../platform/errors";
import type { AnalyticsTable } from "./schema";
/** A physical identifier that has been VALIDATED, not merely received. */
const IDENT_RE = /^[A-Za-z_][A-Za-z0-9_]{0,62}$/;
export function assertSafeIdentifier(value: string, what: string): string {
if (!IDENT_RE.test(value)) {
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", `invalid ${what} identifier`);
}
return value;
}
export const METRIC_AGGREGATES = ["sum", "avg", "min", "max", "count", "count_distinct"] as const;
export type MetricAggregate = (typeof METRIC_AGGREGATES)[number];
/**
* The dimensions a caller is allowed to name, resolved from
* `DimensionDefinition` rows. The compiler looks a caller's dimension name up
* HERE; a miss is a rejection, never a pass-through.
*/
export interface DimensionCatalog {
/** caller-facing dimension name -> physical dimension key in dim_names. */
resolve(name: string): string | undefined;
}
export function dimensionCatalogFrom(pairs: Record<string, string>): DimensionCatalog {
const map = new Map(Object.entries(pairs));
return {
resolve(name: string): string | undefined {
// Own-property lookup on a Map — no prototype-chain surprises
// ("constructor", "__proto__" resolve to undefined, as they must).
return map.get(name);
},
};
}
export interface MetricQueryRequest {
/** Supplied by the engine from the verified identity, NOT by the caller. */
tenant_id: string;
project_id?: string;
dataset_id?: string;
metric: string;
aggregate: MetricAggregate;
/** Caller-facing dimension names; resolved through the catalog. */
group_by?: string[];
/** Equality filters on dimensions: caller-facing name -> value. */
filters?: Record<string, string>;
from_time?: string;
to_time?: string;
limit?: number;
/** Which BI-P010 table to read. Chosen by the engine, not the caller. */
table?: AnalyticsTable;
}
export interface CompiledQuery {
sql: string;
/** ClickHouse HTTP `param_*` entries. Values NEVER appear in `sql`. */
params: Record<string, string>;
/** The predicate the compiler forced, for assertion in tests. */
forcedTenantPredicate: string;
table: AnalyticsTable;
}
/**
* The exact predicate text every compiled query must contain. Exported so a
* regression test can assert on the literal string — if someone deletes the
* forcing, the test fails on a missing substring rather than on a subtle
* behavioural difference that a mocked engine would happily hide.
*/
export const FORCED_TENANT_PREDICATE = "tenant_id = {tenant:String}";
const AGG_SQL: Record<MetricAggregate, string> = {
sum: "sum(value)",
avg: "avg(value)",
min: "min(value)",
max: "max(value)",
count: "count()",
count_distinct: "uniqExact(event_id)",
};
const MAX_GROUP_BY = 8;
/**
* Compile a closed-DSL metric request into tenant-forced SQL.
*
* @param req the request; `tenant_id` MUST already be the verified identity.
* @param db the analytics database name (engine-controlled, validated).
* @param catalog the tenant's declared dimensions.
* @param maxResultRows the governance ceiling (BI-P012); `limit` is clamped.
*/
export function compileMetricQuery(
req: MetricQueryRequest,
db: string,
catalog: DimensionCatalog,
maxResultRows: number,
): CompiledQuery {
if (!req.tenant_id || !req.tenant_id.trim()) {
// Fail closed. A blank tenant must never compile to "no predicate".
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "tenant_id is required on every analytics query");
}
assertSafeIdentifier(db, "database");
const table: AnalyticsTable = req.table ?? "opt_analytics_event";
assertSafeIdentifier(table, "table");
if (!METRIC_AGGREGATES.includes(req.aggregate)) {
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "unsupported aggregate");
}
const params: Record<string, string> = {
param_tenant: req.tenant_id,
param_metric: req.metric,
};
// ── the forced predicate. First condition, always, no branch. ─────────
const conditions: string[] = [FORCED_TENANT_PREDICATE, "metric = {metric:String}"];
if (req.project_id) {
conditions.push("project_id = {project:String}");
params.param_project = req.project_id;
}
if (req.dataset_id) {
conditions.push("dataset_id = {dataset:String}");
params.param_dataset = req.dataset_id;
}
if (req.from_time) {
conditions.push("event_time >= parseDateTime64BestEffort({fromTime:String})");
params.param_fromTime = req.from_time;
}
if (req.to_time) {
conditions.push("event_time < parseDateTime64BestEffort({toTime:String})");
params.param_toTime = req.to_time;
}
// ── filters: names resolved through the catalog, values parameterized ──
let filterIdx = 0;
for (const [name, value] of Object.entries(req.filters ?? {})) {
const physical = catalog.resolve(name);
if (physical === undefined) {
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", `unknown filter dimension "${name}"`);
}
// `physical` came from the catalog, but validate anyway: a corrupted or
// hostile catalog row must not become an injection vector either.
assertSafeIdentifier(physical, "dimension");
const nameKey = `f${filterIdx}n`;
const valKey = `f${filterIdx}v`;
conditions.push(
`dim_values[indexOf(dim_names, {${nameKey}:String})] = {${valKey}:String}`,
);
params[`param_${nameKey}`] = physical;
params[`param_${valKey}`] = value;
filterIdx += 1;
}
// ── group by: same resolution, plus a hard cardinality ceiling ─────────
const groupBy = req.group_by ?? [];
if (groupBy.length > MAX_GROUP_BY) {
throw new SiaxError(
"INVALID_REQUEST",
"0pt.analytics",
`at most ${MAX_GROUP_BY} group-by dimensions (got ${groupBy.length})`,
);
}
const selectParts: string[] = [];
const groupParts: string[] = [];
groupBy.forEach((name, i) => {
const physical = catalog.resolve(name);
if (physical === undefined) {
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", `unknown group_by dimension "${name}"`);
}
assertSafeIdentifier(physical, "dimension");
const key = `g${i}`;
// The ALIAS is compiler-generated (`grp0`, `grp1`), never caller text.
selectParts.push(`dim_values[indexOf(dim_names, {${key}:String})] AS grp${i}`);
groupParts.push(`grp${i}`);
params[`param_${key}`] = physical;
});
const limit = Math.max(1, Math.min(Math.floor(req.limit ?? 1000), maxResultRows));
const sql =
`SELECT ${selectParts.length ? selectParts.join(", ") + ", " : ""}${AGG_SQL[req.aggregate]} AS value ` +
`FROM ${db}.${table} ` +
`WHERE ${conditions.join(" AND ")} ` +
(groupParts.length ? `GROUP BY ${groupParts.join(", ")} ORDER BY value DESC ` : "") +
`LIMIT ${limit} FORMAT JSON`;
return { sql, params, forcedTenantPredicate: FORCED_TENANT_PREDICATE, table };
}
/**
* Compile a projection lookup (SERVING tier, BI-P010's opt_dataset_projection).
* Same forcing, different table and key.
*/
export function compileProjectionQuery(
req: { tenant_id: string; project_id?: string; dataset_id: string; entity_keys?: string[]; limit?: number },
db: string,
maxResultRows: number,
): CompiledQuery {
if (!req.tenant_id || !req.tenant_id.trim()) {
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "tenant_id is required on every analytics query");
}
assertSafeIdentifier(db, "database");
const params: Record<string, string> = {
param_tenant: req.tenant_id,
param_dataset: req.dataset_id,
};
const conditions = [FORCED_TENANT_PREDICATE, "dataset_id = {dataset:String}"];
if (req.project_id) {
conditions.push("project_id = {project:String}");
params.param_project = req.project_id;
}
if (req.entity_keys && req.entity_keys.length > 0) {
conditions.push("has({entityKeys:Array(String)}, entity_key)");
params.param_entityKeys = JSON.stringify(req.entity_keys);
}
const limit = Math.max(1, Math.min(Math.floor(req.limit ?? 1000), maxResultRows));
const sql =
`SELECT entity_key, as_of, version, attributes, measures ` +
`FROM ${db}.opt_dataset_projection ` +
`WHERE ${conditions.join(" AND ")} ` +
`ORDER BY entity_key LIMIT ${limit} FORMAT JSON`;
return { sql, params, forcedTenantPredicate: FORCED_TENANT_PREDICATE, table: "opt_dataset_projection" };
}
/**
* BI-P011 layer-3 bootstrap: the DDL for a RESTRICTED per-tenant ClickHouse
* user with a row policy pinning `tenant_id`.
*
* This is the layer that survives a compiler bug. The compiler forcing the
* predicate is layer 2; if a future refactor drops it, layer 3 still returns
* zero foreign rows because the SERVER filters them. The BI-P011 proof runs
* the adversarial suite twice — once through an unrestricted user (hole open)
* and once through this restricted user (hole closed) — because a claim of
* defence in depth that has never been tested with one layer removed is not
* evidence.
*
* `tenantId` is validated as an identifier-safe token AND embedded as a
* quoted literal; ClickHouse's CREATE ROW POLICY takes no bound parameters,
* so validation is the only protection available and it is applied strictly.
*/
export function restrictedTenantUserDdl(args: {
db: string;
tenantId: string;
userName: string;
password: string;
tables: readonly string[];
settingsProfile?: string;
}): string[] {
assertSafeIdentifier(args.db, "database");
assertSafeIdentifier(args.userName, "user");
// A tenant id is `t_<slug>` in this estate; enforce that shape strictly
// rather than trusting the caller.
if (!/^[A-Za-z0-9_-]{1,64}$/.test(args.tenantId)) {
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "tenant id is not policy-safe");
}
if (args.password.includes("'") || args.password.includes("\\")) {
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "password contains characters that cannot be embedded");
}
const profile = args.settingsProfile;
if (profile) assertSafeIdentifier(profile, "settings profile");
const stmts: string[] = [
`CREATE USER IF NOT EXISTS ${args.userName} IDENTIFIED WITH sha256_password BY '${args.password}'` +
(profile ? ` SETTINGS PROFILE ${profile}` : ` SETTINGS readonly = 1`),
];
for (const table of args.tables) {
assertSafeIdentifier(table, "table");
stmts.push(
`CREATE ROW POLICY IF NOT EXISTS ${args.userName}_${table}_rp ON ${args.db}.${table} ` +
`FOR SELECT USING tenant_id = '${args.tenantId}' TO ${args.userName}`,
);
stmts.push(`GRANT SELECT ON ${args.db}.${table} TO ${args.userName}`);
}
return stmts;
}
+222
View File
@@ -0,0 +1,222 @@
/**
* src/analytics/engine/resource-governance.ts
* ============================================
* BI-P012 — CLICKHOUSE RESOURCE GOVERNANCE.
*
* Every setting BI-P012 names, in one place, with a value:
*
* max_execution_time max_result_rows max_rows_to_read
* max_bytes_to_read max_memory_usage max_concurrent_queries
*
* Three layers, and the order matters:
*
* 1. SERVER-SIDE SETTINGS travel with every statement on the HTTP
* interface. ClickHouse enforces them; the client cannot opt out
* because the client is what writes them. `readonly=1` on top means a
* SELECT cannot re-set them mid-query (readonly=1 forbids SETTINGS
* changes as well as writes — that is the whole reason it is here and
* not merely `readonly=2`).
* 2. `max_concurrent_queries` is NOT a per-query setting in ClickHouse; it
* is a per-USER quota. `concurrencySlotSql` renders the CREATE
* SETTINGS PROFILE that carries it, applied to the tenant's restricted
* user (BI-P011 layer 3) so one tenant's fan-out cannot consume the
* whole server.
* 3. A local admission gate (`ConcurrencyGate`) refuses the N+1st
* in-flight query per tenant before it ever reaches the network. Two
* layers because the ClickHouse quota is per-user and one 0pt process
* can still queue work faster than the server rejects it.
*
* Budget tiers exist because "one limit for everyone" is how a small tenant
* subsidises a query bomb. `INTERACTIVE` is the dashboard default and is
* deliberately mean.
*/
import { SiaxError } from "../../platform/errors";
export interface ResourceBudget {
/** Seconds. ClickHouse `max_execution_time`. */
maxExecutionTimeSec: number;
/** Rows returned to the caller. ClickHouse `max_result_rows`. */
maxResultRows: number;
/** Rows SCANNED (not returned). ClickHouse `max_rows_to_read`. */
maxRowsToRead: number;
/** Bytes scanned. ClickHouse `max_bytes_to_read`. */
maxBytesToRead: number;
/** Bytes of RAM per query. ClickHouse `max_memory_usage`. */
maxMemoryUsageBytes: number;
/** In-flight queries per tenant. Enforced by quota + local gate. */
maxConcurrentQueries: number;
}
export const BUDGET_TIERS = ["INTERACTIVE", "ANALYTICAL", "BATCH"] as const;
export type BudgetTier = (typeof BUDGET_TIERS)[number];
/**
* Defaults. Chosen against the shape of the analytics tables in
* `src/analytics/engine/schema.ts`, not plucked from the air:
* INTERACTIVE must answer a dashboard panel inside a human's patience budget,
* so it caps the SCAN, which is what actually costs time on a MergeTree.
*/
export const DEFAULT_BUDGETS: Record<BudgetTier, ResourceBudget> = {
INTERACTIVE: {
maxExecutionTimeSec: 10,
maxResultRows: 10_000,
maxRowsToRead: 50_000_000,
maxBytesToRead: 2 * 1024 * 1024 * 1024,
maxMemoryUsageBytes: 2 * 1024 * 1024 * 1024,
maxConcurrentQueries: 8,
},
ANALYTICAL: {
maxExecutionTimeSec: 60,
maxResultRows: 500_000,
maxRowsToRead: 2_000_000_000,
maxBytesToRead: 32 * 1024 * 1024 * 1024,
maxMemoryUsageBytes: 8 * 1024 * 1024 * 1024,
maxConcurrentQueries: 4,
},
BATCH: {
maxExecutionTimeSec: 900,
maxResultRows: 10_000_000,
maxRowsToRead: 50_000_000_000,
maxBytesToRead: 512 * 1024 * 1024 * 1024,
maxMemoryUsageBytes: 24 * 1024 * 1024 * 1024,
maxConcurrentQueries: 2,
},
};
/**
* A CL0UD-issued quota may only ever TIGHTEN a tier, never loosen it. A
* control-plane bug that hands out a generous number must not be able to
* disarm the engine's own ceiling.
*/
export function applyQuota(base: ResourceBudget, quota: Partial<ResourceBudget> | undefined): ResourceBudget {
if (!quota) return { ...base };
const pick = (b: number, q: number | undefined): number =>
typeof q === "number" && Number.isFinite(q) && q > 0 ? Math.min(b, q) : b;
return {
maxExecutionTimeSec: pick(base.maxExecutionTimeSec, quota.maxExecutionTimeSec),
maxResultRows: pick(base.maxResultRows, quota.maxResultRows),
maxRowsToRead: pick(base.maxRowsToRead, quota.maxRowsToRead),
maxBytesToRead: pick(base.maxBytesToRead, quota.maxBytesToRead),
maxMemoryUsageBytes: pick(base.maxMemoryUsageBytes, quota.maxMemoryUsageBytes),
maxConcurrentQueries: pick(base.maxConcurrentQueries, quota.maxConcurrentQueries),
};
}
/**
* The settings that ride along on the HTTP request. `readonly: "1"` is what
* makes them un-overridable from inside the query text.
*
* `timeout_overflow_mode` / `result_overflow_mode` / `read_overflow_mode` are
* set to `throw` EXPLICITLY: ClickHouse's default for some of these is
* `break`, which silently returns a PARTIAL result. A partial result that
* looks complete is the analytics equivalent of a false green, so the engine
* refuses it.
*/
export function clickhouseSettings(budget: ResourceBudget): Record<string, string> {
return {
readonly: "1",
max_execution_time: String(budget.maxExecutionTimeSec),
timeout_overflow_mode: "throw",
max_result_rows: String(budget.maxResultRows),
result_overflow_mode: "throw",
max_rows_to_read: String(budget.maxRowsToRead),
max_bytes_to_read: String(budget.maxBytesToRead),
read_overflow_mode: "throw",
max_memory_usage: String(budget.maxMemoryUsageBytes),
};
}
/**
* Settings for a PLANNER call (`EXPLAIN ESTIMATE`, `EXPLAIN PLAN`).
*
* Measured, not assumed: running `EXPLAIN ESTIMATE` under the caller's own
* `max_rows_to_read` makes ClickHouse refuse the ESTIMATE itself with code 158
* whenever the estimate would be over budget — i.e. exactly the case the
* estimate exists to detect. An estimator that cannot describe an expensive
* query is useless, so the read ceilings are deliberately NOT applied here.
* The planner still runs `readonly = 1` under a short deadline and a memory
* cap, so this is a narrower envelope, never an unbounded one.
*/
export function planningSettings(budget: ResourceBudget): Record<string, string> {
return {
readonly: "1",
max_execution_time: String(Math.min(budget.maxExecutionTimeSec, 15)),
timeout_overflow_mode: "throw",
max_memory_usage: String(budget.maxMemoryUsageBytes),
};
}
/** Every setting name BI-P012 requires, for the DoD assertion. */
export const REQUIRED_GOVERNANCE_SETTINGS = [
"max_execution_time",
"max_result_rows",
"max_rows_to_read",
"max_bytes_to_read",
"max_memory_usage",
"max_concurrent_queries",
] as const;
/**
* `max_concurrent_queries` lives on a SETTINGS PROFILE, not on a statement.
* Rendered here so BI-P011's restricted-user bootstrap and BI-P012's budget
* come from one source.
*/
export function concurrencyProfileSql(profileName: string, budget: ResourceBudget): string[] {
return [
`CREATE SETTINGS PROFILE IF NOT EXISTS ${profileName} SETTINGS ` +
`max_execution_time = ${budget.maxExecutionTimeSec}, ` +
`max_result_rows = ${budget.maxResultRows}, ` +
`max_rows_to_read = ${budget.maxRowsToRead}, ` +
`max_bytes_to_read = ${budget.maxBytesToRead}, ` +
`max_memory_usage = ${budget.maxMemoryUsageBytes}, ` +
`readonly = 1`,
`CREATE QUOTA IF NOT EXISTS ${profileName}_quota ` +
`FOR INTERVAL 1 SECOND MAX queries = ${budget.maxConcurrentQueries * 20} ` +
`TO ALL EXCEPT default`,
];
}
/**
* Local admission control. One instance per process; keyed by tenant so a
* noisy tenant is throttled without touching anyone else's slots — which is
* exactly BI-P012's DoD wording ("without degrading unrelated tenant").
*/
export class ConcurrencyGate {
private readonly inFlight = new Map<string, number>();
constructor(private readonly limitFor: (tenantId: string) => number) {}
/** Returns a release function. Throws RATE_LIMITED when the tenant is full. */
acquire(tenantId: string): () => void {
const limit = this.limitFor(tenantId);
const current = this.inFlight.get(tenantId) ?? 0;
if (current >= limit) {
throw new SiaxError(
"RATE_LIMITED",
"0pt.analytics",
`tenant already has ${current} analytics queries in flight (limit ${limit})`,
);
}
this.inFlight.set(tenantId, current + 1);
let released = false;
return () => {
if (released) return;
released = true;
const n = (this.inFlight.get(tenantId) ?? 1) - 1;
if (n <= 0) this.inFlight.delete(tenantId);
else this.inFlight.set(tenantId, n);
};
}
/** Observability: in-flight count for a tenant (0 when idle). */
inFlightFor(tenantId: string): number {
return this.inFlight.get(tenantId) ?? 0;
}
/** Total in-flight across all tenants. */
totalInFlight(): number {
let n = 0;
for (const v of this.inFlight.values()) n += v;
return n;
}
}
+169
View File
@@ -0,0 +1,169 @@
/**
* src/analytics/engine/schema.ts
* ===============================
* BI-P010 — CLICKHOUSE PRODUCTION SCHEMA.
*
* BI-P010's explicit warning is "Avoid one giant unbounded table for every
* analytics use case." The existing `opt_metric_events` (owned by
* src/analytics/clickhouse-analytics.ts, PRESERVED untouched) is exactly one
* table for one job: low-cardinality capability metrics. It stays, and it is
* NOT extended into the catch-all.
*
* Four tables, four jobs, four retention profiles:
*
* opt_metric_events (PRE-EXISTING, not defined here) — capability
* metrics, Map(String,String) dimensions, 90d.
* opt_analytics_event — high-volume per-tenant event stream. Wide
* dimension arrays, SHORT TTL (raw events are not
* the serving tier; they age into the rollups).
* opt_metric_daily — AggregatingMergeTree rollup fed by a
* MATERIALIZED VIEW off opt_analytics_event.
* This is what a dashboard time-series reads;
* long TTL, tiny footprint.
* opt_dataset_projection — the SERVING tier (BI-P003) for a CURATED
* dataset: one row per (tenant, dataset, entity),
* ReplacingMergeTree so a refresh is idempotent.
*
* Why the split is load-bearing, not cosmetic:
* * different ORDER BY keys. A dashboard filtering (tenant, metric, day)
* and a projection looked up by (tenant, dataset, entity_key) cannot
* share one primary key without one of them degenerating to a full scan.
* * different TTLs. Raw events must expire fast; rollups must not.
* * different cardinality. `LowCardinality(String)` is a win on
* tenant/metric/dataset and a LOSS on a free-form entity key, so they
* cannot be the same column type in the same table.
*
* EVERY table leads its ORDER BY with `tenant_id`. That is not stylistic: it
* is what makes the forced tenant predicate (BI-P011) a primary-key range
* scan instead of a filter applied after reading everyone's rows, and it is
* what makes a per-tenant row policy cheap.
*
* LEGAL BOUNDARY (inherited verbatim from clickhouse-analytics.ts): this
* schema is for generic 0pt analytics. It MUST NEVER be pointed at the b0ts
* ClickHouse pipelines or quarantined market data.
*/
export interface AnalyticsSchemaOptions {
database: string;
/** Raw event retention. Deliberately short — rollups carry the history. */
eventRetentionDays?: number;
/** Rollup retention. */
rollupRetentionDays?: number;
/** Serving-projection retention. */
projectionRetentionDays?: number;
}
/** Physical tables this module owns (opt_metric_events is NOT one of them). */
export const ANALYTICS_TABLES = [
"opt_analytics_event",
"opt_metric_daily",
"opt_dataset_projection",
] as const;
export type AnalyticsTable = (typeof ANALYTICS_TABLES)[number];
/** The materialized view wiring opt_analytics_event -> opt_metric_daily. */
export const ANALYTICS_MATERIALIZED_VIEWS = ["opt_metric_daily_mv"] as const;
/**
* DDL, in dependency order. Every statement is CREATE ... IF NOT EXISTS so
* applying it to a live database is idempotent (the same contract
* `ClickHouseAnalytics.ensureSchema` already keeps).
*/
export function analyticsSchemaDdl(opts: AnalyticsSchemaOptions): string[] {
const db = opts.database;
const eventTtl = Math.floor(opts.eventRetentionDays ?? 30);
const rollupTtl = Math.floor(opts.rollupRetentionDays ?? 1095);
const projectionTtl = Math.floor(opts.projectionRetentionDays ?? 365);
return [
`CREATE DATABASE IF NOT EXISTS ${db}`,
// ── raw event stream ────────────────────────────────────────────────
// Dimensions as parallel arrays (not Map) so ClickHouse can prune on
// dimension VALUES via the bloom filter index; Map(String,String) forces
// reading the whole map. `event_id` makes deduplication possible on
// replay without a ReplacingMergeTree's merge latency.
`CREATE TABLE IF NOT EXISTS ${db}.opt_analytics_event (
tenant_id LowCardinality(String),
project_id LowCardinality(String),
dataset_id LowCardinality(String),
metric LowCardinality(String),
event_id String,
event_time DateTime64(3, 'UTC'),
value Float64,
dim_names Array(LowCardinality(String)),
dim_values Array(String),
ingested_at DateTime64(3, 'UTC') DEFAULT now64(3),
INDEX idx_dim_values dim_values TYPE bloom_filter(0.01) GRANULARITY 4
)
ENGINE = MergeTree()
PARTITION BY toYYYYMM(event_time)
ORDER BY (tenant_id, project_id, dataset_id, metric, event_time)
TTL toDateTime(event_time) + INTERVAL ${eventTtl} DAY
SETTINGS index_granularity = 8192`,
// ── daily rollup (serving tier for time series) ─────────────────────
`CREATE TABLE IF NOT EXISTS ${db}.opt_metric_daily (
tenant_id LowCardinality(String),
project_id LowCardinality(String),
dataset_id LowCardinality(String),
metric LowCardinality(String),
day Date,
sum_state AggregateFunction(sum, Float64),
count_state AggregateFunction(count, Float64),
min_state AggregateFunction(min, Float64),
max_state AggregateFunction(max, Float64)
)
ENGINE = AggregatingMergeTree()
PARTITION BY toYYYYMM(day)
ORDER BY (tenant_id, project_id, dataset_id, metric, day)
TTL day + INTERVAL ${rollupTtl} DAY`,
`CREATE MATERIALIZED VIEW IF NOT EXISTS ${db}.opt_metric_daily_mv
TO ${db}.opt_metric_daily AS
SELECT
tenant_id,
project_id,
dataset_id,
metric,
toDate(event_time) AS day,
sumState(value) AS sum_state,
countState(value) AS count_state,
minState(value) AS min_state,
maxState(value) AS max_state
FROM ${db}.opt_analytics_event
GROUP BY tenant_id, project_id, dataset_id, metric, day`,
// ── serving projection for a CURATED dataset ────────────────────────
// ReplacingMergeTree(version) so refreshProjection is idempotent: the
// same (tenant, dataset, entity_key) re-ingested at a higher version
// supersedes rather than duplicates.
`CREATE TABLE IF NOT EXISTS ${db}.opt_dataset_projection (
tenant_id LowCardinality(String),
project_id LowCardinality(String),
dataset_id LowCardinality(String),
entity_key String,
as_of DateTime64(3, 'UTC'),
version UInt64,
attributes Map(String, String),
measures Map(String, Float64),
INDEX idx_entity entity_key TYPE bloom_filter(0.01) GRANULARITY 4
)
ENGINE = ReplacingMergeTree(version)
PARTITION BY (tenant_id)
ORDER BY (tenant_id, project_id, dataset_id, entity_key)
TTL toDateTime(as_of) + INTERVAL ${projectionTtl} DAY`,
];
}
/**
* Statement that proves a table's ORDER BY leads with tenant_id — used by the
* BI-P011 test suite so "tenant-first primary key" is asserted against the
* live server, not against this file's own comments.
*/
export function sortingKeySql(db: string, table: string): string {
return (
`SELECT sorting_key FROM system.tables ` +
`WHERE database = '${db}' AND name = '${table}' FORMAT TSV`
);
}
@@ -0,0 +1,180 @@
/**
* src/analytics/engine/siax-analytics-engine.ts
* ==============================================
* BI-P009 — the provider-neutral analytics engine interface.
*
* Eight capabilities, named exactly as BI-P009 lists them:
*
* health executeMetric executeQuery estimateQuery
* explain cancelQuery ingestProjection refreshProjection
*
* ClickHouse is the CANONICAL provider (BI-P009); DuckDB (BI-P014) and Trino
* (BI-P015) implement the same interface later. Nothing above this interface
* may import a provider module directly — that is what makes "replaceable
* provider" true rather than aspirational.
*
* NORMATIVE OWNERSHIP (BI program §0): every method takes an
* `AnalyticsIdentity` that the CALLER cannot forge, because it is produced by
* CL0UD verification upstream, not by request-body parsing. Product repos
* (VehIQ) call an API in front of this interface; they never hold a provider
* credential and never construct an engine.
*/
import type { CompiledQuery, MetricAggregate } from "./query-compiler";
import type { AnalyticsTable } from "./schema";
import type { BudgetTier, ResourceBudget } from "./resource-governance";
/**
* The VERIFIED caller. `tenant_id` here is the one the engine forces into
* every predicate — it comes from CL0UD identity verification (BI-P011 layer
* 1), never from a request body. This estate has already been burned once by
* a tenant read out of a request body (INF0 Wave 2, cross-tenant RAG access),
* which is why the type is separate from the request rather than a field on
* it.
*/
export interface AnalyticsIdentity {
tenant_id: string;
project_id?: string;
/** CL0UD principal, for audit + per-principal quota. */
principal_id: string;
correlation_id?: string;
/** Budget tier granted by CL0UD entitlements (BI-P012). */
budget_tier?: BudgetTier;
/** CL0UD quota, which may only TIGHTEN the tier. */
quota?: Partial<ResourceBudget>;
}
export interface MetricRequest {
metric: string;
aggregate: MetricAggregate;
dataset_id?: string;
group_by?: string[];
filters?: Record<string, string>;
from_time?: string;
to_time?: string;
limit?: number;
table?: AnalyticsTable;
}
export interface AnalyticsRow {
/** Resolved group values, keyed by the CALLER-FACING dimension name. */
groups?: Record<string, string>;
value: number;
}
export interface EngineResult {
rows: AnalyticsRow[];
/** Engine-side id; the handle `cancelQuery` needs. */
query_id: string;
/** True when the governance ceiling cut the result short (BI-P012). */
truncated: boolean;
stats: QueryStats;
}
export interface QueryStats {
duration_ms: number;
rows_read?: number;
bytes_read?: number;
result_rows: number;
}
export interface QueryEstimate {
/** Rows the engine expects to SCAN. */
estimated_rows_to_read: number;
estimated_bytes_to_read: number;
/** False when the estimate exceeds the caller's budget (BI-P012). */
within_budget: boolean;
budget: ResourceBudget;
/** Why it was refused, when `within_budget` is false. */
reason?: string;
}
export interface QueryExplanation {
/** Provider's plan text. Never contains caller-supplied SQL fragments. */
plan: string;
/** The predicate the compiler forced — assertable, not narrated. */
forced_tenant_predicate: string;
table: string;
}
export interface ProjectionRow {
entity_key: string;
as_of?: string;
version?: number;
attributes?: Record<string, string>;
measures?: Record<string, number>;
}
export interface ProjectionIngestRequest {
dataset_id: string;
rows: ProjectionRow[];
}
export interface ProjectionRefreshResult {
dataset_id: string;
/** Rows visible after the refresh settled. */
row_count: number;
refreshed_at: string;
}
export interface HealthReport {
healthy: boolean;
provider: string;
/** Provider version when reachable. */
version?: string;
/** Short operator-facing reason. Never a raw engine body. */
detail?: string;
}
/**
* A raw, ANALYST-facing query (BI-P017 will govern its SQL dialect). It is a
* SEPARATE method from `executeMetric` on purpose: `executeMetric` has no
* raw-SQL surface at all, and keeping them apart means the closed DSL cannot
* silently grow one.
*
* The canonical ClickHouse provider in this pass DELIBERATELY refuses this
* method (`CAPABILITY_UNAVAILABLE`): opening an analyst SQL surface before
* BI-P017's dialect contract and BI-P033's authorization exist would be
* exactly the "raw SQL attempt" BI-P011's adversarial list names. Declaring
* it and refusing it is honest; implementing it early is not.
*/
export interface AnalystQueryRequest {
sql: string;
dataset_ids: string[];
limit?: number;
}
export interface SiaxAnalyticsEngine {
readonly provider: string;
health(): Promise<HealthReport>;
executeMetric(identity: AnalyticsIdentity, request: MetricRequest): Promise<EngineResult>;
executeQuery(identity: AnalyticsIdentity, request: AnalystQueryRequest): Promise<EngineResult>;
estimateQuery(identity: AnalyticsIdentity, request: MetricRequest): Promise<QueryEstimate>;
cancelQuery(identity: AnalyticsIdentity, queryId: string): Promise<boolean>;
explain(identity: AnalyticsIdentity, request: MetricRequest): Promise<QueryExplanation>;
ingestProjection(identity: AnalyticsIdentity, request: ProjectionIngestRequest): Promise<{ ingested: number }>;
refreshProjection(identity: AnalyticsIdentity, dataset_id: string): Promise<ProjectionRefreshResult>;
}
/** The eight capability names, for the provider contract test. */
export const SIAX_ANALYTICS_ENGINE_CAPABILITIES = [
"health",
"executeMetric",
"executeQuery",
"estimateQuery",
"cancelQuery",
"explain",
"ingestProjection",
"refreshProjection",
] as const;
export type SiaxAnalyticsEngineCapability = (typeof SIAX_ANALYTICS_ENGINE_CAPABILITIES)[number];
/** Re-exported so provider modules do not reach across into the compiler. */
export type { CompiledQuery };