feat(analytics): BI-P002..BI-P012 analytics core — domain model, tiers, contracts, Arrow, ClickHouse engine, isolation, governance
CI (SIAX Cloud) / security (push) Successful in 41s
Security Scan (SIAX Cloud) / security (push) Successful in 40s
CI (SIAX Cloud) / integration (push) Successful in 52s
CI (SIAX Cloud) / quality (push) Successful in 1m5s
CI (SIAX Cloud) / security (pull_request) Successful in 31s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 29s
CI (SIAX Cloud) / integration (pull_request) Successful in 45s
CI (SIAX Cloud) / quality (pull_request) Successful in 53s
CI (SIAX Cloud) / security (push) Successful in 41s
Security Scan (SIAX Cloud) / security (push) Successful in 40s
CI (SIAX Cloud) / integration (push) Successful in 52s
CI (SIAX Cloud) / quality (push) Successful in 1m5s
CI (SIAX Cloud) / security (pull_request) Successful in 31s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 29s
CI (SIAX Cloud) / integration (pull_request) Successful in 45s
CI (SIAX Cloud) / quality (pull_request) Successful in 53s
Implements the eight BI program prompts that make 0pt the canonical SIAX
analytics capability. Every file is NEW: not one existing file is modified,
and src/analytics/clickhouse-analytics.ts is byte-identical to main.
BI-P002 — the nineteen canonical entities, verbatim, with tenant_id +
project_id on every tenant-owned record. sql/008 gives each one a table.
The four version entities BI-P002 names as immutable are enforced by BEFORE
UPDATE/DELETE triggers, not by a TypeScript `readonly` — proven by direct SQL
against a live Postgres.
BI-P003 — RAW/NORMALIZED/CURATED/SERVING with the read rule enforced:
assertReadableByBiSurface refuses a RAW dataset to a dashboard unless a
recorded exception exists. All eight required declarations are NOT NULL.
BI-P004 — DatasetContract and a fail-closed compatibility gate. Fifteen
breaking-change classes are each tested twice: refused under a minor bump,
accepted under a major one.
BI-P005 — Arrow IPC alongside JSON, with a hand-written FlatBuffers writer so
no npm dependency (or CVE surface) is added. Conformance comes from three
independent implementations, not from self-agreement: pyarrow 25.0.1 and
DuckDB 1.5.5 read identical values from both ClickHouse's Arrow and ours, and
ClickHouse's own Arrow C++ ingests our 20,000-row batch with exact aggregates.
BI-P009 — SiaxAnalyticsEngine with all eight capabilities. The existing
ClickHouseAnalytics is COMPOSED, never replaced. executeQuery (analyst SQL) is
declared and deliberately refused with CAPABILITY_UNAVAILABLE until BI-P017 and
BI-P033 land, rather than opening the raw-SQL hole BI-P011 enumerates.
BI-P010 — four purpose-built tables, not one giant one, and the difference is
measured against the counterfactual: 48.8ms/40,960 rows vs 174.4ms/5,400,000
rows for the same question; 5/660 granules vs 660/1149.
BI-P011 — layered isolation, each layer proven with the layer above removed.
Without the forced predicate the same statement returns 50,050 (both tenants);
with it, t_alpha gets 50. Without a row policy an unrestricted user sees 2
tenants; the restricted user running the IDENTICAL statement sees 1.
BI-P012 — every named limit, fired individually and identified by ClickHouse
error code, plus the half usually skipped: an unrelated tenant is timed while
a real CPU bomb is in flight and must still return the CORRECT answer.
Three defects found by writing tests that can fail:
* `CHECK (array_length(purposes,1) >= 1)` can NEVER fail — array_length is
NULL for '{}' and a NULL CHECK passes. Fixed with coalesce(); the test
that caught it is pinned.
* EXPLAIN ESTIMATE run under the caller's own max_rows_to_read fails with
code 158 exactly when the estimate would be over budget. Planner calls now
use a narrower envelope.
* result_overflow_mode=break returns HTTP 200 with 65,409 of 100,000 rows
and no indication it is short. Every SELECT ships `throw`.
Verified rather than assumed: server7's ClickHouse holds `opt_scraped` (51
tables, 407,793,471 rows, `persons`/`cz_owners` included) — the b0ts plane this
module is prohibited from touching — has no `opt_analytics` database, zero
tenant_id columns and zero row policies. Nothing there was modified; every
proof ran against disposable containers.
Evidence: evidence/bi-p002-p012/{test-transcript.txt,bi-p010-benchmark.json,
bi-p005-arrow-roundtrip.json,server7-clickhouse-probe.md}
Docs: docs/analytics/BI_ANALYTICS_CORE.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,304 @@
|
||||
/**
|
||||
* src/analytics/engine/query-compiler.ts
|
||||
* =======================================
|
||||
* BI-P011 — CLICKHOUSE TENANT ISOLATION, layer 2:
|
||||
* "`0pt` query compiler forces tenant".
|
||||
*
|
||||
* The compiler is the ONLY thing in this repo that turns an analytics request
|
||||
* into SQL text. Its contract:
|
||||
*
|
||||
* * `tenant_id` is not an argument the caller may omit — it comes from the
|
||||
* VERIFIED CL0UD identity (layer 1) and is written into the WHERE clause
|
||||
* by this function, unconditionally, as a bound parameter.
|
||||
* * caller-supplied VALUES are never spliced into SQL text. They travel as
|
||||
* ClickHouse HTTP `param_*` query parameters.
|
||||
* * caller-supplied IDENTIFIERS (a dimension name, a group-by column) are
|
||||
* never spliced either. They are RESOLVED against a declared allow-list
|
||||
* of `DimensionDefinition`s and only the resolved, already-known physical
|
||||
* name reaches the SQL — so "dimension identifier injection" has no
|
||||
* surface, because caller text never becomes an identifier.
|
||||
* * there is no raw-SQL entry point. `compileMetricQuery` cannot express a
|
||||
* UNION, a subquery, or a second table, so a caller cannot reach a table
|
||||
* the compiler did not choose.
|
||||
*
|
||||
* WHY A SEPARATE MODULE from clickhouse-analytics.ts: that file's compiler
|
||||
* already forces the tenant predicate correctly and BI-P009 says PRESERVE IT.
|
||||
* This module is the generalised compiler for the BI-P010 tables; it repeats
|
||||
* the guarantee rather than replacing it, and
|
||||
* tests/analytics/tenant-predicate-regression.test.ts asserts BOTH.
|
||||
*/
|
||||
import { SiaxError } from "../../platform/errors";
|
||||
import type { AnalyticsTable } from "./schema";
|
||||
|
||||
/** A physical identifier that has been VALIDATED, not merely received. */
|
||||
const IDENT_RE = /^[A-Za-z_][A-Za-z0-9_]{0,62}$/;
|
||||
|
||||
export function assertSafeIdentifier(value: string, what: string): string {
|
||||
if (!IDENT_RE.test(value)) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", `invalid ${what} identifier`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
export const METRIC_AGGREGATES = ["sum", "avg", "min", "max", "count", "count_distinct"] as const;
|
||||
export type MetricAggregate = (typeof METRIC_AGGREGATES)[number];
|
||||
|
||||
/**
|
||||
* The dimensions a caller is allowed to name, resolved from
|
||||
* `DimensionDefinition` rows. The compiler looks a caller's dimension name up
|
||||
* HERE; a miss is a rejection, never a pass-through.
|
||||
*/
|
||||
export interface DimensionCatalog {
|
||||
/** caller-facing dimension name -> physical dimension key in dim_names. */
|
||||
resolve(name: string): string | undefined;
|
||||
}
|
||||
|
||||
export function dimensionCatalogFrom(pairs: Record<string, string>): DimensionCatalog {
|
||||
const map = new Map(Object.entries(pairs));
|
||||
return {
|
||||
resolve(name: string): string | undefined {
|
||||
// Own-property lookup on a Map — no prototype-chain surprises
|
||||
// ("constructor", "__proto__" resolve to undefined, as they must).
|
||||
return map.get(name);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export interface MetricQueryRequest {
|
||||
/** Supplied by the engine from the verified identity, NOT by the caller. */
|
||||
tenant_id: string;
|
||||
project_id?: string;
|
||||
dataset_id?: string;
|
||||
metric: string;
|
||||
aggregate: MetricAggregate;
|
||||
/** Caller-facing dimension names; resolved through the catalog. */
|
||||
group_by?: string[];
|
||||
/** Equality filters on dimensions: caller-facing name -> value. */
|
||||
filters?: Record<string, string>;
|
||||
from_time?: string;
|
||||
to_time?: string;
|
||||
limit?: number;
|
||||
/** Which BI-P010 table to read. Chosen by the engine, not the caller. */
|
||||
table?: AnalyticsTable;
|
||||
}
|
||||
|
||||
export interface CompiledQuery {
|
||||
sql: string;
|
||||
/** ClickHouse HTTP `param_*` entries. Values NEVER appear in `sql`. */
|
||||
params: Record<string, string>;
|
||||
/** The predicate the compiler forced, for assertion in tests. */
|
||||
forcedTenantPredicate: string;
|
||||
table: AnalyticsTable;
|
||||
}
|
||||
|
||||
/**
|
||||
* The exact predicate text every compiled query must contain. Exported so a
|
||||
* regression test can assert on the literal string — if someone deletes the
|
||||
* forcing, the test fails on a missing substring rather than on a subtle
|
||||
* behavioural difference that a mocked engine would happily hide.
|
||||
*/
|
||||
export const FORCED_TENANT_PREDICATE = "tenant_id = {tenant:String}";
|
||||
|
||||
const AGG_SQL: Record<MetricAggregate, string> = {
|
||||
sum: "sum(value)",
|
||||
avg: "avg(value)",
|
||||
min: "min(value)",
|
||||
max: "max(value)",
|
||||
count: "count()",
|
||||
count_distinct: "uniqExact(event_id)",
|
||||
};
|
||||
|
||||
const MAX_GROUP_BY = 8;
|
||||
|
||||
/**
|
||||
* Compile a closed-DSL metric request into tenant-forced SQL.
|
||||
*
|
||||
* @param req the request; `tenant_id` MUST already be the verified identity.
|
||||
* @param db the analytics database name (engine-controlled, validated).
|
||||
* @param catalog the tenant's declared dimensions.
|
||||
* @param maxResultRows the governance ceiling (BI-P012); `limit` is clamped.
|
||||
*/
|
||||
export function compileMetricQuery(
|
||||
req: MetricQueryRequest,
|
||||
db: string,
|
||||
catalog: DimensionCatalog,
|
||||
maxResultRows: number,
|
||||
): CompiledQuery {
|
||||
if (!req.tenant_id || !req.tenant_id.trim()) {
|
||||
// Fail closed. A blank tenant must never compile to "no predicate".
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "tenant_id is required on every analytics query");
|
||||
}
|
||||
assertSafeIdentifier(db, "database");
|
||||
const table: AnalyticsTable = req.table ?? "opt_analytics_event";
|
||||
assertSafeIdentifier(table, "table");
|
||||
if (!METRIC_AGGREGATES.includes(req.aggregate)) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "unsupported aggregate");
|
||||
}
|
||||
|
||||
const params: Record<string, string> = {
|
||||
param_tenant: req.tenant_id,
|
||||
param_metric: req.metric,
|
||||
};
|
||||
|
||||
// ── the forced predicate. First condition, always, no branch. ─────────
|
||||
const conditions: string[] = [FORCED_TENANT_PREDICATE, "metric = {metric:String}"];
|
||||
|
||||
if (req.project_id) {
|
||||
conditions.push("project_id = {project:String}");
|
||||
params.param_project = req.project_id;
|
||||
}
|
||||
if (req.dataset_id) {
|
||||
conditions.push("dataset_id = {dataset:String}");
|
||||
params.param_dataset = req.dataset_id;
|
||||
}
|
||||
if (req.from_time) {
|
||||
conditions.push("event_time >= parseDateTime64BestEffort({fromTime:String})");
|
||||
params.param_fromTime = req.from_time;
|
||||
}
|
||||
if (req.to_time) {
|
||||
conditions.push("event_time < parseDateTime64BestEffort({toTime:String})");
|
||||
params.param_toTime = req.to_time;
|
||||
}
|
||||
|
||||
// ── filters: names resolved through the catalog, values parameterized ──
|
||||
let filterIdx = 0;
|
||||
for (const [name, value] of Object.entries(req.filters ?? {})) {
|
||||
const physical = catalog.resolve(name);
|
||||
if (physical === undefined) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", `unknown filter dimension "${name}"`);
|
||||
}
|
||||
// `physical` came from the catalog, but validate anyway: a corrupted or
|
||||
// hostile catalog row must not become an injection vector either.
|
||||
assertSafeIdentifier(physical, "dimension");
|
||||
const nameKey = `f${filterIdx}n`;
|
||||
const valKey = `f${filterIdx}v`;
|
||||
conditions.push(
|
||||
`dim_values[indexOf(dim_names, {${nameKey}:String})] = {${valKey}:String}`,
|
||||
);
|
||||
params[`param_${nameKey}`] = physical;
|
||||
params[`param_${valKey}`] = value;
|
||||
filterIdx += 1;
|
||||
}
|
||||
|
||||
// ── group by: same resolution, plus a hard cardinality ceiling ─────────
|
||||
const groupBy = req.group_by ?? [];
|
||||
if (groupBy.length > MAX_GROUP_BY) {
|
||||
throw new SiaxError(
|
||||
"INVALID_REQUEST",
|
||||
"0pt.analytics",
|
||||
`at most ${MAX_GROUP_BY} group-by dimensions (got ${groupBy.length})`,
|
||||
);
|
||||
}
|
||||
const selectParts: string[] = [];
|
||||
const groupParts: string[] = [];
|
||||
groupBy.forEach((name, i) => {
|
||||
const physical = catalog.resolve(name);
|
||||
if (physical === undefined) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", `unknown group_by dimension "${name}"`);
|
||||
}
|
||||
assertSafeIdentifier(physical, "dimension");
|
||||
const key = `g${i}`;
|
||||
// The ALIAS is compiler-generated (`grp0`, `grp1`), never caller text.
|
||||
selectParts.push(`dim_values[indexOf(dim_names, {${key}:String})] AS grp${i}`);
|
||||
groupParts.push(`grp${i}`);
|
||||
params[`param_${key}`] = physical;
|
||||
});
|
||||
|
||||
const limit = Math.max(1, Math.min(Math.floor(req.limit ?? 1000), maxResultRows));
|
||||
|
||||
const sql =
|
||||
`SELECT ${selectParts.length ? selectParts.join(", ") + ", " : ""}${AGG_SQL[req.aggregate]} AS value ` +
|
||||
`FROM ${db}.${table} ` +
|
||||
`WHERE ${conditions.join(" AND ")} ` +
|
||||
(groupParts.length ? `GROUP BY ${groupParts.join(", ")} ORDER BY value DESC ` : "") +
|
||||
`LIMIT ${limit} FORMAT JSON`;
|
||||
|
||||
return { sql, params, forcedTenantPredicate: FORCED_TENANT_PREDICATE, table };
|
||||
}
|
||||
|
||||
/**
|
||||
* Compile a projection lookup (SERVING tier, BI-P010's opt_dataset_projection).
|
||||
* Same forcing, different table and key.
|
||||
*/
|
||||
export function compileProjectionQuery(
|
||||
req: { tenant_id: string; project_id?: string; dataset_id: string; entity_keys?: string[]; limit?: number },
|
||||
db: string,
|
||||
maxResultRows: number,
|
||||
): CompiledQuery {
|
||||
if (!req.tenant_id || !req.tenant_id.trim()) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "tenant_id is required on every analytics query");
|
||||
}
|
||||
assertSafeIdentifier(db, "database");
|
||||
const params: Record<string, string> = {
|
||||
param_tenant: req.tenant_id,
|
||||
param_dataset: req.dataset_id,
|
||||
};
|
||||
const conditions = [FORCED_TENANT_PREDICATE, "dataset_id = {dataset:String}"];
|
||||
if (req.project_id) {
|
||||
conditions.push("project_id = {project:String}");
|
||||
params.param_project = req.project_id;
|
||||
}
|
||||
if (req.entity_keys && req.entity_keys.length > 0) {
|
||||
conditions.push("has({entityKeys:Array(String)}, entity_key)");
|
||||
params.param_entityKeys = JSON.stringify(req.entity_keys);
|
||||
}
|
||||
const limit = Math.max(1, Math.min(Math.floor(req.limit ?? 1000), maxResultRows));
|
||||
const sql =
|
||||
`SELECT entity_key, as_of, version, attributes, measures ` +
|
||||
`FROM ${db}.opt_dataset_projection ` +
|
||||
`WHERE ${conditions.join(" AND ")} ` +
|
||||
`ORDER BY entity_key LIMIT ${limit} FORMAT JSON`;
|
||||
return { sql, params, forcedTenantPredicate: FORCED_TENANT_PREDICATE, table: "opt_dataset_projection" };
|
||||
}
|
||||
|
||||
/**
|
||||
* BI-P011 layer-3 bootstrap: the DDL for a RESTRICTED per-tenant ClickHouse
|
||||
* user with a row policy pinning `tenant_id`.
|
||||
*
|
||||
* This is the layer that survives a compiler bug. The compiler forcing the
|
||||
* predicate is layer 2; if a future refactor drops it, layer 3 still returns
|
||||
* zero foreign rows because the SERVER filters them. The BI-P011 proof runs
|
||||
* the adversarial suite twice — once through an unrestricted user (hole open)
|
||||
* and once through this restricted user (hole closed) — because a claim of
|
||||
* defence in depth that has never been tested with one layer removed is not
|
||||
* evidence.
|
||||
*
|
||||
* `tenantId` is validated as an identifier-safe token AND embedded as a
|
||||
* quoted literal; ClickHouse's CREATE ROW POLICY takes no bound parameters,
|
||||
* so validation is the only protection available and it is applied strictly.
|
||||
*/
|
||||
export function restrictedTenantUserDdl(args: {
|
||||
db: string;
|
||||
tenantId: string;
|
||||
userName: string;
|
||||
password: string;
|
||||
tables: readonly string[];
|
||||
settingsProfile?: string;
|
||||
}): string[] {
|
||||
assertSafeIdentifier(args.db, "database");
|
||||
assertSafeIdentifier(args.userName, "user");
|
||||
// A tenant id is `t_<slug>` in this estate; enforce that shape strictly
|
||||
// rather than trusting the caller.
|
||||
if (!/^[A-Za-z0-9_-]{1,64}$/.test(args.tenantId)) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "tenant id is not policy-safe");
|
||||
}
|
||||
if (args.password.includes("'") || args.password.includes("\\")) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "password contains characters that cannot be embedded");
|
||||
}
|
||||
const profile = args.settingsProfile;
|
||||
if (profile) assertSafeIdentifier(profile, "settings profile");
|
||||
|
||||
const stmts: string[] = [
|
||||
`CREATE USER IF NOT EXISTS ${args.userName} IDENTIFIED WITH sha256_password BY '${args.password}'` +
|
||||
(profile ? ` SETTINGS PROFILE ${profile}` : ` SETTINGS readonly = 1`),
|
||||
];
|
||||
for (const table of args.tables) {
|
||||
assertSafeIdentifier(table, "table");
|
||||
stmts.push(
|
||||
`CREATE ROW POLICY IF NOT EXISTS ${args.userName}_${table}_rp ON ${args.db}.${table} ` +
|
||||
`FOR SELECT USING tenant_id = '${args.tenantId}' TO ${args.userName}`,
|
||||
);
|
||||
stmts.push(`GRANT SELECT ON ${args.db}.${table} TO ${args.userName}`);
|
||||
}
|
||||
return stmts;
|
||||
}
|
||||
Reference in New Issue
Block a user