CI (SIAX Cloud) / quality (push) Successful in 50s
CI (SIAX Cloud) / integration (push) Successful in 35s
CI (SIAX Cloud) / security (push) Successful in 21s
Security Scan (SIAX Cloud) / security (push) Successful in 26s
CI (SIAX Cloud) / quality (pull_request) Successful in 46s
CI (SIAX Cloud) / integration (pull_request) Successful in 35s
CI (SIAX Cloud) / security (pull_request) Successful in 19s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 16s
AnalyticsSQLSession is a SEPARATE class, not a method on SiaxAnalyticsEngine.
executeMetric has no raw-SQL surface at all; on the same object the closed DSL
would be one refactor away from growing one. ClickHouseAnalyticsEngine
.executeQuery stays a documented refusal.
FOUR LAYERS, and the guard is deliberately called the weakest:
1 role — the CL0UD analyst role is checked when the SESSION IS OPENED, so
a caller without it never obtains an object that can run SQL.
2 guard — statement admission. Comments and literals removed with
ClickHouse's own lexical rules, ASCII-only outside literals, one
statement, SELECT/WITH only, and every table reference POSITIVELY
resolved against the tenant's catalog. Deny-lists are a second
opinion, never the mechanism.
3 grants — a per-tenant ClickHouse user with SELECT on that tenant's catalog
alone, plus a restrictive row policy on the base tables.
4 views — the tenant catalog holds only VIEWS, each baking
`WHERE tenant_id = '<tenant>'`, and tenant_id is not projected.
61 bypass attempts, in one shared corpus, run TWICE: against the guard, and
against a live ClickHouse 24.8.14.39 with LAYER 2 REMOVED. The live assertion
is not "the engine errored" — it is "no canary came back", with BETA_CANARY
seeded into the other tenant's rows and SECRET_CANARY into an ungranted table.
The counterfactual runs first: as admin the same base table returns 100 rows
and BETA_CANARY IS readable, so the isolation numbers mean something.
Result: 0 of 61 leaked. Evidence: evidence/bi-p017/analyst-sql-live-bypass.json.
FOUR DEFECTS FOUND BY WRITING TESTS THAT CAN FAIL — three of them mine:
* `CREATE ROW POLICY IF NOT EXISTS` is a SILENT NO-OP on re-provisioning.
Dropping the user empties the policy's apply-to list; the next run skips
re-creation, so the policy survives with `apply_to_list = []`, applying to
nobody, while `system.row_policies` still shows the correct select_filter.
The restricted user read 100 of 100 rows — BOTH TENANTS — from the base
table. Fixed with OR REPLACE + `rowPolicyVerificationSql`, which turns
"the policy is applied" from an assumption into a query, and a test that
asserts applied_to_user = 1 per object.
* `CREATE SETTINGS PROFILE IF NOT EXISTS` has the identical shape: a changed
budget never lands. A stale 60-row ceiling survived re-provisioning and
starved every legitimate statement, with the DDL reporting success
throughout. Also OR REPLACE now.
* `describe.skipIf(!reachable)` where `reachable` is set in `beforeAll`
skips ALWAYS — vitest evaluates skipIf at collection time. Ten tests
reported green while skipping against a live, reachable server. Replaced
with the repo's existing env-var gate, which FAILS when CLICKHOUSE_URL is
set and the server is down rather than degrading to a skip.
* MEASURED, and it changes the design: a ClickHouse user whose profile
carries `readonly = 1` cannot have ANY setting changed from the request
(Code: 164). The BI-P012 ceilings therefore cannot travel with an analyst
statement; they are baked into the user's SETTINGS PROFILE at provisioning.
Proven both ways: over-budget scan fails with code 158 from the profile
with no setting sent, and an attempt to raise it from the request fails
with code 164.
Honest about what the guard does NOT catch: `SELECT tenant_id FROM
t_alpha_sql.sales` is structurally legitimate and IS admitted. It fails at the
engine because the view does not project the column. Recorded in its own
GUARD_ADMITS_ENGINE_REFUSES list and asserted live, rather than deleted from
the corpus.
evidence/bi-p017 is pinned by tests/analytics/analyst-sql-evidence.test.ts,
which never skips: adding an attack without re-running the live suite breaks
the build, and the pin also checks the counterfactual numbers so the evidence
cannot be a recording of an empty database.
246 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
307 lines
12 KiB
TypeScript
307 lines
12 KiB
TypeScript
/**
|
|
* src/analytics/engine/query-compiler.ts
|
|
* =======================================
|
|
* BI-P011 — CLICKHOUSE TENANT ISOLATION, layer 2:
|
|
* "`0pt` query compiler forces tenant".
|
|
*
|
|
* The compiler is the ONLY thing in this repo that turns an analytics request
|
|
* into SQL text. Its contract:
|
|
*
|
|
* * `tenant_id` is not an argument the caller may omit — it comes from the
|
|
* VERIFIED CL0UD identity (layer 1) and is written into the WHERE clause
|
|
* by this function, unconditionally, as a bound parameter.
|
|
* * caller-supplied VALUES are never spliced into SQL text. They travel as
|
|
* ClickHouse HTTP `param_*` query parameters.
|
|
* * caller-supplied IDENTIFIERS (a dimension name, a group-by column) are
|
|
* never spliced either. They are RESOLVED against a declared allow-list
|
|
* of `DimensionDefinition`s and only the resolved, already-known physical
|
|
* name reaches the SQL — so "dimension identifier injection" has no
|
|
* surface, because caller text never becomes an identifier.
|
|
* * there is no raw-SQL entry point. `compileMetricQuery` cannot express a
|
|
* UNION, a subquery, or a second table, so a caller cannot reach a table
|
|
* the compiler did not choose.
|
|
*
|
|
* WHY A SEPARATE MODULE from clickhouse-analytics.ts: that file's compiler
|
|
* already forces the tenant predicate correctly and BI-P009 says PRESERVE IT.
|
|
* This module is the generalised compiler for the BI-P010 tables; it repeats
|
|
* the guarantee rather than replacing it, and
|
|
* tests/analytics/tenant-predicate-regression.test.ts asserts BOTH.
|
|
*/
|
|
import { SiaxError } from '../../platform/errors';
|
|
import type { AnalyticsTable } from './schema';
|
|
|
|
/** A physical identifier that has been VALIDATED, not merely received. */
|
|
const IDENT_RE = /^[A-Za-z_][A-Za-z0-9_]{0,62}$/;
|
|
|
|
export function assertSafeIdentifier(value: string, what: string): string {
|
|
if (!IDENT_RE.test(value)) {
|
|
throw new SiaxError('INVALID_REQUEST', '0pt.analytics', `invalid ${what} identifier`);
|
|
}
|
|
return value;
|
|
}
|
|
|
|
export const METRIC_AGGREGATES = ['sum', 'avg', 'min', 'max', 'count', 'count_distinct'] as const;
|
|
export type MetricAggregate = (typeof METRIC_AGGREGATES)[number];
|
|
|
|
/**
|
|
* The dimensions a caller is allowed to name, resolved from
|
|
* `DimensionDefinition` rows. The compiler looks a caller's dimension name up
|
|
* HERE; a miss is a rejection, never a pass-through.
|
|
*/
|
|
export interface DimensionCatalog {
|
|
/** caller-facing dimension name -> physical dimension key in dim_names. */
|
|
resolve(name: string): string | undefined;
|
|
}
|
|
|
|
export function dimensionCatalogFrom(pairs: Record<string, string>): DimensionCatalog {
|
|
const map = new Map(Object.entries(pairs));
|
|
return {
|
|
resolve(name: string): string | undefined {
|
|
// Own-property lookup on a Map — no prototype-chain surprises
|
|
// ("constructor", "__proto__" resolve to undefined, as they must).
|
|
return map.get(name);
|
|
},
|
|
};
|
|
}
|
|
|
|
export interface MetricQueryRequest {
|
|
/** Supplied by the engine from the verified identity, NOT by the caller. */
|
|
tenant_id: string;
|
|
project_id?: string;
|
|
dataset_id?: string;
|
|
metric: string;
|
|
aggregate: MetricAggregate;
|
|
/** Caller-facing dimension names; resolved through the catalog. */
|
|
group_by?: string[];
|
|
/** Equality filters on dimensions: caller-facing name -> value. */
|
|
filters?: Record<string, string>;
|
|
from_time?: string;
|
|
to_time?: string;
|
|
limit?: number;
|
|
/** Which BI-P010 table to read. Chosen by the engine, not the caller. */
|
|
table?: AnalyticsTable;
|
|
}
|
|
|
|
export interface CompiledQuery {
|
|
sql: string;
|
|
/** ClickHouse HTTP `param_*` entries. Values NEVER appear in `sql`. */
|
|
params: Record<string, string>;
|
|
/** The predicate the compiler forced, for assertion in tests. */
|
|
forcedTenantPredicate: string;
|
|
table: AnalyticsTable;
|
|
}
|
|
|
|
/**
|
|
* The exact predicate text every compiled query must contain. Exported so a
|
|
* regression test can assert on the literal string — if someone deletes the
|
|
* forcing, the test fails on a missing substring rather than on a subtle
|
|
* behavioural difference that a mocked engine would happily hide.
|
|
*/
|
|
export const FORCED_TENANT_PREDICATE = 'tenant_id = {tenant:String}';
|
|
|
|
const AGG_SQL: Record<MetricAggregate, string> = {
|
|
sum: 'sum(value)',
|
|
avg: 'avg(value)',
|
|
min: 'min(value)',
|
|
max: 'max(value)',
|
|
count: 'count()',
|
|
count_distinct: 'uniqExact(event_id)',
|
|
};
|
|
|
|
const MAX_GROUP_BY = 8;
|
|
|
|
/**
|
|
* Compile a closed-DSL metric request into tenant-forced SQL.
|
|
*
|
|
* @param req the request; `tenant_id` MUST already be the verified identity.
|
|
* @param db the analytics database name (engine-controlled, validated).
|
|
* @param catalog the tenant's declared dimensions.
|
|
* @param maxResultRows the governance ceiling (BI-P012); `limit` is clamped.
|
|
*/
|
|
export function compileMetricQuery(
|
|
req: MetricQueryRequest,
|
|
db: string,
|
|
catalog: DimensionCatalog,
|
|
maxResultRows: number,
|
|
): CompiledQuery {
|
|
if (!req.tenant_id || !req.tenant_id.trim()) {
|
|
// Fail closed. A blank tenant must never compile to "no predicate".
|
|
throw new SiaxError('INVALID_REQUEST', '0pt.analytics', 'tenant_id is required on every analytics query');
|
|
}
|
|
assertSafeIdentifier(db, 'database');
|
|
const table: AnalyticsTable = req.table ?? 'opt_analytics_event';
|
|
assertSafeIdentifier(table, 'table');
|
|
if (!METRIC_AGGREGATES.includes(req.aggregate)) {
|
|
throw new SiaxError('INVALID_REQUEST', '0pt.analytics', 'unsupported aggregate');
|
|
}
|
|
|
|
const params: Record<string, string> = {
|
|
param_tenant: req.tenant_id,
|
|
param_metric: req.metric,
|
|
};
|
|
|
|
// ── the forced predicate. First condition, always, no branch. ─────────
|
|
const conditions: string[] = [FORCED_TENANT_PREDICATE, 'metric = {metric:String}'];
|
|
|
|
if (req.project_id) {
|
|
conditions.push('project_id = {project:String}');
|
|
params.param_project = req.project_id;
|
|
}
|
|
if (req.dataset_id) {
|
|
conditions.push('dataset_id = {dataset:String}');
|
|
params.param_dataset = req.dataset_id;
|
|
}
|
|
if (req.from_time) {
|
|
conditions.push('event_time >= parseDateTime64BestEffort({fromTime:String})');
|
|
params.param_fromTime = req.from_time;
|
|
}
|
|
if (req.to_time) {
|
|
conditions.push('event_time < parseDateTime64BestEffort({toTime:String})');
|
|
params.param_toTime = req.to_time;
|
|
}
|
|
|
|
// ── filters: names resolved through the catalog, values parameterized ──
|
|
let filterIdx = 0;
|
|
for (const [name, value] of Object.entries(req.filters ?? {})) {
|
|
const physical = catalog.resolve(name);
|
|
if (physical === undefined) {
|
|
throw new SiaxError('INVALID_REQUEST', '0pt.analytics', `unknown filter dimension "${name}"`);
|
|
}
|
|
// `physical` came from the catalog, but validate anyway: a corrupted or
|
|
// hostile catalog row must not become an injection vector either.
|
|
assertSafeIdentifier(physical, 'dimension');
|
|
const nameKey = `f${filterIdx}n`;
|
|
const valKey = `f${filterIdx}v`;
|
|
conditions.push(`dim_values[indexOf(dim_names, {${nameKey}:String})] = {${valKey}:String}`);
|
|
params[`param_${nameKey}`] = physical;
|
|
params[`param_${valKey}`] = value;
|
|
filterIdx += 1;
|
|
}
|
|
|
|
// ── group by: same resolution, plus a hard cardinality ceiling ─────────
|
|
const groupBy = req.group_by ?? [];
|
|
if (groupBy.length > MAX_GROUP_BY) {
|
|
throw new SiaxError(
|
|
'INVALID_REQUEST',
|
|
'0pt.analytics',
|
|
`at most ${MAX_GROUP_BY} group-by dimensions (got ${groupBy.length})`,
|
|
);
|
|
}
|
|
const selectParts: string[] = [];
|
|
const groupParts: string[] = [];
|
|
groupBy.forEach((name, i) => {
|
|
const physical = catalog.resolve(name);
|
|
if (physical === undefined) {
|
|
throw new SiaxError('INVALID_REQUEST', '0pt.analytics', `unknown group_by dimension "${name}"`);
|
|
}
|
|
assertSafeIdentifier(physical, 'dimension');
|
|
const key = `g${i}`;
|
|
// The ALIAS is compiler-generated (`grp0`, `grp1`), never caller text.
|
|
selectParts.push(`dim_values[indexOf(dim_names, {${key}:String})] AS grp${i}`);
|
|
groupParts.push(`grp${i}`);
|
|
params[`param_${key}`] = physical;
|
|
});
|
|
|
|
const limit = Math.max(1, Math.min(Math.floor(req.limit ?? 1000), maxResultRows));
|
|
|
|
const sql =
|
|
`SELECT ${selectParts.length ? selectParts.join(', ') + ', ' : ''}${AGG_SQL[req.aggregate]} AS value ` +
|
|
`FROM ${db}.${table} ` +
|
|
`WHERE ${conditions.join(' AND ')} ` +
|
|
(groupParts.length ? `GROUP BY ${groupParts.join(', ')} ORDER BY value DESC ` : '') +
|
|
`LIMIT ${limit} FORMAT JSON`;
|
|
|
|
return { sql, params, forcedTenantPredicate: FORCED_TENANT_PREDICATE, table };
|
|
}
|
|
|
|
/**
|
|
* Compile a projection lookup (SERVING tier, BI-P010's opt_dataset_projection).
|
|
* Same forcing, different table and key.
|
|
*/
|
|
export function compileProjectionQuery(
|
|
req: { tenant_id: string; project_id?: string; dataset_id: string; entity_keys?: string[]; limit?: number },
|
|
db: string,
|
|
maxResultRows: number,
|
|
): CompiledQuery {
|
|
if (!req.tenant_id || !req.tenant_id.trim()) {
|
|
throw new SiaxError('INVALID_REQUEST', '0pt.analytics', 'tenant_id is required on every analytics query');
|
|
}
|
|
assertSafeIdentifier(db, 'database');
|
|
const params: Record<string, string> = {
|
|
param_tenant: req.tenant_id,
|
|
param_dataset: req.dataset_id,
|
|
};
|
|
const conditions = [FORCED_TENANT_PREDICATE, 'dataset_id = {dataset:String}'];
|
|
if (req.project_id) {
|
|
conditions.push('project_id = {project:String}');
|
|
params.param_project = req.project_id;
|
|
}
|
|
if (req.entity_keys && req.entity_keys.length > 0) {
|
|
conditions.push('has({entityKeys:Array(String)}, entity_key)');
|
|
params.param_entityKeys = JSON.stringify(req.entity_keys);
|
|
}
|
|
const limit = Math.max(1, Math.min(Math.floor(req.limit ?? 1000), maxResultRows));
|
|
const sql =
|
|
`SELECT entity_key, as_of, version, attributes, measures ` +
|
|
`FROM ${db}.opt_dataset_projection ` +
|
|
`WHERE ${conditions.join(' AND ')} ` +
|
|
`ORDER BY entity_key LIMIT ${limit} FORMAT JSON`;
|
|
return { sql, params, forcedTenantPredicate: FORCED_TENANT_PREDICATE, table: 'opt_dataset_projection' };
|
|
}
|
|
|
|
/**
|
|
* BI-P011 layer-3 bootstrap: the DDL for a RESTRICTED per-tenant ClickHouse
|
|
* user with a row policy pinning `tenant_id`.
|
|
*
|
|
* This is the layer that survives a compiler bug. The compiler forcing the
|
|
* predicate is layer 2; if a future refactor drops it, layer 3 still returns
|
|
* zero foreign rows because the SERVER filters them. The BI-P011 proof runs
|
|
* the adversarial suite twice — once through an unrestricted user (hole open)
|
|
* and once through this restricted user (hole closed) — because a claim of
|
|
* defence in depth that has never been tested with one layer removed is not
|
|
* evidence.
|
|
*
|
|
* `tenantId` is validated as an identifier-safe token AND embedded as a
|
|
* quoted literal; ClickHouse's CREATE ROW POLICY takes no bound parameters,
|
|
* so validation is the only protection available and it is applied strictly.
|
|
*/
|
|
export function restrictedTenantUserDdl(args: {
|
|
db: string;
|
|
tenantId: string;
|
|
userName: string;
|
|
password: string;
|
|
tables: readonly string[];
|
|
settingsProfile?: string;
|
|
}): string[] {
|
|
assertSafeIdentifier(args.db, 'database');
|
|
assertSafeIdentifier(args.userName, 'user');
|
|
// A tenant id is `t_<slug>` in this estate; enforce that shape strictly
|
|
// rather than trusting the caller.
|
|
if (!/^[A-Za-z0-9_-]{1,64}$/.test(args.tenantId)) {
|
|
throw new SiaxError('INVALID_REQUEST', '0pt.analytics', 'tenant id is not policy-safe');
|
|
}
|
|
if (args.password.includes("'") || args.password.includes('\\')) {
|
|
throw new SiaxError(
|
|
'INVALID_REQUEST',
|
|
'0pt.analytics',
|
|
'password contains characters that cannot be embedded',
|
|
);
|
|
}
|
|
const profile = args.settingsProfile;
|
|
if (profile) assertSafeIdentifier(profile, 'settings profile');
|
|
|
|
const stmts: string[] = [
|
|
`CREATE USER IF NOT EXISTS ${args.userName} IDENTIFIED WITH sha256_password BY '${args.password}'` +
|
|
(profile ? ` SETTINGS PROFILE ${profile}` : ` SETTINGS readonly = 1`),
|
|
];
|
|
for (const table of args.tables) {
|
|
assertSafeIdentifier(table, 'table');
|
|
stmts.push(
|
|
`CREATE ROW POLICY IF NOT EXISTS ${args.userName}_${table}_rp ON ${args.db}.${table} ` +
|
|
`FOR SELECT USING tenant_id = '${args.tenantId}' TO ${args.userName}`,
|
|
);
|
|
stmts.push(`GRANT SELECT ON ${args.db}.${table} TO ${args.userName}`);
|
|
}
|
|
return stmts;
|
|
}
|