feat(analytics): BI-P002..BI-P012 analytics core — domain model, tiers, contracts, Arrow, ClickHouse engine, isolation, governance
CI (SIAX Cloud) / security (push) Successful in 41s
Security Scan (SIAX Cloud) / security (push) Successful in 40s
CI (SIAX Cloud) / integration (push) Successful in 52s
CI (SIAX Cloud) / quality (push) Successful in 1m5s
CI (SIAX Cloud) / security (pull_request) Successful in 31s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 29s
CI (SIAX Cloud) / integration (pull_request) Successful in 45s
CI (SIAX Cloud) / quality (pull_request) Successful in 53s

Implements the eight BI program prompts that make 0pt the canonical SIAX
analytics capability. Every file is NEW: not one existing file is modified,
and src/analytics/clickhouse-analytics.ts is byte-identical to main.

BI-P002 — the nineteen canonical entities, verbatim, with tenant_id +
project_id on every tenant-owned record. sql/008 gives each one a table.
The four version entities BI-P002 names as immutable are enforced by BEFORE
UPDATE/DELETE triggers, not by a TypeScript `readonly` — proven by direct SQL
against a live Postgres.

BI-P003 — RAW/NORMALIZED/CURATED/SERVING with the read rule enforced:
assertReadableByBiSurface refuses a RAW dataset to a dashboard unless a
recorded exception exists. All eight required declarations are NOT NULL.

BI-P004 — DatasetContract and a fail-closed compatibility gate. Fifteen
breaking-change classes are each tested twice: refused under a minor bump,
accepted under a major one.

BI-P005 — Arrow IPC alongside JSON, with a hand-written FlatBuffers writer so
no npm dependency (or CVE surface) is added. Conformance comes from three
independent implementations, not from self-agreement: pyarrow 25.0.1 and
DuckDB 1.5.5 read identical values from both ClickHouse's Arrow and ours, and
ClickHouse's own Arrow C++ ingests our 20,000-row batch with exact aggregates.

BI-P009 — SiaxAnalyticsEngine with all eight capabilities. The existing
ClickHouseAnalytics is COMPOSED, never replaced. executeQuery (analyst SQL) is
declared and deliberately refused with CAPABILITY_UNAVAILABLE until BI-P017 and
BI-P033 land, rather than opening the raw-SQL hole BI-P011 enumerates.

BI-P010 — four purpose-built tables, not one giant one, and the difference is
measured against the counterfactual: 48.8ms/40,960 rows vs 174.4ms/5,400,000
rows for the same question; 5/660 granules vs 660/1149.

BI-P011 — layered isolation, each layer proven with the layer above removed.
Without the forced predicate the same statement returns 50,050 (both tenants);
with it, t_alpha gets 50. Without a row policy an unrestricted user sees 2
tenants; the restricted user running the IDENTICAL statement sees 1.

BI-P012 — every named limit, fired individually and identified by ClickHouse
error code, plus the half usually skipped: an unrelated tenant is timed while
a real CPU bomb is in flight and must still return the CORRECT answer.

Three defects found by writing tests that can fail:
  * `CHECK (array_length(purposes,1) >= 1)` can NEVER fail — array_length is
    NULL for '{}' and a NULL CHECK passes. Fixed with coalesce(); the test
    that caught it is pinned.
  * EXPLAIN ESTIMATE run under the caller's own max_rows_to_read fails with
    code 158 exactly when the estimate would be over budget. Planner calls now
    use a narrower envelope.
  * result_overflow_mode=break returns HTTP 200 with 65,409 of 100,000 rows
    and no indication it is short. Every SELECT ships `throw`.

Verified rather than assumed: server7's ClickHouse holds `opt_scraped` (51
tables, 407,793,471 rows, `persons`/`cz_owners` included) — the b0ts plane this
module is prohibited from touching — has no `opt_analytics` database, zero
tenant_id columns and zero row policies. Nothing there was modified; every
proof ran against disposable containers.

Evidence: evidence/bi-p002-p012/{test-transcript.txt,bi-p010-benchmark.json,
bi-p005-arrow-roundtrip.json,server7-clickhouse-probe.md}
Docs: docs/analytics/BI_ANALYTICS_CORE.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
2026-09-06 23:01:27 +02:00
parent 8d0a288668
commit 84bb9faa4f
32 changed files with 7844 additions and 0 deletions
+237
View File
@@ -0,0 +1,237 @@
/**
* src/analytics/arrow/flatbuffer-builder.ts
* ==========================================
* A minimal FlatBuffers writer, sufficient for the Arrow IPC metadata
* messages this repo emits (BI-P005).
*
* WHY HAND-ROLLED rather than `apache-arrow` from npm: adding a runtime
* dependency to 0pt's blocking-typechecked capability layer means a lockfile
* change, a new supply-chain surface, and a new CVE surface for every
* consumer of this repo — for a format whose METADATA side is a few hundred
* bytes of well-specified binary and whose DATA side is plain columnar memory
* either way.
*
* The correctness bar is NOT "this looks right". The BI-P005 evidence script
* (scripts/bi-p005-arrow-roundtrip.mjs) decodes this writer's output with
* pyarrow AND DuckDB — two independent implementations of the specification —
* so a mistake here surfaces as a failed decode by somebody else's parser,
* not as a green test of my own assumptions.
*
* Layout facts implemented (FlatBuffers binary format, little-endian):
* * the buffer is built BACK TO FRONT; `offset` counts bytes from the end;
* * a uoffset stored at a field site holds `site_offset + 4 - target_offset`
* so a reader resolves it as `position + value`;
* * a table begins with an soffset `table_start - vtable_start`;
* * a vtable is `[vtable_len:i16][table_len:i16][field_offsets:i16...]`,
* with 0 meaning "field absent, use the default".
*/
export type Offset = number;
const SIZEOF_INT = 4;
export class FlatBufferBuilder {
private buf: Uint8Array;
private view: DataView;
/** Bytes written so far, counted from the END of `buf`. */
private used = 0;
private minAlign = 1;
private vtable: number[] = [];
private objectStart = 0;
private inObject = false;
/** Offsets of previously written vtables, for deduplication. */
private vtables: number[] = [];
constructor(initial = 1024) {
this.buf = new Uint8Array(initial);
this.view = new DataView(this.buf.buffer);
}
/** Bytes written so far (FlatBuffers "offset"). */
get offset(): number {
return this.used;
}
private ensure(extra: number): void {
if (this.used + extra <= this.buf.length) return;
let size = this.buf.length;
while (size < this.used + extra) size *= 2;
const next = new Uint8Array(size);
next.set(this.buf.subarray(this.buf.length - this.used), size - this.used);
this.buf = next;
this.view = new DataView(this.buf.buffer);
}
private pad(n: number): void {
if (n <= 0) return;
this.ensure(n);
for (let i = 0; i < n; i++) {
this.used += 1;
this.buf[this.buf.length - this.used] = 0;
}
}
/** FlatBuffers `prep`: pad so that `additional` more bytes land aligned. */
private prep(size: number, additional = 0): void {
if (size > this.minAlign) this.minAlign = size;
const alignSize = (~(this.used + additional) + 1) & (size - 1);
this.pad(alignSize);
}
writeInt8(v: number): void {
this.ensure(1);
this.used += 1;
this.view.setInt8(this.buf.length - this.used, v);
}
writeInt16(v: number): void {
this.ensure(2);
this.used += 2;
this.view.setInt16(this.buf.length - this.used, v, true);
}
writeInt32(v: number): void {
this.ensure(4);
this.used += 4;
this.view.setInt32(this.buf.length - this.used, v, true);
}
writeInt64(v: bigint): void {
this.ensure(8);
this.used += 8;
this.view.setBigInt64(this.buf.length - this.used, v, true);
}
writeBytes(bytes: Uint8Array): void {
this.ensure(bytes.length);
this.used += bytes.length;
this.buf.set(bytes, this.buf.length - this.used);
}
addInt8(slot: number, value: number, def: number): void {
if (value === def) return;
this.prep(1);
this.writeInt8(value);
this.slot(slot);
}
addInt16(slot: number, value: number, def: number): void {
if (value === def) return;
this.prep(2);
this.writeInt16(value);
this.slot(slot);
}
addInt32(slot: number, value: number, def: number): void {
if (value === def) return;
this.prep(4);
this.writeInt32(value);
this.slot(slot);
}
addInt64(slot: number, value: bigint, def: bigint): void {
if (value === def) return;
this.prep(8);
this.writeInt64(value);
this.slot(slot);
}
addOffset(slot: number, value: Offset, def: number): void {
if (value === def) return;
this.prep(SIZEOF_INT);
this.writeInt32(this.used + SIZEOF_INT - value);
this.slot(slot);
}
private slot(slotNum: number): void {
this.vtable[slotNum] = this.used;
}
startObject(numFields: number): void {
if (this.inObject) throw new Error("nested startObject");
this.vtable = new Array<number>(numFields).fill(0);
this.objectStart = this.used;
this.inObject = true;
}
endObject(): Offset {
if (!this.inObject) throw new Error("endObject outside an object");
this.prep(SIZEOF_INT);
this.writeInt32(0); // soffset placeholder, patched below
const objectOffset = this.used;
let trimmed = this.vtable.length;
while (trimmed > 0 && this.vtable[trimmed - 1] === 0) trimmed -= 1;
for (let i = trimmed - 1; i >= 0; i--) {
this.writeInt16(this.vtable[i] !== 0 ? objectOffset - this.vtable[i] : 0);
}
this.writeInt16(objectOffset - this.objectStart); // table byte length
const vtLen = (trimmed + 2) * 2;
this.writeInt16(vtLen); // vtable byte length — lands at the vtable start
const vtStart = this.used;
const vtAbs = this.buf.length - vtStart;
let existing = 0;
for (const candidate of this.vtables) {
const candAbs = this.buf.length - candidate;
if (this.view.getInt16(candAbs, true) !== vtLen) continue;
let same = true;
for (let i = 2; i < vtLen; i += 2) {
if (this.view.getInt16(vtAbs + i, true) !== this.view.getInt16(candAbs + i, true)) {
same = false;
break;
}
}
if (same) {
existing = candidate;
break;
}
}
const objAbs = this.buf.length - objectOffset;
if (existing !== 0) {
this.used -= vtLen; // discard the duplicate vtable just written
this.view.setInt32(objAbs, existing - objectOffset, true);
} else {
this.vtables.push(vtStart);
this.view.setInt32(objAbs, this.used - objectOffset, true);
}
this.inObject = false;
return objectOffset;
}
createString(s: string): Offset {
const bytes = new TextEncoder().encode(s);
this.writeInt8(0); // null terminator (readers rely on it)
this.startVector(1, bytes.length, 1);
this.writeBytes(bytes);
return this.endVector(bytes.length);
}
startVector(elemSize: number, numElems: number, alignment: number): void {
this.prep(SIZEOF_INT, elemSize * numElems);
this.prep(alignment, elemSize * numElems);
}
endVector(numElems: number): Offset {
this.writeInt32(numElems);
return this.used;
}
/** Vector of table offsets. Elements are written in reverse build order. */
createOffsetVector(items: Offset[]): Offset {
this.startVector(SIZEOF_INT, items.length, SIZEOF_INT);
for (let i = items.length - 1; i >= 0; i--) {
this.writeInt32(this.used + SIZEOF_INT - items[i]);
}
return this.endVector(items.length);
}
/** Finish the buffer with `root` as the root table. Returns the bytes. */
finish(root: Offset): Uint8Array {
this.prep(this.minAlign, SIZEOF_INT);
this.prep(SIZEOF_INT);
this.writeInt32(this.used + SIZEOF_INT - root);
return this.buf.slice(this.buf.length - this.used);
}
}
+547
View File
@@ -0,0 +1,547 @@
/**
* src/analytics/arrow/ipc.ts
* ===========================
* BI-P005 — APACHE ARROW INTERCHANGE.
*
* Canonical columnar result encoding ALONGSIDE JSON (JSON is not removed —
* BI-P005 says "alongside"). Two paths, and knowing which is which matters:
*
* PASS-THROUGH (preferred for ClickHouse results): ClickHouse emits real
* Arrow IPC itself via `FORMAT ArrowStream`. Those bytes were produced by
* the Arrow C++ implementation; 0pt streams them out unchanged, so there is
* no encoder of ours in the path at all. `clickhouseArrowStreamFormat()`
* and `arrowStreamSettings()` are that path.
*
* ENCODED (for results 0pt itself assembles — projections, JSON-sourced
* rows, exports): `encodeArrowStream()` writes the IPC stream here.
*
* Both paths are decodable by `decodeArrowStream()`, which exists so 0pt can
* count rows and read the schema of an Arrow payload WITHOUT shelling out to
* another runtime — needed to enforce BI-P012's `max_result_rows` on the
* pass-through path, where the row count is not otherwise visible.
*
* "Supported types" (the DoD's qualifier) is `ARROW_TYPE_SUPPORT` below: the
* exact ClickHouse -> Arrow -> DuckDB/Python mapping, including where
* semantic loss WOULD occur, stated rather than hidden.
*
* IPC stream layout (Arrow format 1.5, metadata V5):
* for each message: 0xFFFFFFFF, int32 metadata length, FlatBuffer Message
* padded so the body starts 8-byte aligned, then the body buffers.
* Terminated by 0xFFFFFFFF 0x00000000.
*/
import { FlatBufferBuilder, type Offset } from "./flatbuffer-builder";
import { SiaxError } from "../../platform/errors";
// ── FlatBuffer enum values from org.apache.arrow.flatbuf ────────────────────
const METADATA_VERSION_V5 = 4;
const MESSAGE_HEADER_SCHEMA = 1;
const MESSAGE_HEADER_RECORD_BATCH = 3;
const ENDIANNESS_LITTLE = 0;
const TYPE_INT = 2;
const TYPE_FLOATING_POINT = 3;
const TYPE_UTF8 = 5;
const TYPE_BOOL = 6;
const TYPE_TIMESTAMP = 10;
const PRECISION_DOUBLE = 2;
// BigInt LITERALS (`0n`) require target >= ES2020; this repo targets ES6, so
// every 64-bit constant is built with BigInt(...) instead. Behaviour is
// identical — this is a compiler-target accommodation, not a semantic choice.
const BIG_ZERO = BigInt(0);
const INT64_MAX = BigInt("9223372036854775807");
const INT64_MIN = BigInt("-9223372036854775808");
const TIME_UNIT_MILLISECOND = 1;
// ── public column model ────────────────────────────────────────────────────
export type ArrowColumnType = "int64" | "float64" | "utf8" | "bool" | "timestamp_ms";
export interface ArrowColumn {
name: string;
type: ArrowColumnType;
/** `null` marks a null slot; the validity bitmap is derived from it. */
values: (number | bigint | string | boolean | Date | null)[];
}
/**
* The type table BI-P005's "supported types" qualifier refers to. Every row
* has been exercised end to end by scripts/bi-p005-arrow-roundtrip.mjs.
*/
export const ARROW_TYPE_SUPPORT: {
clickhouse: string;
arrow: ArrowColumnType;
duckdb: string;
python: string;
note: string;
}[] = [
{
clickhouse: "Int64 / UInt32 / Int32",
arrow: "int64",
duckdb: "BIGINT",
python: "int",
note: "exact; UInt64 above 2^63 is NOT supported and is rejected rather than silently wrapped",
},
{
clickhouse: "Float64",
arrow: "float64",
duckdb: "DOUBLE",
python: "float",
note: "exact IEEE-754 round trip",
},
{
clickhouse: "String / LowCardinality(String)",
arrow: "utf8",
duckdb: "VARCHAR",
python: "str",
note: "LowCardinality is decoded to plain utf8 — dictionary encoding is lost, values are not",
},
{
clickhouse: "Bool / UInt8(0|1)",
arrow: "bool",
duckdb: "BOOLEAN",
python: "bool",
note: "bit-packed one bit per value; ClickHouse UInt8 other than 0/1 is NOT a bool and must be sent as int64",
},
{
clickhouse: "DateTime64(3,'UTC')",
arrow: "timestamp_ms",
duckdb: "TIMESTAMP",
python: "datetime",
note: "millisecond precision; DateTime64(6/9) loses sub-ms and is therefore NOT in the supported set",
},
];
// ── ClickHouse pass-through ────────────────────────────────────────────────
/** ClickHouse output format producing an Arrow IPC *stream*. */
export function clickhouseArrowStreamFormat(): string {
return "ArrowStream";
}
/**
* Settings for the pass-through path. `output_format_arrow_string_as_string`
* matters: without it ClickHouse emits String columns as Arrow *Binary*,
* which pyarrow and DuckDB both read as bytes rather than text — semantic
* loss that would slip through a test that only compares row counts.
*/
export function arrowStreamSettings(): Record<string, string> {
return {
output_format_arrow_string_as_string: "1",
output_format_arrow_low_cardinality_as_dictionary: "0",
};
}
// ── encoder ────────────────────────────────────────────────────────────────
function alignTo8(n: number): number {
return (n + 7) & ~7;
}
interface EncodedBuffer {
bytes: Uint8Array;
/** Logical (unpadded) length recorded in the RecordBatch metadata. */
length: number;
}
function validityBuffer(values: readonly unknown[]): { buf: EncodedBuffer; nullCount: number } {
let nullCount = 0;
const byteLen = Math.ceil(values.length / 8);
const bytes = new Uint8Array(alignTo8(byteLen));
for (let i = 0; i < values.length; i++) {
if (values[i] === null || values[i] === undefined) {
nullCount += 1;
} else {
bytes[i >> 3] |= 1 << (i & 7);
}
}
return { buf: { bytes, length: byteLen }, nullCount };
}
function encodeColumn(col: ArrowColumn): { buffers: EncodedBuffer[]; nullCount: number } {
const { buf: validity, nullCount } = validityBuffer(col.values);
const n = col.values.length;
switch (col.type) {
case "int64": {
const data = new Uint8Array(alignTo8(n * 8));
const dv = new DataView(data.buffer);
col.values.forEach((v, i) => {
if (v === null || v === undefined) return;
const b = typeof v === "bigint" ? v : BigInt(Math.trunc(Number(v)));
if (b > INT64_MAX || b < INT64_MIN) {
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", `value out of int64 range in column "${col.name}"`);
}
dv.setBigInt64(i * 8, b, true);
});
return { buffers: [validity, { bytes: data, length: n * 8 }], nullCount };
}
case "float64": {
const data = new Uint8Array(alignTo8(n * 8));
const dv = new DataView(data.buffer);
col.values.forEach((v, i) => {
if (v === null || v === undefined) return;
dv.setFloat64(i * 8, Number(v), true);
});
return { buffers: [validity, { bytes: data, length: n * 8 }], nullCount };
}
case "timestamp_ms": {
const data = new Uint8Array(alignTo8(n * 8));
const dv = new DataView(data.buffer);
col.values.forEach((v, i) => {
if (v === null || v === undefined) return;
const ms = v instanceof Date ? v.getTime() : typeof v === "bigint" ? Number(v) : Number(v);
dv.setBigInt64(i * 8, BigInt(Math.trunc(ms)), true);
});
return { buffers: [validity, { bytes: data, length: n * 8 }], nullCount };
}
case "bool": {
const byteLen = Math.ceil(n / 8);
const data = new Uint8Array(alignTo8(byteLen));
col.values.forEach((v, i) => {
if (v === null || v === undefined) return;
if (v) data[i >> 3] |= 1 << (i & 7);
});
return { buffers: [validity, { bytes: data, length: byteLen }], nullCount };
}
case "utf8": {
const encoded = col.values.map((v) =>
v === null || v === undefined ? new Uint8Array(0) : new TextEncoder().encode(String(v)),
);
const total = encoded.reduce((a, b) => a + b.length, 0);
const offsets = new Uint8Array(alignTo8((n + 1) * 4));
const odv = new DataView(offsets.buffer);
const data = new Uint8Array(alignTo8(total));
let pos = 0;
odv.setInt32(0, 0, true);
encoded.forEach((bytes, i) => {
data.set(bytes, pos);
pos += bytes.length;
odv.setInt32((i + 1) * 4, pos, true);
});
return {
buffers: [validity, { bytes: offsets, length: (n + 1) * 4 }, { bytes: data, length: total }],
nullCount,
};
}
default: {
const never: never = col.type;
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", `unsupported arrow type ${String(never)}`);
}
}
}
function buildTypeTable(b: FlatBufferBuilder, type: ArrowColumnType): { typeType: number; offset: Offset } {
switch (type) {
case "int64": {
b.startObject(2);
b.addInt32(0, 64, 0); // bitWidth
b.addInt8(1, 1, 0); // is_signed
return { typeType: TYPE_INT, offset: b.endObject() };
}
case "float64": {
b.startObject(1);
b.addInt16(0, PRECISION_DOUBLE, 0); // precision = DOUBLE
return { typeType: TYPE_FLOATING_POINT, offset: b.endObject() };
}
case "utf8": {
b.startObject(0);
return { typeType: TYPE_UTF8, offset: b.endObject() };
}
case "bool": {
b.startObject(0);
return { typeType: TYPE_BOOL, offset: b.endObject() };
}
case "timestamp_ms": {
// Build the timezone string BEFORE startObject — FlatBuffers forbids
// creating a nested object while a table is open.
const tz = b.createString("UTC");
b.startObject(2);
b.addInt16(0, TIME_UNIT_MILLISECOND, 0);
b.addOffset(1, tz, 0);
return { typeType: TYPE_TIMESTAMP, offset: b.endObject() };
}
default: {
const never: never = type;
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", `unsupported arrow type ${String(never)}`);
}
}
}
function buildSchemaMessage(columns: readonly ArrowColumn[]): Uint8Array {
const b = new FlatBufferBuilder(2048);
const fieldOffsets: Offset[] = columns.map((col) => {
const nameOffset = b.createString(col.name);
const { typeType, offset: typeOffset } = buildTypeTable(b, col.type);
const children = b.createOffsetVector([]);
b.startObject(7);
b.addOffset(0, nameOffset, 0);
b.addInt8(1, 1, 0); // nullable = true
b.addInt8(2, typeType, 0);
b.addOffset(3, typeOffset, 0);
b.addOffset(5, children, 0);
return b.endObject();
});
const fieldsVector = b.createOffsetVector(fieldOffsets);
b.startObject(4);
b.addInt16(0, ENDIANNESS_LITTLE, 0);
b.addOffset(1, fieldsVector, 0);
const schema = b.endObject();
b.startObject(5);
b.addInt16(0, METADATA_VERSION_V5, 0);
b.addInt8(1, MESSAGE_HEADER_SCHEMA, 0);
b.addOffset(2, schema, 0);
b.addInt64(3, BIG_ZERO, BIG_ZERO); // bodyLength
const message = b.endObject();
return b.finish(message);
}
function buildRecordBatchMessage(
rowCount: number,
nodes: { length: number; nullCount: number }[],
buffers: { offset: number; length: number }[],
bodyLength: number,
): Uint8Array {
const b = new FlatBufferBuilder(2048);
// FieldNode[] — inline structs, written in reverse.
b.startVector(16, nodes.length, 8);
for (let i = nodes.length - 1; i >= 0; i--) {
b.writeInt64(BigInt(nodes[i].nullCount));
b.writeInt64(BigInt(nodes[i].length));
}
const nodesVector = b.endVector(nodes.length);
// Buffer[] — inline structs, written in reverse.
b.startVector(16, buffers.length, 8);
for (let i = buffers.length - 1; i >= 0; i--) {
b.writeInt64(BigInt(buffers[i].length));
b.writeInt64(BigInt(buffers[i].offset));
}
const buffersVector = b.endVector(buffers.length);
b.startObject(5);
b.addInt64(0, BigInt(rowCount), BIG_ZERO);
b.addOffset(1, nodesVector, 0);
b.addOffset(2, buffersVector, 0);
const recordBatch = b.endObject();
b.startObject(5);
b.addInt16(0, METADATA_VERSION_V5, 0);
b.addInt8(1, MESSAGE_HEADER_RECORD_BATCH, 0);
b.addOffset(2, recordBatch, 0);
b.addInt64(3, BigInt(bodyLength), BIG_ZERO);
const message = b.endObject();
return b.finish(message);
}
function framedMessage(metadata: Uint8Array, body: Uint8Array): Uint8Array {
// continuation(4) + length(4) + metadata, padded so the body is 8-aligned.
const prefix = 8;
const padded = alignTo8(prefix + metadata.length) - prefix;
const out = new Uint8Array(prefix + padded + body.length);
const dv = new DataView(out.buffer);
dv.setUint32(0, 0xffffffff, true);
dv.setInt32(4, padded, true);
out.set(metadata, 8);
out.set(body, prefix + padded);
return out;
}
/**
* Encode columns as a complete Arrow IPC *stream* (schema message, one record
* batch, end-of-stream marker).
*/
export function encodeArrowStream(columns: readonly ArrowColumn[]): Uint8Array {
if (columns.length === 0) {
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "an Arrow stream needs at least one column");
}
const rowCount = columns[0].values.length;
for (const c of columns) {
if (c.values.length !== rowCount) {
throw new SiaxError(
"INVALID_REQUEST",
"0pt.analytics",
`column "${c.name}" has ${c.values.length} values, expected ${rowCount}`,
);
}
}
const schemaMsg = buildSchemaMessage(columns);
const nodes: { length: number; nullCount: number }[] = [];
const bufferMeta: { offset: number; length: number }[] = [];
const bodyParts: Uint8Array[] = [];
let bodyOffset = 0;
for (const col of columns) {
const { buffers, nullCount } = encodeColumn(col);
nodes.push({ length: rowCount, nullCount });
for (const buf of buffers) {
bufferMeta.push({ offset: bodyOffset, length: buf.length });
bodyParts.push(buf.bytes);
bodyOffset += buf.bytes.length; // already 8-aligned by encodeColumn
}
}
const body = new Uint8Array(bodyOffset);
let pos = 0;
for (const part of bodyParts) {
body.set(part, pos);
pos += part.length;
}
const batchMsg = buildRecordBatchMessage(rowCount, nodes, bufferMeta, body.length);
const schemaFrame = framedMessage(schemaMsg, new Uint8Array(0));
const batchFrame = framedMessage(batchMsg, body);
const eos = new Uint8Array(8);
new DataView(eos.buffer).setUint32(0, 0xffffffff, true); // length 0 = EOS
const out = new Uint8Array(schemaFrame.length + batchFrame.length + eos.length);
out.set(schemaFrame, 0);
out.set(batchFrame, schemaFrame.length);
out.set(eos, schemaFrame.length + batchFrame.length);
return out;
}
// ── decoder (structural: schema names/types + row counts) ──────────────────
export interface ArrowStreamInfo {
fields: { name: string; typeType: number }[];
/** Total rows across every record batch in the stream. */
rowCount: number;
batchCount: number;
}
/** Minimal FlatBuffers table reader — enough for Message/Schema/RecordBatch. */
class FbTable {
constructor(
readonly dv: DataView,
readonly pos: number,
) {}
fieldPos(slot: number): number {
const vtable = this.pos - this.dv.getInt32(this.pos, true);
const vtLen = this.dv.getInt16(vtable, true);
const idx = 4 + slot * 2;
if (idx >= vtLen) return 0;
const rel = this.dv.getInt16(vtable + idx, true);
return rel === 0 ? 0 : this.pos + rel;
}
readInt8(slot: number, def = 0): number {
const p = this.fieldPos(slot);
return p === 0 ? def : this.dv.getInt8(p);
}
readInt16(slot: number, def = 0): number {
const p = this.fieldPos(slot);
return p === 0 ? def : this.dv.getInt16(p, true);
}
readInt64(slot: number, def: bigint = BIG_ZERO): bigint {
const p = this.fieldPos(slot);
return p === 0 ? def : this.dv.getBigInt64(p, true);
}
/** Follow a uoffset to a nested table. */
readTable(slot: number): FbTable | undefined {
const p = this.fieldPos(slot);
if (p === 0) return undefined;
return new FbTable(this.dv, p + this.dv.getInt32(p, true));
}
vectorLength(slot: number): number {
const p = this.fieldPos(slot);
if (p === 0) return 0;
const vec = p + this.dv.getInt32(p, true);
return this.dv.getInt32(vec, true);
}
vectorElementTable(slot: number, index: number): FbTable {
const p = this.fieldPos(slot);
const vec = p + this.dv.getInt32(p, true);
const elem = vec + 4 + index * 4;
return new FbTable(this.dv, elem + this.dv.getInt32(elem, true));
}
readString(slot: number): string | undefined {
const p = this.fieldPos(slot);
if (p === 0) return undefined;
const str = p + this.dv.getInt32(p, true);
const len = this.dv.getInt32(str, true);
const bytes = new Uint8Array(this.dv.buffer, this.dv.byteOffset + str + 4, len);
return new TextDecoder().decode(bytes);
}
}
/**
* Parse an Arrow IPC stream's structure. Used to enforce BI-P012's result-row
* ceiling on Arrow payloads (including ClickHouse pass-through bytes, where
* the count is otherwise invisible to 0pt) and by the roundtrip tests.
*/
export function decodeArrowStream(bytes: Uint8Array): ArrowStreamInfo {
const dv = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
let pos = 0;
let fields: { name: string; typeType: number }[] = [];
let rowCount = 0;
let batchCount = 0;
while (pos + 8 <= bytes.byteLength) {
const continuation = dv.getUint32(pos, true);
if (continuation !== 0xffffffff) {
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "not an Arrow IPC stream (missing continuation marker)");
}
const metaLen = dv.getInt32(pos + 4, true);
if (metaLen === 0) break; // end of stream
const metaStart = pos + 8;
const root = metaStart + dv.getInt32(metaStart, true);
const message = new FbTable(dv, root);
const headerType = message.readInt8(1, 0);
const bodyLength = Number(message.readInt64(3, BIG_ZERO));
if (headerType === MESSAGE_HEADER_SCHEMA) {
const schema = message.readTable(2);
if (schema) {
const n = schema.vectorLength(1);
fields = [];
for (let i = 0; i < n; i++) {
const f = schema.vectorElementTable(1, i);
fields.push({ name: f.readString(0) ?? "", typeType: f.readInt8(2, 0) });
}
}
} else if (headerType === MESSAGE_HEADER_RECORD_BATCH) {
const rb = message.readTable(2);
if (rb) {
rowCount += Number(rb.readInt64(0, BIG_ZERO));
batchCount += 1;
}
}
pos = metaStart + metaLen + bodyLength;
}
return { fields, rowCount, batchCount };
}
/**
* BI-P012 on the Arrow path: refuse a payload that exceeds the caller's
* result-row ceiling. Without this, asking for Arrow would be a way to walk
* around `max_result_rows`.
*/
export function assertArrowWithinResultBudget(bytes: Uint8Array, maxResultRows: number): ArrowStreamInfo {
const info = decodeArrowStream(bytes);
if (info.rowCount > maxResultRows) {
throw new SiaxError(
"QUOTA_EXCEEDED",
"0pt.analytics",
`Arrow result has ${info.rowCount} rows, over the ${maxResultRows}-row budget`,
);
}
return info;
}
+474
View File
@@ -0,0 +1,474 @@
/**
* src/analytics/domain/contract.ts
* =================================
* BI-P004 — ANALYTICS DATA CONTRACT.
*
* `DatasetContract` is the machine-checkable promise a dataset makes to every
* consumer downstream of it. BI-P004's DoD is a GATE, not a document:
*
* "Breaking dataset schema changes fail compatibility gate unless
* explicitly versioned."
*
* so `checkContractCompatibility` is the deliverable and the prose is the
* commentary. The gate is fail-closed: an unrecognised change is BREAKING,
* never "probably fine".
*
* Versioning is semver-shaped (`major.minor.patch`) with one rule that is not
* negotiable: a BREAKING change requires a MAJOR bump. A breaking change
* shipped under a minor bump is exactly the failure this gate exists to
* catch, so it is reported as a violation rather than tolerated.
*/
import crypto from "node:crypto";
import { SiaxError } from "../../platform/errors";
import type { AnalyticsFieldType, FieldRole, PiiClassification } from "./entities";
import type { AnalyticsTier } from "./tiers";
export interface ContractField {
name: string;
type: AnalyticsFieldType;
nullable: boolean;
role: FieldRole;
/** Part of the dataset's primary identifier. */
primary_identifier?: boolean;
classification?: PiiClassification;
description?: string;
}
export interface ContractQualityRule {
rule_id: string;
kind: string;
field_name?: string;
severity: "warn" | "block";
}
/** BI-P004's named contents, one field each. */
export interface DatasetContract {
dataset_id: string;
/** semver `major.minor.patch`. */
version: string;
tier: AnalyticsTier;
fields: ContractField[];
/** Field names forming the primary identifier, in order. */
primary_identifiers: string[];
/** Field names usable as dimensions (subset of `fields`). */
dimensions: string[];
/** Field names usable as measures (subset of `fields`). */
measures: string[];
/** Freshness SLO in seconds. */
freshness_slo_seconds: number;
retention_days: number;
/** Dataset-level PII classification (the max of its fields, at least). */
pii_classification: PiiClassification;
/** Upstream refs: source ids, dataset ids, external system refs. */
source_lineage: string[];
quality_rules: ContractQualityRule[];
/** GDPR purpose limitation — what this data may be USED for. */
allowed_purposes: string[];
}
export interface SemVer {
major: number;
minor: number;
patch: number;
}
const SEMVER_RE = /^(\d+)\.(\d+)\.(\d+)$/;
export function parseSemVer(v: string): SemVer {
const m = SEMVER_RE.exec(v);
if (!m) {
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", `contract version "${v}" is not major.minor.patch`);
}
return { major: Number(m[1]), minor: Number(m[2]), patch: Number(m[3]) };
}
// ── change classification ──────────────────────────────────────────────────
export type ChangeSeverity = "compatible" | "breaking";
export interface ContractChange {
severity: ChangeSeverity;
/** Stable machine code so callers can allow-list specific changes. */
code: string;
field?: string;
detail: string;
}
/**
* Type widening that does NOT break a reader. Deliberately conservative:
* int64 -> float64 loses exactness for large integers, so it is NOT here.
* Everything absent from this table is breaking.
*/
const SAFE_TYPE_WIDENING: Partial<Record<AnalyticsFieldType, readonly AnalyticsFieldType[]>> = {
date: ["timestamp"],
};
function typeChangeIsCompatible(from: AnalyticsFieldType, to: AnalyticsFieldType): boolean {
if (from === to) return true;
return (SAFE_TYPE_WIDENING[from] ?? []).includes(to);
}
/**
* Diff two contracts for the SAME dataset and classify every change.
*
* Breaking (a reader written against `previous` can now fail or silently
* change meaning):
* - field removed
* - field type changed other than a listed safe widening
* - nullable false -> true (readers may not handle nulls)
* - a field's role changes (a measure becoming a dimension changes meaning)
* - primary identifier set changes
* - a field leaves `measures` or `dimensions`
* - freshness SLO loosened (a consumer's own SLO may depend on it)
* - retention shortened (history a consumer relied on disappears)
* - an allowed purpose is removed (a legitimate consumer becomes illegitimate)
* - PII classification raised (consumers may lack clearance)
* - a blocking quality rule is removed or downgraded to warn
*
* Compatible: additive nullable fields, new dimensions/measures, new quality
* rules, loosened PII classification, added purposes, longer retention,
* tightened freshness.
*/
export function diffContracts(previous: DatasetContract, next: DatasetContract): ContractChange[] {
if (previous.dataset_id !== next.dataset_id) {
throw new SiaxError(
"INVALID_REQUEST",
"0pt.analytics",
"contract compatibility is only defined between versions of the SAME dataset",
);
}
const changes: ContractChange[] = [];
const prevFields = new Map(previous.fields.map((f) => [f.name, f]));
const nextFields = new Map(next.fields.map((f) => [f.name, f]));
for (const [name, pf] of prevFields) {
const nf = nextFields.get(name);
if (!nf) {
changes.push({ severity: "breaking", code: "field_removed", field: name, detail: `field "${name}" removed` });
continue;
}
if (!typeChangeIsCompatible(pf.type, nf.type)) {
changes.push({
severity: "breaking",
code: "field_type_changed",
field: name,
detail: `field "${name}" type ${pf.type} -> ${nf.type}`,
});
} else if (pf.type !== nf.type) {
changes.push({
severity: "compatible",
code: "field_type_widened",
field: name,
detail: `field "${name}" widened ${pf.type} -> ${nf.type}`,
});
}
if (!pf.nullable && nf.nullable) {
changes.push({
severity: "breaking",
code: "field_became_nullable",
field: name,
detail: `field "${name}" became nullable`,
});
}
if (pf.nullable && !nf.nullable) {
changes.push({
severity: "compatible",
code: "field_became_required",
field: name,
detail: `field "${name}" is no longer nullable`,
});
}
if (pf.role !== nf.role) {
changes.push({
severity: "breaking",
code: "field_role_changed",
field: name,
detail: `field "${name}" role ${pf.role} -> ${nf.role}`,
});
}
}
for (const [name, nf] of nextFields) {
if (prevFields.has(name)) continue;
if (nf.nullable) {
changes.push({
severity: "compatible",
code: "nullable_field_added",
field: name,
detail: `nullable field "${name}" added`,
});
} else {
// A required new field breaks producers, and breaks readers that
// reconstruct rows (they now have no value to supply).
changes.push({
severity: "breaking",
code: "required_field_added",
field: name,
detail: `required (non-nullable) field "${name}" added`,
});
}
}
const idsEqual =
previous.primary_identifiers.length === next.primary_identifiers.length &&
previous.primary_identifiers.every((v, i) => next.primary_identifiers[i] === v);
if (!idsEqual) {
changes.push({
severity: "breaking",
code: "primary_identifier_changed",
detail: `primary identifier [${previous.primary_identifiers.join(",")}] -> [${next.primary_identifiers.join(",")}]`,
});
}
for (const d of previous.dimensions) {
if (!next.dimensions.includes(d)) {
changes.push({ severity: "breaking", code: "dimension_removed", field: d, detail: `dimension "${d}" removed` });
}
}
for (const d of next.dimensions) {
if (!previous.dimensions.includes(d)) {
changes.push({ severity: "compatible", code: "dimension_added", field: d, detail: `dimension "${d}" added` });
}
}
for (const m of previous.measures) {
if (!next.measures.includes(m)) {
changes.push({ severity: "breaking", code: "measure_removed", field: m, detail: `measure "${m}" removed` });
}
}
for (const m of next.measures) {
if (!previous.measures.includes(m)) {
changes.push({ severity: "compatible", code: "measure_added", field: m, detail: `measure "${m}" added` });
}
}
if (next.freshness_slo_seconds > previous.freshness_slo_seconds) {
changes.push({
severity: "breaking",
code: "freshness_slo_loosened",
detail: `freshness SLO ${previous.freshness_slo_seconds}s -> ${next.freshness_slo_seconds}s`,
});
}
if (next.retention_days < previous.retention_days) {
changes.push({
severity: "breaking",
code: "retention_shortened",
detail: `retention ${previous.retention_days}d -> ${next.retention_days}d`,
});
}
for (const p of previous.allowed_purposes) {
if (!next.allowed_purposes.includes(p)) {
changes.push({ severity: "breaking", code: "purpose_removed", detail: `allowed purpose "${p}" removed` });
}
}
const PII_RANK: Record<PiiClassification, number> = {
none: 0,
pseudonymous: 1,
personal: 2,
sensitive: 3,
};
if (PII_RANK[next.pii_classification] > PII_RANK[previous.pii_classification]) {
changes.push({
severity: "breaking",
code: "pii_classification_raised",
detail: `PII classification ${previous.pii_classification} -> ${next.pii_classification}`,
});
}
const prevRules = new Map(previous.quality_rules.map((r) => [r.rule_id, r]));
for (const [id, pr] of prevRules) {
const nr = next.quality_rules.find((r) => r.rule_id === id);
if (!nr && pr.severity === "block") {
changes.push({
severity: "breaking",
code: "blocking_quality_rule_removed",
detail: `blocking quality rule "${id}" removed`,
});
} else if (nr && pr.severity === "block" && nr.severity === "warn") {
changes.push({
severity: "breaking",
code: "blocking_quality_rule_downgraded",
detail: `quality rule "${id}" downgraded block -> warn`,
});
}
}
if (previous.tier !== next.tier) {
changes.push({
severity: "breaking",
code: "tier_changed",
detail: `tier ${previous.tier} -> ${next.tier}`,
});
}
return changes;
}
export interface CompatibilityVerdict {
compatible: boolean;
/** True when a MAJOR bump legitimises the breaking changes. */
explicitly_versioned: boolean;
changes: ContractChange[];
breaking: ContractChange[];
previous_version: string;
next_version: string;
/** Human-readable reason when `compatible` is false. */
reason?: string;
}
/**
* BI-P004's gate. Returns a verdict; `assertContractCompatible` throws.
*
* Rules, in order:
* 1. the version must move FORWARD (equal or lower is rejected outright);
* 2. breaking changes REQUIRE a major bump — that is what "explicitly
* versioned" means;
* 3. a major bump with no breaking change is allowed (over-declaring is
* safe; under-declaring is what we refuse).
*/
export function checkContractCompatibility(
previous: DatasetContract,
next: DatasetContract,
): CompatibilityVerdict {
const pv = parseSemVer(previous.version);
const nv = parseSemVer(next.version);
const changes = diffContracts(previous, next);
const breaking = changes.filter((c) => c.severity === "breaking");
const forward =
nv.major > pv.major ||
(nv.major === pv.major && nv.minor > pv.minor) ||
(nv.major === pv.major && nv.minor === pv.minor && nv.patch > pv.patch);
if (!forward) {
return {
compatible: false,
explicitly_versioned: false,
changes,
breaking,
previous_version: previous.version,
next_version: next.version,
reason: `version must move forward: ${previous.version} -> ${next.version}`,
};
}
const majorBumped = nv.major > pv.major;
if (breaking.length > 0 && !majorBumped) {
return {
compatible: false,
explicitly_versioned: false,
changes,
breaking,
previous_version: previous.version,
next_version: next.version,
reason:
`${breaking.length} breaking change(s) require a MAJOR version bump ` +
`(${previous.version} -> ${next.version} is not one): ` +
breaking.map((b) => b.code).join(", "),
};
}
return {
compatible: true,
explicitly_versioned: majorBumped && breaking.length > 0,
changes,
breaking,
previous_version: previous.version,
next_version: next.version,
};
}
/** Throwing form for call sites that treat incompatibility as a failure. */
export function assertContractCompatible(previous: DatasetContract, next: DatasetContract): CompatibilityVerdict {
const verdict = checkContractCompatibility(previous, next);
if (!verdict.compatible) {
throw new SiaxError("CONFLICT", "0pt.analytics", verdict.reason ?? "dataset contract compatibility gate failed");
}
return verdict;
}
/**
* Structural validation of a single contract (independent of history).
* Catches the contract that is internally inconsistent before it is ever
* compared to anything.
*/
export function validateContract(contract: DatasetContract): void {
parseSemVer(contract.version);
if (!contract.dataset_id.trim()) {
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "contract needs a dataset_id");
}
if (contract.fields.length === 0) {
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", `contract ${contract.dataset_id} declares no fields`);
}
const names = new Set<string>();
for (const f of contract.fields) {
if (names.has(f.name)) {
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", `duplicate field "${f.name}"`);
}
names.add(f.name);
}
for (const label of ["primary_identifiers", "dimensions", "measures"] as const) {
for (const n of contract[label]) {
if (!names.has(n)) {
throw new SiaxError(
"INVALID_REQUEST",
"0pt.analytics",
`${label} references unknown field "${n}" in ${contract.dataset_id}`,
);
}
}
}
if (contract.primary_identifiers.length === 0) {
throw new SiaxError(
"INVALID_REQUEST",
"0pt.analytics",
`contract ${contract.dataset_id} declares no primary identifier`,
);
}
if (contract.allowed_purposes.length === 0) {
throw new SiaxError(
"INVALID_REQUEST",
"0pt.analytics",
`contract ${contract.dataset_id} declares no allowed_purposes (purpose limitation is not optional)`,
);
}
if (contract.freshness_slo_seconds <= 0) {
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "freshness_slo_seconds must be positive");
}
if (contract.retention_days < 0) {
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "retention_days must not be negative");
}
}
/**
* Stable hash of a contract's MEANING (field order-insensitive), used as
* `DatasetVersion.contract_hash` so two versions that differ only in comment
* text do not look like schema drift.
*/
export function contractHash(contract: DatasetContract): string {
const canonical = {
dataset_id: contract.dataset_id,
version: contract.version,
tier: contract.tier,
fields: [...contract.fields]
.map((f) => ({
name: f.name,
type: f.type,
nullable: f.nullable,
role: f.role,
primary_identifier: f.primary_identifier ?? false,
classification: f.classification ?? "none",
}))
.sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0)),
primary_identifiers: contract.primary_identifiers,
dimensions: [...contract.dimensions].sort(),
measures: [...contract.measures].sort(),
freshness_slo_seconds: contract.freshness_slo_seconds,
retention_days: contract.retention_days,
pii_classification: contract.pii_classification,
source_lineage: [...contract.source_lineage].sort(),
quality_rules: [...contract.quality_rules].sort((a, b) => (a.rule_id < b.rule_id ? -1 : 1)),
allowed_purposes: [...contract.allowed_purposes].sort(),
};
return crypto.createHash("sha256").update(JSON.stringify(canonical)).digest("hex");
}
+536
View File
@@ -0,0 +1,536 @@
/**
* src/analytics/domain/entities.ts
* =================================
* BI-P002 — ANALYTICS DOMAIN MODEL.
*
* The nineteen canonical analytics entities, named EXACTLY as the program
* lists them (no renaming, no merging, no "we already have something like
* that"):
*
* AnalyticsDataset DatasetVersion DatasetField
* DatasetGrant MetricDefinition MetricVersion
* DimensionDefinition SemanticModel AnalyticsQuery
* QueryExecution QueryResult DashboardDefinition
* DashboardVersion ReportDefinition ReportRun
* DataQualityRule DataQualityRun LineageReference
* BIApplicationBinding
*
* Every tenant-owned record carries `tenant_id` + `project_id` where
* applicable. The four version entities the program names as immutable
* (MetricVersion, SemanticModel version, DashboardVersion, DatasetVersion)
* are `readonly` here AND immutable in the database — sql/008 installs BEFORE
* UPDATE/DELETE triggers, because a TypeScript `readonly` is a compiler
* opinion, not a storage guarantee.
*
* NAME COLLISION, deliberate: src/services/dataset-version-service.ts already
* exports an interface called `DatasetVersion` for the ST0RE-artifact-backed
* dataset versioning that predates this program. That one is about *dataset
* content artifacts*; this one is about *analytics dataset schema versions*
* and lives in its own module namespace (`AnalyticsDatasetVersion` is NOT
* used as the name — BI-P002 lists `DatasetVersion` verbatim and that is what
* ships). Import them qualified when both are needed in one file.
*
* ARCHITECTURE BOUNDARY: these entities are owned by 0pt. A product repo
* (VehIQ) supplies domain MEANING through `BIApplicationBinding`; it never
* defines its own copies of these tables and never receives engine
* credentials.
*/
import type { AnalyticsTier } from "./tiers";
/** Every tenant-owned analytics record. */
export interface TenantScoped {
tenant_id: string;
project_id: string;
}
// ── field/typing vocabulary ────────────────────────────────────────────────
export const ANALYTICS_FIELD_TYPES = [
"string",
"int64",
"float64",
"bool",
"date",
"timestamp",
"decimal",
"json",
] as const;
export type AnalyticsFieldType = (typeof ANALYTICS_FIELD_TYPES)[number];
/** A field's analytical role. Measures aggregate; dimensions group. */
export const FIELD_ROLES = ["identifier", "dimension", "measure", "attribute"] as const;
export type FieldRole = (typeof FIELD_ROLES)[number];
/** PII classification, aligned with the SIAX risk vocabulary (L0 = none). */
export const PII_CLASSIFICATIONS = ["none", "pseudonymous", "personal", "sensitive"] as const;
export type PiiClassification = (typeof PII_CLASSIFICATIONS)[number];
// ── 1. AnalyticsDataset ────────────────────────────────────────────────────
export const DATASET_STATUSES = ["draft", "active", "deprecated", "retired"] as const;
export type AnalyticsDatasetStatus = (typeof DATASET_STATUSES)[number];
/**
* BI-P003 DoD: "Every dataset declares tier, schema, owner, source, freshness,
* quality, retention, classification." All eight are REQUIRED fields here —
* an optional declaration is not a declaration.
*/
export interface AnalyticsDataset extends TenantScoped {
dataset_id: string;
name: string;
/** BI-P003 tier. */
tier: AnalyticsTier;
/** Contract id + version this dataset's schema conforms to (BI-P004). */
schema_contract_id: string;
schema_contract_version: string;
/** Accountable human/team, not the person who typed the create call. */
owner_id: string;
/** Upstream: a 0pt source_id, another dataset_id, or an external ref. */
source_ref: string;
/** Freshness SLO in seconds — how stale the data may be before it is late. */
freshness_slo_seconds: number;
/** DataQualityRule ids that gate this dataset. */
quality_rule_ids: string[];
/** Retention in days. 0 = keep indefinitely (must be justified in policy). */
retention_days: number;
classification: PiiClassification;
status: AnalyticsDatasetStatus;
/** BI-P003 escape hatch for RAW/NORMALIZED exposure to a BI surface. */
raw_access_exception?: string;
description?: string;
created_at: string;
updated_at?: string;
created_by?: string;
}
// ── 2. DatasetVersion (IMMUTABLE) ──────────────────────────────────────────
export interface DatasetVersion extends TenantScoped {
readonly dataset_version_id: string;
readonly dataset_id: string;
/** Monotonic per dataset, starting at 1. */
readonly version: number;
/** The frozen DatasetContract JSON at this version (BI-P004). */
readonly contract_hash: string;
readonly schema_contract_version: string;
readonly record_count?: number;
readonly created_at: string;
readonly created_by?: string;
}
// ── 3. DatasetField ────────────────────────────────────────────────────────
export interface DatasetField extends TenantScoped {
dataset_field_id: string;
dataset_id: string;
/** The dataset version this field description belongs to. */
dataset_version: number;
name: string;
field_type: AnalyticsFieldType;
nullable: boolean;
role: FieldRole;
/** True for the field(s) forming the dataset's primary identifier. */
primary_identifier: boolean;
classification: PiiClassification;
description?: string;
}
// ── 4. DatasetGrant ────────────────────────────────────────────────────────
export const GRANT_LEVELS = ["read", "query", "export", "admin"] as const;
export type GrantLevel = (typeof GRANT_LEVELS)[number];
/**
* Who may use a dataset, and FOR WHAT. `allowed_purposes` is the GDPR purpose
* limitation carried into analytics — a grant without a purpose is refused
* by the repository layer.
*/
export interface DatasetGrant extends TenantScoped {
grant_id: string;
dataset_id: string;
/** Subject: a CL0UD principal, role, or BI application binding id. */
subject_id: string;
subject_type: "user" | "role" | "service" | "bi_application";
level: GrantLevel;
allowed_purposes: string[];
expires_at?: string;
created_at: string;
created_by?: string;
}
// ── 5. MetricDefinition ────────────────────────────────────────────────────
export const METRIC_AGGREGATIONS = [
"sum",
"avg",
"min",
"max",
"count",
"count_distinct",
"ratio",
"derived",
] as const;
export type MetricAggregation = (typeof METRIC_AGGREGATIONS)[number];
export interface MetricDefinition extends TenantScoped {
metric_id: string;
/** Stable business name; unique per tenant. */
name: string;
dataset_id: string;
aggregation: MetricAggregation;
/** Measure field aggregated, or the numerator for ratio metrics. */
measure_field: string;
/** Denominator for ratio metrics. */
denominator_field?: string;
/** Dimensions this metric may legitimately be sliced by. */
allowed_dimension_ids: string[];
/** Current live version number (points at a MetricVersion). */
current_version: number;
owner_id: string;
description?: string;
created_at: string;
updated_at?: string;
}
// ── 6. MetricVersion (IMMUTABLE) ───────────────────────────────────────────
export interface MetricVersion extends TenantScoped {
readonly metric_version_id: string;
readonly metric_id: string;
readonly version: number;
/** Frozen definition JSON — what this metric MEANT at this version. */
readonly definition_json: string;
readonly definition_hash: string;
/** Free-text reason the metric changed; required for version > 1. */
readonly change_reason?: string;
readonly created_at: string;
readonly created_by?: string;
}
// ── 7. DimensionDefinition ─────────────────────────────────────────────────
export interface DimensionDefinition extends TenantScoped {
dimension_id: string;
name: string;
dataset_id: string;
/**
* Physical column/key this dimension resolves to. This is the ONLY place a
* caller-supplied dimension name becomes a physical identifier — the query
* compiler resolves through here and never splices caller input (BI-P011
* "dimension identifier injection").
*/
field_name: string;
field_type: AnalyticsFieldType;
classification: PiiClassification;
description?: string;
created_at: string;
}
// ── 8. SemanticModel (versioned; versions IMMUTABLE) ───────────────────────
export interface SemanticModel extends TenantScoped {
semantic_model_id: string;
name: string;
/** Immutable version number — a change mints a new row, never an UPDATE. */
readonly version: number;
/** MetricFlow/Cube-compatible model document (BI-P029/BI-P032 consume it). */
readonly model_json: string;
readonly model_hash: string;
dataset_ids: string[];
owner_id: string;
readonly created_at: string;
readonly created_by?: string;
}
// ── 9. AnalyticsQuery ──────────────────────────────────────────────────────
export const QUERY_SURFACES = ["metric", "sql", "projection"] as const;
export type QuerySurface = (typeof QUERY_SURFACES)[number];
/**
* A SAVED, reusable query. The request that actually runs is
* `src/analytics/engine/query-compiler.ts`'s `MetricQueryRequest`; this entity
* is its durable, governed form.
*/
export interface AnalyticsQuery extends TenantScoped {
query_id: string;
name: string;
surface: QuerySurface;
dataset_id: string;
metric_id?: string;
/** Serialized closed-DSL request. Never raw SQL for surface="metric". */
request_json: string;
owner_id: string;
created_at: string;
updated_at?: string;
}
// ── 10. QueryExecution ─────────────────────────────────────────────────────
export const EXECUTION_STATES = [
"queued",
"running",
"succeeded",
"failed",
"cancelled",
"rejected",
] as const;
export type ExecutionState = (typeof EXECUTION_STATES)[number];
export interface QueryExecution extends TenantScoped {
execution_id: string;
query_id?: string;
/** Engine-side identifier used to CANCEL this execution (BI-P009). */
engine_query_id: string;
state: ExecutionState;
/** Which provider ran it: "clickhouse" | "duckdb" | "trino". */
provider: string;
submitted_at: string;
started_at?: string;
finished_at?: string;
duration_ms?: number;
rows_read?: number;
bytes_read?: number;
result_rows?: number;
/** SiaxErrorCode when state="failed"/"rejected". Never a raw engine body. */
error_code?: string;
correlation_id?: string;
}
// ── 11. QueryResult ────────────────────────────────────────────────────────
export const RESULT_ENCODINGS = ["json", "arrow_ipc"] as const;
export type ResultEncoding = (typeof RESULT_ENCODINGS)[number];
export interface QueryResult extends TenantScoped {
result_id: string;
execution_id: string;
encoding: ResultEncoding;
row_count: number;
/** Canonical ST0RE ref when materialised; inline results carry no bytes here. */
artifact_ref?: string;
/** True when the engine truncated at max_result_rows (BI-P012). */
truncated: boolean;
expires_at?: string;
created_at: string;
}
// ── 12. DashboardDefinition ────────────────────────────────────────────────
export interface DashboardDefinition extends TenantScoped {
dashboard_id: string;
name: string;
/** Live version pointer. */
current_version: number;
owner_id: string;
/** Datasets the dashboard reads — checked against BI-P003 tier rules. */
dataset_ids: string[];
description?: string;
created_at: string;
updated_at?: string;
}
// ── 13. DashboardVersion (IMMUTABLE) ───────────────────────────────────────
export interface DashboardVersion extends TenantScoped {
readonly dashboard_version_id: string;
readonly dashboard_id: string;
readonly version: number;
/** Frozen layout/panel document. */
readonly layout_json: string;
readonly layout_hash: string;
readonly created_at: string;
readonly created_by?: string;
}
// ── 14. ReportDefinition ───────────────────────────────────────────────────
export const REPORT_FORMATS = ["pdf", "csv", "xlsx", "parquet", "arrow_ipc"] as const;
export type ReportFormat = (typeof REPORT_FORMATS)[number];
export interface ReportDefinition extends TenantScoped {
report_id: string;
name: string;
dashboard_id?: string;
query_id?: string;
format: ReportFormat;
/** Cron expression; undefined = on-demand only. */
schedule_cron?: string;
owner_id: string;
created_at: string;
updated_at?: string;
}
// ── 15. ReportRun ──────────────────────────────────────────────────────────
export interface ReportRun extends TenantScoped {
report_run_id: string;
report_id: string;
state: ExecutionState;
started_at: string;
finished_at?: string;
/** ST0RE artifact ref of the produced report. */
artifact_ref?: string;
error_code?: string;
correlation_id?: string;
}
// ── 16. DataQualityRule ────────────────────────────────────────────────────
export const QUALITY_RULE_KINDS = [
"not_null",
"unique",
"accepted_values",
"range",
"freshness",
"row_count",
"referential",
"custom",
] as const;
export type QualityRuleKind = (typeof QUALITY_RULE_KINDS)[number];
export const QUALITY_SEVERITIES = ["warn", "block"] as const;
export type QualitySeverity = (typeof QUALITY_SEVERITIES)[number];
export interface DataQualityRule extends TenantScoped {
rule_id: string;
dataset_id: string;
kind: QualityRuleKind;
/** Field the rule applies to; omitted for dataset-level rules. */
field_name?: string;
/** Rule parameters (bounds, accepted values, max age). */
params_json: string;
severity: QualitySeverity;
created_at: string;
created_by?: string;
}
// ── 17. DataQualityRun ─────────────────────────────────────────────────────
export interface DataQualityRun extends TenantScoped {
quality_run_id: string;
rule_id: string;
dataset_id: string;
dataset_version?: number;
passed: boolean;
/** Rows that violated the rule. */
violation_count: number;
evaluated_rows: number;
started_at: string;
finished_at?: string;
correlation_id?: string;
}
// ── 18. LineageReference ───────────────────────────────────────────────────
export const LINEAGE_DIRECTIONS = ["input", "output"] as const;
export type LineageDirection = (typeof LINEAGE_DIRECTIONS)[number];
/**
* The queryable projection of an OpenLineage edge (BI-P027). AUD0 stays the
* system of record; this is 0pt's local index so "what fed this dashboard"
* is answerable without leaving the analytics plane.
*/
export interface LineageReference extends TenantScoped {
lineage_reference_id: string;
/** OpenLineage run id this edge came from. */
run_id: string;
job_name: string;
direction: LineageDirection;
/** The analytics entity this edge touches. */
entity_type: "dataset" | "metric" | "dashboard" | "report";
entity_id: string;
/** Namespaced dataset name on the other side of the edge. */
counterpart_ref: string;
observed_at: string;
}
// ── 19. BIApplicationBinding ───────────────────────────────────────────────
export const BI_APPLICATION_KINDS = ["superset", "evidence", "des1gn", "notebook", "custom"] as const;
export type BiApplicationKind = (typeof BI_APPLICATION_KINDS)[number];
/**
* How an external BI surface is attached to the analytics plane.
*
* NORMATIVE (BI program §0 + the SIAX/0pt ownership boundary): this binding
* carries NO engine credential. A bound application authenticates to the 0pt
* Analytics API with a CL0UD identity and receives compiled, tenant-forced
* results — it never learns a ClickHouse DSN, user, or password. The absence
* of a credential field here is the enforcement, not a comment.
*/
export interface BIApplicationBinding extends TenantScoped {
binding_id: string;
kind: BiApplicationKind;
name: string;
/** CL0UD principal the application authenticates as. */
principal_id: string;
/** Datasets/metrics this application may reach at all. */
dataset_ids: string[];
metric_ids: string[];
allowed_purposes: string[];
enabled: boolean;
created_at: string;
created_by?: string;
}
// ── registry of the canonical nineteen ─────────────────────────────────────
/**
* The canonical entity names, verbatim from BI-P002. Persisted-schema tests
* assert sql/008 defines a table for every one of these, so a future rename
* or omission is a test failure rather than drift.
*/
export const ANALYTICS_DOMAIN_ENTITIES = [
"AnalyticsDataset",
"DatasetVersion",
"DatasetField",
"DatasetGrant",
"MetricDefinition",
"MetricVersion",
"DimensionDefinition",
"SemanticModel",
"AnalyticsQuery",
"QueryExecution",
"QueryResult",
"DashboardDefinition",
"DashboardVersion",
"ReportDefinition",
"ReportRun",
"DataQualityRule",
"DataQualityRun",
"LineageReference",
"BIApplicationBinding",
] as const;
export type AnalyticsDomainEntity = (typeof ANALYTICS_DOMAIN_ENTITIES)[number];
/** Entity -> physical table in sql/008_analytics_domain_model.sql. */
export const ANALYTICS_ENTITY_TABLES: Record<AnalyticsDomainEntity, string> = {
AnalyticsDataset: "opt_analytics_dataset",
DatasetVersion: "opt_analytics_dataset_version",
DatasetField: "opt_analytics_dataset_field",
DatasetGrant: "opt_analytics_dataset_grant",
MetricDefinition: "opt_analytics_metric_definition",
MetricVersion: "opt_analytics_metric_version",
DimensionDefinition: "opt_analytics_dimension_definition",
SemanticModel: "opt_analytics_semantic_model",
AnalyticsQuery: "opt_analytics_query",
QueryExecution: "opt_analytics_query_execution",
QueryResult: "opt_analytics_query_result",
DashboardDefinition: "opt_analytics_dashboard_definition",
DashboardVersion: "opt_analytics_dashboard_version",
ReportDefinition: "opt_analytics_report_definition",
ReportRun: "opt_analytics_report_run",
DataQualityRule: "opt_analytics_data_quality_rule",
DataQualityRun: "opt_analytics_data_quality_run",
LineageReference: "opt_analytics_lineage_reference",
BIApplicationBinding: "opt_analytics_bi_application_binding",
};
/** The four entities BI-P002 requires to be immutable once written. */
export const IMMUTABLE_VERSION_ENTITIES: readonly AnalyticsDomainEntity[] = [
"DatasetVersion",
"MetricVersion",
"SemanticModel",
"DashboardVersion",
];
+83
View File
@@ -0,0 +1,83 @@
/**
* src/analytics/domain/tiers.ts
* ==============================
* BI-P003 — ANALYTICS DATA TIERS.
*
* The four canonical tiers every analytics dataset in SIAX belongs to. The
* tier is a CONTRACT about where a dataset may be read from and by whom, not
* a naming convention:
*
* RAW -> ST0RE Parquet/Iceberg (as-ingested, no business meaning)
* NORMALIZED -> ST0RE Iceberg (typed, deduplicated, still source-shaped)
* CURATED -> ST0RE Iceberg (business entities, modelled)
* SERVING -> ClickHouse / projection (query-optimised, dashboard-facing)
*
* BI-P003's normative sentence is "No business dashboard reads arbitrary raw
* ingestion tables by default" — `assertReadableByBiSurface` is where that
* sentence stops being prose. A BI surface (dashboard, report, embedded
* chart) may read SERVING and CURATED; RAW and NORMALIZED require an
* explicit, recorded exception on the dataset itself.
*
* ARCHITECTURE BOUNDARY (BI program §0): 0pt owns tier semantics. Product
* repos (VehIQ et al) own what the rows MEAN; they never re-declare tiers.
*/
import { SiaxError } from "../../platform/errors";
export const ANALYTICS_TIERS = ["RAW", "NORMALIZED", "CURATED", "SERVING"] as const;
export type AnalyticsTier = (typeof ANALYTICS_TIERS)[number];
/** Physical home a tier is expected to live in (BI-P003 "suggested" mapping). */
export const TIER_PHYSICAL_HOME: Record<AnalyticsTier, string> = {
RAW: "st0re:iceberg|parquet",
NORMALIZED: "st0re:iceberg",
CURATED: "st0re:iceberg",
SERVING: "clickhouse:projection",
};
/** Tiers a BI presentation surface may read without an explicit exception. */
export const BI_READABLE_TIERS: readonly AnalyticsTier[] = ["SERVING", "CURATED"];
export function isAnalyticsTier(v: unknown): v is AnalyticsTier {
return typeof v === "string" && (ANALYTICS_TIERS as readonly string[]).includes(v);
}
/**
* Rank used for promotion checks: a dataset may only be BUILT FROM a tier at
* or below its own (SERVING can be built from CURATED; CURATED must never be
* built from SERVING, which would make the projection its own source).
*/
export const TIER_RANK: Record<AnalyticsTier, number> = {
RAW: 0,
NORMALIZED: 1,
CURATED: 2,
SERVING: 3,
};
/**
* BI-P003 DoD gate. `raw_access_exception` is the recorded, auditable escape
* hatch — absent it, a RAW/NORMALIZED dataset is invisible to dashboards.
*/
export function assertReadableByBiSurface(
dataset: { dataset_id: string; tier: AnalyticsTier; raw_access_exception?: string },
surface: string,
): void {
if (BI_READABLE_TIERS.includes(dataset.tier)) return;
if (dataset.raw_access_exception && dataset.raw_access_exception.trim().length > 0) return;
throw new SiaxError(
"AUTHORIZATION_DENIED",
"0pt.analytics",
`BI surface "${surface}" may not read ${dataset.tier} dataset ${dataset.dataset_id} ` +
`without a recorded raw_access_exception (BI-P003)`,
);
}
/** A dataset may not declare a source tier ABOVE its own tier. */
export function assertTierLineageSane(datasetTier: AnalyticsTier, sourceTier: AnalyticsTier): void {
if (TIER_RANK[sourceTier] > TIER_RANK[datasetTier]) {
throw new SiaxError(
"INVALID_REQUEST",
"0pt.analytics",
`a ${datasetTier} dataset cannot be built from a ${sourceTier} dataset (BI-P003 tier order)`,
);
}
}
+563
View File
@@ -0,0 +1,563 @@
/**
* src/analytics/engine/clickhouse-engine.ts
* ==========================================
* BI-P009 — ClickHouse as the CANONICAL `SiaxAnalyticsEngine` provider.
*
* PRESERVATION (BI-P009's first sentence): `ClickHouseAnalytics`
* (src/analytics/clickhouse-analytics.ts) is untouched by this change — not
* one line. This engine COMPOSES it:
*
* * `health()` delegates to `ClickHouseAnalytics.health()`.
* * `executeMetric()` on the pre-existing `opt_metric_events` table
* delegates to `ClickHouseAnalytics.queryMetric()`, i.e. keeps the
* already-proven forced-tenant path exactly as it is.
* * everything else (the BI-P010 tables) goes through
* `src/analytics/engine/query-compiler.ts`, which forces the same
* predicate for the same reason.
*
* "Do not regress the existing forced tenant predicate" is enforced by
* tests/analytics/tenant-predicate-regression.test.ts, which fails if either
* compiler stops emitting `tenant_id = {tenant:String}`.
*
* BI-P012 governance is applied on EVERY statement, including the delegated
* ones — the settings ride on the HTTP request, and `readonly=1` means the
* query text cannot re-set them.
*
* LEGAL BOUNDARY (inherited): generic 0pt analytics only. Never the b0ts
* ClickHouse pipelines, never quarantined market data.
*/
import crypto from "node:crypto";
import { SiaxError, siaxErrorCodeForUpstreamStatus } from "../../platform/errors";
import { ClickHouseAnalytics } from "../clickhouse-analytics";
import {
compileMetricQuery,
compileProjectionQuery,
dimensionCatalogFrom,
FORCED_TENANT_PREDICATE,
type DimensionCatalog,
} from "./query-compiler";
import { analyticsSchemaDdl, type AnalyticsSchemaOptions } from "./schema";
import {
applyQuota,
clickhouseSettings,
ConcurrencyGate,
DEFAULT_BUDGETS,
planningSettings,
type ResourceBudget,
} from "./resource-governance";
import type {
AnalystQueryRequest,
AnalyticsIdentity,
EngineResult,
HealthReport,
MetricRequest,
ProjectionIngestRequest,
ProjectionRefreshResult,
QueryEstimate,
QueryExplanation,
SiaxAnalyticsEngine,
} from "./siax-analytics-engine";
export interface ClickHouseEngineOptions {
baseUrl: string;
/** "user:password" — from an env/secret ref, never a literal in code. */
basicAuth?: string;
database?: string;
timeoutMs?: number;
/** Dimension allow-list per tenant. Absent tenant = no dimensions allowed. */
dimensionCatalogs?: Record<string, Record<string, string>>;
schema?: Omit<AnalyticsSchemaOptions, "database">;
}
const EMPTY_CATALOG: DimensionCatalog = { resolve: () => undefined };
export class ClickHouseAnalyticsEngine implements SiaxAnalyticsEngine {
readonly provider = "clickhouse";
/** PRESERVED pre-existing implementation, composed not replaced. */
private readonly legacy: ClickHouseAnalytics;
private readonly baseUrl: string;
private readonly authHeader?: string;
private readonly database: string;
private readonly timeoutMs: number;
private readonly dimensionCatalogs: Record<string, Record<string, string>>;
private readonly schemaOpts: Omit<AnalyticsSchemaOptions, "database">;
private readonly gate: ConcurrencyGate;
/** query_id -> AbortController, so cancelQuery can stop the client side too. */
private readonly inFlight = new Map<string, AbortController>();
constructor(opts: ClickHouseEngineOptions) {
this.baseUrl = opts.baseUrl.replace(/\/+$/, "");
this.authHeader = opts.basicAuth ? `Basic ${Buffer.from(opts.basicAuth).toString("base64")}` : undefined;
this.database = opts.database ?? "opt_analytics";
this.timeoutMs = opts.timeoutMs ?? 30_000;
this.dimensionCatalogs = opts.dimensionCatalogs ?? {};
this.schemaOpts = opts.schema ?? {};
this.legacy = new ClickHouseAnalytics({
baseUrl: opts.baseUrl,
basicAuth: opts.basicAuth,
database: this.database,
timeoutMs: this.timeoutMs,
});
this.gate = new ConcurrencyGate((tenantId) => this.budgetForTenant(tenantId).maxConcurrentQueries);
}
/** Access to the preserved implementation, for callers that still use it. */
get legacyAnalytics(): ClickHouseAnalytics {
return this.legacy;
}
private budgetForTenant(_tenantId: string): ResourceBudget {
return DEFAULT_BUDGETS.INTERACTIVE;
}
private budgetFor(identity: AnalyticsIdentity): ResourceBudget {
const tier = identity.budget_tier ?? "INTERACTIVE";
return applyQuota(DEFAULT_BUDGETS[tier], identity.quota);
}
private catalogFor(identity: AnalyticsIdentity): DimensionCatalog {
const pairs = this.dimensionCatalogs[identity.tenant_id];
return pairs ? dimensionCatalogFrom(pairs) : EMPTY_CATALOG;
}
/** Fail closed: a missing/blank verified tenant is never a wildcard. */
private assertIdentity(identity: AnalyticsIdentity): void {
if (!identity?.tenant_id || !identity.tenant_id.trim()) {
throw new SiaxError(
"UNAUTHENTICATED",
"0pt.analytics",
"analytics requires a verified CL0UD identity carrying tenant_id",
);
}
if (!identity.principal_id || !identity.principal_id.trim()) {
throw new SiaxError("UNAUTHENTICATED", "0pt.analytics", "analytics requires a verified principal_id");
}
}
// ── schema ───────────────────────────────────────────────────────────────
/** Apply BI-P010 DDL (idempotent) plus the preserved legacy table. */
async ensureSchema(): Promise<void> {
await this.legacy.ensureSchema();
for (const stmt of analyticsSchemaDdl({ database: this.database, ...this.schemaOpts })) {
await this.ddl(stmt);
}
}
// ── BI-P009 capability 1: health ─────────────────────────────────────────
async health(): Promise<HealthReport> {
try {
const legacyOk = await this.legacy.health();
if (!legacyOk) return { healthy: false, provider: this.provider, detail: "legacy health probe returned false" };
const v = await this.raw("SELECT version() AS v FORMAT JSON", {}, {});
const version = (v as { data?: { v?: string }[] })?.data?.[0]?.v;
return { healthy: true, provider: this.provider, version };
} catch (err) {
return {
healthy: false,
provider: this.provider,
detail: err instanceof SiaxError ? err.code : "unreachable",
};
}
}
// ── BI-P009 capability 2: executeMetric ──────────────────────────────────
async executeMetric(identity: AnalyticsIdentity, request: MetricRequest): Promise<EngineResult> {
this.assertIdentity(identity);
const budget = this.budgetFor(identity);
const release = this.gate.acquire(identity.tenant_id);
const started = Date.now();
const queryId = this.newQueryId(identity);
try {
const compiled = compileMetricQuery(
{
tenant_id: identity.tenant_id, // FORCED from the verified identity
project_id: identity.project_id,
dataset_id: request.dataset_id,
metric: request.metric,
aggregate: request.aggregate,
group_by: request.group_by,
filters: request.filters,
from_time: request.from_time,
to_time: request.to_time,
limit: request.limit,
table: request.table,
},
this.database,
this.catalogFor(identity),
budget.maxResultRows,
);
const json = (await this.raw(compiled.sql, compiled.params, clickhouseSettings(budget), queryId)) as
| ClickHouseJsonResponse
| undefined;
const groupNames = request.group_by ?? [];
const rows = (json?.data ?? []).map((r) => {
const groups: Record<string, string> = {};
groupNames.forEach((name, i) => {
const v = (r as Record<string, unknown>)[`grp${i}`];
groups[name] = v === undefined || v === null ? "" : String(v);
});
return {
...(groupNames.length ? { groups } : {}),
value: Number((r as { value: number | string }).value),
};
});
const limit = Math.max(1, Math.min(Math.floor(request.limit ?? 1000), budget.maxResultRows));
return {
rows,
query_id: queryId,
truncated: rows.length >= limit,
stats: {
duration_ms: Date.now() - started,
rows_read: json?.statistics?.rows_read,
bytes_read: json?.statistics?.bytes_read,
result_rows: rows.length,
},
};
} finally {
release();
this.inFlight.delete(queryId);
}
}
/**
* Delegating path to the PRESERVED implementation for the pre-existing
* `opt_metric_events` table. Kept so no existing caller has to migrate and
* the already-proven forced-predicate path stays in service.
*/
async executeLegacyMetric(
identity: AnalyticsIdentity,
request: { metric: string; aggregate: "sum" | "avg" | "min" | "max" | "count"; dataset_id?: string; group_by?: string; from_time?: string; to_time?: string; limit?: number },
): Promise<EngineResult> {
this.assertIdentity(identity);
const started = Date.now();
const rows = await this.legacy.queryMetric({
tenant_id: identity.tenant_id, // FORCED from the verified identity
metric: request.metric,
aggregate: request.aggregate,
dataset_id: request.dataset_id,
group_by: request.group_by,
from_time: request.from_time,
to_time: request.to_time,
limit: request.limit,
});
return {
rows: rows.map((r) => ({
...(r.group !== undefined && request.group_by ? { groups: { [request.group_by]: r.group } } : {}),
value: r.value,
})),
query_id: this.newQueryId(identity),
truncated: false,
stats: { duration_ms: Date.now() - started, result_rows: rows.length },
};
}
// ── BI-P009 capability 3: executeQuery (analyst SQL) ─────────────────────
async executeQuery(identity: AnalyticsIdentity, _request: AnalystQueryRequest): Promise<EngineResult> {
this.assertIdentity(identity);
// Deliberate, documented refusal — see AnalystQueryRequest's doc comment.
// Opening a raw-SQL surface before BI-P017 (SQL dialect contract) and
// BI-P033 (analytics authorization) exist would create precisely the
// "raw SQL attempt" hole BI-P011's adversarial list enumerates.
throw new SiaxError(
"CAPABILITY_UNAVAILABLE",
"0pt.analytics",
"analyst SQL is not enabled on the ClickHouse provider until BI-P017/BI-P033 land",
);
}
// ── BI-P009 capability 4: estimateQuery ──────────────────────────────────
async estimateQuery(identity: AnalyticsIdentity, request: MetricRequest): Promise<QueryEstimate> {
this.assertIdentity(identity);
const budget = this.budgetFor(identity);
const compiled = compileMetricQuery(
{
tenant_id: identity.tenant_id,
project_id: identity.project_id,
dataset_id: request.dataset_id,
metric: request.metric,
aggregate: request.aggregate,
group_by: request.group_by,
filters: request.filters,
from_time: request.from_time,
to_time: request.to_time,
limit: request.limit,
table: request.table,
},
this.database,
this.catalogFor(identity),
budget.maxResultRows,
);
// `EXPLAIN ESTIMATE` reports the parts/rows/marks the planner will read
// AFTER primary-key pruning — which is why tenant_id leads every ORDER BY
// in schema.ts: the estimate for one tenant does not include everyone.
const estimateSql = compiled.sql.replace(/ FORMAT JSON$/, "");
let rowsToRead = 0;
let partsRead = 0;
try {
const res = (await this.raw(
`EXPLAIN ESTIMATE ${estimateSql} FORMAT JSON`,
compiled.params,
planningSettings(budget),
)) as { data?: { rows?: number | string; parts?: number | string }[] } | undefined;
for (const row of res?.data ?? []) {
rowsToRead += Number(row.rows ?? 0);
partsRead += Number(row.parts ?? 0);
}
} catch (err) {
// An estimate that cannot be produced must not read as "cheap".
throw err instanceof SiaxError
? err
: new SiaxError("DEPENDENCY_UNAVAILABLE", "clickhouse", "could not estimate query");
}
// Rough byte estimate: ClickHouse's ESTIMATE reports rows, not bytes.
// 128 B/row is a deliberately PESSIMISTIC constant for these tables
// (measured mean in the BI-P010 benchmark is lower), so the gate errs
// toward refusing rather than admitting an oversized query.
const bytesToRead = rowsToRead * 128;
const within = rowsToRead <= budget.maxRowsToRead && bytesToRead <= budget.maxBytesToRead;
return {
estimated_rows_to_read: rowsToRead,
estimated_bytes_to_read: bytesToRead,
within_budget: within,
budget,
...(within
? {}
: {
reason:
`estimate ${rowsToRead} rows / ${bytesToRead} bytes over budget ` +
`(${budget.maxRowsToRead} rows / ${budget.maxBytesToRead} bytes), parts=${partsRead}`,
}),
};
}
// ── BI-P009 capability 5: cancelQuery ────────────────────────────────────
/**
* Two-sided cancellation. The client-side abort stops us waiting; the
* server-side KILL QUERY stops ClickHouse burning CPU. Cancelling is
* tenant-scoped: the query id embeds the tenant and is checked, so tenant A
* cannot cancel tenant B's work.
*/
async cancelQuery(identity: AnalyticsIdentity, queryId: string): Promise<boolean> {
this.assertIdentity(identity);
if (!queryId.startsWith(this.queryIdPrefix(identity))) {
throw new SiaxError("AUTHORIZATION_DENIED", "0pt.analytics", "query does not belong to this tenant");
}
this.inFlight.get(queryId)?.abort();
this.inFlight.delete(queryId);
try {
// KILL QUERY is a mutation of server state, so it cannot ride the
// readonly=1 settings the SELECT path uses.
await this.raw(
`KILL QUERY WHERE query_id = {killId:String} SYNC FORMAT JSON`,
{ param_killId: queryId },
{ max_execution_time: "10" },
);
return true;
} catch {
return false;
}
}
// ── BI-P009 capability 6: explain ────────────────────────────────────────
async explain(identity: AnalyticsIdentity, request: MetricRequest): Promise<QueryExplanation> {
this.assertIdentity(identity);
const budget = this.budgetFor(identity);
const compiled = compileMetricQuery(
{
tenant_id: identity.tenant_id,
project_id: identity.project_id,
dataset_id: request.dataset_id,
metric: request.metric,
aggregate: request.aggregate,
group_by: request.group_by,
filters: request.filters,
from_time: request.from_time,
to_time: request.to_time,
limit: request.limit,
table: request.table,
},
this.database,
this.catalogFor(identity),
budget.maxResultRows,
);
const sql = compiled.sql.replace(/ FORMAT JSON$/, "");
const res = (await this.raw(
`EXPLAIN PLAN indexes = 1 ${sql} FORMAT TSVRaw`,
compiled.params,
planningSettings(budget),
)) as string | undefined;
return {
plan: typeof res === "string" ? res : JSON.stringify(res ?? ""),
forced_tenant_predicate: FORCED_TENANT_PREDICATE,
table: compiled.table,
};
}
// ── BI-P009 capability 7: ingestProjection ───────────────────────────────
async ingestProjection(
identity: AnalyticsIdentity,
request: ProjectionIngestRequest,
): Promise<{ ingested: number }> {
this.assertIdentity(identity);
if (request.rows.length === 0) return { ingested: 0 };
const now = new Date();
const version = Math.floor(now.getTime());
const body = request.rows
.map((r) => {
if (!r.entity_key || !r.entity_key.trim()) {
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "every projection row needs an entity_key");
}
return JSON.stringify({
// tenant/project come from the VERIFIED identity, never the row —
// a row-supplied tenant_id is the INF0 Wave-2 bug class.
tenant_id: identity.tenant_id,
project_id: identity.project_id ?? "",
dataset_id: request.dataset_id,
entity_key: r.entity_key,
as_of: (r.as_of ?? now.toISOString()).replace("T", " ").replace(/Z$/, ""),
version: r.version ?? version,
attributes: r.attributes ?? {},
measures: r.measures ?? {},
});
})
.join("\n");
await this.raw(
`INSERT INTO ${this.database}.opt_dataset_projection FORMAT JSONEachRow`,
{},
{},
undefined,
body,
);
return { ingested: request.rows.length };
}
// ── BI-P009 capability 8: refreshProjection ──────────────────────────────
/**
* ReplacingMergeTree deduplicates on a background merge, so a read straight
* after an ingest can still see superseded rows. `OPTIMIZE ... FINAL` forces
* the merge for THIS tenant's partition only — the table is
* `PARTITION BY (tenant_id)` precisely so one tenant's refresh does not
* rewrite another tenant's parts (BI-P012's "without degrading unrelated
* tenant" applies to maintenance too, not only to queries).
*/
async refreshProjection(identity: AnalyticsIdentity, dataset_id: string): Promise<ProjectionRefreshResult> {
this.assertIdentity(identity);
await this.raw(
`OPTIMIZE TABLE ${this.database}.opt_dataset_projection PARTITION {tenant:String} FINAL`,
{ param_tenant: identity.tenant_id },
{ max_execution_time: "120" },
);
const compiled = compileProjectionQuery(
{ tenant_id: identity.tenant_id, project_id: identity.project_id, dataset_id, limit: 1 },
this.database,
1,
);
const countSql =
`SELECT count() AS value FROM ${this.database}.opt_dataset_projection FINAL ` +
`WHERE ${FORCED_TENANT_PREDICATE} AND dataset_id = {dataset:String} FORMAT JSON`;
const res = (await this.raw(countSql, compiled.params, clickhouseSettings(this.budgetFor(identity)))) as
| { data?: { value: number | string }[] }
| undefined;
return {
dataset_id,
row_count: Number(res?.data?.[0]?.value ?? 0),
refreshed_at: new Date().toISOString(),
};
}
// ── transport ────────────────────────────────────────────────────────────
private queryIdPrefix(identity: AnalyticsIdentity): string {
return `opt_${crypto.createHash("sha256").update(identity.tenant_id).digest("hex").slice(0, 12)}_`;
}
private newQueryId(identity: AnalyticsIdentity): string {
return `${this.queryIdPrefix(identity)}${crypto.randomUUID().replace(/-/g, "")}`;
}
/** DDL path: no readonly, no result parsing beyond errors. */
private async ddl(sql: string): Promise<void> {
await this.raw(sql, {}, { max_execution_time: "60" });
}
private async raw(
sql: string,
params: Record<string, string>,
settings: Record<string, string>,
queryId?: string,
body?: string,
): Promise<unknown> {
const url = new URL(this.baseUrl);
url.searchParams.set("query", sql);
for (const [k, v] of Object.entries(settings)) url.searchParams.set(k, v);
for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
if (queryId) url.searchParams.set("query_id", queryId);
const headers: Record<string, string> = {};
if (this.authHeader) headers["Authorization"] = this.authHeader;
const controller = new AbortController();
if (queryId) this.inFlight.set(queryId, controller);
const timeout = setTimeout(() => controller.abort(), this.timeoutMs);
let res: Response;
try {
res = await fetch(url.toString(), {
method: "POST",
headers,
body: body ?? "",
signal: controller.signal,
});
} catch (cause) {
throw new SiaxError("DEPENDENCY_UNAVAILABLE", "clickhouse", "clickhouse unreachable", { cause });
} finally {
clearTimeout(timeout);
}
const text = await res.text();
if (!res.ok) {
// ClickHouse error text can echo the query; keep only the numeric code
// so a caller never receives another tenant's identifiers back.
const code = /Code: (\d+)/.exec(text)?.[1] ?? String(res.status);
throw new SiaxError(
siaxErrorCodeForUpstreamStatus(res.status),
"clickhouse",
`query failed (CH code ${code})`,
);
}
if (!text) return undefined;
let parsed: unknown;
try {
parsed = JSON.parse(text);
} catch {
return text;
}
// DEFENCE IN DEPTH against a partial result that looks complete.
// ClickHouse normally answers a breached limit with an HTTP error status
// (measured on 24.8: 408 for max_execution_time, 500 for
// max_rows_to_read), which `res.ok` above already catches. But when rows
// have already been flushed, the JSON body can carry an `exception` key
// under a 200 — a truncated answer indistinguishable from a whole one.
// Treat any body carrying an exception as a failure, whatever the status.
if (parsed && typeof parsed === "object" && "exception" in parsed) {
const ex = String((parsed as { exception?: unknown }).exception ?? "");
const code = /Code: (\d+)/.exec(ex)?.[1] ?? "unknown";
throw new SiaxError("PERMANENT_FAILURE", "clickhouse", `query failed mid-stream (CH code ${code})`);
}
return parsed;
}
}
interface ClickHouseJsonResponse {
data?: Record<string, unknown>[];
statistics?: { rows_read?: number; bytes_read?: number; elapsed?: number };
}
+304
View File
@@ -0,0 +1,304 @@
/**
* src/analytics/engine/query-compiler.ts
* =======================================
* BI-P011 — CLICKHOUSE TENANT ISOLATION, layer 2:
* "`0pt` query compiler forces tenant".
*
* The compiler is the ONLY thing in this repo that turns an analytics request
* into SQL text. Its contract:
*
* * `tenant_id` is not an argument the caller may omit — it comes from the
* VERIFIED CL0UD identity (layer 1) and is written into the WHERE clause
* by this function, unconditionally, as a bound parameter.
* * caller-supplied VALUES are never spliced into SQL text. They travel as
* ClickHouse HTTP `param_*` query parameters.
* * caller-supplied IDENTIFIERS (a dimension name, a group-by column) are
* never spliced either. They are RESOLVED against a declared allow-list
* of `DimensionDefinition`s and only the resolved, already-known physical
* name reaches the SQL — so "dimension identifier injection" has no
* surface, because caller text never becomes an identifier.
* * there is no raw-SQL entry point. `compileMetricQuery` cannot express a
* UNION, a subquery, or a second table, so a caller cannot reach a table
* the compiler did not choose.
*
* WHY A SEPARATE MODULE from clickhouse-analytics.ts: that file's compiler
* already forces the tenant predicate correctly and BI-P009 says PRESERVE IT.
* This module is the generalised compiler for the BI-P010 tables; it repeats
* the guarantee rather than replacing it, and
* tests/analytics/tenant-predicate-regression.test.ts asserts BOTH.
*/
import { SiaxError } from "../../platform/errors";
import type { AnalyticsTable } from "./schema";
/** A physical identifier that has been VALIDATED, not merely received. */
const IDENT_RE = /^[A-Za-z_][A-Za-z0-9_]{0,62}$/;
export function assertSafeIdentifier(value: string, what: string): string {
if (!IDENT_RE.test(value)) {
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", `invalid ${what} identifier`);
}
return value;
}
export const METRIC_AGGREGATES = ["sum", "avg", "min", "max", "count", "count_distinct"] as const;
export type MetricAggregate = (typeof METRIC_AGGREGATES)[number];
/**
* The dimensions a caller is allowed to name, resolved from
* `DimensionDefinition` rows. The compiler looks a caller's dimension name up
* HERE; a miss is a rejection, never a pass-through.
*/
export interface DimensionCatalog {
/** caller-facing dimension name -> physical dimension key in dim_names. */
resolve(name: string): string | undefined;
}
export function dimensionCatalogFrom(pairs: Record<string, string>): DimensionCatalog {
const map = new Map(Object.entries(pairs));
return {
resolve(name: string): string | undefined {
// Own-property lookup on a Map — no prototype-chain surprises
// ("constructor", "__proto__" resolve to undefined, as they must).
return map.get(name);
},
};
}
export interface MetricQueryRequest {
/** Supplied by the engine from the verified identity, NOT by the caller. */
tenant_id: string;
project_id?: string;
dataset_id?: string;
metric: string;
aggregate: MetricAggregate;
/** Caller-facing dimension names; resolved through the catalog. */
group_by?: string[];
/** Equality filters on dimensions: caller-facing name -> value. */
filters?: Record<string, string>;
from_time?: string;
to_time?: string;
limit?: number;
/** Which BI-P010 table to read. Chosen by the engine, not the caller. */
table?: AnalyticsTable;
}
export interface CompiledQuery {
sql: string;
/** ClickHouse HTTP `param_*` entries. Values NEVER appear in `sql`. */
params: Record<string, string>;
/** The predicate the compiler forced, for assertion in tests. */
forcedTenantPredicate: string;
table: AnalyticsTable;
}
/**
* The exact predicate text every compiled query must contain. Exported so a
* regression test can assert on the literal string — if someone deletes the
* forcing, the test fails on a missing substring rather than on a subtle
* behavioural difference that a mocked engine would happily hide.
*/
export const FORCED_TENANT_PREDICATE = "tenant_id = {tenant:String}";
const AGG_SQL: Record<MetricAggregate, string> = {
sum: "sum(value)",
avg: "avg(value)",
min: "min(value)",
max: "max(value)",
count: "count()",
count_distinct: "uniqExact(event_id)",
};
const MAX_GROUP_BY = 8;
/**
* Compile a closed-DSL metric request into tenant-forced SQL.
*
* @param req the request; `tenant_id` MUST already be the verified identity.
* @param db the analytics database name (engine-controlled, validated).
* @param catalog the tenant's declared dimensions.
* @param maxResultRows the governance ceiling (BI-P012); `limit` is clamped.
*/
export function compileMetricQuery(
req: MetricQueryRequest,
db: string,
catalog: DimensionCatalog,
maxResultRows: number,
): CompiledQuery {
if (!req.tenant_id || !req.tenant_id.trim()) {
// Fail closed. A blank tenant must never compile to "no predicate".
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "tenant_id is required on every analytics query");
}
assertSafeIdentifier(db, "database");
const table: AnalyticsTable = req.table ?? "opt_analytics_event";
assertSafeIdentifier(table, "table");
if (!METRIC_AGGREGATES.includes(req.aggregate)) {
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "unsupported aggregate");
}
const params: Record<string, string> = {
param_tenant: req.tenant_id,
param_metric: req.metric,
};
// ── the forced predicate. First condition, always, no branch. ─────────
const conditions: string[] = [FORCED_TENANT_PREDICATE, "metric = {metric:String}"];
if (req.project_id) {
conditions.push("project_id = {project:String}");
params.param_project = req.project_id;
}
if (req.dataset_id) {
conditions.push("dataset_id = {dataset:String}");
params.param_dataset = req.dataset_id;
}
if (req.from_time) {
conditions.push("event_time >= parseDateTime64BestEffort({fromTime:String})");
params.param_fromTime = req.from_time;
}
if (req.to_time) {
conditions.push("event_time < parseDateTime64BestEffort({toTime:String})");
params.param_toTime = req.to_time;
}
// ── filters: names resolved through the catalog, values parameterized ──
let filterIdx = 0;
for (const [name, value] of Object.entries(req.filters ?? {})) {
const physical = catalog.resolve(name);
if (physical === undefined) {
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", `unknown filter dimension "${name}"`);
}
// `physical` came from the catalog, but validate anyway: a corrupted or
// hostile catalog row must not become an injection vector either.
assertSafeIdentifier(physical, "dimension");
const nameKey = `f${filterIdx}n`;
const valKey = `f${filterIdx}v`;
conditions.push(
`dim_values[indexOf(dim_names, {${nameKey}:String})] = {${valKey}:String}`,
);
params[`param_${nameKey}`] = physical;
params[`param_${valKey}`] = value;
filterIdx += 1;
}
// ── group by: same resolution, plus a hard cardinality ceiling ─────────
const groupBy = req.group_by ?? [];
if (groupBy.length > MAX_GROUP_BY) {
throw new SiaxError(
"INVALID_REQUEST",
"0pt.analytics",
`at most ${MAX_GROUP_BY} group-by dimensions (got ${groupBy.length})`,
);
}
const selectParts: string[] = [];
const groupParts: string[] = [];
groupBy.forEach((name, i) => {
const physical = catalog.resolve(name);
if (physical === undefined) {
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", `unknown group_by dimension "${name}"`);
}
assertSafeIdentifier(physical, "dimension");
const key = `g${i}`;
// The ALIAS is compiler-generated (`grp0`, `grp1`), never caller text.
selectParts.push(`dim_values[indexOf(dim_names, {${key}:String})] AS grp${i}`);
groupParts.push(`grp${i}`);
params[`param_${key}`] = physical;
});
const limit = Math.max(1, Math.min(Math.floor(req.limit ?? 1000), maxResultRows));
const sql =
`SELECT ${selectParts.length ? selectParts.join(", ") + ", " : ""}${AGG_SQL[req.aggregate]} AS value ` +
`FROM ${db}.${table} ` +
`WHERE ${conditions.join(" AND ")} ` +
(groupParts.length ? `GROUP BY ${groupParts.join(", ")} ORDER BY value DESC ` : "") +
`LIMIT ${limit} FORMAT JSON`;
return { sql, params, forcedTenantPredicate: FORCED_TENANT_PREDICATE, table };
}
/**
* Compile a projection lookup (SERVING tier, BI-P010's opt_dataset_projection).
* Same forcing, different table and key.
*/
export function compileProjectionQuery(
req: { tenant_id: string; project_id?: string; dataset_id: string; entity_keys?: string[]; limit?: number },
db: string,
maxResultRows: number,
): CompiledQuery {
if (!req.tenant_id || !req.tenant_id.trim()) {
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "tenant_id is required on every analytics query");
}
assertSafeIdentifier(db, "database");
const params: Record<string, string> = {
param_tenant: req.tenant_id,
param_dataset: req.dataset_id,
};
const conditions = [FORCED_TENANT_PREDICATE, "dataset_id = {dataset:String}"];
if (req.project_id) {
conditions.push("project_id = {project:String}");
params.param_project = req.project_id;
}
if (req.entity_keys && req.entity_keys.length > 0) {
conditions.push("has({entityKeys:Array(String)}, entity_key)");
params.param_entityKeys = JSON.stringify(req.entity_keys);
}
const limit = Math.max(1, Math.min(Math.floor(req.limit ?? 1000), maxResultRows));
const sql =
`SELECT entity_key, as_of, version, attributes, measures ` +
`FROM ${db}.opt_dataset_projection ` +
`WHERE ${conditions.join(" AND ")} ` +
`ORDER BY entity_key LIMIT ${limit} FORMAT JSON`;
return { sql, params, forcedTenantPredicate: FORCED_TENANT_PREDICATE, table: "opt_dataset_projection" };
}
/**
* BI-P011 layer-3 bootstrap: the DDL for a RESTRICTED per-tenant ClickHouse
* user with a row policy pinning `tenant_id`.
*
* This is the layer that survives a compiler bug. The compiler forcing the
* predicate is layer 2; if a future refactor drops it, layer 3 still returns
* zero foreign rows because the SERVER filters them. The BI-P011 proof runs
* the adversarial suite twice — once through an unrestricted user (hole open)
* and once through this restricted user (hole closed) — because a claim of
* defence in depth that has never been tested with one layer removed is not
* evidence.
*
* `tenantId` is validated as an identifier-safe token AND embedded as a
* quoted literal; ClickHouse's CREATE ROW POLICY takes no bound parameters,
* so validation is the only protection available and it is applied strictly.
*/
export function restrictedTenantUserDdl(args: {
db: string;
tenantId: string;
userName: string;
password: string;
tables: readonly string[];
settingsProfile?: string;
}): string[] {
assertSafeIdentifier(args.db, "database");
assertSafeIdentifier(args.userName, "user");
// A tenant id is `t_<slug>` in this estate; enforce that shape strictly
// rather than trusting the caller.
if (!/^[A-Za-z0-9_-]{1,64}$/.test(args.tenantId)) {
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "tenant id is not policy-safe");
}
if (args.password.includes("'") || args.password.includes("\\")) {
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "password contains characters that cannot be embedded");
}
const profile = args.settingsProfile;
if (profile) assertSafeIdentifier(profile, "settings profile");
const stmts: string[] = [
`CREATE USER IF NOT EXISTS ${args.userName} IDENTIFIED WITH sha256_password BY '${args.password}'` +
(profile ? ` SETTINGS PROFILE ${profile}` : ` SETTINGS readonly = 1`),
];
for (const table of args.tables) {
assertSafeIdentifier(table, "table");
stmts.push(
`CREATE ROW POLICY IF NOT EXISTS ${args.userName}_${table}_rp ON ${args.db}.${table} ` +
`FOR SELECT USING tenant_id = '${args.tenantId}' TO ${args.userName}`,
);
stmts.push(`GRANT SELECT ON ${args.db}.${table} TO ${args.userName}`);
}
return stmts;
}
+222
View File
@@ -0,0 +1,222 @@
/**
* src/analytics/engine/resource-governance.ts
* ============================================
* BI-P012 — CLICKHOUSE RESOURCE GOVERNANCE.
*
* Every setting BI-P012 names, in one place, with a value:
*
* max_execution_time max_result_rows max_rows_to_read
* max_bytes_to_read max_memory_usage max_concurrent_queries
*
* Three layers, and the order matters:
*
* 1. SERVER-SIDE SETTINGS travel with every statement on the HTTP
* interface. ClickHouse enforces them; the client cannot opt out
* because the client is what writes them. `readonly=1` on top means a
* SELECT cannot re-set them mid-query (readonly=1 forbids SETTINGS
* changes as well as writes — that is the whole reason it is here and
* not merely `readonly=2`).
* 2. `max_concurrent_queries` is NOT a per-query setting in ClickHouse; it
* is a per-USER quota. `concurrencySlotSql` renders the CREATE
* SETTINGS PROFILE that carries it, applied to the tenant's restricted
* user (BI-P011 layer 3) so one tenant's fan-out cannot consume the
* whole server.
* 3. A local admission gate (`ConcurrencyGate`) refuses the N+1st
* in-flight query per tenant before it ever reaches the network. Two
* layers because the ClickHouse quota is per-user and one 0pt process
* can still queue work faster than the server rejects it.
*
* Budget tiers exist because "one limit for everyone" is how a small tenant
* subsidises a query bomb. `INTERACTIVE` is the dashboard default and is
* deliberately mean.
*/
import { SiaxError } from "../../platform/errors";
export interface ResourceBudget {
/** Seconds. ClickHouse `max_execution_time`. */
maxExecutionTimeSec: number;
/** Rows returned to the caller. ClickHouse `max_result_rows`. */
maxResultRows: number;
/** Rows SCANNED (not returned). ClickHouse `max_rows_to_read`. */
maxRowsToRead: number;
/** Bytes scanned. ClickHouse `max_bytes_to_read`. */
maxBytesToRead: number;
/** Bytes of RAM per query. ClickHouse `max_memory_usage`. */
maxMemoryUsageBytes: number;
/** In-flight queries per tenant. Enforced by quota + local gate. */
maxConcurrentQueries: number;
}
export const BUDGET_TIERS = ["INTERACTIVE", "ANALYTICAL", "BATCH"] as const;
export type BudgetTier = (typeof BUDGET_TIERS)[number];
/**
* Defaults. Chosen against the shape of the analytics tables in
* `src/analytics/engine/schema.ts`, not plucked from the air:
* INTERACTIVE must answer a dashboard panel inside a human's patience budget,
* so it caps the SCAN, which is what actually costs time on a MergeTree.
*/
export const DEFAULT_BUDGETS: Record<BudgetTier, ResourceBudget> = {
INTERACTIVE: {
maxExecutionTimeSec: 10,
maxResultRows: 10_000,
maxRowsToRead: 50_000_000,
maxBytesToRead: 2 * 1024 * 1024 * 1024,
maxMemoryUsageBytes: 2 * 1024 * 1024 * 1024,
maxConcurrentQueries: 8,
},
ANALYTICAL: {
maxExecutionTimeSec: 60,
maxResultRows: 500_000,
maxRowsToRead: 2_000_000_000,
maxBytesToRead: 32 * 1024 * 1024 * 1024,
maxMemoryUsageBytes: 8 * 1024 * 1024 * 1024,
maxConcurrentQueries: 4,
},
BATCH: {
maxExecutionTimeSec: 900,
maxResultRows: 10_000_000,
maxRowsToRead: 50_000_000_000,
maxBytesToRead: 512 * 1024 * 1024 * 1024,
maxMemoryUsageBytes: 24 * 1024 * 1024 * 1024,
maxConcurrentQueries: 2,
},
};
/**
* A CL0UD-issued quota may only ever TIGHTEN a tier, never loosen it. A
* control-plane bug that hands out a generous number must not be able to
* disarm the engine's own ceiling.
*/
export function applyQuota(base: ResourceBudget, quota: Partial<ResourceBudget> | undefined): ResourceBudget {
if (!quota) return { ...base };
const pick = (b: number, q: number | undefined): number =>
typeof q === "number" && Number.isFinite(q) && q > 0 ? Math.min(b, q) : b;
return {
maxExecutionTimeSec: pick(base.maxExecutionTimeSec, quota.maxExecutionTimeSec),
maxResultRows: pick(base.maxResultRows, quota.maxResultRows),
maxRowsToRead: pick(base.maxRowsToRead, quota.maxRowsToRead),
maxBytesToRead: pick(base.maxBytesToRead, quota.maxBytesToRead),
maxMemoryUsageBytes: pick(base.maxMemoryUsageBytes, quota.maxMemoryUsageBytes),
maxConcurrentQueries: pick(base.maxConcurrentQueries, quota.maxConcurrentQueries),
};
}
/**
* The settings that ride along on the HTTP request. `readonly: "1"` is what
* makes them un-overridable from inside the query text.
*
* `timeout_overflow_mode` / `result_overflow_mode` / `read_overflow_mode` are
* set to `throw` EXPLICITLY: ClickHouse's default for some of these is
* `break`, which silently returns a PARTIAL result. A partial result that
* looks complete is the analytics equivalent of a false green, so the engine
* refuses it.
*/
export function clickhouseSettings(budget: ResourceBudget): Record<string, string> {
return {
readonly: "1",
max_execution_time: String(budget.maxExecutionTimeSec),
timeout_overflow_mode: "throw",
max_result_rows: String(budget.maxResultRows),
result_overflow_mode: "throw",
max_rows_to_read: String(budget.maxRowsToRead),
max_bytes_to_read: String(budget.maxBytesToRead),
read_overflow_mode: "throw",
max_memory_usage: String(budget.maxMemoryUsageBytes),
};
}
/**
* Settings for a PLANNER call (`EXPLAIN ESTIMATE`, `EXPLAIN PLAN`).
*
* Measured, not assumed: running `EXPLAIN ESTIMATE` under the caller's own
* `max_rows_to_read` makes ClickHouse refuse the ESTIMATE itself with code 158
* whenever the estimate would be over budget — i.e. exactly the case the
* estimate exists to detect. An estimator that cannot describe an expensive
* query is useless, so the read ceilings are deliberately NOT applied here.
* The planner still runs `readonly = 1` under a short deadline and a memory
* cap, so this is a narrower envelope, never an unbounded one.
*/
export function planningSettings(budget: ResourceBudget): Record<string, string> {
return {
readonly: "1",
max_execution_time: String(Math.min(budget.maxExecutionTimeSec, 15)),
timeout_overflow_mode: "throw",
max_memory_usage: String(budget.maxMemoryUsageBytes),
};
}
/** Every setting name BI-P012 requires, for the DoD assertion. */
export const REQUIRED_GOVERNANCE_SETTINGS = [
"max_execution_time",
"max_result_rows",
"max_rows_to_read",
"max_bytes_to_read",
"max_memory_usage",
"max_concurrent_queries",
] as const;
/**
* `max_concurrent_queries` lives on a SETTINGS PROFILE, not on a statement.
* Rendered here so BI-P011's restricted-user bootstrap and BI-P012's budget
* come from one source.
*/
export function concurrencyProfileSql(profileName: string, budget: ResourceBudget): string[] {
return [
`CREATE SETTINGS PROFILE IF NOT EXISTS ${profileName} SETTINGS ` +
`max_execution_time = ${budget.maxExecutionTimeSec}, ` +
`max_result_rows = ${budget.maxResultRows}, ` +
`max_rows_to_read = ${budget.maxRowsToRead}, ` +
`max_bytes_to_read = ${budget.maxBytesToRead}, ` +
`max_memory_usage = ${budget.maxMemoryUsageBytes}, ` +
`readonly = 1`,
`CREATE QUOTA IF NOT EXISTS ${profileName}_quota ` +
`FOR INTERVAL 1 SECOND MAX queries = ${budget.maxConcurrentQueries * 20} ` +
`TO ALL EXCEPT default`,
];
}
/**
* Local admission control. One instance per process; keyed by tenant so a
* noisy tenant is throttled without touching anyone else's slots — which is
* exactly BI-P012's DoD wording ("without degrading unrelated tenant").
*/
export class ConcurrencyGate {
private readonly inFlight = new Map<string, number>();
constructor(private readonly limitFor: (tenantId: string) => number) {}
/** Returns a release function. Throws RATE_LIMITED when the tenant is full. */
acquire(tenantId: string): () => void {
const limit = this.limitFor(tenantId);
const current = this.inFlight.get(tenantId) ?? 0;
if (current >= limit) {
throw new SiaxError(
"RATE_LIMITED",
"0pt.analytics",
`tenant already has ${current} analytics queries in flight (limit ${limit})`,
);
}
this.inFlight.set(tenantId, current + 1);
let released = false;
return () => {
if (released) return;
released = true;
const n = (this.inFlight.get(tenantId) ?? 1) - 1;
if (n <= 0) this.inFlight.delete(tenantId);
else this.inFlight.set(tenantId, n);
};
}
/** Observability: in-flight count for a tenant (0 when idle). */
inFlightFor(tenantId: string): number {
return this.inFlight.get(tenantId) ?? 0;
}
/** Total in-flight across all tenants. */
totalInFlight(): number {
let n = 0;
for (const v of this.inFlight.values()) n += v;
return n;
}
}
+169
View File
@@ -0,0 +1,169 @@
/**
* src/analytics/engine/schema.ts
* ===============================
* BI-P010 — CLICKHOUSE PRODUCTION SCHEMA.
*
* BI-P010's explicit warning is "Avoid one giant unbounded table for every
* analytics use case." The existing `opt_metric_events` (owned by
* src/analytics/clickhouse-analytics.ts, PRESERVED untouched) is exactly one
* table for one job: low-cardinality capability metrics. It stays, and it is
* NOT extended into the catch-all.
*
* Four tables, four jobs, four retention profiles:
*
* opt_metric_events (PRE-EXISTING, not defined here) — capability
* metrics, Map(String,String) dimensions, 90d.
* opt_analytics_event — high-volume per-tenant event stream. Wide
* dimension arrays, SHORT TTL (raw events are not
* the serving tier; they age into the rollups).
* opt_metric_daily — AggregatingMergeTree rollup fed by a
* MATERIALIZED VIEW off opt_analytics_event.
* This is what a dashboard time-series reads;
* long TTL, tiny footprint.
* opt_dataset_projection — the SERVING tier (BI-P003) for a CURATED
* dataset: one row per (tenant, dataset, entity),
* ReplacingMergeTree so a refresh is idempotent.
*
* Why the split is load-bearing, not cosmetic:
* * different ORDER BY keys. A dashboard filtering (tenant, metric, day)
* and a projection looked up by (tenant, dataset, entity_key) cannot
* share one primary key without one of them degenerating to a full scan.
* * different TTLs. Raw events must expire fast; rollups must not.
* * different cardinality. `LowCardinality(String)` is a win on
* tenant/metric/dataset and a LOSS on a free-form entity key, so they
* cannot be the same column type in the same table.
*
* EVERY table leads its ORDER BY with `tenant_id`. That is not stylistic: it
* is what makes the forced tenant predicate (BI-P011) a primary-key range
* scan instead of a filter applied after reading everyone's rows, and it is
* what makes a per-tenant row policy cheap.
*
* LEGAL BOUNDARY (inherited verbatim from clickhouse-analytics.ts): this
* schema is for generic 0pt analytics. It MUST NEVER be pointed at the b0ts
* ClickHouse pipelines or quarantined market data.
*/
export interface AnalyticsSchemaOptions {
database: string;
/** Raw event retention. Deliberately short — rollups carry the history. */
eventRetentionDays?: number;
/** Rollup retention. */
rollupRetentionDays?: number;
/** Serving-projection retention. */
projectionRetentionDays?: number;
}
/** Physical tables this module owns (opt_metric_events is NOT one of them). */
export const ANALYTICS_TABLES = [
"opt_analytics_event",
"opt_metric_daily",
"opt_dataset_projection",
] as const;
export type AnalyticsTable = (typeof ANALYTICS_TABLES)[number];
/** The materialized view wiring opt_analytics_event -> opt_metric_daily. */
export const ANALYTICS_MATERIALIZED_VIEWS = ["opt_metric_daily_mv"] as const;
/**
* DDL, in dependency order. Every statement is CREATE ... IF NOT EXISTS so
* applying it to a live database is idempotent (the same contract
* `ClickHouseAnalytics.ensureSchema` already keeps).
*/
export function analyticsSchemaDdl(opts: AnalyticsSchemaOptions): string[] {
const db = opts.database;
const eventTtl = Math.floor(opts.eventRetentionDays ?? 30);
const rollupTtl = Math.floor(opts.rollupRetentionDays ?? 1095);
const projectionTtl = Math.floor(opts.projectionRetentionDays ?? 365);
return [
`CREATE DATABASE IF NOT EXISTS ${db}`,
// ── raw event stream ────────────────────────────────────────────────
// Dimensions as parallel arrays (not Map) so ClickHouse can prune on
// dimension VALUES via the bloom filter index; Map(String,String) forces
// reading the whole map. `event_id` makes deduplication possible on
// replay without a ReplacingMergeTree's merge latency.
`CREATE TABLE IF NOT EXISTS ${db}.opt_analytics_event (
tenant_id LowCardinality(String),
project_id LowCardinality(String),
dataset_id LowCardinality(String),
metric LowCardinality(String),
event_id String,
event_time DateTime64(3, 'UTC'),
value Float64,
dim_names Array(LowCardinality(String)),
dim_values Array(String),
ingested_at DateTime64(3, 'UTC') DEFAULT now64(3),
INDEX idx_dim_values dim_values TYPE bloom_filter(0.01) GRANULARITY 4
)
ENGINE = MergeTree()
PARTITION BY toYYYYMM(event_time)
ORDER BY (tenant_id, project_id, dataset_id, metric, event_time)
TTL toDateTime(event_time) + INTERVAL ${eventTtl} DAY
SETTINGS index_granularity = 8192`,
// ── daily rollup (serving tier for time series) ─────────────────────
`CREATE TABLE IF NOT EXISTS ${db}.opt_metric_daily (
tenant_id LowCardinality(String),
project_id LowCardinality(String),
dataset_id LowCardinality(String),
metric LowCardinality(String),
day Date,
sum_state AggregateFunction(sum, Float64),
count_state AggregateFunction(count, Float64),
min_state AggregateFunction(min, Float64),
max_state AggregateFunction(max, Float64)
)
ENGINE = AggregatingMergeTree()
PARTITION BY toYYYYMM(day)
ORDER BY (tenant_id, project_id, dataset_id, metric, day)
TTL day + INTERVAL ${rollupTtl} DAY`,
`CREATE MATERIALIZED VIEW IF NOT EXISTS ${db}.opt_metric_daily_mv
TO ${db}.opt_metric_daily AS
SELECT
tenant_id,
project_id,
dataset_id,
metric,
toDate(event_time) AS day,
sumState(value) AS sum_state,
countState(value) AS count_state,
minState(value) AS min_state,
maxState(value) AS max_state
FROM ${db}.opt_analytics_event
GROUP BY tenant_id, project_id, dataset_id, metric, day`,
// ── serving projection for a CURATED dataset ────────────────────────
// ReplacingMergeTree(version) so refreshProjection is idempotent: the
// same (tenant, dataset, entity_key) re-ingested at a higher version
// supersedes rather than duplicates.
`CREATE TABLE IF NOT EXISTS ${db}.opt_dataset_projection (
tenant_id LowCardinality(String),
project_id LowCardinality(String),
dataset_id LowCardinality(String),
entity_key String,
as_of DateTime64(3, 'UTC'),
version UInt64,
attributes Map(String, String),
measures Map(String, Float64),
INDEX idx_entity entity_key TYPE bloom_filter(0.01) GRANULARITY 4
)
ENGINE = ReplacingMergeTree(version)
PARTITION BY (tenant_id)
ORDER BY (tenant_id, project_id, dataset_id, entity_key)
TTL toDateTime(as_of) + INTERVAL ${projectionTtl} DAY`,
];
}
/**
* Statement that proves a table's ORDER BY leads with tenant_id — used by the
* BI-P011 test suite so "tenant-first primary key" is asserted against the
* live server, not against this file's own comments.
*/
export function sortingKeySql(db: string, table: string): string {
return (
`SELECT sorting_key FROM system.tables ` +
`WHERE database = '${db}' AND name = '${table}' FORMAT TSV`
);
}
@@ -0,0 +1,180 @@
/**
* src/analytics/engine/siax-analytics-engine.ts
* ==============================================
* BI-P009 — the provider-neutral analytics engine interface.
*
* Eight capabilities, named exactly as BI-P009 lists them:
*
* health executeMetric executeQuery estimateQuery
* explain cancelQuery ingestProjection refreshProjection
*
* ClickHouse is the CANONICAL provider (BI-P009); DuckDB (BI-P014) and Trino
* (BI-P015) implement the same interface later. Nothing above this interface
* may import a provider module directly — that is what makes "replaceable
* provider" true rather than aspirational.
*
* NORMATIVE OWNERSHIP (BI program §0): every method takes an
* `AnalyticsIdentity` that the CALLER cannot forge, because it is produced by
* CL0UD verification upstream, not by request-body parsing. Product repos
* (VehIQ) call an API in front of this interface; they never hold a provider
* credential and never construct an engine.
*/
import type { CompiledQuery, MetricAggregate } from "./query-compiler";
import type { AnalyticsTable } from "./schema";
import type { BudgetTier, ResourceBudget } from "./resource-governance";
/**
* The VERIFIED caller. `tenant_id` here is the one the engine forces into
* every predicate — it comes from CL0UD identity verification (BI-P011 layer
* 1), never from a request body. This estate has already been burned once by
* a tenant read out of a request body (INF0 Wave 2, cross-tenant RAG access),
* which is why the type is separate from the request rather than a field on
* it.
*/
export interface AnalyticsIdentity {
tenant_id: string;
project_id?: string;
/** CL0UD principal, for audit + per-principal quota. */
principal_id: string;
correlation_id?: string;
/** Budget tier granted by CL0UD entitlements (BI-P012). */
budget_tier?: BudgetTier;
/** CL0UD quota, which may only TIGHTEN the tier. */
quota?: Partial<ResourceBudget>;
}
export interface MetricRequest {
metric: string;
aggregate: MetricAggregate;
dataset_id?: string;
group_by?: string[];
filters?: Record<string, string>;
from_time?: string;
to_time?: string;
limit?: number;
table?: AnalyticsTable;
}
export interface AnalyticsRow {
/** Resolved group values, keyed by the CALLER-FACING dimension name. */
groups?: Record<string, string>;
value: number;
}
export interface EngineResult {
rows: AnalyticsRow[];
/** Engine-side id; the handle `cancelQuery` needs. */
query_id: string;
/** True when the governance ceiling cut the result short (BI-P012). */
truncated: boolean;
stats: QueryStats;
}
export interface QueryStats {
duration_ms: number;
rows_read?: number;
bytes_read?: number;
result_rows: number;
}
export interface QueryEstimate {
/** Rows the engine expects to SCAN. */
estimated_rows_to_read: number;
estimated_bytes_to_read: number;
/** False when the estimate exceeds the caller's budget (BI-P012). */
within_budget: boolean;
budget: ResourceBudget;
/** Why it was refused, when `within_budget` is false. */
reason?: string;
}
export interface QueryExplanation {
/** Provider's plan text. Never contains caller-supplied SQL fragments. */
plan: string;
/** The predicate the compiler forced — assertable, not narrated. */
forced_tenant_predicate: string;
table: string;
}
export interface ProjectionRow {
entity_key: string;
as_of?: string;
version?: number;
attributes?: Record<string, string>;
measures?: Record<string, number>;
}
export interface ProjectionIngestRequest {
dataset_id: string;
rows: ProjectionRow[];
}
export interface ProjectionRefreshResult {
dataset_id: string;
/** Rows visible after the refresh settled. */
row_count: number;
refreshed_at: string;
}
export interface HealthReport {
healthy: boolean;
provider: string;
/** Provider version when reachable. */
version?: string;
/** Short operator-facing reason. Never a raw engine body. */
detail?: string;
}
/**
* A raw, ANALYST-facing query (BI-P017 will govern its SQL dialect). It is a
* SEPARATE method from `executeMetric` on purpose: `executeMetric` has no
* raw-SQL surface at all, and keeping them apart means the closed DSL cannot
* silently grow one.
*
* The canonical ClickHouse provider in this pass DELIBERATELY refuses this
* method (`CAPABILITY_UNAVAILABLE`): opening an analyst SQL surface before
* BI-P017's dialect contract and BI-P033's authorization exist would be
* exactly the "raw SQL attempt" BI-P011's adversarial list names. Declaring
* it and refusing it is honest; implementing it early is not.
*/
export interface AnalystQueryRequest {
sql: string;
dataset_ids: string[];
limit?: number;
}
export interface SiaxAnalyticsEngine {
readonly provider: string;
health(): Promise<HealthReport>;
executeMetric(identity: AnalyticsIdentity, request: MetricRequest): Promise<EngineResult>;
executeQuery(identity: AnalyticsIdentity, request: AnalystQueryRequest): Promise<EngineResult>;
estimateQuery(identity: AnalyticsIdentity, request: MetricRequest): Promise<QueryEstimate>;
cancelQuery(identity: AnalyticsIdentity, queryId: string): Promise<boolean>;
explain(identity: AnalyticsIdentity, request: MetricRequest): Promise<QueryExplanation>;
ingestProjection(identity: AnalyticsIdentity, request: ProjectionIngestRequest): Promise<{ ingested: number }>;
refreshProjection(identity: AnalyticsIdentity, dataset_id: string): Promise<ProjectionRefreshResult>;
}
/** The eight capability names, for the provider contract test. */
export const SIAX_ANALYTICS_ENGINE_CAPABILITIES = [
"health",
"executeMetric",
"executeQuery",
"estimateQuery",
"cancelQuery",
"explain",
"ingestProjection",
"refreshProjection",
] as const;
export type SiaxAnalyticsEngineCapability = (typeof SIAX_ANALYTICS_ENGINE_CAPABILITIES)[number];
/** Re-exported so provider modules do not reach across into the compiler. */
export type { CompiledQuery };
@@ -0,0 +1,356 @@
/**
* src/analytics/repositories/analytics-domain-repository.ts
* ==========================================================
* BI-P002 persistence for the analytics domain model (sql/008).
*
* Two rules the type system alone cannot keep, so they live here:
*
* 1. EVERY read and write is tenant-scoped. There is no method that takes a
* dataset_id without also taking the tenant it must belong to, and every
* WHERE carries `tenant_id = $tenant`. A caller cannot "forget" the
* tenant because there is no signature that lets them.
* 2. VERSION ROWS ARE APPEND-ONLY from this layer too. `createDatasetVersion`
* /`createMetricVersion` compute the next version number inside the same
* statement (`SELECT coalesce(max(version),0)+1`) so two concurrent
* minters collide on the UNIQUE(dataset_id, version) constraint instead
* of silently overwriting — and the database triggers from sql/008 catch
* anything that still tries to UPDATE.
*
* Uses postgres.js through `RepositorySql`, the same connection type the other
* P0 repositories take (src/repositories/db.ts).
*/
import crypto from "node:crypto";
import { SiaxError } from "../../platform/errors";
import type { RepositorySql } from "../../repositories/db";
import type {
AnalyticsDataset,
DatasetVersion,
DimensionDefinition,
MetricDefinition,
MetricVersion,
PiiClassification,
} from "../domain/entities";
import type { AnalyticsTier } from "../domain/tiers";
/** The verified tenant scope every call must present. */
export interface TenantScope {
tenant_id: string;
project_id: string;
}
function assertScope(scope: TenantScope): void {
if (!scope?.tenant_id?.trim() || !scope?.project_id?.trim()) {
throw new SiaxError(
"UNAUTHENTICATED",
"0pt.analytics",
"analytics persistence requires a verified tenant_id + project_id",
);
}
}
function iso(v: string | Date | null | undefined): string | undefined {
if (v === null || v === undefined) return undefined;
return v instanceof Date ? v.toISOString() : v;
}
interface DatasetRow {
dataset_id: string;
tenant_id: string;
project_id: string;
name: string;
tier: string;
schema_contract_id: string;
schema_contract_version: string;
owner_id: string;
source_ref: string;
freshness_slo_seconds: number;
quality_rule_ids: string[];
retention_days: number;
classification: string;
status: string;
raw_access_exception: string | null;
description: string | null;
created_at: string | Date;
updated_at: string | Date | null;
created_by: string | null;
}
function toDataset(r: DatasetRow): AnalyticsDataset {
return {
dataset_id: r.dataset_id,
tenant_id: r.tenant_id,
project_id: r.project_id,
name: r.name,
tier: r.tier as AnalyticsTier,
schema_contract_id: r.schema_contract_id,
schema_contract_version: r.schema_contract_version,
owner_id: r.owner_id,
source_ref: r.source_ref,
freshness_slo_seconds: Number(r.freshness_slo_seconds),
quality_rule_ids: r.quality_rule_ids ?? [],
retention_days: Number(r.retention_days),
classification: r.classification as PiiClassification,
status: r.status as AnalyticsDataset["status"],
...(r.raw_access_exception ? { raw_access_exception: r.raw_access_exception } : {}),
...(r.description ? { description: r.description } : {}),
created_at: iso(r.created_at)!,
...(r.updated_at ? { updated_at: iso(r.updated_at) } : {}),
...(r.created_by ? { created_by: r.created_by } : {}),
};
}
export class AnalyticsDomainRepository {
constructor(private readonly sql: RepositorySql) {}
// ── AnalyticsDataset ─────────────────────────────────────────────────────
async createDataset(scope: TenantScope, input: Omit<AnalyticsDataset, "tenant_id" | "project_id" | "created_at" | "updated_at">): Promise<AnalyticsDataset> {
assertScope(scope);
const rows = await this.sql<DatasetRow[]>`
INSERT INTO opt_analytics_dataset
(dataset_id, tenant_id, project_id, name, tier, schema_contract_id, schema_contract_version,
owner_id, source_ref, freshness_slo_seconds, quality_rule_ids, retention_days,
classification, status, raw_access_exception, description, created_by)
VALUES
(${input.dataset_id}, ${scope.tenant_id}, ${scope.project_id}, ${input.name}, ${input.tier},
${input.schema_contract_id}, ${input.schema_contract_version}, ${input.owner_id}, ${input.source_ref},
${input.freshness_slo_seconds}, ${this.sql.array(input.quality_rule_ids ?? [])}, ${input.retention_days},
${input.classification}, ${input.status}, ${input.raw_access_exception ?? null},
${input.description ?? null}, ${input.created_by ?? null})
RETURNING *`;
return toDataset(rows[0]);
}
/** Tenant-scoped read. A foreign dataset_id returns undefined, never a row. */
async getDataset(scope: TenantScope, datasetId: string): Promise<AnalyticsDataset | undefined> {
assertScope(scope);
const rows = await this.sql<DatasetRow[]>`
SELECT * FROM opt_analytics_dataset
WHERE dataset_id = ${datasetId}
AND tenant_id = ${scope.tenant_id}
AND project_id = ${scope.project_id}
LIMIT 1`;
return rows[0] ? toDataset(rows[0]) : undefined;
}
async listDatasets(scope: TenantScope, filter?: { tier?: AnalyticsTier; status?: string }): Promise<AnalyticsDataset[]> {
assertScope(scope);
const rows = await this.sql<DatasetRow[]>`
SELECT * FROM opt_analytics_dataset
WHERE tenant_id = ${scope.tenant_id}
AND project_id = ${scope.project_id}
${filter?.tier ? this.sql`AND tier = ${filter.tier}` : this.sql``}
${filter?.status ? this.sql`AND status = ${filter.status}` : this.sql``}
ORDER BY created_at DESC`;
return rows.map(toDataset);
}
// ── DatasetVersion (append-only) ─────────────────────────────────────────
/**
* Mint the next version. The version number is computed inside the INSERT,
* so two racing callers cannot both read "3" and both write it — the loser
* hits UNIQUE(dataset_id, version) and gets a CONFLICT, which is the honest
* outcome.
*/
async createDatasetVersion(
scope: TenantScope,
input: { dataset_id: string; contract_hash: string; schema_contract_version: string; record_count?: number; created_by?: string },
): Promise<DatasetVersion> {
assertScope(scope);
const owner = await this.getDataset(scope, input.dataset_id);
if (!owner) {
throw new SiaxError("NOT_FOUND", "0pt.analytics", "dataset not found in this tenant");
}
const id = `dsv_${crypto.randomUUID().replace(/-/g, "")}`;
try {
const rows = await this.sql<
{
dataset_version_id: string;
dataset_id: string;
tenant_id: string;
project_id: string;
version: number;
contract_hash: string;
schema_contract_version: string;
record_count: string | number | null;
created_at: string | Date;
created_by: string | null;
}[]
>`
INSERT INTO opt_analytics_dataset_version
(dataset_version_id, dataset_id, tenant_id, project_id, version,
contract_hash, schema_contract_version, record_count, created_by)
SELECT ${id}, ${input.dataset_id}, ${scope.tenant_id}, ${scope.project_id},
coalesce(max(version), 0) + 1,
${input.contract_hash}, ${input.schema_contract_version},
${input.record_count ?? null}, ${input.created_by ?? null}
FROM opt_analytics_dataset_version
WHERE dataset_id = ${input.dataset_id} AND tenant_id = ${scope.tenant_id}
RETURNING *`;
const r = rows[0];
return {
dataset_version_id: r.dataset_version_id,
dataset_id: r.dataset_id,
tenant_id: r.tenant_id,
project_id: r.project_id,
version: Number(r.version),
contract_hash: r.contract_hash,
schema_contract_version: r.schema_contract_version,
...(r.record_count !== null ? { record_count: Number(r.record_count) } : {}),
created_at: iso(r.created_at)!,
...(r.created_by ? { created_by: r.created_by } : {}),
};
} catch (err) {
if (String(err).includes("duplicate key")) {
throw new SiaxError("CONFLICT", "0pt.analytics", "a concurrent writer minted this dataset version");
}
throw err;
}
}
async listDatasetVersions(scope: TenantScope, datasetId: string): Promise<DatasetVersion[]> {
assertScope(scope);
const rows = await this.sql<
{ dataset_version_id: string; dataset_id: string; tenant_id: string; project_id: string; version: number; contract_hash: string; schema_contract_version: string; record_count: string | number | null; created_at: string | Date; created_by: string | null }[]
>`
SELECT * FROM opt_analytics_dataset_version
WHERE dataset_id = ${datasetId}
AND tenant_id = ${scope.tenant_id}
AND project_id = ${scope.project_id}
ORDER BY version ASC`;
return rows.map((r) => ({
dataset_version_id: r.dataset_version_id,
dataset_id: r.dataset_id,
tenant_id: r.tenant_id,
project_id: r.project_id,
version: Number(r.version),
contract_hash: r.contract_hash,
schema_contract_version: r.schema_contract_version,
...(r.record_count !== null ? { record_count: Number(r.record_count) } : {}),
created_at: iso(r.created_at)!,
...(r.created_by ? { created_by: r.created_by } : {}),
}));
}
// ── MetricDefinition / MetricVersion ─────────────────────────────────────
async createMetric(
scope: TenantScope,
input: Omit<MetricDefinition, "tenant_id" | "project_id" | "created_at" | "updated_at" | "current_version">,
): Promise<MetricDefinition> {
assertScope(scope);
const rows = await this.sql<
{ metric_id: string; tenant_id: string; project_id: string; name: string; dataset_id: string; aggregation: string; measure_field: string; denominator_field: string | null; allowed_dimension_ids: string[]; current_version: number; owner_id: string; description: string | null; created_at: string | Date; updated_at: string | Date | null }[]
>`
INSERT INTO opt_analytics_metric_definition
(metric_id, tenant_id, project_id, name, dataset_id, aggregation, measure_field,
denominator_field, allowed_dimension_ids, current_version, owner_id, description)
VALUES
(${input.metric_id}, ${scope.tenant_id}, ${scope.project_id}, ${input.name}, ${input.dataset_id},
${input.aggregation}, ${input.measure_field}, ${input.denominator_field ?? null},
${this.sql.array(input.allowed_dimension_ids ?? [])}, 1, ${input.owner_id}, ${input.description ?? null})
RETURNING *`;
const r = rows[0];
return {
metric_id: r.metric_id,
tenant_id: r.tenant_id,
project_id: r.project_id,
name: r.name,
dataset_id: r.dataset_id,
aggregation: r.aggregation as MetricDefinition["aggregation"],
measure_field: r.measure_field,
...(r.denominator_field ? { denominator_field: r.denominator_field } : {}),
allowed_dimension_ids: r.allowed_dimension_ids ?? [],
current_version: Number(r.current_version),
owner_id: r.owner_id,
...(r.description ? { description: r.description } : {}),
created_at: iso(r.created_at)!,
...(r.updated_at ? { updated_at: iso(r.updated_at) } : {}),
};
}
/** Append-only. `change_reason` is required for version > 1 (DB CHECK). */
async createMetricVersion(
scope: TenantScope,
input: { metric_id: string; definition_json: string; change_reason?: string; created_by?: string },
): Promise<MetricVersion> {
assertScope(scope);
const hash = crypto.createHash("sha256").update(input.definition_json).digest("hex");
const id = `mv_${crypto.randomUUID().replace(/-/g, "")}`;
const rows = await this.sql<
{ metric_version_id: string; metric_id: string; tenant_id: string; project_id: string; version: number; definition_json: string; definition_hash: string; change_reason: string | null; created_at: string | Date; created_by: string | null }[]
>`
INSERT INTO opt_analytics_metric_version
(metric_version_id, metric_id, tenant_id, project_id, version, definition_json,
definition_hash, change_reason, created_by)
SELECT ${id}, ${input.metric_id}, ${scope.tenant_id}, ${scope.project_id},
coalesce(max(version), 0) + 1, ${input.definition_json}, ${hash},
${input.change_reason ?? null}, ${input.created_by ?? null}
FROM opt_analytics_metric_version
WHERE metric_id = ${input.metric_id} AND tenant_id = ${scope.tenant_id}
RETURNING *`;
const r = rows[0];
return {
metric_version_id: r.metric_version_id,
metric_id: r.metric_id,
tenant_id: r.tenant_id,
project_id: r.project_id,
version: Number(r.version),
definition_json: r.definition_json,
definition_hash: r.definition_hash,
...(r.change_reason ? { change_reason: r.change_reason } : {}),
created_at: iso(r.created_at)!,
...(r.created_by ? { created_by: r.created_by } : {}),
};
}
// ── DimensionDefinition (the query compiler's allow-list) ────────────────
async createDimension(
scope: TenantScope,
input: Omit<DimensionDefinition, "tenant_id" | "project_id" | "created_at">,
): Promise<DimensionDefinition> {
assertScope(scope);
const rows = await this.sql<
{ dimension_id: string; tenant_id: string; project_id: string; name: string; dataset_id: string; field_name: string; field_type: string; classification: string; description: string | null; created_at: string | Date }[]
>`
INSERT INTO opt_analytics_dimension_definition
(dimension_id, tenant_id, project_id, name, dataset_id, field_name, field_type, classification, description)
VALUES
(${input.dimension_id}, ${scope.tenant_id}, ${scope.project_id}, ${input.name}, ${input.dataset_id},
${input.field_name}, ${input.field_type}, ${input.classification}, ${input.description ?? null})
RETURNING *`;
const r = rows[0];
return {
dimension_id: r.dimension_id,
tenant_id: r.tenant_id,
project_id: r.project_id,
name: r.name,
dataset_id: r.dataset_id,
field_name: r.field_name,
field_type: r.field_type as DimensionDefinition["field_type"],
classification: r.classification as PiiClassification,
...(r.description ? { description: r.description } : {}),
created_at: iso(r.created_at)!,
};
}
/**
* The dimension allow-list the compiler resolves through, for one tenant.
* Returns caller-facing name -> physical field name. A tenant with no
* declared dimensions gets an EMPTY map, which means every dimension
* reference is refused — fail closed, not "allow everything".
*/
async dimensionCatalog(scope: TenantScope, datasetId?: string): Promise<Record<string, string>> {
assertScope(scope);
const rows = await this.sql<{ name: string; field_name: string }[]>`
SELECT name, field_name FROM opt_analytics_dimension_definition
WHERE tenant_id = ${scope.tenant_id}
AND project_id = ${scope.project_id}
${datasetId ? this.sql`AND dataset_id = ${datasetId}` : this.sql``}`;
const out: Record<string, string> = Object.create(null);
for (const r of rows) out[r.name] = r.field_name;
return out;
}
}