feat(analytics): BI-P002..BI-P012 analytics core — domain model, tiers, contracts, Arrow, ClickHouse engine, isolation, governance
CI (SIAX Cloud) / security (push) Successful in 41s
Security Scan (SIAX Cloud) / security (push) Successful in 40s
CI (SIAX Cloud) / integration (push) Successful in 52s
CI (SIAX Cloud) / quality (push) Successful in 1m5s
CI (SIAX Cloud) / security (pull_request) Successful in 31s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 29s
CI (SIAX Cloud) / integration (pull_request) Successful in 45s
CI (SIAX Cloud) / quality (pull_request) Successful in 53s
CI (SIAX Cloud) / security (push) Successful in 41s
Security Scan (SIAX Cloud) / security (push) Successful in 40s
CI (SIAX Cloud) / integration (push) Successful in 52s
CI (SIAX Cloud) / quality (push) Successful in 1m5s
CI (SIAX Cloud) / security (pull_request) Successful in 31s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 29s
CI (SIAX Cloud) / integration (pull_request) Successful in 45s
CI (SIAX Cloud) / quality (pull_request) Successful in 53s
Implements the eight BI program prompts that make 0pt the canonical SIAX
analytics capability. Every file is NEW: not one existing file is modified,
and src/analytics/clickhouse-analytics.ts is byte-identical to main.
BI-P002 — the nineteen canonical entities, verbatim, with tenant_id +
project_id on every tenant-owned record. sql/008 gives each one a table.
The four version entities BI-P002 names as immutable are enforced by BEFORE
UPDATE/DELETE triggers, not by a TypeScript `readonly` — proven by direct SQL
against a live Postgres.
BI-P003 — RAW/NORMALIZED/CURATED/SERVING with the read rule enforced:
assertReadableByBiSurface refuses a RAW dataset to a dashboard unless a
recorded exception exists. All eight required declarations are NOT NULL.
BI-P004 — DatasetContract and a fail-closed compatibility gate. Fifteen
breaking-change classes are each tested twice: refused under a minor bump,
accepted under a major one.
BI-P005 — Arrow IPC alongside JSON, with a hand-written FlatBuffers writer so
no npm dependency (or CVE surface) is added. Conformance comes from three
independent implementations, not from self-agreement: pyarrow 25.0.1 and
DuckDB 1.5.5 read identical values from both ClickHouse's Arrow and ours, and
ClickHouse's own Arrow C++ ingests our 20,000-row batch with exact aggregates.
BI-P009 — SiaxAnalyticsEngine with all eight capabilities. The existing
ClickHouseAnalytics is COMPOSED, never replaced. executeQuery (analyst SQL) is
declared and deliberately refused with CAPABILITY_UNAVAILABLE until BI-P017 and
BI-P033 land, rather than opening the raw-SQL hole BI-P011 enumerates.
BI-P010 — four purpose-built tables, not one giant one, and the difference is
measured against the counterfactual: 48.8ms/40,960 rows vs 174.4ms/5,400,000
rows for the same question; 5/660 granules vs 660/1149.
BI-P011 — layered isolation, each layer proven with the layer above removed.
Without the forced predicate the same statement returns 50,050 (both tenants);
with it, t_alpha gets 50. Without a row policy an unrestricted user sees 2
tenants; the restricted user running the IDENTICAL statement sees 1.
BI-P012 — every named limit, fired individually and identified by ClickHouse
error code, plus the half usually skipped: an unrelated tenant is timed while
a real CPU bomb is in flight and must still return the CORRECT answer.
Three defects found by writing tests that can fail:
* `CHECK (array_length(purposes,1) >= 1)` can NEVER fail — array_length is
NULL for '{}' and a NULL CHECK passes. Fixed with coalesce(); the test
that caught it is pinned.
* EXPLAIN ESTIMATE run under the caller's own max_rows_to_read fails with
code 158 exactly when the estimate would be over budget. Planner calls now
use a narrower envelope.
* result_overflow_mode=break returns HTTP 200 with 65,409 of 100,000 rows
and no indication it is short. Every SELECT ships `throw`.
Verified rather than assumed: server7's ClickHouse holds `opt_scraped` (51
tables, 407,793,471 rows, `persons`/`cz_owners` included) — the b0ts plane this
module is prohibited from touching — has no `opt_analytics` database, zero
tenant_id columns and zero row policies. Nothing there was modified; every
proof ran against disposable containers.
Evidence: evidence/bi-p002-p012/{test-transcript.txt,bi-p010-benchmark.json,
bi-p005-arrow-roundtrip.json,server7-clickhouse-probe.md}
Docs: docs/analytics/BI_ANALYTICS_CORE.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,237 @@
|
||||
/**
|
||||
* src/analytics/arrow/flatbuffer-builder.ts
|
||||
* ==========================================
|
||||
* A minimal FlatBuffers writer, sufficient for the Arrow IPC metadata
|
||||
* messages this repo emits (BI-P005).
|
||||
*
|
||||
* WHY HAND-ROLLED rather than `apache-arrow` from npm: adding a runtime
|
||||
* dependency to 0pt's blocking-typechecked capability layer means a lockfile
|
||||
* change, a new supply-chain surface, and a new CVE surface for every
|
||||
* consumer of this repo — for a format whose METADATA side is a few hundred
|
||||
* bytes of well-specified binary and whose DATA side is plain columnar memory
|
||||
* either way.
|
||||
*
|
||||
* The correctness bar is NOT "this looks right". The BI-P005 evidence script
|
||||
* (scripts/bi-p005-arrow-roundtrip.mjs) decodes this writer's output with
|
||||
* pyarrow AND DuckDB — two independent implementations of the specification —
|
||||
* so a mistake here surfaces as a failed decode by somebody else's parser,
|
||||
* not as a green test of my own assumptions.
|
||||
*
|
||||
* Layout facts implemented (FlatBuffers binary format, little-endian):
|
||||
* * the buffer is built BACK TO FRONT; `offset` counts bytes from the end;
|
||||
* * a uoffset stored at a field site holds `site_offset + 4 - target_offset`
|
||||
* so a reader resolves it as `position + value`;
|
||||
* * a table begins with an soffset `table_start - vtable_start`;
|
||||
* * a vtable is `[vtable_len:i16][table_len:i16][field_offsets:i16...]`,
|
||||
* with 0 meaning "field absent, use the default".
|
||||
*/
|
||||
|
||||
export type Offset = number;
|
||||
|
||||
const SIZEOF_INT = 4;
|
||||
|
||||
export class FlatBufferBuilder {
|
||||
private buf: Uint8Array;
|
||||
private view: DataView;
|
||||
/** Bytes written so far, counted from the END of `buf`. */
|
||||
private used = 0;
|
||||
private minAlign = 1;
|
||||
private vtable: number[] = [];
|
||||
private objectStart = 0;
|
||||
private inObject = false;
|
||||
/** Offsets of previously written vtables, for deduplication. */
|
||||
private vtables: number[] = [];
|
||||
|
||||
constructor(initial = 1024) {
|
||||
this.buf = new Uint8Array(initial);
|
||||
this.view = new DataView(this.buf.buffer);
|
||||
}
|
||||
|
||||
/** Bytes written so far (FlatBuffers "offset"). */
|
||||
get offset(): number {
|
||||
return this.used;
|
||||
}
|
||||
|
||||
private ensure(extra: number): void {
|
||||
if (this.used + extra <= this.buf.length) return;
|
||||
let size = this.buf.length;
|
||||
while (size < this.used + extra) size *= 2;
|
||||
const next = new Uint8Array(size);
|
||||
next.set(this.buf.subarray(this.buf.length - this.used), size - this.used);
|
||||
this.buf = next;
|
||||
this.view = new DataView(this.buf.buffer);
|
||||
}
|
||||
|
||||
private pad(n: number): void {
|
||||
if (n <= 0) return;
|
||||
this.ensure(n);
|
||||
for (let i = 0; i < n; i++) {
|
||||
this.used += 1;
|
||||
this.buf[this.buf.length - this.used] = 0;
|
||||
}
|
||||
}
|
||||
|
||||
/** FlatBuffers `prep`: pad so that `additional` more bytes land aligned. */
|
||||
private prep(size: number, additional = 0): void {
|
||||
if (size > this.minAlign) this.minAlign = size;
|
||||
const alignSize = (~(this.used + additional) + 1) & (size - 1);
|
||||
this.pad(alignSize);
|
||||
}
|
||||
|
||||
writeInt8(v: number): void {
|
||||
this.ensure(1);
|
||||
this.used += 1;
|
||||
this.view.setInt8(this.buf.length - this.used, v);
|
||||
}
|
||||
|
||||
writeInt16(v: number): void {
|
||||
this.ensure(2);
|
||||
this.used += 2;
|
||||
this.view.setInt16(this.buf.length - this.used, v, true);
|
||||
}
|
||||
|
||||
writeInt32(v: number): void {
|
||||
this.ensure(4);
|
||||
this.used += 4;
|
||||
this.view.setInt32(this.buf.length - this.used, v, true);
|
||||
}
|
||||
|
||||
writeInt64(v: bigint): void {
|
||||
this.ensure(8);
|
||||
this.used += 8;
|
||||
this.view.setBigInt64(this.buf.length - this.used, v, true);
|
||||
}
|
||||
|
||||
writeBytes(bytes: Uint8Array): void {
|
||||
this.ensure(bytes.length);
|
||||
this.used += bytes.length;
|
||||
this.buf.set(bytes, this.buf.length - this.used);
|
||||
}
|
||||
|
||||
addInt8(slot: number, value: number, def: number): void {
|
||||
if (value === def) return;
|
||||
this.prep(1);
|
||||
this.writeInt8(value);
|
||||
this.slot(slot);
|
||||
}
|
||||
|
||||
addInt16(slot: number, value: number, def: number): void {
|
||||
if (value === def) return;
|
||||
this.prep(2);
|
||||
this.writeInt16(value);
|
||||
this.slot(slot);
|
||||
}
|
||||
|
||||
addInt32(slot: number, value: number, def: number): void {
|
||||
if (value === def) return;
|
||||
this.prep(4);
|
||||
this.writeInt32(value);
|
||||
this.slot(slot);
|
||||
}
|
||||
|
||||
addInt64(slot: number, value: bigint, def: bigint): void {
|
||||
if (value === def) return;
|
||||
this.prep(8);
|
||||
this.writeInt64(value);
|
||||
this.slot(slot);
|
||||
}
|
||||
|
||||
addOffset(slot: number, value: Offset, def: number): void {
|
||||
if (value === def) return;
|
||||
this.prep(SIZEOF_INT);
|
||||
this.writeInt32(this.used + SIZEOF_INT - value);
|
||||
this.slot(slot);
|
||||
}
|
||||
|
||||
private slot(slotNum: number): void {
|
||||
this.vtable[slotNum] = this.used;
|
||||
}
|
||||
|
||||
startObject(numFields: number): void {
|
||||
if (this.inObject) throw new Error("nested startObject");
|
||||
this.vtable = new Array<number>(numFields).fill(0);
|
||||
this.objectStart = this.used;
|
||||
this.inObject = true;
|
||||
}
|
||||
|
||||
endObject(): Offset {
|
||||
if (!this.inObject) throw new Error("endObject outside an object");
|
||||
this.prep(SIZEOF_INT);
|
||||
this.writeInt32(0); // soffset placeholder, patched below
|
||||
const objectOffset = this.used;
|
||||
|
||||
let trimmed = this.vtable.length;
|
||||
while (trimmed > 0 && this.vtable[trimmed - 1] === 0) trimmed -= 1;
|
||||
for (let i = trimmed - 1; i >= 0; i--) {
|
||||
this.writeInt16(this.vtable[i] !== 0 ? objectOffset - this.vtable[i] : 0);
|
||||
}
|
||||
this.writeInt16(objectOffset - this.objectStart); // table byte length
|
||||
const vtLen = (trimmed + 2) * 2;
|
||||
this.writeInt16(vtLen); // vtable byte length — lands at the vtable start
|
||||
|
||||
const vtStart = this.used;
|
||||
const vtAbs = this.buf.length - vtStart;
|
||||
let existing = 0;
|
||||
for (const candidate of this.vtables) {
|
||||
const candAbs = this.buf.length - candidate;
|
||||
if (this.view.getInt16(candAbs, true) !== vtLen) continue;
|
||||
let same = true;
|
||||
for (let i = 2; i < vtLen; i += 2) {
|
||||
if (this.view.getInt16(vtAbs + i, true) !== this.view.getInt16(candAbs + i, true)) {
|
||||
same = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (same) {
|
||||
existing = candidate;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
const objAbs = this.buf.length - objectOffset;
|
||||
if (existing !== 0) {
|
||||
this.used -= vtLen; // discard the duplicate vtable just written
|
||||
this.view.setInt32(objAbs, existing - objectOffset, true);
|
||||
} else {
|
||||
this.vtables.push(vtStart);
|
||||
this.view.setInt32(objAbs, this.used - objectOffset, true);
|
||||
}
|
||||
this.inObject = false;
|
||||
return objectOffset;
|
||||
}
|
||||
|
||||
createString(s: string): Offset {
|
||||
const bytes = new TextEncoder().encode(s);
|
||||
this.writeInt8(0); // null terminator (readers rely on it)
|
||||
this.startVector(1, bytes.length, 1);
|
||||
this.writeBytes(bytes);
|
||||
return this.endVector(bytes.length);
|
||||
}
|
||||
|
||||
startVector(elemSize: number, numElems: number, alignment: number): void {
|
||||
this.prep(SIZEOF_INT, elemSize * numElems);
|
||||
this.prep(alignment, elemSize * numElems);
|
||||
}
|
||||
|
||||
endVector(numElems: number): Offset {
|
||||
this.writeInt32(numElems);
|
||||
return this.used;
|
||||
}
|
||||
|
||||
/** Vector of table offsets. Elements are written in reverse build order. */
|
||||
createOffsetVector(items: Offset[]): Offset {
|
||||
this.startVector(SIZEOF_INT, items.length, SIZEOF_INT);
|
||||
for (let i = items.length - 1; i >= 0; i--) {
|
||||
this.writeInt32(this.used + SIZEOF_INT - items[i]);
|
||||
}
|
||||
return this.endVector(items.length);
|
||||
}
|
||||
|
||||
/** Finish the buffer with `root` as the root table. Returns the bytes. */
|
||||
finish(root: Offset): Uint8Array {
|
||||
this.prep(this.minAlign, SIZEOF_INT);
|
||||
this.prep(SIZEOF_INT);
|
||||
this.writeInt32(this.used + SIZEOF_INT - root);
|
||||
return this.buf.slice(this.buf.length - this.used);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,547 @@
|
||||
/**
|
||||
* src/analytics/arrow/ipc.ts
|
||||
* ===========================
|
||||
* BI-P005 — APACHE ARROW INTERCHANGE.
|
||||
*
|
||||
* Canonical columnar result encoding ALONGSIDE JSON (JSON is not removed —
|
||||
* BI-P005 says "alongside"). Two paths, and knowing which is which matters:
|
||||
*
|
||||
* PASS-THROUGH (preferred for ClickHouse results): ClickHouse emits real
|
||||
* Arrow IPC itself via `FORMAT ArrowStream`. Those bytes were produced by
|
||||
* the Arrow C++ implementation; 0pt streams them out unchanged, so there is
|
||||
* no encoder of ours in the path at all. `clickhouseArrowStreamFormat()`
|
||||
* and `arrowStreamSettings()` are that path.
|
||||
*
|
||||
* ENCODED (for results 0pt itself assembles — projections, JSON-sourced
|
||||
* rows, exports): `encodeArrowStream()` writes the IPC stream here.
|
||||
*
|
||||
* Both paths are decodable by `decodeArrowStream()`, which exists so 0pt can
|
||||
* count rows and read the schema of an Arrow payload WITHOUT shelling out to
|
||||
* another runtime — needed to enforce BI-P012's `max_result_rows` on the
|
||||
* pass-through path, where the row count is not otherwise visible.
|
||||
*
|
||||
* "Supported types" (the DoD's qualifier) is `ARROW_TYPE_SUPPORT` below: the
|
||||
* exact ClickHouse -> Arrow -> DuckDB/Python mapping, including where
|
||||
* semantic loss WOULD occur, stated rather than hidden.
|
||||
*
|
||||
* IPC stream layout (Arrow format 1.5, metadata V5):
|
||||
* for each message: 0xFFFFFFFF, int32 metadata length, FlatBuffer Message
|
||||
* padded so the body starts 8-byte aligned, then the body buffers.
|
||||
* Terminated by 0xFFFFFFFF 0x00000000.
|
||||
*/
|
||||
import { FlatBufferBuilder, type Offset } from "./flatbuffer-builder";
|
||||
import { SiaxError } from "../../platform/errors";
|
||||
|
||||
// ── FlatBuffer enum values from org.apache.arrow.flatbuf ────────────────────
|
||||
|
||||
const METADATA_VERSION_V5 = 4;
|
||||
const MESSAGE_HEADER_SCHEMA = 1;
|
||||
const MESSAGE_HEADER_RECORD_BATCH = 3;
|
||||
const ENDIANNESS_LITTLE = 0;
|
||||
|
||||
const TYPE_INT = 2;
|
||||
const TYPE_FLOATING_POINT = 3;
|
||||
const TYPE_UTF8 = 5;
|
||||
const TYPE_BOOL = 6;
|
||||
const TYPE_TIMESTAMP = 10;
|
||||
|
||||
const PRECISION_DOUBLE = 2;
|
||||
|
||||
// BigInt LITERALS (`0n`) require target >= ES2020; this repo targets ES6, so
|
||||
// every 64-bit constant is built with BigInt(...) instead. Behaviour is
|
||||
// identical — this is a compiler-target accommodation, not a semantic choice.
|
||||
const BIG_ZERO = BigInt(0);
|
||||
const INT64_MAX = BigInt("9223372036854775807");
|
||||
const INT64_MIN = BigInt("-9223372036854775808");
|
||||
const TIME_UNIT_MILLISECOND = 1;
|
||||
|
||||
// ── public column model ────────────────────────────────────────────────────
|
||||
|
||||
export type ArrowColumnType = "int64" | "float64" | "utf8" | "bool" | "timestamp_ms";
|
||||
|
||||
export interface ArrowColumn {
|
||||
name: string;
|
||||
type: ArrowColumnType;
|
||||
/** `null` marks a null slot; the validity bitmap is derived from it. */
|
||||
values: (number | bigint | string | boolean | Date | null)[];
|
||||
}
|
||||
|
||||
/**
|
||||
* The type table BI-P005's "supported types" qualifier refers to. Every row
|
||||
* has been exercised end to end by scripts/bi-p005-arrow-roundtrip.mjs.
|
||||
*/
|
||||
export const ARROW_TYPE_SUPPORT: {
|
||||
clickhouse: string;
|
||||
arrow: ArrowColumnType;
|
||||
duckdb: string;
|
||||
python: string;
|
||||
note: string;
|
||||
}[] = [
|
||||
{
|
||||
clickhouse: "Int64 / UInt32 / Int32",
|
||||
arrow: "int64",
|
||||
duckdb: "BIGINT",
|
||||
python: "int",
|
||||
note: "exact; UInt64 above 2^63 is NOT supported and is rejected rather than silently wrapped",
|
||||
},
|
||||
{
|
||||
clickhouse: "Float64",
|
||||
arrow: "float64",
|
||||
duckdb: "DOUBLE",
|
||||
python: "float",
|
||||
note: "exact IEEE-754 round trip",
|
||||
},
|
||||
{
|
||||
clickhouse: "String / LowCardinality(String)",
|
||||
arrow: "utf8",
|
||||
duckdb: "VARCHAR",
|
||||
python: "str",
|
||||
note: "LowCardinality is decoded to plain utf8 — dictionary encoding is lost, values are not",
|
||||
},
|
||||
{
|
||||
clickhouse: "Bool / UInt8(0|1)",
|
||||
arrow: "bool",
|
||||
duckdb: "BOOLEAN",
|
||||
python: "bool",
|
||||
note: "bit-packed one bit per value; ClickHouse UInt8 other than 0/1 is NOT a bool and must be sent as int64",
|
||||
},
|
||||
{
|
||||
clickhouse: "DateTime64(3,'UTC')",
|
||||
arrow: "timestamp_ms",
|
||||
duckdb: "TIMESTAMP",
|
||||
python: "datetime",
|
||||
note: "millisecond precision; DateTime64(6/9) loses sub-ms and is therefore NOT in the supported set",
|
||||
},
|
||||
];
|
||||
|
||||
// ── ClickHouse pass-through ────────────────────────────────────────────────
|
||||
|
||||
/** ClickHouse output format producing an Arrow IPC *stream*. */
|
||||
export function clickhouseArrowStreamFormat(): string {
|
||||
return "ArrowStream";
|
||||
}
|
||||
|
||||
/**
|
||||
* Settings for the pass-through path. `output_format_arrow_string_as_string`
|
||||
* matters: without it ClickHouse emits String columns as Arrow *Binary*,
|
||||
* which pyarrow and DuckDB both read as bytes rather than text — semantic
|
||||
* loss that would slip through a test that only compares row counts.
|
||||
*/
|
||||
export function arrowStreamSettings(): Record<string, string> {
|
||||
return {
|
||||
output_format_arrow_string_as_string: "1",
|
||||
output_format_arrow_low_cardinality_as_dictionary: "0",
|
||||
};
|
||||
}
|
||||
|
||||
// ── encoder ────────────────────────────────────────────────────────────────
|
||||
|
||||
function alignTo8(n: number): number {
|
||||
return (n + 7) & ~7;
|
||||
}
|
||||
|
||||
interface EncodedBuffer {
|
||||
bytes: Uint8Array;
|
||||
/** Logical (unpadded) length recorded in the RecordBatch metadata. */
|
||||
length: number;
|
||||
}
|
||||
|
||||
function validityBuffer(values: readonly unknown[]): { buf: EncodedBuffer; nullCount: number } {
|
||||
let nullCount = 0;
|
||||
const byteLen = Math.ceil(values.length / 8);
|
||||
const bytes = new Uint8Array(alignTo8(byteLen));
|
||||
for (let i = 0; i < values.length; i++) {
|
||||
if (values[i] === null || values[i] === undefined) {
|
||||
nullCount += 1;
|
||||
} else {
|
||||
bytes[i >> 3] |= 1 << (i & 7);
|
||||
}
|
||||
}
|
||||
return { buf: { bytes, length: byteLen }, nullCount };
|
||||
}
|
||||
|
||||
function encodeColumn(col: ArrowColumn): { buffers: EncodedBuffer[]; nullCount: number } {
|
||||
const { buf: validity, nullCount } = validityBuffer(col.values);
|
||||
const n = col.values.length;
|
||||
|
||||
switch (col.type) {
|
||||
case "int64": {
|
||||
const data = new Uint8Array(alignTo8(n * 8));
|
||||
const dv = new DataView(data.buffer);
|
||||
col.values.forEach((v, i) => {
|
||||
if (v === null || v === undefined) return;
|
||||
const b = typeof v === "bigint" ? v : BigInt(Math.trunc(Number(v)));
|
||||
if (b > INT64_MAX || b < INT64_MIN) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", `value out of int64 range in column "${col.name}"`);
|
||||
}
|
||||
dv.setBigInt64(i * 8, b, true);
|
||||
});
|
||||
return { buffers: [validity, { bytes: data, length: n * 8 }], nullCount };
|
||||
}
|
||||
case "float64": {
|
||||
const data = new Uint8Array(alignTo8(n * 8));
|
||||
const dv = new DataView(data.buffer);
|
||||
col.values.forEach((v, i) => {
|
||||
if (v === null || v === undefined) return;
|
||||
dv.setFloat64(i * 8, Number(v), true);
|
||||
});
|
||||
return { buffers: [validity, { bytes: data, length: n * 8 }], nullCount };
|
||||
}
|
||||
case "timestamp_ms": {
|
||||
const data = new Uint8Array(alignTo8(n * 8));
|
||||
const dv = new DataView(data.buffer);
|
||||
col.values.forEach((v, i) => {
|
||||
if (v === null || v === undefined) return;
|
||||
const ms = v instanceof Date ? v.getTime() : typeof v === "bigint" ? Number(v) : Number(v);
|
||||
dv.setBigInt64(i * 8, BigInt(Math.trunc(ms)), true);
|
||||
});
|
||||
return { buffers: [validity, { bytes: data, length: n * 8 }], nullCount };
|
||||
}
|
||||
case "bool": {
|
||||
const byteLen = Math.ceil(n / 8);
|
||||
const data = new Uint8Array(alignTo8(byteLen));
|
||||
col.values.forEach((v, i) => {
|
||||
if (v === null || v === undefined) return;
|
||||
if (v) data[i >> 3] |= 1 << (i & 7);
|
||||
});
|
||||
return { buffers: [validity, { bytes: data, length: byteLen }], nullCount };
|
||||
}
|
||||
case "utf8": {
|
||||
const encoded = col.values.map((v) =>
|
||||
v === null || v === undefined ? new Uint8Array(0) : new TextEncoder().encode(String(v)),
|
||||
);
|
||||
const total = encoded.reduce((a, b) => a + b.length, 0);
|
||||
const offsets = new Uint8Array(alignTo8((n + 1) * 4));
|
||||
const odv = new DataView(offsets.buffer);
|
||||
const data = new Uint8Array(alignTo8(total));
|
||||
let pos = 0;
|
||||
odv.setInt32(0, 0, true);
|
||||
encoded.forEach((bytes, i) => {
|
||||
data.set(bytes, pos);
|
||||
pos += bytes.length;
|
||||
odv.setInt32((i + 1) * 4, pos, true);
|
||||
});
|
||||
return {
|
||||
buffers: [validity, { bytes: offsets, length: (n + 1) * 4 }, { bytes: data, length: total }],
|
||||
nullCount,
|
||||
};
|
||||
}
|
||||
default: {
|
||||
const never: never = col.type;
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", `unsupported arrow type ${String(never)}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function buildTypeTable(b: FlatBufferBuilder, type: ArrowColumnType): { typeType: number; offset: Offset } {
|
||||
switch (type) {
|
||||
case "int64": {
|
||||
b.startObject(2);
|
||||
b.addInt32(0, 64, 0); // bitWidth
|
||||
b.addInt8(1, 1, 0); // is_signed
|
||||
return { typeType: TYPE_INT, offset: b.endObject() };
|
||||
}
|
||||
case "float64": {
|
||||
b.startObject(1);
|
||||
b.addInt16(0, PRECISION_DOUBLE, 0); // precision = DOUBLE
|
||||
return { typeType: TYPE_FLOATING_POINT, offset: b.endObject() };
|
||||
}
|
||||
case "utf8": {
|
||||
b.startObject(0);
|
||||
return { typeType: TYPE_UTF8, offset: b.endObject() };
|
||||
}
|
||||
case "bool": {
|
||||
b.startObject(0);
|
||||
return { typeType: TYPE_BOOL, offset: b.endObject() };
|
||||
}
|
||||
case "timestamp_ms": {
|
||||
// Build the timezone string BEFORE startObject — FlatBuffers forbids
|
||||
// creating a nested object while a table is open.
|
||||
const tz = b.createString("UTC");
|
||||
b.startObject(2);
|
||||
b.addInt16(0, TIME_UNIT_MILLISECOND, 0);
|
||||
b.addOffset(1, tz, 0);
|
||||
return { typeType: TYPE_TIMESTAMP, offset: b.endObject() };
|
||||
}
|
||||
default: {
|
||||
const never: never = type;
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", `unsupported arrow type ${String(never)}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function buildSchemaMessage(columns: readonly ArrowColumn[]): Uint8Array {
|
||||
const b = new FlatBufferBuilder(2048);
|
||||
|
||||
const fieldOffsets: Offset[] = columns.map((col) => {
|
||||
const nameOffset = b.createString(col.name);
|
||||
const { typeType, offset: typeOffset } = buildTypeTable(b, col.type);
|
||||
const children = b.createOffsetVector([]);
|
||||
b.startObject(7);
|
||||
b.addOffset(0, nameOffset, 0);
|
||||
b.addInt8(1, 1, 0); // nullable = true
|
||||
b.addInt8(2, typeType, 0);
|
||||
b.addOffset(3, typeOffset, 0);
|
||||
b.addOffset(5, children, 0);
|
||||
return b.endObject();
|
||||
});
|
||||
|
||||
const fieldsVector = b.createOffsetVector(fieldOffsets);
|
||||
b.startObject(4);
|
||||
b.addInt16(0, ENDIANNESS_LITTLE, 0);
|
||||
b.addOffset(1, fieldsVector, 0);
|
||||
const schema = b.endObject();
|
||||
|
||||
b.startObject(5);
|
||||
b.addInt16(0, METADATA_VERSION_V5, 0);
|
||||
b.addInt8(1, MESSAGE_HEADER_SCHEMA, 0);
|
||||
b.addOffset(2, schema, 0);
|
||||
b.addInt64(3, BIG_ZERO, BIG_ZERO); // bodyLength
|
||||
const message = b.endObject();
|
||||
|
||||
return b.finish(message);
|
||||
}
|
||||
|
||||
function buildRecordBatchMessage(
|
||||
rowCount: number,
|
||||
nodes: { length: number; nullCount: number }[],
|
||||
buffers: { offset: number; length: number }[],
|
||||
bodyLength: number,
|
||||
): Uint8Array {
|
||||
const b = new FlatBufferBuilder(2048);
|
||||
|
||||
// FieldNode[] — inline structs, written in reverse.
|
||||
b.startVector(16, nodes.length, 8);
|
||||
for (let i = nodes.length - 1; i >= 0; i--) {
|
||||
b.writeInt64(BigInt(nodes[i].nullCount));
|
||||
b.writeInt64(BigInt(nodes[i].length));
|
||||
}
|
||||
const nodesVector = b.endVector(nodes.length);
|
||||
|
||||
// Buffer[] — inline structs, written in reverse.
|
||||
b.startVector(16, buffers.length, 8);
|
||||
for (let i = buffers.length - 1; i >= 0; i--) {
|
||||
b.writeInt64(BigInt(buffers[i].length));
|
||||
b.writeInt64(BigInt(buffers[i].offset));
|
||||
}
|
||||
const buffersVector = b.endVector(buffers.length);
|
||||
|
||||
b.startObject(5);
|
||||
b.addInt64(0, BigInt(rowCount), BIG_ZERO);
|
||||
b.addOffset(1, nodesVector, 0);
|
||||
b.addOffset(2, buffersVector, 0);
|
||||
const recordBatch = b.endObject();
|
||||
|
||||
b.startObject(5);
|
||||
b.addInt16(0, METADATA_VERSION_V5, 0);
|
||||
b.addInt8(1, MESSAGE_HEADER_RECORD_BATCH, 0);
|
||||
b.addOffset(2, recordBatch, 0);
|
||||
b.addInt64(3, BigInt(bodyLength), BIG_ZERO);
|
||||
const message = b.endObject();
|
||||
|
||||
return b.finish(message);
|
||||
}
|
||||
|
||||
function framedMessage(metadata: Uint8Array, body: Uint8Array): Uint8Array {
|
||||
// continuation(4) + length(4) + metadata, padded so the body is 8-aligned.
|
||||
const prefix = 8;
|
||||
const padded = alignTo8(prefix + metadata.length) - prefix;
|
||||
const out = new Uint8Array(prefix + padded + body.length);
|
||||
const dv = new DataView(out.buffer);
|
||||
dv.setUint32(0, 0xffffffff, true);
|
||||
dv.setInt32(4, padded, true);
|
||||
out.set(metadata, 8);
|
||||
out.set(body, prefix + padded);
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Encode columns as a complete Arrow IPC *stream* (schema message, one record
|
||||
* batch, end-of-stream marker).
|
||||
*/
|
||||
export function encodeArrowStream(columns: readonly ArrowColumn[]): Uint8Array {
|
||||
if (columns.length === 0) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "an Arrow stream needs at least one column");
|
||||
}
|
||||
const rowCount = columns[0].values.length;
|
||||
for (const c of columns) {
|
||||
if (c.values.length !== rowCount) {
|
||||
throw new SiaxError(
|
||||
"INVALID_REQUEST",
|
||||
"0pt.analytics",
|
||||
`column "${c.name}" has ${c.values.length} values, expected ${rowCount}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const schemaMsg = buildSchemaMessage(columns);
|
||||
|
||||
const nodes: { length: number; nullCount: number }[] = [];
|
||||
const bufferMeta: { offset: number; length: number }[] = [];
|
||||
const bodyParts: Uint8Array[] = [];
|
||||
let bodyOffset = 0;
|
||||
for (const col of columns) {
|
||||
const { buffers, nullCount } = encodeColumn(col);
|
||||
nodes.push({ length: rowCount, nullCount });
|
||||
for (const buf of buffers) {
|
||||
bufferMeta.push({ offset: bodyOffset, length: buf.length });
|
||||
bodyParts.push(buf.bytes);
|
||||
bodyOffset += buf.bytes.length; // already 8-aligned by encodeColumn
|
||||
}
|
||||
}
|
||||
const body = new Uint8Array(bodyOffset);
|
||||
let pos = 0;
|
||||
for (const part of bodyParts) {
|
||||
body.set(part, pos);
|
||||
pos += part.length;
|
||||
}
|
||||
|
||||
const batchMsg = buildRecordBatchMessage(rowCount, nodes, bufferMeta, body.length);
|
||||
|
||||
const schemaFrame = framedMessage(schemaMsg, new Uint8Array(0));
|
||||
const batchFrame = framedMessage(batchMsg, body);
|
||||
const eos = new Uint8Array(8);
|
||||
new DataView(eos.buffer).setUint32(0, 0xffffffff, true); // length 0 = EOS
|
||||
|
||||
const out = new Uint8Array(schemaFrame.length + batchFrame.length + eos.length);
|
||||
out.set(schemaFrame, 0);
|
||||
out.set(batchFrame, schemaFrame.length);
|
||||
out.set(eos, schemaFrame.length + batchFrame.length);
|
||||
return out;
|
||||
}
|
||||
|
||||
// ── decoder (structural: schema names/types + row counts) ──────────────────
|
||||
|
||||
export interface ArrowStreamInfo {
|
||||
fields: { name: string; typeType: number }[];
|
||||
/** Total rows across every record batch in the stream. */
|
||||
rowCount: number;
|
||||
batchCount: number;
|
||||
}
|
||||
|
||||
/** Minimal FlatBuffers table reader — enough for Message/Schema/RecordBatch. */
|
||||
class FbTable {
|
||||
constructor(
|
||||
readonly dv: DataView,
|
||||
readonly pos: number,
|
||||
) {}
|
||||
|
||||
fieldPos(slot: number): number {
|
||||
const vtable = this.pos - this.dv.getInt32(this.pos, true);
|
||||
const vtLen = this.dv.getInt16(vtable, true);
|
||||
const idx = 4 + slot * 2;
|
||||
if (idx >= vtLen) return 0;
|
||||
const rel = this.dv.getInt16(vtable + idx, true);
|
||||
return rel === 0 ? 0 : this.pos + rel;
|
||||
}
|
||||
|
||||
readInt8(slot: number, def = 0): number {
|
||||
const p = this.fieldPos(slot);
|
||||
return p === 0 ? def : this.dv.getInt8(p);
|
||||
}
|
||||
|
||||
readInt16(slot: number, def = 0): number {
|
||||
const p = this.fieldPos(slot);
|
||||
return p === 0 ? def : this.dv.getInt16(p, true);
|
||||
}
|
||||
|
||||
readInt64(slot: number, def: bigint = BIG_ZERO): bigint {
|
||||
const p = this.fieldPos(slot);
|
||||
return p === 0 ? def : this.dv.getBigInt64(p, true);
|
||||
}
|
||||
|
||||
/** Follow a uoffset to a nested table. */
|
||||
readTable(slot: number): FbTable | undefined {
|
||||
const p = this.fieldPos(slot);
|
||||
if (p === 0) return undefined;
|
||||
return new FbTable(this.dv, p + this.dv.getInt32(p, true));
|
||||
}
|
||||
|
||||
vectorLength(slot: number): number {
|
||||
const p = this.fieldPos(slot);
|
||||
if (p === 0) return 0;
|
||||
const vec = p + this.dv.getInt32(p, true);
|
||||
return this.dv.getInt32(vec, true);
|
||||
}
|
||||
|
||||
vectorElementTable(slot: number, index: number): FbTable {
|
||||
const p = this.fieldPos(slot);
|
||||
const vec = p + this.dv.getInt32(p, true);
|
||||
const elem = vec + 4 + index * 4;
|
||||
return new FbTable(this.dv, elem + this.dv.getInt32(elem, true));
|
||||
}
|
||||
|
||||
readString(slot: number): string | undefined {
|
||||
const p = this.fieldPos(slot);
|
||||
if (p === 0) return undefined;
|
||||
const str = p + this.dv.getInt32(p, true);
|
||||
const len = this.dv.getInt32(str, true);
|
||||
const bytes = new Uint8Array(this.dv.buffer, this.dv.byteOffset + str + 4, len);
|
||||
return new TextDecoder().decode(bytes);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse an Arrow IPC stream's structure. Used to enforce BI-P012's result-row
|
||||
* ceiling on Arrow payloads (including ClickHouse pass-through bytes, where
|
||||
* the count is otherwise invisible to 0pt) and by the roundtrip tests.
|
||||
*/
|
||||
export function decodeArrowStream(bytes: Uint8Array): ArrowStreamInfo {
|
||||
const dv = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
|
||||
let pos = 0;
|
||||
let fields: { name: string; typeType: number }[] = [];
|
||||
let rowCount = 0;
|
||||
let batchCount = 0;
|
||||
|
||||
while (pos + 8 <= bytes.byteLength) {
|
||||
const continuation = dv.getUint32(pos, true);
|
||||
if (continuation !== 0xffffffff) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "not an Arrow IPC stream (missing continuation marker)");
|
||||
}
|
||||
const metaLen = dv.getInt32(pos + 4, true);
|
||||
if (metaLen === 0) break; // end of stream
|
||||
const metaStart = pos + 8;
|
||||
const root = metaStart + dv.getInt32(metaStart, true);
|
||||
const message = new FbTable(dv, root);
|
||||
const headerType = message.readInt8(1, 0);
|
||||
const bodyLength = Number(message.readInt64(3, BIG_ZERO));
|
||||
|
||||
if (headerType === MESSAGE_HEADER_SCHEMA) {
|
||||
const schema = message.readTable(2);
|
||||
if (schema) {
|
||||
const n = schema.vectorLength(1);
|
||||
fields = [];
|
||||
for (let i = 0; i < n; i++) {
|
||||
const f = schema.vectorElementTable(1, i);
|
||||
fields.push({ name: f.readString(0) ?? "", typeType: f.readInt8(2, 0) });
|
||||
}
|
||||
}
|
||||
} else if (headerType === MESSAGE_HEADER_RECORD_BATCH) {
|
||||
const rb = message.readTable(2);
|
||||
if (rb) {
|
||||
rowCount += Number(rb.readInt64(0, BIG_ZERO));
|
||||
batchCount += 1;
|
||||
}
|
||||
}
|
||||
pos = metaStart + metaLen + bodyLength;
|
||||
}
|
||||
|
||||
return { fields, rowCount, batchCount };
|
||||
}
|
||||
|
||||
/**
|
||||
* BI-P012 on the Arrow path: refuse a payload that exceeds the caller's
|
||||
* result-row ceiling. Without this, asking for Arrow would be a way to walk
|
||||
* around `max_result_rows`.
|
||||
*/
|
||||
export function assertArrowWithinResultBudget(bytes: Uint8Array, maxResultRows: number): ArrowStreamInfo {
|
||||
const info = decodeArrowStream(bytes);
|
||||
if (info.rowCount > maxResultRows) {
|
||||
throw new SiaxError(
|
||||
"QUOTA_EXCEEDED",
|
||||
"0pt.analytics",
|
||||
`Arrow result has ${info.rowCount} rows, over the ${maxResultRows}-row budget`,
|
||||
);
|
||||
}
|
||||
return info;
|
||||
}
|
||||
@@ -0,0 +1,474 @@
|
||||
/**
|
||||
* src/analytics/domain/contract.ts
|
||||
* =================================
|
||||
* BI-P004 — ANALYTICS DATA CONTRACT.
|
||||
*
|
||||
* `DatasetContract` is the machine-checkable promise a dataset makes to every
|
||||
* consumer downstream of it. BI-P004's DoD is a GATE, not a document:
|
||||
*
|
||||
* "Breaking dataset schema changes fail compatibility gate unless
|
||||
* explicitly versioned."
|
||||
*
|
||||
* so `checkContractCompatibility` is the deliverable and the prose is the
|
||||
* commentary. The gate is fail-closed: an unrecognised change is BREAKING,
|
||||
* never "probably fine".
|
||||
*
|
||||
* Versioning is semver-shaped (`major.minor.patch`) with one rule that is not
|
||||
* negotiable: a BREAKING change requires a MAJOR bump. A breaking change
|
||||
* shipped under a minor bump is exactly the failure this gate exists to
|
||||
* catch, so it is reported as a violation rather than tolerated.
|
||||
*/
|
||||
import crypto from "node:crypto";
|
||||
import { SiaxError } from "../../platform/errors";
|
||||
import type { AnalyticsFieldType, FieldRole, PiiClassification } from "./entities";
|
||||
import type { AnalyticsTier } from "./tiers";
|
||||
|
||||
export interface ContractField {
|
||||
name: string;
|
||||
type: AnalyticsFieldType;
|
||||
nullable: boolean;
|
||||
role: FieldRole;
|
||||
/** Part of the dataset's primary identifier. */
|
||||
primary_identifier?: boolean;
|
||||
classification?: PiiClassification;
|
||||
description?: string;
|
||||
}
|
||||
|
||||
export interface ContractQualityRule {
|
||||
rule_id: string;
|
||||
kind: string;
|
||||
field_name?: string;
|
||||
severity: "warn" | "block";
|
||||
}
|
||||
|
||||
/** BI-P004's named contents, one field each. */
|
||||
export interface DatasetContract {
|
||||
dataset_id: string;
|
||||
/** semver `major.minor.patch`. */
|
||||
version: string;
|
||||
tier: AnalyticsTier;
|
||||
fields: ContractField[];
|
||||
/** Field names forming the primary identifier, in order. */
|
||||
primary_identifiers: string[];
|
||||
/** Field names usable as dimensions (subset of `fields`). */
|
||||
dimensions: string[];
|
||||
/** Field names usable as measures (subset of `fields`). */
|
||||
measures: string[];
|
||||
/** Freshness SLO in seconds. */
|
||||
freshness_slo_seconds: number;
|
||||
retention_days: number;
|
||||
/** Dataset-level PII classification (the max of its fields, at least). */
|
||||
pii_classification: PiiClassification;
|
||||
/** Upstream refs: source ids, dataset ids, external system refs. */
|
||||
source_lineage: string[];
|
||||
quality_rules: ContractQualityRule[];
|
||||
/** GDPR purpose limitation — what this data may be USED for. */
|
||||
allowed_purposes: string[];
|
||||
}
|
||||
|
||||
export interface SemVer {
|
||||
major: number;
|
||||
minor: number;
|
||||
patch: number;
|
||||
}
|
||||
|
||||
const SEMVER_RE = /^(\d+)\.(\d+)\.(\d+)$/;
|
||||
|
||||
export function parseSemVer(v: string): SemVer {
|
||||
const m = SEMVER_RE.exec(v);
|
||||
if (!m) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", `contract version "${v}" is not major.minor.patch`);
|
||||
}
|
||||
return { major: Number(m[1]), minor: Number(m[2]), patch: Number(m[3]) };
|
||||
}
|
||||
|
||||
// ── change classification ──────────────────────────────────────────────────
|
||||
|
||||
export type ChangeSeverity = "compatible" | "breaking";
|
||||
|
||||
export interface ContractChange {
|
||||
severity: ChangeSeverity;
|
||||
/** Stable machine code so callers can allow-list specific changes. */
|
||||
code: string;
|
||||
field?: string;
|
||||
detail: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Type widening that does NOT break a reader. Deliberately conservative:
|
||||
* int64 -> float64 loses exactness for large integers, so it is NOT here.
|
||||
* Everything absent from this table is breaking.
|
||||
*/
|
||||
const SAFE_TYPE_WIDENING: Partial<Record<AnalyticsFieldType, readonly AnalyticsFieldType[]>> = {
|
||||
date: ["timestamp"],
|
||||
};
|
||||
|
||||
function typeChangeIsCompatible(from: AnalyticsFieldType, to: AnalyticsFieldType): boolean {
|
||||
if (from === to) return true;
|
||||
return (SAFE_TYPE_WIDENING[from] ?? []).includes(to);
|
||||
}
|
||||
|
||||
/**
|
||||
* Diff two contracts for the SAME dataset and classify every change.
|
||||
*
|
||||
* Breaking (a reader written against `previous` can now fail or silently
|
||||
* change meaning):
|
||||
* - field removed
|
||||
* - field type changed other than a listed safe widening
|
||||
* - nullable false -> true (readers may not handle nulls)
|
||||
* - a field's role changes (a measure becoming a dimension changes meaning)
|
||||
* - primary identifier set changes
|
||||
* - a field leaves `measures` or `dimensions`
|
||||
* - freshness SLO loosened (a consumer's own SLO may depend on it)
|
||||
* - retention shortened (history a consumer relied on disappears)
|
||||
* - an allowed purpose is removed (a legitimate consumer becomes illegitimate)
|
||||
* - PII classification raised (consumers may lack clearance)
|
||||
* - a blocking quality rule is removed or downgraded to warn
|
||||
*
|
||||
* Compatible: additive nullable fields, new dimensions/measures, new quality
|
||||
* rules, loosened PII classification, added purposes, longer retention,
|
||||
* tightened freshness.
|
||||
*/
|
||||
export function diffContracts(previous: DatasetContract, next: DatasetContract): ContractChange[] {
|
||||
if (previous.dataset_id !== next.dataset_id) {
|
||||
throw new SiaxError(
|
||||
"INVALID_REQUEST",
|
||||
"0pt.analytics",
|
||||
"contract compatibility is only defined between versions of the SAME dataset",
|
||||
);
|
||||
}
|
||||
const changes: ContractChange[] = [];
|
||||
const prevFields = new Map(previous.fields.map((f) => [f.name, f]));
|
||||
const nextFields = new Map(next.fields.map((f) => [f.name, f]));
|
||||
|
||||
for (const [name, pf] of prevFields) {
|
||||
const nf = nextFields.get(name);
|
||||
if (!nf) {
|
||||
changes.push({ severity: "breaking", code: "field_removed", field: name, detail: `field "${name}" removed` });
|
||||
continue;
|
||||
}
|
||||
if (!typeChangeIsCompatible(pf.type, nf.type)) {
|
||||
changes.push({
|
||||
severity: "breaking",
|
||||
code: "field_type_changed",
|
||||
field: name,
|
||||
detail: `field "${name}" type ${pf.type} -> ${nf.type}`,
|
||||
});
|
||||
} else if (pf.type !== nf.type) {
|
||||
changes.push({
|
||||
severity: "compatible",
|
||||
code: "field_type_widened",
|
||||
field: name,
|
||||
detail: `field "${name}" widened ${pf.type} -> ${nf.type}`,
|
||||
});
|
||||
}
|
||||
if (!pf.nullable && nf.nullable) {
|
||||
changes.push({
|
||||
severity: "breaking",
|
||||
code: "field_became_nullable",
|
||||
field: name,
|
||||
detail: `field "${name}" became nullable`,
|
||||
});
|
||||
}
|
||||
if (pf.nullable && !nf.nullable) {
|
||||
changes.push({
|
||||
severity: "compatible",
|
||||
code: "field_became_required",
|
||||
field: name,
|
||||
detail: `field "${name}" is no longer nullable`,
|
||||
});
|
||||
}
|
||||
if (pf.role !== nf.role) {
|
||||
changes.push({
|
||||
severity: "breaking",
|
||||
code: "field_role_changed",
|
||||
field: name,
|
||||
detail: `field "${name}" role ${pf.role} -> ${nf.role}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
for (const [name, nf] of nextFields) {
|
||||
if (prevFields.has(name)) continue;
|
||||
if (nf.nullable) {
|
||||
changes.push({
|
||||
severity: "compatible",
|
||||
code: "nullable_field_added",
|
||||
field: name,
|
||||
detail: `nullable field "${name}" added`,
|
||||
});
|
||||
} else {
|
||||
// A required new field breaks producers, and breaks readers that
|
||||
// reconstruct rows (they now have no value to supply).
|
||||
changes.push({
|
||||
severity: "breaking",
|
||||
code: "required_field_added",
|
||||
field: name,
|
||||
detail: `required (non-nullable) field "${name}" added`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
const idsEqual =
|
||||
previous.primary_identifiers.length === next.primary_identifiers.length &&
|
||||
previous.primary_identifiers.every((v, i) => next.primary_identifiers[i] === v);
|
||||
if (!idsEqual) {
|
||||
changes.push({
|
||||
severity: "breaking",
|
||||
code: "primary_identifier_changed",
|
||||
detail: `primary identifier [${previous.primary_identifiers.join(",")}] -> [${next.primary_identifiers.join(",")}]`,
|
||||
});
|
||||
}
|
||||
|
||||
for (const d of previous.dimensions) {
|
||||
if (!next.dimensions.includes(d)) {
|
||||
changes.push({ severity: "breaking", code: "dimension_removed", field: d, detail: `dimension "${d}" removed` });
|
||||
}
|
||||
}
|
||||
for (const d of next.dimensions) {
|
||||
if (!previous.dimensions.includes(d)) {
|
||||
changes.push({ severity: "compatible", code: "dimension_added", field: d, detail: `dimension "${d}" added` });
|
||||
}
|
||||
}
|
||||
for (const m of previous.measures) {
|
||||
if (!next.measures.includes(m)) {
|
||||
changes.push({ severity: "breaking", code: "measure_removed", field: m, detail: `measure "${m}" removed` });
|
||||
}
|
||||
}
|
||||
for (const m of next.measures) {
|
||||
if (!previous.measures.includes(m)) {
|
||||
changes.push({ severity: "compatible", code: "measure_added", field: m, detail: `measure "${m}" added` });
|
||||
}
|
||||
}
|
||||
|
||||
if (next.freshness_slo_seconds > previous.freshness_slo_seconds) {
|
||||
changes.push({
|
||||
severity: "breaking",
|
||||
code: "freshness_slo_loosened",
|
||||
detail: `freshness SLO ${previous.freshness_slo_seconds}s -> ${next.freshness_slo_seconds}s`,
|
||||
});
|
||||
}
|
||||
if (next.retention_days < previous.retention_days) {
|
||||
changes.push({
|
||||
severity: "breaking",
|
||||
code: "retention_shortened",
|
||||
detail: `retention ${previous.retention_days}d -> ${next.retention_days}d`,
|
||||
});
|
||||
}
|
||||
|
||||
for (const p of previous.allowed_purposes) {
|
||||
if (!next.allowed_purposes.includes(p)) {
|
||||
changes.push({ severity: "breaking", code: "purpose_removed", detail: `allowed purpose "${p}" removed` });
|
||||
}
|
||||
}
|
||||
|
||||
const PII_RANK: Record<PiiClassification, number> = {
|
||||
none: 0,
|
||||
pseudonymous: 1,
|
||||
personal: 2,
|
||||
sensitive: 3,
|
||||
};
|
||||
if (PII_RANK[next.pii_classification] > PII_RANK[previous.pii_classification]) {
|
||||
changes.push({
|
||||
severity: "breaking",
|
||||
code: "pii_classification_raised",
|
||||
detail: `PII classification ${previous.pii_classification} -> ${next.pii_classification}`,
|
||||
});
|
||||
}
|
||||
|
||||
const prevRules = new Map(previous.quality_rules.map((r) => [r.rule_id, r]));
|
||||
for (const [id, pr] of prevRules) {
|
||||
const nr = next.quality_rules.find((r) => r.rule_id === id);
|
||||
if (!nr && pr.severity === "block") {
|
||||
changes.push({
|
||||
severity: "breaking",
|
||||
code: "blocking_quality_rule_removed",
|
||||
detail: `blocking quality rule "${id}" removed`,
|
||||
});
|
||||
} else if (nr && pr.severity === "block" && nr.severity === "warn") {
|
||||
changes.push({
|
||||
severity: "breaking",
|
||||
code: "blocking_quality_rule_downgraded",
|
||||
detail: `quality rule "${id}" downgraded block -> warn`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (previous.tier !== next.tier) {
|
||||
changes.push({
|
||||
severity: "breaking",
|
||||
code: "tier_changed",
|
||||
detail: `tier ${previous.tier} -> ${next.tier}`,
|
||||
});
|
||||
}
|
||||
|
||||
return changes;
|
||||
}
|
||||
|
||||
export interface CompatibilityVerdict {
|
||||
compatible: boolean;
|
||||
/** True when a MAJOR bump legitimises the breaking changes. */
|
||||
explicitly_versioned: boolean;
|
||||
changes: ContractChange[];
|
||||
breaking: ContractChange[];
|
||||
previous_version: string;
|
||||
next_version: string;
|
||||
/** Human-readable reason when `compatible` is false. */
|
||||
reason?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* BI-P004's gate. Returns a verdict; `assertContractCompatible` throws.
|
||||
*
|
||||
* Rules, in order:
|
||||
* 1. the version must move FORWARD (equal or lower is rejected outright);
|
||||
* 2. breaking changes REQUIRE a major bump — that is what "explicitly
|
||||
* versioned" means;
|
||||
* 3. a major bump with no breaking change is allowed (over-declaring is
|
||||
* safe; under-declaring is what we refuse).
|
||||
*/
|
||||
export function checkContractCompatibility(
|
||||
previous: DatasetContract,
|
||||
next: DatasetContract,
|
||||
): CompatibilityVerdict {
|
||||
const pv = parseSemVer(previous.version);
|
||||
const nv = parseSemVer(next.version);
|
||||
const changes = diffContracts(previous, next);
|
||||
const breaking = changes.filter((c) => c.severity === "breaking");
|
||||
|
||||
const forward =
|
||||
nv.major > pv.major ||
|
||||
(nv.major === pv.major && nv.minor > pv.minor) ||
|
||||
(nv.major === pv.major && nv.minor === pv.minor && nv.patch > pv.patch);
|
||||
if (!forward) {
|
||||
return {
|
||||
compatible: false,
|
||||
explicitly_versioned: false,
|
||||
changes,
|
||||
breaking,
|
||||
previous_version: previous.version,
|
||||
next_version: next.version,
|
||||
reason: `version must move forward: ${previous.version} -> ${next.version}`,
|
||||
};
|
||||
}
|
||||
|
||||
const majorBumped = nv.major > pv.major;
|
||||
if (breaking.length > 0 && !majorBumped) {
|
||||
return {
|
||||
compatible: false,
|
||||
explicitly_versioned: false,
|
||||
changes,
|
||||
breaking,
|
||||
previous_version: previous.version,
|
||||
next_version: next.version,
|
||||
reason:
|
||||
`${breaking.length} breaking change(s) require a MAJOR version bump ` +
|
||||
`(${previous.version} -> ${next.version} is not one): ` +
|
||||
breaking.map((b) => b.code).join(", "),
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
compatible: true,
|
||||
explicitly_versioned: majorBumped && breaking.length > 0,
|
||||
changes,
|
||||
breaking,
|
||||
previous_version: previous.version,
|
||||
next_version: next.version,
|
||||
};
|
||||
}
|
||||
|
||||
/** Throwing form for call sites that treat incompatibility as a failure. */
|
||||
export function assertContractCompatible(previous: DatasetContract, next: DatasetContract): CompatibilityVerdict {
|
||||
const verdict = checkContractCompatibility(previous, next);
|
||||
if (!verdict.compatible) {
|
||||
throw new SiaxError("CONFLICT", "0pt.analytics", verdict.reason ?? "dataset contract compatibility gate failed");
|
||||
}
|
||||
return verdict;
|
||||
}
|
||||
|
||||
/**
|
||||
* Structural validation of a single contract (independent of history).
|
||||
* Catches the contract that is internally inconsistent before it is ever
|
||||
* compared to anything.
|
||||
*/
|
||||
export function validateContract(contract: DatasetContract): void {
|
||||
parseSemVer(contract.version);
|
||||
if (!contract.dataset_id.trim()) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "contract needs a dataset_id");
|
||||
}
|
||||
if (contract.fields.length === 0) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", `contract ${contract.dataset_id} declares no fields`);
|
||||
}
|
||||
const names = new Set<string>();
|
||||
for (const f of contract.fields) {
|
||||
if (names.has(f.name)) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", `duplicate field "${f.name}"`);
|
||||
}
|
||||
names.add(f.name);
|
||||
}
|
||||
for (const label of ["primary_identifiers", "dimensions", "measures"] as const) {
|
||||
for (const n of contract[label]) {
|
||||
if (!names.has(n)) {
|
||||
throw new SiaxError(
|
||||
"INVALID_REQUEST",
|
||||
"0pt.analytics",
|
||||
`${label} references unknown field "${n}" in ${contract.dataset_id}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
if (contract.primary_identifiers.length === 0) {
|
||||
throw new SiaxError(
|
||||
"INVALID_REQUEST",
|
||||
"0pt.analytics",
|
||||
`contract ${contract.dataset_id} declares no primary identifier`,
|
||||
);
|
||||
}
|
||||
if (contract.allowed_purposes.length === 0) {
|
||||
throw new SiaxError(
|
||||
"INVALID_REQUEST",
|
||||
"0pt.analytics",
|
||||
`contract ${contract.dataset_id} declares no allowed_purposes (purpose limitation is not optional)`,
|
||||
);
|
||||
}
|
||||
if (contract.freshness_slo_seconds <= 0) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "freshness_slo_seconds must be positive");
|
||||
}
|
||||
if (contract.retention_days < 0) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "retention_days must not be negative");
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Stable hash of a contract's MEANING (field order-insensitive), used as
|
||||
* `DatasetVersion.contract_hash` so two versions that differ only in comment
|
||||
* text do not look like schema drift.
|
||||
*/
|
||||
export function contractHash(contract: DatasetContract): string {
|
||||
const canonical = {
|
||||
dataset_id: contract.dataset_id,
|
||||
version: contract.version,
|
||||
tier: contract.tier,
|
||||
fields: [...contract.fields]
|
||||
.map((f) => ({
|
||||
name: f.name,
|
||||
type: f.type,
|
||||
nullable: f.nullable,
|
||||
role: f.role,
|
||||
primary_identifier: f.primary_identifier ?? false,
|
||||
classification: f.classification ?? "none",
|
||||
}))
|
||||
.sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0)),
|
||||
primary_identifiers: contract.primary_identifiers,
|
||||
dimensions: [...contract.dimensions].sort(),
|
||||
measures: [...contract.measures].sort(),
|
||||
freshness_slo_seconds: contract.freshness_slo_seconds,
|
||||
retention_days: contract.retention_days,
|
||||
pii_classification: contract.pii_classification,
|
||||
source_lineage: [...contract.source_lineage].sort(),
|
||||
quality_rules: [...contract.quality_rules].sort((a, b) => (a.rule_id < b.rule_id ? -1 : 1)),
|
||||
allowed_purposes: [...contract.allowed_purposes].sort(),
|
||||
};
|
||||
return crypto.createHash("sha256").update(JSON.stringify(canonical)).digest("hex");
|
||||
}
|
||||
@@ -0,0 +1,536 @@
|
||||
/**
|
||||
* src/analytics/domain/entities.ts
|
||||
* =================================
|
||||
* BI-P002 — ANALYTICS DOMAIN MODEL.
|
||||
*
|
||||
* The nineteen canonical analytics entities, named EXACTLY as the program
|
||||
* lists them (no renaming, no merging, no "we already have something like
|
||||
* that"):
|
||||
*
|
||||
* AnalyticsDataset DatasetVersion DatasetField
|
||||
* DatasetGrant MetricDefinition MetricVersion
|
||||
* DimensionDefinition SemanticModel AnalyticsQuery
|
||||
* QueryExecution QueryResult DashboardDefinition
|
||||
* DashboardVersion ReportDefinition ReportRun
|
||||
* DataQualityRule DataQualityRun LineageReference
|
||||
* BIApplicationBinding
|
||||
*
|
||||
* Every tenant-owned record carries `tenant_id` + `project_id` where
|
||||
* applicable. The four version entities the program names as immutable
|
||||
* (MetricVersion, SemanticModel version, DashboardVersion, DatasetVersion)
|
||||
* are `readonly` here AND immutable in the database — sql/008 installs BEFORE
|
||||
* UPDATE/DELETE triggers, because a TypeScript `readonly` is a compiler
|
||||
* opinion, not a storage guarantee.
|
||||
*
|
||||
* NAME COLLISION, deliberate: src/services/dataset-version-service.ts already
|
||||
* exports an interface called `DatasetVersion` for the ST0RE-artifact-backed
|
||||
* dataset versioning that predates this program. That one is about *dataset
|
||||
* content artifacts*; this one is about *analytics dataset schema versions*
|
||||
* and lives in its own module namespace (`AnalyticsDatasetVersion` is NOT
|
||||
* used as the name — BI-P002 lists `DatasetVersion` verbatim and that is what
|
||||
* ships). Import them qualified when both are needed in one file.
|
||||
*
|
||||
* ARCHITECTURE BOUNDARY: these entities are owned by 0pt. A product repo
|
||||
* (VehIQ) supplies domain MEANING through `BIApplicationBinding`; it never
|
||||
* defines its own copies of these tables and never receives engine
|
||||
* credentials.
|
||||
*/
|
||||
import type { AnalyticsTier } from "./tiers";
|
||||
|
||||
/** Every tenant-owned analytics record. */
|
||||
export interface TenantScoped {
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
}
|
||||
|
||||
// ── field/typing vocabulary ────────────────────────────────────────────────
|
||||
|
||||
export const ANALYTICS_FIELD_TYPES = [
|
||||
"string",
|
||||
"int64",
|
||||
"float64",
|
||||
"bool",
|
||||
"date",
|
||||
"timestamp",
|
||||
"decimal",
|
||||
"json",
|
||||
] as const;
|
||||
export type AnalyticsFieldType = (typeof ANALYTICS_FIELD_TYPES)[number];
|
||||
|
||||
/** A field's analytical role. Measures aggregate; dimensions group. */
|
||||
export const FIELD_ROLES = ["identifier", "dimension", "measure", "attribute"] as const;
|
||||
export type FieldRole = (typeof FIELD_ROLES)[number];
|
||||
|
||||
/** PII classification, aligned with the SIAX risk vocabulary (L0 = none). */
|
||||
export const PII_CLASSIFICATIONS = ["none", "pseudonymous", "personal", "sensitive"] as const;
|
||||
export type PiiClassification = (typeof PII_CLASSIFICATIONS)[number];
|
||||
|
||||
// ── 1. AnalyticsDataset ────────────────────────────────────────────────────
|
||||
|
||||
export const DATASET_STATUSES = ["draft", "active", "deprecated", "retired"] as const;
|
||||
export type AnalyticsDatasetStatus = (typeof DATASET_STATUSES)[number];
|
||||
|
||||
/**
|
||||
* BI-P003 DoD: "Every dataset declares tier, schema, owner, source, freshness,
|
||||
* quality, retention, classification." All eight are REQUIRED fields here —
|
||||
* an optional declaration is not a declaration.
|
||||
*/
|
||||
export interface AnalyticsDataset extends TenantScoped {
|
||||
dataset_id: string;
|
||||
name: string;
|
||||
/** BI-P003 tier. */
|
||||
tier: AnalyticsTier;
|
||||
/** Contract id + version this dataset's schema conforms to (BI-P004). */
|
||||
schema_contract_id: string;
|
||||
schema_contract_version: string;
|
||||
/** Accountable human/team, not the person who typed the create call. */
|
||||
owner_id: string;
|
||||
/** Upstream: a 0pt source_id, another dataset_id, or an external ref. */
|
||||
source_ref: string;
|
||||
/** Freshness SLO in seconds — how stale the data may be before it is late. */
|
||||
freshness_slo_seconds: number;
|
||||
/** DataQualityRule ids that gate this dataset. */
|
||||
quality_rule_ids: string[];
|
||||
/** Retention in days. 0 = keep indefinitely (must be justified in policy). */
|
||||
retention_days: number;
|
||||
classification: PiiClassification;
|
||||
status: AnalyticsDatasetStatus;
|
||||
/** BI-P003 escape hatch for RAW/NORMALIZED exposure to a BI surface. */
|
||||
raw_access_exception?: string;
|
||||
description?: string;
|
||||
created_at: string;
|
||||
updated_at?: string;
|
||||
created_by?: string;
|
||||
}
|
||||
|
||||
// ── 2. DatasetVersion (IMMUTABLE) ──────────────────────────────────────────
|
||||
|
||||
export interface DatasetVersion extends TenantScoped {
|
||||
readonly dataset_version_id: string;
|
||||
readonly dataset_id: string;
|
||||
/** Monotonic per dataset, starting at 1. */
|
||||
readonly version: number;
|
||||
/** The frozen DatasetContract JSON at this version (BI-P004). */
|
||||
readonly contract_hash: string;
|
||||
readonly schema_contract_version: string;
|
||||
readonly record_count?: number;
|
||||
readonly created_at: string;
|
||||
readonly created_by?: string;
|
||||
}
|
||||
|
||||
// ── 3. DatasetField ────────────────────────────────────────────────────────
|
||||
|
||||
export interface DatasetField extends TenantScoped {
|
||||
dataset_field_id: string;
|
||||
dataset_id: string;
|
||||
/** The dataset version this field description belongs to. */
|
||||
dataset_version: number;
|
||||
name: string;
|
||||
field_type: AnalyticsFieldType;
|
||||
nullable: boolean;
|
||||
role: FieldRole;
|
||||
/** True for the field(s) forming the dataset's primary identifier. */
|
||||
primary_identifier: boolean;
|
||||
classification: PiiClassification;
|
||||
description?: string;
|
||||
}
|
||||
|
||||
// ── 4. DatasetGrant ────────────────────────────────────────────────────────
|
||||
|
||||
export const GRANT_LEVELS = ["read", "query", "export", "admin"] as const;
|
||||
export type GrantLevel = (typeof GRANT_LEVELS)[number];
|
||||
|
||||
/**
|
||||
* Who may use a dataset, and FOR WHAT. `allowed_purposes` is the GDPR purpose
|
||||
* limitation carried into analytics — a grant without a purpose is refused
|
||||
* by the repository layer.
|
||||
*/
|
||||
export interface DatasetGrant extends TenantScoped {
|
||||
grant_id: string;
|
||||
dataset_id: string;
|
||||
/** Subject: a CL0UD principal, role, or BI application binding id. */
|
||||
subject_id: string;
|
||||
subject_type: "user" | "role" | "service" | "bi_application";
|
||||
level: GrantLevel;
|
||||
allowed_purposes: string[];
|
||||
expires_at?: string;
|
||||
created_at: string;
|
||||
created_by?: string;
|
||||
}
|
||||
|
||||
// ── 5. MetricDefinition ────────────────────────────────────────────────────
|
||||
|
||||
export const METRIC_AGGREGATIONS = [
|
||||
"sum",
|
||||
"avg",
|
||||
"min",
|
||||
"max",
|
||||
"count",
|
||||
"count_distinct",
|
||||
"ratio",
|
||||
"derived",
|
||||
] as const;
|
||||
export type MetricAggregation = (typeof METRIC_AGGREGATIONS)[number];
|
||||
|
||||
export interface MetricDefinition extends TenantScoped {
|
||||
metric_id: string;
|
||||
/** Stable business name; unique per tenant. */
|
||||
name: string;
|
||||
dataset_id: string;
|
||||
aggregation: MetricAggregation;
|
||||
/** Measure field aggregated, or the numerator for ratio metrics. */
|
||||
measure_field: string;
|
||||
/** Denominator for ratio metrics. */
|
||||
denominator_field?: string;
|
||||
/** Dimensions this metric may legitimately be sliced by. */
|
||||
allowed_dimension_ids: string[];
|
||||
/** Current live version number (points at a MetricVersion). */
|
||||
current_version: number;
|
||||
owner_id: string;
|
||||
description?: string;
|
||||
created_at: string;
|
||||
updated_at?: string;
|
||||
}
|
||||
|
||||
// ── 6. MetricVersion (IMMUTABLE) ───────────────────────────────────────────
|
||||
|
||||
export interface MetricVersion extends TenantScoped {
|
||||
readonly metric_version_id: string;
|
||||
readonly metric_id: string;
|
||||
readonly version: number;
|
||||
/** Frozen definition JSON — what this metric MEANT at this version. */
|
||||
readonly definition_json: string;
|
||||
readonly definition_hash: string;
|
||||
/** Free-text reason the metric changed; required for version > 1. */
|
||||
readonly change_reason?: string;
|
||||
readonly created_at: string;
|
||||
readonly created_by?: string;
|
||||
}
|
||||
|
||||
// ── 7. DimensionDefinition ─────────────────────────────────────────────────
|
||||
|
||||
export interface DimensionDefinition extends TenantScoped {
|
||||
dimension_id: string;
|
||||
name: string;
|
||||
dataset_id: string;
|
||||
/**
|
||||
* Physical column/key this dimension resolves to. This is the ONLY place a
|
||||
* caller-supplied dimension name becomes a physical identifier — the query
|
||||
* compiler resolves through here and never splices caller input (BI-P011
|
||||
* "dimension identifier injection").
|
||||
*/
|
||||
field_name: string;
|
||||
field_type: AnalyticsFieldType;
|
||||
classification: PiiClassification;
|
||||
description?: string;
|
||||
created_at: string;
|
||||
}
|
||||
|
||||
// ── 8. SemanticModel (versioned; versions IMMUTABLE) ───────────────────────
|
||||
|
||||
export interface SemanticModel extends TenantScoped {
|
||||
semantic_model_id: string;
|
||||
name: string;
|
||||
/** Immutable version number — a change mints a new row, never an UPDATE. */
|
||||
readonly version: number;
|
||||
/** MetricFlow/Cube-compatible model document (BI-P029/BI-P032 consume it). */
|
||||
readonly model_json: string;
|
||||
readonly model_hash: string;
|
||||
dataset_ids: string[];
|
||||
owner_id: string;
|
||||
readonly created_at: string;
|
||||
readonly created_by?: string;
|
||||
}
|
||||
|
||||
// ── 9. AnalyticsQuery ──────────────────────────────────────────────────────
|
||||
|
||||
export const QUERY_SURFACES = ["metric", "sql", "projection"] as const;
|
||||
export type QuerySurface = (typeof QUERY_SURFACES)[number];
|
||||
|
||||
/**
|
||||
* A SAVED, reusable query. The request that actually runs is
|
||||
* `src/analytics/engine/query-compiler.ts`'s `MetricQueryRequest`; this entity
|
||||
* is its durable, governed form.
|
||||
*/
|
||||
export interface AnalyticsQuery extends TenantScoped {
|
||||
query_id: string;
|
||||
name: string;
|
||||
surface: QuerySurface;
|
||||
dataset_id: string;
|
||||
metric_id?: string;
|
||||
/** Serialized closed-DSL request. Never raw SQL for surface="metric". */
|
||||
request_json: string;
|
||||
owner_id: string;
|
||||
created_at: string;
|
||||
updated_at?: string;
|
||||
}
|
||||
|
||||
// ── 10. QueryExecution ─────────────────────────────────────────────────────
|
||||
|
||||
export const EXECUTION_STATES = [
|
||||
"queued",
|
||||
"running",
|
||||
"succeeded",
|
||||
"failed",
|
||||
"cancelled",
|
||||
"rejected",
|
||||
] as const;
|
||||
export type ExecutionState = (typeof EXECUTION_STATES)[number];
|
||||
|
||||
export interface QueryExecution extends TenantScoped {
|
||||
execution_id: string;
|
||||
query_id?: string;
|
||||
/** Engine-side identifier used to CANCEL this execution (BI-P009). */
|
||||
engine_query_id: string;
|
||||
state: ExecutionState;
|
||||
/** Which provider ran it: "clickhouse" | "duckdb" | "trino". */
|
||||
provider: string;
|
||||
submitted_at: string;
|
||||
started_at?: string;
|
||||
finished_at?: string;
|
||||
duration_ms?: number;
|
||||
rows_read?: number;
|
||||
bytes_read?: number;
|
||||
result_rows?: number;
|
||||
/** SiaxErrorCode when state="failed"/"rejected". Never a raw engine body. */
|
||||
error_code?: string;
|
||||
correlation_id?: string;
|
||||
}
|
||||
|
||||
// ── 11. QueryResult ────────────────────────────────────────────────────────
|
||||
|
||||
export const RESULT_ENCODINGS = ["json", "arrow_ipc"] as const;
|
||||
export type ResultEncoding = (typeof RESULT_ENCODINGS)[number];
|
||||
|
||||
export interface QueryResult extends TenantScoped {
|
||||
result_id: string;
|
||||
execution_id: string;
|
||||
encoding: ResultEncoding;
|
||||
row_count: number;
|
||||
/** Canonical ST0RE ref when materialised; inline results carry no bytes here. */
|
||||
artifact_ref?: string;
|
||||
/** True when the engine truncated at max_result_rows (BI-P012). */
|
||||
truncated: boolean;
|
||||
expires_at?: string;
|
||||
created_at: string;
|
||||
}
|
||||
|
||||
// ── 12. DashboardDefinition ────────────────────────────────────────────────
|
||||
|
||||
export interface DashboardDefinition extends TenantScoped {
|
||||
dashboard_id: string;
|
||||
name: string;
|
||||
/** Live version pointer. */
|
||||
current_version: number;
|
||||
owner_id: string;
|
||||
/** Datasets the dashboard reads — checked against BI-P003 tier rules. */
|
||||
dataset_ids: string[];
|
||||
description?: string;
|
||||
created_at: string;
|
||||
updated_at?: string;
|
||||
}
|
||||
|
||||
// ── 13. DashboardVersion (IMMUTABLE) ───────────────────────────────────────
|
||||
|
||||
export interface DashboardVersion extends TenantScoped {
|
||||
readonly dashboard_version_id: string;
|
||||
readonly dashboard_id: string;
|
||||
readonly version: number;
|
||||
/** Frozen layout/panel document. */
|
||||
readonly layout_json: string;
|
||||
readonly layout_hash: string;
|
||||
readonly created_at: string;
|
||||
readonly created_by?: string;
|
||||
}
|
||||
|
||||
// ── 14. ReportDefinition ───────────────────────────────────────────────────
|
||||
|
||||
export const REPORT_FORMATS = ["pdf", "csv", "xlsx", "parquet", "arrow_ipc"] as const;
|
||||
export type ReportFormat = (typeof REPORT_FORMATS)[number];
|
||||
|
||||
export interface ReportDefinition extends TenantScoped {
|
||||
report_id: string;
|
||||
name: string;
|
||||
dashboard_id?: string;
|
||||
query_id?: string;
|
||||
format: ReportFormat;
|
||||
/** Cron expression; undefined = on-demand only. */
|
||||
schedule_cron?: string;
|
||||
owner_id: string;
|
||||
created_at: string;
|
||||
updated_at?: string;
|
||||
}
|
||||
|
||||
// ── 15. ReportRun ──────────────────────────────────────────────────────────
|
||||
|
||||
export interface ReportRun extends TenantScoped {
|
||||
report_run_id: string;
|
||||
report_id: string;
|
||||
state: ExecutionState;
|
||||
started_at: string;
|
||||
finished_at?: string;
|
||||
/** ST0RE artifact ref of the produced report. */
|
||||
artifact_ref?: string;
|
||||
error_code?: string;
|
||||
correlation_id?: string;
|
||||
}
|
||||
|
||||
// ── 16. DataQualityRule ────────────────────────────────────────────────────
|
||||
|
||||
export const QUALITY_RULE_KINDS = [
|
||||
"not_null",
|
||||
"unique",
|
||||
"accepted_values",
|
||||
"range",
|
||||
"freshness",
|
||||
"row_count",
|
||||
"referential",
|
||||
"custom",
|
||||
] as const;
|
||||
export type QualityRuleKind = (typeof QUALITY_RULE_KINDS)[number];
|
||||
|
||||
export const QUALITY_SEVERITIES = ["warn", "block"] as const;
|
||||
export type QualitySeverity = (typeof QUALITY_SEVERITIES)[number];
|
||||
|
||||
export interface DataQualityRule extends TenantScoped {
|
||||
rule_id: string;
|
||||
dataset_id: string;
|
||||
kind: QualityRuleKind;
|
||||
/** Field the rule applies to; omitted for dataset-level rules. */
|
||||
field_name?: string;
|
||||
/** Rule parameters (bounds, accepted values, max age). */
|
||||
params_json: string;
|
||||
severity: QualitySeverity;
|
||||
created_at: string;
|
||||
created_by?: string;
|
||||
}
|
||||
|
||||
// ── 17. DataQualityRun ─────────────────────────────────────────────────────
|
||||
|
||||
export interface DataQualityRun extends TenantScoped {
|
||||
quality_run_id: string;
|
||||
rule_id: string;
|
||||
dataset_id: string;
|
||||
dataset_version?: number;
|
||||
passed: boolean;
|
||||
/** Rows that violated the rule. */
|
||||
violation_count: number;
|
||||
evaluated_rows: number;
|
||||
started_at: string;
|
||||
finished_at?: string;
|
||||
correlation_id?: string;
|
||||
}
|
||||
|
||||
// ── 18. LineageReference ───────────────────────────────────────────────────
|
||||
|
||||
export const LINEAGE_DIRECTIONS = ["input", "output"] as const;
|
||||
export type LineageDirection = (typeof LINEAGE_DIRECTIONS)[number];
|
||||
|
||||
/**
|
||||
* The queryable projection of an OpenLineage edge (BI-P027). AUD0 stays the
|
||||
* system of record; this is 0pt's local index so "what fed this dashboard"
|
||||
* is answerable without leaving the analytics plane.
|
||||
*/
|
||||
export interface LineageReference extends TenantScoped {
|
||||
lineage_reference_id: string;
|
||||
/** OpenLineage run id this edge came from. */
|
||||
run_id: string;
|
||||
job_name: string;
|
||||
direction: LineageDirection;
|
||||
/** The analytics entity this edge touches. */
|
||||
entity_type: "dataset" | "metric" | "dashboard" | "report";
|
||||
entity_id: string;
|
||||
/** Namespaced dataset name on the other side of the edge. */
|
||||
counterpart_ref: string;
|
||||
observed_at: string;
|
||||
}
|
||||
|
||||
// ── 19. BIApplicationBinding ───────────────────────────────────────────────
|
||||
|
||||
export const BI_APPLICATION_KINDS = ["superset", "evidence", "des1gn", "notebook", "custom"] as const;
|
||||
export type BiApplicationKind = (typeof BI_APPLICATION_KINDS)[number];
|
||||
|
||||
/**
|
||||
* How an external BI surface is attached to the analytics plane.
|
||||
*
|
||||
* NORMATIVE (BI program §0 + the SIAX/0pt ownership boundary): this binding
|
||||
* carries NO engine credential. A bound application authenticates to the 0pt
|
||||
* Analytics API with a CL0UD identity and receives compiled, tenant-forced
|
||||
* results — it never learns a ClickHouse DSN, user, or password. The absence
|
||||
* of a credential field here is the enforcement, not a comment.
|
||||
*/
|
||||
export interface BIApplicationBinding extends TenantScoped {
|
||||
binding_id: string;
|
||||
kind: BiApplicationKind;
|
||||
name: string;
|
||||
/** CL0UD principal the application authenticates as. */
|
||||
principal_id: string;
|
||||
/** Datasets/metrics this application may reach at all. */
|
||||
dataset_ids: string[];
|
||||
metric_ids: string[];
|
||||
allowed_purposes: string[];
|
||||
enabled: boolean;
|
||||
created_at: string;
|
||||
created_by?: string;
|
||||
}
|
||||
|
||||
// ── registry of the canonical nineteen ─────────────────────────────────────
|
||||
|
||||
/**
|
||||
* The canonical entity names, verbatim from BI-P002. Persisted-schema tests
|
||||
* assert sql/008 defines a table for every one of these, so a future rename
|
||||
* or omission is a test failure rather than drift.
|
||||
*/
|
||||
export const ANALYTICS_DOMAIN_ENTITIES = [
|
||||
"AnalyticsDataset",
|
||||
"DatasetVersion",
|
||||
"DatasetField",
|
||||
"DatasetGrant",
|
||||
"MetricDefinition",
|
||||
"MetricVersion",
|
||||
"DimensionDefinition",
|
||||
"SemanticModel",
|
||||
"AnalyticsQuery",
|
||||
"QueryExecution",
|
||||
"QueryResult",
|
||||
"DashboardDefinition",
|
||||
"DashboardVersion",
|
||||
"ReportDefinition",
|
||||
"ReportRun",
|
||||
"DataQualityRule",
|
||||
"DataQualityRun",
|
||||
"LineageReference",
|
||||
"BIApplicationBinding",
|
||||
] as const;
|
||||
export type AnalyticsDomainEntity = (typeof ANALYTICS_DOMAIN_ENTITIES)[number];
|
||||
|
||||
/** Entity -> physical table in sql/008_analytics_domain_model.sql. */
|
||||
export const ANALYTICS_ENTITY_TABLES: Record<AnalyticsDomainEntity, string> = {
|
||||
AnalyticsDataset: "opt_analytics_dataset",
|
||||
DatasetVersion: "opt_analytics_dataset_version",
|
||||
DatasetField: "opt_analytics_dataset_field",
|
||||
DatasetGrant: "opt_analytics_dataset_grant",
|
||||
MetricDefinition: "opt_analytics_metric_definition",
|
||||
MetricVersion: "opt_analytics_metric_version",
|
||||
DimensionDefinition: "opt_analytics_dimension_definition",
|
||||
SemanticModel: "opt_analytics_semantic_model",
|
||||
AnalyticsQuery: "opt_analytics_query",
|
||||
QueryExecution: "opt_analytics_query_execution",
|
||||
QueryResult: "opt_analytics_query_result",
|
||||
DashboardDefinition: "opt_analytics_dashboard_definition",
|
||||
DashboardVersion: "opt_analytics_dashboard_version",
|
||||
ReportDefinition: "opt_analytics_report_definition",
|
||||
ReportRun: "opt_analytics_report_run",
|
||||
DataQualityRule: "opt_analytics_data_quality_rule",
|
||||
DataQualityRun: "opt_analytics_data_quality_run",
|
||||
LineageReference: "opt_analytics_lineage_reference",
|
||||
BIApplicationBinding: "opt_analytics_bi_application_binding",
|
||||
};
|
||||
|
||||
/** The four entities BI-P002 requires to be immutable once written. */
|
||||
export const IMMUTABLE_VERSION_ENTITIES: readonly AnalyticsDomainEntity[] = [
|
||||
"DatasetVersion",
|
||||
"MetricVersion",
|
||||
"SemanticModel",
|
||||
"DashboardVersion",
|
||||
];
|
||||
@@ -0,0 +1,83 @@
|
||||
/**
|
||||
* src/analytics/domain/tiers.ts
|
||||
* ==============================
|
||||
* BI-P003 — ANALYTICS DATA TIERS.
|
||||
*
|
||||
* The four canonical tiers every analytics dataset in SIAX belongs to. The
|
||||
* tier is a CONTRACT about where a dataset may be read from and by whom, not
|
||||
* a naming convention:
|
||||
*
|
||||
* RAW -> ST0RE Parquet/Iceberg (as-ingested, no business meaning)
|
||||
* NORMALIZED -> ST0RE Iceberg (typed, deduplicated, still source-shaped)
|
||||
* CURATED -> ST0RE Iceberg (business entities, modelled)
|
||||
* SERVING -> ClickHouse / projection (query-optimised, dashboard-facing)
|
||||
*
|
||||
* BI-P003's normative sentence is "No business dashboard reads arbitrary raw
|
||||
* ingestion tables by default" — `assertReadableByBiSurface` is where that
|
||||
* sentence stops being prose. A BI surface (dashboard, report, embedded
|
||||
* chart) may read SERVING and CURATED; RAW and NORMALIZED require an
|
||||
* explicit, recorded exception on the dataset itself.
|
||||
*
|
||||
* ARCHITECTURE BOUNDARY (BI program §0): 0pt owns tier semantics. Product
|
||||
* repos (VehIQ et al) own what the rows MEAN; they never re-declare tiers.
|
||||
*/
|
||||
import { SiaxError } from "../../platform/errors";
|
||||
|
||||
export const ANALYTICS_TIERS = ["RAW", "NORMALIZED", "CURATED", "SERVING"] as const;
|
||||
export type AnalyticsTier = (typeof ANALYTICS_TIERS)[number];
|
||||
|
||||
/** Physical home a tier is expected to live in (BI-P003 "suggested" mapping). */
|
||||
export const TIER_PHYSICAL_HOME: Record<AnalyticsTier, string> = {
|
||||
RAW: "st0re:iceberg|parquet",
|
||||
NORMALIZED: "st0re:iceberg",
|
||||
CURATED: "st0re:iceberg",
|
||||
SERVING: "clickhouse:projection",
|
||||
};
|
||||
|
||||
/** Tiers a BI presentation surface may read without an explicit exception. */
|
||||
export const BI_READABLE_TIERS: readonly AnalyticsTier[] = ["SERVING", "CURATED"];
|
||||
|
||||
export function isAnalyticsTier(v: unknown): v is AnalyticsTier {
|
||||
return typeof v === "string" && (ANALYTICS_TIERS as readonly string[]).includes(v);
|
||||
}
|
||||
|
||||
/**
|
||||
* Rank used for promotion checks: a dataset may only be BUILT FROM a tier at
|
||||
* or below its own (SERVING can be built from CURATED; CURATED must never be
|
||||
* built from SERVING, which would make the projection its own source).
|
||||
*/
|
||||
export const TIER_RANK: Record<AnalyticsTier, number> = {
|
||||
RAW: 0,
|
||||
NORMALIZED: 1,
|
||||
CURATED: 2,
|
||||
SERVING: 3,
|
||||
};
|
||||
|
||||
/**
|
||||
* BI-P003 DoD gate. `raw_access_exception` is the recorded, auditable escape
|
||||
* hatch — absent it, a RAW/NORMALIZED dataset is invisible to dashboards.
|
||||
*/
|
||||
export function assertReadableByBiSurface(
|
||||
dataset: { dataset_id: string; tier: AnalyticsTier; raw_access_exception?: string },
|
||||
surface: string,
|
||||
): void {
|
||||
if (BI_READABLE_TIERS.includes(dataset.tier)) return;
|
||||
if (dataset.raw_access_exception && dataset.raw_access_exception.trim().length > 0) return;
|
||||
throw new SiaxError(
|
||||
"AUTHORIZATION_DENIED",
|
||||
"0pt.analytics",
|
||||
`BI surface "${surface}" may not read ${dataset.tier} dataset ${dataset.dataset_id} ` +
|
||||
`without a recorded raw_access_exception (BI-P003)`,
|
||||
);
|
||||
}
|
||||
|
||||
/** A dataset may not declare a source tier ABOVE its own tier. */
|
||||
export function assertTierLineageSane(datasetTier: AnalyticsTier, sourceTier: AnalyticsTier): void {
|
||||
if (TIER_RANK[sourceTier] > TIER_RANK[datasetTier]) {
|
||||
throw new SiaxError(
|
||||
"INVALID_REQUEST",
|
||||
"0pt.analytics",
|
||||
`a ${datasetTier} dataset cannot be built from a ${sourceTier} dataset (BI-P003 tier order)`,
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,563 @@
|
||||
/**
|
||||
* src/analytics/engine/clickhouse-engine.ts
|
||||
* ==========================================
|
||||
* BI-P009 — ClickHouse as the CANONICAL `SiaxAnalyticsEngine` provider.
|
||||
*
|
||||
* PRESERVATION (BI-P009's first sentence): `ClickHouseAnalytics`
|
||||
* (src/analytics/clickhouse-analytics.ts) is untouched by this change — not
|
||||
* one line. This engine COMPOSES it:
|
||||
*
|
||||
* * `health()` delegates to `ClickHouseAnalytics.health()`.
|
||||
* * `executeMetric()` on the pre-existing `opt_metric_events` table
|
||||
* delegates to `ClickHouseAnalytics.queryMetric()`, i.e. keeps the
|
||||
* already-proven forced-tenant path exactly as it is.
|
||||
* * everything else (the BI-P010 tables) goes through
|
||||
* `src/analytics/engine/query-compiler.ts`, which forces the same
|
||||
* predicate for the same reason.
|
||||
*
|
||||
* "Do not regress the existing forced tenant predicate" is enforced by
|
||||
* tests/analytics/tenant-predicate-regression.test.ts, which fails if either
|
||||
* compiler stops emitting `tenant_id = {tenant:String}`.
|
||||
*
|
||||
* BI-P012 governance is applied on EVERY statement, including the delegated
|
||||
* ones — the settings ride on the HTTP request, and `readonly=1` means the
|
||||
* query text cannot re-set them.
|
||||
*
|
||||
* LEGAL BOUNDARY (inherited): generic 0pt analytics only. Never the b0ts
|
||||
* ClickHouse pipelines, never quarantined market data.
|
||||
*/
|
||||
import crypto from "node:crypto";
|
||||
import { SiaxError, siaxErrorCodeForUpstreamStatus } from "../../platform/errors";
|
||||
import { ClickHouseAnalytics } from "../clickhouse-analytics";
|
||||
import {
|
||||
compileMetricQuery,
|
||||
compileProjectionQuery,
|
||||
dimensionCatalogFrom,
|
||||
FORCED_TENANT_PREDICATE,
|
||||
type DimensionCatalog,
|
||||
} from "./query-compiler";
|
||||
import { analyticsSchemaDdl, type AnalyticsSchemaOptions } from "./schema";
|
||||
import {
|
||||
applyQuota,
|
||||
clickhouseSettings,
|
||||
ConcurrencyGate,
|
||||
DEFAULT_BUDGETS,
|
||||
planningSettings,
|
||||
type ResourceBudget,
|
||||
} from "./resource-governance";
|
||||
import type {
|
||||
AnalystQueryRequest,
|
||||
AnalyticsIdentity,
|
||||
EngineResult,
|
||||
HealthReport,
|
||||
MetricRequest,
|
||||
ProjectionIngestRequest,
|
||||
ProjectionRefreshResult,
|
||||
QueryEstimate,
|
||||
QueryExplanation,
|
||||
SiaxAnalyticsEngine,
|
||||
} from "./siax-analytics-engine";
|
||||
|
||||
export interface ClickHouseEngineOptions {
|
||||
baseUrl: string;
|
||||
/** "user:password" — from an env/secret ref, never a literal in code. */
|
||||
basicAuth?: string;
|
||||
database?: string;
|
||||
timeoutMs?: number;
|
||||
/** Dimension allow-list per tenant. Absent tenant = no dimensions allowed. */
|
||||
dimensionCatalogs?: Record<string, Record<string, string>>;
|
||||
schema?: Omit<AnalyticsSchemaOptions, "database">;
|
||||
}
|
||||
|
||||
const EMPTY_CATALOG: DimensionCatalog = { resolve: () => undefined };
|
||||
|
||||
export class ClickHouseAnalyticsEngine implements SiaxAnalyticsEngine {
|
||||
readonly provider = "clickhouse";
|
||||
|
||||
/** PRESERVED pre-existing implementation, composed not replaced. */
|
||||
private readonly legacy: ClickHouseAnalytics;
|
||||
private readonly baseUrl: string;
|
||||
private readonly authHeader?: string;
|
||||
private readonly database: string;
|
||||
private readonly timeoutMs: number;
|
||||
private readonly dimensionCatalogs: Record<string, Record<string, string>>;
|
||||
private readonly schemaOpts: Omit<AnalyticsSchemaOptions, "database">;
|
||||
private readonly gate: ConcurrencyGate;
|
||||
/** query_id -> AbortController, so cancelQuery can stop the client side too. */
|
||||
private readonly inFlight = new Map<string, AbortController>();
|
||||
|
||||
constructor(opts: ClickHouseEngineOptions) {
|
||||
this.baseUrl = opts.baseUrl.replace(/\/+$/, "");
|
||||
this.authHeader = opts.basicAuth ? `Basic ${Buffer.from(opts.basicAuth).toString("base64")}` : undefined;
|
||||
this.database = opts.database ?? "opt_analytics";
|
||||
this.timeoutMs = opts.timeoutMs ?? 30_000;
|
||||
this.dimensionCatalogs = opts.dimensionCatalogs ?? {};
|
||||
this.schemaOpts = opts.schema ?? {};
|
||||
this.legacy = new ClickHouseAnalytics({
|
||||
baseUrl: opts.baseUrl,
|
||||
basicAuth: opts.basicAuth,
|
||||
database: this.database,
|
||||
timeoutMs: this.timeoutMs,
|
||||
});
|
||||
this.gate = new ConcurrencyGate((tenantId) => this.budgetForTenant(tenantId).maxConcurrentQueries);
|
||||
}
|
||||
|
||||
/** Access to the preserved implementation, for callers that still use it. */
|
||||
get legacyAnalytics(): ClickHouseAnalytics {
|
||||
return this.legacy;
|
||||
}
|
||||
|
||||
private budgetForTenant(_tenantId: string): ResourceBudget {
|
||||
return DEFAULT_BUDGETS.INTERACTIVE;
|
||||
}
|
||||
|
||||
private budgetFor(identity: AnalyticsIdentity): ResourceBudget {
|
||||
const tier = identity.budget_tier ?? "INTERACTIVE";
|
||||
return applyQuota(DEFAULT_BUDGETS[tier], identity.quota);
|
||||
}
|
||||
|
||||
private catalogFor(identity: AnalyticsIdentity): DimensionCatalog {
|
||||
const pairs = this.dimensionCatalogs[identity.tenant_id];
|
||||
return pairs ? dimensionCatalogFrom(pairs) : EMPTY_CATALOG;
|
||||
}
|
||||
|
||||
/** Fail closed: a missing/blank verified tenant is never a wildcard. */
|
||||
private assertIdentity(identity: AnalyticsIdentity): void {
|
||||
if (!identity?.tenant_id || !identity.tenant_id.trim()) {
|
||||
throw new SiaxError(
|
||||
"UNAUTHENTICATED",
|
||||
"0pt.analytics",
|
||||
"analytics requires a verified CL0UD identity carrying tenant_id",
|
||||
);
|
||||
}
|
||||
if (!identity.principal_id || !identity.principal_id.trim()) {
|
||||
throw new SiaxError("UNAUTHENTICATED", "0pt.analytics", "analytics requires a verified principal_id");
|
||||
}
|
||||
}
|
||||
|
||||
// ── schema ───────────────────────────────────────────────────────────────
|
||||
|
||||
/** Apply BI-P010 DDL (idempotent) plus the preserved legacy table. */
|
||||
async ensureSchema(): Promise<void> {
|
||||
await this.legacy.ensureSchema();
|
||||
for (const stmt of analyticsSchemaDdl({ database: this.database, ...this.schemaOpts })) {
|
||||
await this.ddl(stmt);
|
||||
}
|
||||
}
|
||||
|
||||
// ── BI-P009 capability 1: health ─────────────────────────────────────────
|
||||
|
||||
async health(): Promise<HealthReport> {
|
||||
try {
|
||||
const legacyOk = await this.legacy.health();
|
||||
if (!legacyOk) return { healthy: false, provider: this.provider, detail: "legacy health probe returned false" };
|
||||
const v = await this.raw("SELECT version() AS v FORMAT JSON", {}, {});
|
||||
const version = (v as { data?: { v?: string }[] })?.data?.[0]?.v;
|
||||
return { healthy: true, provider: this.provider, version };
|
||||
} catch (err) {
|
||||
return {
|
||||
healthy: false,
|
||||
provider: this.provider,
|
||||
detail: err instanceof SiaxError ? err.code : "unreachable",
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
// ── BI-P009 capability 2: executeMetric ──────────────────────────────────
|
||||
|
||||
async executeMetric(identity: AnalyticsIdentity, request: MetricRequest): Promise<EngineResult> {
|
||||
this.assertIdentity(identity);
|
||||
const budget = this.budgetFor(identity);
|
||||
const release = this.gate.acquire(identity.tenant_id);
|
||||
const started = Date.now();
|
||||
const queryId = this.newQueryId(identity);
|
||||
try {
|
||||
const compiled = compileMetricQuery(
|
||||
{
|
||||
tenant_id: identity.tenant_id, // FORCED from the verified identity
|
||||
project_id: identity.project_id,
|
||||
dataset_id: request.dataset_id,
|
||||
metric: request.metric,
|
||||
aggregate: request.aggregate,
|
||||
group_by: request.group_by,
|
||||
filters: request.filters,
|
||||
from_time: request.from_time,
|
||||
to_time: request.to_time,
|
||||
limit: request.limit,
|
||||
table: request.table,
|
||||
},
|
||||
this.database,
|
||||
this.catalogFor(identity),
|
||||
budget.maxResultRows,
|
||||
);
|
||||
const json = (await this.raw(compiled.sql, compiled.params, clickhouseSettings(budget), queryId)) as
|
||||
| ClickHouseJsonResponse
|
||||
| undefined;
|
||||
const groupNames = request.group_by ?? [];
|
||||
const rows = (json?.data ?? []).map((r) => {
|
||||
const groups: Record<string, string> = {};
|
||||
groupNames.forEach((name, i) => {
|
||||
const v = (r as Record<string, unknown>)[`grp${i}`];
|
||||
groups[name] = v === undefined || v === null ? "" : String(v);
|
||||
});
|
||||
return {
|
||||
...(groupNames.length ? { groups } : {}),
|
||||
value: Number((r as { value: number | string }).value),
|
||||
};
|
||||
});
|
||||
const limit = Math.max(1, Math.min(Math.floor(request.limit ?? 1000), budget.maxResultRows));
|
||||
return {
|
||||
rows,
|
||||
query_id: queryId,
|
||||
truncated: rows.length >= limit,
|
||||
stats: {
|
||||
duration_ms: Date.now() - started,
|
||||
rows_read: json?.statistics?.rows_read,
|
||||
bytes_read: json?.statistics?.bytes_read,
|
||||
result_rows: rows.length,
|
||||
},
|
||||
};
|
||||
} finally {
|
||||
release();
|
||||
this.inFlight.delete(queryId);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Delegating path to the PRESERVED implementation for the pre-existing
|
||||
* `opt_metric_events` table. Kept so no existing caller has to migrate and
|
||||
* the already-proven forced-predicate path stays in service.
|
||||
*/
|
||||
async executeLegacyMetric(
|
||||
identity: AnalyticsIdentity,
|
||||
request: { metric: string; aggregate: "sum" | "avg" | "min" | "max" | "count"; dataset_id?: string; group_by?: string; from_time?: string; to_time?: string; limit?: number },
|
||||
): Promise<EngineResult> {
|
||||
this.assertIdentity(identity);
|
||||
const started = Date.now();
|
||||
const rows = await this.legacy.queryMetric({
|
||||
tenant_id: identity.tenant_id, // FORCED from the verified identity
|
||||
metric: request.metric,
|
||||
aggregate: request.aggregate,
|
||||
dataset_id: request.dataset_id,
|
||||
group_by: request.group_by,
|
||||
from_time: request.from_time,
|
||||
to_time: request.to_time,
|
||||
limit: request.limit,
|
||||
});
|
||||
return {
|
||||
rows: rows.map((r) => ({
|
||||
...(r.group !== undefined && request.group_by ? { groups: { [request.group_by]: r.group } } : {}),
|
||||
value: r.value,
|
||||
})),
|
||||
query_id: this.newQueryId(identity),
|
||||
truncated: false,
|
||||
stats: { duration_ms: Date.now() - started, result_rows: rows.length },
|
||||
};
|
||||
}
|
||||
|
||||
// ── BI-P009 capability 3: executeQuery (analyst SQL) ─────────────────────
|
||||
|
||||
async executeQuery(identity: AnalyticsIdentity, _request: AnalystQueryRequest): Promise<EngineResult> {
|
||||
this.assertIdentity(identity);
|
||||
// Deliberate, documented refusal — see AnalystQueryRequest's doc comment.
|
||||
// Opening a raw-SQL surface before BI-P017 (SQL dialect contract) and
|
||||
// BI-P033 (analytics authorization) exist would create precisely the
|
||||
// "raw SQL attempt" hole BI-P011's adversarial list enumerates.
|
||||
throw new SiaxError(
|
||||
"CAPABILITY_UNAVAILABLE",
|
||||
"0pt.analytics",
|
||||
"analyst SQL is not enabled on the ClickHouse provider until BI-P017/BI-P033 land",
|
||||
);
|
||||
}
|
||||
|
||||
// ── BI-P009 capability 4: estimateQuery ──────────────────────────────────
|
||||
|
||||
async estimateQuery(identity: AnalyticsIdentity, request: MetricRequest): Promise<QueryEstimate> {
|
||||
this.assertIdentity(identity);
|
||||
const budget = this.budgetFor(identity);
|
||||
const compiled = compileMetricQuery(
|
||||
{
|
||||
tenant_id: identity.tenant_id,
|
||||
project_id: identity.project_id,
|
||||
dataset_id: request.dataset_id,
|
||||
metric: request.metric,
|
||||
aggregate: request.aggregate,
|
||||
group_by: request.group_by,
|
||||
filters: request.filters,
|
||||
from_time: request.from_time,
|
||||
to_time: request.to_time,
|
||||
limit: request.limit,
|
||||
table: request.table,
|
||||
},
|
||||
this.database,
|
||||
this.catalogFor(identity),
|
||||
budget.maxResultRows,
|
||||
);
|
||||
// `EXPLAIN ESTIMATE` reports the parts/rows/marks the planner will read
|
||||
// AFTER primary-key pruning — which is why tenant_id leads every ORDER BY
|
||||
// in schema.ts: the estimate for one tenant does not include everyone.
|
||||
const estimateSql = compiled.sql.replace(/ FORMAT JSON$/, "");
|
||||
let rowsToRead = 0;
|
||||
let partsRead = 0;
|
||||
try {
|
||||
const res = (await this.raw(
|
||||
`EXPLAIN ESTIMATE ${estimateSql} FORMAT JSON`,
|
||||
compiled.params,
|
||||
planningSettings(budget),
|
||||
)) as { data?: { rows?: number | string; parts?: number | string }[] } | undefined;
|
||||
for (const row of res?.data ?? []) {
|
||||
rowsToRead += Number(row.rows ?? 0);
|
||||
partsRead += Number(row.parts ?? 0);
|
||||
}
|
||||
} catch (err) {
|
||||
// An estimate that cannot be produced must not read as "cheap".
|
||||
throw err instanceof SiaxError
|
||||
? err
|
||||
: new SiaxError("DEPENDENCY_UNAVAILABLE", "clickhouse", "could not estimate query");
|
||||
}
|
||||
// Rough byte estimate: ClickHouse's ESTIMATE reports rows, not bytes.
|
||||
// 128 B/row is a deliberately PESSIMISTIC constant for these tables
|
||||
// (measured mean in the BI-P010 benchmark is lower), so the gate errs
|
||||
// toward refusing rather than admitting an oversized query.
|
||||
const bytesToRead = rowsToRead * 128;
|
||||
const within = rowsToRead <= budget.maxRowsToRead && bytesToRead <= budget.maxBytesToRead;
|
||||
return {
|
||||
estimated_rows_to_read: rowsToRead,
|
||||
estimated_bytes_to_read: bytesToRead,
|
||||
within_budget: within,
|
||||
budget,
|
||||
...(within
|
||||
? {}
|
||||
: {
|
||||
reason:
|
||||
`estimate ${rowsToRead} rows / ${bytesToRead} bytes over budget ` +
|
||||
`(${budget.maxRowsToRead} rows / ${budget.maxBytesToRead} bytes), parts=${partsRead}`,
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
// ── BI-P009 capability 5: cancelQuery ────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Two-sided cancellation. The client-side abort stops us waiting; the
|
||||
* server-side KILL QUERY stops ClickHouse burning CPU. Cancelling is
|
||||
* tenant-scoped: the query id embeds the tenant and is checked, so tenant A
|
||||
* cannot cancel tenant B's work.
|
||||
*/
|
||||
async cancelQuery(identity: AnalyticsIdentity, queryId: string): Promise<boolean> {
|
||||
this.assertIdentity(identity);
|
||||
if (!queryId.startsWith(this.queryIdPrefix(identity))) {
|
||||
throw new SiaxError("AUTHORIZATION_DENIED", "0pt.analytics", "query does not belong to this tenant");
|
||||
}
|
||||
this.inFlight.get(queryId)?.abort();
|
||||
this.inFlight.delete(queryId);
|
||||
try {
|
||||
// KILL QUERY is a mutation of server state, so it cannot ride the
|
||||
// readonly=1 settings the SELECT path uses.
|
||||
await this.raw(
|
||||
`KILL QUERY WHERE query_id = {killId:String} SYNC FORMAT JSON`,
|
||||
{ param_killId: queryId },
|
||||
{ max_execution_time: "10" },
|
||||
);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
// ── BI-P009 capability 6: explain ────────────────────────────────────────
|
||||
|
||||
async explain(identity: AnalyticsIdentity, request: MetricRequest): Promise<QueryExplanation> {
|
||||
this.assertIdentity(identity);
|
||||
const budget = this.budgetFor(identity);
|
||||
const compiled = compileMetricQuery(
|
||||
{
|
||||
tenant_id: identity.tenant_id,
|
||||
project_id: identity.project_id,
|
||||
dataset_id: request.dataset_id,
|
||||
metric: request.metric,
|
||||
aggregate: request.aggregate,
|
||||
group_by: request.group_by,
|
||||
filters: request.filters,
|
||||
from_time: request.from_time,
|
||||
to_time: request.to_time,
|
||||
limit: request.limit,
|
||||
table: request.table,
|
||||
},
|
||||
this.database,
|
||||
this.catalogFor(identity),
|
||||
budget.maxResultRows,
|
||||
);
|
||||
const sql = compiled.sql.replace(/ FORMAT JSON$/, "");
|
||||
const res = (await this.raw(
|
||||
`EXPLAIN PLAN indexes = 1 ${sql} FORMAT TSVRaw`,
|
||||
compiled.params,
|
||||
planningSettings(budget),
|
||||
)) as string | undefined;
|
||||
return {
|
||||
plan: typeof res === "string" ? res : JSON.stringify(res ?? ""),
|
||||
forced_tenant_predicate: FORCED_TENANT_PREDICATE,
|
||||
table: compiled.table,
|
||||
};
|
||||
}
|
||||
|
||||
// ── BI-P009 capability 7: ingestProjection ───────────────────────────────
|
||||
|
||||
async ingestProjection(
|
||||
identity: AnalyticsIdentity,
|
||||
request: ProjectionIngestRequest,
|
||||
): Promise<{ ingested: number }> {
|
||||
this.assertIdentity(identity);
|
||||
if (request.rows.length === 0) return { ingested: 0 };
|
||||
const now = new Date();
|
||||
const version = Math.floor(now.getTime());
|
||||
const body = request.rows
|
||||
.map((r) => {
|
||||
if (!r.entity_key || !r.entity_key.trim()) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "every projection row needs an entity_key");
|
||||
}
|
||||
return JSON.stringify({
|
||||
// tenant/project come from the VERIFIED identity, never the row —
|
||||
// a row-supplied tenant_id is the INF0 Wave-2 bug class.
|
||||
tenant_id: identity.tenant_id,
|
||||
project_id: identity.project_id ?? "",
|
||||
dataset_id: request.dataset_id,
|
||||
entity_key: r.entity_key,
|
||||
as_of: (r.as_of ?? now.toISOString()).replace("T", " ").replace(/Z$/, ""),
|
||||
version: r.version ?? version,
|
||||
attributes: r.attributes ?? {},
|
||||
measures: r.measures ?? {},
|
||||
});
|
||||
})
|
||||
.join("\n");
|
||||
await this.raw(
|
||||
`INSERT INTO ${this.database}.opt_dataset_projection FORMAT JSONEachRow`,
|
||||
{},
|
||||
{},
|
||||
undefined,
|
||||
body,
|
||||
);
|
||||
return { ingested: request.rows.length };
|
||||
}
|
||||
|
||||
// ── BI-P009 capability 8: refreshProjection ──────────────────────────────
|
||||
|
||||
/**
|
||||
* ReplacingMergeTree deduplicates on a background merge, so a read straight
|
||||
* after an ingest can still see superseded rows. `OPTIMIZE ... FINAL` forces
|
||||
* the merge for THIS tenant's partition only — the table is
|
||||
* `PARTITION BY (tenant_id)` precisely so one tenant's refresh does not
|
||||
* rewrite another tenant's parts (BI-P012's "without degrading unrelated
|
||||
* tenant" applies to maintenance too, not only to queries).
|
||||
*/
|
||||
async refreshProjection(identity: AnalyticsIdentity, dataset_id: string): Promise<ProjectionRefreshResult> {
|
||||
this.assertIdentity(identity);
|
||||
await this.raw(
|
||||
`OPTIMIZE TABLE ${this.database}.opt_dataset_projection PARTITION {tenant:String} FINAL`,
|
||||
{ param_tenant: identity.tenant_id },
|
||||
{ max_execution_time: "120" },
|
||||
);
|
||||
const compiled = compileProjectionQuery(
|
||||
{ tenant_id: identity.tenant_id, project_id: identity.project_id, dataset_id, limit: 1 },
|
||||
this.database,
|
||||
1,
|
||||
);
|
||||
const countSql =
|
||||
`SELECT count() AS value FROM ${this.database}.opt_dataset_projection FINAL ` +
|
||||
`WHERE ${FORCED_TENANT_PREDICATE} AND dataset_id = {dataset:String} FORMAT JSON`;
|
||||
const res = (await this.raw(countSql, compiled.params, clickhouseSettings(this.budgetFor(identity)))) as
|
||||
| { data?: { value: number | string }[] }
|
||||
| undefined;
|
||||
return {
|
||||
dataset_id,
|
||||
row_count: Number(res?.data?.[0]?.value ?? 0),
|
||||
refreshed_at: new Date().toISOString(),
|
||||
};
|
||||
}
|
||||
|
||||
// ── transport ────────────────────────────────────────────────────────────
|
||||
|
||||
private queryIdPrefix(identity: AnalyticsIdentity): string {
|
||||
return `opt_${crypto.createHash("sha256").update(identity.tenant_id).digest("hex").slice(0, 12)}_`;
|
||||
}
|
||||
|
||||
private newQueryId(identity: AnalyticsIdentity): string {
|
||||
return `${this.queryIdPrefix(identity)}${crypto.randomUUID().replace(/-/g, "")}`;
|
||||
}
|
||||
|
||||
/** DDL path: no readonly, no result parsing beyond errors. */
|
||||
private async ddl(sql: string): Promise<void> {
|
||||
await this.raw(sql, {}, { max_execution_time: "60" });
|
||||
}
|
||||
|
||||
private async raw(
|
||||
sql: string,
|
||||
params: Record<string, string>,
|
||||
settings: Record<string, string>,
|
||||
queryId?: string,
|
||||
body?: string,
|
||||
): Promise<unknown> {
|
||||
const url = new URL(this.baseUrl);
|
||||
url.searchParams.set("query", sql);
|
||||
for (const [k, v] of Object.entries(settings)) url.searchParams.set(k, v);
|
||||
for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
|
||||
if (queryId) url.searchParams.set("query_id", queryId);
|
||||
|
||||
const headers: Record<string, string> = {};
|
||||
if (this.authHeader) headers["Authorization"] = this.authHeader;
|
||||
|
||||
const controller = new AbortController();
|
||||
if (queryId) this.inFlight.set(queryId, controller);
|
||||
const timeout = setTimeout(() => controller.abort(), this.timeoutMs);
|
||||
|
||||
let res: Response;
|
||||
try {
|
||||
res = await fetch(url.toString(), {
|
||||
method: "POST",
|
||||
headers,
|
||||
body: body ?? "",
|
||||
signal: controller.signal,
|
||||
});
|
||||
} catch (cause) {
|
||||
throw new SiaxError("DEPENDENCY_UNAVAILABLE", "clickhouse", "clickhouse unreachable", { cause });
|
||||
} finally {
|
||||
clearTimeout(timeout);
|
||||
}
|
||||
const text = await res.text();
|
||||
if (!res.ok) {
|
||||
// ClickHouse error text can echo the query; keep only the numeric code
|
||||
// so a caller never receives another tenant's identifiers back.
|
||||
const code = /Code: (\d+)/.exec(text)?.[1] ?? String(res.status);
|
||||
throw new SiaxError(
|
||||
siaxErrorCodeForUpstreamStatus(res.status),
|
||||
"clickhouse",
|
||||
`query failed (CH code ${code})`,
|
||||
);
|
||||
}
|
||||
if (!text) return undefined;
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(text);
|
||||
} catch {
|
||||
return text;
|
||||
}
|
||||
// DEFENCE IN DEPTH against a partial result that looks complete.
|
||||
// ClickHouse normally answers a breached limit with an HTTP error status
|
||||
// (measured on 24.8: 408 for max_execution_time, 500 for
|
||||
// max_rows_to_read), which `res.ok` above already catches. But when rows
|
||||
// have already been flushed, the JSON body can carry an `exception` key
|
||||
// under a 200 — a truncated answer indistinguishable from a whole one.
|
||||
// Treat any body carrying an exception as a failure, whatever the status.
|
||||
if (parsed && typeof parsed === "object" && "exception" in parsed) {
|
||||
const ex = String((parsed as { exception?: unknown }).exception ?? "");
|
||||
const code = /Code: (\d+)/.exec(ex)?.[1] ?? "unknown";
|
||||
throw new SiaxError("PERMANENT_FAILURE", "clickhouse", `query failed mid-stream (CH code ${code})`);
|
||||
}
|
||||
return parsed;
|
||||
}
|
||||
}
|
||||
|
||||
interface ClickHouseJsonResponse {
|
||||
data?: Record<string, unknown>[];
|
||||
statistics?: { rows_read?: number; bytes_read?: number; elapsed?: number };
|
||||
}
|
||||
@@ -0,0 +1,304 @@
|
||||
/**
|
||||
* src/analytics/engine/query-compiler.ts
|
||||
* =======================================
|
||||
* BI-P011 — CLICKHOUSE TENANT ISOLATION, layer 2:
|
||||
* "`0pt` query compiler forces tenant".
|
||||
*
|
||||
* The compiler is the ONLY thing in this repo that turns an analytics request
|
||||
* into SQL text. Its contract:
|
||||
*
|
||||
* * `tenant_id` is not an argument the caller may omit — it comes from the
|
||||
* VERIFIED CL0UD identity (layer 1) and is written into the WHERE clause
|
||||
* by this function, unconditionally, as a bound parameter.
|
||||
* * caller-supplied VALUES are never spliced into SQL text. They travel as
|
||||
* ClickHouse HTTP `param_*` query parameters.
|
||||
* * caller-supplied IDENTIFIERS (a dimension name, a group-by column) are
|
||||
* never spliced either. They are RESOLVED against a declared allow-list
|
||||
* of `DimensionDefinition`s and only the resolved, already-known physical
|
||||
* name reaches the SQL — so "dimension identifier injection" has no
|
||||
* surface, because caller text never becomes an identifier.
|
||||
* * there is no raw-SQL entry point. `compileMetricQuery` cannot express a
|
||||
* UNION, a subquery, or a second table, so a caller cannot reach a table
|
||||
* the compiler did not choose.
|
||||
*
|
||||
* WHY A SEPARATE MODULE from clickhouse-analytics.ts: that file's compiler
|
||||
* already forces the tenant predicate correctly and BI-P009 says PRESERVE IT.
|
||||
* This module is the generalised compiler for the BI-P010 tables; it repeats
|
||||
* the guarantee rather than replacing it, and
|
||||
* tests/analytics/tenant-predicate-regression.test.ts asserts BOTH.
|
||||
*/
|
||||
import { SiaxError } from "../../platform/errors";
|
||||
import type { AnalyticsTable } from "./schema";
|
||||
|
||||
/** A physical identifier that has been VALIDATED, not merely received. */
|
||||
const IDENT_RE = /^[A-Za-z_][A-Za-z0-9_]{0,62}$/;
|
||||
|
||||
export function assertSafeIdentifier(value: string, what: string): string {
|
||||
if (!IDENT_RE.test(value)) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", `invalid ${what} identifier`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
export const METRIC_AGGREGATES = ["sum", "avg", "min", "max", "count", "count_distinct"] as const;
|
||||
export type MetricAggregate = (typeof METRIC_AGGREGATES)[number];
|
||||
|
||||
/**
|
||||
* The dimensions a caller is allowed to name, resolved from
|
||||
* `DimensionDefinition` rows. The compiler looks a caller's dimension name up
|
||||
* HERE; a miss is a rejection, never a pass-through.
|
||||
*/
|
||||
export interface DimensionCatalog {
|
||||
/** caller-facing dimension name -> physical dimension key in dim_names. */
|
||||
resolve(name: string): string | undefined;
|
||||
}
|
||||
|
||||
export function dimensionCatalogFrom(pairs: Record<string, string>): DimensionCatalog {
|
||||
const map = new Map(Object.entries(pairs));
|
||||
return {
|
||||
resolve(name: string): string | undefined {
|
||||
// Own-property lookup on a Map — no prototype-chain surprises
|
||||
// ("constructor", "__proto__" resolve to undefined, as they must).
|
||||
return map.get(name);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export interface MetricQueryRequest {
|
||||
/** Supplied by the engine from the verified identity, NOT by the caller. */
|
||||
tenant_id: string;
|
||||
project_id?: string;
|
||||
dataset_id?: string;
|
||||
metric: string;
|
||||
aggregate: MetricAggregate;
|
||||
/** Caller-facing dimension names; resolved through the catalog. */
|
||||
group_by?: string[];
|
||||
/** Equality filters on dimensions: caller-facing name -> value. */
|
||||
filters?: Record<string, string>;
|
||||
from_time?: string;
|
||||
to_time?: string;
|
||||
limit?: number;
|
||||
/** Which BI-P010 table to read. Chosen by the engine, not the caller. */
|
||||
table?: AnalyticsTable;
|
||||
}
|
||||
|
||||
export interface CompiledQuery {
|
||||
sql: string;
|
||||
/** ClickHouse HTTP `param_*` entries. Values NEVER appear in `sql`. */
|
||||
params: Record<string, string>;
|
||||
/** The predicate the compiler forced, for assertion in tests. */
|
||||
forcedTenantPredicate: string;
|
||||
table: AnalyticsTable;
|
||||
}
|
||||
|
||||
/**
|
||||
* The exact predicate text every compiled query must contain. Exported so a
|
||||
* regression test can assert on the literal string — if someone deletes the
|
||||
* forcing, the test fails on a missing substring rather than on a subtle
|
||||
* behavioural difference that a mocked engine would happily hide.
|
||||
*/
|
||||
export const FORCED_TENANT_PREDICATE = "tenant_id = {tenant:String}";
|
||||
|
||||
const AGG_SQL: Record<MetricAggregate, string> = {
|
||||
sum: "sum(value)",
|
||||
avg: "avg(value)",
|
||||
min: "min(value)",
|
||||
max: "max(value)",
|
||||
count: "count()",
|
||||
count_distinct: "uniqExact(event_id)",
|
||||
};
|
||||
|
||||
const MAX_GROUP_BY = 8;
|
||||
|
||||
/**
|
||||
* Compile a closed-DSL metric request into tenant-forced SQL.
|
||||
*
|
||||
* @param req the request; `tenant_id` MUST already be the verified identity.
|
||||
* @param db the analytics database name (engine-controlled, validated).
|
||||
* @param catalog the tenant's declared dimensions.
|
||||
* @param maxResultRows the governance ceiling (BI-P012); `limit` is clamped.
|
||||
*/
|
||||
export function compileMetricQuery(
|
||||
req: MetricQueryRequest,
|
||||
db: string,
|
||||
catalog: DimensionCatalog,
|
||||
maxResultRows: number,
|
||||
): CompiledQuery {
|
||||
if (!req.tenant_id || !req.tenant_id.trim()) {
|
||||
// Fail closed. A blank tenant must never compile to "no predicate".
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "tenant_id is required on every analytics query");
|
||||
}
|
||||
assertSafeIdentifier(db, "database");
|
||||
const table: AnalyticsTable = req.table ?? "opt_analytics_event";
|
||||
assertSafeIdentifier(table, "table");
|
||||
if (!METRIC_AGGREGATES.includes(req.aggregate)) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "unsupported aggregate");
|
||||
}
|
||||
|
||||
const params: Record<string, string> = {
|
||||
param_tenant: req.tenant_id,
|
||||
param_metric: req.metric,
|
||||
};
|
||||
|
||||
// ── the forced predicate. First condition, always, no branch. ─────────
|
||||
const conditions: string[] = [FORCED_TENANT_PREDICATE, "metric = {metric:String}"];
|
||||
|
||||
if (req.project_id) {
|
||||
conditions.push("project_id = {project:String}");
|
||||
params.param_project = req.project_id;
|
||||
}
|
||||
if (req.dataset_id) {
|
||||
conditions.push("dataset_id = {dataset:String}");
|
||||
params.param_dataset = req.dataset_id;
|
||||
}
|
||||
if (req.from_time) {
|
||||
conditions.push("event_time >= parseDateTime64BestEffort({fromTime:String})");
|
||||
params.param_fromTime = req.from_time;
|
||||
}
|
||||
if (req.to_time) {
|
||||
conditions.push("event_time < parseDateTime64BestEffort({toTime:String})");
|
||||
params.param_toTime = req.to_time;
|
||||
}
|
||||
|
||||
// ── filters: names resolved through the catalog, values parameterized ──
|
||||
let filterIdx = 0;
|
||||
for (const [name, value] of Object.entries(req.filters ?? {})) {
|
||||
const physical = catalog.resolve(name);
|
||||
if (physical === undefined) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", `unknown filter dimension "${name}"`);
|
||||
}
|
||||
// `physical` came from the catalog, but validate anyway: a corrupted or
|
||||
// hostile catalog row must not become an injection vector either.
|
||||
assertSafeIdentifier(physical, "dimension");
|
||||
const nameKey = `f${filterIdx}n`;
|
||||
const valKey = `f${filterIdx}v`;
|
||||
conditions.push(
|
||||
`dim_values[indexOf(dim_names, {${nameKey}:String})] = {${valKey}:String}`,
|
||||
);
|
||||
params[`param_${nameKey}`] = physical;
|
||||
params[`param_${valKey}`] = value;
|
||||
filterIdx += 1;
|
||||
}
|
||||
|
||||
// ── group by: same resolution, plus a hard cardinality ceiling ─────────
|
||||
const groupBy = req.group_by ?? [];
|
||||
if (groupBy.length > MAX_GROUP_BY) {
|
||||
throw new SiaxError(
|
||||
"INVALID_REQUEST",
|
||||
"0pt.analytics",
|
||||
`at most ${MAX_GROUP_BY} group-by dimensions (got ${groupBy.length})`,
|
||||
);
|
||||
}
|
||||
const selectParts: string[] = [];
|
||||
const groupParts: string[] = [];
|
||||
groupBy.forEach((name, i) => {
|
||||
const physical = catalog.resolve(name);
|
||||
if (physical === undefined) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", `unknown group_by dimension "${name}"`);
|
||||
}
|
||||
assertSafeIdentifier(physical, "dimension");
|
||||
const key = `g${i}`;
|
||||
// The ALIAS is compiler-generated (`grp0`, `grp1`), never caller text.
|
||||
selectParts.push(`dim_values[indexOf(dim_names, {${key}:String})] AS grp${i}`);
|
||||
groupParts.push(`grp${i}`);
|
||||
params[`param_${key}`] = physical;
|
||||
});
|
||||
|
||||
const limit = Math.max(1, Math.min(Math.floor(req.limit ?? 1000), maxResultRows));
|
||||
|
||||
const sql =
|
||||
`SELECT ${selectParts.length ? selectParts.join(", ") + ", " : ""}${AGG_SQL[req.aggregate]} AS value ` +
|
||||
`FROM ${db}.${table} ` +
|
||||
`WHERE ${conditions.join(" AND ")} ` +
|
||||
(groupParts.length ? `GROUP BY ${groupParts.join(", ")} ORDER BY value DESC ` : "") +
|
||||
`LIMIT ${limit} FORMAT JSON`;
|
||||
|
||||
return { sql, params, forcedTenantPredicate: FORCED_TENANT_PREDICATE, table };
|
||||
}
|
||||
|
||||
/**
|
||||
* Compile a projection lookup (SERVING tier, BI-P010's opt_dataset_projection).
|
||||
* Same forcing, different table and key.
|
||||
*/
|
||||
export function compileProjectionQuery(
|
||||
req: { tenant_id: string; project_id?: string; dataset_id: string; entity_keys?: string[]; limit?: number },
|
||||
db: string,
|
||||
maxResultRows: number,
|
||||
): CompiledQuery {
|
||||
if (!req.tenant_id || !req.tenant_id.trim()) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "tenant_id is required on every analytics query");
|
||||
}
|
||||
assertSafeIdentifier(db, "database");
|
||||
const params: Record<string, string> = {
|
||||
param_tenant: req.tenant_id,
|
||||
param_dataset: req.dataset_id,
|
||||
};
|
||||
const conditions = [FORCED_TENANT_PREDICATE, "dataset_id = {dataset:String}"];
|
||||
if (req.project_id) {
|
||||
conditions.push("project_id = {project:String}");
|
||||
params.param_project = req.project_id;
|
||||
}
|
||||
if (req.entity_keys && req.entity_keys.length > 0) {
|
||||
conditions.push("has({entityKeys:Array(String)}, entity_key)");
|
||||
params.param_entityKeys = JSON.stringify(req.entity_keys);
|
||||
}
|
||||
const limit = Math.max(1, Math.min(Math.floor(req.limit ?? 1000), maxResultRows));
|
||||
const sql =
|
||||
`SELECT entity_key, as_of, version, attributes, measures ` +
|
||||
`FROM ${db}.opt_dataset_projection ` +
|
||||
`WHERE ${conditions.join(" AND ")} ` +
|
||||
`ORDER BY entity_key LIMIT ${limit} FORMAT JSON`;
|
||||
return { sql, params, forcedTenantPredicate: FORCED_TENANT_PREDICATE, table: "opt_dataset_projection" };
|
||||
}
|
||||
|
||||
/**
|
||||
* BI-P011 layer-3 bootstrap: the DDL for a RESTRICTED per-tenant ClickHouse
|
||||
* user with a row policy pinning `tenant_id`.
|
||||
*
|
||||
* This is the layer that survives a compiler bug. The compiler forcing the
|
||||
* predicate is layer 2; if a future refactor drops it, layer 3 still returns
|
||||
* zero foreign rows because the SERVER filters them. The BI-P011 proof runs
|
||||
* the adversarial suite twice — once through an unrestricted user (hole open)
|
||||
* and once through this restricted user (hole closed) — because a claim of
|
||||
* defence in depth that has never been tested with one layer removed is not
|
||||
* evidence.
|
||||
*
|
||||
* `tenantId` is validated as an identifier-safe token AND embedded as a
|
||||
* quoted literal; ClickHouse's CREATE ROW POLICY takes no bound parameters,
|
||||
* so validation is the only protection available and it is applied strictly.
|
||||
*/
|
||||
export function restrictedTenantUserDdl(args: {
|
||||
db: string;
|
||||
tenantId: string;
|
||||
userName: string;
|
||||
password: string;
|
||||
tables: readonly string[];
|
||||
settingsProfile?: string;
|
||||
}): string[] {
|
||||
assertSafeIdentifier(args.db, "database");
|
||||
assertSafeIdentifier(args.userName, "user");
|
||||
// A tenant id is `t_<slug>` in this estate; enforce that shape strictly
|
||||
// rather than trusting the caller.
|
||||
if (!/^[A-Za-z0-9_-]{1,64}$/.test(args.tenantId)) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "tenant id is not policy-safe");
|
||||
}
|
||||
if (args.password.includes("'") || args.password.includes("\\")) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.analytics", "password contains characters that cannot be embedded");
|
||||
}
|
||||
const profile = args.settingsProfile;
|
||||
if (profile) assertSafeIdentifier(profile, "settings profile");
|
||||
|
||||
const stmts: string[] = [
|
||||
`CREATE USER IF NOT EXISTS ${args.userName} IDENTIFIED WITH sha256_password BY '${args.password}'` +
|
||||
(profile ? ` SETTINGS PROFILE ${profile}` : ` SETTINGS readonly = 1`),
|
||||
];
|
||||
for (const table of args.tables) {
|
||||
assertSafeIdentifier(table, "table");
|
||||
stmts.push(
|
||||
`CREATE ROW POLICY IF NOT EXISTS ${args.userName}_${table}_rp ON ${args.db}.${table} ` +
|
||||
`FOR SELECT USING tenant_id = '${args.tenantId}' TO ${args.userName}`,
|
||||
);
|
||||
stmts.push(`GRANT SELECT ON ${args.db}.${table} TO ${args.userName}`);
|
||||
}
|
||||
return stmts;
|
||||
}
|
||||
@@ -0,0 +1,222 @@
|
||||
/**
|
||||
* src/analytics/engine/resource-governance.ts
|
||||
* ============================================
|
||||
* BI-P012 — CLICKHOUSE RESOURCE GOVERNANCE.
|
||||
*
|
||||
* Every setting BI-P012 names, in one place, with a value:
|
||||
*
|
||||
* max_execution_time max_result_rows max_rows_to_read
|
||||
* max_bytes_to_read max_memory_usage max_concurrent_queries
|
||||
*
|
||||
* Three layers, and the order matters:
|
||||
*
|
||||
* 1. SERVER-SIDE SETTINGS travel with every statement on the HTTP
|
||||
* interface. ClickHouse enforces them; the client cannot opt out
|
||||
* because the client is what writes them. `readonly=1` on top means a
|
||||
* SELECT cannot re-set them mid-query (readonly=1 forbids SETTINGS
|
||||
* changes as well as writes — that is the whole reason it is here and
|
||||
* not merely `readonly=2`).
|
||||
* 2. `max_concurrent_queries` is NOT a per-query setting in ClickHouse; it
|
||||
* is a per-USER quota. `concurrencySlotSql` renders the CREATE
|
||||
* SETTINGS PROFILE that carries it, applied to the tenant's restricted
|
||||
* user (BI-P011 layer 3) so one tenant's fan-out cannot consume the
|
||||
* whole server.
|
||||
* 3. A local admission gate (`ConcurrencyGate`) refuses the N+1st
|
||||
* in-flight query per tenant before it ever reaches the network. Two
|
||||
* layers because the ClickHouse quota is per-user and one 0pt process
|
||||
* can still queue work faster than the server rejects it.
|
||||
*
|
||||
* Budget tiers exist because "one limit for everyone" is how a small tenant
|
||||
* subsidises a query bomb. `INTERACTIVE` is the dashboard default and is
|
||||
* deliberately mean.
|
||||
*/
|
||||
import { SiaxError } from "../../platform/errors";
|
||||
|
||||
export interface ResourceBudget {
|
||||
/** Seconds. ClickHouse `max_execution_time`. */
|
||||
maxExecutionTimeSec: number;
|
||||
/** Rows returned to the caller. ClickHouse `max_result_rows`. */
|
||||
maxResultRows: number;
|
||||
/** Rows SCANNED (not returned). ClickHouse `max_rows_to_read`. */
|
||||
maxRowsToRead: number;
|
||||
/** Bytes scanned. ClickHouse `max_bytes_to_read`. */
|
||||
maxBytesToRead: number;
|
||||
/** Bytes of RAM per query. ClickHouse `max_memory_usage`. */
|
||||
maxMemoryUsageBytes: number;
|
||||
/** In-flight queries per tenant. Enforced by quota + local gate. */
|
||||
maxConcurrentQueries: number;
|
||||
}
|
||||
|
||||
export const BUDGET_TIERS = ["INTERACTIVE", "ANALYTICAL", "BATCH"] as const;
|
||||
export type BudgetTier = (typeof BUDGET_TIERS)[number];
|
||||
|
||||
/**
|
||||
* Defaults. Chosen against the shape of the analytics tables in
|
||||
* `src/analytics/engine/schema.ts`, not plucked from the air:
|
||||
* INTERACTIVE must answer a dashboard panel inside a human's patience budget,
|
||||
* so it caps the SCAN, which is what actually costs time on a MergeTree.
|
||||
*/
|
||||
export const DEFAULT_BUDGETS: Record<BudgetTier, ResourceBudget> = {
|
||||
INTERACTIVE: {
|
||||
maxExecutionTimeSec: 10,
|
||||
maxResultRows: 10_000,
|
||||
maxRowsToRead: 50_000_000,
|
||||
maxBytesToRead: 2 * 1024 * 1024 * 1024,
|
||||
maxMemoryUsageBytes: 2 * 1024 * 1024 * 1024,
|
||||
maxConcurrentQueries: 8,
|
||||
},
|
||||
ANALYTICAL: {
|
||||
maxExecutionTimeSec: 60,
|
||||
maxResultRows: 500_000,
|
||||
maxRowsToRead: 2_000_000_000,
|
||||
maxBytesToRead: 32 * 1024 * 1024 * 1024,
|
||||
maxMemoryUsageBytes: 8 * 1024 * 1024 * 1024,
|
||||
maxConcurrentQueries: 4,
|
||||
},
|
||||
BATCH: {
|
||||
maxExecutionTimeSec: 900,
|
||||
maxResultRows: 10_000_000,
|
||||
maxRowsToRead: 50_000_000_000,
|
||||
maxBytesToRead: 512 * 1024 * 1024 * 1024,
|
||||
maxMemoryUsageBytes: 24 * 1024 * 1024 * 1024,
|
||||
maxConcurrentQueries: 2,
|
||||
},
|
||||
};
|
||||
|
||||
/**
|
||||
* A CL0UD-issued quota may only ever TIGHTEN a tier, never loosen it. A
|
||||
* control-plane bug that hands out a generous number must not be able to
|
||||
* disarm the engine's own ceiling.
|
||||
*/
|
||||
export function applyQuota(base: ResourceBudget, quota: Partial<ResourceBudget> | undefined): ResourceBudget {
|
||||
if (!quota) return { ...base };
|
||||
const pick = (b: number, q: number | undefined): number =>
|
||||
typeof q === "number" && Number.isFinite(q) && q > 0 ? Math.min(b, q) : b;
|
||||
return {
|
||||
maxExecutionTimeSec: pick(base.maxExecutionTimeSec, quota.maxExecutionTimeSec),
|
||||
maxResultRows: pick(base.maxResultRows, quota.maxResultRows),
|
||||
maxRowsToRead: pick(base.maxRowsToRead, quota.maxRowsToRead),
|
||||
maxBytesToRead: pick(base.maxBytesToRead, quota.maxBytesToRead),
|
||||
maxMemoryUsageBytes: pick(base.maxMemoryUsageBytes, quota.maxMemoryUsageBytes),
|
||||
maxConcurrentQueries: pick(base.maxConcurrentQueries, quota.maxConcurrentQueries),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The settings that ride along on the HTTP request. `readonly: "1"` is what
|
||||
* makes them un-overridable from inside the query text.
|
||||
*
|
||||
* `timeout_overflow_mode` / `result_overflow_mode` / `read_overflow_mode` are
|
||||
* set to `throw` EXPLICITLY: ClickHouse's default for some of these is
|
||||
* `break`, which silently returns a PARTIAL result. A partial result that
|
||||
* looks complete is the analytics equivalent of a false green, so the engine
|
||||
* refuses it.
|
||||
*/
|
||||
export function clickhouseSettings(budget: ResourceBudget): Record<string, string> {
|
||||
return {
|
||||
readonly: "1",
|
||||
max_execution_time: String(budget.maxExecutionTimeSec),
|
||||
timeout_overflow_mode: "throw",
|
||||
max_result_rows: String(budget.maxResultRows),
|
||||
result_overflow_mode: "throw",
|
||||
max_rows_to_read: String(budget.maxRowsToRead),
|
||||
max_bytes_to_read: String(budget.maxBytesToRead),
|
||||
read_overflow_mode: "throw",
|
||||
max_memory_usage: String(budget.maxMemoryUsageBytes),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Settings for a PLANNER call (`EXPLAIN ESTIMATE`, `EXPLAIN PLAN`).
|
||||
*
|
||||
* Measured, not assumed: running `EXPLAIN ESTIMATE` under the caller's own
|
||||
* `max_rows_to_read` makes ClickHouse refuse the ESTIMATE itself with code 158
|
||||
* whenever the estimate would be over budget — i.e. exactly the case the
|
||||
* estimate exists to detect. An estimator that cannot describe an expensive
|
||||
* query is useless, so the read ceilings are deliberately NOT applied here.
|
||||
* The planner still runs `readonly = 1` under a short deadline and a memory
|
||||
* cap, so this is a narrower envelope, never an unbounded one.
|
||||
*/
|
||||
export function planningSettings(budget: ResourceBudget): Record<string, string> {
|
||||
return {
|
||||
readonly: "1",
|
||||
max_execution_time: String(Math.min(budget.maxExecutionTimeSec, 15)),
|
||||
timeout_overflow_mode: "throw",
|
||||
max_memory_usage: String(budget.maxMemoryUsageBytes),
|
||||
};
|
||||
}
|
||||
|
||||
/** Every setting name BI-P012 requires, for the DoD assertion. */
|
||||
export const REQUIRED_GOVERNANCE_SETTINGS = [
|
||||
"max_execution_time",
|
||||
"max_result_rows",
|
||||
"max_rows_to_read",
|
||||
"max_bytes_to_read",
|
||||
"max_memory_usage",
|
||||
"max_concurrent_queries",
|
||||
] as const;
|
||||
|
||||
/**
|
||||
* `max_concurrent_queries` lives on a SETTINGS PROFILE, not on a statement.
|
||||
* Rendered here so BI-P011's restricted-user bootstrap and BI-P012's budget
|
||||
* come from one source.
|
||||
*/
|
||||
export function concurrencyProfileSql(profileName: string, budget: ResourceBudget): string[] {
|
||||
return [
|
||||
`CREATE SETTINGS PROFILE IF NOT EXISTS ${profileName} SETTINGS ` +
|
||||
`max_execution_time = ${budget.maxExecutionTimeSec}, ` +
|
||||
`max_result_rows = ${budget.maxResultRows}, ` +
|
||||
`max_rows_to_read = ${budget.maxRowsToRead}, ` +
|
||||
`max_bytes_to_read = ${budget.maxBytesToRead}, ` +
|
||||
`max_memory_usage = ${budget.maxMemoryUsageBytes}, ` +
|
||||
`readonly = 1`,
|
||||
`CREATE QUOTA IF NOT EXISTS ${profileName}_quota ` +
|
||||
`FOR INTERVAL 1 SECOND MAX queries = ${budget.maxConcurrentQueries * 20} ` +
|
||||
`TO ALL EXCEPT default`,
|
||||
];
|
||||
}
|
||||
|
||||
/**
|
||||
* Local admission control. One instance per process; keyed by tenant so a
|
||||
* noisy tenant is throttled without touching anyone else's slots — which is
|
||||
* exactly BI-P012's DoD wording ("without degrading unrelated tenant").
|
||||
*/
|
||||
export class ConcurrencyGate {
|
||||
private readonly inFlight = new Map<string, number>();
|
||||
|
||||
constructor(private readonly limitFor: (tenantId: string) => number) {}
|
||||
|
||||
/** Returns a release function. Throws RATE_LIMITED when the tenant is full. */
|
||||
acquire(tenantId: string): () => void {
|
||||
const limit = this.limitFor(tenantId);
|
||||
const current = this.inFlight.get(tenantId) ?? 0;
|
||||
if (current >= limit) {
|
||||
throw new SiaxError(
|
||||
"RATE_LIMITED",
|
||||
"0pt.analytics",
|
||||
`tenant already has ${current} analytics queries in flight (limit ${limit})`,
|
||||
);
|
||||
}
|
||||
this.inFlight.set(tenantId, current + 1);
|
||||
let released = false;
|
||||
return () => {
|
||||
if (released) return;
|
||||
released = true;
|
||||
const n = (this.inFlight.get(tenantId) ?? 1) - 1;
|
||||
if (n <= 0) this.inFlight.delete(tenantId);
|
||||
else this.inFlight.set(tenantId, n);
|
||||
};
|
||||
}
|
||||
|
||||
/** Observability: in-flight count for a tenant (0 when idle). */
|
||||
inFlightFor(tenantId: string): number {
|
||||
return this.inFlight.get(tenantId) ?? 0;
|
||||
}
|
||||
|
||||
/** Total in-flight across all tenants. */
|
||||
totalInFlight(): number {
|
||||
let n = 0;
|
||||
for (const v of this.inFlight.values()) n += v;
|
||||
return n;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,169 @@
|
||||
/**
|
||||
* src/analytics/engine/schema.ts
|
||||
* ===============================
|
||||
* BI-P010 — CLICKHOUSE PRODUCTION SCHEMA.
|
||||
*
|
||||
* BI-P010's explicit warning is "Avoid one giant unbounded table for every
|
||||
* analytics use case." The existing `opt_metric_events` (owned by
|
||||
* src/analytics/clickhouse-analytics.ts, PRESERVED untouched) is exactly one
|
||||
* table for one job: low-cardinality capability metrics. It stays, and it is
|
||||
* NOT extended into the catch-all.
|
||||
*
|
||||
* Four tables, four jobs, four retention profiles:
|
||||
*
|
||||
* opt_metric_events (PRE-EXISTING, not defined here) — capability
|
||||
* metrics, Map(String,String) dimensions, 90d.
|
||||
* opt_analytics_event — high-volume per-tenant event stream. Wide
|
||||
* dimension arrays, SHORT TTL (raw events are not
|
||||
* the serving tier; they age into the rollups).
|
||||
* opt_metric_daily — AggregatingMergeTree rollup fed by a
|
||||
* MATERIALIZED VIEW off opt_analytics_event.
|
||||
* This is what a dashboard time-series reads;
|
||||
* long TTL, tiny footprint.
|
||||
* opt_dataset_projection — the SERVING tier (BI-P003) for a CURATED
|
||||
* dataset: one row per (tenant, dataset, entity),
|
||||
* ReplacingMergeTree so a refresh is idempotent.
|
||||
*
|
||||
* Why the split is load-bearing, not cosmetic:
|
||||
* * different ORDER BY keys. A dashboard filtering (tenant, metric, day)
|
||||
* and a projection looked up by (tenant, dataset, entity_key) cannot
|
||||
* share one primary key without one of them degenerating to a full scan.
|
||||
* * different TTLs. Raw events must expire fast; rollups must not.
|
||||
* * different cardinality. `LowCardinality(String)` is a win on
|
||||
* tenant/metric/dataset and a LOSS on a free-form entity key, so they
|
||||
* cannot be the same column type in the same table.
|
||||
*
|
||||
* EVERY table leads its ORDER BY with `tenant_id`. That is not stylistic: it
|
||||
* is what makes the forced tenant predicate (BI-P011) a primary-key range
|
||||
* scan instead of a filter applied after reading everyone's rows, and it is
|
||||
* what makes a per-tenant row policy cheap.
|
||||
*
|
||||
* LEGAL BOUNDARY (inherited verbatim from clickhouse-analytics.ts): this
|
||||
* schema is for generic 0pt analytics. It MUST NEVER be pointed at the b0ts
|
||||
* ClickHouse pipelines or quarantined market data.
|
||||
*/
|
||||
|
||||
export interface AnalyticsSchemaOptions {
|
||||
database: string;
|
||||
/** Raw event retention. Deliberately short — rollups carry the history. */
|
||||
eventRetentionDays?: number;
|
||||
/** Rollup retention. */
|
||||
rollupRetentionDays?: number;
|
||||
/** Serving-projection retention. */
|
||||
projectionRetentionDays?: number;
|
||||
}
|
||||
|
||||
/** Physical tables this module owns (opt_metric_events is NOT one of them). */
|
||||
export const ANALYTICS_TABLES = [
|
||||
"opt_analytics_event",
|
||||
"opt_metric_daily",
|
||||
"opt_dataset_projection",
|
||||
] as const;
|
||||
export type AnalyticsTable = (typeof ANALYTICS_TABLES)[number];
|
||||
|
||||
/** The materialized view wiring opt_analytics_event -> opt_metric_daily. */
|
||||
export const ANALYTICS_MATERIALIZED_VIEWS = ["opt_metric_daily_mv"] as const;
|
||||
|
||||
/**
|
||||
* DDL, in dependency order. Every statement is CREATE ... IF NOT EXISTS so
|
||||
* applying it to a live database is idempotent (the same contract
|
||||
* `ClickHouseAnalytics.ensureSchema` already keeps).
|
||||
*/
|
||||
export function analyticsSchemaDdl(opts: AnalyticsSchemaOptions): string[] {
|
||||
const db = opts.database;
|
||||
const eventTtl = Math.floor(opts.eventRetentionDays ?? 30);
|
||||
const rollupTtl = Math.floor(opts.rollupRetentionDays ?? 1095);
|
||||
const projectionTtl = Math.floor(opts.projectionRetentionDays ?? 365);
|
||||
|
||||
return [
|
||||
`CREATE DATABASE IF NOT EXISTS ${db}`,
|
||||
|
||||
// ── raw event stream ────────────────────────────────────────────────
|
||||
// Dimensions as parallel arrays (not Map) so ClickHouse can prune on
|
||||
// dimension VALUES via the bloom filter index; Map(String,String) forces
|
||||
// reading the whole map. `event_id` makes deduplication possible on
|
||||
// replay without a ReplacingMergeTree's merge latency.
|
||||
`CREATE TABLE IF NOT EXISTS ${db}.opt_analytics_event (
|
||||
tenant_id LowCardinality(String),
|
||||
project_id LowCardinality(String),
|
||||
dataset_id LowCardinality(String),
|
||||
metric LowCardinality(String),
|
||||
event_id String,
|
||||
event_time DateTime64(3, 'UTC'),
|
||||
value Float64,
|
||||
dim_names Array(LowCardinality(String)),
|
||||
dim_values Array(String),
|
||||
ingested_at DateTime64(3, 'UTC') DEFAULT now64(3),
|
||||
INDEX idx_dim_values dim_values TYPE bloom_filter(0.01) GRANULARITY 4
|
||||
)
|
||||
ENGINE = MergeTree()
|
||||
PARTITION BY toYYYYMM(event_time)
|
||||
ORDER BY (tenant_id, project_id, dataset_id, metric, event_time)
|
||||
TTL toDateTime(event_time) + INTERVAL ${eventTtl} DAY
|
||||
SETTINGS index_granularity = 8192`,
|
||||
|
||||
// ── daily rollup (serving tier for time series) ─────────────────────
|
||||
`CREATE TABLE IF NOT EXISTS ${db}.opt_metric_daily (
|
||||
tenant_id LowCardinality(String),
|
||||
project_id LowCardinality(String),
|
||||
dataset_id LowCardinality(String),
|
||||
metric LowCardinality(String),
|
||||
day Date,
|
||||
sum_state AggregateFunction(sum, Float64),
|
||||
count_state AggregateFunction(count, Float64),
|
||||
min_state AggregateFunction(min, Float64),
|
||||
max_state AggregateFunction(max, Float64)
|
||||
)
|
||||
ENGINE = AggregatingMergeTree()
|
||||
PARTITION BY toYYYYMM(day)
|
||||
ORDER BY (tenant_id, project_id, dataset_id, metric, day)
|
||||
TTL day + INTERVAL ${rollupTtl} DAY`,
|
||||
|
||||
`CREATE MATERIALIZED VIEW IF NOT EXISTS ${db}.opt_metric_daily_mv
|
||||
TO ${db}.opt_metric_daily AS
|
||||
SELECT
|
||||
tenant_id,
|
||||
project_id,
|
||||
dataset_id,
|
||||
metric,
|
||||
toDate(event_time) AS day,
|
||||
sumState(value) AS sum_state,
|
||||
countState(value) AS count_state,
|
||||
minState(value) AS min_state,
|
||||
maxState(value) AS max_state
|
||||
FROM ${db}.opt_analytics_event
|
||||
GROUP BY tenant_id, project_id, dataset_id, metric, day`,
|
||||
|
||||
// ── serving projection for a CURATED dataset ────────────────────────
|
||||
// ReplacingMergeTree(version) so refreshProjection is idempotent: the
|
||||
// same (tenant, dataset, entity_key) re-ingested at a higher version
|
||||
// supersedes rather than duplicates.
|
||||
`CREATE TABLE IF NOT EXISTS ${db}.opt_dataset_projection (
|
||||
tenant_id LowCardinality(String),
|
||||
project_id LowCardinality(String),
|
||||
dataset_id LowCardinality(String),
|
||||
entity_key String,
|
||||
as_of DateTime64(3, 'UTC'),
|
||||
version UInt64,
|
||||
attributes Map(String, String),
|
||||
measures Map(String, Float64),
|
||||
INDEX idx_entity entity_key TYPE bloom_filter(0.01) GRANULARITY 4
|
||||
)
|
||||
ENGINE = ReplacingMergeTree(version)
|
||||
PARTITION BY (tenant_id)
|
||||
ORDER BY (tenant_id, project_id, dataset_id, entity_key)
|
||||
TTL toDateTime(as_of) + INTERVAL ${projectionTtl} DAY`,
|
||||
];
|
||||
}
|
||||
|
||||
/**
|
||||
* Statement that proves a table's ORDER BY leads with tenant_id — used by the
|
||||
* BI-P011 test suite so "tenant-first primary key" is asserted against the
|
||||
* live server, not against this file's own comments.
|
||||
*/
|
||||
export function sortingKeySql(db: string, table: string): string {
|
||||
return (
|
||||
`SELECT sorting_key FROM system.tables ` +
|
||||
`WHERE database = '${db}' AND name = '${table}' FORMAT TSV`
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,180 @@
|
||||
/**
|
||||
* src/analytics/engine/siax-analytics-engine.ts
|
||||
* ==============================================
|
||||
* BI-P009 — the provider-neutral analytics engine interface.
|
||||
*
|
||||
* Eight capabilities, named exactly as BI-P009 lists them:
|
||||
*
|
||||
* health executeMetric executeQuery estimateQuery
|
||||
* explain cancelQuery ingestProjection refreshProjection
|
||||
*
|
||||
* ClickHouse is the CANONICAL provider (BI-P009); DuckDB (BI-P014) and Trino
|
||||
* (BI-P015) implement the same interface later. Nothing above this interface
|
||||
* may import a provider module directly — that is what makes "replaceable
|
||||
* provider" true rather than aspirational.
|
||||
*
|
||||
* NORMATIVE OWNERSHIP (BI program §0): every method takes an
|
||||
* `AnalyticsIdentity` that the CALLER cannot forge, because it is produced by
|
||||
* CL0UD verification upstream, not by request-body parsing. Product repos
|
||||
* (VehIQ) call an API in front of this interface; they never hold a provider
|
||||
* credential and never construct an engine.
|
||||
*/
|
||||
import type { CompiledQuery, MetricAggregate } from "./query-compiler";
|
||||
import type { AnalyticsTable } from "./schema";
|
||||
import type { BudgetTier, ResourceBudget } from "./resource-governance";
|
||||
|
||||
/**
|
||||
* The VERIFIED caller. `tenant_id` here is the one the engine forces into
|
||||
* every predicate — it comes from CL0UD identity verification (BI-P011 layer
|
||||
* 1), never from a request body. This estate has already been burned once by
|
||||
* a tenant read out of a request body (INF0 Wave 2, cross-tenant RAG access),
|
||||
* which is why the type is separate from the request rather than a field on
|
||||
* it.
|
||||
*/
|
||||
export interface AnalyticsIdentity {
|
||||
tenant_id: string;
|
||||
project_id?: string;
|
||||
/** CL0UD principal, for audit + per-principal quota. */
|
||||
principal_id: string;
|
||||
correlation_id?: string;
|
||||
/** Budget tier granted by CL0UD entitlements (BI-P012). */
|
||||
budget_tier?: BudgetTier;
|
||||
/** CL0UD quota, which may only TIGHTEN the tier. */
|
||||
quota?: Partial<ResourceBudget>;
|
||||
}
|
||||
|
||||
export interface MetricRequest {
|
||||
metric: string;
|
||||
aggregate: MetricAggregate;
|
||||
dataset_id?: string;
|
||||
group_by?: string[];
|
||||
filters?: Record<string, string>;
|
||||
from_time?: string;
|
||||
to_time?: string;
|
||||
limit?: number;
|
||||
table?: AnalyticsTable;
|
||||
}
|
||||
|
||||
export interface AnalyticsRow {
|
||||
/** Resolved group values, keyed by the CALLER-FACING dimension name. */
|
||||
groups?: Record<string, string>;
|
||||
value: number;
|
||||
}
|
||||
|
||||
export interface EngineResult {
|
||||
rows: AnalyticsRow[];
|
||||
/** Engine-side id; the handle `cancelQuery` needs. */
|
||||
query_id: string;
|
||||
/** True when the governance ceiling cut the result short (BI-P012). */
|
||||
truncated: boolean;
|
||||
stats: QueryStats;
|
||||
}
|
||||
|
||||
export interface QueryStats {
|
||||
duration_ms: number;
|
||||
rows_read?: number;
|
||||
bytes_read?: number;
|
||||
result_rows: number;
|
||||
}
|
||||
|
||||
export interface QueryEstimate {
|
||||
/** Rows the engine expects to SCAN. */
|
||||
estimated_rows_to_read: number;
|
||||
estimated_bytes_to_read: number;
|
||||
/** False when the estimate exceeds the caller's budget (BI-P012). */
|
||||
within_budget: boolean;
|
||||
budget: ResourceBudget;
|
||||
/** Why it was refused, when `within_budget` is false. */
|
||||
reason?: string;
|
||||
}
|
||||
|
||||
export interface QueryExplanation {
|
||||
/** Provider's plan text. Never contains caller-supplied SQL fragments. */
|
||||
plan: string;
|
||||
/** The predicate the compiler forced — assertable, not narrated. */
|
||||
forced_tenant_predicate: string;
|
||||
table: string;
|
||||
}
|
||||
|
||||
export interface ProjectionRow {
|
||||
entity_key: string;
|
||||
as_of?: string;
|
||||
version?: number;
|
||||
attributes?: Record<string, string>;
|
||||
measures?: Record<string, number>;
|
||||
}
|
||||
|
||||
export interface ProjectionIngestRequest {
|
||||
dataset_id: string;
|
||||
rows: ProjectionRow[];
|
||||
}
|
||||
|
||||
export interface ProjectionRefreshResult {
|
||||
dataset_id: string;
|
||||
/** Rows visible after the refresh settled. */
|
||||
row_count: number;
|
||||
refreshed_at: string;
|
||||
}
|
||||
|
||||
export interface HealthReport {
|
||||
healthy: boolean;
|
||||
provider: string;
|
||||
/** Provider version when reachable. */
|
||||
version?: string;
|
||||
/** Short operator-facing reason. Never a raw engine body. */
|
||||
detail?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* A raw, ANALYST-facing query (BI-P017 will govern its SQL dialect). It is a
|
||||
* SEPARATE method from `executeMetric` on purpose: `executeMetric` has no
|
||||
* raw-SQL surface at all, and keeping them apart means the closed DSL cannot
|
||||
* silently grow one.
|
||||
*
|
||||
* The canonical ClickHouse provider in this pass DELIBERATELY refuses this
|
||||
* method (`CAPABILITY_UNAVAILABLE`): opening an analyst SQL surface before
|
||||
* BI-P017's dialect contract and BI-P033's authorization exist would be
|
||||
* exactly the "raw SQL attempt" BI-P011's adversarial list names. Declaring
|
||||
* it and refusing it is honest; implementing it early is not.
|
||||
*/
|
||||
export interface AnalystQueryRequest {
|
||||
sql: string;
|
||||
dataset_ids: string[];
|
||||
limit?: number;
|
||||
}
|
||||
|
||||
export interface SiaxAnalyticsEngine {
|
||||
readonly provider: string;
|
||||
|
||||
health(): Promise<HealthReport>;
|
||||
|
||||
executeMetric(identity: AnalyticsIdentity, request: MetricRequest): Promise<EngineResult>;
|
||||
|
||||
executeQuery(identity: AnalyticsIdentity, request: AnalystQueryRequest): Promise<EngineResult>;
|
||||
|
||||
estimateQuery(identity: AnalyticsIdentity, request: MetricRequest): Promise<QueryEstimate>;
|
||||
|
||||
cancelQuery(identity: AnalyticsIdentity, queryId: string): Promise<boolean>;
|
||||
|
||||
explain(identity: AnalyticsIdentity, request: MetricRequest): Promise<QueryExplanation>;
|
||||
|
||||
ingestProjection(identity: AnalyticsIdentity, request: ProjectionIngestRequest): Promise<{ ingested: number }>;
|
||||
|
||||
refreshProjection(identity: AnalyticsIdentity, dataset_id: string): Promise<ProjectionRefreshResult>;
|
||||
}
|
||||
|
||||
/** The eight capability names, for the provider contract test. */
|
||||
export const SIAX_ANALYTICS_ENGINE_CAPABILITIES = [
|
||||
"health",
|
||||
"executeMetric",
|
||||
"executeQuery",
|
||||
"estimateQuery",
|
||||
"cancelQuery",
|
||||
"explain",
|
||||
"ingestProjection",
|
||||
"refreshProjection",
|
||||
] as const;
|
||||
export type SiaxAnalyticsEngineCapability = (typeof SIAX_ANALYTICS_ENGINE_CAPABILITIES)[number];
|
||||
|
||||
/** Re-exported so provider modules do not reach across into the compiler. */
|
||||
export type { CompiledQuery };
|
||||
@@ -0,0 +1,356 @@
|
||||
/**
|
||||
* src/analytics/repositories/analytics-domain-repository.ts
|
||||
* ==========================================================
|
||||
* BI-P002 persistence for the analytics domain model (sql/008).
|
||||
*
|
||||
* Two rules the type system alone cannot keep, so they live here:
|
||||
*
|
||||
* 1. EVERY read and write is tenant-scoped. There is no method that takes a
|
||||
* dataset_id without also taking the tenant it must belong to, and every
|
||||
* WHERE carries `tenant_id = $tenant`. A caller cannot "forget" the
|
||||
* tenant because there is no signature that lets them.
|
||||
* 2. VERSION ROWS ARE APPEND-ONLY from this layer too. `createDatasetVersion`
|
||||
* /`createMetricVersion` compute the next version number inside the same
|
||||
* statement (`SELECT coalesce(max(version),0)+1`) so two concurrent
|
||||
* minters collide on the UNIQUE(dataset_id, version) constraint instead
|
||||
* of silently overwriting — and the database triggers from sql/008 catch
|
||||
* anything that still tries to UPDATE.
|
||||
*
|
||||
* Uses postgres.js through `RepositorySql`, the same connection type the other
|
||||
* P0 repositories take (src/repositories/db.ts).
|
||||
*/
|
||||
import crypto from "node:crypto";
|
||||
import { SiaxError } from "../../platform/errors";
|
||||
import type { RepositorySql } from "../../repositories/db";
|
||||
import type {
|
||||
AnalyticsDataset,
|
||||
DatasetVersion,
|
||||
DimensionDefinition,
|
||||
MetricDefinition,
|
||||
MetricVersion,
|
||||
PiiClassification,
|
||||
} from "../domain/entities";
|
||||
import type { AnalyticsTier } from "../domain/tiers";
|
||||
|
||||
/** The verified tenant scope every call must present. */
|
||||
export interface TenantScope {
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
}
|
||||
|
||||
function assertScope(scope: TenantScope): void {
|
||||
if (!scope?.tenant_id?.trim() || !scope?.project_id?.trim()) {
|
||||
throw new SiaxError(
|
||||
"UNAUTHENTICATED",
|
||||
"0pt.analytics",
|
||||
"analytics persistence requires a verified tenant_id + project_id",
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function iso(v: string | Date | null | undefined): string | undefined {
|
||||
if (v === null || v === undefined) return undefined;
|
||||
return v instanceof Date ? v.toISOString() : v;
|
||||
}
|
||||
|
||||
interface DatasetRow {
|
||||
dataset_id: string;
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
name: string;
|
||||
tier: string;
|
||||
schema_contract_id: string;
|
||||
schema_contract_version: string;
|
||||
owner_id: string;
|
||||
source_ref: string;
|
||||
freshness_slo_seconds: number;
|
||||
quality_rule_ids: string[];
|
||||
retention_days: number;
|
||||
classification: string;
|
||||
status: string;
|
||||
raw_access_exception: string | null;
|
||||
description: string | null;
|
||||
created_at: string | Date;
|
||||
updated_at: string | Date | null;
|
||||
created_by: string | null;
|
||||
}
|
||||
|
||||
function toDataset(r: DatasetRow): AnalyticsDataset {
|
||||
return {
|
||||
dataset_id: r.dataset_id,
|
||||
tenant_id: r.tenant_id,
|
||||
project_id: r.project_id,
|
||||
name: r.name,
|
||||
tier: r.tier as AnalyticsTier,
|
||||
schema_contract_id: r.schema_contract_id,
|
||||
schema_contract_version: r.schema_contract_version,
|
||||
owner_id: r.owner_id,
|
||||
source_ref: r.source_ref,
|
||||
freshness_slo_seconds: Number(r.freshness_slo_seconds),
|
||||
quality_rule_ids: r.quality_rule_ids ?? [],
|
||||
retention_days: Number(r.retention_days),
|
||||
classification: r.classification as PiiClassification,
|
||||
status: r.status as AnalyticsDataset["status"],
|
||||
...(r.raw_access_exception ? { raw_access_exception: r.raw_access_exception } : {}),
|
||||
...(r.description ? { description: r.description } : {}),
|
||||
created_at: iso(r.created_at)!,
|
||||
...(r.updated_at ? { updated_at: iso(r.updated_at) } : {}),
|
||||
...(r.created_by ? { created_by: r.created_by } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
export class AnalyticsDomainRepository {
|
||||
constructor(private readonly sql: RepositorySql) {}
|
||||
|
||||
// ── AnalyticsDataset ─────────────────────────────────────────────────────
|
||||
|
||||
async createDataset(scope: TenantScope, input: Omit<AnalyticsDataset, "tenant_id" | "project_id" | "created_at" | "updated_at">): Promise<AnalyticsDataset> {
|
||||
assertScope(scope);
|
||||
const rows = await this.sql<DatasetRow[]>`
|
||||
INSERT INTO opt_analytics_dataset
|
||||
(dataset_id, tenant_id, project_id, name, tier, schema_contract_id, schema_contract_version,
|
||||
owner_id, source_ref, freshness_slo_seconds, quality_rule_ids, retention_days,
|
||||
classification, status, raw_access_exception, description, created_by)
|
||||
VALUES
|
||||
(${input.dataset_id}, ${scope.tenant_id}, ${scope.project_id}, ${input.name}, ${input.tier},
|
||||
${input.schema_contract_id}, ${input.schema_contract_version}, ${input.owner_id}, ${input.source_ref},
|
||||
${input.freshness_slo_seconds}, ${this.sql.array(input.quality_rule_ids ?? [])}, ${input.retention_days},
|
||||
${input.classification}, ${input.status}, ${input.raw_access_exception ?? null},
|
||||
${input.description ?? null}, ${input.created_by ?? null})
|
||||
RETURNING *`;
|
||||
return toDataset(rows[0]);
|
||||
}
|
||||
|
||||
/** Tenant-scoped read. A foreign dataset_id returns undefined, never a row. */
|
||||
async getDataset(scope: TenantScope, datasetId: string): Promise<AnalyticsDataset | undefined> {
|
||||
assertScope(scope);
|
||||
const rows = await this.sql<DatasetRow[]>`
|
||||
SELECT * FROM opt_analytics_dataset
|
||||
WHERE dataset_id = ${datasetId}
|
||||
AND tenant_id = ${scope.tenant_id}
|
||||
AND project_id = ${scope.project_id}
|
||||
LIMIT 1`;
|
||||
return rows[0] ? toDataset(rows[0]) : undefined;
|
||||
}
|
||||
|
||||
async listDatasets(scope: TenantScope, filter?: { tier?: AnalyticsTier; status?: string }): Promise<AnalyticsDataset[]> {
|
||||
assertScope(scope);
|
||||
const rows = await this.sql<DatasetRow[]>`
|
||||
SELECT * FROM opt_analytics_dataset
|
||||
WHERE tenant_id = ${scope.tenant_id}
|
||||
AND project_id = ${scope.project_id}
|
||||
${filter?.tier ? this.sql`AND tier = ${filter.tier}` : this.sql``}
|
||||
${filter?.status ? this.sql`AND status = ${filter.status}` : this.sql``}
|
||||
ORDER BY created_at DESC`;
|
||||
return rows.map(toDataset);
|
||||
}
|
||||
|
||||
// ── DatasetVersion (append-only) ─────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Mint the next version. The version number is computed inside the INSERT,
|
||||
* so two racing callers cannot both read "3" and both write it — the loser
|
||||
* hits UNIQUE(dataset_id, version) and gets a CONFLICT, which is the honest
|
||||
* outcome.
|
||||
*/
|
||||
async createDatasetVersion(
|
||||
scope: TenantScope,
|
||||
input: { dataset_id: string; contract_hash: string; schema_contract_version: string; record_count?: number; created_by?: string },
|
||||
): Promise<DatasetVersion> {
|
||||
assertScope(scope);
|
||||
const owner = await this.getDataset(scope, input.dataset_id);
|
||||
if (!owner) {
|
||||
throw new SiaxError("NOT_FOUND", "0pt.analytics", "dataset not found in this tenant");
|
||||
}
|
||||
const id = `dsv_${crypto.randomUUID().replace(/-/g, "")}`;
|
||||
try {
|
||||
const rows = await this.sql<
|
||||
{
|
||||
dataset_version_id: string;
|
||||
dataset_id: string;
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
version: number;
|
||||
contract_hash: string;
|
||||
schema_contract_version: string;
|
||||
record_count: string | number | null;
|
||||
created_at: string | Date;
|
||||
created_by: string | null;
|
||||
}[]
|
||||
>`
|
||||
INSERT INTO opt_analytics_dataset_version
|
||||
(dataset_version_id, dataset_id, tenant_id, project_id, version,
|
||||
contract_hash, schema_contract_version, record_count, created_by)
|
||||
SELECT ${id}, ${input.dataset_id}, ${scope.tenant_id}, ${scope.project_id},
|
||||
coalesce(max(version), 0) + 1,
|
||||
${input.contract_hash}, ${input.schema_contract_version},
|
||||
${input.record_count ?? null}, ${input.created_by ?? null}
|
||||
FROM opt_analytics_dataset_version
|
||||
WHERE dataset_id = ${input.dataset_id} AND tenant_id = ${scope.tenant_id}
|
||||
RETURNING *`;
|
||||
const r = rows[0];
|
||||
return {
|
||||
dataset_version_id: r.dataset_version_id,
|
||||
dataset_id: r.dataset_id,
|
||||
tenant_id: r.tenant_id,
|
||||
project_id: r.project_id,
|
||||
version: Number(r.version),
|
||||
contract_hash: r.contract_hash,
|
||||
schema_contract_version: r.schema_contract_version,
|
||||
...(r.record_count !== null ? { record_count: Number(r.record_count) } : {}),
|
||||
created_at: iso(r.created_at)!,
|
||||
...(r.created_by ? { created_by: r.created_by } : {}),
|
||||
};
|
||||
} catch (err) {
|
||||
if (String(err).includes("duplicate key")) {
|
||||
throw new SiaxError("CONFLICT", "0pt.analytics", "a concurrent writer minted this dataset version");
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
async listDatasetVersions(scope: TenantScope, datasetId: string): Promise<DatasetVersion[]> {
|
||||
assertScope(scope);
|
||||
const rows = await this.sql<
|
||||
{ dataset_version_id: string; dataset_id: string; tenant_id: string; project_id: string; version: number; contract_hash: string; schema_contract_version: string; record_count: string | number | null; created_at: string | Date; created_by: string | null }[]
|
||||
>`
|
||||
SELECT * FROM opt_analytics_dataset_version
|
||||
WHERE dataset_id = ${datasetId}
|
||||
AND tenant_id = ${scope.tenant_id}
|
||||
AND project_id = ${scope.project_id}
|
||||
ORDER BY version ASC`;
|
||||
return rows.map((r) => ({
|
||||
dataset_version_id: r.dataset_version_id,
|
||||
dataset_id: r.dataset_id,
|
||||
tenant_id: r.tenant_id,
|
||||
project_id: r.project_id,
|
||||
version: Number(r.version),
|
||||
contract_hash: r.contract_hash,
|
||||
schema_contract_version: r.schema_contract_version,
|
||||
...(r.record_count !== null ? { record_count: Number(r.record_count) } : {}),
|
||||
created_at: iso(r.created_at)!,
|
||||
...(r.created_by ? { created_by: r.created_by } : {}),
|
||||
}));
|
||||
}
|
||||
|
||||
// ── MetricDefinition / MetricVersion ─────────────────────────────────────
|
||||
|
||||
async createMetric(
|
||||
scope: TenantScope,
|
||||
input: Omit<MetricDefinition, "tenant_id" | "project_id" | "created_at" | "updated_at" | "current_version">,
|
||||
): Promise<MetricDefinition> {
|
||||
assertScope(scope);
|
||||
const rows = await this.sql<
|
||||
{ metric_id: string; tenant_id: string; project_id: string; name: string; dataset_id: string; aggregation: string; measure_field: string; denominator_field: string | null; allowed_dimension_ids: string[]; current_version: number; owner_id: string; description: string | null; created_at: string | Date; updated_at: string | Date | null }[]
|
||||
>`
|
||||
INSERT INTO opt_analytics_metric_definition
|
||||
(metric_id, tenant_id, project_id, name, dataset_id, aggregation, measure_field,
|
||||
denominator_field, allowed_dimension_ids, current_version, owner_id, description)
|
||||
VALUES
|
||||
(${input.metric_id}, ${scope.tenant_id}, ${scope.project_id}, ${input.name}, ${input.dataset_id},
|
||||
${input.aggregation}, ${input.measure_field}, ${input.denominator_field ?? null},
|
||||
${this.sql.array(input.allowed_dimension_ids ?? [])}, 1, ${input.owner_id}, ${input.description ?? null})
|
||||
RETURNING *`;
|
||||
const r = rows[0];
|
||||
return {
|
||||
metric_id: r.metric_id,
|
||||
tenant_id: r.tenant_id,
|
||||
project_id: r.project_id,
|
||||
name: r.name,
|
||||
dataset_id: r.dataset_id,
|
||||
aggregation: r.aggregation as MetricDefinition["aggregation"],
|
||||
measure_field: r.measure_field,
|
||||
...(r.denominator_field ? { denominator_field: r.denominator_field } : {}),
|
||||
allowed_dimension_ids: r.allowed_dimension_ids ?? [],
|
||||
current_version: Number(r.current_version),
|
||||
owner_id: r.owner_id,
|
||||
...(r.description ? { description: r.description } : {}),
|
||||
created_at: iso(r.created_at)!,
|
||||
...(r.updated_at ? { updated_at: iso(r.updated_at) } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
/** Append-only. `change_reason` is required for version > 1 (DB CHECK). */
|
||||
async createMetricVersion(
|
||||
scope: TenantScope,
|
||||
input: { metric_id: string; definition_json: string; change_reason?: string; created_by?: string },
|
||||
): Promise<MetricVersion> {
|
||||
assertScope(scope);
|
||||
const hash = crypto.createHash("sha256").update(input.definition_json).digest("hex");
|
||||
const id = `mv_${crypto.randomUUID().replace(/-/g, "")}`;
|
||||
const rows = await this.sql<
|
||||
{ metric_version_id: string; metric_id: string; tenant_id: string; project_id: string; version: number; definition_json: string; definition_hash: string; change_reason: string | null; created_at: string | Date; created_by: string | null }[]
|
||||
>`
|
||||
INSERT INTO opt_analytics_metric_version
|
||||
(metric_version_id, metric_id, tenant_id, project_id, version, definition_json,
|
||||
definition_hash, change_reason, created_by)
|
||||
SELECT ${id}, ${input.metric_id}, ${scope.tenant_id}, ${scope.project_id},
|
||||
coalesce(max(version), 0) + 1, ${input.definition_json}, ${hash},
|
||||
${input.change_reason ?? null}, ${input.created_by ?? null}
|
||||
FROM opt_analytics_metric_version
|
||||
WHERE metric_id = ${input.metric_id} AND tenant_id = ${scope.tenant_id}
|
||||
RETURNING *`;
|
||||
const r = rows[0];
|
||||
return {
|
||||
metric_version_id: r.metric_version_id,
|
||||
metric_id: r.metric_id,
|
||||
tenant_id: r.tenant_id,
|
||||
project_id: r.project_id,
|
||||
version: Number(r.version),
|
||||
definition_json: r.definition_json,
|
||||
definition_hash: r.definition_hash,
|
||||
...(r.change_reason ? { change_reason: r.change_reason } : {}),
|
||||
created_at: iso(r.created_at)!,
|
||||
...(r.created_by ? { created_by: r.created_by } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
// ── DimensionDefinition (the query compiler's allow-list) ────────────────
|
||||
|
||||
async createDimension(
|
||||
scope: TenantScope,
|
||||
input: Omit<DimensionDefinition, "tenant_id" | "project_id" | "created_at">,
|
||||
): Promise<DimensionDefinition> {
|
||||
assertScope(scope);
|
||||
const rows = await this.sql<
|
||||
{ dimension_id: string; tenant_id: string; project_id: string; name: string; dataset_id: string; field_name: string; field_type: string; classification: string; description: string | null; created_at: string | Date }[]
|
||||
>`
|
||||
INSERT INTO opt_analytics_dimension_definition
|
||||
(dimension_id, tenant_id, project_id, name, dataset_id, field_name, field_type, classification, description)
|
||||
VALUES
|
||||
(${input.dimension_id}, ${scope.tenant_id}, ${scope.project_id}, ${input.name}, ${input.dataset_id},
|
||||
${input.field_name}, ${input.field_type}, ${input.classification}, ${input.description ?? null})
|
||||
RETURNING *`;
|
||||
const r = rows[0];
|
||||
return {
|
||||
dimension_id: r.dimension_id,
|
||||
tenant_id: r.tenant_id,
|
||||
project_id: r.project_id,
|
||||
name: r.name,
|
||||
dataset_id: r.dataset_id,
|
||||
field_name: r.field_name,
|
||||
field_type: r.field_type as DimensionDefinition["field_type"],
|
||||
classification: r.classification as PiiClassification,
|
||||
...(r.description ? { description: r.description } : {}),
|
||||
created_at: iso(r.created_at)!,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The dimension allow-list the compiler resolves through, for one tenant.
|
||||
* Returns caller-facing name -> physical field name. A tenant with no
|
||||
* declared dimensions gets an EMPTY map, which means every dimension
|
||||
* reference is refused — fail closed, not "allow everything".
|
||||
*/
|
||||
async dimensionCatalog(scope: TenantScope, datasetId?: string): Promise<Record<string, string>> {
|
||||
assertScope(scope);
|
||||
const rows = await this.sql<{ name: string; field_name: string }[]>`
|
||||
SELECT name, field_name FROM opt_analytics_dimension_definition
|
||||
WHERE tenant_id = ${scope.tenant_id}
|
||||
AND project_id = ${scope.project_id}
|
||||
${datasetId ? this.sql`AND dataset_id = ${datasetId}` : this.sql``}`;
|
||||
const out: Record<string, string> = Object.create(null);
|
||||
for (const r of rows) out[r.name] = r.field_name;
|
||||
return out;
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user