feat(analytics): BI-P002..BI-P012 analytics core — domain model, tiers, contracts, Arrow, ClickHouse engine, isolation, governance
CI (SIAX Cloud) / security (push) Successful in 41s
Security Scan (SIAX Cloud) / security (push) Successful in 40s
CI (SIAX Cloud) / integration (push) Successful in 52s
CI (SIAX Cloud) / quality (push) Successful in 1m5s
CI (SIAX Cloud) / security (pull_request) Successful in 31s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 29s
CI (SIAX Cloud) / integration (pull_request) Successful in 45s
CI (SIAX Cloud) / quality (pull_request) Successful in 53s
CI (SIAX Cloud) / security (push) Successful in 41s
Security Scan (SIAX Cloud) / security (push) Successful in 40s
CI (SIAX Cloud) / integration (push) Successful in 52s
CI (SIAX Cloud) / quality (push) Successful in 1m5s
CI (SIAX Cloud) / security (pull_request) Successful in 31s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 29s
CI (SIAX Cloud) / integration (pull_request) Successful in 45s
CI (SIAX Cloud) / quality (pull_request) Successful in 53s
Implements the eight BI program prompts that make 0pt the canonical SIAX
analytics capability. Every file is NEW: not one existing file is modified,
and src/analytics/clickhouse-analytics.ts is byte-identical to main.
BI-P002 — the nineteen canonical entities, verbatim, with tenant_id +
project_id on every tenant-owned record. sql/008 gives each one a table.
The four version entities BI-P002 names as immutable are enforced by BEFORE
UPDATE/DELETE triggers, not by a TypeScript `readonly` — proven by direct SQL
against a live Postgres.
BI-P003 — RAW/NORMALIZED/CURATED/SERVING with the read rule enforced:
assertReadableByBiSurface refuses a RAW dataset to a dashboard unless a
recorded exception exists. All eight required declarations are NOT NULL.
BI-P004 — DatasetContract and a fail-closed compatibility gate. Fifteen
breaking-change classes are each tested twice: refused under a minor bump,
accepted under a major one.
BI-P005 — Arrow IPC alongside JSON, with a hand-written FlatBuffers writer so
no npm dependency (or CVE surface) is added. Conformance comes from three
independent implementations, not from self-agreement: pyarrow 25.0.1 and
DuckDB 1.5.5 read identical values from both ClickHouse's Arrow and ours, and
ClickHouse's own Arrow C++ ingests our 20,000-row batch with exact aggregates.
BI-P009 — SiaxAnalyticsEngine with all eight capabilities. The existing
ClickHouseAnalytics is COMPOSED, never replaced. executeQuery (analyst SQL) is
declared and deliberately refused with CAPABILITY_UNAVAILABLE until BI-P017 and
BI-P033 land, rather than opening the raw-SQL hole BI-P011 enumerates.
BI-P010 — four purpose-built tables, not one giant one, and the difference is
measured against the counterfactual: 48.8ms/40,960 rows vs 174.4ms/5,400,000
rows for the same question; 5/660 granules vs 660/1149.
BI-P011 — layered isolation, each layer proven with the layer above removed.
Without the forced predicate the same statement returns 50,050 (both tenants);
with it, t_alpha gets 50. Without a row policy an unrestricted user sees 2
tenants; the restricted user running the IDENTICAL statement sees 1.
BI-P012 — every named limit, fired individually and identified by ClickHouse
error code, plus the half usually skipped: an unrelated tenant is timed while
a real CPU bomb is in flight and must still return the CORRECT answer.
Three defects found by writing tests that can fail:
* `CHECK (array_length(purposes,1) >= 1)` can NEVER fail — array_length is
NULL for '{}' and a NULL CHECK passes. Fixed with coalesce(); the test
that caught it is pinned.
* EXPLAIN ESTIMATE run under the caller's own max_rows_to_read fails with
code 158 exactly when the estimate would be over budget. Planner calls now
use a narrower envelope.
* result_overflow_mode=break returns HTTP 200 with 65,409 of 100,000 rows
and no indication it is short. Every SELECT ships `throw`.
Verified rather than assumed: server7's ClickHouse holds `opt_scraped` (51
tables, 407,793,471 rows, `persons`/`cz_owners` included) — the b0ts plane this
module is prohibited from touching — has no `opt_analytics` database, zero
tenant_id columns and zero row policies. Nothing there was modified; every
proof ran against disposable containers.
Evidence: evidence/bi-p002-p012/{test-transcript.txt,bi-p010-benchmark.json,
bi-p005-arrow-roundtrip.json,server7-clickhouse-probe.md}
Docs: docs/analytics/BI_ANALYTICS_CORE.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,237 @@
|
||||
/**
|
||||
* src/analytics/arrow/flatbuffer-builder.ts
|
||||
* ==========================================
|
||||
* A minimal FlatBuffers writer, sufficient for the Arrow IPC metadata
|
||||
* messages this repo emits (BI-P005).
|
||||
*
|
||||
* WHY HAND-ROLLED rather than `apache-arrow` from npm: adding a runtime
|
||||
* dependency to 0pt's blocking-typechecked capability layer means a lockfile
|
||||
* change, a new supply-chain surface, and a new CVE surface for every
|
||||
* consumer of this repo — for a format whose METADATA side is a few hundred
|
||||
* bytes of well-specified binary and whose DATA side is plain columnar memory
|
||||
* either way.
|
||||
*
|
||||
* The correctness bar is NOT "this looks right". The BI-P005 evidence script
|
||||
* (scripts/bi-p005-arrow-roundtrip.mjs) decodes this writer's output with
|
||||
* pyarrow AND DuckDB — two independent implementations of the specification —
|
||||
* so a mistake here surfaces as a failed decode by somebody else's parser,
|
||||
* not as a green test of my own assumptions.
|
||||
*
|
||||
* Layout facts implemented (FlatBuffers binary format, little-endian):
|
||||
* * the buffer is built BACK TO FRONT; `offset` counts bytes from the end;
|
||||
* * a uoffset stored at a field site holds `site_offset + 4 - target_offset`
|
||||
* so a reader resolves it as `position + value`;
|
||||
* * a table begins with an soffset `table_start - vtable_start`;
|
||||
* * a vtable is `[vtable_len:i16][table_len:i16][field_offsets:i16...]`,
|
||||
* with 0 meaning "field absent, use the default".
|
||||
*/
|
||||
|
||||
export type Offset = number;
|
||||
|
||||
const SIZEOF_INT = 4;
|
||||
|
||||
export class FlatBufferBuilder {
|
||||
private buf: Uint8Array;
|
||||
private view: DataView;
|
||||
/** Bytes written so far, counted from the END of `buf`. */
|
||||
private used = 0;
|
||||
private minAlign = 1;
|
||||
private vtable: number[] = [];
|
||||
private objectStart = 0;
|
||||
private inObject = false;
|
||||
/** Offsets of previously written vtables, for deduplication. */
|
||||
private vtables: number[] = [];
|
||||
|
||||
constructor(initial = 1024) {
|
||||
this.buf = new Uint8Array(initial);
|
||||
this.view = new DataView(this.buf.buffer);
|
||||
}
|
||||
|
||||
/** Bytes written so far (FlatBuffers "offset"). */
|
||||
get offset(): number {
|
||||
return this.used;
|
||||
}
|
||||
|
||||
private ensure(extra: number): void {
|
||||
if (this.used + extra <= this.buf.length) return;
|
||||
let size = this.buf.length;
|
||||
while (size < this.used + extra) size *= 2;
|
||||
const next = new Uint8Array(size);
|
||||
next.set(this.buf.subarray(this.buf.length - this.used), size - this.used);
|
||||
this.buf = next;
|
||||
this.view = new DataView(this.buf.buffer);
|
||||
}
|
||||
|
||||
private pad(n: number): void {
|
||||
if (n <= 0) return;
|
||||
this.ensure(n);
|
||||
for (let i = 0; i < n; i++) {
|
||||
this.used += 1;
|
||||
this.buf[this.buf.length - this.used] = 0;
|
||||
}
|
||||
}
|
||||
|
||||
/** FlatBuffers `prep`: pad so that `additional` more bytes land aligned. */
|
||||
private prep(size: number, additional = 0): void {
|
||||
if (size > this.minAlign) this.minAlign = size;
|
||||
const alignSize = (~(this.used + additional) + 1) & (size - 1);
|
||||
this.pad(alignSize);
|
||||
}
|
||||
|
||||
writeInt8(v: number): void {
|
||||
this.ensure(1);
|
||||
this.used += 1;
|
||||
this.view.setInt8(this.buf.length - this.used, v);
|
||||
}
|
||||
|
||||
writeInt16(v: number): void {
|
||||
this.ensure(2);
|
||||
this.used += 2;
|
||||
this.view.setInt16(this.buf.length - this.used, v, true);
|
||||
}
|
||||
|
||||
writeInt32(v: number): void {
|
||||
this.ensure(4);
|
||||
this.used += 4;
|
||||
this.view.setInt32(this.buf.length - this.used, v, true);
|
||||
}
|
||||
|
||||
writeInt64(v: bigint): void {
|
||||
this.ensure(8);
|
||||
this.used += 8;
|
||||
this.view.setBigInt64(this.buf.length - this.used, v, true);
|
||||
}
|
||||
|
||||
writeBytes(bytes: Uint8Array): void {
|
||||
this.ensure(bytes.length);
|
||||
this.used += bytes.length;
|
||||
this.buf.set(bytes, this.buf.length - this.used);
|
||||
}
|
||||
|
||||
addInt8(slot: number, value: number, def: number): void {
|
||||
if (value === def) return;
|
||||
this.prep(1);
|
||||
this.writeInt8(value);
|
||||
this.slot(slot);
|
||||
}
|
||||
|
||||
addInt16(slot: number, value: number, def: number): void {
|
||||
if (value === def) return;
|
||||
this.prep(2);
|
||||
this.writeInt16(value);
|
||||
this.slot(slot);
|
||||
}
|
||||
|
||||
addInt32(slot: number, value: number, def: number): void {
|
||||
if (value === def) return;
|
||||
this.prep(4);
|
||||
this.writeInt32(value);
|
||||
this.slot(slot);
|
||||
}
|
||||
|
||||
addInt64(slot: number, value: bigint, def: bigint): void {
|
||||
if (value === def) return;
|
||||
this.prep(8);
|
||||
this.writeInt64(value);
|
||||
this.slot(slot);
|
||||
}
|
||||
|
||||
addOffset(slot: number, value: Offset, def: number): void {
|
||||
if (value === def) return;
|
||||
this.prep(SIZEOF_INT);
|
||||
this.writeInt32(this.used + SIZEOF_INT - value);
|
||||
this.slot(slot);
|
||||
}
|
||||
|
||||
private slot(slotNum: number): void {
|
||||
this.vtable[slotNum] = this.used;
|
||||
}
|
||||
|
||||
startObject(numFields: number): void {
|
||||
if (this.inObject) throw new Error("nested startObject");
|
||||
this.vtable = new Array<number>(numFields).fill(0);
|
||||
this.objectStart = this.used;
|
||||
this.inObject = true;
|
||||
}
|
||||
|
||||
endObject(): Offset {
|
||||
if (!this.inObject) throw new Error("endObject outside an object");
|
||||
this.prep(SIZEOF_INT);
|
||||
this.writeInt32(0); // soffset placeholder, patched below
|
||||
const objectOffset = this.used;
|
||||
|
||||
let trimmed = this.vtable.length;
|
||||
while (trimmed > 0 && this.vtable[trimmed - 1] === 0) trimmed -= 1;
|
||||
for (let i = trimmed - 1; i >= 0; i--) {
|
||||
this.writeInt16(this.vtable[i] !== 0 ? objectOffset - this.vtable[i] : 0);
|
||||
}
|
||||
this.writeInt16(objectOffset - this.objectStart); // table byte length
|
||||
const vtLen = (trimmed + 2) * 2;
|
||||
this.writeInt16(vtLen); // vtable byte length — lands at the vtable start
|
||||
|
||||
const vtStart = this.used;
|
||||
const vtAbs = this.buf.length - vtStart;
|
||||
let existing = 0;
|
||||
for (const candidate of this.vtables) {
|
||||
const candAbs = this.buf.length - candidate;
|
||||
if (this.view.getInt16(candAbs, true) !== vtLen) continue;
|
||||
let same = true;
|
||||
for (let i = 2; i < vtLen; i += 2) {
|
||||
if (this.view.getInt16(vtAbs + i, true) !== this.view.getInt16(candAbs + i, true)) {
|
||||
same = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (same) {
|
||||
existing = candidate;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
const objAbs = this.buf.length - objectOffset;
|
||||
if (existing !== 0) {
|
||||
this.used -= vtLen; // discard the duplicate vtable just written
|
||||
this.view.setInt32(objAbs, existing - objectOffset, true);
|
||||
} else {
|
||||
this.vtables.push(vtStart);
|
||||
this.view.setInt32(objAbs, this.used - objectOffset, true);
|
||||
}
|
||||
this.inObject = false;
|
||||
return objectOffset;
|
||||
}
|
||||
|
||||
createString(s: string): Offset {
|
||||
const bytes = new TextEncoder().encode(s);
|
||||
this.writeInt8(0); // null terminator (readers rely on it)
|
||||
this.startVector(1, bytes.length, 1);
|
||||
this.writeBytes(bytes);
|
||||
return this.endVector(bytes.length);
|
||||
}
|
||||
|
||||
startVector(elemSize: number, numElems: number, alignment: number): void {
|
||||
this.prep(SIZEOF_INT, elemSize * numElems);
|
||||
this.prep(alignment, elemSize * numElems);
|
||||
}
|
||||
|
||||
endVector(numElems: number): Offset {
|
||||
this.writeInt32(numElems);
|
||||
return this.used;
|
||||
}
|
||||
|
||||
/** Vector of table offsets. Elements are written in reverse build order. */
|
||||
createOffsetVector(items: Offset[]): Offset {
|
||||
this.startVector(SIZEOF_INT, items.length, SIZEOF_INT);
|
||||
for (let i = items.length - 1; i >= 0; i--) {
|
||||
this.writeInt32(this.used + SIZEOF_INT - items[i]);
|
||||
}
|
||||
return this.endVector(items.length);
|
||||
}
|
||||
|
||||
/** Finish the buffer with `root` as the root table. Returns the bytes. */
|
||||
finish(root: Offset): Uint8Array {
|
||||
this.prep(this.minAlign, SIZEOF_INT);
|
||||
this.prep(SIZEOF_INT);
|
||||
this.writeInt32(this.used + SIZEOF_INT - root);
|
||||
return this.buf.slice(this.buf.length - this.used);
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user