CI (SIAX Cloud) / security (push) Successful in 41s
Security Scan (SIAX Cloud) / security (push) Successful in 40s
CI (SIAX Cloud) / integration (push) Successful in 52s
CI (SIAX Cloud) / quality (push) Successful in 1m5s
CI (SIAX Cloud) / security (pull_request) Successful in 31s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 29s
CI (SIAX Cloud) / integration (pull_request) Successful in 45s
CI (SIAX Cloud) / quality (pull_request) Successful in 53s
Implements the eight BI program prompts that make 0pt the canonical SIAX
analytics capability. Every file is NEW: not one existing file is modified,
and src/analytics/clickhouse-analytics.ts is byte-identical to main.
BI-P002 — the nineteen canonical entities, verbatim, with tenant_id +
project_id on every tenant-owned record. sql/008 gives each one a table.
The four version entities BI-P002 names as immutable are enforced by BEFORE
UPDATE/DELETE triggers, not by a TypeScript `readonly` — proven by direct SQL
against a live Postgres.
BI-P003 — RAW/NORMALIZED/CURATED/SERVING with the read rule enforced:
assertReadableByBiSurface refuses a RAW dataset to a dashboard unless a
recorded exception exists. All eight required declarations are NOT NULL.
BI-P004 — DatasetContract and a fail-closed compatibility gate. Fifteen
breaking-change classes are each tested twice: refused under a minor bump,
accepted under a major one.
BI-P005 — Arrow IPC alongside JSON, with a hand-written FlatBuffers writer so
no npm dependency (or CVE surface) is added. Conformance comes from three
independent implementations, not from self-agreement: pyarrow 25.0.1 and
DuckDB 1.5.5 read identical values from both ClickHouse's Arrow and ours, and
ClickHouse's own Arrow C++ ingests our 20,000-row batch with exact aggregates.
BI-P009 — SiaxAnalyticsEngine with all eight capabilities. The existing
ClickHouseAnalytics is COMPOSED, never replaced. executeQuery (analyst SQL) is
declared and deliberately refused with CAPABILITY_UNAVAILABLE until BI-P017 and
BI-P033 land, rather than opening the raw-SQL hole BI-P011 enumerates.
BI-P010 — four purpose-built tables, not one giant one, and the difference is
measured against the counterfactual: 48.8ms/40,960 rows vs 174.4ms/5,400,000
rows for the same question; 5/660 granules vs 660/1149.
BI-P011 — layered isolation, each layer proven with the layer above removed.
Without the forced predicate the same statement returns 50,050 (both tenants);
with it, t_alpha gets 50. Without a row policy an unrestricted user sees 2
tenants; the restricted user running the IDENTICAL statement sees 1.
BI-P012 — every named limit, fired individually and identified by ClickHouse
error code, plus the half usually skipped: an unrelated tenant is timed while
a real CPU bomb is in flight and must still return the CORRECT answer.
Three defects found by writing tests that can fail:
* `CHECK (array_length(purposes,1) >= 1)` can NEVER fail — array_length is
NULL for '{}' and a NULL CHECK passes. Fixed with coalesce(); the test
that caught it is pinned.
* EXPLAIN ESTIMATE run under the caller's own max_rows_to_read fails with
code 158 exactly when the estimate would be over budget. Planner calls now
use a narrower envelope.
* result_overflow_mode=break returns HTTP 200 with 65,409 of 100,000 rows
and no indication it is short. Every SELECT ships `throw`.
Verified rather than assumed: server7's ClickHouse holds `opt_scraped` (51
tables, 407,793,471 rows, `persons`/`cz_owners` included) — the b0ts plane this
module is prohibited from touching — has no `opt_analytics` database, zero
tenant_id columns and zero row policies. Nothing there was modified; every
proof ran against disposable containers.
Evidence: evidence/bi-p002-p012/{test-transcript.txt,bi-p010-benchmark.json,
bi-p005-arrow-roundtrip.json,server7-clickhouse-probe.md}
Docs: docs/analytics/BI_ANALYTICS_CORE.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
102 lines
4.8 KiB
TypeScript
102 lines
4.8 KiB
TypeScript
/**
|
|
* BI-P005 — APACHE ARROW INTERCHANGE, the part provable without a live engine.
|
|
*
|
|
* Honest scope note: the tests here run 0pt's encoder against 0pt's decoder,
|
|
* which proves internal consistency and NOT specification conformance. The
|
|
* conformance evidence comes from two places where an INDEPENDENT
|
|
* implementation reads the bytes:
|
|
*
|
|
* * tests/integration/analytics-arrow-clickhouse.integration.test.ts —
|
|
* ClickHouse (Arrow C++) both produces Arrow this decoder reads and
|
|
* consumes Arrow this encoder writes. Runs on every push in CI.
|
|
* * evidence/bi-p002-p012/bi-p005-arrow-roundtrip.json — pyarrow and DuckDB
|
|
* decoding the same bytes, executed against real containers.
|
|
*
|
|
* That distinction is the point: a self-roundtrip that agrees with its own
|
|
* mistakes is exactly the kind of green this estate has been burned by.
|
|
*/
|
|
import { describe, it, expect } from "vitest";
|
|
import {
|
|
ARROW_TYPE_SUPPORT,
|
|
arrowStreamSettings,
|
|
assertArrowWithinResultBudget,
|
|
clickhouseArrowStreamFormat,
|
|
decodeArrowStream,
|
|
encodeArrowStream,
|
|
} from "@/src/analytics/arrow/ipc";
|
|
|
|
describe("BI-P005 Arrow IPC", () => {
|
|
it("emits a well-formed IPC stream: continuation markers, schema, batch, EOS", () => {
|
|
const bytes = encodeArrowStream([{ name: "n", type: "int64", values: [1, 2, 3] }]);
|
|
const dv = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
|
|
expect(dv.getUint32(0, true)).toBe(0xffffffff);
|
|
// Last 8 bytes are the end-of-stream marker: 0xFFFFFFFF then length 0.
|
|
expect(dv.getUint32(bytes.length - 8, true)).toBe(0xffffffff);
|
|
expect(dv.getInt32(bytes.length - 4, true)).toBe(0);
|
|
});
|
|
|
|
it("round-trips every supported column type with nulls", () => {
|
|
const bytes = encodeArrowStream([
|
|
{ name: "tenant_id", type: "utf8", values: ["t_alpha", "t_beta", null] },
|
|
{ name: "n", type: "int64", values: [1, 2, 3] },
|
|
{ name: "v", type: "float64", values: [1.5, 2.5, null] },
|
|
{ name: "flag", type: "bool", values: [true, false, true] },
|
|
{ name: "ts", type: "timestamp_ms", values: [new Date("2026-01-01T00:00:00Z"), null, new Date("2026-09-06T12:34:56.789Z")] },
|
|
]);
|
|
const info = decodeArrowStream(bytes);
|
|
expect(info.rowCount).toBe(3);
|
|
expect(info.batchCount).toBe(1);
|
|
expect(info.fields.map((f) => f.name)).toEqual(["tenant_id", "n", "v", "flag", "ts"]);
|
|
// Arrow Type union tags: Int=2, FloatingPoint=3, Utf8=5, Bool=6, Timestamp=10.
|
|
expect(info.fields.map((f) => f.typeType)).toEqual([5, 2, 3, 6, 10]);
|
|
});
|
|
|
|
it("handles an empty result set without producing a malformed stream", () => {
|
|
const bytes = encodeArrowStream([{ name: "n", type: "int64", values: [] }]);
|
|
const info = decodeArrowStream(bytes);
|
|
expect(info.rowCount).toBe(0);
|
|
expect(info.fields).toEqual([{ name: "n", typeType: 2 }]);
|
|
});
|
|
|
|
it("rejects ragged columns rather than emitting a stream that decodes to garbage", () => {
|
|
expect(() =>
|
|
encodeArrowStream([
|
|
{ name: "a", type: "int64", values: [1, 2] },
|
|
{ name: "b", type: "int64", values: [1] },
|
|
]),
|
|
).toThrowError(/expected 2/);
|
|
expect(() => encodeArrowStream([])).toThrowError(/at least one column/);
|
|
});
|
|
|
|
it("refuses an int64 value outside the representable range instead of wrapping it", () => {
|
|
expect(() =>
|
|
encodeArrowStream([{ name: "n", type: "int64", values: [BigInt("9223372036854775808")] }]),
|
|
).toThrowError(/out of int64 range/);
|
|
});
|
|
|
|
it("rejects bytes that are not an Arrow IPC stream", () => {
|
|
expect(() => decodeArrowStream(new Uint8Array([1, 2, 3, 4, 5, 6, 7, 8]))).toThrowError(/not an Arrow IPC stream/);
|
|
});
|
|
|
|
it("BI-P012: Arrow is not a way around max_result_rows", () => {
|
|
const bytes = encodeArrowStream([{ name: "n", type: "int64", values: [1, 2, 3, 4, 5] }]);
|
|
expect(() => assertArrowWithinResultBudget(bytes, 3)).toThrowError(/over the 3-row budget/);
|
|
expect(assertArrowWithinResultBudget(bytes, 5).rowCount).toBe(5);
|
|
});
|
|
|
|
it("the ClickHouse pass-through asks for text Strings, not Binary", () => {
|
|
expect(clickhouseArrowStreamFormat()).toBe("ArrowStream");
|
|
// Without this setting ClickHouse emits String as Arrow Binary, which
|
|
// pyarrow and DuckDB read as bytes — silent semantic loss.
|
|
expect(arrowStreamSettings().output_format_arrow_string_as_string).toBe("1");
|
|
});
|
|
|
|
it("documents the supported type mapping including where loss WOULD occur", () => {
|
|
expect(ARROW_TYPE_SUPPORT.length).toBeGreaterThanOrEqual(5);
|
|
const arrowTypes = ARROW_TYPE_SUPPORT.map((r) => r.arrow);
|
|
expect(new Set(arrowTypes)).toEqual(new Set(["int64", "float64", "utf8", "bool", "timestamp_ms"]));
|
|
// Every row states its caveat; an empty note would be a silent promise.
|
|
for (const row of ARROW_TYPE_SUPPORT) expect(row.note.length).toBeGreaterThan(10);
|
|
});
|
|
});
|