Files
0pt/tests/analytics/arrow-fixture.test.ts
T
Claude Opus 5 84bb9faa4f
CI (SIAX Cloud) / security (push) Successful in 41s
Security Scan (SIAX Cloud) / security (push) Successful in 40s
CI (SIAX Cloud) / integration (push) Successful in 52s
CI (SIAX Cloud) / quality (push) Successful in 1m5s
CI (SIAX Cloud) / security (pull_request) Successful in 31s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 29s
CI (SIAX Cloud) / integration (pull_request) Successful in 45s
CI (SIAX Cloud) / quality (pull_request) Successful in 53s
feat(analytics): BI-P002..BI-P012 analytics core — domain model, tiers, contracts, Arrow, ClickHouse engine, isolation, governance
Implements the eight BI program prompts that make 0pt the canonical SIAX
analytics capability. Every file is NEW: not one existing file is modified,
and src/analytics/clickhouse-analytics.ts is byte-identical to main.

BI-P002 — the nineteen canonical entities, verbatim, with tenant_id +
project_id on every tenant-owned record. sql/008 gives each one a table.
The four version entities BI-P002 names as immutable are enforced by BEFORE
UPDATE/DELETE triggers, not by a TypeScript `readonly` — proven by direct SQL
against a live Postgres.

BI-P003 — RAW/NORMALIZED/CURATED/SERVING with the read rule enforced:
assertReadableByBiSurface refuses a RAW dataset to a dashboard unless a
recorded exception exists. All eight required declarations are NOT NULL.

BI-P004 — DatasetContract and a fail-closed compatibility gate. Fifteen
breaking-change classes are each tested twice: refused under a minor bump,
accepted under a major one.

BI-P005 — Arrow IPC alongside JSON, with a hand-written FlatBuffers writer so
no npm dependency (or CVE surface) is added. Conformance comes from three
independent implementations, not from self-agreement: pyarrow 25.0.1 and
DuckDB 1.5.5 read identical values from both ClickHouse's Arrow and ours, and
ClickHouse's own Arrow C++ ingests our 20,000-row batch with exact aggregates.

BI-P009 — SiaxAnalyticsEngine with all eight capabilities. The existing
ClickHouseAnalytics is COMPOSED, never replaced. executeQuery (analyst SQL) is
declared and deliberately refused with CAPABILITY_UNAVAILABLE until BI-P017 and
BI-P033 land, rather than opening the raw-SQL hole BI-P011 enumerates.

BI-P010 — four purpose-built tables, not one giant one, and the difference is
measured against the counterfactual: 48.8ms/40,960 rows vs 174.4ms/5,400,000
rows for the same question; 5/660 granules vs 660/1149.

BI-P011 — layered isolation, each layer proven with the layer above removed.
Without the forced predicate the same statement returns 50,050 (both tenants);
with it, t_alpha gets 50. Without a row policy an unrestricted user sees 2
tenants; the restricted user running the IDENTICAL statement sees 1.

BI-P012 — every named limit, fired individually and identified by ClickHouse
error code, plus the half usually skipped: an unrelated tenant is timed while
a real CPU bomb is in flight and must still return the CORRECT answer.

Three defects found by writing tests that can fail:
  * `CHECK (array_length(purposes,1) >= 1)` can NEVER fail — array_length is
    NULL for '{}' and a NULL CHECK passes. Fixed with coalesce(); the test
    that caught it is pinned.
  * EXPLAIN ESTIMATE run under the caller's own max_rows_to_read fails with
    code 158 exactly when the estimate would be over budget. Planner calls now
    use a narrower envelope.
  * result_overflow_mode=break returns HTTP 200 with 65,409 of 100,000 rows
    and no indication it is short. Every SELECT ships `throw`.

Verified rather than assumed: server7's ClickHouse holds `opt_scraped` (51
tables, 407,793,471 rows, `persons`/`cz_owners` included) — the b0ts plane this
module is prohibited from touching — has no `opt_analytics` database, zero
tenant_id columns and zero row policies. Nothing there was modified; every
proof ran against disposable containers.

Evidence: evidence/bi-p002-p012/{test-transcript.txt,bi-p010-benchmark.json,
bi-p005-arrow-roundtrip.json,server7-clickhouse-probe.md}
Docs: docs/analytics/BI_ANALYTICS_CORE.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-06 23:01:27 +02:00

55 lines
2.2 KiB
TypeScript

/**
* The canonical BI-P005 fixture, in one place.
*
* Two jobs:
* 1. assert the encoder is DETERMINISTIC — the same columns must produce
* byte-identical output, or the "roundtrip" evidence would be comparing
* two different payloads run to run;
* 2. when ARROW_FIXTURE_DIR is set, write those exact bytes to disk so
* scripts/bi-p005-arrow-roundtrip.mjs hands pyarrow and DuckDB the bytes
* the SHIPPED module produces, not a copy of the encoder living in a
* script.
*/
import { describe, it, expect } from "vitest";
import fs from "node:fs";
import path from "node:path";
import crypto from "node:crypto";
import { type ArrowColumn, encodeArrowStream } from "@/src/analytics/arrow/ipc";
/** The same three rows the ClickHouse side of the DoD produces. */
export const BI_P005_FIXTURE: ArrowColumn[] = [
{ name: "tenant_id", type: "utf8", values: ["t_alpha", "t_beta", "t_gamma"] },
{ name: "n", type: "int64", values: [1, 2, 3] },
{ name: "v", type: "float64", values: [1.5, 2.5, 3.25] },
{ name: "flag", type: "bool", values: [true, false, true] },
{
name: "ts",
type: "timestamp_ms",
values: [
new Date("2026-01-01T00:00:00.000Z"),
new Date("2026-06-15T08:09:10.123Z"),
new Date("2026-09-06T12:34:56.789Z"),
],
},
];
describe("BI-P005 canonical Arrow fixture", () => {
it("encodes deterministically", () => {
const a = encodeArrowStream(BI_P005_FIXTURE);
const b = encodeArrowStream(BI_P005_FIXTURE);
expect(Buffer.from(a).equals(Buffer.from(b))).toBe(true);
expect(a.length).toBeGreaterThan(0);
});
it("writes the fixture for the external roundtrip when asked", () => {
const dir = process.env.ARROW_FIXTURE_DIR;
if (!dir) return; // not an error: the normal test run has nowhere to put it
const bytes = encodeArrowStream(BI_P005_FIXTURE);
const out = path.join(dir, "02_0pt_encoded.arrow");
fs.writeFileSync(out, bytes);
expect(fs.statSync(out).size).toBe(bytes.length);
// Print the digest so the evidence file can be tied to these exact bytes.
console.log(`ARROW_FIXTURE_SHA256=${crypto.createHash("sha256").update(bytes).digest("hex")}`);
});
});