feat(analytics): BI-P002..BI-P012 analytics core — domain model, tiers, contracts, Arrow, ClickHouse engine, isolation, governance
CI (SIAX Cloud) / security (push) Successful in 41s
Security Scan (SIAX Cloud) / security (push) Successful in 40s
CI (SIAX Cloud) / integration (push) Successful in 52s
CI (SIAX Cloud) / quality (push) Successful in 1m5s
CI (SIAX Cloud) / security (pull_request) Successful in 31s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 29s
CI (SIAX Cloud) / integration (pull_request) Successful in 45s
CI (SIAX Cloud) / quality (pull_request) Successful in 53s

Implements the eight BI program prompts that make 0pt the canonical SIAX
analytics capability. Every file is NEW: not one existing file is modified,
and src/analytics/clickhouse-analytics.ts is byte-identical to main.

BI-P002 — the nineteen canonical entities, verbatim, with tenant_id +
project_id on every tenant-owned record. sql/008 gives each one a table.
The four version entities BI-P002 names as immutable are enforced by BEFORE
UPDATE/DELETE triggers, not by a TypeScript `readonly` — proven by direct SQL
against a live Postgres.

BI-P003 — RAW/NORMALIZED/CURATED/SERVING with the read rule enforced:
assertReadableByBiSurface refuses a RAW dataset to a dashboard unless a
recorded exception exists. All eight required declarations are NOT NULL.

BI-P004 — DatasetContract and a fail-closed compatibility gate. Fifteen
breaking-change classes are each tested twice: refused under a minor bump,
accepted under a major one.

BI-P005 — Arrow IPC alongside JSON, with a hand-written FlatBuffers writer so
no npm dependency (or CVE surface) is added. Conformance comes from three
independent implementations, not from self-agreement: pyarrow 25.0.1 and
DuckDB 1.5.5 read identical values from both ClickHouse's Arrow and ours, and
ClickHouse's own Arrow C++ ingests our 20,000-row batch with exact aggregates.

BI-P009 — SiaxAnalyticsEngine with all eight capabilities. The existing
ClickHouseAnalytics is COMPOSED, never replaced. executeQuery (analyst SQL) is
declared and deliberately refused with CAPABILITY_UNAVAILABLE until BI-P017 and
BI-P033 land, rather than opening the raw-SQL hole BI-P011 enumerates.

BI-P010 — four purpose-built tables, not one giant one, and the difference is
measured against the counterfactual: 48.8ms/40,960 rows vs 174.4ms/5,400,000
rows for the same question; 5/660 granules vs 660/1149.

BI-P011 — layered isolation, each layer proven with the layer above removed.
Without the forced predicate the same statement returns 50,050 (both tenants);
with it, t_alpha gets 50. Without a row policy an unrestricted user sees 2
tenants; the restricted user running the IDENTICAL statement sees 1.

BI-P012 — every named limit, fired individually and identified by ClickHouse
error code, plus the half usually skipped: an unrelated tenant is timed while
a real CPU bomb is in flight and must still return the CORRECT answer.

Three defects found by writing tests that can fail:
  * `CHECK (array_length(purposes,1) >= 1)` can NEVER fail — array_length is
    NULL for '{}' and a NULL CHECK passes. Fixed with coalesce(); the test
    that caught it is pinned.
  * EXPLAIN ESTIMATE run under the caller's own max_rows_to_read fails with
    code 158 exactly when the estimate would be over budget. Planner calls now
    use a narrower envelope.
  * result_overflow_mode=break returns HTTP 200 with 65,409 of 100,000 rows
    and no indication it is short. Every SELECT ships `throw`.

Verified rather than assumed: server7's ClickHouse holds `opt_scraped` (51
tables, 407,793,471 rows, `persons`/`cz_owners` included) — the b0ts plane this
module is prohibited from touching — has no `opt_analytics` database, zero
tenant_id columns and zero row policies. Nothing there was modified; every
proof ran against disposable containers.

Evidence: evidence/bi-p002-p012/{test-transcript.txt,bi-p010-benchmark.json,
bi-p005-arrow-roundtrip.json,server7-clickhouse-probe.md}
Docs: docs/analytics/BI_ANALYTICS_CORE.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
2026-09-06 23:01:27 +02:00
parent 8d0a288668
commit 84bb9faa4f
32 changed files with 7844 additions and 0 deletions
+271
View File
@@ -0,0 +1,271 @@
# 0pt Analytics Core — BI-P002 … BI-P012
Status of this document: it describes what **is in this repository and has been
run**, and states separately what has **not** been deployed. Nothing here is a
plan written in the present tense.
---
## 0. The ownership boundary (normative, not negotiable)
```
SIAX / 0pt owns the analytics engine, the query compiler, the tenant forcing,
the resource governance and the semantic/contract layer.
Product repos (VehIQ and the rest) own DOMAIN MEANING — what a row means,
which metric a dealer cares about, how a screen reads.
```
Concretely, and enforced rather than asserted:
* `BIApplicationBinding` (BI-P002 entity 19) has **no credential field**, in
TypeScript or in `sql/008_analytics_domain_model.sql`. A bound BI
application authenticates to the 0pt Analytics API with a CL0UD identity and
receives compiled, tenant-forced results. It never learns a ClickHouse DSN,
user, or password. Two tests assert the absence of such a column — one over
the migration text, one over `information_schema.columns` on a live
Postgres — so adding one later is a test failure.
* `AnalyticsIdentity` (BI-P009) is a **separate parameter** from the request,
never a field on it. This estate has already shipped a live cross-tenant
hole by reading a tenant out of a request body (INF0 Wave 2); the type
shape is what stops that recurring here.
---
## 1. What each prompt produced
| Prompt | Deliverable | Where |
|---|---|---|
| BI-P002 | 19 canonical entities + migrations + tenant-scoped persistence + restore drill | `src/analytics/domain/entities.ts`, `sql/008_analytics_domain_model.sql`, `src/analytics/repositories/analytics-domain-repository.ts` |
| BI-P003 | RAW / NORMALIZED / CURATED / SERVING, with the BI-surface read rule enforced | `src/analytics/domain/tiers.ts` |
| BI-P004 | `DatasetContract` + a fail-closed compatibility gate | `src/analytics/domain/contract.ts` |
| BI-P005 | Arrow IPC encode/decode + ClickHouse pass-through | `src/analytics/arrow/ipc.ts`, `src/analytics/arrow/flatbuffer-builder.ts` |
| BI-P009 | `SiaxAnalyticsEngine` interface + ClickHouse provider, existing implementation **preserved** | `src/analytics/engine/siax-analytics-engine.ts`, `src/analytics/engine/clickhouse-engine.ts` |
| BI-P010 | Four purpose-built ClickHouse tables + rollup MV, benchmarked | `src/analytics/engine/schema.ts`, `scripts/bi-p010-benchmark.mjs` |
| BI-P011 | Layered tenant isolation: verified identity, forced compiler predicate, restricted users, row policies | `src/analytics/engine/query-compiler.ts` |
| BI-P012 | Every named ClickHouse limit + CL0UD quota + per-tenant admission control | `src/analytics/engine/resource-governance.ts` |
---
## 2. BI-P009 — what "preserve" means here
`src/analytics/clickhouse-analytics.ts` is **byte-identical to main**. The new
engine composes it:
* `health()` delegates to it;
* the pre-existing `opt_metric_events` path is reachable through
`executeLegacyMetric()`, still using its own compiler;
* everything else uses the new compiler for the BI-P010 tables.
"Do not regress the existing forced tenant predicate" is enforced by
`tests/analytics/tenant-predicate-regression.test.ts`, which asserts the exact
string `tenant_id = {tenant:String}` on the URL the adapter builds. It has been
run with the forcing deliberately deleted from both compilers and goes RED
(2 of 6 tests fail); the transcript is
`evidence/bi-p002-p012/test-transcript.txt` §1–§2.
The eight capabilities are all implemented, with one **deliberate refusal**:
`executeQuery` (analyst SQL) throws `CAPABILITY_UNAVAILABLE` until BI-P017
(SQL dialect contract) and BI-P033 (analytics authorization) exist. Shipping a
raw-SQL surface first would create exactly the "raw SQL attempt" hole BI-P011
enumerates. It is declared and refused, not silently missing.
---
## 3. BI-P010 — why four tables, measured
BI-P010 warns against "one giant unbounded table for every analytics use
case". `scripts/bi-p010-benchmark.mjs` builds both fixtures and the
counterfactual, and times them. Measured on a disposable
`clickhouse/clickhouse-server:24.8` (full transcript:
`evidence/bi-p002-p012/bi-p010-benchmark.json`):
| Question | Purpose-built | One giant table |
|---|---|---|
| 90-day metric total, one tenant | **48.8 ms p50**, 40 960 rows read | 174.4 ms p50, **5 400 000** rows read |
| Vehicle point lookup by key, one tenant | **49.3 ms p50**, 8 192 rows read | 236.6 ms p50, **7 401 152** rows read |
| Avg asking price by make over 2M vehicles | **389.6 ms p50** | 680.8 ms p50 |
| Granules read for the metric total | **5 / 660** | 660 / 1 149 |
Fixtures: 5 400 000 SaaS events (200 tenants × 12 metrics × 90 days) and
4 000 000 vehicle projections (2M for the tenant under test + 2M decoys owned
by 50 other tenants).
**Honest finding that does not flatter the design:** at this scale the
`AggregatingMergeTree` rollup (37.98 ms p50, 21 416 rows read) was *not* faster
in wall time than scanning raw events (32.55 ms p50, 40 960 rows read). It
reads about half the rows, and its value is the long TTL — raw events expire
in 30 days, the rollup keeps three years — so the rollup earns its keep on
retention and on scale beyond this fixture, not on latency at 5.4M rows. Do
not quote it as a speed win.
---
## 4. BI-P011 — the isolation proof
Four layers, and each one is tested with the layer above it *removed*:
1. **Verified CL0UD identity.** `AnalyticsIdentity` is not part of the
request; a blank tenant or principal is `UNAUTHENTICATED`, never a wildcard.
2. **The 0pt query compiler forces the tenant.** Every compiled statement
contains `tenant_id = {tenant:String}` as its first WHERE condition, with
the value bound as an HTTP parameter — the tenant string never appears in
the SQL text.
3. **Restricted database users + row policies.**
`restrictedTenantUserDdl()` renders a per-tenant ClickHouse user with
`readonly = 1` and a `CREATE ROW POLICY ... USING tenant_id = '<tenant>'`
on every analytics table.
4. **Separate database/cluster tier** for very-high-isolation customers:
supported by construction — the engine takes a `database` and a `baseUrl`
per instance — but **not yet operated**, because there is no deployed
ClickHouse for 0pt analytics at all (see §7).
The proof, run against a disposable ClickHouse with 50 rows for `t_alpha`
(value 1 each) and 50 for `t_beta` (value 1000 each):
* **hole, layer 2 removed:** the same statement minus the forced predicate
returns **50 050** — both tenants in one number;
* **closed, layer 2 present:** `t_alpha` gets **50**, `t_beta` gets **50 000**;
* **hole, layer 3 absent:** an unrestricted user issuing a predicate-free
`uniqExact(tenant_id)` sees **2** tenants;
* **closed, layer 3 present:** the restricted user issuing the *identical*
statement sees **1**, and `sum(value)` returns 50 rather than 50 050.
Adversarial cases from BI-P011's list — tenant A querying B's dataset, forged
dataset IDs, raw SQL, SQL injection, dimension identifier injection, filter
injection — are covered in
`tests/analytics/query-compiler-injection.test.ts` (compiler layer) and
`tests/integration/analytics-tenant-isolation.integration.test.ts` (live).
Dimension names are **resolved through a per-tenant allow-list**
(`DimensionDefinition`), never escaped and never spliced. A tenant that has
declared no dimensions may reference none — fail closed.
---
## 5. BI-P012 — governance, and the partial-result trap
Every setting BI-P012 names is applied. Five ride on the statement itself
(`max_execution_time`, `max_result_rows`, `max_rows_to_read`,
`max_bytes_to_read`, `max_memory_usage`) under `readonly = 1`, so the query
text cannot re-raise them; `max_concurrent_queries` is a per-user quota plus a
local admission gate.
Overflow modes are **`throw`, never `break`**. That is not a style preference.
Measured on ClickHouse 24.8: with `result_overflow_mode = break` and
`max_result_rows = 10`, a 100 000-row query returns **HTTP 200 with 65 409
rows** and nothing in the payload says it is short. A truncated answer that
looks complete is the analytics form of a false green.
Query-bomb outcomes, each fired individually and identified by ClickHouse
error code: 158 `TOO_MANY_ROWS`, 307 `TOO_MANY_BYTES`, 159 `TIMEOUT_EXCEEDED`,
396 `TOO_MANY_ROWS_OR_BYTES`, 241 `MEMORY_LIMIT_EXCEEDED`.
The second half of the DoD — "without degrading unrelated tenant beyond
declared SLO" — is timed, not assumed: an unrelated tenant's interactive query
is run five times *while* an 8-second CPU bomb is in flight, must still return
the **correct** answer, and must stay under a declared 5000 ms ceiling (measured ~30 ms on an idle
server; the ceiling is set wide for a shared CI runner, and a genuinely
starved tenant would take seconds or fail).
Two measured details worth keeping:
* `EXPLAIN ESTIMATE` must **not** run under the caller's own
`max_rows_to_read`. It fails with code 158 exactly when the estimate would
be over budget — i.e. in the only case the estimate matters. Planner calls
use `planningSettings()`, a narrower envelope, instead.
* `SELECT count() FROM numbers(2e10)` is *not* a query bomb — ClickHouse
answers it in about 5 seconds because it can skip the values. The bomb in
the test hashes every row, which cannot be short-circuited.
---
## 6. BI-P005 — Arrow, and why it is not self-certified
Two paths:
* **pass-through** — ClickHouse emits Arrow IPC itself (`FORMAT ArrowStream`);
0pt streams those bytes unchanged. `output_format_arrow_string_as_string=1`
is required, or String columns arrive as Arrow **Binary** and both pyarrow
and DuckDB read them as bytes: silent semantic loss.
* **encoded** — for results 0pt assembles itself, `encodeArrowStream()` writes
the IPC stream, including a small hand-written FlatBuffers writer (no new
npm dependency, no new supply-chain surface).
A hand-rolled encoder is only worth having if somebody else's parser agrees
with it, so conformance is established by three independent implementations:
* **pyarrow 25.0.1** and **DuckDB 1.5.5** decode both ClickHouse's bytes and
0pt's bytes to *identical* values and types
(`evidence/bi-p002-p012/bi-p005-arrow-roundtrip.json`, verdict `PASS`);
* **ClickHouse's own Arrow C++** ingests 0pt-encoded Arrow — including a
20 000-row batch whose aggregates come back exact — in
`tests/integration/analytics-arrow-clickhouse.integration.test.ts`, which
runs on every push.
Supported types and their caveats are the table in `ARROW_TYPE_SUPPORT`;
`DateTime64(6|9)` and `UInt64` above 2^63 are deliberately **outside** the
supported set rather than silently lossy.
Arrow is also not an escape hatch from BI-P012:
`assertArrowWithinResultBudget()` decodes the row count and refuses a payload
over the caller's `max_result_rows`.
---
## 7. What is NOT done — verified, not assumed
* **There is no deployed ClickHouse for 0pt analytics.** Probed 2026-09-06 on
server7, the only ClickHouse in the estate: databases are
`opt_scraped`, `default`, `system`, `INFORMATION_SCHEMA`,
`information_schema`. There is no `opt_analytics` database, so
`ClickHouseAnalytics` — the "existing working" provider — has never been
run against a production instance. Everything in this document was proven
against disposable containers.
* **server7's ClickHouse must not be used for this.** It holds
`opt_scraped` — 51 tables, 407 793 471 rows, including `persons` and
`cz_owners`. That is the b0ts scraping plane, which
`src/analytics/clickhouse-analytics.ts`'s own header already prohibits this
module from touching (Ratsit/person-PII legal case, written DPIA
prohibition on quarantined market data). It also has **zero** columns named
`tenant_id` or `project_id` anywhere on the server and **zero** row
policies, so it could not satisfy BI-P011 even if it were legally
available. Nothing on that server was modified by this work.
* **BI-P011 layer 5** (separate database/cluster tier per high-isolation
customer) is supported by construction but not operated.
* **`executeQuery`** is refused pending BI-P017/BI-P033 (§2).
* The domain repository covers `AnalyticsDataset`, `DatasetVersion`,
`MetricDefinition`, `MetricVersion` and `DimensionDefinition`. The other
fourteen entities have tables, constraints and types but no repository
methods yet; their consumers (dashboards, reports, quality runs, lineage)
arrive with BI-P025 … BI-P035.
---
## 8. Running the proofs yourself
```bash
# unit suites (no infrastructure)
pnpm vitest run tests/analytics
# live ClickHouse suites
docker run -d --name opt-bi-ch -p 18123:8123 \
-e CLICKHOUSE_USER=opt_admin -e CLICKHOUSE_PASSWORD=<throwaway> \
-e CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1 clickhouse/clickhouse-server:24.8
CLICKHOUSE_URL=http://localhost:18123 CLICKHOUSE_AUTH=opt_admin:<throwaway> \
pnpm vitest run tests/integration
# BI-P002 restore drill
docker run -d --name opt-bi-pg -p 15455:5432 \
-e POSTGRES_USER=opt -e POSTGRES_PASSWORD=<throwaway> -e POSTGRES_DB=opt_bi_test \
postgres:16-alpine
ANALYTICS_TEST_DATABASE_URL=postgres://opt:<throwaway>@localhost:15455/opt_bi_test \
pnpm vitest run tests/analytics/analytics-domain-postgres.realinfra.test.ts
# BI-P010 benchmark and BI-P005 external roundtrip (rewrite the evidence files)
CLICKHOUSE_URL=... CLICKHOUSE_AUTH=... node scripts/bi-p010-benchmark.mjs
CLICKHOUSE_URL=... CLICKHOUSE_AUTH=... node scripts/bi-p005-arrow-roundtrip.mjs
```
Both scripts CREATE and DROP a database. Never point either at server7 or at
any ClickHouse holding b0ts or quarantined market data.