# 0pt Analytics Core — BI-P002 … BI-P012 Status of this document: it describes what **is in this repository and has been run**, and states separately what has **not** been deployed. Nothing here is a plan written in the present tense. --- ## 0. The ownership boundary (normative, not negotiable) ``` SIAX / 0pt owns the analytics engine, the query compiler, the tenant forcing, the resource governance and the semantic/contract layer. Product repos (VehIQ and the rest) own DOMAIN MEANING — what a row means, which metric a dealer cares about, how a screen reads. ``` Concretely, and enforced rather than asserted: * `BIApplicationBinding` (BI-P002 entity 19) has **no credential field**, in TypeScript or in `sql/008_analytics_domain_model.sql`. A bound BI application authenticates to the 0pt Analytics API with a CL0UD identity and receives compiled, tenant-forced results. It never learns a ClickHouse DSN, user, or password. Two tests assert the absence of such a column — one over the migration text, one over `information_schema.columns` on a live Postgres — so adding one later is a test failure. * `AnalyticsIdentity` (BI-P009) is a **separate parameter** from the request, never a field on it. This estate has already shipped a live cross-tenant hole by reading a tenant out of a request body (INF0 Wave 2); the type shape is what stops that recurring here. --- ## 1. What each prompt produced | Prompt | Deliverable | Where | |---|---|---| | BI-P002 | 19 canonical entities + migrations + tenant-scoped persistence + restore drill | `src/analytics/domain/entities.ts`, `sql/008_analytics_domain_model.sql`, `src/analytics/repositories/analytics-domain-repository.ts` | | BI-P003 | RAW / NORMALIZED / CURATED / SERVING, with the BI-surface read rule enforced | `src/analytics/domain/tiers.ts` | | BI-P004 | `DatasetContract` + a fail-closed compatibility gate | `src/analytics/domain/contract.ts` | | BI-P005 | Arrow IPC encode/decode + ClickHouse pass-through | `src/analytics/arrow/ipc.ts`, `src/analytics/arrow/flatbuffer-builder.ts` | | BI-P009 | `SiaxAnalyticsEngine` interface + ClickHouse provider, existing implementation **preserved** | `src/analytics/engine/siax-analytics-engine.ts`, `src/analytics/engine/clickhouse-engine.ts` | | BI-P010 | Four purpose-built ClickHouse tables + rollup MV, benchmarked | `src/analytics/engine/schema.ts`, `scripts/bi-p010-benchmark.mjs` | | BI-P011 | Layered tenant isolation: verified identity, forced compiler predicate, restricted users, row policies | `src/analytics/engine/query-compiler.ts` | | BI-P012 | Every named ClickHouse limit + CL0UD quota + per-tenant admission control | `src/analytics/engine/resource-governance.ts` | --- ## 2. BI-P009 — what "preserve" means here `src/analytics/clickhouse-analytics.ts` is **byte-identical to main**. The new engine composes it: * `health()` delegates to it; * the pre-existing `opt_metric_events` path is reachable through `executeLegacyMetric()`, still using its own compiler; * everything else uses the new compiler for the BI-P010 tables. "Do not regress the existing forced tenant predicate" is enforced by `tests/analytics/tenant-predicate-regression.test.ts`, which asserts the exact string `tenant_id = {tenant:String}` on the URL the adapter builds. It has been run with the forcing deliberately deleted from both compilers and goes RED (2 of 6 tests fail); the transcript is `evidence/bi-p002-p012/test-transcript.txt` §1–§2. The eight capabilities are all implemented, with one **deliberate refusal**: `executeQuery` (analyst SQL) throws `CAPABILITY_UNAVAILABLE` until BI-P017 (SQL dialect contract) and BI-P033 (analytics authorization) exist. Shipping a raw-SQL surface first would create exactly the "raw SQL attempt" hole BI-P011 enumerates. It is declared and refused, not silently missing. --- ## 3. BI-P010 — why four tables, measured BI-P010 warns against "one giant unbounded table for every analytics use case". `scripts/bi-p010-benchmark.mjs` builds both fixtures and the counterfactual, and times them. Measured on a disposable `clickhouse/clickhouse-server:24.8` (full transcript: `evidence/bi-p002-p012/bi-p010-benchmark.json`): | Question | Purpose-built | One giant table | |---|---|---| | 90-day metric total, one tenant | **48.8 ms p50**, 40 960 rows read | 174.4 ms p50, **5 400 000** rows read | | Vehicle point lookup by key, one tenant | **49.3 ms p50**, 8 192 rows read | 236.6 ms p50, **7 401 152** rows read | | Avg asking price by make over 2M vehicles | **389.6 ms p50** | 680.8 ms p50 | | Granules read for the metric total | **5 / 660** | 660 / 1 149 | Fixtures: 5 400 000 SaaS events (200 tenants × 12 metrics × 90 days) and 4 000 000 vehicle projections (2M for the tenant under test + 2M decoys owned by 50 other tenants). **Honest finding that does not flatter the design:** at this scale the `AggregatingMergeTree` rollup (37.98 ms p50, 21 416 rows read) was *not* faster in wall time than scanning raw events (32.55 ms p50, 40 960 rows read). It reads about half the rows, and its value is the long TTL — raw events expire in 30 days, the rollup keeps three years — so the rollup earns its keep on retention and on scale beyond this fixture, not on latency at 5.4M rows. Do not quote it as a speed win. --- ## 4. BI-P011 — the isolation proof Four layers, and each one is tested with the layer above it *removed*: 1. **Verified CL0UD identity.** `AnalyticsIdentity` is not part of the request; a blank tenant or principal is `UNAUTHENTICATED`, never a wildcard. 2. **The 0pt query compiler forces the tenant.** Every compiled statement contains `tenant_id = {tenant:String}` as its first WHERE condition, with the value bound as an HTTP parameter — the tenant string never appears in the SQL text. 3. **Restricted database users + row policies.** `restrictedTenantUserDdl()` renders a per-tenant ClickHouse user with `readonly = 1` and a `CREATE ROW POLICY ... USING tenant_id = ''` on every analytics table. 4. **Separate database/cluster tier** for very-high-isolation customers: supported by construction — the engine takes a `database` and a `baseUrl` per instance — but **not yet operated**, because there is no deployed ClickHouse for 0pt analytics at all (see §7). The proof, run against a disposable ClickHouse with 50 rows for `t_alpha` (value 1 each) and 50 for `t_beta` (value 1000 each): * **hole, layer 2 removed:** the same statement minus the forced predicate returns **50 050** — both tenants in one number; * **closed, layer 2 present:** `t_alpha` gets **50**, `t_beta` gets **50 000**; * **hole, layer 3 absent:** an unrestricted user issuing a predicate-free `uniqExact(tenant_id)` sees **2** tenants; * **closed, layer 3 present:** the restricted user issuing the *identical* statement sees **1**, and `sum(value)` returns 50 rather than 50 050. Adversarial cases from BI-P011's list — tenant A querying B's dataset, forged dataset IDs, raw SQL, SQL injection, dimension identifier injection, filter injection — are covered in `tests/analytics/query-compiler-injection.test.ts` (compiler layer) and `tests/integration/analytics-tenant-isolation.integration.test.ts` (live). Dimension names are **resolved through a per-tenant allow-list** (`DimensionDefinition`), never escaped and never spliced. A tenant that has declared no dimensions may reference none — fail closed. --- ## 5. BI-P012 — governance, and the partial-result trap Every setting BI-P012 names is applied. Five ride on the statement itself (`max_execution_time`, `max_result_rows`, `max_rows_to_read`, `max_bytes_to_read`, `max_memory_usage`) under `readonly = 1`, so the query text cannot re-raise them; `max_concurrent_queries` is a per-user quota plus a local admission gate. Overflow modes are **`throw`, never `break`**. That is not a style preference. Measured on ClickHouse 24.8: with `result_overflow_mode = break` and `max_result_rows = 10`, a 100 000-row query returns **HTTP 200 with 65 409 rows** and nothing in the payload says it is short. A truncated answer that looks complete is the analytics form of a false green. Query-bomb outcomes, each fired individually and identified by ClickHouse error code: 158 `TOO_MANY_ROWS`, 307 `TOO_MANY_BYTES`, 159 `TIMEOUT_EXCEEDED`, 396 `TOO_MANY_ROWS_OR_BYTES`, 241 `MEMORY_LIMIT_EXCEEDED`. The second half of the DoD — "without degrading unrelated tenant beyond declared SLO" — is timed, not assumed: an unrelated tenant's interactive query is run five times *while* an 8-second CPU bomb is in flight, must still return the **correct** answer, and must stay under a declared 5000 ms ceiling (measured ~30 ms on an idle server; the ceiling is set wide for a shared CI runner, and a genuinely starved tenant would take seconds or fail). Two measured details worth keeping: * `EXPLAIN ESTIMATE` must **not** run under the caller's own `max_rows_to_read`. It fails with code 158 exactly when the estimate would be over budget — i.e. in the only case the estimate matters. Planner calls use `planningSettings()`, a narrower envelope, instead. * `SELECT count() FROM numbers(2e10)` is *not* a query bomb — ClickHouse answers it in about 5 seconds because it can skip the values. The bomb in the test hashes every row, which cannot be short-circuited. --- ## 6. BI-P005 — Arrow, and why it is not self-certified Two paths: * **pass-through** — ClickHouse emits Arrow IPC itself (`FORMAT ArrowStream`); 0pt streams those bytes unchanged. `output_format_arrow_string_as_string=1` is required, or String columns arrive as Arrow **Binary** and both pyarrow and DuckDB read them as bytes: silent semantic loss. * **encoded** — for results 0pt assembles itself, `encodeArrowStream()` writes the IPC stream, including a small hand-written FlatBuffers writer (no new npm dependency, no new supply-chain surface). A hand-rolled encoder is only worth having if somebody else's parser agrees with it, so conformance is established by three independent implementations: * **pyarrow 25.0.1** and **DuckDB 1.5.5** decode both ClickHouse's bytes and 0pt's bytes to *identical* values and types (`evidence/bi-p002-p012/bi-p005-arrow-roundtrip.json`, verdict `PASS`); * **ClickHouse's own Arrow C++** ingests 0pt-encoded Arrow — including a 20 000-row batch whose aggregates come back exact — in `tests/integration/analytics-arrow-clickhouse.integration.test.ts`, which runs on every push. Supported types and their caveats are the table in `ARROW_TYPE_SUPPORT`; `DateTime64(6|9)` and `UInt64` above 2^63 are deliberately **outside** the supported set rather than silently lossy. Arrow is also not an escape hatch from BI-P012: `assertArrowWithinResultBudget()` decodes the row count and refuses a payload over the caller's `max_result_rows`. --- ## 7. What is NOT done — verified, not assumed * **There is no deployed ClickHouse for 0pt analytics.** Probed 2026-09-06 on server7, the only ClickHouse in the estate: databases are `opt_scraped`, `default`, `system`, `INFORMATION_SCHEMA`, `information_schema`. There is no `opt_analytics` database, so `ClickHouseAnalytics` — the "existing working" provider — has never been run against a production instance. Everything in this document was proven against disposable containers. * **server7's ClickHouse must not be used for this.** It holds `opt_scraped` — 51 tables, 407 793 471 rows, including `persons` and `cz_owners`. That is the b0ts scraping plane, which `src/analytics/clickhouse-analytics.ts`'s own header already prohibits this module from touching (Ratsit/person-PII legal case, written DPIA prohibition on quarantined market data). It also has **zero** columns named `tenant_id` or `project_id` anywhere on the server and **zero** row policies, so it could not satisfy BI-P011 even if it were legally available. Nothing on that server was modified by this work. * **BI-P011 layer 5** (separate database/cluster tier per high-isolation customer) is supported by construction but not operated. * **`executeQuery`** is refused pending BI-P017/BI-P033 (§2). * The domain repository covers `AnalyticsDataset`, `DatasetVersion`, `MetricDefinition`, `MetricVersion` and `DimensionDefinition`. The other fourteen entities have tables, constraints and types but no repository methods yet; their consumers (dashboards, reports, quality runs, lineage) arrive with BI-P025 … BI-P035. --- ## 8. Running the proofs yourself ```bash # unit suites (no infrastructure) pnpm vitest run tests/analytics # live ClickHouse suites docker run -d --name opt-bi-ch -p 18123:8123 \ -e CLICKHOUSE_USER=opt_admin -e CLICKHOUSE_PASSWORD= \ -e CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1 clickhouse/clickhouse-server:24.8 CLICKHOUSE_URL=http://localhost:18123 CLICKHOUSE_AUTH=opt_admin: \ pnpm vitest run tests/integration # BI-P002 restore drill docker run -d --name opt-bi-pg -p 15455:5432 \ -e POSTGRES_USER=opt -e POSTGRES_PASSWORD= -e POSTGRES_DB=opt_bi_test \ postgres:16-alpine ANALYTICS_TEST_DATABASE_URL=postgres://opt:@localhost:15455/opt_bi_test \ pnpm vitest run tests/analytics/analytics-domain-postgres.realinfra.test.ts # BI-P010 benchmark and BI-P005 external roundtrip (rewrite the evidence files) CLICKHOUSE_URL=... CLICKHOUSE_AUTH=... node scripts/bi-p010-benchmark.mjs CLICKHOUSE_URL=... CLICKHOUSE_AUTH=... node scripts/bi-p005-arrow-roundtrip.mjs ``` Both scripts CREATE and DROP a database. Never point either at server7 or at any ClickHouse holding b0ts or quarantined market data.