fix(recovery): restore source removed by production documentation merge

This commit is contained in:
2026-11-10 03:42:10 +01:00
parent 585bdd4b44
commit 3d2fc6f913
957 changed files with 115754 additions and 0 deletions
+271
View File
@@ -0,0 +1,271 @@
# 0pt Analytics Core — BI-P002 … BI-P012
Status of this document: it describes what **is in this repository and has been
run**, and states separately what has **not** been deployed. Nothing here is a
plan written in the present tense.
---
## 0. The ownership boundary (normative, not negotiable)
```
SIAX / 0pt owns the analytics engine, the query compiler, the tenant forcing,
the resource governance and the semantic/contract layer.
Product repos (VehIQ and the rest) own DOMAIN MEANING — what a row means,
which metric a dealer cares about, how a screen reads.
```
Concretely, and enforced rather than asserted:
* `BIApplicationBinding` (BI-P002 entity 19) has **no credential field**, in
TypeScript or in `sql/008_analytics_domain_model.sql`. A bound BI
application authenticates to the 0pt Analytics API with a CL0UD identity and
receives compiled, tenant-forced results. It never learns a ClickHouse DSN,
user, or password. Two tests assert the absence of such a column — one over
the migration text, one over `information_schema.columns` on a live
Postgres — so adding one later is a test failure.
* `AnalyticsIdentity` (BI-P009) is a **separate parameter** from the request,
never a field on it. This estate has already shipped a live cross-tenant
hole by reading a tenant out of a request body (INF0 Wave 2); the type
shape is what stops that recurring here.
---
## 1. What each prompt produced
| Prompt | Deliverable | Where |
|---|---|---|
| BI-P002 | 19 canonical entities + migrations + tenant-scoped persistence + restore drill | `src/analytics/domain/entities.ts`, `sql/008_analytics_domain_model.sql`, `src/analytics/repositories/analytics-domain-repository.ts` |
| BI-P003 | RAW / NORMALIZED / CURATED / SERVING, with the BI-surface read rule enforced | `src/analytics/domain/tiers.ts` |
| BI-P004 | `DatasetContract` + a fail-closed compatibility gate | `src/analytics/domain/contract.ts` |
| BI-P005 | Arrow IPC encode/decode + ClickHouse pass-through | `src/analytics/arrow/ipc.ts`, `src/analytics/arrow/flatbuffer-builder.ts` |
| BI-P009 | `SiaxAnalyticsEngine` interface + ClickHouse provider, existing implementation **preserved** | `src/analytics/engine/siax-analytics-engine.ts`, `src/analytics/engine/clickhouse-engine.ts` |
| BI-P010 | Four purpose-built ClickHouse tables + rollup MV, benchmarked | `src/analytics/engine/schema.ts`, `scripts/bi-p010-benchmark.mjs` |
| BI-P011 | Layered tenant isolation: verified identity, forced compiler predicate, restricted users, row policies | `src/analytics/engine/query-compiler.ts` |
| BI-P012 | Every named ClickHouse limit + CL0UD quota + per-tenant admission control | `src/analytics/engine/resource-governance.ts` |
---
## 2. BI-P009 — what "preserve" means here
`src/analytics/clickhouse-analytics.ts` is **byte-identical to main**. The new
engine composes it:
* `health()` delegates to it;
* the pre-existing `opt_metric_events` path is reachable through
`executeLegacyMetric()`, still using its own compiler;
* everything else uses the new compiler for the BI-P010 tables.
"Do not regress the existing forced tenant predicate" is enforced by
`tests/analytics/tenant-predicate-regression.test.ts`, which asserts the exact
string `tenant_id = {tenant:String}` on the URL the adapter builds. It has been
run with the forcing deliberately deleted from both compilers and goes RED
(2 of 6 tests fail); the transcript is
`evidence/bi-p002-p012/test-transcript.txt` §1–§2.
The eight capabilities are all implemented, with one **deliberate refusal**:
`executeQuery` (analyst SQL) throws `CAPABILITY_UNAVAILABLE` until BI-P017
(SQL dialect contract) and BI-P033 (analytics authorization) exist. Shipping a
raw-SQL surface first would create exactly the "raw SQL attempt" hole BI-P011
enumerates. It is declared and refused, not silently missing.
---
## 3. BI-P010 — why four tables, measured
BI-P010 warns against "one giant unbounded table for every analytics use
case". `scripts/bi-p010-benchmark.mjs` builds both fixtures and the
counterfactual, and times them. Measured on a disposable
`clickhouse/clickhouse-server:24.8` (full transcript:
`evidence/bi-p002-p012/bi-p010-benchmark.json`):
| Question | Purpose-built | One giant table |
|---|---|---|
| 90-day metric total, one tenant | **48.8 ms p50**, 40 960 rows read | 174.4 ms p50, **5 400 000** rows read |
| Vehicle point lookup by key, one tenant | **49.3 ms p50**, 8 192 rows read | 236.6 ms p50, **7 401 152** rows read |
| Avg asking price by make over 2M vehicles | **389.6 ms p50** | 680.8 ms p50 |
| Granules read for the metric total | **5 / 660** | 660 / 1 149 |
Fixtures: 5 400 000 SaaS events (200 tenants × 12 metrics × 90 days) and
4 000 000 vehicle projections (2M for the tenant under test + 2M decoys owned
by 50 other tenants).
**Honest finding that does not flatter the design:** at this scale the
`AggregatingMergeTree` rollup (37.98 ms p50, 21 416 rows read) was *not* faster
in wall time than scanning raw events (32.55 ms p50, 40 960 rows read). It
reads about half the rows, and its value is the long TTL — raw events expire
in 30 days, the rollup keeps three years — so the rollup earns its keep on
retention and on scale beyond this fixture, not on latency at 5.4M rows. Do
not quote it as a speed win.
---
## 4. BI-P011 — the isolation proof
Four layers, and each one is tested with the layer above it *removed*:
1. **Verified CL0UD identity.** `AnalyticsIdentity` is not part of the
request; a blank tenant or principal is `UNAUTHENTICATED`, never a wildcard.
2. **The 0pt query compiler forces the tenant.** Every compiled statement
contains `tenant_id = {tenant:String}` as its first WHERE condition, with
the value bound as an HTTP parameter — the tenant string never appears in
the SQL text.
3. **Restricted database users + row policies.**
`restrictedTenantUserDdl()` renders a per-tenant ClickHouse user with
`readonly = 1` and a `CREATE ROW POLICY ... USING tenant_id = '<tenant>'`
on every analytics table.
4. **Separate database/cluster tier** for very-high-isolation customers:
supported by construction — the engine takes a `database` and a `baseUrl`
per instance — but **not yet operated**, because there is no deployed
ClickHouse for 0pt analytics at all (see §7).
The proof, run against a disposable ClickHouse with 50 rows for `t_alpha`
(value 1 each) and 50 for `t_beta` (value 1000 each):
* **hole, layer 2 removed:** the same statement minus the forced predicate
returns **50 050** — both tenants in one number;
* **closed, layer 2 present:** `t_alpha` gets **50**, `t_beta` gets **50 000**;
* **hole, layer 3 absent:** an unrestricted user issuing a predicate-free
`uniqExact(tenant_id)` sees **2** tenants;
* **closed, layer 3 present:** the restricted user issuing the *identical*
statement sees **1**, and `sum(value)` returns 50 rather than 50 050.
Adversarial cases from BI-P011's list — tenant A querying B's dataset, forged
dataset IDs, raw SQL, SQL injection, dimension identifier injection, filter
injection — are covered in
`tests/analytics/query-compiler-injection.test.ts` (compiler layer) and
`tests/integration/analytics-tenant-isolation.integration.test.ts` (live).
Dimension names are **resolved through a per-tenant allow-list**
(`DimensionDefinition`), never escaped and never spliced. A tenant that has
declared no dimensions may reference none — fail closed.
---
## 5. BI-P012 — governance, and the partial-result trap
Every setting BI-P012 names is applied. Five ride on the statement itself
(`max_execution_time`, `max_result_rows`, `max_rows_to_read`,
`max_bytes_to_read`, `max_memory_usage`) under `readonly = 1`, so the query
text cannot re-raise them; `max_concurrent_queries` is a per-user quota plus a
local admission gate.
Overflow modes are **`throw`, never `break`**. That is not a style preference.
Measured on ClickHouse 24.8: with `result_overflow_mode = break` and
`max_result_rows = 10`, a 100 000-row query returns **HTTP 200 with 65 409
rows** and nothing in the payload says it is short. A truncated answer that
looks complete is the analytics form of a false green.
Query-bomb outcomes, each fired individually and identified by ClickHouse
error code: 158 `TOO_MANY_ROWS`, 307 `TOO_MANY_BYTES`, 159 `TIMEOUT_EXCEEDED`,
396 `TOO_MANY_ROWS_OR_BYTES`, 241 `MEMORY_LIMIT_EXCEEDED`.
The second half of the DoD — "without degrading unrelated tenant beyond
declared SLO" — is timed, not assumed: an unrelated tenant's interactive query
is run five times *while* an 8-second CPU bomb is in flight, must still return
the **correct** answer, and must stay under a declared 5000 ms ceiling (measured ~30 ms on an idle
server; the ceiling is set wide for a shared CI runner, and a genuinely
starved tenant would take seconds or fail).
Two measured details worth keeping:
* `EXPLAIN ESTIMATE` must **not** run under the caller's own
`max_rows_to_read`. It fails with code 158 exactly when the estimate would
be over budget — i.e. in the only case the estimate matters. Planner calls
use `planningSettings()`, a narrower envelope, instead.
* `SELECT count() FROM numbers(2e10)` is *not* a query bomb — ClickHouse
answers it in about 5 seconds because it can skip the values. The bomb in
the test hashes every row, which cannot be short-circuited.
---
## 6. BI-P005 — Arrow, and why it is not self-certified
Two paths:
* **pass-through** — ClickHouse emits Arrow IPC itself (`FORMAT ArrowStream`);
0pt streams those bytes unchanged. `output_format_arrow_string_as_string=1`
is required, or String columns arrive as Arrow **Binary** and both pyarrow
and DuckDB read them as bytes: silent semantic loss.
* **encoded** — for results 0pt assembles itself, `encodeArrowStream()` writes
the IPC stream, including a small hand-written FlatBuffers writer (no new
npm dependency, no new supply-chain surface).
A hand-rolled encoder is only worth having if somebody else's parser agrees
with it, so conformance is established by three independent implementations:
* **pyarrow 25.0.1** and **DuckDB 1.5.5** decode both ClickHouse's bytes and
0pt's bytes to *identical* values and types
(`evidence/bi-p002-p012/bi-p005-arrow-roundtrip.json`, verdict `PASS`);
* **ClickHouse's own Arrow C++** ingests 0pt-encoded Arrow — including a
20 000-row batch whose aggregates come back exact — in
`tests/integration/analytics-arrow-clickhouse.integration.test.ts`, which
runs on every push.
Supported types and their caveats are the table in `ARROW_TYPE_SUPPORT`;
`DateTime64(6|9)` and `UInt64` above 2^63 are deliberately **outside** the
supported set rather than silently lossy.
Arrow is also not an escape hatch from BI-P012:
`assertArrowWithinResultBudget()` decodes the row count and refuses a payload
over the caller's `max_result_rows`.
---
## 7. What is NOT done — verified, not assumed
* **There is no deployed ClickHouse for 0pt analytics.** Probed 2026-09-06 on
server7, the only ClickHouse in the estate: databases are
`opt_scraped`, `default`, `system`, `INFORMATION_SCHEMA`,
`information_schema`. There is no `opt_analytics` database, so
`ClickHouseAnalytics` — the "existing working" provider — has never been
run against a production instance. Everything in this document was proven
against disposable containers.
* **server7's ClickHouse must not be used for this.** It holds
`opt_scraped` — 51 tables, 407 793 471 rows, including `persons` and
`cz_owners`. That is the b0ts scraping plane, which
`src/analytics/clickhouse-analytics.ts`'s own header already prohibits this
module from touching (Ratsit/person-PII legal case, written DPIA
prohibition on quarantined market data). It also has **zero** columns named
`tenant_id` or `project_id` anywhere on the server and **zero** row
policies, so it could not satisfy BI-P011 even if it were legally
available. Nothing on that server was modified by this work.
* **BI-P011 layer 5** (separate database/cluster tier per high-isolation
customer) is supported by construction but not operated.
* **`executeQuery`** is refused pending BI-P017/BI-P033 (§2).
* The domain repository covers `AnalyticsDataset`, `DatasetVersion`,
`MetricDefinition`, `MetricVersion` and `DimensionDefinition`. The other
fourteen entities have tables, constraints and types but no repository
methods yet; their consumers (dashboards, reports, quality runs, lineage)
arrive with BI-P025 … BI-P035.
---
## 8. Running the proofs yourself
```bash
# unit suites (no infrastructure)
pnpm vitest run tests/analytics
# live ClickHouse suites
docker run -d --name opt-bi-ch -p 18123:8123 \
-e CLICKHOUSE_USER=opt_admin -e CLICKHOUSE_PASSWORD=<throwaway> \
-e CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1 clickhouse/clickhouse-server:24.8
CLICKHOUSE_URL=http://localhost:18123 CLICKHOUSE_AUTH=opt_admin:<throwaway> \
pnpm vitest run tests/integration
# BI-P002 restore drill
docker run -d --name opt-bi-pg -p 15455:5432 \
-e POSTGRES_USER=opt -e POSTGRES_PASSWORD=<throwaway> -e POSTGRES_DB=opt_bi_test \
postgres:16-alpine
ANALYTICS_TEST_DATABASE_URL=postgres://opt:<throwaway>@localhost:15455/opt_bi_test \
pnpm vitest run tests/analytics/analytics-domain-postgres.realinfra.test.ts
# BI-P010 benchmark and BI-P005 external roundtrip (rewrite the evidence files)
CLICKHOUSE_URL=... CLICKHOUSE_AUTH=... node scripts/bi-p010-benchmark.mjs
CLICKHOUSE_URL=... CLICKHOUSE_AUTH=... node scripts/bi-p005-arrow-roundtrip.mjs
```
Both scripts CREATE and DROP a database. Never point either at server7 or at
any ClickHouse holding b0ts or quarantined market data.
@@ -0,0 +1,257 @@
# BI-P060, P061, P103, P104, P105 — dashboards, SLOs, benchmarks, pre-aggregation
Extends the analytics core (`0pt#34`, BI-P002..P012) and the query/metric/quality
layer (`0pt#35`). Nothing there is rebuilt: `DashboardDefinition` and
`DashboardVersion` were already canonical entities with tables in `sql/008`, and
these prompts give them behaviour.
Organised, like its predecessor, around what is **proven** and what is **not**.
---
## Where things live
| prompt | module | live proof |
| ---------------------------- | ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| BI-P060 dashboard model | `src/analytics/dashboards/dashboard-document.ts` | unit (pure model + planner) |
| BI-P061 dashboard versioning | `src/analytics/dashboards/dashboard-release.ts`, `sql/010` | `tests/analytics/dashboard-release-postgres.realinfra.test.ts`, Postgres 16.15 |
| BI-P103 BI SLO | `src/analytics/slo/bi-slo.ts`, `slo/usage-events.ts` | unit; measurement path is `aud0#34`'s seven event types |
| BI-P104 benchmark harness | `src/analytics/bench/benchmark-harness.ts` | `tests/integration/bi-p104-benchmark.integration.test.ts`, ClickHouse 26.3.10.60 + DuckDB 1.5.5 |
| BI-P105 materialization | `src/analytics/materialization/` | `tests/integration/materialization-drift-clickhouse.integration.test.ts` |
Evidence: `evidence/bi-p061/dashboard-release-race.json`,
`evidence/bi-p104/benchmark.json`, `evidence/bi-p105/materialization-drift.json`.
---
## BI-P060 — the dashboard is a document, not a Superset export
`layout_json` was a string, and a string holds anything. The canonical document
carries `layout / filters / widgets / parameters / refresh / permissions /
version`, and a widget references **a metric, a contract query and a
visualization spec** — never a vendor's internal id.
The DoD is a function. `planDashboard(document, { parameters })` returns, per
widget, a runnable `AnalyticsQueryRequest` (BI-P016) plus a render spec. If it
returns a plan, the dashboard can be drawn by anything that executes the
contract and draws the twelve canonical visualization types.
Rules that are enforced rather than documented:
- a widget whose only content is a vendor id is **refused** — that is exactly
the document that cannot be reconstructed;
- `vendor_ref` may carry a Superset/Metabase id for round-tripping, but it is
excluded from the canonical hash, so re-exporting into a new Superset instance
does not fabricate a new dashboard version;
- vendor-native keys (`viz_type`, `form_data`, `position_json`, `slice_id`, …)
are refused with a message that names the rule;
- query widgets go through `parseAnalyticsQueryRequest`, so a dashboard cannot
become an SQL surface and cannot name a tenant;
- a locked filter cannot be switched off by a viewer — locked filters are how a
saved dashboard pins its scope, and removing one silently widens the result.
---
## BI-P061 — releases are a ledger
`DRAFT → PUBLISHED → SUPERSEDED`, stored in `opt_analytics_dashboard_release`
(`sql/010`) rather than as a column on the version row: `sql/008` already makes
`DashboardVersion` immutable at the database, and a lifecycle column there would
have meant weakening that. The **document** is frozen; the **release** moves.
The database enforces the state machine, not the application:
| attempted | result |
| ------------------------------------------- | -------------------------------------------------- |
| `published → draft` (un-publishing) | refused — it is how a rollback gets faked |
| `superseded → published` | refused |
| repointing a release at another version | refused (every content column is frozen on UPDATE) |
| rewriting `document_hash` or `published_at` | refused |
| `DELETE` on a release | refused — the ledger has no eraser |
| a `published` row with no `published_at` | refused by CHECK |
| a second open draft for one dashboard | refused by a partial unique index |
**Rollback mints a new version.** It does not resurrect the old release row.
"We went back to v3" becomes a timestamped fact (v5 carries
`rollback_of_version = 3`) instead of a hole in the history where v4 used to be.
The restore is verified by BI-P060 canonical hash, and `documentOf()` refuses to
serve a stored document whose hash no longer matches its release.
### The concurrency claim, measured both ways
`DashboardReleaseService.publish()` checks for a live release before writing.
That check is a read-then-write race and is **not** the guarantee.
`evidence/bi-p061/dashboard-release-race.json`, Postgres 16.15:
| run | partial unique index | transactions committed | published releases after |
| --------------- | -------------------- | ---------------------- | --------------------------- |
| `guard_absent` | dropped | 2 | **2 — two live dashboards** |
| `guard_present` | in place | 1 (loser `23505`) | 1 |
Reproduce with `node scripts/bi-p061-release-race.mjs`.
---
## BI-P103 — SLOs measured from AUD0, or declared unmeasured
The measurement path is `aud0#34`'s seven BI usage event types (BI-P076). aud0's
own doc records that zero producers emit them; `src/analytics/slo/usage-events.ts`
makes 0pt the producer. There is deliberately no second telemetry path — two
measurement systems disagree, and the disagreement surfaces during an incident.
| workload | objectives |
| --------------------- | ---------------------------------------------------------------- |
| interactive dashboard | availability, p95 first query, p95 cached, error rate, freshness |
| exploration | p95, p99, max result, timeout |
| report | completion SLA, delivery SLA |
Three verdicts that are not "met":
- **`insufficient_data`** — an availability objective over an empty window is
silence, not 100%. A started query with no terminal event is _unresolved_, not
a success.
- **`not_measurable_here`** — dashboard **freshness** is a property of the
dataset (BI-P026 / BI-P105 watermark) and report **delivery** is N0TIFY's
receipt. Neither exists in the seven event types, so both are declared
unmeasured rather than proxied by completion.
- **`low_confidence`** — a p99 over 12 samples is the 12th-slowest request
wearing a percentile's clothes. Reported with its sample size, not judged.
The producer is an allowlist built field by named field. `compiled_sql`,
`error_message`, `tenant_id`, filter **values** and result samples exist on the
0pt-side telemetry type and are never mapped; a `query_hash` that is not a
64-hex digest is dropped rather than reshaped.
### Honest status
`BI_SLO_CATALOGUE`'s targets are **starting thresholds, not derived baselines**.
No production BI traffic exists yet, so the first real baselines will come from
the first weeks of emitted events. The evaluator is built so that absence shows
up as `insufficient_data` until then, rather than as a green dashboard.
---
## BI-P104 — the harness refuses to quote one number
Six dataset profiles × two engines, 2 warmups discarded, 9 timed repetitions,
statistics from the shipped module. `evidence/bi-p104/benchmark.json`.
- **Overlapping ranges are `indistinguishable`**, whatever the medians say.
Three of the six real comparisons came out that way, one with a 1.6x median
gap — the headline a noisy host manufactures for you.
- **Every sentence carries the range, the sample size and the stability**, so it
cannot be trimmed to "4.4x faster" without visibly deleting the caveats.
- **Under 5 samples nothing is comparable.** A three-run benchmark is an anecdote.
- **Every engine BI-P104 names appears**, and the five nothing measured carry a
reason each.
Measured (warm cache, one idle host, these query shapes only):
| dataset | verdict, DuckDB 1.5.5 vs ClickHouse 26.3.10.60 |
| --------------- | ------------------------------------------------- |
| small_saas | faster, 1.6x–8.2x (median 4.4x) |
| medium_business | indistinguishable (medians 1.6x apart) |
| vehiq_market | faster, 2.4x–13.3x (median 5.8x) |
| large_event | **slower**, 0.2x–0.9x (`count DISTINCT`, 5M rows) |
| wide_table | indistinguishable |
| geospatial | indistinguishable |
Not exercised, with reasons in the evidence: **trino** (the running instance
belongs to another lane; benchmarking through it would measure their
configuration), **semantic_query** (MetricFlow has never executed in 0pt),
**superset**, **native_charts** (DES1GN owns rendering), **perspective**.
---
## BI-P105 — freshness semantics, and why freshness is not correctness
### The vocabulary
| term | definition |
| ------------------------------ | ------------------------------------------------------------------------------- |
| `data_watermark` | the newest **source** timestamp the aggregate represents. Absent = **UNKNOWN**. |
| staleness | `now − data_watermark`, in seconds. `undefined` when the watermark is unknown. |
| `max_staleness_seconds` | the freshness contract. Older than this and the aggregate stops serving. |
| `drift_check_interval_seconds` | how often the aggregate must be re-proved against its source to keep serving. |
| `last_drift_verdict` | `match` / `drifted` / `insufficient_data` from the last comparison. |
**UNKNOWN is never treated as current.** This is the same doctrine BI-P026
applies to result freshness: a missing watermark is a refusal, not a zero.
### The routing rule
A materialization serves **only** while it is provably current _and_ provably
undrifted. Eight named refusals, each a real way an aggregate becomes a
confident liar:
`coverage_miss` (metric/dimension/grain not carried — including a filter on a
dimension the aggregate already summed across) · `watermark_unknown` ·
`staleness_exceeded` · `window_beyond_watermark` (the question reaches past the
newest data; answering would silently shorten the period) ·
`drift_never_verified` · `drift_verification_stale` · `drift_detected` ·
`state_not_serving`.
`drift_never_verified` is the one people leave out and the one that matters on
launch day: a brand-new aggregate is exactly as unproven as a drifted one.
Grain rollup is restricted, not assumed: a `day` aggregate answers `day` and
coarser, never `hour`; a `week` aggregate answers only `week`, because weekly
buckets do not compose into calendar months.
### Drift detection
- **A missing bucket is drift** even when every value present is correct — late
arrivals and a materialized view created after the backfill both look like
this, and a value-only comparison passes while the total is wrong.
- **An extra bucket is drift** — the source rows are gone and the aggregate is not.
- **An empty comparison is `insufficient_data`, never `match`** (the BI-P025
false green, restated for pre-aggregates), and it takes the aggregate out of
service rather than annotating it.
- **Both tolerances must hold.** 0.1% of a 40M bucket is 40 000 kr of invisible
error; an absolute-only bound flags every rounding difference on a large bucket.
### The proof, on a live ClickHouse
`evidence/bi-p105/materialization-drift.json`, ClickHouse 26.3.10.60, 2M rows,
a real materialized view:
| step | result |
| ------------------------ | ----------------------------------------------------------------------------------- |
| after backfill | `match`, 48 buckets, 801 571 002 on both sides |
| GDPR erasure | `ALTER TABLE sales DELETE …`. An MV is an INSERT trigger and never sees a mutation. |
| what would be served | 801 571 002 against a true 761 580 554 — **39 990 448 overstated, 5.251%** |
| staleness at that moment | **1 second.** Every freshness check passed. |
| guarded router | `base_table` (`state_not_serving`) |
| counterfactual | a staleness-only router **serves it** — measured, not assumed |
| after rebuild | `match`, and routing returns to the materialization |
That last row of the counterfactual is the whole point of the prompt: staleness
is not correctness. The aggregate was one second old and wrong by 5%.
### Measured improvement, honestly
Same drill, base table vs pre-aggregate, 9 repetitions interleaved: p50 3.4x,
but the run-to-run range spans **0.6x–7.5x** on an otherwise idle host — one
sample had the pre-aggregate slower than the base table's best run. Quoting
"3.4x" as a property of the system would be misleading, which is why BI-P104's
harness exists.
---
## What is NOT proven here
- **No dashboard has been rendered.** BI-P060 produces a plan; drawing it is
DES1GN (`DES1GN#9`) and `siax-portal#5`. Nothing in this PR proves a pixel.
- **The release service has no Postgres adapter yet.** The lifecycle is proven
in-memory and the _schema_ is proven against live Postgres; wiring
`DashboardReleaseStore` to `postgres.js` is the next step and is the only
place where the two could drift apart.
- **Zero real SLO events have been emitted in production.** The producer and the
evaluator are tested; no BI surface calls the producer yet, so every objective
would today report `insufficient_data`. That is the honest empty state, and it
is deliberately visible rather than hidden behind a default.
- **No materialization is deployed.** The router and detector are proven against
a live ClickHouse in a throwaway database. Nothing schedules a refresh or a
periodic drift check yet — that belongs with ACT0/N0D scheduling.
- **Trino, Superset, MetricFlow and Perspective are unmeasured**, for the
reasons recorded in the benchmark evidence.
+160
View File
@@ -0,0 +1,160 @@
# BI-P014, P016–P022, P025–P026, P028–P031 — query, metric, quality
Extends the analytics core merged in `0pt#34` (BI-P002..P012). Nothing there is
rebuilt.
This document is organised around what is **proven** and what is **not**,
because the second list is the one that gets dropped.
---
## Where things live
| prompt | module | live proof |
| -------------------------------------- | ----------------------------------------------------------------------------------------- | ----------------------------------------------------------------- |
| BI-P014 DuckDB provider | `src/analytics/providers/duckdb-provider.ts` | `tests/duckdb/duckdb-provider.live.test.ts` |
| BI-P016 query request contract | `src/analytics/contract/query-request.ts`, `compile-clickhouse.ts`, `compile-semantic.ts` | unit only (pure compilation) |
| BI-P017 analyst SQL contract | `src/analytics/sql/` | `tests/integration/analyst-sql-clickhouse.integration.test.ts` |
| BI-P018 execution model | `src/analytics/execution/query-execution-record.ts` | `sql/009` + Postgres realinfra |
| BI-P019 cancellation | `src/analytics/execution/cancellation.ts` | `tests/integration/query-cancellation-budget.integration.test.ts` |
| BI-P020 query cache | `src/analytics/cache/query-cache.ts` | unit only (no engine involved) |
| BI-P021 cost / budget | `src/analytics/cost/query-budget.ts` | same live suite as BI-P019 |
| BI-P022 dbt contract | `src/analytics/transform/dbt-contract.ts`, `analytics/dbt/` | contract only — execution is `n0d#38` |
| BI-P025 quality provider | `src/analytics/quality/` | `tests/integration/data-quality-clickhouse.integration.test.ts` |
| BI-P026 freshness | `src/analytics/quality/freshness.ts` | unit + `sql/009` defaults |
| BI-P028/P031 metric model + governance | `src/analytics/metrics/metric-model.ts` | Postgres trigger, realinfra |
| BI-P029 semantic provider | `src/analytics/metrics/semantic-provider.ts` | **partially — see below** |
| BI-P030 semantic registry | `src/analytics/metrics/semantic-registry.ts` | unit + Postgres constraints |
---
## Five findings that changed the implementation
Each was found by writing a test that could fail, and each is pinned so it
cannot quietly stop being true.
### 1. ClickHouse `readonly = 1` on a user profile forbids **every** request setting
Measured on 24.8.14.39. A user whose profile carries `readonly = 1` answers
`Code: 164. Cannot modify 'max_rows_to_read' setting in readonly mode` to any
setting sent with the statement. So BI-P012's ceilings **cannot travel with an
analyst statement** — they are baked into the user's `SETTINGS PROFILE` at
provisioning. Proven both ways: an over-budget scan fails with code 158 from
the profile with no setting sent, and an attempt to raise it from the request
fails with 164.
### 2. `CREATE ROW POLICY IF NOT EXISTS` is a silent no-op on re-provisioning
Dropping a user empties the policy's apply-to list. The next provisioning run
sees the policy exists and does nothing, so it survives with
`apply_to_list = []` — applying to **nobody** — while `system.row_policies`
still shows the correct `select_filter`. The restricted user read 100 of 100
rows, **both tenants**, from the base table.
Fixed with `OR REPLACE`, plus `rowPolicyVerificationSql()` which turns "the
policy is applied" from an assumption into a query, and a test asserting
`applied_to_user = 1` per object. `CREATE SETTINGS PROFILE IF NOT EXISTS` has
the identical shape and is fixed the same way.
### 3. DuckDB's `allowed_directories` **alone** is a no-op
Measured on 1.5.5. With `allowed_directories` set to one mount directory and
nothing else, `read_csv('/etc/passwd')` **returned the file** and an `https://`
URL was fetched over the network. Only `enable_external_access = false`
alongside it makes the allow-list bind.
The obvious spelling of this control appears in `duckdb_settings()`, looks like
a sandbox, and permits everything. The measurement is pinned by a test that
never skips, so simplifying the bootstrap breaks the build rather than the
guarantee.
### 4. `pg_constraint.conname` is not schema-scoped
`SELECT 1 FROM pg_constraint WHERE conname = '...'` matches a same-named
constraint in **any** schema, so `sql/009` applied into a second schema
silently skipped every guarded `CHECK`. Every guard now scopes via
`conrelid = '<table>'::regclass`.
### 5. `describe.skipIf(flagSetInBeforeAll)` skips **always**
Vitest evaluates `skipIf` at collection time. Ten tests reported green while
skipping against a live, reachable ClickHouse. Replaced with the repo's
env-var gate, which **fails** when the variable is set and the server is down.
---
## What is NOT proven
- **BI-P029 / MetricFlow was never executed.** No Python runtime is
provisioned for this repo's CI, and BI-P029's "pin a currently verified
version" needs a licence and supply-chain review that has not happened.
`MetricFlowProvider.providerReadiness()` returns `verified: false` and names
all four open items. What _is_ proven is that the engine-neutral spec
compiles to a MetricFlow-shaped query and that canonical SIAX metric ids
survive a provider swap — tested against a second, deliberately
different-named provider so the equality is not trivially true.
- **The Superset integration path and the AI Analyst comparison** (BI-P029's
other two DoD surfaces) are not implemented.
- **dbt is not run here.** `DBT_RUNTIME_PIN` has both review flags `false`, and
`assertDbtRuntimeCertified` refuses a production run until they are true.
- **DuckDB Arrow IPC** is unproven: the `arrow` extension is a network install
the offline environment could not download. `readiness()` says so.
- **BI-P021's budget is a bound, not a hard cap.** In-flight enforcement is
granular to the poll interval, so the query overshoots between polls — and on
a columnar engine the overshoot is large. In the committed run
(`evidence/bi-p019-p021/`) a 1,000,000-row window cancelled the query at
**153,318,696 rows**, 55 ms in: a **153×** overshoot. The figure is
machine-dependent and moves between runs (an earlier run on the same box
recorded 131 M / 130×), which is itself the point — the bound is "roughly one
poll interval of engine throughput", not a number. It is still a bound: the
same query, unwatched, read **4,227,318,261 rows** before cancellation in the
BI-P019 case. Anyone quoting it as a hard cap is wrong.
- **The per-process window store does not bind across replicas.**
`describeEnforcement()` reports `"with N replicas a tenant can spend up to N
times this budget"`. A shared store (Valkey, or the CL0UD quota service)
behind `SharedWindowStore` is required for it to bind.
---
## Running the live suites
```bash
docker run -d --name ch -p 28123:8123 -e CLICKHOUSE_SKIP_USER_SETUP=1 \
clickhouse/clickhouse-server:24.8
docker run -d --name pg -p 25433:5432 -e POSTGRES_PASSWORD=test \
-e POSTGRES_DB=opt postgres:16-alpine
CLICKHOUSE_URL=http://127.0.0.1:28123 pnpm vitest run tests/integration
ANALYTICS_TEST_DATABASE_URL=postgres://postgres:test@127.0.0.1:25433/opt \
pnpm vitest run tests/analytics/analytics-governance-postgres.realinfra.test.ts
python -m venv .venv && .venv/bin/pip install duckdb pyarrow 'pyiceberg[sql-sqlite]'
DUCKDB_PYTHON=.venv/bin/python pnpm vitest run tests/duckdb
```
Every one of these **fails** rather than skipping when its variable is set and
the backend is unreachable.
---
## One note for whoever maintains `.gitea/workflows/ci.yml`
Not changed here — it is a protected file, and this branch touched none.
The integration job's guard is:
```sh
if grep -qE "skipped" /tmp/itest.log && ! grep -qE "Tests [0-9]+ passed" /tmp/itest.log
```
Vitest writes that summary with ANSI colour codes, so the literal text is
`Tests [22m [1m[32m81 passed[39m…`. The second pattern therefore **never
matches**, and the guard is effectively "any skip fails the job" rather than
the "skipped _and_ nothing passed" it reads as. That is stricter than intended,
not looser, so nothing is currently unguarded — but the escape hatch does not
work, and someone will eventually be puzzled by it. Piping through
`sed 's/\x1b\[[0-9;]*m//g'` before the grep would restore the intended
behaviour.
This is how the DuckDB suite's placement was caught, so the guard earned its
keep either way.