fix(recovery): restore source removed by production documentation merge
This commit is contained in:
@@ -0,0 +1,271 @@
|
||||
# 0pt Analytics Core — BI-P002 … BI-P012
|
||||
|
||||
Status of this document: it describes what **is in this repository and has been
|
||||
run**, and states separately what has **not** been deployed. Nothing here is a
|
||||
plan written in the present tense.
|
||||
|
||||
---
|
||||
|
||||
## 0. The ownership boundary (normative, not negotiable)
|
||||
|
||||
```
|
||||
SIAX / 0pt owns the analytics engine, the query compiler, the tenant forcing,
|
||||
the resource governance and the semantic/contract layer.
|
||||
|
||||
Product repos (VehIQ and the rest) own DOMAIN MEANING — what a row means,
|
||||
which metric a dealer cares about, how a screen reads.
|
||||
```
|
||||
|
||||
Concretely, and enforced rather than asserted:
|
||||
|
||||
* `BIApplicationBinding` (BI-P002 entity 19) has **no credential field**, in
|
||||
TypeScript or in `sql/008_analytics_domain_model.sql`. A bound BI
|
||||
application authenticates to the 0pt Analytics API with a CL0UD identity and
|
||||
receives compiled, tenant-forced results. It never learns a ClickHouse DSN,
|
||||
user, or password. Two tests assert the absence of such a column — one over
|
||||
the migration text, one over `information_schema.columns` on a live
|
||||
Postgres — so adding one later is a test failure.
|
||||
* `AnalyticsIdentity` (BI-P009) is a **separate parameter** from the request,
|
||||
never a field on it. This estate has already shipped a live cross-tenant
|
||||
hole by reading a tenant out of a request body (INF0 Wave 2); the type
|
||||
shape is what stops that recurring here.
|
||||
|
||||
---
|
||||
|
||||
## 1. What each prompt produced
|
||||
|
||||
| Prompt | Deliverable | Where |
|
||||
|---|---|---|
|
||||
| BI-P002 | 19 canonical entities + migrations + tenant-scoped persistence + restore drill | `src/analytics/domain/entities.ts`, `sql/008_analytics_domain_model.sql`, `src/analytics/repositories/analytics-domain-repository.ts` |
|
||||
| BI-P003 | RAW / NORMALIZED / CURATED / SERVING, with the BI-surface read rule enforced | `src/analytics/domain/tiers.ts` |
|
||||
| BI-P004 | `DatasetContract` + a fail-closed compatibility gate | `src/analytics/domain/contract.ts` |
|
||||
| BI-P005 | Arrow IPC encode/decode + ClickHouse pass-through | `src/analytics/arrow/ipc.ts`, `src/analytics/arrow/flatbuffer-builder.ts` |
|
||||
| BI-P009 | `SiaxAnalyticsEngine` interface + ClickHouse provider, existing implementation **preserved** | `src/analytics/engine/siax-analytics-engine.ts`, `src/analytics/engine/clickhouse-engine.ts` |
|
||||
| BI-P010 | Four purpose-built ClickHouse tables + rollup MV, benchmarked | `src/analytics/engine/schema.ts`, `scripts/bi-p010-benchmark.mjs` |
|
||||
| BI-P011 | Layered tenant isolation: verified identity, forced compiler predicate, restricted users, row policies | `src/analytics/engine/query-compiler.ts` |
|
||||
| BI-P012 | Every named ClickHouse limit + CL0UD quota + per-tenant admission control | `src/analytics/engine/resource-governance.ts` |
|
||||
|
||||
---
|
||||
|
||||
## 2. BI-P009 — what "preserve" means here
|
||||
|
||||
`src/analytics/clickhouse-analytics.ts` is **byte-identical to main**. The new
|
||||
engine composes it:
|
||||
|
||||
* `health()` delegates to it;
|
||||
* the pre-existing `opt_metric_events` path is reachable through
|
||||
`executeLegacyMetric()`, still using its own compiler;
|
||||
* everything else uses the new compiler for the BI-P010 tables.
|
||||
|
||||
"Do not regress the existing forced tenant predicate" is enforced by
|
||||
`tests/analytics/tenant-predicate-regression.test.ts`, which asserts the exact
|
||||
string `tenant_id = {tenant:String}` on the URL the adapter builds. It has been
|
||||
run with the forcing deliberately deleted from both compilers and goes RED
|
||||
(2 of 6 tests fail); the transcript is
|
||||
`evidence/bi-p002-p012/test-transcript.txt` §1–§2.
|
||||
|
||||
The eight capabilities are all implemented, with one **deliberate refusal**:
|
||||
`executeQuery` (analyst SQL) throws `CAPABILITY_UNAVAILABLE` until BI-P017
|
||||
(SQL dialect contract) and BI-P033 (analytics authorization) exist. Shipping a
|
||||
raw-SQL surface first would create exactly the "raw SQL attempt" hole BI-P011
|
||||
enumerates. It is declared and refused, not silently missing.
|
||||
|
||||
---
|
||||
|
||||
## 3. BI-P010 — why four tables, measured
|
||||
|
||||
BI-P010 warns against "one giant unbounded table for every analytics use
|
||||
case". `scripts/bi-p010-benchmark.mjs` builds both fixtures and the
|
||||
counterfactual, and times them. Measured on a disposable
|
||||
`clickhouse/clickhouse-server:24.8` (full transcript:
|
||||
`evidence/bi-p002-p012/bi-p010-benchmark.json`):
|
||||
|
||||
| Question | Purpose-built | One giant table |
|
||||
|---|---|---|
|
||||
| 90-day metric total, one tenant | **48.8 ms p50**, 40 960 rows read | 174.4 ms p50, **5 400 000** rows read |
|
||||
| Vehicle point lookup by key, one tenant | **49.3 ms p50**, 8 192 rows read | 236.6 ms p50, **7 401 152** rows read |
|
||||
| Avg asking price by make over 2M vehicles | **389.6 ms p50** | 680.8 ms p50 |
|
||||
| Granules read for the metric total | **5 / 660** | 660 / 1 149 |
|
||||
|
||||
Fixtures: 5 400 000 SaaS events (200 tenants × 12 metrics × 90 days) and
|
||||
4 000 000 vehicle projections (2M for the tenant under test + 2M decoys owned
|
||||
by 50 other tenants).
|
||||
|
||||
**Honest finding that does not flatter the design:** at this scale the
|
||||
`AggregatingMergeTree` rollup (37.98 ms p50, 21 416 rows read) was *not* faster
|
||||
in wall time than scanning raw events (32.55 ms p50, 40 960 rows read). It
|
||||
reads about half the rows, and its value is the long TTL — raw events expire
|
||||
in 30 days, the rollup keeps three years — so the rollup earns its keep on
|
||||
retention and on scale beyond this fixture, not on latency at 5.4M rows. Do
|
||||
not quote it as a speed win.
|
||||
|
||||
---
|
||||
|
||||
## 4. BI-P011 — the isolation proof
|
||||
|
||||
Four layers, and each one is tested with the layer above it *removed*:
|
||||
|
||||
1. **Verified CL0UD identity.** `AnalyticsIdentity` is not part of the
|
||||
request; a blank tenant or principal is `UNAUTHENTICATED`, never a wildcard.
|
||||
2. **The 0pt query compiler forces the tenant.** Every compiled statement
|
||||
contains `tenant_id = {tenant:String}` as its first WHERE condition, with
|
||||
the value bound as an HTTP parameter — the tenant string never appears in
|
||||
the SQL text.
|
||||
3. **Restricted database users + row policies.**
|
||||
`restrictedTenantUserDdl()` renders a per-tenant ClickHouse user with
|
||||
`readonly = 1` and a `CREATE ROW POLICY ... USING tenant_id = '<tenant>'`
|
||||
on every analytics table.
|
||||
4. **Separate database/cluster tier** for very-high-isolation customers:
|
||||
supported by construction — the engine takes a `database` and a `baseUrl`
|
||||
per instance — but **not yet operated**, because there is no deployed
|
||||
ClickHouse for 0pt analytics at all (see §7).
|
||||
|
||||
The proof, run against a disposable ClickHouse with 50 rows for `t_alpha`
|
||||
(value 1 each) and 50 for `t_beta` (value 1000 each):
|
||||
|
||||
* **hole, layer 2 removed:** the same statement minus the forced predicate
|
||||
returns **50 050** — both tenants in one number;
|
||||
* **closed, layer 2 present:** `t_alpha` gets **50**, `t_beta` gets **50 000**;
|
||||
* **hole, layer 3 absent:** an unrestricted user issuing a predicate-free
|
||||
`uniqExact(tenant_id)` sees **2** tenants;
|
||||
* **closed, layer 3 present:** the restricted user issuing the *identical*
|
||||
statement sees **1**, and `sum(value)` returns 50 rather than 50 050.
|
||||
|
||||
Adversarial cases from BI-P011's list — tenant A querying B's dataset, forged
|
||||
dataset IDs, raw SQL, SQL injection, dimension identifier injection, filter
|
||||
injection — are covered in
|
||||
`tests/analytics/query-compiler-injection.test.ts` (compiler layer) and
|
||||
`tests/integration/analytics-tenant-isolation.integration.test.ts` (live).
|
||||
|
||||
Dimension names are **resolved through a per-tenant allow-list**
|
||||
(`DimensionDefinition`), never escaped and never spliced. A tenant that has
|
||||
declared no dimensions may reference none — fail closed.
|
||||
|
||||
---
|
||||
|
||||
## 5. BI-P012 — governance, and the partial-result trap
|
||||
|
||||
Every setting BI-P012 names is applied. Five ride on the statement itself
|
||||
(`max_execution_time`, `max_result_rows`, `max_rows_to_read`,
|
||||
`max_bytes_to_read`, `max_memory_usage`) under `readonly = 1`, so the query
|
||||
text cannot re-raise them; `max_concurrent_queries` is a per-user quota plus a
|
||||
local admission gate.
|
||||
|
||||
Overflow modes are **`throw`, never `break`**. That is not a style preference.
|
||||
Measured on ClickHouse 24.8: with `result_overflow_mode = break` and
|
||||
`max_result_rows = 10`, a 100 000-row query returns **HTTP 200 with 65 409
|
||||
rows** and nothing in the payload says it is short. A truncated answer that
|
||||
looks complete is the analytics form of a false green.
|
||||
|
||||
Query-bomb outcomes, each fired individually and identified by ClickHouse
|
||||
error code: 158 `TOO_MANY_ROWS`, 307 `TOO_MANY_BYTES`, 159 `TIMEOUT_EXCEEDED`,
|
||||
396 `TOO_MANY_ROWS_OR_BYTES`, 241 `MEMORY_LIMIT_EXCEEDED`.
|
||||
|
||||
The second half of the DoD — "without degrading unrelated tenant beyond
|
||||
declared SLO" — is timed, not assumed: an unrelated tenant's interactive query
|
||||
is run five times *while* an 8-second CPU bomb is in flight, must still return
|
||||
the **correct** answer, and must stay under a declared 5000 ms ceiling (measured ~30 ms on an idle
|
||||
server; the ceiling is set wide for a shared CI runner, and a genuinely
|
||||
starved tenant would take seconds or fail).
|
||||
|
||||
Two measured details worth keeping:
|
||||
|
||||
* `EXPLAIN ESTIMATE` must **not** run under the caller's own
|
||||
`max_rows_to_read`. It fails with code 158 exactly when the estimate would
|
||||
be over budget — i.e. in the only case the estimate matters. Planner calls
|
||||
use `planningSettings()`, a narrower envelope, instead.
|
||||
* `SELECT count() FROM numbers(2e10)` is *not* a query bomb — ClickHouse
|
||||
answers it in about 5 seconds because it can skip the values. The bomb in
|
||||
the test hashes every row, which cannot be short-circuited.
|
||||
|
||||
---
|
||||
|
||||
## 6. BI-P005 — Arrow, and why it is not self-certified
|
||||
|
||||
Two paths:
|
||||
|
||||
* **pass-through** — ClickHouse emits Arrow IPC itself (`FORMAT ArrowStream`);
|
||||
0pt streams those bytes unchanged. `output_format_arrow_string_as_string=1`
|
||||
is required, or String columns arrive as Arrow **Binary** and both pyarrow
|
||||
and DuckDB read them as bytes: silent semantic loss.
|
||||
* **encoded** — for results 0pt assembles itself, `encodeArrowStream()` writes
|
||||
the IPC stream, including a small hand-written FlatBuffers writer (no new
|
||||
npm dependency, no new supply-chain surface).
|
||||
|
||||
A hand-rolled encoder is only worth having if somebody else's parser agrees
|
||||
with it, so conformance is established by three independent implementations:
|
||||
|
||||
* **pyarrow 25.0.1** and **DuckDB 1.5.5** decode both ClickHouse's bytes and
|
||||
0pt's bytes to *identical* values and types
|
||||
(`evidence/bi-p002-p012/bi-p005-arrow-roundtrip.json`, verdict `PASS`);
|
||||
* **ClickHouse's own Arrow C++** ingests 0pt-encoded Arrow — including a
|
||||
20 000-row batch whose aggregates come back exact — in
|
||||
`tests/integration/analytics-arrow-clickhouse.integration.test.ts`, which
|
||||
runs on every push.
|
||||
|
||||
Supported types and their caveats are the table in `ARROW_TYPE_SUPPORT`;
|
||||
`DateTime64(6|9)` and `UInt64` above 2^63 are deliberately **outside** the
|
||||
supported set rather than silently lossy.
|
||||
|
||||
Arrow is also not an escape hatch from BI-P012:
|
||||
`assertArrowWithinResultBudget()` decodes the row count and refuses a payload
|
||||
over the caller's `max_result_rows`.
|
||||
|
||||
---
|
||||
|
||||
## 7. What is NOT done — verified, not assumed
|
||||
|
||||
* **There is no deployed ClickHouse for 0pt analytics.** Probed 2026-09-06 on
|
||||
server7, the only ClickHouse in the estate: databases are
|
||||
`opt_scraped`, `default`, `system`, `INFORMATION_SCHEMA`,
|
||||
`information_schema`. There is no `opt_analytics` database, so
|
||||
`ClickHouseAnalytics` — the "existing working" provider — has never been
|
||||
run against a production instance. Everything in this document was proven
|
||||
against disposable containers.
|
||||
* **server7's ClickHouse must not be used for this.** It holds
|
||||
`opt_scraped` — 51 tables, 407 793 471 rows, including `persons` and
|
||||
`cz_owners`. That is the b0ts scraping plane, which
|
||||
`src/analytics/clickhouse-analytics.ts`'s own header already prohibits this
|
||||
module from touching (Ratsit/person-PII legal case, written DPIA
|
||||
prohibition on quarantined market data). It also has **zero** columns named
|
||||
`tenant_id` or `project_id` anywhere on the server and **zero** row
|
||||
policies, so it could not satisfy BI-P011 even if it were legally
|
||||
available. Nothing on that server was modified by this work.
|
||||
* **BI-P011 layer 5** (separate database/cluster tier per high-isolation
|
||||
customer) is supported by construction but not operated.
|
||||
* **`executeQuery`** is refused pending BI-P017/BI-P033 (§2).
|
||||
* The domain repository covers `AnalyticsDataset`, `DatasetVersion`,
|
||||
`MetricDefinition`, `MetricVersion` and `DimensionDefinition`. The other
|
||||
fourteen entities have tables, constraints and types but no repository
|
||||
methods yet; their consumers (dashboards, reports, quality runs, lineage)
|
||||
arrive with BI-P025 … BI-P035.
|
||||
|
||||
---
|
||||
|
||||
## 8. Running the proofs yourself
|
||||
|
||||
```bash
|
||||
# unit suites (no infrastructure)
|
||||
pnpm vitest run tests/analytics
|
||||
|
||||
# live ClickHouse suites
|
||||
docker run -d --name opt-bi-ch -p 18123:8123 \
|
||||
-e CLICKHOUSE_USER=opt_admin -e CLICKHOUSE_PASSWORD=<throwaway> \
|
||||
-e CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1 clickhouse/clickhouse-server:24.8
|
||||
CLICKHOUSE_URL=http://localhost:18123 CLICKHOUSE_AUTH=opt_admin:<throwaway> \
|
||||
pnpm vitest run tests/integration
|
||||
|
||||
# BI-P002 restore drill
|
||||
docker run -d --name opt-bi-pg -p 15455:5432 \
|
||||
-e POSTGRES_USER=opt -e POSTGRES_PASSWORD=<throwaway> -e POSTGRES_DB=opt_bi_test \
|
||||
postgres:16-alpine
|
||||
ANALYTICS_TEST_DATABASE_URL=postgres://opt:<throwaway>@localhost:15455/opt_bi_test \
|
||||
pnpm vitest run tests/analytics/analytics-domain-postgres.realinfra.test.ts
|
||||
|
||||
# BI-P010 benchmark and BI-P005 external roundtrip (rewrite the evidence files)
|
||||
CLICKHOUSE_URL=... CLICKHOUSE_AUTH=... node scripts/bi-p010-benchmark.mjs
|
||||
CLICKHOUSE_URL=... CLICKHOUSE_AUTH=... node scripts/bi-p005-arrow-roundtrip.mjs
|
||||
```
|
||||
|
||||
Both scripts CREATE and DROP a database. Never point either at server7 or at
|
||||
any ClickHouse holding b0ts or quarantined market data.
|
||||
@@ -0,0 +1,257 @@
|
||||
# BI-P060, P061, P103, P104, P105 — dashboards, SLOs, benchmarks, pre-aggregation
|
||||
|
||||
Extends the analytics core (`0pt#34`, BI-P002..P012) and the query/metric/quality
|
||||
layer (`0pt#35`). Nothing there is rebuilt: `DashboardDefinition` and
|
||||
`DashboardVersion` were already canonical entities with tables in `sql/008`, and
|
||||
these prompts give them behaviour.
|
||||
|
||||
Organised, like its predecessor, around what is **proven** and what is **not**.
|
||||
|
||||
---
|
||||
|
||||
## Where things live
|
||||
|
||||
| prompt | module | live proof |
|
||||
| ---------------------------- | ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
|
||||
| BI-P060 dashboard model | `src/analytics/dashboards/dashboard-document.ts` | unit (pure model + planner) |
|
||||
| BI-P061 dashboard versioning | `src/analytics/dashboards/dashboard-release.ts`, `sql/010` | `tests/analytics/dashboard-release-postgres.realinfra.test.ts`, Postgres 16.15 |
|
||||
| BI-P103 BI SLO | `src/analytics/slo/bi-slo.ts`, `slo/usage-events.ts` | unit; measurement path is `aud0#34`'s seven event types |
|
||||
| BI-P104 benchmark harness | `src/analytics/bench/benchmark-harness.ts` | `tests/integration/bi-p104-benchmark.integration.test.ts`, ClickHouse 26.3.10.60 + DuckDB 1.5.5 |
|
||||
| BI-P105 materialization | `src/analytics/materialization/` | `tests/integration/materialization-drift-clickhouse.integration.test.ts` |
|
||||
|
||||
Evidence: `evidence/bi-p061/dashboard-release-race.json`,
|
||||
`evidence/bi-p104/benchmark.json`, `evidence/bi-p105/materialization-drift.json`.
|
||||
|
||||
---
|
||||
|
||||
## BI-P060 — the dashboard is a document, not a Superset export
|
||||
|
||||
`layout_json` was a string, and a string holds anything. The canonical document
|
||||
carries `layout / filters / widgets / parameters / refresh / permissions /
|
||||
version`, and a widget references **a metric, a contract query and a
|
||||
visualization spec** — never a vendor's internal id.
|
||||
|
||||
The DoD is a function. `planDashboard(document, { parameters })` returns, per
|
||||
widget, a runnable `AnalyticsQueryRequest` (BI-P016) plus a render spec. If it
|
||||
returns a plan, the dashboard can be drawn by anything that executes the
|
||||
contract and draws the twelve canonical visualization types.
|
||||
|
||||
Rules that are enforced rather than documented:
|
||||
|
||||
- a widget whose only content is a vendor id is **refused** — that is exactly
|
||||
the document that cannot be reconstructed;
|
||||
- `vendor_ref` may carry a Superset/Metabase id for round-tripping, but it is
|
||||
excluded from the canonical hash, so re-exporting into a new Superset instance
|
||||
does not fabricate a new dashboard version;
|
||||
- vendor-native keys (`viz_type`, `form_data`, `position_json`, `slice_id`, …)
|
||||
are refused with a message that names the rule;
|
||||
- query widgets go through `parseAnalyticsQueryRequest`, so a dashboard cannot
|
||||
become an SQL surface and cannot name a tenant;
|
||||
- a locked filter cannot be switched off by a viewer — locked filters are how a
|
||||
saved dashboard pins its scope, and removing one silently widens the result.
|
||||
|
||||
---
|
||||
|
||||
## BI-P061 — releases are a ledger
|
||||
|
||||
`DRAFT → PUBLISHED → SUPERSEDED`, stored in `opt_analytics_dashboard_release`
|
||||
(`sql/010`) rather than as a column on the version row: `sql/008` already makes
|
||||
`DashboardVersion` immutable at the database, and a lifecycle column there would
|
||||
have meant weakening that. The **document** is frozen; the **release** moves.
|
||||
|
||||
The database enforces the state machine, not the application:
|
||||
|
||||
| attempted | result |
|
||||
| ------------------------------------------- | -------------------------------------------------- |
|
||||
| `published → draft` (un-publishing) | refused — it is how a rollback gets faked |
|
||||
| `superseded → published` | refused |
|
||||
| repointing a release at another version | refused (every content column is frozen on UPDATE) |
|
||||
| rewriting `document_hash` or `published_at` | refused |
|
||||
| `DELETE` on a release | refused — the ledger has no eraser |
|
||||
| a `published` row with no `published_at` | refused by CHECK |
|
||||
| a second open draft for one dashboard | refused by a partial unique index |
|
||||
|
||||
**Rollback mints a new version.** It does not resurrect the old release row.
|
||||
"We went back to v3" becomes a timestamped fact (v5 carries
|
||||
`rollback_of_version = 3`) instead of a hole in the history where v4 used to be.
|
||||
The restore is verified by BI-P060 canonical hash, and `documentOf()` refuses to
|
||||
serve a stored document whose hash no longer matches its release.
|
||||
|
||||
### The concurrency claim, measured both ways
|
||||
|
||||
`DashboardReleaseService.publish()` checks for a live release before writing.
|
||||
That check is a read-then-write race and is **not** the guarantee.
|
||||
`evidence/bi-p061/dashboard-release-race.json`, Postgres 16.15:
|
||||
|
||||
| run | partial unique index | transactions committed | published releases after |
|
||||
| --------------- | -------------------- | ---------------------- | --------------------------- |
|
||||
| `guard_absent` | dropped | 2 | **2 — two live dashboards** |
|
||||
| `guard_present` | in place | 1 (loser `23505`) | 1 |
|
||||
|
||||
Reproduce with `node scripts/bi-p061-release-race.mjs`.
|
||||
|
||||
---
|
||||
|
||||
## BI-P103 — SLOs measured from AUD0, or declared unmeasured
|
||||
|
||||
The measurement path is `aud0#34`'s seven BI usage event types (BI-P076). aud0's
|
||||
own doc records that zero producers emit them; `src/analytics/slo/usage-events.ts`
|
||||
makes 0pt the producer. There is deliberately no second telemetry path — two
|
||||
measurement systems disagree, and the disagreement surfaces during an incident.
|
||||
|
||||
| workload | objectives |
|
||||
| --------------------- | ---------------------------------------------------------------- |
|
||||
| interactive dashboard | availability, p95 first query, p95 cached, error rate, freshness |
|
||||
| exploration | p95, p99, max result, timeout |
|
||||
| report | completion SLA, delivery SLA |
|
||||
|
||||
Three verdicts that are not "met":
|
||||
|
||||
- **`insufficient_data`** — an availability objective over an empty window is
|
||||
silence, not 100%. A started query with no terminal event is _unresolved_, not
|
||||
a success.
|
||||
- **`not_measurable_here`** — dashboard **freshness** is a property of the
|
||||
dataset (BI-P026 / BI-P105 watermark) and report **delivery** is N0TIFY's
|
||||
receipt. Neither exists in the seven event types, so both are declared
|
||||
unmeasured rather than proxied by completion.
|
||||
- **`low_confidence`** — a p99 over 12 samples is the 12th-slowest request
|
||||
wearing a percentile's clothes. Reported with its sample size, not judged.
|
||||
|
||||
The producer is an allowlist built field by named field. `compiled_sql`,
|
||||
`error_message`, `tenant_id`, filter **values** and result samples exist on the
|
||||
0pt-side telemetry type and are never mapped; a `query_hash` that is not a
|
||||
64-hex digest is dropped rather than reshaped.
|
||||
|
||||
### Honest status
|
||||
|
||||
`BI_SLO_CATALOGUE`'s targets are **starting thresholds, not derived baselines**.
|
||||
No production BI traffic exists yet, so the first real baselines will come from
|
||||
the first weeks of emitted events. The evaluator is built so that absence shows
|
||||
up as `insufficient_data` until then, rather than as a green dashboard.
|
||||
|
||||
---
|
||||
|
||||
## BI-P104 — the harness refuses to quote one number
|
||||
|
||||
Six dataset profiles × two engines, 2 warmups discarded, 9 timed repetitions,
|
||||
statistics from the shipped module. `evidence/bi-p104/benchmark.json`.
|
||||
|
||||
- **Overlapping ranges are `indistinguishable`**, whatever the medians say.
|
||||
Three of the six real comparisons came out that way, one with a 1.6x median
|
||||
gap — the headline a noisy host manufactures for you.
|
||||
- **Every sentence carries the range, the sample size and the stability**, so it
|
||||
cannot be trimmed to "4.4x faster" without visibly deleting the caveats.
|
||||
- **Under 5 samples nothing is comparable.** A three-run benchmark is an anecdote.
|
||||
- **Every engine BI-P104 names appears**, and the five nothing measured carry a
|
||||
reason each.
|
||||
|
||||
Measured (warm cache, one idle host, these query shapes only):
|
||||
|
||||
| dataset | verdict, DuckDB 1.5.5 vs ClickHouse 26.3.10.60 |
|
||||
| --------------- | ------------------------------------------------- |
|
||||
| small_saas | faster, 1.6x–8.2x (median 4.4x) |
|
||||
| medium_business | indistinguishable (medians 1.6x apart) |
|
||||
| vehiq_market | faster, 2.4x–13.3x (median 5.8x) |
|
||||
| large_event | **slower**, 0.2x–0.9x (`count DISTINCT`, 5M rows) |
|
||||
| wide_table | indistinguishable |
|
||||
| geospatial | indistinguishable |
|
||||
|
||||
Not exercised, with reasons in the evidence: **trino** (the running instance
|
||||
belongs to another lane; benchmarking through it would measure their
|
||||
configuration), **semantic_query** (MetricFlow has never executed in 0pt),
|
||||
**superset**, **native_charts** (DES1GN owns rendering), **perspective**.
|
||||
|
||||
---
|
||||
|
||||
## BI-P105 — freshness semantics, and why freshness is not correctness
|
||||
|
||||
### The vocabulary
|
||||
|
||||
| term | definition |
|
||||
| ------------------------------ | ------------------------------------------------------------------------------- |
|
||||
| `data_watermark` | the newest **source** timestamp the aggregate represents. Absent = **UNKNOWN**. |
|
||||
| staleness | `now − data_watermark`, in seconds. `undefined` when the watermark is unknown. |
|
||||
| `max_staleness_seconds` | the freshness contract. Older than this and the aggregate stops serving. |
|
||||
| `drift_check_interval_seconds` | how often the aggregate must be re-proved against its source to keep serving. |
|
||||
| `last_drift_verdict` | `match` / `drifted` / `insufficient_data` from the last comparison. |
|
||||
|
||||
**UNKNOWN is never treated as current.** This is the same doctrine BI-P026
|
||||
applies to result freshness: a missing watermark is a refusal, not a zero.
|
||||
|
||||
### The routing rule
|
||||
|
||||
A materialization serves **only** while it is provably current _and_ provably
|
||||
undrifted. Eight named refusals, each a real way an aggregate becomes a
|
||||
confident liar:
|
||||
|
||||
`coverage_miss` (metric/dimension/grain not carried — including a filter on a
|
||||
dimension the aggregate already summed across) · `watermark_unknown` ·
|
||||
`staleness_exceeded` · `window_beyond_watermark` (the question reaches past the
|
||||
newest data; answering would silently shorten the period) ·
|
||||
`drift_never_verified` · `drift_verification_stale` · `drift_detected` ·
|
||||
`state_not_serving`.
|
||||
|
||||
`drift_never_verified` is the one people leave out and the one that matters on
|
||||
launch day: a brand-new aggregate is exactly as unproven as a drifted one.
|
||||
|
||||
Grain rollup is restricted, not assumed: a `day` aggregate answers `day` and
|
||||
coarser, never `hour`; a `week` aggregate answers only `week`, because weekly
|
||||
buckets do not compose into calendar months.
|
||||
|
||||
### Drift detection
|
||||
|
||||
- **A missing bucket is drift** even when every value present is correct — late
|
||||
arrivals and a materialized view created after the backfill both look like
|
||||
this, and a value-only comparison passes while the total is wrong.
|
||||
- **An extra bucket is drift** — the source rows are gone and the aggregate is not.
|
||||
- **An empty comparison is `insufficient_data`, never `match`** (the BI-P025
|
||||
false green, restated for pre-aggregates), and it takes the aggregate out of
|
||||
service rather than annotating it.
|
||||
- **Both tolerances must hold.** 0.1% of a 40M bucket is 40 000 kr of invisible
|
||||
error; an absolute-only bound flags every rounding difference on a large bucket.
|
||||
|
||||
### The proof, on a live ClickHouse
|
||||
|
||||
`evidence/bi-p105/materialization-drift.json`, ClickHouse 26.3.10.60, 2M rows,
|
||||
a real materialized view:
|
||||
|
||||
| step | result |
|
||||
| ------------------------ | ----------------------------------------------------------------------------------- |
|
||||
| after backfill | `match`, 48 buckets, 801 571 002 on both sides |
|
||||
| GDPR erasure | `ALTER TABLE sales DELETE …`. An MV is an INSERT trigger and never sees a mutation. |
|
||||
| what would be served | 801 571 002 against a true 761 580 554 — **39 990 448 overstated, 5.251%** |
|
||||
| staleness at that moment | **1 second.** Every freshness check passed. |
|
||||
| guarded router | `base_table` (`state_not_serving`) |
|
||||
| counterfactual | a staleness-only router **serves it** — measured, not assumed |
|
||||
| after rebuild | `match`, and routing returns to the materialization |
|
||||
|
||||
That last row of the counterfactual is the whole point of the prompt: staleness
|
||||
is not correctness. The aggregate was one second old and wrong by 5%.
|
||||
|
||||
### Measured improvement, honestly
|
||||
|
||||
Same drill, base table vs pre-aggregate, 9 repetitions interleaved: p50 3.4x,
|
||||
but the run-to-run range spans **0.6x–7.5x** on an otherwise idle host — one
|
||||
sample had the pre-aggregate slower than the base table's best run. Quoting
|
||||
"3.4x" as a property of the system would be misleading, which is why BI-P104's
|
||||
harness exists.
|
||||
|
||||
---
|
||||
|
||||
## What is NOT proven here
|
||||
|
||||
- **No dashboard has been rendered.** BI-P060 produces a plan; drawing it is
|
||||
DES1GN (`DES1GN#9`) and `siax-portal#5`. Nothing in this PR proves a pixel.
|
||||
- **The release service has no Postgres adapter yet.** The lifecycle is proven
|
||||
in-memory and the _schema_ is proven against live Postgres; wiring
|
||||
`DashboardReleaseStore` to `postgres.js` is the next step and is the only
|
||||
place where the two could drift apart.
|
||||
- **Zero real SLO events have been emitted in production.** The producer and the
|
||||
evaluator are tested; no BI surface calls the producer yet, so every objective
|
||||
would today report `insufficient_data`. That is the honest empty state, and it
|
||||
is deliberately visible rather than hidden behind a default.
|
||||
- **No materialization is deployed.** The router and detector are proven against
|
||||
a live ClickHouse in a throwaway database. Nothing schedules a refresh or a
|
||||
periodic drift check yet — that belongs with ACT0/N0D scheduling.
|
||||
- **Trino, Superset, MetricFlow and Perspective are unmeasured**, for the
|
||||
reasons recorded in the benchmark evidence.
|
||||
@@ -0,0 +1,160 @@
|
||||
# BI-P014, P016–P022, P025–P026, P028–P031 — query, metric, quality
|
||||
|
||||
Extends the analytics core merged in `0pt#34` (BI-P002..P012). Nothing there is
|
||||
rebuilt.
|
||||
|
||||
This document is organised around what is **proven** and what is **not**,
|
||||
because the second list is the one that gets dropped.
|
||||
|
||||
---
|
||||
|
||||
## Where things live
|
||||
|
||||
| prompt | module | live proof |
|
||||
| -------------------------------------- | ----------------------------------------------------------------------------------------- | ----------------------------------------------------------------- |
|
||||
| BI-P014 DuckDB provider | `src/analytics/providers/duckdb-provider.ts` | `tests/duckdb/duckdb-provider.live.test.ts` |
|
||||
| BI-P016 query request contract | `src/analytics/contract/query-request.ts`, `compile-clickhouse.ts`, `compile-semantic.ts` | unit only (pure compilation) |
|
||||
| BI-P017 analyst SQL contract | `src/analytics/sql/` | `tests/integration/analyst-sql-clickhouse.integration.test.ts` |
|
||||
| BI-P018 execution model | `src/analytics/execution/query-execution-record.ts` | `sql/009` + Postgres realinfra |
|
||||
| BI-P019 cancellation | `src/analytics/execution/cancellation.ts` | `tests/integration/query-cancellation-budget.integration.test.ts` |
|
||||
| BI-P020 query cache | `src/analytics/cache/query-cache.ts` | unit only (no engine involved) |
|
||||
| BI-P021 cost / budget | `src/analytics/cost/query-budget.ts` | same live suite as BI-P019 |
|
||||
| BI-P022 dbt contract | `src/analytics/transform/dbt-contract.ts`, `analytics/dbt/` | contract only — execution is `n0d#38` |
|
||||
| BI-P025 quality provider | `src/analytics/quality/` | `tests/integration/data-quality-clickhouse.integration.test.ts` |
|
||||
| BI-P026 freshness | `src/analytics/quality/freshness.ts` | unit + `sql/009` defaults |
|
||||
| BI-P028/P031 metric model + governance | `src/analytics/metrics/metric-model.ts` | Postgres trigger, realinfra |
|
||||
| BI-P029 semantic provider | `src/analytics/metrics/semantic-provider.ts` | **partially — see below** |
|
||||
| BI-P030 semantic registry | `src/analytics/metrics/semantic-registry.ts` | unit + Postgres constraints |
|
||||
|
||||
---
|
||||
|
||||
## Five findings that changed the implementation
|
||||
|
||||
Each was found by writing a test that could fail, and each is pinned so it
|
||||
cannot quietly stop being true.
|
||||
|
||||
### 1. ClickHouse `readonly = 1` on a user profile forbids **every** request setting
|
||||
|
||||
Measured on 24.8.14.39. A user whose profile carries `readonly = 1` answers
|
||||
`Code: 164. Cannot modify 'max_rows_to_read' setting in readonly mode` to any
|
||||
setting sent with the statement. So BI-P012's ceilings **cannot travel with an
|
||||
analyst statement** — they are baked into the user's `SETTINGS PROFILE` at
|
||||
provisioning. Proven both ways: an over-budget scan fails with code 158 from
|
||||
the profile with no setting sent, and an attempt to raise it from the request
|
||||
fails with 164.
|
||||
|
||||
### 2. `CREATE ROW POLICY IF NOT EXISTS` is a silent no-op on re-provisioning
|
||||
|
||||
Dropping a user empties the policy's apply-to list. The next provisioning run
|
||||
sees the policy exists and does nothing, so it survives with
|
||||
`apply_to_list = []` — applying to **nobody** — while `system.row_policies`
|
||||
still shows the correct `select_filter`. The restricted user read 100 of 100
|
||||
rows, **both tenants**, from the base table.
|
||||
|
||||
Fixed with `OR REPLACE`, plus `rowPolicyVerificationSql()` which turns "the
|
||||
policy is applied" from an assumption into a query, and a test asserting
|
||||
`applied_to_user = 1` per object. `CREATE SETTINGS PROFILE IF NOT EXISTS` has
|
||||
the identical shape and is fixed the same way.
|
||||
|
||||
### 3. DuckDB's `allowed_directories` **alone** is a no-op
|
||||
|
||||
Measured on 1.5.5. With `allowed_directories` set to one mount directory and
|
||||
nothing else, `read_csv('/etc/passwd')` **returned the file** and an `https://`
|
||||
URL was fetched over the network. Only `enable_external_access = false`
|
||||
alongside it makes the allow-list bind.
|
||||
|
||||
The obvious spelling of this control appears in `duckdb_settings()`, looks like
|
||||
a sandbox, and permits everything. The measurement is pinned by a test that
|
||||
never skips, so simplifying the bootstrap breaks the build rather than the
|
||||
guarantee.
|
||||
|
||||
### 4. `pg_constraint.conname` is not schema-scoped
|
||||
|
||||
`SELECT 1 FROM pg_constraint WHERE conname = '...'` matches a same-named
|
||||
constraint in **any** schema, so `sql/009` applied into a second schema
|
||||
silently skipped every guarded `CHECK`. Every guard now scopes via
|
||||
`conrelid = '<table>'::regclass`.
|
||||
|
||||
### 5. `describe.skipIf(flagSetInBeforeAll)` skips **always**
|
||||
|
||||
Vitest evaluates `skipIf` at collection time. Ten tests reported green while
|
||||
skipping against a live, reachable ClickHouse. Replaced with the repo's
|
||||
env-var gate, which **fails** when the variable is set and the server is down.
|
||||
|
||||
---
|
||||
|
||||
## What is NOT proven
|
||||
|
||||
- **BI-P029 / MetricFlow was never executed.** No Python runtime is
|
||||
provisioned for this repo's CI, and BI-P029's "pin a currently verified
|
||||
version" needs a licence and supply-chain review that has not happened.
|
||||
`MetricFlowProvider.providerReadiness()` returns `verified: false` and names
|
||||
all four open items. What _is_ proven is that the engine-neutral spec
|
||||
compiles to a MetricFlow-shaped query and that canonical SIAX metric ids
|
||||
survive a provider swap — tested against a second, deliberately
|
||||
different-named provider so the equality is not trivially true.
|
||||
- **The Superset integration path and the AI Analyst comparison** (BI-P029's
|
||||
other two DoD surfaces) are not implemented.
|
||||
- **dbt is not run here.** `DBT_RUNTIME_PIN` has both review flags `false`, and
|
||||
`assertDbtRuntimeCertified` refuses a production run until they are true.
|
||||
- **DuckDB Arrow IPC** is unproven: the `arrow` extension is a network install
|
||||
the offline environment could not download. `readiness()` says so.
|
||||
- **BI-P021's budget is a bound, not a hard cap.** In-flight enforcement is
|
||||
granular to the poll interval, so the query overshoots between polls — and on
|
||||
a columnar engine the overshoot is large. In the committed run
|
||||
(`evidence/bi-p019-p021/`) a 1,000,000-row window cancelled the query at
|
||||
**153,318,696 rows**, 55 ms in: a **153×** overshoot. The figure is
|
||||
machine-dependent and moves between runs (an earlier run on the same box
|
||||
recorded 131 M / 130×), which is itself the point — the bound is "roughly one
|
||||
poll interval of engine throughput", not a number. It is still a bound: the
|
||||
same query, unwatched, read **4,227,318,261 rows** before cancellation in the
|
||||
BI-P019 case. Anyone quoting it as a hard cap is wrong.
|
||||
- **The per-process window store does not bind across replicas.**
|
||||
`describeEnforcement()` reports `"with N replicas a tenant can spend up to N
|
||||
times this budget"`. A shared store (Valkey, or the CL0UD quota service)
|
||||
behind `SharedWindowStore` is required for it to bind.
|
||||
|
||||
---
|
||||
|
||||
## Running the live suites
|
||||
|
||||
```bash
|
||||
docker run -d --name ch -p 28123:8123 -e CLICKHOUSE_SKIP_USER_SETUP=1 \
|
||||
clickhouse/clickhouse-server:24.8
|
||||
docker run -d --name pg -p 25433:5432 -e POSTGRES_PASSWORD=test \
|
||||
-e POSTGRES_DB=opt postgres:16-alpine
|
||||
|
||||
CLICKHOUSE_URL=http://127.0.0.1:28123 pnpm vitest run tests/integration
|
||||
ANALYTICS_TEST_DATABASE_URL=postgres://postgres:test@127.0.0.1:25433/opt \
|
||||
pnpm vitest run tests/analytics/analytics-governance-postgres.realinfra.test.ts
|
||||
|
||||
python -m venv .venv && .venv/bin/pip install duckdb pyarrow 'pyiceberg[sql-sqlite]'
|
||||
DUCKDB_PYTHON=.venv/bin/python pnpm vitest run tests/duckdb
|
||||
```
|
||||
|
||||
Every one of these **fails** rather than skipping when its variable is set and
|
||||
the backend is unreachable.
|
||||
|
||||
---
|
||||
|
||||
## One note for whoever maintains `.gitea/workflows/ci.yml`
|
||||
|
||||
Not changed here — it is a protected file, and this branch touched none.
|
||||
|
||||
The integration job's guard is:
|
||||
|
||||
```sh
|
||||
if grep -qE "skipped" /tmp/itest.log && ! grep -qE "Tests [0-9]+ passed" /tmp/itest.log
|
||||
```
|
||||
|
||||
Vitest writes that summary with ANSI colour codes, so the literal text is
|
||||
`Tests [22m [1m[32m81 passed[39m…`. The second pattern therefore **never
|
||||
matches**, and the guard is effectively "any skip fails the job" rather than
|
||||
the "skipped _and_ nothing passed" it reads as. That is stricter than intended,
|
||||
not looser, so nothing is currently unguarded — but the escape hatch does not
|
||||
work, and someone will eventually be puzzled by it. Piping through
|
||||
`sed 's/\x1b\[[0-9;]*m//g'` before the grep would restore the intended
|
||||
behaviour.
|
||||
|
||||
This is how the DuckDB suite's placement was caught, so the guard earned its
|
||||
keep either way.
|
||||
Reference in New Issue
Block a user