From 02b993faf9912098a27baf571e887228ded5d8da Mon Sep 17 00:00:00 2001 From: Claude Opus 5 Date: Mon, 7 Sep 2026 04:08:22 +0200 Subject: [PATCH] =?UTF-8?q?docs(analytics):=20BI-P060/P061/P103/P104/P105?= =?UTF-8?q?=20=E2=80=94=20what=20is=20proven,=20and=20what=20is=20not?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Includes BI-P105's freshness semantics as its DoD asks: watermark, staleness, the eight refusal reasons, grain rollup limits, and the sentence the evidence exists to support — the drifted aggregate was ONE SECOND old and overstated revenue by 5.251%. Co-Authored-By: Claude Opus 5 --- .../BI_DASHBOARDS_SLO_MATERIALIZATION.md | 257 ++++++++++++++++++ 1 file changed, 257 insertions(+) create mode 100644 docs/analytics/BI_DASHBOARDS_SLO_MATERIALIZATION.md diff --git a/docs/analytics/BI_DASHBOARDS_SLO_MATERIALIZATION.md b/docs/analytics/BI_DASHBOARDS_SLO_MATERIALIZATION.md new file mode 100644 index 0000000..0cd7e45 --- /dev/null +++ b/docs/analytics/BI_DASHBOARDS_SLO_MATERIALIZATION.md @@ -0,0 +1,257 @@ +# BI-P060, P061, P103, P104, P105 — dashboards, SLOs, benchmarks, pre-aggregation + +Extends the analytics core (`0pt#34`, BI-P002..P012) and the query/metric/quality +layer (`0pt#35`). Nothing there is rebuilt: `DashboardDefinition` and +`DashboardVersion` were already canonical entities with tables in `sql/008`, and +these prompts give them behaviour. + +Organised, like its predecessor, around what is **proven** and what is **not**. + +--- + +## Where things live + +| prompt | module | live proof | +| ---------------------------- | ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | +| BI-P060 dashboard model | `src/analytics/dashboards/dashboard-document.ts` | unit (pure model + planner) | +| BI-P061 dashboard versioning | `src/analytics/dashboards/dashboard-release.ts`, `sql/010` | `tests/analytics/dashboard-release-postgres.realinfra.test.ts`, Postgres 16.15 | +| BI-P103 BI SLO | `src/analytics/slo/bi-slo.ts`, `slo/usage-events.ts` | unit; measurement path is `aud0#34`'s seven event types | +| BI-P104 benchmark harness | `src/analytics/bench/benchmark-harness.ts` | `tests/integration/bi-p104-benchmark.integration.test.ts`, ClickHouse 26.3.10.60 + DuckDB 1.5.5 | +| BI-P105 materialization | `src/analytics/materialization/` | `tests/integration/materialization-drift-clickhouse.integration.test.ts` | + +Evidence: `evidence/bi-p061/dashboard-release-race.json`, +`evidence/bi-p104/benchmark.json`, `evidence/bi-p105/materialization-drift.json`. + +--- + +## BI-P060 — the dashboard is a document, not a Superset export + +`layout_json` was a string, and a string holds anything. The canonical document +carries `layout / filters / widgets / parameters / refresh / permissions / +version`, and a widget references **a metric, a contract query and a +visualization spec** — never a vendor's internal id. + +The DoD is a function. `planDashboard(document, { parameters })` returns, per +widget, a runnable `AnalyticsQueryRequest` (BI-P016) plus a render spec. If it +returns a plan, the dashboard can be drawn by anything that executes the +contract and draws the twelve canonical visualization types. + +Rules that are enforced rather than documented: + +- a widget whose only content is a vendor id is **refused** — that is exactly + the document that cannot be reconstructed; +- `vendor_ref` may carry a Superset/Metabase id for round-tripping, but it is + excluded from the canonical hash, so re-exporting into a new Superset instance + does not fabricate a new dashboard version; +- vendor-native keys (`viz_type`, `form_data`, `position_json`, `slice_id`, …) + are refused with a message that names the rule; +- query widgets go through `parseAnalyticsQueryRequest`, so a dashboard cannot + become an SQL surface and cannot name a tenant; +- a locked filter cannot be switched off by a viewer — locked filters are how a + saved dashboard pins its scope, and removing one silently widens the result. + +--- + +## BI-P061 — releases are a ledger + +`DRAFT → PUBLISHED → SUPERSEDED`, stored in `opt_analytics_dashboard_release` +(`sql/010`) rather than as a column on the version row: `sql/008` already makes +`DashboardVersion` immutable at the database, and a lifecycle column there would +have meant weakening that. The **document** is frozen; the **release** moves. + +The database enforces the state machine, not the application: + +| attempted | result | +| ------------------------------------------- | -------------------------------------------------- | +| `published → draft` (un-publishing) | refused — it is how a rollback gets faked | +| `superseded → published` | refused | +| repointing a release at another version | refused (every content column is frozen on UPDATE) | +| rewriting `document_hash` or `published_at` | refused | +| `DELETE` on a release | refused — the ledger has no eraser | +| a `published` row with no `published_at` | refused by CHECK | +| a second open draft for one dashboard | refused by a partial unique index | + +**Rollback mints a new version.** It does not resurrect the old release row. +"We went back to v3" becomes a timestamped fact (v5 carries +`rollback_of_version = 3`) instead of a hole in the history where v4 used to be. +The restore is verified by BI-P060 canonical hash, and `documentOf()` refuses to +serve a stored document whose hash no longer matches its release. + +### The concurrency claim, measured both ways + +`DashboardReleaseService.publish()` checks for a live release before writing. +That check is a read-then-write race and is **not** the guarantee. +`evidence/bi-p061/dashboard-release-race.json`, Postgres 16.15: + +| run | partial unique index | transactions committed | published releases after | +| --------------- | -------------------- | ---------------------- | --------------------------- | +| `guard_absent` | dropped | 2 | **2 — two live dashboards** | +| `guard_present` | in place | 1 (loser `23505`) | 1 | + +Reproduce with `node scripts/bi-p061-release-race.mjs`. + +--- + +## BI-P103 — SLOs measured from AUD0, or declared unmeasured + +The measurement path is `aud0#34`'s seven BI usage event types (BI-P076). aud0's +own doc records that zero producers emit them; `src/analytics/slo/usage-events.ts` +makes 0pt the producer. There is deliberately no second telemetry path — two +measurement systems disagree, and the disagreement surfaces during an incident. + +| workload | objectives | +| --------------------- | ---------------------------------------------------------------- | +| interactive dashboard | availability, p95 first query, p95 cached, error rate, freshness | +| exploration | p95, p99, max result, timeout | +| report | completion SLA, delivery SLA | + +Three verdicts that are not "met": + +- **`insufficient_data`** — an availability objective over an empty window is + silence, not 100%. A started query with no terminal event is _unresolved_, not + a success. +- **`not_measurable_here`** — dashboard **freshness** is a property of the + dataset (BI-P026 / BI-P105 watermark) and report **delivery** is N0TIFY's + receipt. Neither exists in the seven event types, so both are declared + unmeasured rather than proxied by completion. +- **`low_confidence`** — a p99 over 12 samples is the 12th-slowest request + wearing a percentile's clothes. Reported with its sample size, not judged. + +The producer is an allowlist built field by named field. `compiled_sql`, +`error_message`, `tenant_id`, filter **values** and result samples exist on the +0pt-side telemetry type and are never mapped; a `query_hash` that is not a +64-hex digest is dropped rather than reshaped. + +### Honest status + +`BI_SLO_CATALOGUE`'s targets are **starting thresholds, not derived baselines**. +No production BI traffic exists yet, so the first real baselines will come from +the first weeks of emitted events. The evaluator is built so that absence shows +up as `insufficient_data` until then, rather than as a green dashboard. + +--- + +## BI-P104 — the harness refuses to quote one number + +Six dataset profiles × two engines, 2 warmups discarded, 9 timed repetitions, +statistics from the shipped module. `evidence/bi-p104/benchmark.json`. + +- **Overlapping ranges are `indistinguishable`**, whatever the medians say. + Three of the six real comparisons came out that way, one with a 1.6x median + gap — the headline a noisy host manufactures for you. +- **Every sentence carries the range, the sample size and the stability**, so it + cannot be trimmed to "4.4x faster" without visibly deleting the caveats. +- **Under 5 samples nothing is comparable.** A three-run benchmark is an anecdote. +- **Every engine BI-P104 names appears**, and the five nothing measured carry a + reason each. + +Measured (warm cache, one idle host, these query shapes only): + +| dataset | verdict, DuckDB 1.5.5 vs ClickHouse 26.3.10.60 | +| --------------- | ------------------------------------------------- | +| small_saas | faster, 1.6x–8.2x (median 4.4x) | +| medium_business | indistinguishable (medians 1.6x apart) | +| vehiq_market | faster, 2.4x–13.3x (median 5.8x) | +| large_event | **slower**, 0.2x–0.9x (`count DISTINCT`, 5M rows) | +| wide_table | indistinguishable | +| geospatial | indistinguishable | + +Not exercised, with reasons in the evidence: **trino** (the running instance +belongs to another lane; benchmarking through it would measure their +configuration), **semantic_query** (MetricFlow has never executed in 0pt), +**superset**, **native_charts** (DES1GN owns rendering), **perspective**. + +--- + +## BI-P105 — freshness semantics, and why freshness is not correctness + +### The vocabulary + +| term | definition | +| ------------------------------ | ------------------------------------------------------------------------------- | +| `data_watermark` | the newest **source** timestamp the aggregate represents. Absent = **UNKNOWN**. | +| staleness | `now − data_watermark`, in seconds. `undefined` when the watermark is unknown. | +| `max_staleness_seconds` | the freshness contract. Older than this and the aggregate stops serving. | +| `drift_check_interval_seconds` | how often the aggregate must be re-proved against its source to keep serving. | +| `last_drift_verdict` | `match` / `drifted` / `insufficient_data` from the last comparison. | + +**UNKNOWN is never treated as current.** This is the same doctrine BI-P026 +applies to result freshness: a missing watermark is a refusal, not a zero. + +### The routing rule + +A materialization serves **only** while it is provably current _and_ provably +undrifted. Eight named refusals, each a real way an aggregate becomes a +confident liar: + +`coverage_miss` (metric/dimension/grain not carried — including a filter on a +dimension the aggregate already summed across) · `watermark_unknown` · +`staleness_exceeded` · `window_beyond_watermark` (the question reaches past the +newest data; answering would silently shorten the period) · +`drift_never_verified` · `drift_verification_stale` · `drift_detected` · +`state_not_serving`. + +`drift_never_verified` is the one people leave out and the one that matters on +launch day: a brand-new aggregate is exactly as unproven as a drifted one. + +Grain rollup is restricted, not assumed: a `day` aggregate answers `day` and +coarser, never `hour`; a `week` aggregate answers only `week`, because weekly +buckets do not compose into calendar months. + +### Drift detection + +- **A missing bucket is drift** even when every value present is correct — late + arrivals and a materialized view created after the backfill both look like + this, and a value-only comparison passes while the total is wrong. +- **An extra bucket is drift** — the source rows are gone and the aggregate is not. +- **An empty comparison is `insufficient_data`, never `match`** (the BI-P025 + false green, restated for pre-aggregates), and it takes the aggregate out of + service rather than annotating it. +- **Both tolerances must hold.** 0.1% of a 40M bucket is 40 000 kr of invisible + error; an absolute-only bound flags every rounding difference on a large bucket. + +### The proof, on a live ClickHouse + +`evidence/bi-p105/materialization-drift.json`, ClickHouse 26.3.10.60, 2M rows, +a real materialized view: + +| step | result | +| ------------------------ | ----------------------------------------------------------------------------------- | +| after backfill | `match`, 48 buckets, 801 571 002 on both sides | +| GDPR erasure | `ALTER TABLE sales DELETE …`. An MV is an INSERT trigger and never sees a mutation. | +| what would be served | 801 571 002 against a true 761 580 554 — **39 990 448 overstated, 5.251%** | +| staleness at that moment | **1 second.** Every freshness check passed. | +| guarded router | `base_table` (`state_not_serving`) | +| counterfactual | a staleness-only router **serves it** — measured, not assumed | +| after rebuild | `match`, and routing returns to the materialization | + +That last row of the counterfactual is the whole point of the prompt: staleness +is not correctness. The aggregate was one second old and wrong by 5%. + +### Measured improvement, honestly + +Same drill, base table vs pre-aggregate, 9 repetitions interleaved: p50 3.4x, +but the run-to-run range spans **0.6x–7.5x** on an otherwise idle host — one +sample had the pre-aggregate slower than the base table's best run. Quoting +"3.4x" as a property of the system would be misleading, which is why BI-P104's +harness exists. + +--- + +## What is NOT proven here + +- **No dashboard has been rendered.** BI-P060 produces a plan; drawing it is + DES1GN (`DES1GN#9`) and `siax-portal#5`. Nothing in this PR proves a pixel. +- **The release service has no Postgres adapter yet.** The lifecycle is proven + in-memory and the _schema_ is proven against live Postgres; wiring + `DashboardReleaseStore` to `postgres.js` is the next step and is the only + place where the two could drift apart. +- **Zero real SLO events have been emitted in production.** The producer and the + evaluator are tested; no BI surface calls the producer yet, so every objective + would today report `insufficient_data`. That is the honest empty state, and it + is deliberately visible rather than hidden behind a default. +- **No materialization is deployed.** The router and detector are proven against + a live ClickHouse in a throwaway database. Nothing schedules a refresh or a + periodic drift check yet — that belongs with ACT0/N0D scheduling. +- **Trino, Superset, MetricFlow and Perspective are unmeasured**, for the + reasons recorded in the benchmark evidence.