admin/0pt
5
0
Fork 0
Code Issues 10 Pull Requests Actions 3.8k Packages Projects Releases Wiki Activity

docs(analytics): BI-P060/P061/P103/P104/P105 — what is proven, and what is not
CI (SIAX Cloud) / security (push) Successful in 27s
Details
Security Scan (SIAX Cloud) / security (push) Successful in 28s
Details
CI (SIAX Cloud) / integration (push) Successful in 47s
Details
CI (SIAX Cloud) / security (pull_request) Successful in 28s
Details
CI (SIAX Cloud) / quality (push) Successful in 1m6s
Details
CI (SIAX Cloud) / integration (pull_request) Successful in 58s
Details
Security Scan (SIAX Cloud) / security (pull_request) Successful in 40s
Details
CI (SIAX Cloud) / quality (pull_request) Successful in 1m13s
Details

Browse Source 
Includes BI-P105's freshness semantics as its DoD asks: watermark, staleness,
the eight refusal reasons, grain rollup limits, and the sentence the evidence
exists to support — the drifted aggregate was ONE SECOND old and overstated
revenue by 5.251%.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
2026-09-07 04:08:22 +02:00
parent f33cc8d32f
commit 02b993faf9
1 changed files with 257 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/analytics/BI_DASHBOARDS_SLO_MATERIALIZATION.md
+257
View File
@@ -0,0 +1,257 @@
# BI-P060, P061, P103, P104, P105 — dashboards, SLOs, benchmarks, pre-aggregation
Extends the analytics core (`0pt#34`, BI-P002..P012) and the query/metric/quality
layer (`0pt#35`). Nothing there is rebuilt: `DashboardDefinition` and
`DashboardVersion` were already canonical entities with tables in `sql/008`, and
these prompts give them behaviour.
Organised, like its predecessor, around what is **proven** and what is **not**.
---
## Where things live
| prompt | module | live proof |
| ---------------------------- | ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| BI-P060 dashboard model | `src/analytics/dashboards/dashboard-document.ts` | unit (pure model + planner) |
| BI-P061 dashboard versioning | `src/analytics/dashboards/dashboard-release.ts`, `sql/010` | `tests/analytics/dashboard-release-postgres.realinfra.test.ts`, Postgres 16.15 |
| BI-P103 BI SLO | `src/analytics/slo/bi-slo.ts`, `slo/usage-events.ts` | unit; measurement path is `aud0#34`'s seven event types |
| BI-P104 benchmark harness | `src/analytics/bench/benchmark-harness.ts` | `tests/integration/bi-p104-benchmark.integration.test.ts`, ClickHouse 26.3.10.60 + DuckDB 1.5.5 |
| BI-P105 materialization | `src/analytics/materialization/` | `tests/integration/materialization-drift-clickhouse.integration.test.ts` |
Evidence: `evidence/bi-p061/dashboard-release-race.json`,
`evidence/bi-p104/benchmark.json`, `evidence/bi-p105/materialization-drift.json`.
---
## BI-P060 — the dashboard is a document, not a Superset export
`layout_json` was a string, and a string holds anything. The canonical document
carries `layout / filters / widgets / parameters / refresh / permissions /
version`, and a widget references **a metric, a contract query and a
visualization spec** — never a vendor's internal id.
The DoD is a function. `planDashboard(document, { parameters })` returns, per
widget, a runnable `AnalyticsQueryRequest` (BI-P016) plus a render spec. If it
returns a plan, the dashboard can be drawn by anything that executes the
contract and draws the twelve canonical visualization types.
Rules that are enforced rather than documented:
- a widget whose only content is a vendor id is **refused** — that is exactly
the document that cannot be reconstructed;
- `vendor_ref` may carry a Superset/Metabase id for round-tripping, but it is
excluded from the canonical hash, so re-exporting into a new Superset instance
does not fabricate a new dashboard version;
- vendor-native keys (`viz_type`, `form_data`, `position_json`, `slice_id`, …)
are refused with a message that names the rule;
- query widgets go through `parseAnalyticsQueryRequest`, so a dashboard cannot
become an SQL surface and cannot name a tenant;
- a locked filter cannot be switched off by a viewer — locked filters are how a
saved dashboard pins its scope, and removing one silently widens the result.
---
## BI-P061 — releases are a ledger
`DRAFT → PUBLISHED → SUPERSEDED`, stored in `opt_analytics_dashboard_release`
(`sql/010`) rather than as a column on the version row: `sql/008` already makes
`DashboardVersion` immutable at the database, and a lifecycle column there would
have meant weakening that. The **document** is frozen; the **release** moves.
The database enforces the state machine, not the application:
| attempted | result |
| ------------------------------------------- | -------------------------------------------------- |
| `published → draft` (un-publishing) | refused — it is how a rollback gets faked |
| `superseded → published` | refused |
| repointing a release at another version | refused (every content column is frozen on UPDATE) |
| rewriting `document_hash` or `published_at` | refused |
| `DELETE` on a release | refused — the ledger has no eraser |
| a `published` row with no `published_at` | refused by CHECK |
| a second open draft for one dashboard | refused by a partial unique index |
**Rollback mints a new version.** It does not resurrect the old release row.
"We went back to v3" becomes a timestamped fact (v5 carries
`rollback_of_version = 3`) instead of a hole in the history where v4 used to be.
The restore is verified by BI-P060 canonical hash, and `documentOf()` refuses to
serve a stored document whose hash no longer matches its release.
### The concurrency claim, measured both ways
`DashboardReleaseService.publish()` checks for a live release before writing.
That check is a read-then-write race and is **not** the guarantee.
`evidence/bi-p061/dashboard-release-race.json`, Postgres 16.15:
| run | partial unique index | transactions committed | published releases after |
| --------------- | -------------------- | ---------------------- | --------------------------- |
| `guard_absent` | dropped | 2 | **2 — two live dashboards** |
| `guard_present` | in place | 1 (loser `23505`) | 1 |
Reproduce with `node scripts/bi-p061-release-race.mjs`.
---
## BI-P103 — SLOs measured from AUD0, or declared unmeasured
The measurement path is `aud0#34`'s seven BI usage event types (BI-P076). aud0's
own doc records that zero producers emit them; `src/analytics/slo/usage-events.ts`
makes 0pt the producer. There is deliberately no second telemetry path — two
measurement systems disagree, and the disagreement surfaces during an incident.
| workload | objectives |
| --------------------- | ---------------------------------------------------------------- |
| interactive dashboard | availability, p95 first query, p95 cached, error rate, freshness |
| exploration | p95, p99, max result, timeout |
| report | completion SLA, delivery SLA |
Three verdicts that are not "met":
- **`insufficient_data`** — an availability objective over an empty window is
silence, not 100%. A started query with no terminal event is _unresolved_, not
a success.
- **`not_measurable_here`** — dashboard **freshness** is a property of the
dataset (BI-P026 / BI-P105 watermark) and report **delivery** is N0TIFY's
receipt. Neither exists in the seven event types, so both are declared
unmeasured rather than proxied by completion.
- **`low_confidence`** — a p99 over 12 samples is the 12th-slowest request
wearing a percentile's clothes. Reported with its sample size, not judged.
The producer is an allowlist built field by named field. `compiled_sql`,
`error_message`, `tenant_id`, filter **values** and result samples exist on the
0pt-side telemetry type and are never mapped; a `query_hash` that is not a
64-hex digest is dropped rather than reshaped.
### Honest status
`BI_SLO_CATALOGUE`'s targets are **starting thresholds, not derived baselines**.
No production BI traffic exists yet, so the first real baselines will come from
the first weeks of emitted events. The evaluator is built so that absence shows
up as `insufficient_data` until then, rather than as a green dashboard.
---
## BI-P104 — the harness refuses to quote one number
Six dataset profiles × two engines, 2 warmups discarded, 9 timed repetitions,
statistics from the shipped module. `evidence/bi-p104/benchmark.json`.
- **Overlapping ranges are `indistinguishable`**, whatever the medians say.
Three of the six real comparisons came out that way, one with a 1.6x median
gap — the headline a noisy host manufactures for you.
- **Every sentence carries the range, the sample size and the stability**, so it
cannot be trimmed to "4.4x faster" without visibly deleting the caveats.
- **Under 5 samples nothing is comparable.** A three-run benchmark is an anecdote.
- **Every engine BI-P104 names appears**, and the five nothing measured carry a
reason each.
Measured (warm cache, one idle host, these query shapes only):
| dataset | verdict, DuckDB 1.5.5 vs ClickHouse 26.3.10.60 |
| --------------- | ------------------------------------------------- |
| small_saas | faster, 1.6x–8.2x (median 4.4x) |
| medium_business | indistinguishable (medians 1.6x apart) |
| vehiq_market | faster, 2.4x–13.3x (median 5.8x) |
| large_event | **slower**, 0.2x–0.9x (`count DISTINCT`, 5M rows) |
| wide_table | indistinguishable |
| geospatial | indistinguishable |
Not exercised, with reasons in the evidence: **trino** (the running instance
belongs to another lane; benchmarking through it would measure their
configuration), **semantic_query** (MetricFlow has never executed in 0pt),
**superset**, **native_charts** (DES1GN owns rendering), **perspective**.
---
## BI-P105 — freshness semantics, and why freshness is not correctness
### The vocabulary
| term | definition |
| ------------------------------ | ------------------------------------------------------------------------------- |
| `data_watermark` | the newest **source** timestamp the aggregate represents. Absent = **UNKNOWN**. |
| staleness | `now − data_watermark`, in seconds. `undefined` when the watermark is unknown. |
| `max_staleness_seconds` | the freshness contract. Older than this and the aggregate stops serving. |
| `drift_check_interval_seconds` | how often the aggregate must be re-proved against its source to keep serving. |
| `last_drift_verdict` | `match` / `drifted` / `insufficient_data` from the last comparison. |
**UNKNOWN is never treated as current.** This is the same doctrine BI-P026
applies to result freshness: a missing watermark is a refusal, not a zero.
### The routing rule
A materialization serves **only** while it is provably current _and_ provably
undrifted. Eight named refusals, each a real way an aggregate becomes a
confident liar:
`coverage_miss` (metric/dimension/grain not carried — including a filter on a
dimension the aggregate already summed across) · `watermark_unknown` ·
`staleness_exceeded` · `window_beyond_watermark` (the question reaches past the
newest data; answering would silently shorten the period) ·
`drift_never_verified` · `drift_verification_stale` · `drift_detected` ·
`state_not_serving`.
`drift_never_verified` is the one people leave out and the one that matters on
launch day: a brand-new aggregate is exactly as unproven as a drifted one.
Grain rollup is restricted, not assumed: a `day` aggregate answers `day` and
coarser, never `hour`; a `week` aggregate answers only `week`, because weekly
buckets do not compose into calendar months.
### Drift detection
- **A missing bucket is drift** even when every value present is correct — late
arrivals and a materialized view created after the backfill both look like
this, and a value-only comparison passes while the total is wrong.
- **An extra bucket is drift** — the source rows are gone and the aggregate is not.
- **An empty comparison is `insufficient_data`, never `match`** (the BI-P025
false green, restated for pre-aggregates), and it takes the aggregate out of
service rather than annotating it.
- **Both tolerances must hold.** 0.1% of a 40M bucket is 40 000 kr of invisible
error; an absolute-only bound flags every rounding difference on a large bucket.
### The proof, on a live ClickHouse
`evidence/bi-p105/materialization-drift.json`, ClickHouse 26.3.10.60, 2M rows,
a real materialized view:
| step | result |
| ------------------------ | ----------------------------------------------------------------------------------- |
| after backfill | `match`, 48 buckets, 801 571 002 on both sides |
| GDPR erasure | `ALTER TABLE sales DELETE …`. An MV is an INSERT trigger and never sees a mutation. |
| what would be served | 801 571 002 against a true 761 580 554 — **39 990 448 overstated, 5.251%** |
| staleness at that moment | **1 second.** Every freshness check passed. |
| guarded router | `base_table` (`state_not_serving`) |
| counterfactual | a staleness-only router **serves it** — measured, not assumed |
| after rebuild | `match`, and routing returns to the materialization |
That last row of the counterfactual is the whole point of the prompt: staleness
is not correctness. The aggregate was one second old and wrong by 5%.
### Measured improvement, honestly
Same drill, base table vs pre-aggregate, 9 repetitions interleaved: p50 3.4x,
but the run-to-run range spans **0.6x–7.5x** on an otherwise idle host — one
sample had the pre-aggregate slower than the base table's best run. Quoting
"3.4x" as a property of the system would be misleading, which is why BI-P104's
harness exists.
---
## What is NOT proven here
- **No dashboard has been rendered.** BI-P060 produces a plan; drawing it is
DES1GN (`DES1GN#9`) and `siax-portal#5`. Nothing in this PR proves a pixel.
- **The release service has no Postgres adapter yet.** The lifecycle is proven
in-memory and the _schema_ is proven against live Postgres; wiring
`DashboardReleaseStore` to `postgres.js` is the next step and is the only
place where the two could drift apart.
- **Zero real SLO events have been emitted in production.** The producer and the
evaluator are tested; no BI surface calls the producer yet, so every objective
would today report `insufficient_data`. That is the honest empty state, and it
is deliberately visible rather than hidden behind a default.
- **No materialization is deployed.** The router and detector are proven against
a live ClickHouse in a throwaway database. Nothing schedules a refresh or a
periodic drift check yet — that belongs with ACT0/N0D scheduling.
- **Trino, Superset, MetricFlow and Perspective are unmeasured**, for the
reasons recorded in the benchmark evidence.