Files
0pt/docs/analytics/BI_QUERY_METRIC_QUALITY.md
T
Claude Opus 5 c4862fb7a7
CI (SIAX Cloud) / quality (push) Successful in 48s
CI (SIAX Cloud) / integration (push) Successful in 37s
CI (SIAX Cloud) / security (push) Successful in 17s
Security Scan (SIAX Cloud) / security (push) Successful in 17s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m16s
CI (SIAX Cloud) / integration (pull_request) Successful in 55s
CI (SIAX Cloud) / security (pull_request) Successful in 35s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 29s
docs(analytics): quote the numbers the COMMITTED evidence actually contains
The live suites regenerate their evidence on every run, so the BI-P021
overshoot figures moved when they were last re-run — and the doc and module
comment still quoted the earlier ones (130,883,409 rows / 130x / 92ms) while
the committed file said 153,318,696 / 153x / 55ms. Prose that disagrees with
the evidence it cites is worse than prose with no number in it.

Both now match the committed file, and both say what the earlier version
implied but did not state: the multiple is MACHINE-DEPENDENT and moves between
runs on the same box. That is the actual finding. The bound is "roughly one
poll interval of engine throughput", not a number, and quoting any specific
multiple as if it were a property of the system is the mistake this correction
exists to prevent.

The unwatched-baseline figure is corrected the same way: 4,227,318,261 rows
read before cancellation in the current BI-P019 record, not the 6.5 billion of
the earlier run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 03:18:12 +02:00

161 lines
9.4 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# BI-P014, P016–P022, P025–P026, P028–P031 — query, metric, quality
Extends the analytics core merged in `0pt#34` (BI-P002..P012). Nothing there is
rebuilt.
This document is organised around what is **proven** and what is **not**,
because the second list is the one that gets dropped.
---
## Where things live
| prompt | module | live proof |
| -------------------------------------- | ----------------------------------------------------------------------------------------- | ----------------------------------------------------------------- |
| BI-P014 DuckDB provider | `src/analytics/providers/duckdb-provider.ts` | `tests/duckdb/duckdb-provider.live.test.ts` |
| BI-P016 query request contract | `src/analytics/contract/query-request.ts`, `compile-clickhouse.ts`, `compile-semantic.ts` | unit only (pure compilation) |
| BI-P017 analyst SQL contract | `src/analytics/sql/` | `tests/integration/analyst-sql-clickhouse.integration.test.ts` |
| BI-P018 execution model | `src/analytics/execution/query-execution-record.ts` | `sql/009` + Postgres realinfra |
| BI-P019 cancellation | `src/analytics/execution/cancellation.ts` | `tests/integration/query-cancellation-budget.integration.test.ts` |
| BI-P020 query cache | `src/analytics/cache/query-cache.ts` | unit only (no engine involved) |
| BI-P021 cost / budget | `src/analytics/cost/query-budget.ts` | same live suite as BI-P019 |
| BI-P022 dbt contract | `src/analytics/transform/dbt-contract.ts`, `analytics/dbt/` | contract only — execution is `n0d#38` |
| BI-P025 quality provider | `src/analytics/quality/` | `tests/integration/data-quality-clickhouse.integration.test.ts` |
| BI-P026 freshness | `src/analytics/quality/freshness.ts` | unit + `sql/009` defaults |
| BI-P028/P031 metric model + governance | `src/analytics/metrics/metric-model.ts` | Postgres trigger, realinfra |
| BI-P029 semantic provider | `src/analytics/metrics/semantic-provider.ts` | **partially — see below** |
| BI-P030 semantic registry | `src/analytics/metrics/semantic-registry.ts` | unit + Postgres constraints |
---
## Five findings that changed the implementation
Each was found by writing a test that could fail, and each is pinned so it
cannot quietly stop being true.
### 1. ClickHouse `readonly = 1` on a user profile forbids **every** request setting
Measured on 24.8.14.39. A user whose profile carries `readonly = 1` answers
`Code: 164. Cannot modify 'max_rows_to_read' setting in readonly mode` to any
setting sent with the statement. So BI-P012's ceilings **cannot travel with an
analyst statement** — they are baked into the user's `SETTINGS PROFILE` at
provisioning. Proven both ways: an over-budget scan fails with code 158 from
the profile with no setting sent, and an attempt to raise it from the request
fails with 164.
### 2. `CREATE ROW POLICY IF NOT EXISTS` is a silent no-op on re-provisioning
Dropping a user empties the policy's apply-to list. The next provisioning run
sees the policy exists and does nothing, so it survives with
`apply_to_list = []` — applying to **nobody** — while `system.row_policies`
still shows the correct `select_filter`. The restricted user read 100 of 100
rows, **both tenants**, from the base table.
Fixed with `OR REPLACE`, plus `rowPolicyVerificationSql()` which turns "the
policy is applied" from an assumption into a query, and a test asserting
`applied_to_user = 1` per object. `CREATE SETTINGS PROFILE IF NOT EXISTS` has
the identical shape and is fixed the same way.
### 3. DuckDB's `allowed_directories` **alone** is a no-op
Measured on 1.5.5. With `allowed_directories` set to one mount directory and
nothing else, `read_csv('/etc/passwd')` **returned the file** and an `https://`
URL was fetched over the network. Only `enable_external_access = false`
alongside it makes the allow-list bind.
The obvious spelling of this control appears in `duckdb_settings()`, looks like
a sandbox, and permits everything. The measurement is pinned by a test that
never skips, so simplifying the bootstrap breaks the build rather than the
guarantee.
### 4. `pg_constraint.conname` is not schema-scoped
`SELECT 1 FROM pg_constraint WHERE conname = '...'` matches a same-named
constraint in **any** schema, so `sql/009` applied into a second schema
silently skipped every guarded `CHECK`. Every guard now scopes via
`conrelid = '<table>'::regclass`.
### 5. `describe.skipIf(flagSetInBeforeAll)` skips **always**
Vitest evaluates `skipIf` at collection time. Ten tests reported green while
skipping against a live, reachable ClickHouse. Replaced with the repo's
env-var gate, which **fails** when the variable is set and the server is down.
---
## What is NOT proven
- **BI-P029 / MetricFlow was never executed.** No Python runtime is
provisioned for this repo's CI, and BI-P029's "pin a currently verified
version" needs a licence and supply-chain review that has not happened.
`MetricFlowProvider.providerReadiness()` returns `verified: false` and names
all four open items. What _is_ proven is that the engine-neutral spec
compiles to a MetricFlow-shaped query and that canonical SIAX metric ids
survive a provider swap — tested against a second, deliberately
different-named provider so the equality is not trivially true.
- **The Superset integration path and the AI Analyst comparison** (BI-P029's
other two DoD surfaces) are not implemented.
- **dbt is not run here.** `DBT_RUNTIME_PIN` has both review flags `false`, and
`assertDbtRuntimeCertified` refuses a production run until they are true.
- **DuckDB Arrow IPC** is unproven: the `arrow` extension is a network install
the offline environment could not download. `readiness()` says so.
- **BI-P021's budget is a bound, not a hard cap.** In-flight enforcement is
granular to the poll interval, so the query overshoots between polls — and on
a columnar engine the overshoot is large. In the committed run
(`evidence/bi-p019-p021/`) a 1,000,000-row window cancelled the query at
**153,318,696 rows**, 55 ms in: a **153×** overshoot. The figure is
machine-dependent and moves between runs (an earlier run on the same box
recorded 131 M / 130×), which is itself the point — the bound is "roughly one
poll interval of engine throughput", not a number. It is still a bound: the
same query, unwatched, read **4,227,318,261 rows** before cancellation in the
BI-P019 case. Anyone quoting it as a hard cap is wrong.
- **The per-process window store does not bind across replicas.**
`describeEnforcement()` reports `"with N replicas a tenant can spend up to N
times this budget"`. A shared store (Valkey, or the CL0UD quota service)
behind `SharedWindowStore` is required for it to bind.
---
## Running the live suites
```bash
docker run -d --name ch -p 28123:8123 -e CLICKHOUSE_SKIP_USER_SETUP=1 \
clickhouse/clickhouse-server:24.8
docker run -d --name pg -p 25433:5432 -e POSTGRES_PASSWORD=test \
-e POSTGRES_DB=opt postgres:16-alpine
CLICKHOUSE_URL=http://127.0.0.1:28123 pnpm vitest run tests/integration
ANALYTICS_TEST_DATABASE_URL=postgres://postgres:test@127.0.0.1:25433/opt \
pnpm vitest run tests/analytics/analytics-governance-postgres.realinfra.test.ts
python -m venv .venv && .venv/bin/pip install duckdb pyarrow 'pyiceberg[sql-sqlite]'
DUCKDB_PYTHON=.venv/bin/python pnpm vitest run tests/duckdb
```
Every one of these **fails** rather than skipping when its variable is set and
the backend is unreachable.
---
## One note for whoever maintains `.gitea/workflows/ci.yml`
Not changed here — it is a protected file, and this branch touched none.
The integration job's guard is:
```sh
if grep -qE "skipped" /tmp/itest.log && ! grep -qE "Tests [0-9]+ passed" /tmp/itest.log
```
Vitest writes that summary with ANSI colour codes, so the literal text is
`Tests [22m [1m[32m81 passed[39m…`. The second pattern therefore **never
matches**, and the guard is effectively "any skip fails the job" rather than
the "skipped _and_ nothing passed" it reads as. That is stricter than intended,
not looser, so nothing is currently unguarded — but the escape hatch does not
work, and someone will eventually be puzzled by it. Piping through
`sed 's/\x1b\[[0-9;]*m//g'` before the grep would restore the intended
behaviour.
This is how the DuckDB suite's placement was caught, so the guard earned its
keep either way.