# BI-P014, P016–P022, P025–P026, P028–P031 — query, metric, quality Extends the analytics core merged in `0pt#34` (BI-P002..P012). Nothing there is rebuilt. This document is organised around what is **proven** and what is **not**, because the second list is the one that gets dropped. --- ## Where things live | prompt | module | live proof | | -------------------------------------- | ----------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | | BI-P014 DuckDB provider | `src/analytics/providers/duckdb-provider.ts` | `tests/duckdb/duckdb-provider.live.test.ts` | | BI-P016 query request contract | `src/analytics/contract/query-request.ts`, `compile-clickhouse.ts`, `compile-semantic.ts` | unit only (pure compilation) | | BI-P017 analyst SQL contract | `src/analytics/sql/` | `tests/integration/analyst-sql-clickhouse.integration.test.ts` | | BI-P018 execution model | `src/analytics/execution/query-execution-record.ts` | `sql/009` + Postgres realinfra | | BI-P019 cancellation | `src/analytics/execution/cancellation.ts` | `tests/integration/query-cancellation-budget.integration.test.ts` | | BI-P020 query cache | `src/analytics/cache/query-cache.ts` | unit only (no engine involved) | | BI-P021 cost / budget | `src/analytics/cost/query-budget.ts` | same live suite as BI-P019 | | BI-P022 dbt contract | `src/analytics/transform/dbt-contract.ts`, `analytics/dbt/` | contract only — execution is `n0d#38` | | BI-P025 quality provider | `src/analytics/quality/` | `tests/integration/data-quality-clickhouse.integration.test.ts` | | BI-P026 freshness | `src/analytics/quality/freshness.ts` | unit + `sql/009` defaults | | BI-P028/P031 metric model + governance | `src/analytics/metrics/metric-model.ts` | Postgres trigger, realinfra | | BI-P029 semantic provider | `src/analytics/metrics/semantic-provider.ts` | **partially — see below** | | BI-P030 semantic registry | `src/analytics/metrics/semantic-registry.ts` | unit + Postgres constraints | --- ## Five findings that changed the implementation Each was found by writing a test that could fail, and each is pinned so it cannot quietly stop being true. ### 1. ClickHouse `readonly = 1` on a user profile forbids **every** request setting Measured on 24.8.14.39. A user whose profile carries `readonly = 1` answers `Code: 164. Cannot modify 'max_rows_to_read' setting in readonly mode` to any setting sent with the statement. So BI-P012's ceilings **cannot travel with an analyst statement** — they are baked into the user's `SETTINGS PROFILE` at provisioning. Proven both ways: an over-budget scan fails with code 158 from the profile with no setting sent, and an attempt to raise it from the request fails with 164. ### 2. `CREATE ROW POLICY IF NOT EXISTS` is a silent no-op on re-provisioning Dropping a user empties the policy's apply-to list. The next provisioning run sees the policy exists and does nothing, so it survives with `apply_to_list = []` — applying to **nobody** — while `system.row_policies` still shows the correct `select_filter`. The restricted user read 100 of 100 rows, **both tenants**, from the base table. Fixed with `OR REPLACE`, plus `rowPolicyVerificationSql()` which turns "the policy is applied" from an assumption into a query, and a test asserting `applied_to_user = 1` per object. `CREATE SETTINGS PROFILE IF NOT EXISTS` has the identical shape and is fixed the same way. ### 3. DuckDB's `allowed_directories` **alone** is a no-op Measured on 1.5.5. With `allowed_directories` set to one mount directory and nothing else, `read_csv('/etc/passwd')` **returned the file** and an `https://` URL was fetched over the network. Only `enable_external_access = false` alongside it makes the allow-list bind. The obvious spelling of this control appears in `duckdb_settings()`, looks like a sandbox, and permits everything. The measurement is pinned by a test that never skips, so simplifying the bootstrap breaks the build rather than the guarantee. ### 4. `pg_constraint.conname` is not schema-scoped `SELECT 1 FROM pg_constraint WHERE conname = '...'` matches a same-named constraint in **any** schema, so `sql/009` applied into a second schema silently skipped every guarded `CHECK`. Every guard now scopes via `conrelid = ''::regclass`. ### 5. `describe.skipIf(flagSetInBeforeAll)` skips **always** Vitest evaluates `skipIf` at collection time. Ten tests reported green while skipping against a live, reachable ClickHouse. Replaced with the repo's env-var gate, which **fails** when the variable is set and the server is down. --- ## What is NOT proven - **BI-P029 / MetricFlow was never executed.** No Python runtime is provisioned for this repo's CI, and BI-P029's "pin a currently verified version" needs a licence and supply-chain review that has not happened. `MetricFlowProvider.providerReadiness()` returns `verified: false` and names all four open items. What _is_ proven is that the engine-neutral spec compiles to a MetricFlow-shaped query and that canonical SIAX metric ids survive a provider swap — tested against a second, deliberately different-named provider so the equality is not trivially true. - **The Superset integration path and the AI Analyst comparison** (BI-P029's other two DoD surfaces) are not implemented. - **dbt is not run here.** `DBT_RUNTIME_PIN` has both review flags `false`, and `assertDbtRuntimeCertified` refuses a production run until they are true. - **DuckDB Arrow IPC** is unproven: the `arrow` extension is a network install the offline environment could not download. `readiness()` says so. - **BI-P021's budget is a bound, not a hard cap.** In-flight enforcement is granular to the poll interval, so the query overshoots between polls — and on a columnar engine the overshoot is large. In the committed run (`evidence/bi-p019-p021/`) a 1,000,000-row window cancelled the query at **153,318,696 rows**, 55 ms in: a **153×** overshoot. The figure is machine-dependent and moves between runs (an earlier run on the same box recorded 131 M / 130×), which is itself the point — the bound is "roughly one poll interval of engine throughput", not a number. It is still a bound: the same query, unwatched, read **4,227,318,261 rows** before cancellation in the BI-P019 case. Anyone quoting it as a hard cap is wrong. - **The per-process window store does not bind across replicas.** `describeEnforcement()` reports `"with N replicas a tenant can spend up to N times this budget"`. A shared store (Valkey, or the CL0UD quota service) behind `SharedWindowStore` is required for it to bind. --- ## Running the live suites ```bash docker run -d --name ch -p 28123:8123 -e CLICKHOUSE_SKIP_USER_SETUP=1 \ clickhouse/clickhouse-server:24.8 docker run -d --name pg -p 25433:5432 -e POSTGRES_PASSWORD=test \ -e POSTGRES_DB=opt postgres:16-alpine CLICKHOUSE_URL=http://127.0.0.1:28123 pnpm vitest run tests/integration ANALYTICS_TEST_DATABASE_URL=postgres://postgres:test@127.0.0.1:25433/opt \ pnpm vitest run tests/analytics/analytics-governance-postgres.realinfra.test.ts python -m venv .venv && .venv/bin/pip install duckdb pyarrow 'pyiceberg[sql-sqlite]' DUCKDB_PYTHON=.venv/bin/python pnpm vitest run tests/duckdb ``` Every one of these **fails** rather than skipping when its variable is set and the backend is unreachable. --- ## One note for whoever maintains `.gitea/workflows/ci.yml` Not changed here — it is a protected file, and this branch touched none. The integration job's guard is: ```sh if grep -qE "skipped" /tmp/itest.log && ! grep -qE "Tests [0-9]+ passed" /tmp/itest.log ``` Vitest writes that summary with ANSI colour codes, so the literal text is `Tests [22m [1m[32m81 passed[39m…`. The second pattern therefore **never matches**, and the guard is effectively "any skip fails the job" rather than the "skipped _and_ nothing passed" it reads as. That is stricter than intended, not looser, so nothing is currently unguarded — but the escape hatch does not work, and someone will eventually be puzzled by it. Piping through `sed 's/\x1b\[[0-9;]*m//g'` before the grep would restore the intended behaviour. This is how the DuckDB suite's placement was caught, so the guard earned its keep either way.