The live suites regenerate their evidence on every run, so the BI-P021 overshoot figures moved when they were last re-run — and the doc and module comment still quoted the earlier ones (130,883,409 rows / 130x / 92ms) while the committed file said 153,318,696 / 153x / 55ms. Prose that disagrees with the evidence it cites is worse than prose with no number in it. Both now match the committed file, and both say what the earlier version implied but did not state: the multiple is MACHINE-DEPENDENT and moves between runs on the same box. That is the actual finding. The bound is "roughly one poll interval of engine throughput", not a number, and quoting any specific multiple as if it were a property of the system is the mistake this correction exists to prevent. The unwatched-baseline figure is corrected the same way: 4,227,318,261 rows read before cancellation in the current BI-P019 record, not the 6.5 billion of the earlier run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
9.4 KiB
BI-P014, P016–P022, P025–P026, P028–P031 — query, metric, quality
Extends the analytics core merged in 0pt#34 (BI-P002..P012). Nothing there is
rebuilt.
This document is organised around what is proven and what is not, because the second list is the one that gets dropped.
Where things live
| prompt | module | live proof |
|---|---|---|
| BI-P014 DuckDB provider | src/analytics/providers/duckdb-provider.ts |
tests/duckdb/duckdb-provider.live.test.ts |
| BI-P016 query request contract | src/analytics/contract/query-request.ts, compile-clickhouse.ts, compile-semantic.ts |
unit only (pure compilation) |
| BI-P017 analyst SQL contract | src/analytics/sql/ |
tests/integration/analyst-sql-clickhouse.integration.test.ts |
| BI-P018 execution model | src/analytics/execution/query-execution-record.ts |
sql/009 + Postgres realinfra |
| BI-P019 cancellation | src/analytics/execution/cancellation.ts |
tests/integration/query-cancellation-budget.integration.test.ts |
| BI-P020 query cache | src/analytics/cache/query-cache.ts |
unit only (no engine involved) |
| BI-P021 cost / budget | src/analytics/cost/query-budget.ts |
same live suite as BI-P019 |
| BI-P022 dbt contract | src/analytics/transform/dbt-contract.ts, analytics/dbt/ |
contract only — execution is n0d#38 |
| BI-P025 quality provider | src/analytics/quality/ |
tests/integration/data-quality-clickhouse.integration.test.ts |
| BI-P026 freshness | src/analytics/quality/freshness.ts |
unit + sql/009 defaults |
| BI-P028/P031 metric model + governance | src/analytics/metrics/metric-model.ts |
Postgres trigger, realinfra |
| BI-P029 semantic provider | src/analytics/metrics/semantic-provider.ts |
partially — see below |
| BI-P030 semantic registry | src/analytics/metrics/semantic-registry.ts |
unit + Postgres constraints |
Five findings that changed the implementation
Each was found by writing a test that could fail, and each is pinned so it cannot quietly stop being true.
1. ClickHouse readonly = 1 on a user profile forbids every request setting
Measured on 24.8.14.39. A user whose profile carries readonly = 1 answers
Code: 164. Cannot modify 'max_rows_to_read' setting in readonly mode to any
setting sent with the statement. So BI-P012's ceilings cannot travel with an
analyst statement — they are baked into the user's SETTINGS PROFILE at
provisioning. Proven both ways: an over-budget scan fails with code 158 from
the profile with no setting sent, and an attempt to raise it from the request
fails with 164.
2. CREATE ROW POLICY IF NOT EXISTS is a silent no-op on re-provisioning
Dropping a user empties the policy's apply-to list. The next provisioning run
sees the policy exists and does nothing, so it survives with
apply_to_list = [] — applying to nobody — while system.row_policies
still shows the correct select_filter. The restricted user read 100 of 100
rows, both tenants, from the base table.
Fixed with OR REPLACE, plus rowPolicyVerificationSql() which turns "the
policy is applied" from an assumption into a query, and a test asserting
applied_to_user = 1 per object. CREATE SETTINGS PROFILE IF NOT EXISTS has
the identical shape and is fixed the same way.
3. DuckDB's allowed_directories alone is a no-op
Measured on 1.5.5. With allowed_directories set to one mount directory and
nothing else, read_csv('/etc/passwd') returned the file and an https://
URL was fetched over the network. Only enable_external_access = false
alongside it makes the allow-list bind.
The obvious spelling of this control appears in duckdb_settings(), looks like
a sandbox, and permits everything. The measurement is pinned by a test that
never skips, so simplifying the bootstrap breaks the build rather than the
guarantee.
4. pg_constraint.conname is not schema-scoped
SELECT 1 FROM pg_constraint WHERE conname = '...' matches a same-named
constraint in any schema, so sql/009 applied into a second schema
silently skipped every guarded CHECK. Every guard now scopes via
conrelid = '<table>'::regclass.
5. describe.skipIf(flagSetInBeforeAll) skips always
Vitest evaluates skipIf at collection time. Ten tests reported green while
skipping against a live, reachable ClickHouse. Replaced with the repo's
env-var gate, which fails when the variable is set and the server is down.
What is NOT proven
- BI-P029 / MetricFlow was never executed. No Python runtime is
provisioned for this repo's CI, and BI-P029's "pin a currently verified
version" needs a licence and supply-chain review that has not happened.
MetricFlowProvider.providerReadiness()returnsverified: falseand names all four open items. What is proven is that the engine-neutral spec compiles to a MetricFlow-shaped query and that canonical SIAX metric ids survive a provider swap — tested against a second, deliberately different-named provider so the equality is not trivially true. - The Superset integration path and the AI Analyst comparison (BI-P029's other two DoD surfaces) are not implemented.
- dbt is not run here.
DBT_RUNTIME_PINhas both review flagsfalse, andassertDbtRuntimeCertifiedrefuses a production run until they are true. - DuckDB Arrow IPC is unproven: the
arrowextension is a network install the offline environment could not download.readiness()says so. - BI-P021's budget is a bound, not a hard cap. In-flight enforcement is
granular to the poll interval, so the query overshoots between polls — and on
a columnar engine the overshoot is large. In the committed run
(
evidence/bi-p019-p021/) a 1,000,000-row window cancelled the query at 153,318,696 rows, 55 ms in: a 153× overshoot. The figure is machine-dependent and moves between runs (an earlier run on the same box recorded 131 M / 130×), which is itself the point — the bound is "roughly one poll interval of engine throughput", not a number. It is still a bound: the same query, unwatched, read 4,227,318,261 rows before cancellation in the BI-P019 case. Anyone quoting it as a hard cap is wrong. - The per-process window store does not bind across replicas.
describeEnforcement()reports"with N replicas a tenant can spend up to N times this budget". A shared store (Valkey, or the CL0UD quota service) behindSharedWindowStoreis required for it to bind.
Running the live suites
docker run -d --name ch -p 28123:8123 -e CLICKHOUSE_SKIP_USER_SETUP=1 \
clickhouse/clickhouse-server:24.8
docker run -d --name pg -p 25433:5432 -e POSTGRES_PASSWORD=test \
-e POSTGRES_DB=opt postgres:16-alpine
CLICKHOUSE_URL=http://127.0.0.1:28123 pnpm vitest run tests/integration
ANALYTICS_TEST_DATABASE_URL=postgres://postgres:test@127.0.0.1:25433/opt \
pnpm vitest run tests/analytics/analytics-governance-postgres.realinfra.test.ts
python -m venv .venv && .venv/bin/pip install duckdb pyarrow 'pyiceberg[sql-sqlite]'
DUCKDB_PYTHON=.venv/bin/python pnpm vitest run tests/duckdb
Every one of these fails rather than skipping when its variable is set and the backend is unreachable.
One note for whoever maintains .gitea/workflows/ci.yml
Not changed here — it is a protected file, and this branch touched none.
The integration job's guard is:
if grep -qE "skipped" /tmp/itest.log && ! grep -qE "Tests [0-9]+ passed" /tmp/itest.log
Vitest writes that summary with ANSI colour codes, so the literal text is
Tests [22m [1m[32m81 passed[39m…. The second pattern therefore never
matches, and the guard is effectively "any skip fails the job" rather than
the "skipped and nothing passed" it reads as. That is stricter than intended,
not looser, so nothing is currently unguarded — but the escape hatch does not
work, and someone will eventually be puzzled by it. Piping through
sed 's/\x1b\[[0-9;]*m//g' before the grep would restore the intended
behaviour.
This is how the DuckDB suite's placement was caught, so the guard earned its keep either way.