Files
0pt/docs/analytics/BI_QUERY_METRIC_QUALITY.md
T
Claude Opus 5 c4862fb7a7
CI (SIAX Cloud) / quality (push) Successful in 48s
CI (SIAX Cloud) / integration (push) Successful in 37s
CI (SIAX Cloud) / security (push) Successful in 17s
Security Scan (SIAX Cloud) / security (push) Successful in 17s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m16s
CI (SIAX Cloud) / integration (pull_request) Successful in 55s
CI (SIAX Cloud) / security (pull_request) Successful in 35s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 29s
docs(analytics): quote the numbers the COMMITTED evidence actually contains
The live suites regenerate their evidence on every run, so the BI-P021
overshoot figures moved when they were last re-run — and the doc and module
comment still quoted the earlier ones (130,883,409 rows / 130x / 92ms) while
the committed file said 153,318,696 / 153x / 55ms. Prose that disagrees with
the evidence it cites is worse than prose with no number in it.

Both now match the committed file, and both say what the earlier version
implied but did not state: the multiple is MACHINE-DEPENDENT and moves between
runs on the same box. That is the actual finding. The bound is "roughly one
poll interval of engine throughput", not a number, and quoting any specific
multiple as if it were a property of the system is the mistake this correction
exists to prevent.

The unwatched-baseline figure is corrected the same way: 4,227,318,261 rows
read before cancellation in the current BI-P019 record, not the 6.5 billion of
the earlier run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 03:18:12 +02:00

9.4 KiB
Raw Blame History

BI-P014, P016–P022, P025–P026, P028–P031 — query, metric, quality

Extends the analytics core merged in 0pt#34 (BI-P002..P012). Nothing there is rebuilt.

This document is organised around what is proven and what is not, because the second list is the one that gets dropped.


Where things live

prompt module live proof
BI-P014 DuckDB provider src/analytics/providers/duckdb-provider.ts tests/duckdb/duckdb-provider.live.test.ts
BI-P016 query request contract src/analytics/contract/query-request.ts, compile-clickhouse.ts, compile-semantic.ts unit only (pure compilation)
BI-P017 analyst SQL contract src/analytics/sql/ tests/integration/analyst-sql-clickhouse.integration.test.ts
BI-P018 execution model src/analytics/execution/query-execution-record.ts sql/009 + Postgres realinfra
BI-P019 cancellation src/analytics/execution/cancellation.ts tests/integration/query-cancellation-budget.integration.test.ts
BI-P020 query cache src/analytics/cache/query-cache.ts unit only (no engine involved)
BI-P021 cost / budget src/analytics/cost/query-budget.ts same live suite as BI-P019
BI-P022 dbt contract src/analytics/transform/dbt-contract.ts, analytics/dbt/ contract only — execution is n0d#38
BI-P025 quality provider src/analytics/quality/ tests/integration/data-quality-clickhouse.integration.test.ts
BI-P026 freshness src/analytics/quality/freshness.ts unit + sql/009 defaults
BI-P028/P031 metric model + governance src/analytics/metrics/metric-model.ts Postgres trigger, realinfra
BI-P029 semantic provider src/analytics/metrics/semantic-provider.ts partially — see below
BI-P030 semantic registry src/analytics/metrics/semantic-registry.ts unit + Postgres constraints

Five findings that changed the implementation

Each was found by writing a test that could fail, and each is pinned so it cannot quietly stop being true.

1. ClickHouse readonly = 1 on a user profile forbids every request setting

Measured on 24.8.14.39. A user whose profile carries readonly = 1 answers Code: 164. Cannot modify 'max_rows_to_read' setting in readonly mode to any setting sent with the statement. So BI-P012's ceilings cannot travel with an analyst statement — they are baked into the user's SETTINGS PROFILE at provisioning. Proven both ways: an over-budget scan fails with code 158 from the profile with no setting sent, and an attempt to raise it from the request fails with 164.

2. CREATE ROW POLICY IF NOT EXISTS is a silent no-op on re-provisioning

Dropping a user empties the policy's apply-to list. The next provisioning run sees the policy exists and does nothing, so it survives with apply_to_list = [] — applying to nobody — while system.row_policies still shows the correct select_filter. The restricted user read 100 of 100 rows, both tenants, from the base table.

Fixed with OR REPLACE, plus rowPolicyVerificationSql() which turns "the policy is applied" from an assumption into a query, and a test asserting applied_to_user = 1 per object. CREATE SETTINGS PROFILE IF NOT EXISTS has the identical shape and is fixed the same way.

3. DuckDB's allowed_directories alone is a no-op

Measured on 1.5.5. With allowed_directories set to one mount directory and nothing else, read_csv('/etc/passwd') returned the file and an https:// URL was fetched over the network. Only enable_external_access = false alongside it makes the allow-list bind.

The obvious spelling of this control appears in duckdb_settings(), looks like a sandbox, and permits everything. The measurement is pinned by a test that never skips, so simplifying the bootstrap breaks the build rather than the guarantee.

4. pg_constraint.conname is not schema-scoped

SELECT 1 FROM pg_constraint WHERE conname = '...' matches a same-named constraint in any schema, so sql/009 applied into a second schema silently skipped every guarded CHECK. Every guard now scopes via conrelid = '<table>'::regclass.

5. describe.skipIf(flagSetInBeforeAll) skips always

Vitest evaluates skipIf at collection time. Ten tests reported green while skipping against a live, reachable ClickHouse. Replaced with the repo's env-var gate, which fails when the variable is set and the server is down.


What is NOT proven

  • BI-P029 / MetricFlow was never executed. No Python runtime is provisioned for this repo's CI, and BI-P029's "pin a currently verified version" needs a licence and supply-chain review that has not happened. MetricFlowProvider.providerReadiness() returns verified: false and names all four open items. What is proven is that the engine-neutral spec compiles to a MetricFlow-shaped query and that canonical SIAX metric ids survive a provider swap — tested against a second, deliberately different-named provider so the equality is not trivially true.
  • The Superset integration path and the AI Analyst comparison (BI-P029's other two DoD surfaces) are not implemented.
  • dbt is not run here. DBT_RUNTIME_PIN has both review flags false, and assertDbtRuntimeCertified refuses a production run until they are true.
  • DuckDB Arrow IPC is unproven: the arrow extension is a network install the offline environment could not download. readiness() says so.
  • BI-P021's budget is a bound, not a hard cap. In-flight enforcement is granular to the poll interval, so the query overshoots between polls — and on a columnar engine the overshoot is large. In the committed run (evidence/bi-p019-p021/) a 1,000,000-row window cancelled the query at 153,318,696 rows, 55 ms in: a 153× overshoot. The figure is machine-dependent and moves between runs (an earlier run on the same box recorded 131 M / 130×), which is itself the point — the bound is "roughly one poll interval of engine throughput", not a number. It is still a bound: the same query, unwatched, read 4,227,318,261 rows before cancellation in the BI-P019 case. Anyone quoting it as a hard cap is wrong.
  • The per-process window store does not bind across replicas. describeEnforcement() reports "with N replicas a tenant can spend up to N times this budget". A shared store (Valkey, or the CL0UD quota service) behind SharedWindowStore is required for it to bind.

Running the live suites

docker run -d --name ch -p 28123:8123 -e CLICKHOUSE_SKIP_USER_SETUP=1 \
  clickhouse/clickhouse-server:24.8
docker run -d --name pg -p 25433:5432 -e POSTGRES_PASSWORD=test \
  -e POSTGRES_DB=opt postgres:16-alpine

CLICKHOUSE_URL=http://127.0.0.1:28123 pnpm vitest run tests/integration
ANALYTICS_TEST_DATABASE_URL=postgres://postgres:test@127.0.0.1:25433/opt \
  pnpm vitest run tests/analytics/analytics-governance-postgres.realinfra.test.ts

python -m venv .venv && .venv/bin/pip install duckdb pyarrow 'pyiceberg[sql-sqlite]'
DUCKDB_PYTHON=.venv/bin/python pnpm vitest run tests/duckdb

Every one of these fails rather than skipping when its variable is set and the backend is unreachable.


One note for whoever maintains .gitea/workflows/ci.yml

Not changed here — it is a protected file, and this branch touched none.

The integration job's guard is:

if grep -qE "skipped" /tmp/itest.log && ! grep -qE "Tests  [0-9]+ passed" /tmp/itest.log

Vitest writes that summary with ANSI colour codes, so the literal text is Tests [22m [1m[32m81 passed[39m…. The second pattern therefore never matches, and the guard is effectively "any skip fails the job" rather than the "skipped and nothing passed" it reads as. That is stricter than intended, not looser, so nothing is currently unguarded — but the escape hatch does not work, and someone will eventually be puzzled by it. Piping through sed 's/\x1b\[[0-9;]*m//g' before the grep would restore the intended behaviour.

This is how the DuckDB suite's placement was caught, so the guard earned its keep either way.