CI (SIAX Cloud) / quality (push) Successful in 48s
CI (SIAX Cloud) / integration (push) Successful in 37s
CI (SIAX Cloud) / security (push) Successful in 17s
Security Scan (SIAX Cloud) / security (push) Successful in 17s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m16s
CI (SIAX Cloud) / integration (pull_request) Successful in 55s
CI (SIAX Cloud) / security (pull_request) Successful in 35s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 29s
The live suites regenerate their evidence on every run, so the BI-P021 overshoot figures moved when they were last re-run — and the doc and module comment still quoted the earlier ones (130,883,409 rows / 130x / 92ms) while the committed file said 153,318,696 / 153x / 55ms. Prose that disagrees with the evidence it cites is worse than prose with no number in it. Both now match the committed file, and both say what the earlier version implied but did not state: the multiple is MACHINE-DEPENDENT and moves between runs on the same box. That is the actual finding. The bound is "roughly one poll interval of engine throughput", not a number, and quoting any specific multiple as if it were a property of the system is the mistake this correction exists to prevent. The unwatched-baseline figure is corrected the same way: 4,227,318,261 rows read before cancellation in the current BI-P019 record, not the 6.5 billion of the earlier run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
161 lines
9.4 KiB
Markdown
161 lines
9.4 KiB
Markdown
# BI-P014, P016–P022, P025–P026, P028–P031 — query, metric, quality
|
||
|
||
Extends the analytics core merged in `0pt#34` (BI-P002..P012). Nothing there is
|
||
rebuilt.
|
||
|
||
This document is organised around what is **proven** and what is **not**,
|
||
because the second list is the one that gets dropped.
|
||
|
||
---
|
||
|
||
## Where things live
|
||
|
||
| prompt | module | live proof |
|
||
| -------------------------------------- | ----------------------------------------------------------------------------------------- | ----------------------------------------------------------------- |
|
||
| BI-P014 DuckDB provider | `src/analytics/providers/duckdb-provider.ts` | `tests/duckdb/duckdb-provider.live.test.ts` |
|
||
| BI-P016 query request contract | `src/analytics/contract/query-request.ts`, `compile-clickhouse.ts`, `compile-semantic.ts` | unit only (pure compilation) |
|
||
| BI-P017 analyst SQL contract | `src/analytics/sql/` | `tests/integration/analyst-sql-clickhouse.integration.test.ts` |
|
||
| BI-P018 execution model | `src/analytics/execution/query-execution-record.ts` | `sql/009` + Postgres realinfra |
|
||
| BI-P019 cancellation | `src/analytics/execution/cancellation.ts` | `tests/integration/query-cancellation-budget.integration.test.ts` |
|
||
| BI-P020 query cache | `src/analytics/cache/query-cache.ts` | unit only (no engine involved) |
|
||
| BI-P021 cost / budget | `src/analytics/cost/query-budget.ts` | same live suite as BI-P019 |
|
||
| BI-P022 dbt contract | `src/analytics/transform/dbt-contract.ts`, `analytics/dbt/` | contract only — execution is `n0d#38` |
|
||
| BI-P025 quality provider | `src/analytics/quality/` | `tests/integration/data-quality-clickhouse.integration.test.ts` |
|
||
| BI-P026 freshness | `src/analytics/quality/freshness.ts` | unit + `sql/009` defaults |
|
||
| BI-P028/P031 metric model + governance | `src/analytics/metrics/metric-model.ts` | Postgres trigger, realinfra |
|
||
| BI-P029 semantic provider | `src/analytics/metrics/semantic-provider.ts` | **partially — see below** |
|
||
| BI-P030 semantic registry | `src/analytics/metrics/semantic-registry.ts` | unit + Postgres constraints |
|
||
|
||
---
|
||
|
||
## Five findings that changed the implementation
|
||
|
||
Each was found by writing a test that could fail, and each is pinned so it
|
||
cannot quietly stop being true.
|
||
|
||
### 1. ClickHouse `readonly = 1` on a user profile forbids **every** request setting
|
||
|
||
Measured on 24.8.14.39. A user whose profile carries `readonly = 1` answers
|
||
`Code: 164. Cannot modify 'max_rows_to_read' setting in readonly mode` to any
|
||
setting sent with the statement. So BI-P012's ceilings **cannot travel with an
|
||
analyst statement** — they are baked into the user's `SETTINGS PROFILE` at
|
||
provisioning. Proven both ways: an over-budget scan fails with code 158 from
|
||
the profile with no setting sent, and an attempt to raise it from the request
|
||
fails with 164.
|
||
|
||
### 2. `CREATE ROW POLICY IF NOT EXISTS` is a silent no-op on re-provisioning
|
||
|
||
Dropping a user empties the policy's apply-to list. The next provisioning run
|
||
sees the policy exists and does nothing, so it survives with
|
||
`apply_to_list = []` — applying to **nobody** — while `system.row_policies`
|
||
still shows the correct `select_filter`. The restricted user read 100 of 100
|
||
rows, **both tenants**, from the base table.
|
||
|
||
Fixed with `OR REPLACE`, plus `rowPolicyVerificationSql()` which turns "the
|
||
policy is applied" from an assumption into a query, and a test asserting
|
||
`applied_to_user = 1` per object. `CREATE SETTINGS PROFILE IF NOT EXISTS` has
|
||
the identical shape and is fixed the same way.
|
||
|
||
### 3. DuckDB's `allowed_directories` **alone** is a no-op
|
||
|
||
Measured on 1.5.5. With `allowed_directories` set to one mount directory and
|
||
nothing else, `read_csv('/etc/passwd')` **returned the file** and an `https://`
|
||
URL was fetched over the network. Only `enable_external_access = false`
|
||
alongside it makes the allow-list bind.
|
||
|
||
The obvious spelling of this control appears in `duckdb_settings()`, looks like
|
||
a sandbox, and permits everything. The measurement is pinned by a test that
|
||
never skips, so simplifying the bootstrap breaks the build rather than the
|
||
guarantee.
|
||
|
||
### 4. `pg_constraint.conname` is not schema-scoped
|
||
|
||
`SELECT 1 FROM pg_constraint WHERE conname = '...'` matches a same-named
|
||
constraint in **any** schema, so `sql/009` applied into a second schema
|
||
silently skipped every guarded `CHECK`. Every guard now scopes via
|
||
`conrelid = '<table>'::regclass`.
|
||
|
||
### 5. `describe.skipIf(flagSetInBeforeAll)` skips **always**
|
||
|
||
Vitest evaluates `skipIf` at collection time. Ten tests reported green while
|
||
skipping against a live, reachable ClickHouse. Replaced with the repo's
|
||
env-var gate, which **fails** when the variable is set and the server is down.
|
||
|
||
---
|
||
|
||
## What is NOT proven
|
||
|
||
- **BI-P029 / MetricFlow was never executed.** No Python runtime is
|
||
provisioned for this repo's CI, and BI-P029's "pin a currently verified
|
||
version" needs a licence and supply-chain review that has not happened.
|
||
`MetricFlowProvider.providerReadiness()` returns `verified: false` and names
|
||
all four open items. What _is_ proven is that the engine-neutral spec
|
||
compiles to a MetricFlow-shaped query and that canonical SIAX metric ids
|
||
survive a provider swap — tested against a second, deliberately
|
||
different-named provider so the equality is not trivially true.
|
||
- **The Superset integration path and the AI Analyst comparison** (BI-P029's
|
||
other two DoD surfaces) are not implemented.
|
||
- **dbt is not run here.** `DBT_RUNTIME_PIN` has both review flags `false`, and
|
||
`assertDbtRuntimeCertified` refuses a production run until they are true.
|
||
- **DuckDB Arrow IPC** is unproven: the `arrow` extension is a network install
|
||
the offline environment could not download. `readiness()` says so.
|
||
- **BI-P021's budget is a bound, not a hard cap.** In-flight enforcement is
|
||
granular to the poll interval, so the query overshoots between polls — and on
|
||
a columnar engine the overshoot is large. In the committed run
|
||
(`evidence/bi-p019-p021/`) a 1,000,000-row window cancelled the query at
|
||
**153,318,696 rows**, 55 ms in: a **153×** overshoot. The figure is
|
||
machine-dependent and moves between runs (an earlier run on the same box
|
||
recorded 131 M / 130×), which is itself the point — the bound is "roughly one
|
||
poll interval of engine throughput", not a number. It is still a bound: the
|
||
same query, unwatched, read **4,227,318,261 rows** before cancellation in the
|
||
BI-P019 case. Anyone quoting it as a hard cap is wrong.
|
||
- **The per-process window store does not bind across replicas.**
|
||
`describeEnforcement()` reports `"with N replicas a tenant can spend up to N
|
||
times this budget"`. A shared store (Valkey, or the CL0UD quota service)
|
||
behind `SharedWindowStore` is required for it to bind.
|
||
|
||
---
|
||
|
||
## Running the live suites
|
||
|
||
```bash
|
||
docker run -d --name ch -p 28123:8123 -e CLICKHOUSE_SKIP_USER_SETUP=1 \
|
||
clickhouse/clickhouse-server:24.8
|
||
docker run -d --name pg -p 25433:5432 -e POSTGRES_PASSWORD=test \
|
||
-e POSTGRES_DB=opt postgres:16-alpine
|
||
|
||
CLICKHOUSE_URL=http://127.0.0.1:28123 pnpm vitest run tests/integration
|
||
ANALYTICS_TEST_DATABASE_URL=postgres://postgres:test@127.0.0.1:25433/opt \
|
||
pnpm vitest run tests/analytics/analytics-governance-postgres.realinfra.test.ts
|
||
|
||
python -m venv .venv && .venv/bin/pip install duckdb pyarrow 'pyiceberg[sql-sqlite]'
|
||
DUCKDB_PYTHON=.venv/bin/python pnpm vitest run tests/duckdb
|
||
```
|
||
|
||
Every one of these **fails** rather than skipping when its variable is set and
|
||
the backend is unreachable.
|
||
|
||
---
|
||
|
||
## One note for whoever maintains `.gitea/workflows/ci.yml`
|
||
|
||
Not changed here — it is a protected file, and this branch touched none.
|
||
|
||
The integration job's guard is:
|
||
|
||
```sh
|
||
if grep -qE "skipped" /tmp/itest.log && ! grep -qE "Tests [0-9]+ passed" /tmp/itest.log
|
||
```
|
||
|
||
Vitest writes that summary with ANSI colour codes, so the literal text is
|
||
`Tests [22m [1m[32m81 passed[39m…`. The second pattern therefore **never
|
||
matches**, and the guard is effectively "any skip fails the job" rather than
|
||
the "skipped _and_ nothing passed" it reads as. That is stricter than intended,
|
||
not looser, so nothing is currently unguarded — but the escape hatch does not
|
||
work, and someone will eventually be puzzled by it. Piping through
|
||
`sed 's/\x1b\[[0-9;]*m//g'` before the grep would restore the intended
|
||
behaviour.
|
||
|
||
This is how the DuckDB suite's placement was caught, so the guard earned its
|
||
keep either way.
|