Certifieringslanan (BI-P147) hittade det; jag verifierade det oberoende mot
LICENSE-filen vid respektive git-tagg 2026-09-07:
v0.208.1 Business Source License 1.1 — source-available, INTE öppen källkod
v0.209.0 Apache License 2.0
dbt Labs licensierade om MetricFlow vid 0.209.0. Den gamla pinnen låg alltså
exakt en patch under gränsen. För en självhostad plattform vars hela premiss
är att inte ha ett proprietärt runtime-beroende är det en licensdefekt, inte
en smaksak: BUSL begränsar produktionsanvändning fram till sitt change date.
Notera metodpunkten: PyPI redovisar INGET licensfält för någondera versionen,
så en metadatakontroll hade inte hittat något. Skillnaden syns bara i
LICENSE-filen vid taggen.
Ny grind: `tests/unit/metricflow-licence.test.ts` asserterar att pinnen ligger
på eller över `METRICFLOW_FIRST_OPEN_SOURCE_VERSION`, och — som negativ
kontroll — att alla tre kända BUSL-versionerna (0.207.3, 0.208.0, 0.208.1)
faktiskt förkastas av samma jämförelse. Utan den andra assertionen hade den
första kunnat passera med en jämförelse som inte diskriminerar.
Detta gör INTE MetricFlow verifierad: `providerReadiness()` returnerar
fortfarande `verified: false`, ingen supply-chain-granskning är gjord och
providern har aldrig körts (ingen Python-runtime är provisionerad). Att laga
licensen är inte samma sak som att bevisa komponenten.
typecheck rent, 2/2 tester gröna.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
BI-P029 + BI-P103: two prompts cited as done were not (#38)
The BI-P029 parity test and the BI-P103 baseline evidence were both cited in
headers and neither existed. Writing the parity test exposed metric definition
filters (BI-P028 MetricSpec.filters) being dropped by the ClickHouse compile
path, so a metric defined as "revenue excluding internal transfers" executed as
"revenue" -- 11 249 instead of 1 250 on the committed fixture.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Same shape as the BI-P029 finding in the previous commit. `bi-slo.ts` said
"evidence/bi-p103 records the first baselines". There was no evidence/bi-p103.
The catalogue and the evaluator were both real and both well covered — against
SYNTHETIC events. A synthetic event carries whatever duration the test author
typed, so that suite proves the arithmetic and nothing about the system: it
cannot tell you whether a dashboard query takes 8 ms or 8 s. BI-P103's DoD is
one sentence, "measured baselines exist", and it was the part not done.
tests/integration/bi-p103-slo-baseline.integration.test.ts runs 460 real
queries against a live ClickHouse over a 400 000-row fact table -- 120 cold
dashboard queries (each a different bucket so the BI-P020 cache genuinely
misses), the same 120 again through the same shipped cache now hitting, and
220 exploration queries, that last number chosen so explore.p99's min_samples
of 200 is MET rather than reported low-confidence. Every duration is
wall-clock around the real call, and the telemetry goes through the shipped
producer and the shipped evaluator, so the evidence is written by the code
under test.
Measured: 7 of 11 objectives produced a number, `fully_measured` false,
measured_fraction 0.64. The four that did not each say why -- dashboard
freshness is a property of the dataset, the exploration timeout is a
configured ceiling, report completion has no producer in 0pt at all, and
report delivery is N0TIFY's receipt. A baseline table listing only the rows
that worked is how an estate comes to believe it measures more than it does.
The evidence also records what is NARROWER than production, because these
numbers will be quoted: the cached path is the in-process reference cache and
not Valkey, every latency is the engine round trip with no HTTP or
authorization layer, and there is no concurrency.
The suite is adversarial about its own inputs: a "cold" sample that came from
the cache fails, a cache "hit" that missed fails, a cached p95 that is not
faster than the cold p95 fails, and dropping the exploration sample to 60
fails on explore.p99 being low-confidence rather than quietly recording a
p99 over 60 requests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`semantic-provider.ts` cited tests/integration/semantic-parity.integration.test.ts
as the proof of the one BI-P029 DoD surface it claimed was closed, and
`providerReadiness().proven` asserted the comparison had been made against live
ClickHouse. The file did not exist. The citation was the evidence.
Writing it exposed a divergence that had been shipping:
`compileRequestToClickHouse` had no notion of a metric's OWN filters
(BI-P028 `MetricSpec.filters`). `ResolvedMetric` did not carry them and the
compiler never asked, so a metric defined as "invoiced revenue EXCLUDING
internal transfers" executed as "invoiced revenue". On the committed fixture
that is 11 249 instead of 1 250 for one country -- a plausible-looking number,
9x too large, under the correct canonical metric id. The registry, the
governance checks and the MetricFlow manifest all carried the filter; only the
thing that actually ran did not.
* `ResolvedMetric` gains `definition_filters`, already resolved onto the
physical dim_names key so the compilers stay dumb.
* The ClickHouse compiler folds them into THAT metric's conditional
aggregate, never into the shared WHERE -- two metrics in one request may
exclude different rows, and a shared predicate would apply one metric's
meaning to the other. `renderFilter`'s uniquifier becomes a string so
request filter `0` and metric filter `0_0m` cannot collide in the
parameter map.
* `compileRequestToSemantic` carries them through, and
`MetricFlowProvider.compile` emits them per metric.
The test runs both compile paths against one live ClickHouse: 0pt's `sumIf`
compiler, and a renderer driven only by the provider's structured query that
shares no SQL-generation code and emits `sum(if(...))` with inlined literals.
It asserts the two SQL strings differ, so parity is not vacuous. Backing the
fix out reproduces the pre-fix numbers, and the suite records them itself
rather than quoting a console: evidence/bi-p029/semantic-parity.json.
Three of BI-P029's four DoD surfaces stay OPEN and still say so: MetricFlow
itself is not executed (no Python runtime is provisioned here), Superset is not
implemented, and the AI Analyst path has not been compared.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
BI-P129 names four provider kinds (native, dlt, Meltano/Singer, Airbyte-if-
licensed). This commit ships the seam and the `native` provider; dlt and
Singer follow on this branch.
What is deliberately NOT here: a second ingestion stack. 0pt already has the
P061 connector contract, the P062 runner with durable checkpoints and
crash/resume tests, the schema registry, the quality gate and the idempotency
ledger. A provider's job is to hand back a P061 `Connector`; the highway is
the one that exists.
What is new is the question the P061 stack has no opinion about: an ingestion
run has to LAND somewhere, and where it lands is an analytics contract
question. Four fail-closed gates, each attacked with the guard (must refuse)
AND without it (must succeed — otherwise the guard is decoration):
1. tier — source data lands in RAW, or NORMALIZED with a declared
normalization. Never directly in CURATED/SERVING. BI-P003's "no business
dashboard reads arbitrary raw ingestion tables" is defeated in one line
by relabelling the landing table SERVING; the tier is therefore checked
at WRITE, not only at read.
2. contract coverage — every field discovery actually reported must be
declared in the BI-P004 contract. An undeclared column is an
unclassified column. The counterfactual test lands `personnummer` into a
RAW table whose contract says pii_classification "none", and the runner
reports success: that is the harm, shown rather than asserted.
3. physical home — RAW/NORMALIZED land as Parquet/Iceberg in ST0RE
(st0re#45 shapes), never straight into ClickHouse.
4. provider readiness — a provider whose own providerReadiness() says
verified:false cannot run in a production runtime without a recorded,
named exception. This is the structural form of the honesty rule 0pt#35
set with MetricFlow.
Airbyte is a declared kind with NO implementation: its server is ELv2, which
restricts offering the software as a managed service, and that is exactly
what 0pt does. Registering or planning one is refused at the registry, in one
place, so a future licence review has a single constant to change.
BI-P129 DoD — "two real source types ingest into canonical analytics tiers" —
is proven with the two real, heterogeneous P061 connectors: a JSONL file on
actual disk (cursor = line offset, real CDC deletes) and an HTTP JSON API
driven through a real fetch/Response cycle (cursor = opaque server token).
Both run through the real runner into the declared RAW tier, paged, with
9 rows landed each and distinct envelope ids.
Verification: 24 tests. Mutation-checked — neutering gates 1 and 2 fails 5 of
them, so they are load-bearing rather than incidental.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Includes BI-P105's freshness semantics as its DoD asks: watermark, staleness,
the eight refusal reasons, grain rollup limits, and the sentence the evidence
exists to support — the drifted aggregate was ONE SECOND old and overstated
revenue by 5.251%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Six dataset profiles (small SaaS, medium business, VehIQ market, large event,
wide table, geospatial), one representative question each, run on ClickHouse
26.3.10.60 and DuckDB 1.5.5 with 2 warmups discarded and 9 timed repetitions.
evidence/bi-p104/benchmark.json.
The harness's job is to stop the benchmark from lying:
- overlapping ranges are INDISTINGUISHABLE whatever the medians say. Three
of the six real comparisons came out that way, one of them with a 1.6x
median gap — which is exactly the headline a noisy host manufactures.
- every sentence carries the range, the sample size and the stability, so it
cannot be trimmed to "4.4x faster" without visibly deleting the caveats.
- under 5 samples nothing is comparable at all. A three-run benchmark is an
anecdote.
- every engine BI-P104 names appears in the evidence, and the five nothing
measured carry a reason each. The running Trino belongs to another lane,
so benchmarking through it would have measured their configuration.
Measured shape, warm cache, one idle host, these query shapes only: DuckDB
faster on small_saas (1.6-8.2x) and vehiq_market (2.4-13.3x), SLOWER on
large_event (0.2-0.9x, count DISTINCT over 5M rows), indistinguishable on
medium_business, wide_table and geospatial.
One finding from the harness's own tests: an engine declared available that
produced no run came out of buildEvidence() with no reason — the dangerous
case, since it reads as "we could have and chose not to say". It now gets a
generated reason. A second from the runs: DuckDB's progress bar writes to
STDOUT and corrupted the first transcript; it is disabled explicitly and the
JSON is read from a sentinel.
10 unit tests + 4 against real engines.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
aud0#34 merged seven BI usage event types and recorded that zero producers
emit them. This makes 0pt the producer and derives the SLOs from those events
rather than opening a second, private measurement path that would disagree
with aud0 during an incident.
Three honest verdicts instead of one number:
insufficient_data an availability objective over an empty window is
SILENCE, not 100%. Every objective in an empty window
comes back insufficient_data and fully_measured is
false. A started query with no terminal event counts
as unresolved — not as a success.
not_measurable_here dashboard FRESHNESS is a property of the dataset
(BI-P026/BI-P105 watermark) and report DELIVERY is
N0TIFY's receipt. Neither is in aud0's seven types, so
both are declared unmeasured rather than proxied by
completion.
low_confidence a p99 over 12 samples is the 12th-slowest request
wearing a percentile's clothes. Reported with its
sample size and not judged.
The producer is an allowlist built field by named field, never a spread. The
test feeds it an execution record carrying SQL text, a database error message
quoting a personnummer, filter VALUES, a tenant id and a result sample, then
greps the emitted events — after first proving the grep finds those strings in
the input, so a clean result cannot be a broken detector.
14 tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A slow query is a complaint; a drifted pre-aggregate answers instantly, with
confidence, and is wrong. So the router is fail-closed on eight named reasons,
and the one people leave out — drift_never_verified — is the one that matters
on launch day: a brand-new aggregate is exactly as unproven as a drifted one.
Proved against ClickHouse 26.3.10.60, 2M rows, a real materialized view
(evidence/bi-p105/materialization-drift.json):
after backfill match, 48 buckets, 801 571 002 both sides
GDPR erasure ALTER TABLE ... DELETE on the source. An MV is an INSERT
trigger and never sees a mutation, so the aggregate kept
the deleted revenue: it would have reported 801 571 002
against a true 761 580 554 — 39 990 448 overstated,
5.251%, with NOTHING stale. Staleness at the moment of
the decision: 1 second.
guarded router base_table (state_not_serving)
counterfactual a staleness-only router SERVES it — measured, not assumed
after rebuild match, and routing returns to the materialization
Drift detection treats a MISSING bucket as drift even when every value present
is correct (late-arriving rows are the sneakiest failure), and an empty
comparison as insufficient_data rather than match — the same false green
BI-P025 found in a quality check over an empty table.
The measured speed-up is reported as a distribution, not a figure: p50 3.4x,
but the run-to-run range spans 0.6x-7.5x on one idle host, so quoting "3.4x"
as a property of the system would be misleading (BI-P104 doctrine).
19 unit tests + 4 against live ClickHouse.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
DRAFT -> PUBLISHED -> SUPERSEDED as a ledger (sql/010) rather than a column on
the frozen version: sql/008 already makes DashboardVersion immutable at the
database, and a lifecycle column there would have meant weakening that.
Rollback mints a NEW version carrying the old document and publishes it, so
"we went back to v3" is a timestamped fact (v5 = rollback_of_version 3) rather
than a hole where v4 used to be. It is verified by BI-P060 canonical hash, and
documentOf() refuses to serve a stored document whose hash no longer matches
the release — a dashboard that renders an unapproved definition is worse than
one that will not open.
The claim worth attacking is "only one release can be live". The application
check is read-then-write, so it is NOT the guarantee, and the evidence says so
in both directions against Postgres 16.15:
guard_absent partial unique index dropped -> BOTH concurrent publishes
commit, dashboard ends with TWO live versions
guard_present index in place -> loser fails 23505 on
opt_analytics_dashboard_release_one_published, one live
evidence/bi-p061/dashboard-release-race.json, reproducible with
scripts/bi-p061-release-race.mjs.
Two findings while writing the tests, both fixed here rather than papered over:
the one-draft-per-dashboard index makes the two-draft race impossible to even
construct (so the race is run as two create-and-publish transactions, which is
what a "save and publish" button does), and an UPDATE matching zero rows
raises nothing — which reads as "refused" to a careless test, so the
immutability test asserts the row exists first.
23 tests: 8 unit + 15 against live Postgres.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
BI-P002 named DashboardDefinition/DashboardVersion and sql/008 stored them,
but `layout_json` was a string, and a string holds anything — including a
Superset export only Superset can read back. This gives the document a body
and makes the DoD checkable rather than asserted.
planDashboard() is the DoD: a document in, runnable AnalyticsQueryRequest
objects (BI-P016) plus visualization specs out. The test greps the whole
serialized plan for vendor traces after first proving the detector CAN find
one in the source document — a clean grep over a broken detector is the usual
way this test passes for the wrong reason.
The counterfactual is a test too: a widget whose only content is a Superset
slice id is REFUSED, because that is exactly the document that cannot be
reconstructed. vendor_ref survives for round-tripping but is excluded from
the canonical hash, so re-exporting to a new Superset instance does not
fabricate a new dashboard version.
No SQL surface: query widgets are parsed by parseAnalyticsQueryRequest, so
`sql`, `where` and `tenant_id` are refused structurally rather than by
review. 18 tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fourteen BI prompts extending the analytics core merged in #34.
Five findings that changed the implementation, each from a test written so it could fail:
1. ClickHouse readonly=1 on a user profile forbids EVERY request setting (code 164) — BI-P012 ceilings must live on a SETTINGS PROFILE.
2. CREATE ROW POLICY IF NOT EXISTS is a silent no-op on re-provisioning; the restricted user read 100 of 100 rows, both tenants. Fixed with OR REPLACE + a verification query.
3. DuckDB allowed_directories ALONE is a no-op — read_csv('/etc/passwd') returned the file until enable_external_access=false was set alongside it.
4. pg_constraint.conname is not schema-scoped, so sql/009 silently skipped every guarded CHECK in a second schema.
5. describe.skipIf(flagSetInBeforeAll) skips ALWAYS — ten tests green while skipping against a live server.
BI-P017's 64 bypass attempts were run against a live ClickHouse with the guard REMOVED: 0 leaked, judged by seeded canaries rather than by error messages, with the counterfactual asserted first. BI-P019 cancellation is VERIFIED, not requested (4.2B rows read, gone in 162ms, ClickHouse code 394). BI-P021's budget cancels mid-scan, and its ~150x poll-interval overshoot is documented rather than presented as a hard cap.
Honest about what is not proven: MetricFlow was never executed, the Superset and AI-Analyst DoD surfaces are open, dbt's pin has had no licence review, and DuckDB Arrow is unexercised. providerReadiness() and describeEnforcement() report these in code.
957 tests pass against live ClickHouse 24.8, Postgres 16 and DuckDB 1.5.5.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The live suites regenerate their evidence on every run, so the BI-P021
overshoot figures moved when they were last re-run — and the doc and module
comment still quoted the earlier ones (130,883,409 rows / 130x / 92ms) while
the committed file said 153,318,696 / 153x / 55ms. Prose that disagrees with
the evidence it cites is worse than prose with no number in it.
Both now match the committed file, and both say what the earlier version
implied but did not state: the multiple is MACHINE-DEPENDENT and moves between
runs on the same box. That is the actual finding. The bound is "roughly one
poll interval of engine throughput", not a number, and quoting any specific
multiple as if it were a property of the system is the mistake this correction
exists to prevent.
The unwatched-baseline figure is corrected the same way: 4,227,318,261 rows
read before cancellation in the current BI-P019 record, not the 6.5 billion of
the earlier run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
admin
