Files
0pt/evidence/bi-p029/semantic-parity.json
T
Claude Opus 5 468a87c00c
CI (SIAX Cloud) / security (push) Successful in 14s
Security Scan (SIAX Cloud) / security (push) Successful in 15s
CI (SIAX Cloud) / integration (push) Successful in 46s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 20s
CI (SIAX Cloud) / security (pull_request) Successful in 25s
CI (SIAX Cloud) / quality (push) Successful in 54s
CI (SIAX Cloud) / integration (pull_request) Successful in 46s
CI (SIAX Cloud) / quality (pull_request) Successful in 48s
BI-P029: the cited parity test did not exist, and it caught a real bug
`semantic-provider.ts` cited tests/integration/semantic-parity.integration.test.ts
as the proof of the one BI-P029 DoD surface it claimed was closed, and
`providerReadiness().proven` asserted the comparison had been made against live
ClickHouse. The file did not exist. The citation was the evidence.

Writing it exposed a divergence that had been shipping:
`compileRequestToClickHouse` had no notion of a metric's OWN filters
(BI-P028 `MetricSpec.filters`). `ResolvedMetric` did not carry them and the
compiler never asked, so a metric defined as "invoiced revenue EXCLUDING
internal transfers" executed as "invoiced revenue". On the committed fixture
that is 11 249 instead of 1 250 for one country -- a plausible-looking number,
9x too large, under the correct canonical metric id. The registry, the
governance checks and the MetricFlow manifest all carried the filter; only the
thing that actually ran did not.

  * `ResolvedMetric` gains `definition_filters`, already resolved onto the
    physical dim_names key so the compilers stay dumb.
  * The ClickHouse compiler folds them into THAT metric's conditional
    aggregate, never into the shared WHERE -- two metrics in one request may
    exclude different rows, and a shared predicate would apply one metric's
    meaning to the other. `renderFilter`'s uniquifier becomes a string so
    request filter `0` and metric filter `0_0m` cannot collide in the
    parameter map.
  * `compileRequestToSemantic` carries them through, and
    `MetricFlowProvider.compile` emits them per metric.

The test runs both compile paths against one live ClickHouse: 0pt's `sumIf`
compiler, and a renderer driven only by the provider's structured query that
shares no SQL-generation code and emits `sum(if(...))` with inlined literals.
It asserts the two SQL strings differ, so parity is not vacuous. Backing the
fix out reproduces the pre-fix numbers, and the suite records them itself
rather than quoting a console: evidence/bi-p029/semantic-parity.json.

Three of BI-P029's four DoD surfaces stay OPEN and still say so: MetricFlow
itself is not executed (no Python runtime is provisioned here), Superset is not
implemented, and the AI Analyst path has not been compared.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 08:23:43 +02:00

63 lines
1.5 KiB
JSON

{
"prompt": "BI-P029",
"title": "Golden metric parity: 0pt metric compiler vs an independent spec-driven renderer",
"recorded_at": "2026-09-07T06:23:27.839Z",
"clickhouse_version": "26.3.10.60",
"golden_metric": {
"id": "m_revenue",
"name": "net_revenue",
"definition_filters": [
{
"dimension": "channel",
"operator": "ne",
"values": [
"internal"
]
}
],
"semantic_model": {
"model_id": "sm_sales",
"version": 1,
"hash": "c5b9cac1b5fdcfbfb89a1e960b946a9b33b8c29dc14fe9a304c134f7a0fb73a7"
}
},
"expected_from_typescript": {
"SE": 1250,
"NO": 400,
"DK": 125.5
},
"regression_guarded": {
"description": "the same request compiled WITHOUT the metric definition filters — what 0pt returned before this change",
"result": {
"DK": 3458.5,
"NO": 8177,
"SE": 11249
}
},
"path_a": {
"compiler": "0pt compileRequestToClickHouse",
"aggregate": "sumIf",
"result": {
"DK": 125.5,
"NO": 400,
"SE": 1250
}
},
"path_b": {
"compiler": "MetricFlowProvider.compile + independent renderer",
"aggregate": "sum(if(...))",
"result": {
"DK": 125.5,
"NO": 400,
"SE": 1250
}
},
"tolerance": "1e-9 relative",
"verdict": "PASS",
"still_open": [
"MetricFlow itself was not executed (no Python runtime provisioned for this repo)",
"the Superset integration path is not implemented",
"the AI Analyst path (inf0#29) has not been compared against this provider"
]
}