Files
0pt/src/analytics/materialization
Claude Opus 5 96ff601248
CI (SIAX Cloud) / security (push) Successful in 49s
Security Scan (SIAX Cloud) / security (push) Successful in 50s
CI (SIAX Cloud) / integration (push) Successful in 55s
CI (SIAX Cloud) / quality (push) Successful in 1m20s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m21s
CI (SIAX Cloud) / security (pull_request) Successful in 42s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 37s
CI (SIAX Cloud) / integration (pull_request) Successful in 1m10s
feat(analytics): BI-P105 — a pre-aggregate serves only while it is provably undrifted
A slow query is a complaint; a drifted pre-aggregate answers instantly, with
confidence, and is wrong. So the router is fail-closed on eight named reasons,
and the one people leave out — drift_never_verified — is the one that matters
on launch day: a brand-new aggregate is exactly as unproven as a drifted one.

Proved against ClickHouse 26.3.10.60, 2M rows, a real materialized view
(evidence/bi-p105/materialization-drift.json):

  after backfill      match, 48 buckets, 801 571 002 both sides
  GDPR erasure        ALTER TABLE ... DELETE on the source. An MV is an INSERT
                      trigger and never sees a mutation, so the aggregate kept
                      the deleted revenue: it would have reported 801 571 002
                      against a true 761 580 554 — 39 990 448 overstated,
                      5.251%, with NOTHING stale. Staleness at the moment of
                      the decision: 1 second.
  guarded router      base_table (state_not_serving)
  counterfactual      a staleness-only router SERVES it — measured, not assumed
  after rebuild       match, and routing returns to the materialization

Drift detection treats a MISSING bucket as drift even when every value present
is correct (late-arriving rows are the sneakiest failure), and an empty
comparison as insufficient_data rather than match — the same false green
BI-P025 found in a quality check over an empty table.

The measured speed-up is reported as a distribution, not a figure: p50 3.4x,
but the run-to-run range spans 0.6x-7.5x on one idle host, so quoting "3.4x"
as a property of the system would be misleading (BI-P104 doctrine).

19 unit tests + 4 against live ClickHouse.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 03:56:32 +02:00
..