feat(analytics): BI-P105 — a pre-aggregate serves only while it is provably undrifted
CI (SIAX Cloud) / security (push) Successful in 49s
Security Scan (SIAX Cloud) / security (push) Successful in 50s
CI (SIAX Cloud) / integration (push) Successful in 55s
CI (SIAX Cloud) / quality (push) Successful in 1m20s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m21s
CI (SIAX Cloud) / security (pull_request) Successful in 42s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 37s
CI (SIAX Cloud) / integration (pull_request) Successful in 1m10s

A slow query is a complaint; a drifted pre-aggregate answers instantly, with
confidence, and is wrong. So the router is fail-closed on eight named reasons,
and the one people leave out — drift_never_verified — is the one that matters
on launch day: a brand-new aggregate is exactly as unproven as a drifted one.

Proved against ClickHouse 26.3.10.60, 2M rows, a real materialized view
(evidence/bi-p105/materialization-drift.json):

  after backfill      match, 48 buckets, 801 571 002 both sides
  GDPR erasure        ALTER TABLE ... DELETE on the source. An MV is an INSERT
                      trigger and never sees a mutation, so the aggregate kept
                      the deleted revenue: it would have reported 801 571 002
                      against a true 761 580 554 — 39 990 448 overstated,
                      5.251%, with NOTHING stale. Staleness at the moment of
                      the decision: 1 second.
  guarded router      base_table (state_not_serving)
  counterfactual      a staleness-only router SERVES it — measured, not assumed
  after rebuild       match, and routing returns to the materialization

Drift detection treats a MISSING bucket as drift even when every value present
is correct (late-arriving rows are the sneakiest failure), and an empty
comparison as insufficient_data rather than match — the same false green
BI-P025 found in a quality check over an empty table.

The measured speed-up is reported as a distribution, not a figure: p50 3.4x,
but the run-to-run range spans 0.6x-7.5x on one idle host, so quoting "3.4x"
as a property of the system would be misleading (BI-P104 doctrine).

19 unit tests + 4 against live ClickHouse.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
2026-09-07 03:56:32 +02:00
parent 1a12e8a882
commit 96ff601248
5 changed files with 1229 additions and 0 deletions
@@ -0,0 +1,170 @@
/**
* src/analytics/materialization/drift-detector.ts
* ================================================
* BI-P105 — is the pre-aggregate still telling the truth?
*
* The comparison is bucket-by-bucket between the aggregate and a recomputation
* from the source over the same window. Three things make it a real detector
* rather than a ceremony:
*
* 1. A MISSING BUCKET IS DRIFT. The nastiest pre-aggregate failure is not a
* wrong value, it is an absent row: late-arriving data, a materialized view
* that was created after the backfill, a refresh that failed silently. Every
* value it DOES hold is correct, so a value-only comparison passes while the
* total is wrong. Set difference is checked first, before any arithmetic.
*
* 2. ZERO BUCKETS IS `insufficient_data`, NEVER `match`. A comparison over an
* empty window finds no discrepancies — the same false green BI-P025 found
* in a quality check over an empty table. The router treats
* `insufficient_data` as "do not serve", not as "fine".
*
* 3. RELATIVE AND ABSOLUTE TOLERANCE BOTH HAVE TO HOLD. A 0.1% tolerance on a
* revenue bucket of 40M is 40 000 kr of invisible error; an absolute-only
* tolerance flags every rounding difference on a large bucket. Requiring
* both means a discrepancy has to be small in kronor AND small in
* proportion.
*/
import type { DriftTolerance, DriftVerdict, MaterializationStatus } from './materialization';
/** One aggregated row, keyed by its dimension tuple. */
export interface AggregateBucket {
/** Stable key for the dimension tuple, e.g. "2026-09-01|SE". */
key: string;
/** metric name -> value. */
values: Record<string, number>;
}
export interface DriftDiscrepancy {
key: string;
metric: string;
source_value: number;
aggregate_value: number;
absolute_delta: number;
/** |delta| / |source|, or Infinity when the source value is 0 and the aggregate is not. */
relative_delta: number;
}
export interface DriftReport {
verdict: DriftVerdict;
checked_at: string;
buckets_compared: number;
/** In the source recomputation, absent from the aggregate. Always drift. */
missing_in_aggregate: string[];
/** In the aggregate, absent from the source. Always drift (deleted rows). */
extra_in_aggregate: string[];
discrepancies: DriftDiscrepancy[];
worst?: DriftDiscrepancy;
tolerance: DriftTolerance;
/**
* What a caller would have been told if it had trusted the aggregate: the
* total the aggregate reports vs the total the source computes, per metric.
* This is the number that makes a drift report actionable — "0.4% off" means
* nothing until it is "1 284 500 kr reported, 1 271 900 kr true".
*/
totals: Record<string, { source: number; aggregate: number; delta: number }>;
}
const MAX_REPORTED_DISCREPANCIES = 50;
function relative(sourceValue: number, delta: number): number {
if (sourceValue === 0) return delta === 0 ? 0 : Number.POSITIVE_INFINITY;
return Math.abs(delta / sourceValue);
}
/**
* Compare a recomputation from source against the pre-aggregate.
*
* `fromSource` is the truth. Both sides must cover the same window; the caller
* is responsible for that, and `missing_in_aggregate` will catch it loudly if
* they do not.
*/
export function detectDrift(args: {
fromSource: AggregateBucket[];
fromAggregate: AggregateBucket[];
tolerance: DriftTolerance;
metrics: string[];
now?: Date;
}): DriftReport {
const { fromSource, fromAggregate, tolerance, metrics } = args;
const checked_at = (args.now ?? new Date()).toISOString();
const sourceByKey = new Map(fromSource.map((b) => [b.key, b]));
const aggByKey = new Map(fromAggregate.map((b) => [b.key, b]));
const missing_in_aggregate = [...sourceByKey.keys()].filter((k) => !aggByKey.has(k)).sort();
const extra_in_aggregate = [...aggByKey.keys()].filter((k) => !sourceByKey.has(k)).sort();
const discrepancies: DriftDiscrepancy[] = [];
let buckets_compared = 0;
const totals: DriftReport['totals'] = {};
for (const m of metrics) totals[m] = { source: 0, aggregate: 0, delta: 0 };
for (const [key, sourceBucket] of sourceByKey) {
for (const m of metrics) totals[m].source += sourceBucket.values[m] ?? 0;
const aggBucket = aggByKey.get(key);
if (!aggBucket) continue;
buckets_compared += 1;
for (const m of metrics) {
const sv = sourceBucket.values[m] ?? 0;
const av = aggBucket.values[m] ?? 0;
const delta = av - sv;
const rel = relative(sv, delta);
if (Math.abs(delta) > tolerance.absolute || rel > tolerance.relative) {
discrepancies.push({
key,
metric: m,
source_value: sv,
aggregate_value: av,
absolute_delta: delta,
relative_delta: rel,
});
}
}
}
for (const bucket of aggByKey.values()) {
for (const m of metrics) totals[m].aggregate += bucket.values[m] ?? 0;
}
for (const m of metrics) totals[m].delta = totals[m].aggregate - totals[m].source;
discrepancies.sort((a, b) => Math.abs(b.absolute_delta) - Math.abs(a.absolute_delta));
// A comparison that compared nothing has not proved anything. This is the
// BI-P025 false green, restated for pre-aggregates.
const verdict: DriftVerdict =
buckets_compared === 0 && missing_in_aggregate.length === 0 && extra_in_aggregate.length === 0
? 'insufficient_data'
: discrepancies.length > 0 || missing_in_aggregate.length > 0 || extra_in_aggregate.length > 0
? 'drifted'
: 'match';
const report: DriftReport = {
verdict,
checked_at,
buckets_compared,
missing_in_aggregate,
extra_in_aggregate,
discrepancies: discrepancies.slice(0, MAX_REPORTED_DISCREPANCIES),
tolerance,
totals,
};
if (discrepancies.length > 0) report.worst = discrepancies[0];
return report;
}
/**
* Fold a drift report into the runtime status. A drifted aggregate is taken
* OUT of service here rather than merely annotated: leaving it `serving` with
* a bad verdict relies on every future reader remembering to check the
* verdict, and one that forgets serves the wrong number.
*
* `insufficient_data` also stops service. It means the check could not prove
* anything, and an unproven aggregate is not a current one.
*/
export function applyDriftReport(status: MaterializationStatus, report: DriftReport): MaterializationStatus {
return {
...status,
state: report.verdict === 'match' ? status.state : 'drifted',
last_drift_check_at: report.checked_at,
last_drift_verdict: report.verdict,
};
}
@@ -0,0 +1,309 @@
/**
* src/analytics/materialization/materialization.ts
* =================================================
* BI-P105 — MATERIALIZATION / PRE-AGGREGATION.
*
* A pre-aggregate is the one optimisation that fails SILENTLY. A slow query is
* a complaint; a pre-aggregate that has drifted from its source answers
* instantly, with confidence, and is wrong — no error, no timeout, nothing in
* a log. So this module is built around one rule:
*
* A MATERIALIZATION SERVES ONLY WHILE IT IS PROVABLY CURRENT AND PROVABLY
* UNDRIFTED. Every unknown routes to the source table.
*
* Six refusal reasons, each of which is a real way the aggregate becomes a
* confident liar:
*
* coverage_miss it does not carry the metric/dimension/grain
* asked for (a filter on a dimension the
* aggregate collapsed cannot be applied after
* the fact, and applying it anyway is how a
* pre-aggregate returns another slice's number)
* watermark_unknown nobody knows how current it is. BI-P026 already
* refuses to render UNKNOWN age as current; the
* same doctrine applies here.
* staleness_exceeded it is older than its own freshness contract
* window_beyond_watermark the question reaches past the newest data the
* aggregate contains — it would silently answer
* with a shorter period
* drift_never_verified it has never been compared to its source
* drift_verification_stale the last comparison is older than the interval
* the spec requires
* drift_detected the last comparison FAILED
*
* `drift_never_verified` is the one people leave out, and it is the one that
* matters most on day one: a brand-new materialization is exactly as unproven
* as a drifted one, and a router that defaults new aggregates to "serving"
* ships the bug on the day the feature launches.
*
* The DoD also asks for measured improvement and documented freshness
* semantics: scripts/bi-p105-materialization-drift.mjs measures both against a
* live ClickHouse and writes evidence/bi-p105/materialization-drift.json,
* including what happens WITHOUT this router (the drifted aggregate served).
*/
import { SiaxError } from '../../platform/errors';
import type { AnalyticsQueryRequest, TimeGrain } from '../contract/query-request';
export const MATERIALIZATION_STRATEGIES = [
'clickhouse_materialized_view',
'clickhouse_projection',
'dbt_incremental',
'cube_preaggregation',
] as const;
export type MaterializationStrategy = (typeof MATERIALIZATION_STRATEGIES)[number];
export const MATERIALIZATION_STATES = ['building', 'serving', 'drifted', 'disabled'] as const;
export type MaterializationState = (typeof MATERIALIZATION_STATES)[number];
/**
* Grains ordered coarsest-last. An aggregate at `day` can answer `day` and
* anything coarser by re-aggregating; it can NEVER answer `hour`, because the
* detail was thrown away when the rows were rolled up.
*/
const GRAIN_ORDER: readonly TimeGrain[] = ['hour', 'day', 'week', 'month', 'quarter', 'year'];
/**
* `week` does not divide `month` and `month` does not divide `quarter` by row
* count, but both roll up cleanly from `day`. Grains that CANNOT be derived by
* re-aggregating a coarser one are listed here so the router refuses rather
* than quietly returning a mis-bucketed series.
*/
function canRollUp(from: TimeGrain, to: TimeGrain): boolean {
if (from === to) return true;
const fi = GRAIN_ORDER.indexOf(from);
const ti = GRAIN_ORDER.indexOf(to);
if (fi < 0 || ti < 0 || ti < fi) return false;
// Weekly buckets do not compose into calendar months/quarters/years.
if (from === 'week' && to !== 'week') return false;
return true;
}
export interface DriftTolerance {
/** Absolute delta allowed per metric per bucket. */
absolute: number;
/** Relative delta allowed (0.001 = 0.1%). Both must hold. */
relative: number;
}
/** What a materialization IS. Declarative; the engine adapters read it. */
export interface MaterializationSpec {
materialization_id: string;
tenant_id: string;
project_id: string;
/** The dataset it pre-aggregates (BI-P003). */
dataset: string;
strategy: MaterializationStrategy;
/** Physical target (ClickHouse table, dbt model, Cube pre-agg name). */
target_ref: string;
metrics: string[];
dimensions: string[];
time_dimension?: string;
grain?: TimeGrain;
/** The freshness contract. Older than this and it stops serving. */
max_staleness_seconds: number;
/** How often the aggregate must be re-proved against its source. */
drift_check_interval_seconds: number;
tolerance: DriftTolerance;
}
export const DRIFT_VERDICTS = ['match', 'drifted', 'insufficient_data'] as const;
export type DriftVerdict = (typeof DRIFT_VERDICTS)[number];
/** Runtime state. Separate from the spec: the spec is config, this is fact. */
export interface MaterializationStatus {
materialization_id: string;
state: MaterializationState;
/**
* The newest source timestamp the aggregate represents. UNDEFINED means
* UNKNOWN, and unknown is never treated as current.
*/
data_watermark?: string;
last_refresh_at?: string;
last_drift_check_at?: string;
last_drift_verdict?: DriftVerdict;
}
export const ROUTING_REFUSALS = [
'coverage_miss',
'watermark_unknown',
'staleness_exceeded',
'window_beyond_watermark',
'drift_never_verified',
'drift_verification_stale',
'drift_detected',
'state_not_serving',
] as const;
export type RoutingRefusal = (typeof ROUTING_REFUSALS)[number];
export type RoutingDecision =
| {
source: 'materialization';
materialization_id: string;
target_ref: string;
/** Reported so a caller can show it; never inferred, always measured. */
staleness_seconds: number;
}
| { source: 'base_table'; refusal: RoutingRefusal; detail: string };
function refuse(refusal: RoutingRefusal, detail: string): RoutingDecision {
return { source: 'base_table', refusal, detail };
}
/**
* Staleness of an aggregate, in seconds, or `undefined` when it cannot be
* computed. Deliberately NOT defaulted to 0: "we could not tell" and "it is
* current" are different facts, and conflating them is the bug.
*/
export function materializationStaleness(status: MaterializationStatus, now: Date): number | undefined {
if (!status.data_watermark) return undefined;
const t = Date.parse(status.data_watermark);
if (Number.isNaN(t)) return undefined;
return Math.max(0, Math.floor((now.getTime() - t) / 1000));
}
/**
* Decide whether a request may be answered from the pre-aggregate.
*
* Fail-closed by construction: the function returns `materialization` only
* after every check has passed, and every unknown is a refusal with a named
* reason. Callers get the reason so it can be logged and counted (BI-P103
* reads the counts).
*/
export function selectQuerySource(args: {
request: AnalyticsQueryRequest;
spec: MaterializationSpec;
status: MaterializationStatus;
now: Date;
}): RoutingDecision {
const { request, spec, status, now } = args;
if (request.dataset !== spec.dataset) {
return refuse('coverage_miss', `materialization covers "${spec.dataset}", not "${request.dataset}"`);
}
if (status.state !== 'serving') {
return refuse('state_not_serving', `materialization state is "${status.state}"`);
}
// ── coverage ────────────────────────────────────────────────────────────
const metrics = new Set(spec.metrics);
for (const m of request.metrics) {
if (!metrics.has(m)) return refuse('coverage_miss', `metric "${m}" is not in the pre-aggregate`);
}
const dimensions = new Set(spec.dimensions);
for (const d of request.dimensions) {
if (!dimensions.has(d)) return refuse('coverage_miss', `dimension "${d}" is not in the pre-aggregate`);
}
for (const f of request.filters) {
if (!dimensions.has(f.dimension)) {
// The aggregate summed across this dimension. Filtering afterwards would
// return a number computed over rows the filter should have excluded.
return refuse(
'coverage_miss',
`filter on "${f.dimension}" cannot be applied: the pre-aggregate has already summed across it`,
);
}
}
const requestedGrain = request.timeRange?.grain;
if (requestedGrain) {
if (!spec.grain) return refuse('coverage_miss', 'the pre-aggregate has no time grain');
if (!canRollUp(spec.grain, requestedGrain)) {
return refuse(
'coverage_miss',
`a "${spec.grain}" pre-aggregate cannot answer at "${requestedGrain}" grain`,
);
}
}
// ── freshness ───────────────────────────────────────────────────────────
const staleness = materializationStaleness(status, now);
if (staleness === undefined) {
return refuse(
'watermark_unknown',
'the pre-aggregate has no data watermark — unknown age is not current age',
);
}
if (staleness > spec.max_staleness_seconds) {
return refuse(
'staleness_exceeded',
`${staleness}s stale, contract allows ${spec.max_staleness_seconds}s`,
);
}
const to = request.timeRange?.to;
if (to) {
const toMs = Date.parse(to);
const watermarkMs = Date.parse(status.data_watermark!);
if (!Number.isNaN(toMs) && toMs > watermarkMs) {
return refuse(
'window_beyond_watermark',
`the question runs to ${to} but the pre-aggregate only contains data to ${status.data_watermark}`,
);
}
}
// ── drift ───────────────────────────────────────────────────────────────
if (!status.last_drift_check_at || !status.last_drift_verdict) {
return refuse(
'drift_never_verified',
'the pre-aggregate has never been compared to its source; a new aggregate is as unproven as a drifted one',
);
}
if (status.last_drift_verdict !== 'match') {
return refuse('drift_detected', `last drift check returned "${status.last_drift_verdict}"`);
}
const checkedAgo = Math.floor((now.getTime() - Date.parse(status.last_drift_check_at)) / 1000);
if (Number.isNaN(checkedAgo)) {
return refuse('drift_never_verified', 'last_drift_check_at is not a timestamp');
}
if (checkedAgo > spec.drift_check_interval_seconds) {
return refuse(
'drift_verification_stale',
`last verified ${checkedAgo}s ago, spec requires every ${spec.drift_check_interval_seconds}s`,
);
}
return {
source: 'materialization',
materialization_id: spec.materialization_id,
target_ref: spec.target_ref,
staleness_seconds: staleness,
};
}
/**
* The routing decision as a freshness envelope for the result (BI-P026 shape):
* a consumer must be able to see WHICH source answered and how old it was.
* A result that does not say is a result nobody can audit.
*/
export function describeRouting(decision: RoutingDecision): {
served_from: 'materialization' | 'base_table';
materialization_id?: string;
staleness_seconds?: number;
refusal?: RoutingRefusal;
detail?: string;
} {
return decision.source === 'materialization'
? {
served_from: 'materialization',
materialization_id: decision.materialization_id,
staleness_seconds: decision.staleness_seconds,
}
: { served_from: 'base_table', refusal: decision.refusal, detail: decision.detail };
}
export function assertSpec(spec: MaterializationSpec): void {
const invalid = (d: string): never => {
throw new SiaxError('INVALID_REQUEST', '0pt.analytics.materialization', d);
};
if (!spec.materialization_id?.trim()) invalid('materialization_id is required');
if (!spec.tenant_id?.trim() || !spec.project_id?.trim()) invalid('a materialization is tenant-scoped');
if (spec.metrics.length === 0) invalid('a materialization with no metrics pre-aggregates nothing');
if (spec.max_staleness_seconds <= 0) invalid('max_staleness_seconds must be positive');
if (spec.drift_check_interval_seconds <= 0) {
invalid(
'drift_check_interval_seconds must be positive — an aggregate that is never re-verified is one ' +
'source mutation away from serving wrong numbers forever',
);
}
if (spec.tolerance.absolute < 0 || spec.tolerance.relative < 0) invalid('tolerance must be non-negative');
if (spec.grain && !GRAIN_ORDER.includes(spec.grain)) invalid('grain is not a time grain');
}