feat(analytics): BI-P104 — a benchmark harness that refuses to quote one number
CI (SIAX Cloud) / security (push) Successful in 30s
CI (SIAX Cloud) / integration (push) Successful in 44s
Security Scan (SIAX Cloud) / security (push) Successful in 36s
CI (SIAX Cloud) / quality (push) Successful in 1m14s
CI (SIAX Cloud) / security (pull_request) Successful in 34s
CI (SIAX Cloud) / integration (pull_request) Successful in 50s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 27s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m7s

Six dataset profiles (small SaaS, medium business, VehIQ market, large event,
wide table, geospatial), one representative question each, run on ClickHouse
26.3.10.60 and DuckDB 1.5.5 with 2 warmups discarded and 9 timed repetitions.
evidence/bi-p104/benchmark.json.

The harness's job is to stop the benchmark from lying:

  - overlapping ranges are INDISTINGUISHABLE whatever the medians say. Three
    of the six real comparisons came out that way, one of them with a 1.6x
    median gap — which is exactly the headline a noisy host manufactures.
  - every sentence carries the range, the sample size and the stability, so it
    cannot be trimmed to "4.4x faster" without visibly deleting the caveats.
  - under 5 samples nothing is comparable at all. A three-run benchmark is an
    anecdote.
  - every engine BI-P104 names appears in the evidence, and the five nothing
    measured carry a reason each. The running Trino belongs to another lane,
    so benchmarking through it would have measured their configuration.

Measured shape, warm cache, one idle host, these query shapes only: DuckDB
faster on small_saas (1.6-8.2x) and vehiq_market (2.4-13.3x), SLOWER on
large_event (0.2-0.9x, count DISTINCT over 5M rows), indistinguishable on
medium_business, wide_table and geospatial.

One finding from the harness's own tests: an engine declared available that
produced no run came out of buildEvidence() with no reason — the dangerous
case, since it reads as "we could have and chose not to say". It now gets a
generated reason. A second from the runs: DuckDB's progress bar writes to
STDOUT and corrupted the first transcript; it is disabled explicitly and the
JSON is read from a sentinel.

10 unit tests + 4 against real engines.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
2026-09-07 04:06:43 +02:00
parent ed6e62809d
commit f33cc8d32f
4 changed files with 1517 additions and 0 deletions
+341
View File
@@ -0,0 +1,341 @@
/**
* src/analytics/bench/benchmark-harness.ts
* =========================================
* BI-P104 — BI PERFORMANCE BENCHMARK HARNESS. DoD: "results committed as
* machine-readable evidence."
*
* The harness exists because of a specific way benchmarks lie. Yesterday's
* lane measured the same budget-enforcement overshoot at ~150x on one run and
* ~130x on the next, on the same machine, and correctly reported the SHAPE
* rather than a number. That is the doctrine encoded here:
*
* 1. A SINGLE FIGURE IS NOT A SYSTEM PROPERTY. `summarizeSamples` returns a
* distribution — n, min, p50, p95, p99, max, mean, stdev, CV, IQR — and
* there is no function anywhere in this module that returns one number
* for a measurement. `describeComparison` renders a RANGE and the sample
* size into every sentence it produces.
*
* 2. OVERLAPPING DISTRIBUTIONS ARE INDISTINGUISHABLE, whatever the medians
* say. If the candidate's p95 is not below the baseline's minimum, the
* verdict is `indistinguishable` even when p50 differs by 2x — because a
* 2x median gap inside a 10x spread is noise with a story attached.
*
* 3. TOO FEW SAMPLES IS A REFUSAL. Under MIN_SAMPLES the summary carries
* `stability: 'insufficient_samples'` and comparisons refuse to judge.
* A three-run benchmark is an anecdote.
*
* 4. AN ENGINE THAT WAS NOT RUN IS RECORDED AS NOT RUN. `EngineAvailability`
* carries a reason. The evidence file lists every engine BI-P104 names,
* including the ones nothing measured — an omitted engine reads as "we
* chose not to report it".
*/
/** The six dataset profiles BI-P104 names, with what makes each one distinct. */
export const BENCH_DATASETS = [
{
id: 'small_saas',
label: 'small SaaS',
approx_rows: 100_000,
shape: 'narrow fact table, one tenant, high selectivity',
},
{
id: 'medium_business',
label: 'medium business',
approx_rows: 2_000_000,
shape: 'multi-tenant fact table with a date and a few dimensions',
},
{
id: 'vehiq_market',
label: 'VehIQ market',
approx_rows: 1_000_000,
shape: 'vehicle listings: make/model/year/price/mileage, skewed cardinalities',
},
{
id: 'large_event',
label: 'large event',
approx_rows: 5_000_000,
shape: 'append-only event stream, timestamp-ordered, sparse dimensions',
},
{
id: 'wide_table',
label: 'wide table',
approx_rows: 200_000,
shape: 'many columns, few read per query — the columnar-store best case',
},
{
id: 'geospatial',
label: 'geospatial',
approx_rows: 500_000,
shape: 'lat/lon points with bounding-box and distance predicates',
},
] as const;
export type BenchDatasetId = (typeof BENCH_DATASETS)[number]['id'];
/** Every engine BI-P104 lists. All of them appear in the evidence, run or not. */
export const BENCH_ENGINES = [
'clickhouse',
'duckdb',
'trino',
'semantic_query',
'superset',
'native_charts',
'perspective',
] as const;
export type BenchEngine = (typeof BENCH_ENGINES)[number];
export interface EngineAvailability {
engine: BenchEngine;
available: boolean;
/** Required when unavailable. "Unknown" is not a reason. */
reason?: string;
version?: string;
}
export const STABILITY_VERDICTS = ['stable', 'noisy', 'unstable', 'insufficient_samples'] as const;
export type StabilityVerdict = (typeof STABILITY_VERDICTS)[number];
/** Below this a run is an anecdote, not a measurement. */
export const MIN_SAMPLES = 5;
/** Coefficient of variation thresholds. Wide on purpose: real hosts are noisy. */
const CV_STABLE = 0.15;
const CV_NOISY = 0.5;
export interface Distribution {
n: number;
min: number;
p50: number;
p95: number;
p99: number;
max: number;
mean: number;
stdev: number;
/** stdev / mean. The number that decides whether a comparison is meaningful. */
cv: number;
iqr: number;
/** max / min. Reported because it is what a reader feels. */
spread_ratio: number;
stability: StabilityVerdict;
}
/** Nearest-rank; no interpolation, so every reported value really happened. */
function pct(sorted: number[], q: number): number {
const rank = Math.ceil(q * sorted.length);
return sorted[Math.min(sorted.length - 1, Math.max(0, rank - 1))];
}
export function summarizeSamples(samples: readonly number[]): Distribution {
if (samples.length === 0) {
return {
n: 0,
min: 0,
p50: 0,
p95: 0,
p99: 0,
max: 0,
mean: 0,
stdev: 0,
cv: 0,
iqr: 0,
spread_ratio: 0,
stability: 'insufficient_samples',
};
}
const s = [...samples].sort((a, b) => a - b);
const mean = s.reduce((a, b) => a + b, 0) / s.length;
const variance = s.reduce((acc, v) => acc + (v - mean) ** 2, 0) / s.length;
const stdev = Math.sqrt(variance);
const cv = mean === 0 ? 0 : stdev / mean;
const stability: StabilityVerdict =
s.length < MIN_SAMPLES
? 'insufficient_samples'
: cv <= CV_STABLE
? 'stable'
: cv <= CV_NOISY
? 'noisy'
: 'unstable';
return {
n: s.length,
min: s[0],
p50: pct(s, 0.5),
p95: pct(s, 0.95),
p99: pct(s, 0.99),
max: s[s.length - 1],
mean,
stdev,
cv,
iqr: pct(s, 0.75) - pct(s, 0.25),
spread_ratio: s[0] === 0 ? Number.POSITIVE_INFINITY : s[s.length - 1] / s[0],
stability,
};
}
export const COMPARISON_VERDICTS = ['faster', 'slower', 'indistinguishable', 'not_comparable'] as const;
export type ComparisonVerdict = (typeof COMPARISON_VERDICTS)[number];
export interface Comparison {
verdict: ComparisonVerdict;
/** Ratio of medians. Present for information; never the claim on its own. */
p50_ratio: number;
/** Worst case for the candidate: baseline.min / candidate.max. */
conservative_ratio: number;
/** Best case for the candidate: baseline.max / candidate.min. */
optimistic_ratio: number;
/** True when the observed ranges overlap at all. */
ranges_overlap: boolean;
baseline: Distribution;
candidate: Distribution;
}
/**
* Compare two measured distributions.
*
* `faster` requires clean separation — the candidate's p95 below the
* baseline's minimum. Anything less is `indistinguishable`, however flattering
* the medians look, because that is exactly the claim a noisy host will
* manufacture for you.
*/
export function compareDistributions(baseline: Distribution, candidate: Distribution): Comparison {
const ratio = (a: number, b: number) => (b === 0 ? Number.POSITIVE_INFINITY : a / b);
const base: Omit<Comparison, 'verdict'> = {
p50_ratio: ratio(baseline.p50, candidate.p50),
conservative_ratio: ratio(baseline.min, candidate.max),
optimistic_ratio: ratio(baseline.max, candidate.min),
ranges_overlap: !(candidate.max < baseline.min || baseline.max < candidate.min),
baseline,
candidate,
};
if (baseline.stability === 'insufficient_samples' || candidate.stability === 'insufficient_samples') {
return { ...base, verdict: 'not_comparable' };
}
if (candidate.p95 < baseline.min) return { ...base, verdict: 'faster' };
if (baseline.p95 < candidate.min) return { ...base, verdict: 'slower' };
return { ...base, verdict: 'indistinguishable' };
}
/**
* The sentence a human is allowed to quote. It ALWAYS carries the range, the
* sample size and the stability verdict, so it cannot be trimmed down to "3.4x
* faster" without visibly removing the caveats.
*/
export function describeComparison(c: Comparison, label = 'candidate'): string {
const r = (v: number) => (Number.isFinite(v) ? v.toFixed(1) : '∞');
if (c.verdict === 'not_comparable') {
return (
`${label}: NOT COMPARABLE — ${c.baseline.n} baseline and ${c.candidate.n} candidate samples, ` +
`fewer than the ${MIN_SAMPLES} required. A run this short is an anecdote.`
);
}
if (c.verdict === 'indistinguishable') {
return (
`${label}: INDISTINGUISHABLE at this sample size. Medians differ by ${r(c.p50_ratio)}x but the ` +
`observed ranges overlap (baseline ${c.baseline.min.toFixed(1)}-${c.baseline.max.toFixed(1)}ms, ` +
`candidate ${c.candidate.min.toFixed(1)}-${c.candidate.max.toFixed(1)}ms; n=${c.baseline.n}/${c.candidate.n}, ` +
`CV ${c.baseline.cv.toFixed(2)}/${c.candidate.cv.toFixed(2)}).`
);
}
return (
`${label}: ${c.verdict.toUpperCase()} — ${r(c.conservative_ratio)}x to ${r(c.optimistic_ratio)}x ` +
`(median ${r(c.p50_ratio)}x) over n=${c.baseline.n}/${c.candidate.n}, ` +
`stability ${c.baseline.stability}/${c.candidate.stability}. Warm cache, one host, this query shape.`
);
}
export interface BenchmarkRun {
engine: BenchEngine;
dataset: BenchDatasetId;
query_label: string;
/** Discarded from the statistics; recorded so the discard is visible. */
warmup_ms: number[];
samples_ms: number[];
distribution: Distribution;
rows_returned?: number;
}
/**
* Run one measurement. Warmups are executed and RECORDED but excluded from the
* distribution: a first run that pays for cold marks and page cache is a
* different measurement, and silently folding it in is how a p95 becomes a
* story about disk.
*/
export async function measure(args: {
engine: BenchEngine;
dataset: BenchDatasetId;
query_label: string;
run: () => Promise<{ rows?: number }>;
warmups?: number;
repetitions?: number;
now?: () => number;
}): Promise<BenchmarkRun> {
const now = args.now ?? (() => performance.now());
const warmups = args.warmups ?? 2;
const repetitions = args.repetitions ?? 9;
const warmup_ms: number[] = [];
const samples_ms: number[] = [];
let rows: number | undefined;
for (let i = 0; i < warmups; i += 1) {
const t0 = now();
await args.run();
warmup_ms.push(now() - t0);
}
for (let i = 0; i < repetitions; i += 1) {
const t0 = now();
const out = await args.run();
samples_ms.push(now() - t0);
rows = out.rows ?? rows;
}
return {
engine: args.engine,
dataset: args.dataset,
query_label: args.query_label,
warmup_ms,
samples_ms,
distribution: summarizeSamples(samples_ms),
...(rows !== undefined ? { rows_returned: rows } : {}),
};
}
export interface BenchmarkEvidence {
recorded_at: string;
host_note: string;
datasets: typeof BENCH_DATASETS;
engines: EngineAvailability[];
runs: BenchmarkRun[];
comparisons: Array<{ dataset: BenchDatasetId; query_label: string; description: string } & Comparison>;
/** Engines BI-P104 names that produced no measurement, each with a reason. */
not_exercised: EngineAvailability[];
}
/**
* Assemble the evidence document. Every engine BI-P104 names appears, and the
* ones with no runs are listed under `not_exercised` with their reason — an
* engine quietly missing from a benchmark table reads as a choice.
*/
export function buildEvidence(args: {
host_note: string;
engines: EngineAvailability[];
runs: BenchmarkRun[];
comparisons?: BenchmarkEvidence['comparisons'];
now?: Date;
}): BenchmarkEvidence {
const measured = new Set(args.runs.map((r) => r.engine));
const declared = new Map(args.engines.map((e) => [e.engine, e]));
return {
recorded_at: (args.now ?? new Date()).toISOString(),
host_note: args.host_note,
datasets: BENCH_DATASETS,
engines: BENCH_ENGINES.map(
(engine) => declared.get(engine) ?? { engine, available: false, reason: 'not declared by this run' },
),
runs: args.runs,
comparisons: args.comparisons ?? [],
not_exercised: BENCH_ENGINES.filter((e) => !measured.has(e)).map((engine) => {
const d = declared.get(engine);
if (!d) return { engine, available: false, reason: 'not present in this environment' };
// An engine declared AVAILABLE that produced no measurement is the
// dangerous case: it reads as "we could have, and chose not to say".
// It gets a reason too, generated here rather than left blank.
return d.reason ? d : { ...d, reason: 'declared available but produced no measurement in this run' };
}),
};
}