Files
0pt/src/analytics/bench/benchmark-harness.ts
T
Claude Opus 5 f33cc8d32f
CI (SIAX Cloud) / security (push) Successful in 30s
CI (SIAX Cloud) / integration (push) Successful in 44s
Security Scan (SIAX Cloud) / security (push) Successful in 36s
CI (SIAX Cloud) / quality (push) Successful in 1m14s
CI (SIAX Cloud) / security (pull_request) Successful in 34s
CI (SIAX Cloud) / integration (pull_request) Successful in 50s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 27s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m7s
feat(analytics): BI-P104 — a benchmark harness that refuses to quote one number
Six dataset profiles (small SaaS, medium business, VehIQ market, large event,
wide table, geospatial), one representative question each, run on ClickHouse
26.3.10.60 and DuckDB 1.5.5 with 2 warmups discarded and 9 timed repetitions.
evidence/bi-p104/benchmark.json.

The harness's job is to stop the benchmark from lying:

  - overlapping ranges are INDISTINGUISHABLE whatever the medians say. Three
    of the six real comparisons came out that way, one of them with a 1.6x
    median gap — which is exactly the headline a noisy host manufactures.
  - every sentence carries the range, the sample size and the stability, so it
    cannot be trimmed to "4.4x faster" without visibly deleting the caveats.
  - under 5 samples nothing is comparable at all. A three-run benchmark is an
    anecdote.
  - every engine BI-P104 names appears in the evidence, and the five nothing
    measured carry a reason each. The running Trino belongs to another lane,
    so benchmarking through it would have measured their configuration.

Measured shape, warm cache, one idle host, these query shapes only: DuckDB
faster on small_saas (1.6-8.2x) and vehiq_market (2.4-13.3x), SLOWER on
large_event (0.2-0.9x, count DISTINCT over 5M rows), indistinguishable on
medium_business, wide_table and geospatial.

One finding from the harness's own tests: an engine declared available that
produced no run came out of buildEvidence() with no reason — the dangerous
case, since it reads as "we could have and chose not to say". It now gets a
generated reason. A second from the runs: DuckDB's progress bar writes to
STDOUT and corrupted the first transcript; it is disabled explicitly and the
JSON is read from a sentinel.

10 unit tests + 4 against real engines.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 04:06:43 +02:00

342 lines
12 KiB
TypeScript

/**
* src/analytics/bench/benchmark-harness.ts
* =========================================
* BI-P104 — BI PERFORMANCE BENCHMARK HARNESS. DoD: "results committed as
* machine-readable evidence."
*
* The harness exists because of a specific way benchmarks lie. Yesterday's
* lane measured the same budget-enforcement overshoot at ~150x on one run and
* ~130x on the next, on the same machine, and correctly reported the SHAPE
* rather than a number. That is the doctrine encoded here:
*
* 1. A SINGLE FIGURE IS NOT A SYSTEM PROPERTY. `summarizeSamples` returns a
* distribution — n, min, p50, p95, p99, max, mean, stdev, CV, IQR — and
* there is no function anywhere in this module that returns one number
* for a measurement. `describeComparison` renders a RANGE and the sample
* size into every sentence it produces.
*
* 2. OVERLAPPING DISTRIBUTIONS ARE INDISTINGUISHABLE, whatever the medians
* say. If the candidate's p95 is not below the baseline's minimum, the
* verdict is `indistinguishable` even when p50 differs by 2x — because a
* 2x median gap inside a 10x spread is noise with a story attached.
*
* 3. TOO FEW SAMPLES IS A REFUSAL. Under MIN_SAMPLES the summary carries
* `stability: 'insufficient_samples'` and comparisons refuse to judge.
* A three-run benchmark is an anecdote.
*
* 4. AN ENGINE THAT WAS NOT RUN IS RECORDED AS NOT RUN. `EngineAvailability`
* carries a reason. The evidence file lists every engine BI-P104 names,
* including the ones nothing measured — an omitted engine reads as "we
* chose not to report it".
*/
/** The six dataset profiles BI-P104 names, with what makes each one distinct. */
export const BENCH_DATASETS = [
{
id: 'small_saas',
label: 'small SaaS',
approx_rows: 100_000,
shape: 'narrow fact table, one tenant, high selectivity',
},
{
id: 'medium_business',
label: 'medium business',
approx_rows: 2_000_000,
shape: 'multi-tenant fact table with a date and a few dimensions',
},
{
id: 'vehiq_market',
label: 'VehIQ market',
approx_rows: 1_000_000,
shape: 'vehicle listings: make/model/year/price/mileage, skewed cardinalities',
},
{
id: 'large_event',
label: 'large event',
approx_rows: 5_000_000,
shape: 'append-only event stream, timestamp-ordered, sparse dimensions',
},
{
id: 'wide_table',
label: 'wide table',
approx_rows: 200_000,
shape: 'many columns, few read per query — the columnar-store best case',
},
{
id: 'geospatial',
label: 'geospatial',
approx_rows: 500_000,
shape: 'lat/lon points with bounding-box and distance predicates',
},
] as const;
export type BenchDatasetId = (typeof BENCH_DATASETS)[number]['id'];
/** Every engine BI-P104 lists. All of them appear in the evidence, run or not. */
export const BENCH_ENGINES = [
'clickhouse',
'duckdb',
'trino',
'semantic_query',
'superset',
'native_charts',
'perspective',
] as const;
export type BenchEngine = (typeof BENCH_ENGINES)[number];
export interface EngineAvailability {
engine: BenchEngine;
available: boolean;
/** Required when unavailable. "Unknown" is not a reason. */
reason?: string;
version?: string;
}
export const STABILITY_VERDICTS = ['stable', 'noisy', 'unstable', 'insufficient_samples'] as const;
export type StabilityVerdict = (typeof STABILITY_VERDICTS)[number];
/** Below this a run is an anecdote, not a measurement. */
export const MIN_SAMPLES = 5;
/** Coefficient of variation thresholds. Wide on purpose: real hosts are noisy. */
const CV_STABLE = 0.15;
const CV_NOISY = 0.5;
export interface Distribution {
n: number;
min: number;
p50: number;
p95: number;
p99: number;
max: number;
mean: number;
stdev: number;
/** stdev / mean. The number that decides whether a comparison is meaningful. */
cv: number;
iqr: number;
/** max / min. Reported because it is what a reader feels. */
spread_ratio: number;
stability: StabilityVerdict;
}
/** Nearest-rank; no interpolation, so every reported value really happened. */
function pct(sorted: number[], q: number): number {
const rank = Math.ceil(q * sorted.length);
return sorted[Math.min(sorted.length - 1, Math.max(0, rank - 1))];
}
export function summarizeSamples(samples: readonly number[]): Distribution {
if (samples.length === 0) {
return {
n: 0,
min: 0,
p50: 0,
p95: 0,
p99: 0,
max: 0,
mean: 0,
stdev: 0,
cv: 0,
iqr: 0,
spread_ratio: 0,
stability: 'insufficient_samples',
};
}
const s = [...samples].sort((a, b) => a - b);
const mean = s.reduce((a, b) => a + b, 0) / s.length;
const variance = s.reduce((acc, v) => acc + (v - mean) ** 2, 0) / s.length;
const stdev = Math.sqrt(variance);
const cv = mean === 0 ? 0 : stdev / mean;
const stability: StabilityVerdict =
s.length < MIN_SAMPLES
? 'insufficient_samples'
: cv <= CV_STABLE
? 'stable'
: cv <= CV_NOISY
? 'noisy'
: 'unstable';
return {
n: s.length,
min: s[0],
p50: pct(s, 0.5),
p95: pct(s, 0.95),
p99: pct(s, 0.99),
max: s[s.length - 1],
mean,
stdev,
cv,
iqr: pct(s, 0.75) - pct(s, 0.25),
spread_ratio: s[0] === 0 ? Number.POSITIVE_INFINITY : s[s.length - 1] / s[0],
stability,
};
}
export const COMPARISON_VERDICTS = ['faster', 'slower', 'indistinguishable', 'not_comparable'] as const;
export type ComparisonVerdict = (typeof COMPARISON_VERDICTS)[number];
export interface Comparison {
verdict: ComparisonVerdict;
/** Ratio of medians. Present for information; never the claim on its own. */
p50_ratio: number;
/** Worst case for the candidate: baseline.min / candidate.max. */
conservative_ratio: number;
/** Best case for the candidate: baseline.max / candidate.min. */
optimistic_ratio: number;
/** True when the observed ranges overlap at all. */
ranges_overlap: boolean;
baseline: Distribution;
candidate: Distribution;
}
/**
* Compare two measured distributions.
*
* `faster` requires clean separation — the candidate's p95 below the
* baseline's minimum. Anything less is `indistinguishable`, however flattering
* the medians look, because that is exactly the claim a noisy host will
* manufacture for you.
*/
export function compareDistributions(baseline: Distribution, candidate: Distribution): Comparison {
const ratio = (a: number, b: number) => (b === 0 ? Number.POSITIVE_INFINITY : a / b);
const base: Omit<Comparison, 'verdict'> = {
p50_ratio: ratio(baseline.p50, candidate.p50),
conservative_ratio: ratio(baseline.min, candidate.max),
optimistic_ratio: ratio(baseline.max, candidate.min),
ranges_overlap: !(candidate.max < baseline.min || baseline.max < candidate.min),
baseline,
candidate,
};
if (baseline.stability === 'insufficient_samples' || candidate.stability === 'insufficient_samples') {
return { ...base, verdict: 'not_comparable' };
}
if (candidate.p95 < baseline.min) return { ...base, verdict: 'faster' };
if (baseline.p95 < candidate.min) return { ...base, verdict: 'slower' };
return { ...base, verdict: 'indistinguishable' };
}
/**
* The sentence a human is allowed to quote. It ALWAYS carries the range, the
* sample size and the stability verdict, so it cannot be trimmed down to "3.4x
* faster" without visibly removing the caveats.
*/
export function describeComparison(c: Comparison, label = 'candidate'): string {
const r = (v: number) => (Number.isFinite(v) ? v.toFixed(1) : '∞');
if (c.verdict === 'not_comparable') {
return (
`${label}: NOT COMPARABLE — ${c.baseline.n} baseline and ${c.candidate.n} candidate samples, ` +
`fewer than the ${MIN_SAMPLES} required. A run this short is an anecdote.`
);
}
if (c.verdict === 'indistinguishable') {
return (
`${label}: INDISTINGUISHABLE at this sample size. Medians differ by ${r(c.p50_ratio)}x but the ` +
`observed ranges overlap (baseline ${c.baseline.min.toFixed(1)}-${c.baseline.max.toFixed(1)}ms, ` +
`candidate ${c.candidate.min.toFixed(1)}-${c.candidate.max.toFixed(1)}ms; n=${c.baseline.n}/${c.candidate.n}, ` +
`CV ${c.baseline.cv.toFixed(2)}/${c.candidate.cv.toFixed(2)}).`
);
}
return (
`${label}: ${c.verdict.toUpperCase()} — ${r(c.conservative_ratio)}x to ${r(c.optimistic_ratio)}x ` +
`(median ${r(c.p50_ratio)}x) over n=${c.baseline.n}/${c.candidate.n}, ` +
`stability ${c.baseline.stability}/${c.candidate.stability}. Warm cache, one host, this query shape.`
);
}
export interface BenchmarkRun {
engine: BenchEngine;
dataset: BenchDatasetId;
query_label: string;
/** Discarded from the statistics; recorded so the discard is visible. */
warmup_ms: number[];
samples_ms: number[];
distribution: Distribution;
rows_returned?: number;
}
/**
* Run one measurement. Warmups are executed and RECORDED but excluded from the
* distribution: a first run that pays for cold marks and page cache is a
* different measurement, and silently folding it in is how a p95 becomes a
* story about disk.
*/
export async function measure(args: {
engine: BenchEngine;
dataset: BenchDatasetId;
query_label: string;
run: () => Promise<{ rows?: number }>;
warmups?: number;
repetitions?: number;
now?: () => number;
}): Promise<BenchmarkRun> {
const now = args.now ?? (() => performance.now());
const warmups = args.warmups ?? 2;
const repetitions = args.repetitions ?? 9;
const warmup_ms: number[] = [];
const samples_ms: number[] = [];
let rows: number | undefined;
for (let i = 0; i < warmups; i += 1) {
const t0 = now();
await args.run();
warmup_ms.push(now() - t0);
}
for (let i = 0; i < repetitions; i += 1) {
const t0 = now();
const out = await args.run();
samples_ms.push(now() - t0);
rows = out.rows ?? rows;
}
return {
engine: args.engine,
dataset: args.dataset,
query_label: args.query_label,
warmup_ms,
samples_ms,
distribution: summarizeSamples(samples_ms),
...(rows !== undefined ? { rows_returned: rows } : {}),
};
}
export interface BenchmarkEvidence {
recorded_at: string;
host_note: string;
datasets: typeof BENCH_DATASETS;
engines: EngineAvailability[];
runs: BenchmarkRun[];
comparisons: Array<{ dataset: BenchDatasetId; query_label: string; description: string } & Comparison>;
/** Engines BI-P104 names that produced no measurement, each with a reason. */
not_exercised: EngineAvailability[];
}
/**
* Assemble the evidence document. Every engine BI-P104 names appears, and the
* ones with no runs are listed under `not_exercised` with their reason — an
* engine quietly missing from a benchmark table reads as a choice.
*/
export function buildEvidence(args: {
host_note: string;
engines: EngineAvailability[];
runs: BenchmarkRun[];
comparisons?: BenchmarkEvidence['comparisons'];
now?: Date;
}): BenchmarkEvidence {
const measured = new Set(args.runs.map((r) => r.engine));
const declared = new Map(args.engines.map((e) => [e.engine, e]));
return {
recorded_at: (args.now ?? new Date()).toISOString(),
host_note: args.host_note,
datasets: BENCH_DATASETS,
engines: BENCH_ENGINES.map(
(engine) => declared.get(engine) ?? { engine, available: false, reason: 'not declared by this run' },
),
runs: args.runs,
comparisons: args.comparisons ?? [],
not_exercised: BENCH_ENGINES.filter((e) => !measured.has(e)).map((engine) => {
const d = declared.get(engine);
if (!d) return { engine, available: false, reason: 'not present in this environment' };
// An engine declared AVAILABLE that produced no measurement is the
// dangerous case: it reads as "we could have, and chose not to say".
// It gets a reason too, generated here rather than left blank.
return d.reason ? d : { ...d, reason: 'declared available but produced no measurement in this run' };
}),
};
}