CI (SIAX Cloud) / security (push) Successful in 30s
CI (SIAX Cloud) / integration (push) Successful in 44s
Security Scan (SIAX Cloud) / security (push) Successful in 36s
CI (SIAX Cloud) / quality (push) Successful in 1m14s
CI (SIAX Cloud) / security (pull_request) Successful in 34s
CI (SIAX Cloud) / integration (pull_request) Successful in 50s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 27s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m7s
Six dataset profiles (small SaaS, medium business, VehIQ market, large event,
wide table, geospatial), one representative question each, run on ClickHouse
26.3.10.60 and DuckDB 1.5.5 with 2 warmups discarded and 9 timed repetitions.
evidence/bi-p104/benchmark.json.
The harness's job is to stop the benchmark from lying:
- overlapping ranges are INDISTINGUISHABLE whatever the medians say. Three
of the six real comparisons came out that way, one of them with a 1.6x
median gap — which is exactly the headline a noisy host manufactures.
- every sentence carries the range, the sample size and the stability, so it
cannot be trimmed to "4.4x faster" without visibly deleting the caveats.
- under 5 samples nothing is comparable at all. A three-run benchmark is an
anecdote.
- every engine BI-P104 names appears in the evidence, and the five nothing
measured carry a reason each. The running Trino belongs to another lane,
so benchmarking through it would have measured their configuration.
Measured shape, warm cache, one idle host, these query shapes only: DuckDB
faster on small_saas (1.6-8.2x) and vehiq_market (2.4-13.3x), SLOWER on
large_event (0.2-0.9x, count DISTINCT over 5M rows), indistinguishable on
medium_business, wide_table and geospatial.
One finding from the harness's own tests: an engine declared available that
produced no run came out of buildEvidence() with no reason — the dangerous
case, since it reads as "we could have and chose not to say". It now gets a
generated reason. A second from the runs: DuckDB's progress bar writes to
STDOUT and corrupted the first transcript; it is disabled explicitly and the
JSON is read from a sentinel.
10 unit tests + 4 against real engines.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
342 lines
12 KiB
TypeScript
342 lines
12 KiB
TypeScript
/**
|
|
* src/analytics/bench/benchmark-harness.ts
|
|
* =========================================
|
|
* BI-P104 — BI PERFORMANCE BENCHMARK HARNESS. DoD: "results committed as
|
|
* machine-readable evidence."
|
|
*
|
|
* The harness exists because of a specific way benchmarks lie. Yesterday's
|
|
* lane measured the same budget-enforcement overshoot at ~150x on one run and
|
|
* ~130x on the next, on the same machine, and correctly reported the SHAPE
|
|
* rather than a number. That is the doctrine encoded here:
|
|
*
|
|
* 1. A SINGLE FIGURE IS NOT A SYSTEM PROPERTY. `summarizeSamples` returns a
|
|
* distribution — n, min, p50, p95, p99, max, mean, stdev, CV, IQR — and
|
|
* there is no function anywhere in this module that returns one number
|
|
* for a measurement. `describeComparison` renders a RANGE and the sample
|
|
* size into every sentence it produces.
|
|
*
|
|
* 2. OVERLAPPING DISTRIBUTIONS ARE INDISTINGUISHABLE, whatever the medians
|
|
* say. If the candidate's p95 is not below the baseline's minimum, the
|
|
* verdict is `indistinguishable` even when p50 differs by 2x — because a
|
|
* 2x median gap inside a 10x spread is noise with a story attached.
|
|
*
|
|
* 3. TOO FEW SAMPLES IS A REFUSAL. Under MIN_SAMPLES the summary carries
|
|
* `stability: 'insufficient_samples'` and comparisons refuse to judge.
|
|
* A three-run benchmark is an anecdote.
|
|
*
|
|
* 4. AN ENGINE THAT WAS NOT RUN IS RECORDED AS NOT RUN. `EngineAvailability`
|
|
* carries a reason. The evidence file lists every engine BI-P104 names,
|
|
* including the ones nothing measured — an omitted engine reads as "we
|
|
* chose not to report it".
|
|
*/
|
|
|
|
/** The six dataset profiles BI-P104 names, with what makes each one distinct. */
|
|
export const BENCH_DATASETS = [
|
|
{
|
|
id: 'small_saas',
|
|
label: 'small SaaS',
|
|
approx_rows: 100_000,
|
|
shape: 'narrow fact table, one tenant, high selectivity',
|
|
},
|
|
{
|
|
id: 'medium_business',
|
|
label: 'medium business',
|
|
approx_rows: 2_000_000,
|
|
shape: 'multi-tenant fact table with a date and a few dimensions',
|
|
},
|
|
{
|
|
id: 'vehiq_market',
|
|
label: 'VehIQ market',
|
|
approx_rows: 1_000_000,
|
|
shape: 'vehicle listings: make/model/year/price/mileage, skewed cardinalities',
|
|
},
|
|
{
|
|
id: 'large_event',
|
|
label: 'large event',
|
|
approx_rows: 5_000_000,
|
|
shape: 'append-only event stream, timestamp-ordered, sparse dimensions',
|
|
},
|
|
{
|
|
id: 'wide_table',
|
|
label: 'wide table',
|
|
approx_rows: 200_000,
|
|
shape: 'many columns, few read per query — the columnar-store best case',
|
|
},
|
|
{
|
|
id: 'geospatial',
|
|
label: 'geospatial',
|
|
approx_rows: 500_000,
|
|
shape: 'lat/lon points with bounding-box and distance predicates',
|
|
},
|
|
] as const;
|
|
export type BenchDatasetId = (typeof BENCH_DATASETS)[number]['id'];
|
|
|
|
/** Every engine BI-P104 lists. All of them appear in the evidence, run or not. */
|
|
export const BENCH_ENGINES = [
|
|
'clickhouse',
|
|
'duckdb',
|
|
'trino',
|
|
'semantic_query',
|
|
'superset',
|
|
'native_charts',
|
|
'perspective',
|
|
] as const;
|
|
export type BenchEngine = (typeof BENCH_ENGINES)[number];
|
|
|
|
export interface EngineAvailability {
|
|
engine: BenchEngine;
|
|
available: boolean;
|
|
/** Required when unavailable. "Unknown" is not a reason. */
|
|
reason?: string;
|
|
version?: string;
|
|
}
|
|
|
|
export const STABILITY_VERDICTS = ['stable', 'noisy', 'unstable', 'insufficient_samples'] as const;
|
|
export type StabilityVerdict = (typeof STABILITY_VERDICTS)[number];
|
|
|
|
/** Below this a run is an anecdote, not a measurement. */
|
|
export const MIN_SAMPLES = 5;
|
|
/** Coefficient of variation thresholds. Wide on purpose: real hosts are noisy. */
|
|
const CV_STABLE = 0.15;
|
|
const CV_NOISY = 0.5;
|
|
|
|
export interface Distribution {
|
|
n: number;
|
|
min: number;
|
|
p50: number;
|
|
p95: number;
|
|
p99: number;
|
|
max: number;
|
|
mean: number;
|
|
stdev: number;
|
|
/** stdev / mean. The number that decides whether a comparison is meaningful. */
|
|
cv: number;
|
|
iqr: number;
|
|
/** max / min. Reported because it is what a reader feels. */
|
|
spread_ratio: number;
|
|
stability: StabilityVerdict;
|
|
}
|
|
|
|
/** Nearest-rank; no interpolation, so every reported value really happened. */
|
|
function pct(sorted: number[], q: number): number {
|
|
const rank = Math.ceil(q * sorted.length);
|
|
return sorted[Math.min(sorted.length - 1, Math.max(0, rank - 1))];
|
|
}
|
|
|
|
export function summarizeSamples(samples: readonly number[]): Distribution {
|
|
if (samples.length === 0) {
|
|
return {
|
|
n: 0,
|
|
min: 0,
|
|
p50: 0,
|
|
p95: 0,
|
|
p99: 0,
|
|
max: 0,
|
|
mean: 0,
|
|
stdev: 0,
|
|
cv: 0,
|
|
iqr: 0,
|
|
spread_ratio: 0,
|
|
stability: 'insufficient_samples',
|
|
};
|
|
}
|
|
const s = [...samples].sort((a, b) => a - b);
|
|
const mean = s.reduce((a, b) => a + b, 0) / s.length;
|
|
const variance = s.reduce((acc, v) => acc + (v - mean) ** 2, 0) / s.length;
|
|
const stdev = Math.sqrt(variance);
|
|
const cv = mean === 0 ? 0 : stdev / mean;
|
|
const stability: StabilityVerdict =
|
|
s.length < MIN_SAMPLES
|
|
? 'insufficient_samples'
|
|
: cv <= CV_STABLE
|
|
? 'stable'
|
|
: cv <= CV_NOISY
|
|
? 'noisy'
|
|
: 'unstable';
|
|
return {
|
|
n: s.length,
|
|
min: s[0],
|
|
p50: pct(s, 0.5),
|
|
p95: pct(s, 0.95),
|
|
p99: pct(s, 0.99),
|
|
max: s[s.length - 1],
|
|
mean,
|
|
stdev,
|
|
cv,
|
|
iqr: pct(s, 0.75) - pct(s, 0.25),
|
|
spread_ratio: s[0] === 0 ? Number.POSITIVE_INFINITY : s[s.length - 1] / s[0],
|
|
stability,
|
|
};
|
|
}
|
|
|
|
export const COMPARISON_VERDICTS = ['faster', 'slower', 'indistinguishable', 'not_comparable'] as const;
|
|
export type ComparisonVerdict = (typeof COMPARISON_VERDICTS)[number];
|
|
|
|
export interface Comparison {
|
|
verdict: ComparisonVerdict;
|
|
/** Ratio of medians. Present for information; never the claim on its own. */
|
|
p50_ratio: number;
|
|
/** Worst case for the candidate: baseline.min / candidate.max. */
|
|
conservative_ratio: number;
|
|
/** Best case for the candidate: baseline.max / candidate.min. */
|
|
optimistic_ratio: number;
|
|
/** True when the observed ranges overlap at all. */
|
|
ranges_overlap: boolean;
|
|
baseline: Distribution;
|
|
candidate: Distribution;
|
|
}
|
|
|
|
/**
|
|
* Compare two measured distributions.
|
|
*
|
|
* `faster` requires clean separation — the candidate's p95 below the
|
|
* baseline's minimum. Anything less is `indistinguishable`, however flattering
|
|
* the medians look, because that is exactly the claim a noisy host will
|
|
* manufacture for you.
|
|
*/
|
|
export function compareDistributions(baseline: Distribution, candidate: Distribution): Comparison {
|
|
const ratio = (a: number, b: number) => (b === 0 ? Number.POSITIVE_INFINITY : a / b);
|
|
const base: Omit<Comparison, 'verdict'> = {
|
|
p50_ratio: ratio(baseline.p50, candidate.p50),
|
|
conservative_ratio: ratio(baseline.min, candidate.max),
|
|
optimistic_ratio: ratio(baseline.max, candidate.min),
|
|
ranges_overlap: !(candidate.max < baseline.min || baseline.max < candidate.min),
|
|
baseline,
|
|
candidate,
|
|
};
|
|
if (baseline.stability === 'insufficient_samples' || candidate.stability === 'insufficient_samples') {
|
|
return { ...base, verdict: 'not_comparable' };
|
|
}
|
|
if (candidate.p95 < baseline.min) return { ...base, verdict: 'faster' };
|
|
if (baseline.p95 < candidate.min) return { ...base, verdict: 'slower' };
|
|
return { ...base, verdict: 'indistinguishable' };
|
|
}
|
|
|
|
/**
|
|
* The sentence a human is allowed to quote. It ALWAYS carries the range, the
|
|
* sample size and the stability verdict, so it cannot be trimmed down to "3.4x
|
|
* faster" without visibly removing the caveats.
|
|
*/
|
|
export function describeComparison(c: Comparison, label = 'candidate'): string {
|
|
const r = (v: number) => (Number.isFinite(v) ? v.toFixed(1) : '∞');
|
|
if (c.verdict === 'not_comparable') {
|
|
return (
|
|
`${label}: NOT COMPARABLE — ${c.baseline.n} baseline and ${c.candidate.n} candidate samples, ` +
|
|
`fewer than the ${MIN_SAMPLES} required. A run this short is an anecdote.`
|
|
);
|
|
}
|
|
if (c.verdict === 'indistinguishable') {
|
|
return (
|
|
`${label}: INDISTINGUISHABLE at this sample size. Medians differ by ${r(c.p50_ratio)}x but the ` +
|
|
`observed ranges overlap (baseline ${c.baseline.min.toFixed(1)}-${c.baseline.max.toFixed(1)}ms, ` +
|
|
`candidate ${c.candidate.min.toFixed(1)}-${c.candidate.max.toFixed(1)}ms; n=${c.baseline.n}/${c.candidate.n}, ` +
|
|
`CV ${c.baseline.cv.toFixed(2)}/${c.candidate.cv.toFixed(2)}).`
|
|
);
|
|
}
|
|
return (
|
|
`${label}: ${c.verdict.toUpperCase()} — ${r(c.conservative_ratio)}x to ${r(c.optimistic_ratio)}x ` +
|
|
`(median ${r(c.p50_ratio)}x) over n=${c.baseline.n}/${c.candidate.n}, ` +
|
|
`stability ${c.baseline.stability}/${c.candidate.stability}. Warm cache, one host, this query shape.`
|
|
);
|
|
}
|
|
|
|
export interface BenchmarkRun {
|
|
engine: BenchEngine;
|
|
dataset: BenchDatasetId;
|
|
query_label: string;
|
|
/** Discarded from the statistics; recorded so the discard is visible. */
|
|
warmup_ms: number[];
|
|
samples_ms: number[];
|
|
distribution: Distribution;
|
|
rows_returned?: number;
|
|
}
|
|
|
|
/**
|
|
* Run one measurement. Warmups are executed and RECORDED but excluded from the
|
|
* distribution: a first run that pays for cold marks and page cache is a
|
|
* different measurement, and silently folding it in is how a p95 becomes a
|
|
* story about disk.
|
|
*/
|
|
export async function measure(args: {
|
|
engine: BenchEngine;
|
|
dataset: BenchDatasetId;
|
|
query_label: string;
|
|
run: () => Promise<{ rows?: number }>;
|
|
warmups?: number;
|
|
repetitions?: number;
|
|
now?: () => number;
|
|
}): Promise<BenchmarkRun> {
|
|
const now = args.now ?? (() => performance.now());
|
|
const warmups = args.warmups ?? 2;
|
|
const repetitions = args.repetitions ?? 9;
|
|
const warmup_ms: number[] = [];
|
|
const samples_ms: number[] = [];
|
|
let rows: number | undefined;
|
|
|
|
for (let i = 0; i < warmups; i += 1) {
|
|
const t0 = now();
|
|
await args.run();
|
|
warmup_ms.push(now() - t0);
|
|
}
|
|
for (let i = 0; i < repetitions; i += 1) {
|
|
const t0 = now();
|
|
const out = await args.run();
|
|
samples_ms.push(now() - t0);
|
|
rows = out.rows ?? rows;
|
|
}
|
|
return {
|
|
engine: args.engine,
|
|
dataset: args.dataset,
|
|
query_label: args.query_label,
|
|
warmup_ms,
|
|
samples_ms,
|
|
distribution: summarizeSamples(samples_ms),
|
|
...(rows !== undefined ? { rows_returned: rows } : {}),
|
|
};
|
|
}
|
|
|
|
export interface BenchmarkEvidence {
|
|
recorded_at: string;
|
|
host_note: string;
|
|
datasets: typeof BENCH_DATASETS;
|
|
engines: EngineAvailability[];
|
|
runs: BenchmarkRun[];
|
|
comparisons: Array<{ dataset: BenchDatasetId; query_label: string; description: string } & Comparison>;
|
|
/** Engines BI-P104 names that produced no measurement, each with a reason. */
|
|
not_exercised: EngineAvailability[];
|
|
}
|
|
|
|
/**
|
|
* Assemble the evidence document. Every engine BI-P104 names appears, and the
|
|
* ones with no runs are listed under `not_exercised` with their reason — an
|
|
* engine quietly missing from a benchmark table reads as a choice.
|
|
*/
|
|
export function buildEvidence(args: {
|
|
host_note: string;
|
|
engines: EngineAvailability[];
|
|
runs: BenchmarkRun[];
|
|
comparisons?: BenchmarkEvidence['comparisons'];
|
|
now?: Date;
|
|
}): BenchmarkEvidence {
|
|
const measured = new Set(args.runs.map((r) => r.engine));
|
|
const declared = new Map(args.engines.map((e) => [e.engine, e]));
|
|
return {
|
|
recorded_at: (args.now ?? new Date()).toISOString(),
|
|
host_note: args.host_note,
|
|
datasets: BENCH_DATASETS,
|
|
engines: BENCH_ENGINES.map(
|
|
(engine) => declared.get(engine) ?? { engine, available: false, reason: 'not declared by this run' },
|
|
),
|
|
runs: args.runs,
|
|
comparisons: args.comparisons ?? [],
|
|
not_exercised: BENCH_ENGINES.filter((e) => !measured.has(e)).map((engine) => {
|
|
const d = declared.get(engine);
|
|
if (!d) return { engine, available: false, reason: 'not present in this environment' };
|
|
// An engine declared AVAILABLE that produced no measurement is the
|
|
// dangerous case: it reads as "we could have, and chose not to say".
|
|
// It gets a reason too, generated here rather than left blank.
|
|
return d.reason ? d : { ...d, reason: 'declared available but produced no measurement in this run' };
|
|
}),
|
|
};
|
|
}
|