Files
0pt/tests/integration/bi-p104-benchmark.integration.test.ts
T
Claude Opus 5 f33cc8d32f
CI (SIAX Cloud) / security (push) Successful in 30s
CI (SIAX Cloud) / integration (push) Successful in 44s
Security Scan (SIAX Cloud) / security (push) Successful in 36s
CI (SIAX Cloud) / quality (push) Successful in 1m14s
CI (SIAX Cloud) / security (pull_request) Successful in 34s
CI (SIAX Cloud) / integration (pull_request) Successful in 50s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 27s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m7s
feat(analytics): BI-P104 — a benchmark harness that refuses to quote one number
Six dataset profiles (small SaaS, medium business, VehIQ market, large event,
wide table, geospatial), one representative question each, run on ClickHouse
26.3.10.60 and DuckDB 1.5.5 with 2 warmups discarded and 9 timed repetitions.
evidence/bi-p104/benchmark.json.

The harness's job is to stop the benchmark from lying:

  - overlapping ranges are INDISTINGUISHABLE whatever the medians say. Three
    of the six real comparisons came out that way, one of them with a 1.6x
    median gap — which is exactly the headline a noisy host manufactures.
  - every sentence carries the range, the sample size and the stability, so it
    cannot be trimmed to "4.4x faster" without visibly deleting the caveats.
  - under 5 samples nothing is comparable at all. A three-run benchmark is an
    anecdote.
  - every engine BI-P104 names appears in the evidence, and the five nothing
    measured carry a reason each. The running Trino belongs to another lane,
    so benchmarking through it would have measured their configuration.

Measured shape, warm cache, one idle host, these query shapes only: DuckDB
faster on small_saas (1.6-8.2x) and vehiq_market (2.4-13.3x), SLOWER on
large_event (0.2-0.9x, count DISTINCT over 5M rows), indistinguishable on
medium_business, wide_table and geospatial.

One finding from the harness's own tests: an engine declared available that
produced no run came out of buildEvidence() with no reason — the dangerous
case, since it reads as "we could have and chose not to say". It now gets a
generated reason. A second from the runs: DuckDB's progress bar writes to
STDOUT and corrupted the first transcript; it is disabled explicitly and the
JSON is read from a sentinel.

10 unit tests + 4 against real engines.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 04:06:43 +02:00

345 lines
16 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
/**
* tests/integration/bi-p104-benchmark.integration.test.ts
* ========================================================
* BI-P104 — the benchmark itself, against real engines. Skips honestly when
* CLICKHOUSE_URL is unset; DuckDB additionally needs DUCKDB_PYTHON (an
* interpreter with the `duckdb` package), matching
* tests/duckdb/duckdb-provider.live.test.ts.
*
* Six dataset profiles, one representative question each, the same logical
* question on both engines. Deliberately NO engine-specific functions in the
* geospatial case (bounding box + grid arithmetic rather than each engine's
* GIS library) so the comparison is about the storage and execution engine
* rather than about who ships the nicer geo function.
*
* Statistics come from src/analytics/bench/benchmark-harness.ts — the shipped
* code, not a copy — so the evidence file is produced by the thing under test.
*
* Trino, Superset, native charts and Perspective are NOT exercised, and each
* appears in the evidence with its reason. An engine omitted from a benchmark
* table reads as a choice, so nothing is omitted.
*
* CLICKHOUSE_URL=http://127.0.0.1:28123 CLICKHOUSE_AUTH=default:test \
* DUCKDB_PYTHON=/path/to/venv/bin/python \
* pnpm vitest run tests/integration/bi-p104-benchmark.integration.test.ts
*
* It CREATEs and DROPs its own ClickHouse database and uses an in-memory
* DuckDB. Never point it at the b0ts ClickHouse or any quarantined data.
*/
import { execFileSync } from 'node:child_process';
import fs from 'node:fs';
import path from 'node:path';
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import {
buildEvidence,
compareDistributions,
describeComparison,
summarizeSamples,
type BenchDatasetId,
type BenchmarkRun,
type EngineAvailability,
} from '@/src/analytics/bench/benchmark-harness';
const CLICKHOUSE_URL = process.env.CLICKHOUSE_URL;
const CLICKHOUSE_AUTH = process.env.CLICKHOUSE_AUTH;
const DUCKDB_PYTHON = process.env.DUCKDB_PYTHON;
const run = CLICKHOUSE_URL ? describe : describe.skip;
const DB = `opt_p104_${Date.now().toString(36)}`;
const WARMUPS = 2;
const REPETITIONS = 9;
const WIDE_COLUMNS = 50;
const headers: Record<string, string> = CLICKHOUSE_AUTH
? { Authorization: `Basic ${Buffer.from(CLICKHOUSE_AUTH).toString('base64')}` }
: {};
async function ch(query: string): Promise<string> {
const res = await fetch(CLICKHOUSE_URL ?? 'http://skipped.invalid', {
method: 'POST',
headers,
body: query,
});
const text = await res.text();
if (!res.ok) throw new Error(`ClickHouse ${res.status}: ${text.slice(0, 300)}`);
return text;
}
/** The six questions, one per profile, phrased for each engine. */
const QUERIES: Record<BenchDatasetId, { label: string; clickhouse: string; duckdb: string }> = {
small_saas: {
label: 'revenue by day',
clickhouse: `SELECT d, sum(amount) AS revenue, count() AS n FROM ${DB}.small_saas WHERE tenant_id='t_1' GROUP BY d ORDER BY d`,
duckdb: `SELECT d, sum(amount) AS revenue, count(*) AS n FROM small_saas WHERE tenant_id='t_1' GROUP BY d ORDER BY d`,
},
medium_business: {
label: 'revenue by month and region for one tenant',
clickhouse: `SELECT toStartOfMonth(d) AS m, region, sum(amount) AS revenue FROM ${DB}.medium_business WHERE tenant_id='t_2' GROUP BY m, region ORDER BY m, region`,
duckdb: `SELECT date_trunc('month', d) AS m, region, sum(amount) AS revenue FROM medium_business WHERE tenant_id='t_2' GROUP BY m, region ORDER BY m, region`,
},
vehiq_market: {
label: 'average asking price by make and model year',
clickhouse: `SELECT make, model_year, avg(price) AS avg_price, count() AS listings FROM ${DB}.vehiq_market WHERE mileage < 150000 GROUP BY make, model_year ORDER BY make, model_year`,
duckdb: `SELECT make, model_year, avg(price) AS avg_price, count(*) AS listings FROM vehiq_market WHERE mileage < 150000 GROUP BY make, model_year ORDER BY make, model_year`,
},
large_event: {
label: 'hourly event volume for one event type',
clickhouse: `SELECT toStartOfHour(ts) AS h, count() AS n, uniqExact(user_id) AS users FROM ${DB}.large_event WHERE event_type='checkout' GROUP BY h ORDER BY h`,
duckdb: `SELECT date_trunc('hour', ts) AS h, count(*) AS n, count(DISTINCT user_id) AS users FROM large_event WHERE event_type='checkout' GROUP BY h ORDER BY h`,
},
wide_table: {
label: 'three columns of fifty, grouped',
clickhouse: `SELECT grp, sum(c7) AS a, sum(c23) AS b, sum(c41) AS c FROM ${DB}.wide_table GROUP BY grp ORDER BY grp`,
duckdb: `SELECT grp, sum(c7) AS a, sum(c23) AS b, sum(c41) AS c FROM wide_table GROUP BY grp ORDER BY grp`,
},
geospatial: {
label: 'bounding box, aggregated to a grid cell',
clickhouse: `SELECT floor(lat*10)/10 AS cell_lat, floor(lon*10)/10 AS cell_lon, count() AS n FROM ${DB}.geospatial WHERE lat BETWEEN 57.5 AND 59.5 AND lon BETWEEN 11.5 AND 18.5 GROUP BY cell_lat, cell_lon ORDER BY cell_lat, cell_lon`,
duckdb: `SELECT floor(lat*10)/10 AS cell_lat, floor(lon*10)/10 AS cell_lon, count(*) AS n FROM geospatial WHERE lat BETWEEN 57.5 AND 59.5 AND lon BETWEEN 11.5 AND 18.5 GROUP BY cell_lat, cell_lon ORDER BY cell_lat, cell_lon`,
},
};
const CH_DDL: Array<[BenchDatasetId, string]> = [
[
'small_saas',
`CREATE TABLE ${DB}.small_saas ENGINE = MergeTree ORDER BY (tenant_id, d) AS
SELECT concat('t_', toString(number % 3)) AS tenant_id,
toDate('2026-01-01') + toIntervalDay(number % 365) AS d,
['free','pro','enterprise'][(number % 3) + 1] AS plan,
toDecimal64((number % 499) + 9, 2) AS amount
FROM numbers(100000)`,
],
[
'medium_business',
`CREATE TABLE ${DB}.medium_business ENGINE = MergeTree ORDER BY (tenant_id, d) AS
SELECT concat('t_', toString(number % 8)) AS tenant_id,
toDate('2025-01-01') + toIntervalDay(number % 730) AS d,
['SE','NO','DK','FI','DE'][(number % 5) + 1] AS region,
toDecimal64((number % 977) + 13, 2) AS amount
FROM numbers(2000000)`,
],
[
'vehiq_market',
`CREATE TABLE ${DB}.vehiq_market ENGINE = MergeTree ORDER BY (make, model_year) AS
SELECT ['Volvo','VW','Toyota','BMW','Kia','Tesla'][(number % 6) + 1] AS make,
toUInt16(2005 + (number % 20)) AS model_year,
toUInt32(30000 + (number % 900000)) AS price,
toUInt32((number * 7) % 400000) AS mileage,
['diesel','petrol','hybrid','electric'][(number % 4) + 1] AS fuel
FROM numbers(1000000)`,
],
[
'large_event',
`CREATE TABLE ${DB}.large_event ENGINE = MergeTree ORDER BY (event_type, ts) AS
SELECT toDateTime('2026-08-01 00:00:00') + toIntervalSecond(number % 2592000) AS ts,
toUInt32(number % 250000) AS user_id,
['view','click','checkout','signup'][(number % 4) + 1] AS event_type,
toFloat64(number % 1000) AS value
FROM numbers(5000000)`,
],
[
'wide_table',
`CREATE TABLE ${DB}.wide_table ENGINE = MergeTree ORDER BY grp AS
SELECT toUInt16(number % 500) AS grp,
${Array.from({ length: WIDE_COLUMNS }, (_, i) => `toFloat64((number + ${i}) % 1000) AS c${i}`).join(',\n ')}
FROM numbers(200000)`,
],
[
'geospatial',
`CREATE TABLE ${DB}.geospatial ENGINE = MergeTree ORDER BY (lat, lon) AS
SELECT 55.0 + (number % 10000) / 1000.0 AS lat,
10.0 + ((number * 13) % 10000) / 1000.0 AS lon,
toUInt32(number) AS point_id
FROM numbers(500000)`,
],
];
/** Runs all six DuckDB measurements inside ONE python process. */
const DUCKDB_PROGRAM = `
import duckdb, json, time, sys
con = duckdb.connect()
# The progress bar writes to STDOUT and corrupts the JSON transcript. It cost
# one run to find, so it is turned off explicitly rather than filtered out.
con.execute("SET enable_progress_bar=false")
WIDE = ${WIDE_COLUMNS}
wide_cols = ", ".join(f"(i + {j}) % 1000.0 AS c{j}" for j in range(WIDE))
ddl = {
"small_saas": "CREATE TABLE small_saas AS SELECT 't_' || (i % 3) AS tenant_id, DATE '2026-01-01' + INTERVAL (i % 365) DAY AS d, ['free','pro','enterprise'][(i % 3) + 1] AS plan, ((i % 499) + 9)::DECIMAL(18,2) AS amount FROM range(100000) t(i)",
"medium_business": "CREATE TABLE medium_business AS SELECT 't_' || (i % 8) AS tenant_id, DATE '2025-01-01' + INTERVAL (i % 730) DAY AS d, ['SE','NO','DK','FI','DE'][(i % 5) + 1] AS region, ((i % 977) + 13)::DECIMAL(18,2) AS amount FROM range(2000000) t(i)",
"vehiq_market": "CREATE TABLE vehiq_market AS SELECT ['Volvo','VW','Toyota','BMW','Kia','Tesla'][(i % 6) + 1] AS make, (2005 + (i % 20))::USMALLINT AS model_year, (30000 + (i % 900000))::UINTEGER AS price, ((i * 7) % 400000)::UINTEGER AS mileage, ['diesel','petrol','hybrid','electric'][(i % 4) + 1] AS fuel FROM range(1000000) t(i)",
"large_event": "CREATE TABLE large_event AS SELECT TIMESTAMP '2026-08-01 00:00:00' + INTERVAL (i % 2592000) SECOND AS ts, (i % 250000)::UINTEGER AS user_id, ['view','click','checkout','signup'][(i % 4) + 1] AS event_type, (i % 1000)::DOUBLE AS value FROM range(5000000) t(i)",
"wide_table": f"CREATE TABLE wide_table AS SELECT (i % 500)::USMALLINT AS grp, {wide_cols} FROM range(200000) t(i)",
"geospatial": "CREATE TABLE geospatial AS SELECT 55.0 + (i % 10000) / 1000.0 AS lat, 10.0 + ((i * 13) % 10000) / 1000.0 AS lon, i::UINTEGER AS point_id FROM range(500000) t(i)",
}
queries = json.loads(sys.stdin.read())
out = {"version": duckdb.__version__, "runs": {}}
for name, sql in ddl.items():
con.execute(sql)
for name, q in queries.items():
warm = []
for _ in range(${WARMUPS}):
t0 = time.perf_counter(); rows = con.execute(q).fetchall(); warm.append((time.perf_counter() - t0) * 1000)
samples = []
for _ in range(${REPETITIONS}):
t0 = time.perf_counter(); rows = con.execute(q).fetchall(); samples.append((time.perf_counter() - t0) * 1000)
out["runs"][name] = {"warmup_ms": warm, "samples_ms": samples, "rows": len(rows)}
sys.stdout.write("<<<BENCH>>>" + json.dumps(out))
`;
const runs: BenchmarkRun[] = [];
const engines: EngineAvailability[] = [];
let duckdbResult:
| { version: string; runs: Record<string, { warmup_ms: number[]; samples_ms: number[]; rows: number }> }
| undefined;
run('BI-P104 benchmark harness against real engines', () => {
beforeAll(async () => {
const version = (await ch('SELECT version()')).trim();
engines.push({ engine: 'clickhouse', available: true, version });
await ch(`CREATE DATABASE IF NOT EXISTS ${DB}`);
for (const [, ddl] of CH_DDL) await ch(ddl);
}, 900_000);
afterAll(async () => {
if (!CLICKHOUSE_URL) return;
try {
await ch(`DROP DATABASE IF EXISTS ${DB}`);
} finally {
const comparisons = duckdbResult
? (Object.keys(QUERIES) as BenchDatasetId[]).flatMap((dataset) => {
const chRun = runs.find((r) => r.engine === 'clickhouse' && r.dataset === dataset);
const duckRun = runs.find((r) => r.engine === 'duckdb' && r.dataset === dataset);
if (!chRun || !duckRun) return [];
const c = compareDistributions(chRun.distribution, duckRun.distribution);
return [
{
dataset,
query_label: QUERIES[dataset].label,
description: describeComparison(c, `duckdb vs clickhouse (${dataset})`),
...c,
},
];
})
: [];
const evidence = buildEvidence({
host_note:
'One idle host, warm caches, both engines local, single-threaded client. These numbers ' +
'describe these query shapes on this data on this machine. They are not engine rankings.',
engines,
runs,
comparisons,
});
const dir = path.join(process.cwd(), 'evidence/bi-p104');
fs.mkdirSync(dir, { recursive: true });
fs.writeFileSync(path.join(dir, 'benchmark.json'), `${JSON.stringify(evidence, null, 2)}\n`);
}
}, 120_000);
it('measures all six dataset profiles on ClickHouse', async () => {
for (const dataset of Object.keys(QUERIES) as BenchDatasetId[]) {
const q = QUERIES[dataset];
const warmup_ms: number[] = [];
const samples_ms: number[] = [];
let rows = 0;
for (let i = 0; i < WARMUPS; i += 1) {
const t0 = performance.now();
await ch(`${q.clickhouse} FORMAT JSONCompact`);
warmup_ms.push(performance.now() - t0);
}
for (let i = 0; i < REPETITIONS; i += 1) {
const t0 = performance.now();
const text = await ch(`${q.clickhouse} FORMAT JSONCompact`);
samples_ms.push(performance.now() - t0);
rows = (JSON.parse(text) as { rows: number }).rows;
}
runs.push({
engine: 'clickhouse',
dataset,
query_label: q.label,
warmup_ms,
samples_ms,
distribution: summarizeSamples(samples_ms),
rows_returned: rows,
});
}
expect(runs.filter((r) => r.engine === 'clickhouse')).toHaveLength(6);
for (const r of runs) {
expect(r.distribution.n).toBe(REPETITIONS);
expect(r.distribution.stability).not.toBe('insufficient_samples');
}
}, 900_000);
it('measures the same six on DuckDB, or records why it did not', async () => {
if (!DUCKDB_PYTHON) {
engines.push({
engine: 'duckdb',
available: false,
reason: 'DUCKDB_PYTHON not set on this runner',
});
expect(engines.find((e) => e.engine === 'duckdb')?.reason).toBeTruthy();
return;
}
const payload = Object.fromEntries(
(Object.keys(QUERIES) as BenchDatasetId[]).map((d) => [d, QUERIES[d].duckdb]),
);
const stdout = execFileSync(DUCKDB_PYTHON, ['-c', DUCKDB_PROGRAM], {
input: JSON.stringify(payload),
encoding: 'utf8',
maxBuffer: 32 * 1024 * 1024,
timeout: 900_000,
});
// Parse from the sentinel: anything DuckDB writes to stdout before it
// (a progress bar, a warning) must not be able to corrupt the transcript.
const marker = stdout.lastIndexOf('<<<BENCH>>>');
expect(marker, 'duckdb transcript marker missing').toBeGreaterThanOrEqual(0);
duckdbResult = JSON.parse(stdout.slice(marker + '<<<BENCH>>>'.length));
engines.push({ engine: 'duckdb', available: true, version: duckdbResult!.version });
for (const [dataset, r] of Object.entries(duckdbResult!.runs)) {
runs.push({
engine: 'duckdb',
dataset: dataset as BenchDatasetId,
query_label: QUERIES[dataset as BenchDatasetId].label,
warmup_ms: r.warmup_ms,
samples_ms: r.samples_ms,
distribution: summarizeSamples(r.samples_ms),
rows_returned: r.rows,
});
}
expect(runs.filter((r) => r.engine === 'duckdb')).toHaveLength(6);
}, 900_000);
it('records the engines BI-P104 names that nothing measured, with reasons', () => {
engines.push(
{
engine: 'trino',
available: false,
reason:
'a Trino is running on this host but belongs to another lane (st0re-verify-trino); ' +
'benchmarking through someone else’s container would measure their configuration',
},
{
engine: 'semantic_query',
available: false,
reason: 'MetricFlow has never been executed in 0pt (BI-P029 honest status)',
},
{ engine: 'superset', available: false, reason: 'no Superset deployment in the estate yet' },
{ engine: 'native_charts', available: false, reason: 'render-side; DES1GN owns it (BI-P051..P057)' },
{ engine: 'perspective', available: false, reason: 'not deployed' },
);
const evidence = buildEvidence({ host_note: 'x', engines, runs });
for (const e of evidence.not_exercised) expect(e.reason, e.engine).toBeTruthy();
expect(evidence.engines).toHaveLength(7);
});
it('reports every comparison as a range with its stability, never a bare figure', () => {
if (!duckdbResult) return;
for (const dataset of Object.keys(QUERIES) as BenchDatasetId[]) {
const a = runs.find((r) => r.engine === 'clickhouse' && r.dataset === dataset)!;
const b = runs.find((r) => r.engine === 'duckdb' && r.dataset === dataset)!;
const sentence = describeComparison(compareDistributions(a.distribution, b.distribution), dataset);
expect(sentence).toMatch(/n=9\/9/);
expect(sentence).toMatch(/INDISTINGUISHABLE|x to /);
}
});
});