Claude Opus 5
f33cc8d32f
CI (SIAX Cloud) / security (push) Successful in 30s
CI (SIAX Cloud) / integration (push) Successful in 44s
Security Scan (SIAX Cloud) / security (push) Successful in 36s
CI (SIAX Cloud) / quality (push) Successful in 1m14s
CI (SIAX Cloud) / security (pull_request) Successful in 34s
CI (SIAX Cloud) / integration (pull_request) Successful in 50s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 27s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m7s
Six dataset profiles (small SaaS, medium business, VehIQ market, large event,
wide table, geospatial), one representative question each, run on ClickHouse
26.3.10.60 and DuckDB 1.5.5 with 2 warmups discarded and 9 timed repetitions.
evidence/bi-p104/benchmark.json.
The harness's job is to stop the benchmark from lying:
- overlapping ranges are INDISTINGUISHABLE whatever the medians say. Three
of the six real comparisons came out that way, one of them with a 1.6x
median gap — which is exactly the headline a noisy host manufactures.
- every sentence carries the range, the sample size and the stability, so it
cannot be trimmed to "4.4x faster" without visibly deleting the caveats.
- under 5 samples nothing is comparable at all. A three-run benchmark is an
anecdote.
- every engine BI-P104 names appears in the evidence, and the five nothing
measured carry a reason each. The running Trino belongs to another lane,
so benchmarking through it would have measured their configuration.
Measured shape, warm cache, one idle host, these query shapes only: DuckDB
faster on small_saas (1.6-8.2x) and vehiq_market (2.4-13.3x), SLOWER on
large_event (0.2-0.9x, count DISTINCT over 5M rows), indistinguishable on
medium_business, wide_table and geospatial.
One finding from the harness's own tests: an engine declared available that
produced no run came out of buildEvidence() with no reason — the dangerous
case, since it reads as "we could have and chose not to say". It now gets a
generated reason. A second from the runs: DuckDB's progress bar writes to
STDOUT and corrupted the first transcript; it is disabled explicitly and the
JSON is read from a sentinel.
10 unit tests + 4 against real engines.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>