Lägger docs/production-readiness.md och siax/production-gaps.json enligt SIAX Master Definition of Done. Underlag: git-, kod- och live-evidens 2026-11-11. Nivån är evidensbaserad, inte ett leveransgodkännande.
Six dataset profiles (small SaaS, medium business, VehIQ market, large event,
wide table, geospatial), one representative question each, run on ClickHouse
26.3.10.60 and DuckDB 1.5.5 with 2 warmups discarded and 9 timed repetitions.
evidence/bi-p104/benchmark.json.
The harness's job is to stop the benchmark from lying:
- overlapping ranges are INDISTINGUISHABLE whatever the medians say. Three
of the six real comparisons came out that way, one of them with a 1.6x
median gap — which is exactly the headline a noisy host manufactures.
- every sentence carries the range, the sample size and the stability, so it
cannot be trimmed to "4.4x faster" without visibly deleting the caveats.
- under 5 samples nothing is comparable at all. A three-run benchmark is an
anecdote.
- every engine BI-P104 names appears in the evidence, and the five nothing
measured carry a reason each. The running Trino belongs to another lane,
so benchmarking through it would have measured their configuration.
Measured shape, warm cache, one idle host, these query shapes only: DuckDB
faster on small_saas (1.6-8.2x) and vehiq_market (2.4-13.3x), SLOWER on
large_event (0.2-0.9x, count DISTINCT over 5M rows), indistinguishable on
medium_business, wide_table and geospatial.
One finding from the harness's own tests: an engine declared available that
produced no run came out of buildEvidence() with no reason — the dangerous
case, since it reads as "we could have and chose not to say". It now gets a
generated reason. A second from the runs: DuckDB's progress bar writes to
STDOUT and corrupted the first transcript; it is disabled explicitly and the
JSON is read from a sentinel.
10 unit tests + 4 against real engines.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>