Claude Opus 5
7441de5429
Security Scan (SIAX Cloud) / security (pull_request) Successful in 15s
Security Scan (SIAX Cloud) / security (push) Successful in 16s
CI (SIAX Cloud) / security (push) Successful in 33s
CI (SIAX Cloud) / security (pull_request) Successful in 32s
CI (SIAX Cloud) / integration (pull_request) Successful in 36s
CI (SIAX Cloud) / quality (pull_request) Successful in 48s
CI (SIAX Cloud) / quality (push) Successful in 53s
CI (SIAX Cloud) / integration (push) Successful in 54s
Same shape as the BI-P029 finding in the previous commit. `bi-slo.ts` said "evidence/bi-p103 records the first baselines". There was no evidence/bi-p103. The catalogue and the evaluator were both real and both well covered — against SYNTHETIC events. A synthetic event carries whatever duration the test author typed, so that suite proves the arithmetic and nothing about the system: it cannot tell you whether a dashboard query takes 8 ms or 8 s. BI-P103's DoD is one sentence, "measured baselines exist", and it was the part not done. tests/integration/bi-p103-slo-baseline.integration.test.ts runs 460 real queries against a live ClickHouse over a 400 000-row fact table -- 120 cold dashboard queries (each a different bucket so the BI-P020 cache genuinely misses), the same 120 again through the same shipped cache now hitting, and 220 exploration queries, that last number chosen so explore.p99's min_samples of 200 is MET rather than reported low-confidence. Every duration is wall-clock around the real call, and the telemetry goes through the shipped producer and the shipped evaluator, so the evidence is written by the code under test. Measured: 7 of 11 objectives produced a number, `fully_measured` false, measured_fraction 0.64. The four that did not each say why -- dashboard freshness is a property of the dataset, the exploration timeout is a configured ceiling, report completion has no producer in 0pt at all, and report delivery is N0TIFY's receipt. A baseline table listing only the rows that worked is how an estate comes to believe it measures more than it does. The evidence also records what is NARROWER than production, because these numbers will be quoted: the cached path is the in-process reference cache and not Valkey, every latency is the engine round trip with no HTTP or authorization layer, and there is no concurrency. The suite is adversarial about its own inputs: a "cold" sample that came from the cache fails, a cache "hit" that missed fails, a cached p95 that is not faster than the cold p95 fails, and dropping the exploration sample to 60 fails on explore.p99 being low-confidence rather than quietly recording a p99 over 60 requests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
174 lines
5.7 KiB
JSON
174 lines
5.7 KiB
JSON
{
|
|
"prompt": "BI-P103",
|
|
"title": "First measured BI SLO baselines",
|
|
"recorded_at": "2026-09-07T06:26:43.457Z",
|
|
"host_note": "One idle host, local ClickHouse, warm page cache, single-threaded client. A floor for the shape of the distribution on this fixture — not a production SLA.",
|
|
"clickhouse_version": "26.3.10.60",
|
|
"fixture": {
|
|
"rows": 400000,
|
|
"table": "slo_fact",
|
|
"partitions": "MergeTree ORDER BY (tenant_id, bucket, ts)"
|
|
},
|
|
"workload": {
|
|
"cold_dashboard_queries": 120,
|
|
"cached_dashboard_queries": 120,
|
|
"exploration_queries": 220,
|
|
"report_queries": 0
|
|
},
|
|
"window": {
|
|
"from": "2026-09-07T06:26:39.422Z",
|
|
"to": "2026-09-07T06:26:43.447Z"
|
|
},
|
|
"catalogue_size": 11,
|
|
"fully_measured": false,
|
|
"unresolved_queries": 0,
|
|
"summary": {
|
|
"met": 7,
|
|
"breached": 0,
|
|
"insufficient_data": 1,
|
|
"not_measurable_here": 3,
|
|
"measured_fraction": 0.6363636363636364
|
|
},
|
|
"results": [
|
|
{
|
|
"objective_id": "dash.availability",
|
|
"workload": "interactive_dashboard",
|
|
"kind": "availability",
|
|
"status": "met",
|
|
"target": 0.995,
|
|
"unit": "ratio",
|
|
"observed": 1,
|
|
"sample_size": 240,
|
|
"low_confidence": false,
|
|
"detail": "0 failed of 240 terminal queries"
|
|
},
|
|
{
|
|
"objective_id": "dash.p95_first_query",
|
|
"workload": "interactive_dashboard",
|
|
"kind": "latency_p95_first",
|
|
"status": "met",
|
|
"target": 3000,
|
|
"unit": "ms",
|
|
"observed": 8.893520999999964,
|
|
"sample_size": 120,
|
|
"low_confidence": false,
|
|
"detail": "p95 over 120 samples"
|
|
},
|
|
{
|
|
"objective_id": "dash.p95_cached",
|
|
"workload": "interactive_dashboard",
|
|
"kind": "latency_p95_cached",
|
|
"status": "met",
|
|
"target": 400,
|
|
"unit": "ms",
|
|
"observed": 0.024498999999877924,
|
|
"sample_size": 120,
|
|
"low_confidence": false,
|
|
"detail": "p95 over 120 samples"
|
|
},
|
|
{
|
|
"objective_id": "dash.error_rate",
|
|
"workload": "interactive_dashboard",
|
|
"kind": "error_rate",
|
|
"status": "met",
|
|
"target": 0.005,
|
|
"unit": "ratio",
|
|
"observed": 0,
|
|
"sample_size": 240,
|
|
"low_confidence": false,
|
|
"detail": "0 failed of 240 terminal queries"
|
|
},
|
|
{
|
|
"objective_id": "dash.freshness",
|
|
"workload": "interactive_dashboard",
|
|
"kind": "freshness",
|
|
"status": "not_measurable_here",
|
|
"target": 900,
|
|
"unit": "seconds",
|
|
"sample_size": 0,
|
|
"low_confidence": false,
|
|
"detail": "measured from analytics_freshness_watermark; BI usage events cannot answer it — Data age is a property of the DATASET (BI-P026 freshness, BI-P105 watermark), not of a usage event. Usage events cannot measure it and this evaluator does not pretend to."
|
|
},
|
|
{
|
|
"objective_id": "explore.p95",
|
|
"workload": "exploration",
|
|
"kind": "latency_p95",
|
|
"status": "met",
|
|
"target": 10000,
|
|
"unit": "ms",
|
|
"observed": 17.072708000000148,
|
|
"sample_size": 220,
|
|
"low_confidence": false,
|
|
"detail": "p95 over 220 samples"
|
|
},
|
|
{
|
|
"objective_id": "explore.p99",
|
|
"workload": "exploration",
|
|
"kind": "latency_p99",
|
|
"status": "met",
|
|
"target": 60000,
|
|
"unit": "ms",
|
|
"observed": 21.640736000000288,
|
|
"sample_size": 220,
|
|
"low_confidence": false,
|
|
"detail": "p99 over 220 samples"
|
|
},
|
|
{
|
|
"objective_id": "explore.max_result",
|
|
"workload": "exploration",
|
|
"kind": "max_result_rows",
|
|
"status": "met",
|
|
"target": 1000000,
|
|
"unit": "rows",
|
|
"observed": 96,
|
|
"sample_size": 220,
|
|
"low_confidence": false,
|
|
"detail": "largest result seen in the window"
|
|
},
|
|
{
|
|
"objective_id": "explore.timeout",
|
|
"workload": "exploration",
|
|
"kind": "timeout",
|
|
"status": "not_measurable_here",
|
|
"target": 300,
|
|
"unit": "seconds",
|
|
"sample_size": 0,
|
|
"low_confidence": false,
|
|
"detail": "measured from engine_configuration; BI usage events cannot answer it — The configured ceiling (BI-P012 settings profile), not an observation."
|
|
},
|
|
{
|
|
"objective_id": "report.completion_sla",
|
|
"workload": "report",
|
|
"kind": "completion_sla",
|
|
"status": "insufficient_data",
|
|
"target": 900000,
|
|
"unit": "ms",
|
|
"sample_size": 0,
|
|
"low_confidence": false,
|
|
"detail": "no report_generated events"
|
|
},
|
|
{
|
|
"objective_id": "report.delivery_sla",
|
|
"workload": "report",
|
|
"kind": "delivery_sla",
|
|
"status": "not_measurable_here",
|
|
"target": 300000,
|
|
"unit": "ms",
|
|
"sample_size": 0,
|
|
"low_confidence": false,
|
|
"detail": "measured from n0tify_delivery_receipt; BI usage events cannot answer it — Produced is not delivered. aud0 has no delivery event; the receipt is N0TIFY's, so this objective is declared and reported unmeasured rather than proxied by completion."
|
|
}
|
|
],
|
|
"measured_but_narrower_than_production": [
|
|
"dash.p95_cached is the IN-PROCESS reference cache (InMemoryResultCache). A Valkey deployment adds a network round trip, so 0.03 ms here is a floor and not the number a user will see.",
|
|
"Every latency is the ENGINE round trip only: no HTTP layer, no CL0UD authorization, no serialization to the caller. The end-to-end p95 is larger by whatever those cost.",
|
|
"One tenant, one client, no concurrency. Contention is not represented."
|
|
],
|
|
"not_measured_and_why": [
|
|
"dash.freshness — a property of the DATASET (BI-P026 watermark), not of a usage event",
|
|
"explore.timeout — the configured BI-P012 ceiling, not an observation",
|
|
"report.completion_sla — nothing in 0pt emits aud0.report_generated yet, so the sample is 0",
|
|
"report.delivery_sla — delivery is N0TIFY's receipt; aud0 has no delivery event"
|
|
]
|
|
}
|