b1884a89a5be3c61aec47a41e3922eca27af715f
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
b1884a89a5 |
feat(analytics): BI-P016 — canonical query request contract, two compile targets
CI (SIAX Cloud) / security (push) Successful in 25s
Security Scan (SIAX Cloud) / security (push) Successful in 24s
CI (SIAX Cloud) / integration (push) Successful in 39s
CI (SIAX Cloud) / security (pull_request) Successful in 25s
CI (SIAX Cloud) / quality (push) Successful in 57s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 20s
CI (SIAX Cloud) / integration (pull_request) Successful in 1m38s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m59s
The eight-field AnalyticsQueryRequest BI-P016 names, verbatim, with the rule
that makes it safe enforced structurally rather than by convention: there is
no field that can carry SQL, and parseAnalyticsQueryRequest REJECTS unknown
keys instead of ignoring them. Ignoring an unsupported field is how it becomes
a supported one after the next refactor.
`tenant`, `tenant_id`, `project_id`, `database`, `catalog` and `table` are
rejected by name. INF0's Wave-2 cross-tenant hole was a tenant read out of a
request body; the fix that sticks is a request type in which the field does
not exist, and a parser that refuses it when a caller sends it anyway.
BI-P016's DoD ("same request compiles to ClickHouse/semantic engine without
consumer rewrite") is tested as an equality, not narrated: both targets are
projected onto a comparable shape and compared. The ClickHouse side of that
projection is obtained by PARSING the emitted SQL and its bound parameters in
the test — not by reading a field the compiler helpfully returned — so a
compiler that silently drops a filter fails the comparison instead of passing
it. The parser throws on any condition it does not recognise, which is what
would catch unbound caller text appearing in the statement.
BI-P011's forced tenant predicate is re-proven against this new path: five
request shapes are enumerated and each must have FORCED_TENANT_PREDICATE as
condition[0]; a blank tenant refuses to compile rather than emitting no
predicate; a hostile filter value appears only in params, never in sql.
45 tests, all failing-capable: every rejection assertion checks the SiaxError
code, and the DoD assertions compare reconstructed meaning rather than strings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
|
84bb9faa4f |
feat(analytics): BI-P002..BI-P012 analytics core — domain model, tiers, contracts, Arrow, ClickHouse engine, isolation, governance
CI (SIAX Cloud) / security (push) Successful in 41s
Security Scan (SIAX Cloud) / security (push) Successful in 40s
CI (SIAX Cloud) / integration (push) Successful in 52s
CI (SIAX Cloud) / quality (push) Successful in 1m5s
CI (SIAX Cloud) / security (pull_request) Successful in 31s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 29s
CI (SIAX Cloud) / integration (pull_request) Successful in 45s
CI (SIAX Cloud) / quality (pull_request) Successful in 53s
Implements the eight BI program prompts that make 0pt the canonical SIAX
analytics capability. Every file is NEW: not one existing file is modified,
and src/analytics/clickhouse-analytics.ts is byte-identical to main.
BI-P002 — the nineteen canonical entities, verbatim, with tenant_id +
project_id on every tenant-owned record. sql/008 gives each one a table.
The four version entities BI-P002 names as immutable are enforced by BEFORE
UPDATE/DELETE triggers, not by a TypeScript `readonly` — proven by direct SQL
against a live Postgres.
BI-P003 — RAW/NORMALIZED/CURATED/SERVING with the read rule enforced:
assertReadableByBiSurface refuses a RAW dataset to a dashboard unless a
recorded exception exists. All eight required declarations are NOT NULL.
BI-P004 — DatasetContract and a fail-closed compatibility gate. Fifteen
breaking-change classes are each tested twice: refused under a minor bump,
accepted under a major one.
BI-P005 — Arrow IPC alongside JSON, with a hand-written FlatBuffers writer so
no npm dependency (or CVE surface) is added. Conformance comes from three
independent implementations, not from self-agreement: pyarrow 25.0.1 and
DuckDB 1.5.5 read identical values from both ClickHouse's Arrow and ours, and
ClickHouse's own Arrow C++ ingests our 20,000-row batch with exact aggregates.
BI-P009 — SiaxAnalyticsEngine with all eight capabilities. The existing
ClickHouseAnalytics is COMPOSED, never replaced. executeQuery (analyst SQL) is
declared and deliberately refused with CAPABILITY_UNAVAILABLE until BI-P017 and
BI-P033 land, rather than opening the raw-SQL hole BI-P011 enumerates.
BI-P010 — four purpose-built tables, not one giant one, and the difference is
measured against the counterfactual: 48.8ms/40,960 rows vs 174.4ms/5,400,000
rows for the same question; 5/660 granules vs 660/1149.
BI-P011 — layered isolation, each layer proven with the layer above removed.
Without the forced predicate the same statement returns 50,050 (both tenants);
with it, t_alpha gets 50. Without a row policy an unrestricted user sees 2
tenants; the restricted user running the IDENTICAL statement sees 1.
BI-P012 — every named limit, fired individually and identified by ClickHouse
error code, plus the half usually skipped: an unrelated tenant is timed while
a real CPU bomb is in flight and must still return the CORRECT answer.
Three defects found by writing tests that can fail:
* `CHECK (array_length(purposes,1) >= 1)` can NEVER fail — array_length is
NULL for '{}' and a NULL CHECK passes. Fixed with coalesce(); the test
that caught it is pinned.
* EXPLAIN ESTIMATE run under the caller's own max_rows_to_read fails with
code 158 exactly when the estimate would be over budget. Planner calls now
use a narrower envelope.
* result_overflow_mode=break returns HTTP 200 with 65,409 of 100,000 rows
and no indication it is short. Every SELECT ships `throw`.
Verified rather than assumed: server7's ClickHouse holds `opt_scraped` (51
tables, 407,793,471 rows, `persons`/`cz_owners` included) — the b0ts plane this
module is prohibited from touching — has no `opt_analytics` database, zero
tenant_id columns and zero row policies. Nothing there was modified; every
proof ran against disposable containers.
Evidence: evidence/bi-p002-p012/{test-transcript.txt,bi-p010-benchmark.json,
bi-p005-arrow-roundtrip.json,server7-clickhouse-probe.md}
Docs: docs/analytics/BI_ANALYTICS_CORE.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
|
b46807da75 |
feat(data-os): P0 capability layer — source registry, search, quality, dataset versions, metadata, analytics, features, ER
- Source Registry (§10): rights/license/purpose metadata, provenance,
freshness SLA, health checks, describe/query; purpose gate FAILS CLOSED
(no rights record => deny); cross-tenant probes => NOT_FOUND.
- Search (§10/§42): OpenSearch REST adapter (no SDK) + SearchService with
MANDATORY tenant + purpose/rights filtering, versioned index definitions,
bulk, generation-per-rebuild (opt-{name}--g{n}) with atomic alias cutover,
rollback, failed-rebuild generation discard. OpenSearch never canonical.
- Quality (§10): versioned rule registry (GE-JSON interchange), local
engine for the GE core subset (never silently passes unsupported types),
runs/results/issues/score/trend, whole-run delegation to the GE backend
via N0D (0pt.quality.ge_run) - never fabricated results.
- Dataset (§10/§5/§53): immutable numbered versions holding canonical
st0re:artifact refs only, idempotent createVersion, export registers the
ST0RE artifact + AUD0 evidence + OpenLineage.
- Metadata (§10): OpenMetadata PUT /api/v1/tables projection adapter +
projectAll rebuild path; projection never canonical.
- Analytics (§10): ClickHouse HTTP adapter, closed metric-query DSL with
FORCED tenant predicate + readonly/max_execution_time/max_result_rows,
TTL retention. Zero connection to b0ts/quarantined pipelines.
- Features (§10): versioned feature-view registry, real Feast
get-online-features adapter, materialize/get-historical as N0D delegation.
- ER (§10): generic candidate API with deterministic blocking + transparent
scoring + per-field explanations, product hard-rule hook (force both
directions), review-band handoff, Splink batch via N0D contract.
93 new unit/contract tests; suite 248 passed / 1 skipped.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|