Files
0pt/src/analytics/domain/entities.ts
T
Claude Opus 5 84bb9faa4f
CI (SIAX Cloud) / security (push) Successful in 41s
Security Scan (SIAX Cloud) / security (push) Successful in 40s
CI (SIAX Cloud) / integration (push) Successful in 52s
CI (SIAX Cloud) / quality (push) Successful in 1m5s
CI (SIAX Cloud) / security (pull_request) Successful in 31s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 29s
CI (SIAX Cloud) / integration (pull_request) Successful in 45s
CI (SIAX Cloud) / quality (pull_request) Successful in 53s
feat(analytics): BI-P002..BI-P012 analytics core — domain model, tiers, contracts, Arrow, ClickHouse engine, isolation, governance
Implements the eight BI program prompts that make 0pt the canonical SIAX
analytics capability. Every file is NEW: not one existing file is modified,
and src/analytics/clickhouse-analytics.ts is byte-identical to main.

BI-P002 — the nineteen canonical entities, verbatim, with tenant_id +
project_id on every tenant-owned record. sql/008 gives each one a table.
The four version entities BI-P002 names as immutable are enforced by BEFORE
UPDATE/DELETE triggers, not by a TypeScript `readonly` — proven by direct SQL
against a live Postgres.

BI-P003 — RAW/NORMALIZED/CURATED/SERVING with the read rule enforced:
assertReadableByBiSurface refuses a RAW dataset to a dashboard unless a
recorded exception exists. All eight required declarations are NOT NULL.

BI-P004 — DatasetContract and a fail-closed compatibility gate. Fifteen
breaking-change classes are each tested twice: refused under a minor bump,
accepted under a major one.

BI-P005 — Arrow IPC alongside JSON, with a hand-written FlatBuffers writer so
no npm dependency (or CVE surface) is added. Conformance comes from three
independent implementations, not from self-agreement: pyarrow 25.0.1 and
DuckDB 1.5.5 read identical values from both ClickHouse's Arrow and ours, and
ClickHouse's own Arrow C++ ingests our 20,000-row batch with exact aggregates.

BI-P009 — SiaxAnalyticsEngine with all eight capabilities. The existing
ClickHouseAnalytics is COMPOSED, never replaced. executeQuery (analyst SQL) is
declared and deliberately refused with CAPABILITY_UNAVAILABLE until BI-P017 and
BI-P033 land, rather than opening the raw-SQL hole BI-P011 enumerates.

BI-P010 — four purpose-built tables, not one giant one, and the difference is
measured against the counterfactual: 48.8ms/40,960 rows vs 174.4ms/5,400,000
rows for the same question; 5/660 granules vs 660/1149.

BI-P011 — layered isolation, each layer proven with the layer above removed.
Without the forced predicate the same statement returns 50,050 (both tenants);
with it, t_alpha gets 50. Without a row policy an unrestricted user sees 2
tenants; the restricted user running the IDENTICAL statement sees 1.

BI-P012 — every named limit, fired individually and identified by ClickHouse
error code, plus the half usually skipped: an unrelated tenant is timed while
a real CPU bomb is in flight and must still return the CORRECT answer.

Three defects found by writing tests that can fail:
  * `CHECK (array_length(purposes,1) >= 1)` can NEVER fail — array_length is
    NULL for '{}' and a NULL CHECK passes. Fixed with coalesce(); the test
    that caught it is pinned.
  * EXPLAIN ESTIMATE run under the caller's own max_rows_to_read fails with
    code 158 exactly when the estimate would be over budget. Planner calls now
    use a narrower envelope.
  * result_overflow_mode=break returns HTTP 200 with 65,409 of 100,000 rows
    and no indication it is short. Every SELECT ships `throw`.

Verified rather than assumed: server7's ClickHouse holds `opt_scraped` (51
tables, 407,793,471 rows, `persons`/`cz_owners` included) — the b0ts plane this
module is prohibited from touching — has no `opt_analytics` database, zero
tenant_id columns and zero row policies. Nothing there was modified; every
proof ran against disposable containers.

Evidence: evidence/bi-p002-p012/{test-transcript.txt,bi-p010-benchmark.json,
bi-p005-arrow-roundtrip.json,server7-clickhouse-probe.md}
Docs: docs/analytics/BI_ANALYTICS_CORE.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-06 23:01:27 +02:00

537 lines
20 KiB
TypeScript

/**
* src/analytics/domain/entities.ts
* =================================
* BI-P002 — ANALYTICS DOMAIN MODEL.
*
* The nineteen canonical analytics entities, named EXACTLY as the program
* lists them (no renaming, no merging, no "we already have something like
* that"):
*
* AnalyticsDataset DatasetVersion DatasetField
* DatasetGrant MetricDefinition MetricVersion
* DimensionDefinition SemanticModel AnalyticsQuery
* QueryExecution QueryResult DashboardDefinition
* DashboardVersion ReportDefinition ReportRun
* DataQualityRule DataQualityRun LineageReference
* BIApplicationBinding
*
* Every tenant-owned record carries `tenant_id` + `project_id` where
* applicable. The four version entities the program names as immutable
* (MetricVersion, SemanticModel version, DashboardVersion, DatasetVersion)
* are `readonly` here AND immutable in the database — sql/008 installs BEFORE
* UPDATE/DELETE triggers, because a TypeScript `readonly` is a compiler
* opinion, not a storage guarantee.
*
* NAME COLLISION, deliberate: src/services/dataset-version-service.ts already
* exports an interface called `DatasetVersion` for the ST0RE-artifact-backed
* dataset versioning that predates this program. That one is about *dataset
* content artifacts*; this one is about *analytics dataset schema versions*
* and lives in its own module namespace (`AnalyticsDatasetVersion` is NOT
* used as the name — BI-P002 lists `DatasetVersion` verbatim and that is what
* ships). Import them qualified when both are needed in one file.
*
* ARCHITECTURE BOUNDARY: these entities are owned by 0pt. A product repo
* (VehIQ) supplies domain MEANING through `BIApplicationBinding`; it never
* defines its own copies of these tables and never receives engine
* credentials.
*/
import type { AnalyticsTier } from "./tiers";
/** Every tenant-owned analytics record. */
export interface TenantScoped {
tenant_id: string;
project_id: string;
}
// ── field/typing vocabulary ────────────────────────────────────────────────
export const ANALYTICS_FIELD_TYPES = [
"string",
"int64",
"float64",
"bool",
"date",
"timestamp",
"decimal",
"json",
] as const;
export type AnalyticsFieldType = (typeof ANALYTICS_FIELD_TYPES)[number];
/** A field's analytical role. Measures aggregate; dimensions group. */
export const FIELD_ROLES = ["identifier", "dimension", "measure", "attribute"] as const;
export type FieldRole = (typeof FIELD_ROLES)[number];
/** PII classification, aligned with the SIAX risk vocabulary (L0 = none). */
export const PII_CLASSIFICATIONS = ["none", "pseudonymous", "personal", "sensitive"] as const;
export type PiiClassification = (typeof PII_CLASSIFICATIONS)[number];
// ── 1. AnalyticsDataset ────────────────────────────────────────────────────
export const DATASET_STATUSES = ["draft", "active", "deprecated", "retired"] as const;
export type AnalyticsDatasetStatus = (typeof DATASET_STATUSES)[number];
/**
* BI-P003 DoD: "Every dataset declares tier, schema, owner, source, freshness,
* quality, retention, classification." All eight are REQUIRED fields here —
* an optional declaration is not a declaration.
*/
export interface AnalyticsDataset extends TenantScoped {
dataset_id: string;
name: string;
/** BI-P003 tier. */
tier: AnalyticsTier;
/** Contract id + version this dataset's schema conforms to (BI-P004). */
schema_contract_id: string;
schema_contract_version: string;
/** Accountable human/team, not the person who typed the create call. */
owner_id: string;
/** Upstream: a 0pt source_id, another dataset_id, or an external ref. */
source_ref: string;
/** Freshness SLO in seconds — how stale the data may be before it is late. */
freshness_slo_seconds: number;
/** DataQualityRule ids that gate this dataset. */
quality_rule_ids: string[];
/** Retention in days. 0 = keep indefinitely (must be justified in policy). */
retention_days: number;
classification: PiiClassification;
status: AnalyticsDatasetStatus;
/** BI-P003 escape hatch for RAW/NORMALIZED exposure to a BI surface. */
raw_access_exception?: string;
description?: string;
created_at: string;
updated_at?: string;
created_by?: string;
}
// ── 2. DatasetVersion (IMMUTABLE) ──────────────────────────────────────────
export interface DatasetVersion extends TenantScoped {
readonly dataset_version_id: string;
readonly dataset_id: string;
/** Monotonic per dataset, starting at 1. */
readonly version: number;
/** The frozen DatasetContract JSON at this version (BI-P004). */
readonly contract_hash: string;
readonly schema_contract_version: string;
readonly record_count?: number;
readonly created_at: string;
readonly created_by?: string;
}
// ── 3. DatasetField ────────────────────────────────────────────────────────
export interface DatasetField extends TenantScoped {
dataset_field_id: string;
dataset_id: string;
/** The dataset version this field description belongs to. */
dataset_version: number;
name: string;
field_type: AnalyticsFieldType;
nullable: boolean;
role: FieldRole;
/** True for the field(s) forming the dataset's primary identifier. */
primary_identifier: boolean;
classification: PiiClassification;
description?: string;
}
// ── 4. DatasetGrant ────────────────────────────────────────────────────────
export const GRANT_LEVELS = ["read", "query", "export", "admin"] as const;
export type GrantLevel = (typeof GRANT_LEVELS)[number];
/**
* Who may use a dataset, and FOR WHAT. `allowed_purposes` is the GDPR purpose
* limitation carried into analytics — a grant without a purpose is refused
* by the repository layer.
*/
export interface DatasetGrant extends TenantScoped {
grant_id: string;
dataset_id: string;
/** Subject: a CL0UD principal, role, or BI application binding id. */
subject_id: string;
subject_type: "user" | "role" | "service" | "bi_application";
level: GrantLevel;
allowed_purposes: string[];
expires_at?: string;
created_at: string;
created_by?: string;
}
// ── 5. MetricDefinition ────────────────────────────────────────────────────
export const METRIC_AGGREGATIONS = [
"sum",
"avg",
"min",
"max",
"count",
"count_distinct",
"ratio",
"derived",
] as const;
export type MetricAggregation = (typeof METRIC_AGGREGATIONS)[number];
export interface MetricDefinition extends TenantScoped {
metric_id: string;
/** Stable business name; unique per tenant. */
name: string;
dataset_id: string;
aggregation: MetricAggregation;
/** Measure field aggregated, or the numerator for ratio metrics. */
measure_field: string;
/** Denominator for ratio metrics. */
denominator_field?: string;
/** Dimensions this metric may legitimately be sliced by. */
allowed_dimension_ids: string[];
/** Current live version number (points at a MetricVersion). */
current_version: number;
owner_id: string;
description?: string;
created_at: string;
updated_at?: string;
}
// ── 6. MetricVersion (IMMUTABLE) ───────────────────────────────────────────
export interface MetricVersion extends TenantScoped {
readonly metric_version_id: string;
readonly metric_id: string;
readonly version: number;
/** Frozen definition JSON — what this metric MEANT at this version. */
readonly definition_json: string;
readonly definition_hash: string;
/** Free-text reason the metric changed; required for version > 1. */
readonly change_reason?: string;
readonly created_at: string;
readonly created_by?: string;
}
// ── 7. DimensionDefinition ─────────────────────────────────────────────────
export interface DimensionDefinition extends TenantScoped {
dimension_id: string;
name: string;
dataset_id: string;
/**
* Physical column/key this dimension resolves to. This is the ONLY place a
* caller-supplied dimension name becomes a physical identifier — the query
* compiler resolves through here and never splices caller input (BI-P011
* "dimension identifier injection").
*/
field_name: string;
field_type: AnalyticsFieldType;
classification: PiiClassification;
description?: string;
created_at: string;
}
// ── 8. SemanticModel (versioned; versions IMMUTABLE) ───────────────────────
export interface SemanticModel extends TenantScoped {
semantic_model_id: string;
name: string;
/** Immutable version number — a change mints a new row, never an UPDATE. */
readonly version: number;
/** MetricFlow/Cube-compatible model document (BI-P029/BI-P032 consume it). */
readonly model_json: string;
readonly model_hash: string;
dataset_ids: string[];
owner_id: string;
readonly created_at: string;
readonly created_by?: string;
}
// ── 9. AnalyticsQuery ──────────────────────────────────────────────────────
export const QUERY_SURFACES = ["metric", "sql", "projection"] as const;
export type QuerySurface = (typeof QUERY_SURFACES)[number];
/**
* A SAVED, reusable query. The request that actually runs is
* `src/analytics/engine/query-compiler.ts`'s `MetricQueryRequest`; this entity
* is its durable, governed form.
*/
export interface AnalyticsQuery extends TenantScoped {
query_id: string;
name: string;
surface: QuerySurface;
dataset_id: string;
metric_id?: string;
/** Serialized closed-DSL request. Never raw SQL for surface="metric". */
request_json: string;
owner_id: string;
created_at: string;
updated_at?: string;
}
// ── 10. QueryExecution ─────────────────────────────────────────────────────
export const EXECUTION_STATES = [
"queued",
"running",
"succeeded",
"failed",
"cancelled",
"rejected",
] as const;
export type ExecutionState = (typeof EXECUTION_STATES)[number];
export interface QueryExecution extends TenantScoped {
execution_id: string;
query_id?: string;
/** Engine-side identifier used to CANCEL this execution (BI-P009). */
engine_query_id: string;
state: ExecutionState;
/** Which provider ran it: "clickhouse" | "duckdb" | "trino". */
provider: string;
submitted_at: string;
started_at?: string;
finished_at?: string;
duration_ms?: number;
rows_read?: number;
bytes_read?: number;
result_rows?: number;
/** SiaxErrorCode when state="failed"/"rejected". Never a raw engine body. */
error_code?: string;
correlation_id?: string;
}
// ── 11. QueryResult ────────────────────────────────────────────────────────
export const RESULT_ENCODINGS = ["json", "arrow_ipc"] as const;
export type ResultEncoding = (typeof RESULT_ENCODINGS)[number];
export interface QueryResult extends TenantScoped {
result_id: string;
execution_id: string;
encoding: ResultEncoding;
row_count: number;
/** Canonical ST0RE ref when materialised; inline results carry no bytes here. */
artifact_ref?: string;
/** True when the engine truncated at max_result_rows (BI-P012). */
truncated: boolean;
expires_at?: string;
created_at: string;
}
// ── 12. DashboardDefinition ────────────────────────────────────────────────
export interface DashboardDefinition extends TenantScoped {
dashboard_id: string;
name: string;
/** Live version pointer. */
current_version: number;
owner_id: string;
/** Datasets the dashboard reads — checked against BI-P003 tier rules. */
dataset_ids: string[];
description?: string;
created_at: string;
updated_at?: string;
}
// ── 13. DashboardVersion (IMMUTABLE) ───────────────────────────────────────
export interface DashboardVersion extends TenantScoped {
readonly dashboard_version_id: string;
readonly dashboard_id: string;
readonly version: number;
/** Frozen layout/panel document. */
readonly layout_json: string;
readonly layout_hash: string;
readonly created_at: string;
readonly created_by?: string;
}
// ── 14. ReportDefinition ───────────────────────────────────────────────────
export const REPORT_FORMATS = ["pdf", "csv", "xlsx", "parquet", "arrow_ipc"] as const;
export type ReportFormat = (typeof REPORT_FORMATS)[number];
export interface ReportDefinition extends TenantScoped {
report_id: string;
name: string;
dashboard_id?: string;
query_id?: string;
format: ReportFormat;
/** Cron expression; undefined = on-demand only. */
schedule_cron?: string;
owner_id: string;
created_at: string;
updated_at?: string;
}
// ── 15. ReportRun ──────────────────────────────────────────────────────────
export interface ReportRun extends TenantScoped {
report_run_id: string;
report_id: string;
state: ExecutionState;
started_at: string;
finished_at?: string;
/** ST0RE artifact ref of the produced report. */
artifact_ref?: string;
error_code?: string;
correlation_id?: string;
}
// ── 16. DataQualityRule ────────────────────────────────────────────────────
export const QUALITY_RULE_KINDS = [
"not_null",
"unique",
"accepted_values",
"range",
"freshness",
"row_count",
"referential",
"custom",
] as const;
export type QualityRuleKind = (typeof QUALITY_RULE_KINDS)[number];
export const QUALITY_SEVERITIES = ["warn", "block"] as const;
export type QualitySeverity = (typeof QUALITY_SEVERITIES)[number];
export interface DataQualityRule extends TenantScoped {
rule_id: string;
dataset_id: string;
kind: QualityRuleKind;
/** Field the rule applies to; omitted for dataset-level rules. */
field_name?: string;
/** Rule parameters (bounds, accepted values, max age). */
params_json: string;
severity: QualitySeverity;
created_at: string;
created_by?: string;
}
// ── 17. DataQualityRun ─────────────────────────────────────────────────────
export interface DataQualityRun extends TenantScoped {
quality_run_id: string;
rule_id: string;
dataset_id: string;
dataset_version?: number;
passed: boolean;
/** Rows that violated the rule. */
violation_count: number;
evaluated_rows: number;
started_at: string;
finished_at?: string;
correlation_id?: string;
}
// ── 18. LineageReference ───────────────────────────────────────────────────
export const LINEAGE_DIRECTIONS = ["input", "output"] as const;
export type LineageDirection = (typeof LINEAGE_DIRECTIONS)[number];
/**
* The queryable projection of an OpenLineage edge (BI-P027). AUD0 stays the
* system of record; this is 0pt's local index so "what fed this dashboard"
* is answerable without leaving the analytics plane.
*/
export interface LineageReference extends TenantScoped {
lineage_reference_id: string;
/** OpenLineage run id this edge came from. */
run_id: string;
job_name: string;
direction: LineageDirection;
/** The analytics entity this edge touches. */
entity_type: "dataset" | "metric" | "dashboard" | "report";
entity_id: string;
/** Namespaced dataset name on the other side of the edge. */
counterpart_ref: string;
observed_at: string;
}
// ── 19. BIApplicationBinding ───────────────────────────────────────────────
export const BI_APPLICATION_KINDS = ["superset", "evidence", "des1gn", "notebook", "custom"] as const;
export type BiApplicationKind = (typeof BI_APPLICATION_KINDS)[number];
/**
* How an external BI surface is attached to the analytics plane.
*
* NORMATIVE (BI program §0 + the SIAX/0pt ownership boundary): this binding
* carries NO engine credential. A bound application authenticates to the 0pt
* Analytics API with a CL0UD identity and receives compiled, tenant-forced
* results — it never learns a ClickHouse DSN, user, or password. The absence
* of a credential field here is the enforcement, not a comment.
*/
export interface BIApplicationBinding extends TenantScoped {
binding_id: string;
kind: BiApplicationKind;
name: string;
/** CL0UD principal the application authenticates as. */
principal_id: string;
/** Datasets/metrics this application may reach at all. */
dataset_ids: string[];
metric_ids: string[];
allowed_purposes: string[];
enabled: boolean;
created_at: string;
created_by?: string;
}
// ── registry of the canonical nineteen ─────────────────────────────────────
/**
* The canonical entity names, verbatim from BI-P002. Persisted-schema tests
* assert sql/008 defines a table for every one of these, so a future rename
* or omission is a test failure rather than drift.
*/
export const ANALYTICS_DOMAIN_ENTITIES = [
"AnalyticsDataset",
"DatasetVersion",
"DatasetField",
"DatasetGrant",
"MetricDefinition",
"MetricVersion",
"DimensionDefinition",
"SemanticModel",
"AnalyticsQuery",
"QueryExecution",
"QueryResult",
"DashboardDefinition",
"DashboardVersion",
"ReportDefinition",
"ReportRun",
"DataQualityRule",
"DataQualityRun",
"LineageReference",
"BIApplicationBinding",
] as const;
export type AnalyticsDomainEntity = (typeof ANALYTICS_DOMAIN_ENTITIES)[number];
/** Entity -> physical table in sql/008_analytics_domain_model.sql. */
export const ANALYTICS_ENTITY_TABLES: Record<AnalyticsDomainEntity, string> = {
AnalyticsDataset: "opt_analytics_dataset",
DatasetVersion: "opt_analytics_dataset_version",
DatasetField: "opt_analytics_dataset_field",
DatasetGrant: "opt_analytics_dataset_grant",
MetricDefinition: "opt_analytics_metric_definition",
MetricVersion: "opt_analytics_metric_version",
DimensionDefinition: "opt_analytics_dimension_definition",
SemanticModel: "opt_analytics_semantic_model",
AnalyticsQuery: "opt_analytics_query",
QueryExecution: "opt_analytics_query_execution",
QueryResult: "opt_analytics_query_result",
DashboardDefinition: "opt_analytics_dashboard_definition",
DashboardVersion: "opt_analytics_dashboard_version",
ReportDefinition: "opt_analytics_report_definition",
ReportRun: "opt_analytics_report_run",
DataQualityRule: "opt_analytics_data_quality_rule",
DataQualityRun: "opt_analytics_data_quality_run",
LineageReference: "opt_analytics_lineage_reference",
BIApplicationBinding: "opt_analytics_bi_application_binding",
};
/** The four entities BI-P002 requires to be immutable once written. */
export const IMMUTABLE_VERSION_ENTITIES: readonly AnalyticsDomainEntity[] = [
"DatasetVersion",
"MetricVersion",
"SemanticModel",
"DashboardVersion",
];