feat(analytics): BI-P022 — dbt canonical transformation CONTRACT and reference project
CI (SIAX Cloud) / security (push) Successful in 22s
CI (SIAX Cloud) / integration (push) Successful in 38s
CI (SIAX Cloud) / quality (push) Successful in 52s
Security Scan (SIAX Cloud) / security (push) Successful in 20s
CI (SIAX Cloud) / security (pull_request) Successful in 22s
CI (SIAX Cloud) / integration (pull_request) Successful in 37s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 20s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m1s

Scope stated in the module's first paragraph: this defines the contract, it
does not run dbt, and nothing in this repo does. Execution is N0D's (n0d#38),
which consumes this from the other side. There is deliberately no DbtExecutor
implementation here — only the interface — so "0pt can run dbt" cannot become
true by accident and the two repos share one versioned shape.

`analytics/dbt/` is a real reference project covering raw -> staging ->
curated -> serving, with the four layers mapped onto BI-P003's existing tiers
rather than inventing a fifth vocabulary. The tenant predicate is applied at
the FIRST modelled layer so every downstream model inherits it and no later
model has to remember — the same forced-predicate rule as the query compiler's,
one layer up.

Two singular tests ship with it, both aimed at false greens:
  * `assert_mart_is_not_empty` — without it every column test passes trivially
    over zero rows and the whole suite reports green while the pipeline is dead;
  * `assert_no_cross_tenant_rows` — which should be impossible to fail, and
    exists precisely for that reason. An isolation guarantee nobody tests is the
    one that turns out to have four holes rather than the one reported.

BI-P022's version rule is a GATE. `DBT_RUNTIME_PIN` records dbt Core 1.10.13
(stable) with dbt-clickhouse 1.9.2 and BOTH review flags FALSE — no licence or
supply-chain review has been performed in this repo and none is claimed.
`assertDbtRuntimeCertified` refuses a production run until they are true, and
refuses an alpha line in production without a separate certification reference.
A version constant on its own would be the documented-but-unenforced pattern;
a test pins the flags at false so flipping them is a visible deliberate act.

The result is treated as UNTRUSTED INPUT, because it arrives from another
repo's executor and this estate has been burned by green statuses that meant
nothing. Refused: `status: success` with ZERO nodes executed (a dbt selector
matching nothing exits 0 — the dbt-shaped false green); an observed runtime
different from the pin (the models mean something else and nothing says so); a
revision that is not an exact 40-char SHA (a branch name is not reproducible,
and reproducibility is the DoD); a missing manifest.json or run_results.json,
without which the run cannot be audited or its tests ingested by BI-P025.

Layer dependencies are checked against the manifest GRAPH, not asserted as a
convention: a mart reading a source directly, or skipping a layer, or a
staging model reading a mart, are each refused.

Selectors are validated rather than passed through — a free-form argument list
is a shell surface, and a shell surface on a job holding warehouse credentials
is the whole problem. The job also cannot name an arbitrary target schema: it
must be scoped to the job's tenant.

39 tests. typecheck + eslint clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
2026-09-07 02:45:41 +02:00
parent 6cb00add57
commit ec26ddc296
14 changed files with 975 additions and 0 deletions
+374
View File
@@ -0,0 +1,374 @@
/**
* src/analytics/transform/dbt-contract.ts
* ========================================
* BI-P022 — DBT CORE CANONICAL TRANSFORMATION.
*
* ── SCOPE, STATED FIRST ───────────────────────────────────────────────────
* This module defines the CONTRACT. It does not run dbt, and nothing in this
* repo does. Execution belongs to N0D (n0d#38), which consumes this contract
* from the other side. There is deliberately no `DbtExecutor` implementation
* here — only the interface — so that "0pt can run dbt" cannot become true by
* accident, and so the two repos have one shared, versioned shape rather than
* two drifting ones.
*
* ── THE VERSION RULE BI-P022 STATES ───────────────────────────────────────
* "Use stable dbt Core production line AFTER current license/security
* verification. Do NOT use alpha development engine in production unless
* separately certified."
*
* So the pin is recorded together with its certification STATE, and the state
* is `unverified` until someone does the review. `assertDbtRuntimeCertified`
* refuses a production run against an uncertified or alpha runtime — which
* makes the sentence above a gate rather than a paragraph. A version constant
* on its own would be exactly the "documented but unenforced" pattern.
*
* ── THE TIER MAPPING ──────────────────────────────────────────────────────
* BI-P022's DoD names raw → staging → curated → serving. BI-P003 (merged)
* already owns RAW/NORMALIZED/CURATED/SERVING, so the dbt layers map onto it
* rather than inventing a fifth vocabulary:
*
* sources/ -> RAW (declared, never modelled)
* staging/ -> NORMALIZED (rename, cast, deduplicate; no business logic)
* intermediate/ -> CURATED (joins, business logic, conformed entities)
* marts/ -> SERVING (what a dashboard or a metric reads)
*
* A model in the wrong directory for its tier is a rejection, not a warning:
* the tier is what BI-P003's read rule gates on, and a mart that is really a
* staging model is how RAW data reaches a dashboard.
*/
import { SiaxError } from '../../platform/errors';
import type { AnalyticsTier } from '../domain/tiers';
/** dbt layers, in dependency order. */
export const DBT_LAYERS = ['sources', 'staging', 'intermediate', 'marts'] as const;
export type DbtLayer = (typeof DBT_LAYERS)[number];
export const DBT_LAYER_TIERS: Record<DbtLayer, AnalyticsTier> = {
sources: 'RAW',
staging: 'NORMALIZED',
intermediate: 'CURATED',
marts: 'SERVING',
};
/** Directories BI-P022 requires under `analytics/dbt/`. */
export const REQUIRED_DBT_DIRECTORIES = [
'models/sources',
'models/staging',
'models/intermediate',
'models/marts',
'tests',
'snapshots',
] as const;
// ── runtime certification ──────────────────────────────────────────────────
export type DbtReleaseLine = 'stable' | 'prerelease' | 'alpha';
export interface DbtRuntimePin {
/** e.g. "1.10.13". Exact, never a range. */
version: string;
line: DbtReleaseLine;
/** Adapter and its pin, e.g. { name: "dbt-clickhouse", version: "1.9.2" }. */
adapter: { name: string; version: string };
/** Has a licence + supply-chain review been completed FOR THIS PIN? */
licenseReviewed: boolean;
securityReviewed: boolean;
reviewedBy?: string;
reviewedAt?: string;
/** Required by BI-P022 to run an alpha line in production at all. */
separateCertificationRef?: string;
}
/**
* The pin this contract is written against.
*
* BOTH review flags are FALSE. No licence or supply-chain review of dbt Core
* has been performed in this repo, and none is claimed. The constant records
* an intended pin; `assertDbtRuntimeCertified` is what stops it being treated
* as an approval. Flipping these to true without doing the review is the
* failure this shape exists to make visible.
*/
export const DBT_RUNTIME_PIN: DbtRuntimePin = {
version: '1.10.13',
line: 'stable',
adapter: { name: 'dbt-clickhouse', version: '1.9.2' },
licenseReviewed: false,
securityReviewed: false,
};
export type DbtRunEnvironment = 'development' | 'ci' | 'production';
/** BI-P022's version rule, as a gate. */
export function assertDbtRuntimeCertified(pin: DbtRuntimePin, environment: DbtRunEnvironment): void {
if (!/^\d+\.\d+\.\d+$/.test(pin.version)) {
throw new SiaxError(
'INVALID_REQUEST',
'0pt.analytics.dbt',
'the dbt runtime pin must be an exact version',
);
}
if (environment !== 'production') return;
if (pin.line === 'alpha' && !pin.separateCertificationRef) {
throw new SiaxError(
'ENTITLEMENT_REQUIRED',
'0pt.analytics.dbt',
'BI-P022: an alpha dbt engine may not run in production without a separate certification reference',
);
}
if (!pin.licenseReviewed || !pin.securityReviewed) {
throw new SiaxError(
'ENTITLEMENT_REQUIRED',
'0pt.analytics.dbt',
`BI-P022: dbt ${pin.version} has not completed licence and security verification; ` +
`it may not run in production`,
);
}
}
// ── the job contract N0D executes ──────────────────────────────────────────
export const DBT_COMMANDS = [
'build',
'run',
'test',
'snapshot',
'seed',
'compile',
'source_freshness',
] as const;
export type DbtCommand = (typeof DBT_COMMANDS)[number];
/**
* What 0pt hands N0D. Deliberately narrow: there is no free-form `args` field,
* because a passthrough argument list is a shell surface, and a shell surface
* on a job that holds warehouse credentials is the whole problem.
*/
export interface DbtJobRequest {
/** Verified, from CL0UD. Never read from a request body. */
tenant_id: string;
project_id: string;
/** Which analytics project to run — a registered name, not a path. */
dbt_project: string;
/** Git ref of the analytics repo the models come from. Exact SHA. */
revision: string;
command: DbtCommand;
/** dbt node selectors. Validated; never concatenated into a shell string. */
select?: string[];
exclude?: string[];
/** Target layer, which N0D maps to a credential with matching write scope. */
target_layer: DbtLayer;
environment: DbtRunEnvironment;
runtime: DbtRuntimePin;
/** Idempotency key, per the standard N0D job contract. */
idempotency_key: string;
correlation_id?: string;
/** Fail the run if it would write outside this tenant's schema. */
expected_target_schema: string;
}
/** The artefacts N0D must return. dbt produces all of them; they are not optional. */
export const REQUIRED_DBT_ARTIFACTS = ['manifest.json', 'run_results.json'] as const;
export interface DbtJobResult {
tenant_id: string;
dbt_project: string;
revision: string;
command: DbtCommand;
status: 'success' | 'error' | 'partial';
started_at: string;
finished_at: string;
/** ST0RE artifact refs, one per REQUIRED_DBT_ARTIFACTS entry. */
artifacts: Record<string, string>;
/** Per-node outcomes, the input to BI-P025's DbtQualityProvider. */
nodes: DbtNodeResult[];
/** dbt's own version, as reported by the runtime that actually ran. */
observed_dbt_version: string;
}
export interface DbtNodeResult {
unique_id: string;
resource_type: 'model' | 'test' | 'snapshot' | 'seed' | 'source';
status: 'success' | 'error' | 'skipped' | 'pass' | 'fail' | 'warn';
rows_affected?: number;
execution_time_s?: number;
message?: string;
}
const SELECTOR_RE = /^[A-Za-z0-9_.:*+@ -]{1,200}$/;
const NAME_RE = /^[a-z][a-z0-9_]{1,62}$/;
const SHA_RE = /^[0-9a-f]{40}$/;
export function assertValidDbtJobRequest(req: DbtJobRequest): void {
if (!req.tenant_id?.trim()) {
throw new SiaxError('UNAUTHENTICATED', '0pt.analytics.dbt', 'a dbt job requires a verified tenant');
}
if (!NAME_RE.test(req.dbt_project)) {
throw new SiaxError(
'INVALID_REQUEST',
'0pt.analytics.dbt',
'dbt_project must be a registered snake_case name',
);
}
if (!SHA_RE.test(req.revision)) {
// A branch name is not reproducible, and BI-P022's DoD is about
// reproducibility. "main" today and "main" tomorrow are different projects.
throw new SiaxError(
'INVALID_REQUEST',
'0pt.analytics.dbt',
'revision must be an exact 40-character commit SHA — a branch name is not reproducible',
);
}
if (!(DBT_COMMANDS as readonly string[]).includes(req.command)) {
throw new SiaxError('INVALID_REQUEST', '0pt.analytics.dbt', `unsupported dbt command "${req.command}"`);
}
if (!(DBT_LAYERS as readonly string[]).includes(req.target_layer)) {
throw new SiaxError('INVALID_REQUEST', '0pt.analytics.dbt', `unknown target layer "${req.target_layer}"`);
}
if (req.target_layer === 'sources') {
throw new SiaxError(
'INVALID_REQUEST',
'0pt.analytics.dbt',
'sources are DECLARED, never built — a job cannot target the sources layer',
);
}
for (const selector of [...(req.select ?? []), ...(req.exclude ?? [])]) {
if (!SELECTOR_RE.test(selector)) {
throw new SiaxError(
'INVALID_REQUEST',
'0pt.analytics.dbt',
`dbt selector "${selector.slice(0, 40)}" contains characters that are not selector syntax`,
);
}
}
if (!req.idempotency_key?.trim()) {
throw new SiaxError(
'INVALID_REQUEST',
'0pt.analytics.dbt',
'the N0D job contract requires an idempotency key',
);
}
if (!req.expected_target_schema.includes(req.tenant_id)) {
// The target schema must be derivable from the tenant. A job that can name
// an arbitrary schema is a cross-tenant write waiting to happen.
throw new SiaxError(
'AUTHORIZATION_DENIED',
'0pt.analytics.dbt',
"expected_target_schema must be scoped to the job's tenant",
);
}
assertDbtRuntimeCertified(req.runtime, req.environment);
}
/**
* Validate what came BACK. A result is untrusted input: it arrives from
* another repo's executor, and the estate has been burned by trusting a green
* status that meant nothing.
*/
export function assertValidDbtJobResult(result: DbtJobResult, request: DbtJobRequest): void {
if (result.tenant_id !== request.tenant_id) {
throw new SiaxError('AUTHORIZATION_DENIED', '0pt.analytics.dbt', 'dbt result is for a different tenant');
}
if (result.revision !== request.revision) {
throw new SiaxError(
'CONFLICT',
'0pt.analytics.dbt',
'dbt result is for a different revision than was requested — the run is not reproducible',
);
}
for (const artifact of REQUIRED_DBT_ARTIFACTS) {
if (!result.artifacts[artifact]) {
throw new SiaxError(
'PERMANENT_FAILURE',
'0pt.analytics.dbt',
`dbt result is missing "${artifact}" — without it the run cannot be audited or its tests ingested`,
);
}
}
if (result.observed_dbt_version !== request.runtime.version) {
// A runtime that is not the pinned one silently changes what the models
// mean. Catching it here is the difference between a pin and a wish.
throw new SiaxError(
'CONFLICT',
'0pt.analytics.dbt',
`dbt ran as ${result.observed_dbt_version} but ${request.runtime.version} was pinned`,
);
}
if (result.status === 'success' && result.nodes.length === 0) {
// A "successful" run that built nothing is the dbt-shaped false green: a
// selector that matched no models exits 0.
throw new SiaxError(
'PERMANENT_FAILURE',
'0pt.analytics.dbt',
'dbt reported success but executed zero nodes — a selector that matches nothing exits 0',
);
}
const failed = result.nodes.filter((n) => n.status === 'error' || n.status === 'fail');
if (result.status === 'success' && failed.length > 0) {
throw new SiaxError(
'PERMANENT_FAILURE',
'0pt.analytics.dbt',
`dbt reported success while ${failed.length} node(s) failed`,
);
}
}
/**
* The executor N0D implements. Declared so both repos compile against one
* shape; deliberately NOT implemented here.
*/
export interface DbtExecutor {
readonly runsIn: 'n0d';
execute(request: DbtJobRequest): Promise<DbtJobResult>;
}
/** Which tier a model path belongs to, or a rejection. */
export function tierForModelPath(modelPath: string): AnalyticsTier {
const match = /(?:^|\/)models\/(sources|staging|intermediate|marts)\//.exec(modelPath);
if (!match) {
throw new SiaxError(
'INVALID_REQUEST',
'0pt.analytics.dbt',
`model "${modelPath}" is not under one of ${DBT_LAYERS.join('/')} — BI-P003 cannot assign it a tier`,
);
}
return DBT_LAYER_TIERS[match[1] as DbtLayer];
}
/**
* A staging model must not reach past sources, a mart must not read a source
* directly, and nothing may skip a layer. Enforced on the dependency graph
* from `manifest.json`, because "we follow the convention" is not a check.
*/
const LAYER_ORDER: DbtLayer[] = ['sources', 'staging', 'intermediate', 'marts'];
export function assertLayerDependenciesLegal(edges: Array<{ from: string; to: string }>): void {
for (const edge of edges) {
const fromLayer = layerOf(edge.from);
const toLayer = layerOf(edge.to);
const fromIdx = LAYER_ORDER.indexOf(fromLayer);
const toIdx = LAYER_ORDER.indexOf(toLayer);
if (toIdx > fromIdx) {
throw new SiaxError(
'INVALID_REQUEST',
'0pt.analytics.dbt',
`"${edge.from}" (${fromLayer}) depends on "${edge.to}" (${toLayer}) — a model may not read a later layer`,
);
}
if (fromIdx - toIdx > 1) {
throw new SiaxError(
'INVALID_REQUEST',
'0pt.analytics.dbt',
`"${edge.from}" (${fromLayer}) skips a layer to read "${edge.to}" (${toLayer}) — ` +
`raw data reaching a mart directly is what BI-P003's tier rule exists to prevent`,
);
}
}
}
function layerOf(path: string): DbtLayer {
const match = /(?:^|\/)models\/(sources|staging|intermediate|marts)\//.exec(path);
if (!match) {
throw new SiaxError('INVALID_REQUEST', '0pt.analytics.dbt', `model "${path}" is not in a known layer`);
}
return match[1] as DbtLayer;
}