feat(analytics): BI-P025/BI-P026 — quality providers that fail on empty, freshness that refuses to be silent
CI (SIAX Cloud) / integration (push) Successful in 43s
CI (SIAX Cloud) / security (push) Successful in 23s
CI (SIAX Cloud) / quality (push) Successful in 57s
Security Scan (SIAX Cloud) / security (push) Successful in 21s
CI (SIAX Cloud) / security (pull_request) Successful in 23s
CI (SIAX Cloud) / integration (pull_request) Successful in 40s
CI (SIAX Cloud) / quality (pull_request) Successful in 55s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 25s
CI (SIAX Cloud) / integration (push) Successful in 43s
CI (SIAX Cloud) / security (push) Successful in 23s
CI (SIAX Cloud) / quality (push) Successful in 57s
Security Scan (SIAX Cloud) / security (push) Successful in 21s
CI (SIAX Cloud) / security (pull_request) Successful in 23s
CI (SIAX Cloud) / integration (pull_request) Successful in 40s
CI (SIAX Cloud) / quality (pull_request) Successful in 55s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 25s
Both prompts are about a FALSE GREEN, so both are tested by constructing the false green and requiring a failure. BI-P025. Three provider kinds, and the interface makes the difference visible instead of implying parity: `providerKind` is "executing" or "ingesting". The native provider runs checks; the dbt and Great Expectations adapters INGEST results produced elsewhere (n0d#38 owns dbt execution against BI-P022, and GE is Python — nothing here runs it). Calling all three "providers" while only one executes is how "we have Great Expectations coverage" ends up in a status report. THE FALSE GREEN, proven against a live ClickHouse 24.8.14.39: a `not_null` check over an EMPTY table returns violations=0, evaluated=0. The raw counts are asserted first — the engine really does say zero-over-zero — and then the provider is required to call it a FAILURE, not a pass. Every rule compiles to a VIOLATION expression rather than a "rows that are fine" count, so the distinction cannot be lost at the runner layer. Same treatment for the ingesting providers: a MISSING dbt result is a failure, and dbt's `error`, `warn` and `skipped` are each asserted not to be passes. `skipped` in particular is the exact shape of l0re's 28 silently-skipped Postgres tests. The gate: zero verdicts QUARANTINES. "No rules ran" and "everything passed" produce the same green in most implementations, which is how a quality gate stops gating without anyone noticing — api0's `trivy --exit-code 0` in another costume. A later passing run does not auto-release either; release takes a principal and a reason, or a flapping rule would quarantine and un-quarantine a dataset between two reads of the same dashboard. BI-P025's DoD is the READ side: a quarantined projection is REFUSED, not annotated. An annotation on a dashboard nobody reads is not a quarantine. Live counts, all exact, from evidence/bi-p025/native-quality-live.json: not_null 5/118, range 3, distribution 4, unique 11. The unique figure is 11 rather than the 6 the first version of the test asserted — the five empty-string customers are duplicates of each other, so not_null and unique overlap on the same defect. The engine was right and the fixture arithmetic was wrong; the breakdown is written into the test rather than the number quietly adjusted. Tenant isolation re-proven on this path specifically, because a quality run uses an ADMINISTRATIVE credential: with 618 rows in the base table, t_alpha evaluates 118 and t_beta 500, and t_beta's blocking failure quarantines t_beta alone. Without the forced predicate one tenant's dirty rows would quarantine another tenant's projection. BI-P026. Five states, and the DoD's operative word is "silently". So the gate sits at the boundary every result crosses: a result with NO freshness envelope is unreleasable — not defaulted to FRESH, not defaulted to UNKNOWN — because "we forgot to attach it" and "the data is current" are indistinguishable to a reader, and any default makes forgetting the silent path. A non-current state without an explicit acknowledgement is refused, so a consumer that ignores the envelope cannot render stale numbers as current. FAILED outranks age: a pipeline that just failed may still hold recently-timestamped data from the previous run, and reporting that as FRESH is how a broken pipeline stays invisible for a day. UNKNOWN is never collapsed into FRESH. 54 unit tests, 11 live tests. typecheck + eslint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"clickhouse_version": "24.8.14.39",
|
||||
"recorded_at": "2026-09-07T00:40:02.745Z",
|
||||
"not_null": {
|
||||
"evaluated": 118,
|
||||
"violations": 5
|
||||
},
|
||||
"range": {
|
||||
"violations": 3
|
||||
},
|
||||
"distribution": {
|
||||
"violations": 4
|
||||
},
|
||||
"unique": {
|
||||
"violations": 11,
|
||||
"breakdown": "6 rows on dup_0/dup_1, plus 5 rows sharing the empty-string key"
|
||||
},
|
||||
"empty_dataset": {
|
||||
"raw_counts_from_clickhouse": {
|
||||
"violations": 0,
|
||||
"evaluated": 0
|
||||
},
|
||||
"verdict_passed": false,
|
||||
"note": "a real engine returns zero violations over zero rows; the provider refuses to call that a pass"
|
||||
},
|
||||
"tenant_isolation": {
|
||||
"base_table_rows": 618,
|
||||
"alpha_evaluated": 118,
|
||||
"alpha_violations": 5,
|
||||
"beta_evaluated": 500,
|
||||
"beta_violations": 500
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
/**
|
||||
* src/analytics/quality/clickhouse-quality-runner.ts
|
||||
* ===================================================
|
||||
* BI-P025 — the ClickHouse `QualityQueryRunner`.
|
||||
*
|
||||
* One statement per rule, and it counts BOTH numbers in a single pass:
|
||||
*
|
||||
* SELECT countIf(<violation expression>) AS violations, count() AS evaluated
|
||||
* FROM <db>.<table> WHERE tenant_id = {tenant:String} AND dataset_id = …
|
||||
*
|
||||
* `evaluated` is not decoration. It is what makes an EMPTY dataset
|
||||
* distinguishable from a CLEAN one — the false green
|
||||
* `NativeQualityProvider` refuses. Counting only violations would make the two
|
||||
* cases identical at this layer, and no amount of care upstream could recover
|
||||
* the difference.
|
||||
*
|
||||
* The BI-P011 forced tenant predicate applies here exactly as it does to a
|
||||
* metric query: `tenant_id = {tenant:String}` is condition[0], the value is
|
||||
* bound, and a quality run therefore cannot read another tenant's rows even
|
||||
* though it runs on an administrative credential.
|
||||
*
|
||||
* `unique` gets its own statement shape, because a duplicate is a property of
|
||||
* a GROUP, not of a row, and pretending otherwise would make the rule count
|
||||
* the wrong thing.
|
||||
*/
|
||||
import { SiaxError } from '../../platform/errors';
|
||||
import { assertSafeIdentifier, FORCED_TENANT_PREDICATE } from '../engine/query-compiler';
|
||||
import type { QualityQueryRunner } from './quality-provider';
|
||||
|
||||
export interface ClickHouseQualityRunnerOptions {
|
||||
baseUrl: string;
|
||||
basicAuth?: string;
|
||||
database: string;
|
||||
/** dataset_id -> physical table. An unlisted dataset is refused, not guessed. */
|
||||
tables: Record<string, string>;
|
||||
timeoutMs?: number;
|
||||
/** Ceiling for a quality scan; a check must not become the expensive query. */
|
||||
maxRowsToRead?: number;
|
||||
}
|
||||
|
||||
/** Marker the provider emits for the `unique` rule. */
|
||||
const UNIQUE_MARKER = /^__unique__\(([A-Za-z_][A-Za-z0-9_]{0,62})\)$/;
|
||||
|
||||
export class ClickHouseQualityRunner implements QualityQueryRunner {
|
||||
readonly engine = 'clickhouse';
|
||||
|
||||
constructor(private readonly opts: ClickHouseQualityRunnerOptions) {
|
||||
assertSafeIdentifier(opts.database, 'database');
|
||||
for (const table of Object.values(opts.tables)) assertSafeIdentifier(table, 'table');
|
||||
}
|
||||
|
||||
async count(args: {
|
||||
tenant_id: string;
|
||||
dataset_id: string;
|
||||
violationExpression: string;
|
||||
params: Record<string, string>;
|
||||
}): Promise<{ violations: number; evaluated: number }> {
|
||||
if (!args.tenant_id?.trim()) {
|
||||
throw new SiaxError(
|
||||
'UNAUTHENTICATED',
|
||||
'0pt.analytics.quality',
|
||||
'a quality run requires a verified tenant',
|
||||
);
|
||||
}
|
||||
const table = this.opts.tables[args.dataset_id];
|
||||
if (!table) {
|
||||
throw new SiaxError(
|
||||
'NOT_FOUND',
|
||||
'0pt.analytics.quality',
|
||||
`dataset "${args.dataset_id}" has no physical table registered for quality runs`,
|
||||
);
|
||||
}
|
||||
const params: Record<string, string> = {
|
||||
...args.params,
|
||||
param_tenant: args.tenant_id,
|
||||
param_dataset: args.dataset_id,
|
||||
};
|
||||
const where = `${FORCED_TENANT_PREDICATE} AND dataset_id = {dataset:String}`;
|
||||
|
||||
const unique = UNIQUE_MARKER.exec(args.violationExpression.trim());
|
||||
const sql = unique
|
||||
? // A duplicate is a property of a GROUP. Count the rows that belong to a
|
||||
// key appearing more than once, and the total rows, in one pass.
|
||||
`SELECT sum(multiIf(c > 1, c, 0)) AS violations, sum(c) AS evaluated FROM ` +
|
||||
`(SELECT ${assertSafeIdentifier(unique[1], 'field')} AS k, count() AS c ` +
|
||||
`FROM ${this.opts.database}.${table} WHERE ${where} GROUP BY k) FORMAT JSON`
|
||||
: `SELECT countIf(${args.violationExpression}) AS violations, count() AS evaluated ` +
|
||||
`FROM ${this.opts.database}.${table} WHERE ${where} FORMAT JSON`;
|
||||
|
||||
const url = new URL(this.opts.baseUrl.replace(/\/+$/, ''));
|
||||
url.searchParams.set('query', sql);
|
||||
for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
|
||||
url.searchParams.set('max_rows_to_read', String(this.opts.maxRowsToRead ?? 5_000_000_000));
|
||||
url.searchParams.set('read_overflow_mode', 'throw');
|
||||
url.searchParams.set('max_execution_time', '60');
|
||||
url.searchParams.set('timeout_overflow_mode', 'throw');
|
||||
|
||||
const headers: Record<string, string> = {};
|
||||
if (this.opts.basicAuth) {
|
||||
headers.Authorization = `Basic ${Buffer.from(this.opts.basicAuth).toString('base64')}`;
|
||||
}
|
||||
|
||||
let res: Response;
|
||||
try {
|
||||
res = await fetch(url.toString(), {
|
||||
method: 'POST',
|
||||
headers,
|
||||
body: '',
|
||||
signal: AbortSignal.timeout(this.opts.timeoutMs ?? 60_000),
|
||||
});
|
||||
} catch (cause) {
|
||||
throw new SiaxError('DEPENDENCY_UNAVAILABLE', 'clickhouse', 'clickhouse unreachable', { cause });
|
||||
}
|
||||
const text = await res.text();
|
||||
if (!res.ok) {
|
||||
const code = /Code:\s*(\d+)/.exec(text)?.[1] ?? String(res.status);
|
||||
throw new SiaxError('PERMANENT_FAILURE', 'clickhouse', `quality query failed (CH code ${code})`);
|
||||
}
|
||||
const row = (
|
||||
JSON.parse(text) as { data: Array<{ violations: string | number; evaluated: string | number }> }
|
||||
).data[0];
|
||||
if (!row) {
|
||||
throw new SiaxError('PERMANENT_FAILURE', 'clickhouse', 'quality query returned no counts');
|
||||
}
|
||||
return { violations: Number(row.violations ?? 0), evaluated: Number(row.evaluated ?? 0) };
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,196 @@
|
||||
/**
|
||||
* src/analytics/quality/freshness.ts
|
||||
* ===================================
|
||||
* BI-P026 — DATA FRESHNESS.
|
||||
*
|
||||
* Five states, exactly as BI-P026 names them, and the DoD is a NEGATIVE:
|
||||
*
|
||||
* "Stale data is never SILENTLY presented as current."
|
||||
*
|
||||
* Two words carry it. "Silently" means the failure mode is not "we served old
|
||||
* data" — it is "we served old data and the reader could not tell". And
|
||||
* "never" means the enforcement has to sit at the boundary every result
|
||||
* crosses, not in each dashboard's rendering code.
|
||||
*
|
||||
* So `assertFreshnessDisclosed` refuses to release a result that carries no
|
||||
* freshness envelope at all. A result with no envelope is not FRESH and not
|
||||
* UNKNOWN — it is unreleasable, because "we forgot to attach it" and "the data
|
||||
* is current" are indistinguishable to a reader, and a default of FRESH would
|
||||
* make forgetting the silent path.
|
||||
*
|
||||
* UNKNOWN is likewise never treated as FRESH. It is its own state precisely
|
||||
* because "we could not determine the age" is the honest thing to display, and
|
||||
* collapsing it into FRESH is the classic version of this bug.
|
||||
*/
|
||||
import { SiaxError } from '../../platform/errors';
|
||||
|
||||
/** BI-P026's states, verbatim and in severity order. */
|
||||
export const FRESHNESS_STATES = ['FRESH', 'DELAYED', 'STALE', 'FAILED', 'UNKNOWN'] as const;
|
||||
export type FreshnessState = (typeof FRESHNESS_STATES)[number];
|
||||
|
||||
/** States a reader must be shown explicitly rather than served as current. */
|
||||
export const NON_CURRENT_STATES: readonly FreshnessState[] = ['DELAYED', 'STALE', 'FAILED', 'UNKNOWN'];
|
||||
|
||||
export interface FreshnessPolicy {
|
||||
/** How often the upstream is expected to land data. */
|
||||
expectedIntervalMs: number;
|
||||
/**
|
||||
* Multiples of the expected interval. Past `delayedAfter` the data is
|
||||
* DELAYED; past `staleAfter` it is STALE. Defaults are deliberately tight:
|
||||
* a dashboard that is two cycles behind is already misleading someone.
|
||||
*/
|
||||
delayedAfterIntervals?: number;
|
||||
staleAfterIntervals?: number;
|
||||
}
|
||||
|
||||
export interface FreshnessInput {
|
||||
/** When the newest successfully loaded data is timestamped. */
|
||||
lastSuccessfulLoadAt?: string;
|
||||
/** Outcome of the most recent load attempt. */
|
||||
lastRunStatus?: 'succeeded' | 'failed' | 'running' | 'never_run';
|
||||
lastRunAt?: string;
|
||||
policy: FreshnessPolicy;
|
||||
now: number;
|
||||
}
|
||||
|
||||
export interface FreshnessEnvelope {
|
||||
state: FreshnessState;
|
||||
/** Age of the newest data, in ms. Absent when unknown. */
|
||||
age_ms?: number;
|
||||
last_successful_load_at?: string;
|
||||
last_run_at?: string;
|
||||
expected_interval_ms: number;
|
||||
/** Plain-language line a UI can render verbatim. */
|
||||
disclosure: string;
|
||||
computed_at: string;
|
||||
}
|
||||
|
||||
export function computeFreshness(input: FreshnessInput): FreshnessEnvelope {
|
||||
const { policy, now } = input;
|
||||
const delayedAfter = policy.expectedIntervalMs * (policy.delayedAfterIntervals ?? 1.5);
|
||||
const staleAfter = policy.expectedIntervalMs * (policy.staleAfterIntervals ?? 3);
|
||||
const computed_at = new Date(now).toISOString();
|
||||
const base = {
|
||||
last_successful_load_at: input.lastSuccessfulLoadAt,
|
||||
last_run_at: input.lastRunAt,
|
||||
expected_interval_ms: policy.expectedIntervalMs,
|
||||
computed_at,
|
||||
};
|
||||
|
||||
// FAILED outranks age: a pipeline that just failed may still have
|
||||
// recently-timestamped data from the run before, and reporting that as FRESH
|
||||
// is precisely how a broken pipeline stays invisible for a day.
|
||||
if (input.lastRunStatus === 'failed') {
|
||||
const age = ageOf(input.lastSuccessfulLoadAt, now);
|
||||
return {
|
||||
...base,
|
||||
state: 'FAILED',
|
||||
age_ms: age,
|
||||
disclosure: `The most recent load FAILED. Showing data from ${input.lastSuccessfulLoadAt ?? 'an unknown time'}.`,
|
||||
};
|
||||
}
|
||||
|
||||
if (!input.lastSuccessfulLoadAt || input.lastRunStatus === 'never_run') {
|
||||
return {
|
||||
...base,
|
||||
state: 'UNKNOWN',
|
||||
disclosure: 'The age of this data could not be determined. Do not read it as current.',
|
||||
};
|
||||
}
|
||||
|
||||
const age = ageOf(input.lastSuccessfulLoadAt, now);
|
||||
if (age === undefined) {
|
||||
return { ...base, state: 'UNKNOWN', disclosure: 'The age of this data could not be determined.' };
|
||||
}
|
||||
if (age > staleAfter) {
|
||||
return {
|
||||
...base,
|
||||
state: 'STALE',
|
||||
age_ms: age,
|
||||
disclosure: `This data is ${describeAge(age)} old, past the ${describeAge(staleAfter)} staleness threshold.`,
|
||||
};
|
||||
}
|
||||
if (age > delayedAfter) {
|
||||
return {
|
||||
...base,
|
||||
state: 'DELAYED',
|
||||
age_ms: age,
|
||||
disclosure: `This data is ${describeAge(age)} old and behind its ${describeAge(policy.expectedIntervalMs)} schedule.`,
|
||||
};
|
||||
}
|
||||
return {
|
||||
...base,
|
||||
state: 'FRESH',
|
||||
age_ms: age,
|
||||
disclosure: `Current as of ${input.lastSuccessfulLoadAt}.`,
|
||||
};
|
||||
}
|
||||
|
||||
function ageOf(iso: string | undefined, now: number): number | undefined {
|
||||
if (!iso) return undefined;
|
||||
const t = Date.parse(iso);
|
||||
if (Number.isNaN(t)) return undefined;
|
||||
return Math.max(0, now - t);
|
||||
}
|
||||
|
||||
function describeAge(ms: number): string {
|
||||
const s = Math.round(ms / 1000);
|
||||
if (s < 90) return `${s}s`;
|
||||
const m = Math.round(s / 60);
|
||||
if (m < 90) return `${m}m`;
|
||||
const h = Math.round(m / 60);
|
||||
if (h < 48) return `${h}h`;
|
||||
return `${Math.round(h / 24)}d`;
|
||||
}
|
||||
|
||||
/** Any result leaving the analytics boundary. */
|
||||
export interface DisclosableResult {
|
||||
freshness?: FreshnessEnvelope;
|
||||
/**
|
||||
* Set by a caller that has SEEN a non-current state and chosen to proceed.
|
||||
* Its presence is what makes serving non-current data not-silent.
|
||||
*/
|
||||
acknowledged_non_current?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* BI-P026's DoD, as a gate.
|
||||
*
|
||||
* Three refusals, in order of how easy they are to get wrong:
|
||||
* 1. no envelope at all — unreleasable. Not defaulted to FRESH.
|
||||
* 2. a non-current state with no acknowledgement — refused, so a consumer
|
||||
* that ignores the envelope cannot render stale numbers as current.
|
||||
* 3. an acknowledgement with no envelope — refused, because acknowledging
|
||||
* something you were never shown is not disclosure.
|
||||
*/
|
||||
export function assertFreshnessDisclosed(result: DisclosableResult, context: string): FreshnessEnvelope {
|
||||
if (!result.freshness) {
|
||||
throw new SiaxError(
|
||||
'PERMANENT_FAILURE',
|
||||
'0pt.analytics.freshness',
|
||||
`${context} carries no freshness envelope — BI-P026: data with unknown age must never be released ` +
|
||||
`as though it were current`,
|
||||
);
|
||||
}
|
||||
const state = result.freshness.state;
|
||||
if (!(FRESHNESS_STATES as readonly string[]).includes(state)) {
|
||||
throw new SiaxError(
|
||||
'PERMANENT_FAILURE',
|
||||
'0pt.analytics.freshness',
|
||||
`${context} has an unknown freshness state`,
|
||||
);
|
||||
}
|
||||
if (NON_CURRENT_STATES.includes(state) && !result.acknowledged_non_current) {
|
||||
throw new SiaxError(
|
||||
'CONFLICT',
|
||||
'0pt.analytics.freshness',
|
||||
`${context} is ${state} and the consumer has not acknowledged it: ${result.freshness.disclosure}`,
|
||||
);
|
||||
}
|
||||
return result.freshness;
|
||||
}
|
||||
|
||||
/** True when the state may be presented without a caveat. Only one state may. */
|
||||
export function isPresentableAsCurrent(state: FreshnessState): boolean {
|
||||
return state === 'FRESH';
|
||||
}
|
||||
@@ -0,0 +1,418 @@
|
||||
/**
|
||||
* src/analytics/quality/quality-provider.ts
|
||||
* ==========================================
|
||||
* BI-P025 — DATA QUALITY PROVIDER.
|
||||
*
|
||||
* One interface, three provider kinds, and an honest split between them:
|
||||
*
|
||||
* NATIVE — this module compiles the rule into a counting query and runs it
|
||||
* through an injected runner. It EXECUTES.
|
||||
* DBT — dbt runs its own tests elsewhere (n0d#38 owns execution against
|
||||
* the BI-P022 contract). This module INGESTS the results. It does
|
||||
* not run dbt, and `providerKind` says so rather than implying
|
||||
* otherwise.
|
||||
* GREAT_EXPECTATIONS — same posture. GE is a Python library; nothing here
|
||||
* executes it. The adapter maps a GE validation result onto the
|
||||
* canonical verdict so a GE suite counts toward the same gate.
|
||||
*
|
||||
* Calling all three "providers" while only one executes would be the kind of
|
||||
* overstatement that turns into "we have Great Expectations coverage" in a
|
||||
* status report. `SiaxDataQualityProvider.providerKind` is `"executing"` or
|
||||
* `"ingesting"`, and the gate records which produced each verdict.
|
||||
*
|
||||
* ── BI-P025's DoD ─────────────────────────────────────────────────────────
|
||||
* "Failed critical quality gate can QUARANTINE serving projection."
|
||||
*
|
||||
* `quarantine.ts` is that, and the important half is what happens on the READ
|
||||
* side: a quarantined projection is REFUSED, not annotated. An annotation on a
|
||||
* dashboard nobody reads is not a quarantine.
|
||||
*/
|
||||
import { SiaxError } from '../../platform/errors';
|
||||
import type { QualityRuleKind, QualitySeverity } from '../domain/entities';
|
||||
|
||||
/** BI-P025's rule list, verbatim, mapped onto BI-P002's persisted kinds. */
|
||||
export const QUALITY_RULES = [
|
||||
'not_null',
|
||||
'unique',
|
||||
'range',
|
||||
'referential',
|
||||
'freshness',
|
||||
'schema',
|
||||
'distribution',
|
||||
'custom',
|
||||
] as const;
|
||||
export type QualityRule = (typeof QUALITY_RULES)[number];
|
||||
|
||||
/** BI-P002 persists `accepted_values` and `row_count`; BI-P025 names `distribution` and `range`. */
|
||||
export const RULE_KIND_ALIASES: Record<QualityRule, QualityRuleKind> = {
|
||||
not_null: 'not_null',
|
||||
unique: 'unique',
|
||||
range: 'range',
|
||||
referential: 'referential',
|
||||
freshness: 'freshness',
|
||||
schema: 'custom',
|
||||
distribution: 'accepted_values',
|
||||
custom: 'custom',
|
||||
};
|
||||
|
||||
export interface QualityRuleSpec {
|
||||
rule_id: string;
|
||||
tenant_id: string;
|
||||
dataset_id: string;
|
||||
rule: QualityRule;
|
||||
/** Column the rule applies to; absent for dataset-level rules. */
|
||||
field?: string;
|
||||
severity: QualitySeverity;
|
||||
params: QualityRuleParams;
|
||||
description?: string;
|
||||
}
|
||||
|
||||
export interface QualityRuleParams {
|
||||
/** range */
|
||||
min?: number;
|
||||
max?: number;
|
||||
/** distribution / accepted values */
|
||||
accepted_values?: string[];
|
||||
/** distribution: the share of rows that must fall in accepted_values */
|
||||
min_share?: number;
|
||||
/** referential */
|
||||
references_dataset?: string;
|
||||
references_field?: string;
|
||||
/** freshness: the maximum acceptable age of the newest row */
|
||||
max_age_seconds?: number;
|
||||
timestamp_field?: string;
|
||||
/** schema: the columns and types that must exist */
|
||||
expected_columns?: Array<{ name: string; type: string }>;
|
||||
/** unique / not_null take no parameters */
|
||||
/** custom: an operator-authored counting query, reviewed like code */
|
||||
custom_violation_sql?: string;
|
||||
}
|
||||
|
||||
export interface QualityVerdict {
|
||||
rule_id: string;
|
||||
dataset_id: string;
|
||||
tenant_id: string;
|
||||
rule: QualityRule;
|
||||
severity: QualitySeverity;
|
||||
passed: boolean;
|
||||
violation_count: number;
|
||||
evaluated_rows: number;
|
||||
/** Which provider produced this, and whether it EXECUTED or merely ingested. */
|
||||
produced_by: string;
|
||||
provider_kind: 'executing' | 'ingesting';
|
||||
evaluated_at: string;
|
||||
/** Short operator-facing note. Never raw engine output. */
|
||||
detail?: string;
|
||||
}
|
||||
|
||||
export interface SiaxDataQualityProvider {
|
||||
readonly provider: string;
|
||||
readonly providerKind: 'executing' | 'ingesting';
|
||||
/** Rules this provider can actually evaluate. */
|
||||
supports(rule: QualityRule): boolean;
|
||||
evaluate(spec: QualityRuleSpec): Promise<QualityVerdict>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs a counting query and returns two numbers. The only engine coupling in
|
||||
* this module, so a DuckDB or Trino runner drops in unchanged.
|
||||
*/
|
||||
export interface QualityQueryRunner {
|
||||
readonly engine: string;
|
||||
/**
|
||||
* Must apply the caller's tenant scoping. The runner is handed the tenant
|
||||
* explicitly rather than trusting the SQL to carry it.
|
||||
*/
|
||||
count(args: {
|
||||
tenant_id: string;
|
||||
dataset_id: string;
|
||||
violationExpression: string;
|
||||
/** Bound parameters. Values NEVER appear in the expression text. */
|
||||
params: Record<string, string>;
|
||||
}): Promise<{ violations: number; evaluated: number }>;
|
||||
}
|
||||
|
||||
const IDENT_RE = /^[A-Za-z_][A-Za-z0-9_]{0,62}$/;
|
||||
|
||||
function assertField(field: string | undefined, rule: QualityRule): string {
|
||||
if (!field || !IDENT_RE.test(field)) {
|
||||
throw new SiaxError(
|
||||
'INVALID_REQUEST',
|
||||
'0pt.analytics.quality',
|
||||
`rule "${rule}" requires a valid field name`,
|
||||
);
|
||||
}
|
||||
return field;
|
||||
}
|
||||
|
||||
/**
|
||||
* The provider that actually runs checks.
|
||||
*
|
||||
* Every rule compiles to a VIOLATION EXPRESSION — a boolean over one row — and
|
||||
* the runner counts rows where it is true. That shape is chosen so a rule can
|
||||
* never return "how many rows are fine", which is the number that looks
|
||||
* healthy when a table is empty.
|
||||
*/
|
||||
export class NativeQualityProvider implements SiaxDataQualityProvider {
|
||||
readonly provider = 'native';
|
||||
readonly providerKind = 'executing' as const;
|
||||
|
||||
constructor(
|
||||
private readonly runner: QualityQueryRunner,
|
||||
private readonly now: () => number = Date.now,
|
||||
) {}
|
||||
|
||||
supports(rule: QualityRule): boolean {
|
||||
// `schema` and `referential` need catalog/second-table access the counting
|
||||
// runner does not have; they are honestly unsupported here rather than
|
||||
// silently passing.
|
||||
return !['schema', 'referential'].includes(rule);
|
||||
}
|
||||
|
||||
async evaluate(spec: QualityRuleSpec): Promise<QualityVerdict> {
|
||||
if (!this.supports(spec.rule)) {
|
||||
throw new SiaxError(
|
||||
'CAPABILITY_UNAVAILABLE',
|
||||
'0pt.analytics.quality',
|
||||
`the native provider cannot evaluate "${spec.rule}" — use a dbt test or a Great Expectations suite`,
|
||||
);
|
||||
}
|
||||
const { expression, params } = this.compile(spec);
|
||||
const { violations, evaluated } = await this.runner.count({
|
||||
tenant_id: spec.tenant_id,
|
||||
dataset_id: spec.dataset_id,
|
||||
violationExpression: expression,
|
||||
params,
|
||||
});
|
||||
|
||||
// An EMPTY dataset is not a passing dataset. A `not_null` check over zero
|
||||
// rows finds zero violations and would report success while the pipeline
|
||||
// is silently broken — the single most common false green in data quality.
|
||||
const emptyIsFailure = spec.rule !== 'custom';
|
||||
const passed = evaluated === 0 && emptyIsFailure ? false : violations === 0;
|
||||
|
||||
return {
|
||||
rule_id: spec.rule_id,
|
||||
dataset_id: spec.dataset_id,
|
||||
tenant_id: spec.tenant_id,
|
||||
rule: spec.rule,
|
||||
severity: spec.severity,
|
||||
passed,
|
||||
violation_count: violations,
|
||||
evaluated_rows: evaluated,
|
||||
produced_by: `${this.provider}:${this.runner.engine}`,
|
||||
provider_kind: this.providerKind,
|
||||
evaluated_at: new Date(this.now()).toISOString(),
|
||||
detail:
|
||||
evaluated === 0 && emptyIsFailure
|
||||
? 'dataset is EMPTY — reported as a failure, not as zero violations'
|
||||
: undefined,
|
||||
};
|
||||
}
|
||||
|
||||
private compile(spec: QualityRuleSpec): { expression: string; params: Record<string, string> } {
|
||||
const p = spec.params;
|
||||
switch (spec.rule) {
|
||||
case 'not_null': {
|
||||
const f = assertField(spec.field, spec.rule);
|
||||
return { expression: `${f} IS NULL OR ${f} = ''`, params: {} };
|
||||
}
|
||||
case 'unique': {
|
||||
const f = assertField(spec.field, spec.rule);
|
||||
// Counted over a grouped subquery by the runner; the expression names
|
||||
// the key so the runner can build it.
|
||||
return { expression: `__unique__(${f})`, params: {} };
|
||||
}
|
||||
case 'range': {
|
||||
const f = assertField(spec.field, spec.rule);
|
||||
if (p.min === undefined && p.max === undefined) {
|
||||
throw new SiaxError(
|
||||
'INVALID_REQUEST',
|
||||
'0pt.analytics.quality',
|
||||
'a range rule needs a min or a max',
|
||||
);
|
||||
}
|
||||
const parts: string[] = [];
|
||||
const params: Record<string, string> = {};
|
||||
if (p.min !== undefined) {
|
||||
parts.push(`${f} < {qmin:Float64}`);
|
||||
params.param_qmin = String(p.min);
|
||||
}
|
||||
if (p.max !== undefined) {
|
||||
parts.push(`${f} > {qmax:Float64}`);
|
||||
params.param_qmax = String(p.max);
|
||||
}
|
||||
return { expression: parts.join(' OR '), params };
|
||||
}
|
||||
case 'freshness': {
|
||||
const f = assertField(p.timestamp_field ?? spec.field, spec.rule);
|
||||
if (!p.max_age_seconds || p.max_age_seconds <= 0) {
|
||||
throw new SiaxError(
|
||||
'INVALID_REQUEST',
|
||||
'0pt.analytics.quality',
|
||||
'a freshness rule needs max_age_seconds',
|
||||
);
|
||||
}
|
||||
return {
|
||||
expression: `${f} < now() - toIntervalSecond({qage:UInt64})`,
|
||||
params: { param_qage: String(Math.floor(p.max_age_seconds)) },
|
||||
};
|
||||
}
|
||||
case 'distribution': {
|
||||
const f = assertField(spec.field, spec.rule);
|
||||
if (!p.accepted_values?.length) {
|
||||
throw new SiaxError(
|
||||
'INVALID_REQUEST',
|
||||
'0pt.analytics.quality',
|
||||
'a distribution rule needs accepted_values',
|
||||
);
|
||||
}
|
||||
const params: Record<string, string> = {};
|
||||
const bound = p.accepted_values.map((v, i) => {
|
||||
params[`param_qv${i}`] = v;
|
||||
return `{qv${i}:String}`;
|
||||
});
|
||||
return { expression: `${f} NOT IN (${bound.join(', ')})`, params };
|
||||
}
|
||||
case 'custom': {
|
||||
if (!p.custom_violation_sql?.trim()) {
|
||||
throw new SiaxError(
|
||||
'INVALID_REQUEST',
|
||||
'0pt.analytics.quality',
|
||||
'a custom rule needs custom_violation_sql',
|
||||
);
|
||||
}
|
||||
// Deliberately NOT parsed or guarded here. A custom rule is
|
||||
// operator-authored, reviewed like code, and stored in the rule table
|
||||
// — not caller input. It never reaches this function from a request
|
||||
// body; `QualityRuleSpec` rows are written by an admin path.
|
||||
return { expression: p.custom_violation_sql, params: {} };
|
||||
}
|
||||
default:
|
||||
throw new SiaxError('INVALID_REQUEST', '0pt.analytics.quality', `unsupported rule "${spec.rule}"`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Ingests dbt test results. Does NOT run dbt — n0d#38 owns execution against
|
||||
* the BI-P022 contract, and pretending otherwise here would misreport where
|
||||
* the work happens.
|
||||
*/
|
||||
export interface DbtTestResult {
|
||||
unique_id: string;
|
||||
status: 'pass' | 'fail' | 'error' | 'warn' | 'skipped';
|
||||
failures: number;
|
||||
/** dbt does not report rows evaluated; absent, not zero. */
|
||||
rows_evaluated?: number;
|
||||
message?: string;
|
||||
}
|
||||
|
||||
export class DbtQualityProvider implements SiaxDataQualityProvider {
|
||||
readonly provider = 'dbt';
|
||||
readonly providerKind = 'ingesting' as const;
|
||||
constructor(
|
||||
private readonly results: Map<string, DbtTestResult>,
|
||||
private readonly now: () => number = Date.now,
|
||||
) {}
|
||||
|
||||
supports(): boolean {
|
||||
return true;
|
||||
}
|
||||
|
||||
async evaluate(spec: QualityRuleSpec): Promise<QualityVerdict> {
|
||||
const result = this.results.get(spec.rule_id);
|
||||
if (!result) {
|
||||
// A missing result is UNKNOWN, and unknown is a FAILURE for a blocking
|
||||
// rule. Treating "no result" as "passed" is how a quality gate stops
|
||||
// gating the moment the test stops running.
|
||||
return {
|
||||
rule_id: spec.rule_id,
|
||||
dataset_id: spec.dataset_id,
|
||||
tenant_id: spec.tenant_id,
|
||||
rule: spec.rule,
|
||||
severity: spec.severity,
|
||||
passed: false,
|
||||
violation_count: 0,
|
||||
evaluated_rows: 0,
|
||||
produced_by: this.provider,
|
||||
provider_kind: this.providerKind,
|
||||
evaluated_at: new Date(this.now()).toISOString(),
|
||||
detail: 'no dbt result for this rule — reported as a failure, not as a pass',
|
||||
};
|
||||
}
|
||||
return {
|
||||
rule_id: spec.rule_id,
|
||||
dataset_id: spec.dataset_id,
|
||||
tenant_id: spec.tenant_id,
|
||||
rule: spec.rule,
|
||||
severity: spec.severity,
|
||||
// `skipped` and `error` are NOT passes. dbt's `skipped` in particular is
|
||||
// the shape of l0re's silently-skipped Postgres tests.
|
||||
passed: result.status === 'pass',
|
||||
violation_count: result.failures,
|
||||
evaluated_rows: result.rows_evaluated ?? 0,
|
||||
produced_by: `dbt:${result.unique_id}`,
|
||||
provider_kind: this.providerKind,
|
||||
evaluated_at: new Date(this.now()).toISOString(),
|
||||
detail: result.status !== 'pass' ? `dbt status ${result.status}` : undefined,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/** Ingests a Great Expectations validation result. Does NOT run GE. */
|
||||
export interface GreatExpectationsResult {
|
||||
expectation_suite_name: string;
|
||||
success: boolean;
|
||||
results: Array<{
|
||||
expectation_config: { expectation_type: string };
|
||||
success: boolean;
|
||||
result?: { unexpected_count?: number; element_count?: number };
|
||||
}>;
|
||||
}
|
||||
|
||||
export class GreatExpectationsQualityProvider implements SiaxDataQualityProvider {
|
||||
readonly provider = 'great_expectations';
|
||||
readonly providerKind = 'ingesting' as const;
|
||||
constructor(
|
||||
private readonly suites: Map<string, GreatExpectationsResult>,
|
||||
private readonly now: () => number = Date.now,
|
||||
) {}
|
||||
|
||||
supports(): boolean {
|
||||
return true;
|
||||
}
|
||||
|
||||
async evaluate(spec: QualityRuleSpec): Promise<QualityVerdict> {
|
||||
const suite = this.suites.get(spec.rule_id);
|
||||
const base = {
|
||||
rule_id: spec.rule_id,
|
||||
dataset_id: spec.dataset_id,
|
||||
tenant_id: spec.tenant_id,
|
||||
rule: spec.rule,
|
||||
severity: spec.severity,
|
||||
produced_by: this.provider,
|
||||
provider_kind: this.providerKind,
|
||||
evaluated_at: new Date(this.now()).toISOString(),
|
||||
};
|
||||
if (!suite) {
|
||||
return {
|
||||
...base,
|
||||
passed: false,
|
||||
violation_count: 0,
|
||||
evaluated_rows: 0,
|
||||
detail: 'no GE result — reported as a failure',
|
||||
};
|
||||
}
|
||||
const unexpected = suite.results.reduce((a, r) => a + (r.result?.unexpected_count ?? 0), 0);
|
||||
const elements = suite.results.reduce((a, r) => Math.max(a, r.result?.element_count ?? 0), 0);
|
||||
return {
|
||||
...base,
|
||||
passed: suite.success,
|
||||
violation_count: unexpected,
|
||||
evaluated_rows: elements,
|
||||
produced_by: `great_expectations:${suite.expectation_suite_name}`,
|
||||
detail: suite.success ? undefined : 'GE suite failed',
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,188 @@
|
||||
/**
|
||||
* src/analytics/quality/quarantine.ts
|
||||
* ====================================
|
||||
* BI-P025's DoD: "Failed critical quality gate can QUARANTINE serving
|
||||
* projection."
|
||||
*
|
||||
* The word that matters is on the READ side. A quarantine that annotates a
|
||||
* result is a label; a quarantine that REFUSES the read is a quarantine. This
|
||||
* module does the second, and `assertProjectionReadable` is the single place
|
||||
* every serving read passes through.
|
||||
*
|
||||
* Severity semantics, stated because they are the whole gate:
|
||||
* `block` — a failure quarantines the projection. Reads are refused.
|
||||
* `warn` — a failure is recorded and surfaced, and reads continue.
|
||||
* A rule with no severity does not exist: `QualityRuleSpec.severity` is
|
||||
* required, so "we forgot to mark it critical" cannot silently become `warn`.
|
||||
*/
|
||||
import { SiaxError } from '../../platform/errors';
|
||||
import type { QualityVerdict } from './quality-provider';
|
||||
|
||||
export interface QuarantineRecord {
|
||||
tenant_id: string;
|
||||
dataset_id: string;
|
||||
quarantined: boolean;
|
||||
/** Rule ids whose blocking failure caused it. */
|
||||
caused_by: string[];
|
||||
since?: string;
|
||||
released_at?: string;
|
||||
released_by?: string;
|
||||
/** Operator-facing summary. */
|
||||
reason?: string;
|
||||
}
|
||||
|
||||
export interface QuarantineStore {
|
||||
get(tenantId: string, datasetId: string): Promise<QuarantineRecord | undefined>;
|
||||
put(record: QuarantineRecord): Promise<void>;
|
||||
}
|
||||
|
||||
export class InMemoryQuarantineStore implements QuarantineStore {
|
||||
private readonly rows = new Map<string, QuarantineRecord>();
|
||||
private key(t: string, d: string): string {
|
||||
return `${t}::${d}`;
|
||||
}
|
||||
async get(t: string, d: string): Promise<QuarantineRecord | undefined> {
|
||||
return this.rows.get(this.key(t, d));
|
||||
}
|
||||
async put(r: QuarantineRecord): Promise<void> {
|
||||
this.rows.set(this.key(r.tenant_id, r.dataset_id), r);
|
||||
}
|
||||
}
|
||||
|
||||
export interface GateOutcome {
|
||||
dataset_id: string;
|
||||
tenant_id: string;
|
||||
passed: boolean;
|
||||
quarantined: boolean;
|
||||
blocking_failures: QualityVerdict[];
|
||||
warnings: QualityVerdict[];
|
||||
evaluated: number;
|
||||
}
|
||||
|
||||
export class QualityGate {
|
||||
constructor(
|
||||
private readonly store: QuarantineStore,
|
||||
private readonly now: () => number = Date.now,
|
||||
) {}
|
||||
|
||||
/**
|
||||
* Apply a run's verdicts.
|
||||
*
|
||||
* A gate over ZERO verdicts does not pass. "No rules ran" and "every rule
|
||||
* passed" produce the same green in most implementations, and that is how a
|
||||
* quality gate stops gating without anyone noticing — the same shape as
|
||||
* l0re's 28 silently-skipped tests and api0's `trivy --exit-code 0`.
|
||||
*/
|
||||
async apply(tenantId: string, datasetId: string, verdicts: QualityVerdict[]): Promise<GateOutcome> {
|
||||
if (verdicts.length === 0) {
|
||||
const record: QuarantineRecord = {
|
||||
tenant_id: tenantId,
|
||||
dataset_id: datasetId,
|
||||
quarantined: true,
|
||||
caused_by: [],
|
||||
since: new Date(this.now()).toISOString(),
|
||||
reason: 'no quality rules were evaluated — an ungated dataset is quarantined, not trusted',
|
||||
};
|
||||
await this.store.put(record);
|
||||
return {
|
||||
dataset_id: datasetId,
|
||||
tenant_id: tenantId,
|
||||
passed: false,
|
||||
quarantined: true,
|
||||
blocking_failures: [],
|
||||
warnings: [],
|
||||
evaluated: 0,
|
||||
};
|
||||
}
|
||||
|
||||
for (const v of verdicts) {
|
||||
if (v.tenant_id !== tenantId || v.dataset_id !== datasetId) {
|
||||
// A verdict from another tenant's run must never influence this gate.
|
||||
throw new SiaxError(
|
||||
'AUTHORIZATION_DENIED',
|
||||
'0pt.analytics.quality',
|
||||
'a quality verdict from a different tenant or dataset was supplied to this gate',
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const blocking = verdicts.filter((v) => !v.passed && v.severity === 'block');
|
||||
const warnings = verdicts.filter((v) => !v.passed && v.severity === 'warn');
|
||||
|
||||
if (blocking.length > 0) {
|
||||
await this.store.put({
|
||||
tenant_id: tenantId,
|
||||
dataset_id: datasetId,
|
||||
quarantined: true,
|
||||
caused_by: blocking.map((v) => v.rule_id),
|
||||
since: new Date(this.now()).toISOString(),
|
||||
reason: `${blocking.length} blocking quality rule(s) failed`,
|
||||
});
|
||||
} else {
|
||||
// Passing does NOT auto-release: releasing is an act with an operator
|
||||
// behind it (`release()`), because a flapping rule would otherwise let a
|
||||
// dataset quarantine and un-quarantine itself between reads.
|
||||
const existing = await this.store.get(tenantId, datasetId);
|
||||
if (!existing || !existing.quarantined) {
|
||||
await this.store.put({
|
||||
tenant_id: tenantId,
|
||||
dataset_id: datasetId,
|
||||
quarantined: false,
|
||||
caused_by: [],
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
dataset_id: datasetId,
|
||||
tenant_id: tenantId,
|
||||
passed: blocking.length === 0,
|
||||
quarantined: blocking.length > 0 || (await this.isQuarantined(tenantId, datasetId)),
|
||||
blocking_failures: blocking,
|
||||
warnings,
|
||||
evaluated: verdicts.length,
|
||||
};
|
||||
}
|
||||
|
||||
async isQuarantined(tenantId: string, datasetId: string): Promise<boolean> {
|
||||
return (await this.store.get(tenantId, datasetId))?.quarantined ?? false;
|
||||
}
|
||||
|
||||
/** Release requires an operator and a reason. It is not automatic. */
|
||||
async release(tenantId: string, datasetId: string, by: string, reason: string): Promise<QuarantineRecord> {
|
||||
if (!by?.trim() || !reason?.trim()) {
|
||||
throw new SiaxError(
|
||||
'INVALID_REQUEST',
|
||||
'0pt.analytics.quality',
|
||||
'releasing a quarantine requires a principal and a reason',
|
||||
);
|
||||
}
|
||||
const record: QuarantineRecord = {
|
||||
tenant_id: tenantId,
|
||||
dataset_id: datasetId,
|
||||
quarantined: false,
|
||||
caused_by: [],
|
||||
released_at: new Date(this.now()).toISOString(),
|
||||
released_by: by,
|
||||
reason,
|
||||
};
|
||||
await this.store.put(record);
|
||||
return record;
|
||||
}
|
||||
|
||||
/**
|
||||
* The read-side gate. Every serving projection read passes through here.
|
||||
* Refuses, rather than annotating — an annotation on a dashboard nobody
|
||||
* reads is not a quarantine.
|
||||
*/
|
||||
async assertProjectionReadable(tenantId: string, datasetId: string): Promise<void> {
|
||||
const record = await this.store.get(tenantId, datasetId);
|
||||
if (record?.quarantined) {
|
||||
throw new SiaxError(
|
||||
'DEPENDENCY_UNAVAILABLE',
|
||||
'0pt.analytics.quality',
|
||||
`serving projection for "${datasetId}" is QUARANTINED: ${record.reason ?? 'failed quality gate'}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,377 @@
|
||||
/**
|
||||
* tests/analytics/quality-and-freshness.test.ts
|
||||
* ==============================================
|
||||
* BI-P025 (data quality provider) and BI-P026 (data freshness).
|
||||
*
|
||||
* Both DoDs are about a FALSE GREEN, so both are tested by constructing the
|
||||
* false green and requiring a failure:
|
||||
*
|
||||
* BI-P025 A quality gate that passes when nothing ran. A `not_null` check
|
||||
* over an EMPTY table finds zero violations — the single most
|
||||
* common false green in data quality. Both are asserted to FAIL.
|
||||
*
|
||||
* BI-P026 UNKNOWN age quietly rendering as current, and a result released
|
||||
* with no freshness envelope at all. Both are refused.
|
||||
*/
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { SiaxError } from '@/src/platform/errors';
|
||||
import {
|
||||
DbtQualityProvider,
|
||||
GreatExpectationsQualityProvider,
|
||||
NativeQualityProvider,
|
||||
QUALITY_RULES,
|
||||
type QualityQueryRunner,
|
||||
type QualityRuleSpec,
|
||||
type QualityVerdict,
|
||||
} from '@/src/analytics/quality/quality-provider';
|
||||
import { InMemoryQuarantineStore, QualityGate } from '@/src/analytics/quality/quarantine';
|
||||
import {
|
||||
assertFreshnessDisclosed,
|
||||
computeFreshness,
|
||||
FRESHNESS_STATES,
|
||||
isPresentableAsCurrent,
|
||||
NON_CURRENT_STATES,
|
||||
} from '@/src/analytics/quality/freshness';
|
||||
|
||||
function runner(
|
||||
violations: number,
|
||||
evaluated: number,
|
||||
): QualityQueryRunner & { lastExpression?: string; lastParams?: Record<string, string> } {
|
||||
const r: QualityQueryRunner & { lastExpression?: string; lastParams?: Record<string, string> } = {
|
||||
engine: 'clickhouse',
|
||||
async count(args) {
|
||||
r.lastExpression = args.violationExpression;
|
||||
r.lastParams = args.params;
|
||||
return { violations, evaluated };
|
||||
},
|
||||
};
|
||||
return r;
|
||||
}
|
||||
|
||||
const RULE: QualityRuleSpec = {
|
||||
rule_id: 'q_not_null_customer',
|
||||
tenant_id: 't_alpha',
|
||||
dataset_id: 'ds_sales',
|
||||
rule: 'not_null',
|
||||
field: 'customer',
|
||||
severity: 'block',
|
||||
params: {},
|
||||
};
|
||||
|
||||
describe('BI-P025 — the native provider EXECUTES, and says so', () => {
|
||||
it('declares every rule BI-P025 names', () => {
|
||||
expect([...QUALITY_RULES]).toEqual([
|
||||
'not_null',
|
||||
'unique',
|
||||
'range',
|
||||
'referential',
|
||||
'freshness',
|
||||
'schema',
|
||||
'distribution',
|
||||
'custom',
|
||||
]);
|
||||
});
|
||||
|
||||
it('passes a clean dataset', async () => {
|
||||
const verdict = await new NativeQualityProvider(runner(0, 1000)).evaluate(RULE);
|
||||
expect(verdict.passed).toBe(true);
|
||||
expect(verdict.provider_kind).toBe('executing');
|
||||
});
|
||||
|
||||
it('FAILS an empty dataset instead of reporting zero violations', async () => {
|
||||
// The false green this whole file exists for: a not-null check over zero
|
||||
// rows finds zero violations and looks healthy while the pipeline is dead.
|
||||
const verdict = await new NativeQualityProvider(runner(0, 0)).evaluate(RULE);
|
||||
expect(verdict.passed).toBe(false);
|
||||
expect(verdict.detail).toContain('EMPTY');
|
||||
});
|
||||
|
||||
it('fails a dataset with violations', async () => {
|
||||
const verdict = await new NativeQualityProvider(runner(7, 1000)).evaluate(RULE);
|
||||
expect(verdict.passed).toBe(false);
|
||||
expect(verdict.violation_count).toBe(7);
|
||||
});
|
||||
|
||||
it('binds rule parameters instead of splicing them into the expression', async () => {
|
||||
const r = runner(0, 10);
|
||||
await new NativeQualityProvider(r).evaluate({
|
||||
...RULE,
|
||||
rule: 'distribution',
|
||||
field: 'channel',
|
||||
params: { accepted_values: ['web', "retail'; DROP TABLE x; --"] },
|
||||
});
|
||||
expect(r.lastExpression).not.toContain('DROP');
|
||||
expect(Object.values(r.lastParams ?? {})).toContain("retail'; DROP TABLE x; --");
|
||||
});
|
||||
|
||||
it('refuses a field name that is not an identifier', async () => {
|
||||
const p = new NativeQualityProvider(runner(0, 10));
|
||||
await expect(p.evaluate({ ...RULE, field: 'customer; DROP TABLE x' })).rejects.toBeInstanceOf(SiaxError);
|
||||
});
|
||||
|
||||
it('refuses an incomplete rule rather than evaluating a meaningless one', async () => {
|
||||
const p = new NativeQualityProvider(runner(0, 10));
|
||||
await expect(p.evaluate({ ...RULE, rule: 'range', params: {} })).rejects.toBeInstanceOf(SiaxError);
|
||||
await expect(p.evaluate({ ...RULE, rule: 'freshness', params: {} })).rejects.toBeInstanceOf(SiaxError);
|
||||
await expect(p.evaluate({ ...RULE, rule: 'distribution', params: {} })).rejects.toBeInstanceOf(SiaxError);
|
||||
await expect(p.evaluate({ ...RULE, rule: 'custom', params: {} })).rejects.toBeInstanceOf(SiaxError);
|
||||
});
|
||||
|
||||
it('says CAPABILITY_UNAVAILABLE for rules it cannot evaluate, rather than passing them', async () => {
|
||||
const p = new NativeQualityProvider(runner(0, 10));
|
||||
for (const rule of ['schema', 'referential'] as const) {
|
||||
await expect(p.evaluate({ ...RULE, rule })).rejects.toBeInstanceOf(SiaxError);
|
||||
expect(p.supports(rule)).toBe(false);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('BI-P025 — the ingesting providers do not pretend to execute', () => {
|
||||
it('dbt and GE declare themselves as ingesting', async () => {
|
||||
expect(new DbtQualityProvider(new Map()).providerKind).toBe('ingesting');
|
||||
expect(new GreatExpectationsQualityProvider(new Map()).providerKind).toBe('ingesting');
|
||||
});
|
||||
|
||||
it('a MISSING dbt result is a failure, not a pass', async () => {
|
||||
// Treating "no result" as "passed" is how a gate stops gating the moment
|
||||
// the test stops running.
|
||||
const verdict = await new DbtQualityProvider(new Map()).evaluate(RULE);
|
||||
expect(verdict.passed).toBe(false);
|
||||
expect(verdict.detail).toContain('no dbt result');
|
||||
});
|
||||
|
||||
it.each(['error', 'warn', 'skipped'] as const)('dbt status %s is NOT a pass', async (status) => {
|
||||
const results = new Map([[RULE.rule_id, { unique_id: 't1', status, failures: 0 }]]);
|
||||
const verdict = await new DbtQualityProvider(results).evaluate(RULE);
|
||||
expect(verdict.passed).toBe(false);
|
||||
});
|
||||
|
||||
it('dbt pass is a pass', async () => {
|
||||
const results = new Map([[RULE.rule_id, { unique_id: 't1', status: 'pass' as const, failures: 0 }]]);
|
||||
expect((await new DbtQualityProvider(results).evaluate(RULE)).passed).toBe(true);
|
||||
});
|
||||
|
||||
it('a missing GE suite is a failure', async () => {
|
||||
expect((await new GreatExpectationsQualityProvider(new Map()).evaluate(RULE)).passed).toBe(false);
|
||||
});
|
||||
|
||||
it("a GE suite's unexpected counts are carried through", async () => {
|
||||
const suites = new Map([
|
||||
[
|
||||
RULE.rule_id,
|
||||
{
|
||||
expectation_suite_name: 'sales_suite',
|
||||
success: false,
|
||||
results: [
|
||||
{
|
||||
expectation_config: { expectation_type: 'expect_column_values_to_not_be_null' },
|
||||
success: false,
|
||||
result: { unexpected_count: 12, element_count: 1000 },
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
]);
|
||||
const verdict = await new GreatExpectationsQualityProvider(suites).evaluate(RULE);
|
||||
expect(verdict.passed).toBe(false);
|
||||
expect(verdict.violation_count).toBe(12);
|
||||
expect(verdict.evaluated_rows).toBe(1000);
|
||||
});
|
||||
});
|
||||
|
||||
describe('BI-P025 DoD — a failed critical gate QUARANTINES the serving projection', () => {
|
||||
function verdict(overrides: Partial<QualityVerdict> = {}): QualityVerdict {
|
||||
return {
|
||||
rule_id: 'q1',
|
||||
dataset_id: 'ds_sales',
|
||||
tenant_id: 't_alpha',
|
||||
rule: 'not_null',
|
||||
severity: 'block',
|
||||
passed: true,
|
||||
violation_count: 0,
|
||||
evaluated_rows: 100,
|
||||
produced_by: 'native:clickhouse',
|
||||
provider_kind: 'executing',
|
||||
evaluated_at: new Date().toISOString(),
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
it('a blocking failure quarantines, and the READ is then REFUSED', async () => {
|
||||
const gate = new QualityGate(new InMemoryQuarantineStore());
|
||||
// Before: readable.
|
||||
await expect(gate.assertProjectionReadable('t_alpha', 'ds_sales')).resolves.toBeUndefined();
|
||||
|
||||
const outcome = await gate.apply('t_alpha', 'ds_sales', [verdict({ passed: false, severity: 'block' })]);
|
||||
expect(outcome.quarantined).toBe(true);
|
||||
expect(outcome.passed).toBe(false);
|
||||
|
||||
// After: refused. Not annotated — refused.
|
||||
await expect(gate.assertProjectionReadable('t_alpha', 'ds_sales')).rejects.toBeInstanceOf(SiaxError);
|
||||
});
|
||||
|
||||
it('a WARN failure does not quarantine, but is surfaced', async () => {
|
||||
const gate = new QualityGate(new InMemoryQuarantineStore());
|
||||
const outcome = await gate.apply('t_alpha', 'ds_sales', [verdict({ passed: false, severity: 'warn' })]);
|
||||
expect(outcome.quarantined).toBe(false);
|
||||
expect(outcome.warnings).toHaveLength(1);
|
||||
await expect(gate.assertProjectionReadable('t_alpha', 'ds_sales')).resolves.toBeUndefined();
|
||||
});
|
||||
|
||||
it("a gate over ZERO verdicts QUARANTINES — 'no rules ran' is not 'everything passed'", async () => {
|
||||
// The same shape as l0re's silently-skipped tests and api0's
|
||||
// `trivy --exit-code 0`: a check that produced nothing reporting green.
|
||||
const gate = new QualityGate(new InMemoryQuarantineStore());
|
||||
const outcome = await gate.apply('t_alpha', 'ds_sales', []);
|
||||
expect(outcome.passed).toBe(false);
|
||||
expect(outcome.quarantined).toBe(true);
|
||||
await expect(gate.assertProjectionReadable('t_alpha', 'ds_sales')).rejects.toBeInstanceOf(SiaxError);
|
||||
});
|
||||
|
||||
it('a later passing run does NOT auto-release a quarantine', async () => {
|
||||
const gate = new QualityGate(new InMemoryQuarantineStore());
|
||||
await gate.apply('t_alpha', 'ds_sales', [verdict({ passed: false })]);
|
||||
await gate.apply('t_alpha', 'ds_sales', [verdict({ passed: true })]);
|
||||
// A flapping rule would otherwise quarantine and un-quarantine the dataset
|
||||
// between two reads of the same dashboard.
|
||||
await expect(gate.assertProjectionReadable('t_alpha', 'ds_sales')).rejects.toBeInstanceOf(SiaxError);
|
||||
});
|
||||
|
||||
it('release requires a principal and a reason', async () => {
|
||||
const gate = new QualityGate(new InMemoryQuarantineStore());
|
||||
await gate.apply('t_alpha', 'ds_sales', [verdict({ passed: false })]);
|
||||
await expect(gate.release('t_alpha', 'ds_sales', '', 'x')).rejects.toBeInstanceOf(SiaxError);
|
||||
await expect(gate.release('t_alpha', 'ds_sales', 'u_1', ' ')).rejects.toBeInstanceOf(SiaxError);
|
||||
const released = await gate.release('t_alpha', 'ds_sales', 'u_1', 'upstream backfilled');
|
||||
expect(released.released_by).toBe('u_1');
|
||||
await expect(gate.assertProjectionReadable('t_alpha', 'ds_sales')).resolves.toBeUndefined();
|
||||
});
|
||||
|
||||
it("quarantine is tenant-scoped: one tenant's failure does not block another", async () => {
|
||||
const gate = new QualityGate(new InMemoryQuarantineStore());
|
||||
await gate.apply('t_alpha', 'ds_sales', [verdict({ passed: false })]);
|
||||
await expect(gate.assertProjectionReadable('t_beta', 'ds_sales')).resolves.toBeUndefined();
|
||||
});
|
||||
|
||||
it('refuses a verdict belonging to another tenant or dataset', async () => {
|
||||
const gate = new QualityGate(new InMemoryQuarantineStore());
|
||||
await expect(
|
||||
gate.apply('t_alpha', 'ds_sales', [verdict({ tenant_id: 't_beta' })]),
|
||||
).rejects.toBeInstanceOf(SiaxError);
|
||||
await expect(
|
||||
gate.apply('t_alpha', 'ds_sales', [verdict({ dataset_id: 'ds_other' })]),
|
||||
).rejects.toBeInstanceOf(SiaxError);
|
||||
});
|
||||
});
|
||||
|
||||
describe('BI-P026 — five states, and UNKNOWN is not FRESH', () => {
|
||||
const HOUR = 3_600_000;
|
||||
const NOW = Date.parse('2026-09-07T12:00:00Z');
|
||||
const policy = { expectedIntervalMs: HOUR };
|
||||
|
||||
it('declares the five states BI-P026 names', () => {
|
||||
expect([...FRESHNESS_STATES]).toEqual(['FRESH', 'DELAYED', 'STALE', 'FAILED', 'UNKNOWN']);
|
||||
});
|
||||
|
||||
it('only FRESH may be presented as current', () => {
|
||||
expect(isPresentableAsCurrent('FRESH')).toBe(true);
|
||||
for (const s of NON_CURRENT_STATES) expect(isPresentableAsCurrent(s)).toBe(false);
|
||||
expect(NON_CURRENT_STATES).toContain('UNKNOWN');
|
||||
});
|
||||
|
||||
const cases: Array<[string, Parameters<typeof computeFreshness>[0], string]> = [
|
||||
[
|
||||
'recent load',
|
||||
{ lastSuccessfulLoadAt: '2026-09-07T11:50:00Z', lastRunStatus: 'succeeded', policy, now: NOW },
|
||||
'FRESH',
|
||||
],
|
||||
[
|
||||
'behind schedule',
|
||||
{ lastSuccessfulLoadAt: '2026-09-07T09:50:00Z', lastRunStatus: 'succeeded', policy, now: NOW },
|
||||
'DELAYED',
|
||||
],
|
||||
[
|
||||
'well past the threshold',
|
||||
{ lastSuccessfulLoadAt: '2026-09-06T12:00:00Z', lastRunStatus: 'succeeded', policy, now: NOW },
|
||||
'STALE',
|
||||
],
|
||||
[
|
||||
'the last run FAILED, even though the data looks recent',
|
||||
{ lastSuccessfulLoadAt: '2026-09-07T11:59:00Z', lastRunStatus: 'failed', policy, now: NOW },
|
||||
'FAILED',
|
||||
],
|
||||
['never run', { lastRunStatus: 'never_run', policy, now: NOW }, 'UNKNOWN'],
|
||||
['no load timestamp at all', { policy, now: NOW }, 'UNKNOWN'],
|
||||
['an unparseable timestamp', { lastSuccessfulLoadAt: 'not a date', policy, now: NOW }, 'UNKNOWN'],
|
||||
];
|
||||
|
||||
it.each(cases)('%s -> %s', (_label, input, expected) => {
|
||||
expect(computeFreshness(input).state).toBe(expected);
|
||||
});
|
||||
|
||||
it('FAILED outranks a recent timestamp — a broken pipeline must not look FRESH', () => {
|
||||
const env = computeFreshness({
|
||||
lastSuccessfulLoadAt: '2026-09-07T11:59:59Z',
|
||||
lastRunStatus: 'failed',
|
||||
policy,
|
||||
now: NOW,
|
||||
});
|
||||
expect(env.state).toBe('FAILED');
|
||||
expect(env.disclosure).toContain('FAILED');
|
||||
});
|
||||
|
||||
it('every state carries a disclosure a UI can render verbatim', () => {
|
||||
for (const [, input] of cases) {
|
||||
const env = computeFreshness(input);
|
||||
expect(env.disclosure.length).toBeGreaterThan(10);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('BI-P026 DoD — stale data is never SILENTLY presented as current', () => {
|
||||
const env = computeFreshness({
|
||||
lastSuccessfulLoadAt: '2026-09-01T00:00:00Z',
|
||||
lastRunStatus: 'succeeded',
|
||||
policy: { expectedIntervalMs: 3_600_000 },
|
||||
now: Date.parse('2026-09-07T12:00:00Z'),
|
||||
});
|
||||
|
||||
it('refuses to release a result with NO freshness envelope', () => {
|
||||
// Not defaulted to FRESH, not defaulted to UNKNOWN: unreleasable. "We
|
||||
// forgot to attach it" and "the data is current" are indistinguishable to
|
||||
// a reader, so a default would make forgetting the silent path.
|
||||
expect(() => assertFreshnessDisclosed({}, 'dashboard panel p1')).toThrowError(SiaxError);
|
||||
});
|
||||
|
||||
it('refuses to release STALE data that the consumer has not acknowledged', () => {
|
||||
expect(env.state).toBe('STALE');
|
||||
let thrown: unknown;
|
||||
try {
|
||||
assertFreshnessDisclosed({ freshness: env }, 'dashboard panel p1');
|
||||
} catch (e) {
|
||||
thrown = e;
|
||||
}
|
||||
expect(thrown).toBeInstanceOf(SiaxError);
|
||||
expect((thrown as SiaxError).message).toContain('STALE');
|
||||
});
|
||||
|
||||
it('releases STALE data once the consumer has acknowledged it', () => {
|
||||
expect(assertFreshnessDisclosed({ freshness: env, acknowledged_non_current: true }, 'p1').state).toBe(
|
||||
'STALE',
|
||||
);
|
||||
});
|
||||
|
||||
it.each(NON_CURRENT_STATES)('refuses an unacknowledged %s result', (state) => {
|
||||
expect(() => assertFreshnessDisclosed({ freshness: { ...env, state } }, 'p1')).toThrowError(SiaxError);
|
||||
});
|
||||
|
||||
it('releases FRESH data without ceremony', () => {
|
||||
expect(assertFreshnessDisclosed({ freshness: { ...env, state: 'FRESH' } }, 'p1').state).toBe('FRESH');
|
||||
});
|
||||
|
||||
it('refuses an envelope carrying a state that is not one of the five', () => {
|
||||
expect(() =>
|
||||
assertFreshnessDisclosed({ freshness: { ...env, state: 'PROBABLY_FINE' as never } }, 'p1'),
|
||||
).toThrowError(SiaxError);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,244 @@
|
||||
/**
|
||||
* tests/integration/data-quality-clickhouse.integration.test.ts
|
||||
* ==============================================================
|
||||
* BI-P025 — the native quality provider against a live ClickHouse.
|
||||
*
|
||||
* The unit suite proves the provider's LOGIC (an empty dataset is a failure, a
|
||||
* missing dbt result is a failure). This suite proves the part a fake runner
|
||||
* cannot: that the compiled expressions are valid ClickHouse and count the
|
||||
* right rows, including the empty-dataset case where a real engine returns
|
||||
* zero violations over zero rows and every naive implementation reports green.
|
||||
*
|
||||
* It also re-proves BI-P011's forced tenant predicate on this path: a quality
|
||||
* run executes on an ADMINISTRATIVE credential (it must read the base table),
|
||||
* so if the predicate were missing here, one tenant's dirty rows would
|
||||
* quarantine another tenant's projection.
|
||||
*
|
||||
* Gated on CLICKHOUSE_URL. With the variable set and the server down, this
|
||||
* suite FAILS rather than skipping.
|
||||
*/
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import { beforeAll, describe, expect, it } from 'vitest';
|
||||
import { NativeQualityProvider, type QualityRuleSpec } from '@/src/analytics/quality/quality-provider';
|
||||
import { ClickHouseQualityRunner } from '@/src/analytics/quality/clickhouse-quality-runner';
|
||||
import { InMemoryQuarantineStore, QualityGate } from '@/src/analytics/quality/quarantine';
|
||||
|
||||
const BASE_URL = process.env.CLICKHOUSE_URL ?? '';
|
||||
const AUTH = process.env.CLICKHOUSE_AUTH ?? 'default:';
|
||||
const suite = BASE_URL ? describe : describe.skip;
|
||||
const DB = 'opt_quality_test';
|
||||
const EVIDENCE_PATH = path.join(process.cwd(), 'evidence/bi-p025/native-quality-live.json');
|
||||
|
||||
async function admin(sql: string): Promise<{ ok: boolean; body: string }> {
|
||||
const url = new URL(BASE_URL);
|
||||
url.searchParams.set('query', sql);
|
||||
const res = await fetch(url.toString(), {
|
||||
method: 'POST',
|
||||
headers: { Authorization: `Basic ${Buffer.from(AUTH).toString('base64')}` },
|
||||
body: '',
|
||||
signal: AbortSignal.timeout(60_000),
|
||||
});
|
||||
return { ok: res.ok, body: await res.text() };
|
||||
}
|
||||
|
||||
const runner = new ClickHouseQualityRunner({
|
||||
baseUrl: BASE_URL || 'http://unset',
|
||||
basicAuth: AUTH,
|
||||
database: DB,
|
||||
tables: { ds_sales: 'sales', ds_empty: 'empty_sales' },
|
||||
});
|
||||
const provider = new NativeQualityProvider(runner);
|
||||
const evidence: Record<string, unknown> = {};
|
||||
|
||||
function rule(over: Partial<QualityRuleSpec>): QualityRuleSpec {
|
||||
return {
|
||||
rule_id: 'q1',
|
||||
tenant_id: 't_alpha',
|
||||
dataset_id: 'ds_sales',
|
||||
rule: 'not_null',
|
||||
field: 'customer',
|
||||
severity: 'block',
|
||||
params: {},
|
||||
...over,
|
||||
};
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
if (!BASE_URL) return;
|
||||
const v = await admin('SELECT version()');
|
||||
if (!v.ok) throw new Error('CLICKHOUSE_URL is set but ClickHouse is not answering');
|
||||
evidence.clickhouse_version = v.body.trim();
|
||||
evidence.recorded_at = new Date().toISOString();
|
||||
|
||||
await admin(`DROP DATABASE IF EXISTS ${DB}`);
|
||||
await admin(`CREATE DATABASE ${DB}`);
|
||||
await admin(
|
||||
`CREATE TABLE ${DB}.sales (tenant_id String, dataset_id String, customer String, amount Float64, ` +
|
||||
`channel String, event_time DateTime) ENGINE = MergeTree ORDER BY (tenant_id, event_time)`,
|
||||
);
|
||||
await admin(
|
||||
`CREATE TABLE ${DB}.empty_sales (tenant_id String, dataset_id String, customer String, amount Float64, ` +
|
||||
`channel String, event_time DateTime) ENGINE = MergeTree ORDER BY (tenant_id, event_time)`,
|
||||
);
|
||||
|
||||
// t_alpha: 100 clean rows, then deliberate defects.
|
||||
await admin(
|
||||
`INSERT INTO ${DB}.sales SELECT 't_alpha', 'ds_sales', 'cust_' || toString(number), number, 'web', now() ` +
|
||||
`FROM numbers(100)`,
|
||||
);
|
||||
// 5 rows with an empty customer (the not_null violation).
|
||||
await admin(
|
||||
`INSERT INTO ${DB}.sales SELECT 't_alpha', 'ds_sales', '', number, 'web', now() FROM numbers(5)`,
|
||||
);
|
||||
// 3 rows out of range (amount > 1000).
|
||||
await admin(
|
||||
`INSERT INTO ${DB}.sales SELECT 't_alpha', 'ds_sales', 'big_' || toString(number), 5000, 'web', now() ` +
|
||||
`FROM numbers(3)`,
|
||||
);
|
||||
// 4 rows with an unaccepted channel.
|
||||
await admin(
|
||||
`INSERT INTO ${DB}.sales SELECT 't_alpha', 'ds_sales', 'ch_' || toString(number), 1, 'carrier_pigeon', now() ` +
|
||||
`FROM numbers(4)`,
|
||||
);
|
||||
// 6 rows sharing two duplicate keys (3 each) — the unique violation.
|
||||
await admin(
|
||||
`INSERT INTO ${DB}.sales SELECT 't_alpha', 'ds_sales', 'dup_' || toString(number % 2), number, 'web', now() ` +
|
||||
`FROM numbers(6)`,
|
||||
);
|
||||
|
||||
// t_beta: FILTHY. Every rule fails here. If the tenant predicate is missing,
|
||||
// t_alpha's counts jump and this tenant quarantines the other one.
|
||||
await admin(
|
||||
`INSERT INTO ${DB}.sales SELECT 't_beta', 'ds_sales', '', 999999, 'carrier_pigeon', now() FROM numbers(500)`,
|
||||
);
|
||||
}, 120_000);
|
||||
|
||||
suite('BI-P025 live — the compiled expressions are valid SQL and count the right rows', () => {
|
||||
it("not_null finds exactly the 5 empty-customer rows, out of t_alpha's 118", async () => {
|
||||
const verdict = await provider.evaluate(rule({ rule: 'not_null', field: 'customer' }));
|
||||
expect(verdict.evaluated_rows).toBe(118);
|
||||
expect(verdict.violation_count).toBe(5);
|
||||
expect(verdict.passed).toBe(false);
|
||||
evidence.not_null = { evaluated: verdict.evaluated_rows, violations: verdict.violation_count };
|
||||
});
|
||||
|
||||
it('range finds exactly the 3 rows over the maximum', async () => {
|
||||
const verdict = await provider.evaluate(
|
||||
rule({ rule: 'range', field: 'amount', params: { min: 0, max: 1000 } }),
|
||||
);
|
||||
expect(verdict.violation_count).toBe(3);
|
||||
evidence.range = { violations: verdict.violation_count };
|
||||
});
|
||||
|
||||
it('distribution finds exactly the 4 rows outside the accepted values', async () => {
|
||||
const verdict = await provider.evaluate(
|
||||
rule({ rule: 'distribution', field: 'channel', params: { accepted_values: ['web', 'retail'] } }),
|
||||
);
|
||||
expect(verdict.violation_count).toBe(4);
|
||||
evidence.distribution = { violations: verdict.violation_count };
|
||||
});
|
||||
|
||||
it('unique counts ROWS sitting on a duplicated key, not the number of keys', async () => {
|
||||
const verdict = await provider.evaluate(rule({ rule: 'unique', field: 'customer' }));
|
||||
// 11, and the arithmetic is worth writing down because the first version
|
||||
// of this test asserted 6 and the engine disagreed:
|
||||
// * `dup_0` and `dup_1` appear 3 times each -> 6 rows
|
||||
// * the five EMPTY customers are all the same key '' -> 5 rows
|
||||
// The empty strings are duplicates of one another, so `not_null` and
|
||||
// `unique` overlap on the same defect. That is the engine being right and
|
||||
// the fixture arithmetic being wrong, not a rule that over-counts.
|
||||
expect(verdict.violation_count).toBe(11);
|
||||
evidence.unique = {
|
||||
violations: verdict.violation_count,
|
||||
breakdown: '6 rows on dup_0/dup_1, plus 5 rows sharing the empty-string key',
|
||||
};
|
||||
});
|
||||
|
||||
it('freshness finds no violation for rows written just now', async () => {
|
||||
const verdict = await provider.evaluate(
|
||||
rule({ rule: 'freshness', params: { timestamp_field: 'event_time', max_age_seconds: 3600 } }),
|
||||
);
|
||||
expect(verdict.violation_count).toBe(0);
|
||||
expect(verdict.passed).toBe(true);
|
||||
});
|
||||
|
||||
it('a hostile accepted-value is bound, not executed', async () => {
|
||||
const verdict = await provider.evaluate(
|
||||
rule({
|
||||
rule: 'distribution',
|
||||
field: 'channel',
|
||||
params: { accepted_values: ['web', 'retail', "'); DROP TABLE sales; --"] },
|
||||
}),
|
||||
);
|
||||
expect(verdict.violation_count).toBe(4);
|
||||
// The table is still there, with its rows.
|
||||
const after = await admin(`SELECT count() FROM ${DB}.sales`);
|
||||
expect(Number(after.body.trim())).toBe(618);
|
||||
});
|
||||
});
|
||||
|
||||
suite('BI-P025 live — the EMPTY dataset false green, against a real engine', () => {
|
||||
it('a real ClickHouse returns 0 violations over 0 rows, and the provider calls it a FAILURE', async () => {
|
||||
// The counterfactual first: the raw counts genuinely look clean.
|
||||
const raw = await runner.count({
|
||||
tenant_id: 't_alpha',
|
||||
dataset_id: 'ds_empty',
|
||||
violationExpression: "customer IS NULL OR customer = ''",
|
||||
params: {},
|
||||
});
|
||||
expect(raw).toEqual({ violations: 0, evaluated: 0 });
|
||||
|
||||
// A naive implementation reports green here. This one does not.
|
||||
const verdict = await provider.evaluate(rule({ dataset_id: 'ds_empty' }));
|
||||
expect(verdict.passed).toBe(false);
|
||||
expect(verdict.detail).toContain('EMPTY');
|
||||
evidence.empty_dataset = {
|
||||
raw_counts_from_clickhouse: raw,
|
||||
verdict_passed: verdict.passed,
|
||||
note: 'a real engine returns zero violations over zero rows; the provider refuses to call that a pass',
|
||||
};
|
||||
});
|
||||
});
|
||||
|
||||
suite("BI-P025 live — a quality run cannot read another tenant's rows", () => {
|
||||
it("t_alpha's counts are unaffected by t_beta's 500 filthy rows", async () => {
|
||||
const all = await admin(`SELECT count() FROM ${DB}.sales`);
|
||||
expect(Number(all.body.trim())).toBe(618); // 118 alpha + 500 beta
|
||||
|
||||
const alpha = await provider.evaluate(
|
||||
rule({ tenant_id: 't_alpha', rule: 'not_null', field: 'customer' }),
|
||||
);
|
||||
const beta = await provider.evaluate(rule({ tenant_id: 't_beta', rule: 'not_null', field: 'customer' }));
|
||||
|
||||
expect(alpha.evaluated_rows).toBe(118);
|
||||
expect(alpha.violation_count).toBe(5);
|
||||
expect(beta.evaluated_rows).toBe(500);
|
||||
expect(beta.violation_count).toBe(500);
|
||||
evidence.tenant_isolation = {
|
||||
base_table_rows: 618,
|
||||
alpha_evaluated: alpha.evaluated_rows,
|
||||
alpha_violations: alpha.violation_count,
|
||||
beta_evaluated: beta.evaluated_rows,
|
||||
beta_violations: beta.violation_count,
|
||||
};
|
||||
});
|
||||
|
||||
it("t_beta's blocking failure quarantines t_beta only", async () => {
|
||||
const gate = new QualityGate(new InMemoryQuarantineStore());
|
||||
const beta = await provider.evaluate(rule({ tenant_id: 't_beta', rule: 'not_null', field: 'customer' }));
|
||||
await gate.apply('t_beta', 'ds_sales', [beta]);
|
||||
await expect(gate.assertProjectionReadable('t_beta', 'ds_sales')).rejects.toThrow(/QUARANTINED/);
|
||||
await expect(gate.assertProjectionReadable('t_alpha', 'ds_sales')).resolves.toBeUndefined();
|
||||
});
|
||||
|
||||
it('refuses a dataset with no registered physical table rather than guessing one', async () => {
|
||||
await expect(provider.evaluate(rule({ dataset_id: 'ds_unregistered' }))).rejects.toThrow();
|
||||
});
|
||||
|
||||
it('writes the evidence file', () => {
|
||||
fs.mkdirSync(path.dirname(EVIDENCE_PATH), { recursive: true });
|
||||
fs.writeFileSync(EVIDENCE_PATH, `${JSON.stringify(evidence, null, 2)}\n`);
|
||||
expect(fs.existsSync(EVIDENCE_PATH)).toBe(true);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user