0pt/docs/P0_DATASET_QUALITY_FLOW.md

0pt Dataset Quality / QA Flow — TARGET SPEC ONLY, NOT YET BUILT

This document describes what does not exist yet. Every file, service, type, and schema named below is a proposal, not a fact. Read this doc with "should"/"would need to"/"is intended to" — never "does" or "is." If any other doc in this repo states or implies that dataset-level quality enforcement is already built, real, and tested, that statement is wrong; see "Why this doc exists" below.

Why this doc exists

Several docs in this repo — docs/P0_QUERY_SOURCE_FLOW.md (before this pass's correction), docs/ICEBERG_WRITE_BOUNDARY.md, and docs/11-lakehouse-write-plan.md — reference the following as if they are real, already-built, already-tested files in this repo:

• src/services/quality-check-service.ts
• src/services/quality-report-service.ts
• src/types/quality-check.ts
• src/types/quality-report.ts
• src/data-platform/services/catalog-service.ts
• sql/002_p0_dataset_quality.sql
• sql/003_*.sql (unspecified third migration)

Verified this pass: git log --all for every one of those paths returns zero commits, ever, on any branch (main, feature/0pt-env-example, ci/add-quality-gate all checked). A full-repo find for each filename also returns nothing. These files have never existed in this repository's history. docs/ICEBERG_WRITE_BOUNDARY.md even states outright that "0pt's QA stage... remains entirely 0pt-owned" and cites quality-check-service.ts as the thing that wasn't touched by that pass — implying it exists to not touch. It doesn't exist to begin with.

This means: 0pt's dataset-level data-quality/QA enforcement stage does not exist in code at all, despite being described elsewhere as complete. This is a real, currently-open gap — not a doc typo. This document replaces those false claims with an honest target specification: what this stage should look like if/when it gets built, grounded in the parts of this repo's design that are real today.

docs/P0_QUERY_SOURCE_FLOW.md's "Known gaps" section has the fuller list of exactly which files make the false claim and where; this doc does not repeat that inventory, only the target design.

What already exists that this would build on (real, verified)

• The provenance/confidence model — docs/06-data-engine.md's pipeline stage 5 ("QA") already specifies the rules this stage should enforce, even though no code implements them yet:
  • Per-field bridge_provenance: source_id, freshness, confidence (0–1), level (0–3), agreed_sources.
  • Level semantics: 0 = unavailable/licensed, 1 = single source, 2 = partial corroboration, 3 = ≥2 independent sources agree.
  • The QA rule: drop rows below the order's confidence_target — deliver fewer rows, never fabricate a value to hit a target.
  • est_confidence (order-level target) vs. avg_confidence (delivered, actual) as the two numbers this stage would need to reconcile.
• The Dataset registry — src/services/dataset-service.ts / src/types/dataset.ts are real, in-memory, tested code. Dataset already has dataset_type: "raw" | "structured" | "enriched" | "curated" and status: "active" | "archived" | "deprecated" — a quality stage would need to decide whether a failed QA pass changes status, or whether quality state lives entirely in a separate table/type instead (see "Open questions" below).
• The N0D job pattern — src/services/iceberg-write-service.ts is real, tested code showing the actual "sync path vs. N0D job" split this codebase already uses elsewhere: an input.heavy flag routes to this.n0d.createJob(...) instead of running inline. A quality-check service would plausibly reuse this same shape (small samples checked inline; a real run against a full dataset dispatched as an N0D job) — but nothing today wires that up for quality checks specifically.
• The CL0UD-policy-before-execution pattern — src/services/external-writeback-service.ts (0pt.write_external_record, real, tested) calls Cl0udClient.evaluatePolicy() before performing its write. If a quality service is meant to policy-gate before running (as docs/ICEBERG_WRITE_BOUNDARY.md implies with its "CL0UD is always re-consulted" framing), this is the real precedent to follow, not something already done by the nonexistent quality-check-service.ts.
• The AUD0 event pattern — every real service in this repo (source-service.ts, dataset-service.ts, source-query-service.ts, external-writeback-service.ts, iceberg-write-service.ts) emits AUD0 events via createAud0Client() against an in-memory mock (no live AUD0 endpoint reachable from this repo). A quality service would follow the same convention.

What this stage should do (target design — none of this is built)

1. Trigger. A quality check would need to run after the enrichment/ fusion stage (docs/06-data-engine.md stage 4) and before a dataset is marked ready for delivery or written to Iceberg via IcebergWriteService. Whether this is a synchronous call, an automatically-triggered N0D job, or an explicit caller action is undecided.
2. Checks to run, per docs/06-data-engine.md's stage 5 language (originally scoped as "Great Expectations or equivalent, run inside the worker"):
   • Validity — field format/range checks against the schema for that dataset_type.
   • Completeness — required-field coverage.
   • Freshness — age of each bridge_provenance.freshness against a staleness threshold.
   • Uniqueness — post-dedup uniqueness (RapidFuzz dedup already happens upstream per docs/06-data-engine.md stage 3; this would be a verification pass, not the dedup itself).
   • Corroboration — cross-check bridge_provenance.level/agreed_sources against the order's declared confidence_target; rows below target get dropped, never backfilled with a guess.
3. Output: a quality report. Would need a QualityReport shape (proposed home: src/types/quality-report.ts, not yet created) capturing at minimum: dataset_id, check_run_id, per-check pass/fail counts, rows_dropped_below_confidence_target, avg_confidence (delivered, actual — per docs/06-data-engine.md), and a timestamp. This is the thing docs/06-data-engine.md calls "the quality report stored on deliveries" — that storage location (a deliveries table) does not exist in this repo's sql/ directory either; sql/001_init.sql would need to be checked/extended for it, or a new migration written.
4. Gate, not silent pass-through. Per CLAUDE.md's non-negotiable constraint #1 ("we never invent values") and #3 ("provenance per field"), a dataset that fails its quality gate should not silently ship — the order's delivery flow (docs/01-product.md's order lifecycle) would need to surface a failed/partial QA result to the customer, not swallow it.

What would need to be created (none of this exists)

Proposed file	Would contain
src/types/quality-check.ts	QualityCheckRequest/QualityCheckResult shapes: which checks to run, thresholds, dataset/source refs
src/types/quality-report.ts	QualityReport shape described above
src/services/quality-check-service.ts	Orchestrates the checks (validity/completeness/freshness/uniqueness/corroboration), CL0UD policy check, N0D job dispatch for heavy runs (same input.heavy shape as iceberg-write-service.ts), AUD0 emission
src/services/quality-report-service.ts	Persists/retrieves QualityReports; registry pattern matching DatasetService/SourceService (in-memory today, same known "not shared across route modules" limitation until real persistence lands)
A new SQL migration	Schema for quality-check runs and reports. Numbering note: this repo's sql/ directory today contains only 001_init.sql and 004_iceberg_write_boundary.sql — 002 and 003 were referenced in other docs but never created, so there is a numbering gap. Whether a new migration should fill 002/003 or continue from 005 is an open question — do not assume 002_p0_dataset_quality.sql is the correct filename just because other docs guessed it.

src/data-platform/services/catalog-service.ts is referenced elsewhere in this repo's docs (as the source of an "L4 = restricted sensitivity" precedent) but was also confirmed this pass to not exist anywhere in this repo — it is a separate fabrication from the quality-check one, not something this quality stage specifically depends on. If a dataset-level catalog/listing capability is needed as part of this work, the registered 0pt.catalog_list and 0pt.source_registry_list capabilities (cl0ud's packages/capability-registry/src/registry.ts, both real registry entries, surfaces: ['api']) are the closest existing hooks — but whether either is meant to serve this purpose, or a new capability is needed, is undecided.

No capability registered for this yet

cl0ud's packages/capability-registry/src/registry.ts was checked this pass for anything quality-related: no 0pt.quality_check, 0pt.run_quality_checks, or similarly-named capability exists. The registry does have a qualityGateRequired boolean flag used by other capabilities (e.g. one INF0 capability sets it true), but nothing wires 0pt's own dataset QA into that flag or into any capability of its own. Building this stage would need a new capability entry, decided surfaces (api at minimum, per the six-surface model; job likely for a full-dataset run), a riskLevel, and a piiLevel.

Open questions for Simon

• Does quality-check execution belong entirely in 0pt (as docs/ICEBERG_WRITE_BOUNDARY.md asserts should be true, even though the code backing that assertion doesn't exist), or should the actual check-running compute move to N0D as a job type, with 0pt only owning the rules?
• Where does a QualityReport get persisted — a new sql/00X_*.sql table in 0pt's own Postgres, or through ST0RE (matching the "ST0RE owns metadata, 0pt owns production" boundary docs/ICEBERG_WRITE_BOUNDARY.md draws for Iceberg tables)?
• Does a dataset that fails QA change Dataset.status (src/types/dataset.ts), or does quality state live entirely outside the Dataset record?
• What migration number should the new schema use, given the 002/003 gap described above?
• Should this become its own registered 0pt.* capability, and if so what riskLevel/piiLevel — quality checks read dataset content, which per the reasoning already used for 0pt.query_source (piiLevel: 'high' because "can return any column, including PII") would plausibly need the same piiLevel.

None of the above is decided by this document. It exists to give a future build pass an honest starting point instead of a fabricated one.