# 0pt Dataset Quality / QA Flow — TARGET SPEC ONLY, NOT YET BUILT > **This document describes what does not exist yet.** Every file, service, > type, and schema named below is a proposal, not a fact. Read this doc with > "should"/"would need to"/"is intended to" — never "does" or "is." If any > other doc in this repo states or implies that dataset-level quality > enforcement is already built, real, and tested, that statement is wrong; > see "Why this doc exists" below. ## Why this doc exists Several docs in this repo — `docs/P0_QUERY_SOURCE_FLOW.md` (before this pass's correction), `docs/ICEBERG_WRITE_BOUNDARY.md`, and `docs/11-lakehouse-write-plan.md` — reference the following as if they are real, already-built, already-tested files in this repo: - `src/services/quality-check-service.ts` - `src/services/quality-report-service.ts` - `src/types/quality-check.ts` - `src/types/quality-report.ts` - `src/data-platform/services/catalog-service.ts` - `sql/002_p0_dataset_quality.sql` - `sql/003_*.sql` (unspecified third migration) **Verified this pass:** `git log --all` for every one of those paths returns **zero commits, ever, on any branch** (`main`, `feature/0pt-env-example`, `ci/add-quality-gate` all checked). A full-repo `find` for each filename also returns nothing. These files have never existed in this repository's history. `docs/ICEBERG_WRITE_BOUNDARY.md` even states outright that "0pt's QA stage... remains entirely 0pt-owned" and cites `quality-check-service.ts` as the thing that wasn't touched by that pass — implying it exists to not touch. It doesn't exist to begin with. This means: **0pt's dataset-level data-quality/QA enforcement stage does not exist in code at all**, despite being described elsewhere as complete. This is a real, currently-open gap — not a doc typo. This document replaces those false claims with an honest target specification: what this stage *should* look like if/when it gets built, grounded in the parts of this repo's design that *are* real today. `docs/P0_QUERY_SOURCE_FLOW.md`'s "Known gaps" section has the fuller list of exactly which files make the false claim and where; this doc does not repeat that inventory, only the target design. ## What already exists that this would build on (real, verified) - **The provenance/confidence model** — `docs/06-data-engine.md`'s pipeline stage 5 ("QA") already specifies the *rules* this stage should enforce, even though no code implements them yet: - Per-field `bridge_provenance`: `source_id`, `freshness`, `confidence` (0–1), `level` (0–3), `agreed_sources`. - Level semantics: 0 = unavailable/licensed, 1 = single source, 2 = partial corroboration, 3 = ≥2 independent sources agree. - The QA rule: **drop rows below the order's `confidence_target`** — deliver fewer rows, never fabricate a value to hit a target. - `est_confidence` (order-level target) vs. `avg_confidence` (delivered, actual) as the two numbers this stage would need to reconcile. - **The Dataset registry** — `src/services/dataset-service.ts` / `src/types/dataset.ts` are real, in-memory, tested code. `Dataset` already has `dataset_type: "raw" | "structured" | "enriched" | "curated"` and `status: "active" | "archived" | "deprecated"` — a quality stage would need to decide whether a failed QA pass changes `status`, or whether quality state lives entirely in a separate table/type instead (see "Open questions" below). - **The N0D job pattern** — `src/services/iceberg-write-service.ts` is real, tested code showing the actual "sync path vs. N0D job" split this codebase already uses elsewhere: an `input.heavy` flag routes to `this.n0d.createJob(...)` instead of running inline. A quality-check service would plausibly reuse this same shape (small samples checked inline; a real run against a full dataset dispatched as an N0D job) — but nothing today wires that up for quality checks specifically. - **The CL0UD-policy-before-execution pattern** — `src/services/external-writeback-service.ts` (`0pt.write_external_record`, real, tested) calls `Cl0udClient.evaluatePolicy()` before performing its write. If a quality service is meant to policy-gate before running (as `docs/ICEBERG_WRITE_BOUNDARY.md` implies with its "CL0UD is always re-consulted" framing), this is the real precedent to follow, not something already done by the nonexistent `quality-check-service.ts`. - **The AUD0 event pattern** — every real service in this repo (`source-service.ts`, `dataset-service.ts`, `source-query-service.ts`, `external-writeback-service.ts`, `iceberg-write-service.ts`) emits AUD0 events via `createAud0Client()` against an in-memory mock (no live AUD0 endpoint reachable from this repo). A quality service would follow the same convention. ## What this stage should do (target design — none of this is built) 1. **Trigger.** A quality check would need to run after the enrichment/ fusion stage (`docs/06-data-engine.md` stage 4) and before a dataset is marked ready for delivery or written to Iceberg via `IcebergWriteService`. Whether this is a synchronous call, an automatically-triggered N0D job, or an explicit caller action is undecided. 2. **Checks to run**, per `docs/06-data-engine.md`'s stage 5 language (originally scoped as "Great Expectations or equivalent, run inside the worker"): - **Validity** — field format/range checks against the schema for that `dataset_type`. - **Completeness** — required-field coverage. - **Freshness** — age of each `bridge_provenance.freshness` against a staleness threshold. - **Uniqueness** — post-dedup uniqueness (RapidFuzz dedup already happens upstream per `docs/06-data-engine.md` stage 3; this would be a verification pass, not the dedup itself). - **Corroboration** — cross-check `bridge_provenance.level`/`agreed_sources` against the order's declared `confidence_target`; rows below target get dropped, never backfilled with a guess. 3. **Output: a quality report.** Would need a `QualityReport` shape (proposed home: `src/types/quality-report.ts`, not yet created) capturing at minimum: `dataset_id`, `check_run_id`, per-check pass/fail counts, `rows_dropped_below_confidence_target`, `avg_confidence` (delivered, actual — per `docs/06-data-engine.md`), and a timestamp. This is the thing `docs/06-data-engine.md` calls "the quality report stored on `deliveries`" — that storage location (a `deliveries` table) does not exist in this repo's `sql/` directory either; `sql/001_init.sql` would need to be checked/extended for it, or a new migration written. 4. **Gate, not silent pass-through.** Per CLAUDE.md's non-negotiable constraint #1 ("we never invent values") and #3 ("provenance per field"), a dataset that fails its quality gate should not silently ship — the order's delivery flow (`docs/01-product.md`'s order lifecycle) would need to surface a failed/partial QA result to the customer, not swallow it. ## What would need to be created (none of this exists) | Proposed file | Would contain | |---|---| | `src/types/quality-check.ts` | `QualityCheckRequest`/`QualityCheckResult` shapes: which checks to run, thresholds, dataset/source refs | | `src/types/quality-report.ts` | `QualityReport` shape described above | | `src/services/quality-check-service.ts` | Orchestrates the checks (validity/completeness/freshness/uniqueness/corroboration), CL0UD policy check, N0D job dispatch for heavy runs (same `input.heavy` shape as `iceberg-write-service.ts`), AUD0 emission | | `src/services/quality-report-service.ts` | Persists/retrieves `QualityReport`s; registry pattern matching `DatasetService`/`SourceService` (in-memory today, same known "not shared across route modules" limitation until real persistence lands) | | A new SQL migration | Schema for quality-check runs and reports. **Numbering note:** this repo's `sql/` directory today contains only `001_init.sql` and `004_iceberg_write_boundary.sql` — `002` and `003` were referenced in other docs but never created, so there is a numbering gap. Whether a new migration should fill `002`/`003` or continue from `005` is an open question — do not assume `002_p0_dataset_quality.sql` is the correct filename just because other docs guessed it. | `src/data-platform/services/catalog-service.ts` is referenced elsewhere in this repo's docs (as the source of an "`L4` = restricted sensitivity" precedent) but was also confirmed this pass to not exist anywhere in this repo — it is a separate fabrication from the quality-check one, not something this quality stage specifically depends on. If a dataset-level catalog/listing capability is needed as part of this work, the registered `0pt.catalog_list` and `0pt.source_registry_list` capabilities (`cl0ud`'s `packages/capability-registry/src/registry.ts`, both real registry entries, `surfaces: ['api']`) are the closest existing hooks — but whether either is meant to serve this purpose, or a new capability is needed, is undecided. ## No capability registered for this yet `cl0ud`'s `packages/capability-registry/src/registry.ts` was checked this pass for anything quality-related: no `0pt.quality_check`, `0pt.run_quality_checks`, or similarly-named capability exists. The registry does have a `qualityGateRequired` boolean flag used by *other* capabilities (e.g. one INF0 capability sets it `true`), but nothing wires `0pt`'s own dataset QA into that flag or into any capability of its own. Building this stage would need a new capability entry, decided surfaces (`api` at minimum, per the six-surface model; `job` likely for a full-dataset run), a `riskLevel`, and a `piiLevel`. ## Open questions for Simon - Does quality-check execution belong entirely in 0pt (as `docs/ICEBERG_WRITE_BOUNDARY.md` asserts should be true, even though the code backing that assertion doesn't exist), or should the actual check-running compute move to N0D as a job type, with 0pt only owning the *rules*? - Where does a `QualityReport` get persisted — a new `sql/00X_*.sql` table in 0pt's own Postgres, or through ST0RE (matching the "ST0RE owns metadata, 0pt owns production" boundary `docs/ICEBERG_WRITE_BOUNDARY.md` draws for Iceberg tables)? - Does a dataset that fails QA change `Dataset.status` (`src/types/dataset.ts`), or does quality state live entirely outside the `Dataset` record? - What migration number should the new schema use, given the `002`/`003` gap described above? - Should this become its own registered `0pt.*` capability, and if so what `riskLevel`/`piiLevel` — quality checks read dataset content, which per the reasoning already used for `0pt.query_source` (`piiLevel: 'high'` because "can return any column, including PII") would plausibly need the same `piiLevel`. None of the above is decided by this document. It exists to give a future build pass an honest starting point instead of a fabricated one.