* docs: full accuracy overhaul, grounded claim-by-claim in real code Every doc in this repo (CLAUDE.md, README.md, SECURITY.md, docs/*.md) was audited and rewritten by a multi-pass process: independent rewrite per cluster, adversarial re-verification of that rewrite against the actual code, then a final cross-doc consistency pass — each stage re-checking claims against the repo directly rather than trusting the previous stage. Two corrections matter most: 1. docs/ICEBERG_WRITE_BOUNDARY.md, docs/11-lakehouse-write-plan.md, and docs/P0_QUERY_SOURCE_FLOW.md asserted src/services/quality-check-service.ts, quality-report-service.ts, src/data-platform/services/catalog-service.ts, and sql/002_p0_dataset_quality.sql/sql/003_*.sql as real, tested code. git log --all across every branch shows zero commits for any of these paths, ever. 0pt's data-quality/QA enforcement does not exist in code. Corrected everywhere, with a new docs/P0_DATASET_QUALITY_FLOW.md added as an explicit target-spec (not-yet-built) replacement. 2. github.com/sax3l/0pt (this repo) and git.siax.io/sax3l/0pt (Gitea) are two diverged remotes sharing part of the same history (services/api/ workers/adaptive_router.py is byte-identical between them). Gitea's main independently accumulated a WAF-bypass/proxy-rotation scraper stack in three 2026-07-13 commits, since removed via a Gitea PR; GitHub's main never had it. Which remote is canonical/deployed is unresolved — flagged consistently across the corpus, not decided here. Also newly and consistently documented: the tenant_id/project_id-read-from- client-body gap on the P0 lakehouse/rate-policy/source-query routes (no auth/session system exists anywhere in this repo yet); that every CL0UD/ N0D/AUD0/ST0RE platform client is real but mock-only (zero process.env reads for any of their *_URL vars); that two P0 mutation routes (lakehouse/mappings, sources/rate-policies) always fail over HTTP today because their prerequisite entity can never be created via any route; a schema bug in sql/004 (FK references to opt_datasets/opt_sources, which don't exist); and that 0pt.app's live deployment was independently confirmed (curled this pass) to be running this exact codebase with DATABASE_URL/REDIS_URL unset in production. New file: docs/12-edge-capability-api.md — the edge-native P0 TypeScript capability contract (app/api/0pt/*, /api/screen, /api/ai/parse), previously undocumented and distinct from docs/04-api.md's FastAPI-engine-only scope. Verified before commit: tsc --noEmit clean; vitest run 88 passed, 1 todo, 0 failures (13 files) — docs-only change, no source touched. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs: reflect PR #5's lakehouse-mapping-registry + status-code fixes Follow-up to keep docs/12-edge-capability-api.md accurate once both PRs land: fix/0pt-full-bug-hunt (#5) fixed the shared-instance bug in the lakehouse mapping routes and added proper status-code mapping to the document extraction route. This doc was written against the pre-fix state in the same pass that added it, so it needs the same correction. Does not claim the lakehouse write path fully works now -- the deeper dataset-registration gap (no route ever calls DatasetService.registerDataset) is unchanged and still blocks it end-to-end; only the shared-instance half of the bug is fixed. --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
181 lines
11 KiB
Markdown
181 lines
11 KiB
Markdown
# 0pt Dataset Quality / QA Flow — TARGET SPEC ONLY, NOT YET BUILT
|
||
|
||
> **This document describes what does not exist yet.** Every file, service,
|
||
> type, and schema named below is a proposal, not a fact. Read this doc with
|
||
> "should"/"would need to"/"is intended to" — never "does" or "is." If any
|
||
> other doc in this repo states or implies that dataset-level quality
|
||
> enforcement is already built, real, and tested, that statement is wrong;
|
||
> see "Why this doc exists" below.
|
||
|
||
## Why this doc exists
|
||
|
||
Several docs in this repo — `docs/P0_QUERY_SOURCE_FLOW.md` (before this
|
||
pass's correction), `docs/ICEBERG_WRITE_BOUNDARY.md`, and
|
||
`docs/11-lakehouse-write-plan.md` — reference the following as if they are
|
||
real, already-built, already-tested files in this repo:
|
||
|
||
- `src/services/quality-check-service.ts`
|
||
- `src/services/quality-report-service.ts`
|
||
- `src/types/quality-check.ts`
|
||
- `src/types/quality-report.ts`
|
||
- `src/data-platform/services/catalog-service.ts`
|
||
- `sql/002_p0_dataset_quality.sql`
|
||
- `sql/003_*.sql` (unspecified third migration)
|
||
|
||
**Verified this pass:** `git log --all` for every one of those paths returns
|
||
**zero commits, ever, on any branch** (`main`, `feature/0pt-env-example`,
|
||
`ci/add-quality-gate` all checked). A full-repo `find` for each filename
|
||
also returns nothing. These files have never existed in this repository's
|
||
history. `docs/ICEBERG_WRITE_BOUNDARY.md` even states outright that "0pt's
|
||
QA stage... remains entirely 0pt-owned" and cites `quality-check-service.ts`
|
||
as the thing that wasn't touched by that pass — implying it exists to not
|
||
touch. It doesn't exist to begin with.
|
||
|
||
This means: **0pt's dataset-level data-quality/QA enforcement stage does
|
||
not exist in code at all**, despite being described elsewhere as complete.
|
||
This is a real, currently-open gap — not a doc typo. This document replaces
|
||
those false claims with an honest target specification: what this stage
|
||
*should* look like if/when it gets built, grounded in the parts of this
|
||
repo's design that *are* real today.
|
||
|
||
`docs/P0_QUERY_SOURCE_FLOW.md`'s "Known gaps" section has the fuller list of
|
||
exactly which files make the false claim and where; this doc does not
|
||
repeat that inventory, only the target design.
|
||
|
||
## What already exists that this would build on (real, verified)
|
||
|
||
- **The provenance/confidence model** — `docs/06-data-engine.md`'s pipeline
|
||
stage 5 ("QA") already specifies the *rules* this stage should enforce,
|
||
even though no code implements them yet:
|
||
- Per-field `bridge_provenance`: `source_id`, `freshness`, `confidence`
|
||
(0–1), `level` (0–3), `agreed_sources`.
|
||
- Level semantics: 0 = unavailable/licensed, 1 = single source, 2 =
|
||
partial corroboration, 3 = ≥2 independent sources agree.
|
||
- The QA rule: **drop rows below the order's `confidence_target`** —
|
||
deliver fewer rows, never fabricate a value to hit a target.
|
||
- `est_confidence` (order-level target) vs. `avg_confidence` (delivered,
|
||
actual) as the two numbers this stage would need to reconcile.
|
||
- **The Dataset registry** — `src/services/dataset-service.ts` /
|
||
`src/types/dataset.ts` are real, in-memory, tested code. `Dataset` already
|
||
has `dataset_type: "raw" | "structured" | "enriched" | "curated"` and
|
||
`status: "active" | "archived" | "deprecated"` — a quality stage would
|
||
need to decide whether a failed QA pass changes `status`, or whether
|
||
quality state lives entirely in a separate table/type instead (see
|
||
"Open questions" below).
|
||
- **The N0D job pattern** — `src/services/iceberg-write-service.ts` is real,
|
||
tested code showing the actual "sync path vs. N0D job" split this
|
||
codebase already uses elsewhere: an `input.heavy` flag routes to
|
||
`this.n0d.createJob(...)` instead of running inline. A quality-check
|
||
service would plausibly reuse this same shape (small samples checked
|
||
inline; a real run against a full dataset dispatched as an N0D job) —
|
||
but nothing today wires that up for quality checks specifically.
|
||
- **The CL0UD-policy-before-execution pattern** — `src/services/external-writeback-service.ts`
|
||
(`0pt.write_external_record`, real, tested) calls
|
||
`Cl0udClient.evaluatePolicy()` before performing its write. If a quality
|
||
service is meant to policy-gate before running (as
|
||
`docs/ICEBERG_WRITE_BOUNDARY.md` implies with its "CL0UD is always
|
||
re-consulted" framing), this is the real precedent to follow, not
|
||
something already done by the nonexistent `quality-check-service.ts`.
|
||
- **The AUD0 event pattern** — every real service in this repo
|
||
(`source-service.ts`, `dataset-service.ts`, `source-query-service.ts`,
|
||
`external-writeback-service.ts`, `iceberg-write-service.ts`) emits AUD0
|
||
events via `createAud0Client()` against an in-memory mock (no live AUD0
|
||
endpoint reachable from this repo). A quality service would follow the
|
||
same convention.
|
||
|
||
## What this stage should do (target design — none of this is built)
|
||
|
||
1. **Trigger.** A quality check would need to run after the enrichment/
|
||
fusion stage (`docs/06-data-engine.md` stage 4) and before a dataset is
|
||
marked ready for delivery or written to Iceberg via
|
||
`IcebergWriteService`. Whether this is a synchronous call, an
|
||
automatically-triggered N0D job, or an explicit caller action is
|
||
undecided.
|
||
2. **Checks to run**, per `docs/06-data-engine.md`'s stage 5 language
|
||
(originally scoped as "Great Expectations or equivalent, run inside the
|
||
worker"):
|
||
- **Validity** — field format/range checks against the schema for that
|
||
`dataset_type`.
|
||
- **Completeness** — required-field coverage.
|
||
- **Freshness** — age of each `bridge_provenance.freshness` against a
|
||
staleness threshold.
|
||
- **Uniqueness** — post-dedup uniqueness (RapidFuzz dedup already
|
||
happens upstream per `docs/06-data-engine.md` stage 3; this would be a
|
||
verification pass, not the dedup itself).
|
||
- **Corroboration** — cross-check `bridge_provenance.level`/`agreed_sources`
|
||
against the order's declared `confidence_target`; rows below target
|
||
get dropped, never backfilled with a guess.
|
||
3. **Output: a quality report.** Would need a `QualityReport` shape
|
||
(proposed home: `src/types/quality-report.ts`, not yet created) capturing
|
||
at minimum: `dataset_id`, `check_run_id`, per-check pass/fail counts,
|
||
`rows_dropped_below_confidence_target`, `avg_confidence` (delivered,
|
||
actual — per `docs/06-data-engine.md`), and a timestamp. This is the
|
||
thing `docs/06-data-engine.md` calls "the quality report stored on
|
||
`deliveries`" — that storage location (a `deliveries` table) does not
|
||
exist in this repo's `sql/` directory either; `sql/001_init.sql` would
|
||
need to be checked/extended for it, or a new migration written.
|
||
4. **Gate, not silent pass-through.** Per CLAUDE.md's non-negotiable
|
||
constraint #1 ("we never invent values") and #3 ("provenance per field"),
|
||
a dataset that fails its quality gate should not silently ship — the
|
||
order's delivery flow (`docs/01-product.md`'s order lifecycle) would need
|
||
to surface a failed/partial QA result to the customer, not swallow it.
|
||
|
||
## What would need to be created (none of this exists)
|
||
|
||
| Proposed file | Would contain |
|
||
|---|---|
|
||
| `src/types/quality-check.ts` | `QualityCheckRequest`/`QualityCheckResult` shapes: which checks to run, thresholds, dataset/source refs |
|
||
| `src/types/quality-report.ts` | `QualityReport` shape described above |
|
||
| `src/services/quality-check-service.ts` | Orchestrates the checks (validity/completeness/freshness/uniqueness/corroboration), CL0UD policy check, N0D job dispatch for heavy runs (same `input.heavy` shape as `iceberg-write-service.ts`), AUD0 emission |
|
||
| `src/services/quality-report-service.ts` | Persists/retrieves `QualityReport`s; registry pattern matching `DatasetService`/`SourceService` (in-memory today, same known "not shared across route modules" limitation until real persistence lands) |
|
||
| A new SQL migration | Schema for quality-check runs and reports. **Numbering note:** this repo's `sql/` directory today contains only `001_init.sql` and `004_iceberg_write_boundary.sql` — `002` and `003` were referenced in other docs but never created, so there is a numbering gap. Whether a new migration should fill `002`/`003` or continue from `005` is an open question — do not assume `002_p0_dataset_quality.sql` is the correct filename just because other docs guessed it. |
|
||
|
||
`src/data-platform/services/catalog-service.ts` is referenced elsewhere in
|
||
this repo's docs (as the source of an "`L4` = restricted sensitivity"
|
||
precedent) but was also confirmed this pass to not exist anywhere in this
|
||
repo — it is a separate fabrication from the quality-check one, not
|
||
something this quality stage specifically depends on. If a dataset-level
|
||
catalog/listing capability is needed as part of this work, the registered
|
||
`0pt.catalog_list` and `0pt.source_registry_list` capabilities
|
||
(`cl0ud`'s `packages/capability-registry/src/registry.ts`, both real
|
||
registry entries, `surfaces: ['api']`) are the closest existing hooks — but
|
||
whether either is meant to serve this purpose, or a new capability is
|
||
needed, is undecided.
|
||
|
||
## No capability registered for this yet
|
||
|
||
`cl0ud`'s `packages/capability-registry/src/registry.ts` was checked this
|
||
pass for anything quality-related: no `0pt.quality_check`,
|
||
`0pt.run_quality_checks`, or similarly-named capability exists. The
|
||
registry does have a `qualityGateRequired` boolean flag used by *other*
|
||
capabilities (e.g. one INF0 capability sets it `true`), but nothing wires
|
||
`0pt`'s own dataset QA into that flag or into any capability of its own.
|
||
Building this stage would need a new capability entry, decided surfaces
|
||
(`api` at minimum, per the six-surface model; `job` likely for a
|
||
full-dataset run), a `riskLevel`, and a `piiLevel`.
|
||
|
||
## Open questions for Simon
|
||
|
||
- Does quality-check execution belong entirely in 0pt (as
|
||
`docs/ICEBERG_WRITE_BOUNDARY.md` asserts should be true, even though the
|
||
code backing that assertion doesn't exist), or should the actual
|
||
check-running compute move to N0D as a job type, with 0pt only owning the
|
||
*rules*?
|
||
- Where does a `QualityReport` get persisted — a new `sql/00X_*.sql` table
|
||
in 0pt's own Postgres, or through ST0RE (matching the "ST0RE owns
|
||
metadata, 0pt owns production" boundary `docs/ICEBERG_WRITE_BOUNDARY.md`
|
||
draws for Iceberg tables)?
|
||
- Does a dataset that fails QA change `Dataset.status`
|
||
(`src/types/dataset.ts`), or does quality state live entirely outside the
|
||
`Dataset` record?
|
||
- What migration number should the new schema use, given the `002`/`003`
|
||
gap described above?
|
||
- Should this become its own registered `0pt.*` capability, and if so what
|
||
`riskLevel`/`piiLevel` — quality checks read dataset content, which per
|
||
the reasoning already used for `0pt.query_source` (`piiLevel: 'high'`
|
||
because "can return any column, including PII") would plausibly need the
|
||
same `piiLevel`.
|
||
|
||
None of the above is decided by this document. It exists to give a future
|
||
build pass an honest starting point instead of a fabricated one.
|