# IMPLEMENTATION_REPORT.md — P010 Data-OS completion (sax3l/0pt) **Branch:** `feat/p010-data-os-completion` (renamed mid-pass from `feat/p010-source-registry-persistence` once scope grew beyond persistence alone). **Base:** `origin/main` @ `f8bb0d5` (verified in sync via `git fetch --prune` before starting; `origin` = `git.siax.io/sax3l/0pt`, the canonical remote per `CLAUDE.md` — the GitHub copy was not touched). **Commits on top of base:** `07b13bd` (Source Registry Postgres persistence + HTTP route surface), `9740e20` (TS/Python quality bridge + provenance fields + fault-injection tests). **Report date:** 2026-09-01. Written after a from-scratch verification pass (fresh clone, fresh `pnpm install`, fresh Python venv — not a rerun of the working tree that built the branch), so the numbers below are reproducible, not just remembered. This report follows `BEFORE_STATE.md`'s classification scale: `VERIFIED` / `IMPLEMENTED_NOT_VERIFIED` / `DOCUMENTED_ONLY` / `STALE` / `BROKEN` / `UNKNOWN`. Read `BEFORE_STATE.md` first — this report only describes the delta from it. --- ## 1. What actually changed, mapped to BEFORE_STATE.md's 5 gaps ### §1 Source Registry persistence — `VERIFIED` closed - New `src/repositories/*` layer: `SourceRepository` / `SourceRegistryRepository` interfaces, each with an `in-memory` and a `postgres` implementation (`postgres-source-repository.ts`, `postgres-source-registry-repository.ts`). `src/repositories/db.ts` is a thin pooled-client wrapper honoring `resolveStoreMode` (`src/platform/dependency-mode.ts`) — the same guard BEFORE_STATE.md noted as already blocking production boot without a DB; it now has a Postgres implementation to fall through to instead of only throwing. - `sql/002_source_registry.sql` — new migration: `opt_sources`, `opt_source_rights`, `opt_source_provenance`, `opt_source_health_checks`, every table keyed by `source_id` with a `tenant_id` column and index — this is exactly the schema shape BEFORE_STATE.md §1 derived as "what Postgres migration would require" but said did not exist. It now exists and is applied. - `src/services/source-service.ts` and `source-registry-service.ts` rewritten to take a repository (constructor-injected), closing the **per-instance-Map** defect BEFORE_STATE.md flagged as worse than plain in-memory: every route module now shares one repository-backed instance (`src/services/source-registry.ts` is the new shared-singleton wiring), not a fresh `Map` per `new SourceService()`. - New route surface closing BEFORE_STATE.md's "effectively zero HTTP routes" finding: `app/api/0pt/sources/route.ts` (create/list — did not exist before), `[sourceId]/route.ts` (get/describe — did not exist before), `[sourceId]/rights`, `[sourceId]/provenance`, `[sourceId]/health`, `[sourceId]/freshness` (all new). `[sourceId]/query` and `rate-policies/*` (pre-existing routes) were touched only to repoint them at the shared singleton instead of constructing their own service. - `tests/p0/source-registry-postgres.realinfra.test.ts` (340 lines, 7 tests) — a **real-Postgres** suite, not a mock. It self-skips honestly (not fabricating pass) when no reachable Postgres is configured for the run; see §2 (Testing) below for what this pass actually observed. **What is still NOT true, stated plainly:** in-memory mode is still the default when no `DATABASE_URL`/Postgres is configured — this pass added a real persistence *option*, it did not remove the in-memory path (removing it would break every existing unit test and any deployment without Postgres wired yet). `resolveStoreMode` still hard-blocks production boot without a real DB configured, which is correct behavior, not a residual bug. ### §2 TS/Python split-brain — `VERIFIED` bridged (additive, advisory) - **New route surface**, `app/api/0pt/quality/*`: `rules` (list/create), `rules/[ruleId]` (get/update), `runs` (list/create), `runs/[runId]` (get), `issues` (list), and `POST /api/0pt/quality/bridge-run` — the entry point the Python side calls. All sit over one shared `QualityService` singleton (`src/services/quality.ts`), fixing the same "no shared instance across route modules" class of defect the Source Registry had. - `src/quality/quality-bridge.ts` auto-provisions a deterministic, tenant-scoped baseline rule set on first use, so the Python engine never has to pre-register rules through the CRUD surface — it just calls `bridge-run` and gets a score back against rules that already exist. - **Python side**, `services/api/workers/quality_bridge.py`: an HTTP client to the route above. - **Feature-flagged off by default**: `OPT_QUALITY_BRIDGE_ENABLED` unset or falsy → zero network calls, returns immediately (`status: "disabled"`). Verified in code (`_enabled()`, `services/api/workers/ quality_bridge.py:59-60`) and in tests (`test_disabled_by_default_makes_zero_network_calls`). - **Fails open, unconditionally.** Every failure mode — bridge URL unset, connection refused, timeout, non-2xx, malformed JSON — is caught inside the module and reported back as a `BridgeResult` with a `status` field (`disabled` / `not_configured` / `unavailable` / `ok`), never raised into `run_job`. `pipeline.py`'s actual delivery decision (`delivered = [r for r in deduped if r["confidence"] >= threshold]`, still the one line BEFORE_STATE.md quoted) is **unchanged** by any bridge outcome — success or failure. Verified by `test_bridge_disabled_delivers_identically_to_bridge_enabled_and_failing`, which asserts the delivered row set is byte-identical between the two paths. - **Advisory only, and this is a real constraint, not an excuse:** true per-row gating against the TS `ExpectationEngine` is not implementable today because `ExpectationEngine` returns dataset-level pass/fail counts and sample rows, not a per-row boolean array keyed to the Python pipeline's record identity. Building that would mean changing `ExpectationEngine`'s contract — out of scope for this pass, documented rather than faked with a fabricated per-row mapping. - **Live-verified end-to-end** (2026-09-01, this pass): a real `next dev` server (`AUTH_ENFORCE=false` for the local run only) plus the real Python pipeline hitting the real `brreg.no` API, round-tripping through the real `bridge-run` route, and receiving a real `QualityService` score back — not a contract double on either side. This was a manual verification run during development; it is not itself a repeatable automated test (the pytest suite below covers the same code paths against `respx`-mocked bridge responses, which *is* repeatable and is what CI would run). **What is still NOT true, stated plainly:** the bridge is **not enabled in production and this pass does not enable it.** `OPT_QUALITY_BRIDGE_ENABLED` stays unset/`false` as shipped. Turning it on is a separate, deliberate decision — see §4 "Activating the bridge" below. Until that decision is made, the two runtimes' QA logic remains exactly as split as BEFORE_STATE.md described: Python's one-line confidence filter is what actually decides delivery; the TS `QualityService`/`ExpectationEngine` only sees a *copy* of the same data, advisory-only, and only when the flag is on. ### §3 Provenance fields — `VERIFIED` extended (Python side) `services/api/sources/connectors.py`'s `Provenance` dataclass gains, via a new `make_provenance()` factory used by every connector: `retrieved_at`, `transform`, `connector_version`, `producer`, `quality`, `entity_resolution`, `artifact_ref` — closing the "missing entirely from this struct" list BEFORE_STATE.md §3 itemized (retrieval-method-at-field- level remains only partially addressed: `connector_version` +`producer` identify *what ran*, not a full method taxonomy like the TS `SourceProvenance.method` enum). - `resolve.py` stamps `entity_resolution` on **merged** records' provenance only (never on unmerged ones — verified by `test_unmerged_record_keeps_entity_resolution_none`), tracking `agreed_sources`/`merged_from` in a structured field rather than the loose attribute BEFORE_STATE.md found. - `pipeline.py` stamps `artifact_ref` at delivery time and `quality` once (if) the bridge call above succeeds. **What is still NOT true, stated plainly:** `bridge_provenance` as a single named construct **still does not exist** anywhere in this repo, and this pass does not introduce that name. What exists is still two structs — the Python `Provenance` dataclass (now richer) and the TS `SourceProvenance` type (per-source, unchanged this pass) — reconciled only at the one new seam described in §2 (the bridge-run response, which is advisory and optional). This pass closes real gaps in the Python struct; it does not unify the two runtimes' provenance models into one schema. Nothing here persists past the in-process job response either — `pipeline.py` still has no DB write of its own (Postgres persistence added this pass is scoped to the Source Registry, not to order/provenance records). ### §4 Failure-mode tests — `VERIFIED` extended, with an honest residual New pytest + respx suite, `services/api/tests/`: | File | Covers | |---|---| | `test_resolve_determinism.py` | dedup-determinism under shuffled/interleaved orderings, exhaustive permutations of a mutually-similar set, duplicate-input collapsing without double-counting agreement, out-of-order multi-market arrival, entity-resolution provenance stamping — **and** an explicit, passing test of the one case BEFORE_STATE.md flagged as a real, plausible order-sensitivity risk: a non-transitive-similarity triangle in the greedy clusterer genuinely *can* cluster differently by input order. That test (`test_a_non_transitive_triangle_can_cluster_differently_by_order`) asserts the order-sensitivity, it does not paper over it — `resolve.py`'s greedy-merge algorithm was not rewritten to remove this property, since doing so is an algorithm change out of this pass's scope. | | `test_pipeline_fault_injection.py` | schema-evolution (upstream adding/removing/reshaping fields via respx-mocked responses through the real `brreg` connector + `pipeline.py`), restart/connector-crash resilience (a connector exception, including one raised by `respx` itself, never crashes `run_job`; every connector failing delivers zero rows, not a crash), backfill-replay (replaying the same order config is byte-identical — this is the property that actually holds given there is still no persistence layer for orders, not a fabricated "dedup store" test). | | `test_quality_bridge.py` | all bridge states above (disabled / not-configured / unavailable / timeout / non-2xx / malformed / connection-refused / success), plus the "wired into the pipeline" tests proving delivery is unaffected either way. | | `test_connectors_provenance.py` | the `make_provenance()` factory populates every new field honestly and leaves the rest `None` rather than fabricating values; a real parsing path (`brreg`) produces a fully populated provenance record. | | `test_known_gaps.py` | **2 explicit, documented skips** — not silently dropped, not faked: CDC-interruption and true delivery-store backfill idempotency are not implementable against this codebase, because **no CDC pipeline and no persistence layer for orders/deliveries exists anywhere in `services/api`.** Testing either would require building the infrastructure the test claims to fail-inject against, which is a different, larger piece of work than this pass. | Net: of BEFORE_STATE.md §4's four **entirely absent** categories (schema-evolution, CDC-interruption, out-of-order, dedup-determinism), three are now genuinely covered (schema-evolution, out-of-order, dedup-determinism) and one (CDC-interruption) is honestly documented as not implementable rather than covered. Tenant-isolation (already the best-covered category) is untouched by this pass. Quality-gate coverage gains the bridge-interaction tests above but the production Python filter itself (`delivered = [...]`) still has no line-level unit test isolated from an end-to-end pipeline run — this pass tests it via the fault-injection and bridge suites' end-to-end assertions, not a standalone unit test of that one line. ### §5 1P/ST0RE/N0D/AUD0/CL0UD integration — unchanged this pass Not touched. BEFORE_STATE.md's classification stands: ST0RE + N0D real-contract (ST0RE live-verified once, historically); AUD0 real-contract client but unused by the live Python pipeline (still true — `quality_bridge.py` does not emit AUD0 events, and neither does the rest of `services/api`); CL0UD real-contract but not platform-wide-enforced; 1P documented-only. --- ## 2. Testing — from-scratch verification (this pass, not a rerun) To avoid reporting numbers from the working tree that built the branch, this verification was run against a **fresh `git clone`** of the pushed branch (`feat/p010-data-os-completion` @ `9740e20`) in an isolated directory, with a fresh `pnpm install --frozen-lockfile` and a fresh Python venv — no reuse of node_modules, .next cache, or an existing venv from development. | Check | Result | Note | |---|---|---| | `pnpm install --frozen-lockfile` | OK | lockfile v9, no resolution drift | | `pnpm run typecheck` (`tsc -p tsconfig.check.json`, **CI-blocking**) | **0 errors** | Clean on the pristine clone. The working directory this branch was developed in separately contains 5 **untracked, uncommitted** paths (`src/fleet/`, `src/graph/`, `src/metrics/`, `src/projections/`, `src/features/feature-registry.ts`) that were already present and flagged, unaudited, in `BEFORE_STATE.md` before this pass started. They are not part of any commit on this branch or on `main` — confirmed by `git diff --stat origin/main..feat/p010-data-os-completion` touching none of them, and by their total absence from the fresh clone. Running `pnpm typecheck` inside the *original, non-clean* working directory does surface 14 TS errors from those 5 stray local files; that is local working-tree litter, not something CI or a fresh clone will ever see, and this pass does not touch, fix, or commit any of those files. | | `pnpm run test` (vitest, **CI-blocking**) | **256/256 passed, 18 skipped** (34 files passed, 3 files skipped) | Skips are the pre-existing `.realinfra`/`.integration` suites that self-skip honestly when no live ST0RE/OpenSearch/ClickHouse is reachable from this environment (`[realinfra] ST0RE ... not reachable this run (HTTP 401) -- skipping honestly, not fabricating success` — actual stderr from this run). 249 of the 256 predate this branch; 7 are new (`tests/p0/quality-bridge.test.ts`). | | `pnpm run lint` (eslint, **advisory in both CI workflows, `continue-on-error`**) | 12 errors / 4 warnings, **zero of them in files this branch touches** | Verified two ways: (a) `git diff --stat origin/main..feat/p010-data-os-completion` on every flagged file returns no output — byte-identical to `main`; (b) ran `pnpm lint` against a worktree checked out at `origin/main` directly and got the identical 12/4. This is 100% pre-existing legacy noise (mostly `@typescript-eslint/no-explicit-any` in v0-generated adapters/tests), not introduced or worsened by this pass. | | `pnpm run build` (`next build`, **CI-blocking in `.gitea/workflows/ci.yml`**; not run by `.github/workflows/ci.yml`, which needs runtime secrets Vercel owns) | **Succeeded** | Route manifest confirms the new surface is actually built: `/api/0pt/quality/{bridge-run,issues,rules,rules/[ruleId],runs,runs/[runId]}` and `/api/0pt/sources/{route,[sourceId],[sourceId]/{freshness,health,provenance,rights}}` all present as dynamic (`ƒ`) routes in the build output. | | `python -m pytest -v` (`services/api`, fresh venv from `requirements-dev.txt`) | **28 passed, 2 skipped** | The 2 skips are `test_known_gaps.py`'s documented CDC/backfill-idempotency gaps (§4 above) — `pytest.skip(reason=...)`, not silent omission. **Not currently wired into either `.gitea/workflows/ci.yml` or `.github/workflows/ci.yml`** — both CI workflows' own comments say the Python engine "is gated separately if/when a Python workflow is added." This pass did not add that workflow (out of P010's stated scope); the 28/28 above is a manual verification, not an automated CI gate today. Flagging this explicitly as a residual gap, not glossing over it. | **Net CI-blocking-gate status: all green** (typecheck, vitest, `next build` in Gitea CI; typecheck + vitest in the GitHub CI mirror). Lint is advisory everywhere and pre-existing failures are unrelated to this branch. Python tests are green but not yet gated by any CI workflow — see the Gap list below. --- ## 3. Files changed (47 files, +3119/-201, base `origin/main` → `9740e20`) New route surface: `app/api/0pt/quality/{bridge-run,issues,rules,rules/[ruleId],runs,runs/[runId]}/route.ts`, `app/api/0pt/sources/{route.ts,[sourceId]/route.ts,[sourceId]/{freshness,health,provenance,rights}/route.ts}`. New Python: `services/api/workers/quality_bridge.py`, `services/api/pytest.ini`, `services/api/requirements-dev.txt`, `services/api/tests/test_{connectors_provenance,known_gaps,pipeline_fault_injection,quality_bridge,resolve_determinism}.py`. New TS repositories/migration: `src/repositories/*` (7 files), `sql/002_source_registry.sql`. New TS quality bridge: `src/quality/quality-bridge.ts`, `src/services/quality.ts`, `src/services/source-registry.ts`. Modified: `services/api/sources/connectors.py`, `services/api/workers/{pipeline,resolve}.py`, `src/services/{source-registry-service,source-service,source-rate-policy-service}.ts`, `src/validation/0pt-schemas.ts`, `src/http/siax-error-response.ts`, `.gitignore` (added `.venv/`, `.pytest_cache/`), `sql/004_iceberg_write_boundary.sql` (minor). Test files modified: `tests/p0/source-registry.test.ts` (repointed at shared singleton), new `tests/p0/quality-bridge.test.ts`, `tests/p0/source-registry-postgres.realinfra.test.ts`. No changes to: `services/scrapers/` (the still-unmerged WAF-bypass deletion PR — untouched, not this pass's scope), the `app/`-root Next.js UI pages (`bestall/`, `katalog/`, `portal/`, etc. — untouched), any of the 5 untracked stray paths in the working tree. --- ## 4. Activating the pipeline bridge — Simon's decision, not made here **`OPT_QUALITY_BRIDGE_ENABLED` ships unset (default `false`) and this pass does not flip it, anywhere, including in any deploy config, `.env.example`, or CI workflow.** This affects real customer orders once on — enabling it in production is a separate decision, deliberately left to Simon. To turn it on later, the following must all be true first (this pass built the mechanism, not the readiness): 1. `OPT_QUALITY_BRIDGE_URL` (or `OPT_QUALITY_BRIDGE_EDGE_URL`, per `quality_bridge.py`'s `_bridge_url()`) must point at a reachable, authenticated `0pt.app` deployment — the Python engine and the Next.js edge app are typically different deploy targets (VPS vs. Vercel/SIAX Deploy Platform per the `siax-0pt` skill's two-plane model), so this is a real cross-service network dependency, not a same-process flag flip. 2. Even fully enabled and reachable, the bridge remains **advisory-only by design** — it stamps a `quality` field on delivered provenance and surfaces a score; it does **not** and, per §2's `ExpectationEngine` contract gap, currently **cannot** change which rows Python's `delivered = [...]` line keeps or drops. Anyone enabling the flag expecting stricter row-level gating will not get it from this code as shipped — that would require the `ExpectationEngine` contract change noted above, which is future work. 3. Recommended rollout, not yet executed: enable against one non-production tenant/order first, confirm `bridge-run` round-trips correctly (the manual live-verification in §1/§2 above did this once, ad hoc, during development — it was not repeated against a staging environment as a formal rollout step), then decide with Simon whether "advisory-only, fails open" is the intended production posture or whether the per-row gating gap needs to close first. **This pass leaves the flag off. No default, config file, or workflow in this PR turns it on.** --- ## 5. Residual gaps (honest, not exhaustive elsewhere in this doc) - Python engine tests (28/28) are not wired into either CI workflow yet — a `.gitea/workflows/`/`.github/workflows/` Python job is future work, not part of this pass. - `bridge_provenance` as a single unified construct still does not exist; this pass narrows but does not close the TS/Python provenance-schema gap (§3 above). - Per-row quality gating from the TS engine into the Python pipeline is not implementable against the current `ExpectationEngine` contract (§2, §4 above) — advisory-only is the ceiling of what this pass could honestly build, not a corner cut. - `resolve.py`'s greedy merge remains genuinely order-sensitive for non-transitive-similarity triangles — tested and documented, not fixed (fixing it is an algorithm change, out of scope here). - CDC-interruption and true delivery-store backfill-idempotency tests remain unimplementable because the underlying infrastructure (a CDC pipeline, a delivery persistence store) does not exist anywhere in `services/api` — documented via `pytest.skip` with reasons in `test_known_gaps.py`, not silently dropped. - 5 untracked/unaudited local paths (`src/fleet/`, `src/graph/`, `src/metrics/`, `src/projections/`, `src/features/feature-registry.ts`) remain exactly as found — not committed, not deleted, not audited by this pass. They are not part of this PR's diff and a fresh clone never sees them. - `services/scrapers/` (WAF-bypass stack) remains on `main`, unmerged deletion PR still open — unrelated to and untouched by this pass, flagged again here only because `BEFORE_STATE.md` flagged it and it has not changed. --- ## 6. Process notes - No PR was open before this pass; one was opened as part of this pass (see PR URL reported separately / in `PRODUCTION_EVIDENCE.json`). - `BEFORE_STATE.md` and the 5 pre-existing untracked/unaudited paths were left untouched and uncommitted throughout, exactly as found at the start. - `.venv/` and `.pytest_cache/` (both created fresh under `services/api/` while running the Python suite) were added to `.gitignore` so they are never accidentally committed; no leftover Python processes. - The `next dev` server and any background verification process used for the live end-to-end bridge check (§1/§2) were both confirmed stopped before this pass ended — no lingering listeners.