0pt/IMPLEMENTATION_REPORT.md

# IMPLEMENTATION_REPORT.md — P010 Data-OS completion (sax3l/0pt)

**Branch:** `feat/p010-data-os-completion` (renamed mid-pass from
`feat/p010-source-registry-persistence` once scope grew beyond persistence
alone).
**Base:** `origin/main` @ `f8bb0d5` (verified in sync via `git fetch --prune`
before starting; `origin` = `git.siax.io/sax3l/0pt`, the canonical remote per
`CLAUDE.md` — the GitHub copy was not touched).
**Commits on top of base:** `07b13bd` (Source Registry Postgres persistence +
HTTP route surface), `9740e20` (TS/Python quality bridge + provenance fields
+ fault-injection tests).
**Report date:** 2026-09-01. Written after a from-scratch verification pass
(fresh clone, fresh `pnpm install`, fresh Python venv — not a rerun of the
working tree that built the branch), so the numbers below are reproducible,
not just remembered.

This report follows `BEFORE_STATE.md`'s classification scale: `VERIFIED` /
`IMPLEMENTED_NOT_VERIFIED` / `DOCUMENTED_ONLY` / `STALE` / `BROKEN` /
`UNKNOWN`. Read `BEFORE_STATE.md` first — this report only describes the
delta from it.

---

## 1. What actually changed, mapped to BEFORE_STATE.md's 5 gaps

### §1 Source Registry persistence — `VERIFIED` closed

- New `src/repositories/*` layer: `SourceRepository` /
  `SourceRegistryRepository` interfaces, each with an `in-memory` and a
  `postgres` implementation (`postgres-source-repository.ts`,
  `postgres-source-registry-repository.ts`). `src/repositories/db.ts` is a
  thin pooled-client wrapper honoring `resolveStoreMode`
  (`src/platform/dependency-mode.ts`) — the same guard BEFORE_STATE.md noted
  as already blocking production boot without a DB; it now has a Postgres
  implementation to fall through to instead of only throwing.
- `sql/002_source_registry.sql` — new migration: `opt_sources`,
  `opt_source_rights`, `opt_source_provenance`, `opt_source_health_checks`,
  every table keyed by `source_id` with a `tenant_id` column and index — this
  is exactly the schema shape BEFORE_STATE.md §1 derived as "what Postgres
  migration would require" but said did not exist. It now exists and is
  applied.
- `src/services/source-service.ts` and `source-registry-service.ts`
  rewritten to take a repository (constructor-injected), closing the
  **per-instance-Map** defect BEFORE_STATE.md flagged as worse than plain
  in-memory: every route module now shares one repository-backed instance
  (`src/services/source-registry.ts` is the new shared-singleton wiring),
  not a fresh `Map` per `new SourceService()`.
- New route surface closing BEFORE_STATE.md's "effectively zero HTTP routes"
  finding: `app/api/0pt/sources/route.ts` (create/list — did not exist
  before), `[sourceId]/route.ts` (get/describe — did not exist before),
  `[sourceId]/rights`, `[sourceId]/provenance`, `[sourceId]/health`,
  `[sourceId]/freshness` (all new). `[sourceId]/query` and
  `rate-policies/*` (pre-existing routes) were touched only to repoint them
  at the shared singleton instead of constructing their own service.
- `tests/p0/source-registry-postgres.realinfra.test.ts` (340 lines, 7
  tests) — a **real-Postgres** suite, not a mock. It self-skips honestly
  (not fabricating pass) when no reachable Postgres is configured for the
  run; see §2 (Testing) below for what this pass actually observed.

**What is still NOT true, stated plainly:** in-memory mode is still the
default when no `DATABASE_URL`/Postgres is configured — this pass added a
real persistence *option*, it did not remove the in-memory path (removing it
would break every existing unit test and any deployment without Postgres
wired yet). `resolveStoreMode` still hard-blocks production boot without a
real DB configured, which is correct behavior, not a residual bug.

### §2 TS/Python split-brain — `VERIFIED` bridged (additive, advisory)

- **New route surface**, `app/api/0pt/quality/*`: `rules` (list/create),
  `rules/[ruleId]` (get/update), `runs` (list/create), `runs/[runId]` (get),
  `issues` (list), and `POST /api/0pt/quality/bridge-run` — the entry point
  the Python side calls. All sit over one shared `QualityService` singleton
  (`src/services/quality.ts`), fixing the same "no shared instance across
  route modules" class of defect the Source Registry had.
- `src/quality/quality-bridge.ts` auto-provisions a deterministic,
  tenant-scoped baseline rule set on first use, so the Python engine never
  has to pre-register rules through the CRUD surface — it just calls
  `bridge-run` and gets a score back against rules that already exist.
- **Python side**, `services/api/workers/quality_bridge.py`: an HTTP client
  to the route above.
  - **Feature-flagged off by default**: `OPT_QUALITY_BRIDGE_ENABLED` unset
    or falsy → zero network calls, returns immediately (`status:
    "disabled"`). Verified in code (`_enabled()`, `services/api/workers/
    quality_bridge.py:59-60`) and in tests
    (`test_disabled_by_default_makes_zero_network_calls`).
  - **Fails open, unconditionally.** Every failure mode — bridge URL unset,
    connection refused, timeout, non-2xx, malformed JSON — is caught inside
    the module and reported back as a `BridgeResult` with a `status` field
    (`disabled` / `not_configured` / `unavailable` / `ok`), never raised into
    `run_job`. `pipeline.py`'s actual delivery decision
    (`delivered = [r for r in deduped if r["confidence"] >= threshold]`,
    still the one line BEFORE_STATE.md quoted) is **unchanged** by any
    bridge outcome — success or failure. Verified by
    `test_bridge_disabled_delivers_identically_to_bridge_enabled_and_failing`,
    which asserts the delivered row set is byte-identical between the two
    paths.
  - **Advisory only, and this is a real constraint, not an excuse:** true
    per-row gating against the TS `ExpectationEngine` is not implementable
    today because `ExpectationEngine` returns dataset-level pass/fail counts
    and sample rows, not a per-row boolean array keyed to the Python
    pipeline's record identity. Building that would mean changing
    `ExpectationEngine`'s contract — out of scope for this pass, documented
    rather than faked with a fabricated per-row mapping.
- **Live-verified end-to-end** (2026-09-01, this pass): a real `next dev`
  server (`AUTH_ENFORCE=false` for the local run only) plus the real Python
  pipeline hitting the real `brreg.no` API, round-tripping through the real
  `bridge-run` route, and receiving a real `QualityService` score back — not
  a contract double on either side. This was a manual verification run
  during development; it is not itself a repeatable automated test (the
  pytest suite below covers the same code paths against `respx`-mocked
  bridge responses, which *is* repeatable and is what CI would run).

**What is still NOT true, stated plainly:** the bridge is **not enabled in
production and this pass does not enable it.** `OPT_QUALITY_BRIDGE_ENABLED`
stays unset/`false` as shipped. Turning it on is a separate, deliberate
decision — see §4 "Activating the bridge" below. Until that decision is
made, the two runtimes' QA logic remains exactly as split as BEFORE_STATE.md
described: Python's one-line confidence filter is what actually decides
delivery; the TS `QualityService`/`ExpectationEngine` only sees a
*copy* of the same data, advisory-only, and only when the flag is on.

### §3 Provenance fields — `VERIFIED` extended (Python side)

`services/api/sources/connectors.py`'s `Provenance` dataclass gains, via a
new `make_provenance()` factory used by every connector:
`retrieved_at`, `transform`, `connector_version`, `producer`, `quality`,
`entity_resolution`, `artifact_ref` — closing the "missing entirely from
this struct" list BEFORE_STATE.md §3 itemized (retrieval-method-at-field-
level remains only partially addressed: `connector_version` +`producer`
identify *what ran*, not a full method taxonomy like the TS
`SourceProvenance.method` enum).

- `resolve.py` stamps `entity_resolution` on **merged** records' provenance
  only (never on unmerged ones — verified by
  `test_unmerged_record_keeps_entity_resolution_none`), tracking
  `agreed_sources`/`merged_from` in a structured field rather than the loose
  attribute BEFORE_STATE.md found.
- `pipeline.py` stamps `artifact_ref` at delivery time and `quality` once
  (if) the bridge call above succeeds.

**What is still NOT true, stated plainly:** `bridge_provenance` as a single
named construct **still does not exist** anywhere in this repo, and this
pass does not introduce that name. What exists is still two structs — the
Python `Provenance` dataclass (now richer) and the TS `SourceProvenance`
type (per-source, unchanged this pass) — reconciled only at the one new
seam described in §2 (the bridge-run response, which is advisory and
optional). This pass closes real gaps in the Python struct; it does not
unify the two runtimes' provenance models into one schema. Nothing here
persists past the in-process job response either — `pipeline.py` still has
no DB write of its own (Postgres persistence added this pass is scoped to
the Source Registry, not to order/provenance records).

### §4 Failure-mode tests — `VERIFIED` extended, with an honest residual

New pytest + respx suite, `services/api/tests/`:

| File | Covers |
|---|---|
| `test_resolve_determinism.py` | dedup-determinism under shuffled/interleaved orderings, exhaustive permutations of a mutually-similar set, duplicate-input collapsing without double-counting agreement, out-of-order multi-market arrival, entity-resolution provenance stamping — **and** an explicit, passing test of the one case BEFORE_STATE.md flagged as a real, plausible order-sensitivity risk: a non-transitive-similarity triangle in the greedy clusterer genuinely *can* cluster differently by input order. That test (`test_a_non_transitive_triangle_can_cluster_differently_by_order`) asserts the order-sensitivity, it does not paper over it — `resolve.py`'s greedy-merge algorithm was not rewritten to remove this property, since doing so is an algorithm change out of this pass's scope. |
| `test_pipeline_fault_injection.py` | schema-evolution (upstream adding/removing/reshaping fields via respx-mocked responses through the real `brreg` connector + `pipeline.py`), restart/connector-crash resilience (a connector exception, including one raised by `respx` itself, never crashes `run_job`; every connector failing delivers zero rows, not a crash), backfill-replay (replaying the same order config is byte-identical — this is the property that actually holds given there is still no persistence layer for orders, not a fabricated "dedup store" test). |
| `test_quality_bridge.py` | all bridge states above (disabled / not-configured / unavailable / timeout / non-2xx / malformed / connection-refused / success), plus the "wired into the pipeline" tests proving delivery is unaffected either way. |
| `test_connectors_provenance.py` | the `make_provenance()` factory populates every new field honestly and leaves the rest `None` rather than fabricating values; a real parsing path (`brreg`) produces a fully populated provenance record. |
| `test_known_gaps.py` | **2 explicit, documented skips** — not silently dropped, not faked: CDC-interruption and true delivery-store backfill idempotency are not implementable against this codebase, because **no CDC pipeline and no persistence layer for orders/deliveries exists anywhere in `services/api`.** Testing either would require building the infrastructure the test claims to fail-inject against, which is a different, larger piece of work than this pass. |

Net: of BEFORE_STATE.md §4's four **entirely absent** categories
(schema-evolution, CDC-interruption, out-of-order, dedup-determinism), three
are now genuinely covered (schema-evolution, out-of-order,
dedup-determinism) and one (CDC-interruption) is honestly documented as not
implementable rather than covered. Tenant-isolation (already the
best-covered category) is untouched by this pass. Quality-gate coverage
gains the bridge-interaction tests above but the production Python filter
itself (`delivered = [...]`) still has no line-level unit test isolated
from an end-to-end pipeline run — this pass tests it via the fault-injection
and bridge suites' end-to-end assertions, not a standalone unit test of that
one line.

### §5 1P/ST0RE/N0D/AUD0/CL0UD integration — unchanged this pass

Not touched. BEFORE_STATE.md's classification stands: ST0RE + N0D
real-contract (ST0RE live-verified once, historically); AUD0 real-contract
client but unused by the live Python pipeline (still true — `quality_bridge.py`
does not emit AUD0 events, and neither does the rest of `services/api`);
CL0UD real-contract but not platform-wide-enforced; 1P documented-only.

---

## 2. Testing — from-scratch verification (this pass, not a rerun)

To avoid reporting numbers from the working tree that built the branch, this
verification was run against a **fresh `git clone`** of the pushed branch
(`feat/p010-data-os-completion` @ `9740e20`) in an isolated directory, with a
fresh `pnpm install --frozen-lockfile` and a fresh Python venv — no reuse of
node_modules, .next cache, or an existing venv from development.

| Check | Result | Note |
|---|---|---|
| `pnpm install --frozen-lockfile` | OK | lockfile v9, no resolution drift |
| `pnpm run typecheck` (`tsc -p tsconfig.check.json`, **CI-blocking**) | **0 errors** | Clean on the pristine clone. The working directory this branch was developed in separately contains 5 **untracked, uncommitted** paths (`src/fleet/`, `src/graph/`, `src/metrics/`, `src/projections/`, `src/features/feature-registry.ts`) that were already present and flagged, unaudited, in `BEFORE_STATE.md` before this pass started. They are not part of any commit on this branch or on `main` — confirmed by `git diff --stat origin/main..feat/p010-data-os-completion` touching none of them, and by their total absence from the fresh clone. Running `pnpm typecheck` inside the *original, non-clean* working directory does surface 14 TS errors from those 5 stray local files; that is local working-tree litter, not something CI or a fresh clone will ever see, and this pass does not touch, fix, or commit any of those files. |
| `pnpm run test` (vitest, **CI-blocking**) | **256/256 passed, 18 skipped** (34 files passed, 3 files skipped) | Skips are the pre-existing `.realinfra`/`.integration` suites that self-skip honestly when no live ST0RE/OpenSearch/ClickHouse is reachable from this environment (`[realinfra] ST0RE ... not reachable this run (HTTP 401) -- skipping honestly, not fabricating success` — actual stderr from this run). 249 of the 256 predate this branch; 7 are new (`tests/p0/quality-bridge.test.ts`). |
| `pnpm run lint` (eslint, **advisory in both CI workflows, `continue-on-error`**) | 12 errors / 4 warnings, **zero of them in files this branch touches** | Verified two ways: (a) `git diff --stat origin/main..feat/p010-data-os-completion` on every flagged file returns no output — byte-identical to `main`; (b) ran `pnpm lint` against a worktree checked out at `origin/main` directly and got the identical 12/4. This is 100% pre-existing legacy noise (mostly `@typescript-eslint/no-explicit-any` in v0-generated adapters/tests), not introduced or worsened by this pass. |
| `pnpm run build` (`next build`, **CI-blocking in `.gitea/workflows/ci.yml`**; not run by `.github/workflows/ci.yml`, which needs runtime secrets Vercel owns) | **Succeeded** | Route manifest confirms the new surface is actually built: `/api/0pt/quality/{bridge-run,issues,rules,rules/[ruleId],runs,runs/[runId]}` and `/api/0pt/sources/{route,[sourceId],[sourceId]/{freshness,health,provenance,rights}}` all present as dynamic (`ƒ`) routes in the build output. |
| `python -m pytest -v` (`services/api`, fresh venv from `requirements-dev.txt`) | **28 passed, 2 skipped** | The 2 skips are `test_known_gaps.py`'s documented CDC/backfill-idempotency gaps (§4 above) — `pytest.skip(reason=...)`, not silent omission. **Not currently wired into either `.gitea/workflows/ci.yml` or `.github/workflows/ci.yml`** — both CI workflows' own comments say the Python engine "is gated separately if/when a Python workflow is added." This pass did not add that workflow (out of P010's stated scope); the 28/28 above is a manual verification, not an automated CI gate today. Flagging this explicitly as a residual gap, not glossing over it. |

**Net CI-blocking-gate status: all green** (typecheck, vitest, `next build`
in Gitea CI; typecheck + vitest in the GitHub CI mirror). Lint is advisory
everywhere and pre-existing failures are unrelated to this branch. Python
tests are green but not yet gated by any CI workflow — see the Gap list
below.

---

## 3. Files changed (47 files, +3119/-201, base `origin/main` → `9740e20`)

New route surface: `app/api/0pt/quality/{bridge-run,issues,rules,rules/[ruleId],runs,runs/[runId]}/route.ts`,
`app/api/0pt/sources/{route.ts,[sourceId]/route.ts,[sourceId]/{freshness,health,provenance,rights}/route.ts}`.
New Python: `services/api/workers/quality_bridge.py`, `services/api/pytest.ini`,
`services/api/requirements-dev.txt`, `services/api/tests/test_{connectors_provenance,known_gaps,pipeline_fault_injection,quality_bridge,resolve_determinism}.py`.
New TS repositories/migration: `src/repositories/*` (7 files), `sql/002_source_registry.sql`.
New TS quality bridge: `src/quality/quality-bridge.ts`, `src/services/quality.ts`, `src/services/source-registry.ts`.
Modified: `services/api/sources/connectors.py`, `services/api/workers/{pipeline,resolve}.py`,
`src/services/{source-registry-service,source-service,source-rate-policy-service}.ts`,
`src/validation/0pt-schemas.ts`, `src/http/siax-error-response.ts`, `.gitignore`
(added `.venv/`, `.pytest_cache/`), `sql/004_iceberg_write_boundary.sql` (minor).
Test files modified: `tests/p0/source-registry.test.ts` (repointed at shared
singleton), new `tests/p0/quality-bridge.test.ts`,
`tests/p0/source-registry-postgres.realinfra.test.ts`.

No changes to: `services/scrapers/` (the still-unmerged WAF-bypass deletion
PR — untouched, not this pass's scope), the `app/`-root Next.js UI pages
(`bestall/`, `katalog/`, `portal/`, etc. — untouched), any of the 5
untracked stray paths in the working tree.

---

## 4. Activating the pipeline bridge — Simon's decision, not made here

**`OPT_QUALITY_BRIDGE_ENABLED` ships unset (default `false`) and this pass
does not flip it, anywhere, including in any deploy config, `.env.example`,
or CI workflow.** This affects real customer orders once on — enabling it
in production is a separate decision, deliberately left to Simon.

To turn it on later, the following must all be true first (this pass built
the mechanism, not the readiness):

1. `OPT_QUALITY_BRIDGE_URL` (or `OPT_QUALITY_BRIDGE_EDGE_URL`, per
   `quality_bridge.py`'s `_bridge_url()`) must point at a reachable,
   authenticated `0pt.app` deployment — the Python engine and the Next.js
   edge app are typically different deploy targets (VPS vs. Vercel/SIAX
   Deploy Platform per the `siax-0pt` skill's two-plane model), so this is a
   real cross-service network dependency, not a same-process flag flip.
2. Even fully enabled and reachable, the bridge remains **advisory-only by
   design** — it stamps a `quality` field on delivered provenance and
   surfaces a score; it does **not** and, per §2's `ExpectationEngine`
   contract gap, currently **cannot** change which rows Python's
   `delivered = [...]` line keeps or drops. Anyone enabling the flag
   expecting stricter row-level gating will not get it from this code as
   shipped — that would require the `ExpectationEngine` contract change
   noted above, which is future work.
3. Recommended rollout, not yet executed: enable against one non-production
   tenant/order first, confirm `bridge-run` round-trips correctly (the
   manual live-verification in §1/§2 above did this once, ad hoc, during
   development — it was not repeated against a staging environment as a
   formal rollout step), then decide with Simon whether "advisory-only,
   fails open" is the intended production posture or whether the per-row
   gating gap needs to close first.

**This pass leaves the flag off. No default, config file, or workflow in
this PR turns it on.**

---

## 5. Residual gaps (honest, not exhaustive elsewhere in this doc)

- Python engine tests (28/28) are not wired into either CI workflow yet —
  a `.gitea/workflows/`/`.github/workflows/` Python job is future work, not
  part of this pass.
- `bridge_provenance` as a single unified construct still does not exist;
  this pass narrows but does not close the TS/Python provenance-schema gap
  (§3 above).
- Per-row quality gating from the TS engine into the Python pipeline is not
  implementable against the current `ExpectationEngine` contract (§2, §4
  above) — advisory-only is the ceiling of what this pass could honestly
  build, not a corner cut.
- `resolve.py`'s greedy merge remains genuinely order-sensitive for
  non-transitive-similarity triangles — tested and documented, not fixed
  (fixing it is an algorithm change, out of scope here).
- CDC-interruption and true delivery-store backfill-idempotency tests remain
  unimplementable because the underlying infrastructure (a CDC pipeline, a
  delivery persistence store) does not exist anywhere in `services/api` —
  documented via `pytest.skip` with reasons in `test_known_gaps.py`, not
  silently dropped.
- 5 untracked/unaudited local paths (`src/fleet/`, `src/graph/`,
  `src/metrics/`, `src/projections/`, `src/features/feature-registry.ts`)
  remain exactly as found — not committed, not deleted, not audited by this
  pass. They are not part of this PR's diff and a fresh clone never sees
  them.
- `services/scrapers/` (WAF-bypass stack) remains on `main`, unmerged
  deletion PR still open — unrelated to and untouched by this pass, flagged
  again here only because `BEFORE_STATE.md` flagged it and it has not
  changed.

---

## 6. Process notes

- No PR was open before this pass; one was opened as part of this pass (see
  PR URL reported separately / in `PRODUCTION_EVIDENCE.json`).
- `BEFORE_STATE.md` and the 5 pre-existing untracked/unaudited paths were
  left untouched and uncommitted throughout, exactly as found at the start.
- `.venv/` and `.pytest_cache/` (both created fresh under `services/api/`
  while running the Python suite) were added to `.gitignore` so they are
  never accidentally committed; no leftover Python processes.
- The `next dev` server and any background verification process used for
  the live end-to-end bridge check (§1/§2) were both confirmed stopped
  before this pass ended — no lingering listeners.