0pt/STATUS.md

# 0pt — capability status (Master-DoD §56 status honesty)

> Classification scale (Master-DoD §2, exactly one per capability):
> `MISSING → DOCUMENTED_ONLY → SKELETON → MOCK_ONLY → IMPLEMENTED → INTEGRATION_TESTED → LIVE_VERIFIED → PRODUCTION_READY`.
> Only `PRODUCTION_READY` may be called DONE. Docs ≠ implementation.
> Last full audit + re-classification: **2026-08-31** (SIAX Platforms P0 wave 2).

## How to read this

- **INTEGRATION_TESTED** here means: verified against a REAL dependency
  (live OpenSearch/ClickHouse container, or a recorded wire-contract double
  pinned to the sibling repo's real handler code on its `main`) in CI.
- **LIVE_VERIFIED** requires a verification against the actual SIAX fleet
  deployment. No new capability in this wave claims it.
- The unit/contract/integration suites are in `tests/` and run in
  `.gitea/workflows/ci.yml` (jobs `quality` + `integration`).

## Cross-product clients (Master-DoD §10 "Cross-product clients")

| Capability | Status | Evidence | Honest gaps |
|---|---|---|---|
| CL0UD client (policy evaluate, `POST /api/policy/evaluate`) | **INTEGRATION_TESTED** (contract) | `src/clients/cl0ud-client.ts`; `tests/contracts/cl0ud-client.contract.test.ts` pins the real camelCase `{data}` envelope; fail-closed (missing decision ⇒ deny) | Not yet exercised against a live CL0UD deployment (no reachable CL0UD URL this session ⇒ not LIVE_VERIFIED) |
| CL0UD capability discovery (`GET /api/capabilities`) | **INTEGRATION_TESTED** (contract) | same suite | 0pt's own capabilities are not yet registered in CL0UD's registry (§3 gap) |
| ST0RE client (artifact register, camelCase `/v1/st0re/artifacts/register`) | **INTEGRATION_TESTED** (contract) | `src/clients/st0re-client.ts`; `tests/contracts/st0re-client.contract.test.ts` | Raw-bytes upload still goes via LocalBlobStore interim (pre-existing, unchanged); presign upload path not consumed yet |
| N0D client (create/get job, get result) | **INTEGRATION_TESTED** (contract) | `src/clients/n0d-client.ts`; `tests/contracts/n0d-client.contract.test.ts` pins `required_role`/`resource_requirements`/body `idempotency_key` | CloudEvents completion callback receiving endpoint not built in 0pt (poll path works) |
| AUD0 client (events + OpenLineage ingest) | **INTEGRATION_TESTED** (contract) | `src/clients/aud0-client.ts`; `tests/contracts/aud0-client.contract.test.ts`; legacy lineage shape converts to real OpenLineage | No durable local outbox for AUD0 outages (§70 "durable local/outbox path" — emission failure is surfaced, not queued) |
| **No production mock fallback (§3/§10)** | **INTEGRATION_TESTED** | `src/platform/dependency-mode.ts`; `tests/contracts/dependency-mode.contract.test.ts` drives NODE_ENV=production for all four factories ⇒ hard `DEPENDENCY_UNAVAILABLE` | — |

**Correction of prior state (2026-08-31):** the pre-existing clients' HTTP
paths were verified against the sibling repos' actual `main` branches and
THREE of four were wrong: cl0ud (`/v1/cl0ud/policies/evaluate` — never
existed), st0re (snake_case body — would 400), aud0 lineage
(`/v1/aud0/lineage/events` — never existed). All fixed and pinned by
contract tests. The earlier MOCK_ONLY classification in the README was
accurate.

## Data-OS capabilities (Master-DoD §10)

| Capability | Status | Evidence | Honest gaps |
|---|---|---|---|
| Source Registry — register/query/describe/health, rights/license/purpose, freshness, provenance | **IMPLEMENTED** | `src/services/source-registry-service.ts`, `src/types/source-registry.ts`; `tests/p0/source-registry.test.ts` (fail-closed purpose gate, cross-tenant NOT_FOUND) | In-memory persistence (per-process); no Postgres store yet — production boot must therefore configure one before this counts further. No API route surface yet |
| Source rate-policy | **IMPLEMENTED** (pre-existing) | `src/services/source-rate-policy-service.ts` | unchanged this wave |
| Search — OpenSearch adapter, index defs, bulk, tenant + purpose/rights filter, rebuild, generation switch | **INTEGRATION_TESTED** | `src/search/*`; unit `tests/p0/search-service.test.ts`; **real OpenSearch 2.11** `tests/integration/search-opensearch.integration.test.ts` incl. projection-disaster rebuild (§68-style) — runs in CI service container | Hybrid (vector) query not built; no metrics emission; index definitions in-memory |
| Quality — versioned rules, runs, results, issues, score, trend, GE interchange | **IMPLEMENTED** (local engine) / **SKELETON** (GE delegation) | `src/quality/*`; `tests/p0/quality-service.test.ts`, `tests/p0/expectation-engine` cases | Local engine covers 7 GE core expectation types; everything else delegates via N0D job `0pt.quality.ge_run` whose EXECUTOR (0pt-motor GE runner) does not exist yet — delegated runs stay "delegated" until it does. Never fabricates results |
| Dataset — create (pre-existing) + version/query/export with `st0re:artifact:` refs | **IMPLEMENTED** | `src/services/dataset-version-service.ts`; `tests/p0/dataset-version-service.test.ts` (idempotency §53, canonical refs §5) | Export assumes bytes already written by a job; no route surface; in-memory persistence |
| Lineage — OpenLineage emission to AUD0's real ingest | **INTEGRATION_TESTED** (contract) | `src/lineage/*`; aud0 contract suite; emitted from dataset version/export, quality runs, search rebuilds | Emission is best-effort (never-throw); no outbox. AUD0-side ingestion verified against contract double, not live AUD0 |
| Metadata — OpenMetadata projection (never canonical) | **IMPLEMENTED** | `src/metadata/openmetadata-adapter.ts`; `tests/contracts/metadata-features.contract.test.ts`; `projectAll` rebuild path | No live OpenMetadata exercised (no local container this pass); lineage/glossary/ownership projection not built — only dataset entities |
| Analytics — ClickHouse, tenant-isolated, query limits, retention | **INTEGRATION_TESTED** | `src/analytics/clickhouse-analytics.ts`; **real ClickHouse 24.8** `tests/integration/analytics-clickhouse.integration.test.ts` (tenant isolation, closed DSL, TTL schema) — runs in CI service container | Closed metric-DSL only (no ad-hoc queries by design); no quota integration with CL0UD |
| Features — registry + Feast adapter | **IMPLEMENTED** (online path, registry) / **SKELETON** (materialize/get-historical) | `src/features/feast-adapter.ts`; contract suite pins Feast's `get-online-features` wire shape | No live Feast server exercised; offline/materialization are N0D delegation contracts whose executor (0pt-motor) does not exist yet; point-in-time correctness therefore NOT yet proven |
| Entity Resolution — candidate API, blocking, explanation, hard-rule hook, review handoff | **IMPLEMENTED** | `src/er/entity-resolution-service.ts`; `tests/p0/entity-resolution.test.ts` (hard rules override both directions, review band, blocking refusals) | Generic reference scorer (Jaccard+Levenshtein), NOT Splink probabilistic linkage — that is the `0pt.er.batch_resolution` N0D contract, executor pending in 0pt-motor. No persistence of review queues |
| Graph (§10 Graph) | **MISSING** | — | Not started this wave (explicitly out of the delivered P0 subset; next in line per §10) |

## Canonical platform plumbing

| Capability | Status | Evidence |
|---|---|---|
| SiaxRequestContext + propagation headers (mirror of cl0ud `@siax/contract`) | **INTEGRATION_TESTED** (contract-pinned) | `src/platform/siax-context.ts`; `tests/contracts/siax-context.contract.test.ts` |
| Canonical 12-code error model + typed `SiaxError` (§6) | **INTEGRATION_TESTED** (contract-pinned) | `src/platform/errors.ts`; same suite |
| Canonical resource refs (§5) | **IMPLEMENTED** | `src/platform/resource-ref.ts`; `tests/p0/dataset-version-service.test.ts` |

## Pre-existing capabilities (unchanged this wave — prior classifications stand)

| Capability | Status | Note |
|---|---|---|
| Iceberg write boundary via ST0RE lakehouse | LIVE_VERIFIED (2026-07-13) | `docs/ICEBERG_WRITE_BOUNDARY.md`; untouched |
| Document capture/extraction | IMPLEMENTED | LocalBlobStore interim persists bytes; ST0RE raw upload still pending |
| Edge order/payment app (Next.js) | see `docs/architecture/ARCHITECTURE.md` | outside this wave's scope |
| FastAPI engine (`services/api`) | IMPLEMENTED | outside this wave's scope |
| `services/scrapers/*` (WAF-bypass stack) | **quarantined, pending deletion PR** | `feature/0pt-remove-noncompliant-scrapers` — NOT touched by this wave; nothing new wires it |

## Global gaps blocking PRODUCTION_READY (§3, tracked, not hidden)

1. Persistent stores REMAINING: quality runs/rules and feature views are
   still per-process in-memory (`resolveStoreMode` guarantees production
   boot FAILS rather than serving Maps). CLOSED since the last revision:
   dataset registry (`opt_datasets`) + immutable versions
   (`opt_dataset_versions`, monotonic under transaction lock, idempotency
   with payload-drift detection, cross-tenant ⇒ NOT_FOUND) and versioned
   search index definitions (`opt_search_index_definitions`, sequential
   schema versions, replay-idempotent, same-version-different-schema ⇒
   CONFLICT; OpenSearch remains a rebuildable projection, never canonical
   truth) are Postgres-canonical with real-Postgres tests.
2. Capability registration in CL0UD + API0 discovery for the new
   capabilities (§3 "Capability är registrerad i CL0UD Capability
   Registry") — PARTIALLY CLOSED: the collaboration/meeting domain was
   canonicalized in CL0UD's registry (cl0ud #73/#74); 0pt's own
   capability set still needs the same registration.
3. CLOSED: AUD0 outbox for evidence durability under AUD0 outage (§70) —
   durable Postgres outbox + dispatcher (at-least-once, bounded batch,
   backoff, dead-letter, SKIP LOCKED leasing, tenant binding, idempotency)
   + real-Postgres tests; dispatcher uses explicit `durable:false` so it
   never re-enqueues its own delivery evidence.
4. CLOSED for datasets/versions/search/ER: `app/api/0pt/datasets*`,
   `app/api/0pt/search/definitions` + tenant-safed query route, and
   `app/api/0pt/er/{candidates,batch}` (+ internal ER route) are live;
   clients cannot raise `rights_tag` via request body (fail-closed to
   `internal`). Quality routes exist (`bridge-run/issues/rules/runs`).
5. 0pt-motor executors for the delegated jobs
   (`0pt.quality.ge_run`, `0pt.features.*`, `0pt.er.batch_resolution`).
6. Observability: no metrics/traces emitted by the new services (§36).

---

## P060-P068 ingestion plane (added 2026-09-04)

Classification scale unchanged. Every row below states what was actually
verified, and by which command.

| Requirement | BEFORE this pass | Status now | Evidence |
|---|---|---|---|
| **P060** Source Registry: durable canonical registry; restart preserves sources; an unclassified source cannot be activated | **PARTIAL / BROKEN.** The registry existed with Postgres persistence (`sql/002_source_registry.sql`, `src/repositories/postgres-source-registry-repository.ts`) and carried source_id/tenant/project/type/status/license/purpose/rights. It had NO authorization, NO schedule, NO checkpoint, NO schema, and no `owner` distinct from `created_by`. The activation DoD was inverted: `postgres-source-repository.ts:70` wrote `${input.status ?? "active"}`, so a source with no rights, no provenance and no authorization was ACTIVE the instant it was registered, and no activation operation existed at all | **INTEGRATION_TESTED** | `sql/007_ingestion_plane.sql` (authorization/schedule/schemas/checkpoints + `ALTER … SET DEFAULT 'pending'` + `owner_id`); `SourceRegistryService.classifySource`/`activateSource`; `tests/p0/source-activation-gate.test.ts` (11); `tests/api/0pt-p060-routes.test.ts` (9); `tests/p0/ingestion-plane-postgres.realinfra.test.ts` against a live Postgres 16 |
| **P061** Connector contract: discover/validate/connect/checkpoint/read/resume/backfill/health/disconnect; two heterogeneous connectors pass one conformance suite | **MISSING.** `services/api/sources/connectors.py` had per-source `fetch_*()` functions only — no lifecycle, no cursor, no resume, no health, no disconnect, no conformance suite | **INTEGRATION_TESTED** | `src/connectors/connector.ts` (all nine operations); `src/connectors/conformance.ts` (15 checks); `HttpJsonConnector` (real node:http server, opaque cursor) and `JsonlFileConnector` (real file, line-offset cursor, CDC deletes) both pass the identical suite — `tests/p0/connector-conformance.test.ts`, which also proves the suite FAILS a fail-open connector |
| **P062** CDC/batch/stream normalized to one envelope; restart resumes from a durable checkpoint without silent gaps | **MISSING**, and documented as such: `services/api/tests/test_known_gaps.py:10-19` skipped the CDC case because no CDC pipeline existed anywhere, and `workers/pipeline.py` had no persistence at all | **INTEGRATION_TESTED** | `src/types/ingestion.ts` (one envelope for all three modes); `src/ingestion/ingestion-runner.ts`; `tests/p0/ingestion-runner.test.ts` kills a run inside the sink and proves the resumed run covers all 10 records exactly once; same proof against real Postgres across fresh processes |
| **P063** Schema versioning + compatibility policy; a breaking schema is detected before consumer corruption | **MISSING.** `schema_version` existed as a free-form string on datasets/Iceberg writes; there was no comparison, no policy, no versioned source schema | **INTEGRATION_TESTED** | `src/schema/compatibility.ts`, `src/schema/schema-registry.ts`; `tests/p0/schema-evolution.test.ts` (12); the runner's schema gate runs before the first read — `tests/p0/ingestion-runner.test.ts` asserts `sinkCalls === 0` on a break |
| **P064** Quality rules behind the SIAX abstraction; failures block or quarantine per declared policy | **PARTIAL.** `src/quality/*` had rules, runs, results, issues, scores and a GE-compatible engine, but a failing rule only opened an ISSUE — the data flowed on regardless. `grep -ri quarantin src/` found one unrelated hit | **INTEGRATION_TESTED** | `src/quality/quality-policy.ts` composes the EXISTING `ExpectationEngine` (not a second engine) and adds the consequence; block / quarantine / warn each proved in `tests/p0/ingestion-runner.test.ts`; quarantine durability proved against real Postgres |
| **P065** Normalization/ER: only the canonical highway; no second simplified production engine | **CLOSED.** `services/api/workers/resolve.py` is now a thin adapter: Python no longer scores identity — it POSTs opaque records to `/api/0pt/internal/er/candidates` and applies ONLY explicit hard-rule `match` decisions (score/fuzzy can never merge records, even from a compromised bridge); canonical engine unreachable ⇒ records return UNMERGED, never fallback-scored. Company semantics `stable-identifiers.v3`: LEI global; org_nr hard identity only within the same registration_country; punctuation normalized; negative stable identity dominates every positive; fuzzy alone ⇒ human review, never auto-merge. Cluster-wide hard-negative constraints: A≈B ∧ B≈C never forces A≈C when canonical identity says A≠C | **INTEGRATION_TESTED** | `tests/p0/entity-resolution.test.ts` (v3 semantics incl. hard-negative transitivity), `tests/p0/entity-resolution-http.test.ts` (internal route fails closed: empty/wrong token ⇒ 401), `services/api/tests/test_resolve_determinism.py` (adapter contract: canonical edges only), shared request validation (`src/er/request-validation.ts`, records 2..5000), blocking pytest gate (`.gitea/workflows/python-engine.yml`) |
| **P066** Iceberg | out of scope — ST0RE is owned by another workflow | **NOT TOUCHED** | Contract read only (`src/services/iceberg-write-service.ts`, `src/clients/st0re-lakehouse-client.ts`) |
| **P067** Lineage: OpenLineage/AUD0 events; source → output lineage is queryable | **PARTIAL.** `src/lineage/lineage-emitter.ts` emitted OpenLineage to AUD0, but nothing in this repo could ANSWER a lineage question — no store, no query, no route | **INTEGRATION_TESTED** | `src/lineage/lineage-edge.ts`, `src/lineage/lineage-store.ts` (local queryable projection; AUD0 stays the system of record), `GET /api/0pt/lineage`; `tests/p0/ingestion-runner.test.ts` queries both directions and proves tenant scoping |
| **P068** Durable bounded backfill/replay; repeated backfill is idempotent | **MISSING**, documented as such: `services/api/tests/test_known_gaps.py:44-52` skipped it because "no delivery-store persistence layer exists to dedupe a backfill against" | **INTEGRATION_TESTED** | `opt_ingestion_ledger` + `IngestionRunner.backfill`; `tests/p0/ingestion-runner.test.ts` (repeat emits 0, live checkpoint untouched, a real content change still re-emits); same against real Postgres |

### What was actually run (not "should pass")

- `pnpm run typecheck` — clean.
- `pnpm run test` — **365 passed, 25 skipped** (was 311/18 before this pass).
- `pnpm run build` — clean; the four new routes appear in the route table.
- `pnpm run lint` — 12 errors / 5 warnings, **all pre-existing** (identical
  count on the base commit; advisory in CI). This pass adds none.
- Real Postgres 16: `tests/p0/ingestion-plane-postgres.realinfra.test.ts`
  with `INGESTION_TEST_DATABASE_URL` — 7 passed.
- **Real container restart**, run manually against that same database:
  `docker restart` the Postgres container, then re-read through the real
  repositories. Before and after were identical — 2 sources (one `active`
  with its owner, one `pending`), 1 checkpoint (cursor 6, 6 committed),
  6 ledger rows, 1 lineage edge, 4 quarantined records, 1 schema version.
  That is P060's "a restart preserves all sources", measured rather than
  asserted.

### A real bug this pass wrote and then caught

The first draft of `IngestionRepository` combined the idempotency check and
its recording into one atomic call, made BEFORE the sink. The crash/resume
test then delivered **7 of 10 records**: the page in flight when the sink
died had already been marked delivered, so the resumed run skipped it as a
duplicate — permanent, silent data loss. The fix is the `peekDeliveries` /
`commitDeliveries` split, with both the ledger commit and the checkpoint
commit after the sink resolves. Recorded here because the failure mode is
the interesting artifact, not the fix.

### Honest gaps in the new plane (not hidden)

1. **Delivery is AT-LEAST-ONCE, not exactly-once.** Two concurrent runs on
   the same (source, partition) can both peek "new" and both emit.
   `envelope_id` is deterministic so a dedup-capable sink survives it. There
   is no per-(source, partition) run lock. Do not describe this plane as
   exactly-once.
2. **No CDC *source* exists.** The envelope, the operations and the runner
   handle change events, and `JsonlFileConnector` emits real deletes from an
   `_op` field — but this repo still has no Debezium/WAL/replication-slot
   consumer. P062's contract is met; a CDC connector is still MISSING.
3. **0pt does not schedule itself.** `opt_source_schedule` is a declaration
   ACT0 would read; there is no cron daemon here.
4. **The sink is caller-supplied.** The runner stops at the sink; the
   lakehouse write boundary is ST0RE's (P066, another workflow's repo).
5. **Nothing here is deployed.** See the deployment note below.
6. **`QualityService`'s own rule/run/issue registry is still in-process
   Maps** (`src/quality/quality-service.ts:47-50`). The new *enforcement*
   path is durable; the older rule registry is not. Unchanged by this pass.
7. **Python tests still run nowhere.** `grep -rn "pytest" .gitea/workflows
   .github/workflows` finds nothing — `services/api/tests/*` (9 files) has
   never been executed by CI. Flagged, not fixed (it needs a workflow change
   and a Python toolchain decision).

### Deployment reality, measured 2026-09-04

`https://0pt.app/api/health` answers **200** from **Vercel** with
`{"db":"not_configured"}`, and `https://0pt.app/api/0pt/sources` answers
**404** — the deployed build predates the entire `app/api/0pt/*` capability
layer. `https://api.0pt.app/health` answers **404** (the FastAPI engine is
not reachable), and `.gitea/workflows/cd.yml` still guards on
`COOLIFY_APP_ID = TODO-no-coolify-app-provisioned-yet`. So: none of the
capability layer — neither this pass's work nor P010/P014's — is running in
production, and P060's durability guarantee cannot be exercised there,
because production has no database configured and does not serve the routes.
Verified with `curl`; not inferred.

### P065 — closure record (2026-09-07)

The removal plan below was executed in full and is verified in CI:

1. ✅ Canonical ER exposed as `POST /api/0pt/er/candidates` (public,
   tenant-safed) AND `POST /api/0pt/internal/er/candidates` (service-token,
   fails closed without `X-Internal-Token`).
2. ✅ Python CI job: `.gitea/workflows/python-engine.yml` runs
   `services/api/tests` as a BLOCKING pytest gate.
3. ✅ `workers/resolve.py` rewritten as the HTTP adapter — fail-closed to
   UNMERGED records when the bridge is unreachable; no fallback scorer
   remains. Cluster-wide hard-negative constraints prevent transitive
   matches from overriding canonical identity decisions.
4. ✅ `services/api/tests/test_resolve_determinism.py` asserts the adapter
   contract: canonical match edges merge; review/non-match edges do not;
   hard negatives are respected transitively.

Residual honesty: the PYTHON-side tests run in CI, but the full
adapter→Next.js route chain has not been exercised against a live deployed
edge in this branch (the route's own fail-closed behavior is unit-proven).
That keeps P065 at INTEGRATION_TESTED, not LIVE_VERIFIED.