Replaces the in-memory (per-instance, not even per-process-shared) Maps in
SourceService/SourceRegistryService with real Postgres persistence, and adds
the register/query/describe/health/rights/provenance/freshness HTTP routes
that never existed (grep for SourceRegistryService in app/ previously
returned nothing).
- sql/002_source_registry.sql: opt_sources, opt_source_rights,
opt_source_provenance, opt_source_health_checks, opt_source_freshness.
Numbered 002 (not 006) because 004's opt_source_rate_policies already
forward-references opt_sources, which no sql/ file defined until now —
fixes 004's pre-existing broken migration ordering (opt_datasets remains
a separate, unrelated, still-open gap in 004, flagged not fixed).
- src/repositories/*: SourceRepository / SourceRegistryRepository seams,
each with an in-memory implementation (business logic extracted verbatim)
and a Postgres implementation, chosen via resolveStoreMode — production
still fails hard with no DATABASE_URL (Master-DoD §3), but a configured
production now actually works instead of always being blocked.
- SourceService / SourceRegistryService: storage moved behind the
repository seam; getHealth/getFreshness are now async and tenant-scoped
at the accessor itself (needed once they're reachable directly over HTTP,
not only after an earlier caller's tenant check).
- src/services/source-registry.ts: shared singleton instances for the
sources/* route group, fixing the "each route module owned its own
private registry" defect (source-rate-policies and .../query previously
couldn't see a source registered via POST /api/0pt/sources).
- New routes: POST/GET /api/0pt/sources, GET /api/0pt/sources/[id],
GET+PUT .../rights, GET+PUT .../provenance, GET+POST .../health,
GET .../freshness.
- Business logic unchanged: fail-closed purpose gate, cross-tenant
NOT_FOUND (no existence oracle, §50), bounded 100-row health history.
Tested against a real Postgres container (not mocked) —
tests/p0/source-registry-postgres.realinfra.test.ts, env-gated on
SOURCE_REGISTRY_TEST_DATABASE_URL, skips honestly when unset. All 249
pre-existing P0 tests + 7 new realinfra tests pass; also smoke-tested the
full HTTP route surface end-to-end against next dev + live Postgres.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Re-runs this session's task spec against 0pt's actual repo state, which
turned out to already have a substantial, tested implementation from a
prior pass (35b6ec6): dataset<->Iceberg-table mapping, source rate policy,
and an ST0RE lakehouse client that was honestly marked "not live-verified".
This pass found api.st0re.siax.io IS reachable from this environment and
used it -- first via curl to safely discover the real contract, then
through 0pt's own updated code -- to drive a real namespace -> table ->
snapshot write against production ST0RE (task item 3's "write ONE first
test dataset" proof). That surfaced and fixed three real bugs the
mock-only pass could not have caught:
- ST0RE requires X-Tenant-Id/X-Actor-Id headers on every lakehouse route;
the previous HTTP client sent none. Fixed via a threaded
St0reRequestContext.
- IcebergTableInput.datasetId is a foreign key into ST0RE's OWN dataset
table, not a free-form string -- sending 0pt's dataset_id 400s live with
a Postgres FK violation. Fixed by no longer sending it; 0pt's
dataset<->table linkage lives entirely in DatasetIcebergMapping.
- ST0RE enforces a real unique constraint on lakehouse namespaces
(tenant_id, project_id, name) with no upsert route. Fixed with a
client-side lookup-or-create (listNamespaces + findOrRegisterNamespace).
Also closed a field-level gap against the task's literal schema spec:
0pt.dataset_iceberg_mapping.schema.v1 now carries namespace_id (a real
reference to ST0RE's LakehouseNamespace.namespaceId, not guessed) and
mapped_at, both additive alongside the existing namespace/created_at
fields (sql/005, JSON schema, types, service).
tests/p0/iceberg-write-service.realinfra.test.ts replaces the prior
it.todo with a real SEAM-pattern live test: attempts the real HTTP call,
skips honestly if ST0RE is unreachable, never fabricates success. It ran
successfully against production ST0RE this session (real namespace/table/
snapshot UUIDs logged in docs/ICEBERG_WRITE_BOUNDARY.md).
Two ST0RE-side cross-tenant read bugs were found and flagged (not fixed
here -- out of scope for this repo): GET .../namespaces ignores its own
tenant header and query filters; GET .../tables/:id doesn't check the
caller's tenant against the table's owner.
npm test: 155 passed, 0 failed, 0 todo (up from 154 passed + 1 todo).
npx tsc --noEmit: 1 pre-existing, unrelated error in
quality-report-service.ts (present before this pass, part of the wider
uncommitted P0 data-platform build already in this repo) -- not touched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Real, tested implementation of docs/11-lakehouse-write-plan.md's items 2-8
(the "future implementation session" that doc named): 0pt is now the
canonical writer of Iceberg tables through ST0RE's real, shipped
/v1/st0re/lakehouse/* table registry (sax3l/st0re commit 9ea0abb), never
persisting lakehouse table/catalog/snapshot state itself.
- src/types + src/services for 0pt.dataset_iceberg_mapping.schema.v1
(dataset -> ST0RE tableId reference) and 0pt.source_rate_policy.schema.v1
(the sourceRatePolicy ownership repair, moved out of N0D per ADR-0030).
- src/clients/st0re-lakehouse-client.ts: new client matching ST0RE's actual
camelCase route contract, with the same mock/HTTP boundary convention as
this repo's existing cl0ud/n0d/aud0/st0re clients. Honest caveat in its
module doc comment: not verified against a live ST0RE instance in this
pass (no reachable ST0RE_URL); every test exercises the deterministic mock.
- src/services/iceberg-write-service.ts: registers tables/snapshots through
ST0RE, requests N0D jobs only for heavy writes (never executes them
in-process), emits all four AUD0 event names
(dataset_iceberg_table_mapped / iceberg_write_requested /
iceberg_snapshot_created / source_rate_policy_created).
- sql/004, packages/schemas/*.json, app/api/0pt/lakehouse + sources/rate-policies
routes, tests/p0/*.test.ts (154 passed + 1 honest it.todo for the live-ST0RE
case, up from 142 baseline, 0 failures).
- Reconciled a real collision: an earlier untracked draft schema/test pair
(tests/p0/enterprise-data-os-expansion.test.ts) asserted a different,
now-superseded source_rate_policy shape; updated that one test block only.
- docs/ICEBERG_WRITE_BOUNDARY.md states the ownership boundary plainly and
is explicit throughout about what's real-and-tested vs. contract-only.
docs/11-lakehouse-write-plan.md gets a status-update section, not a rewrite.
- Data quality rules (quality-check-service.ts, quality-report-service.ts)
were not touched -- confirmed still 0pt-owned.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
adminandClaude Sonnet 5