# Iceberg write-path ownership boundary (0pt) Read `docs/10-lakehouse-decision.md` and `docs/11-lakehouse-write-plan.md` first -- this doc is the implementation record for the plan those wrote. It states the ownership boundary plainly and says exactly what in this pass is real code vs. contract-only, matching the honesty convention already used across this SIAX effort (e.g. sax3l/act0's own client docs, sax3l/st0re's PRODUCTION_READINESS.md). ## The boundary, plainly - **ST0RE owns table/catalog/artifact metadata for the lakehouse.** The Iceberg catalog, table registry, namespace registry, and snapshot registry live in ST0RE (sax3l/st0re, `src/api/routes/lakehouse.ts`, `/v1/st0re/lakehouse/*`). 0pt does not run a catalog, does not keep a second copy of table/snapshot rows, and does not talk to MinIO or an Iceberg REST catalog directly. - **0pt owns dataset production, quality, data contracts, and writes.** 0pt runs sourcing -> extraction -> structuring -> enrichment -> QA (`docs/06-data-engine.md`) as it always has, and is the canonical writer of the Iceberg tables those datasets produce -- but it writes THROUGH ST0RE's table registry API, never around it. - **0pt owns source-rate/crawl-compliance policy.** This is the genuine ownership repair this pass makes concrete: `0pt.source_rate_policy` (per-source rate limits, crawl budgets, compliance notes) is 0pt's own contract. It does not exist in, and must not be reintroduced into, N0D's `ComputePolicy.sourceRatePolicy` (deprecated there per the ADR-0030 audit), CL0UD, or any other product's policy defaults. - **N0D executes.** 0pt may request an N0D job for a heavy write or compaction (`IcebergWriteService.requestIcebergWrite({ heavy: true })`), but N0D never owns the schema, the catalog state, or the decision of what to write -- and 0pt never persists the result of that job anywhere except by registering the resulting snapshot through ST0RE. - **0pt never persists lakehouse table/catalog/snapshot state outside ST0RE.** Not in Postgres (`opt_*` tables), not in memory beyond a single dataset -> table-id pointer (`opt_dataset_iceberg_mappings.iceberg_table_id`, a REFERENCE, never a copy of ST0RE's row). Standing up a parallel catalog in 0pt would repeat the exact mistake this whole cross-repo effort is repairing (N0D's own bootstrap catalog, built because nothing else had one yet -- see sax3l/st0re `docs/LAKEHOUSE_OWNERSHIP_PLAN.md`). ## What this pass built | Concern | File(s) | Status | |---|---|---| | `0pt.dataset_iceberg_mapping.schema.v1` type | `src/types/dataset-iceberg-mapping.ts` | Real | | `0pt.source_rate_policy.schema.v1` type | `src/types/source-rate-policy.ts` | Real | | Dataset<->Iceberg-table mapping service | `src/services/dataset-iceberg-mapping-service.ts` | Real, tested | | Source rate policy service | `src/services/source-rate-policy-service.ts` | Real, tested | | ST0RE lakehouse HTTP/mock client | `src/clients/st0re-lakehouse-client.ts` | Real code, contract matches ST0RE's shipped API; **not live-verified** (see below) | | Write orchestration (map table, request write) | `src/services/iceberg-write-service.ts` | Real, tested against the mock client | | JSON Schemas | `packages/schemas/0pt.dataset_iceberg_mapping.schema.json`, `packages/schemas/0pt.source_rate_policy.schema.json` | Real | | Postgres migration | `sql/004_iceberg_write_boundary.sql` | Real (not applied to a live DB by this pass -- same as `sql/002`/`sql/003`, which this repo's own P0 docs already note are schema-only until `DATABASE_URL` wiring lands) | | API routes | `app/api/0pt/lakehouse/mappings/route.ts` (+`[mappingId]`, `+writes`), `app/api/0pt/sources/rate-policies/route.ts` (+`[sourceRatePolicyId]`) | Real, follow the existing `app/api/0pt/*` convention | | Tests | `tests/p0/dataset-iceberg-mapping.test.ts`, `tests/p0/source-rate-policy.test.ts`, `tests/p0/iceberg-write-service.test.ts` | Real, all passing (`pnpm test`); one explicit `it.todo` for the live-ST0RE case | ## 2026-07-13 update: live verification + field-level gap closure A later pass (same task spec re-run) found `api.st0re.siax.io` reachable from this environment (resolves to `188.40.230.69` via this network's DNS even though the public Cloudflare A record was still pending per this session's known infra state) and used it to actually drive the write path end to end against production ST0RE, then closed every gap that surfaced: - **`namespace_id` + `mapped_at` added** to `0pt.dataset_iceberg_mapping.schema.v1` (`src/types/dataset-iceberg-mapping.ts`, the JSON schema, `sql/005_iceberg_write_boundary_gap_closure.sql`) -- the task spec's literal field list (`dataset_id, iceberg_table_id, namespace_id, mapped_at, mapped_by`) named these explicitly; the first pass only carried a bare `namespace` name string and `created_at`. `namespace_id` is populated from ST0RE's real `LakehouseNamespace.namespaceId`, not guessed. - **Three real ST0RE contract facts found live, not reasoned about**, all now handled correctly by `src/clients/st0re-lakehouse-client.ts` / `src/services/iceberg-write-service.ts` (see that file's header comment for the full detail): 1. Every `/v1/st0re/lakehouse/*` route requires `X-Tenant-Id` + `X-Actor-Id` headers -- the previous `HttpSt0reLakehouseClient` sent none. Fixed by threading a `St0reRequestContext` through every client method. 2. `IcebergTableInput.datasetId` is a foreign key into ST0RE's *own* dataset table, not a free-form string -- the previous service sent 0pt's `dataset_id` there, which 400s live with a Postgres FK violation. Fixed by no longer sending it; the dataset<->table link lives entirely in 0pt's own `DatasetIcebergMapping.iceberg_table_id`. 3. ST0RE enforces a real unique constraint on `(tenant_id, project_id, name)` for namespaces, with no upsert route. Fixed with a client-side lookup-or-create (`IcebergWriteService.findOrRegisterNamespace` + the client's new `listNamespaces`). - **Two ST0RE-side bugs flagged, not fixed** (out of scope -- this task only touches the `0pt` repo): `GET /v1/st0re/lakehouse/namespaces` ignores query filters and returns every tenant's rows regardless of the caller's `X-Tenant-Id` header; `GET /v1/st0re/lakehouse/tables/:tableId` returns a table's data even under a deliberately wrong `X-Tenant-Id`. Both look like missing tenant-scoping on ST0RE's lakehouse read paths -- a real cross-tenant data-exposure risk in a different repo. - **`tests/p0/iceberg-write-service.realinfra.test.ts` replaces the `it.todo`** -- a real live-integration test (SEAM pattern: attempts the real HTTP call, `ctx.skip()`s honestly with a logged reason if ST0RE is unreachable when the suite runs, never fabricates success). This session ran it against live ST0RE and it created real rows: - namespace `aeadcefb-1bce-43b1-bd66-fbdae30f4c6d` (tenant `t_0pt_realinfra_pilot`) - table `5db3fcf8-a63c-4e26-806a-ba94d0d777c3` (`dim_company_se_pilot_`) - snapshot `610c58bf-559b-40dc-82bb-6250f57e1dec` (append, 1 row, 128 bytes) This is task item 3's "write ONE first test dataset to Iceberg through ST0RE's table registry" proof -- executed for real, through 0pt's own code path (not raw curl, though curl was used first to discover the contract facts above safely before changing code). - Full suite after this pass: **155 passed, 0 failed** (up from the 154 passed + 1 todo baseline), including the live realinfra test actually passing against production ST0RE at the time this pass ran. `npx tsc --noEmit` has one **pre-existing, unrelated** error in `src/services/quality-report-service.ts:110` (present before this pass touched anything, part of the wider uncommitted P0 data-platform build already in this repo) -- not touched, out of scope. ## Verification status (real vs. contract-only -- read this before trusting any claim above) - **Real and tested**: every TypeScript file above typechecks (except the one pre-existing, unrelated `quality-report-service.ts` error noted above) and its behavior is exercised by a passing vitest test in `tests/p0/`. The full suite (`pnpm test`) passed at 154 tests + 1 todo after the first pass, up from a 142-test baseline before it, with 0 failures; the 2026-07-13 pass above brought it to 155 passed, 0 failed, 0 todo, with the live ST0RE path now actually exercised. - **Live-verified as of 2026-07-13** (supersedes the "not live-verified" note the first pass left here): `src/clients/st0re-lakehouse-client.ts`'s HTTP path was executed against a running, reachable ST0RE instance (`api.st0re.siax.io`) by a later pass -- not just written against the documented contract. Real IDs it created are listed in the "2026-07-13 update" section above. Unit tests still exercise the deterministic in-memory mock for speed/hermeticity (same convention `cl0ud-client.ts`/`n0d-client.ts`/`aud0-client.ts`/`st0re-client.ts` use); only `tests/p0/iceberg-write-service.realinfra.test.ts` hits the real HTTP path, and only when ST0RE is reachable (honest skip otherwise). - **No longer an open TODO**: the `it.todo(...)` line this section used to point to (`tests/p0/iceberg-write-service.test.ts`) has been replaced by the real `tests/p0/iceberg-write-service.realinfra.test.ts` described above. - **Not built at all in this pass**: actual Iceberg data-file writes (Parquet bytes, manifest generation). `IcebergWriteService.requestIcebergWrite` registers a snapshot POINTER through ST0RE (`manifestUri`, `recordCount`, `sizeBytes` supplied by the caller) -- it does not produce those bytes itself. This mirrors ST0RE's own documented stance on its snapshot route ("ST0RE doesn't itself write the Iceberg metadata.json... this registry tracks the pointer/version history for whatever the caller's own catalog write produced" -- `src/api/routes/lakehouse.ts` in sax3l/st0re). Producing the actual bytes is a distinct, larger piece of work for a future session once a real Iceberg writer library and MinIO credentials are wired into 0pt's engine (`services/api`). ## A test collision this pass found and fixed `tests/p0/enterprise-data-os-expansion.test.ts` (untracked, pre-existing, from an earlier "PROMPT 2" pass) had one test asserting `packages/schemas/0pt.source_rate_policy.schema.json`'s `required` array contained `requests_per_second`/`burst_limit` and its `priority` enum contained `critical` -- the OLD draft shape for this same schema file. This pass's canonical fields (`rate_limit_per_minute`/`crawl_budget`/`compliance_notes`, per this session's explicit task spec) replaced that draft, which would have silently broken that one pre-existing test. Fixed by updating that single test block (not the rest of the file) to assert the new canonical fields instead -- `pnpm test` is 154 passed + 1 todo, 0 failures, after this pass (up from a 142-test baseline before it). See that test file's inline comment for the same note. ## A pre-existing artifact this pass did not touch `packages/schemas/0pt.iceberg_table.schema.json` already existed (untracked, un-pretty-printed) in this repo's working tree before this pass started -- it appears to be an earlier, different attempt at this same concern that mirrors ST0RE's full `IcebergTable` row shape (`storage_uri`, `catalog_ref`, `current_snapshot_id`, etc.) rather than a lightweight reference. If it is ever adopted, it must be documented as a non-authoritative READ CACHE only -- 0pt must never treat it as the source of truth or write to it independently of ST0RE. This pass's actual, owned contract is `0pt.dataset_iceberg_mapping.schema.v1` above; it was left in place, unmodified, rather than deleted or silently rewritten, since it wasn't part of this task's scope and this repo has other, concurrent work in flight (`STATUS.md`, the wider uncommitted P0 data-platform build) that this pass did not want to disturb outside its own files. ## Event names (all four wired, docs/11 item 6) | Event | Emitted by | When | |---|---|---| | `0pt.dataset_iceberg_table_mapped` | `DatasetIcebergMappingService.createMapping` | A dataset -> Iceberg-table mapping is created | | `0pt.iceberg_write_requested` | `IcebergWriteService.requestIcebergWrite` | Before 0pt calls ST0RE to register a snapshot | | `0pt.iceberg_snapshot_created` | `IcebergWriteService.requestIcebergWrite` | After ST0RE confirms the new snapshot | | `0pt.source_rate_policy_created` | `SourceRatePolicyService.createSourceRatePolicy` | A new source rate policy is created | All four go through the existing `Aud0Client` (`src/clients/aud0-client.ts`) -- no-op by default, same as every other AUD0 emission already in this repo (AUD0 has no live ingestion endpoint yet). ## Data quality stays in 0pt (confirmed, not moved) This pass did not modify `src/services/quality-check-service.ts`, `src/services/quality-report-service.ts`, `src/types/quality-check.ts`, or `src/types/quality-report.ts`. 0pt's QA stage (`docs/06-data-engine.md`) remains entirely 0pt-owned; nothing in this pass routes quality logic through N0D, CL0UD, or ST0RE. N0D may still be asked to EXECUTE a quality-check job (`quality-check-service.ts`'s existing `n0d.createJob` call, unrelated to this pass), but the rules being executed are always 0pt's own.