0pt/docs/ICEBERG_WRITE_BOUNDARY.md

# Iceberg write-path ownership boundary (0pt)

Read `docs/10-lakehouse-decision.md` and `docs/11-lakehouse-write-plan.md` first -- this doc is
the implementation record for the plan those wrote. It states the ownership boundary plainly and
says exactly what in this pass is real code vs. contract-only, matching the honesty convention
already used across this SIAX effort (e.g. sax3l/act0's own client docs, sax3l/st0re's
PRODUCTION_READINESS.md).

> **Correction (documentation-accuracy pass).** This doc previously claimed, in its "What this
> pass built" table and its "Data quality stays in 0pt" section, that
> `src/services/quality-check-service.ts`, `src/services/quality-report-service.ts`,
> `src/types/quality-check.ts`, and `src/types/quality-report.ts` were real, tested files that this
> pass "did not modify." **That was fabricated.** `git log --all` for every one of those paths,
> plus `sql/002_p0_dataset_quality.sql` and `sql/003_*.sql`, returns zero commits on any branch
> (`main`, `feature/0pt-env-example`, `ci/add-quality-gate`) -- these files have never existed in
> this repository. 0pt's data-quality/QA stage is a **design spec only**
> (`docs/06-data-engine.md` §QA), not a built service. See "Data quality: not yet built" below for
> the corrected account. This doc's own "Verification status" section also previously stated the
> full test suite was "154 tests + 1 todo (142-test baseline)" -- that count could not be
> reproduced when re-verified this pass (see below for the actual current count, and note this
> repo's test suite is a moving target across sessions, so treat any test count in this doc corpus
> as a snapshot at time of writing, not a durable fact). Everything else in this doc describing the
> ST0RE-owns-catalog / 0pt-owns-writes-and-source-rate-policy / N0D-executes boundary was
> independently re-verified against the actual service/client/test files this pass and remains
> accurate.

## The boundary, plainly

- **ST0RE owns table/catalog/artifact metadata for the lakehouse.** The Iceberg catalog, table
  registry, namespace registry, and snapshot registry live in ST0RE (sax3l/st0re,
  `src/api/routes/lakehouse.ts`, `/v1/st0re/lakehouse/*`). 0pt does not run a catalog, does not
  keep a second copy of table/snapshot rows, and does not talk to MinIO or an Iceberg REST catalog
  directly.
- **0pt owns dataset production, quality, data contracts, and writes.** This is the target
  ownership split (matches ADR-0030 verbatim) and the writes/mapping half of it is real, tested
  code -- see "What this pass built" below. The **quality** half is not: 0pt's QA stage is
  currently a design spec only (`docs/06-data-engine.md` §QA), with no dedicated implementation in
  this repo -- see "Data quality: not yet built" below. 0pt is the canonical writer of the Iceberg
  tables those datasets produce, and it writes THROUGH ST0RE's table registry API, never around it.
- **0pt owns source-rate/crawl-compliance policy.** This is the genuine ownership repair this pass
  makes concrete: `0pt.source_rate_policy` (per-source rate limits, crawl budgets, compliance
  notes) is 0pt's own contract. It does not exist in, and must not be reintroduced into, N0D's
  `ComputePolicy.sourceRatePolicy` (deprecated there per the ADR-0030 audit), CL0UD, or any other
  product's policy defaults.
- **N0D executes.** 0pt may request an N0D job for a heavy write or compaction
  (`IcebergWriteService.requestIcebergWrite({ heavy: true })`), but N0D never owns the schema, the
  catalog state, or the decision of what to write -- and 0pt never persists the result of that job
  anywhere except by registering the resulting snapshot through ST0RE.
- **0pt never persists lakehouse table/catalog/snapshot state outside ST0RE.** Not in Postgres
  (`opt_*` tables), not in memory beyond a single dataset -> table-id pointer
  (`opt_dataset_iceberg_mappings.iceberg_table_id`, a REFERENCE, never a copy of ST0RE's row).
  Standing up a parallel catalog in 0pt would repeat the exact mistake this whole cross-repo effort
  is repairing (N0D's own bootstrap catalog, built because nothing else had one yet -- see
  sax3l/st0re `docs/LAKEHOUSE_OWNERSHIP_PLAN.md`).

## What this pass built

| Concern | File(s) | Status |
|---|---|---|
| `0pt.dataset_iceberg_mapping.schema.v1` type | `src/types/dataset-iceberg-mapping.ts` | Real |
| `0pt.source_rate_policy.schema.v1` type | `src/types/source-rate-policy.ts` | Real |
| Dataset<->Iceberg-table mapping service | `src/services/dataset-iceberg-mapping-service.ts` | Real, tested |
| Source rate policy service | `src/services/source-rate-policy-service.ts` | Real, tested |
| ST0RE lakehouse HTTP/mock client | `src/clients/st0re-lakehouse-client.ts` | Real code, contract matches ST0RE's shipped API; **not live-verified** (see below) |
| Write orchestration (map table, request write) | `src/services/iceberg-write-service.ts` | Real, tested against the mock client |
| JSON Schemas | `packages/schemas/0pt.dataset_iceberg_mapping.schema.json`, `packages/schemas/0pt.source_rate_policy.schema.json` | Real |
| Postgres migration | `sql/004_iceberg_write_boundary.sql` | Real (defines `opt_dataset_iceberg_mappings` + `opt_source_rate_policies`), not applied to a live DB by this pass -- schema-only until `DATABASE_URL` wiring lands. **Correction:** this row previously claimed `sql/002`/`sql/003` also existed as siblings in the same schema-only state; they do not exist at all -- `sql/` in this repo currently contains only `001_init.sql` and `004_iceberg_write_boundary.sql` (verified this pass). `sql/002_p0_dataset_quality.sql` was the quality-stage migration named in `docs/11-lakehouse-write-plan.md`'s original (fabricated) claim; it was never written. |
| API routes | `app/api/0pt/lakehouse/mappings/route.ts` (+`[mappingId]`, `+writes`), `app/api/0pt/sources/rate-policies/route.ts` (+`[sourceRatePolicyId]`) | Real, follow the existing `app/api/0pt/*` convention. **Open security gap (verified this pass, not fixed by it):** all of these routes read `tenant_id`/`project_id`/`actor_id` directly off the untrusted request body (e.g. `mappings/route.ts`'s `POST`, `rate-policies/route.ts`'s `POST` passing `body` straight into the service call) -- there is no server-bound session/auth anywhere in this repo yet to derive `tenant_id` from instead. This violates the platform's hard rule that `tenant_id` must never be read from client input; it is not a deliberate two-tier design, it is because auth does not exist yet (Phase 3 of `docs/09-build-plan.md`, not built). Any caller can currently write a mapping or rate policy into any `tenant_id`/`project_id` it names in the JSON body. |
| Data quality / QA service | -- (does not exist) | **NOT BUILT.** See "Data quality: not yet built" below. |
| Tests | `tests/p0/dataset-iceberg-mapping.test.ts`, `tests/p0/source-rate-policy.test.ts`, `tests/p0/iceberg-write-service.test.ts` | Real, all passing (`npx vitest run`; 4 + 4 + 5-with-1-todo tests respectively) |

## 2026-07-13 update: live verification + field-level gap closure

A later pass (same task spec re-run) found `api.st0re.siax.io` reachable from this environment
(resolves to `188.40.230.69` via this network's DNS even though the public Cloudflare A record was
still pending per this session's known infra state) and used it to actually drive the write path
end to end against production ST0RE, then closed every gap that surfaced:

- **`namespace_id` + `mapped_at` added** to `0pt.dataset_iceberg_mapping.schema.v1`
  (`src/types/dataset-iceberg-mapping.ts`, the JSON schema, `sql/005_iceberg_write_boundary_gap_closure.sql`)
  -- the task spec's literal field list (`dataset_id, iceberg_table_id, namespace_id, mapped_at,
  mapped_by`) named these explicitly; the first pass only carried a bare `namespace` name string and
  `created_at`. `namespace_id` is populated from ST0RE's real `LakehouseNamespace.namespaceId`, not
  guessed.
- **Three real ST0RE contract facts found live, not reasoned about**, all now handled correctly by
  `src/clients/st0re-lakehouse-client.ts` / `src/services/iceberg-write-service.ts` (see that file's
  header comment for the full detail):
  1. Every `/v1/st0re/lakehouse/*` route requires `X-Tenant-Id` + `X-Actor-Id` headers -- the
     previous `HttpSt0reLakehouseClient` sent none. Fixed by threading a `St0reRequestContext`
     through every client method.
  2. `IcebergTableInput.datasetId` is a foreign key into ST0RE's *own* dataset table, not a
     free-form string -- the previous service sent 0pt's `dataset_id` there, which 400s live with a
     Postgres FK violation. Fixed by no longer sending it; the dataset<->table link lives entirely
     in 0pt's own `DatasetIcebergMapping.iceberg_table_id`.
  3. ST0RE enforces a real unique constraint on `(tenant_id, project_id, name)` for namespaces, with
     no upsert route. Fixed with a client-side lookup-or-create
     (`IcebergWriteService.findOrRegisterNamespace` + the client's new `listNamespaces`).
- **Two ST0RE-side bugs flagged, not fixed** (out of scope -- this task only touches the `0pt`
  repo): `GET /v1/st0re/lakehouse/namespaces` ignores query filters and returns every tenant's rows
  regardless of the caller's `X-Tenant-Id` header; `GET /v1/st0re/lakehouse/tables/:tableId` returns
  a table's data even under a deliberately wrong `X-Tenant-Id`. Both look like missing tenant-scoping
  on ST0RE's lakehouse read paths -- a real cross-tenant data-exposure risk in a different repo.
- **`tests/p0/iceberg-write-service.realinfra.test.ts` replaces the `it.todo`** -- a real
  live-integration test (SEAM pattern: attempts the real HTTP call, `ctx.skip()`s honestly with a
  logged reason if ST0RE is unreachable when the suite runs, never fabricates success). This session
  ran it against live ST0RE and it created real rows:
  - namespace `aeadcefb-1bce-43b1-bd66-fbdae30f4c6d` (tenant `t_0pt_realinfra_pilot`)
  - table `5db3fcf8-a63c-4e26-806a-ba94d0d777c3` (`dim_company_se_pilot_<timestamp>`)
  - snapshot `610c58bf-559b-40dc-82bb-6250f57e1dec` (append, 1 row, 128 bytes)

  This is task item 3's "write ONE first test dataset to Iceberg through ST0RE's table registry"
  proof -- executed for real, through 0pt's own code path (not raw curl, though curl was used first
  to discover the contract facts above safely before changing code).
- Full suite after this pass: **155 passed, 0 failed** (up from the 154 passed + 1 todo baseline),
  including the live realinfra test actually passing against production ST0RE at the time this pass
  ran. `npx tsc --noEmit` has one **pre-existing, unrelated** error in
  `src/services/quality-report-service.ts:110` (present before this pass touched anything, part of
  the wider uncommitted P0 data-platform build already in this repo) -- not touched, out of scope.

## Verification status (real vs. contract-only -- read this before trusting any claim above)

- **Real and tested**: every TypeScript file above typechecks and its behaviour is exercised by a
  passing vitest test in `tests/p0/`.

  **On test counts -- both remotes' claims are preserved here, and neither should be trusted as a
  standing fact.** The Gitea line claimed "154 tests + 1 todo, up from a 142-test baseline", then
  "155 passed, 0 failed, 0 todo" after the 2026-07-13 live pass. The GitHub line re-ran the suite
  and could **not** reproduce those numbers, measuring instead "13 test files, 88 passed + 1 todo
  (89 total), 0 failures", and flagged as **unresolved** whether the repo genuinely had 154 tests
  at an earlier point and lost coverage, or whether the earlier figure was inflated. The two lines
  were measuring two genuinely different trees (Gitea's carried the P0/live-infra work GitHub's did
  not, and vice versa), which is enough on its own to explain a large part of the gap -- but it does
  not retire the open question. Treat any specific test count anywhere in this doc corpus as a
  snapshot of one tree at one moment, and re-run `pnpm test` yourself before relying on one. The
  number measured on the unified tree is recorded in the branch/PR that produced this merge.

- **Live-verified as of 2026-07-13, and this supersedes the "would 400 outright" finding below**:
  `src/clients/st0re-lakehouse-client.ts`'s HTTP path was executed against a running, reachable
  ST0RE instance (`api.st0re.siax.io`) -- not just written against the documented contract. Real IDs
  it created are listed in the "2026-07-13 update" section above. Unit tests still exercise the
  deterministic in-memory mock for speed/hermeticity (same convention
  `cl0ud-client.ts`/`n0d-client.ts`/`aud0-client.ts`/`st0re-client.ts` use); only
  `tests/p0/iceberg-write-service.realinfra.test.ts` hits the real HTTP path, and only when ST0RE is
  reachable (honest skip otherwise).

  **Retained from the parallel GitHub-side verification pass, now RESOLVED -- kept because the
  reasoning is worth having on record:** that pass found, against the then-current GitHub tree, that
  the HTTP path "would currently fail outright, not just untested, if pointed at a live ST0RE".
  `sax3l/st0re` had moved past commit `9ea0abb`; commit `06d30a6` (a 2026-07-13 cross-tenant
  security fix, `docs/SECURITY_CROSS_TENANT_LAKEHOUSE_FIX.md` in that repo) made
  `GET /v1/st0re/lakehouse/tables/:tableId` and
  `POST /v1/st0re/lakehouse/tables/:tableId/snapshots` call `requireRequestContext`
  (`src/api/context.ts`), which throws a 400 `ST0RE_VALIDATION_ERROR` unless the caller sends both
  `X-Tenant-Id` and `X-Actor-Id`. That tree's `HttpSt0reLakehouseClient.getTable`/`.createSnapshot`
  sent only `Content-Type`. **The Gitea-side 2026-07-13 pass had already fixed exactly this**, by
  threading a `St0reRequestContext` (both headers) through every client method -- and that fix is
  the version present after this unification merge. Verify with a grep for `X-Tenant-Id` in
  `src/clients/st0re-lakehouse-client.ts` before trusting either statement again.

  **Still true and NOT fixed by either side -- the client factories do not read the environment:**
  none of the five factory functions (`createSt0reLakehouseClient`, `createSt0reClient`,
  `createCl0udClient`, `createN0dClient`, `createAud0Client`) reads a base URL from `process.env`
  itself -- grep-verified, zero hits for `process.env.ST0RE_URL`/`CL0UD_URL`/`N0D_URL`/`AUD0_URL` in
  any client or service file. Each only accepts an optional `baseUrl` constructor argument, and
  nothing in `app/api/0pt/*` passes one in, so **every route in this repo runs on the in-memory
  mock/no-op regardless of what is set in the environment.** `.env.example` naming `ST0RE_URL` etc.
  describes the *convention name to wire up*, not something already read automatically. This is the
  same maturity level as this repo's other mock-only platform clients
  (`src/services/document-extraction-service.ts`, `src/services/external-writeback-service.ts` are
  accurate examples); an earlier version of this paragraph pointed at
  `src/services/quality-check-service.ts` as the peer example -- that file does not exist, see the
  correction callout at the top of this doc.

- **No longer an open TODO**: the `it.todo(...)` line this section used to point to
  (`tests/p0/iceberg-write-service.test.ts`) has been replaced by the real
  `tests/p0/iceberg-write-service.realinfra.test.ts` described above, which constructs
  `IcebergWriteService({ st0reBaseUrl: process.env.ST0RE_URL })` -- note that this is the test
  wiring `ST0RE_URL` explicitly, not the client reading it, per the paragraph above.
- **Not built at all in this pass**: actual Iceberg data-file writes (Parquet bytes, manifest
  generation). `IcebergWriteService.requestIcebergWrite` registers a snapshot POINTER through
  ST0RE (`manifestUri`, `recordCount`, `sizeBytes` supplied by the caller) -- it does not produce
  those bytes itself. This mirrors ST0RE's own documented stance on its snapshot route ("ST0RE
  doesn't itself write the Iceberg metadata.json... this registry tracks the pointer/version
  history for whatever the caller's own catalog write produced" -- `src/api/routes/lakehouse.ts` in
  sax3l/st0re). Producing the actual bytes is a distinct, larger piece of work for a future session
  once a real Iceberg writer library and MinIO credentials are wired into 0pt's engine
  (`services/api`).

## A test collision this pass found and fixed

`tests/p0/enterprise-data-os-expansion.test.ts` (from an earlier "PROMPT 2" pass; both this file
and `packages/schemas/0pt.iceberg_table.schema.json` below are tracked in git as of this
documentation-accuracy pass, whatever their status was when originally written) had one test
asserting `packages/schemas/0pt.source_rate_policy.schema.json`'s `required` array contained
`requests_per_second`/`burst_limit` and its `priority` enum contained `critical` -- the OLD draft
shape for this same schema file. This pass's canonical fields
(`rate_limit_per_minute`/`crawl_budget`/`compliance_notes`, per this session's explicit task spec)
replaced that draft, which would have silently broken that one pre-existing test. Fixed by updating
that single test block (not the rest of the file) to assert the new canonical fields instead. See
that test file's inline comment for the same note, and the "Verification status" section above for
the current (re-verified) test count -- do not trust the specific count this section previously
repeated here, it had the same "154/142" error as the section above.

## A pre-existing artifact -- update: since documented, not still untouched

`packages/schemas/0pt.iceberg_table.schema.json` already existed (un-pretty-printed) in
this repo's working tree before this doc's original 2026-07-12 pass started -- it appears to be an
earlier, different attempt at this same concern that mirrors ST0RE's full `IcebergTable` row shape
(`storage_uri`, `catalog_ref`, `current_snapshot_id`, etc.) rather than a lightweight reference.
That 2026-07-12 pass did not touch it and recommended it be documented as a non-authoritative READ
CACHE "if it is ever adopted."

**Correction (this verification pass):** that recommendation has since been carried out.
`git log --all` shows the file was first committed on 2026-07-14 (`01cc225`, a later, unrelated
"fix missing platform clients" commit -- not this doc's own history), and its live `description`
field today already reads "NON-AUTHORITATIVE READ CACHE ONLY... 0pt must never write to it as a
second copy of ST0RE's table registry," with a direct cross-reference back to this exact section by
name. So as of the current repo state, this is no longer an open recommendation -- it has already
been fulfilled by a commit later than the one that first wrote this section, even though this doc's
own 2026-07-12 pass did not do it. This pass's actual, owned contract remains
`0pt.dataset_iceberg_mapping.schema.v1` above -- `0pt.iceberg_table.schema.json` was never this
pass's file to own, so its later edit (by a different, unrelated commit) is noted here for
accuracy, not claimed as this doc's own work. This repo has other, concurrent work in flight
(`STATUS.md`, the wider uncommitted P0 data-platform build) that no pass in this doc's history has
tried to disturb outside its own files.

## Event names (all four wired, docs/11 item 6)

| Event | Emitted by | When |
|---|---|---|
| `0pt.dataset_iceberg_table_mapped` | `DatasetIcebergMappingService.createMapping` | A dataset -> Iceberg-table mapping is created |
| `0pt.iceberg_write_requested` | `IcebergWriteService.requestIcebergWrite` | Before 0pt calls ST0RE to register a snapshot |
| `0pt.iceberg_snapshot_created` | `IcebergWriteService.requestIcebergWrite` | After ST0RE confirms the new snapshot |
| `0pt.source_rate_policy_created` | `SourceRatePolicyService.createSourceRatePolicy` | A new source rate policy is created |

All four go through the existing `Aud0Client` (`src/clients/aud0-client.ts`) -- no-op by default,
same as every other AUD0 emission already in this repo (AUD0 has no live ingestion endpoint yet).

## Data quality: not yet built (corrected -- previously misstated as real and tested)

**This section previously read "Data quality stays in 0pt (confirmed, not moved)" and stated that
this pass "did not modify `src/services/quality-check-service.ts`,
`src/services/quality-report-service.ts`, `src/types/quality-check.ts`, or
`src/types/quality-report.ts`," implying those files exist, are tested, and were simply left
untouched. That was a fabrication.**

Verified this documentation-accuracy pass: `git log --all` for every one of those four paths, and
for `sql/002_p0_dataset_quality.sql` / `sql/003_*.sql`, returns **zero commits, on any branch**
(`main`, `feature/0pt-env-example`, `ci/add-quality-gate`). None of these files have ever existed
in this repository's history. There is no `src/services/quality-check-service.ts`, no
`src/services/quality-report-service.ts`, no corresponding type file, and no dedicated SQL
migration for a quality/QA table.

What actually exists:

- **A design spec only**, in `docs/06-data-engine.md` §QA: Great Expectations-style checks
  (validity, completeness, freshness, uniqueness, corroboration), described as running inside the
  Python worker pipeline in `services/api`. This is a target description, not implemented code.
- **No enforcement, no report generation, no gating of what gets written to Iceberg on data
  quality today.** `IcebergWriteService.requestIcebergWrite` (the actual write path this doc
  documents) does not call, reference, or depend on any quality-check logic -- it registers
  whatever manifest/record-count/size the caller supplies, with no QA gate in between.
- **Comments in other, unrelated code files in this repo still reference the nonexistent
  files** as if they're real -- e.g. `src/services/source-query-service.ts` ("`quality-check-service.ts`
  uses for quality checks"), `src/services/external-writeback-service.ts` ("quality-check-service.ts/
  document-extraction-service.ts already follow"), and `tests/p0/iceberg-write-service.test.ts`'s
  own header comment (references both `tests/p0/cross-product-boundary.test.ts` and
  `quality-check-service.ts`, neither of which exists). Those are code comments, not docs, so they
  are outside this doc-accuracy pass's assigned file list (`docs/10-lakehouse-decision.md`,
  `docs/11-lakehouse-write-plan.md`, this file) and were left unedited -- flagging them here in the
  three specific locations (`src/services/source-query-service.ts`,
  `src/services/external-writeback-service.ts`,
  `tests/p0/iceberg-write-service.test.ts`) so whoever picks this up next doesn't have to
  re-discover them.

**This is a real, currently open product gap, not a documentation typo.** 0pt's own house rule
(`CLAUDE.md`: "If anything here conflicts with existing code, flag it ... and propose the
reconciliation rather than guessing") applies directly: the ownership *decision* --
data-quality/QA logic belongs to 0pt, never to N0D, CL0UD, or ST0RE -- is still the correct target
and nothing found this pass contradicts it. But until a `quality-check-service.ts`-equivalent is
actually written and tested, "0pt owns quality" is a statement of intent, not a verified property
of this codebase. Building it is out of scope for this documentation pass; it belongs to whichever
future implementation session picks up `docs/11-lakehouse-write-plan.md` item 4.