0pt/docs/ICEBERG_WRITE_BOUNDARY.md

Iceberg write-path ownership boundary (0pt)

Read docs/10-lakehouse-decision.md and docs/11-lakehouse-write-plan.md first -- this doc is the implementation record for the plan those wrote. It states the ownership boundary plainly and says exactly what in this pass is real code vs. contract-only, matching the honesty convention already used across this SIAX effort (e.g. sax3l/act0's own client docs, sax3l/st0re's PRODUCTION_READINESS.md).

Correction (documentation-accuracy pass). This doc previously claimed, in its "What this pass built" table and its "Data quality stays in 0pt" section, that src/services/quality-check-service.ts, src/services/quality-report-service.ts, src/types/quality-check.ts, and src/types/quality-report.ts were real, tested files that this pass "did not modify." That was fabricated. git log --all for every one of those paths, plus sql/002_p0_dataset_quality.sql and sql/003_*.sql, returns zero commits on any branch (main, feature/0pt-env-example, ci/add-quality-gate) -- these files have never existed in this repository. 0pt's data-quality/QA stage is a design spec only (docs/06-data-engine.md §QA), not a built service. See "Data quality: not yet built" below for the corrected account. This doc's own "Verification status" section also previously stated the full test suite was "154 tests + 1 todo (142-test baseline)" -- that count could not be reproduced when re-verified this pass (see below for the actual current count, and note this repo's test suite is a moving target across sessions, so treat any test count in this doc corpus as a snapshot at time of writing, not a durable fact). Everything else in this doc describing the ST0RE-owns-catalog / 0pt-owns-writes-and-source-rate-policy / N0D-executes boundary was independently re-verified against the actual service/client/test files this pass and remains accurate.

The boundary, plainly

• ST0RE owns table/catalog/artifact metadata for the lakehouse. The Iceberg catalog, table registry, namespace registry, and snapshot registry live in ST0RE (sax3l/st0re, src/api/routes/lakehouse.ts, /v1/st0re/lakehouse/*). 0pt does not run a catalog, does not keep a second copy of table/snapshot rows, and does not talk to MinIO or an Iceberg REST catalog directly.
• 0pt owns dataset production, quality, data contracts, and writes. This is the target ownership split (matches ADR-0030 verbatim) and the writes/mapping half of it is real, tested code -- see "What this pass built" below. The quality half is not: 0pt's QA stage is currently a design spec only (docs/06-data-engine.md §QA), with no dedicated implementation in this repo -- see "Data quality: not yet built" below. 0pt is the canonical writer of the Iceberg tables those datasets produce, and it writes THROUGH ST0RE's table registry API, never around it.
• 0pt owns source-rate/crawl-compliance policy. This is the genuine ownership repair this pass makes concrete: 0pt.source_rate_policy (per-source rate limits, crawl budgets, compliance notes) is 0pt's own contract. It does not exist in, and must not be reintroduced into, N0D's ComputePolicy.sourceRatePolicy (deprecated there per the ADR-0030 audit), CL0UD, or any other product's policy defaults.
• N0D executes. 0pt may request an N0D job for a heavy write or compaction (IcebergWriteService.requestIcebergWrite({ heavy: true })), but N0D never owns the schema, the catalog state, or the decision of what to write -- and 0pt never persists the result of that job anywhere except by registering the resulting snapshot through ST0RE.
• 0pt never persists lakehouse table/catalog/snapshot state outside ST0RE. Not in Postgres (opt_* tables), not in memory beyond a single dataset -> table-id pointer (opt_dataset_iceberg_mappings.iceberg_table_id, a REFERENCE, never a copy of ST0RE's row). Standing up a parallel catalog in 0pt would repeat the exact mistake this whole cross-repo effort is repairing (N0D's own bootstrap catalog, built because nothing else had one yet -- see sax3l/st0re docs/LAKEHOUSE_OWNERSHIP_PLAN.md).

What this pass built

Concern	File(s)	Status
0pt.dataset_iceberg_mapping.schema.v1 type	src/types/dataset-iceberg-mapping.ts	Real
0pt.source_rate_policy.schema.v1 type	src/types/source-rate-policy.ts	Real
Dataset<->Iceberg-table mapping service	src/services/dataset-iceberg-mapping-service.ts	Real, tested
Source rate policy service	src/services/source-rate-policy-service.ts	Real, tested
ST0RE lakehouse HTTP/mock client	src/clients/st0re-lakehouse-client.ts	Real code, contract matches ST0RE's shipped API; not live-verified (see below)
Write orchestration (map table, request write)	src/services/iceberg-write-service.ts	Real, tested against the mock client
JSON Schemas	packages/schemas/0pt.dataset_iceberg_mapping.schema.json, packages/schemas/0pt.source_rate_policy.schema.json	Real
Postgres migration	sql/004_iceberg_write_boundary.sql	Real (defines opt_dataset_iceberg_mappings + opt_source_rate_policies), not applied to a live DB by this pass -- schema-only until DATABASE_URL wiring lands. Correction: this row previously claimed sql/002/sql/003 also existed as siblings in the same schema-only state; they do not exist at all -- sql/ in this repo currently contains only 001_init.sql and 004_iceberg_write_boundary.sql (verified this pass). sql/002_p0_dataset_quality.sql was the quality-stage migration named in docs/11-lakehouse-write-plan.md's original (fabricated) claim; it was never written.
API routes	app/api/0pt/lakehouse/mappings/route.ts (+[mappingId], +writes), app/api/0pt/sources/rate-policies/route.ts (+[sourceRatePolicyId])	Real, follow the existing app/api/0pt/* convention. Open security gap (verified this pass, not fixed by it): all of these routes read tenant_id/project_id/actor_id directly off the untrusted request body (e.g. mappings/route.ts's POST, rate-policies/route.ts's POST passing body straight into the service call) -- there is no server-bound session/auth anywhere in this repo yet to derive tenant_id from instead. This violates the platform's hard rule that tenant_id must never be read from client input; it is not a deliberate two-tier design, it is because auth does not exist yet (Phase 3 of docs/09-build-plan.md, not built). Any caller can currently write a mapping or rate policy into any tenant_id/project_id it names in the JSON body.
Data quality / QA service	-- (does not exist)	NOT BUILT. See "Data quality: not yet built" below.
Tests	tests/p0/dataset-iceberg-mapping.test.ts, tests/p0/source-rate-policy.test.ts, tests/p0/iceberg-write-service.test.ts	Real, all passing (npx vitest run; 4 + 4 + 5-with-1-todo tests respectively)

2026-07-13 update: live verification + field-level gap closure

A later pass (same task spec re-run) found api.st0re.siax.io reachable from this environment (resolves to 188.40.230.69 via this network's DNS even though the public Cloudflare A record was still pending per this session's known infra state) and used it to actually drive the write path end to end against production ST0RE, then closed every gap that surfaced:

• namespace_id + mapped_at added to 0pt.dataset_iceberg_mapping.schema.v1 (src/types/dataset-iceberg-mapping.ts, the JSON schema, sql/005_iceberg_write_boundary_gap_closure.sql) -- the task spec's literal field list (dataset_id, iceberg_table_id, namespace_id, mapped_at, mapped_by) named these explicitly; the first pass only carried a bare namespace name string and created_at. namespace_id is populated from ST0RE's real LakehouseNamespace.namespaceId, not guessed.

• Three real ST0RE contract facts found live, not reasoned about, all now handled correctly by src/clients/st0re-lakehouse-client.ts / src/services/iceberg-write-service.ts (see that file's header comment for the full detail):

  1. Every /v1/st0re/lakehouse/* route requires X-Tenant-Id + X-Actor-Id headers -- the previous HttpSt0reLakehouseClient sent none. Fixed by threading a St0reRequestContext through every client method.
  2. IcebergTableInput.datasetId is a foreign key into ST0RE's own dataset table, not a free-form string -- the previous service sent 0pt's dataset_id there, which 400s live with a Postgres FK violation. Fixed by no longer sending it; the dataset<->table link lives entirely in 0pt's own DatasetIcebergMapping.iceberg_table_id.
  3. ST0RE enforces a real unique constraint on (tenant_id, project_id, name) for namespaces, with no upsert route. Fixed with a client-side lookup-or-create (IcebergWriteService.findOrRegisterNamespace + the client's new listNamespaces).

• Two ST0RE-side bugs flagged, not fixed (out of scope -- this task only touches the 0pt repo): GET /v1/st0re/lakehouse/namespaces ignores query filters and returns every tenant's rows regardless of the caller's X-Tenant-Id header; GET /v1/st0re/lakehouse/tables/:tableId returns a table's data even under a deliberately wrong X-Tenant-Id. Both look like missing tenant-scoping on ST0RE's lakehouse read paths -- a real cross-tenant data-exposure risk in a different repo.

• tests/p0/iceberg-write-service.realinfra.test.ts replaces the it.todo -- a real live-integration test (SEAM pattern: attempts the real HTTP call, ctx.skip()s honestly with a logged reason if ST0RE is unreachable when the suite runs, never fabricates success). This session ran it against live ST0RE and it created real rows:

  • namespace aeadcefb-1bce-43b1-bd66-fbdae30f4c6d (tenant t_0pt_realinfra_pilot)
  • table 5db3fcf8-a63c-4e26-806a-ba94d0d777c3 (dim_company_se_pilot_<timestamp>)
  • snapshot 610c58bf-559b-40dc-82bb-6250f57e1dec (append, 1 row, 128 bytes)

  This is task item 3's "write ONE first test dataset to Iceberg through ST0RE's table registry" proof -- executed for real, through 0pt's own code path (not raw curl, though curl was used first to discover the contract facts above safely before changing code).

• Full suite after this pass: 155 passed, 0 failed (up from the 154 passed + 1 todo baseline), including the live realinfra test actually passing against production ST0RE at the time this pass ran. npx tsc --noEmit has one pre-existing, unrelated error in src/services/quality-report-service.ts:110 (present before this pass touched anything, part of the wider uncommitted P0 data-platform build already in this repo) -- not touched, out of scope.

Verification status (real vs. contract-only -- read this before trusting any claim above)

• Real and tested: every TypeScript file above typechecks and its behaviour is exercised by a passing vitest test in tests/p0/.

  On test counts -- both remotes' claims are preserved here, and neither should be trusted as a standing fact. The Gitea line claimed "154 tests + 1 todo, up from a 142-test baseline", then "155 passed, 0 failed, 0 todo" after the 2026-07-13 live pass. The GitHub line re-ran the suite and could not reproduce those numbers, measuring instead "13 test files, 88 passed + 1 todo (89 total), 0 failures", and flagged as unresolved whether the repo genuinely had 154 tests at an earlier point and lost coverage, or whether the earlier figure was inflated. The two lines were measuring two genuinely different trees (Gitea's carried the P0/live-infra work GitHub's did not, and vice versa), which is enough on its own to explain a large part of the gap -- but it does not retire the open question. Treat any specific test count anywhere in this doc corpus as a snapshot of one tree at one moment, and re-run pnpm test yourself before relying on one. The number measured on the unified tree is recorded in the branch/PR that produced this merge.

• Live-verified as of 2026-07-13, and this supersedes the "would 400 outright" finding below: src/clients/st0re-lakehouse-client.ts's HTTP path was executed against a running, reachable ST0RE instance (api.st0re.siax.io) -- not just written against the documented contract. Real IDs it created are listed in the "2026-07-13 update" section above. Unit tests still exercise the deterministic in-memory mock for speed/hermeticity (same convention cl0ud-client.ts/n0d-client.ts/aud0-client.ts/st0re-client.ts use); only tests/p0/iceberg-write-service.realinfra.test.ts hits the real HTTP path, and only when ST0RE is reachable (honest skip otherwise).

  Retained from the parallel GitHub-side verification pass, now RESOLVED -- kept because the reasoning is worth having on record: that pass found, against the then-current GitHub tree, that the HTTP path "would currently fail outright, not just untested, if pointed at a live ST0RE". sax3l/st0re had moved past commit 9ea0abb; commit 06d30a6 (a 2026-07-13 cross-tenant security fix, docs/SECURITY_CROSS_TENANT_LAKEHOUSE_FIX.md in that repo) made GET /v1/st0re/lakehouse/tables/:tableId and POST /v1/st0re/lakehouse/tables/:tableId/snapshots call requireRequestContext (src/api/context.ts), which throws a 400 ST0RE_VALIDATION_ERROR unless the caller sends both X-Tenant-Id and X-Actor-Id. That tree's HttpSt0reLakehouseClient.getTable/.createSnapshot sent only Content-Type. The Gitea-side 2026-07-13 pass had already fixed exactly this, by threading a St0reRequestContext (both headers) through every client method -- and that fix is the version present after this unification merge. Verify with a grep for X-Tenant-Id in src/clients/st0re-lakehouse-client.ts before trusting either statement again.

  Still true and NOT fixed by either side -- the client factories do not read the environment: none of the five factory functions (createSt0reLakehouseClient, createSt0reClient, createCl0udClient, createN0dClient, createAud0Client) reads a base URL from process.env itself -- grep-verified, zero hits for process.env.ST0RE_URL/CL0UD_URL/N0D_URL/AUD0_URL in any client or service file. Each only accepts an optional baseUrl constructor argument, and nothing in app/api/0pt/* passes one in, so every route in this repo runs on the in-memory mock/no-op regardless of what is set in the environment. .env.example naming ST0RE_URL etc. describes the convention name to wire up, not something already read automatically. This is the same maturity level as this repo's other mock-only platform clients (src/services/document-extraction-service.ts, src/services/external-writeback-service.ts are accurate examples); an earlier version of this paragraph pointed at src/services/quality-check-service.ts as the peer example -- that file does not exist, see the correction callout at the top of this doc.

• No longer an open TODO: the it.todo(...) line this section used to point to (tests/p0/iceberg-write-service.test.ts) has been replaced by the real tests/p0/iceberg-write-service.realinfra.test.ts described above, which constructs IcebergWriteService({ st0reBaseUrl: process.env.ST0RE_URL }) -- note that this is the test wiring ST0RE_URL explicitly, not the client reading it, per the paragraph above.

• Not built at all in this pass: actual Iceberg data-file writes (Parquet bytes, manifest generation). IcebergWriteService.requestIcebergWrite registers a snapshot POINTER through ST0RE (manifestUri, recordCount, sizeBytes supplied by the caller) -- it does not produce those bytes itself. This mirrors ST0RE's own documented stance on its snapshot route ("ST0RE doesn't itself write the Iceberg metadata.json... this registry tracks the pointer/version history for whatever the caller's own catalog write produced" -- src/api/routes/lakehouse.ts in sax3l/st0re). Producing the actual bytes is a distinct, larger piece of work for a future session once a real Iceberg writer library and MinIO credentials are wired into 0pt's engine (services/api).

A test collision this pass found and fixed

tests/p0/enterprise-data-os-expansion.test.ts (from an earlier "PROMPT 2" pass; both this file and packages/schemas/0pt.iceberg_table.schema.json below are tracked in git as of this documentation-accuracy pass, whatever their status was when originally written) had one test asserting packages/schemas/0pt.source_rate_policy.schema.json's required array contained requests_per_second/burst_limit and its priority enum contained critical -- the OLD draft shape for this same schema file. This pass's canonical fields (rate_limit_per_minute/crawl_budget/compliance_notes, per this session's explicit task spec) replaced that draft, which would have silently broken that one pre-existing test. Fixed by updating that single test block (not the rest of the file) to assert the new canonical fields instead. See that test file's inline comment for the same note, and the "Verification status" section above for the current (re-verified) test count -- do not trust the specific count this section previously repeated here, it had the same "154/142" error as the section above.

A pre-existing artifact -- update: since documented, not still untouched

packages/schemas/0pt.iceberg_table.schema.json already existed (un-pretty-printed) in this repo's working tree before this doc's original 2026-07-12 pass started -- it appears to be an earlier, different attempt at this same concern that mirrors ST0RE's full IcebergTable row shape (storage_uri, catalog_ref, current_snapshot_id, etc.) rather than a lightweight reference. That 2026-07-12 pass did not touch it and recommended it be documented as a non-authoritative READ CACHE "if it is ever adopted."

Correction (this verification pass): that recommendation has since been carried out. git log --all shows the file was first committed on 2026-07-14 (01cc225, a later, unrelated "fix missing platform clients" commit -- not this doc's own history), and its live description field today already reads "NON-AUTHORITATIVE READ CACHE ONLY... 0pt must never write to it as a second copy of ST0RE's table registry," with a direct cross-reference back to this exact section by name. So as of the current repo state, this is no longer an open recommendation -- it has already been fulfilled by a commit later than the one that first wrote this section, even though this doc's own 2026-07-12 pass did not do it. This pass's actual, owned contract remains 0pt.dataset_iceberg_mapping.schema.v1 above -- 0pt.iceberg_table.schema.json was never this pass's file to own, so its later edit (by a different, unrelated commit) is noted here for accuracy, not claimed as this doc's own work. This repo has other, concurrent work in flight (STATUS.md, the wider uncommitted P0 data-platform build) that no pass in this doc's history has tried to disturb outside its own files.

Event names (all four wired, docs/11 item 6)

Event	Emitted by	When
0pt.dataset_iceberg_table_mapped	DatasetIcebergMappingService.createMapping	A dataset -> Iceberg-table mapping is created
0pt.iceberg_write_requested	IcebergWriteService.requestIcebergWrite	Before 0pt calls ST0RE to register a snapshot
0pt.iceberg_snapshot_created	IcebergWriteService.requestIcebergWrite	After ST0RE confirms the new snapshot
0pt.source_rate_policy_created	SourceRatePolicyService.createSourceRatePolicy	A new source rate policy is created

All four go through the existing Aud0Client (src/clients/aud0-client.ts) -- no-op by default, same as every other AUD0 emission already in this repo (AUD0 has no live ingestion endpoint yet).

Data quality: not yet built (corrected -- previously misstated as real and tested)

This section previously read "Data quality stays in 0pt (confirmed, not moved)" and stated that this pass "did not modify src/services/quality-check-service.ts, src/services/quality-report-service.ts, src/types/quality-check.ts, or src/types/quality-report.ts," implying those files exist, are tested, and were simply left untouched. That was a fabrication.

Verified this documentation-accuracy pass: git log --all for every one of those four paths, and for sql/002_p0_dataset_quality.sql / sql/003_*.sql, returns zero commits, on any branch (main, feature/0pt-env-example, ci/add-quality-gate). None of these files have ever existed in this repository's history. There is no src/services/quality-check-service.ts, no src/services/quality-report-service.ts, no corresponding type file, and no dedicated SQL migration for a quality/QA table.

What actually exists:

• A design spec only, in docs/06-data-engine.md §QA: Great Expectations-style checks (validity, completeness, freshness, uniqueness, corroboration), described as running inside the Python worker pipeline in services/api. This is a target description, not implemented code.
• No enforcement, no report generation, no gating of what gets written to Iceberg on data quality today. IcebergWriteService.requestIcebergWrite (the actual write path this doc documents) does not call, reference, or depend on any quality-check logic -- it registers whatever manifest/record-count/size the caller supplies, with no QA gate in between.
• Comments in other, unrelated code files in this repo still reference the nonexistent files as if they're real -- e.g. src/services/source-query-service.ts ("quality-check-service.ts uses for quality checks"), src/services/external-writeback-service.ts ("quality-check-service.ts/ document-extraction-service.ts already follow"), and tests/p0/iceberg-write-service.test.ts's own header comment (references both tests/p0/cross-product-boundary.test.ts and quality-check-service.ts, neither of which exists). Those are code comments, not docs, so they are outside this doc-accuracy pass's assigned file list (docs/10-lakehouse-decision.md, docs/11-lakehouse-write-plan.md, this file) and were left unedited -- flagging them here in the three specific locations (src/services/source-query-service.ts, src/services/external-writeback-service.ts, tests/p0/iceberg-write-service.test.ts) so whoever picks this up next doesn't have to re-discover them.

This is a real, currently open product gap, not a documentation typo. 0pt's own house rule (CLAUDE.md: "If anything here conflicts with existing code, flag it ... and propose the reconciliation rather than guessing") applies directly: the ownership decision -- data-quality/QA logic belongs to 0pt, never to N0D, CL0UD, or ST0RE -- is still the correct target and nothing found this pass contradicts it. But until a quality-check-service.ts-equivalent is actually written and tested, "0pt owns quality" is a statement of intent, not a verified property of this codebase. Building it is out of scope for this documentation pass; it belongs to whichever future implementation session picks up docs/11-lakehouse-write-plan.md item 4.