Add 0pt Iceberg write-path ownership: dataset<->table mapping + source rate policy

Real, tested implementation of docs/11-lakehouse-write-plan.md's items 2-8
(the "future implementation session" that doc named): 0pt is now the
canonical writer of Iceberg tables through ST0RE's real, shipped
/v1/st0re/lakehouse/* table registry (sax3l/st0re commit 9ea0abb), never
persisting lakehouse table/catalog/snapshot state itself.

- src/types + src/services for 0pt.dataset_iceberg_mapping.schema.v1
  (dataset -> ST0RE tableId reference) and 0pt.source_rate_policy.schema.v1
  (the sourceRatePolicy ownership repair, moved out of N0D per ADR-0030).
- src/clients/st0re-lakehouse-client.ts: new client matching ST0RE's actual
  camelCase route contract, with the same mock/HTTP boundary convention as
  this repo's existing cl0ud/n0d/aud0/st0re clients. Honest caveat in its
  module doc comment: not verified against a live ST0RE instance in this
  pass (no reachable ST0RE_URL); every test exercises the deterministic mock.
- src/services/iceberg-write-service.ts: registers tables/snapshots through
  ST0RE, requests N0D jobs only for heavy writes (never executes them
  in-process), emits all four AUD0 event names
  (dataset_iceberg_table_mapped / iceberg_write_requested /
  iceberg_snapshot_created / source_rate_policy_created).
- sql/004, packages/schemas/*.json, app/api/0pt/lakehouse + sources/rate-policies
  routes, tests/p0/*.test.ts (154 passed + 1 honest it.todo for the live-ST0RE
  case, up from 142 baseline, 0 failures).
- Reconciled a real collision: an earlier untracked draft schema/test pair
  (tests/p0/enterprise-data-os-expansion.test.ts) asserted a different,
  now-superseded source_rate_policy shape; updated that one test block only.
- docs/ICEBERG_WRITE_BOUNDARY.md states the ownership boundary plainly and
  is explicit throughout about what's real-and-tested vs. contract-only.
  docs/11-lakehouse-write-plan.md gets a status-update section, not a rewrite.
- Data quality rules (quality-check-service.ts, quality-report-service.ts)
  were not touched -- confirmed still 0pt-owned.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-12 19:46:47 +02:00
co-authored by Claude Sonnet 5
parent 2545218265
commit 35b6ec63bf
20 changed files with 1695 additions and 0 deletions
+43
View File
@@ -5,6 +5,49 @@ implementation session — it names the schemas/events precisely so that session
guess, but it does not build the full pipeline today. Some items are already true today (quality
rules) and this doc just confirms 0pt intends to keep them that way.
## 2026-07-12 update: what the 0pt write-boundary implementation pass closed
This pass (the "future implementation session" this doc was written for) turned most items below
into real, tested code. Full detail in `docs/ICEBERG_WRITE_BOUNDARY.md`; short version:
- Item 2 (`0pt.dataset_iceberg_mapping`): CLOSED as real code -- `src/types/dataset-iceberg-mapping.ts`,
`src/services/dataset-iceberg-mapping-service.ts`, `packages/schemas/0pt.dataset_iceberg_mapping.schema.json`,
`sql/004_iceberg_write_boundary.sql` (`opt_dataset_iceberg_mappings`), tested in
`tests/p0/dataset-iceberg-mapping.test.ts`. Field names differ slightly from the "suggested shape"
above (`dataset_iceberg_mapping_id` as PK, `iceberg_table_id` instead of `iceberg_table_ref`,
`mapping_status` instead of bare `entity_type`/`market`) -- reconciled against ST0RE's REAL
shipped `IcebergTable.tableId` shape (sax3l/st0re, not the "future" registry this doc assumed) and
against this session's own task spec; see the boundary doc for the full reconciliation note.
- Item 3 (first test dataset write): CLOSED against ST0RE's now-real (but not live-verified from
this environment) `/v1/st0re/lakehouse/*` API -- `src/clients/st0re-lakehouse-client.ts`,
`src/services/iceberg-write-service.ts`, proven in `tests/p0/iceberg-write-service.test.ts`
against that client's mock implementation (same mock/HTTP convention as `cl0ud-client.ts` etc).
NOT verified against a running ST0RE deployment in this pass -- see the boundary doc's
"Verification status" section and the `it.todo` in that test file.
- Item 4 (quality rules stay in 0pt): CONFIRMED, untouched -- `src/services/quality-check-service.ts`
and `src/services/quality-report-service.ts` were not modified by this pass.
- Item 5 (`0pt.source_rate_policy`): CLOSED as real code -- `src/types/source-rate-policy.ts`,
`src/services/source-rate-policy-service.ts`, `packages/schemas/0pt.source_rate_policy.schema.json`,
`sql/004_iceberg_write_boundary.sql` (`opt_source_rate_policies`), tested in
`tests/p0/source-rate-policy.test.ts`. Kept `requests_per_second`-style fields (as `domain`/
`robots_ok`/`access`) as OPTIONAL alongside the task-spec's primary `rate_limit_per_minute`/
`crawl_budget`/`compliance_notes` fields -- both specs for this same contract are satisfied on one
row, not two competing shapes.
- Item 6 (four AUD0 events): CLOSED for all four names -- `0pt.dataset_iceberg_table_mapped` and
`0pt.iceberg_write_requested`/`0pt.iceberg_snapshot_created` emit from
`src/services/iceberg-write-service.ts` and `dataset-iceberg-mapping-service.ts`;
`0pt.source_rate_policy_created` emits from `src/services/source-rate-policy-service.ts`. All via
the existing `Aud0Client` (mock/no-op by default, same as every other AUD0 emission in this repo --
AUD0 itself still has no live ingestion endpoint).
- Item 7 (N0D heavy-write requests): CLOSED as real code -- `IcebergWriteService.requestIcebergWrite`
requests an N0D job (`0pt.iceberg_write_job`) only when `heavy: true`, never executes in-process;
tested in `tests/p0/iceberg-write-service.test.ts`.
- Item 8 (no lakehouse state outside ST0RE): CLOSED as a design property, not just a statement --
`IcebergWriteService` and `DatasetIcebergMappingService` hold zero table/snapshot fields; the only
thing 0pt persists is the mapping's `iceberg_table_id` reference. See the boundary doc.
- Item 1 (Iceberg canonical, Delta rejected): CONFIRMED, unchanged from `docs/10` --
`IcebergWriteService.mapDatasetToIcebergTable` defaults `table_format` to `"iceberg"`.
## To-do list
1. **Use Apache Iceberg as the canonical write format** for datasets 0pt produces, per
+125
View File
@@ -0,0 +1,125 @@
# Iceberg write-path ownership boundary (0pt)
Read `docs/10-lakehouse-decision.md` and `docs/11-lakehouse-write-plan.md` first -- this doc is
the implementation record for the plan those wrote. It states the ownership boundary plainly and
says exactly what in this pass is real code vs. contract-only, matching the honesty convention
already used across this SIAX effort (e.g. sax3l/act0's own client docs, sax3l/st0re's
PRODUCTION_READINESS.md).
## The boundary, plainly
- **ST0RE owns table/catalog/artifact metadata for the lakehouse.** The Iceberg catalog, table
registry, namespace registry, and snapshot registry live in ST0RE (sax3l/st0re,
`src/api/routes/lakehouse.ts`, `/v1/st0re/lakehouse/*`). 0pt does not run a catalog, does not
keep a second copy of table/snapshot rows, and does not talk to MinIO or an Iceberg REST catalog
directly.
- **0pt owns dataset production, quality, data contracts, and writes.** 0pt runs sourcing ->
extraction -> structuring -> enrichment -> QA (`docs/06-data-engine.md`) as it always has, and is
the canonical writer of the Iceberg tables those datasets produce -- but it writes THROUGH
ST0RE's table registry API, never around it.
- **0pt owns source-rate/crawl-compliance policy.** This is the genuine ownership repair this pass
makes concrete: `0pt.source_rate_policy` (per-source rate limits, crawl budgets, compliance
notes) is 0pt's own contract. It does not exist in, and must not be reintroduced into, N0D's
`ComputePolicy.sourceRatePolicy` (deprecated there per the ADR-0030 audit), CL0UD, or any other
product's policy defaults.
- **N0D executes.** 0pt may request an N0D job for a heavy write or compaction
(`IcebergWriteService.requestIcebergWrite({ heavy: true })`), but N0D never owns the schema, the
catalog state, or the decision of what to write -- and 0pt never persists the result of that job
anywhere except by registering the resulting snapshot through ST0RE.
- **0pt never persists lakehouse table/catalog/snapshot state outside ST0RE.** Not in Postgres
(`opt_*` tables), not in memory beyond a single dataset -> table-id pointer
(`opt_dataset_iceberg_mappings.iceberg_table_id`, a REFERENCE, never a copy of ST0RE's row).
Standing up a parallel catalog in 0pt would repeat the exact mistake this whole cross-repo effort
is repairing (N0D's own bootstrap catalog, built because nothing else had one yet -- see
sax3l/st0re `docs/LAKEHOUSE_OWNERSHIP_PLAN.md`).
## What this pass built
| Concern | File(s) | Status |
|---|---|---|
| `0pt.dataset_iceberg_mapping.schema.v1` type | `src/types/dataset-iceberg-mapping.ts` | Real |
| `0pt.source_rate_policy.schema.v1` type | `src/types/source-rate-policy.ts` | Real |
| Dataset<->Iceberg-table mapping service | `src/services/dataset-iceberg-mapping-service.ts` | Real, tested |
| Source rate policy service | `src/services/source-rate-policy-service.ts` | Real, tested |
| ST0RE lakehouse HTTP/mock client | `src/clients/st0re-lakehouse-client.ts` | Real code, contract matches ST0RE's shipped API; **not live-verified** (see below) |
| Write orchestration (map table, request write) | `src/services/iceberg-write-service.ts` | Real, tested against the mock client |
| JSON Schemas | `packages/schemas/0pt.dataset_iceberg_mapping.schema.json`, `packages/schemas/0pt.source_rate_policy.schema.json` | Real |
| Postgres migration | `sql/004_iceberg_write_boundary.sql` | Real (not applied to a live DB by this pass -- same as `sql/002`/`sql/003`, which this repo's own P0 docs already note are schema-only until `DATABASE_URL` wiring lands) |
| API routes | `app/api/0pt/lakehouse/mappings/route.ts` (+`[mappingId]`, `+writes`), `app/api/0pt/sources/rate-policies/route.ts` (+`[sourceRatePolicyId]`) | Real, follow the existing `app/api/0pt/*` convention |
| Tests | `tests/p0/dataset-iceberg-mapping.test.ts`, `tests/p0/source-rate-policy.test.ts`, `tests/p0/iceberg-write-service.test.ts` | Real, all passing (`pnpm test`); one explicit `it.todo` for the live-ST0RE case |
## Verification status (real vs. contract-only -- read this before trusting any claim above)
- **Real and tested**: every TypeScript file above typechecks and its behavior is exercised by a
passing vitest test in `tests/p0/`. The full suite (`pnpm test`) passed at 154 tests + 1 todo
after this pass, up from a 142-test baseline before it, with 0 failures.
- **Real code, not live-verified**: `src/clients/st0re-lakehouse-client.ts`'s HTTP path is written
against ST0RE's actual, committed, shipped route contract (sax3l/st0re commit `9ea0abb`,
`src/api/routes/lakehouse.ts` + `src/types.ts` -- read directly from that repo's source during
this pass, not guessed). ST0RE's own `STATUS.md` documents it as deployed via Coolify on port
8090. This pass did not have a confirmed, reachable `ST0RE_URL` to call, so the HTTP path has
**not been executed against a running ST0RE instance**. Every test in this pass exercises the
client's deterministic in-memory mock instead -- the exact same convention `cl0ud-client.ts`,
`n0d-client.ts`, `aud0-client.ts`, and `st0re-client.ts` already use in this repo for the same
reason (no live CL0UD/N0D/AUD0/ST0RE endpoint is reachable from this repo's test environment
either). This is not a downgrade from the rest of P0 -- it is the same maturity level as
everything else in `src/services/quality-check-service.ts`.
- **Contract-only / explicit TODO**: a live-ST0RE integration test is named but not built --
`tests/p0/iceberg-write-service.test.ts`'s `it.todo(...)` line. A future session with a reachable
`ST0RE_URL` should fill it in by constructing `IcebergWriteService({ st0reBaseUrl:
process.env.ST0RE_URL })` and asserting the same flow against the real HTTP API.
- **Not built at all in this pass**: actual Iceberg data-file writes (Parquet bytes, manifest
generation). `IcebergWriteService.requestIcebergWrite` registers a snapshot POINTER through
ST0RE (`manifestUri`, `recordCount`, `sizeBytes` supplied by the caller) -- it does not produce
those bytes itself. This mirrors ST0RE's own documented stance on its snapshot route ("ST0RE
doesn't itself write the Iceberg metadata.json... this registry tracks the pointer/version
history for whatever the caller's own catalog write produced" -- `src/api/routes/lakehouse.ts` in
sax3l/st0re). Producing the actual bytes is a distinct, larger piece of work for a future session
once a real Iceberg writer library and MinIO credentials are wired into 0pt's engine
(`services/api`).
## A test collision this pass found and fixed
`tests/p0/enterprise-data-os-expansion.test.ts` (untracked, pre-existing, from an earlier "PROMPT
2" pass) had one test asserting `packages/schemas/0pt.source_rate_policy.schema.json`'s `required`
array contained `requests_per_second`/`burst_limit` and its `priority` enum contained `critical` --
the OLD draft shape for this same schema file. This pass's canonical fields
(`rate_limit_per_minute`/`crawl_budget`/`compliance_notes`, per this session's explicit task spec)
replaced that draft, which would have silently broken that one pre-existing test. Fixed by updating
that single test block (not the rest of the file) to assert the new canonical fields instead --
`pnpm test` is 154 passed + 1 todo, 0 failures, after this pass (up from a 142-test baseline before
it). See that test file's inline comment for the same note.
## A pre-existing artifact this pass did not touch
`packages/schemas/0pt.iceberg_table.schema.json` already existed (untracked, un-pretty-printed) in
this repo's working tree before this pass started -- it appears to be an earlier, different attempt
at this same concern that mirrors ST0RE's full `IcebergTable` row shape (`storage_uri`,
`catalog_ref`, `current_snapshot_id`, etc.) rather than a lightweight reference. If it is ever
adopted, it must be documented as a non-authoritative READ CACHE only -- 0pt must never treat it as
the source of truth or write to it independently of ST0RE. This pass's actual, owned contract is
`0pt.dataset_iceberg_mapping.schema.v1` above; it was left in place, unmodified, rather than
deleted or silently rewritten, since it wasn't part of this task's scope and this repo has other,
concurrent work in flight (`STATUS.md`, the wider uncommitted P0 data-platform build) that this
pass did not want to disturb outside its own files.
## Event names (all four wired, docs/11 item 6)
| Event | Emitted by | When |
|---|---|---|
| `0pt.dataset_iceberg_table_mapped` | `DatasetIcebergMappingService.createMapping` | A dataset -> Iceberg-table mapping is created |
| `0pt.iceberg_write_requested` | `IcebergWriteService.requestIcebergWrite` | Before 0pt calls ST0RE to register a snapshot |
| `0pt.iceberg_snapshot_created` | `IcebergWriteService.requestIcebergWrite` | After ST0RE confirms the new snapshot |
| `0pt.source_rate_policy_created` | `SourceRatePolicyService.createSourceRatePolicy` | A new source rate policy is created |
All four go through the existing `Aud0Client` (`src/clients/aud0-client.ts`) -- no-op by default,
same as every other AUD0 emission already in this repo (AUD0 has no live ingestion endpoint yet).
## Data quality stays in 0pt (confirmed, not moved)
This pass did not modify `src/services/quality-check-service.ts`, `src/services/quality-report-service.ts`,
`src/types/quality-check.ts`, or `src/types/quality-report.ts`. 0pt's QA stage
(`docs/06-data-engine.md`) remains entirely 0pt-owned; nothing in this pass routes quality logic
through N0D, CL0UD, or ST0RE. N0D may still be asked to EXECUTE a quality-check job
(`quality-check-service.ts`'s existing `n0d.createJob` call, unrelated to this pass), but the rules
being executed are always 0pt's own.