Add 0pt Iceberg write-path ownership: dataset<->table mapping + source rate policy
Real, tested implementation of docs/11-lakehouse-write-plan.md's items 2-8 (the "future implementation session" that doc named): 0pt is now the canonical writer of Iceberg tables through ST0RE's real, shipped /v1/st0re/lakehouse/* table registry (sax3l/st0re commit 9ea0abb), never persisting lakehouse table/catalog/snapshot state itself. - src/types + src/services for 0pt.dataset_iceberg_mapping.schema.v1 (dataset -> ST0RE tableId reference) and 0pt.source_rate_policy.schema.v1 (the sourceRatePolicy ownership repair, moved out of N0D per ADR-0030). - src/clients/st0re-lakehouse-client.ts: new client matching ST0RE's actual camelCase route contract, with the same mock/HTTP boundary convention as this repo's existing cl0ud/n0d/aud0/st0re clients. Honest caveat in its module doc comment: not verified against a live ST0RE instance in this pass (no reachable ST0RE_URL); every test exercises the deterministic mock. - src/services/iceberg-write-service.ts: registers tables/snapshots through ST0RE, requests N0D jobs only for heavy writes (never executes them in-process), emits all four AUD0 event names (dataset_iceberg_table_mapped / iceberg_write_requested / iceberg_snapshot_created / source_rate_policy_created). - sql/004, packages/schemas/*.json, app/api/0pt/lakehouse + sources/rate-policies routes, tests/p0/*.test.ts (154 passed + 1 honest it.todo for the live-ST0RE case, up from 142 baseline, 0 failures). - Reconciled a real collision: an earlier untracked draft schema/test pair (tests/p0/enterprise-data-os-expansion.test.ts) asserted a different, now-superseded source_rate_policy shape; updated that one test block only. - docs/ICEBERG_WRITE_BOUNDARY.md states the ownership boundary plainly and is explicit throughout about what's real-and-tested vs. contract-only. docs/11-lakehouse-write-plan.md gets a status-update section, not a rewrite. - Data quality rules (quality-check-service.ts, quality-report-service.ts) were not touched -- confirmed still 0pt-owned. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -5,6 +5,49 @@ implementation session — it names the schemas/events precisely so that session
|
||||
guess, but it does not build the full pipeline today. Some items are already true today (quality
|
||||
rules) and this doc just confirms 0pt intends to keep them that way.
|
||||
|
||||
## 2026-07-12 update: what the 0pt write-boundary implementation pass closed
|
||||
|
||||
This pass (the "future implementation session" this doc was written for) turned most items below
|
||||
into real, tested code. Full detail in `docs/ICEBERG_WRITE_BOUNDARY.md`; short version:
|
||||
|
||||
- Item 2 (`0pt.dataset_iceberg_mapping`): CLOSED as real code -- `src/types/dataset-iceberg-mapping.ts`,
|
||||
`src/services/dataset-iceberg-mapping-service.ts`, `packages/schemas/0pt.dataset_iceberg_mapping.schema.json`,
|
||||
`sql/004_iceberg_write_boundary.sql` (`opt_dataset_iceberg_mappings`), tested in
|
||||
`tests/p0/dataset-iceberg-mapping.test.ts`. Field names differ slightly from the "suggested shape"
|
||||
above (`dataset_iceberg_mapping_id` as PK, `iceberg_table_id` instead of `iceberg_table_ref`,
|
||||
`mapping_status` instead of bare `entity_type`/`market`) -- reconciled against ST0RE's REAL
|
||||
shipped `IcebergTable.tableId` shape (sax3l/st0re, not the "future" registry this doc assumed) and
|
||||
against this session's own task spec; see the boundary doc for the full reconciliation note.
|
||||
- Item 3 (first test dataset write): CLOSED against ST0RE's now-real (but not live-verified from
|
||||
this environment) `/v1/st0re/lakehouse/*` API -- `src/clients/st0re-lakehouse-client.ts`,
|
||||
`src/services/iceberg-write-service.ts`, proven in `tests/p0/iceberg-write-service.test.ts`
|
||||
against that client's mock implementation (same mock/HTTP convention as `cl0ud-client.ts` etc).
|
||||
NOT verified against a running ST0RE deployment in this pass -- see the boundary doc's
|
||||
"Verification status" section and the `it.todo` in that test file.
|
||||
- Item 4 (quality rules stay in 0pt): CONFIRMED, untouched -- `src/services/quality-check-service.ts`
|
||||
and `src/services/quality-report-service.ts` were not modified by this pass.
|
||||
- Item 5 (`0pt.source_rate_policy`): CLOSED as real code -- `src/types/source-rate-policy.ts`,
|
||||
`src/services/source-rate-policy-service.ts`, `packages/schemas/0pt.source_rate_policy.schema.json`,
|
||||
`sql/004_iceberg_write_boundary.sql` (`opt_source_rate_policies`), tested in
|
||||
`tests/p0/source-rate-policy.test.ts`. Kept `requests_per_second`-style fields (as `domain`/
|
||||
`robots_ok`/`access`) as OPTIONAL alongside the task-spec's primary `rate_limit_per_minute`/
|
||||
`crawl_budget`/`compliance_notes` fields -- both specs for this same contract are satisfied on one
|
||||
row, not two competing shapes.
|
||||
- Item 6 (four AUD0 events): CLOSED for all four names -- `0pt.dataset_iceberg_table_mapped` and
|
||||
`0pt.iceberg_write_requested`/`0pt.iceberg_snapshot_created` emit from
|
||||
`src/services/iceberg-write-service.ts` and `dataset-iceberg-mapping-service.ts`;
|
||||
`0pt.source_rate_policy_created` emits from `src/services/source-rate-policy-service.ts`. All via
|
||||
the existing `Aud0Client` (mock/no-op by default, same as every other AUD0 emission in this repo --
|
||||
AUD0 itself still has no live ingestion endpoint).
|
||||
- Item 7 (N0D heavy-write requests): CLOSED as real code -- `IcebergWriteService.requestIcebergWrite`
|
||||
requests an N0D job (`0pt.iceberg_write_job`) only when `heavy: true`, never executes in-process;
|
||||
tested in `tests/p0/iceberg-write-service.test.ts`.
|
||||
- Item 8 (no lakehouse state outside ST0RE): CLOSED as a design property, not just a statement --
|
||||
`IcebergWriteService` and `DatasetIcebergMappingService` hold zero table/snapshot fields; the only
|
||||
thing 0pt persists is the mapping's `iceberg_table_id` reference. See the boundary doc.
|
||||
- Item 1 (Iceberg canonical, Delta rejected): CONFIRMED, unchanged from `docs/10` --
|
||||
`IcebergWriteService.mapDatasetToIcebergTable` defaults `table_format` to `"iceberg"`.
|
||||
|
||||
## To-do list
|
||||
|
||||
1. **Use Apache Iceberg as the canonical write format** for datasets 0pt produces, per
|
||||
|
||||
@@ -0,0 +1,125 @@
|
||||
# Iceberg write-path ownership boundary (0pt)
|
||||
|
||||
Read `docs/10-lakehouse-decision.md` and `docs/11-lakehouse-write-plan.md` first -- this doc is
|
||||
the implementation record for the plan those wrote. It states the ownership boundary plainly and
|
||||
says exactly what in this pass is real code vs. contract-only, matching the honesty convention
|
||||
already used across this SIAX effort (e.g. sax3l/act0's own client docs, sax3l/st0re's
|
||||
PRODUCTION_READINESS.md).
|
||||
|
||||
## The boundary, plainly
|
||||
|
||||
- **ST0RE owns table/catalog/artifact metadata for the lakehouse.** The Iceberg catalog, table
|
||||
registry, namespace registry, and snapshot registry live in ST0RE (sax3l/st0re,
|
||||
`src/api/routes/lakehouse.ts`, `/v1/st0re/lakehouse/*`). 0pt does not run a catalog, does not
|
||||
keep a second copy of table/snapshot rows, and does not talk to MinIO or an Iceberg REST catalog
|
||||
directly.
|
||||
- **0pt owns dataset production, quality, data contracts, and writes.** 0pt runs sourcing ->
|
||||
extraction -> structuring -> enrichment -> QA (`docs/06-data-engine.md`) as it always has, and is
|
||||
the canonical writer of the Iceberg tables those datasets produce -- but it writes THROUGH
|
||||
ST0RE's table registry API, never around it.
|
||||
- **0pt owns source-rate/crawl-compliance policy.** This is the genuine ownership repair this pass
|
||||
makes concrete: `0pt.source_rate_policy` (per-source rate limits, crawl budgets, compliance
|
||||
notes) is 0pt's own contract. It does not exist in, and must not be reintroduced into, N0D's
|
||||
`ComputePolicy.sourceRatePolicy` (deprecated there per the ADR-0030 audit), CL0UD, or any other
|
||||
product's policy defaults.
|
||||
- **N0D executes.** 0pt may request an N0D job for a heavy write or compaction
|
||||
(`IcebergWriteService.requestIcebergWrite({ heavy: true })`), but N0D never owns the schema, the
|
||||
catalog state, or the decision of what to write -- and 0pt never persists the result of that job
|
||||
anywhere except by registering the resulting snapshot through ST0RE.
|
||||
- **0pt never persists lakehouse table/catalog/snapshot state outside ST0RE.** Not in Postgres
|
||||
(`opt_*` tables), not in memory beyond a single dataset -> table-id pointer
|
||||
(`opt_dataset_iceberg_mappings.iceberg_table_id`, a REFERENCE, never a copy of ST0RE's row).
|
||||
Standing up a parallel catalog in 0pt would repeat the exact mistake this whole cross-repo effort
|
||||
is repairing (N0D's own bootstrap catalog, built because nothing else had one yet -- see
|
||||
sax3l/st0re `docs/LAKEHOUSE_OWNERSHIP_PLAN.md`).
|
||||
|
||||
## What this pass built
|
||||
|
||||
| Concern | File(s) | Status |
|
||||
|---|---|---|
|
||||
| `0pt.dataset_iceberg_mapping.schema.v1` type | `src/types/dataset-iceberg-mapping.ts` | Real |
|
||||
| `0pt.source_rate_policy.schema.v1` type | `src/types/source-rate-policy.ts` | Real |
|
||||
| Dataset<->Iceberg-table mapping service | `src/services/dataset-iceberg-mapping-service.ts` | Real, tested |
|
||||
| Source rate policy service | `src/services/source-rate-policy-service.ts` | Real, tested |
|
||||
| ST0RE lakehouse HTTP/mock client | `src/clients/st0re-lakehouse-client.ts` | Real code, contract matches ST0RE's shipped API; **not live-verified** (see below) |
|
||||
| Write orchestration (map table, request write) | `src/services/iceberg-write-service.ts` | Real, tested against the mock client |
|
||||
| JSON Schemas | `packages/schemas/0pt.dataset_iceberg_mapping.schema.json`, `packages/schemas/0pt.source_rate_policy.schema.json` | Real |
|
||||
| Postgres migration | `sql/004_iceberg_write_boundary.sql` | Real (not applied to a live DB by this pass -- same as `sql/002`/`sql/003`, which this repo's own P0 docs already note are schema-only until `DATABASE_URL` wiring lands) |
|
||||
| API routes | `app/api/0pt/lakehouse/mappings/route.ts` (+`[mappingId]`, `+writes`), `app/api/0pt/sources/rate-policies/route.ts` (+`[sourceRatePolicyId]`) | Real, follow the existing `app/api/0pt/*` convention |
|
||||
| Tests | `tests/p0/dataset-iceberg-mapping.test.ts`, `tests/p0/source-rate-policy.test.ts`, `tests/p0/iceberg-write-service.test.ts` | Real, all passing (`pnpm test`); one explicit `it.todo` for the live-ST0RE case |
|
||||
|
||||
## Verification status (real vs. contract-only -- read this before trusting any claim above)
|
||||
|
||||
- **Real and tested**: every TypeScript file above typechecks and its behavior is exercised by a
|
||||
passing vitest test in `tests/p0/`. The full suite (`pnpm test`) passed at 154 tests + 1 todo
|
||||
after this pass, up from a 142-test baseline before it, with 0 failures.
|
||||
- **Real code, not live-verified**: `src/clients/st0re-lakehouse-client.ts`'s HTTP path is written
|
||||
against ST0RE's actual, committed, shipped route contract (sax3l/st0re commit `9ea0abb`,
|
||||
`src/api/routes/lakehouse.ts` + `src/types.ts` -- read directly from that repo's source during
|
||||
this pass, not guessed). ST0RE's own `STATUS.md` documents it as deployed via Coolify on port
|
||||
8090. This pass did not have a confirmed, reachable `ST0RE_URL` to call, so the HTTP path has
|
||||
**not been executed against a running ST0RE instance**. Every test in this pass exercises the
|
||||
client's deterministic in-memory mock instead -- the exact same convention `cl0ud-client.ts`,
|
||||
`n0d-client.ts`, `aud0-client.ts`, and `st0re-client.ts` already use in this repo for the same
|
||||
reason (no live CL0UD/N0D/AUD0/ST0RE endpoint is reachable from this repo's test environment
|
||||
either). This is not a downgrade from the rest of P0 -- it is the same maturity level as
|
||||
everything else in `src/services/quality-check-service.ts`.
|
||||
- **Contract-only / explicit TODO**: a live-ST0RE integration test is named but not built --
|
||||
`tests/p0/iceberg-write-service.test.ts`'s `it.todo(...)` line. A future session with a reachable
|
||||
`ST0RE_URL` should fill it in by constructing `IcebergWriteService({ st0reBaseUrl:
|
||||
process.env.ST0RE_URL })` and asserting the same flow against the real HTTP API.
|
||||
- **Not built at all in this pass**: actual Iceberg data-file writes (Parquet bytes, manifest
|
||||
generation). `IcebergWriteService.requestIcebergWrite` registers a snapshot POINTER through
|
||||
ST0RE (`manifestUri`, `recordCount`, `sizeBytes` supplied by the caller) -- it does not produce
|
||||
those bytes itself. This mirrors ST0RE's own documented stance on its snapshot route ("ST0RE
|
||||
doesn't itself write the Iceberg metadata.json... this registry tracks the pointer/version
|
||||
history for whatever the caller's own catalog write produced" -- `src/api/routes/lakehouse.ts` in
|
||||
sax3l/st0re). Producing the actual bytes is a distinct, larger piece of work for a future session
|
||||
once a real Iceberg writer library and MinIO credentials are wired into 0pt's engine
|
||||
(`services/api`).
|
||||
|
||||
## A test collision this pass found and fixed
|
||||
|
||||
`tests/p0/enterprise-data-os-expansion.test.ts` (untracked, pre-existing, from an earlier "PROMPT
|
||||
2" pass) had one test asserting `packages/schemas/0pt.source_rate_policy.schema.json`'s `required`
|
||||
array contained `requests_per_second`/`burst_limit` and its `priority` enum contained `critical` --
|
||||
the OLD draft shape for this same schema file. This pass's canonical fields
|
||||
(`rate_limit_per_minute`/`crawl_budget`/`compliance_notes`, per this session's explicit task spec)
|
||||
replaced that draft, which would have silently broken that one pre-existing test. Fixed by updating
|
||||
that single test block (not the rest of the file) to assert the new canonical fields instead --
|
||||
`pnpm test` is 154 passed + 1 todo, 0 failures, after this pass (up from a 142-test baseline before
|
||||
it). See that test file's inline comment for the same note.
|
||||
|
||||
## A pre-existing artifact this pass did not touch
|
||||
|
||||
`packages/schemas/0pt.iceberg_table.schema.json` already existed (untracked, un-pretty-printed) in
|
||||
this repo's working tree before this pass started -- it appears to be an earlier, different attempt
|
||||
at this same concern that mirrors ST0RE's full `IcebergTable` row shape (`storage_uri`,
|
||||
`catalog_ref`, `current_snapshot_id`, etc.) rather than a lightweight reference. If it is ever
|
||||
adopted, it must be documented as a non-authoritative READ CACHE only -- 0pt must never treat it as
|
||||
the source of truth or write to it independently of ST0RE. This pass's actual, owned contract is
|
||||
`0pt.dataset_iceberg_mapping.schema.v1` above; it was left in place, unmodified, rather than
|
||||
deleted or silently rewritten, since it wasn't part of this task's scope and this repo has other,
|
||||
concurrent work in flight (`STATUS.md`, the wider uncommitted P0 data-platform build) that this
|
||||
pass did not want to disturb outside its own files.
|
||||
|
||||
## Event names (all four wired, docs/11 item 6)
|
||||
|
||||
| Event | Emitted by | When |
|
||||
|---|---|---|
|
||||
| `0pt.dataset_iceberg_table_mapped` | `DatasetIcebergMappingService.createMapping` | A dataset -> Iceberg-table mapping is created |
|
||||
| `0pt.iceberg_write_requested` | `IcebergWriteService.requestIcebergWrite` | Before 0pt calls ST0RE to register a snapshot |
|
||||
| `0pt.iceberg_snapshot_created` | `IcebergWriteService.requestIcebergWrite` | After ST0RE confirms the new snapshot |
|
||||
| `0pt.source_rate_policy_created` | `SourceRatePolicyService.createSourceRatePolicy` | A new source rate policy is created |
|
||||
|
||||
All four go through the existing `Aud0Client` (`src/clients/aud0-client.ts`) -- no-op by default,
|
||||
same as every other AUD0 emission already in this repo (AUD0 has no live ingestion endpoint yet).
|
||||
|
||||
## Data quality stays in 0pt (confirmed, not moved)
|
||||
|
||||
This pass did not modify `src/services/quality-check-service.ts`, `src/services/quality-report-service.ts`,
|
||||
`src/types/quality-check.ts`, or `src/types/quality-report.ts`. 0pt's QA stage
|
||||
(`docs/06-data-engine.md`) remains entirely 0pt-owned; nothing in this pass routes quality logic
|
||||
through N0D, CL0UD, or ST0RE. N0D may still be asked to EXECUTE a quality-check job
|
||||
(`quality-check-service.ts`'s existing `n0d.createJob` call, unrelated to this pass), but the rules
|
||||
being executed are always 0pt's own.
|
||||
Reference in New Issue
Block a user