Re-runs this session's task spec against 0pt's actual repo state, which
turned out to already have a substantial, tested implementation from a
prior pass (35b6ec6): dataset<->Iceberg-table mapping, source rate policy,
and an ST0RE lakehouse client that was honestly marked "not live-verified".
This pass found api.st0re.siax.io IS reachable from this environment and
used it -- first via curl to safely discover the real contract, then
through 0pt's own updated code -- to drive a real namespace -> table ->
snapshot write against production ST0RE (task item 3's "write ONE first
test dataset" proof). That surfaced and fixed three real bugs the
mock-only pass could not have caught:
- ST0RE requires X-Tenant-Id/X-Actor-Id headers on every lakehouse route;
the previous HTTP client sent none. Fixed via a threaded
St0reRequestContext.
- IcebergTableInput.datasetId is a foreign key into ST0RE's OWN dataset
table, not a free-form string -- sending 0pt's dataset_id 400s live with
a Postgres FK violation. Fixed by no longer sending it; 0pt's
dataset<->table linkage lives entirely in DatasetIcebergMapping.
- ST0RE enforces a real unique constraint on lakehouse namespaces
(tenant_id, project_id, name) with no upsert route. Fixed with a
client-side lookup-or-create (listNamespaces + findOrRegisterNamespace).
Also closed a field-level gap against the task's literal schema spec:
0pt.dataset_iceberg_mapping.schema.v1 now carries namespace_id (a real
reference to ST0RE's LakehouseNamespace.namespaceId, not guessed) and
mapped_at, both additive alongside the existing namespace/created_at
fields (sql/005, JSON schema, types, service).
tests/p0/iceberg-write-service.realinfra.test.ts replaces the prior
it.todo with a real SEAM-pattern live test: attempts the real HTTP call,
skips honestly if ST0RE is unreachable, never fabricates success. It ran
successfully against production ST0RE this session (real namespace/table/
snapshot UUIDs logged in docs/ICEBERG_WRITE_BOUNDARY.md).
Two ST0RE-side cross-tenant read bugs were found and flagged (not fixed
here -- out of scope for this repo): GET .../namespaces ignores its own
tenant header and query filters; GET .../tables/:id doesn't check the
caller's tenant against the table's owner.
npm test: 155 passed, 0 failed, 0 todo (up from 154 passed + 1 todo).
npx tsc --noEmit: 1 pre-existing, unrelated error in
quality-report-service.ts (present before this pass, part of the wider
uncommitted P0 data-platform build already in this repo) -- not touched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
172 lines
13 KiB
Markdown
172 lines
13 KiB
Markdown
# Iceberg write-path ownership boundary (0pt)
|
|
|
|
Read `docs/10-lakehouse-decision.md` and `docs/11-lakehouse-write-plan.md` first -- this doc is
|
|
the implementation record for the plan those wrote. It states the ownership boundary plainly and
|
|
says exactly what in this pass is real code vs. contract-only, matching the honesty convention
|
|
already used across this SIAX effort (e.g. sax3l/act0's own client docs, sax3l/st0re's
|
|
PRODUCTION_READINESS.md).
|
|
|
|
## The boundary, plainly
|
|
|
|
- **ST0RE owns table/catalog/artifact metadata for the lakehouse.** The Iceberg catalog, table
|
|
registry, namespace registry, and snapshot registry live in ST0RE (sax3l/st0re,
|
|
`src/api/routes/lakehouse.ts`, `/v1/st0re/lakehouse/*`). 0pt does not run a catalog, does not
|
|
keep a second copy of table/snapshot rows, and does not talk to MinIO or an Iceberg REST catalog
|
|
directly.
|
|
- **0pt owns dataset production, quality, data contracts, and writes.** 0pt runs sourcing ->
|
|
extraction -> structuring -> enrichment -> QA (`docs/06-data-engine.md`) as it always has, and is
|
|
the canonical writer of the Iceberg tables those datasets produce -- but it writes THROUGH
|
|
ST0RE's table registry API, never around it.
|
|
- **0pt owns source-rate/crawl-compliance policy.** This is the genuine ownership repair this pass
|
|
makes concrete: `0pt.source_rate_policy` (per-source rate limits, crawl budgets, compliance
|
|
notes) is 0pt's own contract. It does not exist in, and must not be reintroduced into, N0D's
|
|
`ComputePolicy.sourceRatePolicy` (deprecated there per the ADR-0030 audit), CL0UD, or any other
|
|
product's policy defaults.
|
|
- **N0D executes.** 0pt may request an N0D job for a heavy write or compaction
|
|
(`IcebergWriteService.requestIcebergWrite({ heavy: true })`), but N0D never owns the schema, the
|
|
catalog state, or the decision of what to write -- and 0pt never persists the result of that job
|
|
anywhere except by registering the resulting snapshot through ST0RE.
|
|
- **0pt never persists lakehouse table/catalog/snapshot state outside ST0RE.** Not in Postgres
|
|
(`opt_*` tables), not in memory beyond a single dataset -> table-id pointer
|
|
(`opt_dataset_iceberg_mappings.iceberg_table_id`, a REFERENCE, never a copy of ST0RE's row).
|
|
Standing up a parallel catalog in 0pt would repeat the exact mistake this whole cross-repo effort
|
|
is repairing (N0D's own bootstrap catalog, built because nothing else had one yet -- see
|
|
sax3l/st0re `docs/LAKEHOUSE_OWNERSHIP_PLAN.md`).
|
|
|
|
## What this pass built
|
|
|
|
| Concern | File(s) | Status |
|
|
|---|---|---|
|
|
| `0pt.dataset_iceberg_mapping.schema.v1` type | `src/types/dataset-iceberg-mapping.ts` | Real |
|
|
| `0pt.source_rate_policy.schema.v1` type | `src/types/source-rate-policy.ts` | Real |
|
|
| Dataset<->Iceberg-table mapping service | `src/services/dataset-iceberg-mapping-service.ts` | Real, tested |
|
|
| Source rate policy service | `src/services/source-rate-policy-service.ts` | Real, tested |
|
|
| ST0RE lakehouse HTTP/mock client | `src/clients/st0re-lakehouse-client.ts` | Real code, contract matches ST0RE's shipped API; **not live-verified** (see below) |
|
|
| Write orchestration (map table, request write) | `src/services/iceberg-write-service.ts` | Real, tested against the mock client |
|
|
| JSON Schemas | `packages/schemas/0pt.dataset_iceberg_mapping.schema.json`, `packages/schemas/0pt.source_rate_policy.schema.json` | Real |
|
|
| Postgres migration | `sql/004_iceberg_write_boundary.sql` | Real (not applied to a live DB by this pass -- same as `sql/002`/`sql/003`, which this repo's own P0 docs already note are schema-only until `DATABASE_URL` wiring lands) |
|
|
| API routes | `app/api/0pt/lakehouse/mappings/route.ts` (+`[mappingId]`, `+writes`), `app/api/0pt/sources/rate-policies/route.ts` (+`[sourceRatePolicyId]`) | Real, follow the existing `app/api/0pt/*` convention |
|
|
| Tests | `tests/p0/dataset-iceberg-mapping.test.ts`, `tests/p0/source-rate-policy.test.ts`, `tests/p0/iceberg-write-service.test.ts` | Real, all passing (`pnpm test`); one explicit `it.todo` for the live-ST0RE case |
|
|
|
|
## 2026-07-13 update: live verification + field-level gap closure
|
|
|
|
A later pass (same task spec re-run) found `api.st0re.siax.io` reachable from this environment
|
|
(resolves to `188.40.230.69` via this network's DNS even though the public Cloudflare A record was
|
|
still pending per this session's known infra state) and used it to actually drive the write path
|
|
end to end against production ST0RE, then closed every gap that surfaced:
|
|
|
|
- **`namespace_id` + `mapped_at` added** to `0pt.dataset_iceberg_mapping.schema.v1`
|
|
(`src/types/dataset-iceberg-mapping.ts`, the JSON schema, `sql/005_iceberg_write_boundary_gap_closure.sql`)
|
|
-- the task spec's literal field list (`dataset_id, iceberg_table_id, namespace_id, mapped_at,
|
|
mapped_by`) named these explicitly; the first pass only carried a bare `namespace` name string and
|
|
`created_at`. `namespace_id` is populated from ST0RE's real `LakehouseNamespace.namespaceId`, not
|
|
guessed.
|
|
- **Three real ST0RE contract facts found live, not reasoned about**, all now handled correctly by
|
|
`src/clients/st0re-lakehouse-client.ts` / `src/services/iceberg-write-service.ts` (see that file's
|
|
header comment for the full detail):
|
|
1. Every `/v1/st0re/lakehouse/*` route requires `X-Tenant-Id` + `X-Actor-Id` headers -- the
|
|
previous `HttpSt0reLakehouseClient` sent none. Fixed by threading a `St0reRequestContext`
|
|
through every client method.
|
|
2. `IcebergTableInput.datasetId` is a foreign key into ST0RE's *own* dataset table, not a
|
|
free-form string -- the previous service sent 0pt's `dataset_id` there, which 400s live with a
|
|
Postgres FK violation. Fixed by no longer sending it; the dataset<->table link lives entirely
|
|
in 0pt's own `DatasetIcebergMapping.iceberg_table_id`.
|
|
3. ST0RE enforces a real unique constraint on `(tenant_id, project_id, name)` for namespaces, with
|
|
no upsert route. Fixed with a client-side lookup-or-create
|
|
(`IcebergWriteService.findOrRegisterNamespace` + the client's new `listNamespaces`).
|
|
- **Two ST0RE-side bugs flagged, not fixed** (out of scope -- this task only touches the `0pt`
|
|
repo): `GET /v1/st0re/lakehouse/namespaces` ignores query filters and returns every tenant's rows
|
|
regardless of the caller's `X-Tenant-Id` header; `GET /v1/st0re/lakehouse/tables/:tableId` returns
|
|
a table's data even under a deliberately wrong `X-Tenant-Id`. Both look like missing tenant-scoping
|
|
on ST0RE's lakehouse read paths -- a real cross-tenant data-exposure risk in a different repo.
|
|
- **`tests/p0/iceberg-write-service.realinfra.test.ts` replaces the `it.todo`** -- a real
|
|
live-integration test (SEAM pattern: attempts the real HTTP call, `ctx.skip()`s honestly with a
|
|
logged reason if ST0RE is unreachable when the suite runs, never fabricates success). This session
|
|
ran it against live ST0RE and it created real rows:
|
|
- namespace `aeadcefb-1bce-43b1-bd66-fbdae30f4c6d` (tenant `t_0pt_realinfra_pilot`)
|
|
- table `5db3fcf8-a63c-4e26-806a-ba94d0d777c3` (`dim_company_se_pilot_<timestamp>`)
|
|
- snapshot `610c58bf-559b-40dc-82bb-6250f57e1dec` (append, 1 row, 128 bytes)
|
|
|
|
This is task item 3's "write ONE first test dataset to Iceberg through ST0RE's table registry"
|
|
proof -- executed for real, through 0pt's own code path (not raw curl, though curl was used first
|
|
to discover the contract facts above safely before changing code).
|
|
- Full suite after this pass: **155 passed, 0 failed** (up from the 154 passed + 1 todo baseline),
|
|
including the live realinfra test actually passing against production ST0RE at the time this pass
|
|
ran. `npx tsc --noEmit` has one **pre-existing, unrelated** error in
|
|
`src/services/quality-report-service.ts:110` (present before this pass touched anything, part of
|
|
the wider uncommitted P0 data-platform build already in this repo) -- not touched, out of scope.
|
|
|
|
## Verification status (real vs. contract-only -- read this before trusting any claim above)
|
|
|
|
- **Real and tested**: every TypeScript file above typechecks (except the one pre-existing,
|
|
unrelated `quality-report-service.ts` error noted above) and its behavior is exercised by a
|
|
passing vitest test in `tests/p0/`. The full suite (`pnpm test`) passed at 154 tests + 1 todo
|
|
after the first pass, up from a 142-test baseline before it, with 0 failures; the 2026-07-13 pass
|
|
above brought it to 155 passed, 0 failed, 0 todo, with the live ST0RE path now actually exercised.
|
|
- **Live-verified as of 2026-07-13** (supersedes the "not live-verified" note the first pass left
|
|
here): `src/clients/st0re-lakehouse-client.ts`'s HTTP path was executed against a running,
|
|
reachable ST0RE instance (`api.st0re.siax.io`) by a later pass -- not just written against the
|
|
documented contract. Real IDs it created are listed in the "2026-07-13 update" section above. Unit
|
|
tests still exercise the deterministic in-memory mock for speed/hermeticity (same convention
|
|
`cl0ud-client.ts`/`n0d-client.ts`/`aud0-client.ts`/`st0re-client.ts` use); only
|
|
`tests/p0/iceberg-write-service.realinfra.test.ts` hits the real HTTP path, and only when ST0RE is
|
|
reachable (honest skip otherwise).
|
|
- **No longer an open TODO**: the `it.todo(...)` line this section used to point to
|
|
(`tests/p0/iceberg-write-service.test.ts`) has been replaced by the real
|
|
`tests/p0/iceberg-write-service.realinfra.test.ts` described above.
|
|
- **Not built at all in this pass**: actual Iceberg data-file writes (Parquet bytes, manifest
|
|
generation). `IcebergWriteService.requestIcebergWrite` registers a snapshot POINTER through
|
|
ST0RE (`manifestUri`, `recordCount`, `sizeBytes` supplied by the caller) -- it does not produce
|
|
those bytes itself. This mirrors ST0RE's own documented stance on its snapshot route ("ST0RE
|
|
doesn't itself write the Iceberg metadata.json... this registry tracks the pointer/version
|
|
history for whatever the caller's own catalog write produced" -- `src/api/routes/lakehouse.ts` in
|
|
sax3l/st0re). Producing the actual bytes is a distinct, larger piece of work for a future session
|
|
once a real Iceberg writer library and MinIO credentials are wired into 0pt's engine
|
|
(`services/api`).
|
|
|
|
## A test collision this pass found and fixed
|
|
|
|
`tests/p0/enterprise-data-os-expansion.test.ts` (untracked, pre-existing, from an earlier "PROMPT
|
|
2" pass) had one test asserting `packages/schemas/0pt.source_rate_policy.schema.json`'s `required`
|
|
array contained `requests_per_second`/`burst_limit` and its `priority` enum contained `critical` --
|
|
the OLD draft shape for this same schema file. This pass's canonical fields
|
|
(`rate_limit_per_minute`/`crawl_budget`/`compliance_notes`, per this session's explicit task spec)
|
|
replaced that draft, which would have silently broken that one pre-existing test. Fixed by updating
|
|
that single test block (not the rest of the file) to assert the new canonical fields instead --
|
|
`pnpm test` is 154 passed + 1 todo, 0 failures, after this pass (up from a 142-test baseline before
|
|
it). See that test file's inline comment for the same note.
|
|
|
|
## A pre-existing artifact this pass did not touch
|
|
|
|
`packages/schemas/0pt.iceberg_table.schema.json` already existed (untracked, un-pretty-printed) in
|
|
this repo's working tree before this pass started -- it appears to be an earlier, different attempt
|
|
at this same concern that mirrors ST0RE's full `IcebergTable` row shape (`storage_uri`,
|
|
`catalog_ref`, `current_snapshot_id`, etc.) rather than a lightweight reference. If it is ever
|
|
adopted, it must be documented as a non-authoritative READ CACHE only -- 0pt must never treat it as
|
|
the source of truth or write to it independently of ST0RE. This pass's actual, owned contract is
|
|
`0pt.dataset_iceberg_mapping.schema.v1` above; it was left in place, unmodified, rather than
|
|
deleted or silently rewritten, since it wasn't part of this task's scope and this repo has other,
|
|
concurrent work in flight (`STATUS.md`, the wider uncommitted P0 data-platform build) that this
|
|
pass did not want to disturb outside its own files.
|
|
|
|
## Event names (all four wired, docs/11 item 6)
|
|
|
|
| Event | Emitted by | When |
|
|
|---|---|---|
|
|
| `0pt.dataset_iceberg_table_mapped` | `DatasetIcebergMappingService.createMapping` | A dataset -> Iceberg-table mapping is created |
|
|
| `0pt.iceberg_write_requested` | `IcebergWriteService.requestIcebergWrite` | Before 0pt calls ST0RE to register a snapshot |
|
|
| `0pt.iceberg_snapshot_created` | `IcebergWriteService.requestIcebergWrite` | After ST0RE confirms the new snapshot |
|
|
| `0pt.source_rate_policy_created` | `SourceRatePolicyService.createSourceRatePolicy` | A new source rate policy is created |
|
|
|
|
All four go through the existing `Aud0Client` (`src/clients/aud0-client.ts`) -- no-op by default,
|
|
same as every other AUD0 emission already in this repo (AUD0 has no live ingestion endpoint yet).
|
|
|
|
## Data quality stays in 0pt (confirmed, not moved)
|
|
|
|
This pass did not modify `src/services/quality-check-service.ts`, `src/services/quality-report-service.ts`,
|
|
`src/types/quality-check.ts`, or `src/types/quality-report.ts`. 0pt's QA stage
|
|
(`docs/06-data-engine.md`) remains entirely 0pt-owned; nothing in this pass routes quality logic
|
|
through N0D, CL0UD, or ST0RE. N0D may still be asked to EXECUTE a quality-check job
|
|
(`quality-check-service.ts`'s existing `n0d.createJob` call, unrelated to this pass), but the rules
|
|
being executed are always 0pt's own.
|