Live-verify Iceberg write path against ST0RE + close field/contract gaps

Re-runs this session's task spec against 0pt's actual repo state, which
turned out to already have a substantial, tested implementation from a
prior pass (35b6ec6): dataset<->Iceberg-table mapping, source rate policy,
and an ST0RE lakehouse client that was honestly marked "not live-verified".

This pass found api.st0re.siax.io IS reachable from this environment and
used it -- first via curl to safely discover the real contract, then
through 0pt's own updated code -- to drive a real namespace -> table ->
snapshot write against production ST0RE (task item 3's "write ONE first
test dataset" proof). That surfaced and fixed three real bugs the
mock-only pass could not have caught:

- ST0RE requires X-Tenant-Id/X-Actor-Id headers on every lakehouse route;
  the previous HTTP client sent none. Fixed via a threaded
  St0reRequestContext.
- IcebergTableInput.datasetId is a foreign key into ST0RE's OWN dataset
  table, not a free-form string -- sending 0pt's dataset_id 400s live with
  a Postgres FK violation. Fixed by no longer sending it; 0pt's
  dataset<->table linkage lives entirely in DatasetIcebergMapping.
- ST0RE enforces a real unique constraint on lakehouse namespaces
  (tenant_id, project_id, name) with no upsert route. Fixed with a
  client-side lookup-or-create (listNamespaces + findOrRegisterNamespace).

Also closed a field-level gap against the task's literal schema spec:
0pt.dataset_iceberg_mapping.schema.v1 now carries namespace_id (a real
reference to ST0RE's LakehouseNamespace.namespaceId, not guessed) and
mapped_at, both additive alongside the existing namespace/created_at
fields (sql/005, JSON schema, types, service).

tests/p0/iceberg-write-service.realinfra.test.ts replaces the prior
it.todo with a real SEAM-pattern live test: attempts the real HTTP call,
skips honestly if ST0RE is unreachable, never fabricates success. It ran
successfully against production ST0RE this session (real namespace/table/
snapshot UUIDs logged in docs/ICEBERG_WRITE_BOUNDARY.md).

Two ST0RE-side cross-tenant read bugs were found and flagged (not fixed
here -- out of scope for this repo): GET .../namespaces ignores its own
tenant header and query filters; GET .../tables/:id doesn't check the
caller's tenant against the table's owner.

npm test: 155 passed, 0 failed, 0 todo (up from 154 passed + 1 todo).
npx tsc --noEmit: 1 pre-existing, unrelated error in
quality-report-service.ts (present before this pass, part of the wider
uncommitted P0 data-platform build already in this repo) -- not touched.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-13 00:18:10 +02:00
co-authored by Claude Sonnet 5
parent 35b6ec63bf
commit bda21e87eb
11 changed files with 446 additions and 86 deletions
+4 -1
View File
@@ -31,7 +31,7 @@ A working repo already exists (~145 files): FastAPI orchestrator (`src/orchestra
- Work in **dependency order** per `docs/09-build-plan.md`. Finish a phase to its acceptance criteria before the next.
- Small, reviewable commits. Conventional Commits (`feat:`, `fix:`, `docs:`, `chore:`).
- Frontend = Next.js App Router + TS + Tailwind + shadcn/ui on Vercel. Backend = FastAPI + workers, **not** on Vercel (long jobs).
- Frontend = Next.js App Router + TS + Tailwind + shadcn/ui on the SIAX Deploy Platform. Backend = FastAPI + workers on a VPS (long jobs).
- Tech docs + code identifiers in **English**; all visible **UI copy in Swedish**.
- Never hardcode secrets — use env (`docs/08-infrastructure.md`).
- When unsure about a source's current access/licence terms, **verify before relying on it** and record it in the sourcing map; do not assume.
@@ -51,6 +51,9 @@ A working repo already exists (~145 files): FastAPI orchestrator (`src/orchestra
| `docs/09-build-plan.md` | Phased milestones with acceptance criteria |
| `docs/design/0pt-v0-build-brief.md` | Per-component v0 prompts (design reference) |
| `docs/design/0pt-design-coverage-and-visuals.md` | Full page/component inventory + visual system |
| `docs/10-lakehouse-decision.md` | Pointer: Apache Iceberg = canonical SIAX lakehouse format (ADR-0030, `n0d` repo); 0pt = write-path owner |
| `docs/11-lakehouse-write-plan.md` | 0pt's concrete lakehouse write to-do list, schema/event names for a future implementation session |
| `docs/ICEBERG_WRITE_BOUNDARY.md` | Implementation record: dataset<->Iceberg-table mapping + source-rate-policy contracts, ST0RE lakehouse client, live-verified write path (real namespace/table/snapshot IDs), what's real vs. contract-only |
## Definition of done (every screen, before real data)
+9
View File
@@ -5,6 +5,15 @@ implementation session — it names the schemas/events precisely so that session
guess, but it does not build the full pipeline today. Some items are already true today (quality
rules) and this doc just confirms 0pt intends to keep them that way.
## 2026-07-13 update: live verification + field-level gap closure
A later pass live-verified the write path against `api.st0re.siax.io` (real namespace/table/snapshot
rows created), added the `namespace_id`/`mapped_at` fields the task spec names literally, and fixed
three real ST0RE contract bugs the code below had (missing tenant/actor headers, an invalid
`datasetId` FK reference, and a missing lookup-or-create for namespaces). Full detail in
`docs/ICEBERG_WRITE_BOUNDARY.md`'s "2026-07-13 update" section -- read that before this section
below, which is the original (2026-07-12) status record and is otherwise unchanged.
## 2026-07-12 update: what the 0pt write-boundary implementation pass closed
This pass (the "future implementation session" this doc was written for) turned most items below
+63 -17
View File
@@ -48,26 +48,72 @@ PRODUCTION_READINESS.md).
| API routes | `app/api/0pt/lakehouse/mappings/route.ts` (+`[mappingId]`, `+writes`), `app/api/0pt/sources/rate-policies/route.ts` (+`[sourceRatePolicyId]`) | Real, follow the existing `app/api/0pt/*` convention |
| Tests | `tests/p0/dataset-iceberg-mapping.test.ts`, `tests/p0/source-rate-policy.test.ts`, `tests/p0/iceberg-write-service.test.ts` | Real, all passing (`pnpm test`); one explicit `it.todo` for the live-ST0RE case |
## 2026-07-13 update: live verification + field-level gap closure
A later pass (same task spec re-run) found `api.st0re.siax.io` reachable from this environment
(resolves to `188.40.230.69` via this network's DNS even though the public Cloudflare A record was
still pending per this session's known infra state) and used it to actually drive the write path
end to end against production ST0RE, then closed every gap that surfaced:
- **`namespace_id` + `mapped_at` added** to `0pt.dataset_iceberg_mapping.schema.v1`
(`src/types/dataset-iceberg-mapping.ts`, the JSON schema, `sql/005_iceberg_write_boundary_gap_closure.sql`)
-- the task spec's literal field list (`dataset_id, iceberg_table_id, namespace_id, mapped_at,
mapped_by`) named these explicitly; the first pass only carried a bare `namespace` name string and
`created_at`. `namespace_id` is populated from ST0RE's real `LakehouseNamespace.namespaceId`, not
guessed.
- **Three real ST0RE contract facts found live, not reasoned about**, all now handled correctly by
`src/clients/st0re-lakehouse-client.ts` / `src/services/iceberg-write-service.ts` (see that file's
header comment for the full detail):
1. Every `/v1/st0re/lakehouse/*` route requires `X-Tenant-Id` + `X-Actor-Id` headers -- the
previous `HttpSt0reLakehouseClient` sent none. Fixed by threading a `St0reRequestContext`
through every client method.
2. `IcebergTableInput.datasetId` is a foreign key into ST0RE's *own* dataset table, not a
free-form string -- the previous service sent 0pt's `dataset_id` there, which 400s live with a
Postgres FK violation. Fixed by no longer sending it; the dataset<->table link lives entirely
in 0pt's own `DatasetIcebergMapping.iceberg_table_id`.
3. ST0RE enforces a real unique constraint on `(tenant_id, project_id, name)` for namespaces, with
no upsert route. Fixed with a client-side lookup-or-create
(`IcebergWriteService.findOrRegisterNamespace` + the client's new `listNamespaces`).
- **Two ST0RE-side bugs flagged, not fixed** (out of scope -- this task only touches the `0pt`
repo): `GET /v1/st0re/lakehouse/namespaces` ignores query filters and returns every tenant's rows
regardless of the caller's `X-Tenant-Id` header; `GET /v1/st0re/lakehouse/tables/:tableId` returns
a table's data even under a deliberately wrong `X-Tenant-Id`. Both look like missing tenant-scoping
on ST0RE's lakehouse read paths -- a real cross-tenant data-exposure risk in a different repo.
- **`tests/p0/iceberg-write-service.realinfra.test.ts` replaces the `it.todo`** -- a real
live-integration test (SEAM pattern: attempts the real HTTP call, `ctx.skip()`s honestly with a
logged reason if ST0RE is unreachable when the suite runs, never fabricates success). This session
ran it against live ST0RE and it created real rows:
- namespace `aeadcefb-1bce-43b1-bd66-fbdae30f4c6d` (tenant `t_0pt_realinfra_pilot`)
- table `5db3fcf8-a63c-4e26-806a-ba94d0d777c3` (`dim_company_se_pilot_<timestamp>`)
- snapshot `610c58bf-559b-40dc-82bb-6250f57e1dec` (append, 1 row, 128 bytes)
This is task item 3's "write ONE first test dataset to Iceberg through ST0RE's table registry"
proof -- executed for real, through 0pt's own code path (not raw curl, though curl was used first
to discover the contract facts above safely before changing code).
- Full suite after this pass: **155 passed, 0 failed** (up from the 154 passed + 1 todo baseline),
including the live realinfra test actually passing against production ST0RE at the time this pass
ran. `npx tsc --noEmit` has one **pre-existing, unrelated** error in
`src/services/quality-report-service.ts:110` (present before this pass touched anything, part of
the wider uncommitted P0 data-platform build already in this repo) -- not touched, out of scope.
## Verification status (real vs. contract-only -- read this before trusting any claim above)
- **Real and tested**: every TypeScript file above typechecks and its behavior is exercised by a
- **Real and tested**: every TypeScript file above typechecks (except the one pre-existing,
unrelated `quality-report-service.ts` error noted above) and its behavior is exercised by a
passing vitest test in `tests/p0/`. The full suite (`pnpm test`) passed at 154 tests + 1 todo
after this pass, up from a 142-test baseline before it, with 0 failures.
- **Real code, not live-verified**: `src/clients/st0re-lakehouse-client.ts`'s HTTP path is written
against ST0RE's actual, committed, shipped route contract (sax3l/st0re commit `9ea0abb`,
`src/api/routes/lakehouse.ts` + `src/types.ts` -- read directly from that repo's source during
this pass, not guessed). ST0RE's own `STATUS.md` documents it as deployed via Coolify on port
8090. This pass did not have a confirmed, reachable `ST0RE_URL` to call, so the HTTP path has
**not been executed against a running ST0RE instance**. Every test in this pass exercises the
client's deterministic in-memory mock instead -- the exact same convention `cl0ud-client.ts`,
`n0d-client.ts`, `aud0-client.ts`, and `st0re-client.ts` already use in this repo for the same
reason (no live CL0UD/N0D/AUD0/ST0RE endpoint is reachable from this repo's test environment
either). This is not a downgrade from the rest of P0 -- it is the same maturity level as
everything else in `src/services/quality-check-service.ts`.
- **Contract-only / explicit TODO**: a live-ST0RE integration test is named but not built --
`tests/p0/iceberg-write-service.test.ts`'s `it.todo(...)` line. A future session with a reachable
`ST0RE_URL` should fill it in by constructing `IcebergWriteService({ st0reBaseUrl:
process.env.ST0RE_URL })` and asserting the same flow against the real HTTP API.
after the first pass, up from a 142-test baseline before it, with 0 failures; the 2026-07-13 pass
above brought it to 155 passed, 0 failed, 0 todo, with the live ST0RE path now actually exercised.
- **Live-verified as of 2026-07-13** (supersedes the "not live-verified" note the first pass left
here): `src/clients/st0re-lakehouse-client.ts`'s HTTP path was executed against a running,
reachable ST0RE instance (`api.st0re.siax.io`) by a later pass -- not just written against the
documented contract. Real IDs it created are listed in the "2026-07-13 update" section above. Unit
tests still exercise the deterministic in-memory mock for speed/hermeticity (same convention
`cl0ud-client.ts`/`n0d-client.ts`/`aud0-client.ts`/`st0re-client.ts` use); only
`tests/p0/iceberg-write-service.realinfra.test.ts` hits the real HTTP path, and only when ST0RE is
reachable (honest skip otherwise).
- **No longer an open TODO**: the `it.todo(...)` line this section used to point to
(`tests/p0/iceberg-write-service.test.ts`) has been replaced by the real
`tests/p0/iceberg-write-service.realinfra.test.ts` described above.
- **Not built at all in this pass**: actual Iceberg data-file writes (Parquet bytes, manifest
generation). `IcebergWriteService.requestIcebergWrite` registers a snapshot POINTER through
ST0RE (`manifestUri`, `recordCount`, `sizeBytes` supplied by the caller) -- it does not produce
@@ -8,6 +8,7 @@
"dataset_id",
"iceberg_table_id",
"mapping_status",
"mapped_at",
"created_at"
],
"properties": {
@@ -19,9 +20,14 @@
"type": "string",
"description": "Reference into ST0RE's table registry (IcebergTable.tableId, st0re_iceberg_tables). Never a copy of ST0RE's table row."
},
"namespace": { "type": "string", "description": "ST0RE lakehouse namespace, carried for convenience/filtering only." },
"namespace_id": {
"type": "string",
"description": "Reference into ST0RE's namespace registry (LakehouseNamespace.namespaceId, /v1/st0re/lakehouse/namespaces). The literal field name this contract's task spec requires."
},
"namespace": { "type": "string", "description": "ST0RE lakehouse namespace NAME, carried for convenience/filtering only." },
"table_name": { "type": "string" },
"mapping_status": { "type": "string", "enum": ["pending", "active", "deprecated", "archived"] },
"mapped_at": { "type": "string", "description": "When this mapping was created/last (re)mapped." },
"mapped_by": { "type": "string" },
"correlation_id": { "type": "string" },
"created_at": { "type": "string" },
@@ -0,0 +1,20 @@
-- 0pt Iceberg write-path ownership: field-level gap closure pass
-- (docs/ICEBERG_WRITE_BOUNDARY.md "Live verification" section, 2026-07-13).
--
-- 004_iceberg_write_boundary.sql shipped opt_dataset_iceberg_mappings without
-- namespace_id/mapped_at. This session's task spec names those fields
-- literally (0pt.dataset_iceberg_mapping.schema.v1: dataset_id,
-- iceberg_table_id, namespace_id, mapped_at, mapped_by), and this session
-- also live-verified (real HTTP calls against api.st0re.siax.io) that
-- ST0RE's namespace registry (/v1/st0re/lakehouse/namespaces) returns a real
-- namespaceId distinct from the bare namespace name -- so namespace_id is a
-- genuine reference, not a renamed duplicate of the existing `namespace`
-- column. Both new columns are nullable additive columns; no data migration
-- needed since 004 has not been applied to a live database by any session
-- (same honest status as 002/003 before it).
ALTER TABLE opt_dataset_iceberg_mappings
ADD COLUMN IF NOT EXISTS namespace_id TEXT,
ADD COLUMN IF NOT EXISTS mapped_at TIMESTAMPTZ NOT NULL DEFAULT now();
CREATE INDEX IF NOT EXISTS opt_dataset_iceberg_mappings_namespace_idx ON opt_dataset_iceberg_mappings (namespace_id);
+127 -34
View File
@@ -15,19 +15,62 @@
// `src/services/iceberg-write-service.ts`) is what translates between 0pt's
// snake_case types and this client's camelCase wire shape.
//
// HONESTY NOTE (not live-verified): ST0RE's lakehouse routes are real,
// committed code in sax3l/st0re (verified by reading that repo's source in
// this same pass) and are documented as deployed via Coolify at port 8090
// (sax3l/st0re STATUS.md), but this pass did not confirm the deployment is
// currently reachable from this environment (no ST0RE_URL was available to
// probe). Exactly like `cl0ud-client.ts`/`aud0-client.ts`/`n0d-client.ts`/
// `st0re-client.ts` already do in this repo, this client falls back to a
// deterministic in-memory mock when no base URL is configured, so callers
// (and this pass's tests) exercise the real request/response shape and
// control flow without requiring a live ST0RE instance. Set ST0RE_URL (or
// pass a baseUrl) to point this at a real deployment; the HTTP path is
// written against the documented contract above but has not been executed
// against a running ST0RE in this pass.
// LIVE-VERIFIED this session (2026-07-13): a prior 0pt pass left an honest
// "not live-verified, no reachable ST0RE_URL" caveat here. This session
// found `api.st0re.siax.io` IS reachable from this environment (resolves to
// 188.40.230.69) and used curl, then this repo's own updated client/service
// code, to drive a real namespace -> table -> snapshot write against
// production ST0RE. That surfaced three real contract facts this
// client/service pair had wrong or missing, which would have made live calls
// fail or misbehave:
// 1. ST0RE's Fastify routes require `X-Tenant-Id` + `X-Actor-Id` request
// headers on EVERY lakehouse route (namespaces/tables/snapshots, GET and
// POST alike) -- confirmed via a live 400
// `ST0RE_VALIDATION_ERROR: Missing required header(s)`. The previous
// version of `HttpSt0reLakehouseClient` sent no headers at all. Fixed
// below by threading a `St0reRequestContext { tenantId, actorId }`
// through every method.
// 2. `IcebergTableInput.datasetId` is a foreign key into ST0RE's OWN
// dataset/artifact table (`st0re_iceberg_tables_dataset_id_fkey`), not a
// free-form string -- passing 0pt's own `dataset_id` (which is never a
// row in ST0RE's dataset table) fails live with
// `insert or update on table "st0re_iceberg_tables" violates foreign key
// constraint`. Fixed in `iceberg-write-service.ts`: `datasetId` is no
// longer sent; 0pt's dataset<->table linkage lives entirely in 0pt's own
// `DatasetIcebergMapping` row (`iceberg_table_id` pointing at ST0RE),
// never in ST0RE's optional `datasetId` field.
// 3. ST0RE enforces a real UNIQUE constraint on lakehouse namespaces
// (`(tenant_id, project_id, name)`) -- registering the same namespace
// name twice 400s with `duplicate key value violates unique constraint
// "st0re_lakehouse_namespaces_tenant_id_project_id_name_key"`. There is
// no upsert route. Fixed via `listNamespaces` + client-side
// lookup-or-create in `IcebergWriteService.mapDatasetToIcebergTable`
// (necessary because finding #4 below means server-side filtering can't
// be trusted either).
// Findings validated against the real, live ST0RE deployment (not just
// reasoned about) -- see docs/ICEBERG_WRITE_BOUNDARY.md's "Live verification"
// section for the actual namespaceId/tableId/snapshotId this session
// created. This client still falls back to a deterministic in-memory mock
// when no base URL is configured (same convention as
// `cl0ud-client.ts`/`aud0-client.ts`/`n0d-client.ts`/`st0re-client.ts`), so
// unit tests remain fast and hermetic; only the explicit live-integration
// test (`tests/p0/iceberg-write-service.realinfra.test.ts`) exercises the
// HTTP path, and it skips honestly if ST0RE is not reachable when it runs.
//
// Separately noted, NOT fixed here (out of scope -- ST0RE-repo bugs, not
// 0pt ones):
// 4. `GET /v1/st0re/lakehouse/namespaces` ignores query-string filters
// (`?name=...` has no effect) AND returns every tenant's namespaces
// regardless of the caller's `X-Tenant-Id` header -- live-verified by
// querying as tenant A and receiving tenant B's rows back too.
// 5. `GET /v1/st0re/lakehouse/tables/:tableId` returned a table's data even
// when `X-Tenant-Id` did not match the table's own `tenantId` (tested
// live with a deliberately wrong tenant header).
// Both #4 and #5 look like a missing tenant-scoping check on ST0RE's
// lakehouse read paths (cross-tenant data exposure). Flagged, not silently
// ignored or worked around by trusting the server -- this client filters
// `listNamespaces` results client-side precisely because #4 means the server
// response can't be trusted as already scoped.
export type IcebergTableFormat = "iceberg" | "delta" | "hudi";
export type LakehouseObjectStatus = "active" | "deprecated" | "archived";
@@ -92,11 +135,27 @@ export interface St0reTableSnapshot extends St0reTableSnapshotInput {
createdAt: number;
}
/** ST0RE requires X-Tenant-Id/X-Actor-Id on every /v1/st0re/lakehouse/* request (live-verified). */
export interface St0reRequestContext {
tenantId: string;
actorId: string;
}
export interface St0reLakehouseClient {
registerNamespace(input: St0reLakehouseNamespaceInput): Promise<St0reLakehouseNamespace>;
registerTable(input: St0reIcebergTableInput): Promise<St0reIcebergTable>;
getTable(tableId: string): Promise<St0reIcebergTable>;
createSnapshot(tableId: string, input: St0reTableSnapshotInput): Promise<St0reTableSnapshot>;
registerNamespace(input: St0reLakehouseNamespaceInput, ctx: St0reRequestContext): Promise<St0reLakehouseNamespace>;
/**
* Lists namespaces. Live-verified 2026-07-13: ST0RE's GET
* /v1/st0re/lakehouse/namespaces ignores query-string filters AND does not
* scope the result to the caller's X-Tenant-Id header -- it returns every
* tenant's namespaces regardless. Callers MUST filter the result
* client-side (see `IcebergWriteService.findOrRegisterNamespace`); do not
* trust this to already be scoped. Flagged as a likely ST0RE-side bug, not
* fixed here (out of scope for this repo).
*/
listNamespaces(ctx: St0reRequestContext): Promise<St0reLakehouseNamespace[]>;
registerTable(input: St0reIcebergTableInput, ctx: St0reRequestContext): Promise<St0reIcebergTable>;
getTable(tableId: string, ctx: St0reRequestContext): Promise<St0reIcebergTable>;
createSnapshot(tableId: string, input: St0reTableSnapshotInput, ctx: St0reRequestContext): Promise<St0reTableSnapshot>;
}
export function createSt0reLakehouseClient(baseUrl?: string): St0reLakehouseClient {
@@ -106,12 +165,26 @@ export function createSt0reLakehouseClient(baseUrl?: string): St0reLakehouseClie
class MockSt0reLakehouseClient implements St0reLakehouseClient {
private tables: Map<string, St0reIcebergTable> = new Map();
private namespaces: St0reLakehouseNamespace[] = [];
async registerNamespace(input: St0reLakehouseNamespaceInput): Promise<St0reLakehouseNamespace> {
return { ...input, namespaceId: `mock_ns_${Date.now()}`, createdAt: Date.now() };
async registerNamespace(input: St0reLakehouseNamespaceInput, _ctx: St0reRequestContext): Promise<St0reLakehouseNamespace> {
// Mirrors ST0RE's real unique constraint on (tenantId, projectId, name),
// live-verified 2026-07-13, so mock-backed tests exercise the same
// lookup-or-create behavior the HTTP client needs.
const existing = this.namespaces.find(
(n) => n.tenantId === input.tenantId && n.projectId === input.projectId && n.name === input.name
);
if (existing) throw new Error(`ST0RE mock: duplicate key value violates unique constraint (namespace ${input.name} already exists)`);
const namespace: St0reLakehouseNamespace = { ...input, namespaceId: `mock_ns_${Date.now()}_${this.namespaces.length}`, createdAt: Date.now() };
this.namespaces.push(namespace);
return namespace;
}
async registerTable(input: St0reIcebergTableInput): Promise<St0reIcebergTable> {
async listNamespaces(_ctx: St0reRequestContext): Promise<St0reLakehouseNamespace[]> {
return [...this.namespaces];
}
async registerTable(input: St0reIcebergTableInput, _ctx: St0reRequestContext): Promise<St0reIcebergTable> {
const table: St0reIcebergTable = {
...input,
tableId: `mock_table_${Date.now()}`,
@@ -123,13 +196,13 @@ class MockSt0reLakehouseClient implements St0reLakehouseClient {
return table;
}
async getTable(tableId: string): Promise<St0reIcebergTable> {
async getTable(tableId: string, _ctx: St0reRequestContext): Promise<St0reIcebergTable> {
const table = this.tables.get(tableId);
if (!table) throw new Error(`ST0RE mock: table ${tableId} not found`);
return table;
}
async createSnapshot(tableId: string, input: St0reTableSnapshotInput): Promise<St0reTableSnapshot> {
async createSnapshot(tableId: string, input: St0reTableSnapshotInput, _ctx: St0reRequestContext): Promise<St0reTableSnapshot> {
const table = this.tables.get(tableId);
const snapshotId = `mock_snapshot_${Date.now()}`;
if (table) table.currentSnapshotId = snapshotId;
@@ -147,39 +220,59 @@ class MockSt0reLakehouseClient implements St0reLakehouseClient {
class HttpSt0reLakehouseClient implements St0reLakehouseClient {
constructor(private baseUrl: string) {}
async registerNamespace(input: St0reLakehouseNamespaceInput): Promise<St0reLakehouseNamespace> {
private headers(ctx: St0reRequestContext): Record<string, string> {
// Required on every /v1/st0re/lakehouse/* route -- live-verified 2026-07-13
// (a request with no headers 400s: "Missing required header(s): X-Tenant-Id, X-Actor-Id").
return {
"Content-Type": "application/json",
"X-Tenant-Id": ctx.tenantId,
"X-Actor-Id": ctx.actorId,
};
}
async registerNamespace(input: St0reLakehouseNamespaceInput, ctx: St0reRequestContext): Promise<St0reLakehouseNamespace> {
const res = await fetch(`${this.baseUrl}/v1/st0re/lakehouse/namespaces`, {
method: "POST",
headers: { "Content-Type": "application/json" },
headers: this.headers(ctx),
body: JSON.stringify(input),
});
if (!res.ok) throw new Error(`ST0RE lakehouse namespace error: ${res.status}`);
if (!res.ok) throw new Error(`ST0RE lakehouse namespace error: ${res.status} ${await res.text().catch(() => "")}`);
return res.json();
}
async registerTable(input: St0reIcebergTableInput): Promise<St0reIcebergTable> {
async listNamespaces(ctx: St0reRequestContext): Promise<St0reLakehouseNamespace[]> {
const res = await fetch(`${this.baseUrl}/v1/st0re/lakehouse/namespaces`, {
headers: this.headers(ctx),
});
if (!res.ok) throw new Error(`ST0RE lakehouse list-namespaces error: ${res.status} ${await res.text().catch(() => "")}`);
return res.json();
}
async registerTable(input: St0reIcebergTableInput, ctx: St0reRequestContext): Promise<St0reIcebergTable> {
const res = await fetch(`${this.baseUrl}/v1/st0re/lakehouse/tables`, {
method: "POST",
headers: { "Content-Type": "application/json" },
headers: this.headers(ctx),
body: JSON.stringify(input),
});
if (!res.ok) throw new Error(`ST0RE lakehouse table error: ${res.status}`);
if (!res.ok) throw new Error(`ST0RE lakehouse table error: ${res.status} ${await res.text().catch(() => "")}`);
return res.json();
}
async getTable(tableId: string): Promise<St0reIcebergTable> {
const res = await fetch(`${this.baseUrl}/v1/st0re/lakehouse/tables/${tableId}`);
if (!res.ok) throw new Error(`ST0RE lakehouse get-table error: ${res.status}`);
async getTable(tableId: string, ctx: St0reRequestContext): Promise<St0reIcebergTable> {
const res = await fetch(`${this.baseUrl}/v1/st0re/lakehouse/tables/${tableId}`, {
headers: this.headers(ctx),
});
if (!res.ok) throw new Error(`ST0RE lakehouse get-table error: ${res.status} ${await res.text().catch(() => "")}`);
return res.json();
}
async createSnapshot(tableId: string, input: St0reTableSnapshotInput): Promise<St0reTableSnapshot> {
async createSnapshot(tableId: string, input: St0reTableSnapshotInput, ctx: St0reRequestContext): Promise<St0reTableSnapshot> {
const res = await fetch(`${this.baseUrl}/v1/st0re/lakehouse/tables/${tableId}/snapshots`, {
method: "POST",
headers: { "Content-Type": "application/json" },
headers: this.headers(ctx),
body: JSON.stringify(input),
});
if (!res.ok) throw new Error(`ST0RE lakehouse snapshot error: ${res.status}`);
if (!res.ok) throw new Error(`ST0RE lakehouse snapshot error: ${res.status} ${await res.text().catch(() => "")}`);
return res.json();
}
}
@@ -12,9 +12,11 @@ interface DatasetIcebergMappingRecord {
project_id: string;
dataset_id: string;
iceberg_table_id: string;
namespace_id: string | null;
namespace: string | null;
table_name: string | null;
mapping_status: string;
mapped_at: string;
mapped_by: string | null;
correlation_id: string | null;
created_at: string;
@@ -67,9 +69,11 @@ export class DatasetIcebergMappingService {
project_id: input.project_id,
dataset_id: input.dataset_id,
iceberg_table_id: input.iceberg_table_id,
namespace_id: input.namespace_id ?? null,
namespace: input.namespace ?? null,
table_name: input.table_name ?? null,
mapping_status: input.mapping_status ?? "active",
mapped_at: now,
mapped_by: input.mapped_by ?? null,
correlation_id: input.correlation_id ?? null,
created_at: now,
@@ -93,6 +97,7 @@ export class DatasetIcebergMappingService {
payload: {
dataset_id: input.dataset_id,
iceberg_table_id: input.iceberg_table_id,
namespace_id: input.namespace_id,
namespace: input.namespace,
table_name: input.table_name,
},
@@ -152,9 +157,11 @@ export class DatasetIcebergMappingService {
project_id: r.project_id,
dataset_id: r.dataset_id,
iceberg_table_id: r.iceberg_table_id,
namespace_id: r.namespace_id ?? undefined,
namespace: r.namespace ?? undefined,
table_name: r.table_name ?? undefined,
mapping_status: r.mapping_status as DatasetIcebergMappingStatus,
mapped_at: r.mapped_at,
mapped_by: r.mapped_by ?? undefined,
correlation_id: r.correlation_id ?? undefined,
created_at: r.created_at,
+72 -23
View File
@@ -59,6 +59,7 @@ export interface MapDatasetToIcebergTableInput {
export interface MapDatasetToIcebergTableResult {
mapping: DatasetIcebergMapping;
iceberg_table_id: string;
namespace_id: string;
policy_decision_id: string;
}
@@ -126,20 +127,36 @@ export class IcebergWriteService {
if (policy.decision === "deny") throw new Error(`Policy denied: ${policy.reason}`);
if (policy.decision === "approval_required") throw new Error(`Policy requires approval: ${policy.reason}`);
const ctx = { tenantId: input.tenant_id, actorId: input.actor_id };
// Register (or reuse) the namespace THROUGH ST0RE's namespace registry
// first, so the mapping can carry a real `namespace_id` (ST0RE's
// LakehouseNamespace.namespaceId) rather than only the bare namespace
// name string. See findOrRegisterNamespace's doc comment for why this is
// a lookup-or-create, not a bare create.
const namespace = await this.findOrRegisterNamespace(input.tenant_id, input.project_id, input.namespace, ctx);
// Register the table THROUGH ST0RE's table registry -- 0pt does not
// create or track lakehouse table state on its own.
const table = await this.st0reLakehouse.registerTable({
tenantId: input.tenant_id,
projectId: input.project_id,
namespace: input.namespace,
tableName: input.table_name,
tableFormat: input.table_format ?? "iceberg", // ADR-0030: Iceberg is canonical; Delta/Hudi accepted only for bridge/import scenarios, never chosen by default here.
storageUri: input.storage_uri,
catalogRef: input.catalog_ref,
datasetId: input.dataset_id,
ownerProduct: "0pt",
createdBy: input.actor_id,
});
// create or track lakehouse table state on its own. NOTE: `datasetId` is
// deliberately NOT sent -- it is a foreign key into ST0RE's OWN
// dataset/artifact table (live-verified: sending 0pt's `dataset_id` here
// 400s with a Postgres FK violation, since 0pt datasets are never rows in
// ST0RE's dataset table). 0pt's dataset<->table linkage is carried
// entirely by the DatasetIcebergMapping row created below.
const table = await this.st0reLakehouse.registerTable(
{
tenantId: input.tenant_id,
projectId: input.project_id,
namespace: input.namespace,
tableName: input.table_name,
tableFormat: input.table_format ?? "iceberg", // ADR-0030: Iceberg is canonical; Delta/Hudi accepted only for bridge/import scenarios, never chosen by default here.
storageUri: input.storage_uri,
catalogRef: input.catalog_ref,
ownerProduct: "0pt",
createdBy: input.actor_id,
},
ctx
);
const mapping = await this.mappingService.createMapping({
dataset_iceberg_mapping_id: input.dataset_iceberg_mapping_id,
@@ -147,6 +164,7 @@ export class IcebergWriteService {
project_id: input.project_id,
dataset_id: input.dataset_id,
iceberg_table_id: table.tableId,
namespace_id: namespace.namespaceId,
namespace: input.namespace,
table_name: input.table_name,
mapping_status: "active",
@@ -154,7 +172,7 @@ export class IcebergWriteService {
correlation_id: input.correlation_id,
});
return { mapping, iceberg_table_id: table.tableId, policy_decision_id: policy.decision_id };
return { mapping, iceberg_table_id: table.tableId, namespace_id: namespace.namespaceId, policy_decision_id: policy.decision_id };
}
async requestIcebergWrite(input: RequestIcebergWriteInput): Promise<RequestIcebergWriteResult> {
@@ -222,15 +240,19 @@ export class IcebergWriteService {
// Register the snapshot THROUGH ST0RE's table registry -- the one and
// only place 0pt persists that a write landed.
const snapshot = await this.st0reLakehouse.createSnapshot(mapping.iceberg_table_id, {
snapshotType: input.snapshot_type ?? "append",
schemaVersion: input.schema_version,
manifestUri: input.manifest_uri,
createdByJobId: n0dJobId,
createdByCapability: "0pt.write_iceberg_table",
recordCount: input.record_count,
sizeBytes: input.size_bytes,
});
const snapshot = await this.st0reLakehouse.createSnapshot(
mapping.iceberg_table_id,
{
snapshotType: input.snapshot_type ?? "append",
schemaVersion: input.schema_version,
manifestUri: input.manifest_uri,
createdByJobId: n0dJobId,
createdByCapability: "0pt.write_iceberg_table",
recordCount: input.record_count,
sizeBytes: input.size_bytes,
},
{ tenantId: mapping.tenant_id, actorId: input.actor_id }
);
// 0pt.iceberg_snapshot_created -- fired once the write lands and ST0RE
// has registered a new snapshot, carrying table ref + snapshot id + row
@@ -281,4 +303,31 @@ export class IcebergWriteService {
n0d_job_id: n0dJobId,
};
}
/**
* Looks up an existing ST0RE lakehouse namespace by (tenantId, projectId,
* name) before creating one. ST0RE has no upsert route and enforces a real
* unique constraint on that triple (live-verified 2026-07-13: a second
* `registerNamespace` call for the same triple 400s with
* `duplicate key value violates unique constraint
* "st0re_lakehouse_namespaces_tenant_id_project_id_name_key"`), so a bare
* "always create" call is not idempotent and would break every
* `mapDatasetToIcebergTable` call after the first for a given namespace.
*
* ST0RE's list route does not reliably scope or filter results (see the
* `listNamespaces` doc comment in st0re-lakehouse-client.ts, live-verified
* finding #4) so the match is done here, client-side, against the full
* returned list.
*/
private async findOrRegisterNamespace(
tenantId: string,
projectId: string,
name: string,
ctx: { tenantId: string; actorId: string }
) {
const existing = await this.st0reLakehouse.listNamespaces(ctx);
const match = existing.find((n) => n.tenantId === tenantId && n.projectId === projectId && n.name === name);
if (match) return match;
return this.st0reLakehouse.registerNamespace({ tenantId, projectId, name, ownerProduct: "0pt" }, ctx);
}
}
+12 -1
View File
@@ -17,10 +17,20 @@ export interface DatasetIcebergMapping {
dataset_id: string;
/** Reference into ST0RE's table registry (IcebergTable.tableId). Not resolved or duplicated locally. */
iceberg_table_id: string;
/** ST0RE lakehouse namespace the table lives in, carried for convenience/filtering only -- ST0RE remains the source of truth. */
/**
* Reference into ST0RE's namespace registry (LakehouseNamespace.namespaceId,
* `/v1/st0re/lakehouse/namespaces`) -- the literal field this task's contract
* spec names. Populated from the real namespaceId ST0RE returns when
* `IcebergWriteService.mapDatasetToIcebergTable` registers the namespace,
* not guessed or derived from the `namespace` name string below.
*/
namespace_id?: string;
/** ST0RE lakehouse namespace NAME the table lives in, carried for convenience/filtering only -- ST0RE remains the source of truth. */
namespace?: string;
table_name?: string;
mapping_status: DatasetIcebergMappingStatus;
/** When this mapping was created/last (re)mapped -- same instant as created_at on first creation; the task spec's literal field name. */
mapped_at: string;
mapped_by?: string;
correlation_id?: string;
created_at: string;
@@ -33,6 +43,7 @@ export interface CreateDatasetIcebergMappingInput {
project_id: string;
dataset_id: string;
iceberg_table_id: string;
namespace_id?: string;
namespace?: string;
table_name?: string;
mapping_status?: DatasetIcebergMappingStatus;
@@ -0,0 +1,120 @@
import { describe, it, expect, beforeAll } from "vitest";
import { DatasetService } from "@/src/services/dataset-service";
import { DatasetIcebergMappingService } from "@/src/services/dataset-iceberg-mapping-service";
import { IcebergWriteService } from "@/src/services/iceberg-write-service";
/**
* LIVE ST0RE integration -- the SEAM pattern this session's task spec asked
* for: "attempt the real call, degrade honestly with a clear status if ST0RE
* is not reachable, never fabricate success" (same convention this same
* session verified in api0's clients, and the same convention ST0RE's own
* `integration-realinfra.test.ts` uses for its DATABASE_URL/MINIO_* gate).
*
* This is NOT a mock-client test -- it constructs `IcebergWriteService` with
* a real `st0reBaseUrl` and drives the actual `HttpSt0reLakehouseClient`
* HTTP path against production ST0RE (api.st0re.siax.io, currently reachable
* from this environment -- DNS for that name resolves here even though
* Cloudflare's public A record is still pending per this session's known
* infra state; set ST0RE_URL to override).
*
* Real writes this test produces land in ST0RE's live Postgres under a
* dedicated pilot tenant (`t_0pt_realinfra_pilot` by default, overridable via
* ST0RE_TEST_TENANT_ID) -- that IS the point of task item 3 ("write ONE
* first test dataset to an Iceberg table through ST0RE's table registry").
*
* Honest degrade: a `beforeAll` reachability probe (5s timeout) decides
* whether to run for real or `ctx.skip()` with a logged reason. This test
* never asserts success without having actually made the HTTP calls, and
* never falls back to the mock client silently.
*/
const ST0RE_URL = process.env.ST0RE_URL ?? "https://api.st0re.siax.io";
const TEST_TENANT = process.env.ST0RE_TEST_TENANT_ID ?? "t_0pt_realinfra_pilot";
const TEST_PROJECT = "p_0pt_realinfra";
const TEST_ACTOR = "0pt-realinfra-test";
let reachable = false;
let unreachableReason = "";
beforeAll(async () => {
try {
const res = await fetch(`${ST0RE_URL}/v1/st0re/lakehouse/namespaces`, {
headers: { "X-Tenant-Id": TEST_TENANT, "X-Actor-Id": TEST_ACTOR },
signal: AbortSignal.timeout(5000),
});
reachable = res.ok;
if (!reachable) unreachableReason = `ST0RE responded HTTP ${res.status}`;
} catch (err) {
reachable = false;
unreachableReason = err instanceof Error ? err.message : String(err);
}
}, 10000);
describe("IcebergWriteService against a LIVE ST0RE deployment", () => {
it(
"mapDatasetToIcebergTable + requestIcebergWrite register a real namespace/table/snapshot in ST0RE",
async (ctx) => {
if (!reachable) {
console.warn(
`[realinfra] ST0RE (${ST0RE_URL}) not reachable this run (${unreachableReason}) -- skipping honestly, not fabricating success.`
);
ctx.skip();
return;
}
const datasetService = new DatasetService();
const mappingService = new DatasetIcebergMappingService({ datasetService });
const service = new IcebergWriteService({ datasetService, mappingService, st0reBaseUrl: ST0RE_URL });
const stamp = Date.now();
const datasetId = `ds_realinfra_pilot_${stamp}`;
await datasetService.registerDataset({
dataset_id: datasetId,
tenant_id: TEST_TENANT,
project_id: TEST_PROJECT,
name: "0pt real-ST0RE pilot dataset",
dataset_type: "enriched",
});
const mapped = await service.mapDatasetToIcebergTable({
tenant_id: TEST_TENANT,
project_id: TEST_PROJECT,
dataset_id: datasetId,
dataset_iceberg_mapping_id: `dim_realinfra_${stamp}`,
namespace: "0pt_pilot",
table_name: `dim_company_se_pilot_${stamp}`,
storage_uri: `s3://siax-tenants/prod/tenants/${TEST_TENANT}/projects/${TEST_PROJECT}/lakehouse/0pt_pilot/dim_company_se_pilot_${stamp}`,
catalog_ref: `0pt_pilot.dim_company_se_pilot_${stamp}`,
actor_id: TEST_ACTOR,
correlation_id: `corr_realinfra_${stamp}`,
});
// Real ST0RE issues Postgres UUIDs -- never the mock client's
// "mock_table_<timestamp>" shape. This is the assertion that proves
// this test actually hit the live HTTP path, not the mock fallback.
expect(mapped.iceberg_table_id).not.toContain("mock_table_");
expect(mapped.iceberg_table_id).toMatch(/^[0-9a-f-]{36}$/i);
expect(mapped.namespace_id).not.toContain("mock_ns_");
expect(mapped.namespace_id).toMatch(/^[0-9a-f-]{36}$/i);
expect(mapped.mapping.dataset_id).toBe(datasetId);
// eslint-disable-next-line no-console
console.log(`[realinfra] LIVE ST0RE table registered: tableId=${mapped.iceberg_table_id} namespaceId=${mapped.namespace_id}`);
const write = await service.requestIcebergWrite({
dataset_iceberg_mapping_id: mapped.mapping.dataset_iceberg_mapping_id,
actor_id: TEST_ACTOR,
correlation_id: `corr_realinfra_write_${stamp}`,
schema_version: "1",
manifest_uri: `s3://siax-tenants/prod/tenants/${TEST_TENANT}/projects/${TEST_PROJECT}/lakehouse/0pt_pilot/dim_company_se_pilot_${stamp}/metadata/manifest-1.avro`,
record_count: 1,
size_bytes: 128,
});
expect(write.snapshot_id).not.toContain("mock_snapshot_");
expect(write.snapshot_id).toMatch(/^[0-9a-f-]{36}$/i);
expect(write.iceberg_table_id).toBe(mapped.iceberg_table_id);
// eslint-disable-next-line no-console
console.log(`[realinfra] LIVE ST0RE snapshot registered: snapshotId=${write.snapshot_id}`);
},
20000
);
});
+5 -9
View File
@@ -165,13 +165,9 @@ describe("IcebergWriteService", () => {
).rejects.toThrow("is not active");
});
// TODO(future session): once ST0RE_URL points at a live, reachable ST0RE
// deployment (sax3l/st0re, /v1/st0re/lakehouse/*, port 8090 per that repo's
// STATUS.md), add a live-integration test here that constructs
// IcebergWriteService with { st0reBaseUrl: process.env.ST0RE_URL } and
// asserts the same flow against the real HTTP API. Not built in this pass
// -- no reachable ST0RE endpoint was available to verify against. See
// src/clients/st0re-lakehouse-client.ts's module doc comment and
// docs/ICEBERG_WRITE_BOUNDARY.md "Verification status" section.
it.todo("mapDatasetToIcebergTable against a LIVE ST0RE deployment (ST0RE_URL) -- not live-verified in this pass");
// The live-ST0RE case this `it.todo` used to name is now built:
// tests/p0/iceberg-write-service.realinfra.test.ts attempts a real HTTP
// call against api.st0re.siax.io (SEAM pattern: real call, honest skip if
// unreachable -- see that file's header comment and
// docs/ICEBERG_WRITE_BOUNDARY.md "Live verification" section).
});