Files
0pt/src/types/dataset-iceberg-mapping.ts
T
adminandClaude Sonnet 5 bda21e87eb Live-verify Iceberg write path against ST0RE + close field/contract gaps
Re-runs this session's task spec against 0pt's actual repo state, which
turned out to already have a substantial, tested implementation from a
prior pass (35b6ec6): dataset<->Iceberg-table mapping, source rate policy,
and an ST0RE lakehouse client that was honestly marked "not live-verified".

This pass found api.st0re.siax.io IS reachable from this environment and
used it -- first via curl to safely discover the real contract, then
through 0pt's own updated code -- to drive a real namespace -> table ->
snapshot write against production ST0RE (task item 3's "write ONE first
test dataset" proof). That surfaced and fixed three real bugs the
mock-only pass could not have caught:

- ST0RE requires X-Tenant-Id/X-Actor-Id headers on every lakehouse route;
  the previous HTTP client sent none. Fixed via a threaded
  St0reRequestContext.
- IcebergTableInput.datasetId is a foreign key into ST0RE's OWN dataset
  table, not a free-form string -- sending 0pt's dataset_id 400s live with
  a Postgres FK violation. Fixed by no longer sending it; 0pt's
  dataset<->table linkage lives entirely in DatasetIcebergMapping.
- ST0RE enforces a real unique constraint on lakehouse namespaces
  (tenant_id, project_id, name) with no upsert route. Fixed with a
  client-side lookup-or-create (listNamespaces + findOrRegisterNamespace).

Also closed a field-level gap against the task's literal schema spec:
0pt.dataset_iceberg_mapping.schema.v1 now carries namespace_id (a real
reference to ST0RE's LakehouseNamespace.namespaceId, not guessed) and
mapped_at, both additive alongside the existing namespace/created_at
fields (sql/005, JSON schema, types, service).

tests/p0/iceberg-write-service.realinfra.test.ts replaces the prior
it.todo with a real SEAM-pattern live test: attempts the real HTTP call,
skips honestly if ST0RE is unreachable, never fabricates success. It ran
successfully against production ST0RE this session (real namespace/table/
snapshot UUIDs logged in docs/ICEBERG_WRITE_BOUNDARY.md).

Two ST0RE-side cross-tenant read bugs were found and flagged (not fixed
here -- out of scope for this repo): GET .../namespaces ignores its own
tenant header and query filters; GET .../tables/:id doesn't check the
caller's tenant against the table's owner.

npm test: 155 passed, 0 failed, 0 todo (up from 154 passed + 1 todo).
npx tsc --noEmit: 1 pre-existing, unrelated error in
quality-report-service.ts (present before this pass, part of the wider
uncommitted P0 data-platform build already in this repo) -- not touched.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-13 00:18:10 +02:00

53 lines
2.1 KiB
TypeScript

// 0pt.dataset_iceberg_mapping.schema.v1
//
// Maps a 0pt dataset to the Iceberg table that carries its canonical lakehouse
// write path. `iceberg_table_id` is a REFERENCE into ST0RE's table registry
// (IcebergTable.tableId, st0re_iceberg_tables) -- it is not a copy of ST0RE's
// table metadata. 0pt does not persist table/catalog/snapshot state of its
// own; this row exists only to answer "which Iceberg table does this dataset
// write to", per docs/10-lakehouse-decision.md and
// docs/ICEBERG_WRITE_BOUNDARY.md.
export type DatasetIcebergMappingStatus = "pending" | "active" | "deprecated" | "archived";
export interface DatasetIcebergMapping {
dataset_iceberg_mapping_id: string;
tenant_id: string;
project_id: string;
dataset_id: string;
/** Reference into ST0RE's table registry (IcebergTable.tableId). Not resolved or duplicated locally. */
iceberg_table_id: string;
/**
* Reference into ST0RE's namespace registry (LakehouseNamespace.namespaceId,
* `/v1/st0re/lakehouse/namespaces`) -- the literal field this task's contract
* spec names. Populated from the real namespaceId ST0RE returns when
* `IcebergWriteService.mapDatasetToIcebergTable` registers the namespace,
* not guessed or derived from the `namespace` name string below.
*/
namespace_id?: string;
/** ST0RE lakehouse namespace NAME the table lives in, carried for convenience/filtering only -- ST0RE remains the source of truth. */
namespace?: string;
table_name?: string;
mapping_status: DatasetIcebergMappingStatus;
/** When this mapping was created/last (re)mapped -- same instant as created_at on first creation; the task spec's literal field name. */
mapped_at: string;
mapped_by?: string;
correlation_id?: string;
created_at: string;
updated_at?: string;
}
export interface CreateDatasetIcebergMappingInput {
dataset_iceberg_mapping_id: string;
tenant_id: string;
project_id: string;
dataset_id: string;
iceberg_table_id: string;
namespace_id?: string;
namespace?: string;
table_name?: string;
mapping_status?: DatasetIcebergMappingStatus;
mapped_by?: string;
correlation_id?: string;
}