Files
0pt/docs/11-lakehouse-write-plan.md
T
adminandClaude Sonnet 5 bda21e87eb Live-verify Iceberg write path against ST0RE + close field/contract gaps
Re-runs this session's task spec against 0pt's actual repo state, which
turned out to already have a substantial, tested implementation from a
prior pass (35b6ec6): dataset<->Iceberg-table mapping, source rate policy,
and an ST0RE lakehouse client that was honestly marked "not live-verified".

This pass found api.st0re.siax.io IS reachable from this environment and
used it -- first via curl to safely discover the real contract, then
through 0pt's own updated code -- to drive a real namespace -> table ->
snapshot write against production ST0RE (task item 3's "write ONE first
test dataset" proof). That surfaced and fixed three real bugs the
mock-only pass could not have caught:

- ST0RE requires X-Tenant-Id/X-Actor-Id headers on every lakehouse route;
  the previous HTTP client sent none. Fixed via a threaded
  St0reRequestContext.
- IcebergTableInput.datasetId is a foreign key into ST0RE's OWN dataset
  table, not a free-form string -- sending 0pt's dataset_id 400s live with
  a Postgres FK violation. Fixed by no longer sending it; 0pt's
  dataset<->table linkage lives entirely in DatasetIcebergMapping.
- ST0RE enforces a real unique constraint on lakehouse namespaces
  (tenant_id, project_id, name) with no upsert route. Fixed with a
  client-side lookup-or-create (listNamespaces + findOrRegisterNamespace).

Also closed a field-level gap against the task's literal schema spec:
0pt.dataset_iceberg_mapping.schema.v1 now carries namespace_id (a real
reference to ST0RE's LakehouseNamespace.namespaceId, not guessed) and
mapped_at, both additive alongside the existing namespace/created_at
fields (sql/005, JSON schema, types, service).

tests/p0/iceberg-write-service.realinfra.test.ts replaces the prior
it.todo with a real SEAM-pattern live test: attempts the real HTTP call,
skips honestly if ST0RE is unreachable, never fabricates success. It ran
successfully against production ST0RE this session (real namespace/table/
snapshot UUIDs logged in docs/ICEBERG_WRITE_BOUNDARY.md).

Two ST0RE-side cross-tenant read bugs were found and flagged (not fixed
here -- out of scope for this repo): GET .../namespaces ignores its own
tenant header and query filters; GET .../tables/:id doesn't check the
caller's tenant against the table's owner.

npm test: 155 passed, 0 failed, 0 todo (up from 154 passed + 1 todo).
npx tsc --noEmit: 1 pre-existing, unrelated error in
quality-report-service.ts (present before this pass, part of the wider
uncommitted P0 data-platform build already in this repo) -- not touched.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-13 00:18:10 +02:00

11 KiB

11 · Lakehouse write plan (0pt's to-do list)

Concrete follow-up to docs/10-lakehouse-decision.md. This is a planning doc for a future implementation session — it names the schemas/events precisely so that session doesn't have to guess, but it does not build the full pipeline today. Some items are already true today (quality rules) and this doc just confirms 0pt intends to keep them that way.

2026-07-13 update: live verification + field-level gap closure

A later pass live-verified the write path against api.st0re.siax.io (real namespace/table/snapshot rows created), added the namespace_id/mapped_at fields the task spec names literally, and fixed three real ST0RE contract bugs the code below had (missing tenant/actor headers, an invalid datasetId FK reference, and a missing lookup-or-create for namespaces). Full detail in docs/ICEBERG_WRITE_BOUNDARY.md's "2026-07-13 update" section -- read that before this section below, which is the original (2026-07-12) status record and is otherwise unchanged.

2026-07-12 update: what the 0pt write-boundary implementation pass closed

This pass (the "future implementation session" this doc was written for) turned most items below into real, tested code. Full detail in docs/ICEBERG_WRITE_BOUNDARY.md; short version:

  • Item 2 (0pt.dataset_iceberg_mapping): CLOSED as real code -- src/types/dataset-iceberg-mapping.ts, src/services/dataset-iceberg-mapping-service.ts, packages/schemas/0pt.dataset_iceberg_mapping.schema.json, sql/004_iceberg_write_boundary.sql (opt_dataset_iceberg_mappings), tested in tests/p0/dataset-iceberg-mapping.test.ts. Field names differ slightly from the "suggested shape" above (dataset_iceberg_mapping_id as PK, iceberg_table_id instead of iceberg_table_ref, mapping_status instead of bare entity_type/market) -- reconciled against ST0RE's REAL shipped IcebergTable.tableId shape (sax3l/st0re, not the "future" registry this doc assumed) and against this session's own task spec; see the boundary doc for the full reconciliation note.
  • Item 3 (first test dataset write): CLOSED against ST0RE's now-real (but not live-verified from this environment) /v1/st0re/lakehouse/* API -- src/clients/st0re-lakehouse-client.ts, src/services/iceberg-write-service.ts, proven in tests/p0/iceberg-write-service.test.ts against that client's mock implementation (same mock/HTTP convention as cl0ud-client.ts etc). NOT verified against a running ST0RE deployment in this pass -- see the boundary doc's "Verification status" section and the it.todo in that test file.
  • Item 4 (quality rules stay in 0pt): CONFIRMED, untouched -- src/services/quality-check-service.ts and src/services/quality-report-service.ts were not modified by this pass.
  • Item 5 (0pt.source_rate_policy): CLOSED as real code -- src/types/source-rate-policy.ts, src/services/source-rate-policy-service.ts, packages/schemas/0pt.source_rate_policy.schema.json, sql/004_iceberg_write_boundary.sql (opt_source_rate_policies), tested in tests/p0/source-rate-policy.test.ts. Kept requests_per_second-style fields (as domain/ robots_ok/access) as OPTIONAL alongside the task-spec's primary rate_limit_per_minute/ crawl_budget/compliance_notes fields -- both specs for this same contract are satisfied on one row, not two competing shapes.
  • Item 6 (four AUD0 events): CLOSED for all four names -- 0pt.dataset_iceberg_table_mapped and 0pt.iceberg_write_requested/0pt.iceberg_snapshot_created emit from src/services/iceberg-write-service.ts and dataset-iceberg-mapping-service.ts; 0pt.source_rate_policy_created emits from src/services/source-rate-policy-service.ts. All via the existing Aud0Client (mock/no-op by default, same as every other AUD0 emission in this repo -- AUD0 itself still has no live ingestion endpoint).
  • Item 7 (N0D heavy-write requests): CLOSED as real code -- IcebergWriteService.requestIcebergWrite requests an N0D job (0pt.iceberg_write_job) only when heavy: true, never executes in-process; tested in tests/p0/iceberg-write-service.test.ts.
  • Item 8 (no lakehouse state outside ST0RE): CLOSED as a design property, not just a statement -- IcebergWriteService and DatasetIcebergMappingService hold zero table/snapshot fields; the only thing 0pt persists is the mapping's iceberg_table_id reference. See the boundary doc.
  • Item 1 (Iceberg canonical, Delta rejected): CONFIRMED, unchanged from docs/10 -- IcebergWriteService.mapDatasetToIcebergTable defaults table_format to "iceberg".

To-do list

  1. Use Apache Iceberg as the canonical write format for datasets 0pt produces, per docs/10-lakehouse-decision.md / ADR-0030 (n0d repo). Do not adopt Delta Lake as a native write target; Delta is import/export-bridge-only, ecosystem-wide.

  2. Add a dataset_to_iceberg_table mapping. Every 0pt dataset (today: rows keyed by entity_type + market in dim_company / fact_financials / dim_contact / dim_ubo / dim_vehicle, per docs/03-data-model.md, or a catalog product / order delivery) needs a stable mapping to the Iceberg table(s) it is written to. Schema name: 0pt.dataset_iceberg_mapping.schema.v1. Suggested shape (to be finalized by the implementation session, following the snake_case / uuid PK / created_at+updated_at conventions in docs/03-data-model.md):

    • dataset_id — 0pt-side identifier (catalog product id, or order_id+item_id for custom orders).
    • iceberg_table_ref — fully-qualified table identifier as registered in ST0RE's table registry (catalog.namespace.table).
    • entity_type, market — for routing/filtering.
    • mapped_at, mapped_by.
  3. Write a first test dataset to Iceberg through ST0RE's (future) table registry, once that registry exists. Pick a small, low-risk dataset (e.g. a single-market dim_company slice) as the pilot. This validates the write path end-to-end before any production order relies on it.

  4. Keep data quality rules in 0pt. This is already correct today (per the n0d audit): 0pt's QA stage (docs/06-data-engine.md §QA — Great Expectations checks for validity, completeness, freshness, uniqueness, corroboration) is caller-supplied logic that 0pt owns and hands to whichever engine executes a job; it is not, and must not become, logic owned or embedded by N0D. This doc confirms 0pt intends to keep authoring/owning quality rules itself even after N0D executes heavy writes/compactions on 0pt's behalf (see item 7).

  5. Keep source-rate/crawl-compliance policy logic in 0pt — build a real model here, do not rely on N0D for this. Context: N0D's ComputePolicy.sourceRatePolicy (per-domain crawl rate limits) has been deprecated/removed from N0D's seed data as part of the same audit — that field never belonged to N0D. 0pt already has the raw material for this in sources.yaml (docs/06-data-engine.md, the rate_limit / robots_ok fields per source), but that's currently free-text/per-entry, not a first-class policy object. Formalize it as 0pt.source_rate_policy.schema.v1. Suggested shape, following the same conventions:

    • source_id — references dim_source(source_id) (docs/03-data-model.md).
    • domain — the crawl-target domain (e.g. allabolag.se).
    • requests_per_second / requests_per_minute — the compliance-derived rate limit.
    • robots_ok — whether robots.txt permits the access pattern.
    • access — mirrors dim_source.access (api|bulk|authorized_web|licensed).
    • created_at, updated_at, created_by. This model belongs entirely to 0pt's sourcing/compliance domain (docs/06) — it must not be reintroduced into N0D, CL0UD, or any other product's policy defaults.
  6. Emit lineage for dataset → Iceberg table → snapshot, to AUD0, once AUD0 exists as a real evented product (it does not yet — do not build a lineage sink against it prematurely). Plan for three events, emitted by 0pt at the corresponding step:

    • 0pt.dataset_iceberg_table_mapped — fired when step 2's mapping is created/updated.
    • 0pt.iceberg_write_requested — fired when 0pt requests a write of a produced dataset into its mapped Iceberg table (whether executed by 0pt directly or via an N0D job, per item 7).
    • 0pt.iceberg_snapshot_created — fired once the write lands and a new Iceberg snapshot exists for the table, carrying enough detail (table ref, snapshot id, row count, dataset id, order id if applicable) for AUD0 to build lineage from source → dataset → Iceberg snapshot.
    • 0pt.source_rate_policy_created — fired when a new 0pt.source_rate_policy.schema.v1 record is created (or materially updated), so AUD0/CL0UD have an evented trail of 0pt's crawl-compliance posture without needing to poll 0pt's DB.
  7. Request N0D jobs for heavy writes/compactions if/when needed, rather than 0pt executing them itself. 0pt produces and owns the dataset; if a write or compaction is heavy enough to warrant dedicated compute (large backfills, table maintenance), 0pt should submit that as a job request to N0D's execution layer rather than running it in-process in services/api workers. N0D executes; it does not own the data, the schema, or the decision of what to write.

  8. Do not persist lakehouse table state outside ST0RE once ST0RE's registry exists. Until then, 0pt has no lakehouse table state to persist at all — Postgres (docs/03) remains the only store. Once ST0RE ships its registry, 0pt must not stand up a parallel catalog/metadata store of its own (that would repeat the exact mistake this plan is repairing: N0D building its own Iceberg catalog because nothing else had one yet).

Required schema names (for the future implementation session)

  • 0pt.dataset_iceberg_mapping.schema.v1
  • 0pt.source_rate_policy.schema.v1

Required future event names (for the future implementation session)

  • 0pt.dataset_iceberg_table_mapped
  • 0pt.iceberg_write_requested
  • 0pt.iceberg_snapshot_created
  • 0pt.source_rate_policy_created

Sequencing / dependencies

  • Items 2, 4, 5 have no hard dependency on another product and can be defined/built inside 0pt today (this repo already owns sources.yaml and the QA stage).
  • Items 1, 3, 8 are blocked on ST0RE's table registry existing.
  • Item 6 is blocked on AUD0 existing as a real evented product.
  • Item 7 assumes N0D's job-request surface (see n0d repo) is reachable from 0pt's orchestrator; no such integration exists yet in services/api and none is being added by this doc.

Explicitly out of scope for 0pt (do not build here)

  • An Iceberg catalog implementation, or any direct MinIO/Iceberg client code, ahead of ST0RE's registry existing — that repeats the n0d mistake this plan is repairing.
  • Query, compaction, or table-maintenance execution — that is N0D's job.
  • A lineage UI or lineage storage — that is AUD0's job; 0pt only emits the events above.