Re-runs this session's task spec against 0pt's actual repo state, which
turned out to already have a substantial, tested implementation from a
prior pass (35b6ec6): dataset<->Iceberg-table mapping, source rate policy,
and an ST0RE lakehouse client that was honestly marked "not live-verified".
This pass found api.st0re.siax.io IS reachable from this environment and
used it -- first via curl to safely discover the real contract, then
through 0pt's own updated code -- to drive a real namespace -> table ->
snapshot write against production ST0RE (task item 3's "write ONE first
test dataset" proof). That surfaced and fixed three real bugs the
mock-only pass could not have caught:
- ST0RE requires X-Tenant-Id/X-Actor-Id headers on every lakehouse route;
the previous HTTP client sent none. Fixed via a threaded
St0reRequestContext.
- IcebergTableInput.datasetId is a foreign key into ST0RE's OWN dataset
table, not a free-form string -- sending 0pt's dataset_id 400s live with
a Postgres FK violation. Fixed by no longer sending it; 0pt's
dataset<->table linkage lives entirely in DatasetIcebergMapping.
- ST0RE enforces a real unique constraint on lakehouse namespaces
(tenant_id, project_id, name) with no upsert route. Fixed with a
client-side lookup-or-create (listNamespaces + findOrRegisterNamespace).
Also closed a field-level gap against the task's literal schema spec:
0pt.dataset_iceberg_mapping.schema.v1 now carries namespace_id (a real
reference to ST0RE's LakehouseNamespace.namespaceId, not guessed) and
mapped_at, both additive alongside the existing namespace/created_at
fields (sql/005, JSON schema, types, service).
tests/p0/iceberg-write-service.realinfra.test.ts replaces the prior
it.todo with a real SEAM-pattern live test: attempts the real HTTP call,
skips honestly if ST0RE is unreachable, never fabricates success. It ran
successfully against production ST0RE this session (real namespace/table/
snapshot UUIDs logged in docs/ICEBERG_WRITE_BOUNDARY.md).
Two ST0RE-side cross-tenant read bugs were found and flagged (not fixed
here -- out of scope for this repo): GET .../namespaces ignores its own
tenant header and query filters; GET .../tables/:id doesn't check the
caller's tenant against the table's owner.
npm test: 155 passed, 0 failed, 0 todo (up from 154 passed + 1 todo).
npx tsc --noEmit: 1 pre-existing, unrelated error in
quality-report-service.ts (present before this pass, part of the wider
uncommitted P0 data-platform build already in this repo) -- not touched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
11 KiB
11 · Lakehouse write plan (0pt's to-do list)
Concrete follow-up to docs/10-lakehouse-decision.md. This is a planning doc for a future
implementation session — it names the schemas/events precisely so that session doesn't have to
guess, but it does not build the full pipeline today. Some items are already true today (quality
rules) and this doc just confirms 0pt intends to keep them that way.
2026-07-13 update: live verification + field-level gap closure
A later pass live-verified the write path against api.st0re.siax.io (real namespace/table/snapshot
rows created), added the namespace_id/mapped_at fields the task spec names literally, and fixed
three real ST0RE contract bugs the code below had (missing tenant/actor headers, an invalid
datasetId FK reference, and a missing lookup-or-create for namespaces). Full detail in
docs/ICEBERG_WRITE_BOUNDARY.md's "2026-07-13 update" section -- read that before this section
below, which is the original (2026-07-12) status record and is otherwise unchanged.
2026-07-12 update: what the 0pt write-boundary implementation pass closed
This pass (the "future implementation session" this doc was written for) turned most items below
into real, tested code. Full detail in docs/ICEBERG_WRITE_BOUNDARY.md; short version:
- Item 2 (
0pt.dataset_iceberg_mapping): CLOSED as real code --src/types/dataset-iceberg-mapping.ts,src/services/dataset-iceberg-mapping-service.ts,packages/schemas/0pt.dataset_iceberg_mapping.schema.json,sql/004_iceberg_write_boundary.sql(opt_dataset_iceberg_mappings), tested intests/p0/dataset-iceberg-mapping.test.ts. Field names differ slightly from the "suggested shape" above (dataset_iceberg_mapping_idas PK,iceberg_table_idinstead oficeberg_table_ref,mapping_statusinstead of bareentity_type/market) -- reconciled against ST0RE's REAL shippedIcebergTable.tableIdshape (sax3l/st0re, not the "future" registry this doc assumed) and against this session's own task spec; see the boundary doc for the full reconciliation note. - Item 3 (first test dataset write): CLOSED against ST0RE's now-real (but not live-verified from
this environment)
/v1/st0re/lakehouse/*API --src/clients/st0re-lakehouse-client.ts,src/services/iceberg-write-service.ts, proven intests/p0/iceberg-write-service.test.tsagainst that client's mock implementation (same mock/HTTP convention ascl0ud-client.tsetc). NOT verified against a running ST0RE deployment in this pass -- see the boundary doc's "Verification status" section and theit.todoin that test file. - Item 4 (quality rules stay in 0pt): CONFIRMED, untouched --
src/services/quality-check-service.tsandsrc/services/quality-report-service.tswere not modified by this pass. - Item 5 (
0pt.source_rate_policy): CLOSED as real code --src/types/source-rate-policy.ts,src/services/source-rate-policy-service.ts,packages/schemas/0pt.source_rate_policy.schema.json,sql/004_iceberg_write_boundary.sql(opt_source_rate_policies), tested intests/p0/source-rate-policy.test.ts. Keptrequests_per_second-style fields (asdomain/robots_ok/access) as OPTIONAL alongside the task-spec's primaryrate_limit_per_minute/crawl_budget/compliance_notesfields -- both specs for this same contract are satisfied on one row, not two competing shapes. - Item 6 (four AUD0 events): CLOSED for all four names --
0pt.dataset_iceberg_table_mappedand0pt.iceberg_write_requested/0pt.iceberg_snapshot_createdemit fromsrc/services/iceberg-write-service.tsanddataset-iceberg-mapping-service.ts;0pt.source_rate_policy_createdemits fromsrc/services/source-rate-policy-service.ts. All via the existingAud0Client(mock/no-op by default, same as every other AUD0 emission in this repo -- AUD0 itself still has no live ingestion endpoint). - Item 7 (N0D heavy-write requests): CLOSED as real code --
IcebergWriteService.requestIcebergWriterequests an N0D job (0pt.iceberg_write_job) only whenheavy: true, never executes in-process; tested intests/p0/iceberg-write-service.test.ts. - Item 8 (no lakehouse state outside ST0RE): CLOSED as a design property, not just a statement --
IcebergWriteServiceandDatasetIcebergMappingServicehold zero table/snapshot fields; the only thing 0pt persists is the mapping'siceberg_table_idreference. See the boundary doc. - Item 1 (Iceberg canonical, Delta rejected): CONFIRMED, unchanged from
docs/10--IcebergWriteService.mapDatasetToIcebergTabledefaultstable_formatto"iceberg".
To-do list
-
Use Apache Iceberg as the canonical write format for datasets 0pt produces, per
docs/10-lakehouse-decision.md/ ADR-0030 (n0drepo). Do not adopt Delta Lake as a native write target; Delta is import/export-bridge-only, ecosystem-wide. -
Add a
dataset_to_iceberg_tablemapping. Every 0pt dataset (today: rows keyed byentity_type+ market indim_company/fact_financials/dim_contact/dim_ubo/dim_vehicle, perdocs/03-data-model.md, or a catalog product / order delivery) needs a stable mapping to the Iceberg table(s) it is written to. Schema name:0pt.dataset_iceberg_mapping.schema.v1. Suggested shape (to be finalized by the implementation session, following thesnake_case/uuidPK /created_at+updated_atconventions indocs/03-data-model.md):dataset_id— 0pt-side identifier (catalog product id, ororder_id+item_idfor custom orders).iceberg_table_ref— fully-qualified table identifier as registered in ST0RE's table registry (catalog.namespace.table).entity_type,market— for routing/filtering.mapped_at,mapped_by.
-
Write a first test dataset to Iceberg through ST0RE's (future) table registry, once that registry exists. Pick a small, low-risk dataset (e.g. a single-market
dim_companyslice) as the pilot. This validates the write path end-to-end before any production order relies on it. -
Keep data quality rules in 0pt. This is already correct today (per the
n0daudit): 0pt's QA stage (docs/06-data-engine.md§QA — Great Expectations checks for validity, completeness, freshness, uniqueness, corroboration) is caller-supplied logic that 0pt owns and hands to whichever engine executes a job; it is not, and must not become, logic owned or embedded by N0D. This doc confirms 0pt intends to keep authoring/owning quality rules itself even after N0D executes heavy writes/compactions on 0pt's behalf (see item 7). -
Keep source-rate/crawl-compliance policy logic in 0pt — build a real model here, do not rely on N0D for this. Context: N0D's
ComputePolicy.sourceRatePolicy(per-domain crawl rate limits) has been deprecated/removed from N0D's seed data as part of the same audit — that field never belonged to N0D. 0pt already has the raw material for this insources.yaml(docs/06-data-engine.md, therate_limit/robots_okfields per source), but that's currently free-text/per-entry, not a first-class policy object. Formalize it as0pt.source_rate_policy.schema.v1. Suggested shape, following the same conventions:source_id— referencesdim_source(source_id)(docs/03-data-model.md).domain— the crawl-target domain (e.g.allabolag.se).requests_per_second/requests_per_minute— the compliance-derived rate limit.robots_ok— whetherrobots.txtpermits the access pattern.access— mirrorsdim_source.access(api|bulk|authorized_web|licensed).created_at,updated_at,created_by. This model belongs entirely to 0pt's sourcing/compliance domain (docs/06) — it must not be reintroduced into N0D, CL0UD, or any other product's policy defaults.
-
Emit lineage for dataset → Iceberg table → snapshot, to AUD0, once AUD0 exists as a real evented product (it does not yet — do not build a lineage sink against it prematurely). Plan for three events, emitted by 0pt at the corresponding step:
0pt.dataset_iceberg_table_mapped— fired when step 2's mapping is created/updated.0pt.iceberg_write_requested— fired when 0pt requests a write of a produced dataset into its mapped Iceberg table (whether executed by 0pt directly or via an N0D job, per item 7).0pt.iceberg_snapshot_created— fired once the write lands and a new Iceberg snapshot exists for the table, carrying enough detail (table ref, snapshot id, row count, dataset id, order id if applicable) for AUD0 to build lineage from source → dataset → Iceberg snapshot.0pt.source_rate_policy_created— fired when a new0pt.source_rate_policy.schema.v1record is created (or materially updated), so AUD0/CL0UD have an evented trail of 0pt's crawl-compliance posture without needing to poll 0pt's DB.
-
Request N0D jobs for heavy writes/compactions if/when needed, rather than 0pt executing them itself. 0pt produces and owns the dataset; if a write or compaction is heavy enough to warrant dedicated compute (large backfills, table maintenance), 0pt should submit that as a job request to N0D's execution layer rather than running it in-process in
services/apiworkers. N0D executes; it does not own the data, the schema, or the decision of what to write. -
Do not persist lakehouse table state outside ST0RE once ST0RE's registry exists. Until then, 0pt has no lakehouse table state to persist at all — Postgres (
docs/03) remains the only store. Once ST0RE ships its registry, 0pt must not stand up a parallel catalog/metadata store of its own (that would repeat the exact mistake this plan is repairing: N0D building its own Iceberg catalog because nothing else had one yet).
Required schema names (for the future implementation session)
0pt.dataset_iceberg_mapping.schema.v10pt.source_rate_policy.schema.v1
Required future event names (for the future implementation session)
0pt.dataset_iceberg_table_mapped0pt.iceberg_write_requested0pt.iceberg_snapshot_created0pt.source_rate_policy_created
Sequencing / dependencies
- Items 2, 4, 5 have no hard dependency on another product and can be defined/built inside 0pt
today (this repo already owns
sources.yamland the QA stage). - Items 1, 3, 8 are blocked on ST0RE's table registry existing.
- Item 6 is blocked on AUD0 existing as a real evented product.
- Item 7 assumes N0D's job-request surface (see
n0drepo) is reachable from 0pt's orchestrator; no such integration exists yet inservices/apiand none is being added by this doc.
Explicitly out of scope for 0pt (do not build here)
- An Iceberg catalog implementation, or any direct MinIO/Iceberg client code, ahead of ST0RE's
registry existing — that repeats the
n0dmistake this plan is repairing. - Query, compaction, or table-maintenance execution — that is N0D's job.
- A lineage UI or lineage storage — that is AUD0's job; 0pt only emits the events above.