0pt/docs/11-lakehouse-write-plan.md

11 · Lakehouse write plan (0pt's to-do list)

Concrete follow-up to docs/10-lakehouse-decision.md. This is a planning doc for a future implementation session — it names the schemas/events precisely so that session doesn't have to guess, but it does not build the full pipeline today.

Correction (documentation-accuracy pass). This doc previously stated that item 4 (quality rules staying in 0pt) was "already true today" and, in the update section below, that src/services/quality-check-service.ts and src/services/quality-report-service.ts were real, tested files that a later pass left "untouched." That was fabricated. git log --all for src/services/quality-check-service.ts, src/services/quality-report-service.ts, src/types/quality-check.ts, src/types/quality-report.ts, sql/002_p0_dataset_quality.sql, and sql/003_*.sql returns zero commits, on any branch (main, feature/0pt-env-example, ci/add-quality-gate), ever — these files have never existed in this repository. 0pt's data-quality/QA stage is a design spec only (docs/06-data-engine.md §QA — Great Expectations-style checks, intended to run inside the Python worker pipeline in services/api) and is not yet implemented as any dedicated service in this repo, TypeScript or otherwise. This is a real, currently open build gap, not a doc typo — see item 4's corrected entry below and docs/ICEBERG_WRITE_BOUNDARY.md's "Data quality: not yet built" section for the full account. Everything else on this page (items 1, 2, 3, 5, 6, 7, 8) was independently re-verified against the actual service/client/test files this pass and remains accurate.

2026-07-13 update: live verification + field-level gap closure

A later pass live-verified the write path against api.st0re.siax.io (real namespace/table/snapshot rows created), added the namespace_id/mapped_at fields the task spec names literally, and fixed three real ST0RE contract bugs the code below had (missing tenant/actor headers, an invalid datasetId FK reference, and a missing lookup-or-create for namespaces). Full detail in docs/ICEBERG_WRITE_BOUNDARY.md's "2026-07-13 update" section -- read that before this section below, which is the original (2026-07-12) status record and is otherwise unchanged.

2026-07-12 update: what the 0pt write-boundary implementation pass closed

This pass turned most items below into real, tested code. Full detail in docs/ICEBERG_WRITE_BOUNDARY.md; short version (re-verified this documentation-accuracy pass against the actual files):

• Item 2 (0pt.dataset_iceberg_mapping): CLOSED as real code -- src/types/dataset-iceberg-mapping.ts, src/services/dataset-iceberg-mapping-service.ts, packages/schemas/0pt.dataset_iceberg_mapping.schema.json, sql/004_iceberg_write_boundary.sql (opt_dataset_iceberg_mappings), tested in tests/p0/dataset-iceberg-mapping.test.ts. Field names differ slightly from the "suggested shape" above (dataset_iceberg_mapping_id as PK, iceberg_table_id instead of iceberg_table_ref, mapping_status instead of bare entity_type/market) -- reconciled against ST0RE's REAL shipped IcebergTable.tableId shape (sax3l/st0re, not the "future" registry this doc assumed) and against this session's own task spec; see the boundary doc for the full reconciliation note.
• Item 3 (first test dataset write): CLOSED against ST0RE's now-real (but not live-verified from this environment) /v1/st0re/lakehouse/* API -- src/clients/st0re-lakehouse-client.ts, src/services/iceberg-write-service.ts, proven in tests/p0/iceberg-write-service.test.ts against that client's mock implementation (same mock/HTTP convention as cl0ud-client.ts etc). NOT verified against a running ST0RE deployment in this pass -- see the boundary doc's "Verification status" section and the it.todo in that test file.
• Item 4 (quality rules stay in 0pt): NOT BUILT. src/services/quality-check-service.ts and src/services/quality-report-service.ts do not exist and never have (see the correction callout above) -- there was nothing for this pass to leave "untouched." The intent (0pt owns QA logic, N0D never does) is still the correct target and nothing since has contradicted it, but it is not yet backed by any code. This is an open gap for a future implementation session, not a closed item.
• Item 5 (0pt.source_rate_policy): CLOSED as real code -- src/types/source-rate-policy.ts, src/services/source-rate-policy-service.ts, packages/schemas/0pt.source_rate_policy.schema.json, sql/004_iceberg_write_boundary.sql (opt_source_rate_policies), tested in tests/p0/source-rate-policy.test.ts. Kept requests_per_second-style fields (as domain/ robots_ok/access) as OPTIONAL alongside the task-spec's primary rate_limit_per_minute/ crawl_budget/compliance_notes fields -- both specs for this same contract are satisfied on one row, not two competing shapes.
• Item 6 (four AUD0 events): CLOSED for all four names -- 0pt.dataset_iceberg_table_mapped and 0pt.iceberg_write_requested/0pt.iceberg_snapshot_created emit from src/services/iceberg-write-service.ts and dataset-iceberg-mapping-service.ts; 0pt.source_rate_policy_created emits from src/services/source-rate-policy-service.ts. All via the existing Aud0Client (mock/no-op by default, same as every other AUD0 emission in this repo -- AUD0 itself still has no live ingestion endpoint).
• Item 7 (N0D heavy-write requests): CLOSED as real code -- IcebergWriteService.requestIcebergWrite requests an N0D job (0pt.iceberg_write_job) only when heavy: true, never executes in-process; tested in tests/p0/iceberg-write-service.test.ts.
• Item 8 (no lakehouse state outside ST0RE): CLOSED as a design property, not just a statement -- IcebergWriteService and DatasetIcebergMappingService hold zero table/snapshot fields; the only thing 0pt persists is the mapping's iceberg_table_id reference. See the boundary doc.
• Item 1 (Iceberg canonical, Delta rejected): CONFIRMED, unchanged from docs/10 -- IcebergWriteService.mapDatasetToIcebergTable defaults table_format to "iceberg".

To-do list

1. Use Apache Iceberg as the canonical write format for datasets 0pt produces, per docs/10-lakehouse-decision.md / ADR-0030 (n0d repo). Do not adopt Delta Lake as a native write target; Delta is import/export-bridge-only, ecosystem-wide.

2. Add a dataset_to_iceberg_table mapping. Every 0pt dataset (today: rows keyed by entity_type + market in dim_company / fact_financials / dim_contact / dim_ubo / dim_vehicle, per docs/03-data-model.md, or a catalog product / order delivery) needs a stable mapping to the Iceberg table(s) it is written to. Schema name: 0pt.dataset_iceberg_mapping.schema.v1. Suggested shape (to be finalized by the implementation session, following the snake_case / uuid PK / created_at+updated_at conventions in docs/03-data-model.md):

   • dataset_id — 0pt-side identifier (catalog product id, or order_id+item_id for custom orders).
   • iceberg_table_ref — fully-qualified table identifier as registered in ST0RE's table registry (catalog.namespace.table).
   • entity_type, market — for routing/filtering.
   • mapped_at, mapped_by.

3. Write a first test dataset to Iceberg through ST0RE's table registry. (Status: CLOSED per the update above — tests/p0/iceberg-write-service.test.ts exercises exactly this pilot flow, but only against St0reLakehouseClient's mock; not yet run against a live ST0RE. Treat this item as "code exists," not "validated end-to-end against a real deployment.") Original plan: pick a small, low-risk dataset (e.g. a single-market dim_company slice) as the pilot, to validate the write path end-to-end before any production order relies on it — that live-ST0RE validation still has not happened.

4. Keep data quality rules in 0pt. This is the correct target per the n0d audit, and remains a spec-level design in docs/06-data-engine.md §QA (Great Expectations-style checks for validity, completeness, freshness, uniqueness, corroboration), but it is not built — there is no dedicated quality-check/quality-report service in this repo as of this writing (see the correction callout at the top of this doc). Once it is built, the design intent stands: quality logic must be caller-supplied logic that 0pt owns and hands to whichever engine executes a job; it must not become logic owned or embedded by N0D, even after N0D executes heavy writes/compactions on 0pt's behalf (see item 7). This item stays open until that service exists and is tested.

5. Keep source-rate/crawl-compliance policy logic in 0pt — build a real model here, do not rely on N0D for this. (Status: CLOSED per the update above — SourceRatePolicyService is real and tested.) Context: N0D's ComputePolicy.sourceRatePolicy (per-domain crawl rate limits) has been deprecated in N0D as part of the same audit (the field is still present on N0D's type for backward compatibility, but N0D's own boundary-repair tests assert it now has zero effect on any allow/deny decision — verified by reading n0d's src/policy.ts and src/__tests__/boundary-repair.test.ts) — that field never belonged to N0D. 0pt already has the raw material for this in sources.yaml (docs/06-data-engine.md, the rate_limit / robots_ok fields per source), but that's currently free-text/per-entry, not a first-class policy object. Formalize it as 0pt.source_rate_policy.schema.v1. Suggested shape, following the same conventions:

   • source_id — references dim_source(source_id) (docs/03-data-model.md).
   • domain — the crawl-target domain (e.g. allabolag.se).
   • requests_per_second / requests_per_minute — the compliance-derived rate limit.
   • robots_ok — whether robots.txt permits the access pattern.
   • access — mirrors dim_source.access (api|bulk|authorized_web|licensed).
   • created_at, updated_at, created_by. This model belongs entirely to 0pt's sourcing/compliance domain (docs/06) — it must not be reintroduced into N0D, CL0UD, or any other product's policy defaults.

6. Emit lineage for dataset → Iceberg table → snapshot, to AUD0. (Status: CLOSED per the update above — all four events below are wired and fire through the existing Aud0Client, which is mock/no-op by default because AUD0 still has no live ingestion endpoint. This did not mean standing up a bespoke lineage sink prematurely — it means calling the same shared, already-mocked AUD0 client interface every other 0pt event in this repo already calls.) Plan for three events, emitted by 0pt at the corresponding step:

   • 0pt.dataset_iceberg_table_mapped — fired when step 2's mapping is created/updated.
   • 0pt.iceberg_write_requested — fired when 0pt requests a write of a produced dataset into its mapped Iceberg table (whether executed by 0pt directly or via an N0D job, per item 7).
   • 0pt.iceberg_snapshot_created — fired once the write lands and a new Iceberg snapshot exists for the table, carrying enough detail (table ref, snapshot id, row count, dataset id, order id if applicable) for AUD0 to build lineage from source → dataset → Iceberg snapshot.
   • 0pt.source_rate_policy_created — fired when a new 0pt.source_rate_policy.schema.v1 record is created (or materially updated), so AUD0/CL0UD have an evented trail of 0pt's crawl-compliance posture without needing to poll 0pt's DB.

7. Request N0D jobs for heavy writes/compactions if/when needed, rather than 0pt executing them itself. (Status: CLOSED per the update above — IcebergWriteService.requestIcebergWrite requests an N0D job only when heavy: true, tested.) 0pt produces and owns the dataset; if a write or compaction is heavy enough to warrant dedicated compute (large backfills, table maintenance), 0pt should submit that as a job request to N0D's execution layer rather than running it in-process in services/api workers. N0D executes; it does not own the data, the schema, or the decision of what to write.

8. Do not persist lakehouse table state outside ST0RE. (Status: CLOSED as a design property per the update above — IcebergWriteService/DatasetIcebergMappingService hold zero table or snapshot state; the only thing 0pt persists is a reference iceberg_table_id.) ST0RE's table registry now exists as real code (not yet confirmed live/reachable from this repo). 0pt must not stand up a parallel catalog/metadata store of its own (that would repeat the exact mistake this plan is repairing: N0D building its own Iceberg catalog because nothing else had one yet).

Required schema names (for the future implementation session)

• 0pt.dataset_iceberg_mapping.schema.v1
• 0pt.source_rate_policy.schema.v1

Required future event names (for the future implementation session)

• 0pt.dataset_iceberg_table_mapped
• 0pt.iceberg_write_requested
• 0pt.iceberg_snapshot_created
• 0pt.source_rate_policy_created

Sequencing / dependencies

(Status note: this section described the original plan's dependencies. As of the 2026-07-12 update above, items 1, 2, 3, 5, 6, 7, 8 are built — mock-tested, not live-verified. Item 4 (data-quality/QA service) is the one item that remains genuinely not-started; see the correction callout at the top of this doc. The dependency shape below is kept for the record and because "mock-tested" is not the same as "the dependency is resolved" — see each item's status note above for what's still actually pending.)

• Items 2, 4, 5 have no hard dependency on another product and can be defined/built inside 0pt — 2 and 5 now are; 4 (the QA service) still is not.
• Items 1, 3, 8 depended on ST0RE's table registry existing. It now exists as real code in sax3l/st0re; what remains outstanding is a live, reachable ST0RE_URL to verify against — no code path in this repo reads ST0RE_URL from the environment automatically (same as CL0UD_URL/N0D_URL/AUD0_URL — none of the platform clients have an env fallback wired; a base URL must be passed in explicitly via each service's constructor options). Wiring ST0RE_URL alone would not be enough, either: sax3l/st0re's current main enforces X-Tenant-Id/ X-Actor-Id headers on the getTable/createSnapshot routes this write path calls, and this repo's St0reLakehouseClient does not send them yet — see docs/ICEBERG_WRITE_BOUNDARY.md's "Real code, not live-verified" section for the full account (found during this verification pass).
• Item 6 depended on AUD0 existing as a real evented product; it still does not have a live ingestion endpoint, so the four events fire through Aud0Client's mock/no-op path, same as every other AUD0 emission in this repo.
• Item 7 assumes N0D's job-request surface (see n0d repo) is reachable from 0pt's orchestrator; IcebergWriteService calls N0dClient.createJob, which — like the other platform clients — has no live N0D_URL wired from this repo's environment either.

Explicitly out of scope for 0pt (do not build here)

• An Iceberg catalog implementation, or any direct MinIO/Iceberg client code, ahead of ST0RE's registry existing — that repeats the n0d mistake this plan is repairing.
• Query, compaction, or table-maintenance execution — that is N0D's job.
• A lineage UI or lineage storage — that is AUD0's job; 0pt only emits the events above.