# 11 · Lakehouse write plan (0pt's to-do list) Concrete follow-up to `docs/10-lakehouse-decision.md`. This is a planning doc for a **future** implementation session — it names the schemas/events precisely so that session doesn't have to guess, but it does not build the full pipeline today. > **Correction (documentation-accuracy pass).** This doc previously stated that item 4 (quality > rules staying in 0pt) was "already true today" and, in the update section below, that > `src/services/quality-check-service.ts` and `src/services/quality-report-service.ts` were real, > tested files that a later pass left "untouched." **That was fabricated.** `git log --all` for > `src/services/quality-check-service.ts`, `src/services/quality-report-service.ts`, > `src/types/quality-check.ts`, `src/types/quality-report.ts`, `sql/002_p0_dataset_quality.sql`, > and `sql/003_*.sql` returns zero commits, on any branch (`main`, `feature/0pt-env-example`, > `ci/add-quality-gate`), ever — these files have never existed in this repository. 0pt's > data-quality/QA stage is a **design spec only** (`docs/06-data-engine.md` §QA — Great > Expectations-style checks, intended to run inside the Python worker pipeline in `services/api`) > and is **not yet implemented** as any dedicated service in this repo, TypeScript or otherwise. > This is a real, currently open build gap, not a doc typo — see item 4's corrected entry below > and `docs/ICEBERG_WRITE_BOUNDARY.md`'s "Data quality: not yet built" section for the full > account. Everything else on this page (items 1, 2, 3, 5, 6, 7, 8) was independently re-verified > against the actual service/client/test files this pass and remains accurate. ## 2026-07-13 update: live verification + field-level gap closure A later pass live-verified the write path against `api.st0re.siax.io` (real namespace/table/snapshot rows created), added the `namespace_id`/`mapped_at` fields the task spec names literally, and fixed three real ST0RE contract bugs the code below had (missing tenant/actor headers, an invalid `datasetId` FK reference, and a missing lookup-or-create for namespaces). Full detail in `docs/ICEBERG_WRITE_BOUNDARY.md`'s "2026-07-13 update" section -- read that before this section below, which is the original (2026-07-12) status record and is otherwise unchanged. ## 2026-07-12 update: what the 0pt write-boundary implementation pass closed This pass turned most items below into real, tested code. Full detail in `docs/ICEBERG_WRITE_BOUNDARY.md`; short version (re-verified this documentation-accuracy pass against the actual files): - Item 2 (`0pt.dataset_iceberg_mapping`): CLOSED as real code -- `src/types/dataset-iceberg-mapping.ts`, `src/services/dataset-iceberg-mapping-service.ts`, `packages/schemas/0pt.dataset_iceberg_mapping.schema.json`, `sql/004_iceberg_write_boundary.sql` (`opt_dataset_iceberg_mappings`), tested in `tests/p0/dataset-iceberg-mapping.test.ts`. Field names differ slightly from the "suggested shape" above (`dataset_iceberg_mapping_id` as PK, `iceberg_table_id` instead of `iceberg_table_ref`, `mapping_status` instead of bare `entity_type`/`market`) -- reconciled against ST0RE's REAL shipped `IcebergTable.tableId` shape (sax3l/st0re, not the "future" registry this doc assumed) and against this session's own task spec; see the boundary doc for the full reconciliation note. - Item 3 (first test dataset write): CLOSED against ST0RE's now-real (but not live-verified from this environment) `/v1/st0re/lakehouse/*` API -- `src/clients/st0re-lakehouse-client.ts`, `src/services/iceberg-write-service.ts`, proven in `tests/p0/iceberg-write-service.test.ts` against that client's mock implementation (same mock/HTTP convention as `cl0ud-client.ts` etc). NOT verified against a running ST0RE deployment in this pass -- see the boundary doc's "Verification status" section and the `it.todo` in that test file. - Item 4 (quality rules stay in 0pt): **NOT BUILT.** `src/services/quality-check-service.ts` and `src/services/quality-report-service.ts` do not exist and never have (see the correction callout above) -- there was nothing for this pass to leave "untouched." The *intent* (0pt owns QA logic, N0D never does) is still the correct target and nothing since has contradicted it, but it is not yet backed by any code. This is an open gap for a future implementation session, not a closed item. - Item 5 (`0pt.source_rate_policy`): CLOSED as real code -- `src/types/source-rate-policy.ts`, `src/services/source-rate-policy-service.ts`, `packages/schemas/0pt.source_rate_policy.schema.json`, `sql/004_iceberg_write_boundary.sql` (`opt_source_rate_policies`), tested in `tests/p0/source-rate-policy.test.ts`. Kept `requests_per_second`-style fields (as `domain`/ `robots_ok`/`access`) as OPTIONAL alongside the task-spec's primary `rate_limit_per_minute`/ `crawl_budget`/`compliance_notes` fields -- both specs for this same contract are satisfied on one row, not two competing shapes. - Item 6 (four AUD0 events): CLOSED for all four names -- `0pt.dataset_iceberg_table_mapped` and `0pt.iceberg_write_requested`/`0pt.iceberg_snapshot_created` emit from `src/services/iceberg-write-service.ts` and `dataset-iceberg-mapping-service.ts`; `0pt.source_rate_policy_created` emits from `src/services/source-rate-policy-service.ts`. All via the existing `Aud0Client` (mock/no-op by default, same as every other AUD0 emission in this repo -- AUD0 itself still has no live ingestion endpoint). - Item 7 (N0D heavy-write requests): CLOSED as real code -- `IcebergWriteService.requestIcebergWrite` requests an N0D job (`0pt.iceberg_write_job`) only when `heavy: true`, never executes in-process; tested in `tests/p0/iceberg-write-service.test.ts`. - Item 8 (no lakehouse state outside ST0RE): CLOSED as a design property, not just a statement -- `IcebergWriteService` and `DatasetIcebergMappingService` hold zero table/snapshot fields; the only thing 0pt persists is the mapping's `iceberg_table_id` reference. See the boundary doc. - Item 1 (Iceberg canonical, Delta rejected): CONFIRMED, unchanged from `docs/10` -- `IcebergWriteService.mapDatasetToIcebergTable` defaults `table_format` to `"iceberg"`. ## To-do list 1. **Use Apache Iceberg as the canonical write format** for datasets 0pt produces, per `docs/10-lakehouse-decision.md` / ADR-0030 (`n0d` repo). Do not adopt Delta Lake as a native write target; Delta is import/export-bridge-only, ecosystem-wide. 2. **Add a `dataset_to_iceberg_table` mapping.** Every 0pt dataset (today: rows keyed by `entity_type` + market in `dim_company` / `fact_financials` / `dim_contact` / `dim_ubo` / `dim_vehicle`, per `docs/03-data-model.md`, or a catalog product / order delivery) needs a stable mapping to the Iceberg table(s) it is written to. Schema name: **`0pt.dataset_iceberg_mapping.schema.v1`**. Suggested shape (to be finalized by the implementation session, following the `snake_case` / `uuid` PK / `created_at`+`updated_at` conventions in `docs/03-data-model.md`): - `dataset_id` — 0pt-side identifier (catalog product id, or `order_id`+`item_id` for custom orders). - `iceberg_table_ref` — fully-qualified table identifier as registered in ST0RE's table registry (catalog.namespace.table). - `entity_type`, `market` — for routing/filtering. - `mapped_at`, `mapped_by`. 3. **Write a first test dataset to Iceberg** through ST0RE's table registry. *(Status: CLOSED per the update above — `tests/p0/iceberg-write-service.test.ts` exercises exactly this pilot flow, but only against `St0reLakehouseClient`'s mock; not yet run against a live ST0RE. Treat this item as "code exists," not "validated end-to-end against a real deployment.")* Original plan: pick a small, low-risk dataset (e.g. a single-market `dim_company` slice) as the pilot, to validate the write path end-to-end before any production order relies on it — that live-ST0RE validation still has not happened. 4. **Keep data quality rules in 0pt.** This is the correct *target* per the `n0d` audit, and remains a spec-level design in `docs/06-data-engine.md` §QA (Great Expectations-style checks for validity, completeness, freshness, uniqueness, corroboration), but **it is not built** — there is no dedicated quality-check/quality-report service in this repo as of this writing (see the correction callout at the top of this doc). Once it is built, the design intent stands: quality logic must be caller-supplied logic that 0pt owns and hands to whichever engine executes a job; it must not become logic owned or embedded by N0D, even after N0D executes heavy writes/compactions on 0pt's behalf (see item 7). This item stays open until that service exists and is tested. 5. **Keep source-rate/crawl-compliance policy logic in 0pt** — build a real model here, do not rely on N0D for this. *(Status: CLOSED per the update above — `SourceRatePolicyService` is real and tested.)* Context: N0D's `ComputePolicy.sourceRatePolicy` (per-domain crawl rate limits) has been deprecated in N0D as part of the same audit (the field is still present on N0D's type for backward compatibility, but N0D's own boundary-repair tests assert it now has zero effect on any allow/deny decision — verified by reading `n0d`'s `src/policy.ts` and `src/__tests__/boundary-repair.test.ts`) — that field never belonged to N0D. 0pt already has the raw material for this in `sources.yaml` (`docs/06-data-engine.md`, the `rate_limit` / `robots_ok` fields per source), but that's currently free-text/per-entry, not a first-class policy object. Formalize it as **`0pt.source_rate_policy.schema.v1`**. Suggested shape, following the same conventions: - `source_id` — references `dim_source(source_id)` (`docs/03-data-model.md`). - `domain` — the crawl-target domain (e.g. `allabolag.se`). - `requests_per_second` / `requests_per_minute` — the compliance-derived rate limit. - `robots_ok` — whether `robots.txt` permits the access pattern. - `access` — mirrors `dim_source.access` (`api`|`bulk`|`authorized_web`|`licensed`). - `created_at`, `updated_at`, `created_by`. This model belongs entirely to 0pt's sourcing/compliance domain (`docs/06`) — it must not be reintroduced into N0D, CL0UD, or any other product's policy defaults. 6. **Emit lineage for dataset → Iceberg table → snapshot**, to AUD0. *(Status: CLOSED per the update above — all four events below are wired and fire through the existing `Aud0Client`, which is mock/no-op by default because AUD0 still has no live ingestion endpoint. This did not mean standing up a bespoke lineage sink prematurely — it means calling the same shared, already-mocked AUD0 client interface every other 0pt event in this repo already calls.)* Plan for three events, emitted by 0pt at the corresponding step: - **`0pt.dataset_iceberg_table_mapped`** — fired when step 2's mapping is created/updated. - **`0pt.iceberg_write_requested`** — fired when 0pt requests a write of a produced dataset into its mapped Iceberg table (whether executed by 0pt directly or via an N0D job, per item 7). - **`0pt.iceberg_snapshot_created`** — fired once the write lands and a new Iceberg snapshot exists for the table, carrying enough detail (table ref, snapshot id, row count, dataset id, order id if applicable) for AUD0 to build lineage from source → dataset → Iceberg snapshot. - **`0pt.source_rate_policy_created`** — fired when a new `0pt.source_rate_policy.schema.v1` record is created (or materially updated), so AUD0/CL0UD have an evented trail of 0pt's crawl-compliance posture without needing to poll 0pt's DB. 7. **Request N0D jobs for heavy writes/compactions if/when needed, rather than 0pt executing them itself.** *(Status: CLOSED per the update above — `IcebergWriteService.requestIcebergWrite` requests an N0D job only when `heavy: true`, tested.)* 0pt produces and owns the dataset; if a write or compaction is heavy enough to warrant dedicated compute (large backfills, table maintenance), 0pt should submit that as a job request to N0D's execution layer rather than running it in-process in `services/api` workers. N0D executes; it does not own the data, the schema, or the decision of *what* to write. 8. **Do not persist lakehouse table state outside ST0RE.** *(Status: CLOSED as a design property per the update above — `IcebergWriteService`/`DatasetIcebergMappingService` hold zero table or snapshot state; the only thing 0pt persists is a reference `iceberg_table_id`.)* ST0RE's table registry now exists as real code (not yet confirmed live/reachable from this repo). 0pt must not stand up a parallel catalog/metadata store of its own (that would repeat the exact mistake this plan is repairing: N0D building its own Iceberg catalog because nothing else had one yet). ## Required schema names (for the future implementation session) - `0pt.dataset_iceberg_mapping.schema.v1` - `0pt.source_rate_policy.schema.v1` ## Required future event names (for the future implementation session) - `0pt.dataset_iceberg_table_mapped` - `0pt.iceberg_write_requested` - `0pt.iceberg_snapshot_created` - `0pt.source_rate_policy_created` ## Sequencing / dependencies *(Status note: this section described the original plan's dependencies. As of the 2026-07-12 update above, items 1, 2, 3, 5, 6, 7, 8 are built — mock-tested, not live-verified. Item 4 (data-quality/QA service) is the one item that remains genuinely not-started; see the correction callout at the top of this doc. The dependency shape below is kept for the record and because "mock-tested" is not the same as "the dependency is resolved" — see each item's status note above for what's still actually pending.)* - Items 2, 4, 5 have no hard dependency on another product and can be defined/built inside 0pt — 2 and 5 now are; 4 (the QA service) still is not. - Items 1, 3, 8 depended on **ST0RE's table registry** existing. It now exists as real code in `sax3l/st0re`; what remains outstanding is a *live, reachable* `ST0RE_URL` to verify against — no code path in this repo reads `ST0RE_URL` from the environment automatically (same as `CL0UD_URL`/`N0D_URL`/`AUD0_URL` — none of the platform clients have an env fallback wired; a base URL must be passed in explicitly via each service's constructor options). Wiring `ST0RE_URL` alone would not be enough, either: `sax3l/st0re`'s current `main` enforces `X-Tenant-Id`/ `X-Actor-Id` headers on the `getTable`/`createSnapshot` routes this write path calls, and this repo's `St0reLakehouseClient` does not send them yet — see `docs/ICEBERG_WRITE_BOUNDARY.md`'s "Real code, not live-verified" section for the full account (found during this verification pass). - Item 6 depended on **AUD0** existing as a real evented product; it still does not have a live ingestion endpoint, so the four events fire through `Aud0Client`'s mock/no-op path, same as every other AUD0 emission in this repo. - Item 7 assumes N0D's job-request surface (see `n0d` repo) is reachable from 0pt's orchestrator; `IcebergWriteService` calls `N0dClient.createJob`, which — like the other platform clients — has no live `N0D_URL` wired from this repo's environment either. ## Explicitly out of scope for 0pt (do not build here) - An Iceberg catalog implementation, or any direct MinIO/Iceberg client code, ahead of ST0RE's registry existing — that repeats the `n0d` mistake this plan is repairing. - Query, compaction, or table-maintenance execution — that is N0D's job. - A lineage UI or lineage storage — that is AUD0's job; 0pt only emits the events above.