Document lakehouse write-path ownership decision (ADR-0030)

Points at n0d's ADR-0030 (Apache Iceberg canonical, Delta rejected for
P0/P1) and lays out 0pt's write-path plan: dataset-to-Iceberg-table
mapping, writes through ST0RE's future table registry, keeping QA and
source-rate/crawl-compliance logic in 0pt. Schema/event names specified
precisely for a future implementation session. Written adjacent to, not
touching, the substantial live P0 data-platform build in progress in this
repo (data-contract/dataset/quality-check services).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-11 15:11:15 +02:00
co-authored by Claude Sonnet 5
parent 00982f027d
commit 2545218265
2 changed files with 168 additions and 0 deletions
+60
View File
@@ -0,0 +1,60 @@
# 10 · Lakehouse decision (pointer)
This doc is a **pointer**, not a new architecture. It records an ecosystem-wide decision made
outside this repo and states what it means for 0pt specifically. It does not change anything in
`docs/02-architecture.md` or `docs/03-data-model.md` today — Postgres remains the backbone for
0pt's platform tables and current data entities (`docs/03`). Read this alongside those docs, not
instead of them.
## The decision (made elsewhere — do not re-litigate here)
**Apache Iceberg is the canonical SIAX lakehouse table format**, per **ADR-0030**, authored in the
`n0d` (Compute OS) repo. Delta Lake is explicitly rejected as the canonical P0/P1 format across
SIAX Technology; it may only ever be used later as an import/export bridge, never as the format
0pt writes natively.
Rationale (from the ecosystem ADR, summarized): SIAX needs one vendor-neutral, multi-engine,
catalog-first open table format for the MinIO/ST0RE lakehouse, because several products need to
share the same tables — ST0RE owns catalog/metadata, 0pt owns dataset writes, N0D owns
query/compaction execution, AUD0 owns lineage, API0 owns docs — and because future reads must
work from many engines (Trino/Spark/Flink/DuckDB/etc.), not one.
## Cross-product ownership split (context, not 0pt's to change)
| Product | Owns |
|---|---|
| ST0RE | Table/catalog/artifact metadata (the Iceberg catalog + table registry) |
| **0pt** | **Dataset production and writes into Iceberg tables**, through ST0RE's table registry |
| N0D | Query/compaction/maintenance execution only |
| AUD0 | Lineage |
| CL0UD | Policy |
## What this means for 0pt
0pt is the **write-path owner** for the datasets it produces: 0pt runs sourcing → extraction →
structuring → enrichment → QA (`docs/06-data-engine.md`) as it does today, and — once ST0RE's
table registry exists — writes the resulting datasets as **Iceberg tables through that registry**.
0pt does **not** talk to N0D's catalog or any lakehouse storage directly; it goes through ST0RE.
This does not replace Postgres as the platform backbone (orders, users, jobs, billing — `docs/03`
— stay exactly as they are). It is additive: dataset **output** (the delivered data product, today
exported as CSV/Parquet/JSON per `docs/04-api.md`) gains an additional, canonical Iceberg
write-path once ST0RE's registry ships. Concrete to-dos and required schema/event names are in
`docs/11-lakehouse-write-plan.md`.
## Also relevant: the `sourceRatePolicy` deprecation in N0D
The same ecosystem audit that produced ADR-0030 found N0D's `ComputePolicy` type carrying a
`sourceRatePolicy` field (per-source-domain crawl rate limits, e.g.
`{'allabolag.se': 1, 'ratsit.se': 1}`) hardcoded into N0D's policy defaults. That is 0pt's own
crawl-domain knowledge — it has been deprecated/removed from N0D's seed data. 0pt does not rely on
N0D for source-rate/crawl-compliance policy; 0pt already owns the sourcing chain and its rate
limits today (`sources.yaml`, `docs/06-data-engine.md` §`rate_limit`), and will formalize its own
model rather than depend on anything in N0D for this. See `docs/11-lakehouse-write-plan.md`.
## Non-goals of this doc
- Does not implement Iceberg writes today.
- Does not modify `docs/03-data-model.md`'s Postgres schema.
- Does not require ST0RE's table registry to exist yet — this doc is the pointer; the plan doc
lists what's blocked on ST0RE vs. what 0pt can define now.
+108
View File
@@ -0,0 +1,108 @@
# 11 · Lakehouse write plan (0pt's to-do list)
Concrete follow-up to `docs/10-lakehouse-decision.md`. This is a planning doc for a **future**
implementation session — it names the schemas/events precisely so that session doesn't have to
guess, but it does not build the full pipeline today. Some items are already true today (quality
rules) and this doc just confirms 0pt intends to keep them that way.
## To-do list
1. **Use Apache Iceberg as the canonical write format** for datasets 0pt produces, per
`docs/10-lakehouse-decision.md` / ADR-0030 (`n0d` repo). Do not adopt Delta Lake as a native
write target; Delta is import/export-bridge-only, ecosystem-wide.
2. **Add a `dataset_to_iceberg_table` mapping.** Every 0pt dataset (today: rows keyed by
`entity_type` + market in `dim_company` / `fact_financials` / `dim_contact` / `dim_ubo` /
`dim_vehicle`, per `docs/03-data-model.md`, or a catalog product / order delivery) needs a
stable mapping to the Iceberg table(s) it is written to. Schema name:
**`0pt.dataset_iceberg_mapping.schema.v1`**. Suggested shape (to be finalized by the
implementation session, following the `snake_case` / `uuid` PK / `created_at`+`updated_at`
conventions in `docs/03-data-model.md`):
- `dataset_id` — 0pt-side identifier (catalog product id, or `order_id`+`item_id` for custom
orders).
- `iceberg_table_ref` — fully-qualified table identifier as registered in ST0RE's table
registry (catalog.namespace.table).
- `entity_type`, `market` — for routing/filtering.
- `mapped_at`, `mapped_by`.
3. **Write a first test dataset to Iceberg** through ST0RE's (future) table registry, once that
registry exists. Pick a small, low-risk dataset (e.g. a single-market `dim_company` slice) as
the pilot. This validates the write path end-to-end before any production order relies on it.
4. **Keep data quality rules in 0pt.** This is already correct today (per the `n0d` audit): 0pt's
QA stage (`docs/06-data-engine.md` §QA — Great Expectations checks for validity, completeness,
freshness, uniqueness, corroboration) is caller-supplied logic that 0pt owns and hands to
whichever engine executes a job; it is not, and must not become, logic owned or embedded by
N0D. This doc confirms 0pt intends to keep authoring/owning quality rules itself even after
N0D executes heavy writes/compactions on 0pt's behalf (see item 7).
5. **Keep source-rate/crawl-compliance policy logic in 0pt** — build a real model here, do not
rely on N0D for this. Context: N0D's `ComputePolicy.sourceRatePolicy` (per-domain crawl rate
limits) has been deprecated/removed from N0D's seed data as part of the same audit — that field
never belonged to N0D. 0pt already has the raw material for this in `sources.yaml`
(`docs/06-data-engine.md`, the `rate_limit` / `robots_ok` fields per source), but that's
currently free-text/per-entry, not a first-class policy object. Formalize it as
**`0pt.source_rate_policy.schema.v1`**. Suggested shape, following the same conventions:
- `source_id` — references `dim_source(source_id)` (`docs/03-data-model.md`).
- `domain` — the crawl-target domain (e.g. `allabolag.se`).
- `requests_per_second` / `requests_per_minute` — the compliance-derived rate limit.
- `robots_ok` — whether `robots.txt` permits the access pattern.
- `access` — mirrors `dim_source.access` (`api`|`bulk`|`authorized_web`|`licensed`).
- `created_at`, `updated_at`, `created_by`.
This model belongs entirely to 0pt's sourcing/compliance domain (`docs/06`) — it must not be
reintroduced into N0D, CL0UD, or any other product's policy defaults.
6. **Emit lineage for dataset → Iceberg table → snapshot**, to AUD0, once AUD0 exists as a real
evented product (it does not yet — do not build a lineage sink against it prematurely). Plan
for three events, emitted by 0pt at the corresponding step:
- **`0pt.dataset_iceberg_table_mapped`** — fired when step 2's mapping is created/updated.
- **`0pt.iceberg_write_requested`** — fired when 0pt requests a write of a produced dataset
into its mapped Iceberg table (whether executed by 0pt directly or via an N0D job, per
item 7).
- **`0pt.iceberg_snapshot_created`** — fired once the write lands and a new Iceberg snapshot
exists for the table, carrying enough detail (table ref, snapshot id, row count, dataset id,
order id if applicable) for AUD0 to build lineage from source → dataset → Iceberg
snapshot.
- **`0pt.source_rate_policy_created`** — fired when a new `0pt.source_rate_policy.schema.v1`
record is created (or materially updated), so AUD0/CL0UD have an evented trail of 0pt's
crawl-compliance posture without needing to poll 0pt's DB.
7. **Request N0D jobs for heavy writes/compactions if/when needed, rather than 0pt executing
them itself.** 0pt produces and owns the dataset; if a write or compaction is heavy enough to
warrant dedicated compute (large backfills, table maintenance), 0pt should submit that as a job
request to N0D's execution layer rather than running it in-process in `services/api` workers.
N0D executes; it does not own the data, the schema, or the decision of *what* to write.
8. **Do not persist lakehouse table state outside ST0RE** once ST0RE's registry exists. Until
then, 0pt has no lakehouse table state to persist at all — Postgres (`docs/03`) remains the
only store. Once ST0RE ships its registry, 0pt must not stand up a parallel
catalog/metadata store of its own (that would repeat the exact mistake this plan is repairing:
N0D building its own Iceberg catalog because nothing else had one yet).
## Required schema names (for the future implementation session)
- `0pt.dataset_iceberg_mapping.schema.v1`
- `0pt.source_rate_policy.schema.v1`
## Required future event names (for the future implementation session)
- `0pt.dataset_iceberg_table_mapped`
- `0pt.iceberg_write_requested`
- `0pt.iceberg_snapshot_created`
- `0pt.source_rate_policy_created`
## Sequencing / dependencies
- Items 2, 4, 5 have no hard dependency on another product and can be defined/built inside 0pt
today (this repo already owns `sources.yaml` and the QA stage).
- Items 1, 3, 8 are blocked on **ST0RE's table registry** existing.
- Item 6 is blocked on **AUD0** existing as a real evented product.
- Item 7 assumes N0D's job-request surface (see `n0d` repo) is reachable from 0pt's orchestrator;
no such integration exists yet in `services/api` and none is being added by this doc.
## Explicitly out of scope for 0pt (do not build here)
- An Iceberg catalog implementation, or any direct MinIO/Iceberg client code, ahead of ST0RE's
registry existing — that repeats the `n0d` mistake this plan is repairing.
- Query, compaction, or table-maintenance execution — that is N0D's job.
- A lineage UI or lineage storage — that is AUD0's job; 0pt only emits the events above.