Document lakehouse write-path ownership decision (ADR-0030)
Points at n0d's ADR-0030 (Apache Iceberg canonical, Delta rejected for P0/P1) and lays out 0pt's write-path plan: dataset-to-Iceberg-table mapping, writes through ST0RE's future table registry, keeping QA and source-rate/crawl-compliance logic in 0pt. Schema/event names specified precisely for a future implementation session. Written adjacent to, not touching, the substantial live P0 data-platform build in progress in this repo (data-contract/dataset/quality-check services). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,60 @@
|
||||
# 10 · Lakehouse decision (pointer)
|
||||
|
||||
This doc is a **pointer**, not a new architecture. It records an ecosystem-wide decision made
|
||||
outside this repo and states what it means for 0pt specifically. It does not change anything in
|
||||
`docs/02-architecture.md` or `docs/03-data-model.md` today — Postgres remains the backbone for
|
||||
0pt's platform tables and current data entities (`docs/03`). Read this alongside those docs, not
|
||||
instead of them.
|
||||
|
||||
## The decision (made elsewhere — do not re-litigate here)
|
||||
|
||||
**Apache Iceberg is the canonical SIAX lakehouse table format**, per **ADR-0030**, authored in the
|
||||
`n0d` (Compute OS) repo. Delta Lake is explicitly rejected as the canonical P0/P1 format across
|
||||
SIAX Technology; it may only ever be used later as an import/export bridge, never as the format
|
||||
0pt writes natively.
|
||||
|
||||
Rationale (from the ecosystem ADR, summarized): SIAX needs one vendor-neutral, multi-engine,
|
||||
catalog-first open table format for the MinIO/ST0RE lakehouse, because several products need to
|
||||
share the same tables — ST0RE owns catalog/metadata, 0pt owns dataset writes, N0D owns
|
||||
query/compaction execution, AUD0 owns lineage, API0 owns docs — and because future reads must
|
||||
work from many engines (Trino/Spark/Flink/DuckDB/etc.), not one.
|
||||
|
||||
## Cross-product ownership split (context, not 0pt's to change)
|
||||
|
||||
| Product | Owns |
|
||||
|---|---|
|
||||
| ST0RE | Table/catalog/artifact metadata (the Iceberg catalog + table registry) |
|
||||
| **0pt** | **Dataset production and writes into Iceberg tables**, through ST0RE's table registry |
|
||||
| N0D | Query/compaction/maintenance execution only |
|
||||
| AUD0 | Lineage |
|
||||
| CL0UD | Policy |
|
||||
|
||||
## What this means for 0pt
|
||||
|
||||
0pt is the **write-path owner** for the datasets it produces: 0pt runs sourcing → extraction →
|
||||
structuring → enrichment → QA (`docs/06-data-engine.md`) as it does today, and — once ST0RE's
|
||||
table registry exists — writes the resulting datasets as **Iceberg tables through that registry**.
|
||||
0pt does **not** talk to N0D's catalog or any lakehouse storage directly; it goes through ST0RE.
|
||||
|
||||
This does not replace Postgres as the platform backbone (orders, users, jobs, billing — `docs/03`
|
||||
— stay exactly as they are). It is additive: dataset **output** (the delivered data product, today
|
||||
exported as CSV/Parquet/JSON per `docs/04-api.md`) gains an additional, canonical Iceberg
|
||||
write-path once ST0RE's registry ships. Concrete to-dos and required schema/event names are in
|
||||
`docs/11-lakehouse-write-plan.md`.
|
||||
|
||||
## Also relevant: the `sourceRatePolicy` deprecation in N0D
|
||||
|
||||
The same ecosystem audit that produced ADR-0030 found N0D's `ComputePolicy` type carrying a
|
||||
`sourceRatePolicy` field (per-source-domain crawl rate limits, e.g.
|
||||
`{'allabolag.se': 1, 'ratsit.se': 1}`) hardcoded into N0D's policy defaults. That is 0pt's own
|
||||
crawl-domain knowledge — it has been deprecated/removed from N0D's seed data. 0pt does not rely on
|
||||
N0D for source-rate/crawl-compliance policy; 0pt already owns the sourcing chain and its rate
|
||||
limits today (`sources.yaml`, `docs/06-data-engine.md` §`rate_limit`), and will formalize its own
|
||||
model rather than depend on anything in N0D for this. See `docs/11-lakehouse-write-plan.md`.
|
||||
|
||||
## Non-goals of this doc
|
||||
|
||||
- Does not implement Iceberg writes today.
|
||||
- Does not modify `docs/03-data-model.md`'s Postgres schema.
|
||||
- Does not require ST0RE's table registry to exist yet — this doc is the pointer; the plan doc
|
||||
lists what's blocked on ST0RE vs. what 0pt can define now.
|
||||
@@ -0,0 +1,108 @@
|
||||
# 11 · Lakehouse write plan (0pt's to-do list)
|
||||
|
||||
Concrete follow-up to `docs/10-lakehouse-decision.md`. This is a planning doc for a **future**
|
||||
implementation session — it names the schemas/events precisely so that session doesn't have to
|
||||
guess, but it does not build the full pipeline today. Some items are already true today (quality
|
||||
rules) and this doc just confirms 0pt intends to keep them that way.
|
||||
|
||||
## To-do list
|
||||
|
||||
1. **Use Apache Iceberg as the canonical write format** for datasets 0pt produces, per
|
||||
`docs/10-lakehouse-decision.md` / ADR-0030 (`n0d` repo). Do not adopt Delta Lake as a native
|
||||
write target; Delta is import/export-bridge-only, ecosystem-wide.
|
||||
|
||||
2. **Add a `dataset_to_iceberg_table` mapping.** Every 0pt dataset (today: rows keyed by
|
||||
`entity_type` + market in `dim_company` / `fact_financials` / `dim_contact` / `dim_ubo` /
|
||||
`dim_vehicle`, per `docs/03-data-model.md`, or a catalog product / order delivery) needs a
|
||||
stable mapping to the Iceberg table(s) it is written to. Schema name:
|
||||
**`0pt.dataset_iceberg_mapping.schema.v1`**. Suggested shape (to be finalized by the
|
||||
implementation session, following the `snake_case` / `uuid` PK / `created_at`+`updated_at`
|
||||
conventions in `docs/03-data-model.md`):
|
||||
- `dataset_id` — 0pt-side identifier (catalog product id, or `order_id`+`item_id` for custom
|
||||
orders).
|
||||
- `iceberg_table_ref` — fully-qualified table identifier as registered in ST0RE's table
|
||||
registry (catalog.namespace.table).
|
||||
- `entity_type`, `market` — for routing/filtering.
|
||||
- `mapped_at`, `mapped_by`.
|
||||
|
||||
3. **Write a first test dataset to Iceberg** through ST0RE's (future) table registry, once that
|
||||
registry exists. Pick a small, low-risk dataset (e.g. a single-market `dim_company` slice) as
|
||||
the pilot. This validates the write path end-to-end before any production order relies on it.
|
||||
|
||||
4. **Keep data quality rules in 0pt.** This is already correct today (per the `n0d` audit): 0pt's
|
||||
QA stage (`docs/06-data-engine.md` §QA — Great Expectations checks for validity, completeness,
|
||||
freshness, uniqueness, corroboration) is caller-supplied logic that 0pt owns and hands to
|
||||
whichever engine executes a job; it is not, and must not become, logic owned or embedded by
|
||||
N0D. This doc confirms 0pt intends to keep authoring/owning quality rules itself even after
|
||||
N0D executes heavy writes/compactions on 0pt's behalf (see item 7).
|
||||
|
||||
5. **Keep source-rate/crawl-compliance policy logic in 0pt** — build a real model here, do not
|
||||
rely on N0D for this. Context: N0D's `ComputePolicy.sourceRatePolicy` (per-domain crawl rate
|
||||
limits) has been deprecated/removed from N0D's seed data as part of the same audit — that field
|
||||
never belonged to N0D. 0pt already has the raw material for this in `sources.yaml`
|
||||
(`docs/06-data-engine.md`, the `rate_limit` / `robots_ok` fields per source), but that's
|
||||
currently free-text/per-entry, not a first-class policy object. Formalize it as
|
||||
**`0pt.source_rate_policy.schema.v1`**. Suggested shape, following the same conventions:
|
||||
- `source_id` — references `dim_source(source_id)` (`docs/03-data-model.md`).
|
||||
- `domain` — the crawl-target domain (e.g. `allabolag.se`).
|
||||
- `requests_per_second` / `requests_per_minute` — the compliance-derived rate limit.
|
||||
- `robots_ok` — whether `robots.txt` permits the access pattern.
|
||||
- `access` — mirrors `dim_source.access` (`api`|`bulk`|`authorized_web`|`licensed`).
|
||||
- `created_at`, `updated_at`, `created_by`.
|
||||
This model belongs entirely to 0pt's sourcing/compliance domain (`docs/06`) — it must not be
|
||||
reintroduced into N0D, CL0UD, or any other product's policy defaults.
|
||||
|
||||
6. **Emit lineage for dataset → Iceberg table → snapshot**, to AUD0, once AUD0 exists as a real
|
||||
evented product (it does not yet — do not build a lineage sink against it prematurely). Plan
|
||||
for three events, emitted by 0pt at the corresponding step:
|
||||
- **`0pt.dataset_iceberg_table_mapped`** — fired when step 2's mapping is created/updated.
|
||||
- **`0pt.iceberg_write_requested`** — fired when 0pt requests a write of a produced dataset
|
||||
into its mapped Iceberg table (whether executed by 0pt directly or via an N0D job, per
|
||||
item 7).
|
||||
- **`0pt.iceberg_snapshot_created`** — fired once the write lands and a new Iceberg snapshot
|
||||
exists for the table, carrying enough detail (table ref, snapshot id, row count, dataset id,
|
||||
order id if applicable) for AUD0 to build lineage from source → dataset → Iceberg
|
||||
snapshot.
|
||||
- **`0pt.source_rate_policy_created`** — fired when a new `0pt.source_rate_policy.schema.v1`
|
||||
record is created (or materially updated), so AUD0/CL0UD have an evented trail of 0pt's
|
||||
crawl-compliance posture without needing to poll 0pt's DB.
|
||||
|
||||
7. **Request N0D jobs for heavy writes/compactions if/when needed, rather than 0pt executing
|
||||
them itself.** 0pt produces and owns the dataset; if a write or compaction is heavy enough to
|
||||
warrant dedicated compute (large backfills, table maintenance), 0pt should submit that as a job
|
||||
request to N0D's execution layer rather than running it in-process in `services/api` workers.
|
||||
N0D executes; it does not own the data, the schema, or the decision of *what* to write.
|
||||
|
||||
8. **Do not persist lakehouse table state outside ST0RE** once ST0RE's registry exists. Until
|
||||
then, 0pt has no lakehouse table state to persist at all — Postgres (`docs/03`) remains the
|
||||
only store. Once ST0RE ships its registry, 0pt must not stand up a parallel
|
||||
catalog/metadata store of its own (that would repeat the exact mistake this plan is repairing:
|
||||
N0D building its own Iceberg catalog because nothing else had one yet).
|
||||
|
||||
## Required schema names (for the future implementation session)
|
||||
|
||||
- `0pt.dataset_iceberg_mapping.schema.v1`
|
||||
- `0pt.source_rate_policy.schema.v1`
|
||||
|
||||
## Required future event names (for the future implementation session)
|
||||
|
||||
- `0pt.dataset_iceberg_table_mapped`
|
||||
- `0pt.iceberg_write_requested`
|
||||
- `0pt.iceberg_snapshot_created`
|
||||
- `0pt.source_rate_policy_created`
|
||||
|
||||
## Sequencing / dependencies
|
||||
|
||||
- Items 2, 4, 5 have no hard dependency on another product and can be defined/built inside 0pt
|
||||
today (this repo already owns `sources.yaml` and the QA stage).
|
||||
- Items 1, 3, 8 are blocked on **ST0RE's table registry** existing.
|
||||
- Item 6 is blocked on **AUD0** existing as a real evented product.
|
||||
- Item 7 assumes N0D's job-request surface (see `n0d` repo) is reachable from 0pt's orchestrator;
|
||||
no such integration exists yet in `services/api` and none is being added by this doc.
|
||||
|
||||
## Explicitly out of scope for 0pt (do not build here)
|
||||
|
||||
- An Iceberg catalog implementation, or any direct MinIO/Iceberg client code, ahead of ST0RE's
|
||||
registry existing — that repeats the `n0d` mistake this plan is repairing.
|
||||
- Query, compaction, or table-maintenance execution — that is N0D's job.
|
||||
- A lineage UI or lineage storage — that is AUD0's job; 0pt only emits the events above.
|
||||
Reference in New Issue
Block a user