Files
0pt/docs/10-lakehouse-decision.md
T

7.0 KiB

10 · Lakehouse decision (pointer)

This doc is a pointer, not a new architecture. It records an ecosystem-wide decision made outside this repo and states what it means for 0pt specifically. It does not change anything in docs/02-architecture.md or docs/03-data-model.md today — Postgres remains the backbone for 0pt's platform tables and current data entities (docs/03). Read this alongside those docs, not instead of them.

Status update. Since this pointer doc was first written, a follow-up implementation pass turned most of 0pt's side of this plan into real TypeScript — see docs/11-lakehouse-write-plan.md's "what the implementation pass closed" section and docs/ICEBERG_WRITE_BOUNDARY.md for the current, corrected status. Two things to know before reading the rest of this doc: (1) that write-path code is real and unit-tested against a mock ST0RE client, not verified against a running ST0RE deployment; (2) an earlier version of docs/11 and docs/ICEBERG_WRITE_BOUNDARY.md also claimed 0pt's data-quality/QA stage had a dedicated, tested quality-check-service.ts/quality-report-service.ts implementation — that claim was fabricated. git log --all for those paths (and sql/002_p0_dataset_quality.sql, sql/003_*.sql) returns zero commits, on any branch, ever. Both docs have been corrected; see them for the full account. This doc's own references to "QA" below point only to the spec in docs/06-data-engine.md, not to any implemented service.

The decision (made elsewhere — do not re-litigate here)

Apache Iceberg is the canonical SIAX lakehouse table format, per ADR-0030, authored in the n0d (Compute OS) repo (a dated copy also lives in cl0ud's docs/architecture/, since the boundary audit that motivated it originated in n0d but the ADR itself is recorded as SIAX-wide; both copies describe the same accepted decision — verified by reading both this pass). Delta Lake is explicitly rejected as the canonical P0/P1 format across SIAX Technology; it may only ever be used later as an import/export bridge, never as the format 0pt writes natively.

Rationale (from the ecosystem ADR, summarized): SIAX needs one vendor-neutral, multi-engine, catalog-first open table format for the MinIO/ST0RE lakehouse, because several products need to share the same tables — ST0RE owns catalog/metadata, 0pt owns dataset writes, N0D owns query/compaction execution, AUD0 owns lineage, API0 owns docs — and because future reads must work from many engines (Trino/Spark/Flink/DuckDB/etc.), not one.

Cross-product ownership split (context, not 0pt's to change)

Product Owns
ST0RE Table/catalog/artifact metadata (the Iceberg catalog + table registry)
0pt Dataset production and writes into Iceberg tables, through ST0RE's table registry
N0D Query/compaction/maintenance execution only
AUD0 Lineage
CL0UD Policy

This table condenses ADR-0030 §2.1/§2.4's ownership split for 0pt's purposes — it is a paraphrase, not a verbatim quote (corrected this verification pass; re-checked against cl0ud's copy of the ADR). The ADR's §2.4 table also names INF0 (model registry/experiment tracking, MLflow as an adapter) and its §2.1 also names API0 (schema/docs exposure); both rows are omitted here because neither changes anything for 0pt specifically. One paraphrase is deliberate, not accidental: the ADR's own 0pt row reads "Dataset production, quality, data contracts, writes" — the word "quality" is intentionally dropped from this table's 0pt row above, because 0pt's quality/QA stage is not yet built (see the status callout at the top of this doc). See ADR-0030 §2.4 directly (cl0ud's or n0d's copy) for its own exact wording of every row.

What this means for 0pt

0pt is the write-path owner for the datasets it produces: 0pt runs sourcing → extraction → structuring → enrichment → QA (docs/06-data-engine.md — a design spec, not yet a built service; see the status callout above) as it does today, and writes the resulting datasets as Iceberg tables through ST0RE's table registry. 0pt does not talk to N0D's catalog or any lakehouse storage directly; it goes through ST0RE.

ST0RE's table registry is real, committed code (sax3l/st0re, /v1/st0re/lakehouse/*), and 0pt has a real client and services that target it — src/clients/st0re-lakehouse-client.ts, src/services/iceberg-write-service.ts, src/services/dataset-iceberg-mapping-service.ts. None of this has been exercised against a running ST0RE instance from this repo: every test to date runs against that client's deterministic in-memory mock (no ST0RE_URL was reachable from this environment). Treat "0pt writes Iceberg tables through ST0RE" as real code, mock-tested, not live-verified — see docs/ICEBERG_WRITE_BOUNDARY.md's "Verification status" section for the full detail, including what is and isn't wired end-to-end.

This does not replace Postgres as the platform backbone (orders, users, jobs, billing — docs/03 — stay exactly as they are). It is additive: dataset output (the delivered data product, today exported as CSV/Parquet/JSON per docs/04-api.md) gains an additional, canonical Iceberg write-path. Concrete to-dos, current build status, and required schema/event names are in docs/11-lakehouse-write-plan.md.

Also relevant: the sourceRatePolicy deprecation in N0D

The same ecosystem audit that produced ADR-0030 found N0D's ComputePolicy type carrying a sourceRatePolicy field (per-source-domain crawl rate limits, e.g. {'allabolag.se': 1, 'ratsit.se': 1}) hardcoded into N0D's policy defaults. That is 0pt's own crawl-domain knowledge — it has been deprecated in N0D (the field is still present on the type for backward compatibility, but N0D's own boundary-repair tests assert it now has zero effect on any allow/deny decision; verified by reading n0d's src/policy.ts and src/__tests__/boundary-repair.test.ts this pass). 0pt does not rely on N0D for source-rate/ crawl-compliance policy; 0pt already owns the sourcing chain and its rate limits today (sources.yaml, docs/06-data-engine.md §rate_limit), and has since formalized its own model — src/services/source-rate-policy-service.ts, real and tested, see docs/11-lakehouse-write-plan.md and docs/ICEBERG_WRITE_BOUNDARY.md.

Non-goals of this doc

  • Does not itself implement Iceberg writes — see docs/11-lakehouse-write-plan.md / docs/ICEBERG_WRITE_BOUNDARY.md for what a later pass actually built (real, mock-tested code; not live-verified; data-quality/QA enforcement specifically is not built — see the status callout above).
  • Does not modify docs/03-data-model.md's Postgres schema.
  • Does not require ST0RE's table registry to exist — as of this pass it does exist as real, committed code in sax3l/st0re, though this repo has not confirmed it live/reachable. This doc remains the pointer; docs/11 lists what's still blocked on a live, reachable ST0RE vs. what 0pt has already built against the mock.