# 10 · Lakehouse decision (pointer) This doc is a **pointer**, not a new architecture. It records an ecosystem-wide decision made outside this repo and states what it means for 0pt specifically. It does not change anything in `docs/02-architecture.md` or `docs/03-data-model.md` today — Postgres remains the backbone for 0pt's platform tables and current data entities (`docs/03`). Read this alongside those docs, not instead of them. > **Status update.** Since this pointer doc was first written, a follow-up implementation pass > turned most of 0pt's side of this plan into real TypeScript — see > `docs/11-lakehouse-write-plan.md`'s "what the implementation pass closed" section and > `docs/ICEBERG_WRITE_BOUNDARY.md` for the current, corrected status. Two things to know before > reading the rest of this doc: (1) that write-path code is **real and unit-tested against a mock > ST0RE client, not verified against a running ST0RE deployment**; (2) an earlier version of > `docs/11` and `docs/ICEBERG_WRITE_BOUNDARY.md` also claimed 0pt's data-quality/QA stage had a > dedicated, tested `quality-check-service.ts`/`quality-report-service.ts` implementation — that > claim was **fabricated**. `git log --all` for those paths (and `sql/002_p0_dataset_quality.sql`, > `sql/003_*.sql`) returns zero commits, on any branch, ever. Both docs have been corrected; see > them for the full account. This doc's own references to "QA" below point only to the *spec* in > `docs/06-data-engine.md`, not to any implemented service. ## The decision (made elsewhere — do not re-litigate here) **Apache Iceberg is the canonical SIAX lakehouse table format**, per **ADR-0030**, authored in the `n0d` (Compute OS) repo (a dated copy also lives in `cl0ud`'s `docs/architecture/`, since the boundary audit that motivated it originated in n0d but the ADR itself is recorded as SIAX-wide; both copies describe the same accepted decision — verified by reading both this pass). Delta Lake is explicitly rejected as the canonical P0/P1 format across SIAX Technology; it may only ever be used later as an import/export bridge, never as the format 0pt writes natively. Rationale (from the ecosystem ADR, summarized): SIAX needs one vendor-neutral, multi-engine, catalog-first open table format for the MinIO/ST0RE lakehouse, because several products need to share the same tables — ST0RE owns catalog/metadata, 0pt owns dataset writes, N0D owns query/compaction execution, AUD0 owns lineage, API0 owns docs — and because future reads must work from many engines (Trino/Spark/Flink/DuckDB/etc.), not one. ## Cross-product ownership split (context, not 0pt's to change) | Product | Owns | |---|---| | ST0RE | Table/catalog/artifact metadata (the Iceberg catalog + table registry) | | **0pt** | **Dataset production and writes into Iceberg tables**, through ST0RE's table registry | | N0D | Query/compaction/maintenance execution only | | AUD0 | Lineage | | CL0UD | Policy | This table condenses ADR-0030 §2.1/§2.4's ownership split for 0pt's purposes — it is a paraphrase, not a verbatim quote (corrected this verification pass; re-checked against `cl0ud`'s copy of the ADR). The ADR's §2.4 table also names **INF0** (model registry/experiment tracking, MLflow as an adapter) and its §2.1 also names **API0** (schema/docs exposure); both rows are omitted here because neither changes anything for 0pt specifically. One paraphrase is deliberate, not accidental: the ADR's own 0pt row reads "Dataset production, **quality**, data contracts, writes" — the word "quality" is intentionally dropped from this table's 0pt row above, because 0pt's quality/QA stage is not yet built (see the status callout at the top of this doc). See ADR-0030 §2.4 directly (`cl0ud`'s or `n0d`'s copy) for its own exact wording of every row. ## What this means for 0pt 0pt is the **write-path owner** for the datasets it produces: 0pt runs sourcing → extraction → structuring → enrichment → QA (`docs/06-data-engine.md` — a design spec, not yet a built service; see the status callout above) as it does today, and writes the resulting datasets as **Iceberg tables through ST0RE's table registry**. 0pt does **not** talk to N0D's catalog or any lakehouse storage directly; it goes through ST0RE. ST0RE's table registry is real, committed code (`sax3l/st0re`, `/v1/st0re/lakehouse/*`), and 0pt has a real client and services that target it — `src/clients/st0re-lakehouse-client.ts`, `src/services/iceberg-write-service.ts`, `src/services/dataset-iceberg-mapping-service.ts`. None of this has been exercised against a *running* ST0RE instance from this repo: every test to date runs against that client's deterministic in-memory mock (no `ST0RE_URL` was reachable from this environment). Treat "0pt writes Iceberg tables through ST0RE" as **real code, mock-tested, not live-verified** — see `docs/ICEBERG_WRITE_BOUNDARY.md`'s "Verification status" section for the full detail, including what is and isn't wired end-to-end. This does not replace Postgres as the platform backbone (orders, users, jobs, billing — `docs/03` — stay exactly as they are). It is additive: dataset **output** (the delivered data product, today exported as CSV/Parquet/JSON per `docs/04-api.md`) gains an additional, canonical Iceberg write-path. Concrete to-dos, current build status, and required schema/event names are in `docs/11-lakehouse-write-plan.md`. ## Also relevant: the `sourceRatePolicy` deprecation in N0D The same ecosystem audit that produced ADR-0030 found N0D's `ComputePolicy` type carrying a `sourceRatePolicy` field (per-source-domain crawl rate limits, e.g. `{'allabolag.se': 1, 'ratsit.se': 1}`) hardcoded into N0D's policy defaults. That is 0pt's own crawl-domain knowledge — it has been deprecated in N0D (the field is still present on the type for backward compatibility, but N0D's own boundary-repair tests assert it now has **zero effect** on any allow/deny decision; verified by reading `n0d`'s `src/policy.ts` and `src/__tests__/boundary-repair.test.ts` this pass). 0pt does not rely on N0D for source-rate/ crawl-compliance policy; 0pt already owns the sourcing chain and its rate limits today (`sources.yaml`, `docs/06-data-engine.md` §`rate_limit`), and has since formalized its own model — `src/services/source-rate-policy-service.ts`, real and tested, see `docs/11-lakehouse-write-plan.md` and `docs/ICEBERG_WRITE_BOUNDARY.md`. ## Non-goals of this doc - Does not itself implement Iceberg writes — see `docs/11-lakehouse-write-plan.md` / `docs/ICEBERG_WRITE_BOUNDARY.md` for what a later pass actually built (real, mock-tested code; not live-verified; data-quality/QA enforcement specifically is **not built** — see the status callout above). - Does not modify `docs/03-data-model.md`'s Postgres schema. - Does not require ST0RE's table registry to exist — as of this pass it does exist as real, committed code in `sax3l/st0re`, though this repo has not confirmed it live/reachable. This doc remains the pointer; `docs/11` lists what's still blocked on a *live, reachable* ST0RE vs. what 0pt has already built against the mock.