Files
0pt/docs/11-lakehouse-write-plan.md
T
Claude (SIAX platform session)andClaude Opus 5 bcf285f057
CI (SIAX Cloud) / quality (push) Successful in 38s
CI (SIAX Cloud) / security (push) Successful in 17s
merge: unify GitHub main into Gitea main (0pt)
Merges github/main (6 commits: 17-bug audit fix, CI quality gate,
.env.example + .gitignore fix, missing platform clients + registry
services, docs accuracy overhaul) into gitea/main (8 commits: Iceberg
write boundary + live ST0RE verification, lakehouse mapping routes,
Waves 1-4 foundational hardening).

16 conflicts. Both sides had independently written the four
src/clients/*.ts and the two registry services, so those are combined
rather than picked:

- st0re-client.ts: GitHub's endpoint wins on evidence. POST goes to
  /v1/st0re/artifacts/register, not /v1/st0re/artifacts -- verified in
  sax3l/st0re src/api/routes/artifacts.ts, where the bare path is a
  GET-only list route. Gitea's URL normalisation, error-body text and
  mock kept.
- n0d-client.ts: field is `status`, not `state`, with N0D's real 13-state
  union -- verified in sax3l/n0d packages/schemas/n0d.job.schema.json,
  which says explicitly not to add "succeeded". Gitea's mock echo,
  URL normalisation and error-body text kept.
- cl0ud-client.ts: Gitea's fail-closed HTTP normalisation kept (a 200
  with no `decision` reads as "deny", not as a silent allow). GitHub's
  PolicyDecision/PolicyDecisionValue naming kept; PolicyResult retained
  as an alias.
- aud0-client.ts: permissive union of both input types (optional
  payload/metadata/correlation_id), GitHub's echo return, Gitea's URL
  normalisation and error-body text, plus a JSON-parse fallback so a
  successful POST with an empty body does not throw.
- dataset-service.ts / source-service.ts: GitHub's record/DTO split and
  event naming (0pt.source_created / 0pt.create_source, which the merged
  docs already reference) plus Gitea's name validation, description /
  created_by fields and actor_id derivation. List filters are the union.
- lakehouse mapping routes: GitHub's shared lakehouse-mapping-registry
  singletons (without which every write 400'd "mapping not found") AND
  Gitea's Zod parseBody validation. Both, not either.
- src/types/source.ts, src/types/dataset.ts: unions of both vocabularies;
  GitHub's `access` legal-sourcing field kept.
- .env.example: GitHub's audited inventory plus Gitea's Zitadel /
  AUTH_ENFORCE block.
- CLAUDE.md, docs/ICEBERG_WRITE_BOUNDARY.md: GitHub's corrections kept,
  Gitea's live-verification kept and marked as superseding the
  "would 400 outright" finding (Gitea had already added the required
  X-Tenant-Id/X-Actor-Id headers).
- pnpm-lock.yaml: regenerated with pnpm 9.15.9, not merged.

Also corrects CLAUDE.md's claim that the banned services/scrapers stack
was "since removed" -- the removal PR is open, not merged, and this
branch inherits that code from Gitea main. Nothing here resurrects,
extends or re-wires any of it; services/scrapers/ is byte-identical to
Gitea main.

Verified on the merged tree: typecheck (src/ + tests/) clean;
17 test files, 121 passed, 1 skipped (the realinfra test skipping
honestly on ST0RE HTTP 401), 0 failures. Gitea main baseline was the
same 121/1; GitHub main baseline could not even `pnpm install`
("packages field missing or empty") -- Gitea's pnpm-workspace.yaml fix
resolves that. Whole-repo `typecheck:app` reports 9 errors, an identical
set to Gitea main's baseline, so it is advisory in both CI workflows
rather than deleted. Lint: 16 problems, identical to Gitea baseline.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 12:38:01 +02:00

216 lines
16 KiB
Markdown

# 11 · Lakehouse write plan (0pt's to-do list)
Concrete follow-up to `docs/10-lakehouse-decision.md`. This is a planning doc for a **future**
implementation session — it names the schemas/events precisely so that session doesn't have to
guess, but it does not build the full pipeline today.
> **Correction (documentation-accuracy pass).** This doc previously stated that item 4 (quality
> rules staying in 0pt) was "already true today" and, in the update section below, that
> `src/services/quality-check-service.ts` and `src/services/quality-report-service.ts` were real,
> tested files that a later pass left "untouched." **That was fabricated.** `git log --all` for
> `src/services/quality-check-service.ts`, `src/services/quality-report-service.ts`,
> `src/types/quality-check.ts`, `src/types/quality-report.ts`, `sql/002_p0_dataset_quality.sql`,
> and `sql/003_*.sql` returns zero commits, on any branch (`main`, `feature/0pt-env-example`,
> `ci/add-quality-gate`), ever — these files have never existed in this repository. 0pt's
> data-quality/QA stage is a **design spec only** (`docs/06-data-engine.md` §QA — Great
> Expectations-style checks, intended to run inside the Python worker pipeline in `services/api`)
> and is **not yet implemented** as any dedicated service in this repo, TypeScript or otherwise.
> This is a real, currently open build gap, not a doc typo — see item 4's corrected entry below
> and `docs/ICEBERG_WRITE_BOUNDARY.md`'s "Data quality: not yet built" section for the full
> account. Everything else on this page (items 1, 2, 3, 5, 6, 7, 8) was independently re-verified
> against the actual service/client/test files this pass and remains accurate.
## 2026-07-13 update: live verification + field-level gap closure
A later pass live-verified the write path against `api.st0re.siax.io` (real namespace/table/snapshot
rows created), added the `namespace_id`/`mapped_at` fields the task spec names literally, and fixed
three real ST0RE contract bugs the code below had (missing tenant/actor headers, an invalid
`datasetId` FK reference, and a missing lookup-or-create for namespaces). Full detail in
`docs/ICEBERG_WRITE_BOUNDARY.md`'s "2026-07-13 update" section -- read that before this section
below, which is the original (2026-07-12) status record and is otherwise unchanged.
## 2026-07-12 update: what the 0pt write-boundary implementation pass closed
This pass turned most items below into real, tested code. Full detail in
`docs/ICEBERG_WRITE_BOUNDARY.md`; short version (re-verified this documentation-accuracy pass
against the actual files):
- Item 2 (`0pt.dataset_iceberg_mapping`): CLOSED as real code -- `src/types/dataset-iceberg-mapping.ts`,
`src/services/dataset-iceberg-mapping-service.ts`, `packages/schemas/0pt.dataset_iceberg_mapping.schema.json`,
`sql/004_iceberg_write_boundary.sql` (`opt_dataset_iceberg_mappings`), tested in
`tests/p0/dataset-iceberg-mapping.test.ts`. Field names differ slightly from the "suggested shape"
above (`dataset_iceberg_mapping_id` as PK, `iceberg_table_id` instead of `iceberg_table_ref`,
`mapping_status` instead of bare `entity_type`/`market`) -- reconciled against ST0RE's REAL
shipped `IcebergTable.tableId` shape (sax3l/st0re, not the "future" registry this doc assumed) and
against this session's own task spec; see the boundary doc for the full reconciliation note.
- Item 3 (first test dataset write): CLOSED against ST0RE's now-real (but not live-verified from
this environment) `/v1/st0re/lakehouse/*` API -- `src/clients/st0re-lakehouse-client.ts`,
`src/services/iceberg-write-service.ts`, proven in `tests/p0/iceberg-write-service.test.ts`
against that client's mock implementation (same mock/HTTP convention as `cl0ud-client.ts` etc).
NOT verified against a running ST0RE deployment in this pass -- see the boundary doc's
"Verification status" section and the `it.todo` in that test file.
- Item 4 (quality rules stay in 0pt): **NOT BUILT.** `src/services/quality-check-service.ts` and
`src/services/quality-report-service.ts` do not exist and never have (see the correction callout
above) -- there was nothing for this pass to leave "untouched." The *intent* (0pt owns QA logic,
N0D never does) is still the correct target and nothing since has contradicted it, but it is not
yet backed by any code. This is an open gap for a future implementation session, not a closed
item.
- Item 5 (`0pt.source_rate_policy`): CLOSED as real code -- `src/types/source-rate-policy.ts`,
`src/services/source-rate-policy-service.ts`, `packages/schemas/0pt.source_rate_policy.schema.json`,
`sql/004_iceberg_write_boundary.sql` (`opt_source_rate_policies`), tested in
`tests/p0/source-rate-policy.test.ts`. Kept `requests_per_second`-style fields (as `domain`/
`robots_ok`/`access`) as OPTIONAL alongside the task-spec's primary `rate_limit_per_minute`/
`crawl_budget`/`compliance_notes` fields -- both specs for this same contract are satisfied on one
row, not two competing shapes.
- Item 6 (four AUD0 events): CLOSED for all four names -- `0pt.dataset_iceberg_table_mapped` and
`0pt.iceberg_write_requested`/`0pt.iceberg_snapshot_created` emit from
`src/services/iceberg-write-service.ts` and `dataset-iceberg-mapping-service.ts`;
`0pt.source_rate_policy_created` emits from `src/services/source-rate-policy-service.ts`. All via
the existing `Aud0Client` (mock/no-op by default, same as every other AUD0 emission in this repo --
AUD0 itself still has no live ingestion endpoint).
- Item 7 (N0D heavy-write requests): CLOSED as real code -- `IcebergWriteService.requestIcebergWrite`
requests an N0D job (`0pt.iceberg_write_job`) only when `heavy: true`, never executes in-process;
tested in `tests/p0/iceberg-write-service.test.ts`.
- Item 8 (no lakehouse state outside ST0RE): CLOSED as a design property, not just a statement --
`IcebergWriteService` and `DatasetIcebergMappingService` hold zero table/snapshot fields; the only
thing 0pt persists is the mapping's `iceberg_table_id` reference. See the boundary doc.
- Item 1 (Iceberg canonical, Delta rejected): CONFIRMED, unchanged from `docs/10` --
`IcebergWriteService.mapDatasetToIcebergTable` defaults `table_format` to `"iceberg"`.
## To-do list
1. **Use Apache Iceberg as the canonical write format** for datasets 0pt produces, per
`docs/10-lakehouse-decision.md` / ADR-0030 (`n0d` repo). Do not adopt Delta Lake as a native
write target; Delta is import/export-bridge-only, ecosystem-wide.
2. **Add a `dataset_to_iceberg_table` mapping.** Every 0pt dataset (today: rows keyed by
`entity_type` + market in `dim_company` / `fact_financials` / `dim_contact` / `dim_ubo` /
`dim_vehicle`, per `docs/03-data-model.md`, or a catalog product / order delivery) needs a
stable mapping to the Iceberg table(s) it is written to. Schema name:
**`0pt.dataset_iceberg_mapping.schema.v1`**. Suggested shape (to be finalized by the
implementation session, following the `snake_case` / `uuid` PK / `created_at`+`updated_at`
conventions in `docs/03-data-model.md`):
- `dataset_id` — 0pt-side identifier (catalog product id, or `order_id`+`item_id` for custom
orders).
- `iceberg_table_ref` — fully-qualified table identifier as registered in ST0RE's table
registry (catalog.namespace.table).
- `entity_type`, `market` — for routing/filtering.
- `mapped_at`, `mapped_by`.
3. **Write a first test dataset to Iceberg** through ST0RE's table registry. *(Status: CLOSED per
the update above — `tests/p0/iceberg-write-service.test.ts` exercises exactly this pilot flow,
but only against `St0reLakehouseClient`'s mock; not yet run against a live ST0RE. Treat this
item as "code exists," not "validated end-to-end against a real deployment.")* Original plan:
pick a small, low-risk dataset (e.g. a single-market `dim_company` slice) as the pilot, to
validate the write path end-to-end before any production order relies on it — that live-ST0RE
validation still has not happened.
4. **Keep data quality rules in 0pt.** This is the correct *target* per the `n0d` audit, and
remains a spec-level design in `docs/06-data-engine.md` §QA (Great Expectations-style checks
for validity, completeness, freshness, uniqueness, corroboration), but **it is not built** —
there is no dedicated quality-check/quality-report service in this repo as of this writing (see
the correction callout at the top of this doc). Once it is built, the design intent stands:
quality logic must be caller-supplied logic that 0pt owns and hands to whichever engine executes
a job; it must not become logic owned or embedded by N0D, even after N0D executes heavy
writes/compactions on 0pt's behalf (see item 7). This item stays open until that service exists
and is tested.
5. **Keep source-rate/crawl-compliance policy logic in 0pt** — build a real model here, do not
rely on N0D for this. *(Status: CLOSED per the update above — `SourceRatePolicyService` is real
and tested.)* Context: N0D's `ComputePolicy.sourceRatePolicy` (per-domain crawl rate limits) has
been deprecated in N0D as part of the same audit (the field is still present on N0D's type for
backward compatibility, but N0D's own boundary-repair tests assert it now has zero effect on any
allow/deny decision — verified by reading `n0d`'s `src/policy.ts` and
`src/__tests__/boundary-repair.test.ts`) — that field never belonged to N0D. 0pt already has the
raw material for this in `sources.yaml`
(`docs/06-data-engine.md`, the `rate_limit` / `robots_ok` fields per source), but that's
currently free-text/per-entry, not a first-class policy object. Formalize it as
**`0pt.source_rate_policy.schema.v1`**. Suggested shape, following the same conventions:
- `source_id` — references `dim_source(source_id)` (`docs/03-data-model.md`).
- `domain` — the crawl-target domain (e.g. `allabolag.se`).
- `requests_per_second` / `requests_per_minute` — the compliance-derived rate limit.
- `robots_ok` — whether `robots.txt` permits the access pattern.
- `access` — mirrors `dim_source.access` (`api`|`bulk`|`authorized_web`|`licensed`).
- `created_at`, `updated_at`, `created_by`.
This model belongs entirely to 0pt's sourcing/compliance domain (`docs/06`) — it must not be
reintroduced into N0D, CL0UD, or any other product's policy defaults.
6. **Emit lineage for dataset → Iceberg table → snapshot**, to AUD0. *(Status: CLOSED per the
update above — all four events below are wired and fire through the existing `Aud0Client`,
which is mock/no-op by default because AUD0 still has no live ingestion endpoint. This did not
mean standing up a bespoke lineage sink prematurely — it means calling the same shared,
already-mocked AUD0 client interface every other 0pt event in this repo already calls.)* Plan
for three events, emitted by 0pt at the corresponding step:
- **`0pt.dataset_iceberg_table_mapped`** — fired when step 2's mapping is created/updated.
- **`0pt.iceberg_write_requested`** — fired when 0pt requests a write of a produced dataset
into its mapped Iceberg table (whether executed by 0pt directly or via an N0D job, per
item 7).
- **`0pt.iceberg_snapshot_created`** — fired once the write lands and a new Iceberg snapshot
exists for the table, carrying enough detail (table ref, snapshot id, row count, dataset id,
order id if applicable) for AUD0 to build lineage from source → dataset → Iceberg
snapshot.
- **`0pt.source_rate_policy_created`** — fired when a new `0pt.source_rate_policy.schema.v1`
record is created (or materially updated), so AUD0/CL0UD have an evented trail of 0pt's
crawl-compliance posture without needing to poll 0pt's DB.
7. **Request N0D jobs for heavy writes/compactions if/when needed, rather than 0pt executing
them itself.** *(Status: CLOSED per the update above — `IcebergWriteService.requestIcebergWrite`
requests an N0D job only when `heavy: true`, tested.)* 0pt produces and owns the dataset; if a
write or compaction is heavy enough to warrant dedicated compute (large backfills, table
maintenance), 0pt should submit that as a job request to N0D's execution layer rather than
running it in-process in `services/api` workers. N0D executes; it does not own the data, the
schema, or the decision of *what* to write.
8. **Do not persist lakehouse table state outside ST0RE.** *(Status: CLOSED as a design property
per the update above — `IcebergWriteService`/`DatasetIcebergMappingService` hold zero table or
snapshot state; the only thing 0pt persists is a reference `iceberg_table_id`.)* ST0RE's table
registry now exists as real code (not yet confirmed live/reachable from this repo). 0pt must not
stand up a parallel catalog/metadata store of its own (that would repeat the exact mistake this
plan is repairing: N0D building its own Iceberg catalog because nothing else had one yet).
## Required schema names (for the future implementation session)
- `0pt.dataset_iceberg_mapping.schema.v1`
- `0pt.source_rate_policy.schema.v1`
## Required future event names (for the future implementation session)
- `0pt.dataset_iceberg_table_mapped`
- `0pt.iceberg_write_requested`
- `0pt.iceberg_snapshot_created`
- `0pt.source_rate_policy_created`
## Sequencing / dependencies
*(Status note: this section described the original plan's dependencies. As of the 2026-07-12
update above, items 1, 2, 3, 5, 6, 7, 8 are built — mock-tested, not live-verified. Item 4
(data-quality/QA service) is the one item that remains genuinely not-started; see the correction
callout at the top of this doc. The dependency shape below is kept for the record and because
"mock-tested" is not the same as "the dependency is resolved" — see each item's status note above
for what's still actually pending.)*
- Items 2, 4, 5 have no hard dependency on another product and can be defined/built inside 0pt —
2 and 5 now are; 4 (the QA service) still is not.
- Items 1, 3, 8 depended on **ST0RE's table registry** existing. It now exists as real code in
`sax3l/st0re`; what remains outstanding is a *live, reachable* `ST0RE_URL` to verify against —
no code path in this repo reads `ST0RE_URL` from the environment automatically (same as
`CL0UD_URL`/`N0D_URL`/`AUD0_URL` — none of the platform clients have an env fallback wired; a
base URL must be passed in explicitly via each service's constructor options). Wiring `ST0RE_URL`
alone would not be enough, either: `sax3l/st0re`'s current `main` enforces `X-Tenant-Id`/
`X-Actor-Id` headers on the `getTable`/`createSnapshot` routes this write path calls, and this
repo's `St0reLakehouseClient` does not send them yet — see `docs/ICEBERG_WRITE_BOUNDARY.md`'s
"Real code, not live-verified" section for the full account (found during this verification pass).
- Item 6 depended on **AUD0** existing as a real evented product; it still does not have a live
ingestion endpoint, so the four events fire through `Aud0Client`'s mock/no-op path, same as every
other AUD0 emission in this repo.
- Item 7 assumes N0D's job-request surface (see `n0d` repo) is reachable from 0pt's orchestrator;
`IcebergWriteService` calls `N0dClient.createJob`, which — like the other platform clients — has
no live `N0D_URL` wired from this repo's environment either.
## Explicitly out of scope for 0pt (do not build here)
- An Iceberg catalog implementation, or any direct MinIO/Iceberg client code, ahead of ST0RE's
registry existing — that repeats the `n0d` mistake this plan is repairing.
- Query, compaction, or table-maintenance execution — that is N0D's job.
- A lineage UI or lineage storage — that is AUD0's job; 0pt only emits the events above.