Merges github/main (6 commits: 17-bug audit fix, CI quality gate,
.env.example + .gitignore fix, missing platform clients + registry
services, docs accuracy overhaul) into gitea/main (8 commits: Iceberg
write boundary + live ST0RE verification, lakehouse mapping routes,
Waves 1-4 foundational hardening).
16 conflicts. Both sides had independently written the four
src/clients/*.ts and the two registry services, so those are combined
rather than picked:
- st0re-client.ts: GitHub's endpoint wins on evidence. POST goes to
/v1/st0re/artifacts/register, not /v1/st0re/artifacts -- verified in
sax3l/st0re src/api/routes/artifacts.ts, where the bare path is a
GET-only list route. Gitea's URL normalisation, error-body text and
mock kept.
- n0d-client.ts: field is `status`, not `state`, with N0D's real 13-state
union -- verified in sax3l/n0d packages/schemas/n0d.job.schema.json,
which says explicitly not to add "succeeded". Gitea's mock echo,
URL normalisation and error-body text kept.
- cl0ud-client.ts: Gitea's fail-closed HTTP normalisation kept (a 200
with no `decision` reads as "deny", not as a silent allow). GitHub's
PolicyDecision/PolicyDecisionValue naming kept; PolicyResult retained
as an alias.
- aud0-client.ts: permissive union of both input types (optional
payload/metadata/correlation_id), GitHub's echo return, Gitea's URL
normalisation and error-body text, plus a JSON-parse fallback so a
successful POST with an empty body does not throw.
- dataset-service.ts / source-service.ts: GitHub's record/DTO split and
event naming (0pt.source_created / 0pt.create_source, which the merged
docs already reference) plus Gitea's name validation, description /
created_by fields and actor_id derivation. List filters are the union.
- lakehouse mapping routes: GitHub's shared lakehouse-mapping-registry
singletons (without which every write 400'd "mapping not found") AND
Gitea's Zod parseBody validation. Both, not either.
- src/types/source.ts, src/types/dataset.ts: unions of both vocabularies;
GitHub's `access` legal-sourcing field kept.
- .env.example: GitHub's audited inventory plus Gitea's Zitadel /
AUTH_ENFORCE block.
- CLAUDE.md, docs/ICEBERG_WRITE_BOUNDARY.md: GitHub's corrections kept,
Gitea's live-verification kept and marked as superseding the
"would 400 outright" finding (Gitea had already added the required
X-Tenant-Id/X-Actor-Id headers).
- pnpm-lock.yaml: regenerated with pnpm 9.15.9, not merged.
Also corrects CLAUDE.md's claim that the banned services/scrapers stack
was "since removed" -- the removal PR is open, not merged, and this
branch inherits that code from Gitea main. Nothing here resurrects,
extends or re-wires any of it; services/scrapers/ is byte-identical to
Gitea main.
Verified on the merged tree: typecheck (src/ + tests/) clean;
17 test files, 121 passed, 1 skipped (the realinfra test skipping
honestly on ST0RE HTTP 401), 0 failures. Gitea main baseline was the
same 121/1; GitHub main baseline could not even `pnpm install`
("packages field missing or empty") -- Gitea's pnpm-workspace.yaml fix
resolves that. Whole-repo `typecheck:app` reports 9 errors, an identical
set to Gitea main's baseline, so it is advisory in both CI workflows
rather than deleted. Lint: 16 problems, identical to Gitea baseline.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
216 lines
16 KiB
Markdown
216 lines
16 KiB
Markdown
# 11 · Lakehouse write plan (0pt's to-do list)
|
|
|
|
Concrete follow-up to `docs/10-lakehouse-decision.md`. This is a planning doc for a **future**
|
|
implementation session — it names the schemas/events precisely so that session doesn't have to
|
|
guess, but it does not build the full pipeline today.
|
|
|
|
> **Correction (documentation-accuracy pass).** This doc previously stated that item 4 (quality
|
|
> rules staying in 0pt) was "already true today" and, in the update section below, that
|
|
> `src/services/quality-check-service.ts` and `src/services/quality-report-service.ts` were real,
|
|
> tested files that a later pass left "untouched." **That was fabricated.** `git log --all` for
|
|
> `src/services/quality-check-service.ts`, `src/services/quality-report-service.ts`,
|
|
> `src/types/quality-check.ts`, `src/types/quality-report.ts`, `sql/002_p0_dataset_quality.sql`,
|
|
> and `sql/003_*.sql` returns zero commits, on any branch (`main`, `feature/0pt-env-example`,
|
|
> `ci/add-quality-gate`), ever — these files have never existed in this repository. 0pt's
|
|
> data-quality/QA stage is a **design spec only** (`docs/06-data-engine.md` §QA — Great
|
|
> Expectations-style checks, intended to run inside the Python worker pipeline in `services/api`)
|
|
> and is **not yet implemented** as any dedicated service in this repo, TypeScript or otherwise.
|
|
> This is a real, currently open build gap, not a doc typo — see item 4's corrected entry below
|
|
> and `docs/ICEBERG_WRITE_BOUNDARY.md`'s "Data quality: not yet built" section for the full
|
|
> account. Everything else on this page (items 1, 2, 3, 5, 6, 7, 8) was independently re-verified
|
|
> against the actual service/client/test files this pass and remains accurate.
|
|
|
|
## 2026-07-13 update: live verification + field-level gap closure
|
|
|
|
A later pass live-verified the write path against `api.st0re.siax.io` (real namespace/table/snapshot
|
|
rows created), added the `namespace_id`/`mapped_at` fields the task spec names literally, and fixed
|
|
three real ST0RE contract bugs the code below had (missing tenant/actor headers, an invalid
|
|
`datasetId` FK reference, and a missing lookup-or-create for namespaces). Full detail in
|
|
`docs/ICEBERG_WRITE_BOUNDARY.md`'s "2026-07-13 update" section -- read that before this section
|
|
below, which is the original (2026-07-12) status record and is otherwise unchanged.
|
|
|
|
## 2026-07-12 update: what the 0pt write-boundary implementation pass closed
|
|
|
|
This pass turned most items below into real, tested code. Full detail in
|
|
`docs/ICEBERG_WRITE_BOUNDARY.md`; short version (re-verified this documentation-accuracy pass
|
|
against the actual files):
|
|
|
|
- Item 2 (`0pt.dataset_iceberg_mapping`): CLOSED as real code -- `src/types/dataset-iceberg-mapping.ts`,
|
|
`src/services/dataset-iceberg-mapping-service.ts`, `packages/schemas/0pt.dataset_iceberg_mapping.schema.json`,
|
|
`sql/004_iceberg_write_boundary.sql` (`opt_dataset_iceberg_mappings`), tested in
|
|
`tests/p0/dataset-iceberg-mapping.test.ts`. Field names differ slightly from the "suggested shape"
|
|
above (`dataset_iceberg_mapping_id` as PK, `iceberg_table_id` instead of `iceberg_table_ref`,
|
|
`mapping_status` instead of bare `entity_type`/`market`) -- reconciled against ST0RE's REAL
|
|
shipped `IcebergTable.tableId` shape (sax3l/st0re, not the "future" registry this doc assumed) and
|
|
against this session's own task spec; see the boundary doc for the full reconciliation note.
|
|
- Item 3 (first test dataset write): CLOSED against ST0RE's now-real (but not live-verified from
|
|
this environment) `/v1/st0re/lakehouse/*` API -- `src/clients/st0re-lakehouse-client.ts`,
|
|
`src/services/iceberg-write-service.ts`, proven in `tests/p0/iceberg-write-service.test.ts`
|
|
against that client's mock implementation (same mock/HTTP convention as `cl0ud-client.ts` etc).
|
|
NOT verified against a running ST0RE deployment in this pass -- see the boundary doc's
|
|
"Verification status" section and the `it.todo` in that test file.
|
|
- Item 4 (quality rules stay in 0pt): **NOT BUILT.** `src/services/quality-check-service.ts` and
|
|
`src/services/quality-report-service.ts` do not exist and never have (see the correction callout
|
|
above) -- there was nothing for this pass to leave "untouched." The *intent* (0pt owns QA logic,
|
|
N0D never does) is still the correct target and nothing since has contradicted it, but it is not
|
|
yet backed by any code. This is an open gap for a future implementation session, not a closed
|
|
item.
|
|
- Item 5 (`0pt.source_rate_policy`): CLOSED as real code -- `src/types/source-rate-policy.ts`,
|
|
`src/services/source-rate-policy-service.ts`, `packages/schemas/0pt.source_rate_policy.schema.json`,
|
|
`sql/004_iceberg_write_boundary.sql` (`opt_source_rate_policies`), tested in
|
|
`tests/p0/source-rate-policy.test.ts`. Kept `requests_per_second`-style fields (as `domain`/
|
|
`robots_ok`/`access`) as OPTIONAL alongside the task-spec's primary `rate_limit_per_minute`/
|
|
`crawl_budget`/`compliance_notes` fields -- both specs for this same contract are satisfied on one
|
|
row, not two competing shapes.
|
|
- Item 6 (four AUD0 events): CLOSED for all four names -- `0pt.dataset_iceberg_table_mapped` and
|
|
`0pt.iceberg_write_requested`/`0pt.iceberg_snapshot_created` emit from
|
|
`src/services/iceberg-write-service.ts` and `dataset-iceberg-mapping-service.ts`;
|
|
`0pt.source_rate_policy_created` emits from `src/services/source-rate-policy-service.ts`. All via
|
|
the existing `Aud0Client` (mock/no-op by default, same as every other AUD0 emission in this repo --
|
|
AUD0 itself still has no live ingestion endpoint).
|
|
- Item 7 (N0D heavy-write requests): CLOSED as real code -- `IcebergWriteService.requestIcebergWrite`
|
|
requests an N0D job (`0pt.iceberg_write_job`) only when `heavy: true`, never executes in-process;
|
|
tested in `tests/p0/iceberg-write-service.test.ts`.
|
|
- Item 8 (no lakehouse state outside ST0RE): CLOSED as a design property, not just a statement --
|
|
`IcebergWriteService` and `DatasetIcebergMappingService` hold zero table/snapshot fields; the only
|
|
thing 0pt persists is the mapping's `iceberg_table_id` reference. See the boundary doc.
|
|
- Item 1 (Iceberg canonical, Delta rejected): CONFIRMED, unchanged from `docs/10` --
|
|
`IcebergWriteService.mapDatasetToIcebergTable` defaults `table_format` to `"iceberg"`.
|
|
|
|
## To-do list
|
|
|
|
1. **Use Apache Iceberg as the canonical write format** for datasets 0pt produces, per
|
|
`docs/10-lakehouse-decision.md` / ADR-0030 (`n0d` repo). Do not adopt Delta Lake as a native
|
|
write target; Delta is import/export-bridge-only, ecosystem-wide.
|
|
|
|
2. **Add a `dataset_to_iceberg_table` mapping.** Every 0pt dataset (today: rows keyed by
|
|
`entity_type` + market in `dim_company` / `fact_financials` / `dim_contact` / `dim_ubo` /
|
|
`dim_vehicle`, per `docs/03-data-model.md`, or a catalog product / order delivery) needs a
|
|
stable mapping to the Iceberg table(s) it is written to. Schema name:
|
|
**`0pt.dataset_iceberg_mapping.schema.v1`**. Suggested shape (to be finalized by the
|
|
implementation session, following the `snake_case` / `uuid` PK / `created_at`+`updated_at`
|
|
conventions in `docs/03-data-model.md`):
|
|
- `dataset_id` — 0pt-side identifier (catalog product id, or `order_id`+`item_id` for custom
|
|
orders).
|
|
- `iceberg_table_ref` — fully-qualified table identifier as registered in ST0RE's table
|
|
registry (catalog.namespace.table).
|
|
- `entity_type`, `market` — for routing/filtering.
|
|
- `mapped_at`, `mapped_by`.
|
|
|
|
3. **Write a first test dataset to Iceberg** through ST0RE's table registry. *(Status: CLOSED per
|
|
the update above — `tests/p0/iceberg-write-service.test.ts` exercises exactly this pilot flow,
|
|
but only against `St0reLakehouseClient`'s mock; not yet run against a live ST0RE. Treat this
|
|
item as "code exists," not "validated end-to-end against a real deployment.")* Original plan:
|
|
pick a small, low-risk dataset (e.g. a single-market `dim_company` slice) as the pilot, to
|
|
validate the write path end-to-end before any production order relies on it — that live-ST0RE
|
|
validation still has not happened.
|
|
|
|
4. **Keep data quality rules in 0pt.** This is the correct *target* per the `n0d` audit, and
|
|
remains a spec-level design in `docs/06-data-engine.md` §QA (Great Expectations-style checks
|
|
for validity, completeness, freshness, uniqueness, corroboration), but **it is not built** —
|
|
there is no dedicated quality-check/quality-report service in this repo as of this writing (see
|
|
the correction callout at the top of this doc). Once it is built, the design intent stands:
|
|
quality logic must be caller-supplied logic that 0pt owns and hands to whichever engine executes
|
|
a job; it must not become logic owned or embedded by N0D, even after N0D executes heavy
|
|
writes/compactions on 0pt's behalf (see item 7). This item stays open until that service exists
|
|
and is tested.
|
|
|
|
5. **Keep source-rate/crawl-compliance policy logic in 0pt** — build a real model here, do not
|
|
rely on N0D for this. *(Status: CLOSED per the update above — `SourceRatePolicyService` is real
|
|
and tested.)* Context: N0D's `ComputePolicy.sourceRatePolicy` (per-domain crawl rate limits) has
|
|
been deprecated in N0D as part of the same audit (the field is still present on N0D's type for
|
|
backward compatibility, but N0D's own boundary-repair tests assert it now has zero effect on any
|
|
allow/deny decision — verified by reading `n0d`'s `src/policy.ts` and
|
|
`src/__tests__/boundary-repair.test.ts`) — that field never belonged to N0D. 0pt already has the
|
|
raw material for this in `sources.yaml`
|
|
(`docs/06-data-engine.md`, the `rate_limit` / `robots_ok` fields per source), but that's
|
|
currently free-text/per-entry, not a first-class policy object. Formalize it as
|
|
**`0pt.source_rate_policy.schema.v1`**. Suggested shape, following the same conventions:
|
|
- `source_id` — references `dim_source(source_id)` (`docs/03-data-model.md`).
|
|
- `domain` — the crawl-target domain (e.g. `allabolag.se`).
|
|
- `requests_per_second` / `requests_per_minute` — the compliance-derived rate limit.
|
|
- `robots_ok` — whether `robots.txt` permits the access pattern.
|
|
- `access` — mirrors `dim_source.access` (`api`|`bulk`|`authorized_web`|`licensed`).
|
|
- `created_at`, `updated_at`, `created_by`.
|
|
This model belongs entirely to 0pt's sourcing/compliance domain (`docs/06`) — it must not be
|
|
reintroduced into N0D, CL0UD, or any other product's policy defaults.
|
|
|
|
6. **Emit lineage for dataset → Iceberg table → snapshot**, to AUD0. *(Status: CLOSED per the
|
|
update above — all four events below are wired and fire through the existing `Aud0Client`,
|
|
which is mock/no-op by default because AUD0 still has no live ingestion endpoint. This did not
|
|
mean standing up a bespoke lineage sink prematurely — it means calling the same shared,
|
|
already-mocked AUD0 client interface every other 0pt event in this repo already calls.)* Plan
|
|
for three events, emitted by 0pt at the corresponding step:
|
|
- **`0pt.dataset_iceberg_table_mapped`** — fired when step 2's mapping is created/updated.
|
|
- **`0pt.iceberg_write_requested`** — fired when 0pt requests a write of a produced dataset
|
|
into its mapped Iceberg table (whether executed by 0pt directly or via an N0D job, per
|
|
item 7).
|
|
- **`0pt.iceberg_snapshot_created`** — fired once the write lands and a new Iceberg snapshot
|
|
exists for the table, carrying enough detail (table ref, snapshot id, row count, dataset id,
|
|
order id if applicable) for AUD0 to build lineage from source → dataset → Iceberg
|
|
snapshot.
|
|
- **`0pt.source_rate_policy_created`** — fired when a new `0pt.source_rate_policy.schema.v1`
|
|
record is created (or materially updated), so AUD0/CL0UD have an evented trail of 0pt's
|
|
crawl-compliance posture without needing to poll 0pt's DB.
|
|
|
|
7. **Request N0D jobs for heavy writes/compactions if/when needed, rather than 0pt executing
|
|
them itself.** *(Status: CLOSED per the update above — `IcebergWriteService.requestIcebergWrite`
|
|
requests an N0D job only when `heavy: true`, tested.)* 0pt produces and owns the dataset; if a
|
|
write or compaction is heavy enough to warrant dedicated compute (large backfills, table
|
|
maintenance), 0pt should submit that as a job request to N0D's execution layer rather than
|
|
running it in-process in `services/api` workers. N0D executes; it does not own the data, the
|
|
schema, or the decision of *what* to write.
|
|
|
|
8. **Do not persist lakehouse table state outside ST0RE.** *(Status: CLOSED as a design property
|
|
per the update above — `IcebergWriteService`/`DatasetIcebergMappingService` hold zero table or
|
|
snapshot state; the only thing 0pt persists is a reference `iceberg_table_id`.)* ST0RE's table
|
|
registry now exists as real code (not yet confirmed live/reachable from this repo). 0pt must not
|
|
stand up a parallel catalog/metadata store of its own (that would repeat the exact mistake this
|
|
plan is repairing: N0D building its own Iceberg catalog because nothing else had one yet).
|
|
|
|
## Required schema names (for the future implementation session)
|
|
|
|
- `0pt.dataset_iceberg_mapping.schema.v1`
|
|
- `0pt.source_rate_policy.schema.v1`
|
|
|
|
## Required future event names (for the future implementation session)
|
|
|
|
- `0pt.dataset_iceberg_table_mapped`
|
|
- `0pt.iceberg_write_requested`
|
|
- `0pt.iceberg_snapshot_created`
|
|
- `0pt.source_rate_policy_created`
|
|
|
|
## Sequencing / dependencies
|
|
|
|
*(Status note: this section described the original plan's dependencies. As of the 2026-07-12
|
|
update above, items 1, 2, 3, 5, 6, 7, 8 are built — mock-tested, not live-verified. Item 4
|
|
(data-quality/QA service) is the one item that remains genuinely not-started; see the correction
|
|
callout at the top of this doc. The dependency shape below is kept for the record and because
|
|
"mock-tested" is not the same as "the dependency is resolved" — see each item's status note above
|
|
for what's still actually pending.)*
|
|
|
|
- Items 2, 4, 5 have no hard dependency on another product and can be defined/built inside 0pt —
|
|
2 and 5 now are; 4 (the QA service) still is not.
|
|
- Items 1, 3, 8 depended on **ST0RE's table registry** existing. It now exists as real code in
|
|
`sax3l/st0re`; what remains outstanding is a *live, reachable* `ST0RE_URL` to verify against —
|
|
no code path in this repo reads `ST0RE_URL` from the environment automatically (same as
|
|
`CL0UD_URL`/`N0D_URL`/`AUD0_URL` — none of the platform clients have an env fallback wired; a
|
|
base URL must be passed in explicitly via each service's constructor options). Wiring `ST0RE_URL`
|
|
alone would not be enough, either: `sax3l/st0re`'s current `main` enforces `X-Tenant-Id`/
|
|
`X-Actor-Id` headers on the `getTable`/`createSnapshot` routes this write path calls, and this
|
|
repo's `St0reLakehouseClient` does not send them yet — see `docs/ICEBERG_WRITE_BOUNDARY.md`'s
|
|
"Real code, not live-verified" section for the full account (found during this verification pass).
|
|
- Item 6 depended on **AUD0** existing as a real evented product; it still does not have a live
|
|
ingestion endpoint, so the four events fire through `Aud0Client`'s mock/no-op path, same as every
|
|
other AUD0 emission in this repo.
|
|
- Item 7 assumes N0D's job-request surface (see `n0d` repo) is reachable from 0pt's orchestrator;
|
|
`IcebergWriteService` calls `N0dClient.createJob`, which — like the other platform clients — has
|
|
no live `N0D_URL` wired from this repo's environment either.
|
|
|
|
## Explicitly out of scope for 0pt (do not build here)
|
|
|
|
- An Iceberg catalog implementation, or any direct MinIO/Iceberg client code, ahead of ST0RE's
|
|
registry existing — that repeats the `n0d` mistake this plan is repairing.
|
|
- Query, compaction, or table-maintenance execution — that is N0D's job.
|
|
- A lineage UI or lineage storage — that is AUD0's job; 0pt only emits the events above.
|