Add 0pt Iceberg write-path ownership: dataset<->table mapping + source rate policy

Real, tested implementation of docs/11-lakehouse-write-plan.md's items 2-8
(the "future implementation session" that doc named): 0pt is now the
canonical writer of Iceberg tables through ST0RE's real, shipped
/v1/st0re/lakehouse/* table registry (sax3l/st0re commit 9ea0abb), never
persisting lakehouse table/catalog/snapshot state itself.

- src/types + src/services for 0pt.dataset_iceberg_mapping.schema.v1
  (dataset -> ST0RE tableId reference) and 0pt.source_rate_policy.schema.v1
  (the sourceRatePolicy ownership repair, moved out of N0D per ADR-0030).
- src/clients/st0re-lakehouse-client.ts: new client matching ST0RE's actual
  camelCase route contract, with the same mock/HTTP boundary convention as
  this repo's existing cl0ud/n0d/aud0/st0re clients. Honest caveat in its
  module doc comment: not verified against a live ST0RE instance in this
  pass (no reachable ST0RE_URL); every test exercises the deterministic mock.
- src/services/iceberg-write-service.ts: registers tables/snapshots through
  ST0RE, requests N0D jobs only for heavy writes (never executes them
  in-process), emits all four AUD0 event names
  (dataset_iceberg_table_mapped / iceberg_write_requested /
  iceberg_snapshot_created / source_rate_policy_created).
- sql/004, packages/schemas/*.json, app/api/0pt/lakehouse + sources/rate-policies
  routes, tests/p0/*.test.ts (154 passed + 1 honest it.todo for the live-ST0RE
  case, up from 142 baseline, 0 failures).
- Reconciled a real collision: an earlier untracked draft schema/test pair
  (tests/p0/enterprise-data-os-expansion.test.ts) asserted a different,
  now-superseded source_rate_policy shape; updated that one test block only.
- docs/ICEBERG_WRITE_BOUNDARY.md states the ownership boundary plainly and
  is explicit throughout about what's real-and-tested vs. contract-only.
  docs/11-lakehouse-write-plan.md gets a status-update section, not a rewrite.
- Data quality rules (quality-check-service.ts, quality-report-service.ts)
  were not touched -- confirmed still 0pt-owned.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-12 19:46:47 +02:00
co-authored by Claude Sonnet 5
parent 2545218265
commit 35b6ec63bf
20 changed files with 1695 additions and 0 deletions
+41
View File
@@ -0,0 +1,41 @@
// 0pt.dataset_iceberg_mapping.schema.v1
//
// Maps a 0pt dataset to the Iceberg table that carries its canonical lakehouse
// write path. `iceberg_table_id` is a REFERENCE into ST0RE's table registry
// (IcebergTable.tableId, st0re_iceberg_tables) -- it is not a copy of ST0RE's
// table metadata. 0pt does not persist table/catalog/snapshot state of its
// own; this row exists only to answer "which Iceberg table does this dataset
// write to", per docs/10-lakehouse-decision.md and
// docs/ICEBERG_WRITE_BOUNDARY.md.
export type DatasetIcebergMappingStatus = "pending" | "active" | "deprecated" | "archived";
export interface DatasetIcebergMapping {
dataset_iceberg_mapping_id: string;
tenant_id: string;
project_id: string;
dataset_id: string;
/** Reference into ST0RE's table registry (IcebergTable.tableId). Not resolved or duplicated locally. */
iceberg_table_id: string;
/** ST0RE lakehouse namespace the table lives in, carried for convenience/filtering only -- ST0RE remains the source of truth. */
namespace?: string;
table_name?: string;
mapping_status: DatasetIcebergMappingStatus;
mapped_by?: string;
correlation_id?: string;
created_at: string;
updated_at?: string;
}
export interface CreateDatasetIcebergMappingInput {
dataset_iceberg_mapping_id: string;
tenant_id: string;
project_id: string;
dataset_id: string;
iceberg_table_id: string;
namespace?: string;
table_name?: string;
mapping_status?: DatasetIcebergMappingStatus;
mapped_by?: string;
correlation_id?: string;
}
+57
View File
@@ -0,0 +1,57 @@
// 0pt.source_rate_policy.schema.v1
//
// The genuine ownership repair this contract encodes: crawl-domain rate
// limiting and compliance posture belongs to 0pt, not N0D. N0D's
// `ComputePolicy.sourceRatePolicy` field (per-domain crawl rate limits) is
// deprecated/removed from N0D's seed data per ADR-0030's audit -- 0pt already
// owns the sourcing chain (docs/06-data-engine.md, `sources.yaml`) and is the
// only product that should hold this policy.
//
// Field names reconcile two prior specs written for this same contract:
// docs/11-lakehouse-write-plan.md item 5 (domain/requests_per_second/
// robots_ok/access) and this pass's task spec (rate_limit_per_minute/
// crawl_budget/compliance_notes). Both are kept: the task-spec fields are
// primary, the docs/11 fields are additional optional detail for the same
// row, not a competing shape.
export type SourceRatePolicyStatus = "active" | "paused" | "deprecated";
export interface SourceRatePolicy {
source_rate_policy_id: string;
tenant_id: string;
project_id: string;
/** References source.source_id (src/types/source.ts). */
source_id: string;
rate_limit_per_minute: number;
/** Max requests/pages/records this policy permits per crawl run or per day (unit documented in compliance_notes when non-obvious). */
crawl_budget: number;
/** Free-text compliance rationale: robots.txt posture, licence terms, contact/ToS reference. */
compliance_notes?: string;
/** Crawl-target domain, e.g. "allabolag.se" -- optional convenience field mirroring docs/06-data-engine.md's sources.yaml rate_limit entries. */
domain?: string;
/** Whether robots.txt permits the access pattern this policy governs. */
robots_ok?: boolean;
/** Mirrors dim_source.access (docs/03-data-model.md): api | bulk | authorized_web | licensed. */
access?: "api" | "bulk" | "authorized_web" | "licensed";
status: SourceRatePolicyStatus;
correlation_id?: string;
created_at: string;
updated_at?: string;
created_by?: string;
}
export interface CreateSourceRatePolicyInput {
source_rate_policy_id: string;
tenant_id: string;
project_id: string;
source_id: string;
rate_limit_per_minute: number;
crawl_budget: number;
compliance_notes?: string;
domain?: string;
robots_ok?: boolean;
access?: "api" | "bulk" | "authorized_web" | "licensed";
status?: SourceRatePolicyStatus;
correlation_id?: string;
created_by?: string;
}