Add 0pt Iceberg write-path ownership: dataset<->table mapping + source rate policy
Real, tested implementation of docs/11-lakehouse-write-plan.md's items 2-8 (the "future implementation session" that doc named): 0pt is now the canonical writer of Iceberg tables through ST0RE's real, shipped /v1/st0re/lakehouse/* table registry (sax3l/st0re commit 9ea0abb), never persisting lakehouse table/catalog/snapshot state itself. - src/types + src/services for 0pt.dataset_iceberg_mapping.schema.v1 (dataset -> ST0RE tableId reference) and 0pt.source_rate_policy.schema.v1 (the sourceRatePolicy ownership repair, moved out of N0D per ADR-0030). - src/clients/st0re-lakehouse-client.ts: new client matching ST0RE's actual camelCase route contract, with the same mock/HTTP boundary convention as this repo's existing cl0ud/n0d/aud0/st0re clients. Honest caveat in its module doc comment: not verified against a live ST0RE instance in this pass (no reachable ST0RE_URL); every test exercises the deterministic mock. - src/services/iceberg-write-service.ts: registers tables/snapshots through ST0RE, requests N0D jobs only for heavy writes (never executes them in-process), emits all four AUD0 event names (dataset_iceberg_table_mapped / iceberg_write_requested / iceberg_snapshot_created / source_rate_policy_created). - sql/004, packages/schemas/*.json, app/api/0pt/lakehouse + sources/rate-policies routes, tests/p0/*.test.ts (154 passed + 1 honest it.todo for the live-ST0RE case, up from 142 baseline, 0 failures). - Reconciled a real collision: an earlier untracked draft schema/test pair (tests/p0/enterprise-data-os-expansion.test.ts) asserted a different, now-superseded source_rate_policy shape; updated that one test block only. - docs/ICEBERG_WRITE_BOUNDARY.md states the ownership boundary plainly and is explicit throughout about what's real-and-tested vs. contract-only. docs/11-lakehouse-write-plan.md gets a status-update section, not a rewrite. - Data quality rules (quality-check-service.ts, quality-report-service.ts) were not touched -- confirmed still 0pt-owned. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,41 @@
|
||||
// 0pt.dataset_iceberg_mapping.schema.v1
|
||||
//
|
||||
// Maps a 0pt dataset to the Iceberg table that carries its canonical lakehouse
|
||||
// write path. `iceberg_table_id` is a REFERENCE into ST0RE's table registry
|
||||
// (IcebergTable.tableId, st0re_iceberg_tables) -- it is not a copy of ST0RE's
|
||||
// table metadata. 0pt does not persist table/catalog/snapshot state of its
|
||||
// own; this row exists only to answer "which Iceberg table does this dataset
|
||||
// write to", per docs/10-lakehouse-decision.md and
|
||||
// docs/ICEBERG_WRITE_BOUNDARY.md.
|
||||
|
||||
export type DatasetIcebergMappingStatus = "pending" | "active" | "deprecated" | "archived";
|
||||
|
||||
export interface DatasetIcebergMapping {
|
||||
dataset_iceberg_mapping_id: string;
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
dataset_id: string;
|
||||
/** Reference into ST0RE's table registry (IcebergTable.tableId). Not resolved or duplicated locally. */
|
||||
iceberg_table_id: string;
|
||||
/** ST0RE lakehouse namespace the table lives in, carried for convenience/filtering only -- ST0RE remains the source of truth. */
|
||||
namespace?: string;
|
||||
table_name?: string;
|
||||
mapping_status: DatasetIcebergMappingStatus;
|
||||
mapped_by?: string;
|
||||
correlation_id?: string;
|
||||
created_at: string;
|
||||
updated_at?: string;
|
||||
}
|
||||
|
||||
export interface CreateDatasetIcebergMappingInput {
|
||||
dataset_iceberg_mapping_id: string;
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
dataset_id: string;
|
||||
iceberg_table_id: string;
|
||||
namespace?: string;
|
||||
table_name?: string;
|
||||
mapping_status?: DatasetIcebergMappingStatus;
|
||||
mapped_by?: string;
|
||||
correlation_id?: string;
|
||||
}
|
||||
@@ -0,0 +1,57 @@
|
||||
// 0pt.source_rate_policy.schema.v1
|
||||
//
|
||||
// The genuine ownership repair this contract encodes: crawl-domain rate
|
||||
// limiting and compliance posture belongs to 0pt, not N0D. N0D's
|
||||
// `ComputePolicy.sourceRatePolicy` field (per-domain crawl rate limits) is
|
||||
// deprecated/removed from N0D's seed data per ADR-0030's audit -- 0pt already
|
||||
// owns the sourcing chain (docs/06-data-engine.md, `sources.yaml`) and is the
|
||||
// only product that should hold this policy.
|
||||
//
|
||||
// Field names reconcile two prior specs written for this same contract:
|
||||
// docs/11-lakehouse-write-plan.md item 5 (domain/requests_per_second/
|
||||
// robots_ok/access) and this pass's task spec (rate_limit_per_minute/
|
||||
// crawl_budget/compliance_notes). Both are kept: the task-spec fields are
|
||||
// primary, the docs/11 fields are additional optional detail for the same
|
||||
// row, not a competing shape.
|
||||
|
||||
export type SourceRatePolicyStatus = "active" | "paused" | "deprecated";
|
||||
|
||||
export interface SourceRatePolicy {
|
||||
source_rate_policy_id: string;
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
/** References source.source_id (src/types/source.ts). */
|
||||
source_id: string;
|
||||
rate_limit_per_minute: number;
|
||||
/** Max requests/pages/records this policy permits per crawl run or per day (unit documented in compliance_notes when non-obvious). */
|
||||
crawl_budget: number;
|
||||
/** Free-text compliance rationale: robots.txt posture, licence terms, contact/ToS reference. */
|
||||
compliance_notes?: string;
|
||||
/** Crawl-target domain, e.g. "allabolag.se" -- optional convenience field mirroring docs/06-data-engine.md's sources.yaml rate_limit entries. */
|
||||
domain?: string;
|
||||
/** Whether robots.txt permits the access pattern this policy governs. */
|
||||
robots_ok?: boolean;
|
||||
/** Mirrors dim_source.access (docs/03-data-model.md): api | bulk | authorized_web | licensed. */
|
||||
access?: "api" | "bulk" | "authorized_web" | "licensed";
|
||||
status: SourceRatePolicyStatus;
|
||||
correlation_id?: string;
|
||||
created_at: string;
|
||||
updated_at?: string;
|
||||
created_by?: string;
|
||||
}
|
||||
|
||||
export interface CreateSourceRatePolicyInput {
|
||||
source_rate_policy_id: string;
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
source_id: string;
|
||||
rate_limit_per_minute: number;
|
||||
crawl_budget: number;
|
||||
compliance_notes?: string;
|
||||
domain?: string;
|
||||
robots_ok?: boolean;
|
||||
access?: "api" | "bulk" | "authorized_web" | "licensed";
|
||||
status?: SourceRatePolicyStatus;
|
||||
correlation_id?: string;
|
||||
created_by?: string;
|
||||
}
|
||||
Reference in New Issue
Block a user