Add 0pt Iceberg write-path ownership: dataset<->table mapping + source rate policy

Real, tested implementation of docs/11-lakehouse-write-plan.md's items 2-8
(the "future implementation session" that doc named): 0pt is now the
canonical writer of Iceberg tables through ST0RE's real, shipped
/v1/st0re/lakehouse/* table registry (sax3l/st0re commit 9ea0abb), never
persisting lakehouse table/catalog/snapshot state itself.

- src/types + src/services for 0pt.dataset_iceberg_mapping.schema.v1
  (dataset -> ST0RE tableId reference) and 0pt.source_rate_policy.schema.v1
  (the sourceRatePolicy ownership repair, moved out of N0D per ADR-0030).
- src/clients/st0re-lakehouse-client.ts: new client matching ST0RE's actual
  camelCase route contract, with the same mock/HTTP boundary convention as
  this repo's existing cl0ud/n0d/aud0/st0re clients. Honest caveat in its
  module doc comment: not verified against a live ST0RE instance in this
  pass (no reachable ST0RE_URL); every test exercises the deterministic mock.
- src/services/iceberg-write-service.ts: registers tables/snapshots through
  ST0RE, requests N0D jobs only for heavy writes (never executes them
  in-process), emits all four AUD0 event names
  (dataset_iceberg_table_mapped / iceberg_write_requested /
  iceberg_snapshot_created / source_rate_policy_created).
- sql/004, packages/schemas/*.json, app/api/0pt/lakehouse + sources/rate-policies
  routes, tests/p0/*.test.ts (154 passed + 1 honest it.todo for the live-ST0RE
  case, up from 142 baseline, 0 failures).
- Reconciled a real collision: an earlier untracked draft schema/test pair
  (tests/p0/enterprise-data-os-expansion.test.ts) asserted a different,
  now-superseded source_rate_policy shape; updated that one test block only.
- docs/ICEBERG_WRITE_BOUNDARY.md states the ownership boundary plainly and
  is explicit throughout about what's real-and-tested vs. contract-only.
  docs/11-lakehouse-write-plan.md gets a status-update section, not a rewrite.
- Data quality rules (quality-check-service.ts, quality-report-service.ts)
  were not touched -- confirmed still 0pt-owned.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-12 19:46:47 +02:00
co-authored by Claude Sonnet 5
parent 2545218265
commit 35b6ec63bf
20 changed files with 1695 additions and 0 deletions
+185
View File
@@ -0,0 +1,185 @@
// ST0RE lakehouse client -- 0pt's ONLY path for writing Iceberg table/
// snapshot state. 0pt must never persist lakehouse table/catalog/snapshot
// state anywhere else (see docs/ICEBERG_WRITE_BOUNDARY.md).
//
// Distinct from `./st0re-client.ts` (the generic Artifact registry client,
// snake_case request shape) because it targets a different, newer route
// family on the SAME ST0RE service: `/v1/st0re/lakehouse/*`
// (sax3l/st0re `src/api/routes/lakehouse.ts`, commit 9ea0abb). That route
// family's Fastify handlers parse camelCase JSON bodies matching ST0RE's own
// `IcebergTableInput`/`TableSnapshotInput`/`LakehouseNamespaceInput` types
// (sax3l/st0re `src/types.ts`) -- this client's request/response shapes are
// camelCase deliberately, to match ST0RE's real contract byte-for-byte
// rather than 0pt's own snake_case domain convention. The service layer
// (`src/services/dataset-iceberg-mapping-service.ts`,
// `src/services/iceberg-write-service.ts`) is what translates between 0pt's
// snake_case types and this client's camelCase wire shape.
//
// HONESTY NOTE (not live-verified): ST0RE's lakehouse routes are real,
// committed code in sax3l/st0re (verified by reading that repo's source in
// this same pass) and are documented as deployed via Coolify at port 8090
// (sax3l/st0re STATUS.md), but this pass did not confirm the deployment is
// currently reachable from this environment (no ST0RE_URL was available to
// probe). Exactly like `cl0ud-client.ts`/`aud0-client.ts`/`n0d-client.ts`/
// `st0re-client.ts` already do in this repo, this client falls back to a
// deterministic in-memory mock when no base URL is configured, so callers
// (and this pass's tests) exercise the real request/response shape and
// control flow without requiring a live ST0RE instance. Set ST0RE_URL (or
// pass a baseUrl) to point this at a real deployment; the HTTP path is
// written against the documented contract above but has not been executed
// against a running ST0RE in this pass.
export type IcebergTableFormat = "iceberg" | "delta" | "hudi";
export type LakehouseObjectStatus = "active" | "deprecated" | "archived";
export type TableSnapshotType = "append" | "overwrite" | "delete" | "replace" | "schema_change" | "compaction";
export interface St0reLakehouseNamespaceInput {
tenantId: string;
projectId: string;
name: string;
ownerProduct: string;
}
export interface St0reLakehouseNamespace extends St0reLakehouseNamespaceInput {
namespaceId: string;
createdAt: number;
}
export interface St0reIcebergTableInput {
tenantId: string;
projectId: string;
namespace: string;
tableName: string;
tableFormat: IcebergTableFormat;
storageUri: string;
/** Opaque reference into the external Iceberg catalog -- ST0RE does not resolve it. */
catalogRef: string;
datasetId?: string;
ownerProduct: string;
schemaId?: string;
partitionSpec?: readonly string[];
retentionPolicy?: string;
dataResidency?: string;
piiLevel?: "none" | "low" | "medium" | "high";
createdBy: string;
}
export interface St0reIcebergTable extends St0reIcebergTableInput {
tableId: string;
currentSnapshotId?: string;
status: LakehouseObjectStatus;
createdAt: number;
artifactId?: string;
}
export interface St0reTableSnapshotInput {
snapshotType: TableSnapshotType;
schemaVersion: string;
/** URI of the Iceberg manifest list/file for this snapshot -- 0pt/N0D writes the actual bytes; ST0RE only tracks the pointer. */
manifestUri: string;
createdByJobId?: string;
createdByCapability?: string;
recordCount?: number;
sizeBytes?: number;
aud0EventId?: string;
}
export interface St0reTableSnapshot extends St0reTableSnapshotInput {
snapshotId: string;
tableId: string;
tenantId: string;
projectId: string;
createdAt: number;
}
export interface St0reLakehouseClient {
registerNamespace(input: St0reLakehouseNamespaceInput): Promise<St0reLakehouseNamespace>;
registerTable(input: St0reIcebergTableInput): Promise<St0reIcebergTable>;
getTable(tableId: string): Promise<St0reIcebergTable>;
createSnapshot(tableId: string, input: St0reTableSnapshotInput): Promise<St0reTableSnapshot>;
}
export function createSt0reLakehouseClient(baseUrl?: string): St0reLakehouseClient {
if (!baseUrl) return new MockSt0reLakehouseClient();
return new HttpSt0reLakehouseClient(baseUrl);
}
class MockSt0reLakehouseClient implements St0reLakehouseClient {
private tables: Map<string, St0reIcebergTable> = new Map();
async registerNamespace(input: St0reLakehouseNamespaceInput): Promise<St0reLakehouseNamespace> {
return { ...input, namespaceId: `mock_ns_${Date.now()}`, createdAt: Date.now() };
}
async registerTable(input: St0reIcebergTableInput): Promise<St0reIcebergTable> {
const table: St0reIcebergTable = {
...input,
tableId: `mock_table_${Date.now()}`,
status: "active",
piiLevel: input.piiLevel ?? "none",
createdAt: Date.now(),
};
this.tables.set(table.tableId, table);
return table;
}
async getTable(tableId: string): Promise<St0reIcebergTable> {
const table = this.tables.get(tableId);
if (!table) throw new Error(`ST0RE mock: table ${tableId} not found`);
return table;
}
async createSnapshot(tableId: string, input: St0reTableSnapshotInput): Promise<St0reTableSnapshot> {
const table = this.tables.get(tableId);
const snapshotId = `mock_snapshot_${Date.now()}`;
if (table) table.currentSnapshotId = snapshotId;
return {
...input,
snapshotId,
tableId,
tenantId: table?.tenantId ?? "unknown",
projectId: table?.projectId ?? "unknown",
createdAt: Date.now(),
};
}
}
class HttpSt0reLakehouseClient implements St0reLakehouseClient {
constructor(private baseUrl: string) {}
async registerNamespace(input: St0reLakehouseNamespaceInput): Promise<St0reLakehouseNamespace> {
const res = await fetch(`${this.baseUrl}/v1/st0re/lakehouse/namespaces`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(input),
});
if (!res.ok) throw new Error(`ST0RE lakehouse namespace error: ${res.status}`);
return res.json();
}
async registerTable(input: St0reIcebergTableInput): Promise<St0reIcebergTable> {
const res = await fetch(`${this.baseUrl}/v1/st0re/lakehouse/tables`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(input),
});
if (!res.ok) throw new Error(`ST0RE lakehouse table error: ${res.status}`);
return res.json();
}
async getTable(tableId: string): Promise<St0reIcebergTable> {
const res = await fetch(`${this.baseUrl}/v1/st0re/lakehouse/tables/${tableId}`);
if (!res.ok) throw new Error(`ST0RE lakehouse get-table error: ${res.status}`);
return res.json();
}
async createSnapshot(tableId: string, input: St0reTableSnapshotInput): Promise<St0reTableSnapshot> {
const res = await fetch(`${this.baseUrl}/v1/st0re/lakehouse/tables/${tableId}/snapshots`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(input),
});
if (!res.ok) throw new Error(`ST0RE lakehouse snapshot error: ${res.status}`);
return res.json();
}
}
@@ -0,0 +1,164 @@
import type {
DatasetIcebergMapping,
DatasetIcebergMappingStatus,
CreateDatasetIcebergMappingInput,
} from "../types/dataset-iceberg-mapping";
import { createAud0Client, type Aud0Client } from "../clients/aud0-client";
import { DatasetService } from "./dataset-service";
interface DatasetIcebergMappingRecord {
dataset_iceberg_mapping_id: string;
tenant_id: string;
project_id: string;
dataset_id: string;
iceberg_table_id: string;
namespace: string | null;
table_name: string | null;
mapping_status: string;
mapped_by: string | null;
correlation_id: string | null;
created_at: string;
updated_at: string | null;
}
/**
* Owns 0pt.dataset_iceberg_mapping.schema.v1 -- the pointer from a 0pt
* dataset to the Iceberg table it writes to. `iceberg_table_id` is always a
* REFERENCE into ST0RE's table registry (never a copy of ST0RE's table row);
* this service holds no table/catalog/snapshot state itself. See
* docs/ICEBERG_WRITE_BOUNDARY.md.
*
* This service does not call ST0RE directly -- it records a mapping to an
* `iceberg_table_id` that already exists in ST0RE's registry.
* `IcebergWriteService.mapDatasetToIcebergTable` (src/services/
* iceberg-write-service.ts) is the higher-level flow that registers a NEW
* table in ST0RE first and then calls this service to record the mapping --
* kept as two composable steps so a mapping can also be created against an
* already-registered table (e.g. one N0D or another 0pt job already set up).
*/
export class DatasetIcebergMappingService {
private mappings: Map<string, DatasetIcebergMappingRecord> = new Map();
private aud0: Aud0Client;
private datasetService: DatasetService;
constructor(opts?: { aud0BaseUrl?: string; datasetService?: DatasetService }) {
this.aud0 = createAud0Client(opts?.aud0BaseUrl);
this.datasetService = opts?.datasetService ?? new DatasetService(opts?.aud0BaseUrl);
}
async createMapping(input: CreateDatasetIcebergMappingInput): Promise<DatasetIcebergMapping> {
if (this.mappings.has(input.dataset_iceberg_mapping_id)) {
throw new Error(`Dataset Iceberg mapping ${input.dataset_iceberg_mapping_id} already exists`);
}
if (!input.iceberg_table_id) {
throw new Error("iceberg_table_id is required (reference into ST0RE's table registry)");
}
// Verify the dataset exists -- a mapping with no dataset is meaningless.
try {
await this.datasetService.getDataset(input.dataset_id);
} catch {
throw new Error(`Dataset ${input.dataset_id} not found`);
}
const now = new Date().toISOString();
const record: DatasetIcebergMappingRecord = {
dataset_iceberg_mapping_id: input.dataset_iceberg_mapping_id,
tenant_id: input.tenant_id,
project_id: input.project_id,
dataset_id: input.dataset_id,
iceberg_table_id: input.iceberg_table_id,
namespace: input.namespace ?? null,
table_name: input.table_name ?? null,
mapping_status: input.mapping_status ?? "active",
mapped_by: input.mapped_by ?? null,
correlation_id: input.correlation_id ?? null,
created_at: now,
updated_at: null,
};
this.mappings.set(input.dataset_iceberg_mapping_id, record);
// 0pt.dataset_iceberg_table_mapped -- docs/11-lakehouse-write-plan.md event list,
// docs/ICEBERG_WRITE_BOUNDARY.md. Fired when the mapping is created/updated.
await this.aud0.ingestEvent({
event_id: `evt_0pt_dataset_iceberg_table_mapped_${input.dataset_iceberg_mapping_id}`,
event_type: "0pt.dataset_iceberg_table_mapped",
tenant_id: input.tenant_id,
project_id: input.project_id,
actor_id: input.mapped_by ?? "system",
app_id: "0pt",
capability_id: "0pt.map_dataset_iceberg_table",
resource_type: "dataset_iceberg_mapping",
resource_id: input.dataset_iceberg_mapping_id,
risk_level: "L2",
payload: {
dataset_id: input.dataset_id,
iceberg_table_id: input.iceberg_table_id,
namespace: input.namespace,
table_name: input.table_name,
},
created_at: now,
correlation_id: input.correlation_id ?? "",
});
// Lineage: dataset -> iceberg table.
await this.aud0.ingestLineageEvent({
lineage_event_id: `lineage_0pt_dataset_iceberg_${input.dataset_iceberg_mapping_id}`,
tenant_id: input.tenant_id,
project_id: input.project_id,
app_id: "0pt",
capability_id: "0pt.map_dataset_iceberg_table",
source_type: "dataset",
source_id: input.dataset_id,
target_type: "iceberg_table",
target_id: input.iceberg_table_id,
lineage_type: "stored_as",
metadata: { dataset_iceberg_mapping_id: input.dataset_iceberg_mapping_id },
created_at: now,
correlation_id: input.correlation_id ?? "",
});
return this.toMapping(record);
}
async getMapping(mappingId: string): Promise<DatasetIcebergMapping> {
const record = this.mappings.get(mappingId);
if (!record) throw new Error(`Dataset Iceberg mapping ${mappingId} not found`);
return this.toMapping(record);
}
async getMappingForDataset(datasetId: string): Promise<DatasetIcebergMapping | undefined> {
const record = Array.from(this.mappings.values()).find((r) => r.dataset_id === datasetId && r.mapping_status === "active");
return record ? this.toMapping(record) : undefined;
}
async listMappings(filter?: {
tenant_id?: string;
project_id?: string;
dataset_id?: string;
mapping_status?: DatasetIcebergMappingStatus;
}): Promise<DatasetIcebergMapping[]> {
let results = Array.from(this.mappings.values());
if (filter?.tenant_id) results = results.filter((r) => r.tenant_id === filter.tenant_id);
if (filter?.project_id) results = results.filter((r) => r.project_id === filter.project_id);
if (filter?.dataset_id) results = results.filter((r) => r.dataset_id === filter.dataset_id);
if (filter?.mapping_status) results = results.filter((r) => r.mapping_status === filter.mapping_status);
return results.map((r) => this.toMapping(r));
}
private toMapping(r: DatasetIcebergMappingRecord): DatasetIcebergMapping {
return {
dataset_iceberg_mapping_id: r.dataset_iceberg_mapping_id,
tenant_id: r.tenant_id,
project_id: r.project_id,
dataset_id: r.dataset_id,
iceberg_table_id: r.iceberg_table_id,
namespace: r.namespace ?? undefined,
table_name: r.table_name ?? undefined,
mapping_status: r.mapping_status as DatasetIcebergMappingStatus,
mapped_by: r.mapped_by ?? undefined,
correlation_id: r.correlation_id ?? undefined,
created_at: r.created_at,
updated_at: r.updated_at ?? undefined,
};
}
}
+284
View File
@@ -0,0 +1,284 @@
import type { DatasetIcebergMapping } from "../types/dataset-iceberg-mapping";
import { createCl0udClient, type Cl0udClient } from "../clients/cl0ud-client";
import { createN0dClient, type N0dClient } from "../clients/n0d-client";
import { createAud0Client, type Aud0Client } from "../clients/aud0-client";
import {
createSt0reLakehouseClient,
type St0reLakehouseClient,
type IcebergTableFormat,
type TableSnapshotType,
} from "../clients/st0re-lakehouse-client";
import { DatasetService } from "./dataset-service";
import { DatasetIcebergMappingService } from "./dataset-iceberg-mapping-service";
/**
* IcebergWriteService -- 0pt's write-path entrypoint for Iceberg tables.
*
* This is the concrete implementation of docs/ICEBERG_WRITE_BOUNDARY.md /
* docs/11-lakehouse-write-plan.md items 1, 3, 6, 7, 8: 0pt is the canonical
* writer of Iceberg tables, and it writes THROUGH ST0RE's table registry
* (`St0reLakehouseClient`) rather than persisting any lakehouse table,
* catalog, or snapshot state itself. This service holds NO table/snapshot
* state of its own -- only `DatasetIcebergMappingService`'s lightweight
* dataset-to-table-id pointer, which is a reference, not a copy.
*
* Two entry points:
* - `mapDatasetToIcebergTable` -- registers a NEW Iceberg table in ST0RE for
* a dataset that does not have one yet, then records the mapping. This is
* the "first test dataset written to Iceberg through ST0RE's table
* registry" proof path (see tests/p0/iceberg-write-service.test.ts).
* - `requestIcebergWrite` -- registers a new snapshot (an actual write) for
* an already-mapped dataset, optionally routed through an N0D job when the
* write is heavy (item 7: 0pt REQUESTS N0D jobs for heavy writes/
* compactions, it does not execute them itself, and it never owns the
* result of that job beyond the ST0RE-registered snapshot ref).
*
* Byte-level writes (actually producing Parquet/Iceberg data files) are OUT
* OF SCOPE for this pass, same honest boundary ST0RE itself documents for
* its own snapshot routes ("ST0RE doesn't itself write the Iceberg
* metadata.json... this registry tracks the pointer/version history for
* whatever the caller's own catalog write produced" --
* sax3l/st0re src/api/routes/lakehouse.ts). What this service proves is the
* OWNERSHIP path -- that 0pt requests the write and registers its result
* through ST0RE, never anywhere else.
*/
export interface MapDatasetToIcebergTableInput {
tenant_id: string;
project_id: string;
dataset_id: string;
dataset_iceberg_mapping_id: string;
namespace: string;
table_name: string;
storage_uri: string;
catalog_ref: string;
table_format?: IcebergTableFormat;
actor_id: string;
correlation_id: string;
}
export interface MapDatasetToIcebergTableResult {
mapping: DatasetIcebergMapping;
iceberg_table_id: string;
policy_decision_id: string;
}
export interface RequestIcebergWriteInput {
dataset_iceberg_mapping_id: string;
actor_id: string;
correlation_id: string;
snapshot_type?: TableSnapshotType;
schema_version: string;
manifest_uri: string;
record_count?: number;
size_bytes?: number;
/** True for large backfills/table maintenance -- routes through an N0D job request rather than 0pt executing in-process (docs/11-lakehouse-write-plan.md item 7). */
heavy?: boolean;
}
export interface RequestIcebergWriteResult {
snapshot_id: string;
iceberg_table_id: string;
policy_decision_id: string;
n0d_job_id?: string;
}
export class IcebergWriteService {
private cl0ud: Cl0udClient;
private n0d: N0dClient;
private st0reLakehouse: St0reLakehouseClient;
private aud0: Aud0Client;
private datasetService: DatasetService;
private mappingService: DatasetIcebergMappingService;
constructor(opts?: {
cl0udBaseUrl?: string;
n0dBaseUrl?: string;
st0reBaseUrl?: string;
aud0BaseUrl?: string;
datasetService?: DatasetService;
mappingService?: DatasetIcebergMappingService;
}) {
this.cl0ud = createCl0udClient(opts?.cl0udBaseUrl);
this.n0d = createN0dClient(opts?.n0dBaseUrl);
this.st0reLakehouse = createSt0reLakehouseClient(opts?.st0reBaseUrl);
this.aud0 = createAud0Client(opts?.aud0BaseUrl);
this.datasetService = opts?.datasetService ?? new DatasetService(opts?.aud0BaseUrl);
this.mappingService = opts?.mappingService ?? new DatasetIcebergMappingService({ aud0BaseUrl: opts?.aud0BaseUrl, datasetService: this.datasetService });
}
async mapDatasetToIcebergTable(input: MapDatasetToIcebergTableInput): Promise<MapDatasetToIcebergTableResult> {
const dataset = await this.datasetService.getDataset(input.dataset_id); // throws if missing
// CL0UD policy boundary -- same L3 gate ST0RE's own table-registration
// route applies (sax3l/st0re src/api/routes/lakehouse.ts,
// 'st0re.lakehouse.write_table'), checked here too since 0pt is the
// caller initiating the write, not just relying on ST0RE to gate it.
const policy = await this.cl0ud.evaluatePolicy({
tenant_id: input.tenant_id,
project_id: input.project_id,
actor_id: input.actor_id,
capability_id: "0pt.write_iceberg_table",
resource_type: "dataset",
resource_id: input.dataset_id,
risk_level: "L3",
correlation_id: input.correlation_id,
});
if (policy.decision === "deny") throw new Error(`Policy denied: ${policy.reason}`);
if (policy.decision === "approval_required") throw new Error(`Policy requires approval: ${policy.reason}`);
// Register the table THROUGH ST0RE's table registry -- 0pt does not
// create or track lakehouse table state on its own.
const table = await this.st0reLakehouse.registerTable({
tenantId: input.tenant_id,
projectId: input.project_id,
namespace: input.namespace,
tableName: input.table_name,
tableFormat: input.table_format ?? "iceberg", // ADR-0030: Iceberg is canonical; Delta/Hudi accepted only for bridge/import scenarios, never chosen by default here.
storageUri: input.storage_uri,
catalogRef: input.catalog_ref,
datasetId: input.dataset_id,
ownerProduct: "0pt",
createdBy: input.actor_id,
});
const mapping = await this.mappingService.createMapping({
dataset_iceberg_mapping_id: input.dataset_iceberg_mapping_id,
tenant_id: input.tenant_id,
project_id: input.project_id,
dataset_id: input.dataset_id,
iceberg_table_id: table.tableId,
namespace: input.namespace,
table_name: input.table_name,
mapping_status: "active",
mapped_by: input.actor_id,
correlation_id: input.correlation_id,
});
return { mapping, iceberg_table_id: table.tableId, policy_decision_id: policy.decision_id };
}
async requestIcebergWrite(input: RequestIcebergWriteInput): Promise<RequestIcebergWriteResult> {
const mapping = await this.mappingService.getMapping(input.dataset_iceberg_mapping_id);
if (mapping.mapping_status !== "active") {
throw new Error(`Dataset Iceberg mapping ${input.dataset_iceberg_mapping_id} is not active (status=${mapping.mapping_status})`);
}
const policy = await this.cl0ud.evaluatePolicy({
tenant_id: mapping.tenant_id,
project_id: mapping.project_id,
actor_id: input.actor_id,
capability_id: "0pt.write_iceberg_table",
resource_type: "dataset",
resource_id: mapping.dataset_id,
risk_level: "L3",
correlation_id: input.correlation_id,
});
if (policy.decision === "deny") throw new Error(`Policy denied: ${policy.reason}`);
if (policy.decision === "approval_required") throw new Error(`Policy requires approval: ${policy.reason}`);
// 0pt.iceberg_write_requested -- fired when 0pt requests the write,
// before the write lands (docs/11-lakehouse-write-plan.md event list).
await this.aud0.ingestEvent({
event_id: `evt_0pt_iceberg_write_requested_${input.dataset_iceberg_mapping_id}_${Date.now()}`,
event_type: "0pt.iceberg_write_requested",
tenant_id: mapping.tenant_id,
project_id: mapping.project_id,
actor_id: input.actor_id,
app_id: "0pt",
capability_id: "0pt.write_iceberg_table",
resource_type: "iceberg_table",
resource_id: mapping.iceberg_table_id,
risk_level: "L3",
payload: { dataset_id: mapping.dataset_id, dataset_iceberg_mapping_id: mapping.dataset_iceberg_mapping_id, heavy: input.heavy ?? false },
created_at: new Date().toISOString(),
correlation_id: input.correlation_id,
});
// Heavy writes/compactions are REQUESTED from N0D, never executed
// in-process by 0pt (docs/11-lakehouse-write-plan.md item 7). N0D
// executes; it does not own the schema or the decision of what to
// write -- the snapshot registration below still goes through ST0RE
// regardless of who performed the write.
let n0dJobId: string | undefined;
if (input.heavy) {
const job = await this.n0d.createJob({
tenant_id: mapping.tenant_id,
project_id: mapping.project_id,
job_type: "0pt.iceberg_write_job",
capability_id: "0pt.write_iceberg_table",
requested_by_app_id: "0pt",
requested_by_actor_id: input.actor_id,
input_dataset_ids: [mapping.dataset_id],
output_artifact_type: "iceberg_table_snapshot",
requires_backbone: true,
requires_gpu: false,
required_node_roles: ["compute_general"],
resource_profile: "cpu_io_large",
risk_level: "L3",
correlation_id: input.correlation_id,
});
n0dJobId = job.job_id;
}
// Register the snapshot THROUGH ST0RE's table registry -- the one and
// only place 0pt persists that a write landed.
const snapshot = await this.st0reLakehouse.createSnapshot(mapping.iceberg_table_id, {
snapshotType: input.snapshot_type ?? "append",
schemaVersion: input.schema_version,
manifestUri: input.manifest_uri,
createdByJobId: n0dJobId,
createdByCapability: "0pt.write_iceberg_table",
recordCount: input.record_count,
sizeBytes: input.size_bytes,
});
// 0pt.iceberg_snapshot_created -- fired once the write lands and ST0RE
// has registered a new snapshot, carrying table ref + snapshot id + row
// count + dataset id for AUD0 to build source -> dataset -> snapshot
// lineage.
await this.aud0.ingestEvent({
event_id: `evt_0pt_iceberg_snapshot_created_${snapshot.snapshotId}`,
event_type: "0pt.iceberg_snapshot_created",
tenant_id: mapping.tenant_id,
project_id: mapping.project_id,
actor_id: input.actor_id,
app_id: "0pt",
capability_id: "0pt.write_iceberg_table",
resource_type: "iceberg_snapshot",
resource_id: snapshot.snapshotId,
risk_level: "L2",
payload: {
dataset_id: mapping.dataset_id,
iceberg_table_id: mapping.iceberg_table_id,
snapshot_id: snapshot.snapshotId,
record_count: input.record_count,
n0d_job_id: n0dJobId,
},
created_at: new Date().toISOString(),
correlation_id: input.correlation_id,
});
await this.aud0.ingestLineageEvent({
lineage_event_id: `lineage_0pt_iceberg_snapshot_${snapshot.snapshotId}`,
tenant_id: mapping.tenant_id,
project_id: mapping.project_id,
app_id: "0pt",
capability_id: "0pt.write_iceberg_table",
source_type: "dataset",
source_id: mapping.dataset_id,
target_type: "iceberg_snapshot",
target_id: snapshot.snapshotId,
lineage_type: "produced",
metadata: { iceberg_table_id: mapping.iceberg_table_id, n0d_job_id: n0dJobId },
created_at: new Date().toISOString(),
correlation_id: input.correlation_id,
});
return {
snapshot_id: snapshot.snapshotId,
iceberg_table_id: mapping.iceberg_table_id,
policy_decision_id: policy.decision_id,
n0d_job_id: n0dJobId,
};
}
}
+141
View File
@@ -0,0 +1,141 @@
import type { SourceRatePolicy, SourceRatePolicyStatus, CreateSourceRatePolicyInput } from "../types/source-rate-policy";
import { createAud0Client, type Aud0Client } from "../clients/aud0-client";
import { SourceService } from "./source-service";
interface SourceRatePolicyRecord {
source_rate_policy_id: string;
tenant_id: string;
project_id: string;
source_id: string;
rate_limit_per_minute: number;
crawl_budget: number;
compliance_notes: string | null;
domain: string | null;
robots_ok: boolean | null;
access: string | null;
status: string;
correlation_id: string | null;
created_at: string;
updated_at: string | null;
created_by: string | null;
}
/**
* Owns 0pt.source_rate_policy.schema.v1 -- the genuine ownership repair:
* source/crawl-domain rate limiting is 0pt's own compliance policy, moved
* OUT of N0D's ComputePolicy.sourceRatePolicy (deprecated there per
* ADR-0030's audit). See docs/ICEBERG_WRITE_BOUNDARY.md.
*/
export class SourceRatePolicyService {
private policies: Map<string, SourceRatePolicyRecord> = new Map();
private aud0: Aud0Client;
private sourceService: SourceService;
constructor(opts?: { aud0BaseUrl?: string; sourceService?: SourceService }) {
this.aud0 = createAud0Client(opts?.aud0BaseUrl);
this.sourceService = opts?.sourceService ?? new SourceService(opts?.aud0BaseUrl);
}
async createSourceRatePolicy(input: CreateSourceRatePolicyInput): Promise<SourceRatePolicy> {
if (this.policies.has(input.source_rate_policy_id)) {
throw new Error(`Source rate policy ${input.source_rate_policy_id} already exists`);
}
if (input.rate_limit_per_minute <= 0) {
throw new Error("rate_limit_per_minute must be > 0");
}
if (input.crawl_budget <= 0) {
throw new Error("crawl_budget must be > 0");
}
// Verify the source exists -- a rate policy with no source is meaningless.
try {
await this.sourceService.getSource(input.source_id);
} catch {
throw new Error(`Source ${input.source_id} not found`);
}
const now = new Date().toISOString();
const record: SourceRatePolicyRecord = {
source_rate_policy_id: input.source_rate_policy_id,
tenant_id: input.tenant_id,
project_id: input.project_id,
source_id: input.source_id,
rate_limit_per_minute: input.rate_limit_per_minute,
crawl_budget: input.crawl_budget,
compliance_notes: input.compliance_notes ?? null,
domain: input.domain ?? null,
robots_ok: input.robots_ok ?? null,
access: input.access ?? null,
status: input.status ?? "active",
correlation_id: input.correlation_id ?? null,
created_at: now,
updated_at: null,
created_by: input.created_by ?? null,
};
this.policies.set(input.source_rate_policy_id, record);
// 0pt.source_rate_policy_created -- docs/11-lakehouse-write-plan.md event list,
// docs/ICEBERG_WRITE_BOUNDARY.md. Gives AUD0/CL0UD an evented trail of 0pt's
// crawl-compliance posture without polling 0pt's DB.
await this.aud0.ingestEvent({
event_id: `evt_0pt_source_rate_policy_created_${input.source_rate_policy_id}`,
event_type: "0pt.source_rate_policy_created",
tenant_id: input.tenant_id,
project_id: input.project_id,
actor_id: input.created_by ?? "system",
app_id: "0pt",
capability_id: "0pt.create_source_rate_policy",
resource_type: "source_rate_policy",
resource_id: input.source_rate_policy_id,
risk_level: "L2",
payload: {
source_id: input.source_id,
rate_limit_per_minute: input.rate_limit_per_minute,
crawl_budget: input.crawl_budget,
},
created_at: now,
correlation_id: input.correlation_id ?? "",
});
return this.toPolicy(record);
}
async getSourceRatePolicy(sourceRatePolicyId: string): Promise<SourceRatePolicy> {
const record = this.policies.get(sourceRatePolicyId);
if (!record) throw new Error(`Source rate policy ${sourceRatePolicyId} not found`);
return this.toPolicy(record);
}
async listSourceRatePolicies(filter?: {
tenant_id?: string;
project_id?: string;
source_id?: string;
status?: SourceRatePolicyStatus;
}): Promise<SourceRatePolicy[]> {
let results = Array.from(this.policies.values());
if (filter?.tenant_id) results = results.filter((r) => r.tenant_id === filter.tenant_id);
if (filter?.project_id) results = results.filter((r) => r.project_id === filter.project_id);
if (filter?.source_id) results = results.filter((r) => r.source_id === filter.source_id);
if (filter?.status) results = results.filter((r) => r.status === filter.status);
return results.map((r) => this.toPolicy(r));
}
private toPolicy(r: SourceRatePolicyRecord): SourceRatePolicy {
return {
source_rate_policy_id: r.source_rate_policy_id,
tenant_id: r.tenant_id,
project_id: r.project_id,
source_id: r.source_id,
rate_limit_per_minute: r.rate_limit_per_minute,
crawl_budget: r.crawl_budget,
compliance_notes: r.compliance_notes ?? undefined,
domain: r.domain ?? undefined,
robots_ok: r.robots_ok ?? undefined,
access: (r.access as SourceRatePolicy["access"]) ?? undefined,
status: r.status as SourceRatePolicyStatus,
correlation_id: r.correlation_id ?? undefined,
created_at: r.created_at,
updated_at: r.updated_at ?? undefined,
created_by: r.created_by ?? undefined,
};
}
}
+41
View File
@@ -0,0 +1,41 @@
// 0pt.dataset_iceberg_mapping.schema.v1
//
// Maps a 0pt dataset to the Iceberg table that carries its canonical lakehouse
// write path. `iceberg_table_id` is a REFERENCE into ST0RE's table registry
// (IcebergTable.tableId, st0re_iceberg_tables) -- it is not a copy of ST0RE's
// table metadata. 0pt does not persist table/catalog/snapshot state of its
// own; this row exists only to answer "which Iceberg table does this dataset
// write to", per docs/10-lakehouse-decision.md and
// docs/ICEBERG_WRITE_BOUNDARY.md.
export type DatasetIcebergMappingStatus = "pending" | "active" | "deprecated" | "archived";
export interface DatasetIcebergMapping {
dataset_iceberg_mapping_id: string;
tenant_id: string;
project_id: string;
dataset_id: string;
/** Reference into ST0RE's table registry (IcebergTable.tableId). Not resolved or duplicated locally. */
iceberg_table_id: string;
/** ST0RE lakehouse namespace the table lives in, carried for convenience/filtering only -- ST0RE remains the source of truth. */
namespace?: string;
table_name?: string;
mapping_status: DatasetIcebergMappingStatus;
mapped_by?: string;
correlation_id?: string;
created_at: string;
updated_at?: string;
}
export interface CreateDatasetIcebergMappingInput {
dataset_iceberg_mapping_id: string;
tenant_id: string;
project_id: string;
dataset_id: string;
iceberg_table_id: string;
namespace?: string;
table_name?: string;
mapping_status?: DatasetIcebergMappingStatus;
mapped_by?: string;
correlation_id?: string;
}
+57
View File
@@ -0,0 +1,57 @@
// 0pt.source_rate_policy.schema.v1
//
// The genuine ownership repair this contract encodes: crawl-domain rate
// limiting and compliance posture belongs to 0pt, not N0D. N0D's
// `ComputePolicy.sourceRatePolicy` field (per-domain crawl rate limits) is
// deprecated/removed from N0D's seed data per ADR-0030's audit -- 0pt already
// owns the sourcing chain (docs/06-data-engine.md, `sources.yaml`) and is the
// only product that should hold this policy.
//
// Field names reconcile two prior specs written for this same contract:
// docs/11-lakehouse-write-plan.md item 5 (domain/requests_per_second/
// robots_ok/access) and this pass's task spec (rate_limit_per_minute/
// crawl_budget/compliance_notes). Both are kept: the task-spec fields are
// primary, the docs/11 fields are additional optional detail for the same
// row, not a competing shape.
export type SourceRatePolicyStatus = "active" | "paused" | "deprecated";
export interface SourceRatePolicy {
source_rate_policy_id: string;
tenant_id: string;
project_id: string;
/** References source.source_id (src/types/source.ts). */
source_id: string;
rate_limit_per_minute: number;
/** Max requests/pages/records this policy permits per crawl run or per day (unit documented in compliance_notes when non-obvious). */
crawl_budget: number;
/** Free-text compliance rationale: robots.txt posture, licence terms, contact/ToS reference. */
compliance_notes?: string;
/** Crawl-target domain, e.g. "allabolag.se" -- optional convenience field mirroring docs/06-data-engine.md's sources.yaml rate_limit entries. */
domain?: string;
/** Whether robots.txt permits the access pattern this policy governs. */
robots_ok?: boolean;
/** Mirrors dim_source.access (docs/03-data-model.md): api | bulk | authorized_web | licensed. */
access?: "api" | "bulk" | "authorized_web" | "licensed";
status: SourceRatePolicyStatus;
correlation_id?: string;
created_at: string;
updated_at?: string;
created_by?: string;
}
export interface CreateSourceRatePolicyInput {
source_rate_policy_id: string;
tenant_id: string;
project_id: string;
source_id: string;
rate_limit_per_minute: number;
crawl_budget: number;
compliance_notes?: string;
domain?: string;
robots_ok?: boolean;
access?: "api" | "bulk" | "authorized_web" | "licensed";
status?: SourceRatePolicyStatus;
correlation_id?: string;
created_by?: string;
}