Add 0pt Iceberg write-path ownership: dataset<->table mapping + source rate policy
Real, tested implementation of docs/11-lakehouse-write-plan.md's items 2-8 (the "future implementation session" that doc named): 0pt is now the canonical writer of Iceberg tables through ST0RE's real, shipped /v1/st0re/lakehouse/* table registry (sax3l/st0re commit 9ea0abb), never persisting lakehouse table/catalog/snapshot state itself. - src/types + src/services for 0pt.dataset_iceberg_mapping.schema.v1 (dataset -> ST0RE tableId reference) and 0pt.source_rate_policy.schema.v1 (the sourceRatePolicy ownership repair, moved out of N0D per ADR-0030). - src/clients/st0re-lakehouse-client.ts: new client matching ST0RE's actual camelCase route contract, with the same mock/HTTP boundary convention as this repo's existing cl0ud/n0d/aud0/st0re clients. Honest caveat in its module doc comment: not verified against a live ST0RE instance in this pass (no reachable ST0RE_URL); every test exercises the deterministic mock. - src/services/iceberg-write-service.ts: registers tables/snapshots through ST0RE, requests N0D jobs only for heavy writes (never executes them in-process), emits all four AUD0 event names (dataset_iceberg_table_mapped / iceberg_write_requested / iceberg_snapshot_created / source_rate_policy_created). - sql/004, packages/schemas/*.json, app/api/0pt/lakehouse + sources/rate-policies routes, tests/p0/*.test.ts (154 passed + 1 honest it.todo for the live-ST0RE case, up from 142 baseline, 0 failures). - Reconciled a real collision: an earlier untracked draft schema/test pair (tests/p0/enterprise-data-os-expansion.test.ts) asserted a different, now-superseded source_rate_policy shape; updated that one test block only. - docs/ICEBERG_WRITE_BOUNDARY.md states the ownership boundary plainly and is explicit throughout about what's real-and-tested vs. contract-only. docs/11-lakehouse-write-plan.md gets a status-update section, not a rewrite. - Data quality rules (quality-check-service.ts, quality-report-service.ts) were not touched -- confirmed still 0pt-owned. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,185 @@
|
||||
// ST0RE lakehouse client -- 0pt's ONLY path for writing Iceberg table/
|
||||
// snapshot state. 0pt must never persist lakehouse table/catalog/snapshot
|
||||
// state anywhere else (see docs/ICEBERG_WRITE_BOUNDARY.md).
|
||||
//
|
||||
// Distinct from `./st0re-client.ts` (the generic Artifact registry client,
|
||||
// snake_case request shape) because it targets a different, newer route
|
||||
// family on the SAME ST0RE service: `/v1/st0re/lakehouse/*`
|
||||
// (sax3l/st0re `src/api/routes/lakehouse.ts`, commit 9ea0abb). That route
|
||||
// family's Fastify handlers parse camelCase JSON bodies matching ST0RE's own
|
||||
// `IcebergTableInput`/`TableSnapshotInput`/`LakehouseNamespaceInput` types
|
||||
// (sax3l/st0re `src/types.ts`) -- this client's request/response shapes are
|
||||
// camelCase deliberately, to match ST0RE's real contract byte-for-byte
|
||||
// rather than 0pt's own snake_case domain convention. The service layer
|
||||
// (`src/services/dataset-iceberg-mapping-service.ts`,
|
||||
// `src/services/iceberg-write-service.ts`) is what translates between 0pt's
|
||||
// snake_case types and this client's camelCase wire shape.
|
||||
//
|
||||
// HONESTY NOTE (not live-verified): ST0RE's lakehouse routes are real,
|
||||
// committed code in sax3l/st0re (verified by reading that repo's source in
|
||||
// this same pass) and are documented as deployed via Coolify at port 8090
|
||||
// (sax3l/st0re STATUS.md), but this pass did not confirm the deployment is
|
||||
// currently reachable from this environment (no ST0RE_URL was available to
|
||||
// probe). Exactly like `cl0ud-client.ts`/`aud0-client.ts`/`n0d-client.ts`/
|
||||
// `st0re-client.ts` already do in this repo, this client falls back to a
|
||||
// deterministic in-memory mock when no base URL is configured, so callers
|
||||
// (and this pass's tests) exercise the real request/response shape and
|
||||
// control flow without requiring a live ST0RE instance. Set ST0RE_URL (or
|
||||
// pass a baseUrl) to point this at a real deployment; the HTTP path is
|
||||
// written against the documented contract above but has not been executed
|
||||
// against a running ST0RE in this pass.
|
||||
|
||||
export type IcebergTableFormat = "iceberg" | "delta" | "hudi";
|
||||
export type LakehouseObjectStatus = "active" | "deprecated" | "archived";
|
||||
export type TableSnapshotType = "append" | "overwrite" | "delete" | "replace" | "schema_change" | "compaction";
|
||||
|
||||
export interface St0reLakehouseNamespaceInput {
|
||||
tenantId: string;
|
||||
projectId: string;
|
||||
name: string;
|
||||
ownerProduct: string;
|
||||
}
|
||||
|
||||
export interface St0reLakehouseNamespace extends St0reLakehouseNamespaceInput {
|
||||
namespaceId: string;
|
||||
createdAt: number;
|
||||
}
|
||||
|
||||
export interface St0reIcebergTableInput {
|
||||
tenantId: string;
|
||||
projectId: string;
|
||||
namespace: string;
|
||||
tableName: string;
|
||||
tableFormat: IcebergTableFormat;
|
||||
storageUri: string;
|
||||
/** Opaque reference into the external Iceberg catalog -- ST0RE does not resolve it. */
|
||||
catalogRef: string;
|
||||
datasetId?: string;
|
||||
ownerProduct: string;
|
||||
schemaId?: string;
|
||||
partitionSpec?: readonly string[];
|
||||
retentionPolicy?: string;
|
||||
dataResidency?: string;
|
||||
piiLevel?: "none" | "low" | "medium" | "high";
|
||||
createdBy: string;
|
||||
}
|
||||
|
||||
export interface St0reIcebergTable extends St0reIcebergTableInput {
|
||||
tableId: string;
|
||||
currentSnapshotId?: string;
|
||||
status: LakehouseObjectStatus;
|
||||
createdAt: number;
|
||||
artifactId?: string;
|
||||
}
|
||||
|
||||
export interface St0reTableSnapshotInput {
|
||||
snapshotType: TableSnapshotType;
|
||||
schemaVersion: string;
|
||||
/** URI of the Iceberg manifest list/file for this snapshot -- 0pt/N0D writes the actual bytes; ST0RE only tracks the pointer. */
|
||||
manifestUri: string;
|
||||
createdByJobId?: string;
|
||||
createdByCapability?: string;
|
||||
recordCount?: number;
|
||||
sizeBytes?: number;
|
||||
aud0EventId?: string;
|
||||
}
|
||||
|
||||
export interface St0reTableSnapshot extends St0reTableSnapshotInput {
|
||||
snapshotId: string;
|
||||
tableId: string;
|
||||
tenantId: string;
|
||||
projectId: string;
|
||||
createdAt: number;
|
||||
}
|
||||
|
||||
export interface St0reLakehouseClient {
|
||||
registerNamespace(input: St0reLakehouseNamespaceInput): Promise<St0reLakehouseNamespace>;
|
||||
registerTable(input: St0reIcebergTableInput): Promise<St0reIcebergTable>;
|
||||
getTable(tableId: string): Promise<St0reIcebergTable>;
|
||||
createSnapshot(tableId: string, input: St0reTableSnapshotInput): Promise<St0reTableSnapshot>;
|
||||
}
|
||||
|
||||
export function createSt0reLakehouseClient(baseUrl?: string): St0reLakehouseClient {
|
||||
if (!baseUrl) return new MockSt0reLakehouseClient();
|
||||
return new HttpSt0reLakehouseClient(baseUrl);
|
||||
}
|
||||
|
||||
class MockSt0reLakehouseClient implements St0reLakehouseClient {
|
||||
private tables: Map<string, St0reIcebergTable> = new Map();
|
||||
|
||||
async registerNamespace(input: St0reLakehouseNamespaceInput): Promise<St0reLakehouseNamespace> {
|
||||
return { ...input, namespaceId: `mock_ns_${Date.now()}`, createdAt: Date.now() };
|
||||
}
|
||||
|
||||
async registerTable(input: St0reIcebergTableInput): Promise<St0reIcebergTable> {
|
||||
const table: St0reIcebergTable = {
|
||||
...input,
|
||||
tableId: `mock_table_${Date.now()}`,
|
||||
status: "active",
|
||||
piiLevel: input.piiLevel ?? "none",
|
||||
createdAt: Date.now(),
|
||||
};
|
||||
this.tables.set(table.tableId, table);
|
||||
return table;
|
||||
}
|
||||
|
||||
async getTable(tableId: string): Promise<St0reIcebergTable> {
|
||||
const table = this.tables.get(tableId);
|
||||
if (!table) throw new Error(`ST0RE mock: table ${tableId} not found`);
|
||||
return table;
|
||||
}
|
||||
|
||||
async createSnapshot(tableId: string, input: St0reTableSnapshotInput): Promise<St0reTableSnapshot> {
|
||||
const table = this.tables.get(tableId);
|
||||
const snapshotId = `mock_snapshot_${Date.now()}`;
|
||||
if (table) table.currentSnapshotId = snapshotId;
|
||||
return {
|
||||
...input,
|
||||
snapshotId,
|
||||
tableId,
|
||||
tenantId: table?.tenantId ?? "unknown",
|
||||
projectId: table?.projectId ?? "unknown",
|
||||
createdAt: Date.now(),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
class HttpSt0reLakehouseClient implements St0reLakehouseClient {
|
||||
constructor(private baseUrl: string) {}
|
||||
|
||||
async registerNamespace(input: St0reLakehouseNamespaceInput): Promise<St0reLakehouseNamespace> {
|
||||
const res = await fetch(`${this.baseUrl}/v1/st0re/lakehouse/namespaces`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify(input),
|
||||
});
|
||||
if (!res.ok) throw new Error(`ST0RE lakehouse namespace error: ${res.status}`);
|
||||
return res.json();
|
||||
}
|
||||
|
||||
async registerTable(input: St0reIcebergTableInput): Promise<St0reIcebergTable> {
|
||||
const res = await fetch(`${this.baseUrl}/v1/st0re/lakehouse/tables`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify(input),
|
||||
});
|
||||
if (!res.ok) throw new Error(`ST0RE lakehouse table error: ${res.status}`);
|
||||
return res.json();
|
||||
}
|
||||
|
||||
async getTable(tableId: string): Promise<St0reIcebergTable> {
|
||||
const res = await fetch(`${this.baseUrl}/v1/st0re/lakehouse/tables/${tableId}`);
|
||||
if (!res.ok) throw new Error(`ST0RE lakehouse get-table error: ${res.status}`);
|
||||
return res.json();
|
||||
}
|
||||
|
||||
async createSnapshot(tableId: string, input: St0reTableSnapshotInput): Promise<St0reTableSnapshot> {
|
||||
const res = await fetch(`${this.baseUrl}/v1/st0re/lakehouse/tables/${tableId}/snapshots`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify(input),
|
||||
});
|
||||
if (!res.ok) throw new Error(`ST0RE lakehouse snapshot error: ${res.status}`);
|
||||
return res.json();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,164 @@
|
||||
import type {
|
||||
DatasetIcebergMapping,
|
||||
DatasetIcebergMappingStatus,
|
||||
CreateDatasetIcebergMappingInput,
|
||||
} from "../types/dataset-iceberg-mapping";
|
||||
import { createAud0Client, type Aud0Client } from "../clients/aud0-client";
|
||||
import { DatasetService } from "./dataset-service";
|
||||
|
||||
interface DatasetIcebergMappingRecord {
|
||||
dataset_iceberg_mapping_id: string;
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
dataset_id: string;
|
||||
iceberg_table_id: string;
|
||||
namespace: string | null;
|
||||
table_name: string | null;
|
||||
mapping_status: string;
|
||||
mapped_by: string | null;
|
||||
correlation_id: string | null;
|
||||
created_at: string;
|
||||
updated_at: string | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Owns 0pt.dataset_iceberg_mapping.schema.v1 -- the pointer from a 0pt
|
||||
* dataset to the Iceberg table it writes to. `iceberg_table_id` is always a
|
||||
* REFERENCE into ST0RE's table registry (never a copy of ST0RE's table row);
|
||||
* this service holds no table/catalog/snapshot state itself. See
|
||||
* docs/ICEBERG_WRITE_BOUNDARY.md.
|
||||
*
|
||||
* This service does not call ST0RE directly -- it records a mapping to an
|
||||
* `iceberg_table_id` that already exists in ST0RE's registry.
|
||||
* `IcebergWriteService.mapDatasetToIcebergTable` (src/services/
|
||||
* iceberg-write-service.ts) is the higher-level flow that registers a NEW
|
||||
* table in ST0RE first and then calls this service to record the mapping --
|
||||
* kept as two composable steps so a mapping can also be created against an
|
||||
* already-registered table (e.g. one N0D or another 0pt job already set up).
|
||||
*/
|
||||
export class DatasetIcebergMappingService {
|
||||
private mappings: Map<string, DatasetIcebergMappingRecord> = new Map();
|
||||
private aud0: Aud0Client;
|
||||
private datasetService: DatasetService;
|
||||
|
||||
constructor(opts?: { aud0BaseUrl?: string; datasetService?: DatasetService }) {
|
||||
this.aud0 = createAud0Client(opts?.aud0BaseUrl);
|
||||
this.datasetService = opts?.datasetService ?? new DatasetService(opts?.aud0BaseUrl);
|
||||
}
|
||||
|
||||
async createMapping(input: CreateDatasetIcebergMappingInput): Promise<DatasetIcebergMapping> {
|
||||
if (this.mappings.has(input.dataset_iceberg_mapping_id)) {
|
||||
throw new Error(`Dataset Iceberg mapping ${input.dataset_iceberg_mapping_id} already exists`);
|
||||
}
|
||||
if (!input.iceberg_table_id) {
|
||||
throw new Error("iceberg_table_id is required (reference into ST0RE's table registry)");
|
||||
}
|
||||
// Verify the dataset exists -- a mapping with no dataset is meaningless.
|
||||
try {
|
||||
await this.datasetService.getDataset(input.dataset_id);
|
||||
} catch {
|
||||
throw new Error(`Dataset ${input.dataset_id} not found`);
|
||||
}
|
||||
|
||||
const now = new Date().toISOString();
|
||||
const record: DatasetIcebergMappingRecord = {
|
||||
dataset_iceberg_mapping_id: input.dataset_iceberg_mapping_id,
|
||||
tenant_id: input.tenant_id,
|
||||
project_id: input.project_id,
|
||||
dataset_id: input.dataset_id,
|
||||
iceberg_table_id: input.iceberg_table_id,
|
||||
namespace: input.namespace ?? null,
|
||||
table_name: input.table_name ?? null,
|
||||
mapping_status: input.mapping_status ?? "active",
|
||||
mapped_by: input.mapped_by ?? null,
|
||||
correlation_id: input.correlation_id ?? null,
|
||||
created_at: now,
|
||||
updated_at: null,
|
||||
};
|
||||
this.mappings.set(input.dataset_iceberg_mapping_id, record);
|
||||
|
||||
// 0pt.dataset_iceberg_table_mapped -- docs/11-lakehouse-write-plan.md event list,
|
||||
// docs/ICEBERG_WRITE_BOUNDARY.md. Fired when the mapping is created/updated.
|
||||
await this.aud0.ingestEvent({
|
||||
event_id: `evt_0pt_dataset_iceberg_table_mapped_${input.dataset_iceberg_mapping_id}`,
|
||||
event_type: "0pt.dataset_iceberg_table_mapped",
|
||||
tenant_id: input.tenant_id,
|
||||
project_id: input.project_id,
|
||||
actor_id: input.mapped_by ?? "system",
|
||||
app_id: "0pt",
|
||||
capability_id: "0pt.map_dataset_iceberg_table",
|
||||
resource_type: "dataset_iceberg_mapping",
|
||||
resource_id: input.dataset_iceberg_mapping_id,
|
||||
risk_level: "L2",
|
||||
payload: {
|
||||
dataset_id: input.dataset_id,
|
||||
iceberg_table_id: input.iceberg_table_id,
|
||||
namespace: input.namespace,
|
||||
table_name: input.table_name,
|
||||
},
|
||||
created_at: now,
|
||||
correlation_id: input.correlation_id ?? "",
|
||||
});
|
||||
|
||||
// Lineage: dataset -> iceberg table.
|
||||
await this.aud0.ingestLineageEvent({
|
||||
lineage_event_id: `lineage_0pt_dataset_iceberg_${input.dataset_iceberg_mapping_id}`,
|
||||
tenant_id: input.tenant_id,
|
||||
project_id: input.project_id,
|
||||
app_id: "0pt",
|
||||
capability_id: "0pt.map_dataset_iceberg_table",
|
||||
source_type: "dataset",
|
||||
source_id: input.dataset_id,
|
||||
target_type: "iceberg_table",
|
||||
target_id: input.iceberg_table_id,
|
||||
lineage_type: "stored_as",
|
||||
metadata: { dataset_iceberg_mapping_id: input.dataset_iceberg_mapping_id },
|
||||
created_at: now,
|
||||
correlation_id: input.correlation_id ?? "",
|
||||
});
|
||||
|
||||
return this.toMapping(record);
|
||||
}
|
||||
|
||||
async getMapping(mappingId: string): Promise<DatasetIcebergMapping> {
|
||||
const record = this.mappings.get(mappingId);
|
||||
if (!record) throw new Error(`Dataset Iceberg mapping ${mappingId} not found`);
|
||||
return this.toMapping(record);
|
||||
}
|
||||
|
||||
async getMappingForDataset(datasetId: string): Promise<DatasetIcebergMapping | undefined> {
|
||||
const record = Array.from(this.mappings.values()).find((r) => r.dataset_id === datasetId && r.mapping_status === "active");
|
||||
return record ? this.toMapping(record) : undefined;
|
||||
}
|
||||
|
||||
async listMappings(filter?: {
|
||||
tenant_id?: string;
|
||||
project_id?: string;
|
||||
dataset_id?: string;
|
||||
mapping_status?: DatasetIcebergMappingStatus;
|
||||
}): Promise<DatasetIcebergMapping[]> {
|
||||
let results = Array.from(this.mappings.values());
|
||||
if (filter?.tenant_id) results = results.filter((r) => r.tenant_id === filter.tenant_id);
|
||||
if (filter?.project_id) results = results.filter((r) => r.project_id === filter.project_id);
|
||||
if (filter?.dataset_id) results = results.filter((r) => r.dataset_id === filter.dataset_id);
|
||||
if (filter?.mapping_status) results = results.filter((r) => r.mapping_status === filter.mapping_status);
|
||||
return results.map((r) => this.toMapping(r));
|
||||
}
|
||||
|
||||
private toMapping(r: DatasetIcebergMappingRecord): DatasetIcebergMapping {
|
||||
return {
|
||||
dataset_iceberg_mapping_id: r.dataset_iceberg_mapping_id,
|
||||
tenant_id: r.tenant_id,
|
||||
project_id: r.project_id,
|
||||
dataset_id: r.dataset_id,
|
||||
iceberg_table_id: r.iceberg_table_id,
|
||||
namespace: r.namespace ?? undefined,
|
||||
table_name: r.table_name ?? undefined,
|
||||
mapping_status: r.mapping_status as DatasetIcebergMappingStatus,
|
||||
mapped_by: r.mapped_by ?? undefined,
|
||||
correlation_id: r.correlation_id ?? undefined,
|
||||
created_at: r.created_at,
|
||||
updated_at: r.updated_at ?? undefined,
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,284 @@
|
||||
import type { DatasetIcebergMapping } from "../types/dataset-iceberg-mapping";
|
||||
import { createCl0udClient, type Cl0udClient } from "../clients/cl0ud-client";
|
||||
import { createN0dClient, type N0dClient } from "../clients/n0d-client";
|
||||
import { createAud0Client, type Aud0Client } from "../clients/aud0-client";
|
||||
import {
|
||||
createSt0reLakehouseClient,
|
||||
type St0reLakehouseClient,
|
||||
type IcebergTableFormat,
|
||||
type TableSnapshotType,
|
||||
} from "../clients/st0re-lakehouse-client";
|
||||
import { DatasetService } from "./dataset-service";
|
||||
import { DatasetIcebergMappingService } from "./dataset-iceberg-mapping-service";
|
||||
|
||||
/**
|
||||
* IcebergWriteService -- 0pt's write-path entrypoint for Iceberg tables.
|
||||
*
|
||||
* This is the concrete implementation of docs/ICEBERG_WRITE_BOUNDARY.md /
|
||||
* docs/11-lakehouse-write-plan.md items 1, 3, 6, 7, 8: 0pt is the canonical
|
||||
* writer of Iceberg tables, and it writes THROUGH ST0RE's table registry
|
||||
* (`St0reLakehouseClient`) rather than persisting any lakehouse table,
|
||||
* catalog, or snapshot state itself. This service holds NO table/snapshot
|
||||
* state of its own -- only `DatasetIcebergMappingService`'s lightweight
|
||||
* dataset-to-table-id pointer, which is a reference, not a copy.
|
||||
*
|
||||
* Two entry points:
|
||||
* - `mapDatasetToIcebergTable` -- registers a NEW Iceberg table in ST0RE for
|
||||
* a dataset that does not have one yet, then records the mapping. This is
|
||||
* the "first test dataset written to Iceberg through ST0RE's table
|
||||
* registry" proof path (see tests/p0/iceberg-write-service.test.ts).
|
||||
* - `requestIcebergWrite` -- registers a new snapshot (an actual write) for
|
||||
* an already-mapped dataset, optionally routed through an N0D job when the
|
||||
* write is heavy (item 7: 0pt REQUESTS N0D jobs for heavy writes/
|
||||
* compactions, it does not execute them itself, and it never owns the
|
||||
* result of that job beyond the ST0RE-registered snapshot ref).
|
||||
*
|
||||
* Byte-level writes (actually producing Parquet/Iceberg data files) are OUT
|
||||
* OF SCOPE for this pass, same honest boundary ST0RE itself documents for
|
||||
* its own snapshot routes ("ST0RE doesn't itself write the Iceberg
|
||||
* metadata.json... this registry tracks the pointer/version history for
|
||||
* whatever the caller's own catalog write produced" --
|
||||
* sax3l/st0re src/api/routes/lakehouse.ts). What this service proves is the
|
||||
* OWNERSHIP path -- that 0pt requests the write and registers its result
|
||||
* through ST0RE, never anywhere else.
|
||||
*/
|
||||
export interface MapDatasetToIcebergTableInput {
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
dataset_id: string;
|
||||
dataset_iceberg_mapping_id: string;
|
||||
namespace: string;
|
||||
table_name: string;
|
||||
storage_uri: string;
|
||||
catalog_ref: string;
|
||||
table_format?: IcebergTableFormat;
|
||||
actor_id: string;
|
||||
correlation_id: string;
|
||||
}
|
||||
|
||||
export interface MapDatasetToIcebergTableResult {
|
||||
mapping: DatasetIcebergMapping;
|
||||
iceberg_table_id: string;
|
||||
policy_decision_id: string;
|
||||
}
|
||||
|
||||
export interface RequestIcebergWriteInput {
|
||||
dataset_iceberg_mapping_id: string;
|
||||
actor_id: string;
|
||||
correlation_id: string;
|
||||
snapshot_type?: TableSnapshotType;
|
||||
schema_version: string;
|
||||
manifest_uri: string;
|
||||
record_count?: number;
|
||||
size_bytes?: number;
|
||||
/** True for large backfills/table maintenance -- routes through an N0D job request rather than 0pt executing in-process (docs/11-lakehouse-write-plan.md item 7). */
|
||||
heavy?: boolean;
|
||||
}
|
||||
|
||||
export interface RequestIcebergWriteResult {
|
||||
snapshot_id: string;
|
||||
iceberg_table_id: string;
|
||||
policy_decision_id: string;
|
||||
n0d_job_id?: string;
|
||||
}
|
||||
|
||||
export class IcebergWriteService {
|
||||
private cl0ud: Cl0udClient;
|
||||
private n0d: N0dClient;
|
||||
private st0reLakehouse: St0reLakehouseClient;
|
||||
private aud0: Aud0Client;
|
||||
private datasetService: DatasetService;
|
||||
private mappingService: DatasetIcebergMappingService;
|
||||
|
||||
constructor(opts?: {
|
||||
cl0udBaseUrl?: string;
|
||||
n0dBaseUrl?: string;
|
||||
st0reBaseUrl?: string;
|
||||
aud0BaseUrl?: string;
|
||||
datasetService?: DatasetService;
|
||||
mappingService?: DatasetIcebergMappingService;
|
||||
}) {
|
||||
this.cl0ud = createCl0udClient(opts?.cl0udBaseUrl);
|
||||
this.n0d = createN0dClient(opts?.n0dBaseUrl);
|
||||
this.st0reLakehouse = createSt0reLakehouseClient(opts?.st0reBaseUrl);
|
||||
this.aud0 = createAud0Client(opts?.aud0BaseUrl);
|
||||
this.datasetService = opts?.datasetService ?? new DatasetService(opts?.aud0BaseUrl);
|
||||
this.mappingService = opts?.mappingService ?? new DatasetIcebergMappingService({ aud0BaseUrl: opts?.aud0BaseUrl, datasetService: this.datasetService });
|
||||
}
|
||||
|
||||
async mapDatasetToIcebergTable(input: MapDatasetToIcebergTableInput): Promise<MapDatasetToIcebergTableResult> {
|
||||
const dataset = await this.datasetService.getDataset(input.dataset_id); // throws if missing
|
||||
|
||||
// CL0UD policy boundary -- same L3 gate ST0RE's own table-registration
|
||||
// route applies (sax3l/st0re src/api/routes/lakehouse.ts,
|
||||
// 'st0re.lakehouse.write_table'), checked here too since 0pt is the
|
||||
// caller initiating the write, not just relying on ST0RE to gate it.
|
||||
const policy = await this.cl0ud.evaluatePolicy({
|
||||
tenant_id: input.tenant_id,
|
||||
project_id: input.project_id,
|
||||
actor_id: input.actor_id,
|
||||
capability_id: "0pt.write_iceberg_table",
|
||||
resource_type: "dataset",
|
||||
resource_id: input.dataset_id,
|
||||
risk_level: "L3",
|
||||
correlation_id: input.correlation_id,
|
||||
});
|
||||
if (policy.decision === "deny") throw new Error(`Policy denied: ${policy.reason}`);
|
||||
if (policy.decision === "approval_required") throw new Error(`Policy requires approval: ${policy.reason}`);
|
||||
|
||||
// Register the table THROUGH ST0RE's table registry -- 0pt does not
|
||||
// create or track lakehouse table state on its own.
|
||||
const table = await this.st0reLakehouse.registerTable({
|
||||
tenantId: input.tenant_id,
|
||||
projectId: input.project_id,
|
||||
namespace: input.namespace,
|
||||
tableName: input.table_name,
|
||||
tableFormat: input.table_format ?? "iceberg", // ADR-0030: Iceberg is canonical; Delta/Hudi accepted only for bridge/import scenarios, never chosen by default here.
|
||||
storageUri: input.storage_uri,
|
||||
catalogRef: input.catalog_ref,
|
||||
datasetId: input.dataset_id,
|
||||
ownerProduct: "0pt",
|
||||
createdBy: input.actor_id,
|
||||
});
|
||||
|
||||
const mapping = await this.mappingService.createMapping({
|
||||
dataset_iceberg_mapping_id: input.dataset_iceberg_mapping_id,
|
||||
tenant_id: input.tenant_id,
|
||||
project_id: input.project_id,
|
||||
dataset_id: input.dataset_id,
|
||||
iceberg_table_id: table.tableId,
|
||||
namespace: input.namespace,
|
||||
table_name: input.table_name,
|
||||
mapping_status: "active",
|
||||
mapped_by: input.actor_id,
|
||||
correlation_id: input.correlation_id,
|
||||
});
|
||||
|
||||
return { mapping, iceberg_table_id: table.tableId, policy_decision_id: policy.decision_id };
|
||||
}
|
||||
|
||||
async requestIcebergWrite(input: RequestIcebergWriteInput): Promise<RequestIcebergWriteResult> {
|
||||
const mapping = await this.mappingService.getMapping(input.dataset_iceberg_mapping_id);
|
||||
if (mapping.mapping_status !== "active") {
|
||||
throw new Error(`Dataset Iceberg mapping ${input.dataset_iceberg_mapping_id} is not active (status=${mapping.mapping_status})`);
|
||||
}
|
||||
|
||||
const policy = await this.cl0ud.evaluatePolicy({
|
||||
tenant_id: mapping.tenant_id,
|
||||
project_id: mapping.project_id,
|
||||
actor_id: input.actor_id,
|
||||
capability_id: "0pt.write_iceberg_table",
|
||||
resource_type: "dataset",
|
||||
resource_id: mapping.dataset_id,
|
||||
risk_level: "L3",
|
||||
correlation_id: input.correlation_id,
|
||||
});
|
||||
if (policy.decision === "deny") throw new Error(`Policy denied: ${policy.reason}`);
|
||||
if (policy.decision === "approval_required") throw new Error(`Policy requires approval: ${policy.reason}`);
|
||||
|
||||
// 0pt.iceberg_write_requested -- fired when 0pt requests the write,
|
||||
// before the write lands (docs/11-lakehouse-write-plan.md event list).
|
||||
await this.aud0.ingestEvent({
|
||||
event_id: `evt_0pt_iceberg_write_requested_${input.dataset_iceberg_mapping_id}_${Date.now()}`,
|
||||
event_type: "0pt.iceberg_write_requested",
|
||||
tenant_id: mapping.tenant_id,
|
||||
project_id: mapping.project_id,
|
||||
actor_id: input.actor_id,
|
||||
app_id: "0pt",
|
||||
capability_id: "0pt.write_iceberg_table",
|
||||
resource_type: "iceberg_table",
|
||||
resource_id: mapping.iceberg_table_id,
|
||||
risk_level: "L3",
|
||||
payload: { dataset_id: mapping.dataset_id, dataset_iceberg_mapping_id: mapping.dataset_iceberg_mapping_id, heavy: input.heavy ?? false },
|
||||
created_at: new Date().toISOString(),
|
||||
correlation_id: input.correlation_id,
|
||||
});
|
||||
|
||||
// Heavy writes/compactions are REQUESTED from N0D, never executed
|
||||
// in-process by 0pt (docs/11-lakehouse-write-plan.md item 7). N0D
|
||||
// executes; it does not own the schema or the decision of what to
|
||||
// write -- the snapshot registration below still goes through ST0RE
|
||||
// regardless of who performed the write.
|
||||
let n0dJobId: string | undefined;
|
||||
if (input.heavy) {
|
||||
const job = await this.n0d.createJob({
|
||||
tenant_id: mapping.tenant_id,
|
||||
project_id: mapping.project_id,
|
||||
job_type: "0pt.iceberg_write_job",
|
||||
capability_id: "0pt.write_iceberg_table",
|
||||
requested_by_app_id: "0pt",
|
||||
requested_by_actor_id: input.actor_id,
|
||||
input_dataset_ids: [mapping.dataset_id],
|
||||
output_artifact_type: "iceberg_table_snapshot",
|
||||
requires_backbone: true,
|
||||
requires_gpu: false,
|
||||
required_node_roles: ["compute_general"],
|
||||
resource_profile: "cpu_io_large",
|
||||
risk_level: "L3",
|
||||
correlation_id: input.correlation_id,
|
||||
});
|
||||
n0dJobId = job.job_id;
|
||||
}
|
||||
|
||||
// Register the snapshot THROUGH ST0RE's table registry -- the one and
|
||||
// only place 0pt persists that a write landed.
|
||||
const snapshot = await this.st0reLakehouse.createSnapshot(mapping.iceberg_table_id, {
|
||||
snapshotType: input.snapshot_type ?? "append",
|
||||
schemaVersion: input.schema_version,
|
||||
manifestUri: input.manifest_uri,
|
||||
createdByJobId: n0dJobId,
|
||||
createdByCapability: "0pt.write_iceberg_table",
|
||||
recordCount: input.record_count,
|
||||
sizeBytes: input.size_bytes,
|
||||
});
|
||||
|
||||
// 0pt.iceberg_snapshot_created -- fired once the write lands and ST0RE
|
||||
// has registered a new snapshot, carrying table ref + snapshot id + row
|
||||
// count + dataset id for AUD0 to build source -> dataset -> snapshot
|
||||
// lineage.
|
||||
await this.aud0.ingestEvent({
|
||||
event_id: `evt_0pt_iceberg_snapshot_created_${snapshot.snapshotId}`,
|
||||
event_type: "0pt.iceberg_snapshot_created",
|
||||
tenant_id: mapping.tenant_id,
|
||||
project_id: mapping.project_id,
|
||||
actor_id: input.actor_id,
|
||||
app_id: "0pt",
|
||||
capability_id: "0pt.write_iceberg_table",
|
||||
resource_type: "iceberg_snapshot",
|
||||
resource_id: snapshot.snapshotId,
|
||||
risk_level: "L2",
|
||||
payload: {
|
||||
dataset_id: mapping.dataset_id,
|
||||
iceberg_table_id: mapping.iceberg_table_id,
|
||||
snapshot_id: snapshot.snapshotId,
|
||||
record_count: input.record_count,
|
||||
n0d_job_id: n0dJobId,
|
||||
},
|
||||
created_at: new Date().toISOString(),
|
||||
correlation_id: input.correlation_id,
|
||||
});
|
||||
|
||||
await this.aud0.ingestLineageEvent({
|
||||
lineage_event_id: `lineage_0pt_iceberg_snapshot_${snapshot.snapshotId}`,
|
||||
tenant_id: mapping.tenant_id,
|
||||
project_id: mapping.project_id,
|
||||
app_id: "0pt",
|
||||
capability_id: "0pt.write_iceberg_table",
|
||||
source_type: "dataset",
|
||||
source_id: mapping.dataset_id,
|
||||
target_type: "iceberg_snapshot",
|
||||
target_id: snapshot.snapshotId,
|
||||
lineage_type: "produced",
|
||||
metadata: { iceberg_table_id: mapping.iceberg_table_id, n0d_job_id: n0dJobId },
|
||||
created_at: new Date().toISOString(),
|
||||
correlation_id: input.correlation_id,
|
||||
});
|
||||
|
||||
return {
|
||||
snapshot_id: snapshot.snapshotId,
|
||||
iceberg_table_id: mapping.iceberg_table_id,
|
||||
policy_decision_id: policy.decision_id,
|
||||
n0d_job_id: n0dJobId,
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,141 @@
|
||||
import type { SourceRatePolicy, SourceRatePolicyStatus, CreateSourceRatePolicyInput } from "../types/source-rate-policy";
|
||||
import { createAud0Client, type Aud0Client } from "../clients/aud0-client";
|
||||
import { SourceService } from "./source-service";
|
||||
|
||||
interface SourceRatePolicyRecord {
|
||||
source_rate_policy_id: string;
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
source_id: string;
|
||||
rate_limit_per_minute: number;
|
||||
crawl_budget: number;
|
||||
compliance_notes: string | null;
|
||||
domain: string | null;
|
||||
robots_ok: boolean | null;
|
||||
access: string | null;
|
||||
status: string;
|
||||
correlation_id: string | null;
|
||||
created_at: string;
|
||||
updated_at: string | null;
|
||||
created_by: string | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Owns 0pt.source_rate_policy.schema.v1 -- the genuine ownership repair:
|
||||
* source/crawl-domain rate limiting is 0pt's own compliance policy, moved
|
||||
* OUT of N0D's ComputePolicy.sourceRatePolicy (deprecated there per
|
||||
* ADR-0030's audit). See docs/ICEBERG_WRITE_BOUNDARY.md.
|
||||
*/
|
||||
export class SourceRatePolicyService {
|
||||
private policies: Map<string, SourceRatePolicyRecord> = new Map();
|
||||
private aud0: Aud0Client;
|
||||
private sourceService: SourceService;
|
||||
|
||||
constructor(opts?: { aud0BaseUrl?: string; sourceService?: SourceService }) {
|
||||
this.aud0 = createAud0Client(opts?.aud0BaseUrl);
|
||||
this.sourceService = opts?.sourceService ?? new SourceService(opts?.aud0BaseUrl);
|
||||
}
|
||||
|
||||
async createSourceRatePolicy(input: CreateSourceRatePolicyInput): Promise<SourceRatePolicy> {
|
||||
if (this.policies.has(input.source_rate_policy_id)) {
|
||||
throw new Error(`Source rate policy ${input.source_rate_policy_id} already exists`);
|
||||
}
|
||||
if (input.rate_limit_per_minute <= 0) {
|
||||
throw new Error("rate_limit_per_minute must be > 0");
|
||||
}
|
||||
if (input.crawl_budget <= 0) {
|
||||
throw new Error("crawl_budget must be > 0");
|
||||
}
|
||||
// Verify the source exists -- a rate policy with no source is meaningless.
|
||||
try {
|
||||
await this.sourceService.getSource(input.source_id);
|
||||
} catch {
|
||||
throw new Error(`Source ${input.source_id} not found`);
|
||||
}
|
||||
|
||||
const now = new Date().toISOString();
|
||||
const record: SourceRatePolicyRecord = {
|
||||
source_rate_policy_id: input.source_rate_policy_id,
|
||||
tenant_id: input.tenant_id,
|
||||
project_id: input.project_id,
|
||||
source_id: input.source_id,
|
||||
rate_limit_per_minute: input.rate_limit_per_minute,
|
||||
crawl_budget: input.crawl_budget,
|
||||
compliance_notes: input.compliance_notes ?? null,
|
||||
domain: input.domain ?? null,
|
||||
robots_ok: input.robots_ok ?? null,
|
||||
access: input.access ?? null,
|
||||
status: input.status ?? "active",
|
||||
correlation_id: input.correlation_id ?? null,
|
||||
created_at: now,
|
||||
updated_at: null,
|
||||
created_by: input.created_by ?? null,
|
||||
};
|
||||
this.policies.set(input.source_rate_policy_id, record);
|
||||
|
||||
// 0pt.source_rate_policy_created -- docs/11-lakehouse-write-plan.md event list,
|
||||
// docs/ICEBERG_WRITE_BOUNDARY.md. Gives AUD0/CL0UD an evented trail of 0pt's
|
||||
// crawl-compliance posture without polling 0pt's DB.
|
||||
await this.aud0.ingestEvent({
|
||||
event_id: `evt_0pt_source_rate_policy_created_${input.source_rate_policy_id}`,
|
||||
event_type: "0pt.source_rate_policy_created",
|
||||
tenant_id: input.tenant_id,
|
||||
project_id: input.project_id,
|
||||
actor_id: input.created_by ?? "system",
|
||||
app_id: "0pt",
|
||||
capability_id: "0pt.create_source_rate_policy",
|
||||
resource_type: "source_rate_policy",
|
||||
resource_id: input.source_rate_policy_id,
|
||||
risk_level: "L2",
|
||||
payload: {
|
||||
source_id: input.source_id,
|
||||
rate_limit_per_minute: input.rate_limit_per_minute,
|
||||
crawl_budget: input.crawl_budget,
|
||||
},
|
||||
created_at: now,
|
||||
correlation_id: input.correlation_id ?? "",
|
||||
});
|
||||
|
||||
return this.toPolicy(record);
|
||||
}
|
||||
|
||||
async getSourceRatePolicy(sourceRatePolicyId: string): Promise<SourceRatePolicy> {
|
||||
const record = this.policies.get(sourceRatePolicyId);
|
||||
if (!record) throw new Error(`Source rate policy ${sourceRatePolicyId} not found`);
|
||||
return this.toPolicy(record);
|
||||
}
|
||||
|
||||
async listSourceRatePolicies(filter?: {
|
||||
tenant_id?: string;
|
||||
project_id?: string;
|
||||
source_id?: string;
|
||||
status?: SourceRatePolicyStatus;
|
||||
}): Promise<SourceRatePolicy[]> {
|
||||
let results = Array.from(this.policies.values());
|
||||
if (filter?.tenant_id) results = results.filter((r) => r.tenant_id === filter.tenant_id);
|
||||
if (filter?.project_id) results = results.filter((r) => r.project_id === filter.project_id);
|
||||
if (filter?.source_id) results = results.filter((r) => r.source_id === filter.source_id);
|
||||
if (filter?.status) results = results.filter((r) => r.status === filter.status);
|
||||
return results.map((r) => this.toPolicy(r));
|
||||
}
|
||||
|
||||
private toPolicy(r: SourceRatePolicyRecord): SourceRatePolicy {
|
||||
return {
|
||||
source_rate_policy_id: r.source_rate_policy_id,
|
||||
tenant_id: r.tenant_id,
|
||||
project_id: r.project_id,
|
||||
source_id: r.source_id,
|
||||
rate_limit_per_minute: r.rate_limit_per_minute,
|
||||
crawl_budget: r.crawl_budget,
|
||||
compliance_notes: r.compliance_notes ?? undefined,
|
||||
domain: r.domain ?? undefined,
|
||||
robots_ok: r.robots_ok ?? undefined,
|
||||
access: (r.access as SourceRatePolicy["access"]) ?? undefined,
|
||||
status: r.status as SourceRatePolicyStatus,
|
||||
correlation_id: r.correlation_id ?? undefined,
|
||||
created_at: r.created_at,
|
||||
updated_at: r.updated_at ?? undefined,
|
||||
created_by: r.created_by ?? undefined,
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
// 0pt.dataset_iceberg_mapping.schema.v1
|
||||
//
|
||||
// Maps a 0pt dataset to the Iceberg table that carries its canonical lakehouse
|
||||
// write path. `iceberg_table_id` is a REFERENCE into ST0RE's table registry
|
||||
// (IcebergTable.tableId, st0re_iceberg_tables) -- it is not a copy of ST0RE's
|
||||
// table metadata. 0pt does not persist table/catalog/snapshot state of its
|
||||
// own; this row exists only to answer "which Iceberg table does this dataset
|
||||
// write to", per docs/10-lakehouse-decision.md and
|
||||
// docs/ICEBERG_WRITE_BOUNDARY.md.
|
||||
|
||||
export type DatasetIcebergMappingStatus = "pending" | "active" | "deprecated" | "archived";
|
||||
|
||||
export interface DatasetIcebergMapping {
|
||||
dataset_iceberg_mapping_id: string;
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
dataset_id: string;
|
||||
/** Reference into ST0RE's table registry (IcebergTable.tableId). Not resolved or duplicated locally. */
|
||||
iceberg_table_id: string;
|
||||
/** ST0RE lakehouse namespace the table lives in, carried for convenience/filtering only -- ST0RE remains the source of truth. */
|
||||
namespace?: string;
|
||||
table_name?: string;
|
||||
mapping_status: DatasetIcebergMappingStatus;
|
||||
mapped_by?: string;
|
||||
correlation_id?: string;
|
||||
created_at: string;
|
||||
updated_at?: string;
|
||||
}
|
||||
|
||||
export interface CreateDatasetIcebergMappingInput {
|
||||
dataset_iceberg_mapping_id: string;
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
dataset_id: string;
|
||||
iceberg_table_id: string;
|
||||
namespace?: string;
|
||||
table_name?: string;
|
||||
mapping_status?: DatasetIcebergMappingStatus;
|
||||
mapped_by?: string;
|
||||
correlation_id?: string;
|
||||
}
|
||||
@@ -0,0 +1,57 @@
|
||||
// 0pt.source_rate_policy.schema.v1
|
||||
//
|
||||
// The genuine ownership repair this contract encodes: crawl-domain rate
|
||||
// limiting and compliance posture belongs to 0pt, not N0D. N0D's
|
||||
// `ComputePolicy.sourceRatePolicy` field (per-domain crawl rate limits) is
|
||||
// deprecated/removed from N0D's seed data per ADR-0030's audit -- 0pt already
|
||||
// owns the sourcing chain (docs/06-data-engine.md, `sources.yaml`) and is the
|
||||
// only product that should hold this policy.
|
||||
//
|
||||
// Field names reconcile two prior specs written for this same contract:
|
||||
// docs/11-lakehouse-write-plan.md item 5 (domain/requests_per_second/
|
||||
// robots_ok/access) and this pass's task spec (rate_limit_per_minute/
|
||||
// crawl_budget/compliance_notes). Both are kept: the task-spec fields are
|
||||
// primary, the docs/11 fields are additional optional detail for the same
|
||||
// row, not a competing shape.
|
||||
|
||||
export type SourceRatePolicyStatus = "active" | "paused" | "deprecated";
|
||||
|
||||
export interface SourceRatePolicy {
|
||||
source_rate_policy_id: string;
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
/** References source.source_id (src/types/source.ts). */
|
||||
source_id: string;
|
||||
rate_limit_per_minute: number;
|
||||
/** Max requests/pages/records this policy permits per crawl run or per day (unit documented in compliance_notes when non-obvious). */
|
||||
crawl_budget: number;
|
||||
/** Free-text compliance rationale: robots.txt posture, licence terms, contact/ToS reference. */
|
||||
compliance_notes?: string;
|
||||
/** Crawl-target domain, e.g. "allabolag.se" -- optional convenience field mirroring docs/06-data-engine.md's sources.yaml rate_limit entries. */
|
||||
domain?: string;
|
||||
/** Whether robots.txt permits the access pattern this policy governs. */
|
||||
robots_ok?: boolean;
|
||||
/** Mirrors dim_source.access (docs/03-data-model.md): api | bulk | authorized_web | licensed. */
|
||||
access?: "api" | "bulk" | "authorized_web" | "licensed";
|
||||
status: SourceRatePolicyStatus;
|
||||
correlation_id?: string;
|
||||
created_at: string;
|
||||
updated_at?: string;
|
||||
created_by?: string;
|
||||
}
|
||||
|
||||
export interface CreateSourceRatePolicyInput {
|
||||
source_rate_policy_id: string;
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
source_id: string;
|
||||
rate_limit_per_minute: number;
|
||||
crawl_budget: number;
|
||||
compliance_notes?: string;
|
||||
domain?: string;
|
||||
robots_ok?: boolean;
|
||||
access?: "api" | "bulk" | "authorized_web" | "licensed";
|
||||
status?: SourceRatePolicyStatus;
|
||||
correlation_id?: string;
|
||||
created_by?: string;
|
||||
}
|
||||
Reference in New Issue
Block a user