feat(data-os): P060-P068 durable ingestion plane — source classification gate, connector contract, common envelope, schema evolution, quality enforcement, queryable lineage, idempotent backfill
CI (SIAX Cloud) / integration (push) Successful in 42s
CI (SIAX Cloud) / quality (push) Successful in 1m17s
CI (SIAX Cloud) / security (push) Successful in 46s
Security Scan (SIAX Cloud) / security (push) Successful in 31s
CI (SIAX Cloud) / security (pull_request) Successful in 24s
CI (SIAX Cloud) / integration (pull_request) Successful in 36s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m4s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 27s
CI (SIAX Cloud) / integration (push) Successful in 42s
CI (SIAX Cloud) / quality (push) Successful in 1m17s
CI (SIAX Cloud) / security (push) Successful in 46s
Security Scan (SIAX Cloud) / security (push) Successful in 31s
CI (SIAX Cloud) / security (pull_request) Successful in 24s
CI (SIAX Cloud) / integration (pull_request) Successful in 36s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m4s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 27s
BEFORE_STATE (verified, with evidence):
* P060 PARTIAL/BROKEN. The registry persisted rights/provenance/health/
freshness to Postgres, but carried no authorization, no schedule, no
checkpoint, no schema and no owner distinct from created_by — and the
activation DoD was inverted: postgres-source-repository.ts wrote
`${input.status ?? "active"}`, so a source with no rights, no provenance
and no authorization was ACTIVE the instant it was registered. No
activation operation existed at all.
* P061 MISSING. services/api/sources/connectors.py had per-source fetch_*()
functions only — no lifecycle, no cursor, no resume, no health, no
conformance suite.
* P062 MISSING, already documented as such in
services/api/tests/test_known_gaps.py:10-19 (no CDC pipeline anywhere;
workers/pipeline.py has no persistence at all).
* P063 MISSING. schema_version existed as a free string; no comparison,
no policy, no versioned source schema.
* P064 PARTIAL. Rules/runs/issues/scores existed, but a failing rule only
opened an issue and the data flowed on. No quarantine anywhere.
* P065 DUPLICATE (confirmed). Two ER engines: src/er/entity-resolution-
service.ts (canonical) and services/api/workers/resolve.py (simplified,
still called by pipeline.py:110).
* P067 PARTIAL. Lineage was emitted to AUD0 but nothing here could ANSWER a
lineage question — no store, no query, no route.
* P068 MISSING, documented in test_known_gaps.py:44-52 ("no delivery-store
persistence layer exists to dedupe a backfill against").
BUILT (only what was PARTIAL/BROKEN/MISSING):
* P060 sql/007_ingestion_plane.sql adds authorization/schedule/schemas/
checkpoints/ledger/quarantine/lineage tables, owner_id, and flips the
opt_sources status DEFAULT to 'pending' so even a raw psql INSERT cannot
produce an unclassified-but-active source. SourceRegistryService gains
classifySource/activateSource/deactivateSource + authorization/schedule
accessors; registration NEVER activates (asking for status:"active" goes
through the gate, not around it); a service identity cannot authorize a
source. Routes: activate (GET classification / POST activate),
authorization, schedule.
* P061 src/connectors/connector.ts — all nine operations on one interface.
src/connectors/conformance.ts — a runnable suite (15 checks, no "skip"
verdict). HttpJsonConnector (real HTTP, opaque cursor) and
JsonlFileConnector (real file, line offsets, CDC deletes) both pass it.
The suite caught a real defect while being written: an API that drops
next_cursor at the end left the connector unable to resume, which is why
HttpJsonConnectorConfig has cursor_field.
* P062 src/types/ingestion.ts + src/ingestion/{envelope,ingestion-runner}.ts.
One envelope for batch/stream/cdc; durable per-(source,partition)
checkpoints; resume proven by killing a run inside the sink.
* P063 src/schema/{record-schema,compatibility,schema-registry}.ts. Two
enforcement points, both upstream of any consumer write: a discovery gate
before the first read, and per-record conformance before the page reaches
the sink. A producer cannot widen its own policy retroactively.
* P064 src/quality/quality-policy.ts composes the EXISTING ExpectationEngine
(not a second engine) and adds the missing consequence: block /
quarantine / warn, with quarantine durable and recoverable. No policy is
reported as "none_declared", never as a pass; an unexecutable expectation
counts as a violation, never a silent pass.
* P067 src/lineage/{lineage-edge,lineage-store}.ts + GET /api/0pt/lineage.
AUD0 stays the system of record; this is 0pt's own queryable projection.
* P068 opt_ingestion_ledger + IngestionRunner.backfill: bounded, idempotent
on repeat, leaves the live checkpoint alone, still re-emits real changes.
A BUG THIS PASS WROTE AND CAUGHT: the first IngestionRepository draft
combined the idempotency check and its recording into one atomic call
BEFORE the sink. The crash/resume test then delivered 7 of 10 records — the
page in flight when the sink died was already marked delivered, so the
resumed run skipped it as a duplicate. Permanent, silent data loss. Fixed by
splitting peekDeliveries/commitDeliveries and committing both the ledger and
the checkpoint only after the sink resolves.
NOT DONE, stated rather than deferred quietly:
* P065 is NOT met. resolve.py is still a second simplified production
engine. Closing it means rewriting it as an adapter over the canonical
service, which cannot be verified from this repo today: services/api is
not built by this repo's Dockerfile, api.0pt.app answers 404, and NO CI
workflow runs its pytest suite. The duplication is now stated at both
sites in code and the removal plan is in STATUS.md. The new ingestion
plane performs no ER, so it adds no third engine.
* P066 (Iceberg) untouched — ST0RE is owned by another workflow. Contract
read only.
* Delivery is AT-LEAST-ONCE, not exactly-once (no per-partition run lock).
* No CDC *source* exists; the contract is met, a CDC connector is not built.
* 0pt does not schedule itself — the schedule row is a declaration ACT0
would read.
VERIFIED: pnpm typecheck clean; pnpm test 365 passed / 25 skipped (was
311/18); pnpm build clean with all four new routes in the route table; lint
unchanged at 12 pre-existing errors. Real Postgres 16
(tests/p0/ingestion-plane-postgres.realinfra.test.ts, 7 passed) plus a real
`docker restart` of that container with an identical read-back through the
production code path — that is P060's "a restart preserves all sources",
measured.
DEPLOYMENT REALITY (curl, 2026-09-04): 0pt.app/api/health returns 200 from
Vercel with db:not_configured; 0pt.app/api/0pt/sources returns 404;
api.0pt.app/health returns 404; cd.yml still guards on a placeholder Coolify
app id. None of the capability layer runs in production. Nothing here is
LIVE_VERIFIED and STATUS.md says so.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
c5ebceff38
commit
9267576b7c
@@ -0,0 +1,276 @@
|
||||
/**
|
||||
* src/connectors/conformance.ts
|
||||
* ==============================
|
||||
* P061 DoD: "two heterogeneous connectors pass the same conformance suite."
|
||||
*
|
||||
* This module IS that suite. It is a plain async function returning a report
|
||||
* — not a vitest file — so it can be run three ways without duplication:
|
||||
* 1. from tests (tests/p0/connector-conformance.test.ts runs it per connector),
|
||||
* 2. from an operator script against a real configured source,
|
||||
* 3. from a future capability route as a pre-activation check.
|
||||
*
|
||||
* Each check is a named assertion about the CONTRACT, not about any one
|
||||
* connector's data. A connector that cannot serve a check honestly (e.g. it
|
||||
* has no partitions under the given config) FAILS it — there is no "skip"
|
||||
* verdict, because a skip that reads as a pass is how a conformance suite
|
||||
* becomes decoration.
|
||||
*/
|
||||
import type {
|
||||
BackfillRequest,
|
||||
Connector,
|
||||
ConnectorConfig,
|
||||
ConnectorContext,
|
||||
} from "./connector";
|
||||
|
||||
export interface ConformanceCheck {
|
||||
name: string;
|
||||
passed: boolean;
|
||||
detail: string;
|
||||
}
|
||||
|
||||
export interface ConformanceReport {
|
||||
connector_id: string;
|
||||
connector_version: string;
|
||||
passed: boolean;
|
||||
checks: ConformanceCheck[];
|
||||
}
|
||||
|
||||
export interface ConformanceFixture {
|
||||
/** A config the connector must accept. */
|
||||
validConfig: ConnectorConfig;
|
||||
/** A config the connector must REJECT in validate() without doing IO. */
|
||||
invalidConfig: ConnectorConfig;
|
||||
ctx: ConnectorContext;
|
||||
/**
|
||||
* A bounded backfill range that overlaps data the connector can serve.
|
||||
* `from`/`to` are connector-defined (documented per connector).
|
||||
*/
|
||||
backfill: Omit<BackfillRequest, "partition_key" | "limit"> & { partition_key?: string };
|
||||
/** Total number of records the fixture's first partition holds, so paging can be checked exactly. */
|
||||
expectedRecordCount: number;
|
||||
}
|
||||
|
||||
async function check(
|
||||
checks: ConformanceCheck[],
|
||||
name: string,
|
||||
fn: () => Promise<string> | string,
|
||||
): Promise<void> {
|
||||
try {
|
||||
const detail = await fn();
|
||||
checks.push({ name, passed: true, detail });
|
||||
} catch (err) {
|
||||
checks.push({
|
||||
name,
|
||||
passed: false,
|
||||
detail: err instanceof Error ? err.message : String(err),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
function assert(condition: boolean, message: string): void {
|
||||
if (!condition) throw new Error(message);
|
||||
}
|
||||
|
||||
/**
|
||||
* Run the full conformance suite against one connector instance.
|
||||
* The connector is left disconnected on return, pass or fail.
|
||||
*/
|
||||
export async function runConnectorConformance(
|
||||
connector: Connector,
|
||||
fixture: ConformanceFixture,
|
||||
): Promise<ConformanceReport> {
|
||||
const checks: ConformanceCheck[] = [];
|
||||
|
||||
await check(checks, "identity: non-empty id and version", () => {
|
||||
assert(connector.connector_id.length > 0, "connector_id is empty");
|
||||
assert(connector.connector_version.length > 0, "connector_version is empty");
|
||||
return `${connector.connector_id}@${connector.connector_version}`;
|
||||
});
|
||||
|
||||
await check(checks, "validate: accepts a valid config without IO", () => {
|
||||
const res = connector.validate(fixture.validConfig);
|
||||
assert(res.valid, `valid config rejected: ${res.errors.join("; ")}`);
|
||||
assert(res.errors.length === 0, "valid config produced errors");
|
||||
return "valid config accepted";
|
||||
});
|
||||
|
||||
await check(checks, "validate: rejects an invalid config and says why", () => {
|
||||
const res = connector.validate(fixture.invalidConfig);
|
||||
assert(!res.valid, "invalid config was accepted (fail-open)");
|
||||
assert(res.errors.length > 0, "rejection carried no reason");
|
||||
return `rejected with ${res.errors.length} reason(s)`;
|
||||
});
|
||||
|
||||
await check(checks, "operations before connect() fail closed", async () => {
|
||||
let threw = false;
|
||||
try {
|
||||
await connector.read({ partition_key: "anything", limit: 1 });
|
||||
} catch {
|
||||
threw = true;
|
||||
}
|
||||
assert(threw, "read() succeeded without a session");
|
||||
return "read() refused without a session";
|
||||
});
|
||||
|
||||
await check(checks, "connect: idempotent", async () => {
|
||||
await connector.connect(fixture.validConfig, fixture.ctx);
|
||||
await connector.connect(fixture.validConfig, fixture.ctx);
|
||||
return "connect() called twice without error";
|
||||
});
|
||||
|
||||
let partitionKey = fixture.backfill.partition_key ?? "";
|
||||
await check(checks, "discover: reports at least one partition with a schema", async () => {
|
||||
const discovery = await connector.discover();
|
||||
assert(discovery.partitions.length > 0, "no partitions discovered");
|
||||
assert(discovery.modes.length > 0, "no ingestion modes declared");
|
||||
const first = discovery.partitions[0];
|
||||
assert(first.partition_key.length > 0, "partition_key is empty");
|
||||
assert(
|
||||
Object.keys(first.schema).length > 0,
|
||||
`partition "${first.partition_key}" reported an empty schema`,
|
||||
);
|
||||
if (!partitionKey) partitionKey = first.partition_key;
|
||||
return `${discovery.partitions.length} partition(s), modes: ${discovery.modes.join(",")}`;
|
||||
});
|
||||
|
||||
await check(checks, "health: answers with a status instead of throwing", async () => {
|
||||
const health = await connector.health();
|
||||
assert(
|
||||
["up", "degraded", "down"].includes(health.status),
|
||||
`unexpected health status "${health.status}"`,
|
||||
);
|
||||
return `status=${health.status}`;
|
||||
});
|
||||
|
||||
await check(checks, "read: honours the limit and advances the cursor", async () => {
|
||||
const first = await connector.read({ partition_key: partitionKey, limit: 1 });
|
||||
assert(first.records.length <= 1, `limit 1 returned ${first.records.length} records`);
|
||||
assert(first.records.length === 1, "limit 1 returned nothing — fixture has no data");
|
||||
assert(first.next_cursor !== undefined, "no cursor returned after a non-empty read");
|
||||
const second = await connector.read({
|
||||
partition_key: partitionKey,
|
||||
cursor: first.next_cursor,
|
||||
limit: 1,
|
||||
});
|
||||
if (second.records.length > 0) {
|
||||
assert(
|
||||
second.records[0].record_key !== first.records[0].record_key,
|
||||
"reading from the returned cursor re-delivered the same record (cursor is not exclusive)",
|
||||
);
|
||||
}
|
||||
return `cursor advanced past ${first.records[0].record_key}`;
|
||||
});
|
||||
|
||||
await check(checks, "read: paging covers every record exactly once", async () => {
|
||||
const seen: string[] = [];
|
||||
let cursor: string | undefined;
|
||||
for (let page = 0; page < 100; page++) {
|
||||
const res = await connector.read({ partition_key: partitionKey, cursor, limit: 2 });
|
||||
for (const r of res.records) seen.push(r.record_key);
|
||||
cursor = res.next_cursor;
|
||||
if (res.exhausted || res.records.length === 0) break;
|
||||
}
|
||||
assert(
|
||||
seen.length === fixture.expectedRecordCount,
|
||||
`paged ${seen.length} records, fixture declares ${fixture.expectedRecordCount}`,
|
||||
);
|
||||
assert(new Set(seen).size === seen.length, "paging delivered a duplicate record_key");
|
||||
return `${seen.length} records, no duplicates`;
|
||||
});
|
||||
|
||||
await check(checks, "read: an exhausted partition reports exhausted", async () => {
|
||||
const everything = await connector.read({ partition_key: partitionKey, limit: 1000 });
|
||||
const past = await connector.read({
|
||||
partition_key: partitionKey,
|
||||
cursor: everything.next_cursor,
|
||||
limit: 1000,
|
||||
});
|
||||
assert(past.records.length === 0, "reading past the end returned records");
|
||||
assert(past.exhausted, "reading past the end did not report exhausted");
|
||||
return "end of partition reported honestly";
|
||||
});
|
||||
|
||||
await check(checks, "checkpoint: reflects the session position", async () => {
|
||||
const cp = await connector.checkpoint(partitionKey);
|
||||
assert(cp.partition_key === partitionKey, "checkpoint returned a different partition");
|
||||
return `cursor=${cp.cursor ?? "<none>"}`;
|
||||
});
|
||||
|
||||
await check(checks, "resume: continues from a committed checkpoint without re-delivery", async () => {
|
||||
const firstPage = await connector.read({ partition_key: partitionKey, limit: 2 });
|
||||
assert(firstPage.records.length > 0, "fixture partition is empty");
|
||||
const committed = {
|
||||
source_id: fixture.ctx.source_id,
|
||||
tenant_id: fixture.ctx.tenant_id,
|
||||
partition_key: partitionKey,
|
||||
cursor: firstPage.next_cursor,
|
||||
sequence: firstPage.records.length,
|
||||
records_committed: firstPage.records.length,
|
||||
committed_at: new Date().toISOString(),
|
||||
};
|
||||
const resumed = await connector.resume(committed, 1000);
|
||||
const firstKeys = new Set(firstPage.records.map((r) => r.record_key));
|
||||
for (const r of resumed.records) {
|
||||
assert(
|
||||
!firstKeys.has(r.record_key),
|
||||
`resume re-delivered already-committed record "${r.record_key}"`,
|
||||
);
|
||||
}
|
||||
assert(
|
||||
firstPage.records.length + resumed.records.length === fixture.expectedRecordCount,
|
||||
`resume covered ${firstPage.records.length + resumed.records.length} of ${fixture.expectedRecordCount} records — a gap`,
|
||||
);
|
||||
return `resumed ${resumed.records.length} remaining records with no gap and no overlap`;
|
||||
});
|
||||
|
||||
await check(checks, "backfill: serves a bounded range", async () => {
|
||||
const res = await connector.backfill({
|
||||
partition_key: partitionKey,
|
||||
from: fixture.backfill.from,
|
||||
to: fixture.backfill.to,
|
||||
limit: 1000,
|
||||
});
|
||||
assert(res.records.length > 0, "bounded backfill range returned nothing");
|
||||
return `${res.records.length} record(s) in range`;
|
||||
});
|
||||
|
||||
await check(checks, "backfill: refuses an unbounded range", async () => {
|
||||
let threw = false;
|
||||
try {
|
||||
await connector.backfill({
|
||||
partition_key: partitionKey,
|
||||
from: fixture.backfill.from,
|
||||
to: "",
|
||||
limit: 1000,
|
||||
});
|
||||
} catch {
|
||||
threw = true;
|
||||
}
|
||||
assert(threw, "an unbounded backfill was accepted");
|
||||
return "unbounded backfill refused";
|
||||
});
|
||||
|
||||
await check(checks, "disconnect: idempotent and safe", async () => {
|
||||
await connector.disconnect();
|
||||
await connector.disconnect();
|
||||
return "disconnect() called twice without error";
|
||||
});
|
||||
|
||||
await check(checks, "operations after disconnect() fail closed", async () => {
|
||||
let threw = false;
|
||||
try {
|
||||
await connector.read({ partition_key: partitionKey, limit: 1 });
|
||||
} catch {
|
||||
threw = true;
|
||||
}
|
||||
assert(threw, "read() succeeded after disconnect()");
|
||||
return "read() refused after disconnect()";
|
||||
});
|
||||
|
||||
return {
|
||||
connector_id: connector.connector_id,
|
||||
connector_version: connector.connector_version,
|
||||
passed: checks.every((c) => c.passed),
|
||||
checks,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,163 @@
|
||||
/**
|
||||
* src/connectors/connector.ts
|
||||
* ============================
|
||||
* P061: THE connector contract. Nine operations, one interface, every
|
||||
* connector implements all nine:
|
||||
*
|
||||
* validate — is this config usable? Pure, no IO, no network. Fail-closed.
|
||||
* connect — open whatever session the source needs. Must be idempotent.
|
||||
* discover — what partitions exist, and what shape do their records have?
|
||||
* health — is the source answering right now?
|
||||
* read — produce records from a cursor forward, bounded by `limit`.
|
||||
* checkpoint — the cursor meaning "everything before this is produced".
|
||||
* resume — continue from a previously committed checkpoint.
|
||||
* backfill — re-read a bounded historical range. Bounded, never "all".
|
||||
* disconnect — release the session. Must be idempotent and safe after an error.
|
||||
*
|
||||
* WHY AN INTERFACE AND NOT A BASE CLASS: connectors are heterogeneous by
|
||||
* nature (an HTTP API paginates by opaque token, a file store seeks by byte
|
||||
* offset, a change feed advances an LSN). The contract fixes the SHAPE of
|
||||
* the interaction and the meaning of the cursor; it does not pretend the
|
||||
* mechanics are the same. `src/connectors/conformance.ts` is what proves two
|
||||
* unrelated implementations really do honour the same shape.
|
||||
*
|
||||
* WHAT THIS REPLACES, HONESTLY: services/api/sources/connectors.py has
|
||||
* per-source `async def fetch_x()` functions and nothing else — no lifecycle,
|
||||
* no cursor, no resume, no health, no disconnect. Those functions remain the
|
||||
* Python engine's fetch layer; they are NOT retrofitted here (that engine is
|
||||
* not deployed — api.0pt.app answers 404 — and rewriting it blind would be
|
||||
* the exact "build it again" failure this programme forbids). This contract
|
||||
* is the seam a Python connector is wrapped into when that engine is revived.
|
||||
*/
|
||||
import type { IngestionCheckpoint, IngestionMode } from "../types/ingestion";
|
||||
import type { RecordSchemaFields } from "../schema/record-schema";
|
||||
|
||||
/** Config is connector-specific; the contract only requires it be serializable. */
|
||||
export type ConnectorConfig = Record<string, unknown>;
|
||||
|
||||
export interface ConnectorContext {
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
source_id: string;
|
||||
/** Correlates every operation of one ingestion run. */
|
||||
run_id: string;
|
||||
}
|
||||
|
||||
/** Result of `validate` — never throws, always answers. */
|
||||
export interface ValidationResult {
|
||||
valid: boolean;
|
||||
/** One entry per problem. Empty iff valid. */
|
||||
errors: string[];
|
||||
}
|
||||
|
||||
/** One readable partition (a table, a topic, a file, a page family). */
|
||||
export interface PartitionDescriptor {
|
||||
partition_key: string;
|
||||
/** Human label; may equal partition_key. */
|
||||
name: string;
|
||||
/** Record shape as the connector currently sees it — fed straight into the schema registry (P063). */
|
||||
schema: RecordSchemaFields;
|
||||
}
|
||||
|
||||
export interface DiscoveryResult {
|
||||
partitions: PartitionDescriptor[];
|
||||
/** Modes this connector can actually serve for this config. */
|
||||
modes: IngestionMode[];
|
||||
}
|
||||
|
||||
export interface ConnectorHealth {
|
||||
status: "up" | "degraded" | "down";
|
||||
latency_ms?: number;
|
||||
/** Short operator-facing text — never a raw upstream body. */
|
||||
detail?: string;
|
||||
}
|
||||
|
||||
export interface ReadRequest {
|
||||
partition_key: string;
|
||||
/** Start AFTER this cursor. Undefined = from the beginning of the partition. */
|
||||
cursor?: string;
|
||||
/** Hard upper bound on records returned. Connectors MUST honour it. */
|
||||
limit: number;
|
||||
}
|
||||
|
||||
/** A record as the connector sees it, before the runner wraps it in an envelope. */
|
||||
export interface ConnectorRecord {
|
||||
/** Stable business key within the partition. */
|
||||
record_key: string;
|
||||
/** Null only for a delete in a change feed. */
|
||||
payload: Record<string, unknown> | null;
|
||||
operation?: "upsert" | "insert" | "update" | "delete";
|
||||
occurred_at?: string;
|
||||
/** Cursor value AT this record — the runner records it per envelope. */
|
||||
cursor?: string;
|
||||
}
|
||||
|
||||
export interface ReadResult {
|
||||
records: ConnectorRecord[];
|
||||
/**
|
||||
* Cursor meaning "everything up to and including `records` has been
|
||||
* produced". The runner commits this only after the records are accepted.
|
||||
*/
|
||||
next_cursor?: string;
|
||||
/** False when the connector knows more records are immediately available. */
|
||||
exhausted: boolean;
|
||||
}
|
||||
|
||||
/** A bounded historical range. Unbounded backfills are rejected by the contract. */
|
||||
export interface BackfillRequest {
|
||||
partition_key: string;
|
||||
/** Inclusive lower bound (cursor or ISO timestamp — connector-defined, documented per connector). */
|
||||
from: string;
|
||||
/** Exclusive upper bound. Required: a backfill with no end is a full reload, not a backfill. */
|
||||
to: string;
|
||||
limit: number;
|
||||
}
|
||||
|
||||
export interface Connector {
|
||||
readonly connector_id: string;
|
||||
readonly connector_version: string;
|
||||
|
||||
/** Pure config check. MUST NOT perform IO. MUST NOT throw. */
|
||||
validate(config: ConnectorConfig): ValidationResult;
|
||||
|
||||
/** Open a session. Idempotent: calling it twice is a no-op, not an error. */
|
||||
connect(config: ConnectorConfig, ctx: ConnectorContext): Promise<void>;
|
||||
|
||||
/** Requires a live session. */
|
||||
discover(): Promise<DiscoveryResult>;
|
||||
|
||||
/** Requires a live session. Never throws for an unhealthy source — reports it. */
|
||||
health(): Promise<ConnectorHealth>;
|
||||
|
||||
/** Requires a live session. Honours `limit` exactly. */
|
||||
read(req: ReadRequest): Promise<ReadResult>;
|
||||
|
||||
/**
|
||||
* The connector's own view of where it stands for a partition. The DURABLE
|
||||
* checkpoint is owned by the runner (CheckpointStore); this is the
|
||||
* in-session position, used to cross-check a resume.
|
||||
*/
|
||||
checkpoint(partitionKey: string): Promise<{ partition_key: string; cursor?: string }>;
|
||||
|
||||
/**
|
||||
* Continue from a previously committed checkpoint. Equivalent to
|
||||
* `read({cursor: checkpoint.cursor})` for cursor-based connectors, but
|
||||
* explicit so a connector that must re-establish server-side state (a
|
||||
* replication slot, a scroll context) has a place to do it.
|
||||
*/
|
||||
resume(checkpoint: IngestionCheckpoint, limit: number): Promise<ReadResult>;
|
||||
|
||||
/** Bounded historical re-read. MUST reject an unbounded range. */
|
||||
backfill(req: BackfillRequest): Promise<ReadResult>;
|
||||
|
||||
/** Idempotent; safe to call without a session and after an error. */
|
||||
disconnect(): Promise<void>;
|
||||
}
|
||||
|
||||
/** Thrown by connectors when an operation needs a session that was never opened. */
|
||||
export class ConnectorNotConnectedError extends Error {
|
||||
constructor(connectorId: string, operation: string) {
|
||||
super(`connector ${connectorId}: ${operation} requires connect() first`);
|
||||
this.name = "ConnectorNotConnectedError";
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,274 @@
|
||||
/**
|
||||
* src/connectors/http-json-connector.ts
|
||||
* ======================================
|
||||
* P061 conformance connector #1: a real HTTP JSON API source.
|
||||
*
|
||||
* Heterogeneity (the point of having two): this connector talks to a REMOTE
|
||||
* service over the network, pages by an OPAQUE token the server hands back,
|
||||
* cannot know how many records exist until the server says it is done, and
|
||||
* has a genuine health check (an HTTP round trip). Its sibling
|
||||
* (jsonl-file-connector.ts) does none of those things.
|
||||
*
|
||||
* COMPLIANCE (CLAUDE.md §1): this connector is a plain, self-identifying
|
||||
* JSON client. It sends a fixed User-Agent, honours the caller's timeout, and
|
||||
* treats 401/403/429/451 as a STOP signal for that source — the same posture
|
||||
* services/api/sources/connectors.py's BLOCKED_STATUSES already takes. There
|
||||
* is no retry-harder path, no proxy rotation, no header spoofing, and adding
|
||||
* any would be a compliance violation, not a feature.
|
||||
*
|
||||
* `fetchImpl` is injectable so tests drive a real request/response cycle
|
||||
* without a network. That is a seam for TESTS, not a mock fallback: with no
|
||||
* override it uses global fetch and talks to the configured URL for real.
|
||||
*/
|
||||
import {
|
||||
ConnectorNotConnectedError,
|
||||
type BackfillRequest,
|
||||
type Connector,
|
||||
type ConnectorConfig,
|
||||
type ConnectorContext,
|
||||
type ConnectorHealth,
|
||||
type ConnectorRecord,
|
||||
type DiscoveryResult,
|
||||
type ReadRequest,
|
||||
type ReadResult,
|
||||
type ValidationResult,
|
||||
} from "./connector";
|
||||
import type { IngestionCheckpoint } from "../types/ingestion";
|
||||
import type { RecordSchemaFields } from "../schema/record-schema";
|
||||
|
||||
export const HTTP_JSON_CONNECTOR_VERSION = "http-json-connector@1";
|
||||
|
||||
/** Statuses that mean "stop asking this way" — never "retry harder". */
|
||||
const BLOCKED_STATUSES = new Set([401, 403, 429, 451]);
|
||||
|
||||
export interface HttpJsonConnectorConfig extends ConnectorConfig {
|
||||
/** Absolute https:// (or http:// for a local fixture) base URL. */
|
||||
base_url: string;
|
||||
/** Path serving records, e.g. "/v1/companies". */
|
||||
records_path: string;
|
||||
/** Partition key this endpoint serves. */
|
||||
partition_key: string;
|
||||
/** Declared record shape for the partition — the connector does not infer schemas from data. */
|
||||
schema: RecordSchemaFields;
|
||||
/** JSON field holding the business key of each record. */
|
||||
record_key_field: string;
|
||||
/**
|
||||
* JSON field holding each record's own cursor value (an id, a sequence, an
|
||||
* `updated_at`). Optional, but REQUIRED IN PRACTICE for any API that stops
|
||||
* returning `next_cursor` once a page is exhausted: without it the
|
||||
* connector has nothing to resume from after reading to the end, and
|
||||
* resume() would silently restart at the beginning. The conformance suite
|
||||
* catches exactly that ("read: an exhausted partition reports exhausted"),
|
||||
* which is how this field came to exist.
|
||||
*/
|
||||
cursor_field?: string;
|
||||
/** Request timeout (ms). Default 15000. */
|
||||
timeout_ms?: number;
|
||||
}
|
||||
|
||||
type FetchImpl = (url: string, init: RequestInit) => Promise<Response>;
|
||||
|
||||
/** Wire shape this connector expects from the upstream endpoint. */
|
||||
interface UpstreamPage {
|
||||
records?: Array<Record<string, unknown>>;
|
||||
next_cursor?: string | null;
|
||||
}
|
||||
|
||||
function isConfig(config: ConnectorConfig): config is HttpJsonConnectorConfig {
|
||||
return typeof config.base_url === "string" && typeof config.records_path === "string";
|
||||
}
|
||||
|
||||
export class HttpJsonConnector implements Connector {
|
||||
readonly connector_id = "http_json";
|
||||
readonly connector_version = HTTP_JSON_CONNECTOR_VERSION;
|
||||
|
||||
private config?: HttpJsonConnectorConfig;
|
||||
private ctx?: ConnectorContext;
|
||||
private lastCursor = new Map<string, string | undefined>();
|
||||
|
||||
constructor(private readonly fetchImpl: FetchImpl = (url, init) => fetch(url, init)) {}
|
||||
|
||||
validate(config: ConnectorConfig): ValidationResult {
|
||||
const errors: string[] = [];
|
||||
if (typeof config.base_url !== "string" || config.base_url.length === 0) {
|
||||
errors.push("base_url is required");
|
||||
} else if (!/^https?:\/\//.test(config.base_url)) {
|
||||
errors.push("base_url must be an absolute http(s) URL");
|
||||
}
|
||||
if (typeof config.records_path !== "string" || !config.records_path.startsWith("/")) {
|
||||
errors.push("records_path is required and must start with /");
|
||||
}
|
||||
if (typeof config.partition_key !== "string" || config.partition_key.length === 0) {
|
||||
errors.push("partition_key is required");
|
||||
}
|
||||
if (typeof config.record_key_field !== "string" || config.record_key_field.length === 0) {
|
||||
errors.push("record_key_field is required");
|
||||
}
|
||||
const schema = config.schema;
|
||||
if (typeof schema !== "object" || schema === null || Object.keys(schema).length === 0) {
|
||||
errors.push("schema is required and must declare at least one field");
|
||||
}
|
||||
return { valid: errors.length === 0, errors };
|
||||
}
|
||||
|
||||
async connect(config: ConnectorConfig, ctx: ConnectorContext): Promise<void> {
|
||||
const verdict = this.validate(config);
|
||||
if (!verdict.valid) {
|
||||
throw new Error(`http_json connector config invalid: ${verdict.errors.join("; ")}`);
|
||||
}
|
||||
if (!isConfig(config)) throw new Error("http_json connector config invalid");
|
||||
this.config = config;
|
||||
this.ctx = ctx;
|
||||
}
|
||||
|
||||
async discover(): Promise<DiscoveryResult> {
|
||||
const config = this.requireSession("discover");
|
||||
return {
|
||||
partitions: [
|
||||
{
|
||||
partition_key: config.partition_key,
|
||||
name: `${config.base_url}${config.records_path}`,
|
||||
schema: config.schema,
|
||||
},
|
||||
],
|
||||
// A polled JSON endpoint serves batch and (by repeated polling) stream.
|
||||
// It is NOT a change feed: it carries no delete events, so "cdc" is not
|
||||
// declared. Declaring it would be a lie the runner would act on.
|
||||
modes: ["batch", "stream"],
|
||||
};
|
||||
}
|
||||
|
||||
async health(): Promise<ConnectorHealth> {
|
||||
const config = this.requireSession("health");
|
||||
const started = Date.now();
|
||||
try {
|
||||
const res = await this.request(this.url(config.records_path, { limit: "1" }), config);
|
||||
const latency = Date.now() - started;
|
||||
if (BLOCKED_STATUSES.has(res.status)) {
|
||||
return { status: "down", latency_ms: latency, detail: `source signalled a block (HTTP ${res.status})` };
|
||||
}
|
||||
if (!res.ok) return { status: "degraded", latency_ms: latency, detail: `HTTP ${res.status}` };
|
||||
return { status: "up", latency_ms: latency };
|
||||
} catch (err) {
|
||||
return {
|
||||
status: "down",
|
||||
latency_ms: Date.now() - started,
|
||||
detail: err instanceof Error ? err.message : "unreachable",
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
async read(req: ReadRequest): Promise<ReadResult> {
|
||||
const config = this.requireSession("read");
|
||||
const query: Record<string, string> = { limit: String(req.limit) };
|
||||
if (req.cursor !== undefined) query.cursor = req.cursor;
|
||||
const res = await this.request(this.url(config.records_path, query), config);
|
||||
if (BLOCKED_STATUSES.has(res.status)) {
|
||||
throw new Error(`source signalled a block (HTTP ${res.status}) — stop, do not retry harder`);
|
||||
}
|
||||
if (!res.ok) throw new Error(`upstream returned HTTP ${res.status}`);
|
||||
const page = (await res.json()) as UpstreamPage;
|
||||
const rawRecords = page.records ?? [];
|
||||
// The contract says limit is a HARD bound; a server that over-delivers is
|
||||
// truncated here rather than silently widening the caller's page.
|
||||
const records = rawRecords.slice(0, req.limit).map((row) => this.toRecord(row, config));
|
||||
const serverCursor = page.next_cursor ?? undefined;
|
||||
// When the server stops issuing a cursor at the end of a page, fall back
|
||||
// to the last record's own cursor value so a later resume() starts AFTER
|
||||
// the data already produced instead of at the beginning.
|
||||
const trailing = records.length > 0 ? records[records.length - 1].cursor : undefined;
|
||||
const nextCursor = serverCursor ?? trailing ?? req.cursor;
|
||||
if (records.length > 0) this.lastCursor.set(req.partition_key, nextCursor);
|
||||
return {
|
||||
records,
|
||||
next_cursor: nextCursor,
|
||||
// The SERVER's cursor is what says "there is more"; a fallback cursor
|
||||
// only preserves position, it never claims more data exists.
|
||||
exhausted: serverCursor === undefined || records.length === 0,
|
||||
};
|
||||
}
|
||||
|
||||
async checkpoint(partitionKey: string): Promise<{ partition_key: string; cursor?: string }> {
|
||||
this.requireSession("checkpoint");
|
||||
return { partition_key: partitionKey, cursor: this.lastCursor.get(partitionKey) };
|
||||
}
|
||||
|
||||
async resume(checkpoint: IngestionCheckpoint, limit: number): Promise<ReadResult> {
|
||||
this.requireSession("resume");
|
||||
return this.read({
|
||||
partition_key: checkpoint.partition_key,
|
||||
cursor: checkpoint.cursor,
|
||||
limit,
|
||||
});
|
||||
}
|
||||
|
||||
async backfill(req: BackfillRequest): Promise<ReadResult> {
|
||||
const config = this.requireSession("backfill");
|
||||
if (!req.from || !req.to) {
|
||||
throw new Error("backfill requires both `from` and `to` — an unbounded backfill is a full reload, not a backfill");
|
||||
}
|
||||
const res = await this.request(
|
||||
this.url(config.records_path, { from: req.from, to: req.to, limit: String(req.limit) }),
|
||||
config,
|
||||
);
|
||||
if (BLOCKED_STATUSES.has(res.status)) {
|
||||
throw new Error(`source signalled a block (HTTP ${res.status}) — stop, do not retry harder`);
|
||||
}
|
||||
if (!res.ok) throw new Error(`upstream returned HTTP ${res.status}`);
|
||||
const page = (await res.json()) as UpstreamPage;
|
||||
const records = (page.records ?? []).slice(0, req.limit).map((row) => this.toRecord(row, config));
|
||||
return { records, next_cursor: page.next_cursor ?? undefined, exhausted: true };
|
||||
}
|
||||
|
||||
async disconnect(): Promise<void> {
|
||||
this.config = undefined;
|
||||
this.ctx = undefined;
|
||||
this.lastCursor.clear();
|
||||
}
|
||||
|
||||
// --- internals -----------------------------------------------------------
|
||||
|
||||
private toRecord(row: Record<string, unknown>, config: HttpJsonConnectorConfig): ConnectorRecord {
|
||||
const key = row[config.record_key_field];
|
||||
if (typeof key !== "string" && typeof key !== "number") {
|
||||
throw new Error(`record is missing string/number key field "${config.record_key_field}"`);
|
||||
}
|
||||
let cursor: string | undefined;
|
||||
if (config.cursor_field) {
|
||||
const raw = row[config.cursor_field];
|
||||
if (typeof raw === "string" || typeof raw === "number") cursor = String(raw);
|
||||
}
|
||||
return { record_key: String(key), payload: row, operation: "upsert", cursor };
|
||||
}
|
||||
|
||||
private requireSession(op: string): HttpJsonConnectorConfig {
|
||||
if (!this.config) throw new ConnectorNotConnectedError(this.connector_id, op);
|
||||
return this.config;
|
||||
}
|
||||
|
||||
private url(path: string, query: Record<string, string>): string {
|
||||
const config = this.config;
|
||||
if (!config) throw new ConnectorNotConnectedError(this.connector_id, "url");
|
||||
const url = new URL(path, config.base_url);
|
||||
for (const [k, v] of Object.entries(query)) url.searchParams.set(k, v);
|
||||
return url.toString();
|
||||
}
|
||||
|
||||
private async request(url: string, config: HttpJsonConnectorConfig): Promise<Response> {
|
||||
const controller = new AbortController();
|
||||
const timer = setTimeout(() => controller.abort(), config.timeout_ms ?? 15_000);
|
||||
try {
|
||||
return await this.fetchImpl(url, {
|
||||
method: "GET",
|
||||
signal: controller.signal,
|
||||
headers: {
|
||||
Accept: "application/json",
|
||||
"User-Agent": "0pt-connector/1 (+https://0pt.app; SIAX Technology AB)",
|
||||
...(this.ctx ? { "X-Correlation-Id": this.ctx.run_id } : {}),
|
||||
},
|
||||
});
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,224 @@
|
||||
/**
|
||||
* src/connectors/jsonl-file-connector.ts
|
||||
* ========================================
|
||||
* P061 conformance connector #2: a JSONL file drop on local disk.
|
||||
*
|
||||
* Heterogeneity vs http-json-connector.ts — these two share nothing but the
|
||||
* contract:
|
||||
* - no network, no server, no opaque token; the cursor is a LINE OFFSET the
|
||||
* connector computes itself,
|
||||
* - it knows the full record count up front (the file is finite and local),
|
||||
* so `exhausted` is exact rather than server-reported,
|
||||
* - health is a filesystem stat, not an HTTP round trip,
|
||||
* - it serves `cdc`-shaped records: a JSONL line may carry an `_op` field
|
||||
* ("insert"/"update"/"delete"), so this connector emits real delete
|
||||
* envelopes with a null payload. The HTTP connector cannot,
|
||||
* - backfill bounds are line numbers, not timestamps.
|
||||
*
|
||||
* That last point is why the conformance suite takes `from`/`to` from a
|
||||
* fixture: the contract fixes that a backfill is BOUNDED, not what the
|
||||
* bounds mean. Each connector documents its own bound vocabulary.
|
||||
*/
|
||||
import { promises as fs } from "fs";
|
||||
import path from "path";
|
||||
import {
|
||||
ConnectorNotConnectedError,
|
||||
type BackfillRequest,
|
||||
type Connector,
|
||||
type ConnectorConfig,
|
||||
type ConnectorContext,
|
||||
type ConnectorHealth,
|
||||
type ConnectorRecord,
|
||||
type DiscoveryResult,
|
||||
type ReadRequest,
|
||||
type ReadResult,
|
||||
type ValidationResult,
|
||||
} from "./connector";
|
||||
import type { IngestionCheckpoint } from "../types/ingestion";
|
||||
import type { RecordSchemaFields } from "../schema/record-schema";
|
||||
|
||||
export const JSONL_FILE_CONNECTOR_VERSION = "jsonl-file-connector@1";
|
||||
|
||||
export interface JsonlFileConnectorConfig extends ConnectorConfig {
|
||||
/** Absolute path to the .jsonl file. */
|
||||
file_path: string;
|
||||
partition_key: string;
|
||||
schema: RecordSchemaFields;
|
||||
record_key_field: string;
|
||||
}
|
||||
|
||||
function isConfig(config: ConnectorConfig): config is JsonlFileConnectorConfig {
|
||||
return typeof config.file_path === "string" && typeof config.partition_key === "string";
|
||||
}
|
||||
|
||||
/** Cursor vocabulary: the decimal count of lines already produced. */
|
||||
function parseCursor(cursor: string | undefined): number {
|
||||
if (cursor === undefined) return 0;
|
||||
const n = Number.parseInt(cursor, 10);
|
||||
if (!Number.isInteger(n) || n < 0) throw new Error(`invalid cursor "${cursor}" — expected a line offset`);
|
||||
return n;
|
||||
}
|
||||
|
||||
export class JsonlFileConnector implements Connector {
|
||||
readonly connector_id = "jsonl_file";
|
||||
readonly connector_version = JSONL_FILE_CONNECTOR_VERSION;
|
||||
|
||||
private config?: JsonlFileConnectorConfig;
|
||||
private lastCursor = new Map<string, string | undefined>();
|
||||
|
||||
validate(config: ConnectorConfig): ValidationResult {
|
||||
const errors: string[] = [];
|
||||
if (typeof config.file_path !== "string" || config.file_path.length === 0) {
|
||||
errors.push("file_path is required");
|
||||
} else if (!path.isAbsolute(config.file_path)) {
|
||||
errors.push("file_path must be absolute");
|
||||
} else if (!config.file_path.endsWith(".jsonl")) {
|
||||
errors.push("file_path must name a .jsonl file");
|
||||
}
|
||||
if (typeof config.partition_key !== "string" || config.partition_key.length === 0) {
|
||||
errors.push("partition_key is required");
|
||||
}
|
||||
if (typeof config.record_key_field !== "string" || config.record_key_field.length === 0) {
|
||||
errors.push("record_key_field is required");
|
||||
}
|
||||
const schema = config.schema;
|
||||
if (typeof schema !== "object" || schema === null || Object.keys(schema).length === 0) {
|
||||
errors.push("schema is required and must declare at least one field");
|
||||
}
|
||||
return { valid: errors.length === 0, errors };
|
||||
}
|
||||
|
||||
async connect(config: ConnectorConfig, _ctx: ConnectorContext): Promise<void> {
|
||||
const verdict = this.validate(config);
|
||||
if (!verdict.valid) {
|
||||
throw new Error(`jsonl_file connector config invalid: ${verdict.errors.join("; ")}`);
|
||||
}
|
||||
if (!isConfig(config)) throw new Error("jsonl_file connector config invalid");
|
||||
// Fail at connect time, not at first read: a file drop that is not there
|
||||
// is a connection failure, and the runner must see it as one.
|
||||
await fs.access(config.file_path);
|
||||
this.config = config;
|
||||
}
|
||||
|
||||
async discover(): Promise<DiscoveryResult> {
|
||||
const config = this.requireSession("discover");
|
||||
return {
|
||||
partitions: [
|
||||
{ partition_key: config.partition_key, name: path.basename(config.file_path), schema: config.schema },
|
||||
],
|
||||
// A file drop is a batch source; because a line may carry `_op`, it can
|
||||
// also carry a change log, so cdc is honestly available here.
|
||||
modes: ["batch", "cdc"],
|
||||
};
|
||||
}
|
||||
|
||||
async health(): Promise<ConnectorHealth> {
|
||||
const config = this.requireSession("health");
|
||||
const started = Date.now();
|
||||
try {
|
||||
const stat = await fs.stat(config.file_path);
|
||||
if (stat.size === 0) {
|
||||
return { status: "degraded", latency_ms: Date.now() - started, detail: "file is empty" };
|
||||
}
|
||||
return { status: "up", latency_ms: Date.now() - started };
|
||||
} catch (err) {
|
||||
return {
|
||||
status: "down",
|
||||
latency_ms: Date.now() - started,
|
||||
detail: err instanceof Error ? err.message : "stat failed",
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
async read(req: ReadRequest): Promise<ReadResult> {
|
||||
const lines = await this.lines();
|
||||
const start = parseCursor(req.cursor);
|
||||
const slice = lines.slice(start, start + req.limit);
|
||||
const records = slice.map((line, i) => this.toRecord(line, start + i));
|
||||
const nextOffset = start + slice.length;
|
||||
const cursor = String(nextOffset);
|
||||
this.lastCursor.set(req.partition_key, cursor);
|
||||
return {
|
||||
records,
|
||||
next_cursor: cursor,
|
||||
exhausted: nextOffset >= lines.length,
|
||||
};
|
||||
}
|
||||
|
||||
async checkpoint(partitionKey: string): Promise<{ partition_key: string; cursor?: string }> {
|
||||
this.requireSession("checkpoint");
|
||||
return { partition_key: partitionKey, cursor: this.lastCursor.get(partitionKey) };
|
||||
}
|
||||
|
||||
async resume(checkpoint: IngestionCheckpoint, limit: number): Promise<ReadResult> {
|
||||
this.requireSession("resume");
|
||||
return this.read({ partition_key: checkpoint.partition_key, cursor: checkpoint.cursor, limit });
|
||||
}
|
||||
|
||||
/** Bounds are LINE NUMBERS: `from` inclusive, `to` exclusive. */
|
||||
async backfill(req: BackfillRequest): Promise<ReadResult> {
|
||||
this.requireSession("backfill");
|
||||
if (!req.from || !req.to) {
|
||||
throw new Error("backfill requires both `from` and `to` — an unbounded backfill is a full reload, not a backfill");
|
||||
}
|
||||
const from = parseCursor(req.from);
|
||||
const to = parseCursor(req.to);
|
||||
if (to <= from) throw new Error(`backfill range is empty or inverted (from=${from}, to=${to})`);
|
||||
const lines = await this.lines();
|
||||
const slice = lines.slice(from, Math.min(to, from + req.limit));
|
||||
return {
|
||||
records: slice.map((line, i) => this.toRecord(line, from + i)),
|
||||
next_cursor: String(from + slice.length),
|
||||
exhausted: true,
|
||||
};
|
||||
}
|
||||
|
||||
async disconnect(): Promise<void> {
|
||||
this.config = undefined;
|
||||
this.lastCursor.clear();
|
||||
}
|
||||
|
||||
// --- internals -----------------------------------------------------------
|
||||
|
||||
private requireSession(op: string): JsonlFileConnectorConfig {
|
||||
if (!this.config) throw new ConnectorNotConnectedError(this.connector_id, op);
|
||||
return this.config;
|
||||
}
|
||||
|
||||
private async lines(): Promise<string[]> {
|
||||
const config = this.requireSession("read");
|
||||
const raw = await fs.readFile(config.file_path, "utf8");
|
||||
return raw.split("\n").filter((l) => l.trim().length > 0);
|
||||
}
|
||||
|
||||
private toRecord(line: string, lineNumber: number): ConnectorRecord {
|
||||
const config = this.requireSession("read");
|
||||
let parsed: Record<string, unknown>;
|
||||
try {
|
||||
parsed = JSON.parse(line) as Record<string, unknown>;
|
||||
} catch {
|
||||
throw new Error(`line ${lineNumber + 1} is not valid JSON`);
|
||||
}
|
||||
const key = parsed[config.record_key_field];
|
||||
if (typeof key !== "string" && typeof key !== "number") {
|
||||
throw new Error(`line ${lineNumber + 1} is missing key field "${config.record_key_field}"`);
|
||||
}
|
||||
const op = parsed._op;
|
||||
const operation =
|
||||
op === "insert" || op === "update" || op === "delete" ? op : ("upsert" as const);
|
||||
const occurredAt = typeof parsed._occurred_at === "string" ? parsed._occurred_at : undefined;
|
||||
// `_op`/`_occurred_at` are envelope metadata, not payload fields — leaving
|
||||
// them in the payload would make every record fail schema conformance for
|
||||
// an "unregistered field".
|
||||
const payload: Record<string, unknown> = { ...parsed };
|
||||
delete payload._op;
|
||||
delete payload._occurred_at;
|
||||
return {
|
||||
record_key: String(key),
|
||||
payload: operation === "delete" ? null : payload,
|
||||
operation,
|
||||
occurred_at: occurredAt,
|
||||
cursor: String(lineNumber + 1),
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -23,6 +23,16 @@
|
||||
* GENERIC means GENERIC: no person-matching logic, no Ratsit/PII pipelines —
|
||||
* records are opaque field maps supplied by the calling product under its
|
||||
* own legal basis; this module implements comparison mechanics only.
|
||||
*
|
||||
* P065 STATUS — THIS IS THE CANONICAL ENGINE, AND IT IS NOT YET THE ONLY ONE.
|
||||
* `services/api/workers/resolve.py` is a second, simplified ER engine (fuzzy
|
||||
* threshold, auto-merge, no hard rules, no review band) that
|
||||
* `services/api/workers/pipeline.py` still calls. P065's DoD ("no second
|
||||
* simplified duplicated production engine remains") is therefore NOT met.
|
||||
* See that file's header for why it was not rewritten blind in the
|
||||
* P060-P068 pass and STATUS.md for the removal plan. The ingestion plane
|
||||
* added in that pass (src/ingestion/*) deliberately does no ER of its own —
|
||||
* it stops at the sink, so it adds no THIRD engine.
|
||||
*/
|
||||
import crypto from "crypto";
|
||||
import { SiaxError } from "../platform/errors";
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
/**
|
||||
* src/ingestion/envelope.ts
|
||||
* ==========================
|
||||
* P062: the one place a ConnectorRecord becomes an IngestionEnvelope. Every
|
||||
* connector, every mode, one function — that is what "normalize all ingestion
|
||||
* to a common envelope" means operationally.
|
||||
*
|
||||
* `hashPayload` is the stable content hash the idempotency ledger (P068) and
|
||||
* the deterministic `envelope_id` are both built on. Key order must not
|
||||
* change the hash — two connectors reading the same record can serialize the
|
||||
* same object with different key order — so keys are sorted recursively.
|
||||
* Undefined values are dropped (JSON has no undefined); null is kept, because
|
||||
* "field explicitly null" is different data from "field absent".
|
||||
*/
|
||||
import crypto from "crypto";
|
||||
import type { ConnectorRecord } from "../connectors/connector";
|
||||
import type { EnvelopeOperation, IngestionEnvelope, IngestionMode } from "../types/ingestion";
|
||||
|
||||
/** Recursively key-sorted JSON — stable across key insertion order. */
|
||||
export function stableStringify(value: unknown): string {
|
||||
if (value === null) return "null";
|
||||
if (Array.isArray(value)) return `[${value.map(stableStringify).join(",")}]`;
|
||||
if (typeof value === "object") {
|
||||
const entries = Object.entries(value as Record<string, unknown>)
|
||||
.filter(([, v]) => v !== undefined)
|
||||
.sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0));
|
||||
return `{${entries.map(([k, v]) => `${JSON.stringify(k)}:${stableStringify(v)}`).join(",")}}`;
|
||||
}
|
||||
return JSON.stringify(value) ?? "null";
|
||||
}
|
||||
|
||||
/** Content hash of an envelope payload. A delete (null payload) hashes as "deleted". */
|
||||
export function hashPayload(payload: Record<string, unknown> | null): string {
|
||||
if (payload === null) return "sha256:deleted";
|
||||
return `sha256:${crypto.createHash("sha256").update(stableStringify(payload)).digest("hex")}`;
|
||||
}
|
||||
|
||||
export interface BuildEnvelopeInput {
|
||||
record: ConnectorRecord;
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
mode: IngestionMode;
|
||||
partition_key: string;
|
||||
schema_version: number;
|
||||
sequence: number;
|
||||
connector_id: string;
|
||||
connector_version: string;
|
||||
run_id: string;
|
||||
/** Cursor covering this record, when the connector reports one. */
|
||||
cursor?: string;
|
||||
now?: string;
|
||||
}
|
||||
|
||||
export function buildEnvelope(input: BuildEnvelopeInput): IngestionEnvelope {
|
||||
const operation: EnvelopeOperation = input.record.operation ?? "upsert";
|
||||
const payload = operation === "delete" ? null : input.record.payload;
|
||||
const contentHash = hashPayload(payload);
|
||||
return {
|
||||
// Deterministic: the same record with the same content re-read after a
|
||||
// crash produces the same envelope_id, so a downstream sink that dedups
|
||||
// by id sees a retry, not a new record.
|
||||
envelope_id: `env_${crypto
|
||||
.createHash("sha256")
|
||||
.update(`${input.source_id}|${input.partition_key}|${input.record.record_key}|${contentHash}`)
|
||||
.digest("hex")
|
||||
.slice(0, 32)}`,
|
||||
source_id: input.source_id,
|
||||
tenant_id: input.tenant_id,
|
||||
project_id: input.project_id,
|
||||
mode: input.mode,
|
||||
operation,
|
||||
partition_key: input.partition_key,
|
||||
record_key: input.record.record_key,
|
||||
schema_version: input.schema_version,
|
||||
sequence: input.sequence,
|
||||
occurred_at: input.record.occurred_at,
|
||||
ingested_at: input.now ?? new Date().toISOString(),
|
||||
payload,
|
||||
provenance: {
|
||||
connector_id: input.connector_id,
|
||||
connector_version: input.connector_version,
|
||||
run_id: input.run_id,
|
||||
cursor: input.cursor ?? input.record.cursor,
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,557 @@
|
||||
/**
|
||||
* src/ingestion/ingestion-runner.ts
|
||||
* ==================================
|
||||
* The single ingestion highway (P062, P063 gate, P064 gate, P067 emission,
|
||||
* P068 ledger). One loop, one envelope shape, one durable checkpoint.
|
||||
*
|
||||
* ORDERING IS THE WHOLE CONTRACT. Per page:
|
||||
*
|
||||
* read/resume -> normalize to envelopes -> schema conformance -> quality gate
|
||||
* -> ledger PEEK (filter replays) -> SINK -> ledger COMMIT -> checkpoint COMMIT
|
||||
*
|
||||
* Both commits happen AFTER the sink resolves. That is what makes "a restart
|
||||
* resumes without silent gaps" true: a process that dies anywhere before the
|
||||
* commits leaves the page unconsumed, the next run re-reads it, and nothing is
|
||||
* lost. The two tempting shortcuts are both data-loss bugs, and one of them
|
||||
* was actually written here first:
|
||||
* - committing the CHECKPOINT before the sink skips the page entirely;
|
||||
* - committing the LEDGER before the sink marks records delivered that the
|
||||
* sink never received, so the resumed run drops them as duplicates. This
|
||||
* one shipped in the first draft of this file and was caught by
|
||||
* tests/p0/ingestion-runner.test.ts's crash/resume case (7 of 10 records
|
||||
* delivered). The peek/commit split in IngestionRepository exists because
|
||||
* of it.
|
||||
*
|
||||
* The resulting delivery guarantee is AT-LEAST-ONCE, not exactly-once: two
|
||||
* concurrent runs on the same partition can both peek "new". `envelope_id` is
|
||||
* deterministic so a dedup-capable sink survives that. A per-(source,
|
||||
* partition) run lock does not exist yet — see STATUS.md.
|
||||
*
|
||||
* A BLOCKED quality page also stops WITHOUT committing: a blocked page must
|
||||
* stay unconsumed, not be skipped over.
|
||||
*
|
||||
* WHAT THIS RUNNER DOES NOT DO, stated plainly:
|
||||
* - it does not schedule itself (ACT0 owns scheduling; the source's
|
||||
* `schedule` row is a declaration, not a cron daemon in this repo),
|
||||
* - it does not write to a lakehouse — the `sink` is supplied by the
|
||||
* caller, because the Iceberg write boundary is ST0RE's contract
|
||||
* (src/services/iceberg-write-service.ts) and is owned by another repo
|
||||
* in this programme (P066),
|
||||
* - it does not do entity resolution. The canonical ER engine is
|
||||
* src/er/entity-resolution-service.ts and runs downstream of the sink.
|
||||
*/
|
||||
import crypto from "crypto";
|
||||
import { SiaxError } from "../platform/errors";
|
||||
import { conformanceErrors } from "../schema/compatibility";
|
||||
import type { SchemaRegistry } from "../schema/schema-registry";
|
||||
import { QualityGate, type QualityEnforcementPolicy } from "../quality/quality-policy";
|
||||
import type { IngestionRepository } from "../repositories/ingestion-repository";
|
||||
import type { LineageStore } from "../lineage/lineage-store";
|
||||
import type {
|
||||
BackfillRequest,
|
||||
Connector,
|
||||
ConnectorConfig,
|
||||
ReadResult,
|
||||
} from "../connectors/connector";
|
||||
import type {
|
||||
IngestionCheckpoint,
|
||||
IngestionEnvelope,
|
||||
IngestionMode,
|
||||
QuarantineRecord,
|
||||
} from "../types/ingestion";
|
||||
import { buildEnvelope, hashPayload } from "./envelope";
|
||||
|
||||
/** Where accepted envelopes go. Supplied by the caller — see the module note. */
|
||||
export type IngestionSink = (envelopes: IngestionEnvelope[]) => Promise<void>;
|
||||
|
||||
export interface IngestionRunInput {
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
mode: IngestionMode;
|
||||
connector: Connector;
|
||||
config: ConnectorConfig;
|
||||
sink: IngestionSink;
|
||||
/** Deterministic run id. Defaults to a random one. */
|
||||
run_id?: string;
|
||||
/** Records per connector read. Default 100. */
|
||||
page_size?: number;
|
||||
/** Safety stop. Default 1000 pages. */
|
||||
max_pages?: number;
|
||||
/** Only this partition; default is every partition discover() reports. */
|
||||
partition_key?: string;
|
||||
/** P064. Omitted = no gate runs, and the report says so explicitly. */
|
||||
quality_policy?: QualityEnforcementPolicy;
|
||||
/** Canonical ref of what the sink produces, for lineage (e.g. "0pt:dataset:ds_x"). */
|
||||
output_ref?: string;
|
||||
correlation_id?: string;
|
||||
}
|
||||
|
||||
export interface PartitionRunReport {
|
||||
partition_key: string;
|
||||
schema_version: number;
|
||||
pages_read: number;
|
||||
records_read: number;
|
||||
envelopes_emitted: number;
|
||||
duplicates_skipped: number;
|
||||
changed: number;
|
||||
quarantined: number;
|
||||
quality_violations: number;
|
||||
checkpoint: IngestionCheckpoint;
|
||||
/** Set when the partition stopped early. */
|
||||
stopped_because?: "blocked_by_quality" | "max_pages" | "sink_failed";
|
||||
}
|
||||
|
||||
export interface IngestionRunReport {
|
||||
run_id: string;
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
mode: IngestionMode;
|
||||
status: "succeeded" | "failed";
|
||||
/** Honest marker: "none_declared" when the caller supplied no policy at all. */
|
||||
quality_policy: "block" | "quarantine" | "warn" | "none_declared";
|
||||
partitions: PartitionRunReport[];
|
||||
lineage_edges: number;
|
||||
/** Present when status is "failed". */
|
||||
failure?: { code: string; detail: string };
|
||||
}
|
||||
|
||||
export class IngestionRunner {
|
||||
private readonly gate: QualityGate;
|
||||
|
||||
constructor(
|
||||
private readonly repo: IngestionRepository,
|
||||
private readonly schemas: SchemaRegistry,
|
||||
private readonly lineage?: LineageStore,
|
||||
gate?: QualityGate,
|
||||
) {
|
||||
this.gate = gate ?? new QualityGate();
|
||||
}
|
||||
|
||||
async run(input: IngestionRunInput): Promise<IngestionRunReport> {
|
||||
const runId = input.run_id ?? `run_${crypto.randomUUID()}`;
|
||||
const pageSize = input.page_size ?? 100;
|
||||
const maxPages = input.max_pages ?? 1000;
|
||||
const report: IngestionRunReport = {
|
||||
run_id: runId,
|
||||
source_id: input.source_id,
|
||||
tenant_id: input.tenant_id,
|
||||
mode: input.mode,
|
||||
status: "succeeded",
|
||||
quality_policy: input.quality_policy?.on_violation ?? "none_declared",
|
||||
partitions: [],
|
||||
lineage_edges: 0,
|
||||
};
|
||||
|
||||
const verdict = input.connector.validate(input.config);
|
||||
if (!verdict.valid) {
|
||||
throw new SiaxError(
|
||||
"INVALID_REQUEST",
|
||||
"0pt.ingestion",
|
||||
`connector config rejected: ${verdict.errors.join("; ")}`,
|
||||
);
|
||||
}
|
||||
|
||||
await input.connector.connect(input.config, {
|
||||
tenant_id: input.tenant_id,
|
||||
project_id: input.project_id,
|
||||
source_id: input.source_id,
|
||||
run_id: runId,
|
||||
});
|
||||
|
||||
try {
|
||||
const discovery = await input.connector.discover();
|
||||
if (!discovery.modes.includes(input.mode)) {
|
||||
throw new SiaxError(
|
||||
"INVALID_REQUEST",
|
||||
"0pt.ingestion",
|
||||
`connector ${input.connector.connector_id} does not serve mode "${input.mode}" for this config (serves: ${discovery.modes.join(", ")})`,
|
||||
);
|
||||
}
|
||||
const partitions = input.partition_key
|
||||
? discovery.partitions.filter((p) => p.partition_key === input.partition_key)
|
||||
: discovery.partitions;
|
||||
if (partitions.length === 0) {
|
||||
throw new SiaxError(
|
||||
"NOT_FOUND",
|
||||
"0pt.ingestion",
|
||||
`partition "${input.partition_key}" was not discovered on source ${input.source_id}`,
|
||||
);
|
||||
}
|
||||
|
||||
// ---- P063: schema gate, BEFORE any record is read ------------------
|
||||
// Every partition's shape is checked and registered first. A break
|
||||
// fails the whole run here, so no envelope — and therefore no consumer
|
||||
// write — can happen under a broken schema.
|
||||
const schemaVersions = new Map<string, number>();
|
||||
for (const partition of partitions) {
|
||||
const evaluated = await this.schemas.evaluate(input.source_id, partition.schema);
|
||||
if (evaluated.verdict && !evaluated.verdict.compatible) {
|
||||
throw new SiaxError(
|
||||
"CONFLICT",
|
||||
"0pt.ingestion",
|
||||
`source ${input.source_id} partition "${partition.partition_key}" changed shape in a way that breaks its declared ` +
|
||||
`"${evaluated.verdict.policy}" compatibility policy — refusing to ingest: ` +
|
||||
evaluated.verdict.breaks.map((b) => `[${b.breaks}] ${b.detail}`).join("; "),
|
||||
);
|
||||
}
|
||||
const registered = await this.schemas.register({
|
||||
source_id: input.source_id,
|
||||
tenant_id: input.tenant_id,
|
||||
fields: partition.schema,
|
||||
compatibility: evaluated.previous?.compatibility ?? "backward",
|
||||
registered_by: `run:${runId}`,
|
||||
});
|
||||
schemaVersions.set(partition.partition_key, registered.schema.schema_version);
|
||||
}
|
||||
|
||||
for (const partition of partitions) {
|
||||
report.partitions.push(
|
||||
await this.runPartition(input, {
|
||||
runId,
|
||||
pageSize,
|
||||
maxPages,
|
||||
partitionKey: partition.partition_key,
|
||||
schemaVersion: schemaVersions.get(partition.partition_key) ?? 1,
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
// ---- P067: one lineage event per run, source -> output -------------
|
||||
if (this.lineage) {
|
||||
report.lineage_edges = await this.lineage.record(
|
||||
{
|
||||
eventType: "COMPLETE",
|
||||
eventTime: new Date().toISOString(),
|
||||
producer: "https://git.siax.io/sax3l/0pt",
|
||||
run: { runId },
|
||||
job: { namespace: "0pt", name: "0pt.ingestion.run.v1" },
|
||||
inputs: [
|
||||
{
|
||||
namespace: "0pt",
|
||||
name: `source/${input.source_id}`,
|
||||
facets: { siaxResourceRef: { ref: `0pt:source:${input.source_id}` } },
|
||||
},
|
||||
],
|
||||
outputs: [
|
||||
{
|
||||
namespace: "0pt",
|
||||
name: input.output_ref ?? `ingestion/${input.source_id}`,
|
||||
...(input.output_ref ? { facets: { siaxResourceRef: { ref: input.output_ref } } } : {}),
|
||||
},
|
||||
],
|
||||
},
|
||||
input.tenant_id,
|
||||
input.correlation_id,
|
||||
);
|
||||
}
|
||||
|
||||
return report;
|
||||
} finally {
|
||||
await input.connector.disconnect();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* P068: a bounded historical re-read. Runs through the SAME envelope,
|
||||
* quality and ledger path as `run`, and deliberately does NOT touch the
|
||||
* streaming checkpoint — a backfill must not rewind or advance live
|
||||
* ingestion. Repeating an identical backfill emits nothing: every record
|
||||
* comes back "duplicate" from the ledger.
|
||||
*/
|
||||
async backfill(
|
||||
input: IngestionRunInput & { range: Omit<BackfillRequest, "limit"> },
|
||||
): Promise<PartitionRunReport> {
|
||||
const runId = input.run_id ?? `bkf_${crypto.randomUUID()}`;
|
||||
const pageSize = input.page_size ?? 100;
|
||||
const verdict = input.connector.validate(input.config);
|
||||
if (!verdict.valid) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.ingestion", `connector config rejected: ${verdict.errors.join("; ")}`);
|
||||
}
|
||||
await input.connector.connect(input.config, {
|
||||
tenant_id: input.tenant_id,
|
||||
project_id: input.project_id,
|
||||
source_id: input.source_id,
|
||||
run_id: runId,
|
||||
});
|
||||
try {
|
||||
const latest = await this.schemas.getLatest(input.source_id);
|
||||
const schemaVersion = latest?.schema_version ?? 1;
|
||||
const result = await input.connector.backfill({ ...input.range, limit: pageSize });
|
||||
const existing =
|
||||
(await this.repo.getCheckpoint(input.source_id, input.range.partition_key)) ??
|
||||
this.emptyCheckpoint(input, input.range.partition_key);
|
||||
const outcome = await this.consumePage(input, {
|
||||
runId,
|
||||
result,
|
||||
partitionKey: input.range.partition_key,
|
||||
schemaVersion,
|
||||
startSequence: existing.sequence,
|
||||
schemaFields: latest?.fields,
|
||||
});
|
||||
return {
|
||||
partition_key: input.range.partition_key,
|
||||
schema_version: schemaVersion,
|
||||
pages_read: 1,
|
||||
records_read: result.records.length,
|
||||
envelopes_emitted: outcome.emitted,
|
||||
duplicates_skipped: outcome.duplicates,
|
||||
changed: outcome.changed,
|
||||
quarantined: outcome.quarantined,
|
||||
quality_violations: outcome.violations,
|
||||
// Unchanged on purpose — a backfill leaves the live checkpoint alone.
|
||||
checkpoint: existing,
|
||||
stopped_because: outcome.blocked ? "blocked_by_quality" : undefined,
|
||||
};
|
||||
} finally {
|
||||
await input.connector.disconnect();
|
||||
}
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
|
||||
private emptyCheckpoint(
|
||||
input: { source_id: string; tenant_id: string },
|
||||
partitionKey: string,
|
||||
): IngestionCheckpoint {
|
||||
return {
|
||||
source_id: input.source_id,
|
||||
tenant_id: input.tenant_id,
|
||||
partition_key: partitionKey,
|
||||
cursor: undefined,
|
||||
sequence: 0,
|
||||
records_committed: 0,
|
||||
committed_at: new Date(0).toISOString(),
|
||||
};
|
||||
}
|
||||
|
||||
private async runPartition(
|
||||
input: IngestionRunInput,
|
||||
opts: {
|
||||
runId: string;
|
||||
pageSize: number;
|
||||
maxPages: number;
|
||||
partitionKey: string;
|
||||
schemaVersion: number;
|
||||
},
|
||||
): Promise<PartitionRunReport> {
|
||||
const schema = await this.schemas.getLatest(input.source_id);
|
||||
let checkpoint =
|
||||
(await this.repo.getCheckpoint(input.source_id, opts.partitionKey)) ??
|
||||
this.emptyCheckpoint(input, opts.partitionKey);
|
||||
|
||||
const stats = {
|
||||
pages: 0,
|
||||
records: 0,
|
||||
emitted: 0,
|
||||
duplicates: 0,
|
||||
changed: 0,
|
||||
quarantined: 0,
|
||||
violations: 0,
|
||||
};
|
||||
let stoppedBecause: PartitionRunReport["stopped_because"];
|
||||
|
||||
for (let page = 0; page < opts.maxPages; page++) {
|
||||
// First page of a partition that has a durable checkpoint goes through
|
||||
// resume(), so a connector that must re-establish server-side state has
|
||||
// its chance; later pages are ordinary reads from the running cursor.
|
||||
const result: ReadResult =
|
||||
page === 0 && checkpoint.cursor !== undefined
|
||||
? await input.connector.resume(checkpoint, opts.pageSize)
|
||||
: await input.connector.read({
|
||||
partition_key: opts.partitionKey,
|
||||
cursor: checkpoint.cursor,
|
||||
limit: opts.pageSize,
|
||||
});
|
||||
|
||||
stats.pages++;
|
||||
stats.records += result.records.length;
|
||||
if (result.records.length === 0) break;
|
||||
|
||||
const outcome = await this.consumePage(input, {
|
||||
runId: opts.runId,
|
||||
result,
|
||||
partitionKey: opts.partitionKey,
|
||||
schemaVersion: opts.schemaVersion,
|
||||
startSequence: checkpoint.sequence,
|
||||
schemaFields: schema?.fields,
|
||||
});
|
||||
stats.emitted += outcome.emitted;
|
||||
stats.duplicates += outcome.duplicates;
|
||||
stats.changed += outcome.changed;
|
||||
stats.quarantined += outcome.quarantined;
|
||||
stats.violations += outcome.violations;
|
||||
|
||||
if (outcome.blocked) {
|
||||
// The checkpoint is NOT advanced: the offending page stays unconsumed
|
||||
// so the next run sees it again rather than stepping over it.
|
||||
stoppedBecause = "blocked_by_quality";
|
||||
break;
|
||||
}
|
||||
|
||||
checkpoint = {
|
||||
source_id: input.source_id,
|
||||
tenant_id: input.tenant_id,
|
||||
partition_key: opts.partitionKey,
|
||||
cursor: result.next_cursor,
|
||||
sequence: checkpoint.sequence + result.records.length,
|
||||
records_committed: checkpoint.records_committed + outcome.emitted,
|
||||
committed_at: new Date().toISOString(),
|
||||
};
|
||||
await this.repo.commitCheckpoint(checkpoint);
|
||||
|
||||
if (result.exhausted) break;
|
||||
if (page === opts.maxPages - 1) stoppedBecause = "max_pages";
|
||||
}
|
||||
|
||||
return {
|
||||
partition_key: opts.partitionKey,
|
||||
schema_version: opts.schemaVersion,
|
||||
pages_read: stats.pages,
|
||||
records_read: stats.records,
|
||||
envelopes_emitted: stats.emitted,
|
||||
duplicates_skipped: stats.duplicates,
|
||||
changed: stats.changed,
|
||||
quarantined: stats.quarantined,
|
||||
quality_violations: stats.violations,
|
||||
checkpoint,
|
||||
stopped_because: stoppedBecause,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalize -> schema conformance -> quality gate -> ledger -> sink.
|
||||
* Throws if the sink throws; the caller must NOT have committed a
|
||||
* checkpoint by then.
|
||||
*/
|
||||
private async consumePage(
|
||||
input: IngestionRunInput,
|
||||
opts: {
|
||||
runId: string;
|
||||
result: ReadResult;
|
||||
partitionKey: string;
|
||||
schemaVersion: number;
|
||||
startSequence: number;
|
||||
schemaFields?: Record<string, { type: string; required: boolean }>;
|
||||
},
|
||||
): Promise<{
|
||||
emitted: number;
|
||||
duplicates: number;
|
||||
changed: number;
|
||||
quarantined: number;
|
||||
violations: number;
|
||||
blocked: boolean;
|
||||
}> {
|
||||
const envelopes: IngestionEnvelope[] = opts.result.records.map((record, i) =>
|
||||
buildEnvelope({
|
||||
record,
|
||||
source_id: input.source_id,
|
||||
tenant_id: input.tenant_id,
|
||||
project_id: input.project_id,
|
||||
mode: input.mode,
|
||||
partition_key: opts.partitionKey,
|
||||
schema_version: opts.schemaVersion,
|
||||
sequence: opts.startSequence + i + 1,
|
||||
connector_id: input.connector.connector_id,
|
||||
connector_version: input.connector.connector_version,
|
||||
run_id: opts.runId,
|
||||
cursor: opts.result.next_cursor,
|
||||
}),
|
||||
);
|
||||
|
||||
// ---- P063 per-record conformance ------------------------------------
|
||||
// A source that silently changed shape without a new discovery is caught
|
||||
// here, on the record, before the sink sees anything from this page.
|
||||
if (opts.schemaFields) {
|
||||
for (const envelope of envelopes) {
|
||||
if (envelope.payload === null) continue;
|
||||
const errors = conformanceErrors(
|
||||
opts.schemaFields as never,
|
||||
envelope.payload,
|
||||
);
|
||||
if (errors.length > 0) {
|
||||
throw new SiaxError(
|
||||
"CONFLICT",
|
||||
"0pt.ingestion",
|
||||
`record "${envelope.record_key}" does not conform to registered schema v${opts.schemaVersion} ` +
|
||||
`for source ${input.source_id}: ${errors.join("; ")}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ---- P064 quality gate ----------------------------------------------
|
||||
let accepted = envelopes;
|
||||
let quarantinedCount = 0;
|
||||
let violationCount = 0;
|
||||
let blocked = false;
|
||||
if (input.quality_policy) {
|
||||
const gated = this.gate.evaluate(envelopes, input.quality_policy);
|
||||
accepted = gated.accepted;
|
||||
blocked = gated.blocked;
|
||||
violationCount = gated.violations.length;
|
||||
for (const item of gated.quarantined) {
|
||||
const record: QuarantineRecord = {
|
||||
quarantine_id: `qtn_${crypto
|
||||
.createHash("sha256")
|
||||
.update(`${opts.runId}|${item.envelope.envelope_id}`)
|
||||
.digest("hex")
|
||||
.slice(0, 32)}`,
|
||||
source_id: input.source_id,
|
||||
tenant_id: input.tenant_id,
|
||||
run_id: opts.runId,
|
||||
record_key: item.envelope.record_key,
|
||||
violations: item.violations,
|
||||
envelope: item.envelope,
|
||||
quarantined_at: new Date().toISOString(),
|
||||
};
|
||||
await this.repo.quarantine(record);
|
||||
quarantinedCount++;
|
||||
}
|
||||
if (blocked) {
|
||||
return {
|
||||
emitted: 0,
|
||||
duplicates: 0,
|
||||
changed: 0,
|
||||
quarantined: quarantinedCount,
|
||||
violations: violationCount,
|
||||
blocked: true,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
// ---- P068 idempotency ledger, PEEK ONLY ------------------------------
|
||||
// Read the verdicts, filter replays, and commit NOTHING yet: a record is
|
||||
// only "delivered" once the sink has actually taken it. Committing here
|
||||
// would mark records delivered that a crashing sink never received, and
|
||||
// the resumed run would then skip them as duplicates — silent data loss.
|
||||
const entries = accepted.map((e) => ({
|
||||
record_key: e.record_key,
|
||||
payload_hash: hashPayload(e.payload),
|
||||
}));
|
||||
const decisions = await this.repo.peekDeliveries(input.source_id, entries);
|
||||
const verdictByKey = new Map(decisions.map((d) => [d.record_key, d.verdict]));
|
||||
const toEmit = accepted.filter((e) => verdictByKey.get(e.record_key) !== "duplicate");
|
||||
const duplicates = accepted.length - toEmit.length;
|
||||
const changed = decisions.filter((d) => d.verdict === "changed").length;
|
||||
|
||||
// ---- sink (the consumer write) --------------------------------------
|
||||
if (toEmit.length > 0) await input.sink(toEmit);
|
||||
|
||||
// ---- P068 ledger COMMIT, after the sink resolved ---------------------
|
||||
const emitted = new Set(toEmit.map((e) => e.record_key));
|
||||
const committed = entries.filter((e) => emitted.has(e.record_key));
|
||||
if (committed.length > 0) {
|
||||
await this.repo.commitDeliveries(input.source_id, input.tenant_id, committed);
|
||||
}
|
||||
|
||||
return {
|
||||
emitted: toEmit.length,
|
||||
duplicates,
|
||||
changed,
|
||||
quarantined: quarantinedCount,
|
||||
violations: violationCount,
|
||||
blocked: false,
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
/**
|
||||
* src/lineage/lineage-edge.ts
|
||||
* ============================
|
||||
* P067: the flattened, QUERYABLE form of an OpenLineage run event.
|
||||
*
|
||||
* LineageEmitter (src/lineage/lineage-emitter.ts) already emits spec-shaped
|
||||
* run events to AUD0, and AUD0 remains the SYSTEM OF RECORD for them. But
|
||||
* "emitted to AUD0" is not "answerable from 0pt": nothing in this repo could
|
||||
* answer "which sources fed dataset ds_x?" without asking another product.
|
||||
* These edges are 0pt's own projection — one row per (run, input, output)
|
||||
* pair — written in the same call that emits, so the question is answerable
|
||||
* locally and joins back to AUD0 by `run_id`.
|
||||
*
|
||||
* The projection is deliberately LOSSY (facets are not stored, only the
|
||||
* canonical SIAX ref) — it is an index, not a second source of truth.
|
||||
*/
|
||||
|
||||
export interface LineageEdge {
|
||||
/** Deterministic sha256 of (run_id, input, output) — a re-emitted idempotent run does not duplicate edges. */
|
||||
edge_id: string;
|
||||
tenant_id: string;
|
||||
run_id: string;
|
||||
capability_id: string;
|
||||
input_namespace: string;
|
||||
input_name: string;
|
||||
input_ref?: string;
|
||||
output_namespace: string;
|
||||
output_name: string;
|
||||
output_ref?: string;
|
||||
event_time: string;
|
||||
correlation_id?: string;
|
||||
}
|
||||
|
||||
/** Query one direction of the graph. Exactly one of the two selectors is required. */
|
||||
export interface LineageQuery {
|
||||
tenant_id: string;
|
||||
/** Everything that fed this output (upstream). */
|
||||
output_name?: string;
|
||||
/** Everything this input produced (downstream). */
|
||||
input_name?: string;
|
||||
limit?: number;
|
||||
}
|
||||
@@ -0,0 +1,121 @@
|
||||
/**
|
||||
* src/lineage/lineage-store.ts
|
||||
* =============================
|
||||
* P067 DoD: "source -> output lineage is QUERYABLE."
|
||||
*
|
||||
* LineageEmitter already emits OpenLineage run events to AUD0 and that does
|
||||
* not change — AUD0 stays the system of record. This class adds the missing
|
||||
* half: it flattens the same run event into (input, output) edges, persists
|
||||
* them, and answers both directions of the question.
|
||||
*
|
||||
* `RecordingLineageEmitter` is the composition point: it wraps the existing
|
||||
* emitter so every existing caller (QualityService, DatasetVersionService,
|
||||
* the ingestion runner) gets the local projection for free, without any of
|
||||
* them learning about a second store. Emission to AUD0 keeps its existing
|
||||
* never-throw contract; a projection write failure is likewise reported, not
|
||||
* thrown, because lineage is evidence of work that already happened.
|
||||
*/
|
||||
import crypto from "crypto";
|
||||
import type { IngestionRepository } from "../repositories/ingestion-repository";
|
||||
import type { LineageEdge, LineageQuery } from "./lineage-edge";
|
||||
import type { LineageEmitInput, LineageEmitResult, LineageEmitter } from "./lineage-emitter";
|
||||
import type { OpenLineageRunEvent } from "./openlineage";
|
||||
|
||||
/** Deterministic edge id: the same run re-emitted produces the same edges. */
|
||||
export function lineageEdgeId(runId: string, input: string, output: string): string {
|
||||
return `led_${crypto.createHash("sha256").update(`${runId}|${input}|${output}`).digest("hex").slice(0, 32)}`;
|
||||
}
|
||||
|
||||
function refOf(facets: Record<string, unknown> | undefined): string | undefined {
|
||||
const facet = facets?.siaxResourceRef as { ref?: string } | undefined;
|
||||
return typeof facet?.ref === "string" ? facet.ref : undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Flatten one OpenLineage run event into edges. A run with N inputs and M
|
||||
* outputs yields N*M edges; a run with no inputs or no outputs yields none
|
||||
* (there is no edge to draw), which is why the runner always emits both.
|
||||
*/
|
||||
export function edgesFromRunEvent(
|
||||
event: OpenLineageRunEvent,
|
||||
tenantId: string,
|
||||
correlationId?: string,
|
||||
): LineageEdge[] {
|
||||
const edges: LineageEdge[] = [];
|
||||
for (const input of event.inputs ?? []) {
|
||||
for (const output of event.outputs ?? []) {
|
||||
edges.push({
|
||||
edge_id: lineageEdgeId(event.run.runId, `${input.namespace}/${input.name}`, `${output.namespace}/${output.name}`),
|
||||
tenant_id: tenantId,
|
||||
run_id: event.run.runId,
|
||||
capability_id: event.job.name,
|
||||
input_namespace: input.namespace,
|
||||
input_name: input.name,
|
||||
input_ref: refOf(input.facets),
|
||||
output_namespace: output.namespace,
|
||||
output_name: output.name,
|
||||
output_ref: refOf(output.facets),
|
||||
event_time: event.eventTime,
|
||||
correlation_id: correlationId,
|
||||
});
|
||||
}
|
||||
}
|
||||
return edges;
|
||||
}
|
||||
|
||||
export class LineageStore {
|
||||
constructor(private readonly repo: IngestionRepository) {}
|
||||
|
||||
async record(event: OpenLineageRunEvent, tenantId: string, correlationId?: string): Promise<number> {
|
||||
const edges = edgesFromRunEvent(event, tenantId, correlationId);
|
||||
if (edges.length > 0) await this.repo.appendLineageEdges(edges);
|
||||
return edges.length;
|
||||
}
|
||||
|
||||
/** Everything that fed `outputName`, newest first. */
|
||||
async upstreamOf(tenantId: string, outputName: string, limit?: number): Promise<LineageEdge[]> {
|
||||
return this.repo.queryLineage({ tenant_id: tenantId, output_name: outputName, limit });
|
||||
}
|
||||
|
||||
/** Everything `inputName` produced, newest first. */
|
||||
async downstreamOf(tenantId: string, inputName: string, limit?: number): Promise<LineageEdge[]> {
|
||||
return this.repo.queryLineage({ tenant_id: tenantId, input_name: inputName, limit });
|
||||
}
|
||||
|
||||
async query(query: LineageQuery): Promise<LineageEdge[]> {
|
||||
return this.repo.queryLineage(query);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* A LineageEmitter that also writes the local projection. Drop-in for the
|
||||
* existing emitter: same `emit` signature, same never-throw contract.
|
||||
*/
|
||||
export class RecordingLineageEmitter {
|
||||
constructor(
|
||||
private readonly inner: LineageEmitter,
|
||||
private readonly store: LineageStore,
|
||||
) {}
|
||||
|
||||
buildRunEvent(input: LineageEmitInput): OpenLineageRunEvent {
|
||||
return this.inner.buildRunEvent(input);
|
||||
}
|
||||
|
||||
async emit(input: LineageEmitInput): Promise<LineageEmitResult & { projected: number }> {
|
||||
const event = this.inner.buildRunEvent(input);
|
||||
const result = await this.inner.emit(input);
|
||||
let projected = 0;
|
||||
try {
|
||||
projected = await this.store.record(event, input.tenant_id, input.correlation_id);
|
||||
} catch (err) {
|
||||
return {
|
||||
...result,
|
||||
projected: 0,
|
||||
warning: [result.warning, `lineage projection write failed: ${err instanceof Error ? err.message : String(err)}`]
|
||||
.filter(Boolean)
|
||||
.join("; "),
|
||||
};
|
||||
}
|
||||
return { ...result, projected };
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,129 @@
|
||||
/**
|
||||
* src/quality/quality-policy.ts
|
||||
* ==============================
|
||||
* P064: the DECLARED enforcement policy — what happens to a record that fails
|
||||
* its quality rules — and the gate that applies it on the ingestion path.
|
||||
*
|
||||
* WHAT EXISTED BEFORE (and is reused, not rebuilt): QualityService already
|
||||
* owns the rule registry, runs, results, issues and scores, and
|
||||
* ExpectationEngine already executes the Great Expectations core subset. What
|
||||
* neither of them had was a CONSEQUENCE: a failing rule produced an ISSUE for
|
||||
* a human, and the data flowed on regardless. That is the gap this closes.
|
||||
* This module deliberately does NOT re-implement rule storage or execution —
|
||||
* it composes ExpectationEngine and adds the decision.
|
||||
*
|
||||
* Three declared actions, and nothing implicit:
|
||||
* - `block` — the whole page fails; the checkpoint is NOT advanced, so
|
||||
* the run stops on the bad data instead of stepping over it.
|
||||
* - `quarantine` — the failing envelopes are persisted to durable quarantine
|
||||
* and excluded from the output; clean envelopes continue and
|
||||
* the checkpoint advances past all of them.
|
||||
* - `warn` — everything continues; violations are counted and reported.
|
||||
*
|
||||
* There is no default. A source with no policy gets `block` from the runner
|
||||
* only if a policy was supplied; with no policy at all the runner skips the
|
||||
* gate and SAYS SO in the run report (`quality_policy: "none_declared"`) —
|
||||
* an undeclared policy must never look like a passed one.
|
||||
*/
|
||||
import { ExpectationEngine } from "./expectation-engine";
|
||||
import type { QualityExpectation } from "./quality-types";
|
||||
import type { IngestionEnvelope } from "../types/ingestion";
|
||||
|
||||
export type QualityEnforcementAction = "block" | "quarantine" | "warn";
|
||||
|
||||
export interface QualityEnforcementPolicy {
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
/** Rules applied per record. Expectations are evaluated against a one-row table. */
|
||||
expectations: QualityExpectation[];
|
||||
on_violation: QualityEnforcementAction;
|
||||
}
|
||||
|
||||
export interface EnvelopeViolation {
|
||||
expectation_type: string;
|
||||
detail: string;
|
||||
}
|
||||
|
||||
export interface QualityGateResult {
|
||||
/** Envelopes that may proceed downstream. */
|
||||
accepted: IngestionEnvelope[];
|
||||
/** Envelopes routed to quarantine (only under `quarantine`). */
|
||||
quarantined: Array<{ envelope: IngestionEnvelope; violations: EnvelopeViolation[] }>;
|
||||
/** Every violation seen, including under `warn` where nothing was withheld. */
|
||||
violations: Array<{ record_key: string; violations: EnvelopeViolation[] }>;
|
||||
/** True when `block` fired: the caller must NOT advance the checkpoint. */
|
||||
blocked: boolean;
|
||||
action: QualityEnforcementAction;
|
||||
}
|
||||
|
||||
export class QualityGate {
|
||||
constructor(private readonly engine: ExpectationEngine = new ExpectationEngine()) {}
|
||||
|
||||
/**
|
||||
* Evaluate one page of envelopes against the policy.
|
||||
*
|
||||
* Delete envelopes (payload null) are passed through untouched: a CDC delete
|
||||
* carries no columns to check, and failing it against a column expectation
|
||||
* would quarantine the very deletions a source is asking us to honour.
|
||||
*/
|
||||
evaluate(envelopes: IngestionEnvelope[], policy: QualityEnforcementPolicy): QualityGateResult {
|
||||
const accepted: IngestionEnvelope[] = [];
|
||||
const quarantined: QualityGateResult["quarantined"] = [];
|
||||
const allViolations: QualityGateResult["violations"] = [];
|
||||
|
||||
for (const envelope of envelopes) {
|
||||
if (envelope.payload === null) {
|
||||
accepted.push(envelope);
|
||||
continue;
|
||||
}
|
||||
const violations = this.violationsFor(envelope.payload, policy.expectations);
|
||||
if (violations.length === 0) {
|
||||
accepted.push(envelope);
|
||||
continue;
|
||||
}
|
||||
allViolations.push({ record_key: envelope.record_key, violations });
|
||||
if (policy.on_violation === "quarantine") {
|
||||
quarantined.push({ envelope, violations });
|
||||
} else if (policy.on_violation === "warn") {
|
||||
accepted.push(envelope);
|
||||
}
|
||||
// `block`: neither accepted nor quarantined — the caller stops.
|
||||
}
|
||||
|
||||
const blocked = policy.on_violation === "block" && allViolations.length > 0;
|
||||
return {
|
||||
accepted: blocked ? [] : accepted,
|
||||
quarantined,
|
||||
violations: allViolations,
|
||||
blocked,
|
||||
action: policy.on_violation,
|
||||
};
|
||||
}
|
||||
|
||||
private violationsFor(
|
||||
payload: Record<string, unknown>,
|
||||
expectations: QualityExpectation[],
|
||||
): EnvelopeViolation[] {
|
||||
const violations: EnvelopeViolation[] = [];
|
||||
for (const expectation of expectations) {
|
||||
if (!this.engine.supports(expectation.expectation_type)) {
|
||||
// An expectation this engine cannot execute is a VIOLATION of the
|
||||
// policy's executability, not a silent pass. The record is treated as
|
||||
// failing so an unrunnable rule can never look like a satisfied one.
|
||||
violations.push({
|
||||
expectation_type: expectation.expectation_type,
|
||||
detail: `expectation is not executable by the local engine; delegate it to the GE backend before declaring it in an enforcement policy`,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
const outcome = this.engine.execute(expectation, [payload]);
|
||||
if (!outcome.success) {
|
||||
violations.push({
|
||||
expectation_type: expectation.expectation_type,
|
||||
detail: `failed on column "${String(expectation.kwargs.column ?? "<table>")}" (${outcome.unexpected_count}/${outcome.evaluated_count} unexpected)`,
|
||||
});
|
||||
}
|
||||
}
|
||||
return violations;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,120 @@
|
||||
/**
|
||||
* src/repositories/in-memory-ingestion-repository.ts
|
||||
* ====================================================
|
||||
* Non-production implementation of IngestionRepository (hermetic tests, local
|
||||
* dev). Semantics are kept IDENTICAL to the Postgres implementation on every
|
||||
* point the runner depends on:
|
||||
* - checkpoints are upserted per (source, partition),
|
||||
* - ledger verdicts are new/changed/duplicate against the LAST accepted
|
||||
* payload hash for the key,
|
||||
* - quarantine is append-only, newest-first on read,
|
||||
* - schemas are keyed by (source, version) and "latest" means highest version,
|
||||
* - lineage edges are deduplicated by edge_id.
|
||||
*
|
||||
* Any divergence between the two implementations is a bug in one of them;
|
||||
* tests/p0/ingestion-repository-parity.test.ts is what keeps them honest for
|
||||
* the in-memory side, and tests/p0/ingestion-plane-postgres.realinfra.test.ts
|
||||
* runs the same expectations against real Postgres.
|
||||
*/
|
||||
import type { LineageEdge, LineageQuery } from "../lineage/lineage-edge";
|
||||
import type { RecordSchema } from "../schema/record-schema";
|
||||
import type { IngestionCheckpoint, LedgerDecision, QuarantineRecord } from "../types/ingestion";
|
||||
import type { IngestionRepository, LedgerEntry } from "./ingestion-repository";
|
||||
|
||||
function checkpointKey(sourceId: string, partitionKey: string): string {
|
||||
return `${sourceId}|${partitionKey}`;
|
||||
}
|
||||
|
||||
export class InMemoryIngestionRepository implements IngestionRepository {
|
||||
private readonly checkpoints = new Map<string, IngestionCheckpoint>();
|
||||
/** source_id -> record_key -> last accepted payload hash. */
|
||||
private readonly ledger = new Map<string, Map<string, string>>();
|
||||
private readonly quarantined = new Map<string, QuarantineRecord[]>();
|
||||
private readonly schemas = new Map<string, RecordSchema[]>();
|
||||
private readonly lineage = new Map<string, LineageEdge>();
|
||||
|
||||
async getCheckpoint(sourceId: string, partitionKey: string): Promise<IngestionCheckpoint | undefined> {
|
||||
return this.checkpoints.get(checkpointKey(sourceId, partitionKey));
|
||||
}
|
||||
|
||||
async commitCheckpoint(checkpoint: IngestionCheckpoint): Promise<void> {
|
||||
this.checkpoints.set(checkpointKey(checkpoint.source_id, checkpoint.partition_key), { ...checkpoint });
|
||||
}
|
||||
|
||||
async listCheckpoints(sourceId: string): Promise<IngestionCheckpoint[]> {
|
||||
return [...this.checkpoints.values()]
|
||||
.filter((c) => c.source_id === sourceId)
|
||||
.sort((a, b) => a.partition_key.localeCompare(b.partition_key));
|
||||
}
|
||||
|
||||
async peekDeliveries(sourceId: string, entries: LedgerEntry[]): Promise<LedgerDecision[]> {
|
||||
const keys = this.ledger.get(sourceId);
|
||||
return entries.map((entry) => {
|
||||
const prior = keys?.get(entry.record_key);
|
||||
return {
|
||||
record_key: entry.record_key,
|
||||
verdict: prior === undefined ? "new" : prior === entry.payload_hash ? "duplicate" : "changed",
|
||||
} as LedgerDecision;
|
||||
});
|
||||
}
|
||||
|
||||
async commitDeliveries(sourceId: string, _tenantId: string, entries: LedgerEntry[]): Promise<void> {
|
||||
let keys = this.ledger.get(sourceId);
|
||||
if (!keys) {
|
||||
keys = new Map<string, string>();
|
||||
this.ledger.set(sourceId, keys);
|
||||
}
|
||||
for (const entry of entries) keys.set(entry.record_key, entry.payload_hash);
|
||||
}
|
||||
|
||||
async countLedger(sourceId: string): Promise<number> {
|
||||
return this.ledger.get(sourceId)?.size ?? 0;
|
||||
}
|
||||
|
||||
async quarantine(record: QuarantineRecord): Promise<void> {
|
||||
const list = this.quarantined.get(record.source_id) ?? [];
|
||||
list.push(record);
|
||||
this.quarantined.set(record.source_id, list);
|
||||
}
|
||||
|
||||
async listQuarantine(sourceId: string, limit = 100): Promise<QuarantineRecord[]> {
|
||||
const list = this.quarantined.get(sourceId) ?? [];
|
||||
return [...list].reverse().slice(0, limit);
|
||||
}
|
||||
|
||||
async putSchema(schema: RecordSchema): Promise<void> {
|
||||
const list = this.schemas.get(schema.source_id) ?? [];
|
||||
const existing = list.findIndex((s) => s.schema_version === schema.schema_version);
|
||||
if (existing >= 0) list[existing] = { ...schema };
|
||||
else list.push({ ...schema });
|
||||
list.sort((a, b) => a.schema_version - b.schema_version);
|
||||
this.schemas.set(schema.source_id, list);
|
||||
}
|
||||
|
||||
async getSchema(sourceId: string, version: number): Promise<RecordSchema | undefined> {
|
||||
return this.schemas.get(sourceId)?.find((s) => s.schema_version === version);
|
||||
}
|
||||
|
||||
async getLatestSchema(sourceId: string): Promise<RecordSchema | undefined> {
|
||||
const list = this.schemas.get(sourceId);
|
||||
return list && list.length > 0 ? list[list.length - 1] : undefined;
|
||||
}
|
||||
|
||||
async listSchemas(sourceId: string): Promise<RecordSchema[]> {
|
||||
return [...(this.schemas.get(sourceId) ?? [])];
|
||||
}
|
||||
|
||||
async appendLineageEdges(edges: LineageEdge[]): Promise<void> {
|
||||
for (const edge of edges) this.lineage.set(edge.edge_id, { ...edge });
|
||||
}
|
||||
|
||||
async queryLineage(query: LineageQuery): Promise<LineageEdge[]> {
|
||||
const limit = query.limit ?? 200;
|
||||
return [...this.lineage.values()]
|
||||
.filter((e) => e.tenant_id === query.tenant_id)
|
||||
.filter((e) => (query.output_name ? e.output_name === query.output_name : true))
|
||||
.filter((e) => (query.input_name ? e.input_name === query.input_name : true))
|
||||
.sort((a, b) => b.event_time.localeCompare(a.event_time))
|
||||
.slice(0, limit);
|
||||
}
|
||||
}
|
||||
@@ -10,7 +10,13 @@
|
||||
* - 1]` for "most recent" and scanned backward for consecutive failures --
|
||||
* callers now get the same data pre-ordered instead).
|
||||
*/
|
||||
import type { SourceHealthCheck, SourceProvenance, SourceRights } from "../types/source-registry";
|
||||
import type {
|
||||
SourceAuthorization,
|
||||
SourceHealthCheck,
|
||||
SourceProvenance,
|
||||
SourceRights,
|
||||
SourceSchedule,
|
||||
} from "../types/source-registry";
|
||||
import type { SourceRegistryRepository } from "./source-registry-repository";
|
||||
|
||||
export class InMemorySourceRegistryRepository implements SourceRegistryRepository {
|
||||
@@ -18,6 +24,8 @@ export class InMemorySourceRegistryRepository implements SourceRegistryRepositor
|
||||
private readonly provenance = new Map<string, SourceProvenance>();
|
||||
private readonly healthChecks = new Map<string, SourceHealthCheck[]>();
|
||||
private readonly lastSuccess = new Map<string, string>();
|
||||
private readonly authorizations = new Map<string, SourceAuthorization>();
|
||||
private readonly schedules = new Map<string, SourceSchedule>();
|
||||
|
||||
async setRights(rights: SourceRights): Promise<void> {
|
||||
this.rights.set(rights.source_id, rights);
|
||||
@@ -43,6 +51,22 @@ export class InMemorySourceRegistryRepository implements SourceRegistryRepositor
|
||||
return this.lastSuccess.get(sourceId);
|
||||
}
|
||||
|
||||
async setAuthorization(auth: SourceAuthorization): Promise<void> {
|
||||
this.authorizations.set(auth.source_id, auth);
|
||||
}
|
||||
|
||||
async getAuthorization(sourceId: string): Promise<SourceAuthorization | undefined> {
|
||||
return this.authorizations.get(sourceId);
|
||||
}
|
||||
|
||||
async setSchedule(schedule: SourceSchedule): Promise<void> {
|
||||
this.schedules.set(schedule.source_id, schedule);
|
||||
}
|
||||
|
||||
async getSchedule(sourceId: string): Promise<SourceSchedule | undefined> {
|
||||
return this.schedules.get(sourceId);
|
||||
}
|
||||
|
||||
async appendHealthCheck(check: SourceHealthCheck): Promise<void> {
|
||||
const list = this.healthChecks.get(check.source_id) ?? [];
|
||||
list.push(check);
|
||||
|
||||
@@ -27,6 +27,7 @@ interface SourceRecord {
|
||||
created_at: string;
|
||||
updated_at: string | null;
|
||||
created_by: string | null;
|
||||
owner_id: string | null;
|
||||
}
|
||||
|
||||
export class InMemorySourceRepository implements SourceRepository {
|
||||
@@ -50,11 +51,16 @@ export class InMemorySourceRepository implements SourceRepository {
|
||||
adapter_id: input.adapter_id ?? null,
|
||||
access: input.access ?? null,
|
||||
description: input.description ?? null,
|
||||
status: input.status ?? "active",
|
||||
// P060: "pending", not "active". A source with no rights, no provenance
|
||||
// and no authorization used to be live the instant it was registered;
|
||||
// reaching "active" now goes through
|
||||
// SourceRegistryService.activateSource's classification gate.
|
||||
status: input.status ?? "pending",
|
||||
correlation_id: input.correlation_id ?? null,
|
||||
created_at: now,
|
||||
updated_at: null,
|
||||
created_by: input.created_by ?? null,
|
||||
owner_id: input.owner_id ?? null,
|
||||
};
|
||||
this.sources.set(record.source_id, record);
|
||||
return toSource(record);
|
||||
@@ -66,6 +72,14 @@ export class InMemorySourceRepository implements SourceRepository {
|
||||
return toSource(record);
|
||||
}
|
||||
|
||||
async updateStatus(sourceId: string, status: SourceStatus): Promise<Source> {
|
||||
const record = this.sources.get(sourceId);
|
||||
if (!record) throw new Error(`Source ${sourceId} not found`);
|
||||
record.status = status;
|
||||
record.updated_at = new Date().toISOString();
|
||||
return toSource(record);
|
||||
}
|
||||
|
||||
async list(filter?: SourceListFilter): Promise<Source[]> {
|
||||
let results = Array.from(this.sources.values());
|
||||
if (filter?.tenant_id) results = results.filter((r) => r.tenant_id === filter.tenant_id);
|
||||
@@ -91,5 +105,6 @@ function toSource(r: SourceRecord): Source {
|
||||
created_at: r.created_at,
|
||||
updated_at: r.updated_at ?? undefined,
|
||||
created_by: r.created_by ?? undefined,
|
||||
owner_id: r.owner_id ?? undefined,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
/**
|
||||
* src/repositories/ingestion-repository.ts
|
||||
* ==========================================
|
||||
* The storage seam for the P060-P068 ingestion plane: checkpoints (P060/P062),
|
||||
* the idempotency ledger (P068), quarantine (P064), the schema registry
|
||||
* (P060/P063) and the queryable lineage projection (P067).
|
||||
*
|
||||
* WHY ONE INTERFACE AND NOT FIVE. These five tables are written by ONE
|
||||
* caller — IngestionRunner — inside one run, and their correctness is
|
||||
* relative to each other (a checkpoint that advanced past records the ledger
|
||||
* never accepted is a silent gap; a quarantined record whose checkpoint moved
|
||||
* on is a silently dropped record). Splitting them into five factories would
|
||||
* let a future caller wire four of them to Postgres and one to memory and
|
||||
* still typecheck, which is exactly the failure this plane must not have.
|
||||
* They are chosen once, together, by one `resolveStoreMode` call.
|
||||
*
|
||||
* Same production posture as every other 0pt store: `resolveStoreMode` fails
|
||||
* hard in production with no DATABASE_URL rather than silently serving Maps
|
||||
* (src/platform/dependency-mode.ts, Master-DoD §3/§10).
|
||||
*/
|
||||
import { resolveStoreMode } from "../platform/dependency-mode";
|
||||
import type { LineageEdge, LineageQuery } from "../lineage/lineage-edge";
|
||||
import type { RecordSchema } from "../schema/record-schema";
|
||||
import type { IngestionCheckpoint, LedgerDecision, QuarantineRecord } from "../types/ingestion";
|
||||
import { getRepositorySql, repositoryDatabaseUrl } from "./db";
|
||||
import { InMemoryIngestionRepository } from "./in-memory-ingestion-repository";
|
||||
import { PostgresIngestionRepository } from "./postgres-ingestion-repository";
|
||||
|
||||
/** One record offered to the ledger. */
|
||||
export interface LedgerEntry {
|
||||
record_key: string;
|
||||
/** Stable hash of the payload — see hashPayload() in src/ingestion/ingestion-ledger.ts. */
|
||||
payload_hash: string;
|
||||
}
|
||||
|
||||
export interface IngestionRepository {
|
||||
// --- checkpoints (P060 registry field, P062 durable resume) --------------
|
||||
getCheckpoint(sourceId: string, partitionKey: string): Promise<IngestionCheckpoint | undefined>;
|
||||
commitCheckpoint(checkpoint: IngestionCheckpoint): Promise<void>;
|
||||
listCheckpoints(sourceId: string): Promise<IngestionCheckpoint[]>;
|
||||
|
||||
// --- idempotency ledger (P068) ------------------------------------------
|
||||
/**
|
||||
* READ-ONLY verdicts, in input order:
|
||||
* "new" — this record_key has never been accepted for this source,
|
||||
* "changed" — it has, but with different content (a real update),
|
||||
* "duplicate" — it has, with byte-identical content (a replay).
|
||||
*
|
||||
* Deliberately separate from {@link commitDeliveries}. An earlier draft of
|
||||
* this interface combined them into one atomic check-and-record, which was
|
||||
* WRONG in the way that matters: the runner has to consult the ledger
|
||||
* BEFORE the sink (to filter replays) and commit AFTER it (so a crash
|
||||
* between the two re-delivers rather than loses). A combined call marked a
|
||||
* record "delivered" that the sink never received, and the resumed run then
|
||||
* skipped it as a duplicate — permanent, silent data loss. That is not a
|
||||
* hypothetical: tests/p0/ingestion-runner.test.ts caught it, and the
|
||||
* "resumes with NO gap" case is what pins the fix.
|
||||
*/
|
||||
peekDeliveries(sourceId: string, entries: LedgerEntry[]): Promise<LedgerDecision[]>;
|
||||
|
||||
/**
|
||||
* Record that these records were successfully delivered. Called only after
|
||||
* the sink resolved. Idempotent: re-committing the same entry bumps the
|
||||
* delivery count and leaves the verdict unchanged.
|
||||
*/
|
||||
commitDeliveries(sourceId: string, tenantId: string, entries: LedgerEntry[]): Promise<void>;
|
||||
|
||||
countLedger(sourceId: string): Promise<number>;
|
||||
|
||||
// --- quarantine (P064) --------------------------------------------------
|
||||
quarantine(record: QuarantineRecord): Promise<void>;
|
||||
listQuarantine(sourceId: string, limit?: number): Promise<QuarantineRecord[]>;
|
||||
|
||||
// --- schema registry (P060 registry field, P063 evolution) --------------
|
||||
putSchema(schema: RecordSchema): Promise<void>;
|
||||
getSchema(sourceId: string, version: number): Promise<RecordSchema | undefined>;
|
||||
getLatestSchema(sourceId: string): Promise<RecordSchema | undefined>;
|
||||
listSchemas(sourceId: string): Promise<RecordSchema[]>;
|
||||
|
||||
// --- lineage projection (P067) ------------------------------------------
|
||||
appendLineageEdges(edges: LineageEdge[]): Promise<void>;
|
||||
queryLineage(query: LineageQuery): Promise<LineageEdge[]>;
|
||||
}
|
||||
|
||||
export function createIngestionRepository(databaseUrl?: string): IngestionRepository {
|
||||
const mode = resolveStoreMode("0pt.ingestion", databaseUrl ?? repositoryDatabaseUrl());
|
||||
if (mode.mode === "postgres") {
|
||||
return new PostgresIngestionRepository(getRepositorySql(mode.url));
|
||||
}
|
||||
return new InMemoryIngestionRepository();
|
||||
}
|
||||
@@ -0,0 +1,308 @@
|
||||
/**
|
||||
* src/repositories/postgres-ingestion-repository.ts
|
||||
* ==================================================
|
||||
* Real Postgres backing for IngestionRepository (sql/007_ingestion_plane.sql).
|
||||
*
|
||||
* THE LEDGER IS THE INTERESTING ONE (P068), and it is deliberately SPLIT:
|
||||
* `peekDeliveries` reads verdicts, `commitDeliveries` records them. The
|
||||
* runner peeks before the sink and commits after it, so a crash in between
|
||||
* re-delivers on resume instead of losing the record. An earlier version of
|
||||
* this file combined the two into one atomic check-and-record statement; that
|
||||
* was measurably wrong (see the interface's doc comment) and the crash/resume
|
||||
* test in tests/p0/ingestion-runner.test.ts is what proved it.
|
||||
*
|
||||
* KNOWN LIMIT, stated rather than papered over: the split means two runs on
|
||||
* the SAME (source_id, partition) can both peek "new" and both emit. The
|
||||
* delivery guarantee is therefore AT-LEAST-ONCE, which is what the
|
||||
* deterministic `envelope_id` (src/ingestion/envelope.ts) exists to make
|
||||
* survivable downstream. Exactly-once across concurrent runs needs a run lock
|
||||
* per (source, partition); there is none yet. That is a real open gap, not a
|
||||
* solved one — do not describe this plane as exactly-once.
|
||||
*/
|
||||
import type { LineageEdge, LineageQuery } from "../lineage/lineage-edge";
|
||||
import type {
|
||||
CompatibilityPolicy,
|
||||
RecordSchema,
|
||||
RecordSchemaFields,
|
||||
} from "../schema/record-schema";
|
||||
import type {
|
||||
IngestionCheckpoint,
|
||||
IngestionEnvelope,
|
||||
LedgerDecision,
|
||||
QuarantineRecord,
|
||||
} from "../types/ingestion";
|
||||
import type { RepositorySql } from "./db";
|
||||
import type { IngestionRepository, LedgerEntry } from "./ingestion-repository";
|
||||
|
||||
/** postgres.js's `sql.json()` parameter type, named locally so the two JSONB columns can be passed without importing postgres' internals. */
|
||||
type JsonValue = Parameters<RepositorySql["json"]>[0];
|
||||
|
||||
function toIso(v: string | Date): string {
|
||||
return v instanceof Date ? v.toISOString() : v;
|
||||
}
|
||||
|
||||
interface CheckpointRow {
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
partition_key: string;
|
||||
cursor: string | null;
|
||||
sequence: string | number;
|
||||
records_committed: string | number;
|
||||
committed_at: string | Date;
|
||||
}
|
||||
|
||||
function toCheckpoint(r: CheckpointRow): IngestionCheckpoint {
|
||||
return {
|
||||
source_id: r.source_id,
|
||||
tenant_id: r.tenant_id,
|
||||
partition_key: r.partition_key,
|
||||
cursor: r.cursor ?? undefined,
|
||||
sequence: Number(r.sequence),
|
||||
records_committed: Number(r.records_committed),
|
||||
committed_at: toIso(r.committed_at),
|
||||
};
|
||||
}
|
||||
|
||||
export class PostgresIngestionRepository implements IngestionRepository {
|
||||
constructor(private readonly sql: RepositorySql) {}
|
||||
|
||||
// --- checkpoints ---------------------------------------------------------
|
||||
|
||||
async getCheckpoint(sourceId: string, partitionKey: string): Promise<IngestionCheckpoint | undefined> {
|
||||
const rows = await this.sql<CheckpointRow[]>`
|
||||
SELECT source_id, tenant_id, partition_key, cursor, sequence, records_committed, committed_at
|
||||
FROM opt_ingestion_checkpoints
|
||||
WHERE source_id = ${sourceId} AND partition_key = ${partitionKey}
|
||||
LIMIT 1`;
|
||||
return rows[0] ? toCheckpoint(rows[0]) : undefined;
|
||||
}
|
||||
|
||||
async commitCheckpoint(cp: IngestionCheckpoint): Promise<void> {
|
||||
await this.sql`
|
||||
INSERT INTO opt_ingestion_checkpoints
|
||||
(source_id, partition_key, tenant_id, cursor, sequence, records_committed, committed_at)
|
||||
VALUES
|
||||
(${cp.source_id}, ${cp.partition_key}, ${cp.tenant_id}, ${cp.cursor ?? null},
|
||||
${cp.sequence}, ${cp.records_committed}, ${cp.committed_at})
|
||||
ON CONFLICT (source_id, partition_key) DO UPDATE
|
||||
SET cursor = EXCLUDED.cursor,
|
||||
sequence = EXCLUDED.sequence,
|
||||
records_committed = EXCLUDED.records_committed,
|
||||
committed_at = EXCLUDED.committed_at,
|
||||
tenant_id = EXCLUDED.tenant_id`;
|
||||
}
|
||||
|
||||
async listCheckpoints(sourceId: string): Promise<IngestionCheckpoint[]> {
|
||||
const rows = await this.sql<CheckpointRow[]>`
|
||||
SELECT source_id, tenant_id, partition_key, cursor, sequence, records_committed, committed_at
|
||||
FROM opt_ingestion_checkpoints
|
||||
WHERE source_id = ${sourceId}
|
||||
ORDER BY partition_key ASC`;
|
||||
return rows.map(toCheckpoint);
|
||||
}
|
||||
|
||||
// --- ledger --------------------------------------------------------------
|
||||
|
||||
async peekDeliveries(sourceId: string, entries: LedgerEntry[]): Promise<LedgerDecision[]> {
|
||||
if (entries.length === 0) return [];
|
||||
const keys = entries.map((e) => e.record_key);
|
||||
const rows = await this.sql<{ record_key: string; payload_hash: string }[]>`
|
||||
SELECT record_key, payload_hash FROM opt_ingestion_ledger
|
||||
WHERE source_id = ${sourceId} AND record_key = ANY(${this.sql.array(keys)})`;
|
||||
const priorByKey = new Map(rows.map((r) => [r.record_key, r.payload_hash]));
|
||||
return entries.map((entry) => {
|
||||
const prior = priorByKey.get(entry.record_key);
|
||||
return {
|
||||
record_key: entry.record_key,
|
||||
verdict: prior === undefined ? "new" : prior === entry.payload_hash ? "duplicate" : "changed",
|
||||
} as LedgerDecision;
|
||||
});
|
||||
}
|
||||
|
||||
async commitDeliveries(sourceId: string, tenantId: string, entries: LedgerEntry[]): Promise<void> {
|
||||
for (const entry of entries) {
|
||||
await this.sql`
|
||||
INSERT INTO opt_ingestion_ledger (source_id, record_key, tenant_id, payload_hash)
|
||||
VALUES (${sourceId}, ${entry.record_key}, ${tenantId}, ${entry.payload_hash})
|
||||
ON CONFLICT (source_id, record_key) DO UPDATE
|
||||
SET last_seen_at = now(),
|
||||
delivery_count = opt_ingestion_ledger.delivery_count + 1,
|
||||
payload_hash = EXCLUDED.payload_hash`;
|
||||
}
|
||||
}
|
||||
|
||||
async countLedger(sourceId: string): Promise<number> {
|
||||
const rows = await this.sql<{ n: string }[]>`
|
||||
SELECT count(*)::text AS n FROM opt_ingestion_ledger WHERE source_id = ${sourceId}`;
|
||||
return Number(rows[0]?.n ?? 0);
|
||||
}
|
||||
|
||||
// --- quarantine ----------------------------------------------------------
|
||||
|
||||
async quarantine(record: QuarantineRecord): Promise<void> {
|
||||
await this.sql`
|
||||
INSERT INTO opt_quality_quarantine
|
||||
(quarantine_id, source_id, tenant_id, run_id, record_key, violations, envelope, quarantined_at)
|
||||
VALUES
|
||||
(${record.quarantine_id}, ${record.source_id}, ${record.tenant_id}, ${record.run_id},
|
||||
${record.record_key}, ${this.sql.json(record.violations as unknown as JsonValue)},
|
||||
${this.sql.json(record.envelope as unknown as JsonValue)},
|
||||
${record.quarantined_at})
|
||||
ON CONFLICT (quarantine_id) DO NOTHING`;
|
||||
}
|
||||
|
||||
async listQuarantine(sourceId: string, limit = 100): Promise<QuarantineRecord[]> {
|
||||
const rows = await this.sql<
|
||||
{
|
||||
quarantine_id: string;
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
run_id: string;
|
||||
record_key: string;
|
||||
violations: QuarantineRecord["violations"];
|
||||
envelope: IngestionEnvelope;
|
||||
quarantined_at: string | Date;
|
||||
}[]
|
||||
>`
|
||||
SELECT quarantine_id, source_id, tenant_id, run_id, record_key, violations, envelope, quarantined_at
|
||||
FROM opt_quality_quarantine
|
||||
WHERE source_id = ${sourceId}
|
||||
ORDER BY quarantined_at DESC, quarantine_id DESC
|
||||
LIMIT ${limit}`;
|
||||
return rows.map((r) => ({
|
||||
quarantine_id: r.quarantine_id,
|
||||
source_id: r.source_id,
|
||||
tenant_id: r.tenant_id,
|
||||
run_id: r.run_id,
|
||||
record_key: r.record_key,
|
||||
violations: r.violations,
|
||||
envelope: r.envelope,
|
||||
quarantined_at: toIso(r.quarantined_at),
|
||||
}));
|
||||
}
|
||||
|
||||
// --- schema registry -----------------------------------------------------
|
||||
|
||||
async putSchema(schema: RecordSchema): Promise<void> {
|
||||
await this.sql`
|
||||
INSERT INTO opt_source_schemas
|
||||
(source_id, schema_version, tenant_id, fields, compatibility, registered_at, registered_by)
|
||||
VALUES
|
||||
(${schema.source_id}, ${schema.schema_version}, ${schema.tenant_id},
|
||||
${this.sql.json(schema.fields as unknown as JsonValue)}, ${schema.compatibility},
|
||||
${schema.registered_at}, ${schema.registered_by ?? null})
|
||||
ON CONFLICT (source_id, schema_version) DO UPDATE
|
||||
SET fields = EXCLUDED.fields,
|
||||
compatibility = EXCLUDED.compatibility,
|
||||
registered_at = EXCLUDED.registered_at,
|
||||
registered_by = EXCLUDED.registered_by`;
|
||||
}
|
||||
|
||||
async getSchema(sourceId: string, version: number): Promise<RecordSchema | undefined> {
|
||||
const rows = await this.sql<SchemaRow[]>`
|
||||
SELECT source_id, schema_version, tenant_id, fields, compatibility, registered_at, registered_by
|
||||
FROM opt_source_schemas
|
||||
WHERE source_id = ${sourceId} AND schema_version = ${version}
|
||||
LIMIT 1`;
|
||||
return rows[0] ? toSchema(rows[0]) : undefined;
|
||||
}
|
||||
|
||||
async getLatestSchema(sourceId: string): Promise<RecordSchema | undefined> {
|
||||
const rows = await this.sql<SchemaRow[]>`
|
||||
SELECT source_id, schema_version, tenant_id, fields, compatibility, registered_at, registered_by
|
||||
FROM opt_source_schemas
|
||||
WHERE source_id = ${sourceId}
|
||||
ORDER BY schema_version DESC
|
||||
LIMIT 1`;
|
||||
return rows[0] ? toSchema(rows[0]) : undefined;
|
||||
}
|
||||
|
||||
async listSchemas(sourceId: string): Promise<RecordSchema[]> {
|
||||
const rows = await this.sql<SchemaRow[]>`
|
||||
SELECT source_id, schema_version, tenant_id, fields, compatibility, registered_at, registered_by
|
||||
FROM opt_source_schemas
|
||||
WHERE source_id = ${sourceId}
|
||||
ORDER BY schema_version ASC`;
|
||||
return rows.map(toSchema);
|
||||
}
|
||||
|
||||
// --- lineage -------------------------------------------------------------
|
||||
|
||||
async appendLineageEdges(edges: LineageEdge[]): Promise<void> {
|
||||
for (const e of edges) {
|
||||
await this.sql`
|
||||
INSERT INTO opt_lineage_edges
|
||||
(edge_id, tenant_id, run_id, capability_id, input_namespace, input_name, input_ref,
|
||||
output_namespace, output_name, output_ref, event_time, correlation_id)
|
||||
VALUES
|
||||
(${e.edge_id}, ${e.tenant_id}, ${e.run_id}, ${e.capability_id}, ${e.input_namespace},
|
||||
${e.input_name}, ${e.input_ref ?? null}, ${e.output_namespace}, ${e.output_name},
|
||||
${e.output_ref ?? null}, ${e.event_time}, ${e.correlation_id ?? null})
|
||||
ON CONFLICT (edge_id) DO NOTHING`;
|
||||
}
|
||||
}
|
||||
|
||||
async queryLineage(query: LineageQuery): Promise<LineageEdge[]> {
|
||||
const limit = query.limit ?? 200;
|
||||
const rows = await this.sql<LineageRow[]>`
|
||||
SELECT edge_id, tenant_id, run_id, capability_id, input_namespace, input_name, input_ref,
|
||||
output_namespace, output_name, output_ref, event_time, correlation_id
|
||||
FROM opt_lineage_edges
|
||||
WHERE tenant_id = ${query.tenant_id}
|
||||
${query.output_name ? this.sql`AND output_name = ${query.output_name}` : this.sql``}
|
||||
${query.input_name ? this.sql`AND input_name = ${query.input_name}` : this.sql``}
|
||||
ORDER BY event_time DESC, edge_id DESC
|
||||
LIMIT ${limit}`;
|
||||
return rows.map((r) => ({
|
||||
edge_id: r.edge_id,
|
||||
tenant_id: r.tenant_id,
|
||||
run_id: r.run_id,
|
||||
capability_id: r.capability_id,
|
||||
input_namespace: r.input_namespace,
|
||||
input_name: r.input_name,
|
||||
input_ref: r.input_ref ?? undefined,
|
||||
output_namespace: r.output_namespace,
|
||||
output_name: r.output_name,
|
||||
output_ref: r.output_ref ?? undefined,
|
||||
event_time: toIso(r.event_time),
|
||||
correlation_id: r.correlation_id ?? undefined,
|
||||
}));
|
||||
}
|
||||
}
|
||||
|
||||
interface SchemaRow {
|
||||
source_id: string;
|
||||
schema_version: number;
|
||||
tenant_id: string;
|
||||
fields: RecordSchemaFields;
|
||||
compatibility: string;
|
||||
registered_at: string | Date;
|
||||
registered_by: string | null;
|
||||
}
|
||||
|
||||
function toSchema(r: SchemaRow): RecordSchema {
|
||||
return {
|
||||
source_id: r.source_id,
|
||||
schema_version: Number(r.schema_version),
|
||||
tenant_id: r.tenant_id,
|
||||
fields: r.fields,
|
||||
compatibility: r.compatibility as CompatibilityPolicy,
|
||||
registered_at: toIso(r.registered_at),
|
||||
registered_by: r.registered_by ?? undefined,
|
||||
};
|
||||
}
|
||||
|
||||
interface LineageRow {
|
||||
edge_id: string;
|
||||
tenant_id: string;
|
||||
run_id: string;
|
||||
capability_id: string;
|
||||
input_namespace: string;
|
||||
input_name: string;
|
||||
input_ref: string | null;
|
||||
output_namespace: string;
|
||||
output_name: string;
|
||||
output_ref: string | null;
|
||||
event_time: string | Date;
|
||||
correlation_id: string | null;
|
||||
}
|
||||
@@ -10,10 +10,13 @@
|
||||
* is a single upserted "last success" row per source.
|
||||
*/
|
||||
import type {
|
||||
AuthorizationBasis,
|
||||
SourceAuthorization,
|
||||
SourceHealthCheck,
|
||||
SourceHealthStatus,
|
||||
SourceProvenance,
|
||||
SourceRights,
|
||||
SourceSchedule,
|
||||
} from "../types/source-registry";
|
||||
import type { RepositorySql } from "./db";
|
||||
import type { SourceRegistryRepository } from "./source-registry-repository";
|
||||
@@ -167,6 +170,57 @@ export class PostgresSourceRegistryRepository implements SourceRegistryRepositor
|
||||
return rows[0] ? toIsoString(rows[0].last_success_at) : undefined;
|
||||
}
|
||||
|
||||
// --- P060 authorization / schedule ---------------------------------------
|
||||
|
||||
async setAuthorization(auth: SourceAuthorization): Promise<void> {
|
||||
await this.sql`
|
||||
INSERT INTO opt_source_authorization
|
||||
(source_id, tenant_id, basis, granted_by, granted_at, expires_at, evidence_ref, notes)
|
||||
VALUES
|
||||
(${auth.source_id}, ${auth.tenant_id}, ${auth.basis}, ${auth.granted_by}, ${auth.granted_at},
|
||||
${auth.expires_at ?? null}, ${auth.evidence_ref ?? null}, ${auth.notes ?? null})
|
||||
ON CONFLICT (source_id) DO UPDATE
|
||||
SET tenant_id = EXCLUDED.tenant_id,
|
||||
basis = EXCLUDED.basis,
|
||||
granted_by = EXCLUDED.granted_by,
|
||||
granted_at = EXCLUDED.granted_at,
|
||||
expires_at = EXCLUDED.expires_at,
|
||||
evidence_ref = EXCLUDED.evidence_ref,
|
||||
notes = EXCLUDED.notes,
|
||||
updated_at = now()`;
|
||||
}
|
||||
|
||||
async getAuthorization(sourceId: string): Promise<SourceAuthorization | undefined> {
|
||||
const rows = await this.sql<AuthorizationRow[]>`
|
||||
SELECT source_id, tenant_id, basis, granted_by, granted_at, expires_at, evidence_ref, notes
|
||||
FROM opt_source_authorization WHERE source_id = ${sourceId} LIMIT 1`;
|
||||
return rows[0] ? toAuthorization(rows[0]) : undefined;
|
||||
}
|
||||
|
||||
async setSchedule(schedule: SourceSchedule): Promise<void> {
|
||||
await this.sql`
|
||||
INSERT INTO opt_source_schedule
|
||||
(source_id, tenant_id, mode, interval_seconds, cron, enabled, updated_at)
|
||||
VALUES
|
||||
(${schedule.source_id}, ${schedule.tenant_id}, ${schedule.mode},
|
||||
${schedule.interval_seconds ?? null}, ${schedule.cron ?? null}, ${schedule.enabled},
|
||||
${schedule.updated_at})
|
||||
ON CONFLICT (source_id) DO UPDATE
|
||||
SET tenant_id = EXCLUDED.tenant_id,
|
||||
mode = EXCLUDED.mode,
|
||||
interval_seconds = EXCLUDED.interval_seconds,
|
||||
cron = EXCLUDED.cron,
|
||||
enabled = EXCLUDED.enabled,
|
||||
updated_at = EXCLUDED.updated_at`;
|
||||
}
|
||||
|
||||
async getSchedule(sourceId: string): Promise<SourceSchedule | undefined> {
|
||||
const rows = await this.sql<ScheduleRow[]>`
|
||||
SELECT source_id, tenant_id, mode, interval_seconds, cron, enabled, updated_at
|
||||
FROM opt_source_schedule WHERE source_id = ${sourceId} LIMIT 1`;
|
||||
return rows[0] ? toSchedule(rows[0]) : undefined;
|
||||
}
|
||||
|
||||
async appendHealthCheck(check: SourceHealthCheck): Promise<void> {
|
||||
await this.sql`
|
||||
INSERT INTO opt_source_health_checks (source_id, tenant_id, status, checked_at, latency_ms, detail)
|
||||
@@ -195,3 +249,49 @@ export class PostgresSourceRegistryRepository implements SourceRegistryRepositor
|
||||
return rows.map(toHealthCheck);
|
||||
}
|
||||
}
|
||||
|
||||
interface AuthorizationRow {
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
basis: string;
|
||||
granted_by: string;
|
||||
granted_at: string | Date;
|
||||
expires_at: string | Date | null;
|
||||
evidence_ref: string | null;
|
||||
notes: string | null;
|
||||
}
|
||||
|
||||
function toAuthorization(r: AuthorizationRow): SourceAuthorization {
|
||||
return {
|
||||
source_id: r.source_id,
|
||||
tenant_id: r.tenant_id,
|
||||
basis: r.basis as AuthorizationBasis,
|
||||
granted_by: r.granted_by,
|
||||
granted_at: toIsoString(r.granted_at),
|
||||
expires_at: r.expires_at ? toIsoString(r.expires_at) : undefined,
|
||||
evidence_ref: r.evidence_ref ?? undefined,
|
||||
notes: r.notes ?? undefined,
|
||||
};
|
||||
}
|
||||
|
||||
interface ScheduleRow {
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
mode: string;
|
||||
interval_seconds: number | string | null;
|
||||
cron: string | null;
|
||||
enabled: boolean;
|
||||
updated_at: string | Date;
|
||||
}
|
||||
|
||||
function toSchedule(r: ScheduleRow): SourceSchedule {
|
||||
return {
|
||||
source_id: r.source_id,
|
||||
tenant_id: r.tenant_id,
|
||||
mode: r.mode as SourceSchedule["mode"],
|
||||
interval_seconds: r.interval_seconds === null ? undefined : Number(r.interval_seconds),
|
||||
cron: r.cron ?? undefined,
|
||||
enabled: r.enabled,
|
||||
updated_at: toIsoString(r.updated_at),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -25,6 +25,7 @@ interface SourceRow {
|
||||
created_at: string | Date;
|
||||
updated_at: string | Date | null;
|
||||
created_by: string | null;
|
||||
owner_id: string | null;
|
||||
}
|
||||
|
||||
function toIsoString(v: string | Date): string {
|
||||
@@ -46,6 +47,7 @@ function toSource(r: SourceRow): Source {
|
||||
created_at: toIsoString(r.created_at),
|
||||
updated_at: r.updated_at ? toIsoString(r.updated_at) : undefined,
|
||||
created_by: r.created_by ?? undefined,
|
||||
owner_id: r.owner_id ?? undefined,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -63,29 +65,41 @@ export class PostgresSourceRepository implements SourceRepository {
|
||||
|
||||
const rows = await this.sql<SourceRow[]>`
|
||||
INSERT INTO opt_sources
|
||||
(source_id, tenant_id, project_id, source_type, name, adapter_id, access, description, status, correlation_id, created_by)
|
||||
(source_id, tenant_id, project_id, source_type, name, adapter_id, access, description, status, correlation_id, created_by, owner_id)
|
||||
VALUES
|
||||
(${input.source_id}, ${input.tenant_id}, ${input.project_id}, ${input.source_type}, ${input.name},
|
||||
${input.adapter_id ?? null}, ${input.access ?? null}, ${input.description ?? null},
|
||||
${input.status ?? "active"}, ${input.correlation_id ?? null}, ${input.created_by ?? null})
|
||||
${input.status ?? "pending"}, ${input.correlation_id ?? null}, ${input.created_by ?? null},
|
||||
${input.owner_id ?? null})
|
||||
RETURNING source_id, tenant_id, project_id, source_type, name, adapter_id, access, description,
|
||||
status, correlation_id, created_at, updated_at, created_by`;
|
||||
status, correlation_id, created_at, updated_at, created_by, owner_id`;
|
||||
return toSource(rows[0]);
|
||||
}
|
||||
|
||||
async get(sourceId: string): Promise<Source> {
|
||||
const rows = await this.sql<SourceRow[]>`
|
||||
SELECT source_id, tenant_id, project_id, source_type, name, adapter_id, access, description,
|
||||
status, correlation_id, created_at, updated_at, created_by
|
||||
status, correlation_id, created_at, updated_at, created_by, owner_id
|
||||
FROM opt_sources WHERE source_id = ${sourceId} LIMIT 1`;
|
||||
if (!rows[0]) throw new Error(`Source ${sourceId} not found`);
|
||||
return toSource(rows[0]);
|
||||
}
|
||||
|
||||
async updateStatus(sourceId: string, status: SourceStatus): Promise<Source> {
|
||||
const rows = await this.sql<SourceRow[]>`
|
||||
UPDATE opt_sources
|
||||
SET status = ${status}, updated_at = now()
|
||||
WHERE source_id = ${sourceId}
|
||||
RETURNING source_id, tenant_id, project_id, source_type, name, adapter_id, access, description,
|
||||
status, correlation_id, created_at, updated_at, created_by, owner_id`;
|
||||
if (!rows[0]) throw new Error(`Source ${sourceId} not found`);
|
||||
return toSource(rows[0]);
|
||||
}
|
||||
|
||||
async list(filter?: SourceListFilter): Promise<Source[]> {
|
||||
const rows = await this.sql<SourceRow[]>`
|
||||
SELECT source_id, tenant_id, project_id, source_type, name, adapter_id, access, description,
|
||||
status, correlation_id, created_at, updated_at, created_by
|
||||
status, correlation_id, created_at, updated_at, created_by, owner_id
|
||||
FROM opt_sources
|
||||
WHERE 1=1
|
||||
${filter?.tenant_id ? this.sql`AND tenant_id = ${filter.tenant_id}` : this.sql``}
|
||||
|
||||
@@ -16,7 +16,13 @@
|
||||
* layer trusts the caller, same as the Maps did.
|
||||
*/
|
||||
import { resolveStoreMode } from "../platform/dependency-mode";
|
||||
import type { SourceHealthCheck, SourceProvenance, SourceRights } from "../types/source-registry";
|
||||
import type {
|
||||
SourceAuthorization,
|
||||
SourceHealthCheck,
|
||||
SourceProvenance,
|
||||
SourceRights,
|
||||
SourceSchedule,
|
||||
} from "../types/source-registry";
|
||||
import { getRepositorySql, repositoryDatabaseUrl } from "./db";
|
||||
import { InMemorySourceRegistryRepository } from "./in-memory-source-registry-repository";
|
||||
import { PostgresSourceRegistryRepository } from "./postgres-source-registry-repository";
|
||||
@@ -31,6 +37,14 @@ export interface SourceRegistryRepository {
|
||||
recordSuccess(sourceId: string, tenantId: string, atIso: string): Promise<void>;
|
||||
getLastSuccess(sourceId: string): Promise<string | undefined>;
|
||||
|
||||
/** P060 authorization — the recorded decision that 0pt may connect at all. */
|
||||
setAuthorization(auth: SourceAuthorization): Promise<void>;
|
||||
getAuthorization(sourceId: string): Promise<SourceAuthorization | undefined>;
|
||||
|
||||
/** P060 schedule — ingestion mode + cadence. */
|
||||
setSchedule(schedule: SourceSchedule): Promise<void>;
|
||||
getSchedule(sourceId: string): Promise<SourceSchedule | undefined>;
|
||||
|
||||
appendHealthCheck(check: SourceHealthCheck): Promise<void>;
|
||||
/** Newest-first, bounded to the latest `limit` observations (default 100) -- health is an operational signal, not an audit log; AUD0 owns history. */
|
||||
listHealthChecks(sourceId: string, limit?: number): Promise<SourceHealthCheck[]>;
|
||||
|
||||
@@ -33,6 +33,12 @@ export interface SourceRepository {
|
||||
/** Throws (message contains "not found") when no such source exists -- matches the pre-migration Map behavior every caller already handles. */
|
||||
get(sourceId: string): Promise<Source>;
|
||||
list(filter?: SourceListFilter): Promise<Source[]>;
|
||||
/**
|
||||
* P060: the only way a source's lifecycle status changes. Deliberately NOT
|
||||
* a general `update` -- status is the field the activation gate protects,
|
||||
* and a generic patch method would be the obvious way around it.
|
||||
*/
|
||||
updateStatus(sourceId: string, status: SourceStatus): Promise<Source>;
|
||||
}
|
||||
|
||||
export function createSourceRepository(databaseUrl?: string): SourceRepository {
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
/**
|
||||
* src/schema/compatibility.ts
|
||||
* ============================
|
||||
* P063: the pure comparison. `checkCompatibility(previous, next, policy)`
|
||||
* returns every way `next` breaks readers/writers of `previous` under the
|
||||
* declared policy. No IO, no throwing — the caller decides what a break
|
||||
* means (SchemaRegistry refuses the registration; the ingestion runner
|
||||
* refuses the RUN, before a single envelope reaches a consumer).
|
||||
*/
|
||||
import type {
|
||||
CompatibilityPolicy,
|
||||
CompatibilityVerdict,
|
||||
RecordSchemaFields,
|
||||
SchemaBreak,
|
||||
} from "./record-schema";
|
||||
|
||||
/**
|
||||
* BACKWARD breaks: a consumer written against `next` reading data produced
|
||||
* under `previous` fails when
|
||||
* - `next` adds a REQUIRED field `previous` never produced, or
|
||||
* - `next` changes a field's type (old data has the old type), or
|
||||
* - `next` drops a field that was REQUIRED in `previous` and is therefore
|
||||
* part of the contract consumers rely on.
|
||||
*/
|
||||
function backwardBreaks(previous: RecordSchemaFields, next: RecordSchemaFields): SchemaBreak[] {
|
||||
const breaks: SchemaBreak[] = [];
|
||||
for (const [field, spec] of Object.entries(next)) {
|
||||
const before = previous[field];
|
||||
if (!before) {
|
||||
if (spec.required) {
|
||||
breaks.push({
|
||||
field,
|
||||
breaks: "backward",
|
||||
detail: `new required field "${field}" is absent from every record written under the previous version`,
|
||||
});
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (before.type !== spec.type) {
|
||||
breaks.push({
|
||||
field,
|
||||
breaks: "backward",
|
||||
detail: `field "${field}" changed type ${before.type} -> ${spec.type}; existing data still carries ${before.type}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
for (const [field, before] of Object.entries(previous)) {
|
||||
if (!next[field] && before.required) {
|
||||
breaks.push({
|
||||
field,
|
||||
breaks: "backward",
|
||||
detail: `required field "${field}" was removed; consumers of the previous version still depend on it`,
|
||||
});
|
||||
}
|
||||
}
|
||||
return breaks;
|
||||
}
|
||||
|
||||
/**
|
||||
* FORWARD breaks: a consumer written against `previous` reading data produced
|
||||
* under `next` fails when
|
||||
* - `next` removes a field `previous` marked REQUIRED, or
|
||||
* - `next` changes a field's type.
|
||||
* Adding a required field is forward-SAFE (old readers ignore it).
|
||||
*/
|
||||
function forwardBreaks(previous: RecordSchemaFields, next: RecordSchemaFields): SchemaBreak[] {
|
||||
const breaks: SchemaBreak[] = [];
|
||||
for (const [field, before] of Object.entries(previous)) {
|
||||
const after = next[field];
|
||||
if (!after) {
|
||||
if (before.required) {
|
||||
breaks.push({
|
||||
field,
|
||||
breaks: "forward",
|
||||
detail: `required field "${field}" was removed; readers pinned to the previous version will not find it`,
|
||||
});
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (before.type !== after.type) {
|
||||
breaks.push({
|
||||
field,
|
||||
breaks: "forward",
|
||||
detail: `field "${field}" changed type ${before.type} -> ${after.type}; readers pinned to the previous version expect ${before.type}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
return breaks;
|
||||
}
|
||||
|
||||
/** Deduplicate identical (field, direction, detail) triples produced by both directions under `full`. */
|
||||
function dedupe(breaks: SchemaBreak[]): SchemaBreak[] {
|
||||
const seen = new Set<string>();
|
||||
const out: SchemaBreak[] = [];
|
||||
for (const b of breaks) {
|
||||
const key = `${b.field}|${b.breaks}|${b.detail}`;
|
||||
if (seen.has(key)) continue;
|
||||
seen.add(key);
|
||||
out.push(b);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
export function checkCompatibility(
|
||||
previous: RecordSchemaFields,
|
||||
next: RecordSchemaFields,
|
||||
policy: CompatibilityPolicy,
|
||||
): CompatibilityVerdict {
|
||||
if (policy === "none") {
|
||||
return { compatible: true, policy, breaks: [] };
|
||||
}
|
||||
const breaks: SchemaBreak[] = [];
|
||||
if (policy === "backward" || policy === "full") breaks.push(...backwardBreaks(previous, next));
|
||||
if (policy === "forward" || policy === "full") breaks.push(...forwardBreaks(previous, next));
|
||||
const deduped = dedupe(breaks);
|
||||
return { compatible: deduped.length === 0, policy, breaks: deduped };
|
||||
}
|
||||
|
||||
/**
|
||||
* Does a concrete record conform to a schema? Used by the ingestion runner to
|
||||
* detect that a source has SILENTLY changed shape — i.e. it is emitting data
|
||||
* the registered schema does not describe — which is the case P063 exists to
|
||||
* catch before consumers see it. Returns the problems; empty = conforming.
|
||||
*/
|
||||
export function conformanceErrors(
|
||||
fields: RecordSchemaFields,
|
||||
payload: Record<string, unknown>,
|
||||
): string[] {
|
||||
const errors: string[] = [];
|
||||
for (const [field, spec] of Object.entries(fields)) {
|
||||
const value = payload[field];
|
||||
if (value === undefined || value === null) {
|
||||
if (spec.required) errors.push(`required field "${field}" is missing`);
|
||||
continue;
|
||||
}
|
||||
if (!matchesType(value, spec.type)) {
|
||||
errors.push(`field "${field}" is ${describe(value)}, expected ${spec.type}`);
|
||||
}
|
||||
}
|
||||
for (const field of Object.keys(payload)) {
|
||||
if (!fields[field]) {
|
||||
errors.push(`unregistered field "${field}" appeared in the payload`);
|
||||
}
|
||||
}
|
||||
return errors;
|
||||
}
|
||||
|
||||
function matchesType(value: unknown, type: string): boolean {
|
||||
switch (type) {
|
||||
case "string":
|
||||
return typeof value === "string";
|
||||
case "number":
|
||||
return typeof value === "number" && Number.isFinite(value);
|
||||
case "boolean":
|
||||
return typeof value === "boolean";
|
||||
case "timestamp":
|
||||
return typeof value === "string" && !Number.isNaN(Date.parse(value));
|
||||
case "array":
|
||||
return Array.isArray(value);
|
||||
case "object":
|
||||
return typeof value === "object" && value !== null && !Array.isArray(value);
|
||||
default:
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
function describe(value: unknown): string {
|
||||
if (Array.isArray(value)) return "array";
|
||||
return typeof value;
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
/**
|
||||
* src/schema/record-schema.ts
|
||||
* ============================
|
||||
* P063: the schema 0pt versions per source, and the compatibility policy that
|
||||
* governs what the next version may change.
|
||||
*
|
||||
* Deliberately a SMALL structural schema (field name -> type + required), not
|
||||
* a second JSON-Schema dialect: 0pt already ships JSON Schema in
|
||||
* packages/schemas/ for its own contracts, and Zod in src/validation for HTTP
|
||||
* bodies. Neither of those answers the question P063 actually asks — "is
|
||||
* version N+1 of an UPSTREAM source's record shape safe for the consumers
|
||||
* reading version N?" — which needs a comparable, field-level structure, not
|
||||
* a validator. Anything richer belongs in a real schema registry
|
||||
* (Confluent/Apicurio) and would be an integration, not a rewrite of this.
|
||||
*/
|
||||
|
||||
export type FieldType = "string" | "number" | "boolean" | "timestamp" | "object" | "array";
|
||||
|
||||
export interface FieldSpec {
|
||||
type: FieldType;
|
||||
required: boolean;
|
||||
}
|
||||
|
||||
/** A record shape: field name -> spec. */
|
||||
export type RecordSchemaFields = Record<string, FieldSpec>;
|
||||
|
||||
/**
|
||||
* Compatibility policy, using the standard schema-registry vocabulary:
|
||||
*
|
||||
* - `backward` — new readers can read data written under the OLD schema.
|
||||
* Safe: adding an optional field, removing an optional field, relaxing a
|
||||
* required field to optional. Breaking: adding a REQUIRED field, removing
|
||||
* a field an old writer still emits as required, changing a type.
|
||||
* - `forward` — old readers can read data written under the NEW schema.
|
||||
* Safe: adding a required field, removing an optional field. Breaking:
|
||||
* removing a REQUIRED field, changing a type.
|
||||
* - `full` — both of the above.
|
||||
* - `none` — no check. Recorded, never silently assumed: a source must
|
||||
* declare `none` explicitly, and the runner logs it as an accepted risk.
|
||||
*/
|
||||
export type CompatibilityPolicy = "backward" | "forward" | "full" | "none";
|
||||
|
||||
export interface RecordSchema {
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
schema_version: number;
|
||||
fields: RecordSchemaFields;
|
||||
compatibility: CompatibilityPolicy;
|
||||
registered_at: string;
|
||||
registered_by?: string;
|
||||
}
|
||||
|
||||
/** One incompatibility between two schema versions. */
|
||||
export interface SchemaBreak {
|
||||
field: string;
|
||||
/** Which direction the change breaks. */
|
||||
breaks: "backward" | "forward";
|
||||
detail: string;
|
||||
}
|
||||
|
||||
export interface CompatibilityVerdict {
|
||||
compatible: boolean;
|
||||
policy: CompatibilityPolicy;
|
||||
breaks: SchemaBreak[];
|
||||
}
|
||||
@@ -0,0 +1,144 @@
|
||||
/**
|
||||
* src/schema/schema-registry.ts
|
||||
* ==============================
|
||||
* P063: versioning + compatibility policy for a source's record shape.
|
||||
*
|
||||
* The DoD is "a breaking schema is detected BEFORE consumer corruption", so
|
||||
* this registry is placed on the ingestion path in two places, both of them
|
||||
* upstream of any consumer write:
|
||||
*
|
||||
* 1. `registerFromDiscovery` runs when the runner calls connector.discover(),
|
||||
* i.e. before the first read. A shape change that breaks the declared
|
||||
* policy makes the RUN fail with the field-level reasons — no envelope is
|
||||
* produced at all.
|
||||
* 2. `conformanceErrors` (src/schema/compatibility.ts) runs per envelope, so
|
||||
* a source that changes shape mid-stream without changing its declared
|
||||
* schema is caught on the offending record rather than on the next
|
||||
* discovery, days later.
|
||||
*
|
||||
* A source whose policy is "none" opts out of (1) — explicitly, recorded in
|
||||
* the registry row, never by default.
|
||||
*/
|
||||
import { SiaxError } from "../platform/errors";
|
||||
import type { IngestionRepository } from "../repositories/ingestion-repository";
|
||||
import { checkCompatibility } from "./compatibility";
|
||||
import type {
|
||||
CompatibilityPolicy,
|
||||
CompatibilityVerdict,
|
||||
RecordSchema,
|
||||
RecordSchemaFields,
|
||||
} from "./record-schema";
|
||||
|
||||
export interface RegisterSchemaInput {
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
fields: RecordSchemaFields;
|
||||
/** Policy governing THIS and future versions. Required — there is no default. */
|
||||
compatibility: CompatibilityPolicy;
|
||||
registered_by?: string;
|
||||
}
|
||||
|
||||
export interface RegisterSchemaResult {
|
||||
schema: RecordSchema;
|
||||
/** True when the shape was identical to the current version and no new version was created. */
|
||||
unchanged: boolean;
|
||||
/** Present when a comparison ran (i.e. a previous version existed). */
|
||||
verdict?: CompatibilityVerdict;
|
||||
}
|
||||
|
||||
/** Stable comparison of two field maps, order-independent. */
|
||||
function sameFields(a: RecordSchemaFields, b: RecordSchemaFields): boolean {
|
||||
const ka = Object.keys(a).sort();
|
||||
const kb = Object.keys(b).sort();
|
||||
if (ka.length !== kb.length) return false;
|
||||
for (let i = 0; i < ka.length; i++) {
|
||||
if (ka[i] !== kb[i]) return false;
|
||||
const fa = a[ka[i]];
|
||||
const fb = b[kb[i]];
|
||||
if (fa.type !== fb.type || fa.required !== fb.required) return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
export class SchemaRegistry {
|
||||
constructor(private readonly repo: IngestionRepository) {}
|
||||
|
||||
async getLatest(sourceId: string): Promise<RecordSchema | undefined> {
|
||||
return this.repo.getLatestSchema(sourceId);
|
||||
}
|
||||
|
||||
async list(sourceId: string): Promise<RecordSchema[]> {
|
||||
return this.repo.listSchemas(sourceId);
|
||||
}
|
||||
|
||||
/**
|
||||
* Register a shape. Throws `SiaxError("CONFLICT")` — naming every break —
|
||||
* when the new shape violates the previous version's declared policy.
|
||||
* Nothing is persisted when it throws.
|
||||
*/
|
||||
async register(input: RegisterSchemaInput): Promise<RegisterSchemaResult> {
|
||||
if (Object.keys(input.fields).length === 0) {
|
||||
throw new SiaxError("INVALID_REQUEST", "0pt.schema_registry", "a schema must declare at least one field");
|
||||
}
|
||||
const previous = await this.repo.getLatestSchema(input.source_id);
|
||||
if (!previous) {
|
||||
const schema: RecordSchema = {
|
||||
source_id: input.source_id,
|
||||
tenant_id: input.tenant_id,
|
||||
schema_version: 1,
|
||||
fields: input.fields,
|
||||
compatibility: input.compatibility,
|
||||
registered_at: new Date().toISOString(),
|
||||
registered_by: input.registered_by,
|
||||
};
|
||||
await this.repo.putSchema(schema);
|
||||
return { schema, unchanged: false };
|
||||
}
|
||||
|
||||
if (sameFields(previous.fields, input.fields) && previous.compatibility === input.compatibility) {
|
||||
return { schema: previous, unchanged: true };
|
||||
}
|
||||
|
||||
// The PREVIOUS version's policy governs whether this change is allowed —
|
||||
// a producer must not be able to widen its own policy retroactively by
|
||||
// declaring "none" on the breaking version.
|
||||
const verdict = checkCompatibility(previous.fields, input.fields, previous.compatibility);
|
||||
if (!verdict.compatible) {
|
||||
throw new SiaxError(
|
||||
"CONFLICT",
|
||||
"0pt.schema_registry",
|
||||
`schema v${previous.schema_version + 1} for source ${input.source_id} breaks the declared ` +
|
||||
`"${previous.compatibility}" compatibility policy: ` +
|
||||
verdict.breaks.map((b) => `[${b.breaks}] ${b.detail}`).join("; "),
|
||||
);
|
||||
}
|
||||
|
||||
const schema: RecordSchema = {
|
||||
source_id: input.source_id,
|
||||
tenant_id: input.tenant_id,
|
||||
schema_version: previous.schema_version + 1,
|
||||
fields: input.fields,
|
||||
compatibility: input.compatibility,
|
||||
registered_at: new Date().toISOString(),
|
||||
registered_by: input.registered_by,
|
||||
};
|
||||
await this.repo.putSchema(schema);
|
||||
return { schema, unchanged: false, verdict };
|
||||
}
|
||||
|
||||
/**
|
||||
* Non-throwing form used by the runner's pre-flight: answers whether the
|
||||
* shape a connector just discovered would be accepted, without registering
|
||||
* it. Used to report a break as a RUN failure with a full reason list
|
||||
* rather than an exception message.
|
||||
*/
|
||||
async evaluate(
|
||||
sourceId: string,
|
||||
fields: RecordSchemaFields,
|
||||
): Promise<{ previous?: RecordSchema; verdict?: CompatibilityVerdict; changed: boolean }> {
|
||||
const previous = await this.repo.getLatestSchema(sourceId);
|
||||
if (!previous) return { changed: true };
|
||||
if (sameFields(previous.fields, fields)) return { previous, changed: false };
|
||||
return { previous, verdict: checkCompatibility(previous.fields, fields, previous.compatibility), changed: true };
|
||||
}
|
||||
}
|
||||
@@ -44,6 +44,8 @@ import { SourceService, type SourceServiceOptions } from "./source-service";
|
||||
import { SourceRatePolicyService } from "./source-rate-policy-service";
|
||||
import type { CreateSourceInput, Source } from "../types/source";
|
||||
import type {
|
||||
SourceAuthorization,
|
||||
SourceClassification,
|
||||
SourceDescriptor,
|
||||
SourceFreshness,
|
||||
SourceHealth,
|
||||
@@ -51,13 +53,24 @@ import type {
|
||||
SourceHealthStatus,
|
||||
SourceProvenance,
|
||||
SourceRights,
|
||||
SourceSchedule,
|
||||
} from "../types/source-registry";
|
||||
|
||||
export interface RegisterSourceInput extends CreateSourceInput {
|
||||
rights?: Omit<SourceRights, "source_id" | "tenant_id">;
|
||||
provenance?: Omit<SourceProvenance, "source_id" | "tenant_id" | "updated_at">;
|
||||
authorization?: Omit<SourceAuthorization, "source_id" | "tenant_id">;
|
||||
schedule?: Omit<SourceSchedule, "source_id" | "tenant_id" | "updated_at">;
|
||||
}
|
||||
|
||||
/**
|
||||
* Actor ids that may never appear as an authorization's `granted_by`.
|
||||
* P060 requires an accountable HUMAN decision behind a source going live;
|
||||
* a service identity granting itself access is exactly the rubber stamp the
|
||||
* gate exists to prevent.
|
||||
*/
|
||||
const NON_ACCOUNTABLE_ACTORS = /^(system|svc_|service:|robot|automation|anonymous)/i;
|
||||
|
||||
export class SourceRegistryService {
|
||||
private readonly repo: SourceRegistryRepository;
|
||||
private readonly aud0: Aud0Client;
|
||||
@@ -89,8 +102,16 @@ export class SourceRegistryService {
|
||||
// source.register
|
||||
// -------------------------------------------------------------------------
|
||||
async registerSource(input: RegisterSourceInput): Promise<SourceDescriptor> {
|
||||
const { rights, provenance, ...base } = input;
|
||||
const source = await this.sources.createSource(base);
|
||||
const { rights, provenance, authorization, schedule, ...base } = input;
|
||||
// P060: registration NEVER activates. A caller asking for status "active"
|
||||
// gets the source created "pending" and must go through activateSource,
|
||||
// which checks the classification. Silently honouring `status: "active"`
|
||||
// here would leave the gate wide open to its most obvious bypass.
|
||||
const requestedActive = base.status === "active";
|
||||
const source = await this.sources.createSource({
|
||||
...base,
|
||||
status: requestedActive ? "pending" : base.status,
|
||||
});
|
||||
if (rights) {
|
||||
await this.setRights({ ...rights, source_id: source.source_id, tenant_id: source.tenant_id });
|
||||
}
|
||||
@@ -102,9 +123,165 @@ export class SourceRegistryService {
|
||||
updated_at: new Date().toISOString(),
|
||||
});
|
||||
}
|
||||
if (authorization) {
|
||||
await this.setAuthorization({
|
||||
...authorization,
|
||||
source_id: source.source_id,
|
||||
tenant_id: source.tenant_id,
|
||||
});
|
||||
}
|
||||
if (schedule) {
|
||||
await this.setSchedule({
|
||||
...schedule,
|
||||
source_id: source.source_id,
|
||||
tenant_id: source.tenant_id,
|
||||
updated_at: new Date().toISOString(),
|
||||
});
|
||||
}
|
||||
// A registration that arrived fully classified AND asked to be active is
|
||||
// activated here — through the same gate, not around it.
|
||||
if (requestedActive) {
|
||||
await this.activateSource(source.source_id, source.tenant_id, base.created_by);
|
||||
}
|
||||
return this.describeSource(source.source_id, source.tenant_id);
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
// P060 authorization / schedule
|
||||
// -------------------------------------------------------------------------
|
||||
async setAuthorization(auth: SourceAuthorization): Promise<SourceAuthorization> {
|
||||
await this.assertSourceTenant(auth.source_id, auth.tenant_id);
|
||||
if (!auth.granted_by || NON_ACCOUNTABLE_ACTORS.test(auth.granted_by)) {
|
||||
throw new SiaxError(
|
||||
"INVALID_REQUEST",
|
||||
"0pt.source_registry",
|
||||
`authorization.granted_by must name an accountable actor, not "${auth.granted_by}"`,
|
||||
);
|
||||
}
|
||||
await this.repo.setAuthorization(auth);
|
||||
await this.emit("0pt.source_authorization_set", auth.tenant_id, auth.source_id, {
|
||||
basis: auth.basis,
|
||||
granted_by: auth.granted_by,
|
||||
expires_at: auth.expires_at,
|
||||
evidence_ref: auth.evidence_ref,
|
||||
});
|
||||
return auth;
|
||||
}
|
||||
|
||||
async getAuthorization(sourceId: string, tenantId: string): Promise<SourceAuthorization | undefined> {
|
||||
await this.assertSourceTenant(sourceId, tenantId);
|
||||
return this.repo.getAuthorization(sourceId);
|
||||
}
|
||||
|
||||
async setSchedule(schedule: SourceSchedule): Promise<SourceSchedule> {
|
||||
await this.assertSourceTenant(schedule.source_id, schedule.tenant_id);
|
||||
if (schedule.interval_seconds !== undefined && schedule.interval_seconds <= 0) {
|
||||
throw new SiaxError(
|
||||
"INVALID_REQUEST",
|
||||
"0pt.source_registry",
|
||||
"schedule.interval_seconds must be positive",
|
||||
);
|
||||
}
|
||||
await this.repo.setSchedule(schedule);
|
||||
await this.emit("0pt.source_schedule_set", schedule.tenant_id, schedule.source_id, {
|
||||
mode: schedule.mode,
|
||||
interval_seconds: schedule.interval_seconds,
|
||||
enabled: schedule.enabled,
|
||||
});
|
||||
return schedule;
|
||||
}
|
||||
|
||||
async getSchedule(sourceId: string, tenantId: string): Promise<SourceSchedule | undefined> {
|
||||
await this.assertSourceTenant(sourceId, tenantId);
|
||||
return this.repo.getSchedule(sourceId);
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
// P060 classification + activation gate
|
||||
// -------------------------------------------------------------------------
|
||||
/**
|
||||
* Is this source classified? A source is classified when ALL of the
|
||||
* following are on record:
|
||||
* - an accountable owner,
|
||||
* - an unexpired authorization from an accountable actor,
|
||||
* - rights: a license id AND at least one allowed purpose,
|
||||
* - provenance: how the data is obtained.
|
||||
*
|
||||
* Everything else in the registry (schedule, schema, checkpoint) describes
|
||||
* HOW a source runs, not WHETHER it may — those are deliberately not part
|
||||
* of the gate, and that choice is documented here rather than left implicit.
|
||||
*/
|
||||
async classifySource(
|
||||
sourceId: string,
|
||||
tenantId: string,
|
||||
now: Date = new Date(),
|
||||
): Promise<SourceClassification> {
|
||||
const source = await this.assertSourceTenant(sourceId, tenantId);
|
||||
const missing: string[] = [];
|
||||
|
||||
if (!source.owner_id) missing.push("owner_id is not set");
|
||||
|
||||
const auth = await this.repo.getAuthorization(sourceId);
|
||||
if (!auth) {
|
||||
missing.push("no authorization on record");
|
||||
} else {
|
||||
if (!auth.granted_by || NON_ACCOUNTABLE_ACTORS.test(auth.granted_by)) {
|
||||
missing.push(`authorization.granted_by "${auth.granted_by}" is not an accountable actor`);
|
||||
}
|
||||
if (auth.expires_at && new Date(auth.expires_at).getTime() <= now.getTime()) {
|
||||
missing.push(`authorization expired at ${auth.expires_at}`);
|
||||
}
|
||||
}
|
||||
|
||||
const rights = await this.repo.getRights(sourceId);
|
||||
if (!rights) {
|
||||
missing.push("no rights record (license/purpose) on record");
|
||||
} else {
|
||||
if (!rights.license_id) missing.push("rights.license_id is empty");
|
||||
if (rights.allowed_purposes.length === 0) missing.push("rights.allowed_purposes is empty");
|
||||
}
|
||||
|
||||
const provenance = await this.repo.getProvenance(sourceId);
|
||||
if (!provenance) missing.push("no provenance record (sourcing method) on record");
|
||||
|
||||
return { source_id: sourceId, classified: missing.length === 0, missing };
|
||||
}
|
||||
|
||||
/**
|
||||
* P060 DoD: an UNCLASSIFIED source cannot be activated. Fail-closed —
|
||||
* every missing fact is named at once, and nothing is written when the
|
||||
* check fails.
|
||||
*/
|
||||
async activateSource(sourceId: string, tenantId: string, actorId?: string): Promise<Source> {
|
||||
const classification = await this.classifySource(sourceId, tenantId);
|
||||
if (!classification.classified) {
|
||||
await this.emit("0pt.source_activation_denied", tenantId, sourceId, {
|
||||
missing: classification.missing,
|
||||
});
|
||||
throw new SiaxError(
|
||||
"AUTHORIZATION_DENIED",
|
||||
"0pt.source_registry",
|
||||
`source ${sourceId} is not classified and cannot be activated: ${classification.missing.join("; ")}`,
|
||||
);
|
||||
}
|
||||
const source = await this.sources.updateStatus(sourceId, "active", actorId);
|
||||
await this.emit("0pt.source_activated", tenantId, sourceId, { activated_by: actorId ?? null });
|
||||
return source;
|
||||
}
|
||||
|
||||
/** Take a source out of service. Always allowed — deactivating is never gated. */
|
||||
async deactivateSource(
|
||||
sourceId: string,
|
||||
tenantId: string,
|
||||
status: "paused" | "archived" | "deprecated" = "paused",
|
||||
actorId?: string,
|
||||
): Promise<Source> {
|
||||
await this.assertSourceTenant(sourceId, tenantId);
|
||||
const source = await this.sources.updateStatus(sourceId, status, actorId);
|
||||
await this.emit("0pt.source_deactivated", tenantId, sourceId, { status });
|
||||
return source;
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
// rights / license / purpose metadata
|
||||
// -------------------------------------------------------------------------
|
||||
@@ -262,13 +439,17 @@ export class SourceRegistryService {
|
||||
|
||||
async describeSource(sourceId: string, tenantId: string): Promise<SourceDescriptor> {
|
||||
const source = await this.assertSourceTenant(sourceId, tenantId);
|
||||
const [policies, rights, provenance, health, freshness] = await Promise.all([
|
||||
this.ratePolicies.listSourceRatePolicies({ tenant_id: tenantId, source_id: sourceId }),
|
||||
this.repo.getRights(sourceId),
|
||||
this.repo.getProvenance(sourceId),
|
||||
this.getHealth(sourceId, tenantId),
|
||||
this.getFreshness(sourceId, tenantId),
|
||||
]);
|
||||
const [policies, rights, provenance, health, freshness, authorization, schedule, classification] =
|
||||
await Promise.all([
|
||||
this.ratePolicies.listSourceRatePolicies({ tenant_id: tenantId, source_id: sourceId }),
|
||||
this.repo.getRights(sourceId),
|
||||
this.repo.getProvenance(sourceId),
|
||||
this.getHealth(sourceId, tenantId),
|
||||
this.getFreshness(sourceId, tenantId),
|
||||
this.repo.getAuthorization(sourceId),
|
||||
this.repo.getSchedule(sourceId),
|
||||
this.classifySource(sourceId, tenantId),
|
||||
]);
|
||||
return {
|
||||
source,
|
||||
rights,
|
||||
@@ -276,6 +457,9 @@ export class SourceRegistryService {
|
||||
health,
|
||||
freshness,
|
||||
rate_policy: policies[0],
|
||||
authorization,
|
||||
schedule,
|
||||
classification,
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -28,3 +28,19 @@ export const sourceRegistryService = new SourceRegistryService({
|
||||
sourceService,
|
||||
ratePolicyService: sourceRatePolicyService,
|
||||
});
|
||||
|
||||
/**
|
||||
* P060-P068 ingestion plane singletons. Same reasoning as above: the schema
|
||||
* registry, checkpoint store, ledger, quarantine and lineage projection all
|
||||
* share ONE repository so the in-memory dev/test mode is coherent within a
|
||||
* process, and one `resolveStoreMode` decision in production.
|
||||
*/
|
||||
import { createIngestionRepository } from "../repositories/ingestion-repository";
|
||||
import { SchemaRegistry } from "../schema/schema-registry";
|
||||
import { LineageStore } from "../lineage/lineage-store";
|
||||
import { IngestionRunner } from "../ingestion/ingestion-runner";
|
||||
|
||||
export const ingestionRepository = createIngestionRepository();
|
||||
export const schemaRegistry = new SchemaRegistry(ingestionRepository);
|
||||
export const lineageStore = new LineageStore(ingestionRepository);
|
||||
export const ingestionRunner = new IngestionRunner(ingestionRepository, schemaRegistry, lineageStore);
|
||||
|
||||
@@ -77,6 +77,32 @@ export class SourceService {
|
||||
return this.repo.get(sourceId);
|
||||
}
|
||||
|
||||
/**
|
||||
* P060: the ONLY status transition path. Callers do not get a generic
|
||||
* update method, because status is what the activation gate protects —
|
||||
* SourceRegistryService.activateSource is the only caller that moves a
|
||||
* source to "active", and it runs the classification check first.
|
||||
*/
|
||||
async updateStatus(sourceId: string, status: SourceStatus, actorId?: string): Promise<Source> {
|
||||
const source = await this.repo.updateStatus(sourceId, status);
|
||||
await this.aud0.ingestEvent({
|
||||
event_id: `evt_0pt_source_status_${source.source_id}_${Date.now()}`,
|
||||
event_type: "0pt.source_status_changed",
|
||||
tenant_id: source.tenant_id,
|
||||
project_id: source.project_id,
|
||||
actor_id: actorId ?? source.owner_id ?? "system",
|
||||
app_id: "0pt",
|
||||
capability_id: "0pt.update_source_status",
|
||||
resource_type: "source",
|
||||
resource_id: source.source_id,
|
||||
risk_level: "L2",
|
||||
payload: { status: source.status, name: source.name },
|
||||
created_at: source.updated_at ?? new Date().toISOString(),
|
||||
correlation_id: source.correlation_id,
|
||||
});
|
||||
return source;
|
||||
}
|
||||
|
||||
async listSources(filter?: {
|
||||
tenant_id?: string;
|
||||
project_id?: string;
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
/**
|
||||
* src/types/ingestion.ts
|
||||
* =======================
|
||||
* P062: THE common ingestion envelope. Every record that enters 0pt through
|
||||
* a connector — batch page, stream message, or change event — is normalized
|
||||
* into exactly this shape before anything downstream (quality gate, entity
|
||||
* resolution, dataset write, lineage) sees it. Nothing downstream is allowed
|
||||
* to special-case "this came from a file" vs "this came from a change feed";
|
||||
* that difference is expressed by `mode` + `operation` and nothing else.
|
||||
*
|
||||
* HONEST SCOPE NOTE. `mode: "cdc"` and the delete/update operations are part
|
||||
* of the CONTRACT, and the runner handles them (a delete envelope carries no
|
||||
* payload and is never quality-gated as if it had one). This repo does NOT
|
||||
* ship a CDC *source* — there is no Debezium consumer, no replication-slot
|
||||
* reader, no WAL tailer here, exactly as services/api/tests/test_known_gaps.py
|
||||
* has stated since P010. What changed with P062 is that a CDC connector can
|
||||
* now be added without redesigning the pipeline, and that the checkpoint it
|
||||
* would resume from is durable. A CDC connector itself is still MISSING.
|
||||
*
|
||||
* `checkpoint` on a ReadResult is what makes resume real: the connector hands
|
||||
* back the cursor that means "everything up to and including these envelopes
|
||||
* has been produced", the runner commits it durably AFTER the envelopes are
|
||||
* accepted, and a restart passes it straight back into read(). Committing it
|
||||
* before acceptance would create silent gaps — that ordering is the whole
|
||||
* point and is asserted in tests/p0/ingestion-runner.test.ts.
|
||||
*/
|
||||
|
||||
/** How data arrives from the source. */
|
||||
export type IngestionMode = "batch" | "stream" | "cdc";
|
||||
|
||||
/**
|
||||
* What the record means. Batch/stream sources normally emit "upsert"
|
||||
* (they carry current state); a change feed distinguishes the three CDC
|
||||
* operations. "delete" envelopes carry `payload: null`.
|
||||
*/
|
||||
export type EnvelopeOperation = "upsert" | "insert" | "update" | "delete";
|
||||
|
||||
/** Per-envelope provenance — which code, which version, which run produced it. */
|
||||
export interface EnvelopeProvenance {
|
||||
connector_id: string;
|
||||
connector_version: string;
|
||||
/** The ingestion run that produced this envelope. Joins to lineage + checkpoints. */
|
||||
run_id: string;
|
||||
/** Opaque connector cursor at which this record was produced. */
|
||||
cursor?: string;
|
||||
}
|
||||
|
||||
/** One normalized record. `payload` is null only for `operation: "delete"`. */
|
||||
export interface IngestionEnvelope {
|
||||
envelope_id: string;
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
mode: IngestionMode;
|
||||
operation: EnvelopeOperation;
|
||||
/** Partition/stream within the source (a table, a topic, a file). */
|
||||
partition_key: string;
|
||||
/** Stable business key within the partition — the unit of idempotency (P068). */
|
||||
record_key: string;
|
||||
/** Schema version the payload conforms to (P063). */
|
||||
schema_version: number;
|
||||
/** Monotonic per-(source, partition) sequence assigned by the runner. */
|
||||
sequence: number;
|
||||
/** When the change happened upstream, when known. */
|
||||
occurred_at?: string;
|
||||
/** When 0pt normalized it. */
|
||||
ingested_at: string;
|
||||
payload: Record<string, unknown> | null;
|
||||
provenance: EnvelopeProvenance;
|
||||
}
|
||||
|
||||
/**
|
||||
* The durable resume point for one (source, partition). Committed only after
|
||||
* the envelopes it covers have been accepted downstream.
|
||||
*/
|
||||
export interface IngestionCheckpoint {
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
partition_key: string;
|
||||
/** Opaque connector cursor. `undefined` means "never read". */
|
||||
cursor?: string;
|
||||
/** Highest envelope sequence committed for this partition. */
|
||||
sequence: number;
|
||||
records_committed: number;
|
||||
committed_at: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* P064: an envelope the declared quality policy sent to quarantine instead of
|
||||
* downstream. Quarantine is DURABLE and inspectable — a dropped record that
|
||||
* only exists in a log line is a silently lost record.
|
||||
*/
|
||||
export interface QuarantineRecord {
|
||||
quarantine_id: string;
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
run_id: string;
|
||||
record_key: string;
|
||||
/** One entry per failed expectation. */
|
||||
violations: Array<{ expectation_type: string; detail: string }>;
|
||||
envelope: IngestionEnvelope;
|
||||
quarantined_at: string;
|
||||
}
|
||||
|
||||
/** P068: what the idempotency ledger decided about one delivered record. */
|
||||
export type LedgerVerdict = "new" | "changed" | "duplicate";
|
||||
|
||||
export interface LedgerDecision {
|
||||
record_key: string;
|
||||
verdict: LedgerVerdict;
|
||||
}
|
||||
@@ -87,7 +87,11 @@ export interface SourceFreshness {
|
||||
age_hours?: number;
|
||||
}
|
||||
|
||||
/** The full `source.describe` document (§10): base + rights + provenance + health + freshness + rate policy. */
|
||||
/**
|
||||
* The full `source.describe` document (§10): base + rights + provenance +
|
||||
* health + freshness + rate policy, plus the P060 additions (authorization,
|
||||
* schedule, and the classification verdict the activation gate applies).
|
||||
*/
|
||||
export interface SourceDescriptor {
|
||||
source: import("./source").Source;
|
||||
rights?: SourceRights;
|
||||
@@ -95,4 +99,67 @@ export interface SourceDescriptor {
|
||||
health: SourceHealth;
|
||||
freshness: SourceFreshness;
|
||||
rate_policy?: import("./source-rate-policy").SourceRatePolicy;
|
||||
authorization?: SourceAuthorization;
|
||||
schedule?: SourceSchedule;
|
||||
/** Why this source may (or may not) be activated. */
|
||||
classification?: SourceClassification;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// P060: the registry facts that were missing from the original §10 pass —
|
||||
// authorization, schedule, schema reference, checkpoint — plus the
|
||||
// classification verdict the activation gate produces.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/** Legal/contractual ground on which this tenant may connect to the source at all. */
|
||||
export type AuthorizationBasis =
|
||||
| "contract"
|
||||
| "public_register"
|
||||
| "consent"
|
||||
| "legitimate_interest"
|
||||
| "internal";
|
||||
|
||||
/**
|
||||
* The recorded decision that 0pt MAY connect to this source (P060
|
||||
* "authorization"). Deliberately separate from {@link SourceRights}: rights
|
||||
* say what the DATA may be used for, authorization says who signed off that
|
||||
* the CONNECTION may exist. A source needs both to be classified.
|
||||
*/
|
||||
export interface SourceAuthorization {
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
basis: AuthorizationBasis;
|
||||
/** Accountable actor id — never "system"/"svc_*" (the gate rejects those). */
|
||||
granted_by: string;
|
||||
granted_at: string;
|
||||
/** ISO timestamp; an authorization past this instant does not classify the source. */
|
||||
expires_at?: string;
|
||||
/** AUD0 evidence ref or contract artifact ref backing the decision. */
|
||||
evidence_ref?: string;
|
||||
notes?: string;
|
||||
}
|
||||
|
||||
/** How often 0pt pulls from the source, and in which ingestion mode (P060 "schedule"). */
|
||||
export interface SourceSchedule {
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
mode: "batch" | "stream" | "cdc";
|
||||
/** Poll cadence in seconds. Omitted for a continuously-open stream. */
|
||||
interval_seconds?: number;
|
||||
/** Optional cron expression. Recorded here; 0pt does not execute cron itself (ACT0 owns scheduling). */
|
||||
cron?: string;
|
||||
enabled: boolean;
|
||||
updated_at: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Why a source is (not) classified. `activateSource` refuses on
|
||||
* `classified === false` and names every missing fact, so an operator gets
|
||||
* the whole list at once instead of one error per round trip.
|
||||
*/
|
||||
export interface SourceClassification {
|
||||
source_id: string;
|
||||
classified: boolean;
|
||||
/** Human-readable reasons, one per missing/expired fact. Empty when classified. */
|
||||
missing: string[];
|
||||
}
|
||||
|
||||
+17
-1
@@ -34,8 +34,16 @@ export type SourceType = "database" | "file" | "web" | "api" | "stream" | "manua
|
||||
* Union of both status vocabularies. "archived" and "deprecated" both survive
|
||||
* the unification merge -- they were introduced independently and no code
|
||||
* treats either as canonical.
|
||||
*
|
||||
* "pending" (P060) is the state every newly registered source starts in. It
|
||||
* exists because "active" used to be the CREATE default
|
||||
* (postgres-source-repository.ts / in-memory-source-repository.ts both did
|
||||
* `input.status ?? "active"`), which meant a source with no rights, no
|
||||
* provenance and no authorization was live the instant it was registered.
|
||||
* Reaching "active" now requires passing SourceRegistryService.activateSource's
|
||||
* classification gate.
|
||||
*/
|
||||
export type SourceStatus = "active" | "paused" | "archived" | "deprecated";
|
||||
export type SourceStatus = "pending" | "active" | "paused" | "archived" | "deprecated";
|
||||
|
||||
/** Legal-sourcing vocabulary, mirroring dim_source.access. */
|
||||
export type SourceAccess = "api" | "bulk" | "authorized_web" | "licensed";
|
||||
@@ -60,7 +68,14 @@ export interface Source {
|
||||
correlation_id?: string;
|
||||
created_at: string;
|
||||
updated_at?: string;
|
||||
/** Who registered the source. */
|
||||
created_by?: string;
|
||||
/**
|
||||
* Who is ACCOUNTABLE for the source going forward (P060 "owner"). Distinct
|
||||
* from `created_by`, which is only who typed the registration. Required by
|
||||
* the activation gate.
|
||||
*/
|
||||
owner_id?: string;
|
||||
}
|
||||
|
||||
export interface CreateSourceInput {
|
||||
@@ -75,4 +90,5 @@ export interface CreateSourceInput {
|
||||
status?: SourceStatus;
|
||||
correlation_id?: string;
|
||||
created_by?: string;
|
||||
owner_id?: string;
|
||||
}
|
||||
|
||||
@@ -127,7 +127,8 @@ export const registerSourceSchema = z.object({
|
||||
adapter_id: z.string().min(1).optional(),
|
||||
access: z.enum(['api', 'bulk', 'authorized_web', 'licensed']).optional(),
|
||||
description: z.string().min(1).optional(),
|
||||
status: z.enum(['active', 'paused', 'archived', 'deprecated']).optional(),
|
||||
status: z.enum(['pending', 'active', 'paused', 'archived', 'deprecated']).optional(),
|
||||
owner_id: z.string().min(1).optional(),
|
||||
correlation_id: z.string().min(1).optional(),
|
||||
created_by: z.string().min(1).optional(),
|
||||
rights: sourceRightsInputSchema.optional(),
|
||||
@@ -232,3 +233,33 @@ export const qualityBridgeRunSchema = z.object({
|
||||
rows: z.array(jsonObject),
|
||||
correlation_id: z.string().min(1).optional(),
|
||||
});
|
||||
|
||||
/** Nested `authorization` block on POST /api/0pt/sources, and the body of PUT /api/0pt/sources/[sourceId]/authorization (P060). */
|
||||
const sourceAuthorizationInputSchema = z.object({
|
||||
basis: z.enum(['contract', 'public_register', 'consent', 'legitimate_interest', 'internal']),
|
||||
granted_by: z.string().min(1),
|
||||
granted_at: z.string().min(1),
|
||||
expires_at: z.string().min(1).optional(),
|
||||
evidence_ref: z.string().min(1).optional(),
|
||||
notes: z.string().min(1).optional(),
|
||||
});
|
||||
|
||||
/** PUT /api/0pt/sources/[sourceId]/authorization -- source.set_authorization (P060). */
|
||||
export const setSourceAuthorizationSchema = sourceAuthorizationInputSchema.extend({
|
||||
tenant_id: z.string().min(1),
|
||||
});
|
||||
|
||||
/** PUT /api/0pt/sources/[sourceId]/schedule -- source.set_schedule (P060). */
|
||||
export const setSourceScheduleSchema = z.object({
|
||||
tenant_id: z.string().min(1),
|
||||
mode: z.enum(['batch', 'stream', 'cdc']),
|
||||
interval_seconds: z.number().int().positive().optional(),
|
||||
cron: z.string().min(1).optional(),
|
||||
enabled: z.boolean(),
|
||||
});
|
||||
|
||||
/** POST /api/0pt/sources/[sourceId]/activate -- source.activate (P060 classification gate). */
|
||||
export const activateSourceSchema = z.object({
|
||||
tenant_id: z.string().min(1),
|
||||
actor_id: z.string().min(1).optional(),
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user