feat(data-os): P060-P068 durable ingestion plane — source classification gate, connector contract, common envelope, schema evolution, quality enforcement, queryable lineage, idempotent backfill
CI (SIAX Cloud) / integration (push) Successful in 42s
CI (SIAX Cloud) / quality (push) Successful in 1m17s
CI (SIAX Cloud) / security (push) Successful in 46s
Security Scan (SIAX Cloud) / security (push) Successful in 31s
CI (SIAX Cloud) / security (pull_request) Successful in 24s
CI (SIAX Cloud) / integration (pull_request) Successful in 36s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m4s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 27s
CI (SIAX Cloud) / integration (push) Successful in 42s
CI (SIAX Cloud) / quality (push) Successful in 1m17s
CI (SIAX Cloud) / security (push) Successful in 46s
Security Scan (SIAX Cloud) / security (push) Successful in 31s
CI (SIAX Cloud) / security (pull_request) Successful in 24s
CI (SIAX Cloud) / integration (pull_request) Successful in 36s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m4s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 27s
BEFORE_STATE (verified, with evidence):
* P060 PARTIAL/BROKEN. The registry persisted rights/provenance/health/
freshness to Postgres, but carried no authorization, no schedule, no
checkpoint, no schema and no owner distinct from created_by — and the
activation DoD was inverted: postgres-source-repository.ts wrote
`${input.status ?? "active"}`, so a source with no rights, no provenance
and no authorization was ACTIVE the instant it was registered. No
activation operation existed at all.
* P061 MISSING. services/api/sources/connectors.py had per-source fetch_*()
functions only — no lifecycle, no cursor, no resume, no health, no
conformance suite.
* P062 MISSING, already documented as such in
services/api/tests/test_known_gaps.py:10-19 (no CDC pipeline anywhere;
workers/pipeline.py has no persistence at all).
* P063 MISSING. schema_version existed as a free string; no comparison,
no policy, no versioned source schema.
* P064 PARTIAL. Rules/runs/issues/scores existed, but a failing rule only
opened an issue and the data flowed on. No quarantine anywhere.
* P065 DUPLICATE (confirmed). Two ER engines: src/er/entity-resolution-
service.ts (canonical) and services/api/workers/resolve.py (simplified,
still called by pipeline.py:110).
* P067 PARTIAL. Lineage was emitted to AUD0 but nothing here could ANSWER a
lineage question — no store, no query, no route.
* P068 MISSING, documented in test_known_gaps.py:44-52 ("no delivery-store
persistence layer exists to dedupe a backfill against").
BUILT (only what was PARTIAL/BROKEN/MISSING):
* P060 sql/007_ingestion_plane.sql adds authorization/schedule/schemas/
checkpoints/ledger/quarantine/lineage tables, owner_id, and flips the
opt_sources status DEFAULT to 'pending' so even a raw psql INSERT cannot
produce an unclassified-but-active source. SourceRegistryService gains
classifySource/activateSource/deactivateSource + authorization/schedule
accessors; registration NEVER activates (asking for status:"active" goes
through the gate, not around it); a service identity cannot authorize a
source. Routes: activate (GET classification / POST activate),
authorization, schedule.
* P061 src/connectors/connector.ts — all nine operations on one interface.
src/connectors/conformance.ts — a runnable suite (15 checks, no "skip"
verdict). HttpJsonConnector (real HTTP, opaque cursor) and
JsonlFileConnector (real file, line offsets, CDC deletes) both pass it.
The suite caught a real defect while being written: an API that drops
next_cursor at the end left the connector unable to resume, which is why
HttpJsonConnectorConfig has cursor_field.
* P062 src/types/ingestion.ts + src/ingestion/{envelope,ingestion-runner}.ts.
One envelope for batch/stream/cdc; durable per-(source,partition)
checkpoints; resume proven by killing a run inside the sink.
* P063 src/schema/{record-schema,compatibility,schema-registry}.ts. Two
enforcement points, both upstream of any consumer write: a discovery gate
before the first read, and per-record conformance before the page reaches
the sink. A producer cannot widen its own policy retroactively.
* P064 src/quality/quality-policy.ts composes the EXISTING ExpectationEngine
(not a second engine) and adds the missing consequence: block /
quarantine / warn, with quarantine durable and recoverable. No policy is
reported as "none_declared", never as a pass; an unexecutable expectation
counts as a violation, never a silent pass.
* P067 src/lineage/{lineage-edge,lineage-store}.ts + GET /api/0pt/lineage.
AUD0 stays the system of record; this is 0pt's own queryable projection.
* P068 opt_ingestion_ledger + IngestionRunner.backfill: bounded, idempotent
on repeat, leaves the live checkpoint alone, still re-emits real changes.
A BUG THIS PASS WROTE AND CAUGHT: the first IngestionRepository draft
combined the idempotency check and its recording into one atomic call
BEFORE the sink. The crash/resume test then delivered 7 of 10 records — the
page in flight when the sink died was already marked delivered, so the
resumed run skipped it as a duplicate. Permanent, silent data loss. Fixed by
splitting peekDeliveries/commitDeliveries and committing both the ledger and
the checkpoint only after the sink resolves.
NOT DONE, stated rather than deferred quietly:
* P065 is NOT met. resolve.py is still a second simplified production
engine. Closing it means rewriting it as an adapter over the canonical
service, which cannot be verified from this repo today: services/api is
not built by this repo's Dockerfile, api.0pt.app answers 404, and NO CI
workflow runs its pytest suite. The duplication is now stated at both
sites in code and the removal plan is in STATUS.md. The new ingestion
plane performs no ER, so it adds no third engine.
* P066 (Iceberg) untouched — ST0RE is owned by another workflow. Contract
read only.
* Delivery is AT-LEAST-ONCE, not exactly-once (no per-partition run lock).
* No CDC *source* exists; the contract is met, a CDC connector is not built.
* 0pt does not schedule itself — the schedule row is a declaration ACT0
would read.
VERIFIED: pnpm typecheck clean; pnpm test 365 passed / 25 skipped (was
311/18); pnpm build clean with all four new routes in the route table; lint
unchanged at 12 pre-existing errors. Real Postgres 16
(tests/p0/ingestion-plane-postgres.realinfra.test.ts, 7 passed) plus a real
`docker restart` of that container with an identical read-back through the
production code path — that is P060's "a restart preserves all sources",
measured.
DEPLOYMENT REALITY (curl, 2026-09-04): 0pt.app/api/health returns 200 from
Vercel with db:not_configured; 0pt.app/api/0pt/sources returns 404;
api.0pt.app/health returns 404; cd.yml still guards on a placeholder Coolify
app id. None of the capability layer runs in production. Nothing here is
LIVE_VERIFIED and STATUS.md says so.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
c5ebceff38
commit
9267576b7c
@@ -0,0 +1,111 @@
|
||||
/**
|
||||
* src/types/ingestion.ts
|
||||
* =======================
|
||||
* P062: THE common ingestion envelope. Every record that enters 0pt through
|
||||
* a connector — batch page, stream message, or change event — is normalized
|
||||
* into exactly this shape before anything downstream (quality gate, entity
|
||||
* resolution, dataset write, lineage) sees it. Nothing downstream is allowed
|
||||
* to special-case "this came from a file" vs "this came from a change feed";
|
||||
* that difference is expressed by `mode` + `operation` and nothing else.
|
||||
*
|
||||
* HONEST SCOPE NOTE. `mode: "cdc"` and the delete/update operations are part
|
||||
* of the CONTRACT, and the runner handles them (a delete envelope carries no
|
||||
* payload and is never quality-gated as if it had one). This repo does NOT
|
||||
* ship a CDC *source* — there is no Debezium consumer, no replication-slot
|
||||
* reader, no WAL tailer here, exactly as services/api/tests/test_known_gaps.py
|
||||
* has stated since P010. What changed with P062 is that a CDC connector can
|
||||
* now be added without redesigning the pipeline, and that the checkpoint it
|
||||
* would resume from is durable. A CDC connector itself is still MISSING.
|
||||
*
|
||||
* `checkpoint` on a ReadResult is what makes resume real: the connector hands
|
||||
* back the cursor that means "everything up to and including these envelopes
|
||||
* has been produced", the runner commits it durably AFTER the envelopes are
|
||||
* accepted, and a restart passes it straight back into read(). Committing it
|
||||
* before acceptance would create silent gaps — that ordering is the whole
|
||||
* point and is asserted in tests/p0/ingestion-runner.test.ts.
|
||||
*/
|
||||
|
||||
/** How data arrives from the source. */
|
||||
export type IngestionMode = "batch" | "stream" | "cdc";
|
||||
|
||||
/**
|
||||
* What the record means. Batch/stream sources normally emit "upsert"
|
||||
* (they carry current state); a change feed distinguishes the three CDC
|
||||
* operations. "delete" envelopes carry `payload: null`.
|
||||
*/
|
||||
export type EnvelopeOperation = "upsert" | "insert" | "update" | "delete";
|
||||
|
||||
/** Per-envelope provenance — which code, which version, which run produced it. */
|
||||
export interface EnvelopeProvenance {
|
||||
connector_id: string;
|
||||
connector_version: string;
|
||||
/** The ingestion run that produced this envelope. Joins to lineage + checkpoints. */
|
||||
run_id: string;
|
||||
/** Opaque connector cursor at which this record was produced. */
|
||||
cursor?: string;
|
||||
}
|
||||
|
||||
/** One normalized record. `payload` is null only for `operation: "delete"`. */
|
||||
export interface IngestionEnvelope {
|
||||
envelope_id: string;
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
mode: IngestionMode;
|
||||
operation: EnvelopeOperation;
|
||||
/** Partition/stream within the source (a table, a topic, a file). */
|
||||
partition_key: string;
|
||||
/** Stable business key within the partition — the unit of idempotency (P068). */
|
||||
record_key: string;
|
||||
/** Schema version the payload conforms to (P063). */
|
||||
schema_version: number;
|
||||
/** Monotonic per-(source, partition) sequence assigned by the runner. */
|
||||
sequence: number;
|
||||
/** When the change happened upstream, when known. */
|
||||
occurred_at?: string;
|
||||
/** When 0pt normalized it. */
|
||||
ingested_at: string;
|
||||
payload: Record<string, unknown> | null;
|
||||
provenance: EnvelopeProvenance;
|
||||
}
|
||||
|
||||
/**
|
||||
* The durable resume point for one (source, partition). Committed only after
|
||||
* the envelopes it covers have been accepted downstream.
|
||||
*/
|
||||
export interface IngestionCheckpoint {
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
partition_key: string;
|
||||
/** Opaque connector cursor. `undefined` means "never read". */
|
||||
cursor?: string;
|
||||
/** Highest envelope sequence committed for this partition. */
|
||||
sequence: number;
|
||||
records_committed: number;
|
||||
committed_at: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* P064: an envelope the declared quality policy sent to quarantine instead of
|
||||
* downstream. Quarantine is DURABLE and inspectable — a dropped record that
|
||||
* only exists in a log line is a silently lost record.
|
||||
*/
|
||||
export interface QuarantineRecord {
|
||||
quarantine_id: string;
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
run_id: string;
|
||||
record_key: string;
|
||||
/** One entry per failed expectation. */
|
||||
violations: Array<{ expectation_type: string; detail: string }>;
|
||||
envelope: IngestionEnvelope;
|
||||
quarantined_at: string;
|
||||
}
|
||||
|
||||
/** P068: what the idempotency ledger decided about one delivered record. */
|
||||
export type LedgerVerdict = "new" | "changed" | "duplicate";
|
||||
|
||||
export interface LedgerDecision {
|
||||
record_key: string;
|
||||
verdict: LedgerVerdict;
|
||||
}
|
||||
@@ -87,7 +87,11 @@ export interface SourceFreshness {
|
||||
age_hours?: number;
|
||||
}
|
||||
|
||||
/** The full `source.describe` document (§10): base + rights + provenance + health + freshness + rate policy. */
|
||||
/**
|
||||
* The full `source.describe` document (§10): base + rights + provenance +
|
||||
* health + freshness + rate policy, plus the P060 additions (authorization,
|
||||
* schedule, and the classification verdict the activation gate applies).
|
||||
*/
|
||||
export interface SourceDescriptor {
|
||||
source: import("./source").Source;
|
||||
rights?: SourceRights;
|
||||
@@ -95,4 +99,67 @@ export interface SourceDescriptor {
|
||||
health: SourceHealth;
|
||||
freshness: SourceFreshness;
|
||||
rate_policy?: import("./source-rate-policy").SourceRatePolicy;
|
||||
authorization?: SourceAuthorization;
|
||||
schedule?: SourceSchedule;
|
||||
/** Why this source may (or may not) be activated. */
|
||||
classification?: SourceClassification;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// P060: the registry facts that were missing from the original §10 pass —
|
||||
// authorization, schedule, schema reference, checkpoint — plus the
|
||||
// classification verdict the activation gate produces.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/** Legal/contractual ground on which this tenant may connect to the source at all. */
|
||||
export type AuthorizationBasis =
|
||||
| "contract"
|
||||
| "public_register"
|
||||
| "consent"
|
||||
| "legitimate_interest"
|
||||
| "internal";
|
||||
|
||||
/**
|
||||
* The recorded decision that 0pt MAY connect to this source (P060
|
||||
* "authorization"). Deliberately separate from {@link SourceRights}: rights
|
||||
* say what the DATA may be used for, authorization says who signed off that
|
||||
* the CONNECTION may exist. A source needs both to be classified.
|
||||
*/
|
||||
export interface SourceAuthorization {
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
basis: AuthorizationBasis;
|
||||
/** Accountable actor id — never "system"/"svc_*" (the gate rejects those). */
|
||||
granted_by: string;
|
||||
granted_at: string;
|
||||
/** ISO timestamp; an authorization past this instant does not classify the source. */
|
||||
expires_at?: string;
|
||||
/** AUD0 evidence ref or contract artifact ref backing the decision. */
|
||||
evidence_ref?: string;
|
||||
notes?: string;
|
||||
}
|
||||
|
||||
/** How often 0pt pulls from the source, and in which ingestion mode (P060 "schedule"). */
|
||||
export interface SourceSchedule {
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
mode: "batch" | "stream" | "cdc";
|
||||
/** Poll cadence in seconds. Omitted for a continuously-open stream. */
|
||||
interval_seconds?: number;
|
||||
/** Optional cron expression. Recorded here; 0pt does not execute cron itself (ACT0 owns scheduling). */
|
||||
cron?: string;
|
||||
enabled: boolean;
|
||||
updated_at: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Why a source is (not) classified. `activateSource` refuses on
|
||||
* `classified === false` and names every missing fact, so an operator gets
|
||||
* the whole list at once instead of one error per round trip.
|
||||
*/
|
||||
export interface SourceClassification {
|
||||
source_id: string;
|
||||
classified: boolean;
|
||||
/** Human-readable reasons, one per missing/expired fact. Empty when classified. */
|
||||
missing: string[];
|
||||
}
|
||||
|
||||
+17
-1
@@ -34,8 +34,16 @@ export type SourceType = "database" | "file" | "web" | "api" | "stream" | "manua
|
||||
* Union of both status vocabularies. "archived" and "deprecated" both survive
|
||||
* the unification merge -- they were introduced independently and no code
|
||||
* treats either as canonical.
|
||||
*
|
||||
* "pending" (P060) is the state every newly registered source starts in. It
|
||||
* exists because "active" used to be the CREATE default
|
||||
* (postgres-source-repository.ts / in-memory-source-repository.ts both did
|
||||
* `input.status ?? "active"`), which meant a source with no rights, no
|
||||
* provenance and no authorization was live the instant it was registered.
|
||||
* Reaching "active" now requires passing SourceRegistryService.activateSource's
|
||||
* classification gate.
|
||||
*/
|
||||
export type SourceStatus = "active" | "paused" | "archived" | "deprecated";
|
||||
export type SourceStatus = "pending" | "active" | "paused" | "archived" | "deprecated";
|
||||
|
||||
/** Legal-sourcing vocabulary, mirroring dim_source.access. */
|
||||
export type SourceAccess = "api" | "bulk" | "authorized_web" | "licensed";
|
||||
@@ -60,7 +68,14 @@ export interface Source {
|
||||
correlation_id?: string;
|
||||
created_at: string;
|
||||
updated_at?: string;
|
||||
/** Who registered the source. */
|
||||
created_by?: string;
|
||||
/**
|
||||
* Who is ACCOUNTABLE for the source going forward (P060 "owner"). Distinct
|
||||
* from `created_by`, which is only who typed the registration. Required by
|
||||
* the activation gate.
|
||||
*/
|
||||
owner_id?: string;
|
||||
}
|
||||
|
||||
export interface CreateSourceInput {
|
||||
@@ -75,4 +90,5 @@ export interface CreateSourceInput {
|
||||
status?: SourceStatus;
|
||||
correlation_id?: string;
|
||||
created_by?: string;
|
||||
owner_id?: string;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user