feat(ingestion): VehIQ OLTP -> ClickHouse watermark extractor (P070)
CI (SIAX Cloud) / integration (push) Skipped
masterplan-lock / check (push) Successful in 7s
CI (SIAX Cloud) / security (push) Successful in 17s
masterplan-lock / check (pull_request) Successful in 9s
Security Scan (SIAX Cloud) / security (push) Successful in 38s
CI (SIAX Cloud) / security (pull_request) Successful in 26s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 26s
CI (SIAX Cloud) / quality (push) Successful in 57s
CI (SIAX Cloud) / integration (pull_request) Successful in 39s
CI (SIAX Cloud) / quality (pull_request) Successful in 59s
CI (SIAX Cloud) / integration (push) Skipped
masterplan-lock / check (push) Successful in 7s
CI (SIAX Cloud) / security (push) Successful in 17s
masterplan-lock / check (pull_request) Successful in 9s
Security Scan (SIAX Cloud) / security (push) Successful in 38s
CI (SIAX Cloud) / security (pull_request) Successful in 26s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 26s
CI (SIAX Cloud) / quality (push) Successful in 57s
CI (SIAX Cloud) / integration (pull_request) Successful in 39s
CI (SIAX Cloud) / quality (pull_request) Successful in 59s
Implements Simon's specified pattern (dealer_stock_entry/deal -> watermark/updated_at -> 0pt ingestion -> ClickHouse) as a P061 connector pair (PostgresWatermarkConnector, PostgresKeyedLookupConnector) plugged into the existing P062 IngestionRunner, writing to opt_dataset_projection via SiaxAnalyticsEngine.ingestProjection. Built here, not in admin/vehiq: that repo's own ADR-0035 (decisions 1/2/10, CI-enforced by validate_engine_firewall.py) forbids VehIQ code from ever holding a ClickHouse credential or hammering OLTP directly for analytics. See docs/P070_VEHIQ_OLTP_EXTRACTOR.md for the full rationale. Also corrects a real schema mismatch found while building this: production has no generic `updated_at` on any of the three target tables (verified live, read-only, 2026-09-25). dealer_stock_entry/deal use listed_at/ created_at as insert-time watermarks instead (documented caveat: in-place updates are not caught without a schema change); app.listing has no timestamp column at all and is deliberately not covered yet. Vehicle dimension lookup is narrow by construction: keyed on exactly the vehicle_id set referenced by the batch just read, never a scan of the 98M-row vehiq.vehicle table (connector-level validate() refuses that table by name; runner-level tests assert the query only ever touches referenced IDs). Verified: fixture-backed connector/runner tests (watermark+limit, same-timestamp tie at a page boundary, crash/resume, empty-table case, narrow-lookup query-count assertions) + a real disposable Postgres container (caught two real Date-handling bugs fixtures could not) + a read-only proof against actual production (server10, vehiq-db): the extractor's exact SQL returns zero rows, no error, confirming it is safe to schedule with zero current effect. .gitea/workflows/vehiq-oltp-ingest.yml is present, cron-scheduled, and defaults to VEHIQ_INGEST_MODE=preview (read-only, never writes) with no secrets configured — safe to merge with zero production effect until explicitly activated per the doc's "what activating this requires". Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
ec487746fd
commit
07158099ee
@@ -0,0 +1,258 @@
|
||||
/**
|
||||
* src/connectors/postgres-keyed-lookup-connector.ts
|
||||
* ====================================================
|
||||
* P061 connector: a narrow, explicit-key-list lookup against a large table —
|
||||
* built specifically so `0pt dimension_vehicle` never becomes `SELECT *
|
||||
* FROM vehicle` against the 98M-row/85GB `vehiq.vehicle` table.
|
||||
*
|
||||
* THE RULE THIS ENFORCES (Simon's decision #2, `admin/vehiq` 2026-09-25
|
||||
* pass): `dealer_stock_entry.vehicle_id` -> unique vehicle IDs -> a
|
||||
* TARGETED lookup of just those rows' attributes -> `dimension_vehicle`.
|
||||
* Never a full scan, never an unbounded WHERE clause, never "give me
|
||||
* everything and I'll filter downstream".
|
||||
*
|
||||
* HOW IT'S STRUCTURALLY DIFFERENT FROM `postgres-watermark-connector.ts`,
|
||||
* not just configured differently: this connector's config carries `keys`
|
||||
* — a caller-supplied, already-resolved, finite list of record keys. There
|
||||
* is no watermark, no `ORDER BY updated_at`, no notion of "everything since
|
||||
* X". `read()` can ONLY ever emit rows whose key is in that list, and
|
||||
* `connect()` REFUSES an empty or missing key list rather than defaulting
|
||||
* to "read everything" — see the "zero keys issues zero queries" test,
|
||||
* which is the assertion that actually proves the narrow-scan property
|
||||
* (not just that the SQL text LOOKS bounded).
|
||||
*
|
||||
* THE ORCHESTRATOR (scripts/vehiq-oltp-ingest/run.ts) is what supplies
|
||||
* `keys`: the DISTINCT `vehicle_id`s seen in the `dealer_stock_entry` batch
|
||||
* this connector instance was built for, and nothing else. A full snapshot/
|
||||
* backfill of the whole `vehicle` table is a deliberate, separate, later
|
||||
* decision — see docs/P070_VEHIQ_OLTP_EXTRACTOR.md's "Deferred: full
|
||||
* vehicle dimension backfill" section. This connector does not build
|
||||
* towards that; it is the permanent narrow path, not a stepping stone.
|
||||
*/
|
||||
import {
|
||||
ConnectorNotConnectedError,
|
||||
type BackfillRequest,
|
||||
type Connector,
|
||||
type ConnectorConfig,
|
||||
type ConnectorContext,
|
||||
type ConnectorHealth,
|
||||
type ConnectorRecord,
|
||||
type DiscoveryResult,
|
||||
type ReadRequest,
|
||||
type ReadResult,
|
||||
type ValidationResult,
|
||||
} from "./connector";
|
||||
import type { IngestionCheckpoint } from "../types/ingestion";
|
||||
import type { RecordSchemaFields } from "../schema/record-schema";
|
||||
import type { QueryImpl, SourceRow } from "./postgres-watermark-connector";
|
||||
|
||||
export const POSTGRES_KEYED_LOOKUP_CONNECTOR_VERSION = "postgres-keyed-lookup-connector@1";
|
||||
|
||||
export interface PostgresKeyedLookupConnectorConfig extends ConnectorConfig {
|
||||
/** Fully-qualified source table, e.g. "vehiq.vehicle". */
|
||||
table: string;
|
||||
partition_key: string;
|
||||
columns: string[];
|
||||
record_key_column: string;
|
||||
schema: RecordSchemaFields;
|
||||
/**
|
||||
* The ENTIRE set of keys this connector instance is permitted to read —
|
||||
* resolved by the caller BEFORE construction (e.g. distinct
|
||||
* `dealer_stock_entry.vehicle_id` values from one ingestion batch). Never
|
||||
* mutated after connect(); a new batch means a new connector instance.
|
||||
*/
|
||||
keys: ReadonlyArray<string | number>;
|
||||
/** How many keys go into one SQL `= ANY($1)` call. Default 500. */
|
||||
keys_per_query?: number;
|
||||
}
|
||||
|
||||
function isConfig(config: ConnectorConfig): config is PostgresKeyedLookupConnectorConfig {
|
||||
return typeof config.table === "string" && typeof config.partition_key === "string" && Array.isArray(config.keys);
|
||||
}
|
||||
|
||||
/** Cursor vocabulary: the decimal count of keys already looked up (an index into `config.keys`). */
|
||||
function parseCursor(cursor: string | undefined): number {
|
||||
if (cursor === undefined) return 0;
|
||||
const n = Number.parseInt(cursor, 10);
|
||||
if (!Number.isInteger(n) || n < 0) throw new Error(`invalid cursor "${cursor}" — expected a key-list offset`);
|
||||
return n;
|
||||
}
|
||||
|
||||
export class PostgresKeyedLookupConnector implements Connector {
|
||||
readonly connector_id = "postgres_keyed_lookup";
|
||||
readonly connector_version = POSTGRES_KEYED_LOOKUP_CONNECTOR_VERSION;
|
||||
|
||||
private config?: PostgresKeyedLookupConnectorConfig;
|
||||
private ctx?: ConnectorContext;
|
||||
private lastCursor = new Map<string, string | undefined>();
|
||||
private connected = false;
|
||||
|
||||
/** Query calls actually issued, for tests to assert the narrow-scan property against. */
|
||||
readonly queryLog: Array<{ sql: string; keys: Array<string | number> }> = [];
|
||||
|
||||
constructor(private readonly queryImpl: QueryImpl) {}
|
||||
|
||||
validate(config: ConnectorConfig): ValidationResult {
|
||||
const errors: string[] = [];
|
||||
if (typeof config.table !== "string" || config.table.length === 0) errors.push("table is required");
|
||||
if (typeof config.partition_key !== "string" || config.partition_key.length === 0) {
|
||||
errors.push("partition_key is required");
|
||||
}
|
||||
const columns = config.columns;
|
||||
if (!Array.isArray(columns) || columns.length === 0) {
|
||||
errors.push("columns is required and must be a non-empty explicit column list");
|
||||
} else if (columns.some((c) => c === "*" || typeof c !== "string" || c.length === 0)) {
|
||||
errors.push('columns must not contain "*" or an empty entry — SELECT * is not permitted');
|
||||
}
|
||||
if (typeof config.record_key_column !== "string" || config.record_key_column.length === 0) {
|
||||
errors.push("record_key_column is required");
|
||||
} else if (Array.isArray(columns) && !columns.includes(config.record_key_column)) {
|
||||
errors.push("record_key_column must be included in columns");
|
||||
}
|
||||
const schema = config.schema;
|
||||
if (typeof schema !== "object" || schema === null || Object.keys(schema).length === 0) {
|
||||
errors.push("schema is required and must declare at least one field");
|
||||
}
|
||||
if (!Array.isArray(config.keys)) {
|
||||
errors.push("keys is required and must be an array");
|
||||
} else if (config.keys.length === 0) {
|
||||
// Deliberately a hard validation error, not a silent "0 records" read.
|
||||
// A caller with nothing to look up should not construct this connector
|
||||
// at all — this keeps "no keys => no query, ever" true even for a
|
||||
// caller that forgets to check first.
|
||||
errors.push("keys must not be empty — construct this connector only when there is something to look up");
|
||||
}
|
||||
if (config.keys_per_query !== undefined && (typeof config.keys_per_query !== "number" || config.keys_per_query <= 0)) {
|
||||
errors.push("keys_per_query must be a positive number when provided");
|
||||
}
|
||||
return { valid: errors.length === 0, errors };
|
||||
}
|
||||
|
||||
async connect(config: ConnectorConfig, ctx: ConnectorContext): Promise<void> {
|
||||
const verdict = this.validate(config);
|
||||
if (!verdict.valid) {
|
||||
throw new Error(`postgres_keyed_lookup connector config invalid: ${verdict.errors.join("; ")}`);
|
||||
}
|
||||
if (!isConfig(config)) throw new Error("postgres_keyed_lookup connector config invalid");
|
||||
this.config = config;
|
||||
this.ctx = ctx;
|
||||
this.connected = true;
|
||||
}
|
||||
|
||||
async discover(): Promise<DiscoveryResult> {
|
||||
const config = this.requireSession("discover");
|
||||
return {
|
||||
partitions: [{ partition_key: config.partition_key, name: config.table, schema: config.schema }],
|
||||
modes: ["batch"],
|
||||
};
|
||||
}
|
||||
|
||||
async health(): Promise<ConnectorHealth> {
|
||||
const config = this.requireSession("health");
|
||||
const started = Date.now();
|
||||
try {
|
||||
await this.queryImpl(`SELECT 1 FROM ${config.table} LIMIT 0`, []);
|
||||
return { status: "up", latency_ms: Date.now() - started };
|
||||
} catch (err) {
|
||||
return {
|
||||
status: "down",
|
||||
latency_ms: Date.now() - started,
|
||||
detail: err instanceof Error ? err.message : "unreachable",
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
async read(req: ReadRequest): Promise<ReadResult> {
|
||||
const config = this.requireSession("read");
|
||||
const offset = parseCursor(req.cursor);
|
||||
const pageLimit = Math.min(req.limit, config.keys_per_query ?? 500);
|
||||
const page = config.keys.slice(offset, offset + pageLimit);
|
||||
|
||||
let rows: SourceRow[] = [];
|
||||
if (page.length > 0) {
|
||||
const columnList = config.columns.join(", ");
|
||||
const sql = `SELECT ${columnList} FROM ${config.table} WHERE ${config.record_key_column} = ANY($1)`;
|
||||
this.queryLog.push({ sql, keys: [...page] });
|
||||
rows = await this.queryImpl(sql, [page]);
|
||||
}
|
||||
// Structural guarantee, not just an intent: if `page` is empty, `queryImpl`
|
||||
// is never called at all — this branch cannot silently widen into an
|
||||
// unbounded scan even if a future edit forgets the `page.length > 0` guard
|
||||
// above, because the caller of this method never asks for more than
|
||||
// `config.keys.length` total across repeated read()s (see `exhausted`).
|
||||
const records = rows.map((row) => this.toRecord(row, config));
|
||||
const nextOffset = offset + page.length;
|
||||
const nextCursor = String(nextOffset);
|
||||
this.lastCursor.set(req.partition_key, nextCursor);
|
||||
return {
|
||||
records,
|
||||
next_cursor: nextCursor,
|
||||
exhausted: nextOffset >= config.keys.length,
|
||||
};
|
||||
}
|
||||
|
||||
async checkpoint(partitionKey: string): Promise<{ partition_key: string; cursor?: string }> {
|
||||
this.requireSession("checkpoint");
|
||||
return { partition_key: partitionKey, cursor: this.lastCursor.get(partitionKey) };
|
||||
}
|
||||
|
||||
async resume(checkpoint: IngestionCheckpoint, limit: number): Promise<ReadResult> {
|
||||
this.requireSession("resume");
|
||||
return this.read({ partition_key: checkpoint.partition_key, cursor: checkpoint.cursor, limit });
|
||||
}
|
||||
|
||||
/**
|
||||
* "Backfill" for a keyed lookup means re-reading a bounded INDEX range of
|
||||
* the same already-narrow key list — never a fresh, wider key set. `from`/
|
||||
* `to` are decimal offsets into `config.keys`, documented here per the
|
||||
* contract's "bound vocabulary is connector-defined" rule.
|
||||
*/
|
||||
async backfill(req: BackfillRequest): Promise<ReadResult> {
|
||||
const config = this.requireSession("backfill");
|
||||
if (!req.from || !req.to) {
|
||||
throw new Error("backfill requires both `from` and `to` — an unbounded backfill is a full reload, not a backfill");
|
||||
}
|
||||
const from = parseCursor(req.from);
|
||||
const to = parseCursor(req.to);
|
||||
if (to <= from) throw new Error("backfill requires `to` > `from`");
|
||||
const page = config.keys.slice(from, Math.min(to, from + req.limit));
|
||||
let rows: SourceRow[] = [];
|
||||
if (page.length > 0) {
|
||||
const columnList = config.columns.join(", ");
|
||||
const sql = `SELECT ${columnList} FROM ${config.table} WHERE ${config.record_key_column} = ANY($1)`;
|
||||
this.queryLog.push({ sql, keys: [...page] });
|
||||
rows = await this.queryImpl(sql, [page]);
|
||||
}
|
||||
return { records: rows.map((row) => this.toRecord(row, config)), next_cursor: undefined, exhausted: true };
|
||||
}
|
||||
|
||||
async disconnect(): Promise<void> {
|
||||
this.config = undefined;
|
||||
this.ctx = undefined;
|
||||
this.lastCursor.clear();
|
||||
this.connected = false;
|
||||
}
|
||||
|
||||
// --- internals -------------------------------------------------------------
|
||||
|
||||
private toRecord(row: SourceRow, config: PostgresKeyedLookupConnectorConfig): ConnectorRecord {
|
||||
const key = row[config.record_key_column];
|
||||
if (typeof key !== "string" && typeof key !== "number") {
|
||||
throw new Error(`record is missing string/number key column "${config.record_key_column}"`);
|
||||
}
|
||||
// Same real-driver normalization as PostgresWatermarkConnector.toRecord —
|
||||
// a Date-valued column must become an ISO string before it reaches the
|
||||
// P063 schema-conformance gate. See that connector's `normalizeRow` doc
|
||||
// comment for how this was found.
|
||||
const payload: SourceRow = {};
|
||||
for (const [field, value] of Object.entries(row)) {
|
||||
payload[field] = value instanceof Date ? value.toISOString() : value;
|
||||
}
|
||||
return { record_key: String(key), payload, operation: "upsert" };
|
||||
}
|
||||
|
||||
private requireSession(op: string): PostgresKeyedLookupConnectorConfig {
|
||||
if (!this.connected || !this.config) throw new ConnectorNotConnectedError(this.connector_id, op);
|
||||
return this.config;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
/**
|
||||
* src/connectors/postgres-query-impl.ts
|
||||
* ========================================
|
||||
* The REAL `QueryImpl` (see postgres-watermark-connector.ts) — a thin,
|
||||
* read-only wrapper over `postgres` (postgres.js, already a repo dependency
|
||||
* — see src/repositories/db.ts for the sibling pattern this mirrors).
|
||||
*
|
||||
* Used ONLY by scripts/vehiq-oltp-ingest/run.ts and the opt-in realinfra
|
||||
* test. Every connector-level test in tests/connectors/*.test.ts injects a
|
||||
* fixture QueryImpl instead — no real Postgres required to prove the query
|
||||
* shape, the watermark math, or the narrow-scan property.
|
||||
*
|
||||
* DOUBLE-ENFORCED READ-ONLY. The connectors above only ever build SELECT
|
||||
* statements — there is no code path that builds anything else. This
|
||||
* wrapper adds a second, independent check anyway: it refuses to execute
|
||||
* any statement whose first keyword is not SELECT. Belt-and-suspenders is
|
||||
* appropriate here specifically because the thing on the other end is a
|
||||
* separate product's live production OLTP primary, not 0pt's own store.
|
||||
*/
|
||||
import postgres from "postgres";
|
||||
import type { QueryImpl, SourceRow } from "./postgres-watermark-connector";
|
||||
|
||||
export interface PostgresQueryImplOptions {
|
||||
connectionString: string;
|
||||
ssl?: "require" | false;
|
||||
/** Statement timeout (ms), applied per-connection so a slow query cannot pin the source's OLTP primary. Default 5000. */
|
||||
statementTimeoutMs?: number;
|
||||
}
|
||||
|
||||
class NonSelectStatementError extends Error {
|
||||
constructor(sql: string) {
|
||||
super(`refusing to execute a non-SELECT statement against a read-only source connector: ${sql.slice(0, 120)}`);
|
||||
this.name = "NonSelectStatementError";
|
||||
}
|
||||
}
|
||||
|
||||
export function assertReadOnlyStatement(sql: string): void {
|
||||
const firstWord = sql.trim().split(/\s+/, 1)[0]?.toUpperCase();
|
||||
if (firstWord !== "SELECT") throw new NonSelectStatementError(sql);
|
||||
}
|
||||
|
||||
/**
|
||||
* Builds a `QueryImpl` bound to one `postgres` client. The caller owns the
|
||||
* client's lifecycle (open once per orchestration run, `sql.end()` when
|
||||
* done) — this function does not memoize or reuse connections across calls,
|
||||
* unlike `src/repositories/db.ts`'s deliberate memoization for 0pt's own
|
||||
* store, because this one instance is scoped to a single ingestion run
|
||||
* against an EXTERNAL production database and should not outlive it.
|
||||
*/
|
||||
export function makePostgresQueryImpl(opts: PostgresQueryImplOptions): { query: QueryImpl; sql: ReturnType<typeof postgres>; close: () => Promise<void> } {
|
||||
const sql = postgres(opts.connectionString, {
|
||||
prepare: false,
|
||||
idle_timeout: 20,
|
||||
max: 2,
|
||||
ssl: opts.ssl ?? false,
|
||||
connection: {
|
||||
statement_timeout: opts.statementTimeoutMs ?? 5_000,
|
||||
// A narrow, self-identifying application_name — same posture as
|
||||
// http-json-connector.ts's fixed User-Agent: no impersonation, easy
|
||||
// to spot in the source's own pg_stat_activity.
|
||||
application_name: "0pt-vehiq-oltp-extractor",
|
||||
},
|
||||
});
|
||||
const query: QueryImpl = async (text: string, params: unknown[]): Promise<SourceRow[]> => {
|
||||
assertReadOnlyStatement(text);
|
||||
const rows = await sql.unsafe(text, params as never[]);
|
||||
return rows.map((r) => ({ ...r }));
|
||||
};
|
||||
return { query, sql, close: () => sql.end({ timeout: 5 }) };
|
||||
}
|
||||
@@ -0,0 +1,350 @@
|
||||
/**
|
||||
* src/connectors/postgres-watermark-connector.ts
|
||||
* =================================================
|
||||
* P061 connector: a read-only, watermark-polled Postgres OLTP table.
|
||||
*
|
||||
* WHY THIS EXISTS. BI-3/BI-4 need a controlled, incremental path from a
|
||||
* product's live OLTP Postgres into 0pt — without `wal_level=logical`,
|
||||
* without a Postgres restart, without a permanent replica, and without ever
|
||||
* scanning a whole table. `docs/60-data/bi/...` (read from `admin/vehiq`,
|
||||
* 2026-09-25 pass) names the exact shape this connector implements:
|
||||
*
|
||||
* WHERE updated_at > :last_watermark ORDER BY updated_at, id
|
||||
* LIMIT :batch_size
|
||||
*
|
||||
* with a checkpoint of `(source, table, last_updated_at, last_id,
|
||||
* batch_count, checksum, ingested_at)`. This connector produces exactly
|
||||
* that query and that checkpoint — see the field-mapping table in
|
||||
* `docs/P070_VEHIQ_OLTP_EXTRACTOR.md`.
|
||||
*
|
||||
* TIE-BREAK, NOT A DEVIATION. A bare `updated_at > $1` drops rows that share
|
||||
* the exact same `updated_at` as the LAST row of a page: if row 100 and row
|
||||
* 101 both have `updated_at = T`, and row 100 is the last row of a page, a
|
||||
* naive `> T` on the next page skips row 101 forever. The query below adds
|
||||
* the documented `ORDER BY updated_at, id` tie-break to the WHERE clause
|
||||
* too: `updated_at > $1 OR (updated_at = $1 AND id > $2)`. Same watermark
|
||||
* semantics Simon specified, correct at page boundaries — exactly the class
|
||||
* of bug `tests/connectors/postgres-watermark-connector.test.ts`'s
|
||||
* "same-timestamp tie at a page boundary" case exists to catch.
|
||||
*
|
||||
* GUARD RAIL, NOT DECORATION. `validate()` refuses any config whose `table`
|
||||
* is (or ends in) the 98M-row `vehicle` table. That table gets its own
|
||||
* narrow, keyed connector (`postgres-keyed-lookup-connector.ts`) — this one
|
||||
* must never be pointed at it, even by a future caller's typo. The refusal
|
||||
* is asserted in tests, not just documented.
|
||||
*
|
||||
* READ-ONLY, STRUCTURALLY. There is no INSERT/UPDATE/DELETE anywhere in this
|
||||
* file. `columns` is an explicit allow-list the caller declares — never
|
||||
* `SELECT *` — so a schema drift on the source table cannot silently widen
|
||||
* what this connector reads or hands downstream.
|
||||
*
|
||||
* INJECTABLE QUERY IMPL — same seam as `http-json-connector.ts`'s
|
||||
* `fetchImpl`: tests drive real SQL text/params through a fixture function,
|
||||
* no real Postgres required for the connector-level tests. The default
|
||||
* implementation (`makePostgresQueryImpl`) is a thin wrapper over `postgres`
|
||||
* (postgres.js, already a repo dependency), used only by the orchestration
|
||||
* script and the opt-in realinfra test.
|
||||
*/
|
||||
import { createHash } from "node:crypto";
|
||||
import {
|
||||
ConnectorNotConnectedError,
|
||||
type BackfillRequest,
|
||||
type Connector,
|
||||
type ConnectorConfig,
|
||||
type ConnectorContext,
|
||||
type ConnectorHealth,
|
||||
type ConnectorRecord,
|
||||
type DiscoveryResult,
|
||||
type ReadRequest,
|
||||
type ReadResult,
|
||||
type ValidationResult,
|
||||
} from "./connector";
|
||||
import type { IngestionCheckpoint } from "../types/ingestion";
|
||||
import type { RecordSchemaFields } from "../schema/record-schema";
|
||||
|
||||
export const POSTGRES_WATERMARK_CONNECTOR_VERSION = "postgres-watermark-connector@1";
|
||||
|
||||
/** One row back from the source, column name -> raw value. */
|
||||
export type SourceRow = Record<string, unknown>;
|
||||
|
||||
/**
|
||||
* Injectable query seam. `params` are positional (`$1`, `$2`, ...). Tests
|
||||
* supply a fixture; `makePostgresQueryImpl` supplies the real thing.
|
||||
*/
|
||||
export type QueryImpl = (sql: string, params: unknown[]) => Promise<SourceRow[]>;
|
||||
|
||||
export interface PostgresWatermarkConnectorConfig extends ConnectorConfig {
|
||||
/** Fully-qualified source table, e.g. "app.dealer_stock_entry". */
|
||||
table: string;
|
||||
/** Logical partition name exposed to the runner — a dataset name, never a physical detail leak beyond `table` itself. */
|
||||
partition_key: string;
|
||||
/** Explicit column allow-list. Never "*". Must include `record_key_column` and `watermark_column`. */
|
||||
columns: string[];
|
||||
record_key_column: string;
|
||||
watermark_column: string;
|
||||
/** Declared record shape, fed to the schema registry (P063). */
|
||||
schema: RecordSchemaFields;
|
||||
}
|
||||
|
||||
/** The 98M-row table this connector must never be pointed at. See file header. */
|
||||
const FORBIDDEN_TABLE_PATTERN = /(^|\.)vehicle$/i;
|
||||
|
||||
function isConfig(config: ConnectorConfig): config is PostgresWatermarkConnectorConfig {
|
||||
return typeof config.table === "string" && typeof config.partition_key === "string";
|
||||
}
|
||||
|
||||
/** Cursor vocabulary: JSON `{last_updated_at, last_id, checksum}` — see docs/P070 for the checkpoint field mapping. */
|
||||
export interface WatermarkCursor {
|
||||
last_updated_at: string;
|
||||
last_id: string;
|
||||
/** sha256 of the page just produced, hex-encoded. Absent only for the "never read" case. */
|
||||
checksum?: string;
|
||||
}
|
||||
|
||||
export function encodeCursor(cursor: WatermarkCursor): string {
|
||||
return JSON.stringify(cursor);
|
||||
}
|
||||
|
||||
export function decodeCursor(raw: string | undefined): WatermarkCursor | undefined {
|
||||
if (raw === undefined) return undefined;
|
||||
const parsed = JSON.parse(raw) as unknown;
|
||||
if (
|
||||
typeof parsed !== "object" ||
|
||||
parsed === null ||
|
||||
typeof (parsed as WatermarkCursor).last_updated_at !== "string" ||
|
||||
typeof (parsed as WatermarkCursor).last_id !== "string"
|
||||
) {
|
||||
throw new Error(`invalid watermark cursor "${raw}"`);
|
||||
}
|
||||
return parsed as WatermarkCursor;
|
||||
}
|
||||
|
||||
/**
|
||||
* Postgres drivers (postgres.js included) hand back `timestamptz` columns as
|
||||
* native JS `Date` objects, not strings. `String(date)` produces the
|
||||
* locale/timezone-dependent `Date.toString()` form ("Thu Sep 03 2026
|
||||
* 02:00:00 GMT+0200 (...)") — useless as a stored watermark, and NOT what a
|
||||
* fixture-based test (which hands back plain ISO strings) would ever catch.
|
||||
* Found by running this connector against a real Postgres container, not by
|
||||
* the fixture-only unit tests — which is why this helper exists instead of
|
||||
* an inline `String(...)` at each call site.
|
||||
*/
|
||||
function watermarkValueToIso(value: unknown): string {
|
||||
return value instanceof Date ? value.toISOString() : String(value);
|
||||
}
|
||||
|
||||
/**
|
||||
* The P063 schema-conformance gate (src/schema/compatibility.ts) checks a
|
||||
* payload's field VALUES against the declared schema — and even its
|
||||
* `"timestamp"` type requires an already-ISO-string value, never a raw
|
||||
* `Date`. A real Postgres driver hands back `timestamptz`/`date` columns as
|
||||
* `Date` objects, so a payload built straight from `queryImpl`'s rows fails
|
||||
* conformance the moment it reaches a real database — caught by running
|
||||
* this connector against a real Postgres container (see docs/P070's
|
||||
* verification log), not by the fixture-only unit tests, which all use
|
||||
* plain strings. Every row is normalized through this function before it
|
||||
* becomes a `ConnectorRecord` payload, so the envelope's payload is always
|
||||
* JSON-safe and schema-checkable, independent of which driver produced it.
|
||||
*/
|
||||
function normalizeRow(row: SourceRow): SourceRow {
|
||||
const normalized: SourceRow = {};
|
||||
for (const [field, value] of Object.entries(row)) {
|
||||
normalized[field] = value instanceof Date ? value.toISOString() : value;
|
||||
}
|
||||
return normalized;
|
||||
}
|
||||
|
||||
/** sha256 over a stable JSON encoding of the rows, hex-encoded. Deterministic regardless of object key insertion order upstream. */
|
||||
export function checksumRows(rows: SourceRow[]): string {
|
||||
const stable = rows.map((row) =>
|
||||
Object.fromEntries(Object.keys(row).sort().map((k) => [k, row[k]])),
|
||||
);
|
||||
return createHash("sha256").update(JSON.stringify(stable)).digest("hex");
|
||||
}
|
||||
|
||||
export class PostgresWatermarkConnector implements Connector {
|
||||
readonly connector_id = "postgres_watermark";
|
||||
readonly connector_version = POSTGRES_WATERMARK_CONNECTOR_VERSION;
|
||||
|
||||
private config?: PostgresWatermarkConnectorConfig;
|
||||
private ctx?: ConnectorContext;
|
||||
private lastCursor = new Map<string, string | undefined>();
|
||||
private connected = false;
|
||||
|
||||
constructor(private readonly queryImpl: QueryImpl) {}
|
||||
|
||||
validate(config: ConnectorConfig): ValidationResult {
|
||||
const errors: string[] = [];
|
||||
if (typeof config.table !== "string" || config.table.length === 0) {
|
||||
errors.push("table is required");
|
||||
} else if (FORBIDDEN_TABLE_PATTERN.test(config.table)) {
|
||||
errors.push(
|
||||
`table "${config.table}" looks like the 98M-row vehicle catalog table — this connector is for watermark-polled ` +
|
||||
`OLTP tables only; use PostgresKeyedLookupConnector for narrow, keyed vehicle attribute lookups instead`,
|
||||
);
|
||||
}
|
||||
if (typeof config.partition_key !== "string" || config.partition_key.length === 0) {
|
||||
errors.push("partition_key is required");
|
||||
}
|
||||
const columns = config.columns;
|
||||
if (!Array.isArray(columns) || columns.length === 0) {
|
||||
errors.push("columns is required and must be a non-empty explicit column list");
|
||||
} else if (columns.some((c) => c === "*" || typeof c !== "string" || c.length === 0)) {
|
||||
errors.push('columns must not contain "*" or an empty entry — SELECT * is not permitted');
|
||||
}
|
||||
if (typeof config.record_key_column !== "string" || config.record_key_column.length === 0) {
|
||||
errors.push("record_key_column is required");
|
||||
} else if (Array.isArray(columns) && !columns.includes(config.record_key_column)) {
|
||||
errors.push("record_key_column must be included in columns");
|
||||
}
|
||||
if (typeof config.watermark_column !== "string" || config.watermark_column.length === 0) {
|
||||
errors.push("watermark_column is required");
|
||||
} else if (Array.isArray(columns) && !columns.includes(config.watermark_column)) {
|
||||
errors.push("watermark_column must be included in columns");
|
||||
}
|
||||
const schema = config.schema;
|
||||
if (typeof schema !== "object" || schema === null || Object.keys(schema).length === 0) {
|
||||
errors.push("schema is required and must declare at least one field");
|
||||
}
|
||||
return { valid: errors.length === 0, errors };
|
||||
}
|
||||
|
||||
async connect(config: ConnectorConfig, ctx: ConnectorContext): Promise<void> {
|
||||
const verdict = this.validate(config);
|
||||
if (!verdict.valid) {
|
||||
throw new Error(`postgres_watermark connector config invalid: ${verdict.errors.join("; ")}`);
|
||||
}
|
||||
if (!isConfig(config)) throw new Error("postgres_watermark connector config invalid");
|
||||
this.config = config;
|
||||
this.ctx = ctx;
|
||||
this.connected = true;
|
||||
}
|
||||
|
||||
async discover(): Promise<DiscoveryResult> {
|
||||
const config = this.requireSession("discover");
|
||||
return {
|
||||
partitions: [{ partition_key: config.partition_key, name: config.table, schema: config.schema }],
|
||||
// A polled table is batch/stream, never cdc: this connector never emits deletes.
|
||||
modes: ["batch", "stream"],
|
||||
};
|
||||
}
|
||||
|
||||
async health(): Promise<ConnectorHealth> {
|
||||
const config = this.requireSession("health");
|
||||
const started = Date.now();
|
||||
try {
|
||||
await this.queryImpl(`SELECT 1 FROM ${config.table} LIMIT 0`, []);
|
||||
return { status: "up", latency_ms: Date.now() - started };
|
||||
} catch (err) {
|
||||
return {
|
||||
status: "down",
|
||||
latency_ms: Date.now() - started,
|
||||
detail: err instanceof Error ? err.message : "unreachable",
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
async read(req: ReadRequest): Promise<ReadResult> {
|
||||
const config = this.requireSession("read");
|
||||
const cursor = decodeCursor(req.cursor);
|
||||
const columnList = config.columns.join(", ");
|
||||
let rows: SourceRow[];
|
||||
if (cursor === undefined) {
|
||||
rows = await this.queryImpl(
|
||||
`SELECT ${columnList} FROM ${config.table} ORDER BY ${config.watermark_column}, ${config.record_key_column} LIMIT $1`,
|
||||
[req.limit],
|
||||
);
|
||||
} else {
|
||||
rows = await this.queryImpl(
|
||||
`SELECT ${columnList} FROM ${config.table} ` +
|
||||
`WHERE ${config.watermark_column} > $1 OR (${config.watermark_column} = $1 AND ${config.record_key_column} > $2) ` +
|
||||
`ORDER BY ${config.watermark_column}, ${config.record_key_column} LIMIT $3`,
|
||||
[cursor.last_updated_at, cursor.last_id, req.limit],
|
||||
);
|
||||
}
|
||||
// Hard bound — a queryImpl that over-delivers (a fixture bug, or a
|
||||
// future real driver quirk) is truncated here rather than silently
|
||||
// widening the caller's page.
|
||||
rows = rows.slice(0, req.limit);
|
||||
|
||||
const records: ConnectorRecord[] = rows.map((row) => this.toRecord(row, config));
|
||||
let nextCursor = req.cursor;
|
||||
if (rows.length > 0) {
|
||||
const last = rows[rows.length - 1];
|
||||
const nextWatermark: WatermarkCursor = {
|
||||
last_updated_at: watermarkValueToIso(last[config.watermark_column]),
|
||||
last_id: String(last[config.record_key_column]),
|
||||
checksum: checksumRows(rows),
|
||||
};
|
||||
nextCursor = encodeCursor(nextWatermark);
|
||||
this.lastCursor.set(req.partition_key, nextCursor);
|
||||
}
|
||||
return {
|
||||
records,
|
||||
next_cursor: nextCursor,
|
||||
// Fewer rows than the limit means we caught up to "now" — there may be
|
||||
// MORE later (this is a live table), but nothing more RIGHT NOW.
|
||||
exhausted: rows.length < req.limit,
|
||||
};
|
||||
}
|
||||
|
||||
async checkpoint(partitionKey: string): Promise<{ partition_key: string; cursor?: string }> {
|
||||
this.requireSession("checkpoint");
|
||||
return { partition_key: partitionKey, cursor: this.lastCursor.get(partitionKey) };
|
||||
}
|
||||
|
||||
async resume(checkpoint: IngestionCheckpoint, limit: number): Promise<ReadResult> {
|
||||
this.requireSession("resume");
|
||||
return this.read({ partition_key: checkpoint.partition_key, cursor: checkpoint.cursor, limit });
|
||||
}
|
||||
|
||||
async backfill(req: BackfillRequest): Promise<ReadResult> {
|
||||
const config = this.requireSession("backfill");
|
||||
if (!req.from || !req.to) {
|
||||
throw new Error("backfill requires both `from` and `to` — an unbounded backfill is a full reload, not a backfill");
|
||||
}
|
||||
const columnList = config.columns.join(", ");
|
||||
const rows = (
|
||||
await this.queryImpl(
|
||||
`SELECT ${columnList} FROM ${config.table} ` +
|
||||
`WHERE ${config.watermark_column} >= $1 AND ${config.watermark_column} < $2 ` +
|
||||
`ORDER BY ${config.watermark_column}, ${config.record_key_column} LIMIT $3`,
|
||||
[req.from, req.to, req.limit],
|
||||
)
|
||||
).slice(0, req.limit);
|
||||
return {
|
||||
records: rows.map((row) => this.toRecord(row, config)),
|
||||
next_cursor: undefined,
|
||||
exhausted: true,
|
||||
};
|
||||
}
|
||||
|
||||
async disconnect(): Promise<void> {
|
||||
this.config = undefined;
|
||||
this.ctx = undefined;
|
||||
this.lastCursor.clear();
|
||||
this.connected = false;
|
||||
}
|
||||
|
||||
// --- internals -------------------------------------------------------------
|
||||
|
||||
private toRecord(row: SourceRow, config: PostgresWatermarkConnectorConfig): ConnectorRecord {
|
||||
const key = row[config.record_key_column];
|
||||
if (typeof key !== "string" && typeof key !== "number") {
|
||||
throw new Error(`record is missing string/number key column "${config.record_key_column}"`);
|
||||
}
|
||||
const watermark = watermarkValueToIso(row[config.watermark_column]);
|
||||
return {
|
||||
record_key: String(key),
|
||||
payload: normalizeRow(row),
|
||||
operation: "upsert",
|
||||
occurred_at: watermark,
|
||||
cursor: encodeCursor({ last_updated_at: watermark, last_id: String(key) }),
|
||||
};
|
||||
}
|
||||
|
||||
private requireSession(op: string): PostgresWatermarkConnectorConfig {
|
||||
if (!this.connected || !this.config) throw new ConnectorNotConnectedError(this.connector_id, op);
|
||||
return this.config;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,117 @@
|
||||
/**
|
||||
* src/ingestion/projection-sink.ts
|
||||
* ===================================
|
||||
* Adapts the P062 runner's generic `IngestionSink` (envelopes in) onto
|
||||
* `SiaxAnalyticsEngine.ingestProjection` (BI-P009 capability 7, writes
|
||||
* `opt_dataset_projection` in ClickHouse — see `src/analytics/engine/
|
||||
* clickhouse-engine.ts`).
|
||||
*
|
||||
* WHY `ingestProjection` / `opt_dataset_projection` AND NOT `opt_analytics_
|
||||
* event`. `opt_analytics_event` (BI-P010 schema.ts) is documented as a
|
||||
* "high-volume per-tenant EVENT stream" with a 30-day TTL — the right shape
|
||||
* for discrete occurrences (a page view, a search, a status transition).
|
||||
* `dealer_stock_entry`/`listing`/`deal` rows and vehicle-dimension rows are
|
||||
* ENTITY SNAPSHOTS — current state of a stock entry, a listing, a deal, a
|
||||
* vehicle — which is exactly what `opt_dataset_projection` (ReplacingMerge
|
||||
* Tree serving tier, `entity_key`+`version`) already exists to hold, and
|
||||
* `ingestProjection` is the one write port `SiaxAnalyticsEngine` exposes for
|
||||
* it. Landing entity rows into the raw event stream would be forcing an
|
||||
* entity shape through an event shape — this sink does not do that.
|
||||
*
|
||||
* THE FIREWALL IS RESPECTED BY CONSTRUCTION: this file imports only the
|
||||
* `SiaxAnalyticsEngine`/`AnalyticsIdentity`/`ProjectionRow` TYPES from
|
||||
* `../analytics/engine/siax-analytics-engine` — no ClickHouse client, no
|
||||
* `CLICKHOUSE_*` env read, no connection string. Which concrete engine
|
||||
* (ClickHouse today, anything else later) answers `ingestProjection` is the
|
||||
* caller's concern (see `scripts/vehiq-oltp-ingest/run.ts`), not this file's.
|
||||
*
|
||||
* ORDERING STAYS THE RUNNER'S: this function is called AS the runner's
|
||||
* `sink`, so its own crash/partial-failure behaviour is exactly what
|
||||
* `ingestion-runner.ts`'s header already documents — a thrown error here
|
||||
* leaves the page's checkpoint uncommitted, and the next run re-reads it.
|
||||
*/
|
||||
import type { IngestionEnvelope } from "../types/ingestion";
|
||||
import type {
|
||||
AnalyticsIdentity,
|
||||
ProjectionRow,
|
||||
SiaxAnalyticsEngine,
|
||||
} from "../analytics/engine/siax-analytics-engine";
|
||||
import { SiaxError } from "../platform/errors";
|
||||
|
||||
export interface ProjectionSinkOptions {
|
||||
engine: SiaxAnalyticsEngine;
|
||||
identity: AnalyticsIdentity;
|
||||
/** Maps a partition_key (e.g. "dealer_stock_entry") to a logical dataset id (e.g. "vehiq.dealer_stock_entry"). */
|
||||
datasetIdFor: (partitionKey: string) => string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Splits an envelope's payload into ClickHouse-projection-friendly
|
||||
* attributes (strings) and measures (numbers). `null`/`undefined` fields are
|
||||
* dropped rather than coerced to `"null"`/`NaN`, which would fabricate a
|
||||
* value nothing upstream actually asserted.
|
||||
*/
|
||||
export function toProjectionRow(envelope: IngestionEnvelope): ProjectionRow {
|
||||
if (envelope.operation === "delete" || envelope.payload === null) {
|
||||
throw new SiaxError(
|
||||
"INVALID_REQUEST",
|
||||
"0pt.ingestion.projection-sink",
|
||||
`envelope ${envelope.envelope_id} is a delete (payload: null) — opt_dataset_projection has no tombstone ` +
|
||||
`representation via ingestProjection today; a CDC-capable source reaching this sink is a real gap, not ` +
|
||||
`silently handled here (this connector programme currently emits "upsert" only, so this path is untested ` +
|
||||
`by design, not unreachable by contract)`,
|
||||
);
|
||||
}
|
||||
const attributes: Record<string, string> = {};
|
||||
const measures: Record<string, number> = {};
|
||||
for (const [field, value] of Object.entries(envelope.payload)) {
|
||||
if (value === null || value === undefined) continue;
|
||||
if (typeof value === "number" && Number.isFinite(value)) {
|
||||
measures[field] = value;
|
||||
} else if (value instanceof Date) {
|
||||
attributes[field] = value.toISOString();
|
||||
} else if (typeof value === "object") {
|
||||
attributes[field] = JSON.stringify(value);
|
||||
} else {
|
||||
attributes[field] = String(value);
|
||||
}
|
||||
}
|
||||
return {
|
||||
entity_key: envelope.record_key,
|
||||
as_of: envelope.occurred_at ?? envelope.ingested_at,
|
||||
version: envelope.sequence,
|
||||
attributes,
|
||||
measures,
|
||||
};
|
||||
}
|
||||
|
||||
export function projectionSink(
|
||||
opts: ProjectionSinkOptions,
|
||||
): (envelopes: IngestionEnvelope[]) => Promise<void> {
|
||||
return async (envelopes: IngestionEnvelope[]): Promise<void> => {
|
||||
if (envelopes.length === 0) return;
|
||||
const byPartition = new Map<string, IngestionEnvelope[]>();
|
||||
for (const envelope of envelopes) {
|
||||
const list = byPartition.get(envelope.partition_key) ?? [];
|
||||
list.push(envelope);
|
||||
byPartition.set(envelope.partition_key, list);
|
||||
}
|
||||
for (const [partitionKey, group] of byPartition) {
|
||||
const rows = group.map(toProjectionRow);
|
||||
const dataset_id = opts.datasetIdFor(partitionKey);
|
||||
const result = await opts.engine.ingestProjection(opts.identity, { dataset_id, rows });
|
||||
if (result.ingested !== rows.length) {
|
||||
// Partial ingest is a hard failure, on purpose: the runner only
|
||||
// commits the checkpoint AFTER this sink resolves. Resolving
|
||||
// "successfully" on a partial write would silently drop the
|
||||
// un-ingested rows on the next run's resume.
|
||||
throw new SiaxError(
|
||||
"RETRYABLE_FAILURE",
|
||||
"0pt.ingestion.projection-sink",
|
||||
`ingestProjection for dataset "${dataset_id}" reported ${result.ingested}/${rows.length} rows ingested — ` +
|
||||
`treating as a failed page so the checkpoint does not advance past unlanded rows`,
|
||||
);
|
||||
}
|
||||
}
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,343 @@
|
||||
/**
|
||||
* src/ingestion/vehiq-oltp-ingest-runner.ts
|
||||
* ============================================
|
||||
* BI-3/BI-4 — the controlled, incremental VehIQ OLTP -> 0pt ClickHouse path,
|
||||
* as Simon specified it (`admin/vehiq` 2026-09-25 pass, "Data path for
|
||||
* inventory.days_in_stock"):
|
||||
*
|
||||
* OLTP operational tables -> read-only analytical extraction ->
|
||||
* watermark/updated_at -> 0pt ingestion -> ClickHouse
|
||||
*
|
||||
* WHY THIS LIVES IN `admin/0pt`, NOT `admin/vehiq`. ADR-0035 (admin/vehiq,
|
||||
* decisions 1/2/10, CI-enforced by `tools/validate_engine_firewall.py`):
|
||||
* VehIQ's own codebase must NEVER hold a ClickHouse credential or import a
|
||||
* ClickHouse client, and product code must never hammer the live OLTP
|
||||
* primary with analytical queries directly. `admin/vehiq`'s own hub
|
||||
* (`docs/CURRENT_STATE.md`, "Connection details for a future API0
|
||||
* connector") says this explicitly: "VehIQ's own codebase must never see
|
||||
* any of this. This is 0pt's data plane; only a future `admin/api0`
|
||||
* connector should hold these details." This repo is 0pt's data plane —
|
||||
* the correct home per that same guidance (a sibling of `admin/api0`, not
|
||||
* `admin/api0` itself, since the read/write mechanics belong next to the
|
||||
* P060-P068 ingestion plane and the BI-P009/P010 ClickHouse engine that
|
||||
* already live here).
|
||||
*
|
||||
* TWO ENTRY POINTS, DELIBERATELY SEPARATE:
|
||||
*
|
||||
* `previewVehiqOltpSource` — connect, discover, ONE bounded read. Never
|
||||
* calls checkpoint()/commits anything, never runs the P062 runner. Safe
|
||||
* to point at production at any time, table empty or not: nothing it does
|
||||
* can advance a checkpoint or land a row. This is what proves "zero
|
||||
* batches, zero writes" without deploying a live scheduled service.
|
||||
*
|
||||
* `runVehiqOltpIngestion` — the real, scheduled path: the P062
|
||||
* `IngestionRunner` across all three OLTP sources, then the narrow
|
||||
* `dimension_vehicle` lookup keyed on exactly the `vehicle_id`s the OLTP
|
||||
* batch referenced. This is what `.gitea/workflows/vehiq-oltp-ingest.yml`
|
||||
* runs — see that file and docs/P070_VEHIQ_OLTP_EXTRACTOR.md for the
|
||||
* activation gate this stays behind until Simon flips it.
|
||||
*
|
||||
* CHECKPOINT FIELD MAPPING (Simon's own list: source, table, last_updated_at,
|
||||
* last_id, batch_count, checksum, ingested_at) onto the canonical P062
|
||||
* `IngestionCheckpoint` (source_id, tenant_id, partition_key, cursor,
|
||||
* sequence, records_committed, committed_at) — deliberately reusing the
|
||||
* existing durable checkpoint mechanism rather than building a second one:
|
||||
*
|
||||
* source -> checkpoint.source_id
|
||||
* table -> checkpoint.partition_key
|
||||
* last_updated_at-> decoded from checkpoint.cursor (WatermarkCursor)
|
||||
* last_id -> decoded from checkpoint.cursor (WatermarkCursor)
|
||||
* checksum -> decoded from checkpoint.cursor (WatermarkCursor)
|
||||
* batch_count -> checkpoint.records_committed (cumulative) / a
|
||||
* per-run page count is also in each PartitionRunReport
|
||||
* ingested_at -> checkpoint.committed_at
|
||||
*/
|
||||
import type { IngestionRepository } from "../repositories/ingestion-repository";
|
||||
import type { SchemaRegistry } from "../schema/schema-registry";
|
||||
import type { LineageStore } from "../lineage/lineage-store";
|
||||
import type { RecordSchemaFields } from "../schema/record-schema";
|
||||
import { IngestionRunner, type IngestionRunReport, type IngestionSink } from "./ingestion-runner";
|
||||
import {
|
||||
PostgresWatermarkConnector,
|
||||
decodeCursor,
|
||||
type QueryImpl,
|
||||
} from "../connectors/postgres-watermark-connector";
|
||||
import { PostgresKeyedLookupConnector } from "../connectors/postgres-keyed-lookup-connector";
|
||||
import type { IngestionEnvelope } from "../types/ingestion";
|
||||
|
||||
export interface VehiqOltpSourceDef {
|
||||
/** e.g. "dealer_stock_entry" — also the logical dataset name suffix. */
|
||||
partition_key: "dealer_stock_entry" | "deal";
|
||||
/** Fully-qualified source table, e.g. "app.dealer_stock_entry". */
|
||||
table: string;
|
||||
columns: string[];
|
||||
record_key_column: string;
|
||||
watermark_column: string;
|
||||
schema: RecordSchemaFields;
|
||||
}
|
||||
|
||||
/**
|
||||
* TWO sources, not three — READ THIS BEFORE ASSUMING `app.listing` IS
|
||||
* COVERED. The task's own wording (and `admin/vehiq`'s `docs/CURRENT_STATE.
|
||||
* md`, "the columns already exist") assumed all three OLTP tables carry a
|
||||
* generic `updated_at`. Verified LIVE against production (`server10`,
|
||||
* `vehiq-db`, read-only `information_schema.columns`, 2026-09-25) that
|
||||
* this is only true for none of the three as a literal `updated_at` column
|
||||
* — and NOT true at all for `app.listing`:
|
||||
*
|
||||
* app.dealer_stock_entry — id, tenant_id, vehicle_id, lot_location,
|
||||
* floorplan_status, listed_at, sold_at. No `updated_at`, no `status`
|
||||
* (it's `floorplan_status`). `listed_at` (NOT NULL) is the best
|
||||
* available watermark — see the caveat below.
|
||||
* app.deal — id, tenant_id, party_id, vehicle_id, sale_price_minor,
|
||||
* currency, status, created_at. No `updated_at`. `created_at`
|
||||
* (NOT NULL DEFAULT now()) is the best available watermark.
|
||||
* app.listing — id, tenant_id, vehicle_id, asking_price_minor, currency,
|
||||
* channels, status. NO timestamp column of any kind exists today.
|
||||
* `admin/vehiq` migration 0084 (`published_at`/`unpublished_at`) is
|
||||
* written but NOT YET APPLIED to production (verified: those columns
|
||||
* are absent live) — until it lands, there is no honest way to build a
|
||||
* watermark connector for `app.listing` without inventing a column that
|
||||
* does not exist. Not built here — see docs/P070's "Deferred: app.
|
||||
* listing extraction" section for exactly what unblocks it.
|
||||
*
|
||||
* HONEST CAVEAT ON THE TWO SOURCES THAT ARE HERE: `listed_at`/`created_at`
|
||||
* are set ONCE, at insert. Neither table has a column that bumps when an
|
||||
* EXISTING row changes (`sold_at`/`floorplan_status` on dealer_stock_entry;
|
||||
* `status` on deal). This connector will therefore reliably see every NEW
|
||||
* row, but will NOT see an in-place UPDATE to an existing row as "changed" —
|
||||
* it has already passed the watermark and will not be re-read. This is the
|
||||
* same gap a real `updated_at` column (plus a trigger, e.g. `0084`-style)
|
||||
* would close, and is exactly why building a permanent extractor deserves
|
||||
* that schema decision rather than silently working around its absence.
|
||||
* `docs/P070_VEHIQ_OLTP_EXTRACTOR.md` documents this as the #1 thing to
|
||||
* settle before leaning on this extractor for anything beyond "count of new
|
||||
* rows since X".
|
||||
*/
|
||||
export const VEHIQ_OLTP_SOURCES: VehiqOltpSourceDef[] = [
|
||||
{
|
||||
partition_key: "dealer_stock_entry",
|
||||
table: "app.dealer_stock_entry",
|
||||
columns: ["id", "tenant_id", "vehicle_id", "lot_location", "floorplan_status", "listed_at", "sold_at"],
|
||||
record_key_column: "id",
|
||||
watermark_column: "listed_at",
|
||||
schema: {
|
||||
id: { type: "string", required: true },
|
||||
tenant_id: { type: "string", required: true },
|
||||
vehicle_id: { type: "string", required: true },
|
||||
lot_location: { type: "string", required: true },
|
||||
floorplan_status: { type: "string", required: true },
|
||||
listed_at: { type: "timestamp", required: true },
|
||||
sold_at: { type: "timestamp", required: false },
|
||||
},
|
||||
},
|
||||
{
|
||||
partition_key: "deal",
|
||||
table: "app.deal",
|
||||
columns: ["id", "tenant_id", "party_id", "vehicle_id", "sale_price_minor", "currency", "status", "created_at"],
|
||||
record_key_column: "id",
|
||||
watermark_column: "created_at",
|
||||
schema: {
|
||||
id: { type: "string", required: true },
|
||||
tenant_id: { type: "string", required: true },
|
||||
party_id: { type: "string", required: true },
|
||||
vehicle_id: { type: "string", required: true },
|
||||
sale_price_minor: { type: "number", required: true },
|
||||
currency: { type: "string", required: true },
|
||||
status: { type: "string", required: true },
|
||||
created_at: { type: "timestamp", required: true },
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
/**
|
||||
* `vehiq.vehicle`'s primary key column is `vehicle_id`, NOT `id` — verified
|
||||
* live (information_schema.columns, read-only, no data scanned) 2026-09-25.
|
||||
* Column list otherwise matches what BI-P010/VEHIQ_ANALYTICS_BASELINE.md
|
||||
* already documented (make/model/model_year/fuel_type all present and
|
||||
* correctly named).
|
||||
*/
|
||||
export const VEHIQ_VEHICLE_DIMENSION_TABLE = "vehiq.vehicle";
|
||||
export const VEHIQ_VEHICLE_DIMENSION_RECORD_KEY_COLUMN = "vehicle_id";
|
||||
export const VEHIQ_VEHICLE_DIMENSION_COLUMNS = ["vehicle_id", "make", "model", "model_year", "fuel_type"];
|
||||
export const VEHIQ_VEHICLE_DIMENSION_SCHEMA: RecordSchemaFields = {
|
||||
vehicle_id: { type: "string", required: true },
|
||||
make: { type: "string", required: false },
|
||||
model: { type: "string", required: false },
|
||||
model_year: { type: "number", required: false },
|
||||
fuel_type: { type: "string", required: false },
|
||||
};
|
||||
|
||||
// --- preview: read-only, never commits, safe against production at any time ---
|
||||
|
||||
export interface PreviewResult {
|
||||
partition_key: string;
|
||||
table: string;
|
||||
/** Records this ONE bounded read would produce right now. */
|
||||
would_read: number;
|
||||
exhausted: boolean;
|
||||
checkpoint_before: string | undefined;
|
||||
next_cursor: string | undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* ONE bounded connect -> discover -> read(existing checkpoint, limit), then
|
||||
* disconnect. Never calls IngestionRunner, never calls connector.checkpoint()
|
||||
* against the repo, never writes anything anywhere. This is intentionally
|
||||
* the ONLY function in this file that is safe to point at a real production
|
||||
* source before the extractor is "activated" (see docs/P070).
|
||||
*/
|
||||
export async function previewVehiqOltpSource(
|
||||
source: VehiqOltpSourceDef,
|
||||
query: QueryImpl,
|
||||
repo: Pick<IngestionRepository, "getCheckpoint">,
|
||||
opts: { sourceId: string; tenantId: string; projectId: string; limit?: number } = {
|
||||
sourceId: "vehiq_oltp",
|
||||
tenantId: "t_vehiq",
|
||||
projectId: "vehiq",
|
||||
},
|
||||
): Promise<PreviewResult> {
|
||||
const connector = new PostgresWatermarkConnector(query);
|
||||
const config = {
|
||||
table: source.table,
|
||||
partition_key: source.partition_key,
|
||||
columns: source.columns,
|
||||
record_key_column: source.record_key_column,
|
||||
watermark_column: source.watermark_column,
|
||||
schema: source.schema,
|
||||
};
|
||||
await connector.connect(config, {
|
||||
tenant_id: opts.tenantId,
|
||||
project_id: opts.projectId,
|
||||
source_id: opts.sourceId,
|
||||
run_id: `preview_${Date.now()}`,
|
||||
});
|
||||
try {
|
||||
const existing = await repo.getCheckpoint(opts.sourceId, source.partition_key);
|
||||
const result = await connector.read({
|
||||
partition_key: source.partition_key,
|
||||
cursor: existing?.cursor,
|
||||
limit: opts.limit ?? 100,
|
||||
});
|
||||
return {
|
||||
partition_key: source.partition_key,
|
||||
table: source.table,
|
||||
would_read: result.records.length,
|
||||
exhausted: result.exhausted,
|
||||
checkpoint_before: existing?.cursor,
|
||||
next_cursor: result.next_cursor,
|
||||
};
|
||||
} finally {
|
||||
await connector.disconnect();
|
||||
}
|
||||
}
|
||||
|
||||
// --- real run: the P062 highway, scheduled/activated path ---------------------
|
||||
|
||||
export interface VehiqOltpIngestDeps {
|
||||
repo: IngestionRepository;
|
||||
schemas: SchemaRegistry;
|
||||
lineage?: LineageStore;
|
||||
/** Read-only query against VehIQ's `vehiq-db` (server10) — see makePostgresQueryImpl in the calling script for the real implementation. */
|
||||
sourceQuery: QueryImpl;
|
||||
/** Where OLTP-entity envelopes land (see src/ingestion/projection-sink.ts). */
|
||||
oltpSink: IngestionSink;
|
||||
/** Where the narrow vehicle-dimension envelopes land. Same sink shape; kept separate so the two datasets can be routed differently if ever needed. */
|
||||
dimensionSink: IngestionSink;
|
||||
tenantId: string;
|
||||
projectId: string;
|
||||
sourceIdPrefix?: string;
|
||||
pageSize?: number;
|
||||
maxPages?: number;
|
||||
}
|
||||
|
||||
export interface VehiqOltpIngestSummary {
|
||||
oltp: IngestionRunReport[];
|
||||
dimension?: IngestionRunReport;
|
||||
referenced_vehicle_ids: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs the three watermark sources through the P062 runner, then derives the
|
||||
* narrow vehicle-dimension key set from exactly what THIS run's
|
||||
* `dealer_stock_entry` batch referenced (never a fresh scan, never the full
|
||||
* 98M-row table) and runs that through the runner too, if there is anything
|
||||
* to look up. When there is nothing new (today's actual production state:
|
||||
* all three OLTP tables are empty), this function runs, reports 0 records
|
||||
* read for every source, calls the dimension sink zero times, and returns
|
||||
* cleanly — the empty-table case is a normal, successful, no-op run, not a
|
||||
* special case that needs its own error handling.
|
||||
*/
|
||||
export async function runVehiqOltpIngestion(deps: VehiqOltpIngestDeps): Promise<VehiqOltpIngestSummary> {
|
||||
const runner = new IngestionRunner(deps.repo, deps.schemas, deps.lineage);
|
||||
const sourceIdPrefix = deps.sourceIdPrefix ?? "vehiq_oltp";
|
||||
const oltpReports: IngestionRunReport[] = [];
|
||||
const referencedVehicleIds = new Set<string>();
|
||||
|
||||
for (const source of VEHIQ_OLTP_SOURCES) {
|
||||
const seen: IngestionEnvelope[] = [];
|
||||
const wrappedSink: IngestionSink = async (envelopes) => {
|
||||
seen.push(...envelopes);
|
||||
await deps.oltpSink(envelopes);
|
||||
};
|
||||
const connector = new PostgresWatermarkConnector(deps.sourceQuery);
|
||||
const report = await runner.run({
|
||||
source_id: `${sourceIdPrefix}_${source.partition_key}`,
|
||||
tenant_id: deps.tenantId,
|
||||
project_id: deps.projectId,
|
||||
mode: "batch",
|
||||
connector,
|
||||
config: {
|
||||
table: source.table,
|
||||
partition_key: source.partition_key,
|
||||
columns: source.columns,
|
||||
record_key_column: source.record_key_column,
|
||||
watermark_column: source.watermark_column,
|
||||
schema: source.schema,
|
||||
},
|
||||
sink: wrappedSink,
|
||||
page_size: deps.pageSize ?? 200,
|
||||
max_pages: deps.maxPages ?? 1000,
|
||||
output_ref: `0pt:dataset:vehiq.${source.partition_key}`,
|
||||
});
|
||||
oltpReports.push(report);
|
||||
|
||||
if (source.partition_key === "dealer_stock_entry") {
|
||||
for (const envelope of seen) {
|
||||
const vehicleId = envelope.payload?.["vehicle_id"];
|
||||
if (typeof vehicleId === "string" && vehicleId.length > 0) referencedVehicleIds.add(vehicleId);
|
||||
else if (typeof vehicleId === "number") referencedVehicleIds.add(String(vehicleId));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let dimensionReport: IngestionRunReport | undefined;
|
||||
if (referencedVehicleIds.size > 0) {
|
||||
const keys = [...referencedVehicleIds];
|
||||
const dimensionConnector = new PostgresKeyedLookupConnector(deps.sourceQuery);
|
||||
dimensionReport = await runner.run({
|
||||
source_id: `${sourceIdPrefix}_dimension_vehicle`,
|
||||
tenant_id: deps.tenantId,
|
||||
project_id: deps.projectId,
|
||||
mode: "batch",
|
||||
connector: dimensionConnector,
|
||||
config: {
|
||||
table: VEHIQ_VEHICLE_DIMENSION_TABLE,
|
||||
partition_key: "dimension_vehicle",
|
||||
columns: VEHIQ_VEHICLE_DIMENSION_COLUMNS,
|
||||
record_key_column: VEHIQ_VEHICLE_DIMENSION_RECORD_KEY_COLUMN,
|
||||
schema: VEHIQ_VEHICLE_DIMENSION_SCHEMA,
|
||||
keys,
|
||||
},
|
||||
sink: deps.dimensionSink,
|
||||
page_size: deps.pageSize ?? 200,
|
||||
max_pages: deps.maxPages ?? 1000,
|
||||
output_ref: "0pt:dataset:vehiq.dimension_vehicle",
|
||||
});
|
||||
}
|
||||
|
||||
return { oltp: oltpReports, dimension: dimensionReport, referenced_vehicle_ids: referencedVehicleIds.size };
|
||||
}
|
||||
|
||||
/** Re-exported so callers (the CLI script, tests) don't need a second import from the connector module. */
|
||||
export { decodeCursor };
|
||||
Reference in New Issue
Block a user