feat(ingestion): VehIQ OLTP -> ClickHouse watermark extractor (P070)
CI (SIAX Cloud) / integration (push) Skipped
masterplan-lock / check (push) Successful in 7s
CI (SIAX Cloud) / security (push) Successful in 17s
masterplan-lock / check (pull_request) Successful in 9s
Security Scan (SIAX Cloud) / security (push) Successful in 38s
CI (SIAX Cloud) / security (pull_request) Successful in 26s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 26s
CI (SIAX Cloud) / quality (push) Successful in 57s
CI (SIAX Cloud) / integration (pull_request) Successful in 39s
CI (SIAX Cloud) / quality (pull_request) Successful in 59s

Implements Simon's specified pattern (dealer_stock_entry/deal ->
watermark/updated_at -> 0pt ingestion -> ClickHouse) as a P061 connector
pair (PostgresWatermarkConnector, PostgresKeyedLookupConnector) plugged
into the existing P062 IngestionRunner, writing to opt_dataset_projection
via SiaxAnalyticsEngine.ingestProjection.

Built here, not in admin/vehiq: that repo's own ADR-0035 (decisions 1/2/10,
CI-enforced by validate_engine_firewall.py) forbids VehIQ code from ever
holding a ClickHouse credential or hammering OLTP directly for analytics.
See docs/P070_VEHIQ_OLTP_EXTRACTOR.md for the full rationale.

Also corrects a real schema mismatch found while building this: production
has no generic `updated_at` on any of the three target tables (verified
live, read-only, 2026-09-25). dealer_stock_entry/deal use listed_at/
created_at as insert-time watermarks instead (documented caveat: in-place
updates are not caught without a schema change); app.listing has no
timestamp column at all and is deliberately not covered yet.

Vehicle dimension lookup is narrow by construction: keyed on exactly the
vehicle_id set referenced by the batch just read, never a scan of the
98M-row vehiq.vehicle table (connector-level validate() refuses that table
by name; runner-level tests assert the query only ever touches referenced
IDs).

Verified: fixture-backed connector/runner tests (watermark+limit,
same-timestamp tie at a page boundary, crash/resume, empty-table case,
narrow-lookup query-count assertions) + a real disposable Postgres
container (caught two real Date-handling bugs fixtures could not) + a
read-only proof against actual production (server10, vehiq-db): the
extractor's exact SQL returns zero rows, no error, confirming it is safe
to schedule with zero current effect.

.gitea/workflows/vehiq-oltp-ingest.yml is present, cron-scheduled, and
defaults to VEHIQ_INGEST_MODE=preview (read-only, never writes) with no
secrets configured — safe to merge with zero production effect until
explicitly activated per the doc's "what activating this requires".

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Claude Code
2026-09-25 19:11:15 +02:00
co-authored by Claude Sonnet 5
parent ec487746fd
commit 07158099ee
13 changed files with 1995 additions and 11 deletions
@@ -0,0 +1,258 @@
/**
* src/connectors/postgres-keyed-lookup-connector.ts
* ====================================================
* P061 connector: a narrow, explicit-key-list lookup against a large table —
* built specifically so `0pt dimension_vehicle` never becomes `SELECT *
* FROM vehicle` against the 98M-row/85GB `vehiq.vehicle` table.
*
* THE RULE THIS ENFORCES (Simon's decision #2, `admin/vehiq` 2026-09-25
* pass): `dealer_stock_entry.vehicle_id` -> unique vehicle IDs -> a
* TARGETED lookup of just those rows' attributes -> `dimension_vehicle`.
* Never a full scan, never an unbounded WHERE clause, never "give me
* everything and I'll filter downstream".
*
* HOW IT'S STRUCTURALLY DIFFERENT FROM `postgres-watermark-connector.ts`,
* not just configured differently: this connector's config carries `keys`
* — a caller-supplied, already-resolved, finite list of record keys. There
* is no watermark, no `ORDER BY updated_at`, no notion of "everything since
* X". `read()` can ONLY ever emit rows whose key is in that list, and
* `connect()` REFUSES an empty or missing key list rather than defaulting
* to "read everything" — see the "zero keys issues zero queries" test,
* which is the assertion that actually proves the narrow-scan property
* (not just that the SQL text LOOKS bounded).
*
* THE ORCHESTRATOR (scripts/vehiq-oltp-ingest/run.ts) is what supplies
* `keys`: the DISTINCT `vehicle_id`s seen in the `dealer_stock_entry` batch
* this connector instance was built for, and nothing else. A full snapshot/
* backfill of the whole `vehicle` table is a deliberate, separate, later
* decision — see docs/P070_VEHIQ_OLTP_EXTRACTOR.md's "Deferred: full
* vehicle dimension backfill" section. This connector does not build
* towards that; it is the permanent narrow path, not a stepping stone.
*/
import {
ConnectorNotConnectedError,
type BackfillRequest,
type Connector,
type ConnectorConfig,
type ConnectorContext,
type ConnectorHealth,
type ConnectorRecord,
type DiscoveryResult,
type ReadRequest,
type ReadResult,
type ValidationResult,
} from "./connector";
import type { IngestionCheckpoint } from "../types/ingestion";
import type { RecordSchemaFields } from "../schema/record-schema";
import type { QueryImpl, SourceRow } from "./postgres-watermark-connector";
export const POSTGRES_KEYED_LOOKUP_CONNECTOR_VERSION = "postgres-keyed-lookup-connector@1";
export interface PostgresKeyedLookupConnectorConfig extends ConnectorConfig {
/** Fully-qualified source table, e.g. "vehiq.vehicle". */
table: string;
partition_key: string;
columns: string[];
record_key_column: string;
schema: RecordSchemaFields;
/**
* The ENTIRE set of keys this connector instance is permitted to read —
* resolved by the caller BEFORE construction (e.g. distinct
* `dealer_stock_entry.vehicle_id` values from one ingestion batch). Never
* mutated after connect(); a new batch means a new connector instance.
*/
keys: ReadonlyArray<string | number>;
/** How many keys go into one SQL `= ANY($1)` call. Default 500. */
keys_per_query?: number;
}
function isConfig(config: ConnectorConfig): config is PostgresKeyedLookupConnectorConfig {
return typeof config.table === "string" && typeof config.partition_key === "string" && Array.isArray(config.keys);
}
/** Cursor vocabulary: the decimal count of keys already looked up (an index into `config.keys`). */
function parseCursor(cursor: string | undefined): number {
if (cursor === undefined) return 0;
const n = Number.parseInt(cursor, 10);
if (!Number.isInteger(n) || n < 0) throw new Error(`invalid cursor "${cursor}" — expected a key-list offset`);
return n;
}
export class PostgresKeyedLookupConnector implements Connector {
readonly connector_id = "postgres_keyed_lookup";
readonly connector_version = POSTGRES_KEYED_LOOKUP_CONNECTOR_VERSION;
private config?: PostgresKeyedLookupConnectorConfig;
private ctx?: ConnectorContext;
private lastCursor = new Map<string, string | undefined>();
private connected = false;
/** Query calls actually issued, for tests to assert the narrow-scan property against. */
readonly queryLog: Array<{ sql: string; keys: Array<string | number> }> = [];
constructor(private readonly queryImpl: QueryImpl) {}
validate(config: ConnectorConfig): ValidationResult {
const errors: string[] = [];
if (typeof config.table !== "string" || config.table.length === 0) errors.push("table is required");
if (typeof config.partition_key !== "string" || config.partition_key.length === 0) {
errors.push("partition_key is required");
}
const columns = config.columns;
if (!Array.isArray(columns) || columns.length === 0) {
errors.push("columns is required and must be a non-empty explicit column list");
} else if (columns.some((c) => c === "*" || typeof c !== "string" || c.length === 0)) {
errors.push('columns must not contain "*" or an empty entry — SELECT * is not permitted');
}
if (typeof config.record_key_column !== "string" || config.record_key_column.length === 0) {
errors.push("record_key_column is required");
} else if (Array.isArray(columns) && !columns.includes(config.record_key_column)) {
errors.push("record_key_column must be included in columns");
}
const schema = config.schema;
if (typeof schema !== "object" || schema === null || Object.keys(schema).length === 0) {
errors.push("schema is required and must declare at least one field");
}
if (!Array.isArray(config.keys)) {
errors.push("keys is required and must be an array");
} else if (config.keys.length === 0) {
// Deliberately a hard validation error, not a silent "0 records" read.
// A caller with nothing to look up should not construct this connector
// at all — this keeps "no keys => no query, ever" true even for a
// caller that forgets to check first.
errors.push("keys must not be empty — construct this connector only when there is something to look up");
}
if (config.keys_per_query !== undefined && (typeof config.keys_per_query !== "number" || config.keys_per_query <= 0)) {
errors.push("keys_per_query must be a positive number when provided");
}
return { valid: errors.length === 0, errors };
}
async connect(config: ConnectorConfig, ctx: ConnectorContext): Promise<void> {
const verdict = this.validate(config);
if (!verdict.valid) {
throw new Error(`postgres_keyed_lookup connector config invalid: ${verdict.errors.join("; ")}`);
}
if (!isConfig(config)) throw new Error("postgres_keyed_lookup connector config invalid");
this.config = config;
this.ctx = ctx;
this.connected = true;
}
async discover(): Promise<DiscoveryResult> {
const config = this.requireSession("discover");
return {
partitions: [{ partition_key: config.partition_key, name: config.table, schema: config.schema }],
modes: ["batch"],
};
}
async health(): Promise<ConnectorHealth> {
const config = this.requireSession("health");
const started = Date.now();
try {
await this.queryImpl(`SELECT 1 FROM ${config.table} LIMIT 0`, []);
return { status: "up", latency_ms: Date.now() - started };
} catch (err) {
return {
status: "down",
latency_ms: Date.now() - started,
detail: err instanceof Error ? err.message : "unreachable",
};
}
}
async read(req: ReadRequest): Promise<ReadResult> {
const config = this.requireSession("read");
const offset = parseCursor(req.cursor);
const pageLimit = Math.min(req.limit, config.keys_per_query ?? 500);
const page = config.keys.slice(offset, offset + pageLimit);
let rows: SourceRow[] = [];
if (page.length > 0) {
const columnList = config.columns.join(", ");
const sql = `SELECT ${columnList} FROM ${config.table} WHERE ${config.record_key_column} = ANY($1)`;
this.queryLog.push({ sql, keys: [...page] });
rows = await this.queryImpl(sql, [page]);
}
// Structural guarantee, not just an intent: if `page` is empty, `queryImpl`
// is never called at all — this branch cannot silently widen into an
// unbounded scan even if a future edit forgets the `page.length > 0` guard
// above, because the caller of this method never asks for more than
// `config.keys.length` total across repeated read()s (see `exhausted`).
const records = rows.map((row) => this.toRecord(row, config));
const nextOffset = offset + page.length;
const nextCursor = String(nextOffset);
this.lastCursor.set(req.partition_key, nextCursor);
return {
records,
next_cursor: nextCursor,
exhausted: nextOffset >= config.keys.length,
};
}
async checkpoint(partitionKey: string): Promise<{ partition_key: string; cursor?: string }> {
this.requireSession("checkpoint");
return { partition_key: partitionKey, cursor: this.lastCursor.get(partitionKey) };
}
async resume(checkpoint: IngestionCheckpoint, limit: number): Promise<ReadResult> {
this.requireSession("resume");
return this.read({ partition_key: checkpoint.partition_key, cursor: checkpoint.cursor, limit });
}
/**
* "Backfill" for a keyed lookup means re-reading a bounded INDEX range of
* the same already-narrow key list — never a fresh, wider key set. `from`/
* `to` are decimal offsets into `config.keys`, documented here per the
* contract's "bound vocabulary is connector-defined" rule.
*/
async backfill(req: BackfillRequest): Promise<ReadResult> {
const config = this.requireSession("backfill");
if (!req.from || !req.to) {
throw new Error("backfill requires both `from` and `to` — an unbounded backfill is a full reload, not a backfill");
}
const from = parseCursor(req.from);
const to = parseCursor(req.to);
if (to <= from) throw new Error("backfill requires `to` > `from`");
const page = config.keys.slice(from, Math.min(to, from + req.limit));
let rows: SourceRow[] = [];
if (page.length > 0) {
const columnList = config.columns.join(", ");
const sql = `SELECT ${columnList} FROM ${config.table} WHERE ${config.record_key_column} = ANY($1)`;
this.queryLog.push({ sql, keys: [...page] });
rows = await this.queryImpl(sql, [page]);
}
return { records: rows.map((row) => this.toRecord(row, config)), next_cursor: undefined, exhausted: true };
}
async disconnect(): Promise<void> {
this.config = undefined;
this.ctx = undefined;
this.lastCursor.clear();
this.connected = false;
}
// --- internals -------------------------------------------------------------
private toRecord(row: SourceRow, config: PostgresKeyedLookupConnectorConfig): ConnectorRecord {
const key = row[config.record_key_column];
if (typeof key !== "string" && typeof key !== "number") {
throw new Error(`record is missing string/number key column "${config.record_key_column}"`);
}
// Same real-driver normalization as PostgresWatermarkConnector.toRecord —
// a Date-valued column must become an ISO string before it reaches the
// P063 schema-conformance gate. See that connector's `normalizeRow` doc
// comment for how this was found.
const payload: SourceRow = {};
for (const [field, value] of Object.entries(row)) {
payload[field] = value instanceof Date ? value.toISOString() : value;
}
return { record_key: String(key), payload, operation: "upsert" };
}
private requireSession(op: string): PostgresKeyedLookupConnectorConfig {
if (!this.connected || !this.config) throw new ConnectorNotConnectedError(this.connector_id, op);
return this.config;
}
}
+70
View File
@@ -0,0 +1,70 @@
/**
* src/connectors/postgres-query-impl.ts
* ========================================
* The REAL `QueryImpl` (see postgres-watermark-connector.ts) — a thin,
* read-only wrapper over `postgres` (postgres.js, already a repo dependency
* — see src/repositories/db.ts for the sibling pattern this mirrors).
*
* Used ONLY by scripts/vehiq-oltp-ingest/run.ts and the opt-in realinfra
* test. Every connector-level test in tests/connectors/*.test.ts injects a
* fixture QueryImpl instead — no real Postgres required to prove the query
* shape, the watermark math, or the narrow-scan property.
*
* DOUBLE-ENFORCED READ-ONLY. The connectors above only ever build SELECT
* statements — there is no code path that builds anything else. This
* wrapper adds a second, independent check anyway: it refuses to execute
* any statement whose first keyword is not SELECT. Belt-and-suspenders is
* appropriate here specifically because the thing on the other end is a
* separate product's live production OLTP primary, not 0pt's own store.
*/
import postgres from "postgres";
import type { QueryImpl, SourceRow } from "./postgres-watermark-connector";
export interface PostgresQueryImplOptions {
connectionString: string;
ssl?: "require" | false;
/** Statement timeout (ms), applied per-connection so a slow query cannot pin the source's OLTP primary. Default 5000. */
statementTimeoutMs?: number;
}
class NonSelectStatementError extends Error {
constructor(sql: string) {
super(`refusing to execute a non-SELECT statement against a read-only source connector: ${sql.slice(0, 120)}`);
this.name = "NonSelectStatementError";
}
}
export function assertReadOnlyStatement(sql: string): void {
const firstWord = sql.trim().split(/\s+/, 1)[0]?.toUpperCase();
if (firstWord !== "SELECT") throw new NonSelectStatementError(sql);
}
/**
* Builds a `QueryImpl` bound to one `postgres` client. The caller owns the
* client's lifecycle (open once per orchestration run, `sql.end()` when
* done) — this function does not memoize or reuse connections across calls,
* unlike `src/repositories/db.ts`'s deliberate memoization for 0pt's own
* store, because this one instance is scoped to a single ingestion run
* against an EXTERNAL production database and should not outlive it.
*/
export function makePostgresQueryImpl(opts: PostgresQueryImplOptions): { query: QueryImpl; sql: ReturnType<typeof postgres>; close: () => Promise<void> } {
const sql = postgres(opts.connectionString, {
prepare: false,
idle_timeout: 20,
max: 2,
ssl: opts.ssl ?? false,
connection: {
statement_timeout: opts.statementTimeoutMs ?? 5_000,
// A narrow, self-identifying application_name — same posture as
// http-json-connector.ts's fixed User-Agent: no impersonation, easy
// to spot in the source's own pg_stat_activity.
application_name: "0pt-vehiq-oltp-extractor",
},
});
const query: QueryImpl = async (text: string, params: unknown[]): Promise<SourceRow[]> => {
assertReadOnlyStatement(text);
const rows = await sql.unsafe(text, params as never[]);
return rows.map((r) => ({ ...r }));
};
return { query, sql, close: () => sql.end({ timeout: 5 }) };
}
@@ -0,0 +1,350 @@
/**
* src/connectors/postgres-watermark-connector.ts
* =================================================
* P061 connector: a read-only, watermark-polled Postgres OLTP table.
*
* WHY THIS EXISTS. BI-3/BI-4 need a controlled, incremental path from a
* product's live OLTP Postgres into 0pt — without `wal_level=logical`,
* without a Postgres restart, without a permanent replica, and without ever
* scanning a whole table. `docs/60-data/bi/...` (read from `admin/vehiq`,
* 2026-09-25 pass) names the exact shape this connector implements:
*
* WHERE updated_at > :last_watermark ORDER BY updated_at, id
* LIMIT :batch_size
*
* with a checkpoint of `(source, table, last_updated_at, last_id,
* batch_count, checksum, ingested_at)`. This connector produces exactly
* that query and that checkpoint — see the field-mapping table in
* `docs/P070_VEHIQ_OLTP_EXTRACTOR.md`.
*
* TIE-BREAK, NOT A DEVIATION. A bare `updated_at > $1` drops rows that share
* the exact same `updated_at` as the LAST row of a page: if row 100 and row
* 101 both have `updated_at = T`, and row 100 is the last row of a page, a
* naive `> T` on the next page skips row 101 forever. The query below adds
* the documented `ORDER BY updated_at, id` tie-break to the WHERE clause
* too: `updated_at > $1 OR (updated_at = $1 AND id > $2)`. Same watermark
* semantics Simon specified, correct at page boundaries — exactly the class
* of bug `tests/connectors/postgres-watermark-connector.test.ts`'s
* "same-timestamp tie at a page boundary" case exists to catch.
*
* GUARD RAIL, NOT DECORATION. `validate()` refuses any config whose `table`
* is (or ends in) the 98M-row `vehicle` table. That table gets its own
* narrow, keyed connector (`postgres-keyed-lookup-connector.ts`) — this one
* must never be pointed at it, even by a future caller's typo. The refusal
* is asserted in tests, not just documented.
*
* READ-ONLY, STRUCTURALLY. There is no INSERT/UPDATE/DELETE anywhere in this
* file. `columns` is an explicit allow-list the caller declares — never
* `SELECT *` — so a schema drift on the source table cannot silently widen
* what this connector reads or hands downstream.
*
* INJECTABLE QUERY IMPL — same seam as `http-json-connector.ts`'s
* `fetchImpl`: tests drive real SQL text/params through a fixture function,
* no real Postgres required for the connector-level tests. The default
* implementation (`makePostgresQueryImpl`) is a thin wrapper over `postgres`
* (postgres.js, already a repo dependency), used only by the orchestration
* script and the opt-in realinfra test.
*/
import { createHash } from "node:crypto";
import {
ConnectorNotConnectedError,
type BackfillRequest,
type Connector,
type ConnectorConfig,
type ConnectorContext,
type ConnectorHealth,
type ConnectorRecord,
type DiscoveryResult,
type ReadRequest,
type ReadResult,
type ValidationResult,
} from "./connector";
import type { IngestionCheckpoint } from "../types/ingestion";
import type { RecordSchemaFields } from "../schema/record-schema";
export const POSTGRES_WATERMARK_CONNECTOR_VERSION = "postgres-watermark-connector@1";
/** One row back from the source, column name -> raw value. */
export type SourceRow = Record<string, unknown>;
/**
* Injectable query seam. `params` are positional (`$1`, `$2`, ...). Tests
* supply a fixture; `makePostgresQueryImpl` supplies the real thing.
*/
export type QueryImpl = (sql: string, params: unknown[]) => Promise<SourceRow[]>;
export interface PostgresWatermarkConnectorConfig extends ConnectorConfig {
/** Fully-qualified source table, e.g. "app.dealer_stock_entry". */
table: string;
/** Logical partition name exposed to the runner — a dataset name, never a physical detail leak beyond `table` itself. */
partition_key: string;
/** Explicit column allow-list. Never "*". Must include `record_key_column` and `watermark_column`. */
columns: string[];
record_key_column: string;
watermark_column: string;
/** Declared record shape, fed to the schema registry (P063). */
schema: RecordSchemaFields;
}
/** The 98M-row table this connector must never be pointed at. See file header. */
const FORBIDDEN_TABLE_PATTERN = /(^|\.)vehicle$/i;
function isConfig(config: ConnectorConfig): config is PostgresWatermarkConnectorConfig {
return typeof config.table === "string" && typeof config.partition_key === "string";
}
/** Cursor vocabulary: JSON `{last_updated_at, last_id, checksum}` — see docs/P070 for the checkpoint field mapping. */
export interface WatermarkCursor {
last_updated_at: string;
last_id: string;
/** sha256 of the page just produced, hex-encoded. Absent only for the "never read" case. */
checksum?: string;
}
export function encodeCursor(cursor: WatermarkCursor): string {
return JSON.stringify(cursor);
}
export function decodeCursor(raw: string | undefined): WatermarkCursor | undefined {
if (raw === undefined) return undefined;
const parsed = JSON.parse(raw) as unknown;
if (
typeof parsed !== "object" ||
parsed === null ||
typeof (parsed as WatermarkCursor).last_updated_at !== "string" ||
typeof (parsed as WatermarkCursor).last_id !== "string"
) {
throw new Error(`invalid watermark cursor "${raw}"`);
}
return parsed as WatermarkCursor;
}
/**
* Postgres drivers (postgres.js included) hand back `timestamptz` columns as
* native JS `Date` objects, not strings. `String(date)` produces the
* locale/timezone-dependent `Date.toString()` form ("Thu Sep 03 2026
* 02:00:00 GMT+0200 (...)") — useless as a stored watermark, and NOT what a
* fixture-based test (which hands back plain ISO strings) would ever catch.
* Found by running this connector against a real Postgres container, not by
* the fixture-only unit tests — which is why this helper exists instead of
* an inline `String(...)` at each call site.
*/
function watermarkValueToIso(value: unknown): string {
return value instanceof Date ? value.toISOString() : String(value);
}
/**
* The P063 schema-conformance gate (src/schema/compatibility.ts) checks a
* payload's field VALUES against the declared schema — and even its
* `"timestamp"` type requires an already-ISO-string value, never a raw
* `Date`. A real Postgres driver hands back `timestamptz`/`date` columns as
* `Date` objects, so a payload built straight from `queryImpl`'s rows fails
* conformance the moment it reaches a real database — caught by running
* this connector against a real Postgres container (see docs/P070's
* verification log), not by the fixture-only unit tests, which all use
* plain strings. Every row is normalized through this function before it
* becomes a `ConnectorRecord` payload, so the envelope's payload is always
* JSON-safe and schema-checkable, independent of which driver produced it.
*/
function normalizeRow(row: SourceRow): SourceRow {
const normalized: SourceRow = {};
for (const [field, value] of Object.entries(row)) {
normalized[field] = value instanceof Date ? value.toISOString() : value;
}
return normalized;
}
/** sha256 over a stable JSON encoding of the rows, hex-encoded. Deterministic regardless of object key insertion order upstream. */
export function checksumRows(rows: SourceRow[]): string {
const stable = rows.map((row) =>
Object.fromEntries(Object.keys(row).sort().map((k) => [k, row[k]])),
);
return createHash("sha256").update(JSON.stringify(stable)).digest("hex");
}
export class PostgresWatermarkConnector implements Connector {
readonly connector_id = "postgres_watermark";
readonly connector_version = POSTGRES_WATERMARK_CONNECTOR_VERSION;
private config?: PostgresWatermarkConnectorConfig;
private ctx?: ConnectorContext;
private lastCursor = new Map<string, string | undefined>();
private connected = false;
constructor(private readonly queryImpl: QueryImpl) {}
validate(config: ConnectorConfig): ValidationResult {
const errors: string[] = [];
if (typeof config.table !== "string" || config.table.length === 0) {
errors.push("table is required");
} else if (FORBIDDEN_TABLE_PATTERN.test(config.table)) {
errors.push(
`table "${config.table}" looks like the 98M-row vehicle catalog table — this connector is for watermark-polled ` +
`OLTP tables only; use PostgresKeyedLookupConnector for narrow, keyed vehicle attribute lookups instead`,
);
}
if (typeof config.partition_key !== "string" || config.partition_key.length === 0) {
errors.push("partition_key is required");
}
const columns = config.columns;
if (!Array.isArray(columns) || columns.length === 0) {
errors.push("columns is required and must be a non-empty explicit column list");
} else if (columns.some((c) => c === "*" || typeof c !== "string" || c.length === 0)) {
errors.push('columns must not contain "*" or an empty entry — SELECT * is not permitted');
}
if (typeof config.record_key_column !== "string" || config.record_key_column.length === 0) {
errors.push("record_key_column is required");
} else if (Array.isArray(columns) && !columns.includes(config.record_key_column)) {
errors.push("record_key_column must be included in columns");
}
if (typeof config.watermark_column !== "string" || config.watermark_column.length === 0) {
errors.push("watermark_column is required");
} else if (Array.isArray(columns) && !columns.includes(config.watermark_column)) {
errors.push("watermark_column must be included in columns");
}
const schema = config.schema;
if (typeof schema !== "object" || schema === null || Object.keys(schema).length === 0) {
errors.push("schema is required and must declare at least one field");
}
return { valid: errors.length === 0, errors };
}
async connect(config: ConnectorConfig, ctx: ConnectorContext): Promise<void> {
const verdict = this.validate(config);
if (!verdict.valid) {
throw new Error(`postgres_watermark connector config invalid: ${verdict.errors.join("; ")}`);
}
if (!isConfig(config)) throw new Error("postgres_watermark connector config invalid");
this.config = config;
this.ctx = ctx;
this.connected = true;
}
async discover(): Promise<DiscoveryResult> {
const config = this.requireSession("discover");
return {
partitions: [{ partition_key: config.partition_key, name: config.table, schema: config.schema }],
// A polled table is batch/stream, never cdc: this connector never emits deletes.
modes: ["batch", "stream"],
};
}
async health(): Promise<ConnectorHealth> {
const config = this.requireSession("health");
const started = Date.now();
try {
await this.queryImpl(`SELECT 1 FROM ${config.table} LIMIT 0`, []);
return { status: "up", latency_ms: Date.now() - started };
} catch (err) {
return {
status: "down",
latency_ms: Date.now() - started,
detail: err instanceof Error ? err.message : "unreachable",
};
}
}
async read(req: ReadRequest): Promise<ReadResult> {
const config = this.requireSession("read");
const cursor = decodeCursor(req.cursor);
const columnList = config.columns.join(", ");
let rows: SourceRow[];
if (cursor === undefined) {
rows = await this.queryImpl(
`SELECT ${columnList} FROM ${config.table} ORDER BY ${config.watermark_column}, ${config.record_key_column} LIMIT $1`,
[req.limit],
);
} else {
rows = await this.queryImpl(
`SELECT ${columnList} FROM ${config.table} ` +
`WHERE ${config.watermark_column} > $1 OR (${config.watermark_column} = $1 AND ${config.record_key_column} > $2) ` +
`ORDER BY ${config.watermark_column}, ${config.record_key_column} LIMIT $3`,
[cursor.last_updated_at, cursor.last_id, req.limit],
);
}
// Hard bound — a queryImpl that over-delivers (a fixture bug, or a
// future real driver quirk) is truncated here rather than silently
// widening the caller's page.
rows = rows.slice(0, req.limit);
const records: ConnectorRecord[] = rows.map((row) => this.toRecord(row, config));
let nextCursor = req.cursor;
if (rows.length > 0) {
const last = rows[rows.length - 1];
const nextWatermark: WatermarkCursor = {
last_updated_at: watermarkValueToIso(last[config.watermark_column]),
last_id: String(last[config.record_key_column]),
checksum: checksumRows(rows),
};
nextCursor = encodeCursor(nextWatermark);
this.lastCursor.set(req.partition_key, nextCursor);
}
return {
records,
next_cursor: nextCursor,
// Fewer rows than the limit means we caught up to "now" — there may be
// MORE later (this is a live table), but nothing more RIGHT NOW.
exhausted: rows.length < req.limit,
};
}
async checkpoint(partitionKey: string): Promise<{ partition_key: string; cursor?: string }> {
this.requireSession("checkpoint");
return { partition_key: partitionKey, cursor: this.lastCursor.get(partitionKey) };
}
async resume(checkpoint: IngestionCheckpoint, limit: number): Promise<ReadResult> {
this.requireSession("resume");
return this.read({ partition_key: checkpoint.partition_key, cursor: checkpoint.cursor, limit });
}
async backfill(req: BackfillRequest): Promise<ReadResult> {
const config = this.requireSession("backfill");
if (!req.from || !req.to) {
throw new Error("backfill requires both `from` and `to` — an unbounded backfill is a full reload, not a backfill");
}
const columnList = config.columns.join(", ");
const rows = (
await this.queryImpl(
`SELECT ${columnList} FROM ${config.table} ` +
`WHERE ${config.watermark_column} >= $1 AND ${config.watermark_column} < $2 ` +
`ORDER BY ${config.watermark_column}, ${config.record_key_column} LIMIT $3`,
[req.from, req.to, req.limit],
)
).slice(0, req.limit);
return {
records: rows.map((row) => this.toRecord(row, config)),
next_cursor: undefined,
exhausted: true,
};
}
async disconnect(): Promise<void> {
this.config = undefined;
this.ctx = undefined;
this.lastCursor.clear();
this.connected = false;
}
// --- internals -------------------------------------------------------------
private toRecord(row: SourceRow, config: PostgresWatermarkConnectorConfig): ConnectorRecord {
const key = row[config.record_key_column];
if (typeof key !== "string" && typeof key !== "number") {
throw new Error(`record is missing string/number key column "${config.record_key_column}"`);
}
const watermark = watermarkValueToIso(row[config.watermark_column]);
return {
record_key: String(key),
payload: normalizeRow(row),
operation: "upsert",
occurred_at: watermark,
cursor: encodeCursor({ last_updated_at: watermark, last_id: String(key) }),
};
}
private requireSession(op: string): PostgresWatermarkConnectorConfig {
if (!this.connected || !this.config) throw new ConnectorNotConnectedError(this.connector_id, op);
return this.config;
}
}
+117
View File
@@ -0,0 +1,117 @@
/**
* src/ingestion/projection-sink.ts
* ===================================
* Adapts the P062 runner's generic `IngestionSink` (envelopes in) onto
* `SiaxAnalyticsEngine.ingestProjection` (BI-P009 capability 7, writes
* `opt_dataset_projection` in ClickHouse — see `src/analytics/engine/
* clickhouse-engine.ts`).
*
* WHY `ingestProjection` / `opt_dataset_projection` AND NOT `opt_analytics_
* event`. `opt_analytics_event` (BI-P010 schema.ts) is documented as a
* "high-volume per-tenant EVENT stream" with a 30-day TTL — the right shape
* for discrete occurrences (a page view, a search, a status transition).
* `dealer_stock_entry`/`listing`/`deal` rows and vehicle-dimension rows are
* ENTITY SNAPSHOTS — current state of a stock entry, a listing, a deal, a
* vehicle — which is exactly what `opt_dataset_projection` (ReplacingMerge
* Tree serving tier, `entity_key`+`version`) already exists to hold, and
* `ingestProjection` is the one write port `SiaxAnalyticsEngine` exposes for
* it. Landing entity rows into the raw event stream would be forcing an
* entity shape through an event shape — this sink does not do that.
*
* THE FIREWALL IS RESPECTED BY CONSTRUCTION: this file imports only the
* `SiaxAnalyticsEngine`/`AnalyticsIdentity`/`ProjectionRow` TYPES from
* `../analytics/engine/siax-analytics-engine` — no ClickHouse client, no
* `CLICKHOUSE_*` env read, no connection string. Which concrete engine
* (ClickHouse today, anything else later) answers `ingestProjection` is the
* caller's concern (see `scripts/vehiq-oltp-ingest/run.ts`), not this file's.
*
* ORDERING STAYS THE RUNNER'S: this function is called AS the runner's
* `sink`, so its own crash/partial-failure behaviour is exactly what
* `ingestion-runner.ts`'s header already documents — a thrown error here
* leaves the page's checkpoint uncommitted, and the next run re-reads it.
*/
import type { IngestionEnvelope } from "../types/ingestion";
import type {
AnalyticsIdentity,
ProjectionRow,
SiaxAnalyticsEngine,
} from "../analytics/engine/siax-analytics-engine";
import { SiaxError } from "../platform/errors";
export interface ProjectionSinkOptions {
engine: SiaxAnalyticsEngine;
identity: AnalyticsIdentity;
/** Maps a partition_key (e.g. "dealer_stock_entry") to a logical dataset id (e.g. "vehiq.dealer_stock_entry"). */
datasetIdFor: (partitionKey: string) => string;
}
/**
* Splits an envelope's payload into ClickHouse-projection-friendly
* attributes (strings) and measures (numbers). `null`/`undefined` fields are
* dropped rather than coerced to `"null"`/`NaN`, which would fabricate a
* value nothing upstream actually asserted.
*/
export function toProjectionRow(envelope: IngestionEnvelope): ProjectionRow {
if (envelope.operation === "delete" || envelope.payload === null) {
throw new SiaxError(
"INVALID_REQUEST",
"0pt.ingestion.projection-sink",
`envelope ${envelope.envelope_id} is a delete (payload: null) — opt_dataset_projection has no tombstone ` +
`representation via ingestProjection today; a CDC-capable source reaching this sink is a real gap, not ` +
`silently handled here (this connector programme currently emits "upsert" only, so this path is untested ` +
`by design, not unreachable by contract)`,
);
}
const attributes: Record<string, string> = {};
const measures: Record<string, number> = {};
for (const [field, value] of Object.entries(envelope.payload)) {
if (value === null || value === undefined) continue;
if (typeof value === "number" && Number.isFinite(value)) {
measures[field] = value;
} else if (value instanceof Date) {
attributes[field] = value.toISOString();
} else if (typeof value === "object") {
attributes[field] = JSON.stringify(value);
} else {
attributes[field] = String(value);
}
}
return {
entity_key: envelope.record_key,
as_of: envelope.occurred_at ?? envelope.ingested_at,
version: envelope.sequence,
attributes,
measures,
};
}
export function projectionSink(
opts: ProjectionSinkOptions,
): (envelopes: IngestionEnvelope[]) => Promise<void> {
return async (envelopes: IngestionEnvelope[]): Promise<void> => {
if (envelopes.length === 0) return;
const byPartition = new Map<string, IngestionEnvelope[]>();
for (const envelope of envelopes) {
const list = byPartition.get(envelope.partition_key) ?? [];
list.push(envelope);
byPartition.set(envelope.partition_key, list);
}
for (const [partitionKey, group] of byPartition) {
const rows = group.map(toProjectionRow);
const dataset_id = opts.datasetIdFor(partitionKey);
const result = await opts.engine.ingestProjection(opts.identity, { dataset_id, rows });
if (result.ingested !== rows.length) {
// Partial ingest is a hard failure, on purpose: the runner only
// commits the checkpoint AFTER this sink resolves. Resolving
// "successfully" on a partial write would silently drop the
// un-ingested rows on the next run's resume.
throw new SiaxError(
"RETRYABLE_FAILURE",
"0pt.ingestion.projection-sink",
`ingestProjection for dataset "${dataset_id}" reported ${result.ingested}/${rows.length} rows ingested — ` +
`treating as a failed page so the checkpoint does not advance past unlanded rows`,
);
}
}
};
}
+343
View File
@@ -0,0 +1,343 @@
/**
* src/ingestion/vehiq-oltp-ingest-runner.ts
* ============================================
* BI-3/BI-4 — the controlled, incremental VehIQ OLTP -> 0pt ClickHouse path,
* as Simon specified it (`admin/vehiq` 2026-09-25 pass, "Data path for
* inventory.days_in_stock"):
*
* OLTP operational tables -> read-only analytical extraction ->
* watermark/updated_at -> 0pt ingestion -> ClickHouse
*
* WHY THIS LIVES IN `admin/0pt`, NOT `admin/vehiq`. ADR-0035 (admin/vehiq,
* decisions 1/2/10, CI-enforced by `tools/validate_engine_firewall.py`):
* VehIQ's own codebase must NEVER hold a ClickHouse credential or import a
* ClickHouse client, and product code must never hammer the live OLTP
* primary with analytical queries directly. `admin/vehiq`'s own hub
* (`docs/CURRENT_STATE.md`, "Connection details for a future API0
* connector") says this explicitly: "VehIQ's own codebase must never see
* any of this. This is 0pt's data plane; only a future `admin/api0`
* connector should hold these details." This repo is 0pt's data plane —
* the correct home per that same guidance (a sibling of `admin/api0`, not
* `admin/api0` itself, since the read/write mechanics belong next to the
* P060-P068 ingestion plane and the BI-P009/P010 ClickHouse engine that
* already live here).
*
* TWO ENTRY POINTS, DELIBERATELY SEPARATE:
*
* `previewVehiqOltpSource` — connect, discover, ONE bounded read. Never
* calls checkpoint()/commits anything, never runs the P062 runner. Safe
* to point at production at any time, table empty or not: nothing it does
* can advance a checkpoint or land a row. This is what proves "zero
* batches, zero writes" without deploying a live scheduled service.
*
* `runVehiqOltpIngestion` — the real, scheduled path: the P062
* `IngestionRunner` across all three OLTP sources, then the narrow
* `dimension_vehicle` lookup keyed on exactly the `vehicle_id`s the OLTP
* batch referenced. This is what `.gitea/workflows/vehiq-oltp-ingest.yml`
* runs — see that file and docs/P070_VEHIQ_OLTP_EXTRACTOR.md for the
* activation gate this stays behind until Simon flips it.
*
* CHECKPOINT FIELD MAPPING (Simon's own list: source, table, last_updated_at,
* last_id, batch_count, checksum, ingested_at) onto the canonical P062
* `IngestionCheckpoint` (source_id, tenant_id, partition_key, cursor,
* sequence, records_committed, committed_at) — deliberately reusing the
* existing durable checkpoint mechanism rather than building a second one:
*
* source -> checkpoint.source_id
* table -> checkpoint.partition_key
* last_updated_at-> decoded from checkpoint.cursor (WatermarkCursor)
* last_id -> decoded from checkpoint.cursor (WatermarkCursor)
* checksum -> decoded from checkpoint.cursor (WatermarkCursor)
* batch_count -> checkpoint.records_committed (cumulative) / a
* per-run page count is also in each PartitionRunReport
* ingested_at -> checkpoint.committed_at
*/
import type { IngestionRepository } from "../repositories/ingestion-repository";
import type { SchemaRegistry } from "../schema/schema-registry";
import type { LineageStore } from "../lineage/lineage-store";
import type { RecordSchemaFields } from "../schema/record-schema";
import { IngestionRunner, type IngestionRunReport, type IngestionSink } from "./ingestion-runner";
import {
PostgresWatermarkConnector,
decodeCursor,
type QueryImpl,
} from "../connectors/postgres-watermark-connector";
import { PostgresKeyedLookupConnector } from "../connectors/postgres-keyed-lookup-connector";
import type { IngestionEnvelope } from "../types/ingestion";
export interface VehiqOltpSourceDef {
/** e.g. "dealer_stock_entry" — also the logical dataset name suffix. */
partition_key: "dealer_stock_entry" | "deal";
/** Fully-qualified source table, e.g. "app.dealer_stock_entry". */
table: string;
columns: string[];
record_key_column: string;
watermark_column: string;
schema: RecordSchemaFields;
}
/**
* TWO sources, not three — READ THIS BEFORE ASSUMING `app.listing` IS
* COVERED. The task's own wording (and `admin/vehiq`'s `docs/CURRENT_STATE.
* md`, "the columns already exist") assumed all three OLTP tables carry a
* generic `updated_at`. Verified LIVE against production (`server10`,
* `vehiq-db`, read-only `information_schema.columns`, 2026-09-25) that
* this is only true for none of the three as a literal `updated_at` column
* — and NOT true at all for `app.listing`:
*
* app.dealer_stock_entry — id, tenant_id, vehicle_id, lot_location,
* floorplan_status, listed_at, sold_at. No `updated_at`, no `status`
* (it's `floorplan_status`). `listed_at` (NOT NULL) is the best
* available watermark — see the caveat below.
* app.deal — id, tenant_id, party_id, vehicle_id, sale_price_minor,
* currency, status, created_at. No `updated_at`. `created_at`
* (NOT NULL DEFAULT now()) is the best available watermark.
* app.listing — id, tenant_id, vehicle_id, asking_price_minor, currency,
* channels, status. NO timestamp column of any kind exists today.
* `admin/vehiq` migration 0084 (`published_at`/`unpublished_at`) is
* written but NOT YET APPLIED to production (verified: those columns
* are absent live) — until it lands, there is no honest way to build a
* watermark connector for `app.listing` without inventing a column that
* does not exist. Not built here — see docs/P070's "Deferred: app.
* listing extraction" section for exactly what unblocks it.
*
* HONEST CAVEAT ON THE TWO SOURCES THAT ARE HERE: `listed_at`/`created_at`
* are set ONCE, at insert. Neither table has a column that bumps when an
* EXISTING row changes (`sold_at`/`floorplan_status` on dealer_stock_entry;
* `status` on deal). This connector will therefore reliably see every NEW
* row, but will NOT see an in-place UPDATE to an existing row as "changed" —
* it has already passed the watermark and will not be re-read. This is the
* same gap a real `updated_at` column (plus a trigger, e.g. `0084`-style)
* would close, and is exactly why building a permanent extractor deserves
* that schema decision rather than silently working around its absence.
* `docs/P070_VEHIQ_OLTP_EXTRACTOR.md` documents this as the #1 thing to
* settle before leaning on this extractor for anything beyond "count of new
* rows since X".
*/
export const VEHIQ_OLTP_SOURCES: VehiqOltpSourceDef[] = [
{
partition_key: "dealer_stock_entry",
table: "app.dealer_stock_entry",
columns: ["id", "tenant_id", "vehicle_id", "lot_location", "floorplan_status", "listed_at", "sold_at"],
record_key_column: "id",
watermark_column: "listed_at",
schema: {
id: { type: "string", required: true },
tenant_id: { type: "string", required: true },
vehicle_id: { type: "string", required: true },
lot_location: { type: "string", required: true },
floorplan_status: { type: "string", required: true },
listed_at: { type: "timestamp", required: true },
sold_at: { type: "timestamp", required: false },
},
},
{
partition_key: "deal",
table: "app.deal",
columns: ["id", "tenant_id", "party_id", "vehicle_id", "sale_price_minor", "currency", "status", "created_at"],
record_key_column: "id",
watermark_column: "created_at",
schema: {
id: { type: "string", required: true },
tenant_id: { type: "string", required: true },
party_id: { type: "string", required: true },
vehicle_id: { type: "string", required: true },
sale_price_minor: { type: "number", required: true },
currency: { type: "string", required: true },
status: { type: "string", required: true },
created_at: { type: "timestamp", required: true },
},
},
];
/**
* `vehiq.vehicle`'s primary key column is `vehicle_id`, NOT `id` — verified
* live (information_schema.columns, read-only, no data scanned) 2026-09-25.
* Column list otherwise matches what BI-P010/VEHIQ_ANALYTICS_BASELINE.md
* already documented (make/model/model_year/fuel_type all present and
* correctly named).
*/
export const VEHIQ_VEHICLE_DIMENSION_TABLE = "vehiq.vehicle";
export const VEHIQ_VEHICLE_DIMENSION_RECORD_KEY_COLUMN = "vehicle_id";
export const VEHIQ_VEHICLE_DIMENSION_COLUMNS = ["vehicle_id", "make", "model", "model_year", "fuel_type"];
export const VEHIQ_VEHICLE_DIMENSION_SCHEMA: RecordSchemaFields = {
vehicle_id: { type: "string", required: true },
make: { type: "string", required: false },
model: { type: "string", required: false },
model_year: { type: "number", required: false },
fuel_type: { type: "string", required: false },
};
// --- preview: read-only, never commits, safe against production at any time ---
export interface PreviewResult {
partition_key: string;
table: string;
/** Records this ONE bounded read would produce right now. */
would_read: number;
exhausted: boolean;
checkpoint_before: string | undefined;
next_cursor: string | undefined;
}
/**
* ONE bounded connect -> discover -> read(existing checkpoint, limit), then
* disconnect. Never calls IngestionRunner, never calls connector.checkpoint()
* against the repo, never writes anything anywhere. This is intentionally
* the ONLY function in this file that is safe to point at a real production
* source before the extractor is "activated" (see docs/P070).
*/
export async function previewVehiqOltpSource(
source: VehiqOltpSourceDef,
query: QueryImpl,
repo: Pick<IngestionRepository, "getCheckpoint">,
opts: { sourceId: string; tenantId: string; projectId: string; limit?: number } = {
sourceId: "vehiq_oltp",
tenantId: "t_vehiq",
projectId: "vehiq",
},
): Promise<PreviewResult> {
const connector = new PostgresWatermarkConnector(query);
const config = {
table: source.table,
partition_key: source.partition_key,
columns: source.columns,
record_key_column: source.record_key_column,
watermark_column: source.watermark_column,
schema: source.schema,
};
await connector.connect(config, {
tenant_id: opts.tenantId,
project_id: opts.projectId,
source_id: opts.sourceId,
run_id: `preview_${Date.now()}`,
});
try {
const existing = await repo.getCheckpoint(opts.sourceId, source.partition_key);
const result = await connector.read({
partition_key: source.partition_key,
cursor: existing?.cursor,
limit: opts.limit ?? 100,
});
return {
partition_key: source.partition_key,
table: source.table,
would_read: result.records.length,
exhausted: result.exhausted,
checkpoint_before: existing?.cursor,
next_cursor: result.next_cursor,
};
} finally {
await connector.disconnect();
}
}
// --- real run: the P062 highway, scheduled/activated path ---------------------
export interface VehiqOltpIngestDeps {
repo: IngestionRepository;
schemas: SchemaRegistry;
lineage?: LineageStore;
/** Read-only query against VehIQ's `vehiq-db` (server10) — see makePostgresQueryImpl in the calling script for the real implementation. */
sourceQuery: QueryImpl;
/** Where OLTP-entity envelopes land (see src/ingestion/projection-sink.ts). */
oltpSink: IngestionSink;
/** Where the narrow vehicle-dimension envelopes land. Same sink shape; kept separate so the two datasets can be routed differently if ever needed. */
dimensionSink: IngestionSink;
tenantId: string;
projectId: string;
sourceIdPrefix?: string;
pageSize?: number;
maxPages?: number;
}
export interface VehiqOltpIngestSummary {
oltp: IngestionRunReport[];
dimension?: IngestionRunReport;
referenced_vehicle_ids: number;
}
/**
* Runs the three watermark sources through the P062 runner, then derives the
* narrow vehicle-dimension key set from exactly what THIS run's
* `dealer_stock_entry` batch referenced (never a fresh scan, never the full
* 98M-row table) and runs that through the runner too, if there is anything
* to look up. When there is nothing new (today's actual production state:
* all three OLTP tables are empty), this function runs, reports 0 records
* read for every source, calls the dimension sink zero times, and returns
* cleanly — the empty-table case is a normal, successful, no-op run, not a
* special case that needs its own error handling.
*/
export async function runVehiqOltpIngestion(deps: VehiqOltpIngestDeps): Promise<VehiqOltpIngestSummary> {
const runner = new IngestionRunner(deps.repo, deps.schemas, deps.lineage);
const sourceIdPrefix = deps.sourceIdPrefix ?? "vehiq_oltp";
const oltpReports: IngestionRunReport[] = [];
const referencedVehicleIds = new Set<string>();
for (const source of VEHIQ_OLTP_SOURCES) {
const seen: IngestionEnvelope[] = [];
const wrappedSink: IngestionSink = async (envelopes) => {
seen.push(...envelopes);
await deps.oltpSink(envelopes);
};
const connector = new PostgresWatermarkConnector(deps.sourceQuery);
const report = await runner.run({
source_id: `${sourceIdPrefix}_${source.partition_key}`,
tenant_id: deps.tenantId,
project_id: deps.projectId,
mode: "batch",
connector,
config: {
table: source.table,
partition_key: source.partition_key,
columns: source.columns,
record_key_column: source.record_key_column,
watermark_column: source.watermark_column,
schema: source.schema,
},
sink: wrappedSink,
page_size: deps.pageSize ?? 200,
max_pages: deps.maxPages ?? 1000,
output_ref: `0pt:dataset:vehiq.${source.partition_key}`,
});
oltpReports.push(report);
if (source.partition_key === "dealer_stock_entry") {
for (const envelope of seen) {
const vehicleId = envelope.payload?.["vehicle_id"];
if (typeof vehicleId === "string" && vehicleId.length > 0) referencedVehicleIds.add(vehicleId);
else if (typeof vehicleId === "number") referencedVehicleIds.add(String(vehicleId));
}
}
}
let dimensionReport: IngestionRunReport | undefined;
if (referencedVehicleIds.size > 0) {
const keys = [...referencedVehicleIds];
const dimensionConnector = new PostgresKeyedLookupConnector(deps.sourceQuery);
dimensionReport = await runner.run({
source_id: `${sourceIdPrefix}_dimension_vehicle`,
tenant_id: deps.tenantId,
project_id: deps.projectId,
mode: "batch",
connector: dimensionConnector,
config: {
table: VEHIQ_VEHICLE_DIMENSION_TABLE,
partition_key: "dimension_vehicle",
columns: VEHIQ_VEHICLE_DIMENSION_COLUMNS,
record_key_column: VEHIQ_VEHICLE_DIMENSION_RECORD_KEY_COLUMN,
schema: VEHIQ_VEHICLE_DIMENSION_SCHEMA,
keys,
},
sink: deps.dimensionSink,
page_size: deps.pageSize ?? 200,
max_pages: deps.maxPages ?? 1000,
output_ref: "0pt:dataset:vehiq.dimension_vehicle",
});
}
return { oltp: oltpReports, dimension: dimensionReport, referenced_vehicle_ids: referencedVehicleIds.size };
}
/** Re-exported so callers (the CLI script, tests) don't need a second import from the connector module. */
export { decodeCursor };