Files
0pt/src/connectors/postgres-query-impl.ts
T
Claude CodeandClaude Sonnet 5 07158099ee
CI (SIAX Cloud) / integration (push) Skipped
masterplan-lock / check (push) Successful in 7s
CI (SIAX Cloud) / security (push) Successful in 17s
masterplan-lock / check (pull_request) Successful in 9s
Security Scan (SIAX Cloud) / security (push) Successful in 38s
CI (SIAX Cloud) / security (pull_request) Successful in 26s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 26s
CI (SIAX Cloud) / quality (push) Successful in 57s
CI (SIAX Cloud) / integration (pull_request) Successful in 39s
CI (SIAX Cloud) / quality (pull_request) Successful in 59s
feat(ingestion): VehIQ OLTP -> ClickHouse watermark extractor (P070)
Implements Simon's specified pattern (dealer_stock_entry/deal ->
watermark/updated_at -> 0pt ingestion -> ClickHouse) as a P061 connector
pair (PostgresWatermarkConnector, PostgresKeyedLookupConnector) plugged
into the existing P062 IngestionRunner, writing to opt_dataset_projection
via SiaxAnalyticsEngine.ingestProjection.

Built here, not in admin/vehiq: that repo's own ADR-0035 (decisions 1/2/10,
CI-enforced by validate_engine_firewall.py) forbids VehIQ code from ever
holding a ClickHouse credential or hammering OLTP directly for analytics.
See docs/P070_VEHIQ_OLTP_EXTRACTOR.md for the full rationale.

Also corrects a real schema mismatch found while building this: production
has no generic `updated_at` on any of the three target tables (verified
live, read-only, 2026-09-25). dealer_stock_entry/deal use listed_at/
created_at as insert-time watermarks instead (documented caveat: in-place
updates are not caught without a schema change); app.listing has no
timestamp column at all and is deliberately not covered yet.

Vehicle dimension lookup is narrow by construction: keyed on exactly the
vehicle_id set referenced by the batch just read, never a scan of the
98M-row vehiq.vehicle table (connector-level validate() refuses that table
by name; runner-level tests assert the query only ever touches referenced
IDs).

Verified: fixture-backed connector/runner tests (watermark+limit,
same-timestamp tie at a page boundary, crash/resume, empty-table case,
narrow-lookup query-count assertions) + a real disposable Postgres
container (caught two real Date-handling bugs fixtures could not) + a
read-only proof against actual production (server10, vehiq-db): the
extractor's exact SQL returns zero rows, no error, confirming it is safe
to schedule with zero current effect.

.gitea/workflows/vehiq-oltp-ingest.yml is present, cron-scheduled, and
defaults to VEHIQ_INGEST_MODE=preview (read-only, never writes) with no
secrets configured — safe to merge with zero production effect until
explicitly activated per the doc's "what activating this requires".

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-25 19:11:15 +02:00

71 lines
3.2 KiB
TypeScript

/**
* src/connectors/postgres-query-impl.ts
* ========================================
* The REAL `QueryImpl` (see postgres-watermark-connector.ts) — a thin,
* read-only wrapper over `postgres` (postgres.js, already a repo dependency
* — see src/repositories/db.ts for the sibling pattern this mirrors).
*
* Used ONLY by scripts/vehiq-oltp-ingest/run.ts and the opt-in realinfra
* test. Every connector-level test in tests/connectors/*.test.ts injects a
* fixture QueryImpl instead — no real Postgres required to prove the query
* shape, the watermark math, or the narrow-scan property.
*
* DOUBLE-ENFORCED READ-ONLY. The connectors above only ever build SELECT
* statements — there is no code path that builds anything else. This
* wrapper adds a second, independent check anyway: it refuses to execute
* any statement whose first keyword is not SELECT. Belt-and-suspenders is
* appropriate here specifically because the thing on the other end is a
* separate product's live production OLTP primary, not 0pt's own store.
*/
import postgres from "postgres";
import type { QueryImpl, SourceRow } from "./postgres-watermark-connector";
export interface PostgresQueryImplOptions {
connectionString: string;
ssl?: "require" | false;
/** Statement timeout (ms), applied per-connection so a slow query cannot pin the source's OLTP primary. Default 5000. */
statementTimeoutMs?: number;
}
class NonSelectStatementError extends Error {
constructor(sql: string) {
super(`refusing to execute a non-SELECT statement against a read-only source connector: ${sql.slice(0, 120)}`);
this.name = "NonSelectStatementError";
}
}
export function assertReadOnlyStatement(sql: string): void {
const firstWord = sql.trim().split(/\s+/, 1)[0]?.toUpperCase();
if (firstWord !== "SELECT") throw new NonSelectStatementError(sql);
}
/**
* Builds a `QueryImpl` bound to one `postgres` client. The caller owns the
* client's lifecycle (open once per orchestration run, `sql.end()` when
* done) — this function does not memoize or reuse connections across calls,
* unlike `src/repositories/db.ts`'s deliberate memoization for 0pt's own
* store, because this one instance is scoped to a single ingestion run
* against an EXTERNAL production database and should not outlive it.
*/
export function makePostgresQueryImpl(opts: PostgresQueryImplOptions): { query: QueryImpl; sql: ReturnType<typeof postgres>; close: () => Promise<void> } {
const sql = postgres(opts.connectionString, {
prepare: false,
idle_timeout: 20,
max: 2,
ssl: opts.ssl ?? false,
connection: {
statement_timeout: opts.statementTimeoutMs ?? 5_000,
// A narrow, self-identifying application_name — same posture as
// http-json-connector.ts's fixed User-Agent: no impersonation, easy
// to spot in the source's own pg_stat_activity.
application_name: "0pt-vehiq-oltp-extractor",
},
});
const query: QueryImpl = async (text: string, params: unknown[]): Promise<SourceRow[]> => {
assertReadOnlyStatement(text);
const rows = await sql.unsafe(text, params as never[]);
return rows.map((r) => ({ ...r }));
};
return { query, sql, close: () => sql.end({ timeout: 5 }) };
}