CI (SIAX Cloud) / integration (push) Skipped
masterplan-lock / check (push) Successful in 7s
CI (SIAX Cloud) / security (push) Successful in 17s
masterplan-lock / check (pull_request) Successful in 9s
Security Scan (SIAX Cloud) / security (push) Successful in 38s
CI (SIAX Cloud) / security (pull_request) Successful in 26s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 26s
CI (SIAX Cloud) / quality (push) Successful in 57s
CI (SIAX Cloud) / integration (pull_request) Successful in 39s
CI (SIAX Cloud) / quality (pull_request) Successful in 59s
Implements Simon's specified pattern (dealer_stock_entry/deal -> watermark/updated_at -> 0pt ingestion -> ClickHouse) as a P061 connector pair (PostgresWatermarkConnector, PostgresKeyedLookupConnector) plugged into the existing P062 IngestionRunner, writing to opt_dataset_projection via SiaxAnalyticsEngine.ingestProjection. Built here, not in admin/vehiq: that repo's own ADR-0035 (decisions 1/2/10, CI-enforced by validate_engine_firewall.py) forbids VehIQ code from ever holding a ClickHouse credential or hammering OLTP directly for analytics. See docs/P070_VEHIQ_OLTP_EXTRACTOR.md for the full rationale. Also corrects a real schema mismatch found while building this: production has no generic `updated_at` on any of the three target tables (verified live, read-only, 2026-09-25). dealer_stock_entry/deal use listed_at/ created_at as insert-time watermarks instead (documented caveat: in-place updates are not caught without a schema change); app.listing has no timestamp column at all and is deliberately not covered yet. Vehicle dimension lookup is narrow by construction: keyed on exactly the vehicle_id set referenced by the batch just read, never a scan of the 98M-row vehiq.vehicle table (connector-level validate() refuses that table by name; runner-level tests assert the query only ever touches referenced IDs). Verified: fixture-backed connector/runner tests (watermark+limit, same-timestamp tie at a page boundary, crash/resume, empty-table case, narrow-lookup query-count assertions) + a real disposable Postgres container (caught two real Date-handling bugs fixtures could not) + a read-only proof against actual production (server10, vehiq-db): the extractor's exact SQL returns zero rows, no error, confirming it is safe to schedule with zero current effect. .gitea/workflows/vehiq-oltp-ingest.yml is present, cron-scheduled, and defaults to VEHIQ_INGEST_MODE=preview (read-only, never writes) with no secrets configured — safe to merge with zero production effect until explicitly activated per the doc's "what activating this requires". Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
71 lines
3.2 KiB
TypeScript
71 lines
3.2 KiB
TypeScript
/**
|
|
* src/connectors/postgres-query-impl.ts
|
|
* ========================================
|
|
* The REAL `QueryImpl` (see postgres-watermark-connector.ts) — a thin,
|
|
* read-only wrapper over `postgres` (postgres.js, already a repo dependency
|
|
* — see src/repositories/db.ts for the sibling pattern this mirrors).
|
|
*
|
|
* Used ONLY by scripts/vehiq-oltp-ingest/run.ts and the opt-in realinfra
|
|
* test. Every connector-level test in tests/connectors/*.test.ts injects a
|
|
* fixture QueryImpl instead — no real Postgres required to prove the query
|
|
* shape, the watermark math, or the narrow-scan property.
|
|
*
|
|
* DOUBLE-ENFORCED READ-ONLY. The connectors above only ever build SELECT
|
|
* statements — there is no code path that builds anything else. This
|
|
* wrapper adds a second, independent check anyway: it refuses to execute
|
|
* any statement whose first keyword is not SELECT. Belt-and-suspenders is
|
|
* appropriate here specifically because the thing on the other end is a
|
|
* separate product's live production OLTP primary, not 0pt's own store.
|
|
*/
|
|
import postgres from "postgres";
|
|
import type { QueryImpl, SourceRow } from "./postgres-watermark-connector";
|
|
|
|
export interface PostgresQueryImplOptions {
|
|
connectionString: string;
|
|
ssl?: "require" | false;
|
|
/** Statement timeout (ms), applied per-connection so a slow query cannot pin the source's OLTP primary. Default 5000. */
|
|
statementTimeoutMs?: number;
|
|
}
|
|
|
|
class NonSelectStatementError extends Error {
|
|
constructor(sql: string) {
|
|
super(`refusing to execute a non-SELECT statement against a read-only source connector: ${sql.slice(0, 120)}`);
|
|
this.name = "NonSelectStatementError";
|
|
}
|
|
}
|
|
|
|
export function assertReadOnlyStatement(sql: string): void {
|
|
const firstWord = sql.trim().split(/\s+/, 1)[0]?.toUpperCase();
|
|
if (firstWord !== "SELECT") throw new NonSelectStatementError(sql);
|
|
}
|
|
|
|
/**
|
|
* Builds a `QueryImpl` bound to one `postgres` client. The caller owns the
|
|
* client's lifecycle (open once per orchestration run, `sql.end()` when
|
|
* done) — this function does not memoize or reuse connections across calls,
|
|
* unlike `src/repositories/db.ts`'s deliberate memoization for 0pt's own
|
|
* store, because this one instance is scoped to a single ingestion run
|
|
* against an EXTERNAL production database and should not outlive it.
|
|
*/
|
|
export function makePostgresQueryImpl(opts: PostgresQueryImplOptions): { query: QueryImpl; sql: ReturnType<typeof postgres>; close: () => Promise<void> } {
|
|
const sql = postgres(opts.connectionString, {
|
|
prepare: false,
|
|
idle_timeout: 20,
|
|
max: 2,
|
|
ssl: opts.ssl ?? false,
|
|
connection: {
|
|
statement_timeout: opts.statementTimeoutMs ?? 5_000,
|
|
// A narrow, self-identifying application_name — same posture as
|
|
// http-json-connector.ts's fixed User-Agent: no impersonation, easy
|
|
// to spot in the source's own pg_stat_activity.
|
|
application_name: "0pt-vehiq-oltp-extractor",
|
|
},
|
|
});
|
|
const query: QueryImpl = async (text: string, params: unknown[]): Promise<SourceRow[]> => {
|
|
assertReadOnlyStatement(text);
|
|
const rows = await sql.unsafe(text, params as never[]);
|
|
return rows.map((r) => ({ ...r }));
|
|
};
|
|
return { query, sql, close: () => sql.end({ timeout: 5 }) };
|
|
}
|