Files
0pt/scripts/vehiq-oltp-ingest/run.ts
T
Claude CodeandClaude Sonnet 5 07158099ee
CI (SIAX Cloud) / integration (push) Skipped
masterplan-lock / check (push) Successful in 7s
CI (SIAX Cloud) / security (push) Successful in 17s
masterplan-lock / check (pull_request) Successful in 9s
Security Scan (SIAX Cloud) / security (push) Successful in 38s
CI (SIAX Cloud) / security (pull_request) Successful in 26s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 26s
CI (SIAX Cloud) / quality (push) Successful in 57s
CI (SIAX Cloud) / integration (pull_request) Successful in 39s
CI (SIAX Cloud) / quality (pull_request) Successful in 59s
feat(ingestion): VehIQ OLTP -> ClickHouse watermark extractor (P070)
Implements Simon's specified pattern (dealer_stock_entry/deal ->
watermark/updated_at -> 0pt ingestion -> ClickHouse) as a P061 connector
pair (PostgresWatermarkConnector, PostgresKeyedLookupConnector) plugged
into the existing P062 IngestionRunner, writing to opt_dataset_projection
via SiaxAnalyticsEngine.ingestProjection.

Built here, not in admin/vehiq: that repo's own ADR-0035 (decisions 1/2/10,
CI-enforced by validate_engine_firewall.py) forbids VehIQ code from ever
holding a ClickHouse credential or hammering OLTP directly for analytics.
See docs/P070_VEHIQ_OLTP_EXTRACTOR.md for the full rationale.

Also corrects a real schema mismatch found while building this: production
has no generic `updated_at` on any of the three target tables (verified
live, read-only, 2026-09-25). dealer_stock_entry/deal use listed_at/
created_at as insert-time watermarks instead (documented caveat: in-place
updates are not caught without a schema change); app.listing has no
timestamp column at all and is deliberately not covered yet.

Vehicle dimension lookup is narrow by construction: keyed on exactly the
vehicle_id set referenced by the batch just read, never a scan of the
98M-row vehiq.vehicle table (connector-level validate() refuses that table
by name; runner-level tests assert the query only ever touches referenced
IDs).

Verified: fixture-backed connector/runner tests (watermark+limit,
same-timestamp tie at a page boundary, crash/resume, empty-table case,
narrow-lookup query-count assertions) + a real disposable Postgres
container (caught two real Date-handling bugs fixtures could not) + a
read-only proof against actual production (server10, vehiq-db): the
extractor's exact SQL returns zero rows, no error, confirming it is safe
to schedule with zero current effect.

.gitea/workflows/vehiq-oltp-ingest.yml is present, cron-scheduled, and
defaults to VEHIQ_INGEST_MODE=preview (read-only, never writes) with no
secrets configured — safe to merge with zero production effect until
explicitly activated per the doc's "what activating this requires".

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-25 19:11:15 +02:00

138 lines
5.4 KiB
JavaScript

#!/usr/bin/env -S node
/**
* scripts/vehiq-oltp-ingest/run.ts
* ===================================
* Thin CLI shim over src/ingestion/vehiq-oltp-ingest-runner.ts. All the
* actual logic (connectors, the runner call, the checkpoint mapping) is in
* that blocking-typechecked, tested `src/` module; this file only reads
* `process.env`, wires real implementations, and prints a summary.
*
* TWO MODES, PREVIEW IS THE DEFAULT — this is the activation gate:
*
* VEHIQ_INGEST_MODE=preview (default)
* One bounded read-only connect+discover+read per source, against
* whatever checkpoint already exists. NEVER commits a checkpoint, NEVER
* calls the sink, NEVER touches ClickHouse. Safe to run against real
* production at any time — this is what proves "zero batches, zero
* writes" without activating anything.
*
* VEHIQ_INGEST_MODE=run
* The real, scheduled path: P062 runner, durable checkpoints, writes
* landing in `opt_dataset_projection`. Requires DATABASE_URL (0pt's own
* metadata store, for durable checkpoints) and the OPT_ANALYTICS_
* CLICKHOUSE_* vars (the write target) to be set, or it refuses to run
* rather than silently falling back to in-memory checkpoints or a
* no-op sink — see docs/P070_VEHIQ_OLTP_EXTRACTOR.md "What activating
* this in production actually requires".
*
* NO VEHIQ_SOURCE_DATABASE_URL AT ALL -> exits 0 having done nothing. This
* is what makes the scheduled workflow (.gitea/workflows/vehiq-oltp-
* ingest.yml) safe to leave enabled with no secret configured: cron fires,
* the script logs "not configured", exits 0, no error, no alert.
*/
import {
VEHIQ_OLTP_SOURCES,
previewVehiqOltpSource,
runVehiqOltpIngestion,
} from "../../src/ingestion/vehiq-oltp-ingest-runner";
import { makePostgresQueryImpl } from "../../src/connectors/postgres-query-impl";
import { createIngestionRepository } from "../../src/repositories/ingestion-repository";
import { SchemaRegistry } from "../../src/schema/schema-registry";
import { LineageStore } from "../../src/lineage/lineage-store";
import { ClickHouseAnalyticsEngine } from "../../src/analytics/engine/clickhouse-engine";
import { projectionSink } from "../../src/ingestion/projection-sink";
import type { AnalyticsIdentity } from "../../src/analytics/engine/siax-analytics-engine";
const TENANT_ID = process.env.VEHIQ_INGEST_TENANT_ID ?? "t_vehiq";
const PROJECT_ID = process.env.VEHIQ_INGEST_PROJECT_ID ?? "vehiq";
async function main(): Promise<void> {
const sourceUrl = process.env.VEHIQ_SOURCE_DATABASE_URL;
if (!sourceUrl) {
console.log(
JSON.stringify({ status: "not_configured", detail: "VEHIQ_SOURCE_DATABASE_URL is unset — nothing to do" }),
);
return;
}
const mode = process.env.VEHIQ_INGEST_MODE === "run" ? "run" : "preview";
const { query, close } = makePostgresQueryImpl({
connectionString: sourceUrl,
ssl: process.env.VEHIQ_SOURCE_PGSSL === "require" ? "require" : false,
statementTimeoutMs: process.env.VEHIQ_SOURCE_STATEMENT_TIMEOUT_MS
? Number(process.env.VEHIQ_SOURCE_STATEMENT_TIMEOUT_MS)
: undefined,
});
try {
if (mode === "preview") {
const repo = createIngestionRepository(process.env.DATABASE_URL);
const results = [];
for (const source of VEHIQ_OLTP_SOURCES) {
results.push(
await previewVehiqOltpSource(source, query, repo, {
sourceId: "vehiq_oltp",
tenantId: TENANT_ID,
projectId: PROJECT_ID,
limit: 100,
}),
);
}
console.log(JSON.stringify({ status: "preview", mode, results }, null, 2));
return;
}
// mode === "run" — the activation-gated real path.
if (!process.env.DATABASE_URL) {
throw new Error(
"VEHIQ_INGEST_MODE=run requires DATABASE_URL (0pt's own metadata store, for durable checkpoints) — " +
"refusing to run with in-memory checkpoints, which would silently re-read every row on every scheduled invocation",
);
}
const chBaseUrl = process.env.OPT_ANALYTICS_CLICKHOUSE_BASE_URL;
if (!chBaseUrl) {
throw new Error(
"VEHIQ_INGEST_MODE=run requires OPT_ANALYTICS_CLICKHOUSE_BASE_URL — refusing to run with no write target",
);
}
const repo = createIngestionRepository(process.env.DATABASE_URL);
const schemas = new SchemaRegistry(repo);
const lineage = new LineageStore(repo);
const engine = new ClickHouseAnalyticsEngine({
baseUrl: chBaseUrl,
basicAuth: process.env.OPT_ANALYTICS_CLICKHOUSE_BASIC_AUTH,
database: process.env.OPT_ANALYTICS_CLICKHOUSE_DATABASE ?? "opt_analytics",
});
const identity: AnalyticsIdentity = {
tenant_id: TENANT_ID,
project_id: PROJECT_ID,
principal_id: "svc_vehiq_oltp_extractor",
};
const sink = projectionSink({
engine,
identity,
datasetIdFor: (partitionKey) => `vehiq.${partitionKey}`,
});
const summary = await runVehiqOltpIngestion({
repo,
schemas,
lineage,
sourceQuery: query,
oltpSink: sink,
dimensionSink: sink,
tenantId: TENANT_ID,
projectId: PROJECT_ID,
});
console.log(JSON.stringify({ status: "ran", mode, summary }, null, 2));
} finally {
await close();
}
}
main().catch((err) => {
console.error(JSON.stringify({ status: "error", detail: err instanceof Error ? err.message : String(err) }));
process.exitCode = 1;
});