feat(ingestion): VehIQ OLTP -> ClickHouse watermark extractor (P070)
CI (SIAX Cloud) / integration (push) Skipped
masterplan-lock / check (push) Successful in 7s
CI (SIAX Cloud) / security (push) Successful in 17s
masterplan-lock / check (pull_request) Successful in 9s
Security Scan (SIAX Cloud) / security (push) Successful in 38s
CI (SIAX Cloud) / security (pull_request) Successful in 26s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 26s
CI (SIAX Cloud) / quality (push) Successful in 57s
CI (SIAX Cloud) / integration (pull_request) Successful in 39s
CI (SIAX Cloud) / quality (pull_request) Successful in 59s
CI (SIAX Cloud) / integration (push) Skipped
masterplan-lock / check (push) Successful in 7s
CI (SIAX Cloud) / security (push) Successful in 17s
masterplan-lock / check (pull_request) Successful in 9s
Security Scan (SIAX Cloud) / security (push) Successful in 38s
CI (SIAX Cloud) / security (pull_request) Successful in 26s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 26s
CI (SIAX Cloud) / quality (push) Successful in 57s
CI (SIAX Cloud) / integration (pull_request) Successful in 39s
CI (SIAX Cloud) / quality (pull_request) Successful in 59s
Implements Simon's specified pattern (dealer_stock_entry/deal -> watermark/updated_at -> 0pt ingestion -> ClickHouse) as a P061 connector pair (PostgresWatermarkConnector, PostgresKeyedLookupConnector) plugged into the existing P062 IngestionRunner, writing to opt_dataset_projection via SiaxAnalyticsEngine.ingestProjection. Built here, not in admin/vehiq: that repo's own ADR-0035 (decisions 1/2/10, CI-enforced by validate_engine_firewall.py) forbids VehIQ code from ever holding a ClickHouse credential or hammering OLTP directly for analytics. See docs/P070_VEHIQ_OLTP_EXTRACTOR.md for the full rationale. Also corrects a real schema mismatch found while building this: production has no generic `updated_at` on any of the three target tables (verified live, read-only, 2026-09-25). dealer_stock_entry/deal use listed_at/ created_at as insert-time watermarks instead (documented caveat: in-place updates are not caught without a schema change); app.listing has no timestamp column at all and is deliberately not covered yet. Vehicle dimension lookup is narrow by construction: keyed on exactly the vehicle_id set referenced by the batch just read, never a scan of the 98M-row vehiq.vehicle table (connector-level validate() refuses that table by name; runner-level tests assert the query only ever touches referenced IDs). Verified: fixture-backed connector/runner tests (watermark+limit, same-timestamp tie at a page boundary, crash/resume, empty-table case, narrow-lookup query-count assertions) + a real disposable Postgres container (caught two real Date-handling bugs fixtures could not) + a read-only proof against actual production (server10, vehiq-db): the extractor's exact SQL returns zero rows, no error, confirming it is safe to schedule with zero current effect. .gitea/workflows/vehiq-oltp-ingest.yml is present, cron-scheduled, and defaults to VEHIQ_INGEST_MODE=preview (read-only, never writes) with no secrets configured — safe to merge with zero production effect until explicitly activated per the doc's "what activating this requires". Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
ec487746fd
commit
07158099ee
@@ -0,0 +1,137 @@
|
||||
#!/usr/bin/env -S node
|
||||
/**
|
||||
* scripts/vehiq-oltp-ingest/run.ts
|
||||
* ===================================
|
||||
* Thin CLI shim over src/ingestion/vehiq-oltp-ingest-runner.ts. All the
|
||||
* actual logic (connectors, the runner call, the checkpoint mapping) is in
|
||||
* that blocking-typechecked, tested `src/` module; this file only reads
|
||||
* `process.env`, wires real implementations, and prints a summary.
|
||||
*
|
||||
* TWO MODES, PREVIEW IS THE DEFAULT — this is the activation gate:
|
||||
*
|
||||
* VEHIQ_INGEST_MODE=preview (default)
|
||||
* One bounded read-only connect+discover+read per source, against
|
||||
* whatever checkpoint already exists. NEVER commits a checkpoint, NEVER
|
||||
* calls the sink, NEVER touches ClickHouse. Safe to run against real
|
||||
* production at any time — this is what proves "zero batches, zero
|
||||
* writes" without activating anything.
|
||||
*
|
||||
* VEHIQ_INGEST_MODE=run
|
||||
* The real, scheduled path: P062 runner, durable checkpoints, writes
|
||||
* landing in `opt_dataset_projection`. Requires DATABASE_URL (0pt's own
|
||||
* metadata store, for durable checkpoints) and the OPT_ANALYTICS_
|
||||
* CLICKHOUSE_* vars (the write target) to be set, or it refuses to run
|
||||
* rather than silently falling back to in-memory checkpoints or a
|
||||
* no-op sink — see docs/P070_VEHIQ_OLTP_EXTRACTOR.md "What activating
|
||||
* this in production actually requires".
|
||||
*
|
||||
* NO VEHIQ_SOURCE_DATABASE_URL AT ALL -> exits 0 having done nothing. This
|
||||
* is what makes the scheduled workflow (.gitea/workflows/vehiq-oltp-
|
||||
* ingest.yml) safe to leave enabled with no secret configured: cron fires,
|
||||
* the script logs "not configured", exits 0, no error, no alert.
|
||||
*/
|
||||
import {
|
||||
VEHIQ_OLTP_SOURCES,
|
||||
previewVehiqOltpSource,
|
||||
runVehiqOltpIngestion,
|
||||
} from "../../src/ingestion/vehiq-oltp-ingest-runner";
|
||||
import { makePostgresQueryImpl } from "../../src/connectors/postgres-query-impl";
|
||||
import { createIngestionRepository } from "../../src/repositories/ingestion-repository";
|
||||
import { SchemaRegistry } from "../../src/schema/schema-registry";
|
||||
import { LineageStore } from "../../src/lineage/lineage-store";
|
||||
import { ClickHouseAnalyticsEngine } from "../../src/analytics/engine/clickhouse-engine";
|
||||
import { projectionSink } from "../../src/ingestion/projection-sink";
|
||||
import type { AnalyticsIdentity } from "../../src/analytics/engine/siax-analytics-engine";
|
||||
|
||||
const TENANT_ID = process.env.VEHIQ_INGEST_TENANT_ID ?? "t_vehiq";
|
||||
const PROJECT_ID = process.env.VEHIQ_INGEST_PROJECT_ID ?? "vehiq";
|
||||
|
||||
async function main(): Promise<void> {
|
||||
const sourceUrl = process.env.VEHIQ_SOURCE_DATABASE_URL;
|
||||
if (!sourceUrl) {
|
||||
console.log(
|
||||
JSON.stringify({ status: "not_configured", detail: "VEHIQ_SOURCE_DATABASE_URL is unset — nothing to do" }),
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
const mode = process.env.VEHIQ_INGEST_MODE === "run" ? "run" : "preview";
|
||||
const { query, close } = makePostgresQueryImpl({
|
||||
connectionString: sourceUrl,
|
||||
ssl: process.env.VEHIQ_SOURCE_PGSSL === "require" ? "require" : false,
|
||||
statementTimeoutMs: process.env.VEHIQ_SOURCE_STATEMENT_TIMEOUT_MS
|
||||
? Number(process.env.VEHIQ_SOURCE_STATEMENT_TIMEOUT_MS)
|
||||
: undefined,
|
||||
});
|
||||
|
||||
try {
|
||||
if (mode === "preview") {
|
||||
const repo = createIngestionRepository(process.env.DATABASE_URL);
|
||||
const results = [];
|
||||
for (const source of VEHIQ_OLTP_SOURCES) {
|
||||
results.push(
|
||||
await previewVehiqOltpSource(source, query, repo, {
|
||||
sourceId: "vehiq_oltp",
|
||||
tenantId: TENANT_ID,
|
||||
projectId: PROJECT_ID,
|
||||
limit: 100,
|
||||
}),
|
||||
);
|
||||
}
|
||||
console.log(JSON.stringify({ status: "preview", mode, results }, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
// mode === "run" — the activation-gated real path.
|
||||
if (!process.env.DATABASE_URL) {
|
||||
throw new Error(
|
||||
"VEHIQ_INGEST_MODE=run requires DATABASE_URL (0pt's own metadata store, for durable checkpoints) — " +
|
||||
"refusing to run with in-memory checkpoints, which would silently re-read every row on every scheduled invocation",
|
||||
);
|
||||
}
|
||||
const chBaseUrl = process.env.OPT_ANALYTICS_CLICKHOUSE_BASE_URL;
|
||||
if (!chBaseUrl) {
|
||||
throw new Error(
|
||||
"VEHIQ_INGEST_MODE=run requires OPT_ANALYTICS_CLICKHOUSE_BASE_URL — refusing to run with no write target",
|
||||
);
|
||||
}
|
||||
|
||||
const repo = createIngestionRepository(process.env.DATABASE_URL);
|
||||
const schemas = new SchemaRegistry(repo);
|
||||
const lineage = new LineageStore(repo);
|
||||
const engine = new ClickHouseAnalyticsEngine({
|
||||
baseUrl: chBaseUrl,
|
||||
basicAuth: process.env.OPT_ANALYTICS_CLICKHOUSE_BASIC_AUTH,
|
||||
database: process.env.OPT_ANALYTICS_CLICKHOUSE_DATABASE ?? "opt_analytics",
|
||||
});
|
||||
const identity: AnalyticsIdentity = {
|
||||
tenant_id: TENANT_ID,
|
||||
project_id: PROJECT_ID,
|
||||
principal_id: "svc_vehiq_oltp_extractor",
|
||||
};
|
||||
const sink = projectionSink({
|
||||
engine,
|
||||
identity,
|
||||
datasetIdFor: (partitionKey) => `vehiq.${partitionKey}`,
|
||||
});
|
||||
|
||||
const summary = await runVehiqOltpIngestion({
|
||||
repo,
|
||||
schemas,
|
||||
lineage,
|
||||
sourceQuery: query,
|
||||
oltpSink: sink,
|
||||
dimensionSink: sink,
|
||||
tenantId: TENANT_ID,
|
||||
projectId: PROJECT_ID,
|
||||
});
|
||||
console.log(JSON.stringify({ status: "ran", mode, summary }, null, 2));
|
||||
} finally {
|
||||
await close();
|
||||
}
|
||||
}
|
||||
|
||||
main().catch((err) => {
|
||||
console.error(JSON.stringify({ status: "error", detail: err instanceof Error ? err.message : String(err) }));
|
||||
process.exitCode = 1;
|
||||
});
|
||||
Reference in New Issue
Block a user