CI (SIAX Cloud) / integration (push) Skipped
masterplan-lock / check (push) Successful in 7s
CI (SIAX Cloud) / security (push) Successful in 17s
masterplan-lock / check (pull_request) Successful in 9s
Security Scan (SIAX Cloud) / security (push) Successful in 38s
CI (SIAX Cloud) / security (pull_request) Successful in 26s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 26s
CI (SIAX Cloud) / quality (push) Successful in 57s
CI (SIAX Cloud) / integration (pull_request) Successful in 39s
CI (SIAX Cloud) / quality (pull_request) Successful in 59s
Implements Simon's specified pattern (dealer_stock_entry/deal -> watermark/updated_at -> 0pt ingestion -> ClickHouse) as a P061 connector pair (PostgresWatermarkConnector, PostgresKeyedLookupConnector) plugged into the existing P062 IngestionRunner, writing to opt_dataset_projection via SiaxAnalyticsEngine.ingestProjection. Built here, not in admin/vehiq: that repo's own ADR-0035 (decisions 1/2/10, CI-enforced by validate_engine_firewall.py) forbids VehIQ code from ever holding a ClickHouse credential or hammering OLTP directly for analytics. See docs/P070_VEHIQ_OLTP_EXTRACTOR.md for the full rationale. Also corrects a real schema mismatch found while building this: production has no generic `updated_at` on any of the three target tables (verified live, read-only, 2026-09-25). dealer_stock_entry/deal use listed_at/ created_at as insert-time watermarks instead (documented caveat: in-place updates are not caught without a schema change); app.listing has no timestamp column at all and is deliberately not covered yet. Vehicle dimension lookup is narrow by construction: keyed on exactly the vehicle_id set referenced by the batch just read, never a scan of the 98M-row vehiq.vehicle table (connector-level validate() refuses that table by name; runner-level tests assert the query only ever touches referenced IDs). Verified: fixture-backed connector/runner tests (watermark+limit, same-timestamp tie at a page boundary, crash/resume, empty-table case, narrow-lookup query-count assertions) + a real disposable Postgres container (caught two real Date-handling bugs fixtures could not) + a read-only proof against actual production (server10, vehiq-db): the extractor's exact SQL returns zero rows, no error, confirming it is safe to schedule with zero current effect. .gitea/workflows/vehiq-oltp-ingest.yml is present, cron-scheduled, and defaults to VEHIQ_INGEST_MODE=preview (read-only, never writes) with no secrets configured — safe to merge with zero production effect until explicitly activated per the doc's "what activating this requires". Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
138 lines
5.4 KiB
JavaScript
138 lines
5.4 KiB
JavaScript
#!/usr/bin/env -S node
|
|
/**
|
|
* scripts/vehiq-oltp-ingest/run.ts
|
|
* ===================================
|
|
* Thin CLI shim over src/ingestion/vehiq-oltp-ingest-runner.ts. All the
|
|
* actual logic (connectors, the runner call, the checkpoint mapping) is in
|
|
* that blocking-typechecked, tested `src/` module; this file only reads
|
|
* `process.env`, wires real implementations, and prints a summary.
|
|
*
|
|
* TWO MODES, PREVIEW IS THE DEFAULT — this is the activation gate:
|
|
*
|
|
* VEHIQ_INGEST_MODE=preview (default)
|
|
* One bounded read-only connect+discover+read per source, against
|
|
* whatever checkpoint already exists. NEVER commits a checkpoint, NEVER
|
|
* calls the sink, NEVER touches ClickHouse. Safe to run against real
|
|
* production at any time — this is what proves "zero batches, zero
|
|
* writes" without activating anything.
|
|
*
|
|
* VEHIQ_INGEST_MODE=run
|
|
* The real, scheduled path: P062 runner, durable checkpoints, writes
|
|
* landing in `opt_dataset_projection`. Requires DATABASE_URL (0pt's own
|
|
* metadata store, for durable checkpoints) and the OPT_ANALYTICS_
|
|
* CLICKHOUSE_* vars (the write target) to be set, or it refuses to run
|
|
* rather than silently falling back to in-memory checkpoints or a
|
|
* no-op sink — see docs/P070_VEHIQ_OLTP_EXTRACTOR.md "What activating
|
|
* this in production actually requires".
|
|
*
|
|
* NO VEHIQ_SOURCE_DATABASE_URL AT ALL -> exits 0 having done nothing. This
|
|
* is what makes the scheduled workflow (.gitea/workflows/vehiq-oltp-
|
|
* ingest.yml) safe to leave enabled with no secret configured: cron fires,
|
|
* the script logs "not configured", exits 0, no error, no alert.
|
|
*/
|
|
import {
|
|
VEHIQ_OLTP_SOURCES,
|
|
previewVehiqOltpSource,
|
|
runVehiqOltpIngestion,
|
|
} from "../../src/ingestion/vehiq-oltp-ingest-runner";
|
|
import { makePostgresQueryImpl } from "../../src/connectors/postgres-query-impl";
|
|
import { createIngestionRepository } from "../../src/repositories/ingestion-repository";
|
|
import { SchemaRegistry } from "../../src/schema/schema-registry";
|
|
import { LineageStore } from "../../src/lineage/lineage-store";
|
|
import { ClickHouseAnalyticsEngine } from "../../src/analytics/engine/clickhouse-engine";
|
|
import { projectionSink } from "../../src/ingestion/projection-sink";
|
|
import type { AnalyticsIdentity } from "../../src/analytics/engine/siax-analytics-engine";
|
|
|
|
const TENANT_ID = process.env.VEHIQ_INGEST_TENANT_ID ?? "t_vehiq";
|
|
const PROJECT_ID = process.env.VEHIQ_INGEST_PROJECT_ID ?? "vehiq";
|
|
|
|
async function main(): Promise<void> {
|
|
const sourceUrl = process.env.VEHIQ_SOURCE_DATABASE_URL;
|
|
if (!sourceUrl) {
|
|
console.log(
|
|
JSON.stringify({ status: "not_configured", detail: "VEHIQ_SOURCE_DATABASE_URL is unset — nothing to do" }),
|
|
);
|
|
return;
|
|
}
|
|
|
|
const mode = process.env.VEHIQ_INGEST_MODE === "run" ? "run" : "preview";
|
|
const { query, close } = makePostgresQueryImpl({
|
|
connectionString: sourceUrl,
|
|
ssl: process.env.VEHIQ_SOURCE_PGSSL === "require" ? "require" : false,
|
|
statementTimeoutMs: process.env.VEHIQ_SOURCE_STATEMENT_TIMEOUT_MS
|
|
? Number(process.env.VEHIQ_SOURCE_STATEMENT_TIMEOUT_MS)
|
|
: undefined,
|
|
});
|
|
|
|
try {
|
|
if (mode === "preview") {
|
|
const repo = createIngestionRepository(process.env.DATABASE_URL);
|
|
const results = [];
|
|
for (const source of VEHIQ_OLTP_SOURCES) {
|
|
results.push(
|
|
await previewVehiqOltpSource(source, query, repo, {
|
|
sourceId: "vehiq_oltp",
|
|
tenantId: TENANT_ID,
|
|
projectId: PROJECT_ID,
|
|
limit: 100,
|
|
}),
|
|
);
|
|
}
|
|
console.log(JSON.stringify({ status: "preview", mode, results }, null, 2));
|
|
return;
|
|
}
|
|
|
|
// mode === "run" — the activation-gated real path.
|
|
if (!process.env.DATABASE_URL) {
|
|
throw new Error(
|
|
"VEHIQ_INGEST_MODE=run requires DATABASE_URL (0pt's own metadata store, for durable checkpoints) — " +
|
|
"refusing to run with in-memory checkpoints, which would silently re-read every row on every scheduled invocation",
|
|
);
|
|
}
|
|
const chBaseUrl = process.env.OPT_ANALYTICS_CLICKHOUSE_BASE_URL;
|
|
if (!chBaseUrl) {
|
|
throw new Error(
|
|
"VEHIQ_INGEST_MODE=run requires OPT_ANALYTICS_CLICKHOUSE_BASE_URL — refusing to run with no write target",
|
|
);
|
|
}
|
|
|
|
const repo = createIngestionRepository(process.env.DATABASE_URL);
|
|
const schemas = new SchemaRegistry(repo);
|
|
const lineage = new LineageStore(repo);
|
|
const engine = new ClickHouseAnalyticsEngine({
|
|
baseUrl: chBaseUrl,
|
|
basicAuth: process.env.OPT_ANALYTICS_CLICKHOUSE_BASIC_AUTH,
|
|
database: process.env.OPT_ANALYTICS_CLICKHOUSE_DATABASE ?? "opt_analytics",
|
|
});
|
|
const identity: AnalyticsIdentity = {
|
|
tenant_id: TENANT_ID,
|
|
project_id: PROJECT_ID,
|
|
principal_id: "svc_vehiq_oltp_extractor",
|
|
};
|
|
const sink = projectionSink({
|
|
engine,
|
|
identity,
|
|
datasetIdFor: (partitionKey) => `vehiq.${partitionKey}`,
|
|
});
|
|
|
|
const summary = await runVehiqOltpIngestion({
|
|
repo,
|
|
schemas,
|
|
lineage,
|
|
sourceQuery: query,
|
|
oltpSink: sink,
|
|
dimensionSink: sink,
|
|
tenantId: TENANT_ID,
|
|
projectId: PROJECT_ID,
|
|
});
|
|
console.log(JSON.stringify({ status: "ran", mode, summary }, null, 2));
|
|
} finally {
|
|
await close();
|
|
}
|
|
}
|
|
|
|
main().catch((err) => {
|
|
console.error(JSON.stringify({ status: "error", detail: err instanceof Error ? err.message : String(err) }));
|
|
process.exitCode = 1;
|
|
});
|