0pt/docs/P070_VEHIQ_OLTP_EXTRACTOR.md

P070 — VehIQ OLTP -> 0pt watermark extractor

Status: built, tested, verified read-only against production. Not scheduled/activated in production. Read "What activating this requires" before turning the workflow on.

What this is

A controlled, incremental extractor implementing exactly the pattern Simon specified (admin/vehiq 2026-09-25 pass, "Locked architecture decisions from Simon" / docs/adr/0035-bi-analytics-architecture-product-firewall.md):

OLTP operational tables -> read-only analytical extraction -> watermark ->
0pt ingestion -> ClickHouse (opt_dataset_projection)

Source: app.dealer_stock_entry and app.deal on production Postgres (vehiq-db, server10) — read-only SELECT, never a schema/config change, never touching wal_level or replication, never a permanent replica. Sink: SiaxAnalyticsEngine.ingestProjection (BI-P009, opt_dataset_projection in ClickHouse). Vehicle dimension: a narrow, keyed lookup against vehiq.vehicle (98M rows / 85GB) scoped to exactly the vehicle_ids referenced by the dealer_stock_entry batch just read — never SELECT * FROM vehicle, never a full scan.

Why this lives in admin/0pt, not admin/vehiq

The task that produced this work described building it as a new package inside admin/vehiq (e.g. packages/opt-ingest). Before writing any code, this repo's own docs/adr/0035-bi-analytics-architecture-product-firewall.md (decisions 1, 2, 10 — decided the same day, one-way-door, CI-enforced by tools/validate_engine_firewall.py) was read and is unambiguous:

VehIQ's own codebase must never get its own ClickHouse or DuckDB credentials, and must never import a ClickHouse/DuckDB client directly. ... Product code must never hammer the live OLTP Postgres with analytical queries directly.

And admin/vehiq's docs/CURRENT_STATE.md ("Connection details for a future API0 connector", same pass) says explicitly:

VehIQ's own codebase must never see any of this. This is 0pt's data plane; only a future admin/api0 connector should hold these details.

Building the extractor inside admin/vehiq would have failed validate_engine_firewall.py the moment it imported a ClickHouse client or held a CLICKHOUSE_* env var, and would have been the exact "temporary VehIQ-owned data-access surface" ADR-0035 names as the failure mode it exists to prevent. So this extractor was built here instead — admin/0pt, 0pt's own data plane, a sibling of admin/api0 (the mechanics belong next to the P060-P068 ingestion plane and the BI-P009/P010 ClickHouse engine that already live in this repo). admin/vehiq was not modified by this work.

This is a deliberate deviation from the task's literal repo suggestion, made because the suggestion conflicted with Simon's own same-day, explicit, locked architecture decision discovered while reading the target repo — not a judgment call made against that decision.

Real production schema does not match the task's assumed column names

The task (and admin/vehiq's docs/CURRENT_STATE.md) assumed all three OLTP tables carry a generic updated_at ("the columns already exist"). Verified live against production (server10, vehiq-db, read-only information_schema.columns, 2026-09-25) — this is not true:

table	live columns	usable watermark
app.dealer_stock_entry	id, tenant_id, vehicle_id, lot_location, floorplan_status, listed_at, sold_at	listed_at (NOT NULL, insert-time only)
app.deal	id, tenant_id, party_id, vehicle_id, sale_price_minor, currency, status, created_at	created_at (NOT NULL DEFAULT now(), insert-time only)
app.listing	id, tenant_id, vehicle_id, asking_price_minor, currency, channels, status	none

app.listing has no timestamp column of any kind today. admin/vehiq migration 0084_listing_deal_lifecycle_timestamps.sql (already written, adds published_at/unpublished_at to listing and closed_at/listing_id to deal) exists in the repo but is not yet applied to production (verified: those columns are absent live). Until it lands, there is no honest way to build a watermark connector for app.listing — one is not built here. This extractor covers dealer_stock_entry and deal only, not listing.

Caveat on the two sources that ARE built: listed_at/created_at are set once, at INSERT. Neither table has a column that advances when an existing row changes (sold_at/floorplan_status on dealer_stock_entry; status on deal). This extractor reliably sees every new row; it does not see an in-place UPDATE to a row it already read, because that row has already passed the watermark. Closing this gap needs either listing_id/closed_at (migration 0084, once applied) plus an equivalent for dealer_stock_entry, or a generic updated_at + trigger — a schema decision for Simon, not something this pass made unilaterally (per the hard "never a schema/config change" rule).

Also found and corrected while building: vehiq.vehicle's primary key column is vehicle_id, not id (verified via schema introspection only — no vehicle data rows were read to establish this).

Where the code lives

• src/connectors/postgres-watermark-connector.ts — the P061 Connector implementing WHERE watermark_col > $1 OR (watermark_col = $1 AND key_col > $2) ORDER BY watermark_col, key_col LIMIT $3 (the OR tuple form, not a bare >, so a same-timestamp tie at a page boundary is not silently dropped — see the file's own header for the bug this avoids). Explicit column allow-list, never SELECT *. validate() structurally refuses any config pointed at vehiq.vehicle.
• src/connectors/postgres-keyed-lookup-connector.ts — the narrow vehicle-dimension connector. Config carries an already-resolved, finite keys array; connect() refuses an empty list rather than defaulting to a full read; every query is WHERE vehicle_id = ANY($1) chunked to keys_per_query (default 500).
• src/connectors/postgres-query-impl.ts — the real postgres (postgres.js) implementation of the connectors' injectable QueryImpl, used only by the orchestration script. Refuses to execute anything whose first keyword isn't SELECT, independent of the DB role's own privileges.
• src/ingestion/projection-sink.ts — adapts the P062 runner's IngestionSink onto SiaxAnalyticsEngine.ingestProjection (writes opt_dataset_projection). No ClickHouse import — only the SiaxAnalyticsEngine port type.
• src/ingestion/vehiq-oltp-ingest-runner.ts — the orchestration: previewVehiqOltpSource (one bounded read-only read, never commits, safe against production at any time) and runVehiqOltpIngestion (the real P062-runner path).
• scripts/vehiq-oltp-ingest/run.ts — thin CLI (pnpm run vehiq:oltp-ingest), env-driven, defaults to preview mode.
• Tests: tests/connectors/postgres-watermark-connector.test.ts, tests/connectors/postgres-keyed-lookup-connector.test.ts, tests/ingestion/vehiq-oltp-ingest-runner.test.ts.

Checkpoint field mapping

Simon's own specified checkpoint columns — source, table, last_updated_at, last_id, batch_count, checksum, ingested_at — mapped onto the existing P062 IngestionCheckpoint (reused, not duplicated, per the "no double truths" rule):

Simon's field	Canonical field
source	IngestionCheckpoint.source_id
table	IngestionCheckpoint.partition_key
last_updated_at	decoded from IngestionCheckpoint.cursor (JSON WatermarkCursor)
last_id	decoded from IngestionCheckpoint.cursor
checksum	decoded from IngestionCheckpoint.cursor — sha256 of the page just landed
batch_count	IngestionCheckpoint.records_committed (cumulative); a per-run page count is also in each run's PartitionRunReport
ingested_at	IngestionCheckpoint.committed_at

Durable storage: createIngestionRepository(DATABASE_URL) — Postgres-backed when DATABASE_URL (0pt's own metadata store) is set, in-memory only in non-production/tests. run mode refuses to start without DATABASE_URL explicitly, so a scheduled run can never silently fall back to re-reading everything from the beginning.

Testing

Fixture-backed (no real infra): watermark + limit correctness, a same-timestamp tie at a page boundary, resume-from-checkpoint after a simulated crash (via a sink that throws mid-run, mirroring tests/p0/ingestion-runner.test.ts's own technique), the empty-table case (zero rows, zero envelopes, no throw), and the narrow vehicle-dimension lookup asserted at the query level (every id ever bound into a query is a subset of the referenced set; a vehicle never referenced by the batch is never queried).

Also verified against a real, disposable Postgres container (not just fixtures), both seeded and truncated-to-empty, before touching production — caught and fixed two real bugs the fixture-only tests could not have caught:

1. timestamptz columns come back from the real driver as JS Date objects, not strings — the initial cursor-encoding logic did String(date) (locale-dependent, useless as a stored watermark) instead of .toISOString(). Fixed; regression test added (tests/connectors/postgres-watermark-connector.test.ts, "real Postgres driver value shapes").
2. The same Date-vs-string mismatch tripped the P063 schema-conformance gate on every row (field "listed_at" is object, expected string) until the connector's toRecord() was made to normalize every Date-valued payload field to ISO before building the envelope.

Verified read-only against actual production (server10, vehiq-db, via fleet), 2026-09-25: the connector's exact SQL (SELECT id, tenant_id, vehicle_id, lot_location, floorplan_status, listed_at, sold_at FROM app.dealer_stock_entry ORDER BY listed_at, id LIMIT 100 and the equivalent for app.deal) executed directly, read-only, and returned zero rows, no error — confirming both that the extractor's query shape is valid against the live schema and that there is genuinely nothing to extract yet. vehiq.vehicle's vehicle_id primary-key column name was confirmed via information_schema.columns only — no vehicle data rows were read. The compiled service itself was not run against production — only its exact SQL, via read-only psql, matching this pass's "prove it does nothing, don't deploy a live service yet" scope.

Deferred: full vehicle-dimension backfill

Not built or run in this pass, per the task's explicit instruction. If/when a full dimension_vehicle snapshot is ever wanted (as opposed to the permanent narrow per-batch lookup this extractor does forever), it needs its own deliberate decision:

• Chunking: page through vehiq.vehicle by primary key range (vehicle_id > :last ORDER BY vehicle_id LIMIT :n), never an unbounded scan — the same shape PostgresWatermarkConnector already implements, just pointed at a PK range instead of a timestamp (this connector's own validate() refuses vehiq.vehicle by name specifically to keep this a distinct, deliberate choice rather than an accidental default).
• Throttle: an explicit delay between chunks, watching server10's iostat/vmstat/CPU while it runs — 98M rows / 85GB is enough to matter even read-only.
• Checksum + reconciliation: a row-count and content checksum comparison between source and opt_dataset_projection after the backfill, recorded, before trusting it.
• Watermark handoff: the backfill's end position must hand off cleanly to the narrow per-batch lookup this extractor already does, so nothing is double-counted or gapped at the boundary.

This mirrors the numbered runbook admin/vehiq's docs/CURRENT_STATE.md already wrote for the general CDC case (not executed there either) — the shape is consistent, deliberately, rather than inventing a second convention.

Deferred: app.listing extraction

Blocked on a schema decision, not an engineering gap: app.listing has no timestamp column today. Unblocking it needs one of:

1. Apply admin/vehiq migration 0084_listing_deal_lifecycle_timestamps.sql (adds published_at/unpublished_at) — a decision for whoever owns that repo's migration pipeline, not this pass.
2. Add a generic updated_at + trigger to dealer_stock_entry/deal/listing — closes both this gap and the "insert-only watermark misses in-place updates" caveat above in one move, but is a real schema change on production and needs Simon's explicit sign-off per the hard rule against touching production schema unsupervised.

Once either lands, adding a third VehiqOltpSourceDef entry to VEHIQ_OLTP_SOURCES (src/ingestion/vehiq-oltp-ingest-runner.ts) is the only code change needed — the connector, runner, sink and tests are all already generic across sources.

What activating this in production actually requires

Nothing here is scheduled or running. To turn it on:

1. A read-only Postgres role on vehiq-db (server10) with SELECT on exactly app.dealer_stock_entry and app.deal (not the whole app schema, not vehiq.vehicle beyond what the narrow lookup needs) — creating this role is a production database change and should be coordinated explicitly, the same posture admin/vehiq's own CDC runbook already takes for a bigger version of this same question. Its connection string becomes VEHIQ_SOURCE_DATABASE_URL.
2. DATABASE_URL — 0pt's own metadata Postgres, for durable checkpoints. Already exists for every other 0pt capability; reuse it, do not provision a new one for this.
3. OPT_ANALYTICS_CLICKHOUSE_BASE_URL / OPT_ANALYTICS_CLICKHOUSE_BASIC_AUTH — the already-deployed opt-analytics-clickhouse on server6 (internal-only, reachable from containers on the coolify network — see admin/vehiq's docs/CURRENT_STATE.md, "0pt production ClickHouse 26.8 LTS"). A real least-privilege application user should be provisioned for this connector specifically, not the schema-admin opt_admin user mentioned there.
4. VEHIQ_INGEST_MODE=run on the scheduled workflow (default is preview, which never writes).
5. A decision on whether .gitea/workflows/vehiq-oltp-ingest.yml's cron schedule should actually be enabled (it is present and correctly configured, but Gitea Actions secrets are not set on this repo by this pass — an unset VEHIQ_SOURCE_DATABASE_URL makes every scheduled run a clean no-op, so leaving the workflow file in place with no secrets configured is safe, but is not the same as "this is now extracting real data").

None of steps 1-3 were done in this pass — they require production credentials/role provisioning this pass does not have standing authority to create unsupervised.