Claude CodeandClaude Sonnet 5
07158099ee
CI (SIAX Cloud) / integration (push) Skipped
masterplan-lock / check (push) Successful in 7s
CI (SIAX Cloud) / security (push) Successful in 17s
masterplan-lock / check (pull_request) Successful in 9s
Security Scan (SIAX Cloud) / security (push) Successful in 38s
CI (SIAX Cloud) / security (pull_request) Successful in 26s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 26s
CI (SIAX Cloud) / quality (push) Successful in 57s
CI (SIAX Cloud) / integration (pull_request) Successful in 39s
CI (SIAX Cloud) / quality (pull_request) Successful in 59s
Implements Simon's specified pattern (dealer_stock_entry/deal -> watermark/updated_at -> 0pt ingestion -> ClickHouse) as a P061 connector pair (PostgresWatermarkConnector, PostgresKeyedLookupConnector) plugged into the existing P062 IngestionRunner, writing to opt_dataset_projection via SiaxAnalyticsEngine.ingestProjection. Built here, not in admin/vehiq: that repo's own ADR-0035 (decisions 1/2/10, CI-enforced by validate_engine_firewall.py) forbids VehIQ code from ever holding a ClickHouse credential or hammering OLTP directly for analytics. See docs/P070_VEHIQ_OLTP_EXTRACTOR.md for the full rationale. Also corrects a real schema mismatch found while building this: production has no generic `updated_at` on any of the three target tables (verified live, read-only, 2026-09-25). dealer_stock_entry/deal use listed_at/ created_at as insert-time watermarks instead (documented caveat: in-place updates are not caught without a schema change); app.listing has no timestamp column at all and is deliberately not covered yet. Vehicle dimension lookup is narrow by construction: keyed on exactly the vehicle_id set referenced by the batch just read, never a scan of the 98M-row vehiq.vehicle table (connector-level validate() refuses that table by name; runner-level tests assert the query only ever touches referenced IDs). Verified: fixture-backed connector/runner tests (watermark+limit, same-timestamp tie at a page boundary, crash/resume, empty-table case, narrow-lookup query-count assertions) + a real disposable Postgres container (caught two real Date-handling bugs fixtures could not) + a read-only proof against actual production (server10, vehiq-db): the extractor's exact SQL returns zero rows, no error, confirming it is safe to schedule with zero current effect. .gitea/workflows/vehiq-oltp-ingest.yml is present, cron-scheduled, and defaults to VEHIQ_INGEST_MODE=preview (read-only, never writes) with no secrets configured — safe to merge with zero production effect until explicitly activated per the doc's "what activating this requires". Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
45 lines
1.9 KiB
YAML
45 lines
1.9 KiB
YAML
name: VehIQ OLTP extractor (0pt ingestion)
|
|
|
|
# P070 — scheduled poll of app.dealer_stock_entry/app.deal (VehIQ, server10)
|
|
# into opt_dataset_projection (ClickHouse), via the watermark extractor in
|
|
# src/ingestion/vehiq-oltp-ingest-runner.ts. See docs/P070_VEHIQ_OLTP_
|
|
# EXTRACTOR.md for the full design and "what activating this requires".
|
|
#
|
|
# SAFE BY DEFAULT: VEHIQ_INGEST_MODE defaults to "preview" (read-only, never
|
|
# commits a checkpoint, never writes) below. With no VEHIQ_SOURCE_DATABASE_URL
|
|
# secret configured on this repo, the script logs "not_configured" and exits
|
|
# 0 — this workflow can be left enabled indefinitely with zero effect on
|
|
# anything until BOTH the secret is added AND VEHIQ_INGEST_MODE is changed
|
|
# to "run". Neither is done by this workflow file itself.
|
|
on:
|
|
workflow_dispatch:
|
|
schedule:
|
|
- cron: "*/15 * * * *"
|
|
|
|
jobs:
|
|
extract:
|
|
runs-on: ubuntu-latest
|
|
env:
|
|
VEHIQ_SOURCE_DATABASE_URL: ${{ secrets.VEHIQ_SOURCE_DATABASE_URL }}
|
|
DATABASE_URL: ${{ secrets.DATABASE_URL }}
|
|
OPT_ANALYTICS_CLICKHOUSE_BASE_URL: ${{ secrets.OPT_ANALYTICS_CLICKHOUSE_BASE_URL }}
|
|
OPT_ANALYTICS_CLICKHOUSE_BASIC_AUTH: ${{ secrets.OPT_ANALYTICS_CLICKHOUSE_BASIC_AUTH }}
|
|
OPT_ANALYTICS_CLICKHOUSE_DATABASE: opt_analytics
|
|
# Activation gate — see docs/P070_VEHIQ_OLTP_EXTRACTOR.md. Flip to
|
|
# "run" only once steps 1-4 in that doc's "What activating this
|
|
# requires" section are actually done.
|
|
VEHIQ_INGEST_MODE: preview
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- uses: actions/setup-node@v4
|
|
with:
|
|
node-version: "22"
|
|
- name: Enable corepack + pin pnpm@9
|
|
run: |
|
|
corepack enable
|
|
corepack prepare pnpm@9 --activate
|
|
- name: Install (frozen lockfile)
|
|
run: pnpm install --frozen-lockfile
|
|
- name: Run the extractor (preview mode unless activated)
|
|
run: pnpm run vehiq:oltp-ingest
|