1d682f52c8b64df94324127faf1cd4e2a6eea0ed
9
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
07158099ee |
feat(ingestion): VehIQ OLTP -> ClickHouse watermark extractor (P070)
CI (SIAX Cloud) / integration (push) Skipped
masterplan-lock / check (push) Successful in 7s
CI (SIAX Cloud) / security (push) Successful in 17s
masterplan-lock / check (pull_request) Successful in 9s
Security Scan (SIAX Cloud) / security (push) Successful in 38s
CI (SIAX Cloud) / security (pull_request) Successful in 26s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 26s
CI (SIAX Cloud) / quality (push) Successful in 57s
CI (SIAX Cloud) / integration (pull_request) Successful in 39s
CI (SIAX Cloud) / quality (pull_request) Successful in 59s
Implements Simon's specified pattern (dealer_stock_entry/deal -> watermark/updated_at -> 0pt ingestion -> ClickHouse) as a P061 connector pair (PostgresWatermarkConnector, PostgresKeyedLookupConnector) plugged into the existing P062 IngestionRunner, writing to opt_dataset_projection via SiaxAnalyticsEngine.ingestProjection. Built here, not in admin/vehiq: that repo's own ADR-0035 (decisions 1/2/10, CI-enforced by validate_engine_firewall.py) forbids VehIQ code from ever holding a ClickHouse credential or hammering OLTP directly for analytics. See docs/P070_VEHIQ_OLTP_EXTRACTOR.md for the full rationale. Also corrects a real schema mismatch found while building this: production has no generic `updated_at` on any of the three target tables (verified live, read-only, 2026-09-25). dealer_stock_entry/deal use listed_at/ created_at as insert-time watermarks instead (documented caveat: in-place updates are not caught without a schema change); app.listing has no timestamp column at all and is deliberately not covered yet. Vehicle dimension lookup is narrow by construction: keyed on exactly the vehicle_id set referenced by the batch just read, never a scan of the 98M-row vehiq.vehicle table (connector-level validate() refuses that table by name; runner-level tests assert the query only ever touches referenced IDs). Verified: fixture-backed connector/runner tests (watermark+limit, same-timestamp tie at a page boundary, crash/resume, empty-table case, narrow-lookup query-count assertions) + a real disposable Postgres container (caught two real Date-handling bugs fixtures could not) + a read-only proof against actual production (server10, vehiq-db): the extractor's exact SQL returns zero rows, no error, confirming it is safe to schedule with zero current effect. .gitea/workflows/vehiq-oltp-ingest.yml is present, cron-scheduled, and defaults to VEHIQ_INGEST_MODE=preview (read-only, never writes) with no secrets configured — safe to merge with zero production effect until explicitly activated per the doc's "what activating this requires". Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
655a71ebd7 |
feat(HSP-085): migration-studio — T01 ej importerbart döljs EJ ur nämnaren (ratio = importable/total), T02 session-resume med tenant-check + checkpoint + resolved conflicts, T03 coverage-claim SCOPED (aldrig komplett M365-ersättning), 8 tester
masterplan-lock / check (push) Successful in 7s
masterplan-lock / check (pull_request) Successful in 5s
Security Scan (SIAX Cloud) / security (push) Successful in 15s
CI (SIAX Cloud) / security (push) Successful in 17s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 13s
CI (SIAX Cloud) / security (pull_request) Successful in 15s
CI (SIAX Cloud) / quality (push) Successful in 36s
CI (SIAX Cloud) / quality (pull_request) Successful in 37s
CI (SIAX Cloud) / integration (pull_request) Successful in 43s
CI (SIAX Cloud) / integration (push) Has been skipped
|
||
|
|
85b0e6cd72 |
feat(HSP-084): cutover — T01 detectDualWriters (båda aktiva = STOPPAD), T02 handleLateSourceChanges (carry eller stopp, aldrig tyst förlorad), T03 buildRollbackPlan (targetWritesCarriedBack + käll-writer återaktiveras + konsistensverifiering), 8 tester
CI (SIAX Cloud) / quality (pull_request) Failing after 38s
CI (SIAX Cloud) / integration (push) Has been skipped
masterplan-lock / check (push) Successful in 4s
CI (SIAX Cloud) / security (push) Successful in 15s
Security Scan (SIAX Cloud) / security (push) Successful in 14s
masterplan-lock / check (pull_request) Successful in 5s
CI (SIAX Cloud) / security (pull_request) Successful in 17s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 16s
CI (SIAX Cloud) / integration (pull_request) Successful in 26s
CI (SIAX Cloud) / quality (push) Failing after 29s
|
||
|
|
45dd7dc6e2 |
feat(HSP-067): exit-manifest — T01 open formats only (portabel till icke-SIAX-mål), T02 notice-krav följer tillåtna artefakter, T03 nonExportable EXPLIKIT med skäl (aldrig tyst utelämnat), 7 tester
CI (SIAX Cloud) / integration (push) Has been skipped
masterplan-lock / check (push) Successful in 4s
CI (SIAX Cloud) / security (push) Successful in 15s
masterplan-lock / check (pull_request) Successful in 12s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 14s
CI (SIAX Cloud) / security (pull_request) Successful in 14s
CI (SIAX Cloud) / quality (pull_request) Successful in 32s
Security Scan (SIAX Cloud) / security (push) Successful in 1m21s
CI (SIAX Cloud) / quality (push) Successful in 1m34s
CI (SIAX Cloud) / integration (pull_request) Successful in 1m46s
|
||
|
|
57d5ff36fe |
feat(HSP-083): migration-runner — InMemoryCheckpointStore (T02 cursor commit EFTER durable target-ack), idempotent-sink replay-säkerhet (T01 krasch → replay räknad, inga dubbletter), T03 VERIFIED kräver per-partition content-hash (count-only = UNVERIFIED), 6 tester
CI (SIAX Cloud) / integration (push) Has been skipped
masterplan-lock / check (push) Successful in 4s
CI (SIAX Cloud) / security (push) Successful in 12s
CI (SIAX Cloud) / security (pull_request) Successful in 19s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 18s
CI (SIAX Cloud) / integration (pull_request) Successful in 26s
masterplan-lock / check (pull_request) Successful in 31s
Security Scan (SIAX Cloud) / security (push) Successful in 38s
CI (SIAX Cloud) / quality (push) Successful in 1m15s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m17s
|
||
|
|
23d05d9379 |
feat(HSP-082): migration-mapping — decideEntityMerge (T01 identiska namn EJ ihopslagna utan bevis), transformRecord (T02 åäö byte-exakt + UTC-normalisering med original-offset), aclBlockingCheck (T03 saknad ACL-mappning BLOCKERANDE), 10 tester
CI (SIAX Cloud) / quality (push) Successful in 1m7s
CI (SIAX Cloud) / integration (push) Has been skipped
masterplan-lock / check (push) Successful in 4s
masterplan-lock / check (pull_request) Successful in 5s
CI (SIAX Cloud) / security (pull_request) Successful in 15s
Security Scan (SIAX Cloud) / security (push) Successful in 23s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 18s
CI (SIAX Cloud) / integration (pull_request) Successful in 26s
CI (SIAX Cloud) / security (push) Successful in 39s
CI (SIAX Cloud) / quality (pull_request) Successful in 36s
|
||
|
|
c1c7a33d65 |
feat(HSP-081): discovery-inventory — inventory-report (T01 write-scoped sources EXCLUDED från inventeringen, T02 persondata REDACTED till referenser rekursivt, T03 otillgängliga källor = okända med real fel — aldrig tyst droppade), 6 tester
CI (SIAX Cloud) / integration (push) Has been skipped
masterplan-lock / check (push) Successful in 5s
Security Scan (SIAX Cloud) / security (push) Successful in 15s
CI (SIAX Cloud) / security (push) Successful in 17s
masterplan-lock / check (pull_request) Successful in 4s
CI (SIAX Cloud) / security (pull_request) Successful in 15s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 13s
CI (SIAX Cloud) / quality (push) Successful in 51s
CI (SIAX Cloud) / integration (pull_request) Successful in 30s
CI (SIAX Cloud) / quality (pull_request) Successful in 37s
|
||
|
|
b5ab885a67 |
feat(analytics): BI-P129 — ConnectorProvider, and four gates on where data lands
CI (SIAX Cloud) / security (push) Successful in 27s
Security Scan (SIAX Cloud) / security (push) Successful in 30s
CI (SIAX Cloud) / quality (push) Successful in 39s
CI (SIAX Cloud) / security (pull_request) Successful in 43s
CI (SIAX Cloud) / integration (push) Successful in 51s
CI (SIAX Cloud) / quality (pull_request) Successful in 54s
CI (SIAX Cloud) / integration (pull_request) Successful in 1m15s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 13s
BI-P129 names four provider kinds (native, dlt, Meltano/Singer, Airbyte-if-
licensed). This commit ships the seam and the `native` provider; dlt and
Singer follow on this branch.
What is deliberately NOT here: a second ingestion stack. 0pt already has the
P061 connector contract, the P062 runner with durable checkpoints and
crash/resume tests, the schema registry, the quality gate and the idempotency
ledger. A provider's job is to hand back a P061 `Connector`; the highway is
the one that exists.
What is new is the question the P061 stack has no opinion about: an ingestion
run has to LAND somewhere, and where it lands is an analytics contract
question. Four fail-closed gates, each attacked with the guard (must refuse)
AND without it (must succeed — otherwise the guard is decoration):
1. tier — source data lands in RAW, or NORMALIZED with a declared
normalization. Never directly in CURATED/SERVING. BI-P003's "no business
dashboard reads arbitrary raw ingestion tables" is defeated in one line
by relabelling the landing table SERVING; the tier is therefore checked
at WRITE, not only at read.
2. contract coverage — every field discovery actually reported must be
declared in the BI-P004 contract. An undeclared column is an
unclassified column. The counterfactual test lands `personnummer` into a
RAW table whose contract says pii_classification "none", and the runner
reports success: that is the harm, shown rather than asserted.
3. physical home — RAW/NORMALIZED land as Parquet/Iceberg in ST0RE
(st0re#45 shapes), never straight into ClickHouse.
4. provider readiness — a provider whose own providerReadiness() says
verified:false cannot run in a production runtime without a recorded,
named exception. This is the structural form of the honesty rule 0pt#35
set with MetricFlow.
Airbyte is a declared kind with NO implementation: its server is ELv2, which
restricts offering the software as a managed service, and that is exactly
what 0pt does. Registering or planning one is refused at the registry, in one
place, so a future licence review has a single constant to change.
BI-P129 DoD — "two real source types ingest into canonical analytics tiers" —
is proven with the two real, heterogeneous P061 connectors: a JSONL file on
actual disk (cursor = line offset, real CDC deletes) and an HTTP JSON API
driven through a real fetch/Response cycle (cursor = opaque server token).
Both run through the real runner into the declared RAW tier, paged, with
9 rows landed each and distinct envelope ids.
Verification: 24 tests. Mutation-checked — neutering gates 1 and 2 fails 5 of
them, so they are load-bearing rather than incidental.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
|
9267576b7c |
feat(data-os): P060-P068 durable ingestion plane — source classification gate, connector contract, common envelope, schema evolution, quality enforcement, queryable lineage, idempotent backfill
CI (SIAX Cloud) / integration (push) Successful in 42s
CI (SIAX Cloud) / quality (push) Successful in 1m17s
CI (SIAX Cloud) / security (push) Successful in 46s
Security Scan (SIAX Cloud) / security (push) Successful in 31s
CI (SIAX Cloud) / integration (pull_request) Successful in 36s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m4s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 27s
CI (SIAX Cloud) / security (pull_request) Successful in 24s
BEFORE_STATE (verified, with evidence):
* P060 PARTIAL/BROKEN. The registry persisted rights/provenance/health/
freshness to Postgres, but carried no authorization, no schedule, no
checkpoint, no schema and no owner distinct from created_by — and the
activation DoD was inverted: postgres-source-repository.ts wrote
`${input.status ?? "active"}`, so a source with no rights, no provenance
and no authorization was ACTIVE the instant it was registered. No
activation operation existed at all.
* P061 MISSING. services/api/sources/connectors.py had per-source fetch_*()
functions only — no lifecycle, no cursor, no resume, no health, no
conformance suite.
* P062 MISSING, already documented as such in
services/api/tests/test_known_gaps.py:10-19 (no CDC pipeline anywhere;
workers/pipeline.py has no persistence at all).
* P063 MISSING. schema_version existed as a free string; no comparison,
no policy, no versioned source schema.
* P064 PARTIAL. Rules/runs/issues/scores existed, but a failing rule only
opened an issue and the data flowed on. No quarantine anywhere.
* P065 DUPLICATE (confirmed). Two ER engines: src/er/entity-resolution-
service.ts (canonical) and services/api/workers/resolve.py (simplified,
still called by pipeline.py:110).
* P067 PARTIAL. Lineage was emitted to AUD0 but nothing here could ANSWER a
lineage question — no store, no query, no route.
* P068 MISSING, documented in test_known_gaps.py:44-52 ("no delivery-store
persistence layer exists to dedupe a backfill against").
BUILT (only what was PARTIAL/BROKEN/MISSING):
* P060 sql/007_ingestion_plane.sql adds authorization/schedule/schemas/
checkpoints/ledger/quarantine/lineage tables, owner_id, and flips the
opt_sources status DEFAULT to 'pending' so even a raw psql INSERT cannot
produce an unclassified-but-active source. SourceRegistryService gains
classifySource/activateSource/deactivateSource + authorization/schedule
accessors; registration NEVER activates (asking for status:"active" goes
through the gate, not around it); a service identity cannot authorize a
source. Routes: activate (GET classification / POST activate),
authorization, schedule.
* P061 src/connectors/connector.ts — all nine operations on one interface.
src/connectors/conformance.ts — a runnable suite (15 checks, no "skip"
verdict). HttpJsonConnector (real HTTP, opaque cursor) and
JsonlFileConnector (real file, line offsets, CDC deletes) both pass it.
The suite caught a real defect while being written: an API that drops
next_cursor at the end left the connector unable to resume, which is why
HttpJsonConnectorConfig has cursor_field.
* P062 src/types/ingestion.ts + src/ingestion/{envelope,ingestion-runner}.ts.
One envelope for batch/stream/cdc; durable per-(source,partition)
checkpoints; resume proven by killing a run inside the sink.
* P063 src/schema/{record-schema,compatibility,schema-registry}.ts. Two
enforcement points, both upstream of any consumer write: a discovery gate
before the first read, and per-record conformance before the page reaches
the sink. A producer cannot widen its own policy retroactively.
* P064 src/quality/quality-policy.ts composes the EXISTING ExpectationEngine
(not a second engine) and adds the missing consequence: block /
quarantine / warn, with quarantine durable and recoverable. No policy is
reported as "none_declared", never as a pass; an unexecutable expectation
counts as a violation, never a silent pass.
* P067 src/lineage/{lineage-edge,lineage-store}.ts + GET /api/0pt/lineage.
AUD0 stays the system of record; this is 0pt's own queryable projection.
* P068 opt_ingestion_ledger + IngestionRunner.backfill: bounded, idempotent
on repeat, leaves the live checkpoint alone, still re-emits real changes.
A BUG THIS PASS WROTE AND CAUGHT: the first IngestionRepository draft
combined the idempotency check and its recording into one atomic call
BEFORE the sink. The crash/resume test then delivered 7 of 10 records — the
page in flight when the sink died was already marked delivered, so the
resumed run skipped it as a duplicate. Permanent, silent data loss. Fixed by
splitting peekDeliveries/commitDeliveries and committing both the ledger and
the checkpoint only after the sink resolves.
NOT DONE, stated rather than deferred quietly:
* P065 is NOT met. resolve.py is still a second simplified production
engine. Closing it means rewriting it as an adapter over the canonical
service, which cannot be verified from this repo today: services/api is
not built by this repo's Dockerfile, api.0pt.app answers 404, and NO CI
workflow runs its pytest suite. The duplication is now stated at both
sites in code and the removal plan is in STATUS.md. The new ingestion
plane performs no ER, so it adds no third engine.
* P066 (Iceberg) untouched — ST0RE is owned by another workflow. Contract
read only.
* Delivery is AT-LEAST-ONCE, not exactly-once (no per-partition run lock).
* No CDC *source* exists; the contract is met, a CDC connector is not built.
* 0pt does not schedule itself — the schedule row is a declaration ACT0
would read.
VERIFIED: pnpm typecheck clean; pnpm test 365 passed / 25 skipped (was
311/18); pnpm build clean with all four new routes in the route table; lint
unchanged at 12 pre-existing errors. Real Postgres 16
(tests/p0/ingestion-plane-postgres.realinfra.test.ts, 7 passed) plus a real
`docker restart` of that container with an identical read-back through the
production code path — that is P060's "a restart preserves all sources",
measured.
DEPLOYMENT REALITY (curl, 2026-09-04): 0pt.app/api/health returns 200 from
Vercel with db:not_configured; 0pt.app/api/0pt/sources returns 404;
api.0pt.app/health returns 404; cd.yml still guards on a placeholder Coolify
app id. None of the capability layer runs in production. Nothing here is
LIVE_VERIFIED and STATUS.md says so.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|