Parallellsessions-merge lämnade 12 eslint-errors på main (no-explicit-any i
graph-mailbox-adapter + 2 p0-tester). Graph-JSON:n är genuint ostrukturerad
— medveten, avgränsad eslint-undantag med skäl (DoD §13); p0-testerna
typade med Record/cast. Lint 0 errors; tsc grön; 1141/1141.
Implements Simon's specified pattern (dealer_stock_entry/deal ->
watermark/updated_at -> 0pt ingestion -> ClickHouse) as a P061 connector
pair (PostgresWatermarkConnector, PostgresKeyedLookupConnector) plugged
into the existing P062 IngestionRunner, writing to opt_dataset_projection
via SiaxAnalyticsEngine.ingestProjection.
Built here, not in admin/vehiq: that repo's own ADR-0035 (decisions 1/2/10,
CI-enforced by validate_engine_firewall.py) forbids VehIQ code from ever
holding a ClickHouse credential or hammering OLTP directly for analytics.
See docs/P070_VEHIQ_OLTP_EXTRACTOR.md for the full rationale.
Also corrects a real schema mismatch found while building this: production
has no generic `updated_at` on any of the three target tables (verified
live, read-only, 2026-09-25). dealer_stock_entry/deal use listed_at/
created_at as insert-time watermarks instead (documented caveat: in-place
updates are not caught without a schema change); app.listing has no
timestamp column at all and is deliberately not covered yet.
Vehicle dimension lookup is narrow by construction: keyed on exactly the
vehicle_id set referenced by the batch just read, never a scan of the
98M-row vehiq.vehicle table (connector-level validate() refuses that table
by name; runner-level tests assert the query only ever touches referenced
IDs).
Verified: fixture-backed connector/runner tests (watermark+limit,
same-timestamp tie at a page boundary, crash/resume, empty-table case,
narrow-lookup query-count assertions) + a real disposable Postgres
container (caught two real Date-handling bugs fixtures could not) + a
read-only proof against actual production (server10, vehiq-db): the
extractor's exact SQL returns zero rows, no error, confirming it is safe
to schedule with zero current effect.
.gitea/workflows/vehiq-oltp-ingest.yml is present, cron-scheduled, and
defaults to VEHIQ_INGEST_MODE=preview (read-only, never writes) with no
secrets configured — safe to merge with zero production effect until
explicitly activated per the doc's "what activating this requires".
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Recorded decision (Simon 2026-08-31): personal data may be stored under
legitimate interest ONLY as business contacts bound to a company/org entity.
Gate enforces it: legitimate_interest requires purpose=business_contact and
entity_linked_only=true; recordStored() requires entity_id (orgnr/rgs/LEI,
hashed) and refuses person-only rows. +1 test; suite 167 pass | 1 skip;
typecheck clean.
PiiGateService makes any future personal-data source (ratsit.se and other
directory domains) technically GDPR-OK before the first byte is stored:
- recorded legal basis (art. 6) with basis_ref evidence, no policy => fail-closed
- data minimisation via field allowlist (art. 5(1)(c))
- storage limitation + enforceRetention purge (art. 5(1)(e))
- sha256-only identity keys + log redaction (art. 25/32)
- accessStatement / erase subject-rights surfaces (art. 15/17)
- robots/rate confirmation required for web+api sources; no stealth/bypass
- audit trail via AUD0 at risk L3 (0pt.pii_*)
- gate enabled by default in SourceQueryService
10 new tests; full suite 166 pass | 1 skip; typecheck (blocking gate) clean.
connection_config.sqlEndpoint on POST /api/0pt/sources/{sourceId}/query
was passed straight from the request body to the mssql TDS driver as the
outbound connection target, with only its http(s):// scheme stripped —
no hostname allowlist or private/loopback/link-local/cloud-metadata
guard. This route already requires a Zitadel bearer token + tenant match
(P014 tenant guard), so this was not anonymously exploitable, but it is
the exact SSRF shape found and exploited-in-principle elsewhere in this
estate tonight (press0), so it's worth closing as defense-in-depth.
Adds src/lib/security/ssrf-guard.ts: resolves the endpoint's hostname
and rejects any address in a private/loopback/link-local/cloud-metadata/
unspecified range (RFC1918, 127.0.0.0/8, 169.254.0.0/16 incl. the
169.254.169.254 metadata address, ::1, fc00::/7, fe80::/10, IPv4-mapped
IPv6), failing closed on DNS resolution errors. The range-check logic is
ported 1:1 from press0's src/lib/security/ssrf-guard.ts built the same
night for the same bug class against a fetch()-based URL there; this
module's surface differs because sqlEndpoint is a bare host[:port], not
a fetchable URL. The DNS lookup is injectable (mirrors this adapter's
existing fetchImpl pattern) so tests are hermetic.
FabricOneLakeAdapter.getPool() now calls the guard before loadDriver()/
getToken()/opening the TDS pool, so a blocked target never reaches a
real outbound connection attempt or spends an AAD token exchange.
Tests: new tests/p0/fabric-onelake-ssrf-guard.test.ts unit-tests the
guard module directly (range boundaries, IPv4-mapped-IPv6 bypass
attempt, fail-closed on DNS failure/empty answer). New adapter-level
tests in tests/p0/fabric-onelake-adapter.test.ts prove (a) a
private-IP-resolving sqlEndpoint is rejected before fetch/connect is
ever called, (b) a literal loopback sqlEndpoint is rejected the same
way, and (c) a normal external Fabric hostname still passes the guard
and completes the query — the guard doesn't false-block legitimate
targets. Existing adapter tests are updated to inject a deterministic
DNS resolver so they stay hermetic instead of depending on real DNS.
Full suite: 997 passed, 0 failed, 184 skipped (self-skipping realinfra
tests with no live backend this run). typecheck and lint clean on all
touched files.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The `resolveStoreMode` kill switch (Master-DoD §3/§10) covered the newer
`src/repositories/*` stores but not two older paths, both of which are
live production execution paths today.
1. LEGACY NEXT.JS STORES. `lib/orders-store.ts` and
`lib/subscriptions-store.ts` predate `src/platform/dependency-mode.ts`
and called `getSql()` directly, silently falling back to module-global
`Map`s when no database is configured. For orders, payments and
subscriptions that is not a degraded mode, it is invisible loss of
money-bearing state: the API returns 201 and the row evaporates on the
next restart or on any other replica. The live 0pt deployment
(Coolify app 54) runs NODE_ENV=production with DATABASE_URL,
POSTGRES_URL and POSTGRES_PRISMA_URL all unset — so this is the
behavior in production right now, across 7 API routes.
These now go through `getSqlForDurableStore()`, which reuses the SAME
`resolveStoreMode` decision every `src/repositories/*` store already
uses rather than re-implementing a second production check. The reuse
is load-bearing: `isProductionRuntime` deliberately exempts
`NEXT_PHASE=phase-production-build`, because `next build` runs
page-data collection under NODE_ENV=production while serving nothing.
A naive `process.env.NODE_ENV === "production"` test here would have
failed the BUILD instead of protecting a request path.
2. LOCALBLOBSTORE. It writes document bytes to a container-local disk
while `document-capture-service` registers an artifact pointer
claiming the bytes are stored — durable object storage silently
swapped for ephemeral disk. It is the default blob store for both the
capture and extraction services. In production it now raises
DEPENDENCY_UNAVAILABLE (capability blocked: no ST0RE bytes path
exists yet) instead of fabricating success. Asserted per-operation,
not in the constructor, so merely wiring the services is not fatal —
only actually persisting or reading bytes is.
Evidence: full suite 956 passed / 184 skipped, typecheck clean,
dependency-cruiser 0 errors, lint identical to main's pre-existing
baseline (17 problems, none in changed files). Both guards asserted in
both directions and mutation-tested: bypassing resolveStoreMode -> 2
failed; dropping the write assert -> 2 failed; dropping the read assert
-> 1 failed; all restored -> 8 passed. 7 non-test call sites.
DEPLOYMENT NOTE: merging changes no runtime (no Gitea webhook exists on
any repo; deploys are manual). Deploying this build REQUIRES
provisioning DATABASE_URL for 0pt first, or the order/subscription
routes will correctly return DEPENDENCY_UNAVAILABLE instead of silently
accepting orders into memory. That provisioning is a Simon action.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Certifieringslanan (BI-P147) hittade det; jag verifierade det oberoende mot
LICENSE-filen vid respektive git-tagg 2026-09-07:
v0.208.1 Business Source License 1.1 — source-available, INTE öppen källkod
v0.209.0 Apache License 2.0
dbt Labs licensierade om MetricFlow vid 0.209.0. Den gamla pinnen låg alltså
exakt en patch under gränsen. För en självhostad plattform vars hela premiss
är att inte ha ett proprietärt runtime-beroende är det en licensdefekt, inte
en smaksak: BUSL begränsar produktionsanvändning fram till sitt change date.
Notera metodpunkten: PyPI redovisar INGET licensfält för någondera versionen,
så en metadatakontroll hade inte hittat något. Skillnaden syns bara i
LICENSE-filen vid taggen.
Ny grind: `tests/unit/metricflow-licence.test.ts` asserterar att pinnen ligger
på eller över `METRICFLOW_FIRST_OPEN_SOURCE_VERSION`, och — som negativ
kontroll — att alla tre kända BUSL-versionerna (0.207.3, 0.208.0, 0.208.1)
faktiskt förkastas av samma jämförelse. Utan den andra assertionen hade den
första kunnat passera med en jämförelse som inte diskriminerar.
Detta gör INTE MetricFlow verifierad: `providerReadiness()` returnerar
fortfarande `verified: false`, ingen supply-chain-granskning är gjord och
providern har aldrig körts (ingen Python-runtime är provisionerad). Att laga
licensen är inte samma sak som att bevisa komponenten.
typecheck rent, 2/2 tester gröna.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Same shape as the BI-P029 finding in the previous commit. `bi-slo.ts` said
"evidence/bi-p103 records the first baselines". There was no evidence/bi-p103.
The catalogue and the evaluator were both real and both well covered — against
SYNTHETIC events. A synthetic event carries whatever duration the test author
typed, so that suite proves the arithmetic and nothing about the system: it
cannot tell you whether a dashboard query takes 8 ms or 8 s. BI-P103's DoD is
one sentence, "measured baselines exist", and it was the part not done.
tests/integration/bi-p103-slo-baseline.integration.test.ts runs 460 real
queries against a live ClickHouse over a 400 000-row fact table -- 120 cold
dashboard queries (each a different bucket so the BI-P020 cache genuinely
misses), the same 120 again through the same shipped cache now hitting, and
220 exploration queries, that last number chosen so explore.p99's min_samples
of 200 is MET rather than reported low-confidence. Every duration is
wall-clock around the real call, and the telemetry goes through the shipped
producer and the shipped evaluator, so the evidence is written by the code
under test.
Measured: 7 of 11 objectives produced a number, `fully_measured` false,
measured_fraction 0.64. The four that did not each say why -- dashboard
freshness is a property of the dataset, the exploration timeout is a
configured ceiling, report completion has no producer in 0pt at all, and
report delivery is N0TIFY's receipt. A baseline table listing only the rows
that worked is how an estate comes to believe it measures more than it does.
The evidence also records what is NARROWER than production, because these
numbers will be quoted: the cached path is the in-process reference cache and
not Valkey, every latency is the engine round trip with no HTTP or
authorization layer, and there is no concurrency.
The suite is adversarial about its own inputs: a "cold" sample that came from
the cache fails, a cache "hit" that missed fails, a cached p95 that is not
faster than the cold p95 fails, and dropping the exploration sample to 60
fails on explore.p99 being low-confidence rather than quietly recording a
p99 over 60 requests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`semantic-provider.ts` cited tests/integration/semantic-parity.integration.test.ts
as the proof of the one BI-P029 DoD surface it claimed was closed, and
`providerReadiness().proven` asserted the comparison had been made against live
ClickHouse. The file did not exist. The citation was the evidence.
Writing it exposed a divergence that had been shipping:
`compileRequestToClickHouse` had no notion of a metric's OWN filters
(BI-P028 `MetricSpec.filters`). `ResolvedMetric` did not carry them and the
compiler never asked, so a metric defined as "invoiced revenue EXCLUDING
internal transfers" executed as "invoiced revenue". On the committed fixture
that is 11 249 instead of 1 250 for one country -- a plausible-looking number,
9x too large, under the correct canonical metric id. The registry, the
governance checks and the MetricFlow manifest all carried the filter; only the
thing that actually ran did not.
* `ResolvedMetric` gains `definition_filters`, already resolved onto the
physical dim_names key so the compilers stay dumb.
* The ClickHouse compiler folds them into THAT metric's conditional
aggregate, never into the shared WHERE -- two metrics in one request may
exclude different rows, and a shared predicate would apply one metric's
meaning to the other. `renderFilter`'s uniquifier becomes a string so
request filter `0` and metric filter `0_0m` cannot collide in the
parameter map.
* `compileRequestToSemantic` carries them through, and
`MetricFlowProvider.compile` emits them per metric.
The test runs both compile paths against one live ClickHouse: 0pt's `sumIf`
compiler, and a renderer driven only by the provider's structured query that
shares no SQL-generation code and emits `sum(if(...))` with inlined literals.
It asserts the two SQL strings differ, so parity is not vacuous. Backing the
fix out reproduces the pre-fix numbers, and the suite records them itself
rather than quoting a console: evidence/bi-p029/semantic-parity.json.
Three of BI-P029's four DoD surfaces stay OPEN and still say so: MetricFlow
itself is not executed (no Python runtime is provisioned here), Superset is not
implemented, and the AI Analyst path has not been compared.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
BI-P129 names four provider kinds (native, dlt, Meltano/Singer, Airbyte-if-
licensed). This commit ships the seam and the `native` provider; dlt and
Singer follow on this branch.
What is deliberately NOT here: a second ingestion stack. 0pt already has the
P061 connector contract, the P062 runner with durable checkpoints and
crash/resume tests, the schema registry, the quality gate and the idempotency
ledger. A provider's job is to hand back a P061 `Connector`; the highway is
the one that exists.
What is new is the question the P061 stack has no opinion about: an ingestion
run has to LAND somewhere, and where it lands is an analytics contract
question. Four fail-closed gates, each attacked with the guard (must refuse)
AND without it (must succeed — otherwise the guard is decoration):
1. tier — source data lands in RAW, or NORMALIZED with a declared
normalization. Never directly in CURATED/SERVING. BI-P003's "no business
dashboard reads arbitrary raw ingestion tables" is defeated in one line
by relabelling the landing table SERVING; the tier is therefore checked
at WRITE, not only at read.
2. contract coverage — every field discovery actually reported must be
declared in the BI-P004 contract. An undeclared column is an
unclassified column. The counterfactual test lands `personnummer` into a
RAW table whose contract says pii_classification "none", and the runner
reports success: that is the harm, shown rather than asserted.
3. physical home — RAW/NORMALIZED land as Parquet/Iceberg in ST0RE
(st0re#45 shapes), never straight into ClickHouse.
4. provider readiness — a provider whose own providerReadiness() says
verified:false cannot run in a production runtime without a recorded,
named exception. This is the structural form of the honesty rule 0pt#35
set with MetricFlow.
Airbyte is a declared kind with NO implementation: its server is ELv2, which
restricts offering the software as a managed service, and that is exactly
what 0pt does. Registering or planning one is refused at the registry, in one
place, so a future licence review has a single constant to change.
BI-P129 DoD — "two real source types ingest into canonical analytics tiers" —
is proven with the two real, heterogeneous P061 connectors: a JSONL file on
actual disk (cursor = line offset, real CDC deletes) and an HTTP JSON API
driven through a real fetch/Response cycle (cursor = opaque server token).
Both run through the real runner into the declared RAW tier, paged, with
9 rows landed each and distinct envelope ids.
Verification: 24 tests. Mutation-checked — neutering gates 1 and 2 fails 5 of
them, so they are load-bearing rather than incidental.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Six dataset profiles (small SaaS, medium business, VehIQ market, large event,
wide table, geospatial), one representative question each, run on ClickHouse
26.3.10.60 and DuckDB 1.5.5 with 2 warmups discarded and 9 timed repetitions.
evidence/bi-p104/benchmark.json.
The harness's job is to stop the benchmark from lying:
- overlapping ranges are INDISTINGUISHABLE whatever the medians say. Three
of the six real comparisons came out that way, one of them with a 1.6x
median gap — which is exactly the headline a noisy host manufactures.
- every sentence carries the range, the sample size and the stability, so it
cannot be trimmed to "4.4x faster" without visibly deleting the caveats.
- under 5 samples nothing is comparable at all. A three-run benchmark is an
anecdote.
- every engine BI-P104 names appears in the evidence, and the five nothing
measured carry a reason each. The running Trino belongs to another lane,
so benchmarking through it would have measured their configuration.
Measured shape, warm cache, one idle host, these query shapes only: DuckDB
faster on small_saas (1.6-8.2x) and vehiq_market (2.4-13.3x), SLOWER on
large_event (0.2-0.9x, count DISTINCT over 5M rows), indistinguishable on
medium_business, wide_table and geospatial.
One finding from the harness's own tests: an engine declared available that
produced no run came out of buildEvidence() with no reason — the dangerous
case, since it reads as "we could have and chose not to say". It now gets a
generated reason. A second from the runs: DuckDB's progress bar writes to
STDOUT and corrupted the first transcript; it is disabled explicitly and the
JSON is read from a sentinel.
10 unit tests + 4 against real engines.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
aud0#34 merged seven BI usage event types and recorded that zero producers
emit them. This makes 0pt the producer and derives the SLOs from those events
rather than opening a second, private measurement path that would disagree
with aud0 during an incident.
Three honest verdicts instead of one number:
insufficient_data an availability objective over an empty window is
SILENCE, not 100%. Every objective in an empty window
comes back insufficient_data and fully_measured is
false. A started query with no terminal event counts
as unresolved — not as a success.
not_measurable_here dashboard FRESHNESS is a property of the dataset
(BI-P026/BI-P105 watermark) and report DELIVERY is
N0TIFY's receipt. Neither is in aud0's seven types, so
both are declared unmeasured rather than proxied by
completion.
low_confidence a p99 over 12 samples is the 12th-slowest request
wearing a percentile's clothes. Reported with its
sample size and not judged.
The producer is an allowlist built field by named field, never a spread. The
test feeds it an execution record carrying SQL text, a database error message
quoting a personnummer, filter VALUES, a tenant id and a result sample, then
greps the emitted events — after first proving the grep finds those strings in
the input, so a clean result cannot be a broken detector.
14 tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A slow query is a complaint; a drifted pre-aggregate answers instantly, with
confidence, and is wrong. So the router is fail-closed on eight named reasons,
and the one people leave out — drift_never_verified — is the one that matters
on launch day: a brand-new aggregate is exactly as unproven as a drifted one.
Proved against ClickHouse 26.3.10.60, 2M rows, a real materialized view
(evidence/bi-p105/materialization-drift.json):
after backfill match, 48 buckets, 801 571 002 both sides
GDPR erasure ALTER TABLE ... DELETE on the source. An MV is an INSERT
trigger and never sees a mutation, so the aggregate kept
the deleted revenue: it would have reported 801 571 002
against a true 761 580 554 — 39 990 448 overstated,
5.251%, with NOTHING stale. Staleness at the moment of
the decision: 1 second.
guarded router base_table (state_not_serving)
counterfactual a staleness-only router SERVES it — measured, not assumed
after rebuild match, and routing returns to the materialization
Drift detection treats a MISSING bucket as drift even when every value present
is correct (late-arriving rows are the sneakiest failure), and an empty
comparison as insufficient_data rather than match — the same false green
BI-P025 found in a quality check over an empty table.
The measured speed-up is reported as a distribution, not a figure: p50 3.4x,
but the run-to-run range spans 0.6x-7.5x on one idle host, so quoting "3.4x"
as a property of the system would be misleading (BI-P104 doctrine).
19 unit tests + 4 against live ClickHouse.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
DRAFT -> PUBLISHED -> SUPERSEDED as a ledger (sql/010) rather than a column on
the frozen version: sql/008 already makes DashboardVersion immutable at the
database, and a lifecycle column there would have meant weakening that.
Rollback mints a NEW version carrying the old document and publishes it, so
"we went back to v3" is a timestamped fact (v5 = rollback_of_version 3) rather
than a hole where v4 used to be. It is verified by BI-P060 canonical hash, and
documentOf() refuses to serve a stored document whose hash no longer matches
the release — a dashboard that renders an unapproved definition is worse than
one that will not open.
The claim worth attacking is "only one release can be live". The application
check is read-then-write, so it is NOT the guarantee, and the evidence says so
in both directions against Postgres 16.15:
guard_absent partial unique index dropped -> BOTH concurrent publishes
commit, dashboard ends with TWO live versions
guard_present index in place -> loser fails 23505 on
opt_analytics_dashboard_release_one_published, one live
evidence/bi-p061/dashboard-release-race.json, reproducible with
scripts/bi-p061-release-race.mjs.
Two findings while writing the tests, both fixed here rather than papered over:
the one-draft-per-dashboard index makes the two-draft race impossible to even
construct (so the race is run as two create-and-publish transactions, which is
what a "save and publish" button does), and an UPDATE matching zero rows
raises nothing — which reads as "refused" to a careless test, so the
immutability test asserts the row exists first.
23 tests: 8 unit + 15 against live Postgres.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
BI-P002 named DashboardDefinition/DashboardVersion and sql/008 stored them,
but `layout_json` was a string, and a string holds anything — including a
Superset export only Superset can read back. This gives the document a body
and makes the DoD checkable rather than asserted.
planDashboard() is the DoD: a document in, runnable AnalyticsQueryRequest
objects (BI-P016) plus visualization specs out. The test greps the whole
serialized plan for vendor traces after first proving the detector CAN find
one in the source document — a clean grep over a broken detector is the usual
way this test passes for the wrong reason.
The counterfactual is a test too: a widget whose only content is a Superset
slice id is REFUSED, because that is exactly the document that cannot be
reconstructed. vendor_ref survives for round-tripping but is excluded from
the canonical hash, so re-exporting to a new Superset instance does not
fabricate a new dashboard version.
No SQL surface: query widgets are parsed by parseAnalyticsQueryRequest, so
`sql`, `where` and `tenant_id` are refused structurally rather than by
review. 18 tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The live suites regenerate their evidence on every run, so the BI-P021
overshoot figures moved when they were last re-run — and the doc and module
comment still quoted the earlier ones (130,883,409 rows / 130x / 92ms) while
the committed file said 153,318,696 / 153x / 55ms. Prose that disagrees with
the evidence it cites is worse than prose with no number in it.
Both now match the committed file, and both say what the earlier version
implied but did not state: the multiple is MACHINE-DEPENDENT and moves between
runs on the same box. That is the actual finding. The bound is "roughly one
poll interval of engine throughput", not a number, and quoting any specific
multiple as if it were a property of the system is the mistake this correction
exists to prevent.
The unwatched-baseline figure is corrected the same way: 4,227,318,261 rows
read before cancellation in the current BI-P019 record, not the 6.5 billion of
the earlier run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
admin
siax-bot