52 Commits

Include renames

Author	SHA1	Message	Date
admin	70fd81dc4d	feat: share durable dataset services across route modules	2026-09-07 18:06:21 +02:00
admin	3c5850fbe5	refactor: persist dataset versions and make lineage retry-stable	2026-09-07 18:03:21 +02:00
admin	3ccdcf0cf6	fix: use postgres arrays and normalized idempotency comparison	2026-09-07 18:02:38 +02:00
admin	906fe9ac37	feat: add durable dataset version repository	2026-09-07 18:01:56 +02:00
admin	a3bad9aa12	refactor: extract canonical dataset version types	2026-09-07 18:01:24 +02:00
admin	ea02f4e9f7	refactor: persist DatasetService through repository boundary	2026-09-07 18:00:47 +02:00
admin	5dbd53045c	feat: add durable dataset registry repository	2026-09-07 18:00:29 +02:00
admin	f0958cf708	fix: centralize lineage durability in AUD0 client boundary	2026-09-07 17:54:55 +02:00
admin	020a4be7e8	feat(evidence): make standard AUD0 client durable by default	2026-09-07 17:23:20 +02:00
admin	9f46e12447	feat(lineage): route production evidence through durable AUD0 outbox	2026-09-07 17:14:54 +02:00
admin	79942a2431	feat(evidence): implement durable AUD0 outbox and dispatcher	2026-09-07 17:13:02 +02:00
siax-agentandClaude Opus 5	0e0e1fec56	fix(BI-P147): flytta MetricFlow-pinnen ovanför licensgränsen (0.208.1 var BUSL) ... Certifieringslanan (BI-P147) hittade det; jag verifierade det oberoende mot LICENSE-filen vid respektive git-tagg 2026-09-07: v0.208.1 Business Source License 1.1 — source-available, INTE öppen källkod v0.209.0 Apache License 2.0 dbt Labs licensierade om MetricFlow vid 0.209.0. Den gamla pinnen låg alltså exakt en patch under gränsen. För en självhostad plattform vars hela premiss är att inte ha ett proprietärt runtime-beroende är det en licensdefekt, inte en smaksak: BUSL begränsar produktionsanvändning fram till sitt change date. Notera metodpunkten: PyPI redovisar INGET licensfält för någondera versionen, så en metadatakontroll hade inte hittat något. Skillnaden syns bara i LICENSE-filen vid taggen. Ny grind: `tests/unit/metricflow-licence.test.ts` asserterar att pinnen ligger på eller över `METRICFLOW_FIRST_OPEN_SOURCE_VERSION`, och — som negativ kontroll — att alla tre kända BUSL-versionerna (0.207.3, 0.208.0, 0.208.1) faktiskt förkastas av samma jämförelse. Utan den andra assertionen hade den första kunnat passera med en jämförelse som inte diskriminerar. Detta gör INTE MetricFlow verifierad: `providerReadiness()` returnerar fortfarande `verified: false`, ingen supply-chain-granskning är gjord och providern har aldrig körts (ingen Python-runtime är provisionerad). Att laga licensen är inte samma sak som att bevisa komponenten. typecheck rent, 2/2 tester gröna. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 09:04:30 +02:00
Claude Opus 5	7441de5429	BI-P103: the cited evidence/bi-p103 did not exist — measure it ... Same shape as the BI-P029 finding in the previous commit. `bi-slo.ts` said "evidence/bi-p103 records the first baselines". There was no evidence/bi-p103. The catalogue and the evaluator were both real and both well covered — against SYNTHETIC events. A synthetic event carries whatever duration the test author typed, so that suite proves the arithmetic and nothing about the system: it cannot tell you whether a dashboard query takes 8 ms or 8 s. BI-P103's DoD is one sentence, "measured baselines exist", and it was the part not done. tests/integration/bi-p103-slo-baseline.integration.test.ts runs 460 real queries against a live ClickHouse over a 400 000-row fact table -- 120 cold dashboard queries (each a different bucket so the BI-P020 cache genuinely misses), the same 120 again through the same shipped cache now hitting, and 220 exploration queries, that last number chosen so explore.p99's min_samples of 200 is MET rather than reported low-confidence. Every duration is wall-clock around the real call, and the telemetry goes through the shipped producer and the shipped evaluator, so the evidence is written by the code under test. Measured: 7 of 11 objectives produced a number, `fully_measured` false, measured_fraction 0.64. The four that did not each say why -- dashboard freshness is a property of the dataset, the exploration timeout is a configured ceiling, report completion has no producer in 0pt at all, and report delivery is N0TIFY's receipt. A baseline table listing only the rows that worked is how an estate comes to believe it measures more than it does. The evidence also records what is NARROWER than production, because these numbers will be quoted: the cached path is the in-process reference cache and not Valkey, every latency is the engine round trip with no HTTP or authorization layer, and there is no concurrency. The suite is adversarial about its own inputs: a "cold" sample that came from the cache fails, a cache "hit" that missed fails, a cached p95 that is not faster than the cold p95 fails, and dropping the exploration sample to 60 fails on explore.p99 being low-confidence rather than quietly recording a p99 over 60 requests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 08:27:37 +02:00
Claude Opus 5	468a87c00c	BI-P029: the cited parity test did not exist, and it caught a real bug ... `semantic-provider.ts` cited tests/integration/semantic-parity.integration.test.ts as the proof of the one BI-P029 DoD surface it claimed was closed, and `providerReadiness().proven` asserted the comparison had been made against live ClickHouse. The file did not exist. The citation was the evidence. Writing it exposed a divergence that had been shipping: `compileRequestToClickHouse` had no notion of a metric's OWN filters (BI-P028 `MetricSpec.filters`). `ResolvedMetric` did not carry them and the compiler never asked, so a metric defined as "invoiced revenue EXCLUDING internal transfers" executed as "invoiced revenue". On the committed fixture that is 11 249 instead of 1 250 for one country -- a plausible-looking number, 9x too large, under the correct canonical metric id. The registry, the governance checks and the MetricFlow manifest all carried the filter; only the thing that actually ran did not. * `ResolvedMetric` gains `definition_filters`, already resolved onto the physical dim_names key so the compilers stay dumb. * The ClickHouse compiler folds them into THAT metric's conditional aggregate, never into the shared WHERE -- two metrics in one request may exclude different rows, and a shared predicate would apply one metric's meaning to the other. `renderFilter`'s uniquifier becomes a string so request filter `0` and metric filter `0_0m` cannot collide in the parameter map. * `compileRequestToSemantic` carries them through, and `MetricFlowProvider.compile` emits them per metric. The test runs both compile paths against one live ClickHouse: 0pt's `sumIf` compiler, and a renderer driven only by the provider's structured query that shares no SQL-generation code and emits `sum(if(...))` with inlined literals. It asserts the two SQL strings differ, so parity is not vacuous. Backing the fix out reproduces the pre-fix numbers, and the suite records them itself rather than quoting a console: evidence/bi-p029/semantic-parity.json. Three of BI-P029's four DoD surfaces stay OPEN and still say so: MetricFlow itself is not executed (no Python runtime is provisioned here), Superset is not implemented, and the AI Analyst path has not been compared. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 08:23:43 +02:00
Claude Opus 5	b5ab885a67	feat(analytics): BI-P129 — ConnectorProvider, and four gates on where data lands ... BI-P129 names four provider kinds (native, dlt, Meltano/Singer, Airbyte-if- licensed). This commit ships the seam and the `native` provider; dlt and Singer follow on this branch. What is deliberately NOT here: a second ingestion stack. 0pt already has the P061 connector contract, the P062 runner with durable checkpoints and crash/resume tests, the schema registry, the quality gate and the idempotency ledger. A provider's job is to hand back a P061 `Connector`; the highway is the one that exists. What is new is the question the P061 stack has no opinion about: an ingestion run has to LAND somewhere, and where it lands is an analytics contract question. Four fail-closed gates, each attacked with the guard (must refuse) AND without it (must succeed — otherwise the guard is decoration): 1. tier — source data lands in RAW, or NORMALIZED with a declared normalization. Never directly in CURATED/SERVING. BI-P003's "no business dashboard reads arbitrary raw ingestion tables" is defeated in one line by relabelling the landing table SERVING; the tier is therefore checked at WRITE, not only at read. 2. contract coverage — every field discovery actually reported must be declared in the BI-P004 contract. An undeclared column is an unclassified column. The counterfactual test lands `personnummer` into a RAW table whose contract says pii_classification "none", and the runner reports success: that is the harm, shown rather than asserted. 3. physical home — RAW/NORMALIZED land as Parquet/Iceberg in ST0RE (st0re#45 shapes), never straight into ClickHouse. 4. provider readiness — a provider whose own providerReadiness() says verified:false cannot run in a production runtime without a recorded, named exception. This is the structural form of the honesty rule 0pt#35 set with MetricFlow. Airbyte is a declared kind with NO implementation: its server is ELv2, which restricts offering the software as a managed service, and that is exactly what 0pt does. Registering or planning one is refused at the registry, in one place, so a future licence review has a single constant to change. BI-P129 DoD — "two real source types ingest into canonical analytics tiers" — is proven with the two real, heterogeneous P061 connectors: a JSONL file on actual disk (cursor = line offset, real CDC deletes) and an HTTP JSON API driven through a real fetch/Response cycle (cursor = opaque server token). Both run through the real runner into the declared RAW tier, paged, with 9 rows landed each and distinct envelope ids. Verification: 24 tests. Mutation-checked — neutering gates 1 and 2 fails 5 of them, so they are load-bearing rather than incidental. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 06:22:07 +02:00
Claude Opus 5	f33cc8d32f	feat(analytics): BI-P104 — a benchmark harness that refuses to quote one number ... Six dataset profiles (small SaaS, medium business, VehIQ market, large event, wide table, geospatial), one representative question each, run on ClickHouse 26.3.10.60 and DuckDB 1.5.5 with 2 warmups discarded and 9 timed repetitions. evidence/bi-p104/benchmark.json. The harness's job is to stop the benchmark from lying: - overlapping ranges are INDISTINGUISHABLE whatever the medians say. Three of the six real comparisons came out that way, one of them with a 1.6x median gap — which is exactly the headline a noisy host manufactures. - every sentence carries the range, the sample size and the stability, so it cannot be trimmed to "4.4x faster" without visibly deleting the caveats. - under 5 samples nothing is comparable at all. A three-run benchmark is an anecdote. - every engine BI-P104 names appears in the evidence, and the five nothing measured carry a reason each. The running Trino belongs to another lane, so benchmarking through it would have measured their configuration. Measured shape, warm cache, one idle host, these query shapes only: DuckDB faster on small_saas (1.6-8.2x) and vehiq_market (2.4-13.3x), SLOWER on large_event (0.2-0.9x, count DISTINCT over 5M rows), indistinguishable on medium_business, wide_table and geospatial. One finding from the harness's own tests: an engine declared available that produced no run came out of buildEvidence() with no reason — the dangerous case, since it reads as "we could have and chose not to say". It now gets a generated reason. A second from the runs: DuckDB's progress bar writes to STDOUT and corrupted the first transcript; it is disabled explicitly and the JSON is read from a sentinel. 10 unit tests + 4 against real engines. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 04:06:43 +02:00
Claude Opus 5	ed6e62809d	feat(analytics): BI-P103 — BI SLOs measured from AUD0's usage events, or declared unmeasured ... aud0#34 merged seven BI usage event types and recorded that zero producers emit them. This makes 0pt the producer and derives the SLOs from those events rather than opening a second, private measurement path that would disagree with aud0 during an incident. Three honest verdicts instead of one number: insufficient_data an availability objective over an empty window is SILENCE, not 100%. Every objective in an empty window comes back insufficient_data and fully_measured is false. A started query with no terminal event counts as unresolved — not as a success. not_measurable_here dashboard FRESHNESS is a property of the dataset (BI-P026/BI-P105 watermark) and report DELIVERY is N0TIFY's receipt. Neither is in aud0's seven types, so both are declared unmeasured rather than proxied by completion. low_confidence a p99 over 12 samples is the 12th-slowest request wearing a percentile's clothes. Reported with its sample size and not judged. The producer is an allowlist built field by named field, never a spread. The test feeds it an execution record carrying SQL text, a database error message quoting a personnummer, filter VALUES, a tenant id and a result sample, then greps the emitted events — after first proving the grep finds those strings in the input, so a clean result cannot be a broken detector. 14 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 04:01:16 +02:00
Claude Opus 5	96ff601248	feat(analytics): BI-P105 — a pre-aggregate serves only while it is provably undrifted ... A slow query is a complaint; a drifted pre-aggregate answers instantly, with confidence, and is wrong. So the router is fail-closed on eight named reasons, and the one people leave out — drift_never_verified — is the one that matters on launch day: a brand-new aggregate is exactly as unproven as a drifted one. Proved against ClickHouse 26.3.10.60, 2M rows, a real materialized view (evidence/bi-p105/materialization-drift.json): after backfill match, 48 buckets, 801 571 002 both sides GDPR erasure ALTER TABLE ... DELETE on the source. An MV is an INSERT trigger and never sees a mutation, so the aggregate kept the deleted revenue: it would have reported 801 571 002 against a true 761 580 554 — 39 990 448 overstated, 5.251%, with NOTHING stale. Staleness at the moment of the decision: 1 second. guarded router base_table (state_not_serving) counterfactual a staleness-only router SERVES it — measured, not assumed after rebuild match, and routing returns to the materialization Drift detection treats a MISSING bucket as drift even when every value present is correct (late-arriving rows are the sneakiest failure), and an empty comparison as insufficient_data rather than match — the same false green BI-P025 found in a quality check over an empty table. The measured speed-up is reported as a distribution, not a figure: p50 3.4x, but the run-to-run range spans 0.6x-7.5x on one idle host, so quoting "3.4x" as a property of the system would be misleading (BI-P104 doctrine). 19 unit tests + 4 against live ClickHouse. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 03:56:32 +02:00
Claude Opus 5	1a12e8a882	feat(analytics): BI-P061 — dashboard releases, with the publish race measured both ways ... DRAFT -> PUBLISHED -> SUPERSEDED as a ledger (sql/010) rather than a column on the frozen version: sql/008 already makes DashboardVersion immutable at the database, and a lifecycle column there would have meant weakening that. Rollback mints a NEW version carrying the old document and publishes it, so "we went back to v3" is a timestamped fact (v5 = rollback_of_version 3) rather than a hole where v4 used to be. It is verified by BI-P060 canonical hash, and documentOf() refuses to serve a stored document whose hash no longer matches the release — a dashboard that renders an unapproved definition is worse than one that will not open. The claim worth attacking is "only one release can be live". The application check is read-then-write, so it is NOT the guarantee, and the evidence says so in both directions against Postgres 16.15: guard_absent partial unique index dropped -> BOTH concurrent publishes commit, dashboard ends with TWO live versions guard_present index in place -> loser fails 23505 on opt_analytics_dashboard_release_one_published, one live evidence/bi-p061/dashboard-release-race.json, reproducible with scripts/bi-p061-release-race.mjs. Two findings while writing the tests, both fixed here rather than papered over: the one-draft-per-dashboard index makes the two-draft race impossible to even construct (so the race is run as two create-and-publish transactions, which is what a "save and publish" button does), and an UPDATE matching zero rows raises nothing — which reads as "refused" to a careless test, so the immutability test asserts the row exists first. 23 tests: 8 unit + 15 against live Postgres. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 03:51:04 +02:00
Claude Opus 5	2af126dfc4	feat(analytics): BI-P060 — engine-neutral dashboard model that plans without Superset ... BI-P002 named DashboardDefinition/DashboardVersion and sql/008 stored them, but `layout_json` was a string, and a string holds anything — including a Superset export only Superset can read back. This gives the document a body and makes the DoD checkable rather than asserted. planDashboard() is the DoD: a document in, runnable AnalyticsQueryRequest objects (BI-P016) plus visualization specs out. The test greps the whole serialized plan for vendor traces after first proving the detector CAN find one in the source document — a clean grep over a broken detector is the usual way this test passes for the wrong reason. The counterfactual is a test too: a widget whose only content is a Superset slice id is REFUSED, because that is exactly the document that cannot be reconstructed. vendor_ref survives for round-tripping but is excluded from the canonical hash, so re-exporting to a new Superset instance does not fabricate a new dashboard version. No SQL surface: query widgets are parsed by parseAnalyticsQueryRequest, so `sql`, `where` and `tenant_id` are refused structurally rather than by review. 18 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 03:43:06 +02:00
Claude Opus 5	c4862fb7a7	docs(analytics): quote the numbers the COMMITTED evidence actually contains ... The live suites regenerate their evidence on every run, so the BI-P021 overshoot figures moved when they were last re-run — and the doc and module comment still quoted the earlier ones (130,883,409 rows / 130x / 92ms) while the committed file said 153,318,696 / 153x / 55ms. Prose that disagrees with the evidence it cites is worse than prose with no number in it. Both now match the committed file, and both say what the earlier version implied but did not state: the multiple is MACHINE-DEPENDENT and moves between runs on the same box. That is the actual finding. The bound is "roughly one poll interval of engine throughput", not a number, and quoting any specific multiple as if it were a property of the system is the mistake this correction exists to prevent. The unwatched-baseline figure is corrected the same way: 4,227,318,261 rows read before cancellation in the current BI-P019 record, not the 6.5 billion of the earlier run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 03:18:12 +02:00
Claude Opus 5	c864c8c417	fix(analytics): revert 18 files reformatted by mistake, and actually apply the Iceberg fix a previous message claimed ... Two corrections, both of my own errors. 1. SCOPE CREEP, NOW REVERTED. Running `prettier --write` on whole directories reformatted 18 files this branch had no business touching — the entire merged BI-P002..P012 core (arrow/, domain/, engine/, repositories/, five analytics integration suites) and, absurdly, search-opensearch.integration.test.ts, which has nothing to do with this work. The changes were purely stylistic quote churn, but they added ~1,400 lines of noise to the diff, made the PR harder to review honestly, and invited conflicts with the four other lanes working in this estate. All 18 are now byte-identical to origin/main again (verified with `git diff origin/main --name-only`, which reports nothing for them). Formatting is now applied only to the files this branch actually owns. 2. A COMMIT MESSAGE THAT DESCRIBED A FIX THAT WAS NOT IN THE CODE. Commit 3ee0230 said the "does NOT ship the Iceberg table" assertion had been moved from a filesystem check to `git ls-files`. It had not. The patch anchor used double quotes, prettier had already converted that file to single quotes, so the replacement silently matched nothing and the script printed its success line anyway. The suite then passed — but only because I had just deleted the fixture directory, so the OLD filesystem assertion happened to be true. That is precisely the failure mode this branch keeps writing tests against: a green run that means something different from what it appears to mean. The fix is now genuinely applied and verified the way it should have been the first time — by running the suite WITH the Iceberg fixture present on disk, which is the case that previously failed. It passes, and it now also asserts that the parquet/csv fixtures ARE tracked, so a version shipping no fixture at all would fail rather than pass vacuously. 957 tests pass with ClickHouse, Postgres and DuckDB live. typecheck, eslint and prettier clean on the files this branch owns. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 03:16:30 +02:00
Claude Opus 5	ba5c796b36	docs(analytics): anchor two security claims to the measurements behind them ... Neither changes behaviour; both replace an assertion with a measurement. BI-P014's escaping strategy rests on DuckDB treating a backslash as a literal character, so that doubling `'` is sufficient. Verified on 1.5.5 rather than assumed: `SELECT length('a\\b')` returns 4, and `SELECT 'a\''` is a parser error ("unterminated quoted string") — which is exactly what a NON-escaping backslash produces. In a dialect where `\'` closed the literal, doubling would not be enough, so the claim is worth pinning to a run. BI-P017's FROM/JOIN scan is conservative where it is unsure, and that has a cost worth naming rather than discovering: ClickHouse's `ARRAY JOIN arr` takes a COLUMN, not a table, and this guard refuses it. A false refusal an analyst can see and report is the right side of that trade; a false ADMISSION is not. Supporting it means teaching the scan which JOIN flavours take a non-table target, which is the direction that grows holes — so it stays refused until someone needs it and can add it WITH a corpus entry proving the narrowing did not open anything else. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 03:11:19 +02:00
Claude Opus 5	3ee023017b	fix(analytics): move the DuckDB live suite out of tests/integration, and add the BI docs ... CI's integration job failed, and it failed CORRECTLY. That job provides a live OpenSearch and a live ClickHouse and then asserts that nothing in tests/integration/ skipped, on the stated principle that "a skipped suite must never masquerade as a green integration gate". The DuckDB suite needs a Python interpreter that job does not provide, so it skipped, and the guard fired — the first suite in that directory ever to skip there. Moving the file to tests/duckdb/ respects the guard rather than weakening it. Loosening it to tolerate skips would have traded a real gate for one green job, which is the trade this estate keeps regretting. The suite still runs, still fails rather than skipping when DUCKDB_PYTHON is set and DuckDB is broken, and is still pinned by the committed evidence. Also fixed: "does NOT ship the Iceberg table" asserted against the FILESYSTEM when the claim is about what is COMMITTED. The live suite rebuilds that table locally, so the assertion failed whenever both suites ran in one pass — it was testing the wrong thing rather than catching a real problem. It now asserts against `git ls-files`, and also asserts the parquet/csv fixtures ARE tracked, so a version that stopped shipping any fixture would fail too. docs/analytics/BI_QUERY_METRIC_QUALITY.md is organised around what is PROVEN and what is NOT, because the second list is the one that gets dropped. It records the five findings and the six open items, including that BI-P029's MetricFlow was never executed and that BI-P021's budget is a bound rather than a hard cap (measured 130x overshoot at a 100ms poll). It also reports, without changing it, a latent flaw in .gitea/workflows/ci.yml (a protected file this branch does not touch): the integration guard's second pattern `Tests [0-9]+ passed` never matches, because vitest writes that summary with ANSI colour codes between the words. The guard is therefore STRICTER than it reads — "any skip fails" rather than "skipped and nothing passed" — so nothing is currently unguarded, but the escape hatch does not work and someone will eventually be puzzled by it. 957 tests pass with ClickHouse, Postgres and DuckDB all live. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 03:08:11 +02:00
Claude Opus 5	a9ae7f25a4	feat(analytics): BI-P014 — DuckDB provider, and the measured finding that its sandbox does not work the way it looks ... BI-P014's security requirement is the whole prompt: DuckDB is a local engine with `read_csv('/etc/passwd')` built in. Two layers. LAYER 1, the mount registry: a caller names ARTIFACT IDS, never paths. The registry resolves an id, checks the artifact belongs to the caller's tenant, and realpath()s the result to confirm it is still inside the mount root once symlinks are followed. A symlink to /etc/passwd, a traversing relative path, an absolute path, and a sibling directory sharing the root's prefix (/mnt/artifacts-evil for root /mnt/artifacts) are each refused. A cross-tenant read is AUTHORIZATION_DENIED, deliberately not NOT_FOUND — an operator reading the audit trail needs to see an attempted cross-tenant read as one. LAYER 2, THE FINDING: `SET allowed_directories=[...]` ALONE IS A NO-OP. Measured on DuckDB 1.5.5. With allowed_directories set to a single mount directory and nothing else, `read_csv('/etc/passwd')` RETURNED THE FILE and an https:// URL was fetched over the network. Only when `enable_external_access = false` is ALSO set does the allow-list bind; the same reads then fail with "Permission Error: Cannot access file". `lock_configuration=true` afterwards stops a query re-opening either. So the obvious spelling of this control — set the allow-list and move on — yields a setting that appears in duckdb_settings(), looks like a sandbox, and permits everything. That is `trivy --exit-code 0` in a different costume. The bootstrap is therefore emitted as ONE unit, `assertSessionSandboxed` refuses a session missing any part of it, and the measurement itself is PINNED by a test that never skips: if someone simplifies the bootstrap, the pinned evidence (`read_etc_passwd_succeeded: true` for the naive spelling) keeps the finding attached to the build rather than to a comment that would rot. Twelve bypass attempts run against a real DuckDB with LAYER 1 REMOVED — sent straight to the engine, registry not in the picture. Zero succeeded. The counterfactual runs in the same session and is asserted: the mounted file IS readable, so "nothing succeeded" is not a sandbox that blocks everything, which would be an outage rather than a control. A locked session is then shown unable to re-open its own sandbox, all three ways. DoD, against real fixtures: Parquet aggregates correctly (SE 394.5, NO 105.5), CSV round-trips a value containing a quote, and an ICEBERG SNAPSHOT reads via iceberg_scan on an exact metadata JSON — which is not a workaround: DuckDB refuses to glob for the latest version because globbing can read uncommitted data, and an exact metadata file is what "snapshot" means. Arrow is reported UNPROVEN by `readiness()` with the reason: DuckDB's arrow extension is a network install the offline environment could not download. THREE DEFECTS IN MY OWN TESTS, found and fixed rather than tidied away: * a bypass-corpus entry was `SELECT 1 FROM (SELECT 1) t` — not an attack at all. It succeeded and was reported as a leak. A corpus entry that cannot fail is as useless as a check that cannot. Replaced with real ATTACH and COPY attempts. * the harness ran the session bootstrap outside its try block, so a statement that correctly REFUSED to re-open a locked sandbox killed the subprocess and surfaced as an unreadable "Command failed" instead of as the meaningful refusal it was. * the Iceberg fallback was `expect(evidence.iceberg ?? "unbuilt").toBeDefined()` — a check that can never fail. It let a run without the fixture ERASE the recorded Iceberg result from the evidence file while both suites stayed green. It now rebuilds the fixture when pyiceberg is available and, when it is not, asserts against the previously recorded result and marks it as carried forward. The Iceberg fixture is deliberately NOT committed: pyiceberg writes absolute paths into its metadata and Avro manifests, so a committed table resolves only on the machine that built it, and a fixture that silently fails everywhere else is worse than none. tests/fixtures/duckdb/README.md says so and gives the rebuild command; the recorded result is the artefact instead. Heavy artifacts are refused with CAPABILITY_UNAVAILABLE and pointed at N0D, per BI-P014 — silently running a 40 GB scan in the API process is how a preview engine takes down a control plane. 35 unit tests, 12 live tests. Evidence: evidence/bi-p014/duckdb-sandbox.json. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 02:54:53 +02:00
Claude Opus 5	ec26ddc296	feat(analytics): BI-P022 — dbt canonical transformation CONTRACT and reference project ... Scope stated in the module's first paragraph: this defines the contract, it does not run dbt, and nothing in this repo does. Execution is N0D's (n0d#38), which consumes this from the other side. There is deliberately no DbtExecutor implementation here — only the interface — so "0pt can run dbt" cannot become true by accident and the two repos share one versioned shape. `analytics/dbt/` is a real reference project covering raw -> staging -> curated -> serving, with the four layers mapped onto BI-P003's existing tiers rather than inventing a fifth vocabulary. The tenant predicate is applied at the FIRST modelled layer so every downstream model inherits it and no later model has to remember — the same forced-predicate rule as the query compiler's, one layer up. Two singular tests ship with it, both aimed at false greens: * `assert_mart_is_not_empty` — without it every column test passes trivially over zero rows and the whole suite reports green while the pipeline is dead; * `assert_no_cross_tenant_rows` — which should be impossible to fail, and exists precisely for that reason. An isolation guarantee nobody tests is the one that turns out to have four holes rather than the one reported. BI-P022's version rule is a GATE. `DBT_RUNTIME_PIN` records dbt Core 1.10.13 (stable) with dbt-clickhouse 1.9.2 and BOTH review flags FALSE — no licence or supply-chain review has been performed in this repo and none is claimed. `assertDbtRuntimeCertified` refuses a production run until they are true, and refuses an alpha line in production without a separate certification reference. A version constant on its own would be the documented-but-unenforced pattern; a test pins the flags at false so flipping them is a visible deliberate act. The result is treated as UNTRUSTED INPUT, because it arrives from another repo's executor and this estate has been burned by green statuses that meant nothing. Refused: `status: success` with ZERO nodes executed (a dbt selector matching nothing exits 0 — the dbt-shaped false green); an observed runtime different from the pin (the models mean something else and nothing says so); a revision that is not an exact 40-char SHA (a branch name is not reproducible, and reproducibility is the DoD); a missing manifest.json or run_results.json, without which the run cannot be audited or its tests ingested by BI-P025. Layer dependencies are checked against the manifest GRAPH, not asserted as a convention: a mart reading a source directly, or skipping a layer, or a staging model reading a mart, are each refused. Selectors are validated rather than passed through — a free-form argument list is a shell surface, and a shell surface on a job holding warehouse credentials is the whole problem. The job also cannot name an arbitrary target schema: it must be scoped to the job's tenant. 39 tests. typecheck + eslint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 02:45:41 +02:00
Claude Opus 5	6cb00add57	feat(analytics): BI-P025/BI-P026 — quality providers that fail on empty, freshness that refuses to be silent ... Both prompts are about a FALSE GREEN, so both are tested by constructing the false green and requiring a failure. BI-P025. Three provider kinds, and the interface makes the difference visible instead of implying parity: `providerKind` is "executing" or "ingesting". The native provider runs checks; the dbt and Great Expectations adapters INGEST results produced elsewhere (n0d#38 owns dbt execution against BI-P022, and GE is Python — nothing here runs it). Calling all three "providers" while only one executes is how "we have Great Expectations coverage" ends up in a status report. THE FALSE GREEN, proven against a live ClickHouse 24.8.14.39: a `not_null` check over an EMPTY table returns violations=0, evaluated=0. The raw counts are asserted first — the engine really does say zero-over-zero — and then the provider is required to call it a FAILURE, not a pass. Every rule compiles to a VIOLATION expression rather than a "rows that are fine" count, so the distinction cannot be lost at the runner layer. Same treatment for the ingesting providers: a MISSING dbt result is a failure, and dbt's `error`, `warn` and `skipped` are each asserted not to be passes. `skipped` in particular is the exact shape of l0re's 28 silently-skipped Postgres tests. The gate: zero verdicts QUARANTINES. "No rules ran" and "everything passed" produce the same green in most implementations, which is how a quality gate stops gating without anyone noticing — api0's `trivy --exit-code 0` in another costume. A later passing run does not auto-release either; release takes a principal and a reason, or a flapping rule would quarantine and un-quarantine a dataset between two reads of the same dashboard. BI-P025's DoD is the READ side: a quarantined projection is REFUSED, not annotated. An annotation on a dashboard nobody reads is not a quarantine. Live counts, all exact, from evidence/bi-p025/native-quality-live.json: not_null 5/118, range 3, distribution 4, unique 11. The unique figure is 11 rather than the 6 the first version of the test asserted — the five empty-string customers are duplicates of each other, so not_null and unique overlap on the same defect. The engine was right and the fixture arithmetic was wrong; the breakdown is written into the test rather than the number quietly adjusted. Tenant isolation re-proven on this path specifically, because a quality run uses an ADMINISTRATIVE credential: with 618 rows in the base table, t_alpha evaluates 118 and t_beta 500, and t_beta's blocking failure quarantines t_beta alone. Without the forced predicate one tenant's dirty rows would quarantine another tenant's projection. BI-P026. Five states, and the DoD's operative word is "silently". So the gate sits at the boundary every result crosses: a result with NO freshness envelope is unreleasable — not defaulted to FRESH, not defaulted to UNKNOWN — because "we forgot to attach it" and "the data is current" are indistinguishable to a reader, and any default makes forgetting the silent path. A non-current state without an explicit acknowledgement is refused, so a consumer that ignores the envelope cannot render stale numbers as current. FAILED outranks age: a pipeline that just failed may still hold recently-timestamped data from the previous run, and reporting that as FRESH is how a broken pipeline stays invisible for a day. UNKNOWN is never collapsed into FRESH. 54 unit tests, 11 live tests. typecheck + eslint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 02:40:35 +02:00
Claude Opus 5	65cce86144	feat(analytics): BI-P028/BI-P029/BI-P030/BI-P031 — metric model, semantic registry, governance, semantic provider ... BI-P028 extends the BI-P002 MetricDefinition rather than defining a second, parallel metric — a second metric type would be the exact failure the DoD is about. `MetricSpec` is the fifteen-field shape the prompt names; explicit bridge functions map it onto the persisted row plus the six columns it adds. The DoD ("no production metric exists only inside a dashboard tool") is enforced by something that refuses: a dashboard panel carrying an inline expression, inline aggregation or inline SQL is rejected, as is one naming an unregistered or RETIRED metric. So a metric cannot come into existence inside Superset and quietly become production. Validation refuses the metrics that render as a plausible number and mean nothing: no description, no owner, no time dimension, a ratio without a denominator, a currency format with no currency (which is how SEK gets rendered as USD), a filter on a dimension the metric does not declare. BI-P031's DoD is checked on a MEANING HASH rather than a field diff, so a field this code has never heard of still counts as changing the meaning — fail-closed rather than fail-open on future additions. Eight semantic fields are each mutated and each refused; three editorial fields are each mutated and each allowed, because forcing a version bump to fix a typo is how teams learn to route around governance. A new version starts DRAFT and EXPERIMENTAL: it never inherits the certification, or a certified metric's meaning could change while still wearing the badge. BI-P030 stores the semantic model separately from any engine, and that negative is enforced two ways after the first way turned out to be insufficient: * a VOCABULARY scan rejects a model mentioning metricflow / clickhouse / trino / cube / superset anywhere in its serialized form; * a STRUCTURAL check rejects any top-level key outside BI-P030's field list. The second exists because a test expected the vocabulary scan to catch a MetricFlow-shaped manifest and it did not — the manifest never says "metricflow". The structural check holds regardless of what a document calls itself. The manifest is now also stamped with its generator so a leaked copy is detectable. The replaceability DoD is tested with TWO providers whose naming rules differ (MetricFlow-shaped and Cube-shaped, `ds_sales.amount_sek`): the canonical id set must be identical across both AND the provider bindings must differ — otherwise the equality would hold trivially for a registry that only ever had one provider. BI-P029 — WHAT IS NOT PROVEN IS STATED IN THE MODULE AND RETURNED BY THE CODE. MetricFlow is a Python package and was NOT executed: no Python runtime is provisioned here, and BI-P029's "pin a currently verified version" requires a licence/supply-chain review that has not happened. `providerReadiness()` returns `verified: false` and names all four open items, including that the Superset path is unimplemented and the AI Analyst path uncompared. The constant `METRICFLOW_PINNED_VERSION` is labelled a declared intent, not a certification. A test asserts the readiness report says so, so the honesty cannot rot silently. 353 tests. typecheck + eslint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 02:34:52 +02:00
Claude Opus 5	ca0a3292df	feat(analytics): BI-P020 — provider-neutral query cache that cannot serve a stale or foreign result ... All six key components BI-P020 lists are REQUIRED, and the builder refuses to produce a key when one is missing rather than defaulting it. A defaulted component is how a cache serves a result computed under a metric definition that no longer exists — that answer is not stale, it is WRONG, and it looks current. "Never share tenant result cache" is enforced structurally, in three layers, because the obvious one is a naming convention: * the key is prefixed with the tenant id, derived here from the verified identity and never passed in; * the stored ENTRY records its own tenant id, and every read checks the entry against the caller — so a prefix collision or a shared backing store still cannot serve a foreign result; * a cross-tenant hit is REFUSED and the poisoned entry EVICTED, not returned as a miss. Returning undefined would hide a security event. The third point is the one the test corpus exercises directly: an entry stored under t_alpha's key carrying t_beta's id, and a forged key that a SCAN glob would happily include. ST0RE was relying on naming conventions when it turned out to have four cross-tenant paths rather than the one reported. Staleness is tested by mutating each key component ONE AT A TIME and requiring a miss for every one — and a coverage assertion compares the mutation list against the component type, so adding a field to CacheKeyComponents without adding a mutation fails the suite instead of going untested. ValkeyResultCache takes the client as an injected interface, so no dependency and no CVE surface is added — the same reason BI-P005's Arrow writer is hand-written. It sets a store-side TTL as well as recording one (a recorded timestamp alone leaves expired results resident forever), pages with SCAN rather than blocking on KEYS, and removes a corrupt entry instead of letting it become a permanent silent miss. 32 tests. typecheck + eslint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 02:28:27 +02:00
Claude Opus 5	8f6357044f	feat(analytics): BI-P018/BI-P019/BI-P021 — execution model, verified cancellation, a budget that stops queries ... BI-P018 — every field the prompt names, plus the one it names by its absence. The record carries `compiled_query_hash` and never the statement, and that is ENFORCED: `assertNoRawSql` runs before any record is emitted and refuses one whose `error` field looks like SQL. The realistic leak is ClickHouse's own error text, which echoes the query — the test uses exactly that string. The recorder writes at EVERY transition, because a recorder that only writes at the end loses precisely the executions worth investigating. Illegal state transitions throw rather than overwrite a terminal row. The one place raw SQL is retained stays BI-P017's analyst statement history, where a privileged human ran unreviewed SQL and "who ran what" must survive. Different record, different retention posture, stated in both files. BI-P019 — "stops" is VERIFIED, not requested. Aborting the client stops US waiting, not the server; KILL QUERY without SYNC tells you nothing; even SYNC can return while the query unwinds. So cancelAndVerify aborts, kills, then POLLS the process list until the query is GONE or the deadline passes — and if the deadline passes it returns `verified_stopped: false` rather than a cheerful `true`. The return type is an outcome object, not a boolean, so a caller cannot mistake "we asked" for "it stopped". An unreachable engine is read as "still running" (fail closed), so the kill is still attempted at the moment an operator most needs it to be. Proven live against ClickHouse 24.8.14.39, with the counterfactual first: * an identical bomb left alone IS still running after 2s — so "gone after we cancelled" is not just "the query was short"; * the cancelled query had read 6,559,476,156 rows, and was verified gone 57ms after the request; * system.query_log records ExceptionWhileProcessing with code 394, QUERY_WAS_CANCELLED — ClickHouse's verdict, not ours. BI-P021 — the honest version. A budget checked once before execution bounds nothing, and the module says so in its first paragraph. ClickHouse's max_rows_to_read bounds ONE query; it cannot express "this tenant may scan X per hour", which is what a budget is for. So enforcement polls the engine's live read_rows/read_bytes counters and CANCELS the running query, through BI-P019 with verification, when window spend would be exceeded. Measured live: a 1,000,000-row window cancelled the query at 130,883,409 rows, 92ms in, verified stopped, code 394. THE OVERSHOOT IS DOCUMENTED, in the module and in the evidence: enforcement is granular to the poll interval, so this is a bound of the form "within roughly one poll interval of engine throughput", not a hard cap. Quoting it as a hard cap would be wrong. Two more things stated rather than implied: * `enforceInFlight: false` is reported by describeEnforcement() as "NOT ENFORCED — this budget is advisory", and the constructor REFUSES to accept enforceInFlight without a process control; * an unshared window store is reported as non-binding across replicas ("with N replicas a tenant can spend up to N times this budget"). * a query the budget killed is still charged for the work it burned. Not charging is how a tenant learns the cheapest query is a cancelled one. Evidence: evidence/bi-p019-p021/cancellation-and-budget.json 268 unit tests, 7 live tests. typecheck + eslint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 02:26:05 +02:00
Claude Opus 5	adc7a9be31	fix(analytics): replace the control-character class with a code-point scan (no-control-regex) ... The ASCII-only check used a regex character class of control characters. It tripped eslint's no-control-regex, which the repo's quality gate enforces — and the rule has a point: such a class is unreadable and is exactly the kind of expression a reviewer nods at without checking. Same behaviour, written as an explicit scan naming the permitted set. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 02:19:10 +02:00
Claude Opus 5	d9c8f7190b	feat(analytics): BI-P017 — analyst SQL contract, proven against live ClickHouse with the guard removed ... AnalyticsSQLSession is a SEPARATE class, not a method on SiaxAnalyticsEngine. executeMetric has no raw-SQL surface at all; on the same object the closed DSL would be one refactor away from growing one. ClickHouseAnalyticsEngine .executeQuery stays a documented refusal. FOUR LAYERS, and the guard is deliberately called the weakest: 1 role — the CL0UD analyst role is checked when the SESSION IS OPENED, so a caller without it never obtains an object that can run SQL. 2 guard — statement admission. Comments and literals removed with ClickHouse's own lexical rules, ASCII-only outside literals, one statement, SELECT/WITH only, and every table reference POSITIVELY resolved against the tenant's catalog. Deny-lists are a second opinion, never the mechanism. 3 grants — a per-tenant ClickHouse user with SELECT on that tenant's catalog alone, plus a restrictive row policy on the base tables. 4 views — the tenant catalog holds only VIEWS, each baking `WHERE tenant_id = '<tenant>'`, and tenant_id is not projected. 61 bypass attempts, in one shared corpus, run TWICE: against the guard, and against a live ClickHouse 24.8.14.39 with LAYER 2 REMOVED. The live assertion is not "the engine errored" — it is "no canary came back", with BETA_CANARY seeded into the other tenant's rows and SECRET_CANARY into an ungranted table. The counterfactual runs first: as admin the same base table returns 100 rows and BETA_CANARY IS readable, so the isolation numbers mean something. Result: 0 of 61 leaked. Evidence: evidence/bi-p017/analyst-sql-live-bypass.json. FOUR DEFECTS FOUND BY WRITING TESTS THAT CAN FAIL — three of them mine: * `CREATE ROW POLICY IF NOT EXISTS` is a SILENT NO-OP on re-provisioning. Dropping the user empties the policy's apply-to list; the next run skips re-creation, so the policy survives with `apply_to_list = []`, applying to nobody, while `system.row_policies` still shows the correct select_filter. The restricted user read 100 of 100 rows — BOTH TENANTS — from the base table. Fixed with OR REPLACE + `rowPolicyVerificationSql`, which turns "the policy is applied" from an assumption into a query, and a test that asserts applied_to_user = 1 per object. * `CREATE SETTINGS PROFILE IF NOT EXISTS` has the identical shape: a changed budget never lands. A stale 60-row ceiling survived re-provisioning and starved every legitimate statement, with the DDL reporting success throughout. Also OR REPLACE now. * `describe.skipIf(!reachable)` where `reachable` is set in `beforeAll` skips ALWAYS — vitest evaluates skipIf at collection time. Ten tests reported green while skipping against a live, reachable server. Replaced with the repo's existing env-var gate, which FAILS when CLICKHOUSE_URL is set and the server is down rather than degrading to a skip. * MEASURED, and it changes the design: a ClickHouse user whose profile carries `readonly = 1` cannot have ANY setting changed from the request (Code: 164). The BI-P012 ceilings therefore cannot travel with an analyst statement; they are baked into the user's SETTINGS PROFILE at provisioning. Proven both ways: over-budget scan fails with code 158 from the profile with no setting sent, and an attempt to raise it from the request fails with code 164. Honest about what the guard does NOT catch: `SELECT tenant_id FROM t_alpha_sql.sales` is structurally legitimate and IS admitted. It fails at the engine because the view does not project the column. Recorded in its own GUARD_ADMITS_ENGINE_REFUSES list and asserted live, rather than deleted from the corpus. evidence/bi-p017 is pinned by tests/analytics/analyst-sql-evidence.test.ts, which never skips: adding an attack without re-running the live suite breaks the build, and the pin also checks the counterfactual numbers so the evidence cannot be a recording of an empty database. 246 tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 02:17:07 +02:00
Claude Opus 5	b1884a89a5	feat(analytics): BI-P016 — canonical query request contract, two compile targets ... The eight-field AnalyticsQueryRequest BI-P016 names, verbatim, with the rule that makes it safe enforced structurally rather than by convention: there is no field that can carry SQL, and parseAnalyticsQueryRequest REJECTS unknown keys instead of ignoring them. Ignoring an unsupported field is how it becomes a supported one after the next refactor. `tenant`, `tenant_id`, `project_id`, `database`, `catalog` and `table` are rejected by name. INF0's Wave-2 cross-tenant hole was a tenant read out of a request body; the fix that sticks is a request type in which the field does not exist, and a parser that refuses it when a caller sends it anyway. BI-P016's DoD ("same request compiles to ClickHouse/semantic engine without consumer rewrite") is tested as an equality, not narrated: both targets are projected onto a comparable shape and compared. The ClickHouse side of that projection is obtained by PARSING the emitted SQL and its bound parameters in the test — not by reading a field the compiler helpfully returned — so a compiler that silently drops a filter fails the comparison instead of passing it. The parser throws on any condition it does not recognise, which is what would catch unbound caller text appearing in the statement. BI-P011's forced tenant predicate is re-proven against this new path: five request shapes are enumerated and each must have FORCED_TENANT_PREDICATE as condition[0]; a blank tenant refuses to compile rather than emitting no predicate; a hostile filter value appears only in params, never in sql. 45 tests, all failing-capable: every rejection assertion checks the SiaxError code, and the DoD assertions compare reconstructed meaning rather than strings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-07 02:00:07 +02:00
Claude Opus 5	84bb9faa4f	feat(analytics): BI-P002..BI-P012 analytics core — domain model, tiers, contracts, Arrow, ClickHouse engine, isolation, governance ... Implements the eight BI program prompts that make 0pt the canonical SIAX analytics capability. Every file is NEW: not one existing file is modified, and src/analytics/clickhouse-analytics.ts is byte-identical to main. BI-P002 — the nineteen canonical entities, verbatim, with tenant_id + project_id on every tenant-owned record. sql/008 gives each one a table. The four version entities BI-P002 names as immutable are enforced by BEFORE UPDATE/DELETE triggers, not by a TypeScript `readonly` — proven by direct SQL against a live Postgres. BI-P003 — RAW/NORMALIZED/CURATED/SERVING with the read rule enforced: assertReadableByBiSurface refuses a RAW dataset to a dashboard unless a recorded exception exists. All eight required declarations are NOT NULL. BI-P004 — DatasetContract and a fail-closed compatibility gate. Fifteen breaking-change classes are each tested twice: refused under a minor bump, accepted under a major one. BI-P005 — Arrow IPC alongside JSON, with a hand-written FlatBuffers writer so no npm dependency (or CVE surface) is added. Conformance comes from three independent implementations, not from self-agreement: pyarrow 25.0.1 and DuckDB 1.5.5 read identical values from both ClickHouse's Arrow and ours, and ClickHouse's own Arrow C++ ingests our 20,000-row batch with exact aggregates. BI-P009 — SiaxAnalyticsEngine with all eight capabilities. The existing ClickHouseAnalytics is COMPOSED, never replaced. executeQuery (analyst SQL) is declared and deliberately refused with CAPABILITY_UNAVAILABLE until BI-P017 and BI-P033 land, rather than opening the raw-SQL hole BI-P011 enumerates. BI-P010 — four purpose-built tables, not one giant one, and the difference is measured against the counterfactual: 48.8ms/40,960 rows vs 174.4ms/5,400,000 rows for the same question; 5/660 granules vs 660/1149. BI-P011 — layered isolation, each layer proven with the layer above removed. Without the forced predicate the same statement returns 50,050 (both tenants); with it, t_alpha gets 50. Without a row policy an unrestricted user sees 2 tenants; the restricted user running the IDENTICAL statement sees 1. BI-P012 — every named limit, fired individually and identified by ClickHouse error code, plus the half usually skipped: an unrelated tenant is timed while a real CPU bomb is in flight and must still return the CORRECT answer. Three defects found by writing tests that can fail: * `CHECK (array_length(purposes,1) >= 1)` can NEVER fail — array_length is NULL for '{}' and a NULL CHECK passes. Fixed with coalesce(); the test that caught it is pinned. * EXPLAIN ESTIMATE run under the caller's own max_rows_to_read fails with code 158 exactly when the estimate would be over budget. Planner calls now use a narrower envelope. * result_overflow_mode=break returns HTTP 200 with 65,409 of 100,000 rows and no indication it is short. Every SELECT ships `throw`. Verified rather than assumed: server7's ClickHouse holds `opt_scraped` (51 tables, 407,793,471 rows, `persons`/`cz_owners` included) — the b0ts plane this module is prohibited from touching — has no `opt_analytics` database, zero tenant_id columns and zero row policies. Nothing there was modified; every proof ran against disposable containers. Evidence: evidence/bi-p002-p012/{test-transcript.txt,bi-p010-benchmark.json, bi-p005-arrow-roundtrip.json,server7-clickhouse-probe.md} Docs: docs/analytics/BI_ANALYTICS_CORE.md Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-06 23:01:27 +02:00
Simon AxelssonandClaude Opus 5	9267576b7c	feat(data-os): P060-P068 durable ingestion plane — source classification gate, connector contract, common envelope, schema evolution, quality enforcement, queryable lineage, idempotent backfill ... BEFORE_STATE (verified, with evidence): * P060 PARTIAL/BROKEN. The registry persisted rights/provenance/health/ freshness to Postgres, but carried no authorization, no schedule, no checkpoint, no schema and no owner distinct from created_by — and the activation DoD was inverted: postgres-source-repository.ts wrote `${input.status ?? "active"}`, so a source with no rights, no provenance and no authorization was ACTIVE the instant it was registered. No activation operation existed at all. * P061 MISSING. services/api/sources/connectors.py had per-source fetch_*() functions only — no lifecycle, no cursor, no resume, no health, no conformance suite. * P062 MISSING, already documented as such in services/api/tests/test_known_gaps.py:10-19 (no CDC pipeline anywhere; workers/pipeline.py has no persistence at all). * P063 MISSING. schema_version existed as a free string; no comparison, no policy, no versioned source schema. * P064 PARTIAL. Rules/runs/issues/scores existed, but a failing rule only opened an issue and the data flowed on. No quarantine anywhere. * P065 DUPLICATE (confirmed). Two ER engines: src/er/entity-resolution- service.ts (canonical) and services/api/workers/resolve.py (simplified, still called by pipeline.py:110). * P067 PARTIAL. Lineage was emitted to AUD0 but nothing here could ANSWER a lineage question — no store, no query, no route. * P068 MISSING, documented in test_known_gaps.py:44-52 ("no delivery-store persistence layer exists to dedupe a backfill against"). BUILT (only what was PARTIAL/BROKEN/MISSING): * P060 sql/007_ingestion_plane.sql adds authorization/schedule/schemas/ checkpoints/ledger/quarantine/lineage tables, owner_id, and flips the opt_sources status DEFAULT to 'pending' so even a raw psql INSERT cannot produce an unclassified-but-active source. SourceRegistryService gains classifySource/activateSource/deactivateSource + authorization/schedule accessors; registration NEVER activates (asking for status:"active" goes through the gate, not around it); a service identity cannot authorize a source. Routes: activate (GET classification / POST activate), authorization, schedule. * P061 src/connectors/connector.ts — all nine operations on one interface. src/connectors/conformance.ts — a runnable suite (15 checks, no "skip" verdict). HttpJsonConnector (real HTTP, opaque cursor) and JsonlFileConnector (real file, line offsets, CDC deletes) both pass it. The suite caught a real defect while being written: an API that drops next_cursor at the end left the connector unable to resume, which is why HttpJsonConnectorConfig has cursor_field. * P062 src/types/ingestion.ts + src/ingestion/{envelope,ingestion-runner}.ts. One envelope for batch/stream/cdc; durable per-(source,partition) checkpoints; resume proven by killing a run inside the sink. * P063 src/schema/{record-schema,compatibility,schema-registry}.ts. Two enforcement points, both upstream of any consumer write: a discovery gate before the first read, and per-record conformance before the page reaches the sink. A producer cannot widen its own policy retroactively. * P064 src/quality/quality-policy.ts composes the EXISTING ExpectationEngine (not a second engine) and adds the missing consequence: block / quarantine / warn, with quarantine durable and recoverable. No policy is reported as "none_declared", never as a pass; an unexecutable expectation counts as a violation, never a silent pass. * P067 src/lineage/{lineage-edge,lineage-store}.ts + GET /api/0pt/lineage. AUD0 stays the system of record; this is 0pt's own queryable projection. * P068 opt_ingestion_ledger + IngestionRunner.backfill: bounded, idempotent on repeat, leaves the live checkpoint alone, still re-emits real changes. A BUG THIS PASS WROTE AND CAUGHT: the first IngestionRepository draft combined the idempotency check and its recording into one atomic call BEFORE the sink. The crash/resume test then delivered 7 of 10 records — the page in flight when the sink died was already marked delivered, so the resumed run skipped it as a duplicate. Permanent, silent data loss. Fixed by splitting peekDeliveries/commitDeliveries and committing both the ledger and the checkpoint only after the sink resolves. NOT DONE, stated rather than deferred quietly: * P065 is NOT met. resolve.py is still a second simplified production engine. Closing it means rewriting it as an adapter over the canonical service, which cannot be verified from this repo today: services/api is not built by this repo's Dockerfile, api.0pt.app answers 404, and NO CI workflow runs its pytest suite. The duplication is now stated at both sites in code and the removal plan is in STATUS.md. The new ingestion plane performs no ER, so it adds no third engine. * P066 (Iceberg) untouched — ST0RE is owned by another workflow. Contract read only. * Delivery is AT-LEAST-ONCE, not exactly-once (no per-partition run lock). * No CDC *source* exists; the contract is met, a CDC connector is not built. * 0pt does not schedule itself — the schedule row is a declaration ACT0 would read. VERIFIED: pnpm typecheck clean; pnpm test 365 passed / 25 skipped (was 311/18); pnpm build clean with all four new routes in the route table; lint unchanged at 12 pre-existing errors. Real Postgres 16 (tests/p0/ingestion-plane-postgres.realinfra.test.ts, 7 passed) plus a real `docker restart` of that container with an identical read-back through the production code path — that is P060's "a restart preserves all sources", measured. DEPLOYMENT REALITY (curl, 2026-09-04): 0pt.app/api/health returns 200 from Vercel with db:not_configured; 0pt.app/api/0pt/sources returns 404; api.0pt.app/health returns 404; cd.yml still guards on a placeholder Coolify app id. None of the capability layer runs in production. Nothing here is LIVE_VERIFIED and STATUS.md says so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 17:03:14 +02:00
Simon AxelssonandClaude Sonnet 5	dc6aaa0e10	fix(data-os): P014 — close tenant-isolation + idempotency gaps, execute real restore drill ... Tenant isolation (security-critical): every app/api/0pt/** route reading tenant_id from client-controlled query/body never cross-checked it against the token middleware.ts already verified (x-0pt-verified-tenant). Adds a shared assertTenantMatchesVerified guard and wires it into all 20+ affected routes. Also closes two full-leak-by-omission bugs found in this pass (documents/capture and lakehouse/mappings GETs returned every tenant's rows when tenant_id was simply omitted) and two "no tenant check at all" reads (rate-policies/[id], lakehouse/mappings/[id]) reachable by ID alone. Duplicate-delivery idempotency (correctness-critical): POST /api/orders now honors Idempotency-Key (sql/006, replay-or-409-on-conflict, Postgres + in-memory); the Python orchestrator's POST /orders/{id}/run — the actual delivery trigger — is now idempotent under an asyncio lock, closing a real concurrent-request race, not just the sequential-retry case. Engine-duplication resolution: adaptive_router.py's docstring wrongly invited a bypass ("see b0ts for scraping") contradicting the product's own compliance stance; corrected, and given real (additive, not escalating) block-signal tracking so a rate-limited/blocked source is excluded from the plan instead of hammered. Also fixes a previously-undiscovered production bug: the Python engine's only call into the TS quality bridge (X-Internal-Token) was unconditionally 401'd by middleware.ts, which never checked that credential — the advisory quality bridge was unreachable in any environment enforcing auth. Restore drill: executed for real against throwaway Docker Postgres containers (not simulated). Found and fixed two real blockers along the way: sql/004 referenced opt_datasets, which no migration ever created (chain was never actually runnable end to end); lib/db.ts hardcoded ssl:"require", making any non-TLS Postgres (incl. compose.dev.yaml's own local-dev container) unreachable. After both fixes: full migration chain applies cleanly, pg_dump/pg_restore round-trip is byte-identical (row counts + content checksums), and real application code (lib/orders-store.ts, SourceRegistryService's Postgres repository) reads the restored data back correctly, including cross-tenant isolation still holding. 82 new tests (55 TS + 27 Python), all passing; full existing suite (291 TS + 55 Python) still green; typecheck clean on every changed/added file. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-03 12:00:43 +02:00
Simon AxelssonandClaude Sonnet 5	9740e20818	feat(data-os): TS/Python quality bridge + provenance fields + fault-injection tests ... Continues P010 (branch renamed from feat/p010-source-registry-persistence): closes the three remaining BEFORE_STATE.md gaps (§2 TS/Python split-brain, §3 missing provenance fields, §4 untested failure modes), all additive to the production pipeline — the Python QA filter's delivery decision is unchanged by any of this. TS/Python bridge (§2): - New route surface app/api/0pt/quality/* (rules/runs/issues CRUD + POST /api/0pt/quality/bridge-run) over the shared QualityService singleton (src/services/quality.ts), fixing the same "no shared instance" defect the Source Registry had. - src/quality/quality-bridge.ts auto-provisions a deterministic, tenant-scoped baseline rule set on first use so the Python engine never pre-registers rules through the CRUD surface. - services/api/workers/quality_bridge.py: the Python-side HTTP client. Feature-flagged off by default (OPT_QUALITY_BRIDGE_ENABLED), fails open on any error (never raises into run_job, never changes `delivered`). Advisory only — real per-row gating against the TS engine isn't implementable yet (ExpectationEngine returns dataset-level pass/fail counts + samples, not a per-row boolean array); documented, not faked. - Live-verified end-to-end: `next dev` (AUTH_ENFORCE=false) + the real Python pipeline hitting the real brreg.no API, round-tripping through the real bridge route, receiving a real QualityService score. Provenance (§3): sources/connectors.py's Provenance dataclass gains retrieved_at, transform, connector_version, producer, quality, entity_resolution, artifact_ref (all previously absent from the real dataflow entirely) via a make_provenance() factory used by every connector. resolve.py stamps entity_resolution on merged records' provenance (never on unmerged ones). pipeline.py stamps artifact_ref at delivery and quality once the bridge succeeds. Failure tests (§4), real fault injection against real code (services/api, new pytest + respx suite, not vitest mocks): - dedup-determinism + duplicate-input + out-of-order: resolve.py under shuffled/interleaved/exhaustive-permutation input, plus an explicit, verified test of the one case that IS legitimately order-sensitive (non-transitive similarity in a greedy clusterer) instead of hiding it. - schema-evolution: respx-mocked upstream responses with added/removed/ alien-shaped fields through the real brreg connector + pipeline. - restart: connector exceptions mid-job (including one from respx itself) never crash run_job. - backfill: the real property that holds given no persistence layer exists (replay is compute-idempotent) — NOT a fabricated dedup-store test. - CDC-interruption and true backfill/delivery-store idempotency: explicitly documented as not implementable against this codebase (no CDC pipeline, no delivery store exist anywhere) via skip markers with reasons, in tests/test_known_gaps.py, rather than silently dropped or faked. Testing: 256/256 vitest (249 pre-existing + 7 new bridge route/service tests) and 28/28 pytest (2 documented skips) green locally. tsc -p tsconfig.check.json has zero new errors (14 pre-existing errors, all in the untracked/unaudited src/fleet, src/graph, src/metrics, src/projections paths this pass does not touch, per BEFORE_STATE.md). No PR opened, per instructions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-01 21:02:14 +02:00
Simon AxelssonandClaude Sonnet 5	07b13bd833	feat(data-os): Source Registry Postgres persistence + full HTTP route surface ... Replaces the in-memory (per-instance, not even per-process-shared) Maps in SourceService/SourceRegistryService with real Postgres persistence, and adds the register/query/describe/health/rights/provenance/freshness HTTP routes that never existed (grep for SourceRegistryService in app/ previously returned nothing). - sql/002_source_registry.sql: opt_sources, opt_source_rights, opt_source_provenance, opt_source_health_checks, opt_source_freshness. Numbered 002 (not 006) because 004's opt_source_rate_policies already forward-references opt_sources, which no sql/ file defined until now — fixes 004's pre-existing broken migration ordering (opt_datasets remains a separate, unrelated, still-open gap in 004, flagged not fixed). - src/repositories/*: SourceRepository / SourceRegistryRepository seams, each with an in-memory implementation (business logic extracted verbatim) and a Postgres implementation, chosen via resolveStoreMode — production still fails hard with no DATABASE_URL (Master-DoD §3), but a configured production now actually works instead of always being blocked. - SourceService / SourceRegistryService: storage moved behind the repository seam; getHealth/getFreshness are now async and tenant-scoped at the accessor itself (needed once they're reachable directly over HTTP, not only after an earlier caller's tenant check). - src/services/source-registry.ts: shared singleton instances for the sources/* route group, fixing the "each route module owned its own private registry" defect (source-rate-policies and .../query previously couldn't see a source registered via POST /api/0pt/sources). - New routes: POST/GET /api/0pt/sources, GET /api/0pt/sources/[id], GET+PUT .../rights, GET+PUT .../provenance, GET+POST .../health, GET .../freshness. - Business logic unchanged: fail-closed purpose gate, cross-tenant NOT_FOUND (no existence oracle, §50), bounded 100-row health history. Tested against a real Postgres container (not mocked) — tests/p0/source-registry-postgres.realinfra.test.ts, env-gated on SOURCE_REGISTRY_TEST_DATABASE_URL, skips honestly when unset. All 249 pre-existing P0 tests + 7 new realinfra tests pass; also smoke-tested the full HTTP route surface end-to-end against next dev + live Postgres. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-01 20:42:41 +02:00
Simon AxelssonandClaude Fable 5	c47e3b468e	test(integration)+ci+docs: live OpenSearch/ClickHouse suites, CI service containers, STATUS.md (§56) ... - tests/integration/: 10 tests against REAL OpenSearch 2.11 (tenant isolation, purpose/rights filters, generation rebuild/rollback, projection-disaster rebuild §68) + REAL ClickHouse 24.8 (tenant-isolated metric DSL, TTL schema, injection-surface refusals). Honest skip without env - and CI fails hard if the suites skip instead of running. - .gitea/workflows/ci.yml: new blocking 'integration' job with self-identifying service containers (st0re ci.yml's measured alias-collision discipline: run-unique cluster.name / credentials, getent-enumerated IP selection, no-match = hard fail). - STATUS.md: full 8-level Master-DoD classification per capability incl. the honest gaps blocking PRODUCTION_READY. - dependency-mode: NEXT_PHASE=phase-production-build is not a serving runtime (next build page-data collection must compile; guard re-arms at boot) - pinned by contract test. - docs/02-architecture.md: superseded the accurate-but-now-fixed 'every client runs against its mock' paragraph; .env.example: client URLs now actually read from env + new P0 backend vars. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-31 16:56:33 +02:00
Simon AxelssonandClaude Fable 5	b46807da75	feat(data-os): P0 capability layer — source registry, search, quality, dataset versions, metadata, analytics, features, ER ... - Source Registry (§10): rights/license/purpose metadata, provenance, freshness SLA, health checks, describe/query; purpose gate FAILS CLOSED (no rights record => deny); cross-tenant probes => NOT_FOUND. - Search (§10/§42): OpenSearch REST adapter (no SDK) + SearchService with MANDATORY tenant + purpose/rights filtering, versioned index definitions, bulk, generation-per-rebuild (opt-{name}--g{n}) with atomic alias cutover, rollback, failed-rebuild generation discard. OpenSearch never canonical. - Quality (§10): versioned rule registry (GE-JSON interchange), local engine for the GE core subset (never silently passes unsupported types), runs/results/issues/score/trend, whole-run delegation to the GE backend via N0D (0pt.quality.ge_run) - never fabricated results. - Dataset (§10/§5/§53): immutable numbered versions holding canonical st0re:artifact refs only, idempotent createVersion, export registers the ST0RE artifact + AUD0 evidence + OpenLineage. - Metadata (§10): OpenMetadata PUT /api/v1/tables projection adapter + projectAll rebuild path; projection never canonical. - Analytics (§10): ClickHouse HTTP adapter, closed metric-query DSL with FORCED tenant predicate + readonly/max_execution_time/max_result_rows, TTL retention. Zero connection to b0ts/quarantined pipelines. - Features (§10): versioned feature-view registry, real Feast get-online-features adapter, materialize/get-historical as N0D delegation. - ER (§10): generic candidate API with deterministic blocking + transparent scoring + per-field explanations, product hard-rule hook (force both directions), review-band handoff, Splink batch via N0D contract. 93 new unit/contract tests; suite 248 passed / 1 skipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-31 16:49:32 +02:00
Simon AxelssonandClaude Fable 5	b7038d7675	feat(platform): canonical SIAX context/errors + real-contract clients, no prod mock fallback ... - src/platform/: SiaxRequestContext + propagation headers + 12 canonical error codes (documented mirror of cl0ud packages/contract), SiaxError, resource refs (st0re:artifact:...), and the §3/§10 dependency-mode guard: production runtime + unconfigured dependency/store => hard DEPENDENCY_UNAVAILABLE at construction, never silent mock/in-memory. - src/clients/: all four platform clients rewritten against the REAL wire contracts verified on sibling main branches this session: * cl0ud: POST /api/policy/evaluate ({data} envelope, allow+requiresApproval) - the old /v1/cl0ud/policies/evaluate path never existed on CL0UD * st0re: camelCase ArtifactInput on /v1/st0re/artifacts/register incl. X-Tenant-Id/X-Actor-Id + X-Idempotency-Key - the old snake_case body would have been a 400 against the live registry * n0d: required_role + resource_requirements + body idempotency_key + CloudEvents callback_url per n0d createJobSchema; adds getJob/getJobResult * aud0: /v1/aud0/openlineage is AUD0's only lineage ingest - /v1/aud0/lineage/events never existed; legacy shape converts to a spec-shaped OpenLineage RunEvent (deterministic runId dedup) - src/lineage/: OpenLineage subset types + LineageEmitter (never-throw, §16 evidence semantics) - shared SiaxHttp transport: timeouts, typed SiaxError mapping, context propagation, no raw upstream bodies in errors Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-31 16:35:01 +02:00
adminandClaude Sonnet 5	15a0dbe3b1	fix(security): require auth on GET /api/orders and GET /api/admin/metrics ... A live compliance audit against 0pt.app confirmed both endpoints returned HTTP 200 with zero authentication -- the full order queue and financial pipeline aggregates (deposit/pipeline totals) were readable by anyone. Root cause: middleware.ts's matcher only covers /api/0pt/:path*, by design ("does NOT touch ... the unrelated payment/order APIs"), so nothing upstream of these two handlers ever verified the caller. Fix: both GET handlers now call the same fail-closed Zitadel verifier /api/0pt/* relies on (resolveAuthContext, src/auth/context.ts) directly, via a new thin wrapper (src/auth/require-route-auth.ts) that maps a rejected decision to the same 401/403 JSON shape middleware.ts already returns. The wrapper re-verifies the bearer token itself rather than trusting any x-0pt-verified-* header on the request, since middleware never runs in front of these routes to overwrite a client-supplied one. POST /api/orders (anonymous draft/quote creation before the deposit-payment step) is intentionally left as-is -- only the read paths that leaked data are gated. Tracked as part of docs/compliance/BACKLOG.md P0-3. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-11 04:06:13 +02:00
SIAX Ops	44e83ad1de	resolve: README adopts main (newer, fail-closed auth docs)	2026-08-09 11:30:18 +02:00
Claude (SIAX platform session)andClaude Opus 5	bcf285f057	merge: unify GitHub main into Gitea main (0pt) ... Merges github/main (6 commits: 17-bug audit fix, CI quality gate, .env.example + .gitignore fix, missing platform clients + registry services, docs accuracy overhaul) into gitea/main (8 commits: Iceberg write boundary + live ST0RE verification, lakehouse mapping routes, Waves 1-4 foundational hardening). 16 conflicts. Both sides had independently written the four src/clients/*.ts and the two registry services, so those are combined rather than picked: - st0re-client.ts: GitHub's endpoint wins on evidence. POST goes to /v1/st0re/artifacts/register, not /v1/st0re/artifacts -- verified in sax3l/st0re src/api/routes/artifacts.ts, where the bare path is a GET-only list route. Gitea's URL normalisation, error-body text and mock kept. - n0d-client.ts: field is `status`, not `state`, with N0D's real 13-state union -- verified in sax3l/n0d packages/schemas/n0d.job.schema.json, which says explicitly not to add "succeeded". Gitea's mock echo, URL normalisation and error-body text kept. - cl0ud-client.ts: Gitea's fail-closed HTTP normalisation kept (a 200 with no `decision` reads as "deny", not as a silent allow). GitHub's PolicyDecision/PolicyDecisionValue naming kept; PolicyResult retained as an alias. - aud0-client.ts: permissive union of both input types (optional payload/metadata/correlation_id), GitHub's echo return, Gitea's URL normalisation and error-body text, plus a JSON-parse fallback so a successful POST with an empty body does not throw. - dataset-service.ts / source-service.ts: GitHub's record/DTO split and event naming (0pt.source_created / 0pt.create_source, which the merged docs already reference) plus Gitea's name validation, description / created_by fields and actor_id derivation. List filters are the union. - lakehouse mapping routes: GitHub's shared lakehouse-mapping-registry singletons (without which every write 400'd "mapping not found") AND Gitea's Zod parseBody validation. Both, not either. - src/types/source.ts, src/types/dataset.ts: unions of both vocabularies; GitHub's `access` legal-sourcing field kept. - .env.example: GitHub's audited inventory plus Gitea's Zitadel / AUTH_ENFORCE block. - CLAUDE.md, docs/ICEBERG_WRITE_BOUNDARY.md: GitHub's corrections kept, Gitea's live-verification kept and marked as superseding the "would 400 outright" finding (Gitea had already added the required X-Tenant-Id/X-Actor-Id headers). - pnpm-lock.yaml: regenerated with pnpm 9.15.9, not merged. Also corrects CLAUDE.md's claim that the banned services/scrapers stack was "since removed" -- the removal PR is open, not merged, and this branch inherits that code from Gitea main. Nothing here resurrects, extends or re-wires any of it; services/scrapers/ is byte-identical to Gitea main. Verified on the merged tree: typecheck (src/ + tests/) clean; 17 test files, 121 passed, 1 skipped (the realinfra test skipping honestly on ST0RE HTTP 401), 0 failures. Gitea main baseline was the same 121/1; GitHub main baseline could not even `pnpm install` ("packages field missing or empty") -- Gitea's pnpm-workspace.yaml fix resolves that. Whole-repo `typecheck:app` reports 9 errors, an identical set to Gitea main's baseline, so it is advisory in both CI workflows rather than deleted. Lint: 16 problems, identical to Gitea baseline. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-28 12:38:01 +02:00
Claude (SIAX platform session)	1c1e2cb649	fix(auth): make the AUTH_ENFORCE default fail-closed on /api/0pt/* ... The auth seam read `process.env.AUTH_ENFORCE === 'true'`, so an absent env var meant NO authentication: any fresh container, local run, DR restore or redeploy that dropped the variable served /api/0pt/* anonymously, taking the tenant from the caller-supplied X-Tenant-Id header. 0pt.app is deployed and was answering anonymously (verified: an invalid Bearer token plus an arbitrary X-Tenant-Id returned 200), so this was a live hole, not latent. * authEnforced() now returns true unless AUTH_ENFORCE is explicitly "false" (trimmed, case-insensitive). Unset/empty/misspelled => ENFORCED. * assertAuthConfigSafe() throws AuthMisconfigurationError when NODE_ENV=production and enforcement resolved off, naming AUTH_ENFORCE. It runs at middleware module load (startup) and on the env-driven request path, so a production instance can neither boot nor serve a single unauthenticated request with auth switched off. * .env.example no longer teaches AUTH_ENFORCE=false; README's Deploy section states the fail-closed posture (no DEPLOY.md in this repo). * 7 new tests, incl. the regression case: with AUTH_ENFORCE unset and no `enforce` option, an unauthenticated request is rejected with 401. No test was inverted or deleted -- none asserted the open default; the three enforce:false cases now describe the explicit local-dev escape hatch. Tests: 121 passed / 1 skipped before, 128 passed / 1 skipped after. Typecheck and build clean; advisory lint unchanged (16 pre-existing problems). Scope: auth default only. The matcher still covers /api/0pt/* alone, and the handlers still take tenant_id from the query/body rather than the verified principal -- both described in the PR body, neither fixed here.	2026-07-27 13:28:24 +02:00
SIAX Foundation BotandClaude Opus 4.8	1d76d50f3d	feat(auth,foundation): Wave 2 flag-gated Zitadel JWT verify + validation + eslint + tests ... Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-25 14:32:34 +02:00
SIAX Foundation Bot	844220d9f5	feat(foundation): Wave 3 integrity/persistence/build fixes	2026-07-25 13:54:36 +02:00
adminandClaude Opus 4.8	e6d3101501	fix: 17 confirmed bugs from full-repo audit (edge + Python engine) (#5) ... * docs: correct stale quality-check-service.ts comment references Two source comments still cited src/services/quality-check-service.ts as an existing precedent. It doesn't exist anywhere in this repo's history (git log --all: zero commits, any branch, ever) -- confirmed during the earlier full-doc-corpus accuracy pass. Points each comment at the real precedent instead (document-extraction-service.ts / iceberg-write-service.ts). Applied from task_05ec36aa, spun off during that pass. * fix: 17 confirmed bugs from full-repo audit (edge + Python engine) Found by an 8-finder bug hunt across app/api, src/services, src/clients, frontend, the Python engine, the data layer, config/infra, and tests, then independently adversarially verified per finding (32 raw findings -> 27 confirmed real bugs; 5 refuted). This commit applies the 17 confirmed as safe, mechanical fixes -- no architecture/design decision required. The other 10 need a decision and are reported separately, not touched here. Most consequential, app/api/payments/[order_id]/deposit/route.ts: the client-supplied Origin request header was trusted unsanitized both as a server-side fetch target (SSRF) and as Stripe Checkout success/cancel redirect URLs (open redirect tied to a real payment flow), and a failed order lookup silently fell through to a hardcoded deposit amount instead of 404. Fixed by reading the order directly via lib/orders-store.ts (mirroring the sibling release/route.ts, which already did this correctly) instead of both trusting the Origin header and self-issuing an HTTP fetch. Also fixed: - 5 routes (orders, orders/[id], estimates, subscriptions, ai/parse) that 500'd instead of 400ing on a literal JSON `null` body, since req.json() resolves (not rejects) for that input and bypasses the try/catch guard. - app/api/0pt/lakehouse/mappings/{,[mappingId]/{,writes/}}route.ts each instantiated their own private service instances, so a mapping created via POST was invisible to the sibling GET and writes routes even before the separately-tracked dataset-registration gap. Extracted a shared singleton module, src/services/lakehouse-mapping-registry.ts. - app/api/0pt/documents/[documentId]/extraction/route.ts collapsed every error (not-configured, policy-denied, policy-approval-required, not-found) into a blanket 400; now status-mapped like the sibling sources/[sourceId]/query route. - app/api/vehicles/route.ts: plate query param wasn't URL-encoded before being concatenated into the outbound RDW request, allowing extra query parameters to be injected; added a strict allowlist + encodeURIComponent. - app/api/uploads/route.ts: POST returned a fake 200 success with put_url: null even when "configured", instead of continuing to report 501 until presign generation is actually implemented. - src/adapters/document-intelligence-adapter.ts: the result-polling loop never checked response.ok, so a non-2xx poll response (auth expiry, 429, 5xx) was silently retried until a generic timeout masked the real error. - app/order/[id]/page.tsx: finish() never cleared the production-simulation setInterval, so it kept firing (and re-rendering) forever after delivery. - app/admin/page.tsx: the metrics fetch had no response/shape validation (unlike the adjacent orders fetch), rendering literal "undefined" stat tiles on a malformed response instead of falling back to <ErrorState/>. - services/api/workers/pipeline.py: `config.get("seeds") or DEMO_SEEDS` used truthy-or, so an explicit empty seeds dict was silently replaced with hardcoded demo org numbers instead of being respected as empty. - services/api/workers/resolve.py: corroboration (confidence/agreed_sources bump) no longer fires for two hits from the same source under fuzzy match -- only a genuinely independent source now counts, per docs/06's own ">=2 independent sources agree" definition -- and a merged record's provenance array now actually includes the corroborating source's fields instead of only a bare source-name string. - services/api/orchestrator/main.py: create_order's int(...) on unvalidated client volume/confidence fields raised on any decimal-formatted input (int("92.5") -> ValueError); now coerced via float() first with a safe default. /vehicles/{plate} and /screen also now catch connector exceptions and degrade gracefully, matching pipeline.py's existing pattern for the same connectors. - lib/orders-store.ts: postgres.js (pinned v3.4.9) returns bigint/numeric columns as strings by default; getOrder/listOrders/setStatus were silently handing callers stringified money fields. Added a narrow per-row coercion instead of touching the shared client's global type map (which would affect unrelated tables' bigint/numeric columns too). - lib/subscriptions-store.ts: the in-memory fallback returned oldest-first while the Postgres branch explicitly orders newest-first; now consistent, mirroring the identical fix already in orders-store.ts. - app/api/auth/login/route.ts: DEV_LOGIN_KEY was compared with a plain !==, a non-constant-time comparison; replaced with crypto.timingSafeEqual. Verified before commit: tsc --noEmit clean; vitest run 88 passed, 1 todo, 0 failures (13 files, unchanged from before this commit -- no existing test's assertions needed to change). services/api/workers/resolve.py's source-novelty and provenance-merge logic additionally verified by direct execution against synthetic same-source/cross-source inputs (no Python test suite exists in this repo to add to). services/api/workers/pipeline.py and services/api/orchestrator/main.py verified with python -m py_compile (no httpx/fastapi installed in this environment to run them directly). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(engine): treat explicit seeds:null as missing to avoid 500 in run_job Four-eyes follow-up to this PR's own seeds change. Switching from `config.get('seeds') or DEMO_SEEDS` to `config['seeds'] if 'seeds' in config` correctly stopped an explicit empty {} from being backfilled with DEMO_SEEDS, but regressed on client-controlled `{"seeds": null}`: seeds became None and the downstream `seeds.get(market, [])` raised AttributeError -> 500. That path is reachable (POST /orders stores raw body.config; POST /orders/{id}/run passes it straight to run_job). Using `config.get('seeds') is not None` keeps the intended behaviour (explicit {} honoured, never silently seeded with demo data) while restoring the None-tolerance the old idiom had. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-22 18:52:19 +02:00
Simon AxelssonandClaude Opus 5	01cc225ffc	fix: add missing 0pt platform clients + registry services, wire test/typecheck scripts ... The 0pt data-engine build has been broken on main since 2026-07-08 (commit c897311). document-capture-service.ts, document-extraction-service.ts, source-query-service.ts, external-writeback-service.ts (and, more recently, dataset-iceberg-mapping-service.ts, iceberg-write-service.ts, source-rate-policy-service.ts) all import six platform client modules and two registry services that were never committed: src/clients/{aud0,cl0ud,n0d, st0re}-client.ts, src/services/{dataset,source}-service.ts, plus src/types/{dataset,source}.ts. `next build` failed with 17 "Module not found" errors, and every route under app/api/0pt/* returned HTTP 500 in dev with the Turbopack compile error leaked straight into the JSON response body. This adds the missing files, following the same fetch-based client shape already established by src/clients/st0re-lakehouse-client.ts (thin interface, HTTP implementation, deterministic in-memory mock used when no base URL is configured) and the same "pure core, thin in-memory registry, AUD0-evented mutation" shape the existing SourceRatePolicyService/DatasetIcebergMappingService already use for DatasetService/SourceService. Each new client mock is documented as a mock/degrade path, never a fabricated success, per this repo's CLAUDE.md honesty doctrine. Also wires up "typecheck" and "test" npm scripts (tsc --noEmit / vitest run) and adds vitest as a devDependency -- previously `pnpm test` failed immediately with ERR_PNPM_NO_SCRIPT even though 13 test files (~89 cases) already existed under tests/p0/. Verified: `pnpm run typecheck` clean, `pnpm run test` 88 passed / 1 todo (pre-existing live-ST0RE-only skip), `pnpm run build` exits 0 (39/39 routes compiled, only the pre-existing intentional optional `mssql` import warning remains). Re-verified live on port 4097: GET /api/0pt/sources/rate-policies, GET /api/0pt/lakehouse/mappings, POST /api/0pt/documents/capture, GET /api/0pt/documents/[id]/extraction, and both GET/POST /api/0pt/sources/[id]/query now return real application responses (200/400/405) instead of HTTP 500 with a leaked compile error. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-14 01:16:50 +02:00
adminandClaude Sonnet 5	bda21e87eb	Live-verify Iceberg write path against ST0RE + close field/contract gaps ... Re-runs this session's task spec against 0pt's actual repo state, which turned out to already have a substantial, tested implementation from a prior pass (35b6ec6): dataset<->Iceberg-table mapping, source rate policy, and an ST0RE lakehouse client that was honestly marked "not live-verified". This pass found api.st0re.siax.io IS reachable from this environment and used it -- first via curl to safely discover the real contract, then through 0pt's own updated code -- to drive a real namespace -> table -> snapshot write against production ST0RE (task item 3's "write ONE first test dataset" proof). That surfaced and fixed three real bugs the mock-only pass could not have caught: - ST0RE requires X-Tenant-Id/X-Actor-Id headers on every lakehouse route; the previous HTTP client sent none. Fixed via a threaded St0reRequestContext. - IcebergTableInput.datasetId is a foreign key into ST0RE's OWN dataset table, not a free-form string -- sending 0pt's dataset_id 400s live with a Postgres FK violation. Fixed by no longer sending it; 0pt's dataset<->table linkage lives entirely in DatasetIcebergMapping. - ST0RE enforces a real unique constraint on lakehouse namespaces (tenant_id, project_id, name) with no upsert route. Fixed with a client-side lookup-or-create (listNamespaces + findOrRegisterNamespace). Also closed a field-level gap against the task's literal schema spec: 0pt.dataset_iceberg_mapping.schema.v1 now carries namespace_id (a real reference to ST0RE's LakehouseNamespace.namespaceId, not guessed) and mapped_at, both additive alongside the existing namespace/created_at fields (sql/005, JSON schema, types, service). tests/p0/iceberg-write-service.realinfra.test.ts replaces the prior it.todo with a real SEAM-pattern live test: attempts the real HTTP call, skips honestly if ST0RE is unreachable, never fabricates success. It ran successfully against production ST0RE this session (real namespace/table/ snapshot UUIDs logged in docs/ICEBERG_WRITE_BOUNDARY.md). Two ST0RE-side cross-tenant read bugs were found and flagged (not fixed here -- out of scope for this repo): GET .../namespaces ignores its own tenant header and query filters; GET .../tables/:id doesn't check the caller's tenant against the table's owner. npm test: 155 passed, 0 failed, 0 todo (up from 154 passed + 1 todo). npx tsc --noEmit: 1 pre-existing, unrelated error in quality-report-service.ts (present before this pass, part of the wider uncommitted P0 data-platform build already in this repo) -- not touched. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-13 00:18:10 +02:00