16 Commits

Include renames

Author	SHA1	Message	Date
Claude Opus 5	84bb9faa4f	feat(analytics): BI-P002..BI-P012 analytics core — domain model, tiers, contracts, Arrow, ClickHouse engine, isolation, governance ... Implements the eight BI program prompts that make 0pt the canonical SIAX analytics capability. Every file is NEW: not one existing file is modified, and src/analytics/clickhouse-analytics.ts is byte-identical to main. BI-P002 — the nineteen canonical entities, verbatim, with tenant_id + project_id on every tenant-owned record. sql/008 gives each one a table. The four version entities BI-P002 names as immutable are enforced by BEFORE UPDATE/DELETE triggers, not by a TypeScript `readonly` — proven by direct SQL against a live Postgres. BI-P003 — RAW/NORMALIZED/CURATED/SERVING with the read rule enforced: assertReadableByBiSurface refuses a RAW dataset to a dashboard unless a recorded exception exists. All eight required declarations are NOT NULL. BI-P004 — DatasetContract and a fail-closed compatibility gate. Fifteen breaking-change classes are each tested twice: refused under a minor bump, accepted under a major one. BI-P005 — Arrow IPC alongside JSON, with a hand-written FlatBuffers writer so no npm dependency (or CVE surface) is added. Conformance comes from three independent implementations, not from self-agreement: pyarrow 25.0.1 and DuckDB 1.5.5 read identical values from both ClickHouse's Arrow and ours, and ClickHouse's own Arrow C++ ingests our 20,000-row batch with exact aggregates. BI-P009 — SiaxAnalyticsEngine with all eight capabilities. The existing ClickHouseAnalytics is COMPOSED, never replaced. executeQuery (analyst SQL) is declared and deliberately refused with CAPABILITY_UNAVAILABLE until BI-P017 and BI-P033 land, rather than opening the raw-SQL hole BI-P011 enumerates. BI-P010 — four purpose-built tables, not one giant one, and the difference is measured against the counterfactual: 48.8ms/40,960 rows vs 174.4ms/5,400,000 rows for the same question; 5/660 granules vs 660/1149. BI-P011 — layered isolation, each layer proven with the layer above removed. Without the forced predicate the same statement returns 50,050 (both tenants); with it, t_alpha gets 50. Without a row policy an unrestricted user sees 2 tenants; the restricted user running the IDENTICAL statement sees 1. BI-P012 — every named limit, fired individually and identified by ClickHouse error code, plus the half usually skipped: an unrelated tenant is timed while a real CPU bomb is in flight and must still return the CORRECT answer. Three defects found by writing tests that can fail: * `CHECK (array_length(purposes,1) >= 1)` can NEVER fail — array_length is NULL for '{}' and a NULL CHECK passes. Fixed with coalesce(); the test that caught it is pinned. * EXPLAIN ESTIMATE run under the caller's own max_rows_to_read fails with code 158 exactly when the estimate would be over budget. Planner calls now use a narrower envelope. * result_overflow_mode=break returns HTTP 200 with 65,409 of 100,000 rows and no indication it is short. Every SELECT ships `throw`. Verified rather than assumed: server7's ClickHouse holds `opt_scraped` (51 tables, 407,793,471 rows, `persons`/`cz_owners` included) — the b0ts plane this module is prohibited from touching — has no `opt_analytics` database, zero tenant_id columns and zero row policies. Nothing there was modified; every proof ran against disposable containers. Evidence: evidence/bi-p002-p012/{test-transcript.txt,bi-p010-benchmark.json, bi-p005-arrow-roundtrip.json,server7-clickhouse-probe.md} Docs: docs/analytics/BI_ANALYTICS_CORE.md Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-06 23:01:27 +02:00
Simon AxelssonandClaude Opus 5	9267576b7c	feat(data-os): P060-P068 durable ingestion plane — source classification gate, connector contract, common envelope, schema evolution, quality enforcement, queryable lineage, idempotent backfill ... BEFORE_STATE (verified, with evidence): * P060 PARTIAL/BROKEN. The registry persisted rights/provenance/health/ freshness to Postgres, but carried no authorization, no schedule, no checkpoint, no schema and no owner distinct from created_by — and the activation DoD was inverted: postgres-source-repository.ts wrote `${input.status ?? "active"}`, so a source with no rights, no provenance and no authorization was ACTIVE the instant it was registered. No activation operation existed at all. * P061 MISSING. services/api/sources/connectors.py had per-source fetch_*() functions only — no lifecycle, no cursor, no resume, no health, no conformance suite. * P062 MISSING, already documented as such in services/api/tests/test_known_gaps.py:10-19 (no CDC pipeline anywhere; workers/pipeline.py has no persistence at all). * P063 MISSING. schema_version existed as a free string; no comparison, no policy, no versioned source schema. * P064 PARTIAL. Rules/runs/issues/scores existed, but a failing rule only opened an issue and the data flowed on. No quarantine anywhere. * P065 DUPLICATE (confirmed). Two ER engines: src/er/entity-resolution- service.ts (canonical) and services/api/workers/resolve.py (simplified, still called by pipeline.py:110). * P067 PARTIAL. Lineage was emitted to AUD0 but nothing here could ANSWER a lineage question — no store, no query, no route. * P068 MISSING, documented in test_known_gaps.py:44-52 ("no delivery-store persistence layer exists to dedupe a backfill against"). BUILT (only what was PARTIAL/BROKEN/MISSING): * P060 sql/007_ingestion_plane.sql adds authorization/schedule/schemas/ checkpoints/ledger/quarantine/lineage tables, owner_id, and flips the opt_sources status DEFAULT to 'pending' so even a raw psql INSERT cannot produce an unclassified-but-active source. SourceRegistryService gains classifySource/activateSource/deactivateSource + authorization/schedule accessors; registration NEVER activates (asking for status:"active" goes through the gate, not around it); a service identity cannot authorize a source. Routes: activate (GET classification / POST activate), authorization, schedule. * P061 src/connectors/connector.ts — all nine operations on one interface. src/connectors/conformance.ts — a runnable suite (15 checks, no "skip" verdict). HttpJsonConnector (real HTTP, opaque cursor) and JsonlFileConnector (real file, line offsets, CDC deletes) both pass it. The suite caught a real defect while being written: an API that drops next_cursor at the end left the connector unable to resume, which is why HttpJsonConnectorConfig has cursor_field. * P062 src/types/ingestion.ts + src/ingestion/{envelope,ingestion-runner}.ts. One envelope for batch/stream/cdc; durable per-(source,partition) checkpoints; resume proven by killing a run inside the sink. * P063 src/schema/{record-schema,compatibility,schema-registry}.ts. Two enforcement points, both upstream of any consumer write: a discovery gate before the first read, and per-record conformance before the page reaches the sink. A producer cannot widen its own policy retroactively. * P064 src/quality/quality-policy.ts composes the EXISTING ExpectationEngine (not a second engine) and adds the missing consequence: block / quarantine / warn, with quarantine durable and recoverable. No policy is reported as "none_declared", never as a pass; an unexecutable expectation counts as a violation, never a silent pass. * P067 src/lineage/{lineage-edge,lineage-store}.ts + GET /api/0pt/lineage. AUD0 stays the system of record; this is 0pt's own queryable projection. * P068 opt_ingestion_ledger + IngestionRunner.backfill: bounded, idempotent on repeat, leaves the live checkpoint alone, still re-emits real changes. A BUG THIS PASS WROTE AND CAUGHT: the first IngestionRepository draft combined the idempotency check and its recording into one atomic call BEFORE the sink. The crash/resume test then delivered 7 of 10 records — the page in flight when the sink died was already marked delivered, so the resumed run skipped it as a duplicate. Permanent, silent data loss. Fixed by splitting peekDeliveries/commitDeliveries and committing both the ledger and the checkpoint only after the sink resolves. NOT DONE, stated rather than deferred quietly: * P065 is NOT met. resolve.py is still a second simplified production engine. Closing it means rewriting it as an adapter over the canonical service, which cannot be verified from this repo today: services/api is not built by this repo's Dockerfile, api.0pt.app answers 404, and NO CI workflow runs its pytest suite. The duplication is now stated at both sites in code and the removal plan is in STATUS.md. The new ingestion plane performs no ER, so it adds no third engine. * P066 (Iceberg) untouched — ST0RE is owned by another workflow. Contract read only. * Delivery is AT-LEAST-ONCE, not exactly-once (no per-partition run lock). * No CDC *source* exists; the contract is met, a CDC connector is not built. * 0pt does not schedule itself — the schedule row is a declaration ACT0 would read. VERIFIED: pnpm typecheck clean; pnpm test 365 passed / 25 skipped (was 311/18); pnpm build clean with all four new routes in the route table; lint unchanged at 12 pre-existing errors. Real Postgres 16 (tests/p0/ingestion-plane-postgres.realinfra.test.ts, 7 passed) plus a real `docker restart` of that container with an identical read-back through the production code path — that is P060's "a restart preserves all sources", measured. DEPLOYMENT REALITY (curl, 2026-09-04): 0pt.app/api/health returns 200 from Vercel with db:not_configured; 0pt.app/api/0pt/sources returns 404; api.0pt.app/health returns 404; cd.yml still guards on a placeholder Coolify app id. None of the capability layer runs in production. Nothing here is LIVE_VERIFIED and STATUS.md says so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 17:03:14 +02:00
Simon AxelssonandClaude Sonnet 5	dc6aaa0e10	fix(data-os): P014 — close tenant-isolation + idempotency gaps, execute real restore drill ... Tenant isolation (security-critical): every app/api/0pt/** route reading tenant_id from client-controlled query/body never cross-checked it against the token middleware.ts already verified (x-0pt-verified-tenant). Adds a shared assertTenantMatchesVerified guard and wires it into all 20+ affected routes. Also closes two full-leak-by-omission bugs found in this pass (documents/capture and lakehouse/mappings GETs returned every tenant's rows when tenant_id was simply omitted) and two "no tenant check at all" reads (rate-policies/[id], lakehouse/mappings/[id]) reachable by ID alone. Duplicate-delivery idempotency (correctness-critical): POST /api/orders now honors Idempotency-Key (sql/006, replay-or-409-on-conflict, Postgres + in-memory); the Python orchestrator's POST /orders/{id}/run — the actual delivery trigger — is now idempotent under an asyncio lock, closing a real concurrent-request race, not just the sequential-retry case. Engine-duplication resolution: adaptive_router.py's docstring wrongly invited a bypass ("see b0ts for scraping") contradicting the product's own compliance stance; corrected, and given real (additive, not escalating) block-signal tracking so a rate-limited/blocked source is excluded from the plan instead of hammered. Also fixes a previously-undiscovered production bug: the Python engine's only call into the TS quality bridge (X-Internal-Token) was unconditionally 401'd by middleware.ts, which never checked that credential — the advisory quality bridge was unreachable in any environment enforcing auth. Restore drill: executed for real against throwaway Docker Postgres containers (not simulated). Found and fixed two real blockers along the way: sql/004 referenced opt_datasets, which no migration ever created (chain was never actually runnable end to end); lib/db.ts hardcoded ssl:"require", making any non-TLS Postgres (incl. compose.dev.yaml's own local-dev container) unreachable. After both fixes: full migration chain applies cleanly, pg_dump/pg_restore round-trip is byte-identical (row counts + content checksums), and real application code (lib/orders-store.ts, SourceRegistryService's Postgres repository) reads the restored data back correctly, including cross-tenant isolation still holding. 82 new tests (55 TS + 27 Python), all passing; full existing suite (291 TS + 55 Python) still green; typecheck clean on every changed/added file. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-03 12:00:43 +02:00
Simon AxelssonandClaude Sonnet 5	9740e20818	feat(data-os): TS/Python quality bridge + provenance fields + fault-injection tests ... Continues P010 (branch renamed from feat/p010-source-registry-persistence): closes the three remaining BEFORE_STATE.md gaps (§2 TS/Python split-brain, §3 missing provenance fields, §4 untested failure modes), all additive to the production pipeline — the Python QA filter's delivery decision is unchanged by any of this. TS/Python bridge (§2): - New route surface app/api/0pt/quality/* (rules/runs/issues CRUD + POST /api/0pt/quality/bridge-run) over the shared QualityService singleton (src/services/quality.ts), fixing the same "no shared instance" defect the Source Registry had. - src/quality/quality-bridge.ts auto-provisions a deterministic, tenant-scoped baseline rule set on first use so the Python engine never pre-registers rules through the CRUD surface. - services/api/workers/quality_bridge.py: the Python-side HTTP client. Feature-flagged off by default (OPT_QUALITY_BRIDGE_ENABLED), fails open on any error (never raises into run_job, never changes `delivered`). Advisory only — real per-row gating against the TS engine isn't implementable yet (ExpectationEngine returns dataset-level pass/fail counts + samples, not a per-row boolean array); documented, not faked. - Live-verified end-to-end: `next dev` (AUTH_ENFORCE=false) + the real Python pipeline hitting the real brreg.no API, round-tripping through the real bridge route, receiving a real QualityService score. Provenance (§3): sources/connectors.py's Provenance dataclass gains retrieved_at, transform, connector_version, producer, quality, entity_resolution, artifact_ref (all previously absent from the real dataflow entirely) via a make_provenance() factory used by every connector. resolve.py stamps entity_resolution on merged records' provenance (never on unmerged ones). pipeline.py stamps artifact_ref at delivery and quality once the bridge succeeds. Failure tests (§4), real fault injection against real code (services/api, new pytest + respx suite, not vitest mocks): - dedup-determinism + duplicate-input + out-of-order: resolve.py under shuffled/interleaved/exhaustive-permutation input, plus an explicit, verified test of the one case that IS legitimately order-sensitive (non-transitive similarity in a greedy clusterer) instead of hiding it. - schema-evolution: respx-mocked upstream responses with added/removed/ alien-shaped fields through the real brreg connector + pipeline. - restart: connector exceptions mid-job (including one from respx itself) never crash run_job. - backfill: the real property that holds given no persistence layer exists (replay is compute-idempotent) — NOT a fabricated dedup-store test. - CDC-interruption and true backfill/delivery-store idempotency: explicitly documented as not implementable against this codebase (no CDC pipeline, no delivery store exist anywhere) via skip markers with reasons, in tests/test_known_gaps.py, rather than silently dropped or faked. Testing: 256/256 vitest (249 pre-existing + 7 new bridge route/service tests) and 28/28 pytest (2 documented skips) green locally. tsc -p tsconfig.check.json has zero new errors (14 pre-existing errors, all in the untracked/unaudited src/fleet, src/graph, src/metrics, src/projections paths this pass does not touch, per BEFORE_STATE.md). No PR opened, per instructions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-01 21:02:14 +02:00
Simon AxelssonandClaude Sonnet 5	07b13bd833	feat(data-os): Source Registry Postgres persistence + full HTTP route surface ... Replaces the in-memory (per-instance, not even per-process-shared) Maps in SourceService/SourceRegistryService with real Postgres persistence, and adds the register/query/describe/health/rights/provenance/freshness HTTP routes that never existed (grep for SourceRegistryService in app/ previously returned nothing). - sql/002_source_registry.sql: opt_sources, opt_source_rights, opt_source_provenance, opt_source_health_checks, opt_source_freshness. Numbered 002 (not 006) because 004's opt_source_rate_policies already forward-references opt_sources, which no sql/ file defined until now — fixes 004's pre-existing broken migration ordering (opt_datasets remains a separate, unrelated, still-open gap in 004, flagged not fixed). - src/repositories/*: SourceRepository / SourceRegistryRepository seams, each with an in-memory implementation (business logic extracted verbatim) and a Postgres implementation, chosen via resolveStoreMode — production still fails hard with no DATABASE_URL (Master-DoD §3), but a configured production now actually works instead of always being blocked. - SourceService / SourceRegistryService: storage moved behind the repository seam; getHealth/getFreshness are now async and tenant-scoped at the accessor itself (needed once they're reachable directly over HTTP, not only after an earlier caller's tenant check). - src/services/source-registry.ts: shared singleton instances for the sources/* route group, fixing the "each route module owned its own private registry" defect (source-rate-policies and .../query previously couldn't see a source registered via POST /api/0pt/sources). - New routes: POST/GET /api/0pt/sources, GET /api/0pt/sources/[id], GET+PUT .../rights, GET+PUT .../provenance, GET+POST .../health, GET .../freshness. - Business logic unchanged: fail-closed purpose gate, cross-tenant NOT_FOUND (no existence oracle, §50), bounded 100-row health history. Tested against a real Postgres container (not mocked) — tests/p0/source-registry-postgres.realinfra.test.ts, env-gated on SOURCE_REGISTRY_TEST_DATABASE_URL, skips honestly when unset. All 249 pre-existing P0 tests + 7 new realinfra tests pass; also smoke-tested the full HTTP route surface end-to-end against next dev + live Postgres. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-01 20:42:41 +02:00
Simon AxelssonandClaude Fable 5	c47e3b468e	test(integration)+ci+docs: live OpenSearch/ClickHouse suites, CI service containers, STATUS.md (§56) ... - tests/integration/: 10 tests against REAL OpenSearch 2.11 (tenant isolation, purpose/rights filters, generation rebuild/rollback, projection-disaster rebuild §68) + REAL ClickHouse 24.8 (tenant-isolated metric DSL, TTL schema, injection-surface refusals). Honest skip without env - and CI fails hard if the suites skip instead of running. - .gitea/workflows/ci.yml: new blocking 'integration' job with self-identifying service containers (st0re ci.yml's measured alias-collision discipline: run-unique cluster.name / credentials, getent-enumerated IP selection, no-match = hard fail). - STATUS.md: full 8-level Master-DoD classification per capability incl. the honest gaps blocking PRODUCTION_READY. - dependency-mode: NEXT_PHASE=phase-production-build is not a serving runtime (next build page-data collection must compile; guard re-arms at boot) - pinned by contract test. - docs/02-architecture.md: superseded the accurate-but-now-fixed 'every client runs against its mock' paragraph; .env.example: client URLs now actually read from env + new P0 backend vars. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-31 16:56:33 +02:00
Simon AxelssonandClaude Fable 5	b46807da75	feat(data-os): P0 capability layer — source registry, search, quality, dataset versions, metadata, analytics, features, ER ... - Source Registry (§10): rights/license/purpose metadata, provenance, freshness SLA, health checks, describe/query; purpose gate FAILS CLOSED (no rights record => deny); cross-tenant probes => NOT_FOUND. - Search (§10/§42): OpenSearch REST adapter (no SDK) + SearchService with MANDATORY tenant + purpose/rights filtering, versioned index definitions, bulk, generation-per-rebuild (opt-{name}--g{n}) with atomic alias cutover, rollback, failed-rebuild generation discard. OpenSearch never canonical. - Quality (§10): versioned rule registry (GE-JSON interchange), local engine for the GE core subset (never silently passes unsupported types), runs/results/issues/score/trend, whole-run delegation to the GE backend via N0D (0pt.quality.ge_run) - never fabricated results. - Dataset (§10/§5/§53): immutable numbered versions holding canonical st0re:artifact refs only, idempotent createVersion, export registers the ST0RE artifact + AUD0 evidence + OpenLineage. - Metadata (§10): OpenMetadata PUT /api/v1/tables projection adapter + projectAll rebuild path; projection never canonical. - Analytics (§10): ClickHouse HTTP adapter, closed metric-query DSL with FORCED tenant predicate + readonly/max_execution_time/max_result_rows, TTL retention. Zero connection to b0ts/quarantined pipelines. - Features (§10): versioned feature-view registry, real Feast get-online-features adapter, materialize/get-historical as N0D delegation. - ER (§10): generic candidate API with deterministic blocking + transparent scoring + per-field explanations, product hard-rule hook (force both directions), review-band handoff, Splink batch via N0D contract. 93 new unit/contract tests; suite 248 passed / 1 skipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-31 16:49:32 +02:00
Simon AxelssonandClaude Fable 5	da58176735	test(contracts): per-client wire-contract suites + §3 production-guard tests ... 38 tests: recording provider double answers with the sibling services' real response shapes; each suite asserts the exact request wire shape (path, method, tenant/actor/idempotency headers, camelCase vs snake_case body) so a regression to the pre-rewrite invented shapes fails loudly. Includes the fail-closed policy cases (missing decision => deny, CL0UD down => typed error, never implicit allow) and the production factory guard for all four clients. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-31 16:37:57 +02:00
Simon AxelssonandClaude Fable 5	b7038d7675	feat(platform): canonical SIAX context/errors + real-contract clients, no prod mock fallback ... - src/platform/: SiaxRequestContext + propagation headers + 12 canonical error codes (documented mirror of cl0ud packages/contract), SiaxError, resource refs (st0re:artifact:...), and the §3/§10 dependency-mode guard: production runtime + unconfigured dependency/store => hard DEPENDENCY_UNAVAILABLE at construction, never silent mock/in-memory. - src/clients/: all four platform clients rewritten against the REAL wire contracts verified on sibling main branches this session: * cl0ud: POST /api/policy/evaluate ({data} envelope, allow+requiresApproval) - the old /v1/cl0ud/policies/evaluate path never existed on CL0UD * st0re: camelCase ArtifactInput on /v1/st0re/artifacts/register incl. X-Tenant-Id/X-Actor-Id + X-Idempotency-Key - the old snake_case body would have been a 400 against the live registry * n0d: required_role + resource_requirements + body idempotency_key + CloudEvents callback_url per n0d createJobSchema; adds getJob/getJobResult * aud0: /v1/aud0/openlineage is AUD0's only lineage ingest - /v1/aud0/lineage/events never existed; legacy shape converts to a spec-shaped OpenLineage RunEvent (deterministic runId dedup) - src/lineage/: OpenLineage subset types + LineageEmitter (never-throw, §16 evidence semantics) - shared SiaxHttp transport: timeouts, typed SiaxError mapping, context propagation, no raw upstream bodies in errors Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-31 16:35:01 +02:00
admin	a88ad04867	fix(health): report real per-dependency status instead of hardcoded ok:true ... BACKLOG P0-3: /api/health only ever checked *presence* of DATABASE_URL/ REDIS_URL and always returned ok:true regardless of whether anything was reachable -- ARCHITECTURE.md's own words, "an uptime monitor watching HTTP 200 would show green for a dying instance". lib/health-checks.ts adds real dependency probes: - db: a live `select 1` round trip (not just DATABASE_URL presence) - engine: a real HTTP call to the FastAPI engine's own GET /health - stripe: Stripe's documented lightweight balance.retrieve() liveness call - queue: honestly reports "not_integrated" when REDIS_URL is set but unused -- no Redis client is wired anywhere in this repo, so reporting "ok" would fabricate a check that doesn't exist /api/health stays a liveness check (always 200 while the process can respond; the body's `ok` field is the honest signal). New GET /api/ready fails closed with 503 when the load-bearing dependency (db) isn't both configured and reachable -- the "no /ready route" gap. contracts/openapi.yaml, docs/architecture/ARCHITECTURE.md, and docs/slo.md updated to match. 17 new unit tests (dependency-injected fakes, no real network/DB) + 5 route-level tests.	2026-08-11 13:49:23 +02:00
adminandClaude Sonnet 5	15a0dbe3b1	fix(security): require auth on GET /api/orders and GET /api/admin/metrics ... A live compliance audit against 0pt.app confirmed both endpoints returned HTTP 200 with zero authentication -- the full order queue and financial pipeline aggregates (deposit/pipeline totals) were readable by anyone. Root cause: middleware.ts's matcher only covers /api/0pt/:path*, by design ("does NOT touch ... the unrelated payment/order APIs"), so nothing upstream of these two handlers ever verified the caller. Fix: both GET handlers now call the same fail-closed Zitadel verifier /api/0pt/* relies on (resolveAuthContext, src/auth/context.ts) directly, via a new thin wrapper (src/auth/require-route-auth.ts) that maps a rejected decision to the same 401/403 JSON shape middleware.ts already returns. The wrapper re-verifies the bearer token itself rather than trusting any x-0pt-verified-* header on the request, since middleware never runs in front of these routes to overwrite a client-supplied one. POST /api/orders (anonymous draft/quote creation before the deposit-payment step) is intentionally left as-is -- only the read paths that leaked data are gated. Tracked as part of docs/compliance/BACKLOG.md P0-3. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-11 04:06:13 +02:00
Claude (SIAX platform session)	1c1e2cb649	fix(auth): make the AUTH_ENFORCE default fail-closed on /api/0pt/* ... The auth seam read `process.env.AUTH_ENFORCE === 'true'`, so an absent env var meant NO authentication: any fresh container, local run, DR restore or redeploy that dropped the variable served /api/0pt/* anonymously, taking the tenant from the caller-supplied X-Tenant-Id header. 0pt.app is deployed and was answering anonymously (verified: an invalid Bearer token plus an arbitrary X-Tenant-Id returned 200), so this was a live hole, not latent. * authEnforced() now returns true unless AUTH_ENFORCE is explicitly "false" (trimmed, case-insensitive). Unset/empty/misspelled => ENFORCED. * assertAuthConfigSafe() throws AuthMisconfigurationError when NODE_ENV=production and enforcement resolved off, naming AUTH_ENFORCE. It runs at middleware module load (startup) and on the env-driven request path, so a production instance can neither boot nor serve a single unauthenticated request with auth switched off. * .env.example no longer teaches AUTH_ENFORCE=false; README's Deploy section states the fail-closed posture (no DEPLOY.md in this repo). * 7 new tests, incl. the regression case: with AUTH_ENFORCE unset and no `enforce` option, an unauthenticated request is rejected with 401. No test was inverted or deleted -- none asserted the open default; the three enforce:false cases now describe the explicit local-dev escape hatch. Tests: 121 passed / 1 skipped before, 128 passed / 1 skipped after. Typecheck and build clean; advisory lint unchanged (16 pre-existing problems). Scope: auth default only. The matcher still covers /api/0pt/* alone, and the handlers still take tenant_id from the query/body rather than the verified principal -- both described in the PR body, neither fixed here.	2026-07-27 13:28:24 +02:00
SIAX Foundation BotandClaude Opus 4.8	1d76d50f3d	feat(auth,foundation): Wave 2 flag-gated Zitadel JWT verify + validation + eslint + tests ... Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-25 14:32:34 +02:00
adminandClaude Sonnet 5	bda21e87eb	Live-verify Iceberg write path against ST0RE + close field/contract gaps ... Re-runs this session's task spec against 0pt's actual repo state, which turned out to already have a substantial, tested implementation from a prior pass (35b6ec6): dataset<->Iceberg-table mapping, source rate policy, and an ST0RE lakehouse client that was honestly marked "not live-verified". This pass found api.st0re.siax.io IS reachable from this environment and used it -- first via curl to safely discover the real contract, then through 0pt's own updated code -- to drive a real namespace -> table -> snapshot write against production ST0RE (task item 3's "write ONE first test dataset" proof). That surfaced and fixed three real bugs the mock-only pass could not have caught: - ST0RE requires X-Tenant-Id/X-Actor-Id headers on every lakehouse route; the previous HTTP client sent none. Fixed via a threaded St0reRequestContext. - IcebergTableInput.datasetId is a foreign key into ST0RE's OWN dataset table, not a free-form string -- sending 0pt's dataset_id 400s live with a Postgres FK violation. Fixed by no longer sending it; 0pt's dataset<->table linkage lives entirely in DatasetIcebergMapping. - ST0RE enforces a real unique constraint on lakehouse namespaces (tenant_id, project_id, name) with no upsert route. Fixed with a client-side lookup-or-create (listNamespaces + findOrRegisterNamespace). Also closed a field-level gap against the task's literal schema spec: 0pt.dataset_iceberg_mapping.schema.v1 now carries namespace_id (a real reference to ST0RE's LakehouseNamespace.namespaceId, not guessed) and mapped_at, both additive alongside the existing namespace/created_at fields (sql/005, JSON schema, types, service). tests/p0/iceberg-write-service.realinfra.test.ts replaces the prior it.todo with a real SEAM-pattern live test: attempts the real HTTP call, skips honestly if ST0RE is unreachable, never fabricates success. It ran successfully against production ST0RE this session (real namespace/table/ snapshot UUIDs logged in docs/ICEBERG_WRITE_BOUNDARY.md). Two ST0RE-side cross-tenant read bugs were found and flagged (not fixed here -- out of scope for this repo): GET .../namespaces ignores its own tenant header and query filters; GET .../tables/:id doesn't check the caller's tenant against the table's owner. npm test: 155 passed, 0 failed, 0 todo (up from 154 passed + 1 todo). npx tsc --noEmit: 1 pre-existing, unrelated error in quality-report-service.ts (present before this pass, part of the wider uncommitted P0 data-platform build already in this repo) -- not touched. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-13 00:18:10 +02:00
adminandClaude Sonnet 5	35b6ec63bf	Add 0pt Iceberg write-path ownership: dataset<->table mapping + source rate policy ... Real, tested implementation of docs/11-lakehouse-write-plan.md's items 2-8 (the "future implementation session" that doc named): 0pt is now the canonical writer of Iceberg tables through ST0RE's real, shipped /v1/st0re/lakehouse/* table registry (sax3l/st0re commit 9ea0abb), never persisting lakehouse table/catalog/snapshot state itself. - src/types + src/services for 0pt.dataset_iceberg_mapping.schema.v1 (dataset -> ST0RE tableId reference) and 0pt.source_rate_policy.schema.v1 (the sourceRatePolicy ownership repair, moved out of N0D per ADR-0030). - src/clients/st0re-lakehouse-client.ts: new client matching ST0RE's actual camelCase route contract, with the same mock/HTTP boundary convention as this repo's existing cl0ud/n0d/aud0/st0re clients. Honest caveat in its module doc comment: not verified against a live ST0RE instance in this pass (no reachable ST0RE_URL); every test exercises the deterministic mock. - src/services/iceberg-write-service.ts: registers tables/snapshots through ST0RE, requests N0D jobs only for heavy writes (never executes them in-process), emits all four AUD0 event names (dataset_iceberg_table_mapped / iceberg_write_requested / iceberg_snapshot_created / source_rate_policy_created). - sql/004, packages/schemas/*.json, app/api/0pt/lakehouse + sources/rate-policies routes, tests/p0/*.test.ts (154 passed + 1 honest it.todo for the live-ST0RE case, up from 142 baseline, 0 failures). - Reconciled a real collision: an earlier untracked draft schema/test pair (tests/p0/enterprise-data-os-expansion.test.ts) asserted a different, now-superseded source_rate_policy shape; updated that one test block only. - docs/ICEBERG_WRITE_BOUNDARY.md states the ownership boundary plainly and is explicit throughout about what's real-and-tested vs. contract-only. docs/11-lakehouse-write-plan.md gets a status-update section, not a rewrite. - Data quality rules (quality-check-service.ts, quality-report-service.ts) were not touched -- confirmed still 0pt-owned. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-12 19:46:47 +02:00
adminandClaude Sonnet 5	c89731115c	feat: add 0pt.query_source, 0pt.document_ocr, 0pt.write_external_record capabilities ... Three new, tenant-agnostic P0 capabilities: - 0pt.query_source: bounded, timeout/row-capped ad-hoc SQL query execution against registered database sources, via a Fabric/OneLake adapter (Azure AD credential-based mssql connection), with AUD0 audit events and an app/api/0pt/sources/[sourceId]/query endpoint. - 0pt.document_ocr (document capture + extraction): mailbox-based document intake via Microsoft Graph, OCR/field extraction via Azure Document Intelligence, with capture/extraction API endpoints and services that never fabricate extracted values. - 0pt.write_external_record (external writeback): a generic writeback-adapter contract plus a concrete Odoo bills adapter that matches partners via Fabric, backing an external-writeback-service for pushing records to downstream systems of record. Known interim gaps: - src/adapters/local-blob-store.ts is an interim ST0RE substitute pending the real ST0RE artifact-store integration. - mssql is used by the Fabric/OneLake adapter but deliberately not yet added to package.json (pending `pnpm add mssql` once the concurrent session's package.json/dependency changes settle). A separate, concurrently-active session has unrelated in-flight changes in this same working tree (data-platform catalog/lineage/feature-store, dataset quality, lakehouse/Iceberg docs, deploy scaffolding, package.json/ pnpm-lock.yaml updates, etc.) — none of that is touched or included here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-08 19:32:43 +02:00