13 Commits

Include renames

Author	SHA1	Message	Date
Simon AxelssonandClaude Sonnet 5	9740e20818	feat(data-os): TS/Python quality bridge + provenance fields + fault-injection tests ... Continues P010 (branch renamed from feat/p010-source-registry-persistence): closes the three remaining BEFORE_STATE.md gaps (§2 TS/Python split-brain, §3 missing provenance fields, §4 untested failure modes), all additive to the production pipeline — the Python QA filter's delivery decision is unchanged by any of this. TS/Python bridge (§2): - New route surface app/api/0pt/quality/* (rules/runs/issues CRUD + POST /api/0pt/quality/bridge-run) over the shared QualityService singleton (src/services/quality.ts), fixing the same "no shared instance" defect the Source Registry had. - src/quality/quality-bridge.ts auto-provisions a deterministic, tenant-scoped baseline rule set on first use so the Python engine never pre-registers rules through the CRUD surface. - services/api/workers/quality_bridge.py: the Python-side HTTP client. Feature-flagged off by default (OPT_QUALITY_BRIDGE_ENABLED), fails open on any error (never raises into run_job, never changes `delivered`). Advisory only — real per-row gating against the TS engine isn't implementable yet (ExpectationEngine returns dataset-level pass/fail counts + samples, not a per-row boolean array); documented, not faked. - Live-verified end-to-end: `next dev` (AUTH_ENFORCE=false) + the real Python pipeline hitting the real brreg.no API, round-tripping through the real bridge route, receiving a real QualityService score. Provenance (§3): sources/connectors.py's Provenance dataclass gains retrieved_at, transform, connector_version, producer, quality, entity_resolution, artifact_ref (all previously absent from the real dataflow entirely) via a make_provenance() factory used by every connector. resolve.py stamps entity_resolution on merged records' provenance (never on unmerged ones). pipeline.py stamps artifact_ref at delivery and quality once the bridge succeeds. Failure tests (§4), real fault injection against real code (services/api, new pytest + respx suite, not vitest mocks): - dedup-determinism + duplicate-input + out-of-order: resolve.py under shuffled/interleaved/exhaustive-permutation input, plus an explicit, verified test of the one case that IS legitimately order-sensitive (non-transitive similarity in a greedy clusterer) instead of hiding it. - schema-evolution: respx-mocked upstream responses with added/removed/ alien-shaped fields through the real brreg connector + pipeline. - restart: connector exceptions mid-job (including one from respx itself) never crash run_job. - backfill: the real property that holds given no persistence layer exists (replay is compute-idempotent) — NOT a fabricated dedup-store test. - CDC-interruption and true backfill/delivery-store idempotency: explicitly documented as not implementable against this codebase (no CDC pipeline, no delivery store exist anywhere) via skip markers with reasons, in tests/test_known_gaps.py, rather than silently dropped or faked. Testing: 256/256 vitest (249 pre-existing + 7 new bridge route/service tests) and 28/28 pytest (2 documented skips) green locally. tsc -p tsconfig.check.json has zero new errors (14 pre-existing errors, all in the untracked/unaudited src/fleet, src/graph, src/metrics, src/projections paths this pass does not touch, per BEFORE_STATE.md). No PR opened, per instructions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-01 21:02:14 +02:00
Simon AxelssonandClaude Sonnet 5	07b13bd833	feat(data-os): Source Registry Postgres persistence + full HTTP route surface ... Replaces the in-memory (per-instance, not even per-process-shared) Maps in SourceService/SourceRegistryService with real Postgres persistence, and adds the register/query/describe/health/rights/provenance/freshness HTTP routes that never existed (grep for SourceRegistryService in app/ previously returned nothing). - sql/002_source_registry.sql: opt_sources, opt_source_rights, opt_source_provenance, opt_source_health_checks, opt_source_freshness. Numbered 002 (not 006) because 004's opt_source_rate_policies already forward-references opt_sources, which no sql/ file defined until now — fixes 004's pre-existing broken migration ordering (opt_datasets remains a separate, unrelated, still-open gap in 004, flagged not fixed). - src/repositories/*: SourceRepository / SourceRegistryRepository seams, each with an in-memory implementation (business logic extracted verbatim) and a Postgres implementation, chosen via resolveStoreMode — production still fails hard with no DATABASE_URL (Master-DoD §3), but a configured production now actually works instead of always being blocked. - SourceService / SourceRegistryService: storage moved behind the repository seam; getHealth/getFreshness are now async and tenant-scoped at the accessor itself (needed once they're reachable directly over HTTP, not only after an earlier caller's tenant check). - src/services/source-registry.ts: shared singleton instances for the sources/* route group, fixing the "each route module owned its own private registry" defect (source-rate-policies and .../query previously couldn't see a source registered via POST /api/0pt/sources). - New routes: POST/GET /api/0pt/sources, GET /api/0pt/sources/[id], GET+PUT .../rights, GET+PUT .../provenance, GET+POST .../health, GET .../freshness. - Business logic unchanged: fail-closed purpose gate, cross-tenant NOT_FOUND (no existence oracle, §50), bounded 100-row health history. Tested against a real Postgres container (not mocked) — tests/p0/source-registry-postgres.realinfra.test.ts, env-gated on SOURCE_REGISTRY_TEST_DATABASE_URL, skips honestly when unset. All 249 pre-existing P0 tests + 7 new realinfra tests pass; also smoke-tested the full HTTP route surface end-to-end against next dev + live Postgres. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-01 20:42:41 +02:00
Simon AxelssonandClaude Fable 5	c47e3b468e	test(integration)+ci+docs: live OpenSearch/ClickHouse suites, CI service containers, STATUS.md (§56) ... - tests/integration/: 10 tests against REAL OpenSearch 2.11 (tenant isolation, purpose/rights filters, generation rebuild/rollback, projection-disaster rebuild §68) + REAL ClickHouse 24.8 (tenant-isolated metric DSL, TTL schema, injection-surface refusals). Honest skip without env - and CI fails hard if the suites skip instead of running. - .gitea/workflows/ci.yml: new blocking 'integration' job with self-identifying service containers (st0re ci.yml's measured alias-collision discipline: run-unique cluster.name / credentials, getent-enumerated IP selection, no-match = hard fail). - STATUS.md: full 8-level Master-DoD classification per capability incl. the honest gaps blocking PRODUCTION_READY. - dependency-mode: NEXT_PHASE=phase-production-build is not a serving runtime (next build page-data collection must compile; guard re-arms at boot) - pinned by contract test. - docs/02-architecture.md: superseded the accurate-but-now-fixed 'every client runs against its mock' paragraph; .env.example: client URLs now actually read from env + new P0 backend vars. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-31 16:56:33 +02:00
Simon AxelssonandClaude Fable 5	b46807da75	feat(data-os): P0 capability layer — source registry, search, quality, dataset versions, metadata, analytics, features, ER ... - Source Registry (§10): rights/license/purpose metadata, provenance, freshness SLA, health checks, describe/query; purpose gate FAILS CLOSED (no rights record => deny); cross-tenant probes => NOT_FOUND. - Search (§10/§42): OpenSearch REST adapter (no SDK) + SearchService with MANDATORY tenant + purpose/rights filtering, versioned index definitions, bulk, generation-per-rebuild (opt-{name}--g{n}) with atomic alias cutover, rollback, failed-rebuild generation discard. OpenSearch never canonical. - Quality (§10): versioned rule registry (GE-JSON interchange), local engine for the GE core subset (never silently passes unsupported types), runs/results/issues/score/trend, whole-run delegation to the GE backend via N0D (0pt.quality.ge_run) - never fabricated results. - Dataset (§10/§5/§53): immutable numbered versions holding canonical st0re:artifact refs only, idempotent createVersion, export registers the ST0RE artifact + AUD0 evidence + OpenLineage. - Metadata (§10): OpenMetadata PUT /api/v1/tables projection adapter + projectAll rebuild path; projection never canonical. - Analytics (§10): ClickHouse HTTP adapter, closed metric-query DSL with FORCED tenant predicate + readonly/max_execution_time/max_result_rows, TTL retention. Zero connection to b0ts/quarantined pipelines. - Features (§10): versioned feature-view registry, real Feast get-online-features adapter, materialize/get-historical as N0D delegation. - ER (§10): generic candidate API with deterministic blocking + transparent scoring + per-field explanations, product hard-rule hook (force both directions), review-band handoff, Splink batch via N0D contract. 93 new unit/contract tests; suite 248 passed / 1 skipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-31 16:49:32 +02:00
Simon AxelssonandClaude Fable 5	da58176735	test(contracts): per-client wire-contract suites + §3 production-guard tests ... 38 tests: recording provider double answers with the sibling services' real response shapes; each suite asserts the exact request wire shape (path, method, tenant/actor/idempotency headers, camelCase vs snake_case body) so a regression to the pre-rewrite invented shapes fails loudly. Includes the fail-closed policy cases (missing decision => deny, CL0UD down => typed error, never implicit allow) and the production factory guard for all four clients. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-31 16:37:57 +02:00
Simon AxelssonandClaude Fable 5	b7038d7675	feat(platform): canonical SIAX context/errors + real-contract clients, no prod mock fallback ... - src/platform/: SiaxRequestContext + propagation headers + 12 canonical error codes (documented mirror of cl0ud packages/contract), SiaxError, resource refs (st0re:artifact:...), and the §3/§10 dependency-mode guard: production runtime + unconfigured dependency/store => hard DEPENDENCY_UNAVAILABLE at construction, never silent mock/in-memory. - src/clients/: all four platform clients rewritten against the REAL wire contracts verified on sibling main branches this session: * cl0ud: POST /api/policy/evaluate ({data} envelope, allow+requiresApproval) - the old /v1/cl0ud/policies/evaluate path never existed on CL0UD * st0re: camelCase ArtifactInput on /v1/st0re/artifacts/register incl. X-Tenant-Id/X-Actor-Id + X-Idempotency-Key - the old snake_case body would have been a 400 against the live registry * n0d: required_role + resource_requirements + body idempotency_key + CloudEvents callback_url per n0d createJobSchema; adds getJob/getJobResult * aud0: /v1/aud0/openlineage is AUD0's only lineage ingest - /v1/aud0/lineage/events never existed; legacy shape converts to a spec-shaped OpenLineage RunEvent (deterministic runId dedup) - src/lineage/: OpenLineage subset types + LineageEmitter (never-throw, §16 evidence semantics) - shared SiaxHttp transport: timeouts, typed SiaxError mapping, context propagation, no raw upstream bodies in errors Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-31 16:35:01 +02:00
admin	a88ad04867	fix(health): report real per-dependency status instead of hardcoded ok:true ... BACKLOG P0-3: /api/health only ever checked *presence* of DATABASE_URL/ REDIS_URL and always returned ok:true regardless of whether anything was reachable -- ARCHITECTURE.md's own words, "an uptime monitor watching HTTP 200 would show green for a dying instance". lib/health-checks.ts adds real dependency probes: - db: a live `select 1` round trip (not just DATABASE_URL presence) - engine: a real HTTP call to the FastAPI engine's own GET /health - stripe: Stripe's documented lightweight balance.retrieve() liveness call - queue: honestly reports "not_integrated" when REDIS_URL is set but unused -- no Redis client is wired anywhere in this repo, so reporting "ok" would fabricate a check that doesn't exist /api/health stays a liveness check (always 200 while the process can respond; the body's `ok` field is the honest signal). New GET /api/ready fails closed with 503 when the load-bearing dependency (db) isn't both configured and reachable -- the "no /ready route" gap. contracts/openapi.yaml, docs/architecture/ARCHITECTURE.md, and docs/slo.md updated to match. 17 new unit tests (dependency-injected fakes, no real network/DB) + 5 route-level tests.	2026-08-11 13:49:23 +02:00
adminandClaude Sonnet 5	15a0dbe3b1	fix(security): require auth on GET /api/orders and GET /api/admin/metrics ... A live compliance audit against 0pt.app confirmed both endpoints returned HTTP 200 with zero authentication -- the full order queue and financial pipeline aggregates (deposit/pipeline totals) were readable by anyone. Root cause: middleware.ts's matcher only covers /api/0pt/:path*, by design ("does NOT touch ... the unrelated payment/order APIs"), so nothing upstream of these two handlers ever verified the caller. Fix: both GET handlers now call the same fail-closed Zitadel verifier /api/0pt/* relies on (resolveAuthContext, src/auth/context.ts) directly, via a new thin wrapper (src/auth/require-route-auth.ts) that maps a rejected decision to the same 401/403 JSON shape middleware.ts already returns. The wrapper re-verifies the bearer token itself rather than trusting any x-0pt-verified-* header on the request, since middleware never runs in front of these routes to overwrite a client-supplied one. POST /api/orders (anonymous draft/quote creation before the deposit-payment step) is intentionally left as-is -- only the read paths that leaked data are gated. Tracked as part of docs/compliance/BACKLOG.md P0-3. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-11 04:06:13 +02:00
Claude (SIAX platform session)	1c1e2cb649	fix(auth): make the AUTH_ENFORCE default fail-closed on /api/0pt/* ... The auth seam read `process.env.AUTH_ENFORCE === 'true'`, so an absent env var meant NO authentication: any fresh container, local run, DR restore or redeploy that dropped the variable served /api/0pt/* anonymously, taking the tenant from the caller-supplied X-Tenant-Id header. 0pt.app is deployed and was answering anonymously (verified: an invalid Bearer token plus an arbitrary X-Tenant-Id returned 200), so this was a live hole, not latent. * authEnforced() now returns true unless AUTH_ENFORCE is explicitly "false" (trimmed, case-insensitive). Unset/empty/misspelled => ENFORCED. * assertAuthConfigSafe() throws AuthMisconfigurationError when NODE_ENV=production and enforcement resolved off, naming AUTH_ENFORCE. It runs at middleware module load (startup) and on the env-driven request path, so a production instance can neither boot nor serve a single unauthenticated request with auth switched off. * .env.example no longer teaches AUTH_ENFORCE=false; README's Deploy section states the fail-closed posture (no DEPLOY.md in this repo). * 7 new tests, incl. the regression case: with AUTH_ENFORCE unset and no `enforce` option, an unauthenticated request is rejected with 401. No test was inverted or deleted -- none asserted the open default; the three enforce:false cases now describe the explicit local-dev escape hatch. Tests: 121 passed / 1 skipped before, 128 passed / 1 skipped after. Typecheck and build clean; advisory lint unchanged (16 pre-existing problems). Scope: auth default only. The matcher still covers /api/0pt/* alone, and the handlers still take tenant_id from the query/body rather than the verified principal -- both described in the PR body, neither fixed here.	2026-07-27 13:28:24 +02:00
SIAX Foundation BotandClaude Opus 4.8	1d76d50f3d	feat(auth,foundation): Wave 2 flag-gated Zitadel JWT verify + validation + eslint + tests ... Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-25 14:32:34 +02:00
adminandClaude Sonnet 5	bda21e87eb	Live-verify Iceberg write path against ST0RE + close field/contract gaps ... Re-runs this session's task spec against 0pt's actual repo state, which turned out to already have a substantial, tested implementation from a prior pass (35b6ec6): dataset<->Iceberg-table mapping, source rate policy, and an ST0RE lakehouse client that was honestly marked "not live-verified". This pass found api.st0re.siax.io IS reachable from this environment and used it -- first via curl to safely discover the real contract, then through 0pt's own updated code -- to drive a real namespace -> table -> snapshot write against production ST0RE (task item 3's "write ONE first test dataset" proof). That surfaced and fixed three real bugs the mock-only pass could not have caught: - ST0RE requires X-Tenant-Id/X-Actor-Id headers on every lakehouse route; the previous HTTP client sent none. Fixed via a threaded St0reRequestContext. - IcebergTableInput.datasetId is a foreign key into ST0RE's OWN dataset table, not a free-form string -- sending 0pt's dataset_id 400s live with a Postgres FK violation. Fixed by no longer sending it; 0pt's dataset<->table linkage lives entirely in DatasetIcebergMapping. - ST0RE enforces a real unique constraint on lakehouse namespaces (tenant_id, project_id, name) with no upsert route. Fixed with a client-side lookup-or-create (listNamespaces + findOrRegisterNamespace). Also closed a field-level gap against the task's literal schema spec: 0pt.dataset_iceberg_mapping.schema.v1 now carries namespace_id (a real reference to ST0RE's LakehouseNamespace.namespaceId, not guessed) and mapped_at, both additive alongside the existing namespace/created_at fields (sql/005, JSON schema, types, service). tests/p0/iceberg-write-service.realinfra.test.ts replaces the prior it.todo with a real SEAM-pattern live test: attempts the real HTTP call, skips honestly if ST0RE is unreachable, never fabricates success. It ran successfully against production ST0RE this session (real namespace/table/ snapshot UUIDs logged in docs/ICEBERG_WRITE_BOUNDARY.md). Two ST0RE-side cross-tenant read bugs were found and flagged (not fixed here -- out of scope for this repo): GET .../namespaces ignores its own tenant header and query filters; GET .../tables/:id doesn't check the caller's tenant against the table's owner. npm test: 155 passed, 0 failed, 0 todo (up from 154 passed + 1 todo). npx tsc --noEmit: 1 pre-existing, unrelated error in quality-report-service.ts (present before this pass, part of the wider uncommitted P0 data-platform build already in this repo) -- not touched. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-13 00:18:10 +02:00
adminandClaude Sonnet 5	35b6ec63bf	Add 0pt Iceberg write-path ownership: dataset<->table mapping + source rate policy ... Real, tested implementation of docs/11-lakehouse-write-plan.md's items 2-8 (the "future implementation session" that doc named): 0pt is now the canonical writer of Iceberg tables through ST0RE's real, shipped /v1/st0re/lakehouse/* table registry (sax3l/st0re commit 9ea0abb), never persisting lakehouse table/catalog/snapshot state itself. - src/types + src/services for 0pt.dataset_iceberg_mapping.schema.v1 (dataset -> ST0RE tableId reference) and 0pt.source_rate_policy.schema.v1 (the sourceRatePolicy ownership repair, moved out of N0D per ADR-0030). - src/clients/st0re-lakehouse-client.ts: new client matching ST0RE's actual camelCase route contract, with the same mock/HTTP boundary convention as this repo's existing cl0ud/n0d/aud0/st0re clients. Honest caveat in its module doc comment: not verified against a live ST0RE instance in this pass (no reachable ST0RE_URL); every test exercises the deterministic mock. - src/services/iceberg-write-service.ts: registers tables/snapshots through ST0RE, requests N0D jobs only for heavy writes (never executes them in-process), emits all four AUD0 event names (dataset_iceberg_table_mapped / iceberg_write_requested / iceberg_snapshot_created / source_rate_policy_created). - sql/004, packages/schemas/*.json, app/api/0pt/lakehouse + sources/rate-policies routes, tests/p0/*.test.ts (154 passed + 1 honest it.todo for the live-ST0RE case, up from 142 baseline, 0 failures). - Reconciled a real collision: an earlier untracked draft schema/test pair (tests/p0/enterprise-data-os-expansion.test.ts) asserted a different, now-superseded source_rate_policy shape; updated that one test block only. - docs/ICEBERG_WRITE_BOUNDARY.md states the ownership boundary plainly and is explicit throughout about what's real-and-tested vs. contract-only. docs/11-lakehouse-write-plan.md gets a status-update section, not a rewrite. - Data quality rules (quality-check-service.ts, quality-report-service.ts) were not touched -- confirmed still 0pt-owned. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-12 19:46:47 +02:00
adminandClaude Sonnet 5	c89731115c	feat: add 0pt.query_source, 0pt.document_ocr, 0pt.write_external_record capabilities ... Three new, tenant-agnostic P0 capabilities: - 0pt.query_source: bounded, timeout/row-capped ad-hoc SQL query execution against registered database sources, via a Fabric/OneLake adapter (Azure AD credential-based mssql connection), with AUD0 audit events and an app/api/0pt/sources/[sourceId]/query endpoint. - 0pt.document_ocr (document capture + extraction): mailbox-based document intake via Microsoft Graph, OCR/field extraction via Azure Document Intelligence, with capture/extraction API endpoints and services that never fabricate extracted values. - 0pt.write_external_record (external writeback): a generic writeback-adapter contract plus a concrete Odoo bills adapter that matches partners via Fabric, backing an external-writeback-service for pushing records to downstream systems of record. Known interim gaps: - src/adapters/local-blob-store.ts is an interim ST0RE substitute pending the real ST0RE artifact-store integration. - mssql is used by the Fabric/OneLake adapter but deliberately not yet added to package.json (pending `pnpm add mssql` once the concurrent session's package.json/dependency changes settle). A separate, concurrently-active session has unrelated in-flight changes in this same working tree (data-platform catalog/lineage/feature-store, dataset quality, lakehouse/Iceberg docs, deploy scaffolding, package.json/ pnpm-lock.yaml updates, etc.) — none of that is touched or included here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-08 19:32:43 +02:00