27 Commits

Include renames

Author	SHA1	Message	Date
admin	5ffd41a461	fix(lint): odoo-bills-adapter AUD0-body — disable med skäl (CI-fynd)	2026-09-29 13:27:31 +02:00
admin	e24f8e8af4	fix(lint): typa AUD0-ingest-body i external-writeback-testet (CI-fynd)	2026-09-29 13:17:00 +02:00
admin	3279fa0533	fix(lint): eslint-undantag på rätt rader — json:any i graph-adapter + require-fixtures ... eslint-disable-next-line gällde nästa rad; json:any-linen behövde eget undantag. npm run lint: 0 errors (8→0). 1141/1141.	2026-09-29 12:50:06 +02:00
admin	1b6140a16a	fix(lint): rotfelsa ärvd lint-skuld — graph-adapter + p0-tester (blockerade #128) ... Parallellsessions-merge lämnade 12 eslint-errors på main (no-explicit-any i graph-mailbox-adapter + 2 p0-tester). Graph-JSON:n är genuint ostrukturerad — medveten, avgränsad eslint-undantag med skäl (DoD §13); p0-testerna typade med Record/cast. Lint 0 errors; tsc grön; 1141/1141.	2026-09-29 12:31:34 +02:00
Claude Code	dd4731aa3c	feat(decision): Jev ER auto-match shadow (0PT_DECISION_ER_SHADOW) ... Shadow-eval av D0 auto-match: D0-tröskeln förblir auktoritet, kvaliteten mäts fire-and-forget mot siax-decision (:4600, decisionId 0pt.er.auto_match). Gate 0PT_DECISION_ER_SHADOW=1 (OFF by default). Fail-closed: gate-off/fel/ABSTAINED påverkar aldrig det deterministiska svaret; decision-planen loggar inbyggt (AUD0). - packages/decision: erAutoMatchShadow + erAutoMatchShadowEnabled - src/er/comparePair: void erAutoMatchShadow(...).catch(() => null) - tests/p0/decision-er-shadow.test.ts: gate-off/on, POST-body, DECIDED/ABSTAINED, fetch-fel, fire-and-forget-wire (16 nya tester)	2026-09-23 16:53:55 +02:00
Claude Code	fc1a62bb81	feat: decision-plane wiring — ER-pair decision.match tie-break (@siax/0pt-decision) ... - packages/decision: erMatchDecision (0pt.er.pair_match mot siax-decision :4600) - comparePair async + borderline-band (±0.05 av review_threshold) → decision.match (D1 advisory) — deterministisk status GÄLLER om decision abstainar - OPT_DECISION_ER_ENABLED=1-gate (off tills kalibrering) - tests async-anpassade (await generateCandidates/rejects.toThrowError) - typecheck: 13 fel = pre-existing extraction-route (main-paritet), 0 nya fel - tester: 1108 passed | 184 skipped	2026-09-23 00:15:27 +02:00
admin	422380d0ed	test(BTI): DAT-04/05 dataset version acceptance tests ... - DAT-05-A1: monotonic versions, immutability - DAT-04-A2: estimated schema_version in manifest - 1054 tests green	2026-09-18 01:41:55 +02:00
adminandClaude Sonnet 5	d41f30535b	fix(test): replay full migration chain in source-registry realinfra test ... tests/p0/source-registry-postgres.realinfra.test.ts only applied sql/002_source_registry.sql before running assertions. sql/007_ingestion_plane.sql later added opt_sources.owner_id, and src/repositories/postgres-source-repository.ts's INSERT has named that column ever since -- but nothing caught the drift because SOURCE_REGISTRY_TEST_DATABASE_URL is never set in CI (.gitea/workflows/ci.yml has no job that sets it), so this suite has silently skipped in every CI run. Verified against a real disposable Postgres 16 container: before this fix, all 7 tests failed with `column "owner_id" of relation "opt_sources" does not exist`; after replaying every sql/*.sql file in order (same pattern already used by tests/analytics/analytics-domain-postgres.realinfra.test.ts), all 7 pass. Found while auditing 0pt's Analytics/BI layer (MetricFlow/Cube/ClickHouse/ Trino/DuckDB) and standing up real Postgres/ClickHouse containers to verify those claims for real; this bug is adjacent (source registry, not analytics) but was directly encountered and is a small, safe, test-only fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-15 02:40:52 +02:00
admin	17bd1b40a0	feat(0pt): record legitimate-interest stance — B2B contacts, company-linked only ... Recorded decision (Simon 2026-08-31): personal data may be stored under legitimate interest ONLY as business contacts bound to a company/org entity. Gate enforces it: legitimate_interest requires purpose=business_contact and entity_linked_only=true; recordStored() requires entity_id (orgnr/rgs/LEI, hashed) and refuses person-only rows. +1 test; suite 167 pass | 1 skip; typecheck clean.	2026-09-15 02:40:52 +02:00
admin	12080fc517	feat(0pt): PII-Gate GDPR-guardrail for personal-data sources (fail-closed) ... PiiGateService makes any future personal-data source (ratsit.se and other directory domains) technically GDPR-OK before the first byte is stored: - recorded legal basis (art. 6) with basis_ref evidence, no policy => fail-closed - data minimisation via field allowlist (art. 5(1)(c)) - storage limitation + enforceRetention purge (art. 5(1)(e)) - sha256-only identity keys + log redaction (art. 25/32) - accessStatement / erase subject-rights surfaces (art. 15/17) - robots/rate confirmation required for web+api sources; no stealth/bypass - audit trail via AUD0 at risk L3 (0pt.pii_*) - gate enabled by default in SourceQueryService 10 new tests; full suite 166 pass | 1 skip; typecheck (blocking gate) clean.	2026-09-15 02:40:52 +02:00
adminandClaude Sonnet 5	4caf122129	fix(security): add SSRF guard to Fabric/OneLake adapter's sqlEndpoint ... connection_config.sqlEndpoint on POST /api/0pt/sources/{sourceId}/query was passed straight from the request body to the mssql TDS driver as the outbound connection target, with only its http(s):// scheme stripped — no hostname allowlist or private/loopback/link-local/cloud-metadata guard. This route already requires a Zitadel bearer token + tenant match (P014 tenant guard), so this was not anonymously exploitable, but it is the exact SSRF shape found and exploited-in-principle elsewhere in this estate tonight (press0), so it's worth closing as defense-in-depth. Adds src/lib/security/ssrf-guard.ts: resolves the endpoint's hostname and rejects any address in a private/loopback/link-local/cloud-metadata/ unspecified range (RFC1918, 127.0.0.0/8, 169.254.0.0/16 incl. the 169.254.169.254 metadata address, ::1, fc00::/7, fe80::/10, IPv4-mapped IPv6), failing closed on DNS resolution errors. The range-check logic is ported 1:1 from press0's src/lib/security/ssrf-guard.ts built the same night for the same bug class against a fetch()-based URL there; this module's surface differs because sqlEndpoint is a bare host[:port], not a fetchable URL. The DNS lookup is injectable (mirrors this adapter's existing fetchImpl pattern) so tests are hermetic. FabricOneLakeAdapter.getPool() now calls the guard before loadDriver()/ getToken()/opening the TDS pool, so a blocked target never reaches a real outbound connection attempt or spends an AAD token exchange. Tests: new tests/p0/fabric-onelake-ssrf-guard.test.ts unit-tests the guard module directly (range boundaries, IPv4-mapped-IPv6 bypass attempt, fail-closed on DNS failure/empty answer). New adapter-level tests in tests/p0/fabric-onelake-adapter.test.ts prove (a) a private-IP-resolving sqlEndpoint is rejected before fetch/connect is ever called, (b) a literal loopback sqlEndpoint is rejected the same way, and (c) a normal external Fabric hostname still passes the guard and completes the query — the guard doesn't false-block legitimate targets. Existing adapter tests are updated to inject a deterministic DNS resolver so they stay hermetic instead of depending on real DNS. Full suite: 997 passed, 0 failed, 184 skipped (self-skipping realinfra tests with no live backend this run). typecheck and lint clean on all touched files. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-11 02:18:23 +02:00
admin	2d34cb1629	feat(P-001, P-002): 0pt→N0D quality delegation + internal OpenAPI	2026-09-08 22:38:47 +02:00
admin	3edc56dcdb	fix: slutgranskning av dataset/search/ER-ytorna + STATUS-sanning ... Slutgate inför merge (per fortsättningsplan): Kontrakt: CreateDatasetVersionInput.artifact_refs → readonly string[] (anropare får skicka 'as const'; servicen muterar aldrig); postgres.js tx.array() får muterbar kopia vid INSERT. Testkorrigeringar (stale mot starkare implementation — aldrig svagare): - entity-resolution-http: request(body, undefined) triggade default-param och skickade GILTIGT token → 200. Tomt token = 'utan token' → 401 fail-closed. - entity-resolution: /two records/ → /between 2 and 5000/ (bounds-aware validering) - search-service: omformat purpose-filter-test + NY negativ kontroll — rebuild avvisar dokument utan purpose_tags (fail closed, generation discarded) - dataset-postgres.realinfra: sätter DATABASE_URL i setup så den durabla AUD0-outboxen aktiveras (testet ljög annars om durabilitet) — 5/5 mot live Postgres STATUS.md: AUD0-klient + lineage-rader (outbox nu real-infra testad), gap-listan (persistence/routes/outbox stängda, exakt vad som återstår), P065 STÄNGD med closure-record (thin adapter, v3-semantik, hard-negative transitivitet, fail-closed 401 — 4/4 planpunkter verifierade). Verifierat: typecheck EXIT 0 · 945/945 unit · 5/5 real-Postgres · next build EXIT 0	2026-09-07 19:31:22 +02:00
admin	d45eb74772	test(er): cover country-qualified stable identifiers	2026-09-07 18:09:46 +02:00
admin	7e2b5304b7	test(er): pin review-only fuzzy company boundary	2026-09-07 18:08:24 +02:00
admin	851e43d8db	test: prove dataset registry and versions against real Postgres	2026-09-07 18:05:15 +02:00
admin	6a687212a1	test: pin lineage queued-vs-emitted semantics	2026-09-07 17:59:20 +02:00
admin	2a79ae6e60	test(evidence): verify AUD0 outbox durability against Postgres	2026-09-07 17:13:57 +02:00
admin	76a3cae776	test(evidence): cover AUD0 outbox delivery and failure semantics	2026-09-07 17:13:35 +02:00
Simon AxelssonandClaude Opus 5	9267576b7c	feat(data-os): P060-P068 durable ingestion plane — source classification gate, connector contract, common envelope, schema evolution, quality enforcement, queryable lineage, idempotent backfill ... BEFORE_STATE (verified, with evidence): * P060 PARTIAL/BROKEN. The registry persisted rights/provenance/health/ freshness to Postgres, but carried no authorization, no schedule, no checkpoint, no schema and no owner distinct from created_by — and the activation DoD was inverted: postgres-source-repository.ts wrote `${input.status ?? "active"}`, so a source with no rights, no provenance and no authorization was ACTIVE the instant it was registered. No activation operation existed at all. * P061 MISSING. services/api/sources/connectors.py had per-source fetch_*() functions only — no lifecycle, no cursor, no resume, no health, no conformance suite. * P062 MISSING, already documented as such in services/api/tests/test_known_gaps.py:10-19 (no CDC pipeline anywhere; workers/pipeline.py has no persistence at all). * P063 MISSING. schema_version existed as a free string; no comparison, no policy, no versioned source schema. * P064 PARTIAL. Rules/runs/issues/scores existed, but a failing rule only opened an issue and the data flowed on. No quarantine anywhere. * P065 DUPLICATE (confirmed). Two ER engines: src/er/entity-resolution- service.ts (canonical) and services/api/workers/resolve.py (simplified, still called by pipeline.py:110). * P067 PARTIAL. Lineage was emitted to AUD0 but nothing here could ANSWER a lineage question — no store, no query, no route. * P068 MISSING, documented in test_known_gaps.py:44-52 ("no delivery-store persistence layer exists to dedupe a backfill against"). BUILT (only what was PARTIAL/BROKEN/MISSING): * P060 sql/007_ingestion_plane.sql adds authorization/schedule/schemas/ checkpoints/ledger/quarantine/lineage tables, owner_id, and flips the opt_sources status DEFAULT to 'pending' so even a raw psql INSERT cannot produce an unclassified-but-active source. SourceRegistryService gains classifySource/activateSource/deactivateSource + authorization/schedule accessors; registration NEVER activates (asking for status:"active" goes through the gate, not around it); a service identity cannot authorize a source. Routes: activate (GET classification / POST activate), authorization, schedule. * P061 src/connectors/connector.ts — all nine operations on one interface. src/connectors/conformance.ts — a runnable suite (15 checks, no "skip" verdict). HttpJsonConnector (real HTTP, opaque cursor) and JsonlFileConnector (real file, line offsets, CDC deletes) both pass it. The suite caught a real defect while being written: an API that drops next_cursor at the end left the connector unable to resume, which is why HttpJsonConnectorConfig has cursor_field. * P062 src/types/ingestion.ts + src/ingestion/{envelope,ingestion-runner}.ts. One envelope for batch/stream/cdc; durable per-(source,partition) checkpoints; resume proven by killing a run inside the sink. * P063 src/schema/{record-schema,compatibility,schema-registry}.ts. Two enforcement points, both upstream of any consumer write: a discovery gate before the first read, and per-record conformance before the page reaches the sink. A producer cannot widen its own policy retroactively. * P064 src/quality/quality-policy.ts composes the EXISTING ExpectationEngine (not a second engine) and adds the missing consequence: block / quarantine / warn, with quarantine durable and recoverable. No policy is reported as "none_declared", never as a pass; an unexecutable expectation counts as a violation, never a silent pass. * P067 src/lineage/{lineage-edge,lineage-store}.ts + GET /api/0pt/lineage. AUD0 stays the system of record; this is 0pt's own queryable projection. * P068 opt_ingestion_ledger + IngestionRunner.backfill: bounded, idempotent on repeat, leaves the live checkpoint alone, still re-emits real changes. A BUG THIS PASS WROTE AND CAUGHT: the first IngestionRepository draft combined the idempotency check and its recording into one atomic call BEFORE the sink. The crash/resume test then delivered 7 of 10 records — the page in flight when the sink died was already marked delivered, so the resumed run skipped it as a duplicate. Permanent, silent data loss. Fixed by splitting peekDeliveries/commitDeliveries and committing both the ledger and the checkpoint only after the sink resolves. NOT DONE, stated rather than deferred quietly: * P065 is NOT met. resolve.py is still a second simplified production engine. Closing it means rewriting it as an adapter over the canonical service, which cannot be verified from this repo today: services/api is not built by this repo's Dockerfile, api.0pt.app answers 404, and NO CI workflow runs its pytest suite. The duplication is now stated at both sites in code and the removal plan is in STATUS.md. The new ingestion plane performs no ER, so it adds no third engine. * P066 (Iceberg) untouched — ST0RE is owned by another workflow. Contract read only. * Delivery is AT-LEAST-ONCE, not exactly-once (no per-partition run lock). * No CDC *source* exists; the contract is met, a CDC connector is not built. * 0pt does not schedule itself — the schedule row is a declaration ACT0 would read. VERIFIED: pnpm typecheck clean; pnpm test 365 passed / 25 skipped (was 311/18); pnpm build clean with all four new routes in the route table; lint unchanged at 12 pre-existing errors. Real Postgres 16 (tests/p0/ingestion-plane-postgres.realinfra.test.ts, 7 passed) plus a real `docker restart` of that container with an identical read-back through the production code path — that is P060's "a restart preserves all sources", measured. DEPLOYMENT REALITY (curl, 2026-09-04): 0pt.app/api/health returns 200 from Vercel with db:not_configured; 0pt.app/api/0pt/sources returns 404; api.0pt.app/health returns 404; cd.yml still guards on a placeholder Coolify app id. None of the capability layer runs in production. Nothing here is LIVE_VERIFIED and STATUS.md says so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-04 17:03:14 +02:00
Simon AxelssonandClaude Sonnet 5	9740e20818	feat(data-os): TS/Python quality bridge + provenance fields + fault-injection tests ... Continues P010 (branch renamed from feat/p010-source-registry-persistence): closes the three remaining BEFORE_STATE.md gaps (§2 TS/Python split-brain, §3 missing provenance fields, §4 untested failure modes), all additive to the production pipeline — the Python QA filter's delivery decision is unchanged by any of this. TS/Python bridge (§2): - New route surface app/api/0pt/quality/* (rules/runs/issues CRUD + POST /api/0pt/quality/bridge-run) over the shared QualityService singleton (src/services/quality.ts), fixing the same "no shared instance" defect the Source Registry had. - src/quality/quality-bridge.ts auto-provisions a deterministic, tenant-scoped baseline rule set on first use so the Python engine never pre-registers rules through the CRUD surface. - services/api/workers/quality_bridge.py: the Python-side HTTP client. Feature-flagged off by default (OPT_QUALITY_BRIDGE_ENABLED), fails open on any error (never raises into run_job, never changes `delivered`). Advisory only — real per-row gating against the TS engine isn't implementable yet (ExpectationEngine returns dataset-level pass/fail counts + samples, not a per-row boolean array); documented, not faked. - Live-verified end-to-end: `next dev` (AUTH_ENFORCE=false) + the real Python pipeline hitting the real brreg.no API, round-tripping through the real bridge route, receiving a real QualityService score. Provenance (§3): sources/connectors.py's Provenance dataclass gains retrieved_at, transform, connector_version, producer, quality, entity_resolution, artifact_ref (all previously absent from the real dataflow entirely) via a make_provenance() factory used by every connector. resolve.py stamps entity_resolution on merged records' provenance (never on unmerged ones). pipeline.py stamps artifact_ref at delivery and quality once the bridge succeeds. Failure tests (§4), real fault injection against real code (services/api, new pytest + respx suite, not vitest mocks): - dedup-determinism + duplicate-input + out-of-order: resolve.py under shuffled/interleaved/exhaustive-permutation input, plus an explicit, verified test of the one case that IS legitimately order-sensitive (non-transitive similarity in a greedy clusterer) instead of hiding it. - schema-evolution: respx-mocked upstream responses with added/removed/ alien-shaped fields through the real brreg connector + pipeline. - restart: connector exceptions mid-job (including one from respx itself) never crash run_job. - backfill: the real property that holds given no persistence layer exists (replay is compute-idempotent) — NOT a fabricated dedup-store test. - CDC-interruption and true backfill/delivery-store idempotency: explicitly documented as not implementable against this codebase (no CDC pipeline, no delivery store exist anywhere) via skip markers with reasons, in tests/test_known_gaps.py, rather than silently dropped or faked. Testing: 256/256 vitest (249 pre-existing + 7 new bridge route/service tests) and 28/28 pytest (2 documented skips) green locally. tsc -p tsconfig.check.json has zero new errors (14 pre-existing errors, all in the untracked/unaudited src/fleet, src/graph, src/metrics, src/projections paths this pass does not touch, per BEFORE_STATE.md). No PR opened, per instructions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-01 21:02:14 +02:00
Simon AxelssonandClaude Sonnet 5	07b13bd833	feat(data-os): Source Registry Postgres persistence + full HTTP route surface ... Replaces the in-memory (per-instance, not even per-process-shared) Maps in SourceService/SourceRegistryService with real Postgres persistence, and adds the register/query/describe/health/rights/provenance/freshness HTTP routes that never existed (grep for SourceRegistryService in app/ previously returned nothing). - sql/002_source_registry.sql: opt_sources, opt_source_rights, opt_source_provenance, opt_source_health_checks, opt_source_freshness. Numbered 002 (not 006) because 004's opt_source_rate_policies already forward-references opt_sources, which no sql/ file defined until now — fixes 004's pre-existing broken migration ordering (opt_datasets remains a separate, unrelated, still-open gap in 004, flagged not fixed). - src/repositories/*: SourceRepository / SourceRegistryRepository seams, each with an in-memory implementation (business logic extracted verbatim) and a Postgres implementation, chosen via resolveStoreMode — production still fails hard with no DATABASE_URL (Master-DoD §3), but a configured production now actually works instead of always being blocked. - SourceService / SourceRegistryService: storage moved behind the repository seam; getHealth/getFreshness are now async and tenant-scoped at the accessor itself (needed once they're reachable directly over HTTP, not only after an earlier caller's tenant check). - src/services/source-registry.ts: shared singleton instances for the sources/* route group, fixing the "each route module owned its own private registry" defect (source-rate-policies and .../query previously couldn't see a source registered via POST /api/0pt/sources). - New routes: POST/GET /api/0pt/sources, GET /api/0pt/sources/[id], GET+PUT .../rights, GET+PUT .../provenance, GET+POST .../health, GET .../freshness. - Business logic unchanged: fail-closed purpose gate, cross-tenant NOT_FOUND (no existence oracle, §50), bounded 100-row health history. Tested against a real Postgres container (not mocked) — tests/p0/source-registry-postgres.realinfra.test.ts, env-gated on SOURCE_REGISTRY_TEST_DATABASE_URL, skips honestly when unset. All 249 pre-existing P0 tests + 7 new realinfra tests pass; also smoke-tested the full HTTP route surface end-to-end against next dev + live Postgres. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-01 20:42:41 +02:00
Simon AxelssonandClaude Fable 5	b46807da75	feat(data-os): P0 capability layer — source registry, search, quality, dataset versions, metadata, analytics, features, ER ... - Source Registry (§10): rights/license/purpose metadata, provenance, freshness SLA, health checks, describe/query; purpose gate FAILS CLOSED (no rights record => deny); cross-tenant probes => NOT_FOUND. - Search (§10/§42): OpenSearch REST adapter (no SDK) + SearchService with MANDATORY tenant + purpose/rights filtering, versioned index definitions, bulk, generation-per-rebuild (opt-{name}--g{n}) with atomic alias cutover, rollback, failed-rebuild generation discard. OpenSearch never canonical. - Quality (§10): versioned rule registry (GE-JSON interchange), local engine for the GE core subset (never silently passes unsupported types), runs/results/issues/score/trend, whole-run delegation to the GE backend via N0D (0pt.quality.ge_run) - never fabricated results. - Dataset (§10/§5/§53): immutable numbered versions holding canonical st0re:artifact refs only, idempotent createVersion, export registers the ST0RE artifact + AUD0 evidence + OpenLineage. - Metadata (§10): OpenMetadata PUT /api/v1/tables projection adapter + projectAll rebuild path; projection never canonical. - Analytics (§10): ClickHouse HTTP adapter, closed metric-query DSL with FORCED tenant predicate + readonly/max_execution_time/max_result_rows, TTL retention. Zero connection to b0ts/quarantined pipelines. - Features (§10): versioned feature-view registry, real Feast get-online-features adapter, materialize/get-historical as N0D delegation. - ER (§10): generic candidate API with deterministic blocking + transparent scoring + per-field explanations, product hard-rule hook (force both directions), review-band handoff, Splink batch via N0D contract. 93 new unit/contract tests; suite 248 passed / 1 skipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-31 16:49:32 +02:00
Simon AxelssonandClaude Fable 5	b7038d7675	feat(platform): canonical SIAX context/errors + real-contract clients, no prod mock fallback ... - src/platform/: SiaxRequestContext + propagation headers + 12 canonical error codes (documented mirror of cl0ud packages/contract), SiaxError, resource refs (st0re:artifact:...), and the §3/§10 dependency-mode guard: production runtime + unconfigured dependency/store => hard DEPENDENCY_UNAVAILABLE at construction, never silent mock/in-memory. - src/clients/: all four platform clients rewritten against the REAL wire contracts verified on sibling main branches this session: * cl0ud: POST /api/policy/evaluate ({data} envelope, allow+requiresApproval) - the old /v1/cl0ud/policies/evaluate path never existed on CL0UD * st0re: camelCase ArtifactInput on /v1/st0re/artifacts/register incl. X-Tenant-Id/X-Actor-Id + X-Idempotency-Key - the old snake_case body would have been a 400 against the live registry * n0d: required_role + resource_requirements + body idempotency_key + CloudEvents callback_url per n0d createJobSchema; adds getJob/getJobResult * aud0: /v1/aud0/openlineage is AUD0's only lineage ingest - /v1/aud0/lineage/events never existed; legacy shape converts to a spec-shaped OpenLineage RunEvent (deterministic runId dedup) - src/lineage/: OpenLineage subset types + LineageEmitter (never-throw, §16 evidence semantics) - shared SiaxHttp transport: timeouts, typed SiaxError mapping, context propagation, no raw upstream bodies in errors Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-31 16:35:01 +02:00
adminandClaude Sonnet 5	bda21e87eb	Live-verify Iceberg write path against ST0RE + close field/contract gaps ... Re-runs this session's task spec against 0pt's actual repo state, which turned out to already have a substantial, tested implementation from a prior pass (35b6ec6): dataset<->Iceberg-table mapping, source rate policy, and an ST0RE lakehouse client that was honestly marked "not live-verified". This pass found api.st0re.siax.io IS reachable from this environment and used it -- first via curl to safely discover the real contract, then through 0pt's own updated code -- to drive a real namespace -> table -> snapshot write against production ST0RE (task item 3's "write ONE first test dataset" proof). That surfaced and fixed three real bugs the mock-only pass could not have caught: - ST0RE requires X-Tenant-Id/X-Actor-Id headers on every lakehouse route; the previous HTTP client sent none. Fixed via a threaded St0reRequestContext. - IcebergTableInput.datasetId is a foreign key into ST0RE's OWN dataset table, not a free-form string -- sending 0pt's dataset_id 400s live with a Postgres FK violation. Fixed by no longer sending it; 0pt's dataset<->table linkage lives entirely in DatasetIcebergMapping. - ST0RE enforces a real unique constraint on lakehouse namespaces (tenant_id, project_id, name) with no upsert route. Fixed with a client-side lookup-or-create (listNamespaces + findOrRegisterNamespace). Also closed a field-level gap against the task's literal schema spec: 0pt.dataset_iceberg_mapping.schema.v1 now carries namespace_id (a real reference to ST0RE's LakehouseNamespace.namespaceId, not guessed) and mapped_at, both additive alongside the existing namespace/created_at fields (sql/005, JSON schema, types, service). tests/p0/iceberg-write-service.realinfra.test.ts replaces the prior it.todo with a real SEAM-pattern live test: attempts the real HTTP call, skips honestly if ST0RE is unreachable, never fabricates success. It ran successfully against production ST0RE this session (real namespace/table/ snapshot UUIDs logged in docs/ICEBERG_WRITE_BOUNDARY.md). Two ST0RE-side cross-tenant read bugs were found and flagged (not fixed here -- out of scope for this repo): GET .../namespaces ignores its own tenant header and query filters; GET .../tables/:id doesn't check the caller's tenant against the table's owner. npm test: 155 passed, 0 failed, 0 todo (up from 154 passed + 1 todo). npx tsc --noEmit: 1 pre-existing, unrelated error in quality-report-service.ts (present before this pass, part of the wider uncommitted P0 data-platform build already in this repo) -- not touched. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-13 00:18:10 +02:00
adminandClaude Sonnet 5	35b6ec63bf	Add 0pt Iceberg write-path ownership: dataset<->table mapping + source rate policy ... Real, tested implementation of docs/11-lakehouse-write-plan.md's items 2-8 (the "future implementation session" that doc named): 0pt is now the canonical writer of Iceberg tables through ST0RE's real, shipped /v1/st0re/lakehouse/* table registry (sax3l/st0re commit 9ea0abb), never persisting lakehouse table/catalog/snapshot state itself. - src/types + src/services for 0pt.dataset_iceberg_mapping.schema.v1 (dataset -> ST0RE tableId reference) and 0pt.source_rate_policy.schema.v1 (the sourceRatePolicy ownership repair, moved out of N0D per ADR-0030). - src/clients/st0re-lakehouse-client.ts: new client matching ST0RE's actual camelCase route contract, with the same mock/HTTP boundary convention as this repo's existing cl0ud/n0d/aud0/st0re clients. Honest caveat in its module doc comment: not verified against a live ST0RE instance in this pass (no reachable ST0RE_URL); every test exercises the deterministic mock. - src/services/iceberg-write-service.ts: registers tables/snapshots through ST0RE, requests N0D jobs only for heavy writes (never executes them in-process), emits all four AUD0 event names (dataset_iceberg_table_mapped / iceberg_write_requested / iceberg_snapshot_created / source_rate_policy_created). - sql/004, packages/schemas/*.json, app/api/0pt/lakehouse + sources/rate-policies routes, tests/p0/*.test.ts (154 passed + 1 honest it.todo for the live-ST0RE case, up from 142 baseline, 0 failures). - Reconciled a real collision: an earlier untracked draft schema/test pair (tests/p0/enterprise-data-os-expansion.test.ts) asserted a different, now-superseded source_rate_policy shape; updated that one test block only. - docs/ICEBERG_WRITE_BOUNDARY.md states the ownership boundary plainly and is explicit throughout about what's real-and-tested vs. contract-only. docs/11-lakehouse-write-plan.md gets a status-update section, not a rewrite. - Data quality rules (quality-check-service.ts, quality-report-service.ts) were not touched -- confirmed still 0pt-owned. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-12 19:46:47 +02:00
adminandClaude Sonnet 5	c89731115c	feat: add 0pt.query_source, 0pt.document_ocr, 0pt.write_external_record capabilities ... Three new, tenant-agnostic P0 capabilities: - 0pt.query_source: bounded, timeout/row-capped ad-hoc SQL query execution against registered database sources, via a Fabric/OneLake adapter (Azure AD credential-based mssql connection), with AUD0 audit events and an app/api/0pt/sources/[sourceId]/query endpoint. - 0pt.document_ocr (document capture + extraction): mailbox-based document intake via Microsoft Graph, OCR/field extraction via Azure Document Intelligence, with capture/extraction API endpoints and services that never fabricate extracted values. - 0pt.write_external_record (external writeback): a generic writeback-adapter contract plus a concrete Odoo bills adapter that matches partners via Fabric, backing an external-writeback-service for pushing records to downstream systems of record. Known interim gaps: - src/adapters/local-blob-store.ts is an interim ST0RE substitute pending the real ST0RE artifact-store integration. - mssql is used by the Fabric/OneLake adapter but deliberately not yet added to package.json (pending `pnpm add mssql` once the concurrent session's package.json/dependency changes settle). A separate, concurrently-active session has unrelated in-flight changes in this same working tree (data-platform catalog/lineage/feature-store, dataset quality, lakehouse/Iceberg docs, deploy scaffolding, package.json/ pnpm-lock.yaml updates, etc.) — none of that is touched or included here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-08 19:32:43 +02:00