admin/0pt - 0pt - Gitea: Git with a cup of tea

50 Commits 39 Branches 0 Tags

3 Commits

Include renames

Author SHA1 Message Date

Author	SHA1	Message	Date
adminandClaude Sonnet 5	bda21e87eb	Live-verify Iceberg write path against ST0RE + close field/contract gaps Re-runs this session's task spec against 0pt's actual repo state, which turned out to already have a substantial, tested implementation from a prior pass (`35b6ec6`): dataset<->Iceberg-table mapping, source rate policy, and an ST0RE lakehouse client that was honestly marked "not live-verified". This pass found api.st0re.siax.io IS reachable from this environment and used it -- first via curl to safely discover the real contract, then through 0pt's own updated code -- to drive a real namespace -> table -> snapshot write against production ST0RE (task item 3's "write ONE first test dataset" proof). That surfaced and fixed three real bugs the mock-only pass could not have caught: - ST0RE requires X-Tenant-Id/X-Actor-Id headers on every lakehouse route; the previous HTTP client sent none. Fixed via a threaded St0reRequestContext. - IcebergTableInput.datasetId is a foreign key into ST0RE's OWN dataset table, not a free-form string -- sending 0pt's dataset_id 400s live with a Postgres FK violation. Fixed by no longer sending it; 0pt's dataset<->table linkage lives entirely in DatasetIcebergMapping. - ST0RE enforces a real unique constraint on lakehouse namespaces (tenant_id, project_id, name) with no upsert route. Fixed with a client-side lookup-or-create (listNamespaces + findOrRegisterNamespace). Also closed a field-level gap against the task's literal schema spec: 0pt.dataset_iceberg_mapping.schema.v1 now carries namespace_id (a real reference to ST0RE's LakehouseNamespace.namespaceId, not guessed) and mapped_at, both additive alongside the existing namespace/created_at fields (sql/005, JSON schema, types, service). tests/p0/iceberg-write-service.realinfra.test.ts replaces the prior it.todo with a real SEAM-pattern live test: attempts the real HTTP call, skips honestly if ST0RE is unreachable, never fabricates success. It ran successfully against production ST0RE this session (real namespace/table/ snapshot UUIDs logged in docs/ICEBERG_WRITE_BOUNDARY.md). Two ST0RE-side cross-tenant read bugs were found and flagged (not fixed here -- out of scope for this repo): GET .../namespaces ignores its own tenant header and query filters; GET .../tables/:id doesn't check the caller's tenant against the table's owner. npm test: 155 passed, 0 failed, 0 todo (up from 154 passed + 1 todo). npx tsc --noEmit: 1 pre-existing, unrelated error in quality-report-service.ts (present before this pass, part of the wider uncommitted P0 data-platform build already in this repo) -- not touched. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-13 00:18:10 +02:00
adminandClaude Sonnet 5	35b6ec63bf	Add 0pt Iceberg write-path ownership: dataset<->table mapping + source rate policy Real, tested implementation of docs/11-lakehouse-write-plan.md's items 2-8 (the "future implementation session" that doc named): 0pt is now the canonical writer of Iceberg tables through ST0RE's real, shipped /v1/st0re/lakehouse/* table registry (sax3l/st0re commit 9ea0abb), never persisting lakehouse table/catalog/snapshot state itself. - src/types + src/services for 0pt.dataset_iceberg_mapping.schema.v1 (dataset -> ST0RE tableId reference) and 0pt.source_rate_policy.schema.v1 (the sourceRatePolicy ownership repair, moved out of N0D per ADR-0030). - src/clients/st0re-lakehouse-client.ts: new client matching ST0RE's actual camelCase route contract, with the same mock/HTTP boundary convention as this repo's existing cl0ud/n0d/aud0/st0re clients. Honest caveat in its module doc comment: not verified against a live ST0RE instance in this pass (no reachable ST0RE_URL); every test exercises the deterministic mock. - src/services/iceberg-write-service.ts: registers tables/snapshots through ST0RE, requests N0D jobs only for heavy writes (never executes them in-process), emits all four AUD0 event names (dataset_iceberg_table_mapped / iceberg_write_requested / iceberg_snapshot_created / source_rate_policy_created). - sql/004, packages/schemas/.json, app/api/0pt/lakehouse + sources/rate-policies routes, tests/p0/.test.ts (154 passed + 1 honest it.todo for the live-ST0RE case, up from 142 baseline, 0 failures). - Reconciled a real collision: an earlier untracked draft schema/test pair (tests/p0/enterprise-data-os-expansion.test.ts) asserted a different, now-superseded source_rate_policy shape; updated that one test block only. - docs/ICEBERG_WRITE_BOUNDARY.md states the ownership boundary plainly and is explicit throughout about what's real-and-tested vs. contract-only. docs/11-lakehouse-write-plan.md gets a status-update section, not a rewrite. - Data quality rules (quality-check-service.ts, quality-report-service.ts) were not touched -- confirmed still 0pt-owned. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-12 19:46:47 +02:00
adminandClaude Sonnet 5	c89731115c	feat: add 0pt.query_source, 0pt.document_ocr, 0pt.write_external_record capabilities Three new, tenant-agnostic P0 capabilities: - 0pt.query_source: bounded, timeout/row-capped ad-hoc SQL query execution against registered database sources, via a Fabric/OneLake adapter (Azure AD credential-based mssql connection), with AUD0 audit events and an app/api/0pt/sources/[sourceId]/query endpoint. - 0pt.document_ocr (document capture + extraction): mailbox-based document intake via Microsoft Graph, OCR/field extraction via Azure Document Intelligence, with capture/extraction API endpoints and services that never fabricate extracted values. - 0pt.write_external_record (external writeback): a generic writeback-adapter contract plus a concrete Odoo bills adapter that matches partners via Fabric, backing an external-writeback-service for pushing records to downstream systems of record. Known interim gaps: - src/adapters/local-blob-store.ts is an interim ST0RE substitute pending the real ST0RE artifact-store integration. - mssql is used by the Fabric/OneLake adapter but deliberately not yet added to package.json (pending `pnpm add mssql` once the concurrent session's package.json/dependency changes settle). A separate, concurrently-active session has unrelated in-flight changes in this same working tree (data-platform catalog/lineage/feature-store, dataset quality, lakehouse/Iceberg docs, deploy scaffolding, package.json/ pnpm-lock.yaml updates, etc.) — none of that is touched or included here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-08 19:32:43 +02:00

adminandClaude Sonnet 5

bda21e87eb

Live-verify Iceberg write path against ST0RE + close field/contract gaps

Re-runs this session's task spec against 0pt's actual repo state, which
turned out to already have a substantial, tested implementation from a
prior pass (35b6ec6): dataset<->Iceberg-table mapping, source rate policy,
and an ST0RE lakehouse client that was honestly marked "not live-verified".

This pass found api.st0re.siax.io IS reachable from this environment and
used it -- first via curl to safely discover the real contract, then
through 0pt's own updated code -- to drive a real namespace -> table ->
snapshot write against production ST0RE (task item 3's "write ONE first
test dataset" proof). That surfaced and fixed three real bugs the
mock-only pass could not have caught:

- ST0RE requires X-Tenant-Id/X-Actor-Id headers on every lakehouse route;
  the previous HTTP client sent none. Fixed via a threaded
  St0reRequestContext.
- IcebergTableInput.datasetId is a foreign key into ST0RE's OWN dataset
  table, not a free-form string -- sending 0pt's dataset_id 400s live with
  a Postgres FK violation. Fixed by no longer sending it; 0pt's
  dataset<->table linkage lives entirely in DatasetIcebergMapping.
- ST0RE enforces a real unique constraint on lakehouse namespaces
  (tenant_id, project_id, name) with no upsert route. Fixed with a
  client-side lookup-or-create (listNamespaces + findOrRegisterNamespace).

Also closed a field-level gap against the task's literal schema spec:
0pt.dataset_iceberg_mapping.schema.v1 now carries namespace_id (a real
reference to ST0RE's LakehouseNamespace.namespaceId, not guessed) and
mapped_at, both additive alongside the existing namespace/created_at
fields (sql/005, JSON schema, types, service).

tests/p0/iceberg-write-service.realinfra.test.ts replaces the prior
it.todo with a real SEAM-pattern live test: attempts the real HTTP call,
skips honestly if ST0RE is unreachable, never fabricates success. It ran
successfully against production ST0RE this session (real namespace/table/
snapshot UUIDs logged in docs/ICEBERG_WRITE_BOUNDARY.md).

Two ST0RE-side cross-tenant read bugs were found and flagged (not fixed
here -- out of scope for this repo): GET .../namespaces ignores its own
tenant header and query filters; GET .../tables/:id doesn't check the
caller's tenant against the table's owner.

npm test: 155 passed, 0 failed, 0 todo (up from 154 passed + 1 todo).
npx tsc --noEmit: 1 pre-existing, unrelated error in
quality-report-service.ts (present before this pass, part of the wider
uncommitted P0 data-platform build already in this repo) -- not touched.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

2026-07-13 00:18:10 +02:00

adminandClaude Sonnet 5

35b6ec63bf

Add 0pt Iceberg write-path ownership: dataset<->table mapping + source rate policy

Real, tested implementation of docs/11-lakehouse-write-plan.md's items 2-8
(the "future implementation session" that doc named): 0pt is now the
canonical writer of Iceberg tables through ST0RE's real, shipped
/v1/st0re/lakehouse/* table registry (sax3l/st0re commit 9ea0abb), never
persisting lakehouse table/catalog/snapshot state itself.

- src/types + src/services for 0pt.dataset_iceberg_mapping.schema.v1
  (dataset -> ST0RE tableId reference) and 0pt.source_rate_policy.schema.v1
  (the sourceRatePolicy ownership repair, moved out of N0D per ADR-0030).
- src/clients/st0re-lakehouse-client.ts: new client matching ST0RE's actual
  camelCase route contract, with the same mock/HTTP boundary convention as
  this repo's existing cl0ud/n0d/aud0/st0re clients. Honest caveat in its
  module doc comment: not verified against a live ST0RE instance in this
  pass (no reachable ST0RE_URL); every test exercises the deterministic mock.
- src/services/iceberg-write-service.ts: registers tables/snapshots through
  ST0RE, requests N0D jobs only for heavy writes (never executes them
  in-process), emits all four AUD0 event names
  (dataset_iceberg_table_mapped / iceberg_write_requested /
  iceberg_snapshot_created / source_rate_policy_created).
- sql/004, packages/schemas/*.json, app/api/0pt/lakehouse + sources/rate-policies
  routes, tests/p0/*.test.ts (154 passed + 1 honest it.todo for the live-ST0RE
  case, up from 142 baseline, 0 failures).
- Reconciled a real collision: an earlier untracked draft schema/test pair
  (tests/p0/enterprise-data-os-expansion.test.ts) asserted a different,
  now-superseded source_rate_policy shape; updated that one test block only.
- docs/ICEBERG_WRITE_BOUNDARY.md states the ownership boundary plainly and
  is explicit throughout about what's real-and-tested vs. contract-only.
  docs/11-lakehouse-write-plan.md gets a status-update section, not a rewrite.
- Data quality rules (quality-check-service.ts, quality-report-service.ts)
  were not touched -- confirmed still 0pt-owned.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

2026-07-12 19:46:47 +02:00

adminandClaude Sonnet 5

c89731115c

feat: add 0pt.query_source, 0pt.document_ocr, 0pt.write_external_record capabilities

Three new, tenant-agnostic P0 capabilities:

- 0pt.query_source: bounded, timeout/row-capped ad-hoc SQL query execution
  against registered database sources, via a Fabric/OneLake adapter
  (Azure AD credential-based mssql connection), with AUD0 audit events
  and an app/api/0pt/sources/[sourceId]/query endpoint.

- 0pt.document_ocr (document capture + extraction): mailbox-based document
  intake via Microsoft Graph, OCR/field extraction via Azure Document
  Intelligence, with capture/extraction API endpoints and services that
  never fabricate extracted values.

- 0pt.write_external_record (external writeback): a generic writeback-adapter
  contract plus a concrete Odoo bills adapter that matches partners via
  Fabric, backing an external-writeback-service for pushing records to
  downstream systems of record.

Known interim gaps:
- src/adapters/local-blob-store.ts is an interim ST0RE substitute pending
  the real ST0RE artifact-store integration.
- mssql is used by the Fabric/OneLake adapter but deliberately not yet
  added to package.json (pending `pnpm add mssql` once the concurrent
  session's package.json/dependency changes settle).

A separate, concurrently-active session has unrelated in-flight changes
in this same working tree (data-platform catalog/lineage/feature-store,
dataset quality, lakehouse/Iceberg docs, deploy scaffolding, package.json/
pnpm-lock.yaml updates, etc.) — none of that is touched or included here.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

2026-07-08 19:32:43 +02:00