Files
0pt/docs/P0_QUERY_SOURCE_FLOW.md
T
adminandClaude Sonnet 5 c89731115c feat: add 0pt.query_source, 0pt.document_ocr, 0pt.write_external_record capabilities
Three new, tenant-agnostic P0 capabilities:

- 0pt.query_source: bounded, timeout/row-capped ad-hoc SQL query execution
  against registered database sources, via a Fabric/OneLake adapter
  (Azure AD credential-based mssql connection), with AUD0 audit events
  and an app/api/0pt/sources/[sourceId]/query endpoint.

- 0pt.document_ocr (document capture + extraction): mailbox-based document
  intake via Microsoft Graph, OCR/field extraction via Azure Document
  Intelligence, with capture/extraction API endpoints and services that
  never fabricate extracted values.

- 0pt.write_external_record (external writeback): a generic writeback-adapter
  contract plus a concrete Odoo bills adapter that matches partners via
  Fabric, backing an external-writeback-service for pushing records to
  downstream systems of record.

Known interim gaps:
- src/adapters/local-blob-store.ts is an interim ST0RE substitute pending
  the real ST0RE artifact-store integration.
- mssql is used by the Fabric/OneLake adapter but deliberately not yet
  added to package.json (pending `pnpm add mssql` once the concurrent
  session's package.json/dependency changes settle).

A separate, concurrently-active session has unrelated in-flight changes
in this same working tree (data-platform catalog/lineage/feature-store,
dataset quality, lakehouse/Iceberg docs, deploy scaffolding, package.json/
pnpm-lock.yaml updates, etc.) — none of that is touched or included here.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-08 19:32:43 +02:00

6.0 KiB

0pt.query_source — Bounded Source Query Flow

Additive companion to docs/P0_DATASET_QUALITY_FLOW.md. Adds the capability to actually read data from a registered Source, not just register its metadata.

What this is

0pt.query_source: given a Source already registered via 0pt.create_source (source_type = "database", adapter_id = "fabric_onelake_sql"), run a caller-supplied SQL query against the customer's own Azure Fabric / OneLake SQL analytics endpoint and return rows — synchronously, bounded by a max-row and max-time ceiling.

Tenant-agnostic by construction. Any SIAX customer registers their own Fabric/OneLake warehouse (their own Entra tenant, their own service principal, their own SQL endpoint). Nothing in the adapter or service layer is hardcoded to one customer — every credential and endpoint is a parameter supplied per call, never an env var fallback.

Why bounded + synchronous, not just "run any query"

This path is for small, interactive/preview-style reads (validate a Source is reachable, sample a table, preview a query before scheduling it). It is not a general query engine:

  • max_rows: default 1,000, hard ceiling 10,000 (SOURCE_QUERY_DEFAULT_MAX_ROWS / SOURCE_QUERY_HARD_MAX_ROWS in source-query-service.ts). Excess rows are truncated and the response is flagged truncated: true.
  • timeout_ms: default 10s, hard ceiling 30s (SOURCE_QUERY_DEFAULT_TIMEOUT_MS / SOURCE_QUERY_HARD_MAX_TIMEOUT_MS). A caller-requested value above the ceiling is clamped down, not rejected.
  • Anything heavier — large exports, long scans, scheduled/recurring pulls — must go through N0D as a job instead, the same split quality-check-service.ts already uses (runQualityChecks → CL0UD policy → N0D job, not an inline synchronous call).

Architecture

Adapter:  src/adapters/fabric-onelake-adapter.ts
  - acquireFabricAccessToken(): Entra service-principal OAuth2
    client-credentials flow (POST login.microsoftonline.com/{tenantId}
    /oauth2/v2.0/token) — generalized from v0-datahub's
    scripts/ops/cloud-bootstrap.mjs `aadToken()`.
  - FabricOneLakeAdapter.query(): opens a TDS pool to the Fabric SQL
    analytics endpoint using that token (azure-active-directory-access-token
    auth), same shape as v0-datahub's lib/warehouse/fabric-sql.ts
    `buildConfig()`/`runFabricQuery()`, generalized to caller-supplied
    tenant/client/secret/endpoint/database instead of one customer's env vars.

Service:  src/services/source-query-service.ts
  - SourceQueryService.querySource(): looks up the Source via the existing
    SourceService.getSource() (read-only — does not modify source-service.ts),
    validates source_type/adapter_id, clamps max_rows/timeout_ms, races the
    adapter query against a timeout, truncates rows, emits an AUD0 event,
    always releases the adapter's connection.

Route:    app/api/0pt/sources/[sourceId]/query/route.ts
  - POST { query, params?, tenant_id, project_id, actor_id?, correlation_id?,
    max_rows?, timeout_ms?, connection_config } -> QuerySourceResult
  - 400 on missing query/tenant_id/project_id/connection_config
  - 404 when the source does not exist
  - 422 (SourceQueryUnsupportedError) when source_type != "database" or
    adapter_id != "fabric_onelake_sql"
  - 504 (SourceQueryTimeoutError) on timeout

AUD0:     event_type "0pt.source_queried", capability_id "0pt.query_source",
  resource_type "source", risk_level "L4" (live external data access with a
  tenant's own credentials — one level above the "L3" used for Source/
  Dataset/QualityCheck metadata writes, matching the "L4 = restricted
  sensitivity" precedent in src/data-platform/services/catalog-service.ts).
  Emitted on both success (ok:true, row_count, truncated, duration_ms) and
  failure (ok:false, error) via the same `Aud0Client` pattern as
  source-service.ts / quality-check-service.ts.

Known gaps / deliberately left out

  • mssql is not a declared dependency yet. package.json and pnpm-lock.yaml already have unrelated uncommitted changes from a separate in-flight session at the time this capability was built, so adding a new dependency there was avoided to prevent a collision. The adapter loads mssql lazily via a non-literal dynamic import (works with any bundler/ typecheck without the package installed) and throws a clear, actionable error if it's missing. Once package.json is settled, pnpm add mssql is the only change needed to make query() hit a real Fabric endpoint — no code changes required.
  • In-memory SourceService is not shared across API route modules. Same pre-existing limitation as sources/route.ts / sources/[sourceId]/route.ts (each Next.js route module instantiates its own SourceService, so a Source created via POST /api/0pt/sources is not yet visible to POST /api/0pt/sources/[sourceId]/query in the same running process). document-capture-service.ts hit the same issue and added a shared singleton for its own two routes; doing the same for Source would mean editing source-service.ts and/or the two existing sources/route.ts files, which was out of scope here to avoid touching that flagged area. The real fix is the same one already pointed at by sql/002_p0_dataset_quality.sql (opt_sources table) — back SourceService with real persistence.
  • No CL0UD policy check before executing a query. quality-check-service.ts calls Cl0udClient.evaluatePolicy() before creating a N0D job; this synchronous path does not, since it is scoped as a bounded preview read, not a governed job. Worth revisiting once policy-gating on 0pt.query_source is defined.
  • No SQL rewriting/injection of TOP/LIMIT. Row bounding is applied to whatever the adapter returns, not pushed into the query itself (arbitrary caller SQL can't be safely rewritten). A query that itself returns far more rows than max_rows still executes in full on the warehouse before being truncated client-side — only the response is bounded, not warehouse-side compute. Documented, not silently hidden.