Three new, tenant-agnostic P0 capabilities: - 0pt.query_source: bounded, timeout/row-capped ad-hoc SQL query execution against registered database sources, via a Fabric/OneLake adapter (Azure AD credential-based mssql connection), with AUD0 audit events and an app/api/0pt/sources/[sourceId]/query endpoint. - 0pt.document_ocr (document capture + extraction): mailbox-based document intake via Microsoft Graph, OCR/field extraction via Azure Document Intelligence, with capture/extraction API endpoints and services that never fabricate extracted values. - 0pt.write_external_record (external writeback): a generic writeback-adapter contract plus a concrete Odoo bills adapter that matches partners via Fabric, backing an external-writeback-service for pushing records to downstream systems of record. Known interim gaps: - src/adapters/local-blob-store.ts is an interim ST0RE substitute pending the real ST0RE artifact-store integration. - mssql is used by the Fabric/OneLake adapter but deliberately not yet added to package.json (pending `pnpm add mssql` once the concurrent session's package.json/dependency changes settle). A separate, concurrently-active session has unrelated in-flight changes in this same working tree (data-platform catalog/lineage/feature-store, dataset quality, lakehouse/Iceberg docs, deploy scaffolding, package.json/ pnpm-lock.yaml updates, etc.) — none of that is touched or included here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
6.0 KiB
0pt.query_source — Bounded Source Query Flow
Additive companion to docs/P0_DATASET_QUALITY_FLOW.md. Adds the capability
to actually read data from a registered Source, not just register its
metadata.
What this is
0pt.query_source: given a Source already registered via 0pt.create_source
(source_type = "database", adapter_id = "fabric_onelake_sql"), run a
caller-supplied SQL query against the customer's own Azure Fabric / OneLake
SQL analytics endpoint and return rows — synchronously, bounded by a max-row
and max-time ceiling.
Tenant-agnostic by construction. Any SIAX customer registers their own Fabric/OneLake warehouse (their own Entra tenant, their own service principal, their own SQL endpoint). Nothing in the adapter or service layer is hardcoded to one customer — every credential and endpoint is a parameter supplied per call, never an env var fallback.
Why bounded + synchronous, not just "run any query"
This path is for small, interactive/preview-style reads (validate a Source is reachable, sample a table, preview a query before scheduling it). It is not a general query engine:
max_rows: default 1,000, hard ceiling 10,000 (SOURCE_QUERY_DEFAULT_MAX_ROWS/SOURCE_QUERY_HARD_MAX_ROWSinsource-query-service.ts). Excess rows are truncated and the response is flaggedtruncated: true.timeout_ms: default 10s, hard ceiling 30s (SOURCE_QUERY_DEFAULT_TIMEOUT_MS/SOURCE_QUERY_HARD_MAX_TIMEOUT_MS). A caller-requested value above the ceiling is clamped down, not rejected.- Anything heavier — large exports, long scans, scheduled/recurring pulls —
must go through N0D as a job instead, the same split
quality-check-service.tsalready uses (runQualityChecks→ CL0UD policy → N0D job, not an inline synchronous call).
Architecture
Adapter: src/adapters/fabric-onelake-adapter.ts
- acquireFabricAccessToken(): Entra service-principal OAuth2
client-credentials flow (POST login.microsoftonline.com/{tenantId}
/oauth2/v2.0/token) — generalized from v0-datahub's
scripts/ops/cloud-bootstrap.mjs `aadToken()`.
- FabricOneLakeAdapter.query(): opens a TDS pool to the Fabric SQL
analytics endpoint using that token (azure-active-directory-access-token
auth), same shape as v0-datahub's lib/warehouse/fabric-sql.ts
`buildConfig()`/`runFabricQuery()`, generalized to caller-supplied
tenant/client/secret/endpoint/database instead of one customer's env vars.
Service: src/services/source-query-service.ts
- SourceQueryService.querySource(): looks up the Source via the existing
SourceService.getSource() (read-only — does not modify source-service.ts),
validates source_type/adapter_id, clamps max_rows/timeout_ms, races the
adapter query against a timeout, truncates rows, emits an AUD0 event,
always releases the adapter's connection.
Route: app/api/0pt/sources/[sourceId]/query/route.ts
- POST { query, params?, tenant_id, project_id, actor_id?, correlation_id?,
max_rows?, timeout_ms?, connection_config } -> QuerySourceResult
- 400 on missing query/tenant_id/project_id/connection_config
- 404 when the source does not exist
- 422 (SourceQueryUnsupportedError) when source_type != "database" or
adapter_id != "fabric_onelake_sql"
- 504 (SourceQueryTimeoutError) on timeout
AUD0: event_type "0pt.source_queried", capability_id "0pt.query_source",
resource_type "source", risk_level "L4" (live external data access with a
tenant's own credentials — one level above the "L3" used for Source/
Dataset/QualityCheck metadata writes, matching the "L4 = restricted
sensitivity" precedent in src/data-platform/services/catalog-service.ts).
Emitted on both success (ok:true, row_count, truncated, duration_ms) and
failure (ok:false, error) via the same `Aud0Client` pattern as
source-service.ts / quality-check-service.ts.
Known gaps / deliberately left out
mssqlis not a declared dependency yet. package.json and pnpm-lock.yaml already have unrelated uncommitted changes from a separate in-flight session at the time this capability was built, so adding a new dependency there was avoided to prevent a collision. The adapter loadsmssqllazily via a non-literal dynamic import (works with any bundler/ typecheck without the package installed) and throws a clear, actionable error if it's missing. Once package.json is settled,pnpm add mssqlis the only change needed to makequery()hit a real Fabric endpoint — no code changes required.- In-memory SourceService is not shared across API route modules.
Same pre-existing limitation as
sources/route.ts/sources/[sourceId]/route.ts(each Next.js route module instantiates its ownSourceService, so a Source created viaPOST /api/0pt/sourcesis not yet visible toPOST /api/0pt/sources/[sourceId]/queryin the same running process).document-capture-service.tshit the same issue and added a shared singleton for its own two routes; doing the same for Source would mean editingsource-service.tsand/or the two existingsources/route.tsfiles, which was out of scope here to avoid touching that flagged area. The real fix is the same one already pointed at bysql/002_p0_dataset_quality.sql(opt_sourcestable) — backSourceServicewith real persistence. - No CL0UD policy check before executing a query.
quality-check-service.tscallsCl0udClient.evaluatePolicy()before creating a N0D job; this synchronous path does not, since it is scoped as a bounded preview read, not a governed job. Worth revisiting once policy-gating on0pt.query_sourceis defined. - No SQL rewriting/injection of
TOP/LIMIT. Row bounding is applied to whatever the adapter returns, not pushed into the query itself (arbitrary caller SQL can't be safely rewritten). A query that itself returns far more rows thanmax_rowsstill executes in full on the warehouse before being truncated client-side — only the response is bounded, not warehouse-side compute. Documented, not silently hidden.