0pt/docs/P0_QUERY_SOURCE_FLOW.md

0pt.query_source — Bounded Source Query Flow

Companion-doc note (corrected): this doc originally called itself an "additive companion to docs/P0_DATASET_QUALITY_FLOW.md." That file did not exist anywhere in this repository at the time — git log --all for it returns zero commits, on any branch, ever. This pass adds docs/P0_DATASET_QUALITY_FLOW.md as a clearly-labeled target spec, not yet built (see that file). Treat the two docs as siblings: this one covers the real, working 0pt.query_source read path; the other covers the intended, unbuilt dataset-quality/QA path that several other docs in this repo (wrongly) describe as already implemented. See "Known gaps" below for the details of that wider mixup.

Adds the capability to actually read data from a registered Source, not just register its metadata.

What this is

0pt.query_source: given a Source already registered via 0pt.create_source (source_type = "database", adapter_id = "fabric_onelake_sql"), run a caller-supplied SQL query against the customer's own Azure Fabric / OneLake SQL analytics endpoint and return rows — synchronously, bounded by a max-row and max-time ceiling.

Note: 0pt.create_source (the capability id source-service.ts emits in its AUD0 events) is not one of the 14 0pt.* capabilities currently registered in cl0ud's packages/capability-registry/src/registry.ts — only 0pt.source_registry_list (a read/list capability) is registered for the Source domain. 0pt.query_source itself is registered (see "Capability registration" below). Whether 0pt.create_source needs its own registry entry, or was meant to be folded into an existing one, is unresolved — flagging for Simon rather than guessing.

Tenant-agnostic by construction. Any SIAX customer registers their own Fabric/OneLake warehouse (their own Entra tenant, their own service principal, their own SQL endpoint). Nothing in the adapter or service layer is hardcoded to one customer — every credential and endpoint is a parameter supplied per call, never an env var fallback. (This is orthogonal to the tenant_id/project_id handling gap described below — the warehouse credentials are correctly never defaulted from env; the platform tenant_id used for scoping/audit is a separate, unresolved problem.)

Why bounded + synchronous, not just "run any query"

This path is for small, interactive/preview-style reads (validate a Source is reachable, sample a table, preview a query before scheduling it). It is not a general query engine:

• max_rows: default 1,000, hard ceiling 10,000 (SOURCE_QUERY_DEFAULT_MAX_ROWS / SOURCE_QUERY_HARD_MAX_ROWS in source-query-service.ts). Excess rows are truncated and the response is flagged truncated: true.
• timeout_ms: default 10s, hard ceiling 30s (SOURCE_QUERY_DEFAULT_TIMEOUT_MS / SOURCE_QUERY_HARD_MAX_TIMEOUT_MS). A caller-requested value above the ceiling is clamped down, not rejected.
• Anything heavier — large exports, long scans, scheduled/recurring pulls — is meant to go through N0D as a job instead. The real, working precedent for that "synchronous path vs. N0D job" split in this codebase is src/services/iceberg-write-service.ts: it takes an input.heavy flag and, when set, calls this.n0d.createJob(...) instead of doing the work inline. (An earlier version of this doc cited quality-check-service.ts for this same pattern — that file does not exist; see "Known gaps.") SourceQueryService does not itself request an N0D job for anything today — there is no "heavy" variant of 0pt.query_source yet, only the bounded synchronous path described here.

Architecture

Adapter:  src/adapters/fabric-onelake-adapter.ts
  - acquireFabricAccessToken(): Entra service-principal OAuth2
    client-credentials flow (POST login.microsoftonline.com/{tenantId}
    /oauth2/v2.0/token) — generalized from v0-datahub's
    scripts/ops/cloud-bootstrap.mjs `aadToken()`.
  - FabricOneLakeAdapter.query(): opens a TDS pool to the Fabric SQL
    analytics endpoint using that token (azure-active-directory-access-token
    auth), same shape as v0-datahub's lib/warehouse/fabric-sql.ts
    `buildConfig()`/`runFabricQuery()`, generalized to caller-supplied
    tenant/client/secret/endpoint/database instead of one customer's env vars.

Service:  src/services/source-query-service.ts
  - SourceQueryService.querySource(): looks up the Source via the existing
    SourceService.getSource() (read-only — does not modify source-service.ts),
    validates source_type/adapter_id, clamps max_rows/timeout_ms, races the
    adapter query against a timeout, truncates rows, emits an AUD0 event,
    always releases the adapter's connection.

Route:    app/api/0pt/sources/[sourceId]/query/route.ts
  - POST { query, params?, tenant_id, project_id, actor_id?, correlation_id?,
    max_rows?, timeout_ms?, connection_config } -> QuerySourceResult
  - 400 on missing query/tenant_id/project_id/connection_config
  - 404 when the source does not exist
  - 422 (SourceQueryUnsupportedError) when source_type != "database" or
    adapter_id != "fabric_onelake_sql"
  - 504 (SourceQueryTimeoutError) on timeout

Capability registration (CL0UD registry — verified this pass)

0pt.query_source is registered in cl0ud's packages/capability-registry/src/registry.ts:

Field	Value
surfaces	['api'] only — no cli/mcp/job surface is enabled
sync	true
requiresJob	false
piiLevel	'high' — "ad-hoc SQL against an arbitrary registered source can return any column, including PII" (registry's own comment)
riskLevel	'L2'
mcpEnabled / cliEnabled	both false
auditRequired	true (comment: "emits 0pt.source_queried")
approvalRequired	false

This matches the platform-wide six-surface capability model: every 0pt.* capability registered today exposes api only (plus job for a couple of the heaviest write paths) — none has a live CLI or MCP tool yet. Any claim of a working siax 0pt query-source ... CLI command or an MCP tool for this capability would be describing a target, not current reality.

Discrepancy found this pass: the registry declares riskLevel: 'L2' for 0pt.query_source, but source-query-service.ts's own AUD0 emission (see AUD0 section below) uses risk_level: "L4" on every 0pt.source_queried event — a two-level gap between what the capability is registered as and what it actually reports. Unresolved; needs a decision on which is correct (or whether the registry entry should simply be updated to L4 to match the live code, given "external read with a tenant's own credentials" is arguably closer to L4 than L2).

AUD0 event

event_type: "0pt.source_queried", capability_id: "0pt.query_source", resource_type: "source", risk_level: "L4" in the code as written today (see the registry discrepancy noted above). For comparison, other real, verified risk_level usages elsewhere in this repo's services: source-service.ts and dataset-service.ts use "L2" for their own metadata-write events; document-capture-service.ts and document-extraction-service.ts use "L3"; external-writeback-service.ts (0pt.write_external_record, a real, tested service) uses "L4" for live writes to an external system with a tenant's credentials — that is the closest real precedent for this capability's choice of "L4", not the src/data-platform/services/catalog-service.ts citation an earlier version of this doc made up (that path does not exist anywhere in this repo — confirmed via find).

Emitted on both success (ok:true, row_count, truncated, duration_ms) and failure (ok:false, error) via the same Aud0Client pattern as source-service.ts / external-writeback-service.ts. Like every other Aud0Client use in this repo, this runs against the in-memory mock client — no live AUD0 ingestion endpoint is reachable from this codebase today (no AUD0_URL is read via process.env anywhere in the code). Treat this as "real code, mock-only, not live-verified," not as "integrated."

Known gaps / deliberately left out

• tenant_id/project_id are read straight from the request body, not bound to an authenticated session. app/api/0pt/sources/[sourceId]/query/route.ts requires body.tenant_id and body.project_id as plain JSON fields (see the route's own 400-check) and passes them straight through to SourceQueryService.querySource(), which uses them unchecked for AUD0 scoping. This is not a deliberate two-tier trust design — it is because no auth/session system exists anywhere in this repo yet (app/api/orders/route.ts's own GET handler has the literal comment "org scoping added with auth" — i.e., not yet). Any caller can currently claim to be any tenant simply by putting a different tenant_id in the request body. This is one of several routes in this repo with the same gap: lakehouse/mappings/route.ts and sources/rate-policies/route.ts read tenant_id/project_id directly off the request body exactly like this route does (verified by reading both files); document-capture-service.ts's own route (app/api/0pt/documents/capture/route.ts) does the same (body.tenant_id/body.project_id). lakehouse/mappings/[mappingId]/writes/route.ts is one hop removed — it takes no tenant_id field on its own POST body at all; IcebergWriteService.requestIcebergWrite() instead pulls tenant_id/ project_id off the previously-stored mapping record (looked up by mappingId), whose tenant_id was itself set, unauthenticated, via lakehouse/mappings/route.ts's POST body. Same underlying trust-chain problem, one indirection further away — worth keeping the distinction precise rather than implying all five routes share identical code shape. The fix is an auth/session layer that server-binds tenant_id — that is Simon's call, tracked as Phase 3 of docs/09-build-plan.md, not built anywhere in this repo yet. Do not treat this route as tenant-isolated until that lands.
• mssql is not a declared dependency yet. package.json and pnpm-lock.yaml already have unrelated uncommitted changes from a separate in-flight session at the time this capability was built, so adding a new dependency there was avoided to prevent a collision. The adapter loads mssql lazily via a non-literal dynamic import (works with any bundler/ typecheck without the package installed) and throws a clear, actionable error if it's missing. Once package.json is settled, pnpm add mssql is the only change needed to make query() hit a real Fabric endpoint — no code changes required. As of this pass, this means 0pt.query_source cannot actually execute a live query against Fabric/OneLake yet — the whole path up to the driver load is real and typechecks, but query() will throw the "mssql package is not installed/resolvable" error on any real invocation until the dependency is added.
• In-memory SourceService is not shared across API route modules. Same pre-existing limitation as sources/route.ts / sources/[sourceId]/route.ts (each Next.js route module instantiates its own SourceService, so a Source created via POST /api/0pt/sources is not yet visible to POST /api/0pt/sources/[sourceId]/query in the same running process). document-capture-service.ts hit the same issue and added a shared singleton for its own two routes; doing the same for Source would mean editing source-service.ts and/or the two existing sources/route.ts files, which was out of scope here to avoid touching that flagged area. Correction: an earlier version of this doc, and the route file's own header comment, said the real fix is "already pointed at by sql/002_p0_dataset_quality.sql (opt_sources table)." That file does not exist and never has — git log --all returns zero commits for it, on any branch. The sql/ directory in this repo today contains only 001_init.sql and 004_iceberg_write_boundary.sql; no 002 or 003 migration has ever been created. The real fix is still "back SourceService with real persistence" — just not via a file that exists. (app/api/0pt/sources/[sourceId]/query/route.ts's own header comment still makes this same broken reference; that is a code comment, not a doc, so it's out of scope for this pass to edit, but it should be fixed alongside whichever pass builds real persistence.)
• No CL0UD policy check before executing a query. Unlike external-writeback-service.ts (0pt.write_external_record), which calls Cl0udClient.evaluatePolicy() before executing a write, source-query-service.ts has no cl0ud import and performs no policy check at all — confirmed by reading the file in full. This synchronous read path is scoped as a bounded preview read, not a governed job, but that's a design choice that has not been written down or reviewed, not an enforced policy. Worth revisiting once policy-gating on 0pt.query_source is defined — and note the CL0UD registry already declares piiLevel: 'high' for this capability, which makes the absence of any policy check more notable, not less.
• No SQL rewriting/injection of TOP/LIMIT. Row bounding is applied to whatever the adapter returns, not pushed into the query itself (arbitrary caller SQL can't be safely rewritten). A query that itself returns far more rows than max_rows still executes in full on the warehouse before being truncated client-side — only the response is bounded, not warehouse-side compute. Documented, not silently hidden.
• Cross-file fabrication this pass found and corrected here, but could not fix everywhere it appears. Several files in this repo (some in this doc's own former text, some in code comments this pass does not own) reference src/services/quality-check-service.ts, src/services/quality-report-service.ts, src/types/quality-check.ts, src/types/quality-report.ts, src/data-platform/services/catalog-service.ts, sql/002_p0_dataset_quality.sql, and sql/003_*.sql as if they are real, already-built, already-tested files. git log --all for every one of those paths returns zero commits, ever, on any branch (main, feature/0pt-env-example, ci/add-quality-gate all checked). These files have never existed in this repository. This doc's own text used to make this same false claim in three places; all are corrected above. Re-checked this pass: docs/11-lakehouse-write-plan.md has since been corrected by a parallel pass — it now carries its own "Correction (documentation-accuracy pass)" callout and its item 4 explicitly reads "NOT BUILT ... there is no dedicated quality-check/quality-report service in this repo as of this writing," so it no longer needs the same fix. docs/ICEBERG_WRITE_BOUNDARY.md has also since been fully corrected (re-verified this pass, reading the current file end to end): its former internal contradiction — a top-of-file correction callout next to a body that still asserted the fabricated files as real two sections down — has been resolved. Its "Verification status" section now reads "this section previously pointed at src/services/quality-check-service.ts as a peer example ... that file does not exist," and its data-quality section (retitled "Data quality: not yet built (corrected)") now quotes its own old wording only inside a "this section previously read..." framing before stating plainly that those files don't exist and were never built. Neither section currently asserts the fabricated files as fact. Code comments in app/api/0pt/sources/[sourceId]/query/route.ts (line 14: "sql/002_p0_dataset_quality.sql already defines opt_sources") and src/services/external-writeback-service.ts (line 21: "... quality-check-service.ts/document-extraction-service.ts already follow") still carry the same false reference — confirmed still present as of this pass — and remain outside any of these docs' scope to fix (code, not docs; flagging for whoever next touches those two files). See docs/P0_DATASET_QUALITY_FLOW.md (added this pass) for the honest, target-spec-only replacement for what those files were supposed to do.

Known issue: which 0pt repo is authoritative

This repo has two remotes with diverged history — github.com/sax3l/0pt and git.siax.io/sax3l/0pt (Gitea) — that share origin lineage for at least part of the codebase but have each accumulated commits the other doesn't have. Which one is deployed to 0pt.app, and whether they're meant to stay in sync, is unresolved and is Simon's call, not something this pass attempted to resolve. Noting it here since this doc describes code that lives in "the 0pt repo" as if there's exactly one.