Files
0pt/docs/P0_QUERY_SOURCE_FLOW.md
T
adminandClaude Sonnet 5 2902a26bac docs: full accuracy overhaul, grounded claim-by-claim in real code (#4)
* docs: full accuracy overhaul, grounded claim-by-claim in real code

Every doc in this repo (CLAUDE.md, README.md, SECURITY.md, docs/*.md) was
audited and rewritten by a multi-pass process: independent rewrite per
cluster, adversarial re-verification of that rewrite against the actual
code, then a final cross-doc consistency pass — each stage re-checking
claims against the repo directly rather than trusting the previous stage.

Two corrections matter most:

1. docs/ICEBERG_WRITE_BOUNDARY.md, docs/11-lakehouse-write-plan.md, and
   docs/P0_QUERY_SOURCE_FLOW.md asserted src/services/quality-check-service.ts,
   quality-report-service.ts, src/data-platform/services/catalog-service.ts,
   and sql/002_p0_dataset_quality.sql/sql/003_*.sql as real, tested code.
   git log --all across every branch shows zero commits for any of these
   paths, ever. 0pt's data-quality/QA enforcement does not exist in code.
   Corrected everywhere, with a new docs/P0_DATASET_QUALITY_FLOW.md added
   as an explicit target-spec (not-yet-built) replacement.

2. github.com/sax3l/0pt (this repo) and git.siax.io/sax3l/0pt (Gitea) are
   two diverged remotes sharing part of the same history (services/api/
   workers/adaptive_router.py is byte-identical between them). Gitea's main
   independently accumulated a WAF-bypass/proxy-rotation scraper stack in
   three 2026-07-13 commits, since removed via a Gitea PR; GitHub's main
   never had it. Which remote is canonical/deployed is unresolved — flagged
   consistently across the corpus, not decided here.

Also newly and consistently documented: the tenant_id/project_id-read-from-
client-body gap on the P0 lakehouse/rate-policy/source-query routes (no
auth/session system exists anywhere in this repo yet); that every CL0UD/
N0D/AUD0/ST0RE platform client is real but mock-only (zero process.env
reads for any of their *_URL vars); that two P0 mutation routes
(lakehouse/mappings, sources/rate-policies) always fail over HTTP today
because their prerequisite entity can never be created via any route; a
schema bug in sql/004 (FK references to opt_datasets/opt_sources, which
don't exist); and that 0pt.app's live deployment was independently
confirmed (curled this pass) to be running this exact codebase with
DATABASE_URL/REDIS_URL unset in production.

New file: docs/12-edge-capability-api.md — the edge-native P0 TypeScript
capability contract (app/api/0pt/*, /api/screen, /api/ai/parse), previously
undocumented and distinct from docs/04-api.md's FastAPI-engine-only scope.

Verified before commit: tsc --noEmit clean; vitest run 88 passed, 1 todo,
0 failures (13 files) — docs-only change, no source touched.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* docs: reflect PR #5's lakehouse-mapping-registry + status-code fixes

Follow-up to keep docs/12-edge-capability-api.md accurate once both PRs
land: fix/0pt-full-bug-hunt (#5) fixed the shared-instance bug in the
lakehouse mapping routes and added proper status-code mapping to the
document extraction route. This doc was written against the pre-fix
state in the same pass that added it, so it needs the same correction.

Does not claim the lakehouse write path fully works now -- the deeper
dataset-registration gap (no route ever calls DatasetService.registerDataset)
is unchanged and still blocks it end-to-end; only the shared-instance half
of the bug is fixed.

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-22 18:49:01 +02:00

278 lines
17 KiB
Markdown

# 0pt.query_source — Bounded Source Query Flow
> **Companion-doc note (corrected):** this doc originally called itself an
> "additive companion to `docs/P0_DATASET_QUALITY_FLOW.md`." That file did
> not exist anywhere in this repository at the time — `git log --all` for it
> returns zero commits, on any branch, ever. This pass adds
> `docs/P0_DATASET_QUALITY_FLOW.md` as a clearly-labeled **target spec, not
> yet built** (see that file). Treat the two docs as siblings: this one
> covers the real, working `0pt.query_source` read path; the other covers
> the *intended, unbuilt* dataset-quality/QA path that several other docs in
> this repo (wrongly) describe as already implemented. See "Known gaps"
> below for the details of that wider mixup.
Adds the capability to actually **read data** from a registered Source, not
just register its metadata.
## What this is
`0pt.query_source`: given a Source already registered via `0pt.create_source`
(`source_type = "database"`, `adapter_id = "fabric_onelake_sql"`), run a
caller-supplied SQL query against the customer's own Azure Fabric / OneLake
SQL analytics endpoint and return rows — synchronously, bounded by a max-row
and max-time ceiling.
> Note: `0pt.create_source` (the capability id `source-service.ts` emits in
> its AUD0 events) is **not** one of the 14 `0pt.*` capabilities currently
> registered in `cl0ud`'s `packages/capability-registry/src/registry.ts` —
> only `0pt.source_registry_list` (a read/list capability) is registered for
> the Source domain. `0pt.query_source` itself *is* registered (see
> "Capability registration" below). Whether `0pt.create_source` needs its own
> registry entry, or was meant to be folded into an existing one, is unresolved
> — flagging for Simon rather than guessing.
**Tenant-agnostic by construction.** Any SIAX customer registers their own
Fabric/OneLake warehouse (their own Entra tenant, their own service
principal, their own SQL endpoint). Nothing in the adapter or service layer
is hardcoded to one customer — every credential and endpoint is a parameter
supplied per call, never an env var fallback. (This is orthogonal to the
`tenant_id`/`project_id` handling gap described below — the *warehouse*
credentials are correctly never defaulted from env; the *platform* tenant_id
used for scoping/audit is a separate, unresolved problem.)
## Why bounded + synchronous, not just "run any query"
This path is for small, interactive/preview-style reads (validate a Source
is reachable, sample a table, preview a query before scheduling it). It is
*not* a general query engine:
- `max_rows`: default 1,000, hard ceiling 10,000 (`SOURCE_QUERY_DEFAULT_MAX_ROWS` /
`SOURCE_QUERY_HARD_MAX_ROWS` in `source-query-service.ts`). Excess rows are
truncated and the response is flagged `truncated: true`.
- `timeout_ms`: default 10s, hard ceiling 30s (`SOURCE_QUERY_DEFAULT_TIMEOUT_MS` /
`SOURCE_QUERY_HARD_MAX_TIMEOUT_MS`). A caller-requested value above the
ceiling is clamped down, not rejected.
- Anything heavier — large exports, long scans, scheduled/recurring pulls —
is meant to go through **N0D as a job** instead. The real, working
precedent for that "synchronous path vs. N0D job" split in this codebase
is `src/services/iceberg-write-service.ts`: it takes an `input.heavy`
flag and, when set, calls `this.n0d.createJob(...)` instead of doing the
work inline. (An earlier version of this doc cited `quality-check-service.ts`
for this same pattern — that file does not exist; see "Known gaps.")
`SourceQueryService` does not itself request an N0D job for anything today
— there is no "heavy" variant of `0pt.query_source` yet, only the bounded
synchronous path described here.
## Architecture
```
Adapter: src/adapters/fabric-onelake-adapter.ts
- acquireFabricAccessToken(): Entra service-principal OAuth2
client-credentials flow (POST login.microsoftonline.com/{tenantId}
/oauth2/v2.0/token) — generalized from v0-datahub's
scripts/ops/cloud-bootstrap.mjs `aadToken()`.
- FabricOneLakeAdapter.query(): opens a TDS pool to the Fabric SQL
analytics endpoint using that token (azure-active-directory-access-token
auth), same shape as v0-datahub's lib/warehouse/fabric-sql.ts
`buildConfig()`/`runFabricQuery()`, generalized to caller-supplied
tenant/client/secret/endpoint/database instead of one customer's env vars.
Service: src/services/source-query-service.ts
- SourceQueryService.querySource(): looks up the Source via the existing
SourceService.getSource() (read-only — does not modify source-service.ts),
validates source_type/adapter_id, clamps max_rows/timeout_ms, races the
adapter query against a timeout, truncates rows, emits an AUD0 event,
always releases the adapter's connection.
Route: app/api/0pt/sources/[sourceId]/query/route.ts
- POST { query, params?, tenant_id, project_id, actor_id?, correlation_id?,
max_rows?, timeout_ms?, connection_config } -> QuerySourceResult
- 400 on missing query/tenant_id/project_id/connection_config
- 404 when the source does not exist
- 422 (SourceQueryUnsupportedError) when source_type != "database" or
adapter_id != "fabric_onelake_sql"
- 504 (SourceQueryTimeoutError) on timeout
```
## Capability registration (CL0UD registry — verified this pass)
`0pt.query_source` is registered in `cl0ud`'s
`packages/capability-registry/src/registry.ts`:
| Field | Value |
|---|---|
| `surfaces` | `['api']` only — no `cli`/`mcp`/`job` surface is enabled |
| `sync` | `true` |
| `requiresJob` | `false` |
| `piiLevel` | `'high'` — "ad-hoc SQL against an arbitrary registered source can return any column, including PII" (registry's own comment) |
| `riskLevel` | `'L2'` |
| `mcpEnabled` / `cliEnabled` | both `false` |
| `auditRequired` | `true` (comment: "emits `0pt.source_queried`") |
| `approvalRequired` | `false` |
This matches the platform-wide six-surface capability model: every `0pt.*`
capability registered today exposes `api` only (plus `job` for a couple of
the heaviest write paths) — none has a live CLI or MCP tool yet. Any claim
of a working `siax 0pt query-source ...` CLI command or an MCP tool for this
capability would be describing a target, not current reality.
**Discrepancy found this pass:** the registry declares `riskLevel: 'L2'` for
`0pt.query_source`, but `source-query-service.ts`'s own AUD0 emission (see
AUD0 section below) uses `risk_level: "L4"` on every `0pt.source_queried`
event — a two-level gap between what the capability is registered as and
what it actually reports. Unresolved; needs a decision on which is correct
(or whether the registry entry should simply be updated to `L4` to match
the live code, given "external read with a tenant's own credentials" is
arguably closer to `L4` than `L2`).
## AUD0 event
`event_type: "0pt.source_queried"`, `capability_id: "0pt.query_source"`,
`resource_type: "source"`, `risk_level: "L4"` in the code as written today
(see the registry discrepancy noted above). For comparison, other real,
verified `risk_level` usages elsewhere in this repo's services: `source-service.ts`
and `dataset-service.ts` use `"L2"` for their own metadata-write events;
`document-capture-service.ts` and `document-extraction-service.ts` use
`"L3"`; `external-writeback-service.ts` (`0pt.write_external_record`, a real,
tested service) uses `"L4"` for live writes to an external system with a
tenant's credentials — that is the closest real precedent for this
capability's choice of `"L4"`, not the `src/data-platform/services/catalog-service.ts`
citation an earlier version of this doc made up (that path does not exist
anywhere in this repo — confirmed via `find`).
Emitted on both success (`ok:true, row_count, truncated, duration_ms`) and
failure (`ok:false, error`) via the same `Aud0Client` pattern as
`source-service.ts` / `external-writeback-service.ts`. Like every other
`Aud0Client` use in this repo, this runs against the in-memory mock client —
no live AUD0 ingestion endpoint is reachable from this codebase today (no
`AUD0_URL` is read via `process.env` anywhere in the code). Treat this as
"real code, mock-only, not live-verified," not as "integrated."
## Known gaps / deliberately left out
- **`tenant_id`/`project_id` are read straight from the request body, not
bound to an authenticated session.** `app/api/0pt/sources/[sourceId]/query/route.ts`
requires `body.tenant_id` and `body.project_id` as plain JSON fields (see
the route's own 400-check) and passes them straight through to
`SourceQueryService.querySource()`, which uses them unchecked for AUD0
scoping. This is **not** a deliberate two-tier trust design — it is
because **no auth/session system exists anywhere in this repo yet**
(`app/api/orders/route.ts`'s own `GET` handler has the literal comment
"org scoping added with auth" — i.e., not yet). Any caller can currently
claim to be any tenant simply by putting a different `tenant_id` in the
request body. This is one of several routes in this repo with the same
gap: `lakehouse/mappings/route.ts` and `sources/rate-policies/route.ts`
read `tenant_id`/`project_id` directly off the request body exactly like
this route does (verified by reading both files); `document-capture-service.ts`'s
own route (`app/api/0pt/documents/capture/route.ts`) does the same
(`body.tenant_id`/`body.project_id`). `lakehouse/mappings/[mappingId]/writes/route.ts`
is one hop removed — it takes no `tenant_id` field on its own POST body at
all; `IcebergWriteService.requestIcebergWrite()` instead pulls `tenant_id`/
`project_id` off the previously-stored mapping record (looked up by
`mappingId`), whose `tenant_id` was itself set, unauthenticated, via
`lakehouse/mappings/route.ts`'s POST body. Same underlying trust-chain
problem, one indirection further away — worth keeping the distinction
precise rather than implying all five routes share identical code shape.
The fix is an auth/session layer that server-binds `tenant_id` — that is
Simon's call, tracked as Phase 3 of `docs/09-build-plan.md`, not built
anywhere in this repo yet. Do not treat this route as tenant-isolated
until that lands.
- **`mssql` is not a declared dependency yet.** package.json and
pnpm-lock.yaml already have unrelated uncommitted changes from a separate
in-flight session at the time this capability was built, so adding a new
dependency there was avoided to prevent a collision. The adapter loads
`mssql` lazily via a non-literal dynamic import (works with any bundler/
typecheck without the package installed) and throws a clear, actionable
error if it's missing. Once package.json is settled, `pnpm add mssql` is
the only change needed to make `query()` hit a real Fabric endpoint — no
code changes required. As of this pass, this means `0pt.query_source`
cannot actually execute a live query against Fabric/OneLake yet — the
whole path up to the driver load is real and typechecks, but `query()`
will throw the "mssql package is not installed/resolvable" error on any
real invocation until the dependency is added.
- **In-memory `SourceService` is not shared across API route modules.**
Same pre-existing limitation as `sources/route.ts` /
`sources/[sourceId]/route.ts` (each Next.js route module instantiates its
own `SourceService`, so a Source created via `POST /api/0pt/sources` is not
yet visible to `POST /api/0pt/sources/[sourceId]/query` in the same running
process). `document-capture-service.ts` hit the same issue and added a
shared singleton for its own two routes; doing the same for Source would
mean editing `source-service.ts` and/or the two existing `sources/route.ts`
files, which was out of scope here to avoid touching that flagged area.
**Correction:** an earlier version of this doc, and the route file's own
header comment, said the real fix is "already pointed at by
`sql/002_p0_dataset_quality.sql` (`opt_sources` table)." That file does
not exist and never has — `git log --all` returns zero commits for it, on
any branch. The `sql/` directory in this repo today contains only
`001_init.sql` and `004_iceberg_write_boundary.sql`; no `002` or `003`
migration has ever been created. The real fix is still "back
`SourceService` with real persistence" — just not via a file that exists.
(`app/api/0pt/sources/[sourceId]/query/route.ts`'s own header comment
still makes this same broken reference; that is a code comment, not a doc,
so it's out of scope for this pass to edit, but it should be fixed
alongside whichever pass builds real persistence.)
- **No CL0UD policy check before executing a query.** Unlike
`external-writeback-service.ts` (`0pt.write_external_record`), which calls
`Cl0udClient.evaluatePolicy()` before executing a write, `source-query-service.ts`
has no `cl0ud` import and performs no policy check at all — confirmed by
reading the file in full. This synchronous read path is scoped as a
bounded preview read, not a governed job, but that's a design choice that
has not been written down or reviewed, not an enforced policy. Worth
revisiting once policy-gating on `0pt.query_source` is defined — and note
the CL0UD registry already declares `piiLevel: 'high'` for this
capability, which makes the absence of any policy check more notable, not
less.
- **No SQL rewriting/injection of `TOP`/`LIMIT`.** Row bounding is applied to
whatever the adapter returns, not pushed into the query itself (arbitrary
caller SQL can't be safely rewritten). A query that itself returns far more
rows than `max_rows` still executes in full on the warehouse before being
truncated client-side — only the *response* is bounded, not warehouse-side
compute. Documented, not silently hidden.
- **Cross-file fabrication this pass found and corrected here, but could not
fix everywhere it appears.** Several files in this repo (some in this
doc's own former text, some in code comments this pass does not own)
reference `src/services/quality-check-service.ts`,
`src/services/quality-report-service.ts`, `src/types/quality-check.ts`,
`src/types/quality-report.ts`, `src/data-platform/services/catalog-service.ts`,
`sql/002_p0_dataset_quality.sql`, and `sql/003_*.sql` as if they are real,
already-built, already-tested files. `git log --all` for every one of
those paths returns **zero commits, ever, on any branch** (`main`,
`feature/0pt-env-example`, `ci/add-quality-gate` all checked). These files
have never existed in this repository. This doc's own text used to make
this same false claim in three places; all are corrected above.
**Re-checked this pass:** `docs/11-lakehouse-write-plan.md` has since been
corrected by a parallel pass — it now carries its own "Correction
(documentation-accuracy pass)" callout and its item 4 explicitly reads
"**NOT BUILT** ... there is no dedicated quality-check/quality-report
service in this repo as of this writing," so it no longer needs the same
fix. **`docs/ICEBERG_WRITE_BOUNDARY.md` has *also* since been fully
corrected** (re-verified this pass, reading the current file end to end):
its former internal contradiction — a top-of-file correction callout next
to a body that still asserted the fabricated files as real two sections
down — has been resolved. Its "Verification status" section now reads
"this section previously pointed at `src/services/quality-check-service.ts`
as a peer example ... that file does not exist," and its data-quality
section (retitled "Data quality: not yet built (corrected)") now quotes
its own old wording only inside a "this section previously read..."
framing before stating plainly that those files don't exist and were
never built. Neither section currently asserts the fabricated files as
fact. Code comments in `app/api/0pt/sources/[sourceId]/query/route.ts`
(line 14: "`sql/002_p0_dataset_quality.sql` already defines `opt_sources`")
and `src/services/external-writeback-service.ts` (line 21: "...
`quality-check-service.ts`/`document-extraction-service.ts` already
follow") still carry the same false reference — confirmed still present
as of this pass — and remain outside any of these docs' scope to fix
(code, not docs; flagging for whoever next touches those two files). See
`docs/P0_DATASET_QUALITY_FLOW.md` (added this pass) for the honest,
target-spec-only replacement for what those files were supposed to do.
## Known issue: which 0pt repo is authoritative
This repo has two remotes with diverged history — `github.com/sax3l/0pt` and
`git.siax.io/sax3l/0pt` (Gitea) — that share origin lineage for at least
part of the codebase but have each accumulated commits the other doesn't
have. Which one is deployed to `0pt.app`, and whether they're meant to stay
in sync, is unresolved and is Simon's call, not something this pass
attempted to resolve. Noting it here since this doc describes code that
lives in "the 0pt repo" as if there's exactly one.