* docs: full accuracy overhaul, grounded claim-by-claim in real code Every doc in this repo (CLAUDE.md, README.md, SECURITY.md, docs/*.md) was audited and rewritten by a multi-pass process: independent rewrite per cluster, adversarial re-verification of that rewrite against the actual code, then a final cross-doc consistency pass — each stage re-checking claims against the repo directly rather than trusting the previous stage. Two corrections matter most: 1. docs/ICEBERG_WRITE_BOUNDARY.md, docs/11-lakehouse-write-plan.md, and docs/P0_QUERY_SOURCE_FLOW.md asserted src/services/quality-check-service.ts, quality-report-service.ts, src/data-platform/services/catalog-service.ts, and sql/002_p0_dataset_quality.sql/sql/003_*.sql as real, tested code. git log --all across every branch shows zero commits for any of these paths, ever. 0pt's data-quality/QA enforcement does not exist in code. Corrected everywhere, with a new docs/P0_DATASET_QUALITY_FLOW.md added as an explicit target-spec (not-yet-built) replacement. 2. github.com/sax3l/0pt (this repo) and git.siax.io/sax3l/0pt (Gitea) are two diverged remotes sharing part of the same history (services/api/ workers/adaptive_router.py is byte-identical between them). Gitea's main independently accumulated a WAF-bypass/proxy-rotation scraper stack in three 2026-07-13 commits, since removed via a Gitea PR; GitHub's main never had it. Which remote is canonical/deployed is unresolved — flagged consistently across the corpus, not decided here. Also newly and consistently documented: the tenant_id/project_id-read-from- client-body gap on the P0 lakehouse/rate-policy/source-query routes (no auth/session system exists anywhere in this repo yet); that every CL0UD/ N0D/AUD0/ST0RE platform client is real but mock-only (zero process.env reads for any of their *_URL vars); that two P0 mutation routes (lakehouse/mappings, sources/rate-policies) always fail over HTTP today because their prerequisite entity can never be created via any route; a schema bug in sql/004 (FK references to opt_datasets/opt_sources, which don't exist); and that 0pt.app's live deployment was independently confirmed (curled this pass) to be running this exact codebase with DATABASE_URL/REDIS_URL unset in production. New file: docs/12-edge-capability-api.md — the edge-native P0 TypeScript capability contract (app/api/0pt/*, /api/screen, /api/ai/parse), previously undocumented and distinct from docs/04-api.md's FastAPI-engine-only scope. Verified before commit: tsc --noEmit clean; vitest run 88 passed, 1 todo, 0 failures (13 files) — docs-only change, no source touched. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs: reflect PR #5's lakehouse-mapping-registry + status-code fixes Follow-up to keep docs/12-edge-capability-api.md accurate once both PRs land: fix/0pt-full-bug-hunt (#5) fixed the shared-instance bug in the lakehouse mapping routes and added proper status-code mapping to the document extraction route. This doc was written against the pre-fix state in the same pass that added it, so it needs the same correction. Does not claim the lakehouse write path fully works now -- the deeper dataset-registration gap (no route ever calls DatasetService.registerDataset) is unchanged and still blocks it end-to-end; only the shared-instance half of the bug is fixed. --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
278 lines
17 KiB
Markdown
278 lines
17 KiB
Markdown
# 0pt.query_source — Bounded Source Query Flow
|
|
|
|
> **Companion-doc note (corrected):** this doc originally called itself an
|
|
> "additive companion to `docs/P0_DATASET_QUALITY_FLOW.md`." That file did
|
|
> not exist anywhere in this repository at the time — `git log --all` for it
|
|
> returns zero commits, on any branch, ever. This pass adds
|
|
> `docs/P0_DATASET_QUALITY_FLOW.md` as a clearly-labeled **target spec, not
|
|
> yet built** (see that file). Treat the two docs as siblings: this one
|
|
> covers the real, working `0pt.query_source` read path; the other covers
|
|
> the *intended, unbuilt* dataset-quality/QA path that several other docs in
|
|
> this repo (wrongly) describe as already implemented. See "Known gaps"
|
|
> below for the details of that wider mixup.
|
|
|
|
Adds the capability to actually **read data** from a registered Source, not
|
|
just register its metadata.
|
|
|
|
## What this is
|
|
|
|
`0pt.query_source`: given a Source already registered via `0pt.create_source`
|
|
(`source_type = "database"`, `adapter_id = "fabric_onelake_sql"`), run a
|
|
caller-supplied SQL query against the customer's own Azure Fabric / OneLake
|
|
SQL analytics endpoint and return rows — synchronously, bounded by a max-row
|
|
and max-time ceiling.
|
|
|
|
> Note: `0pt.create_source` (the capability id `source-service.ts` emits in
|
|
> its AUD0 events) is **not** one of the 14 `0pt.*` capabilities currently
|
|
> registered in `cl0ud`'s `packages/capability-registry/src/registry.ts` —
|
|
> only `0pt.source_registry_list` (a read/list capability) is registered for
|
|
> the Source domain. `0pt.query_source` itself *is* registered (see
|
|
> "Capability registration" below). Whether `0pt.create_source` needs its own
|
|
> registry entry, or was meant to be folded into an existing one, is unresolved
|
|
> — flagging for Simon rather than guessing.
|
|
|
|
**Tenant-agnostic by construction.** Any SIAX customer registers their own
|
|
Fabric/OneLake warehouse (their own Entra tenant, their own service
|
|
principal, their own SQL endpoint). Nothing in the adapter or service layer
|
|
is hardcoded to one customer — every credential and endpoint is a parameter
|
|
supplied per call, never an env var fallback. (This is orthogonal to the
|
|
`tenant_id`/`project_id` handling gap described below — the *warehouse*
|
|
credentials are correctly never defaulted from env; the *platform* tenant_id
|
|
used for scoping/audit is a separate, unresolved problem.)
|
|
|
|
## Why bounded + synchronous, not just "run any query"
|
|
|
|
This path is for small, interactive/preview-style reads (validate a Source
|
|
is reachable, sample a table, preview a query before scheduling it). It is
|
|
*not* a general query engine:
|
|
|
|
- `max_rows`: default 1,000, hard ceiling 10,000 (`SOURCE_QUERY_DEFAULT_MAX_ROWS` /
|
|
`SOURCE_QUERY_HARD_MAX_ROWS` in `source-query-service.ts`). Excess rows are
|
|
truncated and the response is flagged `truncated: true`.
|
|
- `timeout_ms`: default 10s, hard ceiling 30s (`SOURCE_QUERY_DEFAULT_TIMEOUT_MS` /
|
|
`SOURCE_QUERY_HARD_MAX_TIMEOUT_MS`). A caller-requested value above the
|
|
ceiling is clamped down, not rejected.
|
|
- Anything heavier — large exports, long scans, scheduled/recurring pulls —
|
|
is meant to go through **N0D as a job** instead. The real, working
|
|
precedent for that "synchronous path vs. N0D job" split in this codebase
|
|
is `src/services/iceberg-write-service.ts`: it takes an `input.heavy`
|
|
flag and, when set, calls `this.n0d.createJob(...)` instead of doing the
|
|
work inline. (An earlier version of this doc cited `quality-check-service.ts`
|
|
for this same pattern — that file does not exist; see "Known gaps.")
|
|
`SourceQueryService` does not itself request an N0D job for anything today
|
|
— there is no "heavy" variant of `0pt.query_source` yet, only the bounded
|
|
synchronous path described here.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Adapter: src/adapters/fabric-onelake-adapter.ts
|
|
- acquireFabricAccessToken(): Entra service-principal OAuth2
|
|
client-credentials flow (POST login.microsoftonline.com/{tenantId}
|
|
/oauth2/v2.0/token) — generalized from v0-datahub's
|
|
scripts/ops/cloud-bootstrap.mjs `aadToken()`.
|
|
- FabricOneLakeAdapter.query(): opens a TDS pool to the Fabric SQL
|
|
analytics endpoint using that token (azure-active-directory-access-token
|
|
auth), same shape as v0-datahub's lib/warehouse/fabric-sql.ts
|
|
`buildConfig()`/`runFabricQuery()`, generalized to caller-supplied
|
|
tenant/client/secret/endpoint/database instead of one customer's env vars.
|
|
|
|
Service: src/services/source-query-service.ts
|
|
- SourceQueryService.querySource(): looks up the Source via the existing
|
|
SourceService.getSource() (read-only — does not modify source-service.ts),
|
|
validates source_type/adapter_id, clamps max_rows/timeout_ms, races the
|
|
adapter query against a timeout, truncates rows, emits an AUD0 event,
|
|
always releases the adapter's connection.
|
|
|
|
Route: app/api/0pt/sources/[sourceId]/query/route.ts
|
|
- POST { query, params?, tenant_id, project_id, actor_id?, correlation_id?,
|
|
max_rows?, timeout_ms?, connection_config } -> QuerySourceResult
|
|
- 400 on missing query/tenant_id/project_id/connection_config
|
|
- 404 when the source does not exist
|
|
- 422 (SourceQueryUnsupportedError) when source_type != "database" or
|
|
adapter_id != "fabric_onelake_sql"
|
|
- 504 (SourceQueryTimeoutError) on timeout
|
|
```
|
|
|
|
## Capability registration (CL0UD registry — verified this pass)
|
|
|
|
`0pt.query_source` is registered in `cl0ud`'s
|
|
`packages/capability-registry/src/registry.ts`:
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| `surfaces` | `['api']` only — no `cli`/`mcp`/`job` surface is enabled |
|
|
| `sync` | `true` |
|
|
| `requiresJob` | `false` |
|
|
| `piiLevel` | `'high'` — "ad-hoc SQL against an arbitrary registered source can return any column, including PII" (registry's own comment) |
|
|
| `riskLevel` | `'L2'` |
|
|
| `mcpEnabled` / `cliEnabled` | both `false` |
|
|
| `auditRequired` | `true` (comment: "emits `0pt.source_queried`") |
|
|
| `approvalRequired` | `false` |
|
|
|
|
This matches the platform-wide six-surface capability model: every `0pt.*`
|
|
capability registered today exposes `api` only (plus `job` for a couple of
|
|
the heaviest write paths) — none has a live CLI or MCP tool yet. Any claim
|
|
of a working `siax 0pt query-source ...` CLI command or an MCP tool for this
|
|
capability would be describing a target, not current reality.
|
|
|
|
**Discrepancy found this pass:** the registry declares `riskLevel: 'L2'` for
|
|
`0pt.query_source`, but `source-query-service.ts`'s own AUD0 emission (see
|
|
AUD0 section below) uses `risk_level: "L4"` on every `0pt.source_queried`
|
|
event — a two-level gap between what the capability is registered as and
|
|
what it actually reports. Unresolved; needs a decision on which is correct
|
|
(or whether the registry entry should simply be updated to `L4` to match
|
|
the live code, given "external read with a tenant's own credentials" is
|
|
arguably closer to `L4` than `L2`).
|
|
|
|
## AUD0 event
|
|
|
|
`event_type: "0pt.source_queried"`, `capability_id: "0pt.query_source"`,
|
|
`resource_type: "source"`, `risk_level: "L4"` in the code as written today
|
|
(see the registry discrepancy noted above). For comparison, other real,
|
|
verified `risk_level` usages elsewhere in this repo's services: `source-service.ts`
|
|
and `dataset-service.ts` use `"L2"` for their own metadata-write events;
|
|
`document-capture-service.ts` and `document-extraction-service.ts` use
|
|
`"L3"`; `external-writeback-service.ts` (`0pt.write_external_record`, a real,
|
|
tested service) uses `"L4"` for live writes to an external system with a
|
|
tenant's credentials — that is the closest real precedent for this
|
|
capability's choice of `"L4"`, not the `src/data-platform/services/catalog-service.ts`
|
|
citation an earlier version of this doc made up (that path does not exist
|
|
anywhere in this repo — confirmed via `find`).
|
|
|
|
Emitted on both success (`ok:true, row_count, truncated, duration_ms`) and
|
|
failure (`ok:false, error`) via the same `Aud0Client` pattern as
|
|
`source-service.ts` / `external-writeback-service.ts`. Like every other
|
|
`Aud0Client` use in this repo, this runs against the in-memory mock client —
|
|
no live AUD0 ingestion endpoint is reachable from this codebase today (no
|
|
`AUD0_URL` is read via `process.env` anywhere in the code). Treat this as
|
|
"real code, mock-only, not live-verified," not as "integrated."
|
|
|
|
## Known gaps / deliberately left out
|
|
|
|
- **`tenant_id`/`project_id` are read straight from the request body, not
|
|
bound to an authenticated session.** `app/api/0pt/sources/[sourceId]/query/route.ts`
|
|
requires `body.tenant_id` and `body.project_id` as plain JSON fields (see
|
|
the route's own 400-check) and passes them straight through to
|
|
`SourceQueryService.querySource()`, which uses them unchecked for AUD0
|
|
scoping. This is **not** a deliberate two-tier trust design — it is
|
|
because **no auth/session system exists anywhere in this repo yet**
|
|
(`app/api/orders/route.ts`'s own `GET` handler has the literal comment
|
|
"org scoping added with auth" — i.e., not yet). Any caller can currently
|
|
claim to be any tenant simply by putting a different `tenant_id` in the
|
|
request body. This is one of several routes in this repo with the same
|
|
gap: `lakehouse/mappings/route.ts` and `sources/rate-policies/route.ts`
|
|
read `tenant_id`/`project_id` directly off the request body exactly like
|
|
this route does (verified by reading both files); `document-capture-service.ts`'s
|
|
own route (`app/api/0pt/documents/capture/route.ts`) does the same
|
|
(`body.tenant_id`/`body.project_id`). `lakehouse/mappings/[mappingId]/writes/route.ts`
|
|
is one hop removed — it takes no `tenant_id` field on its own POST body at
|
|
all; `IcebergWriteService.requestIcebergWrite()` instead pulls `tenant_id`/
|
|
`project_id` off the previously-stored mapping record (looked up by
|
|
`mappingId`), whose `tenant_id` was itself set, unauthenticated, via
|
|
`lakehouse/mappings/route.ts`'s POST body. Same underlying trust-chain
|
|
problem, one indirection further away — worth keeping the distinction
|
|
precise rather than implying all five routes share identical code shape.
|
|
The fix is an auth/session layer that server-binds `tenant_id` — that is
|
|
Simon's call, tracked as Phase 3 of `docs/09-build-plan.md`, not built
|
|
anywhere in this repo yet. Do not treat this route as tenant-isolated
|
|
until that lands.
|
|
- **`mssql` is not a declared dependency yet.** package.json and
|
|
pnpm-lock.yaml already have unrelated uncommitted changes from a separate
|
|
in-flight session at the time this capability was built, so adding a new
|
|
dependency there was avoided to prevent a collision. The adapter loads
|
|
`mssql` lazily via a non-literal dynamic import (works with any bundler/
|
|
typecheck without the package installed) and throws a clear, actionable
|
|
error if it's missing. Once package.json is settled, `pnpm add mssql` is
|
|
the only change needed to make `query()` hit a real Fabric endpoint — no
|
|
code changes required. As of this pass, this means `0pt.query_source`
|
|
cannot actually execute a live query against Fabric/OneLake yet — the
|
|
whole path up to the driver load is real and typechecks, but `query()`
|
|
will throw the "mssql package is not installed/resolvable" error on any
|
|
real invocation until the dependency is added.
|
|
- **In-memory `SourceService` is not shared across API route modules.**
|
|
Same pre-existing limitation as `sources/route.ts` /
|
|
`sources/[sourceId]/route.ts` (each Next.js route module instantiates its
|
|
own `SourceService`, so a Source created via `POST /api/0pt/sources` is not
|
|
yet visible to `POST /api/0pt/sources/[sourceId]/query` in the same running
|
|
process). `document-capture-service.ts` hit the same issue and added a
|
|
shared singleton for its own two routes; doing the same for Source would
|
|
mean editing `source-service.ts` and/or the two existing `sources/route.ts`
|
|
files, which was out of scope here to avoid touching that flagged area.
|
|
**Correction:** an earlier version of this doc, and the route file's own
|
|
header comment, said the real fix is "already pointed at by
|
|
`sql/002_p0_dataset_quality.sql` (`opt_sources` table)." That file does
|
|
not exist and never has — `git log --all` returns zero commits for it, on
|
|
any branch. The `sql/` directory in this repo today contains only
|
|
`001_init.sql` and `004_iceberg_write_boundary.sql`; no `002` or `003`
|
|
migration has ever been created. The real fix is still "back
|
|
`SourceService` with real persistence" — just not via a file that exists.
|
|
(`app/api/0pt/sources/[sourceId]/query/route.ts`'s own header comment
|
|
still makes this same broken reference; that is a code comment, not a doc,
|
|
so it's out of scope for this pass to edit, but it should be fixed
|
|
alongside whichever pass builds real persistence.)
|
|
- **No CL0UD policy check before executing a query.** Unlike
|
|
`external-writeback-service.ts` (`0pt.write_external_record`), which calls
|
|
`Cl0udClient.evaluatePolicy()` before executing a write, `source-query-service.ts`
|
|
has no `cl0ud` import and performs no policy check at all — confirmed by
|
|
reading the file in full. This synchronous read path is scoped as a
|
|
bounded preview read, not a governed job, but that's a design choice that
|
|
has not been written down or reviewed, not an enforced policy. Worth
|
|
revisiting once policy-gating on `0pt.query_source` is defined — and note
|
|
the CL0UD registry already declares `piiLevel: 'high'` for this
|
|
capability, which makes the absence of any policy check more notable, not
|
|
less.
|
|
- **No SQL rewriting/injection of `TOP`/`LIMIT`.** Row bounding is applied to
|
|
whatever the adapter returns, not pushed into the query itself (arbitrary
|
|
caller SQL can't be safely rewritten). A query that itself returns far more
|
|
rows than `max_rows` still executes in full on the warehouse before being
|
|
truncated client-side — only the *response* is bounded, not warehouse-side
|
|
compute. Documented, not silently hidden.
|
|
- **Cross-file fabrication this pass found and corrected here, but could not
|
|
fix everywhere it appears.** Several files in this repo (some in this
|
|
doc's own former text, some in code comments this pass does not own)
|
|
reference `src/services/quality-check-service.ts`,
|
|
`src/services/quality-report-service.ts`, `src/types/quality-check.ts`,
|
|
`src/types/quality-report.ts`, `src/data-platform/services/catalog-service.ts`,
|
|
`sql/002_p0_dataset_quality.sql`, and `sql/003_*.sql` as if they are real,
|
|
already-built, already-tested files. `git log --all` for every one of
|
|
those paths returns **zero commits, ever, on any branch** (`main`,
|
|
`feature/0pt-env-example`, `ci/add-quality-gate` all checked). These files
|
|
have never existed in this repository. This doc's own text used to make
|
|
this same false claim in three places; all are corrected above.
|
|
**Re-checked this pass:** `docs/11-lakehouse-write-plan.md` has since been
|
|
corrected by a parallel pass — it now carries its own "Correction
|
|
(documentation-accuracy pass)" callout and its item 4 explicitly reads
|
|
"**NOT BUILT** ... there is no dedicated quality-check/quality-report
|
|
service in this repo as of this writing," so it no longer needs the same
|
|
fix. **`docs/ICEBERG_WRITE_BOUNDARY.md` has *also* since been fully
|
|
corrected** (re-verified this pass, reading the current file end to end):
|
|
its former internal contradiction — a top-of-file correction callout next
|
|
to a body that still asserted the fabricated files as real two sections
|
|
down — has been resolved. Its "Verification status" section now reads
|
|
"this section previously pointed at `src/services/quality-check-service.ts`
|
|
as a peer example ... that file does not exist," and its data-quality
|
|
section (retitled "Data quality: not yet built (corrected)") now quotes
|
|
its own old wording only inside a "this section previously read..."
|
|
framing before stating plainly that those files don't exist and were
|
|
never built. Neither section currently asserts the fabricated files as
|
|
fact. Code comments in `app/api/0pt/sources/[sourceId]/query/route.ts`
|
|
(line 14: "`sql/002_p0_dataset_quality.sql` already defines `opt_sources`")
|
|
and `src/services/external-writeback-service.ts` (line 21: "...
|
|
`quality-check-service.ts`/`document-extraction-service.ts` already
|
|
follow") still carry the same false reference — confirmed still present
|
|
as of this pass — and remain outside any of these docs' scope to fix
|
|
(code, not docs; flagging for whoever next touches those two files). See
|
|
`docs/P0_DATASET_QUALITY_FLOW.md` (added this pass) for the honest,
|
|
target-spec-only replacement for what those files were supposed to do.
|
|
|
|
## Known issue: which 0pt repo is authoritative
|
|
|
|
This repo has two remotes with diverged history — `github.com/sax3l/0pt` and
|
|
`git.siax.io/sax3l/0pt` (Gitea) — that share origin lineage for at least
|
|
part of the codebase but have each accumulated commits the other doesn't
|
|
have. Which one is deployed to `0pt.app`, and whether they're meant to stay
|
|
in sync, is unresolved and is Simon's call, not something this pass
|
|
attempted to resolve. Noting it here since this doc describes code that
|
|
lives in "the 0pt repo" as if there's exactly one.
|