fix(data-os): P014 — close tenant-isolation + idempotency gaps, execute real restore drill
CI (SIAX Cloud) / quality (push) Successful in 51s
Security Scan (SIAX Cloud) / security (push) Successful in 26s
CI (SIAX Cloud) / security (push) Successful in 28s
CI (SIAX Cloud) / integration (push) Successful in 1m1s
CI (SIAX Cloud) / integration (pull_request) Successful in 30s
CI (SIAX Cloud) / security (pull_request) Successful in 24s
CI (SIAX Cloud) / quality (pull_request) Successful in 48s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 22s

Tenant isolation (security-critical): every app/api/0pt/** route reading
tenant_id from client-controlled query/body never cross-checked it against
the token middleware.ts already verified (x-0pt-verified-tenant). Adds a
shared assertTenantMatchesVerified guard and wires it into all 20+ affected
routes. Also closes two full-leak-by-omission bugs found in this pass
(documents/capture and lakehouse/mappings GETs returned every tenant's rows
when tenant_id was simply omitted) and two "no tenant check at all" reads
(rate-policies/[id], lakehouse/mappings/[id]) reachable by ID alone.

Duplicate-delivery idempotency (correctness-critical): POST /api/orders now
honors Idempotency-Key (sql/006, replay-or-409-on-conflict, Postgres +
in-memory); the Python orchestrator's POST /orders/{id}/run — the actual
delivery trigger — is now idempotent under an asyncio lock, closing a real
concurrent-request race, not just the sequential-retry case.

Engine-duplication resolution: adaptive_router.py's docstring wrongly
invited a bypass ("see b0ts for scraping") contradicting the product's own
compliance stance; corrected, and given real (additive, not escalating)
block-signal tracking so a rate-limited/blocked source is excluded from the
plan instead of hammered. Also fixes a previously-undiscovered production
bug: the Python engine's only call into the TS quality bridge
(X-Internal-Token) was unconditionally 401'd by middleware.ts, which never
checked that credential — the advisory quality bridge was unreachable in any
environment enforcing auth.

Restore drill: executed for real against throwaway Docker Postgres
containers (not simulated). Found and fixed two real blockers along the way:
sql/004 referenced opt_datasets, which no migration ever created (chain was
never actually runnable end to end); lib/db.ts hardcoded ssl:"require",
making any non-TLS Postgres (incl. compose.dev.yaml's own local-dev
container) unreachable. After both fixes: full migration chain applies
cleanly, pg_dump/pg_restore round-trip is byte-identical (row counts +
content checksums), and real application code (lib/orders-store.ts,
SourceRegistryService's Postgres repository) reads the restored data back
correctly, including cross-tenant isolation still holding.

82 new tests (55 TS + 27 Python), all passing; full existing suite (291 TS +
55 Python) still green; typecheck clean on every changed/added file.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Simon Axelsson
2026-09-03 12:00:43 +02:00
co-authored by Claude Sonnet 5
parent e7516697b3
commit dc6aaa0e10
50 changed files with 1820 additions and 52 deletions
+73
View File
@@ -0,0 +1,73 @@
/**
* `src/auth/internal-token.ts` -- verification for the ONE machine-to-machine
* credential 0pt already uses for engine (Python) -> edge (Next.js) calls:
* the `X-Internal-Token` header, checked against the `INTERNAL_TOKEN` env var
* (the same shared secret `lib/engine.ts` and `lib/health-checks.ts` already
* send on the opposite-direction edge -> engine calls).
*
* P014 FIX: `services/api/workers/quality_bridge.py` has always called
* `POST /api/0pt/quality/bridge-run` with `X-Internal-Token`, never an
* `Authorization: Bearer` header. But this route sits under
* `/api/0pt/:path*`, which `middleware.ts` guards unconditionally with real
* Zitadel Bearer verification -- and nothing anywhere checked
* `X-Internal-Token` as an alternative credential. The result: in any
* environment with the default `AUTH_ENFORCE` posture (enforced), the
* Python engine's ONLY call into this bridge was rejected with 401 by
* middleware before the route handler ever ran. `OPT_QUALITY_BRIDGE_ENABLED=true`
* therefore could never actually succeed against a real deployment --
* quality_bridge.py's fail-open design (see its own docstring) silently
* absorbed the 401 as `status: "unavailable"`, so nothing outwardly broke,
* but the whole point of building the bridge (an advisory second opinion
* from the TS QualityService) was unreachable. Neither test suite caught
* this: the TS test (tests/p0/quality-bridge.test.ts) calls the route's
* POST function directly, bypassing middleware.ts entirely; the Python test
* (services/api/tests/test_quality_bridge.py) mocks the HTTP layer and never
* exercises the edge's real auth gate.
*
* This module verifies `X-Internal-Token` in constant time. middleware.ts
* accepts it ONLY for the one route the engine actually calls
* (`/api/0pt/quality/bridge-run`) -- not a blanket bypass for the rest of
* `/api/0pt/*`. Fails closed: an unset/empty `INTERNAL_TOKEN` env var never
* authenticates, even against an empty or missing header (no accidental
* "unset equals unset, so let it through").
*
* Uses only Web APIs (`TextEncoder`), not `node:crypto` -- Next.js
* middleware runs on the Edge runtime, which does not expose Node's
* `crypto.timingSafeEqual`.
*/
/** Minimal header accessor -- Web `Headers` and `NextRequest.headers` satisfy this. */
export interface HeaderLookup {
get(name: string): string | null | undefined;
}
export const INTERNAL_TOKEN_HEADER = "x-internal-token";
/** Constant-time-ish byte comparison (Edge-runtime safe: no node:crypto). */
function timingSafeEqualStr(a: string, b: string): boolean {
const enc = new TextEncoder();
const aBytes = enc.encode(a);
const bBytes = enc.encode(b);
const len = Math.max(aBytes.length, bBytes.length);
let diff = aBytes.length ^ bBytes.length;
for (let i = 0; i < len; i++) {
diff |= (aBytes[i] ?? 0) ^ (bBytes[i] ?? 0);
}
return diff === 0;
}
/**
* True only when `X-Internal-Token` is present AND matches a non-empty
* configured `INTERNAL_TOKEN`. Fails closed on every other combination
* (missing header, missing/empty env var, mismatch).
*/
export function verifyInternalServiceToken(
headers: HeaderLookup,
env: { INTERNAL_TOKEN?: string } = process.env,
): boolean {
const expected = env.INTERNAL_TOKEN?.trim();
if (!expected) return false;
const provided = headers.get(INTERNAL_TOKEN_HEADER);
if (!provided) return false;
return timingSafeEqualStr(provided, expected);
}
+68
View File
@@ -0,0 +1,68 @@
/**
* src/http/tenant-guard.ts
* ==========================
* P014 fix: every route under `app/api/0pt/**` sits behind `middleware.ts`,
* which performs real Zitadel Bearer verification and — only when a
* principal was actually verified — forwards the result as the
* `x-0pt-verified-tenant` request header (see middleware.ts, src/auth/context.ts).
*
* Independently of that, every one of these route handlers reads `tenant_id`
* from CLIENT-CONTROLLED input (a `?tenant_id=` query param or a JSON body
* field) to decide which tenant's rows to read or write. Until this pass,
* NONE of them cross-checked that client-supplied value against the verified
* header — so a caller holding a valid token for tenant A could read or
* write tenant B's sources, rights, provenance, health, quality rules/runs,
* documents, or lakehouse mappings simply by naming `tenant_id=t_b` in the
* request. `SourceRegistryService.assertSourceTenant` (and friends) only
* checks that the object being read *belongs* to the tenant_id it was asked
* to check against — it has no way to know whether the CALLER is actually
* entitled to assert that tenant_id in the first place. That entitlement
* check belongs at the HTTP boundary, once, here.
*
* This mirrors the exact decision `src/auth/context.ts`'s `resolveAuthContext`
* already makes for the enforced-Bearer path ("a disagreeing X-Tenant-Id
* header is a 403"), applied uniformly to every 0pt capability route that
* takes tenant_id as a body/query parameter rather than relying on
* middleware to inject ctx.tenantId directly (most of these routes predate
* that plumbing and read tenant_id straight from the request; fixing that
* more deeply is a larger refactor than this pass — this closes the actual
* authz hole without changing every route's request contract).
*/
import { NextResponse, type NextRequest } from "next/server";
export const VERIFIED_TENANT_HEADER = "x-0pt-verified-tenant";
/**
* Returns a 403 NextResponse when `tenantId` (whatever the route read it
* from — query string or body) disagrees with the tenant middleware.ts
* already verified for this request. Returns `null` when the caller may
* proceed — either because the tenant matches, or because no verified
* tenant is present on the request at all.
*
* The header is absent only in the explicit local-development escape hatch
* (`AUTH_ENFORCE=false` with no bearer token presented) — `assertAuthConfigSafe()`
* refuses to boot that way in production, so an absent header here can only
* happen in a mode that already accepts fully unauthenticated requests.
* Rejecting in that mode would be strictly more restrictive than the
* existing dev escape hatch intends, so this deliberately no-ops rather than
* invents a tenant to compare against.
*/
export function assertTenantMatchesVerified(
request: NextRequest,
tenantId: string | null | undefined,
): NextResponse | null {
const verified = request.headers.get(VERIFIED_TENANT_HEADER);
if (!verified) return null;
if (!tenantId) return null; // handlers already 400 on a missing tenant_id separately
if (tenantId !== verified) {
return NextResponse.json(
{
error: "OPT_ACCESS_DENIED",
message: "tenant_id disagrees with the authenticated token's tenant",
details: { requested_tenant: tenantId, verified_tenant: verified },
},
{ status: 403 },
);
}
return null;
}
@@ -125,9 +125,21 @@ export class DatasetIcebergMappingService {
return this.toMapping(record);
}
async getMapping(mappingId: string): Promise<DatasetIcebergMapping> {
// P014: `tenantId` is optional so trusted in-process callers (existing
// unit tests, and internal callers that already resolved the tenant some
// other way) are unaffected -- but both HTTP routes that reach this method
// now always pass it, closing a real gap: `getMapping` used to be
// reachable by mapping ID alone with NO tenant check anywhere, so any
// authenticated caller could read -- or, via requestIcebergWrite below,
// WRITE a new snapshot onto -- another tenant's dataset<->Iceberg-table
// mapping just by guessing/enumerating dataset_iceberg_mapping_id. A
// mismatched tenantId reports identically to "not found" (no existence
// oracle), matching SourceRegistryService.assertSourceTenant's convention.
async getMapping(mappingId: string, tenantId?: string): Promise<DatasetIcebergMapping> {
const record = this.mappings.get(mappingId);
if (!record) throw new Error(`Dataset Iceberg mapping ${mappingId} not found`);
if (!record || (tenantId && record.tenant_id !== tenantId)) {
throw new Error(`Dataset Iceberg mapping ${mappingId} not found`);
}
return this.toMapping(record);
}
+16 -1
View File
@@ -249,7 +249,22 @@ export class DocumentExtractionService {
return record;
}
async listExtractionsForDocument(documentId: string): Promise<DocumentExtractionRecord[]> {
// P014: `tenantId` is optional so existing trusted in-process callers are
// unaffected, but the HTTP route now always supplies it -- closing a real
// gap: this used to be reachable by documentId alone with NO tenant
// check anywhere, so any authenticated caller could list another
// tenant's OCR/structuring extraction results (potentially containing
// extracted document content, e.g. invoice fields) by guessing or
// enumerating documentId. A mismatched tenantId reports identically to
// "not found" (no existence oracle), same convention as the rest of the
// surface.
async listExtractionsForDocument(documentId: string, tenantId?: string): Promise<DocumentExtractionRecord[]> {
if (tenantId) {
const document = await this.captureService.getDocument(documentId).catch(() => undefined);
if (!document || document.tenant_id !== tenantId) {
throw new Error(`Document ${documentId} not found`);
}
}
return Array.from(this.extractions.values()).filter((r) => r.document_id === documentId);
}
}
+10 -1
View File
@@ -65,6 +65,15 @@ export interface MapDatasetToIcebergTableResult {
export interface RequestIcebergWriteInput {
dataset_iceberg_mapping_id: string;
/**
* P014: optional so existing trusted in-process callers are unaffected,
* but the HTTP route (app/api/0pt/lakehouse/mappings/[mappingId]/writes)
* always supplies it now. When present, it must match the mapping's own
* tenant_id or the write is refused as not-found -- see getMapping's
* doc comment for the gap this closes (a foreign mapping_id used to let
* any caller register a snapshot against another tenant's table).
*/
tenant_id?: string;
actor_id: string;
correlation_id: string;
snapshot_type?: TableSnapshotType;
@@ -176,7 +185,7 @@ export class IcebergWriteService {
}
async requestIcebergWrite(input: RequestIcebergWriteInput): Promise<RequestIcebergWriteResult> {
const mapping = await this.mappingService.getMapping(input.dataset_iceberg_mapping_id);
const mapping = await this.mappingService.getMapping(input.dataset_iceberg_mapping_id, input.tenant_id);
if (mapping.mapping_status !== "active") {
throw new Error(`Dataset Iceberg mapping ${input.dataset_iceberg_mapping_id} is not active (status=${mapping.mapping_status})`);
}
+13 -2
View File
@@ -103,9 +103,20 @@ export class SourceRatePolicyService {
return this.toPolicy(record);
}
async getSourceRatePolicy(sourceRatePolicyId: string): Promise<SourceRatePolicy> {
// P014: `tenantId` is optional so existing internal callers that already
// resolved a policy from a tenant-scoped list (or are trusted, in-process
// callers) are unaffected — but the HTTP route now always passes it, which
// closes a real gap: this lookup used to be reachable by rate-policy ID
// alone with NO tenant check anywhere, so any authenticated caller could
// read another tenant's crawl-budget/rate-limit/compliance-notes record
// just by guessing or enumerating IDs. A mismatched tenantId is reported
// identically to "not found" (no existence oracle), matching the
// convention SourceRegistryService.assertSourceTenant already established.
async getSourceRatePolicy(sourceRatePolicyId: string, tenantId?: string): Promise<SourceRatePolicy> {
const record = this.policies.get(sourceRatePolicyId);
if (!record) throw new Error(`Source rate policy ${sourceRatePolicyId} not found`);
if (!record || (tenantId && record.tenant_id !== tenantId)) {
throw new Error(`Source rate policy ${sourceRatePolicyId} not found`);
}
return this.toPolicy(record);
}
+7
View File
@@ -170,6 +170,13 @@ export const mapDatasetToIcebergTableSchema = z.object({
/** POST /api/0pt/lakehouse/mappings/[mappingId]/writes -- RequestIcebergWriteInput. */
export const requestIcebergWriteSchema = z.object({
// P014: required so the route can both (a) cross-check it against the
// caller's verified tenant (assertTenantMatchesVerified) and (b) thread it
// into IcebergWriteService.requestIcebergWrite -> getMapping, which
// refuses the write as not-found when it disagrees with the mapping's own
// tenant_id. See getMapping's doc comment for the cross-tenant write gap
// this closes.
tenant_id: z.string().min(1),
actor_id: z.string().min(1),
correlation_id: z.string().min(1),
snapshot_type: z.string().min(1).optional(),