fix(data-os): P014 — close tenant-isolation + idempotency gaps, execute real restore drill
CI (SIAX Cloud) / quality (push) Successful in 51s
Security Scan (SIAX Cloud) / security (push) Successful in 26s
CI (SIAX Cloud) / security (push) Successful in 28s
CI (SIAX Cloud) / integration (push) Successful in 1m1s
CI (SIAX Cloud) / integration (pull_request) Successful in 30s
CI (SIAX Cloud) / security (pull_request) Successful in 24s
CI (SIAX Cloud) / quality (pull_request) Successful in 48s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 22s
CI (SIAX Cloud) / quality (push) Successful in 51s
Security Scan (SIAX Cloud) / security (push) Successful in 26s
CI (SIAX Cloud) / security (push) Successful in 28s
CI (SIAX Cloud) / integration (push) Successful in 1m1s
CI (SIAX Cloud) / integration (pull_request) Successful in 30s
CI (SIAX Cloud) / security (pull_request) Successful in 24s
CI (SIAX Cloud) / quality (pull_request) Successful in 48s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 22s
Tenant isolation (security-critical): every app/api/0pt/** route reading
tenant_id from client-controlled query/body never cross-checked it against
the token middleware.ts already verified (x-0pt-verified-tenant). Adds a
shared assertTenantMatchesVerified guard and wires it into all 20+ affected
routes. Also closes two full-leak-by-omission bugs found in this pass
(documents/capture and lakehouse/mappings GETs returned every tenant's rows
when tenant_id was simply omitted) and two "no tenant check at all" reads
(rate-policies/[id], lakehouse/mappings/[id]) reachable by ID alone.
Duplicate-delivery idempotency (correctness-critical): POST /api/orders now
honors Idempotency-Key (sql/006, replay-or-409-on-conflict, Postgres +
in-memory); the Python orchestrator's POST /orders/{id}/run — the actual
delivery trigger — is now idempotent under an asyncio lock, closing a real
concurrent-request race, not just the sequential-retry case.
Engine-duplication resolution: adaptive_router.py's docstring wrongly
invited a bypass ("see b0ts for scraping") contradicting the product's own
compliance stance; corrected, and given real (additive, not escalating)
block-signal tracking so a rate-limited/blocked source is excluded from the
plan instead of hammered. Also fixes a previously-undiscovered production
bug: the Python engine's only call into the TS quality bridge
(X-Internal-Token) was unconditionally 401'd by middleware.ts, which never
checked that credential — the advisory quality bridge was unreachable in any
environment enforcing auth.
Restore drill: executed for real against throwaway Docker Postgres
containers (not simulated). Found and fixed two real blockers along the way:
sql/004 referenced opt_datasets, which no migration ever created (chain was
never actually runnable end to end); lib/db.ts hardcoded ssl:"require",
making any non-TLS Postgres (incl. compose.dev.yaml's own local-dev
container) unreachable. After both fixes: full migration chain applies
cleanly, pg_dump/pg_restore round-trip is byte-identical (row counts +
content checksums), and real application code (lib/orders-store.ts,
SourceRegistryService's Postgres repository) reads the restored data back
correctly, including cross-tenant isolation still holding.
82 new tests (55 TS + 27 Python), all passing; full existing suite (291 TS +
55 Python) still green; typecheck clean on every changed/added file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
e7516697b3
commit
dc6aaa0e10
@@ -0,0 +1,73 @@
|
||||
/**
|
||||
* `src/auth/internal-token.ts` -- verification for the ONE machine-to-machine
|
||||
* credential 0pt already uses for engine (Python) -> edge (Next.js) calls:
|
||||
* the `X-Internal-Token` header, checked against the `INTERNAL_TOKEN` env var
|
||||
* (the same shared secret `lib/engine.ts` and `lib/health-checks.ts` already
|
||||
* send on the opposite-direction edge -> engine calls).
|
||||
*
|
||||
* P014 FIX: `services/api/workers/quality_bridge.py` has always called
|
||||
* `POST /api/0pt/quality/bridge-run` with `X-Internal-Token`, never an
|
||||
* `Authorization: Bearer` header. But this route sits under
|
||||
* `/api/0pt/:path*`, which `middleware.ts` guards unconditionally with real
|
||||
* Zitadel Bearer verification -- and nothing anywhere checked
|
||||
* `X-Internal-Token` as an alternative credential. The result: in any
|
||||
* environment with the default `AUTH_ENFORCE` posture (enforced), the
|
||||
* Python engine's ONLY call into this bridge was rejected with 401 by
|
||||
* middleware before the route handler ever ran. `OPT_QUALITY_BRIDGE_ENABLED=true`
|
||||
* therefore could never actually succeed against a real deployment --
|
||||
* quality_bridge.py's fail-open design (see its own docstring) silently
|
||||
* absorbed the 401 as `status: "unavailable"`, so nothing outwardly broke,
|
||||
* but the whole point of building the bridge (an advisory second opinion
|
||||
* from the TS QualityService) was unreachable. Neither test suite caught
|
||||
* this: the TS test (tests/p0/quality-bridge.test.ts) calls the route's
|
||||
* POST function directly, bypassing middleware.ts entirely; the Python test
|
||||
* (services/api/tests/test_quality_bridge.py) mocks the HTTP layer and never
|
||||
* exercises the edge's real auth gate.
|
||||
*
|
||||
* This module verifies `X-Internal-Token` in constant time. middleware.ts
|
||||
* accepts it ONLY for the one route the engine actually calls
|
||||
* (`/api/0pt/quality/bridge-run`) -- not a blanket bypass for the rest of
|
||||
* `/api/0pt/*`. Fails closed: an unset/empty `INTERNAL_TOKEN` env var never
|
||||
* authenticates, even against an empty or missing header (no accidental
|
||||
* "unset equals unset, so let it through").
|
||||
*
|
||||
* Uses only Web APIs (`TextEncoder`), not `node:crypto` -- Next.js
|
||||
* middleware runs on the Edge runtime, which does not expose Node's
|
||||
* `crypto.timingSafeEqual`.
|
||||
*/
|
||||
|
||||
/** Minimal header accessor -- Web `Headers` and `NextRequest.headers` satisfy this. */
|
||||
export interface HeaderLookup {
|
||||
get(name: string): string | null | undefined;
|
||||
}
|
||||
|
||||
export const INTERNAL_TOKEN_HEADER = "x-internal-token";
|
||||
|
||||
/** Constant-time-ish byte comparison (Edge-runtime safe: no node:crypto). */
|
||||
function timingSafeEqualStr(a: string, b: string): boolean {
|
||||
const enc = new TextEncoder();
|
||||
const aBytes = enc.encode(a);
|
||||
const bBytes = enc.encode(b);
|
||||
const len = Math.max(aBytes.length, bBytes.length);
|
||||
let diff = aBytes.length ^ bBytes.length;
|
||||
for (let i = 0; i < len; i++) {
|
||||
diff |= (aBytes[i] ?? 0) ^ (bBytes[i] ?? 0);
|
||||
}
|
||||
return diff === 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* True only when `X-Internal-Token` is present AND matches a non-empty
|
||||
* configured `INTERNAL_TOKEN`. Fails closed on every other combination
|
||||
* (missing header, missing/empty env var, mismatch).
|
||||
*/
|
||||
export function verifyInternalServiceToken(
|
||||
headers: HeaderLookup,
|
||||
env: { INTERNAL_TOKEN?: string } = process.env,
|
||||
): boolean {
|
||||
const expected = env.INTERNAL_TOKEN?.trim();
|
||||
if (!expected) return false;
|
||||
const provided = headers.get(INTERNAL_TOKEN_HEADER);
|
||||
if (!provided) return false;
|
||||
return timingSafeEqualStr(provided, expected);
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
/**
|
||||
* src/http/tenant-guard.ts
|
||||
* ==========================
|
||||
* P014 fix: every route under `app/api/0pt/**` sits behind `middleware.ts`,
|
||||
* which performs real Zitadel Bearer verification and — only when a
|
||||
* principal was actually verified — forwards the result as the
|
||||
* `x-0pt-verified-tenant` request header (see middleware.ts, src/auth/context.ts).
|
||||
*
|
||||
* Independently of that, every one of these route handlers reads `tenant_id`
|
||||
* from CLIENT-CONTROLLED input (a `?tenant_id=` query param or a JSON body
|
||||
* field) to decide which tenant's rows to read or write. Until this pass,
|
||||
* NONE of them cross-checked that client-supplied value against the verified
|
||||
* header — so a caller holding a valid token for tenant A could read or
|
||||
* write tenant B's sources, rights, provenance, health, quality rules/runs,
|
||||
* documents, or lakehouse mappings simply by naming `tenant_id=t_b` in the
|
||||
* request. `SourceRegistryService.assertSourceTenant` (and friends) only
|
||||
* checks that the object being read *belongs* to the tenant_id it was asked
|
||||
* to check against — it has no way to know whether the CALLER is actually
|
||||
* entitled to assert that tenant_id in the first place. That entitlement
|
||||
* check belongs at the HTTP boundary, once, here.
|
||||
*
|
||||
* This mirrors the exact decision `src/auth/context.ts`'s `resolveAuthContext`
|
||||
* already makes for the enforced-Bearer path ("a disagreeing X-Tenant-Id
|
||||
* header is a 403"), applied uniformly to every 0pt capability route that
|
||||
* takes tenant_id as a body/query parameter rather than relying on
|
||||
* middleware to inject ctx.tenantId directly (most of these routes predate
|
||||
* that plumbing and read tenant_id straight from the request; fixing that
|
||||
* more deeply is a larger refactor than this pass — this closes the actual
|
||||
* authz hole without changing every route's request contract).
|
||||
*/
|
||||
import { NextResponse, type NextRequest } from "next/server";
|
||||
|
||||
export const VERIFIED_TENANT_HEADER = "x-0pt-verified-tenant";
|
||||
|
||||
/**
|
||||
* Returns a 403 NextResponse when `tenantId` (whatever the route read it
|
||||
* from — query string or body) disagrees with the tenant middleware.ts
|
||||
* already verified for this request. Returns `null` when the caller may
|
||||
* proceed — either because the tenant matches, or because no verified
|
||||
* tenant is present on the request at all.
|
||||
*
|
||||
* The header is absent only in the explicit local-development escape hatch
|
||||
* (`AUTH_ENFORCE=false` with no bearer token presented) — `assertAuthConfigSafe()`
|
||||
* refuses to boot that way in production, so an absent header here can only
|
||||
* happen in a mode that already accepts fully unauthenticated requests.
|
||||
* Rejecting in that mode would be strictly more restrictive than the
|
||||
* existing dev escape hatch intends, so this deliberately no-ops rather than
|
||||
* invents a tenant to compare against.
|
||||
*/
|
||||
export function assertTenantMatchesVerified(
|
||||
request: NextRequest,
|
||||
tenantId: string | null | undefined,
|
||||
): NextResponse | null {
|
||||
const verified = request.headers.get(VERIFIED_TENANT_HEADER);
|
||||
if (!verified) return null;
|
||||
if (!tenantId) return null; // handlers already 400 on a missing tenant_id separately
|
||||
if (tenantId !== verified) {
|
||||
return NextResponse.json(
|
||||
{
|
||||
error: "OPT_ACCESS_DENIED",
|
||||
message: "tenant_id disagrees with the authenticated token's tenant",
|
||||
details: { requested_tenant: tenantId, verified_tenant: verified },
|
||||
},
|
||||
{ status: 403 },
|
||||
);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
@@ -125,9 +125,21 @@ export class DatasetIcebergMappingService {
|
||||
return this.toMapping(record);
|
||||
}
|
||||
|
||||
async getMapping(mappingId: string): Promise<DatasetIcebergMapping> {
|
||||
// P014: `tenantId` is optional so trusted in-process callers (existing
|
||||
// unit tests, and internal callers that already resolved the tenant some
|
||||
// other way) are unaffected -- but both HTTP routes that reach this method
|
||||
// now always pass it, closing a real gap: `getMapping` used to be
|
||||
// reachable by mapping ID alone with NO tenant check anywhere, so any
|
||||
// authenticated caller could read -- or, via requestIcebergWrite below,
|
||||
// WRITE a new snapshot onto -- another tenant's dataset<->Iceberg-table
|
||||
// mapping just by guessing/enumerating dataset_iceberg_mapping_id. A
|
||||
// mismatched tenantId reports identically to "not found" (no existence
|
||||
// oracle), matching SourceRegistryService.assertSourceTenant's convention.
|
||||
async getMapping(mappingId: string, tenantId?: string): Promise<DatasetIcebergMapping> {
|
||||
const record = this.mappings.get(mappingId);
|
||||
if (!record) throw new Error(`Dataset Iceberg mapping ${mappingId} not found`);
|
||||
if (!record || (tenantId && record.tenant_id !== tenantId)) {
|
||||
throw new Error(`Dataset Iceberg mapping ${mappingId} not found`);
|
||||
}
|
||||
return this.toMapping(record);
|
||||
}
|
||||
|
||||
|
||||
@@ -249,7 +249,22 @@ export class DocumentExtractionService {
|
||||
return record;
|
||||
}
|
||||
|
||||
async listExtractionsForDocument(documentId: string): Promise<DocumentExtractionRecord[]> {
|
||||
// P014: `tenantId` is optional so existing trusted in-process callers are
|
||||
// unaffected, but the HTTP route now always supplies it -- closing a real
|
||||
// gap: this used to be reachable by documentId alone with NO tenant
|
||||
// check anywhere, so any authenticated caller could list another
|
||||
// tenant's OCR/structuring extraction results (potentially containing
|
||||
// extracted document content, e.g. invoice fields) by guessing or
|
||||
// enumerating documentId. A mismatched tenantId reports identically to
|
||||
// "not found" (no existence oracle), same convention as the rest of the
|
||||
// surface.
|
||||
async listExtractionsForDocument(documentId: string, tenantId?: string): Promise<DocumentExtractionRecord[]> {
|
||||
if (tenantId) {
|
||||
const document = await this.captureService.getDocument(documentId).catch(() => undefined);
|
||||
if (!document || document.tenant_id !== tenantId) {
|
||||
throw new Error(`Document ${documentId} not found`);
|
||||
}
|
||||
}
|
||||
return Array.from(this.extractions.values()).filter((r) => r.document_id === documentId);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -65,6 +65,15 @@ export interface MapDatasetToIcebergTableResult {
|
||||
|
||||
export interface RequestIcebergWriteInput {
|
||||
dataset_iceberg_mapping_id: string;
|
||||
/**
|
||||
* P014: optional so existing trusted in-process callers are unaffected,
|
||||
* but the HTTP route (app/api/0pt/lakehouse/mappings/[mappingId]/writes)
|
||||
* always supplies it now. When present, it must match the mapping's own
|
||||
* tenant_id or the write is refused as not-found -- see getMapping's
|
||||
* doc comment for the gap this closes (a foreign mapping_id used to let
|
||||
* any caller register a snapshot against another tenant's table).
|
||||
*/
|
||||
tenant_id?: string;
|
||||
actor_id: string;
|
||||
correlation_id: string;
|
||||
snapshot_type?: TableSnapshotType;
|
||||
@@ -176,7 +185,7 @@ export class IcebergWriteService {
|
||||
}
|
||||
|
||||
async requestIcebergWrite(input: RequestIcebergWriteInput): Promise<RequestIcebergWriteResult> {
|
||||
const mapping = await this.mappingService.getMapping(input.dataset_iceberg_mapping_id);
|
||||
const mapping = await this.mappingService.getMapping(input.dataset_iceberg_mapping_id, input.tenant_id);
|
||||
if (mapping.mapping_status !== "active") {
|
||||
throw new Error(`Dataset Iceberg mapping ${input.dataset_iceberg_mapping_id} is not active (status=${mapping.mapping_status})`);
|
||||
}
|
||||
|
||||
@@ -103,9 +103,20 @@ export class SourceRatePolicyService {
|
||||
return this.toPolicy(record);
|
||||
}
|
||||
|
||||
async getSourceRatePolicy(sourceRatePolicyId: string): Promise<SourceRatePolicy> {
|
||||
// P014: `tenantId` is optional so existing internal callers that already
|
||||
// resolved a policy from a tenant-scoped list (or are trusted, in-process
|
||||
// callers) are unaffected — but the HTTP route now always passes it, which
|
||||
// closes a real gap: this lookup used to be reachable by rate-policy ID
|
||||
// alone with NO tenant check anywhere, so any authenticated caller could
|
||||
// read another tenant's crawl-budget/rate-limit/compliance-notes record
|
||||
// just by guessing or enumerating IDs. A mismatched tenantId is reported
|
||||
// identically to "not found" (no existence oracle), matching the
|
||||
// convention SourceRegistryService.assertSourceTenant already established.
|
||||
async getSourceRatePolicy(sourceRatePolicyId: string, tenantId?: string): Promise<SourceRatePolicy> {
|
||||
const record = this.policies.get(sourceRatePolicyId);
|
||||
if (!record) throw new Error(`Source rate policy ${sourceRatePolicyId} not found`);
|
||||
if (!record || (tenantId && record.tenant_id !== tenantId)) {
|
||||
throw new Error(`Source rate policy ${sourceRatePolicyId} not found`);
|
||||
}
|
||||
return this.toPolicy(record);
|
||||
}
|
||||
|
||||
|
||||
@@ -170,6 +170,13 @@ export const mapDatasetToIcebergTableSchema = z.object({
|
||||
|
||||
/** POST /api/0pt/lakehouse/mappings/[mappingId]/writes -- RequestIcebergWriteInput. */
|
||||
export const requestIcebergWriteSchema = z.object({
|
||||
// P014: required so the route can both (a) cross-check it against the
|
||||
// caller's verified tenant (assertTenantMatchesVerified) and (b) thread it
|
||||
// into IcebergWriteService.requestIcebergWrite -> getMapping, which
|
||||
// refuses the write as not-found when it disagrees with the mapping's own
|
||||
// tenant_id. See getMapping's doc comment for the cross-tenant write gap
|
||||
// this closes.
|
||||
tenant_id: z.string().min(1),
|
||||
actor_id: z.string().min(1),
|
||||
correlation_id: z.string().min(1),
|
||||
snapshot_type: z.string().min(1).optional(),
|
||||
|
||||
Reference in New Issue
Block a user