fix(data-os): P014 — close tenant-isolation + idempotency gaps, execute real restore drill
CI (SIAX Cloud) / quality (push) Successful in 51s
Security Scan (SIAX Cloud) / security (push) Successful in 26s
CI (SIAX Cloud) / security (push) Successful in 28s
CI (SIAX Cloud) / integration (push) Successful in 1m1s
CI (SIAX Cloud) / integration (pull_request) Successful in 30s
CI (SIAX Cloud) / security (pull_request) Successful in 24s
CI (SIAX Cloud) / quality (pull_request) Successful in 48s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 22s

Tenant isolation (security-critical): every app/api/0pt/** route reading
tenant_id from client-controlled query/body never cross-checked it against
the token middleware.ts already verified (x-0pt-verified-tenant). Adds a
shared assertTenantMatchesVerified guard and wires it into all 20+ affected
routes. Also closes two full-leak-by-omission bugs found in this pass
(documents/capture and lakehouse/mappings GETs returned every tenant's rows
when tenant_id was simply omitted) and two "no tenant check at all" reads
(rate-policies/[id], lakehouse/mappings/[id]) reachable by ID alone.

Duplicate-delivery idempotency (correctness-critical): POST /api/orders now
honors Idempotency-Key (sql/006, replay-or-409-on-conflict, Postgres +
in-memory); the Python orchestrator's POST /orders/{id}/run — the actual
delivery trigger — is now idempotent under an asyncio lock, closing a real
concurrent-request race, not just the sequential-retry case.

Engine-duplication resolution: adaptive_router.py's docstring wrongly
invited a bypass ("see b0ts for scraping") contradicting the product's own
compliance stance; corrected, and given real (additive, not escalating)
block-signal tracking so a rate-limited/blocked source is excluded from the
plan instead of hammered. Also fixes a previously-undiscovered production
bug: the Python engine's only call into the TS quality bridge
(X-Internal-Token) was unconditionally 401'd by middleware.ts, which never
checked that credential — the advisory quality bridge was unreachable in any
environment enforcing auth.

Restore drill: executed for real against throwaway Docker Postgres
containers (not simulated). Found and fixed two real blockers along the way:
sql/004 referenced opt_datasets, which no migration ever created (chain was
never actually runnable end to end); lib/db.ts hardcoded ssl:"require",
making any non-TLS Postgres (incl. compose.dev.yaml's own local-dev
container) unreachable. After both fixes: full migration chain applies
cleanly, pg_dump/pg_restore round-trip is byte-identical (row counts +
content checksums), and real application code (lib/orders-store.ts,
SourceRegistryService's Postgres repository) reads the restored data back
correctly, including cross-tenant isolation still holding.

82 new tests (55 TS + 27 Python), all passing; full existing suite (291 TS +
55 Python) still green; typecheck clean on every changed/added file.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Simon Axelsson
2026-09-03 12:00:43 +02:00
co-authored by Claude Sonnet 5
parent e7516697b3
commit dc6aaa0e10
50 changed files with 1820 additions and 52 deletions
+20 -4
View File
@@ -6,10 +6,26 @@
# not a copy of the full VPS stack (no NocoDB/Metabase/Caddy/worker here,
# those are production concerns, not local dev needs).
#
# UNVERIFIED (2026-08-11): written and reviewed, but not smoke-tested end to
# end in this pass -- no Docker daemon was available in the environment this
# was authored in. Verify with `docker compose -f compose.dev.yaml up` before
# relying on it.
# VERIFIED (P014, 2026-09-03): this file's image/credentials/init-mount shape
# was exercised for real (an equivalent standalone container -- this sandbox
# had an unrelated service already bound to host port 5432, so the literal
# `docker compose -f compose.dev.yaml up` command itself could not be run
# here, but the identical image + `sql/*.sql` init sequence + connection
# shape were run and verified end to end as part of P014's restore drill).
# That drill surfaced and fixed the ACTUAL reason this was unverifiable
# before: `lib/db.ts`'s `getSql()` hardcoded `ssl: "require"`, which this
# plain-Postgres container can never satisfy (no TLS is configured here) --
# every connection attempt failed before a single query ran. Fixed: `ssl` now
# defaults to "require" (unchanged for every real deployment) but honors
# `DATABASE_SSL=false` to opt out. To use this container from the app or a
# script, set BOTH:
# DATABASE_URL=postgresql://opt:opt-dev-only-not-a-secret@localhost:5432/opt
# DATABASE_SSL=false
# `sql/003_datasets.sql` (also added by P014, closing a separate gap where
# `004_iceberg_write_boundary.sql` referenced a table no migration defined)
# is picked up by the `docker-entrypoint-initdb.d` mount automatically --
# Postgres runs `*.sql` in this directory in filename-sorted order, and the
# zero-padded 00N prefixes already sort correctly.
services:
db:
image: pgvector/pgvector:pg16