Files
0pt/services/api/tests/test_connectors_blocked.py
Simon AxelssonandClaude Sonnet 5 dc6aaa0e10
CI (SIAX Cloud) / quality (push) Successful in 51s
Security Scan (SIAX Cloud) / security (push) Successful in 26s
CI (SIAX Cloud) / security (push) Successful in 28s
CI (SIAX Cloud) / integration (push) Successful in 1m1s
CI (SIAX Cloud) / integration (pull_request) Successful in 30s
CI (SIAX Cloud) / security (pull_request) Successful in 24s
CI (SIAX Cloud) / quality (pull_request) Successful in 48s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 22s
fix(data-os): P014 — close tenant-isolation + idempotency gaps, execute real restore drill
Tenant isolation (security-critical): every app/api/0pt/** route reading
tenant_id from client-controlled query/body never cross-checked it against
the token middleware.ts already verified (x-0pt-verified-tenant). Adds a
shared assertTenantMatchesVerified guard and wires it into all 20+ affected
routes. Also closes two full-leak-by-omission bugs found in this pass
(documents/capture and lakehouse/mappings GETs returned every tenant's rows
when tenant_id was simply omitted) and two "no tenant check at all" reads
(rate-policies/[id], lakehouse/mappings/[id]) reachable by ID alone.

Duplicate-delivery idempotency (correctness-critical): POST /api/orders now
honors Idempotency-Key (sql/006, replay-or-409-on-conflict, Postgres +
in-memory); the Python orchestrator's POST /orders/{id}/run — the actual
delivery trigger — is now idempotent under an asyncio lock, closing a real
concurrent-request race, not just the sequential-retry case.

Engine-duplication resolution: adaptive_router.py's docstring wrongly
invited a bypass ("see b0ts for scraping") contradicting the product's own
compliance stance; corrected, and given real (additive, not escalating)
block-signal tracking so a rate-limited/blocked source is excluded from the
plan instead of hammered. Also fixes a previously-undiscovered production
bug: the Python engine's only call into the TS quality bridge
(X-Internal-Token) was unconditionally 401'd by middleware.ts, which never
checked that credential — the advisory quality bridge was unreachable in any
environment enforcing auth.

Restore drill: executed for real against throwaway Docker Postgres
containers (not simulated). Found and fixed two real blockers along the way:
sql/004 referenced opt_datasets, which no migration ever created (chain was
never actually runnable end to end); lib/db.ts hardcoded ssl:"require",
making any non-TLS Postgres (incl. compose.dev.yaml's own local-dev
container) unreachable. After both fixes: full migration chain applies
cleanly, pg_dump/pg_restore round-trip is byte-identical (row counts +
content checksums), and real application code (lib/orders-store.ts,
SourceRegistryService's Postgres repository) reads the restored data back
correctly, including cross-tenant isolation still holding.

82 new tests (55 TS + 27 Python), all passing; full existing suite (291 TS +
55 Python) still green; typecheck clean on every changed/added file.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-03 12:00:43 +02:00

105 lines
3.9 KiB
Python

"""
P014: every connector used by pipeline.py's adaptive plan (gleif, brreg,
companies_house, prh, cvr) used to collapse EVERY non-200 response into a
bare `None`, indistinguishable from "record genuinely doesn't exist". This
tests that the block-status subset (401/403/429/451) now raises
SourceBlockedError instead, while every other non-200 status (404, 500, ...)
still returns None exactly as before -- this is an ADDITIVE signal, not a
behavior change for the non-block case.
"""
from __future__ import annotations
import httpx
import pytest
import respx
from sources.connectors import (
SourceBlockedError,
brreg_by_orgnr,
companies_house_by_number,
cvr_by_number,
gleif_by_lei,
prh_by_businessid,
)
GLEIF_URL = "https://api.gleif.org/api/v1/lei-records/5493001KJTIIGC8Y1R12"
BRREG_URL = "https://data.brreg.no/enhetsregisteret/api/enheter/923609016"
COMPANIES_HOUSE_URL = "https://api.company-information.service.gov.uk/company/00000006"
PRH_URL = "https://avoindata.prh.fi/opendata-ytj-api/v3/companies?businessId=2077474-0"
CVR_URL = "https://cvrapi.dk/api?search=10150817&country=dk"
@pytest.mark.asyncio
@respx.mock
@pytest.mark.parametrize("status", [401, 403, 429, 451])
async def test_gleif_raises_source_blocked_on_block_statuses(status):
respx.get(GLEIF_URL).mock(return_value=httpx.Response(status, json={}))
with pytest.raises(SourceBlockedError) as exc_info:
await gleif_by_lei("5493001KJTIIGC8Y1R12")
assert exc_info.value.source_id == "gleif"
assert exc_info.value.status_code == status
@pytest.mark.asyncio
@respx.mock
async def test_gleif_a_404_still_returns_none_not_blocked(monkeypatch):
respx.get(GLEIF_URL).mock(return_value=httpx.Response(404, json={}))
result = await gleif_by_lei("5493001KJTIIGC8Y1R12")
assert result is None # unchanged pre-P014 behavior for a genuine "not found"
@pytest.mark.asyncio
@respx.mock
async def test_brreg_raises_source_blocked_on_429():
respx.get(BRREG_URL).mock(return_value=httpx.Response(429, json={}))
with pytest.raises(SourceBlockedError) as exc_info:
await brreg_by_orgnr("923609016")
assert exc_info.value.source_id == "brreg"
@pytest.mark.asyncio
@respx.mock
async def test_companies_house_raises_source_blocked_on_403(monkeypatch):
monkeypatch.setenv("COMPANIES_HOUSE_API_KEY", "test-key")
respx.get(COMPANIES_HOUSE_URL).mock(return_value=httpx.Response(403, json={}))
with pytest.raises(SourceBlockedError) as exc_info:
await companies_house_by_number("00000006")
assert exc_info.value.source_id == "companies_house"
@pytest.mark.asyncio
async def test_companies_house_missing_key_still_delivers_honest_level_0_not_a_block():
# No COMPANIES_HOUSE_API_KEY set -- this is a config gap, not a source
# block, and must keep returning the honest "unavailable" result it
# already did (never raise SourceBlockedError for a problem on OUR side).
result = await companies_house_by_number("00000006")
assert result is not None
assert result.provenance[0].level == 0
assert result.provenance[0].transform == "unavailable_no_key"
@pytest.mark.asyncio
@respx.mock
async def test_prh_raises_source_blocked_on_401():
respx.get(PRH_URL).mock(return_value=httpx.Response(401, json={}))
with pytest.raises(SourceBlockedError) as exc_info:
await prh_by_businessid("2077474-0")
assert exc_info.value.source_id == "prh"
@pytest.mark.asyncio
@respx.mock
async def test_cvr_raises_source_blocked_on_451():
respx.get(CVR_URL).mock(return_value=httpx.Response(451, json={}))
with pytest.raises(SourceBlockedError) as exc_info:
await cvr_by_number("10150817")
assert exc_info.value.source_id == "cvr"
@pytest.mark.asyncio
@respx.mock
async def test_cvr_a_500_still_returns_none_not_blocked():
respx.get(CVR_URL).mock(return_value=httpx.Response(500, json={}))
result = await cvr_by_number("10150817")
assert result is None