CI (SIAX Cloud) / quality (push) Successful in 51s
Security Scan (SIAX Cloud) / security (push) Successful in 26s
CI (SIAX Cloud) / security (push) Successful in 28s
CI (SIAX Cloud) / integration (push) Successful in 1m1s
CI (SIAX Cloud) / integration (pull_request) Successful in 30s
CI (SIAX Cloud) / security (pull_request) Successful in 24s
CI (SIAX Cloud) / quality (pull_request) Successful in 48s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 22s
Tenant isolation (security-critical): every app/api/0pt/** route reading
tenant_id from client-controlled query/body never cross-checked it against
the token middleware.ts already verified (x-0pt-verified-tenant). Adds a
shared assertTenantMatchesVerified guard and wires it into all 20+ affected
routes. Also closes two full-leak-by-omission bugs found in this pass
(documents/capture and lakehouse/mappings GETs returned every tenant's rows
when tenant_id was simply omitted) and two "no tenant check at all" reads
(rate-policies/[id], lakehouse/mappings/[id]) reachable by ID alone.
Duplicate-delivery idempotency (correctness-critical): POST /api/orders now
honors Idempotency-Key (sql/006, replay-or-409-on-conflict, Postgres +
in-memory); the Python orchestrator's POST /orders/{id}/run — the actual
delivery trigger — is now idempotent under an asyncio lock, closing a real
concurrent-request race, not just the sequential-retry case.
Engine-duplication resolution: adaptive_router.py's docstring wrongly
invited a bypass ("see b0ts for scraping") contradicting the product's own
compliance stance; corrected, and given real (additive, not escalating)
block-signal tracking so a rate-limited/blocked source is excluded from the
plan instead of hammered. Also fixes a previously-undiscovered production
bug: the Python engine's only call into the TS quality bridge
(X-Internal-Token) was unconditionally 401'd by middleware.ts, which never
checked that credential — the advisory quality bridge was unreachable in any
environment enforcing auth.
Restore drill: executed for real against throwaway Docker Postgres
containers (not simulated). Found and fixed two real blockers along the way:
sql/004 referenced opt_datasets, which no migration ever created (chain was
never actually runnable end to end); lib/db.ts hardcoded ssl:"require",
making any non-TLS Postgres (incl. compose.dev.yaml's own local-dev
container) unreachable. After both fixes: full migration chain applies
cleanly, pg_dump/pg_restore round-trip is byte-identical (row counts +
content checksums), and real application code (lib/orders-store.ts,
SourceRegistryService's Postgres repository) reads the restored data back
correctly, including cross-tenant isolation still holding.
82 new tests (55 TS + 27 Python), all passing; full existing suite (291 TS +
55 Python) still green; typecheck clean on every changed/added file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
105 lines
3.9 KiB
Python
105 lines
3.9 KiB
Python
"""
|
|
P014: every connector used by pipeline.py's adaptive plan (gleif, brreg,
|
|
companies_house, prh, cvr) used to collapse EVERY non-200 response into a
|
|
bare `None`, indistinguishable from "record genuinely doesn't exist". This
|
|
tests that the block-status subset (401/403/429/451) now raises
|
|
SourceBlockedError instead, while every other non-200 status (404, 500, ...)
|
|
still returns None exactly as before -- this is an ADDITIVE signal, not a
|
|
behavior change for the non-block case.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import httpx
|
|
import pytest
|
|
import respx
|
|
|
|
from sources.connectors import (
|
|
SourceBlockedError,
|
|
brreg_by_orgnr,
|
|
companies_house_by_number,
|
|
cvr_by_number,
|
|
gleif_by_lei,
|
|
prh_by_businessid,
|
|
)
|
|
|
|
GLEIF_URL = "https://api.gleif.org/api/v1/lei-records/5493001KJTIIGC8Y1R12"
|
|
BRREG_URL = "https://data.brreg.no/enhetsregisteret/api/enheter/923609016"
|
|
COMPANIES_HOUSE_URL = "https://api.company-information.service.gov.uk/company/00000006"
|
|
PRH_URL = "https://avoindata.prh.fi/opendata-ytj-api/v3/companies?businessId=2077474-0"
|
|
CVR_URL = "https://cvrapi.dk/api?search=10150817&country=dk"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
@respx.mock
|
|
@pytest.mark.parametrize("status", [401, 403, 429, 451])
|
|
async def test_gleif_raises_source_blocked_on_block_statuses(status):
|
|
respx.get(GLEIF_URL).mock(return_value=httpx.Response(status, json={}))
|
|
with pytest.raises(SourceBlockedError) as exc_info:
|
|
await gleif_by_lei("5493001KJTIIGC8Y1R12")
|
|
assert exc_info.value.source_id == "gleif"
|
|
assert exc_info.value.status_code == status
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
@respx.mock
|
|
async def test_gleif_a_404_still_returns_none_not_blocked(monkeypatch):
|
|
respx.get(GLEIF_URL).mock(return_value=httpx.Response(404, json={}))
|
|
result = await gleif_by_lei("5493001KJTIIGC8Y1R12")
|
|
assert result is None # unchanged pre-P014 behavior for a genuine "not found"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
@respx.mock
|
|
async def test_brreg_raises_source_blocked_on_429():
|
|
respx.get(BRREG_URL).mock(return_value=httpx.Response(429, json={}))
|
|
with pytest.raises(SourceBlockedError) as exc_info:
|
|
await brreg_by_orgnr("923609016")
|
|
assert exc_info.value.source_id == "brreg"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
@respx.mock
|
|
async def test_companies_house_raises_source_blocked_on_403(monkeypatch):
|
|
monkeypatch.setenv("COMPANIES_HOUSE_API_KEY", "test-key")
|
|
respx.get(COMPANIES_HOUSE_URL).mock(return_value=httpx.Response(403, json={}))
|
|
with pytest.raises(SourceBlockedError) as exc_info:
|
|
await companies_house_by_number("00000006")
|
|
assert exc_info.value.source_id == "companies_house"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_companies_house_missing_key_still_delivers_honest_level_0_not_a_block():
|
|
# No COMPANIES_HOUSE_API_KEY set -- this is a config gap, not a source
|
|
# block, and must keep returning the honest "unavailable" result it
|
|
# already did (never raise SourceBlockedError for a problem on OUR side).
|
|
result = await companies_house_by_number("00000006")
|
|
assert result is not None
|
|
assert result.provenance[0].level == 0
|
|
assert result.provenance[0].transform == "unavailable_no_key"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
@respx.mock
|
|
async def test_prh_raises_source_blocked_on_401():
|
|
respx.get(PRH_URL).mock(return_value=httpx.Response(401, json={}))
|
|
with pytest.raises(SourceBlockedError) as exc_info:
|
|
await prh_by_businessid("2077474-0")
|
|
assert exc_info.value.source_id == "prh"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
@respx.mock
|
|
async def test_cvr_raises_source_blocked_on_451():
|
|
respx.get(CVR_URL).mock(return_value=httpx.Response(451, json={}))
|
|
with pytest.raises(SourceBlockedError) as exc_info:
|
|
await cvr_by_number("10150817")
|
|
assert exc_info.value.source_id == "cvr"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
@respx.mock
|
|
async def test_cvr_a_500_still_returns_none_not_blocked():
|
|
respx.get(CVR_URL).mock(return_value=httpx.Response(500, json={}))
|
|
result = await cvr_by_number("10150817")
|
|
assert result is None
|