CI (SIAX Cloud) / security (push) Successful in 23s
Security Scan (SIAX Cloud) / security (push) Successful in 22s
CI (SIAX Cloud) / integration (push) Successful in 29s
CI (SIAX Cloud) / quality (push) Successful in 45s
CI (SIAX Cloud) / security (pull_request) Successful in 27s
CI (SIAX Cloud) / integration (pull_request) Successful in 31s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 29s
CI (SIAX Cloud) / quality (pull_request) Successful in 48s
Continues P010 (branch renamed from feat/p010-source-registry-persistence):
closes the three remaining BEFORE_STATE.md gaps (§2 TS/Python split-brain,
§3 missing provenance fields, §4 untested failure modes), all additive to
the production pipeline — the Python QA filter's delivery decision is
unchanged by any of this.
TS/Python bridge (§2):
- New route surface app/api/0pt/quality/* (rules/runs/issues CRUD +
POST /api/0pt/quality/bridge-run) over the shared QualityService
singleton (src/services/quality.ts), fixing the same "no shared
instance" defect the Source Registry had.
- src/quality/quality-bridge.ts auto-provisions a deterministic,
tenant-scoped baseline rule set on first use so the Python engine never
pre-registers rules through the CRUD surface.
- services/api/workers/quality_bridge.py: the Python-side HTTP client.
Feature-flagged off by default (OPT_QUALITY_BRIDGE_ENABLED), fails open
on any error (never raises into run_job, never changes `delivered`).
Advisory only — real per-row gating against the TS engine isn't
implementable yet (ExpectationEngine returns dataset-level pass/fail
counts + samples, not a per-row boolean array); documented, not faked.
- Live-verified end-to-end: `next dev` (AUTH_ENFORCE=false) + the real
Python pipeline hitting the real brreg.no API, round-tripping through
the real bridge route, receiving a real QualityService score.
Provenance (§3): sources/connectors.py's Provenance dataclass gains
retrieved_at, transform, connector_version, producer, quality,
entity_resolution, artifact_ref (all previously absent from the real
dataflow entirely) via a make_provenance() factory used by every connector.
resolve.py stamps entity_resolution on merged records' provenance (never on
unmerged ones). pipeline.py stamps artifact_ref at delivery and quality
once the bridge succeeds.
Failure tests (§4), real fault injection against real code (services/api,
new pytest + respx suite, not vitest mocks):
- dedup-determinism + duplicate-input + out-of-order: resolve.py under
shuffled/interleaved/exhaustive-permutation input, plus an explicit,
verified test of the one case that IS legitimately order-sensitive
(non-transitive similarity in a greedy clusterer) instead of hiding it.
- schema-evolution: respx-mocked upstream responses with added/removed/
alien-shaped fields through the real brreg connector + pipeline.
- restart: connector exceptions mid-job (including one from respx itself)
never crash run_job.
- backfill: the real property that holds given no persistence layer
exists (replay is compute-idempotent) — NOT a fabricated dedup-store
test.
- CDC-interruption and true backfill/delivery-store idempotency:
explicitly documented as not implementable against this codebase
(no CDC pipeline, no delivery store exist anywhere) via skip markers
with reasons, in tests/test_known_gaps.py, rather than silently
dropped or faked.
Testing: 256/256 vitest (249 pre-existing + 7 new bridge route/service
tests) and 28/28 pytest (2 documented skips) green locally. tsc -p
tsconfig.check.json has zero new errors (14 pre-existing errors, all in
the untracked/unaudited src/fleet, src/graph, src/metrics, src/projections
paths this pass does not touch, per BEFORE_STATE.md).
No PR opened, per instructions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
59 lines
2.4 KiB
Python
59 lines
2.4 KiB
Python
"""
|
|
Verifies the P010 provenance additions (BEFORE_STATE.md §3: source,
|
|
retrieval timestamp, transform, version, producer were absent from the real
|
|
dataflow) land correctly on every real connector's output — mocked HTTP,
|
|
real parsing code.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import httpx
|
|
import pytest
|
|
import respx
|
|
|
|
from sources.connectors import brreg_by_orgnr, make_provenance, CONNECTOR_VERSIONS
|
|
|
|
|
|
class TestMakeProvenanceFactory:
|
|
def test_populates_all_new_fields_honestly_and_leaves_the_rest_none(self):
|
|
p = make_provenance("name", "brreg", 0.97, 3)
|
|
assert p.field_name == "name"
|
|
assert p.source_id == "brreg"
|
|
assert p.retrieved_at is not None
|
|
assert p.freshness == p.retrieved_at[:10] # date prefix of the same instant
|
|
assert p.transform == "raw_fetch"
|
|
assert p.connector_version == CONNECTOR_VERSIONS["brreg"]
|
|
assert p.producer == "0pt-engine"
|
|
# Never guessed — these are filled by later pipeline stages, not at fetch time.
|
|
assert p.quality is None
|
|
assert p.entity_resolution is None
|
|
assert p.artifact_ref is None
|
|
|
|
def test_transform_override_is_honored(self):
|
|
p = make_provenance("name", "companies_house", 0.0, 0, transform="unavailable_no_key")
|
|
assert p.transform == "unavailable_no_key"
|
|
|
|
def test_unknown_source_id_has_no_fabricated_version(self):
|
|
p = make_provenance("x", "some_future_source", 0.5, 1)
|
|
assert p.connector_version is None # honestly unknown, not guessed
|
|
|
|
|
|
class TestBrregConnectorProvenance:
|
|
@pytest.mark.asyncio
|
|
@respx.mock
|
|
async def test_real_parsing_path_produces_fully_populated_provenance(self):
|
|
respx.get("https://data.brreg.no/enhetsregisteret/api/enheter/923609016").mock(
|
|
return_value=httpx.Response(200, json={
|
|
"navn": "Volvo Personvagnar Norge AS", "konkurs": False,
|
|
"forretningsadresse": {"adresse": ["Storgata 1"]},
|
|
})
|
|
)
|
|
res = await brreg_by_orgnr("923609016")
|
|
assert res is not None
|
|
assert len(res.provenance) == 3 # name, status, address
|
|
for p in res.provenance:
|
|
assert p.source_id == "brreg"
|
|
assert p.connector_version == "brreg-connector@1"
|
|
assert p.producer == "0pt-engine"
|
|
assert p.retrieved_at is not None
|
|
assert p.transform == "raw_fetch"
|