Files
0pt/services/api/sources/connectors.py
T
Simon AxelssonandClaude Sonnet 5 9740e20818
CI (SIAX Cloud) / security (push) Successful in 23s
Security Scan (SIAX Cloud) / security (push) Successful in 22s
CI (SIAX Cloud) / integration (push) Successful in 29s
CI (SIAX Cloud) / quality (push) Successful in 45s
CI (SIAX Cloud) / security (pull_request) Successful in 27s
CI (SIAX Cloud) / integration (pull_request) Successful in 31s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 29s
CI (SIAX Cloud) / quality (pull_request) Successful in 48s
feat(data-os): TS/Python quality bridge + provenance fields + fault-injection tests
Continues P010 (branch renamed from feat/p010-source-registry-persistence):
closes the three remaining BEFORE_STATE.md gaps (§2 TS/Python split-brain,
§3 missing provenance fields, §4 untested failure modes), all additive to
the production pipeline — the Python QA filter's delivery decision is
unchanged by any of this.

TS/Python bridge (§2):
  - New route surface app/api/0pt/quality/* (rules/runs/issues CRUD +
    POST /api/0pt/quality/bridge-run) over the shared QualityService
    singleton (src/services/quality.ts), fixing the same "no shared
    instance" defect the Source Registry had.
  - src/quality/quality-bridge.ts auto-provisions a deterministic,
    tenant-scoped baseline rule set on first use so the Python engine never
    pre-registers rules through the CRUD surface.
  - services/api/workers/quality_bridge.py: the Python-side HTTP client.
    Feature-flagged off by default (OPT_QUALITY_BRIDGE_ENABLED), fails open
    on any error (never raises into run_job, never changes `delivered`).
    Advisory only — real per-row gating against the TS engine isn't
    implementable yet (ExpectationEngine returns dataset-level pass/fail
    counts + samples, not a per-row boolean array); documented, not faked.
  - Live-verified end-to-end: `next dev` (AUTH_ENFORCE=false) + the real
    Python pipeline hitting the real brreg.no API, round-tripping through
    the real bridge route, receiving a real QualityService score.

Provenance (§3): sources/connectors.py's Provenance dataclass gains
retrieved_at, transform, connector_version, producer, quality,
entity_resolution, artifact_ref (all previously absent from the real
dataflow entirely) via a make_provenance() factory used by every connector.
resolve.py stamps entity_resolution on merged records' provenance (never on
unmerged ones). pipeline.py stamps artifact_ref at delivery and quality
once the bridge succeeds.

Failure tests (§4), real fault injection against real code (services/api,
new pytest + respx suite, not vitest mocks):
  - dedup-determinism + duplicate-input + out-of-order: resolve.py under
    shuffled/interleaved/exhaustive-permutation input, plus an explicit,
    verified test of the one case that IS legitimately order-sensitive
    (non-transitive similarity in a greedy clusterer) instead of hiding it.
  - schema-evolution: respx-mocked upstream responses with added/removed/
    alien-shaped fields through the real brreg connector + pipeline.
  - restart: connector exceptions mid-job (including one from respx itself)
    never crash run_job.
  - backfill: the real property that holds given no persistence layer
    exists (replay is compute-idempotent) — NOT a fabricated dedup-store
    test.
  - CDC-interruption and true backfill/delivery-store idempotency:
    explicitly documented as not implementable against this codebase
    (no CDC pipeline, no delivery store exist anywhere) via skip markers
    with reasons, in tests/test_known_gaps.py, rather than silently
    dropped or faked.

Testing: 256/256 vitest (249 pre-existing + 7 new bridge route/service
tests) and 28/28 pytest (2 documented skips) green locally. tsc -p
tsconfig.check.json has zero new errors (14 pre-existing errors, all in
the untracked/unaudited src/fleet, src/graph, src/metrics, src/projections
paths this pass does not touch, per BEFORE_STATE.md).

No PR opened, per instructions.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-01 21:02:14 +02:00

278 lines
11 KiB
Python

"""
0pt data engine — COMPLIANT source connectors.
Legal sourcing only (CLAUDE.md §1, docs/06). Priority per field/market:
1. official API / bulk 2. authorized public web (robots + rate limits)
3. licensed partner data 4. otherwise -> level 0 "ej tillgänglig / licens".
EXPLICITLY NOT PRESENT (and must never be added here):
WAF/bot bypass, FlareSolverr, stealth/undetected browsers, proxy/fingerprint
rotation to evade bans, scraping behind login walls, fake-account creation.
Every connector returns a normalized company dict + per-field provenance rows
so nothing ships unsourced. `Provenance` (below) carries the full field set
BEFORE_STATE.md §3 found missing from the actual dataflow — this repo's
5-field bridge_provenance was: source, confidence, level, and a DATE-only
freshness; the concept "bridge_provenance" as a named thing did not exist
anywhere in the repo at all. This adds, per field: retrieval TIMESTAMP
(distinct from the legacy date-only `freshness`), the transform applied,
connector/schema version, the producing system, and two slots
(`quality`/`entity_resolution`) that downstream stages (quality bridge,
resolve.py) fill in as the record moves through the pipeline — they start
`None` here and are never backfilled with a guess.
Terms change — VERIFY each source's current access/licence before relying.
"""
from __future__ import annotations
import os
from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Any
import httpx
# A polite client that identifies itself and obeys timeouts/rate limits.
USER_AGENT = "0pt-data-engine/0.1 (+https://0pt.app; SIAX Technology AB; hej@siax.io)"
_TIMEOUT = httpx.Timeout(15.0, connect=10.0)
# Bumped when a connector's parsing/field-mapping logic changes in a way that
# could shift output shape — recorded per-field as `Provenance.connector_version`
# so a downstream consumer can tell which parsing logic produced a value.
CONNECTOR_VERSIONS: dict[str, str] = {
"gleif": "gleif-connector@1",
"brreg": "brreg-connector@1",
"companies_house": "companies-house-connector@1",
"prh": "prh-connector@1",
"cvr": "cvr-connector@1",
"rdw": "rdw-connector@1",
}
PRODUCER = "0pt-engine"
def _client() -> httpx.AsyncClient:
return httpx.AsyncClient(
headers={"User-Agent": USER_AGENT, "Accept": "application/json"},
timeout=_TIMEOUT,
follow_redirects=True,
)
@dataclass
class Provenance:
field_name: str
source_id: str
confidence: float
level: int # 0 unavailable/licensed, 1 single, 2 partial, 3 corroborated
freshness: str | None = None # legacy: retrieval DATE only (kept for back-compat)
retrieved_at: str | None = None # full ISO-8601 UTC timestamp of the actual fetch
transform: str | None = None # e.g. "raw_fetch", "normalized", "entity_resolution_merge"
connector_version: str | None = None # e.g. "brreg-connector@1" — see CONNECTOR_VERSIONS
producer: str = PRODUCER # which system produced this field's value
quality: dict[str, Any] | None = None # filled by the quality bridge, if/when it runs (never guessed)
entity_resolution: dict[str, Any] | None = None # filled by resolve.py on merge, never at fetch time
artifact_ref: str | None = None # delivery artifact reference (ST0RE), set at delivery time — not yet wired
def make_provenance(
field_name: str,
source_id: str,
confidence: float,
level: int,
*,
transform: str = "raw_fetch",
) -> Provenance:
"""Factory used by every connector below so the new provenance fields
(retrieved_at, connector_version, producer) are populated consistently
instead of each call site re-deriving them ad hoc."""
now = datetime.now(timezone.utc)
return Provenance(
field_name=field_name,
source_id=source_id,
confidence=confidence,
level=level,
freshness=now.date().isoformat(),
retrieved_at=now.isoformat(),
transform=transform,
connector_version=CONNECTOR_VERSIONS.get(source_id),
producer=PRODUCER,
)
@dataclass
class CompanyResult:
org_nr: str | None = None
market: str | None = None
lei: str | None = None
name: str | None = None
status: str | None = None
address: dict[str, Any] | None = None
website: str | None = None
provenance: list[Provenance] = field(default_factory=list)
source: str = ""
# ── GLEIF / LEI (global cross-border key) — open API ────────────────────────
async def gleif_by_lei(lei: str) -> CompanyResult | None:
url = f"https://api.gleif.org/api/v1/lei-records/{lei}"
async with _client() as c:
r = await c.get(url, headers={"Accept": "application/vnd.api+json"})
if r.status_code != 200:
return None
a = (r.json().get("data") or {}).get("attributes") or {}
entity = a.get("entity") or {}
legal = entity.get("legalAddress") or {}
res = CompanyResult(
lei=lei,
name=(entity.get("legalName") or {}).get("name"),
status=entity.get("status"),
address={"lines": legal.get("addressLines"), "city": legal.get("city"),
"country": legal.get("country"), "postal": legal.get("postalCode")},
source="gleif",
)
res.provenance += [
make_provenance("lei", "gleif", 0.99, 3),
make_provenance("name", "gleif", 0.97, 3),
]
return res
# ── Brønnøysundregistrene (NO) — open API (NLOD). VERIFY terms. ─────────────
async def brreg_by_orgnr(orgnr: str) -> CompanyResult | None:
url = f"https://data.brreg.no/enhetsregisteret/api/enheter/{orgnr}"
async with _client() as c:
r = await c.get(url)
if r.status_code != 200:
return None
d = r.json()
res = CompanyResult(
org_nr=orgnr, market="NO",
name=d.get("navn"),
status="dissolved" if d.get("konkurs") else "active",
address={"raw": d.get("forretningsadresse")},
source="brreg",
)
res.provenance += [
make_provenance("name", "brreg", 0.97, 3),
make_provenance("status", "brreg", 0.95, 3),
make_provenance("address", "brreg", 0.95, 3),
]
return res
# ── Companies House (UK) — free API, key required. VERIFY terms. ────────────
async def companies_house_by_number(number: str) -> CompanyResult | None:
key = os.getenv("COMPANIES_HOUSE_API_KEY")
if not key:
# No key -> deliver honestly as unavailable, never fabricate.
return CompanyResult(org_nr=number, market="GB", source="companies_house",
provenance=[make_provenance("name", "companies_house", 0.0, 0, transform="unavailable_no_key")])
url = f"https://api.company-information.service.gov.uk/company/{number}"
async with _client() as c:
r = await c.get(url, auth=(key, ""))
if r.status_code != 200:
return None
d = r.json()
res = CompanyResult(
org_nr=number, market="GB",
name=d.get("company_name"),
status=d.get("company_status"),
address={"raw": d.get("registered_office_address")},
source="companies_house",
)
res.provenance += [
make_provenance("name", "companies_house", 0.97, 3),
make_provenance("status", "companies_house", 0.95, 3),
]
return res
# ── PRH avoindata (FI) — open API (CC-BY). VERIFY terms. ────────────────────
async def prh_by_businessid(business_id: str) -> CompanyResult | None:
url = f"https://avoindata.prh.fi/opendata-ytj-api/v3/companies?businessId={business_id}"
async with _client() as c:
r = await c.get(url)
if r.status_code != 200:
return None
items = r.json().get("companies") or r.json().get("results") or []
if not items:
return None
d = items[0]
names = d.get("names") or []
name = names[0].get("name") if names else d.get("name")
res = CompanyResult(org_nr=business_id, market="FI", name=name, status=d.get("status"), source="prh")
if name:
res.provenance.append(make_provenance("name", "prh", 0.95, 3))
return res
# ── CVR (DK) via cvrapi.dk — free for low volume (descriptive UA). VERIFY. ───
async def cvr_by_number(cvr: str) -> CompanyResult | None:
url = f"https://cvrapi.dk/api?search={cvr}&country=dk"
async with _client() as c:
r = await c.get(url)
if r.status_code != 200:
return None
d = r.json()
if not d.get("name"):
return None
res = CompanyResult(
org_nr=cvr, market="DK", name=d.get("name"),
status="ceased" if d.get("enddate") else "active",
address={"raw": d.get("address")}, source="cvr",
)
res.provenance += [make_provenance("name", "cvr", 0.93, 3), make_provenance("address", "cvr", 0.9, 2)]
return res
# ── RDW (NL) open vehicle data — Socrata. Technical/fleet only, no owner PII. ─
async def rdw_vehicle(kenteken: str) -> dict[str, Any] | None:
plate = kenteken.upper().replace("-", "").replace(" ", "")
url = f"https://opendata.rdw.nl/resource/m9d7-ebf2.json?kenteken={plate}"
async with _client() as c:
r = await c.get(url)
if r.status_code != 200:
return None
arr = r.json()
if not arr:
return None
d = arr[0]
p = make_provenance("make", "rdw", 0.9, 3)
return {
"reg_plate": kenteken, "market": "NL",
"make": d.get("merk"), "model": d.get("handelsbenaming"),
"year": (d.get("datum_eerste_toelating") or "")[:4] or None,
"source": "rdw",
# Dict shape (not the Provenance dataclass) — this endpoint predates it
# and is returned directly as an API response body, not merged through
# resolve.py. Field names kept aligned with Provenance's for consistency.
"provenance": [{
"field": p.field_name, "source_id": p.source_id, "confidence": p.confidence, "level": p.level,
"freshness": p.freshness, "retrieved_at": p.retrieved_at, "transform": p.transform,
"connector_version": p.connector_version, "producer": p.producer,
}],
}
# ── OpenSanctions screening — search API. VERIFY commercial terms / key. ─────
async def opensanctions_screen(name: str) -> dict[str, Any]:
url = "https://api.opensanctions.org/search/default"
async with _client() as c:
r = await c.get(url, params={"q": name, "limit": 5})
if r.status_code != 200:
return {"matches": [], "source": "opensanctions"}
results = r.json().get("results", [])
return {
"source": "opensanctions",
"matches": [{"name": x.get("caption"), "score": x.get("score"), "schema": x.get("schema")} for x in results],
}
CONNECTORS = {
"gleif": gleif_by_lei,
"brreg": brreg_by_orgnr,
"companies_house": companies_house_by_number,
"prh": prh_by_businessid,
"cvr": cvr_by_number,
}