0pt/docs/06-data-engine.md

06 · Data engine

The engine turns sources into delivered, provenance-tagged data. It is the legal motor — the whole product positioning depends on it. The real code lives in services/api/ (FastAPI orchestrator + workers), not at the src/workers/ path this doc used to point to — the "Pipeline stages" section below states plainly, stage by stage, what's built vs. still spec.

Known issue — two diverged git remotes for this repo (unresolved, Simon's call)

This doc describes the compliant-only sourcing chain as the law. That law was briefly violated on one of this repo's two remotes, so anyone reading this doc needs to know both exist:

• github.com/sax3l/0pt (this session's origin) — the remote this documentation pass was verified against.
• git.siax.io/sax3l/0pt (Gitea) — a separate, diverged remote that shares origin history with GitHub for at least part of the codebase (services/api/workers/adaptive_router.py is byte-for-byte identical between GitHub's current main and Gitea's post-fix PR branch, so they were once the same commit lineage).

Gitea's main independently accumulated three commits on 2026-07-13 that added a real WAF-bypass / proxy-rotation / fake-identity scraper stack (services/scrapers/*) — exactly the category of code §"Prohibited" below and CLAUDE.md §1 forbid. Those commits never touched GitHub's main. The stack was since removed on Gitea via PR #1, but:

• Which remote is authoritative / deployed to 0pt.app is unknown — not resolved this session.
• Whether the two remotes are supposed to be kept in sync is unknown — not resolved this session.

This pass does not resolve either question. Simon: decide which remote is canonical, confirm the other has no lingering bypass code, and either merge or retire the loser before treating "the 0pt repo" as singular in any future doc pass.

Sourcing chain (strict priority — this is the law)

For each field × market, acquire in this order; stop at the first that works:

1. Official API / bulk — registries with programmatic/bulk access (highest confidence).
2. Authorized public web — clearly public pages, respecting robots.txt and rate limits, no auth walls.
3. Licensed partner data — contractually obtained.
4. Otherwise → "not available / licensed only" — delivered honestly as level 0. Never guessed.

Current implementation covers tier 1 only. All five working connectors in services/api/sources/connectors.py (gleif_by_lei, brreg_by_orgnr, companies_house_by_number, prh_by_businessid, cvr_by_number) call official JSON APIs over httpx, nothing else. There is no HTML-scraping/parsing library anywhere in services/api (no bs4/lxml/selectolax, no playwright, no scrapy/crawlee — checked against requirements.txt and the source tree), so tier 2 (authorized public web) has no code path yet. Tier 3 (licensed partner) also has no connector — sources.yaml's bolagsverket entry is access: licensed with no matching function in connectors.py; it correctly falls through to level 0 today. Both tiers remain target design, not built.

Prohibited (do not build, keep, or extend)

• WAF/bot-detection bypass — including FlareSolverr, "stealth"/patched browsers, undetected-* used to defeat detection.
• Proxy/fingerprint rotation to evade bans; scraping behind login walls; automated fake-account creation (Apollo etc.).
• Reconciliation note (stale — kept for history): this section used to say "if browser_worker.py has an HTTP → FlareSolverr → Playwright chain, drop the FlareSolverr stage." browser_worker.py does not exist anywhere in this repository (verified this pass — no file of that name on GitHub's main). There is nothing to reconcile in the current Python engine. The bypass code that did exist, briefly, lived in services/scrapers/* on the Gitea remote only (see the Known Issue callout above) and has since been removed there via PR #1. adaptive_router.py (below) has never contained a bypass tier on either remote — it chooses among compliant methods only (API > bulk > authorized fetch > licensed), by design.

Pipeline stages (per job)

sourcing → extraction/OCR → structuring → enrichment/fusion → QA

The stage names below are the target architecture. services/api/workers/pipeline.py::run_job is the one function that actually runs today, and it implements a much smaller slice of this than the stage list implies — see the "built vs. not built" status called out under each stage below.

1. Sourcing — adaptive_router.plan_for_market(market) returns a hardcoded, ordered list of compliant source_ids per market (see table below); pipeline.py tries each in order for a given key and stops at the first one that returns a result (break after first hit). This is sequential fallback, not concurrent racing — the phrase "source-racing = query several authorized sources and take the fastest/freshest" describes a target behavior that isn't implemented; today only one source is ever queried per key unless the first fails. Writes raw payload fields directly into the record dict (no separate raw-payload store).
2. Extraction / OCR — not built. There is no ocr_worker, no unstructured, no paddleocr/pytesseract anywhere in this repo (checked services/api/requirements.txt and the full source tree — none present). "Extraction" today is just each connector in connectors.py parsing the JSON response of its own API call into a CompanyResult. Document/PDF/image ingestion and the OCR cascade described here are spec only.
3. Structuring — partially built. services/api/workers/resolve.py::resolve() does real, deterministic entity resolution: block by (market, name[:1].upper()), then rapidfuzz.fuzz.token_sort_ratio(...) >= 92 to merge duplicates, bumping confidence by +0.1 per corroborating match (capped 0.99) and incrementing agreed_sources. That part is real and unit-testable. What's not built: no spaCy/transformers, no NER, no multilingual normalization (sv/no/da/fi/de/fr from the original text), and no explicit LEI-based cross-market entity linking — matching is name-string-similarity only, within the same market. spacy and transformers are not in requirements.txt.
4. Enrichment / fusion — partially built, not fully wired into the job. The confidence/agreement bump described above happens inside resolve(). Separately, opensanctions_screen() (sanctions/PEP) and rdw_vehicle() (NL vehicle data) are real, working connectors in connectors.py — but pipeline.py never imports or calls either one. They're only reachable as their own standalone endpoints (GET /screen, GET /vehicles/{plate} in orchestrator/main.py), not as an automatic enrichment step on an order's records. NACE/SNI classification and "website/tech" enrichment have no code anywhere.
5. QA — not built beyond a single threshold filter. The entire QA stage in pipeline.py is: delivered = [r for r in deduped if r["confidence"] >= threshold]. That's real and does drop below-threshold rows (deliver fewer, never fabricate — that principle holds). But there is no Great Expectations (not in requirements.txt, not imported anywhere), no separate validity/format, completeness, freshness, or post-dedup-uniqueness checks beyond the confidence number already computed in structuring, and no quality-report object emitted or stored anywhere — deliveries (the SQL table defined in sql/001_init.sql) is never written to by this service. This mirrors a confirmed, broader finding this session: 0pt's data-quality/QA layer was at one point documented elsewhere (TypeScript side, src/services/quality-check-service.ts etc.) as if built and tested, but those files do not exist in this repo's history at all (zero commits, any branch — see docs/07-dependencies.md note). docs/ICEBERG_WRITE_BOUNDARY.md, docs/11-lakehouse-write-plan.md, and docs/P0_QUERY_SOURCE_FLOW.md have all since been corrected — each now carries its own explicit "fabricated, not built" callout rather than asserting those files as real — and docs/P0_DATASET_QUALITY_FLOW.md documents this gap as target-spec-only. The Python engine independently confirms the same gap: no real quality-check enforcement exists in this codebase today, in either language.

"Each stage updates jobs ... so the production tracker / SSE can render it live" is not what happens. run_job runs synchronously start-to-finish and returns one final dict (stage: "qa", pct: 100, state: "done", metrics, records). orchestrator/main.py's POST /orders/{oid}/run awaits the whole thing and stores that one result in an in-memory _JOBS dict; GET /orders/{oid}/status just returns whatever's there, or a static {"stage": "queued", "pct": 0, "state": "queued"} placeholder before it's run. There is no per-stage incremental progress, no pct that climbs, and no SSE endpoint exists anywhere in orchestrator/main.py (grepped — none). Live progress rendering is target design, not current behavior.

adaptive_router — actual market coverage (verified against services/api/workers/adaptive_router.py)

MARKET_SOURCES currently has real entries for exactly five markets, each a short compliant list — not the "elaborate multi-source lists" an earlier version of this router may have implied:

Market	Ordered plan	Note
NO	brreg, gleif
GB	companies_house, gleif
FI	prh, gleif
DK	cvr, gleif
SE	gleif only	Bolagsverket bulk is licensed (see above); SE company core relies on GLEIF + a not-yet-built licensed/partner route
anything else	["gleif"]	fallback in plan_for_market()

ACCESS_RANK (api=0, bulk=1, authorized_web=2, licensed=3) is defined in the same file but is not actually used by plan_for_market() to sort anything — the lists above are hand-ordered constants already in the desired priority, and the rank dict is currently dead code (documentation-as-code, not logic).

Confidence & provenance scoring

• Per field, connectors.py's Provenance dataclass carries: field_name, source_id, confidence (0–1), level (0–3), freshness. agreed_sources is not a field on Provenance — it's tracked separately, at the merged-record level, by resolve() (match["agreed_sources"] = ... + 1). If a doc or downstream consumer expects agreed_sources inside each bridge_provenance row, that's a mismatch to reconcile against the actual dataclass shape, not a fact to assume.
• Level: 0 = unavailable/licensed; 1 = single source; 2 = partial corroboration; 3 = ≥2 independent sources agree.
• est_confidence on the order = aggregate target (computed in orchestrator/main.py::compute_estimate); delivered avg_confidence = actual, computed in pipeline.py as the mean confidence of delivered rows.
• None of this is persisted to Postgres by the current service. sql/001_init.sql defines the bridge_provenance table (plus dim_company, dim_source, fact_financials, deliveries, etc.), and asyncpg is a listed dependency — but nothing in services/api actually calls asyncpg or issues a query. DATABASE_URL is read exactly once, in orchestrator/main.py's /health handler, purely to report a boolean ("db": bool(os.getenv("DATABASE_URL"))). orchestrator/main.py stores orders and jobs in in-memory dicts (_ORDERS, _JOBS) that reset on every restart. The schema exists; the wiring from pipeline output → Postgres rows does not yet.

sources.yaml (machine-readable registry)

Single registry that drives the router, the /sources API, and the (still-to-be-built) public sourcing-map page. services/api/sources/sources.yaml currently has 8 entries: gleif, brreg, companies_house, prh, cvr, bolagsverket, rdw, opensanctions. Schema per entry (as actually used, confirmed against the file — no entities/fields mismatch found):

- source_id: gleif
  name: GLEIF (LEI)
  market: global
  access: api            # api | bulk | authorized_web | licensed
  cadence: daily
  licence: "open (CC0) — VERIFY current terms"
  entities: [company]
  fields: [lei, legal_name, address, status]
  base_url: https://api.gleif.org
  robots_ok: true        # for authorized_web only; must be honored
  notes: "cross-border key"

orchestrator/main.py loads this file once at import time (yaml.safe_load(...)) and serves it through GET /sources (id/name/market/access/licence/cadence only) and GET /sources/{market}. It is genuinely the live registry behind those two endpoints — that part of the doc's original claim holds.

Starter set (VERIFY current access/licence per source before relying — terms change)

• GLEIF / LEI — typically open; cross-border key. Connector: gleif_by_lei — real, working.
• Brønnøysund (Brreg, NO) — typically open API. Connector: brreg_by_orgnr — real, working.
• Companies House (UK) — free API (key required, COMPANIES_HOUSE_API_KEY). Connector: companies_house_by_number — real; degrades honestly to level-0 unavailable if the key is unset (does not fabricate).
• PRH (FI) — Connector: prh_by_businessid — real, working.
• CVR / Virk (DK) — connector actually calls cvrapi.dk (a third-party aggregator), not Virk's own API directly, despite the registry entry's base_url: https://datacvr.virk.dk. sources.yaml's base_url for cvr does not match what connectors.py::cvr_by_number actually calls — flag for reconciliation (either update base_url to cvrapi.dk or switch the connector to call Virk directly; not resolved this pass).
• Bolagsverket (SE) — restricted for bulk; sources.yaml marks it access: licensed; no connector exists for it in connectors.py. Consistent with the doc's own "plan licensed/partner route" framing — not a false claim, just confirming it's still unbuilt.
• RDW (NL) vehicle data — open, Socrata-backed. Connector: rdw_vehicle — real, working, but not called by the job pipeline (only reachable via GET /vehicles/{plate}).
• OpenSanctions — global sanctions/PEP screening. Connector: opensanctions_screen — real, working, same caveat: reachable only via GET /screen, not part of run_job.
• Vehicle beyond RDW (Traficom FI, Transportstyrelsen SE, DVLA UK) — no connectors exist yet; (verify each before building; vehicle/PII rules differ).

Do not hardcode access terms as fact. When building each connector, confirm the current API/licence/robots status and record it in sources.yaml. If unclear or disallowed for bulk, route to licensed/partner or mark fields level 0.

Structured financials (XBRL) — real but unwired

services/api/sources/financials.py::companies_house_filings() is a real, working function: given a Companies House number and COMPANIES_HOUSE_API_KEY, it fetches the accounts filing history and returns up to 10 filings. It is not imported or called anywhere else in the codebase (grepped services/api — only financials.py itself references the function name). The file's closing comment describes a further XBRL extraction step ("with arelle installed, download each accounts filing's iXBRL document and pull tagged facts...") that has no code at all — arelle is not in requirements.txt. Treat the whole file as a real but disconnected building block, not a wired pipeline stage.

Libraries used here

Target list only — see docs/07-dependencies.md for what's actually installed today vs. still to be added, with an explicit install-status callout per package. As of this pass, the only Python packages installed for this engine are fastapi, uvicorn, pydantic, httpx, asyncpg (unused), python-dotenv, pyyaml, rapidfuzz, tenacity — i.e. the API-only tier 1 sourcing + structuring-by-fuzzy-match slice described above, nothing from the extraction/OCR/NER/quality layers this section used to list as if already in place.