0pt/docs/06-data-engine.md

06 · Data engine

The engine turns sources into delivered, provenance-tagged data. It is the legal motor — the whole product positioning depends on it. Build on existing src/workers/ (adaptive_router, browser_worker, ocr_worker, db) but enforce the sourcing boundary below.

Sourcing chain (strict priority — this is the law)

For each field × market, acquire in this order; stop at the first that works:

1. Official API / bulk — registries with programmatic/bulk access (highest confidence).
2. Authorized public web — clearly public pages, respecting robots.txt and rate limits, no auth walls.
3. Licensed partner data — contractually obtained.
4. Otherwise → "not available / licensed only" — delivered honestly as level 0. Never guessed.

Prohibited (do not build, keep, or extend)

• WAF/bot-detection bypass — including FlareSolverr, "stealth"/patched browsers, undetected-* used to defeat detection.
• Proxy/fingerprint rotation to evade bans; scraping behind login walls; automated fake-account creation (Apollo etc.).
• Reconciliation: if browser_worker.py has an HTTP → FlareSolverr → Playwright chain, drop the FlareSolverr stage and keep only: polite HTTP fetch (identifies itself, obeys robots + rate limits) → optional Playwright for genuinely public JS-rendered pages (same politeness). The adaptive_router then chooses among compliant methods only (API > bulk > authorized fetch), not bypass tiers.

Pipeline stages (per job)

sourcing → extraction/OCR → structuring → enrichment/fusion → QA

1. Sourcing — adaptive_router picks the compliant method per source; "source-racing" = query several authorized sources and take the fastest/freshest. Writes raw payloads + which source answered.
2. Extraction / OCR — API JSON parsed directly; documents (PDF/HTML/images) via Unstructured + OCR cascade (PaddleOCR → Tesseract → LLM fallback) in ocr_worker. Output: candidate fields.
3. Structuring — NER + entity resolution (spaCy / transformers, sv/no/da/fi/de/fr), normalize to schema, link entities across markets via LEI (GLEIF golden key). Dedup with RapidFuzz.
4. Enrichment / fusion — combine fields from multiple sources; compute multi-source agreement → confidence + level; append optional enrichment (NACE/SNI, sanctions/PEP flag from official lists, website/tech).
5. QA — quality checks (Great Expectations or equivalent, run inside the worker): validity (format/range), completeness, freshness, uniqueness (post-dedup), corroboration. Drop rows below the order's confidence_target (deliver fewer, never fabricate). Emit the quality report stored on deliveries.

Each stage updates jobs (stage, pct, metrics incl. per-market progress, running counts, climbing confidence) so the production tracker / SSE can render it live.

Confidence & provenance scoring

• Per field, write a bridge_provenance row: source_id, freshness, confidence (0–1), level (0–3), agreed_sources.
• Level: 0 = unavailable/licensed; 1 = single source; 2 = partial corroboration; 3 = ≥2 independent sources agree.
• est_confidence on the order = aggregate target; delivered avg_confidence = actual.

sources.yaml (machine-readable registry)

Single registry that drives the router, the /sources API, and the public sourcing-map page. Schema per entry:

- source_id: gleif
  name: GLEIF (LEI)
  market: global
  access: api            # api | bulk | authorized_web | licensed
  cadence: daily
  licence: "open (CC0) — VERIFY current terms"
  entities: [company]
  fields: [lei, legal_name, address]
  base_url: https://api.gleif.org
  robots_ok: true        # for authorized_web only; must be honored
  rate_limit: "per provider terms"
  notes: "cross-border key"

Starter set (VERIFY current access/licence per source before relying — terms change)

• GLEIF / LEI — typically open; cross-border key. (verify)
• Brønnøysund (Brreg, NO) — typically open API. (verify)
• Companies House (UK) — free API (key required). (verify)
• CVR / Virk (DK) — API access with terms. (verify)
• PRH (FI) — varies. (verify)
• Bolagsverket (SE) — more restricted for bulk; plan licensed/partner route. (verify)
• Vehicle: RDW (NL) open-ish, Traficom (FI), Transportstyrelsen (SE), DVLA (UK) — (verify each; vehicle/PII rules differ)

Do not hardcode access terms as fact. When building each connector, confirm the current API/licence/robots status and record it in sources.yaml. If unclear or disallowed for bulk, route to licensed/partner or mark fields level 0.

Libraries used here

Crawlee or Scrapy (orchestration of authorized fetches) · Playwright (public JS pages, politely) · Unstructured (document parsing) · PaddleOCR/Tesseract (OCR) · spaCy / HF transformers (NER, multilingual) · Polars (fast tabular) · RapidFuzz (dedup/matching) · Great Expectations (quality) · dbt (optional transform layer). Versions/install: docs/07-dependencies.md.