0pt/docs/06-data-engine.md

06 · Data engine

The engine turns sources into delivered, provenance-tagged data. It is the legal motor — the whole product positioning depends on it. Build on existing src/workers/ (adaptive_router, browser_worker, ocr_worker, db) but enforce the sourcing boundary below.

Sourcing chain (strict priority — this is the law)

For each field × market, acquire in this order; stop at the first that works:

1. Official API / bulk — registries with programmatic/bulk access (highest confidence).
2. Authorized public web — clearly public pages, respecting robots.txt and rate limits, no auth walls.
3. Licensed partner data — contractually obtained.
4. Otherwise → "not available / licensed only" — delivered honestly as level 0. Never guessed.

Prohibited (do not build, keep, or extend)

• WAF/bot-detection bypass — including FlareSolverr, "stealth"/patched browsers, undetected-* used to defeat detection.
• Proxy/fingerprint rotation to evade bans; scraping behind login walls; automated fake-account creation (Apollo etc.).
• Reconciliation: if browser_worker.py has an HTTP → FlareSolverr → Playwright chain, drop the FlareSolverr stage and keep only: polite HTTP fetch (identifies itself, obeys robots + rate limits) → optional Playwright for genuinely public JS-rendered pages (same politeness). The adaptive_router then chooses among compliant methods only (API > bulk > authorized fetch), not bypass tiers.

Pipeline stages (per job)

sourcing → extraction/OCR → structuring → enrichment/fusion → QA

1. Sourcing — adaptive_router picks the compliant method per source; "source-racing" = query several authorized sources and take the fastest/freshest. Writes raw payloads + which source answered.
2. Extraction / OCR — API JSON parsed directly; documents (PDF/HTML/images) via Unstructured + OCR cascade (PaddleOCR → Tesseract → LLM fallback) in ocr_worker. Output: candidate fields.
3. Structuring — NER + entity resolution (spaCy / transformers, sv/no/da/fi/de/fr), normalize to schema, link entities across markets via LEI (GLEIF golden key). Dedup with RapidFuzz.
4. Enrichment / fusion — combine fields from multiple sources; compute multi-source agreement → confidence + level; append optional enrichment (NACE/SNI, sanctions/PEP flag from official lists, website/tech).
5. QA — quality checks (Great Expectations or equivalent, run inside the worker): validity (format/range), completeness, freshness, uniqueness (post-dedup), corroboration. Drop rows below the order's confidence_target (deliver fewer, never fabricate). Emit the quality report stored on deliveries.

Each stage updates jobs (stage, pct, metrics incl. per-market progress, running counts, climbing confidence) so the production tracker / SSE can render it live.

Confidence & provenance scoring

• Per field, write a bridge_provenance row: source_id, freshness, confidence (0–1), level (0–3), agreed_sources.
• Level: 0 = unavailable/licensed; 1 = single source; 2 = partial corroboration; 3 = ≥2 independent sources agree.
• est_confidence on the order = aggregate target; delivered avg_confidence = actual.

Network-layer provenance: 1p (not yet in use)

bridge_provenance answers which source a field came from. It does not answer what actually happened on the wire — which policy authorized the fetch, whether robots.txt was read and what it said, or whether the bytes we parsed were the page or a bot wall served with HTTP 200. 1p, the SIAX egress measurement point, closes that gap and extends the provenance chain all the way down to the network layer.

1p owns four things: a per-destination policy as code (default DENY; rate, robots/ToS stance, allowed methods), a single egress choke point, block detection, and one provenance record per fetch written to AUD0. Everything else is composed (AUD0, ST0RE, CL0UD).

Why this matters for 0pt specifically — three facts that are hard to assert without it:

Claim 0pt makes	What 1p adds
"acquired from an authorized public page, respecting robots.txt"	The record carries the robots decision plus a sha256 of the robots.txt it was taken from. A policy that says respect and cannot read robots denies the fetch — silence is not permission.
"these are the bytes we parsed"	body_sha256 per fetch, so a delivered field can be tied back to the exact response at the exact time.
"level 1/2/3 corroboration from independent sources"	Responses are classified ok / rate_limited / forbidden / captcha / blocked_soft / error. blocked_soft is the expensive one: HTTP 200 whose body is a block page. Without it, a bot wall gets parsed and scored as corroborating data.

It is a measurement point, not a bypass. This is consistent with, not an exception to, the prohibitions above and in CLAUDE.md §1: 1p exists to detect blocking and prove provenance, never to evade. Its purchased-residential egress pool is an interface with a stub that throws — wiring it up would be a separate, explicit decision. Block detection also alarms on a rising trend, never on single events, and deliberately excludes error (their 5xx, our timeouts) from the blocking measure.

Status: not in use. 1p is Phase 0 — code, contracts, tests; not deployed, no credentials set. No 0pt worker routes through it today, and bridge_provenance remains the only provenance 0pt writes. Adopting it is a later, separate step.

sources.yaml (machine-readable registry)

Single registry that drives the router, the /sources API, and the public sourcing-map page. Schema per entry:

- source_id: gleif
  name: GLEIF (LEI)
  market: global
  access: api            # api | bulk | authorized_web | licensed
  cadence: daily
  licence: "open (CC0) — VERIFY current terms"
  entities: [company]
  fields: [lei, legal_name, address]
  base_url: https://api.gleif.org
  robots_ok: true        # for authorized_web only; must be honored
  rate_limit: "per provider terms"
  notes: "cross-border key"

Starter set (VERIFY current access/licence per source before relying — terms change)

• GLEIF / LEI — typically open; cross-border key. (verify)
• Brønnøysund (Brreg, NO) — typically open API. (verify)
• Companies House (UK) — free API (key required). (verify)
• CVR / Virk (DK) — API access with terms. (verify)
• PRH (FI) — varies. (verify)
• Bolagsverket (SE) — more restricted for bulk; plan licensed/partner route. (verify)
• Vehicle: RDW (NL) open-ish, Traficom (FI), Transportstyrelsen (SE), DVLA (UK) — (verify each; vehicle/PII rules differ)

Do not hardcode access terms as fact. When building each connector, confirm the current API/licence/robots status and record it in sources.yaml. If unclear or disallowed for bulk, route to licensed/partner or mark fields level 0.

Libraries used here

Crawlee or Scrapy (orchestration of authorized fetches) · Playwright (public JS pages, politely) · Unstructured (document parsing) · PaddleOCR/Tesseract (OCR) · spaCy / HF transformers (NER, multilingual) · Polars (fast tabular) · RapidFuzz (dedup/matching) · Great Expectations (quality) · dbt (optional transform layer). Versions/install: docs/07-dependencies.md.