bridge_provenance answers which source a field came from; it does not answer what happened on the wire. 1p (the SIAX egress measurement point) adds the robots decision + sha256 of the robots.txt it was read from, body_sha256 per fetch, and blocked_soft detection — HTTP 200 whose body is a block page, which otherwise gets scored as corroborating data. Consistent with CLAUDE.md §1: a measurement point, not a bypass. Documentation only; not in use, no worker routes through it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
102 lines
7.3 KiB
Markdown
102 lines
7.3 KiB
Markdown
# 06 · Data engine
|
||
|
||
The engine turns sources into delivered, provenance-tagged data. It is the legal motor — the whole product positioning depends on it. Build on existing `src/workers/` (`adaptive_router`, `browser_worker`, `ocr_worker`, `db`) but enforce the sourcing boundary below.
|
||
|
||
## Sourcing chain (strict priority — this is the law)
|
||
|
||
For each field × market, acquire in this order; stop at the first that works:
|
||
|
||
1. **Official API / bulk** — registries with programmatic/bulk access (highest confidence).
|
||
2. **Authorized public web** — clearly public pages, **respecting `robots.txt` and rate limits**, no auth walls.
|
||
3. **Licensed partner data** — contractually obtained.
|
||
4. **Otherwise → "not available / licensed only"** — delivered honestly as level 0. **Never guessed.**
|
||
|
||
### Prohibited (do not build, keep, or extend)
|
||
- WAF/bot-detection bypass — **including FlareSolverr**, "stealth"/patched browsers, undetected-* used to defeat detection.
|
||
- Proxy/fingerprint rotation to evade bans; scraping behind login walls; automated fake-account creation (Apollo etc.).
|
||
- **Reconciliation:** if `browser_worker.py` has an `HTTP → FlareSolverr → Playwright` chain, **drop the FlareSolverr stage** and keep only: polite HTTP fetch (identifies itself, obeys robots + rate limits) → optional Playwright for genuinely public JS-rendered pages (same politeness). The `adaptive_router` then chooses among **compliant** methods only (API > bulk > authorized fetch), not bypass tiers.
|
||
|
||
## Pipeline stages (per job)
|
||
|
||
```
|
||
sourcing → extraction/OCR → structuring → enrichment/fusion → QA
|
||
```
|
||
|
||
1. **Sourcing** — `adaptive_router` picks the compliant method per source; "source-racing" = query several *authorized* sources and take the fastest/freshest. Writes raw payloads + which source answered.
|
||
2. **Extraction / OCR** — API JSON parsed directly; documents (PDF/HTML/images) via **Unstructured** + OCR cascade (PaddleOCR → Tesseract → LLM fallback) in `ocr_worker`. Output: candidate fields.
|
||
3. **Structuring** — NER + entity resolution (spaCy / transformers, sv/no/da/fi/de/fr), normalize to schema, link entities across markets via **LEI** (GLEIF golden key). Dedup with **RapidFuzz**.
|
||
4. **Enrichment / fusion** — combine fields from multiple sources; compute **multi-source agreement** → confidence + level; append optional enrichment (NACE/SNI, sanctions/PEP flag from official lists, website/tech).
|
||
5. **QA** — quality checks (**Great Expectations** or equivalent, run inside the worker): validity (format/range), completeness, freshness, uniqueness (post-dedup), corroboration. Drop rows below the order's `confidence_target` (deliver fewer, never fabricate). Emit the quality report stored on `deliveries`.
|
||
|
||
Each stage updates `jobs` (`stage`, `pct`, `metrics` incl. per-market progress, running counts, climbing confidence) so the production tracker / SSE can render it live.
|
||
|
||
## Confidence & provenance scoring
|
||
|
||
- Per field, write a `bridge_provenance` row: `source_id`, `freshness`, `confidence` (0–1), `level` (0–3), `agreed_sources`.
|
||
- Level: 0 = unavailable/licensed; 1 = single source; 2 = partial corroboration; 3 = ≥2 independent sources agree.
|
||
- `est_confidence` on the order = aggregate target; delivered `avg_confidence` = actual.
|
||
|
||
### Network-layer provenance: `1p` (not yet in use)
|
||
|
||
`bridge_provenance` answers *which source a field came from*. It does not answer *what
|
||
actually happened on the wire* — which policy authorized the fetch, whether `robots.txt` was
|
||
read and what it said, or whether the bytes we parsed were the page or a bot wall served with
|
||
HTTP 200. **[`1p`](https://git.siax.io/sax3l/1p)**, the SIAX egress measurement point, closes
|
||
that gap and extends the provenance chain all the way down to the network layer.
|
||
|
||
1p owns four things: a per-destination policy as code (default DENY; rate, robots/ToS stance,
|
||
allowed methods), a single egress choke point, block detection, and one provenance record per
|
||
fetch written to AUD0. Everything else is composed (AUD0, ST0RE, CL0UD).
|
||
|
||
Why this matters for 0pt specifically — three facts that are hard to assert without it:
|
||
|
||
| Claim 0pt makes | What 1p adds |
|
||
|---|---|
|
||
| "acquired from an authorized public page, respecting `robots.txt`" | The record carries the robots decision **plus a sha256 of the `robots.txt` it was taken from**. A policy that says `respect` and cannot read robots denies the fetch — silence is not permission. |
|
||
| "these are the bytes we parsed" | `body_sha256` per fetch, so a delivered field can be tied back to the exact response at the exact time. |
|
||
| "level 1/2/3 corroboration from independent sources" | Responses are classified `ok` / `rate_limited` / `forbidden` / `captcha` / `blocked_soft` / `error`. `blocked_soft` is the expensive one: **HTTP 200 whose body is a block page**. Without it, a bot wall gets parsed and scored as corroborating data. |
|
||
|
||
**It is a measurement point, not a bypass.** This is consistent with, not an exception to,
|
||
the prohibitions above and in `CLAUDE.md` §1: 1p exists to *detect* blocking and *prove*
|
||
provenance, never to evade. Its purchased-residential egress pool is an interface with a stub
|
||
that throws — wiring it up would be a separate, explicit decision. Block detection also
|
||
alarms on a rising *trend*, never on single events, and deliberately excludes `error`
|
||
(their 5xx, our timeouts) from the blocking measure.
|
||
|
||
**Status: not in use.** 1p is Phase 0 — code, contracts, tests; not deployed, no credentials
|
||
set. No 0pt worker routes through it today, and `bridge_provenance` remains the only
|
||
provenance 0pt writes. Adopting it is a later, separate step.
|
||
|
||
## `sources.yaml` (machine-readable registry)
|
||
|
||
Single registry that drives the router, the `/sources` API, and the public sourcing-map page. Schema per entry:
|
||
|
||
```yaml
|
||
- source_id: gleif
|
||
name: GLEIF (LEI)
|
||
market: global
|
||
access: api # api | bulk | authorized_web | licensed
|
||
cadence: daily
|
||
licence: "open (CC0) — VERIFY current terms"
|
||
entities: [company]
|
||
fields: [lei, legal_name, address]
|
||
base_url: https://api.gleif.org
|
||
robots_ok: true # for authorized_web only; must be honored
|
||
rate_limit: "per provider terms"
|
||
notes: "cross-border key"
|
||
```
|
||
|
||
### Starter set (VERIFY current access/licence per source before relying — terms change)
|
||
- **GLEIF / LEI** — typically open; cross-border key. *(verify)*
|
||
- **Brønnøysund (Brreg, NO)** — typically open API. *(verify)*
|
||
- **Companies House (UK)** — free API (key required). *(verify)*
|
||
- **CVR / Virk (DK)** — API access with terms. *(verify)*
|
||
- **PRH (FI)** — varies. *(verify)*
|
||
- **Bolagsverket (SE)** — more restricted for bulk; plan licensed/partner route. *(verify)*
|
||
- Vehicle: **RDW (NL)** open-ish, **Traficom (FI)**, **Transportstyrelsen (SE)**, **DVLA (UK)** — *(verify each; vehicle/PII rules differ)*
|
||
|
||
> Do not hardcode access terms as fact. When building each connector, confirm the current API/licence/robots status and record it in `sources.yaml`. If unclear or disallowed for bulk, route to licensed/partner or mark fields level 0.
|
||
|
||
## Libraries used here
|
||
Crawlee or Scrapy (orchestration of **authorized** fetches) · Playwright (public JS pages, politely) · Unstructured (document parsing) · PaddleOCR/Tesseract (OCR) · spaCy / HF transformers (NER, multilingual) · Polars (fast tabular) · RapidFuzz (dedup/matching) · Great Expectations (quality) · dbt (optional transform layer). Versions/install: `docs/07-dependencies.md`.
|