Files
0pt/docs/06-data-engine.md
T
adminandClaude Sonnet 5 2902a26bac docs: full accuracy overhaul, grounded claim-by-claim in real code (#4)
* docs: full accuracy overhaul, grounded claim-by-claim in real code

Every doc in this repo (CLAUDE.md, README.md, SECURITY.md, docs/*.md) was
audited and rewritten by a multi-pass process: independent rewrite per
cluster, adversarial re-verification of that rewrite against the actual
code, then a final cross-doc consistency pass — each stage re-checking
claims against the repo directly rather than trusting the previous stage.

Two corrections matter most:

1. docs/ICEBERG_WRITE_BOUNDARY.md, docs/11-lakehouse-write-plan.md, and
   docs/P0_QUERY_SOURCE_FLOW.md asserted src/services/quality-check-service.ts,
   quality-report-service.ts, src/data-platform/services/catalog-service.ts,
   and sql/002_p0_dataset_quality.sql/sql/003_*.sql as real, tested code.
   git log --all across every branch shows zero commits for any of these
   paths, ever. 0pt's data-quality/QA enforcement does not exist in code.
   Corrected everywhere, with a new docs/P0_DATASET_QUALITY_FLOW.md added
   as an explicit target-spec (not-yet-built) replacement.

2. github.com/sax3l/0pt (this repo) and git.siax.io/sax3l/0pt (Gitea) are
   two diverged remotes sharing part of the same history (services/api/
   workers/adaptive_router.py is byte-identical between them). Gitea's main
   independently accumulated a WAF-bypass/proxy-rotation scraper stack in
   three 2026-07-13 commits, since removed via a Gitea PR; GitHub's main
   never had it. Which remote is canonical/deployed is unresolved — flagged
   consistently across the corpus, not decided here.

Also newly and consistently documented: the tenant_id/project_id-read-from-
client-body gap on the P0 lakehouse/rate-policy/source-query routes (no
auth/session system exists anywhere in this repo yet); that every CL0UD/
N0D/AUD0/ST0RE platform client is real but mock-only (zero process.env
reads for any of their *_URL vars); that two P0 mutation routes
(lakehouse/mappings, sources/rate-policies) always fail over HTTP today
because their prerequisite entity can never be created via any route; a
schema bug in sql/004 (FK references to opt_datasets/opt_sources, which
don't exist); and that 0pt.app's live deployment was independently
confirmed (curled this pass) to be running this exact codebase with
DATABASE_URL/REDIS_URL unset in production.

New file: docs/12-edge-capability-api.md — the edge-native P0 TypeScript
capability contract (app/api/0pt/*, /api/screen, /api/ai/parse), previously
undocumented and distinct from docs/04-api.md's FastAPI-engine-only scope.

Verified before commit: tsc --noEmit clean; vitest run 88 passed, 1 todo,
0 failures (13 files) — docs-only change, no source touched.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* docs: reflect PR #5's lakehouse-mapping-registry + status-code fixes

Follow-up to keep docs/12-edge-capability-api.md accurate once both PRs
land: fix/0pt-full-bug-hunt (#5) fixed the shared-instance bug in the
lakehouse mapping routes and added proper status-code mapping to the
document extraction route. This doc was written against the pre-fix
state in the same pass that added it, so it needs the same correction.

Does not claim the lakehouse write path fully works now -- the deeper
dataset-registration gap (no route ever calls DatasetService.registerDataset)
is unchanged and still blocks it end-to-end; only the shared-instance half
of the bug is fixed.

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-22 18:49:01 +02:00

214 lines
16 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 06 · Data engine
The engine turns sources into delivered, provenance-tagged data. It is the legal motor — the whole product positioning depends on it. The real code lives in `services/api/` (FastAPI orchestrator + workers), **not** at the `src/workers/` path this doc used to point to — the "Pipeline stages" section below states plainly, stage by stage, what's built vs. still spec.
> ### Known issue — two diverged git remotes for this repo (unresolved, Simon's call)
> This doc describes the compliant-only sourcing chain as *the law*. That law was briefly violated on one
> of this repo's two remotes, so anyone reading this doc needs to know both exist:
> - `github.com/sax3l/0pt` (this session's `origin`) — the remote this documentation pass was verified against.
> - `git.siax.io/sax3l/0pt` (Gitea) — a **separate, diverged** remote that shares origin history with GitHub for
> at least part of the codebase (`services/api/workers/adaptive_router.py` is byte-for-byte identical between
> GitHub's current `main` and Gitea's post-fix PR branch, so they were once the same commit lineage).
>
> Gitea's `main` independently accumulated three commits on 2026-07-13 that added a real WAF-bypass /
> proxy-rotation / fake-identity scraper stack (`services/scrapers/*`) — exactly the category of code
> §"Prohibited" below and CLAUDE.md §1 forbid. Those commits never touched GitHub's `main`. The stack was
> since removed on Gitea via PR #1, but:
> - **Which remote is authoritative / deployed to `0pt.app` is unknown** — not resolved this session.
> - **Whether the two remotes are supposed to be kept in sync is unknown** — not resolved this session.
>
> This pass does not resolve either question. Simon: decide which remote is canonical, confirm the other has
> no lingering bypass code, and either merge or retire the loser before treating "the 0pt repo" as singular
> in any future doc pass.
## Sourcing chain (strict priority — this is the law)
For each field × market, acquire in this order; stop at the first that works:
1. **Official API / bulk** — registries with programmatic/bulk access (highest confidence).
2. **Authorized public web** — clearly public pages, **respecting `robots.txt` and rate limits**, no auth walls.
3. **Licensed partner data** — contractually obtained.
4. **Otherwise → "not available / licensed only"** — delivered honestly as level 0. **Never guessed.**
**Current implementation covers tier 1 only.** All five working connectors in `services/api/sources/connectors.py`
(`gleif_by_lei`, `brreg_by_orgnr`, `companies_house_by_number`, `prh_by_businessid`, `cvr_by_number`) call
official JSON APIs over `httpx`, nothing else. There is no HTML-scraping/parsing library anywhere in
`services/api` (no `bs4`/`lxml`/`selectolax`, no `playwright`, no `scrapy`/`crawlee` — checked against
`requirements.txt` and the source tree), so tier 2 (authorized public web) has no code path yet. Tier 3
(licensed partner) also has no connector — `sources.yaml`'s `bolagsverket` entry is `access: licensed` with
no matching function in `connectors.py`; it correctly falls through to level 0 today. Both tiers remain
target design, not built.
### Prohibited (do not build, keep, or extend)
- WAF/bot-detection bypass — **including FlareSolverr**, "stealth"/patched browsers, undetected-* used to defeat detection.
- Proxy/fingerprint rotation to evade bans; scraping behind login walls; automated fake-account creation (Apollo etc.).
- **Reconciliation note (stale — kept for history):** this section used to say "if `browser_worker.py` has an
`HTTP → FlareSolverr → Playwright` chain, drop the FlareSolverr stage." **`browser_worker.py` does not exist
anywhere in this repository** (verified this pass — no file of that name on GitHub's `main`). There is
nothing to reconcile in the current Python engine. The bypass code that *did* exist, briefly, lived in
`services/scrapers/*` on the Gitea remote only (see the Known Issue callout above) and has since been
removed there via PR #1. `adaptive_router.py` (below) has never contained a bypass tier on either remote —
it chooses among **compliant** methods only (API > bulk > authorized fetch > licensed), by design.
## Pipeline stages (per job)
```
sourcing → extraction/OCR → structuring → enrichment/fusion → QA
```
The stage names below are the target architecture. `services/api/workers/pipeline.py::run_job` is the one
function that actually runs today, and it implements a much smaller slice of this than the stage list
implies — see the "built vs. not built" status called out under each stage below.
1. **Sourcing** — `adaptive_router.plan_for_market(market)` returns a hardcoded, ordered list of compliant
`source_id`s per market (see table below); `pipeline.py` tries each in order for a given key and **stops at
the first one that returns a result** (`break` after first hit). This is sequential fallback, not
concurrent racing — the phrase "source-racing = query several authorized sources and take the
fastest/freshest" describes a target behavior that isn't implemented; today only one source is ever queried
per key unless the first fails. Writes raw payload fields directly into the record dict (no separate
raw-payload store).
2. **Extraction / OCR — not built.** There is no `ocr_worker`, no `unstructured`, no `paddleocr`/`pytesseract`
anywhere in this repo (checked `services/api/requirements.txt` and the full source tree — none present).
"Extraction" today is just each connector in `connectors.py` parsing the JSON response of its own API call
into a `CompanyResult`. Document/PDF/image ingestion and the OCR cascade described here are spec only.
3. **Structuring — partially built.** `services/api/workers/resolve.py::resolve()` does real, deterministic
entity resolution: block by `(market, name[:1].upper())`, then `rapidfuzz.fuzz.token_sort_ratio(...) >= 92`
to merge duplicates, bumping `confidence` by +0.1 per corroborating match (capped 0.99) and incrementing
`agreed_sources`. That part is real and unit-testable. What's **not** built: no spaCy/transformers, no NER,
no multilingual normalization (`sv/no/da/fi/de/fr` from the original text), and no explicit LEI-based
cross-market entity linking — matching is name-string-similarity only, within the same market. `spacy` and
`transformers` are not in `requirements.txt`.
4. **Enrichment / fusion — partially built, not fully wired into the job.** The confidence/agreement bump
described above happens inside `resolve()`. Separately, `opensanctions_screen()` (sanctions/PEP) and
`rdw_vehicle()` (NL vehicle data) are real, working connectors in `connectors.py` — but `pipeline.py` never
imports or calls either one. They're only reachable as their own standalone endpoints
(`GET /screen`, `GET /vehicles/{plate}` in `orchestrator/main.py`), not as an automatic enrichment step on
an order's records. NACE/SNI classification and "website/tech" enrichment have no code anywhere.
5. **QA — not built beyond a single threshold filter.** The entire QA stage in `pipeline.py` is:
`delivered = [r for r in deduped if r["confidence"] >= threshold]`. That's real and does drop
below-threshold rows (deliver fewer, never fabricate — that principle holds). But there is no
Great Expectations (not in `requirements.txt`, not imported anywhere), no separate validity/format,
completeness, freshness, or post-dedup-uniqueness checks beyond the confidence number already computed in
structuring, and no quality-report object emitted or stored anywhere — `deliveries` (the SQL table defined
in `sql/001_init.sql`) is never written to by this service. This mirrors a confirmed, broader finding this
session: 0pt's data-quality/QA layer was at one point documented elsewhere (TypeScript side,
`src/services/quality-check-service.ts` etc.) as if built and tested, but those files **do not exist** in
this repo's history at all (zero commits, any branch — see `docs/07-dependencies.md` note). `docs/ICEBERG_WRITE_BOUNDARY.md`,
`docs/11-lakehouse-write-plan.md`, and `docs/P0_QUERY_SOURCE_FLOW.md` have all since been corrected — each
now carries its own explicit "fabricated, not built" callout rather than asserting those files as real —
and `docs/P0_DATASET_QUALITY_FLOW.md` documents this gap as target-spec-only. The Python engine
independently confirms the same gap: **no real quality-check enforcement exists in this codebase today**,
in either language.
**"Each stage updates `jobs` ... so the production tracker / SSE can render it live" is not what happens.**
`run_job` runs synchronously start-to-finish and returns one final dict (`stage: "qa", pct: 100, state: "done",
metrics, records`). `orchestrator/main.py`'s `POST /orders/{oid}/run` awaits the whole thing and stores that one
result in an in-memory `_JOBS` dict; `GET /orders/{oid}/status` just returns whatever's there, or a static
`{"stage": "queued", "pct": 0, "state": "queued"}` placeholder before it's run. There is no per-stage
incremental progress, no `pct` that climbs, and **no SSE endpoint exists anywhere in `orchestrator/main.py`**
(grepped — none). Live progress rendering is target design, not current behavior.
### `adaptive_router` — actual market coverage (verified against `services/api/workers/adaptive_router.py`)
`MARKET_SOURCES` currently has real entries for exactly five markets, each a short compliant list — not the
"elaborate multi-source lists" an earlier version of this router may have implied:
| Market | Ordered plan | Note |
|---|---|---|
| NO | `brreg`, `gleif` | |
| GB | `companies_house`, `gleif` | |
| FI | `prh`, `gleif` | |
| DK | `cvr`, `gleif` | |
| SE | `gleif` only | Bolagsverket bulk is licensed (see above); SE company core relies on GLEIF + a not-yet-built licensed/partner route |
| anything else | `["gleif"]` | fallback in `plan_for_market()` |
`ACCESS_RANK` (`api=0, bulk=1, authorized_web=2, licensed=3`) is defined in the same file but is **not
actually used** by `plan_for_market()` to sort anything — the lists above are hand-ordered constants already
in the desired priority, and the rank dict is currently dead code (documentation-as-code, not logic).
## Confidence & provenance scoring
- Per field, `connectors.py`'s `Provenance` dataclass carries: `field_name`, `source_id`, `confidence` (0–1),
`level` (0–3), `freshness`. **`agreed_sources` is not a field on `Provenance`** — it's tracked separately, at
the merged-record level, by `resolve()` (`match["agreed_sources"] = ... + 1`). If a doc or downstream
consumer expects `agreed_sources` inside each `bridge_provenance` row, that's a mismatch to reconcile against
the actual dataclass shape, not a fact to assume.
- Level: 0 = unavailable/licensed; 1 = single source; 2 = partial corroboration; 3 = ≥2 independent sources agree.
- `est_confidence` on the order = aggregate target (computed in `orchestrator/main.py::compute_estimate`);
delivered `avg_confidence` = actual, computed in `pipeline.py` as the mean confidence of delivered rows.
- **None of this is persisted to Postgres by the current service.** `sql/001_init.sql` defines the
`bridge_provenance` table (plus `dim_company`, `dim_source`, `fact_financials`, `deliveries`, etc.), and
`asyncpg` is a listed dependency — but nothing in `services/api` actually calls `asyncpg` or issues a query.
`DATABASE_URL` is read exactly once, in `orchestrator/main.py`'s `/health` handler, purely to report a
boolean (`"db": bool(os.getenv("DATABASE_URL"))`). `orchestrator/main.py` stores orders and jobs in
in-memory dicts (`_ORDERS`, `_JOBS`) that reset on every restart. The schema exists; the wiring from
pipeline output → Postgres rows does not yet.
## `sources.yaml` (machine-readable registry)
Single registry that drives the router, the `/sources` API, and the (still-to-be-built) public sourcing-map
page. `services/api/sources/sources.yaml` currently has **8 entries**: `gleif`, `brreg`, `companies_house`,
`prh`, `cvr`, `bolagsverket`, `rdw`, `opensanctions`. Schema per entry (as actually used, confirmed against the
file — no `entities`/`fields` mismatch found):
```yaml
- source_id: gleif
name: GLEIF (LEI)
market: global
access: api # api | bulk | authorized_web | licensed
cadence: daily
licence: "open (CC0) — VERIFY current terms"
entities: [company]
fields: [lei, legal_name, address, status]
base_url: https://api.gleif.org
robots_ok: true # for authorized_web only; must be honored
notes: "cross-border key"
```
`orchestrator/main.py` loads this file once at import time (`yaml.safe_load(...)`) and serves it through
`GET /sources` (id/name/market/access/licence/cadence only) and `GET /sources/{market}`. It is genuinely the
live registry behind those two endpoints — that part of the doc's original claim holds.
### Starter set (VERIFY current access/licence per source before relying — terms change)
- **GLEIF / LEI** — typically open; cross-border key. Connector: `gleif_by_lei` — real, working.
- **Brønnøysund (Brreg, NO)** — typically open API. Connector: `brreg_by_orgnr` — real, working.
- **Companies House (UK)** — free API (key required, `COMPANIES_HOUSE_API_KEY`). Connector:
`companies_house_by_number` — real; degrades honestly to level-0 unavailable if the key is unset (does not
fabricate).
- **PRH (FI)** — Connector: `prh_by_businessid` — real, working.
- **CVR / Virk (DK)** — connector actually calls `cvrapi.dk` (a third-party aggregator), **not** Virk's own API
directly, despite the registry entry's `base_url: https://datacvr.virk.dk`. `sources.yaml`'s `base_url` for
`cvr` does not match what `connectors.py::cvr_by_number` actually calls — flag for reconciliation (either
update `base_url` to `cvrapi.dk` or switch the connector to call Virk directly; not resolved this pass).
- **Bolagsverket (SE)** — restricted for bulk; `sources.yaml` marks it `access: licensed`; **no connector
exists** for it in `connectors.py`. Consistent with the doc's own "plan licensed/partner route" framing —
not a false claim, just confirming it's still unbuilt.
- **RDW (NL)** vehicle data — open, Socrata-backed. Connector: `rdw_vehicle` — real, working, but **not called
by the job pipeline** (only reachable via `GET /vehicles/{plate}`).
- **OpenSanctions** — global sanctions/PEP screening. Connector: `opensanctions_screen` — real, working, same
caveat: reachable only via `GET /screen`, not part of `run_job`.
- Vehicle beyond RDW (**Traficom FI**, **Transportstyrelsen SE**, **DVLA UK**) — no connectors exist yet;
*(verify each before building; vehicle/PII rules differ)*.
> Do not hardcode access terms as fact. When building each connector, confirm the current API/licence/robots
> status and record it in `sources.yaml`. If unclear or disallowed for bulk, route to licensed/partner or mark
> fields level 0.
## Structured financials (XBRL) — real but unwired
`services/api/sources/financials.py::companies_house_filings()` is a real, working function: given a
Companies House number and `COMPANIES_HOUSE_API_KEY`, it fetches the accounts filing history and returns up to
10 filings. It is **not imported or called anywhere else in the codebase** (grepped `services/api` — only
`financials.py` itself references the function name). The file's closing comment describes a further XBRL
extraction step ("with `arelle` installed, download each accounts filing's iXBRL document and pull tagged
facts...") that has no code at all — `arelle` is not in `requirements.txt`. Treat the whole file as a real but
disconnected building block, not a wired pipeline stage.
## Libraries used here
Target list only — see `docs/07-dependencies.md` for what's actually installed today vs. still to be added,
with an explicit install-status callout per package. As of this pass, the *only* Python packages installed for
this engine are `fastapi`, `uvicorn`, `pydantic`, `httpx`, `asyncpg` (unused), `python-dotenv`, `pyyaml`,
`rapidfuzz`, `tenacity` — i.e. the API-only tier 1 sourcing + structuring-by-fuzzy-match slice described above,
nothing from the extraction/OCR/NER/quality layers this section used to list as if already in place.