adminandClaude Opus 4.8
31128a71c8
FastAPI orchestrator (health/sources/estimates/orders/jobs per docs/04), adaptive router (compliant methods only, no bypass tiers), pipeline (sourcing->extraction->structuring/dedup->fusion->QA with per-field provenance, drops < confidence target), live connectors GLEIF/Brreg/Companies House (PRH/CVR/Bolagsverket gated to level-0 until terms verified). sources.yaml registry. infra/docker-compose + Caddyfile for the VPS. FlareSolverr/stealth/proxy/fake-account paths intentionally NOT carried over (CLAUDE.md §1). py_compile clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
35 lines
1.9 KiB
Markdown
35 lines
1.9 KiB
Markdown
# 0pt data engine (`services/api`)
|
||
|
||
The legal data motor (docs/06). FastAPI orchestrator + worker pipeline that turn
|
||
**compliant** sources into delivered, provenance-tagged data. Runs on a VPS via
|
||
[`../../infra/docker-compose.yml`](../../infra/docker-compose.yml) — **not on Vercel** (long jobs).
|
||
|
||
## Sourcing boundary (CLAUDE.md §1 — non-negotiable)
|
||
|
||
Per field × market, in strict order: **official API/bulk → authorized public web
|
||
(robots + rate limits) → licensed partner data → otherwise level 0 "ej tillgänglig"**.
|
||
We never invent values.
|
||
|
||
**Deliberately excluded (never add here):** WAF/bot-detection bypass, FlareSolverr,
|
||
stealth/undetected browsers, proxy/fingerprint rotation to evade bans, scraping behind
|
||
login walls, automated fake-account creation. This is the honest-accuracy moat.
|
||
|
||
## Layout
|
||
- `sources/connectors.py` — compliant connectors (GLEIF, Brønnøysund, Companies House live; PRH/CVR/Bolagsverket gated on verified terms → level 0 until wired).
|
||
- `sources/sources.yaml` — source registry (access/licence/cadence; VERIFY terms).
|
||
- `orchestrator/main.py` — REST API (health, sources, estimates, orders, jobs) per docs/04.
|
||
- `workers/adaptive_router.py` — picks the cheapest **compliant** method per market (api > bulk > authorized_web > licensed). No bypass tiers.
|
||
- `workers/pipeline.py` — sourcing → extraction → structuring (dedup) → enrichment/fusion → QA (drops rows below the order's confidence target). Writes provenance per field.
|
||
|
||
## Run locally
|
||
```bash
|
||
pip install -r requirements.txt
|
||
uvicorn orchestrator.main:app --reload # API on :8000
|
||
python -m workers.run # run one demo job
|
||
```
|
||
|
||
## Reconciliation note
|
||
Supersedes the earlier `nordic-eu-data-platform` backend: the compliant connectors +
|
||
pipeline are migrated here; the FlareSolverr/stealth/proxy/fake-account paths and the
|
||
`test_bypass_chain`/`test_proxy_pool` examples are intentionally **not** carried over.
|