13 compiled .pyc files under services/{api,scrapers}/**/__pycache__/ and 50
scraped brreg.no JSON pages under services/scrapers/output/ were committed to
git despite __pycache__/*.pyc already being covered by .gitignore -- they
predate that rule and git does not retroactively drop already-tracked files
that start matching an ignore pattern.
Neither set is referenced by any source, test, or doc as a fixture (verified
via repo-wide grep); both are pure runtime artifacts regenerated on demand.
Untrack them with git rm --cached, delete the local copies, and add
services/scrapers/output/ to .gitignore so scraper runs stop reintroducing
this. No source or test file touched; pnpm test still 156 passed / 1 honestly
skipped (realinfra ST0RE check) after the change.
Matches docs/compliance/BACKLOG.md P4-4, explicitly flagged as hygiene doable
without a product decision. Picks up unfinished work from an earlier,
never-merged branch of the same name.
0pt data engine (services/api)
The legal data motor (docs/06). FastAPI orchestrator + worker pipeline that turn
compliant sources into delivered, provenance-tagged data. Runs on a VPS via
../../infra/docker-compose.yml — not on Vercel (long jobs).
Sourcing boundary (CLAUDE.md §1 — non-negotiable)
Per field × market, in strict order: official API/bulk → authorized public web (robots + rate limits) → licensed partner data → otherwise level 0 "ej tillgänglig". We never invent values.
Deliberately excluded (never add here): WAF/bot-detection bypass, FlareSolverr, stealth/undetected browsers, proxy/fingerprint rotation to evade bans, scraping behind login walls, automated fake-account creation. This is the honest-accuracy moat.
Layout
sources/connectors.py— compliant connectors (GLEIF, Brønnøysund, Companies House live; PRH/CVR/Bolagsverket gated on verified terms → level 0 until wired).sources/sources.yaml— source registry (access/licence/cadence; VERIFY terms).orchestrator/main.py— REST API (health, sources, estimates, orders, jobs) per docs/04.workers/adaptive_router.py— picks the cheapest compliant method per market (api > bulk > authorized_web > licensed). No bypass tiers.workers/pipeline.py— sourcing → extraction → structuring (dedup) → enrichment/fusion → QA (drops rows below the order's confidence target). Writes provenance per field.
Run locally
pip install -r requirements.txt
uvicorn orchestrator.main:app --reload # API on :8000
python -m workers.run # run one demo job
Reconciliation note
Supersedes the earlier nordic-eu-data-platform backend: the compliant connectors +
pipeline are migrated here; the FlareSolverr/stealth/proxy/fake-account paths and the
test_bypass_chain/test_proxy_pool examples are intentionally not carried over.