Simon Axelsson
a63530bbe1
Three commits landed directly on main (afd301f,bb6be0f,4a164c5, all 2026-07-13) that violate CLAUDE.md §1 ("Non-negotiable constraints") and this repo's own services/api/README.md ("Deliberately excluded — never add here"): - services/scrapers/waf_bypass.py: FlareSolverr, cloudscraper, Tor SOCKS5 rotation, free-proxy-pool harvesting, fake identity generation, TLS fingerprint impersonation (curl_cffi) — the exact bypass/stealth taxonomy the constraint enumerates. - services/infra/clickhouse/* + docker-compose.data-stack.yml: pipeline built solely to ingest that scraper's output. - services/scrapers/engine.py, engine_v2.py, run_pipeline.py, s3_sync.py, run_benchmark.py, Dockerfile, docker-compose.yml, requirements.txt, __pycache__/*.pyc, output/*.json fixtures: the rest of the standalone scraper stack. - services/scrapers/README.md: pointed at an external "b0ts" repo for "deep scraping" and committed a live Hetzner Server2 IP + SSH key path in plaintext — a confidentiality-rule violation independent of the compliance one. - .gitea/workflows/ci.yml: a no-op "Scraper CI" stub added solely for this code (echoes a string, checks nothing). - services/api/workers/adaptive_router.py: reverted to the pre-4a164c5 compliant version. The removed version dropped the ACCESS_RANK / compliant-only docstring and replaced MARKET_SOURCES with ~25 source ids (proff, gulesider, personensuche, whitepages, northdata, ...) none of which exist in sources.yaml or connectors.py — dangling references to sources with no registered licence/robots/access record. None of the removed source ids were ever registered as compliant connectors, and no doc (README.md, CLAUDE.md, docs/**) references services/scrapers, services/infra, clickhouse, or b0ts — this stack was wired in from outside the documented architecture, not part of any planned build-plan phase. Verified: - Repo-wide grep for flaresolverr|stealth|undetected|proxy.rotat| waf_bypass|cloudscraper|fake.identity|b0ts now only matches the compliant docs/code that document the exclusion (CLAUDE.md, services/api/README.md, services/api/sources/connectors.py, adaptive_router.py's own docstring). - python3 -m py_compile services/api/workers/adaptive_router.py: OK. - adaptive_router.py diffed byte-for-byte against git show 4a164c5^ — identical restore. NOT verified (environment limitation, not introduced by this change): - `pnpm install` cannot complete in this sandbox — no outbound network access to registry.npmjs.org (all package fetches ETIMEDOUT). No tsc/next build/vitest could therefore be run. This is a pre-existing environment constraint: this change touches zero .ts/.tsx files, and tests/p0/*.test.ts already import `vitest`, which is not present in package.json or pnpm-lock.yaml (0 occurrences) — that test suite was not runnable before this change either. Not in scope of this change: services/api's own __pycache__/*.pyc files predate this commit and are a separate, pre-existing hygiene issue.
0pt data engine (services/api)
The legal data motor (docs/06). FastAPI orchestrator + worker pipeline that turn
compliant sources into delivered, provenance-tagged data. Runs on a VPS via
../../infra/docker-compose.yml — not on Vercel (long jobs).
Sourcing boundary (CLAUDE.md §1 — non-negotiable)
Per field × market, in strict order: official API/bulk → authorized public web (robots + rate limits) → licensed partner data → otherwise level 0 "ej tillgänglig". We never invent values.
Deliberately excluded (never add here): WAF/bot-detection bypass, FlareSolverr, stealth/undetected browsers, proxy/fingerprint rotation to evade bans, scraping behind login walls, automated fake-account creation. This is the honest-accuracy moat.
Layout
sources/connectors.py— compliant connectors (GLEIF, Brønnøysund, Companies House live; PRH/CVR/Bolagsverket gated on verified terms → level 0 until wired).sources/sources.yaml— source registry (access/licence/cadence; VERIFY terms).orchestrator/main.py— REST API (health, sources, estimates, orders, jobs) per docs/04.workers/adaptive_router.py— picks the cheapest compliant method per market (api > bulk > authorized_web > licensed). No bypass tiers.workers/pipeline.py— sourcing → extraction → structuring (dedup) → enrichment/fusion → QA (drops rows below the order's confidence target). Writes provenance per field.
Run locally
pip install -r requirements.txt
uvicorn orchestrator.main:app --reload # API on :8000
python -m workers.run # run one demo job
Reconciliation note
Supersedes the earlier nordic-eu-data-platform backend: the compliant connectors +
pipeline are migrated here; the FlareSolverr/stealth/proxy/fake-account paths and the
test_bypass_chain/test_proxy_pool examples are intentionally not carried over.