Simon AxelssonandClaude Opus 5
e848992020
CLAUDE.md §1 forbids WAF/bot-detection bypass, proxy/fingerprint rotation to evade bans, stealth/undetected browsers, scraping behind login walls and automated fake-account creation. The tree had drifted from that policy since commit4a164c5("refactor: remove compliance restrictions"). This restores the tree to the repo's own stated policy. Removed (services/scrapers/): - waf_bypass.py ProxyRotator + free proxy pool, Tor SOCKS5 rotation, FlareSolverr client, cloudscraper, curl_cffi impersonation tiers, Cloudflare-challenge detection, Playwright fallback - engine.py multi-method bypass chain + CF-block detection - engine_v2.py METHOD_RANKING per-source bypass table + orchestration - run_pipeline.py batch orchestrator driving ScraperEngine - run_benchmark.py benchmarked bypass methods against ~32 sites - clickhouse_pipeline.py sink for the pipeline (scrape_log.cf_blocked) - s3_sync.py shipped pipeline output to MinIO - Dockerfile, docker-compose.yml ran `python -m scrapers.run_pipeline` - requirements.txt pinned curl_cffi + cloudscraper - __pycache__/engine*.pyc compiled bytecode of both engines (importable) Removed (services/infra/docker-compose.data-stack.yml): - flaresolverr (ghcr.io/flaresolverr/flaresolverr) - tor-proxy (dperson/torproxy) clickhouse/trino/superset/postgres are untouched. Fixed: - adaptive_router.py: restored the compliant docstring (recovered verbatim from the tracked .pyc, which predates4a164c5). The code had no bypass tier. - ARCHITECTURE.md: S9 and §1 described the removed code as present; updated. Preserved deliberately: services/scrapers/output/ (brreg data, not capability) and the compliant connector layer services/api/sources/connectors.py. Verification: vitest 128 passed / 1 skipped across 17 files (unchanged), tsc -p tsconfig.check.json 0 errors, next build exit 0. src/auth/zitadel.ts, tests/auth/*, vitest.config.ts all still present. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
0pt data engine (services/api)
The legal data motor (docs/06). FastAPI orchestrator + worker pipeline that turn
compliant sources into delivered, provenance-tagged data. Runs on a VPS via
../../infra/docker-compose.yml — not on Vercel (long jobs).
Sourcing boundary (CLAUDE.md §1 — non-negotiable)
Per field × market, in strict order: official API/bulk → authorized public web (robots + rate limits) → licensed partner data → otherwise level 0 "ej tillgänglig". We never invent values.
Deliberately excluded (never add here): WAF/bot-detection bypass, FlareSolverr, stealth/undetected browsers, proxy/fingerprint rotation to evade bans, scraping behind login walls, automated fake-account creation. This is the honest-accuracy moat.
Layout
sources/connectors.py— compliant connectors (GLEIF, Brønnøysund, Companies House live; PRH/CVR/Bolagsverket gated on verified terms → level 0 until wired).sources/sources.yaml— source registry (access/licence/cadence; VERIFY terms).orchestrator/main.py— REST API (health, sources, estimates, orders, jobs) per docs/04.workers/adaptive_router.py— picks the cheapest compliant method per market (api > bulk > authorized_web > licensed). No bypass tiers.workers/pipeline.py— sourcing → extraction → structuring (dedup) → enrichment/fusion → QA (drops rows below the order's confidence target). Writes provenance per field.
Run locally
pip install -r requirements.txt
uvicorn orchestrator.main:app --reload # API on :8000
python -m workers.run # run one demo job
Reconciliation note
Supersedes the earlier nordic-eu-data-platform backend: the compliant connectors +
pipeline are migrated here; the FlareSolverr/stealth/proxy/fake-account paths and the
test_bypass_chain/test_proxy_pool examples are intentionally not carried over.