CLAUDE.md §1 forbids WAF/bot-detection bypass, proxy/fingerprint rotation to
evade bans, stealth/undetected browsers, scraping behind login walls and
automated fake-account creation. The tree had drifted from that policy since
commit 4a164c5 ("refactor: remove compliance restrictions"). This restores the
tree to the repo's own stated policy.
Removed (services/scrapers/):
- waf_bypass.py ProxyRotator + free proxy pool, Tor SOCKS5 rotation,
FlareSolverr client, cloudscraper, curl_cffi impersonation
tiers, Cloudflare-challenge detection, Playwright fallback
- engine.py multi-method bypass chain + CF-block detection
- engine_v2.py METHOD_RANKING per-source bypass table + orchestration
- run_pipeline.py batch orchestrator driving ScraperEngine
- run_benchmark.py benchmarked bypass methods against ~32 sites
- clickhouse_pipeline.py sink for the pipeline (scrape_log.cf_blocked)
- s3_sync.py shipped pipeline output to MinIO
- Dockerfile, docker-compose.yml ran `python -m scrapers.run_pipeline`
- requirements.txt pinned curl_cffi + cloudscraper
- __pycache__/engine*.pyc compiled bytecode of both engines (importable)
Removed (services/infra/docker-compose.data-stack.yml):
- flaresolverr (ghcr.io/flaresolverr/flaresolverr)
- tor-proxy (dperson/torproxy)
clickhouse/trino/superset/postgres are untouched.
Fixed:
- adaptive_router.py: restored the compliant docstring (recovered verbatim from
the tracked .pyc, which predates 4a164c5). The code had no bypass tier.
- ARCHITECTURE.md: S9 and §1 described the removed code as present; updated.
Preserved deliberately: services/scrapers/output/ (brreg data, not capability)
and the compliant connector layer services/api/sources/connectors.py.
Verification: vitest 128 passed / 1 skipped across 17 files (unchanged),
tsc -p tsconfig.check.json 0 errors, next build exit 0.
src/auth/zitadel.ts, tests/auth/*, vitest.config.ts all still present.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Edge /api/vehicles (RDW NL, live now) + /api/screen (OpenSanctions, live now), both defer to engine when ENGINE_URL set. Engine: /vehicles/{plate} + /screen endpoints; workers/resolve (blocking+fuzzy entity resolution, now used by pipeline); workers/queue (in-process job queue); sources/financials (Companies House accounts + XBRL path). py_compile + tsc + next build green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PRH/CVR upgraded from level-0 stubs to live open-API connectors and wired into the pipeline fetch + adaptive router; added RDW vehicle lookup (technical/fleet only, no owner PII) and OpenSanctions name screening. All compliant httpx, VERIFY-terms noted, graceful failure. py_compile clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
FastAPI orchestrator (health/sources/estimates/orders/jobs per docs/04), adaptive router (compliant methods only, no bypass tiers), pipeline (sourcing->extraction->structuring/dedup->fusion->QA with per-field provenance, drops < confidence target), live connectors GLEIF/Brreg/Companies House (PRH/CVR/Bolagsverket gated to level-0 until terms verified). sources.yaml registry. infra/docker-compose + Caddyfile for the VPS. FlareSolverr/stealth/proxy/fake-account paths intentionally NOT carried over (CLAUDE.md §1). py_compile clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>