CLAUDE.md §1 forbids WAF/bot-detection bypass, proxy/fingerprint rotation to evade bans, stealth/undetected browsers, scraping behind login walls and automated fake-account creation. The tree had drifted from that policy since commit4a164c5("refactor: remove compliance restrictions"). This restores the tree to the repo's own stated policy. Removed (services/scrapers/): - waf_bypass.py ProxyRotator + free proxy pool, Tor SOCKS5 rotation, FlareSolverr client, cloudscraper, curl_cffi impersonation tiers, Cloudflare-challenge detection, Playwright fallback - engine.py multi-method bypass chain + CF-block detection - engine_v2.py METHOD_RANKING per-source bypass table + orchestration - run_pipeline.py batch orchestrator driving ScraperEngine - run_benchmark.py benchmarked bypass methods against ~32 sites - clickhouse_pipeline.py sink for the pipeline (scrape_log.cf_blocked) - s3_sync.py shipped pipeline output to MinIO - Dockerfile, docker-compose.yml ran `python -m scrapers.run_pipeline` - requirements.txt pinned curl_cffi + cloudscraper - __pycache__/engine*.pyc compiled bytecode of both engines (importable) Removed (services/infra/docker-compose.data-stack.yml): - flaresolverr (ghcr.io/flaresolverr/flaresolverr) - tor-proxy (dperson/torproxy) clickhouse/trino/superset/postgres are untouched. Fixed: - adaptive_router.py: restored the compliant docstring (recovered verbatim from the tracked .pyc, which predates4a164c5). The code had no bypass tier. - ARCHITECTURE.md: S9 and §1 described the removed code as present; updated. Preserved deliberately: services/scrapers/output/ (brreg data, not capability) and the compliant connector layer services/api/sources/connectors.py. Verification: vitest 128 passed / 1 skipped across 17 files (unchanged), tsc -p tsconfig.check.json 0 errors, next build exit 0. src/auth/zitadel.ts, tests/auth/*, vitest.config.ts all still present. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
services/scrapers — evasion stack removed
The scraping engine that lived here has been removed as prohibited by
CLAUDE.md §1 (legal sourcing only). Removed in the compliance sweep:
| File | What it was |
|---|---|
waf_bypass.py |
ProxyRotator (free proxy pool), Tor SOCKS5 rotation, FlareSolverr client, cloudscraper, curl_cffi TLS-fingerprint impersonation tiers, Cloudflare-challenge detection, Playwright last-resort fetch |
engine.py |
Multi-method bypass chain + CF-block detection |
engine_v2.py |
METHOD_RANKING — a per-source table mapping each source to the impersonation fingerprint that defeats its bot protection — plus the ScraperEngine orchestration |
run_pipeline.py |
Batch orchestrator driving ScraperEngine per country/source |
run_benchmark.py |
Benchmarked the bypass methods against ~32 registry/directory sites |
clickhouse_pipeline.py |
ClickHouse/DuckDB sink for the pipeline output (scrape_log.cf_blocked, scrape_stats.cf_blocked) |
s3_sync.py |
Uploaded pipeline output to MinIO |
Dockerfile, docker-compose.yml |
Ran python -m scrapers.run_pipeline as long-lived services |
requirements.txt |
Pinned curl_cffi + cloudscraper |
__pycache__/engine*.pyc |
Compiled bytecode of the two engines (importable as-is) |
flaresolverr and tor-proxy (dperson/torproxy) were removed from
services/infra/docker-compose.data-stack.yml in the same change.
Do not reintroduce WAF/bot-detection bypass, stealth or undetected browsers,
proxy/fingerprint rotation to evade bans, scraping behind login walls, or
automated fake-account creation. Compliant sourcing only: official/bulk APIs →
authorized fetches of clearly public pages (self-identifying, robots.txt +
rate limits respected) → licensed partner data → otherwise the field is
delivered as "not available / licensed only".
The compliant source plan lives in services/api/workers/adaptive_router.py.
Retained: output/
output/brreg_page_*.json (50 files) is data, not capability — records from
Norway's official open Enhetsregisteret API (data.brreg.no), which is openly
licensed and needs no bypass to read. It was left in place deliberately rather
than deleted as part of a capability removal.
⚠️ Open question for the owner: this data was fetched through the removed
engine (METHOD_RANKING["no-brreg-api"] routed even this official API through
cloudscraper / curl_cffi impersonation), so its provenance chain is tainted
even though the content itself is public open data. Decide separately whether to
re-fetch it via a plain compliant client or drop it.