Files
0pt/services/scrapers
Simon AxelssonandClaude Opus 5 e848992020
CI (SIAX Cloud) / security (pull_request) Successful in 19s
CI (SIAX Cloud) / quality (pull_request) Successful in 41s
CI (SIAX Cloud) / security (push) Successful in 15s
CI (SIAX Cloud) / quality (push) Successful in 40s
chore(compliance): remove the banned WAF-bypass/evasion stack (CLAUDE.md §1)
CLAUDE.md §1 forbids WAF/bot-detection bypass, proxy/fingerprint rotation to
evade bans, stealth/undetected browsers, scraping behind login walls and
automated fake-account creation. The tree had drifted from that policy since
commit 4a164c5 ("refactor: remove compliance restrictions"). This restores the
tree to the repo's own stated policy.

Removed (services/scrapers/):
- waf_bypass.py    ProxyRotator + free proxy pool, Tor SOCKS5 rotation,
                   FlareSolverr client, cloudscraper, curl_cffi impersonation
                   tiers, Cloudflare-challenge detection, Playwright fallback
- engine.py        multi-method bypass chain + CF-block detection
- engine_v2.py     METHOD_RANKING per-source bypass table + orchestration
- run_pipeline.py  batch orchestrator driving ScraperEngine
- run_benchmark.py benchmarked bypass methods against ~32 sites
- clickhouse_pipeline.py  sink for the pipeline (scrape_log.cf_blocked)
- s3_sync.py       shipped pipeline output to MinIO
- Dockerfile, docker-compose.yml  ran `python -m scrapers.run_pipeline`
- requirements.txt pinned curl_cffi + cloudscraper
- __pycache__/engine*.pyc  compiled bytecode of both engines (importable)

Removed (services/infra/docker-compose.data-stack.yml):
- flaresolverr (ghcr.io/flaresolverr/flaresolverr)
- tor-proxy    (dperson/torproxy)
clickhouse/trino/superset/postgres are untouched.

Fixed:
- adaptive_router.py: restored the compliant docstring (recovered verbatim from
  the tracked .pyc, which predates 4a164c5). The code had no bypass tier.
- ARCHITECTURE.md: S9 and §1 described the removed code as present; updated.

Preserved deliberately: services/scrapers/output/ (brreg data, not capability)
and the compliant connector layer services/api/sources/connectors.py.

Verification: vitest 128 passed / 1 skipped across 17 files (unchanged),
tsc -p tsconfig.check.json 0 errors, next build exit 0.
src/auth/zitadel.ts, tests/auth/*, vitest.config.ts all still present.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 23:06:53 +02:00
..

services/scrapers — evasion stack removed

The scraping engine that lived here has been removed as prohibited by CLAUDE.md §1 (legal sourcing only). Removed in the compliance sweep:

File What it was
waf_bypass.py ProxyRotator (free proxy pool), Tor SOCKS5 rotation, FlareSolverr client, cloudscraper, curl_cffi TLS-fingerprint impersonation tiers, Cloudflare-challenge detection, Playwright last-resort fetch
engine.py Multi-method bypass chain + CF-block detection
engine_v2.py METHOD_RANKING — a per-source table mapping each source to the impersonation fingerprint that defeats its bot protection — plus the ScraperEngine orchestration
run_pipeline.py Batch orchestrator driving ScraperEngine per country/source
run_benchmark.py Benchmarked the bypass methods against ~32 registry/directory sites
clickhouse_pipeline.py ClickHouse/DuckDB sink for the pipeline output (scrape_log.cf_blocked, scrape_stats.cf_blocked)
s3_sync.py Uploaded pipeline output to MinIO
Dockerfile, docker-compose.yml Ran python -m scrapers.run_pipeline as long-lived services
requirements.txt Pinned curl_cffi + cloudscraper
__pycache__/engine*.pyc Compiled bytecode of the two engines (importable as-is)

flaresolverr and tor-proxy (dperson/torproxy) were removed from services/infra/docker-compose.data-stack.yml in the same change.

Do not reintroduce WAF/bot-detection bypass, stealth or undetected browsers, proxy/fingerprint rotation to evade bans, scraping behind login walls, or automated fake-account creation. Compliant sourcing only: official/bulk APIs → authorized fetches of clearly public pages (self-identifying, robots.txt + rate limits respected) → licensed partner data → otherwise the field is delivered as "not available / licensed only".

The compliant source plan lives in services/api/workers/adaptive_router.py.

Retained: output/

output/brreg_page_*.json (50 files) is data, not capability — records from Norway's official open Enhetsregisteret API (data.brreg.no), which is openly licensed and needs no bypass to read. It was left in place deliberately rather than deleted as part of a capability removal.

⚠️ Open question for the owner: this data was fetched through the removed engine (METHOD_RANKING["no-brreg-api"] routed even this official API through cloudscraper / curl_cffi impersonation), so its provenance chain is tainted even though the content itself is public open data. Decide separately whether to re-fetch it via a plain compliant client or drop it.