chore(compliance): remove the banned WAF-bypass/evasion stack (CLAUDE.md §1)
CI (SIAX Cloud) / security (pull_request) Successful in 19s
CI (SIAX Cloud) / quality (pull_request) Successful in 41s
CI (SIAX Cloud) / security (push) Successful in 15s
CI (SIAX Cloud) / quality (push) Successful in 40s

CLAUDE.md §1 forbids WAF/bot-detection bypass, proxy/fingerprint rotation to
evade bans, stealth/undetected browsers, scraping behind login walls and
automated fake-account creation. The tree had drifted from that policy since
commit 4a164c5 ("refactor: remove compliance restrictions"). This restores the
tree to the repo's own stated policy.

Removed (services/scrapers/):
- waf_bypass.py    ProxyRotator + free proxy pool, Tor SOCKS5 rotation,
                   FlareSolverr client, cloudscraper, curl_cffi impersonation
                   tiers, Cloudflare-challenge detection, Playwright fallback
- engine.py        multi-method bypass chain + CF-block detection
- engine_v2.py     METHOD_RANKING per-source bypass table + orchestration
- run_pipeline.py  batch orchestrator driving ScraperEngine
- run_benchmark.py benchmarked bypass methods against ~32 sites
- clickhouse_pipeline.py  sink for the pipeline (scrape_log.cf_blocked)
- s3_sync.py       shipped pipeline output to MinIO
- Dockerfile, docker-compose.yml  ran `python -m scrapers.run_pipeline`
- requirements.txt pinned curl_cffi + cloudscraper
- __pycache__/engine*.pyc  compiled bytecode of both engines (importable)

Removed (services/infra/docker-compose.data-stack.yml):
- flaresolverr (ghcr.io/flaresolverr/flaresolverr)
- tor-proxy    (dperson/torproxy)
clickhouse/trino/superset/postgres are untouched.

Fixed:
- adaptive_router.py: restored the compliant docstring (recovered verbatim from
  the tracked .pyc, which predates 4a164c5). The code had no bypass tier.
- ARCHITECTURE.md: S9 and §1 described the removed code as present; updated.

Preserved deliberately: services/scrapers/output/ (brreg data, not capability)
and the compliant connector layer services/api/sources/connectors.py.

Verification: vitest 128 passed / 1 skipped across 17 files (unchanged),
tsc -p tsconfig.check.json 0 errors, next build exit 0.
src/auth/zitadel.ts, tests/auth/*, vitest.config.ts all still present.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Simon Axelsson
2026-08-07 23:06:53 +02:00
co-authored by Claude Opus 5
parent bb58617540
commit e848992020
17 changed files with 85 additions and 1647 deletions
+9 -32
View File
@@ -4,11 +4,12 @@
# Stack:
# ClickHouse - Columnar DB (BigQuery replacement)
# Trino - Distributed SQL query engine
# FlareSolverr - Cloudflare bypass
# Tor - Anonymous proxy rotation
# Apache Superset - BI dashboards
# MinIO - S3 object storage (already exists)
#
# Compliance: this stack contains NO WAF-bypass, stealth-browser or
# proxy/fingerprint-rotation component, by design (CLAUDE.md §1).
#
version: "3.8"
services:
@@ -61,37 +62,13 @@ services:
memory: 4G
# ═══════════════════════════════════════════════
# FLARESOLVERR - Cloudflare bypass
# NOTE: no bypass/evasion services in this stack.
# A `flaresolverr` (Cloudflare bypass) and a `tor-proxy`
# (dperson/torproxy, anonymous proxy rotation) service used to be
# defined here. Both were removed as prohibited by CLAUDE.md §1.
# Do not re-add them, nor any equivalent WAF-bypass, stealth-browser
# or proxy/fingerprint-rotation service.
# ═══════════════════════════════════════════════
flaresolverr:
image: ghcr.io/flaresolverr/flaresolverr:latest
container_name: flaresolverr
restart: unless-stopped
ports:
- "8191:8191"
environment:
- LOG_LEVEL=${LOG_LEVEL:-info}
- TZ=Europe/Stockholm
- CAPTCHA_SOLVER=none
deploy:
resources:
limits:
memory: 1G
# ═══════════════════════════════════════════════
# TOR - Anonymous proxy rotation
# ═══════════════════════════════════════════════
tor-proxy:
image: dperson/torproxy:latest
container_name: tor-proxy
restart: unless-stopped
ports:
- "9050:9050" # SOCKS5
- "9051:9051" # Control port
deploy:
resources:
limits:
memory: 256M
# ═══════════════════════════════════════════════
# APACHE SUPERSET - BI Dashboard