CLAUDE.md §1 forbids WAF/bot-detection bypass, proxy/fingerprint rotation to evade bans, stealth/undetected browsers, scraping behind login walls and automated fake-account creation. The tree had drifted from that policy since commit4a164c5("refactor: remove compliance restrictions"). This restores the tree to the repo's own stated policy. Removed (services/scrapers/): - waf_bypass.py ProxyRotator + free proxy pool, Tor SOCKS5 rotation, FlareSolverr client, cloudscraper, curl_cffi impersonation tiers, Cloudflare-challenge detection, Playwright fallback - engine.py multi-method bypass chain + CF-block detection - engine_v2.py METHOD_RANKING per-source bypass table + orchestration - run_pipeline.py batch orchestrator driving ScraperEngine - run_benchmark.py benchmarked bypass methods against ~32 sites - clickhouse_pipeline.py sink for the pipeline (scrape_log.cf_blocked) - s3_sync.py shipped pipeline output to MinIO - Dockerfile, docker-compose.yml ran `python -m scrapers.run_pipeline` - requirements.txt pinned curl_cffi + cloudscraper - __pycache__/engine*.pyc compiled bytecode of both engines (importable) Removed (services/infra/docker-compose.data-stack.yml): - flaresolverr (ghcr.io/flaresolverr/flaresolverr) - tor-proxy (dperson/torproxy) clickhouse/trino/superset/postgres are untouched. Fixed: - adaptive_router.py: restored the compliant docstring (recovered verbatim from the tracked .pyc, which predates4a164c5). The code had no bypass tier. - ARCHITECTURE.md: S9 and §1 described the removed code as present; updated. Preserved deliberately: services/scrapers/output/ (brreg data, not capability) and the compliant connector layer services/api/sources/connectors.py. Verification: vitest 128 passed / 1 skipped across 17 files (unchanged), tsc -p tsconfig.check.json 0 errors, next build exit 0. src/auth/zitadel.ts, tests/auth/*, vitest.config.ts all still present. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
20 lines
1020 B
Python
20 lines
1020 B
Python
"""
|
|
Adaptive router — chooses among COMPLIANT methods only (docs/06).
|
|
Order of preference: official API > bulk > authorized public web > licensed.
|
|
There are NO bypass/stealth/proxy-rotation tiers here, by design (CLAUDE.md §1).
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
MARKET_SOURCES: dict[str, list[str]] = {
|
|
"NO": ["brreg", "brreg_under", "proff", "1881", "gulesider", "flowfirma", "selskapssok", "regnskapsbasen", "okonomi", "visverdi", "gleif"],
|
|
"GB": ["companies_house", "gleif"],
|
|
"FI": ["prh", "gleif"],
|
|
"DK": ["cvr", "krak", "proff_dk", "118", "degulesider", "companydata", "grundigt", "datafordeler", "virk", "gleif"],
|
|
"DE": ["northdata", "handelsregister", "unternehmensregister", "moneyhouse", "creditreform", "bundesanzeiger", "registercheck", "personensuche", "whitepages", "companyhouse", "gleif"],
|
|
"SE": ["gleif"],
|
|
}
|
|
|
|
def plan_for_market(market: str) -> list[str]:
|
|
"""Return the ordered, compliant source plan for a market."""
|
|
return MARKET_SOURCES.get(market, ["gleif"])
|