Simon AxelssonandClaude Opus 5
e848992020
CLAUDE.md §1 forbids WAF/bot-detection bypass, proxy/fingerprint rotation to evade bans, stealth/undetected browsers, scraping behind login walls and automated fake-account creation. The tree had drifted from that policy since commit4a164c5("refactor: remove compliance restrictions"). This restores the tree to the repo's own stated policy. Removed (services/scrapers/): - waf_bypass.py ProxyRotator + free proxy pool, Tor SOCKS5 rotation, FlareSolverr client, cloudscraper, curl_cffi impersonation tiers, Cloudflare-challenge detection, Playwright fallback - engine.py multi-method bypass chain + CF-block detection - engine_v2.py METHOD_RANKING per-source bypass table + orchestration - run_pipeline.py batch orchestrator driving ScraperEngine - run_benchmark.py benchmarked bypass methods against ~32 sites - clickhouse_pipeline.py sink for the pipeline (scrape_log.cf_blocked) - s3_sync.py shipped pipeline output to MinIO - Dockerfile, docker-compose.yml ran `python -m scrapers.run_pipeline` - requirements.txt pinned curl_cffi + cloudscraper - __pycache__/engine*.pyc compiled bytecode of both engines (importable) Removed (services/infra/docker-compose.data-stack.yml): - flaresolverr (ghcr.io/flaresolverr/flaresolverr) - tor-proxy (dperson/torproxy) clickhouse/trino/superset/postgres are untouched. Fixed: - adaptive_router.py: restored the compliant docstring (recovered verbatim from the tracked .pyc, which predates4a164c5). The code had no bypass tier. - ARCHITECTURE.md: S9 and §1 described the removed code as present; updated. Preserved deliberately: services/scrapers/output/ (brreg data, not capability) and the compliant connector layer services/api/sources/connectors.py. Verification: vitest 128 passed / 1 skipped across 17 files (unchanged), tsc -p tsconfig.check.json 0 errors, next build exit 0. src/auth/zitadel.ts, tests/auth/*, vitest.config.ts all still present. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
115 lines
4.3 KiB
YAML
115 lines
4.3 KiB
YAML
# SIAX Cloud Data Stack - Complete BigQuery Replacement
|
|
# All OPEN SOURCE, self-hosted on SIAX Cloud Server 2
|
|
#
|
|
# Stack:
|
|
# ClickHouse - Columnar DB (BigQuery replacement)
|
|
# Trino - Distributed SQL query engine
|
|
# Apache Superset - BI dashboards
|
|
# MinIO - S3 object storage (already exists)
|
|
#
|
|
# Compliance: this stack contains NO WAF-bypass, stealth-browser or
|
|
# proxy/fingerprint-rotation component, by design (CLAUDE.md §1).
|
|
#
|
|
version: "3.8"
|
|
|
|
services:
|
|
# ═══════════════════════════════════════════════
|
|
# CLICKHOUSE - BigQuery ersättning
|
|
# ═══════════════════════════════════════════════
|
|
clickhouse:
|
|
image: clickhouse/clickhouse-server:24.12-alpine
|
|
container_name: clickhouse
|
|
restart: unless-stopped
|
|
ports:
|
|
- "8123:8123" # HTTP API (REST)
|
|
- "9000:9000" # Native TCP
|
|
- "9090:9090" # Prometheus metrics
|
|
volumes:
|
|
- clickhouse-data:/var/lib/clickhouse
|
|
- clickhouse-logs:/var/log/clickhouse-server
|
|
- ./clickhouse/config.xml:/etc/clickhouse-server/config.xml:ro
|
|
- ./clickhouse/users.xml:/etc/clickhouse-server/users.xml:ro
|
|
environment:
|
|
- CLICKHOUSE_DB=opt_scraped
|
|
- CLICKHOUSE_USER=default
|
|
- CLICKHOUSE_PASSWORD=${CLICKHOUSE_PASSWORD:-changeme}
|
|
ulimits:
|
|
nofile:
|
|
soft: 262144
|
|
hard: 262144
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
memory: 8G
|
|
reservations:
|
|
memory: 4G
|
|
|
|
# ═══════════════════════════════════════════════
|
|
# TRINO - Distributed SQL (optional, for complex queries)
|
|
# ═══════════════════════════════════════════════
|
|
trino:
|
|
image: trinodb/trino:470
|
|
container_name: trino
|
|
restart: unless-stopped
|
|
ports:
|
|
- "8080:8080"
|
|
volumes:
|
|
- ./trino/etc:/etc/trino:ro
|
|
- trino-data:/data/trino
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
memory: 4G
|
|
|
|
# ═══════════════════════════════════════════════
|
|
# NOTE: no bypass/evasion services in this stack.
|
|
# A `flaresolverr` (Cloudflare bypass) and a `tor-proxy`
|
|
# (dperson/torproxy, anonymous proxy rotation) service used to be
|
|
# defined here. Both were removed as prohibited by CLAUDE.md §1.
|
|
# Do not re-add them, nor any equivalent WAF-bypass, stealth-browser
|
|
# or proxy/fingerprint-rotation service.
|
|
# ═══════════════════════════════════════════════
|
|
|
|
# ═══════════════════════════════════════════════
|
|
# APACHE SUPERSET - BI Dashboard
|
|
# ═══════════════════════════════════════════════
|
|
superset:
|
|
image: apache/superset:latest
|
|
container_name: superset
|
|
restart: unless-stopped
|
|
ports:
|
|
- "8088:8088"
|
|
environment:
|
|
- SUPERSET_SECRET_KEY=${SUPERSET_SECRET_KEY:-changeme}
|
|
- SUPERSET_DATABASE_URI=postgresql://superset:${POSTGRES_PASSWORD:-changeme}@postgres:5432/superset
|
|
volumes:
|
|
- superset-data:/app/superset_home
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
memory: 2G
|
|
depends_on:
|
|
- clickhouse
|
|
- postgres
|
|
|
|
# ═══════════════════════════════════════════════
|
|
# POSTGRES for Superset metadata
|
|
# ═══════════════════════════════════════════════
|
|
postgres:
|
|
image: postgres:16-alpine
|
|
container_name: postgres-superset
|
|
restart: unless-stopped
|
|
environment:
|
|
- POSTGRES_DB=superset
|
|
- POSTGRES_USER=superset
|
|
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-changeme}
|
|
volumes:
|
|
- postgres-data:/var/lib/postgresql/data
|
|
|
|
volumes:
|
|
clickhouse-data:
|
|
clickhouse-logs:
|
|
trino-data:
|
|
superset-data:
|
|
postgres-data:
|