CLAUDE.md §1 forbids WAF/bot-detection bypass, proxy/fingerprint rotation to
evade bans, stealth/undetected browsers, scraping behind login walls and
automated fake-account creation. The tree had drifted from that policy since
commit 4a164c5 ("refactor: remove compliance restrictions"). This restores the
tree to the repo's own stated policy.
Removed (services/scrapers/):
- waf_bypass.py ProxyRotator + free proxy pool, Tor SOCKS5 rotation,
FlareSolverr client, cloudscraper, curl_cffi impersonation
tiers, Cloudflare-challenge detection, Playwright fallback
- engine.py multi-method bypass chain + CF-block detection
- engine_v2.py METHOD_RANKING per-source bypass table + orchestration
- run_pipeline.py batch orchestrator driving ScraperEngine
- run_benchmark.py benchmarked bypass methods against ~32 sites
- clickhouse_pipeline.py sink for the pipeline (scrape_log.cf_blocked)
- s3_sync.py shipped pipeline output to MinIO
- Dockerfile, docker-compose.yml ran `python -m scrapers.run_pipeline`
- requirements.txt pinned curl_cffi + cloudscraper
- __pycache__/engine*.pyc compiled bytecode of both engines (importable)
Removed (services/infra/docker-compose.data-stack.yml):
- flaresolverr (ghcr.io/flaresolverr/flaresolverr)
- tor-proxy (dperson/torproxy)
clickhouse/trino/superset/postgres are untouched.
Fixed:
- adaptive_router.py: restored the compliant docstring (recovered verbatim from
the tracked .pyc, which predates 4a164c5). The code had no bypass tier.
- ARCHITECTURE.md: S9 and §1 described the removed code as present; updated.
Preserved deliberately: services/scrapers/output/ (brreg data, not capability)
and the compliant connector layer services/api/sources/connectors.py.
Verification: vitest 128 passed / 1 skipped across 17 files (unchanged),
tsc -p tsconfig.check.json 0 errors, next build exit 0.
src/auth/zitadel.ts, tests/auth/*, vitest.config.ts all still present.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Brings 0pt to the SIAX repo standard. README, LICENSE (proprietary, correct
for this repo), Dockerfile and .gitea/workflows/ci.yml already existed and are
left untouched; no .github/workflows exists.
The architecture document was written by reading the code (entrypoints,
routes, services, migrations, Dockerfile, env, tests) and by probing the live
0pt.app instance -- not from the templates. It therefore contradicts the
existing docs in several places, deliberately:
* /api/0pt/* answers ANONYMOUSLY in production (live-verified 2026-08-07):
GET https://0pt.app/api/0pt returns 200 with no token, and 200 with a
deliberately invalid one. The deployment is current (the newer lakehouse
routes are present), so middleware.ts is simply not running. README lines
49-52 and merge 6531384 claim the opposite. Root cause UNKNOWN -- flagged,
not guessed.
* No handler reads x-0pt-verified-tenant. middleware.ts sets it; nothing else
in the repo reads it. Every /api/0pt/* handler takes tenant_id from the
request body/query instead, so tenant isolation is not implemented.
* Everything outside /api/0pt/* has no auth at all (matcher scope), including
GET /api/admin/metrics and GET /api/orders (both live-verified 200 anon) and
POST /api/payments/{id}/release, which flips an order to "delivered"
regardless of payment.
* CL0UD policy client is fail-open and AUD0 audit is a silent no-op when their
URLs are unset, with no production guard -- unlike AUTH_ENFORCE, which has one.
* connections/db-grant documents AES-GCM encryption it does not implement.
* The FastAPI engine (services/api) has zero authentication despite
lib/engine.ts sending X-Internal-Token.
* services/scrapers/waf_bypass.py (FlareSolverr, Tor, cloudscraper, proxy pool,
undetected browser, fake identities) plus flaresolverr/tor-proxy in
services/infra/docker-compose.data-stack.yml directly contradict CLAUDE.md
§1 and the README's compliance claim. Left in place -- deletion is a Simon
decision, recorded as open question 2.
* Production has NO database (live /api/health reports db:not_configured) and
the whole /api/0pt/* capability layer is per-module in-memory Maps, so state
does not survive a cold start and is not even shared between route modules.
siax.config.json deviates from the template on two fields, on purpose:
deploy.target is "vercel" (live-verified Server: Vercel header), not "coolify",
and designSystem is null because @siax/ui is not a dependency here. Both
deviations are documented in ARCHITECTURE.md §5 and §8.
Docs-only change. No code, CI, LICENSE or workflow files were modified.