Simon Axelsson
a63530bbe1
Three commits landed directly on main (afd301f,bb6be0f,4a164c5, all 2026-07-13) that violate CLAUDE.md §1 ("Non-negotiable constraints") and this repo's own services/api/README.md ("Deliberately excluded — never add here"): - services/scrapers/waf_bypass.py: FlareSolverr, cloudscraper, Tor SOCKS5 rotation, free-proxy-pool harvesting, fake identity generation, TLS fingerprint impersonation (curl_cffi) — the exact bypass/stealth taxonomy the constraint enumerates. - services/infra/clickhouse/* + docker-compose.data-stack.yml: pipeline built solely to ingest that scraper's output. - services/scrapers/engine.py, engine_v2.py, run_pipeline.py, s3_sync.py, run_benchmark.py, Dockerfile, docker-compose.yml, requirements.txt, __pycache__/*.pyc, output/*.json fixtures: the rest of the standalone scraper stack. - services/scrapers/README.md: pointed at an external "b0ts" repo for "deep scraping" and committed a live Hetzner Server2 IP + SSH key path in plaintext — a confidentiality-rule violation independent of the compliance one. - .gitea/workflows/ci.yml: a no-op "Scraper CI" stub added solely for this code (echoes a string, checks nothing). - services/api/workers/adaptive_router.py: reverted to the pre-4a164c5 compliant version. The removed version dropped the ACCESS_RANK / compliant-only docstring and replaced MARKET_SOURCES with ~25 source ids (proff, gulesider, personensuche, whitepages, northdata, ...) none of which exist in sources.yaml or connectors.py — dangling references to sources with no registered licence/robots/access record. None of the removed source ids were ever registered as compliant connectors, and no doc (README.md, CLAUDE.md, docs/**) references services/scrapers, services/infra, clickhouse, or b0ts — this stack was wired in from outside the documented architecture, not part of any planned build-plan phase. Verified: - Repo-wide grep for flaresolverr|stealth|undetected|proxy.rotat| waf_bypass|cloudscraper|fake.identity|b0ts now only matches the compliant docs/code that document the exclusion (CLAUDE.md, services/api/README.md, services/api/sources/connectors.py, adaptive_router.py's own docstring). - python3 -m py_compile services/api/workers/adaptive_router.py: OK. - adaptive_router.py diffed byte-for-byte against git show 4a164c5^ — identical restore. NOT verified (environment limitation, not introduced by this change): - `pnpm install` cannot complete in this sandbox — no outbound network access to registry.npmjs.org (all package fetches ETIMEDOUT). No tsc/next build/vitest could therefore be run. This is a pre-existing environment constraint: this change touches zero .ts/.tsx files, and tests/p0/*.test.ts already import `vitest`, which is not present in package.json or pnpm-lock.yaml (0 occurrences) — that test suite was not runnable before this change either. Not in scope of this change: services/api's own __pycache__/*.pyc files predate this commit and are a separate, pre-existing hygiene issue.
24 lines
925 B
Python
24 lines
925 B
Python
"""
|
|
Adaptive router — chooses among COMPLIANT methods only (docs/06).
|
|
Order of preference: official API > bulk > authorized public web > licensed.
|
|
There are NO bypass/stealth/proxy-rotation tiers here, by design (CLAUDE.md §1).
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
# market -> ordered list of source_ids to try (compliant connectors only)
|
|
MARKET_SOURCES: dict[str, list[str]] = {
|
|
"NO": ["brreg", "gleif"],
|
|
"GB": ["companies_house", "gleif"],
|
|
"FI": ["prh", "gleif"],
|
|
"DK": ["cvr", "gleif"],
|
|
"SE": ["gleif"], # Bolagsverket bulk is licensed; SE company core via GLEIF + licensed/partner
|
|
}
|
|
|
|
# Access rank: lower = preferred (cheapest compliant path first).
|
|
ACCESS_RANK = {"api": 0, "bulk": 1, "authorized_web": 2, "licensed": 3}
|
|
|
|
|
|
def plan_for_market(market: str) -> list[str]:
|
|
"""Return the ordered, compliant source plan for a market."""
|
|
return MARKET_SOURCES.get(market, ["gleif"])
|