- arelle-release + lxml for taxonomi-generisk extraction
- alla tаggade fakta (concept+context+unit+varde+dimensioner) per bolag
- union = hela svenska bastaxonomin (tusentals variabler)
- iter_corpus() for direkt batch over ytterzip->innerzip->XHTML
- batch-runner i workers/ixbrl_batch.py
Three commits landed directly on main (afd301f, bb6be0f, 4a164c5, all
2026-07-13) that violate CLAUDE.md §1 ("Non-negotiable constraints") and
this repo's own services/api/README.md ("Deliberately excluded — never
add here"):
- services/scrapers/waf_bypass.py: FlareSolverr, cloudscraper, Tor SOCKS5
rotation, free-proxy-pool harvesting, fake identity generation, TLS
fingerprint impersonation (curl_cffi) — the exact bypass/stealth taxonomy
the constraint enumerates.
- services/infra/clickhouse/* + docker-compose.data-stack.yml: pipeline
built solely to ingest that scraper's output.
- services/scrapers/engine.py, engine_v2.py, run_pipeline.py, s3_sync.py,
run_benchmark.py, Dockerfile, docker-compose.yml, requirements.txt,
__pycache__/*.pyc, output/*.json fixtures: the rest of the standalone
scraper stack.
- services/scrapers/README.md: pointed at an external "b0ts" repo for
"deep scraping" and committed a live Hetzner Server2 IP + SSH key path
in plaintext — a confidentiality-rule violation independent of the
compliance one.
- .gitea/workflows/ci.yml: a no-op "Scraper CI" stub added solely for
this code (echoes a string, checks nothing).
- services/api/workers/adaptive_router.py: reverted to the pre-4a164c5
compliant version. The removed version dropped the ACCESS_RANK /
compliant-only docstring and replaced MARKET_SOURCES with ~25 source
ids (proff, gulesider, personensuche, whitepages, northdata, ...) none
of which exist in sources.yaml or connectors.py — dangling references
to sources with no registered licence/robots/access record.
None of the removed source ids were ever registered as compliant
connectors, and no doc (README.md, CLAUDE.md, docs/**) references
services/scrapers, services/infra, clickhouse, or b0ts — this stack was
wired in from outside the documented architecture, not part of any
planned build-plan phase.
Verified:
- Repo-wide grep for flaresolverr|stealth|undetected|proxy.rotat|
waf_bypass|cloudscraper|fake.identity|b0ts now only matches the
compliant docs/code that document the exclusion (CLAUDE.md,
services/api/README.md, services/api/sources/connectors.py,
adaptive_router.py's own docstring).
- python3 -m py_compile services/api/workers/adaptive_router.py: OK.
- adaptive_router.py diffed byte-for-byte against git show 4a164c5^ —
identical restore.
NOT verified (environment limitation, not introduced by this change):
- `pnpm install` cannot complete in this sandbox — no outbound network
access to registry.npmjs.org (all package fetches ETIMEDOUT). No
tsc/next build/vitest could therefore be run. This is a pre-existing
environment constraint: this change touches zero .ts/.tsx files, and
tests/p0/*.test.ts already import `vitest`, which is not present in
package.json or pnpm-lock.yaml (0 occurrences) — that test suite was
not runnable before this change either.
Not in scope of this change: services/api's own __pycache__/*.pyc files
predate this commit and are a separate, pre-existing hygiene issue.
Edge /api/vehicles (RDW NL, live now) + /api/screen (OpenSanctions, live now), both defer to engine when ENGINE_URL set. Engine: /vehicles/{plate} + /screen endpoints; workers/resolve (blocking+fuzzy entity resolution, now used by pipeline); workers/queue (in-process job queue); sources/financials (Companies House accounts + XBRL path). py_compile + tsc + next build green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PRH/CVR upgraded from level-0 stubs to live open-API connectors and wired into the pipeline fetch + adaptive router; added RDW vehicle lookup (technical/fleet only, no owner PII) and OpenSanctions name screening. All compliant httpx, VERIFY-terms noted, graceful failure. py_compile clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
FastAPI orchestrator (health/sources/estimates/orders/jobs per docs/04), adaptive router (compliant methods only, no bypass tiers), pipeline (sourcing->extraction->structuring/dedup->fusion->QA with per-field provenance, drops < confidence target), live connectors GLEIF/Brreg/Companies House (PRH/CVR/Bolagsverket gated to level-0 until terms verified). sources.yaml registry. infra/docker-compose + Caddyfile for the VPS. FlareSolverr/stealth/proxy/fake-account paths intentionally NOT carried over (CLAUDE.md §1). py_compile clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Simon Axelsson
admin