Simon Axelsson a63530bbe1 fix: remove non-compliant scraper stack, restore compliant-only sourcing
Three commits landed directly on main (afd301f, bb6be0f, 4a164c5, all
2026-07-13) that violate CLAUDE.md §1 ("Non-negotiable constraints") and
this repo's own services/api/README.md ("Deliberately excluded — never
add here"):

- services/scrapers/waf_bypass.py: FlareSolverr, cloudscraper, Tor SOCKS5
  rotation, free-proxy-pool harvesting, fake identity generation, TLS
  fingerprint impersonation (curl_cffi) — the exact bypass/stealth taxonomy
  the constraint enumerates.
- services/infra/clickhouse/* + docker-compose.data-stack.yml: pipeline
  built solely to ingest that scraper's output.
- services/scrapers/engine.py, engine_v2.py, run_pipeline.py, s3_sync.py,
  run_benchmark.py, Dockerfile, docker-compose.yml, requirements.txt,
  __pycache__/*.pyc, output/*.json fixtures: the rest of the standalone
  scraper stack.
- services/scrapers/README.md: pointed at an external "b0ts" repo for
  "deep scraping" and committed a live Hetzner Server2 IP + SSH key path
  in plaintext — a confidentiality-rule violation independent of the
  compliance one.
- .gitea/workflows/ci.yml: a no-op "Scraper CI" stub added solely for
  this code (echoes a string, checks nothing).
- services/api/workers/adaptive_router.py: reverted to the pre-4a164c5
  compliant version. The removed version dropped the ACCESS_RANK /
  compliant-only docstring and replaced MARKET_SOURCES with ~25 source
  ids (proff, gulesider, personensuche, whitepages, northdata, ...) none
  of which exist in sources.yaml or connectors.py — dangling references
  to sources with no registered licence/robots/access record.

None of the removed source ids were ever registered as compliant
connectors, and no doc (README.md, CLAUDE.md, docs/**) references
services/scrapers, services/infra, clickhouse, or b0ts — this stack was
wired in from outside the documented architecture, not part of any
planned build-plan phase.

Verified:
- Repo-wide grep for flaresolverr|stealth|undetected|proxy.rotat|
  waf_bypass|cloudscraper|fake.identity|b0ts now only matches the
  compliant docs/code that document the exclusion (CLAUDE.md,
  services/api/README.md, services/api/sources/connectors.py,
  adaptive_router.py's own docstring).
- python3 -m py_compile services/api/workers/adaptive_router.py: OK.
- adaptive_router.py diffed byte-for-byte against git show 4a164c5^ —
  identical restore.

NOT verified (environment limitation, not introduced by this change):
- `pnpm install` cannot complete in this sandbox — no outbound network
  access to registry.npmjs.org (all package fetches ETIMEDOUT). No
  tsc/next build/vitest could therefore be run. This is a pre-existing
  environment constraint: this change touches zero .ts/.tsx files, and
  tests/p0/*.test.ts already import `vitest`, which is not present in
  package.json or pnpm-lock.yaml (0 occurrences) — that test suite was
  not runnable before this change either.

Not in scope of this change: services/api's own __pycache__/*.pyc files
predate this commit and are a separate, pre-existing hygiene issue.
2026-07-19 18:16:40 +02:00
2026-06-24 14:47:12 +00:00
2026-06-24 14:47:12 +00:00
2026-06-24 14:47:12 +00:00
2026-06-24 14:47:12 +00:00
2026-06-24 14:47:12 +00:00
2026-06-24 14:47:12 +00:00
2026-06-24 14:51:57 +00:00

0pt — your data

Data services on demand for the Nordics + Europe. Compose a data job, pay 50 % to start, watch production live, validate with full provenance, deliver anywhere. Owned by SIAX Technology AB. Honest accuracy is the moat: every field carries source, freshness and confidence; nothing is guessed.

Live: https://0pt.app · Contract: CLAUDE.md + docs/

Two planes (see docs/02-architecture.md)

  • Edge (Vercel, this app) — marketing, catalog, the order lifecycle, customer portal, admin portal, and the API route handlers. Defers to the data engine when ENGINE_URL is set.
  • Engine (services/api, runs on a VPS) — FastAPI orchestrator + worker pipeline doing the compliant sourcing (official API → bulk → authorized web → licensed → else level 0). Never on Vercel (long jobs). Bypass/stealth/proxy-rotation/fake-account paths are excluded by design (CLAUDE.md §1).

Routes

Area Routes
Marketing / (hero, filosofi, katalog, motorn, visualiseringar, täckning, editorial, bevis, priser, footer)
Catalog /katalog, /katalog/[id]
Sourcing map /kallor
Order lifecycle /bestall → /bestall/forskott → /order/[id] → /order/[id]/validera
Customer portal /portal, /portal/subscriptions, /portal/files, /portal/integrationer, /portal/settings
Admin portal /admin, /admin/sources, /admin/jobs, /admin/customers
API /api/{health,estimates,catalog,sources,orders,orders/[id],subscriptions,admin/metrics,uploads,auth/*,payments/*,webhooks/stripe,connections/db-grant}

Design system

Mono/editorial, light, radius 0, Inter + JetBrains Mono, single red #e40014, animated grain, scroll-reveal motion. No photography — imagery is rendered data (DataArtifact). Every fact shows a provenance square (ProvenanceBadge). Full spec: docs/05-design-system.md.

Run locally

pnpm install
pnpm dev                       # Next.js on http://localhost:3000

# data engine (separate, optional)
cd services/api && pip install -r requirements.txt && uvicorn orchestrator.main:app --reload

Deploy

  • Frontend → Vercel (project 0pt, scope Simon Axelsson's projects), domain 0pt.app.
  • Engine + DB + embeds → one VPS via infra/docker-compose.yml (Postgres+pgvector, Redis, api, worker, NocoDB, Metabase, Caddy).
  • Env inventory: .env.example.

Status

Frontend, full order lifecycle, customer + admin portals, edge API, payment/auth/embed infrastructure, the compliant data engine, schema and infra are all in place and building green. Integrations are env-guarded and activate when their keys/DB are provided:

  1. Database — Vercel → Storage → Postgres (sets POSTGRES_URL) → durable persistence (schema in sql/001_init.sql).
  2. Stripe + Resend keys → payments + email.
  3. VPS for the engine → set ENGINE_URL.
  4. AUTH_SECRET + DEV_LOGIN_KEY → login (full OAuth via Auth.js lands with the DB).

© 2026 SIAX Technology AB. Not affiliated with Protosell.

S
Description
No description provided
Readme
26 MiB
Languages
TypeScript 90.3%
Python 5.5%
JavaScript 2.1%
PLpgSQL 1.4%
Shell 0.3%
Other 0.4%