feat(engine): import compliant data engine (services/api) + infra

FastAPI orchestrator (health/sources/estimates/orders/jobs per docs/04), adaptive router (compliant methods only, no bypass tiers), pipeline (sourcing->extraction->structuring/dedup->fusion->QA with per-field provenance, drops < confidence target), live connectors GLEIF/Brreg/Companies House (PRH/CVR/Bolagsverket gated to level-0 until terms verified). sources.yaml registry. infra/docker-compose + Caddyfile for the VPS. FlareSolverr/stealth/proxy/fake-account paths intentionally NOT carried over (CLAUDE.md §1). py_compile clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-24 20:02:50 +02:00
co-authored by Claude Opus 4.8
parent 9eb1ea6557
commit 31128a71c8
22 changed files with 687 additions and 0 deletions
+13
View File
@@ -0,0 +1,13 @@
# 0pt — reverse proxy + automatic TLS (docs/08). Frontend stays on Vercel (0pt.app);
# these subdomains front the VPS services. Point the DNS A-records at the VPS.
api.0pt.app {
reverse_proxy api:8000
}
data.0pt.app {
reverse_proxy nocodb:8080
}
dash.0pt.app {
reverse_proxy metabase:3000
}
+69
View File
@@ -0,0 +1,69 @@
# 0pt — always-on services for a single small VPS (docs/08). Vercel hosts the
# Next.js frontend; this compose runs the data engine + DB + embeds + proxy.
services:
postgres:
image: pgvector/pgvector:pg16
environment:
POSTGRES_USER: opt
POSTGRES_PASSWORD: ${PG_PASSWORD}
POSTGRES_DB: opt
volumes:
- pgdata:/var/lib/postgresql/data
- ../sql:/docker-entrypoint-initdb.d:ro
restart: unless-stopped
redis:
image: redis:7
restart: unless-stopped
api:
build: ../services/api
command: uvicorn orchestrator.main:app --host 0.0.0.0 --port 8000
environment:
DATABASE_URL: postgres://opt:${PG_PASSWORD}@postgres:5432/opt
COMPANIES_HOUSE_API_KEY: ${COMPANIES_HOUSE_API_KEY:-}
INTERNAL_TOKEN: ${INTERNAL_TOKEN}
depends_on: [postgres, redis]
restart: unless-stopped
worker:
build: ../services/api
command: python -m workers.run
environment:
DATABASE_URL: postgres://opt:${PG_PASSWORD}@postgres:5432/opt
COMPANIES_HOUSE_API_KEY: ${COMPANIES_HOUSE_API_KEY:-}
depends_on: [postgres, redis]
restart: unless-stopped
nocodb:
image: nocodb/nocodb:latest
environment:
NC_DB: "pg://postgres:5432?u=opt&p=${PG_PASSWORD}&d=opt"
NC_PUBLIC_URL: "https://data.0pt.app"
depends_on: [postgres]
restart: unless-stopped
metabase:
image: metabase/metabase:latest
environment:
MB_DB_TYPE: postgres
MB_DB_HOST: postgres
MB_DB_PORT: "5432"
MB_DB_USER: opt
MB_DB_PASS: ${PG_PASSWORD}
MB_DB_DBNAME: opt
depends_on: [postgres]
restart: unless-stopped
caddy:
image: caddy:2
ports: ["80:80", "443:443"]
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile
- caddydata:/data
depends_on: [api, nocodb, metabase]
restart: unless-stopped
volumes:
pgdata: {}
caddydata: {}
+15
View File
@@ -0,0 +1,15 @@
# 0pt data engine — API + worker image (runs on a VPS, NOT Vercel; long jobs).
FROM python:3.12-slim
WORKDIR /app
ENV PYTHONUNBUFFERED=1 PIP_NO_CACHE_DIR=1
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
EXPOSE 8000
# API: uvicorn orchestrator.main:app --host 0.0.0.0 --port 8000
# Worker: python -m workers.run
CMD ["uvicorn", "orchestrator.main:app", "--host", "0.0.0.0", "--port", "8000"]
+34
View File
@@ -0,0 +1,34 @@
# 0pt data engine (`services/api`)
The legal data motor (docs/06). FastAPI orchestrator + worker pipeline that turn
**compliant** sources into delivered, provenance-tagged data. Runs on a VPS via
[`../../infra/docker-compose.yml`](../../infra/docker-compose.yml) — **not on Vercel** (long jobs).
## Sourcing boundary (CLAUDE.md §1 — non-negotiable)
Per field × market, in strict order: **official API/bulk → authorized public web
(robots + rate limits) → licensed partner data → otherwise level 0 "ej tillgänglig"**.
We never invent values.
**Deliberately excluded (never add here):** WAF/bot-detection bypass, FlareSolverr,
stealth/undetected browsers, proxy/fingerprint rotation to evade bans, scraping behind
login walls, automated fake-account creation. This is the honest-accuracy moat.
## Layout
- `sources/connectors.py` — compliant connectors (GLEIF, Brønnøysund, Companies House live; PRH/CVR/Bolagsverket gated on verified terms → level 0 until wired).
- `sources/sources.yaml` — source registry (access/licence/cadence; VERIFY terms).
- `orchestrator/main.py` — REST API (health, sources, estimates, orders, jobs) per docs/04.
- `workers/adaptive_router.py` — picks the cheapest **compliant** method per market (api > bulk > authorized_web > licensed). No bypass tiers.
- `workers/pipeline.py` — sourcing → extraction → structuring (dedup) → enrichment/fusion → QA (drops rows below the order's confidence target). Writes provenance per field.
## Run locally
```bash
pip install -r requirements.txt
uvicorn orchestrator.main:app --reload # API on :8000
python -m workers.run # run one demo job
```
## Reconciliation note
Supersedes the earlier `nordic-eu-data-platform` backend: the compliant connectors +
pipeline are migrated here; the FlareSolverr/stealth/proxy/fake-account paths and the
`test_bypass_chain`/`test_proxy_pool` examples are intentionally **not** carried over.
+1
View File
@@ -0,0 +1 @@
# 0pt data engine — orchestrator package (FastAPI app).
+150
View File
@@ -0,0 +1,150 @@
"""
0pt data engine — FastAPI orchestrator (docs/04). Compliant sourcing only.
Runs on a VPS/worker host, NOT on Vercel (long jobs). Amounts in öre.
Reconciles the Next.js /api routes with the Python data engine: the Next layer
is the customer-facing edge; this service runs the actual pipeline.
"""
from __future__ import annotations
import os
import uuid
from pathlib import Path
from typing import Any
import yaml
from fastapi import FastAPI, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
from workers.pipeline import run_job
app = FastAPI(title="0pt data engine", version="0.1")
app.add_middleware(
CORSMiddleware, allow_origins=["*"], allow_methods=["*"], allow_headers=["*"],
)
_SOURCES = yaml.safe_load((Path(__file__).resolve().parent.parent / "sources" / "sources.yaml").read_text(encoding="utf-8"))["sources"]
# ── estimate model (mirrors apps/web lib/data.computeEstimate) ───────────────
FIELD_RATE = 240
ENTITY_MULT = {"Bolag": 1.0, "Beslutsfattare": 1.4, "Fordon": 1.2}
FIELD_WEIGHT = {"identity": 1.0, "lei": 0.8, "financials": 1.6, "ubo": 1.4, "contact": 1.8, "intent": 1.5}
ENRICH_WEIGHT = {"dedup": 0.1, "crm": 0.2, "sanctions": 0.15, "nace": 0.1, "tech": 0.15}
class EstimateIn(BaseModel):
markets: list[str] = []
entity_type: str = "Bolag"
fields: list[str] = []
enrichment: list[str] = []
volume: int = 1000
frequency: str = "Engångs"
confidence_target: int = 92
def compute_estimate(b: EstimateIn) -> dict[str, Any]:
markets = max(1, len(b.markets))
fields = set(b.fields) | {"identity"}
fw = sum(FIELD_WEIGHT.get(f, 0.5) for f in fields) or 1
em = ENTITY_MULT.get(b.entity_type, 1.0)
cm = 1 + max(0, b.confidence_target - 90) * 0.03
ew = sum(ENRICH_WEIGHT.get(e, 0) for e in b.enrichment)
base = fw * markets * FIELD_RATE * (b.volume / 1000) * em * cm
price = round(base * (1 + ew) / 5) * 5
deposit = round(price / 2)
days = min(14, max(1, round(1 + markets * 0.4 + b.volume / 6000 + len(fields) * 0.25)))
avg_level = sum(3 if f in ("identity", "lei") else 2 if f in ("financials", "ubo") else 1 for f in fields) / len(fields)
ceiling = round(82 + avg_level * 5)
est_conf = min(b.confidence_target, ceiling)
return {
"total_ore": price * 100, "deposit_ore": deposit * 100, "balance_ore": (price - deposit) * 100,
"est_days": days, "est_confidence": est_conf / 100, "recurring": b.frequency != "Engångs",
}
# ── in-memory store (swapped for asyncpg when DATABASE_URL is set) ───────────
_ORDERS: dict[str, dict[str, Any]] = {}
_JOBS: dict[str, dict[str, Any]] = {}
class OrderIn(BaseModel):
flow: str = "custom"
config: dict[str, Any] = {}
@app.get("/health")
async def health():
return {"ok": True, "service": "0pt-data-engine", "db": bool(os.getenv("DATABASE_URL")), "sources": len(_SOURCES)}
@app.get("/sources")
async def sources():
return [{k: s.get(k) for k in ("source_id", "name", "market", "access", "licence", "cadence")} for s in _SOURCES]
@app.get("/sources/{market}")
async def sources_for(market: str):
return [s for s in _SOURCES if s.get("market") in (market, "global")]
@app.post("/estimates")
async def estimates(body: EstimateIn):
return compute_estimate(body)
@app.post("/orders", status_code=201)
async def create_order(body: OrderIn):
est = compute_estimate(EstimateIn(
markets=body.config.get("markets", []),
entity_type=body.config.get("type", "Bolag"),
fields=body.config.get("fieldIds", []),
enrichment=body.config.get("enrichmentIds", []),
volume=int(body.config.get("volume", 1000)),
frequency=body.config.get("frequency", "Engångs"),
confidence_target=int(body.config.get("confidence", 92)),
))
oid = str(uuid.uuid4())
_ORDERS[oid] = {"order_id": oid, "flow": body.flow, "status": "draft", "config": body.config, **est}
return {"order_id": oid, "status": "draft", "deposit_ore": est["deposit_ore"], "total_ore": est["total_ore"]}
@app.get("/orders")
async def list_orders():
return list(_ORDERS.values())
@app.get("/orders/{oid}")
async def get_order(oid: str):
o = _ORDERS.get(oid)
if not o:
raise HTTPException(404, "order saknas")
return {**o, "job": _JOBS.get(oid)}
class StatusIn(BaseModel):
status: str
@app.patch("/orders/{oid}")
async def set_status(oid: str, body: StatusIn):
o = _ORDERS.get(oid)
if not o:
raise HTTPException(404, "order saknas")
o["status"] = body.status
return o
@app.post("/orders/{oid}/run")
async def run(oid: str):
o = _ORDERS.get(oid)
if not o:
raise HTTPException(404, "order saknas")
o["status"] = "in_production"
_JOBS[oid] = await run_job(oid, o["config"])
o["status"] = "review"
return _JOBS[oid]
@app.get("/orders/{oid}/status")
async def job_status(oid: str):
return _JOBS.get(oid) or {"stage": "queued", "pct": 0, "state": "queued"}
+10
View File
@@ -0,0 +1,10 @@
# 0pt data engine — compliant stack only (docs/07). No bypass/stealth/proxy libs.
fastapi==0.115.6
uvicorn[standard]==0.34.0
pydantic==2.10.4
httpx==0.28.1
asyncpg==0.30.0
python-dotenv==1.0.1
pyyaml==6.0.2
rapidfuzz==3.11.0
tenacity==9.0.0
+1
View File
@@ -0,0 +1 @@
# 0pt data engine — sources package (connectors + registry).
+161
View File
@@ -0,0 +1,161 @@
"""
0pt data engine — COMPLIANT source connectors.
Legal sourcing only (CLAUDE.md §1, docs/06). Priority per field/market:
1. official API / bulk 2. authorized public web (robots + rate limits)
3. licensed partner data 4. otherwise -> level 0 "ej tillgänglig / licens".
EXPLICITLY NOT PRESENT (and must never be added here):
WAF/bot bypass, FlareSolverr, stealth/undetected browsers, proxy/fingerprint
rotation to evade bans, scraping behind login walls, fake-account creation.
Every connector returns a normalized company dict + per-field provenance rows
(source_id, confidence 0-1, level 0-3, freshness) so nothing ships unsourced.
Terms change — VERIFY each source's current access/licence before relying.
"""
from __future__ import annotations
import os
from dataclasses import dataclass, field
from datetime import date
from typing import Any
import httpx
# A polite client that identifies itself and obeys timeouts/rate limits.
USER_AGENT = "0pt-data-engine/0.1 (+https://0pt.app; SIAX Technology AB; hej@siax.io)"
_TIMEOUT = httpx.Timeout(15.0, connect=10.0)
def _client() -> httpx.AsyncClient:
return httpx.AsyncClient(
headers={"User-Agent": USER_AGENT, "Accept": "application/json"},
timeout=_TIMEOUT,
follow_redirects=True,
)
@dataclass
class Provenance:
field_name: str
source_id: str
confidence: float
level: int # 0 unavailable/licensed, 1 single, 2 partial, 3 corroborated
freshness: str | None = None
@dataclass
class CompanyResult:
org_nr: str | None = None
market: str | None = None
lei: str | None = None
name: str | None = None
status: str | None = None
address: dict[str, Any] | None = None
website: str | None = None
provenance: list[Provenance] = field(default_factory=list)
source: str = ""
def _today() -> str:
return date.today().isoformat()
# ── GLEIF / LEI (global cross-border key) — open API ────────────────────────
async def gleif_by_lei(lei: str) -> CompanyResult | None:
url = f"https://api.gleif.org/api/v1/lei-records/{lei}"
async with _client() as c:
r = await c.get(url, headers={"Accept": "application/vnd.api+json"})
if r.status_code != 200:
return None
a = (r.json().get("data") or {}).get("attributes") or {}
entity = a.get("entity") or {}
legal = entity.get("legalAddress") or {}
res = CompanyResult(
lei=lei,
name=(entity.get("legalName") or {}).get("name"),
status=entity.get("status"),
address={"lines": legal.get("addressLines"), "city": legal.get("city"),
"country": legal.get("country"), "postal": legal.get("postalCode")},
source="gleif",
)
f = _today()
res.provenance += [
Provenance("lei", "gleif", 0.99, 3, f),
Provenance("name", "gleif", 0.97, 3, f),
]
return res
# ── Brønnøysundregistrene (NO) — open API (NLOD). VERIFY terms. ─────────────
async def brreg_by_orgnr(orgnr: str) -> CompanyResult | None:
url = f"https://data.brreg.no/enhetsregisteret/api/enheter/{orgnr}"
async with _client() as c:
r = await c.get(url)
if r.status_code != 200:
return None
d = r.json()
res = CompanyResult(
org_nr=orgnr, market="NO",
name=d.get("navn"),
status="dissolved" if d.get("konkurs") else "active",
address={"raw": d.get("forretningsadresse")},
source="brreg",
)
f = _today()
res.provenance += [
Provenance("name", "brreg", 0.97, 3, f),
Provenance("status", "brreg", 0.95, 3, f),
Provenance("address", "brreg", 0.95, 3, f),
]
return res
# ── Companies House (UK) — free API, key required. VERIFY terms. ────────────
async def companies_house_by_number(number: str) -> CompanyResult | None:
key = os.getenv("COMPANIES_HOUSE_API_KEY")
if not key:
# No key -> deliver honestly as unavailable, never fabricate.
return CompanyResult(org_nr=number, market="GB", source="companies_house",
provenance=[Provenance("name", "companies_house", 0.0, 0, _today())])
url = f"https://api.company-information.service.gov.uk/company/{number}"
async with _client() as c:
r = await c.get(url, auth=(key, ""))
if r.status_code != 200:
return None
d = r.json()
res = CompanyResult(
org_nr=number, market="GB",
name=d.get("company_name"),
status=d.get("company_status"),
address={"raw": d.get("registered_office_address")},
source="companies_house",
)
f = _today()
res.provenance += [
Provenance("name", "companies_house", 0.97, 3, f),
Provenance("status", "companies_house", 0.95, 3, f),
]
return res
# Registries that require verified terms / keys before bulk use.
# Implement as official-API connectors only; never bypass. Until wired they
# return level-0 so the field is delivered honestly as "ej tillgänglig".
async def prh_by_businessid(business_id: str) -> CompanyResult | None: # FI — VERIFY
return CompanyResult(org_nr=business_id, market="FI", source="prh",
provenance=[Provenance("name", "prh", 0.0, 0, _today())])
async def cvr_by_number(cvr: str) -> CompanyResult | None: # DK — VERIFY (Virk terms)
return CompanyResult(org_nr=cvr, market="DK", source="cvr",
provenance=[Provenance("name", "cvr", 0.0, 0, _today())])
CONNECTORS = {
"gleif": gleif_by_lei,
"brreg": brreg_by_orgnr,
"companies_house": companies_house_by_number,
"prh": prh_by_businessid,
"cvr": cvr_by_number,
}
+99
View File
@@ -0,0 +1,99 @@
# 0pt data engine — sources registry (docs/06). Drives the router, /sources API,
# and the public sourcing-map. access: api | bulk | authorized_web | licensed.
# VERIFY current access/licence/robots per source before relying — terms change.
sources:
- source_id: gleif
name: GLEIF (LEI)
market: global
access: api
cadence: daily
licence: "open (CC0) — VERIFY"
entities: [company]
fields: [lei, legal_name, address, status]
base_url: https://api.gleif.org
robots_ok: true
notes: "cross-border key"
- source_id: brreg
name: Brønnøysundregistrene (Enhetsregisteret)
market: NO
access: api
cadence: daily
licence: "NLOD open — VERIFY"
entities: [company]
fields: [name, status, address, nace]
base_url: https://data.brreg.no
robots_ok: true
notes: "open API + bulk download"
- source_id: companies_house
name: Companies House
market: GB
access: api
cadence: daily
licence: "free, API key required — VERIFY"
entities: [company, officers, financials]
fields: [name, status, address, officers]
base_url: https://api.company-information.service.gov.uk
robots_ok: true
notes: "key via env COMPANIES_HOUSE_API_KEY"
- source_id: prh
name: PRH avoindata (Trade Register)
market: FI
access: api
cadence: daily
licence: "CC-BY — VERIFY"
entities: [company]
fields: [name, status]
base_url: https://avoindata.prh.fi
robots_ok: true
notes: "connector pending verification"
- source_id: cvr
name: CVR / Virk
market: DK
access: api
cadence: daily
licence: "terms apply (Virk) — VERIFY"
entities: [company]
fields: [name, status]
base_url: https://datacvr.virk.dk
robots_ok: true
notes: "connector pending verification"
- source_id: bolagsverket
name: Bolagsverket
market: SE
access: licensed
cadence: daily
licence: "restricted for bulk — partner/licensed route"
entities: [company]
fields: [name, status]
base_url: https://bolagsverket.se
robots_ok: true
notes: "no bulk scrape; licensed/partner only"
- source_id: rdw
name: RDW open vehicle data
market: NL
access: bulk
cadence: daily
licence: "open — VERIFY"
entities: [vehicle]
fields: [make, model, year, specs]
base_url: https://opendata.rdw.nl
robots_ok: true
notes: "technical/fleet vehicle data, not owner PII"
- source_id: opensanctions
name: OpenSanctions
market: global
access: bulk
cadence: daily
licence: "open — VERIFY commercial terms"
entities: [company, person]
fields: [sanction, pep]
base_url: https://www.opensanctions.org
robots_ok: true
notes: "screening/diligence enrichment"
+1
View File
@@ -0,0 +1 @@
# 0pt data engine — workers package (router + pipeline).
Binary file not shown.
+23
View File
@@ -0,0 +1,23 @@
"""
Adaptive router — chooses among COMPLIANT methods only (docs/06).
Order of preference: official API > bulk > authorized public web > licensed.
There are NO bypass/stealth/proxy-rotation tiers here, by design (CLAUDE.md §1).
"""
from __future__ import annotations
# market -> ordered list of source_ids to try (compliant connectors only)
MARKET_SOURCES: dict[str, list[str]] = {
"NO": ["brreg", "gleif"],
"GB": ["companies_house", "gleif"],
"FI": ["prh", "gleif"],
"DK": ["cvr", "gleif"],
"SE": ["gleif"], # Bolagsverket bulk is licensed; SE company core via GLEIF + licensed/partner
}
# Access rank: lower = preferred (cheapest compliant path first).
ACCESS_RANK = {"api": 0, "bulk": 1, "authorized_web": 2, "licensed": 3}
def plan_for_market(market: str) -> list[str]:
"""Return the ordered, compliant source plan for a market."""
return MARKET_SOURCES.get(market, ["gleif"])
+89
View File
@@ -0,0 +1,89 @@
"""
Production pipeline (docs/06): sourcing -> extraction -> structuring ->
enrichment/fusion -> QA. Compliant connectors only; every field gets provenance;
rows below the order's confidence target are dropped (deliver fewer, never fabricate).
"""
from __future__ import annotations
from typing import Any
from rapidfuzz import fuzz
from sources.connectors import (
CompanyResult,
brreg_by_orgnr,
companies_house_by_number,
gleif_by_lei,
)
from workers.adaptive_router import plan_for_market
# Minimal built-in seed so a job is demonstrable without customer input.
DEMO_SEEDS: dict[str, list[str]] = {
"NO": ["923609016"], # brreg orgnr (example)
"GB": ["00000006"], # companies house number (example)
}
async def _fetch(source_id: str, key: str) -> CompanyResult | None:
if source_id == "gleif":
return await gleif_by_lei(key)
if source_id == "brreg":
return await brreg_by_orgnr(key)
if source_id == "companies_house":
return await companies_house_by_number(key)
return None
async def run_job(order_id: str, config: dict[str, Any]) -> dict[str, Any]:
markets: list[str] = config.get("markets", []) or ["NO"]
threshold = float(config.get("confidence", 92)) / 100.0
seeds: dict[str, list[str]] = config.get("seeds") or DEMO_SEEDS
records: list[dict[str, Any]] = []
per_market: dict[str, int] = {}
# 1) sourcing + 2) extraction
for market in markets:
keys = seeds.get(market, [])
plan = plan_for_market(market)
got = 0
for key in keys:
for source_id in plan:
try:
res = await _fetch(source_id, key)
except Exception:
res = None
if res and res.name:
records.append({
"market": market, "name": res.name, "status": res.status,
"lei": res.lei, "org_nr": res.org_nr, "source": res.source,
"provenance": [p.__dict__ for p in res.provenance],
"confidence": max((p.confidence for p in res.provenance), default=0.0),
})
got += 1
break # first compliant source that answers wins
per_market[market] = got
# 3) structuring — dedup by fuzzy name within market
deduped: list[dict[str, Any]] = []
for r in records:
dup = next((d for d in deduped if d["market"] == r["market"] and fuzz.ratio(d["name"], r["name"]) > 92), None)
if dup:
dup["confidence"] = min(0.99, dup["confidence"] + 0.1) # corroboration bump
else:
deduped.append(r)
# 4) enrichment/fusion already folded into confidence above.
# 5) QA — drop rows below the order's confidence threshold.
delivered = [r for r in deduped if r["confidence"] >= threshold]
dropped = len(deduped) - len(delivered)
avg_conf = round(sum(r["confidence"] for r in delivered) / len(delivered), 3) if delivered else 0.0
return {
"stage": "qa", "pct": 100, "state": "done",
"metrics": {
"rows": len(delivered), "dropped_below_threshold": dropped,
"per_market": per_market, "avg_confidence": avg_conf,
},
"records": delivered,
}
+21
View File
@@ -0,0 +1,21 @@
"""
Worker entrypoint. In production this drains a queue of order jobs and runs the
compliant pipeline. For now it exposes a CLI to run a single job locally.
python -m workers.run <order_id>
"""
from __future__ import annotations
import asyncio
import sys
from workers.pipeline import run_job
async def _main() -> None:
order_id = sys.argv[1] if len(sys.argv) > 1 else "demo"
result = await run_job(order_id, {"markets": ["NO", "GB"], "confidence": 90})
print(result["metrics"])
if __name__ == "__main__":
asyncio.run(_main())