Files
0pt/services/api/orchestrator/main.py
T
adminandClaude Opus 4.8 e6d3101501 fix: 17 confirmed bugs from full-repo audit (edge + Python engine) (#5)
* docs: correct stale quality-check-service.ts comment references

Two source comments still cited src/services/quality-check-service.ts as an
existing precedent. It doesn't exist anywhere in this repo's history (git
log --all: zero commits, any branch, ever) -- confirmed during the earlier
full-doc-corpus accuracy pass. Points each comment at the real precedent
instead (document-extraction-service.ts / iceberg-write-service.ts).

Applied from task_05ec36aa, spun off during that pass.

* fix: 17 confirmed bugs from full-repo audit (edge + Python engine)

Found by an 8-finder bug hunt across app/api, src/services, src/clients,
frontend, the Python engine, the data layer, config/infra, and tests, then
independently adversarially verified per finding (32 raw findings -> 27
confirmed real bugs; 5 refuted). This commit applies the 17 confirmed as
safe, mechanical fixes -- no architecture/design decision required. The
other 10 need a decision and are reported separately, not touched here.

Most consequential, app/api/payments/[order_id]/deposit/route.ts:
the client-supplied Origin request header was trusted unsanitized both as
a server-side fetch target (SSRF) and as Stripe Checkout success/cancel
redirect URLs (open redirect tied to a real payment flow), and a failed
order lookup silently fell through to a hardcoded deposit amount instead
of 404. Fixed by reading the order directly via lib/orders-store.ts
(mirroring the sibling release/route.ts, which already did this correctly)
instead of both trusting the Origin header and self-issuing an HTTP fetch.

Also fixed:
- 5 routes (orders, orders/[id], estimates, subscriptions, ai/parse) that
  500'd instead of 400ing on a literal JSON `null` body, since req.json()
  resolves (not rejects) for that input and bypasses the try/catch guard.
- app/api/0pt/lakehouse/mappings/{,[mappingId]/{,writes/}}route.ts each
  instantiated their own private service instances, so a mapping created
  via POST was invisible to the sibling GET and writes routes even before
  the separately-tracked dataset-registration gap. Extracted a shared
  singleton module, src/services/lakehouse-mapping-registry.ts.
- app/api/0pt/documents/[documentId]/extraction/route.ts collapsed every
  error (not-configured, policy-denied, policy-approval-required,
  not-found) into a blanket 400; now status-mapped like the sibling
  sources/[sourceId]/query route.
- app/api/vehicles/route.ts: plate query param wasn't URL-encoded before
  being concatenated into the outbound RDW request, allowing extra query
  parameters to be injected; added a strict allowlist + encodeURIComponent.
- app/api/uploads/route.ts: POST returned a fake 200 success with
  put_url: null even when "configured", instead of continuing to report
  501 until presign generation is actually implemented.
- src/adapters/document-intelligence-adapter.ts: the result-polling loop
  never checked response.ok, so a non-2xx poll response (auth expiry,
  429, 5xx) was silently retried until a generic timeout masked the real
  error.
- app/order/[id]/page.tsx: finish() never cleared the production-simulation
  setInterval, so it kept firing (and re-rendering) forever after delivery.
- app/admin/page.tsx: the metrics fetch had no response/shape validation
  (unlike the adjacent orders fetch), rendering literal "undefined" stat
  tiles on a malformed response instead of falling back to <ErrorState/>.
- services/api/workers/pipeline.py: `config.get("seeds") or DEMO_SEEDS`
  used truthy-or, so an explicit empty seeds dict was silently replaced
  with hardcoded demo org numbers instead of being respected as empty.
- services/api/workers/resolve.py: corroboration (confidence/agreed_sources
  bump) no longer fires for two hits from the same source under fuzzy
  match -- only a genuinely independent source now counts, per docs/06's
  own ">=2 independent sources agree" definition -- and a merged record's
  provenance array now actually includes the corroborating source's
  fields instead of only a bare source-name string.
- services/api/orchestrator/main.py: create_order's int(...) on
  unvalidated client volume/confidence fields raised on any
  decimal-formatted input (int("92.5") -> ValueError); now coerced via
  float() first with a safe default. /vehicles/{plate} and /screen also
  now catch connector exceptions and degrade gracefully, matching
  pipeline.py's existing pattern for the same connectors.
- lib/orders-store.ts: postgres.js (pinned v3.4.9) returns bigint/numeric
  columns as strings by default; getOrder/listOrders/setStatus were
  silently handing callers stringified money fields. Added a narrow
  per-row coercion instead of touching the shared client's global type
  map (which would affect unrelated tables' bigint/numeric columns too).
- lib/subscriptions-store.ts: the in-memory fallback returned
  oldest-first while the Postgres branch explicitly orders newest-first;
  now consistent, mirroring the identical fix already in orders-store.ts.
- app/api/auth/login/route.ts: DEV_LOGIN_KEY was compared with a plain
  !==, a non-constant-time comparison; replaced with crypto.timingSafeEqual.

Verified before commit: tsc --noEmit clean; vitest run 88 passed, 1 todo,
0 failures (13 files, unchanged from before this commit -- no existing
test's assertions needed to change). services/api/workers/resolve.py's
source-novelty and provenance-merge logic additionally verified by direct
execution against synthetic same-source/cross-source inputs (no Python
test suite exists in this repo to add to). services/api/workers/pipeline.py
and services/api/orchestrator/main.py verified with python -m py_compile
(no httpx/fastapi installed in this environment to run them directly).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(engine): treat explicit seeds:null as missing to avoid 500 in run_job

Four-eyes follow-up to this PR's own seeds change. Switching from
`config.get('seeds') or DEMO_SEEDS` to `config['seeds'] if 'seeds' in config`
correctly stopped an explicit empty {} from being backfilled with DEMO_SEEDS,
but regressed on client-controlled `{"seeds": null}`: seeds became None and
the downstream `seeds.get(market, [])` raised AttributeError -> 500. That path
is reachable (POST /orders stores raw body.config; POST /orders/{id}/run passes
it straight to run_job). Using `config.get('seeds') is not None` keeps the
intended behaviour (explicit {} honoured, never silently seeded with demo data)
while restoring the None-tolerance the old idiom had.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-22 18:52:19 +02:00

179 lines
5.7 KiB
Python

"""
0pt data engine — FastAPI orchestrator (docs/04). Compliant sourcing only.
Runs on a VPS/worker host, NOT on Vercel (long jobs). Amounts in öre.
Reconciles the Next.js /api routes with the Python data engine: the Next layer
is the customer-facing edge; this service runs the actual pipeline.
"""
from __future__ import annotations
import os
import uuid
from pathlib import Path
from typing import Any
import yaml
from fastapi import FastAPI, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
from sources.connectors import opensanctions_screen, rdw_vehicle
from workers.pipeline import run_job
app = FastAPI(title="0pt data engine", version="0.1")
app.add_middleware(
CORSMiddleware, allow_origins=["*"], allow_methods=["*"], allow_headers=["*"],
)
_SOURCES = yaml.safe_load((Path(__file__).resolve().parent.parent / "sources" / "sources.yaml").read_text(encoding="utf-8"))["sources"]
def _coerce_int(v: Any, default: int) -> int:
"""int(v) raises on decimal-formatted strings/floats; coerce via float() first
and fall back to `default` on any unparseable, client-controlled input instead
of a 500."""
try:
return int(float(v))
except (TypeError, ValueError):
return default
# ── estimate model (mirrors apps/web lib/data.computeEstimate) ───────────────
FIELD_RATE = 240
ENTITY_MULT = {"Bolag": 1.0, "Beslutsfattare": 1.4, "Fordon": 1.2}
FIELD_WEIGHT = {"identity": 1.0, "lei": 0.8, "financials": 1.6, "ubo": 1.4, "contact": 1.8, "intent": 1.5}
ENRICH_WEIGHT = {"dedup": 0.1, "crm": 0.2, "sanctions": 0.15, "nace": 0.1, "tech": 0.15}
class EstimateIn(BaseModel):
markets: list[str] = []
entity_type: str = "Bolag"
fields: list[str] = []
enrichment: list[str] = []
volume: int = 1000
frequency: str = "Engångs"
confidence_target: int = 92
def compute_estimate(b: EstimateIn) -> dict[str, Any]:
markets = max(1, len(b.markets))
fields = set(b.fields) | {"identity"}
fw = sum(FIELD_WEIGHT.get(f, 0.5) for f in fields) or 1
em = ENTITY_MULT.get(b.entity_type, 1.0)
cm = 1 + max(0, b.confidence_target - 90) * 0.03
ew = sum(ENRICH_WEIGHT.get(e, 0) for e in b.enrichment)
base = fw * markets * FIELD_RATE * (b.volume / 1000) * em * cm
price = round(base * (1 + ew) / 5) * 5
deposit = round(price / 2)
days = min(14, max(1, round(1 + markets * 0.4 + b.volume / 6000 + len(fields) * 0.25)))
avg_level = sum(3 if f in ("identity", "lei") else 2 if f in ("financials", "ubo") else 1 for f in fields) / len(fields)
ceiling = round(82 + avg_level * 5)
est_conf = min(b.confidence_target, ceiling)
return {
"total_ore": price * 100, "deposit_ore": deposit * 100, "balance_ore": (price - deposit) * 100,
"est_days": days, "est_confidence": est_conf / 100, "recurring": b.frequency != "Engångs",
}
# ── in-memory store (swapped for asyncpg when DATABASE_URL is set) ───────────
_ORDERS: dict[str, dict[str, Any]] = {}
_JOBS: dict[str, dict[str, Any]] = {}
class OrderIn(BaseModel):
flow: str = "custom"
config: dict[str, Any] = {}
@app.get("/health")
async def health():
return {"ok": True, "service": "0pt-data-engine", "db": bool(os.getenv("DATABASE_URL")), "sources": len(_SOURCES)}
@app.get("/sources")
async def sources():
return [{k: s.get(k) for k in ("source_id", "name", "market", "access", "licence", "cadence")} for s in _SOURCES]
@app.get("/sources/{market}")
async def sources_for(market: str):
return [s for s in _SOURCES if s.get("market") in (market, "global")]
@app.get("/vehicles/{plate}")
async def vehicle(plate: str):
try:
v = await rdw_vehicle(plate)
except Exception:
v = None
return v or {"vehicle": None}
@app.get("/screen")
async def screen(q: str):
try:
return await opensanctions_screen(q)
except Exception:
return {"matches": [], "source": "opensanctions"}
@app.post("/estimates")
async def estimates(body: EstimateIn):
return compute_estimate(body)
@app.post("/orders", status_code=201)
async def create_order(body: OrderIn):
est = compute_estimate(EstimateIn(
markets=body.config.get("markets", []),
entity_type=body.config.get("type", "Bolag"),
fields=body.config.get("fieldIds", []),
enrichment=body.config.get("enrichmentIds", []),
volume=_coerce_int(body.config.get("volume", 1000), 1000),
frequency=body.config.get("frequency", "Engångs"),
confidence_target=_coerce_int(body.config.get("confidence", 92), 92),
))
oid = str(uuid.uuid4())
_ORDERS[oid] = {"order_id": oid, "flow": body.flow, "status": "draft", "config": body.config, **est}
return {"order_id": oid, "status": "draft", "deposit_ore": est["deposit_ore"], "total_ore": est["total_ore"]}
@app.get("/orders")
async def list_orders():
return list(_ORDERS.values())
@app.get("/orders/{oid}")
async def get_order(oid: str):
o = _ORDERS.get(oid)
if not o:
raise HTTPException(404, "order saknas")
return {**o, "job": _JOBS.get(oid)}
class StatusIn(BaseModel):
status: str
@app.patch("/orders/{oid}")
async def set_status(oid: str, body: StatusIn):
o = _ORDERS.get(oid)
if not o:
raise HTTPException(404, "order saknas")
o["status"] = body.status
return o
@app.post("/orders/{oid}/run")
async def run(oid: str):
o = _ORDERS.get(oid)
if not o:
raise HTTPException(404, "order saknas")
o["status"] = "in_production"
_JOBS[oid] = await run_job(oid, o["config"])
o["status"] = "review"
return _JOBS[oid]
@app.get("/orders/{oid}/status")
async def job_status(oid: str):
return _JOBS.get(oid) or {"stage": "queued", "pct": 0, "state": "queued"}