chore(salvage): preserve uncommitted WIP from /opt/vehiq-work/0pt-p060-p068_0pt before machine cleanup (2026-09-27)
CI (SIAX Cloud) / quality (push) Successful in 49s
CI (SIAX Cloud) / security (push) Successful in 7m5s
Security Scan (SIAX Cloud) / security (push) Successful in 7m5s
CI (SIAX Cloud) / integration (push) Successful in 18m53s
CI (SIAX Cloud) / security (pull_request) Successful in 46s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 58s
CI (SIAX Cloud) / integration (pull_request) Successful in 1m38s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m45s
CI (SIAX Cloud) / quality (push) Successful in 49s
CI (SIAX Cloud) / security (push) Successful in 7m5s
Security Scan (SIAX Cloud) / security (push) Successful in 7m5s
CI (SIAX Cloud) / integration (push) Successful in 18m53s
CI (SIAX Cloud) / security (pull_request) Successful in 46s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 58s
CI (SIAX Cloud) / integration (pull_request) Successful in 1m38s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m45s
This snapshot was found sitting uncommitted in a stale /opt/vehiq-work lane clone dated ~2026-09-09..18, base branch: feat/p060-p068-ingestion-plane. Not reviewed, not verified against current main. Pushed only so the directory could be deleted without losing the diff.
This commit is contained in:
+13
-11
@@ -1,24 +1,25 @@
|
||||
# 0pt data engine (`services/api`)
|
||||
|
||||
The legal data motor (docs/06). FastAPI orchestrator + worker pipeline that turn
|
||||
**compliant** sources into delivered, provenance-tagged data. Runs on a VPS via
|
||||
The governed data motor (docs/06). FastAPI orchestrator + worker pipeline that turn
|
||||
configured sources into delivered, provenance-tagged data. Runs on a VPS via
|
||||
[`../../infra/docker-compose.yml`](../../infra/docker-compose.yml) — **not on Vercel** (long jobs).
|
||||
|
||||
## Sourcing boundary (CLAUDE.md §1 — non-negotiable)
|
||||
## Sourcing and transport boundary
|
||||
|
||||
Per field × market, in strict order: **official API/bulk → authorized public web
|
||||
(robots + rate limits) → licensed partner data → otherwise level 0 "ej tillgänglig"**.
|
||||
Per field × market, the standard profile prefers **official API/bulk → public web
|
||||
→ partner data → otherwise level 0 "ej tillgänglig"**.
|
||||
We never invent values.
|
||||
|
||||
**Deliberately excluded (never add here):** WAF/bot-detection bypass, FlareSolverr,
|
||||
stealth/undetected browsers, proxy/fingerprint rotation to evade bans, scraping behind
|
||||
login walls, automated fake-account creation. This is the honest-accuracy moat.
|
||||
**Canonical ownership:** WAF/challenge handling, FlareSolverr-compatible adapters,
|
||||
stealth browsers, proxy/fingerprint rotation, authenticated sessions and account
|
||||
automation are valid optional profiles owned by b0ts + 1p. They are selected through
|
||||
explicit contracts and are not duplicated inside the standard 0pt worker pipeline.
|
||||
|
||||
## Layout
|
||||
- `sources/connectors.py` — compliant connectors (GLEIF, Brønnøysund, Companies House live; PRH/CVR/Bolagsverket gated on verified terms → level 0 until wired).
|
||||
- `sources/sources.yaml` — source registry (access/licence/cadence; VERIFY terms).
|
||||
- `orchestrator/main.py` — REST API (health, sources, estimates, orders, jobs) per docs/04.
|
||||
- `workers/adaptive_router.py` — picks the cheapest **compliant** method per market (api > bulk > authorized_web > licensed). No bypass tiers.
|
||||
- `workers/adaptive_router.py` — picks the cheapest standard method per market (api > bulk > authorized_web > licensed). High-risk transport profiles are delegated to b0ts + 1p.
|
||||
- `workers/pipeline.py` — sourcing → extraction → structuring (dedup) → enrichment/fusion → QA (drops rows below the order's confidence target). Writes provenance per field.
|
||||
|
||||
## Run locally
|
||||
@@ -30,5 +31,6 @@ python -m workers.run # run one demo job
|
||||
|
||||
## Reconciliation note
|
||||
Supersedes the earlier `nordic-eu-data-platform` backend: the compliant connectors +
|
||||
pipeline are migrated here; the FlareSolverr/stealth/proxy/fake-account paths and the
|
||||
`test_bypass_chain`/`test_proxy_pool` examples are intentionally **not** carried over.
|
||||
pipeline are migrated here; FlareSolverr/stealth/proxy/account-session paths and the
|
||||
`test_bypass_chain`/`test_proxy_pool` examples belong to b0ts + 1p instead of being
|
||||
duplicated in this package.
|
||||
|
||||
Reference in New Issue
Block a user