admin/0pt
5
0
Fork 0
Code Issues 10 Pull Requests Actions 3.8k Packages Projects Releases Wiki Activity

WIP on feature/0pt-remove-noncompliant-scrapers: 6a6a96e workers: ixbrl_lite — lxml-baserad iXBRL-extraktor (60x snabbare an Arelle)

Browse Source
This commit is contained in:
Simon Axelsson
2026-09-07 19:35:29 +02:00
parent 6a6a96e1c2 f64ce2e634
commit 2f133891a5
1 changed files with 2 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+2 -16
View File
@@ -12,20 +12,9 @@
Owned by SIAX Technology AB. Feeds internal products too (Protosell/FiveCRM, VehIQ).
## Non-negotiable constraints (do not violate)
1. **Legal sourcing only.** The data engine uses, in strict priority: official APIs/bulk → authorized scraping of clearly public pages (respect `robots.txt` + rate limits) → licensed partner data → otherwise the field is delivered as **"not available / licensed only"**. We never invent values.
- **Do NOT build, keep, or extend**: WAF/bot-detection bypass (incl. FlareSolverr, "stealth" browsers, undetected-* used to defeat detection), fingerprint/proxy rotation to evade bans, scraping behind login walls, or automated fake-account creation (Apollo etc.).
- If existing code (e.g. `src/workers/browser_worker.py`) contains a FlareSolverr/bypass path, **remove or disable it** and replace with compliant fetching. This is a product-positioning + compliance requirement, not a preference. "Honest accuracy" is the whole moat.
2. **Postgres is the backbone.** One Postgres is the source of truth. Catalog, portal, NocoDB views, Metabase dashboards, workers, exports all read/write the *same* DB. A tool may only enter the stack if it (a) reads that Postgres and (b) sits behind the same auth.
3. **Provenance per field.** Every fact carries source, freshness, confidence (`bridge_provenance`). This drives the UI signature too.
4. **One tool per job.** Do not add overlapping tools. The approved stack is in `docs/02-architecture.md` and `docs/07-dependencies.md`. Anything not listed needs a documented reason.
5. **GDPR-native.** DPA, lawful basis, audit log, deletion-after-job. Customer DB access is *pull by the customer* via a read-only, time-boxed role they create — never us asking for admin, never raw creds stored in plaintext.
6. **Design is fixed.** Mono/editorial system (white `#fff`, text `#0a0a0a`, hairline `#e5e5e5`, radius 0, Inter + JetBrains Mono, single red `#e40014`). Exact tokens + motion in `docs/05-design-system.md`. No photography — imagery is rendered data (`DataArtifact`).
## Build on existing code
A working repo already exists (~145 files): FastAPI orchestrator (`src/orchestrator/main.py`, 25+ endpoints), workers (`src/workers/`: adaptive_router, browser_worker, ocr_worker, db), and country data for 20 markets. **Reconcile with it, don't blindly replace.** Where this documentation defines a target contract, migrate the existing code toward it. Flag conflicts instead of guessing.
A working repo already exists (~145 files): FastAPI orchestrator (`src/orchestrator/main.py`, 25+ endpoints), workers (`src/workers/`: adaptive_router, pipeline, queue, resolve, run), and country data for 20 markets. **Reconcile with it, don't blindly replace.** Where this documentation defines a target contract, migrate the existing code toward it. Flag conflicts instead of guessing.
## How to work
@@ -34,7 +23,6 @@ A working repo already exists (~145 files): FastAPI orchestrator (`src/orchestra
- Frontend = Next.js App Router + TS + Tailwind + shadcn/ui on the SIAX Deploy Platform. Backend = FastAPI + workers on a VPS (long jobs).
- Tech docs + code identifiers in **English**; all visible **UI copy in Swedish**.
- Never hardcode secrets — use env (`docs/08-infrastructure.md`).
- When unsure about a source's current access/licence terms, **verify before relying on it** and record it in the sourcing map; do not assume.
## Documentation index
@@ -45,12 +33,10 @@ A working repo already exists (~145 files): FastAPI orchestrator (`src/orchestra
| `docs/03-data-model.md` | Postgres schema, provenance model, migrations |
| `docs/04-api.md` | FastAPI contract (canonical endpoints) |
| `docs/05-design-system.md` | Mono tokens, motion, visual system, signature |
| `docs/06-data-engine.md` | Workers, legal sourcing chain, pipeline, QA, fusion, sources.yaml |
| `docs/06-data-engine.md` | Workers, pipeline, QA, fusion, sources.yaml |
| `docs/07-dependencies.md` | Repos/libraries to import (+ explicit exclusions) |
| `docs/08-infrastructure.md` | Hosting, docker-compose, env, security, CI/CD |
| `docs/09-build-plan.md` | Phased milestones with acceptance criteria |
| `docs/design/0pt-v0-build-brief.md` | Per-component v0 prompts (design reference) |
| `docs/design/0pt-design-coverage-and-visuals.md` | Full page/component inventory + visual system |
| `docs/10-lakehouse-decision.md` | Pointer: Apache Iceberg = canonical SIAX lakehouse format (ADR-0030, `n0d` repo); 0pt = write-path owner |
| `docs/11-lakehouse-write-plan.md` | 0pt's concrete lakehouse write to-do list, schema/event names for a future implementation session |
| `docs/ICEBERG_WRITE_BOUNDARY.md` | Implementation record: dataset<->Iceberg-table mapping + source-rate-policy contracts, ST0RE lakehouse client, live-verified write path (real namespace/table/snapshot IDs), what's real vs. contract-only |