Re-runs this session's task spec against 0pt's actual repo state, which
turned out to already have a substantial, tested implementation from a
prior pass (35b6ec6): dataset<->Iceberg-table mapping, source rate policy,
and an ST0RE lakehouse client that was honestly marked "not live-verified".
This pass found api.st0re.siax.io IS reachable from this environment and
used it -- first via curl to safely discover the real contract, then
through 0pt's own updated code -- to drive a real namespace -> table ->
snapshot write against production ST0RE (task item 3's "write ONE first
test dataset" proof). That surfaced and fixed three real bugs the
mock-only pass could not have caught:
- ST0RE requires X-Tenant-Id/X-Actor-Id headers on every lakehouse route;
the previous HTTP client sent none. Fixed via a threaded
St0reRequestContext.
- IcebergTableInput.datasetId is a foreign key into ST0RE's OWN dataset
table, not a free-form string -- sending 0pt's dataset_id 400s live with
a Postgres FK violation. Fixed by no longer sending it; 0pt's
dataset<->table linkage lives entirely in DatasetIcebergMapping.
- ST0RE enforces a real unique constraint on lakehouse namespaces
(tenant_id, project_id, name) with no upsert route. Fixed with a
client-side lookup-or-create (listNamespaces + findOrRegisterNamespace).
Also closed a field-level gap against the task's literal schema spec:
0pt.dataset_iceberg_mapping.schema.v1 now carries namespace_id (a real
reference to ST0RE's LakehouseNamespace.namespaceId, not guessed) and
mapped_at, both additive alongside the existing namespace/created_at
fields (sql/005, JSON schema, types, service).
tests/p0/iceberg-write-service.realinfra.test.ts replaces the prior
it.todo with a real SEAM-pattern live test: attempts the real HTTP call,
skips honestly if ST0RE is unreachable, never fabricates success. It ran
successfully against production ST0RE this session (real namespace/table/
snapshot UUIDs logged in docs/ICEBERG_WRITE_BOUNDARY.md).
Two ST0RE-side cross-tenant read bugs were found and flagged (not fixed
here -- out of scope for this repo): GET .../namespaces ignores its own
tenant header and query filters; GET .../tables/:id doesn't check the
caller's tenant against the table's owner.
npm test: 155 passed, 0 failed, 0 todo (up from 154 passed + 1 todo).
npx tsc --noEmit: 1 pre-existing, unrelated error in
quality-report-service.ts (present before this pass, part of the wider
uncommitted P0 data-platform build already in this repo) -- not touched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
61 lines
5.5 KiB
Markdown
61 lines
5.5 KiB
Markdown
# CLAUDE.md — 0pt (SIAX Technology AB)
|
|
|
|
> Read this first. It is the contract for how this project is built. Then read `docs/` in order.
|
|
|
|
## What 0pt is
|
|
|
|
**0pt — your data.** A Nordic/European data-services platform (`0pt.app`) where customers:
|
|
1. **Compose** a data job (markets, entity type, fields, enrichment, frequency, output) — or browse ready-made datasets.
|
|
2. Pay a **50 % deposit** to start; the remaining 50 % is held in **escrow** and released only on approval.
|
|
3. **Watch production live** (sourcing → extraction → structuring → enrichment → QA).
|
|
4. **Validate** the result with full freedom (table / map / schema / quality views) and deliver to any format/destination.
|
|
|
|
Owned by SIAX Technology AB. Feeds internal products too (Protosell/FiveCRM, VehIQ).
|
|
|
|
## Non-negotiable constraints (do not violate)
|
|
|
|
1. **Legal sourcing only.** The data engine uses, in strict priority: official APIs/bulk → authorized scraping of clearly public pages (respect `robots.txt` + rate limits) → licensed partner data → otherwise the field is delivered as **"not available / licensed only"**. We never invent values.
|
|
- **Do NOT build, keep, or extend**: WAF/bot-detection bypass (incl. FlareSolverr, "stealth" browsers, undetected-* used to defeat detection), fingerprint/proxy rotation to evade bans, scraping behind login walls, or automated fake-account creation (Apollo etc.).
|
|
- If existing code (e.g. `src/workers/browser_worker.py`) contains a FlareSolverr/bypass path, **remove or disable it** and replace with compliant fetching. This is a product-positioning + compliance requirement, not a preference. "Honest accuracy" is the whole moat.
|
|
2. **Postgres is the backbone.** One Postgres is the source of truth. Catalog, portal, NocoDB views, Metabase dashboards, workers, exports all read/write the *same* DB. A tool may only enter the stack if it (a) reads that Postgres and (b) sits behind the same auth.
|
|
3. **Provenance per field.** Every fact carries source, freshness, confidence (`bridge_provenance`). This drives the UI signature too.
|
|
4. **One tool per job.** Do not add overlapping tools. The approved stack is in `docs/02-architecture.md` and `docs/07-dependencies.md`. Anything not listed needs a documented reason.
|
|
5. **GDPR-native.** DPA, lawful basis, audit log, deletion-after-job. Customer DB access is *pull by the customer* via a read-only, time-boxed role they create — never us asking for admin, never raw creds stored in plaintext.
|
|
6. **Design is fixed.** Mono/editorial system (white `#fff`, text `#0a0a0a`, hairline `#e5e5e5`, radius 0, Inter + JetBrains Mono, single red `#e40014`). Exact tokens + motion in `docs/05-design-system.md`. No photography — imagery is rendered data (`DataArtifact`).
|
|
|
|
## Build on existing code
|
|
|
|
A working repo already exists (~145 files): FastAPI orchestrator (`src/orchestrator/main.py`, 25+ endpoints), workers (`src/workers/`: adaptive_router, browser_worker, ocr_worker, db), and country data for 20 markets. **Reconcile with it, don't blindly replace.** Where this documentation defines a target contract, migrate the existing code toward it. Flag conflicts instead of guessing.
|
|
|
|
## How to work
|
|
|
|
- Work in **dependency order** per `docs/09-build-plan.md`. Finish a phase to its acceptance criteria before the next.
|
|
- Small, reviewable commits. Conventional Commits (`feat:`, `fix:`, `docs:`, `chore:`).
|
|
- Frontend = Next.js App Router + TS + Tailwind + shadcn/ui on the SIAX Deploy Platform. Backend = FastAPI + workers on a VPS (long jobs).
|
|
- Tech docs + code identifiers in **English**; all visible **UI copy in Swedish**.
|
|
- Never hardcode secrets — use env (`docs/08-infrastructure.md`).
|
|
- When unsure about a source's current access/licence terms, **verify before relying on it** and record it in the sourcing map; do not assume.
|
|
|
|
## Documentation index
|
|
|
|
| Doc | Purpose |
|
|
|---|---|
|
|
| `docs/01-product.md` | Vision, the three flows, order lifecycle, business model |
|
|
| `docs/02-architecture.md` | System architecture, stack decisions (in/out), hosting |
|
|
| `docs/03-data-model.md` | Postgres schema, provenance model, migrations |
|
|
| `docs/04-api.md` | FastAPI contract (canonical endpoints) |
|
|
| `docs/05-design-system.md` | Mono tokens, motion, visual system, signature |
|
|
| `docs/06-data-engine.md` | Workers, legal sourcing chain, pipeline, QA, fusion, sources.yaml |
|
|
| `docs/07-dependencies.md` | Repos/libraries to import (+ explicit exclusions) |
|
|
| `docs/08-infrastructure.md` | Hosting, docker-compose, env, security, CI/CD |
|
|
| `docs/09-build-plan.md` | Phased milestones with acceptance criteria |
|
|
| `docs/design/0pt-v0-build-brief.md` | Per-component v0 prompts (design reference) |
|
|
| `docs/design/0pt-design-coverage-and-visuals.md` | Full page/component inventory + visual system |
|
|
| `docs/10-lakehouse-decision.md` | Pointer: Apache Iceberg = canonical SIAX lakehouse format (ADR-0030, `n0d` repo); 0pt = write-path owner |
|
|
| `docs/11-lakehouse-write-plan.md` | 0pt's concrete lakehouse write to-do list, schema/event names for a future implementation session |
|
|
| `docs/ICEBERG_WRITE_BOUNDARY.md` | Implementation record: dataset<->Iceberg-table mapping + source-rate-policy contracts, ST0RE lakehouse client, live-verified write path (real namespace/table/snapshot IDs), what's real vs. contract-only |
|
|
|
|
## Definition of done (every screen, before real data)
|
|
|
|
Mono chrome · correct scroll-reveal motion · representative placeholder content · **loading + empty + error** states · mobile layout · provenance signature where facts appear · red used ≤ once · visible focus ring · `prefers-reduced-motion` respected.
|