Re-runs this session's task spec against 0pt's actual repo state, which
turned out to already have a substantial, tested implementation from a
prior pass (35b6ec6): dataset<->Iceberg-table mapping, source rate policy,
and an ST0RE lakehouse client that was honestly marked "not live-verified".
This pass found api.st0re.siax.io IS reachable from this environment and
used it -- first via curl to safely discover the real contract, then
through 0pt's own updated code -- to drive a real namespace -> table ->
snapshot write against production ST0RE (task item 3's "write ONE first
test dataset" proof). That surfaced and fixed three real bugs the
mock-only pass could not have caught:
- ST0RE requires X-Tenant-Id/X-Actor-Id headers on every lakehouse route;
the previous HTTP client sent none. Fixed via a threaded
St0reRequestContext.
- IcebergTableInput.datasetId is a foreign key into ST0RE's OWN dataset
table, not a free-form string -- sending 0pt's dataset_id 400s live with
a Postgres FK violation. Fixed by no longer sending it; 0pt's
dataset<->table linkage lives entirely in DatasetIcebergMapping.
- ST0RE enforces a real unique constraint on lakehouse namespaces
(tenant_id, project_id, name) with no upsert route. Fixed with a
client-side lookup-or-create (listNamespaces + findOrRegisterNamespace).
Also closed a field-level gap against the task's literal schema spec:
0pt.dataset_iceberg_mapping.schema.v1 now carries namespace_id (a real
reference to ST0RE's LakehouseNamespace.namespaceId, not guessed) and
mapped_at, both additive alongside the existing namespace/created_at
fields (sql/005, JSON schema, types, service).
tests/p0/iceberg-write-service.realinfra.test.ts replaces the prior
it.todo with a real SEAM-pattern live test: attempts the real HTTP call,
skips honestly if ST0RE is unreachable, never fabricates success. It ran
successfully against production ST0RE this session (real namespace/table/
snapshot UUIDs logged in docs/ICEBERG_WRITE_BOUNDARY.md).
Two ST0RE-side cross-tenant read bugs were found and flagged (not fixed
here -- out of scope for this repo): GET .../namespaces ignores its own
tenant header and query filters; GET .../tables/:id doesn't check the
caller's tenant against the table's owner.
npm test: 155 passed, 0 failed, 0 todo (up from 154 passed + 1 todo).
npx tsc --noEmit: 1 pre-existing, unrelated error in
quality-report-service.ts (present before this pass, part of the wider
uncommitted P0 data-platform build already in this repo) -- not touched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
5.5 KiB
CLAUDE.md — 0pt (SIAX Technology AB)
Read this first. It is the contract for how this project is built. Then read
docs/in order.
What 0pt is
0pt — your data. A Nordic/European data-services platform (0pt.app) where customers:
- Compose a data job (markets, entity type, fields, enrichment, frequency, output) — or browse ready-made datasets.
- Pay a 50 % deposit to start; the remaining 50 % is held in escrow and released only on approval.
- Watch production live (sourcing → extraction → structuring → enrichment → QA).
- Validate the result with full freedom (table / map / schema / quality views) and deliver to any format/destination.
Owned by SIAX Technology AB. Feeds internal products too (Protosell/FiveCRM, VehIQ).
Non-negotiable constraints (do not violate)
- Legal sourcing only. The data engine uses, in strict priority: official APIs/bulk → authorized scraping of clearly public pages (respect
robots.txt+ rate limits) → licensed partner data → otherwise the field is delivered as "not available / licensed only". We never invent values.- Do NOT build, keep, or extend: WAF/bot-detection bypass (incl. FlareSolverr, "stealth" browsers, undetected-* used to defeat detection), fingerprint/proxy rotation to evade bans, scraping behind login walls, or automated fake-account creation (Apollo etc.).
- If existing code (e.g.
src/workers/browser_worker.py) contains a FlareSolverr/bypass path, remove or disable it and replace with compliant fetching. This is a product-positioning + compliance requirement, not a preference. "Honest accuracy" is the whole moat.
- Postgres is the backbone. One Postgres is the source of truth. Catalog, portal, NocoDB views, Metabase dashboards, workers, exports all read/write the same DB. A tool may only enter the stack if it (a) reads that Postgres and (b) sits behind the same auth.
- Provenance per field. Every fact carries source, freshness, confidence (
bridge_provenance). This drives the UI signature too. - One tool per job. Do not add overlapping tools. The approved stack is in
docs/02-architecture.mdanddocs/07-dependencies.md. Anything not listed needs a documented reason. - GDPR-native. DPA, lawful basis, audit log, deletion-after-job. Customer DB access is pull by the customer via a read-only, time-boxed role they create — never us asking for admin, never raw creds stored in plaintext.
- Design is fixed. Mono/editorial system (white
#fff, text#0a0a0a, hairline#e5e5e5, radius 0, Inter + JetBrains Mono, single red#e40014). Exact tokens + motion indocs/05-design-system.md. No photography — imagery is rendered data (DataArtifact).
Build on existing code
A working repo already exists (~145 files): FastAPI orchestrator (src/orchestrator/main.py, 25+ endpoints), workers (src/workers/: adaptive_router, browser_worker, ocr_worker, db), and country data for 20 markets. Reconcile with it, don't blindly replace. Where this documentation defines a target contract, migrate the existing code toward it. Flag conflicts instead of guessing.
How to work
- Work in dependency order per
docs/09-build-plan.md. Finish a phase to its acceptance criteria before the next. - Small, reviewable commits. Conventional Commits (
feat:,fix:,docs:,chore:). - Frontend = Next.js App Router + TS + Tailwind + shadcn/ui on the SIAX Deploy Platform. Backend = FastAPI + workers on a VPS (long jobs).
- Tech docs + code identifiers in English; all visible UI copy in Swedish.
- Never hardcode secrets — use env (
docs/08-infrastructure.md). - When unsure about a source's current access/licence terms, verify before relying on it and record it in the sourcing map; do not assume.
Documentation index
| Doc | Purpose |
|---|---|
docs/01-product.md |
Vision, the three flows, order lifecycle, business model |
docs/02-architecture.md |
System architecture, stack decisions (in/out), hosting |
docs/03-data-model.md |
Postgres schema, provenance model, migrations |
docs/04-api.md |
FastAPI contract (canonical endpoints) |
docs/05-design-system.md |
Mono tokens, motion, visual system, signature |
docs/06-data-engine.md |
Workers, legal sourcing chain, pipeline, QA, fusion, sources.yaml |
docs/07-dependencies.md |
Repos/libraries to import (+ explicit exclusions) |
docs/08-infrastructure.md |
Hosting, docker-compose, env, security, CI/CD |
docs/09-build-plan.md |
Phased milestones with acceptance criteria |
docs/design/0pt-v0-build-brief.md |
Per-component v0 prompts (design reference) |
docs/design/0pt-design-coverage-and-visuals.md |
Full page/component inventory + visual system |
docs/10-lakehouse-decision.md |
Pointer: Apache Iceberg = canonical SIAX lakehouse format (ADR-0030, n0d repo); 0pt = write-path owner |
docs/11-lakehouse-write-plan.md |
0pt's concrete lakehouse write to-do list, schema/event names for a future implementation session |
docs/ICEBERG_WRITE_BOUNDARY.md |
Implementation record: dataset<->Iceberg-table mapping + source-rate-policy contracts, ST0RE lakehouse client, live-verified write path (real namespace/table/snapshot IDs), what's real vs. contract-only |
Definition of done (every screen, before real data)
Mono chrome · correct scroll-reveal motion · representative placeholder content · loading + empty + error states · mobile layout · provenance signature where facts appear · red used ≤ once · visible focus ring · prefers-reduced-motion respected.