Files
0pt/CLAUDE.md
T
adminandClaude Sonnet 5 bda21e87eb Live-verify Iceberg write path against ST0RE + close field/contract gaps
Re-runs this session's task spec against 0pt's actual repo state, which
turned out to already have a substantial, tested implementation from a
prior pass (35b6ec6): dataset<->Iceberg-table mapping, source rate policy,
and an ST0RE lakehouse client that was honestly marked "not live-verified".

This pass found api.st0re.siax.io IS reachable from this environment and
used it -- first via curl to safely discover the real contract, then
through 0pt's own updated code -- to drive a real namespace -> table ->
snapshot write against production ST0RE (task item 3's "write ONE first
test dataset" proof). That surfaced and fixed three real bugs the
mock-only pass could not have caught:

- ST0RE requires X-Tenant-Id/X-Actor-Id headers on every lakehouse route;
  the previous HTTP client sent none. Fixed via a threaded
  St0reRequestContext.
- IcebergTableInput.datasetId is a foreign key into ST0RE's OWN dataset
  table, not a free-form string -- sending 0pt's dataset_id 400s live with
  a Postgres FK violation. Fixed by no longer sending it; 0pt's
  dataset<->table linkage lives entirely in DatasetIcebergMapping.
- ST0RE enforces a real unique constraint on lakehouse namespaces
  (tenant_id, project_id, name) with no upsert route. Fixed with a
  client-side lookup-or-create (listNamespaces + findOrRegisterNamespace).

Also closed a field-level gap against the task's literal schema spec:
0pt.dataset_iceberg_mapping.schema.v1 now carries namespace_id (a real
reference to ST0RE's LakehouseNamespace.namespaceId, not guessed) and
mapped_at, both additive alongside the existing namespace/created_at
fields (sql/005, JSON schema, types, service).

tests/p0/iceberg-write-service.realinfra.test.ts replaces the prior
it.todo with a real SEAM-pattern live test: attempts the real HTTP call,
skips honestly if ST0RE is unreachable, never fabricates success. It ran
successfully against production ST0RE this session (real namespace/table/
snapshot UUIDs logged in docs/ICEBERG_WRITE_BOUNDARY.md).

Two ST0RE-side cross-tenant read bugs were found and flagged (not fixed
here -- out of scope for this repo): GET .../namespaces ignores its own
tenant header and query filters; GET .../tables/:id doesn't check the
caller's tenant against the table's owner.

npm test: 155 passed, 0 failed, 0 todo (up from 154 passed + 1 todo).
npx tsc --noEmit: 1 pre-existing, unrelated error in
quality-report-service.ts (present before this pass, part of the wider
uncommitted P0 data-platform build already in this repo) -- not touched.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-13 00:18:10 +02:00

5.5 KiB

CLAUDE.md — 0pt (SIAX Technology AB)

Read this first. It is the contract for how this project is built. Then read docs/ in order.

What 0pt is

0pt — your data. A Nordic/European data-services platform (0pt.app) where customers:

  1. Compose a data job (markets, entity type, fields, enrichment, frequency, output) — or browse ready-made datasets.
  2. Pay a 50 % deposit to start; the remaining 50 % is held in escrow and released only on approval.
  3. Watch production live (sourcing → extraction → structuring → enrichment → QA).
  4. Validate the result with full freedom (table / map / schema / quality views) and deliver to any format/destination.

Owned by SIAX Technology AB. Feeds internal products too (Protosell/FiveCRM, VehIQ).

Non-negotiable constraints (do not violate)

  1. Legal sourcing only. The data engine uses, in strict priority: official APIs/bulk → authorized scraping of clearly public pages (respect robots.txt + rate limits) → licensed partner data → otherwise the field is delivered as "not available / licensed only". We never invent values.
    • Do NOT build, keep, or extend: WAF/bot-detection bypass (incl. FlareSolverr, "stealth" browsers, undetected-* used to defeat detection), fingerprint/proxy rotation to evade bans, scraping behind login walls, or automated fake-account creation (Apollo etc.).
    • If existing code (e.g. src/workers/browser_worker.py) contains a FlareSolverr/bypass path, remove or disable it and replace with compliant fetching. This is a product-positioning + compliance requirement, not a preference. "Honest accuracy" is the whole moat.
  2. Postgres is the backbone. One Postgres is the source of truth. Catalog, portal, NocoDB views, Metabase dashboards, workers, exports all read/write the same DB. A tool may only enter the stack if it (a) reads that Postgres and (b) sits behind the same auth.
  3. Provenance per field. Every fact carries source, freshness, confidence (bridge_provenance). This drives the UI signature too.
  4. One tool per job. Do not add overlapping tools. The approved stack is in docs/02-architecture.md and docs/07-dependencies.md. Anything not listed needs a documented reason.
  5. GDPR-native. DPA, lawful basis, audit log, deletion-after-job. Customer DB access is pull by the customer via a read-only, time-boxed role they create — never us asking for admin, never raw creds stored in plaintext.
  6. Design is fixed. Mono/editorial system (white #fff, text #0a0a0a, hairline #e5e5e5, radius 0, Inter + JetBrains Mono, single red #e40014). Exact tokens + motion in docs/05-design-system.md. No photography — imagery is rendered data (DataArtifact).

Build on existing code

A working repo already exists (~145 files): FastAPI orchestrator (src/orchestrator/main.py, 25+ endpoints), workers (src/workers/: adaptive_router, browser_worker, ocr_worker, db), and country data for 20 markets. Reconcile with it, don't blindly replace. Where this documentation defines a target contract, migrate the existing code toward it. Flag conflicts instead of guessing.

How to work

  • Work in dependency order per docs/09-build-plan.md. Finish a phase to its acceptance criteria before the next.
  • Small, reviewable commits. Conventional Commits (feat:, fix:, docs:, chore:).
  • Frontend = Next.js App Router + TS + Tailwind + shadcn/ui on the SIAX Deploy Platform. Backend = FastAPI + workers on a VPS (long jobs).
  • Tech docs + code identifiers in English; all visible UI copy in Swedish.
  • Never hardcode secrets — use env (docs/08-infrastructure.md).
  • When unsure about a source's current access/licence terms, verify before relying on it and record it in the sourcing map; do not assume.

Documentation index

Doc Purpose
docs/01-product.md Vision, the three flows, order lifecycle, business model
docs/02-architecture.md System architecture, stack decisions (in/out), hosting
docs/03-data-model.md Postgres schema, provenance model, migrations
docs/04-api.md FastAPI contract (canonical endpoints)
docs/05-design-system.md Mono tokens, motion, visual system, signature
docs/06-data-engine.md Workers, legal sourcing chain, pipeline, QA, fusion, sources.yaml
docs/07-dependencies.md Repos/libraries to import (+ explicit exclusions)
docs/08-infrastructure.md Hosting, docker-compose, env, security, CI/CD
docs/09-build-plan.md Phased milestones with acceptance criteria
docs/design/0pt-v0-build-brief.md Per-component v0 prompts (design reference)
docs/design/0pt-design-coverage-and-visuals.md Full page/component inventory + visual system
docs/10-lakehouse-decision.md Pointer: Apache Iceberg = canonical SIAX lakehouse format (ADR-0030, n0d repo); 0pt = write-path owner
docs/11-lakehouse-write-plan.md 0pt's concrete lakehouse write to-do list, schema/event names for a future implementation session
docs/ICEBERG_WRITE_BOUNDARY.md Implementation record: dataset<->Iceberg-table mapping + source-rate-policy contracts, ST0RE lakehouse client, live-verified write path (real namespace/table/snapshot IDs), what's real vs. contract-only

Definition of done (every screen, before real data)

Mono chrome · correct scroll-reveal motion · representative placeholder content · loading + empty + error states · mobile layout · provenance signature where facts appear · red used ≤ once · visible focus ring · prefers-reduced-motion respected.