0pt/docs/09-build-plan.md

# 09 · Build plan

Phased, in dependency order. **Finish a phase to its acceptance criteria before the next.** Each phase = small, reviewable commits. Reconcile with existing code rather than rewriting wholesale.

> **Status callouts added 2026-07-19.** Each phase below now carries a `Status:` line, grep/read-verified
> against `main` this pass. These are conservative on purpose: a phase is only called **Done** when this
> pass found real, working code satisfying its Accept criteria; anything else is **Partial** (with the
> specific evidence and gap named) or **Not started**. No phase is upgraded to Done without a code
> citation. Where a referenced doc (e.g. `docs/design/...`) doesn't exist, that's flagged rather than
> silently worked around. See `docs/08-infrastructure.md` for the full env/security grounding these
> statuses draw on.

## Phase 0 — Foundation & repo
- Monorepo structure (`docs/08`). Migrate existing `src/orchestrator` + `src/workers` into `services/api`.
- Postgres schema + migrations (`docs/03`); `pgvector` enabled. `.env.example`.
- `docker-compose` up (postgres, redis, api, worker, nocodb, metabase, caddy). `/health` green.
- CI skeleton (lint/typecheck/build).
- **Accept:** `docker compose up` runs the whole stack locally; migrations apply; `/health` returns ok for db + queue.
- **Status: Partial.**
  - Repo structure exists but diverged from the `apps/web`/`packages/shared` layout `docs/08` originally
    sketched — it settled on `app/` (Next.js, repo root), `lib/`, `src/` (platform clients/services), and
    `services/api/` (FastAPI). `docs/08-infrastructure.md` now documents the structure as-built.
  - Migration to `services/api/orchestrator` + `services/api/workers` is **done** (verified: those
    directories exist and contain `main.py`/`adaptive_router.py`/`pipeline.py`/etc.; no `src/orchestrator`
    or `src/workers` remain at the old paths).
  - Postgres schema is real: `sql/001_init.sql` (`CREATE EXTENSION vector`, `CREATE EXTENSION pgcrypto`,
    17 tables incl. `orgs`, `users`, `orders`, `jobs`, `audit_log`, `bridge_provenance`, `db_grants`,
    `payments`) + `sql/004_iceberg_write_boundary.sql`.
    **No `sql/002_*` or `sql/003_*` exists anywhere in this repo's history** — some other docs
    (`docs/ICEBERG_WRITE_BOUNDARY.md` etc.) cite a `sql/002_p0_dataset_quality.sql` that was never built;
    see `docs/08-infrastructure.md`'s "Schema state" section.
  - `.env.example` — **now exists and is merged to `main`** (BACKLOG P3-5: the
    stale link to the unmerged `github.com/sax3l/0pt/pull/3` that used to sit
    here has been removed — that PR has long since landed, `.env.example` is
    a real, current, gitignore-exempted file at the repo root today).
  - `infra/docker-compose.yml` and `infra/Caddyfile` are real, committed files (not sketches) covering
    exactly the services listed. **This pass found no evidence they've ever actually been run** — no
    CI, no deploy log, no reachable host checked. "File exists and looks correct" ≠ "verified to run";
    treat the Accept criterion ("docker compose up runs... /health returns ok") as unverified.
  - CI skeleton — **not built.** There is no `.github/workflows/` directory in this repo at all. `pnpm
    lint`/`typecheck`/`test` exist as local scripts only.

## Phase 1 — Design system in code
- Tailwind + shadcn; `globals.css` (grain + reveal keyframes) and `useReveal` verbatim (`docs/05`).
- Build **DataArtifact** primitive (table/map/schema/chart/grid) and **ProvenanceBadge** (`docs/design` prompts 5.1, 2.11).
- **Accept:** a sample page renders mono tokens, a DataArtifact animates on scroll, reduced-motion respected, focus rings visible.
- **Status: Partial — real components exist; scroll-motion/reduced-motion/focus-ring behavior not
  independently exercised this pass.**
  - Tailwind + shadcn/radix stack: confirmed real in `package.json` (`tailwindcss`, `@radix-ui/*`,
    `class-variance-authority`, `tailwind-merge`, `tailwindcss-animate`).
  - `hooks/use-reveal.ts`, `components/data-artifact.tsx`, `components/provenance-badge.tsx` all exist
    and are used across `app/katalog`, `app/kallor`, `app/bestall`, `app/order/[id]`, and most
    `components/sections/*` files (grep-confirmed).
  - `docs/design/0pt-v0-build-brief.md` and `docs/design/0pt-design-coverage-and-visuals.md` (the
    "prompts 5.1, 2.11" this bullet references) **do not exist anywhere in this repo** — confirmed via
    full-repo search; there is no `docs/design/` directory at all. `CLAUDE.md`'s own doc index has
    already been corrected to flag this as a gap rather than list them as real. Treat any reference to
    specific numbered "build-brief prompts" in this build plan as pointing at a doc that was never
    written — not this file's scope to author that content.
  - Reduced-motion handling, focus-ring visibility, and actual scroll-triggered animation behavior were
    not executed/verified in this pass (no browser run performed) — code presence is confirmed, runtime
    behavior is not.

## Phase 2 — Marketing + catalog (placeholder data)
- Landing components top-to-bottom (build-brief 2.1–2.10): Header, Hero, Philosophy, Catalog teaser, Engine, Visualisation gallery, Coverage, Editorial, Proof, Footer.
- Full **Catalog** page + **Dataset detail** + **Sourcing-map** page (coverage doc 5.4). Postgres full-text search.
- All with loading/empty/error states + mobile.
- **Accept:** marketing site + catalog browse work end-to-end on placeholder catalog rows; one continuous scroll reads well; DoD met per screen.
- **Status: Partial.**
  - Landing section components are real: `components/sections/{hero,philosophy,gallery,editorial,
    pricing,technology,testimonials,footer,featured-products,collection}-section.tsx` all exist.
  - `app/katalog/page.tsx`, `app/katalog/[id]/page.tsx`, `app/kallor/page.tsx` all exist.
  - **Postgres full-text search was not found** — grepped for `to_tsvector`/`websearch_to_tsquery`
    across the whole repo, zero matches. Catalog browsing today reads from `lib/data.ts` placeholder
    data, not a full-text query against Postgres. This Accept criterion's "catalog browse... on
    placeholder catalog rows" half is plausible; the Postgres full-text search half is not built.
  - Loading/empty/error states + mobile + DoD-per-screen were not independently audited screen-by-screen
    this pass — `components/states.tsx` (`LoadingSkeleton`, `ErrorState`) exists and is imported widely,
    which is evidence toward this but not a full verification.

## Phase 3 — Auth & portal shell
- Auth.js + Postgres adapter; sign in/up/verify/forgot. `orgs`/`users`.
- Portal shell + Dashboard ("Mina beställningar") with empty state.
- **Accept:** a user can sign up, land in an empty dashboard, sign out; routes protected.
- **Status: Not done — this is the most consequential gap in the whole repo, not a minor loose end.**
  - **There is no Auth.js integration.** `lib/auth.ts` is a from-scratch, ~40-line HMAC-SHA256 session
    scaffold (Web Crypto, no dependency) whose own comment states: "Production auth = Auth.js + Postgres
    adapter + OAuth (docs/02) — wired once the database is provisioned." `app/api/auth/login/route.ts`
    is a **dev-only** login gated by a shared `DEV_LOGIN_KEY`, not sign-up/verify/forgot.
  - `orgs` and `users` tables exist in `sql/001_init.sql` (schema is ready), but nothing in the app
    reads/writes them for real session-bound identity today.
  - `app/portal/page.tsx` and sibling portal pages exist (shell is real), but routes are **not**
    protected by any real session check tied to that schema.
  - **Concretely, this means the tenant/org isolation several `0pt.*` API routes assume does not exist
    yet.** `app/api/0pt/lakehouse/mappings/route.ts`, `.../writes/route.ts`,
    `app/api/0pt/sources/rate-policies/route.ts`, `app/api/0pt/sources/[sourceId]/query/route.ts`, and
    `src/services/document-capture-service.ts` all read `tenant_id`/`project_id` straight from the
    request body — anyone can claim any tenant. `app/api/orders/route.ts`'s own `GET` handler carries
    the literal comment `// org scoping added with auth`. This is an open security gap, not a
    two-tier design choice — see `docs/08-infrastructure.md`'s "Security & compliance" section for the
    full list. **Do not build any doc or feature that assumes tenant-scoped auth already exists.**

## Phase 4 — Order lifecycle end-to-end (test mode)
- **Configurator** (2.12) wired to `POST /estimates` (real estimate math in FastAPI).
- **Deposit checkout** (2.13) → Stripe **test** Checkout → webhook → order `IN_PRODUCTION`.
- **Production tracker** (2.14) reading `GET /orders/{id}/events` (SSE) — pipeline may be **simulated** at this phase.
- **Validation & delivery** (2.15): tabs render from a sample delivery; **accept** → `POST /payments/release` (test) → `DELIVERED`; **revise** leaves escrow untouched.
- AI-assist (`POST /ai/parse`) prefilling the configurator.
- **Accept:** a full custom order can be configured, deposit paid (test), watched, validated, accepted, and "delivered" — state machine in `docs/01` traversed; escrow split correct.
- **Status: Partial — the payment/escrow spine is genuinely real; the SSE tracker is not.**
  - `app/api/estimates/route.ts` exists.
  - Deposit checkout is **real, working code**: `app/api/payments/[order_id]/deposit/route.ts` calls
    Stripe's Checkout Sessions API directly (not a stub) and `app/api/webhooks/stripe/route.ts` verifies
    the webhook signature and advances order status via `setStatus`.
  - `app/api/payments/[order_id]/release/route.ts` is also real: it enforces the escrow rule in code —
    balance release is rejected with `409 not_accepted` unless `order.status === "accepted"` — matching
    the "escrow split correct" accept criterion's intent.
  - **`GET /api/orders/{id}/events` (SSE) does not exist.** Only `GET`/`PATCH /api/orders/{id}` were
    found (`app/api/orders/[id]/route.ts`); no `/events` sub-route, and no `EventSource`/
    `text/event-stream`/`ReadableStream` usage anywhere in `app/` (grep-verified). If a production
    tracker is live today, it's polling `GET /api/orders/{id}`, not the SSE stream this bullet describes
    — or it hasn't been built. Either way, the SSE contract as written is not implemented.
  - `app/api/ai/parse/route.ts` is real: a genuine (non-fabricated) keyword/regex heuristic that
    prefills a config object, with a documented fallback to `engineFetch("/ai/parse", ...)` when
    `ENGINE_URL` is set. This one holds up as described.

## Phase 5 — Real data engine (Nordic slice)
- Compliant connectors first: **GLEIF + Brreg + Companies House** (verify access terms; record in `sources.yaml`). Remove any FlareSolverr/bypass path (`docs/06`).
- Pipeline stages produce real `dim_company` + `fact_financials` with `bridge_provenance`; QA threshold drops low-confidence rows.
- Production tracker now reflects **real** job metrics.
- **Accept:** a Nordic company dataset is produced for a real order, every field has provenance, quality report generated, level-0 fields shown honestly (not fabricated).
- **Status: Partial.**
  - `services/api/sources/sources.yaml` and `connectors.py` are real: GLEIF, Brreg, Companies House,
    PRH, and CVR are all registered, each with an explicit `licence: "... — VERIFY"` tag rather than an
    assumed-clean claim — matches the "verify access terms" instruction.
  - `services/api/workers/adaptive_router.py`'s own comment confirms: "There are NO bypass/stealth/
    proxy-rotation tiers here, by design (CLAUDE.md §1)" — the FlareSolverr/bypass removal is done, at
    least on this repo's `main` (see `docs/08-infrastructure.md`'s "Known open question" about the
    diverged Gitea remote, which independently added such a stack on its own `main` — unresolved which
    remote is canonical).
  - `services/api/workers/pipeline.py` has a real, working confidence-threshold QA drop: `delivered =
    [r for r in deduped if r["confidence"] >= threshold]`, plus an `avg_confidence` figure in its output.
    This is a genuine, narrow QA mechanism inside the Python engine.
  - **This is not the same thing as the "quality report" / quality-check layer once described in
    `docs/ICEBERG_WRITE_BOUNDARY.md`, `docs/11-lakehouse-write-plan.md`, and
    `docs/P0_QUERY_SOURCE_FLOW.md`.** Those three docs used to cite `src/services/quality-check-service.ts`,
    `src/services/quality-report-service.ts`, `src/types/quality-{check,report}.ts`, and
    `src/data-platform/services/catalog-service.ts` as already-built and tested. **None of those files
    has ever existed in this repo's git history, on any branch** — and all three docs have since been
    corrected to say so explicitly (verified this pass), rather than repeating the original claim. So:
    the narrow Python-side confidence filter is real; the broader TS/lakehouse "quality report" this
    Accept criterion likely also intends is not built (see `docs/P0_DATASET_QUALITY_FLOW.md` for the
    honest target spec). Whether `pipeline.py`'s `avg_confidence` output satisfies "quality report
    generated" is a judgment call for whoever runs this phase — flagging rather than deciding it here.
  - Whether an actual Nordic dataset has been produced end-to-end for a real order was not verified
    this pass (would require running the stack).

## Phase 6 — Delivery surfaces & integrations
- Exports: CSV / Parquet / JSON via R2 presigned URLs.
- Embedded **NocoDB** ("Min data") and **Metabase** dashboards via signed URLs in mono chrome.
- Integrations: **FiveCRM push**, webhooks, Google Sheets.
- File **uploads** (Uppy → R2) for matching/enrichment.
- **Accept:** a delivered dataset is downloadable, viewable in embedded NocoDB + Metabase (no separate login), and pushable to FiveCRM; uploads work.
- **Status: Not done — this phase is UI-shell-only today.**
  - CSV/Parquet/JSON exports via R2 presigned URLs: not found. `app/api/uploads/route.ts` (the closest
    R2-touching code) is a `501` stub with a hardcoded `put_url: null` — no presign logic exists yet,
    for uploads or exports.
  - `components/embed.tsx` (`EmbeddedView`) is real — a sandboxed iframe shell — but the server-side
    signed-URL generator for Metabase/NocoDB (`METABASE_SECRET_KEY`/`NOCODB_API_TOKEN`) that would feed
    it a real `src` was not found anywhere in this pass.
  - "FiveCRM" appears only as a **destination label string** in `lib/data.ts`, `lib/services.ts`,
    `app/portal/integrationer/page.tsx` (explicitly described in that file's own comment as
    "presentationella toggles" — i.e. presentational, not wired), and as one of several regex-matched
    destination names in `app/api/ai/parse/route.ts`'s heuristic. **No actual FiveCRM API push exists.**
  - `uppy` is not a dependency in `package.json` (checked directly) — the Uppy → R2 upload flow this
    bullet describes has no corresponding package installed, let alone implemented.

## Phase 7 — Subscriptions / feeds
- `subscriptions`; recurring jobs (weekly/daily) delivering **deltas**; webhooks on change.
- Freshness monitoring; Resend emails (confirmation, "din data är klar", feedback).
- **Accept:** an order converts to a feed; a scheduled run produces a delta + fires a webhook + email.
- **Status: Partial — data layer real, scheduling and email are not.**
  - `lib/subscriptions-store.ts` is real CRUD against the `subscriptions` table (with an in-memory
    fallback when no DB is configured) — `createSubscription`/`listSubscriptions` genuinely insert/read.
  - No recurring-job scheduler (cron, queue consumer producing deltas, etc.) was found on the TS side of
    this repo. `services/api/workers/queue.py` exists on the Python side but this pass did not trace
    whether it implements actual delta-producing scheduled runs — flagged as unverified rather than
    assumed either way.
  - **Resend is not wired at all** — grepped for `resend`/`Resend` across the whole repo, zero matches
    outside `RESEND_API_KEY` being listed as an env var. No confirmation/"din data är klar"/feedback
    email has been implemented.

## Phase 8 — Admin, hardening, compliance
- Admin: order queue + scope/quote + escrow release; **source registry & freshness** monitoring; worker/job monitoring; customer + catalog management.
- Customer **DB-access grant** flow (encrypted, read-only, time-boxed) + revoke.
- Security pass: RLS, presigned-only files, signed embeds, secret rotation, `audit_log` everywhere; **DPA** published; legal pages.
- Finalize all loading/empty/error states + mobile across the app.
- **Accept:** admin can run the business; DB-grant flow works and self-revokes; security checklist passes; DoD met app-wide.
- **Status: Partial — UI shell real, the security substance is not.**
  - Admin pages exist: `app/admin/{page,jobs/page,sources/page,customers/page,verktyg/page}.tsx`. This
    pass did not verify whether they read real backend data or placeholders.
  - `sql/001_init.sql` defines `audit_log`, `db_grants`, and `payments` tables — schema is ready.
  - **The DB-access grant flow is a stub, not a working feature.**
    `app/api/connections/db-grant/route.ts` returns `501` without `GRANT_ENC_KEY`, and with it set
    returns a hardcoded `201 { grant_id, status: "pending" }` — **no AES-GCM encryption, no read-only
    validation, and no self-revoke logic exist in code today**, despite the route's own comment
    describing that target behavior. See `docs/08-infrastructure.md`'s "Security & compliance" section.
  - RLS enforcement, presigned-only files (blocked on the Phase 6 R2 gap above), signed embeds (blocked
    on the Phase 6 NocoDB/Metabase gap above), and a published DPA were not found as implemented in this
    pass. Treat the "security checklist passes" accept criterion as not met.

## Phase 9 — Launch
- Domain `0pt.app` (Vercel) + subdomains (api/data/dash via Caddy). Sentry + uptime live.
- Soft launch: Protosell internally + a handful of existing dealer relationships.
- **Accept:** a real external customer completes an order start→delivery in production; feedback loop running.
- **Status: Domain is live; soft-launch/customer criterion not verified.**
  - **Verified this pass** (network check performed): `https://0pt.app` returns HTTP 200 and serves this
    repo's own Next.js app (title "0pt — your data", SIAX branding); `GET /api/health` returns
    `{"ok":true,"service":"0pt-web","db":"not_configured","queue":"not_configured",...}`, an exact shape
    match for `app/api/health/route.ts`. README's "Live: https://0pt.app" is confirmed, and the response
    confirms the live deployment has no `DATABASE_URL`/`REDIS_URL` set — see `docs/08-infrastructure.md`'s
    "Hosting topology" section. This does not confirm which git remote/commit is deployed (see the
    diverged-remotes note elsewhere in this doc), or that api/data/dash subdomains or Caddy are live —
    only the root Vercel deployment was checked.
  - **Sentry is not wired** — no `sentry` package in `package.json`, no Sentry config file anywhere in
    the repo (grep-confirmed). `@vercel/analytics` **is** genuinely wired (`app/layout.tsx` renders
    `<Analytics />`), which is a real but much smaller piece of the "Sentry + uptime live" bullet.
  - No evidence either way on the soft-launch/external-customer criterion — out of this doc's ability to
    verify from code alone.

---

## Working conventions (for Claude Code)
- Tech docs + identifiers **English**; UI copy **Swedish**.
- Conventional Commits; one concern per PR; keep diffs reviewable.
- No secrets in code; update `.env.example` when adding env. (`.env.example` itself currently lives on
  `feature/0pt-env-example` / PR #3 — merge that first, or this instruction has nothing to update yet.)
- Add/extend tests for engine logic (estimate math, fusion/confidence, dedup) and API contract.
- When a source's access/licence is unclear, **verify and record in `sources.yaml`** before relying; if disallowed for bulk, route to licensed/partner or mark level 0 — never bypass, never fabricate.
- Touch the design tokens/motion only via `docs/05`; keep red ≤ once per screen; no photography.
- If anything here conflicts with existing code, **flag it** in the PR description and propose the reconciliation rather than guessing.
- **Two git remotes claim to be this repo** (`github.com/sax3l/0pt` and `git.siax.io/sax3l/0pt`), and
  they have independently diverged (see `CLAUDE.md`, `README.md`). Don't assume the checkout you're in
  is the only copy, and don't silently pick one as canonical — that's Simon's decision, flag it if it
  becomes relevant to the phase you're working on.