Files
0pt/docs/09-build-plan.md
T

246 lines
20 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 09 · Build plan
Phased, in dependency order. **Finish a phase to its acceptance criteria before the next.** Each phase = small, reviewable commits. Reconcile with existing code rather than rewriting wholesale.
> **Status callouts added 2026-07-19.** Each phase below now carries a `Status:` line, grep/read-verified
> against `main` this pass. These are conservative on purpose: a phase is only called **Done** when this
> pass found real, working code satisfying its Accept criteria; anything else is **Partial** (with the
> specific evidence and gap named) or **Not started**. No phase is upgraded to Done without a code
> citation. Where a referenced doc (e.g. `docs/design/...`) doesn't exist, that's flagged rather than
> silently worked around. See `docs/08-infrastructure.md` for the full env/security grounding these
> statuses draw on.
## Phase 0 — Foundation & repo
- Monorepo structure (`docs/08`). Migrate existing `src/orchestrator` + `src/workers` into `services/api`.
- Postgres schema + migrations (`docs/03`); `pgvector` enabled. `.env.example`.
- `docker-compose` up (postgres, redis, api, worker, nocodb, metabase, caddy). `/health` green.
- CI skeleton (lint/typecheck/build).
- **Accept:** `docker compose up` runs the whole stack locally; migrations apply; `/health` returns ok for db + queue.
- **Status: Partial.**
- Repo structure exists but diverged from the `apps/web`/`packages/shared` layout `docs/08` originally
sketched — it settled on `app/` (Next.js, repo root), `lib/`, `src/` (platform clients/services), and
`services/api/` (FastAPI). `docs/08-infrastructure.md` now documents the structure as-built.
- Migration to `services/api/orchestrator` + `services/api/workers` is **done** (verified: those
directories exist and contain `main.py`/`adaptive_router.py`/`pipeline.py`/etc.; no `src/orchestrator`
or `src/workers` remain at the old paths).
- Postgres schema is real: `sql/001_init.sql` (`CREATE EXTENSION vector`, `CREATE EXTENSION pgcrypto`,
17 tables incl. `orgs`, `users`, `orders`, `jobs`, `audit_log`, `bridge_provenance`, `db_grants`,
`payments`) + `sql/004_iceberg_write_boundary.sql`.
**No `sql/002_*` or `sql/003_*` exists anywhere in this repo's history** — some other docs
(`docs/ICEBERG_WRITE_BOUNDARY.md` etc.) cite a `sql/002_p0_dataset_quality.sql` that was never built;
see `docs/08-infrastructure.md`'s "Schema state" section.
- `.env.example` — **now exists and is merged to `main`** (BACKLOG P3-5: the
stale link to the unmerged `github.com/sax3l/0pt/pull/3` that used to sit
here has been removed — that PR has long since landed, `.env.example` is
a real, current, gitignore-exempted file at the repo root today).
- `infra/docker-compose.yml` and `infra/Caddyfile` are real, committed files (not sketches) covering
exactly the services listed. **This pass found no evidence they've ever actually been run** — no
CI, no deploy log, no reachable host checked. "File exists and looks correct" ≠ "verified to run";
treat the Accept criterion ("docker compose up runs... /health returns ok") as unverified.
- CI skeleton — **not built.** There is no `.github/workflows/` directory in this repo at all. `pnpm
lint`/`typecheck`/`test` exist as local scripts only.
## Phase 1 — Design system in code
- Tailwind + shadcn; `globals.css` (grain + reveal keyframes) and `useReveal` verbatim (`docs/05`).
- Build **DataArtifact** primitive (table/map/schema/chart/grid) and **ProvenanceBadge** (`docs/design` prompts 5.1, 2.11).
- **Accept:** a sample page renders mono tokens, a DataArtifact animates on scroll, reduced-motion respected, focus rings visible.
- **Status: Partial — real components exist; scroll-motion/reduced-motion/focus-ring behavior not
independently exercised this pass.**
- Tailwind + shadcn/radix stack: confirmed real in `package.json` (`tailwindcss`, `@radix-ui/*`,
`class-variance-authority`, `tailwind-merge`, `tailwindcss-animate`).
- `hooks/use-reveal.ts`, `components/data-artifact.tsx`, `components/provenance-badge.tsx` all exist
and are used across `app/katalog`, `app/kallor`, `app/bestall`, `app/order/[id]`, and most
`components/sections/*` files (grep-confirmed).
- `docs/design/0pt-v0-build-brief.md` and `docs/design/0pt-design-coverage-and-visuals.md` (the
"prompts 5.1, 2.11" this bullet references) **do not exist anywhere in this repo** — confirmed via
full-repo search; there is no `docs/design/` directory at all. `CLAUDE.md`'s own doc index has
already been corrected to flag this as a gap rather than list them as real. Treat any reference to
specific numbered "build-brief prompts" in this build plan as pointing at a doc that was never
written — not this file's scope to author that content.
- Reduced-motion handling, focus-ring visibility, and actual scroll-triggered animation behavior were
not executed/verified in this pass (no browser run performed) — code presence is confirmed, runtime
behavior is not.
## Phase 2 — Marketing + catalog (placeholder data)
- Landing components top-to-bottom (build-brief 2.1–2.10): Header, Hero, Philosophy, Catalog teaser, Engine, Visualisation gallery, Coverage, Editorial, Proof, Footer.
- Full **Catalog** page + **Dataset detail** + **Sourcing-map** page (coverage doc 5.4). Postgres full-text search.
- All with loading/empty/error states + mobile.
- **Accept:** marketing site + catalog browse work end-to-end on placeholder catalog rows; one continuous scroll reads well; DoD met per screen.
- **Status: Partial.**
- Landing section components are real: `components/sections/{hero,philosophy,gallery,editorial,
pricing,technology,testimonials,footer,featured-products,collection}-section.tsx` all exist.
- `app/katalog/page.tsx`, `app/katalog/[id]/page.tsx`, `app/kallor/page.tsx` all exist.
- **Postgres full-text search was not found** — grepped for `to_tsvector`/`websearch_to_tsquery`
across the whole repo, zero matches. Catalog browsing today reads from `lib/data.ts` placeholder
data, not a full-text query against Postgres. This Accept criterion's "catalog browse... on
placeholder catalog rows" half is plausible; the Postgres full-text search half is not built.
- Loading/empty/error states + mobile + DoD-per-screen were not independently audited screen-by-screen
this pass — `components/states.tsx` (`LoadingSkeleton`, `ErrorState`) exists and is imported widely,
which is evidence toward this but not a full verification.
## Phase 3 — Auth & portal shell
- Auth.js + Postgres adapter; sign in/up/verify/forgot. `orgs`/`users`.
- Portal shell + Dashboard ("Mina beställningar") with empty state.
- **Accept:** a user can sign up, land in an empty dashboard, sign out; routes protected.
- **Status: Not done — this is the most consequential gap in the whole repo, not a minor loose end.**
- **There is no Auth.js integration.** `lib/auth.ts` is a from-scratch, ~40-line HMAC-SHA256 session
scaffold (Web Crypto, no dependency) whose own comment states: "Production auth = Auth.js + Postgres
adapter + OAuth (docs/02) — wired once the database is provisioned." `app/api/auth/login/route.ts`
is a **dev-only** login gated by a shared `DEV_LOGIN_KEY`, not sign-up/verify/forgot.
- `orgs` and `users` tables exist in `sql/001_init.sql` (schema is ready), but nothing in the app
reads/writes them for real session-bound identity today.
- `app/portal/page.tsx` and sibling portal pages exist (shell is real), but routes are **not**
protected by any real session check tied to that schema.
- **Concretely, this means the tenant/org isolation several `0pt.*` API routes assume does not exist
yet.** `app/api/0pt/lakehouse/mappings/route.ts`, `.../writes/route.ts`,
`app/api/0pt/sources/rate-policies/route.ts`, `app/api/0pt/sources/[sourceId]/query/route.ts`, and
`src/services/document-capture-service.ts` all read `tenant_id`/`project_id` straight from the
request body — anyone can claim any tenant. `app/api/orders/route.ts`'s own `GET` handler carries
the literal comment `// org scoping added with auth`. This is an open security gap, not a
two-tier design choice — see `docs/08-infrastructure.md`'s "Security & compliance" section for the
full list. **Do not build any doc or feature that assumes tenant-scoped auth already exists.**
## Phase 4 — Order lifecycle end-to-end (test mode)
- **Configurator** (2.12) wired to `POST /estimates` (real estimate math in FastAPI).
- **Deposit checkout** (2.13) → Stripe **test** Checkout → webhook → order `IN_PRODUCTION`.
- **Production tracker** (2.14) reading `GET /orders/{id}/events` (SSE) — pipeline may be **simulated** at this phase.
- **Validation & delivery** (2.15): tabs render from a sample delivery; **accept** → `POST /payments/release` (test) → `DELIVERED`; **revise** leaves escrow untouched.
- AI-assist (`POST /ai/parse`) prefilling the configurator.
- **Accept:** a full custom order can be configured, deposit paid (test), watched, validated, accepted, and "delivered" — state machine in `docs/01` traversed; escrow split correct.
- **Status: Partial — the payment/escrow spine is genuinely real; the SSE tracker is not.**
- `app/api/estimates/route.ts` exists.
- Deposit checkout is **real, working code**: `app/api/payments/[order_id]/deposit/route.ts` calls
Stripe's Checkout Sessions API directly (not a stub) and `app/api/webhooks/stripe/route.ts` verifies
the webhook signature and advances order status via `setStatus`.
- `app/api/payments/[order_id]/release/route.ts` is also real: it enforces the escrow rule in code —
balance release is rejected with `409 not_accepted` unless `order.status === "accepted"` — matching
the "escrow split correct" accept criterion's intent.
- **`GET /api/orders/{id}/events` (SSE) does not exist.** Only `GET`/`PATCH /api/orders/{id}` were
found (`app/api/orders/[id]/route.ts`); no `/events` sub-route, and no `EventSource`/
`text/event-stream`/`ReadableStream` usage anywhere in `app/` (grep-verified). If a production
tracker is live today, it's polling `GET /api/orders/{id}`, not the SSE stream this bullet describes
— or it hasn't been built. Either way, the SSE contract as written is not implemented.
- `app/api/ai/parse/route.ts` is real: a genuine (non-fabricated) keyword/regex heuristic that
prefills a config object, with a documented fallback to `engineFetch("/ai/parse", ...)` when
`ENGINE_URL` is set. This one holds up as described.
## Phase 5 — Real data engine (Nordic slice)
- Compliant connectors first: **GLEIF + Brreg + Companies House** (verify access terms; record in `sources.yaml`). Remove any FlareSolverr/bypass path (`docs/06`).
- Pipeline stages produce real `dim_company` + `fact_financials` with `bridge_provenance`; QA threshold drops low-confidence rows.
- Production tracker now reflects **real** job metrics.
- **Accept:** a Nordic company dataset is produced for a real order, every field has provenance, quality report generated, level-0 fields shown honestly (not fabricated).
- **Status: Partial.**
- `services/api/sources/sources.yaml` and `connectors.py` are real: GLEIF, Brreg, Companies House,
PRH, and CVR are all registered, each with an explicit `licence: "... — VERIFY"` tag rather than an
assumed-clean claim — matches the "verify access terms" instruction.
- `services/api/workers/adaptive_router.py`'s own comment confirms: "There are NO bypass/stealth/
proxy-rotation tiers here, by design (CLAUDE.md §1)" — the FlareSolverr/bypass removal is done, at
least on this repo's `main` (see `docs/08-infrastructure.md`'s "Known open question" about the
diverged Gitea remote, which independently added such a stack on its own `main` — unresolved which
remote is canonical).
- `services/api/workers/pipeline.py` has a real, working confidence-threshold QA drop: `delivered =
[r for r in deduped if r["confidence"] >= threshold]`, plus an `avg_confidence` figure in its output.
This is a genuine, narrow QA mechanism inside the Python engine.
- **This is not the same thing as the "quality report" / quality-check layer once described in
`docs/ICEBERG_WRITE_BOUNDARY.md`, `docs/11-lakehouse-write-plan.md`, and
`docs/P0_QUERY_SOURCE_FLOW.md`.** Those three docs used to cite `src/services/quality-check-service.ts`,
`src/services/quality-report-service.ts`, `src/types/quality-{check,report}.ts`, and
`src/data-platform/services/catalog-service.ts` as already-built and tested. **None of those files
has ever existed in this repo's git history, on any branch** — and all three docs have since been
corrected to say so explicitly (verified this pass), rather than repeating the original claim. So:
the narrow Python-side confidence filter is real; the broader TS/lakehouse "quality report" this
Accept criterion likely also intends is not built (see `docs/P0_DATASET_QUALITY_FLOW.md` for the
honest target spec). Whether `pipeline.py`'s `avg_confidence` output satisfies "quality report
generated" is a judgment call for whoever runs this phase — flagging rather than deciding it here.
- Whether an actual Nordic dataset has been produced end-to-end for a real order was not verified
this pass (would require running the stack).
## Phase 6 — Delivery surfaces & integrations
- Exports: CSV / Parquet / JSON via R2 presigned URLs.
- Embedded **NocoDB** ("Min data") and **Metabase** dashboards via signed URLs in mono chrome.
- Integrations: **FiveCRM push**, webhooks, Google Sheets.
- File **uploads** (Uppy → R2) for matching/enrichment.
- **Accept:** a delivered dataset is downloadable, viewable in embedded NocoDB + Metabase (no separate login), and pushable to FiveCRM; uploads work.
- **Status: Not done — this phase is UI-shell-only today.**
- CSV/Parquet/JSON exports via R2 presigned URLs: not found. `app/api/uploads/route.ts` (the closest
R2-touching code) is a `501` stub with a hardcoded `put_url: null` — no presign logic exists yet,
for uploads or exports.
- `components/embed.tsx` (`EmbeddedView`) is real — a sandboxed iframe shell — but the server-side
signed-URL generator for Metabase/NocoDB (`METABASE_SECRET_KEY`/`NOCODB_API_TOKEN`) that would feed
it a real `src` was not found anywhere in this pass.
- "FiveCRM" appears only as a **destination label string** in `lib/data.ts`, `lib/services.ts`,
`app/portal/integrationer/page.tsx` (explicitly described in that file's own comment as
"presentationella toggles" — i.e. presentational, not wired), and as one of several regex-matched
destination names in `app/api/ai/parse/route.ts`'s heuristic. **No actual FiveCRM API push exists.**
- `uppy` is not a dependency in `package.json` (checked directly) — the Uppy → R2 upload flow this
bullet describes has no corresponding package installed, let alone implemented.
## Phase 7 — Subscriptions / feeds
- `subscriptions`; recurring jobs (weekly/daily) delivering **deltas**; webhooks on change.
- Freshness monitoring; Resend emails (confirmation, "din data är klar", feedback).
- **Accept:** an order converts to a feed; a scheduled run produces a delta + fires a webhook + email.
- **Status: Partial — data layer real, scheduling and email are not.**
- `lib/subscriptions-store.ts` is real CRUD against the `subscriptions` table (with an in-memory
fallback when no DB is configured) — `createSubscription`/`listSubscriptions` genuinely insert/read.
- No recurring-job scheduler (cron, queue consumer producing deltas, etc.) was found on the TS side of
this repo. `services/api/workers/queue.py` exists on the Python side but this pass did not trace
whether it implements actual delta-producing scheduled runs — flagged as unverified rather than
assumed either way.
- **Resend is not wired at all** — grepped for `resend`/`Resend` across the whole repo, zero matches
outside `RESEND_API_KEY` being listed as an env var. No confirmation/"din data är klar"/feedback
email has been implemented.
## Phase 8 — Admin, hardening, compliance
- Admin: order queue + scope/quote + escrow release; **source registry & freshness** monitoring; worker/job monitoring; customer + catalog management.
- Customer **DB-access grant** flow (encrypted, read-only, time-boxed) + revoke.
- Security pass: RLS, presigned-only files, signed embeds, secret rotation, `audit_log` everywhere; **DPA** published; legal pages.
- Finalize all loading/empty/error states + mobile across the app.
- **Accept:** admin can run the business; DB-grant flow works and self-revokes; security checklist passes; DoD met app-wide.
- **Status: Partial — UI shell real, the security substance is not.**
- Admin pages exist: `app/admin/{page,jobs/page,sources/page,customers/page,verktyg/page}.tsx`. This
pass did not verify whether they read real backend data or placeholders.
- `sql/001_init.sql` defines `audit_log`, `db_grants`, and `payments` tables — schema is ready.
- **The DB-access grant flow is a stub, not a working feature.**
`app/api/connections/db-grant/route.ts` returns `501` without `GRANT_ENC_KEY`, and with it set
returns a hardcoded `201 { grant_id, status: "pending" }` — **no AES-GCM encryption, no read-only
validation, and no self-revoke logic exist in code today**, despite the route's own comment
describing that target behavior. See `docs/08-infrastructure.md`'s "Security & compliance" section.
- RLS enforcement, presigned-only files (blocked on the Phase 6 R2 gap above), signed embeds (blocked
on the Phase 6 NocoDB/Metabase gap above), and a published DPA were not found as implemented in this
pass. Treat the "security checklist passes" accept criterion as not met.
## Phase 9 — Launch
- Domain `0pt.app` (Vercel) + subdomains (api/data/dash via Caddy). Sentry + uptime live.
- Soft launch: Protosell internally + a handful of existing dealer relationships.
- **Accept:** a real external customer completes an order start→delivery in production; feedback loop running.
- **Status: Domain is live; soft-launch/customer criterion not verified.**
- **Verified this pass** (network check performed): `https://0pt.app` returns HTTP 200 and serves this
repo's own Next.js app (title "0pt — your data", SIAX branding); `GET /api/health` returns
`{"ok":true,"service":"0pt-web","db":"not_configured","queue":"not_configured",...}`, an exact shape
match for `app/api/health/route.ts`. README's "Live: https://0pt.app" is confirmed, and the response
confirms the live deployment has no `DATABASE_URL`/`REDIS_URL` set — see `docs/08-infrastructure.md`'s
"Hosting topology" section. This does not confirm which git remote/commit is deployed (see the
diverged-remotes note elsewhere in this doc), or that api/data/dash subdomains or Caddy are live —
only the root Vercel deployment was checked.
- **Sentry is not wired** — no `sentry` package in `package.json`, no Sentry config file anywhere in
the repo (grep-confirmed). `@vercel/analytics` **is** genuinely wired (`app/layout.tsx` renders
`<Analytics />`), which is a real but much smaller piece of the "Sentry + uptime live" bullet.
- No evidence either way on the soft-launch/external-customer criterion — out of this doc's ability to
verify from code alone.
---
## Working conventions (for Claude Code)
- Tech docs + identifiers **English**; UI copy **Swedish**.
- Conventional Commits; one concern per PR; keep diffs reviewable.
- No secrets in code; update `.env.example` when adding env. (`.env.example` itself currently lives on
`feature/0pt-env-example` / PR #3 — merge that first, or this instruction has nothing to update yet.)
- Add/extend tests for engine logic (estimate math, fusion/confidence, dedup) and API contract.
- When a source's access/licence is unclear, **verify and record in `sources.yaml`** before relying; if disallowed for bulk, route to licensed/partner or mark level 0 — never bypass, never fabricate.
- Touch the design tokens/motion only via `docs/05`; keep red ≤ once per screen; no photography.
- If anything here conflicts with existing code, **flag it** in the PR description and propose the reconciliation rather than guessing.
- **Two git remotes claim to be this repo** (`github.com/sax3l/0pt` and `git.siax.io/sax3l/0pt`), and
they have independently diverged (see `CLAUDE.md`, `README.md`). Don't assume the checkout you're in
is the only copy, and don't silently pick one as canonical — that's Simon's decision, flag it if it
becomes relevant to the phase you're working on.