246 lines
20 KiB
Markdown
246 lines
20 KiB
Markdown
# 09 · Build plan
|
||
|
||
Phased, in dependency order. **Finish a phase to its acceptance criteria before the next.** Each phase = small, reviewable commits. Reconcile with existing code rather than rewriting wholesale.
|
||
|
||
> **Status callouts added 2026-07-19.** Each phase below now carries a `Status:` line, grep/read-verified
|
||
> against `main` this pass. These are conservative on purpose: a phase is only called **Done** when this
|
||
> pass found real, working code satisfying its Accept criteria; anything else is **Partial** (with the
|
||
> specific evidence and gap named) or **Not started**. No phase is upgraded to Done without a code
|
||
> citation. Where a referenced doc (e.g. `docs/design/...`) doesn't exist, that's flagged rather than
|
||
> silently worked around. See `docs/08-infrastructure.md` for the full env/security grounding these
|
||
> statuses draw on.
|
||
|
||
## Phase 0 — Foundation & repo
|
||
- Monorepo structure (`docs/08`). Migrate existing `src/orchestrator` + `src/workers` into `services/api`.
|
||
- Postgres schema + migrations (`docs/03`); `pgvector` enabled. `.env.example`.
|
||
- `docker-compose` up (postgres, redis, api, worker, nocodb, metabase, caddy). `/health` green.
|
||
- CI skeleton (lint/typecheck/build).
|
||
- **Accept:** `docker compose up` runs the whole stack locally; migrations apply; `/health` returns ok for db + queue.
|
||
- **Status: Partial.**
|
||
- Repo structure exists but diverged from the `apps/web`/`packages/shared` layout `docs/08` originally
|
||
sketched — it settled on `app/` (Next.js, repo root), `lib/`, `src/` (platform clients/services), and
|
||
`services/api/` (FastAPI). `docs/08-infrastructure.md` now documents the structure as-built.
|
||
- Migration to `services/api/orchestrator` + `services/api/workers` is **done** (verified: those
|
||
directories exist and contain `main.py`/`adaptive_router.py`/`pipeline.py`/etc.; no `src/orchestrator`
|
||
or `src/workers` remain at the old paths).
|
||
- Postgres schema is real: `sql/001_init.sql` (`CREATE EXTENSION vector`, `CREATE EXTENSION pgcrypto`,
|
||
17 tables incl. `orgs`, `users`, `orders`, `jobs`, `audit_log`, `bridge_provenance`, `db_grants`,
|
||
`payments`) + `sql/004_iceberg_write_boundary.sql`.
|
||
**No `sql/002_*` or `sql/003_*` exists anywhere in this repo's history** — some other docs
|
||
(`docs/ICEBERG_WRITE_BOUNDARY.md` etc.) cite a `sql/002_p0_dataset_quality.sql` that was never built;
|
||
see `docs/08-infrastructure.md`'s "Schema state" section.
|
||
- `.env.example` — **now exists and is merged to `main`** (BACKLOG P3-5: the
|
||
stale link to the unmerged `github.com/sax3l/0pt/pull/3` that used to sit
|
||
here has been removed — that PR has long since landed, `.env.example` is
|
||
a real, current, gitignore-exempted file at the repo root today).
|
||
- `infra/docker-compose.yml` and `infra/Caddyfile` are real, committed files (not sketches) covering
|
||
exactly the services listed. **This pass found no evidence they've ever actually been run** — no
|
||
CI, no deploy log, no reachable host checked. "File exists and looks correct" ≠ "verified to run";
|
||
treat the Accept criterion ("docker compose up runs... /health returns ok") as unverified.
|
||
- CI skeleton — **not built.** There is no `.github/workflows/` directory in this repo at all. `pnpm
|
||
lint`/`typecheck`/`test` exist as local scripts only.
|
||
|
||
## Phase 1 — Design system in code
|
||
- Tailwind + shadcn; `globals.css` (grain + reveal keyframes) and `useReveal` verbatim (`docs/05`).
|
||
- Build **DataArtifact** primitive (table/map/schema/chart/grid) and **ProvenanceBadge** (`docs/design` prompts 5.1, 2.11).
|
||
- **Accept:** a sample page renders mono tokens, a DataArtifact animates on scroll, reduced-motion respected, focus rings visible.
|
||
- **Status: Partial — real components exist; scroll-motion/reduced-motion/focus-ring behavior not
|
||
independently exercised this pass.**
|
||
- Tailwind + shadcn/radix stack: confirmed real in `package.json` (`tailwindcss`, `@radix-ui/*`,
|
||
`class-variance-authority`, `tailwind-merge`, `tailwindcss-animate`).
|
||
- `hooks/use-reveal.ts`, `components/data-artifact.tsx`, `components/provenance-badge.tsx` all exist
|
||
and are used across `app/katalog`, `app/kallor`, `app/bestall`, `app/order/[id]`, and most
|
||
`components/sections/*` files (grep-confirmed).
|
||
- `docs/design/0pt-v0-build-brief.md` and `docs/design/0pt-design-coverage-and-visuals.md` (the
|
||
"prompts 5.1, 2.11" this bullet references) **do not exist anywhere in this repo** — confirmed via
|
||
full-repo search; there is no `docs/design/` directory at all. `CLAUDE.md`'s own doc index has
|
||
already been corrected to flag this as a gap rather than list them as real. Treat any reference to
|
||
specific numbered "build-brief prompts" in this build plan as pointing at a doc that was never
|
||
written — not this file's scope to author that content.
|
||
- Reduced-motion handling, focus-ring visibility, and actual scroll-triggered animation behavior were
|
||
not executed/verified in this pass (no browser run performed) — code presence is confirmed, runtime
|
||
behavior is not.
|
||
|
||
## Phase 2 — Marketing + catalog (placeholder data)
|
||
- Landing components top-to-bottom (build-brief 2.1–2.10): Header, Hero, Philosophy, Catalog teaser, Engine, Visualisation gallery, Coverage, Editorial, Proof, Footer.
|
||
- Full **Catalog** page + **Dataset detail** + **Sourcing-map** page (coverage doc 5.4). Postgres full-text search.
|
||
- All with loading/empty/error states + mobile.
|
||
- **Accept:** marketing site + catalog browse work end-to-end on placeholder catalog rows; one continuous scroll reads well; DoD met per screen.
|
||
- **Status: Partial.**
|
||
- Landing section components are real: `components/sections/{hero,philosophy,gallery,editorial,
|
||
pricing,technology,testimonials,footer,featured-products,collection}-section.tsx` all exist.
|
||
- `app/katalog/page.tsx`, `app/katalog/[id]/page.tsx`, `app/kallor/page.tsx` all exist.
|
||
- **Postgres full-text search was not found** — grepped for `to_tsvector`/`websearch_to_tsquery`
|
||
across the whole repo, zero matches. Catalog browsing today reads from `lib/data.ts` placeholder
|
||
data, not a full-text query against Postgres. This Accept criterion's "catalog browse... on
|
||
placeholder catalog rows" half is plausible; the Postgres full-text search half is not built.
|
||
- Loading/empty/error states + mobile + DoD-per-screen were not independently audited screen-by-screen
|
||
this pass — `components/states.tsx` (`LoadingSkeleton`, `ErrorState`) exists and is imported widely,
|
||
which is evidence toward this but not a full verification.
|
||
|
||
## Phase 3 — Auth & portal shell
|
||
- Auth.js + Postgres adapter; sign in/up/verify/forgot. `orgs`/`users`.
|
||
- Portal shell + Dashboard ("Mina beställningar") with empty state.
|
||
- **Accept:** a user can sign up, land in an empty dashboard, sign out; routes protected.
|
||
- **Status: Not done — this is the most consequential gap in the whole repo, not a minor loose end.**
|
||
- **There is no Auth.js integration.** `lib/auth.ts` is a from-scratch, ~40-line HMAC-SHA256 session
|
||
scaffold (Web Crypto, no dependency) whose own comment states: "Production auth = Auth.js + Postgres
|
||
adapter + OAuth (docs/02) — wired once the database is provisioned." `app/api/auth/login/route.ts`
|
||
is a **dev-only** login gated by a shared `DEV_LOGIN_KEY`, not sign-up/verify/forgot.
|
||
- `orgs` and `users` tables exist in `sql/001_init.sql` (schema is ready), but nothing in the app
|
||
reads/writes them for real session-bound identity today.
|
||
- `app/portal/page.tsx` and sibling portal pages exist (shell is real), but routes are **not**
|
||
protected by any real session check tied to that schema.
|
||
- **Concretely, this means the tenant/org isolation several `0pt.*` API routes assume does not exist
|
||
yet.** `app/api/0pt/lakehouse/mappings/route.ts`, `.../writes/route.ts`,
|
||
`app/api/0pt/sources/rate-policies/route.ts`, `app/api/0pt/sources/[sourceId]/query/route.ts`, and
|
||
`src/services/document-capture-service.ts` all read `tenant_id`/`project_id` straight from the
|
||
request body — anyone can claim any tenant. `app/api/orders/route.ts`'s own `GET` handler carries
|
||
the literal comment `// org scoping added with auth`. This is an open security gap, not a
|
||
two-tier design choice — see `docs/08-infrastructure.md`'s "Security & compliance" section for the
|
||
full list. **Do not build any doc or feature that assumes tenant-scoped auth already exists.**
|
||
|
||
## Phase 4 — Order lifecycle end-to-end (test mode)
|
||
- **Configurator** (2.12) wired to `POST /estimates` (real estimate math in FastAPI).
|
||
- **Deposit checkout** (2.13) → Stripe **test** Checkout → webhook → order `IN_PRODUCTION`.
|
||
- **Production tracker** (2.14) reading `GET /orders/{id}/events` (SSE) — pipeline may be **simulated** at this phase.
|
||
- **Validation & delivery** (2.15): tabs render from a sample delivery; **accept** → `POST /payments/release` (test) → `DELIVERED`; **revise** leaves escrow untouched.
|
||
- AI-assist (`POST /ai/parse`) prefilling the configurator.
|
||
- **Accept:** a full custom order can be configured, deposit paid (test), watched, validated, accepted, and "delivered" — state machine in `docs/01` traversed; escrow split correct.
|
||
- **Status: Partial — the payment/escrow spine is genuinely real; the SSE tracker is not.**
|
||
- `app/api/estimates/route.ts` exists.
|
||
- Deposit checkout is **real, working code**: `app/api/payments/[order_id]/deposit/route.ts` calls
|
||
Stripe's Checkout Sessions API directly (not a stub) and `app/api/webhooks/stripe/route.ts` verifies
|
||
the webhook signature and advances order status via `setStatus`.
|
||
- `app/api/payments/[order_id]/release/route.ts` is also real: it enforces the escrow rule in code —
|
||
balance release is rejected with `409 not_accepted` unless `order.status === "accepted"` — matching
|
||
the "escrow split correct" accept criterion's intent.
|
||
- **`GET /api/orders/{id}/events` (SSE) does not exist.** Only `GET`/`PATCH /api/orders/{id}` were
|
||
found (`app/api/orders/[id]/route.ts`); no `/events` sub-route, and no `EventSource`/
|
||
`text/event-stream`/`ReadableStream` usage anywhere in `app/` (grep-verified). If a production
|
||
tracker is live today, it's polling `GET /api/orders/{id}`, not the SSE stream this bullet describes
|
||
— or it hasn't been built. Either way, the SSE contract as written is not implemented.
|
||
- `app/api/ai/parse/route.ts` is real: a genuine (non-fabricated) keyword/regex heuristic that
|
||
prefills a config object, with a documented fallback to `engineFetch("/ai/parse", ...)` when
|
||
`ENGINE_URL` is set. This one holds up as described.
|
||
|
||
## Phase 5 — Real data engine (Nordic slice)
|
||
- Compliant connectors first: **GLEIF + Brreg + Companies House** (verify access terms; record in `sources.yaml`). Remove any FlareSolverr/bypass path (`docs/06`).
|
||
- Pipeline stages produce real `dim_company` + `fact_financials` with `bridge_provenance`; QA threshold drops low-confidence rows.
|
||
- Production tracker now reflects **real** job metrics.
|
||
- **Accept:** a Nordic company dataset is produced for a real order, every field has provenance, quality report generated, level-0 fields shown honestly (not fabricated).
|
||
- **Status: Partial.**
|
||
- `services/api/sources/sources.yaml` and `connectors.py` are real: GLEIF, Brreg, Companies House,
|
||
PRH, and CVR are all registered, each with an explicit `licence: "... — VERIFY"` tag rather than an
|
||
assumed-clean claim — matches the "verify access terms" instruction.
|
||
- `services/api/workers/adaptive_router.py`'s own comment confirms: "There are NO bypass/stealth/
|
||
proxy-rotation tiers here, by design (CLAUDE.md §1)" — the FlareSolverr/bypass removal is done, at
|
||
least on this repo's `main` (see `docs/08-infrastructure.md`'s "Known open question" about the
|
||
diverged Gitea remote, which independently added such a stack on its own `main` — unresolved which
|
||
remote is canonical).
|
||
- `services/api/workers/pipeline.py` has a real, working confidence-threshold QA drop: `delivered =
|
||
[r for r in deduped if r["confidence"] >= threshold]`, plus an `avg_confidence` figure in its output.
|
||
This is a genuine, narrow QA mechanism inside the Python engine.
|
||
- **This is not the same thing as the "quality report" / quality-check layer once described in
|
||
`docs/ICEBERG_WRITE_BOUNDARY.md`, `docs/11-lakehouse-write-plan.md`, and
|
||
`docs/P0_QUERY_SOURCE_FLOW.md`.** Those three docs used to cite `src/services/quality-check-service.ts`,
|
||
`src/services/quality-report-service.ts`, `src/types/quality-{check,report}.ts`, and
|
||
`src/data-platform/services/catalog-service.ts` as already-built and tested. **None of those files
|
||
has ever existed in this repo's git history, on any branch** — and all three docs have since been
|
||
corrected to say so explicitly (verified this pass), rather than repeating the original claim. So:
|
||
the narrow Python-side confidence filter is real; the broader TS/lakehouse "quality report" this
|
||
Accept criterion likely also intends is not built (see `docs/P0_DATASET_QUALITY_FLOW.md` for the
|
||
honest target spec). Whether `pipeline.py`'s `avg_confidence` output satisfies "quality report
|
||
generated" is a judgment call for whoever runs this phase — flagging rather than deciding it here.
|
||
- Whether an actual Nordic dataset has been produced end-to-end for a real order was not verified
|
||
this pass (would require running the stack).
|
||
|
||
## Phase 6 — Delivery surfaces & integrations
|
||
- Exports: CSV / Parquet / JSON via R2 presigned URLs.
|
||
- Embedded **NocoDB** ("Min data") and **Metabase** dashboards via signed URLs in mono chrome.
|
||
- Integrations: **FiveCRM push**, webhooks, Google Sheets.
|
||
- File **uploads** (Uppy → R2) for matching/enrichment.
|
||
- **Accept:** a delivered dataset is downloadable, viewable in embedded NocoDB + Metabase (no separate login), and pushable to FiveCRM; uploads work.
|
||
- **Status: Not done — this phase is UI-shell-only today.**
|
||
- CSV/Parquet/JSON exports via R2 presigned URLs: not found. `app/api/uploads/route.ts` (the closest
|
||
R2-touching code) is a `501` stub with a hardcoded `put_url: null` — no presign logic exists yet,
|
||
for uploads or exports.
|
||
- `components/embed.tsx` (`EmbeddedView`) is real — a sandboxed iframe shell — but the server-side
|
||
signed-URL generator for Metabase/NocoDB (`METABASE_SECRET_KEY`/`NOCODB_API_TOKEN`) that would feed
|
||
it a real `src` was not found anywhere in this pass.
|
||
- "FiveCRM" appears only as a **destination label string** in `lib/data.ts`, `lib/services.ts`,
|
||
`app/portal/integrationer/page.tsx` (explicitly described in that file's own comment as
|
||
"presentationella toggles" — i.e. presentational, not wired), and as one of several regex-matched
|
||
destination names in `app/api/ai/parse/route.ts`'s heuristic. **No actual FiveCRM API push exists.**
|
||
- `uppy` is not a dependency in `package.json` (checked directly) — the Uppy → R2 upload flow this
|
||
bullet describes has no corresponding package installed, let alone implemented.
|
||
|
||
## Phase 7 — Subscriptions / feeds
|
||
- `subscriptions`; recurring jobs (weekly/daily) delivering **deltas**; webhooks on change.
|
||
- Freshness monitoring; Resend emails (confirmation, "din data är klar", feedback).
|
||
- **Accept:** an order converts to a feed; a scheduled run produces a delta + fires a webhook + email.
|
||
- **Status: Partial — data layer real, scheduling and email are not.**
|
||
- `lib/subscriptions-store.ts` is real CRUD against the `subscriptions` table (with an in-memory
|
||
fallback when no DB is configured) — `createSubscription`/`listSubscriptions` genuinely insert/read.
|
||
- No recurring-job scheduler (cron, queue consumer producing deltas, etc.) was found on the TS side of
|
||
this repo. `services/api/workers/queue.py` exists on the Python side but this pass did not trace
|
||
whether it implements actual delta-producing scheduled runs — flagged as unverified rather than
|
||
assumed either way.
|
||
- **Resend is not wired at all** — grepped for `resend`/`Resend` across the whole repo, zero matches
|
||
outside `RESEND_API_KEY` being listed as an env var. No confirmation/"din data är klar"/feedback
|
||
email has been implemented.
|
||
|
||
## Phase 8 — Admin, hardening, compliance
|
||
- Admin: order queue + scope/quote + escrow release; **source registry & freshness** monitoring; worker/job monitoring; customer + catalog management.
|
||
- Customer **DB-access grant** flow (encrypted, read-only, time-boxed) + revoke.
|
||
- Security pass: RLS, presigned-only files, signed embeds, secret rotation, `audit_log` everywhere; **DPA** published; legal pages.
|
||
- Finalize all loading/empty/error states + mobile across the app.
|
||
- **Accept:** admin can run the business; DB-grant flow works and self-revokes; security checklist passes; DoD met app-wide.
|
||
- **Status: Partial — UI shell real, the security substance is not.**
|
||
- Admin pages exist: `app/admin/{page,jobs/page,sources/page,customers/page,verktyg/page}.tsx`. This
|
||
pass did not verify whether they read real backend data or placeholders.
|
||
- `sql/001_init.sql` defines `audit_log`, `db_grants`, and `payments` tables — schema is ready.
|
||
- **The DB-access grant flow is a stub, not a working feature.**
|
||
`app/api/connections/db-grant/route.ts` returns `501` without `GRANT_ENC_KEY`, and with it set
|
||
returns a hardcoded `201 { grant_id, status: "pending" }` — **no AES-GCM encryption, no read-only
|
||
validation, and no self-revoke logic exist in code today**, despite the route's own comment
|
||
describing that target behavior. See `docs/08-infrastructure.md`'s "Security & compliance" section.
|
||
- RLS enforcement, presigned-only files (blocked on the Phase 6 R2 gap above), signed embeds (blocked
|
||
on the Phase 6 NocoDB/Metabase gap above), and a published DPA were not found as implemented in this
|
||
pass. Treat the "security checklist passes" accept criterion as not met.
|
||
|
||
## Phase 9 — Launch
|
||
- Domain `0pt.app` (Vercel) + subdomains (api/data/dash via Caddy). Sentry + uptime live.
|
||
- Soft launch: Protosell internally + a handful of existing dealer relationships.
|
||
- **Accept:** a real external customer completes an order start→delivery in production; feedback loop running.
|
||
- **Status: Domain is live; soft-launch/customer criterion not verified.**
|
||
- **Verified this pass** (network check performed): `https://0pt.app` returns HTTP 200 and serves this
|
||
repo's own Next.js app (title "0pt — your data", SIAX branding); `GET /api/health` returns
|
||
`{"ok":true,"service":"0pt-web","db":"not_configured","queue":"not_configured",...}`, an exact shape
|
||
match for `app/api/health/route.ts`. README's "Live: https://0pt.app" is confirmed, and the response
|
||
confirms the live deployment has no `DATABASE_URL`/`REDIS_URL` set — see `docs/08-infrastructure.md`'s
|
||
"Hosting topology" section. This does not confirm which git remote/commit is deployed (see the
|
||
diverged-remotes note elsewhere in this doc), or that api/data/dash subdomains or Caddy are live —
|
||
only the root Vercel deployment was checked.
|
||
- **Sentry is not wired** — no `sentry` package in `package.json`, no Sentry config file anywhere in
|
||
the repo (grep-confirmed). `@vercel/analytics` **is** genuinely wired (`app/layout.tsx` renders
|
||
`<Analytics />`), which is a real but much smaller piece of the "Sentry + uptime live" bullet.
|
||
- No evidence either way on the soft-launch/external-customer criterion — out of this doc's ability to
|
||
verify from code alone.
|
||
|
||
---
|
||
|
||
## Working conventions (for Claude Code)
|
||
- Tech docs + identifiers **English**; UI copy **Swedish**.
|
||
- Conventional Commits; one concern per PR; keep diffs reviewable.
|
||
- No secrets in code; update `.env.example` when adding env. (`.env.example` itself currently lives on
|
||
`feature/0pt-env-example` / PR #3 — merge that first, or this instruction has nothing to update yet.)
|
||
- Add/extend tests for engine logic (estimate math, fusion/confidence, dedup) and API contract.
|
||
- When a source's access/licence is unclear, **verify and record in `sources.yaml`** before relying; if disallowed for bulk, route to licensed/partner or mark level 0 — never bypass, never fabricate.
|
||
- Touch the design tokens/motion only via `docs/05`; keep red ≤ once per screen; no photography.
|
||
- If anything here conflicts with existing code, **flag it** in the PR description and propose the reconciliation rather than guessing.
|
||
- **Two git remotes claim to be this repo** (`github.com/sax3l/0pt` and `git.siax.io/sax3l/0pt`), and
|
||
they have independently diverged (see `CLAUDE.md`, `README.md`). Don't assume the checkout you're in
|
||
is the only copy, and don't silently pick one as canonical — that's Simon's decision, flag it if it
|
||
becomes relevant to the phase you're working on.
|