feat: rebrand MONO->0pt (hero, header, layout) + add project docs/contract
Hero is now a data-native 0pt artifact (no photos, keeps grain/reveal/float); header is the 0pt wordmark with Swedish nav; layout is sv + loads JetBrains Mono + 0pt metadata; globals wires the loaded fonts. Adds CLAUDE.md + docs/01-09 as the build contract. next build green (5 routes). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,49 @@
|
||||
# 01 · Product
|
||||
|
||||
## Vision
|
||||
|
||||
0pt is the most flexible, honest data-services platform for the Nordic + European B2B market. Customers order any kind of data work — from ready-made datasets to bespoke pipelines — and get full visibility into where every value came from and how confident it is. The differentiator is **honest accuracy**: provenance per field, freshness tracking, source transparency. Incumbents (Apollo, Cognism, ZoomInfo) overclaim; 0pt shows its work.
|
||||
|
||||
## The three flows (everything reduces to these)
|
||||
|
||||
Every service maps to one of three intake/delivery flows. The nine "categories" (scraping, structuring, datasets, ICP/leads, BI, ML, documents, subscriptions, custom) are **tags/filters within these flows**, not separate machinery.
|
||||
|
||||
1. **Catalog product** — ready-made, browse → sample → buy → deliver. Exists in Postgres or produced fast from the pipeline. Margin + scale.
|
||||
2. **Custom job** — does not exist yet; must be scoped. `configure (AI-assist) → estimate → manual scope/quote confirm → produce → QA → validate → deliver`. The hero flow. Consulting in a portal.
|
||||
3. **Subscription / feed** — recurring product or job: lead feeds, freshness monitoring, ongoing enrichment. `configure → recurring payment → recurring delivery/webhook`.
|
||||
|
||||
## Order lifecycle (state machine)
|
||||
|
||||
```
|
||||
DRAFT → QUOTED → DEPOSIT_PENDING → IN_PRODUCTION → REVIEW
|
||||
→ (REVISION → IN_PRODUCTION)* → ACCEPTED → DELIVERED → [SUBSCRIPTION_ACTIVE]
|
||||
```
|
||||
|
||||
- **DRAFT**: configurator in progress.
|
||||
- **QUOTED**: estimate produced (catalog = instant; custom = after scope confirm).
|
||||
- **DEPOSIT_PENDING**: awaiting 50 % deposit.
|
||||
- **IN_PRODUCTION**: deposit captured → workers run. Customer watches live.
|
||||
- **REVIEW**: QA passed threshold → customer validates (table/map/schema/quality).
|
||||
- **REVISION**: customer requested changes → escrow untouched → back to production.
|
||||
- **ACCEPTED**: customer approves → remaining 50 % released from escrow.
|
||||
- **DELIVERED**: exports + destinations unlocked; provenance log + quality report attached.
|
||||
- **SUBSCRIPTION_ACTIVE**: if converted to a feed.
|
||||
|
||||
## Business / escrow model
|
||||
|
||||
- **50 / 50.** Deposit (50 %) captured to start production. Balance (50 %) **held** and only captured/released on **ACCEPTED**. Revision never moves the held amount.
|
||||
- Pricing models: pay-per-job, volume, subscription (monthly), custom quote.
|
||||
- Estimate is a function of: selected field weights × markets × volume × entity multiplier × confidence-target multiplier + enrichment. Custom jobs get a manual scope/quote gate before DEPOSIT_PENDING.
|
||||
- Stripe: deposit = capture now; balance = either a held authorization released on accept, or a second invoice on accept (implement as a second Checkout/invoice keyed to the order; see `docs/04-api.md`).
|
||||
|
||||
## Customer DB access (designed, not a checkbox)
|
||||
|
||||
Default: the customer **pushes** data (upload/export) — we rarely need to read their DB. If we must: a **read-only, table-scoped, time-boxed** role the customer creates, IP-allowlisted, ideally against a replica. Creds encrypted, deleted after the job. Everything logged. This is a selling point to compliance-heavy buyers, and it requires a DPA.
|
||||
|
||||
## Personas (brief — for copy + prioritization)
|
||||
|
||||
SaaS-Martin (RevOps, wants clean ICP/leads), Bilhandlar-Bengt (vehicle + company data via VehIQ), Rekryterar-Rebecka (people/firmographics), Investor-Ivar (diligence, sanctions/UBO, scrutinizes compliance), Ekonomi-Erik (financials, scrutinizes compliance), Data-David (raw datasets + API). Ivar and Erik are *why* honest sourcing + DPA must be airtight.
|
||||
|
||||
## First-launch scope (de-risked)
|
||||
|
||||
Do **not** ship "20 markets × 9 categories." Ship the thinnest sellable slice: **Nordic company data** (the open stuff — GLEIF + Brreg + verified sources) + **vehicle data** (VehIQ synergy) as a couple of catalog datasets, **plus the custom-job intake**, Stripe, and delivery to download + into FiveCRM/Data Loader. First customers are Protosell internally + existing dealer relationships — services you already deliver, productized. Then expand markets as the sourcing map fills in.
|
||||
@@ -0,0 +1,70 @@
|
||||
# 02 · Architecture
|
||||
|
||||
## Principle: one database, many lenses
|
||||
|
||||
Postgres is the source of truth. Everything else is a **lens** on it. A tool enters the stack only if it (a) reads that Postgres and (b) sits behind the same auth. This is the difference between an app and a pile of separate logins.
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────┐
|
||||
COLLECTION │ POSTGRES │ LENSES (customer + admin)
|
||||
(legal engine) │ dim_company · dim_contact · dim_vehicle │
|
||||
─────────────── │ fact_financials · dim_ubo │ ─────────────────────────────
|
||||
Official APIs │ dim_source · bridge_provenance │ Next.js portal (Vercel)
|
||||
/bulk ───────▶│ orders · order_items · jobs · deliveries │◀── Catalog + order lifecycle
|
||||
Authorized web │ subscriptions · users · orgs · files │ NocoDB (embedded "Min data")
|
||||
Licensed data │ db_grants · audit_log · pgvector embeddings │ Metabase (embedded dashboards)
|
||||
Customer upload│ │ Export: CSV/Parquet/JSON/API/FiveCRM
|
||||
└───────────────────▲──────────────────────────┘
|
||||
│
|
||||
FastAPI orchestrator + workers
|
||||
(estimates, orders, jobs, AI-assist, webhooks, pipeline)
|
||||
```
|
||||
|
||||
## Components & responsibilities
|
||||
|
||||
- **Next.js (Vercel)** — all customer/admin UI: marketing site, catalog, configurator, checkout, production tracker, validation, portal, admin. Calls FastAPI over an internal token. Embeds NocoDB/Metabase via signed URLs (no separate login shown).
|
||||
- **FastAPI orchestrator** (`src/orchestrator`) — REST contract in `docs/04-api.md`: estimates, orders, AI-assist, uploads, deliveries, webhooks, catalog, sources, job status/SSE. **Not on Vercel** (serverless timeouts kill long jobs).
|
||||
- **Workers** (`src/workers`) — adaptive router + pipeline stages (sourcing → extraction/OCR → structuring → enrichment/fusion → QA). Write to Postgres with provenance. See `docs/06-data-engine.md`.
|
||||
- **Postgres** — backbone (`docs/03-data-model.md`) incl. `pgvector` for matching/embeddings.
|
||||
- **NocoDB** — Airtable-style views over existing tables (no data duplication). Embedded into portal as "Min data".
|
||||
- **Metabase** — customer-facing dashboards. Embedded.
|
||||
- **Cloudflare R2** — file storage (uploads + deliverables) via presigned URLs.
|
||||
- **Stripe** — deposit capture + escrow balance release.
|
||||
- **Resend** — transactional email (order confirmation, "din data är klar", feedback).
|
||||
- **Auth.js** (NextAuth) with a Postgres adapter — single auth for the whole app; NocoDB/Metabase hidden behind it.
|
||||
|
||||
## Stack decisions — IN
|
||||
|
||||
| Concern | Choice | Why |
|
||||
|---|---|---|
|
||||
| Frontend | Next.js App Router + TS + Tailwind + shadcn/ui | Vercel-native, matches existing skills |
|
||||
| Backend/jobs | FastAPI + workers (existing) | already built; long-running jobs |
|
||||
| DB | Postgres (+ pgvector) | backbone; Neon free tier or on-VPS |
|
||||
| Airtable layer | **NocoDB** | sits on existing tables, no duplication |
|
||||
| BI | **Metabase** | customer-facing embedding, simple |
|
||||
| Files | Cloudflare R2 + presigned URLs | cheapest S3-compatible |
|
||||
| Payments | Stripe Checkout + webhook → FastAPI | ~1 day, no SaaS boilerplate needed |
|
||||
| Auth | Auth.js + Postgres adapter | one system |
|
||||
| Email | Resend | best Next.js integration, free tier |
|
||||
| Quality | Great Expectations / checks **inside workers** | scores shipped with each order |
|
||||
|
||||
## Stack decisions — OUT (do not add)
|
||||
|
||||
| Rejected | Reason |
|
||||
|---|---|
|
||||
| Superset | Metabase already covers BI — one tool, not two |
|
||||
| DataHub / OpenMetadata | enterprise-heavy; catalog = a Postgres table in Next.js + NocoDB. Add lineage only if an enterprise customer demands it |
|
||||
| Baserow / Teable | NocoDB chosen — not three Airtable clones |
|
||||
| Meilisearch (v1) | Postgres full-text suffices to a few thousand rows; add later |
|
||||
| Delta Sharing (v1) | unnecessary; customer pull via scoped role + exports |
|
||||
| Trigger.dev | the existing queue already schedules jobs; Resend covers mail |
|
||||
| FlareSolverr / stealth browsers / undetected-* / proxy-fingerprint rotation / fake-account creation | **prohibited** — see CLAUDE.md §1. Replace with compliant fetching |
|
||||
|
||||
## Hosting (cheapest coherent)
|
||||
|
||||
- **Vercel** (Hobby/free) — Next.js + custom domain `0pt.app`.
|
||||
- **One small VPS** (e.g. Hetzner ~€5/mo) running `docker-compose`: FastAPI, workers, NocoDB, Metabase, (Postgres or use Neon), optional Meilisearch later. Per-service managed platforms (Railway) get expensive once you have 4–5 always-on services; a single VPS + Vercel + R2 lands ≈ <€10/mo to start.
|
||||
- **Cloudflare R2** — object storage.
|
||||
- **Postgres** — on the VPS, or **Neon** free tier if you want it managed.
|
||||
|
||||
Full config, env, security, CI/CD: `docs/08-infrastructure.md`.
|
||||
@@ -0,0 +1,201 @@
|
||||
# 03 · Data model
|
||||
|
||||
Postgres is the backbone. Two halves: **data entities** (what we sell) and **platform tables** (orders, users, jobs). Every fact links to `bridge_provenance`. Migrations via Alembic (Python) or `sql/` migrations checked into the repo — pick one and be consistent; align with existing `src/workers/db.py`.
|
||||
|
||||
## Conventions
|
||||
|
||||
- `snake_case`, UTC timestamptz, soft-delete via `deleted_at` where useful.
|
||||
- Natural keys for registries (`org_nr`), surrogate `uuid` PKs for platform tables.
|
||||
- `created_at` / `updated_at` on every table.
|
||||
- All monetary amounts in **öre** (integer) to avoid float errors; currency `SEK` default.
|
||||
|
||||
## Data entities
|
||||
|
||||
```sql
|
||||
-- A company (the core registry entity)
|
||||
CREATE TABLE dim_company (
|
||||
org_nr text PRIMARY KEY, -- national reg. number (per market format)
|
||||
market text NOT NULL, -- ISO country code: SE/NO/DK/FI/...
|
||||
lei text, -- GLEIF cross-border key
|
||||
name text NOT NULL,
|
||||
status text, -- active/dissolved/...
|
||||
nace text, -- NACE/SNI code
|
||||
registered_at date,
|
||||
address jsonb,
|
||||
website text,
|
||||
updated_at timestamptz NOT NULL DEFAULT now(),
|
||||
created_at timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE TABLE fact_financials (
|
||||
org_nr text REFERENCES dim_company(org_nr),
|
||||
fiscal_year int NOT NULL,
|
||||
revenue bigint, -- öre
|
||||
result bigint,
|
||||
employees int,
|
||||
source_id text REFERENCES dim_source(source_id),
|
||||
PRIMARY KEY (org_nr, fiscal_year)
|
||||
);
|
||||
|
||||
CREATE TABLE dim_contact (
|
||||
contact_id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
org_nr text REFERENCES dim_company(org_nr),
|
||||
name text,
|
||||
role text,
|
||||
email text,
|
||||
phone text,
|
||||
linkedin text,
|
||||
-- PII: subject to GDPR; lawful basis recorded per delivery
|
||||
updated_at timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE TABLE dim_ubo ( -- ultimate beneficial owner
|
||||
ubo_id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
org_nr text REFERENCES dim_company(org_nr),
|
||||
owner text,
|
||||
share numeric(5,2)
|
||||
);
|
||||
|
||||
CREATE TABLE dim_vehicle ( -- VehIQ synergy
|
||||
vin text PRIMARY KEY,
|
||||
market text NOT NULL,
|
||||
reg_plate text,
|
||||
make text, model text, year int,
|
||||
owner_org text REFERENCES dim_company(org_nr),
|
||||
specs jsonb,
|
||||
updated_at timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
```
|
||||
|
||||
## Provenance model (the signature — every fact links here)
|
||||
|
||||
```sql
|
||||
CREATE TABLE dim_source (
|
||||
source_id text PRIMARY KEY, -- 'gleif','brreg','companies_house',...
|
||||
name text NOT NULL,
|
||||
market text, -- or 'global'
|
||||
access text, -- 'api'|'bulk'|'authorized_web'|'licensed'
|
||||
licence text, -- short licence note
|
||||
cadence text, -- 'daily'|'weekly'|'annual'|...
|
||||
url text
|
||||
);
|
||||
|
||||
-- One row per (entity field) we deliver, linking value → where it came from + confidence
|
||||
CREATE TABLE bridge_provenance (
|
||||
field_id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
entity_type text NOT NULL, -- 'company'|'contact'|'vehicle'|'financials'
|
||||
entity_key text NOT NULL, -- org_nr / vin / contact_id
|
||||
field_name text NOT NULL, -- 'revenue','email',...
|
||||
source_id text REFERENCES dim_source(source_id),
|
||||
freshness date, -- when this value was confirmed
|
||||
confidence numeric(4,3) NOT NULL, -- 0.000–1.000
|
||||
level smallint NOT NULL, -- 0 unavailable/licensed, 1 single, 2 partial, 3 corroborated
|
||||
agreed_sources int DEFAULT 1, -- # independent sources that agree
|
||||
UNIQUE (entity_type, entity_key, field_name)
|
||||
);
|
||||
CREATE INDEX ON bridge_provenance (entity_type, entity_key);
|
||||
```
|
||||
|
||||
> Rule: **no value ships without a `bridge_provenance` row.** Level 0 = delivered as "not available / licensed", never a guessed value.
|
||||
|
||||
## Platform tables
|
||||
|
||||
```sql
|
||||
CREATE TABLE orgs (
|
||||
org_id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
name text NOT NULL, created_at timestamptz DEFAULT now()
|
||||
);
|
||||
CREATE TABLE users (
|
||||
user_id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
org_id uuid REFERENCES orgs(org_id),
|
||||
email text UNIQUE NOT NULL,
|
||||
role text NOT NULL DEFAULT 'member', -- 'owner'|'admin'|'member'|'siax_admin'
|
||||
created_at timestamptz DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE TABLE orders (
|
||||
order_id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
org_id uuid REFERENCES orgs(org_id),
|
||||
flow text NOT NULL, -- 'catalog'|'custom'|'subscription'
|
||||
status text NOT NULL DEFAULT 'draft', -- see lifecycle in docs/01
|
||||
config jsonb NOT NULL, -- configurator state (markets, fields, ...)
|
||||
total_ore bigint, deposit_ore bigint, balance_ore bigint,
|
||||
est_days int, est_confidence numeric(4,3),
|
||||
created_at timestamptz DEFAULT now(), updated_at timestamptz DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE TABLE order_items ( -- resolved line items / dataset refs
|
||||
item_id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
order_id uuid REFERENCES orders(order_id),
|
||||
kind text, ref text, qty int, meta jsonb
|
||||
);
|
||||
|
||||
CREATE TABLE jobs ( -- one production run per order
|
||||
job_id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
order_id uuid REFERENCES orders(order_id),
|
||||
stage text, -- sourcing|extraction|structuring|enrichment|qa
|
||||
pct numeric(5,2) DEFAULT 0,
|
||||
state text DEFAULT 'queued', -- queued|running|done|failed
|
||||
metrics jsonb, -- rows, confidence, per-market progress
|
||||
started_at timestamptz, finished_at timestamptz
|
||||
);
|
||||
|
||||
CREATE TABLE deliveries (
|
||||
delivery_id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
order_id uuid REFERENCES orders(order_id),
|
||||
formats text[], destinations text[],
|
||||
record_count int, avg_confidence numeric(4,3),
|
||||
quality_report jsonb, provenance_export_url text,
|
||||
files jsonb, -- [{format,url,bytes}]
|
||||
created_at timestamptz DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE TABLE subscriptions (
|
||||
sub_id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
order_id uuid REFERENCES orders(order_id),
|
||||
cadence text, -- weekly|daily
|
||||
webhook_url text, status text DEFAULT 'active',
|
||||
next_run_at timestamptz
|
||||
);
|
||||
|
||||
CREATE TABLE files ( -- customer uploads
|
||||
file_id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
org_id uuid REFERENCES orgs(org_id),
|
||||
order_id uuid REFERENCES orders(order_id),
|
||||
name text, mime text, bytes bigint, r2_key text,
|
||||
created_at timestamptz DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE TABLE db_grants ( -- customer read-only DB access (encrypted)
|
||||
grant_id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
org_id uuid REFERENCES orgs(org_id),
|
||||
order_id uuid REFERENCES orders(order_id),
|
||||
conn_encrypted bytea NOT NULL, -- encrypted connection string
|
||||
scope text, expires_at timestamptz, revoked_at timestamptz,
|
||||
created_at timestamptz DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE TABLE audit_log (
|
||||
id bigserial PRIMARY KEY,
|
||||
actor text, action text, target text, meta jsonb,
|
||||
at timestamptz DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE TABLE payments (
|
||||
payment_id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
order_id uuid REFERENCES orders(order_id),
|
||||
kind text, -- 'deposit'|'balance'
|
||||
amount_ore bigint, stripe_ref text,
|
||||
status text, -- 'pending'|'captured'|'released'|'failed'
|
||||
created_at timestamptz DEFAULT now()
|
||||
);
|
||||
```
|
||||
|
||||
## pgvector
|
||||
|
||||
`CREATE EXTENSION vector;` Use for entity matching/dedup and for the AI-assist (semantic match of NL request → catalog/config). Store embeddings in a `embeddings` table keyed by `(entity_type, entity_key)` or alongside catalog rows.
|
||||
|
||||
## Row-level security & catalog
|
||||
|
||||
- Catalog products: a `catalog` table (or view) over curated datasets with sample rows, counts, coverage, price.
|
||||
- Per-customer data isolation enforced by `org_id` filtering in the API and (where exposed via NocoDB/Metabase) by scoped views/connection params, plus Postgres RLS where practical.
|
||||
@@ -0,0 +1,97 @@
|
||||
# 04 · API contract (FastAPI)
|
||||
|
||||
Canonical REST contract. Reconcile the existing 25+ endpoints in `src/orchestrator/main.py` toward this. JSON in/out, `snake_case`. All amounts in **öre**. Auth: Next.js → FastAPI over a shared internal token (`X-Internal-Token`); customer identity passed as `org_id`/`user_id` resolved from the Auth.js session server-side.
|
||||
|
||||
## Conventions
|
||||
- Errors: `{ "error": { "code", "message" } }`, proper HTTP status. Interface-voice messages (Swedish) are produced in the frontend, not here.
|
||||
- Idempotency: accept `Idempotency-Key` on POSTs that create orders/payments.
|
||||
- Pagination: `?limit=&cursor=`.
|
||||
|
||||
## Catalog
|
||||
```
|
||||
GET /catalog -> [{id,title,markets,entity_type,count,updated,price_ore,confidence}]
|
||||
GET /catalog/{id} -> full dataset (overview, coverage, schema, quality summary)
|
||||
GET /catalog/{id}/sample?n=10 -> sample rows WITH per-field provenance
|
||||
```
|
||||
|
||||
## Estimate (configurator → price/time/confidence)
|
||||
```
|
||||
POST /estimates
|
||||
body: { flow, markets[], entity_type, fields[], enrichment[], volume, frequency, confidence_target, formats[] }
|
||||
-> { total_ore, deposit_ore, balance_ore, est_days, est_confidence, breakdown[] }
|
||||
```
|
||||
|
||||
## AI-assist (NL → config)
|
||||
```
|
||||
POST /ai/parse
|
||||
body: { text } -- "5000 svenska SaaS-bolag med e-post och omsättning"
|
||||
-> { config: <estimate body>, rationale } -- prefilled configurator; uses pgvector match to catalog/fields
|
||||
```
|
||||
|
||||
## Orders (lifecycle)
|
||||
```
|
||||
POST /orders body: { flow, config } -> { order_id, status:'draft'|'quoted' }
|
||||
GET /orders -> [orders for org]
|
||||
GET /orders/{id} -> order + current job + delivery (if any)
|
||||
POST /orders/{id}/scope (custom flow) body:{ notes } -> status:'quoted' (manual quote step)
|
||||
POST /orders/{id}/accept customer approves delivery -> triggers balance release; status:'accepted'
|
||||
POST /orders/{id}/revise body:{ notes } -> status:'revision'; escrow untouched
|
||||
```
|
||||
|
||||
## Payments (50/50 escrow)
|
||||
```
|
||||
POST /payments/{order_id}/deposit -> { checkout_url } (Stripe Checkout for 50%; on success → IN_PRODUCTION)
|
||||
POST /payments/{order_id}/release -> captures/charges the held balance (called on accept)
|
||||
POST /webhooks/stripe (Stripe → here) verify signature; update payments + order status
|
||||
```
|
||||
> Implement balance as either a manual capture on a held PaymentIntent, or a second Checkout/invoice keyed to `order_id`. Never charge balance before `accept`.
|
||||
|
||||
## Jobs / live status
|
||||
```
|
||||
GET /orders/{id}/status -> { stage, pct, per_market[], confidence, eta, rows }
|
||||
GET /orders/{id}/events -> Server-Sent Events stream (preferred for the production tracker)
|
||||
```
|
||||
|
||||
## Uploads (R2 presigned)
|
||||
```
|
||||
POST /uploads body:{ order_id, name, mime, bytes } -> { file_id, put_url } (client PUTs to R2)
|
||||
GET /orders/{id}/files -> [files]
|
||||
```
|
||||
|
||||
## Customer DB access (read-only grant)
|
||||
```
|
||||
POST /connections/db-grant
|
||||
body: { order_id, conn_string, scope, expires_at } -- stored ENCRYPTED; validated read-only
|
||||
-> { grant_id, status:'pending'|'verified' }
|
||||
DELETE /connections/db-grant/{id} -> revoke
|
||||
```
|
||||
|
||||
## Deliveries
|
||||
```
|
||||
GET /orders/{id}/delivery -> { formats, destinations, record_count, avg_confidence, quality_report, files[], provenance_export_url }
|
||||
POST /orders/{id}/delivery/destinations body:{ destinations[] } -> updates targets (CSV/Parquet/JSON/API/NocoDB/Metabase/FiveCRM/Sheets/webhook)
|
||||
```
|
||||
|
||||
## Sources (the transparency moat)
|
||||
```
|
||||
GET /sources -> [{source_id,name,market,access,licence,cadence,confidence}]
|
||||
GET /sources/{market} -> sources + freshness for a market
|
||||
```
|
||||
|
||||
## Subscriptions / feeds
|
||||
```
|
||||
POST /subscriptions body:{ order_id, cadence, webhook_url } -> { sub_id }
|
||||
GET /subscriptions -> [subs]
|
||||
POST /subscriptions/{id}/pause | /resume
|
||||
```
|
||||
|
||||
## Webhooks out (to customers)
|
||||
- On delivery / feed update: POST to `subscriptions.webhook_url` with `{ order_id, delta_url, summary }`.
|
||||
|
||||
## Health
|
||||
```
|
||||
GET /health -> { ok, db, workers, queue }
|
||||
```
|
||||
|
||||
## Existing-code reconciliation
|
||||
Map current endpoints (orders, estimates, ai-assistent, catalog, webhooks, health) onto the names above; rename for consistency, keep behavior. Add: `/orders/{id}/events` (SSE), `/payments/*` escrow split, `/connections/db-grant`, `/sources`, `/subscriptions`. Remove any endpoint that drives a bypass/stealth fetch path.
|
||||
@@ -0,0 +1,87 @@
|
||||
# 05 · Design system (mono)
|
||||
|
||||
Self-contained spec. Component-level v0 prompts live in `docs/design/0pt-v0-build-brief.md`; the full page/component inventory + visual rules in `docs/design/0pt-design-coverage-and-visuals.md`. This file is the source of truth for tokens + motion; build them once and reuse.
|
||||
|
||||
Derived from the **mono / EVASION** template (itself v0-generated). Light, editorial, monochrome, sharp, with a single red accent and animated grain + scroll-reveal motion. **No photography** — imagery is rendered data.
|
||||
|
||||
## Tokens
|
||||
|
||||
```
|
||||
THEME (light, default):
|
||||
--background #ffffff --foreground #0a0a0a --muted-foreground #737373
|
||||
--border #e5e5e5 --card #ffffff --input #f5f5f5
|
||||
THEME (dark sections — footer, one editorial block):
|
||||
--background #0a0a0a --foreground #fafafa --muted-foreground #a3a3a3 --border #262626
|
||||
ACCENT (single, sparing): red --accent #e40014 (one CTA / one number / one marker per screen)
|
||||
PROVENANCE squares (ONLY non-mono color besides the red CTA):
|
||||
level 0 #737373 (unavailable/licensed) · 1 #143aff (single source) · 2 #fcbb00 (partial) · 3 #00d294 (corroborated)
|
||||
RADIUS: 0 everywhere (max 2px on tiny chips). No drop shadows; structure via 1px hairlines.
|
||||
TYPE: Inter (everything; headlines weight 500–600, letter-spacing -0.02em, large) · JetBrains Mono (eyebrows, labels, numbers, codes, table headers — UPPERCASE, letter-spacing 0.12em, ~11px, #737373)
|
||||
GRID: 8px base; graph-paper hairlines; generous whitespace; left-aligned headlines.
|
||||
SECTION SPINE: real ordered indices 00–NN as mono eyebrows.
|
||||
```
|
||||
|
||||
## globals.css (add verbatim — guarantees identical motion + grain)
|
||||
|
||||
```css
|
||||
@keyframes reveal-up{0%{opacity:0;transform:translateY(60px)}to{opacity:1;transform:translateY(0)}}
|
||||
@keyframes reveal-left{0%{opacity:0;transform:translateX(-60px)}to{opacity:1;transform:translateX(0)}}
|
||||
@keyframes reveal-right{0%{opacity:0;transform:translateX(60px)}to{opacity:1;transform:translateX(0)}}
|
||||
@keyframes scale-in{0%{opacity:0;transform:scale(.9)}to{opacity:1;transform:scale(1)}}
|
||||
@keyframes fadeIn{0%{opacity:0;transform:scale(1.02)}to{opacity:1;transform:scale(1)}}
|
||||
@keyframes slideUp{0%{opacity:0;transform:translateY(100%)}to{opacity:1;transform:translateY(0)}}
|
||||
@keyframes float{0%,100%{transform:translateY(0)}50%{transform:translateY(-20px)}}
|
||||
@keyframes grain{0%,100%{transform:translate(0,0)}10%{transform:translate(-5%,-10%)}20%{transform:translate(-15%,5%)}30%{transform:translate(7%,-25%)}40%{transform:translate(-5%,25%)}50%{transform:translate(-15%,10%)}60%{transform:translate(15%,0)}70%{transform:translateY(15%)}80%{transform:translate(3%,35%)}90%{transform:translate(-10%,10%)}}
|
||||
.reveal{opacity:0}
|
||||
.reveal.in-view{animation:.8s ease-out forwards reveal-up}
|
||||
.reveal-left.in-view{animation:.8s ease-out forwards reveal-left}
|
||||
.reveal-right.in-view{animation:.8s ease-out forwards reveal-right}
|
||||
.reveal-scale.in-view{animation:.6s ease-out forwards scale-in}
|
||||
.reveal-fade.in-view{animation:.7s ease-out forwards fadeIn}
|
||||
.delay-1{animation-delay:.1s}.delay-2{animation-delay:.2s}.delay-3{animation-delay:.3s}.delay-4{animation-delay:.4s}.delay-5{animation-delay:.5s}.delay-6{animation-delay:.6s}
|
||||
.animate-float{animation:6s ease-in-out infinite float}
|
||||
.grain-overlay::before{content:"";position:absolute;inset:-100%;opacity:.03;pointer-events:none;background-image:url("data:image/svg+xml,%3Csvg viewBox='0 0 200 200' xmlns='http://www.w3.org/2000/svg'%3E%3Cfilter id='n'%3E%3CfeTurbulence type='fractalNoise' baseFrequency='0.9' numOctaves='4' stitchTiles='stitch'/%3E%3C/filter%3E%3Crect width='100%25' height='100%25' filter='url(%23n)'/%3E%3C/svg%3E");animation:8s steps(10,end) infinite grain}
|
||||
@media(prefers-reduced-motion:reduce){.reveal,.reveal-left,.reveal-right,.reveal-scale,.reveal-fade{opacity:1;animation:none}.animate-float,.grain-overlay::before{animation:none}}
|
||||
```
|
||||
|
||||
## useReveal hook
|
||||
|
||||
```tsx
|
||||
"use client";
|
||||
import { useEffect, useRef } from "react";
|
||||
export function useReveal<T extends HTMLElement = HTMLElement>() {
|
||||
const ref = useRef<T>(null);
|
||||
useEffect(() => {
|
||||
const el = ref.current; if (!el) return;
|
||||
const io = new IntersectionObserver((es) => es.forEach((e) => {
|
||||
if (e.isIntersecting) {
|
||||
el.querySelectorAll(".reveal,.reveal-left,.reveal-right,.reveal-scale,.reveal-fade")
|
||||
.forEach((n) => n.classList.add("in-view"));
|
||||
io.unobserve(el);
|
||||
}
|
||||
}), { threshold: 0.15, rootMargin: "0px 0px -10% 0px" });
|
||||
io.observe(el); return () => io.disconnect();
|
||||
}, []);
|
||||
return ref;
|
||||
}
|
||||
```
|
||||
Every section wraps in a tag using `useReveal()`; children get `reveal[/-left/-right/-scale/-fade]` + `delay-N`.
|
||||
|
||||
## Buttons & labels
|
||||
- Primary: solid `#0a0a0a` bg, white text, square, mono uppercase 12px / spacing 0.1em, padding 14px 26px; hover → bg `#e40014`.
|
||||
- Secondary: transparent, 1px `#0a0a0a` border; hover → bg `#0a0a0a`, white text.
|
||||
- Eyebrow: mono uppercase 11px, spacing 0.14em, `#737373`, with a 2-digit index ("01 — KATALOG").
|
||||
- Headlines: sentence case, never uppercase.
|
||||
|
||||
## Visual system — no photos (`DataArtifact`)
|
||||
Build one generative primitive; every "image" surface uses it, animating with the effect it replaces:
|
||||
- `table` (rows cascade in, provenance square per fact cell) — hero/cards.
|
||||
- `map` (country-code cells, fill bars animate width on view) — coverage.
|
||||
- `schema` (ER mini-diagram, key boxes red border) — schema previews.
|
||||
- `chart` (black bars/lines, exactly ONE red series, grow on view) — dashboards.
|
||||
- `grid` (hairline graph-paper + faint oversized numeral + one floating red square) — pure texture.
|
||||
|
||||
Mapping of template image elements → replacement + effect kept is in the coverage doc. Rules: code-only (CSS/SVG/canvas), color discipline inside visuals (mono + provenance squares + one red series), grain over any "image" surface, `prefers-reduced-motion` settles to static. Optional: add **Lenis** smooth scroll on top — preserves all effects.
|
||||
|
||||
## The signature (carry everywhere)
|
||||
Provenance + confidence is the identity: the provenance square on every fact, one red, mono numerals as texture, section spine, identical motion. A screen that lacks these is off-brand.
|
||||
@@ -0,0 +1,70 @@
|
||||
# 06 · Data engine
|
||||
|
||||
The engine turns sources into delivered, provenance-tagged data. It is the legal motor — the whole product positioning depends on it. Build on existing `src/workers/` (`adaptive_router`, `browser_worker`, `ocr_worker`, `db`) but enforce the sourcing boundary below.
|
||||
|
||||
## Sourcing chain (strict priority — this is the law)
|
||||
|
||||
For each field × market, acquire in this order; stop at the first that works:
|
||||
|
||||
1. **Official API / bulk** — registries with programmatic/bulk access (highest confidence).
|
||||
2. **Authorized public web** — clearly public pages, **respecting `robots.txt` and rate limits**, no auth walls.
|
||||
3. **Licensed partner data** — contractually obtained.
|
||||
4. **Otherwise → "not available / licensed only"** — delivered honestly as level 0. **Never guessed.**
|
||||
|
||||
### Prohibited (do not build, keep, or extend)
|
||||
- WAF/bot-detection bypass — **including FlareSolverr**, "stealth"/patched browsers, undetected-* used to defeat detection.
|
||||
- Proxy/fingerprint rotation to evade bans; scraping behind login walls; automated fake-account creation (Apollo etc.).
|
||||
- **Reconciliation:** if `browser_worker.py` has an `HTTP → FlareSolverr → Playwright` chain, **drop the FlareSolverr stage** and keep only: polite HTTP fetch (identifies itself, obeys robots + rate limits) → optional Playwright for genuinely public JS-rendered pages (same politeness). The `adaptive_router` then chooses among **compliant** methods only (API > bulk > authorized fetch), not bypass tiers.
|
||||
|
||||
## Pipeline stages (per job)
|
||||
|
||||
```
|
||||
sourcing → extraction/OCR → structuring → enrichment/fusion → QA
|
||||
```
|
||||
|
||||
1. **Sourcing** — `adaptive_router` picks the compliant method per source; "source-racing" = query several *authorized* sources and take the fastest/freshest. Writes raw payloads + which source answered.
|
||||
2. **Extraction / OCR** — API JSON parsed directly; documents (PDF/HTML/images) via **Unstructured** + OCR cascade (PaddleOCR → Tesseract → LLM fallback) in `ocr_worker`. Output: candidate fields.
|
||||
3. **Structuring** — NER + entity resolution (spaCy / transformers, sv/no/da/fi/de/fr), normalize to schema, link entities across markets via **LEI** (GLEIF golden key). Dedup with **RapidFuzz**.
|
||||
4. **Enrichment / fusion** — combine fields from multiple sources; compute **multi-source agreement** → confidence + level; append optional enrichment (NACE/SNI, sanctions/PEP flag from official lists, website/tech).
|
||||
5. **QA** — quality checks (**Great Expectations** or equivalent, run inside the worker): validity (format/range), completeness, freshness, uniqueness (post-dedup), corroboration. Drop rows below the order's `confidence_target` (deliver fewer, never fabricate). Emit the quality report stored on `deliveries`.
|
||||
|
||||
Each stage updates `jobs` (`stage`, `pct`, `metrics` incl. per-market progress, running counts, climbing confidence) so the production tracker / SSE can render it live.
|
||||
|
||||
## Confidence & provenance scoring
|
||||
|
||||
- Per field, write a `bridge_provenance` row: `source_id`, `freshness`, `confidence` (0–1), `level` (0–3), `agreed_sources`.
|
||||
- Level: 0 = unavailable/licensed; 1 = single source; 2 = partial corroboration; 3 = ≥2 independent sources agree.
|
||||
- `est_confidence` on the order = aggregate target; delivered `avg_confidence` = actual.
|
||||
|
||||
## `sources.yaml` (machine-readable registry)
|
||||
|
||||
Single registry that drives the router, the `/sources` API, and the public sourcing-map page. Schema per entry:
|
||||
|
||||
```yaml
|
||||
- source_id: gleif
|
||||
name: GLEIF (LEI)
|
||||
market: global
|
||||
access: api # api | bulk | authorized_web | licensed
|
||||
cadence: daily
|
||||
licence: "open (CC0) — VERIFY current terms"
|
||||
entities: [company]
|
||||
fields: [lei, legal_name, address]
|
||||
base_url: https://api.gleif.org
|
||||
robots_ok: true # for authorized_web only; must be honored
|
||||
rate_limit: "per provider terms"
|
||||
notes: "cross-border key"
|
||||
```
|
||||
|
||||
### Starter set (VERIFY current access/licence per source before relying — terms change)
|
||||
- **GLEIF / LEI** — typically open; cross-border key. *(verify)*
|
||||
- **Brønnøysund (Brreg, NO)** — typically open API. *(verify)*
|
||||
- **Companies House (UK)** — free API (key required). *(verify)*
|
||||
- **CVR / Virk (DK)** — API access with terms. *(verify)*
|
||||
- **PRH (FI)** — varies. *(verify)*
|
||||
- **Bolagsverket (SE)** — more restricted for bulk; plan licensed/partner route. *(verify)*
|
||||
- Vehicle: **RDW (NL)** open-ish, **Traficom (FI)**, **Transportstyrelsen (SE)**, **DVLA (UK)** — *(verify each; vehicle/PII rules differ)*
|
||||
|
||||
> Do not hardcode access terms as fact. When building each connector, confirm the current API/licence/robots status and record it in `sources.yaml`. If unclear or disallowed for bulk, route to licensed/partner or mark fields level 0.
|
||||
|
||||
## Libraries used here
|
||||
Crawlee or Scrapy (orchestration of **authorized** fetches) · Playwright (public JS pages, politely) · Unstructured (document parsing) · PaddleOCR/Tesseract (OCR) · spaCy / HF transformers (NER, multilingual) · Polars (fast tabular) · RapidFuzz (dedup/matching) · Great Expectations (quality) · dbt (optional transform layer). Versions/install: `docs/07-dependencies.md`.
|
||||
@@ -0,0 +1,65 @@
|
||||
# 07 · Dependencies & repos to import
|
||||
|
||||
Curated. Each entry: what it is, where it slots in, how to install. **Do not import on reputation/star-count** — several items in earlier brainstorming were unverified or hallucinated (see Exclusions). Verify each repo exists and is maintained before adopting.
|
||||
|
||||
## Frontend (Next.js on Vercel)
|
||||
| Package | Purpose | Install |
|
||||
|---|---|---|
|
||||
| next, react, typescript | App Router base | `npx create-next-app@latest --ts` |
|
||||
| tailwindcss | styling (mono tokens) | per Next setup |
|
||||
| shadcn/ui | base components | `npx shadcn@latest init` |
|
||||
| @tanstack/react-table **or** ag-grid-react | data tables / catalog grid | `npm i @tanstack/react-table` |
|
||||
| @uppy/core @uppy/react @uppy/dashboard @uppy/aws-s3 | file upload → R2 (presigned) | `npm i @uppy/core @uppy/react @uppy/dashboard @uppy/aws-s3` |
|
||||
| next-auth (Auth.js) + @auth/pg-adapter | single auth, Postgres-backed | `npm i next-auth @auth/pg-adapter` |
|
||||
| stripe / @stripe/stripe-js | checkout + portal | `npm i stripe @stripe/stripe-js` |
|
||||
| resend / react-email | transactional email | `npm i resend @react-email/components` |
|
||||
| lenis *(optional)* | smooth scroll (keeps all effects) | `npm i lenis` |
|
||||
| recharts *(only if needed)* | charts inside DataArtifact/dashboards | `npm i recharts` |
|
||||
|
||||
> Catalog/search in v1 = Postgres full-text. Add Meilisearch later only past a few thousand rows.
|
||||
|
||||
## Backend / workers (Python, FastAPI — existing)
|
||||
| Package | Purpose | Install |
|
||||
|---|---|---|
|
||||
| fastapi, uvicorn, pydantic | API (existing) | `pip install fastapi uvicorn pydantic` |
|
||||
| sqlalchemy / asyncpg | Postgres access (align `db.py`) | `pip install sqlalchemy asyncpg` |
|
||||
| alembic | migrations | `pip install alembic` |
|
||||
| celery + redis **or** existing adaptive_router/queue | job queue/concurrency | `pip install celery redis` |
|
||||
| httpx | polite HTTP fetch (identifies itself, obeys rate limits) | `pip install httpx` |
|
||||
| scrapy **or** crawlee | orchestrate **authorized** fetches | `pip install scrapy` |
|
||||
| playwright | public JS-rendered pages, politely | `pip install playwright && playwright install chromium` |
|
||||
| unstructured | parse PDF/HTML/DOCX/images → elements | `pip install "unstructured[all-docs]"` |
|
||||
| paddleocr / pytesseract | OCR cascade | `pip install paddleocr pytesseract` (+ tesseract bin) |
|
||||
| spacy (+ models) / transformers | NER, entity resolution (multilingual) | `pip install spacy transformers` |
|
||||
| polars | fast tabular processing | `pip install polars` |
|
||||
| rapidfuzz | dedup / fuzzy matching | `pip install rapidfuzz` |
|
||||
| great-expectations | data quality checks in pipeline | `pip install great-expectations` |
|
||||
| dbt-postgres *(optional)* | transform layer | `pip install dbt-postgres` |
|
||||
| pgvector (Postgres ext) | embeddings/matching | `CREATE EXTENSION vector;` |
|
||||
| boto3 *(or aioboto3)* | R2 (S3-compatible) | `pip install boto3` |
|
||||
| stripe (python) | webhook handling | `pip install stripe` |
|
||||
|
||||
## Self-hosted services (docker-compose on the VPS)
|
||||
| Service | Purpose | Image |
|
||||
|---|---|---|
|
||||
| **NocoDB** | Airtable-style views over existing Postgres tables; embedded "Min data" | `nocodb/nocodb` |
|
||||
| **Metabase** | customer-facing dashboards; embedded | `metabase/metabase` |
|
||||
| **Postgres** *(or Neon managed)* | backbone (+ pgvector) | `pgvector/pgvector:pg16` |
|
||||
| **Redis** *(if Celery)* | queue | `redis` |
|
||||
| **Meilisearch** *(later)* | catalog search at scale | `getmeili/meilisearch` |
|
||||
|
||||
## External managed (free/cheap tiers)
|
||||
- **Vercel** — Next.js hosting.
|
||||
- **Cloudflare R2** — object storage (presigned URLs).
|
||||
- **Stripe** — payments (test mode first).
|
||||
- **Resend** — email.
|
||||
- **Neon** *(optional)* — managed Postgres.
|
||||
|
||||
## EXCLUSIONS — do not import
|
||||
|
||||
- **Bypass/stealth/evasion**: FlareSolverr, "CloakBrowser", undetected-chromedriver/undetected-playwright (as bypass), proxy/fingerprint-rotation engines, any "WAF bypass" repo. Prohibited by CLAUDE.md §1 — they destroy the honest-accuracy positioning and create legal exposure.
|
||||
- **Fake-account automation** on Apollo/Cognism/etc.
|
||||
- **Unverified / likely-hallucinated** from earlier brainstorming: "CloakBrowser/CloakHQ", "Rota proxy engine", "OpenClaw". Do not adopt unless you confirm a real, maintained project — and even then, not if it falls in the bypass category.
|
||||
- **Overlapping tools** already decided against (Superset, DataHub/OpenMetadata, Baserow/Teable, Trigger.dev, Delta Sharing in v1) — see `docs/02-architecture.md`.
|
||||
|
||||
> Note on real-but-overhyped tools (Firecrawl, browser-use, etc.): they exist, but ignore inflated star-counts and only adopt if they fit a *compliant* role and don't duplicate the stack.
|
||||
@@ -0,0 +1,118 @@
|
||||
# 08 · Infrastructure
|
||||
|
||||
## Repo structure (monorepo)
|
||||
|
||||
```
|
||||
0pt/
|
||||
├─ CLAUDE.md
|
||||
├─ docs/ # this documentation set
|
||||
├─ apps/
|
||||
│ └─ web/ # Next.js (Vercel): marketing, catalog, portal, admin
|
||||
├─ services/
|
||||
│ └─ api/ # FastAPI orchestrator + workers (existing src/* migrates here)
|
||||
│ ├─ orchestrator/ # main.py + routers (docs/04-api.md)
|
||||
│ ├─ workers/ # adaptive_router, fetch_worker, ocr_worker, pipeline, db
|
||||
│ └─ sources/ # sources.yaml + connectors (docs/06)
|
||||
├─ packages/
|
||||
│ └─ shared/ # shared TS types (mirror API contract), design tokens
|
||||
├─ sql/ (or services/api/alembic/) # migrations (docs/03)
|
||||
├─ infra/
|
||||
│ ├─ docker-compose.yml # VPS always-on services
|
||||
│ └─ Caddyfile # reverse proxy + TLS
|
||||
└─ .github/workflows/ # CI/CD
|
||||
```
|
||||
|
||||
## Hosting topology
|
||||
|
||||
- **Vercel** — `apps/web` → `0pt.app`. Free Hobby until paying customers.
|
||||
- **One VPS** (Hetzner ~€5/mo) — `docker-compose`: FastAPI, workers, NocoDB, Metabase, Postgres (or Neon), Redis (if Celery), Caddy. Caddy terminates TLS and proxies `api.0pt.app`, `data.0pt.app` (NocoDB), `dash.0pt.app` (Metabase).
|
||||
- **Cloudflare R2** — files.
|
||||
- **Postgres** — on VPS (`pgvector/pgvector:pg16`) or **Neon** managed.
|
||||
|
||||
## docker-compose (infra/docker-compose.yml — sketch)
|
||||
|
||||
```yaml
|
||||
services:
|
||||
postgres:
|
||||
image: pgvector/pgvector:pg16
|
||||
environment: { POSTGRES_USER: opt, POSTGRES_PASSWORD: ${PG_PASSWORD}, POSTGRES_DB: opt }
|
||||
volumes: ["pgdata:/var/lib/postgresql/data"]
|
||||
redis:
|
||||
image: redis:7
|
||||
api:
|
||||
build: ../services/api
|
||||
command: uvicorn orchestrator.main:app --host 0.0.0.0 --port 8000
|
||||
environment: [DATABASE_URL, R2_*, STRIPE_*, RESEND_API_KEY, INTERNAL_TOKEN]
|
||||
depends_on: [postgres, redis]
|
||||
worker:
|
||||
build: ../services/api
|
||||
command: python -m workers.run # or celery -A workers worker
|
||||
environment: [DATABASE_URL, R2_*]
|
||||
depends_on: [postgres, redis]
|
||||
nocodb:
|
||||
image: nocodb/nocodb
|
||||
environment: { NC_DB: "pg://postgres:5432?u=opt&p=${PG_PASSWORD}&d=opt", NC_PUBLIC_URL: "https://data.0pt.app" }
|
||||
depends_on: [postgres]
|
||||
metabase:
|
||||
image: metabase/metabase
|
||||
environment: { MB_DB_TYPE: postgres, MB_DB_HOST: postgres, MB_DB_USER: opt, MB_DB_PASS: ${PG_PASSWORD}, MB_DB_DBNAME: opt }
|
||||
depends_on: [postgres]
|
||||
caddy:
|
||||
image: caddy:2
|
||||
ports: ["80:80","443:443"]
|
||||
volumes: ["./Caddyfile:/etc/caddy/Caddyfile", "caddydata:/data"]
|
||||
volumes: { pgdata: {}, caddydata: {} }
|
||||
```
|
||||
|
||||
## Environment variables (full inventory)
|
||||
|
||||
```
|
||||
# Postgres
|
||||
DATABASE_URL=postgres://opt:***@host:5432/opt
|
||||
PG_PASSWORD=***
|
||||
# Internal auth (Next ↔ FastAPI)
|
||||
INTERNAL_TOKEN=*** # sent as X-Internal-Token
|
||||
# Auth.js
|
||||
AUTH_SECRET=***
|
||||
AUTH_URL=https://0pt.app
|
||||
# Stripe
|
||||
STRIPE_SECRET_KEY=sk_***
|
||||
STRIPE_WEBHOOK_SECRET=whsec_***
|
||||
NEXT_PUBLIC_STRIPE_PUBLISHABLE_KEY=pk_***
|
||||
# Cloudflare R2 (S3-compatible)
|
||||
R2_ACCOUNT_ID=*** R2_ACCESS_KEY_ID=*** R2_SECRET_ACCESS_KEY=*** R2_BUCKET=opt-files R2_ENDPOINT=https://<acct>.r2.cloudflarestorage.com
|
||||
# Email
|
||||
RESEND_API_KEY=re_***
|
||||
# Embeds (signed URLs)
|
||||
METABASE_SITE_URL=https://dash.0pt.app METABASE_SECRET_KEY=***
|
||||
NOCODB_BASE_URL=https://data.0pt.app NOCODB_API_TOKEN=***
|
||||
# Encryption for db_grants
|
||||
GRANT_ENC_KEY=*** # 32-byte key, AES-GCM
|
||||
# App
|
||||
NEXT_PUBLIC_API_URL=https://api.0pt.app
|
||||
```
|
||||
|
||||
Never hardcode. Vercel env for `apps/web`; VPS `.env` (git-ignored) for compose. Provide `.env.example` with empty values.
|
||||
|
||||
## Security & compliance (build in, not bolt on)
|
||||
|
||||
- **Customer DB grants**: customer creates a read-only, table-scoped, time-boxed DB role; we store the connection string **AES-GCM encrypted** (`GRANT_ENC_KEY`), validate it's read-only, and **delete after the job**. IP-allowlist; prefer a replica. Log access in `audit_log`.
|
||||
- **File delivery**: short-lived **presigned R2 URLs**; never public buckets.
|
||||
- **Isolation**: API filters by `org_id`; Postgres **RLS** where practical; NocoDB/Metabase exposed only via **signed embed URLs** scoped per customer/order (no separate login surfaced).
|
||||
- **PII (contacts/UBO)**: lawful basis recorded per delivery; honor deletion; minimize retention.
|
||||
- **DPA**: ship a Data Processing Agreement; provenance log + audit log are part of the offering.
|
||||
- **Secrets**: rotate; least privilege; no secrets in client bundles (only `NEXT_PUBLIC_*`).
|
||||
|
||||
## Embedding NocoDB / Metabase (so it feels like one app)
|
||||
- Generate a **signed embed URL** server-side (Metabase signed embedding with `METABASE_SECRET_KEY`; NocoDB shared/locked view + token) scoped to the customer's data, and render it in an iframe inside the mono chrome. The customer never sees a NocoDB/Metabase login.
|
||||
|
||||
## CI/CD (.github/workflows)
|
||||
- **web**: lint + typecheck + build → deploy to Vercel on push to `main` (Vercel Git integration).
|
||||
- **api**: lint (ruff) + tests (pytest) → build Docker image → deploy to VPS (SSH `docker compose pull && up -d`, or a small runner).
|
||||
- Run DB migrations as a deploy step.
|
||||
|
||||
## Monitoring (cheap)
|
||||
- Vercel Analytics + Speed Insights (web).
|
||||
- **Sentry** free tier (web + api).
|
||||
- Uptime ping on `/health`.
|
||||
- Worker/job health visible in admin (reads `jobs`).
|
||||
@@ -0,0 +1,75 @@
|
||||
# 09 · Build plan
|
||||
|
||||
Phased, in dependency order. **Finish a phase to its acceptance criteria before the next.** Each phase = small, reviewable commits. Reconcile with existing code rather than rewriting wholesale.
|
||||
|
||||
## Phase 0 — Foundation & repo
|
||||
- Monorepo structure (`docs/08`). Migrate existing `src/orchestrator` + `src/workers` into `services/api`.
|
||||
- Postgres schema + migrations (`docs/03`); `pgvector` enabled. `.env.example`.
|
||||
- `docker-compose` up (postgres, redis, api, worker, nocodb, metabase, caddy). `/health` green.
|
||||
- CI skeleton (lint/typecheck/build).
|
||||
- **Accept:** `docker compose up` runs the whole stack locally; migrations apply; `/health` returns ok for db + queue.
|
||||
|
||||
## Phase 1 — Design system in code
|
||||
- Tailwind + shadcn; `globals.css` (grain + reveal keyframes) and `useReveal` verbatim (`docs/05`).
|
||||
- Build **DataArtifact** primitive (table/map/schema/chart/grid) and **ProvenanceBadge** (`docs/design` prompts 5.1, 2.11).
|
||||
- **Accept:** a sample page renders mono tokens, a DataArtifact animates on scroll, reduced-motion respected, focus rings visible.
|
||||
|
||||
## Phase 2 — Marketing + catalog (placeholder data)
|
||||
- Landing components top-to-bottom (build-brief 2.1–2.10): Header, Hero, Philosophy, Catalog teaser, Engine, Visualisation gallery, Coverage, Editorial, Proof, Footer.
|
||||
- Full **Catalog** page + **Dataset detail** + **Sourcing-map** page (coverage doc 5.4). Postgres full-text search.
|
||||
- All with loading/empty/error states + mobile.
|
||||
- **Accept:** marketing site + catalog browse work end-to-end on placeholder catalog rows; one continuous scroll reads well; DoD met per screen.
|
||||
|
||||
## Phase 3 — Auth & portal shell
|
||||
- Auth.js + Postgres adapter; sign in/up/verify/forgot. `orgs`/`users`.
|
||||
- Portal shell + Dashboard ("Mina beställningar") with empty state.
|
||||
- **Accept:** a user can sign up, land in an empty dashboard, sign out; routes protected.
|
||||
|
||||
## Phase 4 — Order lifecycle end-to-end (test mode)
|
||||
- **Configurator** (2.12) wired to `POST /estimates` (real estimate math in FastAPI).
|
||||
- **Deposit checkout** (2.13) → Stripe **test** Checkout → webhook → order `IN_PRODUCTION`.
|
||||
- **Production tracker** (2.14) reading `GET /orders/{id}/events` (SSE) — pipeline may be **simulated** at this phase.
|
||||
- **Validation & delivery** (2.15): tabs render from a sample delivery; **accept** → `POST /payments/release` (test) → `DELIVERED`; **revise** leaves escrow untouched.
|
||||
- AI-assist (`POST /ai/parse`) prefilling the configurator.
|
||||
- **Accept:** a full custom order can be configured, deposit paid (test), watched, validated, accepted, and "delivered" — state machine in `docs/01` traversed; escrow split correct.
|
||||
|
||||
## Phase 5 — Real data engine (Nordic slice)
|
||||
- Compliant connectors first: **GLEIF + Brreg + Companies House** (verify access terms; record in `sources.yaml`). Remove any FlareSolverr/bypass path (`docs/06`).
|
||||
- Pipeline stages produce real `dim_company` + `fact_financials` with `bridge_provenance`; QA threshold drops low-confidence rows.
|
||||
- Production tracker now reflects **real** job metrics.
|
||||
- **Accept:** a Nordic company dataset is produced for a real order, every field has provenance, quality report generated, level-0 fields shown honestly (not fabricated).
|
||||
|
||||
## Phase 6 — Delivery surfaces & integrations
|
||||
- Exports: CSV / Parquet / JSON via R2 presigned URLs.
|
||||
- Embedded **NocoDB** ("Min data") and **Metabase** dashboards via signed URLs in mono chrome.
|
||||
- Integrations: **FiveCRM push**, webhooks, Google Sheets.
|
||||
- File **uploads** (Uppy → R2) for matching/enrichment.
|
||||
- **Accept:** a delivered dataset is downloadable, viewable in embedded NocoDB + Metabase (no separate login), and pushable to FiveCRM; uploads work.
|
||||
|
||||
## Phase 7 — Subscriptions / feeds
|
||||
- `subscriptions`; recurring jobs (weekly/daily) delivering **deltas**; webhooks on change.
|
||||
- Freshness monitoring; Resend emails (confirmation, "din data är klar", feedback).
|
||||
- **Accept:** an order converts to a feed; a scheduled run produces a delta + fires a webhook + email.
|
||||
|
||||
## Phase 8 — Admin, hardening, compliance
|
||||
- Admin: order queue + scope/quote + escrow release; **source registry & freshness** monitoring; worker/job monitoring; customer + catalog management.
|
||||
- Customer **DB-access grant** flow (encrypted, read-only, time-boxed) + revoke.
|
||||
- Security pass: RLS, presigned-only files, signed embeds, secret rotation, `audit_log` everywhere; **DPA** published; legal pages.
|
||||
- Finalize all loading/empty/error states + mobile across the app.
|
||||
- **Accept:** admin can run the business; DB-grant flow works and self-revokes; security checklist passes; DoD met app-wide.
|
||||
|
||||
## Phase 9 — Launch
|
||||
- Domain `0pt.app` (Vercel) + subdomains (api/data/dash via Caddy). Sentry + uptime live.
|
||||
- Soft launch: Protosell internally + a handful of existing dealer relationships.
|
||||
- **Accept:** a real external customer completes an order start→delivery in production; feedback loop running.
|
||||
|
||||
---
|
||||
|
||||
## Working conventions (for Claude Code)
|
||||
- Tech docs + identifiers **English**; UI copy **Swedish**.
|
||||
- Conventional Commits; one concern per PR; keep diffs reviewable.
|
||||
- No secrets in code; update `.env.example` when adding env.
|
||||
- Add/extend tests for engine logic (estimate math, fusion/confidence, dedup) and API contract.
|
||||
- When a source's access/licence is unclear, **verify and record in `sources.yaml`** before relying; if disallowed for bulk, route to licensed/partner or mark level 0 — never bypass, never fabricate.
|
||||
- Touch the design tokens/motion only via `docs/05`; keep red ≤ once per screen; no photography.
|
||||
- If anything here conflicts with existing code, **flag it** in the PR description and propose the reconciliation rather than guessing.
|
||||
Reference in New Issue
Block a user