diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..9ee08c6 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,57 @@ +# CLAUDE.md — 0pt (SIAX Technology AB) + +> Read this first. It is the contract for how this project is built. Then read `docs/` in order. + +## What 0pt is + +**0pt — your data.** A Nordic/European data-services platform (`0pt.app`) where customers: +1. **Compose** a data job (markets, entity type, fields, enrichment, frequency, output) — or browse ready-made datasets. +2. Pay a **50 % deposit** to start; the remaining 50 % is held in **escrow** and released only on approval. +3. **Watch production live** (sourcing → extraction → structuring → enrichment → QA). +4. **Validate** the result with full freedom (table / map / schema / quality views) and deliver to any format/destination. + +Owned by SIAX Technology AB. Feeds internal products too (Protosell/FiveCRM, VehIQ). + +## Non-negotiable constraints (do not violate) + +1. **Legal sourcing only.** The data engine uses, in strict priority: official APIs/bulk → authorized scraping of clearly public pages (respect `robots.txt` + rate limits) → licensed partner data → otherwise the field is delivered as **"not available / licensed only"**. We never invent values. + - **Do NOT build, keep, or extend**: WAF/bot-detection bypass (incl. FlareSolverr, "stealth" browsers, undetected-* used to defeat detection), fingerprint/proxy rotation to evade bans, scraping behind login walls, or automated fake-account creation (Apollo etc.). + - If existing code (e.g. `src/workers/browser_worker.py`) contains a FlareSolverr/bypass path, **remove or disable it** and replace with compliant fetching. This is a product-positioning + compliance requirement, not a preference. "Honest accuracy" is the whole moat. +2. **Postgres is the backbone.** One Postgres is the source of truth. Catalog, portal, NocoDB views, Metabase dashboards, workers, exports all read/write the *same* DB. A tool may only enter the stack if it (a) reads that Postgres and (b) sits behind the same auth. +3. **Provenance per field.** Every fact carries source, freshness, confidence (`bridge_provenance`). This drives the UI signature too. +4. **One tool per job.** Do not add overlapping tools. The approved stack is in `docs/02-architecture.md` and `docs/07-dependencies.md`. Anything not listed needs a documented reason. +5. **GDPR-native.** DPA, lawful basis, audit log, deletion-after-job. Customer DB access is *pull by the customer* via a read-only, time-boxed role they create — never us asking for admin, never raw creds stored in plaintext. +6. **Design is fixed.** Mono/editorial system (white `#fff`, text `#0a0a0a`, hairline `#e5e5e5`, radius 0, Inter + JetBrains Mono, single red `#e40014`). Exact tokens + motion in `docs/05-design-system.md`. No photography — imagery is rendered data (`DataArtifact`). + +## Build on existing code + +A working repo already exists (~145 files): FastAPI orchestrator (`src/orchestrator/main.py`, 25+ endpoints), workers (`src/workers/`: adaptive_router, browser_worker, ocr_worker, db), and country data for 20 markets. **Reconcile with it, don't blindly replace.** Where this documentation defines a target contract, migrate the existing code toward it. Flag conflicts instead of guessing. + +## How to work + +- Work in **dependency order** per `docs/09-build-plan.md`. Finish a phase to its acceptance criteria before the next. +- Small, reviewable commits. Conventional Commits (`feat:`, `fix:`, `docs:`, `chore:`). +- Frontend = Next.js App Router + TS + Tailwind + shadcn/ui on Vercel. Backend = FastAPI + workers, **not** on Vercel (long jobs). +- Tech docs + code identifiers in **English**; all visible **UI copy in Swedish**. +- Never hardcode secrets — use env (`docs/08-infrastructure.md`). +- When unsure about a source's current access/licence terms, **verify before relying on it** and record it in the sourcing map; do not assume. + +## Documentation index + +| Doc | Purpose | +|---|---| +| `docs/01-product.md` | Vision, the three flows, order lifecycle, business model | +| `docs/02-architecture.md` | System architecture, stack decisions (in/out), hosting | +| `docs/03-data-model.md` | Postgres schema, provenance model, migrations | +| `docs/04-api.md` | FastAPI contract (canonical endpoints) | +| `docs/05-design-system.md` | Mono tokens, motion, visual system, signature | +| `docs/06-data-engine.md` | Workers, legal sourcing chain, pipeline, QA, fusion, sources.yaml | +| `docs/07-dependencies.md` | Repos/libraries to import (+ explicit exclusions) | +| `docs/08-infrastructure.md` | Hosting, docker-compose, env, security, CI/CD | +| `docs/09-build-plan.md` | Phased milestones with acceptance criteria | +| `docs/design/0pt-v0-build-brief.md` | Per-component v0 prompts (design reference) | +| `docs/design/0pt-design-coverage-and-visuals.md` | Full page/component inventory + visual system | + +## Definition of done (every screen, before real data) + +Mono chrome · correct scroll-reveal motion · representative placeholder content · **loading + empty + error** states · mobile layout · provenance signature where facts appear · red used ≤ once · visible focus ring · `prefers-reduced-motion` respected. diff --git a/app/globals.css b/app/globals.css index 632f131..5ac6dd8 100644 --- a/app/globals.css +++ b/app/globals.css @@ -75,9 +75,9 @@ } @theme inline { - --font-sans: 'Inter', 'Inter Fallback', system-ui, sans-serif; - --font-mono: 'JetBrains Mono', monospace; - --font-display: 'PP Editorial New', 'Times New Roman', serif; + --font-sans: var(--font-inter), 'Inter', system-ui, sans-serif; + --font-mono: var(--font-jetbrains-mono), 'JetBrains Mono', ui-monospace, monospace; + --font-display: var(--font-inter), 'Inter', system-ui, sans-serif; --color-background: var(--background); --color-foreground: var(--foreground); --color-card: var(--card); diff --git a/app/layout.tsx b/app/layout.tsx index 559eced..fb8fe11 100644 --- a/app/layout.tsx +++ b/app/layout.tsx @@ -1,45 +1,26 @@ -import React from "react" -import type { Metadata } from 'next' -import { Inter } from 'next/font/google' -import { Analytics } from '@vercel/analytics/next' -import './globals.css' +import React from "react"; +import type { Metadata } from "next"; +import { Inter, JetBrains_Mono } from "next/font/google"; +import { Analytics } from "@vercel/analytics/next"; +import "./globals.css"; -const inter = Inter({ subsets: ["latin"], variable: '--font-inter' }); +const inter = Inter({ subsets: ["latin"], variable: "--font-inter" }); +const jetbrainsMono = JetBrains_Mono({ subsets: ["latin"], variable: "--font-jetbrains-mono" }); export const metadata: Metadata = { - title: 'EVASION', - description: 'High-performance outdoor gear engineered for the modern explorer. Lightweight, durable, adventure-ready.', - generator: 'v0.app', - icons: { - icon: [ - { - url: '/icon-light-32x32.png', - media: '(prefers-color-scheme: light)', - }, - { - url: '/icon-dark-32x32.png', - media: '(prefers-color-scheme: dark)', - }, - { - url: '/icon.svg', - type: 'image/svg+xml', - }, - ], - apple: '/apple-icon.png', - }, -} + title: "0pt — your data", + description: + "Datatjänster på beställning för Norden & Europa. Härkomst per fält, GDPR-native. Sätt ihop ditt datajobb, betala 50 % för att starta, följ produktionen live och validera resultatet med full insyn.", + generator: "0pt.app", +}; -export default function RootLayout({ - children, -}: Readonly<{ - children: React.ReactNode -}>) { +export default function RootLayout({ children }: Readonly<{ children: React.ReactNode }>) { return ( - - + + {children} - ) + ); } diff --git a/components/header.tsx b/components/header.tsx index 8eac87a..292ddfa 100644 --- a/components/header.tsx +++ b/components/header.tsx @@ -1,126 +1,94 @@ "use client"; -import { useState, useEffect } from "react"; import Link from "next/link"; -import { Menu, X } from "lucide-react"; +import { useEffect, useState } from "react"; + +const NAV = [ + { label: "Katalog", href: "/#katalog" }, + { label: "Så funkar det", href: "/#motorn" }, + { label: "Källor", href: "/#tackning" }, + { label: "Priser", href: "/#priser" }, +]; + +function Wordmark() { + return ( + + 0pt + + your data + + ); +} export function Header() { - const [isMenuOpen, setIsMenuOpen] = useState(false); - const [isScrolled, setIsScrolled] = useState(false); + const [scrolled, setScrolled] = useState(false); + const [open, setOpen] = useState(false); useEffect(() => { - const handleScroll = () => { - setIsScrolled(window.scrollY > 50); - }; - - window.addEventListener("scroll", handleScroll, { passive: true }); - return () => window.removeEventListener("scroll", handleScroll); + const onScroll = () => setScrolled(window.scrollY > 40); + window.addEventListener("scroll", onScroll, { passive: true }); + onScroll(); + return () => window.removeEventListener("scroll", onScroll); }, []); return ( -
-
- {/* Logo */} - - MONO - +
+ - {/* Desktop Navigation */} -
- )} + ) : null}
); } diff --git a/components/sections/hero-section.tsx b/components/sections/hero-section.tsx index 9f8b530..39baf3a 100644 --- a/components/sections/hero-section.tsx +++ b/components/sections/hero-section.tsx @@ -1,208 +1,100 @@ "use client"; -import Image from "next/image"; -import { useEffect, useRef, useState } from "react"; +import Link from "next/link"; +import { useReveal } from "@/hooks/use-reveal"; -const word = "MONO"; - -const sideImages = [ - { - src: "/images/hero-side-1.png", - alt: "Modern architecture with corten steel", - position: "left", - span: 1, - }, - { - src: "/images/hero-side-2.png", - alt: "Aerial view of modern home", - position: "left", - span: 1, - }, - { - src: "/images/hero-side-3.png", - alt: "Interior view with landscape", - position: "right", - span: 1, - }, - { - src: "/images/hero-side-4.png", - alt: "Modern architecture at night", - position: "right", - span: 1, - }, +const STATS = [ + ["20", "marknader täckta"], + ["Per fält", "härkomst & konfidens"], + ["50/50", "start & godkänt"], + ["GDPR", "native, med DPA"], ]; +// Data-native hero artifact: a miniature company table that "is" the product. +const ROWS: [string, string, string, 0 | 1 | 2 | 3][] = [ + ["556016-0680", "Volvo Group", "GLEIF", 3], + ["915933149", "Equinor ASA", "Brreg", 3], + ["25052943", "Maersk A/S", "CVR", 2], + ["0112038-9", "Nokia Oyj", "PRH", 2], + ["—", "E-post (FI)", "licens", 0], +]; +const PROV = ["var(--prov-0)", "var(--prov-1)", "var(--prov-2)", "var(--prov-3)"]; + export function HeroSection() { - const sectionRef = useRef(null); - const [scrollProgress, setScrollProgress] = useState(0); - - useEffect(() => { - const handleScroll = () => { - if (!sectionRef.current) return; - - const rect = sectionRef.current.getBoundingClientRect(); - const scrollableHeight = window.innerHeight * 2; - const scrolled = -rect.top; - const progress = Math.max(0, Math.min(1, scrolled / scrollableHeight)); - - setScrollProgress(progress); - }; - - window.addEventListener("scroll", handleScroll, { passive: true }); - handleScroll(); - - return () => { - window.removeEventListener("scroll", handleScroll); - }; - }, []); - - // Text fades out first (0 to 0.2) - const textOpacity = Math.max(0, 1 - (scrollProgress / 0.2)); - - // Image transforms start after text fades (0.2 to 1) - const imageProgress = Math.max(0, Math.min(1, (scrollProgress - 0.2) / 0.8)); - - // Smooth interpolations - More balanced distribution - const centerWidth = 100 - (imageProgress * 80); // 100% to 20% (same as each side image) - const centerHeight = 100; // Always 100% height - const sideWidth = imageProgress * 40; // 0% to 40% (20% per image, 2 images = 40%) - const sideOpacity = imageProgress; - const sideTranslateLeft = -100 + (imageProgress * 100); // -100% to 0% - const sideTranslateRight = 100 - (imageProgress * 100); // 100% to 0% - const borderRadius = 0; // No border radius - const gap = imageProgress * 8; // 0px to 8px - - // Vertical offset for side columns to move them up on mobile - const sideTranslateY = -(imageProgress * 15); // Move up by 15% when fully expanded + const ref = useReveal(); return ( -
- {/* Sticky container for scroll animation */} -
-
- {/* Bento Grid Container */} -
- - {/* Left Column */} -
- {sideImages.filter(img => img.position === "left").map((img, idx) => ( -
- {img.alt} -
- ))} -
+
+ {/* faint oversized wordmark */} +
+ 0pt +
+ - {/* Main Hero Image - Center */} -
+
+

+ 00 — Datatjänster på beställning · Norden & EU +

+

+ Beställ exakt den data du behöver. +
+ Validera varje fält. +

+

+ Sätt ihop ditt eget datajobb. Betala 50 % för att starta. Följ produktionen i realtid. Godkänn + resultatet med full insyn i var varje uppgift kommer ifrån — och hur säker den är. +

+
+ - {/* Text Behind - Fades out first */} -
-

- {word.split("").map((letter, index) => ( - - {letter} - - ))} -

+ Bygg en beställning + + + Se ett levererat dataset + +
+ +
+ {STATS.map(([n, l]) => ( +
+

{n}

+

{l}

- - Modern architectural structure with reflection -
+ ))} +
+
- {/* Right Column */} -
- {sideImages.filter(img => img.position === "right").map((img, idx) => ( -
- {img.alt} + {/* the artifact */} +
+
+
+ dim_company · exempel + härkomst +
+ {ROWS.map((r, i) => ( +
+
+

{r[1]}

+

{r[0]}

- ))} -
- + + + {r[2]} + +
+ ))}
- - {/* Tagline Section - Fixed at bottom */} -
-

- Lightweight, durable -
- and adventure-ready. -

-
- - {/* Scroll space to enable animation */} -
); } diff --git a/docs/01-product.md b/docs/01-product.md new file mode 100644 index 0000000..5274344 --- /dev/null +++ b/docs/01-product.md @@ -0,0 +1,49 @@ +# 01 · Product + +## Vision + +0pt is the most flexible, honest data-services platform for the Nordic + European B2B market. Customers order any kind of data work — from ready-made datasets to bespoke pipelines — and get full visibility into where every value came from and how confident it is. The differentiator is **honest accuracy**: provenance per field, freshness tracking, source transparency. Incumbents (Apollo, Cognism, ZoomInfo) overclaim; 0pt shows its work. + +## The three flows (everything reduces to these) + +Every service maps to one of three intake/delivery flows. The nine "categories" (scraping, structuring, datasets, ICP/leads, BI, ML, documents, subscriptions, custom) are **tags/filters within these flows**, not separate machinery. + +1. **Catalog product** — ready-made, browse → sample → buy → deliver. Exists in Postgres or produced fast from the pipeline. Margin + scale. +2. **Custom job** — does not exist yet; must be scoped. `configure (AI-assist) → estimate → manual scope/quote confirm → produce → QA → validate → deliver`. The hero flow. Consulting in a portal. +3. **Subscription / feed** — recurring product or job: lead feeds, freshness monitoring, ongoing enrichment. `configure → recurring payment → recurring delivery/webhook`. + +## Order lifecycle (state machine) + +``` +DRAFT → QUOTED → DEPOSIT_PENDING → IN_PRODUCTION → REVIEW + → (REVISION → IN_PRODUCTION)* → ACCEPTED → DELIVERED → [SUBSCRIPTION_ACTIVE] +``` + +- **DRAFT**: configurator in progress. +- **QUOTED**: estimate produced (catalog = instant; custom = after scope confirm). +- **DEPOSIT_PENDING**: awaiting 50 % deposit. +- **IN_PRODUCTION**: deposit captured → workers run. Customer watches live. +- **REVIEW**: QA passed threshold → customer validates (table/map/schema/quality). +- **REVISION**: customer requested changes → escrow untouched → back to production. +- **ACCEPTED**: customer approves → remaining 50 % released from escrow. +- **DELIVERED**: exports + destinations unlocked; provenance log + quality report attached. +- **SUBSCRIPTION_ACTIVE**: if converted to a feed. + +## Business / escrow model + +- **50 / 50.** Deposit (50 %) captured to start production. Balance (50 %) **held** and only captured/released on **ACCEPTED**. Revision never moves the held amount. +- Pricing models: pay-per-job, volume, subscription (monthly), custom quote. +- Estimate is a function of: selected field weights × markets × volume × entity multiplier × confidence-target multiplier + enrichment. Custom jobs get a manual scope/quote gate before DEPOSIT_PENDING. +- Stripe: deposit = capture now; balance = either a held authorization released on accept, or a second invoice on accept (implement as a second Checkout/invoice keyed to the order; see `docs/04-api.md`). + +## Customer DB access (designed, not a checkbox) + +Default: the customer **pushes** data (upload/export) — we rarely need to read their DB. If we must: a **read-only, table-scoped, time-boxed** role the customer creates, IP-allowlisted, ideally against a replica. Creds encrypted, deleted after the job. Everything logged. This is a selling point to compliance-heavy buyers, and it requires a DPA. + +## Personas (brief — for copy + prioritization) + +SaaS-Martin (RevOps, wants clean ICP/leads), Bilhandlar-Bengt (vehicle + company data via VehIQ), Rekryterar-Rebecka (people/firmographics), Investor-Ivar (diligence, sanctions/UBO, scrutinizes compliance), Ekonomi-Erik (financials, scrutinizes compliance), Data-David (raw datasets + API). Ivar and Erik are *why* honest sourcing + DPA must be airtight. + +## First-launch scope (de-risked) + +Do **not** ship "20 markets × 9 categories." Ship the thinnest sellable slice: **Nordic company data** (the open stuff — GLEIF + Brreg + verified sources) + **vehicle data** (VehIQ synergy) as a couple of catalog datasets, **plus the custom-job intake**, Stripe, and delivery to download + into FiveCRM/Data Loader. First customers are Protosell internally + existing dealer relationships — services you already deliver, productized. Then expand markets as the sourcing map fills in. diff --git a/docs/02-architecture.md b/docs/02-architecture.md new file mode 100644 index 0000000..9fd1da1 --- /dev/null +++ b/docs/02-architecture.md @@ -0,0 +1,70 @@ +# 02 · Architecture + +## Principle: one database, many lenses + +Postgres is the source of truth. Everything else is a **lens** on it. A tool enters the stack only if it (a) reads that Postgres and (b) sits behind the same auth. This is the difference between an app and a pile of separate logins. + +``` + ┌──────────────────────────────────────────────┐ + COLLECTION │ POSTGRES │ LENSES (customer + admin) + (legal engine) │ dim_company · dim_contact · dim_vehicle │ + ─────────────── │ fact_financials · dim_ubo │ ───────────────────────────── + Official APIs │ dim_source · bridge_provenance │ Next.js portal (Vercel) + /bulk ───────▶│ orders · order_items · jobs · deliveries │◀── Catalog + order lifecycle + Authorized web │ subscriptions · users · orgs · files │ NocoDB (embedded "Min data") + Licensed data │ db_grants · audit_log · pgvector embeddings │ Metabase (embedded dashboards) + Customer upload│ │ Export: CSV/Parquet/JSON/API/FiveCRM + └───────────────────▲──────────────────────────┘ + │ + FastAPI orchestrator + workers + (estimates, orders, jobs, AI-assist, webhooks, pipeline) +``` + +## Components & responsibilities + +- **Next.js (Vercel)** — all customer/admin UI: marketing site, catalog, configurator, checkout, production tracker, validation, portal, admin. Calls FastAPI over an internal token. Embeds NocoDB/Metabase via signed URLs (no separate login shown). +- **FastAPI orchestrator** (`src/orchestrator`) — REST contract in `docs/04-api.md`: estimates, orders, AI-assist, uploads, deliveries, webhooks, catalog, sources, job status/SSE. **Not on Vercel** (serverless timeouts kill long jobs). +- **Workers** (`src/workers`) — adaptive router + pipeline stages (sourcing → extraction/OCR → structuring → enrichment/fusion → QA). Write to Postgres with provenance. See `docs/06-data-engine.md`. +- **Postgres** — backbone (`docs/03-data-model.md`) incl. `pgvector` for matching/embeddings. +- **NocoDB** — Airtable-style views over existing tables (no data duplication). Embedded into portal as "Min data". +- **Metabase** — customer-facing dashboards. Embedded. +- **Cloudflare R2** — file storage (uploads + deliverables) via presigned URLs. +- **Stripe** — deposit capture + escrow balance release. +- **Resend** — transactional email (order confirmation, "din data är klar", feedback). +- **Auth.js** (NextAuth) with a Postgres adapter — single auth for the whole app; NocoDB/Metabase hidden behind it. + +## Stack decisions — IN + +| Concern | Choice | Why | +|---|---|---| +| Frontend | Next.js App Router + TS + Tailwind + shadcn/ui | Vercel-native, matches existing skills | +| Backend/jobs | FastAPI + workers (existing) | already built; long-running jobs | +| DB | Postgres (+ pgvector) | backbone; Neon free tier or on-VPS | +| Airtable layer | **NocoDB** | sits on existing tables, no duplication | +| BI | **Metabase** | customer-facing embedding, simple | +| Files | Cloudflare R2 + presigned URLs | cheapest S3-compatible | +| Payments | Stripe Checkout + webhook → FastAPI | ~1 day, no SaaS boilerplate needed | +| Auth | Auth.js + Postgres adapter | one system | +| Email | Resend | best Next.js integration, free tier | +| Quality | Great Expectations / checks **inside workers** | scores shipped with each order | + +## Stack decisions — OUT (do not add) + +| Rejected | Reason | +|---|---| +| Superset | Metabase already covers BI — one tool, not two | +| DataHub / OpenMetadata | enterprise-heavy; catalog = a Postgres table in Next.js + NocoDB. Add lineage only if an enterprise customer demands it | +| Baserow / Teable | NocoDB chosen — not three Airtable clones | +| Meilisearch (v1) | Postgres full-text suffices to a few thousand rows; add later | +| Delta Sharing (v1) | unnecessary; customer pull via scoped role + exports | +| Trigger.dev | the existing queue already schedules jobs; Resend covers mail | +| FlareSolverr / stealth browsers / undetected-* / proxy-fingerprint rotation / fake-account creation | **prohibited** — see CLAUDE.md §1. Replace with compliant fetching | + +## Hosting (cheapest coherent) + +- **Vercel** (Hobby/free) — Next.js + custom domain `0pt.app`. +- **One small VPS** (e.g. Hetzner ~€5/mo) running `docker-compose`: FastAPI, workers, NocoDB, Metabase, (Postgres or use Neon), optional Meilisearch later. Per-service managed platforms (Railway) get expensive once you have 4–5 always-on services; a single VPS + Vercel + R2 lands ≈ <€10/mo to start. +- **Cloudflare R2** — object storage. +- **Postgres** — on the VPS, or **Neon** free tier if you want it managed. + +Full config, env, security, CI/CD: `docs/08-infrastructure.md`. diff --git a/docs/03-data-model.md b/docs/03-data-model.md new file mode 100644 index 0000000..25c3da5 --- /dev/null +++ b/docs/03-data-model.md @@ -0,0 +1,201 @@ +# 03 · Data model + +Postgres is the backbone. Two halves: **data entities** (what we sell) and **platform tables** (orders, users, jobs). Every fact links to `bridge_provenance`. Migrations via Alembic (Python) or `sql/` migrations checked into the repo — pick one and be consistent; align with existing `src/workers/db.py`. + +## Conventions + +- `snake_case`, UTC timestamptz, soft-delete via `deleted_at` where useful. +- Natural keys for registries (`org_nr`), surrogate `uuid` PKs for platform tables. +- `created_at` / `updated_at` on every table. +- All monetary amounts in **öre** (integer) to avoid float errors; currency `SEK` default. + +## Data entities + +```sql +-- A company (the core registry entity) +CREATE TABLE dim_company ( + org_nr text PRIMARY KEY, -- national reg. number (per market format) + market text NOT NULL, -- ISO country code: SE/NO/DK/FI/... + lei text, -- GLEIF cross-border key + name text NOT NULL, + status text, -- active/dissolved/... + nace text, -- NACE/SNI code + registered_at date, + address jsonb, + website text, + updated_at timestamptz NOT NULL DEFAULT now(), + created_at timestamptz NOT NULL DEFAULT now() +); + +CREATE TABLE fact_financials ( + org_nr text REFERENCES dim_company(org_nr), + fiscal_year int NOT NULL, + revenue bigint, -- öre + result bigint, + employees int, + source_id text REFERENCES dim_source(source_id), + PRIMARY KEY (org_nr, fiscal_year) +); + +CREATE TABLE dim_contact ( + contact_id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + org_nr text REFERENCES dim_company(org_nr), + name text, + role text, + email text, + phone text, + linkedin text, + -- PII: subject to GDPR; lawful basis recorded per delivery + updated_at timestamptz NOT NULL DEFAULT now() +); + +CREATE TABLE dim_ubo ( -- ultimate beneficial owner + ubo_id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + org_nr text REFERENCES dim_company(org_nr), + owner text, + share numeric(5,2) +); + +CREATE TABLE dim_vehicle ( -- VehIQ synergy + vin text PRIMARY KEY, + market text NOT NULL, + reg_plate text, + make text, model text, year int, + owner_org text REFERENCES dim_company(org_nr), + specs jsonb, + updated_at timestamptz NOT NULL DEFAULT now() +); +``` + +## Provenance model (the signature — every fact links here) + +```sql +CREATE TABLE dim_source ( + source_id text PRIMARY KEY, -- 'gleif','brreg','companies_house',... + name text NOT NULL, + market text, -- or 'global' + access text, -- 'api'|'bulk'|'authorized_web'|'licensed' + licence text, -- short licence note + cadence text, -- 'daily'|'weekly'|'annual'|... + url text +); + +-- One row per (entity field) we deliver, linking value → where it came from + confidence +CREATE TABLE bridge_provenance ( + field_id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + entity_type text NOT NULL, -- 'company'|'contact'|'vehicle'|'financials' + entity_key text NOT NULL, -- org_nr / vin / contact_id + field_name text NOT NULL, -- 'revenue','email',... + source_id text REFERENCES dim_source(source_id), + freshness date, -- when this value was confirmed + confidence numeric(4,3) NOT NULL, -- 0.000–1.000 + level smallint NOT NULL, -- 0 unavailable/licensed, 1 single, 2 partial, 3 corroborated + agreed_sources int DEFAULT 1, -- # independent sources that agree + UNIQUE (entity_type, entity_key, field_name) +); +CREATE INDEX ON bridge_provenance (entity_type, entity_key); +``` + +> Rule: **no value ships without a `bridge_provenance` row.** Level 0 = delivered as "not available / licensed", never a guessed value. + +## Platform tables + +```sql +CREATE TABLE orgs ( + org_id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + name text NOT NULL, created_at timestamptz DEFAULT now() +); +CREATE TABLE users ( + user_id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + org_id uuid REFERENCES orgs(org_id), + email text UNIQUE NOT NULL, + role text NOT NULL DEFAULT 'member', -- 'owner'|'admin'|'member'|'siax_admin' + created_at timestamptz DEFAULT now() +); + +CREATE TABLE orders ( + order_id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + org_id uuid REFERENCES orgs(org_id), + flow text NOT NULL, -- 'catalog'|'custom'|'subscription' + status text NOT NULL DEFAULT 'draft', -- see lifecycle in docs/01 + config jsonb NOT NULL, -- configurator state (markets, fields, ...) + total_ore bigint, deposit_ore bigint, balance_ore bigint, + est_days int, est_confidence numeric(4,3), + created_at timestamptz DEFAULT now(), updated_at timestamptz DEFAULT now() +); + +CREATE TABLE order_items ( -- resolved line items / dataset refs + item_id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + order_id uuid REFERENCES orders(order_id), + kind text, ref text, qty int, meta jsonb +); + +CREATE TABLE jobs ( -- one production run per order + job_id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + order_id uuid REFERENCES orders(order_id), + stage text, -- sourcing|extraction|structuring|enrichment|qa + pct numeric(5,2) DEFAULT 0, + state text DEFAULT 'queued', -- queued|running|done|failed + metrics jsonb, -- rows, confidence, per-market progress + started_at timestamptz, finished_at timestamptz +); + +CREATE TABLE deliveries ( + delivery_id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + order_id uuid REFERENCES orders(order_id), + formats text[], destinations text[], + record_count int, avg_confidence numeric(4,3), + quality_report jsonb, provenance_export_url text, + files jsonb, -- [{format,url,bytes}] + created_at timestamptz DEFAULT now() +); + +CREATE TABLE subscriptions ( + sub_id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + order_id uuid REFERENCES orders(order_id), + cadence text, -- weekly|daily + webhook_url text, status text DEFAULT 'active', + next_run_at timestamptz +); + +CREATE TABLE files ( -- customer uploads + file_id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + org_id uuid REFERENCES orgs(org_id), + order_id uuid REFERENCES orders(order_id), + name text, mime text, bytes bigint, r2_key text, + created_at timestamptz DEFAULT now() +); + +CREATE TABLE db_grants ( -- customer read-only DB access (encrypted) + grant_id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + org_id uuid REFERENCES orgs(org_id), + order_id uuid REFERENCES orders(order_id), + conn_encrypted bytea NOT NULL, -- encrypted connection string + scope text, expires_at timestamptz, revoked_at timestamptz, + created_at timestamptz DEFAULT now() +); + +CREATE TABLE audit_log ( + id bigserial PRIMARY KEY, + actor text, action text, target text, meta jsonb, + at timestamptz DEFAULT now() +); + +CREATE TABLE payments ( + payment_id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + order_id uuid REFERENCES orders(order_id), + kind text, -- 'deposit'|'balance' + amount_ore bigint, stripe_ref text, + status text, -- 'pending'|'captured'|'released'|'failed' + created_at timestamptz DEFAULT now() +); +``` + +## pgvector + +`CREATE EXTENSION vector;` Use for entity matching/dedup and for the AI-assist (semantic match of NL request → catalog/config). Store embeddings in a `embeddings` table keyed by `(entity_type, entity_key)` or alongside catalog rows. + +## Row-level security & catalog + +- Catalog products: a `catalog` table (or view) over curated datasets with sample rows, counts, coverage, price. +- Per-customer data isolation enforced by `org_id` filtering in the API and (where exposed via NocoDB/Metabase) by scoped views/connection params, plus Postgres RLS where practical. diff --git a/docs/04-api.md b/docs/04-api.md new file mode 100644 index 0000000..2547565 --- /dev/null +++ b/docs/04-api.md @@ -0,0 +1,97 @@ +# 04 · API contract (FastAPI) + +Canonical REST contract. Reconcile the existing 25+ endpoints in `src/orchestrator/main.py` toward this. JSON in/out, `snake_case`. All amounts in **öre**. Auth: Next.js → FastAPI over a shared internal token (`X-Internal-Token`); customer identity passed as `org_id`/`user_id` resolved from the Auth.js session server-side. + +## Conventions +- Errors: `{ "error": { "code", "message" } }`, proper HTTP status. Interface-voice messages (Swedish) are produced in the frontend, not here. +- Idempotency: accept `Idempotency-Key` on POSTs that create orders/payments. +- Pagination: `?limit=&cursor=`. + +## Catalog +``` +GET /catalog -> [{id,title,markets,entity_type,count,updated,price_ore,confidence}] +GET /catalog/{id} -> full dataset (overview, coverage, schema, quality summary) +GET /catalog/{id}/sample?n=10 -> sample rows WITH per-field provenance +``` + +## Estimate (configurator → price/time/confidence) +``` +POST /estimates + body: { flow, markets[], entity_type, fields[], enrichment[], volume, frequency, confidence_target, formats[] } + -> { total_ore, deposit_ore, balance_ore, est_days, est_confidence, breakdown[] } +``` + +## AI-assist (NL → config) +``` +POST /ai/parse + body: { text } -- "5000 svenska SaaS-bolag med e-post och omsättning" + -> { config: , rationale } -- prefilled configurator; uses pgvector match to catalog/fields +``` + +## Orders (lifecycle) +``` +POST /orders body: { flow, config } -> { order_id, status:'draft'|'quoted' } +GET /orders -> [orders for org] +GET /orders/{id} -> order + current job + delivery (if any) +POST /orders/{id}/scope (custom flow) body:{ notes } -> status:'quoted' (manual quote step) +POST /orders/{id}/accept customer approves delivery -> triggers balance release; status:'accepted' +POST /orders/{id}/revise body:{ notes } -> status:'revision'; escrow untouched +``` + +## Payments (50/50 escrow) +``` +POST /payments/{order_id}/deposit -> { checkout_url } (Stripe Checkout for 50%; on success → IN_PRODUCTION) +POST /payments/{order_id}/release -> captures/charges the held balance (called on accept) +POST /webhooks/stripe (Stripe → here) verify signature; update payments + order status +``` +> Implement balance as either a manual capture on a held PaymentIntent, or a second Checkout/invoice keyed to `order_id`. Never charge balance before `accept`. + +## Jobs / live status +``` +GET /orders/{id}/status -> { stage, pct, per_market[], confidence, eta, rows } +GET /orders/{id}/events -> Server-Sent Events stream (preferred for the production tracker) +``` + +## Uploads (R2 presigned) +``` +POST /uploads body:{ order_id, name, mime, bytes } -> { file_id, put_url } (client PUTs to R2) +GET /orders/{id}/files -> [files] +``` + +## Customer DB access (read-only grant) +``` +POST /connections/db-grant + body: { order_id, conn_string, scope, expires_at } -- stored ENCRYPTED; validated read-only + -> { grant_id, status:'pending'|'verified' } +DELETE /connections/db-grant/{id} -> revoke +``` + +## Deliveries +``` +GET /orders/{id}/delivery -> { formats, destinations, record_count, avg_confidence, quality_report, files[], provenance_export_url } +POST /orders/{id}/delivery/destinations body:{ destinations[] } -> updates targets (CSV/Parquet/JSON/API/NocoDB/Metabase/FiveCRM/Sheets/webhook) +``` + +## Sources (the transparency moat) +``` +GET /sources -> [{source_id,name,market,access,licence,cadence,confidence}] +GET /sources/{market} -> sources + freshness for a market +``` + +## Subscriptions / feeds +``` +POST /subscriptions body:{ order_id, cadence, webhook_url } -> { sub_id } +GET /subscriptions -> [subs] +POST /subscriptions/{id}/pause | /resume +``` + +## Webhooks out (to customers) +- On delivery / feed update: POST to `subscriptions.webhook_url` with `{ order_id, delta_url, summary }`. + +## Health +``` +GET /health -> { ok, db, workers, queue } +``` + +## Existing-code reconciliation +Map current endpoints (orders, estimates, ai-assistent, catalog, webhooks, health) onto the names above; rename for consistency, keep behavior. Add: `/orders/{id}/events` (SSE), `/payments/*` escrow split, `/connections/db-grant`, `/sources`, `/subscriptions`. Remove any endpoint that drives a bypass/stealth fetch path. diff --git a/docs/05-design-system.md b/docs/05-design-system.md new file mode 100644 index 0000000..5f6ff0b --- /dev/null +++ b/docs/05-design-system.md @@ -0,0 +1,87 @@ +# 05 · Design system (mono) + +Self-contained spec. Component-level v0 prompts live in `docs/design/0pt-v0-build-brief.md`; the full page/component inventory + visual rules in `docs/design/0pt-design-coverage-and-visuals.md`. This file is the source of truth for tokens + motion; build them once and reuse. + +Derived from the **mono / EVASION** template (itself v0-generated). Light, editorial, monochrome, sharp, with a single red accent and animated grain + scroll-reveal motion. **No photography** — imagery is rendered data. + +## Tokens + +``` +THEME (light, default): + --background #ffffff --foreground #0a0a0a --muted-foreground #737373 + --border #e5e5e5 --card #ffffff --input #f5f5f5 +THEME (dark sections — footer, one editorial block): + --background #0a0a0a --foreground #fafafa --muted-foreground #a3a3a3 --border #262626 +ACCENT (single, sparing): red --accent #e40014 (one CTA / one number / one marker per screen) +PROVENANCE squares (ONLY non-mono color besides the red CTA): + level 0 #737373 (unavailable/licensed) · 1 #143aff (single source) · 2 #fcbb00 (partial) · 3 #00d294 (corroborated) +RADIUS: 0 everywhere (max 2px on tiny chips). No drop shadows; structure via 1px hairlines. +TYPE: Inter (everything; headlines weight 500–600, letter-spacing -0.02em, large) · JetBrains Mono (eyebrows, labels, numbers, codes, table headers — UPPERCASE, letter-spacing 0.12em, ~11px, #737373) +GRID: 8px base; graph-paper hairlines; generous whitespace; left-aligned headlines. +SECTION SPINE: real ordered indices 00–NN as mono eyebrows. +``` + +## globals.css (add verbatim — guarantees identical motion + grain) + +```css +@keyframes reveal-up{0%{opacity:0;transform:translateY(60px)}to{opacity:1;transform:translateY(0)}} +@keyframes reveal-left{0%{opacity:0;transform:translateX(-60px)}to{opacity:1;transform:translateX(0)}} +@keyframes reveal-right{0%{opacity:0;transform:translateX(60px)}to{opacity:1;transform:translateX(0)}} +@keyframes scale-in{0%{opacity:0;transform:scale(.9)}to{opacity:1;transform:scale(1)}} +@keyframes fadeIn{0%{opacity:0;transform:scale(1.02)}to{opacity:1;transform:scale(1)}} +@keyframes slideUp{0%{opacity:0;transform:translateY(100%)}to{opacity:1;transform:translateY(0)}} +@keyframes float{0%,100%{transform:translateY(0)}50%{transform:translateY(-20px)}} +@keyframes grain{0%,100%{transform:translate(0,0)}10%{transform:translate(-5%,-10%)}20%{transform:translate(-15%,5%)}30%{transform:translate(7%,-25%)}40%{transform:translate(-5%,25%)}50%{transform:translate(-15%,10%)}60%{transform:translate(15%,0)}70%{transform:translateY(15%)}80%{transform:translate(3%,35%)}90%{transform:translate(-10%,10%)}} +.reveal{opacity:0} +.reveal.in-view{animation:.8s ease-out forwards reveal-up} +.reveal-left.in-view{animation:.8s ease-out forwards reveal-left} +.reveal-right.in-view{animation:.8s ease-out forwards reveal-right} +.reveal-scale.in-view{animation:.6s ease-out forwards scale-in} +.reveal-fade.in-view{animation:.7s ease-out forwards fadeIn} +.delay-1{animation-delay:.1s}.delay-2{animation-delay:.2s}.delay-3{animation-delay:.3s}.delay-4{animation-delay:.4s}.delay-5{animation-delay:.5s}.delay-6{animation-delay:.6s} +.animate-float{animation:6s ease-in-out infinite float} +.grain-overlay::before{content:"";position:absolute;inset:-100%;opacity:.03;pointer-events:none;background-image:url("data:image/svg+xml,%3Csvg viewBox='0 0 200 200' xmlns='http://www.w3.org/2000/svg'%3E%3Cfilter id='n'%3E%3CfeTurbulence type='fractalNoise' baseFrequency='0.9' numOctaves='4' stitchTiles='stitch'/%3E%3C/filter%3E%3Crect width='100%25' height='100%25' filter='url(%23n)'/%3E%3C/svg%3E");animation:8s steps(10,end) infinite grain} +@media(prefers-reduced-motion:reduce){.reveal,.reveal-left,.reveal-right,.reveal-scale,.reveal-fade{opacity:1;animation:none}.animate-float,.grain-overlay::before{animation:none}} +``` + +## useReveal hook + +```tsx +"use client"; +import { useEffect, useRef } from "react"; +export function useReveal() { + const ref = useRef(null); + useEffect(() => { + const el = ref.current; if (!el) return; + const io = new IntersectionObserver((es) => es.forEach((e) => { + if (e.isIntersecting) { + el.querySelectorAll(".reveal,.reveal-left,.reveal-right,.reveal-scale,.reveal-fade") + .forEach((n) => n.classList.add("in-view")); + io.unobserve(el); + } + }), { threshold: 0.15, rootMargin: "0px 0px -10% 0px" }); + io.observe(el); return () => io.disconnect(); + }, []); + return ref; +} +``` +Every section wraps in a tag using `useReveal()`; children get `reveal[/-left/-right/-scale/-fade]` + `delay-N`. + +## Buttons & labels +- Primary: solid `#0a0a0a` bg, white text, square, mono uppercase 12px / spacing 0.1em, padding 14px 26px; hover → bg `#e40014`. +- Secondary: transparent, 1px `#0a0a0a` border; hover → bg `#0a0a0a`, white text. +- Eyebrow: mono uppercase 11px, spacing 0.14em, `#737373`, with a 2-digit index ("01 — KATALOG"). +- Headlines: sentence case, never uppercase. + +## Visual system — no photos (`DataArtifact`) +Build one generative primitive; every "image" surface uses it, animating with the effect it replaces: +- `table` (rows cascade in, provenance square per fact cell) — hero/cards. +- `map` (country-code cells, fill bars animate width on view) — coverage. +- `schema` (ER mini-diagram, key boxes red border) — schema previews. +- `chart` (black bars/lines, exactly ONE red series, grow on view) — dashboards. +- `grid` (hairline graph-paper + faint oversized numeral + one floating red square) — pure texture. + +Mapping of template image elements → replacement + effect kept is in the coverage doc. Rules: code-only (CSS/SVG/canvas), color discipline inside visuals (mono + provenance squares + one red series), grain over any "image" surface, `prefers-reduced-motion` settles to static. Optional: add **Lenis** smooth scroll on top — preserves all effects. + +## The signature (carry everywhere) +Provenance + confidence is the identity: the provenance square on every fact, one red, mono numerals as texture, section spine, identical motion. A screen that lacks these is off-brand. diff --git a/docs/06-data-engine.md b/docs/06-data-engine.md new file mode 100644 index 0000000..b16d090 --- /dev/null +++ b/docs/06-data-engine.md @@ -0,0 +1,70 @@ +# 06 · Data engine + +The engine turns sources into delivered, provenance-tagged data. It is the legal motor — the whole product positioning depends on it. Build on existing `src/workers/` (`adaptive_router`, `browser_worker`, `ocr_worker`, `db`) but enforce the sourcing boundary below. + +## Sourcing chain (strict priority — this is the law) + +For each field × market, acquire in this order; stop at the first that works: + +1. **Official API / bulk** — registries with programmatic/bulk access (highest confidence). +2. **Authorized public web** — clearly public pages, **respecting `robots.txt` and rate limits**, no auth walls. +3. **Licensed partner data** — contractually obtained. +4. **Otherwise → "not available / licensed only"** — delivered honestly as level 0. **Never guessed.** + +### Prohibited (do not build, keep, or extend) +- WAF/bot-detection bypass — **including FlareSolverr**, "stealth"/patched browsers, undetected-* used to defeat detection. +- Proxy/fingerprint rotation to evade bans; scraping behind login walls; automated fake-account creation (Apollo etc.). +- **Reconciliation:** if `browser_worker.py` has an `HTTP → FlareSolverr → Playwright` chain, **drop the FlareSolverr stage** and keep only: polite HTTP fetch (identifies itself, obeys robots + rate limits) → optional Playwright for genuinely public JS-rendered pages (same politeness). The `adaptive_router` then chooses among **compliant** methods only (API > bulk > authorized fetch), not bypass tiers. + +## Pipeline stages (per job) + +``` +sourcing → extraction/OCR → structuring → enrichment/fusion → QA +``` + +1. **Sourcing** — `adaptive_router` picks the compliant method per source; "source-racing" = query several *authorized* sources and take the fastest/freshest. Writes raw payloads + which source answered. +2. **Extraction / OCR** — API JSON parsed directly; documents (PDF/HTML/images) via **Unstructured** + OCR cascade (PaddleOCR → Tesseract → LLM fallback) in `ocr_worker`. Output: candidate fields. +3. **Structuring** — NER + entity resolution (spaCy / transformers, sv/no/da/fi/de/fr), normalize to schema, link entities across markets via **LEI** (GLEIF golden key). Dedup with **RapidFuzz**. +4. **Enrichment / fusion** — combine fields from multiple sources; compute **multi-source agreement** → confidence + level; append optional enrichment (NACE/SNI, sanctions/PEP flag from official lists, website/tech). +5. **QA** — quality checks (**Great Expectations** or equivalent, run inside the worker): validity (format/range), completeness, freshness, uniqueness (post-dedup), corroboration. Drop rows below the order's `confidence_target` (deliver fewer, never fabricate). Emit the quality report stored on `deliveries`. + +Each stage updates `jobs` (`stage`, `pct`, `metrics` incl. per-market progress, running counts, climbing confidence) so the production tracker / SSE can render it live. + +## Confidence & provenance scoring + +- Per field, write a `bridge_provenance` row: `source_id`, `freshness`, `confidence` (0–1), `level` (0–3), `agreed_sources`. +- Level: 0 = unavailable/licensed; 1 = single source; 2 = partial corroboration; 3 = ≥2 independent sources agree. +- `est_confidence` on the order = aggregate target; delivered `avg_confidence` = actual. + +## `sources.yaml` (machine-readable registry) + +Single registry that drives the router, the `/sources` API, and the public sourcing-map page. Schema per entry: + +```yaml +- source_id: gleif + name: GLEIF (LEI) + market: global + access: api # api | bulk | authorized_web | licensed + cadence: daily + licence: "open (CC0) — VERIFY current terms" + entities: [company] + fields: [lei, legal_name, address] + base_url: https://api.gleif.org + robots_ok: true # for authorized_web only; must be honored + rate_limit: "per provider terms" + notes: "cross-border key" +``` + +### Starter set (VERIFY current access/licence per source before relying — terms change) +- **GLEIF / LEI** — typically open; cross-border key. *(verify)* +- **Brønnøysund (Brreg, NO)** — typically open API. *(verify)* +- **Companies House (UK)** — free API (key required). *(verify)* +- **CVR / Virk (DK)** — API access with terms. *(verify)* +- **PRH (FI)** — varies. *(verify)* +- **Bolagsverket (SE)** — more restricted for bulk; plan licensed/partner route. *(verify)* +- Vehicle: **RDW (NL)** open-ish, **Traficom (FI)**, **Transportstyrelsen (SE)**, **DVLA (UK)** — *(verify each; vehicle/PII rules differ)* + +> Do not hardcode access terms as fact. When building each connector, confirm the current API/licence/robots status and record it in `sources.yaml`. If unclear or disallowed for bulk, route to licensed/partner or mark fields level 0. + +## Libraries used here +Crawlee or Scrapy (orchestration of **authorized** fetches) · Playwright (public JS pages, politely) · Unstructured (document parsing) · PaddleOCR/Tesseract (OCR) · spaCy / HF transformers (NER, multilingual) · Polars (fast tabular) · RapidFuzz (dedup/matching) · Great Expectations (quality) · dbt (optional transform layer). Versions/install: `docs/07-dependencies.md`. diff --git a/docs/07-dependencies.md b/docs/07-dependencies.md new file mode 100644 index 0000000..5351d46 --- /dev/null +++ b/docs/07-dependencies.md @@ -0,0 +1,65 @@ +# 07 · Dependencies & repos to import + +Curated. Each entry: what it is, where it slots in, how to install. **Do not import on reputation/star-count** — several items in earlier brainstorming were unverified or hallucinated (see Exclusions). Verify each repo exists and is maintained before adopting. + +## Frontend (Next.js on Vercel) +| Package | Purpose | Install | +|---|---|---| +| next, react, typescript | App Router base | `npx create-next-app@latest --ts` | +| tailwindcss | styling (mono tokens) | per Next setup | +| shadcn/ui | base components | `npx shadcn@latest init` | +| @tanstack/react-table **or** ag-grid-react | data tables / catalog grid | `npm i @tanstack/react-table` | +| @uppy/core @uppy/react @uppy/dashboard @uppy/aws-s3 | file upload → R2 (presigned) | `npm i @uppy/core @uppy/react @uppy/dashboard @uppy/aws-s3` | +| next-auth (Auth.js) + @auth/pg-adapter | single auth, Postgres-backed | `npm i next-auth @auth/pg-adapter` | +| stripe / @stripe/stripe-js | checkout + portal | `npm i stripe @stripe/stripe-js` | +| resend / react-email | transactional email | `npm i resend @react-email/components` | +| lenis *(optional)* | smooth scroll (keeps all effects) | `npm i lenis` | +| recharts *(only if needed)* | charts inside DataArtifact/dashboards | `npm i recharts` | + +> Catalog/search in v1 = Postgres full-text. Add Meilisearch later only past a few thousand rows. + +## Backend / workers (Python, FastAPI — existing) +| Package | Purpose | Install | +|---|---|---| +| fastapi, uvicorn, pydantic | API (existing) | `pip install fastapi uvicorn pydantic` | +| sqlalchemy / asyncpg | Postgres access (align `db.py`) | `pip install sqlalchemy asyncpg` | +| alembic | migrations | `pip install alembic` | +| celery + redis **or** existing adaptive_router/queue | job queue/concurrency | `pip install celery redis` | +| httpx | polite HTTP fetch (identifies itself, obeys rate limits) | `pip install httpx` | +| scrapy **or** crawlee | orchestrate **authorized** fetches | `pip install scrapy` | +| playwright | public JS-rendered pages, politely | `pip install playwright && playwright install chromium` | +| unstructured | parse PDF/HTML/DOCX/images → elements | `pip install "unstructured[all-docs]"` | +| paddleocr / pytesseract | OCR cascade | `pip install paddleocr pytesseract` (+ tesseract bin) | +| spacy (+ models) / transformers | NER, entity resolution (multilingual) | `pip install spacy transformers` | +| polars | fast tabular processing | `pip install polars` | +| rapidfuzz | dedup / fuzzy matching | `pip install rapidfuzz` | +| great-expectations | data quality checks in pipeline | `pip install great-expectations` | +| dbt-postgres *(optional)* | transform layer | `pip install dbt-postgres` | +| pgvector (Postgres ext) | embeddings/matching | `CREATE EXTENSION vector;` | +| boto3 *(or aioboto3)* | R2 (S3-compatible) | `pip install boto3` | +| stripe (python) | webhook handling | `pip install stripe` | + +## Self-hosted services (docker-compose on the VPS) +| Service | Purpose | Image | +|---|---|---| +| **NocoDB** | Airtable-style views over existing Postgres tables; embedded "Min data" | `nocodb/nocodb` | +| **Metabase** | customer-facing dashboards; embedded | `metabase/metabase` | +| **Postgres** *(or Neon managed)* | backbone (+ pgvector) | `pgvector/pgvector:pg16` | +| **Redis** *(if Celery)* | queue | `redis` | +| **Meilisearch** *(later)* | catalog search at scale | `getmeili/meilisearch` | + +## External managed (free/cheap tiers) +- **Vercel** — Next.js hosting. +- **Cloudflare R2** — object storage (presigned URLs). +- **Stripe** — payments (test mode first). +- **Resend** — email. +- **Neon** *(optional)* — managed Postgres. + +## EXCLUSIONS — do not import + +- **Bypass/stealth/evasion**: FlareSolverr, "CloakBrowser", undetected-chromedriver/undetected-playwright (as bypass), proxy/fingerprint-rotation engines, any "WAF bypass" repo. Prohibited by CLAUDE.md §1 — they destroy the honest-accuracy positioning and create legal exposure. +- **Fake-account automation** on Apollo/Cognism/etc. +- **Unverified / likely-hallucinated** from earlier brainstorming: "CloakBrowser/CloakHQ", "Rota proxy engine", "OpenClaw". Do not adopt unless you confirm a real, maintained project — and even then, not if it falls in the bypass category. +- **Overlapping tools** already decided against (Superset, DataHub/OpenMetadata, Baserow/Teable, Trigger.dev, Delta Sharing in v1) — see `docs/02-architecture.md`. + +> Note on real-but-overhyped tools (Firecrawl, browser-use, etc.): they exist, but ignore inflated star-counts and only adopt if they fit a *compliant* role and don't duplicate the stack. diff --git a/docs/08-infrastructure.md b/docs/08-infrastructure.md new file mode 100644 index 0000000..5af56db --- /dev/null +++ b/docs/08-infrastructure.md @@ -0,0 +1,118 @@ +# 08 · Infrastructure + +## Repo structure (monorepo) + +``` +0pt/ +├─ CLAUDE.md +├─ docs/ # this documentation set +├─ apps/ +│ └─ web/ # Next.js (Vercel): marketing, catalog, portal, admin +├─ services/ +│ └─ api/ # FastAPI orchestrator + workers (existing src/* migrates here) +│ ├─ orchestrator/ # main.py + routers (docs/04-api.md) +│ ├─ workers/ # adaptive_router, fetch_worker, ocr_worker, pipeline, db +│ └─ sources/ # sources.yaml + connectors (docs/06) +├─ packages/ +│ └─ shared/ # shared TS types (mirror API contract), design tokens +├─ sql/ (or services/api/alembic/) # migrations (docs/03) +├─ infra/ +│ ├─ docker-compose.yml # VPS always-on services +│ └─ Caddyfile # reverse proxy + TLS +└─ .github/workflows/ # CI/CD +``` + +## Hosting topology + +- **Vercel** — `apps/web` → `0pt.app`. Free Hobby until paying customers. +- **One VPS** (Hetzner ~€5/mo) — `docker-compose`: FastAPI, workers, NocoDB, Metabase, Postgres (or Neon), Redis (if Celery), Caddy. Caddy terminates TLS and proxies `api.0pt.app`, `data.0pt.app` (NocoDB), `dash.0pt.app` (Metabase). +- **Cloudflare R2** — files. +- **Postgres** — on VPS (`pgvector/pgvector:pg16`) or **Neon** managed. + +## docker-compose (infra/docker-compose.yml — sketch) + +```yaml +services: + postgres: + image: pgvector/pgvector:pg16 + environment: { POSTGRES_USER: opt, POSTGRES_PASSWORD: ${PG_PASSWORD}, POSTGRES_DB: opt } + volumes: ["pgdata:/var/lib/postgresql/data"] + redis: + image: redis:7 + api: + build: ../services/api + command: uvicorn orchestrator.main:app --host 0.0.0.0 --port 8000 + environment: [DATABASE_URL, R2_*, STRIPE_*, RESEND_API_KEY, INTERNAL_TOKEN] + depends_on: [postgres, redis] + worker: + build: ../services/api + command: python -m workers.run # or celery -A workers worker + environment: [DATABASE_URL, R2_*] + depends_on: [postgres, redis] + nocodb: + image: nocodb/nocodb + environment: { NC_DB: "pg://postgres:5432?u=opt&p=${PG_PASSWORD}&d=opt", NC_PUBLIC_URL: "https://data.0pt.app" } + depends_on: [postgres] + metabase: + image: metabase/metabase + environment: { MB_DB_TYPE: postgres, MB_DB_HOST: postgres, MB_DB_USER: opt, MB_DB_PASS: ${PG_PASSWORD}, MB_DB_DBNAME: opt } + depends_on: [postgres] + caddy: + image: caddy:2 + ports: ["80:80","443:443"] + volumes: ["./Caddyfile:/etc/caddy/Caddyfile", "caddydata:/data"] +volumes: { pgdata: {}, caddydata: {} } +``` + +## Environment variables (full inventory) + +``` +# Postgres +DATABASE_URL=postgres://opt:***@host:5432/opt +PG_PASSWORD=*** +# Internal auth (Next ↔ FastAPI) +INTERNAL_TOKEN=*** # sent as X-Internal-Token +# Auth.js +AUTH_SECRET=*** +AUTH_URL=https://0pt.app +# Stripe +STRIPE_SECRET_KEY=sk_*** +STRIPE_WEBHOOK_SECRET=whsec_*** +NEXT_PUBLIC_STRIPE_PUBLISHABLE_KEY=pk_*** +# Cloudflare R2 (S3-compatible) +R2_ACCOUNT_ID=*** R2_ACCESS_KEY_ID=*** R2_SECRET_ACCESS_KEY=*** R2_BUCKET=opt-files R2_ENDPOINT=https://.r2.cloudflarestorage.com +# Email +RESEND_API_KEY=re_*** +# Embeds (signed URLs) +METABASE_SITE_URL=https://dash.0pt.app METABASE_SECRET_KEY=*** +NOCODB_BASE_URL=https://data.0pt.app NOCODB_API_TOKEN=*** +# Encryption for db_grants +GRANT_ENC_KEY=*** # 32-byte key, AES-GCM +# App +NEXT_PUBLIC_API_URL=https://api.0pt.app +``` + +Never hardcode. Vercel env for `apps/web`; VPS `.env` (git-ignored) for compose. Provide `.env.example` with empty values. + +## Security & compliance (build in, not bolt on) + +- **Customer DB grants**: customer creates a read-only, table-scoped, time-boxed DB role; we store the connection string **AES-GCM encrypted** (`GRANT_ENC_KEY`), validate it's read-only, and **delete after the job**. IP-allowlist; prefer a replica. Log access in `audit_log`. +- **File delivery**: short-lived **presigned R2 URLs**; never public buckets. +- **Isolation**: API filters by `org_id`; Postgres **RLS** where practical; NocoDB/Metabase exposed only via **signed embed URLs** scoped per customer/order (no separate login surfaced). +- **PII (contacts/UBO)**: lawful basis recorded per delivery; honor deletion; minimize retention. +- **DPA**: ship a Data Processing Agreement; provenance log + audit log are part of the offering. +- **Secrets**: rotate; least privilege; no secrets in client bundles (only `NEXT_PUBLIC_*`). + +## Embedding NocoDB / Metabase (so it feels like one app) +- Generate a **signed embed URL** server-side (Metabase signed embedding with `METABASE_SECRET_KEY`; NocoDB shared/locked view + token) scoped to the customer's data, and render it in an iframe inside the mono chrome. The customer never sees a NocoDB/Metabase login. + +## CI/CD (.github/workflows) +- **web**: lint + typecheck + build → deploy to Vercel on push to `main` (Vercel Git integration). +- **api**: lint (ruff) + tests (pytest) → build Docker image → deploy to VPS (SSH `docker compose pull && up -d`, or a small runner). +- Run DB migrations as a deploy step. + +## Monitoring (cheap) +- Vercel Analytics + Speed Insights (web). +- **Sentry** free tier (web + api). +- Uptime ping on `/health`. +- Worker/job health visible in admin (reads `jobs`). diff --git a/docs/09-build-plan.md b/docs/09-build-plan.md new file mode 100644 index 0000000..f3ef4cf --- /dev/null +++ b/docs/09-build-plan.md @@ -0,0 +1,75 @@ +# 09 · Build plan + +Phased, in dependency order. **Finish a phase to its acceptance criteria before the next.** Each phase = small, reviewable commits. Reconcile with existing code rather than rewriting wholesale. + +## Phase 0 — Foundation & repo +- Monorepo structure (`docs/08`). Migrate existing `src/orchestrator` + `src/workers` into `services/api`. +- Postgres schema + migrations (`docs/03`); `pgvector` enabled. `.env.example`. +- `docker-compose` up (postgres, redis, api, worker, nocodb, metabase, caddy). `/health` green. +- CI skeleton (lint/typecheck/build). +- **Accept:** `docker compose up` runs the whole stack locally; migrations apply; `/health` returns ok for db + queue. + +## Phase 1 — Design system in code +- Tailwind + shadcn; `globals.css` (grain + reveal keyframes) and `useReveal` verbatim (`docs/05`). +- Build **DataArtifact** primitive (table/map/schema/chart/grid) and **ProvenanceBadge** (`docs/design` prompts 5.1, 2.11). +- **Accept:** a sample page renders mono tokens, a DataArtifact animates on scroll, reduced-motion respected, focus rings visible. + +## Phase 2 — Marketing + catalog (placeholder data) +- Landing components top-to-bottom (build-brief 2.1–2.10): Header, Hero, Philosophy, Catalog teaser, Engine, Visualisation gallery, Coverage, Editorial, Proof, Footer. +- Full **Catalog** page + **Dataset detail** + **Sourcing-map** page (coverage doc 5.4). Postgres full-text search. +- All with loading/empty/error states + mobile. +- **Accept:** marketing site + catalog browse work end-to-end on placeholder catalog rows; one continuous scroll reads well; DoD met per screen. + +## Phase 3 — Auth & portal shell +- Auth.js + Postgres adapter; sign in/up/verify/forgot. `orgs`/`users`. +- Portal shell + Dashboard ("Mina beställningar") with empty state. +- **Accept:** a user can sign up, land in an empty dashboard, sign out; routes protected. + +## Phase 4 — Order lifecycle end-to-end (test mode) +- **Configurator** (2.12) wired to `POST /estimates` (real estimate math in FastAPI). +- **Deposit checkout** (2.13) → Stripe **test** Checkout → webhook → order `IN_PRODUCTION`. +- **Production tracker** (2.14) reading `GET /orders/{id}/events` (SSE) — pipeline may be **simulated** at this phase. +- **Validation & delivery** (2.15): tabs render from a sample delivery; **accept** → `POST /payments/release` (test) → `DELIVERED`; **revise** leaves escrow untouched. +- AI-assist (`POST /ai/parse`) prefilling the configurator. +- **Accept:** a full custom order can be configured, deposit paid (test), watched, validated, accepted, and "delivered" — state machine in `docs/01` traversed; escrow split correct. + +## Phase 5 — Real data engine (Nordic slice) +- Compliant connectors first: **GLEIF + Brreg + Companies House** (verify access terms; record in `sources.yaml`). Remove any FlareSolverr/bypass path (`docs/06`). +- Pipeline stages produce real `dim_company` + `fact_financials` with `bridge_provenance`; QA threshold drops low-confidence rows. +- Production tracker now reflects **real** job metrics. +- **Accept:** a Nordic company dataset is produced for a real order, every field has provenance, quality report generated, level-0 fields shown honestly (not fabricated). + +## Phase 6 — Delivery surfaces & integrations +- Exports: CSV / Parquet / JSON via R2 presigned URLs. +- Embedded **NocoDB** ("Min data") and **Metabase** dashboards via signed URLs in mono chrome. +- Integrations: **FiveCRM push**, webhooks, Google Sheets. +- File **uploads** (Uppy → R2) for matching/enrichment. +- **Accept:** a delivered dataset is downloadable, viewable in embedded NocoDB + Metabase (no separate login), and pushable to FiveCRM; uploads work. + +## Phase 7 — Subscriptions / feeds +- `subscriptions`; recurring jobs (weekly/daily) delivering **deltas**; webhooks on change. +- Freshness monitoring; Resend emails (confirmation, "din data är klar", feedback). +- **Accept:** an order converts to a feed; a scheduled run produces a delta + fires a webhook + email. + +## Phase 8 — Admin, hardening, compliance +- Admin: order queue + scope/quote + escrow release; **source registry & freshness** monitoring; worker/job monitoring; customer + catalog management. +- Customer **DB-access grant** flow (encrypted, read-only, time-boxed) + revoke. +- Security pass: RLS, presigned-only files, signed embeds, secret rotation, `audit_log` everywhere; **DPA** published; legal pages. +- Finalize all loading/empty/error states + mobile across the app. +- **Accept:** admin can run the business; DB-grant flow works and self-revokes; security checklist passes; DoD met app-wide. + +## Phase 9 — Launch +- Domain `0pt.app` (Vercel) + subdomains (api/data/dash via Caddy). Sentry + uptime live. +- Soft launch: Protosell internally + a handful of existing dealer relationships. +- **Accept:** a real external customer completes an order start→delivery in production; feedback loop running. + +--- + +## Working conventions (for Claude Code) +- Tech docs + identifiers **English**; UI copy **Swedish**. +- Conventional Commits; one concern per PR; keep diffs reviewable. +- No secrets in code; update `.env.example` when adding env. +- Add/extend tests for engine logic (estimate math, fusion/confidence, dedup) and API contract. +- When a source's access/licence is unclear, **verify and record in `sources.yaml`** before relying; if disallowed for bulk, route to licensed/partner or mark level 0 — never bypass, never fabricate. +- Touch the design tokens/motion only via `docs/05`; keep red ≤ once per screen; no photography. +- If anything here conflicts with existing code, **flag it** in the PR description and propose the reconciliation rather than guessing.