0pt/docs/02-architecture.md

02 · Architecture

Known issues affecting this doc, in brief — full detail + evidence in README.md's "Known issues" section: (1) no auth/session system exists yet, so several newer routes trust client-supplied tenant_id (see "Known gaps" below); (2) the CL0UD/N0D/AUD0/ST0RE platform clients are real code but mock-only in this repo; (3) docs/ICEBERG_WRITE_BOUNDARY.md / docs/11-lakehouse-write-plan.md / docs/P0_QUERY_SOURCE_FLOW.md reference several src/services/quality-* and sql/002/sql/003 files that were never actually committed to this repo (out of this doc's file scope to fix — flagged here so it isn't silently repeated); (4) two git remotes (github.com/sax3l/0pt and git.siax.io/sax3l/0pt) have diverged and it is unresolved which is authoritative — Simon's call, not assumed here.

Principle: one database, many lenses

Postgres is the source of truth. Everything else is a lens on it. A tool enters the stack only if it (a) reads that Postgres and (b) sits behind the same auth. This is the difference between an app and a pile of separate logins.

This principle describes — and is largely realized by — the original edge + engine plane: marketing, catalog, order lifecycle, and the FastAPI engine. It does not describe the newer P0 data-platform surface (app/api/0pt/*), which is real, additive, and deliberately does not go through this same path. See "Two API surfaces" below.

                 ┌──────────────────────────────────────────────┐
   COLLECTION    │                  POSTGRES                     │   LENSES (customer + admin)
  (legal engine) │  dim_company · dim_contact · dim_vehicle      │
 ─────────────── │  fact_financials · dim_ubo                    │ ─────────────────────────────
  Official APIs  │  dim_source · bridge_provenance               │   Next.js portal (Vercel)
  /bulk  ───────▶│  orders · order_items · jobs · deliveries     │◀──  Catalog + order lifecycle
  Authorized web │  subscriptions · users · orgs · files         │   NocoDB  (embedded "Min data")
  Licensed data  │  db_grants · audit_log · pgvector embeddings  │   Metabase (embedded dashboards)
  Customer upload│                                               │   Export: CSV/Parquet/JSON/API/FiveCRM
                 └───────────────────▲──────────────────────────┘
                                     │
                        services/api — FastAPI orchestrator + workers
              (estimates, orders, jobs, AI-assist, webhooks, pipeline)

Components & responsibilities

• Next.js (Vercel) — all customer/admin UI: marketing site, catalog, configurator, checkout, production tracker, validation, portal, admin. It also hosts two distinct families of API route handlers under app/api/: the original order/catalog/estimate surface (some of which defer to the FastAPI engine), and the newer P0 data-platform surface under app/api/0pt/* (does not defer to the engine at all — see below).
• FastAPI orchestrator (services/api/orchestrator/main.py) — REST contract in docs/04-api.md: estimates, orders, sources, vehicle lookup, sanctions screening, job run/status. Not on Vercel (serverless timeouts kill long jobs). Correction: earlier drafts of this doc pointed at src/orchestrator; that was the pre-migration location. docs/09-build-plan.md Phase 0 called for moving it into services/api, and that move has happened — the code lives at services/api/orchestrator today.
• Workers (services/api/workers/) — adaptive router (adaptive_router.py, verified: chooses only among compliant methods — official API → bulk → authorized web → licensed, no bypass/stealth/proxy-rotation tier exists in it, by design) + pipeline stages (pipeline.py, queue.py, resolve.py, run.py). See docs/06-data-engine.md.
• Postgres — backbone (docs/03-data-model.md) incl. pgvector for matching/embeddings, for the original edge + engine plane. Wired via lib/db.ts (env-guarded: DATABASE_URL/POSTGRES_URL; falls back to an in-memory, non-durable store when unset, e.g. lib/orders-store.ts).
• NocoDB / Metabase — both are defined as services in infra/docker-compose.yml; embedding them into the portal as "Min data" / customer dashboards is the target, not yet confirmed wired into a portal page in this pass.
• Cloudflare R2 — file storage (uploads + deliverables) via presigned URLs. Env-guarded (app/api/uploads/route.ts checks R2_ACCESS_KEY_ID/R2_BUCKET/R2_ENDPOINT and returns 501 until set); the presign call itself is still a stub (put_url: null) even when the env vars are present — not wired end-to-end yet.
• Stripe — deposit capture + escrow balance release. This one is wired: app/api/payments/[order_id]/deposit/route.ts and .../release/route.ts are real Stripe Checkout calls (env-guarded on STRIPE_SECRET_KEY, 501 until configured). See the correction in docs/01-product.md's business model section for exactly which escrow mechanism is implemented.
• Resend — documented as the transactional-email provider (docs/08-infrastructure.md) but has no call site anywhere in this repo yet. Treat "+ email" as not-yet-implemented, not merely unconfigured.
• Auth.js (NextAuth) with a Postgres adapter — this is the target, listed here as a stack decision, not a built component. What exists today is a lightweight signed-session scaffold (lib/auth.ts: HMAC-SHA256 via Web Crypto, no external deps) behind a guarded dev-only login gate (DEV_LOGIN_KEY). There is no real session-to-org/tenant binding anywhere in this repo. Full Auth.js + OAuth is Phase 3 of docs/09-build-plan.md and has not been built. This matters directly for the tenant-scoping gap described below.

Two API surfaces

This repo currently contains two real, distinct API surfaces, not one. Earlier versions of this doc described the edge as a uniform thin proxy to the FastAPI engine — that was true for part of the original surface and remains true for that part, but a second, newer surface was added that is a deliberate, ADR-documented exception to it. Conflating the two is the single biggest way to misread this codebase's architecture.

1. The original edge → engine surface

A subset of the original app/api/* route handlers defer to the FastAPI engine (services/api) via lib/engine.ts's engineFetch()/hasEngine() bridge when ENGINE_URL is set, and fall back to local computation when it isn't:

• app/api/estimates/route.ts — defers to POST {ENGINE_URL}/estimates, falls back to lib/data.ts's computeEstimate.
• app/api/vehicles/route.ts, app/api/screen/route.ts, app/api/ai/parse/route.ts — same pattern.

Other original routes are not engine-backed at all today: app/api/orders/* reads/writes lib/orders-store.ts directly (Postgres when configured, else in-memory); app/api/catalog/route.ts returns a hardcoded static array. Both of these are entirely edge-local — there is no engine call in either path currently, not "thin proxy with a local fallback."

This surface is documented in full in docs/04-api.md and matches the "one Postgres, many lenses" principle above (once DATABASE_URL is set).

2. The P0 data-platform surface (app/api/0pt/*)

A second, newer surface implements dataset registration, Iceberg lakehouse table mapping/writes, per-source rate-policy management, bounded synchronous source querying, and mailbox-based document capture — directly in the edge app, backed by src/services/* and src/clients/*. It does not call lib/engine.ts/ENGINE_URL and does not go through the FastAPI engine at all. This is deliberate: it is the concrete implementation of ADR-0030 (Apache Iceberg as SIAX's canonical lakehouse table format, authored in the n0d repo) and the ownership split it defines. See docs/10-lakehouse-decision.md (the pointer to the ecosystem decision), docs/11-lakehouse-write-plan.md (the plan), and docs/ICEBERG_WRITE_BOUNDARY.md (the implementation record) for the full boundary — this doc only summarizes it.

Real, existing routes today (verified against the filesystem, not against app/api/0pt/route.ts's own advertised list — see the note below):

Route	Backed by
GET /api/0pt	static index (see note below)
POST/GET /api/0pt/documents/capture	src/services/document-capture-service.ts
.../documents/{documentId}/extraction	src/services/document-extraction-service.ts
POST/GET /api/0pt/lakehouse/mappings	src/services/dataset-iceberg-mapping-service.ts, iceberg-write-service.ts
.../lakehouse/mappings/{mappingId}	same
POST .../lakehouse/mappings/{mappingId}/writes	src/services/iceberg-write-service.ts#requestIcebergWrite
POST/GET /api/0pt/sources/rate-policies	src/services/source-rate-policy-service.ts
.../sources/rate-policies/{sourceRatePolicyId}	same
POST /api/0pt/sources/{sourceId}/query	src/services/source-query-service.ts (bounded, timeout-guarded ad-hoc SQL against a registered database source — see docs/P0_QUERY_SOURCE_FLOW.md)

Code-level inconsistency worth knowing about: app/api/0pt/route.ts's own GET handler advertises an endpoints list including /api/0pt/sources, /api/0pt/datasets, /api/0pt/data-contracts, /api/0pt/quality/checks, /api/0pt/quality/reports, /api/0pt/features/views — none of these exist as routes (confirmed by listing app/api/0pt/ on disk); the lakehouse-mapping and rate-policy routes that do exist aren't listed there either. This is the same "described but never built" pattern as the quality-check-service.ts fabrication flagged below — this doc's route table above reflects what is actually on disk, not what the index endpoint claims.

In-memory scope: every P0 service (DatasetService, SourceService, DatasetIcebergMappingService, SourceRatePolicyService, DocumentCaptureService, …) holds its state in a plain Map inside the Next.js route module that instantiates it. Most route modules construct a fresh instance per module, so a record created via one route is not visible to a different route module in the same running process (each file's own header comment documents this; document-capture-service.ts worked around it for its own two routes with a shared singleton export). A Postgres migration exists (sql/004_iceberg_write_boundary.sql) but is not wired into any route handler yet. So the "one database, many lenses" principle above does not yet hold for this surface.

Six-surface capability model & 0pt's current coverage

SIAX Cloud's target model (defined in cl0ud, docs/architecture/SIAX_CLOUD_UNIFIED_ARCHITECTURE.md) is that one capability can expose up to six surfaces: REST api, cli (siax 0pt ...), mcp (an agent tool), ui (a page/panel), job (an async N0D job type), and sdk (a generated client method). The capability registry's own Surface type (cl0ud packages/capability-registry/src/types.ts) currently encodes five of those six as a declarative field ('api' | 'cli' | 'mcp' | 'ui' | 'job'); sdk is generated separately, not yet a registry field.

0pt's actual coverage today (verified against cl0ud packages/capability-registry/src/registry.ts): 14 distinct 0pt.* capabilities are registered —

0pt.query_source, 0pt.document_ocr, 0pt.write_external_record, 0pt.lakehouse_mapping_manage, 0pt.lakehouse_write, 0pt.source_rate_policy_manage, 0pt.screen_sanctions, 0pt.estimate_compute, 0pt.vehicle_lookup, 0pt.catalog_list, 0pt.source_registry_list, 0pt.order_create, 0pt.order_payment, 0pt.write_iceberg_table

— and, checked entry by entry:

• 11 declare surfaces: ['api'] only.
• 2 (0pt.lakehouse_write, 0pt.write_iceberg_table) declare surfaces: ['api', 'job'].
• 1 (0pt.write_external_record) declares surfaces: ['job'] only — apiEnabled: false, and its own comment states it has "NO dedicated HTTP route in the 0pt app yet"; it exists as a tested TS service (src/services/external-writeback-service.ts) invoked only via direct import today, not a live callable path.
• Every single one has cliEnabled: false and mcpEnabled: false. None has a ui surface entry either.

In plain terms: 0pt is API-only today (plus two job-backed writes, and one job-only capability with no live route at all). If any doc — inside or outside this repo — describes a live siax 0pt ... CLI command or an 0pt.* MCP tool, it is describing a target, not current reality; correct it on sight.

Platform clients: CL0UD / N0D / AUD0 / ST0RE

src/clients/{cl0ud,n0d,aud0,st0re,st0re-lakehouse}-client.ts are real, typechecked TypeScript with accompanying tests (tests/p0/*), each following the same shape: a thin interface, an HTTP implementation written against the target service's real, committed route contract, and a deterministic in-memory mock used whenever no baseUrl is supplied to the constructor.

Verified this pass: none of the five client files, and none of their call sites in src/services/*, read CL0UD_URL, N0D_URL, AUD0_URL, or ST0RE_URL from process.env anywhere (grep -rn "process.env" src/clients src/services turns up nothing for any of those four names outside comments/TODOs). Every constructor call in this repo (createCl0udClient(opts?.cl0udBaseUrl), etc.) is invoked with no baseUrl argument, so every one of these clients runs against its in-memory mock today, in every code path that exists in this repo. Read this as "real code, mock-only, not live-verified" — not "integrated" or "connected" — wherever platform-client status comes up.

What each client is for, concretely:

• CL0UD — policy evaluation (evaluatePolicy). iceberg-write-service.ts and external-writeback-service.ts call it before acting; the mock always returns "allow".
• N0D — job requests (createJob). document-capture-service.ts requests document-extraction jobs; iceberg-write-service.ts requests heavy-write/compaction jobs. 0pt only requests N0D jobs — it never executes them, per docs/ICEBERG_WRITE_BOUNDARY.md's "N0D executes" boundary. No evidence supports a claim that N0D has a dedicated-node ("t_0pt") auth pattern already built for this: n0d's repository was searched in full (all branches, all commits, all files) and contains zero references to t_0pt. Treat any such claim as unconfirmed until proven otherwise.
• AUD0 — audit/lineage events (ingestEvent, ingestLineageEvent). Every mutating P0 service emits these; the mock is a true no-op, matching AUD0's own documented stance that it has no live ingestion endpoint yet.
• ST0RE / ST0RE-lakehouse — artifact-metadata registration and Iceberg table/namespace/snapshot registration, respectively. st0re-lakehouse-client.ts's HTTP path is written against sax3l/st0re's actual shipped route contract (commit 9ea0abb, read directly during a prior pass) — real code, but not executed against a running ST0RE in this repo's test environment either.

Known gaps

tenant_id is not server-bound anywhere in the P0 surface

The platform-wide rule is that tenant_id must be bound to an authenticated session server-side and never trusted from client input. This repo violates that rule today, and it is not a deliberate two-tier design — it's a direct consequence of there being no auth/session system at all yet (see the Auth.js note above; app/api/orders/route.ts's own GET handler literally comments // org scoping added with auth, i.e. not yet).

Concretely: tenant_id/project_id are accepted straight from the client-supplied request body (or query string) at every P0 mutation entry point —

• app/api/0pt/lakehouse/mappings/route.ts (POST)
• app/api/0pt/sources/rate-policies/route.ts (POST)
• app/api/0pt/sources/[sourceId]/query/route.ts (POST)
• app/api/0pt/documents/capture/route.ts → src/services/document-capture-service.ts#captureFromMailbox

app/api/0pt/lakehouse/mappings/[mappingId]/writes/route.ts doesn't read tenant_id from its own body — but it resolves the tenant by looking up mapping.tenant_id from a mapping that was itself created via one of the unauthenticated routes above, so the trust boundary problem is inherited, not avoided.

Do not expose any of these routes to untrusted callers before Phase 3 of docs/09-build-plan.md (Auth.js + org/session binding) lands. This is Simon's call on sequencing, not something to silently patch in a docs pass.

0pt's data-quality/QA stage is a fabrication in sibling docs, not built

Flagged so it is not repeated: src/services/quality-check-service.ts, src/services/quality-report-service.ts, src/types/quality-check.ts, src/types/quality-report.ts, src/data-platform/services/catalog-service.ts, sql/002_p0_dataset_quality.sql, and sql/003_*.sql were at one point referenced in docs/ICEBERG_WRITE_BOUNDARY.md, docs/11-lakehouse-write-plan.md, and docs/P0_QUERY_SOURCE_FLOW.md as if already built and tested. Verified this pass: git log --all for every one of those paths returns zero commits, on any branch, ever — sql/ on disk today contains only 001_init.sql and 004_iceberg_write_boundary.sql. Those files have never existed in this repository. 0pt's data-quality/QA enforcement is spec/target only — a real, open build gap, not a doc typo. All three of the docs above have since been corrected (verified this pass by reading each in full) — none currently asserts those files as real; each carries its own explicit "fabricated, not built" callout, and docs/P0_DATASET_QUALITY_FLOW.md is the honest target-spec replacement.

Two diverged git remotes

origin in local checkouts of this repo is github.com/sax3l/0pt. A separate Gitea remote, git.siax.io/sax3l/0pt, shares at least part of the same history (services/api/workers/adaptive_router.py is byte-for-byte identical between GitHub's current main and a Gitea post-fix PR branch) but independently accumulated commits GitHub's main never received — including, at one point, a WAF-bypass/proxy-rotation/fake-identity scraper stack that directly violated the CLAUDE.md §1 sourcing rule (since removed on Gitea via a separate PR). Which remote is authoritative, which is actually deployed to 0pt.app, and whether the two should be kept in sync is unresolved and is Simon's decision — do not assume origin/GitHub is the sole copy of this codebase, here or in any other doc.

Stack decisions — IN

Concern	Choice	Why
Frontend	Next.js App Router + TS + Tailwind + shadcn/ui	Vercel-native, matches existing skills
Backend/jobs	FastAPI + workers (services/api)	already built; long-running jobs
DB	Postgres (+ pgvector)	backbone; Neon free tier or on-VPS
Airtable layer	NocoDB	sits on existing tables, no duplication
BI	Metabase	customer-facing embedding, simple
Files	Cloudflare R2 + presigned URLs	cheapest S3-compatible
Payments	Stripe Checkout + webhook → Next.js API	~1 day, no SaaS boilerplate needed
Auth	Auth.js + Postgres adapter	one system — target, not yet built (see Components above)
Email	Resend	best Next.js integration, free tier — not yet wired (no call site in this repo)
Quality	Great Expectations / checks inside workers	scores shipped with each order — not yet built for the P0 surface (see "Known gaps" above)

Stack decisions — OUT (do not add)

Rejected	Reason
Superset	Metabase already covers BI — one tool, not two
DataHub / OpenMetadata	enterprise-heavy; catalog = a Postgres table in Next.js + NocoDB. Add lineage only if an enterprise customer demands it
Baserow / Teable	NocoDB chosen — not three Airtable clones
Meilisearch (v1)	Postgres full-text suffices to a few thousand rows; add later
Delta Sharing (v1)	unnecessary; customer pull via scoped role + exports
Trigger.dev	the existing queue already schedules jobs; Resend covers mail
FlareSolverr / stealth browsers / undetected-* / proxy-fingerprint rotation / fake-account creation	prohibited — see CLAUDE.md §1. Replace with compliant fetching. (The Gitea-only scraper stack described under "Two diverged git remotes" above is exactly this — removed there, never present on GitHub main.)

Hosting (cheapest coherent)

• Vercel (Hobby/free) — Next.js + custom domain 0pt.app.
• One small VPS (e.g. Hetzner ~€5/mo) running docker-compose (infra/docker-compose.yml): FastAPI, workers, NocoDB, Metabase, (Postgres or use Neon), optional Meilisearch later. Per-service managed platforms (Railway) get expensive once you have 4–5 always-on services; a single VPS + Vercel + R2 lands ≈ <€10/mo to start.
• Cloudflare R2 — object storage (env-guarded, presign not yet wired — see Components above).
• Postgres — on the VPS, or Neon free tier if you want it managed.

Full config, env, security, CI/CD: docs/08-infrastructure.md. Env var inventory: .env.example exists on branch/PR feature/0pt-env-example (unmerged as of this pass) — until it merges to main, docs/08-infrastructure.md is the source of truth for env var names.