admin/0pt
5
0
Fork 0
Code Issues 10 Pull Requests 1 Actions Packages Projects Releases Wiki Activity
Files
ebffb7df86ee09d0cb0b22fa523e4a4d697c87bc
0pt/docs/07-dependencies.md
T
adminandClaude Sonnet 5 2902a26bac docs: full accuracy overhaul, grounded claim-by-claim in real code (#4) 
* docs: full accuracy overhaul, grounded claim-by-claim in real code

Every doc in this repo (CLAUDE.md, README.md, SECURITY.md, docs/*.md) was
audited and rewritten by a multi-pass process: independent rewrite per
cluster, adversarial re-verification of that rewrite against the actual
code, then a final cross-doc consistency pass — each stage re-checking
claims against the repo directly rather than trusting the previous stage.

Two corrections matter most:

1. docs/ICEBERG_WRITE_BOUNDARY.md, docs/11-lakehouse-write-plan.md, and
   docs/P0_QUERY_SOURCE_FLOW.md asserted src/services/quality-check-service.ts,
   quality-report-service.ts, src/data-platform/services/catalog-service.ts,
   and sql/002_p0_dataset_quality.sql/sql/003_*.sql as real, tested code.
   git log --all across every branch shows zero commits for any of these
   paths, ever. 0pt's data-quality/QA enforcement does not exist in code.
   Corrected everywhere, with a new docs/P0_DATASET_QUALITY_FLOW.md added
   as an explicit target-spec (not-yet-built) replacement.

2. github.com/sax3l/0pt (this repo) and git.siax.io/sax3l/0pt (Gitea) are
   two diverged remotes sharing part of the same history (services/api/
   workers/adaptive_router.py is byte-identical between them). Gitea's main
   independently accumulated a WAF-bypass/proxy-rotation scraper stack in
   three 2026-07-13 commits, since removed via a Gitea PR; GitHub's main
   never had it. Which remote is canonical/deployed is unresolved — flagged
   consistently across the corpus, not decided here.

Also newly and consistently documented: the tenant_id/project_id-read-from-
client-body gap on the P0 lakehouse/rate-policy/source-query routes (no
auth/session system exists anywhere in this repo yet); that every CL0UD/
N0D/AUD0/ST0RE platform client is real but mock-only (zero process.env
reads for any of their *_URL vars); that two P0 mutation routes
(lakehouse/mappings, sources/rate-policies) always fail over HTTP today
because their prerequisite entity can never be created via any route; a
schema bug in sql/004 (FK references to opt_datasets/opt_sources, which
don't exist); and that 0pt.app's live deployment was independently
confirmed (curled this pass) to be running this exact codebase with
DATABASE_URL/REDIS_URL unset in production.

New file: docs/12-edge-capability-api.md — the edge-native P0 TypeScript
capability contract (app/api/0pt/*, /api/screen, /api/ai/parse), previously
undocumented and distinct from docs/04-api.md's FastAPI-engine-only scope.

Verified before commit: tsc --noEmit clean; vitest run 88 passed, 1 todo,
0 failures (13 files) — docs-only change, no source touched.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* docs: reflect PR #5's lakehouse-mapping-registry + status-code fixes

Follow-up to keep docs/12-edge-capability-api.md accurate once both PRs
land: fix/0pt-full-bug-hunt (#5) fixed the shared-instance bug in the
lakehouse mapping routes and added proper status-code mapping to the
document extraction route. This doc was written against the pre-fix
state in the same pass that added it, so it needs the same correction.

Does not claim the lakehouse write path fully works now -- the deeper
dataset-registration gap (no route ever calls DatasetService.registerDataset)
is unchanged and still blocks it end-to-end; only the shared-instance half
of the bug is fixed.

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-22 18:49:01 +02:00

80 lines
12 KiB
Markdown
Raw Blame History

# 07 · Dependencies & repos to import
Curated. Each entry: what it is, where it slots in, how to install. **Do not import on reputation/star-count** — several items in earlier brainstorming were unverified or hallucinated (see Exclusions). Verify each repo exists and is maintained before adopting.
> **This is a plan, not a status report.** Every table below lists what *should* be imported for the target
> architecture in `docs/06-data-engine.md`. A **Status** column has been added to each table (verified this
> pass against `services/api/requirements.txt` and root `package.json` — the actual, current dependency
> manifests, read in full). Read the status column before assuming a package is already wired in; most of the
> data-engine tier (OCR, NER, quality checks, orchestration) is still **not installed**.
## Frontend (Next.js on Vercel)
| Package | Purpose | Install | Status |
|---|---|---|---|
| next, react, typescript | App Router base | `npx create-next-app@latest --ts` | **Installed** — `next@16.0.10`, `react@19.2.0`, `typescript@^5` in `package.json` |
| tailwindcss | styling (mono tokens) | per Next setup | **Installed** — `tailwindcss@^4.1.9` |
| shadcn/ui | base components | `npx shadcn@latest init` | **Installed via copy-in, not as a runtime dep** — no `shadcn` package in `package.json` (expected: the shadcn CLI copies component source into `components/ui/`); the underlying `@radix-ui/react-*` primitives it generates against are installed individually and present |
| @tanstack/react-table **or** ag-grid-react | data tables / catalog grid | `npm i @tanstack/react-table` | **Not installed** — neither package present |
| @uppy/core @uppy/react @uppy/dashboard @uppy/aws-s3 | file upload → R2 (presigned) | `npm i @uppy/core @uppy/react @uppy/dashboard @uppy/aws-s3` | **Not installed** |
| next-auth (Auth.js) + @auth/pg-adapter | single auth, Postgres-backed | `npm i next-auth @auth/pg-adapter` | **Not installed.** Matches a broader, confirmed gap: there is no auth/session system anywhere in this repo yet (`app/api/orders/route.ts`'s own `GET` handler has the literal comment "org scoping added with auth" — i.e. not yet). Several `app/api/0pt/*` routes currently read `tenant_id`/`project_id` straight from the request body as a result — a real, open security gap, not a deliberate design; see `docs/09-build-plan.md` Phase 3 for the planned fix (not built anywhere yet, Simon's call on design). |
| stripe / @stripe/stripe-js | checkout + portal | `npm i stripe @stripe/stripe-js` | **Partially installed** — `stripe@^22.2.3` (server SDK) is present; `@stripe/stripe-js` (client SDK) is not |
| resend / react-email | transactional email | `npm i resend @react-email/components` | **Not installed** |
| lenis *(optional)* | smooth scroll (keeps all effects) | `npm i lenis` | **Not installed** |
| recharts *(only if needed)* | charts inside DataArtifact/dashboards | `npm i recharts` | **Installed** — `recharts@2.15.4` |
> Catalog/search in v1 = Postgres full-text. Add Meilisearch later only past a few thousand rows.
## Backend / workers (Python, FastAPI — existing)
`services/api/requirements.txt` currently pins exactly 9 packages. Everything else in this table is still to
be added — this is the single biggest gap between this doc's plan and the running code (see
`docs/06-data-engine.md`'s "Pipeline stages" section for what that means stage-by-stage: only official-API
sourcing and rapidfuzz-based dedup are real today; extraction/OCR, NER, and quality-checking have no code at all).
| Package | Purpose | Install | Status |
|---|---|---|---|
| fastapi, uvicorn, pydantic | API (existing) | `pip install fastapi uvicorn pydantic` | **Installed** — `fastapi==0.115.6`, `uvicorn[standard]==0.34.0`, `pydantic==2.10.4` |
| httpx | polite HTTP fetch (identifies itself, obeys rate limits) | `pip install httpx` | **Installed and actively used** — `httpx==0.28.1`, every connector in `sources/connectors.py` and `sources/financials.py` goes through it |
| pyyaml | parse `sources.yaml` | `pip install pyyaml` | **Installed and used** — `pyyaml==6.0.2`, loaded once in `orchestrator/main.py` |
| rapidfuzz | dedup / fuzzy matching | `pip install rapidfuzz` | **Installed and used** — `rapidfuzz==3.11.0`, the only matching engine in `workers/resolve.py` |
| python-dotenv, tenacity | env loading, retry helper | — | **Installed**, `python-dotenv==1.0.1` / `tenacity==9.0.0` — present in `requirements.txt` but not yet visibly imported in the worker/orchestrator/sources files read this pass; not confirmed wired up |
| sqlalchemy / asyncpg | Postgres access (align `db.py`) | `pip install sqlalchemy asyncpg` | **`asyncpg==0.30.0` is installed but unused.** No file in `services/api` imports it or issues a query — confirmed by grep. `orchestrator/main.py` reads `DATABASE_URL` once, only to report a boolean in `/health`. Orders/jobs live in in-memory dicts that reset on restart. `sqlalchemy` is **not installed** at all. There is also no `db.py` anywhere in this repo — an older doc reference to `src/workers/db.py` is stale; that path does not exist. |
| alembic | migrations | `pip install alembic` | **Not installed.** Migrations currently ship as plain numbered SQL files in `sql/` (`001_init.sql`, `004_iceberg_write_boundary.sql`), applied ad hoc, not via Alembic. |
| celery + redis **or** existing adaptive_router/queue | job queue/concurrency | `pip install celery redis` | **Neither Celery nor a Python Redis client is installed.** `workers/queue.py` is a small in-process `asyncio.Queue` wrapper (single-worker dev only, by its own docstring). `infra/docker-compose.yml` does run a `redis` *container*, but nothing in `services/api` currently talks to it. |
| scrapy **or** crawlee | orchestrate **authorized** fetches | `pip install scrapy` | **Not installed.** No HTML-scraping library of any kind (also no `bs4`/`lxml`) exists in this codebase — confirmed by grep. Tier-2 "authorized public web" sourcing (docs/06) has no code path yet; only tier-1 official-API connectors exist. |
| playwright | public JS-rendered pages, politely | `pip install playwright && playwright install chromium` | **Not installed.** No `browser_worker.py` file exists anywhere in this repo either — an older doc reference to reconciling a FlareSolverr stage inside it is moot; there's nothing to reconcile in the current Python engine (see `docs/06-data-engine.md`'s Known Issue callout for where a bypass stack *did* briefly exist, on a different git remote). |
| unstructured | parse PDF/HTML/DOCX/images → elements | `pip install "unstructured[all-docs]"` | **Not installed.** |
| paddleocr / pytesseract | OCR cascade | `pip install paddleocr pytesseract` (+ tesseract bin) | **Not installed.** No OCR code or `ocr_worker` module exists anywhere in this repo. |
| spacy (+ models) / transformers | NER, entity resolution (multilingual) | `pip install spacy transformers` | **Not installed.** `workers/resolve.py`'s entity resolution is rapidfuzz string-similarity only — no NER, no multilingual normalization. |
| polars | fast tabular processing | `pip install polars` | **Not installed.** |
| great-expectations | data quality checks in pipeline | `pip install great-expectations` | **Not installed, and nothing equivalent exists either.** The entire QA stage in `workers/pipeline.py` is a single confidence-threshold filter (`r["confidence"] >= threshold`) — no validity/completeness/freshness/uniqueness checks, no quality report emitted or stored. This is a real, currently-open gap, not a doc typo: a broader, confirmed finding this session is that 0pt's data-quality/QA layer, once described elsewhere in this doc corpus as already built and tested (`src/services/quality-check-service.ts` etc.), does not exist in code at all — zero commits in this repo's history, any branch, for any of those files. Every doc that used to make that claim (`docs/ICEBERG_WRITE_BOUNDARY.md`, `docs/11-lakehouse-write-plan.md`, `docs/P0_QUERY_SOURCE_FLOW.md`) has since been corrected. The Python engine independently confirms the same absence. |
| dbt-postgres *(optional)* | transform layer | `pip install dbt-postgres` | **Not installed.** |
| pgvector (Postgres ext) | embeddings/matching | `CREATE EXTENSION vector;` | The `pgvector/pgvector:pg16` image is used in `infra/docker-compose.yml`, so the extension is available in the container, but no code in `services/api` runs a vector query — not exercised by any of the files read this pass |
| boto3 *(or aioboto3)* | R2 (S3-compatible) | `pip install boto3` | **Not installed.** No R2/S3 upload code found in `services/api` this pass. |
| stripe (python) | webhook handling | `pip install stripe` | **Not installed** in `services/api/requirements.txt` (the JS `stripe` package in root `package.json` is separate and unrelated to this table) |
## Self-hosted services (docker-compose on the VPS)
| Service | Purpose | Image | Status |
|---|---|---|---|
| **NocoDB** | Airtable-style views over existing Postgres tables; embedded "Min data" | `nocodb/nocodb` | **Wired in `infra/docker-compose.yml`** (`nocodb/nocodb:latest`, points at the same `postgres` service) — not independently verified running/reachable this pass |
| **Metabase** | customer-facing dashboards; embedded | `metabase/metabase` | **Wired in `infra/docker-compose.yml`** (`metabase/metabase:latest`, same Postgres backend) — not independently verified running/reachable this pass |
| **Postgres** *(or Neon managed)* | backbone (+ pgvector) | `pgvector/pgvector:pg16` | **Wired in `infra/docker-compose.yml`** — but see the backend table above: the FastAPI service that should write to it currently does not (in-memory only) |
| **Redis** *(if Celery)* | queue | `redis` | **Container is wired in `infra/docker-compose.yml`**, but no Python code in `services/api` currently connects to it (no Celery, no `redis` client installed) |
| **Meilisearch** *(later)* | catalog search at scale | `getmeili/meilisearch` | Not in `infra/docker-compose.yml` yet — consistent with this doc's own "later" framing, not a gap |
## External managed (free/cheap tiers)
- **Vercel** — Next.js hosting.
- **Cloudflare R2** — object storage (presigned URLs). No R2 client code found in this pass (`boto3` not installed; no `@uppy/aws-s3` in `package.json`) — treat as not yet wired regardless of account status.
- **Stripe** — payments (test mode first). Server SDK (`stripe` npm package) is installed; whether a live/test key is actually configured was not verified this pass (no secret material read).
- **Resend** — email. Not installed (see frontend table).
- **Neon** *(optional)* — managed Postgres. `infra/docker-compose.yml` runs its own Postgres container instead; no evidence either way of a Neon instance actually in use.
## EXCLUSIONS — do not import
- **Bypass/stealth/evasion**: FlareSolverr, "CloakBrowser", undetected-chromedriver/undetected-playwright (as bypass), proxy/fingerprint-rotation engines, any "WAF bypass" repo. Prohibited by CLAUDE.md §1 — they destroy the honest-accuracy positioning and create legal exposure. **This is not hypothetical**: a real WAF-bypass/proxy-rotation/fake-identity scraper stack (`services/scrapers/*`) briefly existed on this repo's Gitea remote (`git.siax.io/sax3l/0pt`, three commits dated 2026-07-13) and has since been removed there via PR #1. It never touched `origin` (`github.com/sax3l/0pt`), which this doc corpus was verified against. See `docs/06-data-engine.md`'s Known Issue callout for the full, unresolved detail — which remote is canonical is Simon's call, not decided this pass.
- **Fake-account automation** on Apollo/Cognism/etc.
- **Unverified / likely-hallucinated** from earlier brainstorming: "CloakBrowser/CloakHQ", "Rota proxy engine", "OpenClaw". Do not adopt unless you confirm a real, maintained project — and even then, not if it falls in the bypass category.
- **Overlapping tools** already decided against (Superset, DataHub/OpenMetadata, Baserow/Teable, Trigger.dev, Delta Sharing in v1) — see `docs/02-architecture.md`.
> Note on real-but-overhyped tools (Firecrawl, browser-use, etc.): they exist, but ignore inflated star-counts and only adopt if they fit a *compliant* role and don't duplicate the stack.
Reference in New Issue View Git Blame Copy Permalink