admin/0pt

Files

T

adminandClaude Sonnet 5 2902a26bac docs: full accuracy overhaul, grounded claim-by-claim in real code (#4 )

* docs: full accuracy overhaul, grounded claim-by-claim in real code

Every doc in this repo (CLAUDE.md, README.md, SECURITY.md, docs/*.md) was
audited and rewritten by a multi-pass process: independent rewrite per
cluster, adversarial re-verification of that rewrite against the actual
code, then a final cross-doc consistency pass — each stage re-checking
claims against the repo directly rather than trusting the previous stage.

Two corrections matter most:

1. docs/ICEBERG_WRITE_BOUNDARY.md, docs/11-lakehouse-write-plan.md, and
   docs/P0_QUERY_SOURCE_FLOW.md asserted src/services/quality-check-service.ts,
   quality-report-service.ts, src/data-platform/services/catalog-service.ts,
   and sql/002_p0_dataset_quality.sql/sql/003_*.sql as real, tested code.
   git log --all across every branch shows zero commits for any of these
   paths, ever. 0pt's data-quality/QA enforcement does not exist in code.
   Corrected everywhere, with a new docs/P0_DATASET_QUALITY_FLOW.md added
   as an explicit target-spec (not-yet-built) replacement.

2. github.com/sax3l/0pt (this repo) and git.siax.io/sax3l/0pt (Gitea) are
   two diverged remotes sharing part of the same history (services/api/
   workers/adaptive_router.py is byte-identical between them). Gitea's main
   independently accumulated a WAF-bypass/proxy-rotation scraper stack in
   three 2026-07-13 commits, since removed via a Gitea PR; GitHub's main
   never had it. Which remote is canonical/deployed is unresolved — flagged
   consistently across the corpus, not decided here.

Also newly and consistently documented: the tenant_id/project_id-read-from-
client-body gap on the P0 lakehouse/rate-policy/source-query routes (no
auth/session system exists anywhere in this repo yet); that every CL0UD/
N0D/AUD0/ST0RE platform client is real but mock-only (zero process.env
reads for any of their *_URL vars); that two P0 mutation routes
(lakehouse/mappings, sources/rate-policies) always fail over HTTP today
because their prerequisite entity can never be created via any route; a
schema bug in sql/004 (FK references to opt_datasets/opt_sources, which
don't exist); and that 0pt.app's live deployment was independently
confirmed (curled this pass) to be running this exact codebase with
DATABASE_URL/REDIS_URL unset in production.

New file: docs/12-edge-capability-api.md — the edge-native P0 TypeScript
capability contract (app/api/0pt/*, /api/screen, /api/ai/parse), previously
undocumented and distinct from docs/04-api.md's FastAPI-engine-only scope.

Verified before commit: tsc --noEmit clean; vitest run 88 passed, 1 todo,
0 failures (13 files) — docs-only change, no source touched.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* docs: reflect PR #5's lakehouse-mapping-registry + status-code fixes

Follow-up to keep docs/12-edge-capability-api.md accurate once both PRs
land: fix/0pt-full-bug-hunt (#5) fixed the shared-instance bug in the
lakehouse mapping routes and added proper status-code mapping to the
document extraction route. This doc was written against the pre-fix
state in the same pass that added it, so it needs the same correction.

Does not claim the lakehouse write path fully works now -- the deeper
dataset-registration gap (no route ever calls DatasetService.registerDataset)
is unchanged and still blocks it end-to-end; only the shared-instance half
of the bug is fixed.

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

2026-07-22 18:49:01 +02:00

7.0 KiB

Raw Blame History

10 · Lakehouse decision (pointer)

This doc is a pointer, not a new architecture. It records an ecosystem-wide decision made outside this repo and states what it means for 0pt specifically. It does not change anything in docs/02-architecture.md or docs/03-data-model.md today — Postgres remains the backbone for 0pt's platform tables and current data entities (docs/03). Read this alongside those docs, not instead of them.

Status update. Since this pointer doc was first written, a follow-up implementation pass turned most of 0pt's side of this plan into real TypeScript — see docs/11-lakehouse-write-plan.md's "what the implementation pass closed" section and docs/ICEBERG_WRITE_BOUNDARY.md for the current, corrected status. Two things to know before reading the rest of this doc: (1) that write-path code is real and unit-tested against a mock ST0RE client, not verified against a running ST0RE deployment; (2) an earlier version of docs/11 and docs/ICEBERG_WRITE_BOUNDARY.md also claimed 0pt's data-quality/QA stage had a dedicated, tested quality-check-service.ts/quality-report-service.ts implementation — that claim was fabricated. git log --all for those paths (and sql/002_p0_dataset_quality.sql, sql/003_*.sql) returns zero commits, on any branch, ever. Both docs have been corrected; see them for the full account. This doc's own references to "QA" below point only to the spec in docs/06-data-engine.md, not to any implemented service.

The decision (made elsewhere — do not re-litigate here)

Apache Iceberg is the canonical SIAX lakehouse table format, per ADR-0030, authored in the n0d (Compute OS) repo (a dated copy also lives in cl0ud's docs/architecture/, since the boundary audit that motivated it originated in n0d but the ADR itself is recorded as SIAX-wide; both copies describe the same accepted decision — verified by reading both this pass). Delta Lake is explicitly rejected as the canonical P0/P1 format across SIAX Technology; it may only ever be used later as an import/export bridge, never as the format 0pt writes natively.

Rationale (from the ecosystem ADR, summarized): SIAX needs one vendor-neutral, multi-engine, catalog-first open table format for the MinIO/ST0RE lakehouse, because several products need to share the same tables — ST0RE owns catalog/metadata, 0pt owns dataset writes, N0D owns query/compaction execution, AUD0 owns lineage, API0 owns docs — and because future reads must work from many engines (Trino/Spark/Flink/DuckDB/etc.), not one.

Cross-product ownership split (context, not 0pt's to change)

Product	Owns
ST0RE	Table/catalog/artifact metadata (the Iceberg catalog + table registry)
0pt	Dataset production and writes into Iceberg tables, through ST0RE's table registry
N0D	Query/compaction/maintenance execution only
AUD0	Lineage
CL0UD	Policy

This table condenses ADR-0030 §2.1/§2.4's ownership split for 0pt's purposes — it is a paraphrase, not a verbatim quote (corrected this verification pass; re-checked against cl0ud's copy of the ADR). The ADR's §2.4 table also names INF0 (model registry/experiment tracking, MLflow as an adapter) and its §2.1 also names API0 (schema/docs exposure); both rows are omitted here because neither changes anything for 0pt specifically. One paraphrase is deliberate, not accidental: the ADR's own 0pt row reads "Dataset production, quality, data contracts, writes" — the word "quality" is intentionally dropped from this table's 0pt row above, because 0pt's quality/QA stage is not yet built (see the status callout at the top of this doc). See ADR-0030 §2.4 directly (cl0ud's or n0d's copy) for its own exact wording of every row.

What this means for 0pt

0pt is the write-path owner for the datasets it produces: 0pt runs sourcing → extraction → structuring → enrichment → QA (docs/06-data-engine.md — a design spec, not yet a built service; see the status callout above) as it does today, and writes the resulting datasets as Iceberg tables through ST0RE's table registry. 0pt does not talk to N0D's catalog or any lakehouse storage directly; it goes through ST0RE.

ST0RE's table registry is real, committed code (sax3l/st0re, /v1/st0re/lakehouse/*), and 0pt has a real client and services that target it — src/clients/st0re-lakehouse-client.ts, src/services/iceberg-write-service.ts, src/services/dataset-iceberg-mapping-service.ts. None of this has been exercised against a running ST0RE instance from this repo: every test to date runs against that client's deterministic in-memory mock (no ST0RE_URL was reachable from this environment). Treat "0pt writes Iceberg tables through ST0RE" as real code, mock-tested, not live-verified — see docs/ICEBERG_WRITE_BOUNDARY.md's "Verification status" section for the full detail, including what is and isn't wired end-to-end.

This does not replace Postgres as the platform backbone (orders, users, jobs, billing — docs/03 — stay exactly as they are). It is additive: dataset output (the delivered data product, today exported as CSV/Parquet/JSON per docs/04-api.md) gains an additional, canonical Iceberg write-path. Concrete to-dos, current build status, and required schema/event names are in docs/11-lakehouse-write-plan.md.

Also relevant: the `sourceRatePolicy` deprecation in N0D

The same ecosystem audit that produced ADR-0030 found N0D's ComputePolicy type carrying a sourceRatePolicy field (per-source-domain crawl rate limits, e.g. {'allabolag.se': 1, 'ratsit.se': 1}) hardcoded into N0D's policy defaults. That is 0pt's own crawl-domain knowledge — it has been deprecated in N0D (the field is still present on the type for backward compatibility, but N0D's own boundary-repair tests assert it now has zero effect on any allow/deny decision; verified by reading n0d's src/policy.ts and src/__tests__/boundary-repair.test.ts this pass). 0pt does not rely on N0D for source-rate/ crawl-compliance policy; 0pt already owns the sourcing chain and its rate limits today (sources.yaml, docs/06-data-engine.md §rate_limit), and has since formalized its own model — src/services/source-rate-policy-service.ts, real and tested, see docs/11-lakehouse-write-plan.md and docs/ICEBERG_WRITE_BOUNDARY.md.

Non-goals of this doc

Does not itself implement Iceberg writes — see docs/11-lakehouse-write-plan.md / docs/ICEBERG_WRITE_BOUNDARY.md for what a later pass actually built (real, mock-tested code; not live-verified; data-quality/QA enforcement specifically is not built — see the status callout above).
Does not modify docs/03-data-model.md's Postgres schema.
Does not require ST0RE's table registry to exist — as of this pass it does exist as real, committed code in sax3l/st0re, though this repo has not confirmed it live/reachable. This doc remains the pointer; docs/11 lists what's still blocked on a live, reachable ST0RE vs. what 0pt has already built against the mock.

7.0 KiB Raw Blame History