CI (SIAX Cloud) / security (push) Successful in 12s
Security Scan (SIAX Cloud) / security (push) Successful in 14s
CI (SIAX Cloud) / quality (push) Failing after 45s
CI (SIAX Cloud) / integration (push) Successful in 1m59s
CI (SIAX Cloud) / security (pull_request) Successful in 20s
CI (SIAX Cloud) / quality (pull_request) Failing after 26s
CI (SIAX Cloud) / integration (pull_request) Successful in 39s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 33s
Automatiskt commitat innan lokala dubbletter rensades. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
344 lines
31 KiB
Markdown
344 lines
31 KiB
Markdown
# BEFORE_STATE.md — sax3l/0pt recon vs. P010 Master Execution Contract
|
|
|
|
**Scope:** `git.siax.io/sax3l/0pt`, working copy `/home/simon/siax-platforms-work/0pt`.
|
|
**Repo default branch:** `main` (verified via `git remote show origin` → `HEAD branch: main`; not assumed).
|
|
**Sync check:** `git fetch origin --prune` → local `main` HEAD `f8bb0d5` == `origin/main` HEAD `f8bb0d5`. In sync.
|
|
**Working-tree note:** 5 untracked paths present and NOT committed (do not treat as shipped):
|
|
`src/features/feature-registry.ts`, `src/fleet/`, `src/graph/`, `src/metrics/`, `src/projections/`.
|
|
These are additional to what's described below; they were not audited in this pass because they
|
|
are not yet part of any commit history — flagging only, no claims made about their contents.
|
|
|
|
**Classification scale used below** (per P010 instruction): `VERIFIED` (read the actual code/tests
|
|
and confirmed) / `IMPLEMENTED_NOT_VERIFIED` (code exists, but the specific claim wasn't exercised —
|
|
e.g. no live dependency reachable) / `DOCUMENTED_ONLY` (spec/doc exists, no code) / `STALE`
|
|
(doc contradicts current code) / `BROKEN` (code exists but demonstrably doesn't do what it claims)
|
|
/ `UNKNOWN` (not reachable in this pass).
|
|
|
|
**Critical framing already stated by the repo's own `CLAUDE.md`** (read in full before this recon):
|
|
this repo has **two remotes that both claim to be canonical** (`origin` = `github.com/sax3l/0pt`,
|
|
plus `git.siax.io/sax3l/0pt` — the one this recon targets), with independently diverged history that
|
|
Simon has not yet resolved. `CLAUDE.md` also documents a series of *self-corrections* to its own
|
|
prior claims (route counts, file paths, a fabricated `quality-check-service.ts`/`quality-report-service.ts`
|
|
claim that was retracted, a fabricated `sql/002_p0_dataset_quality.sql` claim that was retracted).
|
|
The repo also carries its own honest self-audit, `STATUS.md` (last full pass 2026-08-31, Master-DoD
|
|
§56 status-honesty scale), which independently corroborates nearly everything below — it is quoted
|
|
throughout rather than re-derived where it already says the same thing this recon found by reading code.
|
|
|
|
---
|
|
|
|
## 1. Source Registry
|
|
|
|
**Location:** `src/services/source-registry-service.ts` (rights/provenance/health/freshness/describe,
|
|
Master-DoD §10) composed on top of `src/services/source-service.ts` (base `0pt.source.schema.v1`
|
|
CRUD) and `src/services/source-rate-policy-service.ts`. Types: `src/types/source-registry.ts`,
|
|
`src/types/source.ts`.
|
|
|
|
**Fields that exist today** (`VERIFIED` by reading `src/types/source-registry.ts`):
|
|
- `SourceRights`: `license_id`, `license_url`, `allowed_purposes[]`, `pii_allowed`,
|
|
`redistribution` (`internal|customer_deliverable|public|prohibited`), `attribution`,
|
|
`verified_at`, `verified_by`, `notes`.
|
|
- `SourceProvenance`: `method` (`official_api|bulk_download|authorized_web|licensed_feed|manual_upload`),
|
|
`authority`, `compliance_evidence_ref`, `cadence`, `freshness_sla_hours`, `updated_at`.
|
|
**This is per-SOURCE provenance (how the source itself is obtained), not per-field/per-record
|
|
provenance** — see §3 below for that distinction, which matters for the "bridge_provenance"
|
|
question.
|
|
- `SourceHealth` / `SourceHealthCheck`: status (`up|degraded|down|unknown`), bounded history
|
|
(latest 100 observations kept in-process — explicitly not an audit log; AUD0 owns history).
|
|
- `SourceFreshness`: derived from `provenance.freshness_sla_hours` + last recorded success.
|
|
- `SourceDescriptor` = `source + rights + provenance + health + freshness + rate_policy` — this is
|
|
`source.describe`.
|
|
|
|
**Operations implemented** (`VERIFIED`, reading the class body): `registerSource` (=`source.register`),
|
|
`setRights`/`getRights`, `assertPurposeAllowed` (the fail-closed purpose gate — no rights record or
|
|
purpose outside the allowlist throws `AUTHORIZATION_DENIED`; **VERIFIED** fail-closed, not fail-open),
|
|
`setProvenance`/`getProvenance`, `recordSuccess`/`getFreshness`, `recordHealthCheck`/`getHealth`,
|
|
`querySources` (=`source.query`, tenant-scoped, filterable by purpose/redistribution),
|
|
`describeSource` (=`source.describe`). Tenant isolation is enforced on every read/write via
|
|
`assertSourceTenant`, which returns `NOT_FOUND` (not `403`) on a cross-tenant probe — same
|
|
no-existence-oracle posture ST0RE/N0D use (`VERIFIED` in code; also asserted tested — see §4).
|
|
|
|
**Persistence — confirmed exactly as flagged: `new Map()`, no Postgres.** `VERIFIED`:
|
|
`src/services/source-service.ts` — `private sources: Map<string, SourceRecord> = new Map()`, no
|
|
`pg`/`DATABASE_URL`/Prisma/Drizzle import anywhere in the file. `source-registry-service.ts` itself
|
|
adds four more Maps (`rights`, `provenance`, `healthChecks`, `lastSuccess`) with the same shape.
|
|
|
|
**Worse than "in-memory, no persistence" alone: it is in-memory *per service instance*, not even
|
|
a shared process-wide singleton.** `VERIFIED` — both the class doc-comment and a second, more
|
|
recent, self-correcting comment in `app/api/0pt/sources/[sourceId]/query/route.ts` state this
|
|
explicitly: *"each `new SourceService()` owns its own registry ... a Source registered via one API
|
|
route module is not yet visible to another route module in the same running process."* That same
|
|
route file's comment also retracts an earlier (wrong) claim that a migration
|
|
(`sql/002_p0_dataset_quality.sql`) already existed for this — confirmed by `git log --all` showing
|
|
zero commits ever creating that file; `sql/` today has only `001_init.sql` and
|
|
`004_iceberg_write_boundary.sql`.
|
|
|
|
**What Postgres migration would require** (derived, not yet spec'd anywhere in the repo as a
|
|
migration file): an `opt_sources` table (the shape `SourceRecord` in `source-service.ts` already
|
|
mirrors 1:1 — nullable columns matching every optional field), plus `opt_source_rights`,
|
|
`opt_source_provenance`, `opt_source_health_checks` tables for the four Maps in
|
|
`source-registry-service.ts`, all keyed by `source_id` with a `tenant_id` column and an index
|
|
supporting the existing tenant-scoped query patterns. **No migration for any of this exists.**
|
|
`resolveStoreMode` (referenced in the class doc-comment, `src/platform/dependency-mode.ts`) is the
|
|
only guard in place: it makes a *production* boot throw `DEPENDENCY_UNAVAILABLE` rather than silently
|
|
serving the Maps — i.e. the in-memory mode cannot currently reach production traffic, but that also
|
|
means the Source Registry **cannot run in production at all today**, migrated or not.
|
|
|
|
**HTTP route surface: `VERIFIED` — effectively zero.** `grep -rl "SourceRegistryService" app/` returns
|
|
nothing: no route anywhere calls the rights/provenance/health/describe class. Directory listing shows
|
|
`app/api/0pt/sources/` only contains `rate-policies/route.ts`,
|
|
`[sourceId]/query/route.ts`, and `rate-policies/[sourceRatePolicyId]/route.ts` — there is **no**
|
|
`app/api/0pt/sources/route.ts` (create/list) and **no** `app/api/0pt/sources/[sourceId]/route.ts`
|
|
(get/describe by id). The only other `/api/sources` route in the repo (`app/api/sources/route.ts`,
|
|
no `/0pt/` prefix) is unrelated — a static hardcoded array of 9 national registries for the
|
|
public "transparency moat" catalog page, not backed by `SourceRegistryService` at all.
|
|
`STATUS.md`'s own line for this capability says the same thing independently: *"No API route
|
|
surface yet."*
|
|
|
|
**Classification: `VERIFIED` — in-memory Maps, no Postgres, no HTTP route surface for
|
|
rights/provenance.** The prior claim was accurate; this pass adds two things not previously
|
|
flagged: (a) the in-memory scope is *per-instance*, not just *per-process*, which is a materially
|
|
worse starting point for a Postgres migration (every route module currently constructing its own
|
|
`new SourceService()` needs to be repointed at one shared instance, which is itself a prerequisite
|
|
change before or alongside adding a DB layer); (b) production boot is already hard-blocked without
|
|
a DB (`resolveStoreMode`), so there is no "silently wrong in prod" risk today — the risk is
|
|
"capability unusable in prod," not "capability lies in prod."
|
|
|
|
---
|
|
|
|
## 2. TS/Python split-brain
|
|
|
|
**Confirmed as described, and confirmed with zero bridge — not even an unused one.**
|
|
|
|
**TypeScript QualityService** (`VERIFIED`, `src/quality/quality-service.ts`, 393 lines +
|
|
`quality-types.ts` 114 lines + `expectation-engine.ts`): a genuinely sophisticated, versioned rule
|
|
engine — rule create/update with version bumps and superseded-version history, runs/results/issues,
|
|
severity-weighted scoring (`info=0.25, warning=0.5, error=1, critical=2`) with a score trend series,
|
|
a local Great-Expectations-subset engine (`ExpectationEngine`, 7 GE core expectation types executed
|
|
locally — `VERIFIED` per `STATUS.md`'s own count) that explicitly **never silently passes an
|
|
unsupported expectation type**, and a delegation path to N0D (`0pt.quality.ge_run` job) for anything
|
|
the local engine can't run, plus OpenLineage emission via `LineageEmitter` on every finished run.
|
|
Runs on Node/Next.js (Vercel edge / SIAX Deploy Platform), not Python.
|
|
|
|
**Python pipeline** (`VERIFIED`, `services/api/workers/pipeline.py`, 87 lines total — not literally
|
|
5 lines, but the *entire QA stage* is one line):
|
|
```python
|
|
delivered = [r for r in deduped if r["confidence"] >= threshold]
|
|
```
|
|
Runs inside the FastAPI orchestrator (`services/api/orchestrator/main.py`), which imports
|
|
`run_job` directly (`from workers.pipeline import run_job`) and calls it synchronously from the
|
|
order-run endpoint (`_JOBS[oid] = await run_job(oid, o["config"])`, line 171). This *is* the code
|
|
path that actually executes production orders on the FastAPI engine (`ENGINE_URL`-backed edge
|
|
routes defer here — see below).
|
|
|
|
**Runtimes, confirmed distinct:**
|
|
- TS `QualityService` + the rest of the "P0 Data-OS capability layer" (`src/services/*`,
|
|
`src/quality/*`, `src/er/*`, `src/lineage/*`, `src/metadata/*`, `src/features/*`) → Next.js App
|
|
Router, TypeScript, deployed as the edge app (`0pt.app`, Vercel/SIAX Deploy Platform).
|
|
- `pipeline.py` + `resolve.py` (entity resolution) + `adaptive_router.py` + `connectors.py` →
|
|
FastAPI, Python, `services/api/`, meant for a VPS/SIAX Cloud node (long-running jobs; per
|
|
`siax-0pt` skill orientation, never Vercel).
|
|
|
|
**Bridge that would be needed: none exists. `VERIFIED` by exhaustive grep, not inference:**
|
|
- `grep -rn "engineFetch|hasEngine|ENGINE_URL" src/quality/ src/services/source-registry-service.ts
|
|
src/services/source-service.ts` → **zero matches**. The P0 capability layer never calls the
|
|
engine-bridge helper (`lib/engine.ts`) at all.
|
|
- The engine bridge (`lib/engine.ts`, `hasEngine()`/`engineFetch()`, `X-Internal-Token` header) **does
|
|
exist and is real** — but its only callers are `app/api/estimates/route.ts`,
|
|
`app/api/vehicles/route.ts`, `app/api/screen/route.ts`, `app/api/ai/parse/route.ts`,
|
|
`lib/health-checks.ts`. None of these touch quality/QA.
|
|
- `grep -rn "localhost:3000|/api/0pt|requests\.|httpx\.|aiohttp" services/api/**/*.py` → the only
|
|
`httpx` usage is *outbound* to external registries (GLEIF, Brreg, Companies House, CVR, PRH,
|
|
financial filings) in `connectors.py`/`financials.py`. Python never calls back into the Next.js
|
|
app at all, in either direction.
|
|
- `grep -rn "aud0|AUD0|audit" services/api/**/*.py` → **zero matches.** The production pipeline
|
|
emits **no audit trail whatsoever** — no AUD0 event for a single field it sources, resolves, or
|
|
drops. All of the TS layer's audit/lineage machinery (`aud0-client.ts`, `LineageEmitter`,
|
|
OpenLineage emission on every quality run) is unreachable from the code path that actually runs
|
|
customer orders today.
|
|
- The two runtimes even have **duplicate, independent order systems**: the FastAPI orchestrator has
|
|
its own `POST /orders`, `GET /orders`, `POST /orders/{oid}/run` (which triggers `pipeline.py`),
|
|
`GET /orders/{oid}/status`; the Next.js edge has a *separate* `app/api/orders/route.ts` backed by
|
|
`lib/orders-store.ts` (Postgres via `lib/db.ts`, with an in-memory `Map` fallback "used only until
|
|
a database is provisioned" — this one specific store, unlike the P0 capability layer, does have a
|
|
real DB path). Nothing in the repo currently reconciles an order created via one system with the
|
|
other.
|
|
|
|
**What bridge would actually be needed, concretely** (none of this exists; this is the gap, not a
|
|
design already in place): either (a) an HTTP call from `pipeline.py`'s QA step out to a
|
|
`POST /api/0pt/quality/runs` endpoint on the Next app (which doesn't exist — no such route is wired,
|
|
per §1's route-surface finding pattern repeating here), backed by a real `QualityService` instance
|
|
with Postgres persistence (which also doesn't exist, per its own in-memory Maps); or (b) an N0D job
|
|
(`QualityService` already models this shape for GE delegation — `0pt.quality.ge_run` — but that
|
|
target is 0pt-motor's GE runner, a different codebase, and `STATUS.md` states its executor "does not
|
|
exist yet," so this path is a documented **SKELETON**, not a working bridge, and was never intended
|
|
to connect to `pipeline.py` specifically); or (c) both runtimes sharing one Postgres database, which
|
|
would require building the Postgres layer neither currently has for this data.
|
|
|
|
**Classification: `VERIFIED`.** The split-brain is total: different language, different runtime,
|
|
different deployment target, different order-lifecycle model, zero HTTP calls in either direction,
|
|
zero shared database, zero N0D job connecting them, zero audit trail from the side that actually
|
|
runs. This is not "two systems that drifted" — it's two systems that were never wired together in
|
|
the first place, and the sophisticated one (TS) is the one not in the live path.
|
|
|
|
---
|
|
|
|
## 3. Provenance tracking (source / retrieval / timestamp / transform / version / quality /
|
|
entity-resolution / producer / artifact)
|
|
|
|
**The canonical term `bridge_provenance`** (used by the `siax-0pt` skill and implied by the P010
|
|
prompt) **does not exist anywhere in this codebase.** `VERIFIED`: `grep -rln "bridge_provenance"`
|
|
across the full tree (excluding `node_modules`) returns nothing — no table, type, variable, doc
|
|
reference, or test. Whatever table/concept that name refers to at the Master-DoD level has not been
|
|
implemented under that name in `sax3l/0pt`.
|
|
|
|
**What exists instead, split across the two runtimes (consistent with §2's split-brain):**
|
|
|
|
*Python production path* (`services/api/sources/connectors.py`), `VERIFIED`:
|
|
```python
|
|
@dataclass
|
|
class Provenance:
|
|
field_name: str
|
|
source_id: str
|
|
confidence: float
|
|
level: int # 0 unavailable/licensed, 1 single, 2 partial, 3 corroborated
|
|
freshness: str | None = None
|
|
```
|
|
Covers: **source** (`source_id`) ✓, **quality** (`confidence`, `level`) ✓ (partial — no full QA
|
|
report, just the number), **timestamp** (`freshness`, a plain date string, not a full retrieval
|
|
timestamp with time-of-day) — partial. Missing entirely from this struct: **retrieval method**
|
|
(no equivalent of the TS `SourceProvenance.method` enum at the per-field level — the Python side has
|
|
no concept of *how* a given field was obtained beyond which connector answered), **transform**
|
|
(no record that a field passed through normalization/resolution), **version** (no schema or dataset
|
|
version tag), **producer** (no agent/service identity attached to a field), **artifact** (no link to
|
|
a delivered artifact id — a field's provenance dies with the in-process job; nothing persists it
|
|
past the response payload, since there's no DB write in `pipeline.py` at all).
|
|
|
|
`resolve.py` (entity resolution, part of the live pipeline) does propagate/merge the per-field
|
|
`Provenance` list across merged duplicate records and tracks `agreed_sources`/`merged_from` —
|
|
so **entity-resolution linkage into provenance partially exists** in the live path, but only as an
|
|
in-memory list attribute on a dict that is discarded once the HTTP response is sent; it is never
|
|
written anywhere durable.
|
|
|
|
*TypeScript layer* (disconnected from the above, per §2), `VERIFIED`:
|
|
- `SourceProvenance` (`src/types/source-registry.ts`) — this is **per-source**, not per-field:
|
|
`method`, `authority`, `compliance_evidence_ref`, `cadence`, `freshness_sla_hours`, `updated_at`.
|
|
Covers **source** ✓, **retrieval method** ✓ (at the source level, not the field level),
|
|
**timestamp** ✓ (`updated_at`), but not **transform**, **entity-resolution**, **producer**, or
|
|
**artifact** at all — those aren't in scope for this type, which describes a *source*, not a
|
|
*fact*.
|
|
- `LineageEmitter`/OpenLineage (`src/lineage/lineage-emitter.ts`, `openlineage.ts`) — this is where
|
|
**producer** (`OPT_OPENLINEAGE_PRODUCER` constant), **transform** (`capability_id` becomes
|
|
`job.name`), **version/run** (`run_id`, deterministic per the doc-comment so idempotent retries
|
|
dedupe instead of phantom-duplicating), and **artifact** (canonical refs like `st0re:artifact:...`,
|
|
`0pt:dataset:...` in dataset facets, explicitly "so evidence chains can be joined across
|
|
products") all live — but at the **dataset/run** granularity (one event per finished quality run,
|
|
dataset export, or search rebuild), not the **per-field** granularity the Python side and the
|
|
`bridge_provenance` framing imply. It's real code (`VERIFIED`, wired to a real, contract-tested
|
|
AUD0 client), but it emits from `QualityService`/`DatasetService`/search rebuilds — i.e. from the
|
|
side of the split-brain that isn't in the live order path (§2).
|
|
|
|
**Classification: `IMPLEMENTED_NOT_VERIFIED` / partial, split, and not reconciled.** No single
|
|
component in this repo covers all nine of source/retrieval/timestamp/transform/version/quality/
|
|
entity-resolution/producer/artifact for one fact. The live Python path covers source+quality+
|
|
partial-timestamp+partial-entity-resolution and persists none of it. The disconnected TS lineage
|
|
layer covers producer+transform+version+artifact but at dataset granularity and is contract-tested
|
|
against AUD0 rather than proven end-to-end from a real field. `bridge_provenance` as a named
|
|
construct: **not found — treat any claim that it exists in this repo as false** until someone
|
|
points to where it's meant to live (possibly a Master-DoD-level spec never implemented here, or a
|
|
0pt-motor concept — out of this repo's scope to confirm).
|
|
|
|
---
|
|
|
|
## 4. Failure-mode tests
|
|
|
|
Full test inventory (`VERIFIED`, `find tests -name "*.ts"`, 30 files):
|
|
`tests/api/*` (anonymous-routes-auth, health, ready), `tests/auth/*` (context, zitadel),
|
|
`tests/contracts/*` (aud0/cl0ud/n0d/st0re client contracts, dependency-mode, metadata-features,
|
|
siax-context), `tests/integration/*` (analytics-clickhouse, search-opensearch — both against real
|
|
service containers in CI per `STATUS.md`), `tests/lib/health-checks.test.ts`, `tests/p0/*` (13
|
|
files — dataset-iceberg-mapping, dataset-version-service, document-capture-mailbox,
|
|
document-extraction-{never-fabricates,not-configured}, document-intelligence-adapter,
|
|
enterprise-data-os-expansion, entity-resolution, external-writeback-service, fabric-onelake-adapter,
|
|
graph-mailbox-adapter, iceberg-write-service{,.realinfra}, odoo-bills-adapter, quality-service,
|
|
source-query-service, source-rate-policy, source-registry), `tests/validation/0pt-schemas.test.ts`.
|
|
|
|
Checked against the 10 named failure categories (`VERIFIED` by grep across the whole `tests/` tree,
|
|
case-insensitive, including hyphen/underscore/space variants):
|
|
|
|
| Failure test | Present? | Evidence |
|
|
|---|---|---|
|
|
| Schema-evolution | **Absent** | Zero matches for "schema evolution"/"schema_evolution" anywhere in `tests/`. |
|
|
| CDC-interruption | **Absent** | Zero matches for "CDC" anywhere in `tests/`. |
|
|
| Backfill | **Partial** | Only hit is `tests/p0/iceberg-write-service.test.ts` — scoped to the Iceberg write boundary, not a general backfill-failure-mode test. |
|
|
| Duplicate-input | **Absent** (as a named scenario) | No "duplicate input" test found. Adjacent: `LineageEmitter`'s doc-comment claims idempotent retries dedupe via deterministic `run_id`, and N0D/ST0RE client contract tests pin `idempotency_key` on job/artifact creation bodies — but no test actually POSTs the same input twice and asserts single-row output for the data-ingestion path itself. |
|
|
| Out-of-order | **Absent** | Zero matches. |
|
|
| Tenant-isolation | **VERIFIED PRESENT** | `tests/p0/source-registry.test.ts` — explicit cross-tenant `describe` → `NOT_FOUND` (§50, no existence oracle), rejects a query with empty `tenant_id`, foreign-tenant `querySources` returns empty. `tests/p0/quality-service.test.ts` — "duplicate rule_id is CONFLICT; cross-tenant reads are NOT_FOUND (§50)". Also present in `tests/integration/analytics-clickhouse.integration.test.ts` per its own row in `STATUS.md` ("tenant isolation, closed DSL, TTL schema" against a real ClickHouse container). This is the one category genuinely well covered, and covered against a real dependency in at least one case. |
|
|
| Restart (mid-job resume/crash-recovery) | **Absent as a named scenario** | Broad grep on "restart/resume/idempoten" hits several contract tests, but those test idempotency-key wire shape on client requests (N0D/ST0RE/AUD0), not an actual "job restarts mid-pipeline and resumes correctly" scenario against `pipeline.py` or any TS service. |
|
|
| Dedup-determinism | **Absent** | The only "dedup" hit in the whole tree is inside `tests/contracts/aud0-client.contract.test.ts`, unrelated to data deduplication. `tests/p0/entity-resolution.test.ts` (137 lines) tests blocking, scoring, hard-rule overrides, human-review handoff for mid-band pairs, and refusal of unblocked all-pairs runs — good coverage of ER *behavior*, but no test asserts that re-running resolution on the same input (or the same input in a different order) is deterministic. **Worth flagging as a real, live gap**: `services/api/workers/resolve.py`'s merge loop is order-sensitive by construction (it greedily merges into a growing `out` list scanned in input order), so which record survives as the "representative" of a merged group can plausibly differ if input order changes — untested either way. |
|
|
| Quality-gate | **Partial, and only on the disconnected side** | `tests/p0/quality-service.test.ts` thoroughly tests the TS `QualityService`'s rule-versioning, run/issue/score lifecycle, and GE delegation-never-fakes behavior — real coverage of *that* engine. But per §2, that engine is not in the live order path; the actual production quality gate (`pipeline.py`'s one-line confidence filter) has **no dedicated test at all** — no test in the repo calls `run_job` or exercises the confidence-threshold drop behavior end to end. |
|
|
| Lineage | **Present, contract-level** | `tests/p0/document-extraction-never-fabricates.test.ts`, `document-capture-mailbox.test.ts`, `enterprise-data-os-expansion.test.ts`, `external-writeback-service.test.ts`, and the `aud0-client` contract suite all touch lineage/evidence emission. Per `STATUS.md`: "INTEGRATION_TESTED (contract) ... AUD0-side ingestion verified against contract double, not live AUD0" — i.e. tested against a pinned recorded contract, not a live AUD0 instance. |
|
|
|
|
**Classification: `VERIFIED`.** Of the 10 named categories, genuinely covered: **tenant-isolation**
|
|
(well, and partly against real infra). Partially covered or covered only adjacently:
|
|
backfill, duplicate-input, restart, quality-gate, lineage. **Entirely absent**: schema-evolution,
|
|
CDC-interruption, out-of-order, dedup-determinism. The one category most directly relevant to "does
|
|
the system deliver correct data" — dedup-determinism — is untested against an algorithm
|
|
(`resolve.py`) that has a plausible order-sensitivity risk by inspection.
|
|
|
|
---
|
|
|
|
## 5. 1P / ST0RE / N0D / AUD0 / CL0UD integration status
|
|
|
|
Cross-checked against `STATUS.md`'s own "Cross-product clients" table (2026-08-31 audit), which
|
|
this pass independently corroborates by reading the client source files.
|
|
|
|
| Product | Status | Evidence |
|
|
|---|---|---|
|
|
| **AUD0** | `VERIFIED` real-contract client, `IMPLEMENTED_NOT_VERIFIED` live, **`BROKEN`/absent on the path that matters** | `src/clients/aud0-client.ts` — rewritten 2026-08-31 against AUD0's actual routes (`POST /v1/aud0/events`, `POST /v1/aud0/openlineage`) after discovering the prior client posted to a lineage path (`/v1/aud0/lineage/events`) that **never existed** on AUD0. Contract-pinned in `tests/contracts/aud0-client.contract.test.ts`. Per `STATUS.md`: "No durable local outbox for AUD0 outages — emission failure is surfaced, not queued." **But**, per §2 of this doc: zero calls to AUD0 exist anywhere in `services/api/` (the live production pipeline) — so however solid the client is, the code path that actually processes customer orders emits no audit events at all today. |
|
|
| **ST0RE** | `VERIFIED` real-contract client (two of them: generic artifacts + lakehouse) | `src/clients/st0re-client.ts` (`POST /v1/st0re/artifacts/register`, camelCase, rewritten 2026-08-31 after the prior snake_case body was found to be a guaranteed 400) and `src/clients/st0re-lakehouse-client.ts` (Iceberg table/snapshot writes). The lakehouse client carries a **live-verified** note (2026-07-13 session: `api.st0re.siax.io` reachable, a real namespace→table→snapshot write was driven against production ST0RE, surfacing 3 real contract mismatches that were then fixed). Per `STATUS.md`: "Raw-bytes upload still goes via LocalBlobStore interim ... presign upload path not consumed yet." |
|
|
| **N0D** | `VERIFIED` real-contract client, delegation targets are `SKELETON` | `src/clients/n0d-client.ts` — rewritten 2026-08-31 against N0D's real `createJobSchema` (`required_role`, `resource_requirements{cpu,memory_gb,disk_gb}`, body-only `idempotency_key` — no header). Used by `QualityService` (GE delegation), `EntityResolutionService` (Splink batch delegation), `document-capture-service.ts`, `feast-adapter.ts`, `iceberg-write-service.ts`. Per `STATUS.md` and confirmed in `tests/p0/entity-resolution.test.ts`/`quality-service.test.ts`: the delegation *contract* is real and tested (submits the right job shape, fails hard with `DEPENDENCY_UNAVAILABLE` without a client rather than faking a result) — but the **executors on the 0pt-motor side for `0pt.quality.ge_run`, `0pt.features.*`, `0pt.er.batch_resolution` do not exist yet**, so every delegated job would sit unexecuted today. |
|
|
| **CL0UD** | `VERIFIED` real-contract client, **not wired into the request path** | `src/clients/cl0ud-client.ts` — rewritten 2026-08-31 after discovering the prior client posted to `/v1/cl0ud/policies/evaluate`, which "has never existed" on CL0UD; real contract is `POST /api/policy/evaluate` with a camelCase `{data, meta}` envelope and a separate `requiresApproval` flag. Used by `external-writeback-service.ts`, `document-extraction-service.ts`, `iceberg-write-service.ts`. **Not called from `middleware.ts`** (the actual request-gating layer for `/api/0pt/*`, which does its own Zitadel-only auth check and does not consult CL0UD policy evaluation) — so CL0UD policy decisions are consulted by a few specific services, not enforced platform-wide at the gateway as the `siax-cloud` orientation model ("Auth/RBAC/Tenant/Policy" as one spine step) implies. Per `STATUS.md`: "0pt's own capabilities are not yet registered in CL0UD's registry" — meaning 0pt is a CL0UD *client* in a couple of places but not yet a registered CL0UD *capability* itself. |
|
|
| **1P** | `DOCUMENTED_ONLY`, explicitly self-labeled "not in use" | No client, no import, no usage anywhere in code — the only hit for "1p" in the whole tree (outside docs) is a single comment in `src/types/source-registry.ts` (`compliance_evidence_ref: /** Robots/ToS evidence ref (1p evidence or aud0 evidence ref) where applicable. */`). The design is documented in `docs/06-data-engine.md` under **"Network-layer provenance: `1p` (not yet in use)"**, which the repo itself grades explicitly: *"Status: not in use. 1p is Phase 0 — code, contracts, tests; not deployed, no credentials [issued]."* That doc section was merged via PR #3 (`docs/1p-egress-measurement-point` → `main`, commit `39967cb`) — so the *documentation* is live and current, but it documents a plan, not an integration. 1P itself is a separate repo (`git.siax.io/sax3l/1p`) not in scope for this recon. |
|
|
|
|
**Classification, summarized:** ST0RE and N0D clients: `VERIFIED` real-contract, ST0RE additionally
|
|
`LIVE_VERIFIED` once against production infra. AUD0 client: `VERIFIED` real-contract but
|
|
**functionally `BROKEN`/absent** on the one path where it would matter most (the live order
|
|
pipeline emits nothing to it). CL0UD client: `VERIFIED` real-contract but scoped to a handful of
|
|
services, not the platform-wide policy gate the architecture model calls for; 0pt is not yet a
|
|
registered CL0UD capability. 1P: `DOCUMENTED_ONLY`, self-graded Phase 0, honestly labeled as such by
|
|
the repo itself.
|
|
|
|
---
|
|
|
|
## Other findings observed in passing (not one of the 5 requested areas, flagged for completeness)
|
|
|
|
- **`services/scrapers/` (WAF-bypass stack) still exists on `main`, unreviewed and un-deleted.**
|
|
`git branch -a` shows `feature/0pt-remove-noncompliant-scrapers` as an **open, unmerged** branch;
|
|
`CLAUDE.md` itself documents this precisely and warns "an earlier version of this paragraph
|
|
claimed it was 'since removed', which was false." `VERIFIED` this pass: `services/scrapers/`
|
|
contains `waf_bypass.py`, `engine.py`, `engine_v2.py` on `main` today. `STATUS.md` lists it as
|
|
"quarantined, pending deletion PR — NOT touched by this wave; nothing new wires it." Nothing in
|
|
this recon found any new code path invoking it, but it is still physically present in the tree
|
|
Simon's team would deploy from if `services/scrapers/` were ever imported by mistake. This is a
|
|
live compliance posture question (siax-0pt skill: "No bot-bypass / stealth ... excluded by
|
|
design"), independent of and unrelated to the 0pt.app Vercel-billing auth-bypass issue that this
|
|
P010 pass was explicitly told to ignore — this is a different, code-level, not
|
|
infrastructure-billing, issue.
|
|
- **Two remotes claim canonical status for this repo** (`github.com/sax3l/0pt` vs.
|
|
`git.siax.io/sax3l/0pt`) and `CLAUDE.md` states this is explicitly unresolved and is Simon's
|
|
decision, not something to silently pick a side on. This recon worked against the Gitea remote
|
|
as instructed by the P010 prompt; findings above should not be assumed to hold against the GitHub
|
|
copy without separately checking it.
|
|
|
|
---
|
|
|
|
## Summary table (P010-requested classification, one line each)
|
|
|
|
| Area | Classification |
|
|
|---|---|
|
|
| (1) Source Registry | **VERIFIED**: in-memory Maps (per-instance, not even per-process-shared), no Postgres, zero HTTP route surface for rights/provenance/health/describe. Fail-closed purpose gate and tenant isolation are real and tested. |
|
|
| (2) TS/Python split-brain | **VERIFIED**: total split — different runtime/language/deployment/order-system, zero bridge in either direction, zero shared DB, zero audit emission from the live (Python) path. |
|
|
| (3) Provenance tracking | **VERIFIED partial / split**: no single implementation covers all 9 dimensions for one fact; `bridge_provenance` as a named construct does not exist in this repo. Python path covers source+quality+partial-timestamp+partial-ER, unpersisted. TS lineage layer covers producer+transform+version+artifact at dataset granularity, disconnected from the live path. |
|
|
| (4) Failure-mode tests | **VERIFIED**: tenant-isolation well covered (incl. against real infra in one case); backfill/duplicate-input/restart/quality-gate/lineage partial or adjacent-only; schema-evolution, CDC-interruption, out-of-order, dedup-determinism entirely absent. |
|
|
| (5) 1P/ST0RE/N0D/AUD0/CL0UD | **VERIFIED**: ST0RE + N0D clients real-contract (ST0RE live-verified once); AUD0 client real-contract but unused by the live pipeline; CL0UD client real-contract but not platform-wide-enforced and 0pt not yet a registered CL0UD capability; 1P documented-only, self-graded Phase 0/not in use. |
|