From e2437ab09eb4bec40a5d77ae83b96eeae65f3819 Mon Sep 17 00:00:00 2001 From: Simon Axelsson Date: Mon, 7 Sep 2026 18:22:26 +0200 Subject: [PATCH] docs(er): reconcile ecosystem acquisition plan with canonical ER architecture --- ...DIGITAL_ECOSYSTEM_DATA_ACQUISITION_PLAN.md | 197 ++++++++++++++++++ 1 file changed, 197 insertions(+) create mode 100644 docs/SIAX_DIGITAL_ECOSYSTEM_DATA_ACQUISITION_PLAN.md diff --git a/docs/SIAX_DIGITAL_ECOSYSTEM_DATA_ACQUISITION_PLAN.md b/docs/SIAX_DIGITAL_ECOSYSTEM_DATA_ACQUISITION_PLAN.md new file mode 100644 index 0000000..842f9df --- /dev/null +++ b/docs/SIAX_DIGITAL_ECOSYSTEM_DATA_ACQUISITION_PLAN.md @@ -0,0 +1,197 @@ +# 0pt — SIAX Digital Ecosystem Data Acquisition & Entity Plan + +Status: IMPLEMENTATION CONTRACT / PRODUCTION GATES OPEN +Date: 2026-09-07 + +0pt is the generic external-data acquisition, enrichment, provenance and entity-candidate capability. Product domains own their business entities and decisions. 0pt supplies sourced observations, acquisition receipts and entity-resolution evidence; it must not become a second Work/PRESS0/FUND0/CRM system of record. + +## 1. Ecosystem ownership + +### Work / j0b + +0pt may supply: +- company/employer/end-client enrichment; +- registry/company facts; +- generic source acquisition; +- buyer/supplier enrichment for procurement; +- entity-resolution candidates/evidence; +- public technology/company signals for account intelligence. + +j0b/Work owns opportunities, roles, requirements, consultants, matches, submissions and commercial decisions. + +### PRESS0 + +0pt may supply public-source acquisition, official releases/datasets, source snapshots and factual dataset/source metadata. PRESS0 owns articles, citations, claims and editorial/publication decisions. + +### FUND0 + +0pt may supply public issuer/company/market facts and registry enrichment. FUND0 owns IR projects, disclosures, investor access and data-room state and remains IR_ONLY. + +### FIT0 / CRM + +0pt may supply dated account enrichment and market/buying signals plus entity-candidate evidence. FIT0 owns account/opportunity scoring. CRM remains a projection/operational interaction surface, not the external-fact source of truth. + +## 2. Provenance-first invariant + +Every externally derived fact or observation must preserve enough lineage to answer: +- source/provider; +- external identifier or URL where appropriate; +- observed/fetched time; +- parser/transform version; +- freshness/confidence where meaningful; +- rights/reuse/retention classification; +- payload/artifact hash or ST0RE ref where raw retention is required. + +No enrichment silently overwrites a higher-authority canonical fact without an explicit precedence/provenance rule. + +## 3. Lossless acquisition contract + +When a consumer needs raw upstream retention, 0pt returns or records an immutable acquisition/artifact reference plus metadata rather than retaining only a lossy normalized subset. ST0RE is the preferred custody plane for governed raw payloads once that integration is live verified. + +Target acquisition-receipt fields: +- source/provider; +- external ID/URL; +- fetched/observed times; +- payload/artifact hash/ref; +- parser/version; +- rights/reuse/retention classification; +- data classification; +- extracted observations/provenance; +- correlation/tenant/project scope. + +## 4. Canonical entity-resolution architecture + +There must be one decision architecture, not competing production scorers. + +### 4.1 Synchronous candidate path + +The canonical synchronous engine is the TypeScript `EntityResolutionService` exposed through typed HTTP surfaces. It provides: +- deterministic blocking; +- explainable similarity evidence; +- product hard-rule hooks; +- review-band candidates; +- stable-ID hard match/non-match rules at the product boundary. + +For the Work/company route, fuzzy/similarity evidence is never identity proof. Score-only candidates are review evidence only; only explicit stable-identifier hard rules may emit an automatic match. + +### 4.2 Python ingestion path + +The Python pipeline is a thin adapter to the canonical engine. It must not maintain a second fuzzy identity scorer. If the canonical bridge is unavailable, it returns records unmerged. At the consumer boundary it accepts only hard-rule matches and preserves hard non-match constraints across transitive clustering. + +### 4.3 Heavy/statistical ER + +Heavy probabilistic linkage (for example Splink when the exact runtime/license/security tuple is approved) belongs behind the 0pt → N0D batch job contract. Statistical output is candidate evidence, not unilateral canonical identity mutation. Production certification of a runtime is separate from implementing the contract. + +## 5. Strong company identity keys + +Prefer strong keys where available: +- LEI (global scope); +- country + official registration number; +- VAT/official registry identifiers with their correct jurisdictional scope; +- other verified registry IDs. + +National registration numbers must never be treated as globally unique without jurisdiction. Verified domains may be supporting evidence but are not equivalent to a statutory identifier. + +Negative comparable stable-ID evidence dominates positive similarity and must prevent transitive cluster merges. + +## 6. Human review boundary + +Similarity/fuzzy candidates may be ranked and explained. They do not auto-merge high-risk/canonical company or person identity. Review decisions must be attributable, tenant scoped and auditable when persistence is introduced. + +## 7. Public procurement enrichment + +Support acquisition/normalization references for buyer, supplier, award and contract datasets such as TED/OCDS/official sources. j0b owns commercial opportunity/access logic; 0pt supplies sourced external facts and entity evidence. + +## 8. Source-policy metadata + +Products may pass/receive policy metadata for: +- access method; +- reuse/redistribution; +- retention; +- polling/rate limits; +- terms/license review date; +- purpose/data classification. + +0pt never assumes that publicly reachable content is automatically reusable for every commercial or publication purpose. + +## 9. PRESS0 rights boundary + +Acquisition/extraction is not publication permission. PRESS0 decides what may be quoted, reproduced or published. 0pt preserves source/right metadata needed for that decision. + +## 10. Company verification boundary + +0pt may fetch and reconcile official company facts. ID0 owns verification/KYB decisions. j0b/CRM consume explicit verification references/status; they must not infer `verified` merely because 0pt found a matching registry row. + +## 11. Market/account signals + +Useful dated observations include: +- hiring activity; +- public procurement notices/awards; +- public technology/platform signals; +- company growth/change events; +- leadership/public contact changes where lawful; +- relevant product/market announcements. + +Signals are provenance-bearing observations with observation time and freshness, not permanent assertions. + +## 12. API contract + +Generic typed capabilities should cover: +- company/entity enrichment; +- source acquisition receipt; +- structured observations with provenance; +- entity-resolution candidates; +- batch ER job submission; +- provider/source health/freshness; +- durable dataset/version/search surfaces where implemented. + +Avoid bespoke copies per product when a generic contract is sufficient. Product consumers use APIs/contracts rather than direct database coupling. + +## 13. Security/privacy + +Required controls include: +- tenant/project/purpose scoping; +- PII minimisation; +- controlled egress/allowed-host policy; +- SSRF protection for user-supplied URLs; +- canonical secret storage; +- no raw sensitive payloads in logs; +- retention/deletion support; +- provider/license/data-rights documentation; +- fail-closed identity merge behavior; +- append-only evidence for security/lineage-sensitive mutations when AUD0 integration is required. + +## 14. Production gate + +Code or architecture does not equal deployment. A capability progresses honestly through IMPLEMENTED → TESTED → DEPLOYED → LIVE_VERIFIED. + +Before another product treats 0pt as a mission-critical dependency, the exact branch/commit must pass canonical Gitea gates and the relevant live endpoint/dependency must be verified. Protected workflow changes, database migrations, AUD0 outbox delivery and real external dependencies require canonical runner/live evidence. + +## 15. Implementation order + +1. Reconcile canonical Gitea/main and eliminate duplicate engines/owners. +2. Keep the synchronous TS ER engine as the canonical candidate/decision contract. +3. Keep Python ingestion as a fail-closed adapter, never a second scorer. +4. Complete durable acquisition/provenance/dataset-version/search contracts. +5. Certify a heavy probabilistic ER runtime through N0D only after runtime/license/security review. +6. Wire j0b company/end-client/buyer enrichment through typed APIs. +7. Wire FIT0/CRM signals as projections/observations, not shared mutable truth. +8. Add PRESS0/FUND0 consumers only when their repos/contracts are real. +9. Load/performance/backup/restore/live-verify every mission-critical path. + +## 16. Definition of Done per data capability + +- source/provider documented; +- legal/license/data-use classification recorded; +- provenance/freshness/confidence available where meaningful; +- deterministic/idempotent normalization where expected; +- entity resolution tested against labelled truth data; +- fuzzy evidence cannot silently become identity truth; +- tenant/purpose/security negative tests pass; +- provider outage/schema drift handled fail closed or explicitly degraded; +- raw artifact retention/deletion contract works where required; +- exact production endpoint/dependency is live verified; +- product consumes API/contract, not another product's DB; +- status documentation reflects observed reality. + +Target ecosystem uses in this document do not claim they are already wired or live. \ No newline at end of file