From 5abc842dfd9b386ac694e4b090c69f4bf8a14d29 Mon Sep 17 00:00:00 2001 From: Simon Axelsson Date: Mon, 7 Sep 2026 14:42:09 +0200 Subject: [PATCH] docs: add ecosystem data acquisition and entity plan --- ...DIGITAL_ECOSYSTEM_DATA_ACQUISITION_PLAN.md | 186 ++++++++---------- 1 file changed, 83 insertions(+), 103 deletions(-) diff --git a/docs/SIAX_DIGITAL_ECOSYSTEM_DATA_ACQUISITION_PLAN.md b/docs/SIAX_DIGITAL_ECOSYSTEM_DATA_ACQUISITION_PLAN.md index 842f9df..cfdcdc9 100644 --- a/docs/SIAX_DIGITAL_ECOSYSTEM_DATA_ACQUISITION_PLAN.md +++ b/docs/SIAX_DIGITAL_ECOSYSTEM_DATA_ACQUISITION_PLAN.md @@ -1,54 +1,56 @@ # 0pt — SIAX Digital Ecosystem Data Acquisition & Entity Plan -Status: IMPLEMENTATION CONTRACT / PRODUCTION GATES OPEN +Status: TARGET / IMPLEMENTATION CONTRACT Date: 2026-09-07 -0pt is the generic external-data acquisition, enrichment, provenance and entity-candidate capability. Product domains own their business entities and decisions. 0pt supplies sourced observations, acquisition receipts and entity-resolution evidence; it must not become a second Work/PRESS0/FUND0/CRM system of record. +0pt remains the generic external-data acquisition/enrichment/provenance capability. Product domains own their business entities and decisions. 0pt supplies sourced observations/enrichment/entity-resolution inputs and must not become a second Work/Press/Fund/CRM SSOT. -## 1. Ecosystem ownership +## 1. Ecosystem uses ### Work / j0b -0pt may supply: - company/employer/end-client enrichment; - registry/company facts; -- generic source acquisition; +- source acquisition where generic 0pt infrastructure is appropriate; - buyer/supplier enrichment for procurement; -- entity-resolution candidates/evidence; +- entity-resolution inputs; - public technology/company signals for account intelligence. -j0b/Work owns opportunities, roles, requirements, consultants, matches, submissions and commercial decisions. +j0b owns opportunities, roles, requirements, consultants, matches and submissions. ### PRESS0 -0pt may supply public-source acquisition, official releases/datasets, source snapshots and factual dataset/source metadata. PRESS0 owns articles, citations, claims and editorial/publication decisions. +- public source monitoring/acquisition; +- official releases/datasets; +- source snapshots/enrichment; +- factual dataset/source metadata. + +PRESS0 owns articles/citations/claims/editorial decisions. ### FUND0 -0pt may supply public issuer/company/market facts and registry enrichment. FUND0 owns IR projects, disclosures, investor access and data-room state and remains IR_ONLY. +- issuer/company/public market facts used in project materials; +- public registry/company enrichment. -### FIT0 / CRM +FUND0 owns project/disclosure/investor/data-room state. -0pt may supply dated account enrichment and market/buying signals plus entity-candidate evidence. FIT0 owns account/opportunity scoring. CRM remains a projection/operational interaction surface, not the external-fact source of truth. +### FIT0/CRM + +- account enrichment and market/buying signals; +- company identity resolution. ## 2. Provenance-first invariant -Every externally derived fact or observation must preserve enough lineage to answer: -- source/provider; -- external identifier or URL where appropriate; -- observed/fetched time; -- parser/transform version; -- freshness/confidence where meaningful; -- rights/reuse/retention classification; -- payload/artifact hash or ST0RE ref where raw retention is required. +Retain the existing source hierarchy/principles and ensure each field/observation can carry source, observed/fetched time, freshness/confidence and transformation/version metadata. -No enrichment silently overwrites a higher-authority canonical fact without an explicit precedence/provenance rule. +No enrichment should silently overwrite a higher-authority canonical fact without provenance/precedence rules. ## 3. Lossless acquisition contract -When a consumer needs raw upstream retention, 0pt returns or records an immutable acquisition/artifact reference plus metadata rather than retaining only a lossy normalized subset. ST0RE is the preferred custody plane for governed raw payloads once that integration is live verified. +Where a product needs raw upstream retention, 0pt should return/store an immutable acquisition/artifact reference plus metadata rather than only a lossy normalized subset. ST0RE custody is preferred for large raw payloads when integrated. + +Target acquisition receipt fields include: -Target acquisition-receipt fields: - source/provider; - external ID/URL; - fetched/observed times; @@ -56,55 +58,40 @@ Target acquisition-receipt fields: - parser/version; - rights/reuse/retention classification; - data classification; -- extracted observations/provenance; -- correlation/tenant/project scope. +- extracted observations/provenance. -## 4. Canonical entity-resolution architecture +## 4. Entity resolution -There must be one decision architecture, not competing production scorers. +Add/use Splink or equivalent approved probabilistic entity-resolution capability where it materially improves cross-source linking. -### 4.1 Synchronous candidate path +Primary use cases: -The canonical synchronous engine is the TypeScript `EntityResolutionService` exposed through typed HTTP surfaces. It provides: -- deterministic blocking; -- explainable similarity evidence; -- product hard-rule hooks; -- review-band candidates; -- stable-ID hard match/non-match rules at the product boundary. +- Brainville company = LinkedIn-export/company = JobTech employer = registry company = TED buyer = Twenty account; +- duplicate/similar organizations across countries/registries; +- person/consultant references when legally/purpose appropriate. -For the Work/company route, fuzzy/similarity evidence is never identity proof. Score-only candidates are review evidence only; only explicit stable-identifier hard rules may emit an automatic match. +Entity resolution returns candidates/scores/evidence. Product/canonical identity merge remains policy/domain controlled; do not auto-merge high-risk records solely on fuzzy score. -### 4.2 Python ingestion path - -The Python pipeline is a thin adapter to the canonical engine. It must not maintain a second fuzzy identity scorer. If the canonical bridge is unavailable, it returns records unmerged. At the consumer boundary it accepts only hard-rule matches and preserves hard non-match constraints across transitive clustering. - -### 4.3 Heavy/statistical ER - -Heavy probabilistic linkage (for example Splink when the exact runtime/license/security tuple is approved) belongs behind the 0pt → N0D batch job contract. Statistical output is candidate evidence, not unilateral canonical identity mutation. Production certification of a runtime is separate from implementing the contract. - -## 5. Strong company identity keys +## 5. Company identity keys Prefer strong keys where available: -- LEI (global scope); + - country + official registration number; -- VAT/official registry identifiers with their correct jurisdictional scope; -- other verified registry IDs. +- LEI; +- VAT ID; +- official registry IDs; +- verified domains. -National registration numbers must never be treated as globally unique without jurisdiction. Verified domains may be supporting evidence but are not equivalent to a statutory identifier. +Use probabilistic attributes only when strong IDs are missing/inconsistent. -Negative comparable stable-ID evidence dominates positive similarity and must prevent transitive cluster merges. +## 6. Public procurement enrichment -## 6. Human review boundary +Support acquisition/normalization references for buyer/supplier/award/contract datasets such as TED/OCDS/official sources. j0b procurement domain owns the commercial opportunity/access graph; 0pt supplies sourced external facts. -Similarity/fuzzy candidates may be ranked and explained. They do not auto-merge high-risk/canonical company or person identity. Review decisions must be attributable, tenant scoped and auditable when persistence is introduced. - -## 7. Public procurement enrichment - -Support acquisition/normalization references for buyer, supplier, award and contract datasets such as TED/OCDS/official sources. j0b owns commercial opportunity/access logic; 0pt supplies sourced external facts and entity evidence. - -## 8. Source-policy metadata +## 7. Source-policy metadata Products may pass/receive policy metadata for: + - access method; - reuse/redistribution; - retention; @@ -112,19 +99,20 @@ Products may pass/receive policy metadata for: - terms/license review date; - purpose/data classification. -0pt never assumes that publicly reachable content is automatically reusable for every commercial or publication purpose. +0pt must not globally assume all publicly reachable web content is reusable for every commercial/public purpose. -## 9. PRESS0 rights boundary +## 8. PRESS0 rights boundary -Acquisition/extraction is not publication permission. PRESS0 decides what may be quoted, reproduced or published. 0pt preserves source/right metadata needed for that decision. +Source acquisition/extraction is not publication permission. PRESS0 editorial/source-rights policy decides what may be quoted/reproduced/published. 0pt must preserve enough source metadata to support this decision. -## 10. Company verification boundary +## 9. Company verification boundary -0pt may fetch and reconcile official company facts. ID0 owns verification/KYB decisions. j0b/CRM consume explicit verification references/status; they must not infer `verified` merely because 0pt found a matching registry row. +0pt may fetch/enrich official company facts; ID0 owns verification decisions/KYB status. j0b/CRM consume verification refs/status from ID0, not infer `verified` merely because 0pt found a matching registry row. -## 11. Market/account signals +## 10. Market/account signals + +Target observations useful to FIT0/CRM/j0b: -Useful dated observations include: - hiring activity; - public procurement notices/awards; - public technology/platform signals; @@ -132,66 +120,58 @@ Useful dated observations include: - leadership/public contact changes where lawful; - relevant product/market announcements. -Signals are provenance-bearing observations with observation time and freshness, not permanent assertions. +Signals are dated observations with provenance, not permanent assertions. -## 12. API contract +## 11. API contract + +Fit into actual 0pt architecture. Product consumers should be able to request/receive: -Generic typed capabilities should cover: - company/entity enrichment; - source acquisition receipt; - structured observations with provenance; - entity-resolution candidates; -- batch ER job submission; -- provider/source health/freshness; -- durable dataset/version/search surfaces where implemented. +- provider/source health/freshness where applicable. -Avoid bespoke copies per product when a generic contract is sufficient. Product consumers use APIs/contracts rather than direct database coupling. +Avoid bespoke endpoint copies for each product if generic typed capabilities suffice. -## 13. Security/privacy +## 12. Security/privacy -Required controls include: -- tenant/project/purpose scoping; +- tenant/purpose scoping; - PII minimisation; -- controlled egress/allowed-host policy; -- SSRF protection for user-supplied URLs; -- canonical secret storage; +- egress/allowed-host controls; +- SSRF prevention for user-supplied URLs; +- secrets in canonical secret store; - no raw sensitive payloads in logs; - retention/deletion support; -- provider/license/data-rights documentation; -- fail-closed identity merge behavior; -- append-only evidence for security/lineage-sensitive mutations when AUD0 integration is required. +- provider/license/data-rights documentation. -## 14. Production gate +## 13. Current deployment gate -Code or architecture does not equal deployment. A capability progresses honestly through IMPLEMENTED → TESTED → DEPLOYED → LIVE_VERIFIED. +Before j0b/PRESS0/FUND0 depend on 0pt as a mission-critical production service, reconcile the repo's current deployment/live endpoint status and close any documented routing/auth/hosting gap. Do not treat architecture/code as deployed merely because it exists. -Before another product treats 0pt as a mission-critical dependency, the exact branch/commit must pass canonical Gitea gates and the relevant live endpoint/dependency must be verified. Protected workflow changes, database migrations, AUD0 outbox delivery and real external dependencies require canonical runner/live evidence. +## 14. Implementation order -## 15. Implementation order +1. Reconcile current live deployment, auth and data/provenance APIs. +2. Stabilise generic acquisition receipt/provenance contract. +3. Add Splink/entity-resolution capability with strong-key-first strategy and reviewable candidates. +4. Wire j0b company/end-client/buyer enrichment. +5. Wire FIT0/CRM account signals. +6. Add PRESS0 source/dataset acquisition when press0 exists. +7. Add FUND0 public issuer facts when fund0 exists. +8. Load/performance/backup/restore/live verify. -1. Reconcile canonical Gitea/main and eliminate duplicate engines/owners. -2. Keep the synchronous TS ER engine as the canonical candidate/decision contract. -3. Keep Python ingestion as a fail-closed adapter, never a second scorer. -4. Complete durable acquisition/provenance/dataset-version/search contracts. -5. Certify a heavy probabilistic ER runtime through N0D only after runtime/license/security review. -6. Wire j0b company/end-client/buyer enrichment through typed APIs. -7. Wire FIT0/CRM signals as projections/observations, not shared mutable truth. -8. Add PRESS0/FUND0 consumers only when their repos/contracts are real. -9. Load/performance/backup/restore/live-verify every mission-critical path. - -## 16. Definition of Done per data capability +## 15. Definition of Done per data capability - source/provider documented; - legal/license/data-use classification recorded; -- provenance/freshness/confidence available where meaningful; +- provenance/freshness/confidence available; - deterministic/idempotent normalization where expected; -- entity resolution tested against labelled truth data; -- fuzzy evidence cannot silently become identity truth; -- tenant/purpose/security negative tests pass; -- provider outage/schema drift handled fail closed or explicitly degraded; +- entity resolution tested against labelled truth set; +- tenant/purpose/security tests pass; +- provider outage/schema drift handled; - raw artifact retention/deletion contract works where required; -- exact production endpoint/dependency is live verified; -- product consumes API/contract, not another product's DB; -- status documentation reflects observed reality. +- actual production endpoint/dependency is live verified; +- product consumes API/contract, not DB; +- status docs reflect reality. -Target ecosystem uses in this document do not claim they are already wired or live. \ No newline at end of file +Target ecosystem uses in this document do not claim they are already wired. \ No newline at end of file