docs: add ecosystem data acquisition and entity plan
This commit is contained in:
@@ -1,54 +1,56 @@
|
||||
# 0pt — SIAX Digital Ecosystem Data Acquisition & Entity Plan
|
||||
|
||||
Status: IMPLEMENTATION CONTRACT / PRODUCTION GATES OPEN
|
||||
Status: TARGET / IMPLEMENTATION CONTRACT
|
||||
Date: 2026-09-07
|
||||
|
||||
0pt is the generic external-data acquisition, enrichment, provenance and entity-candidate capability. Product domains own their business entities and decisions. 0pt supplies sourced observations, acquisition receipts and entity-resolution evidence; it must not become a second Work/PRESS0/FUND0/CRM system of record.
|
||||
0pt remains the generic external-data acquisition/enrichment/provenance capability. Product domains own their business entities and decisions. 0pt supplies sourced observations/enrichment/entity-resolution inputs and must not become a second Work/Press/Fund/CRM SSOT.
|
||||
|
||||
## 1. Ecosystem ownership
|
||||
## 1. Ecosystem uses
|
||||
|
||||
### Work / j0b
|
||||
|
||||
0pt may supply:
|
||||
- company/employer/end-client enrichment;
|
||||
- registry/company facts;
|
||||
- generic source acquisition;
|
||||
- source acquisition where generic 0pt infrastructure is appropriate;
|
||||
- buyer/supplier enrichment for procurement;
|
||||
- entity-resolution candidates/evidence;
|
||||
- entity-resolution inputs;
|
||||
- public technology/company signals for account intelligence.
|
||||
|
||||
j0b/Work owns opportunities, roles, requirements, consultants, matches, submissions and commercial decisions.
|
||||
j0b owns opportunities, roles, requirements, consultants, matches and submissions.
|
||||
|
||||
### PRESS0
|
||||
|
||||
0pt may supply public-source acquisition, official releases/datasets, source snapshots and factual dataset/source metadata. PRESS0 owns articles, citations, claims and editorial/publication decisions.
|
||||
- public source monitoring/acquisition;
|
||||
- official releases/datasets;
|
||||
- source snapshots/enrichment;
|
||||
- factual dataset/source metadata.
|
||||
|
||||
PRESS0 owns articles/citations/claims/editorial decisions.
|
||||
|
||||
### FUND0
|
||||
|
||||
0pt may supply public issuer/company/market facts and registry enrichment. FUND0 owns IR projects, disclosures, investor access and data-room state and remains IR_ONLY.
|
||||
- issuer/company/public market facts used in project materials;
|
||||
- public registry/company enrichment.
|
||||
|
||||
### FIT0 / CRM
|
||||
FUND0 owns project/disclosure/investor/data-room state.
|
||||
|
||||
0pt may supply dated account enrichment and market/buying signals plus entity-candidate evidence. FIT0 owns account/opportunity scoring. CRM remains a projection/operational interaction surface, not the external-fact source of truth.
|
||||
### FIT0/CRM
|
||||
|
||||
- account enrichment and market/buying signals;
|
||||
- company identity resolution.
|
||||
|
||||
## 2. Provenance-first invariant
|
||||
|
||||
Every externally derived fact or observation must preserve enough lineage to answer:
|
||||
- source/provider;
|
||||
- external identifier or URL where appropriate;
|
||||
- observed/fetched time;
|
||||
- parser/transform version;
|
||||
- freshness/confidence where meaningful;
|
||||
- rights/reuse/retention classification;
|
||||
- payload/artifact hash or ST0RE ref where raw retention is required.
|
||||
Retain the existing source hierarchy/principles and ensure each field/observation can carry source, observed/fetched time, freshness/confidence and transformation/version metadata.
|
||||
|
||||
No enrichment silently overwrites a higher-authority canonical fact without an explicit precedence/provenance rule.
|
||||
No enrichment should silently overwrite a higher-authority canonical fact without provenance/precedence rules.
|
||||
|
||||
## 3. Lossless acquisition contract
|
||||
|
||||
When a consumer needs raw upstream retention, 0pt returns or records an immutable acquisition/artifact reference plus metadata rather than retaining only a lossy normalized subset. ST0RE is the preferred custody plane for governed raw payloads once that integration is live verified.
|
||||
Where a product needs raw upstream retention, 0pt should return/store an immutable acquisition/artifact reference plus metadata rather than only a lossy normalized subset. ST0RE custody is preferred for large raw payloads when integrated.
|
||||
|
||||
Target acquisition receipt fields include:
|
||||
|
||||
Target acquisition-receipt fields:
|
||||
- source/provider;
|
||||
- external ID/URL;
|
||||
- fetched/observed times;
|
||||
@@ -56,55 +58,40 @@ Target acquisition-receipt fields:
|
||||
- parser/version;
|
||||
- rights/reuse/retention classification;
|
||||
- data classification;
|
||||
- extracted observations/provenance;
|
||||
- correlation/tenant/project scope.
|
||||
- extracted observations/provenance.
|
||||
|
||||
## 4. Canonical entity-resolution architecture
|
||||
## 4. Entity resolution
|
||||
|
||||
There must be one decision architecture, not competing production scorers.
|
||||
Add/use Splink or equivalent approved probabilistic entity-resolution capability where it materially improves cross-source linking.
|
||||
|
||||
### 4.1 Synchronous candidate path
|
||||
Primary use cases:
|
||||
|
||||
The canonical synchronous engine is the TypeScript `EntityResolutionService` exposed through typed HTTP surfaces. It provides:
|
||||
- deterministic blocking;
|
||||
- explainable similarity evidence;
|
||||
- product hard-rule hooks;
|
||||
- review-band candidates;
|
||||
- stable-ID hard match/non-match rules at the product boundary.
|
||||
- Brainville company = LinkedIn-export/company = JobTech employer = registry company = TED buyer = Twenty account;
|
||||
- duplicate/similar organizations across countries/registries;
|
||||
- person/consultant references when legally/purpose appropriate.
|
||||
|
||||
For the Work/company route, fuzzy/similarity evidence is never identity proof. Score-only candidates are review evidence only; only explicit stable-identifier hard rules may emit an automatic match.
|
||||
Entity resolution returns candidates/scores/evidence. Product/canonical identity merge remains policy/domain controlled; do not auto-merge high-risk records solely on fuzzy score.
|
||||
|
||||
### 4.2 Python ingestion path
|
||||
|
||||
The Python pipeline is a thin adapter to the canonical engine. It must not maintain a second fuzzy identity scorer. If the canonical bridge is unavailable, it returns records unmerged. At the consumer boundary it accepts only hard-rule matches and preserves hard non-match constraints across transitive clustering.
|
||||
|
||||
### 4.3 Heavy/statistical ER
|
||||
|
||||
Heavy probabilistic linkage (for example Splink when the exact runtime/license/security tuple is approved) belongs behind the 0pt → N0D batch job contract. Statistical output is candidate evidence, not unilateral canonical identity mutation. Production certification of a runtime is separate from implementing the contract.
|
||||
|
||||
## 5. Strong company identity keys
|
||||
## 5. Company identity keys
|
||||
|
||||
Prefer strong keys where available:
|
||||
- LEI (global scope);
|
||||
|
||||
- country + official registration number;
|
||||
- VAT/official registry identifiers with their correct jurisdictional scope;
|
||||
- other verified registry IDs.
|
||||
- LEI;
|
||||
- VAT ID;
|
||||
- official registry IDs;
|
||||
- verified domains.
|
||||
|
||||
National registration numbers must never be treated as globally unique without jurisdiction. Verified domains may be supporting evidence but are not equivalent to a statutory identifier.
|
||||
Use probabilistic attributes only when strong IDs are missing/inconsistent.
|
||||
|
||||
Negative comparable stable-ID evidence dominates positive similarity and must prevent transitive cluster merges.
|
||||
## 6. Public procurement enrichment
|
||||
|
||||
## 6. Human review boundary
|
||||
Support acquisition/normalization references for buyer/supplier/award/contract datasets such as TED/OCDS/official sources. j0b procurement domain owns the commercial opportunity/access graph; 0pt supplies sourced external facts.
|
||||
|
||||
Similarity/fuzzy candidates may be ranked and explained. They do not auto-merge high-risk/canonical company or person identity. Review decisions must be attributable, tenant scoped and auditable when persistence is introduced.
|
||||
|
||||
## 7. Public procurement enrichment
|
||||
|
||||
Support acquisition/normalization references for buyer, supplier, award and contract datasets such as TED/OCDS/official sources. j0b owns commercial opportunity/access logic; 0pt supplies sourced external facts and entity evidence.
|
||||
|
||||
## 8. Source-policy metadata
|
||||
## 7. Source-policy metadata
|
||||
|
||||
Products may pass/receive policy metadata for:
|
||||
|
||||
- access method;
|
||||
- reuse/redistribution;
|
||||
- retention;
|
||||
@@ -112,19 +99,20 @@ Products may pass/receive policy metadata for:
|
||||
- terms/license review date;
|
||||
- purpose/data classification.
|
||||
|
||||
0pt never assumes that publicly reachable content is automatically reusable for every commercial or publication purpose.
|
||||
0pt must not globally assume all publicly reachable web content is reusable for every commercial/public purpose.
|
||||
|
||||
## 9. PRESS0 rights boundary
|
||||
## 8. PRESS0 rights boundary
|
||||
|
||||
Acquisition/extraction is not publication permission. PRESS0 decides what may be quoted, reproduced or published. 0pt preserves source/right metadata needed for that decision.
|
||||
Source acquisition/extraction is not publication permission. PRESS0 editorial/source-rights policy decides what may be quoted/reproduced/published. 0pt must preserve enough source metadata to support this decision.
|
||||
|
||||
## 10. Company verification boundary
|
||||
## 9. Company verification boundary
|
||||
|
||||
0pt may fetch and reconcile official company facts. ID0 owns verification/KYB decisions. j0b/CRM consume explicit verification references/status; they must not infer `verified` merely because 0pt found a matching registry row.
|
||||
0pt may fetch/enrich official company facts; ID0 owns verification decisions/KYB status. j0b/CRM consume verification refs/status from ID0, not infer `verified` merely because 0pt found a matching registry row.
|
||||
|
||||
## 11. Market/account signals
|
||||
## 10. Market/account signals
|
||||
|
||||
Target observations useful to FIT0/CRM/j0b:
|
||||
|
||||
Useful dated observations include:
|
||||
- hiring activity;
|
||||
- public procurement notices/awards;
|
||||
- public technology/platform signals;
|
||||
@@ -132,66 +120,58 @@ Useful dated observations include:
|
||||
- leadership/public contact changes where lawful;
|
||||
- relevant product/market announcements.
|
||||
|
||||
Signals are provenance-bearing observations with observation time and freshness, not permanent assertions.
|
||||
Signals are dated observations with provenance, not permanent assertions.
|
||||
|
||||
## 12. API contract
|
||||
## 11. API contract
|
||||
|
||||
Fit into actual 0pt architecture. Product consumers should be able to request/receive:
|
||||
|
||||
Generic typed capabilities should cover:
|
||||
- company/entity enrichment;
|
||||
- source acquisition receipt;
|
||||
- structured observations with provenance;
|
||||
- entity-resolution candidates;
|
||||
- batch ER job submission;
|
||||
- provider/source health/freshness;
|
||||
- durable dataset/version/search surfaces where implemented.
|
||||
- provider/source health/freshness where applicable.
|
||||
|
||||
Avoid bespoke copies per product when a generic contract is sufficient. Product consumers use APIs/contracts rather than direct database coupling.
|
||||
Avoid bespoke endpoint copies for each product if generic typed capabilities suffice.
|
||||
|
||||
## 13. Security/privacy
|
||||
## 12. Security/privacy
|
||||
|
||||
Required controls include:
|
||||
- tenant/project/purpose scoping;
|
||||
- tenant/purpose scoping;
|
||||
- PII minimisation;
|
||||
- controlled egress/allowed-host policy;
|
||||
- SSRF protection for user-supplied URLs;
|
||||
- canonical secret storage;
|
||||
- egress/allowed-host controls;
|
||||
- SSRF prevention for user-supplied URLs;
|
||||
- secrets in canonical secret store;
|
||||
- no raw sensitive payloads in logs;
|
||||
- retention/deletion support;
|
||||
- provider/license/data-rights documentation;
|
||||
- fail-closed identity merge behavior;
|
||||
- append-only evidence for security/lineage-sensitive mutations when AUD0 integration is required.
|
||||
- provider/license/data-rights documentation.
|
||||
|
||||
## 14. Production gate
|
||||
## 13. Current deployment gate
|
||||
|
||||
Code or architecture does not equal deployment. A capability progresses honestly through IMPLEMENTED → TESTED → DEPLOYED → LIVE_VERIFIED.
|
||||
Before j0b/PRESS0/FUND0 depend on 0pt as a mission-critical production service, reconcile the repo's current deployment/live endpoint status and close any documented routing/auth/hosting gap. Do not treat architecture/code as deployed merely because it exists.
|
||||
|
||||
Before another product treats 0pt as a mission-critical dependency, the exact branch/commit must pass canonical Gitea gates and the relevant live endpoint/dependency must be verified. Protected workflow changes, database migrations, AUD0 outbox delivery and real external dependencies require canonical runner/live evidence.
|
||||
## 14. Implementation order
|
||||
|
||||
## 15. Implementation order
|
||||
1. Reconcile current live deployment, auth and data/provenance APIs.
|
||||
2. Stabilise generic acquisition receipt/provenance contract.
|
||||
3. Add Splink/entity-resolution capability with strong-key-first strategy and reviewable candidates.
|
||||
4. Wire j0b company/end-client/buyer enrichment.
|
||||
5. Wire FIT0/CRM account signals.
|
||||
6. Add PRESS0 source/dataset acquisition when press0 exists.
|
||||
7. Add FUND0 public issuer facts when fund0 exists.
|
||||
8. Load/performance/backup/restore/live verify.
|
||||
|
||||
1. Reconcile canonical Gitea/main and eliminate duplicate engines/owners.
|
||||
2. Keep the synchronous TS ER engine as the canonical candidate/decision contract.
|
||||
3. Keep Python ingestion as a fail-closed adapter, never a second scorer.
|
||||
4. Complete durable acquisition/provenance/dataset-version/search contracts.
|
||||
5. Certify a heavy probabilistic ER runtime through N0D only after runtime/license/security review.
|
||||
6. Wire j0b company/end-client/buyer enrichment through typed APIs.
|
||||
7. Wire FIT0/CRM signals as projections/observations, not shared mutable truth.
|
||||
8. Add PRESS0/FUND0 consumers only when their repos/contracts are real.
|
||||
9. Load/performance/backup/restore/live-verify every mission-critical path.
|
||||
|
||||
## 16. Definition of Done per data capability
|
||||
## 15. Definition of Done per data capability
|
||||
|
||||
- source/provider documented;
|
||||
- legal/license/data-use classification recorded;
|
||||
- provenance/freshness/confidence available where meaningful;
|
||||
- provenance/freshness/confidence available;
|
||||
- deterministic/idempotent normalization where expected;
|
||||
- entity resolution tested against labelled truth data;
|
||||
- fuzzy evidence cannot silently become identity truth;
|
||||
- tenant/purpose/security negative tests pass;
|
||||
- provider outage/schema drift handled fail closed or explicitly degraded;
|
||||
- entity resolution tested against labelled truth set;
|
||||
- tenant/purpose/security tests pass;
|
||||
- provider outage/schema drift handled;
|
||||
- raw artifact retention/deletion contract works where required;
|
||||
- exact production endpoint/dependency is live verified;
|
||||
- product consumes API/contract, not another product's DB;
|
||||
- status documentation reflects observed reality.
|
||||
- actual production endpoint/dependency is live verified;
|
||||
- product consumes API/contract, not DB;
|
||||
- status docs reflect reality.
|
||||
|
||||
Target ecosystem uses in this document do not claim they are already wired or live.
|
||||
Target ecosystem uses in this document do not claim they are already wired.
|
||||
Reference in New Issue
Block a user