docs: add ecosystem data acquisition and entity plan
This commit is contained in:
@@ -0,0 +1,177 @@
|
||||
# 0pt — SIAX Digital Ecosystem Data Acquisition & Entity Plan
|
||||
|
||||
Status: TARGET / IMPLEMENTATION CONTRACT
|
||||
Date: 2026-09-07
|
||||
|
||||
0pt remains the generic external-data acquisition/enrichment/provenance capability. Product domains own their business entities and decisions. 0pt supplies sourced observations/enrichment/entity-resolution inputs and must not become a second Work/Press/Fund/CRM SSOT.
|
||||
|
||||
## 1. Ecosystem uses
|
||||
|
||||
### Work / j0b
|
||||
|
||||
- company/employer/end-client enrichment;
|
||||
- registry/company facts;
|
||||
- source acquisition where generic 0pt infrastructure is appropriate;
|
||||
- buyer/supplier enrichment for procurement;
|
||||
- entity-resolution inputs;
|
||||
- public technology/company signals for account intelligence.
|
||||
|
||||
j0b owns opportunities, roles, requirements, consultants, matches and submissions.
|
||||
|
||||
### PRESS0
|
||||
|
||||
- public source monitoring/acquisition;
|
||||
- official releases/datasets;
|
||||
- source snapshots/enrichment;
|
||||
- factual dataset/source metadata.
|
||||
|
||||
PRESS0 owns articles/citations/claims/editorial decisions.
|
||||
|
||||
### FUND0
|
||||
|
||||
- issuer/company/public market facts used in project materials;
|
||||
- public registry/company enrichment.
|
||||
|
||||
FUND0 owns project/disclosure/investor/data-room state.
|
||||
|
||||
### FIT0/CRM
|
||||
|
||||
- account enrichment and market/buying signals;
|
||||
- company identity resolution.
|
||||
|
||||
## 2. Provenance-first invariant
|
||||
|
||||
Retain the existing source hierarchy/principles and ensure each field/observation can carry source, observed/fetched time, freshness/confidence and transformation/version metadata.
|
||||
|
||||
No enrichment should silently overwrite a higher-authority canonical fact without provenance/precedence rules.
|
||||
|
||||
## 3. Lossless acquisition contract
|
||||
|
||||
Where a product needs raw upstream retention, 0pt should return/store an immutable acquisition/artifact reference plus metadata rather than only a lossy normalized subset. ST0RE custody is preferred for large raw payloads when integrated.
|
||||
|
||||
Target acquisition receipt fields include:
|
||||
|
||||
- source/provider;
|
||||
- external ID/URL;
|
||||
- fetched/observed times;
|
||||
- payload/artifact hash/ref;
|
||||
- parser/version;
|
||||
- rights/reuse/retention classification;
|
||||
- data classification;
|
||||
- extracted observations/provenance.
|
||||
|
||||
## 4. Entity resolution
|
||||
|
||||
Add/use Splink or equivalent approved probabilistic entity-resolution capability where it materially improves cross-source linking.
|
||||
|
||||
Primary use cases:
|
||||
|
||||
- Brainville company = LinkedIn-export/company = JobTech employer = registry company = TED buyer = Twenty account;
|
||||
- duplicate/similar organizations across countries/registries;
|
||||
- person/consultant references when legally/purpose appropriate.
|
||||
|
||||
Entity resolution returns candidates/scores/evidence. Product/canonical identity merge remains policy/domain controlled; do not auto-merge high-risk records solely on fuzzy score.
|
||||
|
||||
## 5. Company identity keys
|
||||
|
||||
Prefer strong keys where available:
|
||||
|
||||
- country + official registration number;
|
||||
- LEI;
|
||||
- VAT ID;
|
||||
- official registry IDs;
|
||||
- verified domains.
|
||||
|
||||
Use probabilistic attributes only when strong IDs are missing/inconsistent.
|
||||
|
||||
## 6. Public procurement enrichment
|
||||
|
||||
Support acquisition/normalization references for buyer/supplier/award/contract datasets such as TED/OCDS/official sources. j0b procurement domain owns the commercial opportunity/access graph; 0pt supplies sourced external facts.
|
||||
|
||||
## 7. Source-policy metadata
|
||||
|
||||
Products may pass/receive policy metadata for:
|
||||
|
||||
- access method;
|
||||
- reuse/redistribution;
|
||||
- retention;
|
||||
- polling/rate limits;
|
||||
- terms/license review date;
|
||||
- purpose/data classification.
|
||||
|
||||
0pt must not globally assume all publicly reachable web content is reusable for every commercial/public purpose.
|
||||
|
||||
## 8. PRESS0 rights boundary
|
||||
|
||||
Source acquisition/extraction is not publication permission. PRESS0 editorial/source-rights policy decides what may be quoted/reproduced/published. 0pt must preserve enough source metadata to support this decision.
|
||||
|
||||
## 9. Company verification boundary
|
||||
|
||||
0pt may fetch/enrich official company facts; ID0 owns verification decisions/KYB status. j0b/CRM consume verification refs/status from ID0, not infer `verified` merely because 0pt found a matching registry row.
|
||||
|
||||
## 10. Market/account signals
|
||||
|
||||
Target observations useful to FIT0/CRM/j0b:
|
||||
|
||||
- hiring activity;
|
||||
- public procurement notices/awards;
|
||||
- public technology/platform signals;
|
||||
- company growth/change events;
|
||||
- leadership/public contact changes where lawful;
|
||||
- relevant product/market announcements.
|
||||
|
||||
Signals are dated observations with provenance, not permanent assertions.
|
||||
|
||||
## 11. API contract
|
||||
|
||||
Fit into actual 0pt architecture. Product consumers should be able to request/receive:
|
||||
|
||||
- company/entity enrichment;
|
||||
- source acquisition receipt;
|
||||
- structured observations with provenance;
|
||||
- entity-resolution candidates;
|
||||
- provider/source health/freshness where applicable.
|
||||
|
||||
Avoid bespoke endpoint copies for each product if generic typed capabilities suffice.
|
||||
|
||||
## 12. Security/privacy
|
||||
|
||||
- tenant/purpose scoping;
|
||||
- PII minimisation;
|
||||
- egress/allowed-host controls;
|
||||
- SSRF prevention for user-supplied URLs;
|
||||
- secrets in canonical secret store;
|
||||
- no raw sensitive payloads in logs;
|
||||
- retention/deletion support;
|
||||
- provider/license/data-rights documentation.
|
||||
|
||||
## 13. Current deployment gate
|
||||
|
||||
Before j0b/PRESS0/FUND0 depend on 0pt as a mission-critical production service, reconcile the repo's current deployment/live endpoint status and close any documented routing/auth/hosting gap. Do not treat architecture/code as deployed merely because it exists.
|
||||
|
||||
## 14. Implementation order
|
||||
|
||||
1. Reconcile current live deployment, auth and data/provenance APIs.
|
||||
2. Stabilise generic acquisition receipt/provenance contract.
|
||||
3. Add Splink/entity-resolution capability with strong-key-first strategy and reviewable candidates.
|
||||
4. Wire j0b company/end-client/buyer enrichment.
|
||||
5. Wire FIT0/CRM account signals.
|
||||
6. Add PRESS0 source/dataset acquisition when press0 exists.
|
||||
7. Add FUND0 public issuer facts when fund0 exists.
|
||||
8. Load/performance/backup/restore/live verify.
|
||||
|
||||
## 15. Definition of Done per data capability
|
||||
|
||||
- source/provider documented;
|
||||
- legal/license/data-use classification recorded;
|
||||
- provenance/freshness/confidence available;
|
||||
- deterministic/idempotent normalization where expected;
|
||||
- entity resolution tested against labelled truth set;
|
||||
- tenant/purpose/security tests pass;
|
||||
- provider outage/schema drift handled;
|
||||
- raw artifact retention/deletion contract works where required;
|
||||
- actual production endpoint/dependency is live verified;
|
||||
- product consumes API/contract, not DB;
|
||||
- status docs reflect reality.
|
||||
|
||||
Target ecosystem uses in this document do not claim they are already wired.
|
||||
Reference in New Issue
Block a user