feat(0pt): PII-Gate GDPR-guardrail for personal-data sources (fail-closed)
PiiGateService makes any future personal-data source (ratsit.se and other directory domains) technically GDPR-OK before the first byte is stored: - recorded legal basis (art. 6) with basis_ref evidence, no policy => fail-closed - data minimisation via field allowlist (art. 5(1)(c)) - storage limitation + enforceRetention purge (art. 5(1)(e)) - sha256-only identity keys + log redaction (art. 25/32) - accessStatement / erase subject-rights surfaces (art. 15/17) - robots/rate confirmation required for web+api sources; no stealth/bypass - audit trail via AUD0 at risk L3 (0pt.pii_*) - gate enabled by default in SourceQueryService 10 new tests; full suite 166 pass | 1 skip; typecheck (blocking gate) clean.
This commit is contained in:
@@ -0,0 +1,58 @@
|
|||||||
|
# PII Guardrail — GDPR requirement for personal-data sources (0pt)
|
||||||
|
|
||||||
|
Status: **enforced, fail-closed.** No 0pt pipeline is crawling personuppgifter
|
||||||
|
today; the `PiiGateService` ships so that any future source that touches
|
||||||
|
personal data (Ratsit, Merinfo, Hitta, …) is **technically GDPR-OK before the
|
||||||
|
first byte is stored**.
|
||||||
|
|
||||||
|
## The rule
|
||||||
|
|
||||||
|
1. **Classify.** A source is personal-data if it self-flags
|
||||||
|
(`touches_personal_data`) or is a known directory domain (ratsit.se,
|
||||||
|
merinfo.se, hitta.se, mrkoll.se, eniro.se, birthday.se, upplysning.se).
|
||||||
|
2. **No policy → no processing.** `assertCanStore()` throws unless a
|
||||||
|
`PiiPolicy` with a valid `legal_basis` + `basis_ref` is on record.
|
||||||
|
3. **Everything is evidenced.** Each decision is emitted to AUD0 at risk L3
|
||||||
|
(`0pt.pii_policy_created | pii_data_retained | pii_erased |
|
||||||
|
pii_retention_purged`).
|
||||||
|
|
||||||
|
## GDPR article mapping
|
||||||
|
|
||||||
|
| Requirement | Implementation |
|
||||||
|
|---|---|
|
||||||
|
| Legal basis — art. 6 | `legal_basis` ∈ legitimate_interest/consent/contract/legal_obligation + `basis_ref` (LIA doc id, consent record id, contract clause, provision). Cannot be empty. |
|
||||||
|
| Data minimisation — art. 5(1)(c) | `field_allowlist`; storing any other field throws `PiiMinimisationError`. Empty allowlist = nothing stored. |
|
||||||
|
| Storage limitation — art. 5(1)(e) | `retention_days`; `enforceRetention()` purges expired rows. `0` = fetch-and-drop, nothing persisted. |
|
||||||
|
| Integrity/confidentiality — art. 32 | Only sha256 hashes of identity keys stored (`recordStored`); plaintext identity never written; log/trace redaction (`redact()`) for personnummer/e-mail/phone. |
|
||||||
|
| Right of access — art. 15 | `accessStatement(subjectHash)` — what is held on a subject, across sources. |
|
||||||
|
| Right to erasure — art. 17 | `erase({ subject_hash, reason })` tombstones every stored value for the subject. |
|
||||||
|
| Data protection by design — art. 25 | Gate defaults ON in `SourceQueryService`; no bypass knobs. |
|
||||||
|
|
||||||
|
## When someone requests a personal-data source (e.g. Ratsit)
|
||||||
|
|
||||||
|
- [x] Confirm the source is actually needed and no anonymised/official
|
||||||
|
alternative exists (official register > directory listing).
|
||||||
|
- [x] Write the **Legitimate Interest Assessment** (or record consent /
|
||||||
|
contract / legal obligation) and store its id in `basis_ref`.
|
||||||
|
LIA minimum: purpose, necessity, individuals' rights impact, mitigations.
|
||||||
|
- [x] **VERIFY current terms + robots.txt + availability** on the live site.
|
||||||
|
Do not hardcode "catsit.com allows" — re-check each enable.
|
||||||
|
- [x] Set `robots_and_rate_confirmed: true` + a `SourceRatePolicy` matching
|
||||||
|
the source's own limits (legacy N0D default for `ratsit.se` was `1/min` —
|
||||||
|
treat that as a ceiling, not a target).
|
||||||
|
- [x] Define the minimum `field_allowlist` (e.g. only `street_address`, not
|
||||||
|
`phone`).
|
||||||
|
- [x] Invoke `recordStored` for every stored value and run `enforceRetention`
|
||||||
|
on a schedule (e.g. daily cron).
|
||||||
|
- [x] Record a DPIA note if processing is "large scale" (art. 35) — bulk
|
||||||
|
directory scraping of living persons is large-scale by default.
|
||||||
|
|
||||||
|
## Never
|
||||||
|
|
||||||
|
- Pretend a source has a legal basis it does not have; `basis_ref` must point
|
||||||
|
to a real artifact.
|
||||||
|
- Store identity keys in plaintext (`pseudonymize_keys` defaults to true).
|
||||||
|
- Use stealth/bot-bypass/rate-evasion — excluded by CLAUDE.md §1 regardless of
|
||||||
|
GDPR.
|
||||||
|
- Claim compliance on a source we have not re-verified this season (`VERIFY`
|
||||||
|
markers are intentional).
|
||||||
@@ -0,0 +1,324 @@
|
|||||||
|
/**
|
||||||
|
* src/services/pii-gate-service.ts
|
||||||
|
* ==================================
|
||||||
|
* 0pt.pii_* — the GDPR guardrail (fail-closed). Enforced by default wherever
|
||||||
|
* a SourceQueryService is constructed; consulted by workers before they
|
||||||
|
* persist anything for a personal-data source.
|
||||||
|
*
|
||||||
|
* Owns 0pt.pii_policy.schema.v1 and the in-memory identity registry that
|
||||||
|
* backs the subject rights surfaces (access/erasure, GDPR art. 15/17) and
|
||||||
|
* the retention worker (GDPR art. 5(1)(e)). Every decision is emitted to
|
||||||
|
* AUD0 as risk L3 events so CL0UD gets an audit trail without polling 0pt.
|
||||||
|
*
|
||||||
|
* Directory domains known to publish personuppgifter (ratsit.se, …) are
|
||||||
|
* pre-classified as personal: until a PiiPolicy with a recorded lawful
|
||||||
|
* basis exists for them, the gate REJECTS storage/query of those sources.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { createHash } from "node:crypto";
|
||||||
|
import type {
|
||||||
|
PiiPolicy,
|
||||||
|
CreatePiiPolicyInput,
|
||||||
|
PiiIdentityKey,
|
||||||
|
PiiErasureRequest,
|
||||||
|
PiiAccessStatement,
|
||||||
|
} from "../types/pii";
|
||||||
|
import type { Source } from "../types/source";
|
||||||
|
import { createAud0Client, type Aud0Client, type Aud0Event } from "../clients/aud0-client";
|
||||||
|
|
||||||
|
/** Domain publishes personuppgifter (directory/register listings). */
|
||||||
|
const KNOWN_DIRECTORY_DOMAINS = [
|
||||||
|
"ratsit.se",
|
||||||
|
"merinfo.se",
|
||||||
|
"hitta.se",
|
||||||
|
"mrkoll.se",
|
||||||
|
"eniro.se",
|
||||||
|
"birthday.se",
|
||||||
|
"upplysning.se",
|
||||||
|
];
|
||||||
|
|
||||||
|
const DECISION_TEMPLATE = "PII source %s: %s";
|
||||||
|
|
||||||
|
export class PiiPolicyExistsError extends Error {}
|
||||||
|
export class PiiLegalBasisError extends Error {}
|
||||||
|
export class PiiMinimisationError extends Error {}
|
||||||
|
export class PiiRetentionError extends Error {}
|
||||||
|
export class PiiRobotsRateError extends Error {}
|
||||||
|
|
||||||
|
export class PiiGateService {
|
||||||
|
private policies: Map<string, PiiPolicy> = new Map();
|
||||||
|
private registry: PiiIdentityKey[] = [];
|
||||||
|
private aud0: Aud0Client;
|
||||||
|
|
||||||
|
constructor(opts?: { aud0BaseUrl?: string }) {
|
||||||
|
this.aud0 = createAud0Client(opts?.aud0BaseUrl);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** True if the source either self-flags or is a known directory domain. */
|
||||||
|
classifySourcePersonal(source: Pick<Source, "source_id" | "source_type" | "name" | "touches_personal_data">): boolean {
|
||||||
|
if (source.touches_personal_data === true) return true;
|
||||||
|
if (source.source_type !== "web" && source.source_type !== "api") return false;
|
||||||
|
const host = (source.name || "").toLowerCase().replace(/^https?:\/\//, "").replace(/\/.*$/, "");
|
||||||
|
return KNOWN_DIRECTORY_DOMAINS.some((d) => host === d || host.endsWith("." + d));
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ *
|
||||||
|
* Policies
|
||||||
|
* ------------------------------------------------------------------ */
|
||||||
|
|
||||||
|
async createPiiPolicy(input: CreatePiiPolicyInput): Promise<PiiPolicy> {
|
||||||
|
if (this.policies.has(input.source_id)) {
|
||||||
|
throw new Error(`PiiPolicy already exists for source ${input.source_id}`);
|
||||||
|
}
|
||||||
|
// Hard validation — a policy is the lawful-basis gate, so it cannot be
|
||||||
|
// written with a half-legal posture.
|
||||||
|
if (!input.basis_ref.trim()) throw new Error("basis_ref is required (LIA id, consent id, contract clause or legal provision)");
|
||||||
|
if (!Number.isFinite(input.retention_days) || input.retention_days < 0) {
|
||||||
|
throw new Error("retention_days must be a non-negative integer (0 = fetch-and-drop)");
|
||||||
|
}
|
||||||
|
if (!Array.isArray(input.field_allowlist) || input.field_allowlist.length === 0) {
|
||||||
|
throw new Error("field_allowlist must contain at least one field (data minimisation)");
|
||||||
|
}
|
||||||
|
|
||||||
|
const now = new Date().toISOString();
|
||||||
|
const policy: PiiPolicy = {
|
||||||
|
source_id: input.source_id,
|
||||||
|
tenant_id: input.tenant_id,
|
||||||
|
project_id: input.project_id,
|
||||||
|
touches_personal_data: true,
|
||||||
|
legal_basis: input.legal_basis,
|
||||||
|
basis_ref: input.basis_ref,
|
||||||
|
retention_days: input.retention_days,
|
||||||
|
field_allowlist: [...input.field_allowlist],
|
||||||
|
pseudonymize_keys: input.pseudonymize_keys ?? true,
|
||||||
|
robots_and_rate_confirmed: input.robots_and_rate_confirmed ?? false,
|
||||||
|
created_at: now,
|
||||||
|
updated_at: null,
|
||||||
|
created_by: input.created_by ?? null,
|
||||||
|
};
|
||||||
|
this.policies.set(input.source_id, policy);
|
||||||
|
|
||||||
|
await this.safeEmit({
|
||||||
|
event_id: `evt_0pt_pii_policy_created_${input.source_id}`,
|
||||||
|
event_type: "0pt.pii_policy_created",
|
||||||
|
capability_id: "0pt.pii_policy_created",
|
||||||
|
tenant_id: input.tenant_id,
|
||||||
|
project_id: input.project_id,
|
||||||
|
actor_id: input.created_by ?? "system",
|
||||||
|
resource_type: "source_pii_policy",
|
||||||
|
resource_id: input.source_id,
|
||||||
|
risk_level: "L3",
|
||||||
|
payload: { legal_basis: input.legal_basis, retention_days: input.retention_days, field_count: input.field_allowlist.length },
|
||||||
|
});
|
||||||
|
return policy;
|
||||||
|
}
|
||||||
|
|
||||||
|
async getPiiPolicy(sourceId: string): Promise<PiiPolicy | null> {
|
||||||
|
return this.policies.get(sourceId) ?? null;
|
||||||
|
}
|
||||||
|
|
||||||
|
async listPiiPolicies(tenantId?: string): Promise<PiiPolicy[]> {
|
||||||
|
const all = Array.from(this.policies.values());
|
||||||
|
return tenantId ? all.filter((p) => p.tenant_id === tenantId) : all;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ *
|
||||||
|
* Enforcement (fail-closed)
|
||||||
|
* ------------------------------------------------------------------ */
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Gate for a store/query of `fields` against `source`. Throws unless the
|
||||||
|
* source is non-personal or a lawful, minimised, retention-bounded,
|
||||||
|
* robots/rate-confirmed policy is on record.
|
||||||
|
*/
|
||||||
|
async assertCanStore(
|
||||||
|
source: Pick<Source, "source_id" | "source_type" | "name" | "touches_personal_data">,
|
||||||
|
fields: string[],
|
||||||
|
tenantId: string,
|
||||||
|
): Promise<void> {
|
||||||
|
if (!this.classifySourcePersonal(source)) return;
|
||||||
|
|
||||||
|
const policy = this.policies.get(source.source_id);
|
||||||
|
if (!policy) {
|
||||||
|
throw new PiiLegalBasisError(
|
||||||
|
DECISION_TEMPLATE.replace("%s", source.source_id).replace(
|
||||||
|
"%s",
|
||||||
|
`is a personal-data source with no PiiPolicy on record — a lawful basis (${KNOWN_DIRECTORY_DOMAINS.includes(source.name || "") ? "including robots.txt + rate limit confirmation" : "art. 6 GDPR"}) must be registered before any processing`,
|
||||||
|
),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (policy.retention_days === 0 && fields.length > 0) {
|
||||||
|
throw new PiiRetentionError(
|
||||||
|
DECISION_TEMPLATE.replace("%s", source.source_id).replace(
|
||||||
|
"%s",
|
||||||
|
"policy forbids retention (retention_days = 0); no fields may be stored",
|
||||||
|
),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
const disallowed = fields.filter((f) => !policy.field_allowlist.includes(f));
|
||||||
|
if (disallowed.length > 0) {
|
||||||
|
throw new PiiMinimisationError(
|
||||||
|
DECISION_TEMPLATE.replace("%s", source.source_id).replace(
|
||||||
|
"%s",
|
||||||
|
`field(s) outside the allowlist: ${disallowed.join(", ")} (GDPR art. 5(1)(c) minimisation)`,
|
||||||
|
),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
if ((source.source_type === "web" || source.source_type === "api") && !policy.robots_and_rate_confirmed) {
|
||||||
|
throw new PiiRobotsRateError(
|
||||||
|
DECISION_TEMPLATE.replace("%s", source.source_id).replace(
|
||||||
|
"%s",
|
||||||
|
"robots.txt / rate-limit not confirmed — required for web & api personal-data sources; re-check the source terms (do not hardcode)",
|
||||||
|
),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ *
|
||||||
|
* Subject rights: access (art. 15) & erasure (art. 17), retention (art. 5)
|
||||||
|
* ------------------------------------------------------------------ */
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Register that one value was stored. Only hashes are persisted; the
|
||||||
|
* plaintext identity never reaches the registry.
|
||||||
|
*/
|
||||||
|
async recordStored(input: {
|
||||||
|
source: Pick<Source, "source_id" | "source_type" | "name" | "touches_personal_data">;
|
||||||
|
tenantId: string;
|
||||||
|
identityKey: string;
|
||||||
|
subjectId: string;
|
||||||
|
field: string;
|
||||||
|
artifactRef?: string;
|
||||||
|
}): Promise<PiiIdentityKey> {
|
||||||
|
await this.assertCanStore(input.source, [input.field], input.tenantId);
|
||||||
|
const policy = this.policies.get(input.source.source_id)!;
|
||||||
|
|
||||||
|
const now = Date.now();
|
||||||
|
const key: PiiIdentityKey = {
|
||||||
|
key_hash: hash256(`${sourceIdSalt(input.source.source_id)}:${input.identityKey}`),
|
||||||
|
subject_hash: hash256(input.subjectId),
|
||||||
|
source_id: input.source.source_id,
|
||||||
|
tenant_id: input.tenantId,
|
||||||
|
field: input.field,
|
||||||
|
artifact_ref: input.artifactRef ?? null,
|
||||||
|
stored_at: new Date(now).toISOString(),
|
||||||
|
expires_at: new Date(now + policy.retention_days * 86_400_000).toISOString(),
|
||||||
|
status: "active",
|
||||||
|
};
|
||||||
|
this.registry.push(key);
|
||||||
|
|
||||||
|
await this.safeEmit({
|
||||||
|
event_id: `evt_0pt_pii_retained_${input.source.source_id}_${key.key_hash.slice(0, 12)}`,
|
||||||
|
event_type: "0pt.pii_data_retained",
|
||||||
|
capability_id: "0pt.pii_retain_identity",
|
||||||
|
tenant_id: input.tenantId,
|
||||||
|
project_id: "",
|
||||||
|
actor_id: "system",
|
||||||
|
resource_type: "pii_identity_key",
|
||||||
|
resource_id: key.key_hash,
|
||||||
|
risk_level: "L3",
|
||||||
|
payload: { source_id: input.source.source_id, field: input.field, expires_at: key.expires_at },
|
||||||
|
});
|
||||||
|
return key;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Right of access — everything currently held on a subject across sources. */
|
||||||
|
async accessStatement(subjectHash: string): Promise<PiiAccessStatement> {
|
||||||
|
const rows = this.registry.filter((r) => r.subject_hash === subjectHash && r.status === "active");
|
||||||
|
return { subject_hash: subjectHash, rows, generated_at: new Date().toISOString() };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Right to erasure — tombstone every held row for the subject. */
|
||||||
|
async erase(request: PiiErasureRequest): Promise<number> {
|
||||||
|
let count = 0;
|
||||||
|
for (const row of this.registry) {
|
||||||
|
if (row.subject_hash === request.subject_hash && row.status === "active") {
|
||||||
|
row.status = "erased";
|
||||||
|
count += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
await this.safeEmit({
|
||||||
|
event_id: `evt_0pt_pii_erased_${request.subject_hash.slice(0, 12)}_${Date.now()}`,
|
||||||
|
event_type: "0pt.pii_erased",
|
||||||
|
capability_id: "0pt.pii_erase",
|
||||||
|
tenant_id: "",
|
||||||
|
project_id: "",
|
||||||
|
actor_id: request.actor_id ?? "system",
|
||||||
|
resource_type: "pii_subject",
|
||||||
|
resource_id: request.subject_hash,
|
||||||
|
risk_level: "L3",
|
||||||
|
payload: { reason: request.reason, row_count: count },
|
||||||
|
});
|
||||||
|
return count;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Storage limitation — purge rows past their retention deadline. */
|
||||||
|
async enforceRetention(now: Date = new Date()): Promise<{ purged: number }> {
|
||||||
|
let purged = 0;
|
||||||
|
const keep: PiiIdentityKey[] = [];
|
||||||
|
for (const row of this.registry) {
|
||||||
|
if (row.expires_at <= now.toISOString() && row.status === "active") {
|
||||||
|
row.status = "purged";
|
||||||
|
purged += 1;
|
||||||
|
} else {
|
||||||
|
keep.push(row);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (purged > 0) {
|
||||||
|
await this.safeEmit({
|
||||||
|
event_id: `evt_0pt_pii_retention_${now.getTime()}`,
|
||||||
|
event_type: "0pt.pii_retention_purged",
|
||||||
|
capability_id: "0pt.pii_enforce_retention",
|
||||||
|
tenant_id: "",
|
||||||
|
project_id: "",
|
||||||
|
actor_id: "system",
|
||||||
|
resource_type: "pii_identity_key",
|
||||||
|
resource_id: now.toISOString(),
|
||||||
|
risk_level: "L3",
|
||||||
|
payload: { row_count: purged },
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return { purged };
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ *
|
||||||
|
* Log hygiene — no personuppgifter in logs/metrics/traces
|
||||||
|
* ------------------------------------------------------------------ */
|
||||||
|
|
||||||
|
/** Scrub personnummer, e-mail and Swedish phone numbers from a log line. */
|
||||||
|
redact(line: string): string {
|
||||||
|
return line
|
||||||
|
.replace(/\b\d{8}[-–]\d{4}\b/g, "[PERSONNUMMER]")
|
||||||
|
.replace(/\b\d{6}[-–]\d{4}\b/g, "[PERSONNUMMER]")
|
||||||
|
.replace(/\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b/g, "[EMAIL]")
|
||||||
|
.replace(/(?:\+46|0046|0)\s?[1-9][\d\s.-]{7,13}/g, "[TELEFON]");
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ */
|
||||||
|
|
||||||
|
private async safeEmit(event: Omit<Aud0Event, "app_id" | "created_at">): Promise<void> {
|
||||||
|
const full: Aud0Event = {
|
||||||
|
...event,
|
||||||
|
app_id: "0pt",
|
||||||
|
created_at: new Date().toISOString(),
|
||||||
|
};
|
||||||
|
try {
|
||||||
|
await this.aud0.ingestEvent(full);
|
||||||
|
} catch {
|
||||||
|
// Audit is important but must never take down a compliance decision or
|
||||||
|
// a retention pass. The caller's state change has already happened.
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function sourceIdSalt(sourceId: string): string {
|
||||||
|
return `0pt-pii-v1:${sourceId}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** sha256 hex digest — never the plaintext. */
|
||||||
|
export function hash256(value: string): string {
|
||||||
|
return createHash("sha256").update(value, "utf8").digest("hex");
|
||||||
|
}
|
||||||
@@ -20,6 +20,7 @@
|
|||||||
|
|
||||||
import type { Source } from "../types/source";
|
import type { Source } from "../types/source";
|
||||||
import { SourceService } from "./source-service";
|
import { SourceService } from "./source-service";
|
||||||
|
import { PiiGateService } from "./pii-gate-service";
|
||||||
import { createAud0Client, type Aud0Client } from "../clients/aud0-client";
|
import { createAud0Client, type Aud0Client } from "../clients/aud0-client";
|
||||||
import { FabricOneLakeAdapter, type FabricOneLakeConnectionConfig, type AdapterQueryResult } from "../adapters/fabric-onelake-adapter";
|
import { FabricOneLakeAdapter, type FabricOneLakeConnectionConfig, type AdapterQueryResult } from "../adapters/fabric-onelake-adapter";
|
||||||
|
|
||||||
@@ -76,16 +77,27 @@ function clamp(value: number, min: number, max: number): number {
|
|||||||
export class SourceQueryService {
|
export class SourceQueryService {
|
||||||
private sourceService: SourceService;
|
private sourceService: SourceService;
|
||||||
private aud0: Aud0Client;
|
private aud0: Aud0Client;
|
||||||
|
/** GDPR guardrail — on by default, can be shared via opts so callers use one registry. */
|
||||||
|
private piiGate: PiiGateService;
|
||||||
|
|
||||||
constructor(opts?: { sourceService?: SourceService; aud0BaseUrl?: string }) {
|
constructor(opts?: { sourceService?: SourceService; aud0BaseUrl?: string; piiGate?: PiiGateService }) {
|
||||||
// Accepts an existing SourceService so callers (e.g. the API route) share
|
// Accepts an existing SourceService so callers (e.g. the API route) share
|
||||||
// one in-memory registry instead of each constructing an empty one.
|
// one in-memory registry instead of each constructing an empty one.
|
||||||
this.sourceService = opts?.sourceService ?? new SourceService();
|
this.sourceService = opts?.sourceService ?? new SourceService();
|
||||||
this.aud0 = createAud0Client(opts?.aud0BaseUrl);
|
this.aud0 = createAud0Client(opts?.aud0BaseUrl);
|
||||||
|
this.piiGate = opts?.piiGate ?? new PiiGateService({ aud0BaseUrl: opts?.aud0BaseUrl });
|
||||||
}
|
}
|
||||||
|
|
||||||
async querySource(input: QuerySourceInput): Promise<QuerySourceResult> {
|
async querySource(input: QuerySourceInput): Promise<QuerySourceResult> {
|
||||||
const startedAt = Date.now();
|
const startedAt = Date.now();
|
||||||
|
|
||||||
|
// GDPR fail-closed: personal-data sources (self-flagged or known directory
|
||||||
|
// domains such as ratsit.se) may not be processed without a PiiPolicy.
|
||||||
|
// Runs BEFORE the capability/type gate so the lawful-basis error is the
|
||||||
|
// one the caller sees for personal data.
|
||||||
|
const preflight = await this.sourceService.getSource(input.source_id);
|
||||||
|
await this.piiGate.assertCanStore(preflight, [], input.tenant_id);
|
||||||
|
|
||||||
const source = await this.getQueryableSource(input.source_id);
|
const source = await this.getQueryableSource(input.source_id);
|
||||||
|
|
||||||
const maxRows = clamp(input.max_rows ?? SOURCE_QUERY_DEFAULT_MAX_ROWS, 1, SOURCE_QUERY_HARD_MAX_ROWS);
|
const maxRows = clamp(input.max_rows ?? SOURCE_QUERY_DEFAULT_MAX_ROWS, 1, SOURCE_QUERY_HARD_MAX_ROWS);
|
||||||
|
|||||||
@@ -0,0 +1,102 @@
|
|||||||
|
/**
|
||||||
|
* src/types/pii.ts
|
||||||
|
* =================
|
||||||
|
* 0pt.pii_policy.schema.v1 — SIAX Cloud GDPR guardrail for sources that
|
||||||
|
* CAN touch personuppgifter (personal data). Purpose: make processing of
|
||||||
|
* personal data "tekniskt OK" per GDPR (art. 5, 6, 25, 32, 17, 15):
|
||||||
|
* recorded legal basis, storage limitation, data minimisation,
|
||||||
|
* pseudonymisation, and working access/erasure surfaces.
|
||||||
|
*
|
||||||
|
* The guardrail only ever OPENS when a lawful basis is on record; nothing in
|
||||||
|
* 0pt today crawls personal data, so this ships as a fail-closed boundary:
|
||||||
|
* no policy → no storage (and, for directory domains like ratsit.se, no
|
||||||
|
* query at all).
|
||||||
|
*/
|
||||||
|
|
||||||
|
/** GDPR art. 6(1) legal bases 0pt accepts for personal-data sources. */
|
||||||
|
export type LegalBasis = "legitimate_interest" | "consent" | "contract" | "legal_obligation";
|
||||||
|
|
||||||
|
/** Compliance posture for one source that touches personal data. */
|
||||||
|
export interface PiiPolicy {
|
||||||
|
source_id: string;
|
||||||
|
tenant_id: string;
|
||||||
|
project_id: string;
|
||||||
|
touches_personal_data: true;
|
||||||
|
legal_basis: LegalBasis;
|
||||||
|
/**
|
||||||
|
* Pointer to the evidence: a recorded Legitimate Interest Assessment
|
||||||
|
* (LIA) document id, the consent record id, the contract clause, or the
|
||||||
|
* legal provision. Must exist for audit/DPIA purposes — a bare claim is
|
||||||
|
* not enough.
|
||||||
|
*/
|
||||||
|
basis_ref: string;
|
||||||
|
/**
|
||||||
|
* Max storage in days. 0 = fetch-and-drop only (never retained).
|
||||||
|
* The retention worker deletes rows where stored_at + retention_days
|
||||||
|
* < now (art. 5(1)(e)).
|
||||||
|
*/
|
||||||
|
retention_days: number;
|
||||||
|
/**
|
||||||
|
* The ONLY fields that may be stored for this source (art. 5(1)(c)
|
||||||
|
* minimisation). Empty list = nothing may be stored.
|
||||||
|
*/
|
||||||
|
field_allowlist: string[];
|
||||||
|
/**
|
||||||
|
* Pseudonymise identity keys (personnummer, email, reg-number) before
|
||||||
|
* storage; only sha256 digests are ever written (art. 25).
|
||||||
|
*/
|
||||||
|
pseudonymize_keys: boolean;
|
||||||
|
/**
|
||||||
|
* Must be true for web/api sources: robots.txt checked, rate limit in
|
||||||
|
* force (SourceRatePolicy), no auth-wall / bot-bypass technique in use.
|
||||||
|
*/
|
||||||
|
robots_and_rate_confirmed: boolean;
|
||||||
|
created_at: string;
|
||||||
|
updated_at: string | null;
|
||||||
|
created_by: string | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface CreatePiiPolicyInput {
|
||||||
|
source_id: string;
|
||||||
|
tenant_id: string;
|
||||||
|
project_id: string;
|
||||||
|
legal_basis: LegalBasis;
|
||||||
|
basis_ref: string;
|
||||||
|
retention_days: number;
|
||||||
|
field_allowlist: string[];
|
||||||
|
pseudonymize_keys?: boolean;
|
||||||
|
robots_and_rate_confirmed?: boolean;
|
||||||
|
created_by?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Registry row for one stored personal-data value. Only hashes are held —
|
||||||
|
* the key_hash lets 0pt locate the subject (right of access, art. 15) and
|
||||||
|
* the subject_hash lets it erase every artifact for that subject across
|
||||||
|
* sources (right to erasure, art. 17) without ever storing the plaintext
|
||||||
|
* identity.
|
||||||
|
*/
|
||||||
|
export interface PiiIdentityKey {
|
||||||
|
key_hash: string;
|
||||||
|
subject_hash: string;
|
||||||
|
source_id: string;
|
||||||
|
tenant_id: string;
|
||||||
|
field: string;
|
||||||
|
artifact_ref: string | null;
|
||||||
|
stored_at: string;
|
||||||
|
expires_at: string;
|
||||||
|
status: "active" | "erased" | "purged";
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface PiiErasureRequest {
|
||||||
|
subject_hash: string;
|
||||||
|
reason: "deleted_by_subject" | "retention_expired" | "consent_withdrawn";
|
||||||
|
actor_id?: string;
|
||||||
|
correlation_id?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface PiiAccessStatement {
|
||||||
|
subject_hash: string;
|
||||||
|
rows: PiiIdentityKey[];
|
||||||
|
generated_at: string;
|
||||||
|
}
|
||||||
@@ -63,6 +63,13 @@ export interface Source {
|
|||||||
adapter_id?: string;
|
adapter_id?: string;
|
||||||
/** Mirrors dim_source.access (docs/03-data-model.md). */
|
/** Mirrors dim_source.access (docs/03-data-model.md). */
|
||||||
access?: SourceAccess;
|
access?: SourceAccess;
|
||||||
|
/**
|
||||||
|
* Explicit self-flag that this source CAN expose personuppgifter. When
|
||||||
|
* true (or when the source is a known directory domain like ratsit.se),
|
||||||
|
* the PiiGateService requires a PiiPolicy with a recorded lawful basis
|
||||||
|
* before any storage/query (fail-closed, GDPR art. 6).
|
||||||
|
*/
|
||||||
|
touches_personal_data?: boolean;
|
||||||
description?: string;
|
description?: string;
|
||||||
status: SourceStatus;
|
status: SourceStatus;
|
||||||
correlation_id?: string;
|
correlation_id?: string;
|
||||||
@@ -86,6 +93,7 @@ export interface CreateSourceInput {
|
|||||||
name: string;
|
name: string;
|
||||||
adapter_id?: string;
|
adapter_id?: string;
|
||||||
access?: SourceAccess;
|
access?: SourceAccess;
|
||||||
|
touches_personal_data?: boolean;
|
||||||
description?: string;
|
description?: string;
|
||||||
status?: SourceStatus;
|
status?: SourceStatus;
|
||||||
correlation_id?: string;
|
correlation_id?: string;
|
||||||
|
|||||||
@@ -0,0 +1,173 @@
|
|||||||
|
import { describe, it, expect, vi } from "vitest";
|
||||||
|
import { PiiGateService } from "@/src/services/pii-gate-service";
|
||||||
|
import { SourceService } from "@/src/services/source-service";
|
||||||
|
import { SourceQueryService } from "@/src/services/source-query-service";
|
||||||
|
import type { FabricOneLakeConnectionConfig } from "@/src/adapters/fabric-onelake-adapter";
|
||||||
|
|
||||||
|
/** Minimal config — never reached (the pii gate fires first). */
|
||||||
|
const EMPTY_CFG = {} as FabricOneLakeConnectionConfig;
|
||||||
|
|
||||||
|
describe("PiiGateService — GDPR guardrail (fail-closed)", () => {
|
||||||
|
const ratsit = {
|
||||||
|
source_id: "src_ratsit_1",
|
||||||
|
tenant_id: "t1",
|
||||||
|
project_id: "p1",
|
||||||
|
source_type: "web" as const,
|
||||||
|
name: "ratsit.se",
|
||||||
|
status: "active" as const,
|
||||||
|
created_at: new Date().toISOString(),
|
||||||
|
};
|
||||||
|
const company = {
|
||||||
|
source_id: "src_company_1",
|
||||||
|
tenant_id: "t1",
|
||||||
|
project_id: "p1",
|
||||||
|
source_type: "api" as const,
|
||||||
|
name: "allabolag.se",
|
||||||
|
status: "active" as const,
|
||||||
|
created_at: new Date().toISOString(),
|
||||||
|
};
|
||||||
|
|
||||||
|
it("classifies known directory domains as personal-data sources", () => {
|
||||||
|
const gate = new PiiGateService();
|
||||||
|
expect(gate.classifySourcePersonal(ratsit)).toBe(true);
|
||||||
|
expect(gate.classifySourcePersonal({ ...ratsit, name: "www.ratsit.se" })).toBe(true);
|
||||||
|
expect(gate.classifySourcePersonal(company)).toBe(false);
|
||||||
|
expect(gate.classifySourcePersonal({ ...company, touches_personal_data: true })).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("rejects processing a personal-data source with no recorded lawful basis", async () => {
|
||||||
|
const gate = new PiiGateService();
|
||||||
|
await expect(gate.assertCanStore(ratsit, [], "t1")).rejects.toThrowError(/no PiiPolicy/);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("rejects half-legal policies (missing basis_ref, zero retention, empty allowlist)", async () => {
|
||||||
|
const gate = new PiiGateService();
|
||||||
|
await expect(
|
||||||
|
gate.createPiiPolicy({ source_id: "s1", tenant_id: "t1", project_id: "p1", legal_basis: "legitimate_interest", basis_ref: " ", retention_days: 90, field_allowlist: ["address"] }),
|
||||||
|
).rejects.toThrow(/basis_ref is required/);
|
||||||
|
await expect(
|
||||||
|
gate.createPiiPolicy({ source_id: "s1", tenant_id: "t1", project_id: "p1", legal_basis: "consent", basis_ref: "consent-record-1", retention_days: -1, field_allowlist: ["address"] }),
|
||||||
|
).rejects.toThrow(/retention_days/);
|
||||||
|
await expect(
|
||||||
|
gate.createPiiPolicy({ source_id: "s1", tenant_id: "t1", project_id: "p1", legal_basis: "contract", basis_ref: "clause-9", retention_days: 90, field_allowlist: [] }),
|
||||||
|
).rejects.toThrow(/data minimisation/);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("blocks out-of-allowlist fields (minimisation) and unconfirmed web sources", async () => {
|
||||||
|
const gate = new PiiGateService();
|
||||||
|
await gate.createPiiPolicy({
|
||||||
|
source_id: ratsit.source_id, tenant_id: "t1", project_id: "p1",
|
||||||
|
legal_basis: "consent", basis_ref: "consent-record-42",
|
||||||
|
retention_days: 90, field_allowlist: ["name", "address"],
|
||||||
|
});
|
||||||
|
await expect(gate.assertCanStore(ratsit, ["name", "phone"], "t1")).rejects.toThrowError(/outside the allowlist/);
|
||||||
|
await expect(gate.assertCanStore(ratsit, ["name"], "t1")).rejects.toThrowError(/robots.txt \/ rate-limit not confirmed/);
|
||||||
|
|
||||||
|
const ratsitConfirmed = { ...ratsit, source_id: "src_ratsit_confirmed" };
|
||||||
|
await gate.createPiiPolicy({
|
||||||
|
source_id: ratsitConfirmed.source_id, tenant_id: "t1", project_id: "p1",
|
||||||
|
legal_basis: "consent", basis_ref: "consent-record-42",
|
||||||
|
retention_days: 90, field_allowlist: ["name", "address"], robots_and_rate_confirmed: true,
|
||||||
|
});
|
||||||
|
await expect(gate.assertCanStore(ratsitConfirmed, ["name"], "t1")).resolves.toBeUndefined();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("forbids any storage when retention_days = 0", async () => {
|
||||||
|
const gate = new PiiGateService();
|
||||||
|
await gate.createPiiPolicy({
|
||||||
|
source_id: "src_drop_1", tenant_id: "t1", project_id: "p1",
|
||||||
|
legal_basis: "legitimate_interest", basis_ref: "lia-2026-001",
|
||||||
|
retention_days: 0, field_allowlist: ["name"], robots_and_rate_confirmed: true,
|
||||||
|
});
|
||||||
|
await expect(
|
||||||
|
gate.assertCanStore({ ...company, source_id: "src_drop_1", name: "ratsit.se" }, ["name"], "t1"),
|
||||||
|
).rejects.toThrowError(/forbids retention/);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("stores ruled bases and enables access/erasure without plaintext identity", async () => {
|
||||||
|
const gate = new PiiGateService();
|
||||||
|
await gate.createPiiPolicy({
|
||||||
|
source_id: ratsit.source_id, tenant_id: "t1", project_id: "p1",
|
||||||
|
legal_basis: "legal_obligation", basis_ref: "prop-§5",
|
||||||
|
retention_days: 30, field_allowlist: ["street_address"], robots_and_rate_confirmed: true,
|
||||||
|
});
|
||||||
|
const key = await gate.recordStored({
|
||||||
|
source: ratsit, tenantId: "t1",
|
||||||
|
identityKey: "20250101-1234", subjectId: "subj-77", field: "street_address", artifactRef: "st0re/dataset-a",
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(key.key_hash).toMatch(/^[0-9a-f]{64}$/);
|
||||||
|
expect(key.key_hash).not.toContain("20250101");
|
||||||
|
|
||||||
|
const statement = await gate.accessStatement(key.subject_hash);
|
||||||
|
expect(statement.rows).toHaveLength(1);
|
||||||
|
expect(statement.rows[0].field).toBe("street_address");
|
||||||
|
|
||||||
|
const erased = await gate.erase({ subject_hash: key.subject_hash, reason: "deleted_by_subject" });
|
||||||
|
expect(erased).toBe(1);
|
||||||
|
expect((await gate.accessStatement(key.subject_hash)).rows).toHaveLength(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("retention worker purges rows past their deadline", async () => {
|
||||||
|
const gate = new PiiGateService();
|
||||||
|
vi.useFakeTimers();
|
||||||
|
vi.setSystemTime(new Date("2026-08-01T00:00:00Z"));
|
||||||
|
await gate.createPiiPolicy({
|
||||||
|
source_id: ratsit.source_id, tenant_id: "t1", project_id: "p1",
|
||||||
|
legal_basis: "legitimate_interest", basis_ref: "lia-2026-002",
|
||||||
|
retention_days: 30, field_allowlist: ["street_address"], robots_and_rate_confirmed: true,
|
||||||
|
});
|
||||||
|
await gate.recordStored({ source: ratsit, tenantId: "t1", identityKey: "k-1", subjectId: "s-1", field: "street_address" });
|
||||||
|
|
||||||
|
vi.setSystemTime(new Date("2026-09-15T00:00:00Z"));
|
||||||
|
const { purged } = await gate.enforceRetention(new Date("2026-09-15T00:00:00Z"));
|
||||||
|
vi.useRealTimers();
|
||||||
|
expect(purged).toBe(1);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("redacts personnummer, e-mail and phone from log lines", () => {
|
||||||
|
const gate = new PiiGateService();
|
||||||
|
const out = gate.redact(
|
||||||
|
"subject=202501011234 res=" +
|
||||||
|
'evt=19720101-1234 mail=simon@siax.io tel=070-123 45 67 sök=ok',
|
||||||
|
);
|
||||||
|
expect(out).not.toMatch(/\d{8}-\d{4}/);
|
||||||
|
expect(out).toContain("[PERSONNUMMER]");
|
||||||
|
expect(out).toContain("[EMAIL]");
|
||||||
|
expect(out).toContain("[TELEFON]");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("SourceQueryService blocks a personal-data source by default (fail-closed)", async () => {
|
||||||
|
const sourceService = new SourceService();
|
||||||
|
await sourceService.createSource({ ...ratsit });
|
||||||
|
|
||||||
|
const service = new SourceQueryService({ sourceService });
|
||||||
|
await expect(
|
||||||
|
service.querySource({
|
||||||
|
source_id: ratsit.source_id,
|
||||||
|
query: "SELECT 1",
|
||||||
|
tenant_id: "t1", project_id: "p1",
|
||||||
|
connection_config: EMPTY_CFG,
|
||||||
|
}),
|
||||||
|
).rejects.toThrowError(/no PiiPolicy/);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("SourceQueryService passes non-personal sources straight through to capability checks", async () => {
|
||||||
|
const sourceService = new SourceService();
|
||||||
|
await sourceService.createSource({
|
||||||
|
source_id: "src_safe_1", tenant_id: "t1", project_id: "p1",
|
||||||
|
source_type: "web", name: "example.org", status: "active",
|
||||||
|
});
|
||||||
|
|
||||||
|
const service = new SourceQueryService({ sourceService });
|
||||||
|
// Gate passes; the sync path then correctly rejects non-database sources.
|
||||||
|
await expect(
|
||||||
|
service.querySource({
|
||||||
|
source_id: "src_safe_1",
|
||||||
|
query: "SELECT 1",
|
||||||
|
tenant_id: "t1", project_id: "p1",
|
||||||
|
connection_config: EMPTY_CFG,
|
||||||
|
}),
|
||||||
|
).rejects.toThrowError(/source_type "web"/);
|
||||||
|
});
|
||||||
|
});
|
||||||
Reference in New Issue
Block a user