Merge PR #56 (prod-merge-pass): PII-Gate GDPR-guardrail + realinfra-migration-test + security headers & repo-honesty doctor (omsluter PR #44)
Deploy to Coolify / deploy (push) Successful in 1s
masterplan-lock / check (push) Successful in 3s
CI (SIAX Cloud) / security (push) Successful in 13s
Security Scan (SIAX Cloud) / security (push) Successful in 12s
CI (SIAX Cloud) / quality (push) Successful in 38s
CI (SIAX Cloud) / integration (push) Successful in 2m19s
Deploy to Coolify / deploy (push) Successful in 1s
masterplan-lock / check (push) Successful in 3s
CI (SIAX Cloud) / security (push) Successful in 13s
Security Scan (SIAX Cloud) / security (push) Successful in 12s
CI (SIAX Cloud) / quality (push) Successful in 38s
CI (SIAX Cloud) / integration (push) Successful in 2m19s
This commit is contained in:
+45
-5
@@ -25,12 +25,37 @@
|
||||
levereras därför i ett eget npm-steg i Dockerfile, pinnat till versionen i
|
||||
`pnpm-lock.yaml`. Verifierat i containern: `/_next/image` svarar 200 med
|
||||
`image/avif` (1,46 MB PNG -> 6,7 kB).
|
||||
- `chore(deploy)`: `poweredByHeader: false` (ingen `X-Powered-By: Next.js`).
|
||||
- fix(security): response security headers are set by the app itself, in
|
||||
`next.config.mjs` `headers()` — CSP (incl. `frame-ancestors 'none'`,
|
||||
`base-uri 'self'`, `object-src 'none'`, `form-action 'self'`), HSTS,
|
||||
X-Frame-Options `DENY`, X-Content-Type-Options, Referrer-Policy,
|
||||
Permissions-Policy — plus `poweredByHeader: false`. Regression-gated by
|
||||
`tests/security/security-headers.test.ts` (blocking, runs in
|
||||
`pnpm run test`) and re-checkable against a live origin with
|
||||
`node scripts/siax-doctor.mjs --live`. (BACKLOG P0-13, P1-15)
|
||||
|
||||
- fix(security): security headers (CSP/HSTS/X-Frame-Options/etc.) + CI
|
||||
security gates actually enforce (Trivy `--exit-code 1`, gitleaks) instead
|
||||
of advisory-only. Actions pinned to commit SHA. SBOM + license inventory
|
||||
as CI artifacts. (BACKLOG P0-13, P0-16, P0-19, P1-14, P1-15)
|
||||
Correcting this file: from 2026-08 until 2026-09-07 this bullet also
|
||||
claimed the headers themselves, while `main` contained none — a repo-wide
|
||||
grep for `content-security-policy|strict-transport-security|
|
||||
x-frame-options` matched only CHANGELOG.md. The only thing setting HSTS on
|
||||
0pt.app was Vercel's platform default (still observable on the 402
|
||||
responses recorded in `evidence/operational-state.json`), which is not this
|
||||
repo's and does not survive the move to Coolify/Traefik. The claim is true
|
||||
as of this entry; the four claims it was bundled with are not, and are
|
||||
tracked in the section below.
|
||||
|
||||
- fix(repo-honesty): `scripts/siax-doctor.mjs` validated that
|
||||
`siax.repo.json` fields were PRESENT, never what they said — measured:
|
||||
`operationalState: "running"` (false at the time; every declared host
|
||||
answered 402/404) and `"down"` (true) both produced 40/40 and
|
||||
byte-identical output. It now validates values (enums, hostname form),
|
||||
cross-file consistency (declared prod hostnames must appear in this repo's
|
||||
own deploy/contract config, and vice versa), and requires
|
||||
`operationalState` to be backed by a dated probe record in
|
||||
`evidence/operational-state.json` that agrees with it. Offline and
|
||||
deterministic by default; `--live` (opt-in) probes the real hosts.
|
||||
Gated by `tests/repo/siax-doctor.test.ts`, which also asserts the negative
|
||||
cases against fixture trees. (BACKLOG P1-19, control AGENT-04, GOV-06)
|
||||
- fix(health): `/api/health` reports real per-dependency status instead of a
|
||||
hardcoded `ok: true`; new `/api/ready` readiness endpoint. (BACKLOG P0-3)
|
||||
- docs/chore: repo-hygiene pass — schema-valid `siax.repo.json`/`siax.config.json`
|
||||
@@ -38,6 +63,21 @@
|
||||
`docs/architecture/ARCHITECTURE.md` §5 "Ytor och paritet", CODEOWNERS,
|
||||
stale-link cleanup. (BACKLOG P1-1, P1-2, P1-3, P1-19, P1-20, P1-21, P2-5, P3-5)
|
||||
|
||||
### Claimed here before 2026-09-07 but NOT implemented — status 2026-09-15
|
||||
|
||||
These were asserted by the earlier `fix(security)` bullet and were not true on
|
||||
`main` at that time. Current state (measured against `main`, 2026-09-15):
|
||||
|
||||
- CI security gates now ENFORCE: `security.yml` and `nightly.yml` run Trivy
|
||||
with `--exit-code 1` (landed via `ci/make-required-security-gate-real`,
|
||||
2026-09-08). (BACKLOG P0-16 — CLOSED)
|
||||
- gitleaks is still not run anywhere in this repo. (BACKLOG P0-19 — OPEN)
|
||||
- Actions are still NOT pinned to commit SHAs (`actions/checkout@v4` etc.).
|
||||
(BACKLOG P1-14 — OPEN)
|
||||
- No SBOM and no license inventory are produced or uploaded as CI artifacts.
|
||||
(OPEN; the `fix/wire-signed-digest-to-deploy` branch carries a prepared
|
||||
release chain that needs its cosign/registry secrets before it can land.)
|
||||
|
||||
## 2026-08-11 — Evasion-stack decision provenance
|
||||
|
||||
- Attributed and dated the account-owner's keep-decision on `services/scrapers/*`
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
# BACKLOG P1-15 (CI-14): required reviewers for Gitea's branch-protection
|
||||
# "required approvals" setting to have anyone to assign. This file alone
|
||||
# does not turn on required-approvals enforcement -- that is a Gitea
|
||||
# repo-settings change (branch protection), tracked separately; see
|
||||
# docs/compliance/BACKLOG.md P1-15 ("required_approvals: 0").
|
||||
#
|
||||
# Gitea CODEOWNERS syntax matches GitHub's: path pattern, then owner(s).
|
||||
* @sax3l
|
||||
@@ -0,0 +1,77 @@
|
||||
# PII Guardrail — GDPR requirement for personal-data sources (0pt)
|
||||
|
||||
Status: **enforced, fail-closed.** No 0pt pipeline is crawling personuppgifter
|
||||
today; the `PiiGateService` ships so that any future source that touches
|
||||
personal data (Ratsit, Merinfo, Hitta, …) is **technically GDPR-OK before the
|
||||
first byte is stored**.
|
||||
|
||||
## The rule
|
||||
|
||||
1. **Classify.** A source is personal-data if it self-flags
|
||||
(`touches_personal_data`) or is a known directory domain (ratsit.se,
|
||||
merinfo.se, hitta.se, mrkoll.se, eniro.se, birthday.se, upplysning.se).
|
||||
2. **No policy → no processing.** `assertCanStore()` throws unless a
|
||||
`PiiPolicy` with a valid `legal_basis` + `basis_ref` is on record.
|
||||
3. **Everything is evidenced.** Each decision is emitted to AUD0 at risk L3
|
||||
(`0pt.pii_policy_created | pii_data_retained | pii_erased |
|
||||
pii_retention_purged`).
|
||||
|
||||
## Recorded legitimate-interest stance (Simon, 2026-08-31)
|
||||
|
||||
> "Vi får lagra dem om dem är kopplade till företagen i affärssyfte."
|
||||
|
||||
Personal data may be stored under **legitimate interest (art. 6(1)(f))** ONLY
|
||||
as **business contacts bound to a company/org entity** (`entity_linked_only`).
|
||||
The gate enforces this mechanically:
|
||||
|
||||
- `legal_basis: legitimate_interest` ⇒ `purpose` MUST be `business_contact` and
|
||||
`entity_linked_only` MUST be `true`. A lazily-written policy is rejected.
|
||||
- `recordStored()` refuses person-only rows for such policies: an `entity_id`
|
||||
(orgnr/rgs-nummer/LEI) is REQUIRED; only its hash is ever stored.
|
||||
- Stand-alone personal records (no backing company) are not permitted under LI.
|
||||
|
||||
This is the recorded business-policy decision — a written LIA should be kept
|
||||
and its id referenced in `basis_ref` (`lia-YYYY-MM-b2b-NN` naming is fine).
|
||||
|
||||
## GDPR article mapping
|
||||
|
||||
| Requirement | Implementation |
|
||||
|---|---|
|
||||
| Legal basis — art. 6 | `legal_basis` ∈ legitimate_interest/consent/contract/legal_obligation + `basis_ref` (LIA doc id, consent record id, contract clause, provision). Cannot be empty. |
|
||||
| Data minimisation — art. 5(1)(c) | `field_allowlist`; storing any other field throws `PiiMinimisationError`. Empty allowlist = nothing stored. |
|
||||
| Storage limitation — art. 5(1)(e) | `retention_days`; `enforceRetention()` purges expired rows. `0` = fetch-and-drop, nothing persisted. |
|
||||
| Integrity/confidentiality — art. 32 | Only sha256 hashes of identity keys stored (`recordStored`); plaintext identity never written; log/trace redaction (`redact()`) for personnummer/e-mail/phone. |
|
||||
| Right of access — art. 15 | `accessStatement(subjectHash)` — what is held on a subject, across sources. |
|
||||
| Right to erasure — art. 17 | `erase({ subject_hash, reason })` tombstones every stored value for the subject. |
|
||||
| Data protection by design — art. 25 | Gate defaults ON in `SourceQueryService`; no bypass knobs. |
|
||||
|
||||
## When someone requests a personal-data source (e.g. Ratsit)
|
||||
|
||||
- [x] Confirm the source is actually needed and no anonymised/official
|
||||
alternative exists (official register > directory listing).
|
||||
- [x] Write the **Legitimate Interest Assessment** (or record consent /
|
||||
contract / legal obligation) and store its id in `basis_ref`.
|
||||
LIA minimum: purpose, necessity, individuals' rights impact, mitigations.
|
||||
For LI: purpose `business_contact` + `entity_linked_only` (see recorded
|
||||
stance above — an LIA referencing `lia-YYYY-MM-b2b-NN` is good practice).
|
||||
- [x] **VERIFY current terms + robots.txt + availability** on the live site.
|
||||
Do not hardcode "catsit.com allows" — re-check each enable.
|
||||
- [x] Set `robots_and_rate_confirmed: true` + a `SourceRatePolicy` matching
|
||||
the source's own limits (legacy N0D default for `ratsit.se` was `1/min` —
|
||||
treat that as a ceiling, not a target).
|
||||
- [x] Define the minimum `field_allowlist` (e.g. only `street_address`, not
|
||||
`phone`).
|
||||
- [x] Invoke `recordStored` for every stored value and run `enforceRetention`
|
||||
on a schedule (e.g. daily cron).
|
||||
- [x] Record a DPIA note if processing is "large scale" (art. 35) — bulk
|
||||
directory scraping of living persons is large-scale by default.
|
||||
|
||||
## Never
|
||||
|
||||
- Pretend a source has a legal basis it does not have; `basis_ref` must point
|
||||
to a real artifact.
|
||||
- Store identity keys in plaintext (`pseudonymize_keys` defaults to true).
|
||||
- Use stealth/bot-bypass/rate-evasion — excluded by CLAUDE.md §1 regardless of
|
||||
GDPR.
|
||||
- Claim compliance on a source we have not re-verified this season (`VERIFY`
|
||||
markers are intentional).
|
||||
@@ -0,0 +1,49 @@
|
||||
{
|
||||
"$comment": [
|
||||
"GOV-06 substantiation record for siax.repo.json's `operationalState`.",
|
||||
"",
|
||||
"WHY THIS FILE EXISTS: `operationalState` is a claim about the world, and",
|
||||
"scripts/siax-doctor.mjs cannot check the world from CI (no network, and a",
|
||||
"check that needs the internet is a check that goes red for the wrong",
|
||||
"reason). What the doctor CAN do offline is refuse to accept the claim",
|
||||
"unless it is backed by a dated, machine-readable probe result that agrees",
|
||||
"with it. That is this file. Setting operationalState to \"running\" while",
|
||||
"the probes below say 402/404 now fails `pnpm siax:doctor`.",
|
||||
"",
|
||||
"HOW TO REFRESH: `node scripts/siax-doctor.mjs --live --write-evidence`",
|
||||
"(opt-in, needs network). Or edit by hand from a real probe -- but the",
|
||||
"doctor cross-checks `ok` against `httpStatus`, so a hand-written",
|
||||
"`ok: true` next to a 402 is itself a failure."
|
||||
],
|
||||
"control": "GOV-06",
|
||||
"operationalState": "down",
|
||||
"checkedAt": "2026-09-07T18:24:24Z",
|
||||
"method": "curl -sS -o /dev/null -D - --max-time 20 <url>, from the public internet (not the tailnet)",
|
||||
"probes": [
|
||||
{
|
||||
"fqdn": "0pt.app",
|
||||
"env": "prod",
|
||||
"url": "https://0pt.app/api/health",
|
||||
"httpStatus": 402,
|
||||
"ok": false,
|
||||
"note": "server: Vercel; x-vercel-error: DEPLOYMENT_DISABLED. The Vercel deployment is disabled and no Coolify app is serving this hostname yet."
|
||||
},
|
||||
{
|
||||
"fqdn": "api.0pt.app",
|
||||
"env": "prod",
|
||||
"url": "https://api.0pt.app/api/health",
|
||||
"httpStatus": 404,
|
||||
"ok": false,
|
||||
"note": "server: Vercel; x-vercel-error: DEPLOYMENT_NOT_FOUND. Contract-first target host (contracts/openapi.yaml servers), never deployed. Both / and /api/health return the same 404."
|
||||
}
|
||||
],
|
||||
"securityHeadersObserved": {
|
||||
"$comment": "Observed on the 402/404 responses above, i.e. served by Vercel's edge, NOT by this app. This is the evidence for BACKLOG P0-13: everything except Vercel's own HSTS default is absent, and that default does not survive a move to Coolify/Traefik.",
|
||||
"strict-transport-security": "max-age=63072000",
|
||||
"content-security-policy": null,
|
||||
"x-frame-options": null,
|
||||
"x-content-type-options": null,
|
||||
"referrer-policy": null,
|
||||
"permissions-policy": null
|
||||
}
|
||||
}
|
||||
+118
@@ -1,3 +1,113 @@
|
||||
// =============================================================================
|
||||
// 0pt -- Next.js config
|
||||
// =============================================================================
|
||||
// SECURITY HEADERS (BACKLOG P0-13 / SEC-06).
|
||||
//
|
||||
// CHANGELOG.md has claimed since 2026-08 that this repo ships
|
||||
// CSP/HSTS/X-Frame-Options. Until this commit that claim was FALSE on `main`:
|
||||
// a repo-wide grep for `content-security-policy|strict-transport-security|
|
||||
// x-frame-options` matched exactly one file -- CHANGELOG.md itself. There was
|
||||
// no `headers()`, no middleware header injection, no reverse-proxy config in
|
||||
// this repo. The only thing that ever set HSTS on 0pt.app was Vercel's own
|
||||
// platform default, which is not this repo's and disappears the moment the app
|
||||
// moves to Coolify/Traefik (siax.config.json `deploy.target` is already
|
||||
// "coolify"; .gitea/workflows/cd.yml already points at a real Coolify app id).
|
||||
// Setting them here, at the framework level, means they travel with the app
|
||||
// regardless of host -- same idiom as sax3l/siax-io's next.config.mjs.
|
||||
//
|
||||
// CARVE-OUTS, each grep-verified against THIS tree (2026-09-07), not inherited
|
||||
// from an older branch:
|
||||
//
|
||||
// * script-src/style-src 'unsafe-inline' -- Next's App Router injects inline
|
||||
// hydration/RSC-payload scripts and next-themes injects an inline
|
||||
// no-flash script; Tailwind + recharts emit inline style attributes. A
|
||||
// per-request nonce threaded through middleware + layout is the real fix
|
||||
// and is deliberately NOT attempted here: this commit closes "zero
|
||||
// headers", not "zero inline tolerance". Tracked in BACKLOG.
|
||||
// * 'unsafe-eval' is added ONLY in `next dev` (webpack/Turbopack eval source
|
||||
// maps + HMR). Production never gets it. Guarded on NODE_ENV so a
|
||||
// dev-only relaxation can never leak into a built image, whose runtime
|
||||
// stage sets NODE_ENV=production (see Dockerfile).
|
||||
// * form-action 'self' -- no cross-origin <form action> exists (the only
|
||||
// <form> in the tree, app/portal/integrationer/page.tsx, is onSubmit-only).
|
||||
// The Stripe hop in app/bestall/forskott/page.tsx is
|
||||
// `window.location.href = body.url`, i.e. a top-level navigation, which no
|
||||
// shipped CSP directive gates -- `form-action` would not have applied to
|
||||
// it. An earlier draft of this change carried a
|
||||
// `form-action https://checkout.stripe.com` carve-out on that mistaken
|
||||
// premise; it is dropped here rather than kept as decorative permission.
|
||||
// * connect-src 'self' -- every external host in the tree
|
||||
// (api.stripe.com, graph.microsoft.com, login.microsoftonline.com,
|
||||
// id-customers.siax.io, api.opensanctions.org, opendata.rdw.nl) is reached
|
||||
// from server-side route handlers / adapters, never from the browser.
|
||||
// * frame-src -- see FRAME_SRC below. This is the one directive with a live
|
||||
// trap in it, so it is derived from env rather than hardcoded to 'self'.
|
||||
//
|
||||
// WHAT THIS FILE CANNOT DO: it sets response headers. It does not prove they
|
||||
// are served. `tests/security/security-headers.test.ts` asserts this config
|
||||
// keeps them (blocking, runs in "CI (SIAX Cloud) / quality"), and
|
||||
// `node scripts/siax-doctor.mjs --live` re-checks them against a running
|
||||
// instance. Neither replaces the other.
|
||||
// =============================================================================
|
||||
|
||||
const IS_DEV = process.env.NODE_ENV === "development"
|
||||
|
||||
/**
|
||||
* `components/embed.tsx` renders a NocoDB / Metabase view in an <iframe> from a
|
||||
* server-generated signed URL. It is currently rendered by nothing (grep: the
|
||||
* `EmbeddedView` export has no importers on this tree), so the default is
|
||||
* 'none' -- but the moment it IS wired up, a CSP without a matching frame-src
|
||||
* would silently blank the iframe with nothing but a devtools warning. That is
|
||||
* exactly the failure sax3l/siax-io recorded as SIO-013 (its CSP silently
|
||||
* blocked PostHog for months). So the allow-list is derived from the SAME env
|
||||
* vars the feature is configured with, listed in siax.config.json `env`.
|
||||
*
|
||||
* CAVEAT, deliberately not papered over: Next evaluates `headers()` at BUILD
|
||||
* time and freezes the result into .next/routes-manifest.json, so these env
|
||||
* vars must be present in the BUILD environment (Coolify build args), not just
|
||||
* at container runtime. Changing them on a running container has no effect
|
||||
* until the next build.
|
||||
*/
|
||||
function frameSrc() {
|
||||
const origins = []
|
||||
for (const raw of [process.env.NOCODB_BASE_URL, process.env.METABASE_SITE_URL]) {
|
||||
if (!raw) continue
|
||||
try {
|
||||
origins.push(new URL(raw).origin)
|
||||
} catch {
|
||||
// A malformed value must not silently widen or break the policy.
|
||||
}
|
||||
}
|
||||
return origins.length > 0 ? `frame-src ${[...new Set(origins)].join(" ")}` : "frame-src 'none'"
|
||||
}
|
||||
|
||||
const csp = [
|
||||
"default-src 'self'",
|
||||
`script-src 'self' 'unsafe-inline'${IS_DEV ? " 'unsafe-eval'" : ""}`,
|
||||
"style-src 'self' 'unsafe-inline'",
|
||||
"img-src 'self' data: blob:",
|
||||
"font-src 'self' data:",
|
||||
"connect-src 'self'",
|
||||
frameSrc(),
|
||||
"frame-ancestors 'none'",
|
||||
"form-action 'self'",
|
||||
"base-uri 'self'",
|
||||
"object-src 'none'",
|
||||
].join("; ")
|
||||
|
||||
const securityHeaders = [
|
||||
// `preload` is inert until 0pt.app is actually submitted at hstspreload.org.
|
||||
// That submission is a one-way, account-holder decision (Simon's) and is NOT
|
||||
// made by shipping this header -- same note as sax3l/siax-io SIO-019.
|
||||
{ key: "Strict-Transport-Security", value: "max-age=63072000; includeSubDomains; preload" },
|
||||
// DENY, not SAMEORIGIN: nothing in this app frames itself.
|
||||
{ key: "X-Frame-Options", value: "DENY" },
|
||||
{ key: "X-Content-Type-Options", value: "nosniff" },
|
||||
{ key: "Referrer-Policy", value: "strict-origin-when-cross-origin" },
|
||||
{ key: "Permissions-Policy", value: "camera=(), microphone=(), geolocation=(), payment=(), usb=()" },
|
||||
{ key: "Content-Security-Policy", value: csp },
|
||||
]
|
||||
|
||||
/** @type {import('next').NextConfig} */
|
||||
const nextConfig = {
|
||||
// Självhostad drift i SIAX Cloud (Coolify + Traefik) — ersätter Vercel.
|
||||
@@ -56,6 +166,14 @@ const nextConfig = {
|
||||
// aldrig den föråldrade `domains` — den dagen en extern bildvärd tillkommer.
|
||||
formats: ["image/avif", "image/webp"],
|
||||
},
|
||||
async headers() {
|
||||
return [
|
||||
{
|
||||
source: "/(.*)",
|
||||
headers: securityHeaders,
|
||||
},
|
||||
]
|
||||
},
|
||||
}
|
||||
|
||||
export default nextConfig
|
||||
|
||||
+489
-35
@@ -9,31 +9,108 @@
|
||||
// written it wasn't verified reachable/stable from this repo's frozen
|
||||
// lockfile install, and a doctor script that can silently fail to install
|
||||
// is worse than no doctor script. This checks the same class of
|
||||
// machine-checkable facts (required files, manifest shape) using only
|
||||
// Node's built-in `fs`/`path` -- no network, no registry.
|
||||
// machine-checkable facts using only Node's built-ins -- no network in the
|
||||
// default mode, no registry.
|
||||
//
|
||||
// =============================================================================
|
||||
// 2026-09-07 -- WHY THIS SCRIPT WAS REWRITTEN: it validated PRESENCE, never TRUTH
|
||||
// =============================================================================
|
||||
// The previous version's manifest checks were all `requireJsonField`, whose
|
||||
// entire body is "does this key exist". Measured consequence: running it with
|
||||
// siax.repo.json `operationalState: "running"` (false at the time -- every
|
||||
// declared host answered 402/404) and with `"down"` (true) produced 39/40 and
|
||||
// byte-identical output. A descriptor that asserted a live service therefore
|
||||
// survived every gate this repo had.
|
||||
//
|
||||
// DESIGN, and what is deliberately NOT done:
|
||||
//
|
||||
// 1. OFFLINE (default, and the only thing anything automated runs).
|
||||
// Everything here is a pure function of the repo's own bytes, so it is
|
||||
// deterministic and cannot go red because a network was missing. Three
|
||||
// kinds of check:
|
||||
// (a) VALUE checks -- enums, formats, patterns. `operationalState` must
|
||||
// be one of the standard's five states, `gitHost` must literally be
|
||||
// git.siax.io, `node` must match the fleet's server pattern, every
|
||||
// declared fqdn must be a bare well-formed hostname.
|
||||
// (b) INTERNAL CONSISTENCY -- the repo's files must agree with each
|
||||
// other. Every prod hostname in siax.repo.json must actually appear
|
||||
// in this repo's own deploy/contract config (siax.config.json
|
||||
// `environments.*.domain`, contracts/openapi.yaml `servers:`), and
|
||||
// vice versa. A hostname nothing deploys is a hostname nobody owns.
|
||||
// (c) SUBSTANTIATION -- the one that closes the historical hole.
|
||||
// Liveness is not knowable offline. But whether a liveness CLAIM is
|
||||
// BACKED is. `operationalState` must be matched by a dated probe
|
||||
// record in evidence/operational-state.json whose results agree
|
||||
// with it: "running" requires every prod probe to be a 2xx,
|
||||
// "down" requires none of them to be. Claiming "running" next to a
|
||||
// recorded 402 now fails. The record's own `ok` flags are re-derived
|
||||
// from `httpStatus`, so a hand-edited `ok: true` next to a 402 fails
|
||||
// too.
|
||||
//
|
||||
// 2. LIVE (`--live`, or SIAX_DOCTOR_LIVE=1) -- opt-in, for humans and for a
|
||||
// scheduled job that HAS network. Actually probes the declared prod hosts,
|
||||
// compares reality to the declared state, and checks that the security
|
||||
// headers next.config.mjs promises are really on the wire.
|
||||
// `--write-evidence` rewrites evidence/operational-state.json from that
|
||||
// probe. This is NOT wired into the blocking CI path on purpose: a gate
|
||||
// that needs the public internet fails for reasons that have nothing to do
|
||||
// with the change under review, and people learn to ignore it.
|
||||
//
|
||||
// Exit code 0 = every check passed. Exit code 1 = at least one check failed
|
||||
// (see the printed control-ID to find the BACKLOG/standard item).
|
||||
//
|
||||
// NOTE for reviewers: this script was authored alongside two sibling PRs
|
||||
// (fix/honest-health-check adds app/api/ready/route.ts;
|
||||
// ci/security-headers-and-scanning adds CODEOWNERS) that this branch does
|
||||
// not itself contain. Run this on a checkout with all three merged for a
|
||||
// clean 0-failure result -- a couple of FAILs on this branch alone, for
|
||||
// exactly those two files, is expected and not a bug in either PR.
|
||||
import { existsSync, readFileSync } from "node:fs";
|
||||
// `--json` prints a machine-readable result instead of the table; that is what
|
||||
// tests/repo/siax-doctor.test.ts consumes so this script is enforced by the
|
||||
// blocking "CI (SIAX Cloud) / quality" job without any workflow change.
|
||||
// =============================================================================
|
||||
import { existsSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { join, dirname } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const ROOT = join(dirname(fileURLToPath(import.meta.url)), "..");
|
||||
// ROOT is the repo being diagnosed. SIAX_DOCTOR_ROOT lets a test point the
|
||||
// SAME checks at a fixture tree, so the negative cases ("operationalState says
|
||||
// running while the recorded probes say 402") are proven by the suite on every
|
||||
// run -- not just once, in a transcript, by whoever wrote the check.
|
||||
const ROOT = process.env.SIAX_DOCTOR_ROOT
|
||||
? process.env.SIAX_DOCTOR_ROOT
|
||||
: join(dirname(fileURLToPath(import.meta.url)), "..");
|
||||
|
||||
const argv = process.argv.slice(2);
|
||||
const WANT_JSON = argv.includes("--json");
|
||||
const WANT_LIVE = argv.includes("--live") || process.env.SIAX_DOCTOR_LIVE === "1";
|
||||
const WANT_WRITE_EVIDENCE = argv.includes("--write-evidence");
|
||||
const PROBE_TIMEOUT_MS = Number(process.env.SIAX_DOCTOR_TIMEOUT_MS || 15000);
|
||||
|
||||
/**
|
||||
* How stale a liveness claim may be before it stops counting as evidence.
|
||||
* Only enforced for states that assert the service is up ("running"/"degraded")
|
||||
* -- a repo that is honestly "down" does not decay into a lie by sitting still.
|
||||
*/
|
||||
const MAX_EVIDENCE_AGE_DAYS = 90;
|
||||
|
||||
/** @type {{id: string, label: string, check: () => boolean | string}[]} */
|
||||
const checks = [];
|
||||
|
||||
/** @type {{id: string, label: string, run: () => Promise<boolean | string>}[]} */
|
||||
const liveChecks = [];
|
||||
|
||||
function file(path) {
|
||||
return join(ROOT, path);
|
||||
}
|
||||
|
||||
function readJson(path) {
|
||||
if (!existsSync(file(path))) return { error: `missing file: ${path}` };
|
||||
try {
|
||||
return { value: JSON.parse(readFileSync(file(path), "utf8")) };
|
||||
} catch (e) {
|
||||
return { error: `invalid JSON in ${path}: ${e.message}` };
|
||||
}
|
||||
}
|
||||
|
||||
function readText(path) {
|
||||
if (!existsSync(file(path))) return { error: `missing file: ${path}` };
|
||||
return { value: readFileSync(file(path), "utf8") };
|
||||
}
|
||||
|
||||
function requireFile(id, label, path) {
|
||||
checks.push({
|
||||
id,
|
||||
@@ -47,13 +124,8 @@ function requireJsonField(id, label, path, fieldPath) {
|
||||
id,
|
||||
label,
|
||||
check: () => {
|
||||
if (!existsSync(file(path))) return `missing file: ${path}`;
|
||||
let data;
|
||||
try {
|
||||
data = JSON.parse(readFileSync(file(path), "utf8"));
|
||||
} catch (e) {
|
||||
return `invalid JSON in ${path}: ${e.message}`;
|
||||
}
|
||||
const { value: data, error } = readJson(path);
|
||||
if (error) return error;
|
||||
let cur = data;
|
||||
for (const key of fieldPath.split(".")) {
|
||||
if (cur == null || !(key in cur)) return `missing field "${fieldPath}" in ${path}`;
|
||||
@@ -64,18 +136,44 @@ function requireJsonField(id, label, path, fieldPath) {
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* The check the old `requireJsonField` should have been: presence AND a
|
||||
* predicate on the value. `expectation` is prose printed on failure.
|
||||
*/
|
||||
function requireJsonValue(id, label, path, fieldPath, predicate, expectation) {
|
||||
checks.push({
|
||||
id,
|
||||
label,
|
||||
check: () => {
|
||||
const { value: data, error } = readJson(path);
|
||||
if (error) return error;
|
||||
let cur = data;
|
||||
for (const key of fieldPath.split(".")) {
|
||||
if (cur == null || !(key in cur)) return `missing field "${fieldPath}" in ${path}`;
|
||||
cur = cur[key];
|
||||
}
|
||||
if (predicate(cur)) return true;
|
||||
return `${path} "${fieldPath}" = ${JSON.stringify(cur)} -- expected ${expectation}`;
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
function requireTextMatch(id, label, path, pattern, patternLabel) {
|
||||
checks.push({
|
||||
id,
|
||||
label,
|
||||
check: () => {
|
||||
if (!existsSync(file(path))) return `missing file: ${path}`;
|
||||
const text = readFileSync(file(path), "utf8");
|
||||
const { value: text, error } = readText(path);
|
||||
if (error) return error;
|
||||
return pattern.test(text) ? true : `${path} does not contain ${patternLabel}`;
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
function check(id, label, fn) {
|
||||
checks.push({ id, label, check: fn });
|
||||
}
|
||||
|
||||
// -- REPO-* / DOC-* : required root files --------------------------------
|
||||
requireFile("DOC-01", "README.md exists", "README.md");
|
||||
requireFile("REPO-08", "LICENSE exists", "LICENSE");
|
||||
@@ -108,7 +206,7 @@ requireTextMatch(
|
||||
"a dated 'Senast verifierad mot kod' field",
|
||||
);
|
||||
|
||||
// -- P1-1 / REPO-03/04/06 : manifests --------------------------------------
|
||||
// -- P1-1 / REPO-03/04/06 : manifests, PRESENCE ---------------------------
|
||||
requireFile("P1-1/REPO-04", "siax.repo.json exists", "siax.repo.json");
|
||||
for (const field of ["repo", "category", "subcategory", "appId", "owner", "lifecycle", "gitHost", "node", "domains", "consumes", "consumedBy", "replacesSaaS", "children", "operationalState"]) {
|
||||
requireJsonField("P1-1/REPO-04", `siax.repo.json has required field "${field}"`, "siax.repo.json", field);
|
||||
@@ -118,6 +216,241 @@ for (const field of ["framework", "build", "start", "port", "healthcheck", "envi
|
||||
requireJsonField("P1-1/REPO-06", `siax.config.json has required field "${field}"`, "siax.config.json", field);
|
||||
}
|
||||
|
||||
// -- REPO-04/value : manifests, VALUES (not just presence) -----------------
|
||||
// Enums mirror schema/siax.repo.schema.json (standard v2.0). Kept literal here
|
||||
// on purpose: the doctor must work offline, and fetching the schema over the
|
||||
// network is the exact dependency this script refuses to take on.
|
||||
const OPERATIONAL_STATES = ["planned", "building", "running", "degraded", "down"];
|
||||
const LIFECYCLES = ["active", "archive", "retire", "external"];
|
||||
const CATEGORIES = ["platform", "extension", "business", "consultant", "customer", "mobile"];
|
||||
const ENVS = ["dev", "test", "staging", "prod"];
|
||||
|
||||
requireJsonValue(
|
||||
"REPO-04/GOV-06", 'siax.repo.json "operationalState" is one of the standard\'s five states',
|
||||
"siax.repo.json", "operationalState",
|
||||
(v) => OPERATIONAL_STATES.includes(v), `one of ${OPERATIONAL_STATES.join("|")}`,
|
||||
);
|
||||
requireJsonValue(
|
||||
"REPO-04/GOV-06", 'siax.repo.json "lifecycle" is a standard lifecycle',
|
||||
"siax.repo.json", "lifecycle",
|
||||
(v) => LIFECYCLES.includes(v), `one of ${LIFECYCLES.join("|")}`,
|
||||
);
|
||||
requireJsonValue(
|
||||
"REPO-04/GOV-01", 'siax.repo.json "category" is a standard category',
|
||||
"siax.repo.json", "category",
|
||||
(v) => CATEGORIES.includes(v), `one of ${CATEGORIES.join("|")}`,
|
||||
);
|
||||
requireJsonValue(
|
||||
"REPO-04", 'siax.repo.json "gitHost" is git.siax.io (schema const)',
|
||||
"siax.repo.json", "gitHost",
|
||||
(v) => v === "git.siax.io", '"git.siax.io"',
|
||||
);
|
||||
requireJsonValue(
|
||||
"REPO-04", 'siax.repo.json "node" is null or a real fleet node',
|
||||
"siax.repo.json", "node",
|
||||
(v) => v === null || (typeof v === "string" && /^server(5|6|7|8|9|10|11)$/.test(v)),
|
||||
"null, or server5..server11",
|
||||
);
|
||||
requireJsonValue(
|
||||
"REPO-06", 'siax.config.json "port" is a plausible TCP port',
|
||||
"siax.config.json", "port",
|
||||
(v) => Number.isInteger(v) && v > 0 && v < 65536, "an integer in 1..65535",
|
||||
);
|
||||
requireJsonValue(
|
||||
"REPO-06/CD-04", 'siax.config.json "healthcheck.path" is an absolute path',
|
||||
"siax.config.json", "healthcheck.path",
|
||||
(v) => typeof v === "string" && v.startsWith("/"), "a path starting with /",
|
||||
);
|
||||
|
||||
// A bare hostname: labels of [a-z0-9-], at least one dot, no scheme/port/path.
|
||||
const FQDN_RE = /^(?=.{1,253}$)(?!-)[a-z0-9-]{1,63}(?<!-)(\.(?!-)[a-z0-9-]{1,63}(?<!-))+$/;
|
||||
|
||||
check("NET-01", "every siax.repo.json domain entry is well-formed", () => {
|
||||
const { value: repo, error } = readJson("siax.repo.json");
|
||||
if (error) return error;
|
||||
if (!Array.isArray(repo.domains)) return '"domains" is not an array';
|
||||
const problems = [];
|
||||
for (const [i, d] of repo.domains.entries()) {
|
||||
if (!d || typeof d !== "object") { problems.push(`domains[${i}] is not an object`); continue; }
|
||||
if (typeof d.fqdn !== "string" || !FQDN_RE.test(d.fqdn)) {
|
||||
problems.push(`domains[${i}].fqdn=${JSON.stringify(d.fqdn)} is not a bare lowercase hostname (no scheme, port or path)`);
|
||||
}
|
||||
if (!ENVS.includes(d.env)) problems.push(`domains[${i}].env=${JSON.stringify(d.env)} not in ${ENVS.join("|")}`);
|
||||
if (typeof d.purpose !== "string" || d.purpose.trim() === "") problems.push(`domains[${i}].purpose is empty`);
|
||||
}
|
||||
const seen = new Set();
|
||||
for (const d of repo.domains) {
|
||||
if (d && typeof d.fqdn === "string") {
|
||||
if (seen.has(d.fqdn)) problems.push(`duplicate fqdn ${d.fqdn}`);
|
||||
seen.add(d.fqdn);
|
||||
}
|
||||
}
|
||||
return problems.length === 0 ? true : problems.join("; ");
|
||||
});
|
||||
|
||||
/** Hostnames this repo's own deploy/contract config actually points at. */
|
||||
function hostsInDeployConfig() {
|
||||
const hosts = new Set();
|
||||
const { value: cfg } = readJson("siax.config.json");
|
||||
if (cfg && cfg.environments && typeof cfg.environments === "object") {
|
||||
for (const env of Object.values(cfg.environments)) {
|
||||
if (env && typeof env.domain === "string") hosts.add(env.domain.toLowerCase());
|
||||
}
|
||||
}
|
||||
const { value: openapi } = readText("contracts/openapi.yaml");
|
||||
if (openapi) {
|
||||
// `servers:` entries are `- url: https://host`. Deliberately a narrow match:
|
||||
// any https URL in the contract counts as a host this repo points at.
|
||||
for (const m of openapi.matchAll(/^\s*-?\s*url:\s*["']?https?:\/\/([a-z0-9.-]+)/gim)) {
|
||||
hosts.add(m[1].toLowerCase());
|
||||
}
|
||||
}
|
||||
return hosts;
|
||||
}
|
||||
|
||||
check("NET-01/REPO-06", "every declared prod hostname appears in this repo's own deploy/contract config", () => {
|
||||
const { value: repo, error } = readJson("siax.repo.json");
|
||||
if (error) return error;
|
||||
if (!Array.isArray(repo.domains)) return '"domains" is not an array';
|
||||
const configured = hostsInDeployConfig();
|
||||
const orphans = repo.domains
|
||||
.filter((d) => d && d.env === "prod" && typeof d.fqdn === "string")
|
||||
.map((d) => d.fqdn.toLowerCase())
|
||||
.filter((f) => !configured.has(f));
|
||||
return orphans.length === 0
|
||||
? true
|
||||
: `declared but nothing in siax.config.json/contracts/openapi.yaml points at: ${orphans.join(", ")}`;
|
||||
});
|
||||
|
||||
check("REPO-06/NET-01", "every siax.config.json environment domain is declared in siax.repo.json domains", () => {
|
||||
const { value: repo, error: e1 } = readJson("siax.repo.json");
|
||||
if (e1) return e1;
|
||||
const { value: cfg, error: e2 } = readJson("siax.config.json");
|
||||
if (e2) return e2;
|
||||
const declared = new Map(
|
||||
(Array.isArray(repo.domains) ? repo.domains : [])
|
||||
.filter((d) => d && typeof d.fqdn === "string")
|
||||
.map((d) => [d.fqdn.toLowerCase(), d.env]),
|
||||
);
|
||||
const problems = [];
|
||||
for (const [envName, env] of Object.entries(cfg.environments ?? {})) {
|
||||
if (!env || typeof env.domain !== "string") continue;
|
||||
const fqdn = env.domain.toLowerCase();
|
||||
if (!declared.has(fqdn)) {
|
||||
problems.push(`siax.config.json environments.${envName}.domain=${fqdn} is not in siax.repo.json domains`);
|
||||
} else if (declared.get(fqdn) !== envName) {
|
||||
problems.push(`${fqdn} is env=${declared.get(fqdn)} in siax.repo.json but environments.${envName} in siax.config.json`);
|
||||
}
|
||||
}
|
||||
return problems.length === 0 ? true : problems.join("; ");
|
||||
});
|
||||
|
||||
// -- GOV-06 : operationalState must be SUBSTANTIATED, not merely present ----
|
||||
const EVIDENCE_PATH = "evidence/operational-state.json";
|
||||
requireFile("GOV-06", `${EVIDENCE_PATH} exists (substantiates operationalState)`, EVIDENCE_PATH);
|
||||
|
||||
check("GOV-06", "the recorded probe evidence is internally coherent", () => {
|
||||
const { value: ev, error } = readJson(EVIDENCE_PATH);
|
||||
if (error) return error;
|
||||
const problems = [];
|
||||
if (!Array.isArray(ev.probes) || ev.probes.length === 0) problems.push("no probes recorded");
|
||||
const when = Date.parse(ev.checkedAt ?? "");
|
||||
if (Number.isNaN(when)) problems.push(`checkedAt=${JSON.stringify(ev.checkedAt)} is not a parseable timestamp`);
|
||||
else if (when > Date.now() + 86_400_000) problems.push(`checkedAt=${ev.checkedAt} is in the future`);
|
||||
for (const [i, p] of (ev.probes ?? []).entries()) {
|
||||
if (!p || typeof p !== "object") { problems.push(`probes[${i}] is not an object`); continue; }
|
||||
if (typeof p.fqdn !== "string" || !FQDN_RE.test(p.fqdn)) problems.push(`probes[${i}].fqdn is not a hostname`);
|
||||
if (!Number.isInteger(p.httpStatus) && p.httpStatus !== null) problems.push(`probes[${i}].httpStatus must be an integer or null (unreachable)`);
|
||||
if (typeof p.ok !== "boolean") problems.push(`probes[${i}].ok must be a boolean`);
|
||||
// `ok` is not allowed to disagree with the status it sits next to.
|
||||
const derived = Number.isInteger(p.httpStatus) && p.httpStatus >= 200 && p.httpStatus < 400;
|
||||
if (typeof p.ok === "boolean" && p.ok !== derived) {
|
||||
problems.push(`probes[${i}] (${p.fqdn}) records ok=${p.ok} next to httpStatus=${JSON.stringify(p.httpStatus)}`);
|
||||
}
|
||||
}
|
||||
return problems.length === 0 ? true : problems.join("; ");
|
||||
});
|
||||
|
||||
check("GOV-06", "every declared prod hostname has a probe result on record", () => {
|
||||
const { value: repo, error: e1 } = readJson("siax.repo.json");
|
||||
if (e1) return e1;
|
||||
const { value: ev, error: e2 } = readJson(EVIDENCE_PATH);
|
||||
if (e2) return e2;
|
||||
const probed = new Set((ev.probes ?? []).map((p) => String(p?.fqdn ?? "").toLowerCase()));
|
||||
const missing = (Array.isArray(repo.domains) ? repo.domains : [])
|
||||
.filter((d) => d && d.env === "prod" && typeof d.fqdn === "string")
|
||||
.map((d) => d.fqdn.toLowerCase())
|
||||
.filter((f) => !probed.has(f));
|
||||
return missing.length === 0 ? true : `no probe recorded for: ${missing.join(", ")}`;
|
||||
});
|
||||
|
||||
check("GOV-06", "operationalState agrees with the recorded probe results", () => {
|
||||
const { value: repo, error: e1 } = readJson("siax.repo.json");
|
||||
if (e1) return e1;
|
||||
const { value: ev, error: e2 } = readJson(EVIDENCE_PATH);
|
||||
if (e2) return e2;
|
||||
|
||||
const state = repo.operationalState;
|
||||
if (ev.operationalState !== state) {
|
||||
return `siax.repo.json says "${state}" but ${EVIDENCE_PATH} was recorded for "${ev.operationalState}" -- re-probe (\`node scripts/siax-doctor.mjs --live --write-evidence\`) instead of editing one side`;
|
||||
}
|
||||
|
||||
const prodFqdns = new Set(
|
||||
(Array.isArray(repo.domains) ? repo.domains : [])
|
||||
.filter((d) => d && d.env === "prod" && typeof d.fqdn === "string")
|
||||
.map((d) => d.fqdn.toLowerCase()),
|
||||
);
|
||||
const prodProbes = (ev.probes ?? []).filter((p) => prodFqdns.has(String(p?.fqdn ?? "").toLowerCase()));
|
||||
const up = prodProbes.filter((p) => p.ok === true);
|
||||
const down = prodProbes.filter((p) => p.ok !== true);
|
||||
const describe = (list) => list.map((p) => `${p.fqdn}=${p.httpStatus ?? "unreachable"}`).join(", ");
|
||||
|
||||
// Staleness only bites states that ASSERT the service is up. A repo that is
|
||||
// honestly "down" does not become a liar by not being re-probed.
|
||||
if (state === "running" || state === "degraded") {
|
||||
const ageDays = (Date.now() - Date.parse(ev.checkedAt ?? "")) / 86_400_000;
|
||||
if (Number.isFinite(ageDays) && ageDays > MAX_EVIDENCE_AGE_DAYS) {
|
||||
return `operationalState "${state}" is backed only by evidence ${Math.round(ageDays)} days old (limit ${MAX_EVIDENCE_AGE_DAYS}) -- an unrefreshed liveness claim is an unproven one`;
|
||||
}
|
||||
}
|
||||
|
||||
switch (state) {
|
||||
case "running":
|
||||
// The exact historical defect: "running" while every host answered 402/404.
|
||||
return down.length === 0
|
||||
? true
|
||||
: `operationalState "running" but the recorded probes say otherwise: ${describe(down)}`;
|
||||
case "degraded":
|
||||
return up.length > 0
|
||||
? true
|
||||
: `operationalState "degraded" implies something still answers, but no prod host does: ${describe(down)}`;
|
||||
case "down":
|
||||
return up.length === 0
|
||||
? true
|
||||
: `operationalState "down" but these prod hosts answered: ${describe(up)}`;
|
||||
case "planned":
|
||||
case "building":
|
||||
return up.length === 0
|
||||
? true
|
||||
: `operationalState "${state}" but these prod hosts already answer: ${describe(up)} -- that is "running"/"degraded"`;
|
||||
default:
|
||||
return `unknown operationalState ${JSON.stringify(state)}`;
|
||||
}
|
||||
});
|
||||
|
||||
// -- SEC-06 / P0-13 : the security headers CHANGELOG.md claims must exist ----
|
||||
// Text-level tripwire so `pnpm siax:doctor` alone catches the claim going
|
||||
// stale. tests/security/security-headers.test.ts is the strict version (it
|
||||
// evaluates the config and asserts each value).
|
||||
check("SEC-06/P0-13", "next.config.mjs actually sets the security headers CHANGELOG.md claims", () => {
|
||||
const { value: text, error } = readText("next.config.mjs");
|
||||
if (error) return error;
|
||||
if (!/async\s+headers\s*\(/.test(text)) return "next.config.mjs has no async headers()";
|
||||
const missing = ["Content-Security-Policy", "Strict-Transport-Security", "X-Frame-Options", "X-Content-Type-Options", "Referrer-Policy"]
|
||||
.filter((h) => !text.includes(h));
|
||||
return missing.length === 0 ? true : `next.config.mjs never mentions: ${missing.join(", ")}`;
|
||||
});
|
||||
|
||||
// -- AGENT-02/03 : prompts -------------------------------------------------
|
||||
requireFile("AGENT-02", "prompts/000-INDEX.md exists", "prompts/000-INDEX.md");
|
||||
|
||||
@@ -134,21 +467,142 @@ requireFile("A4/DOC", "contracts/openapi.yaml exists", "contracts/openapi.yaml")
|
||||
requireFile("P0-3/CD-04", "app/api/health/route.ts exists", "app/api/health/route.ts");
|
||||
requireFile("P0-3/OBS-04", "app/api/ready/route.ts exists (readiness, not just liveness)", "app/api/ready/route.ts");
|
||||
|
||||
// --------------------------------------------------------------------------
|
||||
// =============================================================================
|
||||
// LIVE MODE (opt-in) -- the half that needs the world
|
||||
// =============================================================================
|
||||
const REQUIRED_LIVE_HEADERS = [
|
||||
"content-security-policy",
|
||||
"strict-transport-security",
|
||||
"x-frame-options",
|
||||
"x-content-type-options",
|
||||
"referrer-policy",
|
||||
];
|
||||
|
||||
let failures = 0;
|
||||
console.log("siax-doctor -- 0pt (repo-local subset, see scripts/siax-doctor.mjs header)\n");
|
||||
for (const { id, label, check } of checks) {
|
||||
const result = check();
|
||||
if (result === true) {
|
||||
console.log(` [PASS] ${id.padEnd(14)} ${label}`);
|
||||
} else {
|
||||
failures += 1;
|
||||
console.log(` [FAIL] ${id.padEnd(14)} ${label} -- ${result}`);
|
||||
async function probe(url) {
|
||||
const ac = new AbortController();
|
||||
const timer = setTimeout(() => ac.abort(), PROBE_TIMEOUT_MS);
|
||||
try {
|
||||
const res = await fetch(url, { method: "GET", redirect: "manual", signal: ac.signal });
|
||||
return { url, httpStatus: res.status, ok: res.status >= 200 && res.status < 400, headers: res.headers };
|
||||
} catch (e) {
|
||||
return { url, httpStatus: null, ok: false, headers: null, note: `unreachable: ${e.message}` };
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
}
|
||||
console.log(`\n${checks.length - failures}/${checks.length} checks passed.`);
|
||||
if (failures > 0) {
|
||||
console.log(`${failures} check(s) failed -- see control-IDs above and cross-reference docs/compliance/BACKLOG.md.`);
|
||||
process.exit(1);
|
||||
|
||||
function prodTargets() {
|
||||
const { value: repo } = readJson("siax.repo.json");
|
||||
const { value: cfg } = readJson("siax.config.json");
|
||||
const healthPath = cfg?.healthcheck?.path ?? "/";
|
||||
return (Array.isArray(repo?.domains) ? repo.domains : [])
|
||||
.filter((d) => d && d.env === "prod" && typeof d.fqdn === "string")
|
||||
.map((d) => ({ fqdn: d.fqdn, env: d.env, url: `https://${d.fqdn}${healthPath}` }));
|
||||
}
|
||||
|
||||
if (WANT_LIVE) {
|
||||
liveChecks.push({
|
||||
id: "GOV-06/live",
|
||||
label: "live probe of every prod host agrees with siax.repo.json operationalState",
|
||||
run: async () => {
|
||||
const { value: repo, error } = readJson("siax.repo.json");
|
||||
if (error) return error;
|
||||
const results = await Promise.all(prodTargets().map((t) => probe(t.url).then((r) => ({ ...t, ...r }))));
|
||||
globalThis.__siaxDoctorLiveResults = results;
|
||||
const up = results.filter((r) => r.ok);
|
||||
const summary = results.map((r) => `${r.fqdn}=${r.httpStatus ?? "unreachable"}`).join(", ");
|
||||
switch (repo.operationalState) {
|
||||
case "running":
|
||||
return up.length === results.length ? true : `declared "running", live probe says: ${summary}`;
|
||||
case "degraded":
|
||||
return up.length > 0 ? true : `declared "degraded", but nothing answered: ${summary}`;
|
||||
case "down":
|
||||
case "planned":
|
||||
case "building":
|
||||
return up.length === 0 ? true : `declared "${repo.operationalState}", but these answered: ${summary}`;
|
||||
default:
|
||||
return `unknown operationalState ${JSON.stringify(repo.operationalState)}`;
|
||||
}
|
||||
},
|
||||
});
|
||||
|
||||
liveChecks.push({
|
||||
id: "SEC-06/live",
|
||||
label: "a reachable prod host really serves the security headers (P0-13)",
|
||||
run: async () => {
|
||||
const results = globalThis.__siaxDoctorLiveResults ?? [];
|
||||
// Only responses THIS app actually produced are in scope. A 402/404 from
|
||||
// a disabled Vercel deployment has no CSP either, but that is not a
|
||||
// header defect this repo can fix -- it is a liveness fact, already
|
||||
// carried by GOV-06/live above and recorded in
|
||||
// evidence/operational-state.json `securityHeadersObserved`. Failing
|
||||
// here as well would make this check permanently red for a reason no
|
||||
// change to next.config.mjs could clear, which is how gates get ignored.
|
||||
const serving = results.filter((r) => r.headers && r.ok);
|
||||
if (serving.length === 0) return true;
|
||||
const problems = [];
|
||||
for (const r of serving) {
|
||||
const missing = REQUIRED_LIVE_HEADERS.filter((h) => !r.headers.get(h));
|
||||
if (missing.length) problems.push(`${r.fqdn} (HTTP ${r.httpStatus}) is missing: ${missing.join(", ")}`);
|
||||
}
|
||||
return problems.length === 0 ? true : problems.join("; ");
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
const results = [];
|
||||
for (const { id, label, check: fn } of checks) {
|
||||
const outcome = fn();
|
||||
results.push({ id, label, mode: "offline", pass: outcome === true, detail: outcome === true ? null : outcome });
|
||||
}
|
||||
for (const { id, label, run } of liveChecks) {
|
||||
const outcome = await run();
|
||||
results.push({ id, label, mode: "live", pass: outcome === true, detail: outcome === true ? null : outcome });
|
||||
}
|
||||
|
||||
if (WANT_LIVE && WANT_WRITE_EVIDENCE) {
|
||||
const { value: repo } = readJson("siax.repo.json");
|
||||
const live = globalThis.__siaxDoctorLiveResults ?? [];
|
||||
const existing = readJson(EVIDENCE_PATH).value ?? {};
|
||||
const next = {
|
||||
...existing,
|
||||
control: "GOV-06",
|
||||
operationalState: repo?.operationalState,
|
||||
checkedAt: new Date().toISOString().replace(/\.\d{3}Z$/, "Z"),
|
||||
method: `scripts/siax-doctor.mjs --live --write-evidence (fetch, ${PROBE_TIMEOUT_MS}ms timeout)`,
|
||||
probes: live.map((r) => ({
|
||||
fqdn: r.fqdn,
|
||||
env: r.env,
|
||||
url: r.url,
|
||||
httpStatus: r.httpStatus,
|
||||
ok: r.ok,
|
||||
note: r.note ?? (r.headers ? `server: ${r.headers.get("server") ?? "unknown"}` : ""),
|
||||
})),
|
||||
securityHeadersObserved: Object.fromEntries(
|
||||
REQUIRED_LIVE_HEADERS.map((h) => [h, live.find((r) => r.headers)?.headers?.get(h) ?? null]),
|
||||
),
|
||||
};
|
||||
writeFileSync(file(EVIDENCE_PATH), `${JSON.stringify(next, null, 2)}\n`);
|
||||
if (!WANT_JSON) console.log(`\nwrote ${EVIDENCE_PATH} from the live probe.\n`);
|
||||
}
|
||||
|
||||
const failures = results.filter((r) => !r.pass);
|
||||
|
||||
if (WANT_JSON) {
|
||||
console.log(JSON.stringify({ total: results.length, failed: failures.length, live: WANT_LIVE, results }, null, 2));
|
||||
} else {
|
||||
console.log("siax-doctor -- 0pt (repo-local subset, see scripts/siax-doctor.mjs header)");
|
||||
console.log(WANT_LIVE ? "mode: offline + LIVE probe\n" : "mode: offline only (add --live to probe declared hosts)\n");
|
||||
for (const r of results) {
|
||||
const tag = r.mode === "live" ? "~" : " ";
|
||||
if (r.pass) console.log(` ${tag}[PASS] ${r.id.padEnd(16)} ${r.label}`);
|
||||
else console.log(` ${tag}[FAIL] ${r.id.padEnd(16)} ${r.label} -- ${r.detail}`);
|
||||
}
|
||||
console.log(`\n${results.length - failures.length}/${results.length} checks passed.`);
|
||||
if (failures.length > 0) {
|
||||
console.log(`${failures.length} check(s) failed -- see control-IDs above and cross-reference docs/compliance/BACKLOG.md.`);
|
||||
}
|
||||
}
|
||||
|
||||
if (failures.length > 0) process.exit(1);
|
||||
|
||||
@@ -0,0 +1,371 @@
|
||||
/**
|
||||
* src/services/pii-gate-service.ts
|
||||
* ==================================
|
||||
* 0pt.pii_* — the GDPR guardrail (fail-closed). Enforced by default wherever
|
||||
* a SourceQueryService is constructed; consulted by workers before they
|
||||
* persist anything for a personal-data source.
|
||||
*
|
||||
* Owns 0pt.pii_policy.schema.v1 and the in-memory identity registry that
|
||||
* backs the subject rights surfaces (access/erasure, GDPR art. 15/17) and
|
||||
* the retention worker (GDPR art. 5(1)(e)). Every decision is emitted to
|
||||
* AUD0 as risk L3 events so CL0UD gets an audit trail without polling 0pt.
|
||||
*
|
||||
* Directory domains known to publish personuppgifter (ratsit.se, …) are
|
||||
* pre-classified as personal: until a PiiPolicy with a recorded lawful
|
||||
* basis exists for them, the gate REJECTS storage/query of those sources.
|
||||
*/
|
||||
|
||||
import { createHash } from "node:crypto";
|
||||
import type {
|
||||
PiiPolicy,
|
||||
CreatePiiPolicyInput,
|
||||
PiiIdentityKey,
|
||||
PiiErasureRequest,
|
||||
PiiAccessStatement,
|
||||
ProcessingPurpose,
|
||||
} from "../types/pii";
|
||||
import type { Source } from "../types/source";
|
||||
import { createAud0Client, type Aud0Client, type Aud0Event } from "../clients/aud0-client";
|
||||
|
||||
/** Domain publishes personuppgifter (directory/register listings). */
|
||||
const KNOWN_DIRECTORY_DOMAINS = [
|
||||
"ratsit.se",
|
||||
"merinfo.se",
|
||||
"hitta.se",
|
||||
"mrkoll.se",
|
||||
"eniro.se",
|
||||
"birthday.se",
|
||||
"upplysning.se",
|
||||
];
|
||||
|
||||
const DECISION_TEMPLATE = "PII source %s: %s";
|
||||
|
||||
export class PiiPolicyExistsError extends Error {}
|
||||
export class PiiLegalBasisError extends Error {}
|
||||
export class PiiMinimisationError extends Error {}
|
||||
export class PiiRetentionError extends Error {}
|
||||
export class PiiRobotsRateError extends Error {}
|
||||
export class PiiEntityLinkageError extends Error {}
|
||||
|
||||
export class PiiGateService {
|
||||
private policies: Map<string, PiiPolicy> = new Map();
|
||||
private registry: PiiIdentityKey[] = [];
|
||||
private aud0: Aud0Client;
|
||||
|
||||
constructor(opts?: { aud0BaseUrl?: string }) {
|
||||
this.aud0 = createAud0Client(opts?.aud0BaseUrl);
|
||||
}
|
||||
|
||||
/** True if the source either self-flags or is a known directory domain. */
|
||||
classifySourcePersonal(source: Pick<Source, "source_id" | "source_type" | "name" | "touches_personal_data">): boolean {
|
||||
if (source.touches_personal_data === true) return true;
|
||||
if (source.source_type !== "web" && source.source_type !== "api") return false;
|
||||
const host = (source.name || "").toLowerCase().replace(/^https?:\/\//, "").replace(/\/.*$/, "");
|
||||
return KNOWN_DIRECTORY_DOMAINS.some((d) => host === d || host.endsWith("." + d));
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ *
|
||||
* Policies
|
||||
* ------------------------------------------------------------------ */
|
||||
|
||||
async createPiiPolicy(input: CreatePiiPolicyInput): Promise<PiiPolicy> {
|
||||
if (this.policies.has(input.source_id)) {
|
||||
throw new Error(`PiiPolicy already exists for source ${input.source_id}`);
|
||||
}
|
||||
// Hard validation — a policy is the lawful-basis gate, so it cannot be
|
||||
// written with a half-legal posture.
|
||||
if (!input.basis_ref.trim()) throw new Error("basis_ref is required (LIA id, consent id, contract clause or legal provision)");
|
||||
if (!Number.isFinite(input.retention_days) || input.retention_days < 0) {
|
||||
throw new Error("retention_days must be a non-negative integer (0 = fetch-and-drop)");
|
||||
}
|
||||
if (!Array.isArray(input.field_allowlist) || input.field_allowlist.length === 0) {
|
||||
throw new Error("field_allowlist must contain at least one field (data minimisation)");
|
||||
}
|
||||
|
||||
// Recorded decision (Simon, 2026-08-31): legitimate interest is accepted for
|
||||
// BUSINESS CONTACTS — personal data only in correspondence with a company/org
|
||||
// entity. Enforce that posture so "B2B-kontakt, företagskopplad" cannot be
|
||||
// bypassed by a lazily-written policy.
|
||||
const purpose: ProcessingPurpose = input.purpose ?? "business_contact";
|
||||
const entityLinkedOnly = input.entity_linked_only ?? input.legal_basis === "legitimate_interest";
|
||||
if (input.legal_basis === "legitimate_interest") {
|
||||
if (purpose !== "business_contact") {
|
||||
throw new Error("legitimate_interest is only accepted with purpose=business_contact (recorded decision 2026-08-31)");
|
||||
}
|
||||
if (!entityLinkedOnly) {
|
||||
throw new Error("legitimate_interest requires entity_linked_only=true — personal data must be bound to a company/org entity");
|
||||
}
|
||||
}
|
||||
|
||||
const now = new Date().toISOString();
|
||||
const policy: PiiPolicy = {
|
||||
source_id: input.source_id,
|
||||
tenant_id: input.tenant_id,
|
||||
project_id: input.project_id,
|
||||
touches_personal_data: true,
|
||||
legal_basis: input.legal_basis,
|
||||
purpose,
|
||||
entity_linked_only: entityLinkedOnly,
|
||||
basis_ref: input.basis_ref,
|
||||
retention_days: input.retention_days,
|
||||
field_allowlist: [...input.field_allowlist],
|
||||
pseudonymize_keys: input.pseudonymize_keys ?? true,
|
||||
robots_and_rate_confirmed: input.robots_and_rate_confirmed ?? false,
|
||||
created_at: now,
|
||||
updated_at: null,
|
||||
created_by: input.created_by ?? null,
|
||||
};
|
||||
this.policies.set(input.source_id, policy);
|
||||
|
||||
await this.safeEmit({
|
||||
event_id: `evt_0pt_pii_policy_created_${input.source_id}`,
|
||||
event_type: "0pt.pii_policy_created",
|
||||
capability_id: "0pt.pii_policy_created",
|
||||
tenant_id: input.tenant_id,
|
||||
project_id: input.project_id,
|
||||
actor_id: input.created_by ?? "system",
|
||||
resource_type: "source_pii_policy",
|
||||
resource_id: input.source_id,
|
||||
risk_level: "L3",
|
||||
payload: {
|
||||
legal_basis: input.legal_basis,
|
||||
purpose,
|
||||
entity_linked_only: entityLinkedOnly,
|
||||
retention_days: input.retention_days,
|
||||
field_count: input.field_allowlist.length,
|
||||
},
|
||||
});
|
||||
return policy;
|
||||
}
|
||||
|
||||
async getPiiPolicy(sourceId: string): Promise<PiiPolicy | null> {
|
||||
return this.policies.get(sourceId) ?? null;
|
||||
}
|
||||
|
||||
async listPiiPolicies(tenantId?: string): Promise<PiiPolicy[]> {
|
||||
const all = Array.from(this.policies.values());
|
||||
return tenantId ? all.filter((p) => p.tenant_id === tenantId) : all;
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ *
|
||||
* Enforcement (fail-closed)
|
||||
* ------------------------------------------------------------------ */
|
||||
|
||||
/**
|
||||
* Gate for a store/query of `fields` against `source`. Throws unless the
|
||||
* source is non-personal or a lawful, minimised, retention-bounded,
|
||||
* robots/rate-confirmed policy is on record.
|
||||
*/
|
||||
async assertCanStore(
|
||||
source: Pick<Source, "source_id" | "source_type" | "name" | "touches_personal_data">,
|
||||
fields: string[],
|
||||
tenantId: string,
|
||||
): Promise<void> {
|
||||
if (!this.classifySourcePersonal(source)) return;
|
||||
|
||||
const policy = this.policies.get(source.source_id);
|
||||
if (!policy) {
|
||||
throw new PiiLegalBasisError(
|
||||
DECISION_TEMPLATE.replace("%s", source.source_id).replace(
|
||||
"%s",
|
||||
`is a personal-data source with no PiiPolicy on record — a lawful basis (${KNOWN_DIRECTORY_DOMAINS.includes(source.name || "") ? "including robots.txt + rate limit confirmation" : "art. 6 GDPR"}) must be registered before any processing`,
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
if (policy.retention_days === 0 && fields.length > 0) {
|
||||
throw new PiiRetentionError(
|
||||
DECISION_TEMPLATE.replace("%s", source.source_id).replace(
|
||||
"%s",
|
||||
"policy forbids retention (retention_days = 0); no fields may be stored",
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
const disallowed = fields.filter((f) => !policy.field_allowlist.includes(f));
|
||||
if (disallowed.length > 0) {
|
||||
throw new PiiMinimisationError(
|
||||
DECISION_TEMPLATE.replace("%s", source.source_id).replace(
|
||||
"%s",
|
||||
`field(s) outside the allowlist: ${disallowed.join(", ")} (GDPR art. 5(1)(c) minimisation)`,
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
if ((source.source_type === "web" || source.source_type === "api") && !policy.robots_and_rate_confirmed) {
|
||||
throw new PiiRobotsRateError(
|
||||
DECISION_TEMPLATE.replace("%s", source.source_id).replace(
|
||||
"%s",
|
||||
"robots.txt / rate-limit not confirmed — required for web & api personal-data sources; re-check the source terms (do not hardcode)",
|
||||
),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ *
|
||||
* Subject rights: access (art. 15) & erasure (art. 17), retention (art. 5)
|
||||
* ------------------------------------------------------------------ */
|
||||
|
||||
/**
|
||||
* Register that one value was stored. Only hashes are persisted; the
|
||||
* plaintext identity never reaches the registry.
|
||||
*/
|
||||
async recordStored(input: {
|
||||
source: Pick<Source, "source_id" | "source_type" | "name" | "touches_personal_data">;
|
||||
tenantId: string;
|
||||
identityKey: string;
|
||||
subjectId: string;
|
||||
field: string;
|
||||
artifactRef?: string;
|
||||
/** Company/org entity the value is bound to (orgnr/rgs/LEI). REQUIRED when the policy is entity_linked_only. */
|
||||
entityId?: string;
|
||||
}): Promise<PiiIdentityKey> {
|
||||
await this.assertCanStore(input.source, [input.field], input.tenantId);
|
||||
const policy = this.policies.get(input.source.source_id)!;
|
||||
|
||||
if (policy.entity_linked_only && !input.entityId) {
|
||||
throw new PiiEntityLinkageError(
|
||||
DECISION_TEMPLATE.replace("%s", input.source.source_id).replace(
|
||||
"%s",
|
||||
"policy requires entity linkage (business contact, recorded decision 2026-08-31) — pass entityId (orgnr/rgs/LEI); personal data may only be stored in correspondence with a company/org entity",
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
const now = Date.now();
|
||||
const key: PiiIdentityKey = {
|
||||
key_hash: hash256(`${sourceIdSalt(input.source.source_id)}:${input.identityKey}`),
|
||||
subject_hash: hash256(input.subjectId),
|
||||
entity_hash: input.entityId ? hash256(normaliseEntity(input.entityId)) : null,
|
||||
source_id: input.source.source_id,
|
||||
tenant_id: input.tenantId,
|
||||
field: input.field,
|
||||
artifact_ref: input.artifactRef ?? null,
|
||||
stored_at: new Date(now).toISOString(),
|
||||
expires_at: new Date(now + policy.retention_days * 86_400_000).toISOString(),
|
||||
status: "active",
|
||||
};
|
||||
this.registry.push(key);
|
||||
|
||||
await this.safeEmit({
|
||||
event_id: `evt_0pt_pii_retained_${input.source.source_id}_${key.key_hash.slice(0, 12)}`,
|
||||
event_type: "0pt.pii_data_retained",
|
||||
capability_id: "0pt.pii_retain_identity",
|
||||
tenant_id: input.tenantId,
|
||||
project_id: "",
|
||||
actor_id: "system",
|
||||
resource_type: "pii_identity_key",
|
||||
resource_id: key.key_hash,
|
||||
risk_level: "L3",
|
||||
payload: {
|
||||
source_id: input.source.source_id,
|
||||
field: input.field,
|
||||
expires_at: key.expires_at,
|
||||
entity_linked: key.entity_hash !== null,
|
||||
},
|
||||
});
|
||||
return key;
|
||||
}
|
||||
|
||||
/** Right of access — everything currently held on a subject across sources. */
|
||||
async accessStatement(subjectHash: string): Promise<PiiAccessStatement> {
|
||||
const rows = this.registry.filter((r) => r.subject_hash === subjectHash && r.status === "active");
|
||||
return { subject_hash: subjectHash, rows, generated_at: new Date().toISOString() };
|
||||
}
|
||||
|
||||
/** Right to erasure — tombstone every held row for the subject. */
|
||||
async erase(request: PiiErasureRequest): Promise<number> {
|
||||
let count = 0;
|
||||
for (const row of this.registry) {
|
||||
if (row.subject_hash === request.subject_hash && row.status === "active") {
|
||||
row.status = "erased";
|
||||
count += 1;
|
||||
}
|
||||
}
|
||||
await this.safeEmit({
|
||||
event_id: `evt_0pt_pii_erased_${request.subject_hash.slice(0, 12)}_${Date.now()}`,
|
||||
event_type: "0pt.pii_erased",
|
||||
capability_id: "0pt.pii_erase",
|
||||
tenant_id: "",
|
||||
project_id: "",
|
||||
actor_id: request.actor_id ?? "system",
|
||||
resource_type: "pii_subject",
|
||||
resource_id: request.subject_hash,
|
||||
risk_level: "L3",
|
||||
payload: { reason: request.reason, row_count: count },
|
||||
});
|
||||
return count;
|
||||
}
|
||||
|
||||
/** Storage limitation — purge rows past their retention deadline. */
|
||||
async enforceRetention(now: Date = new Date()): Promise<{ purged: number }> {
|
||||
let purged = 0;
|
||||
const keep: PiiIdentityKey[] = [];
|
||||
for (const row of this.registry) {
|
||||
if (row.expires_at <= now.toISOString() && row.status === "active") {
|
||||
row.status = "purged";
|
||||
purged += 1;
|
||||
} else {
|
||||
keep.push(row);
|
||||
}
|
||||
}
|
||||
if (purged > 0) {
|
||||
await this.safeEmit({
|
||||
event_id: `evt_0pt_pii_retention_${now.getTime()}`,
|
||||
event_type: "0pt.pii_retention_purged",
|
||||
capability_id: "0pt.pii_enforce_retention",
|
||||
tenant_id: "",
|
||||
project_id: "",
|
||||
actor_id: "system",
|
||||
resource_type: "pii_identity_key",
|
||||
resource_id: now.toISOString(),
|
||||
risk_level: "L3",
|
||||
payload: { row_count: purged },
|
||||
});
|
||||
}
|
||||
return { purged };
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ *
|
||||
* Log hygiene — no personuppgifter in logs/metrics/traces
|
||||
* ------------------------------------------------------------------ */
|
||||
|
||||
/** Scrub personnummer, e-mail and Swedish phone numbers from a log line. */
|
||||
redact(line: string): string {
|
||||
return line
|
||||
.replace(/\b\d{8}[-–]\d{4}\b/g, "[PERSONNUMMER]")
|
||||
.replace(/\b\d{6}[-–]\d{4}\b/g, "[PERSONNUMMER]")
|
||||
.replace(/\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b/g, "[EMAIL]")
|
||||
.replace(/(?:\+46|0046|0)\s?[1-9][\d\s.-]{7,13}/g, "[TELEFON]");
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
|
||||
private async safeEmit(event: Omit<Aud0Event, "app_id" | "created_at">): Promise<void> {
|
||||
const full: Aud0Event = {
|
||||
...event,
|
||||
app_id: "0pt",
|
||||
created_at: new Date().toISOString(),
|
||||
};
|
||||
try {
|
||||
await this.aud0.ingestEvent(full);
|
||||
} catch {
|
||||
// Audit is important but must never take down a compliance decision or
|
||||
// a retention pass. The caller's state change has already happened.
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function sourceIdSalt(sourceId: string): string {
|
||||
return `0pt-pii-v1:${sourceId}`;
|
||||
}
|
||||
|
||||
/** Normalise an org/rgs nummer or LEI to canonical digits/letters before hashing. */
|
||||
function normaliseEntity(entityId: string): string {
|
||||
return entityId.replace(/[^0-9A-Za-z]/g, "").toUpperCase();
|
||||
}
|
||||
|
||||
/** sha256 hex digest — never the plaintext. */
|
||||
export function hash256(value: string): string {
|
||||
return createHash("sha256").update(value, "utf8").digest("hex");
|
||||
}
|
||||
@@ -20,6 +20,7 @@
|
||||
|
||||
import type { Source } from "../types/source";
|
||||
import { SourceService } from "./source-service";
|
||||
import { PiiGateService } from "./pii-gate-service";
|
||||
import { createAud0Client, type Aud0Client } from "../clients/aud0-client";
|
||||
import { FabricOneLakeAdapter, type FabricOneLakeConnectionConfig, type AdapterQueryResult } from "../adapters/fabric-onelake-adapter";
|
||||
|
||||
@@ -76,16 +77,27 @@ function clamp(value: number, min: number, max: number): number {
|
||||
export class SourceQueryService {
|
||||
private sourceService: SourceService;
|
||||
private aud0: Aud0Client;
|
||||
/** GDPR guardrail — on by default, can be shared via opts so callers use one registry. */
|
||||
private piiGate: PiiGateService;
|
||||
|
||||
constructor(opts?: { sourceService?: SourceService; aud0BaseUrl?: string }) {
|
||||
constructor(opts?: { sourceService?: SourceService; aud0BaseUrl?: string; piiGate?: PiiGateService }) {
|
||||
// Accepts an existing SourceService so callers (e.g. the API route) share
|
||||
// one in-memory registry instead of each constructing an empty one.
|
||||
this.sourceService = opts?.sourceService ?? new SourceService();
|
||||
this.aud0 = createAud0Client(opts?.aud0BaseUrl);
|
||||
this.piiGate = opts?.piiGate ?? new PiiGateService({ aud0BaseUrl: opts?.aud0BaseUrl });
|
||||
}
|
||||
|
||||
async querySource(input: QuerySourceInput): Promise<QuerySourceResult> {
|
||||
const startedAt = Date.now();
|
||||
|
||||
// GDPR fail-closed: personal-data sources (self-flagged or known directory
|
||||
// domains such as ratsit.se) may not be processed without a PiiPolicy.
|
||||
// Runs BEFORE the capability/type gate so the lawful-basis error is the
|
||||
// one the caller sees for personal data.
|
||||
const preflight = await this.sourceService.getSource(input.source_id);
|
||||
await this.piiGate.assertCanStore(preflight, [], input.tenant_id);
|
||||
|
||||
const source = await this.getQueryableSource(input.source_id);
|
||||
|
||||
const maxRows = clamp(input.max_rows ?? SOURCE_QUERY_DEFAULT_MAX_ROWS, 1, SOURCE_QUERY_HARD_MAX_ROWS);
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
/**
|
||||
* src/types/pii.ts
|
||||
* =================
|
||||
* 0pt.pii_policy.schema.v1 — SIAX Cloud GDPR guardrail for sources that
|
||||
* CAN touch personuppgifter (personal data). Purpose: make processing of
|
||||
* personal data "tekniskt OK" per GDPR (art. 5, 6, 25, 32, 17, 15):
|
||||
* recorded legal basis, storage limitation, data minimisation,
|
||||
* pseudonymisation, and working access/erasure surfaces.
|
||||
*
|
||||
* The guardrail only ever OPENS when a lawful basis is on record; nothing in
|
||||
* 0pt today crawls personal data, so this ships as a fail-closed boundary:
|
||||
* no policy → no storage (and, for directory domains like ratsit.se, no
|
||||
* query at all).
|
||||
*/
|
||||
|
||||
/** GDPR art. 6(1) legal bases 0pt accepts for personal-data sources. */
|
||||
export type LegalBasis = "legitimate_interest" | "consent" | "contract" | "legal_obligation";
|
||||
|
||||
/**
|
||||
* The purpose of processing. SIAX's recorded decision (Simon, 2026-08-31):
|
||||
* personal data may be stored when LEGITIMATE INTEREST applies for business
|
||||
* contacts — i.e. bound to a company/org entity, for business purposes.
|
||||
* Other bases are allowed but every policy records its purpose.
|
||||
*/
|
||||
export type ProcessingPurpose = "business_contact" | "consent_marketing" | "contract_performance" | "legal_obligation";
|
||||
|
||||
/** Compliance posture for one source that touches personal data. */
|
||||
export interface PiiPolicy {
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
touches_personal_data: true;
|
||||
legal_basis: LegalBasis;
|
||||
purpose: ProcessingPurpose;
|
||||
/**
|
||||
* REQUIRED under legitimate_interest (business_contact): personal data may
|
||||
* only be stored in correspondence with a company/org entity (org/rgs-nr,
|
||||
* GLEIF/LEI). recordStored() rejects stray person-only rows.
|
||||
*/
|
||||
entity_linked_only: boolean;
|
||||
/**
|
||||
* Pointer to the evidence: a recorded Legitimate Interest Assessment
|
||||
* (LIA) document id, the consent record id, the contract clause, or the
|
||||
* legal provision. Must exist for audit/DPIA purposes — a bare claim is
|
||||
* not enough.
|
||||
*/
|
||||
basis_ref: string;
|
||||
/**
|
||||
* Max storage in days. 0 = fetch-and-drop only (never retained).
|
||||
* The retention worker deletes rows where stored_at + retention_days
|
||||
* < now (art. 5(1)(e)).
|
||||
*/
|
||||
retention_days: number;
|
||||
/**
|
||||
* The ONLY fields that may be stored for this source (art. 5(1)(c)
|
||||
* minimisation). Empty list = nothing may be stored.
|
||||
*/
|
||||
field_allowlist: string[];
|
||||
/**
|
||||
* Pseudonymise identity keys (personnummer, email, reg-number) before
|
||||
* storage; only sha256 digests are ever written (art. 25).
|
||||
*/
|
||||
pseudonymize_keys: boolean;
|
||||
/**
|
||||
* Must be true for web/api sources: robots.txt checked, rate limit in
|
||||
* force (SourceRatePolicy), no auth-wall / bot-bypass technique in use.
|
||||
*/
|
||||
robots_and_rate_confirmed: boolean;
|
||||
created_at: string;
|
||||
updated_at: string | null;
|
||||
created_by: string | null;
|
||||
}
|
||||
|
||||
export interface CreatePiiPolicyInput {
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
project_id: string;
|
||||
legal_basis: LegalBasis;
|
||||
basis_ref: string;
|
||||
retention_days: number;
|
||||
field_allowlist: string[];
|
||||
purpose?: ProcessingPurpose;
|
||||
entity_linked_only?: boolean;
|
||||
pseudonymize_keys?: boolean;
|
||||
robots_and_rate_confirmed?: boolean;
|
||||
created_by?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Registry row for one stored personal-data value. Only hashes are held —
|
||||
* the key_hash lets 0pt locate the subject (right of access, art. 15) and
|
||||
* the subject_hash lets it erase every artifact for that subject across
|
||||
* sources (right to erasure, art. 17) without ever storing the plaintext
|
||||
* identity.
|
||||
*/
|
||||
export interface PiiIdentityKey {
|
||||
key_hash: string;
|
||||
subject_hash: string;
|
||||
/** Company/org entity the value is bound to (orgnr/rgs/LEI), hashed. */
|
||||
entity_hash: string | null;
|
||||
source_id: string;
|
||||
tenant_id: string;
|
||||
field: string;
|
||||
artifact_ref: string | null;
|
||||
stored_at: string;
|
||||
expires_at: string;
|
||||
status: "active" | "erased" | "purged";
|
||||
}
|
||||
|
||||
export interface PiiErasureRequest {
|
||||
subject_hash: string;
|
||||
reason: "deleted_by_subject" | "retention_expired" | "consent_withdrawn";
|
||||
actor_id?: string;
|
||||
correlation_id?: string;
|
||||
}
|
||||
|
||||
export interface PiiAccessStatement {
|
||||
subject_hash: string;
|
||||
rows: PiiIdentityKey[];
|
||||
generated_at: string;
|
||||
}
|
||||
@@ -63,6 +63,13 @@ export interface Source {
|
||||
adapter_id?: string;
|
||||
/** Mirrors dim_source.access (docs/03-data-model.md). */
|
||||
access?: SourceAccess;
|
||||
/**
|
||||
* Explicit self-flag that this source CAN expose personuppgifter. When
|
||||
* true (or when the source is a known directory domain like ratsit.se),
|
||||
* the PiiGateService requires a PiiPolicy with a recorded lawful basis
|
||||
* before any storage/query (fail-closed, GDPR art. 6).
|
||||
*/
|
||||
touches_personal_data?: boolean;
|
||||
description?: string;
|
||||
status: SourceStatus;
|
||||
correlation_id?: string;
|
||||
@@ -86,6 +93,7 @@ export interface CreateSourceInput {
|
||||
name: string;
|
||||
adapter_id?: string;
|
||||
access?: SourceAccess;
|
||||
touches_personal_data?: boolean;
|
||||
description?: string;
|
||||
status?: SourceStatus;
|
||||
correlation_id?: string;
|
||||
|
||||
@@ -0,0 +1,201 @@
|
||||
import { describe, it, expect, vi } from "vitest";
|
||||
import { PiiGateService } from "@/src/services/pii-gate-service";
|
||||
import { SourceService } from "@/src/services/source-service";
|
||||
import { SourceQueryService } from "@/src/services/source-query-service";
|
||||
import type { FabricOneLakeConnectionConfig } from "@/src/adapters/fabric-onelake-adapter";
|
||||
|
||||
/** Minimal config — never reached (the pii gate fires first). */
|
||||
const EMPTY_CFG = {} as FabricOneLakeConnectionConfig;
|
||||
|
||||
describe("PiiGateService — GDPR guardrail (fail-closed)", () => {
|
||||
const ratsit = {
|
||||
source_id: "src_ratsit_1",
|
||||
tenant_id: "t1",
|
||||
project_id: "p1",
|
||||
source_type: "web" as const,
|
||||
name: "ratsit.se",
|
||||
status: "active" as const,
|
||||
created_at: new Date().toISOString(),
|
||||
};
|
||||
const company = {
|
||||
source_id: "src_company_1",
|
||||
tenant_id: "t1",
|
||||
project_id: "p1",
|
||||
source_type: "api" as const,
|
||||
name: "allabolag.se",
|
||||
status: "active" as const,
|
||||
created_at: new Date().toISOString(),
|
||||
};
|
||||
|
||||
it("classifies known directory domains as personal-data sources", () => {
|
||||
const gate = new PiiGateService();
|
||||
expect(gate.classifySourcePersonal(ratsit)).toBe(true);
|
||||
expect(gate.classifySourcePersonal({ ...ratsit, name: "www.ratsit.se" })).toBe(true);
|
||||
expect(gate.classifySourcePersonal(company)).toBe(false);
|
||||
expect(gate.classifySourcePersonal({ ...company, touches_personal_data: true })).toBe(true);
|
||||
});
|
||||
|
||||
it("rejects processing a personal-data source with no recorded lawful basis", async () => {
|
||||
const gate = new PiiGateService();
|
||||
await expect(gate.assertCanStore(ratsit, [], "t1")).rejects.toThrowError(/no PiiPolicy/);
|
||||
});
|
||||
|
||||
it("rejects half-legal policies (missing basis_ref, zero retention, empty allowlist)", async () => {
|
||||
const gate = new PiiGateService();
|
||||
await expect(
|
||||
gate.createPiiPolicy({ source_id: "s1", tenant_id: "t1", project_id: "p1", legal_basis: "legitimate_interest", basis_ref: " ", retention_days: 90, field_allowlist: ["address"] }),
|
||||
).rejects.toThrow(/basis_ref is required/);
|
||||
await expect(
|
||||
gate.createPiiPolicy({ source_id: "s1", tenant_id: "t1", project_id: "p1", legal_basis: "consent", basis_ref: "consent-record-1", retention_days: -1, field_allowlist: ["address"] }),
|
||||
).rejects.toThrow(/retention_days/);
|
||||
await expect(
|
||||
gate.createPiiPolicy({ source_id: "s1", tenant_id: "t1", project_id: "p1", legal_basis: "contract", basis_ref: "clause-9", retention_days: 90, field_allowlist: [] }),
|
||||
).rejects.toThrow(/data minimisation/);
|
||||
});
|
||||
|
||||
it("enforces the recorded decision: legitimate_interest = B2B business contact, company-linked only", async () => {
|
||||
const gate = new PiiGateService();
|
||||
await expect(
|
||||
gate.createPiiPolicy({ source_id: "s_li_wrong", tenant_id: "t1", project_id: "p1", legal_basis: "legitimate_interest", basis_ref: "lia-2026-001", retention_days: 90, field_allowlist: ["email"], purpose: "consent_marketing" }),
|
||||
).rejects.toThrowError(/only accepted with purpose=business_contact/);
|
||||
await expect(
|
||||
gate.createPiiPolicy({ source_id: "s_li_wrong2", tenant_id: "t1", project_id: "p1", legal_basis: "legitimate_interest", basis_ref: "lia-2026-001", retention_days: 90, field_allowlist: ["email"], entity_linked_only: false }),
|
||||
).rejects.toThrowError(/entity_linked_only=true/);
|
||||
|
||||
await gate.createPiiPolicy({
|
||||
source_id: "s_li_ok", tenant_id: "t1", project_id: "p1",
|
||||
legal_basis: "legitimate_interest", basis_ref: "lia-2026-b2b-01",
|
||||
retention_days: 365, field_allowlist: ["email", "phone"],
|
||||
});
|
||||
const p = await gate.getPiiPolicy("s_li_ok");
|
||||
expect(p?.purpose).toBe("business_contact");
|
||||
expect(p?.entity_linked_only).toBe(true);
|
||||
|
||||
// Business contact must be bound to a company — stray person-only rows are refused.
|
||||
const liSource = { ...company, source_id: "s_li_ok" };
|
||||
await expect(
|
||||
gate.recordStored({ source: liSource, tenantId: "t1", identityKey: "k", subjectId: "s", field: "email" }),
|
||||
).rejects.toThrowError(/entity linkage/);
|
||||
const linked = await gate.recordStored({ source: liSource, tenantId: "t1", identityKey: "k", subjectId: "s", field: "email", entityId: "556677-8899" });
|
||||
expect(linked.entity_hash).toMatch(/^[0-9a-f]{64}$/);
|
||||
expect(linked.entity_hash).not.toContain("556677");
|
||||
});
|
||||
|
||||
it("blocks out-of-allowlist fields (minimisation) and unconfirmed web sources", async () => {
|
||||
const gate = new PiiGateService();
|
||||
await gate.createPiiPolicy({
|
||||
source_id: ratsit.source_id, tenant_id: "t1", project_id: "p1",
|
||||
legal_basis: "consent", basis_ref: "consent-record-42",
|
||||
retention_days: 90, field_allowlist: ["name", "address"],
|
||||
});
|
||||
await expect(gate.assertCanStore(ratsit, ["name", "phone"], "t1")).rejects.toThrowError(/outside the allowlist/);
|
||||
await expect(gate.assertCanStore(ratsit, ["name"], "t1")).rejects.toThrowError(/robots.txt \/ rate-limit not confirmed/);
|
||||
|
||||
const ratsitConfirmed = { ...ratsit, source_id: "src_ratsit_confirmed" };
|
||||
await gate.createPiiPolicy({
|
||||
source_id: ratsitConfirmed.source_id, tenant_id: "t1", project_id: "p1",
|
||||
legal_basis: "consent", basis_ref: "consent-record-42",
|
||||
retention_days: 90, field_allowlist: ["name", "address"], robots_and_rate_confirmed: true,
|
||||
});
|
||||
await expect(gate.assertCanStore(ratsitConfirmed, ["name"], "t1")).resolves.toBeUndefined();
|
||||
});
|
||||
|
||||
it("forbids any storage when retention_days = 0", async () => {
|
||||
const gate = new PiiGateService();
|
||||
await gate.createPiiPolicy({
|
||||
source_id: "src_drop_1", tenant_id: "t1", project_id: "p1",
|
||||
legal_basis: "legitimate_interest", basis_ref: "lia-2026-001",
|
||||
retention_days: 0, field_allowlist: ["name"], robots_and_rate_confirmed: true,
|
||||
});
|
||||
await expect(
|
||||
gate.assertCanStore({ ...company, source_id: "src_drop_1", name: "ratsit.se" }, ["name"], "t1"),
|
||||
).rejects.toThrowError(/forbids retention/);
|
||||
});
|
||||
|
||||
it("stores ruled bases and enables access/erasure without plaintext identity", async () => {
|
||||
const gate = new PiiGateService();
|
||||
await gate.createPiiPolicy({
|
||||
source_id: ratsit.source_id, tenant_id: "t1", project_id: "p1",
|
||||
legal_basis: "legal_obligation", basis_ref: "prop-§5",
|
||||
retention_days: 30, field_allowlist: ["street_address"], robots_and_rate_confirmed: true,
|
||||
});
|
||||
const key = await gate.recordStored({
|
||||
source: ratsit, tenantId: "t1",
|
||||
identityKey: "20250101-1234", subjectId: "subj-77", field: "street_address", artifactRef: "st0re/dataset-a",
|
||||
});
|
||||
|
||||
expect(key.key_hash).toMatch(/^[0-9a-f]{64}$/);
|
||||
expect(key.key_hash).not.toContain("20250101");
|
||||
|
||||
const statement = await gate.accessStatement(key.subject_hash);
|
||||
expect(statement.rows).toHaveLength(1);
|
||||
expect(statement.rows[0].field).toBe("street_address");
|
||||
|
||||
const erased = await gate.erase({ subject_hash: key.subject_hash, reason: "deleted_by_subject" });
|
||||
expect(erased).toBe(1);
|
||||
expect((await gate.accessStatement(key.subject_hash)).rows).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("retention worker purges rows past their deadline", async () => {
|
||||
const gate = new PiiGateService();
|
||||
vi.useFakeTimers();
|
||||
vi.setSystemTime(new Date("2026-08-01T00:00:00Z"));
|
||||
await gate.createPiiPolicy({
|
||||
source_id: ratsit.source_id, tenant_id: "t1", project_id: "p1",
|
||||
legal_basis: "legitimate_interest", basis_ref: "lia-2026-002",
|
||||
retention_days: 30, field_allowlist: ["street_address"], robots_and_rate_confirmed: true,
|
||||
});
|
||||
await gate.recordStored({ source: ratsit, tenantId: "t1", identityKey: "k-1", subjectId: "s-1", field: "street_address", entityId: "5566778899" });
|
||||
|
||||
vi.setSystemTime(new Date("2026-09-15T00:00:00Z"));
|
||||
const { purged } = await gate.enforceRetention(new Date("2026-09-15T00:00:00Z"));
|
||||
vi.useRealTimers();
|
||||
expect(purged).toBe(1);
|
||||
});
|
||||
|
||||
it("redacts personnummer, e-mail and phone from log lines", () => {
|
||||
const gate = new PiiGateService();
|
||||
const out = gate.redact(
|
||||
"subject=202501011234 res=" +
|
||||
'evt=19720101-1234 mail=simon@siax.io tel=070-123 45 67 sök=ok',
|
||||
);
|
||||
expect(out).not.toMatch(/\d{8}-\d{4}/);
|
||||
expect(out).toContain("[PERSONNUMMER]");
|
||||
expect(out).toContain("[EMAIL]");
|
||||
expect(out).toContain("[TELEFON]");
|
||||
});
|
||||
|
||||
it("SourceQueryService blocks a personal-data source by default (fail-closed)", async () => {
|
||||
const sourceService = new SourceService();
|
||||
await sourceService.createSource({ ...ratsit });
|
||||
|
||||
const service = new SourceQueryService({ sourceService });
|
||||
await expect(
|
||||
service.querySource({
|
||||
source_id: ratsit.source_id,
|
||||
query: "SELECT 1",
|
||||
tenant_id: "t1", project_id: "p1",
|
||||
connection_config: EMPTY_CFG,
|
||||
}),
|
||||
).rejects.toThrowError(/no PiiPolicy/);
|
||||
});
|
||||
|
||||
it("SourceQueryService passes non-personal sources straight through to capability checks", async () => {
|
||||
const sourceService = new SourceService();
|
||||
await sourceService.createSource({
|
||||
source_id: "src_safe_1", tenant_id: "t1", project_id: "p1",
|
||||
source_type: "web", name: "example.org", status: "active",
|
||||
});
|
||||
|
||||
const service = new SourceQueryService({ sourceService });
|
||||
// Gate passes; the sync path then correctly rejects non-database sources.
|
||||
await expect(
|
||||
service.querySource({
|
||||
source_id: "src_safe_1",
|
||||
query: "SELECT 1",
|
||||
tenant_id: "t1", project_id: "p1",
|
||||
connection_config: EMPTY_CFG,
|
||||
}),
|
||||
).rejects.toThrowError(/source_type "web"/);
|
||||
});
|
||||
});
|
||||
@@ -24,10 +24,24 @@
|
||||
* prove the rows are really there, not an artifact of the service's own
|
||||
* in-process bookkeeping.
|
||||
*
|
||||
* beforeAll applies sql/002_source_registry.sql itself (every statement is
|
||||
* `CREATE TABLE/INDEX IF NOT EXISTS`, so this is idempotent and safe to run
|
||||
* against an already-migrated database too) -- this test does not require
|
||||
* the operator to have pre-run migrations.
|
||||
* beforeAll applies every sql/NNN_*.sql file in order (same restore-drill
|
||||
* pattern as tests/analytics/analytics-domain-postgres.realinfra.test.ts),
|
||||
* not just sql/002_source_registry.sql -- this test does not require the
|
||||
* operator to have pre-run migrations.
|
||||
*
|
||||
* CORRECTED (this pass, 2026-09-14): it used to apply ONLY
|
||||
* sql/002_source_registry.sql. That was enough the day this file was
|
||||
* written, but sql/007_ingestion_plane.sql later added `opt_sources.owner_id`
|
||||
* (`ALTER TABLE opt_sources ADD COLUMN IF NOT EXISTS owner_id TEXT`), and
|
||||
* src/repositories/postgres-source-repository.ts's INSERT has named that
|
||||
* column ever since. Because this env var is never set in CI (confirmed by
|
||||
* grep over .gitea/workflows/ci.yml — no job references
|
||||
* SOURCE_REGISTRY_TEST_DATABASE_URL), the drift was never caught: every
|
||||
* assertion below failed with `column "owner_id" of relation "opt_sources"
|
||||
* does not exist` the moment it was actually run against a real, freshly
|
||||
* migrated-to-002-only Postgres. The fix is to stop hand-picking one
|
||||
* migration file and replay the whole chain, exactly like the analytics
|
||||
* suite already does.
|
||||
*/
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import fs from "node:fs";
|
||||
@@ -56,9 +70,23 @@ beforeAll(async () => {
|
||||
adminSql`SELECT 1`,
|
||||
new Promise((_, reject) => setTimeout(() => reject(new Error("connect timeout")), 5000)),
|
||||
]);
|
||||
const migrationPath = path.join(process.cwd(), "sql", "002_source_registry.sql");
|
||||
const migration = fs.readFileSync(migrationPath, "utf8");
|
||||
await adminSql.unsafe(migration);
|
||||
// Replay every migration in order, not just 002 -- see the file header
|
||||
// "CORRECTED" note. Tolerate an individual file failing (mirrors
|
||||
// analytics-domain-postgres.realinfra.test.ts: sql/001 needs the
|
||||
// `vector` extension, which a stock postgres:16-alpine does not carry)
|
||||
// as long as the files this test actually depends on (002, 007) apply.
|
||||
const dir = path.join(process.cwd(), "sql");
|
||||
for (const file of fs
|
||||
.readdirSync(dir)
|
||||
.filter((f) => f.endsWith(".sql"))
|
||||
.sort()) {
|
||||
const body = fs.readFileSync(path.join(dir, file), "utf8");
|
||||
try {
|
||||
await adminSql.unsafe(body);
|
||||
} catch (err) {
|
||||
console.warn(`[realinfra] ${file} did not apply cleanly: ${String(err).slice(0, 200)}`);
|
||||
}
|
||||
}
|
||||
reachable = true;
|
||||
} catch (err) {
|
||||
reachable = false;
|
||||
|
||||
@@ -0,0 +1,209 @@
|
||||
/**
|
||||
* AGENT-04 / GOV-06 -- the doctor is itself gated, and is itself proven able to fail.
|
||||
*
|
||||
* Two things this file exists for:
|
||||
*
|
||||
* 1. `pnpm siax:doctor` is otherwise invoked only by `make doctor`, i.e. by
|
||||
* whoever remembers. Running it here puts it inside `pnpm run test`, which
|
||||
* is the BLOCKING "CI (SIAX Cloud) / quality" job -- no .gitea/workflows
|
||||
* change required (that path is protected on this repo).
|
||||
*
|
||||
* 2. The defect being fixed was a check that COULD NOT FAIL: the old doctor
|
||||
* reported 40/40 and byte-identical output with `operationalState` set to
|
||||
* "running" (false) and to "down" (true). So a green doctor is not enough
|
||||
* evidence here -- the negative cases are asserted too, against a fixture
|
||||
* tree via SIAX_DOCTOR_ROOT.
|
||||
*
|
||||
* Offline only. Nothing here touches the network: `--live` is opt-in and is
|
||||
* deliberately never exercised by CI.
|
||||
*/
|
||||
import { afterAll, describe, expect, it } from "vitest";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import path from "node:path";
|
||||
|
||||
const REPO_ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..");
|
||||
const DOCTOR = path.join(REPO_ROOT, "scripts", "siax-doctor.mjs");
|
||||
|
||||
type DoctorResult = { id: string; label: string; mode: string; pass: boolean; detail: string | null };
|
||||
type DoctorReport = { total: number; failed: number; live: boolean; results: DoctorResult[] };
|
||||
|
||||
function runDoctor(root?: string): { code: number; report: DoctorReport } {
|
||||
const env = { ...process.env, ...(root ? { SIAX_DOCTOR_ROOT: root } : {}) };
|
||||
try {
|
||||
const stdout = execFileSync(process.execPath, [DOCTOR, "--json"], { env, encoding: "utf8" });
|
||||
return { code: 0, report: JSON.parse(stdout) as DoctorReport };
|
||||
} catch (e) {
|
||||
const err = e as { status?: number; stdout?: string };
|
||||
return { code: err.status ?? -1, report: JSON.parse(err.stdout ?? "{}") as DoctorReport };
|
||||
}
|
||||
}
|
||||
|
||||
/** A fixture tree carrying only the two files the GOV-06 checks read. */
|
||||
function fixture(repo: unknown, evidence: unknown): string {
|
||||
const dir = mkdtempSync(path.join(tmpdir(), "siax-doctor-fixture-"));
|
||||
writeFileSync(path.join(dir, "siax.repo.json"), JSON.stringify(repo, null, 2));
|
||||
mkdirSync(path.join(dir, "evidence"), { recursive: true });
|
||||
writeFileSync(path.join(dir, "evidence", "operational-state.json"), JSON.stringify(evidence, null, 2));
|
||||
return dir;
|
||||
}
|
||||
|
||||
const REAL_REPO = JSON.parse(readFileSync(path.join(REPO_ROOT, "siax.repo.json"), "utf8"));
|
||||
const REAL_EVIDENCE = JSON.parse(
|
||||
readFileSync(path.join(REPO_ROOT, "evidence", "operational-state.json"), "utf8"),
|
||||
);
|
||||
|
||||
/** The GOV-06 failure detail for a fixture, or null when GOV-06 passed. */
|
||||
function gov06Failure(root: string): string | null {
|
||||
const { report } = runDoctor(root);
|
||||
const failed = report.results.filter((r) => r.id.startsWith("GOV-06") && !r.pass);
|
||||
return failed.length > 0 ? failed.map((r) => r.detail).join(" | ") : null;
|
||||
}
|
||||
|
||||
describe("AGENT-04: siax-doctor passes on this repo", () => {
|
||||
it("exits 0 with zero failures", () => {
|
||||
const { code, report } = runDoctor();
|
||||
const failures = report.results.filter((r) => !r.pass);
|
||||
expect(failures.map((f) => `${f.id}: ${f.detail}`).join("\n")).toBe("");
|
||||
expect(report.failed).toBe(0);
|
||||
expect(code).toBe(0);
|
||||
});
|
||||
|
||||
it("runs offline by default -- CI must never depend on reaching a host", () => {
|
||||
const { report } = runDoctor();
|
||||
expect(report.live).toBe(false);
|
||||
expect(report.results.every((r) => r.mode === "offline")).toBe(true);
|
||||
});
|
||||
|
||||
it("still carries the truth checks (they cannot be quietly deleted)", () => {
|
||||
const { report } = runDoctor();
|
||||
const ids = new Set(report.results.map((r) => r.id));
|
||||
for (const id of ["GOV-06", "NET-01", "NET-01/REPO-06", "SEC-06/P0-13", "REPO-04/GOV-06"]) {
|
||||
expect(ids.has(id), `siax-doctor no longer runs any "${id}" check`).toBe(true);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("GOV-06: a FALSE operationalState is detected (the historical defect)", () => {
|
||||
const dirs: string[] = [];
|
||||
const track = (d: string) => {
|
||||
dirs.push(d);
|
||||
return d;
|
||||
};
|
||||
afterAll(() => {
|
||||
for (const d of dirs) rmSync(d, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("accepts the repo's real, honest pairing (down + 402/404) as the control", () => {
|
||||
expect(gov06Failure(track(fixture(REAL_REPO, REAL_EVIDENCE)))).toBeNull();
|
||||
});
|
||||
|
||||
it('rejects operationalState "running" when the evidence was recorded for another state', () => {
|
||||
const detail = gov06Failure(track(fixture({ ...REAL_REPO, operationalState: "running" }, REAL_EVIDENCE)));
|
||||
expect(detail).toMatch(/running/);
|
||||
expect(detail).toMatch(/operational-state\.json/);
|
||||
});
|
||||
|
||||
it('rejects operationalState "running" when the recorded probes are 402/404', () => {
|
||||
const detail = gov06Failure(
|
||||
track(
|
||||
fixture(
|
||||
{ ...REAL_REPO, operationalState: "running" },
|
||||
{ ...REAL_EVIDENCE, operationalState: "running" },
|
||||
),
|
||||
),
|
||||
);
|
||||
expect(detail).toMatch(/but the recorded probes say otherwise/);
|
||||
expect(detail).toMatch(/402|404/);
|
||||
});
|
||||
|
||||
it("rejects an `ok: true` hand-written next to a non-2xx status", () => {
|
||||
const detail = gov06Failure(
|
||||
track(
|
||||
fixture(
|
||||
{ ...REAL_REPO, operationalState: "running" },
|
||||
{
|
||||
...REAL_EVIDENCE,
|
||||
operationalState: "running",
|
||||
probes: (REAL_EVIDENCE.probes as { ok: boolean }[]).map((p) => ({ ...p, ok: true })),
|
||||
},
|
||||
),
|
||||
),
|
||||
);
|
||||
expect(detail).toMatch(/records ok=true next to httpStatus/);
|
||||
});
|
||||
|
||||
it('rejects operationalState "down" when a prod host is recorded as answering', () => {
|
||||
const detail = gov06Failure(
|
||||
track(
|
||||
fixture(REAL_REPO, {
|
||||
...REAL_EVIDENCE,
|
||||
probes: (REAL_EVIDENCE.probes as { httpStatus: number; ok: boolean }[]).map((p) => ({
|
||||
...p,
|
||||
httpStatus: 200,
|
||||
ok: true,
|
||||
})),
|
||||
}),
|
||||
),
|
||||
);
|
||||
expect(detail).toMatch(/"down" but these prod hosts answered/);
|
||||
});
|
||||
|
||||
it("rejects a liveness claim backed only by stale evidence", () => {
|
||||
const stale = new Date(Date.now() - 400 * 86_400_000).toISOString();
|
||||
const detail = gov06Failure(
|
||||
track(
|
||||
fixture(
|
||||
{ ...REAL_REPO, operationalState: "running" },
|
||||
{
|
||||
...REAL_EVIDENCE,
|
||||
operationalState: "running",
|
||||
checkedAt: stale,
|
||||
probes: (REAL_EVIDENCE.probes as { httpStatus: number; ok: boolean }[]).map((p) => ({
|
||||
...p,
|
||||
httpStatus: 200,
|
||||
ok: true,
|
||||
})),
|
||||
},
|
||||
),
|
||||
),
|
||||
);
|
||||
expect(detail).toMatch(/unrefreshed liveness claim/);
|
||||
});
|
||||
|
||||
it("rejects a prod hostname with no probe on record at all", () => {
|
||||
const detail = gov06Failure(
|
||||
track(
|
||||
fixture(
|
||||
{
|
||||
...REAL_REPO,
|
||||
domains: [
|
||||
...(REAL_REPO.domains as unknown[]),
|
||||
{ fqdn: "invented.0pt.app", env: "prod", purpose: "x", registrar: "x" },
|
||||
],
|
||||
},
|
||||
REAL_EVIDENCE,
|
||||
),
|
||||
),
|
||||
);
|
||||
expect(detail).toMatch(/no probe recorded for: invented\.0pt\.app/);
|
||||
});
|
||||
|
||||
it("rejects a malformed hostname (NET-01)", () => {
|
||||
const { report } = runDoctor(
|
||||
track(
|
||||
fixture(
|
||||
{
|
||||
...REAL_REPO,
|
||||
domains: [{ fqdn: "https://0pt.app/health", env: "prod", purpose: "x", registrar: "x" }],
|
||||
},
|
||||
REAL_EVIDENCE,
|
||||
),
|
||||
),
|
||||
);
|
||||
const net01 = report.results.find((r) => r.id === "NET-01" && !r.pass);
|
||||
expect(net01?.detail).toMatch(/not a bare lowercase hostname/);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,184 @@
|
||||
/**
|
||||
* BACKLOG P0-13 / SEC-06 -- security-header REGRESSION GATE.
|
||||
*
|
||||
* Why this file exists: CHANGELOG.md asserted "security headers
|
||||
* (CSP/HSTS/X-Frame-Options/etc.)" while `main` contained zero such code for
|
||||
* weeks, and nothing in the repo could tell the difference. Adding the headers
|
||||
* without adding a gate would just reset that clock.
|
||||
*
|
||||
* WHAT THIS PROVES: that `next.config.mjs` -- the object Next.js actually
|
||||
* consumes -- still declares the required headers, with values that are still
|
||||
* strict. It reads the real config module; it does not re-import the header
|
||||
* list and compare it to itself. Every expectation below is written out here,
|
||||
* independently, so deleting or weakening a header in next.config.mjs fails
|
||||
* this test rather than silently changing both sides.
|
||||
*
|
||||
* WHAT IT DOES NOT PROVE: that a running instance serves them. Next freezes
|
||||
* `headers()` into .next/routes-manifest.json at build time, so the manifest
|
||||
* assertion at the bottom runs opportunistically when a build exists (locally,
|
||||
* or in any job that builds before testing). End-to-end, against a live
|
||||
* origin, is `node scripts/siax-doctor.mjs --live`.
|
||||
*
|
||||
* Runs inside `pnpm run test` -> the BLOCKING "CI (SIAX Cloud) / quality" job.
|
||||
*/
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { existsSync, readFileSync } from "node:fs";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import path from "node:path";
|
||||
|
||||
const REPO_ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..");
|
||||
|
||||
type HeaderEntry = { key: string; value: string };
|
||||
type HeaderRule = { source: string; headers: HeaderEntry[] };
|
||||
|
||||
/**
|
||||
* Required headers and how strict each must stay. Hardcoded on purpose --
|
||||
* this is the independent side of the comparison.
|
||||
*/
|
||||
const REQUIRED: { key: string; assert: (value: string) => void }[] = [
|
||||
{
|
||||
key: "Strict-Transport-Security",
|
||||
assert: (v) => {
|
||||
const maxAge = /max-age=(\d+)/.exec(v);
|
||||
expect(maxAge, `HSTS must carry a max-age, got: ${v}`).not.toBeNull();
|
||||
// >= 1 year: below that the header is decorative and ineligible for preload.
|
||||
expect(Number(maxAge![1])).toBeGreaterThanOrEqual(31536000);
|
||||
expect(v).toMatch(/includeSubDomains/i);
|
||||
},
|
||||
},
|
||||
{
|
||||
key: "X-Frame-Options",
|
||||
// SAMEORIGIN would be a weakening for this app: nothing here frames itself.
|
||||
assert: (v) => expect(v.trim().toUpperCase()).toBe("DENY"),
|
||||
},
|
||||
{
|
||||
key: "X-Content-Type-Options",
|
||||
assert: (v) => expect(v.trim().toLowerCase()).toBe("nosniff"),
|
||||
},
|
||||
{
|
||||
key: "Referrer-Policy",
|
||||
assert: (v) =>
|
||||
expect(["no-referrer", "same-origin", "strict-origin", "strict-origin-when-cross-origin"]).toContain(
|
||||
v.trim().toLowerCase(),
|
||||
),
|
||||
},
|
||||
{
|
||||
key: "Permissions-Policy",
|
||||
assert: (v) => {
|
||||
for (const feature of ["camera", "microphone", "geolocation"]) {
|
||||
expect(v, `Permissions-Policy must disable ${feature}`).toMatch(
|
||||
new RegExp(`${feature}\\s*=\\s*\\(\\s*\\)`),
|
||||
);
|
||||
}
|
||||
},
|
||||
},
|
||||
{
|
||||
key: "Content-Security-Policy",
|
||||
assert: (v) => {
|
||||
const directives = new Map(
|
||||
v
|
||||
.split(";")
|
||||
.map((d) => d.trim())
|
||||
.filter(Boolean)
|
||||
.map((d) => {
|
||||
const [name, ...rest] = d.split(/\s+/);
|
||||
return [name.toLowerCase(), rest.join(" ")] as const;
|
||||
}),
|
||||
);
|
||||
expect(directives.get("default-src")).toBe("'self'");
|
||||
// Clickjacking defence that actually applies to modern browsers.
|
||||
expect(directives.get("frame-ancestors")).toBe("'none'");
|
||||
expect(directives.get("base-uri")).toBe("'self'");
|
||||
expect(directives.get("object-src")).toBe("'none'");
|
||||
expect(directives.get("form-action")).toBe("'self'");
|
||||
expect(directives.has("script-src")).toBe(true);
|
||||
expect(directives.has("connect-src")).toBe(true);
|
||||
expect(directives.has("frame-src")).toBe(true);
|
||||
|
||||
// No wildcard sources anywhere: a bare `*` re-opens what this header exists to close.
|
||||
for (const [name, value] of directives) {
|
||||
expect(value.split(/\s+/), `CSP directive ${name} must not allow *`).not.toContain("*");
|
||||
}
|
||||
|
||||
// 'unsafe-eval' is a dev-only relaxation (see next.config.mjs). Vitest runs
|
||||
// with NODE_ENV=test, so it must be absent here; if it ever appears outside
|
||||
// `next dev`, this fails.
|
||||
expect(process.env.NODE_ENV, "vitest is expected to run outside NODE_ENV=development").not.toBe(
|
||||
"development",
|
||||
);
|
||||
expect(v, "'unsafe-eval' must never ship outside next dev").not.toContain("unsafe-eval");
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
/** Rule sources that cover every path Next can serve. */
|
||||
const CATCH_ALL_SOURCES = new Set(["/(.*)", "/:path*", "/(.*)?"]);
|
||||
|
||||
async function loadNextConfig(): Promise<{ poweredByHeader?: boolean; headers?: () => Promise<HeaderRule[]> }> {
|
||||
// Computed specifier on purpose: keeps `next.config.mjs` (a JS file outside
|
||||
// tsconfig.check.json's include) out of the blocking typecheck's graph.
|
||||
const url = new URL("../../next.config.mjs", import.meta.url).href;
|
||||
const mod = (await import(/* @vite-ignore */ url)) as { default: Awaited<ReturnType<typeof loadNextConfig>> };
|
||||
return mod.default;
|
||||
}
|
||||
|
||||
describe("SEC-06: 0pt serves its own security headers (P0-13)", () => {
|
||||
it("next.config.mjs declares a headers() rule covering every path", async () => {
|
||||
const config = await loadNextConfig();
|
||||
expect(typeof config.headers, "next.config.mjs must export an async headers()").toBe("function");
|
||||
|
||||
const rules = await config.headers!();
|
||||
expect(Array.isArray(rules)).toBe(true);
|
||||
|
||||
const catchAll = rules.filter((r) => CATCH_ALL_SOURCES.has(r.source));
|
||||
expect(
|
||||
catchAll.length,
|
||||
`no catch-all header rule; sources were: ${rules.map((r) => r.source).join(", ")}`,
|
||||
).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it("does not advertise the framework via X-Powered-By", async () => {
|
||||
const config = await loadNextConfig();
|
||||
expect(config.poweredByHeader).toBe(false);
|
||||
});
|
||||
|
||||
for (const { key, assert } of REQUIRED) {
|
||||
it(`sets ${key} on every path, and keeps it strict`, async () => {
|
||||
const config = await loadNextConfig();
|
||||
const rules = await config.headers!();
|
||||
const applied = rules
|
||||
.filter((r) => CATCH_ALL_SOURCES.has(r.source))
|
||||
.flatMap((r) => r.headers)
|
||||
.filter((h) => h.key.toLowerCase() === key.toLowerCase());
|
||||
|
||||
expect(
|
||||
applied.length,
|
||||
`${key} is not set for all paths -- CHANGELOG.md claims 0pt ships it. ` +
|
||||
`Either restore it in next.config.mjs or stop claiming it.`,
|
||||
).toBe(1);
|
||||
|
||||
assert(applied[0].value);
|
||||
});
|
||||
}
|
||||
|
||||
it("the built routes-manifest carries the same headers (skipped when no build exists)", async () => {
|
||||
const manifestPath = path.join(REPO_ROOT, ".next", "routes-manifest.json");
|
||||
if (!existsSync(manifestPath)) {
|
||||
// `pnpm run test` runs before `pnpm run build` in .gitea/workflows/ci.yml,
|
||||
// so on a clean CI checkout there is nothing to read. Not a silent pass:
|
||||
// the config-level assertions above already ran and are blocking.
|
||||
return;
|
||||
}
|
||||
const manifest = JSON.parse(readFileSync(manifestPath, "utf8")) as {
|
||||
headers?: { source?: string; regex?: string; headers?: HeaderEntry[] }[];
|
||||
};
|
||||
const built = (manifest.headers ?? []).flatMap((r) => r.headers ?? []);
|
||||
expect(built.length, "a build exists but its routes-manifest has no headers").toBeGreaterThan(0);
|
||||
for (const { key } of REQUIRED) {
|
||||
expect(
|
||||
built.some((h) => h.key.toLowerCase() === key.toLowerCase()),
|
||||
`${key} missing from .next/routes-manifest.json -- it would not be served`,
|
||||
).toBe(true);
|
||||
}
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user