Files
0pt/src/validation/0pt-schemas.ts
T
Simon AxelssonandClaude Sonnet 5 9740e20818
CI (SIAX Cloud) / security (push) Successful in 23s
Security Scan (SIAX Cloud) / security (push) Successful in 22s
CI (SIAX Cloud) / integration (push) Successful in 29s
CI (SIAX Cloud) / quality (push) Successful in 45s
CI (SIAX Cloud) / security (pull_request) Successful in 27s
CI (SIAX Cloud) / integration (pull_request) Successful in 31s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 29s
CI (SIAX Cloud) / quality (pull_request) Successful in 48s
feat(data-os): TS/Python quality bridge + provenance fields + fault-injection tests
Continues P010 (branch renamed from feat/p010-source-registry-persistence):
closes the three remaining BEFORE_STATE.md gaps (§2 TS/Python split-brain,
§3 missing provenance fields, §4 untested failure modes), all additive to
the production pipeline — the Python QA filter's delivery decision is
unchanged by any of this.

TS/Python bridge (§2):
  - New route surface app/api/0pt/quality/* (rules/runs/issues CRUD +
    POST /api/0pt/quality/bridge-run) over the shared QualityService
    singleton (src/services/quality.ts), fixing the same "no shared
    instance" defect the Source Registry had.
  - src/quality/quality-bridge.ts auto-provisions a deterministic,
    tenant-scoped baseline rule set on first use so the Python engine never
    pre-registers rules through the CRUD surface.
  - services/api/workers/quality_bridge.py: the Python-side HTTP client.
    Feature-flagged off by default (OPT_QUALITY_BRIDGE_ENABLED), fails open
    on any error (never raises into run_job, never changes `delivered`).
    Advisory only — real per-row gating against the TS engine isn't
    implementable yet (ExpectationEngine returns dataset-level pass/fail
    counts + samples, not a per-row boolean array); documented, not faked.
  - Live-verified end-to-end: `next dev` (AUTH_ENFORCE=false) + the real
    Python pipeline hitting the real brreg.no API, round-tripping through
    the real bridge route, receiving a real QualityService score.

Provenance (§3): sources/connectors.py's Provenance dataclass gains
retrieved_at, transform, connector_version, producer, quality,
entity_resolution, artifact_ref (all previously absent from the real
dataflow entirely) via a make_provenance() factory used by every connector.
resolve.py stamps entity_resolution on merged records' provenance (never on
unmerged ones). pipeline.py stamps artifact_ref at delivery and quality
once the bridge succeeds.

Failure tests (§4), real fault injection against real code (services/api,
new pytest + respx suite, not vitest mocks):
  - dedup-determinism + duplicate-input + out-of-order: resolve.py under
    shuffled/interleaved/exhaustive-permutation input, plus an explicit,
    verified test of the one case that IS legitimately order-sensitive
    (non-transitive similarity in a greedy clusterer) instead of hiding it.
  - schema-evolution: respx-mocked upstream responses with added/removed/
    alien-shaped fields through the real brreg connector + pipeline.
  - restart: connector exceptions mid-job (including one from respx itself)
    never crash run_job.
  - backfill: the real property that holds given no persistence layer
    exists (replay is compute-idempotent) — NOT a fabricated dedup-store
    test.
  - CDC-interruption and true backfill/delivery-store idempotency:
    explicitly documented as not implementable against this codebase
    (no CDC pipeline, no delivery store exist anywhere) via skip markers
    with reasons, in tests/test_known_gaps.py, rather than silently
    dropped or faked.

Testing: 256/256 vitest (249 pre-existing + 7 new bridge route/service
tests) and 28/28 pytest (2 documented skips) green locally. tsc -p
tsconfig.check.json has zero new errors (14 pre-existing errors, all in
the untracked/unaudited src/fleet, src/graph, src/metrics, src/projections
paths this pass does not touch, per BEFORE_STATE.md).

No PR opened, per instructions.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-01 21:02:14 +02:00

228 lines
8.9 KiB
TypeScript

/**
* `0pt/validation/0pt-schemas.ts` -- Wave 2 input validation for the 0pt
* capability API's MUTATING routes.
*
* Before Wave 2 every POST/PUT handler did `const body = await request.json()`
* and read fields straight off `body` (an implicit `any`), so a malformed
* payload reached the service layer untyped and surfaced as an opaque 500/400.
* These Zod schemas validate the request body at the edge and produce a typed,
* structured 400. This is VALIDATION ONLY -- it re-encodes the shape each
* handler already destructures; it does not add or change business logic.
*/
import { z } from 'zod';
/** A structured 400 parse result. */
export type ParseResult<T> =
| { ok: true; data: T }
| { ok: false; status: 400; body: { error: 'validation_error'; issues: { path: string; message: string }[] } };
/** Validate `body` against `schema`, returning parsed data or a 400 payload. */
export function parseBody<S extends z.ZodTypeAny>(schema: S, body: unknown): ParseResult<z.infer<S>> {
const result = schema.safeParse(body);
if (result.success) return { ok: true, data: result.data };
return {
ok: false,
status: 400,
body: {
error: 'validation_error',
issues: result.error.issues.map((i) => ({
path: i.path.join('.'),
message: i.message,
})),
},
};
}
// A permissive JSON object (used for opaque config blobs whose inner shape is
// owned by a downstream adapter, not this validation layer).
const jsonObject = z.record(z.string(), z.unknown());
/** POST /api/0pt/documents/capture */
export const documentCaptureSchema = z.object({
mailbox: z.object({
tenant_id: z.string().min(1),
client_id: z.string().min(1),
client_secret: z.string().min(1),
mailbox_address: z.string().min(1),
folder: z.string().min(1).optional(),
}),
since_iso: z.string().min(1).optional(),
top: z.number().int().positive().optional(),
max_pages: z.number().int().positive().optional(),
tenant_id: z.string().min(1),
project_id: z.string().min(1),
actor_id: z.string().min(1).optional(),
correlation_id: z.string().min(1).optional(),
});
/** POST /api/0pt/documents/[documentId]/extraction */
export const documentExtractionSchema = z.object({
tenant_id: z.string().min(1),
project_id: z.string().min(1),
actor_id: z.string().min(1).optional(),
correlation_id: z.string().min(1).optional(),
model_id: z.string().min(1).optional(),
});
/** POST /api/0pt/sources/[sourceId]/query */
export const sourceQuerySchema = z.object({
query: z.string().trim().min(1),
params: z.unknown().optional(),
tenant_id: z.string().min(1),
project_id: z.string().min(1),
actor_id: z.string().min(1).optional(),
correlation_id: z.string().min(1).optional(),
max_rows: z.number().int().positive().optional(),
timeout_ms: z.number().int().positive().optional(),
connection_config: jsonObject,
});
/** POST /api/0pt/sources/rate-policies -- mirrors CreateSourceRatePolicyInput. */
export const createSourceRatePolicySchema = z.object({
source_rate_policy_id: z.string().min(1),
tenant_id: z.string().min(1),
project_id: z.string().min(1),
source_id: z.string().min(1),
rate_limit_per_minute: z.number().int().nonnegative(),
crawl_budget: z.number().int().nonnegative(),
compliance_notes: z.string().optional(),
domain: z.string().min(1).optional(),
robots_ok: z.boolean().optional(),
access: z.enum(['api', 'bulk', 'authorized_web', 'licensed']).optional(),
status: z.enum(['active', 'paused', 'deprecated']).optional(),
correlation_id: z.string().min(1).optional(),
created_by: z.string().min(1).optional(),
});
/** Nested `rights` block on POST /api/0pt/sources, and the body of PUT /api/0pt/sources/[sourceId]/rights -- mirrors SourceRights (minus source_id/tenant_id, which come from the URL/outer body). */
const sourceRightsInputSchema = z.object({
license_id: z.string().min(1),
license_url: z.string().min(1).optional(),
allowed_purposes: z.array(z.string().min(1)).default([]),
pii_allowed: z.boolean(),
redistribution: z.enum(['internal', 'customer_deliverable', 'public', 'prohibited']),
attribution: z.string().min(1).optional(),
verified_at: z.string().min(1),
verified_by: z.string().min(1),
notes: z.string().min(1).optional(),
});
/** Nested `provenance` block on POST /api/0pt/sources, and the body of PUT /api/0pt/sources/[sourceId]/provenance -- mirrors SourceProvenance (minus source_id/tenant_id/updated_at, which are set by the service). */
const sourceProvenanceInputSchema = z.object({
method: z.enum(['official_api', 'bulk_download', 'authorized_web', 'licensed_feed', 'manual_upload']),
authority: z.string().min(1).optional(),
compliance_evidence_ref: z.string().min(1).optional(),
cadence: z.string().min(1).optional(),
freshness_sla_hours: z.number().nonnegative().optional(),
});
/** POST /api/0pt/sources -- source.register. */
export const registerSourceSchema = z.object({
source_id: z.string().min(1),
tenant_id: z.string().min(1),
project_id: z.string().min(1),
source_type: z.enum(['database', 'file', 'web', 'api', 'stream', 'manual']),
name: z.string().min(1),
adapter_id: z.string().min(1).optional(),
access: z.enum(['api', 'bulk', 'authorized_web', 'licensed']).optional(),
description: z.string().min(1).optional(),
status: z.enum(['active', 'paused', 'archived', 'deprecated']).optional(),
correlation_id: z.string().min(1).optional(),
created_by: z.string().min(1).optional(),
rights: sourceRightsInputSchema.optional(),
provenance: sourceProvenanceInputSchema.optional(),
});
/** PUT /api/0pt/sources/[sourceId]/rights -- source.set_rights. */
export const setSourceRightsSchema = sourceRightsInputSchema.extend({
tenant_id: z.string().min(1),
});
/** PUT /api/0pt/sources/[sourceId]/provenance -- source.set_provenance. */
export const setSourceProvenanceSchema = sourceProvenanceInputSchema.extend({
tenant_id: z.string().min(1),
});
/** POST /api/0pt/sources/[sourceId]/health -- source.health (record a health check). */
export const recordSourceHealthCheckSchema = z.object({
tenant_id: z.string().min(1),
status: z.enum(['up', 'degraded', 'down', 'unknown']),
checked_at: z.string().min(1).optional(),
latency_ms: z.number().nonnegative().optional(),
detail: z.string().min(1).optional(),
});
/** POST /api/0pt/lakehouse/mappings -- mirrors MapDatasetToIcebergTableInput. */
export const mapDatasetToIcebergTableSchema = z.object({
tenant_id: z.string().min(1),
project_id: z.string().min(1),
dataset_id: z.string().min(1),
dataset_iceberg_mapping_id: z.string().min(1),
namespace: z.string().min(1),
table_name: z.string().min(1),
storage_uri: z.string().min(1),
catalog_ref: z.string().min(1),
table_format: z.string().min(1).optional(),
actor_id: z.string().min(1),
correlation_id: z.string().min(1),
});
/** POST /api/0pt/lakehouse/mappings/[mappingId]/writes -- RequestIcebergWriteInput. */
export const requestIcebergWriteSchema = z.object({
actor_id: z.string().min(1),
correlation_id: z.string().min(1),
snapshot_type: z.string().min(1).optional(),
schema_version: z.string().min(1),
manifest_uri: z.string().min(1),
record_count: z.number().int().nonnegative().optional(),
size_bytes: z.number().int().nonnegative().optional(),
heavy: z.boolean().optional(),
});
/** POST /api/0pt/quality/rules -- mirrors CreateQualityRuleInput (src/quality/quality-types.ts). */
export const createQualityRuleSchema = z.object({
rule_id: z.string().min(1),
tenant_id: z.string().min(1),
project_id: z.string().min(1),
dataset_id: z.string().min(1),
name: z.string().min(1),
expectation: z.object({
expectation_type: z.string().min(1),
kwargs: jsonObject,
}),
severity: z.enum(['info', 'warning', 'error', 'critical']).optional(),
supplied_by: z.string().min(1).optional(),
});
/**
* POST /api/0pt/quality/runs -- runs the currently-registered active rules
* for a dataset against caller-supplied rows (QualityService.runQuality).
*/
export const createQualityRunSchema = z.object({
tenant_id: z.string().min(1),
project_id: z.string().min(1),
dataset_id: z.string().min(1),
dataset_version: z.number().int().optional(),
rows: z.array(jsonObject).optional(),
correlation_id: z.string().min(1).optional(),
});
/**
* POST /api/0pt/quality/bridge-run -- the Python engine's (services/api/workers/pipeline.py)
* bridge into the TS QualityService. Auto-provisions a baseline rule set for
* the (tenant, dataset) pair the first time it is called, so the engine never
* has to pre-register rules through the CRUD surface just to get an advisory
* quality read. `rows` is required here (unlike createQualityRunSchema) --
* the bridge always runs synchronously against real, already-produced rows;
* it never triggers a delegated/no-rows run.
*/
export const qualityBridgeRunSchema = z.object({
tenant_id: z.string().min(1),
project_id: z.string().min(1),
dataset_id: z.string().min(1),
dataset_version: z.number().int().optional(),
rows: z.array(jsonObject),
correlation_id: z.string().min(1).optional(),
});