CI (SIAX Cloud) / security (push) Successful in 23s
Security Scan (SIAX Cloud) / security (push) Successful in 22s
CI (SIAX Cloud) / integration (push) Successful in 29s
CI (SIAX Cloud) / quality (push) Successful in 45s
CI (SIAX Cloud) / security (pull_request) Successful in 27s
CI (SIAX Cloud) / integration (pull_request) Successful in 31s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 29s
CI (SIAX Cloud) / quality (pull_request) Successful in 48s
Continues P010 (branch renamed from feat/p010-source-registry-persistence):
closes the three remaining BEFORE_STATE.md gaps (§2 TS/Python split-brain,
§3 missing provenance fields, §4 untested failure modes), all additive to
the production pipeline — the Python QA filter's delivery decision is
unchanged by any of this.
TS/Python bridge (§2):
- New route surface app/api/0pt/quality/* (rules/runs/issues CRUD +
POST /api/0pt/quality/bridge-run) over the shared QualityService
singleton (src/services/quality.ts), fixing the same "no shared
instance" defect the Source Registry had.
- src/quality/quality-bridge.ts auto-provisions a deterministic,
tenant-scoped baseline rule set on first use so the Python engine never
pre-registers rules through the CRUD surface.
- services/api/workers/quality_bridge.py: the Python-side HTTP client.
Feature-flagged off by default (OPT_QUALITY_BRIDGE_ENABLED), fails open
on any error (never raises into run_job, never changes `delivered`).
Advisory only — real per-row gating against the TS engine isn't
implementable yet (ExpectationEngine returns dataset-level pass/fail
counts + samples, not a per-row boolean array); documented, not faked.
- Live-verified end-to-end: `next dev` (AUTH_ENFORCE=false) + the real
Python pipeline hitting the real brreg.no API, round-tripping through
the real bridge route, receiving a real QualityService score.
Provenance (§3): sources/connectors.py's Provenance dataclass gains
retrieved_at, transform, connector_version, producer, quality,
entity_resolution, artifact_ref (all previously absent from the real
dataflow entirely) via a make_provenance() factory used by every connector.
resolve.py stamps entity_resolution on merged records' provenance (never on
unmerged ones). pipeline.py stamps artifact_ref at delivery and quality
once the bridge succeeds.
Failure tests (§4), real fault injection against real code (services/api,
new pytest + respx suite, not vitest mocks):
- dedup-determinism + duplicate-input + out-of-order: resolve.py under
shuffled/interleaved/exhaustive-permutation input, plus an explicit,
verified test of the one case that IS legitimately order-sensitive
(non-transitive similarity in a greedy clusterer) instead of hiding it.
- schema-evolution: respx-mocked upstream responses with added/removed/
alien-shaped fields through the real brreg connector + pipeline.
- restart: connector exceptions mid-job (including one from respx itself)
never crash run_job.
- backfill: the real property that holds given no persistence layer
exists (replay is compute-idempotent) — NOT a fabricated dedup-store
test.
- CDC-interruption and true backfill/delivery-store idempotency:
explicitly documented as not implementable against this codebase
(no CDC pipeline, no delivery store exist anywhere) via skip markers
with reasons, in tests/test_known_gaps.py, rather than silently
dropped or faked.
Testing: 256/256 vitest (249 pre-existing + 7 new bridge route/service
tests) and 28/28 pytest (2 documented skips) green locally. tsc -p
tsconfig.check.json has zero new errors (14 pre-existing errors, all in
the untracked/unaudited src/fleet, src/graph, src/metrics, src/projections
paths this pass does not touch, per BEFORE_STATE.md).
No PR opened, per instructions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
228 lines
8.9 KiB
TypeScript
228 lines
8.9 KiB
TypeScript
/**
|
|
* `0pt/validation/0pt-schemas.ts` -- Wave 2 input validation for the 0pt
|
|
* capability API's MUTATING routes.
|
|
*
|
|
* Before Wave 2 every POST/PUT handler did `const body = await request.json()`
|
|
* and read fields straight off `body` (an implicit `any`), so a malformed
|
|
* payload reached the service layer untyped and surfaced as an opaque 500/400.
|
|
* These Zod schemas validate the request body at the edge and produce a typed,
|
|
* structured 400. This is VALIDATION ONLY -- it re-encodes the shape each
|
|
* handler already destructures; it does not add or change business logic.
|
|
*/
|
|
|
|
import { z } from 'zod';
|
|
|
|
/** A structured 400 parse result. */
|
|
export type ParseResult<T> =
|
|
| { ok: true; data: T }
|
|
| { ok: false; status: 400; body: { error: 'validation_error'; issues: { path: string; message: string }[] } };
|
|
|
|
/** Validate `body` against `schema`, returning parsed data or a 400 payload. */
|
|
export function parseBody<S extends z.ZodTypeAny>(schema: S, body: unknown): ParseResult<z.infer<S>> {
|
|
const result = schema.safeParse(body);
|
|
if (result.success) return { ok: true, data: result.data };
|
|
return {
|
|
ok: false,
|
|
status: 400,
|
|
body: {
|
|
error: 'validation_error',
|
|
issues: result.error.issues.map((i) => ({
|
|
path: i.path.join('.'),
|
|
message: i.message,
|
|
})),
|
|
},
|
|
};
|
|
}
|
|
|
|
// A permissive JSON object (used for opaque config blobs whose inner shape is
|
|
// owned by a downstream adapter, not this validation layer).
|
|
const jsonObject = z.record(z.string(), z.unknown());
|
|
|
|
/** POST /api/0pt/documents/capture */
|
|
export const documentCaptureSchema = z.object({
|
|
mailbox: z.object({
|
|
tenant_id: z.string().min(1),
|
|
client_id: z.string().min(1),
|
|
client_secret: z.string().min(1),
|
|
mailbox_address: z.string().min(1),
|
|
folder: z.string().min(1).optional(),
|
|
}),
|
|
since_iso: z.string().min(1).optional(),
|
|
top: z.number().int().positive().optional(),
|
|
max_pages: z.number().int().positive().optional(),
|
|
tenant_id: z.string().min(1),
|
|
project_id: z.string().min(1),
|
|
actor_id: z.string().min(1).optional(),
|
|
correlation_id: z.string().min(1).optional(),
|
|
});
|
|
|
|
/** POST /api/0pt/documents/[documentId]/extraction */
|
|
export const documentExtractionSchema = z.object({
|
|
tenant_id: z.string().min(1),
|
|
project_id: z.string().min(1),
|
|
actor_id: z.string().min(1).optional(),
|
|
correlation_id: z.string().min(1).optional(),
|
|
model_id: z.string().min(1).optional(),
|
|
});
|
|
|
|
/** POST /api/0pt/sources/[sourceId]/query */
|
|
export const sourceQuerySchema = z.object({
|
|
query: z.string().trim().min(1),
|
|
params: z.unknown().optional(),
|
|
tenant_id: z.string().min(1),
|
|
project_id: z.string().min(1),
|
|
actor_id: z.string().min(1).optional(),
|
|
correlation_id: z.string().min(1).optional(),
|
|
max_rows: z.number().int().positive().optional(),
|
|
timeout_ms: z.number().int().positive().optional(),
|
|
connection_config: jsonObject,
|
|
});
|
|
|
|
/** POST /api/0pt/sources/rate-policies -- mirrors CreateSourceRatePolicyInput. */
|
|
export const createSourceRatePolicySchema = z.object({
|
|
source_rate_policy_id: z.string().min(1),
|
|
tenant_id: z.string().min(1),
|
|
project_id: z.string().min(1),
|
|
source_id: z.string().min(1),
|
|
rate_limit_per_minute: z.number().int().nonnegative(),
|
|
crawl_budget: z.number().int().nonnegative(),
|
|
compliance_notes: z.string().optional(),
|
|
domain: z.string().min(1).optional(),
|
|
robots_ok: z.boolean().optional(),
|
|
access: z.enum(['api', 'bulk', 'authorized_web', 'licensed']).optional(),
|
|
status: z.enum(['active', 'paused', 'deprecated']).optional(),
|
|
correlation_id: z.string().min(1).optional(),
|
|
created_by: z.string().min(1).optional(),
|
|
});
|
|
|
|
/** Nested `rights` block on POST /api/0pt/sources, and the body of PUT /api/0pt/sources/[sourceId]/rights -- mirrors SourceRights (minus source_id/tenant_id, which come from the URL/outer body). */
|
|
const sourceRightsInputSchema = z.object({
|
|
license_id: z.string().min(1),
|
|
license_url: z.string().min(1).optional(),
|
|
allowed_purposes: z.array(z.string().min(1)).default([]),
|
|
pii_allowed: z.boolean(),
|
|
redistribution: z.enum(['internal', 'customer_deliverable', 'public', 'prohibited']),
|
|
attribution: z.string().min(1).optional(),
|
|
verified_at: z.string().min(1),
|
|
verified_by: z.string().min(1),
|
|
notes: z.string().min(1).optional(),
|
|
});
|
|
|
|
/** Nested `provenance` block on POST /api/0pt/sources, and the body of PUT /api/0pt/sources/[sourceId]/provenance -- mirrors SourceProvenance (minus source_id/tenant_id/updated_at, which are set by the service). */
|
|
const sourceProvenanceInputSchema = z.object({
|
|
method: z.enum(['official_api', 'bulk_download', 'authorized_web', 'licensed_feed', 'manual_upload']),
|
|
authority: z.string().min(1).optional(),
|
|
compliance_evidence_ref: z.string().min(1).optional(),
|
|
cadence: z.string().min(1).optional(),
|
|
freshness_sla_hours: z.number().nonnegative().optional(),
|
|
});
|
|
|
|
/** POST /api/0pt/sources -- source.register. */
|
|
export const registerSourceSchema = z.object({
|
|
source_id: z.string().min(1),
|
|
tenant_id: z.string().min(1),
|
|
project_id: z.string().min(1),
|
|
source_type: z.enum(['database', 'file', 'web', 'api', 'stream', 'manual']),
|
|
name: z.string().min(1),
|
|
adapter_id: z.string().min(1).optional(),
|
|
access: z.enum(['api', 'bulk', 'authorized_web', 'licensed']).optional(),
|
|
description: z.string().min(1).optional(),
|
|
status: z.enum(['active', 'paused', 'archived', 'deprecated']).optional(),
|
|
correlation_id: z.string().min(1).optional(),
|
|
created_by: z.string().min(1).optional(),
|
|
rights: sourceRightsInputSchema.optional(),
|
|
provenance: sourceProvenanceInputSchema.optional(),
|
|
});
|
|
|
|
/** PUT /api/0pt/sources/[sourceId]/rights -- source.set_rights. */
|
|
export const setSourceRightsSchema = sourceRightsInputSchema.extend({
|
|
tenant_id: z.string().min(1),
|
|
});
|
|
|
|
/** PUT /api/0pt/sources/[sourceId]/provenance -- source.set_provenance. */
|
|
export const setSourceProvenanceSchema = sourceProvenanceInputSchema.extend({
|
|
tenant_id: z.string().min(1),
|
|
});
|
|
|
|
/** POST /api/0pt/sources/[sourceId]/health -- source.health (record a health check). */
|
|
export const recordSourceHealthCheckSchema = z.object({
|
|
tenant_id: z.string().min(1),
|
|
status: z.enum(['up', 'degraded', 'down', 'unknown']),
|
|
checked_at: z.string().min(1).optional(),
|
|
latency_ms: z.number().nonnegative().optional(),
|
|
detail: z.string().min(1).optional(),
|
|
});
|
|
|
|
/** POST /api/0pt/lakehouse/mappings -- mirrors MapDatasetToIcebergTableInput. */
|
|
export const mapDatasetToIcebergTableSchema = z.object({
|
|
tenant_id: z.string().min(1),
|
|
project_id: z.string().min(1),
|
|
dataset_id: z.string().min(1),
|
|
dataset_iceberg_mapping_id: z.string().min(1),
|
|
namespace: z.string().min(1),
|
|
table_name: z.string().min(1),
|
|
storage_uri: z.string().min(1),
|
|
catalog_ref: z.string().min(1),
|
|
table_format: z.string().min(1).optional(),
|
|
actor_id: z.string().min(1),
|
|
correlation_id: z.string().min(1),
|
|
});
|
|
|
|
/** POST /api/0pt/lakehouse/mappings/[mappingId]/writes -- RequestIcebergWriteInput. */
|
|
export const requestIcebergWriteSchema = z.object({
|
|
actor_id: z.string().min(1),
|
|
correlation_id: z.string().min(1),
|
|
snapshot_type: z.string().min(1).optional(),
|
|
schema_version: z.string().min(1),
|
|
manifest_uri: z.string().min(1),
|
|
record_count: z.number().int().nonnegative().optional(),
|
|
size_bytes: z.number().int().nonnegative().optional(),
|
|
heavy: z.boolean().optional(),
|
|
});
|
|
|
|
/** POST /api/0pt/quality/rules -- mirrors CreateQualityRuleInput (src/quality/quality-types.ts). */
|
|
export const createQualityRuleSchema = z.object({
|
|
rule_id: z.string().min(1),
|
|
tenant_id: z.string().min(1),
|
|
project_id: z.string().min(1),
|
|
dataset_id: z.string().min(1),
|
|
name: z.string().min(1),
|
|
expectation: z.object({
|
|
expectation_type: z.string().min(1),
|
|
kwargs: jsonObject,
|
|
}),
|
|
severity: z.enum(['info', 'warning', 'error', 'critical']).optional(),
|
|
supplied_by: z.string().min(1).optional(),
|
|
});
|
|
|
|
/**
|
|
* POST /api/0pt/quality/runs -- runs the currently-registered active rules
|
|
* for a dataset against caller-supplied rows (QualityService.runQuality).
|
|
*/
|
|
export const createQualityRunSchema = z.object({
|
|
tenant_id: z.string().min(1),
|
|
project_id: z.string().min(1),
|
|
dataset_id: z.string().min(1),
|
|
dataset_version: z.number().int().optional(),
|
|
rows: z.array(jsonObject).optional(),
|
|
correlation_id: z.string().min(1).optional(),
|
|
});
|
|
|
|
/**
|
|
* POST /api/0pt/quality/bridge-run -- the Python engine's (services/api/workers/pipeline.py)
|
|
* bridge into the TS QualityService. Auto-provisions a baseline rule set for
|
|
* the (tenant, dataset) pair the first time it is called, so the engine never
|
|
* has to pre-register rules through the CRUD surface just to get an advisory
|
|
* quality read. `rows` is required here (unlike createQualityRunSchema) --
|
|
* the bridge always runs synchronously against real, already-produced rows;
|
|
* it never triggers a delegated/no-rows run.
|
|
*/
|
|
export const qualityBridgeRunSchema = z.object({
|
|
tenant_id: z.string().min(1),
|
|
project_id: z.string().min(1),
|
|
dataset_id: z.string().min(1),
|
|
dataset_version: z.number().int().optional(),
|
|
rows: z.array(jsonObject),
|
|
correlation_id: z.string().min(1).optional(),
|
|
});
|