fix: 17 confirmed bugs from full-repo audit (edge + Python engine) (#5)

* docs: correct stale quality-check-service.ts comment references

Two source comments still cited src/services/quality-check-service.ts as an
existing precedent. It doesn't exist anywhere in this repo's history (git
log --all: zero commits, any branch, ever) -- confirmed during the earlier
full-doc-corpus accuracy pass. Points each comment at the real precedent
instead (document-extraction-service.ts / iceberg-write-service.ts).

Applied from task_05ec36aa, spun off during that pass.

* fix: 17 confirmed bugs from full-repo audit (edge + Python engine)

Found by an 8-finder bug hunt across app/api, src/services, src/clients,
frontend, the Python engine, the data layer, config/infra, and tests, then
independently adversarially verified per finding (32 raw findings -> 27
confirmed real bugs; 5 refuted). This commit applies the 17 confirmed as
safe, mechanical fixes -- no architecture/design decision required. The
other 10 need a decision and are reported separately, not touched here.

Most consequential, app/api/payments/[order_id]/deposit/route.ts:
the client-supplied Origin request header was trusted unsanitized both as
a server-side fetch target (SSRF) and as Stripe Checkout success/cancel
redirect URLs (open redirect tied to a real payment flow), and a failed
order lookup silently fell through to a hardcoded deposit amount instead
of 404. Fixed by reading the order directly via lib/orders-store.ts
(mirroring the sibling release/route.ts, which already did this correctly)
instead of both trusting the Origin header and self-issuing an HTTP fetch.

Also fixed:
- 5 routes (orders, orders/[id], estimates, subscriptions, ai/parse) that
  500'd instead of 400ing on a literal JSON `null` body, since req.json()
  resolves (not rejects) for that input and bypasses the try/catch guard.
- app/api/0pt/lakehouse/mappings/{,[mappingId]/{,writes/}}route.ts each
  instantiated their own private service instances, so a mapping created
  via POST was invisible to the sibling GET and writes routes even before
  the separately-tracked dataset-registration gap. Extracted a shared
  singleton module, src/services/lakehouse-mapping-registry.ts.
- app/api/0pt/documents/[documentId]/extraction/route.ts collapsed every
  error (not-configured, policy-denied, policy-approval-required,
  not-found) into a blanket 400; now status-mapped like the sibling
  sources/[sourceId]/query route.
- app/api/vehicles/route.ts: plate query param wasn't URL-encoded before
  being concatenated into the outbound RDW request, allowing extra query
  parameters to be injected; added a strict allowlist + encodeURIComponent.
- app/api/uploads/route.ts: POST returned a fake 200 success with
  put_url: null even when "configured", instead of continuing to report
  501 until presign generation is actually implemented.
- src/adapters/document-intelligence-adapter.ts: the result-polling loop
  never checked response.ok, so a non-2xx poll response (auth expiry,
  429, 5xx) was silently retried until a generic timeout masked the real
  error.
- app/order/[id]/page.tsx: finish() never cleared the production-simulation
  setInterval, so it kept firing (and re-rendering) forever after delivery.
- app/admin/page.tsx: the metrics fetch had no response/shape validation
  (unlike the adjacent orders fetch), rendering literal "undefined" stat
  tiles on a malformed response instead of falling back to <ErrorState/>.
- services/api/workers/pipeline.py: `config.get("seeds") or DEMO_SEEDS`
  used truthy-or, so an explicit empty seeds dict was silently replaced
  with hardcoded demo org numbers instead of being respected as empty.
- services/api/workers/resolve.py: corroboration (confidence/agreed_sources
  bump) no longer fires for two hits from the same source under fuzzy
  match -- only a genuinely independent source now counts, per docs/06's
  own ">=2 independent sources agree" definition -- and a merged record's
  provenance array now actually includes the corroborating source's
  fields instead of only a bare source-name string.
- services/api/orchestrator/main.py: create_order's int(...) on
  unvalidated client volume/confidence fields raised on any
  decimal-formatted input (int("92.5") -> ValueError); now coerced via
  float() first with a safe default. /vehicles/{plate} and /screen also
  now catch connector exceptions and degrade gracefully, matching
  pipeline.py's existing pattern for the same connectors.
- lib/orders-store.ts: postgres.js (pinned v3.4.9) returns bigint/numeric
  columns as strings by default; getOrder/listOrders/setStatus were
  silently handing callers stringified money fields. Added a narrow
  per-row coercion instead of touching the shared client's global type
  map (which would affect unrelated tables' bigint/numeric columns too).
- lib/subscriptions-store.ts: the in-memory fallback returned
  oldest-first while the Postgres branch explicitly orders newest-first;
  now consistent, mirroring the identical fix already in orders-store.ts.
- app/api/auth/login/route.ts: DEV_LOGIN_KEY was compared with a plain
  !==, a non-constant-time comparison; replaced with crypto.timingSafeEqual.

Verified before commit: tsc --noEmit clean; vitest run 88 passed, 1 todo,
0 failures (13 files, unchanged from before this commit -- no existing
test's assertions needed to change). services/api/workers/resolve.py's
source-novelty and provenance-merge logic additionally verified by direct
execution against synthetic same-source/cross-source inputs (no Python
test suite exists in this repo to add to). services/api/workers/pipeline.py
and services/api/orchestrator/main.py verified with python -m py_compile
(no httpx/fastapi installed in this environment to run them directly).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(engine): treat explicit seeds:null as missing to avoid 500 in run_job

Four-eyes follow-up to this PR's own seeds change. Switching from
`config.get('seeds') or DEMO_SEEDS` to `config['seeds'] if 'seeds' in config`
correctly stopped an explicit empty {} from being backfilled with DEMO_SEEDS,
but regressed on client-controlled `{"seeds": null}`: seeds became None and
the downstream `seeds.get(market, [])` raised AttributeError -> 500. That path
is reachable (POST /orders stores raw body.config; POST /orders/{id}/run passes
it straight to run_job). Using `config.get('seeds') is not None` keeps the
intended behaviour (explicit {} honoured, never silently seeded with demo data)
while restoring the None-tolerance the old idiom had.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-22 18:52:19 +02:00
committed by GitHub
co-authored by Claude Opus 4.8
parent 2902a26bac
commit e6d3101501
24 changed files with 177 additions and 58 deletions
+6 -3
View File
@@ -42,10 +42,13 @@ export default function AdminPage() {
const load = useCallback(async () => {
setError(false);
try {
const [o, m] = await Promise.all([
fetch("/api/orders", { cache: "no-store" }).then((r) => r.json()),
fetch("/api/admin/metrics", { cache: "no-store" }).then((r) => r.json()),
const [oRes, mRes] = await Promise.all([
fetch("/api/orders", { cache: "no-store" }),
fetch("/api/admin/metrics", { cache: "no-store" }),
]);
if (!oRes.ok || !mRes.ok) throw new Error("fetch failed");
const [o, m] = await Promise.all([oRes.json(), mRes.json()]);
if (!m || typeof m.total !== "number") throw new Error("bad metrics shape");
setOrders(mergeOrders(Array.isArray(o) ? o : []) as Order[]);
setMetrics(m);
} catch {
@@ -15,7 +15,17 @@ export async function POST(request: NextRequest, { params }: { params: Promise<{
return NextResponse.json(result, { status: 201 });
} catch (error: unknown) {
const message = error instanceof Error ? error.message : "Unknown error";
return NextResponse.json({ error: message }, { status: 400 });
if (/not configured/i.test(message)) {
return NextResponse.json({ error: message }, { status: 501 });
}
if (/^Policy denied/i.test(message)) {
return NextResponse.json({ error: message }, { status: 403 });
}
if (/^Policy requires approval/i.test(message)) {
return NextResponse.json({ error: message }, { status: 409 });
}
const status = /not found/i.test(message) ? 404 : 400;
return NextResponse.json({ error: message }, { status });
}
}
@@ -1,7 +1,5 @@
import { NextResponse } from "next/server";
import { DatasetIcebergMappingService } from "@/src/services/dataset-iceberg-mapping-service";
const mappingService = new DatasetIcebergMappingService();
import { mappingService } from "@/src/services/lakehouse-mapping-registry";
export async function GET(_request: Request, { params }: { params: Promise<{ mappingId: string }> }) {
try {
@@ -1,7 +1,5 @@
import { NextRequest, NextResponse } from "next/server";
import { IcebergWriteService } from "@/src/services/iceberg-write-service";
const writeService = new IcebergWriteService();
import { writeService } from "@/src/services/lakehouse-mapping-registry";
// Registers a new Iceberg snapshot (a write) through ST0RE's table registry
// for an already-mapped dataset. See docs/ICEBERG_WRITE_BOUNDARY.md.
+7 -9
View File
@@ -1,14 +1,12 @@
import { NextRequest, NextResponse } from "next/server";
import { IcebergWriteService } from "@/src/services/iceberg-write-service";
import { DatasetIcebergMappingService } from "@/src/services/dataset-iceberg-mapping-service";
import { mappingService, writeService } from "@/src/services/lakehouse-mapping-registry";
// Shared instances so a mapping created via POST is visible to a later GET
// within the same server process, matching the in-memory-per-module
// convention already used by app/api/0pt/datasets and app/api/0pt/quality
// (Postgres persistence exists in sql/ but is not wired into route handlers
// in this pass -- same limitation those routes already have).
const mappingService = new DatasetIcebergMappingService();
const writeService = new IcebergWriteService({ mappingService });
// Shared instances (src/services/lakehouse-mapping-registry.ts) so a mapping
// created via POST is visible to a later GET, and to the sibling
// /{mappingId}/writes route, within the same server process -- matching the
// in-memory-per-module convention already used by app/api/0pt/datasets and
// app/api/0pt/quality (Postgres persistence exists in sql/ but is not wired
// into route handlers in this pass -- same limitation those routes already have).
// POST registers a NEW Iceberg table in ST0RE's table registry (via
// IcebergWriteService.mapDatasetToIcebergTable) and records the
@@ -10,8 +10,13 @@
* is not shared across route modules yet — see the "Singleton note" in
* src/services/document-capture-service.ts for prior art on this same repo
* limitation). A source registered via POST /api/0pt/sources will not be
* resolvable here until SourceService is backed by shared persistence
* (sql/002_p0_dataset_quality.sql already defines `opt_sources` for that).
* resolvable here until SourceService is backed by shared persistence.
* Correction: this comment previously claimed `sql/002_p0_dataset_quality.sql`
* already defined an `opt_sources` table for that. That file does not exist
* and never has (git log --all: zero commits, any branch, ever). sql/ today
* has only 001_init.sql and 004_iceberg_write_boundary.sql. The real fix —
* backing SourceService with real persistence — is still open; no migration
* for it exists yet.
* Details: docs/P0_QUERY_SOURCE_FLOW.md.
*/
import { NextRequest, NextResponse } from "next/server";
+3 -1
View File
@@ -24,7 +24,9 @@ const DEST: [RegExp, string][] = [
];
export async function POST(req: NextRequest) {
const { text } = await req.json().catch(() => ({ text: "" }));
const parsed = await req.json().catch(() => ({ text: "" }));
const body = parsed && typeof parsed === "object" ? parsed : { text: "" };
const { text } = body as { text?: string };
if (!text) return NextResponse.json({ error: { code: "bad_request", message: "text krävs" } }, { status: 400 });
const eng = await engineFetch("/ai/parse", { method: "POST", body: JSON.stringify({ text }) });
+8 -1
View File
@@ -1,3 +1,4 @@
import { timingSafeEqual } from "node:crypto";
import { NextRequest, NextResponse } from "next/server";
import { signSession, SESSION_COOKIE } from "@/lib/auth";
@@ -9,7 +10,13 @@ export async function POST(req: NextRequest) {
return NextResponse.json({ error: { code: "auth_not_configured", message: "AUTH_SECRET saknas" } }, { status: 501 });
}
const devKey = process.env.DEV_LOGIN_KEY;
if (!devKey || key !== devKey) {
// Constant-time compare: a plain !== leaks the number of matching leading
// bytes via response timing, letting an attacker recover DEV_LOGIN_KEY
// byte-by-byte instead of it being an all-or-nothing guess.
const keyBuf = Buffer.from(typeof key === "string" ? key : "");
const devKeyBuf = Buffer.from(devKey ?? "");
const keysMatch = Boolean(devKey) && keyBuf.length === devKeyBuf.length && timingSafeEqual(keyBuf, devKeyBuf);
if (!keysMatch) {
return NextResponse.json({ error: { code: "forbidden", message: "Dev-login kräver giltig nyckel" } }, { status: 403 });
}
if (!email) {
+3
View File
@@ -11,6 +11,9 @@ export async function POST(req: NextRequest) {
} catch {
return NextResponse.json({ error: { code: "bad_json", message: "Invalid JSON body" } }, { status: 400 });
}
if (!body || typeof body !== "object") {
return NextResponse.json({ error: { code: "bad_json", message: "Invalid JSON body" } }, { status: 400 });
}
// Defer to the FastAPI data engine when it's running; fall back to local math.
const eng = await engineFetch("/estimates", { method: "POST", body: JSON.stringify(body) });
+3
View File
@@ -19,6 +19,9 @@ export async function PATCH(req: NextRequest, ctx: { params: Promise<{ id: strin
} catch {
return NextResponse.json({ error: { code: "bad_json", message: "Invalid JSON" } }, { status: 400 });
}
if (!body || typeof body !== "object") {
return NextResponse.json({ error: { code: "bad_json", message: "Invalid JSON body" } }, { status: 400 });
}
const allowed = ["draft", "quoted", "deposit_pending", "in_production", "review", "revision", "accepted", "delivered"];
if (!body.status || !allowed.includes(body.status)) {
return NextResponse.json({ error: { code: "bad_status", message: "Ogiltig status" } }, { status: 400 });
+3
View File
@@ -9,6 +9,9 @@ export async function POST(req: NextRequest) {
} catch {
return NextResponse.json({ error: { code: "bad_json", message: "Invalid JSON" } }, { status: 400 });
}
if (!body || typeof body !== "object") {
return NextResponse.json({ error: { code: "bad_json", message: "Invalid JSON body" } }, { status: 400 });
}
const flow = body.flow || "custom";
const config = body.config || {};
const order = await createOrder(flow, config);
+9 -12
View File
@@ -1,4 +1,5 @@
import { NextRequest, NextResponse } from "next/server";
import { getOrder } from "@/lib/orders-store";
// POST /api/payments/[order_id]/deposit
// Returns { url } for Stripe Checkout. Returns 501 when STRIPE_SECRET_KEY is absent.
@@ -13,19 +14,15 @@ export async function POST(
return NextResponse.json({ error: "payment_not_configured" }, { status: 501 });
}
const origin =
request.headers.get("origin") ??
process.env.NEXT_PUBLIC_APP_URL ??
"http://localhost:3000";
// Never derive from the client-supplied Origin header (SSRF + open-redirect vector) —
// request.nextUrl.origin reflects the actual request URL, not attacker-controlled input.
const origin = request.nextUrl.origin;
let depositOre = 44750;
try {
const orderRes = await fetch(`${origin}/api/orders/${order_id}`);
if (orderRes.ok) {
const order = (await orderRes.json()) as { deposit_ore?: number; total_ore?: number };
depositOre = order.deposit_ore ?? Math.round((order.total_ore ?? 89500) / 2);
}
} catch {}
const order = await getOrder(order_id);
if (!order) {
return NextResponse.json({ error: { code: "not_found", message: "Order saknas" } }, { status: 404 });
}
const depositOre = order.deposit_ore ?? Math.round((order.total_ore ?? 89500) / 2);
const body = new URLSearchParams({
"payment_method_types[0]": "card",
+2 -1
View File
@@ -7,7 +7,8 @@ export async function GET() {
}
export async function POST(req: NextRequest) {
const body = await req.json().catch(() => ({}));
const parsed = await req.json().catch(() => ({}));
const body = parsed && typeof parsed === "object" ? parsed : {};
if (!body.order_id) {
return NextResponse.json({ error: { code: "bad_request", message: "order_id krävs" } }, { status: 400 });
}
+11 -6
View File
@@ -1,4 +1,4 @@
import { NextRequest, NextResponse } from "next/server";
import { NextResponse } from "next/server";
// GET /api/uploads — list uploaded files. Returns 501 until storage is configured.
export async function GET() {
@@ -10,9 +10,10 @@ export async function GET() {
}
// POST /api/uploads (docs/04) — issue a presigned R2 PUT URL. Env-guarded:
// returns 501 until R2 storage is configured. Client then PUTs the file directly.
export async function POST(req: NextRequest) {
const body = await req.json().catch(() => ({}));
// returns 501 until R2 storage is configured, and until presign generation
// is implemented — a configured-but-unimplemented state must not report
// success. Client then PUTs the file directly.
export async function POST() {
const haveR2 = process.env.R2_ACCESS_KEY_ID && process.env.R2_BUCKET && process.env.R2_ENDPOINT;
if (!haveR2) {
return NextResponse.json(
@@ -20,6 +21,10 @@ export async function POST(req: NextRequest) {
{ status: 501 },
);
}
// With R2 configured: build a presigned PUT (S3-compatible) and return it.
return NextResponse.json({ file_id: crypto.randomUUID(), put_url: null, name: body?.name ?? null });
// TODO: presigned PUT generation not implemented yet — keep returning 501
// instead of a fake 200/null success shape until real R2 presign wiring exists.
return NextResponse.json(
{ error: { code: "storage_not_configured", message: "Presignerad uppladdning är inte implementerad än" } },
{ status: 501 },
);
}
+2 -2
View File
@@ -4,14 +4,14 @@ import { engineFetch } from "@/lib/engine";
// GET /api/vehicles?plate=XX-99-YY — RDW (NL) open vehicle data (technical/fleet
// only, no owner PII). Defers to the engine when running; else calls RDW directly.
export async function GET(req: NextRequest) {
const plate = (req.nextUrl.searchParams.get("plate") ?? "").toUpperCase().replace(/[-\s]/g, "");
const plate = (req.nextUrl.searchParams.get("plate") ?? "").toUpperCase().replace(/[-\s]/g, "").replace(/[^A-Z0-9]/g, "");
if (!plate) return NextResponse.json({ error: { code: "bad_request", message: "plate krävs" } }, { status: 400 });
const eng = await engineFetch(`/vehicles/${plate}`);
if (eng && eng.ok) return NextResponse.json(await eng.json());
try {
const r = await fetch(`https://opendata.rdw.nl/resource/m9d7-ebf2.json?kenteken=${plate}`, {
const r = await fetch(`https://opendata.rdw.nl/resource/m9d7-ebf2.json?kenteken=${encodeURIComponent(plate)}`, {
headers: { "User-Agent": "0pt/0.1 (+https://0pt.app; SIAX)" },
cache: "no-store",
});
+4
View File
@@ -65,6 +65,10 @@ export default function ProductionPage() {
}, [id]);
const finish = () => {
if (timer.current) {
clearInterval(timer.current);
timer.current = null;
}
setProgress([100, 100, 100, 100, 100]);
setMarkets([100, 100, 100, 100]);
setConf(94);
+19 -3
View File
@@ -30,6 +30,21 @@ export interface Order {
// Non-durable fallback used only until a database is provisioned.
const mem = new Map<string, Order>();
// postgres.js (pinned v3.4.9) returns bigint (total_ore/deposit_ore/balance_ore)
// and numeric (est_confidence) columns as strings by default, not JS numbers.
// Coerce only these four fields on rows read back from SQL -- not the shared
// `sql` client's global type map, which would silently affect every other
// table's bigint/numeric columns too (e.g. jobs.pct, deliveries.avg_confidence).
function coerceOrder(row: Order): Order {
return {
...row,
total_ore: Number(row.total_ore),
deposit_ore: Number(row.deposit_ore),
balance_ore: Number(row.balance_ore),
est_confidence: Number(row.est_confidence),
};
}
function estimateFor(config: OrderConfig) {
return computeEstimate({
markets: config.markets ?? [],
@@ -74,7 +89,7 @@ export async function getOrder(id: string): Promise<Order | null> {
const rows = await sql<Order[]>`
SELECT order_id, flow, status, config, total_ore, deposit_ore, balance_ore, est_days, est_confidence, created_at
FROM orders WHERE order_id::text = ${id} LIMIT 1`;
return rows[0] ?? null;
return rows[0] ? coerceOrder(rows[0]) : null;
}
return mem.get(id) ?? null;
}
@@ -82,9 +97,10 @@ export async function getOrder(id: string): Promise<Order | null> {
export async function listOrders(): Promise<Order[]> {
const sql = getSql();
if (sql) {
return sql<Order[]>`
const rows = await sql<Order[]>`
SELECT order_id, flow, status, config, total_ore, deposit_ore, balance_ore, est_days, est_confidence, created_at
FROM orders ORDER BY created_at DESC LIMIT 100`;
return rows.map(coerceOrder);
}
return [...mem.values()].sort((a, b) => b.created_at.localeCompare(a.created_at));
}
@@ -95,7 +111,7 @@ export async function setStatus(id: string, status: string): Promise<Order | nul
const rows = await sql<Order[]>`
UPDATE orders SET status = ${status}, updated_at = now() WHERE order_id::text = ${id}
RETURNING order_id, flow, status, config, total_ore, deposit_ore, balance_ore, est_days, est_confidence, created_at`;
return rows[0] ?? null;
return rows[0] ? coerceOrder(rows[0]) : null;
}
const o = mem.get(id);
if (o) { o.status = status; mem.set(id, o); }
+1 -1
View File
@@ -32,5 +32,5 @@ export async function listSubscriptions(): Promise<Subscription[]> {
if (sql) {
return sql<Subscription[]>`SELECT sub_id, order_id, cadence, webhook_url, status, next_run_at, created_at FROM subscriptions ORDER BY created_at DESC LIMIT 100`;
}
return [...mem.values()];
return [...mem.values()].sort((a, b) => b.created_at.localeCompare(a.created_at));
}
+20 -4
View File
@@ -26,6 +26,16 @@ app.add_middleware(
_SOURCES = yaml.safe_load((Path(__file__).resolve().parent.parent / "sources" / "sources.yaml").read_text(encoding="utf-8"))["sources"]
def _coerce_int(v: Any, default: int) -> int:
"""int(v) raises on decimal-formatted strings/floats; coerce via float() first
and fall back to `default` on any unparseable, client-controlled input instead
of a 500."""
try:
return int(float(v))
except (TypeError, ValueError):
return default
# ── estimate model (mirrors apps/web lib/data.computeEstimate) ───────────────
FIELD_RATE = 240
ENTITY_MULT = {"Bolag": 1.0, "Beslutsfattare": 1.4, "Fordon": 1.2}
@@ -90,13 +100,19 @@ async def sources_for(market: str):
@app.get("/vehicles/{plate}")
async def vehicle(plate: str):
v = await rdw_vehicle(plate)
try:
v = await rdw_vehicle(plate)
except Exception:
v = None
return v or {"vehicle": None}
@app.get("/screen")
async def screen(q: str):
return await opensanctions_screen(q)
try:
return await opensanctions_screen(q)
except Exception:
return {"matches": [], "source": "opensanctions"}
@app.post("/estimates")
@@ -111,9 +127,9 @@ async def create_order(body: OrderIn):
entity_type=body.config.get("type", "Bolag"),
fields=body.config.get("fieldIds", []),
enrichment=body.config.get("enrichmentIds", []),
volume=int(body.config.get("volume", 1000)),
volume=_coerce_int(body.config.get("volume", 1000), 1000),
frequency=body.config.get("frequency", "Engångs"),
confidence_target=int(body.config.get("confidence", 92)),
confidence_target=_coerce_int(body.config.get("confidence", 92), 92),
))
oid = str(uuid.uuid4())
_ORDERS[oid] = {"order_id": oid, "flow": body.flow, "status": "draft", "config": body.config, **est}
+6 -1
View File
@@ -43,7 +43,12 @@ async def _fetch(source_id: str, key: str) -> CompanyResult | None:
async def run_job(order_id: str, config: dict[str, Any]) -> dict[str, Any]:
markets: list[str] = config.get("markets", []) or ["NO"]
threshold = float(config.get("confidence", 92)) / 100.0
seeds: dict[str, list[str]] = config.get("seeds") or DEMO_SEEDS
# Use caller-supplied seeds when present (an explicit empty {} is honoured so a
# real job is never silently backfilled with DEMO_SEEDS), but treat an explicit
# null the same as "missing" -- `config.get("seeds") is not None` avoids a 500
# (None.get) on client-controlled `{"seeds": null}`, which the old `... or ...`
# idiom tolerated and the `"seeds" in config` form regressed.
seeds: dict[str, list[str]] = config["seeds"] if config.get("seeds") is not None else DEMO_SEEDS
records: list[dict[str, Any]] = []
per_market: dict[str, int] = {}
+22 -4
View File
@@ -24,10 +24,28 @@ def resolve(records: list[dict[str, Any]], threshold: int = 92) -> list[dict[str
match = o
break
if match:
# corroboration: independent source agreeing bumps confidence + level
match["confidence"] = min(0.99, match.get("confidence", 0.0) + 0.1)
match["agreed_sources"] = match.get("agreed_sources", 1) + 1
match.setdefault("merged_from", []).append(r.get("source"))
# Merge the losing record's per-field provenance regardless of source
# novelty below, so the surviving record's provenance always reflects
# every contributing record's fields.
existing_provenance = match.setdefault("provenance", [])
seen_provenance = {(p.get("field_name"), p.get("source_id")) for p in existing_provenance}
for p in r.get("provenance", []):
key = (p.get("field_name"), p.get("source_id"))
if key not in seen_provenance:
existing_provenance.append(p)
seen_provenance.add(key)
# Corroboration (confidence/agreed_sources bump) requires an
# *independent* source — two hits from the same connector must not
# count as two agreeing sources.
seen_sources = match.setdefault("_sources_seen", {match.get("source")} - {None})
src = r.get("source")
if src is not None and src not in seen_sources:
match["confidence"] = min(0.99, match.get("confidence", 0.0) + 0.1)
match["agreed_sources"] = match.get("agreed_sources", 1) + 1
seen_sources.add(src)
else:
out.append({**r, "agreed_sources": 1})
out.append({**r, "agreed_sources": 1, "_sources_seen": {r.get("source")} - {None}})
for o in out:
o.pop("_sources_seen", None)
return out
@@ -162,6 +162,10 @@ export class DocumentIntelligenceAdapter {
for (let i = 0; i < maxPollAttempts; i++) {
await sleep(pollIntervalMs);
const pollRes = await fetch(operationLocation, { headers: { "Ocp-Apim-Subscription-Key": this.config.apiKey } });
if (!pollRes.ok) {
const text = await pollRes.text().catch(() => "");
throw new Error(`Document Intelligence poll request failed: ${pollRes.status} ${text.slice(0, 300)}`);
}
const json = (await pollRes.json()) as Record<string, unknown>;
if (json.status === "succeeded") {
payload = json;
+5 -1
View File
@@ -18,7 +18,11 @@
* `writeExternalRecord` runs, a human has already approved the write -- this
* service still re-checks CL0UD policy (per docs/PRODUCT_PLAN.md section
* 4.1's hard rule that CL0UD is always re-consulted, the same rule
* quality-check-service.ts/document-extraction-service.ts already follow),
* document-extraction-service.ts already follows -- confirmed by grepping
* this repo's src/services/ for evaluatePolicy()/Cl0udClient usage; the
* only other real service that does the same is iceberg-write-service.ts.
* An earlier version of this comment also cited `quality-check-service.ts`
* here; that file does not exist anywhere in this repo),
* but does not itself re-verify the upstream Approval record (ACT0 owns
* approvals; 0pt's own capability boundary is "did CL0UD say yes for THIS
* call", same separation of concerns every other 0pt service in this repo
@@ -0,0 +1,19 @@
import { DatasetService } from "./dataset-service";
import { DatasetIcebergMappingService } from "./dataset-iceberg-mapping-service";
import { IcebergWriteService } from "./iceberg-write-service";
/**
* Shared singleton instances for the lakehouse-mapping route group
* (app/api/0pt/lakehouse/mappings/*). Previously each route file in that
* group instantiated its own DatasetService/DatasetIcebergMappingService/
* IcebergWriteService, so a mapping created via POST /mappings was invisible
* to GET /mappings/{id} and to POST /mappings/{id}/writes within the same
* server process (each held a distinct in-memory Map). Constructing all
* three here once, wired together, and importing them everywhere fixes that
* -- matching the in-memory-per-module convention this repo already uses
* elsewhere (see document-capture-service.ts's own singleton note), just
* applied consistently across this route group instead of per-file.
*/
export const datasetService = new DatasetService();
export const mappingService = new DatasetIcebergMappingService({ datasetService });
export const writeService = new IcebergWriteService({ datasetService, mappingService });