feat(agent): telemetry + CI-gate quick wins from the "AI systems that ship" audit (#677)
* feat(agent): telemetry completeness + durability, CI gates, commit_method provenance Quick wins from the "Building AI systems that ship" audit: - mcp.tool_called gains errorMessage (message_sv, truncated 500 chars) on all failure exits; new mcp.skill_loaded event on every gnubok_load_skill (all tiers) so atom usage is finally measurable - event_log: (event_type, created_at) index; cleanup cron keeps mcp.*/agent.* telemetry 180 days (delivery events stay 30) - CI: lint ratchet (npm run check:lint — 60 legacy errors baselined, fails only on NEW errors) and a pg-real coverage gate (migrations touching trigger/RPC/RLS/DEFERRABLE require a *.pg.test.ts change; escape hatch: -- pg-test: covered-by/skip) - journal_entries.commit_method CHECK widened with 'api_key'/'agent'; the MCP approve path records 'api_key' truthfully instead of 'user_accept' (agent_first_vision §8 P0-1). 'agent' is reserved — ALL MCP traffic (incl. claude.ai OAuth, whose access_token is a minted API key) authenticates as api_key today Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(import): derive opening balances from prior-year #UB when SIE lacks #IB (#675) SIE files exported without #IB 0 rows (only #UB -1) previously imported with zero opening balances. getEffectiveOpeningBalances() now derives IB from prior-year UB for balance-sheet accounts when explicit #IB is absent, surfaces the derivation as an info issue in the import preview, and excludes share-capital vouchers from opening-balance detection. Detection regexes are shared between parser and importer so the two checks cannot drift. 507 lib/import tests pass. (Authored in a parallel session in this checkout; included per request.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(review): address PR #677 bot findings — RoPA entry, execFileSync, gate scope note Triage of the compliance-swarm + Greptile findings: Applied: - .compliance/ropa.yaml: new mcp.telemetry processing activity declaring the 180-day mcp.*/agent.* retention, lawful basis, data categories, and the no-args/no-results minimisation (ISO A.8.10, GDPR Art.5(1)(c) — the retention split is now formally documented, referenced from the cron) - check-pg-test-coverage.mjs: execFileSync with argv array — no shell, so a hostile base-ref can't inject (ASVS V13.2.1); verified an injection attempt exits 2 without executing - check-pg-test-coverage.mjs: documented the PR-level (not per-migration) scope of the gate so reviewers know to check coverage per migration when a PR carries several risky migrations (Greptile P2) Acknowledged, no change: - errorMessage PII risk: messages are domain-mapped strings; event_log already persists far richer delivery payloads under the same RLS; now declared in ropa.yaml - cron error envelope: errorResponse maps to the canonical safe envelope and the endpoint is CRON_SECRET-gated - two-pass delete "partial state": TTL deletes are idempotent — the next daily run sweeps whatever a failed pass left behind - skill_loaded actorLabel/sessionId: mirrors the pre-existing mcp.tool_called payload; sessionId is the join key the analytics exist for Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jakob Wennberg
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
076bb169f8
commit
bc61862e76
@@ -36,7 +36,8 @@ const PERSISTED_EVENT_TYPES: CoreEventType[] = [
|
||||
'supplier_invoice.match_confirmed',
|
||||
'supplier_invoice.confirmed',
|
||||
// MCP telemetry — every tool invocation, tools/list call, and resources/read.
|
||||
// Lightweight metadata only; 30-day TTL on event_log bounds the volume.
|
||||
// Lightweight metadata only. mcp.*/agent.* rows are retained 180 days by the
|
||||
// cleanup cron (error-rate trends need more than the 30-day delivery window).
|
||||
'mcp.tool_called',
|
||||
'mcp.tools_list_called',
|
||||
'mcp.resource_read',
|
||||
@@ -46,6 +47,10 @@ const PERSISTED_EVENT_TYPES: CoreEventType[] = [
|
||||
'mcp.workflow_started',
|
||||
'mcp.workflow_completed',
|
||||
'mcp.next_hint_followed',
|
||||
// Every successful gnubok_load_skill, all tiers — which atoms agents
|
||||
// actually load. Joined against mcp.tool_called error rates to measure
|
||||
// whether a loaded atom helps or hurts.
|
||||
'mcp.skill_loaded',
|
||||
// Agent self-reported feedback — surfaces "this tool was missing", "this
|
||||
// description was wrong", etc. Quarterly review → roadmap.
|
||||
'agent.feedback',
|
||||
|
||||
+20
-1
@@ -138,7 +138,8 @@ export type CoreEvent =
|
||||
| { type: 'company.deleted'; payload: { companyId: string; userId: string; archivedAt: string } }
|
||||
| { type: 'account.deleted'; payload: { userId: string; deletedAt: string } }
|
||||
// MCP telemetry — fired from the MCP dispatcher.
|
||||
// Persisted to event_log (30-day TTL) for hot-tool / error-rate / latency analytics.
|
||||
// Persisted to event_log (180-day TTL for mcp.*/agent.* rows, vs 30 days for
|
||||
// delivery events) for hot-tool / error-rate / latency analytics.
|
||||
// Intentionally lightweight: no args, no result body — only metadata.
|
||||
| { type: 'mcp.tool_called'; payload: {
|
||||
tool: string // e.g. 'gnubok_create_invoice'
|
||||
@@ -151,6 +152,9 @@ export type CoreEvent =
|
||||
isError: boolean // matches the JSON-RPC tool-result isError flag returned to the client
|
||||
errorCode: string | null // structured error code from tool-result.toToolError when applicable
|
||||
errorKind: 'execution' | 'scope_denied' | 'unknown_tool' | null
|
||||
errorMessage: string | null // human-readable error message (truncated to 500 chars), null on success.
|
||||
// Raw material for clustering real agent failures into curated gotchas —
|
||||
// errorCode alone can't distinguish "period locked" from "unbalanced".
|
||||
requestId: string | number | null // JSON-RPC request id (helps correlate with client-side logs)
|
||||
userId: string
|
||||
companyId: string
|
||||
@@ -212,6 +216,21 @@ export type CoreEvent =
|
||||
userId: string
|
||||
companyId: string
|
||||
}}
|
||||
// Fires on EVERY successful gnubok_load_skill — all tiers, unlike
|
||||
// mcp.workflow_started which fires only for workflow-tier skills. Records
|
||||
// WHICH skill/atom bodies agents actually pull, the denominator needed to
|
||||
// correlate a loaded atom with downstream tool-error rates (a skill can
|
||||
// make the model worse — measure, don't assume).
|
||||
| { type: 'mcp.skill_loaded'; payload: {
|
||||
slug: string // e.g. 'modifier/holding-ab', 'month-end-close'
|
||||
tier: 'workflow' | 'horizontal' | 'vertical' | 'modifier'
|
||||
sessionId: string | null
|
||||
actorType: 'user' | 'api_key' | 'mcp_oauth' | 'cron'
|
||||
actorId: string | null
|
||||
actorLabel: string | null
|
||||
userId: string
|
||||
companyId: string
|
||||
}}
|
||||
// Fires when the agent's next tool call matches the previous response's
|
||||
// nextHint.tool — measures whether `next` hints are actually followed.
|
||||
// Computed dispatcher-side by comparing the last response shape to the
|
||||
|
||||
Reference in New Issue
Block a user