feat(mcp): worked examples on five high-traffic tools, and in the error that rejects a call (#2100)

#2066 asked for input_examples on the top ~20 tools by call volume. The
binding constraint turned out to be budget, not writing: after #2089
reclaimed 3 763 tokens, its own policy required ratcheting the tools/list
ceiling down with it, so the real headroom was 317 tokens. Ten examples
across five tools cost 199, leaving ~118. The ceiling is not raised.

Tools picked from 30 days of mcp.tool_called crossed with the
combinations the descriptions already warn about and callers still get
wrong: account_override without an explicit vat_treatment (books gross,
no moms line), representation without deltagare and syfte, confirmed on
a high-risk approval, a balanced voucher where the moms leg is its own
line, and get_kpi_report, where one caller sent `metric` 604 times over
seven days to a tool whose only parameter is period_id.

Examples are also surfaced in the unknown-parameter error. That costs
nothing in tools/list, because it only ships on the response to a call
that already failed, and it reaches the caller that most needs it: a key
list told DueCue's agent which parameter was wrong but not what a
correct call looks like, and the same rejected call repeated for a week.

Every example is validated against its own schema by the same
findUnknownArgKeys guard the server runs, plus required/type/enum/pattern
checks. An example our own boundary would reject is worse than none: it
teaches the exact mistake the guard then punishes. That test caught three
invented enum values in this change's own first draft.


Claude-Session: https://claude.ai/code/session_01L3P2hr19PhQuCoTSGoegcY

Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Jakob Wennberg
2026-09-01 20:18:19 +02:00
committed by GitHub
parent e5fba9471e
commit 169e7eaf4e
5 changed files with 231 additions and 2 deletions
+1
View File
@@ -1423,6 +1423,7 @@ One line per decision: `[YYYY-MM-DD] <decision>: <why>`. Appended by agents and
[2026-09-01] multi_user seat gate enforced app-side via a NEW gated RPC (resolve_active_company_gated) instead of editing resolve_active_company/current_active_company_id: the zero-arg RPC and the RLS twin also run on self-hosted DBs where the paywall must never bite, and the app picks the gated overload only when isMultiUserEnforced(). RLS convergence rides the existing used_fallback write-back. company_capability_config deliberately does not apply to multi_user (no expiry to hang the 20-day grace on).
[2026-09-01] EU reverse-charge packs book directly on 4515/4535 instead of adding a 45xx D / 4598 K basbelopp pair (Anders' literal suggestion): same ruta 20/21 outcome, standard BAS practice for a template that owns the cost account anyway, and a 3-business-line pack would return null from convertLibraryToBookingTemplate and silently vanish from the transaction picker. The 4598 motkonto pattern remains the right tool only where the user's own cost account must be preserved (engine-generated bookings, supplier invoices).
[2026-09-01] Floating supplier-invoice underlag gets a standing daily reanchor cron (/api/documents/reanchor/cron) instead of another one-off repair migration: prod case 2026-08-28 (kontantmetod payment verifikat, doc eligible on every static condition, inline anchor silently did nothing, no log line recorded why) is the second time a hand-written sweep (20260727180000, 20260824150000) was needed; the inline anchor is best-effort by design, so the retry belongs in infrastructure. anchorSupplierInvoiceDocument also stops claiming success on a zero-row guarded update and logs its silent bail branches.
[2026-09-01] MCP worked examples ship in the inputSchema for five tools only, not the top twenty (#2066): the tools/list ceiling is the binding constraint, not the writing. After #2089 reclaimed 3 763 tokens the policy in payload-size.bench.test.ts required ratcheting the ceiling down with it, so real headroom was 317 tokens, not the 4 300 the #2089 PR body implied. Ten examples cost 199, leaving ~118. Tools were picked by 30 days of mcp.tool_called crossed with the combinations the descriptions already warn about (account_override without vat_treatment, representation without deltagare, confirmed on a high-risk approval, and get_kpi_report where a caller sent `metric` 604 times to a tool whose only parameter is period_id). Examples are also surfaced in the unknown-parameter error, which costs zero catalog budget and reaches the caller that already failed; a test validates every example against its own schema with the same guard the server runs, because an example our boundary would reject teaches exactly the mistake it then punishes.
[2026-09-01] SKV connector instance wiring (PR6b-2): system (CCG/ombud) auth is deliberately NOT brokered; it stays on the direct path and fails SYSTEM_AUTH_FAILED on a credential-less self-host, because the org certificate and ombud grants are hosted-only. getSkatteverketEnvironment() hard-reports 'prod' in connector mode: the upstream env is resolved from HOSTED's config, and the instance's unset base URLs would otherwise show a Testmiljo badge on real filings (reporting hosted's actual env via /api/connector/status is a #2090 follow-up). buildAuthorizeUrl stays direct-only rather than going async: the connector authorize needs to return the broker's redirect_uri + connector_state for persistence, so index.ts branches to startConnectorAuthorization instead of overloading one function. Connector-layer 4xx bodies (code CONNECTOR_*) are classified BEFORE the SKV-shaped 401/403 sniffing so a broker refusal never tells a self-host operator to check SKATTEVERKET_APIGW_CLIENT_ID; broker refresh 404 CONNECTOR_NOT_OWNED maps to SESSION_EXPIRED (terminal, reconnect fixes) while broker 502 stays a raw error (transient SKV outage must not re-arm the reconnect banner, #1155).
[2026-09-01] multi_user skeptic fixes: Stripe cancel EXPIRES the multi_user stripe grant instead of deleting it (grace anchor; other grants still deleted per freeze-and-retain); app-side state checks go RPC-first via SECURITY DEFINER company_multi_user_state (capability_grants RLS hides team rows from non-team users, byrå clients would misread as frozen); byra-kind teams get a standing team-scoped multi_user grant via backfill + teams trigger (WL-10 assumption made real; partner billing is out-of-band); PGRST202 on resolution fails OPEN (pre-migration DB has no multi_user rows: gated fallback would freeze all non-owners); /api/v1 got the same dormancy gate as MCP. RLS-level enforcement and the mid-session API fallback write-back window stay v2 follow-ups (documented, same class as pre-existing stale-preference fallback).
[2026-09-01] Declined CodeRabbit's UpgradeNote suggestion (PR #1758 follow-up) to append the self-host connector sentence to children instead of replacing them: every caller's children is hosted subscription copy ("... kräver ett abonnemang"), so appending would show subscription wording on a self-host, the exact thing the branch exists to avoid; the "CSV/SIE import stays free" text it cited is a code comment in BankSyncNowButton, not children. Replace-on-self-host stays; a dedicated selfHosted children prop can come when a caller actually needs per-panel reassurance there.
@@ -0,0 +1,157 @@
import { describe, expect, it } from 'vitest'
import { tools, isDefaultCatalogTool } from '../server'
import { findUnknownArgKeys, shortestExampleFor } from '../arg-guard'
/**
* Worked `examples` on the tool inputSchema (#2066).
*
* An example is a call an agent will copy. If our own boundary would reject
* it, the example is worse than none: it teaches the exact mistake the guard
* then punishes. So every example is checked against the schema it ships with,
* with the same unknown-key rule the server enforces at runtime.
*/
interface JsonSchema {
type?: string
properties?: Record<string, JsonSchema>
required?: string[]
enum?: unknown[]
items?: JsonSchema
additionalProperties?: unknown
pattern?: string
examples?: unknown[]
}
const withExamples = tools
.map((t) => ({ name: t.name, schema: t.inputSchema as JsonSchema }))
.filter((t) => Array.isArray(t.schema.examples))
/** Placeholder ids are deliberately short ("3f1a..."): they must not read as real UUIDs. */
const PLACEHOLDER = /^[0-9a-f]{4}\.\.\.$/
function typeOf(value: unknown): string {
if (Array.isArray(value)) return 'array'
if (value === null) return 'null'
return typeof value
}
function checkValue(path: string, value: unknown, schema: JsonSchema, problems: string[]): void {
if (schema.type && schema.type !== typeOf(value)) {
// A placeholder id stands in for a UUID string; still a string.
problems.push(`${path}: expected ${schema.type}, got ${typeOf(value)}`)
return
}
if (schema.enum && !schema.enum.includes(value)) {
problems.push(`${path}: ${JSON.stringify(value)} is not in the declared enum`)
}
if (schema.pattern && typeof value === 'string' && !new RegExp(schema.pattern).test(value)) {
problems.push(`${path}: ${JSON.stringify(value)} does not match ${schema.pattern}`)
}
if (schema.type === 'array' && Array.isArray(value) && schema.items) {
value.forEach((item, i) => checkValue(`${path}[${i}]`, item, schema.items!, problems))
}
if (schema.type === 'object' && schema.properties && value && typeof value === 'object') {
checkObject(path, value as Record<string, unknown>, schema, problems)
}
}
function checkObject(path: string, value: Record<string, unknown>, schema: JsonSchema, problems: string[]): void {
for (const req of schema.required ?? []) {
if (!(req in value)) problems.push(`${path}: missing required property "${req}"`)
}
for (const [key, val] of Object.entries(value)) {
const propSchema = schema.properties?.[key]
if (!propSchema) continue
checkValue(`${path}.${key}`, val, propSchema, problems)
}
}
describe('inputSchema examples are calls the server would accept', () => {
it('ships examples on at least the tools this change targeted', () => {
// Pinned so a rename or a schema rewrite cannot silently drop them.
expect(withExamples.map((t) => t.name).sort()).toEqual([
'gnubok_approve_pending_operation',
'gnubok_categorize_transaction',
'gnubok_create_voucher',
'gnubok_get_kpi_report',
'gnubok_query_journal',
])
})
it('every example survives the unknown-parameter guard that rejects real calls', () => {
const offenders: string[] = []
for (const { name, schema } of withExamples) {
for (const [i, example] of (schema.examples ?? []).entries()) {
const unknown = findUnknownArgKeys(schema as Record<string, unknown>, example as Record<string, unknown>)
if (unknown.length > 0) offenders.push(`${name}[${i}]: ${unknown.join(', ')}`)
}
}
expect(offenders).toEqual([])
})
it('every example satisfies required properties, declared types, enums and patterns', () => {
const problems: string[] = []
for (const { name, schema } of withExamples) {
for (const [i, example] of (schema.examples ?? []).entries()) {
checkObject(`${name}[${i}]`, example as Record<string, unknown>, schema, problems)
}
}
expect(problems).toEqual([])
})
it('examples live only on tools the default catalog actually publishes', () => {
// An example on a search-only tool is budget spent where no agent reads it.
const hidden = withExamples.filter(({ name }) => {
const tool = tools.find((t) => t.name === name)!
return !isDefaultCatalogTool(tool)
})
expect(hidden.map((t) => t.name)).toEqual([])
})
it('uses obvious placeholders for ids, never invented UUIDs', () => {
// A real-looking UUID in an example gets copied verbatim and 404s; worse,
// it could name a row in some other tenant.
const suspicious: string[] = []
const walk = (label: string, value: unknown) => {
if (typeof value === 'string' && /^[0-9a-f]{8}-[0-9a-f]{4}/.test(value)) suspicious.push(`${label}=${value}`)
else if (Array.isArray(value)) value.forEach((v, i) => walk(`${label}[${i}]`, v))
else if (value && typeof value === 'object') {
for (const [k, v] of Object.entries(value)) walk(`${label}.${k}`, v)
}
}
for (const { name, schema } of withExamples) {
for (const [i, example] of (schema.examples ?? []).entries()) walk(`${name}[${i}]`, example)
}
expect(suspicious).toEqual([])
// And the placeholders we do use are recognisable as placeholders.
const ids = (withExamples.find((t) => t.name === 'gnubok_categorize_transaction')!.schema.examples ?? [])
.map((e) => (e as Record<string, string>).transaction_id)
expect(ids.every((id) => PLACEHOLDER.test(id))).toBe(true)
})
})
describe('shortestExampleFor: examples reach the caller that already failed', () => {
it('picks the shortest example, so the error stays readable', () => {
const schema = {
examples: [{ a: 1, b: 2, c: 3, d: 4 }, { a: 1 }],
} as Record<string, unknown>
expect(shortestExampleFor(schema)).toBe('{"a":1}')
})
it('returns nothing when the tool publishes no examples', () => {
expect(shortestExampleFor({})).toBe('')
expect(shortestExampleFor({ examples: [] })).toBe('')
})
it('declines an example too long to help inside an error message', () => {
const long = { note: 'x'.repeat(400) }
expect(shortestExampleFor({ examples: [long] })).toBe('')
})
it('gives the kpi-report caller the empty-object shape it needed', () => {
// The exact prod case: `metric` sent to a tool whose only parameter is
// period_id, 604 times over seven days.
const kpi = tools.find((t) => t.name === 'gnubok_get_kpi_report')!
expect(shortestExampleFor(kpi.inputSchema as Record<string, unknown>)).toBe('{}')
})
})
@@ -328,6 +328,19 @@ describe('tools/list payload size guard', () => {
// zero. Demoting those would hide the year-end flow exactly when it
// is needed. Usage data is necessary here, not sufficient.
//
// * 61.3K to 61.5K by adding worked `examples` to five tools
// (2026-09-01, #2066): categorize_transaction, create_voucher,
// query_journal, approve_pending_operation, get_kpi_report. 199 tokens
// for 10 examples, spending part of what the demotion above reclaimed
// and leaving ~118 under the ceiling. The ceiling is NOT raised.
// Examples were priced against 30 days of mcp.tool_called and aimed at
// the combinations the descriptions already warn about and callers
// still get wrong (account_override without vat_treatment;
// representation without deltagare; confirmed on a high-risk approval;
// `metric` sent to a tool whose only parameter is period_id).
// Cheaper than it looks per example, so the next batch should still
// demote a read first rather than assume there is room.
//
// Long-term answer to growth is no longer a ceiling bump. gnubok_call_tool
// makes `catalogVisibility: 'search'` usable for READ tools on hosts that
// can only invoke what tools/list showed them, which is the constraint that
@@ -26,3 +26,23 @@ export function findUnknownArgKeys(
const allowed = new Set(listArgKeys(inputSchema))
return Object.keys(args).filter((key) => key !== 'company_id' && !allowed.has(key))
}
/**
* The shortest worked example a tool publishes, serialized for an error
* message, or '' when it has none or the shortest is too long to help.
*
* "Valid parameters: period_id" told DueCue's agent which key was wrong but
* not what a correct call looks like, and the same rejected call repeated for
* seven days (604 of them). One example costs nothing in tools/list, because
* it only ships on the response to a call that already failed.
*/
export function shortestExampleFor(inputSchema: Record<string, unknown>, maxChars = 200): string {
const examples = inputSchema.examples
if (!Array.isArray(examples) || examples.length === 0) return ''
const serialized = examples
.map((e) => JSON.stringify(e))
.filter((json): json is string => typeof json === 'string')
.sort((a, b) => a.length - b.length)
const shortest = serialized.find((json) => json.length <= maxChars)
return shortest ?? ''
}
+40 -2
View File
@@ -187,7 +187,7 @@ import {
projectToolInputSchema,
resolveMcpCompanyContext,
} from './company-routing'
import { findUnknownArgKeys, listArgKeys } from './arg-guard'
import { findUnknownArgKeys, listArgKeys, shortestExampleFor } from './arg-guard'
import { findSupplierCandidates, type SupplierRow } from './supplier-candidates'
import {
matchSupplierByIdentity,
@@ -5358,6 +5358,14 @@ export const tools: McpTool[] = [
idempotency_key: { type: 'string', description: 'Optional UUID to dedupe retries: a replayed call returns the already-staged operation instead of staging twice.' },
},
required: ['transaction_id', 'category'],
// The two combinations the prose above describes and callers still get
// wrong: an account_override without an explicit vat_treatment (books
// GROSS, no moms line), and representation without deltagare + syfte.
examples: [
{ transaction_id: '3f1a...', category: 'expense_office' },
{ transaction_id: '3f1a...', category: 'expense_other', account_override: '4600', vat_treatment: 'standard_25' },
{ transaction_id: '3f1a...', category: 'expense_representation', notes: 'Anna Andersson (Acme AB), kundmöte om ramavtal' },
],
},
outputSchema: STAGED_OPERATION_SCHEMA,
annotations: {
@@ -6961,6 +6969,10 @@ export const tools: McpTool[] = [
properties: {
period_id: { type: 'string', description: 'Fiscal period UUID (default: most recent)' },
},
// period_id is the whole surface. Callers have shipped `metric` here for
// days at a time (604 rejected calls, 2026-08-24 to 08-31): the empty
// object is the example that says there is nothing else to pass.
examples: [{}, { period_id: '7c2b...' }],
},
outputSchema: { type: 'object' },
annotations: {
@@ -9237,6 +9249,12 @@ export const tools: McpTool[] = [
group_by_dimension: { type: 'string', description: 'Aggregate by SIE dimension number (e.g. "6" = projekt) from each line\'s dimensions bag; untagged → "(utan dimension)". Mutually exclusive with group_by.' },
limit: { type: 'number', minimum: 1, maximum: 500, description: 'Max lines returned 1-500 (default 100); totals/groups still cover the full match set.' },
},
// Two shapes that cover most ad-hoc questions: an account range over a
// period, and a free-text hunt. status defaults to 'all' on purpose.
examples: [
{ account_from: '4000', account_to: '4999', date_from: '2026-01-01', date_to: '2026-03-31' },
{ text: 'Kjell', status: 'posted' },
],
},
outputSchema: {
type: 'object',
@@ -18187,6 +18205,19 @@ export const tools: McpTool[] = [
},
},
required: ['entry_date', 'description', 'lines'],
// Balance is the rule agents break: sum(debit) === sum(credit), and the
// moms leg is its own line on its own BAS account, never folded in.
examples: [
{
entry_date: '2026-03-31',
description: 'Kontorsmaterial Kjell & Company',
lines: [
{ account_number: '6110', debit_amount: 400 },
{ account_number: '2641', debit_amount: 100 },
{ account_number: '1930', credit_amount: 500 },
],
},
],
},
outputSchema: STAGED_OPERATION_SCHEMA,
annotations: { readOnlyHint: false, destructiveHint: false, idempotentHint: false, openWorldHint: false },
@@ -19350,6 +19381,9 @@ export const tools: McpTool[] = [
},
},
required: ['operation_id'],
// confirmed is not optional for a high-risk operation: without it the
// approval is refused, which reads to an agent as a permissions problem.
examples: [{ operation_id: '9a44...' }, { operation_id: '9a44...', confirmed: true }],
},
outputSchema: {
type: 'object',
@@ -21181,10 +21215,14 @@ export async function handleMcpRequest(request: Request): Promise<Response> {
// VALIDATION_ERROR envelope, never as a half-applied call.
const unknownArgKeys = findUnknownArgKeys(tool.inputSchema as Record<string, unknown>, toolArgs)
if (unknownArgKeys.length > 0) {
// Name the shape, not just the mistake: a caller that already sent a
// wrong key has no way to guess the right one from a key list alone.
const example = shortestExampleFor(tool.inputSchema as Record<string, unknown>)
throw codedError(
'VALIDATION_ERROR',
`Unknown parameter${unknownArgKeys.length > 1 ? 's' : ''} ${unknownArgKeys.map((k) => `"${k}"`).join(', ')} for ${requestedToolName}. ` +
`Valid parameters: ${listArgKeys(tool.inputSchema as Record<string, unknown>).join(', ') || '(none)'}. Unknown keys are rejected, not ignored.`,
`Valid parameters: ${listArgKeys(tool.inputSchema as Record<string, unknown>).join(', ') || '(none)'}. Unknown keys are rejected, not ignored.` +
(example ? ` A working call looks like: ${example}` : ''),
)
}