Files
accounted/lib/receipt-hunt/__tests__/mail-intelligence.test.ts
T
3829b6add3 fix(ai): complete plain-key self-hosting path (#1584)
* feat(ai): resolve the Claude backend from the environment

Tier 1 of #1406: a self-hosted deployment can now run every AI feature on a
plain ANTHROPIC_API_KEY, with no AWS account. Hosted behaviour is unchanged.

lib/ai/provider.ts resolves the backend once, from the environment:

  AI_PROVIDER              explicit override, bedrock|anthropic
  AWS static key pair      Bedrock
  ANTHROPIC_API_KEY        the direct Anthropic API
  nothing set              Bedrock, so the AWS credential provider chain
                           (instance profile, IRSA) still resolves

Bedrock deliberately wins when both credential sets are present. EU residency
in eu-north-1 is a BFL/GDPR posture rather than a default, so adding an
Anthropic key for an experiment must not silently move production inference
out of the region. AI_PROVIDER is the way to say you meant it.

Model ids are written bare in code and prefixed to eu.anthropic.* only for
Bedrock, which needs the cross-region inference profile for on-demand
throughput. An operator override that already carries a prefix passes through
untouched, so BEDROCK_MODEL_ID and friends keep working as written.

Converted call sites: the agent composer, invoice-inbox extraction, the
document-extraction model label, and both receipt-hunt clients. The last two
are not named in the issue, which predates receipt-hunt landing in main.

@anthropic-ai/sdk is declared at 0.95.0, the version @anthropic-ai/bedrock-sdk
0.29.1 already pulled in transitively, so the lockfile dedupes to one copy
with no new download.

scripts/smoke-bedrock.ts becomes scripts/smoke-ai.ts and grows two steps.
Unit tests can only prove which provider and model id get resolved; they
cannot prove the resulting request is one the backend accepts. The script now
sends real traffic over all three shapes the app uses: a plain create, a
streamed turn carrying adaptive thinking, an effort level, an hour-long cache
breakpoint and a tool, and document extraction end to end when given a file.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Bjorn Bergenheim <29535152+bjornbergenheim@users.noreply.github.com>

* docs(self-hosting): document the AI smoke test

The script added alongside the provider split is what closes the #1406
acceptance criterion ("document extraction and the assistant both work"), so
a self-hoster needs to know it exists. Covers both invocations and states
that it exits non-zero, which is what makes it usable as a post-deploy check.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Bjorn Bergenheim <29535152+bjornbergenheim@users.noreply.github.com>

* test(ai): split the smoke test's thinking probe from its tool probe

The combined probe could not falsify what it claimed to. It asked a question
that needs a tool call, so the tool was used and adaptive thinking correctly
declined to reason about it: the zero thinking-block count that came back was
uninformative rather than a signal.

2a keeps the tool and drops thinking. 2b asks a question with several
dependent steps (reverse charge, then a partial deduction, then the affected
boxes) so that a model honouring the parameter must reason, and reports the
thinking text length as well as the block count, since display:"summarized"
can yield blocks with empty text.

The cached system prompt is also padded past the 1024-token minimum cacheable
prefix. Below that the API caches nothing and reports no error, so the old
probe's cache counters read zero whether or not caching worked.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Bjorn Bergenheim <29535152+bjornbergenheim@users.noreply.github.com>

* fix(document-extraction): stop requiring AWS_REGION in the manifest

The extension now needs one of two credential sets, AWS static keys or
ANTHROPIC_API_KEY, and the manifest schema cannot express "one of". Since
requiredEnvVars only drives a build-time warning and never gates anything,
listing AWS_REGION told every self-hoster running the direct API to set a
variable that has no effect for them.

The description was also still promising Sonnet 4.6 via Bedrock specifically,
which is no longer what the extension does.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Bjorn Bergenheim <29535152+bjornbergenheim@users.noreply.github.com>

* fix(ai): read documentKind defensively in the smoke test

The field arrived with the receipt-aware extraction work, so referencing it
directly stops the script compiling against any checkout from before that
landed. tsconfig includes **/*.ts and next.config does not disable type
checking, so on such a checkout this failed the production build rather than
just the script: caught while preparing a test branch for a self-hosted
instance that had not synced yet.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Bjorn Bergenheim <29535152+bjornbergenheim@users.noreply.github.com>

* fix(deps): restore the nested @swc/helpers entry in the lockfile

Declaring @anthropic-ai/sdk with `npm install --package-lock-only` also pruned
node_modules/next-intl/node_modules/@swc/helpers@0.5.23, an optional peer entry
the local npm 11 considers redundant and the image's npm 10.9.8 does not. The
result passed every local check and failed `npm ci` inside the Docker build,
which is the only place the lockfile is actually enforced.

The lockfile is now the previous one plus the single root dependency line,
verified with `npm ci --dry-run`. @anthropic-ai/sdk needed nothing else: it was
already in the tree as a transitive dependency of @anthropic-ai/bedrock-sdk.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Bjorn Bergenheim <29535152+bjornbergenheim@users.noreply.github.com>

* Update DECISIONS.md

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Update Docker documentation for AI provider credentials

Clarify the role of credentials in AI provider selection and document extraction requirements.

* Update SELF-HOSTING.md with smoke-ai script details

Clarify usage of smoke-ai script for credential checks and document extraction.

* Improve error handling and logging in smoke-ai script

* fix(ai): complete plain-key self-hosting path

Signed-off-by: Emil <emilmattsson14@gmail.com>

---------

Signed-off-by: Bjorn Bergenheim <29535152+bjornbergenheim@users.noreply.github.com>
Signed-off-by: Emil <emilmattsson14@gmail.com>
Co-authored-by: Bjorn Bergenheim <29535152+bjornbergenheim@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2026-08-13 15:45:24 +02:00

156 lines
6.1 KiB
TypeScript

/**
* The model reads mails and reports fields. These tests are about what happens
* when it reports something it should not: an id we never offered, a filename
* that is not there, a negative total, a date that is not a date. None of that
* may reach the matcher, because the matcher trusts its inputs.
*/
import { describe, it, expect, vi, beforeEach } from 'vitest'
const mockCreate = vi.fn()
vi.mock('@/lib/ai/provider', () => ({
createAiClient: () => ({ messages: { create: (...args: unknown[]) => mockCreate(...args) } }),
toProviderModelId: (id: string) => id,
}))
import { extractMailDocuments, type CandidateForReview } from '../mail-intelligence'
/** A reply in the shape forced tool use produces. */
function toolReply(input: unknown) {
return { content: [{ type: 'tool_use', name: 'x', id: 'tu', input }] }
}
function candidate(overrides: Partial<CandidateForReview> = {}): CandidateForReview {
return {
messageId: 'msg-1',
mailbox: 'invoice@arcim.io',
subject: 'Fwd: Your receipt from Anthropic, PBC',
from: 'jakob@example.com',
receivedAt: '2026-08-01T10:00:00.000Z',
bodyText: 'Vidarebefordrat meddelande Från: Anthropic, PBC Datum: mån 15 juni 2026 €180.00',
attachmentNames: ['Receipt-2066.pdf'],
...overrides,
}
}
function doc(overrides: Record<string, unknown> = {}) {
return {
message_id: 'msg-1',
attachment_name: 'Receipt-2066.pdf',
is_receipt: true,
vendor: 'Anthropic, PBC',
date: '2026-06-15',
amount: 180,
currency: 'eur',
...overrides,
}
}
beforeEach(() => vi.clearAllMocks())
describe('extractMailDocuments', () => {
it('reports the fields a matcher needs', async () => {
mockCreate.mockResolvedValue(toolReply({ documents: [doc()] }))
const [out] = await extractMailDocuments([candidate()])
expect(out).toMatchObject({
messageId: 'msg-1',
attachmentName: 'Receipt-2066.pdf',
vendor: 'Anthropic, PBC',
date: '2026-06-15',
amount: 180,
currency: 'EUR',
})
})
it('takes the purchase date from the mail, not the forwarding date', async () => {
// The mail was forwarded in August; the purchase was in June. Matching on
// the forwarding date is what made the old design miss five of six repeat
// subscriptions.
mockCreate.mockResolvedValue(toolReply({ documents: [doc()] }))
const [out] = await extractMailDocuments([candidate({ receivedAt: '2026-08-01T10:00:00.000Z' })])
expect(out.date).toBe('2026-06-15')
})
it('drops mail that is not an underlag', async () => {
mockCreate.mockResolvedValue(
toolReply({ documents: [doc({ is_receipt: false, vendor: null, amount: null })] }),
)
await expect(extractMailDocuments([candidate()])).resolves.toEqual([])
})
it('never reports a message id it was not given', async () => {
mockCreate.mockResolvedValue(toolReply({ documents: [doc({ message_id: 'invented' })] }))
await expect(extractMailDocuments([candidate()])).resolves.toEqual([])
})
it('rejects a filename that is not on the message', async () => {
// An invented filename means it was guessing about the contents.
mockCreate.mockResolvedValue(toolReply({ documents: [doc({ attachment_name: 'ghost.pdf' })] }))
await expect(extractMailDocuments([candidate()])).resolves.toEqual([])
})
it('treats a missing amount as unknown rather than as zero', async () => {
// Most receipts state their total only inside the PDF. Null must stay null:
// a zero would score as an amount that disagrees with every transaction.
mockCreate.mockResolvedValue(toolReply({ documents: [doc({ amount: null })] }))
const [out] = await extractMailDocuments([candidate()])
expect(out.amount).toBeNull()
})
it('discards a non-positive total as a misread', async () => {
mockCreate.mockResolvedValue(toolReply({ documents: [doc({ amount: -180 })] }))
const [out] = await extractMailDocuments([candidate()])
expect(out.amount).toBeNull()
})
it('discards a date that is not a date', async () => {
mockCreate.mockResolvedValue(toolReply({ documents: [doc({ date: 'juni 2026' })] }))
const [out] = await extractMailDocuments([candidate()])
expect(out.date).toBeNull()
})
it('reports one document per attachment in a batch forward', async () => {
// "Fwd: Kvitton februari" carries five receipts for five purchases.
mockCreate.mockResolvedValue(
toolReply({
documents: [
doc({ attachment_name: 'a.pdf', amount: 162.02 }),
doc({ attachment_name: 'b.pdf', amount: 425 }),
],
}),
)
const out = await extractMailDocuments([candidate({ attachmentNames: ['a.pdf', 'b.pdf'] })])
expect(out.map((d) => d.amount)).toEqual([162.02, 425])
})
it('refuses to guess which file, when the mail carries several', async () => {
// Without a filename the caller would fetch attachment number one and hope.
// On a batch forward that is a coin flip, so the document is dropped.
mockCreate.mockResolvedValue(toolReply({ documents: [doc({ attachment_name: null })] }))
await expect(
extractMailDocuments([candidate({ attachmentNames: ['a.pdf', 'b.pdf', 'c.pdf'] })]),
).resolves.toEqual([])
})
it('still accepts a body-only receipt, where there is nothing to choose', async () => {
mockCreate.mockResolvedValue(toolReply({ documents: [doc({ attachment_name: null })] }))
const out = await extractMailDocuments([candidate({ attachmentNames: [] })])
expect(out).toHaveLength(1)
})
it('accepts an array the model sent as a JSON string', async () => {
mockCreate.mockResolvedValue(toolReply({ documents: JSON.stringify([doc()]) }))
await expect(extractMailDocuments([candidate()])).resolves.toHaveLength(1)
})
it('reports nothing when the call fails', async () => {
mockCreate.mockRejectedValue(new Error('bedrock timeout'))
await expect(extractMailDocuments([candidate()])).resolves.toEqual([])
})
it('does not call the model when there is nothing to read', async () => {
await expect(extractMailDocuments([])).resolves.toEqual([])
expect(mockCreate).not.toHaveBeenCalled()
})
})