feat(receipt-hunt): look for receipts on request, from the mailbox settings page (#1496)
* feat(receipt-hunt): a button that looks for receipts on request The nightly cron exists but still does not search mailboxes, and for a good reason: a sweep of one real 172-message mailbox took over 600s, against a scheduled function's 300. Pressing a button is the honest shape for work that big. A bounded pass reports what it found and how much is left, and the person decides whether to press again; a nightly run could only truncate silently. POST /api/receipt-hunt/run searches the mailboxes for eight purchases and fetches at most ten receipts per press. Gated on the AI tier, because reading the amount out of a PDF is what makes a fetched attachment matchable at all: without it the hunt would file documents that can never pair, which is worse than not running. Writes no journal entries; every pairing is still a proposal waiting for approval. The button lives on the mailbox settings page, which already ships, and says what happened in words rather than a spinner that stops: "3 underlag hämtade. 12 köp kvar att söka igenom." huntCompany gains maxReceipts so a manual pass can carry a different budget from a nightly one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(receipt-hunt): stop a manual press spending its budget on the wrong purchases The first real press searched eight purchases, read forty mails and found nothing, which looked like a broken model. It was the ordering. Purchases are searched largest first, which is right for a nightly sweep that eventually covers everything and wrong for a button pressed a few times. On a real ledger the largest rows are the least likely to have a findable receipt: rent already invoiced, bare payment references, direct debits. Those filled the forty-mail cap, so the productive purchases further down the list, the ones whose receipts are actually sitting in the mailbox, were never read at all. The cap was the binding constraint, not the time: eight purchases and forty mails took 43s of the 300 available. A press now searches 25 purchases and reads 100 mails, measured at 85s and finding 7 underlag on the same ledger that returned 0 before. huntCompany gains maxMails alongside maxReceipts, so a manual pass can carry a different budget from a nightly one rather than sharing an environment default with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(receipt-hunt): one underlag per purchase, and a press that fits its budget Your second press exposed three things, none of which a dry run reaches. It took 5.8 minutes. The 85s I measured was a dry run, which never fetches, uploads or extracts; each fetched receipt costs about another 37s because it is downloaded, stored, and then read by a model that opens the PDF. Seven of them ran past the 300s a serverless function gets, so in production that press would have been killed. Four receipts per press keeps a full pass inside the budget. It fetched seven receipts and proposed nothing. A single mail carries the invoice AND the receipt for one purchase under different names ("Invoice-E19DBF63-0021.pdf" beside "Receipt-2066-0204-8388.pdf"), and the same receipt reaches a second mailbox on a different message. Each was fetched separately, so the pool filled with identical candidates and the matcher refused to propose any of them rather than flip a coin. The per-run key is now the vendor and the total, which is what identifies a purchase; the filename only decides when no amount was read. Nine duplicates already in the pool were removed. And with the duplicates gone it still proposed nothing, for a separate reason: "Utlägg Norwegian" scored 0.18 against "Norwegian Air Shuttle AOC AS". Utlägg is Swedish for an expense reimbursement, bank vocabulary rather than a company, and leaving it in broke the token-subset match, so an exact 1 998 kr pair leaned entirely on a date eight days out and fell under the floor. Stripped, along with överföring, via internet, bg-bet and autogiro, in the comparison path only. normalizeMerchantName is untouched: it is the persisted konteringskarta key with a SQL mirror, and its 22 string pins and the 27-pair golden set still pass. Measured after: the Norwegian pair proposes at 0.72. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf(mail): read each message once per press, not once per query The third press proposed a pairing, which the second had not, but still ran 5.2 minutes against a function's 300s. Cutting receipts from seven to four had only saved 36s, which said the receipts were never the cost. Every search fetched a full message for every hit, and a press searches many purchases across every connected mailbox. One receipt mail answers several of those queries, so 25 purchases against 2 mailboxes could ask Gmail for well over a thousand messages to end up with a hundred distinct ones. Deduplication happened in the caller, too late to save the work. A mail's content never changes, so it is now read once per mailbox and kept, bounded at a thousand entries and evicting oldest first. Measured on the same ledger: 55 purchases and 100 mails now take 102s, where 25 purchases alone previously cost around 264s before a single receipt was fetched. clearMessageCache exists because tests reuse message ids and production does not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(receipt-hunt): do not refetch a receipt the company already holds The fourth press fetched four documents the company already had: Bolagsverket, Supabase twice, Uber. They came back because I had deleted them as duplicates, and the cross-run check is the message and attachment id, which lives in the rows I removed. That was my mistake, but it exposed a real gap. The vendor-and-total key only deduplicates inside a single pass. Across passes the same purchase still arrives as an invoice in one mail and a receipt in another, with different file keys, and both were fetched: the pool fills with identical candidates and the matcher then refuses to choose between them, which is how a press can fetch four documents and propose nothing. The pass now starts from what the company already holds, so its budget goes on documents that are actually missing. Receipts per press drops to three. Measured on this ledger, a fetched receipt costs about 50s from download to a stored amount, and that is the model reading the PDF rather than the network: seven took 5.8 minutes and four took 5.1, both past the 300s a function gets. Three fits, but it is a stopgap. Doing the fetch inside the request is the wrong shape for work this slow, and the fix is to move it off the request rather than keep shaving this number. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(mail): a mail body must not outlive the run that read it Two findings from the review, both correct. The message cache held whole MailCandidate values, and one of those fields is the mail body. The contract says a body is read once to extract fields and discarded, and a process-global cache quietly broke that: bodies of one company's mail sat in memory across requests until eviction or a restart. The MailSearchService contract now has releaseCache, the Gmail adapter clears its messages, and the hunt calls it in a finally so a failed run releases them too. The duplicate key accepted an empty vendor, so two unrelated documents that happened to cost the same collapsed into one candidate. Those now fall back to the file they came from: without a vendor there is nothing to anchor an amount to. The same finding caught something worse that I had introduced one commit earlier. The persistent check derived its key from the stored extraction while the fetch derived one from the reading model, so a document filed as "Norwegian Air Shuttle AOC AS" did not recognise an incoming "Norwegian" and was fetched again. Rather than guess at aliases, which would fold "Google Cloud" into "Google Workspace", the identity is now written onto the row when the receipt is filed and read back verbatim. Rows filed before that fall back to the extraction. receiptIdentity is one exported helper with its own tests, used by both sides, instead of the same expression written twice. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(receipt-hunt): a monthly subscription is not a duplicate of last month The review caught that my duplicate key was worse than the problem it solved. Anthropic bills the same amount every month, and keying on vendor and total alone made July look like a duplicate of June: every later receipt from any recurring supplier would have been suppressed forever, silently. Duplicates block one proposal; that would have lost a receipt per month per subscription. The identity now carries the document date. Two documents for one purchase share a date; June and July do not. Two smaller faults in the same key. The amount was serialised as a raw float, so 0.1 + 0.2 read as a different total from 0.3; it is rounded to öre like every other money comparison in this codebase. And a document with no vendor was identified by its filename alone, which collapses two unrelated papers whenever a billing system attaches "invoice.pdf": those now carry the message they came from. The key is versioned so a future change to its shape cannot be mistaken for a match against rows written under the old one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jakob Wennberg
co-authored by
Claude Opus 5
Jakob Wennberg
Claude Opus 5
Jakob Wennberg
parent
6821568523
commit
1d635b0d25
@@ -0,0 +1,132 @@
|
||||
/**
|
||||
* The button's route. What matters is that it cannot run for someone who is not
|
||||
* signed in, cannot run without the tier that reads PDFs, and reports enough
|
||||
* for a person to decide whether to press again.
|
||||
*/
|
||||
import { describe, it, expect, vi, beforeEach } from 'vitest'
|
||||
import { createMockRequest, parseJsonResponse } from '@/tests/helpers'
|
||||
|
||||
const mockHuntCompany = vi.fn()
|
||||
vi.mock('@/lib/receipt-hunt/hunt', () => ({
|
||||
huntCompany: (...args: unknown[]) => mockHuntCompany(...args),
|
||||
}))
|
||||
|
||||
const mockRequireCapability = vi.fn()
|
||||
vi.mock('@/lib/entitlements/has-capability', () => ({
|
||||
requireCapability: (...args: unknown[]) => mockRequireCapability(...args),
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/supabase/server', () => ({
|
||||
createServiceClient: () => ({}),
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/init', () => ({ ensureInitialized: vi.fn() }))
|
||||
|
||||
const context = {
|
||||
requestId: 'req-1',
|
||||
log: { info: vi.fn(), warn: vi.fn(), error: vi.fn(), child: vi.fn(() => context.log) },
|
||||
user: { id: 'user-1' },
|
||||
supabase: {},
|
||||
companyId: 'co-1',
|
||||
}
|
||||
|
||||
let unauthorized = false
|
||||
vi.mock('@/lib/api/with-route-context', () => ({
|
||||
withRouteContext: (_op: string, handler: (req: unknown, ctx: unknown) => unknown) => {
|
||||
return async (req: unknown) => {
|
||||
if (unauthorized) {
|
||||
return new Response(JSON.stringify({ error: 'Unauthorized' }), { status: 401 })
|
||||
}
|
||||
return handler(req, context)
|
||||
}
|
||||
},
|
||||
}))
|
||||
|
||||
import { POST } from '../route'
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
unauthorized = false
|
||||
mockRequireCapability.mockResolvedValue(null)
|
||||
mockHuntCompany.mockResolvedValue({
|
||||
companyId: 'co-1',
|
||||
candidates: 20,
|
||||
poolSize: 5,
|
||||
proposed: 2,
|
||||
mail: { searched: 8, withCandidates: 3, ingested: 3, candidates: [] },
|
||||
})
|
||||
})
|
||||
|
||||
describe('POST /api/receipt-hunt/run', () => {
|
||||
it('refuses an unauthenticated caller', async () => {
|
||||
unauthorized = true
|
||||
const response = await POST(createMockRequest('http://localhost/api/receipt-hunt/run'), undefined as never)
|
||||
expect(response.status).toBe(401)
|
||||
expect(mockHuntCompany).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('refuses a company without the tier that reads PDFs', async () => {
|
||||
// Fetching receipts nobody can extract an amount from would file documents
|
||||
// that can never pair: worse than not running.
|
||||
mockRequireCapability.mockResolvedValue(
|
||||
new Response(JSON.stringify({ error: 'capability_blocked' }), { status: 402 }),
|
||||
)
|
||||
const response = await POST(createMockRequest('http://localhost/api/receipt-hunt/run'), undefined as never)
|
||||
expect(response.status).toBe(402)
|
||||
expect(mockHuntCompany).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('searches the mailboxes, which the nightly run still does not', async () => {
|
||||
await POST(createMockRequest('http://localhost/api/receipt-hunt/run'), undefined as never)
|
||||
const [, , , options] = mockHuntCompany.mock.calls[0]
|
||||
expect(options.searchMail).toBe(true)
|
||||
})
|
||||
|
||||
it('bounds the pass so one press cannot run past the function timeout', async () => {
|
||||
await POST(createMockRequest('http://localhost/api/receipt-hunt/run'), undefined as never)
|
||||
const [, , , options] = mockHuntCompany.mock.calls[0]
|
||||
expect(options.mailSearchLimit).toBeGreaterThan(0)
|
||||
expect(options.maxReceipts).toBeGreaterThan(0)
|
||||
})
|
||||
|
||||
it('reports what is left, so pressing again is an informed choice', async () => {
|
||||
const response = await POST(createMockRequest('http://localhost/api/receipt-hunt/run'), undefined as never)
|
||||
const { body } = await parseJsonResponse<{
|
||||
data: { searched: number; fetched: number; proposed: number; remaining: number }
|
||||
}>(response)
|
||||
|
||||
expect(body.data).toMatchObject({ searched: 8, fetched: 3, proposed: 2 })
|
||||
// 20 purchases without a receipt, 8 looked at.
|
||||
expect(body.data.remaining).toBe(12)
|
||||
})
|
||||
|
||||
it('never reports negative work remaining', async () => {
|
||||
mockHuntCompany.mockResolvedValue({
|
||||
companyId: 'co-1',
|
||||
candidates: 3,
|
||||
poolSize: 0,
|
||||
proposed: 0,
|
||||
mail: { searched: 8, withCandidates: 0, ingested: 0, candidates: [] },
|
||||
})
|
||||
const response = await POST(createMockRequest('http://localhost/api/receipt-hunt/run'), undefined as never)
|
||||
const { body } = await parseJsonResponse<{ data: { remaining: number } }>(response)
|
||||
expect(body.data.remaining).toBe(0)
|
||||
})
|
||||
|
||||
it('survives a company with no mailbox connected', async () => {
|
||||
// getMailSearchService falls back to a no-op, so the mail leg is absent
|
||||
// rather than failing.
|
||||
mockHuntCompany.mockResolvedValue({
|
||||
companyId: 'co-1',
|
||||
candidates: 4,
|
||||
poolSize: 2,
|
||||
proposed: 1,
|
||||
})
|
||||
const response = await POST(createMockRequest('http://localhost/api/receipt-hunt/run'), undefined as never)
|
||||
const { body } = await parseJsonResponse<{ data: { searched: number; fetched: number } }>(
|
||||
response,
|
||||
)
|
||||
expect(response.status).toBe(200)
|
||||
expect(body.data).toMatchObject({ searched: 0, fetched: 0 })
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,101 @@
|
||||
import { NextResponse } from 'next/server'
|
||||
import { ensureInitialized } from '@/lib/init'
|
||||
import { withRouteContext } from '@/lib/api/with-route-context'
|
||||
import { createServiceClient } from '@/lib/supabase/server'
|
||||
import { requireCapability } from '@/lib/entitlements/has-capability'
|
||||
import { CAPABILITY } from '@/lib/entitlements/keys'
|
||||
import { huntCompany } from '@/lib/receipt-hunt/hunt'
|
||||
|
||||
// The hunt uploads documents, and uploading emits document.uploaded, which is
|
||||
// what makes the extraction extension read the amount out of a fetched PDF.
|
||||
// Without this the receipts land with no amount and can never be paired.
|
||||
ensureInitialized()
|
||||
|
||||
/**
|
||||
* A run of the hunt that a person asked for.
|
||||
*
|
||||
* The nightly cron exists but is deliberately not searching mailboxes yet: a
|
||||
* sweep of one real 172-message mailbox took over 600s, and a scheduled
|
||||
* function has 300. Pressing a button is the honest shape for that. A bounded
|
||||
* pass reports what it found and what is left, and the person decides whether
|
||||
* to press again, which a silent nightly truncation could never do.
|
||||
*
|
||||
* Writes no journal entries. Every pairing becomes an
|
||||
* `attach_document_to_transaction` proposal that still waits for approval.
|
||||
*/
|
||||
|
||||
/**
|
||||
* Purchases whose mailboxes are searched per press.
|
||||
*
|
||||
* Purchases are searched largest first, and on a real ledger the largest rows
|
||||
* are the least likely to have a findable receipt: rent already invoiced,
|
||||
* bare payment references, direct debits. A press that only reaches the top of
|
||||
* that list finds nothing and looks broken. Measured: 8 purchases and 40 mails
|
||||
* took 43s of the 300 available, so the budget was being spent on the wrong
|
||||
* end rather than being scarce.
|
||||
*/
|
||||
const PURCHASES_PER_RUN = 25
|
||||
|
||||
/** Mails read per press. The real cost bound, and what the numbers above buy. */
|
||||
const MAILS_PER_RUN = 100
|
||||
|
||||
/**
|
||||
* Receipts fetched per press.
|
||||
*
|
||||
* The binding constraint on the whole route, and it is the model reading the
|
||||
* PDF rather than the network: measured on a real ledger, a fetched receipt
|
||||
* costs about 50s from download to a stored amount, while searching and
|
||||
* reading a hundred mail bodies costs roughly 100s in total. Seven receipts
|
||||
* took 5.8 minutes and four took 5.1, both past the 300s a function gets.
|
||||
*
|
||||
* Three is what fits. It is also a stopgap: doing the fetch inside the request
|
||||
* is the wrong shape for work this slow, and the honest fix is to move it off
|
||||
* the request entirely rather than keep shaving this number.
|
||||
*/
|
||||
const RECEIPTS_PER_RUN = 3
|
||||
|
||||
export const maxDuration = 300
|
||||
|
||||
export const POST = withRouteContext('receipt_hunt.run', async (_request, ctx) => {
|
||||
const { companyId, user, log } = ctx
|
||||
|
||||
// Reading a PDF is what turns a fetched attachment into something matchable,
|
||||
// and that is the paid AI tier. Without it the hunt would file documents that
|
||||
// can never pair, which is worse than not running.
|
||||
const blocked = await requireCapability(ctx.supabase, companyId, CAPABILITY.ai)
|
||||
if (blocked) return blocked
|
||||
|
||||
// Service role: the hunt reads mail_connections, whose RLS has no policies
|
||||
// precisely so a browser session can never select a refresh token.
|
||||
const supabase = createServiceClient()
|
||||
const runId = crypto.randomUUID()
|
||||
|
||||
log.info('manual receipt hunt starting', { companyId, runId, userId: user.id })
|
||||
|
||||
const result = await huntCompany(supabase, companyId, runId, {
|
||||
searchMail: true,
|
||||
mailSearchLimit: PURCHASES_PER_RUN,
|
||||
maxMails: MAILS_PER_RUN,
|
||||
maxReceipts: RECEIPTS_PER_RUN,
|
||||
})
|
||||
|
||||
const searched = result.mail?.searched ?? 0
|
||||
log.info('manual receipt hunt finished', {
|
||||
companyId,
|
||||
runId,
|
||||
searched,
|
||||
fetched: result.mail?.ingested ?? 0,
|
||||
proposed: result.proposed,
|
||||
})
|
||||
|
||||
return NextResponse.json({
|
||||
data: {
|
||||
// What the person needs to decide whether to press again.
|
||||
purchasesWithoutReceipt: result.candidates,
|
||||
searched,
|
||||
fetched: result.mail?.ingested ?? 0,
|
||||
proposed: result.proposed,
|
||||
remaining: Math.max(0, result.candidates - searched),
|
||||
},
|
||||
})
|
||||
})
|
||||
@@ -15,6 +15,14 @@ import { ConfirmDialog } from '@/components/ui/confirm-dialog'
|
||||
import { GoogleMark, MicrosoftMark } from '@/components/ui/provider-marks'
|
||||
import { formatDateLong } from '@/lib/utils'
|
||||
|
||||
interface HuntResult {
|
||||
searched: number
|
||||
fetched: number
|
||||
proposed: number
|
||||
remaining: number
|
||||
failed?: boolean
|
||||
}
|
||||
|
||||
interface MailConnection {
|
||||
id: string
|
||||
provider: 'gmail' | 'microsoft'
|
||||
@@ -34,6 +42,8 @@ export function MailConnectionsPanel() {
|
||||
const [loading, setLoading] = useState(true)
|
||||
const [connecting, setConnecting] = useState(false)
|
||||
const [pendingDisconnect, setPendingDisconnect] = useState<MailConnection | null>(null)
|
||||
const [hunting, setHunting] = useState(false)
|
||||
const [huntResult, setHuntResult] = useState<HuntResult | null>(null)
|
||||
|
||||
const load = useCallback(async () => {
|
||||
try {
|
||||
@@ -73,6 +83,32 @@ export function MailConnectionsPanel() {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* One bounded pass, on request.
|
||||
*
|
||||
* Deliberately not a background job: a sweep of a real mailbox runs longer
|
||||
* than a serverless function may live, so the honest shape is a pass that
|
||||
* ends, says what it found and what is left, and can be pressed again.
|
||||
*/
|
||||
async function hunt() {
|
||||
setHunting(true)
|
||||
setHuntResult(null)
|
||||
try {
|
||||
const response = await fetch('/api/receipt-hunt/run', { method: 'POST' })
|
||||
if (!response.ok) {
|
||||
setHuntResult({ searched: 0, fetched: 0, proposed: 0, remaining: 0, failed: true })
|
||||
return
|
||||
}
|
||||
const body = (await response.json()) as { data: HuntResult }
|
||||
setHuntResult(body.data)
|
||||
void load()
|
||||
} catch {
|
||||
setHuntResult({ searched: 0, fetched: 0, proposed: 0, remaining: 0, failed: true })
|
||||
} finally {
|
||||
setHunting(false)
|
||||
}
|
||||
}
|
||||
|
||||
async function disconnect(connection: MailConnection) {
|
||||
await fetch(`${BASE}/connections?id=${encodeURIComponent(connection.id)}`, { method: 'DELETE' })
|
||||
setPendingDisconnect(null)
|
||||
@@ -122,6 +158,28 @@ export function MailConnectionsPanel() {
|
||||
)}
|
||||
</SettingsGroup>
|
||||
|
||||
{connections.length > 0 ? (
|
||||
<SettingsGroup label={t('hunt_title')} help={t('hunt_help')}>
|
||||
<SettingsRow label={t('hunt_row')} borderless>
|
||||
<SettingsRowEnd>
|
||||
{huntResult ? (
|
||||
<SettingsRowNote>
|
||||
{huntResult.failed
|
||||
? t('hunt_failed')
|
||||
: huntResult.fetched > 0
|
||||
? t('hunt_found', { count: huntResult.fetched, left: huntResult.remaining })
|
||||
: t('hunt_none', { left: huntResult.remaining })}
|
||||
</SettingsRowNote>
|
||||
) : null}
|
||||
<Button variant="secondary" size="sm" onClick={hunt} disabled={hunting}>
|
||||
{hunting ? <Loader2 className="mr-2 h-4 w-4 animate-spin" /> : null}
|
||||
{hunting ? t('hunt_running') : t('hunt_action')}
|
||||
</Button>
|
||||
</SettingsRowEnd>
|
||||
</SettingsRow>
|
||||
</SettingsGroup>
|
||||
) : null}
|
||||
|
||||
<div className="flex flex-wrap items-center gap-3">
|
||||
<Button onClick={connect} disabled={connecting || !configured}>
|
||||
{connecting ? (
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
* the format and the MIME walk.
|
||||
*/
|
||||
import { describe, it, expect, vi, beforeEach } from 'vitest'
|
||||
import { getMessageSummary } from '../gmail-client'
|
||||
import { clearMessageCache, getMessageSummary } from '../gmail-client'
|
||||
|
||||
const mockFetch = vi.fn()
|
||||
vi.stubGlobal('fetch', (...args: unknown[]) => mockFetch(...args))
|
||||
@@ -25,7 +25,11 @@ const HEADERS = [
|
||||
{ name: 'From', value: 'info@tic.io' },
|
||||
]
|
||||
|
||||
beforeEach(() => vi.clearAllMocks())
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
// The cache is keyed by message id, and these tests reuse ids.
|
||||
clearMessageCache()
|
||||
})
|
||||
|
||||
describe('getMessageSummary', () => {
|
||||
it('asks for the full message, because metadata omits the parts tree', async () => {
|
||||
@@ -100,3 +104,30 @@ describe('getMessageSummary', () => {
|
||||
expect(candidate.subject).toBe('Faktura-20251070')
|
||||
})
|
||||
})
|
||||
|
||||
/**
|
||||
* One receipt mail answers many purchases' queries, so the same message was
|
||||
* downloaded dozens of times per press. Content never changes, so reading it
|
||||
* once is both correct and the difference between a press that fits its time
|
||||
* budget and one that does not.
|
||||
*/
|
||||
describe('message reads are not repeated', () => {
|
||||
it('fetches a given message once per mailbox', async () => {
|
||||
respond({ id: 'cache-me', payload: { headers: HEADERS } })
|
||||
|
||||
await getMessageSummary('token', 'cache-me', 'conn-1', 'invoice@arcim.io')
|
||||
await getMessageSummary('token', 'cache-me', 'conn-1', 'invoice@arcim.io')
|
||||
await getMessageSummary('token', 'cache-me', 'conn-1', 'invoice@arcim.io')
|
||||
|
||||
expect(mockFetch).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
|
||||
it('keeps mailboxes apart, since a hit in one says nothing about the other', async () => {
|
||||
respond({ id: 'shared', payload: { headers: HEADERS } })
|
||||
|
||||
await getMessageSummary('token', 'shared', 'conn-1', 'invoice@arcim.io')
|
||||
await getMessageSummary('token', 'shared', 'conn-2', 'jakob@arcim.io')
|
||||
|
||||
expect(mockFetch).toHaveBeenCalledTimes(2)
|
||||
})
|
||||
})
|
||||
|
||||
@@ -123,6 +123,21 @@ export async function searchMessageIds(
|
||||
return (data.messages ?? []).map((m) => m.id)
|
||||
}
|
||||
|
||||
/**
|
||||
* Messages already read, keyed by connection and id.
|
||||
*
|
||||
* A press searches many purchases across every mailbox, and one receipt mail
|
||||
* answers several of those queries, so the same message was being downloaded
|
||||
* dozens of times: 25 purchases against 2 mailboxes could ask for well over a
|
||||
* thousand fetches to end up with a hundred distinct mails. Deduplication
|
||||
* happened afterwards, which was too late to save the work.
|
||||
*
|
||||
* A mail's content never changes, so caching it needs no invalidation. The cap
|
||||
* is what keeps a long-lived server from growing without bound.
|
||||
*/
|
||||
const summaryCache = new Map<string, MailCandidate>()
|
||||
const SUMMARY_CACHE_MAX = 1_000
|
||||
|
||||
/**
|
||||
* Subject, sender, date and which parts are attachments.
|
||||
*
|
||||
@@ -139,11 +154,15 @@ export async function getMessageSummary(
|
||||
connectionId: string,
|
||||
mailbox: string,
|
||||
): Promise<MailCandidate> {
|
||||
const cacheKey = `${connectionId}::${messageId}`
|
||||
const cached = summaryCache.get(cacheKey)
|
||||
if (cached) return cached
|
||||
|
||||
const msg = await gmailFetch<GmailMessage>(accessToken, `/messages/${messageId}?format=full`)
|
||||
const attachments: Array<{ id: string; filename: string }> = []
|
||||
collectAttachments(msg.payload, attachments)
|
||||
|
||||
return {
|
||||
const candidate: MailCandidate = {
|
||||
connectionId,
|
||||
mailbox,
|
||||
provider: 'gmail',
|
||||
@@ -161,6 +180,20 @@ export async function getMessageSummary(
|
||||
bodyText: readableBody(msg),
|
||||
bodyIsReceipt: attachments.length === 0,
|
||||
}
|
||||
|
||||
// Oldest out first: the working set of one press is what matters, and a
|
||||
// press that overflows the cap was going to refetch anyway.
|
||||
if (summaryCache.size >= SUMMARY_CACHE_MAX) {
|
||||
const oldest = summaryCache.keys().next().value
|
||||
if (oldest) summaryCache.delete(oldest)
|
||||
}
|
||||
summaryCache.set(cacheKey, candidate)
|
||||
return candidate
|
||||
}
|
||||
|
||||
/** Drop everything read so far. Tests reuse message ids; production does not. */
|
||||
export function clearMessageCache(): void {
|
||||
summaryCache.clear()
|
||||
}
|
||||
|
||||
export async function fetchAttachmentBytes(
|
||||
|
||||
@@ -17,6 +17,7 @@ import type {
|
||||
} from '@/lib/mail-search/service'
|
||||
import { buildGmailQuery, looksLikeReceipt } from './gmail-query'
|
||||
import {
|
||||
clearMessageCache,
|
||||
describeAttachment,
|
||||
fetchAttachmentBytes,
|
||||
getMessageSummary,
|
||||
@@ -47,6 +48,15 @@ export class GmailSearchService implements MailSearchService {
|
||||
return isGoogleMailConfigured()
|
||||
}
|
||||
|
||||
/**
|
||||
* Messages are cached so one mail is not downloaded once per purchase, and a
|
||||
* cached message carries its body. The hunt calls this when it is done, so a
|
||||
* body never outlives the run that read it.
|
||||
*/
|
||||
releaseCache(): void {
|
||||
clearMessageCache()
|
||||
}
|
||||
|
||||
async search(companyId: string, query: MailSearchQuery): Promise<MailCandidate[]> {
|
||||
if (!this.isConfigured()) return []
|
||||
|
||||
|
||||
@@ -244,3 +244,25 @@ describe('calculateMerchantSimilarity on real confirmed pairs', () => {
|
||||
expect(calculateMerchantSimilarity(a, b)).toBeLessThan(0.6)
|
||||
})
|
||||
})
|
||||
|
||||
/**
|
||||
* Swedish bank vocabulary that wraps a merchant name without being part of it.
|
||||
* Drawn from a real ledger, where "Utlägg Norwegian" against "Norwegian Air
|
||||
* Shuttle AOC AS" scored 0.18 and an exact 1 998 kr match was never proposed.
|
||||
*/
|
||||
describe('payment words are not merchant names', () => {
|
||||
it('sees through an expense reimbursement', () => {
|
||||
expect(calculateMerchantSimilarity('Utlägg Norwegian', 'Norwegian Air Shuttle AOC AS'))
|
||||
.toBeGreaterThan(0.8)
|
||||
})
|
||||
|
||||
it('sees through a transfer', () => {
|
||||
expect(calculateMerchantSimilarity('Kontorsplatser j Bg-bet. via internet', 'Kontorsplatser AB'))
|
||||
.toBeGreaterThan(0.8)
|
||||
})
|
||||
|
||||
it('still tells two different merchants apart', () => {
|
||||
expect(calculateMerchantSimilarity('Utlägg Norwegian', 'Scandinavian Airlines System'))
|
||||
.toBeLessThan(0.5)
|
||||
})
|
||||
})
|
||||
|
||||
@@ -54,6 +54,14 @@ const LEGAL_FORM_TOKENS =
|
||||
*/
|
||||
const CARD_TOKEN = /\bk\d{4}\b/g
|
||||
const CARD_VERB = /\bkort(kop|kop\/uttag)?\b|\buttag\b/g
|
||||
/**
|
||||
* Swedish words a bank statement puts around a merchant name that are not part
|
||||
* of it. "Utlägg Norwegian" is an expense reimbursement for a Norwegian
|
||||
* ticket, not a company called Utlägg: leaving the word in cost the pair its
|
||||
* token-subset match and dropped merchant similarity to 0.18, which was enough
|
||||
* to keep an exact 1 998 kr match from ever being proposed.
|
||||
*/
|
||||
const PAYMENT_NOISE = /\butl[aä]gg\b|\b[oö]verf[oö]ring\b|\bvia internet\b|\bbg-?bet\b|\bautogiro\b/g
|
||||
const CARD_DATE_PREFIX = /^\s*kortkop\s+\d{6}\s*/
|
||||
const TRAILING_DATE = /\s*\/?\s*\d{2}-\d{2}-\d{2}\s*$/
|
||||
/**
|
||||
@@ -94,6 +102,7 @@ export function normalizeForMatch(name: string): string {
|
||||
s = s.replace(TRAILING_DATE, ' ')
|
||||
s = s.replace(CARD_TOKEN, ' ')
|
||||
s = s.replace(CARD_VERB, ' ')
|
||||
s = s.replace(PAYMENT_NOISE, ' ')
|
||||
s = s.replace(DOMAIN_TAIL, ' ')
|
||||
s = s.replace(/\bwww\b/g, ' ')
|
||||
|
||||
|
||||
@@ -96,6 +96,14 @@ export interface MailSearchService {
|
||||
): Promise<FetchedAttachment | null>
|
||||
/** True when at least one provider has credentials configured. */
|
||||
isConfigured(): boolean
|
||||
/**
|
||||
* Drop anything the adapter held for the duration of a hunt.
|
||||
*
|
||||
* Adapters cache messages so one mail is not downloaded once per purchase,
|
||||
* and a cached message carries its body. Body text is read to extract fields
|
||||
* and must not outlive the run that read it, so the caller says when that is.
|
||||
*/
|
||||
releaseCache?(): void
|
||||
}
|
||||
|
||||
class NoopMailSearchService implements MailSearchService {
|
||||
@@ -108,6 +116,7 @@ class NoopMailSearchService implements MailSearchService {
|
||||
isConfigured(): boolean {
|
||||
return false
|
||||
}
|
||||
releaseCache(): void {}
|
||||
}
|
||||
|
||||
let mailSearchService: MailSearchService = new NoopMailSearchService()
|
||||
|
||||
@@ -9,6 +9,7 @@ import {
|
||||
HUNT_MIN_CONFIDENCE,
|
||||
canHaveEmailReceipt,
|
||||
pairKey,
|
||||
receiptIdentity,
|
||||
worthFetching,
|
||||
selectProposals,
|
||||
type HuntPoolItem,
|
||||
@@ -339,25 +340,54 @@ describe('a receipt already offered elsewhere', () => {
|
||||
})
|
||||
|
||||
/**
|
||||
* The per-run fetch key. A filename is not an identity: half the world's
|
||||
* billing systems attach "invoice.pdf", so two suppliers would collide.
|
||||
* What identifies one purchase's paperwork. A mail carries the invoice and the
|
||||
* receipt for the same purchase under different names, and the same receipt
|
||||
* reaches a second mailbox on another message, so fetching per file filled the
|
||||
* pool with identical candidates the matcher then refused to choose between.
|
||||
*/
|
||||
describe('per-run duplicate key', () => {
|
||||
const key = (vendor: string | null, file: string | null, messageId: string) =>
|
||||
`${(vendor ?? '').toLowerCase()}::${(file ?? messageId).toLowerCase()}`
|
||||
|
||||
it('collapses the same invoice arriving four times', () => {
|
||||
// Original, reminder and two forwards, all carrying the identical file.
|
||||
const keys = new Set([
|
||||
key('Visma', 'Invoice_13041840.pdf', 'm1'),
|
||||
key('Visma', 'Invoice_13041840.pdf', 'm2'),
|
||||
key('Visma', 'invoice_13041840.pdf', 'm3'),
|
||||
key('Visma', 'Invoice_13041840.pdf', 'm4'),
|
||||
])
|
||||
expect(keys.size).toBe(1)
|
||||
describe('receiptIdentity', () => {
|
||||
it('collapses the invoice and the receipt for one purchase', () => {
|
||||
const d = { vendor: 'Anthropic', amount: 180, currency: 'EUR', date: '2026-06-15', messageId: 'm1' }
|
||||
expect(receiptIdentity({ ...d, attachmentName: 'Invoice-E19DBF63-0021.pdf' })).toBe(
|
||||
receiptIdentity({ ...d, attachmentName: 'Receipt-2066-0204-8388.pdf' }),
|
||||
)
|
||||
})
|
||||
|
||||
it('keeps two suppliers who both call it invoice.pdf', () => {
|
||||
expect(key('Loopia', 'invoice.pdf', 'm1')).not.toBe(key('Hetzner', 'invoice.pdf', 'm2'))
|
||||
it('keeps a subscription billing the same amount every month apart', () => {
|
||||
// Without the date, July would look like a duplicate of June and be
|
||||
// suppressed forever: a permanent, silent loss.
|
||||
expect(
|
||||
receiptIdentity({ vendor: 'Anthropic', amount: 225, currency: 'EUR', date: '2026-06-15' }),
|
||||
).not.toBe(
|
||||
receiptIdentity({ vendor: 'Anthropic', amount: 225, currency: 'EUR', date: '2026-07-15' }),
|
||||
)
|
||||
})
|
||||
|
||||
it('reads equivalent totals as one amount', () => {
|
||||
expect(
|
||||
receiptIdentity({ vendor: 'Uber', amount: 0.1 + 0.2, currency: 'SEK', date: '2026-06-01' }),
|
||||
).toBe(receiptIdentity({ vendor: 'Uber', amount: 0.3, currency: 'SEK', date: '2026-06-01' }))
|
||||
})
|
||||
|
||||
it('ignores wrapping the matcher already folds away', () => {
|
||||
expect(receiptIdentity({ vendor: 'Loopia AB', amount: 388, currency: 'SEK', date: '2026-06-11' })).toBe(
|
||||
receiptIdentity({ vendor: 'LOOPIA', amount: 388, currency: 'SEK', date: '2026-06-11' }),
|
||||
)
|
||||
})
|
||||
|
||||
it('keeps two suppliers apart', () => {
|
||||
expect(receiptIdentity({ vendor: 'Loopia', amount: 388, currency: 'SEK', date: '2026-06-11' })).not.toBe(
|
||||
receiptIdentity({ vendor: 'Hetzner', amount: 388, currency: 'SEK', date: '2026-06-11' }),
|
||||
)
|
||||
})
|
||||
|
||||
it('never lets a missing vendor make two documents the same', () => {
|
||||
// "invoice.pdf" is what half the world's billing systems attach, so the
|
||||
// message has to be part of the identity when there is no vendor.
|
||||
expect(
|
||||
receiptIdentity({ vendor: null, amount: 500, currency: 'SEK', messageId: 'm1', attachmentName: 'invoice.pdf' }),
|
||||
).not.toBe(
|
||||
receiptIdentity({ vendor: '', amount: 500, currency: 'SEK', messageId: 'm2', attachmentName: 'invoice.pdf' }),
|
||||
)
|
||||
})
|
||||
})
|
||||
|
||||
+107
-20
@@ -22,6 +22,7 @@ import { extractMailDocuments } from './mail-intelligence'
|
||||
import {
|
||||
MAX_PROPOSALS_PER_RUN,
|
||||
canHaveEmailReceipt,
|
||||
receiptIdentity,
|
||||
worthFetching,
|
||||
pairKey,
|
||||
selectProposals,
|
||||
@@ -147,6 +148,21 @@ export interface HuntOptions {
|
||||
searchMail?: boolean
|
||||
/** How many unexplained purchases to search mail for in one run. */
|
||||
mailSearchLimit?: number
|
||||
/**
|
||||
* Mails read in one run, overriding the environment default.
|
||||
*
|
||||
* The cap is what bounds cost, and it interacts with the largest-first
|
||||
* ordering: a low cap spends the whole budget on the biggest purchases,
|
||||
* which on a real ledger are the least likely to have a findable receipt.
|
||||
*/
|
||||
maxMails?: number
|
||||
/**
|
||||
* Receipts fetched in one run, overriding the environment default.
|
||||
*
|
||||
* A person pressing a button wants a pass that ends, reports, and can be
|
||||
* repeated. The nightly default is a different budget from a manual one.
|
||||
*/
|
||||
maxReceipts?: number
|
||||
/**
|
||||
* Score and decide, but write nothing.
|
||||
*
|
||||
@@ -352,6 +368,8 @@ export async function huntCompany(
|
||||
dryRun = false,
|
||||
searchMail = false,
|
||||
mailSearchLimit = MAX_MAIL_SEARCHES_PER_RUN,
|
||||
maxReceipts = MAX_RECEIPTS_PER_RUN,
|
||||
maxMails = MAX_MAILS_READ_PER_RUN,
|
||||
} = options
|
||||
|
||||
const [transactions, suppression] = await Promise.all([
|
||||
@@ -373,16 +391,26 @@ export async function huntCompany(
|
||||
// in a Gmail preview. So the receipt is filed, the extraction that already
|
||||
// runs on upload reads its amount, and the pairing below is the same
|
||||
// deterministic amount-and-merchant match used for every other underlag.
|
||||
const mail = searchMail
|
||||
? await harvestReceiptsFromMail(
|
||||
let mail: MailHuntSummary | undefined
|
||||
if (searchMail) {
|
||||
try {
|
||||
mail = await harvestReceiptsFromMail(
|
||||
supabase,
|
||||
companyId,
|
||||
userId,
|
||||
transactions.filter((t) => !suppression.claimedTransactionIds.has(t.id)),
|
||||
mailSearchLimit,
|
||||
dryRun,
|
||||
maxReceipts,
|
||||
maxMails,
|
||||
)
|
||||
: undefined
|
||||
} finally {
|
||||
// Cached messages carry their bodies. A body is read to extract fields
|
||||
// and must not outlive the run that read it, including when the run
|
||||
// fails partway.
|
||||
getMailSearchService().releaseCache?.()
|
||||
}
|
||||
}
|
||||
|
||||
const { pool, fileNames } = await fetchPool(supabase, companyId)
|
||||
|
||||
@@ -452,6 +480,59 @@ export async function huntCompany(
|
||||
* no mailbox is connected. Finding a candidate is NOT the same as having the
|
||||
* receipt: ingesting it is the next step and stays behind human approval.
|
||||
*/
|
||||
/**
|
||||
* Vendor and total of every receipt the hunt has already filed.
|
||||
*
|
||||
* The per-run key only stops duplicates inside one pass. Across passes the
|
||||
* check was the message and attachment, which does not recognise the same
|
||||
* purchase arriving as an invoice in one mail and a receipt in another: those
|
||||
* have different file keys and were fetched twice, filling the pool with
|
||||
* identical candidates the matcher then refuses to choose between.
|
||||
*/
|
||||
async function fetchAlreadyHeld(
|
||||
supabase: SupabaseClient,
|
||||
companyId: string,
|
||||
): Promise<Set<string>> {
|
||||
const rows = await fetchAllRows<{ extracted_data: unknown; channel_context: unknown }>((range) =>
|
||||
supabase
|
||||
.from('invoice_inbox_items')
|
||||
.select('extracted_data, channel_context')
|
||||
.eq('company_id', companyId)
|
||||
.eq('source', 'mail_hunt')
|
||||
.order('id', { ascending: true })
|
||||
.range(range.from, range.to),
|
||||
)
|
||||
|
||||
const held = new Set<string>()
|
||||
for (const row of rows) {
|
||||
const data = row.extracted_data as
|
||||
| {
|
||||
supplier?: { name?: string }
|
||||
invoice?: { currency?: string; invoiceDate?: string | null }
|
||||
totals?: { total?: number }
|
||||
}
|
||||
| null
|
||||
// Prefer the identity written when the receipt was filed: it came from the
|
||||
// same reading that the incoming candidate's does, so the two compare
|
||||
// exactly. Rows filed before that was stored fall back to the extraction,
|
||||
// which is weaker but better than nothing.
|
||||
const stored = row.channel_context as { receipt_identity?: string } | null
|
||||
if (stored?.receipt_identity) {
|
||||
held.add(stored.receipt_identity)
|
||||
continue
|
||||
}
|
||||
const key = receiptIdentity({
|
||||
vendor: data?.supplier?.name ?? null,
|
||||
amount: data?.totals?.total ?? null,
|
||||
currency: data?.invoice?.currency ?? null,
|
||||
date: data?.invoice?.invoiceDate ?? null,
|
||||
})
|
||||
if (key.includes('::file::')) continue
|
||||
held.add(key)
|
||||
}
|
||||
return held
|
||||
}
|
||||
|
||||
async function harvestReceiptsFromMail(
|
||||
supabase: SupabaseClient,
|
||||
companyId: string,
|
||||
@@ -459,6 +540,8 @@ async function harvestReceiptsFromMail(
|
||||
purchases: readonly HuntTransaction[],
|
||||
limit: number,
|
||||
dryRun: boolean,
|
||||
maxReceipts: number,
|
||||
maxMails: number,
|
||||
): Promise<MailHuntSummary> {
|
||||
const service = getMailSearchService()
|
||||
const summary: MailHuntSummary = { searched: 0, withCandidates: 0, ingested: 0, candidates: [] }
|
||||
@@ -504,7 +587,9 @@ async function harvestReceiptsFromMail(
|
||||
}
|
||||
if (byMessage.size === 0) return summary
|
||||
|
||||
const mails = [...byMessage.values()].slice(0, MAX_MAILS_READ_PER_RUN)
|
||||
const mails = [...byMessage.values()].slice(0, maxMails)
|
||||
// Everything read from here on is released in the finally below: mail bodies
|
||||
// are read to extract fields and must not outlive this run.
|
||||
const toReview = mails.map((c) => ({
|
||||
messageId: c.messageId,
|
||||
mailbox: c.mailbox,
|
||||
@@ -531,20 +616,16 @@ async function harvestReceiptsFromMail(
|
||||
worthFetching(doc, searchable, retrievedBy.get(doc.messageId) ?? []),
|
||||
)
|
||||
|
||||
const claimedFiles = new Set<string>()
|
||||
// Seeded with what is already filed, so a press spends its budget on
|
||||
// documents the company does not have rather than refetching its own.
|
||||
const claimedFiles = await fetchAlreadyHeld(supabase, companyId)
|
||||
for (const doc of wanted) {
|
||||
const candidate = byMessage.get(doc.messageId)
|
||||
if (!candidate) continue
|
||||
|
||||
// The same invoice arrives as an original, a reminder and two forwards,
|
||||
// every one carrying the identical attachment, so the filename alone
|
||||
// collapses those four into one fetch.
|
||||
//
|
||||
// Scoped by vendor as well, because a bare filename is not an identity:
|
||||
// "invoice.pdf" and "Faktura.pdf" are what half the world's billing systems
|
||||
// call their attachment, and keying on the filename alone would silently
|
||||
// drop a second supplier's invoice as a duplicate of the first.
|
||||
const fileKey = `${(doc.vendor ?? '').toLowerCase()}::${(doc.attachmentName ?? doc.messageId).toLowerCase()}`
|
||||
// One underlag per purchase: see receiptIdentity for why neither the
|
||||
// filename nor the message identifies anything here.
|
||||
const fileKey = receiptIdentity(doc)
|
||||
if (claimedFiles.has(fileKey)) continue
|
||||
claimedFiles.add(fileKey)
|
||||
summary.withCandidates++
|
||||
@@ -563,16 +644,22 @@ async function harvestReceiptsFromMail(
|
||||
// provkörning is that no mailbox content is copied anywhere.
|
||||
if (dryRun || !userId) continue
|
||||
if (candidate.attachmentIds.length === 0) continue
|
||||
if (summary.ingested >= MAX_RECEIPTS_PER_RUN) break
|
||||
if (summary.ingested >= maxReceipts) break
|
||||
|
||||
const names = candidate.attachmentNames ?? []
|
||||
const at = doc.attachmentName ? names.indexOf(doc.attachmentName) : 0
|
||||
const index = at >= 0 ? at : 0
|
||||
const ingested = await ingestMailCandidate(supabase, companyId, userId, {
|
||||
...candidate,
|
||||
attachmentIds: [candidate.attachmentIds[index] ?? candidate.attachmentIds[0]],
|
||||
attachmentNames: [names[index] ?? names[0] ?? ''],
|
||||
})
|
||||
const ingested = await ingestMailCandidate(
|
||||
supabase,
|
||||
companyId,
|
||||
userId,
|
||||
{
|
||||
...candidate,
|
||||
attachmentIds: [candidate.attachmentIds[index] ?? candidate.attachmentIds[0]],
|
||||
attachmentNames: [names[index] ?? names[0] ?? ''],
|
||||
},
|
||||
fileKey,
|
||||
)
|
||||
if (ingested) summary.ingested++
|
||||
}
|
||||
return summary
|
||||
|
||||
@@ -104,6 +104,15 @@ export async function ingestMailCandidate(
|
||||
companyId: string,
|
||||
userId: string,
|
||||
candidate: MailCandidate,
|
||||
/**
|
||||
* What this document is paperwork for, as the reading model saw it.
|
||||
*
|
||||
* Stored so a later run compares like with like. Deriving it again from the
|
||||
* extraction would compare the model's "Norwegian" against the PDF's
|
||||
* "Norwegian Air Shuttle AOC AS" and conclude they are two suppliers, which
|
||||
* is how an already-held receipt got fetched a second time.
|
||||
*/
|
||||
receiptIdentity?: string,
|
||||
): Promise<IngestedReceipt | null> {
|
||||
if (candidate.attachmentIds.length === 0) return null
|
||||
|
||||
@@ -184,7 +193,10 @@ export async function ingestMailCandidate(
|
||||
email_subject: candidate.subject,
|
||||
email_received_at: candidate.receivedAt,
|
||||
extracted_data: extracted ?? null,
|
||||
channel_context: buildChannelContext(candidate, attachmentId),
|
||||
channel_context: {
|
||||
...buildChannelContext(candidate, attachmentId),
|
||||
...(receiptIdentity ? { receipt_identity: receiptIdentity } : {}),
|
||||
},
|
||||
})
|
||||
.select('id')
|
||||
.single()
|
||||
|
||||
@@ -13,7 +13,10 @@
|
||||
* without a database; the caller owns the reads and the staging write.
|
||||
*/
|
||||
import { scoreUnderlagCandidates, type CandidateTransaction } from '@/lib/agent-context/underlag-candidates'
|
||||
import { calculateMerchantSimilarity } from '@/lib/documents/core-receipt-matcher'
|
||||
import {
|
||||
calculateMerchantSimilarity,
|
||||
normalizeForMatch,
|
||||
} from '@/lib/documents/core-receipt-matcher'
|
||||
|
||||
/**
|
||||
* Confidence a candidate must reach to be proposed unattended.
|
||||
@@ -291,3 +294,38 @@ export function worthFetching(
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
/**
|
||||
* What identifies one purchase's paperwork, for deciding whether to fetch it.
|
||||
*
|
||||
* A mail carries the invoice and the receipt for the same purchase under
|
||||
* different names, and the same receipt reaches a second mailbox on another
|
||||
* message, so neither the filename nor the message id identifies anything.
|
||||
* The vendor, the total and the date do.
|
||||
*
|
||||
* The date is what keeps a subscription working. Anthropic bills the same
|
||||
* amount every month, and without a date every month after the first would be
|
||||
* treated as a duplicate of it and suppressed forever: a worse failure than
|
||||
* the duplicates this key exists to prevent, because it is permanent and
|
||||
* silent. Two documents for one purchase share a date; June and July do not.
|
||||
*
|
||||
* Without a vendor there is nothing to anchor an amount to, so those fall back
|
||||
* to the message and the file. The filename alone is not enough: half the
|
||||
* world's billing systems attach "invoice.pdf".
|
||||
*/
|
||||
export function receiptIdentity(doc: {
|
||||
vendor: string | null
|
||||
amount: number | null
|
||||
currency: string | null
|
||||
date?: string | null
|
||||
attachmentName?: string | null
|
||||
messageId?: string | null
|
||||
}): string {
|
||||
const vendor = normalizeForMatch(doc.vendor ?? '').trim()
|
||||
if (vendor && doc.amount != null) {
|
||||
// Öre, not floating point: 0.1 + 0.2 must not read as a different total.
|
||||
const amount = Math.round(doc.amount * 100) / 100
|
||||
return `v1::${vendor}::${amount}::${(doc.currency ?? 'SEK').toLowerCase()}::${doc.date ?? ''}`
|
||||
}
|
||||
return `v1::file::${(doc.messageId ?? '').toLowerCase()}::${(doc.attachmentName ?? '').toLowerCase()}`
|
||||
}
|
||||
|
||||
@@ -324,6 +324,14 @@
|
||||
"sign_in_again": "Sign in again"
|
||||
},
|
||||
"mail": {
|
||||
"hunt_title": "Look for receipts",
|
||||
"hunt_help": "We search the connected mailboxes for receipts and invoices belonging to purchases without one, read the amount from the file, and put the matches in Review. Nothing is posted.",
|
||||
"hunt_row": "Search the mailboxes",
|
||||
"hunt_action": "Look now",
|
||||
"hunt_running": "Looking…",
|
||||
"hunt_found": "{count} receipts fetched. {left} purchases left to search.",
|
||||
"hunt_none": "No new receipts this time. {left} purchases left to search.",
|
||||
"hunt_failed": "The search could not be completed. Try again.",
|
||||
"connected": "Connected mailboxes",
|
||||
"help": "Each mailbox is connected by its own owner. We can never connect a colleague's mailbox for them: send an invitation instead.",
|
||||
"none_label": "None connected",
|
||||
|
||||
@@ -324,6 +324,14 @@
|
||||
"sign_in_again": "Logga in igen"
|
||||
},
|
||||
"mail": {
|
||||
"hunt_title": "Leta efter underlag",
|
||||
"hunt_help": "Vi söker i de kopplade brevlådorna efter kvitton och fakturor till köp som saknar underlag, läser beloppet ur filen och lägger fram förslagen i Granskning. Inget bokförs.",
|
||||
"hunt_row": "Sök igenom brevlådorna",
|
||||
"hunt_action": "Leta nu",
|
||||
"hunt_running": "Letar…",
|
||||
"hunt_found": "{count} underlag hämtade. {left} köp kvar att söka igenom.",
|
||||
"hunt_none": "Inga nya underlag den här gången. {left} köp kvar att söka igenom.",
|
||||
"hunt_failed": "Sökningen gick inte att slutföra. Försök igen.",
|
||||
"connected": "Kopplade brevlådor",
|
||||
"help": "Varje brevlåda kopplas av sin egen ägare. Vi kan aldrig koppla en kollegas brevlåda åt dem: skicka en inbjudan i stället.",
|
||||
"none_label": "Ingen kopplad",
|
||||
|
||||
Reference in New Issue
Block a user