feat(receipt-hunt): find receipts in connected mailboxes and pair them on the amount (#1492)
* feat(receipt-hunt): nightly matcher pairing unbooked purchases with held receipts Stages an attach_document_to_transaction proposal for every unbooked card purchase whose receipt the company already holds, so the underlag is attached before the transaction is booked and the gap never forms. When the user later books it, categorize-core.ts propagates the document onto the new verifikat through the matched_transaction_id link the executor writes. Deliberately scoped to UNBOOKED transactions. The posted-verifikat backlog is 96% imported history whose originals live in the previous system, so it stays a pull (the verifikat_missing_document worklist) rather than a nightly push. Ranking reuses scoreUnderlagCandidates; the pool is loaded once per company instead of per transaction, which removes both the N+1 and the newest-50 truncation a per-transaction lookup imposes on a deep backlog. Five guards, each mutation-tested: a confidence floor above the shared candidate floor, an ambiguity margin so two equally-good receipts are left to the picker rather than coin-flipped, one-receipt-one-purchase, one live proposal per purchase, and permanent suppression of pairs a human rejected. Suppression is derived from pending_operations history rather than a new table: terminal rows are immutable and a rejection is already the durable "no". Runs 05:30 UTC, after the 05:00 bank sync. Gated on RECEIPT_HUNT_COMPANY_IDS, which hunts nobody when unset so enabling it stays a deliberate act. No migration, no journal writes, no UI: proposals land in the existing Granskning queue. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(receipt-hunt): dry-run mode for provkörning against a real ledger Returns the pairings a run would stage without writing any of them, so a company can see tonight's proposals before they reach the granskningskö and so the matcher can be validated against production data without staging an operation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(matching): fold Swedish bank descriptors so receipts reach their purchases calculateMerchantSimilarity compared raw bank descriptors, so a receipt from "Alviks kött och fisk" scored 0.125 against the bank's own row for it, "Alviks koett och fisk K3667 Kortköp/uttag" — an öre-exact pair no threshold could reach. Adds normalizeForMatch, used for similarity only, which folds what the card rails add and never changes identity: the K#### token, Kortköp/uttag verbs, a leading "Kortköp YYMMDD", trailing /YY-MM-DD dates, reference numbers glued to the name, domain wrappers, legal forms, and the three ways banks mangle Swedish letters (ö, transliterated "oe", and ?? mojibake). Processor markers become spaces because the merchant sits before the star in GOOGLE*PLAY and after it in K*IKEA GALLE. Token-subset containment is scored level with substring containment so a receipt's legal name matches the bank's trading name. normalizeMerchantName is left byte-identical and now documents why: it is a transitive input to categorization_templates.counterparty_name, a persisted UNIQUE key with a hand-written SQL mirror the ledger-context RPC recomputes at query time. Changing it would make stored keys stop equalling computed ones, so the konteringskarta join misses and insertOrUpdateTemplate inserts a second row per merchant instead of migrating the occurrence counts. Aggressive folding is safe because it is applied to both sides of every comparison, so an over-eager fold still matches; the risk is collision between different merchants, which the new tests guard. Measured on 27 receipt/transaction pairs humans actually confirmed in production: recall 27/27, and 0/7 false positives on deliberately similar but distinct merchants. Full unit suite unchanged (13,004 passing), including the 22 string pins on the frozen key path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(mail): read-only Gmail connector so receipts are found without forwarding Forwarding was the only way a receipt reached Accounted, and it is both unpopular (97% of companies with the problem have never used their inbox address) and fragile: Arcim's own forward has been off for weeks and nobody noticed. This lets the hunt look in the mailbox instead. Scope is gmail.readonly and nothing else. It can search and download attachment bytes, and it structurally cannot send, modify or delete: the promise the consent screen makes is enforced by the grant, not by our code being careful. The consequence is deliberate: the agent can prepare a forward for a portal-link receipt but can never send one itself. Query-then-classify, never sync. For each unexplained purchase we run a provider-side search in a -3/+10 day window, pull metadata for a handful of hits, and keep nothing. No mailbox is mirrored and no message body is stored, which is what keeps this inside Google's Limited Use terms and GDPR data minimisation. Mail is searched only for purchases Underlag could not already explain, so a receipt we already hold never costs a mailbox read. The query ORs merchant against amount rather than requiring both: demanding both misses every rebrand and reseller (Anthropic bills as Claude), while the amount alone is a strong filter inside two weeks. mail_connections is service-role only with RLS enabled and zero policies, because the row holds a live refresh token and RLS cannot hide a column. Uniqueness is (company, provider, address) so a second mailbox is additive and a reconnect updates in place. Tokens are AES-256-GCM under their own key by preference, since a mail grant reads correspondence rather than backups. Core reaches the extension through a registered service, mirroring lib/email/service.ts, so lib/receipt-hunt never imports from @/extensions and a zero-extension build still compiles. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(mail): connect UI and ingest, making the hunt reach into the mailbox Two halves that together make the connector usable. Ingest (lib/receipt-hunt/ingest.ts, core): fetches the attachment, files it as a document and an inbox item with source 'mail_hunt', then stages the pairing. It lives in core because it writes documents and inbox items, and an extension may never import another extension; the mail extension only ever hands over bytes. No re-matching for a hunted receipt: it was fetched WHILE SEARCHING for a specific purchase, so the pairing is known by construction. The search is a deliberately broad OR query, which is exactly why the proposal still goes to a human with the mailbox, sender and subject written on it rather than being linked automatically. Provenance goes in channel_context, never extracted_data, because retrying extraction overwrites extracted_data wholesale and the record of which mailbox a receipt came from has to survive that. A partial unique index on (company_id, channel_context->>'mail_message_id') makes re-runs and the same receipt arriving in two mailboxes idempotent, and a 23505 is treated as success rather than an error. Guards, both mutation-tested: a duplicate message costs no provider call, and an oversized attachment is skipped rather than stored. One unreadable attachment falls through to the next and never aborts a night's hunt. UI: /settings/mail lists connected mailboxes with their health, connects a new one through a user-gesture tab (opened before the await, so popup blockers do not eat it), and disconnects behind a ConfirmDialog that states the outcome up front, including that already-approved receipts stay because they belong to the bookkeeping now. Strings in sv and en; the read-only promise is spelled out on the page rather than buried in a consent screen. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(mail): renumber migrations to clear a version collision on main 20260806150000 was already taken by preserve_preset_committed_at, and woocommerce_connections plus enforce_balance_on_posted_insert landed after this branch was cut. Two files sharing a version breaks every fresh database, which only shows up on a clean setup rather than on an already-migrated one. Applied to prod under the new versions (20260807090000 / 20260807090100), so schema_migrations matches these filenames exactly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(receipt-hunt): make the mailbox search actually able to find an underlag A provkörning against a real ledger returned the same seven unrelated messages for every purchase, all reporting no attachments. Three separate causes, each fixed and pinned: 1. `getMessageSummary` asked Gmail for `format=metadata`, which returns headers and omits `payload.parts` entirely. Every message therefore looked attachment-free, `bodyIsReceipt` was always true, and the `found.find(c => c.attachmentIds.length > 0)` guard in the hunt could never select anything: the feature could not file a single receipt. Gmail has no format that returns MIME structure without the body, so the body now comes down the wire; it is read for nothing and stored nowhere. 2. The bank's description is not a merchant name. "Lön Juli Jakob Överföring via internet" searched for "Juli" and matched most of the mailbox. Month names and payment-rail boilerplate are now stopwords. 3. Salary and tax runs are a company's largest outgoing rows, so they consumed the whole search budget hunting receipts that cannot exist. `canHaveEmailReceipt` skips them for the mail leg only. Deliberately narrow: a supplier invoice paid over bankgiro does arrive by mail, and an "Utlägg" reimbursement has a real receipt behind it. Measured on the same ledger: 22 hits, 0 with attachments, 0 ingestable -> 4 hits, all with attachments, 3 of 4 correct (Elgiganten, Sting, Anthropic). The fourth matched a Stockholm billing address, which is why every proposal still waits for a human. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(receipt-hunt): let a model resolve merchants and pick the receipt The keyword hunt was failing for reasons regex tuning cannot reach, all measured against a real mailbox rather than assumed: - `from:anthropic.com` returns 0. Receipts arrive here by being forwarded, so the sender is the user, not the vendor. - The exact charged amount returns 0. The bank posts a converted SEK figure that appears nowhere in a USD receipt. - A date window around the purchase returns 0, while the same merchant search without one returns 10+. A forward is stamped when it was forwarded, sometimes months later. So the query now searches merchant names across the whole mailbox, and precision is restored by judgement rather than by syntax. Two model calls per run, both through forced tool use so the reply is a shape and not prose to be parsed: 1. `planMerchantGroups` resolves bank descriptors to merchants and merges repeats. Six Anthropic subscriptions become one search and one decision instead of six of each. 2. `assignReceipts` decides which mail, and which attachment on it, is the receipt for which charge, and says why in a sentence the reviewer reads. The attachment, not the message, is the unit of an underlag: a single forward routinely carries receipts for several purchases ("Fwd: Kvitton februari" has five). Migration 20260807103000 moves the dedupe key from message to message+attachment, with a backfill, because the old index would have silently blocked every receipt after the first in a forward. The model may not produce any number that reaches the ledger. It returns ids, a confidence and a reason; amounts, dates and the write stay in deterministic code. Its answer is validated, not trusted: an unknown message id, an invented filename or a low confidence drops the pairing, and any failed call proposes nothing at all. Every result still waits for a human. Measured on the same ledger: 0 receipts that could ever be filed -> 3 correct pairings (Elgiganten, Sting office invoice, Anthropic), each with a stated reason. The five remaining Anthropic charges are dated after 2026-06-15, when forwarding to the connected mailbox stopped; the model declined them correctly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(receipt-hunt): amount first, and drop the confidence scoring Three findings from how others build this, applied. Production email search (Superhuman, Haystack 2026) reports that recall comes from loosening retrieval and letting the model filter downstream, not from tightening the query. Retrieval depth per merchant 12 -> 25, and purchases the planner cannot name a merchant for are now searched by amount alone instead of skipped: a line like "1260525758758 Europabetalning" identifies no merchant but is a real supplier payment whose invoice may carry exactly that total. Reconciliation engines weight amount far above date (Midday: 35% vs 5%) because banks post late while amounts do not drift. The Gmail query now leads with the amount and ORs the merchant, rather than dropping the amount whenever a merchant alias exists. Still an OR: a receipt billed in USD never contains the SEK figure the bank charged. The confidence score is gone entirely. Research on verbalised confidence finds it badly calibrated, clustered on round-number anchors and barely better than chance at separating a model's own right answers from its wrong ones. That matched what this ran into: the model anchored on 0.6 / 0.7 / 0.75 / 0.9, and the 0.7 threshold discarded two correct pairings. It is replaced by an observation rather than a self-assessment, whether the charged amount is actually visible in the mail, which is what a reviewer checks first and what sorts the queue. Also fixes a real defect the run exposed: the one-file-one-purchase guard only held within a merchant group, so when the planner split one landlord into "Sting" and "Kontorsplatser" both 15 000 kr charges were assigned the same invoice. A file is now claimed once per run, which is the duplicate underlag BFL forbids. Measured on the same ledger: 3 -> 5 pairings, no duplicate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(receipt-hunt): harvest receipts, then pair them on the amount Splits the mailbox leg in two along the line of what each side can actually know. The model was being asked which purchase a mail belonged to. Deciding that needs the amount; the amount lives inside the PDF; a Gmail preview essentially never shows it. Measured over a real mailbox, every single pairing came back "belopp ej synligt": it was answering without the deciding evidence, which is why it declined five of six repeat subscriptions and why two correct pairings sat just under a threshold. Now it answers only what a subject, a sender and a preview line support: is this mail an underlag, and which attachment is it. Then the receipt is fetched, the extraction that already runs on document.uploaded reads its amount, date and vendor, and the pairing is the same deterministic amount-and-merchant match every other underlag goes through. Amount becomes decisive for real rather than as an instruction the model could not act on. The load-bearing fix is small: ingest now copies the extraction result onto the inbox item. The pool is read from invoice_inbox_items, so a hunted receipt with no extracted_data could never have matched anything, and the whole mail leg was quietly incapable of producing a pairing on amount. Consequences, all deliberate: - Harvesting runs BEFORE the pool is read, so a receipt found tonight is paired tonight rather than a night later. - One staging path instead of two. Mail-sourced proposals carry the same preview and confidence as every other, plus where they came from. - Deduped on the attachment filename, not on the message: the same invoice arrives as an original, a reminder and two forwards, and the old key filed "Invoice_13041840.pdf" four times over. - Capped at 8 receipts per merchant per run. Measured on the same ledger: 5 pairings attempted from thin evidence -> 16 real documents identified, each waiting on an amount it can be checked against. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(receipt-hunt): the model reads mail, arithmetic does the matching Collapses the mailbox leg to one model call that extracts fields, and hands every judgement back to deterministic code. Gone: resolving bank descriptors to merchant names, deciding which mail belongs to which charge, and the confidence score gating the result. Three prompts and two model calls become one, and mail-intelligence.ts drops from 450 lines to 250. What made this possible was measuring what a mail actually contains. The body was being downloaded and thrown away in favour of a 200-character snippet, and the body is where a forwarded receipt quotes its original sender and its original date. That is the purchase date, the thing whose absence forced the date window off entirely and made the old design miss five of six repeat subscriptions. It was there all along. So the model now answers only what text can support: is this an underlag, from whom, when, and for how much if the mail says so. Fields, not judgements. Everything after is arithmetic: - Retrieval is deterministic. No model decides what to search for. - Fetching is gated by worthFetching(): a stated amount is enough on its own, a vendor needs a plausible date, and a mail found by a purchase's own search is evidence in itself. That last rule is what handles a supplier the bank and the invoice name differently ("Kontorsplatser j BG" against "Stockholm Innovation & Growth AB"), which is what the deleted merchant-resolution call used to buy. - The pairing is the existing scorer, reached the same way as every other underlag: fetch, let the extraction that already runs on upload read the PDF, match on the amount. Amount is decisive in fact rather than as an instruction the model could not act on. Also adds the Swedish thousands-space amount formats to the query. Measured: the Sting invoice is findable as "15 000,00" and "15 000" and by no ungrouped form at all, so every amount search was missing them. Measured on the same ledger: 5 thin pairings -> 8 real documents, each with a vendor and a true purchase date, waiting on the amount in its own PDF. Currency is never converted to make a number agree. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(receipt-hunt): trust the bytes, not the mail, when filing an attachment Found by the first live run, which fetched nothing and reported success. Three defects, each invisible to a dry run because a dry run never downloads anything. 1. Gmail declares a forwarded PDF as application/octet-stream, and uploadDocument validates content against the declared type, so the upload was rejected: "Filinnehållet matchar inte den angivna filtypen". Every forwarded receipt with a generic MIME type would have failed this way, silently, since ingest swallows one bad attachment to protect the rest of the run. The type is now sniffed from the magic bytes, then the filename, and only then from what the mail claimed. 2. The filename was re-derived by a second full message fetch inside fetchAttachment, which came back empty and fell back to a generic "underlag.pdf", discarding the real "2332687551.pdf" the search had already reported. The known name now wins. 3. The provkörning script imported lib/init instead of calling ensureInitialized(), so document.uploaded reached no handler and nothing was ever extracted. It also used static imports, which are hoisted and ran before .env.local was read, leaving the extraction extension unable to build a Supabase client. Both are script defects, not product defects: the cron route calls ensureInitialized() at module level as the architecture requires. The script now loads the environment first and imports dynamically. Also makes the per-run fetch cap tunable (RECEIPT_HUNT_MAX_RECEIPTS) so a pilot can be held to a couple of documents, and adds --live to the script, which is the only way it writes anything. Verified end to end against a real ledger, every link exercised for the first time: two attachments fetched from Gmail, stored with their real names and types, extraction run on both, the amount copied onto the inbox item, and the deterministic matcher pairing Elgiganten 21 639,00 kr from the PDF against the -21 639 kr card purchase at 0.85, staged into Granskning as attach_document_to_transaction. The second document, a Bolagsverket filing receipt, carries no total and correctly paired with nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(receipt-hunt): sweep a whole mailbox, and stop lending one receipt twice A backfill on a real ledger, 22 documents fetched from 172 messages. Batches the extraction (25 mails per call) so a first run on an existing company can read the whole mailbox instead of the 40 mails one call can carry, and makes the per-run caps tunable (RECEIPT_HUNT_MAX_MAILS, RECEIPT_HUNT_MAX_RECEIPTS) so a pilot can be bounded. The nightly caps stay where they are: they pace the review queue, and a backlog is a different job from a nightly tick. Two defects the backfill exposed, neither reachable from a dry run: The one-receipt-one-purchase rule only held inside a single run. `spentDocumentIds` is per-invocation, so an H&M receipt was proposed against a -358 kr purchase on one pass and a -354 kr purchase on the next, and approving both would have put the same underlag on two verifikat. A live proposal now claims its document across runs, the same way it already claimed its transaction. A document reported with no filename, on a message carrying five attachments, was not an answer but a shrug: the caller fetched attachment number one and hoped. Those are dropped now. A body-only receipt, where there is nothing to choose between, still passes. Measured after the sweep: 21 of 22 documents read correctly, and the binding constraint on this ledger is no longer retrieval but currency. Ten receipts are in SEK and five of those pair on the amount; twelve are in USD or EUR, where the bank charged a converted figure that appears nowhere in the receipt, so no comparison is possible and none is attempted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(mail): show the provider's own mark on the mailbox settings page Someone connecting a mailbox is picking an account at a provider, and the provider's mark is how they recognise which one. A generic envelope glyph said "mail" when the question is "whose". The Google "G" already existed, drawn inline inside GoogleAuthButton for the sign-in flow. It moves to components/ui/provider-marks so there is one definition rather than two, and a Microsoft square joins it for the Graph connector. Both stay inline: no external host is contacted for an icon before anyone has agreed to anything. These are the only coloured glyphs in an achromatic interface, which is deliberate rather than an oversight. A brand mark is identity, not chrome, and Google's terms require its mark unaltered rather than tinted to match a palette. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(archive): drop the duplicate mail_connections exclusion left by the rebase Main added the table to ARCHIVE_EXCLUDED_TABLES while this branch was open, so rebasing produced the key twice and the zero-extension build failed to type check. Main's entry stays, in its alphabetical place, and keeps the sentence that answers the retention question: the grants are not räkenskapsinformation, but the receipts they find are archived as documents. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(mail): record who disconnected a mailbox, without keeping the token Raised by the compliance review: disconnect() hard-deleted the row with no trace, and which mailboxes feed underlag into the books is a control over how räkenskapsinformation is produced (BFNAR 2013:2 kap 8), so switching one off should be reconstructable years later. Written by hand rather than by the write_audit_log trigger the accounting tables use. That trigger copies the whole row into audit_log, which here would mean copying an encrypted refresh token into a second table and keeping it after the entire point of the delete was to destroy it. The sibling credential table shopify_connections omits the trigger for the same reason. Only the address and provider are recorded, pinned by a test that fails if a credential ever reaches the audit entry. The review's two other flags were checked rather than assumed: nothing purges mail_hunt documents, and categorize-core.ts:403 does carry the attached document onto the verifikat when the transaction is booked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(mail): bound every outbound call, and stop the token widening itself Four findings from the review, each checked against the code first. Neither the Gmail API nor Google's token endpoint had a deadline. Both are awaited inside Promise.all across mailboxes, so one stalled request held the whole company's hunt open until the platform killed the run. Both now carry a 15s AbortSignal, which turns a stall into one mailbox missing from tonight's sweep. `include_granted_scopes: 'true'` let Google fold scopes this app was granted elsewhere into the token issued for a mailbox, so a grant could carry more authority than the consent screen showed. Removed, and pinned by a test asserting the parameter is absent. disconnect() ignored both statement results: a failed delete still wrote an audit entry claiming the mailbox was disconnected while the credential was live, and a failed audit insert passed silently. The delete now throws, so the entry is never written for a delete that did not happen. The audit failure is logged rather than rolled back: the two can now only diverge one way, credential gone and note missing, and recreating a credential to keep them in step would be worse than a missing note. The fifth finding is real and stays open by choice, recorded in DECISIONS.md: the cron still passes searchMail=false. A sweep of one 172-message mailbox took over 600s against a maxDuration of 300, so enabling the mailbox leg nightly would time out mid-run. That flag and RECEIPT_HUNT_COMPANY_IDS get flipped together once the per-company budget is measured. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(receipt-hunt): file each attachment under its own identity Four more findings from the review. The first is a real defect. ingestMailCandidate loops over candidate.attachmentIds, but the dedupe key, the mail_attachment_id provenance and the filename were all read from index 0. Storing the second attachment therefore recorded the first one's key and name, which mislabels the row and, because the key is unique, permanently blocks the first attachment from ever landing. Masked today only because the hunt narrows to a single attachment before calling in, so nothing in the current path exercises it. All three now come from the attachment actually being stored, and the duplicate pre-check moved inside the loop so trying a second attachment is not suppressed by the first already being filed. Mutation-tested. The per-run fetch key was the bare filename, which is not an identity: "invoice.pdf" is what half the world's billing systems attach, so a second supplier's invoice would be dropped as a duplicate of the first. Scoped by vendor as well, keeping the behaviour it was written for, one fetch for an invoice that arrives as an original, a reminder and two forwards. Adds tests/pg/mail-hunt-file-dedupe.pg.test.ts for the new unique index: five attachments from one forward all land, the same attachment is refused twice, two companies hold the same file independently, other inbox sources are untouched by the partial predicate, and the message-scoped predecessor is gone. Written against CI's Postgres; there is no local DATABASE_URL here, so CI is what exercises it. --live now refuses unless RECEIPT_HUNT_CONFIRM names the same company. The script writes to whatever .env.local points at, which for this repo is production, and a recalled command should not be able to fire it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(test): cast the jsonb parameter so Postgres can type it pg-real could not determine the type of $3 inside jsonb_build_object. An explicit ::text is what the other pg tests do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jakob Wennberg
co-authored by
Claude Opus 5
Jakob Wennberg
Claude Opus 5
Jakob Wennberg
parent
6aab1fbe23
commit
38f5d9812e
@@ -0,0 +1,159 @@
|
||||
import { NextResponse } from 'next/server'
|
||||
import type { Extension } from '@/lib/extensions/types'
|
||||
import { registerMailSearchService } from '@/lib/mail-search/service'
|
||||
import { createServiceClientNoCookies } from '@/lib/auth/api-keys'
|
||||
import { GmailSearchService } from './lib/search-service'
|
||||
import { createOAuthState, verifyOAuthState } from './lib/crypto'
|
||||
import {
|
||||
buildAuthorizationUrl,
|
||||
exchangeCodeForTokens,
|
||||
getGoogleOAuthEnv,
|
||||
isGoogleMailConfigured,
|
||||
} from './lib/google-oauth'
|
||||
import { disconnect, listConnections, saveConnection } from './lib/connections'
|
||||
import { resolveCallbackOrigin } from './lib/callback-origin'
|
||||
|
||||
// Registered as soon as the extension loads, so the receipt hunt can search
|
||||
// mail without core ever importing from @/extensions.
|
||||
registerMailSearchService(new GmailSearchService())
|
||||
|
||||
function jsonError(message: string, status = 500): Response {
|
||||
return NextResponse.json({ error: message }, { status })
|
||||
}
|
||||
|
||||
/** How far back a newly connected mailbox may be searched, in days. */
|
||||
const BACKFILL_CHOICES = new Set([30, 90, 365])
|
||||
|
||||
export const mailExtension: Extension = {
|
||||
id: 'mail',
|
||||
name: 'Brevlådor',
|
||||
version: '0.1.0',
|
||||
sector: 'general',
|
||||
|
||||
settingsPanel: {
|
||||
label: 'Brevlådor',
|
||||
path: '/settings/mail',
|
||||
},
|
||||
|
||||
apiRoutes: [
|
||||
// Start the consent flow. Returns the URL rather than redirecting so the
|
||||
// caller can open it in a deliberate, user-gesture tab.
|
||||
{
|
||||
method: 'POST',
|
||||
path: '/oauth/start',
|
||||
handler: async (request, ctx) => {
|
||||
if (!ctx) return jsonError('Missing context', 500)
|
||||
if (!isGoogleMailConfigured()) return jsonError('provider_not_configured', 400)
|
||||
try {
|
||||
const url = new URL(request.url)
|
||||
const origin = resolveCallbackOrigin(url.origin)
|
||||
const state = createOAuthState(ctx.userId, ctx.companyId)
|
||||
const env = getGoogleOAuthEnv(origin)
|
||||
return NextResponse.json({ url: buildAuthorizationUrl(env, state) })
|
||||
} catch (err) {
|
||||
ctx.log.error('mail oauth start failed', err)
|
||||
return jsonError(err instanceof Error ? err.message : 'Could not start OAuth', 500)
|
||||
}
|
||||
},
|
||||
},
|
||||
|
||||
// Google redirects here after consent. Registered in the Google console as
|
||||
// an authorised redirect URI: the `mail` slug and this path are pinned and
|
||||
// must never be renamed without re-registering.
|
||||
{
|
||||
method: 'GET',
|
||||
path: '/oauth/callback',
|
||||
skipAuth: true,
|
||||
handler: async (request) => {
|
||||
const url = new URL(request.url)
|
||||
const code = url.searchParams.get('code')
|
||||
const state = url.searchParams.get('state')
|
||||
const error = url.searchParams.get('error')
|
||||
const settingsUrl = `${resolveCallbackOrigin(url.origin)}/settings/mail`
|
||||
|
||||
// The user declining is a normal outcome, not an error to shout about.
|
||||
if (error) return NextResponse.redirect(`${settingsUrl}?mail=denied`)
|
||||
if (!code || !state) return NextResponse.redirect(`${settingsUrl}?mail=invalid`)
|
||||
|
||||
const verified = verifyOAuthState(state)
|
||||
if (!verified) return NextResponse.redirect(`${settingsUrl}?mail=expired`)
|
||||
|
||||
try {
|
||||
const origin = resolveCallbackOrigin(url.origin)
|
||||
const env = getGoogleOAuthEnv(origin)
|
||||
const tokens = await exchangeCodeForTokens(env, code)
|
||||
if (!tokens.refreshToken) {
|
||||
return NextResponse.redirect(`${settingsUrl}?mail=no_refresh_token`)
|
||||
}
|
||||
if (!tokens.email) {
|
||||
// Without the address we cannot tell two grants apart, and the
|
||||
// unique key depends on it.
|
||||
return NextResponse.redirect(`${settingsUrl}?mail=no_address`)
|
||||
}
|
||||
|
||||
await saveConnection(createServiceClientNoCookies(), {
|
||||
companyId: verified.companyId,
|
||||
userId: verified.userId,
|
||||
provider: 'gmail',
|
||||
emailAddress: tokens.email,
|
||||
refreshToken: tokens.refreshToken,
|
||||
accessToken: tokens.accessToken,
|
||||
expiresAt: tokens.expiresAt,
|
||||
scopes: tokens.scopes,
|
||||
backfillFrom: null,
|
||||
})
|
||||
return NextResponse.redirect(`${settingsUrl}?mail=connected`)
|
||||
} catch {
|
||||
return NextResponse.redirect(`${settingsUrl}?mail=failed`)
|
||||
}
|
||||
},
|
||||
},
|
||||
|
||||
// What this company has connected. Safe projection only: never tokens.
|
||||
{
|
||||
method: 'GET',
|
||||
path: '/connections',
|
||||
handler: async (_request, ctx) => {
|
||||
if (!ctx) return jsonError('Missing context', 500)
|
||||
const connections = await listConnections(createServiceClientNoCookies(), ctx.companyId)
|
||||
return NextResponse.json({ data: { connections, configured: isGoogleMailConfigured() } })
|
||||
},
|
||||
},
|
||||
|
||||
{
|
||||
method: 'DELETE',
|
||||
path: '/connections',
|
||||
handler: async (request, ctx) => {
|
||||
if (!ctx) return jsonError('Missing context', 500)
|
||||
const id = new URL(request.url).searchParams.get('id')
|
||||
if (!id) return jsonError('missing_id', 400)
|
||||
await disconnect(createServiceClientNoCookies(), ctx.companyId, id, ctx.userId)
|
||||
return NextResponse.json({ data: { disconnected: true } })
|
||||
},
|
||||
},
|
||||
|
||||
// How far back a mailbox may be searched once, chosen by the user at
|
||||
// connect time. Bounded to the offered choices so an arbitrary date cannot
|
||||
// widen the grant's reach by hand.
|
||||
{
|
||||
method: 'POST',
|
||||
path: '/connections/backfill',
|
||||
handler: async (request, ctx) => {
|
||||
if (!ctx) return jsonError('Missing context', 500)
|
||||
const body = (await request.json().catch(() => ({}))) as { id?: string; days?: number }
|
||||
if (!body.id || !body.days || !BACKFILL_CHOICES.has(body.days)) {
|
||||
return jsonError('invalid_request', 400)
|
||||
}
|
||||
const from = new Date()
|
||||
from.setDate(from.getDate() - body.days)
|
||||
const supabase = createServiceClientNoCookies()
|
||||
await supabase
|
||||
.from('mail_connections')
|
||||
.update({ backfill_from: from.toISOString().slice(0, 10) })
|
||||
.eq('id', body.id)
|
||||
.eq('company_id', ctx.companyId)
|
||||
return NextResponse.json({ data: { backfill_from: from.toISOString().slice(0, 10) } })
|
||||
},
|
||||
},
|
||||
],
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
/**
|
||||
* Disconnecting a mailbox is a control change over how underlag reaches the
|
||||
* books, so it has to be reconstructable (BFNAR 2013:2 kap 8). What must NOT
|
||||
* happen is the audit entry preserving the credential the delete existed to
|
||||
* destroy.
|
||||
*/
|
||||
import { describe, it, expect, vi, beforeEach } from 'vitest'
|
||||
import { disconnect } from '../connections'
|
||||
|
||||
function mockSupabase(
|
||||
existing: Record<string, unknown> | null,
|
||||
deleteError: { message: string } | null = null,
|
||||
) {
|
||||
const inserted: Array<Record<string, unknown>> = []
|
||||
const deleted: string[] = []
|
||||
const client = {
|
||||
from(table: string) {
|
||||
const chain: Record<string, unknown> = {}
|
||||
for (const m of ['select']) chain[m] = vi.fn(() => chain)
|
||||
let eqCalls = 0
|
||||
chain.eq = vi.fn(() => {
|
||||
eqCalls++
|
||||
// The delete chain resolves after its second .eq(); the select chain
|
||||
// ends in .maybeSingle() instead.
|
||||
return chain.deleting && eqCalls >= 2
|
||||
? Promise.resolve({ error: deleteError })
|
||||
: chain
|
||||
})
|
||||
chain.maybeSingle = vi.fn(() => Promise.resolve({ data: existing, error: null }))
|
||||
chain.delete = vi.fn(() => {
|
||||
deleted.push(table)
|
||||
;(chain as Record<string, unknown>).deleting = true
|
||||
eqCalls = 0
|
||||
return chain
|
||||
})
|
||||
chain.insert = vi.fn((row: Record<string, unknown>) => {
|
||||
if (table === 'audit_log') inserted.push(row)
|
||||
return Promise.resolve({ error: null })
|
||||
})
|
||||
return chain
|
||||
},
|
||||
}
|
||||
return { client: client as never, inserted, deleted }
|
||||
}
|
||||
|
||||
beforeEach(() => vi.clearAllMocks())
|
||||
|
||||
describe('disconnect', () => {
|
||||
it('records who disconnected which mailbox, and when', async () => {
|
||||
const { client, inserted, deleted } = mockSupabase({
|
||||
email_address: 'ekonomi@nordvik.se',
|
||||
provider: 'gmail',
|
||||
})
|
||||
await disconnect(client, 'co-1', 'conn-1', 'user-1')
|
||||
|
||||
expect(deleted).toContain('mail_connections')
|
||||
expect(inserted).toHaveLength(1)
|
||||
expect(inserted[0]).toMatchObject({
|
||||
user_id: 'user-1',
|
||||
company_id: 'co-1',
|
||||
action: 'DELETE',
|
||||
table_name: 'mail_connections',
|
||||
record_id: 'conn-1',
|
||||
})
|
||||
expect(String(inserted[0].description)).toContain('ekonomi@nordvik.se')
|
||||
})
|
||||
|
||||
it('never copies the credential into the audit trail', async () => {
|
||||
// The whole point of disconnecting is that the refresh token is gone.
|
||||
// A write_audit_log trigger would have carried it into a second table.
|
||||
const { client, inserted } = mockSupabase({
|
||||
email_address: 'ekonomi@nordvik.se',
|
||||
provider: 'gmail',
|
||||
encrypted_refresh_token: 'SECRET',
|
||||
})
|
||||
await disconnect(client, 'co-1', 'conn-1', 'user-1')
|
||||
|
||||
const blob = JSON.stringify(inserted[0])
|
||||
expect(blob).not.toContain('SECRET')
|
||||
expect(blob).not.toContain('encrypted_refresh_token')
|
||||
})
|
||||
|
||||
it('does not claim a disconnect that the database refused', async () => {
|
||||
// An audit entry saying the mailbox was disconnected, while the credential
|
||||
// is still live, is worse than no entry at all.
|
||||
const { client, inserted } = mockSupabase(
|
||||
{ email_address: 'ekonomi@nordvik.se', provider: 'gmail' },
|
||||
{ message: 'permission denied' },
|
||||
)
|
||||
await expect(disconnect(client, 'co-1', 'conn-1', 'user-1')).rejects.toThrow('permission denied')
|
||||
expect(inserted).toEqual([])
|
||||
})
|
||||
|
||||
it('writes nothing when there was no such connection', async () => {
|
||||
const { client, inserted } = mockSupabase(null)
|
||||
await disconnect(client, 'co-1', 'missing', 'user-1')
|
||||
expect(inserted).toEqual([])
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,102 @@
|
||||
/**
|
||||
* Reading a message well enough to know whether it carries an underlag.
|
||||
*
|
||||
* The case that matters is the one a provkörning caught: asking Gmail for
|
||||
* `format=metadata` returns headers and no `payload.parts`, so every message
|
||||
* looks attachment-free and the hunt can never file anything. These tests pin
|
||||
* the format and the MIME walk.
|
||||
*/
|
||||
import { describe, it, expect, vi, beforeEach } from 'vitest'
|
||||
import { getMessageSummary } from '../gmail-client'
|
||||
|
||||
const mockFetch = vi.fn()
|
||||
vi.stubGlobal('fetch', (...args: unknown[]) => mockFetch(...args))
|
||||
|
||||
function respond(message: Record<string, unknown>) {
|
||||
mockFetch.mockResolvedValue({
|
||||
ok: true,
|
||||
json: () => Promise.resolve(message),
|
||||
text: () => Promise.resolve(''),
|
||||
})
|
||||
}
|
||||
|
||||
const HEADERS = [
|
||||
{ name: 'Subject', value: 'Faktura-20251070' },
|
||||
{ name: 'From', value: 'info@tic.io' },
|
||||
]
|
||||
|
||||
beforeEach(() => vi.clearAllMocks())
|
||||
|
||||
describe('getMessageSummary', () => {
|
||||
it('asks for the full message, because metadata omits the parts tree', async () => {
|
||||
respond({ id: 'm1', payload: { headers: HEADERS } })
|
||||
await getMessageSummary('token', 'm1', 'conn-1', 'invoice@arcim.io')
|
||||
|
||||
const url = String(mockFetch.mock.calls[0][0])
|
||||
expect(url).toContain('format=full')
|
||||
// The bug this replaces: metadata returns no payload.parts at all.
|
||||
expect(url).not.toContain('format=metadata')
|
||||
})
|
||||
|
||||
it('finds a PDF nested inside a forwarded message', async () => {
|
||||
// Forwarding is how most of these receipts arrive, and it buries the
|
||||
// attachment two levels down inside a message/rfc822 part.
|
||||
respond({
|
||||
id: 'm1',
|
||||
payload: {
|
||||
headers: HEADERS,
|
||||
mimeType: 'multipart/mixed',
|
||||
parts: [
|
||||
{ mimeType: 'text/plain', body: { size: 12 } },
|
||||
{
|
||||
mimeType: 'message/rfc822',
|
||||
parts: [
|
||||
{
|
||||
mimeType: 'multipart/mixed',
|
||||
parts: [
|
||||
{ mimeType: 'text/html', body: { size: 900 } },
|
||||
{
|
||||
mimeType: 'application/pdf',
|
||||
filename: 'faktura.pdf',
|
||||
body: { attachmentId: 'att-deep', size: 51200 },
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
})
|
||||
|
||||
const candidate = await getMessageSummary('token', 'm1', 'conn-1', 'invoice@arcim.io')
|
||||
expect(candidate.attachmentIds).toEqual(['att-deep'])
|
||||
expect(candidate.bodyIsReceipt).toBe(false)
|
||||
})
|
||||
|
||||
it('does not mistake an inline logo for an underlag', async () => {
|
||||
respond({
|
||||
id: 'm1',
|
||||
payload: {
|
||||
headers: HEADERS,
|
||||
parts: [
|
||||
{ mimeType: 'image/png', filename: 'logo.png', body: { attachmentId: 'logo', size: 400 } },
|
||||
],
|
||||
},
|
||||
})
|
||||
|
||||
const candidate = await getMessageSummary('token', 'm1', 'conn-1', 'invoice@arcim.io')
|
||||
expect(candidate.attachmentIds).toEqual([])
|
||||
expect(candidate.bodyIsReceipt).toBe(true)
|
||||
})
|
||||
|
||||
it('reports a genuinely attachment-free mail as a body receipt', async () => {
|
||||
respond({
|
||||
id: 'm1',
|
||||
payload: { headers: HEADERS, mimeType: 'text/html', body: { size: 4000 } },
|
||||
})
|
||||
|
||||
const candidate = await getMessageSummary('token', 'm1', 'conn-1', 'invoice@arcim.io')
|
||||
expect(candidate.bodyIsReceipt).toBe(true)
|
||||
expect(candidate.subject).toBe('Faktura-20251070')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,135 @@
|
||||
/**
|
||||
* The Gmail query is where recall is won or lost, so it is pure and tested
|
||||
* without a mailbox. Cases are drawn from the real bank descriptors and
|
||||
* confirmed receipt pairs observed in production.
|
||||
*/
|
||||
import { describe, it, expect } from 'vitest'
|
||||
import {
|
||||
DAYS_AFTER,
|
||||
DAYS_BEFORE,
|
||||
amountTerms,
|
||||
buildGmailQuery,
|
||||
looksLikeReceipt,
|
||||
merchantTerms,
|
||||
} from '../gmail-query'
|
||||
|
||||
describe('merchantTerms', () => {
|
||||
it('drops legal forms and other tokens that would match the whole mailbox', () => {
|
||||
expect(merchantTerms('circle k')).toEqual(['circle'])
|
||||
expect(merchantTerms('ryde sweden')).toEqual(['ryde'])
|
||||
expect(merchantTerms('')).toEqual([])
|
||||
expect(merchantTerms(null)).toEqual([])
|
||||
})
|
||||
|
||||
it('keeps at most three terms so the query stays selective', () => {
|
||||
expect(merchantTerms('alviks kott och fisk stockholm sodermalm').length).toBeLessThanOrEqual(3)
|
||||
})
|
||||
})
|
||||
|
||||
describe('amountTerms', () => {
|
||||
it('offers both Swedish and international decimal forms', () => {
|
||||
// A Swedish receipt writes 438,75; a SaaS invoice writes 438.75.
|
||||
expect(amountTerms(438.75)).toEqual(expect.arrayContaining(['438.75', '438,75']))
|
||||
})
|
||||
|
||||
it('adds the bare integer when the amount is whole', () => {
|
||||
expect(amountTerms(425)).toEqual(expect.arrayContaining(['425.00', '425,00', '425']))
|
||||
})
|
||||
|
||||
it('ignores the sign, since bank outflows are negative', () => {
|
||||
expect(amountTerms(-425)).toEqual(amountTerms(425))
|
||||
})
|
||||
})
|
||||
|
||||
describe('buildGmailQuery', () => {
|
||||
const base = { merchant: 'circle', amount: 438.75, currency: 'SEK', date: '2026-05-02' }
|
||||
|
||||
it('brackets the purchase with an asymmetric window', () => {
|
||||
const q = buildGmailQuery(base)
|
||||
// Receipts arrive at or after the purchase; the bank may post it late.
|
||||
expect(DAYS_AFTER).toBeGreaterThan(DAYS_BEFORE)
|
||||
expect(q).toContain('after:2026/04/28')
|
||||
expect(q).toContain('before:2026/05/13')
|
||||
})
|
||||
|
||||
it('leads with the amount, then ORs the merchant', () => {
|
||||
// Amount first because it is the one signal that does not drift: banks post
|
||||
// late and receipts get forwarded, so dates move, but the charged figure
|
||||
// does not. Still an OR and never an AND: a receipt billed in USD contains
|
||||
// no SEK amount, and a bank descriptor often names nothing in the receipt.
|
||||
const q = buildGmailQuery(base)
|
||||
expect(q).toMatch(/\("438\.75" OR .*"circle"/)
|
||||
})
|
||||
|
||||
it('still searches on amount alone when the merchant is unusable', () => {
|
||||
const q = buildGmailQuery({ ...base, merchant: 'ab' })
|
||||
expect(q).toContain('"438.75"')
|
||||
expect(q).not.toContain('"ab"')
|
||||
})
|
||||
|
||||
it('does not require an attachment, because many receipts are the body', () => {
|
||||
expect(buildGmailQuery(base)).not.toContain('has:attachment')
|
||||
})
|
||||
})
|
||||
|
||||
describe('looksLikeReceipt', () => {
|
||||
it('keeps plausible receipts', () => {
|
||||
expect(looksLikeReceipt('Ditt kvitto från Circle K', 'no-reply@circlek.se')).toBe(true)
|
||||
expect(looksLikeReceipt('Your receipt from Anthropic', 'billing@anthropic.com')).toBe(true)
|
||||
})
|
||||
|
||||
it('drops the obvious non-receipts before anything expensive reads them', () => {
|
||||
expect(looksLikeReceipt('Nyhetsbrev maj', 'news@example.com')).toBe(false)
|
||||
expect(looksLikeReceipt('Calendar invite: standup', 'cal@example.com')).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
describe('buildAuthorizationUrl', () => {
|
||||
it('carries the CSRF state, which the callback refuses to proceed without', async () => {
|
||||
const { buildAuthorizationUrl } = await import('../google-oauth')
|
||||
const url = buildAuthorizationUrl(
|
||||
{ clientId: 'cid', clientSecret: 'sec', redirectUri: 'https://app.accounted.se/cb' },
|
||||
'signed-state',
|
||||
)
|
||||
const params = new URL(url).searchParams
|
||||
expect(params.get('state')).toBe('signed-state')
|
||||
// Read-only, and offline so a refresh token is actually issued.
|
||||
expect(params.get('scope')).toContain('gmail.readonly')
|
||||
expect(params.get('scope')).not.toContain('gmail.modify')
|
||||
expect(params.get('scope')).not.toContain('gmail.send')
|
||||
expect(params.get('access_type')).toBe('offline')
|
||||
// Not incremental: include_granted_scopes would let Google fold scopes this
|
||||
// app was granted elsewhere into the token issued here, so a mailbox grant
|
||||
// could carry more authority than the consent screen showed.
|
||||
expect(params.get('include_granted_scopes')).toBeNull()
|
||||
})
|
||||
})
|
||||
|
||||
/**
|
||||
* Regression from the first provkörning against a real ledger: the same seven
|
||||
* unrelated messages came back for every purchase, because the bank's
|
||||
* description is not a merchant name and its month and person tokens match most
|
||||
* of a mailbox.
|
||||
*/
|
||||
describe('merchant terms drawn from a real bank description', () => {
|
||||
it('does not search for the month in a salary row', () => {
|
||||
expect(merchantTerms('Lön Juli Jakob Överföring via internet')).not.toContain('Juli')
|
||||
})
|
||||
|
||||
it('drops the payment rail, which every row on the statement shares', () => {
|
||||
const terms = merchantTerms('Kontor Sting apr BG 0000059142596 Bg-bet. via internet')
|
||||
for (const noise of ['apr', 'via', 'internet']) {
|
||||
expect(terms.map((t) => t.toLowerCase())).not.toContain(noise)
|
||||
}
|
||||
})
|
||||
|
||||
it('still keeps what actually names the supplier', () => {
|
||||
// This row matched a real Sting invoice; the fix must not cost that.
|
||||
const terms = merchantTerms('Kontor Sting apr BG 0000059142596 Bg-bet. via internet')
|
||||
expect(terms.map((t) => t.toLowerCase())).toContain('sting')
|
||||
})
|
||||
|
||||
it('leaves an ordinary card purchase alone', () => {
|
||||
expect(merchantTerms('Elgiganten Aktiebolag')).toContain('Elgiganten')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,29 @@
|
||||
/**
|
||||
* Resolve the origin used to build OAuth redirect URIs.
|
||||
*
|
||||
* Both OAuth legs must send the same redirect_uri. Pinning it to the
|
||||
* deployment's canonical app URL also prevents an old domain alias or preview
|
||||
* host from generating a callback that is not registered with the provider.
|
||||
* Self-hosted deployments without NEXT_PUBLIC_APP_URL fall back to the
|
||||
* request origin.
|
||||
*/
|
||||
export function resolveCallbackOrigin(requestOrigin: string): string {
|
||||
const appUrl = process.env.NEXT_PUBLIC_APP_URL
|
||||
if (appUrl && appUrl.trim().length > 0) {
|
||||
try {
|
||||
// Normalizes trailing slashes and strips paths so the provider receives
|
||||
// the same bare origin on the authorization and token-exchange legs.
|
||||
const configuredUrl = new URL(appUrl)
|
||||
if (
|
||||
configuredUrl.protocol !== 'http:' &&
|
||||
configuredUrl.protocol !== 'https:'
|
||||
) {
|
||||
return requestOrigin
|
||||
}
|
||||
return configuredUrl.origin
|
||||
} catch {
|
||||
return requestOrigin
|
||||
}
|
||||
}
|
||||
return requestOrigin
|
||||
}
|
||||
@@ -0,0 +1,242 @@
|
||||
/**
|
||||
* Reading and maintaining mailbox grants.
|
||||
*
|
||||
* Every function here uses the service-role client: `mail_connections` has RLS
|
||||
* enabled with no policies precisely so a live refresh token can never be
|
||||
* selected by a browser session.
|
||||
*/
|
||||
import { createLogger } from '@/lib/logger'
|
||||
import type { SupabaseClient } from '@supabase/supabase-js'
|
||||
import { decryptToken, encryptToken } from './crypto'
|
||||
import {
|
||||
MailTokenRefreshError,
|
||||
getGoogleOAuthEnv,
|
||||
refreshAccessToken,
|
||||
} from './google-oauth'
|
||||
|
||||
const log = createLogger('mail-connections')
|
||||
|
||||
export interface MailConnectionRow {
|
||||
id: string
|
||||
company_id: string
|
||||
provider: 'gmail' | 'microsoft'
|
||||
email_address: string
|
||||
encrypted_refresh_token: string
|
||||
encrypted_access_token: string | null
|
||||
access_token_expires_at: string | null
|
||||
scope_label: string | null
|
||||
status: 'active' | 'needs_reconsent' | 'revoked'
|
||||
}
|
||||
|
||||
/** Safe projection for anything that answers a browser. Never includes tokens. */
|
||||
export interface MailConnectionSummary {
|
||||
id: string
|
||||
provider: 'gmail' | 'microsoft'
|
||||
emailAddress: string
|
||||
scopeLabel: string | null
|
||||
status: 'active' | 'needs_reconsent' | 'revoked'
|
||||
lastSearchedAt: string | null
|
||||
lastErrorCode: string | null
|
||||
}
|
||||
|
||||
export async function listConnections(
|
||||
supabase: SupabaseClient,
|
||||
companyId: string,
|
||||
): Promise<MailConnectionSummary[]> {
|
||||
const { data } = await supabase
|
||||
.from('mail_connections')
|
||||
.select('id, provider, email_address, scope_label, status, last_searched_at, last_error_code')
|
||||
.eq('company_id', companyId)
|
||||
.order('created_at', { ascending: true })
|
||||
|
||||
return ((data ?? []) as Array<Record<string, unknown>>).map((row) => ({
|
||||
id: row.id as string,
|
||||
provider: row.provider as 'gmail' | 'microsoft',
|
||||
emailAddress: row.email_address as string,
|
||||
scopeLabel: (row.scope_label as string | null) ?? null,
|
||||
status: row.status as MailConnectionSummary['status'],
|
||||
lastSearchedAt: (row.last_searched_at as string | null) ?? null,
|
||||
lastErrorCode: (row.last_error_code as string | null) ?? null,
|
||||
}))
|
||||
}
|
||||
|
||||
export async function listActiveConnections(
|
||||
supabase: SupabaseClient,
|
||||
companyId: string,
|
||||
): Promise<MailConnectionRow[]> {
|
||||
const { data } = await supabase
|
||||
.from('mail_connections')
|
||||
.select(
|
||||
'id, company_id, provider, email_address, encrypted_refresh_token, encrypted_access_token, access_token_expires_at, scope_label, status',
|
||||
)
|
||||
.eq('company_id', companyId)
|
||||
.eq('status', 'active')
|
||||
return (data ?? []) as MailConnectionRow[]
|
||||
}
|
||||
|
||||
/**
|
||||
* Upsert on (company, provider, address) so reconnecting the same mailbox
|
||||
* refreshes the grant instead of creating a twin that gets searched twice.
|
||||
*/
|
||||
export async function saveConnection(
|
||||
supabase: SupabaseClient,
|
||||
params: {
|
||||
companyId: string
|
||||
userId: string
|
||||
provider: 'gmail' | 'microsoft'
|
||||
emailAddress: string
|
||||
refreshToken: string
|
||||
accessToken: string
|
||||
expiresAt: Date
|
||||
scopes: string[]
|
||||
backfillFrom: string | null
|
||||
},
|
||||
): Promise<void> {
|
||||
const { error } = await supabase.from('mail_connections').upsert(
|
||||
{
|
||||
company_id: params.companyId,
|
||||
provider: params.provider,
|
||||
// Lowercased here rather than by an expression index, so the upsert's
|
||||
// ON CONFLICT target matches the index exactly (Postgres 42P10 otherwise).
|
||||
email_address: params.emailAddress.trim().toLowerCase(),
|
||||
connected_by: params.userId,
|
||||
encrypted_refresh_token: encryptToken(params.refreshToken),
|
||||
encrypted_access_token: encryptToken(params.accessToken),
|
||||
access_token_expires_at: params.expiresAt.toISOString(),
|
||||
scopes: params.scopes,
|
||||
backfill_from: params.backfillFrom,
|
||||
status: 'active',
|
||||
last_error_code: null,
|
||||
last_error_at: null,
|
||||
},
|
||||
{ onConflict: 'company_id,provider,email_address' },
|
||||
)
|
||||
if (error) throw new Error(`Failed to save mail connection: ${error.message}`)
|
||||
}
|
||||
|
||||
async function markNeedsReconsent(
|
||||
supabase: SupabaseClient,
|
||||
connectionId: string,
|
||||
code: string,
|
||||
): Promise<void> {
|
||||
await supabase
|
||||
.from('mail_connections')
|
||||
.update({ status: 'needs_reconsent', last_error_code: code, last_error_at: new Date().toISOString() })
|
||||
.eq('id', connectionId)
|
||||
}
|
||||
|
||||
/**
|
||||
* A usable access token for one connection, refreshing when it has expired.
|
||||
*
|
||||
* Returns null rather than throwing when the grant is dead: one revoked
|
||||
* mailbox must shrink the hunt, never abort it.
|
||||
*/
|
||||
export async function getAccessToken(
|
||||
supabase: SupabaseClient,
|
||||
connection: MailConnectionRow,
|
||||
origin: string,
|
||||
): Promise<string | null> {
|
||||
const expiresAt = connection.access_token_expires_at
|
||||
? new Date(connection.access_token_expires_at)
|
||||
: null
|
||||
// 60s of slack so a token cannot expire mid-request.
|
||||
if (connection.encrypted_access_token && expiresAt && expiresAt.getTime() - 60_000 > Date.now()) {
|
||||
try {
|
||||
return decryptToken(connection.encrypted_access_token)
|
||||
} catch {
|
||||
// Fall through to a refresh: an undecryptable token means the key
|
||||
// rotated, which a refresh repairs.
|
||||
}
|
||||
}
|
||||
|
||||
try {
|
||||
const env = getGoogleOAuthEnv(origin)
|
||||
const refreshToken = decryptToken(connection.encrypted_refresh_token)
|
||||
const refreshed = await refreshAccessToken(env, refreshToken)
|
||||
await supabase
|
||||
.from('mail_connections')
|
||||
.update({
|
||||
encrypted_access_token: encryptToken(refreshed.accessToken),
|
||||
access_token_expires_at: refreshed.expiresAt.toISOString(),
|
||||
})
|
||||
.eq('id', connection.id)
|
||||
return refreshed.accessToken
|
||||
} catch (error) {
|
||||
if (error instanceof MailTokenRefreshError && error.permanent) {
|
||||
await markNeedsReconsent(supabase, connection.id, 'invalid_grant')
|
||||
}
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
export async function touchSearched(
|
||||
supabase: SupabaseClient,
|
||||
connectionId: string,
|
||||
): Promise<void> {
|
||||
await supabase
|
||||
.from('mail_connections')
|
||||
.update({ last_searched_at: new Date().toISOString() })
|
||||
.eq('id', connectionId)
|
||||
}
|
||||
|
||||
export async function disconnect(
|
||||
supabase: SupabaseClient,
|
||||
companyId: string,
|
||||
connectionId: string,
|
||||
userId: string,
|
||||
): Promise<void> {
|
||||
// Read the address before the row goes, so the audit entry can name the
|
||||
// mailbox that stopped being searched.
|
||||
const { data: existing } = await supabase
|
||||
.from('mail_connections')
|
||||
.select('email_address, provider')
|
||||
.eq('id', connectionId)
|
||||
.eq('company_id', companyId)
|
||||
.maybeSingle()
|
||||
|
||||
// Hard delete: the point of disconnecting is that the token is gone. Receipts
|
||||
// already approved stay, because they belong to the bookkeeping now.
|
||||
const { error: deleteError } = await supabase
|
||||
.from('mail_connections')
|
||||
.delete()
|
||||
.eq('id', connectionId)
|
||||
.eq('company_id', companyId)
|
||||
// A failed delete must not leave an audit entry claiming the mailbox was
|
||||
// disconnected when the credential is still live.
|
||||
if (deleteError) throw new Error(deleteError.message)
|
||||
if (!existing) return
|
||||
|
||||
const row = existing as { email_address: string; provider: string }
|
||||
|
||||
// BFNAR 2013:2 kap 8 (behandlingshistorik): which mailboxes feed underlag into
|
||||
// the books is a control over how räkenskapsinformation is produced, so
|
||||
// switching one off has to be reconstructable years later.
|
||||
//
|
||||
// Written by hand rather than by the write_audit_log trigger the accounting
|
||||
// tables use. That trigger copies the whole row into audit_log, which here
|
||||
// would mean copying an encrypted refresh token into a second table and
|
||||
// keeping it after the point of the delete was to destroy it. The sibling
|
||||
// credential tables (shopify_connections) omit the trigger for the same
|
||||
// reason. Only the safe columns are recorded.
|
||||
const { error: auditError } = await supabase.from('audit_log').insert({
|
||||
user_id: userId,
|
||||
company_id: companyId,
|
||||
action: 'DELETE',
|
||||
table_name: 'mail_connections',
|
||||
record_id: connectionId,
|
||||
description: `Brevlåda frånkopplad: ${row.email_address} (${row.provider})`,
|
||||
old_state: { email_address: row.email_address, provider: row.provider },
|
||||
new_state: null,
|
||||
})
|
||||
// Deliberately not rolled back into one transaction. The two statements can
|
||||
// only diverge one way now: the credential is destroyed and the note about it
|
||||
// is missing. Recreating the credential to keep them in step would be worse
|
||||
// than a missing note, so the gap is surfaced loudly instead of hidden.
|
||||
if (auditError) {
|
||||
log.error('mailbox disconnected but the audit entry failed to write', {
|
||||
connectionId,
|
||||
companyId,
|
||||
error: auditError.message,
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
import crypto from 'crypto'
|
||||
|
||||
/**
|
||||
* AES-256-GCM for mailbox refresh tokens.
|
||||
*
|
||||
* A mail grant is the hottest credential this product holds: it reads someone's
|
||||
* correspondence, not just their backups. So the key is its own env var by
|
||||
* preference (MAIL_TOKEN_ENCRYPTION_KEY, 32 bytes hex) and can be rotated
|
||||
* without touching the database password, following the Skatteverket
|
||||
* token-store rather than cloud-backup's service-role derivation.
|
||||
*
|
||||
* The derived fallback exists so local development and self-hosted deployments
|
||||
* work before anyone sets the variable; it is the same trust boundary as the
|
||||
* database itself, and the purpose string keeps it distinct from every other
|
||||
* derivation in the codebase.
|
||||
*/
|
||||
|
||||
const ALGORITHM = 'aes-256-gcm'
|
||||
|
||||
function getKey(): Buffer {
|
||||
const dedicated = process.env.MAIL_TOKEN_ENCRYPTION_KEY
|
||||
if (dedicated && dedicated.trim().length > 0) {
|
||||
const buf = Buffer.from(dedicated.trim(), 'hex')
|
||||
if (buf.length !== 32) {
|
||||
throw new Error('MAIL_TOKEN_ENCRYPTION_KEY must be 32 bytes of hex (openssl rand -hex 32)')
|
||||
}
|
||||
return buf
|
||||
}
|
||||
const secret = process.env.SUPABASE_SERVICE_ROLE_KEY
|
||||
if (!secret) throw new Error('MAIL_TOKEN_ENCRYPTION_KEY or SUPABASE_SERVICE_ROLE_KEY is required')
|
||||
return crypto.createHash('sha256').update('mail-connections:v1:' + secret).digest()
|
||||
}
|
||||
|
||||
export function encryptToken(plaintext: string): string {
|
||||
const key = getKey()
|
||||
const iv = crypto.randomBytes(12)
|
||||
const cipher = crypto.createCipheriv(ALGORITHM, key, iv)
|
||||
const encrypted = Buffer.concat([cipher.update(plaintext, 'utf8'), cipher.final()])
|
||||
const tag = cipher.getAuthTag()
|
||||
return Buffer.concat([iv, tag, encrypted]).toString('base64url')
|
||||
}
|
||||
|
||||
export function decryptToken(ciphertext: string): string {
|
||||
const key = getKey()
|
||||
const combined = Buffer.from(ciphertext, 'base64url')
|
||||
const iv = combined.subarray(0, 12)
|
||||
const tag = combined.subarray(12, 28)
|
||||
const encrypted = combined.subarray(28)
|
||||
const decipher = crypto.createDecipheriv(ALGORITHM, key, iv)
|
||||
decipher.setAuthTag(tag)
|
||||
return Buffer.concat([decipher.update(encrypted), decipher.final()]).toString('utf8')
|
||||
}
|
||||
|
||||
/**
|
||||
* Short-lived signed state for OAuth CSRF protection. Stateless and
|
||||
* self-expiring, so a callback needs no database round-trip to be trusted.
|
||||
*/
|
||||
const STATE_TTL_MS = 10 * 60 * 1000
|
||||
|
||||
interface StatePayload {
|
||||
u: string
|
||||
c: string
|
||||
e: number
|
||||
}
|
||||
|
||||
export function createOAuthState(userId: string, companyId: string): string {
|
||||
const payload: StatePayload = { u: userId, c: companyId, e: Date.now() + STATE_TTL_MS }
|
||||
return encryptToken(JSON.stringify(payload))
|
||||
}
|
||||
|
||||
export function verifyOAuthState(state: string): { userId: string; companyId: string } | null {
|
||||
try {
|
||||
const payload = JSON.parse(decryptToken(state)) as StatePayload
|
||||
if (Date.now() > payload.e) return null
|
||||
return { userId: payload.u, companyId: payload.c }
|
||||
} catch {
|
||||
return null
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,203 @@
|
||||
/**
|
||||
* The thin slice of the Gmail API the hunt needs: search, read headers, fetch
|
||||
* an attachment. Nothing here writes, because the granted scope cannot.
|
||||
*/
|
||||
import type { MailCandidate } from '@/lib/mail-search/service'
|
||||
|
||||
const API = 'https://gmail.googleapis.com/gmail/v1/users/me'
|
||||
|
||||
/**
|
||||
* Deadline on every Gmail call.
|
||||
*
|
||||
* Mailboxes are searched with Promise.all, so one stalled request would hold
|
||||
* the whole company's hunt open until the platform killed the run. A timeout
|
||||
* turns that into one mailbox missing from tonight's sweep.
|
||||
*/
|
||||
export const GMAIL_TIMEOUT_MS = 15_000
|
||||
|
||||
/** Hits to consider per mailbox per purchase. */
|
||||
export const MAX_RESULTS = 8
|
||||
|
||||
interface GmailHeader {
|
||||
name: string
|
||||
value: string
|
||||
}
|
||||
|
||||
interface GmailPart {
|
||||
filename?: string
|
||||
mimeType?: string
|
||||
body?: { attachmentId?: string; size?: number; data?: string }
|
||||
parts?: GmailPart[]
|
||||
}
|
||||
|
||||
/** Longest body worth carrying: a receipt states its total near the top. */
|
||||
const MAX_BODY_CHARS = 2500
|
||||
|
||||
/**
|
||||
* The readable text of a mail.
|
||||
*
|
||||
* Already on the wire (format=full is required to see the parts tree at all),
|
||||
* so this costs nothing extra, and it is where the two facts a forwarded
|
||||
* receipt hides live: the original sender and the original date, both written
|
||||
* into the "Vidarebefordrat meddelande" header that Gmail's 200-character
|
||||
* snippet cuts off.
|
||||
*/
|
||||
function collectBodyText(part: GmailPart | undefined, out: string[]): void {
|
||||
if (!part) return
|
||||
const type = part.mimeType ?? ''
|
||||
if ((type === 'text/plain' || type === 'text/html') && part.body?.data) {
|
||||
out.push(Buffer.from(part.body.data, 'base64url').toString('utf8'))
|
||||
}
|
||||
for (const child of part.parts ?? []) collectBodyText(child, out)
|
||||
}
|
||||
|
||||
function readableBody(msg: GmailMessage): string {
|
||||
const chunks: string[] = []
|
||||
collectBodyText(msg.payload, chunks)
|
||||
return chunks
|
||||
.join('\n')
|
||||
.replace(/<style[\s\S]*?<\/style>/gi, ' ')
|
||||
.replace(/<[^>]+>/g, ' ')
|
||||
.replace(/ |‌|͏/gi, ' ')
|
||||
.replace(/&/gi, '&')
|
||||
.replace(/\s+/g, ' ')
|
||||
.trim()
|
||||
.slice(0, MAX_BODY_CHARS)
|
||||
}
|
||||
|
||||
interface GmailMessage {
|
||||
id: string
|
||||
internalDate?: string
|
||||
snippet?: string
|
||||
payload?: {
|
||||
headers?: GmailHeader[]
|
||||
filename?: string
|
||||
mimeType?: string
|
||||
body?: { attachmentId?: string; size?: number }
|
||||
parts?: GmailPart[]
|
||||
}
|
||||
}
|
||||
|
||||
function header(msg: GmailMessage, name: string): string | null {
|
||||
const found = msg.payload?.headers?.find((h) => h.name.toLowerCase() === name.toLowerCase())
|
||||
return found?.value ?? null
|
||||
}
|
||||
|
||||
/** Attachments anywhere in the MIME tree, ignoring inline images. */
|
||||
function collectAttachments(
|
||||
part: GmailPart | undefined,
|
||||
out: Array<{ id: string; filename: string }>,
|
||||
): void {
|
||||
if (!part) return
|
||||
const id = part.body?.attachmentId
|
||||
const named = part.filename && part.filename.length > 0
|
||||
const isDocument =
|
||||
named &&
|
||||
!/^image\/(png|gif)$/i.test(part.mimeType ?? '') // inline logos, not receipts
|
||||
if (id && isDocument) out.push({ id, filename: part.filename as string })
|
||||
for (const child of part.parts ?? []) collectAttachments(child, out)
|
||||
}
|
||||
|
||||
async function gmailFetch<T>(accessToken: string, path: string): Promise<T> {
|
||||
const response = await fetch(`${API}${path}`, {
|
||||
headers: { Authorization: `Bearer ${accessToken}` },
|
||||
signal: AbortSignal.timeout(GMAIL_TIMEOUT_MS),
|
||||
})
|
||||
if (!response.ok) {
|
||||
const text = await response.text().catch(() => '')
|
||||
throw new Error(`Gmail ${response.status}: ${text.slice(0, 200)}`)
|
||||
}
|
||||
return (await response.json()) as T
|
||||
}
|
||||
|
||||
export async function searchMessageIds(
|
||||
accessToken: string,
|
||||
query: string,
|
||||
maxResults: number = MAX_RESULTS,
|
||||
): Promise<string[]> {
|
||||
const params = new URLSearchParams({ q: query, maxResults: String(maxResults) })
|
||||
const data = await gmailFetch<{ messages?: Array<{ id: string }> }>(
|
||||
accessToken,
|
||||
`/messages?${params.toString()}`,
|
||||
)
|
||||
return (data.messages ?? []).map((m) => m.id)
|
||||
}
|
||||
|
||||
/**
|
||||
* Subject, sender, date and which parts are attachments.
|
||||
*
|
||||
* `format=full` rather than `format=metadata`: metadata returns headers only
|
||||
* and omits `payload.parts` entirely, so every message came back looking like
|
||||
* it had no attachments and the hunt could never file anything. Gmail offers no
|
||||
* format that returns the MIME structure without the body, so the body does
|
||||
* come down the wire here. It is read for nothing and stored nowhere: only
|
||||
* attachment bytes are ever persisted, and only after a match.
|
||||
*/
|
||||
export async function getMessageSummary(
|
||||
accessToken: string,
|
||||
messageId: string,
|
||||
connectionId: string,
|
||||
mailbox: string,
|
||||
): Promise<MailCandidate> {
|
||||
const msg = await gmailFetch<GmailMessage>(accessToken, `/messages/${messageId}?format=full`)
|
||||
const attachments: Array<{ id: string; filename: string }> = []
|
||||
collectAttachments(msg.payload, attachments)
|
||||
|
||||
return {
|
||||
connectionId,
|
||||
mailbox,
|
||||
provider: 'gmail',
|
||||
messageId: msg.id,
|
||||
subject: header(msg, 'Subject'),
|
||||
from: header(msg, 'From'),
|
||||
receivedAt: msg.internalDate
|
||||
? new Date(Number(msg.internalDate)).toISOString()
|
||||
: header(msg, 'Date'),
|
||||
// A receipt with no attachment is usually the mail body itself; the caller
|
||||
// decides whether to render it.
|
||||
attachmentIds: attachments.map((a) => a.id),
|
||||
attachmentNames: attachments.map((a) => a.filename),
|
||||
snippet: msg.snippet ?? null,
|
||||
bodyText: readableBody(msg),
|
||||
bodyIsReceipt: attachments.length === 0,
|
||||
}
|
||||
}
|
||||
|
||||
export async function fetchAttachmentBytes(
|
||||
accessToken: string,
|
||||
messageId: string,
|
||||
attachmentId: string,
|
||||
): Promise<Buffer | null> {
|
||||
const data = await gmailFetch<{ data?: string; size?: number }>(
|
||||
accessToken,
|
||||
`/messages/${messageId}/attachments/${attachmentId}`,
|
||||
)
|
||||
if (!data.data) return null
|
||||
return Buffer.from(data.data, 'base64url')
|
||||
}
|
||||
|
||||
/**
|
||||
* Filename and MIME type live on the message, not on the attachment response,
|
||||
* so they are read back from the parts tree.
|
||||
*/
|
||||
export async function describeAttachment(
|
||||
accessToken: string,
|
||||
messageId: string,
|
||||
attachmentId: string,
|
||||
): Promise<{ filename: string; mimeType: string } | null> {
|
||||
const msg = await gmailFetch<GmailMessage>(accessToken, `/messages/${messageId}?format=full`)
|
||||
let found: { filename: string; mimeType: string } | null = null
|
||||
const walk = (part: GmailPart | undefined): void => {
|
||||
if (!part || found) return
|
||||
if (part.body?.attachmentId === attachmentId) {
|
||||
found = {
|
||||
filename: part.filename || 'underlag.pdf',
|
||||
mimeType: part.mimeType || 'application/octet-stream',
|
||||
}
|
||||
return
|
||||
}
|
||||
for (const child of part.parts ?? []) walk(child)
|
||||
}
|
||||
walk(msg.payload)
|
||||
return found
|
||||
}
|
||||
@@ -0,0 +1,160 @@
|
||||
/**
|
||||
* Turning a bank purchase into a Gmail search.
|
||||
*
|
||||
* Pure, because this is where recall is won or lost and it must be testable
|
||||
* without a mailbox. The search runs provider-side and only the hits come back:
|
||||
* we never sync or index a mailbox, which is what keeps the feature inside
|
||||
* Google's Limited Use terms and inside GDPR data minimisation.
|
||||
*/
|
||||
import type { MailSearchQuery } from '@/lib/mail-search/service'
|
||||
|
||||
/**
|
||||
* How far around the purchase date to look.
|
||||
*
|
||||
* Asymmetric on purpose: a receipt is normally emailed at or just after the
|
||||
* purchase, while the bank may post the charge a few days late, so the mail can
|
||||
* legitimately predate the transaction date. Card settlement is the reason for
|
||||
* the tail, and Pleo Fetch uses the same shape (-3 / +10).
|
||||
*/
|
||||
export const DAYS_BEFORE = 3
|
||||
export const DAYS_AFTER = 10
|
||||
|
||||
/**
|
||||
* Merchant tokens too generic to search on alone.
|
||||
*
|
||||
* The bank's description is not a merchant name: it is whatever the payer typed
|
||||
* plus the rail it went over. Month names and rail words are in here because a
|
||||
* provkörning on a real ledger showed "Lön Juli Jakob Överföring via internet"
|
||||
* searching for `"Juli"`, which matches most of a mailbox and returned the same
|
||||
* seven unrelated messages for every purchase.
|
||||
*/
|
||||
const STOPWORDS = new Set([
|
||||
'ab', 'hb', 'kb', 'inc', 'llc', 'ltd', 'gmbh', 'oy', 'pbc', 'plc', 'corp', 'co',
|
||||
'the', 'and', 'och', 'group', 'sweden', 'sverige', 'international', 'kortkop',
|
||||
'kortköp', 'uttag', 'betalning', 'payment', 'store', 'shop', 'www', 'com',
|
||||
// Payment rails and the bank's own boilerplate.
|
||||
'överföring', 'overforing', 'internet', 'via', 'bankgiro', 'plusgiro', 'autogiro',
|
||||
'bgbet', 'bg-bet', 'insättning', 'insattning', 'inbetalning', 'utbetalning',
|
||||
'europabetalning', 'swish', 'faktura', 'invoice', 'kortköputtag', 'kortkoputtag',
|
||||
// Months, Swedish and English, full and abbreviated.
|
||||
'januari', 'februari', 'mars', 'april', 'maj', 'juni', 'juli', 'augusti',
|
||||
'september', 'oktober', 'november', 'december',
|
||||
'january', 'february', 'march', 'may', 'june', 'july', 'august', 'october',
|
||||
'jan', 'feb', 'mar', 'apr', 'jun', 'jul', 'aug', 'sep', 'sept', 'okt', 'oct',
|
||||
'nov', 'dec',
|
||||
])
|
||||
|
||||
function isoDay(d: Date): string {
|
||||
return d.toISOString().slice(0, 10)
|
||||
}
|
||||
|
||||
function shiftDays(iso: string, days: number): string {
|
||||
const d = new Date(iso)
|
||||
d.setDate(d.getDate() + days)
|
||||
return isoDay(d)
|
||||
}
|
||||
|
||||
/**
|
||||
* Pick the tokens worth searching for. Short and generic tokens are dropped:
|
||||
* a query for "ab" returns the whole mailbox and costs a page of results for
|
||||
* nothing.
|
||||
*/
|
||||
export function merchantTerms(merchant: string | null): string[] {
|
||||
if (!merchant) return []
|
||||
return merchant
|
||||
.split(/\s+/)
|
||||
.map((t) => t.trim())
|
||||
.filter((t) => t.length >= 3 && !STOPWORDS.has(t.toLowerCase()))
|
||||
.slice(0, 3)
|
||||
}
|
||||
|
||||
/**
|
||||
* Format the amount the way a receipt would write it, so the number itself
|
||||
* becomes a search term. Swedish receipts write 1 234,50; most SaaS invoices
|
||||
* write 1234.50. Both forms are offered.
|
||||
*/
|
||||
export function amountTerms(amount: number): string[] {
|
||||
const abs = Math.abs(amount)
|
||||
const twoDp = abs.toFixed(2)
|
||||
const terms = new Set<string>([twoDp, twoDp.replace('.', ',')])
|
||||
if (Number.isInteger(abs)) terms.add(String(abs))
|
||||
|
||||
// Swedish invoices group thousands with a space: 15 000,00, not 15000,00.
|
||||
// Measured against a real mailbox, the Sting office invoice was findable as
|
||||
// "15 000,00" and "15 000" and by nothing else: every ungrouped form
|
||||
// returned zero. Cheap to add and it is pure recall.
|
||||
const group = (v: string) => v.replace(/\B(?=(\d{3})+(?!\d))/g, ' ')
|
||||
const [whole] = twoDp.split('.')
|
||||
if (whole.length > 3) {
|
||||
terms.add(`${group(whole)},${twoDp.split('.')[1]}`)
|
||||
terms.add(group(whole))
|
||||
}
|
||||
return [...terms]
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the Gmail `q` for one purchase.
|
||||
*
|
||||
* Structure: a date window, then merchant OR amount. It is deliberately an OR:
|
||||
* requiring both misses the common cases where the bank's merchant string does
|
||||
* not appear in the receipt at all (a reseller, a parent company, a brand alias
|
||||
* like Anthropic billing as Claude), and the amount alone is a strong filter
|
||||
* inside a two-week window.
|
||||
*
|
||||
* `has:attachment` is NOT required: plenty of receipts are the mail body itself.
|
||||
*/
|
||||
export function buildGmailQuery(query: MailSearchQuery): string {
|
||||
const parts: string[] = []
|
||||
|
||||
// Off by default now. Receipts reach these mailboxes by being forwarded, and
|
||||
// a forward carries the forwarding date, so windowing on the purchase date
|
||||
// hides the very mail we want. The caller re-establishes precision by having
|
||||
// the model judge the hits instead.
|
||||
if (query.useDateWindow !== false) {
|
||||
const after = shiftDays(query.date, -DAYS_BEFORE - 1) // Gmail's after: is exclusive
|
||||
const before = shiftDays(query.date, DAYS_AFTER + 1)
|
||||
parts.push(`after:${after.replace(/-/g, '/')}`, `before:${before.replace(/-/g, '/')}`)
|
||||
}
|
||||
|
||||
// Merchant OR amount, never one instead of the other.
|
||||
//
|
||||
// The amount is the single strongest signal a reconciliation has: dates drift
|
||||
// because banks post late and mail gets forwarded, but an amount does not
|
||||
// drift. Gmail indexes text inside PDF attachments, so a Swedish receipt is
|
||||
// often findable by its total alone. It is an OR rather than an AND because
|
||||
// neither signal survives every case: a receipt billed in USD never contains
|
||||
// the SEK figure the bank charged, and a bank descriptor frequently names
|
||||
// nothing that appears in the receipt.
|
||||
const aliases = (query.aliases ?? []).map((a) => a.trim()).filter((a) => a.length >= 2)
|
||||
const names = aliases.length > 0 ? aliases : merchantTerms(query.merchant)
|
||||
const alternatives = [
|
||||
...amountTerms(query.amount).map((a) => `"${a}"`),
|
||||
...names.map((t) => `"${t}"`),
|
||||
]
|
||||
|
||||
if (alternatives.length > 0) {
|
||||
parts.push(`(${alternatives.join(' OR ')})`)
|
||||
}
|
||||
|
||||
if (query.requireAttachment) parts.push('has:attachment')
|
||||
|
||||
// Calendar invitations and the user's own outbound mail are never receipts.
|
||||
parts.push('-in:chats')
|
||||
|
||||
return parts.join(' ')
|
||||
}
|
||||
|
||||
/**
|
||||
* Cheap pre-filter on a hit before spending a model call on it.
|
||||
*
|
||||
* Gmail's OR query is broad by design, so most hits are not receipts. Anything
|
||||
* that looks like a newsletter or a calendar notice is dropped here rather than
|
||||
* being read.
|
||||
*/
|
||||
const OBVIOUS_NON_RECEIPT = /\b(nyhetsbrev|newsletter|unsubscribe|prenumerera|kalender|calendar invite|inbjudan)\b/i
|
||||
|
||||
export function looksLikeReceipt(subject: string | null, from: string | null): boolean {
|
||||
const haystack = `${subject ?? ''} ${from ?? ''}`
|
||||
if (OBVIOUS_NON_RECEIPT.test(haystack)) return false
|
||||
return true
|
||||
}
|
||||
@@ -0,0 +1,189 @@
|
||||
/**
|
||||
* Gmail OAuth, read-only.
|
||||
*
|
||||
* The scope is `gmail.readonly` and nothing else. That is enough to search and
|
||||
* to download attachment bytes (verified against Google's method-scope table),
|
||||
* and it structurally cannot send, modify or delete: the promise made in the
|
||||
* consent screen is enforced by the grant, not by our code being careful.
|
||||
*
|
||||
* Consequence worth remembering: because we never hold a send scope, the agent
|
||||
* can prepare a forward for the user but can never send one itself.
|
||||
*/
|
||||
export const GMAIL_READONLY_SCOPE = 'https://www.googleapis.com/auth/gmail.readonly'
|
||||
|
||||
const AUTH_ENDPOINT = 'https://accounts.google.com/o/oauth2/v2/auth'
|
||||
/**
|
||||
* Deadline on the token endpoint.
|
||||
*
|
||||
* A refresh happens inside every mailbox search, and searches run with
|
||||
* Promise.all, so a stalled token endpoint would hold the whole company's hunt
|
||||
* open. On timeout the connection simply yields nothing this run.
|
||||
*/
|
||||
export const TOKEN_TIMEOUT_MS = 15_000
|
||||
|
||||
const TOKEN_ENDPOINT = 'https://oauth2.googleapis.com/token'
|
||||
|
||||
export interface GoogleOAuthEnv {
|
||||
clientId: string
|
||||
clientSecret: string
|
||||
redirectUri: string
|
||||
}
|
||||
|
||||
/**
|
||||
* Deliberately distinct from cloud-backup's GOOGLE_CLIENT_ID: that is a
|
||||
* different OAuth client, in a different project, owned by a different founder,
|
||||
* and sharing the pair would let one integration's credential rotation break
|
||||
* the other.
|
||||
*/
|
||||
export function isGoogleMailConfigured(): boolean {
|
||||
return Boolean(process.env.GOOGLE_MAIL_CLIENT_ID && process.env.GOOGLE_MAIL_CLIENT_SECRET)
|
||||
}
|
||||
|
||||
export function getGoogleOAuthEnv(origin: string): GoogleOAuthEnv {
|
||||
const clientId = process.env.GOOGLE_MAIL_CLIENT_ID
|
||||
const clientSecret = process.env.GOOGLE_MAIL_CLIENT_SECRET
|
||||
if (!clientId || !clientSecret) {
|
||||
throw new Error('Gmail is not configured: set GOOGLE_MAIL_CLIENT_ID and GOOGLE_MAIL_CLIENT_SECRET')
|
||||
}
|
||||
return {
|
||||
clientId,
|
||||
clientSecret,
|
||||
// Must match the string registered in the Google console exactly; the
|
||||
// extension slug `mail` is pinned for that reason.
|
||||
redirectUri: `${origin}/api/extensions/ext/mail/oauth/callback`,
|
||||
}
|
||||
}
|
||||
|
||||
export function buildAuthorizationUrl(env: GoogleOAuthEnv, state: string): string {
|
||||
const params = new URLSearchParams({
|
||||
client_id: env.clientId,
|
||||
redirect_uri: env.redirectUri,
|
||||
response_type: 'code',
|
||||
scope: `openid email ${GMAIL_READONLY_SCOPE}`,
|
||||
// Signed, self-expiring CSRF token. The callback refuses anything without
|
||||
// it, so omitting this breaks the flow as well as the protection.
|
||||
state,
|
||||
// offline + consent is what returns a refresh token at all; without it a
|
||||
// grant dies in an hour and the nightly hunt silently stops.
|
||||
access_type: 'offline',
|
||||
// No include_granted_scopes: it lets Google add scopes this app was granted
|
||||
// elsewhere to the token it returns here, so a mailbox grant could quietly
|
||||
// carry more authority than the consent screen showed.
|
||||
prompt: 'consent',
|
||||
})
|
||||
return `${AUTH_ENDPOINT}?${params.toString()}`
|
||||
}
|
||||
|
||||
export interface GoogleTokens {
|
||||
accessToken: string
|
||||
refreshToken: string | null
|
||||
expiresAt: Date
|
||||
scopes: string[]
|
||||
email: string | null
|
||||
}
|
||||
|
||||
/** Raised when a grant is dead rather than the request being unlucky. */
|
||||
export class MailTokenRefreshError extends Error {
|
||||
constructor(
|
||||
message: string,
|
||||
readonly permanent: boolean,
|
||||
) {
|
||||
super(message)
|
||||
this.name = 'MailTokenRefreshError'
|
||||
}
|
||||
}
|
||||
|
||||
function decodeIdTokenEmail(idToken: string | undefined): string | null {
|
||||
if (!idToken) return null
|
||||
try {
|
||||
const payload = idToken.split('.')[1]
|
||||
const json = JSON.parse(Buffer.from(payload, 'base64url').toString('utf8')) as {
|
||||
email?: string
|
||||
}
|
||||
return json.email ?? null
|
||||
} catch {
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
export async function exchangeCodeForTokens(
|
||||
env: GoogleOAuthEnv,
|
||||
code: string,
|
||||
): Promise<GoogleTokens> {
|
||||
const response = await fetch(TOKEN_ENDPOINT, {
|
||||
method: 'POST',
|
||||
signal: AbortSignal.timeout(TOKEN_TIMEOUT_MS),
|
||||
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
|
||||
body: new URLSearchParams({
|
||||
code,
|
||||
client_id: env.clientId,
|
||||
client_secret: env.clientSecret,
|
||||
redirect_uri: env.redirectUri,
|
||||
grant_type: 'authorization_code',
|
||||
}),
|
||||
})
|
||||
const body = (await response.json()) as {
|
||||
access_token?: string
|
||||
refresh_token?: string
|
||||
expires_in?: number
|
||||
scope?: string
|
||||
id_token?: string
|
||||
error?: string
|
||||
error_description?: string
|
||||
}
|
||||
if (!response.ok || !body.access_token) {
|
||||
throw new Error(body.error_description || body.error || 'Token exchange failed')
|
||||
}
|
||||
if (!body.refresh_token) {
|
||||
// Google withholds it when the user has an older grant for this client.
|
||||
// Say so plainly: the fix is to revoke at myaccount.google.com and retry,
|
||||
// and a connection without one is useless the moment the hour is up.
|
||||
throw new Error(
|
||||
'Google returned no refresh token. Remove the previous access for this app at myaccount.google.com/permissions and connect again.',
|
||||
)
|
||||
}
|
||||
return {
|
||||
accessToken: body.access_token,
|
||||
refreshToken: body.refresh_token,
|
||||
expiresAt: new Date(Date.now() + (body.expires_in ?? 3600) * 1000),
|
||||
scopes: (body.scope ?? '').split(' ').filter(Boolean),
|
||||
email: decodeIdTokenEmail(body.id_token),
|
||||
}
|
||||
}
|
||||
|
||||
export async function refreshAccessToken(
|
||||
env: GoogleOAuthEnv,
|
||||
refreshToken: string,
|
||||
): Promise<{ accessToken: string; expiresAt: Date }> {
|
||||
const response = await fetch(TOKEN_ENDPOINT, {
|
||||
method: 'POST',
|
||||
signal: AbortSignal.timeout(TOKEN_TIMEOUT_MS),
|
||||
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
|
||||
body: new URLSearchParams({
|
||||
refresh_token: refreshToken,
|
||||
client_id: env.clientId,
|
||||
client_secret: env.clientSecret,
|
||||
grant_type: 'refresh_token',
|
||||
}),
|
||||
})
|
||||
const body = (await response.json()) as {
|
||||
access_token?: string
|
||||
expires_in?: number
|
||||
error?: string
|
||||
error_description?: string
|
||||
}
|
||||
if (!response.ok || !body.access_token) {
|
||||
// invalid_grant means revoked, expired or password-changed: retrying every
|
||||
// night would just burn quota, so it is flagged permanent and the
|
||||
// connection is parked as needs_reconsent.
|
||||
const permanent = body.error === 'invalid_grant'
|
||||
throw new MailTokenRefreshError(
|
||||
body.error_description || body.error || 'Token refresh failed',
|
||||
permanent,
|
||||
)
|
||||
}
|
||||
return {
|
||||
accessToken: body.access_token,
|
||||
expiresAt: new Date(Date.now() + (body.expires_in ?? 3600) * 1000),
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,135 @@
|
||||
/**
|
||||
* The Gmail implementation of the core MailSearchService contract.
|
||||
*
|
||||
* Query-then-classify, never sync: for each purchase we run a provider-side
|
||||
* search, pull metadata for the few hits, and let the caller decide. No mailbox
|
||||
* is mirrored, no message body is stored, and nothing is written back to the
|
||||
* mailbox. That is what keeps this inside Google's Limited Use terms and inside
|
||||
* GDPR data minimisation, and it is the promise the consent screen makes.
|
||||
*/
|
||||
import { createServiceClientNoCookies } from '@/lib/auth/api-keys'
|
||||
import { createLogger } from '@/lib/logger'
|
||||
import type {
|
||||
FetchedAttachment,
|
||||
MailCandidate,
|
||||
MailSearchQuery,
|
||||
MailSearchService,
|
||||
} from '@/lib/mail-search/service'
|
||||
import { buildGmailQuery, looksLikeReceipt } from './gmail-query'
|
||||
import {
|
||||
describeAttachment,
|
||||
fetchAttachmentBytes,
|
||||
getMessageSummary,
|
||||
searchMessageIds,
|
||||
} from './gmail-client'
|
||||
import {
|
||||
getAccessToken,
|
||||
listActiveConnections,
|
||||
touchSearched,
|
||||
type MailConnectionRow,
|
||||
} from './connections'
|
||||
import { isGoogleMailConfigured } from './google-oauth'
|
||||
|
||||
const log = createLogger('mail-search')
|
||||
|
||||
/**
|
||||
* Origin used to rebuild the redirect_uri during a token refresh. Google
|
||||
* requires the same value the grant was issued against, so it is derived from
|
||||
* the deployment's canonical URL rather than a request that may not exist
|
||||
* (the hunt runs from a cron, with no browser origin to borrow).
|
||||
*/
|
||||
function canonicalOrigin(): string {
|
||||
return process.env.NEXT_PUBLIC_APP_URL?.trim() || 'http://localhost:3000'
|
||||
}
|
||||
|
||||
export class GmailSearchService implements MailSearchService {
|
||||
isConfigured(): boolean {
|
||||
return isGoogleMailConfigured()
|
||||
}
|
||||
|
||||
async search(companyId: string, query: MailSearchQuery): Promise<MailCandidate[]> {
|
||||
if (!this.isConfigured()) return []
|
||||
|
||||
const supabase = createServiceClientNoCookies()
|
||||
const connections = await listActiveConnections(supabase, companyId)
|
||||
if (connections.length === 0) return []
|
||||
|
||||
const q = buildGmailQuery(query)
|
||||
|
||||
// Mailboxes are searched in parallel: the work is read-only, so there is
|
||||
// nothing to serialise, and one slow account should not delay the rest.
|
||||
const perConnection = await Promise.all(
|
||||
connections.map((connection) => this.searchOne(supabase, connection, q, query.limit)),
|
||||
)
|
||||
return perConnection.flat()
|
||||
}
|
||||
|
||||
private async searchOne(
|
||||
supabase: ReturnType<typeof createServiceClientNoCookies>,
|
||||
connection: MailConnectionRow,
|
||||
q: string,
|
||||
limit?: number,
|
||||
): Promise<MailCandidate[]> {
|
||||
// A dead grant shrinks the hunt rather than aborting it; getAccessToken has
|
||||
// already parked it as needs_reconsent for the UI to surface.
|
||||
const accessToken = await getAccessToken(supabase, connection, canonicalOrigin())
|
||||
if (!accessToken) return []
|
||||
|
||||
try {
|
||||
const ids = await searchMessageIds(accessToken, q, limit)
|
||||
if (ids.length === 0) return []
|
||||
|
||||
const summaries = await Promise.all(
|
||||
ids.map((id) =>
|
||||
getMessageSummary(accessToken, id, connection.id, connection.email_address),
|
||||
),
|
||||
)
|
||||
await touchSearched(supabase, connection.id)
|
||||
|
||||
// Cheap pre-filter before anything expensive looks at these.
|
||||
return summaries.filter((c) => looksLikeReceipt(c.subject, c.from))
|
||||
} catch (error) {
|
||||
// Never let one mailbox's failure surface as the company's failure.
|
||||
log.warn('gmail search failed for connection', {
|
||||
connectionId: connection.id,
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
})
|
||||
return []
|
||||
}
|
||||
}
|
||||
|
||||
async fetchAttachment(
|
||||
connectionId: string,
|
||||
messageId: string,
|
||||
attachmentId: string,
|
||||
): Promise<FetchedAttachment | null> {
|
||||
const supabase = createServiceClientNoCookies()
|
||||
const { data } = await supabase
|
||||
.from('mail_connections')
|
||||
.select(
|
||||
'id, company_id, provider, email_address, encrypted_refresh_token, encrypted_access_token, access_token_expires_at, scope_label, status',
|
||||
)
|
||||
.eq('id', connectionId)
|
||||
.maybeSingle()
|
||||
if (!data) return null
|
||||
|
||||
const accessToken = await getAccessToken(
|
||||
supabase,
|
||||
data as MailConnectionRow,
|
||||
canonicalOrigin(),
|
||||
)
|
||||
if (!accessToken) return null
|
||||
|
||||
const [bytes, described] = await Promise.all([
|
||||
fetchAttachmentBytes(accessToken, messageId, attachmentId),
|
||||
describeAttachment(accessToken, messageId, attachmentId),
|
||||
])
|
||||
if (!bytes) return null
|
||||
|
||||
return {
|
||||
filename: described?.filename ?? 'underlag.pdf',
|
||||
mimeType: described?.mimeType ?? 'application/octet-stream',
|
||||
bytes,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"id": "mail",
|
||||
"sector": "general",
|
||||
"exportName": "mailExtension",
|
||||
"entryPoint": "@/extensions/general/mail",
|
||||
"requiredEnvVars": ["GOOGLE_MAIL_CLIENT_ID", "GOOGLE_MAIL_CLIENT_SECRET"],
|
||||
"optionalEnvVars": ["MAIL_TOKEN_ENCRYPTION_KEY"],
|
||||
"npmDependencies": [],
|
||||
"definition": {
|
||||
"name": "Brevlådor",
|
||||
"category": "operations",
|
||||
"icon": "Mail",
|
||||
"dataPattern": "manual",
|
||||
"hasOwnData": true,
|
||||
"description": "Låt Kvittojakten leta upp kvitton i era brevlådor",
|
||||
"longDescription": "Koppla en eller flera brevlådor, så letar Kvittojakten själv upp kvitton till kortköp som saknar underlag. Åtkomsten är läsbehörighet: agenten kan aldrig skicka, ändra eller radera något i din mejl. Inkorgen kopieras aldrig, utan bara mejl som kan vara ett kvitto till ett visst köp hämtas i stunden och släpps igen. Det som blir underlag arkiveras i ert vanliga sjuåriga arkiv, efter att du godkänt det.",
|
||||
"subscriptionNotice": "Kräver ett Google-konto. Varje brevlåda kopplas av sin egen ägare och kan kopplas från när som helst."
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user