Pressing Leta produced mails=25, documents=0 on a real two-mailbox run. Nothing was found because nothing was searched: every request came back 429 "Too many concurrent requests for user". Two bugs, and the second is the one that matters. The search fanned out with Promise.all over every message id at once, one Gmail request per message, per connection. Gmail enforces a per-user concurrency ceiling as well as a daily quota, and this sailed past it long before any volume worth worrying about. It now runs through a pool of five per connection, which is comfortably under and still finishes a page of results in a couple of round trips. The catch turned each refusal into an empty array, with a comment saying one mailbox's failure must not become the company's. Right instinct, wrong consequence: an empty array is also what an empty mailbox returns, and the manual hunt loop stops on fetched === 0 because that is its signal for "the mailboxes hold nothing more for what is open". So a rate-limited search told the user their receipts do not exist, and stopped looking. searchFailureCount() now separates "could not look" from "nothing there". The run route reports it, and the loop treats a pass with failures as failed rather than finished, so pressing again is the obvious next move instead of a pointless one. This is the failure this feature exists to catch, happening inside the feature: silence that reads as an answer. Restoring the unbounded fan-out fails one test; removing the failure counter fails three. Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
110 lines
3.7 KiB
TypeScript
110 lines
3.7 KiB
TypeScript
'use client'
|
|
|
|
/**
|
|
* Running the receipt hunt from a button.
|
|
*
|
|
* The loop was written for the settings panel and lived there. It belongs to
|
|
* neither surface: the Underlag page is where a person notices that receipts
|
|
* are missing, and asking them to walk to Settings to press the button that
|
|
* fixes it is the kind of seam that makes a feature look broken. Both call this
|
|
* now, so there is one definition of what a run does and when it stops.
|
|
*
|
|
* Why a client loop rather than one long request: a single pass is bounded by
|
|
* the serverless ceiling, and each receipt costs a download plus a model read,
|
|
* which is far too slow to clear a backlog inside one invocation. A queue
|
|
* drained by cron would be the other option, but the finest schedule this app
|
|
* runs is hourly, so pressing the button would mean waiting an hour.
|
|
*
|
|
* It stops when a pass finds nothing new. That is the honest signal that the
|
|
* mailboxes hold nothing more for the purchases still open, and it is why the
|
|
* cap below is a backstop rather than a budget.
|
|
*/
|
|
import { useCallback, useRef, useState } from 'react'
|
|
|
|
export interface HuntResult {
|
|
searched: number
|
|
fetched: number
|
|
proposed: number
|
|
remaining: number
|
|
failed?: boolean
|
|
/** Connections that refused the search. See the stop condition below. */
|
|
searchFailures?: number
|
|
}
|
|
|
|
export interface HuntProgress {
|
|
passes: number
|
|
fetched: number
|
|
proposed: number
|
|
}
|
|
|
|
/**
|
|
* Each pass fetches a few receipts, so this is far more than any real backlog
|
|
* needs. It exists so a pass that keeps reporting work it never completes
|
|
* cannot run forever.
|
|
*/
|
|
export const MAX_PASSES = 25
|
|
|
|
export function useReceiptHunt(onPass?: () => void) {
|
|
const [hunting, setHunting] = useState(false)
|
|
const [progress, setProgress] = useState<HuntProgress | null>(null)
|
|
const [result, setResult] = useState<HuntResult | null>(null)
|
|
const stopped = useRef(false)
|
|
|
|
const stop = useCallback(() => {
|
|
stopped.current = true
|
|
}, [])
|
|
|
|
const hunt = useCallback(async () => {
|
|
setHunting(true)
|
|
setResult(null)
|
|
stopped.current = false
|
|
|
|
let passes = 0
|
|
let fetched = 0
|
|
let proposed = 0
|
|
|
|
try {
|
|
while (!stopped.current && passes < MAX_PASSES) {
|
|
const response = await fetch('/api/receipt-hunt/run', { method: 'POST' })
|
|
if (!response.ok) {
|
|
setResult({ searched: 0, fetched, proposed, remaining: 0, failed: true })
|
|
return
|
|
}
|
|
|
|
const body = (await response.json()) as { data: HuntResult }
|
|
passes++
|
|
fetched += body.data.fetched
|
|
proposed += body.data.proposed
|
|
setProgress({ passes, fetched, proposed })
|
|
// Let the caller refresh whatever the pass just changed, so a long run
|
|
// fills the list as it goes instead of all at once at the end.
|
|
onPass?.()
|
|
|
|
// Nothing new this pass: the mailboxes have no more for what is open.
|
|
//
|
|
// Unless a mailbox refused to be read, in which case zero fetched says
|
|
// nothing about what is in there. Stopping on it, and reporting it as
|
|
// "hittade inget", would tell the user their receipts do not exist
|
|
// because Gmail was busy. Treat it as a failure and let them retry.
|
|
if ((body.data.searchFailures ?? 0) > 0) {
|
|
setResult({ ...body.data, fetched, proposed, failed: true })
|
|
return
|
|
}
|
|
if (body.data.fetched === 0) {
|
|
setResult({ ...body.data, fetched, proposed })
|
|
return
|
|
}
|
|
}
|
|
|
|
setResult({ searched: 0, fetched, proposed, remaining: 0 })
|
|
} catch {
|
|
setResult({ searched: 0, fetched, proposed, remaining: 0, failed: true })
|
|
} finally {
|
|
setHunting(false)
|
|
setProgress(null)
|
|
}
|
|
}, [onPass])
|
|
|
|
return { hunt, stop, hunting, progress, result, setResult }
|
|
}
|