admin/accounted
3
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(parties): selection-step evaluation against document-anchored truth (#2169)

Browse Source 
* feat(parties): selection-step evaluation against document-anchored truth

Scores which similar keys are the same party without human labels: every
key in the set carries an org number OCR-read from a linked invoice, so two
keys are the same party exactly when the org numbers agree. Rules and the
Bedrock model both land at 0.91 pair precision; the residual false merges
are different legal entities sharing a trade name, which text cannot and
should not separate. Results and caveats in the README.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* chore(parties): make the model selector opt-in in the selection eval

Sending voucher key text to the AI provider now requires --llm; the
default run scores the rules selector only and makes no network call.
Documents that the opt-in path uses the same configured provider the
production categorizer already sends the same text to.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Jakob Wennberg
2026-09-02 17:38:20 +02:00
committed by GitHub
co-authored by Claude Fable 5.1 Jakob Wennberg
parent 5291806c37
commit 69d3bba587
3 changed files with 302 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/parties/README.md
+32
View File
@@ -96,6 +96,38 @@ the default for that reason. The model's remaining party misses are text
with a person's name or a card descriptor, exactly where a hard key from a
document would decide instead.
## Selection-step evaluation against document-anchored truth, 2026-09-02
`eval-selection.ts` scores the selection step (which similar keys in the same
company are the same party) without human labels: every key in the set has an
org number read by OCR from an invoice linked to its voucher, so two keys are
the same party exactly when the org numbers agree. Keys mapping to several
org numbers are excluded. 225 anchors, 1,038 candidate pairs (747 same, 291
different), 44 anchors with no true match. Draw: `draw-selection-gold.sql`.
The model selector is opt-in (`--llm`): it sends the same voucher text the
production categorizer already sends to the configured AI provider (Bedrock EU
on hosted), so run it only with an env file whose AI settings you have checked.
| Selector | Pair precision | Pair recall | Non-match recall | Anchors fully right | "None" precision / recall |
|---|---|---|---|---|---|
| Rules (same core name) | 0.909 | 0.776 | 0.801 | 0.547 | 0.40 / 0.86 |
| Sonnet 5 on Bedrock EU, zero-shot | 0.910 | 0.904 | 0.770 | 0.756 | 0.68 / 0.77 |
What the false merges are: both selectors merge "Levfakt Fortnox (325)" with
"Levfakt Fortnox (154)", "Vattenfall Kundservice" with "Vattenfall
Eldistribution", "Jämtkraft" with "Jämtkraft" under another org number. Those
are different legal entities sharing a trade name, or OCR variance in the org
number. Text cannot separate them and should not try: the hard key from the
document is the referee, which is exactly the resolver's precedence order.
What the model catches that the rules miss: "lån themax" versus "lån från
themax", and paraphrases with a changed word order.
Read with one caveat: sales-side vouchers ("kundbet ...") contribute noisy
truth because the org number on the linked document can be the customer's;
the next draw restricts to expense-side vouchers.
Cost: about 1,300 input tokens per anchor, 298k tokens for the run.
## Still to confirm in this phase
- SCB access: the free Företagsregistret API carries F-skattstatus,