723a0f537b
* feat(parties): shadow evaluation of the key pre-classifier Scores a deterministic rule router and the Bedrock model router (zero-shot and with twenty founder examples) against the founder-labelled golden set, on the same held-out rows, reporting strict agreement plus party TPR/TNR. Read-only: reads the gitignored JSONL, calls getAiService(), writes a report next to the input, never opens a database connection. Results and the definitional disagreements are recorded in the README. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(parties): settle the four edge rules from the first labelling round Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
106 lines
5.5 KiB
Markdown
106 lines
5.5 KiB
Markdown
# Parties, phase 0: measure before building
|
|
|
|
Phase 0 of the Kontakter plan (design doc: the "Vem du gör affärer med"
|
|
artifact, 2026-09). Nothing in this phase changes product behaviour. It makes
|
|
the resolver measurable so the model steps in later phases can be judged
|
|
against labelled data instead of opinions.
|
|
|
|
## What ships in this phase
|
|
|
|
- Migration `20260902120000`: enables `pg_trgm` (trigram blocking of
|
|
counterparty keys) and drops the two context-graph tables whose feature code
|
|
was never merged and which prod no longer has. Test:
|
|
`tests/pg/parties-phase0.pg.test.ts`.
|
|
- `draw-golden-set.sql`: the reproducible draw of the labelling sample and the
|
|
payee-change base rate. Read-only.
|
|
|
|
## The golden set (never in git)
|
|
|
|
The draw returns customer voucher text, which can contain person names. The
|
|
repository is public, so the drawn rows and the labels live in
|
|
`dev_docs/parties/golden/` (gitignored). Only aggregates and this vocabulary
|
|
are versioned.
|
|
|
|
Label vocabulary for the pre-classifier, one per key:
|
|
|
|
| label | meaning | example |
|
|
|---|---|---|
|
|
| `party` | a real counterpart the company pays or invoices | `Levfakt Beijer Byggmaterial AB (2089)`, `Loopia` |
|
|
| `category` | an expense description with no counterpart in it | `Inköp av varor`, `Banktjänster`, `Fika` |
|
|
| `payroll` | salary, benefits, expense claims to a person | `Löneutbetalning: 2024.M.01 - anställd: 5` |
|
|
| `adjustment` | periodisering, kostnadsföring, omföring, lagerförändring | `Periodisering av leverantörsfaktura 1311` |
|
|
| `authority` | Skatteverket, Bolagsverket, Transportstyrelsen, kommun | `Bolagsverket ändra bolagsordning` |
|
|
| `bank` | bank fees and bank products | `Baspaket Bank`, `Avgift Amex` |
|
|
| `intermediary` | payment rail or marketplace, not the counterpart | `Klarna`, `Stripe`, `Zettle`, `AmazonMktplc` |
|
|
| `unsure` | cannot tell from the text alone | |
|
|
|
|
A key whose text names a vendor but is booked as a category still gets
|
|
`party`: the resolver decides identity, the account comes from the ledger.
|
|
|
|
Edge rules settled with the founder on 2026-09-02, after the first labelling
|
|
round:
|
|
|
|
- State fees and taxes where the counterpart is fixed (Bolagsverket
|
|
registration fees, Skatteverket, Transportstyrelsen vehicle tax) are
|
|
`authority`. A kommun or a state company that invoices for a service is a
|
|
`party` like any supplier.
|
|
- A marketplace or payment processor that the company actually pays (Amazon
|
|
Marketplace purchases, Stripe or Amex fees) is a `party`. `intermediary` is
|
|
reserved for a rail that only carries someone else's money: Klarna, Swish
|
|
and Zettle payouts, "försäljning via" lines.
|
|
- A card-platform line that carries an employee's name (Pleo, Google Ads with
|
|
the cardholder appended) is a `party`; the name is the cardholder, not the
|
|
counterpart.
|
|
- Insurance and pension premiums are `party`: the insurer is the counterpart,
|
|
whatever the 74xx account says.
|
|
|
|
## Measured on prod, 2026-09-02
|
|
|
|
- 113,032 imported expense vouchers across 385 real companies; 99.5% yield a
|
|
key; 71% belong to a key that repeats within the company.
|
|
- Payee-identity base rate: 365 suppliers have an established bankgiro on two
|
|
or more documents; 13 of them (3.6%) carry two established bankgiro numbers,
|
|
with an average span of 29 days between them, which reads as parallel
|
|
accounts or OCR variance rather than changes over time. Any distinct value
|
|
at all: 8.4%. Consequence for the signal: a new bankgiro is rare enough to
|
|
warrant a "verify" sentence, never a block, and a value must be seen on two
|
|
documents before it counts as known.
|
|
|
|
## Pre-classifier shadow evaluation, 2026-09-02
|
|
|
|
`eval-preclassifier.ts` scores three routers on the same 180 held-out keys
|
|
(20 rows, chosen by md5 of the key, serve as few-shot examples and are never
|
|
scored). Rows the founder marked `unsure` always receive a label, so the
|
|
honest number is the one that excludes them.
|
|
|
|
| Router | Strict agreement | Excluding founder-unsure | Party TPR | Party TNR |
|
|
|---|---|---|---|---|
|
|
| Rules v0, lexicon from BAS account names | 0.917 | 0.965 | 0.992 | 0.930 |
|
|
| Rules v0, lexicon incl. BAS descriptions | 0.911 | 0.959 | 0.984 | 0.930 |
|
|
| Sonnet 5 on Bedrock EU, zero-shot | 0.861 | 0.906 | 0.906 | 0.977 |
|
|
| Sonnet 5 on Bedrock EU, 20 founder examples | 0.878 | 0.924 | 0.930 | 0.977 |
|
|
|
|
Read with two caveats. The rules were written after the labels were seen,
|
|
so their score is optimistic; the model scores are honest. And most of the
|
|
remaining disagreements are definitions, not errors: whether a Bolagsverket
|
|
fee is `authority` or `intermediary`, whether a marketplace or payment
|
|
processor (Amazon Marketplace, Stripe, Amex fees) is `party` or
|
|
`intermediary`, whether a card-platform line with a person's name (Pleo,
|
|
Google Ads) is `party` or `payroll`, and whether a pension insurance premium
|
|
is `party` or `payroll`. Settle those in the vocabulary above, relabel the
|
|
handful of rows, and re-run.
|
|
|
|
BAS description tokens in the lexicon make Google and Facebook look generic
|
|
because the descriptions name them as examples; the account-name lexicon is
|
|
the default for that reason. The model's remaining party misses are text
|
|
with a person's name or a card descriptor, exactly where a hard key from a
|
|
document would decide instead.
|
|
|
|
## Still to confirm in this phase
|
|
|
|
- SCB access: the free Företagsregistret API carries F-skattstatus,
|
|
Momsstatus and Arbetsgivarstatus (verified on scb.se 2026-09-02); access
|
|
requested by email, credentials pending.
|
|
- The pre-classifier and selection agreement bars (0.85 on the labelled set)
|
|
are enforced by the shadow harness in phase 1, not here.
|