feat(parties): phase 0 prerequisites, pg_trgm and golden-set draw (#2157)
* feat(parties): phase 0 prerequisites, pg_trgm and golden-set draw Phase 0 of the Kontakter plan: make the counterparty resolver measurable before building it. - Migration 20260902120000 enables pg_trgm (trigram blocking of counterparty keys) and drops the two context-graph tables from 20260706193007 whose feature code was never merged and which prod no longer has, so fresh replays agree with prod. - tests/pg/parties-phase0.pg.test.ts pins the extension, a sanity check on trigram ranking, and the absence of the graph tables. - scripts/parties/draw-golden-set.sql is the reproducible, read-only draw of the 200-key labelling sample (three strata, md5-ordered) and the payee-identity base rate. The drawn rows contain customer voucher text and are kept in gitignored dev_docs, never in this public repo. - scripts/parties/README.md records the label vocabulary and the numbers measured on prod on 2026-09-02. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(archive): drop the two context-graph tables from the archive contract The migration in this PR removes graph_counterparties and graph_transaction_counterparties, so the full-archive contract must stop classifying them: tests/schema/no-phantom-columns.test.ts asserts that every classified table exists in the migration replay, and the live-DB twin in tests/pg/full-archive-coverage.pg.test.ts asserts the same against information_schema. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Jakob Wennberg
@@ -1491,3 +1491,4 @@ One line per decision: `[YYYY-MM-DD] <decision>: <why>`. Appended by agents and
|
||||
[2026-09-02] MCP OAuth redirect allowlist: registered URIs resolve with a service-role lookup bound explicitly to the consenting user (registrant or a colleague sharing a company) instead of relying on RLS select_own; DB-registered clients default to read-only pre-checks while the built-in Claude/ChatGPT clients keep the full pre-check; member role keeps approve with the SoD acknowledgement recorded at /token, mirroring settings/api-keys.
|
||||
[2026-09-02] BankID signup no longer returns a magic link: the account is created unconfirmed and the typed address must click a mailed link before bankid_linked (and the MFA exemption) is set. Chosen over a pending-signup table: one nullable column (bankid_identities.email_verified_at) and the existing auth callback carry the state.
|
||||
[2026-09-02] Removed the skattekonto drift email (skattekonto.drift_detected event, handler, /api/extensions/skatteverket/skattekonto/drift route, cron hook) instead of fixing it: it alerted on raw saldo-vs-1630 gaps that unbooked rows explain by construction (2026-09-02: Arcim 35 842 kr, 100% explained, while the Hem notice and reconciliation page said nothing was wrong), repeated every 24 h, and was the only surface of a May-2026 feature whose promised dashboard tile was never built. Since 2026-08-25 the reconciliation page and the Hem notice (detectSkvUnexplained, gated on unexplained_difference) are the surface. Considered gating the mail on unexplained_difference + once per episode (built, then dropped): after that gate it only fires on integrity findings the engine itself calls 'never a user task'. skattekonto_drift_tolerance stays (Hem notice reads it); stale skattekonto_drift_last_alert_at rows in extension_data are inert.
|
||||
[2026-09-02] parties phase 0, golden set stays out of git: the labelling sample is prod voucher text with person names (salary, expense claims) and the repo is public, so the draw SQL is versioned but the rows and labels live in gitignored dev_docs/parties/golden/.
|
||||
|
||||
@@ -1197,8 +1197,6 @@ export const ARCHIVE_EXCLUDED_TABLES: Record<string, string> = {
|
||||
'WORM verification log (SHA-256 recompute outcomes); failures reach the archive via audit_log in revision/behandlingshistorik.json',
|
||||
event_log: '30-day TTL event bus log',
|
||||
extension_data: 'extension runtime state (includes this backup\'s own state)',
|
||||
graph_counterparties: 'derived AI context graph, regenerable',
|
||||
graph_transaction_counterparties: 'derived AI context graph, regenerable',
|
||||
idempotency_keys: 'infrastructure',
|
||||
inbox_rate_counters: 'infrastructure',
|
||||
mail_connections:
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
# Parties, phase 0: measure before building
|
||||
|
||||
Phase 0 of the Kontakter plan (design doc: the "Vem du gör affärer med"
|
||||
artifact, 2026-09). Nothing in this phase changes product behaviour. It makes
|
||||
the resolver measurable so the model steps in later phases can be judged
|
||||
against labelled data instead of opinions.
|
||||
|
||||
## What ships in this phase
|
||||
|
||||
- Migration `20260902120000`: enables `pg_trgm` (trigram blocking of
|
||||
counterparty keys) and drops the two context-graph tables whose feature code
|
||||
was never merged and which prod no longer has. Test:
|
||||
`tests/pg/parties-phase0.pg.test.ts`.
|
||||
- `draw-golden-set.sql`: the reproducible draw of the labelling sample and the
|
||||
payee-change base rate. Read-only.
|
||||
|
||||
## The golden set (never in git)
|
||||
|
||||
The draw returns customer voucher text, which can contain person names. The
|
||||
repository is public, so the drawn rows and the labels live in
|
||||
`dev_docs/parties/golden/` (gitignored). Only aggregates and this vocabulary
|
||||
are versioned.
|
||||
|
||||
Label vocabulary for the pre-classifier, one per key:
|
||||
|
||||
| label | meaning | example |
|
||||
|---|---|---|
|
||||
| `party` | a real counterpart the company pays or invoices | `Levfakt Beijer Byggmaterial AB (2089)`, `Loopia` |
|
||||
| `category` | an expense description with no counterpart in it | `Inköp av varor`, `Banktjänster`, `Fika` |
|
||||
| `payroll` | salary, benefits, expense claims to a person | `Löneutbetalning: 2024.M.01 - anställd: 5` |
|
||||
| `adjustment` | periodisering, kostnadsföring, omföring, lagerförändring | `Periodisering av leverantörsfaktura 1311` |
|
||||
| `authority` | Skatteverket, Bolagsverket, Transportstyrelsen, kommun | `Bolagsverket ändra bolagsordning` |
|
||||
| `bank` | bank fees and bank products | `Baspaket Bank`, `Avgift Amex` |
|
||||
| `intermediary` | payment rail or marketplace, not the counterpart | `Klarna`, `Stripe`, `Zettle`, `AmazonMktplc` |
|
||||
| `unsure` | cannot tell from the text alone | |
|
||||
|
||||
A key whose text names a vendor but is booked as a category still gets
|
||||
`party`: the resolver decides identity, the account comes from the ledger.
|
||||
|
||||
## Measured on prod, 2026-09-02
|
||||
|
||||
- 113,032 imported expense vouchers across 385 real companies; 99.5% yield a
|
||||
key; 71% belong to a key that repeats within the company.
|
||||
- Payee-identity base rate: 365 suppliers have an established bankgiro on two
|
||||
or more documents; 13 of them (3.6%) carry two established bankgiro numbers,
|
||||
with an average span of 29 days between them, which reads as parallel
|
||||
accounts or OCR variance rather than changes over time. Any distinct value
|
||||
at all: 8.4%. Consequence for the signal: a new bankgiro is rare enough to
|
||||
warrant a "verify" sentence, never a block, and a value must be seen on two
|
||||
documents before it counts as known.
|
||||
|
||||
## Still to confirm in this phase
|
||||
|
||||
- Which registration flags the free SCB Företagsregistret API exposes
|
||||
(moms, arbetsgivare, F-skatt). This decides how much of rung three can skip
|
||||
TIC.
|
||||
- The pre-classifier and selection agreement bars (0.85 on the labelled set)
|
||||
are enforced by the shadow harness in phase 1, not here.
|
||||
@@ -0,0 +1,77 @@
|
||||
-- Parties, phase 0: draw the golden set for the counterparty resolver.
|
||||
--
|
||||
-- Read-only. Run against prod through the Supabase MCP (execute_sql) or psql.
|
||||
-- The OUTPUT contains customer voucher text, which can include person names
|
||||
-- (salary vouchers, expense claims). It must never be committed to this
|
||||
-- public repository: write it to dev_docs/parties/golden/ (gitignored) and
|
||||
-- label it there. Only aggregate results and the labelling vocabulary belong
|
||||
-- in the repo.
|
||||
--
|
||||
-- Scope: posted SIE-imported vouchers (source_type = 'import') with at least
|
||||
-- one expense debit line (4xxx-7xxx), sandbox tenants excluded. Keys are the
|
||||
-- shipped normalize_counterparty_key() so the sample matches what the
|
||||
-- resolver will see. Three strata so the set is not dominated by one tenant:
|
||||
-- fleet = key seen in 5+ companies (60 rows)
|
||||
-- heavy = 20+ vouchers in fewer companies (60 rows)
|
||||
-- tail = everything else with 3+ vouchers (80 rows)
|
||||
-- Ordering by md5(key) makes the draw deterministic and re-runnable.
|
||||
|
||||
-- 1. Keys for the pre-classifier (party / category / payroll / adjustment /
|
||||
-- authority / bank / intermediary / unsure).
|
||||
WITH real_co AS (SELECT id FROM companies WHERE name NOT ILIKE '%sandl%'),
|
||||
scope AS (
|
||||
SELECT je.id, je.company_id, je.description, public.normalize_counterparty_key(je.description) AS k,
|
||||
(SELECT l.account_number FROM journal_entry_lines l
|
||||
WHERE l.journal_entry_id = je.id AND l.account_number ~ '^[4-7][0-9]{3}$'
|
||||
ORDER BY l.debit_amount DESC LIMIT 1) AS acct,
|
||||
(SELECT sum(l.debit_amount) FROM journal_entry_lines l
|
||||
WHERE l.journal_entry_id = je.id AND l.account_number ~ '^[4-7][0-9]{3}$') AS sek
|
||||
FROM journal_entries je JOIN real_co c ON c.id = je.company_id
|
||||
WHERE je.status = 'posted' AND je.source_type = 'import'
|
||||
AND EXISTS (SELECT 1 FROM journal_entry_lines l
|
||||
WHERE l.journal_entry_id = je.id AND l.account_number ~ '^[4-7][0-9]{3}$' AND l.debit_amount > 0)
|
||||
),
|
||||
keys AS (
|
||||
SELECT k, count(*) AS n, count(DISTINCT company_id) AS cos,
|
||||
mode() WITHIN GROUP (ORDER BY description) AS example,
|
||||
mode() WITHIN GROUP (ORDER BY acct) AS acct, round(sum(sek)) AS sek,
|
||||
(array_agg(DISTINCT left(description, 60)))[1:3] AS variants
|
||||
FROM scope WHERE k <> '' GROUP BY k HAVING count(*) >= 3
|
||||
),
|
||||
strata AS (
|
||||
SELECT *,
|
||||
CASE WHEN cos >= 5 THEN 'fleet' WHEN n >= 20 THEN 'heavy' ELSE 'tail' END AS stratum,
|
||||
row_number() OVER (PARTITION BY (CASE WHEN cos >= 5 THEN 'fleet' WHEN n >= 20 THEN 'heavy' ELSE 'tail' END)
|
||||
ORDER BY md5(k)) AS rn
|
||||
FROM keys
|
||||
)
|
||||
SELECT stratum, k, example, n, cos, acct, sek, variants
|
||||
FROM strata
|
||||
WHERE (stratum = 'fleet' AND rn <= 60) OR (stratum = 'heavy' AND rn <= 60) OR (stratum = 'tail' AND rn <= 80)
|
||||
ORDER BY stratum, rn;
|
||||
|
||||
-- 2. Base rate of supplier bankgiro changes in our own documents, to set the
|
||||
-- severity of the payee-identity signal. "Established" = a bankgiro seen on
|
||||
-- at least two documents for the same org number; a single differing value
|
||||
-- is treated as OCR noise or a one-off account.
|
||||
WITH real_co AS (SELECT id FROM companies WHERE name NOT ILIKE '%sandl%'),
|
||||
docs AS (
|
||||
SELECT d.company_id,
|
||||
regexp_replace(coalesce(d.extracted_data->'supplier'->>'orgNumber',''), '[^0-9]', '', 'g') AS org,
|
||||
regexp_replace(coalesce(d.extracted_data->'supplier'->>'bankgiro',''), '[^0-9]', '', 'g') AS bg,
|
||||
d.created_at::date AS seen
|
||||
FROM document_attachments d JOIN real_co c ON c.id = d.company_id
|
||||
WHERE length(regexp_replace(coalesce(d.extracted_data->'supplier'->>'bankgiro',''), '[^0-9]', '', 'g')) BETWEEN 7 AND 8
|
||||
AND length(regexp_replace(coalesce(d.extracted_data->'supplier'->>'orgNumber',''), '[^0-9]', '', 'g')) = 10
|
||||
),
|
||||
bg_counts AS (SELECT company_id, org, bg, count(*) AS n, min(seen) AS first_seen, max(seen) AS last_seen FROM docs GROUP BY 1,2,3),
|
||||
established AS (SELECT * FROM bg_counts WHERE n >= 2),
|
||||
per_supplier AS (
|
||||
SELECT company_id, org, count(*) AS established_bgs, sum(n) AS docs, max(last_seen) - min(first_seen) AS span_days
|
||||
FROM established GROUP BY 1,2
|
||||
)
|
||||
SELECT count(*) AS suppliers_with_established_bg,
|
||||
count(*) FILTER (WHERE established_bgs >= 2) AS suppliers_with_2_established_bgs,
|
||||
round(100.0 * count(*) FILTER (WHERE established_bgs >= 2) / nullif(count(*),0), 1) AS pct,
|
||||
round(avg(span_days) FILTER (WHERE established_bgs >= 2)) AS avg_span_days_when_changed
|
||||
FROM per_supplier;
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
-- Parties, phase 0: prerequisites for the counterparty resolver.
|
||||
--
|
||||
-- 1. pg_trgm for trigram blocking of counterparty keys. Candidate generation
|
||||
-- runs before any model call and must be cheap; trigram similarity over
|
||||
-- normalised voucher and bank text is the first stage. Installed in the
|
||||
-- `extensions` schema like the other Supabase extensions in this repo.
|
||||
--
|
||||
-- 2. Drop the two context-graph tables from 20260706193007. Their feature
|
||||
-- code (lib/graph/) was never merged and the tables were dropped on prod
|
||||
-- during the context-graph revert, so every fresh replay (CI, preview
|
||||
-- branches, local stacks) has been creating two tables prod does not have.
|
||||
-- The parties layer keys the same identity (company_id + normalised name)
|
||||
-- and must not coexist with a second, divergent substrate. Child first:
|
||||
-- graph_transaction_counterparties references graph_counterparties.
|
||||
|
||||
CREATE EXTENSION IF NOT EXISTS pg_trgm WITH SCHEMA extensions;
|
||||
|
||||
DROP TABLE IF EXISTS public.graph_transaction_counterparties;
|
||||
DROP TABLE IF EXISTS public.graph_counterparties;
|
||||
@@ -0,0 +1,36 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { getPool } from './setup'
|
||||
|
||||
/**
|
||||
* Parties, phase 0 (migration 20260902120000): pg_trgm is available for
|
||||
* trigram blocking of counterparty keys, and the two context-graph tables
|
||||
* whose feature code was never merged are gone, so local replays agree with
|
||||
* prod (where they were dropped during the context-graph revert).
|
||||
*/
|
||||
describe('parties phase 0 (pg)', () => {
|
||||
it('pg_trgm is installed', async () => {
|
||||
const { rows } = await getPool().query<{ n: number }>(
|
||||
`SELECT count(*)::int AS n FROM pg_extension WHERE extname = 'pg_trgm'`,
|
||||
)
|
||||
expect(rows[0]!.n).toBe(1)
|
||||
})
|
||||
|
||||
it('trigram similarity ranks the same vendor above a different one', async () => {
|
||||
const { rows } = await getPool().query<{ same: number; other: number }>(
|
||||
`SELECT extensions.similarity('levfakt beijer byggmaterial 097', 'beijer byggmaterial') AS same,
|
||||
extensions.similarity('levfakt beijer byggmaterial 097', 'skelleftea plat') AS other`,
|
||||
)
|
||||
expect(Number(rows[0]!.same)).toBeGreaterThan(Number(rows[0]!.other))
|
||||
expect(Number(rows[0]!.same)).toBeGreaterThan(0.3)
|
||||
})
|
||||
|
||||
it('the reverted context-graph tables are gone', async () => {
|
||||
const { rows } = await getPool().query<{ n: number }>(
|
||||
`SELECT count(*)::int AS n
|
||||
FROM pg_tables
|
||||
WHERE schemaname = 'public'
|
||||
AND tablename IN ('graph_counterparties', 'graph_transaction_counterparties')`,
|
||||
)
|
||||
expect(rows[0]!.n).toBe(0)
|
||||
})
|
||||
})
|
||||
Reference in New Issue
Block a user