Ai/full autonomous flow (#359)
* Refactor bookkeeping error handling and introduce new error classes - Introduced new error classes for better error categorization: - JournalEntryNotBalancedError - FiscalPeriodNotFoundError - EntryDateOutsideFiscalPeriodError - JournalEntryNotFoundError - CannotReverseNonPostedError - CannotCorrectNonPostedError - EntryAlreadyReversedError - CurrencyRevaluationAlreadyExistsError - InvalidMappingResultError - BookkeepingDatabaseError - Updated existing functions in engine.ts and transaction-entries.ts to throw specific errors instead of generic ones. - Enhanced error response handling in get-error-message.ts to provide localized messages for new error types. - Added unit tests for new error classes and error handling functions to ensure correctness and coverage. * feat(ai): implement AI proposal application and persistence - Add apply.ts to handle the application of AI proposals, including match and booking steps. - Introduce persist.ts for inserting and managing AI requests and proposals, ensuring unique constraints. - Create re-validate.ts for validating proposals before acceptance, checking for stale conditions. - Define database migrations for ai_requests and ai_proposals tables, including constraints and indexes. - Enhance journal_entries with AI provenance tracking, linking entries to AI proposals. - Update categorization_templates to distinguish AI-corrected templates. - Add company settings for toggling AI flow and managing backfill processes. - Extend processing_history to include AI-related events for better tracking. * feat: add uncategorized transactions API and UI for transaction selection - Implemented a new API endpoint for fetching uncategorized transactions with pagination and filtering options. - Created ChangeTransactionDialog component for selecting alternative transactions based on AI proposals. - Developed ReceiptDetailDialog to display detailed information about receipts, including upload functionality. - Added TransactionDetailDialog for viewing transaction details with links to the transaction list. - Introduced receipt quality assessment logic to evaluate extracted receipt data. - Implemented feature flagging for the AI bookkeeping agent to control availability in different environments. * feat: add manual receipt extraction dialog and integrate AWS Textract for expense analysis - Added ManualExtractDialog component for user input when AI fails to extract receipt data. - Implemented ReceiptsList component to manage and display uploaded receipts, including upload and rescan functionalities. - Introduced Textract integration for analyzing expenses, extracting fields like total, vendor, and date. - Updated package.json to include @aws-sdk/client-textract dependency. * fix(ai): handle livsmedel VAT transition (12% → 6%) in booking prompt and re-validate guard Add date-aware guidance to BOOKING_SYSTEM_PROMPT for the temporary livsmedel VAT cut (Prop. 2025/26:55, 2026-04-01 to 2027-12-31), with restaurang/servering carve-out at 12%. Add a re-validate safety net that rejects clearly-stale rate labels for grocery-chain merchants relative to the entry date. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Mattsson
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
ab63da8324
commit
1af977950b
@@ -0,0 +1,77 @@
|
||||
-- ai_requests: structured asks from the AI agent to the user.
|
||||
--
|
||||
-- When the AI agent cannot produce a proposal because something is missing or
|
||||
-- ambiguous (blurry receipt, no candidate transactions, uncertain VAT), it
|
||||
-- creates an ai_requests row instead of an ai_proposals row. The UI renders
|
||||
-- these as actionable cards with typed forms.
|
||||
--
|
||||
-- One open request per (subject, request_type) enforced by a partial unique
|
||||
-- index so the orchestrator can safely re-issue on retries without creating
|
||||
-- duplicates.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS public.ai_requests (
|
||||
id uuid PRIMARY KEY DEFAULT uuid_generate_v4(),
|
||||
company_id uuid NOT NULL REFERENCES public.companies(id) ON DELETE CASCADE,
|
||||
|
||||
-- What the request is about
|
||||
subject_type text NOT NULL
|
||||
CHECK (subject_type IN ('inbox_item')),
|
||||
subject_id uuid NOT NULL,
|
||||
|
||||
-- What the AI is asking for
|
||||
request_type text NOT NULL
|
||||
CHECK (request_type IN (
|
||||
'reupload_document',
|
||||
'pick_transaction',
|
||||
'specify_vat',
|
||||
'clarify_business_private',
|
||||
'needs_manual'
|
||||
)),
|
||||
message text NOT NULL,
|
||||
required_fields jsonb,
|
||||
options jsonb,
|
||||
|
||||
-- Lifecycle
|
||||
status text NOT NULL DEFAULT 'open'
|
||||
CHECK (status IN ('open', 'resolved', 'dismissed')),
|
||||
response_json jsonb,
|
||||
resolved_at timestamptz,
|
||||
resolved_by_user_id uuid REFERENCES auth.users(id) ON DELETE SET NULL,
|
||||
|
||||
-- Provenance
|
||||
model text,
|
||||
prompt_version text,
|
||||
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
updated_at timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
-- Only one open request per (subject, request_type) at a time
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_ai_requests_one_open_per_subject_type
|
||||
ON public.ai_requests (subject_type, subject_id, request_type)
|
||||
WHERE status = 'open';
|
||||
|
||||
-- Lookup by company
|
||||
CREATE INDEX IF NOT EXISTS idx_ai_requests_company_status
|
||||
ON public.ai_requests (company_id, status);
|
||||
|
||||
-- Lookup by subject (for cascading when the inbox item is processed)
|
||||
CREATE INDEX IF NOT EXISTS idx_ai_requests_subject
|
||||
ON public.ai_requests (subject_type, subject_id);
|
||||
|
||||
-- RLS: company-scoped using user_company_ids()
|
||||
ALTER TABLE public.ai_requests ENABLE ROW LEVEL SECURITY;
|
||||
|
||||
CREATE POLICY "ai_requests_select" ON public.ai_requests
|
||||
FOR SELECT USING (company_id IN (SELECT public.user_company_ids()));
|
||||
CREATE POLICY "ai_requests_insert" ON public.ai_requests
|
||||
FOR INSERT WITH CHECK (company_id IN (SELECT public.user_company_ids()));
|
||||
CREATE POLICY "ai_requests_update" ON public.ai_requests
|
||||
FOR UPDATE USING (company_id IN (SELECT public.user_company_ids()));
|
||||
|
||||
-- updated_at trigger
|
||||
CREATE TRIGGER ai_requests_updated_at
|
||||
BEFORE UPDATE ON public.ai_requests
|
||||
FOR EACH ROW EXECUTE FUNCTION public.update_updated_at_column();
|
||||
|
||||
NOTIFY pgrst, 'reload schema';
|
||||
@@ -0,0 +1,92 @@
|
||||
-- ai_proposals: the staging layer for AI-generated bookkeeping proposals.
|
||||
--
|
||||
-- When the AI agent can produce a concrete suggestion for a step in the
|
||||
-- receipt flow (match, booking), it writes a row here with status='pending'.
|
||||
-- The user accepts, rejects, edits, or skips via the /agent-inbox UI.
|
||||
-- Nothing touches the ledger until a pending proposal is explicitly accepted;
|
||||
-- at that point the apply path calls the engine and links applied_entry_id.
|
||||
--
|
||||
-- A partial unique index enforces "one pending proposal per (subject, step)"
|
||||
-- so concurrent generation is idempotent — a new proposal for an already-
|
||||
-- pending (subject, step) pair invalidates the prior one first.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS public.ai_proposals (
|
||||
id uuid PRIMARY KEY DEFAULT uuid_generate_v4(),
|
||||
company_id uuid NOT NULL REFERENCES public.companies(id) ON DELETE CASCADE,
|
||||
user_id uuid NOT NULL REFERENCES auth.users(id) ON DELETE CASCADE,
|
||||
|
||||
-- Subject: what this proposal is about
|
||||
subject_type text NOT NULL
|
||||
CHECK (subject_type IN ('inbox_item')),
|
||||
subject_id uuid NOT NULL,
|
||||
|
||||
-- Step in the agent pipeline: 'match' (document -> transaction) then 'booking' (journal entry)
|
||||
step_type text NOT NULL
|
||||
CHECK (step_type IN ('match', 'booking')),
|
||||
|
||||
-- Lifecycle
|
||||
status text NOT NULL DEFAULT 'pending'
|
||||
CHECK (status IN ('pending', 'accepted', 'rejected', 'skipped', 'invalidated')),
|
||||
version integer NOT NULL DEFAULT 1, -- optimistic-lock counter
|
||||
|
||||
-- Payload: step-shaped JSON (MatchProposalPayload | BookingProposalPayload)
|
||||
proposal_json jsonb NOT NULL,
|
||||
|
||||
-- Confidence is informational only — user always confirms
|
||||
confidence numeric(5,4)
|
||||
CHECK (confidence IS NULL OR (confidence >= 0 AND confidence <= 1)),
|
||||
reasoning text,
|
||||
|
||||
-- Link to an open ai_request when the AI would rather ask than guess
|
||||
ai_request_id uuid REFERENCES public.ai_requests(id) ON DELETE SET NULL,
|
||||
|
||||
-- Provenance (for audit + prompt/model drift analysis)
|
||||
model text NOT NULL,
|
||||
prompt_version text NOT NULL,
|
||||
input_token_count integer NOT NULL DEFAULT 0,
|
||||
output_token_count integer NOT NULL DEFAULT 0,
|
||||
|
||||
-- Outcome tracking
|
||||
edit_diff jsonb, -- set when user edited before accept
|
||||
applied_entry_id uuid REFERENCES public.journal_entries(id) ON DELETE SET NULL,
|
||||
invalidated_reason text,
|
||||
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
accepted_at timestamptz,
|
||||
accepted_by_user_id uuid REFERENCES auth.users(id) ON DELETE SET NULL,
|
||||
rejected_at timestamptz,
|
||||
updated_at timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
-- One pending proposal per (subject, step) — idempotency guard
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_ai_proposals_one_pending_per_step
|
||||
ON public.ai_proposals (subject_type, subject_id, step_type)
|
||||
WHERE status = 'pending';
|
||||
|
||||
-- List queries
|
||||
CREATE INDEX IF NOT EXISTS idx_ai_proposals_company_status
|
||||
ON public.ai_proposals (company_id, status);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_ai_proposals_company_created_at
|
||||
ON public.ai_proposals (company_id, created_at DESC);
|
||||
|
||||
-- Subject lookup (cascade when the inbox item is processed manually)
|
||||
CREATE INDEX IF NOT EXISTS idx_ai_proposals_subject
|
||||
ON public.ai_proposals (subject_type, subject_id);
|
||||
|
||||
-- RLS: company-scoped using user_company_ids()
|
||||
ALTER TABLE public.ai_proposals ENABLE ROW LEVEL SECURITY;
|
||||
|
||||
CREATE POLICY "ai_proposals_select" ON public.ai_proposals
|
||||
FOR SELECT USING (company_id IN (SELECT public.user_company_ids()));
|
||||
CREATE POLICY "ai_proposals_insert" ON public.ai_proposals
|
||||
FOR INSERT WITH CHECK (company_id IN (SELECT public.user_company_ids()));
|
||||
CREATE POLICY "ai_proposals_update" ON public.ai_proposals
|
||||
FOR UPDATE USING (company_id IN (SELECT public.user_company_ids()));
|
||||
|
||||
-- updated_at trigger
|
||||
CREATE TRIGGER ai_proposals_updated_at
|
||||
BEFORE UPDATE ON public.ai_proposals
|
||||
FOR EACH ROW EXECUTE FUNCTION public.update_updated_at_column();
|
||||
|
||||
NOTIFY pgrst, 'reload schema';
|
||||
@@ -0,0 +1,47 @@
|
||||
-- AI provenance on journal entries.
|
||||
--
|
||||
-- Adds two columns to journal_entries so that entries posted via the AI
|
||||
-- agent flow carry a BFL-defensible audit trail: who created this entry,
|
||||
-- and which AI proposal did the user approve to produce it?
|
||||
--
|
||||
-- `created_via` is informational — it describes the *method* of creation,
|
||||
-- not the business event. The existing `source_type` column still describes
|
||||
-- the business event (bank_transaction, invoice_created, supplier_invoice_
|
||||
-- registered, etc.). An AI-proposed booking for a bank transaction will
|
||||
-- have source_type='bank_transaction' AND created_via='ai_proposed'.
|
||||
--
|
||||
-- The existing immutability trigger (migration 017) prevents changes to
|
||||
-- posted entries. These new columns are set while the entry is still in
|
||||
-- draft status and frozen at commit — consistent with how the trigger
|
||||
-- already treats other fields.
|
||||
|
||||
ALTER TABLE public.journal_entries
|
||||
ADD COLUMN IF NOT EXISTS created_via text NOT NULL DEFAULT 'manual';
|
||||
|
||||
ALTER TABLE public.journal_entries
|
||||
ALTER COLUMN created_via SET DEFAULT 'manual';
|
||||
|
||||
-- Re-apply NOT NULL for environments where the column was added out-of-band
|
||||
UPDATE public.journal_entries SET created_via = 'manual' WHERE created_via IS NULL;
|
||||
|
||||
ALTER TABLE public.journal_entries
|
||||
ALTER COLUMN created_via SET NOT NULL;
|
||||
|
||||
ALTER TABLE public.journal_entries
|
||||
DROP CONSTRAINT IF EXISTS journal_entries_created_via_check;
|
||||
|
||||
ALTER TABLE public.journal_entries
|
||||
ADD CONSTRAINT journal_entries_created_via_check
|
||||
CHECK (created_via IN ('manual', 'ai_proposed', 'imported', 'system'));
|
||||
|
||||
-- Nullable FK — only AI-proposed entries link back to a proposal
|
||||
ALTER TABLE public.journal_entries
|
||||
ADD COLUMN IF NOT EXISTS source_proposal_id uuid
|
||||
REFERENCES public.ai_proposals(id) ON DELETE SET NULL;
|
||||
|
||||
-- Audit lookup: "show me all AI-proposed entries from last month"
|
||||
CREATE INDEX IF NOT EXISTS idx_journal_entries_created_via_ai
|
||||
ON public.journal_entries (company_id, created_at DESC)
|
||||
WHERE created_via = 'ai_proposed';
|
||||
|
||||
NOTIFY pgrst, 'reload schema';
|
||||
@@ -0,0 +1,27 @@
|
||||
-- Extend categorization_templates.source to include 'ai_corrected'.
|
||||
--
|
||||
-- When a user edits an AI-generated booking proposal and then agrees to
|
||||
-- "remember this for <counterparty> in future", the resulting
|
||||
-- categorization_templates row is inserted with source='ai_corrected' so
|
||||
-- the origin of the template is distinguishable from the existing
|
||||
-- silent-learning paths (user_approved, auto_learned, sie_import, sni_default).
|
||||
--
|
||||
-- This distinction matters for downstream confidence calibration: AI-
|
||||
-- corrected templates carry stronger user-validation signal than auto_learned
|
||||
-- (which is inferred purely by the AI without explicit user review) and
|
||||
-- comparable signal to user_approved.
|
||||
|
||||
ALTER TABLE public.categorization_templates
|
||||
DROP CONSTRAINT IF EXISTS categorization_templates_source_check;
|
||||
|
||||
ALTER TABLE public.categorization_templates
|
||||
ADD CONSTRAINT categorization_templates_source_check
|
||||
CHECK (source IN (
|
||||
'sie_import',
|
||||
'user_approved',
|
||||
'sni_default',
|
||||
'auto_learned',
|
||||
'ai_corrected'
|
||||
));
|
||||
|
||||
NOTIFY pgrst, 'reload schema';
|
||||
@@ -0,0 +1,41 @@
|
||||
-- Per-company toggle for the AI agent flow.
|
||||
--
|
||||
-- `ai_flow_enabled` is the master switch. When true:
|
||||
-- * newly-classified receipts generate AI proposals (via orchestrator);
|
||||
-- * the auto-book path in lib/transactions/ingest.ts is disabled — every
|
||||
-- uncategorized transaction becomes a review item instead of being
|
||||
-- silently posted at >=0.8 mapping-rule confidence;
|
||||
-- * the /agent-inbox page becomes available.
|
||||
--
|
||||
-- `ai_backfill_cancel_requested` is the kill switch for in-flight backfill
|
||||
-- loops. The backfill endpoint kicks off a fire-and-forget iteration over
|
||||
-- pending receipts; each iteration checks this flag between items so the
|
||||
-- loop can be stopped without a separate job queue.
|
||||
|
||||
ALTER TABLE public.company_settings
|
||||
ADD COLUMN IF NOT EXISTS ai_flow_enabled boolean NOT NULL DEFAULT false;
|
||||
|
||||
ALTER TABLE public.company_settings
|
||||
ALTER COLUMN ai_flow_enabled SET DEFAULT false;
|
||||
|
||||
UPDATE public.company_settings
|
||||
SET ai_flow_enabled = false
|
||||
WHERE ai_flow_enabled IS NULL;
|
||||
|
||||
ALTER TABLE public.company_settings
|
||||
ALTER COLUMN ai_flow_enabled SET NOT NULL;
|
||||
|
||||
ALTER TABLE public.company_settings
|
||||
ADD COLUMN IF NOT EXISTS ai_backfill_cancel_requested boolean NOT NULL DEFAULT false;
|
||||
|
||||
ALTER TABLE public.company_settings
|
||||
ALTER COLUMN ai_backfill_cancel_requested SET DEFAULT false;
|
||||
|
||||
UPDATE public.company_settings
|
||||
SET ai_backfill_cancel_requested = false
|
||||
WHERE ai_backfill_cancel_requested IS NULL;
|
||||
|
||||
ALTER TABLE public.company_settings
|
||||
ALTER COLUMN ai_backfill_cancel_requested SET NOT NULL;
|
||||
|
||||
NOTIFY pgrst, 'reload schema';
|
||||
@@ -0,0 +1,41 @@
|
||||
-- Extend processing_history to cover the AI agent streams.
|
||||
--
|
||||
-- 1) Add 'AIProposal' and 'AIRequest' to the aggregate_type CHECK constraint.
|
||||
-- 2) Register the new event types in processing_event_types:
|
||||
-- AIProposalGenerated, AIProposalAccepted, AIProposalRejected,
|
||||
-- AIProposalSkipped, AIProposalInvalidated, AIRequestCreated,
|
||||
-- AIRequestResolved.
|
||||
--
|
||||
-- Events are written by the orchestrator (lib/ai/orchestrator.ts) and the
|
||||
-- proposal API routes, using the same correlation_id that was threaded
|
||||
-- through the document from ingest onward.
|
||||
|
||||
ALTER TABLE public.processing_history
|
||||
DROP CONSTRAINT IF EXISTS processing_history_aggregate_type_check;
|
||||
|
||||
ALTER TABLE public.processing_history
|
||||
ADD CONSTRAINT processing_history_aggregate_type_check
|
||||
CHECK (aggregate_type IN (
|
||||
'Document',
|
||||
'BankTransaction',
|
||||
'MatchProposal',
|
||||
'Verifikation',
|
||||
'CounterpartyTemplate',
|
||||
'Period',
|
||||
'Migration',
|
||||
'System',
|
||||
'AIProposal',
|
||||
'AIRequest'
|
||||
));
|
||||
|
||||
INSERT INTO public.processing_event_types (event_type) VALUES
|
||||
('AIProposalGenerated'),
|
||||
('AIProposalAccepted'),
|
||||
('AIProposalRejected'),
|
||||
('AIProposalSkipped'),
|
||||
('AIProposalInvalidated'),
|
||||
('AIRequestCreated'),
|
||||
('AIRequestResolved')
|
||||
ON CONFLICT (event_type) DO NOTHING;
|
||||
|
||||
NOTIFY pgrst, 'reload schema';
|
||||
Reference in New Issue
Block a user