feat(ai): job-shaped AI service with OpenAI-compatible backend, extraction-first; stop extracting every inbox document twice (#1740)

* feat(ai): job-shaped AI service with OpenAI-compatible backend, extraction-first; stop extracting every inbox document twice

Sovereign plan WS1 PR1 (#1406 Tier 2, extraction-first, aligned with the
AI surface audit).

lib/ai grows a job-shaped service (generateText / generateStructured /
extractFromDocument; no streaming members yet, see plan rule R3):
- services/anthropic-family delegates to the existing createAiClient()
  and sends the exact request literals the inbox extractor sent before
  (request-shape tests deep-equal them), so hosted Bedrock stays
  byte-identical.
- services/openai-compatible talks to any chat-completions endpoint
  (BYO Swedish provider) via Vercel AI SDK 6.x, exact-pinned and
  guarded: images as parts, PDFs rasterized with poppler (AI_PDF_MODE)
  or sent natively, AI_VISION / AI_STRICT_JSON declared, honest skips
  (ai_no_vision, pdf_rasterizer_missing) instead of fake failures.
- config.ts: AI_PROVIDER/AI_BASE_URL/AI_API_KEY/AI_MODEL and per-tier
  AI_*_MODEL with the legacy BEDROCK_* names kept as the same overrides;
  getAiStatus() is the single source of truth for "is AI wired up".
- provider.ts: openai-compatible in the auto-detect chain (after Bedrock
  and the direct API); createAiClient() refuses it loudly.

Document extraction moves onto the service and gets the audit's fixes:
- Inbox documents were extracted TWICE (pipeline A ran inside
  uploadDocument() before the inbox row existed, so its dedupe branch
  never fired; 3 707 + 1 666 calls / 30 d). The inbox now declares
  extractionOwner on the upload, the extension yields, and the inbox
  mirrors its single outcome onto document_attachments from every
  writer (sync, deferred, attach, retry, MCP).
- Every "no extraction will ever happen" outcome is stamped
  (skipped:no_ai_entitlement / ai_unconfigured / system_generated /
  ...); the status route maps the quiet ones to 'disabled' on the first
  poll instead of a 30 s client timeout. Prod showed 309 of the 327
  never-extracted uploads were the paywall working silently.
- Self-generated documents (our own invoice PDFs, payout files) are no
  longer OCR'd.
- Agent invoke answers 503 ai_unconfigured when the deployment has no
  assistant backend, distinct from the paywall.

Guard: new direct-ai-client antipattern check (shrink-only allowlist of
the pre-abstraction SDK callers) plus exact pins for @anthropic-ai/sdk,
ai and @ai-sdk/openai-compatible.

Verified: 15 958 unit tests green, guards, lint ratchet, typecheck, and a
live smoke against hosted Bedrock through the new service (ping, streamed
tool turn, thinking+cache, PDF extraction).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(ai): make AI_API_KEY optional for OpenAI-compatible endpoints (keyless local model servers)

A local model server (llama.cpp's server, Ollama /v1, LM Studio, vLLM)
usually has no auth. Before, the OpenAI-compatible backend required both
AI_BASE_URL and AI_API_KEY to count as configured, so running Accounted on a
local model meant setting a meaningless placeholder key.

- resolveAiProvider / hasAiCredentials: a base URL alone is now enough.
- services/openai-compatible: only send Authorization: Bearer when AI_API_KEY
  is set, so a keyless server is never handed an empty bearer; a hosted
  provider that needs a key still sets it.
- Docs (SELF-HOSTING Option 3: local-model example, key marked optional),
  DECISIONS.

Verified: with no AI_API_KEY, just AI_BASE_URL + AI_MODEL, getAiStatus()
reports configured=true / provider=openai-compatible (live). lib/ai suite
71 green; tsc, guards, lint clean. Bedrock/Anthropic logic unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

DECISIONS.md

@@ -1119,3 +1119,9 @@ One line per decision: `[YYYY-MM-DD] <decision>: <why>`. Appended by agents and
 [2026-08-20] Unmatched bank rows that no voucher on the account could settle (direction-compatible and equal to the ore) get "Bokfor" linking to /transactions?highlight=<id> instead of a match picker. They are unbooked affarshandelser, not reconciliation work, and the picker held nothing for them. The rule is deliberately strict: a false negative offers booking on a pairable row (a legitimate outcome), a false positive sends the user into an empty picker.
 [2026-08-20] Vercel build heap is raised through vercel.json `buildCommand` (`NODE_OPTIONS=--max-old-space-size=6144 npm run build`), not a project env var and not `build.env`: a project-level NODE_OPTIONS also reaches function runtime (V8 sizes the heap against a limit the function does not have), and `build.env` is marked deprecated in the vercel.json schema; `buildCommand` scopes the flag to the build exactly like core-build.yml's 8192 does for CI. 6144 fits the standard 4-core/8 GB build machine next to the main next process; the type-check needs ~4.5 GB and was hanging at V8's ~4 GB default ceiling (4 production timeouts 2026-08-14..20).
 [2026-08-20] The production build type-checks tsconfig.build.json (tsconfig.json minus tests and mocks) via typescript.tsconfigPath; tsconfig.json stays the editor/ESLint view of the whole repo. Why: Next 16.3 runs the project-local tsc CLI by default (prerequisite for TypeScript 7's native checker, which has no JS API) and that checker checks the complete project it is given, whereas the old API checker silently dropped diagnostics from __tests__/*.test.* files. ~490 real type errors live in test files today (mostly route handlers called without the ctx argument); vitest never type-checks them, so nothing caught them. Excluding tests from the build keeps that debt where it was (invisible) instead of turning it into a red deploy; a separate tsc job for tests is the follow-up that makes it visible. Measured: tests are ~10% of the check's memory, so this is correctness, not the memory fix (that is the vercel.json heap bump).
+[2026-08-20] AI provider abstraction (#1406 Tier 2, Sovereign plan WS1 PR1) ships extraction-first: a job-shaped service in lib/ai (generateText / generateStructured / extractFromDocument, NO streaming members) with an Anthropic-family adapter that delegates to the existing createAiClient() factory and sends the exact request literals the call sites sent before (request-shape tests deep-equal them, so hosted Bedrock stays byte-identical), plus an OpenAI-compatible adapter for BYO Swedish endpoints. Only document extraction moved onto it in this PR; the chat loop, composer, receipt hunt and WhatsApp interpreter stay on the direct SDK behind a shrink-only allowlist in the new direct-ai-client antipattern check, because the AI surface audit (2026-08-18) recommends ripping the chat runtime and making the composer deterministic, and porting streaming + translation for code with a delete recommendation against it would be wasted work. The streaming port is gated on that founder call (plan rule R3).
+[2026-08-20] The OpenAI-compatible adapter uses the Vercel AI SDK 6.x (`ai` 6.0.259 + `@ai-sdk/openai-compatible` 2.0.69), exact-pinned and guarded, and NOT 7.x: 7.0.0 shipped 2026-06-25 (eight weeks old at pin time) while 6.x still receives releases, the plan approved "v6" on 2026-08-18, and the SDK is confined to lib/ai/services/openai-compatible.ts so a later major bump is one reviewed file. Lockfile regenerated with npm@10 (npm 11 drops next-intl's nested @swc/helpers entry, per the dependabot-lockfile memory).
+[2026-08-20] Inbox double extraction fixed by ownership, not by reordering: uploadDocument() awaits the document.uploaded handlers inline, so the document-extraction extension ran BEFORE the invoice-inbox row existed and its copied-from-invoice-inbox branch had fired 0 times ever (every inbox document, web/email/WhatsApp/MCP, was extracted twice: 3 707 + 1 666 calls in 30 d). The inbox now declares `extractionOwner: 'invoice-inbox'` on the upload (event payload), the extension yields, and the inbox mirrors its single outcome onto document_attachments (extracted_data / extracted_at / extraction_model) from every writer (sync, deferred, attach, retry, MCP) via one helper, so the status poll and the agent reads keep working unchanged. A DB trigger was considered and rejected: it would mirror user edits (PUT extracted-data) as if they were AI output and needs a migration + pg test for what is a TS-level ordering problem.
+[2026-08-20] Every "no extraction will ever happen" outcome is now stamped on document_attachments as skipped:<reason> (no_ai_entitlement, ai_unconfigured, system_generated, sandbox, client_opt_out, ai_no_vision, pdf_rasterizer_missing) and the extraction-status route maps the quiet ones to 'disabled' on the first poll. Prod check (30 d, read-only): 309 of the audit's 327 never-extracted uploads belonged to 29 companies with zero successful extractions, i.e. the paywall working silently and the UI spinning 30 s to find out; the remaining 18 are download failures, now stamped failed:storage_download. Self-generated documents (upload_source 'system': our own invoice PDFs, payout files, filings; 59 paid model calls in 30 d) are skipped as system_generated.
+[2026-08-20] Strict JSON on OpenAI-compatible endpoints uses a hand-maintained JSON-schema mirror of the extraction Zod schema, opt-in via AI_STRICT_JSON, never an automatic Zod-to-JSON-schema conversion: the Zod schema carries .catch()/.transform() that have no schema equivalent and a generated schema would drift silently. Zod stays the validator either way; JSON-in-prose + extractJsonObject remains the default everywhere because it works on every model and is what hosted runs.
+[2026-08-20] AI_API_KEY made optional for the OpenAI-compatible backend: a base URL alone now counts as configured (hasAiCredentials / resolveAiProvider), so a local model server (llama.cpp/Ollama/LM Studio/vLLM), which usually has no auth, works with just AI_BASE_URL + AI_MODEL. The openai-compatible service only sends an Authorization: Bearer when AI_API_KEY is set, so a keyless local server is never handed an empty bearer. Hosted providers that require a key still set AI_API_KEY. Bedrock/Anthropic credential logic unchanged.