merge: canonical web product reconstruction engine (GitHub PR #1)
CI (SIAX Cloud) / contracts (push) Successful in 14s
CI (SIAX Cloud) / security (push) Successful in 33s
CI (SIAX Cloud) / quality (push) Successful in 49s

Contracts-first architecture: typed evidence/reconstruction/fidelity
contracts, upstream provenance registry, deterministic fidelity release
gate, canonical architecture CI validator, MASTERPLAN/STANDARDS/
AUTOMATIONS/UPSTREAM-CAPABILITY-MATRIX docs. Rebased over prod cutover
state; broken Coolify auto-deploy removed (manual deploy per DEPLOY.md).
This commit is contained in:
2026-09-16 21:54:30 +02:00
19 changed files with 1072 additions and 56 deletions
+5 -1
View File
@@ -13,6 +13,8 @@ jobs:
version: 9.15.9
- name: Install
run: pnpm install --frozen-lockfile
- name: Canonical architecture policy
run: pnpm run validate:architecture
- name: Typecheck
run: pnpm run typecheck
- name: Test
@@ -39,5 +41,7 @@ jobs:
version: 9.15.9
- name: Install
run: pnpm install --frozen-lockfile
- name: Check schema/contracts
- name: Canonical architecture policy
run: pnpm run validate:architecture
- name: Check schema/contracts and provenance policy
run: pnpm --filter @siax/c0py-core test
-15
View File
@@ -1,15 +0,0 @@
name: Deploy to Coolify
on:
push:
branches: [main]
workflow_dispatch:
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Trigger Coolify redeploy
env:
COOLIFY_TOKEN: ${{ secrets.COOLIFY_TOKEN }}
run: |
if [ -z "$COOLIFY_TOKEN" ]; then echo "::error::COOLIFY_TOKEN is not set"; exit 1; fi
curl -fsS -X POST "https://cloud.siax.io/api/v1/deploy?uuid=c0py&force=false" -H "Authorization: Bearer $COOLIFY_TOKEN" -o /dev/null -w 'coolify deploy HTTP %{http_code}\n'
+65 -17
View File
@@ -1,19 +1,10 @@
# C0PY — AGENTS.md
## Repo-konventioner
## Canonical mandate
- **Package manager**: pnpm (frozen-lockfile)
- **Framework**: Next.js (apps/web), Fastify (apps/api)
- **TypeScript**: strict mode, bundler module resolution
- **CI**: Gitea Actions — `.gitea/workflows/ci.yml`
- **Deploy**: Coolify via `.gitea/workflows/deploy.yml`
- **Secrets**: Infisical (`vault.siax.io`) — ALDRIG i repo
- **Testing**: vitest (apps/api, packages/*)
- **Lint**: eslint flat config
- **OTel**: not yet wired — add bootstrap at apps/api/src/index.ts entry when available
- **Tenant**: bound to session via CL0UD, NEVER from client input
C0PY is the SIAX source of truth for web/product capture, evidence, digital-twin specifications, reconstruction contracts and fidelity verification. Agents MUST follow `docs/MASTERPLAN.md`, `docs/STANDARDS.md`, `docs/AUTOMATIONS.md` and `docs/UPSTREAM-CAPABILITY-MATRIX.md`.
## Drift (Coolify)
Do not create parallel capture ontologies, alternative evidence schemas or competing fidelity models in this or another SIAX repository.
- **App UUID API**: `wvmn6eg40jofssvm3nigmav6`, server6 — `c0py-api.siax.io`
- **App UUID Web**: `58pa82h4qlzcwsc4shreuwk0`, server6 — `c0py-web.siax.io` (`Dockerfile.web`)
@@ -26,13 +17,31 @@
- **⚠️ `c0py.siax.io` apex är Worker-kapad** ("Hello World!") — ej åtgärdbar utan Workers-perms
- **Coolify dockerfile+git-buildpack**: trasig för detta monorepo (build-context saknar repo-träd) — bygg manuellt
## Kontrakt
## Repo conventions
- API-kontrakt: `contracts/openapi.yaml`
- Schema-versionshantering: semantisk versionering i `packages/types`
- Digital Twin Spec: `docs/spec/DIGITAL-TWIN-SPEC.md` (fullständig specifikation för webbplats-analys och identisk reproduktion)
- Package manager: pnpm (frozen-lockfile)
- Framework: Next.js (`apps/web`), Fastify (`apps/api`)
- TypeScript: strict mode
- Canonical shared contracts: `packages/types`
- Core orchestration: `packages/c0py-core`
- CI: Gitea Actions under `.gitea/workflows/`
- Deploy: Coolify for C0PY runtime; reconstruction preview adapters may target Vercel
- Secrets: Infisical (`vault.siax.io`), NEVER in repo
- Testing: vitest
- Tenant: bound to trusted session via CL0UD, NEVER client-controlled
## Fällor
## Architecture rules
1. Deterministic measurement beats AI interpretation.
2. Playwright is the canonical browser automation runtime; use CDP for deeper instrumentation.
3. All significant source-product assertions become typed evidence with provenance.
4. Builders (`v0`, `lovable`, `codex`, generic) consume frozen specs and may not redefine source truth.
5. Semantic agents discover candidate workflows; deterministic capture verifies them.
6. Upstream functionality is classified as `CORE`, `DEPENDENCY`, `CHERRY_PICK`, `ADAPTER`, `REIMPLEMENT`, `REFERENCE_ONLY` or `DROP` before integration.
7. Any source-derived code requires repository, commit SHA, source path, license and verification tests.
8. Copyleft/AGPL implementations stay behind isolated adapters/services unless an explicit licensing decision changes this.
9. Clone mode forbids invented facts. Evidence gaps remain explicit.
10. Verification produces a fidelity vector, never a single release score.
- `NEXT_PUBLIC_*` ska ALDRI innehålla hemligheter
- Alla writes till AUD0/ST0RE via klientskikt, aldrig direkt
@@ -42,3 +51,42 @@
- Auth: preHandler hook kollar `Authorization: Bearer *`-header, returnerar 401 annars
- **GitHub push**: använd `gh auth login` (keyring), INTE inbäddade tokens i URLs
- **Coolify compose på disk kan vara stale** — labels/port regenereras ej vid misslyckad deploy; sed:a compose-filen + `docker compose up -d`, och `docker rm -f` gammal container (den lever kvar vid recreate)
## Canonical evidence confidence
Allowed values:
- `measured`
- `observed`
- `inferred`
- `declared`
- `unknown`
Synthetic values are allowed only in adapt/blend mode and must be explicitly marked as synthetic outside source evidence.
## Capture profiles
Authenticated or variant state must be explicit through capture profiles: role, locale, viewport/device, color scheme and feature flags. Never commit credentials, cookies, tokens or storage-state secrets.
## Contract locations
- API: `contracts/openapi.yaml`
- Shared IR/types: `packages/types/src/index.ts`
- Digital Twin Spec: `docs/spec/DIGITAL-TWIN-SPEC.md`
- Canonical Masterplan: `docs/MASTERPLAN.md`
- Engineering/evidence standards: `docs/STANDARDS.md`
- Automation contracts: `docs/AUTOMATIONS.md`
- Upstream adoption: `docs/UPSTREAM-CAPABILITY-MATRIX.md`
## Drift prevention
When a task proposes a new crawler, browser agent, screenshot comparer, product graph, capture schema or clone workflow, first map it onto existing C0PY contracts. Add a new implementation only when it contributes a distinct capability. Remove or reject duplicate sources of truth.
## Security
- Only capture targets/states the operator is authorized to access.
- Never bypass access controls.
- An authorization failure is evidence, not a prompt to evade the boundary.
- `NEXT_PUBLIC_*` must never contain secrets.
- Writes to platform services use their client/integration layers.
- Docker COPY across build stages can break pnpm symlinks; preserve the existing runtime-stage workaround unless replaced and verified.
+1 -1
View File
@@ -3,7 +3,7 @@
## SIAX Cloud Deploy
- **Config**: [`siax.config.json`](siax.config.json)
- **CI/CD**: `.gitea/workflows/ci.yml` — körs vid push/PR till main (Gitea Actions, git.siax.io)
- **CI/CD**: `.gitea/workflows/ci.yml` — körs vid push/PR till main (Gitea Actions, git.siax.io). Deploy är **manuell** (se nedan) — CI triggar ej Coolify (git-buildpack trasig för detta monorepo, auto-deploy togs bort 2026-09-16).
- **Runtime**: Node 22, port 3000
- **Domain**: `c0py-web.siax.io` (web) / `c0py-api.siax.io` (API)
+52 -18
View File
@@ -1,8 +1,8 @@
# C0PY
**Verktyg för att skapa fullständigt register över en hemsida ut alla aspekter.**
**Canonical SIAX engine for evidence-driven web product capture, digital-twin specification, reconstruction and verification.**
C0PY konsumerar en webbadress och producerar ett komplett register: sida-för-sida innehåll, struktur, prestanda, säkerhet, SEO, metadata och mer. Allt lagras som en tillförlitlig, återläsbar sammanställning.
C0PY consumes an authorized web target and produces a versioned Evidence Graph / Canonical IR covering routes, DOM/runtime state, styles, assets, responsive behavior, interactions, workflows, network contracts, motion, accessibility and product topology. Reconstruction builders consume frozen C0PY specs; they never become the source of truth.
## Snabbkommandon
@@ -12,27 +12,61 @@ pnpm --filter @siax/c0py-api dev
pnpm --filter @siax/c0py-web dev
```
## Arkitektur
## Canonical architecture
```
apps/api/ → Fastify/Next.js API — /v1/c0py/* endpoints
apps/web/ → Next.js UI — registerhantering, dashboard, visualisering
packages/c0py-core → crawling, registry-uppbyggnad, analysmotor
packages/config → Zod-validerad env-laddare
packages/types → delade TypeScript-typer
docs/spec/ → Digital Twin Spec (webbplats-analys och reproduktion)
```text
Source product
-> discovery
-> Playwright/CDP deterministic capture
-> semantic workflow exploration adapters
-> Evidence Graph / Canonical IR
-> frozen reconstruction specs
-> v0 / Lovable / Codex / generic builders
-> preview deployment
-> multidimensional QA
-> typed repair loop
```
## Konsumerade kapabiliteter
## Repo structure
| Kapabilitet | Syfte | Status |
```text
apps/api/ -> Fastify/API surface
apps/web/ -> Next.js console and visualisation
packages/c0py-core -> discovery, capture orchestration, evidence/provenance, product model, verification
packages/config -> validated runtime configuration
packages/types -> canonical shared IR/contracts
docs/spec/ -> full Digital Twin Specification
docs/MASTERPLAN.md -> canonical execution plan
docs/STANDARDS.md -> mandatory engineering/evidence rules
docs/AUTOMATIONS.md -> autonomous pipeline contracts
```
## Capability adoption
External technologies are classified as `CORE`, `DEPENDENCY`, `CHERRY_PICK`, `ADAPTER`, `REIMPLEMENT`, `REFERENCE_ONLY` or `DROP`. See [`docs/UPSTREAM-CAPABILITY-MATRIX.md`](docs/UPSTREAM-CAPABILITY-MATRIX.md).
Primary integrations include Playwright/CDP, Crawlee-compatible crawl orchestration, rrweb, Browser Use/Browser Harness adapters, deterministic pixel/style verification, optional Spector.js WebGL forensics, and isolated archival adapters. Builders include v0, Lovable, Codex and generic agents.
## Operating modes
- `clone` — maximize observed fidelity; synthetic facts are prohibited.
- `adapt` — preserve approved semantics while transforming design/technology.
- `blend` — combine explicitly selected capabilities/patterns with complete provenance.
## Quality
C0PY does not use one visual score as a release verdict. It verifies a fidelity vector across geometry, typography, colors, assets, responsive states, routes, interactions, workflows, network contracts, motion, accessibility, runtime errors and production build health.
## SIAX platform capabilities
| Capability | Purpose | Status |
|---|---|---|
| CL0UD | Autentisering, RBAC, tenant | live |
| AUD0 | Regeleveranshändelser | (ej live) |
| ST0RE | Objektlagring av registerdata | (ej live) |
| INF0 | Innehållsanalys | (ej live) |
| N0D | Asynkrone crawling-jobb | (ej live) |
| CL0UD | Authentication, RBAC, tenant | live |
| AUD0 | Audit/event delivery | planned integration |
| ST0RE | Evidence/object storage | planned integration |
| INF0 | AI/semantic analysis | planned integration |
| N0D | Async capture/reconstruction jobs | planned integration |
## Deploy
SIAX Cloud via Coolify. Se [`DEPLOY.md`](DEPLOY.md).
SIAX Cloud via Coolify. Vercel is supported as a reconstruction preview adapter. See [`DEPLOY.md`](DEPLOY.md).
+98
View File
@@ -0,0 +1,98 @@
# C0PY Automations
## Automation principles
C0PY automations are resumable, idempotent, versioned and evidence-producing. Every run emits a manifest, provenance and machine-readable failures.
## Canonical pipeline
```text
DISCOVER
-> CAPTURE
-> NORMALIZE TO IR
-> MODEL PRODUCT
-> FREEZE SPEC
-> RECONSTRUCT
-> DEPLOY PREVIEW
-> VERIFY
-> REPAIR (if failed)
-> VERIFY
-> RELEASE CANDIDATE
```
## Jobs
### discover-target
Inputs: target URL, capture profile, scope policy.
Outputs: route graph, navigation graph, discovered workflows, crawl queue.
### capture-target
Uses Playwright/CDP and optional adapters. Captures DOM/runtime HTML, styles, accessibility, media/assets, screenshots, network/HAR, storage metadata, events and performance.
### interaction-sweep
Enumerates deterministic interactive states and records before/after deltas. Semantic agents may propose candidate flows; deterministic capture verifies them.
### workflow-discovery
Uses semantic browser adapters to discover product tasks and translates them into explicit action/state graphs with evidence links.
### normalize-evidence
Validates captured artifacts and writes the Canonical IR/Evidence Graph. Rejects untyped or provenance-free critical evidence.
### generate-spec
Creates frozen reconstruction contracts for routes, components, design tokens, responsive states, workflows and network/domain behavior.
### reconstruct
Dispatches versioned specs to selected builders (`v0`, `lovable`, `codex`, `generic`). Generated code is never treated as evidence about the source.
### preview
Builds and deploys a preview target. Vercel is a supported deployment adapter; SIAX/Coolify remains supported for controlled infrastructure.
### verify
Runs the fidelity vector: visual geometry/style, responsive, routes, interactions, workflows, network, motion, accessibility, runtime and build checks.
### repair
Turns verification failures into typed repair tasks, applies the smallest justified patch and sends the result back through verification. Repair may not relax gates without an explicit versioned policy change.
### upstream-reconcile
Scheduled/manual automation that checks registered upstream projects for high-value changes. It creates candidate capability records only; it never auto-merges foreign code without provenance/license/test gates.
### schema-compatibility
On every IR/schema change, validates backward compatibility and requires a version bump for incompatible changes.
### license-provenance
Checks all `CHERRY_PICK` and vendored/adapted source entries for repository, SHA, source path, classification and license/notice requirements.
## Required run manifest
Each automation run records at minimum:
- run id
- target id
- C0PY commit/version
- schema version
- capture profile
- started/completed timestamps
- tool/adapter versions
- evidence object ids
- warnings and typed failures
- retry lineage
- output spec version
- verification result
## Failure policy
No automation may convert a failed measurement into an inferred success. Failed or inaccessible states remain explicit evidence gaps.
## CI/CD gates
Pull requests touching capture, IR, product model, reconstruction contracts or fidelity rules must run:
1. typecheck
2. unit tests
3. schema validation
4. provenance validation
5. license boundary validation
6. representative deterministic fixture capture
7. reconstruction fixture verification when affected
Production promotion is blocked when critical route/workflow/state gates fail.
+143
View File
@@ -0,0 +1,143 @@
# C0PY Masterplan — Canonical Web Product Reconstruction Engine
**Status:** canonical
**Owner:** C0PY
**Rule:** this document and the C0PY IR contracts are the single source of truth for web/product capture and reconstruction across SIAX.
## Mission
C0PY turns an observed web product into an evidence-backed digital twin specification that can be reconstructed, verified and repaired autonomously. The system must cover visual fidelity, responsive behavior, assets, interactions, workflows, network contracts, motion, accessibility and product topology.
## Non-negotiable architecture
```text
Source product
-> discovery
-> deterministic capture
-> semantic exploration
-> Canonical Evidence Graph / IR
-> frozen reconstruction specs
-> builder adapters
-> preview deployment
-> multidimensional verification
-> repair loop
-> production candidate
```
No crawler, AI builder or upstream project owns the truth. The Canonical IR owns the truth.
## Canonical layers
1. **Discovery** — routes, sitemaps, links, navigation, semantic workflow discovery.
2. **Capture** — Playwright + CDP as primary runtime; DOM, CSSOM, computed styles, accessibility tree, screenshots, storage, network, WebSocket/SSE, canvas/media and performance.
3. **Evidence** — every assertion stores source, method, capture context, confidence and provenance.
4. **Product model** — entities, relationships, capabilities, permissions, workflows and state machines inferred from observed behavior.
5. **Specification** — deterministic page/component/workflow contracts generated from the evidence graph.
6. **Reconstruction** — adapters for v0, Lovable, Codex and generic agents. Builders consume specs; they do not inspect the source as authority.
7. **Verification** — geometry, typography, color, assets, routes, state, interaction, workflow, network, motion, accessibility and runtime health.
8. **Repair** — failures become typed root-cause work items and are repaired iteratively.
## Primary technology decisions
- Browser runtime: Playwright.
- Deep browser instrumentation: Chrome DevTools Protocol.
- Crawl orchestration: Crawlee-compatible request graph.
- Session/event replay: rrweb adapter.
- Semantic exploration: Browser Use / Browser Harness adapters.
- Visual comparison: pixelmatch-compatible deterministic diff plus computed-style root cause analysis.
- Optional WebGL forensics: Spector.js adapter.
- Optional archival: SingleFile and Browsertrix only behind process/service adapters due to copyleft boundaries.
- Output target: clean SIAX architecture, normally Next.js + React + TypeScript, regardless of source framework.
## Upstream capability policy
Every borrowed idea or implementation MUST be classified as one of:
- `CORE` — implemented and owned inside C0PY.
- `DEPENDENCY` — consumed as a package/library.
- `CHERRY_PICK` — imported/adapted source with exact provenance and license notice.
- `ADAPTER` — isolated integration behind a C0PY contract.
- `REIMPLEMENT` — behavior/specification reproduced without copying source.
- `REFERENCE_ONLY` — studied, never included in runtime.
- `DROP` — rejected duplicate or lower-value implementation.
See `docs/UPSTREAM-CAPABILITY-MATRIX.md`.
## Canonical package topology
```text
packages/
types/ canonical shared contracts
c0py-core/
discovery/
capture/
evidence/
provenance/
product-model/
specification/
verification/
repair/
adapters/
```
As implementation grows, high-cohesion modules may graduate into their own workspace packages, but they must continue to consume the same IR contracts.
## Evidence model
Evidence confidence MUST use explicit semantics:
- `measured`: deterministic machine measurement.
- `observed`: directly observed behavior/state.
- `inferred`: reasoned from evidence, not directly visible.
- `declared`: supplied by an authorized configuration/specification.
- `unknown`: insufficient evidence.
`invented` is prohibited in clone mode. Adapt/blend mode may create new design/product choices, but they must be marked as synthetic and never confused with source evidence.
## Operating modes
- **clone** — maximize observable fidelity; no creative substitutions without an explicit evidence gap.
- **adapt** — preserve approved product semantics while transforming brand/design/technology.
- **blend** — combine explicitly selected capabilities/patterns from multiple references into a new product; every source lineage remains traceable.
## Quality gates
A single visual score is forbidden as the release criterion. C0PY uses a fidelity vector:
- geometry
- typography
- colors
- assets
- responsive states
- route coverage
- interaction/state coverage
- workflow coverage
- network-contract parity
- motion parity
- accessibility semantics
- console/runtime health
- production build health
Critical routes, contracted workflows and required interaction states must reach 100% coverage before a clone can be marked complete. Visual tolerances are profile-specific and recorded in the manifest.
## Cross-repo rule
Other SIAX repositories may consume C0PY capabilities, evidence and generated specs, but MUST NOT fork a parallel capture ontology, fidelity model or reconstruction masterplan. Cross-repo integrations point back to C0PY contracts/versioned schemas.
## Delivery order
1. Canonical IR + provenance.
2. Playwright/CDP capture.
3. route topology and deterministic extraction.
4. HAR/network evidence and authenticated capture profiles.
5. interaction/state delta engine.
6. rrweb replay/event timeline.
7. semantic workflow discovery adapters.
8. design-system extraction.
9. media/motion/WebGL adapters.
10. specification generator.
11. reconstruction adapters.
12. multidimensional QA.
13. autonomous repair loop.
14. clone/adapt/blend orchestration.
15. MCP/API contracts and fleet automations.
+3 -3
View File
@@ -51,6 +51,6 @@ Skapa OIDC-app under projekt `c0py` (eller motsvarande). Client credentials flow
2. `pnpm run typecheck`
3. `pnpm run test`
4. `pnpm run build`
5. Pusha main → CI → Deploy via `.gitea/workflows/deploy.yml`
6. Verifiera `https://c0py.siax.io/health` → 200 OK
7. Verifiera fail-closed: `https://c0py.siax.io/v1/c0py/registries` → 401 utan token
5. Pusha main → CI (`ci.yml`); deploy sker manuellt enligt [`DEPLOY.md`](../DEPLOY.md) (Coolify git-buildpack är trasigt för detta monorepo — bygg `<app-uuid>:latest` på server6 och kör `docker compose up -d` i appkatalogen)
6. Verifiera `https://c0py-api.siax.io/health` → 200 OK
7. Verifiera fail-closed: `https://c0py-api.siax.io/v1/c0py/registries` → 401 utan token
+127
View File
@@ -0,0 +1,127 @@
# C0PY Standards
## 1. Single-source-of-truth standard
C0PY owns the canonical schemas for capture evidence, product topology, reconstruction specifications and fidelity verification. Other repositories may integrate with these contracts but may not create competing definitions.
## 2. Evidence standard
Every material claim about a source product must be represented with:
- stable evidence id
- evidence kind
- source URL/route
- capture profile
- viewport/device context when relevant
- timestamp
- acquisition method
- confidence class
- provenance/source implementation
- payload or durable object reference
Confidence classes: `measured`, `observed`, `inferred`, `declared`, `unknown`.
In `clone` mode synthetic/invented facts are forbidden. In `adapt` and `blend`, synthetic values must be explicitly labeled.
## 3. Capture standard
Deterministic capture has precedence over AI interpretation.
Preferred order:
1. browser/browser-protocol measurement
2. runtime event/network observation
3. structured source metadata
4. semantic agent observation
5. inference
Playwright is the standard browser automation layer. Raw CDP is used for capabilities that require deeper browser instrumentation.
## 4. Route and workflow standard
Routes and workflows are graph data, not prose. Each discovered route/action/state transition is uniquely identifiable and links to its supporting evidence.
A workflow is complete only when its required preconditions, actions, transitions, resulting states, error paths and relevant network contracts are represented.
## 5. Reconstruction standard
Builders consume frozen, versioned C0PY specifications. v0, Lovable, Codex and future builders are adapters, not authorities.
Generated implementations must preserve:
- component contracts
- responsive contracts
- interaction/state contracts
- route contracts
- network/domain contracts where reconstruction scope includes them
- accessibility semantics
- fidelity thresholds
## 6. Upstream standard
Every external capability is classified as `CORE`, `DEPENDENCY`, `CHERRY_PICK`, `ADAPTER`, `REIMPLEMENT`, `REFERENCE_ONLY` or `DROP`.
Any source-derived code must preserve required copyright/license notices and record exact provenance.
AGPL/copyleft systems such as SingleFile/Browsertrix remain isolated adapters/services unless C0PY deliberately changes licensing strategy.
## 7. Verification standard
Do not reduce fidelity to one number. Required dimensions include:
- geometry
- typography
- color
- assets
- responsive behavior
- route coverage
- interactions and states
- workflows
- network contracts
- motion
- accessibility
- runtime errors
- build health
Release gates are profile/version controlled. Critical route/workflow/state coverage defaults to 100%.
## 8. Failure standard
Failures are typed and machine-actionable. Minimum families:
- `CaptureFailure`
- `DiscoveryFailure`
- `EvidenceGap`
- `ReconstructionFailure`
- `VerificationFailure`
- `AuthorizationBoundary`
- `UnsupportedSurface`
- `DynamicContentVariance`
- `NetworkReplayLimited`
- `AuthSessionMissing`
Every repair loop consumes typed failures rather than unstructured screenshots or generic prompts.
## 9. Security and authorization standard
Only capture resources and product states that the operator is authorized to access. Authentication state is supplied explicitly through capture profiles. Credentials and session material never enter source control.
C0PY does not bypass access controls. Failed authorization is evidence and is represented as such.
## 10. CI standard
Changes to canonical schemas, evidence confidence semantics, capability taxonomy, release gates or adapter boundaries require:
- schema/type checks
- unit tests
- provenance/license checks where applicable
- documentation update
- backward-compatibility note or explicit version bump
## 11. Automation standard
Automations must be idempotent, resumable and evidence-producing. A job may retry execution, but may not silently mutate evidence history or overwrite the lineage of an earlier capture.
## 12. Cross-repo standard
When another SIAX product wants website/product cloning, analysis or reconstruction, it must call/import C0PY instead of adding its own crawler/clone ontology. Shared automation and prompt systems reference this standard and the C0PY Masterplan.
+52
View File
@@ -0,0 +1,52 @@
# C0PY Upstream Capability Matrix
This matrix defines how C0PY consumes high-value upstream ideas without creating duplicate runtimes or conflicting sources of truth.
| Upstream | Classification | Adopt | Do not adopt |
|---|---|---|---|
| Mahanaicoach/ai-site-cloner | CHERRY_PICK / REIMPLEMENT | deterministic extraction, compact state deltas, responsive probes, canvas capture patterns, spec generation/linting, pixel + CSS root-cause QA, resumable manifests | duplicate orchestration/runtime ownership |
| jongko54/webEmbedding | CHERRY_PICK / REIMPLEMENT | HAR capture/replay contracts, authenticated storage-state capture, evidence artifacts, replay-readiness, failure taxonomy, verification patterns, MCP surface | a second canonical evidence model |
| tentenco/ai-website-clone | REIMPLEMENT / REFERENCE_ONLY | clone/adapt/blend modes, evidence-confidence model, reference ledger, motion audit, deterministic acceptance gates | duplicate capture pipeline |
| Omerfaruk-aydn/sitecloner | CHERRY_PICK / REIMPLEMENT | media discovery, yt-dlp/ffmpeg integration patterns, animation extraction, interaction maps, source/clone recording and frame comparison | duplicate browser crawler and visual diff core |
| microsoft/playwright | DEPENDENCY | canonical browser automation/runtime | custom replacement browser runtime |
| Chrome DevTools Protocol | DEPENDENCY | DOMSnapshot, CSS, Animation, Network, Runtime, Accessibility, Performance, Coverage | undocumented browser scraping where a typed CDP domain exists |
| apify/crawlee | DEPENDENCY | request queue, crawl graph, browser crawl orchestration | separate canonical route ontology |
| browser-use/browser-use | ADAPTER | semantic exploration, task/workflow discovery | low-level deterministic measurement |
| browser-use/browser-harness | ADAPTER | authenticated real-browser/CDP exploration | canonical IR ownership |
| rrweb-io/rrweb | DEPENDENCY / ADAPTER | DOM mutation timeline, interaction record/replay | replacing C0PY evidence contracts |
| BabylonJS/Spector.js | ADAPTER | optional WebGL frame forensics | mandatory core dependency |
| mapbox/pixelmatch | DEPENDENCY | deterministic pixel diff primitive | single release score |
| dequelabs/axe-core | DEPENDENCY | accessibility verification | copied implementation |
| Google Lighthouse | DEPENDENCY / TOOL | performance/runtime comparison | product behavior truth |
| gildas-lormeau/SingleFile | ADAPTER | archival/reference snapshot only | copied AGPL code in C0PY core |
| webrecorder/browsertrix-crawler | ADAPTER | optional WARC/WACZ archival service | copied AGPL code in C0PY core |
| Stagehand | ADAPTER | semantic observe/extract/act where deterministic selectors are insufficient | source-of-truth capture engine |
## Import rules
1. Every source-derived implementation must record repository, commit SHA, source files, license, integration method and tests.
2. MIT/ISC/Apache components may be integrated according to their license and notices.
3. Copyleft components must stay behind a separately deployable adapter unless a deliberate licensing decision says otherwise.
4. Prefer dependencies over vendoring when the upstream public API is stable and sufficient.
5. Prefer reimplementation from behavior/specification when importing code would create architectural duplication.
6. No upstream project may define C0PY's schema, ontology or release criteria.
## Provenance record example
```yaml
capability: visual-css-root-cause
classification: CHERRY_PICK
source:
repository: Mahanaicoach/ai-site-cloner
commit: <sha>
files:
- scripts/compare.mjs
license: MIT
integration:
target: packages/c0py-core/src/verification
method: adapted
tests:
- geometry-diff
- typography-diff
- responsive-diff
```
+1 -1
View File
@@ -1 +1 @@
{"name": "c0py", "version": "0.1.0", "private": true, "description": "C0PY — SIAX Cloud. Verktyg för att skapa fullständigt register över en hemsida ut alla aspekter.", "type": "module", "scripts": {"build": "pnpm -r build", "typecheck": "pnpm -r typecheck", "test": "pnpm -r test", "lint": "eslint .", "dev:api": "pnpm --filter @siax/c0py-api dev", "dev:web": "pnpm --filter @siax/c0py-web dev", "start:api": "pnpm --filter @siax/c0py-api start"}, "devDependencies": {"typescript": "^5.7.2", "@types/node": "^24.0.0", "@eslint/js": "^9.18.0", "eslint": "^9.18.0", "eslint-config-prettier": "^10.0.0", "prettier": "^3.4.2", "typescript-eslint": "^8.20.0"}, "engines": {"node": ">=20.17.0"}, "pnpm": {"overrides": {"nanoid@<5.1.16": "5.1.16"}}, "packageManager": "pnpm@9.15.9"}
{"name":"c0py","version":"0.1.0","private":true,"description":"C0PY — canonical SIAX web product capture, digital twin, reconstruction and verification engine.","type":"module","scripts":{"build":"pnpm -r build","typecheck":"pnpm -r typecheck","test":"pnpm -r test","lint":"eslint .","validate:architecture":"node scripts/validate-canonical-architecture.mjs","dev:api":"pnpm --filter @siax/c0py-api dev","dev:web":"pnpm --filter @siax/c0py-web dev","start:api":"pnpm --filter @siax/c0py-api start"},"devDependencies":{"typescript":"^5.7.2","@types/node":"^24.0.0","@eslint/js":"^9.18.0","eslint":"^9.18.0","eslint-config-prettier":"^10.0.0","prettier":"^3.4.2","typescript-eslint":"^8.20.0"},"engines":{"node":">=20.17.0"},"pnpm":{"overrides":{"nanoid@<5.1.16":"5.1.16"}},"packageManager":"pnpm@9.15.9"}
+3
View File
@@ -3,3 +3,6 @@ export * from "./registry/registry";
export * from "./registry/analyzer";
export * from "./crawler/types";
export * from "./registry/types";
export * from "./provenance/registry";
export * from "./reconstruction/contracts";
export * from "./verification/fidelity";
@@ -0,0 +1,50 @@
import type { UpstreamCapabilityRecord } from "@siax/c0py-types";
const COPyleft_ISOLATED = new Set(["AGPL-3.0", "AGPL-3.0-only", "AGPL-3.0-or-later"]);
export interface ProvenanceValidationResult {
valid: boolean;
errors: string[];
warnings: string[];
}
export function validateUpstreamCapability(
record: UpstreamCapabilityRecord,
): ProvenanceValidationResult {
const errors: string[] = [];
const warnings: string[] = [];
const source = record.source;
if (!record.capability.trim()) errors.push("capability is required");
if (!record.rationale.trim()) errors.push("rationale is required");
if (record.disposition !== "DROP" && record.tests.length === 0) {
errors.push("at least one verification test is required");
}
if (["CHERRY_PICK", "REFERENCE_ONLY", "REIMPLEMENT"].includes(record.disposition)) {
if (!source.repository) errors.push("source.repository is required");
if (!source.commit) errors.push("source.commit is required");
}
if (record.disposition === "CHERRY_PICK") {
if (!source.sourcePaths?.length) errors.push("source.sourcePaths is required for CHERRY_PICK");
if (!source.license) errors.push("source.license is required for CHERRY_PICK");
}
if (source.license && COPyleft_ISOLATED.has(source.license) && record.disposition === "CHERRY_PICK") {
errors.push("AGPL/copy-left source must remain behind an adapter/service boundary; do not cherry-pick into core");
}
if (source.license && COPyleft_ISOLATED.has(source.license) && record.disposition !== "ADAPTER") {
warnings.push("copyleft source should normally be classified as ADAPTER");
}
return { valid: errors.length === 0, errors, warnings };
}
export function assertUpstreamCapability(record: UpstreamCapabilityRecord): void {
const result = validateUpstreamCapability(record);
if (!result.valid) {
throw new Error(`Invalid upstream capability ${record.id}: ${result.errors.join("; ")}`);
}
}
@@ -0,0 +1,92 @@
import type {
CaptureProfile,
EvidenceRecord,
FidelityVector,
ReconstructionSpec,
} from "@siax/c0py-types";
export interface TargetDescriptor {
id: string;
url: string;
allowedOrigins: string[];
mode: ReconstructionSpec["mode"];
}
export interface DiscoveryResult {
targetId: string;
routes: string[];
candidateWorkflows: string[];
evidence: EvidenceRecord[];
}
export interface CaptureRequest {
target: TargetDescriptor;
profile: CaptureProfile;
routes: string[];
}
export interface CaptureResult {
targetId: string;
profileId: string;
evidence: EvidenceRecord[];
warnings: PipelineFailure[];
}
export interface VerificationRequest {
spec: ReconstructionSpec;
sourceEvidence: EvidenceRecord[];
implementationUrl: string;
}
export interface VerificationResult {
specId: string;
fidelity: FidelityVector;
failures: PipelineFailure[];
passed: boolean;
}
export type FailureKind =
| "CaptureFailure"
| "DiscoveryFailure"
| "EvidenceGap"
| "ReconstructionFailure"
| "VerificationFailure"
| "AuthorizationBoundary"
| "UnsupportedSurface"
| "DynamicContentVariance"
| "NetworkReplayLimited"
| "AuthSessionMissing";
export interface PipelineFailure {
id: string;
kind: FailureKind;
message: string;
route?: string;
evidenceIds?: string[];
retryable: boolean;
details?: Record<string, unknown>;
}
export interface DiscoveryAdapter {
id: string;
discover(target: TargetDescriptor, profile: CaptureProfile): Promise<DiscoveryResult>;
}
export interface CaptureAdapter {
id: string;
capture(request: CaptureRequest): Promise<CaptureResult>;
}
export interface ReconstructionAdapter {
id: string;
reconstruct(spec: ReconstructionSpec): Promise<{
artifactRef: string;
previewUrl?: string;
commitRef?: string;
}>;
}
export interface VerificationAdapter {
id: string;
verify(request: VerificationRequest): Promise<VerificationResult>;
}
@@ -0,0 +1,58 @@
import type { FidelityVector } from "@siax/c0py-types";
export interface FidelityPolicy {
minimum: Partial<Record<keyof FidelityVector, number | boolean>>;
requireZeroRuntimeErrors?: boolean;
requireProductionBuild?: boolean;
}
export interface FidelityGateResult {
passed: boolean;
failures: string[];
}
export const defaultClonePolicy: FidelityPolicy = {
minimum: {
routeCoverage: 1,
interactionCoverage: 1,
workflowCoverage: 1,
networkContractParity: 1,
responsiveCoverage: 1,
assets: 1,
accessibilityPass: true,
},
requireZeroRuntimeErrors: true,
requireProductionBuild: true,
};
export function evaluateFidelity(
vector: FidelityVector,
policy: FidelityPolicy = defaultClonePolicy,
): FidelityGateResult {
const failures: string[] = [];
for (const [dimension, requirement] of Object.entries(policy.minimum) as Array<
[keyof FidelityVector, number | boolean]
>) {
const actual = vector[dimension];
if (typeof requirement === "boolean") {
if (actual !== requirement) failures.push(`${dimension}: expected ${requirement}, got ${String(actual)}`);
continue;
}
if (typeof actual !== "number" || actual < requirement) {
failures.push(`${dimension}: expected >= ${requirement}, got ${String(actual)}`);
}
}
if (policy.requireZeroRuntimeErrors && vector.runtimeErrorCount !== 0) {
failures.push(`runtimeErrorCount: expected 0, got ${String(vector.runtimeErrorCount)}`);
}
if (policy.requireProductionBuild && vector.productionBuildPass !== true) {
failures.push(`productionBuildPass: expected true, got ${String(vector.productionBuildPass)}`);
}
return { passed: failures.length === 0, failures };
}
+38
View File
@@ -0,0 +1,38 @@
import { describe, expect, it } from "vitest";
import { evaluateFidelity } from "../src/verification/fidelity";
describe("fidelity release gate", () => {
it("passes a complete clone vector", () => {
expect(
evaluateFidelity({
assets: 1,
responsiveCoverage: 1,
routeCoverage: 1,
interactionCoverage: 1,
workflowCoverage: 1,
networkContractParity: 1,
accessibilityPass: true,
runtimeErrorCount: 0,
productionBuildPass: true,
}),
).toEqual({ passed: true, failures: [] });
});
it("fails rather than averaging away missing behavior", () => {
const result = evaluateFidelity({
geometry: 0.999,
assets: 1,
responsiveCoverage: 1,
routeCoverage: 1,
interactionCoverage: 0.8,
workflowCoverage: 1,
networkContractParity: 1,
accessibilityPass: true,
runtimeErrorCount: 0,
productionBuildPass: true,
});
expect(result.passed).toBe(false);
expect(result.failures.join(" ")).toMatch(/interactionCoverage/);
});
});
@@ -0,0 +1,48 @@
import { describe, expect, it } from "vitest";
import { validateUpstreamCapability } from "../src/provenance/registry";
const base = {
id: "visual-css-root-cause",
capability: "visual-css-root-cause",
disposition: "CHERRY_PICK" as const,
source: {
repository: "example/upstream",
commit: "0123456789abcdef",
sourcePaths: ["scripts/compare.mjs"],
license: "MIT",
},
targetModule: "verification",
rationale: "Adds deterministic style-root-cause diagnostics",
tests: ["geometry-diff"],
};
describe("upstream provenance policy", () => {
it("accepts traceable permissive cherry-picks", () => {
expect(validateUpstreamCapability(base)).toEqual({
valid: true,
errors: [],
warnings: [],
});
});
it("rejects AGPL code cherry-picked into core", () => {
const result = validateUpstreamCapability({
...base,
source: { ...base.source, license: "AGPL-3.0" },
});
expect(result.valid).toBe(false);
expect(result.errors.join(" ")).toMatch(/adapter\/service boundary/i);
});
it("requires exact provenance for source-derived implementations", () => {
const result = validateUpstreamCapability({
...base,
source: { license: "MIT", sourcePaths: ["compare.mjs"] },
});
expect(result.valid).toBe(false);
expect(result.errors).toContain("source.repository is required");
expect(result.errors).toContain("source.commit is required");
});
});
+186
View File
@@ -28,3 +28,189 @@ export interface Registry {
createdAt: Date;
updatedAt: Date;
}
export type CapabilityDisposition =
| "CORE"
| "DEPENDENCY"
| "CHERRY_PICK"
| "ADAPTER"
| "REIMPLEMENT"
| "REFERENCE_ONLY"
| "DROP";
export type EvidenceConfidence =
| "measured"
| "observed"
| "inferred"
| "declared"
| "unknown";
export type ReconstructionMode = "clone" | "adapt" | "blend";
export type EvidenceKind =
| "dom"
| "runtime-html"
| "computed-style"
| "stylesheet"
| "accessibility"
| "screenshot"
| "asset"
| "network-request"
| "network-response"
| "har"
| "websocket"
| "sse"
| "storage"
| "interaction"
| "state-transition"
| "workflow"
| "animation"
| "media"
| "canvas"
| "webgl"
| "performance"
| "semantic-observation";
export interface CaptureViewport {
width: number;
height: number;
deviceScaleFactor?: number;
isMobile?: boolean;
hasTouch?: boolean;
}
export interface CaptureProfile {
id: string;
name: string;
authenticated: boolean;
role?: string;
locale?: string;
colorScheme?: "light" | "dark" | "no-preference";
viewport?: CaptureViewport;
featureFlags?: Record<string, string | boolean | number>;
}
export interface ProvenanceSource {
repository?: string;
commit?: string;
sourcePaths?: string[];
license?: string;
classification?: CapabilityDisposition;
adapter?: string;
tool?: string;
toolVersion?: string;
}
export interface EvidenceRecord<T = unknown> {
id: string;
targetId: string;
kind: EvidenceKind;
route?: string;
capturedAt: string;
method: string;
confidence: EvidenceConfidence;
captureProfileId: string;
viewport?: CaptureViewport;
source: ProvenanceSource;
payload: T;
}
export interface RouteNode {
id: string;
path: string;
canonicalUrl?: string;
title?: string;
evidenceIds: string[];
}
export interface RouteEdge {
from: string;
to: string;
trigger: "link" | "redirect" | "action" | "script" | "semantic-discovery";
evidenceIds: string[];
}
export interface StateTransition {
id: string;
fromState: string;
toState: string;
action: string;
evidenceIds: string[];
networkEvidenceIds?: string[];
}
export interface ProductEntity {
id: string;
name: string;
attributes: Record<string, unknown>;
evidenceIds: string[];
confidence: EvidenceConfidence;
}
export interface ProductRelationship {
id: string;
fromEntityId: string;
toEntityId: string;
type: string;
cardinality?: "one-to-one" | "one-to-many" | "many-to-one" | "many-to-many";
evidenceIds: string[];
confidence: EvidenceConfidence;
}
export interface WorkflowStep {
id: string;
action: string;
preconditions?: string[];
resultingState?: string;
evidenceIds: string[];
}
export interface ProductWorkflow {
id: string;
name: string;
steps: WorkflowStep[];
errorPaths?: WorkflowStep[][];
evidenceIds: string[];
confidence: EvidenceConfidence;
}
export interface FidelityVector {
geometry?: number;
typography?: number;
colors?: number;
assets?: number;
responsiveCoverage?: number;
routeCoverage?: number;
interactionCoverage?: number;
workflowCoverage?: number;
networkContractParity?: number;
motionParity?: number;
accessibilityPass?: boolean;
runtimeErrorCount?: number;
productionBuildPass?: boolean;
}
export interface UpstreamCapabilityRecord {
id: string;
capability: string;
disposition: CapabilityDisposition;
source: ProvenanceSource;
targetModule?: string;
rationale: string;
tests: string[];
}
export interface ReconstructionSpec {
id: string;
targetId: string;
version: string;
mode: ReconstructionMode;
schemaVersion: string;
evidenceIds: string[];
routes: RouteNode[];
routeEdges: RouteEdge[];
entities: ProductEntity[];
relationships: ProductRelationship[];
workflows: ProductWorkflow[];
fidelityRequirements: FidelityVector;
}
@@ -0,0 +1,50 @@
import { readFile, access } from "node:fs/promises";
const requiredFiles = [
"docs/MASTERPLAN.md",
"docs/STANDARDS.md",
"docs/AUTOMATIONS.md",
"docs/UPSTREAM-CAPABILITY-MATRIX.md",
"docs/spec/DIGITAL-TWIN-SPEC.md",
"packages/types/src/index.ts",
];
for (const path of requiredFiles) {
await access(path).catch(() => {
throw new Error(`Missing canonical C0PY artifact: ${path}`);
});
}
const types = await readFile("packages/types/src/index.ts", "utf8");
const requiredContracts = [
"CapabilityDisposition",
"EvidenceConfidence",
"EvidenceRecord",
"CaptureProfile",
"ProductWorkflow",
"FidelityVector",
"ReconstructionSpec",
];
for (const contract of requiredContracts) {
if (!types.includes(` ${contract}`)) {
throw new Error(`Missing canonical contract: ${contract}`);
}
}
const masterplan = await readFile("docs/MASTERPLAN.md", "utf8");
for (const disposition of [
"CORE",
"DEPENDENCY",
"CHERRY_PICK",
"ADAPTER",
"REIMPLEMENT",
"REFERENCE_ONLY",
"DROP",
]) {
if (!masterplan.includes(`\`${disposition}\``)) {
throw new Error(`Masterplan is missing capability disposition: ${disposition}`);
}
}
console.log("C0PY canonical architecture policy: OK");