ba5c796b3621749edfc057b0ef67f79d8febb27e
145
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
ba5c796b36 |
docs(analytics): anchor two security claims to the measurements behind them
CI (SIAX Cloud) / security (push) Successful in 30s
CI (SIAX Cloud) / integration (push) Successful in 56s
CI (SIAX Cloud) / quality (push) Successful in 1m23s
CI (SIAX Cloud) / integration (pull_request) Successful in 49s
CI (SIAX Cloud) / security (pull_request) Successful in 24s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m18s
Security Scan (SIAX Cloud) / security (push) Successful in 23s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 23s
Neither changes behaviour; both replace an assertion with a measurement.
BI-P014's escaping strategy rests on DuckDB treating a backslash as a literal
character, so that doubling `'` is sufficient. Verified on 1.5.5 rather than
assumed: `SELECT length('a\\b')` returns 4, and `SELECT 'a\''` is a parser
error ("unterminated quoted string") — which is exactly what a NON-escaping
backslash produces. In a dialect where `\'` closed the literal, doubling would
not be enough, so the claim is worth pinning to a run.
BI-P017's FROM/JOIN scan is conservative where it is unsure, and that has a
cost worth naming rather than discovering: ClickHouse's `ARRAY JOIN arr` takes
a COLUMN, not a table, and this guard refuses it. A false refusal an analyst
can see and report is the right side of that trade; a false ADMISSION is not.
Supporting it means teaching the scan which JOIN flavours take a non-table
target, which is the direction that grows holes — so it stays refused until
someone needs it and can add it WITH a corpus entry proving the narrowing did
not open anything else.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
|
3ee023017b |
fix(analytics): move the DuckDB live suite out of tests/integration, and add the BI docs
CI (SIAX Cloud) / security (push) Successful in 28s
CI (SIAX Cloud) / integration (push) Successful in 42s
Security Scan (SIAX Cloud) / security (push) Successful in 21s
CI (SIAX Cloud) / quality (push) Successful in 1m2s
CI (SIAX Cloud) / security (pull_request) Successful in 23s
CI (SIAX Cloud) / integration (pull_request) Successful in 40s
CI (SIAX Cloud) / quality (pull_request) Successful in 58s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 20s
CI's integration job failed, and it failed CORRECTLY. That job provides a live OpenSearch and a live ClickHouse and then asserts that nothing in tests/integration/ skipped, on the stated principle that "a skipped suite must never masquerade as a green integration gate". The DuckDB suite needs a Python interpreter that job does not provide, so it skipped, and the guard fired — the first suite in that directory ever to skip there. Moving the file to tests/duckdb/ respects the guard rather than weakening it. Loosening it to tolerate skips would have traded a real gate for one green job, which is the trade this estate keeps regretting. The suite still runs, still fails rather than skipping when DUCKDB_PYTHON is set and DuckDB is broken, and is still pinned by the committed evidence. Also fixed: "does NOT ship the Iceberg table" asserted against the FILESYSTEM when the claim is about what is COMMITTED. The live suite rebuilds that table locally, so the assertion failed whenever both suites ran in one pass — it was testing the wrong thing rather than catching a real problem. It now asserts against `git ls-files`, and also asserts the parquet/csv fixtures ARE tracked, so a version that stopped shipping any fixture would fail too. docs/analytics/BI_QUERY_METRIC_QUALITY.md is organised around what is PROVEN and what is NOT, because the second list is the one that gets dropped. It records the five findings and the six open items, including that BI-P029's MetricFlow was never executed and that BI-P021's budget is a bound rather than a hard cap (measured 130x overshoot at a 100ms poll). It also reports, without changing it, a latent flaw in .gitea/workflows/ci.yml (a protected file this branch does not touch): the integration guard's second pattern `Tests [0-9]+ passed` never matches, because vitest writes that summary with ANSI colour codes between the words. The guard is therefore STRICTER than it reads — "any skip fails" rather than "skipped and nothing passed" — so nothing is currently unguarded, but the escape hatch does not work and someone will eventually be puzzled by it. 957 tests pass with ClickHouse, Postgres and DuckDB all live. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
2bfbd3f159 |
feat(analytics): sql/009 — execution metadata, metric governance, semantic registry, quarantine, usage ledger
CI (SIAX Cloud) / integration (push) Failing after 38s
CI (SIAX Cloud) / quality (push) Successful in 1m0s
CI (SIAX Cloud) / security (push) Successful in 25s
Security Scan (SIAX Cloud) / security (push) Successful in 28s
CI (SIAX Cloud) / integration (pull_request) Failing after 48s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m16s
CI (SIAX Cloud) / security (pull_request) Successful in 27s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 23s
The migration the previous commits' comments promised. Additive and idempotent
throughout, so the restore drill can replay it onto a live database.
BI-P018 adds principal_id, workload, compiled_query_hash, datasets, metrics,
cache_status and cost_units to the execution record — and deliberately NO
`query_text` column, with a CHECK pinning the hash to 64 hex characters so a
statement smuggled into that field is refused by the database. A test asserts
the absence of `query_text` and `sql` as columns.
BI-P031's DoD gets a Postgres TRIGGER, not just the TypeScript guard: a
certified metric whose meaning_hash changes without a version bump raises. That
is the layer that holds when something writes without going through the
application — a migration script, a psql session, an admin tool. A compiler
opinion does not survive `psql`; a trigger does.
BI-P030 gets a version-history table whose CONTENT is immutable (status and
valid_from may still move) plus a PARTIAL UNIQUE INDEX enforcing at most one
ACTIVE version per model per tenant. Two active versions is a silent
correctness failure in which half the dashboards read a different model, so it
is an index rather than application discipline.
BI-P021's usage ledger is append-only by trigger: update and delete both raise.
BI-P026's freshness table DEFAULTS TO 'UNKNOWN' — a dataset nobody has computed
freshness for must not read as FRESH — and requires a non-empty disclosure,
because a state with no words attached is a state a dashboard renders as a
silent green dot. BI-P025's quarantine record requires a principal and a reason
on release.
A BUG IN THIS MIGRATION, FOUND BY RUNNING IT:
`SELECT 1 FROM pg_constraint WHERE conname = '...'` is NOT schema-scoped.
conname is unique per schema, not per database, so on a database that already
had these tables in `public` every guarded CHECK was silently skipped when
the file was applied into a second schema. The constraints simply were not
there, and the tests that should have proven them PASSED — because the
INSERTs they expected to fail succeeded, and `expect(message).toContain(...)`
on an undefined message reads as a type error, not as "the constraint is
missing". Every guard now scopes via `conrelid = '<table>'::regclass`, and
the fix is proven by applying the file to `public` FIRST and then running the
suite into a fresh schema: 20/20.
Two flaws in my own test harness, also found by running it:
* `sql.unsafe("SET search_path ...; SELECT ...")` is a multi-statement simple
query, and postgres.js returns the LAST statement's result — so a SELECT
silently returned the connection object and two assertions compared against
nonsense. search_path is now pinned on the CONNECTION.
* the fixture INSERT used column names that do not exist in sql/008's dataset
table, which surfaced as a suite-level failure rather than as 20 skips.
20 tests against a real Postgres 16, gated on ANALYTICS_TEST_DATABASE_URL, and
each of them attempts the write the constraint exists to refuse.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
|
a9ae7f25a4 |
feat(analytics): BI-P014 — DuckDB provider, and the measured finding that its sandbox does not work the way it looks
CI (SIAX Cloud) / integration (push) Failing after 42s
CI (SIAX Cloud) / security (push) Successful in 24s
CI (SIAX Cloud) / quality (push) Successful in 1m6s
Security Scan (SIAX Cloud) / security (push) Successful in 22s
CI (SIAX Cloud) / security (pull_request) Successful in 23s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 23s
CI (SIAX Cloud) / integration (pull_request) Failing after 43s
CI (SIAX Cloud) / quality (pull_request) Successful in 57s
BI-P014's security requirement is the whole prompt: DuckDB is a local engine
with `read_csv('/etc/passwd')` built in. Two layers.
LAYER 1, the mount registry: a caller names ARTIFACT IDS, never paths. The
registry resolves an id, checks the artifact belongs to the caller's tenant,
and realpath()s the result to confirm it is still inside the mount root once
symlinks are followed. A symlink to /etc/passwd, a traversing relative path, an
absolute path, and a sibling directory sharing the root's prefix
(/mnt/artifacts-evil for root /mnt/artifacts) are each refused. A cross-tenant
read is AUTHORIZATION_DENIED, deliberately not NOT_FOUND — an operator reading
the audit trail needs to see an attempted cross-tenant read as one.
LAYER 2, THE FINDING:
`SET allowed_directories=[...]` ALONE IS A NO-OP.
Measured on DuckDB 1.5.5. With allowed_directories set to a single mount
directory and nothing else, `read_csv('/etc/passwd')` RETURNED THE FILE and an
https:// URL was fetched over the network. Only when `enable_external_access
= false` is ALSO set does the allow-list bind; the same reads then fail with
"Permission Error: Cannot access file". `lock_configuration=true` afterwards
stops a query re-opening either.
So the obvious spelling of this control — set the allow-list and move on —
yields a setting that appears in duckdb_settings(), looks like a sandbox, and
permits everything. That is `trivy --exit-code 0` in a different costume. The
bootstrap is therefore emitted as ONE unit, `assertSessionSandboxed` refuses a
session missing any part of it, and the measurement itself is PINNED by a test
that never skips: if someone simplifies the bootstrap, the pinned evidence
(`read_etc_passwd_succeeded: true` for the naive spelling) keeps the finding
attached to the build rather than to a comment that would rot.
Twelve bypass attempts run against a real DuckDB with LAYER 1 REMOVED — sent
straight to the engine, registry not in the picture. Zero succeeded. The
counterfactual runs in the same session and is asserted: the mounted file IS
readable, so "nothing succeeded" is not a sandbox that blocks everything, which
would be an outage rather than a control. A locked session is then shown unable
to re-open its own sandbox, all three ways.
DoD, against real fixtures: Parquet aggregates correctly (SE 394.5, NO 105.5),
CSV round-trips a value containing a quote, and an ICEBERG SNAPSHOT reads via
iceberg_scan on an exact metadata JSON — which is not a workaround: DuckDB
refuses to glob for the latest version because globbing can read uncommitted
data, and an exact metadata file is what "snapshot" means. Arrow is reported
UNPROVEN by `readiness()` with the reason: DuckDB's arrow extension is a
network install the offline environment could not download.
THREE DEFECTS IN MY OWN TESTS, found and fixed rather than tidied away:
* a bypass-corpus entry was `SELECT 1 FROM (SELECT 1) t` — not an attack at
all. It succeeded and was reported as a leak. A corpus entry that cannot
fail is as useless as a check that cannot. Replaced with real ATTACH and
COPY attempts.
* the harness ran the session bootstrap outside its try block, so a statement
that correctly REFUSED to re-open a locked sandbox killed the subprocess
and surfaced as an unreadable "Command failed" instead of as the meaningful
refusal it was.
* the Iceberg fallback was `expect(evidence.iceberg ?? "unbuilt").toBeDefined()`
— a check that can never fail. It let a run without the fixture ERASE the
recorded Iceberg result from the evidence file while both suites stayed
green. It now rebuilds the fixture when pyiceberg is available and, when it
is not, asserts against the previously recorded result and marks it as
carried forward.
The Iceberg fixture is deliberately NOT committed: pyiceberg writes absolute
paths into its metadata and Avro manifests, so a committed table resolves only
on the machine that built it, and a fixture that silently fails everywhere else
is worse than none. tests/fixtures/duckdb/README.md says so and gives the
rebuild command; the recorded result is the artefact instead.
Heavy artifacts are refused with CAPABILITY_UNAVAILABLE and pointed at N0D,
per BI-P014 — silently running a 40 GB scan in the API process is how a preview
engine takes down a control plane.
35 unit tests, 12 live tests. Evidence: evidence/bi-p014/duckdb-sandbox.json.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
|
ec26ddc296 |
feat(analytics): BI-P022 — dbt canonical transformation CONTRACT and reference project
CI (SIAX Cloud) / security (push) Successful in 22s
CI (SIAX Cloud) / integration (push) Successful in 38s
CI (SIAX Cloud) / quality (push) Successful in 52s
Security Scan (SIAX Cloud) / security (push) Successful in 20s
CI (SIAX Cloud) / security (pull_request) Successful in 22s
CI (SIAX Cloud) / integration (pull_request) Successful in 37s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 20s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m1s
Scope stated in the module's first paragraph: this defines the contract, it
does not run dbt, and nothing in this repo does. Execution is N0D's (n0d#38),
which consumes this from the other side. There is deliberately no DbtExecutor
implementation here — only the interface — so "0pt can run dbt" cannot become
true by accident and the two repos share one versioned shape.
`analytics/dbt/` is a real reference project covering raw -> staging ->
curated -> serving, with the four layers mapped onto BI-P003's existing tiers
rather than inventing a fifth vocabulary. The tenant predicate is applied at
the FIRST modelled layer so every downstream model inherits it and no later
model has to remember — the same forced-predicate rule as the query compiler's,
one layer up.
Two singular tests ship with it, both aimed at false greens:
* `assert_mart_is_not_empty` — without it every column test passes trivially
over zero rows and the whole suite reports green while the pipeline is dead;
* `assert_no_cross_tenant_rows` — which should be impossible to fail, and
exists precisely for that reason. An isolation guarantee nobody tests is the
one that turns out to have four holes rather than the one reported.
BI-P022's version rule is a GATE. `DBT_RUNTIME_PIN` records dbt Core 1.10.13
(stable) with dbt-clickhouse 1.9.2 and BOTH review flags FALSE — no licence or
supply-chain review has been performed in this repo and none is claimed.
`assertDbtRuntimeCertified` refuses a production run until they are true, and
refuses an alpha line in production without a separate certification reference.
A version constant on its own would be the documented-but-unenforced pattern;
a test pins the flags at false so flipping them is a visible deliberate act.
The result is treated as UNTRUSTED INPUT, because it arrives from another
repo's executor and this estate has been burned by green statuses that meant
nothing. Refused: `status: success` with ZERO nodes executed (a dbt selector
matching nothing exits 0 — the dbt-shaped false green); an observed runtime
different from the pin (the models mean something else and nothing says so); a
revision that is not an exact 40-char SHA (a branch name is not reproducible,
and reproducibility is the DoD); a missing manifest.json or run_results.json,
without which the run cannot be audited or its tests ingested by BI-P025.
Layer dependencies are checked against the manifest GRAPH, not asserted as a
convention: a mart reading a source directly, or skipping a layer, or a
staging model reading a mart, are each refused.
Selectors are validated rather than passed through — a free-form argument list
is a shell surface, and a shell surface on a job holding warehouse credentials
is the whole problem. The job also cannot name an arbitrary target schema: it
must be scoped to the job's tenant.
39 tests. typecheck + eslint clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
|
6cb00add57 |
feat(analytics): BI-P025/BI-P026 — quality providers that fail on empty, freshness that refuses to be silent
CI (SIAX Cloud) / integration (push) Successful in 43s
CI (SIAX Cloud) / security (push) Successful in 23s
CI (SIAX Cloud) / quality (push) Successful in 57s
Security Scan (SIAX Cloud) / security (push) Successful in 21s
CI (SIAX Cloud) / security (pull_request) Successful in 23s
CI (SIAX Cloud) / integration (pull_request) Successful in 40s
CI (SIAX Cloud) / quality (pull_request) Successful in 55s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 25s
Both prompts are about a FALSE GREEN, so both are tested by constructing the false green and requiring a failure. BI-P025. Three provider kinds, and the interface makes the difference visible instead of implying parity: `providerKind` is "executing" or "ingesting". The native provider runs checks; the dbt and Great Expectations adapters INGEST results produced elsewhere (n0d#38 owns dbt execution against BI-P022, and GE is Python — nothing here runs it). Calling all three "providers" while only one executes is how "we have Great Expectations coverage" ends up in a status report. THE FALSE GREEN, proven against a live ClickHouse 24.8.14.39: a `not_null` check over an EMPTY table returns violations=0, evaluated=0. The raw counts are asserted first — the engine really does say zero-over-zero — and then the provider is required to call it a FAILURE, not a pass. Every rule compiles to a VIOLATION expression rather than a "rows that are fine" count, so the distinction cannot be lost at the runner layer. Same treatment for the ingesting providers: a MISSING dbt result is a failure, and dbt's `error`, `warn` and `skipped` are each asserted not to be passes. `skipped` in particular is the exact shape of l0re's 28 silently-skipped Postgres tests. The gate: zero verdicts QUARANTINES. "No rules ran" and "everything passed" produce the same green in most implementations, which is how a quality gate stops gating without anyone noticing — api0's `trivy --exit-code 0` in another costume. A later passing run does not auto-release either; release takes a principal and a reason, or a flapping rule would quarantine and un-quarantine a dataset between two reads of the same dashboard. BI-P025's DoD is the READ side: a quarantined projection is REFUSED, not annotated. An annotation on a dashboard nobody reads is not a quarantine. Live counts, all exact, from evidence/bi-p025/native-quality-live.json: not_null 5/118, range 3, distribution 4, unique 11. The unique figure is 11 rather than the 6 the first version of the test asserted — the five empty-string customers are duplicates of each other, so not_null and unique overlap on the same defect. The engine was right and the fixture arithmetic was wrong; the breakdown is written into the test rather than the number quietly adjusted. Tenant isolation re-proven on this path specifically, because a quality run uses an ADMINISTRATIVE credential: with 618 rows in the base table, t_alpha evaluates 118 and t_beta 500, and t_beta's blocking failure quarantines t_beta alone. Without the forced predicate one tenant's dirty rows would quarantine another tenant's projection. BI-P026. Five states, and the DoD's operative word is "silently". So the gate sits at the boundary every result crosses: a result with NO freshness envelope is unreleasable — not defaulted to FRESH, not defaulted to UNKNOWN — because "we forgot to attach it" and "the data is current" are indistinguishable to a reader, and any default makes forgetting the silent path. A non-current state without an explicit acknowledgement is refused, so a consumer that ignores the envelope cannot render stale numbers as current. FAILED outranks age: a pipeline that just failed may still hold recently-timestamped data from the previous run, and reporting that as FRESH is how a broken pipeline stays invisible for a day. UNKNOWN is never collapsed into FRESH. 54 unit tests, 11 live tests. typecheck + eslint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
65cce86144 |
feat(analytics): BI-P028/BI-P029/BI-P030/BI-P031 — metric model, semantic registry, governance, semantic provider
CI (SIAX Cloud) / security (push) Successful in 29s
CI (SIAX Cloud) / integration (push) Successful in 52s
CI (SIAX Cloud) / quality (push) Successful in 1m10s
Security Scan (SIAX Cloud) / security (push) Successful in 25s
CI (SIAX Cloud) / security (pull_request) Successful in 29s
CI (SIAX Cloud) / integration (pull_request) Successful in 53s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m13s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 22s
BI-P028 extends the BI-P002 MetricDefinition rather than defining a second,
parallel metric — a second metric type would be the exact failure the DoD is
about. `MetricSpec` is the fifteen-field shape the prompt names; explicit
bridge functions map it onto the persisted row plus the six columns it adds.
The DoD ("no production metric exists only inside a dashboard tool") is
enforced by something that refuses: a dashboard panel carrying an inline
expression, inline aggregation or inline SQL is rejected, as is one naming an
unregistered or RETIRED metric. So a metric cannot come into existence inside
Superset and quietly become production.
Validation refuses the metrics that render as a plausible number and mean
nothing: no description, no owner, no time dimension, a ratio without a
denominator, a currency format with no currency (which is how SEK gets
rendered as USD), a filter on a dimension the metric does not declare.
BI-P031's DoD is checked on a MEANING HASH rather than a field diff, so a
field this code has never heard of still counts as changing the meaning —
fail-closed rather than fail-open on future additions. Eight semantic fields
are each mutated and each refused; three editorial fields are each mutated and
each allowed, because forcing a version bump to fix a typo is how teams learn
to route around governance. A new version starts DRAFT and EXPERIMENTAL: it
never inherits the certification, or a certified metric's meaning could change
while still wearing the badge.
BI-P030 stores the semantic model separately from any engine, and that
negative is enforced two ways after the first way turned out to be
insufficient:
* a VOCABULARY scan rejects a model mentioning metricflow / clickhouse /
trino / cube / superset anywhere in its serialized form;
* a STRUCTURAL check rejects any top-level key outside BI-P030's field list.
The second exists because a test expected the vocabulary scan to catch a
MetricFlow-shaped manifest and it did not — the manifest never says
"metricflow". The structural check holds regardless of what a document calls
itself. The manifest is now also stamped with its generator so a leaked copy
is detectable.
The replaceability DoD is tested with TWO providers whose naming rules differ
(MetricFlow-shaped and Cube-shaped, `ds_sales.amount_sek`): the canonical id
set must be identical across both AND the provider bindings must differ —
otherwise the equality would hold trivially for a registry that only ever had
one provider.
BI-P029 — WHAT IS NOT PROVEN IS STATED IN THE MODULE AND RETURNED BY THE CODE.
MetricFlow is a Python package and was NOT executed: no Python runtime is
provisioned here, and BI-P029's "pin a currently verified version" requires a
licence/supply-chain review that has not happened. `providerReadiness()`
returns `verified: false` and names all four open items, including that the
Superset path is unimplemented and the AI Analyst path uncompared. The
constant `METRICFLOW_PINNED_VERSION` is labelled a declared intent, not a
certification. A test asserts the readiness report says so, so the honesty
cannot rot silently.
353 tests. typecheck + eslint clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
|
ca0a3292df |
feat(analytics): BI-P020 — provider-neutral query cache that cannot serve a stale or foreign result
CI (SIAX Cloud) / security (push) Successful in 23s
Security Scan (SIAX Cloud) / security (push) Successful in 19s
CI (SIAX Cloud) / integration (push) Successful in 38s
CI (SIAX Cloud) / quality (push) Successful in 56s
CI (SIAX Cloud) / security (pull_request) Successful in 22s
CI (SIAX Cloud) / integration (pull_request) Successful in 40s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 22s
CI (SIAX Cloud) / quality (pull_request) Successful in 54s
All six key components BI-P020 lists are REQUIRED, and the builder refuses to
produce a key when one is missing rather than defaulting it. A defaulted
component is how a cache serves a result computed under a metric definition
that no longer exists — that answer is not stale, it is WRONG, and it looks
current.
"Never share tenant result cache" is enforced structurally, in three layers,
because the obvious one is a naming convention:
* the key is prefixed with the tenant id, derived here from the verified
identity and never passed in;
* the stored ENTRY records its own tenant id, and every read checks the
entry against the caller — so a prefix collision or a shared backing store
still cannot serve a foreign result;
* a cross-tenant hit is REFUSED and the poisoned entry EVICTED, not returned
as a miss. Returning undefined would hide a security event.
The third point is the one the test corpus exercises directly: an entry stored
under t_alpha's key carrying t_beta's id, and a forged key that a SCAN glob
would happily include. ST0RE was relying on naming conventions when it turned
out to have four cross-tenant paths rather than the one reported.
Staleness is tested by mutating each key component ONE AT A TIME and requiring
a miss for every one — and a coverage assertion compares the mutation list
against the component type, so adding a field to CacheKeyComponents without
adding a mutation fails the suite instead of going untested.
ValkeyResultCache takes the client as an injected interface, so no dependency
and no CVE surface is added — the same reason BI-P005's Arrow writer is
hand-written. It sets a store-side TTL as well as recording one (a recorded
timestamp alone leaves expired results resident forever), pages with SCAN
rather than blocking on KEYS, and removes a corrupt entry instead of letting
it become a permanent silent miss.
32 tests. typecheck + eslint clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
|
8f6357044f |
feat(analytics): BI-P018/BI-P019/BI-P021 — execution model, verified cancellation, a budget that stops queries
CI (SIAX Cloud) / integration (push) Successful in 43s
CI (SIAX Cloud) / security (push) Successful in 21s
CI (SIAX Cloud) / quality (push) Successful in 1m5s
Security Scan (SIAX Cloud) / security (push) Successful in 23s
CI (SIAX Cloud) / security (pull_request) Successful in 24s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m0s
CI (SIAX Cloud) / integration (pull_request) Successful in 42s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 38s
BI-P018 — every field the prompt names, plus the one it names by its absence.
The record carries `compiled_query_hash` and never the statement, and that is
ENFORCED: `assertNoRawSql` runs before any record is emitted and refuses one
whose `error` field looks like SQL. The realistic leak is ClickHouse's own
error text, which echoes the query — the test uses exactly that string. The
recorder writes at EVERY transition, because a recorder that only writes at
the end loses precisely the executions worth investigating. Illegal state
transitions throw rather than overwrite a terminal row.
The one place raw SQL is retained stays BI-P017's analyst statement history,
where a privileged human ran unreviewed SQL and "who ran what" must survive.
Different record, different retention posture, stated in both files.
BI-P019 — "stops" is VERIFIED, not requested. Aborting the client stops US
waiting, not the server; KILL QUERY without SYNC tells you nothing; even SYNC
can return while the query unwinds. So cancelAndVerify aborts, kills, then
POLLS the process list until the query is GONE or the deadline passes — and
if the deadline passes it returns `verified_stopped: false` rather than a
cheerful `true`. The return type is an outcome object, not a boolean, so a
caller cannot mistake "we asked" for "it stopped". An unreachable engine is
read as "still running" (fail closed), so the kill is still attempted at the
moment an operator most needs it to be.
Proven live against ClickHouse 24.8.14.39, with the counterfactual first:
* an identical bomb left alone IS still running after 2s — so "gone after
we cancelled" is not just "the query was short";
* the cancelled query had read 6,559,476,156 rows, and was verified gone
57ms after the request;
* system.query_log records ExceptionWhileProcessing with code 394,
QUERY_WAS_CANCELLED — ClickHouse's verdict, not ours.
BI-P021 — the honest version. A budget checked once before execution bounds
nothing, and the module says so in its first paragraph. ClickHouse's
max_rows_to_read bounds ONE query; it cannot express "this tenant may scan X
per hour", which is what a budget is for. So enforcement polls the engine's
live read_rows/read_bytes counters and CANCELS the running query, through
BI-P019 with verification, when window spend would be exceeded.
Measured live: a 1,000,000-row window cancelled the query at 130,883,409 rows,
92ms in, verified stopped, code 394. THE OVERSHOOT IS DOCUMENTED, in the module
and in the evidence: enforcement is granular to the poll interval, so this is
a bound of the form "within roughly one poll interval of engine throughput",
not a hard cap. Quoting it as a hard cap would be wrong.
Two more things stated rather than implied:
* `enforceInFlight: false` is reported by describeEnforcement() as
"NOT ENFORCED — this budget is advisory", and the constructor REFUSES to
accept enforceInFlight without a process control;
* an unshared window store is reported as non-binding across replicas
("with N replicas a tenant can spend up to N times this budget").
* a query the budget killed is still charged for the work it burned. Not
charging is how a tenant learns the cheapest query is a cancelled one.
Evidence: evidence/bi-p019-p021/cancellation-and-budget.json
268 unit tests, 7 live tests. typecheck + eslint clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
|
adc7a9be31 |
fix(analytics): replace the control-character class with a code-point scan (no-control-regex)
CI (SIAX Cloud) / quality (push) Successful in 45s
CI (SIAX Cloud) / integration (push) Successful in 34s
CI (SIAX Cloud) / security (push) Successful in 19s
Security Scan (SIAX Cloud) / security (push) Successful in 17s
CI (SIAX Cloud) / quality (pull_request) Successful in 45s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 27s
CI (SIAX Cloud) / security (pull_request) Successful in 37s
CI (SIAX Cloud) / integration (pull_request) Successful in 45s
The ASCII-only check used a regex character class of control characters. It tripped eslint's no-control-regex, which the repo's quality gate enforces — and the rule has a point: such a class is unreadable and is exactly the kind of expression a reviewer nods at without checking. Same behaviour, written as an explicit scan naming the permitted set. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
d9c8f7190b |
feat(analytics): BI-P017 — analyst SQL contract, proven against live ClickHouse with the guard removed
CI (SIAX Cloud) / quality (push) Successful in 50s
CI (SIAX Cloud) / integration (push) Successful in 35s
CI (SIAX Cloud) / security (push) Successful in 21s
Security Scan (SIAX Cloud) / security (push) Successful in 26s
CI (SIAX Cloud) / quality (pull_request) Successful in 46s
CI (SIAX Cloud) / integration (pull_request) Successful in 35s
CI (SIAX Cloud) / security (pull_request) Successful in 19s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 16s
AnalyticsSQLSession is a SEPARATE class, not a method on SiaxAnalyticsEngine.
executeMetric has no raw-SQL surface at all; on the same object the closed DSL
would be one refactor away from growing one. ClickHouseAnalyticsEngine
.executeQuery stays a documented refusal.
FOUR LAYERS, and the guard is deliberately called the weakest:
1 role — the CL0UD analyst role is checked when the SESSION IS OPENED, so
a caller without it never obtains an object that can run SQL.
2 guard — statement admission. Comments and literals removed with
ClickHouse's own lexical rules, ASCII-only outside literals, one
statement, SELECT/WITH only, and every table reference POSITIVELY
resolved against the tenant's catalog. Deny-lists are a second
opinion, never the mechanism.
3 grants — a per-tenant ClickHouse user with SELECT on that tenant's catalog
alone, plus a restrictive row policy on the base tables.
4 views — the tenant catalog holds only VIEWS, each baking
`WHERE tenant_id = '<tenant>'`, and tenant_id is not projected.
61 bypass attempts, in one shared corpus, run TWICE: against the guard, and
against a live ClickHouse 24.8.14.39 with LAYER 2 REMOVED. The live assertion
is not "the engine errored" — it is "no canary came back", with BETA_CANARY
seeded into the other tenant's rows and SECRET_CANARY into an ungranted table.
The counterfactual runs first: as admin the same base table returns 100 rows
and BETA_CANARY IS readable, so the isolation numbers mean something.
Result: 0 of 61 leaked. Evidence: evidence/bi-p017/analyst-sql-live-bypass.json.
FOUR DEFECTS FOUND BY WRITING TESTS THAT CAN FAIL — three of them mine:
* `CREATE ROW POLICY IF NOT EXISTS` is a SILENT NO-OP on re-provisioning.
Dropping the user empties the policy's apply-to list; the next run skips
re-creation, so the policy survives with `apply_to_list = []`, applying to
nobody, while `system.row_policies` still shows the correct select_filter.
The restricted user read 100 of 100 rows — BOTH TENANTS — from the base
table. Fixed with OR REPLACE + `rowPolicyVerificationSql`, which turns
"the policy is applied" from an assumption into a query, and a test that
asserts applied_to_user = 1 per object.
* `CREATE SETTINGS PROFILE IF NOT EXISTS` has the identical shape: a changed
budget never lands. A stale 60-row ceiling survived re-provisioning and
starved every legitimate statement, with the DDL reporting success
throughout. Also OR REPLACE now.
* `describe.skipIf(!reachable)` where `reachable` is set in `beforeAll`
skips ALWAYS — vitest evaluates skipIf at collection time. Ten tests
reported green while skipping against a live, reachable server. Replaced
with the repo's existing env-var gate, which FAILS when CLICKHOUSE_URL is
set and the server is down rather than degrading to a skip.
* MEASURED, and it changes the design: a ClickHouse user whose profile
carries `readonly = 1` cannot have ANY setting changed from the request
(Code: 164). The BI-P012 ceilings therefore cannot travel with an analyst
statement; they are baked into the user's SETTINGS PROFILE at provisioning.
Proven both ways: over-budget scan fails with code 158 from the profile
with no setting sent, and an attempt to raise it from the request fails
with code 164.
Honest about what the guard does NOT catch: `SELECT tenant_id FROM
t_alpha_sql.sales` is structurally legitimate and IS admitted. It fails at the
engine because the view does not project the column. Recorded in its own
GUARD_ADMITS_ENGINE_REFUSES list and asserted live, rather than deleted from
the corpus.
evidence/bi-p017 is pinned by tests/analytics/analyst-sql-evidence.test.ts,
which never skips: adding an attack without re-running the live suite breaks
the build, and the pin also checks the counterfactual numbers so the evidence
cannot be a recording of an empty database.
246 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
|
b1884a89a5 |
feat(analytics): BI-P016 — canonical query request contract, two compile targets
CI (SIAX Cloud) / security (push) Successful in 25s
Security Scan (SIAX Cloud) / security (push) Successful in 24s
CI (SIAX Cloud) / integration (push) Successful in 39s
CI (SIAX Cloud) / security (pull_request) Successful in 25s
CI (SIAX Cloud) / quality (push) Successful in 57s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 20s
CI (SIAX Cloud) / integration (pull_request) Successful in 1m38s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m59s
The eight-field AnalyticsQueryRequest BI-P016 names, verbatim, with the rule
that makes it safe enforced structurally rather than by convention: there is
no field that can carry SQL, and parseAnalyticsQueryRequest REJECTS unknown
keys instead of ignoring them. Ignoring an unsupported field is how it becomes
a supported one after the next refactor.
`tenant`, `tenant_id`, `project_id`, `database`, `catalog` and `table` are
rejected by name. INF0's Wave-2 cross-tenant hole was a tenant read out of a
request body; the fix that sticks is a request type in which the field does
not exist, and a parser that refuses it when a caller sends it anyway.
BI-P016's DoD ("same request compiles to ClickHouse/semantic engine without
consumer rewrite") is tested as an equality, not narrated: both targets are
projected onto a comparable shape and compared. The ClickHouse side of that
projection is obtained by PARSING the emitted SQL and its bound parameters in
the test — not by reading a field the compiler helpfully returned — so a
compiler that silently drops a filter fails the comparison instead of passing
it. The parser throws on any condition it does not recognise, which is what
would catch unbound caller text appearing in the statement.
BI-P011's forced tenant predicate is re-proven against this new path: five
request shapes are enumerated and each must have FORCED_TENANT_PREDICATE as
condition[0]; a blank tenant refuses to compile rather than emitting no
predicate; a hostile filter value appears only in params, never in sql.
45 tests, all failing-capable: every rejection assertion checks the SiaxError
code, and the DoD assertions compare reconstructed meaning rather than strings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
|
b81484e2b5 |
Merge pull request 'feat(analytics): BI-P002..BI-P012 — 0pt analytics core' (#34) from feat/bi-p002-p012-analytics-core into main
Deploy to Coolify / deploy (push) Successful in 4s
CI (SIAX Cloud) / security (push) Successful in 24s
Security Scan (SIAX Cloud) / security (push) Successful in 36s
CI (SIAX Cloud) / integration (push) Successful in 53s
CI (SIAX Cloud) / quality (push) Successful in 57s
|
||
|
|
84bb9faa4f |
feat(analytics): BI-P002..BI-P012 analytics core — domain model, tiers, contracts, Arrow, ClickHouse engine, isolation, governance
CI (SIAX Cloud) / security (push) Successful in 41s
Security Scan (SIAX Cloud) / security (push) Successful in 40s
CI (SIAX Cloud) / integration (push) Successful in 52s
CI (SIAX Cloud) / quality (push) Successful in 1m5s
CI (SIAX Cloud) / security (pull_request) Successful in 31s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 29s
CI (SIAX Cloud) / integration (pull_request) Successful in 45s
CI (SIAX Cloud) / quality (pull_request) Successful in 53s
Implements the eight BI program prompts that make 0pt the canonical SIAX
analytics capability. Every file is NEW: not one existing file is modified,
and src/analytics/clickhouse-analytics.ts is byte-identical to main.
BI-P002 — the nineteen canonical entities, verbatim, with tenant_id +
project_id on every tenant-owned record. sql/008 gives each one a table.
The four version entities BI-P002 names as immutable are enforced by BEFORE
UPDATE/DELETE triggers, not by a TypeScript `readonly` — proven by direct SQL
against a live Postgres.
BI-P003 — RAW/NORMALIZED/CURATED/SERVING with the read rule enforced:
assertReadableByBiSurface refuses a RAW dataset to a dashboard unless a
recorded exception exists. All eight required declarations are NOT NULL.
BI-P004 — DatasetContract and a fail-closed compatibility gate. Fifteen
breaking-change classes are each tested twice: refused under a minor bump,
accepted under a major one.
BI-P005 — Arrow IPC alongside JSON, with a hand-written FlatBuffers writer so
no npm dependency (or CVE surface) is added. Conformance comes from three
independent implementations, not from self-agreement: pyarrow 25.0.1 and
DuckDB 1.5.5 read identical values from both ClickHouse's Arrow and ours, and
ClickHouse's own Arrow C++ ingests our 20,000-row batch with exact aggregates.
BI-P009 — SiaxAnalyticsEngine with all eight capabilities. The existing
ClickHouseAnalytics is COMPOSED, never replaced. executeQuery (analyst SQL) is
declared and deliberately refused with CAPABILITY_UNAVAILABLE until BI-P017 and
BI-P033 land, rather than opening the raw-SQL hole BI-P011 enumerates.
BI-P010 — four purpose-built tables, not one giant one, and the difference is
measured against the counterfactual: 48.8ms/40,960 rows vs 174.4ms/5,400,000
rows for the same question; 5/660 granules vs 660/1149.
BI-P011 — layered isolation, each layer proven with the layer above removed.
Without the forced predicate the same statement returns 50,050 (both tenants);
with it, t_alpha gets 50. Without a row policy an unrestricted user sees 2
tenants; the restricted user running the IDENTICAL statement sees 1.
BI-P012 — every named limit, fired individually and identified by ClickHouse
error code, plus the half usually skipped: an unrelated tenant is timed while
a real CPU bomb is in flight and must still return the CORRECT answer.
Three defects found by writing tests that can fail:
* `CHECK (array_length(purposes,1) >= 1)` can NEVER fail — array_length is
NULL for '{}' and a NULL CHECK passes. Fixed with coalesce(); the test
that caught it is pinned.
* EXPLAIN ESTIMATE run under the caller's own max_rows_to_read fails with
code 158 exactly when the estimate would be over budget. Planner calls now
use a narrower envelope.
* result_overflow_mode=break returns HTTP 200 with 65,409 of 100,000 rows
and no indication it is short. Every SELECT ships `throw`.
Verified rather than assumed: server7's ClickHouse holds `opt_scraped` (51
tables, 407,793,471 rows, `persons`/`cz_owners` included) — the b0ts plane this
module is prohibited from touching — has no `opt_analytics` database, zero
tenant_id columns and zero row policies. Nothing there was modified; every
proof ran against disposable containers.
Evidence: evidence/bi-p002-p012/{test-transcript.txt,bi-p010-benchmark.json,
bi-p005-arrow-roundtrip.json,server7-clickhouse-probe.md}
Docs: docs/analytics/BI_ANALYTICS_CORE.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
|
8d0a288668 |
Merge pull request 'feat(compliance): scaffold Golden Standard artifacts (CI-14, DX-02, DX-03)' (#21) from feat/golden-standard-scaffold into main
Deploy to Coolify / deploy (push) Successful in 5s
CI (SIAX Cloud) / security (push) Successful in 22s
CI (SIAX Cloud) / integration (push) Successful in 39s
CI (SIAX Cloud) / quality (push) Successful in 50s
Security Scan (SIAX Cloud) / security (push) Successful in 24s
Reopened after a false batch-closure on 2026-09-04. The stated reason -- "added-files
present: all; modified files: none" -- was untrue on all four counts: .gitea/CODEOWNERS and
docs/compliance/SCAFFOLD-TODO.md were absent from main, and the other two files differed.
The PR also carried a real merge conflict (pull_request.status=0), which by itself disproves
"already on main".
Resolved against what main says today (
|
||
|
|
e433e7f991 |
Merge main into the Golden Standard scaffold branch — keep only CODEOWNERS
CI (SIAX Cloud) / integration (push) Successful in 29s
CI (SIAX Cloud) / security (push) Successful in 23s
CI (SIAX Cloud) / quality (push) Successful in 53s
CI (SIAX Cloud) / integration (pull_request) Successful in 32s
CI (SIAX Cloud) / security (pull_request) Successful in 24s
CI (SIAX Cloud) / quality (pull_request) Successful in 53s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 21s
Security Scan (SIAX Cloud) / security (push) Successful in 21s
PR #21 was batch-closed 2026-09-04 as "verified against main (added-files
present: all)". That check cannot have run: `.gitea/CODEOWNERS` and
`docs/compliance/SCAFFOLD-TODO.md` were absent from main, and the other two
files differed. Reopened; this merge resolves it against what main says today.
Resolution, file by file, measured against origin/main (
|
||
|
|
08c88f39c8 |
Merge pull request 'docs: the Coolify deploy hook is LIVE — six docs still call it a placeholder' (#33) from docs/coolify-deploy-hook-is-live-opus5 into main
Deploy to Coolify / deploy (push) Successful in 7s
CI (SIAX Cloud) / security (push) Successful in 29s
CI (SIAX Cloud) / integration (push) Successful in 39s
Security Scan (SIAX Cloud) / security (push) Successful in 25s
CI (SIAX Cloud) / quality (push) Successful in 1m5s
Six docs on main claimed no Coolify app was provisioned and that cd.yml's guard fails
on purpose. Measured otherwise: repo secret COOLIFY_TOKEN exists (2026-08-31T11:34:34Z),
repo variable COOLIFY_APP_ID holds a real 24-char UUID, both guards pass, and the push to
main at
|
||
|
|
853f09320c |
docs: probed 0pt.app — Vercel answers with HTTP 402 DEPLOYMENT_DISABLED
CI (SIAX Cloud) / security (push) Successful in 31s
CI (SIAX Cloud) / integration (push) Successful in 47s
CI (SIAX Cloud) / quality (push) Successful in 1m6s
Security Scan (SIAX Cloud) / security (push) Successful in 29s
CI (SIAX Cloud) / security (pull_request) Successful in 26s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 26s
CI (SIAX Cloud) / integration (pull_request) Successful in 41s
CI (SIAX Cloud) / quality (pull_request) Successful in 56s
The previous commit said "nobody has probed the running site". I then probed it, so the docs should not keep hedging. curl -sI https://0pt.app/ HTTP/2 402 server: Vercel x-vercel-error: DEPLOYMENT_DISABLED strict-transport-security: max-age=63072000 https://0pt.app/api/health -> HTTP 402, body "Payment required / DEPLOYMENT_DISABLED" Two facts hold simultaneously and the docs now state both: the Coolify deploy hook fires for real on every push to main, AND the public product domain is dark behind a disabled Vercel deployment with DNS never cut over. Neither cancels the other. Still not established, and still marked as such: whether the Coolify app builds and starts. A deploy endpoint returning 200 means the request was accepted, not that a container came up healthy. Side note worth recording for PR #19: the HSTS header on that 402 response is Vercel's platform default, which is exactly the disappearing-on-cutover risk next.config.mjs in PR #19 was written to remove. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
d7a1c4d5e1 |
docs: the Coolify deploy hook is LIVE — six docs still say it is a placeholder
CI (SIAX Cloud) / security (push) Successful in 31s
CI (SIAX Cloud) / integration (push) Successful in 42s
Security Scan (SIAX Cloud) / security (push) Successful in 44s
CI (SIAX Cloud) / quality (push) Successful in 1m6s
CI (SIAX Cloud) / security (pull_request) Successful in 30s
CI (SIAX Cloud) / integration (pull_request) Successful in 37s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 26s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m2s
Six documents on main assert that no Coolify app has been provisioned for 0pt and
that cd.yml's COOLIFY_APP_ID is a literal placeholder whose guard fails the job on
purpose. That is no longer true, and the gap is not cosmetic: it means a merge to
main silently ships to production while the repo's own docs promise it cannot.
Measured, not assumed:
* repo Actions secret COOLIFY_TOKEN exists, created 2026-08-31T11:34:34Z
(GET /api/v1/repos/sax3l/0pt/actions/secrets)
* repo Actions variable COOLIFY_APP_ID exists with a real 24-character value,
not the "TODO-no-coolify-app-provisioned-yet" literal
(GET /api/v1/repos/sax3l/0pt/actions/variables)
* both guards in .gitea/workflows/cd.yml therefore pass; on the push to main at
|
||
|
|
1ff344107f |
Merge pull request 'docs: withdraw the "Superset / OpenMetadata are OUT" rows — both are canonical' (#32) from docs/bi-canonical-stack-correction-opus5 into main
Deploy to Coolify / deploy (push) Successful in 4s
CI (SIAX Cloud) / quality (push) Successful in 1m9s
CI (SIAX Cloud) / security (push) Successful in 28s
CI (SIAX Cloud) / integration (push) Successful in 46s
Security Scan (SIAX Cloud) / security (push) Successful in 37s
docs/02-architecture.md's "Stack decisions -- OUT (do not add)" table rejected Apache
Superset and DataHub/OpenMetadata, and its IN table named Metabase as the BI choice.
All three contradict 0pt's role as canonical Analytics/Data/Semantic owner (no BI0 repo):
Superset is the canonical BI engine, OpenMetadata the canonical catalog ahead of DataHub,
and Metabase/DataHub/Lightdash/Rill/Grafana are optional App Catalog apps only.
The repo already shipped both "rejected" tools -- src/metadata/openmetadata-adapter.ts
(STATUS.md: IMPLEMENTED) and a superset service in
services/infra/docker-compose.data-stack.yml, inventoried in docs/compliance/OSS-INVENTORY.md
-- so the old rows were self-contradicting, not merely stale.
Docs only. Falsifiable check: FAIL on main, PASS on the branch, FAIL again under each of
two independent mutations. siax-doctor 39/40 on both branch and main (the one failure,
P1-15 CODEOWNERS, is pre-existing and is what PR #19 supplies). All eight CI contexts green
on
|
||
|
|
4bc79c25d3 |
docs: withdraw the "Superset / OpenMetadata are OUT" rows — both are canonical
CI (SIAX Cloud) / integration (push) Successful in 36s
CI (SIAX Cloud) / quality (push) Successful in 1m3s
CI (SIAX Cloud) / security (push) Successful in 23s
Security Scan (SIAX Cloud) / security (push) Successful in 31s
CI (SIAX Cloud) / integration (pull_request) Successful in 32s
CI (SIAX Cloud) / security (pull_request) Successful in 29s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 27s
CI (SIAX Cloud) / quality (pull_request) Successful in 55s
`docs/02-architecture.md` was written when 0pt was planned as a standalone SaaS
product. Its "Stack decisions — OUT (do not add)" table rejects Apache Superset
("Metabase already covers BI") and DataHub/OpenMetadata ("enterprise-heavy"),
and the "IN" table names Metabase as the BI choice.
Under the SIAX Cloud analytics decision, 0pt is the canonical
Analytics/Data/Semantic owner — there is no BI0/Analytics0 repo and no separate
Superset platform. Superset is the canonical BI engine (Apache-2.0, official
embed SDK + guest tokens) and OpenMetadata the canonical catalog, default ahead
of DataHub. Metabase, DataHub, Lightdash, Rill and Grafana are optional App
Catalog apps, never the platform default.
This repository had already shipped both of the "rejected" tools, which is what
makes the old rows self-contradicting rather than merely stale:
* src/metadata/openmetadata-adapter.ts + tests/contracts/metadata-features.contract.test.ts
(STATUS.md classifies the OpenMetadata projection IMPLEMENTED)
* services/infra/docker-compose.data-stack.yml defines a `superset` service
(apache/superset:latest), and docs/compliance/OSS-INVENTORY.md line 74
already inventories `superset | Apache-2.0 | services/infra/data-stack`
Falsifiable check used (asserts the OUT table no longer rejects either tool and
that the IN table's BI row names Superset):
origin/main -> FAIL (4 findings)
this branch -> PASS
mutation: BI row back to Metabase -> FAIL
mutation: OUT rows un-struck -> FAIL
Docs only. No workflow files, no .gitignore, no source. `node scripts/siax-doctor.mjs`
gives 39/40 on this branch and 39/40 on main — the one failure (P1-15 CODEOWNERS
missing) is pre-existing and is what PR #19 supplies.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
|
364b7643fb |
Merge pull request 'fix(deps): update web-stack' (#26) from renovate/web-stack into main
Renovate web-stack bump: next 16.0.10->16.3.3, react/react-dom 19.2.0->19.2.8,
27 @radix-ui/* primitives, plus lockfile (943 package entries before and after;
only 4 new transitive names: @img/sharp-freebsd-wasm32, @img/sharp-webcontainers-wasm32,
@radix-ui/react-use-effect-event, @radix-ui/react-use-is-hydrated).
Diff read by hand: package.json changes are version strings only (hunks @@ -22,33
and @@ -58,12); scripts/engines/overrides untouched. No source files changed.
Both required contexts green on head
|
||
|
|
45a59ca3aa |
fix(deps): update web-stack
renovate/stability-days Updates have met minimum release age requirement
CI (SIAX Cloud) / security (push) Successful in 1m11s
Security Scan (SIAX Cloud) / security (push) Successful in 1m12s
CI (SIAX Cloud) / quality (push) Successful in 1m18s
CI (SIAX Cloud) / security (pull_request) Successful in 53s
CI (SIAX Cloud) / integration (push) Successful in 2m15s
CI (SIAX Cloud) / integration (pull_request) Successful in 1m2s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m4s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 43s
|
||
|
|
83209b0782 |
Merge pull request 'chore(deps): update dependency autoprefixer to v10.5.4' (#13) from renovate/autoprefixer-10.x-lockfile into main
Deploy to Coolify / deploy (push) Successful in 2s
CI (SIAX Cloud) / quality (push) Successful in 51s
CI (SIAX Cloud) / integration (push) Successful in 28s
CI (SIAX Cloud) / security (push) Successful in 22s
Security Scan (SIAX Cloud) / security (push) Successful in 22s
|
||
|
|
ae5841dfe9 |
chore(deps): update dependency autoprefixer to v10.5.4
renovate/stability-days Updates have met minimum release age requirement
CI (SIAX Cloud) / security (push) Successful in 58s
Security Scan (SIAX Cloud) / security (push) Successful in 1m0s
CI (SIAX Cloud) / quality (push) Successful in 1m3s
CI (SIAX Cloud) / security (pull_request) Successful in 36s
CI (SIAX Cloud) / integration (push) Successful in 1m39s
CI (SIAX Cloud) / integration (pull_request) Successful in 39s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 36s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m16s
|
||
|
|
4dbdff0636 |
Merge pull request 'P060-P068: durable ingestion plane — source classification gate, connector contract, common envelope, schema evolution, quality enforcement, queryable lineage, idempotent backfill' (#31) from feat/p060-p068-ingestion-plane into main
CI (SIAX Cloud) / integration (push) Successful in 32s
CI (SIAX Cloud) / quality (push) Successful in 47s
CI (SIAX Cloud) / security (push) Successful in 25s
Deploy to Coolify / deploy (push) Failing after 1m12s
Security Scan (SIAX Cloud) / security (push) Successful in 20s
|
||
|
|
1097231064 |
Merge pull request 'chore(estate): add first siax.repo.v3.json manifest (P003 v3 adoption, REPO-16 NA→PASS)' (#30) from chore/siax-repo-v3-manifest into main
Security Scan (SIAX Cloud) / security (push) Successful in 22s
CI (SIAX Cloud) / quality (push) Successful in 54s
CI (SIAX Cloud) / security (push) Successful in 27s
CI (SIAX Cloud) / integration (push) Successful in 34s
Deploy to Coolify / deploy (push) Failing after 1m6s
|
||
|
|
9267576b7c |
feat(data-os): P060-P068 durable ingestion plane — source classification gate, connector contract, common envelope, schema evolution, quality enforcement, queryable lineage, idempotent backfill
CI (SIAX Cloud) / integration (push) Successful in 42s
CI (SIAX Cloud) / quality (push) Successful in 1m17s
CI (SIAX Cloud) / security (push) Successful in 46s
Security Scan (SIAX Cloud) / security (push) Successful in 31s
CI (SIAX Cloud) / integration (pull_request) Successful in 36s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m4s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 27s
CI (SIAX Cloud) / security (pull_request) Successful in 24s
BEFORE_STATE (verified, with evidence):
* P060 PARTIAL/BROKEN. The registry persisted rights/provenance/health/
freshness to Postgres, but carried no authorization, no schedule, no
checkpoint, no schema and no owner distinct from created_by — and the
activation DoD was inverted: postgres-source-repository.ts wrote
`${input.status ?? "active"}`, so a source with no rights, no provenance
and no authorization was ACTIVE the instant it was registered. No
activation operation existed at all.
* P061 MISSING. services/api/sources/connectors.py had per-source fetch_*()
functions only — no lifecycle, no cursor, no resume, no health, no
conformance suite.
* P062 MISSING, already documented as such in
services/api/tests/test_known_gaps.py:10-19 (no CDC pipeline anywhere;
workers/pipeline.py has no persistence at all).
* P063 MISSING. schema_version existed as a free string; no comparison,
no policy, no versioned source schema.
* P064 PARTIAL. Rules/runs/issues/scores existed, but a failing rule only
opened an issue and the data flowed on. No quarantine anywhere.
* P065 DUPLICATE (confirmed). Two ER engines: src/er/entity-resolution-
service.ts (canonical) and services/api/workers/resolve.py (simplified,
still called by pipeline.py:110).
* P067 PARTIAL. Lineage was emitted to AUD0 but nothing here could ANSWER a
lineage question — no store, no query, no route.
* P068 MISSING, documented in test_known_gaps.py:44-52 ("no delivery-store
persistence layer exists to dedupe a backfill against").
BUILT (only what was PARTIAL/BROKEN/MISSING):
* P060 sql/007_ingestion_plane.sql adds authorization/schedule/schemas/
checkpoints/ledger/quarantine/lineage tables, owner_id, and flips the
opt_sources status DEFAULT to 'pending' so even a raw psql INSERT cannot
produce an unclassified-but-active source. SourceRegistryService gains
classifySource/activateSource/deactivateSource + authorization/schedule
accessors; registration NEVER activates (asking for status:"active" goes
through the gate, not around it); a service identity cannot authorize a
source. Routes: activate (GET classification / POST activate),
authorization, schedule.
* P061 src/connectors/connector.ts — all nine operations on one interface.
src/connectors/conformance.ts — a runnable suite (15 checks, no "skip"
verdict). HttpJsonConnector (real HTTP, opaque cursor) and
JsonlFileConnector (real file, line offsets, CDC deletes) both pass it.
The suite caught a real defect while being written: an API that drops
next_cursor at the end left the connector unable to resume, which is why
HttpJsonConnectorConfig has cursor_field.
* P062 src/types/ingestion.ts + src/ingestion/{envelope,ingestion-runner}.ts.
One envelope for batch/stream/cdc; durable per-(source,partition)
checkpoints; resume proven by killing a run inside the sink.
* P063 src/schema/{record-schema,compatibility,schema-registry}.ts. Two
enforcement points, both upstream of any consumer write: a discovery gate
before the first read, and per-record conformance before the page reaches
the sink. A producer cannot widen its own policy retroactively.
* P064 src/quality/quality-policy.ts composes the EXISTING ExpectationEngine
(not a second engine) and adds the missing consequence: block /
quarantine / warn, with quarantine durable and recoverable. No policy is
reported as "none_declared", never as a pass; an unexecutable expectation
counts as a violation, never a silent pass.
* P067 src/lineage/{lineage-edge,lineage-store}.ts + GET /api/0pt/lineage.
AUD0 stays the system of record; this is 0pt's own queryable projection.
* P068 opt_ingestion_ledger + IngestionRunner.backfill: bounded, idempotent
on repeat, leaves the live checkpoint alone, still re-emits real changes.
A BUG THIS PASS WROTE AND CAUGHT: the first IngestionRepository draft
combined the idempotency check and its recording into one atomic call
BEFORE the sink. The crash/resume test then delivered 7 of 10 records — the
page in flight when the sink died was already marked delivered, so the
resumed run skipped it as a duplicate. Permanent, silent data loss. Fixed by
splitting peekDeliveries/commitDeliveries and committing both the ledger and
the checkpoint only after the sink resolves.
NOT DONE, stated rather than deferred quietly:
* P065 is NOT met. resolve.py is still a second simplified production
engine. Closing it means rewriting it as an adapter over the canonical
service, which cannot be verified from this repo today: services/api is
not built by this repo's Dockerfile, api.0pt.app answers 404, and NO CI
workflow runs its pytest suite. The duplication is now stated at both
sites in code and the removal plan is in STATUS.md. The new ingestion
plane performs no ER, so it adds no third engine.
* P066 (Iceberg) untouched — ST0RE is owned by another workflow. Contract
read only.
* Delivery is AT-LEAST-ONCE, not exactly-once (no per-partition run lock).
* No CDC *source* exists; the contract is met, a CDC connector is not built.
* 0pt does not schedule itself — the schedule row is a declaration ACT0
would read.
VERIFIED: pnpm typecheck clean; pnpm test 365 passed / 25 skipped (was
311/18); pnpm build clean with all four new routes in the route table; lint
unchanged at 12 pre-existing errors. Real Postgres 16
(tests/p0/ingestion-plane-postgres.realinfra.test.ts, 7 passed) plus a real
`docker restart` of that container with an identical read-back through the
production code path — that is P060's "a restart preserves all sources",
measured.
DEPLOYMENT REALITY (curl, 2026-09-04): 0pt.app/api/health returns 200 from
Vercel with db:not_configured; 0pt.app/api/0pt/sources returns 404;
api.0pt.app/health returns 404; cd.yml still guards on a placeholder Coolify
app id. None of the capability layer runs in production. Nothing here is
LIVE_VERIFIED and STATUS.md says so.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
|
cbf0e1a0bf |
chore(estate): add first siax.repo.v3.json manifest (P003 v3 adoption)
CI (SIAX Cloud) / quality (push) Successful in 44s
CI (SIAX Cloud) / integration (push) Successful in 26s
CI (SIAX Cloud) / integration (pull_request) Successful in 27s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 17s
CI (SIAX Cloud) / security (push) Successful in 20s
Security Scan (SIAX Cloud) / security (push) Successful in 17s
CI (SIAX Cloud) / quality (pull_request) Successful in 43s
CI (SIAX Cloud) / security (pull_request) Successful in 19s
First siax.repo.v3.json in the SIAX platform estate (P049 certification found v3 adoption at 0/40, which structurally blocked 8 of the 11 required siax-doctor commands with score 0 / *-NOMANIFEST). Additive only: siax.repo.json (v2) is left untouched per CONTRIBUTING.md. No other file changed. Every field is filled from a real source (siax.repo.json v2, package.json, LICENSE, siax.config.json, docs/runbook/*, docs/slo.md, PRODUCTION_EVIDENCE.json, siax-standard SIAX_ESTATE.yaml + SIAX_TAXONOMY.yaml, cl0ud packages/capability-registry/src/registry.ts, and a live probe of 0pt.app on 2026-09-04). Where the truth is unknown or unproven the honest schema option is used, never a value that merely looks complete: - productionEvidence.verified=false: no SHA->CI->artifact->SBOM->signature-> deployment->running-digest chain exists (PRODUCTION_EVIDENCE.json is a CI verification report, not that chain). - identity.workloadIdentity=false: outbound platform calls use an optional static ST0RE_SERVICE_TOKEN env bearer (src/clients/st0re-client.ts); cl0ud service-auth.ts registers no principal for 0pt. - consumedBy=[]: SIAX_ESTATE.yaml row says consumers UNKNOWN-not-verified; the 4 reverse-lookup candidates are labelled "inferred" there. - reliability.backupFrequency=none / provenRestore=false: production has no database (live /api/health db=not_configured); the P014 drill ran against throwaway containers, not a production backup. - licenses.approved=false: no siax.license-assessment record exists. - operationalState=degraded: live 0pt.app answers liveness 200 but db=not_configured and /api/ready is 404 (deployed build predates main). - node=null / deployment.node=null: live on Vercel, not a fleet node. - observability.metrics=false / otel=false: STATUS.md "no metrics/traces emitted"; /api/admin/metrics is business aggregates, not observability. - capabilities: exactly the 19 ids cl0ud's registry lists with owner '0pt' (5 prospekt.* + 14 0pt.*). NOTE: the repo contains zero "prospekt" code and ships no CLI/MCP surface despite the registry claiming those surfaces for prospekt.* -- flagged in the PR, not silently dropped. - surfaces=[api, ui]: only what actually ships (Next.js route handlers + FastAPI, Next.js pages). No bin, no MCP tools file, no SDK package (packages/schemas is JSON Schema files consumed by path), job execution is delegated to N0D per the registry. siax-doctor (siax-standard main, @siax/schemas): repo REPO-16 NA -> PASS (measuredScore 85 -> 85; REPO-16 is NA-safe) capabilities 0 / CAP-NOMANIFEST -> 100 (CAP-DUP-OWNER PASS, 19 ids) dependencies 0 / DEP-NOMANIFEST -> 100 (DEP-CYCLE/DEP-REVERSE/DEP-ASYM PASS) surfaces 0 / SURF-NODATA -> 0 (needs siax/capabilities/*.json records) engines 0 / ENG-NOMANIFEST -> 0 (ENG-DIRECT-COUPLING NA for core-platform; ENG-REGISTRY UNVERIFIED) licenses 0 / LIC-NOMANIFEST -> 0 (LIC-UNASSESSED UNVERIFIED -- honestly no approved=true claim) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
|
c5ebceff38 |
Merge pull request 'P014: close tenant-isolation + idempotency gaps, execute real restore drill' (#29) from feat/p014-data-os-hardening into main
Deploy to Coolify / deploy (push) Successful in 2s
CI (SIAX Cloud) / integration (push) Successful in 35s
CI (SIAX Cloud) / quality (push) Successful in 48s
Security Scan (SIAX Cloud) / security (push) Successful in 28s
CI (SIAX Cloud) / security (push) Successful in 32s
|
||
|
|
dc6aaa0e10 |
fix(data-os): P014 — close tenant-isolation + idempotency gaps, execute real restore drill
CI (SIAX Cloud) / quality (push) Successful in 51s
Security Scan (SIAX Cloud) / security (push) Successful in 26s
CI (SIAX Cloud) / security (push) Successful in 28s
CI (SIAX Cloud) / integration (push) Successful in 1m1s
CI (SIAX Cloud) / integration (pull_request) Successful in 30s
CI (SIAX Cloud) / security (pull_request) Successful in 24s
CI (SIAX Cloud) / quality (pull_request) Successful in 48s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 22s
Tenant isolation (security-critical): every app/api/0pt/** route reading
tenant_id from client-controlled query/body never cross-checked it against
the token middleware.ts already verified (x-0pt-verified-tenant). Adds a
shared assertTenantMatchesVerified guard and wires it into all 20+ affected
routes. Also closes two full-leak-by-omission bugs found in this pass
(documents/capture and lakehouse/mappings GETs returned every tenant's rows
when tenant_id was simply omitted) and two "no tenant check at all" reads
(rate-policies/[id], lakehouse/mappings/[id]) reachable by ID alone.
Duplicate-delivery idempotency (correctness-critical): POST /api/orders now
honors Idempotency-Key (sql/006, replay-or-409-on-conflict, Postgres +
in-memory); the Python orchestrator's POST /orders/{id}/run — the actual
delivery trigger — is now idempotent under an asyncio lock, closing a real
concurrent-request race, not just the sequential-retry case.
Engine-duplication resolution: adaptive_router.py's docstring wrongly
invited a bypass ("see b0ts for scraping") contradicting the product's own
compliance stance; corrected, and given real (additive, not escalating)
block-signal tracking so a rate-limited/blocked source is excluded from the
plan instead of hammered. Also fixes a previously-undiscovered production
bug: the Python engine's only call into the TS quality bridge
(X-Internal-Token) was unconditionally 401'd by middleware.ts, which never
checked that credential — the advisory quality bridge was unreachable in any
environment enforcing auth.
Restore drill: executed for real against throwaway Docker Postgres
containers (not simulated). Found and fixed two real blockers along the way:
sql/004 referenced opt_datasets, which no migration ever created (chain was
never actually runnable end to end); lib/db.ts hardcoded ssl:"require",
making any non-TLS Postgres (incl. compose.dev.yaml's own local-dev
container) unreachable. After both fixes: full migration chain applies
cleanly, pg_dump/pg_restore round-trip is byte-identical (row counts +
content checksums), and real application code (lib/orders-store.ts,
SourceRegistryService's Postgres repository) reads the restored data back
correctly, including cross-tenant isolation still holding.
82 new tests (55 TS + 27 Python), all passing; full existing suite (291 TS +
55 Python) still green; typecheck clean on every changed/added file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
||
|
|
40c3e2fbee |
Merge pull request 'feat(data-os): P010 completion — Source Registry persistence + TS/Python quality bridge + provenance + fault-injection tests' (#28) from feat/p010-data-os-completion into main
Deploy to Coolify / deploy (push) Successful in 2s
CI (SIAX Cloud) / security (push) Successful in 20s
Security Scan (SIAX Cloud) / security (push) Successful in 20s
CI (SIAX Cloud) / integration (push) Successful in 25s
CI (SIAX Cloud) / quality (push) Successful in 46s
|
||
|
|
e7516697b3 |
docs(p010): implementation report + production evidence for Data-OS completion
CI (SIAX Cloud) / security (push) Successful in 23s
CI (SIAX Cloud) / integration (push) Successful in 30s
Security Scan (SIAX Cloud) / security (push) Successful in 30s
CI (SIAX Cloud) / quality (push) Successful in 53s
CI (SIAX Cloud) / security (pull_request) Successful in 20s
CI (SIAX Cloud) / quality (pull_request) Successful in 51s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 22s
CI (SIAX Cloud) / integration (pull_request) Successful in 32s
From-scratch verification (fresh clone, fresh installs) backing up the P010 completion pass: 256/256 vitest, 28/28 pytest, clean typecheck/build on the committed tree, pre-existing-only lint noise. Documents that OPT_QUALITY_BRIDGE_ENABLED ships off and stays off in this PR -- activating it in production is left as Simon's decision, with a readiness checklist. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
9740e20818 |
feat(data-os): TS/Python quality bridge + provenance fields + fault-injection tests
CI (SIAX Cloud) / security (push) Successful in 23s
Security Scan (SIAX Cloud) / security (push) Successful in 22s
CI (SIAX Cloud) / integration (push) Successful in 29s
CI (SIAX Cloud) / quality (push) Successful in 45s
CI (SIAX Cloud) / security (pull_request) Successful in 27s
CI (SIAX Cloud) / integration (pull_request) Successful in 31s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 29s
CI (SIAX Cloud) / quality (pull_request) Successful in 48s
Continues P010 (branch renamed from feat/p010-source-registry-persistence):
closes the three remaining BEFORE_STATE.md gaps (§2 TS/Python split-brain,
§3 missing provenance fields, §4 untested failure modes), all additive to
the production pipeline — the Python QA filter's delivery decision is
unchanged by any of this.
TS/Python bridge (§2):
- New route surface app/api/0pt/quality/* (rules/runs/issues CRUD +
POST /api/0pt/quality/bridge-run) over the shared QualityService
singleton (src/services/quality.ts), fixing the same "no shared
instance" defect the Source Registry had.
- src/quality/quality-bridge.ts auto-provisions a deterministic,
tenant-scoped baseline rule set on first use so the Python engine never
pre-registers rules through the CRUD surface.
- services/api/workers/quality_bridge.py: the Python-side HTTP client.
Feature-flagged off by default (OPT_QUALITY_BRIDGE_ENABLED), fails open
on any error (never raises into run_job, never changes `delivered`).
Advisory only — real per-row gating against the TS engine isn't
implementable yet (ExpectationEngine returns dataset-level pass/fail
counts + samples, not a per-row boolean array); documented, not faked.
- Live-verified end-to-end: `next dev` (AUTH_ENFORCE=false) + the real
Python pipeline hitting the real brreg.no API, round-tripping through
the real bridge route, receiving a real QualityService score.
Provenance (§3): sources/connectors.py's Provenance dataclass gains
retrieved_at, transform, connector_version, producer, quality,
entity_resolution, artifact_ref (all previously absent from the real
dataflow entirely) via a make_provenance() factory used by every connector.
resolve.py stamps entity_resolution on merged records' provenance (never on
unmerged ones). pipeline.py stamps artifact_ref at delivery and quality
once the bridge succeeds.
Failure tests (§4), real fault injection against real code (services/api,
new pytest + respx suite, not vitest mocks):
- dedup-determinism + duplicate-input + out-of-order: resolve.py under
shuffled/interleaved/exhaustive-permutation input, plus an explicit,
verified test of the one case that IS legitimately order-sensitive
(non-transitive similarity in a greedy clusterer) instead of hiding it.
- schema-evolution: respx-mocked upstream responses with added/removed/
alien-shaped fields through the real brreg connector + pipeline.
- restart: connector exceptions mid-job (including one from respx itself)
never crash run_job.
- backfill: the real property that holds given no persistence layer
exists (replay is compute-idempotent) — NOT a fabricated dedup-store
test.
- CDC-interruption and true backfill/delivery-store idempotency:
explicitly documented as not implementable against this codebase
(no CDC pipeline, no delivery store exist anywhere) via skip markers
with reasons, in tests/test_known_gaps.py, rather than silently
dropped or faked.
Testing: 256/256 vitest (249 pre-existing + 7 new bridge route/service
tests) and 28/28 pytest (2 documented skips) green locally. tsc -p
tsconfig.check.json has zero new errors (14 pre-existing errors, all in
the untracked/unaudited src/fleet, src/graph, src/metrics, src/projections
paths this pass does not touch, per BEFORE_STATE.md).
No PR opened, per instructions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
||
|
|
07b13bd833 |
feat(data-os): Source Registry Postgres persistence + full HTTP route surface
Replaces the in-memory (per-instance, not even per-process-shared) Maps in SourceService/SourceRegistryService with real Postgres persistence, and adds the register/query/describe/health/rights/provenance/freshness HTTP routes that never existed (grep for SourceRegistryService in app/ previously returned nothing). - sql/002_source_registry.sql: opt_sources, opt_source_rights, opt_source_provenance, opt_source_health_checks, opt_source_freshness. Numbered 002 (not 006) because 004's opt_source_rate_policies already forward-references opt_sources, which no sql/ file defined until now — fixes 004's pre-existing broken migration ordering (opt_datasets remains a separate, unrelated, still-open gap in 004, flagged not fixed). - src/repositories/*: SourceRepository / SourceRegistryRepository seams, each with an in-memory implementation (business logic extracted verbatim) and a Postgres implementation, chosen via resolveStoreMode — production still fails hard with no DATABASE_URL (Master-DoD §3), but a configured production now actually works instead of always being blocked. - SourceService / SourceRegistryService: storage moved behind the repository seam; getHealth/getFreshness are now async and tenant-scoped at the accessor itself (needed once they're reachable directly over HTTP, not only after an earlier caller's tenant check). - src/services/source-registry.ts: shared singleton instances for the sources/* route group, fixing the "each route module owned its own private registry" defect (source-rate-policies and .../query previously couldn't see a source registered via POST /api/0pt/sources). - New routes: POST/GET /api/0pt/sources, GET /api/0pt/sources/[id], GET+PUT .../rights, GET+PUT .../provenance, GET+POST .../health, GET .../freshness. - Business logic unchanged: fail-closed purpose gate, cross-tenant NOT_FOUND (no existence oracle, §50), bounded 100-row health history. Tested against a real Postgres container (not mocked) — tests/p0/source-registry-postgres.realinfra.test.ts, env-gated on SOURCE_REGISTRY_TEST_DATABASE_URL, skips honestly when unset. All 249 pre-existing P0 tests + 7 new realinfra tests pass; also smoke-tested the full HTTP route surface end-to-end against next dev + live Postgres. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
f8bb0d5ab9 |
Merge pull request 'feat: SIAX Platforms P0 wave 2 — real-contract platform clients + Data-OS capability layer' (#25) from feat/siax-platforms-p0-data-os into main
Deploy to Coolify / deploy (push) Successful in 3s
CI (SIAX Cloud) / integration (push) Successful in 25s
CI (SIAX Cloud) / security (push) Successful in 18s
CI (SIAX Cloud) / quality (push) Successful in 48s
Security Scan (SIAX Cloud) / security (push) Successful in 19s
|
||
|
|
62f7b0049f |
fix(ci): pin OpenSearch service container workdir — act_runner clobbers WORKDIR-relative ENTRYPOINT
CI (SIAX Cloud) / integration (push) Successful in 22s
CI (SIAX Cloud) / security (push) Successful in 20s
Security Scan (SIAX Cloud) / security (push) Successful in 17s
CI (SIAX Cloud) / quality (push) Successful in 47s
CI (SIAX Cloud) / security (pull_request) Successful in 20s
CI (SIAX Cloud) / integration (pull_request) Successful in 31s
CI (SIAX Cloud) / quality (pull_request) Successful in 45s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 22s
PR #25 CI run 7745 job 'integration' failed before any test ran: 'exec: "./opensearch-docker-entrypoint.sh": stat ...: no such file or directory'. Root cause (confirmed against the job log's merged container.Config, and reproduced locally with plain docker run -w): this runner's act_runner merges the JOB's WorkingDir (/workspace/sax3l/0pt) into every SERVICE container, clobbering the OpenSearch image's own WORKDIR (/usr/share/opensearch) that its ENTRYPOINT ./opensearch-docker-entrypoint.sh is relative to. ClickHouse's entrypoint is an absolute path so it started fine under the same defect -- a green ClickHouse service is not evidence this class of bug is absent elsewhere. Fix: --workdir /usr/share/opensearch in the opt-ci-opensearch service's options, restoring the image's own working directory. Verified locally: 'docker run -w /workspace/sax3l/0pt opensearchproject/opensearch:2.11.1' reproduces the exact failure; '-w /usr/share/opensearch' fixes it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
c47e3b468e |
test(integration)+ci+docs: live OpenSearch/ClickHouse suites, CI service containers, STATUS.md (§56)
CI (SIAX Cloud) / integration (push) Failing after 22s
CI (SIAX Cloud) / security (push) Successful in 22s
Security Scan (SIAX Cloud) / security (push) Successful in 21s
CI (SIAX Cloud) / quality (push) Successful in 43s
CI (SIAX Cloud) / integration (pull_request) Failing after 3s
CI (SIAX Cloud) / security (pull_request) Successful in 17s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 16s
CI (SIAX Cloud) / quality (pull_request) Successful in 43s
- tests/integration/: 10 tests against REAL OpenSearch 2.11 (tenant isolation, purpose/rights filters, generation rebuild/rollback, projection-disaster rebuild §68) + REAL ClickHouse 24.8 (tenant-isolated metric DSL, TTL schema, injection-surface refusals). Honest skip without env - and CI fails hard if the suites skip instead of running. - .gitea/workflows/ci.yml: new blocking 'integration' job with self-identifying service containers (st0re ci.yml's measured alias-collision discipline: run-unique cluster.name / credentials, getent-enumerated IP selection, no-match = hard fail). - STATUS.md: full 8-level Master-DoD classification per capability incl. the honest gaps blocking PRODUCTION_READY. - dependency-mode: NEXT_PHASE=phase-production-build is not a serving runtime (next build page-data collection must compile; guard re-arms at boot) - pinned by contract test. - docs/02-architecture.md: superseded the accurate-but-now-fixed 'every client runs against its mock' paragraph; .env.example: client URLs now actually read from env + new P0 backend vars. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
9f0de72456 | chore(deps): update dependency @tailwindcss/postcss to v4.3.3 (#12) | ||
|
|
6be9992cad | chore(deps): update dev-dependencies (non-major) (#14) | ||
|
|
b46807da75 |
feat(data-os): P0 capability layer — source registry, search, quality, dataset versions, metadata, analytics, features, ER
- Source Registry (§10): rights/license/purpose metadata, provenance,
freshness SLA, health checks, describe/query; purpose gate FAILS CLOSED
(no rights record => deny); cross-tenant probes => NOT_FOUND.
- Search (§10/§42): OpenSearch REST adapter (no SDK) + SearchService with
MANDATORY tenant + purpose/rights filtering, versioned index definitions,
bulk, generation-per-rebuild (opt-{name}--g{n}) with atomic alias cutover,
rollback, failed-rebuild generation discard. OpenSearch never canonical.
- Quality (§10): versioned rule registry (GE-JSON interchange), local
engine for the GE core subset (never silently passes unsupported types),
runs/results/issues/score/trend, whole-run delegation to the GE backend
via N0D (0pt.quality.ge_run) - never fabricated results.
- Dataset (§10/§5/§53): immutable numbered versions holding canonical
st0re:artifact refs only, idempotent createVersion, export registers the
ST0RE artifact + AUD0 evidence + OpenLineage.
- Metadata (§10): OpenMetadata PUT /api/v1/tables projection adapter +
projectAll rebuild path; projection never canonical.
- Analytics (§10): ClickHouse HTTP adapter, closed metric-query DSL with
FORCED tenant predicate + readonly/max_execution_time/max_result_rows,
TTL retention. Zero connection to b0ts/quarantined pipelines.
- Features (§10): versioned feature-view registry, real Feast
get-online-features adapter, materialize/get-historical as N0D delegation.
- ER (§10): generic candidate API with deterministic blocking + transparent
scoring + per-field explanations, product hard-rule hook (force both
directions), review-band handoff, Splink batch via N0D contract.
93 new unit/contract tests; suite 248 passed / 1 skipped.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
|
da58176735 |
test(contracts): per-client wire-contract suites + §3 production-guard tests
38 tests: recording provider double answers with the sibling services' real response shapes; each suite asserts the exact request wire shape (path, method, tenant/actor/idempotency headers, camelCase vs snake_case body) so a regression to the pre-rewrite invented shapes fails loudly. Includes the fail-closed policy cases (missing decision => deny, CL0UD down => typed error, never implicit allow) and the production factory guard for all four clients. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
b7038d7675 |
feat(platform): canonical SIAX context/errors + real-contract clients, no prod mock fallback
- src/platform/: SiaxRequestContext + propagation headers + 12 canonical
error codes (documented mirror of cl0ud packages/contract), SiaxError,
resource refs (st0re:artifact:...), and the §3/§10 dependency-mode guard:
production runtime + unconfigured dependency/store => hard
DEPENDENCY_UNAVAILABLE at construction, never silent mock/in-memory.
- src/clients/: all four platform clients rewritten against the REAL wire
contracts verified on sibling main branches this session:
* cl0ud: POST /api/policy/evaluate ({data} envelope, allow+requiresApproval)
- the old /v1/cl0ud/policies/evaluate path never existed on CL0UD
* st0re: camelCase ArtifactInput on /v1/st0re/artifacts/register incl.
X-Tenant-Id/X-Actor-Id + X-Idempotency-Key - the old snake_case body
would have been a 400 against the live registry
* n0d: required_role + resource_requirements + body idempotency_key +
CloudEvents callback_url per n0d createJobSchema; adds getJob/getJobResult
* aud0: /v1/aud0/openlineage is AUD0's only lineage ingest -
/v1/aud0/lineage/events never existed; legacy shape converts to a
spec-shaped OpenLineage RunEvent (deterministic runId dedup)
- src/lineage/: OpenLineage subset types + LineageEmitter (never-throw,
§16 evidence semantics)
- shared SiaxHttp transport: timeouts, typed SiaxError mapping, context
propagation, no raw upstream bodies in errors
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
|
e452848470 | Merge pull request 'fix: COOLIFY_APP_ID from variables (0pt app live in Coolify)' (#24) from fix/coolify-app-id into main | ||
|
|
18ea0ecc41 |
fix: wire COOLIFY_APP_ID from repo variable (0pt app provisioned via API 2026-08-31)
CI (SIAX Cloud) / security (push) Successful in 46s
CI (SIAX Cloud) / security (pull_request) Successful in 49s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 48s
Security Scan (SIAX Cloud) / security (push) Successful in 21s
CI (SIAX Cloud) / quality (pull_request) Successful in 1m15s
CI (SIAX Cloud) / quality (push) Successful in 1m13s
|
||
|
|
fedaab41ba | fix: auth middleware matcher covers /api/0pt exact path + diag logging for S1 root-cause | ||
|
|
00a1b3997d |
fix(env): NEXT_PUBLIC_API_URL -> live origin (api.0pt.app is dead) (#23)
Co-authored-by: sax3l <simon@siax.io> Co-committed-by: sax3l <simon@siax.io> |
||
|
|
7b1814fef4 |
feat(compliance): scaffold Golden Standard artifacts (CI-14, DX-02, DX-03)
CI (SIAX Cloud) / security (push) Successful in 19s
Security Scan (SIAX Cloud) / security (push) Successful in 17s
CI (SIAX Cloud) / quality (push) Successful in 40s
CI (SIAX Cloud) / security (pull_request) Successful in 24s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 24s
CI (SIAX Cloud) / quality (pull_request) Successful in 48s
Applies siax-standard/packages/scaffold (feat/scaffold-generator) to close
three BLOCK-level gaps found by the doctor:
- .gitea/CODEOWNERS (CI-14) — was completely missing
- compose.dev.yaml (DX-02) — was completely missing, left as `services: {}`
since the scaffold cannot know this repo's real local dependencies
- .devcontainer/devcontainer.json (DX-03) — was completely missing
siax.repo.json, the four docs/runbook/*.md files, .gitea/workflows/security.yml,
prompts/000-INDEX.md and contracts/openapi.yaml already existed with content
and were left untouched (AR-1: the generator never overwrites existing content).
docs/compliance/SCAFFOLD-TODO.md was generated and then manually corrected
before this commit — the generator emits some fixed TODO/warning lines
regardless of actual repo state, and three of them were false for 0pt:
- claimed appId is null needing an appIdJustification field removed; the
EXISTING siax.repo.json already has appId: "0PT" set, no such field exists
- claimed domains: 0 ("none found"); the EXISTING siax.repo.json already
lists 2 domains (0pt.app, api.0pt.app) as strings — the new schema wants
objects, so they fail validation and get silently dropped to 0, which
reads as "no domains configured" when in fact 0pt.app is a real domain
- asked to fill in both consumes[] and consumedBy[] "from scratch"; consumes[]
is already fully populated in the existing file (CL0UD, ST0RE, N0D, AUD0,
INF0) — only consumedBy[] is actually missing
Doctor measuredScore: 74 -> 85 (CI-14, DX-02, DX-03 flip FAIL -> PASS).
REPO-04 stays FAIL: siax.repo.json still needs a human to add the missing
required fields and convert domains to the new object shape (AR-1 will not
touch a file that already has content) — tracked in SCAFFOLD-TODO.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
||
|
|
77e5137c9a | Merge pull request 'docs+chore: repo hygiene -- v2.0 manifests, ADRs, changelog, doctor, guides' (#20) from docs/repo-hygiene-v2-standard into main | ||
|
|
5f98379b80 | Merge pull request 'fix(health): report real per-dependency status instead of hardcoded ok:true' (#18) from fix/honest-health-check into main |