Files
0pt/tests/duckdb/duckdb-provider.live.test.ts
T
Claude Opus 5 3ee023017b
CI (SIAX Cloud) / security (push) Successful in 28s
CI (SIAX Cloud) / integration (push) Successful in 42s
Security Scan (SIAX Cloud) / security (push) Successful in 21s
CI (SIAX Cloud) / quality (push) Successful in 1m2s
CI (SIAX Cloud) / security (pull_request) Successful in 23s
CI (SIAX Cloud) / integration (pull_request) Successful in 40s
CI (SIAX Cloud) / quality (pull_request) Successful in 58s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 20s
fix(analytics): move the DuckDB live suite out of tests/integration, and add the BI docs
CI's integration job failed, and it failed CORRECTLY. That job provides a live
OpenSearch and a live ClickHouse and then asserts that nothing in
tests/integration/ skipped, on the stated principle that "a skipped suite must
never masquerade as a green integration gate". The DuckDB suite needs a Python
interpreter that job does not provide, so it skipped, and the guard fired — the
first suite in that directory ever to skip there.

Moving the file to tests/duckdb/ respects the guard rather than weakening it.
Loosening it to tolerate skips would have traded a real gate for one green job,
which is the trade this estate keeps regretting. The suite still runs, still
fails rather than skipping when DUCKDB_PYTHON is set and DuckDB is broken, and
is still pinned by the committed evidence.

Also fixed: "does NOT ship the Iceberg table" asserted against the FILESYSTEM
when the claim is about what is COMMITTED. The live suite rebuilds that table
locally, so the assertion failed whenever both suites ran in one pass — it was
testing the wrong thing rather than catching a real problem. It now asserts
against `git ls-files`, and also asserts the parquet/csv fixtures ARE tracked,
so a version that stopped shipping any fixture would fail too.

docs/analytics/BI_QUERY_METRIC_QUALITY.md is organised around what is PROVEN
and what is NOT, because the second list is the one that gets dropped. It
records the five findings and the six open items, including that BI-P029's
MetricFlow was never executed and that BI-P021's budget is a bound rather than
a hard cap (measured 130x overshoot at a 100ms poll).

It also reports, without changing it, a latent flaw in .gitea/workflows/ci.yml
(a protected file this branch does not touch): the integration guard's second
pattern `Tests  [0-9]+ passed` never matches, because vitest writes that
summary with ANSI colour codes between the words. The guard is therefore
STRICTER than it reads — "any skip fails" rather than "skipped and nothing
passed" — so nothing is currently unguarded, but the escape hatch does not
work and someone will eventually be puzzled by it.

957 tests pass with ClickHouse, Postgres and DuckDB all live.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 03:08:11 +02:00

397 lines
17 KiB
TypeScript

/**
* tests/duckdb/duckdb-provider.live.test.ts
* ==========================================
* BI-P014 — the DuckDB provider against a real DuckDB.
*
* Two things are proven here that a unit test cannot:
*
* 1. BI-P014's DoD — "Queries work against Parquet / Arrow / Iceberg snapshot
* fixtures" — for the formats this environment can actually build. Parquet
* and CSV run from committed fixtures. The Iceberg case runs when a Python
* interpreter carrying `pyiceberg` is pointed at by PYICEBERG_PYTHON,
* because pyiceberg embeds ABSOLUTE paths in its manifests, so a committed
* Iceberg table would only work from the machine that made it. The recorded
* result of a real Iceberg run is committed as evidence instead, and
* tests/analytics/duckdb-provider.test.ts pins it.
*
* 2. The security claim, WITH LAYER 1 REMOVED. Every bypass attempt is sent
* straight to DuckDB, skipping the mount registry entirely, and layer 2 —
* the session sandbox — must refuse it. Which matters more than usual here,
* because layer 2 does not work the way it looks like it works: see the
* `allowed_directories` finding in duckdb-provider.ts.
*
* Gated on DUCKDB_PYTHON, an interpreter with the `duckdb` package. With it
* set and DuckDB broken, this suite FAILS; it does not degrade to a skip.
*
* ── WHY THIS IS NOT UNDER tests/integration/ ──────────────────────────────
* It was, and CI's integration job failed — correctly. That job provides a
* live OpenSearch and a live ClickHouse and then asserts that NOTHING in
* tests/integration/ skipped, on the stated principle that "a skipped suite
* must never masquerade as a green integration gate". A DuckDB suite needing
* a Python interpreter that job does not provide is not one of the suites that
* job is a gate for, so it skipped, and the guard fired.
*
* Moving the file respects that guard rather than weakening it. Loosening the
* guard to tolerate skips would have traded a real gate for one green job —
* which is the trade this estate keeps regretting.
*/
import { execFileSync } from 'node:child_process';
import fs from 'node:fs';
import path from 'node:path';
import { beforeAll, describe, expect, it } from 'vitest';
import { SiaxError } from '@/src/platform/errors';
import {
ArtifactMountRegistry,
DuckDbAnalyticsProvider,
duckDbSessionBootstrap,
type DuckDbRunner,
} from '@/src/analytics/providers/duckdb-provider';
const PY = process.env.DUCKDB_PYTHON ?? '';
/** An interpreter carrying pyiceberg, used to REBUILD the non-portable fixture. */
const PY_ICEBERG = process.env.PYICEBERG_PYTHON ?? PY;
const suite = PY ? describe : describe.skip;
const FIXTURES = path.join(process.cwd(), 'tests/fixtures/duckdb');
const EVIDENCE_PATH = path.join(process.cwd(), 'evidence/bi-p014/duckdb-sandbox.json');
const IDENTITY = { tenant_id: 't_alpha', principal_id: 'u_1' };
/**
* Runs statements through a real DuckDB. The bootstrap is applied FIRST and in
* order, exactly as the provider emits it.
*/
function runDuck(
bootstrap: readonly string[],
sql: string,
loadIceberg = false,
): { ok: boolean; out: string } {
const script = [
'import duckdb, json, sys',
'c = duckdb.connect()',
...(loadIceberg ? ['c.execute("INSTALL iceberg; LOAD iceberg")'] : []),
// The bootstrap is inside the try as well: a statement that tries to
// RE-OPEN a locked sandbox throws, and that throw is a RESULT to report,
// not a crash of the harness. The first version of this file let it kill
// the subprocess, which turned a meaningful refusal into an unreadable
// "Command failed".
'try:',
...bootstrap.map((s) => ` c.execute(${JSON.stringify(s)})`),
` rows = c.execute(${JSON.stringify(sql)}).fetchall()`,
' print(json.dumps({"ok": True, "rows": [[str(v) for v in r] for r in rows]}))',
'except Exception as e:',
' print(json.dumps({"ok": False, "error": str(e)[:300]}))',
].join('\n');
const out = execFileSync(PY, ['-c', script], { encoding: 'utf8', timeout: 60_000 });
const parsed = JSON.parse(out.trim()) as { ok: boolean; rows?: string[][]; error?: string };
return { ok: parsed.ok, out: JSON.stringify(parsed.rows ?? parsed.error) };
}
const runner: DuckDbRunner = {
engine: 'duckdb',
async run({ bootstrap, sql }) {
const result = runDuck(bootstrap, sql, sql.includes('iceberg_scan'));
if (!result.ok)
throw new SiaxError('PERMANENT_FAILURE', 'duckdb', `query failed: ${result.out.slice(0, 160)}`);
return (JSON.parse(result.out) as string[][]).map((r, i) => ({ _row: i, values: r }));
},
};
const evidence: Record<string, unknown> = {};
let registry: ArtifactMountRegistry;
let provider: DuckDbAnalyticsProvider;
beforeAll(() => {
if (!PY) return;
const version = execFileSync(PY, ['-c', 'import duckdb; print(duckdb.__version__)'], {
encoding: 'utf8',
}).trim();
evidence.duckdb_version = version;
evidence.recorded_at = new Date().toISOString();
expect(fs.existsSync(path.join(FIXTURES, 'sales.parquet')), 'parquet fixture is missing').toBe(true);
// Rebuild the Iceberg fixture, which is deliberately not committed:
// pyiceberg writes ABSOLUTE paths into its metadata and Avro manifests, so a
// committed table resolves only on the machine that made it.
if (PY_ICEBERG && !fs.existsSync(path.join(FIXTURES, 'iceberg_metadata_location.txt'))) {
const build = [
'import os, shutil, pyarrow as pa',
'from pyiceberg.catalog.sql import SqlCatalog',
`root = ${JSON.stringify(FIXTURES)}`,
"wh = os.path.join(root, 'iceberg')",
'shutil.rmtree(wh, ignore_errors=True); os.makedirs(wh)',
"cat = SqlCatalog('local', **{'uri': f'sqlite:///{wh}/catalog.db', 'warehouse': f'file://{wh}'})",
"cat.create_namespace_if_not_exists('bi')",
"d = pa.table({'tenant_id': ['t_alpha']*3, 'country': ['SE','NO','SE'], 'amount': [10.0,20.0,30.0]})",
"t = cat.create_table('bi.sales_iceberg', schema=d.schema); t.append(d)",
"open(os.path.join(root,'iceberg_metadata_location.txt'),'w').write(t.metadata_location)",
"print('built')",
].join('\n');
try {
execFileSync(PY_ICEBERG, ['-c', build], { encoding: 'utf8', timeout: 120_000 });
} catch {
// pyiceberg absent. The Iceberg case then falls back to the committed
// evidence, and it asserts against it rather than passing vacuously.
}
}
registry = new ArtifactMountRegistry(FIXTURES);
registry.register({
artifact_id: 'a_sales_parquet',
tenant_id: 't_alpha',
st0re_ref: 'st0re://t_alpha/sales.parquet',
relative_path: 'sales.parquet',
format: 'parquet',
size_bytes: fs.statSync(path.join(FIXTURES, 'sales.parquet')).size,
});
registry.register({
artifact_id: 'a_customers_csv',
tenant_id: 't_alpha',
st0re_ref: 'st0re://t_alpha/customers.csv',
relative_path: 'customers.csv',
format: 'csv',
size_bytes: fs.statSync(path.join(FIXTURES, 'customers.csv')).size,
});
// Another tenant's artifact, so the cross-tenant refusal is a real refusal
// and not "the id does not exist".
registry.register({
artifact_id: 'a_beta_parquet',
tenant_id: 't_beta',
st0re_ref: 'st0re://t_beta/sales.parquet',
relative_path: 'sales.parquet',
format: 'parquet',
size_bytes: 1,
});
provider = new DuckDbAnalyticsProvider(registry, runner);
}, 60_000);
suite('BI-P014 DoD — queries work against the fixtures', () => {
it('aggregates a Parquet fixture through the closed request', async () => {
const rows = await provider.query(IDENTITY, {
artifact_id: 'a_sales_parquet',
group_by: ['country'],
aggregate: { fn: 'sum', column: 'amount', alias: 'total' },
});
const totals = Object.fromEntries(
rows.map((r) => {
const [country, total] = (r as { values: string[] }).values;
return [country, Number(total)];
}),
);
expect(totals).toEqual({ SE: 394.5, NO: 105.5 });
evidence.parquet = totals;
});
it('reads a CSV fixture, including a value containing a quote', async () => {
const rows = await provider.query(IDENTITY, {
artifact_id: 'a_customers_csv',
select: ['customer_id', 'name'],
});
const names = rows.map((r) => (r as { values: string[] }).values[1]);
expect(names).toContain("O'Brien Ltd");
evidence.csv = { rows: rows.length, includes_quoted_value: true };
});
it('filters with a hostile value without it becoming SQL', async () => {
const rows = await provider.query(IDENTITY, {
artifact_id: 'a_customers_csv',
select: ['customer_id'],
filters: { name: "O'Brien Ltd" },
});
expect(rows).toHaveLength(1);
// The injection attempt: a value that would close the literal and append.
const hostile = await provider.query(IDENTITY, {
artifact_id: 'a_customers_csv',
select: ['customer_id'],
filters: { name: "x' OR '1'='1" },
});
// It matched nothing, because it was DATA. Had the escaping failed, the
// OR would have matched all three rows.
expect(hostile).toHaveLength(0);
evidence.injection_attempt = { matched_rows: hostile.length, note: 'escaped as data; 0 rows, not 3' };
});
it('reads an ICEBERG SNAPSHOT by its exact metadata file', async () => {
const metaFile = path.join(FIXTURES, 'iceberg_metadata_location.txt');
if (!fs.existsSync(metaFile)) {
// No pyiceberg here. Carry the previously RECORDED result forward, marked
// as carried forward, and assert against it — the first version of this
// branch was `expect(evidence.iceberg ?? "unbuilt").toBeDefined()`, which
// can never fail and quietly erased the recorded block from the evidence
// file while both suites stayed green.
const prior = fs.existsSync(EVIDENCE_PATH)
? (JSON.parse(fs.readFileSync(EVIDENCE_PATH, 'utf8')) as {
iceberg?: { totals: Record<string, number> };
})
: {};
expect(
prior.iceberg?.totals,
'no pyiceberg available AND no recorded Iceberg result — the DoD is unproven, not skipped',
).toEqual({ NO: 20, SE: 40 });
evidence.iceberg = { ...prior.iceberg, carried_forward: true };
return;
}
const meta = fs
.readFileSync(metaFile, 'utf8')
.trim()
.replace(/^file:\/\//, '');
const bootstrap = duckDbSessionBootstrap([FIXTURES]);
const res = runDuck(
bootstrap,
`SELECT country, sum(amount) AS s FROM iceberg_scan('${meta}') GROUP BY country ORDER BY country`,
true,
);
expect(res.ok, `iceberg_scan failed: ${res.out}`).toBe(true);
const rows = JSON.parse(res.out) as string[][];
expect(Object.fromEntries(rows.map((r) => [r[0], Number(r[1])]))).toEqual({ NO: 20, SE: 40 });
evidence.iceberg = {
read_by: 'iceberg_scan on the exact metadata JSON',
note:
'an exact metadata file is a SNAPSHOT reference; DuckDB refuses to glob for the latest ' +
'version by default, and that refusal is correct — globbing can read uncommitted data',
totals: { NO: 20, SE: 40 },
};
});
});
suite('BI-P014 — the mount registry (layer 1)', () => {
it('refuses an artifact belonging to another tenant, as AUTHORIZATION_DENIED not NOT_FOUND', async () => {
let thrown: unknown;
try {
await provider.query(IDENTITY, { artifact_id: 'a_beta_parquet' });
} catch (e) {
thrown = e;
}
expect((thrown as SiaxError).code).toBe('AUTHORIZATION_DENIED');
});
it('refuses an unmounted artifact', async () => {
await expect(provider.query(IDENTITY, { artifact_id: 'a_nope' })).rejects.toBeInstanceOf(SiaxError);
});
it('refuses a SYMLINK that escapes the mount root', () => {
const link = path.join(FIXTURES, 'escape.parquet');
fs.rmSync(link, { force: true });
fs.symlinkSync('/etc/passwd', link);
try {
expect(() =>
registry.register({
artifact_id: 'a_escape',
tenant_id: 't_alpha',
st0re_ref: 'st0re://x',
relative_path: 'escape.parquet',
format: 'parquet',
size_bytes: 1,
}),
).toThrowError(SiaxError);
evidence.symlink_escape_refused = true;
} finally {
fs.rmSync(link, { force: true });
}
});
it('refuses a traversing relative path', () => {
expect(() =>
registry.register({
artifact_id: 'a_trav',
tenant_id: 't_alpha',
st0re_ref: 'st0re://x',
relative_path: '../../../../etc/passwd',
format: 'csv',
size_bytes: 1,
}),
).toThrowError(SiaxError);
});
});
suite('BI-P014 — layer 2 alone, WITH THE MOUNT REGISTRY REMOVED', () => {
/** Attempts sent straight to DuckDB. Layer 1 is not in the picture. */
const attempts: Array<{ id: string; sql: string }> = [
{ id: 'read-etc-passwd', sql: "SELECT * FROM read_csv('/etc/passwd')" },
{ id: 'read-etc-shadow', sql: "SELECT * FROM read_csv('/etc/shadow')" },
{ id: 'traversal-from-mount', sql: `SELECT * FROM read_csv('${FIXTURES}/../../../../etc/passwd')` },
{ id: 'glob-root', sql: "SELECT * FROM read_csv('/etc/*')" },
{ id: 'http-exfiltration', sql: "SELECT * FROM read_csv('https://example.com/a.csv')" },
{ id: 's3-read', sql: "SELECT * FROM read_parquet('s3://bucket/a.parquet')" },
// Not a SELECT: ATTACH is the other way a local engine touches a file.
// The first version of this corpus had `SELECT 1 FROM (SELECT 1) t` here,
// which is not an attack at all — it succeeded and was reported as a leak.
// A corpus entry that cannot fail is as useless as a check that cannot.
{ id: 'attach-external-db', sql: "ATTACH '/tmp/duckdb_escape.db' AS evil" },
{ id: 'attach-etc', sql: "ATTACH '/etc/passwd' AS evil2 (READ_ONLY)" },
{ id: 'copy-out-of-mount', sql: "COPY (SELECT 1 AS a) TO '/tmp/duckdb_escape.csv' (FORMAT CSV)" },
{ id: 'read-parquet-outside', sql: "SELECT * FROM read_parquet('/root/x.parquet')" },
{ id: 'json-outside', sql: "SELECT * FROM read_json_auto('/etc/hosts')" },
];
it('no attempt reads anything outside the mount, and the file inside still works', () => {
const bootstrap = duckDbSessionBootstrap([FIXTURES]);
const results: Array<Record<string, unknown>> = [];
const leaks: string[] = [];
for (const attempt of attempts) {
const res = runDuck(bootstrap, attempt.sql);
// A leak is a SUCCESSFUL read. `s3://` and `https://` may fail for other
// reasons too, but the assertion is on success, not on the message.
if (res.ok) leaks.push(attempt.id);
results.push({ id: attempt.id, succeeded: res.ok, engine_message: res.out.slice(0, 140) });
}
// The counterfactual: the SAME session reads the mounted file fine, so
// "nothing succeeded" is not because the sandbox blocks everything.
const inside = runDuck(bootstrap, `SELECT count(*) FROM read_parquet('${FIXTURES}/sales.parquet')`);
expect(inside.ok, 'the sandbox blocked the mounted file too — it is not a sandbox, it is an outage').toBe(
true,
);
expect(JSON.parse(inside.out)[0][0]).toBe('5');
expect(leaks, `DuckDB read outside the mount: ${leaks.join(', ')}`).toEqual([]);
evidence.sandbox_bypass_attempts = results;
evidence.counterfactual_mounted_file_readable = true;
}, 120_000);
it('the NAIVE spelling of the sandbox is a NO-OP — measured, not assumed', () => {
// allowed_directories WITHOUT enable_external_access=false. This is what
// an implementer reaches for first, and it permits everything.
const naive = [`SET allowed_directories=['${FIXTURES}']`];
const res = runDuck(naive, "SELECT * FROM read_csv('/etc/passwd') LIMIT 1");
expect(res.ok, 'if this ever fails, DuckDB changed its behaviour and the header comment is stale').toBe(
true,
);
evidence.naive_allowed_directories_is_a_noop = {
statements: naive,
read_etc_passwd_succeeded: res.ok,
note:
'allowed_directories alone appears in duckdb_settings() and looks like a sandbox. ' +
'It permits /etc/passwd and outbound HTTP. Only enable_external_access=false binds it.',
};
});
it('a locked session cannot re-open its own sandbox', () => {
const bootstrap = duckDbSessionBootstrap([FIXTURES]);
// Sanity: the locked session works.
expect(runDuck(bootstrap, 'SELECT 1').ok).toBe(true);
const reopened: Array<Record<string, unknown>> = [];
for (const reopen of [
"SET allowed_directories=['/etc']",
'SET enable_external_access=true',
'SET lock_configuration=false',
]) {
// Appending the re-open AFTER lock_configuration=true must fail, and it
// must fail on the SET itself — so the following read never happens.
const attempt = runDuck([...bootstrap, reopen], "SELECT * FROM read_csv('/etc/passwd') LIMIT 1");
expect(attempt.ok, `"${reopen}" was accepted after lock_configuration=true`).toBe(false);
reopened.push({ statement: reopen, refused: !attempt.ok, engine_message: attempt.out.slice(0, 120) });
}
evidence.reconfiguration_refused = reopened;
}, 60_000);
it('writes the evidence file', () => {
fs.mkdirSync(path.dirname(EVIDENCE_PATH), { recursive: true });
fs.writeFileSync(EVIDENCE_PATH, `${JSON.stringify(evidence, null, 2)}\n`);
expect(fs.existsSync(EVIDENCE_PATH)).toBe(true);
});
});