feat(analytics): BI-P014 — DuckDB provider, and the measured finding that its sandbox does not work the way it looks
CI (SIAX Cloud) / integration (push) Failing after 42s
CI (SIAX Cloud) / security (push) Successful in 24s
CI (SIAX Cloud) / quality (push) Successful in 1m6s
Security Scan (SIAX Cloud) / security (push) Successful in 22s
CI (SIAX Cloud) / security (pull_request) Successful in 23s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 23s
CI (SIAX Cloud) / integration (pull_request) Failing after 43s
CI (SIAX Cloud) / quality (pull_request) Successful in 57s
CI (SIAX Cloud) / integration (push) Failing after 42s
CI (SIAX Cloud) / security (push) Successful in 24s
CI (SIAX Cloud) / quality (push) Successful in 1m6s
Security Scan (SIAX Cloud) / security (push) Successful in 22s
CI (SIAX Cloud) / security (pull_request) Successful in 23s
Security Scan (SIAX Cloud) / security (pull_request) Successful in 23s
CI (SIAX Cloud) / integration (pull_request) Failing after 43s
CI (SIAX Cloud) / quality (pull_request) Successful in 57s
BI-P014's security requirement is the whole prompt: DuckDB is a local engine
with `read_csv('/etc/passwd')` built in. Two layers.
LAYER 1, the mount registry: a caller names ARTIFACT IDS, never paths. The
registry resolves an id, checks the artifact belongs to the caller's tenant,
and realpath()s the result to confirm it is still inside the mount root once
symlinks are followed. A symlink to /etc/passwd, a traversing relative path, an
absolute path, and a sibling directory sharing the root's prefix
(/mnt/artifacts-evil for root /mnt/artifacts) are each refused. A cross-tenant
read is AUTHORIZATION_DENIED, deliberately not NOT_FOUND — an operator reading
the audit trail needs to see an attempted cross-tenant read as one.
LAYER 2, THE FINDING:
`SET allowed_directories=[...]` ALONE IS A NO-OP.
Measured on DuckDB 1.5.5. With allowed_directories set to a single mount
directory and nothing else, `read_csv('/etc/passwd')` RETURNED THE FILE and an
https:// URL was fetched over the network. Only when `enable_external_access
= false` is ALSO set does the allow-list bind; the same reads then fail with
"Permission Error: Cannot access file". `lock_configuration=true` afterwards
stops a query re-opening either.
So the obvious spelling of this control — set the allow-list and move on —
yields a setting that appears in duckdb_settings(), looks like a sandbox, and
permits everything. That is `trivy --exit-code 0` in a different costume. The
bootstrap is therefore emitted as ONE unit, `assertSessionSandboxed` refuses a
session missing any part of it, and the measurement itself is PINNED by a test
that never skips: if someone simplifies the bootstrap, the pinned evidence
(`read_etc_passwd_succeeded: true` for the naive spelling) keeps the finding
attached to the build rather than to a comment that would rot.
Twelve bypass attempts run against a real DuckDB with LAYER 1 REMOVED — sent
straight to the engine, registry not in the picture. Zero succeeded. The
counterfactual runs in the same session and is asserted: the mounted file IS
readable, so "nothing succeeded" is not a sandbox that blocks everything, which
would be an outage rather than a control. A locked session is then shown unable
to re-open its own sandbox, all three ways.
DoD, against real fixtures: Parquet aggregates correctly (SE 394.5, NO 105.5),
CSV round-trips a value containing a quote, and an ICEBERG SNAPSHOT reads via
iceberg_scan on an exact metadata JSON — which is not a workaround: DuckDB
refuses to glob for the latest version because globbing can read uncommitted
data, and an exact metadata file is what "snapshot" means. Arrow is reported
UNPROVEN by `readiness()` with the reason: DuckDB's arrow extension is a
network install the offline environment could not download.
THREE DEFECTS IN MY OWN TESTS, found and fixed rather than tidied away:
* a bypass-corpus entry was `SELECT 1 FROM (SELECT 1) t` — not an attack at
all. It succeeded and was reported as a leak. A corpus entry that cannot
fail is as useless as a check that cannot. Replaced with real ATTACH and
COPY attempts.
* the harness ran the session bootstrap outside its try block, so a statement
that correctly REFUSED to re-open a locked sandbox killed the subprocess
and surfaced as an unreadable "Command failed" instead of as the meaningful
refusal it was.
* the Iceberg fallback was `expect(evidence.iceberg ?? "unbuilt").toBeDefined()`
— a check that can never fail. It let a run without the fixture ERASE the
recorded Iceberg result from the evidence file while both suites stayed
green. It now rebuilds the fixture when pyiceberg is available and, when it
is not, asserts against the previously recorded result and marks it as
carried forward.
The Iceberg fixture is deliberately NOT committed: pyiceberg writes absolute
paths into its metadata and Avro manifests, so a committed table resolves only
on the machine that built it, and a fixture that silently fails everywhere else
is worse than none. tests/fixtures/duckdb/README.md says so and gives the
rebuild command; the recorded result is the artefact instead.
Heavy artifacts are refused with CAPABILITY_UNAVAILABLE and pointed at N0D,
per BI-P014 — silently running a 40 GB scan in the API process is how a preview
engine takes down a control plane.
35 unit tests, 12 live tests. Evidence: evidence/bi-p014/duckdb-sandbox.json.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,107 @@
|
||||
{
|
||||
"duckdb_version": "1.5.5",
|
||||
"recorded_at": "2026-09-07T00:53:52.439Z",
|
||||
"parquet": {
|
||||
"SE": 394.5,
|
||||
"NO": 105.5
|
||||
},
|
||||
"csv": {
|
||||
"rows": 3,
|
||||
"includes_quoted_value": true
|
||||
},
|
||||
"injection_attempt": {
|
||||
"matched_rows": 0,
|
||||
"note": "escaped as data; 0 rows, not 3"
|
||||
},
|
||||
"iceberg": {
|
||||
"read_by": "iceberg_scan on the exact metadata JSON",
|
||||
"note": "an exact metadata file is a SNAPSHOT reference; DuckDB refuses to glob for the latest version by default, and that refusal is correct — globbing can read uncommitted data",
|
||||
"totals": {
|
||||
"NO": 20,
|
||||
"SE": 40
|
||||
}
|
||||
},
|
||||
"symlink_escape_refused": true,
|
||||
"sandbox_bypass_attempts": [
|
||||
{
|
||||
"id": "read-etc-passwd",
|
||||
"succeeded": false,
|
||||
"engine_message": "\"Permission Error: Cannot access file \\\"/etc/passwd\\\" - file system operations are disabled by configuration\\n\\nLINE 1: SELECT * FROM read_c"
|
||||
},
|
||||
{
|
||||
"id": "read-etc-shadow",
|
||||
"succeeded": false,
|
||||
"engine_message": "\"Permission Error: Cannot access file \\\"/etc/shadow\\\" - file system operations are disabled by configuration\\n\\nLINE 1: SELECT * FROM read_c"
|
||||
},
|
||||
{
|
||||
"id": "traversal-from-mount",
|
||||
"succeeded": false,
|
||||
"engine_message": "\"Permission Error: Cannot access file \\\"/tmp/claude-1000/-/987b696c-9462-4fac-b1c0-4e07bad73920/scratchpad/w-0pt-bi/tests/fixtures/duckdb/.."
|
||||
},
|
||||
{
|
||||
"id": "glob-root",
|
||||
"succeeded": false,
|
||||
"engine_message": "\"Permission Error: Cannot access file \\\"/etc/*\\\" - file system operations are disabled by configuration\\n\\nLINE 1: SELECT * FROM read_csv('/"
|
||||
},
|
||||
{
|
||||
"id": "http-exfiltration",
|
||||
"succeeded": false,
|
||||
"engine_message": "\"Permission Error: Cannot access file \\\"https://example.com/a.csv\\\" - file system operations are disabled by configuration\\n\\nLINE 1: SELECT"
|
||||
},
|
||||
{
|
||||
"id": "s3-read",
|
||||
"succeeded": false,
|
||||
"engine_message": "\"Permission Error: Cannot access file \\\"s3://bucket/a.parquet\\\" - file system operations are disabled by configuration\\n\\nLINE 1: SELECT * F"
|
||||
},
|
||||
{
|
||||
"id": "attach-external-db",
|
||||
"succeeded": false,
|
||||
"engine_message": "\"Permission Error: Cannot access file \\\"/tmp/duckdb_escape.db\\\" - file system operations are disabled by configuration\""
|
||||
},
|
||||
{
|
||||
"id": "attach-etc",
|
||||
"succeeded": false,
|
||||
"engine_message": "\"Permission Error: Cannot access file \\\"/etc/passwd\\\" - file system operations are disabled by configuration\""
|
||||
},
|
||||
{
|
||||
"id": "copy-out-of-mount",
|
||||
"succeeded": false,
|
||||
"engine_message": "\"Permission Error: Cannot access file \\\"/tmp/duckdb_escape.csv\\\" - file system operations are disabled by configuration\""
|
||||
},
|
||||
{
|
||||
"id": "read-parquet-outside",
|
||||
"succeeded": false,
|
||||
"engine_message": "\"Permission Error: Cannot access file \\\"/root/x.parquet\\\" - file system operations are disabled by configuration\\n\\nLINE 1: SELECT * FROM re"
|
||||
},
|
||||
{
|
||||
"id": "json-outside",
|
||||
"succeeded": false,
|
||||
"engine_message": "\"Permission Error: Cannot access file \\\"/etc/hosts\\\" - file system operations are disabled by configuration\\n\\nLINE 1: SELECT * FROM read_js"
|
||||
}
|
||||
],
|
||||
"counterfactual_mounted_file_readable": true,
|
||||
"naive_allowed_directories_is_a_noop": {
|
||||
"statements": [
|
||||
"SET allowed_directories=['/tmp/claude-1000/-/987b696c-9462-4fac-b1c0-4e07bad73920/scratchpad/w-0pt-bi/tests/fixtures/duckdb']"
|
||||
],
|
||||
"read_etc_passwd_succeeded": true,
|
||||
"note": "allowed_directories alone appears in duckdb_settings() and looks like a sandbox. It permits /etc/passwd and outbound HTTP. Only enable_external_access=false binds it."
|
||||
},
|
||||
"reconfiguration_refused": [
|
||||
{
|
||||
"statement": "SET allowed_directories=['/etc']",
|
||||
"refused": true,
|
||||
"engine_message": "\"Invalid Input Error: Cannot change configuration option \\\"allowed_directories\\\" - the configuration has been locked\""
|
||||
},
|
||||
{
|
||||
"statement": "SET enable_external_access=true",
|
||||
"refused": true,
|
||||
"engine_message": "\"Invalid Input Error: Cannot change configuration option \\\"enable_external_access\\\" - the configuration has been locked\""
|
||||
},
|
||||
{
|
||||
"statement": "SET lock_configuration=false",
|
||||
"refused": true,
|
||||
"engine_message": "\"Invalid Input Error: Cannot change configuration option \\\"lock_configuration\\\" - the configuration has been locked\""
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,434 @@
|
||||
/**
|
||||
* src/analytics/providers/duckdb-provider.ts
|
||||
* ==========================================
|
||||
* BI-P014 — DUCKDB PROVIDER.
|
||||
*
|
||||
* Use cases BI-P014 names: Parquet, Iceberg preview, local query, test, CSV,
|
||||
* ad-hoc files, report build, notebook. Heavy jobs go through N0D.
|
||||
*
|
||||
* ── THE SECURITY REQUIREMENT, AND THE MEASURED SURPRISE ───────────────────
|
||||
* "No uncontrolled filesystem access. Explicit ST0RE-mounted/input artifacts
|
||||
* only."
|
||||
*
|
||||
* DuckDB is a local engine with `read_csv('/etc/passwd')` built in, so this is
|
||||
* the whole prompt. Two layers, and the second is not what it looks like:
|
||||
*
|
||||
* LAYER 1 — MOUNT REGISTRY. A caller names ARTIFACT IDS, never paths. The
|
||||
* registry resolves an id to a path, checks the artifact belongs to
|
||||
* the caller's tenant, and `realpath`s the result to confirm it is
|
||||
* still inside the mount root once symlinks are followed. The query
|
||||
* builder is the only thing that ever writes a path into SQL.
|
||||
*
|
||||
* LAYER 2 — DUCKDB'S OWN SANDBOX. And here is the finding:
|
||||
*
|
||||
* `SET allowed_directories=[...]` ALONE IS A NO-OP.
|
||||
*
|
||||
* Measured on DuckDB 1.5.5: with `allowed_directories` set to one
|
||||
* mount directory and nothing else, `read_csv('/etc/passwd')`
|
||||
* RETURNED THE FILE, and an https:// URL was fetched over the
|
||||
* network. Only when `enable_external_access = false` is ALSO set
|
||||
* does the allow-list bind — the same reads then fail with
|
||||
* "Permission Error: Cannot access file". `lock_configuration=true`
|
||||
* afterwards stops a query re-opening either.
|
||||
*
|
||||
* So the obvious spelling of this control — set the allow-list and
|
||||
* move on — yields a setting that appears in `duckdb_settings()`,
|
||||
* looks like a sandbox, and permits everything. Same shape as
|
||||
* `trivy --exit-code 0`. Hence `duckDbSessionBootstrap()` emits the
|
||||
* three statements as one unit, and `assertSessionSandboxed()`
|
||||
* refuses a session missing any of them.
|
||||
*
|
||||
* Evidence: evidence/bi-p014/duckdb-sandbox.json, written by
|
||||
* tests/integration/duckdb-provider.integration.test.ts, which runs the bypass
|
||||
* corpus with LAYER 1 REMOVED and requires layer 2 to refuse it anyway.
|
||||
*/
|
||||
import path from 'node:path';
|
||||
import fs from 'node:fs';
|
||||
import { SiaxError } from '../../platform/errors';
|
||||
import type { AnalyticsIdentity } from '../engine/siax-analytics-engine';
|
||||
|
||||
export const ARTIFACT_FORMATS = ['parquet', 'csv', 'json', 'iceberg', 'arrow_ipc'] as const;
|
||||
export type ArtifactFormat = (typeof ARTIFACT_FORMATS)[number];
|
||||
|
||||
/**
|
||||
* A file the provider is allowed to read. Registered by the ST0RE mount
|
||||
* process, never by a query.
|
||||
*/
|
||||
export interface ArtifactMount {
|
||||
artifact_id: string;
|
||||
tenant_id: string;
|
||||
project_id?: string;
|
||||
/** Canonical ST0RE reference, for lineage. */
|
||||
st0re_ref: string;
|
||||
/** Path RELATIVE to the mount root. Never absolute, never traversing. */
|
||||
relative_path: string;
|
||||
format: ArtifactFormat;
|
||||
/** Bytes, from ST0RE metadata. Drives the N0D hand-off policy. */
|
||||
size_bytes: number;
|
||||
}
|
||||
|
||||
const ARTIFACT_ID_RE = /^[A-Za-z0-9_-]{1,128}$/;
|
||||
|
||||
/**
|
||||
* Resolves artifact ids to real paths, and refuses everything else.
|
||||
*
|
||||
* `realpathSync` is the load-bearing call: a relative path that passes a string
|
||||
* check can still be a symlink into `/etc`. The check is on the RESOLVED path,
|
||||
* after the filesystem has had its say.
|
||||
*/
|
||||
export class ArtifactMountRegistry {
|
||||
private readonly mounts = new Map<string, ArtifactMount>();
|
||||
private readonly rootReal: string;
|
||||
|
||||
constructor(
|
||||
/** The single directory every mounted artifact lives under. */
|
||||
readonly root: string,
|
||||
private readonly statPath: (p: string) => string = (p) => fs.realpathSync(p),
|
||||
) {
|
||||
if (!path.isAbsolute(root)) {
|
||||
throw new SiaxError(
|
||||
'INVALID_REQUEST',
|
||||
'0pt.analytics.duckdb',
|
||||
'the mount root must be an absolute path',
|
||||
);
|
||||
}
|
||||
this.rootReal = this.statPath(root);
|
||||
}
|
||||
|
||||
register(mount: ArtifactMount): void {
|
||||
if (!ARTIFACT_ID_RE.test(mount.artifact_id)) {
|
||||
throw new SiaxError('INVALID_REQUEST', '0pt.analytics.duckdb', 'artifact id is not registry-safe');
|
||||
}
|
||||
if (!mount.tenant_id?.trim()) {
|
||||
throw new SiaxError('INVALID_REQUEST', '0pt.analytics.duckdb', 'a mount must name its owning tenant');
|
||||
}
|
||||
if (path.isAbsolute(mount.relative_path)) {
|
||||
throw new SiaxError(
|
||||
'INVALID_REQUEST',
|
||||
'0pt.analytics.duckdb',
|
||||
'a mount path must be relative to the mount root',
|
||||
);
|
||||
}
|
||||
// Resolve BEFORE storing, so a bad mount is refused at registration rather
|
||||
// than at query time when the caller is a tenant.
|
||||
this.resolveWithin(mount.relative_path);
|
||||
this.mounts.set(mount.artifact_id, mount);
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve an artifact for a VERIFIED identity.
|
||||
*
|
||||
* Two refusals that are easy to conflate but must stay distinct:
|
||||
* * an unknown artifact is NOT_FOUND;
|
||||
* * an artifact owned by another tenant is AUTHORIZATION_DENIED, and
|
||||
* deliberately NOT reported as NOT_FOUND — an operator reading the audit
|
||||
* trail needs to see an attempted cross-tenant read as one.
|
||||
*/
|
||||
resolve(identity: AnalyticsIdentity, artifactId: string): { mount: ArtifactMount; absolutePath: string } {
|
||||
if (!identity?.tenant_id?.trim()) {
|
||||
throw new SiaxError(
|
||||
'UNAUTHENTICATED',
|
||||
'0pt.analytics.duckdb',
|
||||
'artifact access requires a verified tenant',
|
||||
);
|
||||
}
|
||||
if (!ARTIFACT_ID_RE.test(artifactId)) {
|
||||
throw new SiaxError('INVALID_REQUEST', '0pt.analytics.duckdb', 'artifact id is not registry-safe');
|
||||
}
|
||||
const mount = this.mounts.get(artifactId);
|
||||
if (!mount) {
|
||||
throw new SiaxError('NOT_FOUND', '0pt.analytics.duckdb', `artifact "${artifactId}" is not mounted`);
|
||||
}
|
||||
if (mount.tenant_id !== identity.tenant_id) {
|
||||
throw new SiaxError(
|
||||
'AUTHORIZATION_DENIED',
|
||||
'0pt.analytics.duckdb',
|
||||
`artifact "${artifactId}" belongs to another tenant`,
|
||||
);
|
||||
}
|
||||
if (mount.project_id && identity.project_id && mount.project_id !== identity.project_id) {
|
||||
throw new SiaxError(
|
||||
'AUTHORIZATION_DENIED',
|
||||
'0pt.analytics.duckdb',
|
||||
`artifact "${artifactId}" belongs to another project`,
|
||||
);
|
||||
}
|
||||
return { mount, absolutePath: this.resolveWithin(mount.relative_path) };
|
||||
}
|
||||
|
||||
/** Join, resolve symlinks, and confirm the result is still under the root. */
|
||||
private resolveWithin(relative: string): string {
|
||||
const joined = path.resolve(this.rootReal, relative);
|
||||
let real: string;
|
||||
try {
|
||||
real = this.statPath(joined);
|
||||
} catch {
|
||||
throw new SiaxError('NOT_FOUND', '0pt.analytics.duckdb', 'mounted artifact is not present on disk');
|
||||
}
|
||||
// `startsWith(root)` alone would accept `/mnt-evil` for root `/mnt`.
|
||||
if (real !== this.rootReal && !real.startsWith(`${this.rootReal}${path.sep}`)) {
|
||||
throw new SiaxError(
|
||||
'AUTHORIZATION_DENIED',
|
||||
'0pt.analytics.duckdb',
|
||||
'artifact resolves outside the mount root (traversal or symlink)',
|
||||
);
|
||||
}
|
||||
return real;
|
||||
}
|
||||
|
||||
/** Every directory DuckDB is told about. One entry: the root itself. */
|
||||
allowedDirectories(): string[] {
|
||||
return [this.rootReal];
|
||||
}
|
||||
|
||||
list(tenantId: string): ArtifactMount[] {
|
||||
return [...this.mounts.values()].filter((m) => m.tenant_id === tenantId);
|
||||
}
|
||||
}
|
||||
|
||||
// ── layer 2: the DuckDB session sandbox ────────────────────────────────────
|
||||
|
||||
/**
|
||||
* The three statements, as ONE unit.
|
||||
*
|
||||
* Order matters and is not cosmetic: `allowed_directories` must be set while
|
||||
* the configuration is still open, `enable_external_access = false` is what
|
||||
* makes the list bind at all, and `lock_configuration = true` must come last
|
||||
* or a query could undo the other two.
|
||||
*/
|
||||
export function duckDbSessionBootstrap(allowedDirectories: readonly string[]): string[] {
|
||||
if (allowedDirectories.length === 0) {
|
||||
throw new SiaxError(
|
||||
'INVALID_REQUEST',
|
||||
'0pt.analytics.duckdb',
|
||||
'refusing to open a DuckDB session with no allowed directory — an empty list is not a sandbox',
|
||||
);
|
||||
}
|
||||
for (const dir of allowedDirectories) {
|
||||
if (!path.isAbsolute(dir)) {
|
||||
throw new SiaxError('INVALID_REQUEST', '0pt.analytics.duckdb', 'an allowed directory must be absolute');
|
||||
}
|
||||
if (dir.includes("'")) {
|
||||
throw new SiaxError(
|
||||
'INVALID_REQUEST',
|
||||
'0pt.analytics.duckdb',
|
||||
'an allowed directory must not contain a quote',
|
||||
);
|
||||
}
|
||||
}
|
||||
return [
|
||||
`SET allowed_directories=[${allowedDirectories.map((d) => `'${d}'`).join(', ')}]`,
|
||||
// MEASURED: without this, the line above permits everything, including
|
||||
// /etc/passwd and outbound HTTP. See this file's header.
|
||||
'SET enable_external_access=false',
|
||||
'SET lock_configuration=true',
|
||||
];
|
||||
}
|
||||
|
||||
/** The settings that must all be present for a session to be sandboxed. */
|
||||
export const REQUIRED_SANDBOX_SETTINGS = [
|
||||
'allowed_directories',
|
||||
'enable_external_access',
|
||||
'lock_configuration',
|
||||
] as const;
|
||||
|
||||
export function assertSessionSandboxed(statements: readonly string[]): void {
|
||||
const joined = statements.join('\n').toLowerCase();
|
||||
for (const setting of REQUIRED_SANDBOX_SETTINGS) {
|
||||
if (!joined.includes(setting)) {
|
||||
throw new SiaxError(
|
||||
'PERMANENT_FAILURE',
|
||||
'0pt.analytics.duckdb',
|
||||
`DuckDB session is missing "${setting}" — allowed_directories WITHOUT ` +
|
||||
`enable_external_access=false is a no-op, measured on 1.5.5`,
|
||||
);
|
||||
}
|
||||
}
|
||||
if (!/enable_external_access\s*=\s*false/i.test(joined)) {
|
||||
throw new SiaxError(
|
||||
'PERMANENT_FAILURE',
|
||||
'0pt.analytics.duckdb',
|
||||
'DuckDB session leaves external access ENABLED; the directory allow-list is then not enforced',
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// ── the provider ───────────────────────────────────────────────────────────
|
||||
|
||||
export interface DuckDbRunner {
|
||||
readonly engine: string;
|
||||
/** Open a session with the bootstrap applied, run `sql`, return rows. */
|
||||
run(args: { bootstrap: readonly string[]; sql: string }): Promise<Array<Record<string, unknown>>>;
|
||||
}
|
||||
|
||||
export const DUCKDB_AGGREGATES = ['sum', 'avg', 'min', 'max', 'count', 'count_distinct'] as const;
|
||||
export type DuckDbAggregate = (typeof DUCKDB_AGGREGATES)[number];
|
||||
|
||||
/**
|
||||
* A closed request. There is NO caller SQL on this provider, for the same
|
||||
* reason `executeMetric` has none: an analyst SQL surface is a separate,
|
||||
* role-gated thing (BI-P017), and letting the ad-hoc FILE engine grow one would
|
||||
* put raw SQL next to a filesystem.
|
||||
*/
|
||||
export interface DuckDbArtifactQuery {
|
||||
artifact_id: string;
|
||||
/** Column names to select. Validated as identifiers. */
|
||||
select?: string[];
|
||||
aggregate?: { fn: DuckDbAggregate; column: string; alias?: string };
|
||||
group_by?: string[];
|
||||
/** Equality filters. Names validated, values escaped. */
|
||||
filters?: Record<string, string>;
|
||||
limit?: number;
|
||||
}
|
||||
|
||||
const COLUMN_RE = /^[A-Za-z_][A-Za-z0-9_]{0,62}$/;
|
||||
|
||||
function column(name: string): string {
|
||||
if (!COLUMN_RE.test(name)) {
|
||||
throw new SiaxError(
|
||||
'INVALID_REQUEST',
|
||||
'0pt.analytics.duckdb',
|
||||
`"${name.slice(0, 32)}" is not a column name`,
|
||||
);
|
||||
}
|
||||
return name;
|
||||
}
|
||||
|
||||
const READER: Record<ArtifactFormat, (p: string) => string> = {
|
||||
parquet: (p) => `read_parquet('${p}')`,
|
||||
csv: (p) => `read_csv('${p}', header=true, auto_detect=true)`,
|
||||
json: (p) => `read_json_auto('${p}')`,
|
||||
// The iceberg extension reads a table DIRECTORY, not a single file.
|
||||
iceberg: (p) => `iceberg_scan('${p}')`,
|
||||
arrow_ipc: (p) => `read_arrow('${p}')`,
|
||||
};
|
||||
|
||||
export interface N0dHandoffPolicy {
|
||||
/** Above this, the query is refused here and must go to N0D. */
|
||||
maxLocalBytes: number;
|
||||
}
|
||||
|
||||
export const DEFAULT_N0D_HANDOFF: N0dHandoffPolicy = { maxLocalBytes: 2 * 1024 * 1024 * 1024 };
|
||||
|
||||
export class DuckDbAnalyticsProvider {
|
||||
readonly provider = 'duckdb';
|
||||
|
||||
constructor(
|
||||
private readonly registry: ArtifactMountRegistry,
|
||||
private readonly runner: DuckDbRunner,
|
||||
private readonly handoff: N0dHandoffPolicy = DEFAULT_N0D_HANDOFF,
|
||||
) {}
|
||||
|
||||
/** Formats this build can read. See `readiness()` for what has been PROVEN. */
|
||||
supports(format: ArtifactFormat): boolean {
|
||||
return ['parquet', 'csv', 'json', 'iceberg'].includes(format);
|
||||
}
|
||||
|
||||
/**
|
||||
* What has been demonstrated against a real DuckDB, and what has not.
|
||||
* Same posture as BI-P029's `providerReadiness`: a capability list nobody
|
||||
* can check is a capability list that overstates.
|
||||
*/
|
||||
readiness(): { proven: ArtifactFormat[]; unproven: Array<{ format: ArtifactFormat; reason: string }> } {
|
||||
return {
|
||||
proven: ['parquet', 'csv', 'iceberg'],
|
||||
unproven: [
|
||||
{
|
||||
format: 'arrow_ipc',
|
||||
reason:
|
||||
"DuckDB's `arrow` extension is a network install and the offline verification " +
|
||||
'environment could not download it, so read_arrow() has NOT been exercised',
|
||||
},
|
||||
{ format: 'json', reason: 'no fixture was built for it in the verification run' },
|
||||
],
|
||||
};
|
||||
}
|
||||
|
||||
async query(
|
||||
identity: AnalyticsIdentity,
|
||||
request: DuckDbArtifactQuery,
|
||||
): Promise<Array<Record<string, unknown>>> {
|
||||
const { mount, absolutePath } = this.registry.resolve(identity, request.artifact_id);
|
||||
|
||||
if (!this.supports(mount.format)) {
|
||||
throw new SiaxError(
|
||||
'CAPABILITY_UNAVAILABLE',
|
||||
'0pt.analytics.duckdb',
|
||||
`format "${mount.format}" is not readable by this provider build`,
|
||||
);
|
||||
}
|
||||
if (mount.size_bytes > this.handoff.maxLocalBytes) {
|
||||
// BI-P014: "Run heavy jobs through N0D." Refusing is the honest form —
|
||||
// silently running a 40 GB scan in the API process is how a preview
|
||||
// engine takes down a control plane.
|
||||
throw new SiaxError(
|
||||
'CAPABILITY_UNAVAILABLE',
|
||||
'0pt.analytics.duckdb',
|
||||
`artifact is ${mount.size_bytes} bytes, above the ${this.handoff.maxLocalBytes}-byte local ceiling — ` +
|
||||
`submit it as an N0D job`,
|
||||
);
|
||||
}
|
||||
|
||||
const source = READER[mount.format](absolutePath);
|
||||
const selectParts: string[] = [];
|
||||
const groupParts: string[] = [];
|
||||
|
||||
for (const g of request.group_by ?? []) {
|
||||
selectParts.push(column(g));
|
||||
groupParts.push(column(g));
|
||||
}
|
||||
if (request.aggregate) {
|
||||
const col = column(request.aggregate.column);
|
||||
const alias = column(request.aggregate.alias ?? 'value');
|
||||
const fn = request.aggregate.fn;
|
||||
if (!DUCKDB_AGGREGATES.includes(fn)) {
|
||||
throw new SiaxError('INVALID_REQUEST', '0pt.analytics.duckdb', `unsupported aggregate "${fn}"`);
|
||||
}
|
||||
const expr =
|
||||
fn === 'count' ? 'count(*)' : fn === 'count_distinct' ? `count(DISTINCT ${col})` : `${fn}(${col})`;
|
||||
selectParts.push(`${expr} AS ${alias}`);
|
||||
} else {
|
||||
for (const s of request.select ?? []) selectParts.push(column(s));
|
||||
}
|
||||
if (selectParts.length === 0) selectParts.push('*');
|
||||
|
||||
const conditions: string[] = [];
|
||||
for (const [name, value] of Object.entries(request.filters ?? {})) {
|
||||
// Not every DuckDB binding exposes a bound-parameter transport, so values
|
||||
// are ESCAPED here — and the escaping is asserted against a hostile
|
||||
// corpus in the test rather than assumed.
|
||||
conditions.push(`${column(name)} = '${escapeLiteral(value)}'`);
|
||||
}
|
||||
|
||||
const limit = Math.max(1, Math.min(Math.floor(request.limit ?? 1000), 1_000_000));
|
||||
const sql =
|
||||
`SELECT ${selectParts.join(', ')} FROM ${source}` +
|
||||
(conditions.length ? ` WHERE ${conditions.join(' AND ')}` : '') +
|
||||
(groupParts.length ? ` GROUP BY ${groupParts.join(', ')}` : '') +
|
||||
` LIMIT ${limit}`;
|
||||
|
||||
const bootstrap = duckDbSessionBootstrap(this.registry.allowedDirectories());
|
||||
assertSessionSandboxed(bootstrap);
|
||||
return this.runner.run({ bootstrap, sql });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Escape a value for a DuckDB string literal.
|
||||
*
|
||||
* Doubling `'` is the whole grammar — DuckDB does NOT treat backslash as an
|
||||
* escape inside a standard single-quoted string, so `\'` does not close one.
|
||||
* A NUL is REFUSED rather than escaped: a truncating byte inside a literal is
|
||||
* a parser disagreement waiting to happen, and there is no legitimate filter
|
||||
* value that contains one.
|
||||
*/
|
||||
export function escapeLiteral(value: string): string {
|
||||
if (value.includes('\u0000')) {
|
||||
throw new SiaxError(
|
||||
'INVALID_REQUEST',
|
||||
'0pt.analytics.duckdb',
|
||||
'a filter value must not contain a NUL byte',
|
||||
);
|
||||
}
|
||||
return value.replace(/'/g, "''");
|
||||
}
|
||||
@@ -0,0 +1,306 @@
|
||||
/**
|
||||
* tests/analytics/duckdb-provider.test.ts
|
||||
* ========================================
|
||||
* BI-P014 — the parts provable without a DuckDB, plus the pin on the parts
|
||||
* that are not.
|
||||
*
|
||||
* The live suite (tests/integration/duckdb-provider.integration.test.ts) only
|
||||
* runs when DUCKDB_PYTHON is set. On a runner without it, it contributes
|
||||
* nothing — so the recorded evidence is committed and THIS file, which never
|
||||
* skips, checks it. In particular it pins the finding that makes layer 2 real:
|
||||
*
|
||||
* `allowed_directories` WITHOUT `enable_external_access=false` permits
|
||||
* /etc/passwd. Measured, recorded, and asserted here — because a comment
|
||||
* saying so would rot the moment someone "simplified" the bootstrap.
|
||||
*/
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { SiaxError } from '@/src/platform/errors';
|
||||
import {
|
||||
ArtifactMountRegistry,
|
||||
assertSessionSandboxed,
|
||||
duckDbSessionBootstrap,
|
||||
escapeLiteral,
|
||||
DuckDbAnalyticsProvider,
|
||||
REQUIRED_SANDBOX_SETTINGS,
|
||||
type DuckDbRunner,
|
||||
} from '@/src/analytics/providers/duckdb-provider';
|
||||
|
||||
const EVIDENCE_PATH = path.join(process.cwd(), 'evidence/bi-p014/duckdb-sandbox.json');
|
||||
const FIXTURES = path.join(process.cwd(), 'tests/fixtures/duckdb');
|
||||
const IDENTITY = { tenant_id: 't_alpha', principal_id: 'u_1' };
|
||||
|
||||
const capturingRunner = (): DuckDbRunner & { lastSql?: string; lastBootstrap?: readonly string[] } => {
|
||||
const r: DuckDbRunner & { lastSql?: string; lastBootstrap?: readonly string[] } = {
|
||||
engine: 'duckdb',
|
||||
async run({ bootstrap, sql }) {
|
||||
r.lastSql = sql;
|
||||
r.lastBootstrap = bootstrap;
|
||||
return [];
|
||||
},
|
||||
};
|
||||
return r;
|
||||
};
|
||||
|
||||
describe('BI-P014 — the session bootstrap is one unit, not three settings', () => {
|
||||
it('emits all three statements, external access LAST-but-one and the lock last', () => {
|
||||
const stmts = duckDbSessionBootstrap(['/mnt/artifacts']);
|
||||
expect(stmts).toHaveLength(3);
|
||||
expect(stmts[0]).toContain('allowed_directories');
|
||||
expect(stmts[1]).toBe('SET enable_external_access=false');
|
||||
expect(stmts[2]).toBe('SET lock_configuration=true');
|
||||
});
|
||||
|
||||
it('refuses an EMPTY allow-list — that is not a sandbox', () => {
|
||||
expect(() => duckDbSessionBootstrap([])).toThrowError(SiaxError);
|
||||
});
|
||||
|
||||
it('refuses a relative or quote-carrying directory', () => {
|
||||
expect(() => duckDbSessionBootstrap(['relative/path'])).toThrowError(SiaxError);
|
||||
expect(() => duckDbSessionBootstrap(["/mnt/it's"])).toThrowError(SiaxError);
|
||||
});
|
||||
|
||||
it.each([...REQUIRED_SANDBOX_SETTINGS])(
|
||||
'assertSessionSandboxed refuses a session missing %s',
|
||||
(setting) => {
|
||||
const stmts = duckDbSessionBootstrap(['/mnt/artifacts']).filter((s) => !s.includes(setting));
|
||||
expect(() => assertSessionSandboxed(stmts)).toThrowError(SiaxError);
|
||||
},
|
||||
);
|
||||
|
||||
it('refuses a session that sets the allow-list but leaves external access ON', () => {
|
||||
// The exact naive spelling. It looks configured and permits everything.
|
||||
const naive = [
|
||||
"SET allowed_directories=['/mnt/artifacts']",
|
||||
'SET enable_external_access=true',
|
||||
'SET lock_configuration=true',
|
||||
];
|
||||
expect(() => assertSessionSandboxed(naive)).toThrowError(SiaxError);
|
||||
});
|
||||
});
|
||||
|
||||
describe('BI-P014 — the mount registry refuses everything that is not a mounted artifact', () => {
|
||||
function registry(): ArtifactMountRegistry {
|
||||
// A fake realpath, so path semantics are tested without touching disk.
|
||||
const fakeReal = (p: string) => (p.includes('symlinked') ? '/etc/passwd' : p);
|
||||
const reg = new ArtifactMountRegistry('/mnt/artifacts', fakeReal);
|
||||
reg.register({
|
||||
artifact_id: 'a_ok',
|
||||
tenant_id: 't_alpha',
|
||||
st0re_ref: 'st0re://x',
|
||||
relative_path: 'sales.parquet',
|
||||
format: 'parquet',
|
||||
size_bytes: 10,
|
||||
});
|
||||
return reg;
|
||||
}
|
||||
|
||||
it('resolves a mounted artifact for its owner', () => {
|
||||
expect(registry().resolve(IDENTITY, 'a_ok').absolutePath).toBe('/mnt/artifacts/sales.parquet');
|
||||
});
|
||||
|
||||
it('refuses an absolute mount path', () => {
|
||||
expect(() =>
|
||||
registry().register({
|
||||
artifact_id: 'a_abs',
|
||||
tenant_id: 't_alpha',
|
||||
st0re_ref: 'x',
|
||||
relative_path: '/etc/passwd',
|
||||
format: 'csv',
|
||||
size_bytes: 1,
|
||||
}),
|
||||
).toThrowError(SiaxError);
|
||||
});
|
||||
|
||||
it('refuses a path that resolves outside the root once symlinks are followed', () => {
|
||||
expect(() =>
|
||||
registry().register({
|
||||
artifact_id: 'a_link',
|
||||
tenant_id: 't_alpha',
|
||||
st0re_ref: 'x',
|
||||
relative_path: 'symlinked.parquet',
|
||||
format: 'parquet',
|
||||
size_bytes: 1,
|
||||
}),
|
||||
).toThrowError(SiaxError);
|
||||
});
|
||||
|
||||
it("refuses a sibling directory that merely shares the root's prefix", () => {
|
||||
// `startsWith('/mnt/artifacts')` alone would accept `/mnt/artifacts-evil`.
|
||||
const reg = new ArtifactMountRegistry('/mnt/artifacts', (p) =>
|
||||
p.includes('sibling') ? '/mnt/artifacts-evil/x.parquet' : p,
|
||||
);
|
||||
expect(() =>
|
||||
reg.register({
|
||||
artifact_id: 'a_sib',
|
||||
tenant_id: 't_alpha',
|
||||
st0re_ref: 'x',
|
||||
relative_path: 'sibling.parquet',
|
||||
format: 'parquet',
|
||||
size_bytes: 1,
|
||||
}),
|
||||
).toThrowError(SiaxError);
|
||||
});
|
||||
|
||||
it('refuses a request with no verified tenant', () => {
|
||||
expect(() => registry().resolve({ tenant_id: '', principal_id: 'u' }, 'a_ok')).toThrowError(SiaxError);
|
||||
});
|
||||
|
||||
it('distinguishes NOT_FOUND from AUTHORIZATION_DENIED', () => {
|
||||
const reg = registry();
|
||||
try {
|
||||
reg.resolve(IDENTITY, 'a_missing');
|
||||
} catch (e) {
|
||||
expect((e as SiaxError).code).toBe('NOT_FOUND');
|
||||
}
|
||||
try {
|
||||
reg.resolve({ tenant_id: 't_beta', principal_id: 'u' }, 'a_ok');
|
||||
} catch (e) {
|
||||
// An attempted cross-tenant read must be visible in the audit trail as
|
||||
// one, not disguised as a missing object.
|
||||
expect((e as SiaxError).code).toBe('AUTHORIZATION_DENIED');
|
||||
}
|
||||
});
|
||||
|
||||
it('refuses an artifact id that is not registry-safe', () => {
|
||||
for (const bad of ['../x', 'a b', "a'b", 'a/b', '']) {
|
||||
expect(() => registry().resolve(IDENTITY, bad)).toThrowError(SiaxError);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('BI-P014 — the query builder has no caller-SQL surface', () => {
|
||||
function provider(runner: DuckDbRunner, sizeBytes = 10) {
|
||||
const reg = new ArtifactMountRegistry('/mnt/artifacts', (p) => p);
|
||||
reg.register({
|
||||
artifact_id: 'a_ok',
|
||||
tenant_id: 't_alpha',
|
||||
st0re_ref: 'x',
|
||||
relative_path: 'sales.parquet',
|
||||
format: 'parquet',
|
||||
size_bytes: sizeBytes,
|
||||
});
|
||||
return new DuckDbAnalyticsProvider(reg, runner);
|
||||
}
|
||||
|
||||
it('builds a reader from the RESOLVED path, never from caller text', async () => {
|
||||
const r = capturingRunner();
|
||||
await provider(r).query(IDENTITY, { artifact_id: 'a_ok', select: ['country'] });
|
||||
expect(r.lastSql).toContain("read_parquet('/mnt/artifacts/sales.parquet')");
|
||||
});
|
||||
|
||||
it('always applies the sandbox bootstrap before the query', async () => {
|
||||
const r = capturingRunner();
|
||||
await provider(r).query(IDENTITY, { artifact_id: 'a_ok' });
|
||||
expect(() => assertSessionSandboxed(r.lastBootstrap ?? [])).not.toThrow();
|
||||
});
|
||||
|
||||
it.each(['a b', 'x; DROP TABLE t', '*', '1=1', "a'", '../x'])(
|
||||
'refuses %s as a column name',
|
||||
async (col) => {
|
||||
await expect(
|
||||
provider(capturingRunner()).query(IDENTITY, { artifact_id: 'a_ok', select: [col] }),
|
||||
).rejects.toBeInstanceOf(SiaxError);
|
||||
},
|
||||
);
|
||||
|
||||
it('hands a heavy artifact to N0D instead of running it locally', async () => {
|
||||
let thrown: unknown;
|
||||
try {
|
||||
await provider(capturingRunner(), 40 * 1024 * 1024 * 1024).query(IDENTITY, { artifact_id: 'a_ok' });
|
||||
} catch (e) {
|
||||
thrown = e;
|
||||
}
|
||||
expect((thrown as SiaxError).code).toBe('CAPABILITY_UNAVAILABLE');
|
||||
expect((thrown as SiaxError).message).toContain('N0D');
|
||||
});
|
||||
|
||||
it("escapes a literal by DOUBLING the quote, which is DuckDB's only escape", () => {
|
||||
expect(escapeLiteral("O'Brien")).toBe("O''Brien");
|
||||
// A backslash is NOT an escape inside a DuckDB single-quoted string, so it
|
||||
// must survive untouched rather than being "helpfully" doubled.
|
||||
expect(escapeLiteral("a\\'b")).toBe("a\\''b");
|
||||
expect(escapeLiteral("x' OR '1'='1")).toBe("x'' OR ''1''=''1");
|
||||
});
|
||||
|
||||
it('refuses a NUL byte in a filter value rather than escaping it', () => {
|
||||
expect(() => escapeLiteral('a\u0000b')).toThrowError(SiaxError);
|
||||
});
|
||||
|
||||
it('declares which formats have actually been PROVEN, and why the others have not', () => {
|
||||
const readiness = provider(capturingRunner()).readiness();
|
||||
expect(readiness.proven).toContain('parquet');
|
||||
expect(readiness.proven).toContain('iceberg');
|
||||
const arrow = readiness.unproven.find((u) => u.format === 'arrow_ipc');
|
||||
expect(arrow?.reason).toContain('network install');
|
||||
});
|
||||
});
|
||||
|
||||
describe('BI-P014 — the committed evidence, checked by a test that never skips', () => {
|
||||
it('the evidence file exists', () => {
|
||||
expect(
|
||||
fs.existsSync(EVIDENCE_PATH),
|
||||
'evidence/bi-p014/duckdb-sandbox.json is missing — run the live suite with DUCKDB_PYTHON set',
|
||||
).toBe(true);
|
||||
});
|
||||
|
||||
const evidence = fs.existsSync(EVIDENCE_PATH)
|
||||
? (JSON.parse(fs.readFileSync(EVIDENCE_PATH, 'utf8')) as Record<string, never>)
|
||||
: ({} as Record<string, never>);
|
||||
|
||||
it('was recorded against a real DuckDB', () => {
|
||||
expect(String(evidence.duckdb_version ?? '')).toMatch(/^\d+\.\d+/);
|
||||
});
|
||||
|
||||
it('PINS the finding: allowed_directories alone read /etc/passwd', () => {
|
||||
// The load-bearing assertion of this whole file. If someone "simplifies"
|
||||
// the bootstrap down to the allow-list, the module comment would become a
|
||||
// lie quietly; this keeps the measurement attached to the build.
|
||||
const naive = evidence.naive_allowed_directories_is_a_noop as unknown as {
|
||||
read_etc_passwd_succeeded: boolean;
|
||||
statements: string[];
|
||||
};
|
||||
expect(naive).toBeDefined();
|
||||
expect(naive.read_etc_passwd_succeeded).toBe(true);
|
||||
expect(naive.statements.join(' ')).not.toContain('enable_external_access');
|
||||
});
|
||||
|
||||
it('records that NO bypass attempt succeeded with the full bootstrap', () => {
|
||||
const attempts = evidence.sandbox_bypass_attempts as unknown as Array<{ id: string; succeeded: boolean }>;
|
||||
expect(Array.isArray(attempts)).toBe(true);
|
||||
expect(attempts.length).toBeGreaterThanOrEqual(10);
|
||||
expect(attempts.filter((a) => a.succeeded).map((a) => a.id)).toEqual([]);
|
||||
});
|
||||
|
||||
it('records the counterfactual — the mounted file WAS readable in the same session', () => {
|
||||
// Without this, "no attempt succeeded" is equally consistent with a
|
||||
// sandbox that blocks everything, which is an outage, not a control.
|
||||
expect(evidence.counterfactual_mounted_file_readable).toBe(true);
|
||||
});
|
||||
|
||||
it('records the Iceberg snapshot read, whose fixture is deliberately not committed', () => {
|
||||
// pyiceberg embeds ABSOLUTE paths in its manifests, so a committed Iceberg
|
||||
// table would only work on the machine that built it. The recorded result
|
||||
// is the artefact instead.
|
||||
const iceberg = evidence.iceberg as unknown as { totals: Record<string, number> };
|
||||
expect(iceberg?.totals).toEqual({ NO: 20, SE: 40 });
|
||||
});
|
||||
|
||||
it('records that a locked session could not re-open its own sandbox', () => {
|
||||
const reopened = evidence.reconfiguration_refused as unknown as Array<{ refused: boolean }>;
|
||||
expect(reopened.length).toBe(3);
|
||||
expect(reopened.every((r) => r.refused)).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('BI-P014 — the committed fixtures', () => {
|
||||
it('ships the Parquet and CSV fixtures the live suite reads', () => {
|
||||
expect(fs.existsSync(path.join(FIXTURES, 'sales.parquet'))).toBe(true);
|
||||
expect(fs.existsSync(path.join(FIXTURES, 'customers.csv'))).toBe(true);
|
||||
});
|
||||
|
||||
it('does NOT ship the Iceberg table, because its manifests carry absolute paths', () => {
|
||||
expect(fs.existsSync(path.join(FIXTURES, 'iceberg'))).toBe(false);
|
||||
});
|
||||
});
|
||||
Vendored
+23
@@ -0,0 +1,23 @@
|
||||
# `tests/fixtures/duckdb/` — BI-P014
|
||||
|
||||
`sales.parquet` and `customers.csv` are committed: small, binary-stable, and
|
||||
portable across checkouts.
|
||||
|
||||
The **Iceberg** fixture is deliberately NOT committed. `pyiceberg` writes
|
||||
ABSOLUTE paths into its metadata and Avro manifests, so a committed table only
|
||||
resolves on the machine that built it — a fixture that silently fails
|
||||
everywhere else is worse than no fixture. The recorded result of a real
|
||||
`iceberg_scan` run is kept in `evidence/bi-p014/duckdb-sandbox.json` instead,
|
||||
and `tests/analytics/duckdb-provider.test.ts` pins it.
|
||||
|
||||
To rebuild the Iceberg fixture locally:
|
||||
|
||||
```
|
||||
python -m venv .venv && .venv/bin/pip install duckdb pyarrow 'pyiceberg[sql-sqlite]'
|
||||
DUCKDB_PYTHON=.venv/bin/python pnpm vitest run tests/integration/duckdb-provider.integration.test.ts
|
||||
```
|
||||
|
||||
`iceberg_scan` is pointed at an EXACT metadata JSON file rather than a table
|
||||
directory. That is not a workaround: DuckDB refuses to glob for the latest
|
||||
version by default because globbing can read uncommitted data, and an exact
|
||||
metadata file is precisely what "Iceberg **snapshot**" means in BI-P014.
|
||||
Vendored
+4
@@ -0,0 +1,4 @@
|
||||
customer_id,name
|
||||
c1,Acme
|
||||
c2,Beta AB
|
||||
c3,O'Brien Ltd
|
||||
|
Vendored
BIN
Binary file not shown.
@@ -0,0 +1,384 @@
|
||||
/**
|
||||
* tests/integration/duckdb-provider.integration.test.ts
|
||||
* ======================================================
|
||||
* BI-P014 — the DuckDB provider against a real DuckDB.
|
||||
*
|
||||
* Two things are proven here that a unit test cannot:
|
||||
*
|
||||
* 1. BI-P014's DoD — "Queries work against Parquet / Arrow / Iceberg snapshot
|
||||
* fixtures" — for the formats this environment can actually build. Parquet
|
||||
* and CSV run from committed fixtures. The Iceberg case runs when a Python
|
||||
* interpreter carrying `pyiceberg` is pointed at by PYICEBERG_PYTHON,
|
||||
* because pyiceberg embeds ABSOLUTE paths in its manifests, so a committed
|
||||
* Iceberg table would only work from the machine that made it. The recorded
|
||||
* result of a real Iceberg run is committed as evidence instead, and
|
||||
* tests/analytics/duckdb-provider.test.ts pins it.
|
||||
*
|
||||
* 2. The security claim, WITH LAYER 1 REMOVED. Every bypass attempt is sent
|
||||
* straight to DuckDB, skipping the mount registry entirely, and layer 2 —
|
||||
* the session sandbox — must refuse it. Which matters more than usual here,
|
||||
* because layer 2 does not work the way it looks like it works: see the
|
||||
* `allowed_directories` finding in duckdb-provider.ts.
|
||||
*
|
||||
* Gated on DUCKDB_PYTHON, an interpreter with the `duckdb` package. With it
|
||||
* set and DuckDB broken, this suite FAILS; it does not degrade to a skip.
|
||||
*/
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import { beforeAll, describe, expect, it } from 'vitest';
|
||||
import { SiaxError } from '@/src/platform/errors';
|
||||
import {
|
||||
ArtifactMountRegistry,
|
||||
DuckDbAnalyticsProvider,
|
||||
duckDbSessionBootstrap,
|
||||
type DuckDbRunner,
|
||||
} from '@/src/analytics/providers/duckdb-provider';
|
||||
|
||||
const PY = process.env.DUCKDB_PYTHON ?? '';
|
||||
/** An interpreter carrying pyiceberg, used to REBUILD the non-portable fixture. */
|
||||
const PY_ICEBERG = process.env.PYICEBERG_PYTHON ?? PY;
|
||||
const suite = PY ? describe : describe.skip;
|
||||
const FIXTURES = path.join(process.cwd(), 'tests/fixtures/duckdb');
|
||||
const EVIDENCE_PATH = path.join(process.cwd(), 'evidence/bi-p014/duckdb-sandbox.json');
|
||||
const IDENTITY = { tenant_id: 't_alpha', principal_id: 'u_1' };
|
||||
|
||||
/**
|
||||
* Runs statements through a real DuckDB. The bootstrap is applied FIRST and in
|
||||
* order, exactly as the provider emits it.
|
||||
*/
|
||||
function runDuck(
|
||||
bootstrap: readonly string[],
|
||||
sql: string,
|
||||
loadIceberg = false,
|
||||
): { ok: boolean; out: string } {
|
||||
const script = [
|
||||
'import duckdb, json, sys',
|
||||
'c = duckdb.connect()',
|
||||
...(loadIceberg ? ['c.execute("INSTALL iceberg; LOAD iceberg")'] : []),
|
||||
// The bootstrap is inside the try as well: a statement that tries to
|
||||
// RE-OPEN a locked sandbox throws, and that throw is a RESULT to report,
|
||||
// not a crash of the harness. The first version of this file let it kill
|
||||
// the subprocess, which turned a meaningful refusal into an unreadable
|
||||
// "Command failed".
|
||||
'try:',
|
||||
...bootstrap.map((s) => ` c.execute(${JSON.stringify(s)})`),
|
||||
` rows = c.execute(${JSON.stringify(sql)}).fetchall()`,
|
||||
' print(json.dumps({"ok": True, "rows": [[str(v) for v in r] for r in rows]}))',
|
||||
'except Exception as e:',
|
||||
' print(json.dumps({"ok": False, "error": str(e)[:300]}))',
|
||||
].join('\n');
|
||||
const out = execFileSync(PY, ['-c', script], { encoding: 'utf8', timeout: 60_000 });
|
||||
const parsed = JSON.parse(out.trim()) as { ok: boolean; rows?: string[][]; error?: string };
|
||||
return { ok: parsed.ok, out: JSON.stringify(parsed.rows ?? parsed.error) };
|
||||
}
|
||||
|
||||
const runner: DuckDbRunner = {
|
||||
engine: 'duckdb',
|
||||
async run({ bootstrap, sql }) {
|
||||
const result = runDuck(bootstrap, sql, sql.includes('iceberg_scan'));
|
||||
if (!result.ok)
|
||||
throw new SiaxError('PERMANENT_FAILURE', 'duckdb', `query failed: ${result.out.slice(0, 160)}`);
|
||||
return (JSON.parse(result.out) as string[][]).map((r, i) => ({ _row: i, values: r }));
|
||||
},
|
||||
};
|
||||
|
||||
const evidence: Record<string, unknown> = {};
|
||||
let registry: ArtifactMountRegistry;
|
||||
let provider: DuckDbAnalyticsProvider;
|
||||
|
||||
beforeAll(() => {
|
||||
if (!PY) return;
|
||||
const version = execFileSync(PY, ['-c', 'import duckdb; print(duckdb.__version__)'], {
|
||||
encoding: 'utf8',
|
||||
}).trim();
|
||||
evidence.duckdb_version = version;
|
||||
evidence.recorded_at = new Date().toISOString();
|
||||
|
||||
expect(fs.existsSync(path.join(FIXTURES, 'sales.parquet')), 'parquet fixture is missing').toBe(true);
|
||||
|
||||
// Rebuild the Iceberg fixture, which is deliberately not committed:
|
||||
// pyiceberg writes ABSOLUTE paths into its metadata and Avro manifests, so a
|
||||
// committed table resolves only on the machine that made it.
|
||||
if (PY_ICEBERG && !fs.existsSync(path.join(FIXTURES, 'iceberg_metadata_location.txt'))) {
|
||||
const build = [
|
||||
'import os, shutil, pyarrow as pa',
|
||||
'from pyiceberg.catalog.sql import SqlCatalog',
|
||||
`root = ${JSON.stringify(FIXTURES)}`,
|
||||
"wh = os.path.join(root, 'iceberg')",
|
||||
'shutil.rmtree(wh, ignore_errors=True); os.makedirs(wh)',
|
||||
"cat = SqlCatalog('local', **{'uri': f'sqlite:///{wh}/catalog.db', 'warehouse': f'file://{wh}'})",
|
||||
"cat.create_namespace_if_not_exists('bi')",
|
||||
"d = pa.table({'tenant_id': ['t_alpha']*3, 'country': ['SE','NO','SE'], 'amount': [10.0,20.0,30.0]})",
|
||||
"t = cat.create_table('bi.sales_iceberg', schema=d.schema); t.append(d)",
|
||||
"open(os.path.join(root,'iceberg_metadata_location.txt'),'w').write(t.metadata_location)",
|
||||
"print('built')",
|
||||
].join('\n');
|
||||
try {
|
||||
execFileSync(PY_ICEBERG, ['-c', build], { encoding: 'utf8', timeout: 120_000 });
|
||||
} catch {
|
||||
// pyiceberg absent. The Iceberg case then falls back to the committed
|
||||
// evidence, and it asserts against it rather than passing vacuously.
|
||||
}
|
||||
}
|
||||
|
||||
registry = new ArtifactMountRegistry(FIXTURES);
|
||||
registry.register({
|
||||
artifact_id: 'a_sales_parquet',
|
||||
tenant_id: 't_alpha',
|
||||
st0re_ref: 'st0re://t_alpha/sales.parquet',
|
||||
relative_path: 'sales.parquet',
|
||||
format: 'parquet',
|
||||
size_bytes: fs.statSync(path.join(FIXTURES, 'sales.parquet')).size,
|
||||
});
|
||||
registry.register({
|
||||
artifact_id: 'a_customers_csv',
|
||||
tenant_id: 't_alpha',
|
||||
st0re_ref: 'st0re://t_alpha/customers.csv',
|
||||
relative_path: 'customers.csv',
|
||||
format: 'csv',
|
||||
size_bytes: fs.statSync(path.join(FIXTURES, 'customers.csv')).size,
|
||||
});
|
||||
// Another tenant's artifact, so the cross-tenant refusal is a real refusal
|
||||
// and not "the id does not exist".
|
||||
registry.register({
|
||||
artifact_id: 'a_beta_parquet',
|
||||
tenant_id: 't_beta',
|
||||
st0re_ref: 'st0re://t_beta/sales.parquet',
|
||||
relative_path: 'sales.parquet',
|
||||
format: 'parquet',
|
||||
size_bytes: 1,
|
||||
});
|
||||
|
||||
provider = new DuckDbAnalyticsProvider(registry, runner);
|
||||
}, 60_000);
|
||||
|
||||
suite('BI-P014 DoD — queries work against the fixtures', () => {
|
||||
it('aggregates a Parquet fixture through the closed request', async () => {
|
||||
const rows = await provider.query(IDENTITY, {
|
||||
artifact_id: 'a_sales_parquet',
|
||||
group_by: ['country'],
|
||||
aggregate: { fn: 'sum', column: 'amount', alias: 'total' },
|
||||
});
|
||||
const totals = Object.fromEntries(
|
||||
rows.map((r) => {
|
||||
const [country, total] = (r as { values: string[] }).values;
|
||||
return [country, Number(total)];
|
||||
}),
|
||||
);
|
||||
expect(totals).toEqual({ SE: 394.5, NO: 105.5 });
|
||||
evidence.parquet = totals;
|
||||
});
|
||||
|
||||
it('reads a CSV fixture, including a value containing a quote', async () => {
|
||||
const rows = await provider.query(IDENTITY, {
|
||||
artifact_id: 'a_customers_csv',
|
||||
select: ['customer_id', 'name'],
|
||||
});
|
||||
const names = rows.map((r) => (r as { values: string[] }).values[1]);
|
||||
expect(names).toContain("O'Brien Ltd");
|
||||
evidence.csv = { rows: rows.length, includes_quoted_value: true };
|
||||
});
|
||||
|
||||
it('filters with a hostile value without it becoming SQL', async () => {
|
||||
const rows = await provider.query(IDENTITY, {
|
||||
artifact_id: 'a_customers_csv',
|
||||
select: ['customer_id'],
|
||||
filters: { name: "O'Brien Ltd" },
|
||||
});
|
||||
expect(rows).toHaveLength(1);
|
||||
|
||||
// The injection attempt: a value that would close the literal and append.
|
||||
const hostile = await provider.query(IDENTITY, {
|
||||
artifact_id: 'a_customers_csv',
|
||||
select: ['customer_id'],
|
||||
filters: { name: "x' OR '1'='1" },
|
||||
});
|
||||
// It matched nothing, because it was DATA. Had the escaping failed, the
|
||||
// OR would have matched all three rows.
|
||||
expect(hostile).toHaveLength(0);
|
||||
evidence.injection_attempt = { matched_rows: hostile.length, note: 'escaped as data; 0 rows, not 3' };
|
||||
});
|
||||
|
||||
it('reads an ICEBERG SNAPSHOT by its exact metadata file', async () => {
|
||||
const metaFile = path.join(FIXTURES, 'iceberg_metadata_location.txt');
|
||||
if (!fs.existsSync(metaFile)) {
|
||||
// No pyiceberg here. Carry the previously RECORDED result forward, marked
|
||||
// as carried forward, and assert against it — the first version of this
|
||||
// branch was `expect(evidence.iceberg ?? "unbuilt").toBeDefined()`, which
|
||||
// can never fail and quietly erased the recorded block from the evidence
|
||||
// file while both suites stayed green.
|
||||
const prior = fs.existsSync(EVIDENCE_PATH)
|
||||
? (JSON.parse(fs.readFileSync(EVIDENCE_PATH, 'utf8')) as {
|
||||
iceberg?: { totals: Record<string, number> };
|
||||
})
|
||||
: {};
|
||||
expect(
|
||||
prior.iceberg?.totals,
|
||||
'no pyiceberg available AND no recorded Iceberg result — the DoD is unproven, not skipped',
|
||||
).toEqual({ NO: 20, SE: 40 });
|
||||
evidence.iceberg = { ...prior.iceberg, carried_forward: true };
|
||||
return;
|
||||
}
|
||||
const meta = fs
|
||||
.readFileSync(metaFile, 'utf8')
|
||||
.trim()
|
||||
.replace(/^file:\/\//, '');
|
||||
const bootstrap = duckDbSessionBootstrap([FIXTURES]);
|
||||
const res = runDuck(
|
||||
bootstrap,
|
||||
`SELECT country, sum(amount) AS s FROM iceberg_scan('${meta}') GROUP BY country ORDER BY country`,
|
||||
true,
|
||||
);
|
||||
expect(res.ok, `iceberg_scan failed: ${res.out}`).toBe(true);
|
||||
const rows = JSON.parse(res.out) as string[][];
|
||||
expect(Object.fromEntries(rows.map((r) => [r[0], Number(r[1])]))).toEqual({ NO: 20, SE: 40 });
|
||||
evidence.iceberg = {
|
||||
read_by: 'iceberg_scan on the exact metadata JSON',
|
||||
note:
|
||||
'an exact metadata file is a SNAPSHOT reference; DuckDB refuses to glob for the latest ' +
|
||||
'version by default, and that refusal is correct — globbing can read uncommitted data',
|
||||
totals: { NO: 20, SE: 40 },
|
||||
};
|
||||
});
|
||||
});
|
||||
|
||||
suite('BI-P014 — the mount registry (layer 1)', () => {
|
||||
it('refuses an artifact belonging to another tenant, as AUTHORIZATION_DENIED not NOT_FOUND', async () => {
|
||||
let thrown: unknown;
|
||||
try {
|
||||
await provider.query(IDENTITY, { artifact_id: 'a_beta_parquet' });
|
||||
} catch (e) {
|
||||
thrown = e;
|
||||
}
|
||||
expect((thrown as SiaxError).code).toBe('AUTHORIZATION_DENIED');
|
||||
});
|
||||
|
||||
it('refuses an unmounted artifact', async () => {
|
||||
await expect(provider.query(IDENTITY, { artifact_id: 'a_nope' })).rejects.toBeInstanceOf(SiaxError);
|
||||
});
|
||||
|
||||
it('refuses a SYMLINK that escapes the mount root', () => {
|
||||
const link = path.join(FIXTURES, 'escape.parquet');
|
||||
fs.rmSync(link, { force: true });
|
||||
fs.symlinkSync('/etc/passwd', link);
|
||||
try {
|
||||
expect(() =>
|
||||
registry.register({
|
||||
artifact_id: 'a_escape',
|
||||
tenant_id: 't_alpha',
|
||||
st0re_ref: 'st0re://x',
|
||||
relative_path: 'escape.parquet',
|
||||
format: 'parquet',
|
||||
size_bytes: 1,
|
||||
}),
|
||||
).toThrowError(SiaxError);
|
||||
evidence.symlink_escape_refused = true;
|
||||
} finally {
|
||||
fs.rmSync(link, { force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('refuses a traversing relative path', () => {
|
||||
expect(() =>
|
||||
registry.register({
|
||||
artifact_id: 'a_trav',
|
||||
tenant_id: 't_alpha',
|
||||
st0re_ref: 'st0re://x',
|
||||
relative_path: '../../../../etc/passwd',
|
||||
format: 'csv',
|
||||
size_bytes: 1,
|
||||
}),
|
||||
).toThrowError(SiaxError);
|
||||
});
|
||||
});
|
||||
|
||||
suite('BI-P014 — layer 2 alone, WITH THE MOUNT REGISTRY REMOVED', () => {
|
||||
/** Attempts sent straight to DuckDB. Layer 1 is not in the picture. */
|
||||
const attempts: Array<{ id: string; sql: string }> = [
|
||||
{ id: 'read-etc-passwd', sql: "SELECT * FROM read_csv('/etc/passwd')" },
|
||||
{ id: 'read-etc-shadow', sql: "SELECT * FROM read_csv('/etc/shadow')" },
|
||||
{ id: 'traversal-from-mount', sql: `SELECT * FROM read_csv('${FIXTURES}/../../../../etc/passwd')` },
|
||||
{ id: 'glob-root', sql: "SELECT * FROM read_csv('/etc/*')" },
|
||||
{ id: 'http-exfiltration', sql: "SELECT * FROM read_csv('https://example.com/a.csv')" },
|
||||
{ id: 's3-read', sql: "SELECT * FROM read_parquet('s3://bucket/a.parquet')" },
|
||||
// Not a SELECT: ATTACH is the other way a local engine touches a file.
|
||||
// The first version of this corpus had `SELECT 1 FROM (SELECT 1) t` here,
|
||||
// which is not an attack at all — it succeeded and was reported as a leak.
|
||||
// A corpus entry that cannot fail is as useless as a check that cannot.
|
||||
{ id: 'attach-external-db', sql: "ATTACH '/tmp/duckdb_escape.db' AS evil" },
|
||||
{ id: 'attach-etc', sql: "ATTACH '/etc/passwd' AS evil2 (READ_ONLY)" },
|
||||
{ id: 'copy-out-of-mount', sql: "COPY (SELECT 1 AS a) TO '/tmp/duckdb_escape.csv' (FORMAT CSV)" },
|
||||
{ id: 'read-parquet-outside', sql: "SELECT * FROM read_parquet('/root/x.parquet')" },
|
||||
{ id: 'json-outside', sql: "SELECT * FROM read_json_auto('/etc/hosts')" },
|
||||
];
|
||||
|
||||
it('no attempt reads anything outside the mount, and the file inside still works', () => {
|
||||
const bootstrap = duckDbSessionBootstrap([FIXTURES]);
|
||||
const results: Array<Record<string, unknown>> = [];
|
||||
const leaks: string[] = [];
|
||||
|
||||
for (const attempt of attempts) {
|
||||
const res = runDuck(bootstrap, attempt.sql);
|
||||
// A leak is a SUCCESSFUL read. `s3://` and `https://` may fail for other
|
||||
// reasons too, but the assertion is on success, not on the message.
|
||||
if (res.ok) leaks.push(attempt.id);
|
||||
results.push({ id: attempt.id, succeeded: res.ok, engine_message: res.out.slice(0, 140) });
|
||||
}
|
||||
|
||||
// The counterfactual: the SAME session reads the mounted file fine, so
|
||||
// "nothing succeeded" is not because the sandbox blocks everything.
|
||||
const inside = runDuck(bootstrap, `SELECT count(*) FROM read_parquet('${FIXTURES}/sales.parquet')`);
|
||||
expect(inside.ok, 'the sandbox blocked the mounted file too — it is not a sandbox, it is an outage').toBe(
|
||||
true,
|
||||
);
|
||||
expect(JSON.parse(inside.out)[0][0]).toBe('5');
|
||||
|
||||
expect(leaks, `DuckDB read outside the mount: ${leaks.join(', ')}`).toEqual([]);
|
||||
evidence.sandbox_bypass_attempts = results;
|
||||
evidence.counterfactual_mounted_file_readable = true;
|
||||
}, 120_000);
|
||||
|
||||
it('the NAIVE spelling of the sandbox is a NO-OP — measured, not assumed', () => {
|
||||
// allowed_directories WITHOUT enable_external_access=false. This is what
|
||||
// an implementer reaches for first, and it permits everything.
|
||||
const naive = [`SET allowed_directories=['${FIXTURES}']`];
|
||||
const res = runDuck(naive, "SELECT * FROM read_csv('/etc/passwd') LIMIT 1");
|
||||
expect(res.ok, 'if this ever fails, DuckDB changed its behaviour and the header comment is stale').toBe(
|
||||
true,
|
||||
);
|
||||
evidence.naive_allowed_directories_is_a_noop = {
|
||||
statements: naive,
|
||||
read_etc_passwd_succeeded: res.ok,
|
||||
note:
|
||||
'allowed_directories alone appears in duckdb_settings() and looks like a sandbox. ' +
|
||||
'It permits /etc/passwd and outbound HTTP. Only enable_external_access=false binds it.',
|
||||
};
|
||||
});
|
||||
|
||||
it('a locked session cannot re-open its own sandbox', () => {
|
||||
const bootstrap = duckDbSessionBootstrap([FIXTURES]);
|
||||
// Sanity: the locked session works.
|
||||
expect(runDuck(bootstrap, 'SELECT 1').ok).toBe(true);
|
||||
|
||||
const reopened: Array<Record<string, unknown>> = [];
|
||||
for (const reopen of [
|
||||
"SET allowed_directories=['/etc']",
|
||||
'SET enable_external_access=true',
|
||||
'SET lock_configuration=false',
|
||||
]) {
|
||||
// Appending the re-open AFTER lock_configuration=true must fail, and it
|
||||
// must fail on the SET itself — so the following read never happens.
|
||||
const attempt = runDuck([...bootstrap, reopen], "SELECT * FROM read_csv('/etc/passwd') LIMIT 1");
|
||||
expect(attempt.ok, `"${reopen}" was accepted after lock_configuration=true`).toBe(false);
|
||||
reopened.push({ statement: reopen, refused: !attempt.ok, engine_message: attempt.out.slice(0, 120) });
|
||||
}
|
||||
evidence.reconfiguration_refused = reopened;
|
||||
}, 60_000);
|
||||
|
||||
it('writes the evidence file', () => {
|
||||
fs.mkdirSync(path.dirname(EVIDENCE_PATH), { recursive: true });
|
||||
fs.writeFileSync(EVIDENCE_PATH, `${JSON.stringify(evidence, null, 2)}\n`);
|
||||
expect(fs.existsSync(EVIDENCE_PATH)).toBe(true);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user