From f5823cc4d0d7c6216d17c47911b7ad5a9a520462 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Fri, 2 Oct 2026 23:00:31 +0000 Subject: [PATCH 1/3] Initial plan From 0c1c3a176cbfd0937c198f339dc094b2a3eb649c Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Fri, 2 Oct 2026 23:15:32 +0000 Subject: [PATCH 2/3] fix: enforce invocation-private Cloud Hypervisor enclave storage bounds --- docs/cloud-hypervisor-foundation.md | 56 +++++- .../enclave-storage.integration.test.ts | 187 ++++++++++++++++++ src/cloud-hypervisor/enclave-storage.test.ts | 133 +++++++++++++ src/cloud-hypervisor/enclave-storage.ts | 143 ++++++++++++++ .../host-enclave-executor.test.ts | 19 ++ src/cloud-hypervisor/host-enclave-executor.ts | 30 +-- src/cloud-hypervisor/launcher.test.ts | 31 +++ src/cloud-hypervisor/launcher.ts | 10 +- src/cloud-hypervisor/manager-launch.test.ts | 28 +++ src/cloud-hypervisor/manager-start.ts | 1 + src/cloud-hypervisor/virtiofsd.test.ts | 33 ++-- src/cloud-hypervisor/virtiofsd.ts | 43 +--- 12 files changed, 628 insertions(+), 86 deletions(-) create mode 100644 src/cloud-hypervisor/enclave-storage.integration.test.ts create mode 100644 src/cloud-hypervisor/enclave-storage.test.ts create mode 100644 src/cloud-hypervisor/enclave-storage.ts diff --git a/docs/cloud-hypervisor-foundation.md b/docs/cloud-hypervisor-foundation.md index ceed3e48d..b2c0bfcd6 100644 --- a/docs/cloud-hypervisor-foundation.md +++ b/docs/cloud-hypervisor-foundation.md @@ -257,13 +257,55 @@ privileges through executable metadata. Missing mounts, unsupported kernel controls, or verification mismatches abort startup rather than launching without a limit. -Writable virtio-fs storage is accepted only when every writable export is on -the same host filesystem and that filesystem's full capacity is no greater than -the role ceiling. This deliberately strict check fails on typical larger runner -filesystems; the future host executor must provide a genuinely size-bounded -per-invocation backing filesystem or remain unavailable. It is not a per-folder -quota and the guest cannot use `size=` to limit virtio-fs. Tmpfs ceilings are -independent of this host filesystem ceiling. +The trusted host executor mounts one invocation-private Linux **tmpfs** at the +host-derived invocation directory, with `size=1073741824` for scripts (1 GiB) or +`size=536870912` for agents (512 MiB), and `mode=0700,nosuid,nodev,noexec`. +The closed `/output`, `/runtime`, and agent `/session-state` writable virtio-fs +exports are subdirectories of that single backing store, not separate tmpfs +mounts. Request and handoff files also consume its budget. Linux charges allocated +pages across all exports atomically, including concurrent writes and writes into +sparse-file holes; allocation beyond the aggregate ceiling returns `ENOSPC`. +Sparse logical lengths do not allocate pages and do not bypass the allocation +limit. No disk quota, loop device, or guest-only `size=` limit is required, so +the mechanism uses the supported GitHub-hosted Linux/KVM runner's existing +mount/virtio-fs path. Guest-internal tmpfs ceilings remain separate. + +Host tmpfs pages are charged to the writing virtio-fs process's memory cgroup. +The enclave-only host `memory.max` therefore includes the fixed storage ceiling +in addition to 768 MiB guest RAM and the existing 256 MiB VMM overhead: +2 GiB for scripts and 1.5 GiB for agents. This avoids preempting the storage +ceiling with the old shared-cgroup budget; it does not enlarge guest RAM or add +a configurable resource limit. Host memory exhaustion can still terminate an +invocation rather than return `ENOSPC`, and is treated as executor failure. +Primary-agent cgroup budgets are unchanged. + +Before staging inputs and again before starting virtio-fs daemons, AWF verifies +canonical export paths, the exact invocation mount in `/proc/self/mountinfo`, +its tmpfs type, private mount identity, mount options, and exact role capacity +from `statfs`. This is verification of a kernel-enforced backing store, not +a free-space preflight. Missing, undersized, oversized, aliased, nested, or +unverifiable mounts abort startup. There are no caller-selectable storage paths, +sizes, classes, overrides, or fallback to the runner filesystem. + +The existing durable host-executor resource journal records the underlying +directory before mounting and captures the tmpfs mount identity before use. +Completion, timeout, cancellation, and partial startup wait for outstanding +provisioning and VM/virtio-fs teardown before ordinary (never lazy) unmount and +directory removal. Abandoned-run recovery first reaps the VM cleanup record, +then verifies recorded directory/mount ownership and removes invocation storage. +Unmount or ownership-verification failure retains the recovery record, reports +incomplete cleanup, and keeps admissions closed; it never deletes through a +live mount. + +`src/cloud-hypervisor/enclave-storage.integration.test.ts` exercises the actual +host backing paths served by the closed writable exports, including role-sized +aggregate `ENOSPC`, sparse and concurrent writes, invocation isolation, and busy +unmount failure. It also fills storage in the production host cgroup budget while +holding the guest-RAM equivalent resident, with swap disabled, to check that +storage exhaustion is not preempted by a cgroup OOM. Run it as root in a private mount namespace with +`AWF_TEST_ENCLAVE_STORAGE=1 npm test -- --runInBand enclave-storage.integration.test.ts`. +Live guest transport conformance additionally requires the release-attested role +artifacts and KVM runtime wiring. The rootfs build removes package-manager executables, the importable `pip` and `ensurepip` modules (so `python3 -m pip` cannot run or be recreated), diff --git a/src/cloud-hypervisor/enclave-storage.integration.test.ts b/src/cloud-hypervisor/enclave-storage.integration.test.ts new file mode 100644 index 000000000..ef84a6bae --- /dev/null +++ b/src/cloud-hypervisor/enclave-storage.integration.test.ts @@ -0,0 +1,187 @@ +import { promises as fs } from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import execa from 'execa'; +import { + assertBoundedEnclaveWritableExports, + mountBoundedEnclaveStorage, + unmountBoundedEnclaveStorage, +} from './enclave-storage'; +import { CLOUD_HYPERVISOR_ENCLAVE_RESOURCE_PROFILES as profiles } from './workload-profile'; +import type { CloudHypervisorDirectoryExport } from './exports'; +import { CloudHypervisorCgroup } from './launcher'; + +const live = process.env.AWF_TEST_ENCLAVE_STORAGE === '1'; +const tools = { mount: '/usr/bin/mount', umount: '/usr/bin/umount' }; +const page = Buffer.alloc(4096, 1); + +// Exercise the host sources served by the closed guest-visible virtio-fs +// exports. Live VM transport conformance is gated separately on KVM artifacts. +(live ? describe : describe.skip)('host-enforced enclave storage', () => { + let root: string; + const mounted = new Set(); + + beforeAll(async () => { + if (process.getuid?.() !== 0) throw new Error('AWF_TEST_ENCLAVE_STORAGE requires root and mount privileges'); + root = await fs.mkdtemp(path.join(os.tmpdir(), 'awf-enclave-storage-')); + }); + + afterEach(async () => { + for (const directory of [...mounted].reverse()) { + await unmountBoundedEnclaveStorage(directory, tools); + mounted.delete(directory); + } + await fs.rm(root, { recursive: true, force: true }); + root = await fs.mkdtemp(path.join(os.tmpdir(), 'awf-enclave-storage-')); + }); + + afterAll(async () => { + await fs.rm(root, { recursive: true, force: true }); + }); + + async function provision(role: 'script' | 'agent', name: string) { + const directory = path.join(root, name); + await fs.mkdir(directory, { mode: 0o700 }); + const profile = profiles[role]; + await mountBoundedEnclaveStorage( + directory, profile.writableStorageBytes, profile.uid, profile.gid, tools, + ); + mounted.add(directory); + const names = ['output', 'runtime', ...(role === 'agent' ? ['session-state'] : [])]; + const exports: CloudHypervisorDirectoryExport[] = []; + for (const name of names) { + const source = path.join(directory, name); + await fs.mkdir(source, { mode: 0o700 }); + exports.push({ tag: `enclave-${name}`, source, target: `/${name}`, mode: 'rw' }); + } + await assertBoundedEnclaveWritableExports(exports, profile.writableStorageBytes); + return { directory, exports, size: profile.writableStorageBytes }; + } + + it.each(['script', 'agent'] as const)( + '%s writes exhaust one aggregate budget across sparse and concurrent exports', + async (role) => { + const { exports, size } = await provision(role, role); + const files = await Promise.all(exports.map(({ source }) => fs.open(path.join(source, 'sparse'), 'wx'))); + try { + // Logical holes are not allocated storage. Writes into them must still + // consume the same page budget, even far beyond the capacity offset. + await Promise.all(files.map((file) => file.truncate(size * 4))); + const chunk = Buffer.alloc(1024 * 1024, 1); + const pagesPerChunk = chunk.length / page.length; + const chunksPerFile = Math.floor(size / chunk.length / files.length); + await Promise.all(files.map(async (file) => { + for (let chunkIndex = 0; chunkIndex < chunksPerFile; chunkIndex += 1) { + await file.write(chunk, 0, chunk.length, size * 2 + chunkIndex * chunk.length); + } + })); + const allocated = chunksPerFile * files.length * chunk.length; + const remainingPages = (size - allocated) / page.length; + for (let index = 0; index < remainingPages; index += 1) { + await files[0].write(page, 0, page.length, + size * 2 + (chunksPerFile * pagesPerChunk + index) * page.length); + } + const attempts = await Promise.all(files.map((file) => file.write(page, 0, page.length, 0) + .then(() => 'unexpected success', (error: NodeJS.ErrnoException) => error.code))); + expect(attempts).toEqual(files.map(() => 'ENOSPC')); + const stats = await Promise.all(files.map((file) => file.stat())); + expect(stats.reduce((total, stat) => total + stat.blocks * 512, 0)).toBe(size); + } finally { + await Promise.all(files.map((file) => file.close())); + } + }, 120_000, + ); + + it('isolates invocations and rejects an export redirected through another mount', async () => { + const first = await provision('agent', 'first'); + const second = await provision('agent', 'second'); + await fs.writeFile(path.join(first.exports[0].source, 'private'), 'first'); + await expect(fs.readFile(path.join(second.exports[0].source, 'private'))).rejects.toMatchObject({ code: 'ENOENT' }); + expect((await fs.stat(first.directory)).dev).not.toBe((await fs.stat(second.directory)).dev); + const escaped = path.join(first.directory, 'escape'); + await fs.symlink(second.exports[0].source, escaped); + await expect(assertBoundedEnclaveWritableExports([ + { ...first.exports[0], source: escaped }, + ], first.size)).rejects.toThrow(/escapes/); + + const nested = first.exports[1].source; + await execa(tools.mount, ['--bind', second.exports[1].source, nested]); + mounted.add(nested); + await expect(assertBoundedEnclaveWritableExports(first.exports, first.size)) + .rejects.toThrow(/unverifiable/); + }); + + it.each(['script', 'agent'] as const)('reaches %s ENOSPC with resident guest RAM in the host memory cgroup', async (role) => { + const { exports, size } = await provision(role, `cgroup-${role}`); + const profile = profiles[role]; + const cgroup = new CloudHypervisorCgroup( + `/sys/fs/cgroup/awf-cloud-hypervisor/storage-${path.basename(root)}`, + { + memoryMib: profile.memoryMiB, vcpuCount: profile.vcpuCount, + cpuQuotaMilli: profile.cpuQuotaMilli, writableStorageBytes: size, + }, + ); + await cgroup.setup(); + await fs.writeFile(path.join(cgroup.cgroupPath, 'memory.swap.max'), '0'); + // A separate process holds the guest-RAM equivalent resident while filling + // tmpfs. Both charges must fit the same production host memory budget. + const writer = execa(process.execPath, ['-e', ` + const fs = require('fs'); + const { sources, size, memory } = JSON.parse(process.argv[1]); + process.stdin.once('data', () => { + const guestRam = Buffer.alloc(memory, 1); + const chunk = Buffer.alloc(1024 * 1024, 1); + const files = sources.map(source => fs.openSync(source + '/charged', 'wx')); + try { + for (let index = 0; index < size / chunk.length; index++) { + fs.writeSync(files[index % files.length], chunk, 0, chunk.length, + Math.floor(index / files.length) * chunk.length); + } + for (const file of files) { + try { + fs.writeSync(file, chunk, 0, 4096, size); + throw new Error('storage was not bounded'); + } catch (error) { + if (error.code !== 'ENOSPC') throw error; + } + } + if (guestRam[guestRam.length - 1] !== 1) throw new Error('guest RAM missing'); + console.log('ENOSPC'); + } finally { + files.forEach(file => fs.closeSync(file)); + } + }); + `, JSON.stringify({ + sources: exports.map(({ source }) => source), size, memory: profile.memoryMiB * 1024 * 1024, + })], { stdio: ['pipe', 'pipe', 'pipe'] }); + try { + if (writer.pid === undefined) throw new Error('storage writer did not start'); + await cgroup.assign(writer.pid); + writer.stdin!.end('start'); + expect((await writer).stdout.trim()).toBe('ENOSPC'); + expect(await fs.readFile(path.join(cgroup.cgroupPath, 'memory.events'), 'utf8')) + .toMatch(/^oom_kill 0$/m); + expect(BigInt((await fs.readFile(path.join(cgroup.cgroupPath, 'memory.peak'), 'utf8')).trim())) + .toBeGreaterThan(BigInt((profile.memoryMiB + 256) * 1024 * 1024)); + } finally { + writer.kill('SIGKILL'); + await writer.catch(() => undefined); + await cgroup.cleanup(); + } + }, 120_000); + + it('does not report successful cleanup or remove a busy backing store', async () => { + const { directory, exports } = await provision('agent', 'busy'); + const handle = await fs.open(path.join(exports[0].source, 'open'), 'wx'); + try { + await expect(unmountBoundedEnclaveStorage(directory, tools)).rejects.toThrow(/unmount/); + await assertBoundedEnclaveWritableExports(exports, profiles.agent.writableStorageBytes); + } finally { + await handle.close(); + } + await unmountBoundedEnclaveStorage(directory, tools); + mounted.delete(directory); + await fs.rm(directory, { recursive: true }); + await expect(fs.lstat(directory)).rejects.toMatchObject({ code: 'ENOENT' }); + }); +}); diff --git a/src/cloud-hypervisor/enclave-storage.test.ts b/src/cloud-hypervisor/enclave-storage.test.ts new file mode 100644 index 000000000..3d03f3747 --- /dev/null +++ b/src/cloud-hypervisor/enclave-storage.test.ts @@ -0,0 +1,133 @@ +import { + assertBoundedEnclaveStorage, + assertBoundedEnclaveWritableExports, + mountBoundedEnclaveStorage, + type EnclaveStorageDependencies, +} from './enclave-storage'; +import { CLOUD_HYPERVISOR_ENCLAVE_RESOURCE_PROFILES as profiles } from './workload-profile'; + +const directory = '/trusted/invocations/script/0123456789abcdef0123456789abcdef'; +const mountInfo = (point = directory, device = '0:42') => + `30 29 ${device} / ${point} rw,nosuid,nodev,noexec - tmpfs awf-enclave-invocation rw,size=1048576k`; + +function dependencies( + size = profiles.script.writableStorageBytes, + overrides: Partial = {}, +): EnclaveStorageDependencies { + return { + realpath: jest.fn(async (value) => value), + lstat: jest.fn().mockResolvedValue({ uid: 65534, gid: 65534, mode: 0o40700 }), + readMountInfo: jest.fn().mockResolvedValue(mountInfo()), + statfs: jest.fn().mockResolvedValue({ + type: 0x01021994n, blocks: BigInt(size / 4096), bsize: 4096n, + }), + ...overrides, + }; +} + +describe('bounded enclave storage verification', () => { + it.each(['script', 'agent'] as const)('accepts only the exact %s aggregate budget', async (role) => { + const size = profiles[role].writableStorageBytes; + const deps = dependencies(size); + const exports = ['output', 'runtime', ...(role === 'agent' ? ['session-state'] : [])] + .map((name) => `${directory}/${name}`); + await expect(assertBoundedEnclaveStorage(directory, size, exports, deps)).resolves.toBeUndefined(); + expect(deps.statfs).toHaveBeenCalledTimes(exports.length + 1); + expect(deps.readMountInfo).toHaveBeenCalledTimes(1); + }); + + it.each([0, -1, 4096, NaN, Infinity])('rejects non-role size %s', async (size) => { + await expect(assertBoundedEnclaveStorage(directory, size, [], dependencies())) + .rejects.toThrow(/role capacity/); + }); + + it.each([ + ['missing mount', ''], + ['host filesystem', mountInfo().replace('tmpfs', 'ext4')], + ['foreign mount', mountInfo().replace('awf-enclave-invocation', 'tmpfs')], + ['bind mount root', mountInfo().replace(' / ', ' /subdirectory ')], + ['stacked mount', `${mountInfo()}\n${mountInfo()}`], + ['nested mount', `${mountInfo()}\n${mountInfo(`${directory}/runtime`, '0:43')}`], + ['external alias', `${mountInfo()}\n${mountInfo('/another/invocation')}`], + ['missing noexec', mountInfo().replace(',noexec', '')], + ['read-only mount', mountInfo().replace('rw,nosuid', 'ro,nosuid')], + ])('rejects %s rather than falling back', async (_label, info) => { + await expect(assertBoundedEnclaveStorage(directory, profiles.script.writableStorageBytes, [], dependencies( + undefined, { readMountInfo: async () => info }, + ))).rejects.toThrow(/mount/); + }); + + it.each([512 * 1024 * 1024, 2 * 1024 * 1024 * 1024])('rejects incorrectly sized storage %s', async (size) => { + await expect(assertBoundedEnclaveStorage( + directory, profiles.script.writableStorageBytes, [], dependencies(size), + )).rejects.toThrow(/capacity/); + }); + + it('rejects non-tmpfs statfs even when capacity matches', async () => { + await expect(assertBoundedEnclaveStorage( + directory, profiles.script.writableStorageBytes, [], dependencies(undefined, { + statfs: async () => ({ + type: 0xef53n, blocks: 262144n, bsize: 4096n, + }), + }), + )).rejects.toThrow(/capacity/); + }); + + it.each([ + '/', 'relative/path', `${directory}/../elsewhere`, + ])('rejects untrusted invocation path %s', async (value) => { + await expect(assertBoundedEnclaveStorage( + value, profiles.script.writableStorageBytes, [], dependencies(), + )).rejects.toThrow(/identity/); + }); + + it.each([ + '/another/invocation/output', `${directory}/../output`, `${directory}/runtime/child`, + ])('rejects an export outside the closed invocation layout: %s', async (value) => { + await expect(assertBoundedEnclaveStorage( + directory, profiles.script.writableStorageBytes, [value], dependencies(), + )).rejects.toThrow(/escapes/); + }); + + it('rejects symlinked invocation directories and exports', async () => { + const deps = dependencies(undefined, { realpath: async () => '/untrusted/elsewhere' }); + await expect(assertBoundedEnclaveStorage(directory, profiles.script.writableStorageBytes, [], deps)) + .rejects.toThrow(/identity/); + await expect(assertBoundedEnclaveStorage(directory, profiles.script.writableStorageBytes, + [`${directory}/output`], dependencies(undefined, { + realpath: async (value) => value === directory ? value : '/untrusted/elsewhere', + }))).rejects.toThrow(/escapes/); + }); + + it('propagates unverifiable kernel evidence', async () => { + await expect(assertBoundedEnclaveStorage( + directory, profiles.script.writableStorageBytes, [], dependencies(undefined, { + readMountInfo: async () => { throw new Error('mountinfo unavailable'); }, + }), + )).rejects.toThrow('mountinfo unavailable'); + }); + + it.each([ + { uid: 0, gid: 65534, mode: 0o40700 }, + { uid: 65534, gid: 0, mode: 0o40700 }, + { uid: 65534, gid: 65534, mode: 0o40777 }, + ])('rejects incorrect mount ownership %j', async (identity) => { + await expect(assertBoundedEnclaveStorage( + directory, profiles.script.writableStorageBytes, [], dependencies(undefined, { + lstat: async () => identity, + }), + )).rejects.toThrow(/ownership/); + }); + + it('requires writable exports', async () => { + await expect(assertBoundedEnclaveWritableExports([], profiles.script.writableStorageBytes)) + .rejects.toThrow(/requires bounded writable exports/); + }); + + it('rejects caller-selectable sizes or identities before mounting', async () => { + await expect(mountBoundedEnclaveStorage(directory, 4096, 65534, 65534, { mount: '/usr/bin/false' })) + .rejects.toThrow(/closed role profile/); + await expect(mountBoundedEnclaveStorage(directory, profiles.script.writableStorageBytes, + 0, 0, { mount: '/usr/bin/false' })).rejects.toThrow(/closed role profile/); + }); +}); diff --git a/src/cloud-hypervisor/enclave-storage.ts b/src/cloud-hypervisor/enclave-storage.ts new file mode 100644 index 000000000..94ef5500d --- /dev/null +++ b/src/cloud-hypervisor/enclave-storage.ts @@ -0,0 +1,143 @@ +import { promises as fs } from 'fs'; +import * as path from 'path'; +import execa from 'execa'; +import { parseMountInfoLine } from './cleanup-identity'; +import type { CloudHypervisorDirectoryExport } from './exports'; +import { CLOUD_HYPERVISOR_ENCLAVE_RESOURCE_PROFILES } from './workload-profile'; + +const TMPFS_MAGIC = 0x01021994n; +export const ENCLAVE_STORAGE_SOURCE = 'awf-enclave-invocation'; + +export interface EnclaveStorageDependencies { + readonly realpath: (directory: string) => Promise; + readonly lstat: (directory: string) => Promise<{ uid: number; gid: number; mode: number }>; + readonly readMountInfo: () => Promise; + readonly statfs: (directory: string) => Promise<{ + type: bigint; blocks: bigint; bsize: bigint; + }>; +} + +const defaultDependencies: EnclaveStorageDependencies = { + realpath: fs.realpath, + lstat: fs.lstat, + readMountInfo: () => fs.readFile('/proc/self/mountinfo', 'utf8'), + statfs: (directory) => fs.statfs(directory, { bigint: true }), +}; + +export async function mountBoundedEnclaveStorage( + directory: string, + sizeBytes: number, + uid: number, + gid: number, + tools: { readonly mount: string }, +): Promise { + if (!Object.values(CLOUD_HYPERVISOR_ENCLAVE_RESOURCE_PROFILES).some((profile) => + profile.writableStorageBytes === sizeBytes && profile.uid === uid && profile.gid === gid)) { + throw new Error('Enclave storage requires a closed role profile'); + } + if (!path.isAbsolute(directory) || path.normalize(directory) !== directory || + directory === '/' || await fs.realpath(directory) !== directory) { + throw new Error('Enclave storage requires a canonical invocation directory'); + } + const mountOptions = `size=${sizeBytes},mode=0700,uid=${uid},gid=${gid},nosuid,nodev,noexec`; + const result = await execa(tools.mount, [ + '-t', 'tmpfs', + '-o', mountOptions, + ENCLAVE_STORAGE_SOURCE, + directory, + ], { reject: false, stdio: ['ignore', 'pipe', 'pipe'] }); + if (result.exitCode !== 0) { + throw new Error(`Unable to mount bounded enclave invocation storage: ${result.stderr.trim()}`); + } +} + +export async function unmountBoundedEnclaveStorage( + directory: string, + tools: { readonly umount: string }, +): Promise { + // Never detach lazily: a busy mount must retain ownership and block admission. + const result = await execa(tools.umount, [directory], { + reject: false, + stdio: ['ignore', 'pipe', 'pipe'], + }); + if (result.exitCode !== 0) { + throw new Error(`Unable to unmount enclave invocation storage: ${result.stderr.trim()}`); + } +} + +/** + * Verify the host mount, not free space or a guest tmpfs. All writable exports + * must be direct children of one invocation-owned tmpfs with the closed role + * capacity. Nested mounts (including bind mounts) cannot escape that budget. + */ +export async function assertBoundedEnclaveStorage( + invocationDirectory: string, + maximumBytes: number, + writableDirectories: readonly string[] = [], + dependencies: EnclaveStorageDependencies = defaultDependencies, +): Promise { + const profile = Object.values(CLOUD_HYPERVISOR_ENCLAVE_RESOURCE_PROFILES) + .find((profile) => profile.writableStorageBytes === maximumBytes); + if ( + !profile || + !path.isAbsolute(invocationDirectory) || + path.normalize(invocationDirectory) !== invocationDirectory || + invocationDirectory === '/' || + await dependencies.realpath(invocationDirectory) !== invocationDirectory + ) { + throw new Error('Invalid bounded enclave storage identity or role capacity'); + } + for (const directory of writableDirectories) { + if ( + path.dirname(directory) !== invocationDirectory || + path.normalize(directory) !== directory || + await dependencies.realpath(directory) !== directory + ) { + throw new Error('Writable enclave export escapes its invocation storage'); + } + } + const identity = await dependencies.lstat(invocationDirectory); + if (identity.uid !== profile.uid || identity.gid !== profile.gid || + (identity.mode & 0o7777) !== 0o700) { + throw new Error('Bounded enclave storage ownership does not match the closed role profile'); + } + const lines = (await dependencies.readMountInfo()).trim().split('\n').filter(Boolean); + const mounts = lines.map(parseMountInfoLine); + const invocationMounts = mounts.filter(({ mountPoint }) => mountPoint === invocationDirectory); + const mount = invocationMounts[0]; + if ( + invocationMounts.length !== 1 || + !mount || + mount.root !== '/' || + mount.filesystemType !== 'tmpfs' || + mount.source !== ENCLAVE_STORAGE_SOURCE || + mounts.some(({ mountPoint }) => mountPoint.startsWith(`${invocationDirectory}/`)) || + mounts.some((other) => other !== mount && other.device === mount.device) + ) { + throw new Error('Invocation-private bounded enclave tmpfs mount is missing or unverifiable'); + } + const options = lines[mounts.indexOf(mount)].split(' ')[5].split(','); + if (!['rw', 'nosuid', 'nodev', 'noexec'].every((option) => options.includes(option))) { + throw new Error('Bounded enclave tmpfs mount options are unverifiable'); + } + for (const directory of [invocationDirectory, ...writableDirectories]) { + const filesystem = await dependencies.statfs(directory); + if ( + filesystem.type !== TMPFS_MAGIC || + filesystem.blocks * filesystem.bsize !== BigInt(maximumBytes) + ) { + throw new Error('Bounded enclave tmpfs capacity does not match the closed role limit'); + } + } +} + +export async function assertBoundedEnclaveWritableExports( + exports: readonly CloudHypervisorDirectoryExport[], + maximumBytes: number, +): Promise { + const directories = exports.filter((entry) => entry.mode === 'rw').map((entry) => entry.source); + if (directories.length === 0) { + throw new Error('Cloud Hypervisor enclave requires bounded writable exports'); + } + await assertBoundedEnclaveStorage(path.dirname(directories[0]), maximumBytes, directories); +} diff --git a/src/cloud-hypervisor/host-enclave-executor.test.ts b/src/cloud-hypervisor/host-enclave-executor.test.ts index 8e08fc9a5..5c724ee31 100644 --- a/src/cloud-hypervisor/host-enclave-executor.test.ts +++ b/src/cloud-hypervisor/host-enclave-executor.test.ts @@ -223,6 +223,7 @@ describe('readBoundedCloudHypervisorEnclaveResult', () => { mountTmpfs: async (_directory, _size, mountUid, mountGid) => { mountedIdentity = { uid: mountUid, gid: mountGid }; }, + verifyStorage: jest.fn().mockResolvedValue(undefined), unmount: async () => { unmounted = true; }, chown: async (filePathValue) => { chownedPaths.push(filePathValue.toString()); }, resolveIdentity: () => ({ uid, gid }), @@ -342,6 +343,24 @@ describe('readBoundedCloudHypervisorEnclaveResult', () => { bearer: `${agentPolicy.githubBearer}\n`, }); } else if (expectedOutcome === 'success') { + stopped = false; + unmounted = false; + snapshotRemoved = false; + const invalidStorageBackend = new CloudHypervisorHostEnclaveExecutorBackend( + backendOptions, + { + ...dependencies, + verifyStorage: async () => { throw new Error('Unverifiable bounded storage'); }, + createManager: () => { throw new Error('VM must not be created'); }, + }, + ); + await expect(invalidStorageBackend.execute(plan, new AbortController().signal)) + .resolves.toEqual({ outcome: 'executor-failure' }); + expect(unmounted).toBe(true); + expect(snapshotRemoved).toBe(false); + expect(await fs.lstat(invocationHostDir).catch(() => undefined)).toBeUndefined(); + await invalidStorageBackend.close(); + expect(workloadProfile?.kind).toBe('script-enclave'); if (expectedOutcome === 'success') { expect(executionRequest?.argv).toEqual([scriptArtifact.entrypoint]); diff --git a/src/cloud-hypervisor/host-enclave-executor.ts b/src/cloud-hypervisor/host-enclave-executor.ts index cd46ae37a..990bc129a 100644 --- a/src/cloud-hypervisor/host-enclave-executor.ts +++ b/src/cloud-hypervisor/host-enclave-executor.ts @@ -21,6 +21,9 @@ import { reapHostExecutorResources, } from '../enclave/host-executor-journal'; import { DurableCloudHypervisorCleanupRegistry } from './cleanup-registry'; +import { + assertBoundedEnclaveStorage, mountBoundedEnclaveStorage, unmountBoundedEnclaveStorage, +} from './enclave-storage'; import { ENCLAVE_AGENT_API_PROXY_IP, ENCLAVE_AGENT_GITHUB_MCP_IP, @@ -166,6 +169,7 @@ export interface HostEnclaveExecutorDependencies { tools: CloudHypervisorHostToolPaths, ) => Promise; readonly unmount: (directory: string, tools: CloudHypervisorHostToolPaths) => Promise; + readonly verifyStorage: typeof assertBoundedEnclaveStorage; readonly mkdir: typeof fs.mkdir; readonly realpath: typeof fs.realpath; readonly lstat: typeof fs.lstat; @@ -200,28 +204,9 @@ const defaultDependencies: HostEnclaveExecutorDependencies = { artifacts, true, ), - mountTmpfs: async (directory, sizeBytes, uid, gid, tools) => { - const mountOptions = - `size=${sizeBytes},mode=0700,uid=${uid},gid=${gid},nosuid,nodev,noexec`; - const result = await execa(tools.mount, [ - '-t', 'tmpfs', - '-o', mountOptions, - 'awf-enclave-invocation', - directory, - ], { reject: false, stdio: ['ignore', 'pipe', 'pipe'] }); - if (result.exitCode !== 0) { - throw new Error(`Unable to mount bounded enclave invocation storage: ${result.stderr.trim()}`); - } - }, - unmount: async (directory, tools) => { - const result = await execa(tools.umount, [directory], { - reject: false, - stdio: ['ignore', 'pipe', 'pipe'], - }); - if (result.exitCode !== 0) { - throw new Error(`Unable to unmount enclave invocation storage: ${result.stderr.trim()}`); - } - }, + mountTmpfs: mountBoundedEnclaveStorage, + unmount: unmountBoundedEnclaveStorage, + verifyStorage: assertBoundedEnclaveStorage, mkdir: fs.mkdir, realpath: fs.realpath, lstat: fs.lstat, @@ -565,6 +550,7 @@ async function prepareInvocationFilesystem( tools, ); await onMounted(); + await dependencies.verifyStorage(plan.invocationHostDir, resourceProfile.writableStorageBytes); for (const name of ['request', 'output', 'runtime']) { const directory = filePath(plan.invocationHostDir, name); await dependencies.mkdir(directory, { mode: 0o700 }); diff --git a/src/cloud-hypervisor/launcher.test.ts b/src/cloud-hypervisor/launcher.test.ts index 2df220b4e..09fa65835 100644 --- a/src/cloud-hypervisor/launcher.test.ts +++ b/src/cloud-hypervisor/launcher.test.ts @@ -4,6 +4,7 @@ import { computeCloudHypervisorLandlockRules, type CloudHypervisorCgroupDependencies, } from './launcher'; +import { CLOUD_HYPERVISOR_ENCLAVE_RESOURCE_PROFILES as profiles } from './workload-profile'; describe('buildCloudHypervisorLaunchCommand', () => { const baseOptions = { @@ -228,6 +229,36 @@ describe('CloudHypervisorCgroup', () => { ).expectedLimits()).toThrow(/resource limits are invalid/); }); + it.each(['script', 'agent'] as const)('budgets host tmpfs pages separately from %s guest memory', async (role) => { + const profile = profiles[role]; + const deps = dependencies(); + const cgroup = new CloudHypervisorCgroup( + '/sys/fs/cgroup/awf-cloud-hypervisor/enclave-storage', + { + memoryMib: profile.memoryMiB, + vcpuCount: profile.vcpuCount, + cpuQuotaMilli: profile.cpuQuotaMilli, + writableStorageBytes: profile.writableStorageBytes, + }, + deps, + ); + await cgroup.setup(); + expect(cgroup.expectedLimits().memoryMax).toBe( + String((profile.memoryMiB + 256) * 1024 * 1024 + profile.writableStorageBytes), + ); + expect(deps.writeFile).toHaveBeenCalledWith( + '/sys/fs/cgroup/awf-cloud-hypervisor/enclave-storage/memory.max', + cgroup.expectedLimits().memoryMax, + ); + }); + + it.each([0, -1, NaN, Infinity, Number.MAX_SAFE_INTEGER])('rejects invalid storage memory budget %s', (size) => { + expect(() => new CloudHypervisorCgroup( + '/sys/fs/cgroup/awf-cloud-hypervisor/invalid', + { memoryMib: 768, vcpuCount: 1, writableStorageBytes: size }, + ).expectedLimits()).toThrow(/resource limits are invalid/); + }); + it('assigns a PID into cgroup.procs and rejects invalid PIDs', async () => { const deps = dependencies(); const cgroup = new CloudHypervisorCgroup('/sys/fs/cgroup/awf-cloud-hypervisor/run-1', { memoryMib: 512, vcpuCount: 2 }, deps); diff --git a/src/cloud-hypervisor/launcher.ts b/src/cloud-hypervisor/launcher.ts index a8e10229e..75554a07d 100644 --- a/src/cloud-hypervisor/launcher.ts +++ b/src/cloud-hypervisor/launcher.ts @@ -218,6 +218,8 @@ export interface CloudHypervisorResourceLimits { readonly memoryMib: number; readonly vcpuCount: number; readonly cpuQuotaMilli?: number; + /** Host tmpfs pages charged to virtio-fs, separate from guest RAM. */ + readonly writableStorageBytes?: number; } export interface CloudHypervisorCgroupLimits { @@ -287,6 +289,8 @@ export function computeCloudHypervisorCgroupLimits( limits.memoryMib < 1 || !Number.isSafeInteger(limits.vcpuCount) || limits.vcpuCount < 1 || + (limits.writableStorageBytes !== undefined && + (!Number.isSafeInteger(limits.writableStorageBytes) || limits.writableStorageBytes < 1)) || (limits.cpuQuotaMilli !== undefined && (!Number.isSafeInteger(limits.cpuQuotaMilli) || limits.cpuQuotaMilli < 1 || @@ -294,7 +298,11 @@ export function computeCloudHypervisorCgroupLimits( ) { throw new Error('Cloud Hypervisor cgroup resource limits are invalid'); } - const memoryMaxBytes = (limits.memoryMib + CGROUP_MEMORY_HEADROOM_MIB) * 1024 * 1024; + const memoryMaxBytes = (limits.memoryMib + CGROUP_MEMORY_HEADROOM_MIB) * 1024 * 1024 + + (limits.writableStorageBytes ?? 0); + if (!Number.isSafeInteger(memoryMaxBytes)) { + throw new Error('Cloud Hypervisor cgroup resource limits are invalid'); + } const cpuQuotaUs = limits.cpuQuotaMilli === undefined ? limits.vcpuCount * CGROUP_V2_PERIOD_US + CGROUP_CPU_HEADROOM_QUOTA_US : limits.cpuQuotaMilli * CGROUP_V2_PERIOD_US / 1000; diff --git a/src/cloud-hypervisor/manager-launch.test.ts b/src/cloud-hypervisor/manager-launch.test.ts index e71dd940d..4f2d8944a 100644 --- a/src/cloud-hypervisor/manager-launch.test.ts +++ b/src/cloud-hypervisor/manager-launch.test.ts @@ -300,6 +300,34 @@ import { expect(deps.launch).not.toHaveBeenCalled(); }); + it('includes the closed storage budget only in the trusted enclave host cgroup', async () => { + const deps = dependencies(); + const profile = createScriptEnclaveCloudHypervisorProfile({ + enclaveId: 'script-entry', + invocationId: 'b'.repeat(32), + guest: { + supervisorBinaryPath: '/opt/awf-supervisor', + supervisorSha256: 'a'.repeat(64), + }, + exportPlan: enclaveExportPlan('script'), + }); + const manager = new CloudHypervisorManager( + config(), '/tmp/awf', deps, 'script-storage', profile, undefined, undefined, true, + ); + const client = await manager.start(); + expect(deps.createCgroup).toHaveBeenCalledWith( + expect.any(String), + { + memoryMib: 768, vcpuCount: 1, cpuQuotaMilli: 500, + writableStorageBytes: 1024 * 1024 * 1024, + }, + ); + expect(client.vmCreate).toHaveBeenCalledWith(expect.objectContaining({ + memory: expect.objectContaining({ size: 768 * 1024 * 1024 }), + })); + await manager.stop(); + }); + it('configures one rootfs disk and virtio-fs devices, then stops daemons after the VMM', async () => { const order: string[] = []; const child = processMock(); diff --git a/src/cloud-hypervisor/manager-start.ts b/src/cloud-hypervisor/manager-start.ts index 2a4fa3eea..edd5a8d55 100644 --- a/src/cloud-hypervisor/manager-start.ts +++ b/src/cloud-hypervisor/manager-start.ts @@ -226,6 +226,7 @@ export async function startCloudHypervisor( memoryMib: guestConfig.enclaveResources.memoryMiB, vcpuCount: guestConfig.enclaveResources.vcpuCount, cpuQuotaMilli: guestConfig.enclaveResources.cpuQuotaMilli, + writableStorageBytes: guestConfig.enclaveResources.writableStorageBytes, } : { memoryMib: config.memoryMib, vcpuCount: config.vcpuCount }, ); diff --git a/src/cloud-hypervisor/virtiofsd.test.ts b/src/cloud-hypervisor/virtiofsd.test.ts index 5d3560851..02e1dc710 100644 --- a/src/cloud-hypervisor/virtiofsd.test.ts +++ b/src/cloud-hypervisor/virtiofsd.test.ts @@ -109,10 +109,7 @@ function dependencies( isFile: () => false, isSymbolicLink: () => false, }), - statWritableFilesystem: jest.fn().mockResolvedValue({ - device: '8:1', - capacityBytes: 1024, - }), + assertWritableStorageBound: jest.fn().mockResolvedValue(undefined), realpath: jest.fn(async (filePath: string) => filePath), readFile, readlink: jest.fn(async (filePath: string) => { @@ -171,26 +168,28 @@ const enforcement = { }; describe('VirtiofsdManager', () => { - it('requires all writable enclave exports to use one filesystem within the storage budget', async () => { + it('verifies bounded enclave storage before launching any daemon', async () => { const deps = dependencies(); const boundedManager = manager(deps); await expect(boundedManager.start([workspace], undefined, 2048)).resolves.toHaveLength(1); + expect(deps.assertWritableStorageBound).toHaveBeenCalledWith([workspace], 2048); await boundedManager.stop(); - await expect(manager(dependencies({ - statWritableFilesystem: jest.fn().mockResolvedValue({ - device: '8:1', - capacityBytes: 4096, - }), - })).start([workspace], undefined, 2048)).rejects.toThrow(/exceeds its .*byte limit/); + const rejected = dependencies({ + assertWritableStorageBound: jest.fn().mockRejectedValue(new Error('Unverifiable bounded storage')), + }); + await expect(manager(rejected).start([workspace], undefined, 2048)) + .rejects.toThrow('Unverifiable bounded storage'); + expect(rejected.launch).not.toHaveBeenCalled(); + }); - await expect(manager(dependencies({ - statWritableFilesystem: jest.fn() - .mockResolvedValueOnce({ device: '8:1', capacityBytes: 1024 }) - .mockResolvedValueOnce({ device: '8:2', capacityBytes: 1024 }), - })).start([workspace, { ...workspace, tag: 'another', source: '/host/other' }], undefined, 2048)) - .rejects.toThrow(/share one bounded host filesystem/); + it('does not provision or verify bounded storage for primary-agent exports', async () => { + const deps = dependencies(); + const primary = manager(deps); + await primary.start([workspace]); + expect(deps.assertWritableStorageBound).not.toHaveBeenCalled(); + await primary.stop(); }); it('uses explicit sandbox, seccomp, cache, and inode policy', () => { diff --git a/src/cloud-hypervisor/virtiofsd.ts b/src/cloud-hypervisor/virtiofsd.ts index bd2749e4b..31815e485 100644 --- a/src/cloud-hypervisor/virtiofsd.ts +++ b/src/cloud-hypervisor/virtiofsd.ts @@ -4,6 +4,7 @@ import execa, { type ExecaChildProcess } from 'execa'; import type { CloudHypervisorCgroup } from './launcher'; import type { CloudHypervisorDirectoryExport } from './exports'; import type { CloudHypervisorCleanupHandle } from './cleanup-registry'; +import { assertBoundedEnclaveWritableExports } from './enclave-storage'; import { StagedHostMountTree, selectMountPlan, @@ -64,7 +65,7 @@ export interface VirtiofsdDependencies runTool(command: string, args: readonly string[]): Promise; captureTool(command: string, args: readonly string[]): Promise; statPath(filePath: string): Promise; - statWritableFilesystem(filePath: string): Promise<{ device: string; capacityBytes: number }>; + assertWritableStorageBound: typeof assertBoundedEnclaveWritableExports; realpath(filePath: string): Promise; readMountInfo(): Promise; } @@ -110,17 +111,7 @@ const defaultDependencies: VirtiofsdDependencies = { return result.stdout; }, statPath: fs.lstat, - statWritableFilesystem: async (filePath) => { - const [stat, filesystem] = await Promise.all([ - fs.stat(filePath, { bigint: true }), - fs.statfs(filePath, { bigint: true }), - ]); - const capacityBytes = filesystem.blocks * filesystem.bsize; - if (capacityBytes > BigInt(Number.MAX_SAFE_INTEGER)) { - throw new Error(`Writable enclave filesystem capacity is not safely representable: ${filePath}`); - } - return { device: String(stat.dev), capacityBytes: Number(capacityBytes) }; - }, + assertWritableStorageBound: assertBoundedEnclaveWritableExports, realpath: fs.realpath, readMountInfo: () => fs.readFile('/proc/self/mountinfo', 'utf8'), sleep: (milliseconds) => new Promise((resolve) => setTimeout(resolve, milliseconds)), @@ -172,7 +163,7 @@ export class VirtiofsdManager { try { assertPlansMatchExports(enforcement, exports); if (writableStorageLimitBytes !== undefined) { - await this.assertWritableStorageBound(exports, writableStorageLimitBytes); + await this.dependencies.assertWritableStorageBound(exports, writableStorageLimitBytes); } for (const [index, directoryExport] of exports.entries()) { await this.startOne(directoryExport, index, selectMountPlan(enforcement, directoryExport.tag)); @@ -195,32 +186,6 @@ export class VirtiofsdManager { } } - private async assertWritableStorageBound( - exports: readonly CloudHypervisorDirectoryExport[], - maximumBytes: number, - ): Promise { - if (!Number.isSafeInteger(maximumBytes) || maximumBytes < 1) { - throw new Error('Cloud Hypervisor enclave writable-storage limit is invalid'); - } - const writable = exports.filter((entry) => entry.mode === 'rw'); - if (writable.length === 0) { - throw new Error('Cloud Hypervisor enclave requires bounded writable exports'); - } - const filesystems = await Promise.all(writable.map(({ source }) => - this.dependencies.statWritableFilesystem(source))); - const first = filesystems[0]; - if (filesystems.some(({ device, capacityBytes }) => - device !== first.device || capacityBytes !== first.capacityBytes)) { - throw new Error('Cloud Hypervisor enclave writable exports must share one bounded host filesystem'); - } - if (first.capacityBytes > maximumBytes) { - throw new Error( - `Cloud Hypervisor enclave writable export filesystem capacity ${first.capacityBytes} ` + - `exceeds its ${maximumBytes}-byte limit`, - ); - } - } - async stop(): Promise { const errors: unknown[] = []; const remaining: RunningDaemon[] = []; From 70fd0fe207203d7459c6acddb36b24ce87fb8ec4 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Fri, 2 Oct 2026 23:40:13 +0000 Subject: [PATCH 3/3] test: run privileged enclave storage suite in CI --- .github/workflows/test-cloud-hypervisor.yml | 18 ++++++++++++++++++ docs/cloud-hypervisor-foundation.md | 2 ++ src/enclave/host-executor-broker.test.ts | 1 + 3 files changed, 21 insertions(+) diff --git a/.github/workflows/test-cloud-hypervisor.yml b/.github/workflows/test-cloud-hypervisor.yml index 281cb3655..62e74dd5e 100644 --- a/.github/workflows/test-cloud-hypervisor.yml +++ b/.github/workflows/test-cloud-hypervisor.yml @@ -69,6 +69,24 @@ jobs: "$RUNNER_TEMP/microvm-supervisor.test" \ -test.run '^TestEnclaveGuestLimitsLive$' -test.count=1 -test.v + - name: Set up Node.js + uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 + with: + node-version: '22' + cache: npm + + - name: Run privileged enclave host storage enforcement suite + # Mounts the real role-sized invocation tmpfs backing stores and + # host memory cgroups in a private mount namespace, then proves + # aggregate ENOSPC, isolation, cgroup accounting, and busy-unmount + # behavior. The suite is skipped without AWF_TEST_ENCLAVE_STORAGE. + run: | + npm ci + sudo unshare --mount --propagation private \ + env "PATH=$PATH" AWF_TEST_ENCLAVE_STORAGE=1 \ + npx jest --ci --runInBand \ + src/cloud-hypervisor/enclave-storage.integration.test.ts + - name: Install deterministic guest build prerequisites run: | sudo apt-get update diff --git a/docs/cloud-hypervisor-foundation.md b/docs/cloud-hypervisor-foundation.md index b2c0bfcd6..68d1c0870 100644 --- a/docs/cloud-hypervisor-foundation.md +++ b/docs/cloud-hypervisor-foundation.md @@ -304,6 +304,8 @@ unmount failure. It also fills storage in the production host cgroup budget whil holding the guest-RAM equivalent resident, with swap disabled, to check that storage exhaustion is not preempted by a cgroup OOM. Run it as root in a private mount namespace with `AWF_TEST_ENCLAVE_STORAGE=1 npm test -- --runInBand enclave-storage.integration.test.ts`. +The `build-test-artifacts` job in `.github/workflows/test-cloud-hypervisor.yml` +runs this suite on every matching pull request via `sudo unshare --mount --propagation private`. Live guest transport conformance additionally requires the release-attested role artifacts and KVM runtime wiring. diff --git a/src/enclave/host-executor-broker.test.ts b/src/enclave/host-executor-broker.test.ts index 7872526f4..b109da2e4 100644 --- a/src/enclave/host-executor-broker.test.ts +++ b/src/enclave/host-executor-broker.test.ts @@ -119,6 +119,7 @@ describe('finite-disclosure broker → authenticated Unix host → concrete micr copySparseFile: noop, removeArtifactSnapshot: async (directory) => fs.rm(directory, { recursive: true, force: true }), mountTmpfs: noop, + verifyStorage: noop, unmount: async () => { cleaned(); if (cleanupError) throw new Error('PRIVATE_RAW_CLEANUP_ERROR');