Files
PaperClipAI/server/src/services/native-runtime/remote-launch-assets.ts
T
DottaandClaude Opus 5.5 73dc33c620 fix(runner): keep a disk reserve so a full sandbox disk does not strand the runner (#22068)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents in a Paperclip-managed sandbox run under the Paperclip runner
(`runnerd`), which keeps durable state on the sandbox disk
> - An agent can fill that disk with its own work. In one case the agent
installed a Rust toolchain and a second checkout with dependencies on a
10 GiB sandbox
> - `runnerd` then could not write its state and exited. Each relaunch
tried to upload its launch files to the full disk and failed, 83 times
in 7 minutes
> - The run ended as a generic provider failure that asked a person to
inspect it, and Retry repeated the same failure
> - This pull request keeps a small disk reserve that belongs to
Paperclip. Recovery releases it to relaunch the runner, and stops at
once with a named code when nothing is left to free
> - The benefit is that the agent's turn can continue and see the full
disk in its own tool output, instead of the run dying with it

## Linked Issues or Issue Description

**What happened?**
An agent in a managed Daytona sandbox filled the 10 GiB disk. `runnerd`
exited with `failed to persist bounded diagnostic: No space left on
device (os error 28)`. Runner process recovery then retried for the full
420000 ms reconnect grace. Every attempt failed with
`.paperclip-upload-<id>.tar.gz: create: ... no space left on device`.
The run failed as `native_provider_terminal_failed`, and the recovery
asked a board user to verify that the provider stopped.

**Steps to reproduce**
1. Run an agent in a managed sandbox and have it fill the disk (for
example a Rust toolchain plus a full dependency install).
2. `runnerd` exits when it next writes its state.
3. Watch recovery retry the relaunch upload until the grace expires.

**Expected behavior**
Paperclip relaunches its runner on a disk the agent filled, so the turn
continues and the agent sees the full disk in its tools. When the runner
truly cannot start, recovery says so at once with a named cause instead
of retrying for 7 minutes.

**Environment**
Managed Daytona sandbox (fleet snapshot, 4 CPU, 8 GiB memory, 10 GiB
disk), `paperclip_runner` native runtime.

## What Changed

- New `server/src/services/native-runtime/remote-runner-disk-reserve.ts`
owns the reserve: hold/release scripts, the full-disk classifier, and
`createRemoteRunnerDiskReserve`. A release waits for an in-flight hold,
and no hold starts after a release.
- Both remote runner launchers (agent runs and Dot) get a reserve path
in managed sandboxes only:
`<workspace>/.paperclip-runtime/paperclip-runner/disk-reserve`.
Workspace saves and Git already exclude `.paperclip-runtime`. SSH hosts
get no reserve.
- After the first launch's identity is confirmed, the launcher holds 256
MiB in the background, only while 2 GiB more is free. It is off the
startup path.
- A relaunch releases the reserve before it stages. A launch that hits a
full disk while staging releases the reserve and stages once more.
- Release tries twice and reports `released`, `not_held` or `failed`.
Only `not_held` ends the launch with `sandbox_disk_full`. A release that
cannot run rethrows the original error, so recovery still retries.
- Command-based staging (no `syncIn`) keeps a full disk recognizable
with an `ENOSPC` code instead of a bare
`runner_remote_*_staging_failed`.
- `SANDBOX_DISK_FULL_FAILURE_CODE` and `isSandboxDiskFullFailure` live
in the runner package. Runner process recovery checks the error code and
stops at once instead of retrying for the whole grace. Other staging
failures still retry.

## Verification

- Red before the fix, green after:
-
`server/src/services/native-runtime/remote-runner-disk-reserve.test.ts`:
release-and-restage, release before relaunch, `sandbox_disk_full` when
release frees too little, no retry for a failure that is not a full
disk, hold/release scripts in a real shell, no hold below the free-space
floor. Before the fix, the launch rejected with the exact upload error
from the incident.
- `packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts`
"fails at once when a replacement cannot be staged because the sandbox
disk is full": before the fix it was still retrying after 4 s; after, it
fails with `sandbox_disk_full` after one replacement.
- Kept: "retries a replacement whose staging failed before any process
launched" still passes, so a raw transient staging failure still
retries.
- Updated: "confirms no replacement process exists when staging fails
before the launch RPC" now expects `sandbox_disk_full` with the upload
error as cause. Exit is still confirmed and no launch RPC runs.
- Real sandbox from the same fleet snapshot (Daytona, 10 GiB): hold
created a 256 MiB reserve. The disk was filled to 100%. A 5 MB write
then failed. Release returned `true`, and a 50 MB write then succeeded.
A second release returned `false`. Hold on the now 98% full disk did not
create a reserve.
- `pnpm --filter @paperclipai/server exec tsc --noEmit` passes. `pnpm
verify` result is in a comment below.
- Review round: tests for a lost release RPC, hold/release ordering,
command-based staging on a full disk, the hold after a dispatched first
launch (not on relaunch), and no reserve on SSH. Each launcher-level
test failed with its fix removed.
- Focused suites on the review revision: transport 335/335, launcher
130/130 (filtered), launch assets 35/35, reserve 14/14. Server and
runner typecheck pass.
- Not run: a full product run in which an agent fills the disk and the
relaunched `runnerd` resumes the turn end to end.

## Risks

- Each managed sandbox holds 256 MiB more disk while it has more than 2
GiB free. That is 2.5% of a 10 GiB sandbox.
- A relaunch for any reason releases the reserve for the rest of that
run. The next run's launch takes it again.
- If the agent keeps writing after the relaunch, the disk can fill
again. The second time, nothing is left to release, and the run fails at
once with `sandbox_disk_full` instead of after 7 minutes.
- The run's error code is still `native_provider_terminal_failed`. The
message now names `sandbox_disk_full`. A dedicated failure code and
recovery text are a follow-up.
- `runnerd` loses its own error message when it cannot persist its
diagnostic. This PR does not change the Rust binary.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 15:45:20 -06:00

207 lines
9.8 KiB
TypeScript

import { existsSync } from "node:fs";
import { lstat, readdir } from "node:fs/promises";
import { posix, resolve } from "node:path";
import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime";
import type { NativeRuntimeContextSnapshot } from "../../vendor/paperclip-runner/index.js";
import type { NativeRunTrace } from "./native-run-trace.js";
import { remoteStagingFailure } from "./remote-runner-disk-reserve.js";
type StageAsset = {
sourcePath: string;
targetPath: string;
mode: number;
};
type LaunchAssetsInput = {
scope?: "launch" | "recovery";
root: string;
transport: "ssh" | "sandbox";
runner: CommandManagedRuntimeRunner;
persistenceDirectories: readonly {
name: string;
location: "runner" | "filesystem";
targetPath: string | null;
}[];
runtimeContext: {
source: NativeRuntimeContextSnapshot;
target: NativeRuntimeContextSnapshot;
filesystemRoot: string;
} | null;
caBundleMapping: { sourcePath: string; targetPath: string } | null;
stageFile: (input: StageAsset) => Promise<void>;
stageDirectory: (input: StageAsset) => Promise<void>;
isolatedCodexSkills?: { workspaceRoot: string };
trace?: NativeRunTrace;
};
async function assertSelectedSkillsTree(root: string, names: string[]): Promise<void> {
const found = new Set<string>();
const reject = () => { throw new Error("runner_remote_codex_skills_invalid"); };
async function visit(relativePath: string) {
const path = relativePath ? resolve(root, relativePath) : root;
const stat = await lstat(path);
if (stat.isSymbolicLink()) reject();
if (stat.isDirectory()) {
if (relativePath && !names.some(name => relativePath === name || relativePath.startsWith(`${name}/`) || name.startsWith(`${relativePath}/`))) reject();
for (const entry of await readdir(path)) await visit(relativePath ? `${relativePath}/${entry}` : entry);
} else {
if (!stat.isFile() || (stat.mode & 0o222) !== 0 || !names.some(name => relativePath.startsWith(`${name}/`))) reject();
for (const name of names) if (relativePath === `${name}/SKILL.md`) found.add(name);
}
}
await visit("");
if (found.size !== names.length) reject();
}
// Launch preparation owns this isolated directory and runs only after the old
// provider has stopped. Never follow a retained symlink while removing stale
// assignments; auth, config and provider conversation history are not touched.
const validateRemoteCodexHome = `
const fs = require("node:fs"), path = require("node:path");
const home = path.resolve(process.argv[1]), workspace = path.resolve(process.argv[2]);
const relative = path.relative(workspace, home);
if (!relative || relative === ".." || relative.startsWith(".." + path.sep) || path.isAbsolute(relative)) throw Error("unsafe_codex_home");
// The execution target authorizes the workspace anchor. Resolve only that
// anchor (which can have an OS alias), then refuse links below it.
let current = fs.realpathSync(workspace);
if (!fs.lstatSync(current).isDirectory()) throw Error("unsafe_workspace_root");
for (const segment of relative.split(path.sep)) {
current = path.join(current, segment);
if (!fs.lstatSync(current, { throwIfNoEntry: false })) fs.mkdirSync(current, { mode: 0o700 });
const stat = fs.lstatSync(current);
if (!stat.isDirectory() || stat.isSymbolicLink()) throw Error("unsafe_codex_home");
}
for (const name of ["auth.json", "config.toml"]) {
const file = fs.lstatSync(path.join(current, name), { throwIfNoEntry: false });
if (file && (!file.isFile() || file.isSymbolicLink() || file.nlink > 1)) throw Error("unsafe_codex_launch_file");
}
`;
const clearRemoteCodexSkills = `${validateRemoteCodexHome}
const skills = path.join(current, "skills");
function unseal(directory) {
const stat = fs.lstatSync(directory);
if (stat.isSymbolicLink() || !stat.isDirectory()) throw Error("unsafe_skills_root");
fs.chmodSync(directory, 0o700);
for (const name of fs.readdirSync(directory)) {
const child = path.join(directory, name), stat = fs.lstatSync(child);
if (stat.isDirectory() && !stat.isSymbolicLink()) unseal(child);
}
}
if (fs.lstatSync(skills, { throwIfNoEntry: false })) {
unseal(skills);
fs.rmSync(skills, { recursive: true });
}
// Native directory transfers omit the source root and may ignore its mapping
// mode. Create the private discovery root before their mkdir/extract step.
fs.mkdirSync(skills, { mode: 0o700 });
fs.chmodSync(skills, 0o700);
`;
const prepareRemoteCodexRecoveryHome = `${validateRemoteCodexHome}
if (process.argv[3] === "remove-auth") fs.rmSync(path.join(current, "auth.json"), { force: true });
`;
export async function stageRemoteLaunchAssets(input: LaunchAssetsInput): Promise<void> {
const measure = async (name: string, operation: () => Promise<void>) =>
input.trace ? input.trace.measure(name, operation) : operation();
const materializeLaunchState = () => measure("harness_state.launch_material", async () => {
for (const directory of input.persistenceDirectories) {
if (directory.location !== "filesystem") continue;
const targetPath = directory.targetPath;
if (!targetPath) throw new Error("runner_harness_state_mismatch: bootstrap_target_unavailable");
const escapedTarget = targetPath.replaceAll("'", "'\\''");
const created = await input.runner.execute({
command: "sh",
args: ["-c", `umask 077; install -d -m 0700 '${escapedTarget}'`],
bypassSession: true,
timeoutMs: 10_000,
});
if (created.exitCode !== 0 || created.timedOut) {
throw remoteStagingFailure("runner_remote_directory_staging_failed", created);
}
if (directory.name !== "codex-home") continue;
const localDirectory = resolve(input.root, directory.name);
for (const name of input.scope === "recovery" ? ["config.toml", "auth.json"] : ["auth.json", "config.toml"]) {
const sourcePath = resolve(localDirectory, name);
if (input.scope === "recovery" && name === "config.toml" && !existsSync(sourcePath)) throw new Error("runner_remote_codex_recovery_config_missing");
if (!existsSync(sourcePath)) continue;
await input.stageFile({ sourcePath, targetPath: posix.join(targetPath, name), mode: 0o600 });
}
}
});
if (input.scope === "recovery") {
const home = input.persistenceDirectories.find(directory => directory.name === "codex-home" && directory.location === "filesystem");
if (!home?.targetPath || !input.isolatedCodexSkills) throw new Error("runner_remote_codex_credentials_path_invalid");
const validated = await input.runner.execute({
command: "node",
args: ["-e", prepareRemoteCodexRecoveryHome, home.targetPath, input.isolatedCodexSkills.workspaceRoot,
existsSync(resolve(input.root, "codex-home", "auth.json")) ? "keep-auth" : "remove-auth"],
bypassSession: true,
timeoutMs: 10_000,
});
if (validated.exitCode !== 0 || validated.timedOut) throw new Error("runner_remote_codex_credentials_path_invalid");
await materializeLaunchState();
return;
}
const stageRuntimeContext = async () => {
const context = input.runtimeContext;
if (!context) return;
await measure("stage.asset.runtime_context", async () => {
await input.stageDirectory({
sourcePath: context.source.instructions.bundle.rootPath,
targetPath: context.target.instructions.bundle.rootPath,
mode: 0o555,
});
for (let index = 0; index < context.source.skills.length; index += 1) {
await input.stageDirectory({
sourcePath: context.source.skills[index]!.bundle.rootPath,
targetPath: context.target.skills[index]!.bundle.rootPath,
mode: 0o555,
});
}
await input.stageFile({
sourcePath: resolve(input.root, "runtime-context.json"),
targetPath: posix.join(context.filesystemRoot, "runtime-context.json"),
mode: 0o600,
});
});
};
const stageCodexSkills = async () => {
if (!input.isolatedCodexSkills) return;
const directory = input.persistenceDirectories.find(entry => entry.name === "codex-home" && entry.location === "filesystem");
if (!directory?.targetPath) throw new Error("runner_harness_state_mismatch: bootstrap_target_unavailable");
const sourcePath = resolve(input.root, "codex-home", "skills");
await measure("stage.asset.codex_skills.verify", () =>
assertSelectedSkillsTree(sourcePath, input.runtimeContext?.source.skills.map(skill => skill.runtimeName) ?? []));
await measure("stage.asset.codex_skills", async () => {
const cleared = await input.runner.execute({
command: "node",
args: ["-e", clearRemoteCodexSkills, directory.targetPath!, input.isolatedCodexSkills!.workspaceRoot],
bypassSession: true,
timeoutMs: 10_000,
});
if (cleared.exitCode !== 0 || cleared.timedOut) throw new Error("runner_remote_codex_skills_reset_failed");
// Match prepareIsolatedCodexHome: the discovery root is private/writable
// for Codex's own built-ins; each selected package remains sealed.
await input.stageDirectory({ sourcePath, targetPath: posix.join(directory.targetPath!, "skills"), mode: 0o700 });
});
};
// Resolve the owned home and remove obsolete assignments before any launch
// file can be written through a retained, redirected path.
await stageCodexSkills();
if (input.transport === "sandbox" && input.runner.allowConcurrentSyncOperations === true) {
const outcomes = await Promise.allSettled([materializeLaunchState(), stageRuntimeContext()]);
for (const outcome of outcomes) {
if (outcome.status === "rejected") throw outcome.reason;
}
} else {
await materializeLaunchState();
await stageRuntimeContext();
}
if (input.caBundleMapping) {
const mapping = input.caBundleMapping;
await measure("stage.asset.ca_bundle", () => input.stageFile({ ...mapping, mode: 0o600 }));
}
}