mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-21 02:43:25 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents in a Paperclip-managed sandbox run under the Paperclip runner (`runnerd`), which keeps durable state on the sandbox disk > - An agent can fill that disk with its own work. In one case the agent installed a Rust toolchain and a second checkout with dependencies on a 10 GiB sandbox > - `runnerd` then could not write its state and exited. Each relaunch tried to upload its launch files to the full disk and failed, 83 times in 7 minutes > - The run ended as a generic provider failure that asked a person to inspect it, and Retry repeated the same failure > - This pull request keeps a small disk reserve that belongs to Paperclip. Recovery releases it to relaunch the runner, and stops at once with a named code when nothing is left to free > - The benefit is that the agent's turn can continue and see the full disk in its own tool output, instead of the run dying with it ## Linked Issues or Issue Description **What happened?** An agent in a managed Daytona sandbox filled the 10 GiB disk. `runnerd` exited with `failed to persist bounded diagnostic: No space left on device (os error 28)`. Runner process recovery then retried for the full 420000 ms reconnect grace. Every attempt failed with `.paperclip-upload-<id>.tar.gz: create: ... no space left on device`. The run failed as `native_provider_terminal_failed`, and the recovery asked a board user to verify that the provider stopped. **Steps to reproduce** 1. Run an agent in a managed sandbox and have it fill the disk (for example a Rust toolchain plus a full dependency install). 2. `runnerd` exits when it next writes its state. 3. Watch recovery retry the relaunch upload until the grace expires. **Expected behavior** Paperclip relaunches its runner on a disk the agent filled, so the turn continues and the agent sees the full disk in its tools. When the runner truly cannot start, recovery says so at once with a named cause instead of retrying for 7 minutes. **Environment** Managed Daytona sandbox (fleet snapshot, 4 CPU, 8 GiB memory, 10 GiB disk), `paperclip_runner` native runtime. ## What Changed - New `server/src/services/native-runtime/remote-runner-disk-reserve.ts` owns the reserve: hold/release scripts, the full-disk classifier, and `createRemoteRunnerDiskReserve`. A release waits for an in-flight hold, and no hold starts after a release. - Both remote runner launchers (agent runs and Dot) get a reserve path in managed sandboxes only: `<workspace>/.paperclip-runtime/paperclip-runner/disk-reserve`. Workspace saves and Git already exclude `.paperclip-runtime`. SSH hosts get no reserve. - After the first launch's identity is confirmed, the launcher holds 256 MiB in the background, only while 2 GiB more is free. It is off the startup path. - A relaunch releases the reserve before it stages. A launch that hits a full disk while staging releases the reserve and stages once more. - Release tries twice and reports `released`, `not_held` or `failed`. Only `not_held` ends the launch with `sandbox_disk_full`. A release that cannot run rethrows the original error, so recovery still retries. - Command-based staging (no `syncIn`) keeps a full disk recognizable with an `ENOSPC` code instead of a bare `runner_remote_*_staging_failed`. - `SANDBOX_DISK_FULL_FAILURE_CODE` and `isSandboxDiskFullFailure` live in the runner package. Runner process recovery checks the error code and stops at once instead of retrying for the whole grace. Other staging failures still retry. ## Verification - Red before the fix, green after: - `server/src/services/native-runtime/remote-runner-disk-reserve.test.ts`: release-and-restage, release before relaunch, `sandbox_disk_full` when release frees too little, no retry for a failure that is not a full disk, hold/release scripts in a real shell, no hold below the free-space floor. Before the fix, the launch rejected with the exact upload error from the incident. - `packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts` "fails at once when a replacement cannot be staged because the sandbox disk is full": before the fix it was still retrying after 4 s; after, it fails with `sandbox_disk_full` after one replacement. - Kept: "retries a replacement whose staging failed before any process launched" still passes, so a raw transient staging failure still retries. - Updated: "confirms no replacement process exists when staging fails before the launch RPC" now expects `sandbox_disk_full` with the upload error as cause. Exit is still confirmed and no launch RPC runs. - Real sandbox from the same fleet snapshot (Daytona, 10 GiB): hold created a 256 MiB reserve. The disk was filled to 100%. A 5 MB write then failed. Release returned `true`, and a 50 MB write then succeeded. A second release returned `false`. Hold on the now 98% full disk did not create a reserve. - `pnpm --filter @paperclipai/server exec tsc --noEmit` passes. `pnpm verify` result is in a comment below. - Review round: tests for a lost release RPC, hold/release ordering, command-based staging on a full disk, the hold after a dispatched first launch (not on relaunch), and no reserve on SSH. Each launcher-level test failed with its fix removed. - Focused suites on the review revision: transport 335/335, launcher 130/130 (filtered), launch assets 35/35, reserve 14/14. Server and runner typecheck pass. - Not run: a full product run in which an agent fills the disk and the relaunched `runnerd` resumes the turn end to end. ## Risks - Each managed sandbox holds 256 MiB more disk while it has more than 2 GiB free. That is 2.5% of a 10 GiB sandbox. - A relaunch for any reason releases the reserve for the rest of that run. The next run's launch takes it again. - If the agent keeps writing after the relaunch, the disk can fill again. The second time, nothing is left to release, and the run fails at once with `sandbox_disk_full` instead of after 7 minutes. - The run's error code is still `native_provider_terminal_failed`. The message now names `sandbox_disk_full`. A dedicated failure code and recovery text are a follow-up. - `runnerd` loses its own error message when it cannot persist its diagnostic. This PR does not change the Rust binary. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
207 lines
9.8 KiB
TypeScript
207 lines
9.8 KiB
TypeScript
import { existsSync } from "node:fs";
|
|
import { lstat, readdir } from "node:fs/promises";
|
|
import { posix, resolve } from "node:path";
|
|
import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime";
|
|
import type { NativeRuntimeContextSnapshot } from "../../vendor/paperclip-runner/index.js";
|
|
import type { NativeRunTrace } from "./native-run-trace.js";
|
|
import { remoteStagingFailure } from "./remote-runner-disk-reserve.js";
|
|
|
|
type StageAsset = {
|
|
sourcePath: string;
|
|
targetPath: string;
|
|
mode: number;
|
|
};
|
|
|
|
type LaunchAssetsInput = {
|
|
scope?: "launch" | "recovery";
|
|
root: string;
|
|
transport: "ssh" | "sandbox";
|
|
runner: CommandManagedRuntimeRunner;
|
|
persistenceDirectories: readonly {
|
|
name: string;
|
|
location: "runner" | "filesystem";
|
|
targetPath: string | null;
|
|
}[];
|
|
runtimeContext: {
|
|
source: NativeRuntimeContextSnapshot;
|
|
target: NativeRuntimeContextSnapshot;
|
|
filesystemRoot: string;
|
|
} | null;
|
|
caBundleMapping: { sourcePath: string; targetPath: string } | null;
|
|
stageFile: (input: StageAsset) => Promise<void>;
|
|
stageDirectory: (input: StageAsset) => Promise<void>;
|
|
isolatedCodexSkills?: { workspaceRoot: string };
|
|
trace?: NativeRunTrace;
|
|
};
|
|
|
|
async function assertSelectedSkillsTree(root: string, names: string[]): Promise<void> {
|
|
const found = new Set<string>();
|
|
const reject = () => { throw new Error("runner_remote_codex_skills_invalid"); };
|
|
async function visit(relativePath: string) {
|
|
const path = relativePath ? resolve(root, relativePath) : root;
|
|
const stat = await lstat(path);
|
|
if (stat.isSymbolicLink()) reject();
|
|
if (stat.isDirectory()) {
|
|
if (relativePath && !names.some(name => relativePath === name || relativePath.startsWith(`${name}/`) || name.startsWith(`${relativePath}/`))) reject();
|
|
for (const entry of await readdir(path)) await visit(relativePath ? `${relativePath}/${entry}` : entry);
|
|
} else {
|
|
if (!stat.isFile() || (stat.mode & 0o222) !== 0 || !names.some(name => relativePath.startsWith(`${name}/`))) reject();
|
|
for (const name of names) if (relativePath === `${name}/SKILL.md`) found.add(name);
|
|
}
|
|
}
|
|
await visit("");
|
|
if (found.size !== names.length) reject();
|
|
}
|
|
|
|
// Launch preparation owns this isolated directory and runs only after the old
|
|
// provider has stopped. Never follow a retained symlink while removing stale
|
|
// assignments; auth, config and provider conversation history are not touched.
|
|
const validateRemoteCodexHome = `
|
|
const fs = require("node:fs"), path = require("node:path");
|
|
const home = path.resolve(process.argv[1]), workspace = path.resolve(process.argv[2]);
|
|
const relative = path.relative(workspace, home);
|
|
if (!relative || relative === ".." || relative.startsWith(".." + path.sep) || path.isAbsolute(relative)) throw Error("unsafe_codex_home");
|
|
// The execution target authorizes the workspace anchor. Resolve only that
|
|
// anchor (which can have an OS alias), then refuse links below it.
|
|
let current = fs.realpathSync(workspace);
|
|
if (!fs.lstatSync(current).isDirectory()) throw Error("unsafe_workspace_root");
|
|
for (const segment of relative.split(path.sep)) {
|
|
current = path.join(current, segment);
|
|
if (!fs.lstatSync(current, { throwIfNoEntry: false })) fs.mkdirSync(current, { mode: 0o700 });
|
|
const stat = fs.lstatSync(current);
|
|
if (!stat.isDirectory() || stat.isSymbolicLink()) throw Error("unsafe_codex_home");
|
|
}
|
|
for (const name of ["auth.json", "config.toml"]) {
|
|
const file = fs.lstatSync(path.join(current, name), { throwIfNoEntry: false });
|
|
if (file && (!file.isFile() || file.isSymbolicLink() || file.nlink > 1)) throw Error("unsafe_codex_launch_file");
|
|
}
|
|
`;
|
|
|
|
const clearRemoteCodexSkills = `${validateRemoteCodexHome}
|
|
const skills = path.join(current, "skills");
|
|
function unseal(directory) {
|
|
const stat = fs.lstatSync(directory);
|
|
if (stat.isSymbolicLink() || !stat.isDirectory()) throw Error("unsafe_skills_root");
|
|
fs.chmodSync(directory, 0o700);
|
|
for (const name of fs.readdirSync(directory)) {
|
|
const child = path.join(directory, name), stat = fs.lstatSync(child);
|
|
if (stat.isDirectory() && !stat.isSymbolicLink()) unseal(child);
|
|
}
|
|
}
|
|
if (fs.lstatSync(skills, { throwIfNoEntry: false })) {
|
|
unseal(skills);
|
|
fs.rmSync(skills, { recursive: true });
|
|
}
|
|
// Native directory transfers omit the source root and may ignore its mapping
|
|
// mode. Create the private discovery root before their mkdir/extract step.
|
|
fs.mkdirSync(skills, { mode: 0o700 });
|
|
fs.chmodSync(skills, 0o700);
|
|
`;
|
|
|
|
const prepareRemoteCodexRecoveryHome = `${validateRemoteCodexHome}
|
|
if (process.argv[3] === "remove-auth") fs.rmSync(path.join(current, "auth.json"), { force: true });
|
|
`;
|
|
|
|
export async function stageRemoteLaunchAssets(input: LaunchAssetsInput): Promise<void> {
|
|
const measure = async (name: string, operation: () => Promise<void>) =>
|
|
input.trace ? input.trace.measure(name, operation) : operation();
|
|
const materializeLaunchState = () => measure("harness_state.launch_material", async () => {
|
|
for (const directory of input.persistenceDirectories) {
|
|
if (directory.location !== "filesystem") continue;
|
|
const targetPath = directory.targetPath;
|
|
if (!targetPath) throw new Error("runner_harness_state_mismatch: bootstrap_target_unavailable");
|
|
const escapedTarget = targetPath.replaceAll("'", "'\\''");
|
|
const created = await input.runner.execute({
|
|
command: "sh",
|
|
args: ["-c", `umask 077; install -d -m 0700 '${escapedTarget}'`],
|
|
bypassSession: true,
|
|
timeoutMs: 10_000,
|
|
});
|
|
if (created.exitCode !== 0 || created.timedOut) {
|
|
throw remoteStagingFailure("runner_remote_directory_staging_failed", created);
|
|
}
|
|
if (directory.name !== "codex-home") continue;
|
|
const localDirectory = resolve(input.root, directory.name);
|
|
for (const name of input.scope === "recovery" ? ["config.toml", "auth.json"] : ["auth.json", "config.toml"]) {
|
|
const sourcePath = resolve(localDirectory, name);
|
|
if (input.scope === "recovery" && name === "config.toml" && !existsSync(sourcePath)) throw new Error("runner_remote_codex_recovery_config_missing");
|
|
if (!existsSync(sourcePath)) continue;
|
|
await input.stageFile({ sourcePath, targetPath: posix.join(targetPath, name), mode: 0o600 });
|
|
}
|
|
}
|
|
});
|
|
if (input.scope === "recovery") {
|
|
const home = input.persistenceDirectories.find(directory => directory.name === "codex-home" && directory.location === "filesystem");
|
|
if (!home?.targetPath || !input.isolatedCodexSkills) throw new Error("runner_remote_codex_credentials_path_invalid");
|
|
const validated = await input.runner.execute({
|
|
command: "node",
|
|
args: ["-e", prepareRemoteCodexRecoveryHome, home.targetPath, input.isolatedCodexSkills.workspaceRoot,
|
|
existsSync(resolve(input.root, "codex-home", "auth.json")) ? "keep-auth" : "remove-auth"],
|
|
bypassSession: true,
|
|
timeoutMs: 10_000,
|
|
});
|
|
if (validated.exitCode !== 0 || validated.timedOut) throw new Error("runner_remote_codex_credentials_path_invalid");
|
|
await materializeLaunchState();
|
|
return;
|
|
}
|
|
const stageRuntimeContext = async () => {
|
|
const context = input.runtimeContext;
|
|
if (!context) return;
|
|
await measure("stage.asset.runtime_context", async () => {
|
|
await input.stageDirectory({
|
|
sourcePath: context.source.instructions.bundle.rootPath,
|
|
targetPath: context.target.instructions.bundle.rootPath,
|
|
mode: 0o555,
|
|
});
|
|
for (let index = 0; index < context.source.skills.length; index += 1) {
|
|
await input.stageDirectory({
|
|
sourcePath: context.source.skills[index]!.bundle.rootPath,
|
|
targetPath: context.target.skills[index]!.bundle.rootPath,
|
|
mode: 0o555,
|
|
});
|
|
}
|
|
await input.stageFile({
|
|
sourcePath: resolve(input.root, "runtime-context.json"),
|
|
targetPath: posix.join(context.filesystemRoot, "runtime-context.json"),
|
|
mode: 0o600,
|
|
});
|
|
});
|
|
};
|
|
const stageCodexSkills = async () => {
|
|
if (!input.isolatedCodexSkills) return;
|
|
const directory = input.persistenceDirectories.find(entry => entry.name === "codex-home" && entry.location === "filesystem");
|
|
if (!directory?.targetPath) throw new Error("runner_harness_state_mismatch: bootstrap_target_unavailable");
|
|
const sourcePath = resolve(input.root, "codex-home", "skills");
|
|
await measure("stage.asset.codex_skills.verify", () =>
|
|
assertSelectedSkillsTree(sourcePath, input.runtimeContext?.source.skills.map(skill => skill.runtimeName) ?? []));
|
|
await measure("stage.asset.codex_skills", async () => {
|
|
const cleared = await input.runner.execute({
|
|
command: "node",
|
|
args: ["-e", clearRemoteCodexSkills, directory.targetPath!, input.isolatedCodexSkills!.workspaceRoot],
|
|
bypassSession: true,
|
|
timeoutMs: 10_000,
|
|
});
|
|
if (cleared.exitCode !== 0 || cleared.timedOut) throw new Error("runner_remote_codex_skills_reset_failed");
|
|
// Match prepareIsolatedCodexHome: the discovery root is private/writable
|
|
// for Codex's own built-ins; each selected package remains sealed.
|
|
await input.stageDirectory({ sourcePath, targetPath: posix.join(directory.targetPath!, "skills"), mode: 0o700 });
|
|
});
|
|
};
|
|
// Resolve the owned home and remove obsolete assignments before any launch
|
|
// file can be written through a retained, redirected path.
|
|
await stageCodexSkills();
|
|
if (input.transport === "sandbox" && input.runner.allowConcurrentSyncOperations === true) {
|
|
const outcomes = await Promise.allSettled([materializeLaunchState(), stageRuntimeContext()]);
|
|
for (const outcome of outcomes) {
|
|
if (outcome.status === "rejected") throw outcome.reason;
|
|
}
|
|
} else {
|
|
await materializeLaunchState();
|
|
await stageRuntimeContext();
|
|
}
|
|
if (input.caBundleMapping) {
|
|
const mapping = input.caBundleMapping;
|
|
await measure("stage.asset.ca_bundle", () => input.stageFile({ ...mapping, mode: 0o600 }));
|
|
}
|
|
}
|