From 784bc06276d93c608ea7ee082e2f5cb5e8e8fe26 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 16 Nov 2026 19:56:57 -0600 Subject: [PATCH] perf(runner): reuse the sandbox's identical runnerd instead of uploading it on every cold boot (#21949) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Remote agent runs start `paperclip-runnerd` inside a sandbox, such as a Daytona sandbox > - Agent boot time is a measured product constraint, and cold boot on Daytona grew from about 11 s to 25–75 s in restart-chaos waves > - The span that grew is `runner.artifact.prepare`: the controller uploaded its 31 MB `paperclip-runnerd` to every new sandbox > - The controller uploads this file whenever `PAPERCLIP_RUNNER_REMOTE_BINARY_PATH` is set, even when the sandbox image already has the same bytes > - The upload time depends on the sandbox file-upload throughput. That throughput changed from about 1 s to 5–48 s for the same 31 MB, on the same image and with no runner change > - This pull request makes the controller use the image's `paperclip-runnerd` when its SHA-256 equals the configured artifact > - The benefit is a cold boot that does not depend on sandbox upload throughput: about 9–11 s instead of 33–45 s in the same hour ## Linked Issues or Issue Description No issue exists. This describes the bug, following `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** Cold `agent_boot` on Daytona became 2–7 times slower. In restart-chaos waves on master `c5218f5ec4`, the cold boot p50 was about 11 s for claude, about 10 s for codex and 15–25 s for OpenCode. On master `3452639a81` it was 25 s, then 55 s, then 70–78 s for claude and codex over five waves. OpenCode reached 238–265 s. The run spans show that the extra time for every adapter is in `runner.artifact.prepare` (codex example: 42.7 s, before it was 1.2 s). In this span the controller uploads its own 31 MB `paperclip-runnerd` to the new sandbox. `PAPERCLIP_RUNNER_REMOTE_BINARY_PATH` is set, so `mayUsePreinstalledRunnerArtifact` returned `false` and the controller never used the identical `paperclip-runnerd` in the sandbox image. The upload time follows the sandbox upload throughput, which changes over time. None of the 17 commits between the two masters changed this path. **Expected behavior** When the sandbox already has a `paperclip-runnerd` with the same SHA-256 as the configured artifact, the controller links it and does not upload it. Cold boot does not depend on upload throughput. **Steps to reproduce** 1. Set `PAPERCLIP_RUNNER_REMOTE_BINARY_PATH` to the `paperclip-runnerd` from the Daytona runner image. 2. Start a cold agent run in a new Daytona sandbox from that image. 3. Look at the run spans. `runner.artifact.prepare` holds a full upload of the runner (5–48 s in our waves), and there is no `runner.artifact.discover` span. **Paperclip version or commit** `3452639a81` (also current master `6205b24d54`). Image `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:710deab7375a…`. **Agent adapter(s) involved** All remote adapters: claude, codex and OpenCode. ## What Changed - `mayUsePreinstalledRunnerArtifact` accepts a preinstalled `paperclip-runnerd` when an artifact is configured and the controller has that artifact on disk. - In that case the existing SHA-256 check (`runner_remote_preinstalled_artifact_mismatch`) is mandatory. A different, missing, or unreadable image copy still falls back to the upload. - When an artifact is configured, a failed discovery probe counts as "no preinstalled runnerd", so the controller uploads the configured file as before. - Tests: a unit case for the gate, and three cases in the runner wiring test: identical configured artifact (no `syncIn`), mismatched configured artifact (upload), and failed discovery with a configured artifact (upload). - `docs/deploy/environment-variables.md` describes the new behavior. ## Verification - Red/green: `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "server-owned replacement artifact|contract-compatible runnerd|configured artifact the controller holds"`. Without the fix 3 tests fail. With the fix 27 pass. - `pnpm verify` passes. - Restart-chaos waves on Daytona, image `710deab7375a`, 6 tasks (2 claude, 2 codex, 2 opencode), A1/K0, run back to back in one hour: | Wave | Fix | claude cold boot p50 | codex | opencode | `runner.artifact.prepare` | | --- | --- | --- | --- | --- | --- | | w-20261117T005138 | no | 38.4 s | 33.2 s | 36.0 s | not recorded | | w-20261117T005627 | yes | 9.5 s | 8.8 s | 10.6 s | 0.5–0.6 s, all runs use the image copy | | w-20261117T005956 | no | 44.1 s | 13.1 s (max 47.1 s) | 45.5 s | 4.9–37.8 s upload | All 18 tasks passed. ## Risks - Low risk. The controller uses the image copy only when its SHA-256 equals the configured artifact, so the runner bytes do not change. Every other case keeps the current upload. - One extra `command -v` probe and one `sha256sum` probe run when an artifact is configured. Each takes about 0.1 s. - Saved run state is not changed. A run saved by the previous release resumes with the same runner identity, because the bytes are the same. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- docs/deploy/environment-variables.md | 2 +- .../native-runtime/native-session-executor.test.ts | 12 +++++++++--- .../native-runtime/native-session-executor.ts | 13 +++++++++---- 3 files changed, 19 insertions(+), 8 deletions(-) diff --git a/docs/deploy/environment-variables.md b/docs/deploy/environment-variables.md index 70d804fc346..2489a0bd880 100644 --- a/docs/deploy/environment-variables.md +++ b/docs/deploy/environment-variables.md @@ -22,7 +22,7 @@ All environment variables that Paperclip uses for server configuration. | `PAPERCLIP_CHAT_WEBHOOK_PUBLIC_URL` | (board public origin) | Optional HTTPS origin for native chat provider webhooks when ingress and the board use different hosts. Must have no credentials, path, query, or fragment; invalid configuration refuses startup. Used only for provider callback URLs, not board links, authentication, trusted hosts, or identity confirmation. In `authenticated` mode with `private` exposure, also add the ingress hostname to `PAPERCLIP_ALLOWED_HOSTNAMES`. Otherwise the private-hostname guard rejects provider callbacks with 403 before they reach the webhook route. The guard reads `X-Forwarded-Host` before `Host`, so a reverse proxy that rewrites `Host` but forwards the original `X-Forwarded-Host` still gets 403. | | `PAPERCLIP_RUNNER_PUBLIC_URL` | (unset) | Explicit `wss://` base URL used only when a remote `paperclip_runner` target dials Paperclip directly. Paperclip appends `/api/runner/v1/connect/`; the reverse proxy must forward WebSocket upgrades for that route. This value is never inferred from request headers. Daytona ignores it and uses provider ingress. | | `PAPERCLIP_RUNNER_CA_BUNDLE_PATH` | (unset) | Optional PEM CA bundle for direct runner WSS. Platform roots remain enabled. There is no insecure TLS bypass. | -| `PAPERCLIP_RUNNER_REMOTE_BINARY_PATH` | (host build) | Host-local path to a `paperclip-runnerd` artifact built for the remote target OS and architecture. Required when Paperclip and the remote sandbox do not share a compatible platform; build metadata and the required transport mode are verified before launch. | +| `PAPERCLIP_RUNNER_REMOTE_BINARY_PATH` | (host build) | Host-local path to a `paperclip-runnerd` artifact built for the remote target OS and architecture. Required when Paperclip and the remote sandbox do not share a compatible platform; build metadata and the required transport mode are verified before launch. When the sandbox image already has a `paperclip-runnerd` with the same SHA-256, Paperclip uses that copy and does not upload this file. | | `PAPERCLIP_RUNNER_REMOTE_CODEX_PATH` | (unset) | Optional host-local path to a Codex executable built for the remote target OS and architecture. For remote Codex-backed runners, Paperclip stages and verifies this executable beside `paperclip-runnerd`. | | `PAPERCLIP_RUNNER_REMOTE_CODEX_NPM_SPEC` | (unset) | Optional pinned npm package spec (for example, `@openai/codex@0.160.0`) installed inside each fresh remote lease when its Codex harness is not baked into the sandbox image. Mutually exclusive with `PAPERCLIP_RUNNER_REMOTE_CODEX_PATH`; Paperclip verifies the installed executable before starting `runnerd`. | | `PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH` | `/opt/paperclip-runner/provider-pack` in Docker; otherwise unset | Host-local path to the immutable provider pack built by `pnpm --filter @paperclipai/paperclip-runner build:provider-pack`. Stamped standard Docker images include the pack; downstream compositions and the `cloud` target inherit it. Unstamped local Docker builds skip pack generation. The pack includes its target-built Node 24.11+ runtime, locked production dependencies, OpenCode proxy/executable, and ACPX sidecar. Remote OpenCode and ACPX fail closed without it. A preinstalled pack is accepted only when its complete digested manifest matches this build-owned pack; otherwise Paperclip stages this pack into the sandbox. | diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index 13fb4614f48..0dd70c67740 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -5400,6 +5400,11 @@ describe("remote preinstalled executable discovery", () => { expect(mayUsePreinstalledRunnerArtifact(" ")).toBe(true); expect(mayUsePreinstalledRunnerArtifact(undefined)).toBe(true); }); + + it("checks a preinstalled runnerd against a configured artifact the controller holds", () => { + expect(mayUsePreinstalledRunnerArtifact("/artifacts/paperclip-runnerd", true)).toBe(true); + expect(mayUsePreinstalledRunnerArtifact("/artifacts/paperclip-runnerd", false)).toBe(false); + }); }); describe("remote runner build metadata", () => { @@ -23175,7 +23180,7 @@ describe("runnerd provider runtime wiring", () => { }); it.each([ - ...["current", "preinstalled-exact", "preinstalled-mismatch", "preinstalled-error", "preinstalled-timeout", "stale", "missing", "retained", "retained-mismatch", "retained-error", "retained-timeout", "retained-explicit"] + ...["current", "preinstalled-exact", "preinstalled-mismatch", "preinstalled-error", "preinstalled-timeout", "preinstalled-explicit", "preinstalled-explicit-mismatch", "preinstalled-explicit-discover-fails", "stale", "missing", "retained", "retained-mismatch", "retained-error", "retained-timeout", "retained-explicit"] .map((image) => ({ image, version: "0.160.0" })), ...["0.149.0", "0.149.1", "0.153.4", "0.156.0", "0.160.1"] .map((version) => ({ image: "current", version })), @@ -23184,7 +23189,7 @@ describe("runnerd provider runtime wiring", () => { ])("uses shared Codex and the server-owned replacement artifact (image=$image, Codex=$version)", async ({ image, version }) => { const retained = image.startsWith("retained"); const exactRetained = image === "retained" || image === "retained-explicit"; - const needsReplacement = image !== "current" && image !== "preinstalled-exact" && !exactRetained; + const needsReplacement = image !== "current" && image !== "preinstalled-exact" && image !== "preinstalled-explicit" && !exactRetained; // The mocked remote executes metadata probes; artifact staging only needs bytes. // Keep this regression independent of a locally compiled Rust runner binary. const controllerArtifact = join(isolatedStateDirectory, "paperclip-runnerd"); @@ -23225,6 +23230,7 @@ describe("runnerd provider runtime wiring", () => { } else if (script === "uname -s; uname -m") { stdout = `${process.platform === "darwin" ? "Darwin" : "Linux"}\n${process.arch === "arm64" ? "arm64" : "x86_64"}\n`; } else if (script.includes("command -v paperclip-runnerd")) { + if (image === "preinstalled-explicit-discover-fails") throw new Error("sandbox provider unavailable"); stdout = image === "missing" ? "" : "/usr/local/bin/paperclip-runnerd\n"; } else if (script.includes("command -v codex")) { stdout = script.includes("/opt/paperclip-runner/bin/codex") @@ -23250,7 +23256,7 @@ describe("runnerd provider runtime wiring", () => { execution, runnerInstanceId: "runner-image-runtime", onLog, - ...(image === "retained-explicit" ? { runnerRemoteBinaryPath: controllerArtifact } : {}), + ...(image.includes("explicit") ? { runnerRemoteBinaryPath: controllerArtifact } : {}), runnerIngressAuthorized: true, runnerExecutionTarget: { kind: "remote", diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index c4a80912ee2..bee52eec15b 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -14239,8 +14239,9 @@ export function buildRemoteCodexLauncherCommand( export function mayUsePreinstalledRunnerArtifact( configuredRemoteBinaryPath: string | null | undefined, + configuredArtifactAvailable = false, ): boolean { - return !configuredRemoteBinaryPath?.trim(); + return !configuredRemoteBinaryPath?.trim() || configuredArtifactAvailable; } const RUNNERD_BUILD_METADATA_SCHEMA = @@ -19306,11 +19307,15 @@ export function createRemoteNativeArtifactPreparation(input: { ); } const explicitRemoteBinary = input.runnerRemoteBinaryPath?.trim() || null; - if (!runnerArtifactPrepared && mayUsePreinstalledRunnerArtifact(explicitRemoteBinary)) { + const controllerArtifactAvailable = existsSync(controllerRunnerBinary); + if (!runnerArtifactPrepared && + mayUsePreinstalledRunnerArtifact(explicitRemoteBinary, controllerArtifactAvailable)) { const preinstalledRunner = await measureNativeRunnerSpan( input.trace, "runner.artifact.discover", - () => discoverPreinstalledExecutable("paperclip-runnerd"), + () => explicitRemoteBinary + ? discoverPreinstalledExecutable("paperclip-runnerd").catch(() => null) + : discoverPreinstalledExecutable("paperclip-runnerd"), ); if (preinstalledRunner) { try { @@ -19319,7 +19324,7 @@ export function createRemoteNativeArtifactPreparation(input: { "runner.artifact.verify_preinstalled", () => verifyRemoteRunner(requiredMode, preinstalledRunner), ); - if (existsSync(controllerRunnerBinary) && + if (controllerArtifactAvailable && !await matchesControllerRunnerArtifact(preinstalledRunner)) { throw new Error("runner_remote_preinstalled_artifact_mismatch"); }