From 7f66a30ad7723c1fbcb6ef64b3281eaebddada29 Mon Sep 17 00:00:00 2001 From: Dotta Date: Tue, 6 Oct 2026 12:11:15 -0500 Subject: [PATCH] fix(runner): keep sandbox GitHub housekeeping in runner runtime Use one runtime path for sandbox launcher staging, callback transport, and cleanup. Preserve the native observer scope and retained failures. Co-Authored-By: Paperclip --- .../2026-10-02-pi-production-readiness.md | 74 ++++++++++++++++--- .../src/execution-target.test.ts | 2 +- .../adapter-utils/src/execution-target.ts | 6 +- .../src/github-launcher-environment.test.ts | 33 +++++++++ .../native-runtime/native-github-access.ts | 4 +- tests/runner-e2e/README.md | 5 +- 6 files changed, 106 insertions(+), 18 deletions(-) diff --git a/doc/plans/2026-10-02-pi-production-readiness.md b/doc/plans/2026-10-02-pi-production-readiness.md index a7e91b1e8a5..85450e95f45 100644 --- a/doc/plans/2026-10-02-pi-production-readiness.md +++ b/doc/plans/2026-10-02-pi-production-readiness.md @@ -5,7 +5,58 @@ set of release gates. Pi remains a candidate until every required gate passes. Do not expand this work to additional models, widgets, images, Cursor or Copilot qualification. The existing draft stack must be reviewed in dependency order. -## Rebuilt candidate and exact write arguments — 2026-10-06 +## Sandbox GitHub housekeeping correction — 2026-10-06 + +The next shipping correction moves sandbox GitHub launchers, upload locks and +per-command configuration below `.paperclip-runtime/paperclip-runner/github`. +One shared path now binds staging, the native callback bridge and cleanup. +Local and SSH locations keep their existing behavior. The native watcher, +excluded subtree, exact byte grades, timeouts and lifecycle assertions do not +change. Shipping inputs change, so rebuild and freeze packages and the Linux +image before any further paid qualification. The new set starts at Product +0/26 and Runner 0/7; do not carry passes across changed shipping inputs. + +The retained `0cff2b20b` set has Product 1/26 and Runner 3/7 passes. Its corrected +memory case writes exactly 33 bytes with LF, durably saves them and reads them +from a fresh task after server restart. Runner context, context-before-action +and document creation pass. The finish-task case applies exactly one successful +task mutation but never reaches a terminal turn before its unchanged 120-second +deadline. Its cause remains unproven. The corrected Linux restart retains its +pending request and accepts the exact browser answer, but still fails terminal +evidence and canonical cleanup. Its new closed diagnostic is +`unwatched_directory`; all observed runner processes have retired. Independent +cleanup proves the child sandbox, controller process and temporary root absent. +Neither failure is regraded or retried unchanged. + +Two zero-model Linux controls identify a concrete housekeeping defect. Restaging +the same launcher bytes creates temporary upload locks outside the excluded +runtime and reproduces `unwatched_directory` with 22 workspace mutations. A Git +command creates the same failure with five mutations. Both processes retire. +With the correction, the unchanged actual Linux observer returns complete +terminal evidence for both controls; launcher bytes and user files stay intact. +All 35 affected Linux lifecycle and launcher tests pass, including the real +watcher regression and its outside-runtime negative control. All 60 affected +Mac launcher, credential and native-session lifecycle tests pass; the Linux +inotify regression is platform-specific. Shared and adapter-utils compile. +This proves the correction, not the cause of an unrecorded path in an older +failure. A fresh live restart measurement is still required after the rebuild. + +The old-set Linux build and recursive typecheck pass. Its full test command +retains 15,076 passes, 57 skips and one disk-reserve failure. Removing only the +unused owned pnpm download store restores space without changing installed +files, source or assertions. The failed test then passes, and the canonical +full test command runs again with its unchanged 90-minute deadline. The owned +sandbox retention covers that command; host settings and unrelated services +remain untouched. Latest-head checks and the complete commands must also pass +for the newly committed source. Keep all old results as historical evidence. + +No paid attempt runs during this correction. The last provisional dedicated-key +accounting is $0.34323782 used of its $5 lifetime cap. The signed-in default +workspace still has 72 unconfigured BYOK providers. Keep the approved $100 +campaign cap, frozen model, low thinking and zero automatic paid retries. +Rollout remains held. Do not merge or release. + +## Historical rebuilt candidate and exact write arguments — 2026-10-06 Shipping source is now `0cff2b20b5ea785880ad119dde59e6f011a3f9e1`. All 18 public workspace tarballs are freshly packed from that source. The Linux @@ -1384,17 +1435,16 @@ Keep rollout held until every release gate is proven. ## Remaining work in order -1. Isolate the two current failed cases using retained evidence and zero-model - checks. Memory omits the final LF despite intact prompt bytes; the frozen - parser/validator/write/read path preserves it. Restart lacks a complete - independent terminal receipt despite a successful original run. The new - capture order makes future failures inspectable but alone permits no paid - retry. Do not relax graders or replace a case with a passing surrogate. -2. After a concrete correction, run the affected exact case once. Then complete - all 26 Product and seven Runner cases on the source-bound profile-15 runtime, - and fresh Intel public install/admission. Reuse only proven equivalent inputs - with exact recorded source and definition identities. Keep original failures, - zero automatic retries and the approved lifetime/campaign spending limits. +1. Build and freeze the corrected sandbox GitHub runtime packages and image. + Complete the normal platform installations and zero-model admission. Preserve + the old-set memory pass and restart/finish-task failures. Identify a concrete + correction for the finish-task terminal timeout before another paid attempt + of that failed case. Do not relax graders or use a passing surrogate. +2. Run the corrected exact restart case once, then complete all 26 Product and + seven Runner cases on one new frozen source-bound profile-15 set. Require the + exact LF and fresh-task memory proof again on that set. Verify the Intel + package path and distinguish Rosetta from physical Intel hardware evidence. + Keep original failures, zero automatic retries and the approved spending caps. 3. Finish latest-head CI, full workspace verification, current review and the prerequisite admission finding. Keep rollout held until every gate passes. No merge or release is authorized. diff --git a/packages/adapter-utils/src/execution-target.test.ts b/packages/adapter-utils/src/execution-target.test.ts index 5d89d97d3f5..46a4df7d6d5 100644 --- a/packages/adapter-utils/src/execution-target.test.ts +++ b/packages/adapter-utils/src/execution-target.test.ts @@ -431,7 +431,7 @@ describe("GitHub launcher lifecycle", () => { providerKey: "e2b", remoteCwd: "/remote/workspace", runner }; await cleanupGitHubOperationLaunchers({ runId: "finished-run", target }); expect(runner.execute).toHaveBeenCalledWith({ command: "sh", - args: ["-c", "rm -rf -- '/remote/workspace/.paperclip-runtime/github/finished-run'"], + args: ["-c", "rm -rf -- '/remote/workspace/.paperclip-runtime/paperclip-runner/github/finished-run'"], cwd: "/remote/workspace", timeoutMs: 5_000 }); await expect(cleanupGitHubOperationLaunchers({ runId: "../other", target })).rejects.toThrow("Invalid GitHub launcher run ID"); expect(runner.execute).toHaveBeenCalledTimes(1); diff --git a/packages/adapter-utils/src/execution-target.ts b/packages/adapter-utils/src/execution-target.ts index 07c122bde7b..de82970c8f6 100644 --- a/packages/adapter-utils/src/execution-target.ts +++ b/packages/adapter-utils/src/execution-target.ts @@ -1559,11 +1559,13 @@ type GitHubLauncherLocation = { runId: string; target: AdapterExecutionTarget | null | undefined; }; -function githubOperationLauncherDirectory(input: GitHubLauncherLocation): string { +/** Keep sandbox housekeeping within the producer-owned Runner runtime. */ +export function githubOperationLauncherDirectory(input: GitHubLauncherLocation): string { // Only controller-generated run IDs may name a removable directory. if (!/^[a-zA-Z0-9_-]+$/.test(input.runId)) throw new Error("Invalid GitHub launcher run ID"); return input.target?.kind === "remote" - ? path.posix.join(input.target.remoteCwd, ".paperclip-runtime", "github", input.runId) + ? path.posix.join(input.target.remoteCwd, ".paperclip-runtime", + ...(input.target.transport === "sandbox" ? ["paperclip-runner"] : []), "github", input.runId) : path.join(os.tmpdir(), "paperclip-github-runtime", input.runId); } diff --git a/packages/adapter-utils/src/github-launcher-environment.test.ts b/packages/adapter-utils/src/github-launcher-environment.test.ts index 2dab7d4af0f..7259d65451b 100644 --- a/packages/adapter-utils/src/github-launcher-environment.test.ts +++ b/packages/adapter-utils/src/github-launcher-environment.test.ts @@ -1,4 +1,5 @@ import { execFile } from "node:child_process"; +import { watch } from "node:fs"; import { mkdtemp, mkdir, readFile, realpath, rm, writeFile } from "node:fs/promises"; import os from "node:os"; import path from "node:path"; @@ -67,6 +68,38 @@ async function sandbox(layout: string) { } describe("managed GitHub launcher environment", () => { + // The Daytona native directory oracle uses Linux inotify. Run its real + // filesystem regression on that platform; shell/launcher tests cover both. + it.runIf(process.platform === "linux")("keeps launcher restaging and command scratch inside the runner runtime", async () => { + const fixture = await sandbox("usr/bin"); + const privateRoot = path.join(fixture.root, ".paperclip-runtime"); + await mkdir(path.join(privateRoot, "paperclip-runner"), { recursive: true }); + await writeFile(path.join(fixture.root, "user.txt"), "Existing work\n"); + const mutations: string[] = []; + const watchers = [fixture.root, privateRoot].map((directory) => watch(directory, (_kind, name) => { + const changed = path.relative(fixture.root, path.join(directory, String(name))); + if (changed !== ".paperclip-runtime/paperclip-runner") mutations.push(changed); + })); + try { + const input = { runId: "restart-same-launchers", target: fixture.target, cwd: fixture.root, + env: githubBrokerEnvironment({}, { url: "", token: "" }) }; + const first = await prepareGitHubOperationLaunchers(input); + const original = await readFile(path.join(first.PAPERCLIP_GITHUB_LAUNCHER_DIR, "git"), "utf8"); + const restored = await prepareGitHubOperationLaunchers(input); + const command = await fixture.runner.execute({ + command: path.join(restored.PAPERCLIP_GITHUB_LAUNCHER_DIR, "git"), args: ["--version"], env: restored, + }); + expect(command.exitCode, command.stderr).toBe(0); + expect(await readFile(path.join(restored.PAPERCLIP_GITHUB_LAUNCHER_DIR, "git"), "utf8")).toBe(original); + expect(await readFile(path.join(fixture.root, "user.txt"), "utf8")).toBe("Existing work\n"); + await new Promise((resolve) => setTimeout(resolve, 50)); + expect(mutations).toEqual([]); + // The watcher must still notice housekeeping outside that exact runtime. + await mkdir(path.join(privateRoot, "outside-runtime-control")); + await vi.waitFor(() => expect(mutations).toContain(".paperclip-runtime/outside-runtime-control")); + } finally { for (const watcher of watchers) watcher.close(); } + }); + it.each(["module", "commonjs"])("runs managed GitHub launchers inside a %s project", async (type) => { const fixture = await sandbox("usr/bin"); const packageJson = JSON.stringify({ type }); diff --git a/server/src/services/native-runtime/native-github-access.ts b/server/src/services/native-runtime/native-github-access.ts index f0245291f41..a9711329ff3 100644 --- a/server/src/services/native-runtime/native-github-access.ts +++ b/server/src/services/native-runtime/native-github-access.ts @@ -1,8 +1,8 @@ import { randomBytes, randomUUID, timingSafeEqual } from "node:crypto"; import { createServer } from "node:http"; -import path from "node:path"; import { cleanupGitHubOperationLaunchers, + githubOperationLauncherDirectory, prepareGitHubOperationLaunchers, startAdapterExecutionTargetPaperclipBridge, } from "@paperclipai/adapter-utils/execution-target"; @@ -98,7 +98,7 @@ export async function createNativeGitHubAccess(input: { bridge = await startBridge({ ...location, runtimeRootDir: input.target?.kind === "remote" - ? path.posix.join(input.target.remoteCwd, ".paperclip-runtime", "github", location.runId) + ? githubOperationLauncherDirectory(location) : null, adapterKey: "native-github", hostApiToken: token, diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index d422a6853e3..466881849eb 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -1583,7 +1583,10 @@ The observer watches registered targets (including transient create/delete), use workspace changes and the exact runner process plus descendants. The single controller-owned `.paperclip-runtime/paperclip-runner` subtree is excluded from user-file inventory and mutation counts: it holds the binary, provider pack, -context and active runtime state. Its location and identity remain checked, and +context, active runtime state, and sandbox GitHub launchers, upload locks and +per-command configuration. Launcher restaging after controller recovery uses +that same runtime path. SSH and local launcher locations remain separate. +Its location and identity remain checked, and test targets cannot use it. This exclusion is recorded in evidence; it does not claim that runtime-internal writes are covered by the user-file oracle.