From 572bef4ba7679451ac1543db25cb8ebd533bf919 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Tue, 20 Oct 2026 16:01:48 -0700 Subject: [PATCH] fix: omit re-downloadable Codex remote plugins from failover backups (#17026) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native Codex sessions in remote sandboxes copy the sandbox's Codex home into a host-side failover backup after every turn > - Ephemeral sandboxes restore that backup at the start of every turn, so its size affects both disk use and turn latency > - Most of each backup is remote plugin data that Codex itself treats as a cache and re-downloads on startup > - This pull request leaves that data out of Codex checkpoints, keeping everything a resumed session needs > - The benefit is backups roughly 80–90% smaller, less data moved per turn, and much slower disk growth ## Linked Issues or Issue Description Bug report described in-PR (bug report template). Related: #16866 (expires failover backups after their issues close). **What happened?** Failover backups grow quickly because each Codex checkpoint includes `$CODEX_HOME` remote plugin data. On one instance, 204 backups held 22.1 GB: - `codex-home/plugins/cache/openai-curated-remote`: 8.3 GB (about 29 MB per backup); - `codex-home/cache/remote_plugin_catalog`: 6.1 GB (one 38 MB catalog file per backup); - `codex-home/sessions` (the conversation history): 0.31 GB. About 80 new backups (9.8 GB) were written per day. Ephemeral sandboxes also copy all of this into a fresh sandbox before each turn. **Expected behavior** Checkpoints keep what a resumed session needs and omit data that Codex re-downloads itself. **Steps to reproduce** 1. Run a Codex agent with ChatGPT auth in a remote sandbox environment. 2. Inspect `/runtime/paperclip-runner/durable-sessions/*/failover-backups/current/codex-home`. 3. Most bytes are under `plugins/cache/openai-curated-remote` and `cache/remote_plugin_catalog`. **Paperclip version or commit:** `master` at f89c37791c5. **Deployment mode:** server with remote sandbox environments. **Agent adapter(s) involved:** Codex (native app-server and ACPX). ## What Changed - **`server/src/services/native-runtime/native-session-executor.ts`:** Codex checkpoints (direct and under ACPX) now also exclude: - `plugins/cache/` for Codex's six remote marketplaces (`openai-curated-remote`, `created-by-me-remote`, `workspace-directory`, `workspace-shared-with-me`, `workspace-shared-with-me-private`, `workspace-shared-with-me-unlisted`); - `cache/remote_plugin_catalog`; - `plugins/.remote-plugin-install-staging`, and `.remote-plugin-install-staging` (the location older Codex versions use). - **Still kept:** local and bundled plugin caches, `plugins/data`, the Apps tool cache (`cache/codex_apps_tools`, which avoids waiting up to 30 s for the Apps MCP server), session rollouts, and the state databases. - **Tests:** a real-tar checkpoint test covers kept and omitted entries; the profile expectations are updated. - **`doc/recoverable-disk-cleanup.md`:** describes what Codex failover copies omit and the first-turn effect. - No code comments were added. **Why the omitted data is safe to drop** (Codex source at commit d4e03acb): - **Server-side source of truth:** the installed remote plugin list comes from the server (`/backend-api/ps/plugins/installed`) and is held in memory; these directories are not the source of truth. - **Self-healing startup:** at `app-server` startup, a background task reconciles missing bundles and force-refreshes the catalog (`core-plugins/src/manager.rs`, `remote_installed_plugin_sync.rs`). - **No path dependencies:** rollouts and state store plugin ids, not paths into these directories. ## Verification - `pnpm exec vitest run src/services/native-runtime/native-session-executor.test.ts` (in `server/`): 766 passed. - **Mutation check:** dropping one remote marketplace from the exclusion list fails both the new checkpoint test and the profile test. - `pnpm run typecheck` (in `server/`): pass. - **Manual resume with Codex CLI 0.162.1 and ChatGPT auth:** 1. Started a session in a fresh `CODEX_HOME`. This downloaded 28 MB of remote plugins. 2. Copied the home without the excluded entries: 34 MB became 4.9 MB. 3. Ran `codex exec resume --last`. It recalled the conversation in 3 s, and the remote plugins were re-downloaded in the background without errors. - **Compatibility:** restore compares only top-level directory names (`compatibleNativeHarnessBackupManifests`), so existing backups that still contain the plugins restore unchanged. ## Risks - **First turn after a restore:** remote plugin skills and tools can be missing until the background download finishes, typically seconds. Codex leaves an unavailable remote plugin out of a turn; it does not fail the turn. Later turns include it. - **Network and auth:** re-downloading requires ChatGPT auth and network access from the sandbox, which remote plugins require anyway, since Codex fetches the installed list from the server on every start. - **Per-turn download:** ephemeral sandboxes download the remote plugin bundles each turn, about 28 MB in the observed case, in the background. In exchange, about 90 MB less is copied from the host before each turn. - **Rollback:** revert. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code with tool use (shell, file edits, tests, Codex source reading, a local Codex resume test, read-only measurements on a live instance). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/recoverable-disk-cleanup.md | 9 ++- .../native-session-executor.test.ts | 72 ++++++++++++++++++- .../native-runtime/native-session-executor.ts | 22 +++++- 3 files changed, 97 insertions(+), 6 deletions(-) diff --git a/doc/recoverable-disk-cleanup.md b/doc/recoverable-disk-cleanup.md index 81c3e6cb532..e6e450bd37d 100644 --- a/doc/recoverable-disk-cleanup.md +++ b/doc/recoverable-disk-cleanup.md @@ -51,8 +51,13 @@ this worker never scans or removes them. After each remote turn, the server copies the sandbox's harness state (runner state plus the provider's session home, such as `codex-home`) to `/runtime/paperclip-runner/durable-sessions//failover-backups/current`. -A replacement sandbox restores the session from this copy. Each session keeps one -copy, and a single copy can reach several gigabytes. +A replacement sandbox restores the session from this copy, and an ephemeral sandbox +restores it at the start of every turn. Each session keeps one copy, and a single +copy can reach several gigabytes. Codex copies omit credentials, scratch +directories, and data Codex downloads again on startup: remote plugin bundles under +`plugins/cache/`, the remote plugin catalog cache, and plugin +install staging. After a restore, remote plugins can be unavailable to a turn until +Codex finishes downloading them. An hourly background sweep expires the copy once the session can only resume by reopening closed work. It checks at most 200 sessions and expires at most 50 per diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index c1286212309..d2b6b7eeb21 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -1787,12 +1787,26 @@ describe("native harness persistence profiles", () => { }, ); - it("excludes disposable Codex scratch trees and launch-time credentials", () => { + it("excludes disposable Codex scratch trees, launch-time credentials and re-downloadable remote plugins", () => { const codex = profile({ kind: "codex" }, "codex_app_server"); expect( codex.directories.find((directory) => directory.name === "codex-home"), ).toMatchObject({ - excludeEntries: ["tmp", ".tmp", "auth.json", "config.toml"], + excludeEntries: [ + "tmp", + ".tmp", + "auth.json", + "config.toml", + ".remote-plugin-install-staging", + "cache/remote_plugin_catalog", + "plugins/.remote-plugin-install-staging", + "plugins/cache/openai-curated-remote", + "plugins/cache/created-by-me-remote", + "plugins/cache/workspace-directory", + "plugins/cache/workspace-shared-with-me", + "plugins/cache/workspace-shared-with-me-private", + "plugins/cache/workspace-shared-with-me-unlisted", + ], }); }); @@ -1807,6 +1821,15 @@ describe("native harness persistence profiles", () => { `acpx/${sessionDirectory}/codex-home/.tmp`, `acpx/${sessionDirectory}/codex-home/auth.json`, `acpx/${sessionDirectory}/codex-home/config.toml`, + `acpx/${sessionDirectory}/codex-home/.remote-plugin-install-staging`, + `acpx/${sessionDirectory}/codex-home/cache/remote_plugin_catalog`, + `acpx/${sessionDirectory}/codex-home/plugins/.remote-plugin-install-staging`, + `acpx/${sessionDirectory}/codex-home/plugins/cache/openai-curated-remote`, + `acpx/${sessionDirectory}/codex-home/plugins/cache/created-by-me-remote`, + `acpx/${sessionDirectory}/codex-home/plugins/cache/workspace-directory`, + `acpx/${sessionDirectory}/codex-home/plugins/cache/workspace-shared-with-me`, + `acpx/${sessionDirectory}/codex-home/plugins/cache/workspace-shared-with-me-private`, + `acpx/${sessionDirectory}/codex-home/plugins/cache/workspace-shared-with-me-unlisted`, ], }); expect( @@ -2530,6 +2553,51 @@ describe("remote provider checkpoint snapshots", () => { expect(await readFile(join(home, "logs", "unified.jsonl"), "utf8")).toBe("AUTH_SENTINEL"); } finally { await rm(root, { recursive: true, force: true }); } }); + it("keeps Codex session state while omitting re-downloadable remote plugins from the checkpoint", async () => { + const root = await mkdtemp(join(tmpdir(), "codex-plugin-checkpoint-")); + try { + const source = join(root, "codex-home"); + const target = join(root, "backup"); + const files: Record = { + "sessions/2026/10/20/rollout.jsonl": "resume-state", + "state_5.sqlite": "state", + "cache/codex_apps_tools/tools.json": "apps", + "cache/remote_plugin_catalog/catalog.json": "catalog", + "plugins/data/sites/data.json": "plugin-data", + "plugins/cache/openai-bundled/browser/1.0.0/plugin.json": "bundled", + "plugins/cache/openai-curated-remote/sites/1.0.2/plugin.json": "remote", + "plugins/cache/workspace-shared-with-me/notes/0.1.0/plugin.json": "remote", + "plugins/.remote-plugin-install-staging/partial/plugin.json": "staging", + ".remote-plugin-install-staging/partial/plugin.json": "staging", + }; + for (const [path, contents] of Object.entries(files)) { + await mkdir(join(source, path, ".."), { recursive: true }); + await writeFile(join(source, path), contents); + } + const codexProfile = resolveNativeHarnessPersistenceProfile({ + provider: { kind: "codex" }, + session: { normalizedSessionId: "session", driverKind: "codex_app_server" }, + } as unknown as NativeExecutionInputV1); + const execute = async ({ command, args }: { command: string; args: string[] }) => ({ + exitCode: 0, timedOut: false, stdout: execFileSync(command, args, { encoding: "utf8" }), stderr: "", + }); + const syncOut = async (operations: Array<{ files: Array<{ sourcePath: string; targetPath: string }> }>) => { + for (const operation of operations) for (const file of operation.files) + await cp(file.sourcePath, file.targetPath, { recursive: true }); + }; + await syncRemoteRunnerDirectoryOut({ runner: { execute, syncOut } as never, + sourcePath: source, targetPath: target, mode: 0o700, + excludeEntries: codexProfile.directories.find(directory => directory.name === "codex-home")!.excludeEntries }); + for (const kept of ["sessions/2026/10/20/rollout.jsonl", "state_5.sqlite", "cache/codex_apps_tools/tools.json", + "plugins/data/sites/data.json", "plugins/cache/openai-bundled/browser/1.0.0/plugin.json"]) + expect(await readFile(join(target, kept), "utf8")).toBe(files[kept]); + for (const omitted of ["cache/remote_plugin_catalog", "plugins/cache/openai-curated-remote", + "plugins/cache/workspace-shared-with-me", "plugins/.remote-plugin-install-staging", ".remote-plugin-install-staging"]) + await expect(access(join(target, omitted))).rejects.toMatchObject({ code: "ENOENT" }); + expect(await readFile(join(source, "plugins/cache/openai-curated-remote/sites/1.0.2/plugin.json"), "utf8")).toBe("remote"); + } finally { await rm(root, { recursive: true, force: true }); } + }); + it("excludes Codex scratch and credential state without mutating the live provider home", async () => { const execute = vi .fn() diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index b28eec17206..fbe8d832d1d 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -336,6 +336,24 @@ const CODEX_HOME_NON_PERSISTENT_ENTRIES = [ "auth.json", "config.toml", ] as const; +const CODEX_REMOTE_PLUGIN_MARKETPLACES = [ + "openai-curated-remote", + "created-by-me-remote", + "workspace-directory", + "workspace-shared-with-me", + "workspace-shared-with-me-private", + "workspace-shared-with-me-unlisted", +] as const; +const CODEX_HOME_REDOWNLOADABLE_ENTRIES = [ + ".remote-plugin-install-staging", + "cache/remote_plugin_catalog", + "plugins/.remote-plugin-install-staging", + ...CODEX_REMOTE_PLUGIN_MARKETPLACES.map((marketplace) => `plugins/cache/${marketplace}`), +] as const; +const CODEX_HOME_CHECKPOINT_EXCLUDED_ENTRIES = [ + ...CODEX_HOME_NON_PERSISTENT_ENTRIES, + ...CODEX_HOME_REDOWNLOADABLE_ENTRIES, +] as const; const RUNNERD_CONTROL_PLANE_STATE_SCHEMA = "paperclip.runner.durable.control-plane-state.v1"; const RUNNERD_STATE_SCHEMA = "paperclip.runner.durable.state.v1"; @@ -5518,7 +5536,7 @@ export function resolveNativeHarnessPersistenceProfile( // credential source and config.toml can contain the native MCP // bearer token. Re-materialize both for a replacement sandbox // instead of putting credentials into the disaster-recovery copy. - excludeEntries: CODEX_HOME_NON_PERSISTENT_ENTRIES, + excludeEntries: CODEX_HOME_CHECKPOINT_EXCLUDED_ENTRIES, }) : execution.provider.kind === "opencode" ? { @@ -5541,7 +5559,7 @@ export function resolveNativeHarnessPersistenceProfile( ? ["auth.json", "auth-refresh.json", "auth-refresh.json.tmp", "config.toml", "logs"].map((entry) => `acpx/${acpxRuntimeSessionDirectoryName(nativeSessionKey(execution))}/grok-home/${entry}`) : execution.provider.agent === "codex" - ? CODEX_HOME_NON_PERSISTENT_ENTRIES.map( + ? CODEX_HOME_CHECKPOINT_EXCLUDED_ENTRIES.map( (entry) => `acpx/${acpxRuntimeSessionDirectoryName(nativeSessionKey(execution))}/codex-home/${entry}`, )