fix: omit re-downloadable Codex remote plugins from failover backups (#17026)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex sessions in remote sandboxes copy the sandbox's Codex
home into a host-side failover backup after every turn
> - Ephemeral sandboxes restore that backup at the start of every turn,
so its size affects both disk use and turn latency
> - Most of each backup is remote plugin data that Codex itself treats
as a cache and re-downloads on startup
> - This pull request leaves that data out of Codex checkpoints, keeping
everything a resumed session needs
> - The benefit is backups roughly 80–90% smaller, less data moved per
turn, and much slower disk growth

## Linked Issues or Issue Description

Bug report described in-PR (bug report template). Related: #16866
(expires failover backups after their issues close).

**What happened?**
Failover backups grow quickly because each Codex checkpoint includes
`$CODEX_HOME` remote plugin data. On one instance, 204 backups held 22.1
GB:
- `codex-home/plugins/cache/openai-curated-remote`: 8.3 GB (about 29 MB
per backup);
- `codex-home/cache/remote_plugin_catalog`: 6.1 GB (one 38 MB catalog
file per backup);
- `codex-home/sessions` (the conversation history): 0.31 GB.

About 80 new backups (9.8 GB) were written per day. Ephemeral sandboxes
also copy all of this into a fresh sandbox before each turn.

**Expected behavior**
Checkpoints keep what a resumed session needs and omit data that Codex
re-downloads itself.

**Steps to reproduce**
1. Run a Codex agent with ChatGPT auth in a remote sandbox environment.
2. Inspect
`<instance>/runtime/paperclip-runner/durable-sessions/*/failover-backups/current/codex-home`.
3. Most bytes are under `plugins/cache/openai-curated-remote` and
`cache/remote_plugin_catalog`.

**Paperclip version or commit:** `master` at f89c37791c.
**Deployment mode:** server with remote sandbox environments.
**Agent adapter(s) involved:** Codex (native app-server and ACPX).

## What Changed

- **`server/src/services/native-runtime/native-session-executor.ts`:**
Codex checkpoints (direct and under ACPX) now also exclude:
- `plugins/cache/<marketplace>` for Codex's six remote marketplaces
(`openai-curated-remote`, `created-by-me-remote`, `workspace-directory`,
`workspace-shared-with-me`, `workspace-shared-with-me-private`,
`workspace-shared-with-me-unlisted`);
  - `cache/remote_plugin_catalog`;
- `plugins/.remote-plugin-install-staging`, and
`.remote-plugin-install-staging` (the location older Codex versions
use).
- **Still kept:** local and bundled plugin caches, `plugins/data`, the
Apps tool cache (`cache/codex_apps_tools`, which avoids waiting up to 30
s for the Apps MCP server), session rollouts, and the state databases.
- **Tests:** a real-tar checkpoint test covers kept and omitted entries;
the profile expectations are updated.
- **`doc/recoverable-disk-cleanup.md`:** describes what Codex failover
copies omit and the first-turn effect.
- No code comments were added.

**Why the omitted data is safe to drop** (Codex source at commit
d4e03acb):
- **Server-side source of truth:** the installed remote plugin list
comes from the server (`/backend-api/ps/plugins/installed`) and is held
in memory; these directories are not the source of truth.
- **Self-healing startup:** at `app-server` startup, a background task
reconciles missing bundles and force-refreshes the catalog
(`core-plugins/src/manager.rs`, `remote_installed_plugin_sync.rs`).
- **No path dependencies:** rollouts and state store plugin ids, not
paths into these directories.

## Verification

- `pnpm exec vitest run
src/services/native-runtime/native-session-executor.test.ts` (in
`server/`): 766 passed.
- **Mutation check:** dropping one remote marketplace from the exclusion
list fails both the new checkpoint test and the profile test.
- `pnpm run typecheck` (in `server/`): pass.
- **Manual resume with Codex CLI 0.162.1 and ChatGPT auth:**
1. Started a session in a fresh `CODEX_HOME`. This downloaded 28 MB of
remote plugins.
  2. Copied the home without the excluded entries: 34 MB became 4.9 MB.
3. Ran `codex exec resume --last`. It recalled the conversation in 3 s,
and the remote plugins were re-downloaded in the background without
errors.
- **Compatibility:** restore compares only top-level directory names
(`compatibleNativeHarnessBackupManifests`), so existing backups that
still contain the plugins restore unchanged.

## Risks

- **First turn after a restore:** remote plugin skills and tools can be
missing until the background download finishes, typically seconds. Codex
leaves an unavailable remote plugin out of a turn; it does not fail the
turn. Later turns include it.
- **Network and auth:** re-downloading requires ChatGPT auth and network
access from the sandbox, which remote plugins require anyway, since
Codex fetches the installed list from the server on every start.
- **Per-turn download:** ephemeral sandboxes download the remote plugin
bundles each turn, about 28 MB in the observed case, in the background.
In exchange, about 90 MB less is copied from the host before each turn.
- **Rollback:** revert.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, Codex source reading, a local
Codex resume test, read-only measurements on a live instance).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

server/src/services/native-runtime/native-session-executor.test.ts

@@ -1787,12 +1787,26 @@ describe("native harness persistence profiles", () => {
     },
   );
 
-  it("excludes disposable Codex scratch trees and launch-time credentials", () => {
+  it("excludes disposable Codex scratch trees, launch-time credentials and re-downloadable remote plugins", () => {
     const codex = profile({ kind: "codex" }, "codex_app_server");
     expect(
       codex.directories.find((directory) => directory.name === "codex-home"),
     ).toMatchObject({
-      excludeEntries: ["tmp", ".tmp", "auth.json", "config.toml"],
+      excludeEntries: [
+        "tmp",
+        ".tmp",
+        "auth.json",
+        "config.toml",
+        ".remote-plugin-install-staging",
+        "cache/remote_plugin_catalog",
+        "plugins/.remote-plugin-install-staging",
+        "plugins/cache/openai-curated-remote",
+        "plugins/cache/created-by-me-remote",
+        "plugins/cache/workspace-directory",
+        "plugins/cache/workspace-shared-with-me",
+        "plugins/cache/workspace-shared-with-me-private",
+        "plugins/cache/workspace-shared-with-me-unlisted",
+      ],
     });
   });
 
@@ -1807,6 +1821,15 @@ describe("native harness persistence profiles", () => {
         `acpx/${sessionDirectory}/codex-home/.tmp`,
         `acpx/${sessionDirectory}/codex-home/auth.json`,
         `acpx/${sessionDirectory}/codex-home/config.toml`,
+        `acpx/${sessionDirectory}/codex-home/.remote-plugin-install-staging`,
+        `acpx/${sessionDirectory}/codex-home/cache/remote_plugin_catalog`,
+        `acpx/${sessionDirectory}/codex-home/plugins/.remote-plugin-install-staging`,
+        `acpx/${sessionDirectory}/codex-home/plugins/cache/openai-curated-remote`,
+        `acpx/${sessionDirectory}/codex-home/plugins/cache/created-by-me-remote`,
+        `acpx/${sessionDirectory}/codex-home/plugins/cache/workspace-directory`,
+        `acpx/${sessionDirectory}/codex-home/plugins/cache/workspace-shared-with-me`,
+        `acpx/${sessionDirectory}/codex-home/plugins/cache/workspace-shared-with-me-private`,
+        `acpx/${sessionDirectory}/codex-home/plugins/cache/workspace-shared-with-me-unlisted`,
       ],
     });
     expect(
@@ -2530,6 +2553,51 @@ describe("remote provider checkpoint snapshots", () => {
       expect(await readFile(join(home, "logs", "unified.jsonl"), "utf8")).toBe("AUTH_SENTINEL");
     } finally { await rm(root, { recursive: true, force: true }); }
   });
+  it("keeps Codex session state while omitting re-downloadable remote plugins from the checkpoint", async () => {
+    const root = await mkdtemp(join(tmpdir(), "codex-plugin-checkpoint-"));
+    try {
+      const source = join(root, "codex-home");
+      const target = join(root, "backup");
+      const files: Record<string, string> = {
+        "sessions/2026/10/20/rollout.jsonl": "resume-state",
+        "state_5.sqlite": "state",
+        "cache/codex_apps_tools/tools.json": "apps",
+        "cache/remote_plugin_catalog/catalog.json": "catalog",
+        "plugins/data/sites/data.json": "plugin-data",
+        "plugins/cache/openai-bundled/browser/1.0.0/plugin.json": "bundled",
+        "plugins/cache/openai-curated-remote/sites/1.0.2/plugin.json": "remote",
+        "plugins/cache/workspace-shared-with-me/notes/0.1.0/plugin.json": "remote",
+        "plugins/.remote-plugin-install-staging/partial/plugin.json": "staging",
+        ".remote-plugin-install-staging/partial/plugin.json": "staging",
+      };
+      for (const [path, contents] of Object.entries(files)) {
+        await mkdir(join(source, path, ".."), { recursive: true });
+        await writeFile(join(source, path), contents);
+      }
+      const codexProfile = resolveNativeHarnessPersistenceProfile({
+        provider: { kind: "codex" },
+        session: { normalizedSessionId: "session", driverKind: "codex_app_server" },
+      } as unknown as NativeExecutionInputV1);
+      const execute = async ({ command, args }: { command: string; args: string[] }) => ({
+        exitCode: 0, timedOut: false, stdout: execFileSync(command, args, { encoding: "utf8" }), stderr: "",
+      });
+      const syncOut = async (operations: Array<{ files: Array<{ sourcePath: string; targetPath: string }> }>) => {
+        for (const operation of operations) for (const file of operation.files)
+          await cp(file.sourcePath, file.targetPath, { recursive: true });
+      };
+      await syncRemoteRunnerDirectoryOut({ runner: { execute, syncOut } as never,
+        sourcePath: source, targetPath: target, mode: 0o700,
+        excludeEntries: codexProfile.directories.find(directory => directory.name === "codex-home")!.excludeEntries });
+      for (const kept of ["sessions/2026/10/20/rollout.jsonl", "state_5.sqlite", "cache/codex_apps_tools/tools.json",
+        "plugins/data/sites/data.json", "plugins/cache/openai-bundled/browser/1.0.0/plugin.json"])
+        expect(await readFile(join(target, kept), "utf8")).toBe(files[kept]);
+      for (const omitted of ["cache/remote_plugin_catalog", "plugins/cache/openai-curated-remote",
+        "plugins/cache/workspace-shared-with-me", "plugins/.remote-plugin-install-staging", ".remote-plugin-install-staging"])
+        await expect(access(join(target, omitted))).rejects.toMatchObject({ code: "ENOENT" });
+      expect(await readFile(join(source, "plugins/cache/openai-curated-remote/sites/1.0.2/plugin.json"), "utf8")).toBe("remote");
+    } finally { await rm(root, { recursive: true, force: true }); }
+  });
+
   it("excludes Codex scratch and credential state without mutating the live provider home", async () => {
     const execute = vi
       .fn()

server/src/services/native-runtime/native-session-executor.ts

@@ -336,6 +336,24 @@ const CODEX_HOME_NON_PERSISTENT_ENTRIES = [
   "auth.json",
   "config.toml",
 ] as const;
+const CODEX_REMOTE_PLUGIN_MARKETPLACES = [
+  "openai-curated-remote",
+  "created-by-me-remote",
+  "workspace-directory",
+  "workspace-shared-with-me",
+  "workspace-shared-with-me-private",
+  "workspace-shared-with-me-unlisted",
+] as const;
+const CODEX_HOME_REDOWNLOADABLE_ENTRIES = [
+  ".remote-plugin-install-staging",
+  "cache/remote_plugin_catalog",
+  "plugins/.remote-plugin-install-staging",
+  ...CODEX_REMOTE_PLUGIN_MARKETPLACES.map((marketplace) => `plugins/cache/${marketplace}`),
+] as const;
+const CODEX_HOME_CHECKPOINT_EXCLUDED_ENTRIES = [
+  ...CODEX_HOME_NON_PERSISTENT_ENTRIES,
+  ...CODEX_HOME_REDOWNLOADABLE_ENTRIES,
+] as const;
 const RUNNERD_CONTROL_PLANE_STATE_SCHEMA =
   "paperclip.runner.durable.control-plane-state.v1";
 const RUNNERD_STATE_SCHEMA = "paperclip.runner.durable.state.v1";
@@ -5518,7 +5536,7 @@ export function resolveNativeHarnessPersistenceProfile(
           // credential source and config.toml can contain the native MCP
           // bearer token. Re-materialize both for a replacement sandbox
           // instead of putting credentials into the disaster-recovery copy.
-          excludeEntries: CODEX_HOME_NON_PERSISTENT_ENTRIES,
+          excludeEntries: CODEX_HOME_CHECKPOINT_EXCLUDED_ENTRIES,
         })
       : execution.provider.kind === "opencode"
         ? {
@@ -5541,7 +5559,7 @@ export function resolveNativeHarnessPersistenceProfile(
                   ? ["auth.json", "auth-refresh.json", "auth-refresh.json.tmp", "config.toml", "logs"].map((entry) =>
                     `acpx/${acpxRuntimeSessionDirectoryName(nativeSessionKey(execution))}/grok-home/${entry}`)
                   : execution.provider.agent === "codex"
-                  ? CODEX_HOME_NON_PERSISTENT_ENTRIES.map(
+                  ? CODEX_HOME_CHECKPOINT_EXCLUDED_ENTRIES.map(
                       (entry) =>
                         `acpx/${acpxRuntimeSessionDirectoryName(nativeSessionKey(execution))}/codex-home/${entry}`,
                     )