## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex sessions in remote sandboxes copy the sandbox's Codex
home into a host-side failover backup after every turn
> - Ephemeral sandboxes restore that backup at the start of every turn,
so its size affects both disk use and turn latency
> - Most of each backup is remote plugin data that Codex itself treats
as a cache and re-downloads on startup
> - This pull request leaves that data out of Codex checkpoints, keeping
everything a resumed session needs
> - The benefit is backups roughly 80–90% smaller, less data moved per
turn, and much slower disk growth
## Linked Issues or Issue Description
Bug report described in-PR (bug report template). Related: #16866
(expires failover backups after their issues close).
**What happened?**
Failover backups grow quickly because each Codex checkpoint includes
`$CODEX_HOME` remote plugin data. On one instance, 204 backups held 22.1
GB:
- `codex-home/plugins/cache/openai-curated-remote`: 8.3 GB (about 29 MB
per backup);
- `codex-home/cache/remote_plugin_catalog`: 6.1 GB (one 38 MB catalog
file per backup);
- `codex-home/sessions` (the conversation history): 0.31 GB.
About 80 new backups (9.8 GB) were written per day. Ephemeral sandboxes
also copy all of this into a fresh sandbox before each turn.
**Expected behavior**
Checkpoints keep what a resumed session needs and omit data that Codex
re-downloads itself.
**Steps to reproduce**
1. Run a Codex agent with ChatGPT auth in a remote sandbox environment.
2. Inspect
`<instance>/runtime/paperclip-runner/durable-sessions/*/failover-backups/current/codex-home`.
3. Most bytes are under `plugins/cache/openai-curated-remote` and
`cache/remote_plugin_catalog`.
**Paperclip version or commit:** `master` at f89c37791c.
**Deployment mode:** server with remote sandbox environments.
**Agent adapter(s) involved:** Codex (native app-server and ACPX).
## What Changed
- **`server/src/services/native-runtime/native-session-executor.ts`:**
Codex checkpoints (direct and under ACPX) now also exclude:
- `plugins/cache/<marketplace>` for Codex's six remote marketplaces
(`openai-curated-remote`, `created-by-me-remote`, `workspace-directory`,
`workspace-shared-with-me`, `workspace-shared-with-me-private`,
`workspace-shared-with-me-unlisted`);
- `cache/remote_plugin_catalog`;
- `plugins/.remote-plugin-install-staging`, and
`.remote-plugin-install-staging` (the location older Codex versions
use).
- **Still kept:** local and bundled plugin caches, `plugins/data`, the
Apps tool cache (`cache/codex_apps_tools`, which avoids waiting up to 30
s for the Apps MCP server), session rollouts, and the state databases.
- **Tests:** a real-tar checkpoint test covers kept and omitted entries;
the profile expectations are updated.
- **`doc/recoverable-disk-cleanup.md`:** describes what Codex failover
copies omit and the first-turn effect.
- No code comments were added.
**Why the omitted data is safe to drop** (Codex source at commit
d4e03acb):
- **Server-side source of truth:** the installed remote plugin list
comes from the server (`/backend-api/ps/plugins/installed`) and is held
in memory; these directories are not the source of truth.
- **Self-healing startup:** at `app-server` startup, a background task
reconciles missing bundles and force-refreshes the catalog
(`core-plugins/src/manager.rs`, `remote_installed_plugin_sync.rs`).
- **No path dependencies:** rollouts and state store plugin ids, not
paths into these directories.
## Verification
- `pnpm exec vitest run
src/services/native-runtime/native-session-executor.test.ts` (in
`server/`): 766 passed.
- **Mutation check:** dropping one remote marketplace from the exclusion
list fails both the new checkpoint test and the profile test.
- `pnpm run typecheck` (in `server/`): pass.
- **Manual resume with Codex CLI 0.162.1 and ChatGPT auth:**
1. Started a session in a fresh `CODEX_HOME`. This downloaded 28 MB of
remote plugins.
2. Copied the home without the excluded entries: 34 MB became 4.9 MB.
3. Ran `codex exec resume --last`. It recalled the conversation in 3 s,
and the remote plugins were re-downloaded in the background without
errors.
- **Compatibility:** restore compares only top-level directory names
(`compatibleNativeHarnessBackupManifests`), so existing backups that
still contain the plugins restore unchanged.
## Risks
- **First turn after a restore:** remote plugin skills and tools can be
missing until the background download finishes, typically seconds. Codex
leaves an unavailable remote plugin out of a turn; it does not fail the
turn. Later turns include it.
- **Network and auth:** re-downloading requires ChatGPT auth and network
access from the sandbox, which remote plugins require anyway, since
Codex fetches the installed list from the server on every start.
- **Per-turn download:** ephemeral sandboxes download the remote plugin
bundles each turn, about 28 MB in the observed case, in the background.
In exchange, about 90 MB less is copied from the host before each turn.
- **Rollback:** revert.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, Codex source reading, a local
Codex resume test, read-only measurements on a live instance).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent sessions that run in a remote sandbox keep a host-side
failover copy of the sandbox's harness state, so a replacement sandbox
can resume the conversation
> - Each session keeps that copy forever, and one copy can reach several
gigabytes
> - On a busy instance these copies fill the volume, and no existing
cleanup touches them
> - Once a session's work is closed, its copy matters only if someone
reopens that work later
> - This pull request expires the copy after the session's issues have
been closed for a configurable window, and makes a reopened issue start
a fresh session
> - The benefit is bounded disk use without breaking active,
recoverable, or leased sessions
## Linked Issues or Issue Description
Bug report described in-PR (bug report template).
**What happened?**
After every remote turn, `checkpointRemoteRunner` writes the sandbox's
runner state and provider session home (for example `codex-home`) to
`durable-sessions/<scope>/failover-backups/current`. Only the same
session's next checkpoint replaces it. No code path deletes a session's
copy when the session ends, its issue closes, or its agent goes away, so
the instance volume fills. On one instance:
- 246 sessions held 17 GB of copies, 84 of them older than a week;
- 24 new sessions added 5 GB in a single day, one copy alone 3.2 GB.
**Expected behavior**
Disk used by failover copies stays bounded. A copy is kept while its
session can still resume (open issue, live run, pending recovery, or
live lease), and expires once the session's issues have been closed for
a while. A reopened issue then starts a fresh native session, instead of
failing with `runner_harness_backup_unavailable` on a replacement
sandbox.
**Steps to reproduce**
1. Run agents with a native runtime in a remote sandbox environment.
2. Let them complete many issues.
3. Watch
`<instance>/runtime/paperclip-runner/durable-sessions/*/failover-backups/current`
grow without bound after the issues close.
**Paperclip version or commit:** `master` at 9179d8dcc3.
**Deployment mode:** server with remote sandbox environments.
**Agent adapter(s) involved:** not adapter-specific (core bug); observed
with Codex sessions.
**Breaking changes**
- Reopening an issue that has been closed for longer than the window
(default 7 days) starts a fresh agent session instead of resuming the
old conversation.
- Disable with
`PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED=false`, or lengthen
the window with `PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DAYS`.
## What Changed
-
**`server/src/services/native-runtime/native-harness-backup-expiration.ts`
(new):** the sweep. It checks at most 200 sessions and expires at most
50 per batch, continuing from a cursor. A copy expires only when all of
these hold:
- every issue the session ran on is `done` or `cancelled`, closed before
the window;
- the copy was written before the window;
- no run of the session is queued, running, or unfinished;
- every native run finalization of the session is settled (`committed`
or `terminal_failure`), with no recovery state or lease owner;
- no `active`, `released`, `retained`, or `pending_cleanup` environment
lease references the copy (by backup stamp, session run, or recorded
lease);
- the session is not executing, initializing, warm, or closing in this
process, and its directory has no pending cleanup activation.
- **One transaction:** the database checks share-lock the issues and
delete the agent task session that points at the session, so a reopened
issue cannot resume the expired session.
- **Atomic, non-blocking removal:** the copy is renamed to
`failover-backups/.retiring-<UUID>`, and a final check looks for a run
of the session. If one appeared, or the check throws, the rename is
undone. Only a confirmed copy becomes `.expired-<UUID>` and is deleted
with asynchronous file operations, so a multi-gigabyte removal does not
block the event loop. A later sweep restores an unconfirmed
`.retiring-*` copy when `current` is missing, drops it when a newer
`current` exists, and finishes an interrupted deletion. The rest of the
session directory is kept.
- **`native-session-executor.ts`:** exports
`nativeSessionScopeDigestIsBusy`, which checks the in-process executing,
initializing, warm, and closing session maps.
- **`native-harness-backup-stamp.ts`:** exports the state base path the
sweep scans.
- **`config.ts` and `index.ts`:** three environment controls
(`_ENABLED`, `_DRY_RUN`, `_DAYS`) and the scheduled sweep. It starts two
minutes after boot, honors shutdown, standby, and idle drain, and logs
candidates, reclaimed bytes, cleared task sessions, and skip reasons.
- **Docs:** `doc/recoverable-disk-cleanup.md` describes the sweep;
`doc/DEVELOPING.md` lists the variables.
- No code comments were added.
## Verification
- `pnpm exec vitest run
src/__tests__/native-harness-backup-expiration.test.ts
src/__tests__/native-harness-backup-expiration-config.test.ts` (in
`server/`): 30 passed, 24 of them against embedded Postgres plus 6
config tests. Cases:
- expiry for done and cancelled issues, including clearing the task
session;
- each keep condition: open issue, recently closed, live run, pending
finalization, recovery in progress, live lease, recent copy, cleanup
activation, in-process session;
- settled finalizations and expired leases do not block;
- dry run changes nothing;
- bounded batches continue across ticks;
- an interrupted deletion is finished, and an unconfirmed retiring copy
is restored or dropped;
- the configured clock controls the closed window.
- **Mutation checks:** treating `in_progress` as closed, ignoring live
leases, ignoring the configured clock, or not restoring a retiring copy
each fail the corresponding tests.
- `pnpm exec vitest run native-harness-backup config` (in `server/`): 19
files, 293 tests passed.
- `pnpm run typecheck` (in `server/`): pass.
- **Real server:** after building and bundling,
`scripts/server-boot-smoke.sh node server/dist-runtime/index.mjs`
reached healthy.
- **Manual equivalent on a live instance:** applying these checks by
hand (with a 3-day window) to 242 copies freed 7.1 GB. Most of the
remainder belonged to open issues or live leases.
## Risks
- **Reopen after the window:** the agent starts a fresh session and
loses the old conversation's provider history. Issue comments,
documents, and work products are unaffected.
- **Race with a wake:** the checks share-lock the issues, so a reopen
waits for the transaction. A run created for the session between commit
and rename is caught by the post-rename check, which restores the copy.
Such a run would find no task session and start fresh.
- **Missing owner run:** if the owning run row is gone (for example,
after an agent was deleted), the copy is kept, because the sweep cannot
prove the session is closed.
- **Untouched:** quarantined session roots and `continuity-breaks`
archives are not touched by this change.
- **Rollback:** set
`PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED=false`, or revert.
Expired copies cannot be restored.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, local builds, tests and boots,
read-only measurements on a live instance).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
