mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-23 17:33:51 +02:00
Devin FoleyandPaperclip
572bef4ba7
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex sessions in remote sandboxes copy the sandbox's Codex
home into a host-side failover backup after every turn
> - Ephemeral sandboxes restore that backup at the start of every turn,
so its size affects both disk use and turn latency
> - Most of each backup is remote plugin data that Codex itself treats
as a cache and re-downloads on startup
> - This pull request leaves that data out of Codex checkpoints, keeping
everything a resumed session needs
> - The benefit is backups roughly 80–90% smaller, less data moved per
turn, and much slower disk growth
## Linked Issues or Issue Description
Bug report described in-PR (bug report template). Related: #16866
(expires failover backups after their issues close).
**What happened?**
Failover backups grow quickly because each Codex checkpoint includes
`$CODEX_HOME` remote plugin data. On one instance, 204 backups held 22.1
GB:
- `codex-home/plugins/cache/openai-curated-remote`: 8.3 GB (about 29 MB
per backup);
- `codex-home/cache/remote_plugin_catalog`: 6.1 GB (one 38 MB catalog
file per backup);
- `codex-home/sessions` (the conversation history): 0.31 GB.
About 80 new backups (9.8 GB) were written per day. Ephemeral sandboxes
also copy all of this into a fresh sandbox before each turn.
**Expected behavior**
Checkpoints keep what a resumed session needs and omit data that Codex
re-downloads itself.
**Steps to reproduce**
1. Run a Codex agent with ChatGPT auth in a remote sandbox environment.
2. Inspect
`<instance>/runtime/paperclip-runner/durable-sessions/*/failover-backups/current/codex-home`.
3. Most bytes are under `plugins/cache/openai-curated-remote` and
`cache/remote_plugin_catalog`.
**Paperclip version or commit:** `master` at f89c37791c.
**Deployment mode:** server with remote sandbox environments.
**Agent adapter(s) involved:** Codex (native app-server and ACPX).
## What Changed
- **`server/src/services/native-runtime/native-session-executor.ts`:**
Codex checkpoints (direct and under ACPX) now also exclude:
- `plugins/cache/<marketplace>` for Codex's six remote marketplaces
(`openai-curated-remote`, `created-by-me-remote`, `workspace-directory`,
`workspace-shared-with-me`, `workspace-shared-with-me-private`,
`workspace-shared-with-me-unlisted`);
- `cache/remote_plugin_catalog`;
- `plugins/.remote-plugin-install-staging`, and
`.remote-plugin-install-staging` (the location older Codex versions
use).
- **Still kept:** local and bundled plugin caches, `plugins/data`, the
Apps tool cache (`cache/codex_apps_tools`, which avoids waiting up to 30
s for the Apps MCP server), session rollouts, and the state databases.
- **Tests:** a real-tar checkpoint test covers kept and omitted entries;
the profile expectations are updated.
- **`doc/recoverable-disk-cleanup.md`:** describes what Codex failover
copies omit and the first-turn effect.
- No code comments were added.
**Why the omitted data is safe to drop** (Codex source at commit
d4e03acb):
- **Server-side source of truth:** the installed remote plugin list
comes from the server (`/backend-api/ps/plugins/installed`) and is held
in memory; these directories are not the source of truth.
- **Self-healing startup:** at `app-server` startup, a background task
reconciles missing bundles and force-refreshes the catalog
(`core-plugins/src/manager.rs`, `remote_installed_plugin_sync.rs`).
- **No path dependencies:** rollouts and state store plugin ids, not
paths into these directories.
## Verification
- `pnpm exec vitest run
src/services/native-runtime/native-session-executor.test.ts` (in
`server/`): 766 passed.
- **Mutation check:** dropping one remote marketplace from the exclusion
list fails both the new checkpoint test and the profile test.
- `pnpm run typecheck` (in `server/`): pass.
- **Manual resume with Codex CLI 0.162.1 and ChatGPT auth:**
1. Started a session in a fresh `CODEX_HOME`. This downloaded 28 MB of
remote plugins.
2. Copied the home without the excluded entries: 34 MB became 4.9 MB.
3. Ran `codex exec resume --last`. It recalled the conversation in 3 s,
and the remote plugins were re-downloaded in the background without
errors.
- **Compatibility:** restore compares only top-level directory names
(`compatibleNativeHarnessBackupManifests`), so existing backups that
still contain the plugins restore unchanged.
## Risks
- **First turn after a restore:** remote plugin skills and tools can be
missing until the background download finishes, typically seconds. Codex
leaves an unavailable remote plugin out of a turn; it does not fail the
turn. Later turns include it.
- **Network and auth:** re-downloading requires ChatGPT auth and network
access from the sandbox, which remote plugins require anyway, since
Codex fetches the installed list from the server on every start.
- **Per-turn download:** ephemeral sandboxes download the remote plugin
bundles each turn, about 28 MB in the observed case, in the background.
In exchange, about 90 MB less is copied from the host before each turn.
- **Rollback:** revert.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, Codex source reading, a local
Codex resume test, read-only measurements on a live instance).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
95 lines
5.8 KiB
Markdown
95 lines
5.8 KiB
Markdown
# Recoverable disk cleanup
|
|
|
|
The server materializes immutable skill versions under
|
|
`<instance>/skills/<company>/__versions__/<skill>/<version>`. These are disposable
|
|
disk copies: the company-scoped version and repository snapshot rows retain the
|
|
contents, including binary files and executable flags. Normal skill resolution
|
|
rebuilds a missing copy from the database without contacting its upstream source.
|
|
|
|
## Skill version caches
|
|
|
|
An hourly background sweep considers at most 50 directories per batch, yielding
|
|
between batches. It starts after server readiness, and respects the existing
|
|
worker shutdown, standby, and idle-drain controls. A non-current version becomes
|
|
eligible after one hour without use. Current versions, active-run pins and
|
|
unsettled recovery pins are retained. If an active run's version selection is
|
|
unknown, that company's caches remain. Non-versioned skill directories and
|
|
database snapshots are never removed by this worker.
|
|
|
|
Last-use records live separately under `__version_access__`. Older installations
|
|
without those records receive a full one-hour observation period. Preparation,
|
|
pin persistence and eviction share a company-scoped database lock across server
|
|
processes. Snapshot validation and byte counting run outside that lock; cleanup
|
|
rechecks current versions, run pins, last use, and directory identity before rename. Audits hold the lock through their file reads. Eviction atomically
|
|
moves a cache into unique `.trash-<UUID>` directories; trash deletion uses its own lock so normal
|
|
preparation can restore files while deletion finishes. A later sweep finishes
|
|
interrupted trash removal without deleting a newly materialized version.
|
|
|
|
Controls:
|
|
|
|
| Environment variable | Default | Effect |
|
|
| --- | --- | --- |
|
|
| `PAPERCLIP_SKILL_VERSION_RETENTION_ENABLED` | `true` | Set `false` to stop future sweeps; restoration stays available |
|
|
| `PAPERCLIP_SKILL_VERSION_RETENTION_DRY_RUN` | `false` | Set `true` to report candidates without changing cache or access records |
|
|
|
|
The internal `skillVersionRetentionService` supports company-scoped diagnosis,
|
|
bounded continuations, cancellation between entries, and dry run. Explicit agent
|
|
pins stay cached. Operators can also retain newest versions with
|
|
`PAPERCLIP_SKILL_VERSION_RETENTION_KEEP` (default `0`), or change the one-hour
|
|
window with `PAPERCLIP_SKILL_VERSION_RETENTION_GRACE_MINUTES` (default `60`). Each batch reports
|
|
candidate and reclaimed allocated bytes, eviction counts, and skip reasons in
|
|
the server log. Compare volume usage before and after to verify physical savings;
|
|
concurrent writes and filesystem allocation can affect the net change. A dry run
|
|
does not start the legacy observation period.
|
|
|
|
Roll out with dry run enabled, inspect candidates/skips, then enable removal.
|
|
Recovery archives and runner backups have separate ownership and retention;
|
|
this worker never scans or removes them.
|
|
|
|
## Native session failover backups
|
|
|
|
After each remote turn, the server copies the sandbox's harness state (runner
|
|
state plus the provider's session home, such as `codex-home`) to
|
|
`<instance>/runtime/paperclip-runner/durable-sessions/<scope>/failover-backups/current`.
|
|
A replacement sandbox restores the session from this copy, and an ephemeral sandbox
|
|
restores it at the start of every turn. Each session keeps one copy, and a single
|
|
copy can reach several gigabytes. Codex copies omit credentials, scratch
|
|
directories, and data Codex downloads again on startup: remote plugin bundles under
|
|
`plugins/cache/<remote marketplace>`, the remote plugin catalog cache, and plugin
|
|
install staging. After a restore, remote plugins can be unavailable to a turn until
|
|
Codex finishes downloading them.
|
|
|
|
An hourly background sweep expires the copy once the session can only resume by
|
|
reopening closed work. It checks at most 200 sessions and expires at most 50 per
|
|
batch, starting two minutes after readiness. A copy expires when all of these hold:
|
|
|
|
- every issue the session ran on is `done` or `cancelled`, and closed before the window;
|
|
- the copy itself was written before the window;
|
|
- no run of the session is queued, running, or unfinished;
|
|
- every native run finalization of the session is settled, with no recovery in progress;
|
|
- no active, released, retained, or pending-cleanup environment lease references the copy;
|
|
- the session is not executing, initializing, warm, or closing in this process, and
|
|
no cleanup activation is pending in its directory.
|
|
|
|
A session whose control-plane state is missing, unreadable, or larger than 4 MB
|
|
keeps its copy, because the sweep cannot identify its runs.
|
|
|
|
The sweep checks and locks these rows in one transaction, and removes the agent task
|
|
session that points at the session there. A reopened issue therefore starts a fresh
|
|
native session instead of resuming the expired one. The copy is atomically renamed to
|
|
`failover-backups/.retiring-<UUID>`, and a final check looks for a run of the session.
|
|
If one appeared, or the check fails, the rename is undone. Only a confirmed copy is
|
|
renamed to `.expired-<UUID>` and deleted asynchronously. A later sweep restores an
|
|
unconfirmed `.retiring-*` copy when `current` is missing, drops it when a newer
|
|
`current` exists, and finishes an interrupted deletion.
|
|
The rest of the session directory, including its control-plane state, is kept.
|
|
|
|
| Environment variable | Default | Effect |
|
|
| --- | --- | --- |
|
|
| `PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED` | `true` | Set `false` to stop expiring copies |
|
|
| `PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DRY_RUN` | `false` | Set `true` to report candidates without deleting copies or task sessions |
|
|
| `PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DAYS` | `7` | Days an issue must be closed before its sessions' copies expire (minimum `1`) |
|
|
|
|
Each batch that finds candidates logs them with candidate and reclaimed allocated
|
|
bytes, cleared task sessions, and skip reasons.
|