## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runner sessions keep durable state under
`<instance>/runtime/paperclip-runner/durable-sessions/`
> - When a session starts a fresh provider session, the retired state is
moved into `<session>/continuity-breaks/<epoch>-<uuid>/`; when state
loses authority, it is moved into
`quarantine/<session>.<reason>.<epoch>.<pid>.<uuid>/`
> - Nothing ever removes either, and each can hold a full Codex home of
several gigabytes
> - A sleep-aware sweep already expires failover backup copies in the
same tree
> - This pull request extends that sweep to remove both kinds of
evidence after a retention window, while keeping anything recovery may
still read
> - The benefit is that native runner disk use stops growing without
bound
## Linked Issues or Issue Description
Bug report described in-PR (bug report template).
**What happened?**
On one instance, one day of native runs produced 20 continuity-break
archives totaling 4.8 GB, one session alone holding 4 GB. The quarantine
folder held 12 entries totaling 2.1 GB, the oldest a month old. Neither
is read again by normal operation (continuity-break archives are written
by `archiveContinuityState` and never read), and nothing deletes them.
They were removed by hand to free space and will regrow.
**Expected behavior**
Evidence older than a bounded window is removed automatically, as
failover backup copies already are, without removing anything a live
session or recovery path still needs.
**Steps to reproduce**
1. Run a native Codex session that hits a continuity break (fresh
provider session), or whose state is quarantined (for example
`identity_indeterminate`).
2. Inspect `durable-sessions/<session>/continuity-breaks/` and
`durable-sessions/quarantine/`.
3. Wait any length of time: the entries stay forever.
**Paperclip version or commit:** `master` at e335265568.
**Deployment mode:** any with native runner sessions.
**Agent adapter(s) involved:** native runner (Codex, and any provider
whose state lands in these folders).
## What Changed
-
`server/src/services/native-runtime/native-harness-backup-expiration.ts`:
the existing tick also retires evidence.
- **Continuity-break archives** (`<epoch ms>-<uuid>` under each
session): removed when older than the window. Age is the epoch in the
name, or the folder mtime when that epoch is in the future.
`archiveContinuityState` refuses to archive state with warm-transition
evidence, and no code path reads `continuity-breaks`.
- **Quarantine entries**
(`<session>.<identity_indeterminate|identity_mismatch>.<epoch
ms>.<pid>.<uuid>`): removed by the same age rule. Native cleanup source
archives (`<session>.identity_indeterminate.cleanup.<uuid>`) are
referenced from recovery history, have no epoch, and keep their original
inode's mtime, so they are always kept.
- Every entry is skipped while its session is executing, initializing,
warm, or closing in this process; this is checked again just before
removal. Symlinked entries and symlinked
`continuity-breaks`/`quarantine` folders are reported as `unsafe_path`
and never followed.
- Removal renames to `.expired-<uuid>` in the same folder and then
deletes; later sweeps finish interrupted deletions. An empty
`continuity-breaks` folder is removed.
- Archives, quarantine entries, and leftover deletions count toward
`maxEntries` and `maxRemovals`, and stop on `shouldContinue`. Paging
resumes inside a session's archives and then in quarantine, so a bounded
or dry-run batch always makes progress and never recounts a session's
backup.
- New result fields: `evidenceCandidates`, `evidenceCandidateBytes`,
`evidenceRemoved`, `evidenceReclaimedBytes`, and skip reasons
`recent_evidence` and `cleanup_archive`. The service also exposes
`noteEvidence(writtenAt)`.
- `server/src/services/native-runtime/native-runner-evidence-signal.ts`
(new): a process-local signal. `native-session-executor.ts` raises it
with the entry's epoch right after `archiveContinuityState` creates an
archive and after `quarantineRunnerdStateRoot` moves state into
quarantine; `server/src/index.ts` subscribes the sweep and unsubscribes
on shutdown.
- `server/src/config.ts`:
`PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS` (default `7`, minimum
`1`).
- `server/src/index.ts`: passes the window and logs a batch when it
found evidence candidates.
- `doc/recoverable-disk-cleanup.md`, `doc/DEVELOPING.md`: document the
variable and behavior.
**Sleep-aware background work**
(`rules/sleep-aware-background-work.mdc`):
- *Trigger:* each entry's own expiry deadline (its epoch plus the
window), plus a direct call from the executor when it writes new
evidence. No timer, interval, or polling loop is added; the pass runs
inside the existing sweep's tick and keeps its drain, warm-standby, and
shutdown handling.
- *Source of truth:* the filesystem entries themselves. After a restart
the first sweep reconstructs the next deadline from them.
- *Pending-work predicate / next deadline:* after a full pass the sweep
keeps the earliest due time of the entries it retained; ticks before
that time skip the evidence pass entirely (no readdir, no SQL). A busy
session, a failed removal, or a full batch makes it due on the next
sweep while that obligation remains.
- *Quiet condition:* no evidence, or only evidence not yet due. The pass
never queries the database, so retained evidence never causes recurring
SQL. Dry run reports each candidate once rather than every hour.
No code comments were added.
## Verification
- `pnpm exec vitest run src/__tests__/native-harness-backup-expiration`
(in `server/`, embedded Postgres): 52 passed. New cases:
- old archives and quarantine entries are removed, recent ones kept; the
empty `continuity-breaks` folder is removed;
- age comes from the name, with mtime used for a future epoch;
non-archive folders are ignored;
- a busy session keeps its evidence, including when it becomes busy
after the first check, and it is retried on the next tick;
- dry run reports candidates and bytes, changes nothing (including
leftovers), and reports each candidate once until new evidence comes
due;
- symlinked entries and symlinked evidence folders are skipped without
touching the target;
- an old quarantine entry is removed even when its session's issue is
still open; cleanup source archives are kept;
- the pass stays quiet until retained evidence is due, and a
`noteEvidence` signal schedules newly written evidence;
- inspected archives and leftovers count against `maxEntries`, and
leftover removal stops on `shouldContinue` and resumes next tick;
- a due pass stopped by drain before any session stays pending;
one-entry batches finish without recounting a resumed session;
- interrupted `.expired-*` deletions are finished; paging across ticks
with `maxRemovals`, in normal and dry run, with no recount of the backup
when resuming.
- `native-session-executor.test.ts`: quarantining a legacy root raises
the evidence signal with the epoch in the entry name.
- `native-harness-backup-expiration-config`: default, explicit, minimum,
and fallback parsing.
- **Mutation checks**, each failing at least one test: removing the age
check, the future-epoch fallback, either busy check, the busy retry
deadline, the recent-entry deadline, the due gate, the `noteEvidence`
update, merging the pass deadline, the entry and both folder symlink
checks, the dry-run return, the dry-run leftover guard, leftover
removal, the leftover budget check and its retry deadline, counting
archives against `maxEntries`, the cleanup-archive skip, the
empty-folder removal, the resume cursor, the per-archive budget check,
the resume skip of the backup pass, not charging a resumed session,
keeping the pass active across drain, and the executor's quarantine
signal.
- `pnpm exec vitest run config native-harness
native-workspace-sync-retention native-session-executor server-startup`
(in `server/`): 1272 passed on the previous commit; `config
native-harness native-workspace-sync-retention server-startup`: 450
passed on the head commit.
- `pnpm run typecheck` (in `server/`): pass.
- Not run: server build and boot smoke. The continuity-archive signal
call is exercised only by a runner integration test that needs the
runnerd binary.
## Risks
- **Quarantine as a fail-closed fence:** recovery
(`recoverQuiescentRunnerdState`) and failed-run retry
(`nativeFailedRunRetryStateIsSafe`) treat a quarantined entry for a
session as a reason not to bootstrap fresh. Once such an entry ages out,
a later run of that session can start a fresh provider session instead
of failing with `quarantine_owner_unverified`. This applies only after
the window, matching what backup expiry already accepts for closed work.
- **Signal coverage:** the signal is process-local. Evidence written by
another process sharing the same state directory is picked up when the
next retained deadline is due or after a restart, not immediately.
- **Legacy-path quarantine:** entries quarantined from legacy roots
carry a legacy digest, so the in-process busy check cannot match them;
they are removed by age only.
- **Remote sandboxes:** archives moved aside on a remote runner
(`<root>.continuity-break-<token>`) are outside this sweep.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent sessions that run in a remote sandbox keep a host-side
failover copy of the sandbox's harness state, so a replacement sandbox
can resume the conversation
> - Each session keeps that copy forever, and one copy can reach several
gigabytes
> - On a busy instance these copies fill the volume, and no existing
cleanup touches them
> - Once a session's work is closed, its copy matters only if someone
reopens that work later
> - This pull request expires the copy after the session's issues have
been closed for a configurable window, and makes a reopened issue start
a fresh session
> - The benefit is bounded disk use without breaking active,
recoverable, or leased sessions
## Linked Issues or Issue Description
Bug report described in-PR (bug report template).
**What happened?**
After every remote turn, `checkpointRemoteRunner` writes the sandbox's
runner state and provider session home (for example `codex-home`) to
`durable-sessions/<scope>/failover-backups/current`. Only the same
session's next checkpoint replaces it. No code path deletes a session's
copy when the session ends, its issue closes, or its agent goes away, so
the instance volume fills. On one instance:
- 246 sessions held 17 GB of copies, 84 of them older than a week;
- 24 new sessions added 5 GB in a single day, one copy alone 3.2 GB.
**Expected behavior**
Disk used by failover copies stays bounded. A copy is kept while its
session can still resume (open issue, live run, pending recovery, or
live lease), and expires once the session's issues have been closed for
a while. A reopened issue then starts a fresh native session, instead of
failing with `runner_harness_backup_unavailable` on a replacement
sandbox.
**Steps to reproduce**
1. Run agents with a native runtime in a remote sandbox environment.
2. Let them complete many issues.
3. Watch
`<instance>/runtime/paperclip-runner/durable-sessions/*/failover-backups/current`
grow without bound after the issues close.
**Paperclip version or commit:** `master` at 9179d8dcc3.
**Deployment mode:** server with remote sandbox environments.
**Agent adapter(s) involved:** not adapter-specific (core bug); observed
with Codex sessions.
**Breaking changes**
- Reopening an issue that has been closed for longer than the window
(default 7 days) starts a fresh agent session instead of resuming the
old conversation.
- Disable with
`PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED=false`, or lengthen
the window with `PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DAYS`.
## What Changed
-
**`server/src/services/native-runtime/native-harness-backup-expiration.ts`
(new):** the sweep. It checks at most 200 sessions and expires at most
50 per batch, continuing from a cursor. A copy expires only when all of
these hold:
- every issue the session ran on is `done` or `cancelled`, closed before
the window;
- the copy was written before the window;
- no run of the session is queued, running, or unfinished;
- every native run finalization of the session is settled (`committed`
or `terminal_failure`), with no recovery state or lease owner;
- no `active`, `released`, `retained`, or `pending_cleanup` environment
lease references the copy (by backup stamp, session run, or recorded
lease);
- the session is not executing, initializing, warm, or closing in this
process, and its directory has no pending cleanup activation.
- **One transaction:** the database checks share-lock the issues and
delete the agent task session that points at the session, so a reopened
issue cannot resume the expired session.
- **Atomic, non-blocking removal:** the copy is renamed to
`failover-backups/.retiring-<UUID>`, and a final check looks for a run
of the session. If one appeared, or the check throws, the rename is
undone. Only a confirmed copy becomes `.expired-<UUID>` and is deleted
with asynchronous file operations, so a multi-gigabyte removal does not
block the event loop. A later sweep restores an unconfirmed
`.retiring-*` copy when `current` is missing, drops it when a newer
`current` exists, and finishes an interrupted deletion. The rest of the
session directory is kept.
- **`native-session-executor.ts`:** exports
`nativeSessionScopeDigestIsBusy`, which checks the in-process executing,
initializing, warm, and closing session maps.
- **`native-harness-backup-stamp.ts`:** exports the state base path the
sweep scans.
- **`config.ts` and `index.ts`:** three environment controls
(`_ENABLED`, `_DRY_RUN`, `_DAYS`) and the scheduled sweep. It starts two
minutes after boot, honors shutdown, standby, and idle drain, and logs
candidates, reclaimed bytes, cleared task sessions, and skip reasons.
- **Docs:** `doc/recoverable-disk-cleanup.md` describes the sweep;
`doc/DEVELOPING.md` lists the variables.
- No code comments were added.
## Verification
- `pnpm exec vitest run
src/__tests__/native-harness-backup-expiration.test.ts
src/__tests__/native-harness-backup-expiration-config.test.ts` (in
`server/`): 30 passed, 24 of them against embedded Postgres plus 6
config tests. Cases:
- expiry for done and cancelled issues, including clearing the task
session;
- each keep condition: open issue, recently closed, live run, pending
finalization, recovery in progress, live lease, recent copy, cleanup
activation, in-process session;
- settled finalizations and expired leases do not block;
- dry run changes nothing;
- bounded batches continue across ticks;
- an interrupted deletion is finished, and an unconfirmed retiring copy
is restored or dropped;
- the configured clock controls the closed window.
- **Mutation checks:** treating `in_progress` as closed, ignoring live
leases, ignoring the configured clock, or not restoring a retiring copy
each fail the corresponding tests.
- `pnpm exec vitest run native-harness-backup config` (in `server/`): 19
files, 293 tests passed.
- `pnpm run typecheck` (in `server/`): pass.
- **Real server:** after building and bundling,
`scripts/server-boot-smoke.sh node server/dist-runtime/index.mjs`
reached healthy.
- **Manual equivalent on a live instance:** applying these checks by
hand (with a 3-day window) to 242 copies freed 7.1 GB. Most of the
remainder belonged to open issues or live leases.
## Risks
- **Reopen after the window:** the agent starts a fresh session and
loses the old conversation's provider history. Issue comments,
documents, and work products are unaffected.
- **Race with a wake:** the checks share-lock the issues, so a reopen
waits for the transaction. A run created for the session between commit
and rename is caught by the post-rename check, which restores the copy.
Such a run would find no task session and start fresh.
- **Missing owner run:** if the owning run row is gone (for example,
after an agent was deleted), the copy is kept, because the sweep cannot
prove the session is closed.
- **Untouched:** quarantined session roots and `continuity-breaks`
archives are not touched by this change.
- **Rollback:** set
`PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED=false`, or revert.
Expired copies cannot be restored.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, local builds, tests and boots,
read-only measurements on a live instance).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
