Files
PaperClipAI/server/src/services/native-runtime/native-harness-backup-stamp.ts
T
Devin FoleyandPaperclip 1f3695ac67 fix: expire native session failover backups after their issues close (#16866)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent sessions that run in a remote sandbox keep a host-side
failover copy of the sandbox's harness state, so a replacement sandbox
can resume the conversation
> - Each session keeps that copy forever, and one copy can reach several
gigabytes
> - On a busy instance these copies fill the volume, and no existing
cleanup touches them
> - Once a session's work is closed, its copy matters only if someone
reopens that work later
> - This pull request expires the copy after the session's issues have
been closed for a configurable window, and makes a reopened issue start
a fresh session
> - The benefit is bounded disk use without breaking active,
recoverable, or leased sessions

## Linked Issues or Issue Description

Bug report described in-PR (bug report template).

**What happened?**
After every remote turn, `checkpointRemoteRunner` writes the sandbox's
runner state and provider session home (for example `codex-home`) to
`durable-sessions/<scope>/failover-backups/current`. Only the same
session's next checkpoint replaces it. No code path deletes a session's
copy when the session ends, its issue closes, or its agent goes away, so
the instance volume fills. On one instance:
- 246 sessions held 17 GB of copies, 84 of them older than a week;
- 24 new sessions added 5 GB in a single day, one copy alone 3.2 GB.

**Expected behavior**
Disk used by failover copies stays bounded. A copy is kept while its
session can still resume (open issue, live run, pending recovery, or
live lease), and expires once the session's issues have been closed for
a while. A reopened issue then starts a fresh native session, instead of
failing with `runner_harness_backup_unavailable` on a replacement
sandbox.

**Steps to reproduce**
1. Run agents with a native runtime in a remote sandbox environment.
2. Let them complete many issues.
3. Watch
`<instance>/runtime/paperclip-runner/durable-sessions/*/failover-backups/current`
grow without bound after the issues close.

**Paperclip version or commit:** `master` at 9179d8dcc3.
**Deployment mode:** server with remote sandbox environments.
**Agent adapter(s) involved:** not adapter-specific (core bug); observed
with Codex sessions.

**Breaking changes**
- Reopening an issue that has been closed for longer than the window
(default 7 days) starts a fresh agent session instead of resuming the
old conversation.
- Disable with
`PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED=false`, or lengthen
the window with `PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DAYS`.

## What Changed

-
**`server/src/services/native-runtime/native-harness-backup-expiration.ts`
(new):** the sweep. It checks at most 200 sessions and expires at most
50 per batch, continuing from a cursor. A copy expires only when all of
these hold:
- every issue the session ran on is `done` or `cancelled`, closed before
the window;
  - the copy was written before the window;
  - no run of the session is queued, running, or unfinished;
- every native run finalization of the session is settled (`committed`
or `terminal_failure`), with no recovery state or lease owner;
- no `active`, `released`, `retained`, or `pending_cleanup` environment
lease references the copy (by backup stamp, session run, or recorded
lease);
- the session is not executing, initializing, warm, or closing in this
process, and its directory has no pending cleanup activation.
- **One transaction:** the database checks share-lock the issues and
delete the agent task session that points at the session, so a reopened
issue cannot resume the expired session.
- **Atomic, non-blocking removal:** the copy is renamed to
`failover-backups/.retiring-<UUID>`, and a final check looks for a run
of the session. If one appeared, or the check throws, the rename is
undone. Only a confirmed copy becomes `.expired-<UUID>` and is deleted
with asynchronous file operations, so a multi-gigabyte removal does not
block the event loop. A later sweep restores an unconfirmed
`.retiring-*` copy when `current` is missing, drops it when a newer
`current` exists, and finishes an interrupted deletion. The rest of the
session directory is kept.
- **`native-session-executor.ts`:** exports
`nativeSessionScopeDigestIsBusy`, which checks the in-process executing,
initializing, warm, and closing session maps.
- **`native-harness-backup-stamp.ts`:** exports the state base path the
sweep scans.
- **`config.ts` and `index.ts`:** three environment controls
(`_ENABLED`, `_DRY_RUN`, `_DAYS`) and the scheduled sweep. It starts two
minutes after boot, honors shutdown, standby, and idle drain, and logs
candidates, reclaimed bytes, cleared task sessions, and skip reasons.
- **Docs:** `doc/recoverable-disk-cleanup.md` describes the sweep;
`doc/DEVELOPING.md` lists the variables.
- No code comments were added.

## Verification

- `pnpm exec vitest run
src/__tests__/native-harness-backup-expiration.test.ts
src/__tests__/native-harness-backup-expiration-config.test.ts` (in
`server/`): 30 passed, 24 of them against embedded Postgres plus 6
config tests. Cases:
- expiry for done and cancelled issues, including clearing the task
session;
- each keep condition: open issue, recently closed, live run, pending
finalization, recovery in progress, live lease, recent copy, cleanup
activation, in-process session;
  - settled finalizations and expired leases do not block;
  - dry run changes nothing;
  - bounded batches continue across ticks;
- an interrupted deletion is finished, and an unconfirmed retiring copy
is restored or dropped;
  - the configured clock controls the closed window.
- **Mutation checks:** treating `in_progress` as closed, ignoring live
leases, ignoring the configured clock, or not restoring a retiring copy
each fail the corresponding tests.
- `pnpm exec vitest run native-harness-backup config` (in `server/`): 19
files, 293 tests passed.
- `pnpm run typecheck` (in `server/`): pass.
- **Real server:** after building and bundling,
`scripts/server-boot-smoke.sh node server/dist-runtime/index.mjs`
reached healthy.
- **Manual equivalent on a live instance:** applying these checks by
hand (with a 3-day window) to 242 copies freed 7.1 GB. Most of the
remainder belonged to open issues or live leases.

## Risks

- **Reopen after the window:** the agent starts a fresh session and
loses the old conversation's provider history. Issue comments,
documents, and work products are unaffected.
- **Race with a wake:** the checks share-lock the issues, so a reopen
waits for the transaction. A run created for the session between commit
and rename is caught by the post-rename check, which restores the copy.
Such a run would find no task session and start fresh.
- **Missing owner run:** if the owning run row is gone (for example,
after an agent was deleted), the copy is kept, because the sweep cannot
prove the session is closed.
- **Untouched:** quarantined session roots and `continuity-breaks`
archives are not touched by this change.
- **Rollback:** set
`PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED=false`, or revert.
Expired copies cannot be restored.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, local builds, tests and boots,
read-only measurements on a live instance).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-20 07:57:43 -07:00

229 lines
7.7 KiB
TypeScript

import { createHash } from "node:crypto";
import { lstatSync, readFileSync, readdirSync, readlinkSync } from "node:fs";
import { dirname, resolve } from "node:path";
import { resolvePaperclipInstanceRoot } from "../../home-paths.js";
export interface NativeHarnessBackupStamp {
schema: "paperclip.native-harness-backup-stamp.v2";
normalizedSessionId: string;
runnerInstanceId: string;
sessionScopeSha256: string;
sourceProviderLeaseId: string;
authorizedProviderLeaseId: string;
manifestSha256: string;
completedAt: string;
}
export function nativeHarnessStateBase(): string {
return resolve(
process.env.PAPERCLIP_RUNNER_STATE_DIR ??
resolve(
resolvePaperclipInstanceRoot(),
"runtime",
"paperclip-runner",
"durable-sessions",
),
);
}
function stateRootFromDigest(digest: string): string | null {
if (!/^[0-9a-f]{64}$/.test(digest)) return null;
const base = nativeHarnessStateBase();
const root = resolve(base, digest);
return dirname(root) === base ? root : null;
}
function isRealDirectory(path: string): boolean {
try {
const metadata = lstatSync(path);
return metadata.isDirectory() && !metadata.isSymbolicLink();
} catch {
return false;
}
}
function isRealFile(path: string): boolean {
try {
const metadata = lstatSync(path);
return metadata.isFile() && !metadata.isSymbolicLink();
} catch {
return false;
}
}
function digestDirectory(directory: string): { sha256: string; bytes: number } {
const hash = createHash("sha256");
let bytes = 0;
const visit = (current: string, relative: string) => {
const entries = readdirSync(current, { withFileTypes: true }).sort(
(left, right) => left.name.localeCompare(right.name),
);
if (entries.length === 0) hash.update(`directory:${relative}\0`);
for (const entry of entries) {
const path = resolve(current, entry.name);
const relativePath = relative ? `${relative}/${entry.name}` : entry.name;
const stats = lstatSync(path);
if (entry.isDirectory()) {
hash.update(`directory:${relativePath}:${stats.mode & 0o777}\0`);
visit(path, relativePath);
} else if (entry.isSymbolicLink()) {
hash.update(`symlink:${relativePath}:${readlinkSync(path)}\0`);
} else if (entry.isFile()) {
const contents = readFileSync(path);
bytes += contents.byteLength;
hash.update(
`file:${relativePath}:${stats.mode & 0o777}:${contents.byteLength}\0`,
);
hash.update(contents);
} else {
throw new Error(
`runner_harness_backup_unsupported_entry:${relativePath}`,
);
}
}
};
visit(directory, "");
return { sha256: `sha256:${hash.digest("hex")}`, bytes };
}
export function createNativeHarnessBackupStamp(input: {
manifestPath: string;
sessionScopeId: string;
authorizedProviderLeaseId: string;
normalizedSessionId: string;
runnerInstanceId: string;
completedAt: string;
}): NativeHarnessBackupStamp {
if (!input.authorizedProviderLeaseId) {
throw new Error("runner_harness_backup_lease_missing");
}
const sessionScopeSha256 = createHash("sha256")
.update(input.sessionScopeId)
.digest("hex");
const root = stateRootFromDigest(sessionScopeSha256);
if (!root || !isRealDirectory(root)) {
throw new Error("runner_harness_backup_scope_invalid");
}
const resolvedManifestPath = resolve(input.manifestPath);
const allowedManifestPaths = ["current", "previous"].map((candidate) =>
resolve(root, "failover-backups", candidate, "manifest.json"),
);
if (
!allowedManifestPaths.includes(resolvedManifestPath) ||
!isRealDirectory(dirname(resolvedManifestPath)) ||
!isRealFile(resolvedManifestPath)
) {
throw new Error("runner_harness_backup_scope_invalid");
}
const manifestBytes = readFileSync(input.manifestPath);
const manifest = JSON.parse(manifestBytes.toString("utf8")) as Record<
string,
unknown
>;
if (
manifest.schema !== "paperclip.native-harness-backup.v1" ||
manifest.normalizedSessionId !== input.normalizedSessionId ||
manifest.runnerInstanceId !== input.runnerInstanceId ||
manifest.completedAt !== input.completedAt ||
typeof manifest.sourceProviderLeaseId !== "string" ||
!manifest.sourceProviderLeaseId
) {
throw new Error("runner_harness_backup_scope_invalid");
}
return {
schema: "paperclip.native-harness-backup-stamp.v2",
normalizedSessionId: input.normalizedSessionId,
runnerInstanceId: input.runnerInstanceId,
sessionScopeSha256,
sourceProviderLeaseId: manifest.sourceProviderLeaseId,
authorizedProviderLeaseId: input.authorizedProviderLeaseId,
manifestSha256: `sha256:${createHash("sha256").update(manifestBytes).digest("hex")}`,
completedAt: input.completedAt,
};
}
export function verifyNativeHarnessBackupStamp(
value: unknown,
expectedProviderLeaseId: string,
): boolean {
if (!expectedProviderLeaseId) return false;
if (!value || typeof value !== "object" || Array.isArray(value)) return false;
const stamp = value as Record<string, unknown>;
if (
stamp.schema !== "paperclip.native-harness-backup-stamp.v2" ||
typeof stamp.normalizedSessionId !== "string" ||
!stamp.normalizedSessionId ||
typeof stamp.runnerInstanceId !== "string" ||
!stamp.runnerInstanceId ||
typeof stamp.sessionScopeSha256 !== "string" ||
typeof stamp.sourceProviderLeaseId !== "string" ||
!stamp.sourceProviderLeaseId ||
stamp.authorizedProviderLeaseId !== expectedProviderLeaseId ||
typeof stamp.manifestSha256 !== "string" ||
!stamp.manifestSha256.startsWith("sha256:")
)
return false;
const root = stateRootFromDigest(stamp.sessionScopeSha256);
if (!root || !isRealDirectory(root)) return false;
const backupRoot = resolve(root, "failover-backups");
for (const candidate of [
resolve(backupRoot, "current"),
resolve(backupRoot, "previous"),
]) {
const manifestPath = resolve(candidate, "manifest.json");
if (!isRealDirectory(candidate) || !isRealFile(manifestPath)) continue;
try {
const bytes = readFileSync(manifestPath);
const manifestSha256 = `sha256:${createHash("sha256").update(bytes).digest("hex")}`;
if (manifestSha256 !== stamp.manifestSha256) continue;
const manifest = JSON.parse(bytes.toString("utf8")) as Record<
string,
unknown
>;
if (
manifest.schema !== "paperclip.native-harness-backup.v1" ||
manifest.normalizedSessionId !== stamp.normalizedSessionId ||
manifest.runnerInstanceId !== stamp.runnerInstanceId ||
manifest.sourceProviderLeaseId !== stamp.sourceProviderLeaseId ||
!Array.isArray(manifest.directories) ||
manifest.directories.length === 0
)
continue;
let valid = true;
for (const entry of manifest.directories) {
if (!entry || typeof entry !== "object" || Array.isArray(entry)) {
valid = false;
break;
}
const declared = entry as Record<string, unknown>;
if (
typeof declared.name !== "string" ||
!/^[A-Za-z0-9._-]+$/.test(declared.name) ||
typeof declared.sha256 !== "string" ||
typeof declared.bytes !== "number"
) {
valid = false;
break;
}
const directory = resolve(candidate, declared.name);
if (!isRealDirectory(directory)) {
valid = false;
break;
}
const actual = digestDirectory(directory);
if (
actual.sha256 !== declared.sha256 ||
actual.bytes !== declared.bytes
) {
valid = false;
break;
}
}
if (valid) return true;
} catch {
// Try the previous atomically-published backup.
}
}
return false;
}