mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-30 21:38:42 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent sessions that run in a remote sandbox keep a host-side
failover copy of the sandbox's harness state, so a replacement sandbox
can resume the conversation
> - Each session keeps that copy forever, and one copy can reach several
gigabytes
> - On a busy instance these copies fill the volume, and no existing
cleanup touches them
> - Once a session's work is closed, its copy matters only if someone
reopens that work later
> - This pull request expires the copy after the session's issues have
been closed for a configurable window, and makes a reopened issue start
a fresh session
> - The benefit is bounded disk use without breaking active,
recoverable, or leased sessions
## Linked Issues or Issue Description
Bug report described in-PR (bug report template).
**What happened?**
After every remote turn, `checkpointRemoteRunner` writes the sandbox's
runner state and provider session home (for example `codex-home`) to
`durable-sessions/<scope>/failover-backups/current`. Only the same
session's next checkpoint replaces it. No code path deletes a session's
copy when the session ends, its issue closes, or its agent goes away, so
the instance volume fills. On one instance:
- 246 sessions held 17 GB of copies, 84 of them older than a week;
- 24 new sessions added 5 GB in a single day, one copy alone 3.2 GB.
**Expected behavior**
Disk used by failover copies stays bounded. A copy is kept while its
session can still resume (open issue, live run, pending recovery, or
live lease), and expires once the session's issues have been closed for
a while. A reopened issue then starts a fresh native session, instead of
failing with `runner_harness_backup_unavailable` on a replacement
sandbox.
**Steps to reproduce**
1. Run agents with a native runtime in a remote sandbox environment.
2. Let them complete many issues.
3. Watch
`<instance>/runtime/paperclip-runner/durable-sessions/*/failover-backups/current`
grow without bound after the issues close.
**Paperclip version or commit:** `master` at 9179d8dcc3.
**Deployment mode:** server with remote sandbox environments.
**Agent adapter(s) involved:** not adapter-specific (core bug); observed
with Codex sessions.
**Breaking changes**
- Reopening an issue that has been closed for longer than the window
(default 7 days) starts a fresh agent session instead of resuming the
old conversation.
- Disable with
`PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED=false`, or lengthen
the window with `PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DAYS`.
## What Changed
-
**`server/src/services/native-runtime/native-harness-backup-expiration.ts`
(new):** the sweep. It checks at most 200 sessions and expires at most
50 per batch, continuing from a cursor. A copy expires only when all of
these hold:
- every issue the session ran on is `done` or `cancelled`, closed before
the window;
- the copy was written before the window;
- no run of the session is queued, running, or unfinished;
- every native run finalization of the session is settled (`committed`
or `terminal_failure`), with no recovery state or lease owner;
- no `active`, `released`, `retained`, or `pending_cleanup` environment
lease references the copy (by backup stamp, session run, or recorded
lease);
- the session is not executing, initializing, warm, or closing in this
process, and its directory has no pending cleanup activation.
- **One transaction:** the database checks share-lock the issues and
delete the agent task session that points at the session, so a reopened
issue cannot resume the expired session.
- **Atomic, non-blocking removal:** the copy is renamed to
`failover-backups/.retiring-<UUID>`, and a final check looks for a run
of the session. If one appeared, or the check throws, the rename is
undone. Only a confirmed copy becomes `.expired-<UUID>` and is deleted
with asynchronous file operations, so a multi-gigabyte removal does not
block the event loop. A later sweep restores an unconfirmed
`.retiring-*` copy when `current` is missing, drops it when a newer
`current` exists, and finishes an interrupted deletion. The rest of the
session directory is kept.
- **`native-session-executor.ts`:** exports
`nativeSessionScopeDigestIsBusy`, which checks the in-process executing,
initializing, warm, and closing session maps.
- **`native-harness-backup-stamp.ts`:** exports the state base path the
sweep scans.
- **`config.ts` and `index.ts`:** three environment controls
(`_ENABLED`, `_DRY_RUN`, `_DAYS`) and the scheduled sweep. It starts two
minutes after boot, honors shutdown, standby, and idle drain, and logs
candidates, reclaimed bytes, cleared task sessions, and skip reasons.
- **Docs:** `doc/recoverable-disk-cleanup.md` describes the sweep;
`doc/DEVELOPING.md` lists the variables.
- No code comments were added.
## Verification
- `pnpm exec vitest run
src/__tests__/native-harness-backup-expiration.test.ts
src/__tests__/native-harness-backup-expiration-config.test.ts` (in
`server/`): 30 passed, 24 of them against embedded Postgres plus 6
config tests. Cases:
- expiry for done and cancelled issues, including clearing the task
session;
- each keep condition: open issue, recently closed, live run, pending
finalization, recovery in progress, live lease, recent copy, cleanup
activation, in-process session;
- settled finalizations and expired leases do not block;
- dry run changes nothing;
- bounded batches continue across ticks;
- an interrupted deletion is finished, and an unconfirmed retiring copy
is restored or dropped;
- the configured clock controls the closed window.
- **Mutation checks:** treating `in_progress` as closed, ignoring live
leases, ignoring the configured clock, or not restoring a retiring copy
each fail the corresponding tests.
- `pnpm exec vitest run native-harness-backup config` (in `server/`): 19
files, 293 tests passed.
- `pnpm run typecheck` (in `server/`): pass.
- **Real server:** after building and bundling,
`scripts/server-boot-smoke.sh node server/dist-runtime/index.mjs`
reached healthy.
- **Manual equivalent on a live instance:** applying these checks by
hand (with a 3-day window) to 242 copies freed 7.1 GB. Most of the
remainder belonged to open issues or live leases.
## Risks
- **Reopen after the window:** the agent starts a fresh session and
loses the old conversation's provider history. Issue comments,
documents, and work products are unaffected.
- **Race with a wake:** the checks share-lock the issues, so a reopen
waits for the transaction. A run created for the session between commit
and rename is caught by the post-rename check, which restores the copy.
Such a run would find no task session and start fresh.
- **Missing owner run:** if the owning run row is gone (for example,
after an agent was deleted), the copy is kept, because the sweep cannot
prove the session is closed.
- **Untouched:** quarantined session roots and `continuity-breaks`
archives are not touched by this change.
- **Rollback:** set
`PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED=false`, or revert.
Expired copies cannot be restored.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, local builds, tests and boots,
read-only measurements on a live instance).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
229 lines
7.7 KiB
TypeScript
229 lines
7.7 KiB
TypeScript
import { createHash } from "node:crypto";
|
|
import { lstatSync, readFileSync, readdirSync, readlinkSync } from "node:fs";
|
|
import { dirname, resolve } from "node:path";
|
|
import { resolvePaperclipInstanceRoot } from "../../home-paths.js";
|
|
|
|
export interface NativeHarnessBackupStamp {
|
|
schema: "paperclip.native-harness-backup-stamp.v2";
|
|
normalizedSessionId: string;
|
|
runnerInstanceId: string;
|
|
sessionScopeSha256: string;
|
|
sourceProviderLeaseId: string;
|
|
authorizedProviderLeaseId: string;
|
|
manifestSha256: string;
|
|
completedAt: string;
|
|
}
|
|
|
|
export function nativeHarnessStateBase(): string {
|
|
return resolve(
|
|
process.env.PAPERCLIP_RUNNER_STATE_DIR ??
|
|
resolve(
|
|
resolvePaperclipInstanceRoot(),
|
|
"runtime",
|
|
"paperclip-runner",
|
|
"durable-sessions",
|
|
),
|
|
);
|
|
}
|
|
|
|
function stateRootFromDigest(digest: string): string | null {
|
|
if (!/^[0-9a-f]{64}$/.test(digest)) return null;
|
|
const base = nativeHarnessStateBase();
|
|
const root = resolve(base, digest);
|
|
return dirname(root) === base ? root : null;
|
|
}
|
|
|
|
function isRealDirectory(path: string): boolean {
|
|
try {
|
|
const metadata = lstatSync(path);
|
|
return metadata.isDirectory() && !metadata.isSymbolicLink();
|
|
} catch {
|
|
return false;
|
|
}
|
|
}
|
|
|
|
function isRealFile(path: string): boolean {
|
|
try {
|
|
const metadata = lstatSync(path);
|
|
return metadata.isFile() && !metadata.isSymbolicLink();
|
|
} catch {
|
|
return false;
|
|
}
|
|
}
|
|
|
|
function digestDirectory(directory: string): { sha256: string; bytes: number } {
|
|
const hash = createHash("sha256");
|
|
let bytes = 0;
|
|
const visit = (current: string, relative: string) => {
|
|
const entries = readdirSync(current, { withFileTypes: true }).sort(
|
|
(left, right) => left.name.localeCompare(right.name),
|
|
);
|
|
if (entries.length === 0) hash.update(`directory:${relative}\0`);
|
|
for (const entry of entries) {
|
|
const path = resolve(current, entry.name);
|
|
const relativePath = relative ? `${relative}/${entry.name}` : entry.name;
|
|
const stats = lstatSync(path);
|
|
if (entry.isDirectory()) {
|
|
hash.update(`directory:${relativePath}:${stats.mode & 0o777}\0`);
|
|
visit(path, relativePath);
|
|
} else if (entry.isSymbolicLink()) {
|
|
hash.update(`symlink:${relativePath}:${readlinkSync(path)}\0`);
|
|
} else if (entry.isFile()) {
|
|
const contents = readFileSync(path);
|
|
bytes += contents.byteLength;
|
|
hash.update(
|
|
`file:${relativePath}:${stats.mode & 0o777}:${contents.byteLength}\0`,
|
|
);
|
|
hash.update(contents);
|
|
} else {
|
|
throw new Error(
|
|
`runner_harness_backup_unsupported_entry:${relativePath}`,
|
|
);
|
|
}
|
|
}
|
|
};
|
|
visit(directory, "");
|
|
return { sha256: `sha256:${hash.digest("hex")}`, bytes };
|
|
}
|
|
|
|
export function createNativeHarnessBackupStamp(input: {
|
|
manifestPath: string;
|
|
sessionScopeId: string;
|
|
authorizedProviderLeaseId: string;
|
|
normalizedSessionId: string;
|
|
runnerInstanceId: string;
|
|
completedAt: string;
|
|
}): NativeHarnessBackupStamp {
|
|
if (!input.authorizedProviderLeaseId) {
|
|
throw new Error("runner_harness_backup_lease_missing");
|
|
}
|
|
const sessionScopeSha256 = createHash("sha256")
|
|
.update(input.sessionScopeId)
|
|
.digest("hex");
|
|
const root = stateRootFromDigest(sessionScopeSha256);
|
|
if (!root || !isRealDirectory(root)) {
|
|
throw new Error("runner_harness_backup_scope_invalid");
|
|
}
|
|
const resolvedManifestPath = resolve(input.manifestPath);
|
|
const allowedManifestPaths = ["current", "previous"].map((candidate) =>
|
|
resolve(root, "failover-backups", candidate, "manifest.json"),
|
|
);
|
|
if (
|
|
!allowedManifestPaths.includes(resolvedManifestPath) ||
|
|
!isRealDirectory(dirname(resolvedManifestPath)) ||
|
|
!isRealFile(resolvedManifestPath)
|
|
) {
|
|
throw new Error("runner_harness_backup_scope_invalid");
|
|
}
|
|
const manifestBytes = readFileSync(input.manifestPath);
|
|
const manifest = JSON.parse(manifestBytes.toString("utf8")) as Record<
|
|
string,
|
|
unknown
|
|
>;
|
|
if (
|
|
manifest.schema !== "paperclip.native-harness-backup.v1" ||
|
|
manifest.normalizedSessionId !== input.normalizedSessionId ||
|
|
manifest.runnerInstanceId !== input.runnerInstanceId ||
|
|
manifest.completedAt !== input.completedAt ||
|
|
typeof manifest.sourceProviderLeaseId !== "string" ||
|
|
!manifest.sourceProviderLeaseId
|
|
) {
|
|
throw new Error("runner_harness_backup_scope_invalid");
|
|
}
|
|
return {
|
|
schema: "paperclip.native-harness-backup-stamp.v2",
|
|
normalizedSessionId: input.normalizedSessionId,
|
|
runnerInstanceId: input.runnerInstanceId,
|
|
sessionScopeSha256,
|
|
sourceProviderLeaseId: manifest.sourceProviderLeaseId,
|
|
authorizedProviderLeaseId: input.authorizedProviderLeaseId,
|
|
manifestSha256: `sha256:${createHash("sha256").update(manifestBytes).digest("hex")}`,
|
|
completedAt: input.completedAt,
|
|
};
|
|
}
|
|
|
|
export function verifyNativeHarnessBackupStamp(
|
|
value: unknown,
|
|
expectedProviderLeaseId: string,
|
|
): boolean {
|
|
if (!expectedProviderLeaseId) return false;
|
|
if (!value || typeof value !== "object" || Array.isArray(value)) return false;
|
|
const stamp = value as Record<string, unknown>;
|
|
if (
|
|
stamp.schema !== "paperclip.native-harness-backup-stamp.v2" ||
|
|
typeof stamp.normalizedSessionId !== "string" ||
|
|
!stamp.normalizedSessionId ||
|
|
typeof stamp.runnerInstanceId !== "string" ||
|
|
!stamp.runnerInstanceId ||
|
|
typeof stamp.sessionScopeSha256 !== "string" ||
|
|
typeof stamp.sourceProviderLeaseId !== "string" ||
|
|
!stamp.sourceProviderLeaseId ||
|
|
stamp.authorizedProviderLeaseId !== expectedProviderLeaseId ||
|
|
typeof stamp.manifestSha256 !== "string" ||
|
|
!stamp.manifestSha256.startsWith("sha256:")
|
|
)
|
|
return false;
|
|
const root = stateRootFromDigest(stamp.sessionScopeSha256);
|
|
if (!root || !isRealDirectory(root)) return false;
|
|
const backupRoot = resolve(root, "failover-backups");
|
|
for (const candidate of [
|
|
resolve(backupRoot, "current"),
|
|
resolve(backupRoot, "previous"),
|
|
]) {
|
|
const manifestPath = resolve(candidate, "manifest.json");
|
|
if (!isRealDirectory(candidate) || !isRealFile(manifestPath)) continue;
|
|
try {
|
|
const bytes = readFileSync(manifestPath);
|
|
const manifestSha256 = `sha256:${createHash("sha256").update(bytes).digest("hex")}`;
|
|
if (manifestSha256 !== stamp.manifestSha256) continue;
|
|
const manifest = JSON.parse(bytes.toString("utf8")) as Record<
|
|
string,
|
|
unknown
|
|
>;
|
|
if (
|
|
manifest.schema !== "paperclip.native-harness-backup.v1" ||
|
|
manifest.normalizedSessionId !== stamp.normalizedSessionId ||
|
|
manifest.runnerInstanceId !== stamp.runnerInstanceId ||
|
|
manifest.sourceProviderLeaseId !== stamp.sourceProviderLeaseId ||
|
|
!Array.isArray(manifest.directories) ||
|
|
manifest.directories.length === 0
|
|
)
|
|
continue;
|
|
let valid = true;
|
|
for (const entry of manifest.directories) {
|
|
if (!entry || typeof entry !== "object" || Array.isArray(entry)) {
|
|
valid = false;
|
|
break;
|
|
}
|
|
const declared = entry as Record<string, unknown>;
|
|
if (
|
|
typeof declared.name !== "string" ||
|
|
!/^[A-Za-z0-9._-]+$/.test(declared.name) ||
|
|
typeof declared.sha256 !== "string" ||
|
|
typeof declared.bytes !== "number"
|
|
) {
|
|
valid = false;
|
|
break;
|
|
}
|
|
const directory = resolve(candidate, declared.name);
|
|
if (!isRealDirectory(directory)) {
|
|
valid = false;
|
|
break;
|
|
}
|
|
const actual = digestDirectory(directory);
|
|
if (
|
|
actual.sha256 !== declared.sha256 ||
|
|
actual.bytes !== declared.bytes
|
|
) {
|
|
valid = false;
|
|
break;
|
|
}
|
|
}
|
|
if (valid) return true;
|
|
} catch {
|
|
// Try the previous atomically-published backup.
|
|
}
|
|
}
|
|
return false;
|
|
}
|