test(runner-e2e): delete each runner E2E instance when it ends and expire kept instances after 6 hours (#20246)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness boots a full temporary Paperclip instance
per wave: embedded Postgres, workspaces and caches, often 5–13 GB
> - At wave end it deleted that instance only when the wave passed with
no harness error and no `--keep`
> - Almost every chaos wave has at least one failing task, so almost
every instance stayed on disk
> - Over Nov 6–9, 359 `paperclip-runner-e2e-*` directories filled the
machine's disk (ENOSPC), which stopped a running storm wave
> - This pull request deletes the instance at the end of every wave,
smoke and baseline run unless it is explicitly kept or its processes
could not be stopped, and expires kept instances after 6 hours
> - The benefit is that chaos waves can run back to back without filling
the disk

## Linked Issues or Issue Description

**What happened?**
`main.ts` kept the temporary root whenever `keep || anyFailure ||
harnessError`. A failing task is the normal output of a chaos wave, so
instances accumulated until the disk was full. The codex K13 storm
w-20261109T113925-s9403 died after about a minute with every task cut
off.

**Steps to reproduce**
1. Run any wave in which at least one task fails.
2. The `paperclip-runner-e2e-*` directory under the OS temp dir stays
behind, with no expiry.

**Expected behavior**
The instance is removed when the wave ends. Evidence is already in
`results/`. Only an explicit `--keep`, or a stop failure that may leave
processes running, keeps it, and kept instances expire.

**Actual behavior**
Every failing wave leaks its instance permanently.

## What Changed

- New shared `tests/runner-e2e/instance-retention.ts`:
- `releaseInstance` deletes the root unless `--keep` was passed or the
processes could not be stopped. In those cases it writes a
`.paperclip-runner-e2e-kept.json` marker (owner, reason, time) and sets
the directory to 0700. An optional `retain` list keeps small evidence
files and removes everything else.
- `sweepExpiredKeptInstances` removes `paperclip-runner-e2e-*`
directories that carry the marker, are older than 6 hours, and have no
live `postmaster.pid` anywhere inside them, and that no live process
command line references. Each root is made removable first, and one
failed delete does not stop the sweep. Unmarked directories are never
touched, so existing directories from older runs stay until someone
decides on them. `recovery_state` markers never expire.
- `createRunnerE2ETemporaryRoot` runs the sweep, so every caller gets
it.
- Callers:
- A wave whose Daytona cleanup is still pending (environment DELETE
blocked, sandboxes still running after cleanup, or cleanup errors) keeps
its instance as `recovery_state`, because its database owns that
cleanup.
- The wave releases its instance before writing the ledger and summary,
so a failed report write cannot skip cleanup.
- Restart chaos `main.ts` (crash-point suite included) releases the
instance in its final cleanup, which also runs on failure and
interrupts. The crash-point task-count check moved above supervisor
creation so that error no longer leaks a fresh root.
- `ttft-baseline.ts` releases after stopping the supervisor, including
on SIGINT and SIGTERM.
- The rolling-deployment and controller-replacement smokes now delete
their roots, keeping only the evidence report, logs and screenshots. If
cleanup is unconfirmed, the root is kept with a marker.
- `launch.ts` marks its unconfirmed-remote-cleanup root as
`recovery_state`, which never expires, and its
`PAPERCLIP_RUNNER_E2E_KEEP_FAILED_PRIVATE` root as `keep`. Its existing
deletion path is unchanged, and the file gets one line shorter.
- README: the instance lifecycle and `--keep`.

## Verification

- `npx vitest run --config tests/runner-e2e/vitest.config.ts`: 328
files, 6915 passed. The new retention tests cover:
  - deleting a finished instance, and retaining only listed evidence;
  - keeping it with a marker and mode 0700;
- sweeping only marked, expired instances whose Postgres isn't running,
while sparing fresh, live, recovery-state, unmarked and unrelated
directories;
  - a stale `postmaster.pid`;
  - sweeping on `createRunnerE2ETemporaryRoot`.

The create-sweep test failed before `createRunnerE2ETemporaryRoot` ran
the sweep.
- `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 221
passed.
- `node scripts/typecheck-test-projects.mjs tests/runner-e2e` and
`tests/restart-chaos`: clean.
- No wave or smoke was run, because the disk is still nearly full.

## Risks

- A failed wave's instance is no longer available for post-mortem DB
inspection unless the wave ran with `--keep`. Evidence bundles in
`results/` are unchanged.
- The sweep relies on `postmaster.pid`. An instance whose Postgres was
started outside the harness, without that file, could be removed after 6
hours.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

tests/runner-e2e/controller-replacement-smoke.ts

@@ -17,6 +17,7 @@ import { paperclipConfigSchema } from "../../packages/shared/src/config-schema.j
 import { configuredControllerOutage, validControllerOutageReceipt, type ControllerOutage } from "./controller-outage.js";
 import { runnerE2EServerControlPaths } from "./harness-env.js";
 import { reserveRunnerE2EServerPort } from "./ports.js";
+import { releaseInstance } from "./instance-retention.js";
 import { createRunnerE2ETemporaryRoot, prepareRunnerE2EServerConfig } from "./server-config.js";
 import { createRunnerE2EServerStopper } from "./server-stop.js";
 import { cleanupControllerSmokeDatabase } from "./controller-smoke-cleanup.js";
@@ -196,7 +197,12 @@ try {
   report.cleanupConfirmed = cleanupErrors.length === 0;
   if (cleanupErrors.length) { report.cleanupErrors = cleanupErrors; report.passed = false; }
   await new Promise<void>(resolve => log.end(resolve));
-  await writeFile(reportPath, `${JSON.stringify(report, null, 2)}\n`, { mode: 0o600 });
+  try {
+    await writeFile(reportPath, `${JSON.stringify(report, null, 2)}\n`, { mode: 0o600 });
+  } finally {
+    await releaseInstance(temporaryRoot, { keep: false, stopFailed: !report.cleanupConfirmed, owner: "controller-replacement-smoke",
+      retain: entry => entry === path.basename(reportPath) || entry.endsWith(".log") }).catch(error => console.warn(`Temporary root cleanup failed at ${temporaryRoot}: ${String(error)}`));
+  }
   console.info(JSON.stringify({ passed: report.passed, evidence: reportPath, cleanupConfirmed: report.cleanupConfirmed }));
 }
 if (failure) throw failure;

tests/runner-e2e/instance-retention.test.ts

@@ -0,0 +1,112 @@
+import { existsSync } from "node:fs";
+import { chmod, mkdir, mkdtemp, rm, stat, writeFile } from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { afterEach, describe, expect, it } from "vitest";
+import { KEPT_INSTANCE_MARKER, KEPT_INSTANCE_MAX_AGE_MS, keepInstance, releaseInstance, sweepExpiredKeptInstances } from "./instance-retention.js";
+import { createRunnerE2ETemporaryRoot } from "./server-config.js";
+
+const roots: string[] = [];
+afterEach(async () => { await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true }))); });
+
+async function parent() {
+  const root = await mkdtemp(path.join(os.tmpdir(), "runner-e2e-retention-"));
+  roots.push(root);
+  return root;
+}
+async function instance(under: string, name: string) {
+  const root = path.join(under, `paperclip-runner-e2e-${name}`);
+  await mkdir(path.join(root, "home", "instances", "default", "db"), { recursive: true });
+  return root;
+}
+
+describe("instance retention", () => {
+  it("deletes a finished instance unless --keep or a failed stop asks to keep it", async () => {
+    const under = await parent();
+    const finished = await instance(under, "done");
+    expect(await releaseInstance(finished, { keep: false, stopFailed: false, owner: "w1" })).toEqual({ kept: false });
+    expect(existsSync(finished)).toBe(false);
+    const kept = await instance(under, "kept");
+    expect(await releaseInstance(kept, { keep: true, stopFailed: false, owner: "w2" }, 1_000)).toEqual({ kept: true, reason: "keep" });
+    expect(existsSync(path.join(kept, KEPT_INSTANCE_MARKER))).toBe(true);
+    expect((await stat(kept)).mode & 0o777).toBe(0o700);
+    const evidence = await instance(under, "evidence");
+    await writeFile(path.join(evidence, "report.json"), "{}");
+    expect(await releaseInstance(evidence, { keep: false, stopFailed: false, owner: "smoke", retain: entry => entry === "report.json" })).toEqual({ kept: false });
+    expect(existsSync(path.join(evidence, "report.json"))).toBe(true);
+    expect(existsSync(path.join(evidence, "home"))).toBe(false);
+    const remote = await instance(under, "remote");
+    expect(await releaseInstance(remote, { keep: false, stopFailed: false, remoteCleanupPending: true, owner: "w4" }))
+      .toEqual({ kept: true, reason: "recovery_state" });
+    const stuck = await instance(under, "stuck");
+    expect(await releaseInstance(stuck, { keep: false, stopFailed: true, owner: "w3" }, 1_000)).toEqual({ kept: true, reason: "stop_failed" });
+  });
+
+  it("sweeps only marked kept instances past their expiry whose Postgres is not running", async () => {
+    const under = await parent();
+    const now = 10 * KEPT_INSTANCE_MAX_AGE_MS;
+    const old = now - KEPT_INSTANCE_MAX_AGE_MS - 1;
+    const expired = await instance(under, "expired");
+    await keepInstance(expired, { owner: "w1", reason: "keep" }, old);
+    const fresh = await instance(under, "fresh");
+    await keepInstance(fresh, { owner: "w2", reason: "keep" }, now - 1_000);
+    const live = await instance(under, "live");
+    await keepInstance(live, { owner: "w3", reason: "keep" }, old);
+    await mkdir(path.join(live, "a", "home"), { recursive: true });
+    await writeFile(path.join(live, "a", "home", "postmaster.pid"), `${process.pid}\n/data\n`);
+    const recovery = await instance(under, "recovery");
+    await keepInstance(recovery, { owner: "launch", reason: "recovery_state" }, old);
+    const unmarked = await instance(under, "unmarked");
+    const unrelated = path.join(under, "something-else");
+    await mkdir(unrelated);
+    await writeFile(path.join(unrelated, KEPT_INSTANCE_MARKER), JSON.stringify({ keptAt: 0 }));
+
+    const runner = await instance(under, "runner");
+    await keepInstance(runner, { owner: "w4", reason: "stop_failed" }, old);
+
+    expect(await sweepExpiredKeptInstances(under, now, KEPT_INSTANCE_MAX_AGE_MS, () => [`node ${runner}/provider-bin/runnerd`])).toEqual([expired]);
+    expect(existsSync(expired)).toBe(false);
+    for (const survivor of [fresh, live, recovery, unmarked, unrelated, runner]) expect(existsSync(survivor)).toBe(true);
+  });
+
+  it("releases a finished instance whose immutable runtime bundles are read-only", async () => {
+    const under = await parent();
+    const finished = await instance(under, "bundles");
+    const bundle = path.join(finished, "home", "instances", "default", "runtime-context-assets", "bundles", "sha256-abc");
+    await mkdir(bundle, { recursive: true });
+    await writeFile(path.join(bundle, "AGENTS.md"), "x");
+    await chmod(bundle, 0o555);
+    await chmod(path.dirname(bundle), 0o555);
+    expect(await releaseInstance(finished, { keep: false, stopFailed: false, owner: "w1" })).toEqual({ kept: false });
+    expect(existsSync(finished)).toBe(false);
+  });
+
+  it("sweeps an expired instance that holds read-only directories", async () => {
+    const under = await parent();
+    const locked = await instance(under, "locked");
+    await keepInstance(locked, { owner: "w1", reason: "keep" }, 0);
+    const readOnly = path.join(locked, "home", "cache");
+    await mkdir(readOnly, { recursive: true });
+    await writeFile(path.join(readOnly, "entry"), "x");
+    await chmod(readOnly, 0o500);
+    expect(await sweepExpiredKeptInstances(under, KEPT_INSTANCE_MAX_AGE_MS + 1)).toEqual([locked]);
+    expect(existsSync(locked)).toBe(false);
+  });
+
+  it("treats a stale postmaster.pid whose process is gone as not running", async () => {
+    const under = await parent();
+    const stale = await instance(under, "stale");
+    await keepInstance(stale, { owner: "w1", reason: "stop_failed" }, 0);
+    await writeFile(path.join(stale, "home", "instances", "default", "db", "postmaster.pid"), "2147483646\n/data\n");
+    expect(await sweepExpiredKeptInstances(under, KEPT_INSTANCE_MAX_AGE_MS + 1)).toEqual([stale]);
+  });
+
+  it("sweeps expired kept instances whenever a caller creates a new temporary root", async () => {
+    const under = await parent();
+    const expired = await instance(under, "expired");
+    await keepInstance(expired, { owner: "w1", reason: "keep" }, Date.now() - KEPT_INSTANCE_MAX_AGE_MS - 1);
+    const created = await createRunnerE2ETemporaryRoot(under);
+    expect(existsSync(expired)).toBe(false);
+    expect(path.basename(created).startsWith("paperclip-runner-e2e-")).toBe(true);
+  });
+});

tests/runner-e2e/instance-retention.ts

@@ -0,0 +1,89 @@
+import { execFileSync } from "node:child_process";
+import { chmod, readdir, readFile, rm, writeFile } from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { makeDirectoryTreeRemovable } from "./directory-cleanup.js";
+
+export const KEPT_INSTANCE_MARKER = ".paperclip-runner-e2e-kept.json";
+export const KEPT_INSTANCE_MAX_AGE_MS = 6 * 60 * 60_000;
+export const RUNNER_E2E_INSTANCE_PREFIX = "paperclip-runner-e2e-";
+const POSTMASTER_SEARCH_DEPTH = 6;
+
+export type KeepReason = "keep" | "stop_failed" | "recovery_state";
+
+export async function keepInstance(root: string, input: { owner: string; reason: KeepReason }, now = Date.now()): Promise<void> {
+  await chmod(root, 0o700);
+  await writeFile(path.join(root, KEPT_INSTANCE_MARKER), `${JSON.stringify({ owner: input.owner, reason: input.reason, keptAt: now })}\n`, { mode: 0o600 });
+}
+
+export async function releaseInstance(root: string, input: { keep: boolean; stopFailed: boolean; owner: string;
+  remoteCleanupPending?: boolean; retain?: (entry: string) => boolean }, now = Date.now()): Promise<{ kept: false } | { kept: true; reason: KeepReason }> {
+  const reason: KeepReason | null = input.remoteCleanupPending ? "recovery_state" : input.stopFailed ? "stop_failed" : input.keep ? "keep" : null;
+  if (!reason) {
+    await makeDirectoryTreeRemovable(root).catch(() => undefined);
+    if (!input.retain) {
+      await rm(root, { recursive: true, force: true });
+      return { kept: false };
+    }
+    for (const entry of await readdir(root)) if (!input.retain(entry)) await rm(path.join(root, entry), { recursive: true, force: true });
+    return { kept: false };
+  }
+  await keepInstance(root, { owner: input.owner, reason }, now);
+  return { kept: true, reason };
+}
+
+function processAlive(pid: number): boolean {
+  try {
+    process.kill(pid, 0);
+    return true;
+  } catch (error) {
+    return (error as NodeJS.ErrnoException).code === "EPERM";
+  }
+}
+
+async function postgresRunning(directory: string, depth = POSTMASTER_SEARCH_DEPTH): Promise<boolean> {
+  const entries = await readdir(directory, { withFileTypes: true }).catch(() => []);
+  for (const entry of entries) {
+    if (entry.isFile() && entry.name === "postmaster.pid") {
+      const pid = Number((await readFile(path.join(directory, entry.name), "utf8").catch(() => "")).split("\n")[0]);
+      if (Number.isInteger(pid) && pid > 0 && processAlive(pid)) return true;
+    }
+  }
+  if (depth === 0) return false;
+  for (const entry of entries) {
+    if (entry.isDirectory() && !entry.isSymbolicLink() && await postgresRunning(path.join(directory, entry.name), depth - 1)) return true;
+  }
+  return false;
+}
+
+function liveProcessCommands(): string[] {
+  try {
+    return execFileSync("ps", ["-axww", "-o", "command="], { encoding: "utf8", maxBuffer: 64 * 1024 * 1024 }).split("\n");
+  } catch {
+    return [];
+  }
+}
+
+export async function sweepExpiredKeptInstances(parent = os.tmpdir(), now = Date.now(), maxAgeMs = KEPT_INSTANCE_MAX_AGE_MS,
+  processCommands: () => readonly string[] = liveProcessCommands): Promise<string[]> {
+  const removed: string[] = [];
+  let commands: readonly string[] | null = null;
+  for (const entry of await readdir(parent, { withFileTypes: true }).catch(() => [])) {
+    if (!entry.isDirectory() || !entry.name.startsWith(RUNNER_E2E_INSTANCE_PREFIX)) continue;
+    const root = path.join(parent, entry.name);
+    const marker = await readFile(path.join(root, KEPT_INSTANCE_MARKER), "utf8")
+      .then(text => JSON.parse(text) as { keptAt?: unknown; reason?: unknown }).catch(() => null);
+    if (typeof marker?.keptAt !== "number" || marker.reason === "recovery_state" || now - marker.keptAt <= maxAgeMs) continue;
+    if (await postgresRunning(root)) continue;
+    commands ??= processCommands();
+    if (commands.some(command => command.includes(root))) continue;
+    try {
+      await makeDirectoryTreeRemovable(root).catch(() => undefined);
+      await rm(root, { recursive: true, force: true });
+      removed.push(root);
+    } catch {
+      continue;
+    }
+  }
+  return removed;
+}

tests/runner-e2e/launch.ts

@@ -10,7 +10,7 @@ import { runnerE2EPlaywrightCaseTimeoutMs, runnerE2EPlaywrightInvocation, runner
 import { verifyInstalledDaytonaPlugin } from "./installed-daytona-plugin.js";
 import { assertInstalledCliSelection, assertInstalledStartupOnly, verifyInstalledCli } from "./installed-cli.js";
 import { createProcessTreeOwner, stopOwnedProcessTree } from "./process-tree-owner.js";
-import { createRunnerE2ETemporaryRoot } from "./server-config.js";
+import { createRunnerE2ETemporaryRoot, keepInstance } from "./server-config.js";
 import { createExternalControllerDatabase } from "./external-controller-database.js";
 import { createHash, randomBytes } from "node:crypto";
 import { prepareCodexCiSandbox, requiresCodexCiSandbox } from "./codex-ci-sandbox.js";
@@ -25,7 +25,6 @@ import os from "node:os";
 import path from "node:path";
 import {
   access,
-  chmod,
   cp,
   lstat,
   mkdir,
@@ -915,7 +914,7 @@ async function runAttempt(input: {
     if (preserveRecovery || cleanupError) {
       // Remote allocation cleanup is journaled in this instance's database.
       // Deleting it after the controller exits would strand uncertain creates.
-      await chmod(temporaryRoot, 0o700);
+      await keepInstance(temporaryRoot, { owner: "runner-e2e-launch", reason: "recovery_state" });
       cleanupError = new Error(`Preserving private recovery state after unconfirmed cleanup: ${temporaryRoot}`);
     } else if (shouldKeepFailedDiagnostics({
       enabled: process.env.PAPERCLIP_RUNNER_E2E_KEEP_FAILED_PRIVATE === "1",
@@ -924,7 +923,7 @@ async function runAttempt(input: {
     })) {
       // Explicit diagnosis only. Confirmed resource cleanup stays confirmed;
       // keep private traces for investigation without publishing provider data.
-      await chmod(temporaryRoot, 0o700);
+      await keepInstance(temporaryRoot, { owner: "runner-e2e-launch", reason: "keep" });
       console.warn(`Retained private failed-case diagnostics: ${temporaryRoot}`);
     } else if (
       temporaryRoot.startsWith(`${temporaryParent}${path.sep}paperclip-runner-e2e-`)

tests/runner-e2e/rolling-deployment-smoke.ts

@@ -12,6 +12,7 @@ import { setTimeout as delay } from "node:timers/promises";
 import { startEmbeddedPostgresTestDatabase } from "../../packages/db/src/index.js";
 import { createExternalControllerDatabase } from "./external-controller-database.js";
 import { reserveRunnerE2EServerPort } from "./ports.js";
+import { releaseInstance } from "./instance-retention.js";
 import { createRunnerE2ETemporaryRoot } from "./server-config.js";
 import { openRollingDeploymentBrowser } from "./rolling-deployment-browser.js";
 import { startRollingDeploymentProcess } from "./rolling-deployment-process.js";
@@ -305,7 +306,12 @@ try {
   report.cleanupConfirmed = cleanupErrors.length === 0;
   if (cleanupErrors.length) report.cleanupErrors = cleanupErrors;
   report.passed = !report.error && checks.length > 0 && checks.every(item => item.passed) && report.cleanupConfirmed;
-  await saveReport();
+  try {
+    await saveReport();
+  } finally {
+    await releaseInstance(root, { keep: false, stopFailed: !report.cleanupConfirmed, owner: "rolling-deployment-smoke", retain: entry => entry === path.basename(reportPath) || /\.(log|png)$/.test(entry) })
+      .catch(error => console.warn(`Temporary root cleanup failed at ${root}: ${String(error)}`));
+  }
   console.info(JSON.stringify({ passed: report.passed, checks, evidence: reportPath, cleanupConfirmed: report.cleanupConfirmed }));
 }
 if (!report.passed) process.exitCode = 1;

tests/runner-e2e/server-config.ts

@@ -2,10 +2,13 @@ import { mkdir, mkdtemp, realpath, readFile, writeFile } from "node:fs/promises"
 import os from "node:os";
 import path from "node:path";
 import { paperclipConfigSchema } from "../../packages/shared/src/config-schema.js";
+import { sweepExpiredKeptInstances } from "./instance-retention.js";
+export { keepInstance } from "./instance-retention.js";
 import { reserveRunnerE2EDatabasePort } from "./ports.js";
 
 /** Runtime instructions and native filesystem grants must use the same path. */
 export async function createRunnerE2ETemporaryRoot(parent = os.tmpdir()) {
+  await sweepExpiredKeptInstances(parent).catch(() => []);
   return await realpath(await mkdtemp(path.join(parent, "paperclip-runner-e2e-")));
 }