fix(native): preserve parked Runners across controller restarts (#20586)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native execution parks a released sandbox Runner for a later turn.
> - The warm idle timer detaches Core and leaves Runner retention to the
host.
> - Controller shutdown instead stops recently parked Runners through
checkpoint cleanup.
> - This makes reuse depend on whether the idle timer fired before
shutdown.
> - This pull request uses the same detach operation for idle expiry and
shutdown.
> - The benefit is consistent Runner retention without a new retention
or recovery mechanism.

## Linked Issues or Issue Description

Fixes: #20012.
Related: #20005 unified cached and adopted process reuse. Closed #19975
included this shutdown change within a broader retention proposal.

## What Changed

- Detach confirmed released entries during controller shutdown. Keep
busy, unreleased, and close-marked entries on the existing cleanup path.
- Share the detach operation with idle expiry and cached-owner handoff.
- Record that a local handle has started detaching. Later cleanup cannot
fall back to stopping its Runner after a partial detach failure.
- Preserve explicit environment deletion, release proof, admission, and
process-reuse checks.
- Add Codex and Claude restart, detach-failure, and environment-deletion
regressions. Add real Runner adoption cases after the restart sweep,
including managed GitHub rebinding.
- Document the lifetime rule and make the shutdown warning cover both
detach and checkpoint failures.

## Verification

- Final source revision: `e4bb6ecbd7`. Local environment: macOS arm64,
Node 26.12.0.
- New controller-restart and detach-failure tests fail against unchanged
master: 4 expected failures. Old shutdown calls close or reports success
instead of reporting detach failure.
- Focused executor tests: 22 passed.
- Full native session executor suite: 1,328 passed.
- `cd server && node scripts/typecheck-layers.mjs`: passed all source
and test layers.
- `node scripts/check-module-boundaries.mjs` and `git diff --check`:
passed.
- Manual diff review and added-line credential/PII scan: passed.
- `cd server && node ../node_modules/vitest/vitest.mjs run
src/services/native-runtime/plugin-runner-release-resume.integration.test.ts
--bail=1 --reporter=verbose`: 14 passed on this revision. This includes
new Codex and Claude restart cases and managed GitHub rebinding.
- Failure history: an earlier concurrent run had 4 passes and 10
failures. Its first failure was the existing Claude cached case
exceeding 60 seconds. Later cases encountered a leftover prototype spy,
recursive calls, and startup timeouts. No source changed before the
passing full rerun. A clean worktree of unchanged master (`e01fe7f84a`)
passed all 11 original cases. The first timeout's exact cause remains
unknown; the later pass does not establish that cause.
- Standard Greptile review: 5/5 on
`e4bb6ecbd7bd965b5f56507d2c42c580e8a1d547`, with no actionable findings.
- `node scripts/local-verify.mjs --explain` returned exit 2 because the
execution-semantics document requires broad CI. Required CI will supply
broad typecheck, test, and build coverage.
- A hosted deployment and an actual controller process replacement have
not been run. The added integration cases exercise the shutdown sweep
and a new adopting transport against a real Runner.

## Risks

- SIGINT and SIGTERM preserve confirmed parked Runners, including
ordinary service stops. The host still owns eviction and retention.
- Detach failure retains a fenced local owner and reports failure. Core
does not infer permission to stop the Runner from that failure.
- Reuse remains conditional on current admission, process configuration,
and host availability.
- No persisted schema, checkpoint format, launch identity, or migration
changes. State from the previous release uses the existing adoption or
verified restore paths.

## Model Used

- OpenAI Codex, GPT-6, with reasoning, repository inspection, tool use,
and local test execution. The exact deployment identifier and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Nicky LeachandPaperclip authored and GitHub committed 2026-11-10 16:48:30 +00:00
1 parent 765ccd8691
commit 2e3cbe57a4
5 files changed
+65 -20

No files matched your search

@@ -16910,14 +16910,19 @@ describe("runnerd provider runtime wiring", () => {
} finally {owner.mockRestore();await rm(home,{recursive:true,force:true});}
});
it.each((["codex", "claude"] as const).flatMap(provider => [[false, false, false, false], [true, false, false, false], [false, true, false, false], [false, false, true, false], ...(provider === "codex" ? [[false, false, false, true]] : [])].map(([unknown, shutdown, journalFailure, deadline]) => [provider, unknown, shutdown, journalFailure, deadline] as const)))(
"releases settled %s only under its current owner and detaches its cache locally (unknown=%s, shutdown=%s, journalFailure=%s, deadline=%s)", async (provider, unknown, shutdown, journalFailure, deadline) => {
it.each((["codex", "claude"] as const).flatMap(provider => [
...[[false, false, false, false], [true, false, false, false], [false, true, false, false], [false, false, true, false], ...(provider === "codex" ? [[false, false, false, true]] : [])]
.map(([unknown, shutdown, journalFailure, deadline]) => [provider, unknown, shutdown, journalFailure, deadline, "idle timeout"] as const),
...(["controller restart", "detach failure", "environment deletion"] as const)
.map(retirement => [provider, false, false, false, false, retirement] as const),
]))(
"releases settled %s only under its current owner and retires its cache (unknown=%s, shutdown=%s, journalFailure=%s, deadline=%s, retirement=%s)", async (provider, unknown, shutdown, journalFailure, deadline, retirement) => {
await closeIdleWarmNativeSessionsForRestart();
expect(readNativeSessionIdleWork()).toBe("none");
const previousHome = process.env.PAPERCLIP_HOME;
process.env.PAPERCLIP_HOME = isolatedStateDirectory;
const current = scopedNativeExecution(`released-${provider}-${unknown}`);
current.session.lifecyclePolicy = { mode: "warm", idleTimeoutMs: unknown ? 750 : 250 };
current.session.lifecyclePolicy = { mode: "warm", idleTimeoutMs: retirement === "idle timeout" ? unknown ? 750 : 250 : 60_000 };
if (provider === "claude") current.provider = { kind: "acpx", agent: "claude", model: "fixture", permissionMode: "deny-all" } as never;
const source = { runnerInstanceId: "released-runner", environmentLeaseId: "released-lease", runId: current.binding.runId,
normalizedSessionId: current.session.normalizedSessionId!, turnId: "settled-turn", itemId: "settled-item" };
@@ -17055,7 +17060,28 @@ describe("runnerd provider runtime wiring", () => {
expect(state.execute).toHaveBeenCalledTimes(1);
expect(JSON.parse(await readFile(path, "utf8")).releasedRunner).toEqual(envelope.releasedRunner);
}
await vi.waitFor(() => expect(detachControllerForRestart).toHaveBeenCalledOnce());
if (retirement === "environment deletion") {
expect(await closeWarmNativeSessionsForEnvironment({ environmentId: target.environmentId, reason: "environment deleted" }))
.toEqual({ closed: 1, busy: 0, failed: 0 });
expect(close).toHaveBeenCalledExactlyOnceWith({ reason: "environment deleted" });
expect(detachControllerForRestart).not.toHaveBeenCalled();
return;
}
if (retirement === "detach failure") {
detachControllerForRestart.mockRejectedValue(new Error("fixture journal drain failed"));
for (let attempt = 0; attempt < 2; attempt++) {
expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 0, busy: 0, failed: 1 });
expect(close).not.toHaveBeenCalled();
expect(readNativeSessionIdleWork()).toBe("present");
expect(JSON.parse(await readFile(path, "utf8")).releasedRunner).toEqual(envelope.releasedRunner);
}
detachControllerForRestart.mockResolvedValue(undefined);
}
if (retirement !== "idle timeout") {
expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 1, busy: 0, failed: 0 });
expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 0, busy: 0, failed: 0 });
} else await vi.waitFor(() => expect(detachControllerForRestart).toHaveBeenCalledOnce());
expect(readNativeSessionIdleWork()).toBe("none");
expect(close).not.toHaveBeenCalled();
expect(task.mock.calls.map(([operation]) => (operation as { kind: string }).kind)).toEqual(["describe"]);
expect(JSON.parse(await readFile(path, "utf8")).releasedRunner).toEqual(envelope.releasedRunner);
@@ -17088,6 +17114,7 @@ describe("runnerd provider runtime wiring", () => {
await expect(owners.mock.calls.at(-1)![0].onResumed!(source)).rejects.toThrow("released_source_mismatch");
expect(JSON.parse(await readFile(path, "utf8")).releasedRunner).toEqual(replacement.releasedRunner);
} finally {
detachControllerForRestart.mockResolvedValue(undefined);
allowRelease(); allowPhysicalStop(); await execute.catch(() => undefined); await closeIdleWarmNativeSessionsForRestart(); owners.mockRestore();
if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = previousHome;
}