fix(native): preserve parked Runners across controller restarts (#20586)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native execution parks a released sandbox Runner for a later turn.
> - The warm idle timer detaches Core and leaves Runner retention to the
host.
> - Controller shutdown instead stops recently parked Runners through
checkpoint cleanup.
> - This makes reuse depend on whether the idle timer fired before
shutdown.
> - This pull request uses the same detach operation for idle expiry and
shutdown.
> - The benefit is consistent Runner retention without a new retention
or recovery mechanism.

## Linked Issues or Issue Description

Fixes: #20012.
Related: #20005 unified cached and adopted process reuse. Closed #19975
included this shutdown change within a broader retention proposal.

## What Changed

- Detach confirmed released entries during controller shutdown. Keep
busy, unreleased, and close-marked entries on the existing cleanup path.
- Share the detach operation with idle expiry and cached-owner handoff.
- Record that a local handle has started detaching. Later cleanup cannot
fall back to stopping its Runner after a partial detach failure.
- Preserve explicit environment deletion, release proof, admission, and
process-reuse checks.
- Add Codex and Claude restart, detach-failure, and environment-deletion
regressions. Add real Runner adoption cases after the restart sweep,
including managed GitHub rebinding.
- Document the lifetime rule and make the shutdown warning cover both
detach and checkpoint failures.

## Verification

- Final source revision: `e4bb6ecbd7`. Local environment: macOS arm64,
Node 26.12.0.
- New controller-restart and detach-failure tests fail against unchanged
master: 4 expected failures. Old shutdown calls close or reports success
instead of reporting detach failure.
- Focused executor tests: 22 passed.
- Full native session executor suite: 1,328 passed.
- `cd server && node scripts/typecheck-layers.mjs`: passed all source
and test layers.
- `node scripts/check-module-boundaries.mjs` and `git diff --check`:
passed.
- Manual diff review and added-line credential/PII scan: passed.
- `cd server && node ../node_modules/vitest/vitest.mjs run
src/services/native-runtime/plugin-runner-release-resume.integration.test.ts
--bail=1 --reporter=verbose`: 14 passed on this revision. This includes
new Codex and Claude restart cases and managed GitHub rebinding.
- Failure history: an earlier concurrent run had 4 passes and 10
failures. Its first failure was the existing Claude cached case
exceeding 60 seconds. Later cases encountered a leftover prototype spy,
recursive calls, and startup timeouts. No source changed before the
passing full rerun. A clean worktree of unchanged master (`e01fe7f84a`)
passed all 11 original cases. The first timeout's exact cause remains
unknown; the later pass does not establish that cause.
- Standard Greptile review: 5/5 on
`e4bb6ecbd7bd965b5f56507d2c42c580e8a1d547`, with no actionable findings.
- `node scripts/local-verify.mjs --explain` returned exit 2 because the
execution-semantics document requires broad CI. Required CI will supply
broad typecheck, test, and build coverage.
- A hosted deployment and an actual controller process replacement have
not been run. The added integration cases exercise the shutdown sweep
and a new adopting transport against a real Runner.

## Risks

- SIGINT and SIGTERM preserve confirmed parked Runners, including
ordinary service stops. The host still owns eviction and retention.
- Detach failure retains a fenced local owner and reports failure. Core
does not infer permission to stop the Runner from that failure.
- Reuse remains conditional on current admission, process configuration,
and host availability.
- No persisted schema, checkpoint format, launch identity, or migration
changes. State from the previous release uses the existing adoption or
verified restore paths.

## Model Used

- OpenAI Codex, GPT-6, with reasoning, repository inspection, tool use,
and local test execution. The exact deployment identifier and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Nicky LeachandPaperclip authored and GitHub committed 2026-11-10 16:48:30 +00:00
1 parent 765ccd8691
commit 2e3cbe57a4
5 files changed
+65 -20

No files matched your search

@@ -693,6 +693,7 @@ type WarmNativeSession = {
environmentId: string | null;
busy: boolean;
released?: boolean;
controllerDetachedForRestart?: boolean;
finishRelease?: () => Promise<void>;
closeOnReleaseReason?: string;
idleTimer: ReturnType<typeof setTimeout> | null;
@@ -703,6 +704,7 @@ async function closeWarmNativeSession(entry: WarmNativeSession, reason: string,
preserveInstructionsForRunId?: string;
discardUnsettledSession?: boolean;
}) {
if (entry.controllerDetachedForRestart) return detachReleasedWarmNativeSession(entry);
// Revoke before awaiting process retirement/checkpoint IO.
const stopping = entry.githubAccess?.stop();
try {
@@ -728,6 +730,13 @@ async function closeWarmNativeSession(entry: WarmNativeSession, reason: string,
finally { await stopping; }
}
async function detachReleasedWarmNativeSession(entry: WarmNativeSession) {
entry.controllerDetachedForRestart = true;
if (!entry.session.detachControllerForRestart) throw new Error("native_session_released_detach_unavailable");
await entry.session.detachControllerForRestart();
await entry.githubAccess?.stop();
}
const warmNativeSessions = new Map<string, WarmNativeSession>();
function forgetWarmNativeSession(scope: string, entry: WarmNativeSession) {
if (warmNativeSessions.get(scope) !== entry) return;
@@ -895,7 +904,7 @@ export async function closeWarmNativeSessionsForEnvironment(input: {
return closeIdleWarmNativeSessions(input);
}
/** Suspend idle owners and persist their remote backup before a controller
/** Detach released owners and checkpoint unreleased owners before a controller
* exits. Active turns keep their separate authenticated restart handoff. */
export async function closeIdleWarmNativeSessionsForRestart(): Promise<{
closed: number; busy: number; failed: number;
@@ -903,6 +912,7 @@ export async function closeIdleWarmNativeSessionsForRestart(): Promise<{
return closeIdleWarmNativeSessions({
reason: "controller restart",
closeBusyOnRelease: true,
detachReleased: true,
});
}
@@ -916,6 +926,7 @@ async function closeIdleWarmNativeSessions(input: {
environmentId?: string;
reason: string;
closeBusyOnRelease?: boolean;
detachReleased?: boolean;
canRetire?: () => boolean;
}): Promise<{ closed: number; busy: number; failed: number }> {
let closed = 0;
@@ -937,13 +948,14 @@ async function closeIdleWarmNativeSessions(input: {
}
// The host owns retention after release; a quiet local handle needs no retirement for sleep.
if (input.canRetire && entry.released && !entry.closeOnReleaseReason) continue;
const detach = input.detachReleased === true && entry.released === true && !entry.closeOnReleaseReason;
if (entry.idleTimer !== null) clearTimeout(entry.idleTimer);
// Keep a fenced owner until both retirement and stopped collection succeed.
// Failure must not make a live directory eligible for a replacement owner.
entry.busy = true;
entry.closeOnReleaseReason = input.reason;
if (!detach) entry.closeOnReleaseReason = input.reason;
const closing = Promise.resolve().then(() =>
closeWarmNativeSession(entry, input.reason),
detach ? detachReleasedWarmNativeSession(entry) : closeWarmNativeSession(entry, input.reason),
);
closingWarmNativeSessions.set(sessionId, closing);
try {
@@ -951,6 +963,7 @@ async function closeIdleWarmNativeSessions(input: {
forgetWarmNativeSession(sessionId, entry);
closed += 1;
} catch {
if (detach) entry.closeOnReleaseReason = "released Runner local detach is unconfirmed";
failed += 1;
} finally {
entry.busy = false;
@@ -8020,11 +8033,7 @@ function armWarmNativeSessionIdleTimer(
current.busy = true;
const released = current.released === true;
if (!released) current.closeOnReleaseReason = "warm native session idle timeout";
const closing = (released ? (async () => {
if (!current.session.detachControllerForRestart) throw new Error("native_session_released_detach_unavailable");
await current.session.detachControllerForRestart();
await current.githubAccess?.stop();
})() : closeWarmNativeSession(current, current.closeOnReleaseReason!))
const closing = (released ? detachReleasedWarmNativeSession(current) : closeWarmNativeSession(current, current.closeOnReleaseReason!))
.then(() => { forgetWarmNativeSession(sessionId, current); })
.catch(() => {
if (released) current.closeOnReleaseReason = "released Runner local detach is unconfirmed";
@@ -9898,10 +9907,8 @@ async function executePaperclipNativeSessionAttempt(
const resumeFrom = await resolvePluginTaskResume(input.db, { companyId: input.execution.binding.companyId,
leaseId: target.leaseId, priorIdentity: proof.source, inspectCheckpoint: true });
if (resumeFrom !== proof.taskId) throw new Error("native_session_released_owner_scope_mismatch");
if (!cachedReleasedOwner.session.detachControllerForRestart) throw new Error("native_session_released_detach_unavailable");
try {
await cachedReleasedOwner.session.detachControllerForRestart();
await cachedReleasedOwner.githubAccess?.stop();
await detachReleasedWarmNativeSession(cachedReleasedOwner);
} catch (error) {
cachedReleasedOwner.closeOnReleaseReason = "released Runner local detach is unconfirmed";
throw error;