mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-14 08:53:19 +01:00
fix(native): preserve parked Runners across controller restarts (#20586)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native execution parks a released sandbox Runner for a later turn. > - The warm idle timer detaches Core and leaves Runner retention to the host. > - Controller shutdown instead stops recently parked Runners through checkpoint cleanup. > - This makes reuse depend on whether the idle timer fired before shutdown. > - This pull request uses the same detach operation for idle expiry and shutdown. > - The benefit is consistent Runner retention without a new retention or recovery mechanism. ## Linked Issues or Issue Description Fixes: #20012. Related: #20005 unified cached and adopted process reuse. Closed #19975 included this shutdown change within a broader retention proposal. ## What Changed - Detach confirmed released entries during controller shutdown. Keep busy, unreleased, and close-marked entries on the existing cleanup path. - Share the detach operation with idle expiry and cached-owner handoff. - Record that a local handle has started detaching. Later cleanup cannot fall back to stopping its Runner after a partial detach failure. - Preserve explicit environment deletion, release proof, admission, and process-reuse checks. - Add Codex and Claude restart, detach-failure, and environment-deletion regressions. Add real Runner adoption cases after the restart sweep, including managed GitHub rebinding. - Document the lifetime rule and make the shutdown warning cover both detach and checkpoint failures. ## Verification - Final source revision: `e4bb6ecbd7`. Local environment: macOS arm64, Node 26.12.0. - New controller-restart and detach-failure tests fail against unchanged master: 4 expected failures. Old shutdown calls close or reports success instead of reporting detach failure. - Focused executor tests: 22 passed. - Full native session executor suite: 1,328 passed. - `cd server && node scripts/typecheck-layers.mjs`: passed all source and test layers. - `node scripts/check-module-boundaries.mjs` and `git diff --check`: passed. - Manual diff review and added-line credential/PII scan: passed. - `cd server && node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/plugin-runner-release-resume.integration.test.ts --bail=1 --reporter=verbose`: 14 passed on this revision. This includes new Codex and Claude restart cases and managed GitHub rebinding. - Failure history: an earlier concurrent run had 4 passes and 10 failures. Its first failure was the existing Claude cached case exceeding 60 seconds. Later cases encountered a leftover prototype spy, recursive calls, and startup timeouts. No source changed before the passing full rerun. A clean worktree of unchanged master (`e01fe7f84a`) passed all 11 original cases. The first timeout's exact cause remains unknown; the later pass does not establish that cause. - Standard Greptile review: 5/5 on `e4bb6ecbd7bd965b5f56507d2c42c580e8a1d547`, with no actionable findings. - `node scripts/local-verify.mjs --explain` returned exit 2 because the execution-semantics document requires broad CI. Required CI will supply broad typecheck, test, and build coverage. - A hosted deployment and an actual controller process replacement have not been run. The added integration cases exercise the shutdown sweep and a new adopting transport against a real Runner. ## Risks - SIGINT and SIGTERM preserve confirmed parked Runners, including ordinary service stops. The host still owns eviction and retention. - Detach failure retains a fenced local owner and reports failure. Core does not infer permission to stop the Runner from that failure. - Reuse remains conditional on current admission, process configuration, and host availability. - No persisted schema, checkpoint format, launch identity, or migration changes. State from the previous release uses the existing adoption or verified restore paths. ## Model Used - OpenAI Codex, GPT-6, with reasoning, repository inspection, tool use, and local test execution. The exact deployment identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
765ccd8691
commit
2e3cbe57a4
5 files changed
+65
-20
No files matched your search
@@ -693,6 +693,7 @@ type WarmNativeSession = {
|
||||
environmentId: string | null;
|
||||
busy: boolean;
|
||||
released?: boolean;
|
||||
controllerDetachedForRestart?: boolean;
|
||||
finishRelease?: () => Promise<void>;
|
||||
closeOnReleaseReason?: string;
|
||||
idleTimer: ReturnType<typeof setTimeout> | null;
|
||||
@@ -703,6 +704,7 @@ async function closeWarmNativeSession(entry: WarmNativeSession, reason: string,
|
||||
preserveInstructionsForRunId?: string;
|
||||
discardUnsettledSession?: boolean;
|
||||
}) {
|
||||
if (entry.controllerDetachedForRestart) return detachReleasedWarmNativeSession(entry);
|
||||
// Revoke before awaiting process retirement/checkpoint IO.
|
||||
const stopping = entry.githubAccess?.stop();
|
||||
try {
|
||||
@@ -728,6 +730,13 @@ async function closeWarmNativeSession(entry: WarmNativeSession, reason: string,
|
||||
finally { await stopping; }
|
||||
}
|
||||
|
||||
async function detachReleasedWarmNativeSession(entry: WarmNativeSession) {
|
||||
entry.controllerDetachedForRestart = true;
|
||||
if (!entry.session.detachControllerForRestart) throw new Error("native_session_released_detach_unavailable");
|
||||
await entry.session.detachControllerForRestart();
|
||||
await entry.githubAccess?.stop();
|
||||
}
|
||||
|
||||
const warmNativeSessions = new Map<string, WarmNativeSession>();
|
||||
function forgetWarmNativeSession(scope: string, entry: WarmNativeSession) {
|
||||
if (warmNativeSessions.get(scope) !== entry) return;
|
||||
@@ -895,7 +904,7 @@ export async function closeWarmNativeSessionsForEnvironment(input: {
|
||||
return closeIdleWarmNativeSessions(input);
|
||||
}
|
||||
|
||||
/** Suspend idle owners and persist their remote backup before a controller
|
||||
/** Detach released owners and checkpoint unreleased owners before a controller
|
||||
* exits. Active turns keep their separate authenticated restart handoff. */
|
||||
export async function closeIdleWarmNativeSessionsForRestart(): Promise<{
|
||||
closed: number; busy: number; failed: number;
|
||||
@@ -903,6 +912,7 @@ export async function closeIdleWarmNativeSessionsForRestart(): Promise<{
|
||||
return closeIdleWarmNativeSessions({
|
||||
reason: "controller restart",
|
||||
closeBusyOnRelease: true,
|
||||
detachReleased: true,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -916,6 +926,7 @@ async function closeIdleWarmNativeSessions(input: {
|
||||
environmentId?: string;
|
||||
reason: string;
|
||||
closeBusyOnRelease?: boolean;
|
||||
detachReleased?: boolean;
|
||||
canRetire?: () => boolean;
|
||||
}): Promise<{ closed: number; busy: number; failed: number }> {
|
||||
let closed = 0;
|
||||
@@ -937,13 +948,14 @@ async function closeIdleWarmNativeSessions(input: {
|
||||
}
|
||||
// The host owns retention after release; a quiet local handle needs no retirement for sleep.
|
||||
if (input.canRetire && entry.released && !entry.closeOnReleaseReason) continue;
|
||||
const detach = input.detachReleased === true && entry.released === true && !entry.closeOnReleaseReason;
|
||||
if (entry.idleTimer !== null) clearTimeout(entry.idleTimer);
|
||||
// Keep a fenced owner until both retirement and stopped collection succeed.
|
||||
// Failure must not make a live directory eligible for a replacement owner.
|
||||
entry.busy = true;
|
||||
entry.closeOnReleaseReason = input.reason;
|
||||
if (!detach) entry.closeOnReleaseReason = input.reason;
|
||||
const closing = Promise.resolve().then(() =>
|
||||
closeWarmNativeSession(entry, input.reason),
|
||||
detach ? detachReleasedWarmNativeSession(entry) : closeWarmNativeSession(entry, input.reason),
|
||||
);
|
||||
closingWarmNativeSessions.set(sessionId, closing);
|
||||
try {
|
||||
@@ -951,6 +963,7 @@ async function closeIdleWarmNativeSessions(input: {
|
||||
forgetWarmNativeSession(sessionId, entry);
|
||||
closed += 1;
|
||||
} catch {
|
||||
if (detach) entry.closeOnReleaseReason = "released Runner local detach is unconfirmed";
|
||||
failed += 1;
|
||||
} finally {
|
||||
entry.busy = false;
|
||||
@@ -8020,11 +8033,7 @@ function armWarmNativeSessionIdleTimer(
|
||||
current.busy = true;
|
||||
const released = current.released === true;
|
||||
if (!released) current.closeOnReleaseReason = "warm native session idle timeout";
|
||||
const closing = (released ? (async () => {
|
||||
if (!current.session.detachControllerForRestart) throw new Error("native_session_released_detach_unavailable");
|
||||
await current.session.detachControllerForRestart();
|
||||
await current.githubAccess?.stop();
|
||||
})() : closeWarmNativeSession(current, current.closeOnReleaseReason!))
|
||||
const closing = (released ? detachReleasedWarmNativeSession(current) : closeWarmNativeSession(current, current.closeOnReleaseReason!))
|
||||
.then(() => { forgetWarmNativeSession(sessionId, current); })
|
||||
.catch(() => {
|
||||
if (released) current.closeOnReleaseReason = "released Runner local detach is unconfirmed";
|
||||
@@ -9898,10 +9907,8 @@ async function executePaperclipNativeSessionAttempt(
|
||||
const resumeFrom = await resolvePluginTaskResume(input.db, { companyId: input.execution.binding.companyId,
|
||||
leaseId: target.leaseId, priorIdentity: proof.source, inspectCheckpoint: true });
|
||||
if (resumeFrom !== proof.taskId) throw new Error("native_session_released_owner_scope_mismatch");
|
||||
if (!cachedReleasedOwner.session.detachControllerForRestart) throw new Error("native_session_released_detach_unavailable");
|
||||
try {
|
||||
await cachedReleasedOwner.session.detachControllerForRestart();
|
||||
await cachedReleasedOwner.githubAccess?.stop();
|
||||
await detachReleasedWarmNativeSession(cachedReleasedOwner);
|
||||
} catch (error) {
|
||||
cachedReleasedOwner.closeOnReleaseReason = "released Runner local detach is unconfirmed";
|
||||
throw error;
|
||||
|
||||
Reference in new issue
Block a user