fix(native): let a same-run reattachment take over its own retained remote Runner cleanup (#21328)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) use a detached
`paperclip-runnerd`. After a recoverable failure, the server resumes the
same run and reattaches to that Runner
> - #20923 keeps an unreachable remote Runner alive when the failed
attempt closes, so that the same-run reattachment can adopt it
> - The same close also leaves the session in the in-process cleanup
quarantine. The quarantine refuses every new session of the task, and
that includes the reattachment that the Runner was kept for
> - Close retries cannot settle it. After the first close fails, the
Runner transport returns the same rejected close for every retry
> - On staging, this was the largest failure class: 14 Codex runs in one
hour failed with `native_session_cleanup_quarantined`, while only 5 runs
succeeded. Each task then needed Retry
> - This pull request lets the same run's reattachment to that Runner,
in the same remote sandbox, take over its own retained cleanup
> - The benefit is that the task continues on its own, with no spent
attempt, no failed run and no Retry

## Linked Issues or Issue Description

Refs #20923, #21031

**What happened?**
A remote native run (Codex on Daytona) failed a turn with a recoverable
error, for example `runnerd did not report the provider turn identity`
or `native_finalization_missing: session returned no semantic result`.
The close then logged `runner did not prove durable suspension before
checkpoint` and `remote runner left alive unsettled for same-run
reattachment`. Each later same-run reattachment was refused at
admission:
- When the close error was retryable, admission waited out the grace and
failed with `native_session_cleanup_quarantined: prior session cleanup
remains incomplete`. That spent a same-run attempt. After the attempt
budget, the run failed. All close retries replayed the same failed close
within milliseconds (`runner transport close requested: native session
quarantined cleanup recovery`), so they could never settle the
quarantine.
- When the close error was `native_session_close_unrecoverable`
(operator-required), the next attempt failed at once with
`native_session_cleanup_quarantined: the previous session is still
shutting down; Paperclip finishes the cleanup automatically and the
saved work is kept.` That code is a permanent failure.

"Paperclip finishes the cleanup automatically" did not happen in the
process. Only a server restart (a deploy) removed the in-memory
quarantine, and by then the run had failed and the task waited for
Retry. Staging restarts every 20 to 45 minutes, so attempts often end
mid-cleanup and meet this gate.

**Expected behavior**
The same run's reattachment adopts the Runner that its earlier attempt
kept for it. It does not wait for a close that cannot settle, and it
does not spend an attempt. The task continues without Retry. Other runs
of the task and other sandboxes stay gated.

**Steps to reproduce**
1. Start a remote native run, and let its turn fail with a recoverable
transport error while the Runner cannot prove suspension.
2. The close keeps the Runner alive and quarantines the session.
3. The server schedules the same-run reattachment 30 s later.
4. The reattachment fails with `native_session_cleanup_quarantined`, and
the task needs Retry.

**Paperclip version or commit**
Deployed Core `c730872974` on staging. Reproduced on `master` at
`49f0030f06`.

**Deployment mode**
Hosted cloud (staging), Daytona remote sandboxes.

**Agent adapter(s) involved**
`paperclip_runner` native runtime with the Codex app server backend.

## What Changed

- `packages/paperclip-runner/src/native-session-runtime.ts`:
`executeNativeSession` takes a new option `retainedRunnerReattachment`.
With a remote cleanup scope, admission retires quarantine entries that
belong to the same run in the same remote scope. Removal stops later
close retries. A close attempt that is still in flight stays an
admission owner until it settles, so the old and new controllers do not
overlap. Other runs, other sandboxes and local backends keep the gate.
- `server/src/services/native-runtime/native-session-executor.ts`: sets
`retainedRunnerReattachment` only for a `reattach_remote_runner`
recovery of the same run.
- `doc/execution-semantics.md`: records the takeover.
- Tests:
- `packages/paperclip-runner/src/native-session-runtime.test.ts`: four
new cases. An operator-required close with a retained Runner. A
retryable close whose retries replay one failure. A close that a deploy
interrupts while it is in flight (the reattachment waits for it, then
takes over). Controls: another run, a run without the takeover option
and a local backend stay refused.
- `server/src/services/native-runtime/native-session-executor.test.ts`:
asserts that the option is set for a same-run remote reattachment and is
not set for a fresh sandbox run.

## Verification

- Red on `master` (runtime source restored): 3 of the 4 new runtime
cases fail with exactly the two staging errors (`the previous session is
still shutting down…` and `prior session cleanup remains incomplete`).
The control case passes.
- Red on `master` (executor source restored): the 3 executor assertions
fail.
- Green with this change:
- `cd packages/paperclip-runner && npx vitest run
src/native-session-runtime.test.ts` (294 passed)
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "exact
Stop cleanup|fresh sandbox|retirement"` (46 passed)
- `cd server && npx vitest run
src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts
src/services/quarantined-task-retry.test.ts
src/__tests__/completed-turn-cleanup-hold-settle.test.ts` (61 passed)
- `npx tsc --noEmit` in `packages/paperclip-runner` and `server` passes.
- Staging evidence: runs `7b7ded99` and `27c487ef` show the sequence
above in their events and logs.

## Risks

- The takeover applies only when the server recovers the same run with
`reattach_remote_runner` in the same remote cleanup scope. This is the
case that #20923 keeps the Runner alive for. After a deploy the
quarantine is already gone and the same reattachment runs today, so the
takeover gives the in-process retry the same behavior as a restart.
- A close attempt that is in flight is never overlapped. The
reattachment waits for it within the existing admission grace.
- The reattachment still verifies Runner ownership and its controller
generation before it adopts the Runner. If the Runner is gone, the
existing relaunch or reconciliation path handles it.
- No persisted shape changes. Saved state from the previous release is
not affected because the quarantine lives only in process memory.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 03:15:40 -06:00
1 parent d1d71a7755
commit 0986c3eb3a
5 files changed
+196 -1

No files matched your search

+11
View File
@@ -1812,6 +1812,17 @@ by the deploy, and the task then lost its session to a fresh one. A Stop, any
identity or authority refusal, any other recovery failure, and a protocol
integrity fault still stop it.
The failed attempt's close keeps the session quarantined in the server process,
and later close retries only replay that failed close, so they never settle it.
The same run's reattachment to that Runner (in the same remote sandbox) takes
over the quarantine instead of waiting for it: it lets any close attempt still
in flight settle, then retires the attempt's cleanup and adopts the Runner.
Before this, the reattachment was refused with `native_session_cleanup_quarantined`
(`prior session cleanup remains incomplete` while retries were scheduled, or
`the previous session is still shutting down` once the close was
operator-required), each refusal spent a same-run attempt, and the task needed
Retry. Other runs of the task and other sandboxes stay gated as before.
A kept Runner can answer its WebSocket pings but no recovery command. On
staging, a reattached Codex Runner authenticated and restored its session, then
left `session.snapshot` unanswered (`pending ahead 0; last runner event seq 338
@@ -11132,3 +11132,137 @@ describe("remote Runner retention after a failed run", () => {
expect(retainedOn(close).every(retained => retained === false)).toBe(true);
});
});
describe("same-run reattachment to a Runner the failed attempt retained", () => {
const capabilities = { resume: true, typedEvents: true, steering: false, interruption: false, structuredResult: true };
const controlPlane = (): ControlPlanePort => ({
openRun: async () => {}, checkpointSession: async () => {},
appendEvent: async () => ({ cursor: 0, highestContiguousSourceSeq: 0, disposition: "committed" }),
replayEvents: async () => ({ events: [], highestContiguousSourceSeq: 0 }), completeRun: vi.fn(async () => {}),
});
const attemptSession = (runIdentity: NativeRunIdentity, failure: string, close: () => Promise<void>): NativeSession => ({
identity: () => runIdentity,
capabilities: async () => capabilities,
async *events() { throw new Error(failure); },
startTurn: async () => ({ turnId: "retained-turn" }),
result: async () => null,
async snapshot() { return { backendKind: "mock", sessionId: runIdentity.sessionId, identity: runIdentity,
providerSessionId: "retained-provider", cursor: null, activeTurnId: null, pendingRuntimeRequests: [], lineage: [] }; },
close: vi.fn(close),
});
const setup = (name: string) => {
const runIdentity = { ...identity, companyId: `${name}-company`, runId: `${name}-run` };
const scopedInput = { ...input, binding: { ...input.binding, companyId: runIdentity.companyId, runId: runIdentity.runId } };
const block = { companyId: runIdentity.companyId, issueId: scopedInput.binding.issueId };
const reattached = attemptSession(runIdentity, "reattached attempt reached the Runner", async () => {});
const options = (session: NativeSession, extra: Partial<ExecuteNativeSessionOptions> = {}): ExecuteNativeSessionOptions => ({
input: scopedInput,
backend: { descriptor: async () => ({ kind: "runner", name: `${name}-backend`, version: "1", capabilities }),
openSession: vi.fn(async () => session) },
controlPlane: controlPlane(), runnerInstanceId: `${name}-runner`, controlPlaneInstanceId: "control",
requireSessionCloseBeforeReturn: true, remoteCleanupScope: `${name}-sandbox`, ...extra,
});
return { runIdentity, scopedInput, block, reattached, options };
};
it("admits the run's own reattachment after a close that left the remote Runner alive and unsettled", async () => {
const { runIdentity, scopedInput, block, reattached, options } = setup("retained-unsettled");
const retained = new NativeSessionCloseUnrecoverableError({ runnerSuspended: false, runnerRetained: true });
const failed = attemptSession(runIdentity, "runnerd did not report the provider turn identity", async () => { throw retained; });
await expect(executeNativeSession(options(failed))).rejects.toMatchObject({
message: "runnerd did not report the provider turn identity", cleanupError: retained });
expect(nativeSessionCleanupBlock(block)).toBe("operator_required");
const otherRun = options(reattached, { retainedRunnerReattachment: true,
input: { ...scopedInput, binding: { ...scopedInput.binding, runId: "another-run" } } });
await expect(executeNativeSession(otherRun)).rejects.toBeInstanceOf(NativeSessionCleanupQuarantinedError);
const reattachment = options(reattached, { retainedRunnerReattachment: true });
await expect(executeNativeSession(reattachment)).rejects.toThrow("reattached attempt reached the Runner");
expect(reattachment.backend.openSession).toHaveBeenCalledOnce();
expect(failed.close).toHaveBeenCalledOnce();
expect(nativeSessionCleanupBlock(block)).toBeNull();
});
it("keeps refusing a run that is not reattaching to its own retained remote Runner", async () => {
const remote = setup("retained-without-takeover");
const retained = new NativeSessionCloseUnrecoverableError({ runnerSuspended: false, runnerRetained: true });
const failed = attemptSession(remote.runIdentity, "transport interrupted", async () => { throw retained; });
await expect(executeNativeSession(remote.options(failed))).rejects.toThrow("transport interrupted");
const fresh = remote.options(remote.reattached);
await expect(executeNativeSession(fresh)).rejects.toBeInstanceOf(NativeSessionCleanupQuarantinedError);
expect(fresh.backend.openSession).not.toHaveBeenCalled();
expect(nativeSessionCleanupBlock(remote.block)).toBe("operator_required");
completeTerminatedRemoteNativeSessionCleanup({ companyId: remote.runIdentity.companyId, runId: remote.runIdentity.runId,
remoteCleanupScope: "retained-without-takeover-sandbox" });
const local = setup("retained-local");
const localFailed = attemptSession(local.runIdentity, "local transport interrupted", async () => { throw retained; });
await expect(executeNativeSession(local.options(localFailed, { remoteCleanupScope: undefined })))
.rejects.toThrow("local transport interrupted");
const localReattachment = local.options(local.reattached, { remoteCleanupScope: undefined, retainedRunnerReattachment: true });
await expect(executeNativeSession(localReattachment)).rejects.toBeInstanceOf(NativeSessionCleanupQuarantinedError);
expect(localReattachment.backend.openSession).not.toHaveBeenCalled();
completeTerminatedLocalNativeSessionCleanup({ companyId: local.runIdentity.companyId, runId: local.runIdentity.runId,
runnerInstanceId: "retained-local-runner" });
});
it("admits the run's own reattachment instead of waiting out close retries that replay one failed close", async () => {
vi.useFakeTimers();
try {
const { runIdentity, block, reattached, options } = setup("retained-replayed-close");
const drainTimedOut = new Error("Controller journal drain timed out; preserve the session before replacing its writer.");
const failed = attemptSession(runIdentity, "runnerd did not report the provider turn identity", async () => { throw drainTimedOut; });
const first = expect(executeNativeSession(options(failed))).rejects.toThrow("runnerd did not report the provider turn identity");
await vi.advanceTimersByTimeAsync(10_000);
await first;
expect(nativeSessionCleanupBlock(block)).toBe("pending");
const closesBeforeReattachment = vi.mocked(failed.close).mock.calls.length;
const reattachment = options(reattached, { retainedRunnerReattachment: true });
const admitted = expect(executeNativeSession(reattachment)).rejects.toThrow("reattached attempt reached the Runner");
await vi.advanceTimersByTimeAsync(2_000);
await admitted;
expect(reattachment.backend.openSession).toHaveBeenCalledOnce();
expect(nativeSessionCleanupBlock(block)).toBeNull();
await vi.advanceTimersByTimeAsync(600_000);
expect(vi.mocked(failed.close).mock.calls.length).toBe(closesBeforeReattachment);
} finally {
vi.useRealTimers();
}
});
it("lets a close that a deploy interrupted settle before the reattachment takes the Runner over", async () => {
vi.useFakeTimers();
try {
const { runIdentity, block, reattached, options } = setup("retained-deploy-mid-cleanup");
const interrupted = new Error("runner transport closed by deploy");
let rejectClose!: (error: Error) => void;
const inFlightClose = new Promise<void>((_resolve, reject) => { rejectClose = reject; });
void inFlightClose.catch(() => undefined);
const failed = attemptSession(runIdentity, "server restarting for deploy", () => inFlightClose);
const first = executeNativeSession(options(failed, { requireSessionCloseBeforeReturn: false }));
void first.catch(() => undefined);
await vi.waitFor(() => expect(failed.close).toHaveBeenCalledOnce());
expect(nativeSessionCleanupBlock(block)).toBe("pending");
const reattachment = options(reattached, { retainedRunnerReattachment: true });
let settled = false;
const admitted = executeNativeSession(reattachment).catch((error: unknown) => { settled = true; return error; });
await vi.advanceTimersByTimeAsync(1_000);
expect(settled).toBe(false);
expect(reattachment.backend.openSession).not.toHaveBeenCalled();
rejectClose(interrupted);
await vi.advanceTimersByTimeAsync(10_000);
await expect(admitted).resolves.toMatchObject({ message: "reattached attempt reached the Runner" });
expect(reattachment.backend.openSession).toHaveBeenCalledOnce();
expect(nativeSessionCleanupBlock(block)).toBeNull();
const closesAfterTakeover = vi.mocked(failed.close).mock.calls.length;
await vi.advanceTimersByTimeAsync(600_000);
expect(vi.mocked(failed.close).mock.calls.length).toBe(closesAfterTakeover);
} finally {
vi.useRealTimers();
}
});
});
@@ -329,6 +329,11 @@ export interface ExecuteNativeSessionOptions {
/** Trusted provider resource identity: independent remote sandboxes must not
* inherit each other's process-cleanup gates. Omit for local backends. */
remoteCleanupScope?: string;
/** This execution is the same run reattaching to the remote Runner that its
* earlier attempt's close left alive for exactly this reattachment. It takes
* over that attempt's retained cleanup in `remoteCleanupScope` instead of
* waiting for a close that can no longer settle. Other runs stay gated. */
retainedRunnerReattachment?: boolean;
/** Operation bound; explicit values also preserve the legacy turn bound. */
timeoutMs?: number;
/** Total turn duration. Zero or no configured bound allows long-running work. */
@@ -1016,9 +1021,42 @@ function scheduleQuarantinedSessionCleanup(
cleanup.timer.unref?.();
}
function quarantinedCleanupRunId(cleanup: QuarantinedSessionCleanup): string | null {
try {
return cleanup.session.identity().runId;
} catch {
return null;
}
}
/** The same run's reattachment owns the remote Runner its earlier attempt
* retained, so it retires that attempt's cleanup in the same remote scope.
* Removal stops later close retries; an in-flight attempt stays an admission
* owner until it settles, so the old and new controllers never overlap. */
function takeOverRetainedRunnerCleanups(
cleanupDomain: NativeSessionCleanupDomain,
runId: string,
): void {
if (JSON.parse(cleanupDomain.key).length !== 4) return;
for (const cleanup of [...quarantinedSessionCleanups]) {
if (
cleanup.domain.key !== cleanupDomain.key ||
!nativeSessionCleanupScopeBlocks(cleanup.domain, cleanupDomain) ||
quarantinedCleanupRunId(cleanup) !== runId
)
continue;
if (cleanup.timer) clearTimeout(cleanup.timer);
cleanup.timer = null;
quarantinedSessionCleanups.delete(cleanup);
if (!cleanup.attempt && !cleanup.recovery)
announceCleanupSettled(cleanup.domain);
}
}
async function retryQuarantinedSessionCleanups(
cleanupDomain: NativeSessionCleanupDomain,
signal?: AbortSignal,
reattachingRunId?: string,
): Promise<void> {
// A close attempt becomes admission-visible as soon as the runtime retains
// its exact cleanup owner. It may not have rejected yet, so it may not have
@@ -1030,6 +1068,8 @@ async function retryQuarantinedSessionCleanups(
let observedOwnerPhases = 0;
while (true) {
signal?.throwIfAborted();
if (reattachingRunId)
takeOverRetainedRunnerCleanups(cleanupDomain, reattachingRunId);
if (
[...quarantinedSessionCleanups].some(
(cleanup) =>
@@ -2353,7 +2393,13 @@ export async function executeNativeSession(
]),
],
};
await retryQuarantinedSessionCleanups(cleanupDomain, options.signal);
await retryQuarantinedSessionCleanups(
cleanupDomain,
options.signal,
options.retainedRunnerReattachment && options.remoteCleanupScope
? input.binding.runId
: undefined,
);
if ("runtimeContext" in input) {
const capabilities = descriptor.runtimeContextCapabilities;
const unsupported = (["instructions", "skills", "mcp"] as const).filter(
@@ -12662,6 +12662,7 @@ describe("native warm session supervision", () => {
expect(state.stageNativeRunnerWakeAttachments).not.toHaveBeenCalled();
expect(cancel).toHaveBeenCalledTimes(invalidated ? 0 : 1);
expect(submit).not.toHaveBeenCalled();
expect(state.execute.mock.calls.at(-1)?.[0].retainedRunnerReattachment).toBe(true);
} finally {
state.stageNativeRunnerWakeAttachments.mockReset().mockImplementation(originalStaging);
close.mockImplementation(async () => undefined);
@@ -12965,6 +12966,7 @@ describe("native warm session supervision", () => {
expect(state.execute).toHaveBeenCalledTimes(2);
expect(state.execute.mock.calls.at(-1)?.[0].existingSession).toBeUndefined();
expect(close).toHaveBeenCalledTimes(closes);
expect(state.execute.mock.calls.at(-1)?.[0].retainedRunnerReattachment).toBe(false);
});
it.each(["companyId", "agentId", "issueId", "executionWorkspaceId"] as const)(
@@ -12353,6 +12353,8 @@ async function executePaperclipNativeSessionAttempt(
input: runnerExecution,
startup: executionContext && (!usesPluginTask || executionContext.pluginSupported) ? executionContext : undefined,
remoteCleanupScope: remoteCleanupLease ? remoteLeaseCleanupScope(remoteCleanupLease) : undefined,
retainedRunnerReattachment: input.restartRecovery?.kind === "reattach_remote_runner"
&& input.restartRecovery.runId === input.execution.binding.runId,
turnTimeoutMs: input.turnTimeoutMs,
backend:
input.backend ??