fix: retire failed warm sessions before sandbox stop (#14747)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runner sessions can stay warm between turns in a reusable
sandbox.
> - Heartbeat stops that sandbox when a turn fails or is cancelled.
> - The native executor treated every returned terminal result as a
successful warm release.
> - A failed session could therefore retain a transport for a stopped
sandbox.
> - This pull request retires unsuccessful sessions before heartbeat
stops their sandbox.
> - A retry can start without inheriting that stale transport.
Successful turns stay warm.

## Linked Issues or Issue Description

**What happened?**

A structured failed or cancelled terminal result did not throw. The host
kept its native session in the warm cache even though heartbeat stopped
the reusable sandbox. Later session retirement could use the stopped
provider transport and reject the retry.

**Expected behavior**

Retire the unsuccessful session and collect its managed files before
returning to heartbeat. Keep successful sessions warm.

**Steps to reproduce**

1. Run a native session with a warm lifecycle and a reusable sandbox.
2. Return a structured failed or cancelled result from the provider.
3. Stop the sandbox after the executor returns.
4. Retry with a changed native session identity.
5. Verify that the previous session was closed before step 3 and is not
closed again during retry.

**Paperclip version or commit**

Developed from `b54b2dc35`, the current `origin/master` at
implementation time.

**Deployment mode**

Native runner with a reusable Daytona sandbox. The same warm-session
release path also serves local providers.

Related work: #14735 added incremental managed-file checkpoints for warm
turns. #14734 preserves tool outcomes during shutdown. This change fixes
the host's handling of unsuccessful terminal results.

## What Changed

- Retain a warm session only when its terminal run state is `succeeded`.
- Use the existing failed-session retirement path for structured
failures and cancellations.
- Preserve the existing checkpoint-before-retirement path, including
when provider shutdown fails. Collect stopped files after successful
shutdown, including edits made during shutdown.
- Retire the warm owner if the checkpoint or its receipt callback
rejects, then propagate the initiating error.
- Add ten regression cases for failure and cancellation, with and
without managed files. They check cleanup order, cache removal, retry
with a new session identity, and edits preserved when close rejects.
- Document the lifecycle rule.

## Verification

- The failed and cancelled managed-file regressions fail on unpatched
master because the provider is not closed.
- All 516 native executor tests pass, including ten new regressions.
- `pnpm -r typecheck` and `pnpm build` pass. `pnpm test:run` finished
its general-server group with 14,537 passed, 75 skipped, and one
existing failure in `native-session-resume.test.ts`
(`retainedNativeCleanupJournalMatches`, line 1125). A separate run using
the unmodified `origin/master` executor fails the same assertion. The
command stops before later local groups; all equivalent CI groups pass
on this head.
- All 56 PR checks are successful or intentionally skipped. Greptile
reviewed this head at 5/5 with no remaining findings.
- A real Daytona public-API probe forced a Codex authentication failure,
waited for a confirmed sandbox stop, corrected the credential, and
retried successfully in the same resumed sandbox. It verified the agent
note in canonical storage and deleted the test sandbox.
- Pumpkin staging on this exact commit: forced a structured
authentication failure, confirmed sandbox stop, corrected the
credential, and retried successfully in that same sandbox. Three
successful turns kept the same live runner PID/start ticks, native
session, runner instance, and provider session. Canonical downloads
verified the note, all 8 MiB binary bytes, and deletion persistence.
Subsequent checkpoints hashed/copied only 52 and 78 bytes. After sandbox
deletion, canonical files still matched. Temporary secrets, agent
configuration, and test policy were cleaned up or restored.
- Unpatched live probes with both 5-second and 90-second retry delays
also succeeded. The unit regressions prove incorrect retention; the
original stopped-lease exception was not reproduced in those probes. The
staging recovery campaign used the public retry API after an explicit
reset of the disposable task’s already-deleted old sandbox/session. The
UI did not expose a Retry control on the inspected task or run detail
views.
- Temporary API-key authentication was used for the campaign. This
change does not repair the original managed-subscription authentication
401.

## Risks

- Failed and cancelled turns now close their provider sooner. Successful
warm-turn behavior is unchanged.
- Cleanup uses the existing failed-session path. Its existing handling
of cleanup errors is unchanged.
- No database migration, API contract change, or dependency change is
included.

## Model Used

OpenAI Codex, GPT-6, with reasoning, tool use, code execution, and test
analysis. The exact served model ID and context-window size are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (516 relevant executor
tests; the full local suite has the independently reproduced baseline
failure documented above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-09-30 15:36:06 -05:00
1 parent ad55d0a281
commit 5edf55d735
3 files changed
+161 -3

No files matched your search

+5
View File
@@ -1240,6 +1240,11 @@ that classification finishes.
A resumed sandbox lease can contain a workspace whose provider never started. A new attempt may create its exact session directory only when durable control-plane evidence proves zero connections, zero events, and untouched bootstrap commands, and no backup or remote session directory exists. Directory creation is atomic; partial state or uncertain ownership remains blocked.
Only successful turns retain a live warm provider session. A structured failed
or cancelled result must retire that session and collect its managed files before
heartbeat stops the reusable sandbox. A later retry can resume the same sandbox
without inheriting the stopped provider transport.
Run the credential-free real-process restart suite with:
```sh
@@ -5919,6 +5919,145 @@ describe("native session same-turn steering", () => {
});
describe("native warm session supervision", () => {
it.each([
{ runTerminalState: "failed", managedFiles: true },
{ runTerminalState: "cancelled", managedFiles: true },
{ runTerminalState: "failed", managedFiles: false },
{ runTerminalState: "cancelled", managedFiles: false },
] as const)("retires a structured $runTerminalState turn before its reusable sandbox stops (managed files: $managedFiles)", async ({ runTerminalState, managedFiles }) => {
const name = `warm-terminal-${runTerminalState}-${managedFiles}`;
const current = { ...execution,
binding: { ...execution.binding, runId: `${name}-one`, executionWorkspaceId: name },
session: { ...execution.session, normalizedSessionId: name, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 } },
} as NativeExecutionInputV1;
const target = { kind: "remote" as const, transport: "sandbox" as const, environmentId: name,
remoteCwd: `/tmp/${name}`, sandboxLeaseAcquisition: { outcome: "created" as const, providerLeaseId: name } };
let sandboxStopped = false;
const close = vi.fn(async () => {
if (sandboxStopped) throw new Error(`Daytona sandbox lease ${name} is no longer active.`);
});
const collectStopped = vi.fn(async () => undefined);
const checkpointWarm = vi.fn(async () => true);
const copy = { runId: current.binding.runId, root: `/tmp/${name}/home`, collectStopped, checkpointWarm, hasChanges: vi.fn() };
const terminalResult = { result: { summary: "provider stopped" }, terminal: { runTerminalState },
turnId: name, normalizedSessionId: name, providerSessionId: name, driverKind: "test", driverVersion: "1",
nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null };
state.execute.mockReset().mockImplementationOnce(async options => {
await options.onSession?.({ close });
return terminalResult;
});
try {
const result = await executePaperclipNativeSession({ db: leaseDb(current), execution: current,
runnerInstanceId: name, runnerExecutionTarget: target, ...(managedFiles ? { instructionWorkingCopy: copy } : {}) });
expect(result.exitCode).toBe(1);
// Heartbeat stops failed/cancelled sandboxes after the native executor
// returns. Their sessions must already be retired and files collected.
expect(close).toHaveBeenCalledOnce();
expect(collectStopped).toHaveBeenCalledTimes(managedFiles ? 1 : 0);
if (managedFiles) expect(close.mock.invocationCallOrder[0]).toBeLessThan(collectStopped.mock.invocationCallOrder[0]!);
expect(checkpointWarm).toHaveBeenCalledTimes(managedFiles ? 1 : 0);
if (managedFiles) expect(checkpointWarm.mock.invocationCallOrder[0]).toBeLessThan(close.mock.invocationCallOrder[0]!);
sandboxStopped = true;
expect(await reserveWarmNativeInstructionDirectory({ companyId: current.binding.companyId,
agentId: current.binding.agentId, previousRunId: current.binding.runId, runId: `${name}-two`,
target, canReuse: async () => true })).toBeNull();
// A corrected credential rotates the native identity on the next run.
// It must not try to close the stopped prior owner again at startup.
const next = { ...current, binding: { ...current.binding, runId: `${name}-two` },
session: { ...current.session, normalizedSessionId: `${name}-replacement` } };
state.execute.mockImplementationOnce(async options => {
expect(options.existingSession).toBeUndefined();
return { ...terminalResult, terminal: { runTerminalState: "succeeded" } };
});
await expect(executePaperclipNativeSession({ db: leaseDb(next), execution: next,
runnerInstanceId: `${name}-two`, runnerExecutionTarget: target })).resolves.toMatchObject({ exitCode: 0 });
expect(close).toHaveBeenCalledOnce();
} finally {
sandboxStopped = false;
await closeWarmNativeSessionsForEnvironment({ environmentId: name, reason: "test cleanup" });
}
});
it.each(["failed", "cancelled"] as const)("checkpoints managed edits before retiring a %s session whose close rejects", async (runTerminalState) => {
const name = `warm-terminal-close-fails-${runTerminalState}`;
const current = { ...execution,
binding: { ...execution.binding, runId: name, executionWorkspaceId: name },
session: { ...execution.session, normalizedSessionId: name, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 } },
} as NativeExecutionInputV1;
let canonicalNote = "original";
const checkpointWarm = vi.fn(async () => { canonicalNote = "edited during failed turn"; return true; });
const close = vi.fn(async () => { throw new Error("provider shutdown failed"); });
const collectStopped = vi.fn(async () => undefined);
state.execute.mockReset().mockImplementationOnce(async options => {
await options.onSession?.({ close });
return { result: { summary: "provider failed" }, terminal: { runTerminalState },
turnId: name, normalizedSessionId: name, providerSessionId: name, driverKind: "test", driverVersion: "1",
nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null };
});
await expect(executePaperclipNativeSession({ db: leaseDb(current), execution: current, runnerInstanceId: name,
instructionWorkingCopy: { runId: name, root: `/tmp/${name}`, checkpointWarm, collectStopped, hasChanges: vi.fn() },
})).resolves.toMatchObject({ exitCode: 1 });
expect(checkpointWarm).toHaveBeenCalledOnce();
expect(close).toHaveBeenCalledOnce();
expect(checkpointWarm.mock.invocationCallOrder[0]).toBeLessThan(close.mock.invocationCallOrder[0]!);
expect(canonicalNote).toBe("edited during failed turn");
// Failed close is not proof that the process stopped. The preceding warm
// checkpoint preserves the edits without calling stopped-only collection.
expect(collectStopped).not.toHaveBeenCalled();
});
it.each(["failed", "cancelled"] as const)("retires a %s session even if its warm checkpoint rejects", async (runTerminalState) => {
const name = `warm-checkpoint-rejects-${runTerminalState}`;
const current = { ...execution,
binding: { ...execution.binding, runId: name, executionWorkspaceId: name },
session: { ...execution.session, normalizedSessionId: name, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 } },
} as NativeExecutionInputV1;
const close = vi.fn(async () => undefined);
const collectStopped = vi.fn(async () => undefined);
const failure = new Error("instruction-save receipt failed");
state.execute.mockReset().mockImplementationOnce(async options => {
await options.onSession?.({ close });
return { result: { summary: "provider failed" }, terminal: { runTerminalState },
turnId: name, normalizedSessionId: name, providerSessionId: name, driverKind: "test", driverVersion: "1",
nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null };
});
await expect(executePaperclipNativeSession({ db: leaseDb(current), execution: current, runnerInstanceId: name,
instructionWorkingCopy: { runId: name, root: `/tmp/${name}`, collectStopped, hasChanges: vi.fn(),
checkpointWarm: async () => { throw failure; } },
})).rejects.toBe(failure);
expect(close).toHaveBeenCalledOnce();
expect(collectStopped).toHaveBeenCalledOnce();
expect(close.mock.invocationCallOrder[0]).toBeLessThan(collectStopped.mock.invocationCallOrder[0]!);
expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 0, busy: 0, failed: 0 });
});
it.each(["failed", "cancelled"] as const)("collects edits written during %s provider shutdown after the warm checkpoint", async (runTerminalState) => {
const name = `warm-shutdown-edits-${runTerminalState}`;
const current = { ...execution,
binding: { ...execution.binding, runId: name, executionWorkspaceId: name },
session: { ...execution.session, normalizedSessionId: name, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 } },
} as NativeExecutionInputV1;
let remoteNote = "before shutdown";
let canonicalNote = "original";
const checkpointWarm = vi.fn(async () => { canonicalNote = remoteNote; return true; });
const close = vi.fn(async () => { remoteNote += "\nafter shutdown"; });
const collectStopped = vi.fn(async () => { canonicalNote = remoteNote; });
state.execute.mockReset().mockImplementationOnce(async options => {
await options.onSession?.({ close });
return { result: { summary: "provider failed" }, terminal: { runTerminalState },
turnId: name, normalizedSessionId: name, providerSessionId: name, driverKind: "test", driverVersion: "1",
nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null };
});
await expect(executePaperclipNativeSession({ db: leaseDb(current), execution: current, runnerInstanceId: name,
instructionWorkingCopy: { runId: name, root: `/tmp/${name}`, checkpointWarm, collectStopped, hasChanges: vi.fn() },
})).resolves.toMatchObject({ exitCode: 1 });
expect(checkpointWarm).toHaveBeenCalledOnce();
expect(collectStopped).toHaveBeenCalledOnce();
expect(checkpointWarm.mock.invocationCallOrder[0]).toBeLessThan(close.mock.invocationCallOrder[0]!);
expect(close.mock.invocationCallOrder[0]).toBeLessThan(collectStopped.mock.invocationCallOrder[0]!);
expect(canonicalNote).toBe("before shutdown\nafter shutdown");
});
describe("managed directory warm checkpoints", () => {
const result = { result: { summary: "completed" }, terminal: { runTerminalState: "succeeded" },
turnId: "turn", normalizedSessionId: "managed", providerSessionId: "provider", driverKind: "test", driverVersion: "1",
@@ -9158,10 +9158,24 @@ async function executePaperclipNativeSessionWithinScope(
// A following run cannot attach until the prior run's durable finalization
// is committed. Provider completion alone is not an authority boundary.
if (warmSessionId !== null && lifecyclePolicy.mode === "warm") {
// A structured failed/cancelled result completes the protocol without
// throwing. Heartbeat still stops its sandbox, so retire the provider now
// while its transport can collect files and checkpoint the suspended runner.
// Leaving it in the warm map makes the next turn retire a stopped transport.
const retainSession = native.terminal.runTerminalState === "succeeded";
const instructionCopy = input.instructionWorkingCopy;
const ownedSession = warmNativeSessions.get(warmSessionId);
const collectInstructions = Boolean(instructionCopy && ownedSession?.ownerToken === warmSessionOwnerToken &&
(instructionCopy.checkpointWarm ? !await instructionCopy.checkpointWarm() : await instructionCopy.hasChanges()));
let collectInstructions: boolean;
try {
collectInstructions = Boolean(instructionCopy && ownedSession?.ownerToken === warmSessionOwnerToken &&
(instructionCopy.checkpointWarm ? !await instructionCopy.checkpointWarm() : await instructionCopy.hasChanges()));
} catch (error) {
// A checkpoint or its receipt callback can reject. Retire this owner
// before heartbeat stops the sandbox, preserving the initiating error.
await releaseWarmNativeSession(warmSessionId, warmSessionOwnerToken, lifecyclePolicy.idleTimeoutMs, true)
.catch(() => undefined);
throw error;
}
const collectedByOwner = ownedSession?.instructionCopy?.collectStopped === instructionCopy?.collectStopped;
if (collectInstructions && ownedSession) {
// Keep the unchanged warm path intact. A changed private instruction copy
@@ -9172,7 +9186,7 @@ async function executePaperclipNativeSessionWithinScope(
warmSessionId,
warmSessionOwnerToken,
lifecyclePolicy.idleTimeoutMs,
false,
!retainSession,
);
if (collectInstructions && !collectedByOwner) await instructionCopy!.collectStopped();
}