mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 11:54:35 +01:00
fix: retire failed warm sessions before sandbox stop (#14747)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runner sessions can stay warm between turns in a reusable sandbox. > - Heartbeat stops that sandbox when a turn fails or is cancelled. > - The native executor treated every returned terminal result as a successful warm release. > - A failed session could therefore retain a transport for a stopped sandbox. > - This pull request retires unsuccessful sessions before heartbeat stops their sandbox. > - A retry can start without inheriting that stale transport. Successful turns stay warm. ## Linked Issues or Issue Description **What happened?** A structured failed or cancelled terminal result did not throw. The host kept its native session in the warm cache even though heartbeat stopped the reusable sandbox. Later session retirement could use the stopped provider transport and reject the retry. **Expected behavior** Retire the unsuccessful session and collect its managed files before returning to heartbeat. Keep successful sessions warm. **Steps to reproduce** 1. Run a native session with a warm lifecycle and a reusable sandbox. 2. Return a structured failed or cancelled result from the provider. 3. Stop the sandbox after the executor returns. 4. Retry with a changed native session identity. 5. Verify that the previous session was closed before step 3 and is not closed again during retry. **Paperclip version or commit** Developed from `b54b2dc35`, the current `origin/master` at implementation time. **Deployment mode** Native runner with a reusable Daytona sandbox. The same warm-session release path also serves local providers. Related work: #14735 added incremental managed-file checkpoints for warm turns. #14734 preserves tool outcomes during shutdown. This change fixes the host's handling of unsuccessful terminal results. ## What Changed - Retain a warm session only when its terminal run state is `succeeded`. - Use the existing failed-session retirement path for structured failures and cancellations. - Preserve the existing checkpoint-before-retirement path, including when provider shutdown fails. Collect stopped files after successful shutdown, including edits made during shutdown. - Retire the warm owner if the checkpoint or its receipt callback rejects, then propagate the initiating error. - Add ten regression cases for failure and cancellation, with and without managed files. They check cleanup order, cache removal, retry with a new session identity, and edits preserved when close rejects. - Document the lifecycle rule. ## Verification - The failed and cancelled managed-file regressions fail on unpatched master because the provider is not closed. - All 516 native executor tests pass, including ten new regressions. - `pnpm -r typecheck` and `pnpm build` pass. `pnpm test:run` finished its general-server group with 14,537 passed, 75 skipped, and one existing failure in `native-session-resume.test.ts` (`retainedNativeCleanupJournalMatches`, line 1125). A separate run using the unmodified `origin/master` executor fails the same assertion. The command stops before later local groups; all equivalent CI groups pass on this head. - All 56 PR checks are successful or intentionally skipped. Greptile reviewed this head at 5/5 with no remaining findings. - A real Daytona public-API probe forced a Codex authentication failure, waited for a confirmed sandbox stop, corrected the credential, and retried successfully in the same resumed sandbox. It verified the agent note in canonical storage and deleted the test sandbox. - Pumpkin staging on this exact commit: forced a structured authentication failure, confirmed sandbox stop, corrected the credential, and retried successfully in that same sandbox. Three successful turns kept the same live runner PID/start ticks, native session, runner instance, and provider session. Canonical downloads verified the note, all 8 MiB binary bytes, and deletion persistence. Subsequent checkpoints hashed/copied only 52 and 78 bytes. After sandbox deletion, canonical files still matched. Temporary secrets, agent configuration, and test policy were cleaned up or restored. - Unpatched live probes with both 5-second and 90-second retry delays also succeeded. The unit regressions prove incorrect retention; the original stopped-lease exception was not reproduced in those probes. The staging recovery campaign used the public retry API after an explicit reset of the disposable task’s already-deleted old sandbox/session. The UI did not expose a Retry control on the inspected task or run detail views. - Temporary API-key authentication was used for the campaign. This change does not repair the original managed-subscription authentication 401. ## Risks - Failed and cancelled turns now close their provider sooner. Successful warm-turn behavior is unchanged. - Cleanup uses the existing failed-session path. Its existing handling of cleanup errors is unchanged. - No database migration, API contract change, or dependency change is included. ## Model Used OpenAI Codex, GPT-6, with reasoning, tool use, code execution, and test analysis. The exact served model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (516 relevant executor tests; the full local suite has the independently reproduced baseline failure documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
ad55d0a281
commit
5edf55d735
3 files changed
+161
-3
No files matched your search
@@ -1240,6 +1240,11 @@ that classification finishes.
|
||||
|
||||
A resumed sandbox lease can contain a workspace whose provider never started. A new attempt may create its exact session directory only when durable control-plane evidence proves zero connections, zero events, and untouched bootstrap commands, and no backup or remote session directory exists. Directory creation is atomic; partial state or uncertain ownership remains blocked.
|
||||
|
||||
Only successful turns retain a live warm provider session. A structured failed
|
||||
or cancelled result must retire that session and collect its managed files before
|
||||
heartbeat stops the reusable sandbox. A later retry can resume the same sandbox
|
||||
without inheriting the stopped provider transport.
|
||||
|
||||
Run the credential-free real-process restart suite with:
|
||||
|
||||
```sh
|
||||
|
||||
@@ -5919,6 +5919,145 @@ describe("native session same-turn steering", () => {
|
||||
});
|
||||
|
||||
describe("native warm session supervision", () => {
|
||||
it.each([
|
||||
{ runTerminalState: "failed", managedFiles: true },
|
||||
{ runTerminalState: "cancelled", managedFiles: true },
|
||||
{ runTerminalState: "failed", managedFiles: false },
|
||||
{ runTerminalState: "cancelled", managedFiles: false },
|
||||
] as const)("retires a structured $runTerminalState turn before its reusable sandbox stops (managed files: $managedFiles)", async ({ runTerminalState, managedFiles }) => {
|
||||
const name = `warm-terminal-${runTerminalState}-${managedFiles}`;
|
||||
const current = { ...execution,
|
||||
binding: { ...execution.binding, runId: `${name}-one`, executionWorkspaceId: name },
|
||||
session: { ...execution.session, normalizedSessionId: name, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 } },
|
||||
} as NativeExecutionInputV1;
|
||||
const target = { kind: "remote" as const, transport: "sandbox" as const, environmentId: name,
|
||||
remoteCwd: `/tmp/${name}`, sandboxLeaseAcquisition: { outcome: "created" as const, providerLeaseId: name } };
|
||||
let sandboxStopped = false;
|
||||
const close = vi.fn(async () => {
|
||||
if (sandboxStopped) throw new Error(`Daytona sandbox lease ${name} is no longer active.`);
|
||||
});
|
||||
const collectStopped = vi.fn(async () => undefined);
|
||||
const checkpointWarm = vi.fn(async () => true);
|
||||
const copy = { runId: current.binding.runId, root: `/tmp/${name}/home`, collectStopped, checkpointWarm, hasChanges: vi.fn() };
|
||||
const terminalResult = { result: { summary: "provider stopped" }, terminal: { runTerminalState },
|
||||
turnId: name, normalizedSessionId: name, providerSessionId: name, driverKind: "test", driverVersion: "1",
|
||||
nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null };
|
||||
state.execute.mockReset().mockImplementationOnce(async options => {
|
||||
await options.onSession?.({ close });
|
||||
return terminalResult;
|
||||
});
|
||||
try {
|
||||
const result = await executePaperclipNativeSession({ db: leaseDb(current), execution: current,
|
||||
runnerInstanceId: name, runnerExecutionTarget: target, ...(managedFiles ? { instructionWorkingCopy: copy } : {}) });
|
||||
expect(result.exitCode).toBe(1);
|
||||
// Heartbeat stops failed/cancelled sandboxes after the native executor
|
||||
// returns. Their sessions must already be retired and files collected.
|
||||
expect(close).toHaveBeenCalledOnce();
|
||||
expect(collectStopped).toHaveBeenCalledTimes(managedFiles ? 1 : 0);
|
||||
if (managedFiles) expect(close.mock.invocationCallOrder[0]).toBeLessThan(collectStopped.mock.invocationCallOrder[0]!);
|
||||
expect(checkpointWarm).toHaveBeenCalledTimes(managedFiles ? 1 : 0);
|
||||
if (managedFiles) expect(checkpointWarm.mock.invocationCallOrder[0]).toBeLessThan(close.mock.invocationCallOrder[0]!);
|
||||
sandboxStopped = true;
|
||||
expect(await reserveWarmNativeInstructionDirectory({ companyId: current.binding.companyId,
|
||||
agentId: current.binding.agentId, previousRunId: current.binding.runId, runId: `${name}-two`,
|
||||
target, canReuse: async () => true })).toBeNull();
|
||||
// A corrected credential rotates the native identity on the next run.
|
||||
// It must not try to close the stopped prior owner again at startup.
|
||||
const next = { ...current, binding: { ...current.binding, runId: `${name}-two` },
|
||||
session: { ...current.session, normalizedSessionId: `${name}-replacement` } };
|
||||
state.execute.mockImplementationOnce(async options => {
|
||||
expect(options.existingSession).toBeUndefined();
|
||||
return { ...terminalResult, terminal: { runTerminalState: "succeeded" } };
|
||||
});
|
||||
await expect(executePaperclipNativeSession({ db: leaseDb(next), execution: next,
|
||||
runnerInstanceId: `${name}-two`, runnerExecutionTarget: target })).resolves.toMatchObject({ exitCode: 0 });
|
||||
expect(close).toHaveBeenCalledOnce();
|
||||
} finally {
|
||||
sandboxStopped = false;
|
||||
await closeWarmNativeSessionsForEnvironment({ environmentId: name, reason: "test cleanup" });
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["failed", "cancelled"] as const)("checkpoints managed edits before retiring a %s session whose close rejects", async (runTerminalState) => {
|
||||
const name = `warm-terminal-close-fails-${runTerminalState}`;
|
||||
const current = { ...execution,
|
||||
binding: { ...execution.binding, runId: name, executionWorkspaceId: name },
|
||||
session: { ...execution.session, normalizedSessionId: name, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 } },
|
||||
} as NativeExecutionInputV1;
|
||||
let canonicalNote = "original";
|
||||
const checkpointWarm = vi.fn(async () => { canonicalNote = "edited during failed turn"; return true; });
|
||||
const close = vi.fn(async () => { throw new Error("provider shutdown failed"); });
|
||||
const collectStopped = vi.fn(async () => undefined);
|
||||
state.execute.mockReset().mockImplementationOnce(async options => {
|
||||
await options.onSession?.({ close });
|
||||
return { result: { summary: "provider failed" }, terminal: { runTerminalState },
|
||||
turnId: name, normalizedSessionId: name, providerSessionId: name, driverKind: "test", driverVersion: "1",
|
||||
nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null };
|
||||
});
|
||||
await expect(executePaperclipNativeSession({ db: leaseDb(current), execution: current, runnerInstanceId: name,
|
||||
instructionWorkingCopy: { runId: name, root: `/tmp/${name}`, checkpointWarm, collectStopped, hasChanges: vi.fn() },
|
||||
})).resolves.toMatchObject({ exitCode: 1 });
|
||||
expect(checkpointWarm).toHaveBeenCalledOnce();
|
||||
expect(close).toHaveBeenCalledOnce();
|
||||
expect(checkpointWarm.mock.invocationCallOrder[0]).toBeLessThan(close.mock.invocationCallOrder[0]!);
|
||||
expect(canonicalNote).toBe("edited during failed turn");
|
||||
// Failed close is not proof that the process stopped. The preceding warm
|
||||
// checkpoint preserves the edits without calling stopped-only collection.
|
||||
expect(collectStopped).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each(["failed", "cancelled"] as const)("retires a %s session even if its warm checkpoint rejects", async (runTerminalState) => {
|
||||
const name = `warm-checkpoint-rejects-${runTerminalState}`;
|
||||
const current = { ...execution,
|
||||
binding: { ...execution.binding, runId: name, executionWorkspaceId: name },
|
||||
session: { ...execution.session, normalizedSessionId: name, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 } },
|
||||
} as NativeExecutionInputV1;
|
||||
const close = vi.fn(async () => undefined);
|
||||
const collectStopped = vi.fn(async () => undefined);
|
||||
const failure = new Error("instruction-save receipt failed");
|
||||
state.execute.mockReset().mockImplementationOnce(async options => {
|
||||
await options.onSession?.({ close });
|
||||
return { result: { summary: "provider failed" }, terminal: { runTerminalState },
|
||||
turnId: name, normalizedSessionId: name, providerSessionId: name, driverKind: "test", driverVersion: "1",
|
||||
nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null };
|
||||
});
|
||||
await expect(executePaperclipNativeSession({ db: leaseDb(current), execution: current, runnerInstanceId: name,
|
||||
instructionWorkingCopy: { runId: name, root: `/tmp/${name}`, collectStopped, hasChanges: vi.fn(),
|
||||
checkpointWarm: async () => { throw failure; } },
|
||||
})).rejects.toBe(failure);
|
||||
expect(close).toHaveBeenCalledOnce();
|
||||
expect(collectStopped).toHaveBeenCalledOnce();
|
||||
expect(close.mock.invocationCallOrder[0]).toBeLessThan(collectStopped.mock.invocationCallOrder[0]!);
|
||||
expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 0, busy: 0, failed: 0 });
|
||||
});
|
||||
|
||||
it.each(["failed", "cancelled"] as const)("collects edits written during %s provider shutdown after the warm checkpoint", async (runTerminalState) => {
|
||||
const name = `warm-shutdown-edits-${runTerminalState}`;
|
||||
const current = { ...execution,
|
||||
binding: { ...execution.binding, runId: name, executionWorkspaceId: name },
|
||||
session: { ...execution.session, normalizedSessionId: name, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 } },
|
||||
} as NativeExecutionInputV1;
|
||||
let remoteNote = "before shutdown";
|
||||
let canonicalNote = "original";
|
||||
const checkpointWarm = vi.fn(async () => { canonicalNote = remoteNote; return true; });
|
||||
const close = vi.fn(async () => { remoteNote += "\nafter shutdown"; });
|
||||
const collectStopped = vi.fn(async () => { canonicalNote = remoteNote; });
|
||||
state.execute.mockReset().mockImplementationOnce(async options => {
|
||||
await options.onSession?.({ close });
|
||||
return { result: { summary: "provider failed" }, terminal: { runTerminalState },
|
||||
turnId: name, normalizedSessionId: name, providerSessionId: name, driverKind: "test", driverVersion: "1",
|
||||
nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null };
|
||||
});
|
||||
await expect(executePaperclipNativeSession({ db: leaseDb(current), execution: current, runnerInstanceId: name,
|
||||
instructionWorkingCopy: { runId: name, root: `/tmp/${name}`, checkpointWarm, collectStopped, hasChanges: vi.fn() },
|
||||
})).resolves.toMatchObject({ exitCode: 1 });
|
||||
expect(checkpointWarm).toHaveBeenCalledOnce();
|
||||
expect(collectStopped).toHaveBeenCalledOnce();
|
||||
expect(checkpointWarm.mock.invocationCallOrder[0]).toBeLessThan(close.mock.invocationCallOrder[0]!);
|
||||
expect(close.mock.invocationCallOrder[0]).toBeLessThan(collectStopped.mock.invocationCallOrder[0]!);
|
||||
expect(canonicalNote).toBe("before shutdown\nafter shutdown");
|
||||
});
|
||||
|
||||
describe("managed directory warm checkpoints", () => {
|
||||
const result = { result: { summary: "completed" }, terminal: { runTerminalState: "succeeded" },
|
||||
turnId: "turn", normalizedSessionId: "managed", providerSessionId: "provider", driverKind: "test", driverVersion: "1",
|
||||
|
||||
@@ -9158,10 +9158,24 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
// A following run cannot attach until the prior run's durable finalization
|
||||
// is committed. Provider completion alone is not an authority boundary.
|
||||
if (warmSessionId !== null && lifecyclePolicy.mode === "warm") {
|
||||
// A structured failed/cancelled result completes the protocol without
|
||||
// throwing. Heartbeat still stops its sandbox, so retire the provider now
|
||||
// while its transport can collect files and checkpoint the suspended runner.
|
||||
// Leaving it in the warm map makes the next turn retire a stopped transport.
|
||||
const retainSession = native.terminal.runTerminalState === "succeeded";
|
||||
const instructionCopy = input.instructionWorkingCopy;
|
||||
const ownedSession = warmNativeSessions.get(warmSessionId);
|
||||
const collectInstructions = Boolean(instructionCopy && ownedSession?.ownerToken === warmSessionOwnerToken &&
|
||||
(instructionCopy.checkpointWarm ? !await instructionCopy.checkpointWarm() : await instructionCopy.hasChanges()));
|
||||
let collectInstructions: boolean;
|
||||
try {
|
||||
collectInstructions = Boolean(instructionCopy && ownedSession?.ownerToken === warmSessionOwnerToken &&
|
||||
(instructionCopy.checkpointWarm ? !await instructionCopy.checkpointWarm() : await instructionCopy.hasChanges()));
|
||||
} catch (error) {
|
||||
// A checkpoint or its receipt callback can reject. Retire this owner
|
||||
// before heartbeat stops the sandbox, preserving the initiating error.
|
||||
await releaseWarmNativeSession(warmSessionId, warmSessionOwnerToken, lifecyclePolicy.idleTimeoutMs, true)
|
||||
.catch(() => undefined);
|
||||
throw error;
|
||||
}
|
||||
const collectedByOwner = ownedSession?.instructionCopy?.collectStopped === instructionCopy?.collectStopped;
|
||||
if (collectInstructions && ownedSession) {
|
||||
// Keep the unchanged warm path intact. A changed private instruction copy
|
||||
@@ -9172,7 +9186,7 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
warmSessionId,
|
||||
warmSessionOwnerToken,
|
||||
lifecyclePolicy.idleTimeoutMs,
|
||||
false,
|
||||
!retainSession,
|
||||
);
|
||||
if (collectInstructions && !collectedByOwner) await instructionCopy!.collectStopped();
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user