Start follow-up runs normally when a warm sandbox's window ends during its idle close (#21504)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm sandbox keeps an agent's Runner alive between runs, so the
next run on the task starts fast
> - When a run ends, the server closes the idle warm Runner in-process.
A separate SQL worker stops the sandbox when its warm window ends
(#21410, merged today)
> - Both steps start at the same moment. The sandbox stop wins, so the
idle close fails. The failed owner stays registered, and the next run
throws `native_instruction_materialization_handoff_unavailable` before
any provider work
> - Recovery must just work: a follow-up run must never fail because a
predecessor's warm copy is gone
> - This pull request lets the idle close finish before the sandbox
stops, prepares instructions fresh when the warm copy cannot be handed
off, and releases an owner whose sandbox the host already gave up
> - The benefit is that automatic follow-up runs start normally after a
warm window ends

## Linked Issues or Issue Description

Refs #21472 (base of this stacked PR), #21410 (introduced the
warm-window stop).

**What happened?**
In a Daytona two-server chaos wave (`w-20261113T200824-s7106`), three
automatic follow-up runs failed before any provider work with
`native_instruction_materialization_handoff_unavailable`: run
`5f68e194`, run `d6bd5a97` and run `d1198e40`. None of them was at a
release swap. In each case, the predecessor's idle close had failed
first with `provider_state_unreadable` / "Daytona sandbox lease … is no
longer active". The cause: the warm-window stop from #21410 stopped the
sandbox at the same moment that the idle close started. For the first
run's predecessor, the close was requested at 20:30:34 and the sandbox
got SIGTERM at 20:30:37.8.

**What did you expect?**
The follow-up run prepares its instructions fresh and continues. The
warm-window stop does not interrupt the host's own idle close.

**Steps to reproduce**
1. Run a warm Daytona task until it ends, so its warm window starts.
2. Let the warm window end while the idle timer closes the Runner.
3. Start a follow-up run on the same task. It fails with
`native_instruction_materialization_handoff_unavailable`.

Two of the three tasks then ended stranded. That downstream part (the
claim-gate cancellation with `execution_reconciliation_required` and the
stranded notice) is owned and fixed by another session. It is **not** in
this PR.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`
`claimWarmNativeInstructionCopy`: after it retires a stale warm owner
whose copy cannot be adopted, it returns `null`. The caller then
prepares instructions fresh. Before, it threw
`native_instruction_materialization_handoff_unavailable`.
- `server/src/services/warm-sandbox-expiry.ts`: the warm-window stop now
waits `WARM_SANDBOX_IDLE_CLOSE_MARGIN_MS` (2 minutes) past the window.
The in-process idle close (close grace plus the 30 s proof read)
finishes first.
- `native-session-executor.ts` idle timer: when an idle close fails, it
reads the lease rows for that sandbox. If every row is no longer
`active` or `retained` (the host gave the sandbox up), it releases the
in-process owner. The next run then prepares fresh.
- `doc/execution-semantics.md`: one paragraph about the stop margin and
fresh preparation.

## Verification

Red → green (each test fails on the base `35ba76d0ef` and passes on this
branch):

- `native-session-executor.test.ts` › "hands off collection before
probing or retiring a warm owner: unavailable" — red: throws
`handoff_unavailable`; green: returns `null` (fresh preparation).
- `native-session-executor.test.ts` › "forgets a warm owner whose idle
close failed only once the host gave up its sandbox: pending_cleanup /
released / retained" — red on all three; green: the owner is released
for `pending_cleanup` and `released`, and kept (retired by the
successor) for `retained`.
- `warm-sandbox-expiry.test.ts` › "leaves the host's idle close of the
warm runner time to finish before stopping its sandbox" — red: the
sandbox stops 3 s after the window; green: it stops only after the
margin. Two existing tests ("keeps a sandbox still inside its warm
window, and wakes when the window ends", "reconciles a warm sandbox left
running across a restart") now expect the deadline with the margin.

Local runs on this branch:

- `server`: `tsc --noEmit` clean.
- `vitest run src/__tests__/warm-sandbox-expiry.test.ts
src/services/heartbeat/instruction-working-copy.test.ts`: 24/24 passed.
- `vitest run
src/services/native-runtime/native-session-executor.test.ts -t "hands
off collection before probing or retiring a warm owner|forgets a warm
owner whose idle close failed"`: 10/10 passed.

Not run: a Daytona wave with this change.

## Risks

- Warm sandboxes now stop up to 2 minutes later than the configured
window. This is a small cost increase per warm sandbox. I chose a fixed
margin over an in-process "is a close running?" check because the stop
is pure SQL. The margin works when two servers run at the same time (the
server that runs the stop may not be the server that owns the Runner),
and it needs no busy-loop or cross-server signal.
- The margin does not cover a re-armed idle timer or a close that takes
longer than 2 minutes. Fresh preparation (fix 1) and the owner release
(fix 3) cover those cases.
- Not run on Daytona. The evidence is from the wave above and the unit
tests.
- In the wave, a `restore_failed` copy-back onto the just-restarted
sandbox was also seen. I did not find its root cause. It is not changed
here.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, test runs) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 19:47:42 -06:00
1 parent a4c570aede
commit 1f388ed385
5 files changed
+103 -12

No files matched your search

+8
View File
@@ -1639,6 +1639,14 @@ the configured interval until the provider confirms the new one. An explicit `0`
(auto-stop disabled), a longer configured interval, per-turn, and non-reusable
sandboxes are unchanged by the warm floor, and idle sandboxes still auto-stop.
The host's own warm-window stop of a retained sandbox waits 2 minutes past the
warm window, so the host's idle close of the warm runner (suspend and its proof
read) finishes before the sandbox stops. When an idle close still fails and every
lease row for that sandbox is no longer active or retained, the host has given the
sandbox up: the warm owner is released and the next run on the task prepares its
instructions fresh. A follow-up whose predecessor's warm instruction copy cannot be
handed off retires that owner and prepares fresh instead of failing.
A remote runner also keeps working while its controller is offline, and the host
reattaches it within the remote runner reconnect grace (7 minutes). Daytona counts
only SDK and lifecycle calls as activity, so with no controller its idle auto-stop
@@ -8,6 +8,7 @@ import { heartbeatService, type HeartbeatEnvironmentRuntime } from "../services/
import {
nextWarmSandboxStopAt,
stopWarmSandboxesPastWindow,
WARM_SANDBOX_IDLE_CLOSE_MARGIN_MS,
WARM_SANDBOX_IDLE_TIMEOUT_KEY,
warmSandboxWindowMetadata,
} from "../services/warm-sandbox-expiry.js";
@@ -81,14 +82,14 @@ suite("warm sandbox expiry", () => {
expect(await stopWarmSandboxesPastWindow(db)).toEqual({ stopping: 0 });
expect(await row(leaseId)).toEqual(before);
const windowEnd = before.lastUsedAt.getTime() + 5 * MINUTE;
const windowEnd = before.lastUsedAt.getTime() + 5 * MINUTE + WARM_SANDBOX_IDLE_CLOSE_MARGIN_MS;
expect(await nextWarmSandboxStopAt(db)).toBe(windowEnd);
// The cleanup worker sleeps until exactly that deadline: no interval scan.
expect(await nextEnvironmentCleanupAt(db)).toBe(windowEnd);
});
it("stops a sandbox once its warm window has passed and keeps it for the next turn", async () => {
const leaseId = await warmRetained({ releasedAgoMs: 6 * MINUTE, windowMs: 5 * MINUTE });
const leaseId = await warmRetained({ releasedAgoMs: 8 * MINUTE, windowMs: 5 * MINUTE });
expect(await stopWarmSandboxesPastWindow(db)).toEqual({ stopping: 1 });
const stopping = await row(leaseId);
@@ -99,8 +100,21 @@ suite("warm sandbox expiry", () => {
expect(await nextWarmSandboxStopAt(db)).toBeNull();
});
it("leaves the host's idle close of the warm runner time to finish before stopping its sandbox", async () => {
const closing = await warmRetained({ releasedAgoMs: 5 * MINUTE + 3_000, windowMs: 5 * MINUTE });
const before = await row(closing);
expect(await stopWarmSandboxesPastWindow(db)).toEqual({ stopping: 0 });
expect(await row(closing)).toEqual(before);
expect(await nextWarmSandboxStopAt(db)).toBeGreaterThanOrEqual(before.lastUsedAt.getTime() + 5 * MINUTE + 90_000);
await db.update(environmentLeases).set({ lastUsedAt: new Date(Date.now() - 5 * MINUTE - WARM_SANDBOX_IDLE_CLOSE_MARGIN_MS - 1_000) })
.where(eq(environmentLeases.id, closing));
expect(await stopWarmSandboxesPastWindow(db)).toEqual({ stopping: 1 });
});
it("stops the sandbox at the provider through the cleanup sweep", async () => {
const leaseId = await warmRetained({ releasedAgoMs: 6 * MINUTE, windowMs: 5 * MINUTE });
const leaseId = await warmRetained({ releasedAgoMs: 8 * MINUTE, windowMs: 5 * MINUTE });
const teardown = vi.fn(async ({ lease }: { lease: { providerLeaseId: string | null } }) =>
({ providerLeaseId: lease.providerLeaseId, state: "stopped" }));
@@ -118,7 +132,7 @@ suite("warm sandbox expiry", () => {
it("keeps a sandbox past its window while a live run holds it", async () => {
const providerLeaseId = `sandbox://fake/${randomUUID()}`;
const leaseId = await warmRetained({ releasedAgoMs: 6 * MINUTE, providerLeaseId });
const leaseId = await warmRetained({ releasedAgoMs: 8 * MINUTE, providerLeaseId });
const liveRun = await run("running");
await db.insert(environmentLeases).values({ companyId, environmentId, issueId, heartbeatRunId: liveRun,
status: "active", leasePolicy: "reuse_by_environment", provider: "fake", providerLeaseId,
@@ -132,7 +146,7 @@ suite("warm sandbox expiry", () => {
});
it("keeps a sandbox past its window while a run on the task is starting", async () => {
const leaseId = await warmRetained({ releasedAgoMs: 6 * MINUTE });
const leaseId = await warmRetained({ releasedAgoMs: 8 * MINUTE });
await run("running");
const before = await row(leaseId);
@@ -141,7 +155,7 @@ suite("warm sandbox expiry", () => {
});
it("keeps a sandbox whose own run has not ended", async () => {
const leaseId = await warmRetained({ releasedAgoMs: 6 * MINUTE, runStatus: "running" });
const leaseId = await warmRetained({ releasedAgoMs: 8 * MINUTE, runStatus: "running" });
expect(await stopWarmSandboxesPastWindow(db)).toEqual({ stopping: 0 });
expect((await row(leaseId)).status).toBe("retained");
@@ -150,7 +164,7 @@ suite("warm sandbox expiry", () => {
it("leaves a plugin Runner task lease to its own cleanup", async () => {
// Its release path does not use the stop-only contract, so expiry must not
// hand it over: the plugin could free the allocation and the task's files.
const leaseId = await warmRetained({ releasedAgoMs: 6 * MINUTE });
const leaseId = await warmRetained({ releasedAgoMs: 8 * MINUTE });
await db.update(environmentLeases).set({ metadata: { driver: "plugin", pluginId: "fake-plugin",
runnerTaskLease: true, [WARM_SANDBOX_IDLE_TIMEOUT_KEY]: 5 * MINUTE } }).where(eq(environmentLeases.id, leaseId));
const before = await row(leaseId);
@@ -178,7 +192,7 @@ suite("warm sandbox expiry", () => {
expect(await row(leaked)).toMatchObject({ status: "released", cleanupStatus: "success" });
expect((await row(recent)).status).toBe("retained");
const recentRow = await row(recent);
expect(await nextWarmSandboxStopAt(db)).toBe(recentRow.lastUsedAt.getTime() + 24 * HOUR);
expect(await nextWarmSandboxStopAt(db)).toBe(recentRow.lastUsedAt.getTime() + 24 * HOUR + WARM_SANDBOX_IDLE_CLOSE_MARGIN_MS);
});
it("records the run's warm window on the lease only for a warm run", () => {
@@ -11503,6 +11503,51 @@ describe("native warm session supervision", () => {
}
});
it.each([
{ lease: "pending_cleanup", forgotten: true },
{ lease: "released", forgotten: true },
{ lease: "retained", forgotten: false },
])("forgets a warm owner whose idle close failed only once the host gave up its sandbox: $lease", async ({ lease, forgotten }) => {
const identity = `warm-idle-close-sandbox-${lease}`;
const warmExecution = { ...execution, binding: { ...execution.binding, runId: identity, executionWorkspaceId: `${identity}-workspace` },
session: { ...execution.session, normalizedSessionId: identity, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 50 } },
} as NativeExecutionInputV1;
let allowClose = false;
const session = { close: vi.fn(async () => { if (!allowClose) throw new Error("runner_harness_state_mismatch: provider_state_unreadable"); }) };
state.execute.mockReset().mockImplementation(async (options) => {
await options.onSession?.(session);
return { result: { summary: "completed" }, terminal: { runTerminalState: "succeeded" },
turnId: identity, normalizedSessionId: identity, providerSessionId: identity,
driverKind: "test", driverVersion: "1", nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null };
});
const copy = (runId: string) => ({ runId, root: `/tmp/${identity}`, preparationKey: "same", hasChanges: async () => false,
collectStopped: async () => undefined, retirementFailed: async () => undefined });
let leaseRows: Array<{ status: string }> = [];
const base = leaseDb(warmExecution);
const db = { ...base, select: (fields?: unknown) => ({ from: (table: unknown) => table === environmentLeases
? { where: async () => leaseRows } : base.select(fields as never).from(table as never) }) } as unknown as Db;
try {
await executePaperclipNativeSession({ db, execution: warmExecution, runnerInstanceId: identity,
runnerExecutionTarget: { kind: "remote", transport: "sandbox", environmentId: identity, remoteCwd: `/tmp/${identity}`,
sandboxLeaseAcquisition: { providerLeaseId: `${identity}-sandbox` } } as never,
instructionWorkingCopy: copy(identity) });
leaseRows = [{ status: lease }];
await vi.waitFor(() => expect(session.close).toHaveBeenCalledTimes(1));
const followUpRunId = `${identity}-follow-up`;
const adopt = vi.fn(async () => null);
allowClose = true;
await expect(claimWarmNativeInstructionCopy({ priorExecution: warmExecution,
companyId: warmExecution.binding.companyId, agentId: warmExecution.binding.agentId,
executionWorkspaceId: warmExecution.binding.executionWorkspaceId, workspace: warmExecution.workspace,
environmentId: identity, runId: followUpRunId, preparationKey: "same", adopt })).resolves.toBeNull();
expect(adopt).toHaveBeenCalledTimes(forgotten ? 0 : 1);
expect(session.close).toHaveBeenCalledTimes(forgotten ? 1 : 2);
} finally {
allowClose = true;
await closeWarmNativeSessionsForEnvironment({ environmentId: identity, reason: "test cleanup" });
}
});
it("admits the follow-up's execution only after the previous owner's final cleanup hands its scopes back", async () => {
const identity = "warm-release-scope-handback";
const warmExecution = { ...execution, binding: { ...execution.binding, runId: identity, executionWorkspaceId: `${identity}-workspace` },
@@ -11895,8 +11940,8 @@ describe("native warm session supervision", () => {
const claiming = claimWarmNativeInstructionCopy({ priorExecution: first, companyId: first.binding.companyId, agentId: first.binding.agentId,
executionWorkspaceId: first.binding.executionWorkspaceId, workspace: first.workspace, environmentId: identity,
runId: current.runId, preparationKey: current.preparationKey, forceRetirement: scenario === "reset", adopt });
if (["unavailable", "auth", "wrong-capability", "retirement-failure"].includes(scenario)) {
await expect(claiming).rejects.toThrow(({ unavailable: "handoff_unavailable", auth: "authorization denied",
if (["auth", "wrong-capability", "retirement-failure"].includes(scenario)) {
await expect(claiming).rejects.toThrow(({ auth: "authorization denied",
"wrong-capability": "handoff_mismatch", "retirement-failure": "stop unconfirmed" } as Record<string, string>)[scenario]);
} else expect(await claiming).toBeNull();
expect(priorCopy.hasChanges).not.toHaveBeenCalled();
@@ -716,6 +716,7 @@ type WarmNativeSession = {
pluginTaskId?: string;
/** The sandbox that holds this owner's Runner, when it runs in one. */
providerLeaseId?: string;
sandboxLeaseDb?: Db;
/** A successor's retirement of this owner failed; it stays registered for its own sandbox. */
retirementUnproven?: boolean;
observabilityCompatibilityKey?: string;
@@ -1002,7 +1003,7 @@ export async function claimWarmNativeInstructionCopy(input: {
const adopted = await input.adopt(entry.instructionWorkingCopy.runId);
if (!adopted) {
await retire("warm instruction materialization cannot be handed off");
throw new Error("native_instruction_materialization_handoff_unavailable");
return null;
}
if (adopted.runId !== input.runId || adopted.preparationKey !== input.preparationKey) throw new Error("native_instruction_materialization_handoff_mismatch");
entry.instructionWorkingCopy = adopted;
@@ -1088,6 +1089,16 @@ export async function closeWarmNativeSessionsForEnvironment(input: {
return closeIdleWarmNativeSessions(input);
}
async function warmSandboxGivenUpByHost(entry: WarmNativeSession): Promise<boolean> {
if (!entry.sandboxLeaseDb || !entry.providerLeaseId || !entry.environmentId) return false;
const leases = await entry.sandboxLeaseDb.select({ status: environmentLeases.status }).from(environmentLeases).where(and(
eq(environmentLeases.companyId, entry.companyId),
eq(environmentLeases.environmentId, entry.environmentId),
eq(environmentLeases.providerLeaseId, entry.providerLeaseId),
));
return leases.length > 0 && leases.every(lease => lease.status !== "active" && lease.status !== "retained");
}
async function warmSandboxDestroyed(db: Db, entry: WarmNativeSession): Promise<boolean> {
if (!entry.providerLeaseId || !entry.environmentId) return false;
const leases = await db.select().from(environmentLeases).where(and(
@@ -8777,6 +8788,16 @@ function armWarmNativeSessionIdleTimer(
logger.warn({ runId: current.runId, issueId: current.issueId, environmentId: current.environmentId,
providerLeaseId: current.providerLeaseId ?? null, closeFailure: describeWarmCloseFailure(error) },
released ? "idle warm native session detach failed" : "idle warm native session close failed");
if (released) return;
return warmSandboxGivenUpByHost(current).then(givenUp => {
if (!givenUp) return;
logger.warn({ runId: current.runId, issueId: current.issueId, environmentId: current.environmentId,
providerLeaseId: current.providerLeaseId ?? null },
"warm native session's sandbox was stopped by the host; released its owner");
forgetWarmNativeSession(sessionId, current);
}, proofError => {
logger.warn({ err: proofError, runId: current.runId }, "could not read the warm session's sandbox lease");
});
})
.finally(() => {
current.busy = false;
@@ -12602,6 +12623,7 @@ async function executePaperclipNativeSessionAttempt(
pluginTaskId: usesPluginTask && input.runnerExecutionTarget?.kind === "remote" && input.runnerExecutionTarget.transport === "sandbox"
? input.runnerExecutionTarget.sandboxLeaseAcquisition?.providerLeaseId : undefined,
providerLeaseId: sandboxProviderLeaseId(input.runnerExecutionTarget) ?? undefined,
sandboxLeaseDb: input.db,
runId: input.execution.binding.runId,
issueId: input.execution.binding.issueId,
instructionWorkingCopy: input.instructionWorkingCopy?.checkpointWarm ? undefined : input.instructionWorkingCopy,
+3 -1
View File
@@ -40,6 +40,8 @@ export const WARM_SANDBOX_IDLE_TIMEOUT_KEY = "warmSandboxIdleTimeoutMs";
/** Marker on the stop-and-retain intent: the warm window ended. */
export const WARM_WINDOW_EXPIRED_KEY = "warmWindowExpired";
export const WARM_SANDBOX_IDLE_CLOSE_MARGIN_MS = 2 * 60_000;
const STOP_PAGE_SIZE = 25;
function positiveWindowSql(value: ReturnType<typeof sql>) {
@@ -52,7 +54,7 @@ export function warmWindowEndsAtSql() {
const window = sql`least(coalesce(
${positiveWindowSql(sql`${environmentLeases.metadata}->${WARM_SANDBOX_IDLE_TIMEOUT_KEY}`)},
${PAPERCLIP_RUNNER_IDLE_TIMEOUT_MAX_MS}), ${PAPERCLIP_RUNNER_IDLE_TIMEOUT_MAX_MS})`;
return sql`(extract(epoch from ${environmentLeases.lastUsedAt}) * 1000 + ${window})`;
return sql`(extract(epoch from ${environmentLeases.lastUsedAt}) * 1000 + ${window} + ${WARM_SANDBOX_IDLE_CLOSE_MARGIN_MS})`;
}
/**