fix(environments): continue on a new sandbox when an idle runner sandbox never starts (#21323)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task that runs on a reusable cloud sandbox resumes the same
sandbox on the next wake
> - On staging, Daytona left two idle sandboxes in the `archiving` state
for more than three hours, so no resume call could succeed
> - The native runner leases had no verified harness backup, so the run
never replaced the sandbox, even though the task files were saved
> - Each retry spent six minutes on the same stuck sandbox, the bounded
retries ran out, and the task waited for a person to press Retry
> - This pull request continues on a new sandbox when the provider has
proven it cannot start the old one and the workspace is saved, and it
keeps the old sandbox
> - The benefit is that the task recovers by itself and a sandbox that
may hold unsaved runner state is never destroyed

## Linked Issues or Issue Description

Refs #20626, #21113

**What happened?**
Eight Codie runs on staging failed in one hour with
`reusable_sandbox_resume_failed`: `Reusable sandbox lease "<id>" could
not be resumed (cause=transport_timeout); the lease was preserved and no
replacement was created.` They came in two clusters (05:59 to 06:00Z and
06:32 to 06:38Z). Each run took about six minutes, so the 330-second
idle resume budget from #21113 was applied. All runs were already
`transient_failure_retry` wakes.

Investigation of one instance (lease `c23d6120…`, run `d16c272e…`):
- The lease was `released` at 02:37Z with a confirmed stop-and-retain.
No live run held it, so the lease was idle and the extended budget
applied.
- A read-only Daytona API read at 06:5xZ showed the sandbox in state
`archiving` (desired state `archived`), last updated 03:38Z, one hour
after the stop (the auto-archive interval is 60 minutes). The second
sandbox (`e0e6cfe5…`) showed the same state since 03:35Z. Daytona does
not start a sandbox while it archives, so every resume call timed out.
More time or a new HTTP connection could not help.
- The task workspace was clean and had a committed saved snapshot.
- The agent uses `paperclip_runner`. The last run on the sandbox failed
while it saved the runner session, so the lease had no
`nativeHarnessBackup` stamp. `reusableSandboxReplacementIsSafe` requires
a verified harness backup for a runner lease, so the run never replaced
the sandbox. It failed strictly, the bounded retries (2) ran out, and
the task waited for a manual Retry.

**Expected behavior**
When an idle sandbox never starts through the whole provider start
deadline and the task files are saved, the run continues on a new
sandbox automatically. A sandbox that may hold unsaved runner state is
not destroyed.

**Steps to reproduce**
1. Run a `paperclip_runner` task on a reusable Daytona sandbox. Let the
last run end without a harness backup (for example, the runner exits
while it saves the session). The workspace is clean with a committed
snapshot.
2. Let Daytona start to archive the stopped sandbox and get stuck in
`archiving`.
3. Wake the task. Every `environmentResumeLease` call times out, the run
fails after about 330 seconds with `reusable_sandbox_resume_failed`, and
the retries fail the same way.

**Paperclip version or commit**
Staging Core `0c04ff76eb` (current `master` has the same code path).

**Deployment mode**
Hosted staging (cloud), Daytona sandbox provider plugin, reusable leases
on.

**Agent adapter(s) involved**
`paperclip_runner` (native runner) on a Daytona sandbox environment.

**Root cause**
`server/src/services/reusable-sandbox-resume-replacement.ts`: a runner
lease without a verified harness backup was never replaceable, for any
failure. The only other path was to retry the same sandbox, which cannot
work while the provider is stuck. Replacement also always retired the
old lease to `pending_cleanup`, so allowing it without a backup would
have destroyed the only copy of the runner session state.

## What Changed

- `reusable-sandbox-resume-replacement.ts`:
`reusableSandboxReplacementIsSafe` becomes
`reusableSandboxReplacementPlan`. It returns `retire` (same rules as
before), `keep`, or `null`. It returns `keep` for a runner lease without
a verified harness backup only when the caller passes
`providerCannotStart`. All other conditions still apply: idle lease,
same workspace, clean workspace, committed snapshot, this run is the
writer, and no access refusal.
- Add `keepReplacedReusableSandboxLease`. After the replacement lease is
recorded, one guarded write marks the old lease with
`reusableSandboxKeptAfterReplacement`. The lease keeps its status and
its sandbox. Nothing destroys it.
- `environment-runtime.ts`: set `providerCannotStart` only when the idle
start deadline was extended and the run then used all of it on timeouts.
Lease selection skips a lease with the keep marker, so later runs do not
spend time on it and obsolete-lease cleanup does not destroy it.
- The replacement records `replacement` / `resume_failed` with no
harness backup. The native executor already starts a fresh session in a
new sandbox in this case (`freshReplacement`), and workspace preparation
restores the committed snapshot.
- The keep plan needs every resume call to time out. A rate limit, a
reset, or any other failure in between keeps the strict behavior.
- `reusable-sandbox-kept-marker.ts`: the marker key and a SQL guard.
Every automatic teardown path skips a marked lease: retained-orphan
listing and settle, `destroyReusableSandboxLease` (obsolete cleanup,
expiry, run destroy), `destroyClaimedSandboxLease` (issue and workspace
teardown), recovered-workspace cleanup, and abandoned-file release. Only
an operator-consented environment delete still destroys it.
- `doc/execution-semantics.md`: document the rule.

## Verification

- Red before the fix: the new staging-shape test in
`server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts`
(released runner lease, no harness backup, clean committed snapshot,
every resume times out) failed with the staging error `could not be
resumed (cause=transport_timeout); the lease was preserved and no
replacement was created.`
- Review round: "keeps an idle sandbox with no harness backup when some
resume calls failed for another reason", "never destroys a sandbox kept
after replacement through the run or workspace teardown paths", and the
retained-orphan case "keeps a warm sandbox while a replacement kept it"
each failed before the fix and pass after it.
`environment-retained-orphan-sandbox.test.ts`,
`abandoned-workspace-files-hold.test.ts`,
`workspace-recovery-preparation-gates.test.ts`,
`workspace-recovery-admission.test.ts`: all pass.
- Green after the fix: 22/22 pass in the replacement suite. The run gets
a new sandbox within 330 seconds, makes no destroy call, the old lease
stays `released` with the keep marker, and the next run resumes the new
sandbox only. With unsaved work or no saved snapshot, the run still
fails strictly and does not mark the old lease. Earlier tests (a refusal
without a backup, a live owner, a dirty workspace with a timeout) are
unchanged.
- `pnpm exec vitest run src/__tests__/environment-runtime.test.ts
src/services/environment-runtime-reusable-lease-probe.test.ts
src/__tests__/reusable-leases-default.test.ts`: 215 pass.
- `tsc -p server/tsconfig.json --noEmit` passes.
- Not run: a live replacement on staging. It is verified after deploy on
the next wake of the affected tasks.

## Risks

- A runner task that hits this path starts a fresh runner session. The
files come from the saved snapshot, and the task, comments, and run
history are kept. Before this change, the task stopped and waited for a
person.
- The kept sandbox stays with the provider (archived storage) until
someone removes it. This is the cost of never destroying unsaved runner
state.
- The rule needs the full extended start deadline (about 330 seconds) of
timeouts, so a slow but healthy start still resumes the same sandbox.
- No schema change. Old leases follow the same rule.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 02:06:25 -06:00
1 parent 503a736195
commit e73fc71331
9 files changed
+267 -35

No files matched your search

+15 -1
View File
@@ -1472,7 +1472,21 @@ so the cleanup sweep destroys the old sandbox later and no provider delete delay
the first turn. It creates a new sandbox, records the acquisition as
`replacement` with reason `resume_failed`, and workspace preparation restores the
saved snapshot into it. Leases saved by older releases have the same fields and
follow the same rule; one without a verified backup stays strict.
follow the same rule.
A native runner session without a verified harness backup is replaced only when
the provider has proven it cannot start the sandbox: the lease is idle and every
resume call timed out (no rate limit, reset, or other failure in between) through
the whole extended start deadline (for example a
Daytona sandbox stuck in `archiving`). The other conditions above still apply, so
the task's files come from the committed saved snapshot. The new sandbox starts a
fresh runner session. The old lease keeps its status and its sandbox, and is never
destroyed: one guarded write marks it `reusableSandboxKeptAfterReplacement`, and
later runs no longer offer it for resume. Every automatic teardown path skips a
marked lease: obsolete-lease cleanup, run and workspace teardown, expiry,
retained-orphan settlement, recovered-workspace cleanup, and abandoned-file
release. Only an operator-consented environment delete destroys it. Without that proof, a lease without a
verified backup stays strict.
Otherwise (unsaved work, a live owner, a workspace with no committed saved
version, an access refusal, or an unavailable worker) the refusal keeps the strict
@@ -308,7 +308,7 @@ suite("retained orphan sandbox on the shared ownership path", () => {
* is the exact row shape releases before this change wrote; it carries no
* new marker. */
async function warmLease(runId: string, input: { status?: string; workspaceId: string | null; receipt?: boolean;
providerLeaseId?: string; createdAt?: Date }) {
providerLeaseId?: string; createdAt?: Date; metadata?: Record<string, unknown> }) {
const id = randomUUID(), providerLeaseId = input.providerLeaseId ?? `sandbox://fake/${id}`;
const now = input.createdAt ?? new Date();
await db.insert(environmentLeases).values({ id, companyId, environmentId, issueId, heartbeatRunId: runId,
@@ -317,7 +317,7 @@ suite("retained orphan sandbox on the shared ownership path", () => {
createdAt: now, updatedAt: now,
metadata: { agentId, reuseLease: true, ...(input.receipt === false ? {} : { remoteExecutionTermination: {
schema: "paperclip.remote-termination.v1", companyId, runId, leaseId: id, provider: "fake", providerLeaseId,
state: "stopped", confirmedAt: now.toISOString() } }) } });
state: "stopped", confirmedAt: now.toISOString() } }), ...input.metadata } });
return id;
}
@@ -374,6 +374,13 @@ suite("retained orphan sandbox on the shared ownership path", () => {
await db.insert(issues).values({ companyId, title: "Sibling on the shared workspace", executionWorkspaceId: shared });
return [await warmLease(await run("failed"), { workspaceId: shared })];
}],
["a replacement kept it because it may hold the only runner session", async () => {
const [oldWorkspace, newWorkspace] = [await workspace(), await workspace()];
await db.update(issues).set({ executionWorkspaceId: newWorkspace }).where(eq(issues.id, issueId));
return [await warmLease(await run("failed"), { workspaceId: oldWorkspace, metadata: {
reusableSandboxKeptAfterReplacement: { runId: randomUUID(), replacementLeaseId: randomUUID(), recordedAt: new Date().toISOString() },
} })];
}],
["the task still has a run in flight", async () => {
const [oldWorkspace, newWorkspace] = [await workspace(), await workspace()];
await db.update(issues).set({ executionWorkspaceId: newWorkspace }).where(eq(issues.id, issueId));
@@ -394,6 +394,119 @@ describeEmbeddedPostgres("reusable sandbox resume failure replacement", () => {
});
});
it("continues on a new sandbox and keeps the old one when an idle sandbox never starts and has no harness backup (staging shape)", async () => {
const seeded = await seed({ leaseStatus: "released", backupStamp: false });
const clock = rpcTimeoutClock();
const { call, runtime } = worker(seeded, async (_attempt, timeoutMs) => { throw clock.timeOut(timeoutMs); });
const acquired = await acquire(seeded, runtime);
expect(acquired.lease.providerLeaseId).toBe("replacement-sandbox");
expect(acquired.lease.metadata?.sandboxLeaseAcquisition).toEqual({ outcome: "replacement", reason: "resume_failed" });
expect(acquired.lease.metadata?.nativeHarnessBackup).toBeUndefined();
expect(call.mock.calls.filter(([, method]) => method === "environmentResumeLease").length).toBeGreaterThan(1);
expect(clock.elapsed()).toBeLessThanOrEqual(330_000);
expect(call).not.toHaveBeenCalledWith(seeded.pluginId, "environmentDestroyLease", expect.anything(), expect.anything());
const kept = await environmentService(db).getLeaseById(seeded.lease.id);
expect(kept).toMatchObject({ status: "released", providerLeaseId: seeded.providerLeaseId, cleanupStatus: null });
expect(kept?.metadata?.reusableSandboxKeptAfterReplacement).toMatchObject({
runId: seeded.runId, replacementLeaseId: acquired.lease.id,
});
const nextRunId = randomUUID();
await db.insert(heartbeatRuns).values({ id: nextRunId, companyId: seeded.companyId, agentId: seeded.agentId,
invocationSource: "assignment", status: "running" });
await environmentService(db).releaseLease(acquired.lease.id, "released");
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, seeded.runId));
const next = worker(seeded, async () => ({ providerLeaseId: "replacement-sandbox",
metadata: { remoteCwd: "/home/daytona/paperclip-workspace" } }));
const resumed = await acquire(seeded, next.runtime, nextRunId);
expect(resumed.lease.providerLeaseId).toBe("replacement-sandbox");
const nextResumes = next.call.mock.calls.filter(([, method]) => method === "environmentResumeLease");
expect(nextResumes).toHaveLength(1);
expect(nextResumes[0]![2]).toMatchObject({ providerLeaseId: "replacement-sandbox" });
await expect(environmentService(db).getLeaseById(seeded.lease.id)).resolves.toMatchObject({
status: "released", providerLeaseId: seeded.providerLeaseId,
});
});
it("never destroys a sandbox kept after replacement through the run or workspace teardown paths", async () => {
const seeded = await seed({ leaseStatus: "released", backupStamp: false });
const clock = rpcTimeoutClock();
const { call, runtime } = worker(seeded, async (_attempt, timeoutMs) => { throw clock.timeOut(timeoutMs); });
const acquired = await acquire(seeded, runtime);
await environmentService(db).releaseLease(acquired.lease.id, "released");
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, seeded.runId));
const kept = (await environmentService(db).getLeaseById(seeded.lease.id))!;
await expect(runtime.destroyRunLease({ environment: seeded.environment, lease: kept })).resolves.toBeNull();
const destroyed = await runtime.destroyReusableSandboxLeases({ companyId: seeded.companyId,
executionWorkspaceId: seeded.executionWorkspaceId });
expect(destroyed.map(record => record.lease.id)).toEqual([acquired.lease.id]);
expect(call.mock.calls.filter(([, method, params]) => method === "environmentDestroyLease"
&& (params as { providerLeaseId?: string }).providerLeaseId === seeded.providerLeaseId)).toHaveLength(0);
await expect(environmentService(db).getLeaseById(seeded.lease.id)).resolves.toMatchObject({
status: "released", providerLeaseId: seeded.providerLeaseId,
});
});
it("destroys a sandbox kept after replacement when an operator approves the environment delete", async () => {
const seeded = await seed({ leaseStatus: "released", backupStamp: false });
const clock = rpcTimeoutClock();
const { runtime } = worker(seeded, async (_attempt, timeoutMs) => { throw clock.timeOut(timeoutMs); });
const acquired = await acquire(seeded, runtime);
await environmentService(db).releaseLease(acquired.lease.id, "released");
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, seeded.runId));
const teardown = worker(seeded, async () => { throw new Error("no resume during delete"); });
teardown.call.mockImplementation(async (_pluginId: string, method: string, params: unknown) => {
if (method === "environmentDestroyLease") return { providerLeaseId: (params as { providerLeaseId: string }).providerLeaseId };
throw new Error(`Unexpected plugin method: ${method}`);
});
const result = await teardown.runtime.destroyReusableSandboxLeasesForEnvironment({
environmentId: seeded.environment.id, consentedBy: "operator" });
expect(result).toMatchObject({ destroyed: 2, failed: 0 });
expect(teardown.call.mock.calls.filter(([, method, params]) => method === "environmentDestroyLease"
&& (params as { providerLeaseId?: string }).providerLeaseId === seeded.providerLeaseId)).toHaveLength(1);
const deleted = await environmentService(db).getLeaseById(seeded.lease.id);
expect(deleted?.status).not.toBe("pending_cleanup");
expect(deleted?.metadata?.reusableSandboxKeptAfterReplacement).toBeUndefined();
});
it("keeps an idle sandbox with no harness backup when some resume calls failed for another reason", async () => {
const seeded = await seed({ leaseStatus: "released", backupStamp: false });
const clock = rpcTimeoutClock();
const { call, runtime } = worker(seeded, async (attempt, timeoutMs) => {
if (attempt === 1) throw clock.timeOut(timeoutMs);
throw Object.assign(clock.timeOut(timeoutMs), { message: "connection reset by peer (ECONNRESET)" });
});
await expect(acquire(seeded, runtime)).rejects.toThrow(/the lease was preserved and no replacement was created\.$/);
expect(call.mock.calls.filter(([, method]) => method === "environmentResumeLease").length).toBeGreaterThan(1);
expect(call).not.toHaveBeenCalledWith(seeded.pluginId, "environmentAcquireLease", expect.anything(), expect.anything());
const kept = await environmentService(db).getLeaseById(seeded.lease.id);
expect(kept?.metadata?.reusableSandboxKeptAfterReplacement).toBeUndefined();
});
it.each([
["unsaved work on the sandbox", { dirty: true }],
["no saved workspace version", { snapshot: null }],
] as Array<[string, Shape]>)("keeps an idle sandbox with no harness backup that never starts and fails strictly with %s", async (_label, shape) => {
const seeded = await seed({ leaseStatus: "released", backupStamp: false, ...shape });
const clock = rpcTimeoutClock();
const { call, runtime } = worker(seeded, async (_attempt, timeoutMs) => { throw clock.timeOut(timeoutMs); });
await expect(acquire(seeded, runtime)).rejects.toThrow(
/could not be resumed \(cause=transport_timeout\); the lease was preserved and no replacement was created\.$/,
);
expect(call).not.toHaveBeenCalledWith(seeded.pluginId, "environmentAcquireLease", expect.anything(), expect.anything());
const kept = await environmentService(db).getLeaseById(seeded.lease.id);
expect(kept).toMatchObject({ status: "released", providerLeaseId: seeded.providerLeaseId });
expect(kept?.metadata?.reusableSandboxKeptAfterReplacement).toBeUndefined();
});
it("does not extend the wait for a sandbox that a live run still holds", async () => {
const seeded = await seed({ previousRunStatus: "running", dirty: true });
const clock = rpcTimeoutClock();
@@ -7,6 +7,7 @@ import type { OwnerProof } from "./owner-proof.js";
import type { InProgressRecord, SettleOutcome } from "./recovery/ownership-registry.js";
import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
import { ORPHANED_SANDBOX_RECOVERY_KEY, prepareSupersededWarmSandboxStop } from "./sandbox-stop-and-retain.js";
import { REUSABLE_SANDBOX_KEPT_AFTER_REPLACEMENT_KEY } from "./reusable-sandbox-kept-marker.js";
/** Failure reason on a retained orphan sandbox released for destruction. */
export const RETAINED_ORPHAN_RELEASED_REASON = "retained_orphan_sandbox_released";
@@ -111,7 +112,7 @@ function softlyHeldSql(now?: Date) {
function alwaysHeldSql() {
return sql`(${environmentLeases.metadata} ?| array['nativeWorkspaceExportResume', 'sandboxStopAndRetain',
'workspaceRestoreRecovery', 'workspaceRecoveryCleanup', 'runnerTaskSourceConsumer']
'workspaceRestoreRecovery', 'workspaceRecoveryCleanup', 'runnerTaskSourceConsumer', ${REUSABLE_SANDBOX_KEPT_AFTER_REPLACEMENT_KEY}]
or ${abandonedWorkspaceFilesHeldOnLeaseRowSql()}
or coalesce(${environmentLeases.metadata}->>'pendingCleanupInFlight', 'false') = 'true'
or coalesce(${environmentLeases.metadata}#>>'{legacyShutdownWorkspaceSettlement,state}' = 'pending', false)
@@ -241,6 +242,7 @@ async function heldBy(tx: Db, lease: LeaseRow, runId: string, now: Date): Promis
if (metadata.workspaceRestoreRecovery !== undefined) return "workspace_restore_recovery";
if (metadata.workspaceRecoveryCleanup !== undefined) return "workspace_recovery_cleanup";
if (metadata.runnerTaskSourceConsumer !== undefined) return "source_recovery_consumer";
if (metadata[REUSABLE_SANDBOX_KEPT_AFTER_REPLACEMENT_KEY] !== undefined) return "kept_after_replacement";
const shutdown = metadata.legacyShutdownWorkspaceSettlement as Record<string, unknown> | undefined;
if (shutdown?.state === "pending") return "legacy_shutdown_fence";
const [other] = await tx.select({ id: environmentLeases.id }).from(environmentLeases).where(and(
+44 -12
View File
@@ -7,10 +7,15 @@ import { sandboxProviderFailureLabel } from "./remote-workspace-transport.js";
import { NATIVE_PREDECESSOR_RETIREMENT_UNPROVEN_CODE, readReplaceBeforeReuseMarker } from "./native-runtime/predecessor-retirement.js";
import {
REUSABLE_SANDBOX_IDLE_RESUME_BUDGET_MS,
REUSABLE_SANDBOX_KEPT_AFTER_REPLACEMENT_KEY,
REUSABLE_SANDBOX_RESUME_BUDGET_MS,
keepReplacedReusableSandboxLease,
retireReplacedReusableSandboxLease,
reusableSandboxKeptAfterReplacement,
reusableSandboxLeaseIsIdle,
reusableSandboxReplacementIsSafe,
reusableSandboxNotKeptAfterReplacementSql,
reusableSandboxReplacementPlan,
type ReusableSandboxReplacementPlan,
sandboxStateTransitionFailure,
} from "./reusable-sandbox-resume-replacement.js";
import { PluginWorkerUnavailableError } from "./plugin-worker-unavailable-error.js";
@@ -1808,6 +1813,7 @@ function createSandboxEnvironmentDriver(
? (await environmentsSvc.listLeases(input.environment.id))
.filter((lease) =>
lease.leasePolicy === "reuse_by_environment" &&
!reusableSandboxKeptAfterReplacement(lease) &&
reusableLeaseCanBeResumed({ lease, heartbeatRunId: input.heartbeatRunId }) &&
lease.executionWorkspaceId === input.executionWorkspaceId &&
(input.executionWorkspaceId !== null || lease.issueId === input.issueId) &&
@@ -1859,6 +1865,7 @@ function createSandboxEnvironmentDriver(
let providerLease: PluginEnvironmentLease | null = null;
let replacingUnretiredOwner = false;
let replacementPlan: ReusableSandboxReplacementPlan | null = null;
let replacementReason:
| "not_found"
| "expired"
@@ -1888,13 +1895,16 @@ function createSandboxEnvironmentDriver(
// sandbox asked for a fresh one. Replace it under the same safety
// rule as a failed resume; otherwise resume it as before.
replacingUnretiredOwner = readReplaceBeforeReuseMarker(reusableLease.metadata) !== null
&& await reusableSandboxReplacementIsSafe({
&& await reusableSandboxReplacementPlan({
db, lease: reusableLease, companyId: input.companyId, heartbeatRunId: input.heartbeatRunId,
executionWorkspaceId: input.executionWorkspaceId, adapterType: input.adapterType,
cause: new Error(NATIVE_PREDECESSOR_RETIREMENT_UNPROVEN_CODE),
});
if (replacingUnretiredOwner) replacementReason = "resume_failed";
else try {
}) === "retire";
let idleStartWaitSpent = false;
if (replacingUnretiredOwner) {
replacementReason = "resume_failed";
replacementPlan = "retire";
} else try {
// One run spends a bounded budget resuming, retries included; the
// provider's own start deadline (five minutes on Daytona) is not it.
const providerStartTimeoutMs = resolvePluginSandboxRpcTimeoutMs(
@@ -1909,6 +1919,8 @@ function createSandboxEnvironmentDriver(
Math.min(providerStartTimeoutMs, REUSABLE_SANDBOX_IDLE_RESUME_BUDGET_MS),
);
let retryDelayMs = 250;
let idleStartWaitExtended = false;
let everyResumeTimedOut = true;
let resumed: PluginEnvironmentLease;
while (true) {
try {
@@ -1932,6 +1944,7 @@ function createSandboxEnvironmentDriver(
break;
} catch (error) {
if (!transientSandboxResumeFailure(error)) throw error;
if (sandboxProviderFailureLabel(error).label !== "transport_timeout") everyResumeTimedOut = false;
const retryAt = Date.now() + retryDelayMs * 1.25;
if (retryAt >= resumeDeadline) {
const waitForIdleStart = resumeDeadline < idleResumeDeadline && retryAt < idleResumeDeadline
@@ -1939,8 +1952,12 @@ function createSandboxEnvironmentDriver(
&& await reusableSandboxLeaseIsIdle({
db, lease: reusableLease, companyId: input.companyId, heartbeatRunId: input.heartbeatRunId,
});
if (!waitForIdleStart) throw error;
if (!waitForIdleStart) {
idleStartWaitSpent = idleStartWaitExtended && everyResumeTimedOut;
throw error;
}
resumeDeadline = idleResumeDeadline;
idleStartWaitExtended = true;
}
const jitteredDelayMs = Math.max(
1,
@@ -1967,10 +1984,12 @@ function createSandboxEnvironmentDriver(
} catch (error) {
// A refusal is not proof the sandbox is lost. Replace it only when
// nothing lives solely on it; otherwise preserve it for the retry.
if (!await reusableSandboxReplacementIsSafe({
replacementPlan = await reusableSandboxReplacementPlan({
db, lease: reusableLease, companyId: input.companyId, heartbeatRunId: input.heartbeatRunId,
executionWorkspaceId: input.executionWorkspaceId, adapterType: input.adapterType, cause: error,
})) {
providerCannotStart: idleStartWaitSpent,
});
if (!replacementPlan) {
throw new ReusableSandboxResumeError({
provider: parsed.config.provider,
providerLeaseId: reusableLease.providerLeaseId,
@@ -2142,9 +2161,16 @@ function createSandboxEnvironmentDriver(
// carries the harness backup, is recorded. A failed creation or a
// restart before this point leaves it eligible for the next attempt.
if (replacementReason === "resume_failed" && reusableLease && input.heartbeatRunId) {
await retireReplacedReusableSandboxLease({
db, lease: reusableLease, companyId: input.companyId, heartbeatRunId: input.heartbeatRunId,
});
if (replacementPlan === "keep") {
await keepReplacedReusableSandboxLease({
db, lease: reusableLease, companyId: input.companyId, heartbeatRunId: input.heartbeatRunId,
replacementLeaseId: recordedLease.id,
});
} else {
await retireReplacedReusableSandboxLease({
db, lease: reusableLease, companyId: input.companyId, heartbeatRunId: input.heartbeatRunId,
});
}
}
return recordedLease;
}
@@ -2195,6 +2221,7 @@ function createSandboxEnvironmentDriver(
? (await environmentsSvc.listLeases(input.environment.id))
.filter((lease) =>
lease.leasePolicy === "reuse_by_environment" &&
!reusableSandboxKeptAfterReplacement(lease) &&
reusableLeaseCanBeResumed({ lease, heartbeatRunId: input.heartbeatRunId }) &&
lease.executionWorkspaceId === input.executionWorkspaceId &&
(input.executionWorkspaceId !== null || lease.issueId === input.issueId) &&
@@ -3216,6 +3243,7 @@ function createSandboxEnvironmentDriver(
lease: EnvironmentLease;
failureReason: string;
}): Promise<EnvironmentLease | null> {
if (reusableSandboxKeptAfterReplacement(input.lease)) return null;
if (await legacyShutdownWorkspaceResourceProtected(db, input.lease)) return null;
if (await abandonedWorkspaceFilesHeldOnLease(db, input.lease)) {
await parkAbandonedWorkspaceFilesLease(db, input.lease, ["released", "retained", "failed", "pending_cleanup"]);
@@ -3932,6 +3960,7 @@ export function environmentRuntimeService(
runId?: string;
failureReason: string;
anyLeasePolicy?: boolean;
releasesKeptSandbox?: boolean;
}): Promise<EnvironmentLease | null> {
if (await nativeWorkspaceExportPendingForLease(db, input.leaseRow)) return null;
if (!(await allowLegacyShutdownWorkspaceCleanup(db, input.leaseRow))) return null;
@@ -3952,7 +3981,8 @@ export function environmentRuntimeService(
lastUsedAt: now,
updatedAt: now,
metadata: sql`(case when jsonb_typeof(${environmentLeases.metadata}) = 'object'
then ${environmentLeases.metadata} else '{}'::jsonb end - 'remoteExecutionTermination')
then ${environmentLeases.metadata} else '{}'::jsonb end - 'remoteExecutionTermination'
- ${input.releasesKeptSandbox ? REUSABLE_SANDBOX_KEPT_AFTER_REPLACEMENT_KEY : ""}::text)
|| ${JSON.stringify({ pendingCleanupAttemptId: attemptId, pendingCleanupInFlight: true,
pendingCleanupLeaseExpiresAtMs: Date.now() + 15 * 60_000, pendingCleanupController: controller })}::jsonb`,
}).where(and(
@@ -3963,6 +3993,7 @@ export function environmentRuntimeService(
: input.anyLeasePolicy ? undefined : eq(environmentLeases.leasePolicy, "reuse_by_environment"),
eq(environmentLeases.status, input.leaseRow.status),
input.scopeCondition,
input.releasesKeptSandbox ? undefined : reusableSandboxNotKeptAfterReplacementSql(),
sql`coalesce(${environmentLeases.metadata}->>'pendingCleanupInFlight', 'false') != 'true'`,
@@ -4602,6 +4633,7 @@ export function environmentRuntimeService(
continue;
}
const lease = await destroyClaimedSandboxLease({ environment, leaseRow, failureReason, anyLeasePolicy: true,
releasesKeptSandbox: true,
scopeCondition: inArray(environmentLeases.leasePolicy, [...KEPT_SANDBOX_LEASE_POLICIES]) });
if (!lease) {
// Lost to a racing resume or a concurrent terminal transition — the
@@ -3,6 +3,7 @@ import { companies, environmentLeases, executionWorkspaces, heartbeatRuns, issue
import { logger } from "../middleware/logger.js";
import { logActivity } from "./activity-log.js";
import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
import { reusableSandboxNotKeptAfterReplacementSql } from "./reusable-sandbox-kept-marker.js";
import {
ABANDONED_WORKSPACE_FILES_RETENTION_DAYS, ABANDONED_WORKSPACE_FILES_RETENTION_MS, REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA,
} from "./remote-workspace-epoch-abandonment.js";
@@ -129,7 +130,7 @@ export async function releaseAbandonedWorkspaceFiles(db: Db, input: {
|| ${JSON.stringify({ [ABANDONED_WORKSPACE_FILES_RELEASE_KEY]: marker })}::jsonb`,
}).where(and(eq(environmentLeases.companyId, input.companyId), eq(environmentLeases.providerLeaseId, providerLeaseId),
eq(environmentLeases.status, "retained"), sql`${environmentLeases.metadata} ? ${ABANDONED_WORKSPACE_FILES_PARKED_KEY}`,
sql`not ${abandonedWorkspaceFilesHeldOnLeaseRowSql()}`)).returning({ id: environmentLeases.id }),
sql`not ${abandonedWorkspaceFilesHeldOnLeaseRowSql()}`, reusableSandboxNotKeptAfterReplacementSql())).returning({ id: environmentLeases.id }),
{ companyId: input.companyId });
const scheduled = unparked.length > 0;
const summary = input.reason === "retention_expired"
@@ -0,0 +1,14 @@
import { sql } from "drizzle-orm";
import { environmentLeases } from "@paperclipai/db";
import type { EnvironmentLease } from "@paperclipai/shared";
export const REUSABLE_SANDBOX_KEPT_AFTER_REPLACEMENT_KEY = "reusableSandboxKeptAfterReplacement";
export function reusableSandboxKeptAfterReplacement(lease: Pick<EnvironmentLease, "metadata">): boolean {
const marker = lease.metadata?.[REUSABLE_SANDBOX_KEPT_AFTER_REPLACEMENT_KEY];
return typeof marker === "object" && marker !== null && !Array.isArray(marker);
}
export function reusableSandboxNotKeptAfterReplacementSql() {
return sql`not coalesce(${environmentLeases.metadata} ? ${REUSABLE_SANDBOX_KEPT_AFTER_REPLACEMENT_KEY}, false)`;
}
@@ -9,6 +9,13 @@ import { verifyRecoverableNativeHarnessBackup } from "./native-runtime/native-ha
import { findPluginWorkerUnavailableError } from "./plugin-worker-unavailable-error.js";
import { abandonedWorkspaceFilesHeldOnLeaseRowSql, ABANDONED_WORKSPACE_FILES_PARKED_KEY, ABANDONED_WORKSPACE_FILES_PARKED_REASON } from "./remote-workspace-abandoned-files-hold.js";
import { sandboxProviderFailureLabel } from "./remote-workspace-transport.js";
import { REUSABLE_SANDBOX_KEPT_AFTER_REPLACEMENT_KEY } from "./reusable-sandbox-kept-marker.js";
export {
REUSABLE_SANDBOX_KEPT_AFTER_REPLACEMENT_KEY,
reusableSandboxKeptAfterReplacement,
reusableSandboxNotKeptAfterReplacementSql,
} from "./reusable-sandbox-kept-marker.js";
/**
* Total time one run spends resuming a preserved reusable sandbox, including
@@ -23,6 +30,10 @@ export const REUSABLE_SANDBOX_IDLE_RESUME_BUDGET_MS = 330_000;
/** Failure reason recorded on a preserved lease retired in favor of a replacement. */
export const REUSABLE_SANDBOX_RESUME_REPLACED_REASON = "resume_failed";
export type ReusableSandboxReplacementPlan = "retire" | "keep";
/**
* A provider refused to start a sandbox because it is still changing state,
* for example auto-stopping or archiving after an idle period. Waiting can
@@ -55,21 +66,23 @@ export async function reusableSandboxLeaseIsIdle(input: {
/**
* Decide whether a preserved reusable sandbox that could not be resumed may be
* replaced.
* replaced, and what happens to it.
*
* Replacement is safe only when nothing lives solely on that sandbox:
* - its last run has ended (released or retained, and no live run holds it);
* - the task workspace is clean and its saved version is a committed snapshot,
* so a new sandbox restores exactly the same files;
* - a native runner session has a verified, lease-bound harness backup.
* - a native runner session has a verified, lease-bound harness backup, or the
* provider proved it cannot start the sandbox, which is then kept, not retired.
* Anything else (unsaved work, a live owner, a legacy workspace with no saved
* version, an access refusal) keeps the strict preserve-and-retry behavior.
*
* This only reads. The caller creates and records the replacement first, then
* calls `retireReplacedReusableSandboxLease`, so a failed creation or a restart
* in between leaves the preserved lease eligible for the next attempt.
* calls `retireReplacedReusableSandboxLease` or `keepReplacedReusableSandboxLease`,
* so a failed creation or a restart in between leaves the preserved lease
* eligible for the next attempt.
*/
export async function reusableSandboxReplacementIsSafe(input: {
export async function reusableSandboxReplacementPlan(input: {
db: Db;
lease: EnvironmentLease;
companyId: string;
@@ -77,17 +90,18 @@ export async function reusableSandboxReplacementIsSafe(input: {
executionWorkspaceId: string | null;
adapterType: string | null | undefined;
cause: unknown;
}): Promise<boolean> {
providerCannotStart?: boolean;
}): Promise<ReusableSandboxReplacementPlan | null> {
const { db, lease } = input;
if (!lease.providerLeaseId || !await reusableSandboxLeaseIsIdle(input)) return false;
if (!input.executionWorkspaceId || lease.executionWorkspaceId !== input.executionWorkspaceId) return false;
if (!lease.providerLeaseId || !await reusableSandboxLeaseIsIdle(input)) return null;
if (!input.executionWorkspaceId || lease.executionWorkspaceId !== input.executionWorkspaceId) return null;
// A replacement cannot succeed where the resume was refused for access, and
// an unavailable worker cannot create one either.
if (findPluginWorkerUnavailableError(input.cause)) return false;
if (findPluginWorkerUnavailableError(input.cause)) return null;
const failure = sandboxProviderFailureLabel(input.cause);
const status = failure.status ?? readEnvironmentResumeErrorDiagnostic(input.cause)?.httpStatus ?? null;
if (failure.label === "provider_access_denied" || failure.label === "permission_denied"
|| status === 401 || status === 403) return false;
|| status === 401 || status === 403) return null;
const [workspace] = await db.select({ metadata: executionWorkspaces.metadata }).from(executionWorkspaces)
.where(and(eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, input.executionWorkspaceId)));
@@ -95,25 +109,27 @@ export async function reusableSandboxReplacementIsSafe(input: {
try {
authority = readRemoteWorkspaceAuthority(workspace?.metadata);
} catch {
return false;
return null;
}
if (!authority || authority.dirty || !authority.snapshotId) return false;
if (authority.writerRunId !== null && authority.writerRunId !== input.heartbeatRunId) return false;
if (authority.location && authority.location.providerLeaseId !== lease.providerLeaseId) return false;
if (!authority || authority.dirty || !authority.snapshotId) return null;
if (authority.writerRunId !== null && authority.writerRunId !== input.heartbeatRunId) return null;
if (authority.location && authority.location.providerLeaseId !== lease.providerLeaseId) return null;
const [snapshot] = await db.select({ id: executionWorkspaceSnapshots.id }).from(executionWorkspaceSnapshots).where(and(
eq(executionWorkspaceSnapshots.companyId, input.companyId),
eq(executionWorkspaceSnapshots.executionWorkspaceId, input.executionWorkspaceId),
eq(executionWorkspaceSnapshots.id, authority.snapshotId),
eq(executionWorkspaceSnapshots.status, "committed"),
));
if (!snapshot) return false;
if (!snapshot) return null;
const scope = lease.metadata?.reusableSandboxLease as { adapterType?: unknown } | undefined;
if ((input.adapterType === "paperclip_runner" || scope?.adapterType === "paperclip_runner")
&& !await verifyRecoverableNativeHarnessBackup({ db, companyId: lease.companyId,
stamp: lease.metadata?.nativeHarnessBackup, providerLeaseId: lease.providerLeaseId })) return false;
stamp: lease.metadata?.nativeHarnessBackup, providerLeaseId: lease.providerLeaseId })) {
return input.providerCannotStart === true ? "keep" : null;
}
return true;
return "retire";
}
/**
@@ -158,3 +174,34 @@ export async function retireReplacedReusableSandboxLease(input: {
)).returning({ id: environmentLeases.id }), { companyId: input.companyId, runId: lease.heartbeatRunId ?? undefined });
return Boolean(claimed);
}
export async function keepReplacedReusableSandboxLease(input: {
db: Db;
lease: EnvironmentLease;
companyId: string;
heartbeatRunId: string;
replacementLeaseId: string;
}): Promise<boolean> {
const { db, lease } = input;
if (!lease.providerLeaseId) return false;
const now = new Date();
const marker = { [REUSABLE_SANDBOX_KEPT_AFTER_REPLACEMENT_KEY]: {
runId: input.heartbeatRunId, replacementLeaseId: input.replacementLeaseId, recordedAt: now.toISOString(),
} };
const [kept] = await db.update(environmentLeases).set({
metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify(marker)}::jsonb`,
updatedAt: now,
}).where(and(
eq(environmentLeases.id, lease.id),
eq(environmentLeases.companyId, input.companyId),
eq(environmentLeases.providerLeaseId, lease.providerLeaseId),
inArray(environmentLeases.status, ["released", "retained"]),
sql`NOT EXISTS (
SELECT 1 FROM ${heartbeatRuns}
WHERE ${heartbeatRuns.id} = ${environmentLeases.heartbeatRunId}
AND ${heartbeatRuns.id} <> ${input.heartbeatRunId}
AND ${heartbeatRuns.status} IN ('queued', 'scheduled_retry', 'running')
)`,
)).returning({ id: environmentLeases.id });
return Boolean(kept);
}
+3 -1
View File
@@ -40,6 +40,7 @@ import { capturePolicyRearmDueCondition, retainedSourceFallbackDueCondition, sto
import { rearmExhaustedCapturePolicy } from "./workspace-recovery-capture-policy.js";
import { rearmExhaustedStopProof } from "./workspace-recovery-stop-proof.js";
import { settleRecoveredWorkspaceRestoreHolds } from "./workspace-restore-hold-settlement.js";
import { reusableSandboxNotKeptAfterReplacementSql } from "./reusable-sandbox-kept-marker.js";
const exportCauses = ["native_workspace_sync_out_unrecoverable", "native_workspace_sync_out_retry_exhausted"];
const terminal = ["failed", "interrupted", "timed_out", "cancelled", "succeeded"];
@@ -600,7 +601,8 @@ export async function scheduleRecoveredWorkspaceCleanup(db: Db, run: typeof hear
eq(environmentLeases.heartbeatRunId, run.id), eq(environmentLeases.providerLeaseId, ref.providerLeaseId),
eq(environmentLeases.status, "released"), eq(environmentLeases.cleanupStatus, "success"),
sql`${environmentLeases.metadata} = ${JSON.stringify(lease.metadata)}::jsonb`,
sql`${environmentLeases.metadata}->>'workspaceRecoveryCleanup' is null`)), { companyId: run.companyId, runId: run.id });
sql`${environmentLeases.metadata}->>'workspaceRecoveryCleanup' is null`, reusableSandboxNotKeptAfterReplacementSql())),
{ companyId: run.companyId, runId: run.id });
}
/** `dueOnly` narrows the obligations to exhausted holds whose re-arm or
* fallback is due now. Cleanup, interrupted preparation and saved input that