fix(recovery): wait out a sandbox the provider could not start, with no Retry (#22194)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task that runs in a sandbox environment first gets a lease from
the environment provider (for example Daytona)
> - Sometimes the provider creates the sandbox but cannot start it.
Daytona then puts the sandbox in its `error` state
> - The Daytona plugin deletes that failed sandbox and rethrows the
provider error. The error has no HTTP status and no errno, so the host
cannot tell that it is a provider-side failure
> - The run fails with `setup_failed`, and the task rests on "Retry when
ready". No agent work had started, and a fresh sandbox works, so a
person must click Retry for something the system can do itself
> - Recovery must just work, and the target for runs that need Retry is
0%
> - This pull request makes the plugin report a removed sandbox that the
provider could not start as structured evidence (the step and the
provider state). The host classifies it as a transient provider failure.
The task then uses the existing bounded retries and the provider-outage
wait from #22139
> - The benefit is that a provider start failure delays the task by
seconds instead of stranding it on Retry

## Linked Issues or Issue Description

Refs #22139 (the provider-outage wait lane that this failure now
reaches). Refs #22128 (setup failure after work; this failure happens
before any work, so that path did not apply).

**What happened?**
In a restart-chaos wave with graceful release swaps on 2026-11-18, two
tasks started a run at 14:51:15Z and 14:51:24Z. Both runs failed at
14:51:53Z with `setup_failed` and `failureCause: lease_acquire_failed`.
The cause was `Sandbox <id> failed to start with status: error, error
reason: null` from the Daytona SDK, for two different sandboxes at the
same moment. Both tasks rested on Retry. A Retry click started each task
again, and both tasks completed.

**Expected behavior**
A sandbox that the provider cannot start is a provider failure before
any agent work. Paperclip deletes it and tries again with a new sandbox,
with no Retry. If the provider keeps failing, the task waits for the
provider without a Retry, and a person gets a working Retry only after
the 30-minute ceiling.

**Steps to reproduce**
1. Make the Daytona create call fail while the new sandbox is in the
provider's `error` state.
2. Assign a task to an agent that uses that environment.
3. The run fails with `setup_failed`, and the task rests on Retry.

## What Changed

- Plugin SDK (`environment-creation-cleanup.ts`,
`environment-acquire-error.ts`, `index.ts`): new
`PluginEnvironmentAcquisitionError`. It carries the acquisition
diagnostic (step, elapsed time, budget, provider state) for a failed
allocation that the provider already removed, with no cleanup envelope.
The acquire RPC error data (`paperclip/environment-acquire-error/v1`)
now carries this diagnostic. `readEnvironmentAcquisitionDiagnostic`
reads it from that envelope. The existing cleanup envelope path is
unchanged. No provider text crosses the RPC.
- Daytona plugin (`plugin.ts`): `destroyFailedCreation` returns the
state of the sandbox that it deleted. When a create fails without an
HTTP status and the deleted sandbox was in the `error` state, the plugin
throws `PluginEnvironmentAcquisitionError` with `providerState:
"error"`. A `build_failed` sandbox (a deterministic snapshot build
failure) and any other state keep the original error.
- Server (`environment-runtime.ts`): the acquire path records the
diagnostic for this run also when the error has no cleanup envelope. A
cleanup envelope for another run, company or environment still records
nothing.
- Server (`environment-provider-unavailable-error.ts`): an acquisition
diagnostic with `providerState: "error"` is the transient reason
`provider_start_failed` ("the provider could not start the sandbox"). It
becomes `EnvironmentProviderUnavailableError`, so the run uses the two
bounded retries and then the provider-outage wait lane (one probe per
environment, 30-minute ceiling).
- `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pins for
the changed SDK sources (generated by `node
tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --write`).

Module boundaries: the change to the shared SDK contract is additive.
The Daytona plugin owns the provider state. The host reads only the
closed diagnostic fields. A plugin that does not use the new class
behaves as before.

## Verification

- `packages/plugins/sandbox-providers/daytona`: `npx vitest run
src/plugin.test.ts -t "failed sandbox creation cleanup"` (37 passed).
New: "reports a sandbox that failed to start as a removed allocation
with its observed step" (red before the plugin change, green after) and
"keeps the create error when the removed sandbox was build_failed /
started".
- `packages/plugins/sdk`: `npx vitest run
tests/environment-acquire-error.test.ts
tests/environment-creation-cleanup.test.ts` (50 passed). New: the
diagnostic crosses the real worker RPC without provider text, and forged
or unstructured envelopes give no diagnostic.
- `server`: `npx vitest run
src/services/environment-provider-unavailable-error.test.ts
src/__tests__/environment-acquisition-diagnostics.test.ts` (74 passed).
New: the reason is `provider_start_failed` with a transient provider
failure, and a diagnostic that was not recorded for this run is not
retried.
- `server`: `npx vitest run src/__tests__/environment-runtime.test.ts -t
"provider could not start|retains uncertain plugin creation"` (12
passed). New: "records the step of a sandbox the provider could not
start and removed" for a worker error (red before the runtime change,
green after) and for a forged error (not recorded).
- `server`: `npx vitest run
src/services/__tests__/run-failure-diagnostics.test.ts
src/__tests__/heartbeat-process-recovery.test.ts -t
"provider|acquisition|diagnostic"` (167 passed).
- Typecheck: `server` `tsc --noEmit`, `pnpm run typecheck` in
`packages/plugins/sdk` and in the Daytona plugin all pass.
- `pnpm verify` exits 2 (the SDK input registry is a global CI input).
Required CI supplies the broad coverage.

## Risks

- **Automatic retries for a deterministic start failure.** If a provider
image or configuration made every sandbox go to `error`, the task now
waits in the provider-outage lane instead of resting on Retry at once.
The lane uses one probe per environment on a 1, 2, then 5 minute
backoff, and it hands the task to a person with a working Retry after 30
minutes. The aggregate automatic-run brake is unchanged. `build_failed`
is not retried by this change.
- **Shared SDK contract.** The new field is additive. An older host
ignores it, and the run fails as before.
- No database migration. No UI change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in
Claude Code with tool use (shell, file edits) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-18 11:47:33 -06:00
1 parent a9dc0df9da
commit f0bd99ba8a
11 files changed
+182 -26

No files matched your search

@@ -1,5 +1,6 @@
import { JsonRpcCallError, environmentResumeErrorData } from "@paperclipai/plugin-sdk";
import { collectRunFailureDiagnostics } from "../services/run-failure-diagnostics.js";
import { readTransientLeaseAcquisitionFailure } from "../services/environment-provider-unavailable-error.js";
import { PluginWorkerUnavailableError } from "../services/plugin-worker-unavailable-error.js";
import { createSandboxOrphanCleanupSpool } from "../services/sandbox-orphan-cleanup-spool.js";
import { idleWorkSnapshot } from "../services/task-admission.js";
@@ -1496,6 +1497,37 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
.toEqual([expect.objectContaining({ state: "pending", originalEnvironmentId: seeded.environment.id, runId: seeded.runId })]);
});
it.each(["worker", "forged error", "invalid worker diagnostic"])("records the step of a sandbox the provider could not start and removed: %s", async (mode) => {
const seeded = await seedReusablePluginSandboxLease();
await environmentService(db).releaseLease(seeded.reusableLease.id, "expired");
const data = { schema: "paperclip/environment-acquire-error/v1",
acquisitionDiagnostic: { phase: mode === "invalid worker diagnostic" ? "private-command" : "create",
elapsedMs: 38_000, budgetMs: 90_000, providerState: "error" } };
const failure = mode === "forged error"
? Object.assign(new Error("Daytona could not start the sandbox; it was removed."), { data })
: new JsonRpcCallError({ code: -32002, message: "Daytona could not start the sandbox; it was removed.", data });
const call = vi.fn(async (_id: string, method: string) => {
if (method === "environmentAcquireLease") throw failure;
throw new Error(`Unexpected method ${method}`);
});
const worker = { isRunning: () => true, getWorker: () => ({ supportedMethods: ["environmentDestroyLease"] }), call } as unknown as PluginWorkerManager;
const caught = await environmentRuntimeService(db, { pluginWorkerManager: worker }).acquireRunLease({
companyId: seeded.companyId, environment: seeded.environment, issueId: null, heartbeatRunId: seeded.runId,
persistedExecutionWorkspace: null }).catch(error => error);
expect(caught).toBe(failure);
const run = { id: seeded.runId, companyId: seeded.companyId };
const execution = collectRunFailureDiagnostics({ ...run, errorCode: "setup_failed", resultJson: null,
} as typeof heartbeatRuns.$inferSelect, { error: caught, phase: "setup" }).execution;
if (mode === "worker") {
expect(execution).toMatchObject({ environmentAcquisitionPhase: "create", environmentAcquisitionProviderState: "error" });
expect(readTransientLeaseAcquisitionFailure(caught, run)).toEqual({ reason: "provider_start_failed" });
} else {
expect(execution).not.toHaveProperty("environmentAcquisitionPhase");
expect(readTransientLeaseAcquisitionFailure(caught, run)).toBeNull();
}
expect(await db.select().from(environmentLeases).where(eq(environmentLeases.status, "pending_cleanup"))).toHaveLength(0);
});
it.each(["deleted receipt lost", "observation response lost", "journal write failed", "foreign observation"])(
"journals observed creation identity before deletion and recovers: %s", async (fault) => {
const seeded = await seedReusablePluginSandboxLease();
@@ -154,6 +154,36 @@ describe("readTransientLeaseAcquisitionFailure", () => {
expect(readTransientLeaseAcquisitionFailure(creationFailure(12_000), run)).toEqual({ reason: "acquisition_incomplete" });
});
it("retries a sandbox the provider could not start and already removed", () => {
const error = new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.WORKER_ERROR,
message: "Daytona could not start the sandbox; it was removed.",
data: { schema: "paperclip/environment-acquire-error/v1",
acquisitionDiagnostic: { phase: "create", elapsedMs: 38_000, budgetMs: 90_000, providerState: "error" } } });
captureEnvironmentAcquisitionDiagnostic(error, { companyId: run.companyId, environmentId: "env-1", runId: run.id },
readEnvironmentAcquisitionDiagnostic(error));
expect(readTransientLeaseAcquisitionFailure(new Error("lease", { cause: error }), run)).toEqual({ reason: "provider_start_failed" });
const unavailable = new EnvironmentProviderUnavailableError({ environmentName: "Daytona warm", driver: "sandbox",
evidence: { reason: "provider_start_failed" }, cause: error });
expect(unavailable.providerFailure).toMatchObject({ retryability: "transient", providerCode: "provider_start_failed" });
expect(unavailable.message).toContain("the provider could not start the sandbox");
});
it("names a later-step failure an unfinished acquisition, not a start failure", () => {
const error = new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.WORKER_ERROR, message: "x",
data: { schema: "paperclip/environment-acquire-error/v1",
acquisitionDiagnostic: { phase: "shell", elapsedMs: 38_000, budgetMs: 90_000, providerState: "error" } } });
captureEnvironmentAcquisitionDiagnostic(error, { companyId: run.companyId, environmentId: "env-1", runId: run.id },
readEnvironmentAcquisitionDiagnostic(error));
expect(readTransientLeaseAcquisitionFailure(error, run)).toEqual({ reason: "acquisition_incomplete" });
});
it("does not retry a removed sandbox whose step was never recorded for this run", () => {
const error = new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.WORKER_ERROR, message: "x",
data: { schema: "paperclip/environment-acquire-error/v1",
acquisitionDiagnostic: { phase: "create", elapsedMs: 38_000, budgetMs: 90_000, providerState: "error" } } });
expect(readTransientLeaseAcquisitionFailure(error, run)).toBeNull();
});
it("ignores an unfinished-creation diagnostic recorded for another run", () => {
expect(readTransientLeaseAcquisitionFailure(creationFailure(12_000, "run-2"), run)).toBeNull();
});
@@ -19,7 +19,7 @@ export const ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE = "environment_provider_unava
* the provider reported a provisional creation that did not finish, or a
* control-plane database query failed while the lease was taken.
*/
type TransientLeaseFailureReason = "acquisition_incomplete" | "database_error";
type TransientLeaseFailureReason = "acquisition_incomplete" | "provider_start_failed" | "database_error";
type TransientProviderEvidence = { httpStatus?: number; errorCode?: string; acquisitionTimedOut?: true; reason?: TransientLeaseFailureReason };
function own(value: unknown, key: string): unknown {
@@ -104,7 +104,8 @@ export function readTransientLeaseAcquisitionFailure(
for (let cause = error, depth = 0; cause && depth < 5; cause = own(cause, "cause"), depth++) {
const acquisition = getEnvironmentAcquisitionDiagnostic(cause, run);
if (acquisition) {
return acquisition.elapsedMs >= acquisition.budgetMs ? { acquisitionTimedOut: true } : { reason: "acquisition_incomplete" };
if (acquisition.elapsedMs >= acquisition.budgetMs) return { acquisitionTimedOut: true };
return { reason: acquisition.phase === "create" && acquisition.providerState === "error" ? "provider_start_failed" : "acquisition_incomplete" };
}
if (cause instanceof JsonRpcCallError && cause.code === PLUGIN_RPC_ERROR_CODES.TIMEOUT) return { acquisitionTimedOut: true };
}
@@ -113,6 +114,7 @@ export function readTransientLeaseAcquisitionFailure(
const REASON_LABELS: Record<TransientLeaseFailureReason, string> = {
acquisition_incomplete: "sandbox creation did not finish",
provider_start_failed: "the provider could not start the sandbox",
database_error: "a database query failed while recording the lease",
};
+3 -2
View File
@@ -2047,8 +2047,9 @@ function createSandboxEnvironmentDriver(
resolvePluginSandboxRpcTimeoutMs(workerConfig, pluginProvider.resolved.driver.defaultAcquireTimeoutMs));
} catch (error) {
const cleanup = readEnvironmentCreationCleanupError(error);
const diagnostic = error instanceof JsonRpcCallError && cleanup?.companyId === input.companyId &&
cleanup.environmentId === input.environment.id && cleanup.runId === identity.runId
const ownsReceipt = cleanup === null || (cleanup.companyId === input.companyId &&
cleanup.environmentId === input.environment.id && cleanup.runId === identity.runId);
const diagnostic = error instanceof JsonRpcCallError && ownsReceipt
? readEnvironmentAcquisitionDiagnostic(error) : null;
captureEnvironmentAcquisitionDiagnostic(error, {
companyId: input.companyId, environmentId: input.environment.id, runId: identity.runId,