mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-19 16:29:42 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task that runs in a sandbox environment first gets a lease from the environment provider (for example Daytona) > - Sometimes the provider creates the sandbox but cannot start it. Daytona then puts the sandbox in its `error` state > - The Daytona plugin deletes that failed sandbox and rethrows the provider error. The error has no HTTP status and no errno, so the host cannot tell that it is a provider-side failure > - The run fails with `setup_failed`, and the task rests on "Retry when ready". No agent work had started, and a fresh sandbox works, so a person must click Retry for something the system can do itself > - Recovery must just work, and the target for runs that need Retry is 0% > - This pull request makes the plugin report a removed sandbox that the provider could not start as structured evidence (the step and the provider state). The host classifies it as a transient provider failure. The task then uses the existing bounded retries and the provider-outage wait from #22139 > - The benefit is that a provider start failure delays the task by seconds instead of stranding it on Retry ## Linked Issues or Issue Description Refs #22139 (the provider-outage wait lane that this failure now reaches). Refs #22128 (setup failure after work; this failure happens before any work, so that path did not apply). **What happened?** In a restart-chaos wave with graceful release swaps on 2026-11-18, two tasks started a run at 14:51:15Z and 14:51:24Z. Both runs failed at 14:51:53Z with `setup_failed` and `failureCause: lease_acquire_failed`. The cause was `Sandbox <id> failed to start with status: error, error reason: null` from the Daytona SDK, for two different sandboxes at the same moment. Both tasks rested on Retry. A Retry click started each task again, and both tasks completed. **Expected behavior** A sandbox that the provider cannot start is a provider failure before any agent work. Paperclip deletes it and tries again with a new sandbox, with no Retry. If the provider keeps failing, the task waits for the provider without a Retry, and a person gets a working Retry only after the 30-minute ceiling. **Steps to reproduce** 1. Make the Daytona create call fail while the new sandbox is in the provider's `error` state. 2. Assign a task to an agent that uses that environment. 3. The run fails with `setup_failed`, and the task rests on Retry. ## What Changed - Plugin SDK (`environment-creation-cleanup.ts`, `environment-acquire-error.ts`, `index.ts`): new `PluginEnvironmentAcquisitionError`. It carries the acquisition diagnostic (step, elapsed time, budget, provider state) for a failed allocation that the provider already removed, with no cleanup envelope. The acquire RPC error data (`paperclip/environment-acquire-error/v1`) now carries this diagnostic. `readEnvironmentAcquisitionDiagnostic` reads it from that envelope. The existing cleanup envelope path is unchanged. No provider text crosses the RPC. - Daytona plugin (`plugin.ts`): `destroyFailedCreation` returns the state of the sandbox that it deleted. When a create fails without an HTTP status and the deleted sandbox was in the `error` state, the plugin throws `PluginEnvironmentAcquisitionError` with `providerState: "error"`. A `build_failed` sandbox (a deterministic snapshot build failure) and any other state keep the original error. - Server (`environment-runtime.ts`): the acquire path records the diagnostic for this run also when the error has no cleanup envelope. A cleanup envelope for another run, company or environment still records nothing. - Server (`environment-provider-unavailable-error.ts`): an acquisition diagnostic with `providerState: "error"` is the transient reason `provider_start_failed` ("the provider could not start the sandbox"). It becomes `EnvironmentProviderUnavailableError`, so the run uses the two bounded retries and then the provider-outage wait lane (one probe per environment, 30-minute ceiling). - `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pins for the changed SDK sources (generated by `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --write`). Module boundaries: the change to the shared SDK contract is additive. The Daytona plugin owns the provider state. The host reads only the closed diagnostic fields. A plugin that does not use the new class behaves as before. ## Verification - `packages/plugins/sandbox-providers/daytona`: `npx vitest run src/plugin.test.ts -t "failed sandbox creation cleanup"` (37 passed). New: "reports a sandbox that failed to start as a removed allocation with its observed step" (red before the plugin change, green after) and "keeps the create error when the removed sandbox was build_failed / started". - `packages/plugins/sdk`: `npx vitest run tests/environment-acquire-error.test.ts tests/environment-creation-cleanup.test.ts` (50 passed). New: the diagnostic crosses the real worker RPC without provider text, and forged or unstructured envelopes give no diagnostic. - `server`: `npx vitest run src/services/environment-provider-unavailable-error.test.ts src/__tests__/environment-acquisition-diagnostics.test.ts` (74 passed). New: the reason is `provider_start_failed` with a transient provider failure, and a diagnostic that was not recorded for this run is not retried. - `server`: `npx vitest run src/__tests__/environment-runtime.test.ts -t "provider could not start|retains uncertain plugin creation"` (12 passed). New: "records the step of a sandbox the provider could not start and removed" for a worker error (red before the runtime change, green after) and for a forged error (not recorded). - `server`: `npx vitest run src/services/__tests__/run-failure-diagnostics.test.ts src/__tests__/heartbeat-process-recovery.test.ts -t "provider|acquisition|diagnostic"` (167 passed). - Typecheck: `server` `tsc --noEmit`, `pnpm run typecheck` in `packages/plugins/sdk` and in the Daytona plugin all pass. - `pnpm verify` exits 2 (the SDK input registry is a global CI input). Required CI supplies the broad coverage. ## Risks - **Automatic retries for a deterministic start failure.** If a provider image or configuration made every sandbox go to `error`, the task now waits in the provider-outage lane instead of resting on Retry at once. The lane uses one probe per environment on a 1, 2, then 5 minute backoff, and it hands the task to a person with a working Retry after 30 minutes. The aggregate automatic-run brake is unchanged. `build_failed` is not retried by this change. - **Shared SDK contract.** The new field is additive. An older host ignores it, and the run fails as before. - No database migration. No UI change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in Claude Code with tool use (shell, file edits) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
96 lines
5.4 KiB
TypeScript
96 lines
5.4 KiB
TypeScript
import { createInterface } from "node:readline";
|
|
import { PassThrough } from "node:stream";
|
|
import { describe, expect, it } from "vitest";
|
|
import { definePlugin } from "../src/define-plugin.js";
|
|
import { readEnvironmentAcquireErrorDiagnostic } from "../src/environment-acquire-error.js";
|
|
import {
|
|
PluginEnvironmentAcquisitionError,
|
|
PluginEnvironmentCreationCleanupError,
|
|
readEnvironmentAcquisitionDiagnostic,
|
|
readEnvironmentCreationCleanupError,
|
|
} from "../src/environment-creation-cleanup.js";
|
|
import { createRequest, isJsonRpcResponse, JsonRpcCallError, parseMessage, serializeMessage, type JsonRpcResponse } from "../src/protocol.js";
|
|
import { startWorkerRpcHost } from "../src/worker-rpc-host.js";
|
|
|
|
async function acquire(error: unknown) {
|
|
const stdin = new PassThrough();
|
|
const stdout = new PassThrough();
|
|
const lines = createInterface({ input: stdout });
|
|
const worker = startWorkerRpcHost({ stdin, stdout, plugin: definePlugin({
|
|
async setup() {},
|
|
async onEnvironmentAcquireLease() { throw error; },
|
|
}) });
|
|
try {
|
|
const response = new Promise<JsonRpcResponse>((resolve) => {
|
|
lines.on("line", (line) => {
|
|
const message = parseMessage(line);
|
|
if (isJsonRpcResponse(message)) resolve(message);
|
|
});
|
|
});
|
|
stdin.write(serializeMessage(createRequest("environmentAcquireLease", {}, "acquire-test")));
|
|
const result = await response;
|
|
if (!("error" in result) || !result.error) throw new Error("Expected RPC error");
|
|
return new JsonRpcCallError(result.error);
|
|
} finally {
|
|
worker.stop(); lines.close(); stdin.destroy(); stdout.destroy();
|
|
}
|
|
}
|
|
|
|
const cleanup = {
|
|
providerLeaseId: "paperclip-create-05f81d77", companyId: "company-1", environmentId: "environment-1",
|
|
attemptId: "05f81d77", accountFingerprint: "a".repeat(64), labels: { "paperclip-provider": "daytona" },
|
|
};
|
|
|
|
describe("environment acquisition refusal diagnostics", () => {
|
|
it("carries a create status across the real acquire RPC without provider text", async () => {
|
|
const providerError = Object.assign(new Error("Bad gateway at https://private.example/api"), { statusCode: 502 });
|
|
const callError = await acquire(providerError);
|
|
expect(readEnvironmentAcquireErrorDiagnostic(callError)).toEqual({ httpStatus: 502 });
|
|
expect(JSON.stringify(callError.data)).not.toContain("private.example");
|
|
});
|
|
|
|
it("reports the create status, not the compensating lookup's, and keeps the cleanup envelope", async () => {
|
|
const createError = Object.assign(new Error("create failed"), { statusCode: 502 });
|
|
const lookupError = Object.assign(new Error("not found"), { statusCode: 404 });
|
|
const callError = await acquire(new PluginEnvironmentCreationCleanupError([createError, lookupError],
|
|
"Daytona sandbox creation failed (HTTP 502); cleanup could not be confirmed for paperclip-create-05f81d77", cleanup));
|
|
expect(readEnvironmentAcquireErrorDiagnostic(callError)).toEqual({ httpStatus: 502 });
|
|
// The ownership envelope that drives cleanup reconciliation is unchanged.
|
|
expect(readEnvironmentCreationCleanupError(callError)).toMatchObject({ providerLeaseId: cleanup.providerLeaseId, attemptId: cleanup.attemptId });
|
|
});
|
|
|
|
it("carries a closed connection errno and omits unstructured failures", async () => {
|
|
expect(readEnvironmentAcquireErrorDiagnostic(await acquire(Object.assign(new Error("reset"), { code: "ECONNRESET" }))))
|
|
.toEqual({ errorCode: "ECONNRESET" });
|
|
const plain = await acquire(new Error("Daytona sandbox creation failed (HTTP 502)"));
|
|
expect(plain.data).toBeUndefined();
|
|
expect(readEnvironmentAcquireErrorDiagnostic(plain)).toBeNull();
|
|
});
|
|
|
|
it("carries the observed step of a removed failed allocation without provider text", async () => {
|
|
const callError = await acquire(new PluginEnvironmentAcquisitionError("Daytona could not start the sandbox.",
|
|
{ phase: "create", elapsedMs: 38_000, budgetMs: 90_000, providerState: "error" },
|
|
new Error("Sandbox private-sandbox-id failed to start with status: error")));
|
|
expect(readEnvironmentAcquisitionDiagnostic(callError)).toEqual({ phase: "create", elapsedMs: 38_000, budgetMs: 90_000, providerState: "error" });
|
|
expect(readEnvironmentCreationCleanupError(callError)).toBeNull();
|
|
expect(readEnvironmentAcquireErrorDiagnostic(callError)).toBeNull();
|
|
expect(JSON.stringify(callError.data)).not.toContain("private-sandbox-id");
|
|
});
|
|
|
|
it("reads no acquisition step from an unstructured or forged refusal", async () => {
|
|
expect(readEnvironmentAcquisitionDiagnostic(await acquire(new Error("plain")))).toBeNull();
|
|
const forged = (data: unknown) => Object.assign(new Error("x"), { data });
|
|
expect(readEnvironmentAcquisitionDiagnostic(forged({ schema: "other",
|
|
acquisitionDiagnostic: { phase: "create", elapsedMs: 1, budgetMs: 2 } }))).toBeNull();
|
|
expect(readEnvironmentAcquisitionDiagnostic(forged({ schema: "paperclip/environment-acquire-error/v1",
|
|
acquisitionDiagnostic: { phase: "anything", elapsedMs: 1, budgetMs: 2 } }))).toBeNull();
|
|
});
|
|
|
|
it("rejects forged or out-of-range envelopes", () => {
|
|
const forged = (data: unknown) => Object.assign(new Error("x"), { data });
|
|
expect(readEnvironmentAcquireErrorDiagnostic(forged({ schema: "other", providerDiagnostic: { httpStatus: 502 } }))).toBeNull();
|
|
expect(readEnvironmentAcquireErrorDiagnostic(forged({ schema: "paperclip/environment-acquire-error/v1",
|
|
providerDiagnostic: { httpStatus: 9000, errorCode: "EVIL" } }))).toBeNull();
|
|
});
|
|
});
|