mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - On a Daytona sandbox environment, every new task run first creates a fresh sandbox > - When one Daytona create did not start, the plugin waited the full 300 s acquisition budget > - The run then failed `setup_failed` with `lease_acquire_failed`. No agent work had started, but nothing retried it > - On staging this hit two agents in 15 minutes. Each attempt used about 5 minutes, and the task stopped until a person pressed Retry. The manual Retry then got a sandbox in 28 s > - This pull request ends a stuck create early, adopts a create that started late, and routes the timeout to the bounded bootstrap retry that already exists for transient provider refusals > - The benefit is that a stuck create costs about 90 s and recovers by itself. The task shows Retry only after the bounded attempts are used up ## Linked Issues or Issue Description Refs #20626 (bounded resume of a reusable sandbox), Refs #20212 (wait out a Daytona state change), Refs #20609 (force a stuck stop). **What happened?** A run on a Daytona sandbox environment failed after 307 s with `Failed to acquire lease for environment "Paperclip Sandbox" (sandbox): Daytona lease acquisition timed out; allocation cleanup is pending`. The run recorded `errorCode: setup_failed`, `failureCause: lease_acquire_failed`, and `executionRecovery: { kind: "bootstrap", providerWorkStarted: false }`. The task became blocked with Retry. No automatic retry was scheduled. **Steps to reproduce** 1. Use a Daytona sandbox environment with the default `timeoutMs` (300,000). 2. Start a run while Daytona keeps the new sandbox in `creating`. 3. See that the run waits 300 s, fails `setup_failed`, and is not retried. **Expected behavior** A create that has not started before provider work must not cost 5 minutes. The run must retry by itself within bounds. A late sandbox that has started must be used. A sandbox that was abandoned must be removed. **Root cause** - `packages/plugins/sandbox-providers/daytona/src/plugin.ts` (`onEnvironmentAcquireLease`): the only bound on the create step was the whole acquisition budget (`timeoutMs`, default 300 s). - `server/src/services/environment-provider-unavailable-error.ts` (`readTransientEnvironmentProviderFailure`): only an HTTP status or errno on the create call counted as transient. The timeout envelope carries neither, so the run fell through to generic `setup_failed`, which `isTransientWorkspacePreparationCode` does not retry. ## What Changed - Daytona: one fresh acquisition is now capped at 180 s. A create that has not returned after 90 s is looked up once by its creation name. A sandbox in `started` is adopted, and setup continues. A sandbox in `pending_build`, `building_snapshot`, or `pulling_snapshot` keeps waiting inside the cap. Any other state ends the attempt with the existing cleanup envelope and an acquisition diagnostic whose elapsed time reached the expired budget. - Daytona: a late SDK failure no longer runs the compensating delete for a sandbox that was adopted. - Host: `readTransientEnvironmentProviderFailure` reads the validated acquisition diagnostic. An acquisition whose first failure was its own deadline is transient. The run fails with `environment_provider_unavailable`, category `timeout`, and uses the bounded bootstrap retry (30 s, then 60 s). An earlier setup failure stays `setup_failed`. Message text is never parsed. - Host: the task reason for this case is plain: "The sandbox for ... did not finish starting in time. No agent work had started. Paperclip retries automatically ...". - Docs: `doc/execution-semantics.md` and `doc/plugins/SANDBOX_PROVIDER_CAPABILITIES.md`. ## Verification Red before the fix, green after: - `server/src/services/heartbeat/run-completion.test.ts` "routes a fresh sandbox create that timed out through bounded bootstrap recovery" uses the staging error shape through the orchestrator. Red: `errorCode` was `setup_failed`. Green: `environment_provider_unavailable`, category `timeout`, bounded retry scheduled. The `started` and `early_failure` cases stay `setup_failed` with no retry. - `server/src/services/environment-provider-unavailable-error.test.ts`: deadline diagnostic is transient (red, then green). An early failure is not. The task message is plain. - `packages/plugins/sandbox-providers/daytona/src/plugin.test.ts`: "ends a create still pending at 90 seconds" (red: still pending at 90 s; green) and "adopts a create that started by 90 seconds" (red; green, and a late SDK rejection does not delete it). "keeps waiting for a create that is still preparing its snapshot" is a safety case. - Existing tests that encoded the 300 s acquisition budget now use the 180 s cap. Commands run locally: - `npx vitest run src/plugin.test.ts` (Daytona): 414 passed. - `npx vitest run src/services/heartbeat/run-completion.test.ts src/services/environment-provider-unavailable-error.test.ts src/__tests__/environment-acquisition-diagnostics.test.ts src/__tests__/task-interruption-query.test.ts`: 229 tests passed. The run-completion file reports an `afterAll` embedded-Postgres teardown timeout. The same timeout occurs on unchanged `origin/master`, so it is a local platform fault. - Daytona plugin `pnpm typecheck` and server `pnpm typecheck`: pass. - `pnpm verify`: asked for broader CI coverage, because `doc/execution-semantics.md` is a global CI input. Required CI supplies that coverage. Upgrade safety: nothing new is persisted. A task already blocked by this failure keeps `canRetry: true`. Its Retry starts a new run with a new creation record, and that run uses the new path. On staging, the manual Retry of the affected task got a fresh lease in 28 s. The creation record of the earlier attempt stays with the creation reconciler. Not verified: a live Daytona create that stays in `creating`. That cannot be forced on demand. ## Risks - A legitimately slow create that stays in `creating` or `starting` for more than 90 s now ends the attempt and retries. Snapshot preparation states still wait inside the 180 s cap. Staging creates normally take about 20 to 30 s. - An abandoned create is removed by the creation reconciler. It can stay alive until that reconciler settles the record (the existing settle window). This PR does not change that window. - Other sandbox providers get the host classification only when they send the same acquisition diagnostic. ## Model Used Claude Opus 5.5 (claude-opus-5-5), Anthropic, used through Claude Code with tool use (shell, file edits, browser for read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
103 lines
5.7 KiB
TypeScript
103 lines
5.7 KiB
TypeScript
import { readEnvironmentAcquireErrorDiagnostic, readEnvironmentAcquisitionDiagnostic } from "@paperclipai/plugin-sdk";
|
|
import type { ProviderFailure } from "@paperclipai/shared";
|
|
|
|
/**
|
|
* Setup failure code for a lease acquisition that the environment provider
|
|
* refused with a transient status (HTTP 408/425/429/5xx or a connection-level
|
|
* errno), or that reached its own acquisition deadline, before any agent work
|
|
* started. The run is retried with the bounded
|
|
* bootstrap budget. A provisional allocation whose cleanup could not be
|
|
* confirmed stays owned by its own creation record, which the cleanup
|
|
* reconciler settles independently; every retry is a new creation attempt
|
|
* with its own idempotent identity, so it can neither reuse nor leak it.
|
|
*/
|
|
export const ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE = "environment_provider_unavailable";
|
|
|
|
const TRANSIENT_HTTP_STATUSES = new Set([408, 425, 429, 500, 502, 503, 504]);
|
|
// Connection-level failures only. ENOTFOUND (a wrong host) and ABORT_ERR (our
|
|
// own cancellation) are excluded: retrying cannot repair either.
|
|
const TRANSIENT_ERROR_CODES = new Set([
|
|
"ECONNRESET", "ECONNREFUSED", "ETIMEDOUT", "EPIPE", "EAI_AGAIN", "ENETUNREACH",
|
|
"UND_ERR_CONNECT_TIMEOUT", "UND_ERR_SOCKET",
|
|
]);
|
|
|
|
type TransientProviderEvidence = { httpStatus?: number; errorCode?: string; acquisitionTimedOut?: true };
|
|
|
|
function own(value: unknown, key: string): unknown {
|
|
try {
|
|
return value && typeof value === "object" ? Object.getOwnPropertyDescriptor(value, key)?.value : undefined;
|
|
} catch { return undefined; }
|
|
}
|
|
|
|
/**
|
|
* Read transient provider evidence from structured fields only: the validated
|
|
* plugin RPC envelope, or the status/errno an in-process provider attached to
|
|
* its error. An acquisition whose first failure was its own deadline is
|
|
* transient too. Message text is never parsed.
|
|
*/
|
|
export function readTransientEnvironmentProviderFailure(error: unknown): TransientProviderEvidence | null {
|
|
for (let cause = error, depth = 0; cause && depth < 5; cause = own(cause, "cause"), depth++) {
|
|
const acquisition = readEnvironmentAcquisitionDiagnostic(cause);
|
|
if (acquisition && acquisition.elapsedMs >= acquisition.budgetMs) return { acquisitionTimedOut: true };
|
|
const envelope = readEnvironmentAcquireErrorDiagnostic(cause);
|
|
const statusValue = envelope?.httpStatus ?? [own(cause, "status"), own(cause, "statusCode")]
|
|
.find(value => Number.isInteger(value));
|
|
const codeValue = envelope?.errorCode ?? own(cause, "code");
|
|
const httpStatus = typeof statusValue === "number" && TRANSIENT_HTTP_STATUSES.has(statusValue) ? statusValue : undefined;
|
|
const errorCode = typeof codeValue === "string" && TRANSIENT_ERROR_CODES.has(codeValue) ? codeValue : undefined;
|
|
if (httpStatus !== undefined || errorCode !== undefined) {
|
|
return { ...(httpStatus !== undefined ? { httpStatus } : {}), ...(errorCode !== undefined ? { errorCode } : {}) };
|
|
}
|
|
// A definite non-transient status is authoritative for this layer.
|
|
if (typeof statusValue === "number" && statusValue >= 400) return null;
|
|
}
|
|
return null;
|
|
}
|
|
|
|
function category(evidence: TransientProviderEvidence): ProviderFailure["category"] {
|
|
if (evidence.httpStatus === 429) return "rate_limit";
|
|
if (evidence.acquisitionTimedOut || evidence.httpStatus === 408 || evidence.httpStatus === 504
|
|
|| evidence.errorCode === "ETIMEDOUT" || evidence.errorCode === "UND_ERR_CONNECT_TIMEOUT") return "timeout";
|
|
return "overloaded";
|
|
}
|
|
|
|
/** A lease acquisition the environment provider refused transiently. */
|
|
export class EnvironmentProviderUnavailableError extends Error {
|
|
readonly code = ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE;
|
|
readonly httpStatus: number | null;
|
|
readonly errorCode: string | null;
|
|
/** Read by the setup-failure path and shown on the task as the reason. */
|
|
readonly providerFailure: ProviderFailure;
|
|
|
|
constructor(input: { environmentName: string; driver: string; evidence: TransientProviderEvidence; cause: unknown }) {
|
|
const status = input.evidence.httpStatus !== undefined ? `HTTP ${input.evidence.httpStatus}` : input.evidence.errorCode;
|
|
super(status
|
|
? `Environment provider for "${input.environmentName}" was temporarily unavailable (${status}).`
|
|
: `Environment provider for "${input.environmentName}" did not finish starting the sandbox in time.`, { cause: input.cause });
|
|
this.name = "EnvironmentProviderUnavailableError";
|
|
this.httpStatus = input.evidence.httpStatus ?? null;
|
|
this.errorCode = input.evidence.errorCode ?? null;
|
|
this.providerFailure = {
|
|
schema: "paperclip.provider-failure.v1",
|
|
provider: /^[a-zA-Z0-9_.:/-]{1,80}$/.test(input.driver) ? input.driver : "environment",
|
|
phase: "startup",
|
|
category: category(input.evidence),
|
|
message: (status
|
|
? `The provider for "${input.environmentName}" was temporarily unavailable while Paperclip was starting it (${status}). `
|
|
: `The sandbox for "${input.environmentName}" did not finish starting in time. `)
|
|
+ "No agent work had started. Paperclip retries automatically; if those retries are used up, use Retry once the provider is available.",
|
|
...(input.evidence.httpStatus !== undefined ? { httpStatus: input.evidence.httpStatus } : {}),
|
|
...(input.evidence.errorCode !== undefined ? { providerCode: input.evidence.errorCode } : {}),
|
|
retryability: "transient",
|
|
};
|
|
}
|
|
}
|
|
|
|
/** Find a transient provider refusal through the orchestrator's error wrappers. */
|
|
export function findEnvironmentProviderUnavailableError(error: unknown): EnvironmentProviderUnavailableError | null {
|
|
for (let cause = error, depth = 0; cause && depth < 4; cause = (cause as { cause?: unknown }).cause, depth++) {
|
|
if (cause instanceof EnvironmentProviderUnavailableError) return cause;
|
|
}
|
|
return null;
|
|
}
|