mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 18:26:40 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A task run takes an environment lease (for example a Daytona sandbox) before the agent starts. No provider work happens before that lease exists. > - Only a provider HTTP status or a connection errno counted as a transient lease failure. #20828 (merged while this was open) added the Daytona acquisition deadline. A database error while the lease was recorded, a plugin RPC deadline, and an unfinished provisional creation still did not count. > - Those failures became a generic `setup_failed` with no bounded retry. And when the bounded retries of any `environment_provider_unavailable` failure were used up, recovery still put the task on a board-owned `stranded_assigned_issue` action. > - A hosted instance saw this for six runs in 45 minutes when concurrent runs rose from about 8 to about 20. Two tasks stayed blocked and waited for the board. > - This pull request classifies every lease acquisition failure that has structured transient evidence as `environment_provider_unavailable`. When the bounded retries are used up, the task rests on a system-owned action and a working Retry. > - It also adds the restart-chaos fault EP1, so the harness covers lease acquisition failures from now on. > - The benefit is that a busy or slow sandbox provider no longer strands tasks on the board. ## Linked Issues or Issue Description No public issue exists. **What happened?** Five runs failed with `Failed to acquire lease for environment "Paperclip Sandbox" (sandbox): Daytona lease acquisition timed out; allocation cleanup is pending` after 5 to 6 minutes in `preparing`. One run failed with `Database query failed (lease_acquire_failed)` within 10 seconds. Every run recorded `executionRecovery: { kind: "bootstrap", providerWorkStarted: false }`, but the error code was `setup_failed`. No run got a bounded retry. Two tasks were left `blocked` with a board-owned `stranded_assigned_issue` recovery action. **Expected behavior** A lease acquisition failure happens before any provider work. The run must retry with a bounded, visible backoff. When the bound is used up, the task must rest with a Retry that works, and no board-owned recovery action. **Steps to reproduce** 1. Make lease acquisition fail with the provider's acquisition deadline (the validated acquisition diagnostic that the environment runtime records), or with a database driver error. 2. Let the run start. 3. Before this change, the run fails with `setup_failed`, no retry is scheduled, and recovery creates a board-owned `stranded_assigned_issue` action. ## What Changed **How this PR and #20828 divide the problem.** #20828 (merged) makes the Daytona plugin end a stuck create early and classifies the acquisition-deadline envelope as `environment_provider_unavailable`. This PR does not change the Daytona plugin or that deadline path. It adds only what #20828 leaves out: - a lease-acquisition failure that is not a create timeout: the plugin RPC deadline (`-32003`) and an unfinished provisional creation recorded for this run; - a transient database error while the lease is recorded; - the exhausted state: a system-owned recovery action and Retry instead of a board-owned `stranded_assigned_issue`, for every `environment_provider_unavailable` cause (including #20828's); - the chaos fault EP1 (EP1, EP1N, EP1D). - `readTransientLeaseAcquisitionFailure` (`server/src/services/environment-provider-unavailable-error.ts`) classifies a lease acquisition failure as transient when it has structured evidence: the provider status or errno (as before), the run's own validated acquisition diagnostic (deadline or unfinished provisional creation), the plugin RPC deadline, or a transient database error (a postgres.js connection code, or SQLSTATE class 08 or 53, 57P01-57P03, 57014, 55P03, 40001, 40P01). A deterministic SQL error (syntax, schema, constraint) is not retried. A definite non-transient status (for example HTTP 401 or 404) is still `setup_failed`. Message text is not parsed. - `environment-run-orchestrator.ts` uses this classifier, so these failures get `environment_provider_unavailable` and the existing bounded bootstrap retry (30 s, then 60 s, with jitter). The task shows "retry scheduled" while it waits. - When the retries are used up, recovery uses the cause `environment_provider_unavailable` with a system-owned action and no board routing. The notice says "Retry when ready" and does not ask for a board decision. The task's Retry starts a new acquisition. - Restart-chaos fault EP1: a test-mode server seam (`environment-lease-fault-injection.ts`, read from `PAPERCLIP_CHAOS_LEASE_FAULT_CONTROL`, ignored in production) throws the staging error shapes before any provider call. `tests/restart-chaos/env-provider-wave.ts` runs EP1 (deadline twice, then success), EP1N (deadline forever, then Retry after disarm) and EP1D (database error once). The oracle fails on any `setup_failed` run, any board-owned recovery action, a wrong retry count, or a task that does not complete. - Docs: `doc/execution-semantics.md` and `tests/restart-chaos/README.md`. ## Verification - RED on `origin/master` (`1d67e309e0`, before #20828): the three new tests in `heartbeat-process-recovery.test.ts` ("transient sandbox provider refusal during lease acquisition") fail with the staging result: `expected [ [ 'failed', 'setup_failed' ] ] to deeply equal [ [ 'failed', 'environment_provider_unavailable' ], [ 'scheduled_retry', null ] ]`. On current master the deadline now retries (#20828), but the exhausted case still rests on a board-owned `stranded_assigned_issue` action (the existing HTTP 502 exhaustion test showed `ownerType: "board"` before the recovery change), and the database error is still `setup_failed`. - GREEN: `pnpm --filter @paperclipai/server exec vitest run src/__tests__/heartbeat-process-recovery.test.ts -t "lease acquisition"` passes 5 of 5. This includes the deadline and the database error that retry once and then complete, and the deadline that never clears. That task rests `blocked` with a system-owned action, `canRetry: true`, no execution blocker, and Retry completes it. - Unit tests: `environment-provider-unavailable-error.test.ts` + `environment-lease-fault-injection.test.ts` (56), `tests/restart-chaos/env-provider-faults.test.ts` (6) pass. - Related suites pass: `environment-run-orchestrator`, `task-interruption-query`, `issue-recovery-actions`, `issue-recovery-deadlines`, `stranded-notice`, `heartbeat-native-resume-lease-failure`. - `server` and `tests/restart-chaos` typecheck pass. - Chaos waves (local, Codex `gpt-6.1-sol`, head `1bd387e98f`; the later head changes only database classification and the notice), all pass: - EP1 `w-20261111T171803-s71-ep`, 2 tasks: each had exactly 2 `environment_provider_unavailable` runs, then succeeded with no user action, no board-owned recovery action, brief oracle pass. - EP1D `w-20261111T172850-s73-ep`, 1 task: 1 failed run on the database error, then succeeded. - EP1N `w-20261111T173346-s74-ep` (and `w-20261111T162752-s72-ep` before the rebase): 3 failed runs, the task rested `blocked` with `canRetry: true` and no board-owned action; after disarm, Retry completed it. - No Daytona wave was run. The fault is injected in the server before the provider call, so a local instance reaches the same retry and recovery code. ## Risks - More failures now retry, all bounded. Only transient SQLSTATEs retry. The staging database error recorded no SQLSTATE. If it was a deterministic SQL error, it still fails `setup_failed` and goes to the board as before; that is intended, because a retry cannot fix it. - Each retry is a new creation attempt with its own creation record. A provisional allocation whose cleanup is pending stays with its record and the creation reconciler, as before. - The retry bound is the shared bootstrap bound (two retries). A provider outage longer than about 15 minutes still rests on Retry, but no longer on a board action. - The fault seam reads a file only when the environment variable is set, and never when `NODE_ENV=production`. - Saved state from the previous release: no persisted shape changes. A task already stranded on a board action keeps it (`preserveExistingOwner`). New failures use the new cause. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic) through Claude Code, model ID `claude-opus-5-5`, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
175 lines
9.7 KiB
TypeScript
175 lines
9.7 KiB
TypeScript
import { JsonRpcCallError, PLUGIN_RPC_ERROR_CODES, readEnvironmentAcquireErrorDiagnostic, readEnvironmentAcquisitionDiagnostic } from "@paperclipai/plugin-sdk";
|
|
import type { ProviderFailure } from "@paperclipai/shared";
|
|
import { getEnvironmentAcquisitionDiagnostic } from "./environment-acquisition-diagnostics.js";
|
|
|
|
/**
|
|
* Setup failure code for a lease acquisition that the environment provider
|
|
* refused with a transient status (HTTP 408/425/429/5xx or a connection-level
|
|
* errno), or that reached its own acquisition deadline, before any agent work
|
|
* started. The run is retried with the bounded
|
|
* bootstrap budget. A provisional allocation whose cleanup could not be
|
|
* confirmed stays owned by its own creation record, which the cleanup
|
|
* reconciler settles independently; every retry is a new creation attempt
|
|
* with its own idempotent identity, so it can neither reuse nor leak it.
|
|
*/
|
|
export const ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE = "environment_provider_unavailable";
|
|
|
|
const TRANSIENT_HTTP_STATUSES = new Set([408, 425, 429, 500, 502, 503, 504]);
|
|
// Connection-level failures only. ENOTFOUND (a wrong host) and ABORT_ERR (our
|
|
// own cancellation) are excluded: retrying cannot repair either.
|
|
const TRANSIENT_ERROR_CODES = new Set([
|
|
"ECONNRESET", "ECONNREFUSED", "ETIMEDOUT", "EPIPE", "EAI_AGAIN", "ENETUNREACH",
|
|
"UND_ERR_CONNECT_TIMEOUT", "UND_ERR_SOCKET",
|
|
]);
|
|
|
|
/**
|
|
* `reason` names evidence that has neither a provider status nor a deadline:
|
|
* the provider reported a provisional creation that did not finish, or a
|
|
* control-plane database query failed while the lease was taken.
|
|
*/
|
|
type TransientLeaseFailureReason = "acquisition_incomplete" | "database_error";
|
|
type TransientProviderEvidence = { httpStatus?: number; errorCode?: string; acquisitionTimedOut?: true; reason?: TransientLeaseFailureReason };
|
|
|
|
function own(value: unknown, key: string): unknown {
|
|
try {
|
|
return value && typeof value === "object" ? Object.getOwnPropertyDescriptor(value, key)?.value : undefined;
|
|
} catch { return undefined; }
|
|
}
|
|
|
|
/**
|
|
* Read transient provider evidence from structured fields only: the validated
|
|
* plugin RPC envelope, or the status/errno an in-process provider attached to
|
|
* its error. An acquisition whose first failure was its own deadline is
|
|
* transient too. Message text is never parsed.
|
|
*/
|
|
export function readTransientEnvironmentProviderFailure(error: unknown): TransientProviderEvidence | null {
|
|
for (let cause = error, depth = 0; cause && depth < 5; cause = own(cause, "cause"), depth++) {
|
|
const acquisition = readEnvironmentAcquisitionDiagnostic(cause);
|
|
if (acquisition && acquisition.elapsedMs >= acquisition.budgetMs) return { acquisitionTimedOut: true };
|
|
const envelope = readEnvironmentAcquireErrorDiagnostic(cause);
|
|
const statusValue = envelope?.httpStatus ?? [own(cause, "status"), own(cause, "statusCode")]
|
|
.find(value => Number.isInteger(value));
|
|
const codeValue = envelope?.errorCode ?? own(cause, "code");
|
|
const httpStatus = typeof statusValue === "number" && TRANSIENT_HTTP_STATUSES.has(statusValue) ? statusValue : undefined;
|
|
const errorCode = typeof codeValue === "string" && TRANSIENT_ERROR_CODES.has(codeValue) ? codeValue : undefined;
|
|
if (httpStatus !== undefined || errorCode !== undefined) {
|
|
return { ...(httpStatus !== undefined ? { httpStatus } : {}), ...(errorCode !== undefined ? { errorCode } : {}) };
|
|
}
|
|
// A definite non-transient status is authoritative for this layer.
|
|
if (typeof statusValue === "number" && statusValue >= 400) return null;
|
|
}
|
|
return null;
|
|
}
|
|
|
|
/** A definite non-transient HTTP status anywhere on the chain is authoritative: retrying cannot change it. */
|
|
function definiteNonTransientStatus(error: unknown): boolean {
|
|
for (let cause = error, depth = 0; cause && depth < 5; cause = own(cause, "cause"), depth++) {
|
|
const status = readEnvironmentAcquireErrorDiagnostic(cause)?.httpStatus
|
|
?? [own(cause, "status"), own(cause, "statusCode")].find(value => Number.isInteger(value));
|
|
if (typeof status === "number" && status >= 400 && !TRANSIENT_HTTP_STATUSES.has(status)) return true;
|
|
}
|
|
return false;
|
|
}
|
|
|
|
// postgres.js connection-level codes, and SQLSTATEs a repeated statement can
|
|
// pass: connection (08), insufficient resources (53), operator intervention
|
|
// (57P01-57P03, 57014 query cancelled or timed out), lock not available (55P03),
|
|
// serialization failure and deadlock (40001, 40P01). A deterministic SQL error
|
|
// (syntax, schema, constraint) is not retried.
|
|
const DATABASE_CONNECTION_CODES = new Set(["CONNECTION_CLOSED", "CONNECTION_ENDED", "CONNECTION_DESTROYED", "CONNECT_TIMEOUT"]);
|
|
const TRANSIENT_SQLSTATES = new Set(["57P01", "57P02", "57P03", "57014", "55P03", "40001", "40P01"]);
|
|
|
|
function transientDatabaseFailure(error: unknown): boolean {
|
|
for (let cause = error, depth = 0; cause && depth < 5; cause = own(cause, "cause"), depth++) {
|
|
const code = own(cause, "code");
|
|
if (typeof code !== "string") continue;
|
|
if (DATABASE_CONNECTION_CODES.has(code)) return true;
|
|
if (/^[0-9A-Z]{5}$/.test(code) && (code.startsWith("08") || code.startsWith("53") || TRANSIENT_SQLSTATES.has(code))) {
|
|
// A five-character code is a SQLSTATE only on a database driver error.
|
|
const name = own(cause, "name");
|
|
if (name === "PostgresError" || name === "DatabaseError" || own(cause, "severity") !== undefined) return true;
|
|
}
|
|
}
|
|
return false;
|
|
}
|
|
|
|
/**
|
|
* Classify a failed lease acquisition for the bounded bootstrap retry. Lease
|
|
* acquisition precedes any provider (agent) work by construction, and every
|
|
* retry is a new creation attempt with its own idempotent identity, so a
|
|
* failure is retried unless it carries a definite non-transient status. Only
|
|
* structured evidence is read: the provider's status or errno, the validated
|
|
* acquisition diagnostic the runtime recorded for this run, the plugin RPC
|
|
* deadline, or a transient database driver code. Message text is never parsed.
|
|
*/
|
|
export function readTransientLeaseAcquisitionFailure(
|
|
error: unknown,
|
|
run: { id: string; companyId: string },
|
|
): TransientProviderEvidence | null {
|
|
const provider = readTransientEnvironmentProviderFailure(error);
|
|
if (provider) return provider;
|
|
if (definiteNonTransientStatus(error)) return null;
|
|
for (let cause = error, depth = 0; cause && depth < 5; cause = own(cause, "cause"), depth++) {
|
|
const acquisition = getEnvironmentAcquisitionDiagnostic(cause, run);
|
|
if (acquisition) {
|
|
return acquisition.elapsedMs >= acquisition.budgetMs ? { acquisitionTimedOut: true } : { reason: "acquisition_incomplete" };
|
|
}
|
|
if (cause instanceof JsonRpcCallError && cause.code === PLUGIN_RPC_ERROR_CODES.TIMEOUT) return { acquisitionTimedOut: true };
|
|
}
|
|
return transientDatabaseFailure(error) ? { reason: "database_error" } : null;
|
|
}
|
|
|
|
const REASON_LABELS: Record<TransientLeaseFailureReason, string> = {
|
|
acquisition_incomplete: "sandbox creation did not finish",
|
|
database_error: "a database query failed while recording the lease",
|
|
};
|
|
|
|
function category(evidence: TransientProviderEvidence): ProviderFailure["category"] {
|
|
if (evidence.httpStatus === 429) return "rate_limit";
|
|
if (evidence.acquisitionTimedOut || evidence.httpStatus === 408 || evidence.httpStatus === 504
|
|
|| evidence.errorCode === "ETIMEDOUT" || evidence.errorCode === "UND_ERR_CONNECT_TIMEOUT") return "timeout";
|
|
return "overloaded";
|
|
}
|
|
|
|
/** A lease acquisition the environment provider refused transiently. */
|
|
export class EnvironmentProviderUnavailableError extends Error {
|
|
readonly code = ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE;
|
|
readonly httpStatus: number | null;
|
|
readonly errorCode: string | null;
|
|
/** Read by the setup-failure path and shown on the task as the reason. */
|
|
readonly providerFailure: ProviderFailure;
|
|
|
|
constructor(input: { environmentName: string; driver: string; evidence: TransientProviderEvidence; cause: unknown }) {
|
|
const status = input.evidence.httpStatus !== undefined ? `HTTP ${input.evidence.httpStatus}`
|
|
: input.evidence.errorCode ?? (input.evidence.reason ? REASON_LABELS[input.evidence.reason] : undefined);
|
|
super(status
|
|
? `Environment provider for "${input.environmentName}" was temporarily unavailable (${status}).`
|
|
: `Environment provider for "${input.environmentName}" did not finish starting the sandbox in time.`, { cause: input.cause });
|
|
this.name = "EnvironmentProviderUnavailableError";
|
|
this.httpStatus = input.evidence.httpStatus ?? null;
|
|
this.errorCode = input.evidence.errorCode ?? null;
|
|
const providerCode = input.evidence.errorCode ?? (input.evidence.httpStatus === undefined ? input.evidence.reason : undefined);
|
|
this.providerFailure = {
|
|
schema: "paperclip.provider-failure.v1",
|
|
provider: /^[a-zA-Z0-9_.:/-]{1,80}$/.test(input.driver) ? input.driver : "environment",
|
|
phase: "startup",
|
|
category: category(input.evidence),
|
|
message: (status
|
|
? `The provider for "${input.environmentName}" was temporarily unavailable while Paperclip was starting it (${status}). `
|
|
: `The sandbox for "${input.environmentName}" did not finish starting in time. `)
|
|
+ "No agent work had started. Paperclip retries automatically; if those retries are used up, use Retry once the provider is available.",
|
|
...(input.evidence.httpStatus !== undefined ? { httpStatus: input.evidence.httpStatus } : {}),
|
|
...(providerCode !== undefined ? { providerCode } : {}),
|
|
retryability: "transient",
|
|
};
|
|
}
|
|
}
|
|
|
|
/** Find a transient provider refusal through the orchestrator's error wrappers. */
|
|
export function findEnvironmentProviderUnavailableError(error: unknown): EnvironmentProviderUnavailableError | null {
|
|
for (let cause = error, depth = 0; cause && depth < 4; cause = (cause as { cause?: unknown }).cause, depth++) {
|
|
if (cause instanceof EnvironmentProviderUnavailableError) return cause;
|
|
}
|
|
return null;
|
|
}
|