Files
PaperClipAI/server/src/services/environment-provider-unavailable-error.test.ts
T
DottaandClaude Fable 5.2 ffe05b1b7c fix(heartbeat): retry every pre-provider lease acquisition failure and rest it on Retry (#20857)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A task run takes an environment lease (for example a Daytona
sandbox) before the agent starts. No provider work happens before that
lease exists.
> - Only a provider HTTP status or a connection errno counted as a
transient lease failure. #20828 (merged while this was open) added the
Daytona acquisition deadline. A database error while the lease was
recorded, a plugin RPC deadline, and an unfinished provisional creation
still did not count.
> - Those failures became a generic `setup_failed` with no bounded
retry. And when the bounded retries of any
`environment_provider_unavailable` failure were used up, recovery still
put the task on a board-owned `stranded_assigned_issue` action.
> - A hosted instance saw this for six runs in 45 minutes when
concurrent runs rose from about 8 to about 20. Two tasks stayed blocked
and waited for the board.
> - This pull request classifies every lease acquisition failure that
has structured transient evidence as `environment_provider_unavailable`.
When the bounded retries are used up, the task rests on a system-owned
action and a working Retry.
> - It also adds the restart-chaos fault EP1, so the harness covers
lease acquisition failures from now on.
> - The benefit is that a busy or slow sandbox provider no longer
strands tasks on the board.

## Linked Issues or Issue Description

No public issue exists.

**What happened?**

Five runs failed with `Failed to acquire lease for environment
"Paperclip Sandbox" (sandbox): Daytona lease acquisition timed out;
allocation cleanup is pending` after 5 to 6 minutes in `preparing`. One
run failed with `Database query failed (lease_acquire_failed)` within 10
seconds. Every run recorded `executionRecovery: { kind: "bootstrap",
providerWorkStarted: false }`, but the error code was `setup_failed`. No
run got a bounded retry. Two tasks were left `blocked` with a
board-owned `stranded_assigned_issue` recovery action.

**Expected behavior**

A lease acquisition failure happens before any provider work. The run
must retry with a bounded, visible backoff. When the bound is used up,
the task must rest with a Retry that works, and no board-owned recovery
action.

**Steps to reproduce**

1. Make lease acquisition fail with the provider's acquisition deadline
(the validated acquisition diagnostic that the environment runtime
records), or with a database driver error.
2. Let the run start.
3. Before this change, the run fails with `setup_failed`, no retry is
scheduled, and recovery creates a board-owned `stranded_assigned_issue`
action.

## What Changed

**How this PR and #20828 divide the problem.** #20828 (merged) makes the
Daytona plugin end a stuck create early and classifies the
acquisition-deadline envelope as `environment_provider_unavailable`.
This PR does not change the Daytona plugin or that deadline path. It
adds only what #20828 leaves out:
- a lease-acquisition failure that is not a create timeout: the plugin
RPC deadline (`-32003`) and an unfinished provisional creation recorded
for this run;
- a transient database error while the lease is recorded;
- the exhausted state: a system-owned recovery action and Retry instead
of a board-owned `stranded_assigned_issue`, for every
`environment_provider_unavailable` cause (including #20828's);
- the chaos fault EP1 (EP1, EP1N, EP1D).

- `readTransientLeaseAcquisitionFailure`
(`server/src/services/environment-provider-unavailable-error.ts`)
classifies a lease acquisition failure as transient when it has
structured evidence: the provider status or errno (as before), the run's
own validated acquisition diagnostic (deadline or unfinished provisional
creation), the plugin RPC deadline, or a transient database error (a
postgres.js connection code, or SQLSTATE class 08 or 53, 57P01-57P03,
57014, 55P03, 40001, 40P01). A deterministic SQL error (syntax, schema,
constraint) is not retried. A definite non-transient status (for example
HTTP 401 or 404) is still `setup_failed`. Message text is not parsed.
- `environment-run-orchestrator.ts` uses this classifier, so these
failures get `environment_provider_unavailable` and the existing bounded
bootstrap retry (30 s, then 60 s, with jitter). The task shows "retry
scheduled" while it waits.
- When the retries are used up, recovery uses the cause
`environment_provider_unavailable` with a system-owned action and no
board routing. The notice says "Retry when ready" and does not ask for a
board decision. The task's Retry starts a new acquisition.
- Restart-chaos fault EP1: a test-mode server seam
(`environment-lease-fault-injection.ts`, read from
`PAPERCLIP_CHAOS_LEASE_FAULT_CONTROL`, ignored in production) throws the
staging error shapes before any provider call.
`tests/restart-chaos/env-provider-wave.ts` runs EP1 (deadline twice,
then success), EP1N (deadline forever, then Retry after disarm) and EP1D
(database error once). The oracle fails on any `setup_failed` run, any
board-owned recovery action, a wrong retry count, or a task that does
not complete.
- Docs: `doc/execution-semantics.md` and
`tests/restart-chaos/README.md`.

## Verification

- RED on `origin/master` (`1d67e309e0`, before #20828): the three new
tests in `heartbeat-process-recovery.test.ts` ("transient sandbox
provider refusal during lease acquisition") fail with the staging
result: `expected [ [ 'failed', 'setup_failed' ] ] to deeply equal [ [
'failed', 'environment_provider_unavailable' ], [ 'scheduled_retry',
null ] ]`. On current master the deadline now retries (#20828), but the
exhausted case still rests on a board-owned `stranded_assigned_issue`
action (the existing HTTP 502 exhaustion test showed `ownerType:
"board"` before the recovery change), and the database error is still
`setup_failed`.
- GREEN: `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/heartbeat-process-recovery.test.ts -t "lease acquisition"`
passes 5 of 5. This includes the deadline and the database error that
retry once and then complete, and the deadline that never clears. That
task rests `blocked` with a system-owned action, `canRetry: true`, no
execution blocker, and Retry completes it.
- Unit tests: `environment-provider-unavailable-error.test.ts` +
`environment-lease-fault-injection.test.ts` (56),
`tests/restart-chaos/env-provider-faults.test.ts` (6) pass.
- Related suites pass: `environment-run-orchestrator`,
`task-interruption-query`, `issue-recovery-actions`,
`issue-recovery-deadlines`, `stranded-notice`,
`heartbeat-native-resume-lease-failure`.
- `server` and `tests/restart-chaos` typecheck pass.
- Chaos waves (local, Codex `gpt-6.1-sol`, head `1bd387e98f`; the later
head changes only database classification and the notice), all pass:
- EP1 `w-20261111T171803-s71-ep`, 2 tasks: each had exactly 2
`environment_provider_unavailable` runs, then succeeded with no user
action, no board-owned recovery action, brief oracle pass.
- EP1D `w-20261111T172850-s73-ep`, 1 task: 1 failed run on the database
error, then succeeded.
- EP1N `w-20261111T173346-s74-ep` (and `w-20261111T162752-s72-ep` before
the rebase): 3 failed runs, the task rested `blocked` with `canRetry:
true` and no board-owned action; after disarm, Retry completed it.
- No Daytona wave was run. The fault is injected in the server before
the provider call, so a local instance reaches the same retry and
recovery code.

## Risks

- More failures now retry, all bounded. Only transient SQLSTATEs retry.
The staging database error recorded no SQLSTATE. If it was a
deterministic SQL error, it still fails `setup_failed` and goes to the
board as before; that is intended, because a retry cannot fix it.
- Each retry is a new creation attempt with its own creation record. A
provisional allocation whose cleanup is pending stays with its record
and the creation reconciler, as before.
- The retry bound is the shared bootstrap bound (two retries). A
provider outage longer than about 15 minutes still rests on Retry, but
no longer on a board action.
- The fault seam reads a file only when the environment variable is set,
and never when `NODE_ENV=production`.
- Saved state from the previous release: no persisted shape changes. A
task already stranded on a board action keeps it
(`preserveExistingOwner`). New failures use the new cause.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic) through Claude Code, model ID `claude-opus-5-5`,
with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 12:05:54 -06:00

193 lines
9.8 KiB
TypeScript

import { describe, expect, it } from "vitest";
import { JsonRpcCallError, PLUGIN_RPC_ERROR_CODES, readEnvironmentAcquisitionDiagnostic } from "@paperclipai/plugin-sdk";
import { captureEnvironmentAcquisitionDiagnostic } from "./environment-acquisition-diagnostics.js";
import {
EnvironmentProviderUnavailableError,
findEnvironmentProviderUnavailableError,
readTransientEnvironmentProviderFailure,
readTransientLeaseAcquisitionFailure,
} from "./environment-provider-unavailable-error.js";
function withStatus(status: number, key: "status" | "statusCode" = "status") {
return Object.assign(new Error("provider refused"), { [key]: status });
}
describe("readTransientEnvironmentProviderFailure", () => {
it.each([408, 425, 429, 500, 502, 503, 504])("treats HTTP %i as transient", (status) => {
expect(readTransientEnvironmentProviderFailure(withStatus(status))).toEqual({ httpStatus: status });
});
it("reads statusCode as well as status", () => {
expect(readTransientEnvironmentProviderFailure(withStatus(503, "statusCode"))).toEqual({ httpStatus: 503 });
});
it.each(["ECONNRESET", "ECONNREFUSED", "ETIMEDOUT", "EPIPE", "EAI_AGAIN", "ENETUNREACH",
"UND_ERR_CONNECT_TIMEOUT", "UND_ERR_SOCKET"])("treats errno %s as transient", (code) => {
expect(readTransientEnvironmentProviderFailure(Object.assign(new Error("closed"), { code })))
.toEqual({ errorCode: code });
});
it.each(["ENOTFOUND", "ABORT_ERR", "EACCES"])("does not retry errno %s", (code) => {
expect(readTransientEnvironmentProviderFailure(Object.assign(new Error("x"), { code }))).toBeNull();
});
it.each([400, 401, 403, 404, 409, 422, 501])("does not retry HTTP %i", (status) => {
expect(readTransientEnvironmentProviderFailure(withStatus(status))).toBeNull();
});
it("finds transient evidence on a wrapped cause", () => {
const wrapped = new Error("outer", { cause: new Error("middle", { cause: withStatus(502) }) });
expect(readTransientEnvironmentProviderFailure(wrapped)).toEqual({ httpStatus: 502 });
});
it("stops at a definite non-transient status instead of reading deeper causes", () => {
const outer = Object.assign(new Error("bad request", { cause: withStatus(503) }), { status: 400 });
expect(readTransientEnvironmentProviderFailure(outer)).toBeNull();
});
it("reads the validated plugin RPC envelope", () => {
const rpcError = Object.assign(new Error("rpc failed"), {
data: { schema: "paperclip/environment-acquire-error/v1", providerDiagnostic: { httpStatus: 429 } },
});
expect(readTransientEnvironmentProviderFailure(rpcError)).toEqual({ httpStatus: 429 });
});
const stagingLeaseTimeout = (diagnostic: { phase: string; elapsedMs: number; budgetMs: number; providerState?: string }) =>
Object.assign(new Error("Daytona lease acquisition timed out; allocation cleanup is pending"), {
data: { schema: "paperclip/environment-creation-cleanup/v1",
cleanup: { providerLeaseId: "paperclip-create-05f81d77", companyId: "company", environmentId: "environment",
runId: "f41857b6", attemptId: "05f81d77", accountFingerprint: "a".repeat(64),
labels: { "paperclip-provider": "daytona", "paperclip-create-attempt": "05f81d77" } },
acquisitionDiagnostic: diagnostic },
});
it("treats an acquisition that reached its deadline as transient", () => {
const timeout = stagingLeaseTimeout({ phase: "create", elapsedMs: 300_004, budgetMs: 300_000, providerState: "creating" });
expect(readTransientEnvironmentProviderFailure(new Error("outer", { cause: timeout }))).toEqual({ acquisitionTimedOut: true });
});
it("does not retry an acquisition whose first failure came before its deadline", () => {
expect(readTransientEnvironmentProviderFailure(stagingLeaseTimeout({ phase: "workspace", elapsedMs: 100, budgetMs: 300_000 }))).toBeNull();
});
it("never parses message text", () => {
expect(readTransientEnvironmentProviderFailure(new Error("HTTP 502 Bad Gateway ECONNRESET"))).toBeNull();
});
});
describe("EnvironmentProviderUnavailableError", () => {
it.each([
[{ httpStatus: 429 }, "rate_limit"],
[{ httpStatus: 408 }, "timeout"],
[{ httpStatus: 504 }, "timeout"],
[{ errorCode: "ETIMEDOUT" }, "timeout"],
[{ httpStatus: 502 }, "overloaded"],
[{ errorCode: "ECONNRESET" }, "overloaded"],
] as const)("classifies %o as %s", (evidence, category) => {
const error = new EnvironmentProviderUnavailableError({
environmentName: "env", driver: "daytona", evidence, cause: null,
});
expect(error.providerFailure.category).toBe(category);
expect(error.providerFailure.retryability).toBe("transient");
});
it("names an acquisition timeout in plain words", () => {
const error = new EnvironmentProviderUnavailableError({
environmentName: "Paperclip Sandbox", driver: "sandbox", evidence: { acquisitionTimedOut: true }, cause: null,
});
expect(error.providerFailure).toMatchObject({ category: "timeout", retryability: "transient", phase: "startup" });
expect(error.providerFailure.message).toContain("did not finish starting");
expect(error.providerFailure.message).not.toMatch(/ETIMEDOUT|undefined/);
});
it("falls back to a generic provider label for an unsafe driver key", () => {
const error = new EnvironmentProviderUnavailableError({
environmentName: "env", driver: "bad driver <x>", evidence: { httpStatus: 502 }, cause: null,
});
expect(error.providerFailure.provider).toBe("environment");
});
it("is found through wrapper causes", () => {
const inner = new EnvironmentProviderUnavailableError({
environmentName: "env", driver: "ssh", evidence: { errorCode: "ECONNREFUSED" }, cause: null,
});
expect(findEnvironmentProviderUnavailableError(new Error("outer", { cause: inner }))).toBe(inner);
expect(findEnvironmentProviderUnavailableError(new Error("plain"))).toBeNull();
});
});
describe("readTransientLeaseAcquisitionFailure", () => {
const run = { id: "run-1", companyId: "company-1" };
const creationFailure = (elapsedMs: number, runId = run.id) => {
const error = new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.WORKER_ERROR,
message: "Daytona lease acquisition timed out; allocation cleanup is pending",
data: { schema: "paperclip/environment-creation-cleanup/v1",
cleanup: { providerLeaseId: "paperclip-create-1", companyId: run.companyId, environmentId: "env-1", runId,
attemptId: "attempt-1", accountFingerprint: "a".repeat(64), labels: {} },
acquisitionDiagnostic: { phase: "create", elapsedMs, budgetMs: 300_000 } } });
captureEnvironmentAcquisitionDiagnostic(error, { companyId: run.companyId, environmentId: "env-1", runId },
readEnvironmentAcquisitionDiagnostic(error));
return error;
};
it("keeps the provider's transient status", () => {
expect(readTransientLeaseAcquisitionFailure(withStatus(503), run)).toEqual({ httpStatus: 503 });
});
it("retries a creation that hit the acquisition deadline", () => {
expect(readTransientLeaseAcquisitionFailure(new Error("wrapped", { cause: creationFailure(300_000) }), run))
.toEqual({ acquisitionTimedOut: true });
});
it("retries a provisional creation that did not finish", () => {
expect(readTransientLeaseAcquisitionFailure(creationFailure(12_000), run)).toEqual({ reason: "acquisition_incomplete" });
});
it("ignores an unfinished-creation diagnostic recorded for another run", () => {
expect(readTransientLeaseAcquisitionFailure(creationFailure(12_000, "run-2"), run)).toBeNull();
});
it("retries a plugin RPC deadline", () => {
const timeout = new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.TIMEOUT, message: "RPC call timed out" });
expect(readTransientLeaseAcquisitionFailure(timeout, run)).toEqual({ acquisitionTimedOut: true });
});
const pg = (code: string) => Object.assign(new Error("database error"), { name: "PostgresError", code });
it.each([
["a terminated connection", pg("57P01")],
["a connection failure", pg("08006")],
["too many connections", pg("53300")],
["a lock timeout", pg("55P03")],
["a closed connection", Object.assign(new Error("closed"), { code: "CONNECTION_CLOSED" })],
["a Drizzle wrapper around a statement timeout", new Error("Failed query: select 1\nparams: ", { cause: pg("57014") })],
])("retries %s while recording the lease", (_label, cause) => {
expect(readTransientLeaseAcquisitionFailure(new Error("lease", { cause }), run)).toEqual({ reason: "database_error" });
});
it.each([
["a unique violation", pg("23505")],
["an undefined column", pg("42703")],
["a Drizzle wrapper around a syntax error", new Error("Failed query: select\nparams: ", { cause: pg("42601") })],
["a bare Drizzle wrapper", new Error("Failed query: select 1\nparams: ")],
["an errno that looks like a SQLSTATE", Object.assign(new Error("x"), { code: "EPERM" })],
])("does not retry %s", (_label, cause) => {
expect(readTransientLeaseAcquisitionFailure(new Error("lease", { cause }), run)).toBeNull();
});
it("does not retry a definite non-transient status, even with other evidence", () => {
const refused = Object.assign(new Error("unauthorized", { cause: creationFailure(300_000) }), { status: 401 });
expect(readTransientLeaseAcquisitionFailure(refused, run)).toBeNull();
});
it("does not retry an error without structured evidence", () => {
expect(readTransientLeaseAcquisitionFailure(new Error("Daytona lease acquisition timed out"), run)).toBeNull();
});
it("names the reason in the board-facing failure", () => {
const error = new EnvironmentProviderUnavailableError({ environmentName: "Sandbox", driver: "daytona",
evidence: { reason: "database_error" }, cause: null });
expect(error.providerFailure).toMatchObject({ category: "overloaded", providerCode: "database_error" });
expect(error.message).toContain("a database query failed while recording the lease");
});
});