mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-14 00:26:28 +01:00
fix(heartbeat): retry every pre-provider lease acquisition failure and rest it on Retry (#20857)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A task run takes an environment lease (for example a Daytona sandbox) before the agent starts. No provider work happens before that lease exists. > - Only a provider HTTP status or a connection errno counted as a transient lease failure. #20828 (merged while this was open) added the Daytona acquisition deadline. A database error while the lease was recorded, a plugin RPC deadline, and an unfinished provisional creation still did not count. > - Those failures became a generic `setup_failed` with no bounded retry. And when the bounded retries of any `environment_provider_unavailable` failure were used up, recovery still put the task on a board-owned `stranded_assigned_issue` action. > - A hosted instance saw this for six runs in 45 minutes when concurrent runs rose from about 8 to about 20. Two tasks stayed blocked and waited for the board. > - This pull request classifies every lease acquisition failure that has structured transient evidence as `environment_provider_unavailable`. When the bounded retries are used up, the task rests on a system-owned action and a working Retry. > - It also adds the restart-chaos fault EP1, so the harness covers lease acquisition failures from now on. > - The benefit is that a busy or slow sandbox provider no longer strands tasks on the board. ## Linked Issues or Issue Description No public issue exists. **What happened?** Five runs failed with `Failed to acquire lease for environment "Paperclip Sandbox" (sandbox): Daytona lease acquisition timed out; allocation cleanup is pending` after 5 to 6 minutes in `preparing`. One run failed with `Database query failed (lease_acquire_failed)` within 10 seconds. Every run recorded `executionRecovery: { kind: "bootstrap", providerWorkStarted: false }`, but the error code was `setup_failed`. No run got a bounded retry. Two tasks were left `blocked` with a board-owned `stranded_assigned_issue` recovery action. **Expected behavior** A lease acquisition failure happens before any provider work. The run must retry with a bounded, visible backoff. When the bound is used up, the task must rest with a Retry that works, and no board-owned recovery action. **Steps to reproduce** 1. Make lease acquisition fail with the provider's acquisition deadline (the validated acquisition diagnostic that the environment runtime records), or with a database driver error. 2. Let the run start. 3. Before this change, the run fails with `setup_failed`, no retry is scheduled, and recovery creates a board-owned `stranded_assigned_issue` action. ## What Changed **How this PR and #20828 divide the problem.** #20828 (merged) makes the Daytona plugin end a stuck create early and classifies the acquisition-deadline envelope as `environment_provider_unavailable`. This PR does not change the Daytona plugin or that deadline path. It adds only what #20828 leaves out: - a lease-acquisition failure that is not a create timeout: the plugin RPC deadline (`-32003`) and an unfinished provisional creation recorded for this run; - a transient database error while the lease is recorded; - the exhausted state: a system-owned recovery action and Retry instead of a board-owned `stranded_assigned_issue`, for every `environment_provider_unavailable` cause (including #20828's); - the chaos fault EP1 (EP1, EP1N, EP1D). - `readTransientLeaseAcquisitionFailure` (`server/src/services/environment-provider-unavailable-error.ts`) classifies a lease acquisition failure as transient when it has structured evidence: the provider status or errno (as before), the run's own validated acquisition diagnostic (deadline or unfinished provisional creation), the plugin RPC deadline, or a transient database error (a postgres.js connection code, or SQLSTATE class 08 or 53, 57P01-57P03, 57014, 55P03, 40001, 40P01). A deterministic SQL error (syntax, schema, constraint) is not retried. A definite non-transient status (for example HTTP 401 or 404) is still `setup_failed`. Message text is not parsed. - `environment-run-orchestrator.ts` uses this classifier, so these failures get `environment_provider_unavailable` and the existing bounded bootstrap retry (30 s, then 60 s, with jitter). The task shows "retry scheduled" while it waits. - When the retries are used up, recovery uses the cause `environment_provider_unavailable` with a system-owned action and no board routing. The notice says "Retry when ready" and does not ask for a board decision. The task's Retry starts a new acquisition. - Restart-chaos fault EP1: a test-mode server seam (`environment-lease-fault-injection.ts`, read from `PAPERCLIP_CHAOS_LEASE_FAULT_CONTROL`, ignored in production) throws the staging error shapes before any provider call. `tests/restart-chaos/env-provider-wave.ts` runs EP1 (deadline twice, then success), EP1N (deadline forever, then Retry after disarm) and EP1D (database error once). The oracle fails on any `setup_failed` run, any board-owned recovery action, a wrong retry count, or a task that does not complete. - Docs: `doc/execution-semantics.md` and `tests/restart-chaos/README.md`. ## Verification - RED on `origin/master` (`1d67e309e0`, before #20828): the three new tests in `heartbeat-process-recovery.test.ts` ("transient sandbox provider refusal during lease acquisition") fail with the staging result: `expected [ [ 'failed', 'setup_failed' ] ] to deeply equal [ [ 'failed', 'environment_provider_unavailable' ], [ 'scheduled_retry', null ] ]`. On current master the deadline now retries (#20828), but the exhausted case still rests on a board-owned `stranded_assigned_issue` action (the existing HTTP 502 exhaustion test showed `ownerType: "board"` before the recovery change), and the database error is still `setup_failed`. - GREEN: `pnpm --filter @paperclipai/server exec vitest run src/__tests__/heartbeat-process-recovery.test.ts -t "lease acquisition"` passes 5 of 5. This includes the deadline and the database error that retry once and then complete, and the deadline that never clears. That task rests `blocked` with a system-owned action, `canRetry: true`, no execution blocker, and Retry completes it. - Unit tests: `environment-provider-unavailable-error.test.ts` + `environment-lease-fault-injection.test.ts` (56), `tests/restart-chaos/env-provider-faults.test.ts` (6) pass. - Related suites pass: `environment-run-orchestrator`, `task-interruption-query`, `issue-recovery-actions`, `issue-recovery-deadlines`, `stranded-notice`, `heartbeat-native-resume-lease-failure`. - `server` and `tests/restart-chaos` typecheck pass. - Chaos waves (local, Codex `gpt-6.1-sol`, head `1bd387e98f`; the later head changes only database classification and the notice), all pass: - EP1 `w-20261111T171803-s71-ep`, 2 tasks: each had exactly 2 `environment_provider_unavailable` runs, then succeeded with no user action, no board-owned recovery action, brief oracle pass. - EP1D `w-20261111T172850-s73-ep`, 1 task: 1 failed run on the database error, then succeeded. - EP1N `w-20261111T173346-s74-ep` (and `w-20261111T162752-s72-ep` before the rebase): 3 failed runs, the task rested `blocked` with `canRetry: true` and no board-owned action; after disarm, Retry completed it. - No Daytona wave was run. The fault is injected in the server before the provider call, so a local instance reaches the same retry and recovery code. ## Risks - More failures now retry, all bounded. Only transient SQLSTATEs retry. The staging database error recorded no SQLSTATE. If it was a deterministic SQL error, it still fails `setup_failed` and goes to the board as before; that is intended, because a retry cannot fix it. - Each retry is a new creation attempt with its own creation record. A provisional allocation whose cleanup is pending stays with its record and the creation reconciler, as before. - The retry bound is the shared bootstrap bound (two retries). A provider outage longer than about 15 minutes still rests on Retry, but no longer on a board action. - The fault seam reads a file only when the environment variable is set, and never when `NODE_ENV=production`. - Saved state from the previous release: no persisted shape changes. A task already stranded on a board action keeps it (`preserveExistingOwner`). New failures use the new cause. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic) through Claude Code, model ID `claude-opus-5-5`, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Fable 5.2 authored and GitHub committed
1 parent
b2edea5754
commit
ffe05b1b7c
14 files changed
+825
-13
No files matched your search
@@ -28,6 +28,7 @@ import { recordLegacyWorkspaceRestoreFailure, terminalizeLegacyExecution } from
|
||||
import { issueService } from "../services/issues.js";
|
||||
import { getExecutionBlocker } from "../services/execution-blocker.js";
|
||||
import { JsonRpcCallError } from "@paperclipai/plugin-sdk";
|
||||
import { leaseFaultError } from "../services/environment-lease-fault-injection.js";
|
||||
import { taskInterruptionPresentation } from "../services/task-interruption-presentation.js";
|
||||
import { requestTaskRetry } from "../services/task-retry.js";
|
||||
import { adapterExecutionControls, createAdapterExecutionControl } from "../services/adapter-execution-control.js";
|
||||
@@ -5903,7 +5904,20 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
|
||||
message: "Daytona sandbox creation failed (HTTP 502); cleanup could not be confirmed for paperclip-create-05f81d77",
|
||||
data: { schema: "paperclip/environment-acquire-error/v1", providerDiagnostic: { httpStatus: 502 } } });
|
||||
|
||||
function failLeaseAcquisition(times: number) {
|
||||
// EP1: Daytona's own acquisition deadline expired while the provisional
|
||||
// sandbox was still being created, as staging recorded at 14:00-14:45Z when
|
||||
// concurrent runs rose from about 8 to about 20. The environment runtime
|
||||
// records the validated acquisition diagnostic for this run before rethrowing.
|
||||
// The chaos fault EP1 injects these same shapes (environment-lease-fault-injection.ts).
|
||||
type LeaseFault = (input: { companyId: string; environment: { id: string }; heartbeatRunId?: string | null }) => Error;
|
||||
const daytonaAcquisitionDeadline: LeaseFault = input => leaseFaultError("timeout",
|
||||
{ companyId: input.companyId, environmentId: input.environment.id, heartbeatRunId: input.heartbeatRunId! });
|
||||
// EP1 database: a control-plane query failed while the lease was being
|
||||
// recorded (staging: "Database query failed (lease_acquire_failed)").
|
||||
const leaseRecordQueryFailure: LeaseFault = input => leaseFaultError("database",
|
||||
{ companyId: input.companyId, environmentId: input.environment.id, heartbeatRunId: input.heartbeatRunId! });
|
||||
|
||||
function failLeaseAcquisition(times: number, fault: LeaseFault = daytonaCreate502) {
|
||||
let remaining = times;
|
||||
const acquisitions = { refused: 0, delegated: 0 };
|
||||
const originalFactory = environmentOrchestrator.environmentRunOrchestrator;
|
||||
@@ -5913,7 +5927,7 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
|
||||
if (remaining > 0) {
|
||||
remaining -= 1;
|
||||
acquisitions.refused += 1;
|
||||
throw daytonaCreate502();
|
||||
throw fault(input);
|
||||
}
|
||||
acquisitions.delegated += 1;
|
||||
return runtime.acquireRunLease(input);
|
||||
@@ -6028,6 +6042,81 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
|
||||
mockAdapterExecute.mockClear();
|
||||
}
|
||||
}, 90_000);
|
||||
it.each([
|
||||
["a provider acquisition deadline", daytonaAcquisitionDeadline, "timeout"],
|
||||
["a database error while recording the lease", leaseRecordQueryFailure, "overloaded"],
|
||||
] as const)("retries %s before provider work with a bounded, visible backoff", async (_label, fault, category) => {
|
||||
const { companyId, agentId, runId, issueId } = await seedQueuedIssueRunFixture();
|
||||
const lease = failLeaseAcquisition(1, fault);
|
||||
try {
|
||||
const heartbeat = heartbeatService(db);
|
||||
await heartbeat.resumeQueuedRuns();
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
|
||||
const afterFailure = await runsFor(agentId);
|
||||
expect(afterFailure.map(run => [run.status, run.errorCode])).toEqual([
|
||||
["failed", "environment_provider_unavailable"], ["scheduled_retry", null]]);
|
||||
expect(afterFailure[0]!.resultJson).toMatchObject({ providerFailure: { phase: "startup", category, retryability: "transient" },
|
||||
executionRecovery: { kind: "bootstrap", providerWorkStarted: false } });
|
||||
expect(afterFailure[1]).toMatchObject({ retryOfRunId: runId, scheduledRetryAttempt: 1, scheduledRetryReason: "transient_failure" });
|
||||
const [waiting] = await db.select().from(issues).where(eq(issues.id, issueId));
|
||||
expect(waiting).toMatchObject({ status: "in_progress", executionRunId: afterFailure[1]!.id });
|
||||
expect(await taskInterruptionPresentation(db, waiting!)).toMatchObject({ state: "retry_scheduled", responsibility: "system" });
|
||||
expect(await getExecutionBlocker(db, companyId, issueId)).toBeNull();
|
||||
|
||||
await promoteLatestScheduledRetry(heartbeat, agentId);
|
||||
const successors = (await runsFor(agentId)).filter(run => run.retryOfRunId === runId);
|
||||
expect(successors).toHaveLength(1);
|
||||
expect(successors[0]).toMatchObject({ status: "succeeded" });
|
||||
expect(lease.acquisitions.refused).toBe(1);
|
||||
// The mock adapter's later disposition repair is unrelated; no setup failure reached recovery.
|
||||
expect((await selectRecoveryActionsForIssue(issueId)).filter(action => action.kind === "stranded_assigned_issue")).toEqual([]);
|
||||
} finally {
|
||||
lease.restore();
|
||||
mockAdapterExecute.mockClear();
|
||||
}
|
||||
}, 60_000);
|
||||
|
||||
it("rests a provider deadline that never clears on a working Retry, never on a board recovery action", async () => {
|
||||
const { companyId, agentId, issueId } = await seedQueuedIssueRunFixture();
|
||||
const lease = failLeaseAcquisition(BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS + 1, daytonaAcquisitionDeadline);
|
||||
try {
|
||||
const heartbeat = heartbeatService(db);
|
||||
await heartbeat.resumeQueuedRuns();
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
for (let attempt = 0; attempt < BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS; attempt += 1) {
|
||||
await promoteLatestScheduledRetry(heartbeat, agentId);
|
||||
}
|
||||
const exhausted = await runsFor(agentId);
|
||||
expect(exhausted.map(run => run.errorCode)).toEqual(
|
||||
Array(BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS + 1).fill("environment_provider_unavailable"));
|
||||
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
||||
|
||||
const blocked = await waitForValue(async () => {
|
||||
const [row] = await db.select().from(issues).where(eq(issues.id, issueId));
|
||||
return row?.status === "blocked" ? row : null;
|
||||
});
|
||||
const presentation = await taskInterruptionPresentation(db, blocked!);
|
||||
expect(presentation).toMatchObject({ state: "failed", responsibility: "system", canRetry: true,
|
||||
observedRunId: exhausted.at(-1)!.id, providerFailure: { category: "timeout", retryability: "transient" } });
|
||||
expect(await getExecutionBlocker(db, companyId, issueId)).toBeNull();
|
||||
// The task rests on a system-owned action and Retry; nothing waits on the board.
|
||||
expect((await selectRecoveryActionsForIssue(issueId)).map(action => [action.ownerType, action.cause]))
|
||||
.toEqual([["system", "environment_provider_unavailable"]]);
|
||||
|
||||
const receipt = await requestTaskRetry({ db, companyId, issueId, actorId: "responsible-user",
|
||||
idempotencyKey: randomUUID(), observedRunId: presentation!.observedRunId });
|
||||
expect(receipt.accepted).toBe(true);
|
||||
await heartbeat.resumeTaskRetry(companyId, receipt.requestId!);
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
expect(await waitForValue(async () => (await runsFor(agentId)).find(run => run.status === "succeeded") ?? null, 10_000))
|
||||
.not.toBeNull();
|
||||
expect(lease.acquisitions.delegated).toBe(1);
|
||||
} finally {
|
||||
lease.restore();
|
||||
mockAdapterExecute.mockClear();
|
||||
}
|
||||
}, 90_000);
|
||||
});
|
||||
|
||||
it("classifies only the installed-but-not-ready sandbox provider plugin message as a configuration gap", () => {
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
import { mkdtemp, rm, writeFile } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { parseLeaseFaultControl } from "./environment-lease-fault-injection.js";
|
||||
|
||||
describe("lease acquisition fault injection (chaos EP1)", () => {
|
||||
let directory: string;
|
||||
let control: string;
|
||||
let seam: typeof import("./environment-lease-fault-injection.js");
|
||||
// Loaded after the reset, so it shares the seam's diagnostic receipts.
|
||||
let readTransientLeaseAcquisitionFailure: typeof import("./environment-provider-unavailable-error.js").readTransientLeaseAcquisitionFailure;
|
||||
const input = (issueId: string | null = "issue-1") => ({ companyId: "company-1", environmentId: "env-1", heartbeatRunId: "run-1", issueId });
|
||||
|
||||
beforeAll(async () => {
|
||||
directory = await mkdtemp(path.join(os.tmpdir(), "lease-fault-"));
|
||||
control = path.join(directory, "control.json");
|
||||
vi.stubEnv("PAPERCLIP_CHAOS_LEASE_FAULT_CONTROL", control);
|
||||
vi.resetModules();
|
||||
seam = await import("./environment-lease-fault-injection.js");
|
||||
({ readTransientLeaseAcquisitionFailure } = await import("./environment-provider-unavailable-error.js"));
|
||||
});
|
||||
afterAll(async () => {
|
||||
vi.unstubAllEnvs();
|
||||
await rm(directory, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("parses only a well-formed control", () => {
|
||||
expect(parseLeaseFaultControl('{"token":"a","kind":"timeout","failures":2}')).toEqual({ token: "a", kind: "timeout", failures: 2 });
|
||||
expect(parseLeaseFaultControl('{"token":"a","kind":"database","failures":null}')).toEqual({ token: "a", kind: "database", failures: null });
|
||||
expect(parseLeaseFaultControl('{"token":"a","kind":"other","failures":1}')).toBeNull();
|
||||
expect(parseLeaseFaultControl('{"token":"a b","kind":"timeout","failures":1}')).toBeNull();
|
||||
expect(parseLeaseFaultControl("")).toBeNull();
|
||||
});
|
||||
|
||||
it("is disarmed without a control file", () => {
|
||||
expect(() => seam.injectLeaseAcquisitionFault(input())).not.toThrow();
|
||||
});
|
||||
|
||||
it("fails each task's first N acquisitions with a retryable staging shape, then lets them through", async () => {
|
||||
await writeFile(control, JSON.stringify({ token: "t1", kind: "timeout", failures: 2 }));
|
||||
for (let attempt = 0; attempt < 2; attempt += 1) {
|
||||
let thrown: unknown;
|
||||
try { seam.injectLeaseAcquisitionFault(input()); } catch (error) { thrown = error; }
|
||||
expect(readTransientLeaseAcquisitionFailure(thrown, { id: "run-1", companyId: "company-1" })).toEqual({ acquisitionTimedOut: true });
|
||||
}
|
||||
expect(() => seam.injectLeaseAcquisitionFault(input())).not.toThrow();
|
||||
// Another task keeps its own count; a run with no task is never faulted.
|
||||
expect(() => seam.injectLeaseAcquisitionFault(input("issue-2"))).toThrow(/lease acquisition timed out/);
|
||||
expect(() => seam.injectLeaseAcquisitionFault(input(null))).not.toThrow();
|
||||
});
|
||||
|
||||
it("re-arms with a new token, fails forever with null, and disarms on an empty file", async () => {
|
||||
await writeFile(control, JSON.stringify({ token: "t2", kind: "database", failures: null }));
|
||||
for (let attempt = 0; attempt < 5; attempt += 1) {
|
||||
let thrown: unknown;
|
||||
try { seam.injectLeaseAcquisitionFault(input()); } catch (error) { thrown = error; }
|
||||
expect(readTransientLeaseAcquisitionFailure(thrown, { id: "run-1", companyId: "company-1" })).toEqual({ reason: "database_error" });
|
||||
}
|
||||
await writeFile(control, "");
|
||||
expect(() => seam.injectLeaseAcquisitionFault(input())).not.toThrow();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,76 @@
|
||||
/**
|
||||
* Restart-chaos fault EP1 (tests/restart-chaos/env-provider-faults.ts): make a
|
||||
* run's lease acquisition fail before any provider call, in the exact shapes
|
||||
* staging recorded, so the bounded retry and Retry paths can be exercised on a
|
||||
* local instance without a paid sandbox provider.
|
||||
*
|
||||
* Arming (test mode only; ignored when NODE_ENV=production):
|
||||
*
|
||||
* PAPERCLIP_CHAOS_LEASE_FAULT_CONTROL=<file>
|
||||
*
|
||||
* Each acquisition reads the file. Missing, empty or malformed means disarmed.
|
||||
* Content: {"token":"<id>","kind":"timeout"|"database","failures":<n>|null}.
|
||||
* The first <n> acquisitions of each task under one token fail (null: every
|
||||
* one), so writing a new token re-arms with fresh counts and an empty file
|
||||
* disarms. Counts live in this process, so a restart starts them again.
|
||||
*
|
||||
* This is test instrumentation, not Telemetry or Observability
|
||||
* (rules/data-paths.mdc): nothing is recorded or sent anywhere.
|
||||
*/
|
||||
import { readFileSync } from "node:fs";
|
||||
import { JsonRpcCallError, PLUGIN_RPC_ERROR_CODES, readEnvironmentAcquisitionDiagnostic } from "@paperclipai/plugin-sdk";
|
||||
import { captureEnvironmentAcquisitionDiagnostic } from "./environment-acquisition-diagnostics.js";
|
||||
|
||||
export const LEASE_FAULT_CONTROL_ENV = "PAPERCLIP_CHAOS_LEASE_FAULT_CONTROL";
|
||||
|
||||
export type LeaseFaultControl = { token: string; kind: "timeout" | "database"; failures: number | null };
|
||||
|
||||
export function parseLeaseFaultControl(text: string): LeaseFaultControl | null {
|
||||
try {
|
||||
const value = JSON.parse(text) as Record<string, unknown>;
|
||||
if (typeof value.token !== "string" || !/^[A-Za-z0-9_-]{1,80}$/.test(value.token)) return null;
|
||||
if (value.kind !== "timeout" && value.kind !== "database") return null;
|
||||
const failures = value.failures;
|
||||
if (failures !== null && !(Number.isSafeInteger(failures) && (failures as number) >= 0)) return null;
|
||||
return { token: value.token, kind: value.kind, failures: failures as number | null };
|
||||
} catch { return null; }
|
||||
}
|
||||
|
||||
const controlPath = process.env.NODE_ENV === "production" ? null : process.env[LEASE_FAULT_CONTROL_ENV]?.trim() || null;
|
||||
const counts = new Map<string, number>();
|
||||
|
||||
/** The staging shapes: Daytona's acquisition deadline, and a failed control-plane query. */
|
||||
export function leaseFaultError(kind: LeaseFaultControl["kind"], input: {
|
||||
companyId: string; environmentId: string; heartbeatRunId: string;
|
||||
}): Error {
|
||||
if (kind === "database") {
|
||||
return new Error("Failed query: insert into \"environment_leases\" (\"id\") values ($1)\nparams: <chaos>", {
|
||||
cause: Object.assign(new Error("terminating connection due to administrator command"), { name: "PostgresError", code: "57P01" }),
|
||||
});
|
||||
}
|
||||
const error = new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.WORKER_ERROR,
|
||||
message: "Daytona lease acquisition timed out; allocation cleanup is pending",
|
||||
data: { schema: "paperclip/environment-creation-cleanup/v1",
|
||||
cleanup: { providerLeaseId: `chaos-ep1-${input.heartbeatRunId.slice(0, 8)}`, companyId: input.companyId,
|
||||
environmentId: input.environmentId, runId: input.heartbeatRunId, attemptId: input.heartbeatRunId,
|
||||
accountFingerprint: "0".repeat(64), labels: {} },
|
||||
acquisitionDiagnostic: { phase: "create", elapsedMs: 300_000, budgetMs: 300_000, providerState: "creating" } } });
|
||||
captureEnvironmentAcquisitionDiagnostic(error, { companyId: input.companyId, environmentId: input.environmentId,
|
||||
runId: input.heartbeatRunId }, readEnvironmentAcquisitionDiagnostic(error));
|
||||
return error;
|
||||
}
|
||||
|
||||
/** Throws the armed fault for this acquisition; a single null check when disarmed. */
|
||||
export function injectLeaseAcquisitionFault(input: {
|
||||
companyId: string; environmentId: string; heartbeatRunId: string; issueId: string | null;
|
||||
}): void {
|
||||
if (controlPath === null || !input.issueId) return;
|
||||
let control: LeaseFaultControl | null;
|
||||
try { control = parseLeaseFaultControl(readFileSync(controlPath, "utf8")); } catch { return; }
|
||||
if (!control) return;
|
||||
const key = `${control.token}:${input.issueId}`;
|
||||
const seen = counts.get(key) ?? 0;
|
||||
if (control.failures !== null && seen >= control.failures) return;
|
||||
counts.set(key, seen + 1);
|
||||
throw leaseFaultError(control.kind, input);
|
||||
}
|
||||
@@ -1,8 +1,11 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { JsonRpcCallError, PLUGIN_RPC_ERROR_CODES, readEnvironmentAcquisitionDiagnostic } from "@paperclipai/plugin-sdk";
|
||||
import { captureEnvironmentAcquisitionDiagnostic } from "./environment-acquisition-diagnostics.js";
|
||||
import {
|
||||
EnvironmentProviderUnavailableError,
|
||||
findEnvironmentProviderUnavailableError,
|
||||
readTransientEnvironmentProviderFailure,
|
||||
readTransientLeaseAcquisitionFailure,
|
||||
} from "./environment-provider-unavailable-error.js";
|
||||
|
||||
function withStatus(status: number, key: "status" | "statusCode" = "status") {
|
||||
@@ -112,3 +115,78 @@ describe("EnvironmentProviderUnavailableError", () => {
|
||||
expect(findEnvironmentProviderUnavailableError(new Error("plain"))).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("readTransientLeaseAcquisitionFailure", () => {
|
||||
const run = { id: "run-1", companyId: "company-1" };
|
||||
const creationFailure = (elapsedMs: number, runId = run.id) => {
|
||||
const error = new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.WORKER_ERROR,
|
||||
message: "Daytona lease acquisition timed out; allocation cleanup is pending",
|
||||
data: { schema: "paperclip/environment-creation-cleanup/v1",
|
||||
cleanup: { providerLeaseId: "paperclip-create-1", companyId: run.companyId, environmentId: "env-1", runId,
|
||||
attemptId: "attempt-1", accountFingerprint: "a".repeat(64), labels: {} },
|
||||
acquisitionDiagnostic: { phase: "create", elapsedMs, budgetMs: 300_000 } } });
|
||||
captureEnvironmentAcquisitionDiagnostic(error, { companyId: run.companyId, environmentId: "env-1", runId },
|
||||
readEnvironmentAcquisitionDiagnostic(error));
|
||||
return error;
|
||||
};
|
||||
|
||||
it("keeps the provider's transient status", () => {
|
||||
expect(readTransientLeaseAcquisitionFailure(withStatus(503), run)).toEqual({ httpStatus: 503 });
|
||||
});
|
||||
|
||||
it("retries a creation that hit the acquisition deadline", () => {
|
||||
expect(readTransientLeaseAcquisitionFailure(new Error("wrapped", { cause: creationFailure(300_000) }), run))
|
||||
.toEqual({ acquisitionTimedOut: true });
|
||||
});
|
||||
|
||||
it("retries a provisional creation that did not finish", () => {
|
||||
expect(readTransientLeaseAcquisitionFailure(creationFailure(12_000), run)).toEqual({ reason: "acquisition_incomplete" });
|
||||
});
|
||||
|
||||
it("ignores an unfinished-creation diagnostic recorded for another run", () => {
|
||||
expect(readTransientLeaseAcquisitionFailure(creationFailure(12_000, "run-2"), run)).toBeNull();
|
||||
});
|
||||
|
||||
it("retries a plugin RPC deadline", () => {
|
||||
const timeout = new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.TIMEOUT, message: "RPC call timed out" });
|
||||
expect(readTransientLeaseAcquisitionFailure(timeout, run)).toEqual({ acquisitionTimedOut: true });
|
||||
});
|
||||
|
||||
const pg = (code: string) => Object.assign(new Error("database error"), { name: "PostgresError", code });
|
||||
it.each([
|
||||
["a terminated connection", pg("57P01")],
|
||||
["a connection failure", pg("08006")],
|
||||
["too many connections", pg("53300")],
|
||||
["a lock timeout", pg("55P03")],
|
||||
["a closed connection", Object.assign(new Error("closed"), { code: "CONNECTION_CLOSED" })],
|
||||
["a Drizzle wrapper around a statement timeout", new Error("Failed query: select 1\nparams: ", { cause: pg("57014") })],
|
||||
])("retries %s while recording the lease", (_label, cause) => {
|
||||
expect(readTransientLeaseAcquisitionFailure(new Error("lease", { cause }), run)).toEqual({ reason: "database_error" });
|
||||
});
|
||||
|
||||
it.each([
|
||||
["a unique violation", pg("23505")],
|
||||
["an undefined column", pg("42703")],
|
||||
["a Drizzle wrapper around a syntax error", new Error("Failed query: select\nparams: ", { cause: pg("42601") })],
|
||||
["a bare Drizzle wrapper", new Error("Failed query: select 1\nparams: ")],
|
||||
["an errno that looks like a SQLSTATE", Object.assign(new Error("x"), { code: "EPERM" })],
|
||||
])("does not retry %s", (_label, cause) => {
|
||||
expect(readTransientLeaseAcquisitionFailure(new Error("lease", { cause }), run)).toBeNull();
|
||||
});
|
||||
|
||||
it("does not retry a definite non-transient status, even with other evidence", () => {
|
||||
const refused = Object.assign(new Error("unauthorized", { cause: creationFailure(300_000) }), { status: 401 });
|
||||
expect(readTransientLeaseAcquisitionFailure(refused, run)).toBeNull();
|
||||
});
|
||||
|
||||
it("does not retry an error without structured evidence", () => {
|
||||
expect(readTransientLeaseAcquisitionFailure(new Error("Daytona lease acquisition timed out"), run)).toBeNull();
|
||||
});
|
||||
|
||||
it("names the reason in the board-facing failure", () => {
|
||||
const error = new EnvironmentProviderUnavailableError({ environmentName: "Sandbox", driver: "daytona",
|
||||
evidence: { reason: "database_error" }, cause: null });
|
||||
expect(error.providerFailure).toMatchObject({ category: "overloaded", providerCode: "database_error" });
|
||||
expect(error.message).toContain("a database query failed while recording the lease");
|
||||
});
|
||||
});
|
||||
@@ -1,5 +1,6 @@
|
||||
import { readEnvironmentAcquireErrorDiagnostic, readEnvironmentAcquisitionDiagnostic } from "@paperclipai/plugin-sdk";
|
||||
import { JsonRpcCallError, PLUGIN_RPC_ERROR_CODES, readEnvironmentAcquireErrorDiagnostic, readEnvironmentAcquisitionDiagnostic } from "@paperclipai/plugin-sdk";
|
||||
import type { ProviderFailure } from "@paperclipai/shared";
|
||||
import { getEnvironmentAcquisitionDiagnostic } from "./environment-acquisition-diagnostics.js";
|
||||
|
||||
/**
|
||||
* Setup failure code for a lease acquisition that the environment provider
|
||||
@@ -21,7 +22,13 @@ const TRANSIENT_ERROR_CODES = new Set([
|
||||
"UND_ERR_CONNECT_TIMEOUT", "UND_ERR_SOCKET",
|
||||
]);
|
||||
|
||||
type TransientProviderEvidence = { httpStatus?: number; errorCode?: string; acquisitionTimedOut?: true };
|
||||
/**
|
||||
* `reason` names evidence that has neither a provider status nor a deadline:
|
||||
* the provider reported a provisional creation that did not finish, or a
|
||||
* control-plane database query failed while the lease was taken.
|
||||
*/
|
||||
type TransientLeaseFailureReason = "acquisition_incomplete" | "database_error";
|
||||
type TransientProviderEvidence = { httpStatus?: number; errorCode?: string; acquisitionTimedOut?: true; reason?: TransientLeaseFailureReason };
|
||||
|
||||
function own(value: unknown, key: string): unknown {
|
||||
try {
|
||||
@@ -54,6 +61,69 @@ export function readTransientEnvironmentProviderFailure(error: unknown): Transie
|
||||
return null;
|
||||
}
|
||||
|
||||
/** A definite non-transient HTTP status anywhere on the chain is authoritative: retrying cannot change it. */
|
||||
function definiteNonTransientStatus(error: unknown): boolean {
|
||||
for (let cause = error, depth = 0; cause && depth < 5; cause = own(cause, "cause"), depth++) {
|
||||
const status = readEnvironmentAcquireErrorDiagnostic(cause)?.httpStatus
|
||||
?? [own(cause, "status"), own(cause, "statusCode")].find(value => Number.isInteger(value));
|
||||
if (typeof status === "number" && status >= 400 && !TRANSIENT_HTTP_STATUSES.has(status)) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
// postgres.js connection-level codes, and SQLSTATEs a repeated statement can
|
||||
// pass: connection (08), insufficient resources (53), operator intervention
|
||||
// (57P01-57P03, 57014 query cancelled or timed out), lock not available (55P03),
|
||||
// serialization failure and deadlock (40001, 40P01). A deterministic SQL error
|
||||
// (syntax, schema, constraint) is not retried.
|
||||
const DATABASE_CONNECTION_CODES = new Set(["CONNECTION_CLOSED", "CONNECTION_ENDED", "CONNECTION_DESTROYED", "CONNECT_TIMEOUT"]);
|
||||
const TRANSIENT_SQLSTATES = new Set(["57P01", "57P02", "57P03", "57014", "55P03", "40001", "40P01"]);
|
||||
|
||||
function transientDatabaseFailure(error: unknown): boolean {
|
||||
for (let cause = error, depth = 0; cause && depth < 5; cause = own(cause, "cause"), depth++) {
|
||||
const code = own(cause, "code");
|
||||
if (typeof code !== "string") continue;
|
||||
if (DATABASE_CONNECTION_CODES.has(code)) return true;
|
||||
if (/^[0-9A-Z]{5}$/.test(code) && (code.startsWith("08") || code.startsWith("53") || TRANSIENT_SQLSTATES.has(code))) {
|
||||
// A five-character code is a SQLSTATE only on a database driver error.
|
||||
const name = own(cause, "name");
|
||||
if (name === "PostgresError" || name === "DatabaseError" || own(cause, "severity") !== undefined) return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* Classify a failed lease acquisition for the bounded bootstrap retry. Lease
|
||||
* acquisition precedes any provider (agent) work by construction, and every
|
||||
* retry is a new creation attempt with its own idempotent identity, so a
|
||||
* failure is retried unless it carries a definite non-transient status. Only
|
||||
* structured evidence is read: the provider's status or errno, the validated
|
||||
* acquisition diagnostic the runtime recorded for this run, the plugin RPC
|
||||
* deadline, or a transient database driver code. Message text is never parsed.
|
||||
*/
|
||||
export function readTransientLeaseAcquisitionFailure(
|
||||
error: unknown,
|
||||
run: { id: string; companyId: string },
|
||||
): TransientProviderEvidence | null {
|
||||
const provider = readTransientEnvironmentProviderFailure(error);
|
||||
if (provider) return provider;
|
||||
if (definiteNonTransientStatus(error)) return null;
|
||||
for (let cause = error, depth = 0; cause && depth < 5; cause = own(cause, "cause"), depth++) {
|
||||
const acquisition = getEnvironmentAcquisitionDiagnostic(cause, run);
|
||||
if (acquisition) {
|
||||
return acquisition.elapsedMs >= acquisition.budgetMs ? { acquisitionTimedOut: true } : { reason: "acquisition_incomplete" };
|
||||
}
|
||||
if (cause instanceof JsonRpcCallError && cause.code === PLUGIN_RPC_ERROR_CODES.TIMEOUT) return { acquisitionTimedOut: true };
|
||||
}
|
||||
return transientDatabaseFailure(error) ? { reason: "database_error" } : null;
|
||||
}
|
||||
|
||||
const REASON_LABELS: Record<TransientLeaseFailureReason, string> = {
|
||||
acquisition_incomplete: "sandbox creation did not finish",
|
||||
database_error: "a database query failed while recording the lease",
|
||||
};
|
||||
|
||||
function category(evidence: TransientProviderEvidence): ProviderFailure["category"] {
|
||||
if (evidence.httpStatus === 429) return "rate_limit";
|
||||
if (evidence.acquisitionTimedOut || evidence.httpStatus === 408 || evidence.httpStatus === 504
|
||||
@@ -70,13 +140,15 @@ export class EnvironmentProviderUnavailableError extends Error {
|
||||
readonly providerFailure: ProviderFailure;
|
||||
|
||||
constructor(input: { environmentName: string; driver: string; evidence: TransientProviderEvidence; cause: unknown }) {
|
||||
const status = input.evidence.httpStatus !== undefined ? `HTTP ${input.evidence.httpStatus}` : input.evidence.errorCode;
|
||||
const status = input.evidence.httpStatus !== undefined ? `HTTP ${input.evidence.httpStatus}`
|
||||
: input.evidence.errorCode ?? (input.evidence.reason ? REASON_LABELS[input.evidence.reason] : undefined);
|
||||
super(status
|
||||
? `Environment provider for "${input.environmentName}" was temporarily unavailable (${status}).`
|
||||
: `Environment provider for "${input.environmentName}" did not finish starting the sandbox in time.`, { cause: input.cause });
|
||||
this.name = "EnvironmentProviderUnavailableError";
|
||||
this.httpStatus = input.evidence.httpStatus ?? null;
|
||||
this.errorCode = input.evidence.errorCode ?? null;
|
||||
const providerCode = input.evidence.errorCode ?? (input.evidence.httpStatus === undefined ? input.evidence.reason : undefined);
|
||||
this.providerFailure = {
|
||||
schema: "paperclip.provider-failure.v1",
|
||||
provider: /^[a-zA-Z0-9_.:/-]{1,80}$/.test(input.driver) ? input.driver : "environment",
|
||||
@@ -87,7 +159,7 @@ export class EnvironmentProviderUnavailableError extends Error {
|
||||
: `The sandbox for "${input.environmentName}" did not finish starting in time. `)
|
||||
+ "No agent work had started. Paperclip retries automatically; if those retries are used up, use Retry once the provider is available.",
|
||||
...(input.evidence.httpStatus !== undefined ? { httpStatus: input.evidence.httpStatus } : {}),
|
||||
...(input.evidence.errorCode !== undefined ? { providerCode: input.evidence.errorCode } : {}),
|
||||
...(providerCode !== undefined ? { providerCode } : {}),
|
||||
retryability: "transient",
|
||||
};
|
||||
}
|
||||
|
||||
@@ -22,7 +22,8 @@ import { environmentLeases, type Db } from "@paperclipai/db";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { remoteExecutionHasStopped } from "./remote-execution-termination.js";
|
||||
import { remoteRecoveryLeaseMismatch, type RemoteRecoveryLeaseMismatchField } from "./native-runtime/remote-recovery-lease-mismatch.js";
|
||||
import { EnvironmentProviderUnavailableError, readTransientEnvironmentProviderFailure } from "./environment-provider-unavailable-error.js";
|
||||
import { EnvironmentProviderUnavailableError, readTransientLeaseAcquisitionFailure } from "./environment-provider-unavailable-error.js";
|
||||
import { injectLeaseAcquisitionFault } from "./environment-lease-fault-injection.js";
|
||||
import { findReusableSandboxResumeError } from "./reusable-sandbox-resume-error.js";
|
||||
import { findPluginWorkerUnavailableError } from "./plugin-worker-unavailable-error.js";
|
||||
import type {
|
||||
@@ -232,13 +233,18 @@ export function environmentRunOrchestrator(
|
||||
runnerReconnectGraceMs?: number;
|
||||
}): Promise<EnvironmentRuntimeLeaseRecord> {
|
||||
try {
|
||||
// Restart-chaos EP1 only; disarmed (always in production) this is one null check.
|
||||
injectLeaseAcquisitionFault({ companyId: input.companyId, environmentId: input.environment.id,
|
||||
heartbeatRunId: input.heartbeatRunId, issueId: input.issueId });
|
||||
return await environmentRuntime.acquireRunLease({ ...input, assertCompanyBinding: true });
|
||||
} catch (err) {
|
||||
// A transient provider refusal (HTTP 5xx/429/timeout) is retried by the
|
||||
// setup path with its bounded bootstrap budget. Saved-sandbox resume and
|
||||
// Lease acquisition precedes any provider work, so a failure without a
|
||||
// definite non-transient status (HTTP 5xx/429, errno, acquisition
|
||||
// deadline, unfinished creation, database error) is retried by the setup
|
||||
// path with its bounded bootstrap budget. Saved-sandbox resume and
|
||||
// stopped-worker refusals keep their own, more specific classifications.
|
||||
const transient = findReusableSandboxResumeError(err) || findPluginWorkerUnavailableError(err)
|
||||
? null : readTransientEnvironmentProviderFailure(err);
|
||||
? null : readTransientLeaseAcquisitionFailure(err, { id: input.heartbeatRunId, companyId: input.companyId });
|
||||
throw new EnvironmentRunError(
|
||||
"lease_acquire_failed",
|
||||
`Failed to acquire lease for environment "${input.environment.name}" (${input.environment.driver}): ${err instanceof Error ? err.message : String(err)}`,
|
||||
|
||||
@@ -299,6 +299,7 @@ export type StrandedRecoveryCause =
|
||||
| "deliberate_wait_without_target"
|
||||
| "process_lost"
|
||||
| "provider_quota"
|
||||
| typeof ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE
|
||||
| "codex_output_inactivity_monitor"
|
||||
| "workspace_validation_failed"
|
||||
| "configuration_incomplete"
|
||||
@@ -444,12 +445,24 @@ function isProviderQuotaRecovery(latestRun: LatestIssueRun) {
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Lease acquisition failed before any provider work, and its bounded retry
|
||||
* budget is spent. Nothing needs inspection or repair by the board: the task
|
||||
* rests on a system-owned action and the normal Retry, never on a board one.
|
||||
*/
|
||||
function isPreDispatchEnvironmentProviderFailure(latestRun: LatestIssueRun) {
|
||||
if (latestRun?.errorCode !== ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE) return false;
|
||||
const recovery = parseObject(parseObject(latestRun.resultJson).executionRecovery);
|
||||
return recovery.kind === "bootstrap" && recovery.providerWorkStarted === false;
|
||||
}
|
||||
|
||||
function resolveStrandedRecoveryCause(
|
||||
latestRun: LatestIssueRun,
|
||||
explicitCause?: StrandedRecoveryCause,
|
||||
): StrandedRecoveryCause {
|
||||
if (explicitCause) return explicitCause;
|
||||
if (isProviderQuotaRecovery(latestRun)) return "provider_quota";
|
||||
if (isPreDispatchEnvironmentProviderFailure(latestRun)) return ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE;
|
||||
if (latestRun?.errorCode === "process_lost") return "process_lost";
|
||||
if (latestRun?.errorCode === "codex_output_inactivity_monitor") {
|
||||
return "codex_output_inactivity_monitor";
|
||||
@@ -2553,6 +2566,7 @@ export function recoveryService(
|
||||
latestRun: input.latestRun,
|
||||
});
|
||||
const isProviderQuotaWait = recoveryCause === "provider_quota";
|
||||
const isEnvironmentProviderWait = recoveryCause === ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE;
|
||||
const now = new Date();
|
||||
const actions = database === db ? recoveryActionsSvc : issueRecoveryActionService(database);
|
||||
const action = await actions.upsertSourceScoped({
|
||||
@@ -2565,7 +2579,7 @@ export function recoveryService(
|
||||
supersedeOnIdentityChange: recoveryCause === "configuration_incomplete",
|
||||
preserveExistingOwner: true,
|
||||
kind: strandedRecoveryActionKind(recoveryCause),
|
||||
ownerType: isProviderQuotaWait ? "system" : "board",
|
||||
ownerType: isProviderQuotaWait || isEnvironmentProviderWait ? "system" : "board",
|
||||
ownerAgentId: null,
|
||||
ownerUserId: null,
|
||||
previousOwnerAgentId: input.issue.assigneeAgentId,
|
||||
@@ -2587,7 +2601,7 @@ export function recoveryService(
|
||||
failureSummary:
|
||||
summarizeRunFailureForIssueComment(input.latestRun)?.trim() ?? null,
|
||||
},
|
||||
evidenceOnCreate: isProviderQuotaWait
|
||||
evidenceOnCreate: isProviderQuotaWait || isEnvironmentProviderWait
|
||||
? {}
|
||||
: { routingPolicy: STRANDED_BOARD_ESCALATION_POLICY },
|
||||
nextAction:
|
||||
@@ -2597,6 +2611,8 @@ export function recoveryService(
|
||||
? "Board operator: inspect the retry history, then explicitly retry the original owner, reassign, or intentionally resolve the task."
|
||||
: recoveryCause === "provider_quota"
|
||||
? "Wait for provider quota recovery, then retry the original assignee; do not wake a takeover owner."
|
||||
: isEnvironmentProviderWait
|
||||
? "No agent work started. Retry the task once the environment provider is available."
|
||||
: recoveryCause === "codex_output_inactivity_monitor"
|
||||
? "Board operator: inspect the inactivity evidence, then explicitly retry the original owner, reassign, or intentionally resolve the task."
|
||||
: recoveryCause === "workspace_validation_failed"
|
||||
@@ -2632,6 +2648,8 @@ export function recoveryService(
|
||||
type: "monitor_only",
|
||||
reason: recoveryCause,
|
||||
}
|
||||
: isEnvironmentProviderWait
|
||||
? { type: "manual_retry", reason: recoveryCause, preservesSourceAssignee: true }
|
||||
: {
|
||||
type: "board_escalation",
|
||||
reason: recoveryCause,
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { noticeMetadataReferencesRecoveryAction } from "./successful-run-handoff.js";
|
||||
import { ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE } from "../environment-provider-unavailable-error.js";
|
||||
import {
|
||||
DEFAULT_STRANDED_RECOVERY_NOTICE_BODY,
|
||||
buildConfigurationIncompleteRecoveryNoticeSeed,
|
||||
@@ -147,6 +148,19 @@ describe("buildStrandedRecoveryEscalationNotice", () => {
|
||||
expect(notice.body).toContain("before the agent started");
|
||||
expect(JSON.stringify(notice.metadata)).toContain("worker status");
|
||||
});
|
||||
it("asks for Retry, not a board decision, when the environment provider stayed unavailable", () => {
|
||||
const notice = buildStrandedRecoveryEscalationNotice({
|
||||
seed: buildImmediateExecutionPathRecoveryNoticeSeed({ status: "in_progress" }),
|
||||
recoveryCause: ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE,
|
||||
recoveryActionId: "provider-recovery",
|
||||
recoveryOwner: null,
|
||||
sourceRun: { id: "failed-run", status: "failed", errorCode: ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE },
|
||||
});
|
||||
const metadata = JSON.stringify(notice.metadata);
|
||||
expect(notice.presentation.title).toBe("Environment provider was temporarily unavailable");
|
||||
expect(metadata).toContain("Retry when ready");
|
||||
expect(metadata).not.toContain("Board decision required");
|
||||
});
|
||||
const actionId = "6a2f8e64-6f5e-4b58-b7fd-111111111111";
|
||||
const owner = { id: "9b1c2d3e-4f50-4a61-8b72-222222222222", name: "CTO" };
|
||||
const sourceRun = {
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import type { IssueCommentMetadata, IssueCommentPresentation } from "@paperclipai/shared";
|
||||
import { ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE } from "../environment-provider-unavailable-error.js";
|
||||
import {
|
||||
agentLinkRow,
|
||||
keyValueRow,
|
||||
@@ -281,7 +282,8 @@ export function buildStrandedRecoveryEscalationNotice(input: {
|
||||
? agentLinkRow("Recovery owner", input.recoveryOwner)
|
||||
: keyValueRow(
|
||||
"Recovery owner",
|
||||
"Board decision required",
|
||||
// A pre-dispatch provider outage needs no board decision, only Retry.
|
||||
input.recoveryCause === ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE ? "Retry when ready" : "Board decision required",
|
||||
),
|
||||
keyValueRow(
|
||||
"Next action",
|
||||
|
||||
Reference in new issue
Block a user