Files
PaperClipAI/ui/src/components/IssueScheduledRetryCard.test.tsx
T
DottaandClaude Opus 5.5 a665ed6a1b fix(recovery): wait out a sandbox provider outage with no Retry (#22139)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task that runs in a sandbox environment first gets a lease from
the environment provider (for example Daytona)
> - When the provider refuses the lease with a transient error
(`environment_provider_unavailable`), Paperclip retries twice, after 30
s and 60 s
> - After those two retries the task rests on "Retry when ready". A
provider outage longer than about 90 s therefore needs a person to click
Retry
> - Recovery must just work, and the target for runs that need Retry is
0%. The fix must not multiply automatic runs into a crash loop
> - This pull request adds a provider-outage wait after the fast
retries. The task stays system-owned, one waiter per environment checks
the provider on a capped backoff, and the other waiters continue by
themselves once the provider gives out a lease again. A person gets
Retry only after a 30-minute ceiling
> - The benefit is that a provider outage of up to 30 minutes delays
work instead of stranding it, without extra load on the provider

## Linked Issues or Issue Description

Refs #22087 (it classifies Daytona connection errors as transient, so
they reach this path).

**What happened?**
In a restart-chaos wave on 2026-11-17, the Daytona API host names did
not resolve from 19:40:28Z to 19:52:17Z (182 `ENOTFOUND` errors for the
API and proxy hosts). Tasks that started in that window failed with
`environment_provider_unavailable`. The two bounded retries (30 s, then
60 s) ran inside the outage, so these tasks rested on "Retry when ready"
and needed a person to continue them.

**Expected behavior**
Paperclip continues the task by itself when the provider is reachable
again. The task shows that it waits for the provider. It shows no Retry.
Only a long outage hands the task to a person, and then Retry works.

**Steps to reproduce**
1. Make lease acquisition fail with a transient provider error (HTTP 502
or a connection errno) for longer than about 90 s.
2. Assign a task to an agent that uses that environment.
3. After the second automatic retry fails, the task is blocked with a
Retry button, although the provider becomes available again later.

## What Changed

- `server/src/services/environment-provider-wait.ts` (new): the
`environment_provider_wait` retry lane. The delays are 1 min, 2 min,
then 5 min. The lane stops at 30 minutes after the wait started, with at
most 12 attempts. It also builds the task text "Waiting for
"<environment name>" to come back. Paperclip continues this task
automatically when it does."
- `scheduleBoundedRetryForRun` (`heartbeat/retries.ts`): when a run
fails with `environment_provider_unavailable` after the fast retries are
spent, it schedules the next attempt in the wait lane. The retry context
records the environment and the time the wait started. The time is kept
across attempts. The lane is uncharged
(`execution-recovery-attempt.ts`), so it does not spend the failure
budget. Other lanes drop the wait context.
- The environment orchestrator puts the environment id on
`EnvironmentProviderUnavailableError`, and the setup-failure path saves
it in `resultJson.environmentProvider`.
- Run dispatch (`run-dispatch/adapters/environment-provider-wait.ts`,
new): a due wait stays parked while an earlier waiter for the same
environment is still in line or still acquiring its lease, and no lease
was acquired in that environment since this waiter last failed. Parked
waiters check again every 30 s. This uses database reads only, and it
adds no work to the run start path. When a lease is acquired, the parked
waiters are promoted in arrival order. The automatic-run brake then
meters their starts as before.
- The probe must be able to start. A queued earlier waiter blocks later
waiters for only 60 s after promotion, and a running one only while it
is still acquiring its lease (at most 10 minutes). If the probe's agent
is busy, the next runnable waiter becomes the probe. A waiter past the
30-minute ceiling is never parked, so it makes its last attempt and the
exhaustion path runs.
- The notify-and-update mechanics of parking are in one helper,
`parkScheduledRetryInTx`
(`run-dispatch/adapters/scheduled-retry-park.ts`). The workspace-busy
wait and the provider wait use it and only decide whether to park and
until when.
- UI: `IssueScheduledRetryCard` shows nothing for
`environment_provider_wait` (as for `workspace_busy`). `IssueProperties`
shows "Sandbox: Waiting for the provider" instead of the scheduled-retry
picker. `IssueMonitorBanner` shows a steady "Waiting for the sandbox
provider" with no countdown and no Check now.
- Task presentation: a task in the wait lane is "working", system-owned,
with no Retry, and shows the wait text. After the ceiling the existing
path applies: the task is blocked with the system-owned notice and a
Retry that works. The notice and the provider-failure text now say that
Paperclip retried for 30 minutes.
- The per-task transient cap, the automatic-run brake and the bounded
fast retries are unchanged.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/heartbeat-process-recovery.test.ts -t "transient sandbox
provider refusal"` (embedded Postgres):
- New: "resumes ten tasks by itself when a five-minute provider outage
ends, one probe at a time and within the automatic-run brake". Ten tasks
in one company share one environment. The provider refuses leases for 5
minutes of simulated time (the test shifts all stored timestamps). All
ten tasks complete with no Retry. No task is blocked and no task offers
Retry while it waits. At most one waiter per tick is refused by the
provider. While the brake is open, at most one automatic start happens
per tick. All ten finish within 5 minutes after the outage ends. Red on
master: tasks become `blocked` after the fast retries. Green with this
change.
- New: "lets one waiter per environment probe the provider and releases
the others once a lease is acquired". The second waiter stays parked
while the first one is queued or running without a lease. It is promoted
after a lease is acquired. Red without the park step, green with it.
- Updated: "waits for the provider without a Retry, and offers a working
Retry only after the 30-minute ceiling". After the fast retries the task
is working with the wait text and no Retry. The backoff grows from 1 to
2 minutes. After the ceiling the task is blocked, Retry is offered, and
Retry runs the task.
- Updated: "rests a provider deadline that never clears on a working
Retry, never on a board recovery action". It now reaches the ceiling
first.
- New: "lets the next waiter probe when the first waiter cannot start
because its agent is busy" (red before, green after).
- New: "hands a parked task to a person with a working Retry at the
30-minute ceiling even while the leader stays queued" (red before, green
after).
- `src/services/environment-provider-wait.test.ts`: backoff, ceiling and
text.
- UI: `IssueScheduledRetryCard.test.tsx` ("offers no Retry while the
task waits for the sandbox provider to come back") and
`IssueMonitorBanner.test.tsx` ("shows a steady provider wait with no
countdown"). `pnpm check:token-gates` is clean.
- Also run locally: `heartbeat/retries.test.ts`,
`heartbeat/run-completion.test.ts` (all 163 tests pass; the file-level
`afterAll` teardown timed out on this loaded machine),
`task-interruption-query.test.ts`, `stranded-notice.test.ts`,
`execution-recovery-attempt.test.ts`,
`environment-provider-unavailable-error.test.ts`,
`task-interruption-presentation.test.ts`, the automatic-run brake tests,
and `src/modules/run-dispatch`.
- Server `tsc --noEmit` passes.

## Risks

- **Automatic-run volume:** while a provider is down, each environment
has one waiter that calls the provider on the backoff (1, 2, then 5
minutes). The other waiters only read the database every 30 s. Before
this change the volume was zero after the second retry, because the task
waited for a person. The new volume per environment is at most about 8
provider calls in 30 minutes. It does not grow with the number of tasks.
The automatic-run brake still opens after 5 consecutive provider
refusals, and it limits the probe to one start per minute.
- After the outage, waiting tasks start again as the brake allows (a
probe, then a release ramp of 2, 4, 8 and more starts per minute). They
do not all start at once.
- A run that was in the old fast lane when this deploys continues in the
new lane when its second retry fails. A run that is already blocked
keeps its Retry.
- No migration. The new fields are optional JSON in `resultJson` and
`contextSnapshot`.

## Model Used

- Claude Opus 5.5 (provider Anthropic, model ID `claude-opus-5-5`), with
tool use and code execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 01:24:13 -06:00

283 lines
10 KiB
TypeScript

// @vitest-environment jsdom
import { act } from "react";
import type { ComponentProps, ReactNode } from "react";
import { createRoot, type Root } from "react-dom/client";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import type { IssueRetryNowOutcome, IssueScheduledRetry } from "@paperclipai/shared";
import { IssueScheduledRetryCard } from "./IssueScheduledRetryCard";
import { ToastProvider } from "../context/ToastContext";
const retryNowMock = vi.hoisted(() => vi.fn());
vi.mock("@/lib/router", () => ({
Link: ({ children, to, ...props }: { children: ReactNode; to: string } & ComponentProps<"a">) => (
<a href={to} {...props}>{children}</a>
),
}));
vi.mock("../api/issues", () => ({
issuesApi: {
retryScheduledRetryNow: retryNowMock,
},
}));
(globalThis as unknown as { IS_REACT_ACT_ENVIRONMENT: boolean }).IS_REACT_ACT_ENVIRONMENT = true;
let container: HTMLDivElement;
let root: Root;
let dateNowSpy: ReturnType<typeof vi.spyOn> | null = null;
const SYSTEM_NOW = new Date("2026-04-18T20:00:00.000Z").getTime();
const baseRetry: IssueScheduledRetry = {
runId: "run-00000000",
status: "scheduled_retry",
agentId: "agent-1",
agentName: "ClaudeCoder",
retryOfRunId: "run-prev-1234567",
scheduledRetryAt: "2026-04-18T20:15:00.000Z",
scheduledRetryAttempt: 4,
scheduledRetryReason: "transient_failure",
retryExhaustedReason: null,
error: "Upstream provider rate limited",
errorCode: "rate_limited",
};
function buildRetryResponse(outcome: IssueRetryNowOutcome) {
return {
outcome,
message:
outcome === "promoted"
? "Promoted scheduled retry"
: outcome === "already_promoted"
? "Scheduled retry already promoted"
: outcome === "no_scheduled_retry"
? "No scheduled retry"
: "Promotion suppressed by gate",
scheduledRetry:
outcome === "promoted" || outcome === "already_promoted"
? { ...baseRetry, status: "queued" as const }
: null,
};
}
async function waitForUi(assertion: () => void) {
await vi.waitFor(async () => {
await act(async () => {
await Promise.resolve();
});
assertion();
});
}
async function waitForRetryButtonText(expected: string) {
for (let i = 0; i < 20; i += 1) {
if ((getRetryNowButton()?.textContent ?? "").includes(expected)) return;
// eslint-disable-next-line no-await-in-loop
await act(async () => {
await new Promise((resolve) => setTimeout(resolve, 0));
});
}
expect(getRetryNowButton()!.textContent ?? "").toContain(expected);
}
function renderWithProviders(ui: ReactNode) {
const queryClient = new QueryClient({
defaultOptions: { queries: { retry: false }, mutations: { retry: false } },
});
act(() => {
root.render(
<QueryClientProvider client={queryClient}>
<ToastProvider>{ui}</ToastProvider>
</QueryClientProvider>,
);
});
}
beforeEach(() => {
dateNowSpy = vi.spyOn(Date, "now").mockReturnValue(SYSTEM_NOW);
retryNowMock.mockReset();
container = document.createElement("div");
document.body.appendChild(container);
root = createRoot(container);
});
afterEach(() => {
act(() => root.unmount());
container.remove();
dateNowSpy?.mockRestore();
});
function getCard() {
return container.querySelector('[data-testid="issue-scheduled-retry-card"]');
}
function getRetryNowButton() {
return container.querySelector<HTMLButtonElement>(
'[data-testid="issue-scheduled-retry-card-retry-now"]',
);
}
describe("IssueScheduledRetryCard", () => {
it("raises no conversation notice for workspace contention the user cannot act on (N-14)", () => {
renderWithProviders(<IssueScheduledRetryCard issueId="issue-1" scheduledRetry={{ ...baseRetry, scheduledRetryReason: "workspace_busy" }} />);
expect(container.textContent).toBe("");
expect(getRetryNowButton()).toBeNull();
});
it("offers no Retry while the task waits for the sandbox provider to come back", () => {
renderWithProviders(<IssueScheduledRetryCard issueId="issue-1" scheduledRetry={{ ...baseRetry, scheduledRetryReason: "environment_provider_wait" }} />);
expect(container.textContent).toBe("");
expect(getRetryNowButton()).toBeNull();
});
it("surfaces model capacity and the automatic retry without a generic failure", () => {
renderWithProviders(<IssueScheduledRetryCard issueId="issue-1" scheduledRetry={{ ...baseRetry, scheduledRetryAttempt: 1, scheduledRetryReason: "native_provider_overloaded",
error: "Selected model is at capacity. Please try a different model.", errorCode: "native_provider_overloaded" }} />);
expect(container.textContent).toContain("Model at capacity");
expect(container.textContent).toContain("Selected model is at capacity");
expect(container.textContent).toContain("Paperclip will retry automatically");
expect(container.textContent).not.toContain("Native session failed");
});
it("renders nothing when there is no scheduled retry", () => {
renderWithProviders(<IssueScheduledRetryCard issueId="issue-1" scheduledRetry={null} />);
expect(getCard()).toBeNull();
});
it("renders nothing when status is not scheduled_retry", () => {
renderWithProviders(
<IssueScheduledRetryCard
issueId="issue-1"
scheduledRetry={{ ...baseRetry, status: "queued" }}
/>,
);
expect(getCard()).toBeNull();
});
it("shows attempt count, reason, absolute and relative timestamps", () => {
renderWithProviders(
<IssueScheduledRetryCard issueId="issue-1" scheduledRetry={baseRetry} />,
);
const card = getCard();
expect(card).not.toBeNull();
const text = card!.textContent ?? "";
expect(text).toContain("Retry scheduled");
expect(text).toContain("Attempt 4");
expect(text).toContain("Transient failure");
expect(text).toContain("Automatic retry in 15m");
expect(text).toContain("run-prev");
});
it("uses continuation copy for max-turn continuations", () => {
renderWithProviders(
<IssueScheduledRetryCard
issueId="issue-1"
scheduledRetry={{ ...baseRetry, scheduledRetryReason: "max_turns_continuation" }}
/>,
);
const text = getCard()?.textContent ?? "";
expect(text).toContain("Continuation scheduled");
expect(text).toContain("Automatic continuation");
expect(text).toContain("Pulls continuation forward immediately");
});
it("uses 'due now' label when scheduledRetryAt is at the current time", () => {
renderWithProviders(
<IssueScheduledRetryCard
issueId="issue-1"
scheduledRetry={{ ...baseRetry, scheduledRetryAt: "2026-04-18T20:00:10.000Z" }}
/>,
);
const text = getCard()?.textContent ?? "";
expect(text).toContain("Automatic retry due now");
});
it("invokes retry-now and shows promoted state on success", async () => {
retryNowMock.mockResolvedValue(buildRetryResponse("promoted"));
renderWithProviders(
<IssueScheduledRetryCard issueId="issue-1" scheduledRetry={baseRetry} />,
);
const button = getRetryNowButton();
expect(button).not.toBeNull();
act(() => {
button!.click();
});
await waitForRetryButtonText("Promoted");
expect(retryNowMock).toHaveBeenCalledWith("issue-1");
const finalButton = getRetryNowButton();
expect(finalButton!.textContent ?? "").toContain("Promoted");
expect(finalButton!.disabled).toBe(true);
});
it("shows already promoted state when backend reports duplicate click", async () => {
retryNowMock.mockResolvedValue(buildRetryResponse("already_promoted"));
renderWithProviders(
<IssueScheduledRetryCard issueId="issue-1" scheduledRetry={baseRetry} />,
);
act(() => {
getRetryNowButton()!.click();
});
await waitForRetryButtonText("Already promoted");
expect(getRetryNowButton()!.textContent ?? "").toContain("Already promoted");
expect(container.querySelector('[data-testid="issue-scheduled-retry-error-band"]')).toBeNull();
});
it("shows cleanup waiting without claiming success or disabling Retry now", async () => {
const message = "Waiting for execution cleanup. Paperclip will retry automatically once cleanup finishes.";
retryNowMock.mockResolvedValue({ outcome: "waiting", message, scheduledRetry: baseRetry });
renderWithProviders(<IssueScheduledRetryCard issueId="issue-1" scheduledRetry={baseRetry} />);
act(() => { getRetryNowButton()!.click(); });
await waitForUi(() => {
expect(getCard()?.textContent).toContain(message);
expect(getCard()?.textContent).not.toContain("run starting");
expect(getRetryNowButton()!.disabled).toBe(false);
expect(container.querySelector('[data-testid="issue-scheduled-retry-error-band"]')).toBeNull();
});
});
it("explains when there was no scheduled retry left to start", async () => {
const message = "No scheduled retry is waiting for this task.";
retryNowMock.mockResolvedValue({ outcome: "no_scheduled_retry", message, scheduledRetry: null });
renderWithProviders(<IssueScheduledRetryCard issueId="issue-1" scheduledRetry={baseRetry} />);
act(() => { getRetryNowButton()!.click(); });
await waitForUi(() => {
expect(getCard()?.textContent).toContain(message);
expect(getCard()?.textContent).not.toContain("run starting");
});
});
it("renders an inline error band on backend failure", async () => {
retryNowMock.mockRejectedValue(new Error("Server error"));
renderWithProviders(
<IssueScheduledRetryCard issueId="issue-1" scheduledRetry={baseRetry} />,
);
act(() => {
getRetryNowButton()!.click();
});
await waitForUi(() => {
const band = container.querySelector('[data-testid="issue-scheduled-retry-error-band"]');
expect(band).not.toBeNull();
expect((band?.textContent ?? "")).toContain("Server error");
expect(getRetryNowButton()!.disabled).toBe(false);
});
});
it("surfaces gate-suppressed outcome via the inline error band", async () => {
retryNowMock.mockResolvedValue(buildRetryResponse("gate_suppressed"));
renderWithProviders(
<IssueScheduledRetryCard issueId="issue-1" scheduledRetry={baseRetry} />,
);
act(() => {
getRetryNowButton()!.click();
});
await waitForUi(() => {
const band = container.querySelector('[data-testid="issue-scheduled-retry-error-band"]');
expect(band).not.toBeNull();
expect((band?.textContent ?? "")).toContain("Promotion suppressed");
expect(getRetryNowButton()!.disabled).toBe(false);
});
});
});