mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-20 15:09:57 +02:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs preserve provider events and results, then decide task state. > - Codex can fail a turn because the selected model is temporarily at capacity. > - Paperclip displayed a generic native-session failure and left the task in recovery. > - A known capacity failure needs a clear message and a durable delayed retry. > - This pull request uses committed terminal evidence, the status-effect ledger, and existing dispatch gates. > - The task can continue automatically without an unbounded retry loop or a model switch. ## Linked Issues or Issue Description Related: #13993 adds launcher capacity deferrals before provider startup. This change handles a committed native Codex `serverOverloaded` terminal after provider work has started. **What happened?** A native Codex turn ended with `codexErrorInfo: serverOverloaded` and “Selected model is at capacity. Please try a different model.” Paperclip saved the result, showed a generic failure, and required recovery instead of waiting and retrying. **Expected behavior** Show the capacity error directly. Schedule bounded retries after a short delay. Preserve saved work and honor current execution gates. **Steps to reproduce** 1. Run a native Codex task. 2. Commit a `turn.failed` event with `serverOverloaded`, then accept a failed result for the same turn. 3. Inspect the task error and its recovery state. The regression test reproduces this without a paid provider call. **Paperclip version or commit** Reproduced against master `16b7db35ffa0f9a95913c8cbdeea3d595435691f`. ## What Changed - Classify capacity failures from committed runner events and the pinned execution identity. Preserve accepted results and display the specific capacity error. - Atomically persist one scheduled successor with the status decision. Retry after one minute, then two minutes. Share the existing failure budget and stop after two automatic retries. - Reuse dispatch gates for ownership, task holds, dependencies, budget, and locks. Wait for predecessor execution, finalization, and cleanup before claiming a retry. - Preserve pending reviewer authority and suppress retries after reassignment or a successor claim. - Consume the failed run's resume receipt and delivered wake input. Rebuild ordinary continuation from the failed run so explicitly resumed tasks can retry without borrowing one-run authorization. - Label scheduled retries “Model at capacity.” Add a Storybook example and avoid duplicate punctuation in the existing retry card. - Add regression coverage and document the runtime contract. ## Verification - Targeted recovery and UI suites: 125 tests passed. Additional final cleanup and lock regressions passed. - Latest-head continuation and authorization regressions: 57 tests passed, including all four capacity integration tests against an isolated PostgreSQL database. - Rechecked all four capacity integration tests with the full runner's isolated `PAPERCLIP_HOME`, config, temporary directory, and serial fork settings: passed. - `pnpm -r typecheck`: passed. Final server typecheck passed. - `pnpm build`: passed. - `pnpm check:token-gates`: passed. - Browser: checked the real Storybook card. It shows the capacity message, automatic retry time, and existing Retry now action. - `pnpm test:run`: attempted and restarted after an interruption. The resumed run started before the final review correction and was stopped after recorded workspace/native test failures and the five-minute Git streaming timeout already documented on master. Exit 130; no complete local full-suite pass is claimed. Current-head isolated capacity tests and the complete CI suite pass. - A combined local recovery-suite attempt also hit PostgreSQL initialization failures at this macOS host's global shared-memory limit (32 slots). The focused recovery suites passed separately; final-head capacity tests also pass with the full runner's isolated environment settings. - Latest-head GitHub checks: all 56 checks green, including general and serialized test shards, browser E2E, typecheck, build, Runner verification, and canary dry run. No merge conflicts. - Greptile: 5/5 on `813a470b8c18c05aeb7e31e63e573c3a3a2f5cac`, with zero unresolved findings after fixing the resumed-task continuation issue. ## Risks - Capacity retries can repeat a task turn after partial work. They start a fresh provider session with task history and wait for predecessor cleanup. They do not resume the failed turn. - Retries retain the configured model unless an operator changes configuration. Persistent overload consumes the existing failure budget and then needs an explicit retry or model change. - Only run-bound native Codex `serverOverloaded` failures qualify. Usage-limit exhaustion, model/account incompatibility, unbound text, and unknown provider failures retain their current recovery behavior. - No schema migration is required. ## Model Used - OpenAI GPT-6 through Codex. The exact model ID and context-window size are not exposed to this session. Used reasoning, repository tools, code execution, and browser verification. No subagents. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
256 lines
9.0 KiB
TypeScript
256 lines
9.0 KiB
TypeScript
// @vitest-environment jsdom
|
|
|
|
import { act } from "react";
|
|
import type { ComponentProps, ReactNode } from "react";
|
|
import { createRoot, type Root } from "react-dom/client";
|
|
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
|
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
|
|
import type { IssueRetryNowOutcome, IssueScheduledRetry } from "@paperclipai/shared";
|
|
import { IssueScheduledRetryCard } from "./IssueScheduledRetryCard";
|
|
import { ToastProvider } from "../context/ToastContext";
|
|
|
|
const retryNowMock = vi.hoisted(() => vi.fn());
|
|
|
|
vi.mock("@/lib/router", () => ({
|
|
Link: ({ children, to, ...props }: { children: ReactNode; to: string } & ComponentProps<"a">) => (
|
|
<a href={to} {...props}>{children}</a>
|
|
),
|
|
}));
|
|
|
|
vi.mock("../api/issues", () => ({
|
|
issuesApi: {
|
|
retryScheduledRetryNow: retryNowMock,
|
|
},
|
|
}));
|
|
|
|
(globalThis as unknown as { IS_REACT_ACT_ENVIRONMENT: boolean }).IS_REACT_ACT_ENVIRONMENT = true;
|
|
|
|
let container: HTMLDivElement;
|
|
let root: Root;
|
|
let dateNowSpy: ReturnType<typeof vi.spyOn> | null = null;
|
|
|
|
const SYSTEM_NOW = new Date("2026-04-18T20:00:00.000Z").getTime();
|
|
|
|
const baseRetry: IssueScheduledRetry = {
|
|
runId: "run-00000000",
|
|
status: "scheduled_retry",
|
|
agentId: "agent-1",
|
|
agentName: "ClaudeCoder",
|
|
retryOfRunId: "run-prev-1234567",
|
|
scheduledRetryAt: "2026-04-18T20:15:00.000Z",
|
|
scheduledRetryAttempt: 4,
|
|
scheduledRetryReason: "transient_failure",
|
|
retryExhaustedReason: null,
|
|
error: "Upstream provider rate limited",
|
|
errorCode: "rate_limited",
|
|
};
|
|
|
|
function buildRetryResponse(outcome: IssueRetryNowOutcome) {
|
|
return {
|
|
outcome,
|
|
message:
|
|
outcome === "promoted"
|
|
? "Promoted scheduled retry"
|
|
: outcome === "already_promoted"
|
|
? "Scheduled retry already promoted"
|
|
: outcome === "no_scheduled_retry"
|
|
? "No scheduled retry"
|
|
: "Promotion suppressed by gate",
|
|
scheduledRetry:
|
|
outcome === "promoted" || outcome === "already_promoted"
|
|
? { ...baseRetry, status: "queued" as const }
|
|
: null,
|
|
};
|
|
}
|
|
|
|
async function waitForUi(assertion: () => void) {
|
|
await vi.waitFor(async () => {
|
|
await act(async () => {
|
|
await Promise.resolve();
|
|
});
|
|
assertion();
|
|
});
|
|
}
|
|
|
|
async function waitForRetryButtonText(expected: string) {
|
|
for (let i = 0; i < 20; i += 1) {
|
|
if ((getRetryNowButton()?.textContent ?? "").includes(expected)) return;
|
|
// eslint-disable-next-line no-await-in-loop
|
|
await act(async () => {
|
|
await new Promise((resolve) => setTimeout(resolve, 0));
|
|
});
|
|
}
|
|
expect(getRetryNowButton()!.textContent ?? "").toContain(expected);
|
|
}
|
|
|
|
function renderWithProviders(ui: ReactNode) {
|
|
const queryClient = new QueryClient({
|
|
defaultOptions: { queries: { retry: false }, mutations: { retry: false } },
|
|
});
|
|
act(() => {
|
|
root.render(
|
|
<QueryClientProvider client={queryClient}>
|
|
<ToastProvider>{ui}</ToastProvider>
|
|
</QueryClientProvider>,
|
|
);
|
|
});
|
|
}
|
|
|
|
beforeEach(() => {
|
|
dateNowSpy = vi.spyOn(Date, "now").mockReturnValue(SYSTEM_NOW);
|
|
retryNowMock.mockReset();
|
|
container = document.createElement("div");
|
|
document.body.appendChild(container);
|
|
root = createRoot(container);
|
|
});
|
|
|
|
afterEach(() => {
|
|
act(() => root.unmount());
|
|
container.remove();
|
|
dateNowSpy?.mockRestore();
|
|
});
|
|
|
|
function getCard() {
|
|
return container.querySelector('[data-testid="issue-scheduled-retry-card"]');
|
|
}
|
|
|
|
function getRetryNowButton() {
|
|
return container.querySelector<HTMLButtonElement>(
|
|
'[data-testid="issue-scheduled-retry-card-retry-now"]',
|
|
);
|
|
}
|
|
|
|
describe("IssueScheduledRetryCard", () => {
|
|
it("shows workspace contention as an automatic wait without failure or retry controls", () => {
|
|
renderWithProviders(<IssueScheduledRetryCard issueId="issue-1" scheduledRetry={{ ...baseRetry, scheduledRetryReason: "workspace_busy" }} />);
|
|
expect(container.textContent).toContain("Waiting for workspace");
|
|
expect(container.textContent).toContain("Work starts automatically");
|
|
expect(container.textContent).not.toMatch(/failed|Retry|Attempt|Replaces run/);
|
|
expect(getRetryNowButton()).toBeNull();
|
|
});
|
|
it("surfaces model capacity and the automatic retry without a generic failure", () => {
|
|
renderWithProviders(<IssueScheduledRetryCard issueId="issue-1" scheduledRetry={{ ...baseRetry, scheduledRetryAttempt: 1, scheduledRetryReason: "native_provider_overloaded",
|
|
error: "Selected model is at capacity. Please try a different model.", errorCode: "native_provider_overloaded" }} />);
|
|
expect(container.textContent).toContain("Model at capacity");
|
|
expect(container.textContent).toContain("Selected model is at capacity");
|
|
expect(container.textContent).toContain("Paperclip will retry automatically");
|
|
expect(container.textContent).not.toContain("Native session failed");
|
|
});
|
|
|
|
it("renders nothing when there is no scheduled retry", () => {
|
|
renderWithProviders(<IssueScheduledRetryCard issueId="issue-1" scheduledRetry={null} />);
|
|
expect(getCard()).toBeNull();
|
|
});
|
|
|
|
it("renders nothing when status is not scheduled_retry", () => {
|
|
renderWithProviders(
|
|
<IssueScheduledRetryCard
|
|
issueId="issue-1"
|
|
scheduledRetry={{ ...baseRetry, status: "queued" }}
|
|
/>,
|
|
);
|
|
expect(getCard()).toBeNull();
|
|
});
|
|
|
|
it("shows attempt count, reason, absolute and relative timestamps", () => {
|
|
renderWithProviders(
|
|
<IssueScheduledRetryCard issueId="issue-1" scheduledRetry={baseRetry} />,
|
|
);
|
|
const card = getCard();
|
|
expect(card).not.toBeNull();
|
|
const text = card!.textContent ?? "";
|
|
expect(text).toContain("Retry scheduled");
|
|
expect(text).toContain("Attempt 4");
|
|
expect(text).toContain("Transient failure");
|
|
expect(text).toContain("Automatic retry in 15m");
|
|
expect(text).toContain("run-prev");
|
|
});
|
|
|
|
it("uses continuation copy for max-turn continuations", () => {
|
|
renderWithProviders(
|
|
<IssueScheduledRetryCard
|
|
issueId="issue-1"
|
|
scheduledRetry={{ ...baseRetry, scheduledRetryReason: "max_turns_continuation" }}
|
|
/>,
|
|
);
|
|
const text = getCard()?.textContent ?? "";
|
|
expect(text).toContain("Continuation scheduled");
|
|
expect(text).toContain("Automatic continuation");
|
|
expect(text).toContain("Pulls continuation forward immediately");
|
|
});
|
|
|
|
it("uses 'due now' label when scheduledRetryAt is at the current time", () => {
|
|
renderWithProviders(
|
|
<IssueScheduledRetryCard
|
|
issueId="issue-1"
|
|
scheduledRetry={{ ...baseRetry, scheduledRetryAt: "2026-04-18T20:00:10.000Z" }}
|
|
/>,
|
|
);
|
|
const text = getCard()?.textContent ?? "";
|
|
expect(text).toContain("Automatic retry due now");
|
|
});
|
|
|
|
it("invokes retry-now and shows promoted state on success", async () => {
|
|
retryNowMock.mockResolvedValue(buildRetryResponse("promoted"));
|
|
renderWithProviders(
|
|
<IssueScheduledRetryCard issueId="issue-1" scheduledRetry={baseRetry} />,
|
|
);
|
|
const button = getRetryNowButton();
|
|
expect(button).not.toBeNull();
|
|
|
|
act(() => {
|
|
button!.click();
|
|
});
|
|
await waitForRetryButtonText("Promoted");
|
|
expect(retryNowMock).toHaveBeenCalledWith("issue-1");
|
|
const finalButton = getRetryNowButton();
|
|
expect(finalButton!.textContent ?? "").toContain("Promoted");
|
|
expect(finalButton!.disabled).toBe(true);
|
|
});
|
|
|
|
it("shows already promoted state when backend reports duplicate click", async () => {
|
|
retryNowMock.mockResolvedValue(buildRetryResponse("already_promoted"));
|
|
renderWithProviders(
|
|
<IssueScheduledRetryCard issueId="issue-1" scheduledRetry={baseRetry} />,
|
|
);
|
|
act(() => {
|
|
getRetryNowButton()!.click();
|
|
});
|
|
await waitForRetryButtonText("Already promoted");
|
|
expect(getRetryNowButton()!.textContent ?? "").toContain("Already promoted");
|
|
expect(container.querySelector('[data-testid="issue-scheduled-retry-error-band"]')).toBeNull();
|
|
});
|
|
|
|
it("renders an inline error band on backend failure", async () => {
|
|
retryNowMock.mockRejectedValue(new Error("Server error"));
|
|
renderWithProviders(
|
|
<IssueScheduledRetryCard issueId="issue-1" scheduledRetry={baseRetry} />,
|
|
);
|
|
act(() => {
|
|
getRetryNowButton()!.click();
|
|
});
|
|
await waitForUi(() => {
|
|
const band = container.querySelector('[data-testid="issue-scheduled-retry-error-band"]');
|
|
expect(band).not.toBeNull();
|
|
expect((band?.textContent ?? "")).toContain("Server error");
|
|
expect(getRetryNowButton()!.disabled).toBe(false);
|
|
});
|
|
});
|
|
|
|
it("surfaces gate-suppressed outcome via the inline error band", async () => {
|
|
retryNowMock.mockResolvedValue(buildRetryResponse("gate_suppressed"));
|
|
renderWithProviders(
|
|
<IssueScheduledRetryCard issueId="issue-1" scheduledRetry={baseRetry} />,
|
|
);
|
|
act(() => {
|
|
getRetryNowButton()!.click();
|
|
});
|
|
await waitForUi(() => {
|
|
const band = container.querySelector('[data-testid="issue-scheduled-retry-error-band"]');
|
|
expect(band).not.toBeNull();
|
|
expect((band?.textContent ?? "")).toContain("Promotion suppressed");
|
|
expect(getRetryNowButton()!.disabled).toBe(false);
|
|
});
|
|
});
|
|
});
|