mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-21 00:05:49 +02:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Recovery schedules bounded retries after a provider disconnects. > - A stopped run can still hold its environment lease while cleanup runs. > - Retrying before that lease is released cancels the new run before it starts and spends another retry. > - Restoring the task can also send the worker repair instructions from an already resolved recovery action. > - This pull request preserves the waiting retry and removes settled recovery instructions from later wakes. > - The task can continue after cleanup without an operator repairing the same incident again. ## Linked Issues or Issue Description **What happened?** A legacy conversation run disconnected while it was doing ordinary work. Its environment cleanup took longer than the retry delay. Two retries were cancelled before dispatch with `execution_reconciliation_required`. Those cancellations exhausted the failure budget. After an operator restored the task, the wake still told the original worker to repair the runtime and hand the task back to itself. **Expected behavior** Cleanup waits preserve the pending attempt. The same retry can continue after ownership is released, subject to all current gates. Once a recovery action is resolved or cancelled, subsequent task wakes omit its repair instructions. **Steps to reproduce** 1. Fail a legacy conversation run while its environment lease remains in `pending_cleanup`. 2. Schedule a bounded retry and run promotion before cleanup releases that lease. 3. Repeat the scheduler sweep. Before this fix, retries promote and then cancel without starting. 4. Resolve a stranded-task recovery action and build the restored task wake with that action ID. Before this fix, the wake still includes the settled repair instructions. **Paperclip version or commit** Reproduced with database regressions against `ceabc3bc880` on master. Related: #15019 restores a skipped assignment handoff after lease release. #15235 filters stale handoff evidence in the recovery sweep. This change preserves an existing scheduled conversation retry and corrects restored wake content. It does not create a new handoff wake. ## What Changed - Keep an unstarted legacy conversation retry on the same durable row while prior execution ownership remains active. Recheck after 30 seconds without increasing retry accounting. - Return a queued retry to scheduled state if it encounters that hold at the claim gate. Retain its issue claim and publish the status change. - Record one local lifecycle diagnostic per blocking run. Remove that wait marker on promotion. - Include recovery action metadata only while the referenced action is active or escalated. - Return an explicit `waiting` response and the saved schedule when Retry now meets cleanup. Show the wait inline without a false success or disabled button. - Add database, rendered-prompt, route, and UI regressions. Document the execution and run-log contracts. ## Verification - Five cleanup and restored-wake regressions fail against the original production code. The Retry now route and UI regressions also fail before their correction. - Related retry, dispatch, stale-queue, and recovery suites: 321 tests pass across seven files. All ten focused cleanup/restored-wake cases pass after rebase. The final dispatch adjustment passes all 46 adapter tests. - Retry now routes and affected UI suites: all 46 tests pass. The tests cover repeated clicks, the saved schedule, unchanged accounting, promotion after release, and no false success or error state. - `pnpm -r typecheck`, `pnpm build`, `pnpm build-storybook`, and `pnpm check:token-gates` pass. - Full local `pnpm test:run` was started and then stopped after the final commit passed all GitHub CI test shards. No complete local full-suite result is claimed; CI supplies the complete test result for the final commit. - Final head `6f1058a332c039e33c4f002b296d20a5554e760e`: all 55 GitHub checks are green or intentionally skipped. Apex review is 5/5 after two reviews, with no unresolved threads. The PR has no merge conflicts. ## Risks The wait applies only to unstarted legacy conversation retries. Native runs and non-conversation execution keep their existing recovery rules. Cleanup must actually release ownership before execution can resume. The wait does not fix a cleanup service that never finishes. Promotion and dispatch still enforce cancellation, reassignment, pause, budget, and reconciliation gates. No schema change is required. The Retry now response adds a `waiting` outcome; the shared contract and all three UI controls handle it. ## Model Used OpenAI Codex based on GPT-6, with repository analysis, tool use, and local code execution. The runtime does not expose the precise serving model ID or context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused suites; complete final-head test coverage in CI) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
269 lines
9.7 KiB
TypeScript
269 lines
9.7 KiB
TypeScript
// @vitest-environment jsdom
|
|
|
|
import { act } from "react";
|
|
import type { ComponentProps, ReactNode } from "react";
|
|
import { createRoot, type Root } from "react-dom/client";
|
|
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
|
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
|
|
import type { IssueRetryNowOutcome, IssueScheduledRetry } from "@paperclipai/shared";
|
|
import { IssueScheduledRetryCard } from "./IssueScheduledRetryCard";
|
|
import { ToastProvider } from "../context/ToastContext";
|
|
|
|
const retryNowMock = vi.hoisted(() => vi.fn());
|
|
|
|
vi.mock("@/lib/router", () => ({
|
|
Link: ({ children, to, ...props }: { children: ReactNode; to: string } & ComponentProps<"a">) => (
|
|
<a href={to} {...props}>{children}</a>
|
|
),
|
|
}));
|
|
|
|
vi.mock("../api/issues", () => ({
|
|
issuesApi: {
|
|
retryScheduledRetryNow: retryNowMock,
|
|
},
|
|
}));
|
|
|
|
(globalThis as unknown as { IS_REACT_ACT_ENVIRONMENT: boolean }).IS_REACT_ACT_ENVIRONMENT = true;
|
|
|
|
let container: HTMLDivElement;
|
|
let root: Root;
|
|
let dateNowSpy: ReturnType<typeof vi.spyOn> | null = null;
|
|
|
|
const SYSTEM_NOW = new Date("2026-04-18T20:00:00.000Z").getTime();
|
|
|
|
const baseRetry: IssueScheduledRetry = {
|
|
runId: "run-00000000",
|
|
status: "scheduled_retry",
|
|
agentId: "agent-1",
|
|
agentName: "ClaudeCoder",
|
|
retryOfRunId: "run-prev-1234567",
|
|
scheduledRetryAt: "2026-04-18T20:15:00.000Z",
|
|
scheduledRetryAttempt: 4,
|
|
scheduledRetryReason: "transient_failure",
|
|
retryExhaustedReason: null,
|
|
error: "Upstream provider rate limited",
|
|
errorCode: "rate_limited",
|
|
};
|
|
|
|
function buildRetryResponse(outcome: IssueRetryNowOutcome) {
|
|
return {
|
|
outcome,
|
|
message:
|
|
outcome === "promoted"
|
|
? "Promoted scheduled retry"
|
|
: outcome === "already_promoted"
|
|
? "Scheduled retry already promoted"
|
|
: outcome === "no_scheduled_retry"
|
|
? "No scheduled retry"
|
|
: "Promotion suppressed by gate",
|
|
scheduledRetry:
|
|
outcome === "promoted" || outcome === "already_promoted"
|
|
? { ...baseRetry, status: "queued" as const }
|
|
: null,
|
|
};
|
|
}
|
|
|
|
async function waitForUi(assertion: () => void) {
|
|
await vi.waitFor(async () => {
|
|
await act(async () => {
|
|
await Promise.resolve();
|
|
});
|
|
assertion();
|
|
});
|
|
}
|
|
|
|
async function waitForRetryButtonText(expected: string) {
|
|
for (let i = 0; i < 20; i += 1) {
|
|
if ((getRetryNowButton()?.textContent ?? "").includes(expected)) return;
|
|
// eslint-disable-next-line no-await-in-loop
|
|
await act(async () => {
|
|
await new Promise((resolve) => setTimeout(resolve, 0));
|
|
});
|
|
}
|
|
expect(getRetryNowButton()!.textContent ?? "").toContain(expected);
|
|
}
|
|
|
|
function renderWithProviders(ui: ReactNode) {
|
|
const queryClient = new QueryClient({
|
|
defaultOptions: { queries: { retry: false }, mutations: { retry: false } },
|
|
});
|
|
act(() => {
|
|
root.render(
|
|
<QueryClientProvider client={queryClient}>
|
|
<ToastProvider>{ui}</ToastProvider>
|
|
</QueryClientProvider>,
|
|
);
|
|
});
|
|
}
|
|
|
|
beforeEach(() => {
|
|
dateNowSpy = vi.spyOn(Date, "now").mockReturnValue(SYSTEM_NOW);
|
|
retryNowMock.mockReset();
|
|
container = document.createElement("div");
|
|
document.body.appendChild(container);
|
|
root = createRoot(container);
|
|
});
|
|
|
|
afterEach(() => {
|
|
act(() => root.unmount());
|
|
container.remove();
|
|
dateNowSpy?.mockRestore();
|
|
});
|
|
|
|
function getCard() {
|
|
return container.querySelector('[data-testid="issue-scheduled-retry-card"]');
|
|
}
|
|
|
|
function getRetryNowButton() {
|
|
return container.querySelector<HTMLButtonElement>(
|
|
'[data-testid="issue-scheduled-retry-card-retry-now"]',
|
|
);
|
|
}
|
|
|
|
describe("IssueScheduledRetryCard", () => {
|
|
it("shows workspace contention as an automatic wait without failure or retry controls", () => {
|
|
renderWithProviders(<IssueScheduledRetryCard issueId="issue-1" scheduledRetry={{ ...baseRetry, scheduledRetryReason: "workspace_busy" }} />);
|
|
expect(container.textContent).toContain("Waiting for workspace");
|
|
expect(container.textContent).toContain("Work starts automatically");
|
|
expect(container.textContent).not.toMatch(/failed|Retry|Attempt|Replaces run/);
|
|
expect(getRetryNowButton()).toBeNull();
|
|
});
|
|
it("surfaces model capacity and the automatic retry without a generic failure", () => {
|
|
renderWithProviders(<IssueScheduledRetryCard issueId="issue-1" scheduledRetry={{ ...baseRetry, scheduledRetryAttempt: 1, scheduledRetryReason: "native_provider_overloaded",
|
|
error: "Selected model is at capacity. Please try a different model.", errorCode: "native_provider_overloaded" }} />);
|
|
expect(container.textContent).toContain("Model at capacity");
|
|
expect(container.textContent).toContain("Selected model is at capacity");
|
|
expect(container.textContent).toContain("Paperclip will retry automatically");
|
|
expect(container.textContent).not.toContain("Native session failed");
|
|
});
|
|
|
|
it("renders nothing when there is no scheduled retry", () => {
|
|
renderWithProviders(<IssueScheduledRetryCard issueId="issue-1" scheduledRetry={null} />);
|
|
expect(getCard()).toBeNull();
|
|
});
|
|
|
|
it("renders nothing when status is not scheduled_retry", () => {
|
|
renderWithProviders(
|
|
<IssueScheduledRetryCard
|
|
issueId="issue-1"
|
|
scheduledRetry={{ ...baseRetry, status: "queued" }}
|
|
/>,
|
|
);
|
|
expect(getCard()).toBeNull();
|
|
});
|
|
|
|
it("shows attempt count, reason, absolute and relative timestamps", () => {
|
|
renderWithProviders(
|
|
<IssueScheduledRetryCard issueId="issue-1" scheduledRetry={baseRetry} />,
|
|
);
|
|
const card = getCard();
|
|
expect(card).not.toBeNull();
|
|
const text = card!.textContent ?? "";
|
|
expect(text).toContain("Retry scheduled");
|
|
expect(text).toContain("Attempt 4");
|
|
expect(text).toContain("Transient failure");
|
|
expect(text).toContain("Automatic retry in 15m");
|
|
expect(text).toContain("run-prev");
|
|
});
|
|
|
|
it("uses continuation copy for max-turn continuations", () => {
|
|
renderWithProviders(
|
|
<IssueScheduledRetryCard
|
|
issueId="issue-1"
|
|
scheduledRetry={{ ...baseRetry, scheduledRetryReason: "max_turns_continuation" }}
|
|
/>,
|
|
);
|
|
const text = getCard()?.textContent ?? "";
|
|
expect(text).toContain("Continuation scheduled");
|
|
expect(text).toContain("Automatic continuation");
|
|
expect(text).toContain("Pulls continuation forward immediately");
|
|
});
|
|
|
|
it("uses 'due now' label when scheduledRetryAt is at the current time", () => {
|
|
renderWithProviders(
|
|
<IssueScheduledRetryCard
|
|
issueId="issue-1"
|
|
scheduledRetry={{ ...baseRetry, scheduledRetryAt: "2026-04-18T20:00:10.000Z" }}
|
|
/>,
|
|
);
|
|
const text = getCard()?.textContent ?? "";
|
|
expect(text).toContain("Automatic retry due now");
|
|
});
|
|
|
|
it("invokes retry-now and shows promoted state on success", async () => {
|
|
retryNowMock.mockResolvedValue(buildRetryResponse("promoted"));
|
|
renderWithProviders(
|
|
<IssueScheduledRetryCard issueId="issue-1" scheduledRetry={baseRetry} />,
|
|
);
|
|
const button = getRetryNowButton();
|
|
expect(button).not.toBeNull();
|
|
|
|
act(() => {
|
|
button!.click();
|
|
});
|
|
await waitForRetryButtonText("Promoted");
|
|
expect(retryNowMock).toHaveBeenCalledWith("issue-1");
|
|
const finalButton = getRetryNowButton();
|
|
expect(finalButton!.textContent ?? "").toContain("Promoted");
|
|
expect(finalButton!.disabled).toBe(true);
|
|
});
|
|
|
|
it("shows already promoted state when backend reports duplicate click", async () => {
|
|
retryNowMock.mockResolvedValue(buildRetryResponse("already_promoted"));
|
|
renderWithProviders(
|
|
<IssueScheduledRetryCard issueId="issue-1" scheduledRetry={baseRetry} />,
|
|
);
|
|
act(() => {
|
|
getRetryNowButton()!.click();
|
|
});
|
|
await waitForRetryButtonText("Already promoted");
|
|
expect(getRetryNowButton()!.textContent ?? "").toContain("Already promoted");
|
|
expect(container.querySelector('[data-testid="issue-scheduled-retry-error-band"]')).toBeNull();
|
|
});
|
|
|
|
it("shows cleanup waiting without claiming success or disabling Retry now", async () => {
|
|
const message = "Waiting for execution cleanup. Paperclip will retry automatically once cleanup finishes.";
|
|
retryNowMock.mockResolvedValue({ outcome: "waiting", message, scheduledRetry: baseRetry });
|
|
renderWithProviders(<IssueScheduledRetryCard issueId="issue-1" scheduledRetry={baseRetry} />);
|
|
act(() => { getRetryNowButton()!.click(); });
|
|
await waitForUi(() => {
|
|
expect(getCard()?.textContent).toContain(message);
|
|
expect(getCard()?.textContent).not.toContain("run starting");
|
|
expect(getRetryNowButton()!.disabled).toBe(false);
|
|
expect(container.querySelector('[data-testid="issue-scheduled-retry-error-band"]')).toBeNull();
|
|
});
|
|
});
|
|
|
|
it("renders an inline error band on backend failure", async () => {
|
|
retryNowMock.mockRejectedValue(new Error("Server error"));
|
|
renderWithProviders(
|
|
<IssueScheduledRetryCard issueId="issue-1" scheduledRetry={baseRetry} />,
|
|
);
|
|
act(() => {
|
|
getRetryNowButton()!.click();
|
|
});
|
|
await waitForUi(() => {
|
|
const band = container.querySelector('[data-testid="issue-scheduled-retry-error-band"]');
|
|
expect(band).not.toBeNull();
|
|
expect((band?.textContent ?? "")).toContain("Server error");
|
|
expect(getRetryNowButton()!.disabled).toBe(false);
|
|
});
|
|
});
|
|
|
|
it("surfaces gate-suppressed outcome via the inline error band", async () => {
|
|
retryNowMock.mockResolvedValue(buildRetryResponse("gate_suppressed"));
|
|
renderWithProviders(
|
|
<IssueScheduledRetryCard issueId="issue-1" scheduledRetry={baseRetry} />,
|
|
);
|
|
act(() => {
|
|
getRetryNowButton()!.click();
|
|
});
|
|
await waitForUi(() => {
|
|
const band = container.querySelector('[data-testid="issue-scheduled-retry-error-band"]');
|
|
expect(band).not.toBeNull();
|
|
expect((band?.textContent ?? "")).toContain("Promotion suppressed");
|
|
expect(getRetryNowButton()!.disabled).toBe(false);
|
|
});
|
|
});
|
|
});
|