mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-23 00:53:38 +02:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs preserve provider events and results, then decide task state. > - Codex can fail a turn because the selected model is temporarily at capacity. > - Paperclip displayed a generic native-session failure and left the task in recovery. > - A known capacity failure needs a clear message and a durable delayed retry. > - This pull request uses committed terminal evidence, the status-effect ledger, and existing dispatch gates. > - The task can continue automatically without an unbounded retry loop or a model switch. ## Linked Issues or Issue Description Related: #13993 adds launcher capacity deferrals before provider startup. This change handles a committed native Codex `serverOverloaded` terminal after provider work has started. **What happened?** A native Codex turn ended with `codexErrorInfo: serverOverloaded` and “Selected model is at capacity. Please try a different model.” Paperclip saved the result, showed a generic failure, and required recovery instead of waiting and retrying. **Expected behavior** Show the capacity error directly. Schedule bounded retries after a short delay. Preserve saved work and honor current execution gates. **Steps to reproduce** 1. Run a native Codex task. 2. Commit a `turn.failed` event with `serverOverloaded`, then accept a failed result for the same turn. 3. Inspect the task error and its recovery state. The regression test reproduces this without a paid provider call. **Paperclip version or commit** Reproduced against master `16b7db35ffa0f9a95913c8cbdeea3d595435691f`. ## What Changed - Classify capacity failures from committed runner events and the pinned execution identity. Preserve accepted results and display the specific capacity error. - Atomically persist one scheduled successor with the status decision. Retry after one minute, then two minutes. Share the existing failure budget and stop after two automatic retries. - Reuse dispatch gates for ownership, task holds, dependencies, budget, and locks. Wait for predecessor execution, finalization, and cleanup before claiming a retry. - Preserve pending reviewer authority and suppress retries after reassignment or a successor claim. - Consume the failed run's resume receipt and delivered wake input. Rebuild ordinary continuation from the failed run so explicitly resumed tasks can retry without borrowing one-run authorization. - Label scheduled retries “Model at capacity.” Add a Storybook example and avoid duplicate punctuation in the existing retry card. - Add regression coverage and document the runtime contract. ## Verification - Targeted recovery and UI suites: 125 tests passed. Additional final cleanup and lock regressions passed. - Latest-head continuation and authorization regressions: 57 tests passed, including all four capacity integration tests against an isolated PostgreSQL database. - Rechecked all four capacity integration tests with the full runner's isolated `PAPERCLIP_HOME`, config, temporary directory, and serial fork settings: passed. - `pnpm -r typecheck`: passed. Final server typecheck passed. - `pnpm build`: passed. - `pnpm check:token-gates`: passed. - Browser: checked the real Storybook card. It shows the capacity message, automatic retry time, and existing Retry now action. - `pnpm test:run`: attempted and restarted after an interruption. The resumed run started before the final review correction and was stopped after recorded workspace/native test failures and the five-minute Git streaming timeout already documented on master. Exit 130; no complete local full-suite pass is claimed. Current-head isolated capacity tests and the complete CI suite pass. - A combined local recovery-suite attempt also hit PostgreSQL initialization failures at this macOS host's global shared-memory limit (32 slots). The focused recovery suites passed separately; final-head capacity tests also pass with the full runner's isolated environment settings. - Latest-head GitHub checks: all 56 checks green, including general and serialized test shards, browser E2E, typecheck, build, Runner verification, and canary dry run. No merge conflicts. - Greptile: 5/5 on `813a470b8c18c05aeb7e31e63e573c3a3a2f5cac`, with zero unresolved findings after fixing the resumed-task continuation issue. ## Risks - Capacity retries can repeat a task turn after partial work. They start a fresh provider session with task history and wait for predecessor cleanup. They do not resume the failed turn. - Retries retain the configured model unless an operator changes configuration. Persistent overload consumes the existing failure budget and then needs an explicit retry or model change. - Only run-bound native Codex `serverOverloaded` failures qualify. Usage-limit exhaustion, model/account incompatibility, unbound text, and unknown provider failures retain their current recovery behavior. - No schema migration is required. ## Model Used - OpenAI GPT-6 through Codex. The exact model ID and context-window size are not exposed to this session. Used reasoning, repository tools, code execution, and browser verification. No subagents. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
205 lines
7.8 KiB
TypeScript
205 lines
7.8 KiB
TypeScript
import { Clock, RotateCcw, AlertCircle, Loader2, CheckCircle2 } from "lucide-react";
|
|
import { Link } from "@/lib/router";
|
|
import { Button } from "@/components/ui/button";
|
|
import { cn, formatDateTime } from "@/lib/utils";
|
|
import { formatMonitorOffset } from "@/lib/issue-monitor";
|
|
import { formatRetryReason } from "@/lib/runRetryState";
|
|
import type { IssueScheduledRetry } from "@paperclipai/shared";
|
|
import { useRetryNowMutation, type RetryNowError } from "../hooks/useRetryNowMutation";
|
|
import { Badge } from "@/components/ui/badge";
|
|
import { InlineBanner } from "@/components/InlineBanner";
|
|
|
|
const MAX_TURN_CONTINUATION = "max_turns_continuation";
|
|
|
|
function isContinuationReason(reason: string | null | undefined) {
|
|
return reason === MAX_TURN_CONTINUATION;
|
|
}
|
|
|
|
function shortRunId(runId: string | null | undefined) {
|
|
return typeof runId === "string" && runId.length >= 8 ? runId.slice(0, 8) : runId ?? "";
|
|
}
|
|
|
|
interface IssueScheduledRetryCardProps {
|
|
issueId: string | null | undefined;
|
|
scheduledRetry: IssueScheduledRetry | null | undefined;
|
|
}
|
|
|
|
export function IssueScheduledRetryCard({
|
|
issueId,
|
|
scheduledRetry,
|
|
}: IssueScheduledRetryCardProps) {
|
|
const retryNow = useRetryNowMutation(issueId);
|
|
|
|
if (!scheduledRetry || !issueId) return null;
|
|
if (scheduledRetry.status !== "scheduled_retry") return null;
|
|
|
|
if (scheduledRetry.scheduledRetryReason === "workspace_busy") {
|
|
return (
|
|
<InlineBanner tone="info" icon={Clock} title="Waiting for workspace" className="mb-3">
|
|
Another task is using this workspace. Work starts automatically when it is available.
|
|
</InlineBanner>
|
|
);
|
|
}
|
|
|
|
const continuation = isContinuationReason(scheduledRetry.scheduledRetryReason);
|
|
const dueAtIso = scheduledRetry.scheduledRetryAt
|
|
? new Date(scheduledRetry.scheduledRetryAt).toISOString()
|
|
: null;
|
|
const relative = dueAtIso ? formatMonitorOffset(dueAtIso) : null;
|
|
const absolute = scheduledRetry.scheduledRetryAt
|
|
? formatDateTime(scheduledRetry.scheduledRetryAt)
|
|
: null;
|
|
const reason = formatRetryReason(scheduledRetry.scheduledRetryReason);
|
|
const attempt =
|
|
typeof scheduledRetry.scheduledRetryAttempt === "number"
|
|
&& Number.isFinite(scheduledRetry.scheduledRetryAttempt)
|
|
&& scheduledRetry.scheduledRetryAttempt > 0
|
|
? scheduledRetry.scheduledRetryAttempt
|
|
: null;
|
|
|
|
const badgeLabel = continuation ? "Continuation scheduled" : "Retry scheduled";
|
|
const titleAction = continuation ? "Automatic continuation" : "Automatic retry";
|
|
let titleSuffix: string;
|
|
if (relative === "now") {
|
|
titleSuffix = "due now";
|
|
} else if (relative) {
|
|
titleSuffix = relative;
|
|
} else {
|
|
titleSuffix = "pending schedule";
|
|
}
|
|
const title = `${titleAction} ${titleSuffix}`;
|
|
|
|
const helperIdle = continuation
|
|
? "Pulls continuation forward immediately"
|
|
: "Pulls retry forward immediately";
|
|
const isError = retryNow.isError || retryNow.lastError !== null;
|
|
const isSuccessTransient = retryNow.isSuccess
|
|
&& (retryNow.data?.outcome === "promoted" || retryNow.data?.outcome === "already_promoted");
|
|
|
|
return (
|
|
<div
|
|
data-testid="issue-scheduled-retry-card"
|
|
className="mb-3 rounded-lg border border-blue-500/30 bg-blue-500/5 px-3 py-3"
|
|
>
|
|
<div className="flex flex-col gap-2 sm:flex-row sm:items-start sm:justify-between">
|
|
<div className="min-w-0 flex-1">
|
|
<div className="flex flex-wrap items-center gap-2 text-xs">
|
|
<Badge variant="outline" className="border-blue-500/30 bg-blue-500/10 text-blue-700 dark:text-blue-300">
|
|
<Clock className="h-3 w-3" aria-hidden="true" />
|
|
{badgeLabel}
|
|
</Badge>
|
|
{attempt !== null ? (
|
|
<span className="text-muted-foreground">Attempt {attempt}</span>
|
|
) : null}
|
|
{reason ? (
|
|
<span className="text-muted-foreground">{reason}</span>
|
|
) : null}
|
|
</div>
|
|
<div className="mt-1 text-sm font-medium text-foreground">{title}</div>
|
|
{(absolute || scheduledRetry.retryOfRunId) ? (
|
|
<div className="mt-0.5 text-xs text-muted-foreground">
|
|
{absolute ? <span>{absolute}</span> : null}
|
|
{absolute && scheduledRetry.retryOfRunId ? <span>{" · "}</span> : null}
|
|
{scheduledRetry.retryOfRunId ? (
|
|
<span>
|
|
Replaces run{" "}
|
|
<Link
|
|
to={`/agents/${scheduledRetry.agentId}/runs/${scheduledRetry.retryOfRunId}`}
|
|
className="font-mono text-foreground hover:underline"
|
|
>
|
|
{shortRunId(scheduledRetry.retryOfRunId)}
|
|
</Link>
|
|
</span>
|
|
) : null}
|
|
</div>
|
|
) : null}
|
|
{scheduledRetry.error ? (
|
|
<div className="mt-1 text-xs text-muted-foreground">
|
|
Last attempt failed: {scheduledRetry.error}{/[.!?]$/.test(scheduledRetry.error.trim()) ? "" : "."} Paperclip will retry automatically.
|
|
</div>
|
|
) : null}
|
|
{isError ? (
|
|
<RetryErrorBand
|
|
error={retryNow.lastError}
|
|
onRetry={() => {
|
|
retryNow.reset();
|
|
retryNow.mutate();
|
|
}}
|
|
/>
|
|
) : null}
|
|
</div>
|
|
<div className="flex flex-col items-stretch gap-1 sm:items-end">
|
|
<Button
|
|
type="button"
|
|
variant="outline"
|
|
size="sm"
|
|
className="shrink-0 shadow-none"
|
|
onClick={() => retryNow.mutate()}
|
|
disabled={retryNow.isPending || isSuccessTransient}
|
|
data-testid="issue-scheduled-retry-card-retry-now"
|
|
>
|
|
{retryNow.isPending ? (
|
|
<span className="inline-flex items-center gap-1.5">
|
|
<Loader2 className="h-3.5 w-3.5 animate-spin" aria-hidden="true" />
|
|
Retrying…
|
|
</span>
|
|
) : isSuccessTransient ? (
|
|
<span className="inline-flex items-center gap-1.5">
|
|
<CheckCircle2 className="h-3.5 w-3.5" aria-hidden="true" />
|
|
{retryNow.data?.outcome === "already_promoted" ? "Already promoted" : "Promoted"}
|
|
</span>
|
|
) : (
|
|
<span className="inline-flex items-center gap-1.5">
|
|
<RotateCcw className="h-3.5 w-3.5" aria-hidden="true" />
|
|
Retry now
|
|
</span>
|
|
)}
|
|
</Button>
|
|
<span className="text-right text-xs text-muted-foreground sm:max-w-(--sz-12rem)">
|
|
{retryNow.isPending
|
|
? "Promoting scheduled retry"
|
|
: isSuccessTransient
|
|
? retryNow.data?.outcome === "already_promoted"
|
|
? "Already promoted — run starting"
|
|
: "Promoted — run starting"
|
|
: helperIdle}
|
|
</span>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
);
|
|
}
|
|
|
|
interface RetryErrorBandProps {
|
|
error: RetryNowError | null;
|
|
onRetry: () => void;
|
|
className?: string;
|
|
}
|
|
|
|
export function RetryErrorBand({ error, onRetry, className }: RetryErrorBandProps) {
|
|
if (!error) return null;
|
|
return (
|
|
<div
|
|
className={cn(
|
|
"mt-2 flex items-start gap-2 rounded-md border border-rose-500/30 bg-rose-500/5 px-2 py-1.5 text-xs text-rose-700 dark:text-rose-300",
|
|
className,
|
|
)}
|
|
role="alert"
|
|
data-testid="issue-scheduled-retry-error-band"
|
|
>
|
|
<AlertCircle className="mt-0.5 h-3.5 w-3.5 shrink-0" aria-hidden="true" />
|
|
<div className="min-w-0 flex-1">
|
|
<div className="font-medium">Couldn't retry now</div>
|
|
<div className="mt-0.5 text-muted-foreground">{error.message}</div>
|
|
</div>
|
|
<button
|
|
type="button"
|
|
onClick={onRetry}
|
|
className="shrink-0 font-medium text-rose-700 hover:underline dark:text-rose-300"
|
|
>
|
|
Try again
|
|
</button>
|
|
</div>
|
|
);
|
|
}
|