mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-21 00:05:49 +02:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs preserve provider events and results, then decide task state. > - Codex can fail a turn because the selected model is temporarily at capacity. > - Paperclip displayed a generic native-session failure and left the task in recovery. > - A known capacity failure needs a clear message and a durable delayed retry. > - This pull request uses committed terminal evidence, the status-effect ledger, and existing dispatch gates. > - The task can continue automatically without an unbounded retry loop or a model switch. ## Linked Issues or Issue Description Related: #13993 adds launcher capacity deferrals before provider startup. This change handles a committed native Codex `serverOverloaded` terminal after provider work has started. **What happened?** A native Codex turn ended with `codexErrorInfo: serverOverloaded` and “Selected model is at capacity. Please try a different model.” Paperclip saved the result, showed a generic failure, and required recovery instead of waiting and retrying. **Expected behavior** Show the capacity error directly. Schedule bounded retries after a short delay. Preserve saved work and honor current execution gates. **Steps to reproduce** 1. Run a native Codex task. 2. Commit a `turn.failed` event with `serverOverloaded`, then accept a failed result for the same turn. 3. Inspect the task error and its recovery state. The regression test reproduces this without a paid provider call. **Paperclip version or commit** Reproduced against master `16b7db35ffa0f9a95913c8cbdeea3d595435691f`. ## What Changed - Classify capacity failures from committed runner events and the pinned execution identity. Preserve accepted results and display the specific capacity error. - Atomically persist one scheduled successor with the status decision. Retry after one minute, then two minutes. Share the existing failure budget and stop after two automatic retries. - Reuse dispatch gates for ownership, task holds, dependencies, budget, and locks. Wait for predecessor execution, finalization, and cleanup before claiming a retry. - Preserve pending reviewer authority and suppress retries after reassignment or a successor claim. - Consume the failed run's resume receipt and delivered wake input. Rebuild ordinary continuation from the failed run so explicitly resumed tasks can retry without borrowing one-run authorization. - Label scheduled retries “Model at capacity.” Add a Storybook example and avoid duplicate punctuation in the existing retry card. - Add regression coverage and document the runtime contract. ## Verification - Targeted recovery and UI suites: 125 tests passed. Additional final cleanup and lock regressions passed. - Latest-head continuation and authorization regressions: 57 tests passed, including all four capacity integration tests against an isolated PostgreSQL database. - Rechecked all four capacity integration tests with the full runner's isolated `PAPERCLIP_HOME`, config, temporary directory, and serial fork settings: passed. - `pnpm -r typecheck`: passed. Final server typecheck passed. - `pnpm build`: passed. - `pnpm check:token-gates`: passed. - Browser: checked the real Storybook card. It shows the capacity message, automatic retry time, and existing Retry now action. - `pnpm test:run`: attempted and restarted after an interruption. The resumed run started before the final review correction and was stopped after recorded workspace/native test failures and the five-minute Git streaming timeout already documented on master. Exit 130; no complete local full-suite pass is claimed. Current-head isolated capacity tests and the complete CI suite pass. - A combined local recovery-suite attempt also hit PostgreSQL initialization failures at this macOS host's global shared-memory limit (32 slots). The focused recovery suites passed separately; final-head capacity tests also pass with the full runner's isolated environment settings. - Latest-head GitHub checks: all 56 checks green, including general and serialized test shards, browser E2E, typecheck, build, Runner verification, and canary dry run. No merge conflicts. - Greptile: 5/5 on `813a470b8c18c05aeb7e31e63e573c3a3a2f5cac`, with zero unresolved findings after fixing the resumed-task continuation issue. ## Risks - Capacity retries can repeat a task turn after partial work. They start a fresh provider session with task history and wait for predecessor cleanup. They do not resume the failed turn. - Retries retain the configured model unless an operator changes configuration. Persistent overload consumes the existing failure budget and then needs an explicit retry or model change. - Only run-bound native Codex `serverOverloaded` failures qualify. Usage-limit exhaustion, model/account incompatibility, unbound text, and unknown provider failures retain their current recovery behavior. - No schema migration is required. ## Model Used - OpenAI GPT-6 through Codex. The exact model ID and context-window size are not exposed to this session. Used reasoning, repository tools, code execution, and browser verification. No subagents. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
102 lines
3.8 KiB
TypeScript
102 lines
3.8 KiB
TypeScript
import { formatDateTime } from "./utils";
|
|
|
|
type RetryAwareRun = {
|
|
status: string;
|
|
retryOfRunId?: string | null;
|
|
scheduledRetryAt?: string | Date | null;
|
|
scheduledRetryAttempt?: number | null;
|
|
scheduledRetryReason?: string | null;
|
|
retryExhaustedReason?: string | null;
|
|
};
|
|
|
|
export type RunRetryStateSummary = {
|
|
kind: "scheduled" | "exhausted" | "attempted";
|
|
badgeLabel: string;
|
|
tone: string;
|
|
detail: string | null;
|
|
secondary: string | null;
|
|
retryOfRunId: string | null;
|
|
};
|
|
|
|
const RETRY_REASON_LABELS: Record<string, string> = {
|
|
native_provider_overloaded: "Model at capacity",
|
|
ai_connection_pool_wait: "Waiting for pool usage to recover",
|
|
transient_failure: "Transient failure",
|
|
missing_issue_comment: "Missing task comment",
|
|
process_lost: "Process lost",
|
|
assignment_recovery: "Assignment recovery",
|
|
issue_continuation_needed: "Continuation needed",
|
|
max_turns_continuation: "Max-turn continuation",
|
|
};
|
|
|
|
function readNonEmptyString(value: unknown) {
|
|
return typeof value === "string" && value.trim().length > 0 ? value.trim() : null;
|
|
}
|
|
|
|
function joinFragments(parts: Array<string | null>) {
|
|
const filtered = parts.filter((part): part is string => Boolean(part));
|
|
return filtered.length > 0 ? filtered.join(" · ") : null;
|
|
}
|
|
|
|
export function formatRetryReason(reason: string | null | undefined) {
|
|
const normalized = readNonEmptyString(reason);
|
|
if (!normalized) return null;
|
|
return RETRY_REASON_LABELS[normalized] ?? normalized.replace(/_/g, " ");
|
|
}
|
|
|
|
export function describeRunRetryState(run: RetryAwareRun): RunRetryStateSummary | null {
|
|
const attempt =
|
|
typeof run.scheduledRetryAttempt === "number" && Number.isFinite(run.scheduledRetryAttempt) && run.scheduledRetryAttempt > 0
|
|
? run.scheduledRetryAttempt
|
|
: null;
|
|
const attemptLabel = attempt ? `Attempt ${attempt}` : null;
|
|
const reasonLabel = formatRetryReason(run.scheduledRetryReason);
|
|
const retryOfRunId = readNonEmptyString(run.retryOfRunId);
|
|
const exhaustedReason = readNonEmptyString(run.retryExhaustedReason);
|
|
const dueAt = run.scheduledRetryAt ? formatDateTime(run.scheduledRetryAt) : null;
|
|
const isMaxTurnContinuation = run.scheduledRetryReason === "max_turns_continuation";
|
|
const hasRetryMetadata =
|
|
Boolean(retryOfRunId)
|
|
|| Boolean(reasonLabel)
|
|
|| Boolean(dueAt)
|
|
|| Boolean(attemptLabel)
|
|
|| Boolean(exhaustedReason);
|
|
|
|
if (!hasRetryMetadata) return null;
|
|
|
|
if (run.status === "scheduled_retry") {
|
|
return {
|
|
kind: "scheduled",
|
|
badgeLabel: run.scheduledRetryReason === "ai_connection_pool_wait" ? "Pool exhausted" : isMaxTurnContinuation ? "Continuation scheduled" : "Retry scheduled",
|
|
tone: "border-blue-500/30 bg-blue-500/10 text-blue-700 dark:text-blue-300",
|
|
detail: joinFragments([attemptLabel, reasonLabel]),
|
|
secondary: dueAt
|
|
? `${isMaxTurnContinuation ? "Next continuation" : "Next retry"} ${dueAt}`
|
|
: `${isMaxTurnContinuation ? "Next continuation" : "Next retry"} pending schedule`,
|
|
retryOfRunId,
|
|
};
|
|
}
|
|
|
|
if (exhaustedReason) {
|
|
return {
|
|
kind: "exhausted",
|
|
badgeLabel: isMaxTurnContinuation ? "Continuation exhausted" : "Retry exhausted",
|
|
tone: "border-amber-500/30 bg-amber-500/10 text-amber-700 dark:text-amber-300",
|
|
detail: joinFragments([attemptLabel, reasonLabel, "Automatic retries exhausted"]),
|
|
secondary: exhaustedReason.includes("Manual intervention required")
|
|
? exhaustedReason
|
|
: `${exhaustedReason} Manual intervention required.`,
|
|
retryOfRunId,
|
|
};
|
|
}
|
|
|
|
return {
|
|
kind: "attempted",
|
|
badgeLabel: isMaxTurnContinuation ? "Continued run" : "Retried run",
|
|
tone: "border-slate-500/20 bg-slate-500/10 text-slate-700 dark:text-slate-300",
|
|
detail: joinFragments([attemptLabel, reasonLabel]),
|
|
secondary: null,
|
|
retryOfRunId,
|
|
};
|
|
}
|