mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A deploy replaces the server. The old server stops its running legacy runs at SIGTERM (`server_shutdown_interrupted`). The new server finds runs whose owner died without that drain (`process_lost`, "server may have restarted"). > - Each of these runs got its follow-up from the bounded transient retry. That retry spends the failure budget (2 attempts) and waits 30 to 60 seconds. > - So the first deploy spent one failure retry. A run that was already the last automatic retry got no follow-up at all. The task then rested on a manual Retry. > - A deploy is routine. An interruption by the server is not a failed attempt, and it must never leave a task waiting for a person. > - This pull request moves these runs to their own `server_restart` retry lane. The lane never spends the failure budget, starts the first continuation at once, and has its own bound. > - The benefit is that a task continues by itself through several deploys a day, and a real loop still stops and shows why. ## Linked Issues or Issue Description **What happened?** A zero-downtime deploy replaced a hosted server. In the 11 minutes around the swap, 11 legacy runs ended with `server_shutdown_interrupted` and 7 ended with `process_lost`. At least 7 tasks were left with no follow-up run and no scheduled retry, so each needed a manual Retry. Two of the stranded runs were already automatic retries. **Expected behavior** Each interrupted run continues once, automatically, on the surviving server, with its original wake. **Paperclip version or commit** Core commit7d9973405c. The behavior is the same on masterbf18b348c5. **Steps to reproduce** 1. Start a legacy run with a conversation adapter (for example `codex_local`) as the second automatic retry of its chain (failure budget spent). 2. Send SIGTERM to the server (`drainRunningRunsForShutdown`), or let a new server boot find the run without its owner (`reapOrphanedRuns`). 3. See that no successor run is created. The scheduler logs "Bounded retry exhausted". **Root cause** - `enqueueProcessLossRetry` (`server/src/services/heartbeat/recovery.ts`) calls `scheduleBoundedRetryForRun`. That function put these runs in the `transient_failure` lane, with `maxAttempts = 2` counted from `executionRetryAccounting.failureRetries` (`server/src/services/heartbeat/retries.ts`). A first attempt got a charged retry 30 to 60 seconds later. A run with 2 spent failure retries got `retry_exhausted`. - The orphan reaper (`recoverActiveExecutions`) skipped the retry when `failedRunRetryBudgetSpent(run)` was true. It also skipped a conversation run that had no local child process recorded (for example a run that was dispatching), because `shouldRetry` needed a local child, a preparing stage, or a monitor wake. - The stranded-task sweeper then saw the spent budget and moved the task to `blocked`. ## What Changed - `execution-recovery-attempt.ts`: add the `server_restart` retry lane. It is an uncharged lane. Its counter (`serverRestarts`, `serverRestartWindowStartedAt`) is saved in `executionRetryAccounting`, so it is visible on each run. The bound is 6 continuations in one 60-minute window. A later interruption after the window starts a new window. - New `server-replacement-interruption.ts`: `isServerReplacementInterruption(run)` names a legacy `server_shutdown_interrupted` run, or a `process_lost` run whose owner was another server boot. It reads the new `processLossDiagnostic.controllerReplaced` verdict. For rows saved by an older release, it reads the owner boot id or `runPredatesObserver`. - `retries.ts`: `scheduleBoundedRetryForRun` uses the `server_restart` lane for these runs (`usesServerRestartLane`). The first continuation is due at once; later ones in the window back off like the interrupted-preparation lane. `failedRunRetryBudgetSpent` reports the restart bound for these runs, so the sweeper escalates only when that bound trips. An interrupted preparation keeps its own lane while its chain has failure budget left. - `recovery.ts` (orphan reaper): record `controllerReplaced`. A conversation run that a restart ended is eligible for continuation even without a local child process, and it checks the restart bound, not the failure budget. - `wake-queue/adapters/postgres.ts`: carry the restart counter to successors that the wake queue creates. - No schema change. No change to how runs start (no new waits before a turn). Behavior that did not change: - At most one successor per run: every path still uses the predecessor row lock and the `retry_of_run_id` check in `scheduleBoundedRetryForRun`. Two servers alive at once reuse the same successor. - A non-conversation adapter that reached dispatch stays held for reconciliation, because a replay can repeat its actions. - Native runtime runs keep their own recovery. - A server that began shutdown already starts no queued run (`closeProcessRunAdmission` in `server/src/index.ts`, checked in `heartbeat/queue.ts`). Its queued and newly scheduled runs stay in the database for the new server. ## Verification - Red on `origin/master`, green on this branch (`server/src/__tests__/heartbeat-process-recovery.test.ts`, block "runs ended by a server restart or replacement"): - Graceful shutdown, first attempt: master scheduled a charged `transient_failure` retry. Now one immediate `server_restart` continuation. - Graceful shutdown, already the last retry: master scheduled nothing. Now one continuation; the failure count stays 2. - Owner lost at boot (`process_lost`), first attempt and already-retried: same results, with two servers running the boot sweep at the same time. Only one successor. - Old server exited after it committed the interruption and before it scheduled the continuation: the new server's sweep continues the run. - Run saved by an older release (no `controllerReplaced`, spent failure budget): the sweep continues it instead of moving the task to `blocked`. - The restart bound trips after 6 in one window, records `retryReason: server_restart` on the run, and the sweeper escalates. The same count in a closed window continues. - A run that failed by itself (`adapter_failed`) with a spent failure budget still gets no retry. - Each continuation's execution continuation reports the original wake (`trigger.retriedReason = issue_monitor_due`). - Unit tests: `server/src/services/server-replacement-interruption.test.ts`. - Six earlier tests asserted escalation after "three deploy restarts" spent the failure budget. They now spend the restart bound instead, so they still prove escalation when the bound trips. - Existing test `server/src/services/heartbeat/queue.test.ts` covers "starts no run once shutdown closed run admission". - `pnpm verify`: see the PR checks. ## Risks - A run that a restart ends now continues more often (up to 6 times per hour per chain, not 2 in total). A run that itself crashes the server is still stopped by the bound, and the stop is shown on the task. - Tasks that an older release already moved to `blocked` stay blocked. Tasks that are still `in_progress` or `todo` with a stranded run continue on the next sweep. - A `process_lost` row from an older release whose owner was the same boot that later restarted is read as a restart. That case is a lost in-memory handle, which is also not a failure of the agent's work. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, test execution) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
168 lines
9.3 KiB
TypeScript
168 lines
9.3 KiB
TypeScript
import { RUNNER_REVISION_WAIT_RETRY_REASON } from "./runner-revision-mismatch.js";
|
|
import { STORAGE_UNAVAILABLE_WAIT_RETRY_REASON } from "../storage/storage-unavailable.js";
|
|
import { PROVIDER_ATTEMPT_CREDIT_GENERATION_LIMIT } from "./native-runtime/provider-attempt-outcome.js";
|
|
import { AGENT_DIRECTORY_BUSY_RETRY_REASON } from "../modules/run-dispatch/index.js";
|
|
|
|
/**
|
|
* A preparation that a restart interrupted before any provider work started
|
|
* (its certificate records `providerWorkStarted: false`) spent no provider
|
|
* work, so it never charges the failure budget (durable-ownership plan,
|
|
* section F). Its own counter keeps a crash loop bounded, at the same limit
|
|
* as the native controller-generation cap.
|
|
*/
|
|
export const INTERRUPTED_PREPARATION_RETRY_REASON = "interrupted_preparation";
|
|
export const INTERRUPTED_PREPARATION_RETRY_MAX_ATTEMPTS = PROVIDER_ATTEMPT_CREDIT_GENERATION_LIMIT;
|
|
export const AGENT_DIRECTORY_WAIT_RETRY_REASON = AGENT_DIRECTORY_BUSY_RETRY_REASON;
|
|
|
|
/**
|
|
* A run that ended only because its server process was replaced or restarted
|
|
* (a deploy's graceful shutdown, or a run found without its owner after a
|
|
* restart) did not fail. Its continuation never spends the failure budget. Its
|
|
* own bound stops a loop: at most SERVER_RESTART_RETRY_MAX_ATTEMPTS
|
|
* continuations inside one SERVER_RESTART_RETRY_WINDOW_MS window. A chain that
|
|
* is interrupted again after the window has passed starts a new window, so a
|
|
* long task survives several deploys a day.
|
|
*/
|
|
export const SERVER_RESTART_RETRY_REASON = "server_restart";
|
|
export const SERVER_RESTART_RETRY_MAX_ATTEMPTS = 6;
|
|
export const SERVER_RESTART_RETRY_WINDOW_MS = 60 * 60_000;
|
|
|
|
/** Retry lanes that keep their own counter and never spend failures. */
|
|
const UNCHARGED_RETRY_LANES: ReadonlySet<string> = new Set([
|
|
"workspace_busy", "ai_connection_busy", "ai_connection_pool_wait", AGENT_DIRECTORY_WAIT_RETRY_REASON, RUNNER_REVISION_WAIT_RETRY_REASON,
|
|
INTERRUPTED_PREPARATION_RETRY_REASON, STORAGE_UNAVAILABLE_WAIT_RETRY_REASON, SERVER_RESTART_RETRY_REASON,
|
|
]);
|
|
|
|
type RetryRun = {
|
|
scheduledRetryAttempt?: number | null;
|
|
scheduledRetryReason?: string | null;
|
|
contextSnapshot?: Record<string, unknown> | null;
|
|
};
|
|
|
|
/** Server-owned counts for one automatic continuation chain. Repair slots live
|
|
* in legacyDispositionEpisode and are never charged to either counter here. */
|
|
export interface ExecutionRetryAccounting {
|
|
version: 1;
|
|
failureRetries: number;
|
|
maxTurnContinuations: number;
|
|
/**
|
|
* Interrupted-preparation retries in this chain. Saved with the chain, so a
|
|
* wait retry in between never resets it. Omitted while zero.
|
|
*/
|
|
interruptedPreparations?: number;
|
|
/** Storage-outage waits in this chain, kept across intervening resource waits. Omitted while zero. */
|
|
storageWaits?: number;
|
|
/** Agent-directory waits in this chain, kept across intervening resource waits. Omitted while zero. */
|
|
agentDirectoryWaits?: number;
|
|
/** Server-restart continuations in the current window. Omitted while zero. */
|
|
serverRestarts?: number;
|
|
/** Start of the current server-restart window (ISO time). Saved with serverRestarts. */
|
|
serverRestartWindowStartedAt?: string;
|
|
}
|
|
|
|
function count(value: unknown): number | null {
|
|
return typeof value === "number" && Number.isSafeInteger(value) && value >= 0 ? value : null;
|
|
}
|
|
|
|
function savedAccounting(run: RetryRun): ExecutionRetryAccounting | null {
|
|
const value = run.contextSnapshot?.executionRetryAccounting;
|
|
if (!value || typeof value !== "object" || Array.isArray(value)) return null;
|
|
const saved = value as Record<string, unknown>;
|
|
const failureRetries = count(saved.failureRetries);
|
|
const maxTurnContinuations = count(saved.maxTurnContinuations);
|
|
if (saved.version !== 1 || failureRetries === null || maxTurnContinuations === null) return null;
|
|
const interruptedPreparations = count(saved.interruptedPreparations) ?? 0;
|
|
const storageWaits = count(saved.storageWaits) ?? 0;
|
|
const agentDirectoryWaits = count(saved.agentDirectoryWaits) ?? 0;
|
|
const serverRestarts = count(saved.serverRestarts) ?? 0;
|
|
const windowStartedAt = typeof saved.serverRestartWindowStartedAt === "string"
|
|
&& Number.isFinite(Date.parse(saved.serverRestartWindowStartedAt)) ? saved.serverRestartWindowStartedAt : null;
|
|
return { version: 1, failureRetries, maxTurnContinuations,
|
|
...(interruptedPreparations > 0 ? { interruptedPreparations } : {}),
|
|
...(storageWaits > 0 ? { storageWaits } : {}),
|
|
...(agentDirectoryWaits > 0 ? { agentDirectoryWaits } : {}),
|
|
...(serverRestarts > 0 && windowStartedAt ? { serverRestarts, serverRestartWindowStartedAt: windowStartedAt } : {}) };
|
|
}
|
|
|
|
function historicalFailureCount(run: RetryRun): number {
|
|
if (run.scheduledRetryReason === "max_turns_continuation" || run.scheduledRetryReason === "issue_disposition_repair") return 0;
|
|
if (run.scheduledRetryReason === "ai_connection_busy" || run.scheduledRetryReason === "ai_connection_pool_wait") {
|
|
const saved = count(run.contextSnapshot?.failureRetriesBeforeAiConnectionWait);
|
|
if (saved !== null) return saved;
|
|
}
|
|
if (run.scheduledRetryReason === "workspace_busy") {
|
|
const saved = count(run.contextSnapshot?.failureRetriesBeforeWorkspaceWait);
|
|
if (saved !== null) return saved;
|
|
}
|
|
// Historical ambiguous counters remain conservative rather than resetting.
|
|
return count(run.scheduledRetryAttempt) ?? 0;
|
|
}
|
|
|
|
export function executionRetryAccounting(run: RetryRun): ExecutionRetryAccounting {
|
|
const saved = savedAccounting(run);
|
|
const reason = run.scheduledRetryReason ?? "";
|
|
const nonFailureLane = reason === "max_turns_continuation" || reason === "issue_disposition_repair" || UNCHARGED_RETRY_LANES.has(reason);
|
|
const interruptedPreparations = Math.max(saved?.interruptedPreparations ?? 0,
|
|
reason === INTERRUPTED_PREPARATION_RETRY_REASON ? count(run.scheduledRetryAttempt) ?? 0 : 0);
|
|
const storageWaits = Math.max(saved?.storageWaits ?? 0,
|
|
reason === STORAGE_UNAVAILABLE_WAIT_RETRY_REASON ? count(run.scheduledRetryAttempt) ?? 0 : 0);
|
|
const agentDirectoryWaits = Math.max(saved?.agentDirectoryWaits ?? 0,
|
|
reason === AGENT_DIRECTORY_WAIT_RETRY_REASON ? count(run.scheduledRetryAttempt) ?? 0 : 0);
|
|
return {
|
|
version: 1,
|
|
failureRetries: Math.max(saved?.failureRetries ?? 0, saved && nonFailureLane ? 0 : historicalFailureCount(run)),
|
|
maxTurnContinuations: Math.max(saved?.maxTurnContinuations ?? 0,
|
|
run.scheduledRetryReason === "max_turns_continuation" ? count(run.scheduledRetryAttempt) ?? 0 : 0),
|
|
...(interruptedPreparations > 0 ? { interruptedPreparations } : {}),
|
|
...(storageWaits > 0 ? { storageWaits } : {}),
|
|
...(agentDirectoryWaits > 0 ? { agentDirectoryWaits } : {}),
|
|
...(saved?.serverRestarts ? { serverRestarts: saved.serverRestarts,
|
|
serverRestartWindowStartedAt: saved.serverRestartWindowStartedAt } : {}),
|
|
};
|
|
}
|
|
|
|
/** Server-restart continuations already spent in the window that is open at `now`. */
|
|
export function serverRestartAttemptCount(run: RetryRun, now: Date): number {
|
|
const accounting = executionRetryAccounting(run);
|
|
const startedAt = Date.parse(accounting.serverRestartWindowStartedAt ?? "");
|
|
if (!accounting.serverRestarts || !Number.isFinite(startedAt)) return 0;
|
|
return now.getTime() - startedAt < SERVER_RESTART_RETRY_WINDOW_MS ? accounting.serverRestarts : 0;
|
|
}
|
|
|
|
/** True when another server-restart continuation would exceed the window's bound. */
|
|
export function serverRestartBudgetSpent(run: RetryRun, now = new Date()): boolean {
|
|
return serverRestartAttemptCount(run, now) >= SERVER_RESTART_RETRY_MAX_ATTEMPTS;
|
|
}
|
|
|
|
/** Resource waits, repairs and productive continuations do not spend failures. */
|
|
export function executionFailureRetryCount(run: RetryRun): number {
|
|
return executionRetryAccounting(run).failureRetries;
|
|
}
|
|
|
|
export function executionRetryAttemptCount(run: RetryRun, reason: string): number {
|
|
if (reason === INTERRUPTED_PREPARATION_RETRY_REASON) return executionRetryAccounting(run).interruptedPreparations ?? 0;
|
|
if (reason === STORAGE_UNAVAILABLE_WAIT_RETRY_REASON) return executionRetryAccounting(run).storageWaits ?? 0;
|
|
if (reason === AGENT_DIRECTORY_WAIT_RETRY_REASON) return executionRetryAccounting(run).agentDirectoryWaits ?? 0;
|
|
if (UNCHARGED_RETRY_LANES.has(reason)) {
|
|
return run.scheduledRetryReason === reason ? count(run.scheduledRetryAttempt) ?? 0 : 0;
|
|
}
|
|
const accounting = executionRetryAccounting(run);
|
|
return reason === "max_turns_continuation" ? accounting.maxTurnContinuations : accounting.failureRetries;
|
|
}
|
|
|
|
export function accountingForScheduledRetry(run: RetryRun, reason: string, attempt: number, now = new Date()): ExecutionRetryAccounting {
|
|
const accounting = executionRetryAccounting(run);
|
|
if (reason === SERVER_RESTART_RETRY_REASON) {
|
|
// Attempt 1 opens a new window; a later attempt stays in the open one.
|
|
if (attempt <= 1 || serverRestartAttemptCount(run, now) === 0) accounting.serverRestartWindowStartedAt = now.toISOString();
|
|
accounting.serverRestarts = attempt;
|
|
return accounting;
|
|
}
|
|
if (reason === "max_turns_continuation") accounting.maxTurnContinuations = attempt;
|
|
else if (reason === INTERRUPTED_PREPARATION_RETRY_REASON) accounting.interruptedPreparations = attempt;
|
|
else if (reason === STORAGE_UNAVAILABLE_WAIT_RETRY_REASON) accounting.storageWaits = attempt;
|
|
else if (reason === AGENT_DIRECTORY_WAIT_RETRY_REASON) accounting.agentDirectoryWaits = attempt;
|
|
else if (!UNCHARGED_RETRY_LANES.has(reason)) accounting.failureRetries = attempt;
|
|
return accounting;
|
|
}
|