Files
PaperClipAI/server/src/services/server-replacement-interruption.test.ts
T
DottaandClaude Opus 5.5 156453fe7c fix(recovery): continue runs that a server restart or replacement ended, without spending the failure budget (#21019)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A deploy replaces the server. The old server stops its running
legacy runs at SIGTERM (`server_shutdown_interrupted`). The new server
finds runs whose owner died without that drain (`process_lost`, "server
may have restarted").
> - Each of these runs got its follow-up from the bounded transient
retry. That retry spends the failure budget (2 attempts) and waits 30 to
60 seconds.
> - So the first deploy spent one failure retry. A run that was already
the last automatic retry got no follow-up at all. The task then rested
on a manual Retry.
> - A deploy is routine. An interruption by the server is not a failed
attempt, and it must never leave a task waiting for a person.
> - This pull request moves these runs to their own `server_restart`
retry lane. The lane never spends the failure budget, starts the first
continuation at once, and has its own bound.
> - The benefit is that a task continues by itself through several
deploys a day, and a real loop still stops and shows why.

## Linked Issues or Issue Description

**What happened?**
A zero-downtime deploy replaced a hosted server. In the 11 minutes
around the swap, 11 legacy runs ended with `server_shutdown_interrupted`
and 7 ended with `process_lost`. At least 7 tasks were left with no
follow-up run and no scheduled retry, so each needed a manual Retry. Two
of the stranded runs were already automatic retries.

**Expected behavior**
Each interrupted run continues once, automatically, on the surviving
server, with its original wake.

**Paperclip version or commit**
Core commit 7d9973405c. The behavior is
the same on master bf18b348c5.

**Steps to reproduce**
1. Start a legacy run with a conversation adapter (for example
`codex_local`) as the second automatic retry of its chain (failure
budget spent).
2. Send SIGTERM to the server (`drainRunningRunsForShutdown`), or let a
new server boot find the run without its owner (`reapOrphanedRuns`).
3. See that no successor run is created. The scheduler logs "Bounded
retry exhausted".

**Root cause**
- `enqueueProcessLossRetry`
(`server/src/services/heartbeat/recovery.ts`) calls
`scheduleBoundedRetryForRun`. That function put these runs in the
`transient_failure` lane, with `maxAttempts = 2` counted from
`executionRetryAccounting.failureRetries`
(`server/src/services/heartbeat/retries.ts`). A first attempt got a
charged retry 30 to 60 seconds later. A run with 2 spent failure retries
got `retry_exhausted`.
- The orphan reaper (`recoverActiveExecutions`) skipped the retry when
`failedRunRetryBudgetSpent(run)` was true. It also skipped a
conversation run that had no local child process recorded (for example a
run that was dispatching), because `shouldRetry` needed a local child, a
preparing stage, or a monitor wake.
- The stranded-task sweeper then saw the spent budget and moved the task
to `blocked`.

## What Changed

- `execution-recovery-attempt.ts`: add the `server_restart` retry lane.
It is an uncharged lane. Its counter (`serverRestarts`,
`serverRestartWindowStartedAt`) is saved in `executionRetryAccounting`,
so it is visible on each run. The bound is 6 continuations in one
60-minute window. A later interruption after the window starts a new
window.
- New `server-replacement-interruption.ts`:
`isServerReplacementInterruption(run)` names a legacy
`server_shutdown_interrupted` run, or a `process_lost` run whose owner
was another server boot. It reads the new
`processLossDiagnostic.controllerReplaced` verdict. For rows saved by an
older release, it reads the owner boot id or `runPredatesObserver`.
- `retries.ts`: `scheduleBoundedRetryForRun` uses the `server_restart`
lane for these runs (`usesServerRestartLane`). The first continuation is
due at once; later ones in the window back off like the
interrupted-preparation lane. `failedRunRetryBudgetSpent` reports the
restart bound for these runs, so the sweeper escalates only when that
bound trips. An interrupted preparation keeps its own lane while its
chain has failure budget left.
- `recovery.ts` (orphan reaper): record `controllerReplaced`. A
conversation run that a restart ended is eligible for continuation even
without a local child process, and it checks the restart bound, not the
failure budget.
- `wake-queue/adapters/postgres.ts`: carry the restart counter to
successors that the wake queue creates.
- No schema change. No change to how runs start (no new waits before a
turn).

Behavior that did not change:
- At most one successor per run: every path still uses the predecessor
row lock and the `retry_of_run_id` check in
`scheduleBoundedRetryForRun`. Two servers alive at once reuse the same
successor.
- A non-conversation adapter that reached dispatch stays held for
reconciliation, because a replay can repeat its actions.
- Native runtime runs keep their own recovery.
- A server that began shutdown already starts no queued run
(`closeProcessRunAdmission` in `server/src/index.ts`, checked in
`heartbeat/queue.ts`). Its queued and newly scheduled runs stay in the
database for the new server.

## Verification

- Red on `origin/master`, green on this branch
(`server/src/__tests__/heartbeat-process-recovery.test.ts`, block "runs
ended by a server restart or replacement"):
- Graceful shutdown, first attempt: master scheduled a charged
`transient_failure` retry. Now one immediate `server_restart`
continuation.
- Graceful shutdown, already the last retry: master scheduled nothing.
Now one continuation; the failure count stays 2.
- Owner lost at boot (`process_lost`), first attempt and
already-retried: same results, with two servers running the boot sweep
at the same time. Only one successor.
- Old server exited after it committed the interruption and before it
scheduled the continuation: the new server's sweep continues the run.
- Run saved by an older release (no `controllerReplaced`, spent failure
budget): the sweep continues it instead of moving the task to `blocked`.
- The restart bound trips after 6 in one window, records `retryReason:
server_restart` on the run, and the sweeper escalates. The same count in
a closed window continues.
- A run that failed by itself (`adapter_failed`) with a spent failure
budget still gets no retry.
- Each continuation's execution continuation reports the original wake
(`trigger.retriedReason = issue_monitor_due`).
- Unit tests:
`server/src/services/server-replacement-interruption.test.ts`.
- Six earlier tests asserted escalation after "three deploy restarts"
spent the failure budget. They now spend the restart bound instead, so
they still prove escalation when the bound trips.
- Existing test `server/src/services/heartbeat/queue.test.ts` covers
"starts no run once shutdown closed run admission".
- `pnpm verify`: see the PR checks.

## Risks

- A run that a restart ends now continues more often (up to 6 times per
hour per chain, not 2 in total). A run that itself crashes the server is
still stopped by the bound, and the stop is shown on the task.
- Tasks that an older release already moved to `blocked` stay blocked.
Tasks that are still `in_progress` or `todo` with a stranded run
continue on the next sweep.
- A `process_lost` row from an older release whose owner was the same
boot that later restarted is read as a restart. That case is a lost
in-memory handle, which is also not a failure of the agent's work.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:56:49 -06:00

72 lines
5.1 KiB
TypeScript

import { describe, expect, it } from "vitest";
import { isServerReplacementInterruption } from "./server-replacement-interruption.js";
import {
accountingForScheduledRetry, executionFailureRetryCount, SERVER_RESTART_RETRY_MAX_ATTEMPTS, SERVER_RESTART_RETRY_REASON,
SERVER_RESTART_RETRY_WINDOW_MS, serverRestartAttemptCount, serverRestartBudgetSpent,
} from "./execution-recovery-attempt.js";
import { failedRunRetryBudgetSpent, usesServerRestartLane } from "./heartbeat/retries.js";
const thisBoot = "boot-current";
const run = (patch: Record<string, unknown>) => ({ runtimeMode: "legacy", status: "failed", errorCode: "process_lost",
resultJson: null, controllerBootId: null, ...patch }) as Parameters<typeof isServerReplacementInterruption>[0];
describe("isServerReplacementInterruption", () => {
it("names a graceful shutdown and a restart-caused process loss, never a run's own failure", () => {
expect(isServerReplacementInterruption(run({ status: "interrupted", errorCode: "server_shutdown_interrupted" }), thisBoot)).toBe(true);
expect(isServerReplacementInterruption(run({ resultJson: { processLossDiagnostic: { controllerReplaced: true } } }), thisBoot)).toBe(true);
expect(isServerReplacementInterruption(run({ resultJson: { processLossDiagnostic: { controllerReplaced: false } },
controllerBootId: "boot-old" }), thisBoot)).toBe(false);
expect(isServerReplacementInterruption(run({ errorCode: "adapter_failed", controllerBootId: "boot-old" }), thisBoot)).toBe(false);
expect(isServerReplacementInterruption(run({ runtimeMode: "native", status: "interrupted",
errorCode: "server_shutdown_interrupted" }), thisBoot)).toBe(false);
});
it("reads rows saved by an older release from the owner boot or the observer start", () => {
expect(isServerReplacementInterruption(run({ controllerBootId: "boot-old" }), thisBoot)).toBe(true);
expect(isServerReplacementInterruption(run({ controllerBootId: thisBoot }), thisBoot)).toBe(false);
expect(isServerReplacementInterruption(run({ resultJson: { processLossDiagnostic: { runPredatesObserver: true } } }), thisBoot)).toBe(true);
expect(isServerReplacementInterruption(run({ resultJson: { processLossDiagnostic: { runPredatesObserver: false } } }), thisBoot)).toBe(false);
});
});
describe("server-restart retry accounting", () => {
const now = new Date("2026-11-12T04:00:00.000Z");
it("counts restarts in a window, keeps the failure count, and opens a new window after it closes", () => {
const base = { contextSnapshot: { executionRetryAccounting: { version: 1, failureRetries: 2, maxTurnContinuations: 0 } } };
const first = accountingForScheduledRetry(base, SERVER_RESTART_RETRY_REASON, 1, now);
expect(first).toMatchObject({ failureRetries: 2, serverRestarts: 1, serverRestartWindowStartedAt: now.toISOString() });
const chained = { contextSnapshot: { executionRetryAccounting: first } };
expect(executionFailureRetryCount(chained)).toBe(2);
const later = new Date(now.getTime() + 5 * 60_000);
expect(serverRestartAttemptCount(chained, later)).toBe(1);
const second = accountingForScheduledRetry(chained, SERVER_RESTART_RETRY_REASON, 2, later);
expect(second).toMatchObject({ serverRestarts: 2, serverRestartWindowStartedAt: now.toISOString() });
const afterWindow = new Date(now.getTime() + SERVER_RESTART_RETRY_WINDOW_MS);
expect(serverRestartAttemptCount({ contextSnapshot: { executionRetryAccounting: second } }, afterWindow)).toBe(0);
});
it("trips at the bound inside one window", () => {
const spent = { contextSnapshot: { executionRetryAccounting: { version: 1, failureRetries: 0, maxTurnContinuations: 0,
serverRestarts: SERVER_RESTART_RETRY_MAX_ATTEMPTS, serverRestartWindowStartedAt: now.toISOString() } } };
expect(serverRestartBudgetSpent(spent, new Date(now.getTime() + 60_000))).toBe(true);
expect(serverRestartBudgetSpent(spent, new Date(now.getTime() + SERVER_RESTART_RETRY_WINDOW_MS))).toBe(false);
});
it("charges only the restart bound to a restart, and the failure budget to a failure", () => {
const spentFailures = { executionRetryAccounting: { version: 1, failureRetries: 2, maxTurnContinuations: 0 } };
const interrupted = { ...run({ status: "interrupted", errorCode: "server_shutdown_interrupted" }), contextSnapshot: spentFailures };
expect(usesServerRestartLane(interrupted)).toBe(true);
expect(failedRunRetryBudgetSpent(interrupted, now)).toBe(false);
const failed = { ...run({ errorCode: "adapter_failed" }), contextSnapshot: spentFailures };
expect(usesServerRestartLane(failed)).toBe(false);
expect(failedRunRetryBudgetSpent(failed, now)).toBe(true);
// A preparation that started no provider work keeps its own lane while
// the chain has failure budget left.
const preparation = { ...run({ status: "interrupted", errorCode: "server_shutdown_interrupted",
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } }), contextSnapshot: {} };
expect(usesServerRestartLane(preparation)).toBe(false);
expect(usesServerRestartLane({ ...preparation, contextSnapshot: spentFailures })).toBe(true);
});
});