fix(native): continue again when a deploy interrupts an automatic continuation (#21314)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs execute in a remote Runner (Daytona) that the
server controls; a deploy restarts the server while runs are in flight
> - When the restart's reattachment proves the Runner process gone
mid-turn, #21039 makes the server request one automatic Retry that
continues in a fresh session from the saved work, without replaying the
turn
> - Staging deploys every 20 to 45 minutes, and Codex runs often take
longer, so that automatic continuation is often interrupted by the next
deploy too
> - The hook treated that second loss as "automatic Retry already used"
and left the task blocked for a person
(`native_continuation_requires_reconciliation`)
> - This pull request gives an automatic continuation one more
fresh-session continuation each time a server restart or deploy took its
controller, and a deploy spends no automatic budget
> - The benefit is that a task interrupted by repeated deploys continues
by itself, and no provider turn that may have run is sent again

## Linked Issues or Issue Description

Refs #21039, Refs #21052, Refs #21272, Refs #21023, Refs #21120, Refs
#21161

**What happened?**

On staging, the largest failure class for a Codex agent on Daytona was
`runner_remote_process_identity_unavailable` with error code
`native_continuation_requires_reconciliation`: 18 runs in 3 hours. The
error name is misleading. It is the shared code for every refused remote
reattachment (`REMOTE_RUNNER_RECOVERY_ERROR_CODE`). In every sampled run
the refusal was `process_verification` / `process_not_running`: the
Runner process was gone, its stderr log was empty, and the last turn
event was `turn.submitted` or `turn.accepted` (unsettled). It was not a
missing identity file.

Sampled 12 such runs. In 9 the #21039 hook worked: it recorded
`native.interrupted_turn_outcome_unknown` and requested the
fresh-session Retry. The tasks that stayed blocked had one shape:

1. A run lost its Runner after a deploy. The hook requested the
automatic Retry (key `task-retry:restart-interrupted-turn:<run>`).
2. The Retry run started a fresh session. The next deploy came during
its turn (a graceful `SIGTERM` shutdown transition, then "Recovering the
existing native runner process after server restart").
3. The same-run reattachment found that Runner gone too
(`process_not_running`).
4. The hook returned `automatic_retry_used`, because the run's Retry
intent came from the hook. The task stayed blocked.

Examples: a `task_retry` run created 40 s after its predecessor failed
(the hook's Retry), interrupted by the 05:17 deploy, failed at 05:18,
task blocked. A second chain (assignment, then the hook's Retry) failed
the same way at 05:12.

**Expected behavior**

A deploy is not the task's failure. When a server restart or deploy took
the controller of an automatic continuation and its Runner is proven
gone, the server continues the task again in a fresh session from the
saved work. It never replays the turn. A loss without restart evidence
keeps the existing one-time rule.

**Steps to reproduce**

1. Start a long Codex run in a remote Daytona Runner.
2. Deploy the server during its provider turn so that the reattachment
finds the Runner gone. The server requests the automatic Retry.
3. Deploy again during the Retry run's turn so that its Runner is also
lost.
4. See the Retry run fail with
`runner_remote_process_identity_unavailable` and the task stay blocked
with no new run.

**Paperclip version or commit**

Staging Core `25e9a9c2e9`. Fixed on `master` after `731b52453c`.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`, `codex_app_server`).

## What Changed

- `restart-interrupted-turn.ts`: `lostControllerToRestart` finds restart
evidence on the run: a `native.recovery.transition` with `restartKind:
"graceful"` (shutdown), or a restart claim (`reattach_remote_runner`,
`reattach_existing_runner`, `resume_dead_runner`,
`bootstrap_incomplete`) without the `same_run_retry` trigger.
- Both `automatic_retry_used` refusals (a person's Retry that the hook
requested, and a system retry of owed work) now continue when the run
has restart evidence. A deploy interruption spends no budget, so a task
that only deploys interrupt never runs out. Each continuation needs a
new deploy during its run, so deploys bound it. A loss without restart
evidence keeps the one-time rule. All other gates are unchanged: proven
Runner exit, unknown-turn record, the issue still assigned and open,
idempotency per run, and the Retry's admission gates.
- `doc/execution-semantics.md` describes the rule.

## Verification

- Red before the fix, green after:
`server/src/services/runner-lost-mid-turn-continuation.test.ts` › "an
automatic continuation that the next deploy interrupted". The test seeds
the staging sequence: a lost run, the hook's Retry run with a
`turn.submitted` turn, the shutdown and restart transitions with staging
payloads, then a same-run reattachment that proves the Runner gone. On
the old code the two continuation tests return `automatic_retry_used`.
With the fix they request one fresh-session Retry, record the unknown
turn outcome once, recover the saved work and queue the successor with
`forceFreshSession: true`.
- A chain of 12 deploy continuations, each with a system retry that
inherits its Retry receipt, still continues (red on the first revision
of this PR, which had a budget of 8). Losses without deploy evidence
stay refused (with and without earlier deploy continuations).
- `npx vitest run
src/services/runner-lost-mid-turn-continuation.test.ts` 23/23 pass.
`governed-wait-runner-loss.test.ts` and
`workspace-recovery-retry-copyback.test.ts` 103/103 pass.
- `pnpm --filter @paperclipai/server typecheck` passes.
- `pnpm verify` exits 2 (a global doc input changed), so required CI
supplies the broad coverage.

## Risks

- Low to medium. A task whose Runner is lost on every deploy now
continues after each deploy instead of once. Each continuation is a full
run, so it costs tokens. The deploy count bounds it, because each
continuation needs a new restart during its run.
- Not fixed here: why the reattached Runner is gone after a deploy. In
the sampled runs the reattachment after the restart hung in `native
session provider recovery timed out after 900000ms`, and the next
attempt found the process gone with empty stderr. That needs its own
investigation.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, browser read-only API GETs on staging).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 00:41:32 -06:00
1 parent 03f9b83e52
commit b555087f0f
3 files changed
+131 -5

No files matched your search

+6 -1
View File
@@ -1842,7 +1842,12 @@ first, and the Retry starts a fresh provider session over the saved work, so the
turn is never replayed. Activity records the request
as `restart-recovery` on behalf of that user. Admission keeps every gate: saved-work
recovery, stop proof, approvals, pauses, budget and ownership. A Retry that the
server requested gets no second automatic attempt. A system wake (for example a
server requested gets no second automatic attempt, unless a server restart or
deploy took that continuation's controller while it ran (a shutdown
`native.recovery.transition` with `restartKind: "graceful"`, or a restart
scan's claim that is not a same-run retry). Each such deploy earns one more
fresh-session continuation and spends no budget, so a task that only deploys
interrupt never runs out. A system wake (for example a
monitor) continues this way only with a positive
record that no turn was ever issued: the refused reattachment proved the Runner
process gone, and the run's controller journal, which records each turn command
@@ -83,7 +83,7 @@ async function recordUnknownTurnOutcome(db: Db, run: { companyId: string; id: st
...(typeof runnerExit.runnerLifecycle === "string" ? { runnerLifecycle: runnerExit.runnerLifecycle } : {}) } } });
}
/** Automatic continuation is requested at most once per chain of runs. */
/** Automatic continuation is requested at most once per chain of runs, plus once for each deploy that interrupts a continuation. */
export const RESTART_RETRY_KEY_PREFIX = "restart-interrupted-turn:";
const MAX_RETRY_CHAIN = 32;
@@ -108,6 +108,19 @@ async function chainUsedAutomaticRetry(db: Db, run: { companyId: string; id: str
return visited.size >= MAX_RETRY_CHAIN;
}
const RESTART_CLAIM_DISPOSITIONS = ["reattach_remote_runner", "reattach_existing_runner", "resume_dead_runner",
"bootstrap_incomplete"];
async function lostControllerToRestart(db: Db, run: { companyId: string; id: string }) {
const [evidence] = await db.select({ id: heartbeatRunEvents.id }).from(heartbeatRunEvents).where(and(
eq(heartbeatRunEvents.companyId, run.companyId), eq(heartbeatRunEvents.runId, run.id),
eq(heartbeatRunEvents.eventType, "native.recovery.transition"),
sql`(${heartbeatRunEvents.payload}->>'restartKind' = 'graceful' or (${heartbeatRunEvents.payload}->>'trigger' is null
and ${heartbeatRunEvents.payload}->>'runnerDisposition' in (${sql.join(RESTART_CLAIM_DISPOSITIONS.map(kind => sql`${kind}`), sql`, `)})))`,
)).limit(1);
return evidence !== undefined;
}
type PrepareChatRetry = NonNullable<Parameters<typeof requestTaskRetry>[0]["prepareChatRetry"]>;
/** A connected conversation's Retry goes through the chat service, which the
@@ -188,15 +201,17 @@ export async function requestInterruptedNativeTurnRetry(db: Db, runId: string):
if (!wake || reason === "task_retry" || !run.responsibleUserId
|| !(owedRedelivery || await hasProvenNoProviderTurn(db, run, runnerExit)))
return { requested: false, reason: "not_user_request" };
if (owedRedelivery && await chainUsedAutomaticRetry(db, run)) return { requested: false, reason: "automatic_retry_used" };
if (owedRedelivery && await chainUsedAutomaticRetry(db, run) && !await lostControllerToRestart(db, run))
return { requested: false, reason: "automatic_retry_used" };
actorId = run.responsibleUserId;
} else if (reason === "task_retry") {
// A Retry this hook already requested gets no second automatic attempt.
// A Retry this hook already requested gets no second automatic attempt unless a deploy interrupted it.
const retryId = typeof context.taskRetryRequestId === "string" ? context.taskRetryRequestId : null;
const [receipt] = retryId ? await db.select({ key: agentWakeupRequests.idempotencyKey }).from(agentWakeupRequests).where(and(
eq(agentWakeupRequests.companyId, run.companyId), sql`${agentWakeupRequests.id}::text = ${retryId}`)) : [];
if (!receipt) return { requested: false, reason: "retry_source_missing" };
if (receipt.key?.startsWith(`task-retry:${RESTART_RETRY_KEY_PREFIX}`)) return { requested: false, reason: "automatic_retry_used" };
if (receipt.key?.startsWith(`task-retry:${RESTART_RETRY_KEY_PREFIX}`) && !await lostControllerToRestart(db, run))
return { requested: false, reason: "automatic_retry_used" };
} else if (isUserCommentContinuationReason(reason)) {
const commentId = typeof context.wakeCommentId === "string" ? context.wakeCommentId
: typeof wake!.payload?.commentId === "string" ? wake!.payload.commentId : null;
@@ -170,6 +170,112 @@ describe("a remote Runner that died in the middle of a provider turn", () => {
eq(agentWakeupRequests.reason, "task_retry")))).filter(retry => retry.id !== hookIntent.id)).toHaveLength(0);
}, 30_000);
describe("an automatic continuation that the next deploy interrupted", () => {
async function deployDuringRun(f: { companyId: string; runId: string; agentId: string }) {
for (const [message, payload] of [
["Server shutdown suspended native controller ownership without cancelling provider work", { signal: "SIGTERM",
processPid: 1048, restartKind: "graceful", processGroupId: null, retryRunCreated: false,
processStartedAt: "2026-11-13T05:28:12.307Z", runnerDisposition: "awaiting_runner_reattach" }],
["Recovering the existing native runner process after server restart", { restartKind: "hard", providerAttempt: 2,
recoveryRequestId: null, runnerDisposition: "reattach_remote_runner", controllerGeneration: 3 }],
["Reattaching to the existing remote runner to resume this run", { trigger: "same_run_retry", restartKind: "hard",
providerAttempt: 3, recoveryRequestId: null, runnerDisposition: "reattach_remote_runner", controllerGeneration: 4 }],
] as const) await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.runId, agentId: f.agentId,
eventType: "native.recovery.transition", stream: "system", level: "info", message, payload });
}
async function automaticContinuation(input: { deploy: boolean; earlierContinuations?: number; systemRetries?: boolean }) {
const f = await runnerLostMidTurn({ wake: "user_retry", events: ["turn.submitted"], refusalPayload: {
remoteRunnerRecovery: { stage: "process_verification", reason: "process_not_running", runnerSettled: false,
runnerLifecycle: "ready", runnerDiagnostics: ["WebSocket peer closed the connection"],
runnerExitEvidence: "== runnerd.process-stderr.log" } } });
let previousRunId: string | null = null;
for (let index = 0; index <= (input.earlierContinuations ?? 0); index += 1) {
const [lost]: Array<typeof heartbeatRuns.$inferSelect> = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, issueId: f.issueId,
status: "failed", runtimeMode: "native", error: "runner_remote_process_identity_unavailable",
createdAt: new Date(Date.now() - (120 - index) * 60_000), finishedAt: new Date(Date.now() - (110 - index) * 60_000),
errorCode: "native_continuation_requires_reconciliation", contextSnapshot: { issueId: f.issueId,
wakeReason: index === 0 ? "issue_assigned" : "task_retry", ...(previousRunId ? { previousRunId } : {}) } }).returning();
if (previousRunId) {
const [hookIntent]: Array<typeof agentWakeupRequests.$inferSelect> = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId,
source: "on_demand", triggerDetail: "manual", reason: "task_retry", status: "completed",
idempotencyKey: `task-retry:restart-interrupted-turn:${previousRunId}`, requestedByActorType: "user",
requestedByActorId: "local-board", payload: { issueId: f.issueId } }).returning();
await db.update(heartbeatRuns).set({ contextSnapshot: { ...lost.contextSnapshot, taskRetryRequestId: hookIntent.id } })
.where(eq(heartbeatRuns.id, lost.id));
if (input.systemRetries) {
const [systemRetry]: Array<typeof heartbeatRuns.$inferSelect> = await db.insert(heartbeatRuns).values({
companyId: f.companyId, agentId: f.agentId, issueId: f.issueId, status: "failed", runtimeMode: "native",
retryOfRunId: lost.id, error: "runner_remote_process_identity_unavailable",
createdAt: new Date(Date.now() - (120 - index) * 60_000 + 30_000),
finishedAt: new Date(Date.now() - (110 - index) * 60_000 + 30_000),
errorCode: "native_continuation_requires_reconciliation", contextSnapshot: { issueId: f.issueId,
wakeReason: "transient_failure_retry", taskRetryRequestId: hookIntent.id, retryOfRunId: lost.id } }).returning();
previousRunId = systemRetry.id;
continue;
}
}
previousRunId = lost.id;
}
await db.update(agentWakeupRequests).set({ idempotencyKey: `task-retry:restart-interrupted-turn:${previousRunId}` })
.where(eq(agentWakeupRequests.id, f.retryIntent!.id));
const current = await run(f.runId);
await db.update(heartbeatRuns).set({ contextSnapshot: { ...current.contextSnapshot, previousRunId } })
.where(eq(heartbeatRuns.id, f.runId));
if (input.deploy) await deployDuringRun(f);
return f;
}
it("continues again in a fresh session when a deploy took the continuation's Runner mid-turn", async () => {
const f = await automaticContinuation({ deploy: true });
expect(await requestRestartInterruptedTurnRetry(db, f.claim)).toEqual({ requested: true, requestId: expect.any(String) });
expect(await requestInterruptedNativeTurnRetry(db, f.runId)).toMatchObject({ requested: true });
expect(await unknownOutcomeEvents(f.runId)).toEqual([expect.objectContaining({ payload: expect.objectContaining({
lastTurnEvent: "turn.submitted", replayed: false }) })]);
await vi.waitFor(async () => {
await Promise.all(await reconcileRetainedWorkspaces(db, f.environmentRuntime, current => f.service.resumeRemoteStopComments(current)) ?? []);
expect(readSystemRecovery((await action(f.actionId)).evidence)?.state).toBe("complete");
}, { timeout: 10_000, interval: 10 });
await vi.waitFor(async () => expect(await queued(f.companyId)).toHaveLength(1), { timeout: 10_000, interval: 50 });
const [successor] = await queued(f.companyId);
expect(successor.contextSnapshot).toMatchObject({ previousRunId: f.runId, wakeReason: "task_retry", forceFreshSession: true });
const retries = await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, f.companyId),
eq(agentWakeupRequests.idempotencyKey, `task-retry:restart-interrupted-turn:${f.runId}`)));
expect(retries).toEqual([expect.objectContaining({ requestedByActorType: "user", requestedByActorId: "local-board" })]);
}, 30_000);
it("continues a system retry of owed work that inherited the continuation when a deploy took its Runner", async () => {
const f = await runnerLostMidTurn({ wake: "system_retry" });
const current = await run(f.runId);
const [hookIntent] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId,
source: "on_demand", triggerDetail: "manual", reason: "task_retry", status: "completed",
idempotencyKey: `task-retry:restart-interrupted-turn:${randomUUID()}`, requestedByActorType: "user",
requestedByActorId: "local-board", payload: { issueId: f.issueId } }).returning();
await db.update(heartbeatRuns).set({ contextSnapshot: { ...current.contextSnapshot, taskRetryRequestId: hookIntent.id } })
.where(eq(heartbeatRuns.id, f.runId));
await deployDuringRun(f);
expect(await requestRestartInterruptedTurnRetry(db, f.claim)).toEqual({ requested: true, requestId: expect.any(String) });
expect(await unknownOutcomeEvents(f.runId)).toHaveLength(1);
}, 30_000);
it("never runs out when only deploys interrupted a long chain whose system retries inherit each continuation", async () => {
const f = await automaticContinuation({ deploy: true, earlierContinuations: 12, systemRetries: true });
expect(await requestRestartInterruptedTurnRetry(db, f.claim)).toEqual({ requested: true, requestId: expect.any(String) });
expect(await unknownOutcomeEvents(f.runId)).toHaveLength(1);
}, 30_000);
it.each([
{ title: "no deploy interrupted it", earlierContinuations: 0 },
{ title: "no deploy interrupted it after earlier deploy continuations", earlierContinuations: 4 },
])("keeps the visible Retry when $title", async ({ earlierContinuations }) => {
const f = await automaticContinuation({ deploy: false, earlierContinuations, systemRetries: true });
expect(await requestRestartInterruptedTurnRetry(db, f.claim)).toEqual({ requested: false, reason: "automatic_retry_used" });
expect(await unknownOutcomeEvents(f.runId)).toHaveLength(0);
expect(await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, f.companyId),
eq(agentWakeupRequests.idempotencyKey, `task-retry:restart-interrupted-turn:${f.runId}`)))).toHaveLength(0);
}, 30_000);
});
// The staging chat task runs through a connected Slack conversation, whose
// endpoint restricts external execution: its Retry goes through the chat service.
describe("an external chat message in a connected conversation", () => {