fix(native): continue a turn lost on restore in a proven same-run resume (#20645)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run keeps its provider conversation when its local runner
dies. When the restored runner finds that the active turn was lost, it
reports `provider_turn_lost_on_restore` and does not invent a result.
> - After a control-plane restart, the admitted restart attempt may send
one continuation in that same conversation (#15297). The agent then
reconciles unfinished work. The task is not resent.
> - A runner can also be lost without a server restart, for example when
`runner.drain` times out. The run then resumes itself after the retry
delay from its exact checkpoint. That path never allowed the
continuation, because the executor passed `resumeInterruptedTurn` only
for a `resume_dead_runner` restart claim.
> - So the lost turn failed with `native_provider_terminal_failed`,
automatic replacement was refused (`uncertain_provider_action`), and the
user had to click Retry.
> - This pull request allows the same single continuation for that
resume, but only when durable records prove the same three facts that a
restart claim proves.
> - The benefit is that a runner lost mid-turn continues in the same
conversation with no Retry. No case that is refused today becomes
allowed without that proof.

## Linked Issues or Issue Description

Refs #15297, #20625

**What happened?**
Restart-chaos wave L7 (seed 9107, local), task codex B5 TM2, run on
master after #20625. A TM3 restart resumed the run (attempt 1). During
the TM2 freeze the TS runtime lost its runner:
`native_session_recovery_failed: runner_local_connect_failed: PRP
command runner.drain timed out`. The executor saved `retryable_failure`
with `recoveryMode: "exact_checkpoint_resume"`. After the delay,
recovery verified the runner process stopped
(`native.local_process_stopped`). The same run then resumed as attempt 2
(`native.process_start_requested`, `providerAttempt: 2`). The restored
runner reported `turn.failed` with `provider_turn_lost_on_restore`
(`recoverable: true`). No continuation was sent. The run failed with
`native_provider_terminal_failed`, recovery stopped with
`uncertain_provider_action` / `preserve_without_replay_v1`, and the task
needed Retry.

**Expected behavior**
The run sends the one same-conversation continuation, exactly as it does
after a restart claim, and finishes without Retry.

**Steps to reproduce**
`node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts
--env local --adapters codex,claude --workloads A1,B5,C1 --faults
TM1,TM2,TM3 --time-age-hours 1 --concurrency 3 --tasks 6 --seed 9107`
(timing-dependent: the drain timeout must land during an active turn).

## What Changed

- New
`server/src/services/native-runtime/scheduled-resume-continuation.ts`.
It decides whether a same-run scheduled resume may continue a lost turn.
Each of the three proofs comes from a durable record:
1. **The previous runner is stopped.** The latest process record of the
run in `heartbeat_run_events` (`native.process_start_requested`,
`native.process_identity_recorded`, `native.local_process_stopped`) must
be `native.local_process_stopped`. Its `processPid` must equal its
`processGroupId`, and neither the pid nor the group may be alive now.
This is the existing `readNativeLocalProcessStop` check, read in the
claim transaction before this attempt records its own
`native.process_start_requested`.
2. **The attempt was admitted under the bounded budget.** The executor's
claim moves `native_run_finalizations` from phase `retryable_failure` to
`observed`. It charges a provider attempt, and the claim refuses any
attempt above 3 (`native_session_retry_exhausted`). The decision also
requires `1 <= attempt <= NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT`.
3. **It is the same provider conversation.** The saved `failure_detail`
must show `recoveryMode: "exact_checkpoint_resume"`, `recoverable:
true`, `providerSessionEstablished: true` and `checkpointExists: true`,
with no `successorRunId` or `replacementDenied`. At run time the
existing runtime guards still apply: the backend must recover the
checkpointed session (`recovered`, not a continuity-break replacement),
and the lost turn must equal the checkpoint's `activeTurnId` and the
snapshot's interrupted turn. The provider kinds stay codex, acpx and
opencode, with no compaction, goal control, held result or pending
runtime request.
- `native-session-executor.ts`: computes that decision in the
coordinator claim. It passes `resumeInterruptedTurn` when either the
restart claim is `resume_dead_runner` (unchanged) or the decision allows
it.
- `doc/execution-semantics.md`: describes the rule and the refused
cases.

**Cases that stay refused** (each has a test in
`scheduled-resume-continuation.test.ts`):
- a restart-recovery claim (it keeps its own existing rule);
- a remote execution target;
- a coordinator that is not `retryable_failure`, including a fresh run;
- `recoveryMode` `bootstrap_retry` or `ambiguous_state`, or no saved
failure detail;
- a failure that is not recoverable;
- no established provider session;
- no checkpoint;
- a recorded replacement (`successorRunId` or `replacementDenied`);
- an attempt outside 1..3;
- no stop receipt, a stop receipt superseded by a newer launch, a
stopped process that is still alive, or a run that never had a process.

The runtime-side refusals are unchanged and already covered by
`native-session-runtime.test.ts`: real provider errors, historical
error-null failures, and no restart permission.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/scheduled-resume-continuation.test.ts`: 22
tests, covering the decision table and the stop receipt in embedded
Postgres.
- `cd server && npx tsc --noEmit` passes.
- Live: not proven. An L7 wave on this branch (seed 9107) did not
reproduce the drain-timeout path, so this pull request has unit and
database tests only. In that wave the same cell (codex B5 TM2) needed
Retry for a different reason: TM2 aged its restart-recovery claim lease
before dispatch, and budget admission refused with `Native recovery no
longer owns provider dispatch`. This pull request does not change that
path. Four other tasks passed. Claude C1 TM2 rested blocked without a
Retry button, which is a separate cleanup-quarantine issue.

## Risks

- This allows one more path to send the same-conversation continuation.
It is limited to local runs. It needs the same proofs as a restart
claim, and it uses the attempt the resume already charged. It does not
change the generation cap or `restartResumeKeepsAttempt`.
- A rolling deploy reads `failure_detail` and stop events written by
earlier releases. Those fields and events already have this shape, so an
older saved failure is judged by the same rule.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs). Extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
DottaandClaude authored and GitHub committed 2026-11-10 15:16:51 -06:00
1 parent 353945c86d
commit 37c50f70c2
7 files changed
+330 -4

No files matched your search

@@ -119,6 +119,7 @@ import { stoppedCodexTurnIsTextOnly } from "./stopped-codex-turn.js";
import { prepareVerifiedRemoteProviderPack, useVerifiedPreinstalledRemoteProviderPack, shouldUseBundledAcpxImageAssets } from "./remote-provider-pack.js";
import { uploadVerifiedProviderPack } from "./provider-pack-upload.js";
import { selectRemotePiCompanion } from "./remote-pi-companion.js";
import { scheduledResumeContinuationAllowed } from "./scheduled-resume-continuation.js";
import { readNativeLocalProcessStop, recordedLocalExecutionStopped, PRIOR_OWNER_FENCE_REFUSED, PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED, withReleasedNativeRunProcess } from "../native-local-process-stop.js";
import { remoteLeaseCleanupScope } from "../remote-execution-termination.js";
import { connectorToolDefinitionsForSkills } from "../connector-runtime.js";
@@ -10063,6 +10064,7 @@ async function executePaperclipNativeSessionAttempt(
let confirmedLeaseDeadline = 0;
let claimedStartupContext: NativeExecutionContext | undefined;
let freshClaimAttempt: number | undefined;
let scheduledResumeContinuation = false;
try {
attempt = await trace.measure(
"native.coordinator.claim",
@@ -10212,6 +10214,15 @@ async function executePaperclipNativeSessionAttempt(
);
if (nextAttempt > 3)
throw new Error("native_session_retry_exhausted");
scheduledResumeContinuation = (await scheduledResumeContinuationAllowed(tx as unknown as Db, {
companyId: input.execution.binding.companyId,
runId: input.execution.binding.runId,
restartRecovery: Boolean(recovering),
targetKind: input.runnerExecutionTarget?.kind === "remote" ? "remote" : "local",
coordinatorPhase: coordinator.phase,
failureDetail: coordinator.failureDetail,
admittedAttempt: nextAttempt,
})).allowed;
const nextControllerGeneration = recovering
? recovering.controllerGeneration
: coordinator.controllerBootId === controller.bootId
@@ -11302,7 +11313,7 @@ async function executePaperclipNativeSessionAttempt(
trace.activate(runnerSessionStartupScope);
const runExecution = () =>
executeNativeSession({
resumeInterruptedTurn: input.restartRecovery?.kind === "resume_dead_runner",
resumeInterruptedTurn: input.restartRecovery?.kind === "resume_dead_runner" || scheduledResumeContinuation,
getFreshSessionHandoff: input.getFreshSessionHandoff,
// A failed seal commits nothing: the messages stay queued for the
// turn boundary instead of failing this run before its turn.