mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-19 07:42:38 +01:00
fix(native-runtime): a remote Runner reattachment continues a turn the Runner can recover (#21362)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime runs agent turns through a Runner, often in a remote sandbox, and must recover runs after a restart without help from a person > - When a Runner process is lost, a new Runner restores the conversation. If the provider turn is gone, the Runner sends `turn.failed` with `provider_turn_lost_on_restore` and `recoverable: true`. The Runner sends this only after it proves that the old process ended > - The runtime can continue such a turn in the same conversation, but only when the server sets `resumeInterruptedTurn`. The server set it only for a local `resume_dead_runner` claim or a scheduled resume, and a scheduled remote resume is refused. So on a remote Runner reattachment the runtime did not continue the turn, and the run ended > - This pull request sets `resumeInterruptedTurn` for a `reattach_remote_runner` claim too > - The benefit is that a remote run continues the lost turn in the same conversation immediately, instead of failing and then restarting 3 to 6 minutes later ## Linked Issues or Issue Description **What happened?** On hosted staging (build `d1d71a7755`, read-only trace), 10 runs in 3 hours ended with `native_provider_terminal_failed` and the message "The runner restored the conversation after process loss, but the previous turn is no longer active." The Runner marked each of these events `recoverable: true`. Example: run 646e2efb. A new Runner process started at 08:32:04. `session.resumed` came at 08:34:49. `turn.failed provider_turn_lost_on_restore` (recoverable) came at 08:34:51. The run failed at 08:35:21. Example: run 11fe2cdc. `session.resumed` came at 09:23:24. `turn.failed` came at 09:23:25. The run failed at 09:23:52. In each case, a separate recovery path continued the task only 3 to 6 minutes later. **Expected behavior** When the Runner marks a lost turn as recoverable after a restore on a remote reattachment, the run continues the turn in the same conversation. **Steps to reproduce** 1. Start a native run on a remote sandbox Runner. 2. Lose the Runner process while a turn is active, so that a new Runner reattaches through a `reattach_remote_runner` claim. 3. The new Runner restores the conversation and reports `provider_turn_lost_on_restore` with `recoverable: true`. 4. See that the run ends with `native_provider_terminal_failed` and does not continue the turn. **Agent adapter(s) involved** Not adapter-specific (core bug). Seen with Codex on a Daytona remote sandbox. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts`: `resumeInterruptedTurn` is now also true for a `reattach_remote_runner` restart-recovery claim. The runtime still continues the turn only when the Runner itself sends `turn.failed provider_turn_lost_on_restore` with `recoverable: true` for the checkpointed active turn. - `doc/execution-semantics.md`: one paragraph describes this behaviour, and says that a lost turn that still ends the run keeps `native_provider_terminal_failed`. - Tests: - `native-scheduled-resume-continuation.integration.test.ts`: "continues a lost turn on a same-run remote Runner reattachment" (2 cases: a plain reattachment and a `replaceUnansweredRunner` reattachment). - New `native-recoverable-lost-turn.integration.test.ts`: a guard through `executePaperclipNativeSession`. The runtime saves a failed-terminal checkpoint and then raises the lost turn (recoverable and not recoverable). The run ends as `native_provider_terminal_failed`, and the hold cause stays one that the replacement lane continues. - An earlier revision of this PR also mapped a recoverable lost turn to `native_session_interrupted`. Review showed that this does not make the run retryable, and the hold it writes is not picked up by any automatic continuation. That hunk is removed. ## Verification - Red without the product change: the two integration cases fail with `passed` = `[false]` (expected `[true]`). - Green with the change: - `cd server && npx vitest run src/services/native-runtime/native-scheduled-resume-continuation.integration.test.ts src/services/native-runtime/native-recoverable-lost-turn.integration.test.ts` — 14 passed. - `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts` — passed (with the scheduled-resume file: 224 passed, 14 skipped, pre-existing skips). - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts` — 1469 passed, 7 skipped. - `cd server && npx tsc --noEmit` — exit 0. - Staging evidence (hosted staging, build `d1d71a7755`, read-only trace): 10 runs in 3 hours ended with `native_provider_terminal_failed` although the Runner marked the lost turn `recoverable: true`. Run 646e2efb: new Runner pid 08:32:04, `session.resumed` 08:34:49, `turn.failed provider_turn_lost_on_restore` (recoverable) 08:34:51, run failed 08:35:21. Run 11fe2cdc: `session.resumed` 09:23:24, `turn.failed` 09:23:25, run failed 09:23:52. The task continued only 3 to 6 minutes later through a separate recovery path. ## Risks - This change has not run live on staging or on a Daytona sandbox yet. The evidence is the staging trace and the local tests. - This PR does not introduce or widen the gap where a recoverable lost turn still ends the run. On master, that run already ends as `native_provider_terminal_failed`, and the replacement lane continues it after its decision (3 to 6 minutes on staging). This PR removes the run-ending case whenever the Runner reports the turn recoverable for the checkpointed active turn on a remote reattachment. Where the run still ends (for example, when the continuation itself is lost), the behaviour is unchanged, and the guard test `native-recoverable-lost-turn.integration.test.ts` proves that it stays `native_provider_terminal_failed`. - The remaining case is tracked with the owner of the recovery path: recovery evidence will accept `provider_turn_lost_on_restore` / `recoverable` directly, after their ownership-deadline PR #21364. An earlier hunk here tried to shorten that case. An integration test through the real path showed that it would create a board-owned ambiguous hold that no automatic pass continues, so the hunk is removed. - This change does not fix why the turn is lost. In staging, a retry replaced a live Runner. A separate fix for that is in progress. - The runtime continues a turn only when the Runner sends the recoverable lost-turn signal. The Runner sends it only after it proves that the old process ended. So no provider turn runs twice. - No Runner image change. No schema change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, in Claude Code with tool use (shell, file edit, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
d9b42b50fd
commit
5ba26600ef
4 files changed
+104
-3
No files matched your search
@@ -1845,6 +1845,13 @@ mid-turn can leave provider or tool processes that the fence cannot find, so it
|
||||
keeps the fail-closed path below, and the task continues in a fresh session from
|
||||
the saved work.
|
||||
|
||||
On every `reattach_remote_runner` attempt, a turn the Runner reports as
|
||||
`provider_turn_lost_on_restore` with `recoverable: true` continues once in the same
|
||||
conversation, as on a local dead-runner resume. The Runner reports that cause only
|
||||
after it proved the old process ended and restored the conversation. If the turn
|
||||
still ends the run, the run keeps `native_provider_terminal_failed`, and the safe
|
||||
replacement lane continues the task.
|
||||
|
||||
The first unanswered read can come on the last of the three provider attempts,
|
||||
for example when a server restart used the second attempt. The run then gets one
|
||||
more attempt for the replacement only: the failure stays retryable, the claim
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { agents, companies, completionContracts, createDb, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations } from "@paperclipai/db";
|
||||
import type { ControlPlanePort, NativeExecutionInputV1, PersistedNativeSession } from "@paperclipai/paperclip-runner";
|
||||
import { EXECUTION_RECONCILIATION_CAUSES } from "@paperclipai/shared";
|
||||
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
||||
import { NativeProviderTerminalFailure } from "../../vendor/paperclip-runner/index.js";
|
||||
|
||||
const provider = vi.hoisted(() => ({ execute: vi.fn() }));
|
||||
vi.mock("../../vendor/paperclip-runner/index.js", async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import("../../vendor/paperclip-runner/index.js")>()), executeNativeSession: provider.execute,
|
||||
}));
|
||||
import { executePaperclipNativeSession } from "./native-session-executor.js";
|
||||
import { buildNativeCompletionContract } from "./completion-contracts.js";
|
||||
import { prepareNativeHeartbeatRun } from "./prepare-native-run.js";
|
||||
|
||||
const LOST_TURN_MESSAGE = "The runner restored the conversation after process loss, but the previous turn is no longer active.";
|
||||
|
||||
describe("a lost turn that still ends the run after a restore", () => {
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
const companyId = randomUUID(), agentId = randomUUID();
|
||||
beforeAll(async () => {
|
||||
temporary = await startEmbeddedPostgresTestDatabase("native-recoverable-lost-turn-");
|
||||
db = createDb(temporary.connectionString);
|
||||
await db.insert(companies).values({ id: companyId, name: "Lost turn", issuePrefix: "LST" });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Lost turn worker", status: "active", adapterType: "paperclip_runner",
|
||||
adapterConfig: { provider: "codex", model: "test-model" } });
|
||||
});
|
||||
afterAll(async () => { await temporary?.cleanup(); });
|
||||
|
||||
it.each([true, false])("keeps a lost turn (recoverable %s) on the hold the replacement lane continues", async (recoverable) => {
|
||||
const issueId = randomUUID(), runId = randomUUID();
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Lost turn task", status: "in_progress", assigneeAgentId: agentId });
|
||||
const [run] = await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", runtimeMode: "native",
|
||||
nativeIssueId: issueId, contextSnapshot: { issueId } }).returning();
|
||||
const [issue] = await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId)).returning();
|
||||
const native = await prepareNativeHeartbeatRun({ db, run, issue, environmentLeaseId: randomUUID() });
|
||||
const [contract] = await db.select().from(completionContracts).where(eq(completionContracts.issueId, issueId));
|
||||
const contractInput = buildNativeCompletionContract(issue, { revision: contract.revision });
|
||||
await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed" });
|
||||
const execution: NativeExecutionInputV1 = {
|
||||
schema: "paperclip.native-execution-input.v1", binding: { companyId, runId, issueId, agentId, executionWorkspaceId: runId },
|
||||
provider: { kind: "codex", model: "test-model" },
|
||||
task: { identifier: issueId, title: issue.title, description: null, prompt: "Continue the task.", workMode: "standard" },
|
||||
workspace: { cwd: "/tmp", repoUrl: null, repoRef: null, branchName: null },
|
||||
session: { normalizedSessionId: native.normalizedSessionId, driverKind: "codex_app_server", protocolVersion: 1, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
|
||||
completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1", contract: contractInput },
|
||||
interactionResponses: [], credentialBindings: [],
|
||||
};
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeExecutionInput: execution } }).where(eq(heartbeatRuns.id, runId));
|
||||
const identity = { companyId, agentId, issueId, runId, sessionId: native.normalizedSessionId };
|
||||
const snapshot: PersistedNativeSession = { backendKind: "mock", sessionId: native.normalizedSessionId, identity,
|
||||
providerSessionId: "provider-thread", activeTurnId: null, pendingRuntimeRequests: [], goal: null, semanticResult: null,
|
||||
terminal: { schema: "paperclip.prp.terminal.v1", turnTerminalState: "failed", runTerminalState: "failed", reportedWorkDisposition: "yielded" } };
|
||||
provider.execute.mockReset().mockImplementation(async (options: { controlPlane: ControlPlanePort }) => {
|
||||
await options.controlPlane.openRun({ identity, backendKind: "mock", sourceInstanceId: native.runnerInstanceId });
|
||||
await options.controlPlane.checkpointSession?.(snapshot);
|
||||
throw new NativeProviderTerminalFailure("provider_turn_lost_on_restore", recoverable, LOST_TURN_MESSAGE);
|
||||
});
|
||||
const unexpected = async () => { throw new Error("Unexpected provider call"); };
|
||||
await executePaperclipNativeSession({ db, execution, runnerInstanceId: native.runnerInstanceId, backend: { descriptor: unexpected, openSession: unexpected } })
|
||||
.catch(() => undefined);
|
||||
expect(provider.execute).toHaveBeenCalledTimes(1);
|
||||
const [saved] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
||||
expect(saved.runnerProfileJson).toMatchObject({ sessionCheckpoint: { terminal: { runTerminalState: "failed" } } });
|
||||
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId));
|
||||
expect(coordinator).toMatchObject({ phase: "terminal_failure", failureCode: "native_provider_terminal_failed", nextAttemptAt: null });
|
||||
const holds = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId));
|
||||
expect(holds).toEqual([expect.objectContaining({ kind: "active_run_watchdog", cause: "native_provider_terminal_failed",
|
||||
evidence: expect.objectContaining({ runId }) })]);
|
||||
expect(EXECUTION_RECONCILIATION_CAUSES).toContain(holds[0]!.cause);
|
||||
});
|
||||
});
|
||||
+20
-2
@@ -18,6 +18,7 @@ import { executePaperclipNativeSession } from "./native-session-executor.js";
|
||||
import { buildNativeCompletionContract } from "./completion-contracts.js";
|
||||
import { prepareNativeHeartbeatRun } from "./prepare-native-run.js";
|
||||
import { claimNativeSessionResumptions } from "./native-finalization-reconciler.js";
|
||||
import { currentNativeControllerIdentity, type NativeRestartRecoveryClaim } from "./native-restart-recovery.js";
|
||||
|
||||
const EXITED_PID = 4_194_304;
|
||||
|
||||
@@ -40,7 +41,8 @@ describe("a scheduled same-run resume reaching the provider runtime", () => {
|
||||
checkpointExists: true, originalFailureCode: "native_session_recovery_failed" };
|
||||
|
||||
async function resumeOnce(options: { failureDetail: Record<string, unknown>; stopRecorded: boolean; viaReconciler?: boolean;
|
||||
failedRelaunch?: boolean; unidentifiedStopRecorded?: boolean; liveRunnerForRun?: boolean }) {
|
||||
failedRelaunch?: boolean; unidentifiedStopRecorded?: boolean; liveRunnerForRun?: boolean;
|
||||
remoteReattach?: { replaceUnansweredRunner?: true } }) {
|
||||
const issueId = randomUUID();
|
||||
const runId = randomUUID();
|
||||
const [issue] = await db.insert(issues).values({ id: issueId, companyId, title: "Resume the lost turn",
|
||||
@@ -99,12 +101,23 @@ describe("a scheduled same-run resume reaching the provider runtime", () => {
|
||||
? spawn(process.execPath, ["-e", "setTimeout(() => {}, 60_000)", "--", "--run-id", runId], { stdio: "ignore" }) : null;
|
||||
if (liveRunner) await once(liveRunner, "spawn");
|
||||
let leaseOwner: string | undefined;
|
||||
let restartRecovery: NativeRestartRecoveryClaim | undefined;
|
||||
if (options.remoteReattach) {
|
||||
const controller = await currentNativeControllerIdentity();
|
||||
leaseOwner = `reattach-${runId}`;
|
||||
await db.update(nativeRunFinalizations).set({ phase: "observed", leaseOwner, leaseExpiresAt: new Date(Date.now() + 60_000),
|
||||
controllerBootId: controller.bootId, controllerPid: controller.pid, controllerGeneration: 7 })
|
||||
.where(eq(nativeRunFinalizations.runId, runId));
|
||||
restartRecovery = { kind: "reattach_remote_runner", runId, leaseOwner, controllerGeneration: 7, providerAttempt: 2,
|
||||
restartKind: "hard", recoveryRequestId: null, remote: { providerLeaseId: "sandbox", remoteCwd: "/workspace" },
|
||||
...options.remoteReattach };
|
||||
}
|
||||
if (options.viaReconciler) {
|
||||
const claims = await claimNativeSessionResumptions({ db, runnerInstanceId: native.runnerInstanceId, runIds: [runId] });
|
||||
expect(claims).toEqual([expect.objectContaining({ runId })]);
|
||||
leaseOwner = claims[0]!.leaseOwner;
|
||||
}
|
||||
await executePaperclipNativeSession({ db, execution, runnerInstanceId: native.runnerInstanceId, leaseOwner, backend: {
|
||||
await executePaperclipNativeSession({ db, execution, runnerInstanceId: native.runnerInstanceId, leaseOwner, restartRecovery, backend: {
|
||||
descriptor: async () => { throw new Error("Unexpected provider call"); },
|
||||
openSession: async () => { throw new Error("Unexpected provider call"); },
|
||||
} }).catch(() => undefined).finally(() => { liveRunner?.kill("SIGKILL"); });
|
||||
@@ -135,6 +148,11 @@ describe("a scheduled same-run resume reaching the provider runtime", () => {
|
||||
expect(passed).toEqual([false]);
|
||||
});
|
||||
|
||||
it.each([{}, { replaceUnansweredRunner: true as const }])("continues a lost turn on a same-run remote Runner reattachment (%j)", async (remoteReattach) => {
|
||||
const { passed } = await resumeOnce({ failureDetail: exactResume, stopRecorded: false, remoteReattach });
|
||||
expect(passed).toEqual([true]);
|
||||
});
|
||||
|
||||
it("keeps the lost turn failing when the reconciler claimed a resume without a stop receipt", async () => {
|
||||
const { passed } = await resumeOnce({ failureDetail: exactResume, stopRecorded: false, viaReconciler: true });
|
||||
expect(passed).toEqual([false]);
|
||||
|
||||
@@ -12321,7 +12321,8 @@ async function executePaperclipNativeSessionAttempt(
|
||||
trace.activate(runnerSessionStartupScope);
|
||||
const runExecution = () =>
|
||||
executeNativeSession({
|
||||
resumeInterruptedTurn: input.restartRecovery?.kind === "resume_dead_runner" || scheduledResumeContinuation,
|
||||
resumeInterruptedTurn: input.restartRecovery?.kind === "resume_dead_runner"
|
||||
|| input.restartRecovery?.kind === "reattach_remote_runner" || scheduledResumeContinuation,
|
||||
getFreshSessionHandoff: input.getFreshSessionHandoff,
|
||||
// A failed seal commits nothing: the messages stay queued for the
|
||||
// turn boundary instead of failing this run before its turn.
|
||||
|
||||
Reference in new issue
Block a user