mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
fix(native): finish the run from a provider turn a timed-out start left running (#21338)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native Codex run talks to a remote Runner through a controller. A
deploy or a same-run retry can reattach a new controller to the live
Runner.
> - A slow turn start can make the first controller give up with
"runnerd did not report the provider turn identity". Codex still runs
that turn.
> - The retry reattaches the Runner and adopts that turn, because the
Runner accepted it for this run's latest turn.start.
> - If the adopted turn finishes while recovery still runs, the saved
checkpoint has no active turn. The runtime then tries to start a fresh
turn, and the run fails with "session cannot start another turn". The
Runner is lost and the task needs a Retry.
> - This pull request makes the runtime treat a terminal that the
recovered session newly settled as the adopted provider terminal when
the saved checkpoint bound no turn.
> - The benefit is that a turn which a timed-out start left running
finishes the same run on the reattached Runner, with its session
context, and with no second turn/start.
## Linked Issues or Issue Description
**What happened?**
On staging, Codex runs on a remote Runner lost the Runner after a
same-run reattachment. Each chain started with a turn.start that timed
out ("runnerd did not report the provider turn identity") while Codex
ran the turn anyway. After the reattachment, the run failed with a
protocol integrity fault (thread_binding_mismatch,
tool_binding_mismatch, or native_event_replay_conflict "Provider
authoritative notification item/started did not name its execution
owner"). The task then fell back to a fresh session and lost its
context. A local reproduction with the real runnerd and the fake Codex
app server shows the same trigger also fails when the abandoned turn
finishes during the reattachment: the runtime tries a new turn and fails
with "session cannot start another turn".
**Expected behavior**
A turn that a timed-out turn.start left running belongs to the same run.
The reattached run adopts it, binds its late notifications, and finishes
from its outcome. Recovery does not kill the Runner and does not need a
manual Retry. A real cross-run or cross-task binding violation still
fails closed.
**Steps to reproduce**
1. Start a native Codex run on a remote Runner whose provider is slow to
emit turn/started after it accepts turn/start.
2. Let the controller give up with "runnerd did not report the provider
turn identity" while the Runner stays alive.
3. Let the same-run retry reattach the Runner.
4. Let the provider turn start and finish after the reattachment.
5. See the run fail instead of finishing from the adopted turn.
**Paperclip version or commit**
master at 22c9300dac (after #21321 and #21329)
**Deployment mode**
Paperclip Cloud staging, remote Runner in a Daytona sandbox
**Agent adapter(s) involved**
Codex (codex_app_server through paperclip-runnerd)
## What Changed
- `native-session-runtime.ts`: a terminal that the recovered session
newly settled now counts as the adopted provider terminal when the saved
checkpoint had no active turn. The runtime consumes that terminal
instead of starting a fresh turn.
- New `runnerd-abandoned-turn-start-reattach.test.ts` with the real
runnerd and the fake Codex app server. Each attempt runs in a fresh
module graph, as each attempt runs in its own executor process in
production. The first attempt times out on turn.start against an
unreachable remote Runner and leaves it alive.
- Red/green: the provider turn starts and finishes only after the
reattachment. The run now succeeds from `provider-turn-1` with one
turn/start and no duplicate Runner launch. Before the fix it failed with
"session cannot start another turn".
- Guard: output the abandoned turn reported while no controller was
attached binds to the reattached run's adopted turn with no
`session.failed`, and the Runner stays alive.
## Verification
- `npx vitest run
src/live/runnerd-abandoned-turn-start-reattach.test.ts` in
`packages/paperclip-runner`: red without the runtime change ("session
cannot start another turn"), green with it.
- `npx vitest run src/native-session-runtime.test.ts
src/live/runnerd-restart-continuation.test.ts
src/live/runnerd-abandoned-turn-start-reattach.test.ts
src/live/runnerd-turn-start-timeout-retention.test.ts
src/live/runnerd-codex-transport-cut-turn-start.test.ts
src/drivers/codex/codex-app-server-driver.terminal-replay.test.ts
src/drivers/codex/codex-app-server-driver.recovery.test.ts`: 7 files,
386 tests passed.
- `tsc -p tsconfig.test.json --noEmit` in `packages/paperclip-runner`:
no new errors (the existing `src/eval/workflow-harness.ts` errors need
the eval kernel build).
## Risks
- The runtime now adopts a newly settled terminal when the saved
checkpoint bound no turn. The driver records a turn terminal only for a
turn that it bound as active (the persisted active turn, the
uncheckpointed disposition turn, or this run's own accepted turn). A
turn of another run or task is never bound, so it still fails closed in
the driver.
- The staging chains failed with binding-mismatch codes. The local
reproduction reaches the same trigger and the adoption path, but the
fake provider cannot hold a live turn while it answers other requests,
so the exact staging codes did not reproduce locally. Late items that
reach a reattached controller before it knows the provider turn identity
can still fail closed. That path needs a separate fix if it recurs after
this change.
## Model Used
- Claude Opus 5.5 (claude-opus-5-5) in Claude Code, with tool use
(shell, file edits, browser reads of staging run events).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Refs #21321, #21329
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
d16becac04
commit
cb94108484
2 files changed
+246
No files matched your search
@@ -0,0 +1,245 @@
|
||||
import { execFileSync, spawn } from "node:child_process";
|
||||
import { createServer } from "node:http";
|
||||
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { expect, it, vi } from "vitest";
|
||||
import type { NativeExecutionInputV1 } from "../contracts/native-execution.js";
|
||||
import type { PersistedNativeSession } from "../contracts/native-session-backend.js";
|
||||
import type * as durableControlPlane from "../control-plane/durable-prp-control-plane.js";
|
||||
import type { PrpEvent } from "../protocol/replay-contract.js";
|
||||
import type { executeNativeSession } from "../native-session-runtime.js";
|
||||
import type { createCapabilityRunnerdCodexTransport } from "./runnerd-codex-transport.js";
|
||||
import { defaultCapabilityRunnerdBinary, fakeCodex, fakeCodexArgs } from "../../test/support/runnerd-transport.js";
|
||||
|
||||
type ControlPlaneModule = typeof import("../control-plane/durable-prp-control-plane.js");
|
||||
type Transport = ReturnType<typeof createCapabilityRunnerdCodexTransport>;
|
||||
|
||||
function alive(pid: number | null | undefined): boolean {
|
||||
if (!pid) return false;
|
||||
try { process.kill(pid, 0); } catch { return false; }
|
||||
try {
|
||||
return !execFileSync("ps", ["-o", "stat=", "-p", String(pid)], { encoding: "utf8" }).trim().startsWith("Z");
|
||||
} catch { return false; }
|
||||
}
|
||||
|
||||
const kill = (pid: number | null | undefined) => {
|
||||
if (!pid) return;
|
||||
try { process.kill(pid, "SIGKILL"); } catch (error) { if (!["ESRCH", "EPERM"].includes(String((error as NodeJS.ErrnoException).code))) throw error; }
|
||||
};
|
||||
|
||||
async function freshExecutorProcess() {
|
||||
vi.resetModules();
|
||||
const [backend, control, runtime, transport] = await Promise.all([
|
||||
import("../backends/codex-native-backend.js"),
|
||||
import("../control-plane/durable-prp-control-plane.js"),
|
||||
import("../native-session-runtime.js"),
|
||||
import("./runnerd-codex-transport.js"),
|
||||
]);
|
||||
return {
|
||||
createBackend: backend.createRunnerdNativeSessionBackend,
|
||||
control,
|
||||
execute: runtime.executeNativeSession,
|
||||
createTransport: transport.createCapabilityRunnerdCodexTransport,
|
||||
};
|
||||
}
|
||||
|
||||
function withUnreachableRemoteRunner(control: ControlPlaneModule) {
|
||||
let unreachable = true;
|
||||
const OriginalCore = control.DurablePrpControlPlane;
|
||||
const spy = vi.spyOn(control, "DurablePrpControlPlane")
|
||||
.mockImplementation(function(options: ConstructorParameters<typeof OriginalCore>[0]) {
|
||||
const core = new OriginalCore(options);
|
||||
const queueCommand = core.queueCommand.bind(core);
|
||||
core.queueCommand = ((type: string, ...rest: unknown[]) => {
|
||||
if (unreachable && type === "session.snapshot") throw new Error("runner snapshot unavailable");
|
||||
if (unreachable && (type === "runner.suspend" || type === "turn.stop")) return undefined;
|
||||
return (queueCommand as (...args: unknown[]) => unknown)(type, ...rest);
|
||||
}) as typeof core.queueCommand;
|
||||
return core;
|
||||
} as unknown as typeof OriginalCore);
|
||||
return { restore: () => { unreachable = false; spy.mockRestore(); } };
|
||||
}
|
||||
|
||||
async function abandonTurnStart(label: string, providerArgs: string[]) {
|
||||
const root = await mkdtemp(join(tmpdir(), `runnerd-abandoned-${label}-`));
|
||||
const gate = join(root, "turn-started-gate");
|
||||
const callLog = join(root, "calls.log");
|
||||
const identity = { runId: `run-${label}`, sessionId: `session-${label}`, companyId: `company-${label}`,
|
||||
issueId: `issue-${label}`, agentId: `agent-${label}` };
|
||||
const execution: NativeExecutionInputV1 = {
|
||||
schema: "paperclip.native-execution-input.v1",
|
||||
binding: { runId: identity.runId, companyId: identity.companyId, issueId: identity.issueId, agentId: identity.agentId,
|
||||
executionWorkspaceId: `workspace-${label}` },
|
||||
session: { normalizedSessionId: identity.sessionId, driverKind: "codex_app_server", protocolVersion: 1,
|
||||
lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
|
||||
task: { identifier: "TEST-21", title: "Finish checks", description: null, prompt: "Finish the checks.", workMode: "standard" },
|
||||
workspace: { cwd: root, repoUrl: null, repoRef: null, branchName: null },
|
||||
provider: { kind: "codex", model: null },
|
||||
completionContract: { id: `contract-${label}`, sha256: "digest", schemaVersion: "paperclip.completion-contract.v1", contract: {
|
||||
revision: "1", objective: "Finish checks", criteria: [{ id: "checks", requirement: "Check the outcome" }],
|
||||
} },
|
||||
interactionResponses: [], credentialBindings: [],
|
||||
};
|
||||
const children: ReturnType<typeof spawn>[] = [];
|
||||
const launch = vi.fn((spec: durableControlPlane.RunnerProcessLaunchSpec) => {
|
||||
const child = spawn(spec.command, [...spec.args], { cwd: spec.cwd, env: spec.environment, stdio: "ignore" });
|
||||
children.push(child);
|
||||
return {
|
||||
child,
|
||||
completion: new Promise<durableControlPlane.RunnerProcessResult>((resolveExit, rejectExit) => {
|
||||
child.once("error", rejectExit);
|
||||
child.once("exit", (code, exitSignal) => resolveExit({ code, signal: exitSignal, stdout: "", stderr: "" }));
|
||||
}),
|
||||
isExitConfirmed: () => child.exitCode !== null || child.signalCode !== null,
|
||||
};
|
||||
});
|
||||
const server = createServer();
|
||||
let authority: durableControlPlane.DurablePrpControlPlane | null = null;
|
||||
server.on("upgrade", (request, socket, head) => {
|
||||
if (!authority) { socket.destroy(); return; }
|
||||
authority.handleUpgrade(request, socket, "/runner", head);
|
||||
});
|
||||
await new Promise<void>(resolveListen => server.listen(0, "127.0.0.1", resolveListen));
|
||||
const address = server.address();
|
||||
if (!address || typeof address === "string") throw new Error("Expected a control-plane listener");
|
||||
const options = {
|
||||
controlPlaneRegistration: async (next: durableControlPlane.DurablePrpControlPlane) => {
|
||||
authority = next;
|
||||
return { connectUrl: `ws://127.0.0.1:${address.port}/runner`, release: async () => { if (authority === next) authority = null; } };
|
||||
},
|
||||
runnerBinary: defaultCapabilityRunnerdBinary(), codexCommand: fakeCodex,
|
||||
codexArgs: fakeCodexArgs(root, "--turn-started-gate", gate, "--call-log", callLog, ...providerArgs),
|
||||
stateDirectory: root, runnerReconnectGraceMs: 60_000, closeGraceMs: 600, turnStartTimeoutMs: 2_000,
|
||||
prpIdentity: { runnerInstanceId: `runner-${label}`, environmentLeaseId: `lease-${label}`, runId: identity.runId,
|
||||
normalizedSessionId: identity.sessionId, turnId: `turn-${label}`, itemId: `item-${label}` },
|
||||
};
|
||||
const events: PrpEvent[] = [];
|
||||
let checkpoint: PersistedNativeSession | null = null;
|
||||
const controlPlane: Parameters<typeof executeNativeSession>[0]["controlPlane"] = {
|
||||
async openRun() {},
|
||||
async checkpointSession(snapshot) { checkpoint = structuredClone(snapshot); },
|
||||
async appendEvent(event) {
|
||||
if (!("sourceSeq" in event)) throw new Error("Expected a durable PRP event");
|
||||
events.push(event);
|
||||
return { cursor: events.length, highestContiguousSourceSeq: event.sourceSeq, disposition: "committed" };
|
||||
},
|
||||
async replayEvents(request) {
|
||||
const source = events.filter(event => event.sourceInstanceId === request.sourceInstanceId);
|
||||
return { events: source.filter(event => event.sourceSeq > request.afterSourceSeq),
|
||||
highestContiguousSourceSeq: source.at(-1)?.sourceSeq ?? 0 };
|
||||
},
|
||||
async completeRun() {},
|
||||
};
|
||||
const transports: Transport[] = [];
|
||||
const cleanup = async () => {
|
||||
await rm(gate, { force: true });
|
||||
for (const bundle of [...transports].reverse()) await bundle.transport.close().catch(() => undefined);
|
||||
for (const child of children) kill(child.pid);
|
||||
for (const bundle of transports) { kill(bundle.evidence().runnerPid); kill(bundle.evidence().providerPid); }
|
||||
await new Promise<void>(resolveClose => server.close(() => resolveClose()));
|
||||
await rm(root, { recursive: true, force: true });
|
||||
};
|
||||
try {
|
||||
const firstProcess = await freshExecutorProcess();
|
||||
const remote = withUnreachableRemoteRunner(firstProcess.control);
|
||||
try {
|
||||
const first = firstProcess.createTransport({ ...options, runnerProcessLauncher: launch });
|
||||
transports.push(first);
|
||||
await expect(firstProcess.execute({
|
||||
input: execution, runnerInstanceId: options.prpIdentity.runnerInstanceId, controlPlaneInstanceId: `controller-${label}`,
|
||||
backend: firstProcess.createBackend(execution, { runnerInstanceId: options.prpIdentity.runnerInstanceId, transportFactory: () => first.transport }),
|
||||
timeoutMs: 30_000, requireSessionCloseBeforeReturn: true, controlPlane,
|
||||
})).rejects.toThrow("runnerd did not report the provider turn identity");
|
||||
} finally {
|
||||
remote.restore();
|
||||
}
|
||||
const runner = children[0]!;
|
||||
expect(alive(runner.pid)).toBe(true);
|
||||
const startedAt = transports[0]!.transport.processInfo!().startedAt;
|
||||
const saved = checkpoint as PersistedNativeSession | null;
|
||||
if (saved === null) throw new Error("Expected the first attempt to checkpoint its session");
|
||||
expect(saved.activeTurnId ?? null).toBeNull();
|
||||
const reattach = async () => {
|
||||
const secondProcess = await freshExecutorProcess();
|
||||
const duplicateLauncher = vi.fn(() => { throw new Error("duplicate runner spawn attempted"); });
|
||||
const adopted = secondProcess.createTransport({
|
||||
...options,
|
||||
resumeDynamicTools: [],
|
||||
resumeActiveTurnId: saved.activeTurnId ?? null,
|
||||
resumeProviderSession: { driverSessionId: saved.sessionId, providerSessionId: saved.providerSessionId ?? null },
|
||||
runnerProcessLauncher: duplicateLauncher,
|
||||
adoptExistingRunner: {
|
||||
pid: runner.pid!, processGroupId: runner.pid!, startedAt, signal: () => true,
|
||||
isAlive: () => alive(runner.pid),
|
||||
},
|
||||
});
|
||||
transports.push(adopted);
|
||||
const backend = secondProcess.createBackend(execution, {
|
||||
runnerInstanceId: options.prpIdentity.runnerInstanceId, transportFactory: () => adopted.transport,
|
||||
});
|
||||
return {
|
||||
adopted, duplicateLauncher, backend,
|
||||
execute: (persistedSession: PersistedNativeSession) => secondProcess.execute({
|
||||
input: execution, persistedSession, resumeInterruptedTurn: true, runnerInstanceId: options.prpIdentity.runnerInstanceId,
|
||||
controlPlaneInstanceId: `controller-${label}-2`, timeoutMs: 30_000, controlPlane, backend,
|
||||
}),
|
||||
};
|
||||
};
|
||||
return { root, gate, callLog, runner, saved, events, reattach, cleanup };
|
||||
} catch (error) {
|
||||
await cleanup();
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
it("finishes the run from the provider turn a timed-out turn.start left running once it completes after the reattachment", async () => {
|
||||
const attempt = await abandonTurnStart("completed-start", []);
|
||||
let release: ReturnType<typeof setInterval> | undefined;
|
||||
try {
|
||||
const second = await attempt.reattach();
|
||||
release = setInterval(() => {
|
||||
if (!second.adopted.evidence().diagnostics.some(line => line.includes("restored its durable PRP session"))) return;
|
||||
clearInterval(release);
|
||||
void writeFile(attempt.gate, "start");
|
||||
}, 20);
|
||||
release.unref();
|
||||
const completion = await second.execute(attempt.saved);
|
||||
expect(completion).toMatchObject({ turnId: "provider-turn-1", terminal: { runTerminalState: "succeeded" } });
|
||||
expect(second.duplicateLauncher).not.toHaveBeenCalled();
|
||||
expect(attempt.events.some(event => event.eventType === "session.failed")).toBe(false);
|
||||
const calls = (await readFile(attempt.callLog, "utf8")).trim().split(/\r?\n/);
|
||||
expect(calls.filter(call => call === "turn/start")).toHaveLength(1);
|
||||
} finally {
|
||||
clearInterval(release);
|
||||
await attempt.cleanup();
|
||||
}
|
||||
}, 90_000);
|
||||
|
||||
it("binds output the abandoned provider turn reported while no controller was attached to the reattached run", async () => {
|
||||
const attempt = await abandonTurnStart("buffered-output", ["--hold-turn", "--held-final-message"]);
|
||||
try {
|
||||
await writeFile(attempt.gate, "start");
|
||||
await vi.waitFor(async () => {
|
||||
expect(await readFile(join(attempt.root, "runner", "codex-provider-state.json"), "utf8")).toContain("Durable Cloud final");
|
||||
}, { timeout: 15_000, interval: 50 });
|
||||
const second = await attempt.reattach();
|
||||
const recovery = await second.backend.recoverSession!(attempt.saved, { signal: new AbortController().signal });
|
||||
expect(recovery).toMatchObject({ recovered: true });
|
||||
const session = recovery.session!;
|
||||
const received: PrpEvent[] = [];
|
||||
const reading = (async () => { for await (const event of session.events()) received.push(event); })().catch(() => undefined);
|
||||
await vi.waitFor(() => {
|
||||
expect(received.find(event => event.eventType === "session.failed")).toBeUndefined();
|
||||
expect(received.some(event => event.eventType === "item.completed" && event.turnId === "provider-turn-1"
|
||||
&& JSON.stringify(event.payload).includes("Durable Cloud final"))).toBe(true);
|
||||
}, { timeout: 15_000, interval: 50 });
|
||||
expect((await session.snapshot()).activeTurnId).toBe("provider-turn-1");
|
||||
expect(alive(attempt.runner.pid)).toBe(true);
|
||||
expect(second.duplicateLauncher).not.toHaveBeenCalled();
|
||||
await session.close({ reason: "test complete" }).catch(() => undefined);
|
||||
await reading;
|
||||
} finally {
|
||||
await attempt.cleanup();
|
||||
}
|
||||
}, 90_000);
|
||||
@@ -2987,6 +2987,7 @@ export async function executeNativeSession(
|
||||
(persistedTerminal) => persistedTerminal.turnId === terminal.turnId,
|
||||
) &&
|
||||
(terminal.turnId === persistedSession?.activeTurnId ||
|
||||
!persistedSession?.activeTurnId ||
|
||||
recoveredSnapshot.dispositionOnlyRecoveryConsumed),
|
||||
),
|
||||
);
|
||||
|
||||
Reference in new issue
Block a user