mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 07:10:34 +01:00
fix(native): tell a restart continuation that earlier processes may be gone (#19804)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a control-plane restart loses a turn, the native runtime sends a same-conversation continuation (#19501, #19699, #19771) > - A restart can also kill shell or background processes that earlier turns started > - In three restart storms, the continuation agent could not inspect its killed `bench.sh` and set the task to blocked > - The continuation prompt told the agent not to "blindly rerun an action whose outcome is unknown", so the agent did not re-run the command > - This pull request tells the agent that earlier processes may be gone, and to check the workspace outputs and re-run a command whose expected output is missing > - The benefit is that a task can finish after a restart instead of blocking on a process that no longer exists ## Linked Issues or Issue Description Refs #19501, #19699, #19771 **What happened?** Restart chaos workload B5 (Codex and Claude) failed this way three times: storm 6b t14, s7111 t16 and storm 10 t12. The agent started `bench.sh` in the background. A restart killed it. The restart continuation agent tried to find its tool session and got "process inspection is denied". It then blocked the task. The comment read: "Blocked by the Paperclip runtime restart: the benchmark session is unavailable…". It added that it "cannot verify whether the original run is still active or recover its output, so I have not rerun it". **Expected behavior** The agent knows it is in a resumed session after a restart. It checks the workspace for the command's outputs. If the expected output is missing, it re-runs the command and finishes the task. **Root cause** `packages/paperclip-runner/src/native-session-runtime.ts`, `restartContinuation`: the prompt did not say that processes from earlier turns may be gone. It told the agent to inspect each unfinished command's "current state" and not to "blindly rerun an action whose outcome is unknown". With the process gone, the agent could not inspect it, so it blocked. ## What Changed - The restart continuation prompt now says: - This is a resumed session after a Paperclip control-plane restart. - Shell commands and background processes from earlier turns may no longer be running, and their tool sessions may be gone. Do not block only because one cannot be found or inspected. - Check the workspace for the outputs or artifacts the command should have produced. If the expected output is present, use it. If the expected output is missing, re-run the command. - Do not repeat completed work. Before repeating an action with external effects (a push, a message, a deployment), check whether it already took effect. If you cannot tell, report the blocker. - The prompt does not say that re-running is always safe. Process teardown (#19727) has not merged. - `doc/execution-semantics.md`: one additive sentence. ## Verification - `native-session-runtime.test.ts`: the three restart continuation tests (Codex, ACPX, OpenCode) now check the new guidance. They were red before the prompt change and are green after it. - Guard: "uses the admitted session cwd in the task envelope" now checks that fresh turns and ordinary resumed turns do not contain the restart guidance. - `npx vitest run src/native-session-runtime.test.ts src/drivers/codex/ src/live/runnerd-restart-continuation.test.ts` in `packages/paperclip-runner`: 603 passed. - `tsc -p tsconfig.json` and `tsc -p tsconfig.test.json` for `packages/paperclip-runner` are clean. - Not run: a live restart-chaos replay with a real model. ## Risks - Low. This changes only the prompt text of the restart continuation. Ordinary turns do not change. - An agent may now re-run a command whose earlier run had partial side effects. The prompt asks it to check external effects first, and to re-run only when the expected output is missing. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
6fb6220d96
commit
2468b3f907
3 files changed
+26
-6
No files matched your search
@@ -2643,7 +2643,10 @@ session's own protocol failure ends a pending turn start, the run reports
|
||||
|
||||
The continuation tells the agent to finish in that turn, not to end it while
|
||||
waiting for a background command or a notification, and to report one
|
||||
`paperclip_finish` or `paperclip_block`. The continuation only borrows the
|
||||
`paperclip_finish` or `paperclip_block`. It also says that processes started in
|
||||
earlier turns may no longer run after the restart: the agent checks the
|
||||
workspace for their outputs and re-runs a command when its expected output is
|
||||
missing, instead of blocking on the lost process. The continuation only borrows the
|
||||
one-shot disposition marker so a crash can adopt its accepted turn. If the
|
||||
continuation completes without a semantic result, the driver returns the marker
|
||||
and the same attempt sends one disposition request. That request does not use
|
||||
|
||||
@@ -212,6 +212,20 @@ function canonicalTestJson(value: unknown): string {
|
||||
return JSON.stringify(value) ?? "undefined";
|
||||
}
|
||||
|
||||
// A restart can kill processes that earlier turns started. The continuation
|
||||
// must point the agent at workspace outputs instead of the lost process.
|
||||
const RESTART_LOST_PROCESS_GUIDANCE = "may no longer be running";
|
||||
function expectRestartContinuationGuidance(text: string) {
|
||||
const events = String(JSON.parse(text).events);
|
||||
expect(events).toContain("This is a resumed session after a Paperclip control-plane restart.");
|
||||
expect(events).toContain("Shell commands and background processes started in earlier turns " + RESTART_LOST_PROCESS_GUIDANCE);
|
||||
expect(events).toContain("check the workspace for the outputs or artifacts");
|
||||
expect(events).toContain("If the expected output is missing, re-run the command.");
|
||||
expect(events).toContain("Do not block only because such a process or its tool session cannot be found or inspected.");
|
||||
expect(events).not.toContain("blindly rerun");
|
||||
expect(events).toContain("Reconcile any unfinished tool or command");
|
||||
}
|
||||
|
||||
function runnerEvent(
|
||||
sourceSeq: number,
|
||||
eventType: PrpEvent["eventType"],
|
||||
@@ -334,6 +348,8 @@ describe("executeNativeSession recovery", () => {
|
||||
await executeNativeSession({ input: executionInput, backend, controlPlane: port,
|
||||
runnerInstanceId: "runner-recovery", controlPlaneInstanceId: "control-recovery" });
|
||||
expect(JSON.parse(startTurn.mock.calls[0]![0].message.text).workspace).toEqual({ cwd: cwd ?? originalWorkspace.cwd });
|
||||
// Fresh and ordinary resumed turns do not carry restart-continuation guidance.
|
||||
expect(startTurn.mock.calls[0]![0].message.text).not.toContain(RESTART_LOST_PROCESS_GUIDANCE);
|
||||
expect(executionInput.workspace).toEqual(originalWorkspace);
|
||||
});
|
||||
|
||||
@@ -563,7 +579,7 @@ describe("executeNativeSession recovery", () => {
|
||||
expect(startTurn).toHaveBeenCalledOnce();
|
||||
const submitted = startTurn.mock.calls[0]![0] as Parameters<NativeSession["startTurn"]>[0];
|
||||
expect(submitted.continuation).toBe(true);
|
||||
expect(submitted.message.text).toContain("Reconcile any unfinished tool or command");
|
||||
expectRestartContinuationGuidance(submitted.message.text);
|
||||
expect(submitted.message.text).not.toContain(input.task.prompt);
|
||||
expect(events.filter(event => event.eventType === "turn.failed")).toHaveLength(1);
|
||||
expect(events.filter(event => event.eventType === "run.terminal")).toHaveLength(1);
|
||||
@@ -631,7 +647,7 @@ describe("executeNativeSession recovery", () => {
|
||||
expect(startTurn).toHaveBeenCalledOnce();
|
||||
const submitted = startTurn.mock.calls[0]![0] as Parameters<NativeSession["startTurn"]>[0];
|
||||
expect(submitted.continuation).toBe(true);
|
||||
expect(submitted.message.text).toContain("Reconcile any unfinished tool or command");
|
||||
expectRestartContinuationGuidance(submitted.message.text);
|
||||
expect(nativeRestartInterruptedTurnId(interrupted)).toBe("turn-recovery");
|
||||
});
|
||||
|
||||
|
||||
@@ -2649,10 +2649,11 @@ export async function executeNativeSession(
|
||||
message: { role: "user" as const, text: JSON.stringify({
|
||||
schema: "paperclip.native-continuation.v1",
|
||||
events: [
|
||||
"Paperclip restarted and restored this same conversation. The previous turn was interrupted.",
|
||||
"This is a resumed session after a Paperclip control-plane restart. Paperclip restored this same conversation and workspace; the restart interrupted the previous turn.",
|
||||
"Continue the current user request from the recorded progress and preserved workspace.",
|
||||
"Reconcile any unfinished tool or command before proceeding: inspect its current state and recorded outcomes.",
|
||||
"Do not repeat completed work or blindly rerun an action whose outcome is unknown. If an external action cannot be reconciled, report the specific blocker.",
|
||||
"Shell commands and background processes started in earlier turns may no longer be running, and their tool sessions may be gone. Do not block only because such a process or its tool session cannot be found or inspected.",
|
||||
"Reconcile any unfinished tool or command before proceeding: check the workspace for the outputs or artifacts it should have produced. If the expected output is present, use it. If the expected output is missing, re-run the command.",
|
||||
"Do not repeat completed work. Before you repeat an action with external effects, such as a push, a message or a deployment, check whether it already took effect. If you cannot determine that, report the specific blocker.",
|
||||
"Finish in this turn. Nothing resumes this conversation after the turn ends, so do not end it to wait for a background command or a notification; wait for or check that work first.",
|
||||
"Then invoke exactly one paperclip_finish or paperclip_block with the accurate disposition.",
|
||||
].join("\n"),
|
||||
|
||||
Reference in new issue
Block a user