From 59ded526be0368bf10c21ddec2d25e131d028cc6 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 11 Nov 2026 04:22:09 -0600 Subject: [PATCH] fix(native): ask once to re-check Paperclip before resting blocked on its own stall (#20779) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs report their result through a completion report. The server checks the report before it accepts it. > - When the Paperclip server pauses or restarts, the agent's Paperclip tool calls stay unanswered. They return later, all together. > - The agent can read that silence as an outage. It then reports a blocker owned by a "Paperclip ... service" with reasonCode `external_service_unavailable`. > - The server accepts that report. The task rests blocked with no Retry. Paperclip's own stall stops the task. Our rule says that must not happen. > - This pull request rejects that blocker report one time per run. The correction tells the agent that Paperclip is reachable now and to re-check. > - The benefit is that the agent continues in the same turn and reports the real outcome. ## Linked Issues or Issue Description **What happened?** In a restart-chaos test, the server was paused (SIGSTOP) for about 2 to 5 minutes while a Codex agent saved a task document. `write_document` and `read_document` did not return during the pause. After the pause, the save had succeeded, but the agent reported a blocker: owner "Paperclip document service", reasonCode `external_service_unavailable`. The run succeeded with disposition `blocked`. The task rested blocked with `canRetry=false` and no automatic recovery. A second test (two long server holds, then a same-run resume) gave the same result. **Expected behavior** A Paperclip stall or restart must not leave a task stuck. The agent must re-check Paperclip after it is reachable and finish the task. **Steps to reproduce** 1. Start a native Codex task that saves a task document. 2. Pause the server process with SIGSTOP for about 2 minutes during the save, then resume it. 3. The agent reports a blocker owned by a Paperclip service. The task rests blocked with no Retry. **Paperclip version or commit** master at 62b7733161 (test base) and eb75c9bb85. ## What Changed - `nativeCompletionFeedback` now rejects a `blocked` report one time per run when all of these are true: the blocker owner kind is `system`, the owner name contains "Paperclip", the reasonCode is `external_service_unavailable`, the run is still running, and it owns the task. - The correction tells the agent: Paperclip answered this report, so it is reachable now. Re-read the current state before writing again. Repeat a needed write with the same `idempotencyKey`. Report blocked only if a fresh Paperclip call fails now. - The second identical report in the same run is accepted as before, so a persistent real failure still rests blocked. - Added a regression test. It also covers concatenated owner names such as "PaperclipAI document service" and "paperclip_runner" (the match is a case-insensitive substring). ## Verification - `cd server && npx vitest run src/services/native-runtime/native-completion-feedback.test.ts` passes (32 tests). - Red on master: the new test fails with `promise resolved "'Blocker report accepted. ...'" instead of rejecting`. Green with this change. - `cd server && npx tsc --noEmit -p tsconfig.json` passes. ## Risks - Low risk. The change only adds one correction message inside the same turn. It does not change a status, a database row, or a schedule. - The once-per-run limit is kept in process memory. After a server restart, a run can get the correction one more time. This is bounded and harmless. - The match uses the agent's own owner name. That only selects the correction. It does not prove anything or replay a side effect. The agent's next Paperclip calls give the real proof. Writes use idempotency keys, so a repeated write does not apply twice. - This does not add a Retry to a task that already rests blocked from an older report. That is a separate product decision. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (no user docs change; the behavior is explained in code comments) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green (pending on the latest head) - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups (pending on the latest head) - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude --- .../native-completion-feedback.test.ts | 30 +++++++++++++++++++ .../native-completion-feedback.ts | 29 ++++++++++++++++++ 2 files changed, 59 insertions(+) diff --git a/server/src/services/native-runtime/native-completion-feedback.test.ts b/server/src/services/native-runtime/native-completion-feedback.test.ts index 841878e195b..1b0eb6fa773 100644 --- a/server/src/services/native-runtime/native-completion-feedback.test.ts +++ b/server/src/services/native-runtime/native-completion-feedback.test.ts @@ -410,6 +410,36 @@ describe("native final-response feedback", () => { await db.update(issues).set({ status: "in_progress", executionRunId: newerRunId }).where(eq(issues.id, value.issueId)); await expect(nativeCompletionFeedback(db, value.runId, report)).resolves.toContain("newer run owns"); }); + it("asks once to re-check Paperclip instead of resting blocked on its own stall", async () => { + // Chaos run 2b600576: a SIGSTOPped server left write_document/read_document + // unanswered; the agent reported the Paperclip document service as an + // external outage and the task rested blocked with no Retry. + const value = await fixture(); + const report: PrpStructuredRunResult = { ...done, reportedWorkDisposition: "blocked", + completionClaim: { ...done.completionClaim, objectiveSatisfied: false }, + blocker: { reasonCode: "external_service_unavailable", owner: { kind: "system", name: "Paperclip document service" }, + scope: "task_wide", unblockAction: "Restore document API responsiveness" } }; + await expect(nativeCompletionFeedback(db, value.runId, report)).rejects.toThrow(/Paperclip answered this report[\s\S]*same idempotencyKey/); + // Once per run: a Paperclip failure that persists after the re-check rests blocked. + await expect(nativeCompletionFeedback(db, value.runId, report)).resolves.toContain("Blocker report accepted"); + // Concatenated names such as "PaperclipAI document service" also name a Paperclip service. + const concatenated = await fixture(); + await expect(nativeCompletionFeedback(db, concatenated.runId, { ...report, blocker: { ...report.blocker!, owner: { kind: "system", name: "PaperclipAI document service" } } })) + .rejects.toThrow("Paperclip answered this report"); + const runnerNamed = await fixture(); + await expect(nativeCompletionFeedback(db, runnerNamed.runId, { ...report, blocker: { ...report.blocker!, owner: { kind: "system", name: "paperclip_runner" } } })) + .rejects.toThrow("Paperclip answered this report"); + // A genuinely external service, a non-system owner, or a finished run is not re-checked. + const other = await fixture(); + await expect(nativeCompletionFeedback(db, other.runId, { ...report, blocker: { ...report.blocker!, owner: { kind: "system", name: "GitHub API" } } })) + .resolves.toContain("Blocker report accepted"); + await expect(nativeCompletionFeedback(db, other.runId, { ...report, blocker: { ...report.blocker!, owner: { kind: "user", name: "Paperclip admin" } } })) + .resolves.toContain("Blocker report accepted"); + await expect(nativeCompletionFeedback(db, other.runId, { ...report, blocker: { ...report.blocker!, reasonCode: "missing_access" } })) + .resolves.toContain("Blocker report accepted"); + await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, other.runId)); + await expect(nativeCompletionFeedback(db, other.runId, report)).resolves.toContain("Blocker report accepted"); + }); it("asks a blocked provider to explain the cause and action instead of describing completed work", async () => { const value = await fixture(); const feedback = await nativeCompletionFeedback(db, value.runId, { ...done, reportedWorkDisposition: "blocked", diff --git a/server/src/services/native-runtime/native-completion-feedback.ts b/server/src/services/native-runtime/native-completion-feedback.ts index 9d3de432a54..18969541880 100644 --- a/server/src/services/native-runtime/native-completion-feedback.ts +++ b/server/src/services/native-runtime/native-completion-feedback.ts @@ -32,6 +32,31 @@ import { const repositoryYieldCorrection = 'This task has a saved repository preparation for its next admission. This is routine setup, not a blocker. Call paperclip_finish with reportedWorkDisposition="yielded" and omit continuation (or use null); Paperclip will select the saved workspace continuation. Preserve the completed work and explain what remains. Do not repeat setup, set a timer, or ask the user to restart.'; +// A Paperclip stall or restart leaves the agent's Paperclip tool calls +// unanswered until the server is back; they then return together. The agent can +// read that silence as an outage and report it as a blocker. This report reached +// the server and is being answered, which is the server's own proof that its +// control plane is reachable now. Ask once per run to re-check with fresh +// Paperclip calls instead of resting the task blocked on a past stall. +const paperclipServiceRecheckCorrection = "Paperclip answered this report, so its control plane is reachable now. Paperclip tool calls that did not return earlier were most likely delayed by a Paperclip restart or pause, and their results may have arrived since. Do not report this as a blocker. Re-read the current Paperclip state (for example read_document) before writing again; when a write is still needed, repeat it with the same idempotencyKey so it cannot apply twice. Then continue the task and report its real outcome. Report blocked only if a fresh Paperclip call fails now."; +const PAPERCLIP_SERVICE_RECHECK_LIMIT = 1_000; +const paperclipServiceRechecked = new Set(); + +function isPaperclipServiceOutageBlocker(result: PrpStructuredRunResult) { + const blocker = result.blocker; + return result.reportedWorkDisposition === "blocked" && blocker?.owner.kind === "system" + && blocker.reasonCode === "external_service_unavailable" && (blocker.owner.name ?? "").toLowerCase().includes("paperclip"); +} + +/** Once per run (per server process), so a real persistent failure still rests blocked. */ +function claimPaperclipServiceRecheck(runId: string) { + if (paperclipServiceRechecked.has(runId)) return false; + if (paperclipServiceRechecked.size >= PAPERCLIP_SERVICE_RECHECK_LIMIT) + paperclipServiceRechecked.delete(paperclipServiceRechecked.values().next().value!); + paperclipServiceRechecked.add(runId); + return true; +} + function record(value: unknown): Record { return value && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; @@ -323,6 +348,10 @@ export async function nativeCompletionFeedback( throw new Error(repositoryYieldCorrection); } } + if (isPaperclipServiceOutageBlocker(result) && issue.executionRunId === run.id && run.status === "running" + && claimPaperclipServiceRecheck(run.id)) { + throw new Error(paperclipServiceRecheckCorrection); + } if (result.reportedWorkDisposition === "yielded" && result.continuation?.kind === "monitor") { return `Monitor wait accepted. The task remains active and Paperclip will wake its assignee at or after ${issue.monitorNextCheckAt!.toISOString()} with issue_monitor_due. End this turn; do not poll or mark the task done.`; }