fix(retry): continue an accepted Retry past chat input held because it predates the stop (#21129)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person's Retry on a connected task (for example a GitHub pull
request review) must start work. #21080 made a Retry whose chat replay
is refused after the click continue from the recovered saved work
> - After #21080 was deployed, accepted Retries on a busy GitHub review
agent still did not start
> - Those tasks had a saved chat request that arrived while the failed
run was still running. Admission holds such a request as
`message_predates_stop`, so it can never start alone
> - #21080 refused to continue while any saved chat input existed, so
this held input blocked the continuation. For a second group, the latest
run was an admission that the recovery hold cancelled before dispatch,
and the recovered saved work belongs to the earlier failed run
> - This pull request lets the continuation adopt input that was held
because it predates the stop, and lets the recovered run that held a
cancelled admission count as the saved work
> - The benefit is that an accepted Retry starts work on these tasks too

## Linked Issues or Issue Description

Follow-up to #21080. No public issue exists. This is the bug
description, in the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

After #21080 was deployed, `POST /api/issues/:id/retry` returned 202 for
blocked tasks of a GitHub review agent (`paperclip_runner`), but no run
started. Each Retry receipt changed to `failed` about 5 seconds later
(on the 14:48 Retries, 5 seconds for one task and about 75 seconds for
two others), with no run and no `issue.retry_continued_from_saved_work`
activity. Every affected task had a saved chat request (a newer GitHub
review request) with an `issue.execution_waiting` record of reason
`message_predates_stop`. The request arrived while the failed run was
running, so admission holds it permanently. The interruption projection
kept pointing at that held request. In a second group of tasks, the
latest run was cancelled before dispatch with
`execution_reconciliation_required`, and the recovered saved work
belonged to the earlier failed run that held it.

Root cause: in `continueRefusedTaskChatRetry`
(`server/src/services/task-retry.ts`), any saved chat input from
`selectTaskRetryInput` stopped the continuation, so the receipt settled
`failed`. The held input can never start by itself, so nothing ran. The
recovered-work check also looked only at the latest run, so it missed
the recovered run that held a cancelled admission.

**Expected behavior**

An accepted Retry starts work. Saved input that admission holds because
it predates the stop joins the continuation, and the task continues from
the recovered saved work.

**Steps to reproduce**

1. Bind a task to a GitHub chat conversation. Let a native run fail with
`native_continuation_requires_reconciliation`, and recover its saved
work.
2. Save a chat request while that run is still running, so admission
holds it with `message_predates_stop`.
3. Click Retry while the chat identity lock is busy, so the source check
is deferred and then refused.
4. The route returns 202. The Retry receipt changes to `failed`, and no
run starts.

**Paperclip version or commit**

a65644d1e1 (the deployed #21080). Reproduced on ef2b53b675.

**Deployment mode**

Hosted (cloud) instance with Daytona remote sandboxes and a GitHub chat
connection.

**Agent adapter(s) involved**

`paperclip_runner` (native runs, and legacy admissions cancelled before
dispatch). The fix is in core Retry code and is not adapter-specific.

## What Changed

- `server/src/services/task-retry.ts` (`continueRefusedTaskChatRetry`):
- Saved chat input whose source comment was written no later than the
failed run's stop boundary (`userStopBoundaryAt`: the recorded user Stop
request, otherwise `finishedAt`) no longer stops the continuation. This
is the same boundary and timestamp that admission uses. Admission holds
that input as `message_predates_stop`. The continuation coalesces its
receipts into the Retry receipt (`coalescedIntoWakeupRequestId`), so the
task stops waiting on it. The receipt ids are recorded in the activity
details.
- Saved input created after the stop, and unadmitted chat input, still
settle the Retry as failed, as before.
- When the latest run is an admission cancelled before dispatch
(`execution_reconciliation_required`, never started), the recovered-work
check also accepts the run named by that run's
`executionWait.recoveryActionId`.
- `doc/execution-semantics.md`: both rules.
- Tests in `server/src/__tests__/chat-channels.integration.test.ts`: two
new variants of the #21080 regression test.
`person_with_input_before_stop` saves a chat request before the failed
run stops. `person_after_held_admission` adds a legacy admission that
the recovery hold cancelled. Both start one run that resumes the
recovered workspace (`resume_preserved_workspace`). The held-admission
fixture binds the task to the recovered workspace, as on the hosted
tasks. Refusal edges `input_after_stop` and `input_inside_user_stop`
settle the Retry failed with no run and leave the input deferred.

## Verification

- Red before the fix: both new variants fail. The Retry receipt is
`failed` (or stays held), and no run starts.
- Green after the fix: both variants pass, and the held input receipt is
coalesced into the Retry receipt.
- `npx vitest run src/__tests__/chat-channels.integration.test.ts -t
"[Rr]etry"` with `src/services/task-retry.test.ts` passes 369/369 on the
final commit.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- Staging was read with GET requests only.

## Risks

- Behavior change: input that arrived during a failed run no longer
starts its own turn through the chat retry. It is adopted into the
saved-work continuation, and the comment stays in the task thread.
Before this change, that input could never start at all.
- Input that arrived after the stop keeps its own admission path.
- No migration. Receipts that already failed stay failed. The next click
takes the new path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-12 10:34:36 -06:00
1 parent d5bd38468d
commit f2e449f8f6
3 files changed
+73 -15

No files matched your search

@@ -65892,11 +65892,19 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
},
);
it.each(["person", "person_paused_after_accept", "restart_recovery", "restart_recovery_joined_by_person"] as const)("continues an accepted GitHub task Retry from recovered saved work when its chat replay is refused after the click: %s", async initiator => {
const f = await currentGitHubTaskRetryFixture(false);
it.each(["person", "person_with_input_before_stop", "person_after_held_admission", "person_paused_after_accept", "restart_recovery", "restart_recovery_joined_by_person"] as const)("continues an accepted GitHub task Retry from recovered saved work when its chat replay is refused after the click: %s", async initiator => {
const f = await currentGitHubTaskRetryFixture(initiator === "person_with_input_before_stop");
try {
const companyId = f.fixture.companyId, issueId = f.issue.id;
const leaseId = randomUUID(), providerLeaseId = randomUUID(), workspaceId = randomUUID(), snapshotId = randomUUID();
if (f.pending) {
await db.update(agentWakeupRequests).set({ requestedAt: new Date(Date.now() - 60_000) })
.where(eq(agentWakeupRequests.id, f.pending.id));
await db.update(chatActions).set({ createdAt: new Date(Date.now() - 60_000) }).where(eq(chatActions.id, f.pending.id));
await db.update(issueComments).set({ createdAt: new Date(Date.now() - 60_000) })
.where(eq(issueComments.id, String(f.pending.payload.commentId)));
await db.update(heartbeatRuns).set({ finishedAt: new Date(Date.now() - 30_000) }).where(eq(heartbeatRuns.id, f.runId));
}
await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, f.runId));
await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, sourceIssueId: issueId,
name: "Review workspace", providerType: "cloud_sandbox", mode: "isolated_workspace", strategyType: "project_primary",
@@ -65932,6 +65940,19 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
executionWorkspaceId: workspaceId, phase: "workspace_recovery", status: "succeeded", finishedAt: new Date(),
metadata: { recoveryActionId, leaseId, providerLeaseId, snapshotId, generation: 1, sha256: "c".repeat(64),
storage: "remote_workspace", epoch: 2 } });
let sourceRunId: string = f.runId;
if (initiator === "person_after_held_admission") {
const [failed] = await readHeartbeatRunsById(f.runId);
const [held] = await db.insert(heartbeatRuns).values({ companyId, agentId: f.fixture.assignedAgentId,
status: "cancelled", errorCode: "execution_reconciliation_required", runtimeMode: "legacy", issueId,
wakeupRequestId: failed.wakeupRequestId, contextSnapshot: failed.contextSnapshot, finishedAt: new Date(),
createdAt: new Date(failed.createdAt.getTime() + 1_000),
resultJson: { stopReason: "execution_reconciliation_required", timeoutSource: "stale_queued_run_gate",
executionWait: { issueId, recoveryActionId } } }).returning();
await db.update(agentWakeupRequests).set({ runId: held.id }).where(eq(agentWakeupRequests.id, failed.wakeupRequestId!));
await db.update(issues).set({ executionWorkspaceId: workspaceId }).where(eq(issues.id, issueId));
sourceRunId = held.id;
}
const { getExecutionBlocker } = await import("../services/execution-blocker.js");
const blocker = await getExecutionBlocker(db, companyId, issueId);
const interruption = await taskInterruptionPresentation(db, f.issue);
@@ -65965,7 +65986,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
let request: Awaited<ReturnType<typeof requestTaskRetry>>;
try {
request = await requestTaskRetry({ db, companyId, issueId, actorId: "owner-user", actorSource: "cloud_tenant",
observedRunId: f.runId, idempotencyKey: randomUUID(), prepareChatRetry: f.service.prepareFailedChatRunRetry,
observedRunId: sourceRunId, idempotencyKey: randomUUID(), prepareChatRetry: f.service.prepareFailedChatRunRetry,
...(initiator.startsWith("person") ? {} : { initiatedBy: "restart_recovery" as const }) });
} finally { await releaseIdentity(); }
if (initiator === "restart_recovery_joined_by_person") {
@@ -66020,12 +66041,14 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
eq(activityLog.action, "issue.retry_continued_from_saved_work")));
expect(activity).toHaveLength(1);
expect(activity[0].details).toMatchObject({ requestId: request.requestId, refusedChatActionId: request.chatActionId });
if (f.pending) expect((await readAgentWakeupRequestsById(f.pending.id))[0]).toMatchObject({ status: "coalesced",
runId: null, payload: { coalescedIntoWakeupRequestId: request.requestId } });
} finally { await f.service.shutdown(); }
}, 90_000);
it.each(["unrecovered_workspace", "access_revoked", "paused_agent"] as const)(
it.each(["unrecovered_workspace", "access_revoked", "paused_agent", "input_after_stop", "input_inside_user_stop"] as const)(
"settles a refused GitHub task Retry by its own gates: %s", async gate => {
const f = await currentGitHubTaskRetryFixture(false);
const f = await currentGitHubTaskRetryFixture(gate.startsWith("input_"));
try {
const companyId = f.fixture.companyId, issueId = f.issue.id;
await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: issueId, kind: "active_run_watchdog",
@@ -66038,6 +66061,12 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: f.runId,
status: gate === "unrecovered_workspace" ? "recovering" : "recovered",
stage: gate === "unrecovered_workspace" ? "restoring_files" : "saving_receipts", attempt: 1, requestIds: [] } } });
if (gate === "input_inside_user_stop") {
const [failed] = await readHeartbeatRunsById(f.runId);
await db.update(heartbeatRuns).set({ finishedAt: new Date(), resultJson: { cancellation: { source: "operator",
expected: true, initiator: { type: "user", id: "owner-user" }, reason: "Stopped by the user",
recordedAt: failed.finishedAt!.toISOString() } } }).where(eq(heartbeatRuns.id, f.runId));
}
const request = await requestTaskRetry({ db, companyId, issueId, actorId: "owner-user", actorSource: "cloud_tenant",
observedRunId: f.runId, idempotencyKey: randomUUID(), prepareChatRetry: f.service.prepareFailedChatRunRetry });
const [chat] = await db.update(chatActions).set({ status: "failed",
@@ -66054,6 +66083,8 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
if (gate !== "paused_agent") {
expect((await readAgentWakeupRequestsById(request.requestId!))[0]).toMatchObject({ status: "failed", runId: null });
expect(await retryRuns()).toHaveLength(0);
if (f.pending) expect((await readAgentWakeupRequestsById(f.pending.id))[0])
.toMatchObject({ status: "deferred_issue_execution", runId: null });
return;
}
expect((await readAgentWakeupRequestsById(request.requestId!))[0]).toMatchObject({ status: "deferred_issue_execution", runId: null });
+36 -9
View File
@@ -3,6 +3,7 @@ import { notifyExternalConversationSettlement } from "./external-conversation-se
import { writeExecutionControl } from "./execution-control-notifications.js";
import { DELIVERY_QUEUES } from "./delivery-work-notifications.js";
import { z } from "zod";
import { userStopBoundaryAt } from "./run-cancellation.js";
import { readWorkspaceRecovery } from "./workspace-recovery-receipt.js";
import { randomUUID } from "node:crypto";
import { taskRetryRunAncestry } from "./recovery-run-ancestry.js";
@@ -520,11 +521,12 @@ async function continueRefusedTaskChatRetry(db: Db, wake: typeof agentWakeupRequ
const [issue] = await tx.select().from(issues).where(and(eq(issues.id, issueId),
eq(issues.companyId, wake.companyId))).for("update");
if (!issue || issue.assigneeAgentId !== wake.agentId || ["done", "cancelled"].includes(issue.status)) return false;
const [latest] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
const [latest] = await tx.select({ id: heartbeatRuns.id, finishedAt: heartbeatRuns.finishedAt, status: heartbeatRuns.status,
startedAt: heartbeatRuns.startedAt, errorCode: heartbeatRuns.errorCode, resultJson: heartbeatRuns.resultJson }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, wake.companyId), or(eq(heartbeatRuns.issueId, issueId), eq(heartbeatRuns.nativeIssueId, issueId),
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`)))
.orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
if (latest?.id !== intent.sourceRunId) return false;
if (latest?.id !== intent.sourceRunId || !latest.finishedAt) return false;
if (intent.initiatedBy) {
const [joined] = await tx.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
eq(agentWakeupRequests.companyId, wake.companyId), eq(agentWakeupRequests.agentId, wake.agentId),
@@ -535,19 +537,38 @@ async function continueRefusedTaskChatRetry(db: Db, wake: typeof agentWakeupRequ
sql`${agentWakeupRequests.payload}#>'{taskRetry,initiatedBy}' is null`)).limit(1);
if (!joined) return false;
}
const heldBy = latest.status === "cancelled" && !latest.startedAt && latest.errorCode === "execution_reconciliation_required"
? z.string().uuid().safeParse((latest.resultJson?.executionWait as Record<string, unknown> | undefined)?.recoveryActionId) : null;
const [blockingAction] = heldBy?.success ? await tx.select({ evidence: issueRecoveryActions.evidence })
.from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, wake.companyId),
eq(issueRecoveryActions.sourceIssueId, issueId), eq(issueRecoveryActions.id, heldBy.data))) : [];
const recoveredRunIds = [intent.sourceRunId, ...(typeof blockingAction?.evidence.runId === "string"
? [blockingAction.evidence.runId] : [])];
const recoveries = await tx.select({ evidence: issueRecoveryActions.evidence }).from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, wake.companyId), eq(issueRecoveryActions.sourceIssueId, issueId),
sql`${issueRecoveryActions.evidence}->>'runId' = ${intent.sourceRunId}`));
if (!recoveries.some(({ evidence }) => readSystemRecovery(evidence)?.state === "complete" &&
readWorkspaceRecovery(evidence)?.runId === intent.sourceRunId &&
readWorkspaceRecovery(evidence)?.status === "recovered")) return false;
inArray(sql<string>`${issueRecoveryActions.evidence}->>'runId'`, recoveredRunIds)));
if (!recoveries.some(({ evidence }) => {
const workspace = readWorkspaceRecovery(evidence);
return readSystemRecovery(evidence)?.state === "complete" && workspace?.status === "recovered" &&
workspace.runId === evidence.runId && recoveredRunIds.includes(workspace.runId);
})) return false;
if (!await taskRetryMutationAllowed(tx as unknown as Db, { companyId: wake.companyId, issueId,
actorId: intent.actorId, actorSource: intent.actorSource })) return false;
const [chatReceipt] = await tx.select({ status: agentWakeupRequests.status, runId: agentWakeupRequests.runId })
.from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, wake.companyId), eq(agentWakeupRequests.id, chat.id)));
if (chatReceipt && (chatReceipt.runId || !["failed", "skipped", "cancelled"].includes(chatReceipt.status))) return false;
if (await taskHasUnadmittedChatInput(tx as unknown as Db, { companyId: wake.companyId, issueId,
agentId: wake.agentId, failedRunId: intent.sourceRunId })) return false;
const [unadmitted] = await tx.select({ id: issues.id }).from(issues).where(and(eq(issues.id, issueId),
eq(issues.companyId, wake.companyId), unadmittedChatWakeupCondition(issues.id, issues.companyId))).limit(1);
if (unadmitted) return false;
const savedInputIds = (await selectTaskRetryInput(tx, { companyId: wake.companyId, issueId, agentId: wake.agentId,
failedRunId: intent.sourceRunId }))?.sourceActionIds ?? [];
const stopBoundary = userStopBoundaryAt(latest);
const savedInputs = savedInputIds.length ? await tx.select({ id: chatActions.id, authorizedAt: issueComments.createdAt })
.from(chatActions).innerJoin(issueComments, and(eq(issueComments.companyId, chatActions.companyId),
eq(issueComments.issueId, issueId), sql`${issueComments.id}::text = ${chatActions.payload}->>'commentId'`))
.where(and(eq(chatActions.companyId, wake.companyId), inArray(chatActions.id, savedInputIds))) : [];
if (!stopBoundary || savedInputs.length !== savedInputIds.length ||
savedInputs.some(input => input.authorizedAt > stopBoundary)) return false;
const { chatActionId: _replaced, initiatedBy: _automatic, ...continuation } = intent;
await notifyDeferredExecutionWork(tx, { companyId: wake.companyId, requestId: wake.id });
const [converted] = await tx.update(agentWakeupRequests).set({ updatedAt: new Date(),
@@ -558,9 +579,15 @@ async function continueRefusedTaskChatRetry(db: Db, wake: typeof agentWakeupRequ
sql`${agentWakeupRequests.payload}#>>'{taskRetry,chatActionId}' = ${chat.id}`))
.returning({ id: agentWakeupRequests.id });
if (!converted) return false;
if (savedInputIds.length) await tx.update(agentWakeupRequests).set({ status: "coalesced", finishedAt: new Date(),
updatedAt: new Date(), payload: sql`coalesce(${agentWakeupRequests.payload}, '{}'::jsonb) || ${JSON.stringify({
coalescedIntoWakeupRequestId: wake.id })}::jsonb` }).where(and(eq(agentWakeupRequests.companyId, wake.companyId),
inArray(agentWakeupRequests.id, savedInputIds), eq(agentWakeupRequests.status, "deferred_issue_execution"),
isNull(agentWakeupRequests.runId)));
await persistActivity(tx as unknown as Db, { companyId: wake.companyId, actorType: "system", actorId: "task-retry",
action: "issue.retry_continued_from_saved_work", entityType: "issue", entityId: issueId, runId: intent.sourceRunId,
details: { requestId: wake.id, sourceRunId: intent.sourceRunId, refusedChatActionId: chat.id } });
details: { requestId: wake.id, sourceRunId: intent.sourceRunId, refusedChatActionId: chat.id,
...(savedInputIds.length ? { heldInputRequestIds: savedInputIds } : {}) } });
return true;
});
}