mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
fix(retry): continue an accepted Retry past chat input held because it predates the stop (#21129)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person's Retry on a connected task (for example a GitHub pull request review) must start work. #21080 made a Retry whose chat replay is refused after the click continue from the recovered saved work > - After #21080 was deployed, accepted Retries on a busy GitHub review agent still did not start > - Those tasks had a saved chat request that arrived while the failed run was still running. Admission holds such a request as `message_predates_stop`, so it can never start alone > - #21080 refused to continue while any saved chat input existed, so this held input blocked the continuation. For a second group, the latest run was an admission that the recovery hold cancelled before dispatch, and the recovered saved work belongs to the earlier failed run > - This pull request lets the continuation adopt input that was held because it predates the stop, and lets the recovered run that held a cancelled admission count as the saved work > - The benefit is that an accepted Retry starts work on these tasks too ## Linked Issues or Issue Description Follow-up to #21080. No public issue exists. This is the bug description, in the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** After #21080 was deployed, `POST /api/issues/:id/retry` returned 202 for blocked tasks of a GitHub review agent (`paperclip_runner`), but no run started. Each Retry receipt changed to `failed` about 5 seconds later (on the 14:48 Retries, 5 seconds for one task and about 75 seconds for two others), with no run and no `issue.retry_continued_from_saved_work` activity. Every affected task had a saved chat request (a newer GitHub review request) with an `issue.execution_waiting` record of reason `message_predates_stop`. The request arrived while the failed run was running, so admission holds it permanently. The interruption projection kept pointing at that held request. In a second group of tasks, the latest run was cancelled before dispatch with `execution_reconciliation_required`, and the recovered saved work belonged to the earlier failed run that held it. Root cause: in `continueRefusedTaskChatRetry` (`server/src/services/task-retry.ts`), any saved chat input from `selectTaskRetryInput` stopped the continuation, so the receipt settled `failed`. The held input can never start by itself, so nothing ran. The recovered-work check also looked only at the latest run, so it missed the recovered run that held a cancelled admission. **Expected behavior** An accepted Retry starts work. Saved input that admission holds because it predates the stop joins the continuation, and the task continues from the recovered saved work. **Steps to reproduce** 1. Bind a task to a GitHub chat conversation. Let a native run fail with `native_continuation_requires_reconciliation`, and recover its saved work. 2. Save a chat request while that run is still running, so admission holds it with `message_predates_stop`. 3. Click Retry while the chat identity lock is busy, so the source check is deferred and then refused. 4. The route returns 202. The Retry receipt changes to `failed`, and no run starts. **Paperclip version or commit**a65644d1e1(the deployed #21080). Reproduced onef2b53b675. **Deployment mode** Hosted (cloud) instance with Daytona remote sandboxes and a GitHub chat connection. **Agent adapter(s) involved** `paperclip_runner` (native runs, and legacy admissions cancelled before dispatch). The fix is in core Retry code and is not adapter-specific. ## What Changed - `server/src/services/task-retry.ts` (`continueRefusedTaskChatRetry`): - Saved chat input whose source comment was written no later than the failed run's stop boundary (`userStopBoundaryAt`: the recorded user Stop request, otherwise `finishedAt`) no longer stops the continuation. This is the same boundary and timestamp that admission uses. Admission holds that input as `message_predates_stop`. The continuation coalesces its receipts into the Retry receipt (`coalescedIntoWakeupRequestId`), so the task stops waiting on it. The receipt ids are recorded in the activity details. - Saved input created after the stop, and unadmitted chat input, still settle the Retry as failed, as before. - When the latest run is an admission cancelled before dispatch (`execution_reconciliation_required`, never started), the recovered-work check also accepts the run named by that run's `executionWait.recoveryActionId`. - `doc/execution-semantics.md`: both rules. - Tests in `server/src/__tests__/chat-channels.integration.test.ts`: two new variants of the #21080 regression test. `person_with_input_before_stop` saves a chat request before the failed run stops. `person_after_held_admission` adds a legacy admission that the recovery hold cancelled. Both start one run that resumes the recovered workspace (`resume_preserved_workspace`). The held-admission fixture binds the task to the recovered workspace, as on the hosted tasks. Refusal edges `input_after_stop` and `input_inside_user_stop` settle the Retry failed with no run and leave the input deferred. ## Verification - Red before the fix: both new variants fail. The Retry receipt is `failed` (or stays held), and no run starts. - Green after the fix: both variants pass, and the held input receipt is coalesced into the Retry receipt. - `npx vitest run src/__tests__/chat-channels.integration.test.ts -t "[Rr]etry"` with `src/services/task-retry.test.ts` passes 369/369 on the final commit. - `npx tsc --noEmit -p server/tsconfig.json` passes. - Staging was read with GET requests only. ## Risks - Behavior change: input that arrived during a failed run no longer starts its own turn through the chat retry. It is adopted into the saved-work continuation, and the comment stays in the task thread. Before this change, that input could never start at all. - Input that arrived after the stop keeps its own admission path. - No migration. Receipts that already failed stay failed. The next click takes the new path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
d5bd38468d
commit
f2e449f8f6
3 files changed
+73
-15
No files matched your search
@@ -65892,11 +65892,19 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["person", "person_paused_after_accept", "restart_recovery", "restart_recovery_joined_by_person"] as const)("continues an accepted GitHub task Retry from recovered saved work when its chat replay is refused after the click: %s", async initiator => {
|
||||
const f = await currentGitHubTaskRetryFixture(false);
|
||||
it.each(["person", "person_with_input_before_stop", "person_after_held_admission", "person_paused_after_accept", "restart_recovery", "restart_recovery_joined_by_person"] as const)("continues an accepted GitHub task Retry from recovered saved work when its chat replay is refused after the click: %s", async initiator => {
|
||||
const f = await currentGitHubTaskRetryFixture(initiator === "person_with_input_before_stop");
|
||||
try {
|
||||
const companyId = f.fixture.companyId, issueId = f.issue.id;
|
||||
const leaseId = randomUUID(), providerLeaseId = randomUUID(), workspaceId = randomUUID(), snapshotId = randomUUID();
|
||||
if (f.pending) {
|
||||
await db.update(agentWakeupRequests).set({ requestedAt: new Date(Date.now() - 60_000) })
|
||||
.where(eq(agentWakeupRequests.id, f.pending.id));
|
||||
await db.update(chatActions).set({ createdAt: new Date(Date.now() - 60_000) }).where(eq(chatActions.id, f.pending.id));
|
||||
await db.update(issueComments).set({ createdAt: new Date(Date.now() - 60_000) })
|
||||
.where(eq(issueComments.id, String(f.pending.payload.commentId)));
|
||||
await db.update(heartbeatRuns).set({ finishedAt: new Date(Date.now() - 30_000) }).where(eq(heartbeatRuns.id, f.runId));
|
||||
}
|
||||
await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, f.runId));
|
||||
await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, sourceIssueId: issueId,
|
||||
name: "Review workspace", providerType: "cloud_sandbox", mode: "isolated_workspace", strategyType: "project_primary",
|
||||
@@ -65932,6 +65940,19 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
executionWorkspaceId: workspaceId, phase: "workspace_recovery", status: "succeeded", finishedAt: new Date(),
|
||||
metadata: { recoveryActionId, leaseId, providerLeaseId, snapshotId, generation: 1, sha256: "c".repeat(64),
|
||||
storage: "remote_workspace", epoch: 2 } });
|
||||
let sourceRunId: string = f.runId;
|
||||
if (initiator === "person_after_held_admission") {
|
||||
const [failed] = await readHeartbeatRunsById(f.runId);
|
||||
const [held] = await db.insert(heartbeatRuns).values({ companyId, agentId: f.fixture.assignedAgentId,
|
||||
status: "cancelled", errorCode: "execution_reconciliation_required", runtimeMode: "legacy", issueId,
|
||||
wakeupRequestId: failed.wakeupRequestId, contextSnapshot: failed.contextSnapshot, finishedAt: new Date(),
|
||||
createdAt: new Date(failed.createdAt.getTime() + 1_000),
|
||||
resultJson: { stopReason: "execution_reconciliation_required", timeoutSource: "stale_queued_run_gate",
|
||||
executionWait: { issueId, recoveryActionId } } }).returning();
|
||||
await db.update(agentWakeupRequests).set({ runId: held.id }).where(eq(agentWakeupRequests.id, failed.wakeupRequestId!));
|
||||
await db.update(issues).set({ executionWorkspaceId: workspaceId }).where(eq(issues.id, issueId));
|
||||
sourceRunId = held.id;
|
||||
}
|
||||
const { getExecutionBlocker } = await import("../services/execution-blocker.js");
|
||||
const blocker = await getExecutionBlocker(db, companyId, issueId);
|
||||
const interruption = await taskInterruptionPresentation(db, f.issue);
|
||||
@@ -65965,7 +65986,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
let request: Awaited<ReturnType<typeof requestTaskRetry>>;
|
||||
try {
|
||||
request = await requestTaskRetry({ db, companyId, issueId, actorId: "owner-user", actorSource: "cloud_tenant",
|
||||
observedRunId: f.runId, idempotencyKey: randomUUID(), prepareChatRetry: f.service.prepareFailedChatRunRetry,
|
||||
observedRunId: sourceRunId, idempotencyKey: randomUUID(), prepareChatRetry: f.service.prepareFailedChatRunRetry,
|
||||
...(initiator.startsWith("person") ? {} : { initiatedBy: "restart_recovery" as const }) });
|
||||
} finally { await releaseIdentity(); }
|
||||
if (initiator === "restart_recovery_joined_by_person") {
|
||||
@@ -66020,12 +66041,14 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
eq(activityLog.action, "issue.retry_continued_from_saved_work")));
|
||||
expect(activity).toHaveLength(1);
|
||||
expect(activity[0].details).toMatchObject({ requestId: request.requestId, refusedChatActionId: request.chatActionId });
|
||||
if (f.pending) expect((await readAgentWakeupRequestsById(f.pending.id))[0]).toMatchObject({ status: "coalesced",
|
||||
runId: null, payload: { coalescedIntoWakeupRequestId: request.requestId } });
|
||||
} finally { await f.service.shutdown(); }
|
||||
}, 90_000);
|
||||
|
||||
it.each(["unrecovered_workspace", "access_revoked", "paused_agent"] as const)(
|
||||
it.each(["unrecovered_workspace", "access_revoked", "paused_agent", "input_after_stop", "input_inside_user_stop"] as const)(
|
||||
"settles a refused GitHub task Retry by its own gates: %s", async gate => {
|
||||
const f = await currentGitHubTaskRetryFixture(false);
|
||||
const f = await currentGitHubTaskRetryFixture(gate.startsWith("input_"));
|
||||
try {
|
||||
const companyId = f.fixture.companyId, issueId = f.issue.id;
|
||||
await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: issueId, kind: "active_run_watchdog",
|
||||
@@ -66038,6 +66061,12 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: f.runId,
|
||||
status: gate === "unrecovered_workspace" ? "recovering" : "recovered",
|
||||
stage: gate === "unrecovered_workspace" ? "restoring_files" : "saving_receipts", attempt: 1, requestIds: [] } } });
|
||||
if (gate === "input_inside_user_stop") {
|
||||
const [failed] = await readHeartbeatRunsById(f.runId);
|
||||
await db.update(heartbeatRuns).set({ finishedAt: new Date(), resultJson: { cancellation: { source: "operator",
|
||||
expected: true, initiator: { type: "user", id: "owner-user" }, reason: "Stopped by the user",
|
||||
recordedAt: failed.finishedAt!.toISOString() } } }).where(eq(heartbeatRuns.id, f.runId));
|
||||
}
|
||||
const request = await requestTaskRetry({ db, companyId, issueId, actorId: "owner-user", actorSource: "cloud_tenant",
|
||||
observedRunId: f.runId, idempotencyKey: randomUUID(), prepareChatRetry: f.service.prepareFailedChatRunRetry });
|
||||
const [chat] = await db.update(chatActions).set({ status: "failed",
|
||||
@@ -66054,6 +66083,8 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
if (gate !== "paused_agent") {
|
||||
expect((await readAgentWakeupRequestsById(request.requestId!))[0]).toMatchObject({ status: "failed", runId: null });
|
||||
expect(await retryRuns()).toHaveLength(0);
|
||||
if (f.pending) expect((await readAgentWakeupRequestsById(f.pending.id))[0])
|
||||
.toMatchObject({ status: "deferred_issue_execution", runId: null });
|
||||
return;
|
||||
}
|
||||
expect((await readAgentWakeupRequestsById(request.requestId!))[0]).toMatchObject({ status: "deferred_issue_execution", runId: null });
|
||||
|
||||
@@ -3,6 +3,7 @@ import { notifyExternalConversationSettlement } from "./external-conversation-se
|
||||
import { writeExecutionControl } from "./execution-control-notifications.js";
|
||||
import { DELIVERY_QUEUES } from "./delivery-work-notifications.js";
|
||||
import { z } from "zod";
|
||||
import { userStopBoundaryAt } from "./run-cancellation.js";
|
||||
import { readWorkspaceRecovery } from "./workspace-recovery-receipt.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { taskRetryRunAncestry } from "./recovery-run-ancestry.js";
|
||||
@@ -520,11 +521,12 @@ async function continueRefusedTaskChatRetry(db: Db, wake: typeof agentWakeupRequ
|
||||
const [issue] = await tx.select().from(issues).where(and(eq(issues.id, issueId),
|
||||
eq(issues.companyId, wake.companyId))).for("update");
|
||||
if (!issue || issue.assigneeAgentId !== wake.agentId || ["done", "cancelled"].includes(issue.status)) return false;
|
||||
const [latest] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
||||
const [latest] = await tx.select({ id: heartbeatRuns.id, finishedAt: heartbeatRuns.finishedAt, status: heartbeatRuns.status,
|
||||
startedAt: heartbeatRuns.startedAt, errorCode: heartbeatRuns.errorCode, resultJson: heartbeatRuns.resultJson }).from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.companyId, wake.companyId), or(eq(heartbeatRuns.issueId, issueId), eq(heartbeatRuns.nativeIssueId, issueId),
|
||||
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`)))
|
||||
.orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
|
||||
if (latest?.id !== intent.sourceRunId) return false;
|
||||
if (latest?.id !== intent.sourceRunId || !latest.finishedAt) return false;
|
||||
if (intent.initiatedBy) {
|
||||
const [joined] = await tx.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
|
||||
eq(agentWakeupRequests.companyId, wake.companyId), eq(agentWakeupRequests.agentId, wake.agentId),
|
||||
@@ -535,19 +537,38 @@ async function continueRefusedTaskChatRetry(db: Db, wake: typeof agentWakeupRequ
|
||||
sql`${agentWakeupRequests.payload}#>'{taskRetry,initiatedBy}' is null`)).limit(1);
|
||||
if (!joined) return false;
|
||||
}
|
||||
const heldBy = latest.status === "cancelled" && !latest.startedAt && latest.errorCode === "execution_reconciliation_required"
|
||||
? z.string().uuid().safeParse((latest.resultJson?.executionWait as Record<string, unknown> | undefined)?.recoveryActionId) : null;
|
||||
const [blockingAction] = heldBy?.success ? await tx.select({ evidence: issueRecoveryActions.evidence })
|
||||
.from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, wake.companyId),
|
||||
eq(issueRecoveryActions.sourceIssueId, issueId), eq(issueRecoveryActions.id, heldBy.data))) : [];
|
||||
const recoveredRunIds = [intent.sourceRunId, ...(typeof blockingAction?.evidence.runId === "string"
|
||||
? [blockingAction.evidence.runId] : [])];
|
||||
const recoveries = await tx.select({ evidence: issueRecoveryActions.evidence }).from(issueRecoveryActions).where(and(
|
||||
eq(issueRecoveryActions.companyId, wake.companyId), eq(issueRecoveryActions.sourceIssueId, issueId),
|
||||
sql`${issueRecoveryActions.evidence}->>'runId' = ${intent.sourceRunId}`));
|
||||
if (!recoveries.some(({ evidence }) => readSystemRecovery(evidence)?.state === "complete" &&
|
||||
readWorkspaceRecovery(evidence)?.runId === intent.sourceRunId &&
|
||||
readWorkspaceRecovery(evidence)?.status === "recovered")) return false;
|
||||
inArray(sql<string>`${issueRecoveryActions.evidence}->>'runId'`, recoveredRunIds)));
|
||||
if (!recoveries.some(({ evidence }) => {
|
||||
const workspace = readWorkspaceRecovery(evidence);
|
||||
return readSystemRecovery(evidence)?.state === "complete" && workspace?.status === "recovered" &&
|
||||
workspace.runId === evidence.runId && recoveredRunIds.includes(workspace.runId);
|
||||
})) return false;
|
||||
if (!await taskRetryMutationAllowed(tx as unknown as Db, { companyId: wake.companyId, issueId,
|
||||
actorId: intent.actorId, actorSource: intent.actorSource })) return false;
|
||||
const [chatReceipt] = await tx.select({ status: agentWakeupRequests.status, runId: agentWakeupRequests.runId })
|
||||
.from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, wake.companyId), eq(agentWakeupRequests.id, chat.id)));
|
||||
if (chatReceipt && (chatReceipt.runId || !["failed", "skipped", "cancelled"].includes(chatReceipt.status))) return false;
|
||||
if (await taskHasUnadmittedChatInput(tx as unknown as Db, { companyId: wake.companyId, issueId,
|
||||
agentId: wake.agentId, failedRunId: intent.sourceRunId })) return false;
|
||||
const [unadmitted] = await tx.select({ id: issues.id }).from(issues).where(and(eq(issues.id, issueId),
|
||||
eq(issues.companyId, wake.companyId), unadmittedChatWakeupCondition(issues.id, issues.companyId))).limit(1);
|
||||
if (unadmitted) return false;
|
||||
const savedInputIds = (await selectTaskRetryInput(tx, { companyId: wake.companyId, issueId, agentId: wake.agentId,
|
||||
failedRunId: intent.sourceRunId }))?.sourceActionIds ?? [];
|
||||
const stopBoundary = userStopBoundaryAt(latest);
|
||||
const savedInputs = savedInputIds.length ? await tx.select({ id: chatActions.id, authorizedAt: issueComments.createdAt })
|
||||
.from(chatActions).innerJoin(issueComments, and(eq(issueComments.companyId, chatActions.companyId),
|
||||
eq(issueComments.issueId, issueId), sql`${issueComments.id}::text = ${chatActions.payload}->>'commentId'`))
|
||||
.where(and(eq(chatActions.companyId, wake.companyId), inArray(chatActions.id, savedInputIds))) : [];
|
||||
if (!stopBoundary || savedInputs.length !== savedInputIds.length ||
|
||||
savedInputs.some(input => input.authorizedAt > stopBoundary)) return false;
|
||||
const { chatActionId: _replaced, initiatedBy: _automatic, ...continuation } = intent;
|
||||
await notifyDeferredExecutionWork(tx, { companyId: wake.companyId, requestId: wake.id });
|
||||
const [converted] = await tx.update(agentWakeupRequests).set({ updatedAt: new Date(),
|
||||
@@ -558,9 +579,15 @@ async function continueRefusedTaskChatRetry(db: Db, wake: typeof agentWakeupRequ
|
||||
sql`${agentWakeupRequests.payload}#>>'{taskRetry,chatActionId}' = ${chat.id}`))
|
||||
.returning({ id: agentWakeupRequests.id });
|
||||
if (!converted) return false;
|
||||
if (savedInputIds.length) await tx.update(agentWakeupRequests).set({ status: "coalesced", finishedAt: new Date(),
|
||||
updatedAt: new Date(), payload: sql`coalesce(${agentWakeupRequests.payload}, '{}'::jsonb) || ${JSON.stringify({
|
||||
coalescedIntoWakeupRequestId: wake.id })}::jsonb` }).where(and(eq(agentWakeupRequests.companyId, wake.companyId),
|
||||
inArray(agentWakeupRequests.id, savedInputIds), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
||||
isNull(agentWakeupRequests.runId)));
|
||||
await persistActivity(tx as unknown as Db, { companyId: wake.companyId, actorType: "system", actorId: "task-retry",
|
||||
action: "issue.retry_continued_from_saved_work", entityType: "issue", entityId: issueId, runId: intent.sourceRunId,
|
||||
details: { requestId: wake.id, sourceRunId: intent.sourceRunId, refusedChatActionId: chat.id } });
|
||||
details: { requestId: wake.id, sourceRunId: intent.sourceRunId, refusedChatActionId: chat.id,
|
||||
...(savedInputIds.length ? { heldInputRequestIds: savedInputIds } : {}) } });
|
||||
return true;
|
||||
});
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user