fix(retry): continue a connected task from saved work when its exact chat request cannot be repeated (#21038)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Tasks that come from a connected chat (for example GitHub pull
request reviews) show a Retry button when a run fails
> - The task projection offers Retry, but the Retry route sends a
connected task to the chat retry owner, which only replays the exact
failed chat request
> - When that exact request cannot be replayed (the attempt already
saved a result, its output was published, the latest run did not start
from a chat message, or the requester's access changed), the route
returns 409 and tells the person to check the connection by hand
> - A Retry button that is shown must work, and recovery must not ask
the person to inspect anything
> - This pull request makes a person's Retry continue the task from its
saved work when the exact chat replay is refused, the same way a new
message on the task would
> - The benefit is that the projection and the route agree, and blocked
connected tasks become retryable after deploy with no migration

## Linked Issues or Issue Description

No public issue exists. This is the bug description, in the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

On a hosted deploy, five blocked tasks that came from a connected GitHub
conversation showed Retry (`interruption.canRetry: true`). Each click on
Retry returned 409 `chat_failed_run_retry_not_authorized`. The message
was "The exact failed chat request can no longer be retried safely.
Check its current connection, access, and recovery state." or "This
attempt already recorded a result. Send a new message on the task to
continue from the saved work." The latest runs were legacy sandbox setup
failures, native runs that needed reconciliation, a run that started
from an issue monitor instead of a chat message, and a native run that
already saved its result. Root cause: `admitTaskRetry` in
`server/src/services/task-retry.ts` called the chat retry owner for
every connected task and passed its exact-replay refusal straight to the
person. The projection cannot predict that refusal.

**Expected behavior**

A shown Retry works. If the exact chat request cannot be repeated, Retry
continues the task from its saved work, the same way a new message on
the task would. Users do not get operator instructions.

**Steps to reproduce**

1. Bind a task to a restricted chat conversation.
2. Make its latest run fail so that the chat retry owner refuses the
exact replay. For example, use a run whose source is not a chat message,
or a native run with an accepted result.
3. Click Retry (`POST /api/issues/:id/retry`).
4. The route returns 409 `chat_failed_run_retry_not_authorized`.

**Paperclip version or commit**

`master` at 5a7378af5 (the deployed revision), reproduced on 51d77648bc.

**Deployment mode**

Hosted (cloud) instance with Daytona remote sandboxes and a GitHub chat
connection.

**Agent adapter(s) involved**

Not adapter-specific (core bug). Seen with `paperclip_runner` legacy and
native runs.

## What Changed

- `server/src/services/task-retry.ts`: a person's Retry runs the exact
chat replay inside a savepoint. On a
`chat_failed_run_retry_not_authorized` refusal, the savepoint rolls back
and the same Retry receipt stays an ordinary board continuation. The
clicking person is the responsible user, the run starts a fresh session
(`forceFreshSession`) from saved work, and no chat action, chat
authority, or output replay is created. Retry that Paperclip requests by
itself (`initiatedBy: restart_recovery | provider_limit_reset`) keeps
the exact-replay requirement. Other errors (403, lock contention,
database errors) still propagate.
- `server/src/services/chat-channels/failed-run-retry.ts`: plain refusal
wording. Runs saved by an earlier release with the old diagnostic are
still recognized by the pre-provider check.
- Tests: new red/green cases in `server/src/services/task-retry.test.ts`
(four refusal reasons on a chat task whose latest run came from an issue
monitor, plus automatic Retry still refused). Updated two real-service
route cases in `server/src/__tests__/chat-channels.integration.test.ts`
(accepted native result on an inherited GitHub task, and an unknown chat
source) from expecting 409 to expecting a board continuation without a
chat replay.
- A person's Retry on that person's own saved board turn (`source:
issue.comment.reopen`) that cannot be certified for replay also
continues from saved work, without replaying the saved message.
Automatic Retry still gets 409 `saved_board_message_not_replayable`.
- A person's Retry when no chat retry owner is wired also continues from
saved work. Automatic Retry still gets 409.
- The fallback is not used while the conversation holds saved or
unadmitted chat input for the task. That input keeps its own admission
path and the existing refusal.
- Docs: `doc/execution-semantics.md` and `doc/connections/GITHUB.md`.

### 409 paths in the task Retry route after this change

| Path | Who sees it | Result |
| --- | --- | --- |
| `chat_failed_run_retry_not_authorized` from the chat retry owner |
Person | Continues from saved work (board continuation) |
| `saved_board_message_not_replayable` | Person | Continues from saved
work |
| No chat retry owner wired | Person | Continues from saved work |
| Any of the three above while saved or unadmitted chat input is pending
| Person | 409. The pending input continues through its own admission
and delivery. |
| Any of the three above | Automatic Retry (restart recovery, provider
limit reset) | 409, recorded by the hook. No button. |
| `task_retry_busy` (chat authorization lock held) | Person | 409 with
`retryable: true` after 10 internal attempts. A new click works. |
| Idempotency key reused for a different task or user | Client error |
409. The UI makes a new key for each click. |

## Verification

- Red before the fix: `npx vitest run src/services/task-retry.test.ts -t
"connected task"` failed 4/4 with `Error: refused` (409). The board-turn
and missing-owner cases failed with "This saved message cannot be
retried safely..." and "Retry is unavailable for this connected
conversation." `src/__tests__/chat-channels.integration.test.ts -t
"continues an inherited GitHub task"` failed with `expected 202
"Accepted", got 409 "Conflict"`.
- Green after the fix: `src/services/task-retry.test.ts`,
`src/__tests__/durable-chat-wakeup.test.ts`,
`src/services/budget-resume-continuation.test.ts`: 160/160 passed.
- Full `src/__tests__/chat-channels.integration.test.ts`: 1949/1951
passed. The 2 local failures (`binds Slack tools...`, `materializes
direct external-chat finals...`) read `.claude/skills/slack/SKILL.md`
from the parent checkout of a nested local worktree (ENOENT). They do
not touch Retry and depend on the local checkout layout.
- `npx tsc --noEmit -p server/tsconfig.json`: passed.
- `pnpm verify` exited 2 (a doc file is a global CI input), so required
CI supplies broad coverage.
- Staging was inspected with GET requests only. No POST was sent to
staging.

## Risks

- Behavior change: a person's Retry on a connected task can now start a
board continuation where it was refused before. It uses the same
authority as a new message from that person, and dispatch keeps all
normal gates (task access, executor containment, recovery holds,
unadmitted chat input).
- The continuation does not publish the earlier result again and does
not reuse the chat requester's authority.
- No migration. Existing blocked tasks become retryable at the next
click.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-12 01:27:54 -06:00
1 parent b2cb8f8f56
commit 3c058c37e6
6 files changed
+214 -41

No files matched your search

+9 -4
View File
@@ -386,10 +386,15 @@ exact executor termination proof. Successful results and uncertain delivery do
not grant a replay. A certified failed-chat Retry retains the exact accepted
message identity when its predecessor stopped before identity initialization,
so a published review can expose its authorized passive response wait.
If an inherited task-resume attempt already recorded a result, ordinary Retry
explains that exact replay is unavailable and directs the user to send a new
message to continue from saved work. That new turn still passes normal current
requester, access and executor-containment checks.
When the exact failed GitHub request cannot be repeated (for example, the
attempt already recorded a result, its output was already published, the run
did not start from a GitHub message, or the requester's GitHub access changed),
a person's Retry continues the task from its saved work instead, the same way a
new message on the task would. That continuation runs as the person who clicked
Retry in a fresh session, never with the GitHub requester's authority, and never
repeats the original request or its output. It still passes the normal current
access and executor-containment checks. Automatic Retry on a person's behalf
does not take this path; it continues only with the exact GitHub request.
If that successor failed at Retry authorization before runtime or provider
startup, the next normal Retry follows its canonical source receipt. The exact
+4
View File
@@ -1611,6 +1611,10 @@ same preparation retains its directory and receipt check. A published workspace
on the same allocation retains its recorded directory. Automatic bootstrap
timeout retries still require a fresh allocation under the policy above.
### Connected task Retry without an exact chat replay
A person's task Retry on a connected task first asks the chat retry owner to repeat the exact failed chat request. When the latest run is that person's own saved board turn that cannot be certified for replay, when no chat retry owner is available, or when that owner refuses (`chat_failed_run_retry_not_authorized`), the refusal and any writes it made are rolled back and the same Retry receipt becomes an ordinary board continuation: the clicking person is the responsible user, the run starts a fresh session from saved work, and no chat action, chat authority, or output replay is created. This is the same authority as a new message from that person. When the conversation still holds saved or unadmitted chat input for the task, Retry keeps the exact-replay path and its refusal, because that input continues through its own admission. Dispatch keeps every normal gate (current task access, executor containment, recovery holds, unadmitted chat input). Projection and the route therefore agree: a shown Retry on a connected task is accepted. Retry that Paperclip requests by itself on a person's behalf (restart recovery, provider limit reset) keeps the exact-replay requirement and still settles on the refusal. A budget resume continues the person's own Retry, so it takes the board continuation too. No migration is needed; tasks blocked by an earlier release become retryable at the next click.
### Saved chat Retry preparation
Task Retry can meet a busy chat identity or endpoint lock before its source is authorized.
@@ -65471,9 +65471,14 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.retryOfRunId === f.runId)).toHaveLength(1);
} else {
expect([202, 409], JSON.stringify(response.body)).toContain(response.status);
if (response.status === 202) {
if (response.status === 202 && response.body.chatActionId) {
const outcome = await f.service.processFailedChatRunRetry(response.body.chatActionId);
expect(outcome.status).not.toBe("queued");
} else if (response.status === 202) {
const [intent] = await readAgentWakeupRequestsById(response.body.requestId);
expect(intent).toMatchObject({ reason: "task_retry", requestedByActorId: "owner-user",
payload: { issueId, taskRetry: { sourceRunId: f.runId } } });
expect((intent.payload!.taskRetry as Record<string, unknown>).chatActionId).toBeUndefined();
}
expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.retryOfRunId === f.runId)).toHaveLength(0);
}
@@ -66290,7 +66295,12 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
}
const response = await send();
if (["principal_revoked", "accepted_result"].includes(mode)) {
expect(response.status).toBe(409);
expect(response.status, JSON.stringify(response.body)).toBe(202);
expect(response.body.chatActionId).toBeUndefined();
const [intent] = await readAgentWakeupRequestsById(response.body.requestId);
expect(intent).toMatchObject({ reason: "task_retry", requestedByActorId: "owner-user",
payload: { issueId, taskRetry: { sourceRunId: f.runId } } });
expect((intent.payload!.taskRetry as Record<string, unknown>).chatActionId).toBeUndefined();
} else {
expect(response.status).toBe(202);
if (mode === "source_edited") await db.update(issueComments).set({ body: "Changed after Retry acceptance" }).where(eq(issueComments.id, String(f.action.payload.commentId)));
@@ -66378,11 +66388,26 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
const denySource = async (apply: () => Promise<unknown>, restore: () => Promise<unknown>, stale = false) => {
try {
await apply();
const denied = await request(app).post(`/api/issues/${issueId}/retry`).send({ idempotencyKey: randomUUID(), observedRunId: current!.id }).expect(stale ? 200 : 409);
if (stale) expect(denied.body).toMatchObject({ accepted: false, stale: true });
expect(await db.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.reason, "task_retry")))).toHaveLength(0);
} finally { await restore(); }
const denied = await request(app).post(`/api/issues/${issueId}/retry`).send({ idempotencyKey: randomUUID(), observedRunId: current!.id }).expect(stale ? 200 : 202);
const retries = await db.select().from(agentWakeupRequests).where(and(
eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.reason, "task_retry")));
if (stale) {
expect(denied.body).toMatchObject({ accepted: false, stale: true });
expect(retries).toHaveLength(0);
} else {
expect(denied.body.chatActionId).toBeUndefined();
expect(retries).toHaveLength(1);
expect(retries[0]).toMatchObject({ id: denied.body.requestId, requestedByActorId: "owner-user",
payload: { issueId, taskRetry: { sourceRunId: current!.id } } });
const taskRetry = retries[0].payload!.taskRetry as Record<string, unknown>;
expect(taskRetry.boardMessageOrigin).toBeUndefined();
expect(taskRetry.chatActionId).toBeUndefined();
}
} finally {
await db.delete(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, companyId),
eq(agentWakeupRequests.reason, "task_retry")));
await restore();
}
};
await denySource(() => db.update(issueComments).set({ body: "Edited after acceptance", updatedAt: new Date() }).where(eq(issueComments.id, comment.id)),
() => db.update(issueComments).set({ body: comment.body, updatedAt: sql`${issueComments.createdAt}` }).where(eq(issueComments.id, comment.id)));
@@ -66493,17 +66518,26 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
const [firstRetry] = await readHeartbeatRunsById(successor.id);
const nextRequest = () => request(app).post(`/api/issues/${issueId}/retry`).send({
idempotencyKey: randomUUID(), observedRunId: firstRetry.id });
const continueWithoutReplay = async () => {
const continued = await nextRequest().expect(202);
expect(continued.body.chatActionId).toBeUndefined();
const [intent] = await readAgentWakeupRequestsById(continued.body.requestId);
expect(intent).toMatchObject({ reason: "task_retry", requestedByActorId: "owner-user",
payload: { issueId, taskRetry: { sourceRunId: firstRetry.id } } });
expect((intent.payload!.taskRetry as Record<string, unknown>).boardMessageOrigin).toBeUndefined();
await db.delete(agentWakeupRequests).where(eq(agentWakeupRequests.id, intent.id));
};
await db.update(heartbeatRuns).set({ sessionIdAfter: "contradictory-provider-session" }).where(eq(heartbeatRuns.id, firstRetry.id));
await nextRequest().expect(409);
await continueWithoutReplay();
await db.update(heartbeatRuns).set({ sessionIdAfter: null }).where(eq(heartbeatRuns.id, firstRetry.id));
await db.update(issueComments).set({ body: "Changed original accepted message", updatedAt: new Date() }).where(eq(issueComments.id, comment.id));
await nextRequest().expect(409);
await continueWithoutReplay();
await db.update(issueComments).set({ body: comment.body, updatedAt: sql`${issueComments.createdAt}` }).where(eq(issueComments.id, comment.id));
const [firstIntent] = await readAgentWakeupRequestsById(retry.id);
await db.update(agentWakeupRequests).set({ payload: { ...firstIntent.payload,
taskRetry: { ...(firstIntent.payload!.taskRetry as object), sourceRunId: firstRetry.id } } }).where(eq(agentWakeupRequests.id, firstIntent.id));
await db.update(heartbeatRuns).set({ contextSnapshot: { ...firstRetry.contextSnapshot, previousRunId: firstRetry.id } }).where(eq(heartbeatRuns.id, firstRetry.id));
await nextRequest().expect(409);
await continueWithoutReplay();
await db.update(agentWakeupRequests).set({ payload: firstIntent.payload }).where(eq(agentWakeupRequests.id, firstIntent.id));
await db.update(heartbeatRuns).set({ contextSnapshot: firstRetry.contextSnapshot }).where(eq(heartbeatRuns.id, firstRetry.id));
const second = await nextRequest().expect(202);
@@ -66544,7 +66578,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
}
});
it("distinguishes an inherited GitHub requester from permission to replay a committed provider failure", async () => {
it("continues an inherited GitHub task from its recorded result instead of replaying the committed provider failure", async () => {
const f = await failedChatRetryFixture("github", "42", true, false, true);
try {
const companyId = f.fixture.companyId, agentId = f.fixture.assignedAgentId, issueId = f.issue.id;
@@ -66593,19 +66627,30 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
app.use((req, _res, next) => { req.actor = boardActor(companyId); next(); });
app.use("/api", issueRoutes(db, createStorageService().storage, { chatRunRetries: f.service }));
app.use(errorHandler);
const denied = await request(app).post(`/api/issues/${issueId}/retry`).send({
const continued = await request(app).post(`/api/issues/${issueId}/retry`).send({
idempotencyKey: input.idempotencyKey, observedRunId: input.observedRunId,
}).expect(409);
expect(denied.body).toMatchObject({ error: "This attempt already recorded a result. Send a new message on the task to continue from the saved work.",
details: { code: "chat_failed_run_retry_not_authorized", reason: "accepted_result_without_new_input" } });
// Isolate the later accepted-result guard using only owned fixture data.
// Resolving inherited provenance cannot by itself authorize replay.
}).expect(202);
expect(continued.body).toMatchObject({ accepted: true, stale: false, observedRunId: resumed!.id });
expect(continued.body.chatActionId).toBeUndefined();
const assertBoardContinuation = async (requestId: string) => {
const [intent] = await readAgentWakeupRequestsById(requestId);
expect(intent).toMatchObject({ reason: "task_retry", requestedByActorType: "user", requestedByActorId: "owner-user",
payload: { issueId, taskRetry: { sourceRunId: resumed!.id, actorId: "owner-user" } } });
expect((intent.payload!.taskRetry as Record<string, unknown>).chatActionId).toBeUndefined();
expect(await db.select().from(chatActions).where(and(eq(chatActions.companyId, companyId),
eq(chatActions.kind, "failed_run_retry")))).toHaveLength(0);
await db.update(agentWakeupRequests).set({ status: "failed", finishedAt: new Date() }).where(eq(agentWakeupRequests.id, requestId));
};
await assertBoardContinuation(continued.body.requestId);
// The later accepted-result guard also continues from the saved result
// instead of repeating the exact chat request.
await db.update(heartbeatRuns).set({ contextSnapshot: { ...failedResume.contextSnapshot,
source: "chat:github", wakeCommentId: f.action.payload.commentId, wakeCommentIds: [f.action.payload.commentId] } })
.where(eq(heartbeatRuns.id, failedResume.id));
try {
await expect(requestTaskRetry({ ...input, idempotencyKey: randomUUID() })).rejects.toMatchObject({ details: {
code: "chat_failed_run_retry_not_authorized", reason: "accepted_result_without_new_input" } });
const receipt = await requestTaskRetry({ ...input, idempotencyKey: randomUUID() });
expect(receipt).toMatchObject({ accepted: true, chatActionId: undefined });
await assertBoardContinuation(receipt.requestId!);
} finally { await db.update(heartbeatRuns).set({ contextSnapshot: failedResume.contextSnapshot }).where(eq(heartbeatRuns.id, failedResume.id)); }
expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.retryOfRunId === resumed!.id)).toHaveLength(0);
} finally { await f.service.shutdown(); }
@@ -67164,9 +67209,12 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
} finally { unsubscribes.forEach(unsubscribe => unsubscribe()); }
}
if (mode !== "committed_board_existing" && mode !== "committed_board_existing_coalesced" && mode !== "committed_board_existing_identity_contended" && mode !== "committed_board_existing_identity_expired") {
await expect(requestTaskRetry(nextInput)).rejects.toMatchObject({
details: { code: "chat_failed_run_retry_not_authorized" },
});
const continued = await requestTaskRetry(nextInput);
expect(continued).toMatchObject({ accepted: true, chatActionId: undefined });
const [intent] = await readAgentWakeupRequestsById(continued.requestId!);
expect(intent).toMatchObject({ reason: "task_retry", requestedByActorId: "owner-user",
payload: { taskRetry: { sourceRunId: failed.id } } });
expect((intent.payload!.taskRetry as Record<string, unknown>).chatActionId).toBeUndefined();
expect(await db.select().from(agentWakeupRequests).where(inArray(agentWakeupRequests.id, inputIds)).orderBy(asc(agentWakeupRequests.id)))
.toEqual(before);
return;
@@ -102,6 +102,9 @@ type DbOrTransaction = Db | Parameters<Parameters<Db["transaction"]>[0]>[0];
type DeliveryRow = typeof chatDeliveries.$inferSelect;
type RetryRuntimeFence = { credentialFingerprint: string; generation: number };
const FAILED_RUN_RETRY_IDENTITY_CONTENTION_MS = 60_000;
const FAILED_CHAT_RETRY_DIAGNOSTIC = "This chat request can't be repeated exactly as it was sent. Retry continues the task from its saved work instead.";
const FAILED_CHAT_RETRY_DIAGNOSTICS = [FAILED_CHAT_RETRY_DIAGNOSTIC,
"The exact failed chat request can no longer be retried safely. Check its current connection, access, and recovery state."];
export type ChatFailedRetrySource = Awaited<ReturnType<ReturnType<typeof createChatFailedRunRetry>["failedChatRetrySource"]>>;
type ChatFailedRunRetryOptions = {
@@ -139,8 +142,8 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti
"prior_result_publication" | "prior_interaction" | "source_authorization", authorityReason?: ChatWakeAuthorityReason) =>
conflict(
reason === "accepted_result_without_new_input"
? "This attempt already recorded a result. Send a new message on the task to continue from the saved work."
: "The exact failed chat request can no longer be retried safely. Check its current connection, access, and recovery state.",
? "This attempt already saved its result, so it is not repeated. Retry continues from that saved result."
: FAILED_CHAT_RETRY_DIAGNOSTIC,
{ code: "chat_failed_run_retry_not_authorized", ...(reason ? { reason } : {}), ...(reason === "source_authorization" && authorityReason ? { authorityReason } : {}) },
);
class FailedChatRetryPublicationReadError extends Error {
@@ -227,8 +230,9 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti
run: typeof heartbeatRuns.$inferSelect,
allowWorkspacePreparation = false,
): Promise<boolean> {
const retryDiagnostic = "The exact failed chat request can no longer be retried safely. Check its current connection, access, and recovery state.";
const failedRetry = run.errorCode === "chat_failed_run_retry_not_authorized" && run.error === retryDiagnostic;
const failedRetry = run.errorCode === "chat_failed_run_retry_not_authorized" &&
FAILED_CHAT_RETRY_DIAGNOSTICS.includes(run.error ?? "");
const retryDiagnostic = failedRetry ? run.error : null;
const validation = run.resultJson?.workspaceValidation as Record<string, unknown> | undefined;
const recovery = run.resultJson?.executionRecovery as Record<string, unknown> | undefined;
const workspaceValidation = run.errorCode === "workspace_validation_failed" &&
+90 -1
View File
@@ -1,7 +1,7 @@
import { randomUUID } from "node:crypto";
import { and, eq, sql } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { agents, agentWakeupRequests, chatConversations, chatEndpoints, companies, budgetPolicies, costEvents, createDb, type Db, environmentLeases, environments, executionWorkspaces, executionWorkspaceSnapshots, workspaceOperations, heartbeatRunEvents, heartbeatRuns, issueComments, issues, approvals, issueApprovals, issueRecoveryActions, nativeRunFinalizations, toolApplications, toolConnections, toolInvocations, applyPendingMigrations } from "@paperclipai/db";
import { agents, agentWakeupRequests, chatActions, chatConversations, chatEndpoints, companies, budgetPolicies, costEvents, createDb, type Db, environmentLeases, environments, executionWorkspaces, executionWorkspaceSnapshots, workspaceOperations, heartbeatRunEvents, heartbeatRuns, issueComments, issues, approvals, issueApprovals, issueRecoveryActions, nativeRunFinalizations, toolApplications, toolConnections, toolInvocations, applyPendingMigrations } from "@paperclipai/db";
import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
import { priorRecoveryApiReceipt, unresolvedRecoveryToolInvocation } from "./recovery-effect-receipts.js";
import { requestTaskRetry } from "./task-retry.js";
@@ -937,6 +937,95 @@ describe("durable task Retry admission", () => {
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, input.companyId))).toHaveLength(0);
});
const refusedExactChatReplay = (reason?: string) => async (tx: Db, scope: { companyId: string; issueId: string; failedRunId: string }) => {
const [endpoint] = await tx.select().from(chatEndpoints).where(eq(chatEndpoints.companyId, scope.companyId));
await tx.insert(chatActions).values({ companyId: scope.companyId, endpointId: endpoint.id, kind: "failed_run_retry",
status: "issued", providerActionId: randomUUID(), payload: { issueId: scope.issueId, failedRunId: scope.failedRunId } });
throw new HttpError(409, "refused", { code: "chat_failed_run_retry_not_authorized", ...(reason ? { reason } : {}) });
};
it.each([undefined, "accepted_result_without_new_input", "prior_result_publication", "source_authorization"])(
"continues a connected task from saved work when its exact chat request cannot be repeated: %s", async reason => {
const input = await seedChatTask();
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.observedRunId));
await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: input.issueId, source: "issue.monitor",
wakeReason: "issue_monitor_due" } }).where(eq(heartbeatRuns.id, source.id));
const receipt = await requestTaskRetry({ ...input, actorSource: "local_implicit", prepareChatRetry: refusedExactChatReplay(reason) });
expect(receipt).toMatchObject({ accepted: true, stale: false, observedRunId: source.id });
expect(receipt.chatActionId).toBeUndefined();
expect(await db.select().from(chatActions).where(eq(chatActions.companyId, input.companyId))).toHaveLength(0);
const [intent] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId!));
expect(intent).toMatchObject({ status: "deferred_issue_execution", reason: "task_retry", requestedByActorId: input.actorId,
payload: { issueId: input.issueId, taskRetry: { sourceRunId: source.id, actorId: input.actorId } } });
expect((intent.payload!.taskRetry as Record<string, unknown>).chatActionId).toBeUndefined();
await db.update(agents).set({ runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } } }).where(eq(agents.id, source.agentId));
await db.insert(heartbeatRuns).values({ companyId: input.companyId, agentId: source.agentId, status: "running" });
await heartbeatService(db).resumeTaskRetry(input.companyId, receipt.requestId!);
const queued = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId),
eq(heartbeatRuns.status, "queued")));
expect(queued).toHaveLength(1);
expect(queued[0].contextSnapshot).toMatchObject({ issueId: input.issueId, previousRunId: source.id,
taskRetryRequestId: receipt.requestId, forceFreshSession: true, responsibleUserId: input.actorId });
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId!)))[0])
.toMatchObject({ status: "coalesced", runId: queued[0].id });
});
it.each(["person", "automatic"] as const)(
"continues a person's saved board turn on a connected task from saved work without replaying it: %s", async initiator => {
const input = await seedChatTask();
const [source] = await db.update(heartbeatRuns).set({ runtimeMode: "legacy", errorCode: "adapter_failed",
error: "Provider stopped", responsibleUserId: input.actorId, sessionIdAfter: "provider-session",
contextSnapshot: { issueId: input.issueId, source: "issue.comment.reopen", wakeCommentId: randomUUID() } })
.where(eq(heartbeatRuns.id, input.observedRunId)).returning();
let calls = 0;
const request = { ...input, actorSource: "local_implicit" as const,
...(initiator === "automatic" ? { initiatedBy: "restart_recovery" as const } : {}),
prepareChatRetry: async () => { calls += 1; return { actionId: randomUUID() }; } };
if (initiator === "automatic") {
await expect(requestTaskRetry(request)).rejects.toMatchObject({ status: 409,
details: { code: "chat_failed_run_retry_not_authorized", reason: "saved_board_message_not_replayable" } });
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, input.companyId))).toHaveLength(0);
expect(calls).toBe(0);
return;
}
const receipt = await requestTaskRetry(request);
expect(receipt).toMatchObject({ accepted: true, stale: false, observedRunId: source.id });
expect(receipt.chatActionId).toBeUndefined();
expect(calls).toBe(0);
const [intent] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId!));
const taskRetry = intent.payload!.taskRetry as Record<string, unknown>;
expect(taskRetry).toMatchObject({ sourceRunId: source.id, actorId: input.actorId });
expect(taskRetry.chatActionId).toBeUndefined();
expect(taskRetry.boardMessageOrigin).toBeUndefined();
await db.update(agents).set({ runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } } }).where(eq(agents.id, source.agentId));
await db.insert(heartbeatRuns).values({ companyId: input.companyId, agentId: source.agentId, status: "running" });
await heartbeatService(db).resumeTaskRetry(input.companyId, receipt.requestId!);
const queued = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId),
eq(heartbeatRuns.status, "queued")));
expect(queued).toHaveLength(1);
expect(queued[0].contextSnapshot).toMatchObject({ previousRunId: source.id, taskRetryRequestId: receipt.requestId,
forceFreshSession: true, responsibleUserId: input.actorId });
expect(queued[0].contextSnapshot?.boardMessageRetry).toBeUndefined();
expect(queued[0].contextSnapshot?.wakeCommentId).toBeUndefined();
});
it("continues a person's Retry on a connected task when no chat retry owner is available", async () => {
const input = await seedChatTask();
const receipt = await requestTaskRetry({ ...input, actorSource: "local_implicit" });
expect(receipt).toMatchObject({ accepted: true, chatActionId: undefined });
const automatic = await seedChatTask();
await expect(requestTaskRetry({ ...automatic, initiatedBy: "provider_limit_reset" })).rejects.toMatchObject({ status: 409 });
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, automatic.companyId))).toHaveLength(0);
});
it("keeps an automatic Retry from continuing a connected task without the exact chat request", async () => {
const input = await seedChatTask();
await expect(requestTaskRetry({ ...input, initiatedBy: "restart_recovery", prepareChatRetry: refusedExactChatReplay() }))
.rejects.toMatchObject({ status: 409, details: { code: "chat_failed_run_retry_not_authorized" } });
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, input.companyId))).toHaveLength(0);
expect(await db.select().from(chatActions).where(eq(chatActions.companyId, input.companyId))).toHaveLength(0);
});
it("does not retry an authorization denial", async () => {
const input = await seedChatTask();
let calls = 0;
+33 -10
View File
@@ -18,13 +18,14 @@ import { preserveNativeFailedWorkspace, preserveTaskRemoteWorkspaceWriter } from
import { NativeWorkspaceFinalizationBusyError } from "./native-runtime/native-workspace-finalization-ownership.js";
import { adapterExecutionControls } from "./adapter-execution-control.js";
import { hasRequiredWorkspaceRecovery } from "./workspace-restore-recovery-state.js";
import { deferredChatInputPredicate, resumeDeferredChatWake } from "./durable-chat-wakeup.js";
import { deferredChatInputPredicate, resumeDeferredChatWake, unadmittedChatWakeupCondition } from "./durable-chat-wakeup.js";
import { notifyDeferredExecutionWork } from "./deferred-execution-notifications.js";
import { hasPendingRunCleanup } from "./execution-blocker.js";
import { PersistedExecutionWaitRefusal } from "./deferred-execution-wait.js";
import { rearmExhaustedNativeFinalization } from "./native-runtime/exhausted-finalization-retry.js";
import { NativeChatReviewPresentationContentionError } from "./native-runtime/native-chat-review-presentation.js";
import { isExternalChatWaitAuthorizationContention } from "./native-runtime/chat-attachment-reuse.js";
import { selectTaskRetryInput } from "./chat-channels/task-retry-input.js";
export async function taskRetryMutationAllowed(db: Db, input: {
companyId: string; issueId: string; actorId: string; actorSource: TaskRetryIntent["actorSource"];
@@ -241,6 +242,11 @@ function automaticRetryActor(input: Pick<TaskRetryRequest, "initiatedBy" | "acto
const TASK_RETRY_CONTENTION_ATTEMPTS = 10;
const TASK_RETRY_CONTENTION_DELAY_MS = 100;
function isExactChatReplayRefusal(error: unknown) {
return error instanceof HttpError && error.status === 409 &&
(error.details as Record<string, unknown> | undefined)?.code === "chat_failed_run_retry_not_authorized";
}
function isTaskRetryContention(error: unknown) {
return error instanceof NativeChatReviewPresentationContentionError || isExternalChatWaitAuthorizationContention(error);
}
@@ -418,9 +424,15 @@ function admitTaskRetry(input: TaskRetryRequest) {
await notifyDeferredExecutionWork(tx, { companyId: issue.companyId, requestId: id });
}
}
if (chat && !active && !busy && !boardMessageOrigin && !intent.queuedChatRequestId && (latest.contextSnapshot?.source === "issue.comment.reopen" || latest.contextSnapshot?.boardMessageRetry === true) &&
latest.responsibleUserId === input.actorId) {
throw conflict("This saved message cannot be retried safely. Send a new message on the task to continue from saved work.",
const savedBoardTurn = Boolean(chat && !active && !busy && !boardMessageOrigin && !intent.queuedChatRequestId &&
(latest.contextSnapshot?.source === "issue.comment.reopen" || latest.contextSnapshot?.boardMessageRetry === true) &&
latest.responsibleUserId === input.actorId);
const hasUnadmittedChatInput = async () => (await tx.select({ id: issues.id }).from(issues).where(and(eq(issues.id, issue.id),
eq(issues.companyId, issue.companyId), unadmittedChatWakeupCondition(issues.id, issues.companyId))).limit(1)).length > 0 ||
Boolean((await selectTaskRetryInput(tx, { companyId: issue.companyId, issueId: issue.id,
agentId: issue.assigneeAgentId!, failedRunId: latest.id }))?.sourceActionIds.length);
if (savedBoardTurn && (input.initiatedBy || await hasUnadmittedChatInput())) {
throw conflict("This saved message can't be repeated exactly as it was sent. Retry continues the task from its saved work instead.",
{ code: "chat_failed_run_retry_not_authorized", reason: "saved_board_message_not_replayable" });
}
await tx.insert(agentWakeupRequests).values({ id, companyId: issue.companyId, agentId: issue.assigneeAgentId,
@@ -429,12 +441,23 @@ function admitTaskRetry(input: TaskRetryRequest) {
status: active || busy ? "coalesced" : "deferred_issue_execution",
runId: active?.runId ?? (busy ? latest.id : null),
payload: { issueId: issue.id, taskRetry: intent }, finishedAt: active || busy ? new Date() : null });
if (chat && !boardMessageOrigin && !intent.queuedChatRequestId && !active && !busy) {
if (!input.prepareChatRetry) throw conflict("Retry is unavailable for this connected conversation.");
intent.chatActionId = (await input.prepareChatRetry(db, { companyId: issue.companyId, issueId: issue.id,
agentId: issue.assigneeAgentId, failedRunId: latest.id, initiatedByUserId: input.actorId, retryIntentId: id })).actionId;
await tx.update(agentWakeupRequests).set({ payload: { issueId: issue.id, taskRetry: intent } })
.where(eq(agentWakeupRequests.id, id));
if (chat && !boardMessageOrigin && !intent.queuedChatRequestId && !active && !busy && !savedBoardTurn) {
if (!input.prepareChatRetry && input.initiatedBy) throw conflict("Retry is unavailable for this connected conversation.");
const prepareChatRetry = input.prepareChatRetry ?? (() => { throw conflict("Retry is unavailable for this connected conversation.",
{ code: "chat_failed_run_retry_not_authorized" }); });
const scope = { companyId: issue.companyId, issueId: issue.id, agentId: issue.assigneeAgentId,
failedRunId: latest.id, initiatedByUserId: input.actorId, retryIntentId: id };
const prepared = await tx.transaction(savepoint => prepareChatRetry(savepoint as unknown as Db, scope))
.catch(async error => {
if (input.initiatedBy || !isExactChatReplayRefusal(error)) throw error;
if (await hasUnadmittedChatInput()) throw error;
return null;
});
if (prepared) {
intent.chatActionId = prepared.actionId;
await tx.update(agentWakeupRequests).set({ payload: { issueId: issue.id, taskRetry: intent } })
.where(eq(agentWakeupRequests.id, id));
}
}
if (!busy) {
for (const action of records) {