mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
fix(retry): continue a connected task from saved work when its exact chat request cannot be repeated (#21038)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Tasks that come from a connected chat (for example GitHub pull request reviews) show a Retry button when a run fails > - The task projection offers Retry, but the Retry route sends a connected task to the chat retry owner, which only replays the exact failed chat request > - When that exact request cannot be replayed (the attempt already saved a result, its output was published, the latest run did not start from a chat message, or the requester's access changed), the route returns 409 and tells the person to check the connection by hand > - A Retry button that is shown must work, and recovery must not ask the person to inspect anything > - This pull request makes a person's Retry continue the task from its saved work when the exact chat replay is refused, the same way a new message on the task would > - The benefit is that the projection and the route agree, and blocked connected tasks become retryable after deploy with no migration ## Linked Issues or Issue Description No public issue exists. This is the bug description, in the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** On a hosted deploy, five blocked tasks that came from a connected GitHub conversation showed Retry (`interruption.canRetry: true`). Each click on Retry returned 409 `chat_failed_run_retry_not_authorized`. The message was "The exact failed chat request can no longer be retried safely. Check its current connection, access, and recovery state." or "This attempt already recorded a result. Send a new message on the task to continue from the saved work." The latest runs were legacy sandbox setup failures, native runs that needed reconciliation, a run that started from an issue monitor instead of a chat message, and a native run that already saved its result. Root cause: `admitTaskRetry` in `server/src/services/task-retry.ts` called the chat retry owner for every connected task and passed its exact-replay refusal straight to the person. The projection cannot predict that refusal. **Expected behavior** A shown Retry works. If the exact chat request cannot be repeated, Retry continues the task from its saved work, the same way a new message on the task would. Users do not get operator instructions. **Steps to reproduce** 1. Bind a task to a restricted chat conversation. 2. Make its latest run fail so that the chat retry owner refuses the exact replay. For example, use a run whose source is not a chat message, or a native run with an accepted result. 3. Click Retry (`POST /api/issues/:id/retry`). 4. The route returns 409 `chat_failed_run_retry_not_authorized`. **Paperclip version or commit** `master` at5a7378af5(the deployed revision), reproduced on51d77648bc. **Deployment mode** Hosted (cloud) instance with Daytona remote sandboxes and a GitHub chat connection. **Agent adapter(s) involved** Not adapter-specific (core bug). Seen with `paperclip_runner` legacy and native runs. ## What Changed - `server/src/services/task-retry.ts`: a person's Retry runs the exact chat replay inside a savepoint. On a `chat_failed_run_retry_not_authorized` refusal, the savepoint rolls back and the same Retry receipt stays an ordinary board continuation. The clicking person is the responsible user, the run starts a fresh session (`forceFreshSession`) from saved work, and no chat action, chat authority, or output replay is created. Retry that Paperclip requests by itself (`initiatedBy: restart_recovery | provider_limit_reset`) keeps the exact-replay requirement. Other errors (403, lock contention, database errors) still propagate. - `server/src/services/chat-channels/failed-run-retry.ts`: plain refusal wording. Runs saved by an earlier release with the old diagnostic are still recognized by the pre-provider check. - Tests: new red/green cases in `server/src/services/task-retry.test.ts` (four refusal reasons on a chat task whose latest run came from an issue monitor, plus automatic Retry still refused). Updated two real-service route cases in `server/src/__tests__/chat-channels.integration.test.ts` (accepted native result on an inherited GitHub task, and an unknown chat source) from expecting 409 to expecting a board continuation without a chat replay. - A person's Retry on that person's own saved board turn (`source: issue.comment.reopen`) that cannot be certified for replay also continues from saved work, without replaying the saved message. Automatic Retry still gets 409 `saved_board_message_not_replayable`. - A person's Retry when no chat retry owner is wired also continues from saved work. Automatic Retry still gets 409. - The fallback is not used while the conversation holds saved or unadmitted chat input for the task. That input keeps its own admission path and the existing refusal. - Docs: `doc/execution-semantics.md` and `doc/connections/GITHUB.md`. ### 409 paths in the task Retry route after this change | Path | Who sees it | Result | | --- | --- | --- | | `chat_failed_run_retry_not_authorized` from the chat retry owner | Person | Continues from saved work (board continuation) | | `saved_board_message_not_replayable` | Person | Continues from saved work | | No chat retry owner wired | Person | Continues from saved work | | Any of the three above while saved or unadmitted chat input is pending | Person | 409. The pending input continues through its own admission and delivery. | | Any of the three above | Automatic Retry (restart recovery, provider limit reset) | 409, recorded by the hook. No button. | | `task_retry_busy` (chat authorization lock held) | Person | 409 with `retryable: true` after 10 internal attempts. A new click works. | | Idempotency key reused for a different task or user | Client error | 409. The UI makes a new key for each click. | ## Verification - Red before the fix: `npx vitest run src/services/task-retry.test.ts -t "connected task"` failed 4/4 with `Error: refused` (409). The board-turn and missing-owner cases failed with "This saved message cannot be retried safely..." and "Retry is unavailable for this connected conversation." `src/__tests__/chat-channels.integration.test.ts -t "continues an inherited GitHub task"` failed with `expected 202 "Accepted", got 409 "Conflict"`. - Green after the fix: `src/services/task-retry.test.ts`, `src/__tests__/durable-chat-wakeup.test.ts`, `src/services/budget-resume-continuation.test.ts`: 160/160 passed. - Full `src/__tests__/chat-channels.integration.test.ts`: 1949/1951 passed. The 2 local failures (`binds Slack tools...`, `materializes direct external-chat finals...`) read `.claude/skills/slack/SKILL.md` from the parent checkout of a nested local worktree (ENOENT). They do not touch Retry and depend on the local checkout layout. - `npx tsc --noEmit -p server/tsconfig.json`: passed. - `pnpm verify` exited 2 (a doc file is a global CI input), so required CI supplies broad coverage. - Staging was inspected with GET requests only. No POST was sent to staging. ## Risks - Behavior change: a person's Retry on a connected task can now start a board continuation where it was refused before. It uses the same authority as a new message from that person, and dispatch keeps all normal gates (task access, executor containment, recovery holds, unadmitted chat input). - The continuation does not publish the earlier result again and does not reuse the chat requester's authority. - No migration. Existing blocked tasks become retryable at the next click. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
b2cb8f8f56
commit
3c058c37e6
6 files changed
+214
-41
No files matched your search
@@ -386,10 +386,15 @@ exact executor termination proof. Successful results and uncertain delivery do
|
||||
not grant a replay. A certified failed-chat Retry retains the exact accepted
|
||||
message identity when its predecessor stopped before identity initialization,
|
||||
so a published review can expose its authorized passive response wait.
|
||||
If an inherited task-resume attempt already recorded a result, ordinary Retry
|
||||
explains that exact replay is unavailable and directs the user to send a new
|
||||
message to continue from saved work. That new turn still passes normal current
|
||||
requester, access and executor-containment checks.
|
||||
When the exact failed GitHub request cannot be repeated (for example, the
|
||||
attempt already recorded a result, its output was already published, the run
|
||||
did not start from a GitHub message, or the requester's GitHub access changed),
|
||||
a person's Retry continues the task from its saved work instead, the same way a
|
||||
new message on the task would. That continuation runs as the person who clicked
|
||||
Retry in a fresh session, never with the GitHub requester's authority, and never
|
||||
repeats the original request or its output. It still passes the normal current
|
||||
access and executor-containment checks. Automatic Retry on a person's behalf
|
||||
does not take this path; it continues only with the exact GitHub request.
|
||||
|
||||
If that successor failed at Retry authorization before runtime or provider
|
||||
startup, the next normal Retry follows its canonical source receipt. The exact
|
||||
|
||||
@@ -1611,6 +1611,10 @@ same preparation retains its directory and receipt check. A published workspace
|
||||
on the same allocation retains its recorded directory. Automatic bootstrap
|
||||
timeout retries still require a fresh allocation under the policy above.
|
||||
|
||||
### Connected task Retry without an exact chat replay
|
||||
|
||||
A person's task Retry on a connected task first asks the chat retry owner to repeat the exact failed chat request. When the latest run is that person's own saved board turn that cannot be certified for replay, when no chat retry owner is available, or when that owner refuses (`chat_failed_run_retry_not_authorized`), the refusal and any writes it made are rolled back and the same Retry receipt becomes an ordinary board continuation: the clicking person is the responsible user, the run starts a fresh session from saved work, and no chat action, chat authority, or output replay is created. This is the same authority as a new message from that person. When the conversation still holds saved or unadmitted chat input for the task, Retry keeps the exact-replay path and its refusal, because that input continues through its own admission. Dispatch keeps every normal gate (current task access, executor containment, recovery holds, unadmitted chat input). Projection and the route therefore agree: a shown Retry on a connected task is accepted. Retry that Paperclip requests by itself on a person's behalf (restart recovery, provider limit reset) keeps the exact-replay requirement and still settles on the refusal. A budget resume continues the person's own Retry, so it takes the board continuation too. No migration is needed; tasks blocked by an earlier release become retryable at the next click.
|
||||
|
||||
### Saved chat Retry preparation
|
||||
|
||||
Task Retry can meet a busy chat identity or endpoint lock before its source is authorized.
|
||||
|
||||
@@ -65471,9 +65471,14 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.retryOfRunId === f.runId)).toHaveLength(1);
|
||||
} else {
|
||||
expect([202, 409], JSON.stringify(response.body)).toContain(response.status);
|
||||
if (response.status === 202) {
|
||||
if (response.status === 202 && response.body.chatActionId) {
|
||||
const outcome = await f.service.processFailedChatRunRetry(response.body.chatActionId);
|
||||
expect(outcome.status).not.toBe("queued");
|
||||
} else if (response.status === 202) {
|
||||
const [intent] = await readAgentWakeupRequestsById(response.body.requestId);
|
||||
expect(intent).toMatchObject({ reason: "task_retry", requestedByActorId: "owner-user",
|
||||
payload: { issueId, taskRetry: { sourceRunId: f.runId } } });
|
||||
expect((intent.payload!.taskRetry as Record<string, unknown>).chatActionId).toBeUndefined();
|
||||
}
|
||||
expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.retryOfRunId === f.runId)).toHaveLength(0);
|
||||
}
|
||||
@@ -66290,7 +66295,12 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
}
|
||||
const response = await send();
|
||||
if (["principal_revoked", "accepted_result"].includes(mode)) {
|
||||
expect(response.status).toBe(409);
|
||||
expect(response.status, JSON.stringify(response.body)).toBe(202);
|
||||
expect(response.body.chatActionId).toBeUndefined();
|
||||
const [intent] = await readAgentWakeupRequestsById(response.body.requestId);
|
||||
expect(intent).toMatchObject({ reason: "task_retry", requestedByActorId: "owner-user",
|
||||
payload: { issueId, taskRetry: { sourceRunId: f.runId } } });
|
||||
expect((intent.payload!.taskRetry as Record<string, unknown>).chatActionId).toBeUndefined();
|
||||
} else {
|
||||
expect(response.status).toBe(202);
|
||||
if (mode === "source_edited") await db.update(issueComments).set({ body: "Changed after Retry acceptance" }).where(eq(issueComments.id, String(f.action.payload.commentId)));
|
||||
@@ -66378,11 +66388,26 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
const denySource = async (apply: () => Promise<unknown>, restore: () => Promise<unknown>, stale = false) => {
|
||||
try {
|
||||
await apply();
|
||||
const denied = await request(app).post(`/api/issues/${issueId}/retry`).send({ idempotencyKey: randomUUID(), observedRunId: current!.id }).expect(stale ? 200 : 409);
|
||||
if (stale) expect(denied.body).toMatchObject({ accepted: false, stale: true });
|
||||
expect(await db.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
|
||||
eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.reason, "task_retry")))).toHaveLength(0);
|
||||
} finally { await restore(); }
|
||||
const denied = await request(app).post(`/api/issues/${issueId}/retry`).send({ idempotencyKey: randomUUID(), observedRunId: current!.id }).expect(stale ? 200 : 202);
|
||||
const retries = await db.select().from(agentWakeupRequests).where(and(
|
||||
eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.reason, "task_retry")));
|
||||
if (stale) {
|
||||
expect(denied.body).toMatchObject({ accepted: false, stale: true });
|
||||
expect(retries).toHaveLength(0);
|
||||
} else {
|
||||
expect(denied.body.chatActionId).toBeUndefined();
|
||||
expect(retries).toHaveLength(1);
|
||||
expect(retries[0]).toMatchObject({ id: denied.body.requestId, requestedByActorId: "owner-user",
|
||||
payload: { issueId, taskRetry: { sourceRunId: current!.id } } });
|
||||
const taskRetry = retries[0].payload!.taskRetry as Record<string, unknown>;
|
||||
expect(taskRetry.boardMessageOrigin).toBeUndefined();
|
||||
expect(taskRetry.chatActionId).toBeUndefined();
|
||||
}
|
||||
} finally {
|
||||
await db.delete(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, companyId),
|
||||
eq(agentWakeupRequests.reason, "task_retry")));
|
||||
await restore();
|
||||
}
|
||||
};
|
||||
await denySource(() => db.update(issueComments).set({ body: "Edited after acceptance", updatedAt: new Date() }).where(eq(issueComments.id, comment.id)),
|
||||
() => db.update(issueComments).set({ body: comment.body, updatedAt: sql`${issueComments.createdAt}` }).where(eq(issueComments.id, comment.id)));
|
||||
@@ -66493,17 +66518,26 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
const [firstRetry] = await readHeartbeatRunsById(successor.id);
|
||||
const nextRequest = () => request(app).post(`/api/issues/${issueId}/retry`).send({
|
||||
idempotencyKey: randomUUID(), observedRunId: firstRetry.id });
|
||||
const continueWithoutReplay = async () => {
|
||||
const continued = await nextRequest().expect(202);
|
||||
expect(continued.body.chatActionId).toBeUndefined();
|
||||
const [intent] = await readAgentWakeupRequestsById(continued.body.requestId);
|
||||
expect(intent).toMatchObject({ reason: "task_retry", requestedByActorId: "owner-user",
|
||||
payload: { issueId, taskRetry: { sourceRunId: firstRetry.id } } });
|
||||
expect((intent.payload!.taskRetry as Record<string, unknown>).boardMessageOrigin).toBeUndefined();
|
||||
await db.delete(agentWakeupRequests).where(eq(agentWakeupRequests.id, intent.id));
|
||||
};
|
||||
await db.update(heartbeatRuns).set({ sessionIdAfter: "contradictory-provider-session" }).where(eq(heartbeatRuns.id, firstRetry.id));
|
||||
await nextRequest().expect(409);
|
||||
await continueWithoutReplay();
|
||||
await db.update(heartbeatRuns).set({ sessionIdAfter: null }).where(eq(heartbeatRuns.id, firstRetry.id));
|
||||
await db.update(issueComments).set({ body: "Changed original accepted message", updatedAt: new Date() }).where(eq(issueComments.id, comment.id));
|
||||
await nextRequest().expect(409);
|
||||
await continueWithoutReplay();
|
||||
await db.update(issueComments).set({ body: comment.body, updatedAt: sql`${issueComments.createdAt}` }).where(eq(issueComments.id, comment.id));
|
||||
const [firstIntent] = await readAgentWakeupRequestsById(retry.id);
|
||||
await db.update(agentWakeupRequests).set({ payload: { ...firstIntent.payload,
|
||||
taskRetry: { ...(firstIntent.payload!.taskRetry as object), sourceRunId: firstRetry.id } } }).where(eq(agentWakeupRequests.id, firstIntent.id));
|
||||
await db.update(heartbeatRuns).set({ contextSnapshot: { ...firstRetry.contextSnapshot, previousRunId: firstRetry.id } }).where(eq(heartbeatRuns.id, firstRetry.id));
|
||||
await nextRequest().expect(409);
|
||||
await continueWithoutReplay();
|
||||
await db.update(agentWakeupRequests).set({ payload: firstIntent.payload }).where(eq(agentWakeupRequests.id, firstIntent.id));
|
||||
await db.update(heartbeatRuns).set({ contextSnapshot: firstRetry.contextSnapshot }).where(eq(heartbeatRuns.id, firstRetry.id));
|
||||
const second = await nextRequest().expect(202);
|
||||
@@ -66544,7 +66578,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("distinguishes an inherited GitHub requester from permission to replay a committed provider failure", async () => {
|
||||
it("continues an inherited GitHub task from its recorded result instead of replaying the committed provider failure", async () => {
|
||||
const f = await failedChatRetryFixture("github", "42", true, false, true);
|
||||
try {
|
||||
const companyId = f.fixture.companyId, agentId = f.fixture.assignedAgentId, issueId = f.issue.id;
|
||||
@@ -66593,19 +66627,30 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
app.use((req, _res, next) => { req.actor = boardActor(companyId); next(); });
|
||||
app.use("/api", issueRoutes(db, createStorageService().storage, { chatRunRetries: f.service }));
|
||||
app.use(errorHandler);
|
||||
const denied = await request(app).post(`/api/issues/${issueId}/retry`).send({
|
||||
const continued = await request(app).post(`/api/issues/${issueId}/retry`).send({
|
||||
idempotencyKey: input.idempotencyKey, observedRunId: input.observedRunId,
|
||||
}).expect(409);
|
||||
expect(denied.body).toMatchObject({ error: "This attempt already recorded a result. Send a new message on the task to continue from the saved work.",
|
||||
details: { code: "chat_failed_run_retry_not_authorized", reason: "accepted_result_without_new_input" } });
|
||||
// Isolate the later accepted-result guard using only owned fixture data.
|
||||
// Resolving inherited provenance cannot by itself authorize replay.
|
||||
}).expect(202);
|
||||
expect(continued.body).toMatchObject({ accepted: true, stale: false, observedRunId: resumed!.id });
|
||||
expect(continued.body.chatActionId).toBeUndefined();
|
||||
const assertBoardContinuation = async (requestId: string) => {
|
||||
const [intent] = await readAgentWakeupRequestsById(requestId);
|
||||
expect(intent).toMatchObject({ reason: "task_retry", requestedByActorType: "user", requestedByActorId: "owner-user",
|
||||
payload: { issueId, taskRetry: { sourceRunId: resumed!.id, actorId: "owner-user" } } });
|
||||
expect((intent.payload!.taskRetry as Record<string, unknown>).chatActionId).toBeUndefined();
|
||||
expect(await db.select().from(chatActions).where(and(eq(chatActions.companyId, companyId),
|
||||
eq(chatActions.kind, "failed_run_retry")))).toHaveLength(0);
|
||||
await db.update(agentWakeupRequests).set({ status: "failed", finishedAt: new Date() }).where(eq(agentWakeupRequests.id, requestId));
|
||||
};
|
||||
await assertBoardContinuation(continued.body.requestId);
|
||||
// The later accepted-result guard also continues from the saved result
|
||||
// instead of repeating the exact chat request.
|
||||
await db.update(heartbeatRuns).set({ contextSnapshot: { ...failedResume.contextSnapshot,
|
||||
source: "chat:github", wakeCommentId: f.action.payload.commentId, wakeCommentIds: [f.action.payload.commentId] } })
|
||||
.where(eq(heartbeatRuns.id, failedResume.id));
|
||||
try {
|
||||
await expect(requestTaskRetry({ ...input, idempotencyKey: randomUUID() })).rejects.toMatchObject({ details: {
|
||||
code: "chat_failed_run_retry_not_authorized", reason: "accepted_result_without_new_input" } });
|
||||
const receipt = await requestTaskRetry({ ...input, idempotencyKey: randomUUID() });
|
||||
expect(receipt).toMatchObject({ accepted: true, chatActionId: undefined });
|
||||
await assertBoardContinuation(receipt.requestId!);
|
||||
} finally { await db.update(heartbeatRuns).set({ contextSnapshot: failedResume.contextSnapshot }).where(eq(heartbeatRuns.id, failedResume.id)); }
|
||||
expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.retryOfRunId === resumed!.id)).toHaveLength(0);
|
||||
} finally { await f.service.shutdown(); }
|
||||
@@ -67164,9 +67209,12 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
} finally { unsubscribes.forEach(unsubscribe => unsubscribe()); }
|
||||
}
|
||||
if (mode !== "committed_board_existing" && mode !== "committed_board_existing_coalesced" && mode !== "committed_board_existing_identity_contended" && mode !== "committed_board_existing_identity_expired") {
|
||||
await expect(requestTaskRetry(nextInput)).rejects.toMatchObject({
|
||||
details: { code: "chat_failed_run_retry_not_authorized" },
|
||||
});
|
||||
const continued = await requestTaskRetry(nextInput);
|
||||
expect(continued).toMatchObject({ accepted: true, chatActionId: undefined });
|
||||
const [intent] = await readAgentWakeupRequestsById(continued.requestId!);
|
||||
expect(intent).toMatchObject({ reason: "task_retry", requestedByActorId: "owner-user",
|
||||
payload: { taskRetry: { sourceRunId: failed.id } } });
|
||||
expect((intent.payload!.taskRetry as Record<string, unknown>).chatActionId).toBeUndefined();
|
||||
expect(await db.select().from(agentWakeupRequests).where(inArray(agentWakeupRequests.id, inputIds)).orderBy(asc(agentWakeupRequests.id)))
|
||||
.toEqual(before);
|
||||
return;
|
||||
|
||||
@@ -102,6 +102,9 @@ type DbOrTransaction = Db | Parameters<Parameters<Db["transaction"]>[0]>[0];
|
||||
type DeliveryRow = typeof chatDeliveries.$inferSelect;
|
||||
type RetryRuntimeFence = { credentialFingerprint: string; generation: number };
|
||||
const FAILED_RUN_RETRY_IDENTITY_CONTENTION_MS = 60_000;
|
||||
const FAILED_CHAT_RETRY_DIAGNOSTIC = "This chat request can't be repeated exactly as it was sent. Retry continues the task from its saved work instead.";
|
||||
const FAILED_CHAT_RETRY_DIAGNOSTICS = [FAILED_CHAT_RETRY_DIAGNOSTIC,
|
||||
"The exact failed chat request can no longer be retried safely. Check its current connection, access, and recovery state."];
|
||||
export type ChatFailedRetrySource = Awaited<ReturnType<ReturnType<typeof createChatFailedRunRetry>["failedChatRetrySource"]>>;
|
||||
|
||||
type ChatFailedRunRetryOptions = {
|
||||
@@ -139,8 +142,8 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti
|
||||
"prior_result_publication" | "prior_interaction" | "source_authorization", authorityReason?: ChatWakeAuthorityReason) =>
|
||||
conflict(
|
||||
reason === "accepted_result_without_new_input"
|
||||
? "This attempt already recorded a result. Send a new message on the task to continue from the saved work."
|
||||
: "The exact failed chat request can no longer be retried safely. Check its current connection, access, and recovery state.",
|
||||
? "This attempt already saved its result, so it is not repeated. Retry continues from that saved result."
|
||||
: FAILED_CHAT_RETRY_DIAGNOSTIC,
|
||||
{ code: "chat_failed_run_retry_not_authorized", ...(reason ? { reason } : {}), ...(reason === "source_authorization" && authorityReason ? { authorityReason } : {}) },
|
||||
);
|
||||
class FailedChatRetryPublicationReadError extends Error {
|
||||
@@ -227,8 +230,9 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti
|
||||
run: typeof heartbeatRuns.$inferSelect,
|
||||
allowWorkspacePreparation = false,
|
||||
): Promise<boolean> {
|
||||
const retryDiagnostic = "The exact failed chat request can no longer be retried safely. Check its current connection, access, and recovery state.";
|
||||
const failedRetry = run.errorCode === "chat_failed_run_retry_not_authorized" && run.error === retryDiagnostic;
|
||||
const failedRetry = run.errorCode === "chat_failed_run_retry_not_authorized" &&
|
||||
FAILED_CHAT_RETRY_DIAGNOSTICS.includes(run.error ?? "");
|
||||
const retryDiagnostic = failedRetry ? run.error : null;
|
||||
const validation = run.resultJson?.workspaceValidation as Record<string, unknown> | undefined;
|
||||
const recovery = run.resultJson?.executionRecovery as Record<string, unknown> | undefined;
|
||||
const workspaceValidation = run.errorCode === "workspace_validation_failed" &&
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import { agents, agentWakeupRequests, chatConversations, chatEndpoints, companies, budgetPolicies, costEvents, createDb, type Db, environmentLeases, environments, executionWorkspaces, executionWorkspaceSnapshots, workspaceOperations, heartbeatRunEvents, heartbeatRuns, issueComments, issues, approvals, issueApprovals, issueRecoveryActions, nativeRunFinalizations, toolApplications, toolConnections, toolInvocations, applyPendingMigrations } from "@paperclipai/db";
|
||||
import { agents, agentWakeupRequests, chatActions, chatConversations, chatEndpoints, companies, budgetPolicies, costEvents, createDb, type Db, environmentLeases, environments, executionWorkspaces, executionWorkspaceSnapshots, workspaceOperations, heartbeatRunEvents, heartbeatRuns, issueComments, issues, approvals, issueApprovals, issueRecoveryActions, nativeRunFinalizations, toolApplications, toolConnections, toolInvocations, applyPendingMigrations } from "@paperclipai/db";
|
||||
import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
|
||||
import { priorRecoveryApiReceipt, unresolvedRecoveryToolInvocation } from "./recovery-effect-receipts.js";
|
||||
import { requestTaskRetry } from "./task-retry.js";
|
||||
@@ -937,6 +937,95 @@ describe("durable task Retry admission", () => {
|
||||
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, input.companyId))).toHaveLength(0);
|
||||
});
|
||||
|
||||
const refusedExactChatReplay = (reason?: string) => async (tx: Db, scope: { companyId: string; issueId: string; failedRunId: string }) => {
|
||||
const [endpoint] = await tx.select().from(chatEndpoints).where(eq(chatEndpoints.companyId, scope.companyId));
|
||||
await tx.insert(chatActions).values({ companyId: scope.companyId, endpointId: endpoint.id, kind: "failed_run_retry",
|
||||
status: "issued", providerActionId: randomUUID(), payload: { issueId: scope.issueId, failedRunId: scope.failedRunId } });
|
||||
throw new HttpError(409, "refused", { code: "chat_failed_run_retry_not_authorized", ...(reason ? { reason } : {}) });
|
||||
};
|
||||
|
||||
it.each([undefined, "accepted_result_without_new_input", "prior_result_publication", "source_authorization"])(
|
||||
"continues a connected task from saved work when its exact chat request cannot be repeated: %s", async reason => {
|
||||
const input = await seedChatTask();
|
||||
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.observedRunId));
|
||||
await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: input.issueId, source: "issue.monitor",
|
||||
wakeReason: "issue_monitor_due" } }).where(eq(heartbeatRuns.id, source.id));
|
||||
const receipt = await requestTaskRetry({ ...input, actorSource: "local_implicit", prepareChatRetry: refusedExactChatReplay(reason) });
|
||||
expect(receipt).toMatchObject({ accepted: true, stale: false, observedRunId: source.id });
|
||||
expect(receipt.chatActionId).toBeUndefined();
|
||||
expect(await db.select().from(chatActions).where(eq(chatActions.companyId, input.companyId))).toHaveLength(0);
|
||||
const [intent] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId!));
|
||||
expect(intent).toMatchObject({ status: "deferred_issue_execution", reason: "task_retry", requestedByActorId: input.actorId,
|
||||
payload: { issueId: input.issueId, taskRetry: { sourceRunId: source.id, actorId: input.actorId } } });
|
||||
expect((intent.payload!.taskRetry as Record<string, unknown>).chatActionId).toBeUndefined();
|
||||
await db.update(agents).set({ runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } } }).where(eq(agents.id, source.agentId));
|
||||
await db.insert(heartbeatRuns).values({ companyId: input.companyId, agentId: source.agentId, status: "running" });
|
||||
await heartbeatService(db).resumeTaskRetry(input.companyId, receipt.requestId!);
|
||||
const queued = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId),
|
||||
eq(heartbeatRuns.status, "queued")));
|
||||
expect(queued).toHaveLength(1);
|
||||
expect(queued[0].contextSnapshot).toMatchObject({ issueId: input.issueId, previousRunId: source.id,
|
||||
taskRetryRequestId: receipt.requestId, forceFreshSession: true, responsibleUserId: input.actorId });
|
||||
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId!)))[0])
|
||||
.toMatchObject({ status: "coalesced", runId: queued[0].id });
|
||||
});
|
||||
|
||||
it.each(["person", "automatic"] as const)(
|
||||
"continues a person's saved board turn on a connected task from saved work without replaying it: %s", async initiator => {
|
||||
const input = await seedChatTask();
|
||||
const [source] = await db.update(heartbeatRuns).set({ runtimeMode: "legacy", errorCode: "adapter_failed",
|
||||
error: "Provider stopped", responsibleUserId: input.actorId, sessionIdAfter: "provider-session",
|
||||
contextSnapshot: { issueId: input.issueId, source: "issue.comment.reopen", wakeCommentId: randomUUID() } })
|
||||
.where(eq(heartbeatRuns.id, input.observedRunId)).returning();
|
||||
let calls = 0;
|
||||
const request = { ...input, actorSource: "local_implicit" as const,
|
||||
...(initiator === "automatic" ? { initiatedBy: "restart_recovery" as const } : {}),
|
||||
prepareChatRetry: async () => { calls += 1; return { actionId: randomUUID() }; } };
|
||||
if (initiator === "automatic") {
|
||||
await expect(requestTaskRetry(request)).rejects.toMatchObject({ status: 409,
|
||||
details: { code: "chat_failed_run_retry_not_authorized", reason: "saved_board_message_not_replayable" } });
|
||||
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, input.companyId))).toHaveLength(0);
|
||||
expect(calls).toBe(0);
|
||||
return;
|
||||
}
|
||||
const receipt = await requestTaskRetry(request);
|
||||
expect(receipt).toMatchObject({ accepted: true, stale: false, observedRunId: source.id });
|
||||
expect(receipt.chatActionId).toBeUndefined();
|
||||
expect(calls).toBe(0);
|
||||
const [intent] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId!));
|
||||
const taskRetry = intent.payload!.taskRetry as Record<string, unknown>;
|
||||
expect(taskRetry).toMatchObject({ sourceRunId: source.id, actorId: input.actorId });
|
||||
expect(taskRetry.chatActionId).toBeUndefined();
|
||||
expect(taskRetry.boardMessageOrigin).toBeUndefined();
|
||||
await db.update(agents).set({ runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } } }).where(eq(agents.id, source.agentId));
|
||||
await db.insert(heartbeatRuns).values({ companyId: input.companyId, agentId: source.agentId, status: "running" });
|
||||
await heartbeatService(db).resumeTaskRetry(input.companyId, receipt.requestId!);
|
||||
const queued = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId),
|
||||
eq(heartbeatRuns.status, "queued")));
|
||||
expect(queued).toHaveLength(1);
|
||||
expect(queued[0].contextSnapshot).toMatchObject({ previousRunId: source.id, taskRetryRequestId: receipt.requestId,
|
||||
forceFreshSession: true, responsibleUserId: input.actorId });
|
||||
expect(queued[0].contextSnapshot?.boardMessageRetry).toBeUndefined();
|
||||
expect(queued[0].contextSnapshot?.wakeCommentId).toBeUndefined();
|
||||
});
|
||||
|
||||
it("continues a person's Retry on a connected task when no chat retry owner is available", async () => {
|
||||
const input = await seedChatTask();
|
||||
const receipt = await requestTaskRetry({ ...input, actorSource: "local_implicit" });
|
||||
expect(receipt).toMatchObject({ accepted: true, chatActionId: undefined });
|
||||
const automatic = await seedChatTask();
|
||||
await expect(requestTaskRetry({ ...automatic, initiatedBy: "provider_limit_reset" })).rejects.toMatchObject({ status: 409 });
|
||||
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, automatic.companyId))).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("keeps an automatic Retry from continuing a connected task without the exact chat request", async () => {
|
||||
const input = await seedChatTask();
|
||||
await expect(requestTaskRetry({ ...input, initiatedBy: "restart_recovery", prepareChatRetry: refusedExactChatReplay() }))
|
||||
.rejects.toMatchObject({ status: 409, details: { code: "chat_failed_run_retry_not_authorized" } });
|
||||
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, input.companyId))).toHaveLength(0);
|
||||
expect(await db.select().from(chatActions).where(eq(chatActions.companyId, input.companyId))).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("does not retry an authorization denial", async () => {
|
||||
const input = await seedChatTask();
|
||||
let calls = 0;
|
||||
|
||||
@@ -18,13 +18,14 @@ import { preserveNativeFailedWorkspace, preserveTaskRemoteWorkspaceWriter } from
|
||||
import { NativeWorkspaceFinalizationBusyError } from "./native-runtime/native-workspace-finalization-ownership.js";
|
||||
import { adapterExecutionControls } from "./adapter-execution-control.js";
|
||||
import { hasRequiredWorkspaceRecovery } from "./workspace-restore-recovery-state.js";
|
||||
import { deferredChatInputPredicate, resumeDeferredChatWake } from "./durable-chat-wakeup.js";
|
||||
import { deferredChatInputPredicate, resumeDeferredChatWake, unadmittedChatWakeupCondition } from "./durable-chat-wakeup.js";
|
||||
import { notifyDeferredExecutionWork } from "./deferred-execution-notifications.js";
|
||||
import { hasPendingRunCleanup } from "./execution-blocker.js";
|
||||
import { PersistedExecutionWaitRefusal } from "./deferred-execution-wait.js";
|
||||
import { rearmExhaustedNativeFinalization } from "./native-runtime/exhausted-finalization-retry.js";
|
||||
import { NativeChatReviewPresentationContentionError } from "./native-runtime/native-chat-review-presentation.js";
|
||||
import { isExternalChatWaitAuthorizationContention } from "./native-runtime/chat-attachment-reuse.js";
|
||||
import { selectTaskRetryInput } from "./chat-channels/task-retry-input.js";
|
||||
|
||||
export async function taskRetryMutationAllowed(db: Db, input: {
|
||||
companyId: string; issueId: string; actorId: string; actorSource: TaskRetryIntent["actorSource"];
|
||||
@@ -241,6 +242,11 @@ function automaticRetryActor(input: Pick<TaskRetryRequest, "initiatedBy" | "acto
|
||||
const TASK_RETRY_CONTENTION_ATTEMPTS = 10;
|
||||
const TASK_RETRY_CONTENTION_DELAY_MS = 100;
|
||||
|
||||
function isExactChatReplayRefusal(error: unknown) {
|
||||
return error instanceof HttpError && error.status === 409 &&
|
||||
(error.details as Record<string, unknown> | undefined)?.code === "chat_failed_run_retry_not_authorized";
|
||||
}
|
||||
|
||||
function isTaskRetryContention(error: unknown) {
|
||||
return error instanceof NativeChatReviewPresentationContentionError || isExternalChatWaitAuthorizationContention(error);
|
||||
}
|
||||
@@ -418,9 +424,15 @@ function admitTaskRetry(input: TaskRetryRequest) {
|
||||
await notifyDeferredExecutionWork(tx, { companyId: issue.companyId, requestId: id });
|
||||
}
|
||||
}
|
||||
if (chat && !active && !busy && !boardMessageOrigin && !intent.queuedChatRequestId && (latest.contextSnapshot?.source === "issue.comment.reopen" || latest.contextSnapshot?.boardMessageRetry === true) &&
|
||||
latest.responsibleUserId === input.actorId) {
|
||||
throw conflict("This saved message cannot be retried safely. Send a new message on the task to continue from saved work.",
|
||||
const savedBoardTurn = Boolean(chat && !active && !busy && !boardMessageOrigin && !intent.queuedChatRequestId &&
|
||||
(latest.contextSnapshot?.source === "issue.comment.reopen" || latest.contextSnapshot?.boardMessageRetry === true) &&
|
||||
latest.responsibleUserId === input.actorId);
|
||||
const hasUnadmittedChatInput = async () => (await tx.select({ id: issues.id }).from(issues).where(and(eq(issues.id, issue.id),
|
||||
eq(issues.companyId, issue.companyId), unadmittedChatWakeupCondition(issues.id, issues.companyId))).limit(1)).length > 0 ||
|
||||
Boolean((await selectTaskRetryInput(tx, { companyId: issue.companyId, issueId: issue.id,
|
||||
agentId: issue.assigneeAgentId!, failedRunId: latest.id }))?.sourceActionIds.length);
|
||||
if (savedBoardTurn && (input.initiatedBy || await hasUnadmittedChatInput())) {
|
||||
throw conflict("This saved message can't be repeated exactly as it was sent. Retry continues the task from its saved work instead.",
|
||||
{ code: "chat_failed_run_retry_not_authorized", reason: "saved_board_message_not_replayable" });
|
||||
}
|
||||
await tx.insert(agentWakeupRequests).values({ id, companyId: issue.companyId, agentId: issue.assigneeAgentId,
|
||||
@@ -429,12 +441,23 @@ function admitTaskRetry(input: TaskRetryRequest) {
|
||||
status: active || busy ? "coalesced" : "deferred_issue_execution",
|
||||
runId: active?.runId ?? (busy ? latest.id : null),
|
||||
payload: { issueId: issue.id, taskRetry: intent }, finishedAt: active || busy ? new Date() : null });
|
||||
if (chat && !boardMessageOrigin && !intent.queuedChatRequestId && !active && !busy) {
|
||||
if (!input.prepareChatRetry) throw conflict("Retry is unavailable for this connected conversation.");
|
||||
intent.chatActionId = (await input.prepareChatRetry(db, { companyId: issue.companyId, issueId: issue.id,
|
||||
agentId: issue.assigneeAgentId, failedRunId: latest.id, initiatedByUserId: input.actorId, retryIntentId: id })).actionId;
|
||||
await tx.update(agentWakeupRequests).set({ payload: { issueId: issue.id, taskRetry: intent } })
|
||||
.where(eq(agentWakeupRequests.id, id));
|
||||
if (chat && !boardMessageOrigin && !intent.queuedChatRequestId && !active && !busy && !savedBoardTurn) {
|
||||
if (!input.prepareChatRetry && input.initiatedBy) throw conflict("Retry is unavailable for this connected conversation.");
|
||||
const prepareChatRetry = input.prepareChatRetry ?? (() => { throw conflict("Retry is unavailable for this connected conversation.",
|
||||
{ code: "chat_failed_run_retry_not_authorized" }); });
|
||||
const scope = { companyId: issue.companyId, issueId: issue.id, agentId: issue.assigneeAgentId,
|
||||
failedRunId: latest.id, initiatedByUserId: input.actorId, retryIntentId: id };
|
||||
const prepared = await tx.transaction(savepoint => prepareChatRetry(savepoint as unknown as Db, scope))
|
||||
.catch(async error => {
|
||||
if (input.initiatedBy || !isExactChatReplayRefusal(error)) throw error;
|
||||
if (await hasUnadmittedChatInput()) throw error;
|
||||
return null;
|
||||
});
|
||||
if (prepared) {
|
||||
intent.chatActionId = prepared.actionId;
|
||||
await tx.update(agentWakeupRequests).set({ payload: { issueId: issue.id, taskRetry: intent } })
|
||||
.where(eq(agentWakeupRequests.id, id));
|
||||
}
|
||||
}
|
||||
if (!busy) {
|
||||
for (const action of records) {
|
||||
|
||||
Reference in new issue
Block a user