mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
fix(native): continue a routine assignment or chat message whose remote Runner an upgrade lost mid-turn (#21052)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on remote sandboxes (Daytona) use a detached Runner. When the server is replaced during an upgrade, the new server reattaches to that Runner > - On staging, upgrades every 15–40 minutes lose the in-flight Runner. The same-run reattachment then finds `process_verification` / `process_not_running`, the run fails with `runner_remote_process_identity_unavailable`, and the task is blocked > - #21039 made these runs continue once in a fresh session (the turn is recorded as having an unknown outcome and is not replayed). It covers user wakes and automatic retries (`retryOfRunId`) only > - A routine's system assignment (`issue_assigned`, requested by the system) and an external chat message wake have no `retryOfRunId`. They got `not_user_request` and stayed blocked for a person > - This pull request treats those two wakes as owed task work, so they continue the same way once the Runner is proven gone > - The benefit is that an upgrade no longer leaves these tasks waiting for a Retry click ## Linked Issues or Issue Description Refs #21039, #21023, #20923 **What happened?** Staging replaced its server at about 07:10Z. 17 remote native runs were in flight. Each new controller reached its Runner 114–152 s after detach. Recovery failed with "PRP command session.snapshot timed out". The same-run reattachment then found the Runner process gone, and each run failed with `native_continuation_requires_reconciliation`. #21039 continued 14 of the 17 automatically. The other 3 were a routine's assignment (two runs) and an external chat message. They have no `retryOfRunId`, so `requestInterruptedNativeTurnRetry` returned `not_user_request`. Those tasks are still blocked with no retry. **What did you expect to happen?** The Runner is proven gone, so nothing can still run the turn. The run should continue once in a fresh session, the same as an automatic retry of owed work. **Steps to reproduce** 1. Start a remote native run from a routine assignment, or from an external chat message. 2. Replace the server while the provider turn is active, so that the reattachment finds `process_not_running`. 3. The run fails and the task stays blocked. No automatic Retry is requested. ## What Changed - `server/src/services/heartbeat/restart-interrupted-turn.ts`: after the Runner is proven gone, a system wake also continues when it is a system assignment (`issue_assigned`), or when it is a comment or external chat message wake whose message is still on the task (not deleted, not written by a run). It runs under the run's responsible user, at most once per chain of runs, like an automatic retry already does. A linked person's external chat message continues under that person when the message is still on the task. Monitors and other system wakes are unchanged. - The same file and `server/src/app.ts`: the hook now passes the chat service's Retry callback (`prepareFailedChatRunRetry`, registered at app startup next to the budget-resume callback). A task in a connected conversation whose endpoint restricts external execution then continues through the chat Retry path. Before, `requestTaskRetry` refused it (`retry_refused`) and the task stayed blocked. - `server/src/services/runner-lost-mid-turn-continuation.test.ts`: production-shape cases for a system assignment, an unlinked external chat sender, and a linked person's chat message. Negative cases: a deleted message, a message written by a run, a deleted linked message, and a system assignment whose Runner was not proven gone. - `doc/execution-semantics.md`: describes the rule. ## Verification - Red on `origin/master` with only the new tests: `cd server && npx vitest run src/services/runner-lost-mid-turn-continuation.test.ts` fails 4 of 16. The three new continuation cases return `{ requested: false }`. The deleted linked-message case returns `not_user_request` instead of `message_unavailable`. - Green with the fix: 16 of 16. Two more cases bind the task to a real restricted Slack endpoint and conversation. With the chat callback registered, the run continues and the Retry carries the chat action (18 of 18). Without the callback, the result is `retry_refused`, which was the behavior before for every connected conversation. - Related suites pass: `runner-lost-mid-turn-continuation.test.ts`, `heartbeat-native-remote-failure-retention.test.ts`, `workspace-recovery-retry-copyback.test.ts`, `governed-wait-runner-loss.test.ts` (116 tests). - `cd server && npx tsc --noEmit` passes. - Production evidence was read from staging run events through the read-only API: 3 of 17 losses at the 07:10Z swap took this path. After this change the next post-upgrade audit should show an automatic `task_retry` for every `native_continuation_requires_reconciliation` run whose refusal is `process_not_running`, including routine assignments and chat messages. ## Risks - Low. The change applies only after the Runner process is proven gone (`process_verification` / `process_not_running`), so no Runner can still run the turn. The turn is not replayed. The fresh session continues over the saved work, as in #21039. - The task can contain partial tool work from the lost turn. This is the same accepted risk as #21039. - Previous-release story: no persisted shape changes. The rule reads the wake's reason and comment id, which earlier releases already write. A run an older release already settled keeps its visible Retry. - This does not stop the Runner loss itself. That is a separate fix. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic) via Claude Code, with tool use (shell, file edits, test runs, read-only staging API reads). Model ID as attributed in the commit trailer: Claude Fable 5.2. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
1 parent
4e519bd428
commit
fd5ee7e385
4 files changed
+130
-22
No files matched your search
@@ -1771,7 +1771,7 @@ turn is never replayed. Activity records the request
|
||||
as `restart-recovery` on behalf of that user. Admission keeps every gate: saved-work
|
||||
recovery, stop proof, approvals, pauses, budget and ownership. A Retry that the
|
||||
server requested gets no second automatic attempt. A system wake (for example a
|
||||
monitor or an automatic assignment) continues this way only with a positive
|
||||
monitor) continues this way only with a positive
|
||||
record that no turn was ever issued: the refused reattachment proved the Runner
|
||||
process gone, and the run's controller journal, which records each turn command
|
||||
before it is sent, holds no turn command and no turn event
|
||||
@@ -1779,11 +1779,15 @@ before it is sent, holds no turn command and no turn event
|
||||
must also have no turn or tool event, and the Retry runs under the run's
|
||||
responsible user. Missing events alone never qualify, because a turn can reach
|
||||
the provider before its events are stored. An automatic retry of owed work (a
|
||||
system wake whose run has `retryOfRunId`, such as `transient_failure_retry`)
|
||||
also continues under the run's responsible user once the refusal proved its
|
||||
Runner process gone, because losing the Runner to a restart is not the task's
|
||||
failure. Other system wakes, and in-flight provider turns without that proof,
|
||||
do not continue automatically. For these failed runs, a
|
||||
system wake whose run has `retryOfRunId`, such as `transient_failure_retry`, a
|
||||
system assignment such as a routine's dispatch (`issue_assigned`), or a comment
|
||||
or external chat message wake whose message is still on the task and was not
|
||||
written by a run) also continues under the run's responsible user once the
|
||||
refusal proved its Runner process gone, because losing the Runner to a restart
|
||||
is not the task's failure. A linked person's external chat message continues
|
||||
under that person once the message is still on the task. Monitors and other
|
||||
system wakes, and in-flight provider turns without that proof, do not continue
|
||||
automatically. Each chain of runs gets at most one such automatic Retry. For these failed runs, a
|
||||
recovered workspace hold offers Retry (`canRetry`) and says "Retry or send a
|
||||
message to continue." An intentional Stop keeps "send a message" as its only
|
||||
continuation.
|
||||
|
||||
@@ -226,6 +226,7 @@ import { chatInboundWakeNoticeWork } from "./services/chat-inbound-wake-notice-w
|
||||
import { chatFailedRetryNoticeWork } from "./services/chat-failed-retry-notice-work.js";
|
||||
import { chatRunMilestoneWork } from "./services/chat-run-milestone-work.js";
|
||||
import { heartbeatService } from "./services/heartbeat.js";
|
||||
import { registerInterruptedTurnChatRetry } from "./services/heartbeat/restart-interrupted-turn.js";
|
||||
import { pluginLifecycleManager } from "./services/plugin-lifecycle.js";
|
||||
import { registerPluginLifecycleDelivery } from "./services/plugin-lifecycle-delivery.js";
|
||||
import { createPluginJobCoordinator } from "./services/plugin-job-coordinator.js";
|
||||
@@ -603,6 +604,8 @@ export async function createApp(
|
||||
storage: opts.storageService,
|
||||
});
|
||||
if (opts.budgetResumeDelivery) opts.budgetResumeDelivery.prepareChatRetry = chatChannels.prepareFailedChatRunRetry;
|
||||
// A restart's automatic Retry of a connected conversation needs the same callback.
|
||||
registerInterruptedTurnChatRetry(chatChannels.prepareFailedChatRunRetry);
|
||||
// Voice capabilities are verified by the signed callback route, not by
|
||||
// board/agent bearer authentication. Mount before actorMiddleware so its
|
||||
// per-session Bearer cannot be mistaken for an agent API key.
|
||||
|
||||
@@ -4,6 +4,7 @@ import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js";
|
||||
import type { NativeRestartRecoveryClaim } from "../native-runtime/native-restart-recovery.js";
|
||||
import { isUserCommentContinuationReason } from "../explicit-native-continuation.js";
|
||||
import { requestTaskRetry } from "../task-retry.js";
|
||||
import { EXTERNAL_CHAT_MESSAGE_WAKE_REASON } from "../queued-comment-delivery.js";
|
||||
import { HttpError } from "../../errors.js";
|
||||
import { logger } from "../../middleware/logger.js";
|
||||
import { NATIVE_REMOTE_RECOVERY_LEASE_MISMATCH } from "../native-runtime/remote-recovery-lease-mismatch.js";
|
||||
@@ -107,7 +108,30 @@ async function chainUsedAutomaticRetry(db: Db, run: { companyId: string; id: str
|
||||
return visited.size >= MAX_RETRY_CHAIN;
|
||||
}
|
||||
|
||||
type Outcome = { requested: true; requestId: string } | { requested: false; reason: string };
|
||||
type PrepareChatRetry = NonNullable<Parameters<typeof requestTaskRetry>[0]["prepareChatRetry"]>;
|
||||
|
||||
/** A connected conversation's Retry goes through the chat service, which the
|
||||
* app registers at startup. Without it, a connected task keeps its Retry. */
|
||||
let interruptedTurnChatRetry: PrepareChatRetry | undefined;
|
||||
export function registerInterruptedTurnChatRetry(prepare: PrepareChatRetry | undefined) {
|
||||
interruptedTurnChatRetry = prepare;
|
||||
}
|
||||
|
||||
/** The chat or comment message that woke the run is still on the task and was
|
||||
* not written by a run, so the run still owes it an answer. */
|
||||
async function wakeMessageStillOwed(db: Db, companyId: string, issueId: string, reason: string | null | undefined,
|
||||
context: Record<string, unknown>, wake: { payload: Record<string, unknown> | null }) {
|
||||
if (reason !== EXTERNAL_CHAT_MESSAGE_WAKE_REASON && !isUserCommentContinuationReason(reason)) return false;
|
||||
const commentId = typeof context.wakeCommentId === "string" ? context.wakeCommentId
|
||||
: typeof wake.payload?.commentId === "string" ? wake.payload.commentId : null;
|
||||
if (!commentId) return false;
|
||||
const [comment] = await db.select({ id: issueComments.id }).from(issueComments).where(and(
|
||||
eq(issueComments.companyId, companyId), eq(issueComments.issueId, issueId),
|
||||
sql`${issueComments.id}::text = ${commentId}`, isNull(issueComments.deletedAt), isNull(issueComments.createdByRunId)));
|
||||
return comment !== undefined;
|
||||
}
|
||||
|
||||
type Outcome ={ requested: true; requestId: string } | { requested: false; reason: string };
|
||||
|
||||
/**
|
||||
* Reattachment runs after a server restart and for a same-run retry of a remote
|
||||
@@ -118,9 +142,9 @@ type Outcome = { requested: true; requestId: string } | { requested: false; reas
|
||||
* proven gone; an in-flight turn's outcome is then unknown and recorded, and the
|
||||
* Retry's fresh session never replays it. Without that proof the in-flight turn
|
||||
* keeps the visible Retry. A system wake continues this way only when the
|
||||
* controller journal proved that no turn was ever issued, or when it was an
|
||||
* automatic retry of owed work whose Runner is proven gone, under the run's
|
||||
* responsible user. Admission keeps every gate, including stop proof and
|
||||
* controller journal proved that no turn was ever issued, or when it carried
|
||||
* owed work (an automatic retry, a system assignment, or a chat message still
|
||||
* on the task) whose Runner is proven gone, under the run's responsible user. Admission keeps every gate, including stop proof and
|
||||
* saved-work recovery before the next turn.
|
||||
*/
|
||||
export async function requestRestartInterruptedTurnRetry(db: Db, claim: NativeRestartRecoveryClaim): Promise<Outcome> {
|
||||
@@ -155,7 +179,12 @@ export async function requestInterruptedNativeTurnRetry(db: Db, runId: string):
|
||||
// ever issued: nothing the run did can be repeated, so the Retry runs once
|
||||
// under the run's responsible user. Missing events alone prove nothing,
|
||||
// because a turn can reach the provider before its events are stored.
|
||||
const owedRedelivery = run.retryOfRunId !== null && runnerExit !== null;
|
||||
// Work the wake still owes the task also continues once the Runner is
|
||||
// proven gone: an automatic retry of earlier owed work, a system
|
||||
// assignment (a routine's dispatch), or a chat message that is still on
|
||||
// the task. Monitors and other system wakes keep their policy.
|
||||
const owedRedelivery = runnerExit !== null && wake !== undefined && (run.retryOfRunId !== null
|
||||
|| reason === "issue_assigned" || await wakeMessageStillOwed(db, run.companyId, issueId, reason, context, wake));
|
||||
if (!wake || reason === "task_retry" || !run.responsibleUserId
|
||||
|| !(owedRedelivery || await hasProvenNoProviderTurn(db, run, runnerExit)))
|
||||
return { requested: false, reason: "not_user_request" };
|
||||
@@ -177,6 +206,11 @@ export async function requestInterruptedNativeTurnRetry(db: Db, runId: string):
|
||||
eq(issueComments.authorUserId, userActorId), isNull(issueComments.deletedAt), isNull(issueComments.createdByRunId),
|
||||
)) : [];
|
||||
if (!comment && reason !== "issue_assigned") return { requested: false, reason: "message_unavailable" };
|
||||
} else if (reason === EXTERNAL_CHAT_MESSAGE_WAKE_REASON) {
|
||||
// A linked person's chat message is stored by the chat bridge, not always
|
||||
// as that person's own comment, so the message itself must still be there.
|
||||
if (!await wakeMessageStillOwed(db, run.companyId, issueId, reason, context, wake!))
|
||||
return { requested: false, reason: "message_unavailable" };
|
||||
} else return { requested: false, reason: "not_user_request" };
|
||||
if (!actorId) return { requested: false, reason: "not_user_request" };
|
||||
const [issue] = await db.select({ status: issues.status, assigneeAgentId: issues.assigneeAgentId }).from(issues)
|
||||
@@ -187,7 +221,8 @@ export async function requestInterruptedNativeTurnRetry(db: Db, runId: string):
|
||||
try {
|
||||
const receipt = await requestTaskRetry({ db, companyId: run.companyId, issueId, actorId,
|
||||
actorSource: actorId === "local-board" ? "local_implicit" : "session",
|
||||
idempotencyKey: `${RESTART_RETRY_KEY_PREFIX}${run.id}`, observedRunId: run.id, initiatedBy: "restart_recovery" });
|
||||
idempotencyKey: `${RESTART_RETRY_KEY_PREFIX}${run.id}`, observedRunId: run.id, initiatedBy: "restart_recovery",
|
||||
prepareChatRetry: interruptedTurnChatRetry });
|
||||
return receipt.accepted && receipt.requestId
|
||||
? { requested: true, requestId: receipt.requestId } : { requested: false, reason: "retry_stale" };
|
||||
} catch (error) {
|
||||
|
||||
@@ -4,10 +4,12 @@ import path from "node:path";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import type { AdapterSandboxExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
|
||||
import { agentWakeupRequests, heartbeatRunEvents, heartbeatRuns } from "@paperclipai/db";
|
||||
import { agentWakeupRequests, chatConversations, chatEndpoints, heartbeatRunEvents, heartbeatRuns, issueComments,
|
||||
toolApplications, toolConnections } from "@paperclipai/db";
|
||||
import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
|
||||
import { heartbeatService } from "./heartbeat.js";
|
||||
import { requestInterruptedNativeTurnRetry, requestRestartInterruptedTurnRetry } from "./heartbeat/restart-interrupted-turn.js";
|
||||
import { registerInterruptedTurnChatRetry, requestInterruptedNativeTurnRetry, requestRestartInterruptedTurnRetry }
|
||||
from "./heartbeat/restart-interrupted-turn.js";
|
||||
import { readSystemRecovery } from "./system-recovery-state.js";
|
||||
import { reconcileRetainedWorkspaces } from "./workspace-recovery.js";
|
||||
import { useWorkspaceRecoveryFixture, action, db, run, seed } from "./__tests__/workspace-recovery-fixture.js";
|
||||
@@ -31,14 +33,24 @@ const stagingRefusal = (reason: "process_not_running" | "process_identity_change
|
||||
describe("a remote Runner that died in the middle of a provider turn", () => {
|
||||
useWorkspaceRecoveryFixture(targets);
|
||||
|
||||
async function runnerLostMidTurn(input: { wake: "user" | "user_retry" | "system_retry" | "system_monitor";
|
||||
async function runnerLostMidTurn(input: { wake: "user" | "user_retry" | "system_retry" | "system_monitor"
|
||||
| "system_assignment" | "system_chat" | "user_chat";
|
||||
refusal?: "process_not_running" | "process_identity_changed" | null;
|
||||
chatMessage?: "present" | "deleted" | "written_by_run";
|
||||
events?: string[]; refusalPayload?: Record<string, unknown> }) {
|
||||
const f = await seed("native");
|
||||
const original = await run(f.runId);
|
||||
const system = input.wake !== "user" && input.wake !== "user_retry";
|
||||
const reason = input.wake === "user" ? "issue_assigned" : input.wake === "user_retry" ? "task_retry"
|
||||
const system = input.wake !== "user" && input.wake !== "user_retry" && input.wake !== "user_chat";
|
||||
const chat = input.wake === "system_chat" || input.wake === "user_chat";
|
||||
const reason = input.wake === "user" || input.wake === "system_assignment" ? "issue_assigned"
|
||||
: input.wake === "user_retry" ? "task_retry"
|
||||
: chat ? "External chat message received"
|
||||
: input.wake === "system_retry" ? "transient_failure_retry" : "issue_monitor_due";
|
||||
// The chat bridge stores the inbound message as a task comment before it wakes the agent.
|
||||
const [chatComment] = chat ? await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId,
|
||||
body: "Can you also check the release notes?", authorType: "system",
|
||||
...(input.chatMessage === "deleted" ? { deletedAt: new Date() } : {}),
|
||||
...(input.chatMessage === "written_by_run" ? { createdByRunId: f.runId } : {}) }).returning() : [];
|
||||
const [retryIntent] = input.wake === "user_retry" ? await db.insert(agentWakeupRequests).values({ companyId: f.companyId,
|
||||
agentId: original.agentId, source: "on_demand", triggerDetail: "manual", reason: "task_retry", status: "completed",
|
||||
idempotencyKey: `task-retry:${randomUUID()}`, requestedByActorType: "user", requestedByActorId: "local-board",
|
||||
@@ -48,13 +60,16 @@ describe("a remote Runner that died in the middle of a provider turn", () => {
|
||||
finishedAt: new Date(Date.now() - 50 * 60_000),
|
||||
errorCode: "environment_provider_unavailable", contextSnapshot: { issueId: f.issueId, wakeReason: "issue_assigned" } }).returning() : [];
|
||||
const [wake] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: original.agentId,
|
||||
source: system ? "automation" : "assignment", triggerDetail: "system", reason, status: "failed", runId: f.runId,
|
||||
source: system && input.wake !== "system_assignment" && !chat ? "automation" : "assignment", triggerDetail: "system",
|
||||
reason, status: "failed", runId: f.runId,
|
||||
requestedByActorType: system ? "system" : "user", requestedByActorId: system ? null : "local-board",
|
||||
finishedAt: new Date(), payload: { issueId: f.issueId } }).returning();
|
||||
finishedAt: new Date(), payload: { issueId: f.issueId, ...(chatComment ? { commentId: chatComment.id } : {}) } }).returning();
|
||||
await db.update(heartbeatRuns).set({ errorCode: "native_continuation_requires_reconciliation",
|
||||
error: "runner_remote_process_identity_unavailable", wakeupRequestId: wake.id, responsibleUserId: "local-board",
|
||||
retryOfRunId: predecessor?.id ?? null, startedAt: new Date(Date.now() - 45 * 60_000),
|
||||
contextSnapshot: { issueId: f.issueId, wakeReason: reason, ...(retryIntent ? { taskRetryRequestId: retryIntent.id,
|
||||
contextSnapshot: { issueId: f.issueId, wakeReason: reason,
|
||||
...(chatComment ? { wakeCommentId: chatComment.id, commentId: chatComment.id, wakeSource: "assignment" } : {}),
|
||||
...(retryIntent ? { taskRetryRequestId: retryIntent.id,
|
||||
forceFreshSession: true } : {}), ...(predecessor ? { retryOfRunId: predecessor.id,
|
||||
retryReason: "transient_failure", scheduledRetryAttempt: 1 } : {}) },
|
||||
}).where(eq(heartbeatRuns.id, f.runId));
|
||||
@@ -85,6 +100,11 @@ describe("a remote Runner that died in the middle of a provider turn", () => {
|
||||
{ title: "a user's assignment", wake: "user" as const, viaClaim: true },
|
||||
{ title: "an automatic retry of the assignment (the staging shape)", wake: "system_retry" as const, viaClaim: true },
|
||||
{ title: "a same-run retry of a user's assignment", wake: "user" as const, viaClaim: false },
|
||||
// Staging after the upgrade fix: a routine's system assignment and an
|
||||
// external chat message, neither a retry, stayed blocked for a person.
|
||||
{ title: "a routine's system assignment", wake: "system_assignment" as const, viaClaim: true },
|
||||
{ title: "an external chat message from an unlinked sender", wake: "system_chat" as const, viaClaim: true },
|
||||
{ title: "an external chat message from a linked person", wake: "user_chat" as const, viaClaim: false },
|
||||
])("continues $title once in a fresh session and records that the turn's outcome is unknown", async ({ wake, viaClaim }) => {
|
||||
const f = await runnerLostMidTurn({ wake });
|
||||
const request = () => viaClaim ? requestRestartInterruptedTurnRetry(db, f.claim) : requestInterruptedNativeTurnRetry(db, f.runId);
|
||||
@@ -150,14 +170,60 @@ describe("a remote Runner that died in the middle of a provider turn", () => {
|
||||
eq(agentWakeupRequests.reason, "task_retry")))).filter(retry => retry.id !== hookIntent.id)).toHaveLength(0);
|
||||
}, 30_000);
|
||||
|
||||
// The staging chat task runs through a connected Slack conversation, whose
|
||||
// endpoint restricts external execution: its Retry goes through the chat service.
|
||||
describe("an external chat message in a connected conversation", () => {
|
||||
async function connectConversation(f: { companyId: string; issueId: string; agentId: string }) {
|
||||
const [application] = await db.insert(toolApplications).values({ companyId: f.companyId, name: "Slack", type: "chat" }).returning();
|
||||
const [connection] = await db.insert(toolConnections).values({ companyId: f.companyId, applicationId: application.id,
|
||||
name: "Slack", uid: randomUUID(), transport: "chat_sdk", connectionPurpose: "channel" }).returning();
|
||||
const [endpoint] = await db.insert(chatEndpoints).values({ companyId: f.companyId, connectionId: connection.id,
|
||||
provider: "slack", publicId: randomUUID(), assignedAgentId: f.agentId, status: "active" }).returning();
|
||||
expect(endpoint.externalExecutionPolicy).toBe("restricted");
|
||||
await db.insert(chatConversations).values({ companyId: f.companyId, endpointId: endpoint.id, issueId: f.issueId,
|
||||
externalConversationId: "channel", externalThreadId: randomUUID(), externalLabel: "#work", state: "waiting" });
|
||||
}
|
||||
|
||||
it("continues through the chat service's Retry", async () => {
|
||||
const f = await runnerLostMidTurn({ wake: "system_chat" });
|
||||
await connectConversation(f);
|
||||
const actionId = randomUUID();
|
||||
const prepared: Array<{ failedRunId: string; initiatedByUserId: string }> = [];
|
||||
registerInterruptedTurnChatRetry(async (_tx, scope) => { prepared.push(scope); return { actionId }; });
|
||||
try {
|
||||
const outcome = await requestRestartInterruptedTurnRetry(db, f.claim);
|
||||
expect(outcome).toEqual({ requested: true, requestId: expect.any(String) });
|
||||
expect(prepared).toEqual([expect.objectContaining({ failedRunId: f.runId, initiatedByUserId: "local-board" })]);
|
||||
const [retry] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id,
|
||||
(outcome as { requestId: string }).requestId));
|
||||
expect(retry.payload).toMatchObject({ taskRetry: { chatActionId: actionId } });
|
||||
expect(await unknownOutcomeEvents(f.runId)).toHaveLength(1);
|
||||
} finally { registerInterruptedTurnChatRetry(undefined); }
|
||||
}, 30_000);
|
||||
|
||||
it("keeps the visible Retry when the chat service is not available", async () => {
|
||||
const f = await runnerLostMidTurn({ wake: "system_chat" });
|
||||
await connectConversation(f);
|
||||
expect(await requestRestartInterruptedTurnRetry(db, f.claim)).toEqual({ requested: false, reason: "retry_refused" });
|
||||
}, 30_000);
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ title: "the refusal did not prove the Runner process gone", wake: "user" as const, refusal: "process_identity_changed" as const,
|
||||
reason: "provider_turn_unsettled" },
|
||||
{ title: "no reattachment refusal was recorded", wake: "user" as const, refusal: null, reason: "provider_turn_unsettled" },
|
||||
{ title: "the source was a monitor wake, not a retry of owed work", wake: "system_monitor" as const, refusal: "process_not_running" as const,
|
||||
reason: "not_user_request" },
|
||||
])("keeps the visible Retry and records nothing when $title", async ({ wake, refusal, reason }) => {
|
||||
const f = await runnerLostMidTurn({ wake, refusal });
|
||||
{ title: "a system chat wake's message was deleted", wake: "system_chat" as const, refusal: "process_not_running" as const,
|
||||
chatMessage: "deleted" as const, reason: "not_user_request" },
|
||||
{ title: "a system chat wake's message was written by a run", wake: "system_chat" as const,
|
||||
refusal: "process_not_running" as const, chatMessage: "written_by_run" as const, reason: "not_user_request" },
|
||||
{ title: "a linked person's chat message was deleted", wake: "user_chat" as const, refusal: "process_not_running" as const,
|
||||
chatMessage: "deleted" as const, reason: "message_unavailable" },
|
||||
{ title: "a system assignment's Runner was not proven gone", wake: "system_assignment" as const,
|
||||
refusal: "process_identity_changed" as const, reason: "provider_turn_unsettled" },
|
||||
])("keeps the visible Retry and records nothing when $title", async ({ wake, refusal, reason, ...rest }) => {
|
||||
const f = await runnerLostMidTurn({ wake, refusal, chatMessage: "chatMessage" in rest ? rest.chatMessage : undefined });
|
||||
expect(await requestRestartInterruptedTurnRetry(db, f.claim)).toEqual({ requested: false, reason });
|
||||
expect(await unknownOutcomeEvents(f.runId)).toHaveLength(0);
|
||||
expect(await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, f.companyId),
|
||||
|
||||
Reference in new issue
Block a user