fix(native): continue a routine assignment or chat message whose remote Runner an upgrade lost mid-turn (#21052)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on remote sandboxes (Daytona) use a detached Runner.
When the server is replaced during an upgrade, the new server reattaches
to that Runner
> - On staging, upgrades every 15–40 minutes lose the in-flight Runner.
The same-run reattachment then finds `process_verification` /
`process_not_running`, the run fails with
`runner_remote_process_identity_unavailable`, and the task is blocked
> - #21039 made these runs continue once in a fresh session (the turn is
recorded as having an unknown outcome and is not replayed). It covers
user wakes and automatic retries (`retryOfRunId`) only
> - A routine's system assignment (`issue_assigned`, requested by the
system) and an external chat message wake have no `retryOfRunId`. They
got `not_user_request` and stayed blocked for a person
> - This pull request treats those two wakes as owed task work, so they
continue the same way once the Runner is proven gone
> - The benefit is that an upgrade no longer leaves these tasks waiting
for a Retry click

## Linked Issues or Issue Description

Refs #21039, #21023, #20923

**What happened?**
Staging replaced its server at about 07:10Z. 17 remote native runs were
in flight. Each new controller reached its Runner 114–152 s after
detach. Recovery failed with "PRP command session.snapshot timed out".
The same-run reattachment then found the Runner process gone, and each
run failed with `native_continuation_requires_reconciliation`. #21039
continued 14 of the 17 automatically. The other 3 were a routine's
assignment (two runs) and an external chat message. They have no
`retryOfRunId`, so `requestInterruptedNativeTurnRetry` returned
`not_user_request`. Those tasks are still blocked with no retry.

**What did you expect to happen?**
The Runner is proven gone, so nothing can still run the turn. The run
should continue once in a fresh session, the same as an automatic retry
of owed work.

**Steps to reproduce**
1. Start a remote native run from a routine assignment, or from an
external chat message.
2. Replace the server while the provider turn is active, so that the
reattachment finds `process_not_running`.
3. The run fails and the task stays blocked. No automatic Retry is
requested.

## What Changed

- `server/src/services/heartbeat/restart-interrupted-turn.ts`: after the
Runner is proven gone, a system wake also continues when it is a system
assignment (`issue_assigned`), or when it is a comment or external chat
message wake whose message is still on the task (not deleted, not
written by a run). It runs under the run's responsible user, at most
once per chain of runs, like an automatic retry already does. A linked
person's external chat message continues under that person when the
message is still on the task. Monitors and other system wakes are
unchanged.
- The same file and `server/src/app.ts`: the hook now passes the chat
service's Retry callback (`prepareFailedChatRunRetry`, registered at app
startup next to the budget-resume callback). A task in a connected
conversation whose endpoint restricts external execution then continues
through the chat Retry path. Before, `requestTaskRetry` refused it
(`retry_refused`) and the task stayed blocked.
- `server/src/services/runner-lost-mid-turn-continuation.test.ts`:
production-shape cases for a system assignment, an unlinked external
chat sender, and a linked person's chat message. Negative cases: a
deleted message, a message written by a run, a deleted linked message,
and a system assignment whose Runner was not proven gone.
- `doc/execution-semantics.md`: describes the rule.

## Verification

- Red on `origin/master` with only the new tests: `cd server && npx
vitest run src/services/runner-lost-mid-turn-continuation.test.ts` fails
4 of 16. The three new continuation cases return `{ requested: false }`.
The deleted linked-message case returns `not_user_request` instead of
`message_unavailable`.
- Green with the fix: 16 of 16. Two more cases bind the task to a real
restricted Slack endpoint and conversation. With the chat callback
registered, the run continues and the Retry carries the chat action (18
of 18). Without the callback, the result is `retry_refused`, which was
the behavior before for every connected conversation.
- Related suites pass: `runner-lost-mid-turn-continuation.test.ts`,
`heartbeat-native-remote-failure-retention.test.ts`,
`workspace-recovery-retry-copyback.test.ts`,
`governed-wait-runner-loss.test.ts` (116 tests).
- `cd server && npx tsc --noEmit` passes.
- Production evidence was read from staging run events through the
read-only API: 3 of 17 losses at the 07:10Z swap took this path. After
this change the next post-upgrade audit should show an automatic
`task_retry` for every `native_continuation_requires_reconciliation` run
whose refusal is `process_not_running`, including routine assignments
and chat messages.

## Risks

- Low. The change applies only after the Runner process is proven gone
(`process_verification` / `process_not_running`), so no Runner can still
run the turn. The turn is not replayed. The fresh session continues over
the saved work, as in #21039.
- The task can contain partial tool work from the lost turn. This is the
same accepted risk as #21039.
- Previous-release story: no persisted shape changes. The rule reads the
wake's reason and comment id, which earlier releases already write. A
run an older release already settled keeps its visible Retry.
- This does not stop the Runner loss itself. That is a separate fix.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic) via Claude Code, with tool use (shell, file edits,
test runs, read-only staging API reads). Model ID as attributed in the
commit trailer: Claude Fable 5.2.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Fable 5.2 authored and GitHub committed 2026-11-12 02:11:28 -06:00
1 parent 4e519bd428
commit fd5ee7e385
4 files changed
+130 -22

No files matched your search

+10 -6
View File
@@ -1771,7 +1771,7 @@ turn is never replayed. Activity records the request
as `restart-recovery` on behalf of that user. Admission keeps every gate: saved-work
recovery, stop proof, approvals, pauses, budget and ownership. A Retry that the
server requested gets no second automatic attempt. A system wake (for example a
monitor or an automatic assignment) continues this way only with a positive
monitor) continues this way only with a positive
record that no turn was ever issued: the refused reattachment proved the Runner
process gone, and the run's controller journal, which records each turn command
before it is sent, holds no turn command and no turn event
@@ -1779,11 +1779,15 @@ before it is sent, holds no turn command and no turn event
must also have no turn or tool event, and the Retry runs under the run's
responsible user. Missing events alone never qualify, because a turn can reach
the provider before its events are stored. An automatic retry of owed work (a
system wake whose run has `retryOfRunId`, such as `transient_failure_retry`)
also continues under the run's responsible user once the refusal proved its
Runner process gone, because losing the Runner to a restart is not the task's
failure. Other system wakes, and in-flight provider turns without that proof,
do not continue automatically. For these failed runs, a
system wake whose run has `retryOfRunId`, such as `transient_failure_retry`, a
system assignment such as a routine's dispatch (`issue_assigned`), or a comment
or external chat message wake whose message is still on the task and was not
written by a run) also continues under the run's responsible user once the
refusal proved its Runner process gone, because losing the Runner to a restart
is not the task's failure. A linked person's external chat message continues
under that person once the message is still on the task. Monitors and other
system wakes, and in-flight provider turns without that proof, do not continue
automatically. Each chain of runs gets at most one such automatic Retry. For these failed runs, a
recovered workspace hold offers Retry (`canRetry`) and says "Retry or send a
message to continue." An intentional Stop keeps "send a message" as its only
continuation.
+3
View File
@@ -226,6 +226,7 @@ import { chatInboundWakeNoticeWork } from "./services/chat-inbound-wake-notice-w
import { chatFailedRetryNoticeWork } from "./services/chat-failed-retry-notice-work.js";
import { chatRunMilestoneWork } from "./services/chat-run-milestone-work.js";
import { heartbeatService } from "./services/heartbeat.js";
import { registerInterruptedTurnChatRetry } from "./services/heartbeat/restart-interrupted-turn.js";
import { pluginLifecycleManager } from "./services/plugin-lifecycle.js";
import { registerPluginLifecycleDelivery } from "./services/plugin-lifecycle-delivery.js";
import { createPluginJobCoordinator } from "./services/plugin-job-coordinator.js";
@@ -603,6 +604,8 @@ export async function createApp(
storage: opts.storageService,
});
if (opts.budgetResumeDelivery) opts.budgetResumeDelivery.prepareChatRetry = chatChannels.prepareFailedChatRunRetry;
// A restart's automatic Retry of a connected conversation needs the same callback.
registerInterruptedTurnChatRetry(chatChannels.prepareFailedChatRunRetry);
// Voice capabilities are verified by the signed callback route, not by
// board/agent bearer authentication. Mount before actorMiddleware so its
// per-session Bearer cannot be mistaken for an agent API key.
@@ -4,6 +4,7 @@ import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js";
import type { NativeRestartRecoveryClaim } from "../native-runtime/native-restart-recovery.js";
import { isUserCommentContinuationReason } from "../explicit-native-continuation.js";
import { requestTaskRetry } from "../task-retry.js";
import { EXTERNAL_CHAT_MESSAGE_WAKE_REASON } from "../queued-comment-delivery.js";
import { HttpError } from "../../errors.js";
import { logger } from "../../middleware/logger.js";
import { NATIVE_REMOTE_RECOVERY_LEASE_MISMATCH } from "../native-runtime/remote-recovery-lease-mismatch.js";
@@ -107,7 +108,30 @@ async function chainUsedAutomaticRetry(db: Db, run: { companyId: string; id: str
return visited.size >= MAX_RETRY_CHAIN;
}
type Outcome = { requested: true; requestId: string } | { requested: false; reason: string };
type PrepareChatRetry = NonNullable<Parameters<typeof requestTaskRetry>[0]["prepareChatRetry"]>;
/** A connected conversation's Retry goes through the chat service, which the
* app registers at startup. Without it, a connected task keeps its Retry. */
let interruptedTurnChatRetry: PrepareChatRetry | undefined;
export function registerInterruptedTurnChatRetry(prepare: PrepareChatRetry | undefined) {
interruptedTurnChatRetry = prepare;
}
/** The chat or comment message that woke the run is still on the task and was
* not written by a run, so the run still owes it an answer. */
async function wakeMessageStillOwed(db: Db, companyId: string, issueId: string, reason: string | null | undefined,
context: Record<string, unknown>, wake: { payload: Record<string, unknown> | null }) {
if (reason !== EXTERNAL_CHAT_MESSAGE_WAKE_REASON && !isUserCommentContinuationReason(reason)) return false;
const commentId = typeof context.wakeCommentId === "string" ? context.wakeCommentId
: typeof wake.payload?.commentId === "string" ? wake.payload.commentId : null;
if (!commentId) return false;
const [comment] = await db.select({ id: issueComments.id }).from(issueComments).where(and(
eq(issueComments.companyId, companyId), eq(issueComments.issueId, issueId),
sql`${issueComments.id}::text = ${commentId}`, isNull(issueComments.deletedAt), isNull(issueComments.createdByRunId)));
return comment !== undefined;
}
type Outcome ={ requested: true; requestId: string } | { requested: false; reason: string };
/**
* Reattachment runs after a server restart and for a same-run retry of a remote
@@ -118,9 +142,9 @@ type Outcome = { requested: true; requestId: string } | { requested: false; reas
* proven gone; an in-flight turn's outcome is then unknown and recorded, and the
* Retry's fresh session never replays it. Without that proof the in-flight turn
* keeps the visible Retry. A system wake continues this way only when the
* controller journal proved that no turn was ever issued, or when it was an
* automatic retry of owed work whose Runner is proven gone, under the run's
* responsible user. Admission keeps every gate, including stop proof and
* controller journal proved that no turn was ever issued, or when it carried
* owed work (an automatic retry, a system assignment, or a chat message still
* on the task) whose Runner is proven gone, under the run's responsible user. Admission keeps every gate, including stop proof and
* saved-work recovery before the next turn.
*/
export async function requestRestartInterruptedTurnRetry(db: Db, claim: NativeRestartRecoveryClaim): Promise<Outcome> {
@@ -155,7 +179,12 @@ export async function requestInterruptedNativeTurnRetry(db: Db, runId: string):
// ever issued: nothing the run did can be repeated, so the Retry runs once
// under the run's responsible user. Missing events alone prove nothing,
// because a turn can reach the provider before its events are stored.
const owedRedelivery = run.retryOfRunId !== null && runnerExit !== null;
// Work the wake still owes the task also continues once the Runner is
// proven gone: an automatic retry of earlier owed work, a system
// assignment (a routine's dispatch), or a chat message that is still on
// the task. Monitors and other system wakes keep their policy.
const owedRedelivery = runnerExit !== null && wake !== undefined && (run.retryOfRunId !== null
|| reason === "issue_assigned" || await wakeMessageStillOwed(db, run.companyId, issueId, reason, context, wake));
if (!wake || reason === "task_retry" || !run.responsibleUserId
|| !(owedRedelivery || await hasProvenNoProviderTurn(db, run, runnerExit)))
return { requested: false, reason: "not_user_request" };
@@ -177,6 +206,11 @@ export async function requestInterruptedNativeTurnRetry(db: Db, runId: string):
eq(issueComments.authorUserId, userActorId), isNull(issueComments.deletedAt), isNull(issueComments.createdByRunId),
)) : [];
if (!comment && reason !== "issue_assigned") return { requested: false, reason: "message_unavailable" };
} else if (reason === EXTERNAL_CHAT_MESSAGE_WAKE_REASON) {
// A linked person's chat message is stored by the chat bridge, not always
// as that person's own comment, so the message itself must still be there.
if (!await wakeMessageStillOwed(db, run.companyId, issueId, reason, context, wake!))
return { requested: false, reason: "message_unavailable" };
} else return { requested: false, reason: "not_user_request" };
if (!actorId) return { requested: false, reason: "not_user_request" };
const [issue] = await db.select({ status: issues.status, assigneeAgentId: issues.assigneeAgentId }).from(issues)
@@ -187,7 +221,8 @@ export async function requestInterruptedNativeTurnRetry(db: Db, runId: string):
try {
const receipt = await requestTaskRetry({ db, companyId: run.companyId, issueId, actorId,
actorSource: actorId === "local-board" ? "local_implicit" : "session",
idempotencyKey: `${RESTART_RETRY_KEY_PREFIX}${run.id}`, observedRunId: run.id, initiatedBy: "restart_recovery" });
idempotencyKey: `${RESTART_RETRY_KEY_PREFIX}${run.id}`, observedRunId: run.id, initiatedBy: "restart_recovery",
prepareChatRetry: interruptedTurnChatRetry });
return receipt.accepted && receipt.requestId
? { requested: true, requestId: receipt.requestId } : { requested: false, reason: "retry_stale" };
} catch (error) {
@@ -4,10 +4,12 @@ import path from "node:path";
import { and, eq } from "drizzle-orm";
import { describe, expect, it, vi } from "vitest";
import type { AdapterSandboxExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
import { agentWakeupRequests, heartbeatRunEvents, heartbeatRuns } from "@paperclipai/db";
import { agentWakeupRequests, chatConversations, chatEndpoints, heartbeatRunEvents, heartbeatRuns, issueComments,
toolApplications, toolConnections } from "@paperclipai/db";
import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
import { heartbeatService } from "./heartbeat.js";
import { requestInterruptedNativeTurnRetry, requestRestartInterruptedTurnRetry } from "./heartbeat/restart-interrupted-turn.js";
import { registerInterruptedTurnChatRetry, requestInterruptedNativeTurnRetry, requestRestartInterruptedTurnRetry }
from "./heartbeat/restart-interrupted-turn.js";
import { readSystemRecovery } from "./system-recovery-state.js";
import { reconcileRetainedWorkspaces } from "./workspace-recovery.js";
import { useWorkspaceRecoveryFixture, action, db, run, seed } from "./__tests__/workspace-recovery-fixture.js";
@@ -31,14 +33,24 @@ const stagingRefusal = (reason: "process_not_running" | "process_identity_change
describe("a remote Runner that died in the middle of a provider turn", () => {
useWorkspaceRecoveryFixture(targets);
async function runnerLostMidTurn(input: { wake: "user" | "user_retry" | "system_retry" | "system_monitor";
async function runnerLostMidTurn(input: { wake: "user" | "user_retry" | "system_retry" | "system_monitor"
| "system_assignment" | "system_chat" | "user_chat";
refusal?: "process_not_running" | "process_identity_changed" | null;
chatMessage?: "present" | "deleted" | "written_by_run";
events?: string[]; refusalPayload?: Record<string, unknown> }) {
const f = await seed("native");
const original = await run(f.runId);
const system = input.wake !== "user" && input.wake !== "user_retry";
const reason = input.wake === "user" ? "issue_assigned" : input.wake === "user_retry" ? "task_retry"
const system = input.wake !== "user" && input.wake !== "user_retry" && input.wake !== "user_chat";
const chat = input.wake === "system_chat" || input.wake === "user_chat";
const reason = input.wake === "user" || input.wake === "system_assignment" ? "issue_assigned"
: input.wake === "user_retry" ? "task_retry"
: chat ? "External chat message received"
: input.wake === "system_retry" ? "transient_failure_retry" : "issue_monitor_due";
// The chat bridge stores the inbound message as a task comment before it wakes the agent.
const [chatComment] = chat ? await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId,
body: "Can you also check the release notes?", authorType: "system",
...(input.chatMessage === "deleted" ? { deletedAt: new Date() } : {}),
...(input.chatMessage === "written_by_run" ? { createdByRunId: f.runId } : {}) }).returning() : [];
const [retryIntent] = input.wake === "user_retry" ? await db.insert(agentWakeupRequests).values({ companyId: f.companyId,
agentId: original.agentId, source: "on_demand", triggerDetail: "manual", reason: "task_retry", status: "completed",
idempotencyKey: `task-retry:${randomUUID()}`, requestedByActorType: "user", requestedByActorId: "local-board",
@@ -48,13 +60,16 @@ describe("a remote Runner that died in the middle of a provider turn", () => {
finishedAt: new Date(Date.now() - 50 * 60_000),
errorCode: "environment_provider_unavailable", contextSnapshot: { issueId: f.issueId, wakeReason: "issue_assigned" } }).returning() : [];
const [wake] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: original.agentId,
source: system ? "automation" : "assignment", triggerDetail: "system", reason, status: "failed", runId: f.runId,
source: system && input.wake !== "system_assignment" && !chat ? "automation" : "assignment", triggerDetail: "system",
reason, status: "failed", runId: f.runId,
requestedByActorType: system ? "system" : "user", requestedByActorId: system ? null : "local-board",
finishedAt: new Date(), payload: { issueId: f.issueId } }).returning();
finishedAt: new Date(), payload: { issueId: f.issueId, ...(chatComment ? { commentId: chatComment.id } : {}) } }).returning();
await db.update(heartbeatRuns).set({ errorCode: "native_continuation_requires_reconciliation",
error: "runner_remote_process_identity_unavailable", wakeupRequestId: wake.id, responsibleUserId: "local-board",
retryOfRunId: predecessor?.id ?? null, startedAt: new Date(Date.now() - 45 * 60_000),
contextSnapshot: { issueId: f.issueId, wakeReason: reason, ...(retryIntent ? { taskRetryRequestId: retryIntent.id,
contextSnapshot: { issueId: f.issueId, wakeReason: reason,
...(chatComment ? { wakeCommentId: chatComment.id, commentId: chatComment.id, wakeSource: "assignment" } : {}),
...(retryIntent ? { taskRetryRequestId: retryIntent.id,
forceFreshSession: true } : {}), ...(predecessor ? { retryOfRunId: predecessor.id,
retryReason: "transient_failure", scheduledRetryAttempt: 1 } : {}) },
}).where(eq(heartbeatRuns.id, f.runId));
@@ -85,6 +100,11 @@ describe("a remote Runner that died in the middle of a provider turn", () => {
{ title: "a user's assignment", wake: "user" as const, viaClaim: true },
{ title: "an automatic retry of the assignment (the staging shape)", wake: "system_retry" as const, viaClaim: true },
{ title: "a same-run retry of a user's assignment", wake: "user" as const, viaClaim: false },
// Staging after the upgrade fix: a routine's system assignment and an
// external chat message, neither a retry, stayed blocked for a person.
{ title: "a routine's system assignment", wake: "system_assignment" as const, viaClaim: true },
{ title: "an external chat message from an unlinked sender", wake: "system_chat" as const, viaClaim: true },
{ title: "an external chat message from a linked person", wake: "user_chat" as const, viaClaim: false },
])("continues $title once in a fresh session and records that the turn's outcome is unknown", async ({ wake, viaClaim }) => {
const f = await runnerLostMidTurn({ wake });
const request = () => viaClaim ? requestRestartInterruptedTurnRetry(db, f.claim) : requestInterruptedNativeTurnRetry(db, f.runId);
@@ -150,14 +170,60 @@ describe("a remote Runner that died in the middle of a provider turn", () => {
eq(agentWakeupRequests.reason, "task_retry")))).filter(retry => retry.id !== hookIntent.id)).toHaveLength(0);
}, 30_000);
// The staging chat task runs through a connected Slack conversation, whose
// endpoint restricts external execution: its Retry goes through the chat service.
describe("an external chat message in a connected conversation", () => {
async function connectConversation(f: { companyId: string; issueId: string; agentId: string }) {
const [application] = await db.insert(toolApplications).values({ companyId: f.companyId, name: "Slack", type: "chat" }).returning();
const [connection] = await db.insert(toolConnections).values({ companyId: f.companyId, applicationId: application.id,
name: "Slack", uid: randomUUID(), transport: "chat_sdk", connectionPurpose: "channel" }).returning();
const [endpoint] = await db.insert(chatEndpoints).values({ companyId: f.companyId, connectionId: connection.id,
provider: "slack", publicId: randomUUID(), assignedAgentId: f.agentId, status: "active" }).returning();
expect(endpoint.externalExecutionPolicy).toBe("restricted");
await db.insert(chatConversations).values({ companyId: f.companyId, endpointId: endpoint.id, issueId: f.issueId,
externalConversationId: "channel", externalThreadId: randomUUID(), externalLabel: "#work", state: "waiting" });
}
it("continues through the chat service's Retry", async () => {
const f = await runnerLostMidTurn({ wake: "system_chat" });
await connectConversation(f);
const actionId = randomUUID();
const prepared: Array<{ failedRunId: string; initiatedByUserId: string }> = [];
registerInterruptedTurnChatRetry(async (_tx, scope) => { prepared.push(scope); return { actionId }; });
try {
const outcome = await requestRestartInterruptedTurnRetry(db, f.claim);
expect(outcome).toEqual({ requested: true, requestId: expect.any(String) });
expect(prepared).toEqual([expect.objectContaining({ failedRunId: f.runId, initiatedByUserId: "local-board" })]);
const [retry] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id,
(outcome as { requestId: string }).requestId));
expect(retry.payload).toMatchObject({ taskRetry: { chatActionId: actionId } });
expect(await unknownOutcomeEvents(f.runId)).toHaveLength(1);
} finally { registerInterruptedTurnChatRetry(undefined); }
}, 30_000);
it("keeps the visible Retry when the chat service is not available", async () => {
const f = await runnerLostMidTurn({ wake: "system_chat" });
await connectConversation(f);
expect(await requestRestartInterruptedTurnRetry(db, f.claim)).toEqual({ requested: false, reason: "retry_refused" });
}, 30_000);
});
it.each([
{ title: "the refusal did not prove the Runner process gone", wake: "user" as const, refusal: "process_identity_changed" as const,
reason: "provider_turn_unsettled" },
{ title: "no reattachment refusal was recorded", wake: "user" as const, refusal: null, reason: "provider_turn_unsettled" },
{ title: "the source was a monitor wake, not a retry of owed work", wake: "system_monitor" as const, refusal: "process_not_running" as const,
reason: "not_user_request" },
])("keeps the visible Retry and records nothing when $title", async ({ wake, refusal, reason }) => {
const f = await runnerLostMidTurn({ wake, refusal });
{ title: "a system chat wake's message was deleted", wake: "system_chat" as const, refusal: "process_not_running" as const,
chatMessage: "deleted" as const, reason: "not_user_request" },
{ title: "a system chat wake's message was written by a run", wake: "system_chat" as const,
refusal: "process_not_running" as const, chatMessage: "written_by_run" as const, reason: "not_user_request" },
{ title: "a linked person's chat message was deleted", wake: "user_chat" as const, refusal: "process_not_running" as const,
chatMessage: "deleted" as const, reason: "message_unavailable" },
{ title: "a system assignment's Runner was not proven gone", wake: "system_assignment" as const,
refusal: "process_identity_changed" as const, reason: "provider_turn_unsettled" },
])("keeps the visible Retry and records nothing when $title", async ({ wake, refusal, reason, ...rest }) => {
const f = await runnerLostMidTurn({ wake, refusal, chatMessage: "chatMessage" in rest ? rest.chatMessage : undefined });
expect(await requestRestartInterruptedTurnRetry(db, f.claim)).toEqual({ requested: false, reason });
expect(await unknownOutcomeEvents(f.runId)).toHaveLength(0);
expect(await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, f.companyId),