From 8eabb13ebdb6868afd5e159350fe0660b92d6e4a Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 11 Nov 2026 12:22:51 -0600 Subject: [PATCH] fix(recovery): retry a system-woken run once when it lost its runner before any turn (#20873) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run on a remote sandbox can lose its Runner process before its same-run retry reattaches. The run then fails with `runner_remote_process_identity_unavailable` > - After saved-work recovery, the server requests one Retry by itself, but only for user wakes. System wakes (monitors, automatic assignments) rest on the visible Retry > - On staging, a monitor wake's run lost its Runner before it started any provider turn. Nothing it did could be repeated, but the task stayed blocked until a person clicked Retry > - This pull request lets that one case continue automatically, once, under the run's responsible user > - The benefit is that a system-woken task whose run never reached a provider turn continues without a person ## Linked Issues or Issue Description **What happened?** On staging, a monitor wake (`issue_monitor_due`) started a remote native run. Its `session.snapshot` command timed out before any turn. The same-run retry found the Runner process gone (`process_not_running`, lifecycle `ready`) and the run failed with `runner_remote_process_identity_unavailable`. The run had no turn event and no tool event. Saved-work recovery recovered the files, but `requestInterruptedNativeTurnRetry` refused with `not_user_request`, so the task stayed blocked on a manual Retry. **What did you expect to happen?** When the run proves that it did no provider turn, the server requests the Retry once by itself, as it does for user wakes. **Steps to reproduce** 1. A system wake starts a remote native run. 2. The run fails before any turn event, and the same-run retry finds the Runner gone. 3. Saved-work recovery completes. No Retry is requested; the task waits for a person. Refs #20872, #20853 ## What Changed The rule that changed, exactly: in `requestInterruptedNativeTurnRetry` (`server/src/services/heartbeat/restart-interrupted-turn.ts`), a run whose wake was not requested by a user used to stop with `not_user_request` in all cases. Now it requests the task Retry only when all of these are true: - the run failed with one of the existing reattachment-loss codes (unchanged gate); - a positive durable record proves that no turn was ever issued: the run's last `native.remote_runner_recovery_failed` event carries `remoteRunnerRecovery` with `stage: process_verification`, `reason: process_not_running` and `providerTurnIssued: false`; - the run also recorded no turn event and no tool event; - the run has a `responsibleUserId`; the Retry runs under that user; - the run is not itself a `task_retry` run; - the task is still open and assigned to the same agent (unchanged gate). An absence of run events is never enough: a turn can reach the provider before its events are stored (Greptile, first review). - New `withRemoteRunnerTurnProof` (`native-session-executor.ts`): when a same-run reattachment is refused because the sandbox proved the Runner process gone (exit 45, not an unreachable sandbox), the executor reads the run's controller journal. The controller records each turn command there before it sends it. Only when `runnerdStateProvesNoRemoteProviderTurn` holds (just `run.prepare`/`session.open` and maintenance commands such as `session.snapshot`, and only session-open events) does it add `providerTurnIssued: false` to the diagnostic. This is the same proof that restart recovery already uses for `bootstrap_incomplete`. `RemoteRunnerRecoveryDiagnostic` keeps the field only as `false`. - Which staging shapes this covers: a run whose journal holds no turn command. Whether c14bab19 (the `session.snapshot` timeout) qualifies depends on its controller journal, which staging does not expose; if its journal held a `turn.start`, it stays on the manual Retry floor. 32376b3a (`turn.submitted`, then a `turn.start` timeout) always stays on the floor. - Everything else about system wakes is unchanged: without the journal proof, with any turn or tool event, or without a responsible user, a system wake still rests on the visible Retry. - Bounded: the Retry uses the existing idempotency key `restart-interrupted-turn:`, so it is requested at most once per run. Its successor is a user-attributed `task_retry` run, so if that run also loses its Runner, the existing `automatic_retry_used` rule leaves it on the visible Retry. No loop is possible. - Attribution: `requestTaskRetry` records `issue.retry_requested` with actor `restart-recovery` on behalf of the run's responsible user. Admission keeps every gate (saved-work recovery, stop proof, approvals, pause, budget, ownership). - Tests in `server/src/services/workspace-recovery-retry-copyback.test.ts`: a positive case (monitor wake, Runner proven gone, journal proof), including the second-loss `automatic_retry_used` check. Negative cases: no run events but no journal proof (the Greptile shape), a settled turn, a tool event, no responsible user. - New `server/src/services/native-runtime/remote-runner-turn-proof.test.ts`: the proof is recorded only for a journal of session commands; not for a `turn.start` command, a committed turn event, an unreadable journal, or an unreachable sandbox (`probe_transport_failed`). - `doc/execution-semantics.md`: describes the rule. ## Verification - `cd server && pnpm exec vitest run src/services/workspace-recovery-retry-copyback.test.ts`: 83 passed. - Red on master (product hunk removed): the positive case fails with `expected { requested: false, … } to match object { requested: true }`. - Red on the first head (a244ad3b): the "no run events but no journal proof" case is requested automatically; CI's `run-completion.test.ts` ("replays saved user intent … system_wake") failed for the same reason. Both pass now. - `pnpm exec vitest run src/services/native-runtime/remote-runner-turn-proof.test.ts`: 4 passed. `native-session-executor.test.ts`, `run-completion.test.ts`, `runtime-execution.test.ts`: 1571 tests passed (one local `afterAll` hook timeout under load). - `cd server && pnpm exec tsc --noEmit -p tsconfig.json`: passes. ## Risks - Low to medium. A system-woken task can now start one new run without a person. This happens only when the controller journal proves that no turn command was ever issued, so no provider work or tool side effect from that run can be repeated. The Retry starts a fresh provider session, like a manual Retry. - The Retry runs with the run's responsible user's authority. That user already authorized the failed run. If the user lost access, `requestTaskRetry` refuses and the visible Retry remains. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in an agent harness. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/execution-semantics.md | 10 +++- .../heartbeat/restart-interrupted-turn.ts | 46 +++++++++++++--- .../native-runtime/native-session-executor.ts | 28 ++++++++-- .../remote-runner-recovery-diagnostics.ts | 5 +- .../remote-runner-turn-proof.test.ts | 52 +++++++++++++++++++ .../workspace-recovery-retry-copyback.test.ts | 37 ++++++++++--- 6 files changed, 158 insertions(+), 20 deletions(-) create mode 100644 server/src/services/native-runtime/remote-runner-turn-proof.test.ts diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 6f1334faebd..f86cdf04cf0 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -1725,7 +1725,15 @@ assigned the task, or clicked Retry. It does this only when the run recorded no provider turn or its last provider turn is settled. Activity records the request as `restart-recovery` on behalf of that user. Admission keeps every gate: saved-work recovery, stop proof, approvals, pauses, budget and ownership. A Retry that the -server requested gets no second automatic attempt. System wakes and runs with an +server requested gets no second automatic attempt. A system wake (for example a +monitor or an automatic assignment) continues this way only with a positive +record that no turn was ever issued: the refused reattachment proved the Runner +process gone, and the run's controller journal, which records each turn command +before it is sent, holds no turn command and no turn event +(`providerTurnIssued: false` on `native.remote_runner_recovery_failed`). The run +must also have no turn or tool event, and the Retry runs under the run's +responsible user. Missing events alone never qualify, because a turn can reach +the provider before its events are stored. Other system wakes and runs with an in-flight provider turn do not continue automatically. For these failed runs, a recovered workspace hold offers Retry (`canRetry`) and says "Retry or send a message to continue." An intentional Stop keeps "send a message" as its only diff --git a/server/src/services/heartbeat/restart-interrupted-turn.ts b/server/src/services/heartbeat/restart-interrupted-turn.ts index a1458a27826..16564be7238 100644 --- a/server/src/services/heartbeat/restart-interrupted-turn.ts +++ b/server/src/services/heartbeat/restart-interrupted-turn.ts @@ -36,6 +36,29 @@ export function isProvenTurnSettlement(event: { eventType: string; payload: unkn asRecord(asRecord(payload.prpEvent).payload).providerTerminalObserved !== false; } +/** Any turn or tool event: the run may have done provider work that a new turn could repeat. */ +const PROVIDER_TURN_ACTIVITY_EVENTS = [...TURN_STARTED_EVENTS, ...TURN_SETTLED_EVENTS, + "tool.execution.started", "tool.execution.completed"]; + +/** + * The run's last refused reattachment proved its Runner process gone and its + * controller journal proved that no turn command was ever issued + * (`providerTurnIssued: false`, recorded by the executor). No turn or tool + * event may exist either. + */ +async function hasProvenNoProviderTurn(db: Db, run: { companyId: string; id: string }) { + const [activity] = await db.select({ id: heartbeatRunEvents.id }).from(heartbeatRunEvents).where(and( + eq(heartbeatRunEvents.companyId, run.companyId), eq(heartbeatRunEvents.runId, run.id), + inArray(heartbeatRunEvents.eventType, PROVIDER_TURN_ACTIVITY_EVENTS))).limit(1); + if (activity) return false; + const [refusal] = await db.select({ payload: heartbeatRunEvents.payload }).from(heartbeatRunEvents).where(and( + eq(heartbeatRunEvents.companyId, run.companyId), eq(heartbeatRunEvents.runId, run.id), + eq(heartbeatRunEvents.eventType, "native.remote_runner_recovery_failed"), + )).orderBy(desc(heartbeatRunEvents.seq)).limit(1); + const proof = asRecord(asRecord(refusal?.payload).remoteRunnerRecovery); + return proof.stage === "process_verification" && proof.reason === "process_not_running" && proof.providerTurnIssued === false; +} + /** Automatic continuation is requested at most once per chain of runs. */ export const RESTART_RETRY_KEY_PREFIX = "restart-interrupted-turn:"; @@ -47,7 +70,9 @@ type Outcome = { requested: true; requestId: string } | { requested: false; reas * request the same Retry a user would click, once and idempotently per run, * under the requesting user's authority. Only when the run's user input was not * yet acted on, or its last provider turn is settled; an in-flight turn's - * outcome is unknown, so it keeps the visible Retry. Admission keeps every gate, + * outcome is unknown, so it keeps the visible Retry. A system wake continues + * this way only when the controller journal proved that no turn was ever + * issued, under the run's responsible user. Admission keeps every gate, * including stop proof and saved-work recovery before the next turn. */ export async function requestRestartInterruptedTurnRetry(db: Db, claim: NativeRestartRecoveryClaim): Promise { @@ -71,11 +96,19 @@ export async function requestInterruptedNativeTurnRetry(db: Db, runId: string): const [wake] = await db.select().from(agentWakeupRequests).where(and( eq(agentWakeupRequests.companyId, run.companyId), eq(agentWakeupRequests.agentId, run.agentId), eq(agentWakeupRequests.id, run.wakeupRequestId), eq(agentWakeupRequests.runId, run.id))); - const actorId = wake?.requestedByActorType === "user" ? wake.requestedByActorId : null; + const userActorId = wake?.requestedByActorType === "user" ? wake.requestedByActorId : null; const reason = typeof context.wakeReason === "string" ? context.wakeReason : wake?.reason; - // System wakes keep their policy: only a new user turn continues them. - if (!actorId) return { requested: false, reason: "not_user_request" }; - if (reason === "task_retry") { + let actorId = userActorId; + if (!userActorId) { + // System wakes keep their policy (only a new user turn continues them), + // except when the controller journal positively proves that no turn was + // ever issued: nothing the run did can be repeated, so the Retry runs once + // under the run's responsible user. Missing events alone prove nothing, + // because a turn can reach the provider before its events are stored. + if (!wake || reason === "task_retry" || !run.responsibleUserId || !await hasProvenNoProviderTurn(db, run)) + return { requested: false, reason: "not_user_request" }; + actorId = run.responsibleUserId; + } else if (reason === "task_retry") { // A Retry this hook already requested gets no second automatic attempt. const retryId = typeof context.taskRetryRequestId === "string" ? context.taskRetryRequestId : null; const [receipt] = retryId ? await db.select({ key: agentWakeupRequests.idempotencyKey }).from(agentWakeupRequests).where(and( @@ -88,10 +121,11 @@ export async function requestInterruptedNativeTurnRetry(db: Db, runId: string): const [comment] = commentId ? await db.select({ id: issueComments.id }).from(issueComments).where(and( eq(issueComments.companyId, run.companyId), eq(issueComments.issueId, issueId), sql`${issueComments.id}::text = ${commentId}`, eq(issueComments.authorType, "user"), - eq(issueComments.authorUserId, actorId), isNull(issueComments.deletedAt), isNull(issueComments.createdByRunId), + eq(issueComments.authorUserId, userActorId), isNull(issueComments.deletedAt), isNull(issueComments.createdByRunId), )) : []; if (!comment && reason !== "issue_assigned") return { requested: false, reason: "message_unavailable" }; } else return { requested: false, reason: "not_user_request" }; + if (!actorId) return { requested: false, reason: "not_user_request" }; const [issue] = await db.select({ status: issues.status, assigneeAgentId: issues.assigneeAgentId }).from(issues) .where(and(eq(issues.companyId, run.companyId), eq(issues.id, issueId))); if (!issue || issue.assigneeAgentId !== run.agentId || ["done", "cancelled"].includes(issue.status)) diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index add78046e35..c46e7976c96 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -6252,11 +6252,15 @@ async function migrateRunnerdStateRootForExecution(input: { await verifyPluginRunnerTaskReattachment(target.runnerTask, identity); return; } - await verifyRemoteRunnerOrSettledExit({ - claim: input.restartRecovery, target: input.runnerExecutionTarget, identity, - runId: input.execution.binding.runId, normalizedSessionId: nativeSessionKey(input.execution), - providerAttempt: input.restartRecovery.providerAttempt, - }); + try { + await verifyRemoteRunnerOrSettledExit({ + claim: input.restartRecovery, target: input.runnerExecutionTarget, identity, + runId: input.execution.binding.runId, normalizedSessionId: nativeSessionKey(input.execution), + providerAttempt: input.restartRecovery.providerAttempt, + }); + } catch (error) { + throw withRemoteRunnerTurnProof(error, scoped); + } return; } if (durableIdentityMatchesExecution(identity, input.execution)) { @@ -6858,6 +6862,20 @@ export async function stopStaleRemoteRunnerBeforeBootstrap(input: { } } +/** + * A refused reattachment whose Runner process is proven gone records whether + * this run's controller journal proves that no turn command was ever issued. + * The controller writes each turn command there before sending it, so only + * that positive record (never an absence of run events) allows an automatic + * relaunch that cannot repeat provider work. + */ +export function withRemoteRunnerTurnProof(error: unknown, root: string): unknown { + const diagnostic = readRemoteRunnerRecoveryError(error); + if (diagnostic?.stage !== "process_verification" || diagnostic.reason !== "process_not_running" || + !runnerdStateProvesNoRemoteProviderTurn(root)) return error; + return attachRemoteRunnerRecoveryDiagnostic(error, { ...diagnostic, providerTurnIssued: false }); +} + /** The remote controller record proves its runner was never sent a turn. */ export function runnerdStateProvesNoRemoteProviderTurn(root: string): boolean { try { diff --git a/server/src/services/native-runtime/remote-runner-recovery-diagnostics.ts b/server/src/services/native-runtime/remote-runner-recovery-diagnostics.ts index 78d660b5bfc..cd6b7710842 100644 --- a/server/src/services/native-runtime/remote-runner-recovery-diagnostics.ts +++ b/server/src/services/native-runtime/remote-runner-recovery-diagnostics.ts @@ -21,6 +21,8 @@ export interface RemoteRunnerRecoveryDiagnostic { reason: typeof reasons[number]; runnerLifecycle?: typeof runnerLifecycles[number]; runnerSettled?: boolean; + /** Present only as `false`: the controller journal proves no turn command was ever issued. */ + providerTurnIssued?: false; } export function describeExitedRemoteRunner(state: Record): Pick; + const { stage, reason, runnerLifecycle, runnerSettled, providerTurnIssued } = value as Record; if (!stages.some(candidate => candidate === stage) || !reasons.some(candidate => candidate === reason)) return null; return { stage, reason, ...(runnerLifecycles.some(candidate => candidate === runnerLifecycle) ? { runnerLifecycle } : {}), ...(typeof runnerSettled === "boolean" ? { runnerSettled } : {}), + ...(providerTurnIssued === false ? { providerTurnIssued } : {}), } as RemoteRunnerRecoveryDiagnostic; } catch { return null; diff --git a/server/src/services/native-runtime/remote-runner-turn-proof.test.ts b/server/src/services/native-runtime/remote-runner-turn-proof.test.ts new file mode 100644 index 00000000000..ff585e5058d --- /dev/null +++ b/server/src/services/native-runtime/remote-runner-turn-proof.test.ts @@ -0,0 +1,52 @@ +import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, describe, expect, it } from "vitest"; +import { withRemoteRunnerTurnProof } from "./native-session-executor.js"; +import { RemoteRunnerRecoveryError, readRemoteRunnerRecoveryError } from "./remote-runner-recovery-diagnostics.js"; + +const roots: string[] = []; +afterEach(async () => { await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true }))); }); + +async function journal(commands: Array<{ type: string; status: string }>, eventTypes = ["session.started"]) { + const root = await mkdtemp(path.join(os.tmpdir(), "turn-proof-")); + roots.push(root); + await mkdir(path.join(root, "control-plane"), { recursive: true }); + await writeFile(path.join(root, "control-plane", "control-plane-state.json"), JSON.stringify({ + schema: "paperclip.runner.durable.control-plane-state.v2", connectionCount: 1, + commands, committedEvents: eventTypes.map((eventType, index) => ({ sourceSeq: index + 1, eventType })), + ackedSourceSeq: eventTypes.length, + })); + return root; +} +const exited = () => new RemoteRunnerRecoveryError({ stage: "process_verification", reason: "process_not_running", + runnerLifecycle: "ready", runnerSettled: false }); + +// Staging class T: the Runner process was proven gone. Only the controller +// journal, which records each turn command before sending it, can prove no +// turn was issued; an absence of run events cannot. +describe("withRemoteRunnerTurnProof", () => { + it("records the proof when the journal holds only session commands", async () => { + const root = await journal([{ type: "run.prepare", status: "completed" }, { type: "session.open", status: "completed" }, + { type: "session.snapshot", status: "pending" }]); + expect(readRemoteRunnerRecoveryError(withRemoteRunnerTurnProof(exited(), root))).toMatchObject({ + reason: "process_not_running", providerTurnIssued: false }); + }); + + it.each([ + ["a turn command was issued", [{ type: "session.open", status: "completed" }, { type: "turn.start", status: "pending" }], ["session.started"]], + ["a turn event was committed", [{ type: "session.open", status: "completed" }], ["session.started", "turn.started"]], + ])("records nothing when %s", async (_label, commands, events) => { + const root = await journal(commands, events); + expect(readRemoteRunnerRecoveryError(withRemoteRunnerTurnProof(exited(), root))).not.toHaveProperty("providerTurnIssued"); + }); + + it("records nothing without a readable journal or without a proven exit", async () => { + const empty = await mkdtemp(path.join(os.tmpdir(), "turn-proof-")); + roots.push(empty); + expect(readRemoteRunnerRecoveryError(withRemoteRunnerTurnProof(exited(), empty))).not.toHaveProperty("providerTurnIssued"); + const root = await journal([{ type: "session.open", status: "completed" }]); + const unreachable = new RemoteRunnerRecoveryError({ stage: "process_verification", reason: "probe_transport_failed" }); + expect(readRemoteRunnerRecoveryError(withRemoteRunnerTurnProof(unreachable, root))).not.toHaveProperty("providerTurnIssued"); + }); +}); diff --git a/server/src/services/workspace-recovery-retry-copyback.test.ts b/server/src/services/workspace-recovery-retry-copyback.test.ts index 39bbad5f77b..def29ff74d5 100644 --- a/server/src/services/workspace-recovery-retry-copyback.test.ts +++ b/server/src/services/workspace-recovery-retry-copyback.test.ts @@ -114,6 +114,17 @@ describe("automatic retained workspace recovery", () => { } const queuedSuccessors = async (companyId: string) => db.select().from(heartbeatRuns) .where(and(eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.status, "queued"))); + async function systemWake(f: Awaited>, proof?: { providerTurnIssued?: false }) { + await db.update(agentWakeupRequests).set({ requestedByActorType: "system", requestedByActorId: null, + reason: "issue_monitor_due" }).where(eq(agentWakeupRequests.id, f.wake.id)); + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, wakeReason: "issue_monitor_due" } }) + .where(eq(heartbeatRuns.id, f.runId)); + // The executor's refused reattachment (see withRemoteRunnerTurnProof). + if (proof) await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.runId, agentId: f.agentId, + eventType: "native.remote_runner_recovery_failed", stream: "system", level: "error", message: "Could not verify the existing remote runner.", + payload: { remoteRunnerRecovery: { stage: "process_verification", reason: "process_not_running", + runnerLifecycle: "ready", runnerSettled: false, ...proof } } }); + } it.each([ { title: "the first attachment found the previous runner gone", turn: "none" as const, error: "runner_remote_process_identity_unavailable", initial: true }, @@ -122,8 +133,14 @@ describe("automatic retained workspace recovery", () => { { title: "a same-run retry found the settled turn's remote runner gone", turn: "settled" as const, error: "runner_remote_process_identity_unavailable", initial: false }, { title: "a same-run retry found the settled turn's sandbox lease gone", turn: "settled" as const, error: "native_remote_recovery_lease_mismatch", initial: false }, { title: "a same-run retry named the lease check that failed", turn: "settled" as const, error: "native_remote_recovery_lease_mismatch: stage=environment_acquire field=lease_status", initial: false }, - ])("continues a user turn once, without a second message, after $title and recovery saved the work", async ({ turn, error, initial }) => { + // Staging class T (c14bab19): a monitor wake's session snapshot timed out + // before any turn, then the same-run retry found the Runner gone. Its + // controller journal proved no turn command was ever issued, so nothing can + // be repeated and the run's responsible user's Retry continues it once. + { title: "a system wake's run found its runner gone and its journal proved no turn", turn: "none" as const, error: "runner_remote_process_identity_unavailable", initial: false, system: true }, + ])("continues a user turn once, without a second message, after $title and recovery saved the work", async ({ turn, error, initial, system }) => { const f = await restartAllocationLostSource(turn, error); + if (system) await systemWake(f, { providerTurnIssued: false }); const service = heartbeatService(db, { environmentRuntime: f.environmentRuntime }); // Keep the agent's only slot busy so the successor stays observable as queued. await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" }); @@ -164,18 +181,24 @@ describe("automatic retained workspace recovery", () => { it.each([ { title: "its provider turn was still in flight", change: "turn" }, { title: "its provider turn ended without an observed provider terminal", change: "unproven" }, - { title: "the source was a system wake", change: "system" }, + { title: "the source was a system wake whose provider turn settled", change: "system" }, + // No run events is not proof: a turn can reach the provider before its + // events are stored. Only the controller journal can prove it. + { title: "the source was a system wake with no run events but no journal proof", change: "system_unproven" }, + { title: "the source was a system wake whose run used a tool before its turn event", change: "system_tool" }, + { title: "the source was a system wake whose run had no responsible user", change: "system_unowned" }, { title: "the failure was not a lost runner or allocation", change: "error" }, // Staging class P: the new sandbox had no Runner state to resume the saved // provider session. The Runner now names it, so the run ends with the // permanent code and Retry continues in a fresh provider session. { title: "the Runner had no saved state to resume the provider session", change: "unrestorable" }, ])("does not continue automatically when $title, and offers a Retry that works on first use", async ({ change }) => { - const f = await restartAllocationLostSource(change === "turn" ? "in_flight" : change === "unproven" ? "unproven" : "none"); - if (change === "system") await db.update(agentWakeupRequests).set({ requestedByActorType: "system", requestedByActorId: null, - reason: "issue_children_completed" }).where(eq(agentWakeupRequests.id, f.wake.id)); - if (change === "system") await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, wakeReason: "issue_children_completed" } }) - .where(eq(heartbeatRuns.id, f.runId)); + const f = await restartAllocationLostSource(change === "turn" ? "in_flight" : change === "unproven" ? "unproven" + : change === "system" ? "settled" : "none"); + if (change.startsWith("system")) await systemWake(f, change === "system_unproven" ? undefined : { providerTurnIssued: false }); + if (change === "system_tool") await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.runId, + agentId: f.agentId, eventType: "tool.execution.started", stream: "system", level: "info", message: "tool" }); + if (change === "system_unowned") await db.update(heartbeatRuns).set({ responsibleUserId: null }).where(eq(heartbeatRuns.id, f.runId)); if (change === "error") await db.update(heartbeatRuns).set({ error: "native_provider_terminal_failed" }).where(eq(heartbeatRuns.id, f.runId)); if (change === "unrestorable") await db.update(heartbeatRuns).set({ errorCode: "native_provider_state_incompatible", error: "native_session_recovery_failed: NativeRunnerResumeStateUnavailableError: PRP provider resume state is unavailable" })