From b8e8ab4641b4cb6aac5cf2b64bd10630f2713eaa Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 5 Nov 2026 05:08:03 -0600 Subject: [PATCH] fix(recovery): continue saved input after a recovered sandbox stops, not after its destroy (#19224) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A board user can stop or interrupt a native run and send a message for the next turn > - When the stop leaves sandbox files unsaved, automatic recovery saves them, re-stops the sandbox, and the saved message continues > - For agents whose sandbox is destroyed after each turn (an ephemeral lease), recovery then queues the destroy. That step removed the lease's stop receipt > - The continuation gate waits for a stop receipt, so the message waited for the destroy and its cleanup retry, about 5 to 8 minutes > - Agents with retained sandboxes continued within seconds, so the delay looked agent specific > - This pull request keeps the verified stop on the cleanup lease and lets the continuation and the next remote writer accept it > - The benefit is that a redirect, Retry or interrupt continues right after its files are saved, for every lease policy ## Linked Issues or Issue Description Refs #19206 **What happened?** On hosted staging, a Stop or `interrupt: true` comment on a native run left a follow-up turn that started 5 to 8 minutes later for some agents, and 25 to 40 seconds later for others. The slow agents use ephemeral sandbox leases. In each slow case, the stopped run's lease was released by `pending_cleanup_retry` about 5 minutes after recovery, and the saved message started one second later. The fast agents use retained leases, which recovery releases at once. **Expected behavior** The saved message continues right after recovery saves the files and the sandbox is stopped. The destroy of a stopped sandbox is resource cleanup and must not hold the next turn. **Steps to reproduce** 1. Use an agent whose sandbox lease is ephemeral (destroy after each turn). 2. Interrupt or stop its native run so that recovery must save the sandbox files. 3. Post a message. See the follow-up start only after the sandbox destroy finishes. **Paperclip version or commit** Hosted, at `56517dda`. ## What Changed - `server/src/services/workspace-recovery.ts`: `scheduleRecoveredWorkspaceCleanup` keeps the verified stop receipt as `workspaceRecoveryCleanup.stopReceipt` when it queues the destroy. - `server/src/services/remote-execution-termination.ts`: new `hasRecoveredCleanupStopReceipt(lease)`. It is true only when the lease has recovery cleanup for its own run, was released, and the carried receipt matches the lease, run, provider and provider lease id. - `server/src/services/explicit-native-continuation.ts`: the `remote_cleanup` gate accepts that receipt. - `server/src/services/execution-blocker.ts`: `hasPendingRunCleanup` ignores a lease with that valid receipt, so a saved native interrupt send (and Retry or fresh-workspace checks) does not wait for the destroy. Other cleanup holds are unchanged. - `server/src/services/remote-workspaces.ts`: the next remote writer accepts that receipt for the ended writer's lease. - `server/src/services/execution-blocker.ts` (from #19206 review, not in the merged head): pre-turn evidence uses the exact startup lifecycle event list, so turn-scoped events such as `workspace.diff.recorded` count as provider work. - `doc/execution-semantics.md`: describes the rule. ## Verification - Red before the fix, green after: `cd server && npx vitest run src/services/workspace-recovery.test.ts -t "admits the interrupt redirect"`. An interrupted native run with recovered files and a queued destroy blocked the user message with `remote_cleanup` before the fix. After the fix the message is admitted. A lease without the stop receipt, or with a receipt for another run, still blocks with `remote_cleanup`. The test runs the same input as a plain message and as a saved native interrupt send (`queuedCommentInterruptId`); `hasPendingRunCleanup` was true before the fix and blocked the interrupt send with `cleanup_pending`. - `cd server && npx vitest run src/services/remote-workspace-reservation.test.ts -t "ended writer"`: the next writer reserves the workspace from an ended writer whose recovered lease awaits destroy; a missing or foreign receipt still blocks. Removing the new branch fails the positive case. - Updated `saves the authoritative remote workspace after native failed ...`: Retry now continues before the destroy, and the cleanup lease carries the stop receipt for its run and lease. - `workspace-recovery.test.ts -t "provider work cannot be excluded"` includes `workspace.diff.recorded` as provider work. - Related suites pass: `workspace-recovery.test.ts`, `explicit-native-continuation.test.ts`, `remote-workspace-reservation.test.ts`, `remote-workspaces.integration.test.ts`, `remote-workspace-lifecycle.test.ts`, `task-retry*.test.ts`, `native-cancelled-workspace-preservation.test.ts`, `workspace-recovery-source.test.ts`. `cd server && npx tsc --noEmit` passes. - Hosted evidence (read only): slow cases have ephemeral leases released by `pending_cleanup_retry` about 5 minutes after `issue.workspace_recovered`, with settlement one second later. Fast cases have retained leases released by recovery. ## Risks - Low to medium. The next turn can start while the old stopped sandbox is still being destroyed. The receipt proves that the sandbox was stopped after the files were saved, and an ephemeral successor uses a new sandbox. A destroy failure stays with the existing cleanup retry. - Leases without the carried receipt behave as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/execution-semantics.md | 10 ++- server/src/services/execution-blocker.ts | 32 +++++--- .../services/explicit-native-continuation.ts | 4 +- .../services/remote-execution-termination.ts | 13 ++++ .../remote-workspace-reservation.test.ts | 28 ++++++- server/src/services/remote-workspaces.ts | 4 +- .../src/services/workspace-recovery.test.ts | 74 ++++++++++++++++--- server/src/services/workspace-recovery.ts | 4 +- 8 files changed, 140 insertions(+), 29 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index d027cdbfaea..b400dc6bd3a 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -2158,7 +2158,7 @@ A Stop or reassignment can reach a native run while its provider session is still opening. The startup owner then fences the late handle before any prompt, and the Stop acknowledgement records `nativeCancellation.beforeProviderTurn`. A reattached provider from restart recovery never gets this proof. When the run -also has no runner event outside the startup lifecycle (harness, session, workspace, +also has no runner event outside the exact startup lifecycle events (harness, session, workspace ready, runtime phase, credential refresh), no turn or provider event, and no native result, no provider outcome is unknown. The completed copyback receipt then releases the hold for the stopped @@ -2171,6 +2171,14 @@ complete, so recovery and that turn cannot both wait for each other. Running turns, and queued turns for other tasks that share the workspace, still block recovery. +After recovery saves the files of a run whose sandbox is destroyed after each +turn (an ephemeral lease), it re-stops the sandbox and then queues the destroy. +The lease carries the verified stop forward as +`workspaceRecoveryCleanup.stopReceipt`. The saved message, Retry and the next +remote workspace writer wait for that stop, not for the destroy. So they continue +right after recovery, instead of after the cleanup retry. A lease without a +matching stop receipt still blocks them. + The deadline is a backstop, not the normal failure path. When native task preparation fails after a fresh controller claim and before any provider turn, the controller stops renewal and releases only that exact claim. Ordinary diff --git a/server/src/services/execution-blocker.ts b/server/src/services/execution-blocker.ts index b67d5c16f2b..ed1630f6f07 100644 --- a/server/src/services/execution-blocker.ts +++ b/server/src/services/execution-blocker.ts @@ -11,7 +11,7 @@ import { queuedCommentIdsFromWakePayload } from "./issue-queued-comment-queue.js import { EXECUTION_RECONCILIATION_CAUSES, type ExecutionBlocker } from "@paperclipai/shared"; import { hasRequiredWorkspaceRecovery, LEGACY_WORKSPACE_RECOVERY_SCHEMA, nativeRunHasNoWorkspaceAccess } from "./workspace-restore-recovery-state.js"; import { unreconciledNativeExportPredicate } from "./native-runtime/native-workspace-reconstruction.js"; -import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js"; +import { hasRecoveredCleanupStopReceipt, hasRemoteTerminationReceipt } from "./remote-execution-termination.js"; import { readNativeWorkspaceSyncReference } from "./native-runtime/native-workspace-sync.js"; import { completedWorkspaceRecoveryCondition, hasVerifiedWorkspaceRepair, workspaceRecoveryDisplay } from "./workspace-recovery-receipt.js"; import { nativeManualAdmissionHoldsExecution } from "./native-runtime/native-runner-ownership.js"; @@ -69,10 +69,17 @@ const blockerRunFields = { export async function hasPendingRunCleanup(db: Db, companyId: string, runId: string): Promise { if (await nativeManualAdmissionHoldsExecution(db, { companyId, runId })) return true; + // Recovery saved the files and re-stopped this sandbox before queueing its + // destroy. That destroy is resource cleanup, not a live owner of the run. + const recoveredStopped = (await db.select().from(environmentLeases).where(and( + eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, runId), + sql`${environmentLeases.metadata}->'workspaceRecoveryCleanup' is not null`, + ))).filter(hasRecoveredCleanupStopReceipt).map(lease => lease.id); const [state] = await db.execute<{ pending: boolean }>(sql`SELECT ${heartbeatRuns.runnerProfileJson}->>'nativeWorkspaceFinalizationOwner' IS NOT NULL OR EXISTS (SELECT 1 FROM ${environmentLeases} WHERE ${environmentLeases.companyId} = ${companyId} AND ${environmentLeases.heartbeatRunId} = ${runId} + ${recoveredStopped.length ? sql`AND ${environmentLeases.id}::text NOT IN (${sql.join(recoveredStopped.map(id => sql`${id}`), sql`, `)})` : sql``} AND (( ${environmentLeases.releasedAt} IS NULL AND NOT COALESCE( ${environmentLeases.status} = 'retained' AND ${environmentLeases.leasePolicy} = 'reuse_by_environment' AND ${environmentLeases.cleanupStatus} = 'success', false)) @@ -126,14 +133,18 @@ export function nativeExecutionRecoveryActionPredicate() { and ${heartbeatRuns.runtimeMode} = 'native')`; } -// Runner startup lifecycle families: they open no turn. Every other event the -// runner sourced counts as provider work, so a new or unlisted family keeps -// the hold (fail closed). Server-side records without a source event (spans, -// process identity, lifecycle) count only when they name turn or provider work. -export const PROVIDER_STARTUP_EVENT_PREFIXES = ["harness.", "session.", "workspace.", "runtime.phase.", - "runtime.activity.", "provider.notice.", "auth.credentials.", "mcp_app.discovered", "mcp_app.initializing", "mcp_app.ready", - "mcp_app.resource.", "mcp_app.host_context.", "mcp_app.teardown", "mcp_app.failed"] as const; -export const PROVIDER_STARTUP_EVENT_TYPES = ["run.attached", "run.detached"] as const; +// Exact runner startup lifecycle events (event-v3 schema): they open no turn. +// Every other runner-sourced event counts as provider work, so a new, unlisted +// or turn-scoped event (for example workspace.diff.recorded or a session goal) +// keeps the hold (fail closed). Server-side records without a source event +// (spans, process identity, lifecycle) count only when they name turn or +// provider work. +export const PROVIDER_STARTUP_EVENT_TYPES = ["harness.starting", "harness.ready", "harness.diagnostic", "harness.exited", + "session.starting", "session.started", "session.resuming", "session.resumed", "session.reconciled", "session.updated", + "session.capabilities.updated", "session.closed", "session.failed", "workspace.ready", "runtime.phase.changed", + "runtime.activity.recorded", "provider.notice.recorded", "auth.credentials.updated", "run.attached", "run.detached", + "mcp_app.discovered", "mcp_app.initializing", "mcp_app.ready", "mcp_app.resource.resolved", + "mcp_app.host_context.changed", "mcp_app.teardown", "mcp_app.failed"] as const; const SERVER_TURN_WORK_EVENT_PREFIXES = ["turn.", "item.", "tool.", "semantic_tool.", "run.result."] as const; const SERVER_TURN_WORK_EVENT_TYPES = ["provider.event", "provider.rpc_result"] as const; @@ -144,8 +155,7 @@ const anyType = (eventType: SQL, types: readonly string[]) => /** SQL truth for an event row (alias columns) that may prove provider turn work. */ export function providerTurnWorkEventCondition(eventType: SQL, sourceEventId: SQL) { - return sql`((${sourceEventId} is not null and not (${anyType(eventType, PROVIDER_STARTUP_EVENT_TYPES)} - or ${anyPrefix(eventType, PROVIDER_STARTUP_EVENT_PREFIXES)})) + return sql`((${sourceEventId} is not null and not ${anyType(eventType, PROVIDER_STARTUP_EVENT_TYPES)}) or ${anyType(eventType, SERVER_TURN_WORK_EVENT_TYPES)} or ${anyPrefix(eventType, SERVER_TURN_WORK_EVENT_PREFIXES)})`; } diff --git a/server/src/services/explicit-native-continuation.ts b/server/src/services/explicit-native-continuation.ts index 7593411ffb5..eec6f8c3dde 100644 --- a/server/src/services/explicit-native-continuation.ts +++ b/server/src/services/explicit-native-continuation.ts @@ -17,7 +17,7 @@ import { readQueuedInteractionResponse } from "./queued-interaction-response.js" import { isCancelledNativeStartup, isFailedNativePreparation, isFailedNativePreparationRetryAncestor } from "./cancelled-native-startup.js"; import { hasNativeLocalProcessStop, hasHistoricalSuspendedNativeSession, recordedLocalExecutionStopped } from "./native-local-process-stop.js"; import { completeTerminatedRemoteNativeSessionCleanup } from "../vendor/paperclip-runner/index.js"; -import { hasRemoteTerminationReceipt, remoteLeaseCleanupScope } from "./remote-execution-termination.js"; +import { hasRecoveredCleanupStopReceipt, hasRemoteTerminationReceipt, remoteLeaseCleanupScope } from "./remote-execution-termination.js"; import { z } from "zod"; import { and, desc, eq, gt, inArray, isNull, ne, not, or, sql } from "drizzle-orm"; import { @@ -435,7 +435,7 @@ export async function admitExplicitNativeContinuation(input: { const remote = leases.some(lease => lease.provider !== "local"); if (remote) { // Never interpret remote PIDs using the control-plane host's process table. - if (!leases.every(hasRemoteTerminationReceipt)) return blocked("remote_cleanup", "Waiting for the previous environment to stop. Your message will start automatically."); + if (!leases.every(lease => hasRemoteTerminationReceipt(lease) || hasRecoveredCleanupStopReceipt(lease))) return blocked("remote_cleanup", "Waiting for the previous environment to stop. Your message will start automatically."); if (run.runtimeMode === "native" && !input.dryRun && !leases.every(lease => completeTerminatedRemoteNativeSessionCleanup({ companyId, runId: run.id, remoteCleanupScope: remoteLeaseCleanupScope(lease)!, }))) return null; diff --git a/server/src/services/remote-execution-termination.ts b/server/src/services/remote-execution-termination.ts index d30b60ae697..e18a6fdaae9 100644 --- a/server/src/services/remote-execution-termination.ts +++ b/server/src/services/remote-execution-termination.ts @@ -37,6 +37,19 @@ export function hasRemoteTerminationReceipt(lease: LeaseIdentity & { remoteTerminationReceipt(lease, receipt)); } +/** Workspace recovery saved this run's files, re-stopped its sandbox, and then + * queued the stopped sandbox for destroy. The destroy is resource cleanup: the + * stop receipt it carried forward still proves no provider can run there. */ +export function hasRecoveredCleanupStopReceipt(lease: LeaseIdentity & { + releasedAt: unknown; status: string; metadata: Record | null; +}): boolean { + const cleanup = lease.metadata?.workspaceRecoveryCleanup as Record | undefined; + if (!cleanup || cleanup.runId !== lease.heartbeatRunId || !lease.releasedAt || + !["pending_cleanup", "released", "expired", "failed"].includes(lease.status)) return false; + return hasRemoteTerminationReceipt({ ...lease, status: "released", cleanupStatus: "success", + metadata: { remoteExecutionTermination: cleanup.stopReceipt } }); +} + export function remoteLeaseCleanupScope(lease: Pick) { return lease.provider && lease.provider !== "local" && lease.providerLeaseId ? JSON.stringify([lease.provider, lease.providerLeaseId]) : undefined; diff --git a/server/src/services/remote-workspace-reservation.test.ts b/server/src/services/remote-workspace-reservation.test.ts index 76b7d0d5eed..b1980b44041 100644 --- a/server/src/services/remote-workspace-reservation.test.ts +++ b/server/src/services/remote-workspace-reservation.test.ts @@ -7,7 +7,7 @@ import request from "supertest"; import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; import { eq } from "drizzle-orm"; import { agents, companies, companyMemberships, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, environments, - executionWorkspaces, heartbeatRuns, issues, projects, projectWorkspaces } from "@paperclipai/db"; + environmentLeases, executionWorkspaces, heartbeatRuns, issues, projects, projectWorkspaces } from "@paperclipai/db"; import { readRemoteWorkspaceAuthority } from "@paperclipai/shared"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js"; import { createLocalDiskStorageProvider } from "../storage/local-disk-provider.js"; @@ -15,6 +15,7 @@ import { createStorageService } from "../storage/service.js"; import { instanceSettingsService } from "./instance-settings.js"; import { issueService } from "./issues.js"; import { reserveRemoteWorkspace } from "./remote-workspaces.js"; +import { remoteTerminationReceipt } from "./remote-execution-termination.js"; import { executionWorkspaceService } from "./execution-workspaces.js"; import { accessService } from "./access.js"; import { taskWorkspaceRoutes } from "../routes/task-workspaces.js"; @@ -152,6 +153,31 @@ describe.skipIf(!support.supported)("remote workspace reservation authority", () await expect(reserveRemoteWorkspace(f.input)).resolves.toMatchObject({ id: f.workspace.id }); }); + it.each(["verified-stop", "no-stop-receipt", "foreign-receipt"] as const) + ("hands over from an ended writer whose recovered sandbox awaits destroy (%s)", async kind => { + const f = await fixture(); + const [writer] = await db.insert(heartbeatRuns).values({ companyId: f.company.id, agentId: f.agent.id, + issueId: f.parent.id, status: "cancelled", finishedAt: new Date() }).returning(); + const providerLeaseId = randomUUID(); + const [lease] = await db.insert(environmentLeases).values({ companyId: f.company.id, environmentId: f.input.environmentId, + heartbeatRunId: writer.id, issueId: f.parent.id, provider: "daytona", providerLeaseId, + status: "pending_cleanup", cleanupStatus: "failed", leasePolicy: "ephemeral", releasedAt: new Date() }).returning(); + const receipt = remoteTerminationReceipt(lease, { providerLeaseId, state: "stopped" })!; + const stopReceipt = kind === "verified-stop" ? receipt : kind === "foreign-receipt" ? { ...receipt, runId: randomUUID() } : undefined; + await db.update(environmentLeases).set({ metadata: { workspaceRecoveryCleanup: { runId: writer.id, stopReceipt } } }) + .where(eq(environmentLeases.id, lease.id)); + // Recovery saved the files, so the writer is clean but not yet destroyed. + const ended = { ...f.authority, writerRunId: writer.id, dirty: false }; + await db.update(executionWorkspaces).set({ metadata: { remoteWorkspace: ended } }).where(eq(executionWorkspaces.id, f.workspace.id)); + if (kind === "verified-stop") { + await expect(reserveRemoteWorkspace(f.input)).resolves.toMatchObject({ id: f.workspace.id }); + expect(readRemoteWorkspaceAuthority((await current(f.workspace.id)).metadata)).toMatchObject({ writerRunId: f.run.id, epoch: 4 }); + } else { + await expect(reserveRemoteWorkspace(f.input)).rejects.toThrow(); + expect(readRemoteWorkspaceAuthority((await current(f.workspace.id)).metadata)).toEqual(ended); + } + }); + it("grants only one writer when two bound children reserve concurrently", async () => { const f = await fixture(); const sibling = await issueService(db).create(f.company.id, { parentId: f.parent.id, diff --git a/server/src/services/remote-workspaces.ts b/server/src/services/remote-workspaces.ts index f75f2ce5503..298e1eed7ec 100644 --- a/server/src/services/remote-workspaces.ts +++ b/server/src/services/remote-workspaces.ts @@ -22,7 +22,7 @@ import { logger } from "../middleware/logger.js"; import { createGitRemoteAuthProvider } from "./git-credentials.js"; import { notifyCancelledWorkspacePreservation } from "./cancelled-workspace-preservation-notifications.js"; import { assertExecutionWorkspaceRecordAccess, assertTaskWorkspaceExecutionAccess } from "./task-workspace-access.js"; -import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js"; +import { hasRecoveredCleanupStopReceipt, hasRemoteTerminationReceipt } from "./remote-execution-termination.js"; import { canonicalExecutionUuid } from "./execution-identity.js"; import { persistActivity } from "./activity-log.js"; import { @@ -240,7 +240,7 @@ export async function reserveRemoteWorkspace(input: { && prior.errorCode === REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE && bootstrap.kind === "bootstrap" && bootstrap.providerWorkStarted === false; const unusedReservation = !authority.dirty && prior && ["failed", "cancelled", "interrupted", "timed_out", "succeeded"].includes(prior.status) - && leases.every(lease => hasRemoteTerminationReceipt(lease) || (pristinePreparation + && leases.every(lease => hasRemoteTerminationReceipt(lease) || hasRecoveredCleanupStopReceipt(lease) || (pristinePreparation && lease.releasedAt && lease.cleanupStatus === "success" && ["released", "expired", "failed"].includes(lease.status))); if (!unusedReservation) { // An ended writer still holds the workspace until its save and sandbox diff --git a/server/src/services/workspace-recovery.test.ts b/server/src/services/workspace-recovery.test.ts index feec7f5a4b3..635091541d0 100644 --- a/server/src/services/workspace-recovery.test.ts +++ b/server/src/services/workspace-recovery.test.ts @@ -84,7 +84,7 @@ import { preserveRemoteWorkspacePreparationFailure } from "./native-runtime/nati import { canonicalProviderPackJson } from "./native-runtime/remote-provider-pack-verification.js"; import { REMOTE_WORKSPACE_SANDBOX_DISK_FULL } from "./remote-workspace-transport.js"; import { withDedicatedDbConnection } from "@paperclipai/db"; -import { getExecutionBlocker } from "./execution-blocker.js"; +import { getExecutionBlocker, hasPendingRunCleanup } from "./execution-blocker.js"; import { buildExecutionContinuation } from "./execution-continuation.js"; import { createHeartbeatRunOutput } from "./heartbeat/run-output.js"; import { objectLogFixture } from "./heartbeat/__tests__/object-log-fixture.js"; @@ -2496,11 +2496,9 @@ describe("automatic retained workspace recovery", () => { await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: source.agentId, status: "running" }); const retry = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issueId, actorId: "local-board", actorSource: "local_implicit", observedRunId: source.id, idempotencyKey: randomUUID() }); - await heartbeatService(db).resumeTaskRetry(f.companyId, retry.requestId!); - expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, retry.requestId!)))[0]) - .toMatchObject({ status: "deferred_issue_execution", payload: { executionWait: { reason: "remote_cleanup" } } }); - await environmentService(db).releaseLease(settledLease.id, "released", { cleanupStatus: "success", - remoteExecutionTermination: remoteTerminationReceipt(settledLease, { providerLeaseId: settledLease.providerLeaseId, state: "destroyed" }) }); + // The saved, re-stopped sandbox only awaits destroy. That cleanup does not + // hold the successor: it continues now instead of on the cleanup retry. + expect(settledLease.metadata?.workspaceRecoveryCleanup).toMatchObject({ stopReceipt: { runId: f.runId, leaseId: settledLease.id } }); await heartbeatService(db).resumeTaskRetry(f.companyId, retry.requestId!); const [successor] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued"))); expect(successor).toBeDefined(); @@ -3688,7 +3686,7 @@ describe("automatic retained workspace recovery", () => { expect((await pendingTurns(f)).map(item => item.id)).toEqual([handoff.id]); }, 30_000); - it.each(["no-proof", "turn-event", "provider-event", "unlisted-provider-event", "running-writer", "other-task-writer", + it.each(["no-proof", "turn-event", "provider-event", "unlisted-provider-event", "workspace-diff-event", "running-writer", "other-task-writer", "unscoped-queued-writer", "non-gating-hold"] as const) ("keeps the stopped agent held when provider work cannot be excluded (%s)", async kind => { const f = await seed(); @@ -3698,10 +3696,11 @@ describe("automatic retained workspace recovery", () => { eventType: "turn.started", stream: "system", level: "info", message: "turn started" }); if (kind === "provider-event") await db.insert(heartbeatRunEvents).values({ companyId: f.companyId, runId: f.runId, agentId, seq: 9_999, eventType: "item.completed", stream: "system", level: "info", message: "provider item", sourceEventId: randomUUID() }); - // A runner family outside the startup lifecycle counts as work even when unlisted. - if (kind === "unlisted-provider-event") await db.insert(heartbeatRunEvents).values({ companyId: f.companyId, runId: f.runId, - agentId, seq: 9_999, eventType: "delegation.completed", stream: "system", level: "info", message: "delegated", - sourceEventId: randomUUID() }); + // A runner event outside the startup lifecycle counts as work even when unlisted, + // including turn-scoped events that share a startup family name. + if (kind === "unlisted-provider-event" || kind === "workspace-diff-event") await db.insert(heartbeatRunEvents).values({ + companyId: f.companyId, runId: f.runId, agentId, seq: 9_999, stream: "system", level: "info", message: kind, + eventType: kind === "unlisted-provider-event" ? "delegation.completed" : "workspace.diff.recorded", sourceEventId: randomUUID() }); if (kind === "running-writer" || kind === "other-task-writer" || kind === "unscoped-queued-writer") { // A started turn, or a queued one that this task's dispatch gate does not cover, is a real writer. await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId, runtimeMode: "legacy", @@ -3729,6 +3728,59 @@ describe("automatic retained workspace recovery", () => { expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ recoveryActionId: f.actionId, runId: f.runId }); }, 30_000); + it.each(["verified-stop", "no-stop-receipt", "foreign-receipt"] as const) + ("admits the interrupt redirect once recovery saved and stopped the sandbox, before its destroy finishes (%s)", async kind => { + // An interrupt comment after the session opened: the run did provider work, + // its files were recovered, and the stopped sandbox was queued for destroy. + const f = await seed(); + const source = await run(f.runId); + const finishedAt = new Date(Date.now() - 60_000); + await db.update(heartbeatRuns).set({ status: "cancelled", errorCode: "operator_interrupted", finishedAt, + resultJson: { ...source.resultJson, operatorInterrupted: true, interruptionSource: "issue_comment_interrupt", + nativeCancellation: { schema: "paperclip.native-cancellation.v1", runId: f.runId, companyId: f.companyId, + issueId: f.issueId, scope: "run", reasonCode: "cancellation_run_only", dispatchState: "acknowledged", + dispatched: true, intentAuditId: randomUUID(), acknowledgementAuditId: randomUUID() } }, + }).where(eq(heartbeatRuns.id, f.runId)); + await db.update(issues).set({ status: "in_progress" }).where(eq(issues.id, f.issueId)); + await recoverRetainedWorkspace(f.input); + expect(readSystemRecovery((await action(f.actionId)).evidence)).toMatchObject({ state: "complete" }); + await scheduleRecoveredWorkspaceCleanup(db, await run(f.runId)); + const [queued] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)); + expect(queued).toMatchObject({ status: "pending_cleanup", metadata: { workspaceRecoveryCleanup: { runId: f.runId } } }); + if (kind !== "verified-stop") { + const cleanup = queued.metadata!.workspaceRecoveryCleanup as Record; + const stopReceipt = kind === "foreign-receipt" + ? { ...(cleanup.stopReceipt as Record), runId: randomUUID() } : undefined; + await db.update(environmentLeases).set({ metadata: { ...queued.metadata, + workspaceRecoveryCleanup: { ...cleanup, stopReceipt } } }).where(eq(environmentLeases.id, f.lease.id)); + } + const [comment] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId, + authorType: "user", authorUserId: "operator", body: "Use the other approach instead." }).returning(); + // The same input as a plain message and as a saved native interrupt send. + const queueId = randomUUID(); + await db.insert(agentWakeupRequests).values({ id: queueId, companyId: f.companyId, agentId: source.agentId, + source: "automation", reason: "issue_execution_deferred", status: "deferred_issue_execution", + requestedByActorType: "system", requestedByActorId: "heartbeat", + payload: { issueId: f.issueId, _paperclipWakeContext: { wakeCommentIds: [comment.id] }, + queuedCommentInterrupt: { actorId: "operator", requestedAt: new Date().toISOString() } } }); + expect(await hasPendingRunCleanup(db, f.companyId, f.runId)).toBe(kind !== "verified-stop"); + for (const interrupt of [false, true]) { + const reasons: string[] = []; + const admitted = await admitExplicitNativeContinuation({ db, companyId: f.companyId, issueId: f.issueId, + agentId: source.agentId, actorType: "user", actorId: "operator", reason: "issue_commented", + commentId: comment.id, successorRunId: randomUUID(), dryRun: true, onBlocked: reason => reasons.push(reason), + ...(interrupt ? { queuedCommentInterruptId: queueId } : {}) }); + if (kind === "verified-stop") { + // Destroying a stopped sandbox whose files are saved is cleanup, not ownership. + expect(reasons, JSON.stringify({ interrupt, reasons })).toEqual([]); + expect(admitted).toMatchObject({ previousRunId: f.runId, commentId: comment.id }); + } else { + expect(admitted).toBeNull(); + expect(reasons).toContain("remote_cleanup"); + } + } + }, 30_000); + it("keeps a snapshot-only recovery held for a new assignee because the task workspace lacks the saved files", async () => { const f = await seed("native", false); await db.update(heartbeatRuns).set({ status: "cancelled", errorCode: "issue_reassigned" }).where(eq(heartbeatRuns.id, f.runId)); diff --git a/server/src/services/workspace-recovery.ts b/server/src/services/workspace-recovery.ts index 9372a62c97d..955a20a222b 100644 --- a/server/src/services/workspace-recovery.ts +++ b/server/src/services/workspace-recovery.ts @@ -461,7 +461,9 @@ export async function scheduleRecoveredWorkspaceCleanup(db: Db, run: typeof hear if (other) return; await writeEnvironmentCleanup(db, tx => tx.update(environmentLeases).set({ status: "pending_cleanup", cleanupStatus: "failed", leasePolicy: "ephemeral", updatedAt: new Date(), metadata: sql`(${environmentLeases.metadata} - 'workspaceRestoreRecovery' - 'remoteExecutionTermination') || - ${JSON.stringify({ workspaceRecoveryCleanup: { runId: run.id, ...(legacy ? { descriptorSha256: legacy.descriptorSha256 } + ${JSON.stringify({ workspaceRecoveryCleanup: { runId: run.id, + // Keep the verified stop: successors wait on the stop, not on the destroy. + stopReceipt: lease.metadata?.remoteExecutionTermination, ...(legacy ? { descriptorSha256: legacy.descriptorSha256 } : { snapshotId: remote!.snapshotId, epoch: remote!.epoch }) }, pendingCleanupInFlight: false, pendingCleanupRetryAfterMs: 0, pendingCleanupRetryAttempts: 0, pendingCleanupLeaseExpiresAtMs: 0 })}::jsonb`, }).where(and(eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.id, ref.leaseId),