From 5de779848b8764366e37334ff8407b5ba3e2c536 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 5 Nov 2026 12:54:47 -0600 Subject: [PATCH] fix(recovery): save a reassignment-stopped writer's files from its stopped reusable sandbox (#19304) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A remote task workspace stays held by its last writer run until that writer's files are saved > - A reassignment stops the old owner's run. The stopped writer still holds the workspace with unsaved files. > - When the run used a reusable (warm) sandbox, the stop leaves its lease `released` under the reusable policy, with a stop-and-retain receipt > - Cancelled-workspace preservation only admitted `active` or `ephemeral` sources. It never recorded the save for this stopped writer, so no workspace recovery ran, and the hold stayed forever > - The new owner waited, then failed setup after the one-hour cutoff with "The workspace still has an unfinished writer or save operation" > - This pull request admits the stopped reusable source, so Paperclip saves the stopped writer's files and releases the hold. A waiter keeps waiting while that save is in progress. > - The benefit is that a reassigned task continues with the old owner's saved work, and the new owner does not fail setup ## Linked Issues or Issue Description Refs #19285 (parks a waiter behind an ended writer). Refs #19074, #18820. **What happened?** A board user reassigned a task during a native run on a reusable sandbox. The old run was cancelled with `issue_reassigned`. Its sandbox was stopped and its lease was released under `reuse_by_environment`. The workspace still named the stopped run as its dirty writer. No save operation and no workspace recovery action were ever recorded for that run. The new owner's runs were cancelled `workspace_busy` repeatedly. After more than an hour, a new run failed setup: "The workspace still has an unfinished writer or save operation." **Expected behavior** Paperclip saves the stopped writer's files from its stopped sandbox within a bounded time and releases the hold. The new owner waits for that save and then continues. If the save cannot finish, its retries end with a named board recovery action. **Steps to reproduce** 1. Give a native run a prepared remote workspace on a lease with policy `reuse_by_environment`. 2. Stop the run (for example by reassignment), so the lease is stopped and released with a stop-and-retain receipt. 3. Run cancelled-workspace preservation. Before this fix, it records nothing, and the writer hold stays. ## What Changed - `native-failed-workspace-recovery.ts`: Cancelled-workspace preservation (the delivery worker and the startup scan) now admits a `released` reusable lease with a stopped termination receipt and a stop-and-retain receipt. - `preserveNativeFailedWorkspace` retains such a reusable source only with the exact confirmed stop-and-retain receipt, and only while the workspace is still bound to that writer (`preparedRemoteWorkspaceSource`). Otherwise it writes nothing. - `remote-workspace-writer-hold.ts`: For an ended writer, the holder cutoff now uses the latest progress of an in-progress save of that writer's retained files (a workspace recovery that is not complete, exhausted, or recovered). Reservation and the parked `workspace_busy` wait both read this rule. A waiter waits for the save instead of failing setup. - `doc/execution-semantics.md`: Documents the stopped reusable source and the save-progress cutoff. ## Verification - Red before the fix, green after: - `npx vitest run src/services/native-runtime/native-cancelled-workspace-preservation.test.ts` — new cases "preserves a reassignment-stopped writer's files from its stopped reusable sandbox" (worker and startup scan) failed before the fix: no `workspaceRestoreRecovery` was recorded. - `npx vitest run src/__tests__/heartbeat-remote-workspace-busy.test.ts` — new cases "keeps a waiter parked while an ended writer's save is in progress" failed before the fix: the parked retry was released past the cutoff. - Safety cases: - A reusable source without a confirmed stop, a workspace now owned by another writer, and an already saved workspace are not retained, and the run row does not change. (The writer and saved cases fail if the new guard is removed.) - Preservation does not replay the stopped run, does not create a wake, keeps the stop receipt and lease policy unchanged, and keeps the writer hold until the save publishes. - An exhausted or complete save does not extend the cutoff. The existing past-cutoff test (no save in progress) still fails setup and keeps the held files. - Related suites: workspace recovery, remote workspace reservation, native workspace reconstruction, native failed-workspace recovery, legacy workspace restore recovery. - Not verified on staging: this change is not deployed. The staging task needs this fix deployed. Then the preservation worker records the save, workspace recovery saves the files, and the waiting owner continues. ## Risks - More cancelled runs are now admitted for preservation. The retention guard requires the exact stop-and-retain receipt and the workspace bound to this writer, so a reusable sandbox that another run resumed or that no longer holds unsaved work is not retained. - A stopped reusable source stays `reuse_by_environment` while Paperclip saves it. This matches the existing retention path for reusable native sources. - The writer-hold read adds one indexed query (`issue_recovery_actions_source_run_idx`) for an ended writer only. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, code editing, test execution) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- doc/execution-semantics.md | 13 ++++ .../heartbeat-remote-workspace-busy.test.ts | 45 ++++++++++- ...e-cancelled-workspace-preservation.test.ts | 76 +++++++++++++++++++ .../native-failed-workspace-recovery.ts | 29 +++++-- .../services/remote-workspace-writer-hold.ts | 27 +++++-- 5 files changed, 178 insertions(+), 12 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 6d50ff984fd..538c675cd66 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -502,6 +502,19 @@ This wait is also one parked retry, for example after a reassignment stops the writer. It is re-checked every minute while the ended writer's hold settles, is admitted once the hold releases, and stops parking at the same cutoff. +A writer stopped mid-run, for example by reassignment, keeps its unsaved files +in its stopped sandbox. When the run used a reusable (warm) sandbox, the stop +leaves that lease `released` under its reusable policy with a stop-and-retain +receipt. Cancelled-workspace preservation admits that stopped source, as it does +an active or ephemeral one. It records the save obligation for the exact writer +run while the workspace is still bound to that writer. Paperclip's workspace +recovery then saves those files, which releases the hold. It never replays the +stopped run. While that save is in progress, its last progress counts as the +ended writer's activity for the holder cutoff. A waiter keeps waiting instead of +failing setup. The save's retry budget bounds that wait. A save that exhausts +its retries is handed to the board as a named recovery action. After that hand-off, +the cutoff no longer advances. + An instance admission drain keeps accepted new turns queued and displays **Waiting for maintenance**. Ending the audited drain requests admission for these existing runs immediately. The shared delivery coordinator owns the drain's diff --git a/server/src/__tests__/heartbeat-remote-workspace-busy.test.ts b/server/src/__tests__/heartbeat-remote-workspace-busy.test.ts index b0d106027f1..7a404ad6e36 100644 --- a/server/src/__tests__/heartbeat-remote-workspace-busy.test.ts +++ b/server/src/__tests__/heartbeat-remote-workspace-busy.test.ts @@ -5,7 +5,7 @@ import { join } from "node:path"; import { eq } from "drizzle-orm"; import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { agents, authUsers, companies, companyMemberships, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, - environmentLeases, environments, executionWorkspaces, heartbeatRuns, issues, type Db } from "@paperclipai/db"; + environmentLeases, environments, executionWorkspaces, heartbeatRuns, issueRecoveryActions, issues, type Db } from "@paperclipai/db"; import { readRemoteWorkspaceAuthority } from "@paperclipai/shared"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; import { heartbeatService } from "../services/heartbeat.js"; @@ -213,6 +213,49 @@ describe.skipIf(!support.supported)("inherited remote workspace heartbeat wait", await heartbeat.drainActiveRunExecutions(); }); + // A writer stopped by reassignment whose retained files Paperclip is still + // saving (an in-progress workspace recovery) is still settling, even after the + // one-hour cutoff from its end. A waiter already parked for it stays parked + // instead of being released into a setup failure; the save's own retry budget + // bounds the wait. A finished or board-handed-off save no longer extends it. + it.each([ + { state: undefined, waits: true }, + { state: "scheduled", waits: true }, + { state: "awaiting_evidence", waits: true }, + { state: "exhausted", waits: false }, + { state: "complete", waits: false }, + ] as const)("keeps a waiter parked while an ended writer's save is in progress ($state)", async ({ state, waits }) => { + const f = await terminalWriterFixture(new Date()); + const [retry] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.run.id)); + expect(retry).toMatchObject({ status: "scheduled_retry", scheduledRetryReason: "workspace_busy" }); + // The writer ended over an hour ago; its save recovery recorded progress a minute ago. + const endedAt = new Date(Date.now() - 2 * 60 * 60_000); + await db.update(heartbeatRuns).set({ status: "cancelled", errorCode: "issue_reassigned", finishedAt: endedAt, + startedAt: new Date(endedAt.getTime() - 60_000), lastOutputAt: endedAt }).where(eq(heartbeatRuns.id, f.writer.id)); + await db.insert(issueRecoveryActions).values({ companyId: f.run.companyId, sourceIssueId: f.workspace.sourceIssueId!, + kind: "active_run_watchdog", status: "resolved", outcome: "blocked", ownerType: "board", + cause: "native_continuation_requires_reconciliation", fingerprint: `native-workspace:${f.writer.id}`, + resolvedAt: new Date(), updatedAt: new Date(Date.now() - 60_000), nextAction: "Save the retained files.", + evidence: { runId: f.writer.id, workspaceRestoreFailure: "restore_failed", + workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [randomUUID()] }, + automaticRecovery: { policy: "preserve_without_replay_v1", runId: f.writer.id, replay: "blocked" }, + ...(state ? { systemRecovery: { state } } : {}) } }); + const checkAt = new Date(retry.scheduledRetryAt!.getTime() + 1_000); + const promoted = (await heartbeat.promoteDueScheduledRetries(checkAt)).runIds; + if (waits) { + expect(promoted).not.toContain(retry.id); + const parked = await heartbeat.getRun(retry.id); + expect(parked?.status).toBe("scheduled_retry"); + // Bounded by the cutoff measured from the save's last progress. + expect(new Date(parked!.scheduledRetryAt!).getTime()).toBeLessThanOrEqual(Date.now() + 60 * 60_000); + } else expect(promoted).toContain(retry.id); + await heartbeat.cancelRun(retry.id); + await heartbeat.drainActiveRunExecutions(); + // Waiting never takes over or discards the unsaved writer's files. + expect(readRemoteWorkspaceAuthority((await db.select().from(executionWorkspaces) + .where(eq(executionWorkspaces.id, f.workspace.id)))[0].metadata)).toEqual(f.authority); + }); + it("stops waiting for an ended writer whose save stayed unsettled past the holder cutoff", async () => { const f = await terminalWriterFixture(new Date(Date.now() - 2 * 60 * 60_000)); expect(f.run).toMatchObject({ status: "failed", error: "The workspace still has an unfinished writer or save operation." }); diff --git a/server/src/services/native-runtime/native-cancelled-workspace-preservation.test.ts b/server/src/services/native-runtime/native-cancelled-workspace-preservation.test.ts index 0916bd8c90f..91579705ac4 100644 --- a/server/src/services/native-runtime/native-cancelled-workspace-preservation.test.ts +++ b/server/src/services/native-runtime/native-cancelled-workspace-preservation.test.ts @@ -10,6 +10,9 @@ import { requestWorkspaceRecovery } from "../workspace-recovery.js"; import { inspectWorkspaceRecoverySource } from "../workspace-recovery-source.js"; import { preserveCancelledNativeWorkspaces, preserveNativeFailedWorkspace, registerCancelledWorkspacePreservationWork } from "./native-failed-workspace-recovery.js"; +import { prepareSandboxStopAndRetain, settleStopOnlyCleanup } from "../sandbox-stop-and-retain.js"; +import { readRemoteWorkspaceWriterHold } from "../remote-workspace-writer-hold.js"; +import { readRemoteWorkspaceAuthority } from "@paperclipai/shared"; (recoveryPostgresSupport.supported ? describe : describe.skip)("lifecycle-cancelled workspace preservation", () => { let database: Awaited>; @@ -158,6 +161,79 @@ import { preserveCancelledNativeWorkspaces, preserveNativeFailedWorkspace, } finally { await worker.stop(); } }, 120_000); + // A warm (reusable) run that ends without success stops its sandbox and keeps + // the lease `released` under its reusable policy. Reassignment Stop reaches + // that state with the workspace still dirty and bound to the stopped writer. + async function stopReusableSource(f: Awaited>) { + const [reusable] = await db.update(environmentLeases).set({ leasePolicy: "reuse_by_environment" }) + .where(eq(environmentLeases.id, f.leaseId)).returning(); + const stopping = (await prepareSandboxStopAndRetain(db, reusable))!; + const stopped = (await settleStopOnlyCleanup(db, stopping, { + attemptId: String(stopping.metadata!.pendingCleanupAttemptId), + receipt: { providerLeaseId: f.reference.providerLeaseId, state: "stopped" }, + }))!; + expect(stopped).toMatchObject({ status: "released", leasePolicy: "reuse_by_environment", cleanupStatus: "success" }); + return stopped; + } + + it.each(["worker", "startup-scan"] as const)("preserves a reassignment-stopped writer's files from its stopped reusable sandbox (%s)", async kind => { + const f = await seed(); + const resultJson = { ...f.run.resultJson, reassignmentStopRequested: true }; + await db.update(heartbeatRuns).set({ errorCode: "issue_reassigned", resultJson }).where(eq(heartbeatRuns.id, f.runId)); + const stopped = await stopReusableSource(f); + const worker = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => { throw error; } }); + try { + if (kind === "startup-scan") await preserveCancelledNativeWorkspaces(db); + else await registerCancelledWorkspacePreservationWork(db, worker, () => true, async () => true).ready; + const [retained] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + expect(retained.resultJson).toMatchObject({ ...resultJson, workspaceRestoreFailure: "restore_failed", + workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [f.leaseId] } }); + expect(retained).toMatchObject({ status: "cancelled", errorCode: "issue_reassigned", finishedAt: f.run.finishedAt }); + // The stop receipt and reusable policy stay exactly as the release left them. + expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId)))[0]).toEqual(stopped); + const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.companyId, f.companyId)); + expect(action).toMatchObject({ ownerType: "board", evidence: { runId: f.runId, + automaticRecovery: { replay: "blocked" }, workspaceRestoreRecovery: { leaseIds: [f.leaseId] } } }); + expect((await inspectWorkspaceRecoverySource(db, { companyId: f.companyId, issueId: f.issueId, + runId: f.runId, actionId: action.id })).sources).toHaveLength(1); + // Preservation records the save obligation only. The stopped writer keeps + // the workspace until the saved files are published; nothing replays it. + const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, f.workspaceId)); + expect(await readRemoteWorkspaceWriterHold(db, { companyId: f.companyId, + authority: readRemoteWorkspaceAuthority(workspace.metadata)! })).toMatchObject({ kind: "held", writer: { runId: f.runId, ended: true } }); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(0); + expect(worker.nextWakeAt()).toBeNull(); + } finally { await worker.stop(); } + }, 120_000); + + it.each(["unconfirmed-stop", "writer-released", "workspace-saved"] as const)("does not retain a reusable source (%s)", async kind => { + const f = await seed(); + let released: typeof environmentLeases.$inferSelect; + if (kind === "unconfirmed-stop") { + [released] = await db.update(environmentLeases).set({ leasePolicy: "reuse_by_environment", status: "released", + cleanupStatus: "success", releasedAt: new Date() }).where(eq(environmentLeases.id, f.leaseId)).returning(); + } else { + released = await stopReusableSource(f); + // The workspace moved on: another writer owns it, or the writer's save already published. + const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, f.workspaceId)); + const authority = readRemoteWorkspaceAuthority(workspace.metadata)!; + await db.update(executionWorkspaces).set({ metadata: { ...workspace.metadata, remoteWorkspace: kind === "writer-released" + ? { ...authority, writerRunId: randomUUID() } : { ...authority, writerRunId: null, dirty: false } } }) + .where(eq(executionWorkspaces.id, f.workspaceId)); + } + const before = (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]; + await preserveCancelledNativeWorkspaces(db); + const worker = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => { throw error; } }); + try { await registerCancelledWorkspacePreservationWork(db, worker, () => true, async () => true).ready; } + finally { await worker.stop(); } + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]).toEqual(before); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + expect(run.resultJson).not.toHaveProperty("workspaceRestoreRecovery"); + expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId)))[0]).toEqual(released); + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.companyId, f.companyId))).toHaveLength(0); + }); + it.each(["pending", "missing-dispatch", "invalid-dispatch", "foreign-company", "foreign-issue", "foreign-run", "turn-scope", "empty-intent-audit", "missing-ack-audit", "empty-ack-audit", "live-owner", "live-controller", "accepted-result", "foreign-allocation", "foreign-workspace", "foreign-cwd", "saved-reference", "unknown-reference", "running"] as const) diff --git a/server/src/services/native-runtime/native-failed-workspace-recovery.ts b/server/src/services/native-runtime/native-failed-workspace-recovery.ts index 9681695bef3..1c95720902a 100644 --- a/server/src/services/native-runtime/native-failed-workspace-recovery.ts +++ b/server/src/services/native-runtime/native-failed-workspace-recovery.ts @@ -16,6 +16,10 @@ import { logger } from "../../middleware/logger.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js"; import { preserveRemoteWorkspacePreparationFailure } from "./native-workspace-finalization-state.js"; import { pinFailedObservedWorkspace } from "./native-observed-workspace-preservation.js"; +import { hasConfirmedSandboxStopAndRetain } from "../sandbox-stop-and-retain.js"; +import { preparedRemoteWorkspaceSource } from "../remote-workspace-lifecycle.js"; + +const reusableLeasePolicies = ["reuse_by_environment", "reuse_by_execution_workspace"]; function workspaceRecoveryReference(profile: Record | null | undefined) { @@ -93,6 +97,10 @@ export async function preserveNativeFailedWorkspace(db: Db, selected: Pick(); +/** Leases that may still hold a cancelled writer's prepared files. A reusable + * (warm) lease that ends without success is stopped and kept `released` under + * its reusable policy; a reassignment Stop leaves the writer's unsaved files + * there. Admission only finds candidates: retention still requires the exact + * stop-and-retain receipt and the workspace still bound to this writer. */ +function cancelledWorkspaceSourceLease() { + return or(eq(environmentLeases.status, "active"), + and(inArray(environmentLeases.status, ["pending_cleanup", "released"]), eq(environmentLeases.leasePolicy, "ephemeral")), + and(eq(environmentLeases.status, "released"), eq(environmentLeases.cleanupStatus, "success"), + inArray(environmentLeases.leasePolicy, reusableLeasePolicies), + sql`${environmentLeases.metadata}->'remoteExecutionTermination'->>'state' = 'stopped'`, + sql`${environmentLeases.metadata} ? 'sandboxStopAndRetainReceipt'`)); +} + /** Older controllers could acknowledge Stop without pinning prepared files. * Record only the file obligation; the existing stop-only cleanup worker owns * provider teardown. This never resumes a turn or changes the task's status. */ @@ -178,9 +200,7 @@ export async function preserveCancelledNativeWorkspaces(db: Db) { eq(heartbeatRuns.id, environmentLeases.heartbeatRunId), eq(heartbeatRuns.companyId, environmentLeases.companyId), )).innerJoin(nativeRunFinalizations, and(eq(nativeRunFinalizations.runId, heartbeatRuns.id), eq(nativeRunFinalizations.companyId, heartbeatRuns.companyId), eq(nativeRunFinalizations.issueId, heartbeatRuns.nativeIssueId))) - .where(and(or(eq(environmentLeases.status, "active"), and( - inArray(environmentLeases.status, ["pending_cleanup", "released"]), eq(environmentLeases.leasePolicy, "ephemeral"), - )), eq(heartbeatRuns.runtimeMode, "native"), + .where(and(cancelledWorkspaceSourceLease(), eq(heartbeatRuns.runtimeMode, "native"), eq(heartbeatRuns.status, "cancelled"), eq(nativeRunFinalizations.phase, "terminal_failure"), isNull(nativeRunFinalizations.resultId), isNull(nativeRunFinalizations.leaseOwner), sql`(${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync'->>'state' = 'prepared' OR ${heartbeatRuns.runnerProfileJson}->'remoteWorkspace'->>'state' = 'prepared')`, @@ -259,8 +279,7 @@ export function registerCancelledWorkspacePreservationWork(db: Db, return { ...entry, dueAt: Date.now() + 30_000 }; const [source] = await db.select({ id: environmentLeases.id }).from(environmentLeases).where(and( eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id), - or(eq(environmentLeases.status, "active"), and(inArray(environmentLeases.status, ["pending_cleanup", "released"]), - eq(environmentLeases.leasePolicy, "ephemeral"))), + cancelledWorkspaceSourceLease(), )).limit(1); if (source) await preserveNativeFailedWorkspace(db, run, { allowPreterminal: true, requireCancelledStop: true }); const [current] = await db.select().from(heartbeatRuns) diff --git a/server/src/services/remote-workspace-writer-hold.ts b/server/src/services/remote-workspace-writer-hold.ts index fa4c69bd4a1..6fa682ab7c3 100644 --- a/server/src/services/remote-workspace-writer-hold.ts +++ b/server/src/services/remote-workspace-writer-hold.ts @@ -1,5 +1,5 @@ import { and, eq, sql } from "drizzle-orm"; -import { environmentLeases, heartbeatRuns, issues, type Db } from "@paperclipai/db"; +import { environmentLeases, heartbeatRuns, issueRecoveryActions, issues, type Db } from "@paperclipai/db"; import type { RemoteWorkspaceAuthority } from "@paperclipai/shared"; import { hasRecoveredCleanupStopReceipt, hasRemoteTerminationReceipt } from "./remote-execution-termination.js"; import { canonicalExecutionUuid } from "./execution-identity.js"; @@ -55,10 +55,25 @@ export async function readRemoteWorkspaceWriterHold(tx: Db, input: { && lease.releasedAt && lease.cleanupStatus === "success" && ["released", "expired", "failed"].includes(lease.status))); if (unusedReservation) return { kind: "free" }; // An ended writer still holds the workspace until its save and sandbox - // release settle, within the same bounded cutoff as a live writer. - if (prior && writerIssue) return { kind: "held", writer: { - runId: authority.writerRunId, agentId: prior.agentId, issueId: writerIssue.id, issueIdentifier: writerIssue.identifier, - lastActivityAt: prior.finishedAt ?? prior.lastOutputAt ?? prior.startedAt ?? prior.createdAt, ended: true, - } }; + // release settle, within the same bounded cutoff as a live writer. While + // Paperclip is still saving the writer's retained files, that save's last + // progress is the writer's activity, so a waiter keeps waiting instead of + // failing setup. The save's own retry budget bounds it; a completed or + // board-handed-off save no longer extends the cutoff. + if (prior && writerIssue) { + const ended = new Date(prior.finishedAt ?? prior.lastOutputAt ?? prior.startedAt ?? prior.createdAt); + const [save] = await tx.select({ progressedAt: sql`max(${issueRecoveryActions.updatedAt})` }) + .from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, writerIssue.id), + sql`${issueRecoveryActions.evidence}->>'runId' = ${authority.writerRunId}`, + sql`${issueRecoveryActions.evidence}->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1'`, + sql`coalesce(${issueRecoveryActions.evidence}#>>'{systemRecovery,state}', 'scheduled') in ('scheduled', 'running', 'awaiting_evidence')`, + sql`${issueRecoveryActions.evidence}#>>'{workspaceRecovery,status}' is distinct from 'recovered'`)); + const saving = save?.progressedAt ? new Date(save.progressedAt) : null; + return { kind: "held", writer: { + runId: authority.writerRunId, agentId: prior.agentId, issueId: writerIssue.id, issueIdentifier: writerIssue.identifier, + lastActivityAt: saving && saving.getTime() > ended.getTime() ? saving : ended, ended: true, + } }; + } return { kind: "unattributed" }; }