From 2b374caba01e53258d90f423b20a15666e946ae7 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 2 Nov 2026 16:03:09 -0600 Subject: [PATCH] fix(recovery): preserve unfinished remote exports before Retry (#18678) ## Thinking Path > - Paperclip manages agent work and preserves each task's saved files. > - A completed native result can still need workspace export. > - Remote workspaces now use a different saved descriptor from legacy copyback. > - The execution hold checked only the legacy descriptor. > - Task Retry could admit another run before the original export finished. > - A later failure could then supersede the original recovery record. > - This change keeps unfinished remote exports fenced and permits explicit historical file recovery. > - Recovery keeps human review and newer execution authority intact. ## Linked Issues or Issue Description **What happened?** A native run completed successfully, but its remote workspace export failed. The recovery action used `native_workspace_sync_out_retry_exhausted`. Task Retry created a second task run while the original files still needed preservation. The newer failure then superseded the original action. A board file-recovery request could queue that cancelled record, but the worker excluded it from discovery. **Expected behavior** Keep Retry behind the unfinished export. Save the exact source and finalize the existing accepted result once. Preserve pending human review. An explicit request for a superseded source must save its files without applying an old result over newer work. **Steps to reproduce** 1. Save an accepted successful native result with a prepared `remoteWorkspace` descriptor. 2. Fail its final export and retain its exact sandbox lease. 3. Select Task Retry before preservation completes. 4. Observe another task run before the original workspace is saved. 5. Supersede the original action and request file recovery for its exact source. Observe queued work that the worker does not select. **Paperclip version** Observed at `1f8f2e195f4897cd1c237bdc068de95780ab0c3c`. The regression is present on current master before this patch. **Deployment mode** Authenticated hosted instance with native Paperclip Runner and remote sandbox workspaces. Related: #18403 introduced the remote workspace path. #18644 reduces export scratch disk use. #17916 covers a separate legacy warm-copyback policy case. This change repairs remote export admission and explicit historical preservation. ## What Changed - Include prepared remote descriptors in the existing native export hold. - Retain remote source identity in the blocker and continuation read projections. - Let the existing worker select a cancelled export-repair record only after an explicit board file-recovery request. Keep the current action and workspace binding unchanged. - Add real PostgreSQL, HTTP, filesystem and Git regressions. Update execution semantics and recovery maps. ## Verification - Red before the admission fix: HTTP Retry creates two task runs instead of one. - Red before the historical-source fix: the explicit recovery stays scheduled and saves no snapshot. - All 588 focused tests pass across the workspace recovery, recovery source, explicit continuation and native export retry suites. The three new integration cases verify duplicate request identity, original result finalization, restored bytes and Git HEAD, pending human review, current-writer exclusion, and preservation without older-result commitment over newer work. - `pnpm -r typecheck`, `pnpm build`, map validation and `git diff --check` pass. - Full hosted CI passes on `c598718379fa939ec539e8e91a15a94d90321b59`, including all server, workspace, serialized, Runner and browser lanes ([run](https://github.com/paperclipai/paperclip/actions/runs/41529739604)). Greptile reviewed that commit at 5/5 with no findings; there are no unresolved review threads. Paperclip Review also published a complete 5/5 assessment for that exact commit with zero findings. Its check conclusion is advisory/neutral, titled `5/5`. The first attempt lacked a structured assessment; one normal completion request recovered the saved analysis and published the validated assessment. - The local `pnpm test:run` passed all six chat partitions, then was stopped during the general server group after equivalent full CI completed. Its partial log is retained; it is not reported as a completed local full-suite pass. - The test provider boundary uses real temporary files and the production archive path. It simulates provider resume/stop calls. The deployed fix and original hosted task recovery remain unverified until merge and rollout. - The older local Node 24.11.0 runtime failed to load the installed tsx worker. Node 26.4.0 passed the isolated worker check. Required repository checks use an isolated Node 24.22.2 runtime. The earlier failure logs remain recorded. ## Risks - Recognizing another unfinished export can keep previously admitted work waiting. This is intentional until persistence is verified. - Explicit historical recovery can resume the exact stopped sandbox to read files. It retains source, allocation, ownership, writer and bounded-attempt checks. It starts no provider turn. - A newer run still prevents commitment of an older accepted result. This change does not infer that a cancelled run had no external effects. - No schema, credential, deployment or fleet configuration changes. ## Model Used OpenAI Codex, GPT-6 family. The runtime does not expose the exact model ID or context-window size. Reasoning, repository inspection, tool use and code execution assisted this change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- doc/execution-semantics.md | 14 ++ maps/agent-access-map/recovery.md | 6 + maps/feature-map/recovery.md | 11 ++ server/src/services/execution-blocker.ts | 2 +- .../services/explicit-native-continuation.ts | 2 +- .../native-workspace-reconstruction.ts | 3 +- .../src/services/workspace-recovery-source.ts | 4 + .../src/services/workspace-recovery.test.ts | 147 ++++++++++++++++++ 8 files changed, 186 insertions(+), 3 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index a72120c36e6..5889dc4634a 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -1724,6 +1724,20 @@ sources without this proof remain ineligible. The preservation owner still requires an independently stopped, exclusively bound allocation and no host writer; it saves files and resumes the existing finalizer without another provider turn. +Both the legacy native copyback descriptor and the remote-workspace descriptor +keep an unfinished export under the execution hold. Task Retry saves its intent +behind that hold; an accepted successful result resumes its existing finalizer +after verified preservation instead of admitting another task run. Pending human +review remains effective when the result commits. + +If a later failure supersedes the original recovery action, an explicit board +`recover-workspace` request can still preserve that exact historical export. +The cancelled action remains historical and the current action retains attention +ownership. Preservation checks current writers and source identity, records a +durable snapshot, and leaves the task's newer workspace binding unchanged. It +does not apply an older accepted result over a newer run or authorize continuation. +Cancelled history without a saved explicit recovery request remains ineligible. + Startup and the retained-workspace sweep reconstruct these obligations through the retained workspace index. Committed Retry, recovery admission, and recovery progress writes wake a separate preservation worker. It follows the persisted diff --git a/maps/agent-access-map/recovery.md b/maps/agent-access-map/recovery.md index d551c6b1824..18894d62573 100644 --- a/maps/agent-access-map/recovery.md +++ b/maps/agent-access-map/recovery.md @@ -326,6 +326,12 @@ API: `present` — POST /api/issues/{id}/recovery-actions/recover-workspace with Agent access: `restricted` — board task-read and issue:mutate; not runtime:manage. Agents denied through REST/native fallback. +An explicit request may also preserve a cancelled export-repair action that a +newer failure superseded. It retains the exact original source checks, current +writer exclusion and bounded maintenance budget. It does not change the newer +task workspace or recovery owner, apply an older result over newer execution, +or admit a provider turn. A cancelled record without that request is ineligible. + Policy: Exact terminal retained source/lease/reference; independent stop and physical ownership checks; no newer writer. Bounded queued repair preserves newer files and has retry state. Matching key/recovered state dedupes; permanent blocked source refuses. Recovery starts no provider turn. For native export repair, a reusable lease is eligible only with its exact descriptor and the finalizer-selected accepted successful result under the same company/task/run/completion contract. This does not change actor access or permit another allocation owner or host writer. Invalid Git metadata is a non-retryable persistence failure requiring service repair; it cannot use unsafe-archive omission to complete the result. Run-detail/activity projections retain the allowlisted code, including bounded large-result retrieval, and cancelled-preservation discovery honors the same hold. Evidence: `source-only` — [recovery routes](../../server/src/routes/issues.ts), [retained recovery](../../server/src/services/workspace-recovery.ts); reviewed at `3bbc2104b65caf9f7763938a01260a5673e12f7a`. No live provider proof. diff --git a/maps/feature-map/recovery.md b/maps/feature-map/recovery.md index b97ac0d9a08..1ba9594a4d7 100644 --- a/maps/feature-map/recovery.md +++ b/maps/feature-map/recovery.md @@ -429,6 +429,17 @@ Verify saved and newer local files, one preservation receipt, and no additional agent turn. The PostgreSQL regression covers this boundary; the opt-in provider probe also uses this runtime path with the protection marker. +Repeat Task Retry with a completed native result whose unfinished export uses +the remote-workspace descriptor. Before preservation, require the original run +to remain the only task run. After preservation, verify restored file bytes and +Git HEAD, commitment of that same result, and an unanswered human review keeping +the task In Review. For an action superseded by a newer failure, request exact +historical file recovery through the board API: verify a committed snapshot, +unchanged newer workspace binding and attention owner, no old-result commitment, +and no provider turn. An active current writer or no explicit request keeps the +historical source waiting. The PostgreSQL/HTTP regression is in +[workspace recovery](../../server/src/services/workspace-recovery.test.ts). + For a missing managed allocation, verify a fresh Cloud-signed absence observation under the exact physical recovery owner. The existing Retry may then select an independent workspace through normal admission. Verify that the original run diff --git a/server/src/services/execution-blocker.ts b/server/src/services/execution-blocker.ts index c22437f1a47..4e57d9b0c88 100644 --- a/server/src/services/execution-blocker.ts +++ b/server/src/services/execution-blocker.ts @@ -16,7 +16,7 @@ import { hasVerifiedWorkspaceRepair, workspaceRecoveryDisplay } from "./workspac const blockerContextKeys = ["issueId", "taskId", "source", "connectionIntentResolution", "interactionKind", "interactionStatus", "mutation", "interactionId"] as const; -const blockerProfileKeys = ["adapterDispatch", "nativeExecutionInput", "nativeWorkspaceSync"] as const; +const blockerProfileKeys = ["adapterDispatch", "nativeExecutionInput", "nativeWorkspaceSync", "remoteWorkspace"] as const; const blockerContextColumn = sql | null>`case when ${heartbeatRuns.contextSnapshot} is null then null else ( select coalesce(jsonb_object_agg(field.key, field.value), '{}'::jsonb) diff --git a/server/src/services/explicit-native-continuation.ts b/server/src/services/explicit-native-continuation.ts index 084a25e457f..c93ec6b8681 100644 --- a/server/src/services/explicit-native-continuation.ts +++ b/server/src/services/explicit-native-continuation.ts @@ -52,7 +52,7 @@ const preflightProfile = sql`case then jsonb_build_object('adapterType', field.value->'adapterType') else field.value end), '{}'::jsonb) from jsonb_each(case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object' then ${heartbeatRuns.runnerProfileJson} else '{}'::jsonb end) as field(key, value) - where field.key in ('adapterDispatch', 'nativeWorkspaceSync') + where field.key in ('adapterDispatch', 'nativeWorkspaceSync', 'remoteWorkspace') ) end`; const continuationRunFields = { id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, diff --git a/server/src/services/native-runtime/native-workspace-reconstruction.ts b/server/src/services/native-runtime/native-workspace-reconstruction.ts index 2daf462972f..aa20b852e14 100644 --- a/server/src/services/native-runtime/native-workspace-reconstruction.ts +++ b/server/src/services/native-runtime/native-workspace-reconstruction.ts @@ -36,7 +36,8 @@ export function unreconciledNativeExportPredicate() { and ${heartbeatRuns.nativeIssueId} = ${issueRecoveryActions.sourceIssueId} and ${heartbeatRuns.id}::text = ${issueRecoveryActions.evidence}->>'runId' and ${heartbeatRuns.runtimeMode} = 'native' and ${heartbeatRuns.status} = 'failed' - and ${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync'->>'state' = 'prepared')`, + and (${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync'->>'state' = 'prepared' + or ${heartbeatRuns.runnerProfileJson}->'remoteWorkspace'->>'state' = 'prepared'))`, ); } diff --git a/server/src/services/workspace-recovery-source.ts b/server/src/services/workspace-recovery-source.ts index 3cf14003e49..ad65d438d2b 100644 --- a/server/src/services/workspace-recovery-source.ts +++ b/server/src/services/workspace-recovery-source.ts @@ -25,6 +25,10 @@ export class WorkspaceSourceInspectionLimit extends Error {} export function workspaceRecoverySourceEligible(includeCompleted = false) { const actionEligible = sql`(${issueRecoveryActions.status} in ('active', 'escalated') or + (${issueRecoveryActions.status} = 'cancelled' + and ${inArray(issueRecoveryActions.cause, workspaceExportCauses)} + and jsonb_typeof(${issueRecoveryActions.evidence}#>'{workspaceRecovery,requestIds}') = 'array' + and ${issueRecoveryActions.evidence}#>'{workspaceRecovery,requestIds}' <> '[]'::jsonb) or (${issueRecoveryActions.status} = 'resolved' and ( (${issueRecoveryActions.evidence}#>>'{automaticRecovery,policy}' = 'preserve_without_replay_v1' and ${issueRecoveryActions.evidence}#>>'{automaticRecovery,replay}' = 'blocked' diff --git a/server/src/services/workspace-recovery.test.ts b/server/src/services/workspace-recovery.test.ts index 8648acad550..5c732d4f922 100644 --- a/server/src/services/workspace-recovery.test.ts +++ b/server/src/services/workspace-recovery.test.ts @@ -49,6 +49,10 @@ import { and, eq, inArray, sql } from "drizzle-orm"; import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; import { agents, approvals, budgetPolicies, costEvents, completionContracts, issueApprovals, issueTreeHolds, issueTreeHoldMembers, companies, createDb, applyPendingMigrations, environmentLeases, environments, heartbeatRuns, issueComments, issueRecoveryActions, issues, nativeRunFinalizations, nativeRunResults, agentWakeupRequests, workspaceOperations } from "@paperclipai/db"; import { resumePreservedResultFinalization } from "./preserved-result-finalization.js"; +import { finalizeNativeRun } from "./native-runtime/native-run-finalizer.js"; +import { CONTROL_PLANE_CONFORMANCE_RESULT, CONTROL_PLANE_CONFORMANCE_TERMINAL } from "../vendor/paperclip-runner/testing.js"; +import { issueThreadInteractions } from "@paperclipai/db"; +import { issueRecoveryActionService } from "./issue-recovery-actions.js"; import type { AdapterSandboxExecutionTarget } from "@paperclipai/adapter-utils/execution-target"; import type { EnvironmentLease } from "@paperclipai/shared"; import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js"; @@ -190,6 +194,149 @@ describe("automatic retained workspace recovery", () => { const action = async (id: string) => (await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, id)))[0]!; const run = async (id: string) => (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)))[0]!; + function boardRecoveryApp() { + const app = express(); + app.use(express.json()); + app.use((req, _res, next) => { req.actor = { type: "board", source: "local_implicit", userId: "local-board" }; next(); }); + app.use("/api", issueRoutes(db, {} as Parameters[1])); + app.use(errorHandler); + return app; + } + + async function acceptedRemoteExport() { + const f = await seed("native", false); + const source = await run(f.runId), workspaceId = randomUUID(); + const remoteCwd = await realpath(f.remote); + const git = (args: string[], cwd = remoteCwd) => new Promise((resolve, reject) => + execFile("git", args, { cwd }, (error, stdout) => error ? reject(error) : resolve(stdout.trim()))); + await git(["init"]); + await git(["add", "."]); + await git(["-c", "user.name=Recovery fixture", "-c", "user.email=recovery@example.invalid", "commit", "-m", "Completed work"]); + const commit = await git(["rev-parse", "HEAD"]); + targets.get(f.lease.id)!.remoteCwd = remoteCwd; + f.runtime.resumeRunLease.mockImplementation(async () => ({ providerLeaseId: f.providerLeaseId, metadata: { remoteCwd } })); + const authority = { schema: "paperclip.remote-workspace.v1", epoch: 1, snapshotId: null, + source: { kind: "empty" }, writerRunId: f.runId, dirty: true, + location: { environmentId: f.lease.environmentId!, leaseId: f.lease.id, providerLeaseId: f.providerLeaseId, remoteCwd } }; + await db.insert(executionWorkspaces).values({ id: workspaceId, companyId: f.companyId, sourceIssueId: f.issueId, + name: "Completed remote work", cwd: remoteCwd, providerType: "cloud_sandbox", mode: "isolated_workspace", + strategyType: "project_primary", status: "active", metadata: { remoteWorkspace: authority } }); + await db.update(issues).set({ executionWorkspaceId: workspaceId }).where(eq(issues.id, f.issueId)); + const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)); + await db.update(environmentLeases).set({ executionWorkspaceId: workspaceId, leasePolicy: "reuse_by_environment", + metadata: { ...lease.metadata, remoteCwd, agentId: source.agentId } }).where(eq(environmentLeases.id, lease.id)); + const contractId = randomUUID(), resultId = randomUUID(); + await db.insert(completionContracts).values({ id: contractId, companyId: f.companyId, issueId: f.issueId, + revision: 1, schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test", risk: "low", + completionAuthority: "agent_claim_policy", incompleteCriteriaPolicy: "preserve_non_terminal", + contractJson: { revision: CONTROL_PLANE_CONFORMANCE_RESULT.completionClaim.contractRevision, + objective: "Complete the work", criteria: [{ id: "objective", requirement: "Complete the task" }] }, + canonicalSha256: "contract", createdByActorType: "system", createdByActorId: "test" }); + await db.update(heartbeatRuns).set({ completionContractId: contractId, resultJson: {}, runnerProfileJson: { + adapterDispatch: { adapterType: "paperclip_runner" }, + remoteWorkspace: { schema: "paperclip.remote-workspace-run.v1", workspaceId, leaseId: lease.id, + providerLeaseId: f.providerLeaseId, remoteCwd, epoch: 1, state: "prepared", snapshotId: null }, + } }).where(eq(heartbeatRuns.id, f.runId)); + await db.insert(nativeRunResults).values({ id: resultId, companyId: f.companyId, issueId: f.issueId, runId: f.runId, + completionContractId: contractId, schemaStatus: "accepted", serverFingerprint: "result", canonicalSha256: "result", + resultJson: { terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL, result: CONTROL_PLANE_CONFORMANCE_RESULT } }); + await db.update(nativeRunFinalizations).set({ resultId, failureCode: "native_workspace_sync_out_retry_exhausted" }) + .where(eq(nativeRunFinalizations.runId, f.runId)); + await db.update(issueRecoveryActions).set({ cause: "native_workspace_sync_out_retry_exhausted", evidence: { runId: f.runId } }) + .where(eq(issueRecoveryActions.id, f.actionId)); + await adoptSystemRecovery(db, f); + return { ...f, source, workspaceId, resultId, git, commit }; + } + + it.each([false, true])("keeps Retry behind an accepted remote export and resumes only its saved result (review pending: %s)", async reviewPending => { + const f = await acceptedRemoteExport(); + const { source, workspaceId, resultId, git, commit } = f; + const interactionId = randomUUID(); + if (reviewPending) await db.insert(issueThreadInteractions).values({ id: interactionId, companyId: f.companyId, + issueId: f.issueId, sourceRunId: f.runId, createdByAgentId: source.agentId, kind: "request_confirmation", + title: "Review saved work", effectiveResolverPolicy: "human_only", + payload: { version: 1, prompt: "Review before publishing", rejectRequiresReason: true, + target: { type: "custom", key: "layout_review", label: "Review saved work" } } }); + await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: source.agentId, status: "running" }); + const app = boardRecoveryApp(); + const intent = { observedRunId: f.runId, idempotencyKey: randomUUID() }; + const retry = (await request(app).post(`/api/issues/${f.issueId}/retry`).send(intent).expect(202)).body; + expect((await request(app).post(`/api/issues/${f.issueId}/retry`).send(intent).expect(202)).body.requestId).toBe(retry.requestId); + await heartbeatService(db).resumeTaskRetry(f.companyId, retry.requestId!); + expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), + sql`coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.issueId}::text, + ${heartbeatRuns.contextSnapshot}->>'issueId') = ${f.issueId}`))).toHaveLength(1); + expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ runId: f.runId, workspaceRepairRequired: true }); + await Promise.all(await reconcileRetainedWorkspaces(db, f.environmentRuntime, async () => { + await heartbeatService(db).resumeTaskRetry(f.companyId, retry.requestId!); + }) ?? []); + expect(readSystemRecovery((await action(f.actionId)).evidence)?.state).toBe("complete"); + const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId)); + const saved = readRemoteWorkspaceAuthority(workspace.metadata)!; + const restored = await createSavedWorkspaceLoader(db)({ companyId: f.companyId, workspaceId, snapshotId: saved.snapshotId! }); + try { + expect(await readFile(path.join(restored.root, "saved.txt"), "utf8")).toBe("finished work\n"); + expect(await git(["rev-parse", "HEAD"], restored.root)).toBe(commit); + } + finally { await restored.cleanup(); } + const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(coordinator).toMatchObject({ phase: "workspace_finalizing", resultId }); + expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, retry.requestId!)))[0]) + .toMatchObject({ status: "coalesced", runId: f.runId, payload: { taskRetry: { strategy: "finalize_result" } } }); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(2); + await finalizeNativeRun({ db, runId: f.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0]) + .toMatchObject({ status: reviewPending ? "in_review" : "done", executionRunId: null, checkoutRunId: null }); + expect((await run(f.runId)).status).toBe("succeeded"); + if (reviewPending) expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, interactionId)))[0]) + .toMatchObject({ status: "pending", result: null, resolvedAt: null }); + }, 30_000); + + it("preserves an explicitly requested superseded remote export without applying its result over newer work", async () => { + const f = await acceptedRemoteExport(); + const [later] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.source.agentId, + issueId: f.issueId, status: "cancelled", runtimeMode: "legacy", finishedAt: new Date(), + contextSnapshot: { issueId: f.issueId }, createdAt: new Date(f.source.createdAt.getTime() + 1_000) }).returning(); + const [currentWorkspace] = await db.insert(executionWorkspaces).values({ companyId: f.companyId, + sourceIssueId: f.issueId, name: "Newer workspace", cwd: path.join(root, later.id), providerType: "cloud_sandbox", + mode: "isolated_workspace", strategyType: "project_primary", status: "active" }).returning(); + await db.update(issues).set({ executionWorkspaceId: currentWorkspace.id }).where(eq(issues.id, f.issueId)); + const laterAction = await issueRecoveryActionService(db).upsertSourceScoped({ companyId: f.companyId, + sourceIssueId: f.issueId, kind: "active_run_watchdog", cause: "legacy_execution_requires_reconciliation", + fingerprint: later.id, evidence: { runId: later.id }, nextAction: "Inspect the stopped work", + ownerType: "board", supersedeOnIdentityChange: true }); + expect((await action(f.actionId)).status).toBe("cancelled"); + await Promise.all(await reconcileRetainedWorkspaces(db, f.environmentRuntime, async () => {}) ?? []); + expect(f.runtime.resumeRunLease).not.toHaveBeenCalled(); + const app = boardRecoveryApp(); + const recoveryRequest = { actionId: f.actionId, runId: f.runId, idempotencyKey: randomUUID() }; + await request(app).post(`/api/issues/${f.issueId}/recovery-actions/recover-workspace`).send(recoveryRequest).expect(202); + const admitted = await action(f.actionId); + await request(app).post(`/api/issues/${f.issueId}/recovery-actions/recover-workspace`).send(recoveryRequest).expect(202); + expect(await action(f.actionId)).toEqual(admitted); + await db.update(heartbeatRuns).set({ status: "running", finishedAt: null }).where(eq(heartbeatRuns.id, later.id)); + await Promise.all(await reconcileRetainedWorkspaces(db, f.environmentRuntime, async () => {}) ?? []); + expect(readSystemRecovery((await action(f.actionId)).evidence)?.failure?.code).toBe("workspace_writer_active"); + expect(f.runtime.resumeRunLease).not.toHaveBeenCalled(); + await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(eq(heartbeatRuns.id, later.id)); + await makeRetryDue(f.actionId); + await Promise.all(await reconcileRetainedWorkspaces(db, f.environmentRuntime, async () => {}) ?? []); + expect(readSystemRecovery((await action(f.actionId)).evidence)?.state).toBe("complete"); + const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, f.workspaceId)); + const saved = readRemoteWorkspaceAuthority(workspace.metadata)!; + const restored = await createSavedWorkspaceLoader(db)({ companyId: f.companyId, workspaceId: f.workspaceId, snapshotId: saved.snapshotId! }); + try { + expect(await f.git(["rev-parse", "HEAD"], restored.root)).toBe(f.commit); + expect(await readFile(path.join(restored.root, "saved.txt"), "utf8")).toBe("finished work\n"); + } finally { await restored.cleanup(); } + expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0]) + .toMatchObject({ status: "blocked", executionWorkspaceId: currentWorkspace.id }); + expect(await action(laterAction.id)).toEqual(expect.objectContaining({ status: "active", evidence: { runId: later.id } })); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]) + .toMatchObject({ phase: "terminal_failure", resultId: f.resultId }); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toEqual([]); + }, 30_000); + it.each([ { status: "failed", runtimeMode: "native", uploadFailure: "none" }, { status: "cancelled", runtimeMode: "native", uploadFailure: "none" },