fix(recovery): continue a recovered preparation that failed with a transient code (#22096)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a remote run starts its provider, Paperclip prepares the
sandbox: it restores the saved workspace and stages the agent files
> - When a legacy Runner preparation fails after the sandbox exists,
Paperclip keeps the sandbox, saves its files, and holds the task with
"preserve without replay"
> - A preparation that a restart or a lost process stopped continues by
itself after the files are saved. A preparation that failed with a
transient error does not: it waits for a person to click Retry, even
though its own bounded retry would continue it
> - The agent-file Git exclusion timeout also did not count as
transient. It failed with the generic `adapter_failed` code, so nothing
retried it, and the wake that started the run was lost
> - This pull request continues a recovered preparation that failed with
a transient preparation code, in the same bounded lane as a lost
preparation, and names the exclusion timeout as a transient preparation
timeout
> - The benefit is that a slow sandbox no longer stops a task until a
person notices it

## Linked Issues or Issue Description

Refs #22039, which retries the timed-out Git exclusion in place. This
change covers the case where all three attempts time out.

**What happened?**
A run failed in the preparing stage with "Could not exclude agent files
from task Git staging". No provider work started. The sandbox was kept,
and workspace recovery later saved its files with no changes. The
recovery hold recorded `replay: blocked`, so no new run started and the
wake was lost.

**Steps to reproduce**
1. Start a legacy Runner run on a sandbox that keeps its lease on
failure.
2. Make the agent-file Git exclusion time out on every attempt.
3. Let workspace recovery save the retained sandbox.

**Expected behavior**
After the files are saved, the run continues once in its bounded retry
lane, as a preparation stopped by a restart does.

**Actual behavior**
The run fails as `adapter_failed`. After recovery, the task waits for a
manual Retry.

## What Changed

- `failed-preparation-retry.ts` (new):
`isTransientWorkspacePreparationCode` and
`failedPreparationOwnsBoundedRetry` move here unchanged from
`heartbeat/retries.ts`, which re-exports them. The code list is now an
array, so SQL can use it. The move lets `cancelled-native-startup.ts`
use the predicate without an import cycle, because `retries.ts` already
imports that file.
- `cancelled-native-startup.ts`:
`isInterruptedNativePreparationCandidate` also accepts a failed
preparation that has a required workspace recovery and whose own bounded
retry owns it. All other conditions stay the same: legacy Runner, still
preparing, no provider evidence, released leases with stop receipts, and
a verified workspace repair before replay.
- `workspace-recovery.ts`: the recovery sweep lists the same runs, so
their continuation is delivered after the files are saved.
- `run-completion.ts`: a `RemoteWorkspacePreparationTimeoutError` that a
legacy run raises before it enters the adapter now records
`remote_workspace_prepare_timeout`, the same code the setup path already
records.
- `agent-instruction-files.ts`: after the last timed-out attempt, the
exclusion throws `RemoteWorkspacePreparationTimeoutError`. The message
stays "... (reason=timed_out)". `RemoteWorkspacePreparationTimeoutError`
takes an optional message for this.
- Tests: a recovered preparation that failed with
`remote_workspace_prepare_timeout` is listed by the sweep and continues
once; one that failed with `adapter_failed` still waits for Retry.
`failRun` names the timeout only when the adapter did not start. The
agent-file test checks the error type.

## Verification

- `npx vitest run` on the new and changed cases: the
retained-preparation continuation, the `failRun` classification, and the
agent-file suites. All pass.
- Each new case fails without its source change. With only the sweep
query reverted, the listing assertion fails.
- `npx vitest run` on 62 related server test files (workspace recovery
and its preparation suites, legacy execution recovery, run completion,
retry scheduling, recovery service, remote workspaces, task retry, fresh
continuation): 2361 pass, 16 skipped.
- `npx tsc --noEmit` passes for the server.

## Risks

- Low to medium. A failed preparation that kept its sandbox now
continues once its files are saved, if its failure code is transient.
The continuation uses the interrupted-preparation lane, which is bounded
to two attempts, and the existing failure budget still applies.
- A failure that is not transient (`adapter_failed`, `setup_failed`) is
unchanged: it still waits for Retry.
- A run whose failure is not a pre-provider remote preparation timeout
keeps `adapter_failed`.
- A task that is `done` or `cancelled` is unchanged. A scheduled retry
never runs on a closed task, so a wake on a closed task is still not
replayed after a failed preparation.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Devin FoleyandPaperclip authored and GitHub committed 2026-11-17 16:04:19 -08:00
1 parent 323504e41e
commit a3f86e5cc8
10 files changed
+108 -45

No files matched your search

@@ -29,6 +29,7 @@ import { AgentDirectoryReuseInvalidatedError, agentDirectoryWorkingCopyService }
import { withDirectoryMergeLock } from "@paperclipai/adapter-utils/workspace-restore-merge";
import { heartbeatRunEvents } from "@paperclipai/db";
import { logger } from "../middleware/logger.js";
import { RemoteWorkspacePreparationTimeoutError } from "../services/remote-workspace-transport.js";
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_AGENT_DIRECTORY_TEST_DATABASE_URL");
@@ -1505,8 +1506,9 @@ describe("persistent agent directories", () => {
try {
const runId = randomUUID();
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "on_demand", responsibleUserId: userId });
await expect(copies.prepare({ companyId, agentId, runId, cwd: home, target: sshTarget("/fixture/slow-task") }))
.rejects.toThrow("Could not exclude agent files from task Git staging (reason=timed_out)");
const failure = copies.prepare({ companyId, agentId, runId, cwd: home, target: sshTarget("/fixture/slow-task") });
await expect(failure).rejects.toThrow("Could not exclude agent files from task Git staging (reason=timed_out)");
await expect(failure).rejects.toBeInstanceOf(RemoteWorkspacePreparationTimeoutError);
expect(exclusions(shell)).toHaveLength(3);
expect((await copies.get(companyId, runId))!.state).toBe("unavailable");
} finally { shell.mockRestore(); }
@@ -4,6 +4,7 @@ import path from "node:path";
import { randomUUID } from "node:crypto";
import { unprocessable } from "../errors.js";
import { runAdapterExecutionTargetShellCommand, type AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
import { RemoteWorkspaceOperationTimeoutError, RemoteWorkspacePreparationTimeoutError } from "./remote-workspace-transport.js";
export const MAX_INSTRUCTION_BYTES = 1024 * 1024;
export function instructionPath(value: string): string {
@@ -193,15 +194,18 @@ export function instructionGitExcludeFailureReason(result: {
}
const REMOTE_GIT_EXCLUDE_ATTEMPTS = 3;
const REMOTE_GIT_EXCLUDE_TIMEOUT_SEC = 30;
export async function excludeRuntimeFromRemoteGit(runId: string, target: Exclude<AdapterExecutionTarget, { kind: "local" }>, failure: string) {
const quote = (value: string) => `'${value.replaceAll("'", `'"'"'`)}'`;
for (let attempt = 1; ; attempt++) {
const result = await runAdapterExecutionTargetShellCommand(runId, target,
`node -e ${quote(instructionGitExcludeProgram)} ${quote(target.remoteCwd)}`, { cwd: target.remoteCwd, env: {}, timeoutSec: 30 });
`node -e ${quote(instructionGitExcludeProgram)} ${quote(target.remoteCwd)}`,
{ cwd: target.remoteCwd, env: {}, timeoutSec: REMOTE_GIT_EXCLUDE_TIMEOUT_SEC });
if (result.exitCode === 0 && !result.timedOut) return;
if (!result.timedOut || attempt === REMOTE_GIT_EXCLUDE_ATTEMPTS) {
throw new Error(`${failure} (reason=${instructionGitExcludeFailureReason(result)})`);
}
const message = `${failure} (reason=${instructionGitExcludeFailureReason(result)})`;
if (!result.timedOut) throw new Error(message);
if (attempt === REMOTE_GIT_EXCLUDE_ATTEMPTS) throw new RemoteWorkspacePreparationTimeoutError(
new RemoteWorkspaceOperationTimeoutError("prepare", REMOTE_GIT_EXCLUDE_TIMEOUT_SEC * 1000), message);
}
}
@@ -11,6 +11,7 @@ import { CONTROLLER_LEASE_LOST_FAILURE_CODE, legacyControllerBootId, legacyContr
import { hasRequiredWorkspaceRecovery } from "./workspace-restore-recovery-state.js";
import { hasVerifiedWorkspaceRepair } from "./workspace-recovery-receipt.js";
import { readFailedPreparationSettlement } from "./failed-preparation-settlement.js";
import { failedPreparationOwnsBoundedRetry } from "./failed-preparation-retry.js";
import { hasVerifiedPreparationLog } from "./verified-preparation-log.js";
import { executionStartedEventPredicate, isPreDispatchResourceWait, noRetainedExecutionPredicate } from "./pre-dispatch-execution-evidence.js";
@@ -19,7 +20,7 @@ type Run = Pick<typeof heartbeatRuns.$inferSelect,
"runtimeModeResolvedAt" | "errorCode" | "error" | "resultJson" | "runnerProfileJson" |
"contextSnapshot" | "processPid" | "processGroupId" | "processStartedAt" |
"sessionIdAfter" | "issueId" | "nativeIssueId" | "nativeSessionId" | "executionStage" |
"controllerBootId" | "controllerLeaseExpiresAt">;
"controllerBootId" | "controllerLeaseExpiresAt"> & { runnerInstanceId?: string | null };
type Coordinator = typeof nativeRunFinalizations.$inferSelect;
const EMPTY_LOG_SHA256 = createHash("sha256").digest("hex");
@@ -135,7 +136,9 @@ export function isInterruptedNativePreparationCandidate(run: Run): boolean {
// Restart recovery records this certificate only after revoking a preparing
// controller. A generic process-lost error cannot establish safe replay.
const lostPreparation = run.status === "failed"
&& (run.errorCode === "process_lost" || run.errorCode === CONTROLLER_LEASE_LOST_FAILURE_CODE)
&& (run.errorCode === "process_lost" || run.errorCode === CONTROLLER_LEASE_LOST_FAILURE_CODE
|| hasRequiredWorkspaceRecovery(run.resultJson)
&& failedPreparationOwnsBoundedRetry({ ...run, runnerInstanceId: run.runnerInstanceId ?? null }))
&& recovery?.kind === "bootstrap" && recovery.providerWorkStarted === false;
// Older shutdown records predate structured cancellation. Current records
// identify unexpected system interruption without relying on an error label.
@@ -0,0 +1,50 @@
import type { heartbeatRuns } from "@paperclipai/db";
import { parseObject } from "../adapters/utils.js";
import { isResolvedInteractionContinuationWakeContext } from "../modules/run-dispatch/index.js";
import { STORAGE_UNAVAILABLE_CODE } from "../storage/storage-unavailable.js";
import { ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE } from "./environment-provider-unavailable-error.js";
import { PLUGIN_WORKER_UNAVAILABLE_CODE } from "./plugin-worker-unavailable-error.js";
import { REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE, REMOTE_WORKSPACE_RESTORE_FAILED_CODE } from "./remote-workspace-transport.js";
import { REPOSITORY_CATALOG_UNAVAILABLE_CODE } from "./repository-catalog-unavailable-error.js";
import { REUSABLE_SANDBOX_RESUME_FAILED_CODE } from "./reusable-sandbox-resume-error.js";
import { REVIEWED_CHAT_BINDING_NOT_READY } from "./reviewed-chat-binding-diagnostics.js";
import { RUNNER_REVISION_MISMATCH_CODE } from "./runner-revision-mismatch.js";
import { WORKSPACE_GIT_SCAN_ERROR_CODES } from "./workspace-git-operation-scheduler.js";
type HeartbeatRun = typeof heartbeatRuns.$inferSelect;
export const TRANSIENT_WORKSPACE_PREPARATION_CODES: readonly string[] = [
WORKSPACE_GIT_SCAN_ERROR_CODES.timeout, WORKSPACE_GIT_SCAN_ERROR_CODES.saturated,
REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE, REUSABLE_SANDBOX_RESUME_FAILED_CODE,
REMOTE_WORKSPACE_RESTORE_FAILED_CODE, PLUGIN_WORKER_UNAVAILABLE_CODE,
RUNNER_REVISION_MISMATCH_CODE, ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE,
REPOSITORY_CATALOG_UNAVAILABLE_CODE, REVIEWED_CHAT_BINDING_NOT_READY,
STORAGE_UNAVAILABLE_CODE,
];
export function isTransientWorkspacePreparationCode(code: string | null | undefined): boolean {
return typeof code === "string" && TRANSIENT_WORKSPACE_PREPARATION_CODES.includes(code);
}
/**
* The persisted form of the setup-failure decision in run completion: this
* failed legacy run stopped before provider work with a transient preparation
* code, so its own bounded retry (scheduleBoundedRetryForRun) continues it.
* The stranded-task sweep uses it to offer a run whose executor has not
* scheduled that retry (still settling, or its server stopped) to the same
* idempotent scheduler instead of escalating it.
*/
export function failedPreparationOwnsBoundedRetry(run: Pick<HeartbeatRun,
"status" | "runtimeMode" | "nativeSessionId" | "runnerInstanceId" | "errorCode" | "resultJson" | "contextSnapshot">): boolean {
if (run.status !== "failed" || run.runtimeMode !== "legacy" || run.nativeSessionId !== null || run.runnerInstanceId !== null
|| !isTransientWorkspacePreparationCode(run.errorCode)) return false;
const recovery = parseObject(run.resultJson?.executionRecovery);
if (recovery.kind !== "bootstrap" || recovery.providerWorkStarted !== false) return false;
// A restore failure is retried only when its owner proved it retryable.
if (run.errorCode === REMOTE_WORKSPACE_RESTORE_FAILED_CODE)
return parseObject(run.resultJson?.remoteWorkspaceRestore).retryable === true;
// An accepted interaction keeps its own continuation retry.
return !((run.errorCode === PLUGIN_WORKER_UNAVAILABLE_CODE || run.errorCode === ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE
|| run.errorCode === REPOSITORY_CATALOG_UNAVAILABLE_CODE || run.errorCode === REVIEWED_CHAT_BINDING_NOT_READY)
&& isResolvedInteractionContinuationWakeContext(run.contextSnapshot));
}
+2 -34
View File
@@ -32,9 +32,7 @@ import {
serverRestartBudgetSpent,
} from "../execution-recovery-attempt.js";
import { isReplacedPreparation, isServerReplacementInterruption } from "../server-replacement-interruption.js";
import { WORKSPACE_GIT_SCAN_ERROR_CODES } from "../workspace-git-operation-scheduler.js";
import { REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE, REMOTE_WORKSPACE_RESTORE_FAILED_CODE } from "../remote-workspace-transport.js";
import { REUSABLE_SANDBOX_RESUME_FAILED_CODE } from "../reusable-sandbox-resume-error.js";
import { failedPreparationOwnsBoundedRetry, isTransientWorkspacePreparationCode } from "../failed-preparation-retry.js";
import { PLUGIN_WORKER_UNAVAILABLE_CODE } from "../plugin-worker-unavailable-error.js";
import { ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE } from "../environment-provider-unavailable-error.js";
import { REPOSITORY_CATALOG_UNAVAILABLE_CODE } from "../repository-catalog-unavailable-error.js";
@@ -153,37 +151,7 @@ const BOUNDED_TRANSIENT_HEARTBEAT_RETRY_REASON = "transient_failure";
const BOUNDED_TRANSIENT_HEARTBEAT_RETRY_WAKE_REASON = "transient_failure_retry";
export function isTransientWorkspacePreparationCode(code: string | null | undefined): boolean {
return code === WORKSPACE_GIT_SCAN_ERROR_CODES.timeout || code === WORKSPACE_GIT_SCAN_ERROR_CODES.saturated
|| code === REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE || code === REUSABLE_SANDBOX_RESUME_FAILED_CODE
|| code === REMOTE_WORKSPACE_RESTORE_FAILED_CODE || code === PLUGIN_WORKER_UNAVAILABLE_CODE
|| code === RUNNER_REVISION_MISMATCH_CODE || code === ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE
|| code === REPOSITORY_CATALOG_UNAVAILABLE_CODE || code === REVIEWED_CHAT_BINDING_NOT_READY
|| code === STORAGE_UNAVAILABLE_CODE;
}
/**
* The persisted form of the setup-failure decision in run completion: this
* failed legacy run stopped before provider work with a transient preparation
* code, so its own bounded retry (scheduleBoundedRetryForRun) continues it.
* The stranded-task sweep uses it to offer a run whose executor has not
* scheduled that retry (still settling, or its server stopped) to the same
* idempotent scheduler instead of escalating it.
*/
export function failedPreparationOwnsBoundedRetry(run: Pick<HeartbeatRun,
"status" | "runtimeMode" | "nativeSessionId" | "runnerInstanceId" | "errorCode" | "resultJson" | "contextSnapshot">): boolean {
if (run.status !== "failed" || run.runtimeMode !== "legacy" || run.nativeSessionId !== null || run.runnerInstanceId !== null
|| !isTransientWorkspacePreparationCode(run.errorCode)) return false;
const recovery = parseObject(run.resultJson?.executionRecovery);
if (recovery.kind !== "bootstrap" || recovery.providerWorkStarted !== false) return false;
// A restore failure is retried only when its owner proved it retryable.
if (run.errorCode === REMOTE_WORKSPACE_RESTORE_FAILED_CODE)
return parseObject(run.resultJson?.remoteWorkspaceRestore).retryable === true;
// An accepted interaction keeps its own continuation retry.
return !((run.errorCode === PLUGIN_WORKER_UNAVAILABLE_CODE || run.errorCode === ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE
|| run.errorCode === REPOSITORY_CATALOG_UNAVAILABLE_CODE || run.errorCode === REVIEWED_CHAT_BINDING_NOT_READY)
&& isResolvedInteractionContinuationWakeContext(run.contextSnapshot));
}
export { failedPreparationOwnsBoundedRetry, isTransientWorkspacePreparationCode };
export const BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS =
BOUNDED_TRANSIENT_HEARTBEAT_RETRY_DELAYS_MS.length;
@@ -1370,6 +1370,18 @@ describe.skipIf(!support.supported && !process.env.PAPERCLIP_TEST_DATABASE_URL)(
} else expect(f.deps.scheduleBoundedRetryForRun).not.toHaveBeenCalled();
});
it.each(["before_adapter", "adapter_entered", "untrusted", "native"] as const)("names a remote preparation timeout raised %s during execution", async scenario => {
const f = await fixture({ runtimeMode: scenario === "native" ? "native" : "legacy" });
const cause = new RemoteWorkspaceOperationTimeoutError("prepare", 30_000);
const message = "Could not exclude agent files from task Git staging (reason=timed_out)";
const error = scenario === "untrusted" ? new Error(message) : new RemoteWorkspacePreparationTimeoutError(cause, message);
await f.completion.failRun({ ...f.failure(error), requiredWorkspaceRestoreEvidence: {}, legacyAdapterEntered: scenario === "adapter_entered" });
const failed = await read(f.run.id);
expect(failed).toMatchObject({ status: "failed", error: message,
errorCode: scenario === "before_adapter" ? "remote_workspace_prepare_timeout" : expect.not.stringMatching(/^remote_workspace_prepare_timeout$/) });
if (scenario === "before_adapter") expect(failed.resultJson).toMatchObject({ executionRecovery: { kind: "bootstrap", providerWorkStarted: false } });
});
it.each(["fresh", "started", "untrusted", "retained"] as const)("routes a %s preserved sandbox resume failure through bounded bootstrap recovery", async scenario => {
const f = await fixture({ runtimeMode: scenario === "retained" ? "native" : "legacy" });
const providerError = Object.assign(new Error("Sandbox abc is not in a valid state: stopping /home/daytona/secret"), { statusCode: 400 });
@@ -1323,6 +1323,9 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
? { ...exactResumeCause, originalMessage: redactFailureText(exactResumeCause.originalMessage) }
: null;
const storageUnavailable = isStorageUnavailableError(err);
const remoteWorkspacePreparationTimeout = !legacyAdapterEntered && run.runtimeMode === "legacy"
&& run.nativeSessionId === null && run.runnerInstanceId === null && err instanceof RemoteWorkspacePreparationTimeoutError
? err : null;
const failureErrorCode =
workspaceValidationFailure?.code ??
configurationIncompleteFailure?.code ??
@@ -1330,6 +1333,7 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
recordedResponsibleUserDenialCode ??
nativeTerminalFailureCode ??
(storageUnavailable ? STORAGE_UNAVAILABLE_CODE : null) ??
remoteWorkspacePreparationTimeout?.code ??
"adapter_failed";
// Only a `storage_unavailable` run gets a storage wait. Under any other code
// (for example a native recovery hold that keeps the files without
@@ -133,10 +133,11 @@ export const REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE = "remote_workspace_prepa
export class RemoteWorkspacePreparationTimeoutError extends RemoteWorkspaceOperationTimeoutError {
readonly code = REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE;
constructor(error: RemoteWorkspaceOperationTimeoutError) {
constructor(error: RemoteWorkspaceOperationTimeoutError, message?: string) {
super("prepare", error.timeoutMs);
this.name = "RemoteWorkspacePreparationTimeoutError";
this.cause = error;
if (message) this.message = message;
}
}
@@ -419,6 +419,23 @@ describe("automatic retained workspace recovery", () => {
} finally { await restarted.$client.end(); }
}, 30_000);
it.each([["remote_workspace_prepare_timeout", 1], ["adapter_failed", 0]] as const)(
"continues a recovered preparation that failed with %s only when its bounded retry owns it", async (errorCode, continuations) => {
const f = await interruptedRemotePreparation("process_lost");
await db.update(heartbeatRuns).set({ errorCode, error: "Could not exclude agent files from task Git staging (reason=timed_out)" })
.where(eq(heartbeatRuns.id, f.runId));
await recoverRetainedWorkspace(f.input);
expect(await hasVerifiedWorkspaceRepair(db, await run(f.runId))).toBe(true);
const listed = await retainedWorkspaceCandidates(db, undefined, false, true);
expect(listed.some(row => row.run.id === f.runId)).toBe(continuations === 1);
const heartbeat = heartbeatService(db);
await Promise.all([heartbeat.resumeRemoteStopComments(await run(f.runId)),
heartbeat.resumeRemoteStopComments(await run(f.runId))]);
const successors = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.runId));
expect(successors).toHaveLength(continuations);
if (continuations) expect(successors[0]).toMatchObject({ status: "scheduled_retry", scheduledRetryAttempt: 1 });
}, 30_000);
it.each(["restart", "pause", "claimed", "duplicate"] as const)("retains preparation continuation after settlement before enqueue (%s)", async mode => {
const f = await interruptedRemotePreparation("process_lost"), withheld = await withheldPreparationRetry(f);
await recoverRetainedWorkspace(f.input);
+4 -2
View File
@@ -22,7 +22,8 @@ import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
import { confirmSandboxRecoveryRestart, hasStopOnlyCleanup, prepareHistoricalSandboxStopVerification, prepareSandboxStopAndRetain, prepareStoppedSandboxForRecovery, restoreSandboxStopAfterFailedRestart, settleAbsentSandbox, settleStopOnlyCleanup } from "./sandbox-stop-and-retain.js";
import { requestSandboxAbsenceProof } from "./sandbox-absence-proof.js";
import { persistActivity, publishActivity } from "./activity-log.js";
import { hasRequiredWorkspaceRecovery } from "./workspace-restore-recovery-state.js";
import { hasRequiredWorkspaceRecovery, LEGACY_WORKSPACE_RECOVERY_SCHEMA } from "./workspace-restore-recovery-state.js";
import { TRANSIENT_WORKSPACE_PREPARATION_CODES } from "./failed-preparation-retry.js";
import { hasVerifiedWorkspaceRepair, readWorkspaceRecovery, type WorkspaceRecovery } from "./workspace-recovery-receipt.js";
import { withNativeWorkspaceFinalizationOwnership } from "./native-runtime/native-workspace-finalization-ownership.js";
import { readNativeWorkspaceSyncReference, readNativeWorkspaceSyncLocalCwd, resumeNativeWorkspaceSync } from "./native-runtime/native-workspace-sync.js";
@@ -630,7 +631,8 @@ export function retainedWorkspaceCandidates(db: Db, cursor?: string, unfinishedO
and ${agentWakeupRequests.payload}->>'issueId' = ${issueRecoveryActions.sourceIssueId}::text)`;
const interruptedPreparation = sql`(${heartbeatRuns.runtimeMode} = 'legacy'
and (${heartbeatRuns.status} = 'interrupted' or (${heartbeatRuns.status} = 'failed'
and ${heartbeatRuns.errorCode} = 'process_lost'
and (${heartbeatRuns.errorCode} = 'process_lost' or (${inArray(heartbeatRuns.errorCode, [...TRANSIENT_WORKSPACE_PREPARATION_CODES])}
and ${heartbeatRuns.resultJson}#>>'{workspaceRestoreRecovery,schema}' = ${LEGACY_WORKSPACE_RECOVERY_SCHEMA}))
and ${heartbeatRuns.resultJson}#>>'{executionRecovery,kind}' = 'bootstrap'
and ${heartbeatRuns.resultJson}#>'{executionRecovery,providerWorkStarted}' = 'false'::jsonb))
and ${heartbeatRuns.runtimeModeResolvedAt} is null and ${heartbeatRuns.executionStage} = 'preparing'