mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 09:50:48 +01:00
fix(recovery): continue a recovered preparation that failed with a transient code (#22096)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before a remote run starts its provider, Paperclip prepares the sandbox: it restores the saved workspace and stages the agent files > - When a legacy Runner preparation fails after the sandbox exists, Paperclip keeps the sandbox, saves its files, and holds the task with "preserve without replay" > - A preparation that a restart or a lost process stopped continues by itself after the files are saved. A preparation that failed with a transient error does not: it waits for a person to click Retry, even though its own bounded retry would continue it > - The agent-file Git exclusion timeout also did not count as transient. It failed with the generic `adapter_failed` code, so nothing retried it, and the wake that started the run was lost > - This pull request continues a recovered preparation that failed with a transient preparation code, in the same bounded lane as a lost preparation, and names the exclusion timeout as a transient preparation timeout > - The benefit is that a slow sandbox no longer stops a task until a person notices it ## Linked Issues or Issue Description Refs #22039, which retries the timed-out Git exclusion in place. This change covers the case where all three attempts time out. **What happened?** A run failed in the preparing stage with "Could not exclude agent files from task Git staging". No provider work started. The sandbox was kept, and workspace recovery later saved its files with no changes. The recovery hold recorded `replay: blocked`, so no new run started and the wake was lost. **Steps to reproduce** 1. Start a legacy Runner run on a sandbox that keeps its lease on failure. 2. Make the agent-file Git exclusion time out on every attempt. 3. Let workspace recovery save the retained sandbox. **Expected behavior** After the files are saved, the run continues once in its bounded retry lane, as a preparation stopped by a restart does. **Actual behavior** The run fails as `adapter_failed`. After recovery, the task waits for a manual Retry. ## What Changed - `failed-preparation-retry.ts` (new): `isTransientWorkspacePreparationCode` and `failedPreparationOwnsBoundedRetry` move here unchanged from `heartbeat/retries.ts`, which re-exports them. The code list is now an array, so SQL can use it. The move lets `cancelled-native-startup.ts` use the predicate without an import cycle, because `retries.ts` already imports that file. - `cancelled-native-startup.ts`: `isInterruptedNativePreparationCandidate` also accepts a failed preparation that has a required workspace recovery and whose own bounded retry owns it. All other conditions stay the same: legacy Runner, still preparing, no provider evidence, released leases with stop receipts, and a verified workspace repair before replay. - `workspace-recovery.ts`: the recovery sweep lists the same runs, so their continuation is delivered after the files are saved. - `run-completion.ts`: a `RemoteWorkspacePreparationTimeoutError` that a legacy run raises before it enters the adapter now records `remote_workspace_prepare_timeout`, the same code the setup path already records. - `agent-instruction-files.ts`: after the last timed-out attempt, the exclusion throws `RemoteWorkspacePreparationTimeoutError`. The message stays "... (reason=timed_out)". `RemoteWorkspacePreparationTimeoutError` takes an optional message for this. - Tests: a recovered preparation that failed with `remote_workspace_prepare_timeout` is listed by the sweep and continues once; one that failed with `adapter_failed` still waits for Retry. `failRun` names the timeout only when the adapter did not start. The agent-file test checks the error type. ## Verification - `npx vitest run` on the new and changed cases: the retained-preparation continuation, the `failRun` classification, and the agent-file suites. All pass. - Each new case fails without its source change. With only the sweep query reverted, the listing assertion fails. - `npx vitest run` on 62 related server test files (workspace recovery and its preparation suites, legacy execution recovery, run completion, retry scheduling, recovery service, remote workspaces, task retry, fresh continuation): 2361 pass, 16 skipped. - `npx tsc --noEmit` passes for the server. ## Risks - Low to medium. A failed preparation that kept its sandbox now continues once its files are saved, if its failure code is transient. The continuation uses the interrupted-preparation lane, which is bounded to two attempts, and the existing failure budget still applies. - A failure that is not transient (`adapter_failed`, `setup_failed`) is unchanged: it still waits for Retry. - A run whose failure is not a pre-provider remote preparation timeout keeps `adapter_failed`. - A task that is `done` or `cancelled` is unchanged. A scheduled retry never runs on a closed task, so a wake on a closed task is still not replayed after a failed preparation. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
323504e41e
commit
a3f86e5cc8
10 files changed
+108
-45
No files matched your search
@@ -4,6 +4,7 @@ import path from "node:path";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { unprocessable } from "../errors.js";
|
||||
import { runAdapterExecutionTargetShellCommand, type AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
|
||||
import { RemoteWorkspaceOperationTimeoutError, RemoteWorkspacePreparationTimeoutError } from "./remote-workspace-transport.js";
|
||||
|
||||
export const MAX_INSTRUCTION_BYTES = 1024 * 1024;
|
||||
export function instructionPath(value: string): string {
|
||||
@@ -193,15 +194,18 @@ export function instructionGitExcludeFailureReason(result: {
|
||||
}
|
||||
|
||||
const REMOTE_GIT_EXCLUDE_ATTEMPTS = 3;
|
||||
const REMOTE_GIT_EXCLUDE_TIMEOUT_SEC = 30;
|
||||
|
||||
export async function excludeRuntimeFromRemoteGit(runId: string, target: Exclude<AdapterExecutionTarget, { kind: "local" }>, failure: string) {
|
||||
const quote = (value: string) => `'${value.replaceAll("'", `'"'"'`)}'`;
|
||||
for (let attempt = 1; ; attempt++) {
|
||||
const result = await runAdapterExecutionTargetShellCommand(runId, target,
|
||||
`node -e ${quote(instructionGitExcludeProgram)} ${quote(target.remoteCwd)}`, { cwd: target.remoteCwd, env: {}, timeoutSec: 30 });
|
||||
`node -e ${quote(instructionGitExcludeProgram)} ${quote(target.remoteCwd)}`,
|
||||
{ cwd: target.remoteCwd, env: {}, timeoutSec: REMOTE_GIT_EXCLUDE_TIMEOUT_SEC });
|
||||
if (result.exitCode === 0 && !result.timedOut) return;
|
||||
if (!result.timedOut || attempt === REMOTE_GIT_EXCLUDE_ATTEMPTS) {
|
||||
throw new Error(`${failure} (reason=${instructionGitExcludeFailureReason(result)})`);
|
||||
}
|
||||
const message = `${failure} (reason=${instructionGitExcludeFailureReason(result)})`;
|
||||
if (!result.timedOut) throw new Error(message);
|
||||
if (attempt === REMOTE_GIT_EXCLUDE_ATTEMPTS) throw new RemoteWorkspacePreparationTimeoutError(
|
||||
new RemoteWorkspaceOperationTimeoutError("prepare", REMOTE_GIT_EXCLUDE_TIMEOUT_SEC * 1000), message);
|
||||
}
|
||||
}
|
||||
@@ -11,6 +11,7 @@ import { CONTROLLER_LEASE_LOST_FAILURE_CODE, legacyControllerBootId, legacyContr
|
||||
import { hasRequiredWorkspaceRecovery } from "./workspace-restore-recovery-state.js";
|
||||
import { hasVerifiedWorkspaceRepair } from "./workspace-recovery-receipt.js";
|
||||
import { readFailedPreparationSettlement } from "./failed-preparation-settlement.js";
|
||||
import { failedPreparationOwnsBoundedRetry } from "./failed-preparation-retry.js";
|
||||
import { hasVerifiedPreparationLog } from "./verified-preparation-log.js";
|
||||
import { executionStartedEventPredicate, isPreDispatchResourceWait, noRetainedExecutionPredicate } from "./pre-dispatch-execution-evidence.js";
|
||||
|
||||
@@ -19,7 +20,7 @@ type Run = Pick<typeof heartbeatRuns.$inferSelect,
|
||||
"runtimeModeResolvedAt" | "errorCode" | "error" | "resultJson" | "runnerProfileJson" |
|
||||
"contextSnapshot" | "processPid" | "processGroupId" | "processStartedAt" |
|
||||
"sessionIdAfter" | "issueId" | "nativeIssueId" | "nativeSessionId" | "executionStage" |
|
||||
"controllerBootId" | "controllerLeaseExpiresAt">;
|
||||
"controllerBootId" | "controllerLeaseExpiresAt"> & { runnerInstanceId?: string | null };
|
||||
type Coordinator = typeof nativeRunFinalizations.$inferSelect;
|
||||
const EMPTY_LOG_SHA256 = createHash("sha256").digest("hex");
|
||||
|
||||
@@ -135,7 +136,9 @@ export function isInterruptedNativePreparationCandidate(run: Run): boolean {
|
||||
// Restart recovery records this certificate only after revoking a preparing
|
||||
// controller. A generic process-lost error cannot establish safe replay.
|
||||
const lostPreparation = run.status === "failed"
|
||||
&& (run.errorCode === "process_lost" || run.errorCode === CONTROLLER_LEASE_LOST_FAILURE_CODE)
|
||||
&& (run.errorCode === "process_lost" || run.errorCode === CONTROLLER_LEASE_LOST_FAILURE_CODE
|
||||
|| hasRequiredWorkspaceRecovery(run.resultJson)
|
||||
&& failedPreparationOwnsBoundedRetry({ ...run, runnerInstanceId: run.runnerInstanceId ?? null }))
|
||||
&& recovery?.kind === "bootstrap" && recovery.providerWorkStarted === false;
|
||||
// Older shutdown records predate structured cancellation. Current records
|
||||
// identify unexpected system interruption without relying on an error label.
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
import type { heartbeatRuns } from "@paperclipai/db";
|
||||
import { parseObject } from "../adapters/utils.js";
|
||||
import { isResolvedInteractionContinuationWakeContext } from "../modules/run-dispatch/index.js";
|
||||
import { STORAGE_UNAVAILABLE_CODE } from "../storage/storage-unavailable.js";
|
||||
import { ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE } from "./environment-provider-unavailable-error.js";
|
||||
import { PLUGIN_WORKER_UNAVAILABLE_CODE } from "./plugin-worker-unavailable-error.js";
|
||||
import { REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE, REMOTE_WORKSPACE_RESTORE_FAILED_CODE } from "./remote-workspace-transport.js";
|
||||
import { REPOSITORY_CATALOG_UNAVAILABLE_CODE } from "./repository-catalog-unavailable-error.js";
|
||||
import { REUSABLE_SANDBOX_RESUME_FAILED_CODE } from "./reusable-sandbox-resume-error.js";
|
||||
import { REVIEWED_CHAT_BINDING_NOT_READY } from "./reviewed-chat-binding-diagnostics.js";
|
||||
import { RUNNER_REVISION_MISMATCH_CODE } from "./runner-revision-mismatch.js";
|
||||
import { WORKSPACE_GIT_SCAN_ERROR_CODES } from "./workspace-git-operation-scheduler.js";
|
||||
|
||||
type HeartbeatRun = typeof heartbeatRuns.$inferSelect;
|
||||
|
||||
export const TRANSIENT_WORKSPACE_PREPARATION_CODES: readonly string[] = [
|
||||
WORKSPACE_GIT_SCAN_ERROR_CODES.timeout, WORKSPACE_GIT_SCAN_ERROR_CODES.saturated,
|
||||
REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE, REUSABLE_SANDBOX_RESUME_FAILED_CODE,
|
||||
REMOTE_WORKSPACE_RESTORE_FAILED_CODE, PLUGIN_WORKER_UNAVAILABLE_CODE,
|
||||
RUNNER_REVISION_MISMATCH_CODE, ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE,
|
||||
REPOSITORY_CATALOG_UNAVAILABLE_CODE, REVIEWED_CHAT_BINDING_NOT_READY,
|
||||
STORAGE_UNAVAILABLE_CODE,
|
||||
];
|
||||
|
||||
export function isTransientWorkspacePreparationCode(code: string | null | undefined): boolean {
|
||||
return typeof code === "string" && TRANSIENT_WORKSPACE_PREPARATION_CODES.includes(code);
|
||||
}
|
||||
|
||||
/**
|
||||
* The persisted form of the setup-failure decision in run completion: this
|
||||
* failed legacy run stopped before provider work with a transient preparation
|
||||
* code, so its own bounded retry (scheduleBoundedRetryForRun) continues it.
|
||||
* The stranded-task sweep uses it to offer a run whose executor has not
|
||||
* scheduled that retry (still settling, or its server stopped) to the same
|
||||
* idempotent scheduler instead of escalating it.
|
||||
*/
|
||||
export function failedPreparationOwnsBoundedRetry(run: Pick<HeartbeatRun,
|
||||
"status" | "runtimeMode" | "nativeSessionId" | "runnerInstanceId" | "errorCode" | "resultJson" | "contextSnapshot">): boolean {
|
||||
if (run.status !== "failed" || run.runtimeMode !== "legacy" || run.nativeSessionId !== null || run.runnerInstanceId !== null
|
||||
|| !isTransientWorkspacePreparationCode(run.errorCode)) return false;
|
||||
const recovery = parseObject(run.resultJson?.executionRecovery);
|
||||
if (recovery.kind !== "bootstrap" || recovery.providerWorkStarted !== false) return false;
|
||||
// A restore failure is retried only when its owner proved it retryable.
|
||||
if (run.errorCode === REMOTE_WORKSPACE_RESTORE_FAILED_CODE)
|
||||
return parseObject(run.resultJson?.remoteWorkspaceRestore).retryable === true;
|
||||
// An accepted interaction keeps its own continuation retry.
|
||||
return !((run.errorCode === PLUGIN_WORKER_UNAVAILABLE_CODE || run.errorCode === ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE
|
||||
|| run.errorCode === REPOSITORY_CATALOG_UNAVAILABLE_CODE || run.errorCode === REVIEWED_CHAT_BINDING_NOT_READY)
|
||||
&& isResolvedInteractionContinuationWakeContext(run.contextSnapshot));
|
||||
}
|
||||
@@ -32,9 +32,7 @@ import {
|
||||
serverRestartBudgetSpent,
|
||||
} from "../execution-recovery-attempt.js";
|
||||
import { isReplacedPreparation, isServerReplacementInterruption } from "../server-replacement-interruption.js";
|
||||
import { WORKSPACE_GIT_SCAN_ERROR_CODES } from "../workspace-git-operation-scheduler.js";
|
||||
import { REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE, REMOTE_WORKSPACE_RESTORE_FAILED_CODE } from "../remote-workspace-transport.js";
|
||||
import { REUSABLE_SANDBOX_RESUME_FAILED_CODE } from "../reusable-sandbox-resume-error.js";
|
||||
import { failedPreparationOwnsBoundedRetry, isTransientWorkspacePreparationCode } from "../failed-preparation-retry.js";
|
||||
import { PLUGIN_WORKER_UNAVAILABLE_CODE } from "../plugin-worker-unavailable-error.js";
|
||||
import { ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE } from "../environment-provider-unavailable-error.js";
|
||||
import { REPOSITORY_CATALOG_UNAVAILABLE_CODE } from "../repository-catalog-unavailable-error.js";
|
||||
@@ -153,37 +151,7 @@ const BOUNDED_TRANSIENT_HEARTBEAT_RETRY_REASON = "transient_failure";
|
||||
|
||||
const BOUNDED_TRANSIENT_HEARTBEAT_RETRY_WAKE_REASON = "transient_failure_retry";
|
||||
|
||||
export function isTransientWorkspacePreparationCode(code: string | null | undefined): boolean {
|
||||
return code === WORKSPACE_GIT_SCAN_ERROR_CODES.timeout || code === WORKSPACE_GIT_SCAN_ERROR_CODES.saturated
|
||||
|| code === REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE || code === REUSABLE_SANDBOX_RESUME_FAILED_CODE
|
||||
|| code === REMOTE_WORKSPACE_RESTORE_FAILED_CODE || code === PLUGIN_WORKER_UNAVAILABLE_CODE
|
||||
|| code === RUNNER_REVISION_MISMATCH_CODE || code === ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE
|
||||
|| code === REPOSITORY_CATALOG_UNAVAILABLE_CODE || code === REVIEWED_CHAT_BINDING_NOT_READY
|
||||
|| code === STORAGE_UNAVAILABLE_CODE;
|
||||
}
|
||||
|
||||
/**
|
||||
* The persisted form of the setup-failure decision in run completion: this
|
||||
* failed legacy run stopped before provider work with a transient preparation
|
||||
* code, so its own bounded retry (scheduleBoundedRetryForRun) continues it.
|
||||
* The stranded-task sweep uses it to offer a run whose executor has not
|
||||
* scheduled that retry (still settling, or its server stopped) to the same
|
||||
* idempotent scheduler instead of escalating it.
|
||||
*/
|
||||
export function failedPreparationOwnsBoundedRetry(run: Pick<HeartbeatRun,
|
||||
"status" | "runtimeMode" | "nativeSessionId" | "runnerInstanceId" | "errorCode" | "resultJson" | "contextSnapshot">): boolean {
|
||||
if (run.status !== "failed" || run.runtimeMode !== "legacy" || run.nativeSessionId !== null || run.runnerInstanceId !== null
|
||||
|| !isTransientWorkspacePreparationCode(run.errorCode)) return false;
|
||||
const recovery = parseObject(run.resultJson?.executionRecovery);
|
||||
if (recovery.kind !== "bootstrap" || recovery.providerWorkStarted !== false) return false;
|
||||
// A restore failure is retried only when its owner proved it retryable.
|
||||
if (run.errorCode === REMOTE_WORKSPACE_RESTORE_FAILED_CODE)
|
||||
return parseObject(run.resultJson?.remoteWorkspaceRestore).retryable === true;
|
||||
// An accepted interaction keeps its own continuation retry.
|
||||
return !((run.errorCode === PLUGIN_WORKER_UNAVAILABLE_CODE || run.errorCode === ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE
|
||||
|| run.errorCode === REPOSITORY_CATALOG_UNAVAILABLE_CODE || run.errorCode === REVIEWED_CHAT_BINDING_NOT_READY)
|
||||
&& isResolvedInteractionContinuationWakeContext(run.contextSnapshot));
|
||||
}
|
||||
export { failedPreparationOwnsBoundedRetry, isTransientWorkspacePreparationCode };
|
||||
|
||||
export const BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS =
|
||||
BOUNDED_TRANSIENT_HEARTBEAT_RETRY_DELAYS_MS.length;
|
||||
|
||||
@@ -1370,6 +1370,18 @@ describe.skipIf(!support.supported && !process.env.PAPERCLIP_TEST_DATABASE_URL)(
|
||||
} else expect(f.deps.scheduleBoundedRetryForRun).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each(["before_adapter", "adapter_entered", "untrusted", "native"] as const)("names a remote preparation timeout raised %s during execution", async scenario => {
|
||||
const f = await fixture({ runtimeMode: scenario === "native" ? "native" : "legacy" });
|
||||
const cause = new RemoteWorkspaceOperationTimeoutError("prepare", 30_000);
|
||||
const message = "Could not exclude agent files from task Git staging (reason=timed_out)";
|
||||
const error = scenario === "untrusted" ? new Error(message) : new RemoteWorkspacePreparationTimeoutError(cause, message);
|
||||
await f.completion.failRun({ ...f.failure(error), requiredWorkspaceRestoreEvidence: {}, legacyAdapterEntered: scenario === "adapter_entered" });
|
||||
const failed = await read(f.run.id);
|
||||
expect(failed).toMatchObject({ status: "failed", error: message,
|
||||
errorCode: scenario === "before_adapter" ? "remote_workspace_prepare_timeout" : expect.not.stringMatching(/^remote_workspace_prepare_timeout$/) });
|
||||
if (scenario === "before_adapter") expect(failed.resultJson).toMatchObject({ executionRecovery: { kind: "bootstrap", providerWorkStarted: false } });
|
||||
});
|
||||
|
||||
it.each(["fresh", "started", "untrusted", "retained"] as const)("routes a %s preserved sandbox resume failure through bounded bootstrap recovery", async scenario => {
|
||||
const f = await fixture({ runtimeMode: scenario === "retained" ? "native" : "legacy" });
|
||||
const providerError = Object.assign(new Error("Sandbox abc is not in a valid state: stopping /home/daytona/secret"), { statusCode: 400 });
|
||||
|
||||
@@ -1323,6 +1323,9 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
|
||||
? { ...exactResumeCause, originalMessage: redactFailureText(exactResumeCause.originalMessage) }
|
||||
: null;
|
||||
const storageUnavailable = isStorageUnavailableError(err);
|
||||
const remoteWorkspacePreparationTimeout = !legacyAdapterEntered && run.runtimeMode === "legacy"
|
||||
&& run.nativeSessionId === null && run.runnerInstanceId === null && err instanceof RemoteWorkspacePreparationTimeoutError
|
||||
? err : null;
|
||||
const failureErrorCode =
|
||||
workspaceValidationFailure?.code ??
|
||||
configurationIncompleteFailure?.code ??
|
||||
@@ -1330,6 +1333,7 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
|
||||
recordedResponsibleUserDenialCode ??
|
||||
nativeTerminalFailureCode ??
|
||||
(storageUnavailable ? STORAGE_UNAVAILABLE_CODE : null) ??
|
||||
remoteWorkspacePreparationTimeout?.code ??
|
||||
"adapter_failed";
|
||||
// Only a `storage_unavailable` run gets a storage wait. Under any other code
|
||||
// (for example a native recovery hold that keeps the files without
|
||||
|
||||
@@ -133,10 +133,11 @@ export const REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE = "remote_workspace_prepa
|
||||
export class RemoteWorkspacePreparationTimeoutError extends RemoteWorkspaceOperationTimeoutError {
|
||||
readonly code = REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE;
|
||||
|
||||
constructor(error: RemoteWorkspaceOperationTimeoutError) {
|
||||
constructor(error: RemoteWorkspaceOperationTimeoutError, message?: string) {
|
||||
super("prepare", error.timeoutMs);
|
||||
this.name = "RemoteWorkspacePreparationTimeoutError";
|
||||
this.cause = error;
|
||||
if (message) this.message = message;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -419,6 +419,23 @@ describe("automatic retained workspace recovery", () => {
|
||||
} finally { await restarted.$client.end(); }
|
||||
}, 30_000);
|
||||
|
||||
it.each([["remote_workspace_prepare_timeout", 1], ["adapter_failed", 0]] as const)(
|
||||
"continues a recovered preparation that failed with %s only when its bounded retry owns it", async (errorCode, continuations) => {
|
||||
const f = await interruptedRemotePreparation("process_lost");
|
||||
await db.update(heartbeatRuns).set({ errorCode, error: "Could not exclude agent files from task Git staging (reason=timed_out)" })
|
||||
.where(eq(heartbeatRuns.id, f.runId));
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
expect(await hasVerifiedWorkspaceRepair(db, await run(f.runId))).toBe(true);
|
||||
const listed = await retainedWorkspaceCandidates(db, undefined, false, true);
|
||||
expect(listed.some(row => row.run.id === f.runId)).toBe(continuations === 1);
|
||||
const heartbeat = heartbeatService(db);
|
||||
await Promise.all([heartbeat.resumeRemoteStopComments(await run(f.runId)),
|
||||
heartbeat.resumeRemoteStopComments(await run(f.runId))]);
|
||||
const successors = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.runId));
|
||||
expect(successors).toHaveLength(continuations);
|
||||
if (continuations) expect(successors[0]).toMatchObject({ status: "scheduled_retry", scheduledRetryAttempt: 1 });
|
||||
}, 30_000);
|
||||
|
||||
it.each(["restart", "pause", "claimed", "duplicate"] as const)("retains preparation continuation after settlement before enqueue (%s)", async mode => {
|
||||
const f = await interruptedRemotePreparation("process_lost"), withheld = await withheldPreparationRetry(f);
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
|
||||
@@ -22,7 +22,8 @@ import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
|
||||
import { confirmSandboxRecoveryRestart, hasStopOnlyCleanup, prepareHistoricalSandboxStopVerification, prepareSandboxStopAndRetain, prepareStoppedSandboxForRecovery, restoreSandboxStopAfterFailedRestart, settleAbsentSandbox, settleStopOnlyCleanup } from "./sandbox-stop-and-retain.js";
|
||||
import { requestSandboxAbsenceProof } from "./sandbox-absence-proof.js";
|
||||
import { persistActivity, publishActivity } from "./activity-log.js";
|
||||
import { hasRequiredWorkspaceRecovery } from "./workspace-restore-recovery-state.js";
|
||||
import { hasRequiredWorkspaceRecovery, LEGACY_WORKSPACE_RECOVERY_SCHEMA } from "./workspace-restore-recovery-state.js";
|
||||
import { TRANSIENT_WORKSPACE_PREPARATION_CODES } from "./failed-preparation-retry.js";
|
||||
import { hasVerifiedWorkspaceRepair, readWorkspaceRecovery, type WorkspaceRecovery } from "./workspace-recovery-receipt.js";
|
||||
import { withNativeWorkspaceFinalizationOwnership } from "./native-runtime/native-workspace-finalization-ownership.js";
|
||||
import { readNativeWorkspaceSyncReference, readNativeWorkspaceSyncLocalCwd, resumeNativeWorkspaceSync } from "./native-runtime/native-workspace-sync.js";
|
||||
@@ -630,7 +631,8 @@ export function retainedWorkspaceCandidates(db: Db, cursor?: string, unfinishedO
|
||||
and ${agentWakeupRequests.payload}->>'issueId' = ${issueRecoveryActions.sourceIssueId}::text)`;
|
||||
const interruptedPreparation = sql`(${heartbeatRuns.runtimeMode} = 'legacy'
|
||||
and (${heartbeatRuns.status} = 'interrupted' or (${heartbeatRuns.status} = 'failed'
|
||||
and ${heartbeatRuns.errorCode} = 'process_lost'
|
||||
and (${heartbeatRuns.errorCode} = 'process_lost' or (${inArray(heartbeatRuns.errorCode, [...TRANSIENT_WORKSPACE_PREPARATION_CODES])}
|
||||
and ${heartbeatRuns.resultJson}#>>'{workspaceRestoreRecovery,schema}' = ${LEGACY_WORKSPACE_RECOVERY_SCHEMA}))
|
||||
and ${heartbeatRuns.resultJson}#>>'{executionRecovery,kind}' = 'bootstrap'
|
||||
and ${heartbeatRuns.resultJson}#>'{executionRecovery,providerWorkStarted}' = 'false'::jsonb))
|
||||
and ${heartbeatRuns.runtimeModeResolvedAt} is null and ${heartbeatRuns.executionStage} = 'preparing'
|
||||
|
||||
Reference in new issue
Block a user