mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 01:57:31 +01:00
fix(workspaces): name remote restore failures and leave a board recovery action (#19207)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A remote task restores its saved workspace into a sandbox before the agent starts. > - When that restore fails, the run records only `remote_workspace_operation_failed: operation=restore exit=1`. > - The run then fails as `setup_failed`. No retry starts, and a `blocked` task gets no recovery action. > - An operator cannot tell which step failed, and nobody owns the next step. > - This pull request makes the sandbox scripts name their failed step in a closed grammar, and it gives a restore failure a bounded retry or a named board recovery action. > - The benefit is a visible cause and a clear owner. Saved work stays protected. ## Linked Issues or Issue Description Refs #19152 and #19187. Those changes name copy-back (save) failures. Refs #18852 and #19195. Those changes added the bounded bootstrap retry that this change reuses. **What happened?** A board reply woke an agent on a `blocked` parent task. The run failed during setup 13 seconds later. The only run event was `remote_workspace_operation_failed: operation=restore exit=1`. The run had no workspace operation row. The task stayed `blocked`, and its recovery actions showed `active: null`. The same opaque family also appeared for workspace preparation (`exit=1` only). **Expected behavior** The run names which restore step failed, without command output or paths. A failure before provider work uses a bounded retry when a retry can help. Otherwise, and when the retry budget is spent, the board gets a named recovery action. The saved version is never changed or overwritten. **Steps to reproduce** 1. Save a remote workspace, then start a new run on a different allocation. 2. Put a folder at the published workspace path in the sandbox before restore. 3. Observe that the base code records only `operation=restore exit=1`, then `setup_failed`, with no recovery action for a `blocked` task. **Paperclip version or commit** Base: `87dfba6818`. **Deployment mode** Legacy runtime with sandbox (Daytona) remote workspace storage. ## What Changed - The prepare, repository, and restore sandbox scripts track their current stage. On failure they print `remote_workspace_<op>_failed: stage=<stage>` plus closed fields: `reason=` (the script's own snake_case refusal), `archive=` (the archive helper's fixed reason), `code=` (errno), `disk=full`, `git=` (a fixed label chosen from Git output), and the existing `exit`/`signal`/`program` fields. - The server accepts that text only when it matches a strict grammar for the same operation. Other output is dropped, as before. - Provider refusals during the restore upload and restore command are recorded by closed labels, like copy-back. - A failed saved-workspace restore raises `RemoteWorkspaceRestoreError` with the server step (`snapshot_lookup`, `snapshot_download`, `sandbox_restore`). It is retryable only when the owner re-reads the same writer, epoch, and saved version with no unsaved work, and the sandbox did not name a refusal reason. - Setup failure handling classifies a pre-dispatch legacy restore failure as `remote_workspace_restore_failed`. A retryable failure uses the existing bounded bootstrap retry. A non-retryable or exhausted failure records a board-owned recovery action with the redacted cause, when stranded-work recovery does not scan the task status (for example `blocked`). - Stranded-work recovery gets a named notice for this code. Generic continuation recovery does not grant it a second budget, and legacy reconciliation does not invent provider actions for it. - `doc/execution-semantics.md` documents the contract. ## Verification - Red: the new transport tests failed on the base code with `remote_workspace_operation_failed: operation=restore exit=1` (and the same for prepare and repository). The new completion tests failed because the run became `setup_failed` with no retry and no recovery action. - Green: `npx vitest run src/services/remote-workspace-transport.test.ts src/services/heartbeat/run-completion.test.ts src/services/remote-workspaces.integration.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts src/services/legacy-execution-recovery.test.ts src/services/recovery src/services/remote-workspace-save-concurrency.test.ts src/services/native-runtime/native-workspace-finalizer-failure.test.ts src/__tests__/native-workspace-finalization-work.test.ts` passes (563 tests). - The transport tests use real sandbox commands. They prove an occupied folder reports `stage=inspect_root reason=occupied_root` and keeps its files. They prove a wrong manifest reports `stage=extract archive=manifest_mismatch`. They prove a missing branch reports `stage=checkout_branch`, and a changed origin reports `stage=verify_origin`. Forged output with a path is dropped. - The real-PostgreSQL cold-restore test proves an occupied root gives a non-retryable `sandbox_restore` failure and keeps the unknown files. - The completion tests cover retried, exhausted, refused, post-dispatch, and `in_progress` cases. - `tsc --noEmit` for the server reports no errors in the changed files. ## Risks - A retry keeps the same restore path. If the cause is deterministic but unnamed, the run uses its two retries before the board action. Named refusals skip the retries. - The board action is recorded only for statuses that stranded-work recovery does not scan, and only when no other action is active. This prevents a second recovery identity for one failure. - Script stderr grammar is new. Old sandboxes run the script that the server sends, so no runner update is necessary. - No migration, deployment, or credential change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use, code execution, and browser tools for read-only staging evidence. Context window and reasoning mode are host-managed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed
1 parent
9e9a628a57
commit
2ce5f03cd8
16 files changed
+404
-58
No files matched your search
@@ -1042,6 +1042,21 @@ replacement. The run records the refusal by closed labels only (cause label, HTT
|
||||
status, errno-style code), never by provider text. On exhaustion, the task gets a
|
||||
specific recovery notice, and generic recovery does not grant another budget.
|
||||
|
||||
A saved workspace that cannot be restored into a new allocation before provider
|
||||
dispatch fails with `remote_workspace_restore_failed`. The run error names the
|
||||
server step (`snapshot_lookup`, `snapshot_download`, or `sandbox_restore`) and,
|
||||
for sandbox commands, the script stage plus closed fields such as
|
||||
`reason=occupied_root`, `archive=manifest_mismatch`, `code=EIO`, `disk=full`, or
|
||||
`git=auth_failed`. Command output, paths, and provider text are never copied;
|
||||
prepare and repository commands use the same grammar. A failed restore never
|
||||
publishes over an existing folder and never changes the saved version. The run
|
||||
uses the bounded bootstrap budget only when the owner re-reads the same writer,
|
||||
epoch, and saved version with no unsaved work, and the sandbox did not name a
|
||||
refusal reason. A missing saved version, a named refusal (for example an occupied
|
||||
folder), or an exhausted budget leaves a board-owned recovery action. Stranded-work
|
||||
recovery names this failure for the statuses it scans; for other statuses, such as
|
||||
a `blocked` task woken by a reply, the failure path records the action directly.
|
||||
|
||||
The provider's own idle auto-stop must not coincide with the warm runner idle
|
||||
window. The host forwards the run's effective warm idle window (environment
|
||||
`runnerIdleTimeoutMs`, else the agent's `idleTimeoutMs`) to the sandbox provider
|
||||
|
||||
@@ -493,7 +493,7 @@ describeEmbeddedPostgres("heartbeat bounded retry scheduling", () => {
|
||||
return { companyId, agentId, issueId, runId, now };
|
||||
}
|
||||
|
||||
it.each(["remote_workspace_prepare_timeout", "reusable_sandbox_resume_failed"].flatMap(errorCode =>
|
||||
it.each(["remote_workspace_prepare_timeout", "reusable_sandbox_resume_failed", "remote_workspace_restore_failed"].flatMap(errorCode =>
|
||||
(["resume", "paused", "Stop", "dependency"] as const).map(action => [errorCode, action] as const)))("bounds %s bootstrap retries across restart and honors %s", async (errorCode, action) => {
|
||||
const { companyId, agentId, issueId, runId, now } = await seedMaxTurnFixture();
|
||||
await db.update(agents).set({ adapterType: "paperclip_runner", adapterConfig: { provider: "codex", model: "gpt-6.1-sol" } })
|
||||
|
||||
@@ -21,7 +21,7 @@ import {
|
||||
accountingForScheduledRetry,
|
||||
} from "../execution-recovery-attempt.js";
|
||||
import { WORKSPACE_GIT_SCAN_ERROR_CODES } from "../workspace-git-operation-scheduler.js";
|
||||
import { REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE } from "../remote-workspace-transport.js";
|
||||
import { REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE, REMOTE_WORKSPACE_RESTORE_FAILED_CODE } from "../remote-workspace-transport.js";
|
||||
import { REUSABLE_SANDBOX_RESUME_FAILED_CODE } from "../reusable-sandbox-resume-error.js";
|
||||
import { CONTROLLER_LEASE_LOST_FAILURE_CODE } from "../legacy-controller-lease.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
@@ -134,7 +134,8 @@ const BOUNDED_TRANSIENT_HEARTBEAT_RETRY_WAKE_REASON = "transient_failure_retry";
|
||||
|
||||
export function isTransientWorkspacePreparationCode(code: string | null | undefined): boolean {
|
||||
return code === WORKSPACE_GIT_SCAN_ERROR_CODES.timeout || code === WORKSPACE_GIT_SCAN_ERROR_CODES.saturated
|
||||
|| code === REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE || code === REUSABLE_SANDBOX_RESUME_FAILED_CODE;
|
||||
|| code === REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE || code === REUSABLE_SANDBOX_RESUME_FAILED_CODE
|
||||
|| code === REMOTE_WORKSPACE_RESTORE_FAILED_CODE;
|
||||
}
|
||||
|
||||
export const BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS =
|
||||
|
||||
@@ -16,7 +16,7 @@ import { createHash, randomUUID } from "node:crypto";
|
||||
import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js";
|
||||
import { createNativeRunLogCoverageVerifier, formatNativeRunLogEvent } from "../native-runtime/native-run-log-export.js";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { activityLog, agents, agentWakeupRequests, budgetReservations, caseHandoffs, cases, companies, completionContracts, costEvents, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRuns, heartbeatRunEvents, issues, nativeRunFinalizations } from "@paperclipai/db";
|
||||
import { activityLog, agents, agentWakeupRequests, budgetReservations, caseHandoffs, cases, companies, completionContracts, costEvents, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRuns, heartbeatRunEvents, issueRecoveryActions, issues, nativeRunFinalizations } from "@paperclipai/db";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, it, onTestFinished, vi } from "vitest";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
||||
@@ -26,7 +26,7 @@ import { UnresolvedWorkspaceBaseRefError } from "../workspace-runtime.js";
|
||||
import { LegacyControllerLeaseLostError } from "../legacy-controller-lease.js";
|
||||
import { requestedRunCancellation } from "../run-cancellation.js";
|
||||
import { ProjectAccessDeniedError } from "../project-access-denial.js";
|
||||
import { RemoteWorkspaceOperationTimeoutError, RemoteWorkspacePreparationTimeoutError } from "../remote-workspace-transport.js";
|
||||
import { RemoteWorkspaceOperationError, RemoteWorkspaceOperationTimeoutError, RemoteWorkspacePreparationTimeoutError, RemoteWorkspaceRestoreError } from "../remote-workspace-transport.js";
|
||||
import { EnvironmentRunError } from "../environment-run-orchestrator.js";
|
||||
import { ReusableSandboxResumeError } from "../reusable-sandbox-resume-error.js";
|
||||
import { createHeartbeatLifecycle, type HeartbeatLifecycleDependencies } from "./run-lifecycle.js";
|
||||
@@ -949,6 +949,61 @@ describe.skipIf(!support.supported && !process.env.PAPERCLIP_TEST_DATABASE_URL)(
|
||||
} else expect(f.deps.scheduleBoundedRetryForRun).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each(["retried", "exhausted", "refused", "started", "in progress"] as const)("names a %s saved workspace restore failure and leaves recovery", async scenario => {
|
||||
const f = await fixture();
|
||||
const [issue] = await db.insert(issues).values({ companyId: f.run.companyId, title: "Parent task", status: scenario === "in progress" ? "in_progress" : "blocked",
|
||||
assigneeAgentId: f.agent.id }).returning();
|
||||
await db.update(heartbeatRuns).set({ issueId: issue.id, contextSnapshot: { issueId: issue.id } }).where(eq(heartbeatRuns.id, f.run.id));
|
||||
const sandboxCause = scenario === "refused"
|
||||
? "remote_workspace_operation_failed: operation=restore exit=1; remote_workspace_restore_failed: stage=inspect_root reason=occupied_root"
|
||||
: "remote_workspace_operation_failed: operation=restore exit=1; remote_workspace_restore_failed: stage=extract code=EIO exit=1";
|
||||
const error = new RemoteWorkspaceRestoreError("sandbox_restore", scenario !== "refused", new RemoteWorkspaceOperationError(sandboxCause));
|
||||
if (scenario === "retried") f.deps.scheduleBoundedRetryForRun.mockResolvedValueOnce({ outcome: "scheduled" } as never);
|
||||
else f.deps.scheduleBoundedRetryForRun.mockResolvedValueOnce({ outcome: "retry_exhausted", attempt: 3, maxAttempts: 2 } as never);
|
||||
await f.completion.failRunSetup(f.setup(error, scenario === "started"));
|
||||
const failed = await read(f.run.id);
|
||||
// The run names the restore step and the sandbox step instead of a bare exit status.
|
||||
expect(failed.error).toBe(`remote_workspace_restore_failed: step=sandbox_restore; ${sandboxCause}`);
|
||||
const actions = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issue.id));
|
||||
if (scenario === "started") {
|
||||
expect(failed.errorCode).toBe("setup_failed");
|
||||
expect(f.deps.scheduleBoundedRetryForRun).not.toHaveBeenCalled();
|
||||
// Started work keeps its existing reconciliation path.
|
||||
expect(actions.filter(action => action.cause === "remote_workspace_restore_failed")).toHaveLength(0);
|
||||
return;
|
||||
}
|
||||
expect(failed).toMatchObject({ status: "failed", errorCode: "remote_workspace_restore_failed" });
|
||||
expect(failed.resultJson).toMatchObject({ executionRecovery: { kind: "bootstrap", providerWorkStarted: false },
|
||||
remoteWorkspaceRestore: { step: "sandbox_restore", retryable: scenario !== "refused" } });
|
||||
if (scenario === "refused") expect(f.deps.scheduleBoundedRetryForRun).not.toHaveBeenCalled();
|
||||
else expect(f.deps.scheduleBoundedRetryForRun).toHaveBeenCalledWith(expect.objectContaining({ id: f.run.id }), f.agent);
|
||||
// Stranded-work recovery owns scanned statuses; a retry owns the next step.
|
||||
if (scenario === "retried" || scenario === "in progress") { expect(actions).toHaveLength(0); return; }
|
||||
expect(actions).toHaveLength(1);
|
||||
expect(actions[0]).toMatchObject({ status: "active", ownerType: "board", cause: "remote_workspace_restore_failed",
|
||||
kind: "stranded_assigned_issue", returnOwnerAgentId: f.agent.id });
|
||||
expect(actions[0]!.evidence).toMatchObject({ runId: f.run.id, restoreFailure: failed.error });
|
||||
expect(actions[0]!.nextAction).toContain("saved workspace version was kept");
|
||||
});
|
||||
|
||||
it("keeps an active recovery action that another writer created for the task", async () => {
|
||||
const f = await fixture();
|
||||
const [issue] = await db.insert(issues).values({ companyId: f.run.companyId, title: "Parent task", status: "blocked",
|
||||
assigneeAgentId: f.agent.id }).returning();
|
||||
await db.update(heartbeatRuns).set({ issueId: issue.id, contextSnapshot: { issueId: issue.id } }).where(eq(heartbeatRuns.id, f.run.id));
|
||||
const [existing] = await db.insert(issueRecoveryActions).values({ companyId: f.run.companyId, sourceIssueId: issue.id,
|
||||
kind: "active_run_watchdog", status: "active", ownerType: "board", cause: "other_failure", fingerprint: "other",
|
||||
evidence: { runId: "other-run" }, nextAction: "Handle the other failure.", attemptCount: 1 }).returning();
|
||||
const error = new RemoteWorkspaceRestoreError("sandbox_restore", false, new RemoteWorkspaceOperationError(
|
||||
"remote_workspace_operation_failed: operation=restore exit=1; remote_workspace_restore_failed: stage=inspect_root reason=occupied_root"));
|
||||
await f.completion.failRunSetup(f.setup(error));
|
||||
expect((await read(f.run.id)).errorCode).toBe("remote_workspace_restore_failed");
|
||||
const actions = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issue.id));
|
||||
expect(actions).toHaveLength(1);
|
||||
expect(actions[0]).toMatchObject({ id: existing.id, cause: "other_failure", fingerprint: "other", nextAction: "Handle the other failure.",
|
||||
attemptCount: 1, evidence: { runId: "other-run" } });
|
||||
});
|
||||
|
||||
it("records a lease lost before dispatch as a retryable host stop, not an opaque setup failure", async () => {
|
||||
const f = await fixture();
|
||||
f.controller.abort(new LegacyControllerLeaseLostError("renewal_rejected"));
|
||||
|
||||
@@ -9,7 +9,8 @@ import { pendingCaseHandoffForRun } from "./case-handoff-lifecycle.js";
|
||||
import { NON_RETRYABLE_PREFLIGHT_FAILURE_CODES, isHeartbeatRunTerminalStatus } from "./run-lifecycle.js";
|
||||
import { ChatControlRecoveryUnresolvedError } from "./queue.js";
|
||||
import { readTransientRecoveryContractFromRun, readHeartbeatRunErrorFamily, isTransientWorkspacePreparationCode } from "./retries.js";
|
||||
import { RemoteWorkspacePreparationTimeoutError } from "../remote-workspace-transport.js";
|
||||
import { RemoteWorkspacePreparationTimeoutError, REMOTE_WORKSPACE_RESTORE_FAILED_CODE, findRemoteWorkspaceRestoreError } from "../remote-workspace-transport.js";
|
||||
import { escalateRemoteWorkspaceRestoreFailure } from "../remote-workspace-restore-recovery.js";
|
||||
import { findReusableSandboxResumeError } from "../reusable-sandbox-resume-error.js";
|
||||
import { type RunSessionOutcome, resolveNextSessionState, normalizeUsageTotals, resolveCacheAdjustedCostUsd, resolveLedgerBiller, resolveLedgerCostStatus, normalizeLedgerBillingType } from "./run-state.js";
|
||||
import { readGitConnectionFailure } from "../git-connection-failure.js";
|
||||
@@ -1363,10 +1364,17 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
|
||||
// same recorded lease; it never creates a replacement or a second executor.
|
||||
const reusableSandboxResumeFailure = !providerDispatchStarted && run.runtimeMode === "legacy"
|
||||
&& run.nativeSessionId === null && run.runnerInstanceId === null ? findReusableSandboxResumeError(outerErr) : null;
|
||||
// A saved workspace that could not be restored before provider dispatch
|
||||
// keeps its saved version. Retry it within the bootstrap budget only when
|
||||
// the owner proved the same clean writer and saved version; otherwise, and
|
||||
// on exhaustion, the board gets a named recovery action.
|
||||
const remoteWorkspaceRestoreFailure = !providerDispatchStarted && run.runtimeMode === "legacy"
|
||||
&& run.nativeSessionId === null && run.runnerInstanceId === null ? findRemoteWorkspaceRestoreError(outerErr) : null;
|
||||
const setupFailureErrorCode =
|
||||
(controllerLeaseLost ? CONTROLLER_LEASE_LOST_FAILURE_CODE : null) ??
|
||||
remoteWorkspacePreparationTimeout?.code ??
|
||||
reusableSandboxResumeFailure?.code ??
|
||||
remoteWorkspaceRestoreFailure?.code ??
|
||||
workspaceGitScanFailure?.code ??
|
||||
workspaceValidationSetupFailure?.code ??
|
||||
configurationIncompleteSetupFailure?.code ??
|
||||
@@ -1420,6 +1428,8 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
|
||||
...controllerLeaseLossEvidence(outerErr),
|
||||
...controllerLeaseLossResultJson(signal),
|
||||
...reviewedChatBindingDenialResultJson(outerErr),
|
||||
...(remoteWorkspaceRestoreFailure ? { remoteWorkspaceRestore: {
|
||||
step: remoteWorkspaceRestoreFailure.step, retryable: remoteWorkspaceRestoreFailure.retryable } } : {}),
|
||||
...(!providerDispatchStarted
|
||||
? { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } }
|
||||
: {}),
|
||||
@@ -1509,7 +1519,19 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
|
||||
() => undefined,
|
||||
);
|
||||
}
|
||||
if (livenessRun.errorCode !== PROJECT_ACCESS_DENIED) {
|
||||
if (livenessRun.errorCode === REMOTE_WORKSPACE_RESTORE_FAILED_CODE) {
|
||||
const retry = remoteWorkspaceRestoreFailure?.retryable
|
||||
? await scheduleBoundedRetryForRun(livenessRun, failedAgent).catch((retryError) => {
|
||||
logger.warn({ err: retryError, runId: livenessRun.id }, "failed to schedule a retry after a remote workspace restore failure");
|
||||
return null;
|
||||
})
|
||||
: null;
|
||||
if (retry?.outcome !== "scheduled") {
|
||||
await escalateRemoteWorkspaceRestoreFailure(db, livenessRun).catch((escalationError) => {
|
||||
logger.error({ err: escalationError, runId: livenessRun.id }, "failed to record a recovery action for a remote workspace restore failure");
|
||||
});
|
||||
}
|
||||
} else if (livenessRun.errorCode !== PROJECT_ACCESS_DENIED) {
|
||||
await (isTransientWorkspacePreparationCode(livenessRun.errorCode)
|
||||
? scheduleBoundedRetryForRun(livenessRun, failedAgent)
|
||||
: scheduleInteractionContinuationInfrastructureRetryIfEligible(livenessRun, failedAgent)
|
||||
|
||||
@@ -66,6 +66,9 @@ export type UpsertIssueRecoveryActionInput = {
|
||||
// one. The new failure then gets a distinct recovery identity and a fresh
|
||||
// operator notice, and the prior identity stays as a resolved record.
|
||||
supersedeOnIdentityChange?: boolean;
|
||||
// When true, an existing active action is returned unchanged. A concurrent
|
||||
// writer that wins the insert keeps its owner, cause, and evidence.
|
||||
createOnly?: boolean;
|
||||
// Rollout compatibility for active pre-policy actions. Refresh their
|
||||
// evidence/attempt metadata without silently changing the recorded owner or
|
||||
// the wake/monitor contract that made that owner authoritative.
|
||||
@@ -377,6 +380,7 @@ export function issueRecoveryActionService(db: DbOrTransaction) {
|
||||
): Promise<IssueRecoveryAction> {
|
||||
const existing = await getActiveForIssue(input.companyId, input.sourceIssueId);
|
||||
const now = new Date();
|
||||
if (existing && input.createOnly) return existing;
|
||||
const ownerType = input.ownerType ?? (input.ownerAgentId ? "agent" : "board");
|
||||
if (existing) {
|
||||
// A distinct failure identity must not overwrite the active action of a
|
||||
|
||||
@@ -10,7 +10,7 @@ const stopped = {
|
||||
},
|
||||
};
|
||||
|
||||
it.each(["workspace_git_scan_timeout", "workspace_git_scan_saturated", "remote_workspace_prepare_timeout", "reusable_sandbox_resume_failed"])("does not invent unknown provider actions after exhausted %s bootstrap retries", (errorCode) => {
|
||||
it.each(["workspace_git_scan_timeout", "workspace_git_scan_saturated", "remote_workspace_prepare_timeout", "reusable_sandbox_resume_failed", "remote_workspace_restore_failed"])("does not invent unknown provider actions after exhausted %s bootstrap retries", (errorCode) => {
|
||||
const run = { runtimeMode: "legacy", status: "failed", errorCode, scheduledRetryAttempt: 2,
|
||||
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } };
|
||||
expect(legacyExecutionNeedsReconciliation(run)).toBe(false);
|
||||
|
||||
@@ -6,7 +6,7 @@ import { hasRequiredWorkspaceRecovery, preserveWorkspaceRestoreRecoveryMetadata,
|
||||
import { hasUnrestoredRemoteWorkspace, preserveLegacyWorkspaceRestoreSources, type LegacyWorkspaceRestoreSource } from "./legacy-workspace-restore-recovery.js";
|
||||
import { isPreDispatchReviewWaitVerified } from "./pre-dispatch-review-wait.js";
|
||||
import { isPreDispatchExecutionWaitVerified } from "./pre-dispatch-execution-wait.js";
|
||||
import { REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE } from "./remote-workspace-transport.js";
|
||||
import { REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE, REMOTE_WORKSPACE_RESTORE_FAILED_CODE } from "./remote-workspace-transport.js";
|
||||
import { REUSABLE_SANDBOX_RESUME_FAILED_CODE } from "./reusable-sandbox-resume-error.js";
|
||||
import { hasWorkspaceRestoreFailure } from "@paperclipai/shared";
|
||||
import { normalizeMaxTurnStopReason } from "./heartbeat-stop-metadata.js";
|
||||
@@ -77,7 +77,8 @@ export function legacyExecutionNeedsReconciliation(
|
||||
// exhaustion needs workspace repair, not reconciliation of provider actions
|
||||
// that the bootstrap evidence proves never started. Keep unknown outcomes held.
|
||||
if ((run.errorCode === "workspace_git_scan_timeout" || run.errorCode === "workspace_git_scan_saturated"
|
||||
|| run.errorCode === REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE || run.errorCode === REUSABLE_SANDBOX_RESUME_FAILED_CODE) &&
|
||||
|| run.errorCode === REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE || run.errorCode === REUSABLE_SANDBOX_RESUME_FAILED_CODE
|
||||
|| run.errorCode === REMOTE_WORKSPACE_RESTORE_FAILED_CODE) &&
|
||||
evidence?.kind === "bootstrap" && evidence.providerWorkStarted === false) return false;
|
||||
if (executionFailureRetryCount(run) >= 2) return true;
|
||||
return !(
|
||||
|
||||
@@ -5,7 +5,7 @@ const run = (errorCode: string | null) =>
|
||||
({ errorCode } as unknown as Parameters<typeof classifyContinuationFailure>[0]);
|
||||
|
||||
describe("pause durability: continuation retry classification", () => {
|
||||
it.each(["workspace_git_scan_timeout", "workspace_git_scan_saturated", "workspace_git_scan_failed", "workspace_git_scan_output_limit", "workspace_git_scan_cancelled", "remote_workspace_prepare_timeout", "reusable_sandbox_resume_failed"])("does not grant %s another recovery budget", (code) => {
|
||||
it.each(["workspace_git_scan_timeout", "workspace_git_scan_saturated", "workspace_git_scan_failed", "workspace_git_scan_output_limit", "workspace_git_scan_cancelled", "remote_workspace_prepare_timeout", "reusable_sandbox_resume_failed", "remote_workspace_restore_failed"])("does not grant %s another recovery budget", (code) => {
|
||||
expect(classifyContinuationFailure(run(code))).toMatchObject({ kind: "non_retryable", maxAttempts: 0 });
|
||||
});
|
||||
it("agent_paused is retryable so work resumes (Option A: Resume Continues Work)", () => {
|
||||
|
||||
@@ -7,7 +7,7 @@ import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-pre
|
||||
import { notifyInstructionCollection } from "../instruction-collection-notifications.js";
|
||||
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
|
||||
import { boundRunIssueId } from "../run-task-binding.js";
|
||||
import { REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE } from "../remote-workspace-transport.js";
|
||||
import { REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE, REMOTE_WORKSPACE_RESTORE_FAILED_CODE } from "../remote-workspace-transport.js";
|
||||
import { REUSABLE_SANDBOX_RESUME_FAILED_CODE } from "../reusable-sandbox-resume-error.js";
|
||||
import { recoveryRunContextColumn, recoveryRunVersionColumn } from "./run-context-read.js";
|
||||
import { recoveryRunResultColumn } from "./run-result-read.js";
|
||||
@@ -557,6 +557,7 @@ const NON_RETRYABLE_CONTINUATION_ERROR_CODES = new Set<string>([
|
||||
"workspace_git_scan_failed",
|
||||
REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE,
|
||||
REUSABLE_SANDBOX_RESUME_FAILED_CODE,
|
||||
REMOTE_WORKSPACE_RESTORE_FAILED_CODE,
|
||||
"low_trust_isolation_unavailable",
|
||||
"low_trust_requires_isolated_workspace",
|
||||
"low_trust_boundary_mismatch",
|
||||
|
||||
@@ -8,6 +8,11 @@ import {
|
||||
type NoticeMetadataSection,
|
||||
} from "./notice-format.js";
|
||||
|
||||
/** The saved version is unchanged by a failed restore; the run names the failed step. */
|
||||
export const REMOTE_WORKSPACE_RESTORE_NEXT_ACTION =
|
||||
"Open the failed run to see which restore step failed. The saved workspace version was kept. "
|
||||
+ "If the sandbox folder is already occupied, inspect and move it aside, then retry the task.";
|
||||
|
||||
// Short human-readable body plus the presentation header for one recovery
|
||||
// family. The escalation path merges in the metadata rows only it knows
|
||||
// (recovery action, owner, source run) via buildStrandedRecoveryEscalationNotice.
|
||||
@@ -50,6 +55,7 @@ const STRANDED_RECOVERY_NOTICE_TITLES_BY_RUN_ERROR_CODE: Record<string, string>
|
||||
const WORKSPACE_SCAN_NOTICES: Record<string, { title: string; nextAction: string }> = {
|
||||
workspace_git_scan_timeout: { title: "Workspace scan timed out", nextAction: "Check repository access and server load, then retry the task." },
|
||||
remote_workspace_prepare_timeout: { title: "Workspace setup timed out", nextAction: "Inspect the failed run and check repository access and sandbox load before retrying the task." },
|
||||
remote_workspace_restore_failed: { title: "Saved workspace could not be restored", nextAction: REMOTE_WORKSPACE_RESTORE_NEXT_ACTION },
|
||||
reusable_sandbox_resume_failed: { title: "Saved sandbox could not be resumed", nextAction: "Inspect the failed run and the sandbox provider; the saved sandbox was kept. Retry the task once the provider can start it." },
|
||||
workspace_git_scan_saturated: { title: "Workspace scan queue is full", nextAction: "Check server load and the workspace scan queue, then retry the task." },
|
||||
workspace_git_scan_output_limit: { title: "Workspace scan exceeded its limit", nextAction: "Check the repository size and workspace scan output limit before retrying the task." },
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { heartbeatRuns, issues, type Db } from "@paperclipai/db";
|
||||
import { issueRecoveryActionService } from "./issue-recovery-actions.js";
|
||||
import { REMOTE_WORKSPACE_RESTORE_FAILED_CODE } from "./remote-workspace-transport.js";
|
||||
import { REMOTE_WORKSPACE_RESTORE_NEXT_ACTION } from "./recovery/stranded-notice.js";
|
||||
|
||||
type Run = typeof heartbeatRuns.$inferSelect;
|
||||
|
||||
// Stranded-work recovery scans these statuses and names this failure itself.
|
||||
const STRANDED_RECOVERY_STATUSES = new Set(["todo", "in_progress", "in_review"]);
|
||||
const TERMINAL_STATUSES = new Set(["done", "cancelled"]);
|
||||
|
||||
/**
|
||||
* Leave a named board recovery action for a saved workspace that could not be
|
||||
* restored and will not be retried automatically. Stranded-work recovery owns
|
||||
* the statuses it scans; this covers the rest (for example a `blocked` task
|
||||
* woken by a reply), which would otherwise keep no recovery action at all.
|
||||
* The task row is locked while its status and assignee are checked, and an
|
||||
* active action, including one a concurrent writer just created, is kept as is.
|
||||
*/
|
||||
export async function escalateRemoteWorkspaceRestoreFailure(db: Db, run: Run): Promise<boolean> {
|
||||
if (run.errorCode !== REMOTE_WORKSPACE_RESTORE_FAILED_CODE) return false;
|
||||
const contextIssueId = run.contextSnapshot?.issueId;
|
||||
const issueId = run.nativeIssueId ?? run.issueId ?? (typeof contextIssueId === "string" ? contextIssueId : null);
|
||||
if (!issueId) return false;
|
||||
return db.transaction(async tx => {
|
||||
const [issue] = await tx.select({ id: issues.id, status: issues.status, assigneeAgentId: issues.assigneeAgentId,
|
||||
assigneeUserId: issues.assigneeUserId })
|
||||
.from(issues).where(and(eq(issues.companyId, run.companyId), eq(issues.id, issueId))).for("update");
|
||||
if (!issue || TERMINAL_STATUSES.has(issue.status) || STRANDED_RECOVERY_STATUSES.has(issue.status)
|
||||
|| issue.assigneeUserId || issue.assigneeAgentId !== run.agentId) return false;
|
||||
const workspaceId = run.contextSnapshot?.executionWorkspaceId;
|
||||
const action = await issueRecoveryActionService(tx as unknown as Db).upsertSourceScoped({
|
||||
companyId: run.companyId,
|
||||
sourceIssueId: issue.id,
|
||||
kind: "stranded_assigned_issue",
|
||||
ownerType: "board",
|
||||
returnOwnerAgentId: run.agentId,
|
||||
cause: REMOTE_WORKSPACE_RESTORE_FAILED_CODE,
|
||||
fingerprint: `remote-workspace-restore:${typeof workspaceId === "string" ? workspaceId : issue.id}`,
|
||||
evidence: {
|
||||
runId: run.id,
|
||||
originalFailureCode: run.errorCode,
|
||||
// The run error is the server-built, redacted restore cause.
|
||||
restoreFailure: run.error,
|
||||
executionRecovery: { kind: "bootstrap", providerWorkStarted: false },
|
||||
},
|
||||
nextAction: REMOTE_WORKSPACE_RESTORE_NEXT_ACTION,
|
||||
maxAttempts: null,
|
||||
wakePolicy: null,
|
||||
createOnly: true,
|
||||
});
|
||||
return action.cause === REMOTE_WORKSPACE_RESTORE_FAILED_CODE && action.evidence?.runId === run.id;
|
||||
});
|
||||
}
|
||||
@@ -241,6 +241,73 @@ describe("remote workspace transport safety with real sandbox commands", () => {
|
||||
} finally { await captured.cleanup(); }
|
||||
});
|
||||
|
||||
it("names the restore step that refused an occupied folder without touching its files", async () => {
|
||||
await fs.writeFile(path.join(workspace, "saved.txt"), "saved synthetic work\n");
|
||||
const { remote } = target();
|
||||
const captured = await captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() });
|
||||
try {
|
||||
const root = path.join(scratch, "occupied-secret-folder");
|
||||
await fs.mkdir(root); await fs.writeFile(path.join(root, "later.txt"), "unsaved later work\n");
|
||||
const failure = await restoreRemoteWorkspaceDirectory({ target: remote, remoteCwd: root, archivePath: captured.archivePath,
|
||||
manifestDigest: captured.manifestDigest, restorationIdentity: "e".repeat(64) }).then(() => null, (error: unknown) => error as Error);
|
||||
if (!failure) throw new Error("restore unexpectedly succeeded");
|
||||
expect(failure.message).toBe("remote_workspace_operation_failed: operation=restore exit=1; "
|
||||
+ "remote_workspace_restore_failed: stage=inspect_root reason=occupied_root");
|
||||
expect(remoteWorkspaceFailureCause(failure)).toBe(failure.message);
|
||||
expect(await fs.readdir(root)).toEqual(["later.txt"]);
|
||||
} finally { await captured.cleanup(); }
|
||||
});
|
||||
|
||||
it("names a rejected archive during restore by its closed archive reason", async () => {
|
||||
await fs.writeFile(path.join(workspace, "saved.txt"), "saved synthetic work\n");
|
||||
const { remote } = target();
|
||||
const captured = await captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() });
|
||||
try {
|
||||
const root = path.join(scratch, "restored-with-wrong-digest");
|
||||
await expect(restoreRemoteWorkspaceDirectory({ target: remote, remoteCwd: root, archivePath: captured.archivePath,
|
||||
manifestDigest: "f".repeat(64), restorationIdentity: "e".repeat(64) }))
|
||||
.rejects.toThrow(/^remote_workspace_operation_failed: operation=restore exit=1; remote_workspace_restore_failed: stage=extract archive=manifest_mismatch exit=1$/);
|
||||
expect(await fs.lstat(root).catch(() => null)).toBeNull();
|
||||
} finally { await captured.cleanup(); }
|
||||
});
|
||||
|
||||
it("names the preparation step that could not select a missing branch", async () => {
|
||||
const origin = await repository(), { remote } = target(), root = path.join(workspace, "missing-branch-cause");
|
||||
await expect(prepareRemoteWorkspaceDirectory({ target: remote, remoteCwd: root,
|
||||
preparationIdentity: createHash("sha256").update("synthetic-missing-branch-cause").digest("hex"),
|
||||
source: { kind: "git", repoUrl: origin, ref: "main", branchName: "feature/missing", branchMode: "existing" },
|
||||
resolveGitAuth: async () => null }))
|
||||
.rejects.toThrow(/^remote_workspace_operation_failed: operation=prepare exit=1; remote_workspace_prepare_failed: stage=checkout_branch exit=1 program=git$/);
|
||||
});
|
||||
|
||||
it("names the repository step that found a changed origin", async () => {
|
||||
const origin = await repository(), { remote } = target(), root = path.join(workspace, "repos", "changed-origin");
|
||||
await prepareRemoteRepository({ target: remote, remoteCwd: root, repoUrl: origin, ref: "HEAD", resolveGitAuth: async () => null });
|
||||
await expect(prepareRemoteRepository({ target: remote, remoteCwd: root, repoUrl: `${origin}-moved`, ref: "HEAD",
|
||||
resolveGitAuth: async () => null }))
|
||||
.rejects.toThrow(/^remote_workspace_operation_failed: operation=repository exit=1; remote_workspace_repository_failed: stage=verify_origin reason=repository_origin_changed$/);
|
||||
});
|
||||
|
||||
it.each(["prepare", "repository", "restore"] as const)("drops %s output that is outside the closed cause grammar", async operation => {
|
||||
const { remote, execute } = target();
|
||||
const forged = `remote_workspace_${operation}_failed: stage=extract reason=/home/secret`;
|
||||
if (operation === "restore") execute.mockResolvedValueOnce({ exitCode: 0, stdout: "", stderr: "", timedOut: false,
|
||||
signal: null, pid: process.pid, startedAt: new Date().toISOString() });
|
||||
execute.mockResolvedValueOnce({ exitCode: 1, stdout: "", stderr: forged, timedOut: false,
|
||||
signal: null, pid: process.pid, startedAt: new Date().toISOString() });
|
||||
const request = operation === "prepare"
|
||||
? prepareRemoteWorkspaceDirectory({ target: remote, remoteCwd: workspace, source: { kind: "empty" },
|
||||
preparationIdentity: "a".repeat(64), resolveGitAuth: async () => null })
|
||||
: operation === "repository"
|
||||
? prepareRemoteRepository({ target: remote, remoteCwd: workspace, repoUrl: "https://example.invalid/fixture.git", ref: null, resolveGitAuth: async () => null })
|
||||
: (async () => {
|
||||
const archivePath = path.join(scratch, "forged.pcws"); await fs.writeFile(archivePath, "synthetic archive");
|
||||
return restoreRemoteWorkspaceDirectory({ target: remote, remoteCwd: path.join(scratch, "forged"), archivePath,
|
||||
manifestDigest: "a".repeat(64), restorationIdentity: "b".repeat(64) });
|
||||
})();
|
||||
await expect(request).rejects.toThrow(new RegExp(`^remote_workspace_operation_failed: operation=${operation} exit=1$`));
|
||||
});
|
||||
|
||||
it("publishes one complete capture pair when the same operation starts simultaneously", async () => {
|
||||
await fs.writeFile(path.join(workspace, "saved.txt"), "simultaneously captured work\n");
|
||||
const barrier = path.join(scratch, "capture-start-barrier"); await fs.mkdir(barrier);
|
||||
|
||||
@@ -31,8 +31,26 @@ const CAPTURE_FAILURE_CHECK = `const captureFailure=${remoteWorkspaceCaptureFail
|
||||
// The only sandbox text a capture failure may add: a fixed stage plus bounded
|
||||
// structured fields from remoteWorkspaceFailureDetail. Anything else is dropped.
|
||||
const SANDBOX_CAPTURE_CAUSE = /^remote_workspace_capture_failed: stage=(?:helper|archive|publish)(?: (?:helper_)?(?:code|exit|signal|syscall|killed|program|name|reason)=[A-Za-z0-9_]{1,64}){0,16}$/;
|
||||
// Setup scripts name the step they were in plus closed, structured fields. A
|
||||
// reason is only a script's own snake_case refusal code; Git output selects a
|
||||
// fixed label and is never copied. The server drops anything outside this grammar.
|
||||
const SETUP_FAILURE_CHECK = String.raw`const setupFailure=(prefix,stage,error)=>{const fields=['stage='+stage];
|
||||
const message=typeof error?.message==='string'?error.message:'',stderr=typeof error?.stderr==='string'?error.stderr.trim():'';
|
||||
if(/^[a-z][a-z_]{2,47}$/.test(message))fields.push('reason='+message);
|
||||
const archive=/^remote_workspace_archive_([a-z_]{1,48})$/.exec(stderr)?.[1];if(archive)fields.push('archive='+archive);
|
||||
const errno=/^(E[A-Z0-9]{1,20}):/.exec(stderr)?.[1];if(errno)fields.push('code='+errno);
|
||||
if(diskFull(error))fields.push('disk=full');
|
||||
if(typeof error?.cmd==='string'&&error.cmd.startsWith('git ')){
|
||||
const git=[[/Authentication failed|could not read Username|Permission denied \(publickey\)|terminal prompts disabled|returned error: 40[13]/i,'auth_failed'],
|
||||
[/Repository not found|does not appear to be a git repository|returned error: 404/i,'repository_not_found'],
|
||||
[/Could not resolve host|Connection (?:timed out|refused|reset)|Failed to connect|unable to access/i,'network'],
|
||||
[/unknown revision|not a valid ref|Needed a single revision|invalid reference|did not match any|couldn't find remote ref/i,'ref_not_found']]
|
||||
.find(([pattern])=>pattern.test(stderr))?.[1];if(git)fields.push('git='+git);}
|
||||
for(const field of failureDetail(error).split(' ').filter(Boolean))if(!fields.some(value=>value.startsWith(field.split('=')[0]+'=')))fields.push(field);
|
||||
return prefix+': '+fields.join(' ');};`;
|
||||
const SANDBOX_SETUP_CAUSE = /^remote_workspace_(prepare|repository|restore)_failed: stage=[a-z_]{1,32}(?: (?:reason|archive|disk|git|code|exit|signal|syscall|killed|program|name)=[A-Za-z0-9_]{1,64}){0,16}$/;
|
||||
|
||||
type RemoteWorkspaceOperation = "prepare" | "repository" | "archive_helper" | "capture" | "download" | "restore" | "cleanup";
|
||||
type RemoteWorkspaceOperation = "prepare" | "repository" | "archive_helper" | "capture" | "download" | "upload" | "restore" | "cleanup";
|
||||
|
||||
export class RemoteWorkspaceOperationTimeoutError extends Error {
|
||||
constructor(readonly operation: RemoteWorkspaceOperation, readonly timeoutMs: number) {
|
||||
@@ -57,7 +75,8 @@ export class RemoteWorkspaceOperationError extends Error {
|
||||
* error text is not one of these redacted forms and must stay off the record.
|
||||
*/
|
||||
export function remoteWorkspaceFailureCause(error: unknown): string | null {
|
||||
if (error instanceof RemoteWorkspaceOperationError || error instanceof RemoteWorkspaceOperationTimeoutError) return error.message;
|
||||
if (error instanceof RemoteWorkspaceOperationError || error instanceof RemoteWorkspaceOperationTimeoutError
|
||||
|| error instanceof RemoteWorkspaceRestoreError) return error.message;
|
||||
if (!(error instanceof Error)) return null;
|
||||
if (remoteWorkspaceCaptureFailureMessage({ message: error.message }) === error.message) return error.message;
|
||||
const operation = providerRefusals.get(error);
|
||||
@@ -76,6 +95,45 @@ export class RemoteWorkspacePreparationTimeoutError extends RemoteWorkspaceOpera
|
||||
}
|
||||
}
|
||||
|
||||
export const REMOTE_WORKSPACE_RESTORE_FAILED_CODE = "remote_workspace_restore_failed";
|
||||
|
||||
/** Where restoring a saved workspace into a new allocation stopped. */
|
||||
export type RemoteWorkspaceRestoreStep = "snapshot_lookup" | "snapshot_download" | "sandbox_restore";
|
||||
|
||||
/**
|
||||
* A saved workspace that could not be restored before the agent started. The
|
||||
* saved version is never changed by a failed restore. The message names the
|
||||
* step and a redacted cause; provider or command text is never copied.
|
||||
* `retryable` is set only when the owner re-read the same writer, epoch, and
|
||||
* saved version with no unsaved work, so a bounded retry cannot overwrite work.
|
||||
*/
|
||||
export class RemoteWorkspaceRestoreError extends Error {
|
||||
readonly code = REMOTE_WORKSPACE_RESTORE_FAILED_CODE;
|
||||
|
||||
constructor(readonly step: RemoteWorkspaceRestoreStep, readonly retryable: boolean, cause: unknown) {
|
||||
super(`${REMOTE_WORKSPACE_RESTORE_FAILED_CODE}: step=${step}; ${remoteWorkspaceRestoreFailureDetail(step, cause)}`);
|
||||
this.name = "RemoteWorkspaceRestoreError";
|
||||
this.cause = cause;
|
||||
}
|
||||
}
|
||||
|
||||
function remoteWorkspaceRestoreFailureDetail(step: RemoteWorkspaceRestoreStep, cause: unknown) {
|
||||
const known = remoteWorkspaceFailureCause(cause);
|
||||
if (known) return known;
|
||||
// Server-side restore refusals are fixed snake_case codes.
|
||||
if (cause instanceof Error && /^remote_workspace_[a-z_]{3,64}$/.test(cause.message)) return cause.message;
|
||||
const { label, code, status } = sandboxProviderFailureLabel(cause);
|
||||
return `${step === "snapshot_download" ? "storage" : "transport"}=${label}${code ? ` code=${code}` : ""}${status ? ` status=${status}` : ""}`;
|
||||
}
|
||||
|
||||
/** The restore failure carried by a setup error, through at most four causes. */
|
||||
export function findRemoteWorkspaceRestoreError(error: unknown): RemoteWorkspaceRestoreError | null {
|
||||
for (let cause = error, depth = 0; cause && depth < 4; cause = (cause as { cause?: unknown }).cause, depth++) {
|
||||
if (cause instanceof RemoteWorkspaceRestoreError) return cause;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
const TRANSPORT_FAILURE_LABELS: Array<[RegExp, string]> = [
|
||||
[/daytona_sandbox_not_found/i, "daytona_sandbox_not_found"],
|
||||
[/sandbox lease \S+ is no longer active/i, "sandbox_lease_inactive"],
|
||||
@@ -137,8 +195,8 @@ async function command(target: RemoteWorkspaceTarget, operation: RemoteWorkspace
|
||||
args: string[], env?: Record<string, string>, timeoutMs = 300_000) {
|
||||
const execute = () => target.runner!.execute({ command: "node", args, cwd: target.remoteCwd,
|
||||
env: { NODE_OPTIONS: "", NODE_PATH: "", ...env }, timeoutMs, bypassSession: true });
|
||||
// Copy-back records why the provider refused the command.
|
||||
const result = operation === "capture" || operation === "archive_helper"
|
||||
// Copy-back and restore record why the provider refused the command.
|
||||
const result = operation === "capture" || operation === "archive_helper" || operation === "restore"
|
||||
? await execute().catch((error: unknown) => { noteProviderRefusal(operation, error); throw error; })
|
||||
: await execute();
|
||||
if (result.timedOut) throw new RemoteWorkspaceOperationTimeoutError(operation, timeoutMs);
|
||||
@@ -152,7 +210,8 @@ async function command(target: RemoteWorkspaceTarget, operation: RemoteWorkspace
|
||||
? output.find(value => remoteWorkspaceCaptureFailureMessage({ message: value }) === value) : undefined);
|
||||
if (reported) throw new Error(reported);
|
||||
const cause = operation === "capture" || operation === "archive_helper"
|
||||
? output.find(value => value.length <= 1024 && SANDBOX_CAPTURE_CAUSE.test(value)) : undefined;
|
||||
? output.find(value => value.length <= 1024 && SANDBOX_CAPTURE_CAUSE.test(value))
|
||||
: output.find(value => value.length <= 1024 && SANDBOX_SETUP_CAUSE.exec(value)?.[1] === operation);
|
||||
const status = [`operation=${operation}`,
|
||||
Number.isSafeInteger(result.exitCode) ? `exit=${result.exitCode}` : null,
|
||||
typeof result.signal === "string" && /^SIG[A-Z0-9]{1,12}$/.test(result.signal) ? `signal=${result.signal}` : null,
|
||||
@@ -165,6 +224,7 @@ async function command(target: RemoteWorkspaceTarget, operation: RemoteWorkspace
|
||||
const PREPARE = String.raw`
|
||||
const fs = require('node:fs/promises'), path = require('node:path'), cp = require('node:child_process');
|
||||
const {promisify}=require('node:util'); const exec=promisify(cp.execFile);
|
||||
${DISK_FULL_CHECK}${CAPTURE_FAILURE_CHECK}${SETUP_FAILURE_CHECK}let stage='validate';
|
||||
(async()=>{
|
||||
const [root,raw]=process.argv.slice(1), input=JSON.parse(raw);
|
||||
if (!path.isAbsolute(root)||!/^[a-f0-9]{64}$/.test(input.preparationIdentity)) throw Error('invalid_root');
|
||||
@@ -172,11 +232,11 @@ const {promisify}=require('node:util'); const exec=promisify(cp.execFile);
|
||||
const isolated={...process.env,...noConfig};for(const key of Object.keys(isolated))if(/^GIT_CONFIG_(KEY|VALUE)_\d+$/.test(key))delete isolated[key];
|
||||
const cloneEnv={...isolated};for(const key of input.authEnvKeys)if(/^GIT_CONFIG_(COUNT|(KEY|VALUE)_\d+)$/.test(key)&&process.env[key]!==undefined)cloneEnv[key]=process.env[key];
|
||||
const localEnv={...isolated};for(const key of input.authEnvKeys)delete localEnv[key];Object.assign(localEnv,noConfig);
|
||||
let current='/'; for(const part of path.dirname(root).split('/').filter(Boolean)) {
|
||||
stage='create_parent';let current='/'; for(const part of path.dirname(root).split('/').filter(Boolean)) {
|
||||
current=path.join(current,part); await fs.mkdir(current,{mode:0o700}).catch(e=>{if(e.code!=='EEXIST')throw e});
|
||||
const s=await fs.lstat(current); if(!s.isDirectory()||s.isSymbolicLink())throw Error('unsafe_root');
|
||||
}
|
||||
const prior=await fs.lstat(root).catch(e=>{if(e.code==='ENOENT')return null;throw e});
|
||||
stage='inspect_existing';const prior=await fs.lstat(root).catch(e=>{if(e.code==='ENOENT')return null;throw e});
|
||||
if(prior){
|
||||
if(!prior.isDirectory()||prior.isSymbolicLink()||await fs.realpath(root)!==root)throw Error('unsafe_root');
|
||||
const runtime=path.join(root,'.paperclip-runtime'),marker=path.join(runtime,'workspace-ready.json');
|
||||
@@ -186,33 +246,34 @@ const {promisify}=require('node:util'); const exec=promisify(cp.execFile);
|
||||
process.stdout.write(JSON.stringify({cwd:root}));return;
|
||||
}
|
||||
if(input.probeOnly){process.stdout.write(JSON.stringify({cloneRequired:true}));return;}
|
||||
const staged=await fs.mkdtemp(root+'.preparing-');
|
||||
stage='stage';const staged=await fs.mkdtemp(root+'.preparing-');
|
||||
if(input.kind==='git'){
|
||||
await exec('git',['-c','core.hooksPath=/dev/null','-c','init.templateDir=',...input.authArgs,'clone','--no-hardlinks','--',input.repoUrl,staged],{env:cloneEnv});
|
||||
stage='clone';await exec('git',['-c','core.hooksPath=/dev/null','-c','init.templateDir=',...input.authArgs,'clone','--no-hardlinks','--',input.repoUrl,staged],{env:cloneEnv});
|
||||
const git=async args=>exec('git',['-c','core.hooksPath=/dev/null','-C',staged,...args],{env:localEnv});
|
||||
if(input.branchMode==='existing'){
|
||||
if(!input.branchName)throw Error('existing_branch_required');
|
||||
stage='checkout_branch';if(!input.branchName)throw Error('existing_branch_required');
|
||||
await git(['check-ref-format','--branch',input.branchName]);
|
||||
const remoteRef='refs/remotes/origin/'+input.branchName;
|
||||
await git(['rev-parse','--verify','--quiet',remoteRef+'^{commit}']);
|
||||
const local=await git(['show-ref','--verify','--quiet','refs/heads/'+input.branchName]).then(()=>true,()=>false);
|
||||
await git(local?['checkout',input.branchName]:['checkout','-b',input.branchName,remoteRef]);
|
||||
}else{
|
||||
stage='checkout_ref';
|
||||
${checkoutConfiguredRefScript}
|
||||
if(input.branchName)await git(['checkout','-B',input.branchName]);
|
||||
}
|
||||
await fs.mkdir(path.join(staged,'.git','info'),{recursive:true,mode:0o700});
|
||||
stage='configure';await fs.mkdir(path.join(staged,'.git','info'),{recursive:true,mode:0o700});
|
||||
await fs.appendFile(path.join(staged,'.git','info','exclude'),'\n/.paperclip-runtime/\n/.paperclip-repositories/\n');
|
||||
}else{
|
||||
await fs.writeFile(path.join(staged,'.gitignore'),'/.paperclip-runtime/\n/.paperclip-repositories/\n',{flag:'wx',mode:0o600});
|
||||
stage='configure';await fs.writeFile(path.join(staged,'.gitignore'),'/.paperclip-runtime/\n/.paperclip-repositories/\n',{flag:'wx',mode:0o600});
|
||||
}
|
||||
const runtime=path.join(staged,'.paperclip-runtime');await fs.mkdir(runtime,{mode:0o700}).catch(e=>{if(e.code!=='EEXIST')throw e});
|
||||
stage='write_receipt';const runtime=path.join(staged,'.paperclip-runtime');await fs.mkdir(runtime,{mode:0o700}).catch(e=>{if(e.code!=='EEXIST')throw e});
|
||||
const runtimeStat=await fs.lstat(runtime);if(!runtimeStat.isDirectory()||runtimeStat.isSymbolicLink())throw Error('unsafe_receipt');
|
||||
await fs.writeFile(path.join(runtime,'workspace-ready.json'),JSON.stringify({preparationIdentity:input.preparationIdentity}),{flag:'wx',mode:0o600});
|
||||
if(await fs.realpath(path.dirname(root))!==path.dirname(root))throw Error('unsafe_root');
|
||||
stage='publish';if(await fs.realpath(path.dirname(root))!==path.dirname(root))throw Error('unsafe_root');
|
||||
await fs.rename(staged,root);
|
||||
process.stdout.write(JSON.stringify({cwd:await fs.realpath(root)}));
|
||||
})().catch(()=>{process.stderr.write('remote_workspace_prepare_failed');process.exitCode=1});`;
|
||||
})().catch(error=>{process.stderr.write(setupFailure('remote_workspace_prepare_failed',stage,error));process.exitCode=1});`;
|
||||
|
||||
export async function prepareRemoteWorkspaceDirectory(input: {
|
||||
target: RemoteWorkspaceTarget; remoteCwd: string; source: RemoteWorkspaceSource; resolveGitAuth: GitRemoteAuthProvider;
|
||||
@@ -249,30 +310,32 @@ export async function prepareRemoteRepository(input: {
|
||||
pinnedCommit?: string; verifyOnly?: boolean; savedArchive?: boolean; resolveGitAuth: GitRemoteAuthProvider;
|
||||
}) {
|
||||
const script = String.raw`const fs=require('node:fs/promises'),path=require('node:path'),cp=require('node:child_process');const {promisify}=require('node:util');const exec=promisify(cp.execFile);
|
||||
${DISK_FULL_CHECK}${CAPTURE_FAILURE_CHECK}${SETUP_FAILURE_CHECK}let stage='validate';
|
||||
(async()=>{const [root,raw]=process.argv.slice(1),input=JSON.parse(raw);const noConfig={GIT_TERMINAL_PROMPT:'0',GIT_CONFIG_NOSYSTEM:'1',GIT_CONFIG_GLOBAL:'/dev/null',GIT_CONFIG_SYSTEM:'/dev/null',GIT_CONFIG_COUNT:'0',GIT_CONFIG_PARAMETERS:'',GIT_TEMPLATE_DIR:'',GIT_ASKPASS:'',SSH_ASKPASS:''};
|
||||
const env={...process.env,...noConfig};for(const key of Object.keys(env))if(/^GIT_CONFIG_(KEY|VALUE)_\d+$/.test(key))delete env[key];
|
||||
const cloneEnv={...env};for(const key of input.authEnvKeys)if(/^GIT_CONFIG_(COUNT|(KEY|VALUE)_\d+)$/.test(key)&&process.env[key]!==undefined)cloneEnv[key]=process.env[key];
|
||||
const localEnv={...env};for(const key of input.authEnvKeys)delete localEnv[key];Object.assign(localEnv,noConfig);const opts={env:localEnv};
|
||||
let parent='/';for(const part of path.dirname(root).split('/').filter(Boolean)){parent=path.join(parent,part);
|
||||
stage='create_parent';let parent='/';for(const part of path.dirname(root).split('/').filter(Boolean)){parent=path.join(parent,part);
|
||||
await fs.mkdir(parent,{mode:0o700}).catch(e=>{if(e.code!=='EEXIST')throw e});const entry=await fs.lstat(parent);
|
||||
if(!entry.isDirectory()||entry.isSymbolicLink()||await fs.realpath(parent)!==parent)throw Error('unsafe_repository_parent');}
|
||||
const existing=await fs.lstat(root).catch(e=>{if(e.code==='ENOENT')return null;throw e});
|
||||
stage='inspect';const existing=await fs.lstat(root).catch(e=>{if(e.code==='ENOENT')return null;throw e});
|
||||
if(existing){if(!existing.isDirectory()||existing.isSymbolicLink())throw Error('unsafe_repository')}
|
||||
else{if(input.verifyOnly||input.savedArchive)throw Error('repository_missing');if(input.probeOnly){process.stdout.write(JSON.stringify({cloneRequired:true}));return;}
|
||||
const staged=await fs.mkdtemp(root+'.preparing-');try{
|
||||
await exec('git',['-c','core.hooksPath=/dev/null','-c','init.templateDir=',...input.authArgs,'clone','--no-hardlinks','--',input.repoUrl,staged],{env:cloneEnv});
|
||||
stage='stage';const staged=await fs.mkdtemp(root+'.preparing-');try{
|
||||
stage='clone';await exec('git',['-c','core.hooksPath=/dev/null','-c','init.templateDir=',...input.authArgs,'clone','--no-hardlinks','--',input.repoUrl,staged],{env:cloneEnv});
|
||||
const git=async args=>exec('git',['-c','core.hooksPath=/dev/null','-C',staged,...args],opts);
|
||||
stage='checkout_ref';
|
||||
${checkoutConfiguredRefScript}
|
||||
await fs.rename(staged,root);
|
||||
stage='publish';await fs.rename(staged,root);
|
||||
}finally{await fs.rm(staged,{recursive:true,force:true});}}
|
||||
const git=async args=>(await exec('git',['-c','core.hooksPath=/dev/null','-C',root,...args],opts)).stdout.trim();
|
||||
if(await fs.realpath(root)!==root)throw Error('unsafe_repository');
|
||||
stage='verify_origin';if(await fs.realpath(root)!==root)throw Error('unsafe_repository');
|
||||
const hasOrigin=(await git(['remote'])).split('\n').includes('origin');
|
||||
const origin=hasOrigin?await git(['remote','get-url','origin']):null;
|
||||
if(origin!==input.repoUrl&&!(input.savedArchive&&origin===null))throw Error('repository_origin_changed');
|
||||
const head=await git(['rev-parse','HEAD']);if(input.pinnedCommit)await git(['cat-file','-e',input.pinnedCommit+'^{commit}']);
|
||||
stage='verify_commit';const head=await git(['rev-parse','HEAD']);if(input.pinnedCommit)await git(['cat-file','-e',input.pinnedCommit+'^{commit}']);
|
||||
const branch=await git(['symbolic-ref','--short','-q','HEAD']).catch(()=>null);process.stdout.write(JSON.stringify({cwd:root,pinnedCommit:input.pinnedCommit||head,branchName:branch}));
|
||||
})().catch(()=>{process.stderr.write('remote_workspace_repository_failed');process.exitCode=1});`;
|
||||
})().catch(error=>{process.stderr.write(setupFailure('remote_workspace_repository_failed',stage,error));process.exitCode=1});`;
|
||||
const payload = { ...input, target: undefined, resolveGitAuth: undefined };
|
||||
const prior = JSON.parse(await command(input.target, "repository", ["-e", script, input.remoteCwd,
|
||||
JSON.stringify({ ...payload, authArgs: [], authEnvKeys: [], probeOnly: true })])) as {
|
||||
@@ -400,10 +463,11 @@ export async function restoreRemoteWorkspaceDirectory(input: {
|
||||
const staged = await installArchiveHelper(input.target, parent, operationId);
|
||||
await input.target.runner!.syncIn!([{ operationId,
|
||||
files: [{ kind: "file", sourcePath: input.archivePath, targetPath: staged.archive }],
|
||||
}]);
|
||||
}]).catch((error: unknown) => { noteProviderRefusal("upload", error); throw error; });
|
||||
// Publish the restored folder and its identity together. A lost successful
|
||||
// reply can then adopt the same files without repeating a destructive restore.
|
||||
const restore = String.raw`const fs=require('node:fs/promises'),path=require('node:path'),cp=require('node:child_process'),crypto=require('node:crypto');
|
||||
${DISK_FULL_CHECK}${CAPTURE_FAILURE_CHECK}${SETUP_FAILURE_CHECK}let stage='validate';
|
||||
(async()=>{const [helper,archive,root,digest,identity,allowEmpty,rawFiles]=process.argv.slice(1),files=JSON.parse(rawFiles);
|
||||
if(!/^[a-f0-9]{64}$/.test(identity)||!Array.isArray(files)||files.length>8)throw Error('invalid_identity');
|
||||
for(const file of files)if(!/^\.paperclip-runtime\/[a-zA-Z0-9_-]+\.json$/.test(file.relativePath)
|
||||
@@ -424,31 +488,31 @@ export async function restoreRemoteWorkspaceDirectory(input: {
|
||||
return {name:path.basename(name),content};}));
|
||||
};
|
||||
const parked=root+'.allocation-'+identity;
|
||||
if(await stat(parked)){if(allowEmpty!=='true')throw Error('unexpected_allocation');await allocation(parked);if(!await stat(root))await fs.rename(parked,root);}
|
||||
const prior=await stat(root);if(prior)await directory(root);
|
||||
stage='allocation';if(await stat(parked)){if(allowEmpty!=='true')throw Error('unexpected_allocation');await allocation(parked);if(!await stat(root))await fs.rename(parked,root);}
|
||||
stage='inspect_root';const prior=await stat(root);if(prior)await directory(root);
|
||||
const marker=path.join(root,'.paperclip-runtime','workspace-restored.json');
|
||||
if(prior&&await stat(marker)){
|
||||
await directory(path.dirname(marker));const file=await fs.lstat(marker);
|
||||
stage='existing_receipt';await directory(path.dirname(marker));const file=await fs.lstat(marker);
|
||||
if(!file.isFile()||file.isSymbolicLink()||file.nlink!==1||file.size>16384)throw Error('unsafe_receipt');
|
||||
const receipt=JSON.parse(await fs.readFile(marker,'utf8'));
|
||||
if(receipt.identity!==identity||receipt.manifestDigest!==digest)throw Error('workspace_owner_changed');
|
||||
if(await stat(parked)){await allocation(parked);await fs.rm(parked,{recursive:true});}return;
|
||||
}
|
||||
if(prior&&allowEmpty!=='true')throw Error('occupied_root');
|
||||
const retained=prior?await allocation(root):[];
|
||||
stage='allocation';const retained=prior?await allocation(root):[];
|
||||
if(!prior&&files.length)throw Error('allocation_missing');
|
||||
const staged=await fs.mkdtemp(root+'.restoring-');
|
||||
stage='extract';const staged=await fs.mkdtemp(root+'.restoring-');
|
||||
try{
|
||||
await require('node:util').promisify(cp.execFile)(process.execPath,[helper,'restore',archive,staged,digest],{maxBuffer:1048576});
|
||||
const runtime=path.join(staged,'.paperclip-runtime');await fs.mkdir(runtime,{mode:0o700});
|
||||
stage='write_receipt';const runtime=path.join(staged,'.paperclip-runtime');await fs.mkdir(runtime,{mode:0o700});
|
||||
for(const file of retained)await fs.writeFile(path.join(runtime,file.name),file.content,{flag:'wx',mode:0o600});
|
||||
await fs.writeFile(path.join(runtime,'workspace-restored.json'),JSON.stringify({identity,manifestDigest:digest}),{flag:'wx',mode:0o600});
|
||||
if(await fs.realpath(path.dirname(root))!==path.dirname(root))throw Error('unsafe_root');
|
||||
stage='publish';if(await fs.realpath(path.dirname(root))!==path.dirname(root))throw Error('unsafe_root');
|
||||
if(prior){await allocation(root);if(await stat(parked))throw Error('occupied_parking');await fs.rename(root,parked);}
|
||||
try{await fs.rename(staged,root);}catch(e){if(prior&&!await stat(root))await fs.rename(parked,root);throw e;}
|
||||
if(prior){await allocation(parked);await fs.rm(parked,{recursive:true});}
|
||||
stage='cleanup_parked';if(prior){await allocation(parked);await fs.rm(parked,{recursive:true});}
|
||||
}finally{await fs.rm(staged,{recursive:true,force:true});}
|
||||
})().catch(()=>{process.stderr.write('remote_workspace_restore_failed');process.exitCode=1});`;
|
||||
})().catch(error=>{process.stderr.write(setupFailure('remote_workspace_restore_failed',stage,error));process.exitCode=1});`;
|
||||
await command(input.target, "restore", ["-e", restore, staged.helper, staged.archive, input.remoteCwd,
|
||||
input.manifestDigest, input.restorationIdentity, String(input.allowEmptyRoot === true), JSON.stringify(input.allocationFiles ?? [])]);
|
||||
// Publish the restored location before cleanup can fail, stall, or lose its
|
||||
|
||||
@@ -26,6 +26,8 @@ import { createSavedWorkspaceLoader, prepareRemoteWorkspace, readRemoteWorkspace
|
||||
import type { createGitRemoteAuthProvider } from "./git-credentials.js";
|
||||
import { reconcileSafeNativeReplacements } from "./native-runtime/native-safe-replacement.js";
|
||||
import { remoteTerminationReceipt } from "./remote-execution-termination.js";
|
||||
import { RemoteWorkspaceRestoreError } from "./remote-workspace-transport.js";
|
||||
import { notFound } from "../errors.js";
|
||||
import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
|
||||
import { createRemoteWorkspaceFiles } from "./remote-workspace-files.js";
|
||||
import { workspaceFileResourceService } from "./workspace-file-resources.js";
|
||||
@@ -391,7 +393,7 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem
|
||||
}
|
||||
}, 60_000);
|
||||
|
||||
it.each(["same root", "different root", "destroyed allocation", "occupied root"] as const)(
|
||||
it.each(["same root", "different root", "destroyed allocation", "occupied root", "missing object", "transient restore"] as const)(
|
||||
"preserves the published workspace path and files on a cold restore (%s)", async mode => {
|
||||
const reserved = await reserve({ kind: "empty" }), first = await target("sandbox-A");
|
||||
const prepared = await prepareRemoteWorkspace({ db, storage, workspace: reserved, runId: ids.run, lease: await lease(), target: first });
|
||||
@@ -424,13 +426,40 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem
|
||||
if (mode === "occupied root") {
|
||||
await fs.mkdir(expectedRemoteRoot, { recursive: true });
|
||||
await fs.writeFile(path.join(expectedRemoteRoot, "unpublished.txt"), "preserve unknown preparation\n");
|
||||
await expect(prepareRemoteWorkspace({ db, storage, workspace: reservedAgain, runId: nextRun, lease: newLease, target: second }))
|
||||
.rejects.toThrow("remote_workspace_operation_failed");
|
||||
const occupied = await prepareRemoteWorkspace({ db, storage, workspace: reservedAgain, runId: nextRun, lease: newLease, target: second })
|
||||
.then(() => null, (error: unknown) => error as RemoteWorkspaceRestoreError);
|
||||
// The restore names its step and the occupied folder; it is not retried.
|
||||
expect(occupied).toBeInstanceOf(RemoteWorkspaceRestoreError);
|
||||
expect(occupied).toMatchObject({ step: "sandbox_restore", retryable: false, message: "remote_workspace_restore_failed: step=sandbox_restore; "
|
||||
+ "remote_workspace_operation_failed: operation=restore exit=1; remote_workspace_restore_failed: stage=inspect_root reason=occupied_root" });
|
||||
expect(await fs.readFile(path.join(expectedRemoteRoot, "unpublished.txt"), "utf8")).toBe("preserve unknown preparation\n");
|
||||
expect(readRemoteWorkspaceAuthority((await workspace(reserved.id)).metadata))
|
||||
.toEqual(readRemoteWorkspaceAuthority(reservedAgain.metadata));
|
||||
await fs.rename(expectedRemoteRoot, path.join(root, "retained-unknown-preparation"));
|
||||
}
|
||||
if (mode === "transient restore") {
|
||||
// An unnamed sandbox failure (an I/O error) is retryable; the retry restores the saved files.
|
||||
const run = second.runner!.execute.bind(second.runner);
|
||||
vi.spyOn(second.runner!, "execute").mockImplementation(async command => command.args?.[1]?.includes("remote_workspace_restore_failed")
|
||||
? { exitCode: 1, stdout: "", stderr: "remote_workspace_restore_failed: stage=extract code=EIO exit=1", timedOut: false, signal: null,
|
||||
pid: process.pid, startedAt: new Date().toISOString() } : run(command));
|
||||
const transient = await prepareRemoteWorkspace({ db, storage, workspace: reservedAgain, runId: nextRun, lease: newLease, target: second })
|
||||
.then(() => null, (error: unknown) => error as RemoteWorkspaceRestoreError);
|
||||
expect(transient).toMatchObject({ step: "sandbox_restore", retryable: true, message: "remote_workspace_restore_failed: step=sandbox_restore; "
|
||||
+ "remote_workspace_operation_failed: operation=restore exit=1; remote_workspace_restore_failed: stage=extract code=EIO exit=1" });
|
||||
expect(await fs.lstat(expectedRemoteRoot).catch(() => null)).toBeNull();
|
||||
vi.mocked(second.runner!.execute).mockRestore();
|
||||
}
|
||||
if (mode === "missing object") {
|
||||
// Storage reports a lost saved object as 404; retrying cannot bring it back.
|
||||
const lost: StorageService = { ...storage, getObject: async () => { throw notFound("Object not found"); } };
|
||||
const missing = await prepareRemoteWorkspace({ db, storage: lost, workspace: reservedAgain, runId: nextRun, lease: newLease, target: second })
|
||||
.then(() => null, (error: unknown) => error as RemoteWorkspaceRestoreError);
|
||||
expect(missing).toBeInstanceOf(RemoteWorkspaceRestoreError);
|
||||
expect(missing).toMatchObject({ step: "snapshot_download", retryable: false,
|
||||
message: "remote_workspace_restore_failed: step=snapshot_download; remote_workspace_snapshot_object_missing" });
|
||||
expect(await fs.lstat(expectedRemoteRoot).catch(() => null)).toBeNull();
|
||||
}
|
||||
const restored = await prepareRemoteWorkspace({ db, storage, workspace: reservedAgain, runId: nextRun, lease: newLease, target: second });
|
||||
expect(restored.remoteRoot).toBe(expectedRemoteRoot);
|
||||
if (mode !== "different root") expect(restored.remoteRoot).toBe(prepared.remoteRoot);
|
||||
|
||||
@@ -28,7 +28,7 @@ import { persistActivity } from "./activity-log.js";
|
||||
import {
|
||||
captureRemoteWorkspaceArchive, cleanupRemoteWorkspaceArchive, prepareRemoteWorkspaceDirectory, prepareRemoteRepository, remoteWorkspaceRepositoryPath,
|
||||
REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE, RemoteWorkspaceOperationTimeoutError, RemoteWorkspacePreparationTimeoutError, remoteWorkspaceFailureCause,
|
||||
requireRemoteWorkspaceTarget, restoreRemoteWorkspaceDirectory,
|
||||
requireRemoteWorkspaceTarget, restoreRemoteWorkspaceDirectory, RemoteWorkspaceRestoreError, type RemoteWorkspaceRestoreStep,
|
||||
} from "./remote-workspace-transport.js";
|
||||
import {
|
||||
beginWorkspaceSnapshot, getWorkspaceSnapshot, loadWorkspaceSnapshotArchive, uploadWorkspaceSnapshot,
|
||||
@@ -401,16 +401,42 @@ export async function prepareRemoteWorkspace(input: {
|
||||
assertOwned: input.assertOwned ?? (async () => undefined) })) throw new Error("remote_workspace_export_cleanup_pending");
|
||||
if (!sameAllocation) {
|
||||
if (initial.snapshotId) {
|
||||
const snapshot = await getWorkspaceSnapshot(input.db, { companyId: input.workspace.companyId, workspaceId: input.workspace.id, snapshotId: initial.snapshotId });
|
||||
if (!snapshot) throw new Error("remote_workspace_snapshot_missing");
|
||||
const scratch = await mkdtemp(path.join(os.tmpdir(), "pc-workspace-restore-"));
|
||||
let step: RemoteWorkspaceRestoreStep = "snapshot_lookup";
|
||||
try {
|
||||
const archivePath = path.join(scratch, "workspace.pcws");
|
||||
await loadWorkspaceSnapshotArchive({ storage: input.storage ?? getStorageService(), reference: snapshot, targetPath: archivePath });
|
||||
restoredExport = await restoreRemoteWorkspaceDirectory({ target, remoteCwd: remoteRoot, archivePath, manifestDigest: snapshot.manifestDigest,
|
||||
restorationIdentity: preparationIdentity, allowEmptyRoot: migration !== null,
|
||||
allocationFiles: migration ? workspaceAllocationFiles(input.lease, remoteRoot) : [] });
|
||||
} finally { await rm(scratch, { recursive: true, force: true }); }
|
||||
const snapshot = await getWorkspaceSnapshot(input.db, { companyId: input.workspace.companyId, workspaceId: input.workspace.id, snapshotId: initial.snapshotId });
|
||||
if (!snapshot) throw new Error("remote_workspace_snapshot_missing");
|
||||
const scratch = await mkdtemp(path.join(os.tmpdir(), "pc-workspace-restore-"));
|
||||
try {
|
||||
const archivePath = path.join(scratch, "workspace.pcws");
|
||||
step = "snapshot_download";
|
||||
await loadWorkspaceSnapshotArchive({ storage: input.storage ?? getStorageService(), reference: snapshot, targetPath: archivePath });
|
||||
step = "sandbox_restore";
|
||||
restoredExport = await restoreRemoteWorkspaceDirectory({ target, remoteCwd: remoteRoot, archivePath, manifestDigest: snapshot.manifestDigest,
|
||||
restorationIdentity: preparationIdentity, allowEmptyRoot: migration !== null,
|
||||
allocationFiles: migration ? workspaceAllocationFiles(input.lease, remoteRoot) : [] });
|
||||
} finally { await rm(scratch, { recursive: true, force: true }); }
|
||||
} catch (error) {
|
||||
// Cancellation keeps its own handling, as do lookup refusals.
|
||||
if ((error instanceof Error && error.name === "AbortError") || (error instanceof HttpError && step === "snapshot_lookup")) throw error;
|
||||
// Storage reports a missing saved object as 404. Retrying cannot restore it.
|
||||
const objectMissing = error instanceof HttpError && step === "snapshot_download" && error.status === 404;
|
||||
if (objectMissing) throw new RemoteWorkspaceRestoreError(step, false, new Error("remote_workspace_snapshot_object_missing"));
|
||||
// A missing saved version cannot be fixed by retrying. Otherwise a retry
|
||||
// is offered only while this run still owns the same unchanged, clean
|
||||
// saved version; a failed restore never publishes over existing files.
|
||||
// A sandbox refusal that names its reason (an occupied folder, a changed
|
||||
// owner, a rejected archive) repeats on retry, so it goes to the board.
|
||||
let retryable = false;
|
||||
const refused = error instanceof Error && / (?:reason|archive)=[a-z_]+/.test(error.message);
|
||||
if (step !== "snapshot_lookup" && !refused) {
|
||||
const [latest] = await input.db.select({ metadata: executionWorkspaces.metadata }).from(executionWorkspaces)
|
||||
.where(and(eq(executionWorkspaces.companyId, input.workspace.companyId), eq(executionWorkspaces.id, input.workspace.id)));
|
||||
const current = readRemoteWorkspaceAuthority(latest?.metadata);
|
||||
retryable = Boolean(current && current.epoch === initial.epoch && current.writerRunId === input.runId
|
||||
&& !current.dirty && current.snapshotId === initial.snapshotId);
|
||||
}
|
||||
throw new RemoteWorkspaceRestoreError(step, retryable, error);
|
||||
}
|
||||
} else {
|
||||
try {
|
||||
await prepareRemoteWorkspaceDirectory({ target, remoteCwd: remoteRoot, source: initial.source,
|
||||
|
||||
Reference in new issue
Block a user