mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-18 14:35:55 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task that runs on a reusable cloud sandbox resumes the same sandbox on the next wake > - On staging, Daytona left two idle sandboxes in the `archiving` state for more than three hours, so no resume call could succeed > - The native runner leases had no verified harness backup, so the run never replaced the sandbox, even though the task files were saved > - Each retry spent six minutes on the same stuck sandbox, the bounded retries ran out, and the task waited for a person to press Retry > - This pull request continues on a new sandbox when the provider has proven it cannot start the old one and the workspace is saved, and it keeps the old sandbox > - The benefit is that the task recovers by itself and a sandbox that may hold unsaved runner state is never destroyed ## Linked Issues or Issue Description Refs #20626, #21113 **What happened?** Eight Codie runs on staging failed in one hour with `reusable_sandbox_resume_failed`: `Reusable sandbox lease "<id>" could not be resumed (cause=transport_timeout); the lease was preserved and no replacement was created.` They came in two clusters (05:59 to 06:00Z and 06:32 to 06:38Z). Each run took about six minutes, so the 330-second idle resume budget from #21113 was applied. All runs were already `transient_failure_retry` wakes. Investigation of one instance (lease `c23d6120…`, run `d16c272e…`): - The lease was `released` at 02:37Z with a confirmed stop-and-retain. No live run held it, so the lease was idle and the extended budget applied. - A read-only Daytona API read at 06:5xZ showed the sandbox in state `archiving` (desired state `archived`), last updated 03:38Z, one hour after the stop (the auto-archive interval is 60 minutes). The second sandbox (`e0e6cfe5…`) showed the same state since 03:35Z. Daytona does not start a sandbox while it archives, so every resume call timed out. More time or a new HTTP connection could not help. - The task workspace was clean and had a committed saved snapshot. - The agent uses `paperclip_runner`. The last run on the sandbox failed while it saved the runner session, so the lease had no `nativeHarnessBackup` stamp. `reusableSandboxReplacementIsSafe` requires a verified harness backup for a runner lease, so the run never replaced the sandbox. It failed strictly, the bounded retries (2) ran out, and the task waited for a manual Retry. **Expected behavior** When an idle sandbox never starts through the whole provider start deadline and the task files are saved, the run continues on a new sandbox automatically. A sandbox that may hold unsaved runner state is not destroyed. **Steps to reproduce** 1. Run a `paperclip_runner` task on a reusable Daytona sandbox. Let the last run end without a harness backup (for example, the runner exits while it saves the session). The workspace is clean with a committed snapshot. 2. Let Daytona start to archive the stopped sandbox and get stuck in `archiving`. 3. Wake the task. Every `environmentResumeLease` call times out, the run fails after about 330 seconds with `reusable_sandbox_resume_failed`, and the retries fail the same way. **Paperclip version or commit** Staging Core `0c04ff76eb` (current `master` has the same code path). **Deployment mode** Hosted staging (cloud), Daytona sandbox provider plugin, reusable leases on. **Agent adapter(s) involved** `paperclip_runner` (native runner) on a Daytona sandbox environment. **Root cause** `server/src/services/reusable-sandbox-resume-replacement.ts`: a runner lease without a verified harness backup was never replaceable, for any failure. The only other path was to retry the same sandbox, which cannot work while the provider is stuck. Replacement also always retired the old lease to `pending_cleanup`, so allowing it without a backup would have destroyed the only copy of the runner session state. ## What Changed - `reusable-sandbox-resume-replacement.ts`: `reusableSandboxReplacementIsSafe` becomes `reusableSandboxReplacementPlan`. It returns `retire` (same rules as before), `keep`, or `null`. It returns `keep` for a runner lease without a verified harness backup only when the caller passes `providerCannotStart`. All other conditions still apply: idle lease, same workspace, clean workspace, committed snapshot, this run is the writer, and no access refusal. - Add `keepReplacedReusableSandboxLease`. After the replacement lease is recorded, one guarded write marks the old lease with `reusableSandboxKeptAfterReplacement`. The lease keeps its status and its sandbox. Nothing destroys it. - `environment-runtime.ts`: set `providerCannotStart` only when the idle start deadline was extended and the run then used all of it on timeouts. Lease selection skips a lease with the keep marker, so later runs do not spend time on it and obsolete-lease cleanup does not destroy it. - The replacement records `replacement` / `resume_failed` with no harness backup. The native executor already starts a fresh session in a new sandbox in this case (`freshReplacement`), and workspace preparation restores the committed snapshot. - The keep plan needs every resume call to time out. A rate limit, a reset, or any other failure in between keeps the strict behavior. - `reusable-sandbox-kept-marker.ts`: the marker key and a SQL guard. Every automatic teardown path skips a marked lease: retained-orphan listing and settle, `destroyReusableSandboxLease` (obsolete cleanup, expiry, run destroy), `destroyClaimedSandboxLease` (issue and workspace teardown), recovered-workspace cleanup, and abandoned-file release. Only an operator-consented environment delete still destroys it. - `doc/execution-semantics.md`: document the rule. ## Verification - Red before the fix: the new staging-shape test in `server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts` (released runner lease, no harness backup, clean committed snapshot, every resume times out) failed with the staging error `could not be resumed (cause=transport_timeout); the lease was preserved and no replacement was created.` - Review round: "keeps an idle sandbox with no harness backup when some resume calls failed for another reason", "never destroys a sandbox kept after replacement through the run or workspace teardown paths", and the retained-orphan case "keeps a warm sandbox while a replacement kept it" each failed before the fix and pass after it. `environment-retained-orphan-sandbox.test.ts`, `abandoned-workspace-files-hold.test.ts`, `workspace-recovery-preparation-gates.test.ts`, `workspace-recovery-admission.test.ts`: all pass. - Green after the fix: 22/22 pass in the replacement suite. The run gets a new sandbox within 330 seconds, makes no destroy call, the old lease stays `released` with the keep marker, and the next run resumes the new sandbox only. With unsaved work or no saved snapshot, the run still fails strictly and does not mark the old lease. Earlier tests (a refusal without a backup, a live owner, a dirty workspace with a timeout) are unchanged. - `pnpm exec vitest run src/__tests__/environment-runtime.test.ts src/services/environment-runtime-reusable-lease-probe.test.ts src/__tests__/reusable-leases-default.test.ts`: 215 pass. - `tsc -p server/tsconfig.json --noEmit` passes. - Not run: a live replacement on staging. It is verified after deploy on the next wake of the affected tasks. ## Risks - A runner task that hits this path starts a fresh runner session. The files come from the saved snapshot, and the task, comments, and run history are kept. Before this change, the task stopped and waited for a person. - The kept sandbox stays with the provider (archived storage) until someone removes it. This is the cost of never destroying unsaved runner state. - The rule needs the full extended start deadline (about 330 seconds) of timeouts, so a slow but healthy start still resumes the same sandbox. - No schema change. Old leases follow the same rule. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
794 lines
58 KiB
TypeScript
794 lines
58 KiB
TypeScript
import { classifyWorkspaceRestoreFailure } from "@paperclipai/adapter-utils/workspace-restore-merge";
|
|
import { createHash } from "node:crypto";
|
|
import { adapterExecutionControls } from "./adapter-execution-control.js";
|
|
import { isAcknowledgedNativeStop } from "./acknowledged-native-stop.js";
|
|
import { notifyNativeResultCommit } from "./native-result-commit-notifications.js";
|
|
import { notifyNativeWorkspaceFinalization } from "./native-workspace-finalization-notifications.js";
|
|
import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js";
|
|
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
|
import { beginIdleTrackedWork } from "./task-admission.js";
|
|
import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
|
import { resumePreservedResultFinalization } from "./preserved-result-finalization.js";
|
|
import { reclaimInactiveNativeProviderCache } from "./native-runtime/retained-provider-cache.js";
|
|
import { REMOTE_WORKSPACE_SANDBOX_DISK_FULL, sandboxProviderFailureLabel } from "./remote-workspace-transport.js";
|
|
import { and, asc, eq, gt, inArray, ne, notInArray, or, sql } from "drizzle-orm";
|
|
import { agentWakeupRequests, environmentLeases, executionWorkspaces, heartbeatRuns, issues, issueRecoveryActions, nativeRunFinalizations, workspaceOperations, type Db } from "@paperclipai/db";
|
|
import { remoteWorkspaceRunReferenceSchema, type Environment, type EnvironmentLease } from "@paperclipai/shared";
|
|
import { conflict } from "../errors.js";
|
|
import { logger } from "../middleware/logger.js";
|
|
import type { EnvironmentRuntimeService } from "./environment-runtime.js";
|
|
import { resolveEnvironmentExecutionTarget } from "./environment-execution-target.js";
|
|
import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
|
|
import { confirmSandboxRecoveryRestart, hasStopOnlyCleanup, prepareHistoricalSandboxStopVerification, prepareSandboxStopAndRetain, prepareStoppedSandboxForRecovery, restoreSandboxStopAfterFailedRestart, settleAbsentSandbox, settleStopOnlyCleanup } from "./sandbox-stop-and-retain.js";
|
|
import { requestSandboxAbsenceProof } from "./sandbox-absence-proof.js";
|
|
import { persistActivity, publishActivity } from "./activity-log.js";
|
|
import { hasRequiredWorkspaceRecovery } from "./workspace-restore-recovery-state.js";
|
|
import { hasVerifiedWorkspaceRepair, readWorkspaceRecovery, type WorkspaceRecovery } from "./workspace-recovery-receipt.js";
|
|
import { withNativeWorkspaceFinalizationOwnership } from "./native-runtime/native-workspace-finalization-ownership.js";
|
|
import { readNativeWorkspaceSyncReference, readNativeWorkspaceSyncLocalCwd, resumeNativeWorkspaceSync } from "./native-runtime/native-workspace-sync.js";
|
|
import { classifyNativeWorkspaceFailure } from "./native-runtime/native-workspace-failure.js";
|
|
|
|
import { handOffExhaustedSystemRecovery, adoptSystemRecovery, loadSystemRecovery, updateSystemRecovery } from "./system-recovery.js";
|
|
import { SYSTEM_RECOVERY_MAX_RETRIES, createSystemRecovery, failSystemRecovery, readSystemRecovery, reserveSystemRecoveryAttempt, SystemRecoveryFailure, type SystemRecoveryState } from "./system-recovery-state.js";
|
|
import { inspectWorkspaceRecoverySource, workspaceRecoverySourceEligible, workspaceSourceBindingFailure } from "./workspace-recovery-source.js";
|
|
import { preserveWorkspaceSnapshot } from "./workspace-recovery-snapshot.js";
|
|
import { adoptHistoricalWorkspaceRecoveries } from "./workspace-recovery-backfill.js";
|
|
import { canAdoptNativeSourceRetention, retainNativeRecoverySource, settleRecoveredNativeSourceRetention } from "./native-runtime/native-source-retention.js";
|
|
import { saveRemoteWorkspace } from "./remote-workspaces.js";
|
|
import { continueFromLastSavedWorkspace, isUnverifiableSourceFailure, recordRetainedSourceFallbackDeferral, settleExhaustedRetainedHold } from "./workspace-recovery-fallback.js";
|
|
import { capturePolicyRearmDueCondition, retainedSourceFallbackDueCondition, stopProofRearmDueCondition } from "./workspace-recovery-fallback-discovery.js";
|
|
import { rearmExhaustedCapturePolicy } from "./workspace-recovery-capture-policy.js";
|
|
import { rearmExhaustedStopProof } from "./workspace-recovery-stop-proof.js";
|
|
import { settleRecoveredWorkspaceRestoreHolds } from "./workspace-restore-hold-settlement.js";
|
|
import { reusableSandboxNotKeptAfterReplacementSql } from "./reusable-sandbox-kept-marker.js";
|
|
|
|
const exportCauses = ["native_workspace_sync_out_unrecoverable", "native_workspace_sync_out_retry_exhausted"];
|
|
const terminal = ["failed", "interrupted", "timed_out", "cancelled", "succeeded"];
|
|
const sandboxDiskFull = /No space left on device|Disk quota exceeded/i;
|
|
export const SANDBOX_DISK_FULL_RECOVERY_CAUSE = { label: "sandbox_disk_full", code: "ENOSPC", status: null } as const;
|
|
const isSandboxDiskFullSave = (error: unknown) => error instanceof Error && error.message === REMOTE_WORKSPACE_SANDBOX_DISK_FULL;
|
|
type Source = { companyId: string; issueId: string; runId: string; actionId: string };
|
|
|
|
type StopEvidenceSource = Omit<Source, "actionId"> & { actionId?: string };
|
|
|
|
async function recheckStoppedWorkspaceRecovery(db: Db, source: StopEvidenceSource, observe: boolean) {
|
|
return db.transaction(async tx => {
|
|
const [issue] = await tx.select({ id: issues.id }).from(issues).where(and(
|
|
eq(issues.id, source.issueId), eq(issues.companyId, source.companyId))).for("update");
|
|
const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, source.runId),
|
|
eq(heartbeatRuns.companyId, source.companyId))).for("update");
|
|
if (!issue || !run || run.runtimeMode !== "native" || run.nativeIssueId !== issue.id ||
|
|
!isAcknowledgedNativeStop(run) || adapterExecutionControls.has(run.id)) return 0;
|
|
const cancellation = run.resultJson?.executionCancellation as { state?: unknown; proof?: unknown; acknowledgedAt?: unknown } | undefined;
|
|
if (cancellation?.state !== "acknowledged" || cancellation.proof !== "provider_termination_receipt") return 0;
|
|
const [coordinator] = await tx.select({ phase: nativeRunFinalizations.phase, leaseOwner: nativeRunFinalizations.leaseOwner })
|
|
.from(nativeRunFinalizations).where(and(eq(nativeRunFinalizations.companyId, source.companyId),
|
|
eq(nativeRunFinalizations.runId, source.runId), eq(nativeRunFinalizations.issueId, source.issueId)));
|
|
if (!coordinator || coordinator.leaseOwner || !["terminal_failure", "applied"].includes(coordinator.phase)) return 0;
|
|
const leases = await tx.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, source.companyId),
|
|
eq(environmentLeases.heartbeatRunId, source.runId)));
|
|
if (!leases.length || leases.some(lease => !lease.provider || lease.provider === "local" || !hasRemoteTerminationReceipt(lease))) return 0;
|
|
const intent = run.resultJson!.nativeCancellation as Record<string, unknown>;
|
|
const fingerprint = createHash("sha256").update(JSON.stringify({ companyId: source.companyId, issueId: source.issueId,
|
|
runId: source.runId, intentAuditId: intent.intentAuditId, acknowledgementAuditId: intent.acknowledgementAuditId,
|
|
acknowledgedAt: cancellation.acknowledgedAt, leases: leases.sort((a, b) => a.id.localeCompare(b.id)).map(lease => {
|
|
const receipt = lease.metadata!.remoteExecutionTermination as Record<string, unknown>;
|
|
return { leaseId: lease.id, provider: lease.provider, providerLeaseId: lease.providerLeaseId,
|
|
state: receipt.state, confirmedAt: receipt.confirmedAt };
|
|
}) })).digest("hex");
|
|
const actions = await tx.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, source.companyId),
|
|
eq(issueRecoveryActions.sourceIssueId, source.issueId), or(inArray(issueRecoveryActions.status, ["active", "escalated"]), and(
|
|
eq(issueRecoveryActions.status, "resolved"), sql`${issueRecoveryActions.evidence}#>>'{automaticRecovery,policy}' = 'preserve_without_replay_v1'`,
|
|
sql`${issueRecoveryActions.evidence}#>>'{automaticRecovery,replay}' = 'blocked'`,
|
|
sql`${issueRecoveryActions.evidence}#>>'{automaticRecovery,runId}' = ${source.runId}`)),
|
|
sql`${issueRecoveryActions.evidence}->>'runId' = ${source.runId}`,
|
|
source.actionId ? eq(issueRecoveryActions.id, source.actionId) : undefined)).for("update");
|
|
let changed = 0;
|
|
for (const action of actions) {
|
|
const current = readSystemRecovery(action.evidence), workspace = readWorkspaceRecovery(action.evidence), now = Date.now();
|
|
if (!current || current.sourceRunId !== source.runId || current.phase !== "verify_execution" ||
|
|
!workspace || workspace.runId !== source.runId || ["complete", "exhausted"].includes(current.state) || current.retriesUsed >= SYSTEM_RECOVERY_MAX_RETRIES ||
|
|
(current.deadlineAt !== null && now >= current.deadlineAt)) continue;
|
|
const stopWait = current.failure?.retryClass === "recheck_evidence" &&
|
|
["agent_stop_unconfirmed", "copyback_stop_unconfirmed"].includes(current.failure.code);
|
|
if (current.state !== "running" && current.failure && !stopWait) continue;
|
|
const prior = action.evidence.workspaceStopEvidence as Record<string, unknown> | undefined;
|
|
const same = prior?.schema === "paperclip.workspace-stop-evidence.v1" && prior.sourceRunId === source.runId &&
|
|
prior.generationId === current.generationId && prior.fingerprint === fingerprint;
|
|
if ((same && prior!.consumed === true) || (!observe && (!same || prior!.consumed !== false))) continue;
|
|
const due = Math.max(now, current.providerCooldownAt ?? 0);
|
|
const advance = current.state === "awaiting_evidence" && stopWait && current.attempts > 0 &&
|
|
current.deadlineAt !== null && due < current.deadlineAt && current.nextAttemptAt !== null && current.nextAttemptAt > due;
|
|
const consumed = current.state !== "running";
|
|
if (!advance && same && prior!.consumed === consumed) continue;
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.workspaceRecovery);
|
|
if (advance && !await updateSystemRecovery(tx as unknown as Db, { ...source, actionId: action.id }, current,
|
|
{ ...current, revision: current.revision + 1, nextAttemptAt: due })) continue;
|
|
const marker = { schema: "paperclip.workspace-stop-evidence.v1", sourceRunId: source.runId,
|
|
generationId: current.generationId, fingerprint, consumed };
|
|
const recorded = await tx.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence} ||
|
|
${JSON.stringify({ workspaceStopEvidence: marker })}::jsonb`, updatedAt: new Date() }).where(and(
|
|
eq(issueRecoveryActions.companyId, source.companyId), eq(issueRecoveryActions.id, action.id),
|
|
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,generationId}' = ${current.generationId}`,
|
|
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,revision}' = ${String(current.revision + (advance ? 1 : 0))}`))
|
|
.returning({ id: issueRecoveryActions.id });
|
|
if (!recorded.length) throw new Error("workspace_stop_evidence_changed");
|
|
changed += 1;
|
|
}
|
|
return changed;
|
|
});
|
|
}
|
|
|
|
export function observeStoppedWorkspaceRecovery(db: Db, source: StopEvidenceSource) {
|
|
return recheckStoppedWorkspaceRecovery(db, source, true);
|
|
}
|
|
|
|
/**
|
|
* Ownership-reconciler settle for the RC-50 copyback-owner hold. The recorded
|
|
* owner was proven gone by the shared prover, so the waiting recheck is due
|
|
* now instead of after its backoff. It grants no attempt and settles nothing
|
|
* itself: the worker still proves the owner and its copyback process group
|
|
* before it takes ownership. Returns whether the recheck was woken.
|
|
*/
|
|
export async function wakeCopybackOwnerRecheck(db: Db, input: {
|
|
companyId: string; actionId: string; now: Date;
|
|
owner: { pid: number | null; processStartedAt: string | null; bootId: string | null; processSpace: string | null };
|
|
}): Promise<boolean> {
|
|
return db.transaction(async tx => {
|
|
const [action] = await tx.select().from(issueRecoveryActions).where(and(
|
|
eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.id, input.actionId),
|
|
inArray(issueRecoveryActions.status, ["active", "escalated"]))).for("update");
|
|
const hold = action?.evidence?.workspaceOwnerRecovery as { state?: unknown; owner?: Record<string, unknown> } | undefined;
|
|
const owner = hold?.owner;
|
|
if (!action || hold?.state !== "operator_required" || !owner || owner.pid !== input.owner.pid
|
|
|| owner.processStartedAt !== input.owner.processStartedAt
|
|
|| (owner.controllerBootId ?? null) !== input.owner.bootId || (owner.hostHash ?? null) !== input.owner.processSpace) return false;
|
|
const current = readSystemRecovery(action.evidence);
|
|
const runId = action.evidence.runId;
|
|
const now = input.now.getTime();
|
|
if (!current || typeof runId !== "string" || current.sourceRunId !== runId || current.state !== "awaiting_evidence"
|
|
|| current.failure?.code !== "copyback_stop_unconfirmed"
|
|
|| (current.deadlineAt !== null && now >= current.deadlineAt)) return false;
|
|
const due = Math.max(now, current.providerCooldownAt ?? 0);
|
|
if (current.nextAttemptAt !== null && current.nextAttemptAt <= due) {
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.workspaceRecovery);
|
|
return true;
|
|
}
|
|
return updateSystemRecovery(tx as unknown as Db, { companyId: input.companyId, issueId: action.sourceIssueId,
|
|
runId, actionId: action.id }, current, { ...current, revision: current.revision + 1, nextAttemptAt: due });
|
|
});
|
|
}
|
|
|
|
/** Admission records intent only. No request handler runs physical recovery. */
|
|
export async function requestWorkspaceRecovery(input: Source & { db: Db; idempotencyKey: string; actorId: string }) {
|
|
return input.db.transaction(async tx => {
|
|
const [issue] = await tx.select().from(issues).where(and(eq(issues.id, input.issueId), eq(issues.companyId, input.companyId))).for("update");
|
|
const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId))).for("update");
|
|
const [action] = await tx.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.id, input.actionId),
|
|
eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, input.issueId))).for("update");
|
|
if (!issue || !run || !action || action.evidence.runId !== run.id
|
|
|| (run.nativeIssueId ?? run.issueId ?? run.contextSnapshot?.issueId) !== issue.id
|
|
|| !terminal.includes(run.status) || (!hasRequiredWorkspaceRecovery(run.resultJson) && !exportCauses.includes(action.cause))) {
|
|
throw conflict("The workspace recovery changed. Refresh the task before retrying.");
|
|
}
|
|
const prior = readWorkspaceRecovery(action.evidence);
|
|
if (prior?.requestIds.includes(input.idempotencyKey) || prior?.status === "recovered") return { actionId: action.id, status: prior.status };
|
|
const pending = prior && ["queued", "recovering"].includes(prior.status);
|
|
const state: WorkspaceRecovery = pending ? { ...prior!, requestIds: [...prior!.requestIds, input.idempotencyKey] }
|
|
: { schema: "paperclip.workspace-recovery.v1", runId: run.id, status: "queued", stage: "verifying_stop",
|
|
attempt: prior?.attempt ?? 0, attemptLimit: (prior?.attempt ?? 0) + 1,
|
|
requestIds: [...(prior?.requestIds ?? []), input.idempotencyKey] };
|
|
const previous = readSystemRecovery(action.evidence);
|
|
const active = previous && ["scheduled", "running", "awaiting_evidence"].includes(previous.state);
|
|
const system = previous?.state === "running" ? previous : active ? { ...previous, revision: previous.revision + 1,
|
|
nextAttemptAt: Math.max(Date.now(), previous.providerCooldownAt ?? 0) } : { ...createSystemRecovery(run.id),
|
|
revision: (previous?.revision ?? -1) + 1,
|
|
preservationOperationIds: previous?.preservationOperationIds ?? [],
|
|
providerCooldownAt: previous?.providerCooldownAt ?? null,
|
|
nextAttemptAt: previous?.providerCooldownAt ?? null };
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.workspaceRecovery);
|
|
await tx.update(issueRecoveryActions).set({ ownerType: "system", ownerAgentId: null, ownerUserId: null,
|
|
evidence: { ...action.evidence, workspaceRecovery: state, systemRecovery: system }, updatedAt: new Date() })
|
|
.where(eq(issueRecoveryActions.id, action.id));
|
|
await persistActivity(tx as unknown as Db, { companyId: input.companyId, actorType: "user", actorId: input.actorId,
|
|
action: "issue.workspace_recovery_requested", entityType: "issue", entityId: issue.id, runId: run.id,
|
|
details: { recoveryActionId: action.id, idempotencyKey: input.idempotencyKey,
|
|
...(previous && previous.generationId !== system.generationId ? { previousGeneration: previous } : {}) } });
|
|
return { actionId: action.id, status: state.status };
|
|
});
|
|
}
|
|
|
|
/** Physical repair remains serialized by the existing source ownership lock.
|
|
* Neither preservation nor cleanup creates an agent turn. */
|
|
export async function recoverRetainedWorkspace(input: Source & { db: Db; environmentRuntime: EnvironmentRuntimeService }) {
|
|
const { db } = input;
|
|
await adoptSystemRecovery(db, input);
|
|
let initial = await loadSystemRecovery(db, input);
|
|
// A hold whose files a sibling saved, or whose source is refused permanently
|
|
// (including holds saved by older releases), settles without a person.
|
|
// An adoptable native source keeps its retention path; only the sibling
|
|
// settlement applies to it.
|
|
if (initial?.state === "exhausted"
|
|
&& await settleExhaustedRetainedHold(input, initial, { fallback: !canAdoptNativeSourceRetention(initial) })) return;
|
|
// An exhausted wait for stop proof is re-armed, bounded, once the run is over.
|
|
if (initial?.state === "exhausted" && await rearmExhaustedStopProof(db, input, initial))
|
|
initial = await loadSystemRecovery(db, input);
|
|
// A save exhausted under an older capture policy is retried once under the current one.
|
|
if (initial?.state === "exhausted" && await rearmExhaustedCapturePolicy(db, input, initial))
|
|
initial = await loadSystemRecovery(db, input);
|
|
if (initial?.state === "exhausted" && !canAdoptNativeSourceRetention(initial)) {
|
|
// A budget spent before this hold existed still names its receipt, so the
|
|
// board confirmation in its handoff can release it (RC-50 wave databases).
|
|
if (initial.failure?.code === "copyback_stop_unconfirmed") await recordCopybackOwnerHold(db, input, initial.generationId);
|
|
await handOffExhaustedSystemRecovery(db, input, initial);
|
|
return;
|
|
}
|
|
if (!initial || initial.state === "complete" ||
|
|
(initial.nextAttemptAt !== null && initial.nextAttemptAt > Date.now())) return;
|
|
const runAttempt = async (ownership: { token: string; assertHeld(): Promise<void> }) => {
|
|
let current = await loadSystemRecovery(db, input);
|
|
if (!current) return;
|
|
if (current.state === "exhausted") {
|
|
const spent = current;
|
|
current = await retainNativeRecoverySource(db, input, current, ownership);
|
|
if (!current) {
|
|
// Not adoptable now. The board owns the spent budget until Retry or a
|
|
// later adoption returns it to the system.
|
|
await handOffExhaustedSystemRecovery(db, input, (await loadSystemRecovery(db, input)) ?? spent);
|
|
return;
|
|
}
|
|
}
|
|
const reserved = reserveSystemRecoveryAttempt(current);
|
|
if (!reserved) {
|
|
if (current.state !== "exhausted" && current.deadlineAt !== null && Date.now() >= current.deadlineAt)
|
|
await updateSystemRecovery(db, input, current, { ...current, revision: current.revision + 1,
|
|
state: "exhausted", nextAttemptAt: null });
|
|
return;
|
|
}
|
|
if (!await updateSystemRecovery(db, input, current, reserved)) return;
|
|
current = reserved;
|
|
const progress = async (patch: Partial<SystemRecoveryState>) => {
|
|
await ownership.assertHeld();
|
|
const next = { ...current!, ...patch, revision: current!.revision + 1 };
|
|
if (!await updateSystemRecovery(db, input, current!, next)) throw new Error("workspace_recovery_generation_changed");
|
|
current = next;
|
|
};
|
|
const observeActive = async () => {
|
|
await ownership.assertHeld();
|
|
await db.update(issueRecoveryActions).set({ evidence: sql`jsonb_set(${issueRecoveryActions.evidence},
|
|
'{systemRecoveryActiveAt}', ${JSON.stringify(Date.now())}::jsonb)` }).where(and(
|
|
eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.id, input.actionId),
|
|
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,attemptId}' = ${reserved.attemptId}`));
|
|
};
|
|
let observing = Promise.resolve();
|
|
let cacheReclaimUnconfirmed = false;
|
|
const observer = setInterval(() => { observing = observing.then(observeActive).catch(() => undefined); }, 15_000);
|
|
try {
|
|
await observeActive();
|
|
const inspected = await inspectWorkspaceRecoverySource(db, input);
|
|
let sourceUnavailable = false;
|
|
for (const source of inspected.sources) {
|
|
const { lease, environment, remoteCwd } = source;
|
|
if (hasRemoteTerminationReceipt(lease)
|
|
&& (lease.metadata?.remoteExecutionTermination as Record<string, unknown> | undefined)?.state === "destroyed") {
|
|
sourceUnavailable = true;
|
|
continue;
|
|
}
|
|
if (lease.provider === "daytona" && lease.providerLeaseId) {
|
|
const absence = await requestSandboxAbsenceProof({ companyId: input.companyId, issueId: input.issueId, runId: input.runId, leaseId: lease.id,
|
|
provider: "daytona", providerLeaseId: lease.providerLeaseId, ownerToken: ownership.token });
|
|
if (absence) {
|
|
await ownership.assertHeld();
|
|
if (!await settleAbsentSandbox(db, lease, absence)) throw new SystemRecoveryFailure("agent_stop_unconfirmed", "recheck_evidence");
|
|
sourceUnavailable = true;
|
|
continue;
|
|
}
|
|
}
|
|
let reference = source.preservationOnly ? null : source.reference;
|
|
if (reference) {
|
|
try { await readNativeWorkspaceSyncLocalCwd({ runId: input.runId, companyId: input.companyId,
|
|
workspaceId: reference.workspaceId, reference }); }
|
|
catch (error) {
|
|
// Missing local merge metadata permits a separate snapshot of the
|
|
// exact bound allocation. Corrupt metadata never weakens binding.
|
|
if ((error as NodeJS.ErrnoException)?.code === "ENOENT") reference = null;
|
|
else throw workspaceSourceBindingFailure("local_descriptor_invalid");
|
|
}
|
|
}
|
|
let stoppedLease = lease;
|
|
if (!hasRemoteTerminationReceipt(stoppedLease)) {
|
|
const pending = await db.transaction(async tx => {
|
|
const [fresh] = await tx.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, input.companyId),
|
|
eq(environmentLeases.id, lease.id))).for("update");
|
|
if (!fresh || fresh.metadata?.pendingCleanupInFlight === true) return null;
|
|
const prepared = hasStopOnlyCleanup(fresh) ? fresh : fresh.status === "released"
|
|
? await prepareHistoricalSandboxStopVerification(tx as unknown as Db, fresh)
|
|
: await prepareSandboxStopAndRetain(tx as unknown as Db, fresh);
|
|
if (!prepared) return null;
|
|
const [claimed] = await tx.update(environmentLeases).set({ metadata: sql`${environmentLeases.metadata} ||
|
|
${JSON.stringify({ pendingCleanupAttemptId: ownership.token, pendingCleanupInFlight: true,
|
|
pendingCleanupLeaseExpiresAtMs: Date.now() + 15 * 60_000 })}::jsonb` })
|
|
.where(eq(environmentLeases.id, lease.id)).returning();
|
|
return claimed;
|
|
});
|
|
if (!pending) throw new SystemRecoveryFailure("agent_stop_unconfirmed", "recheck_evidence");
|
|
// The cleanup service owns exact-allocation termination. No process ID
|
|
// or terminal run status is substituted for its provider receipt.
|
|
const settled = await input.environmentRuntime.retryPendingSandboxTeardown({
|
|
environment: environment as unknown as Environment, lease: pending as EnvironmentLease })
|
|
.then(receipt => settleStopOnlyCleanup(db, pending, { attemptId: ownership.token, receipt }),
|
|
async () => await restoreSandboxStopAfterFailedRestart(db, pending, { attemptId: ownership.token })
|
|
?? await settleStopOnlyCleanup(db, pending, { attemptId: ownership.token }));
|
|
if (!settled || !hasRemoteTerminationReceipt(settled))
|
|
throw new SystemRecoveryFailure("agent_stop_unconfirmed", "recheck_evidence");
|
|
// The provider confirmed the sandbox is gone: its files cannot be read.
|
|
if ((settled.metadata?.remoteExecutionTermination as Record<string, unknown> | undefined)?.state === "destroyed") {
|
|
sourceUnavailable = true;
|
|
continue;
|
|
}
|
|
stoppedLease = settled;
|
|
}
|
|
const target = await resolveEnvironmentExecutionTarget({ db, companyId: input.companyId, adapterType: "paperclip_runner",
|
|
environment, leaseId: lease.id, leaseMetadata: stoppedLease.metadata,
|
|
lease: stoppedLease as EnvironmentLease, environmentRuntime: input.environmentRuntime });
|
|
if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runner || target.remoteCwd !== remoteCwd)
|
|
throw workspaceSourceBindingFailure("execution_target_invalid");
|
|
await ownership.assertHeld();
|
|
let cleanupLease = await db.transaction(async tx => {
|
|
const [fresh] = await tx.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, input.companyId),
|
|
eq(environmentLeases.id, lease.id))).for("update");
|
|
if (!fresh || !hasRemoteTerminationReceipt(fresh) || fresh.providerLeaseId !== lease.providerLeaseId)
|
|
throw new SystemRecoveryFailure("agent_stop_unconfirmed", "recheck_evidence");
|
|
const pending = await prepareStoppedSandboxForRecovery(tx as unknown as Db, fresh);
|
|
if (!pending) throw workspaceSourceBindingFailure("recovery_stop_intent_invalid");
|
|
const [claimed] = await tx.update(environmentLeases).set({ metadata: sql`${environmentLeases.metadata} || ${JSON.stringify({
|
|
pendingCleanupAttemptId: ownership.token, pendingCleanupInFlight: true, pendingCleanupLeaseExpiresAtMs: Date.now() + 15 * 60_000,
|
|
})}::jsonb` }).where(eq(environmentLeases.id, fresh.id)).returning();
|
|
return claimed;
|
|
});
|
|
let renewal = Promise.resolve();
|
|
const keepalive = setInterval(() => { renewal = renewal.then(async () => {
|
|
await ownership.assertHeld();
|
|
await db.update(environmentLeases).set({ metadata: sql`jsonb_set(${environmentLeases.metadata}, '{pendingCleanupLeaseExpiresAtMs}', ${JSON.stringify(Date.now() + 15 * 60_000)}::jsonb)` })
|
|
.where(and(eq(environmentLeases.id, lease.id), sql`${environmentLeases.metadata}->>'pendingCleanupAttemptId' = ${ownership.token}`));
|
|
}).catch(() => undefined); }, 30_000);
|
|
let stopSettled = false;
|
|
try {
|
|
const opened = await input.environmentRuntime.resumeWorkspaceRecoveryLease({ source: input,
|
|
leaseId: cleanupLease.id, environmentId: environment.id, ownership }).catch(error => {
|
|
throw sandboxDiskFull.test((error as Error | undefined)?.message ?? "")
|
|
? new SystemRecoveryFailure("source_unavailable", "use_alternative_strategy") : error;
|
|
});
|
|
const confirmed = await confirmSandboxRecoveryRestart(db, cleanupLease, { attemptId: ownership.token });
|
|
if (!confirmed) throw new SystemRecoveryFailure("agent_stop_unconfirmed", "recheck_evidence");
|
|
cleanupLease = confirmed;
|
|
if (opened?.providerLeaseId !== lease.providerLeaseId) throw workspaceSourceBindingFailure("reopened_allocation_mismatch");
|
|
if (opened.metadata?.remoteCwd !== remoteCwd) throw workspaceSourceBindingFailure("reopened_cwd_mismatch");
|
|
await inspectWorkspaceRecoverySource(db, input);
|
|
await progress({ phase: "preserve_files" });
|
|
let operationId: string;
|
|
if (source.remoteReference && source.remoteWorkspace) {
|
|
const [workspace] = await db.select().from(executionWorkspaces).where(and(
|
|
eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, source.remoteWorkspace.id),
|
|
));
|
|
if (!workspace) throw workspaceSourceBindingFailure("workspace_reference_mismatch");
|
|
const save = () => saveRemoteWorkspace({ db, workspace, runId: input.runId,
|
|
lease: cleanupLease as EnvironmentLease, target, assertOwned: ownership.assertHeld });
|
|
let saved: Awaited<ReturnType<typeof save>>;
|
|
try { saved = await save(); }
|
|
catch (error) {
|
|
if (!isSandboxDiskFullSave(error)) throw error;
|
|
let reclaimed = false;
|
|
try {
|
|
const reclaim = await reclaimInactiveNativeProviderCache({ db, companyId: input.companyId,
|
|
issueId: input.issueId, runId: input.runId, workspaceId: workspace.id, target,
|
|
assertOwned: ownership.assertHeld });
|
|
reclaimed = reclaim.changed;
|
|
cacheReclaimUnconfirmed = !reclaim.confirmed;
|
|
} catch {
|
|
cacheReclaimUnconfirmed = true;
|
|
logger.warn({ runId: input.runId }, "Retained provider cache cleanup could not be confirmed");
|
|
}
|
|
if (!reclaimed) throw error;
|
|
await ownership.assertHeld();
|
|
saved = await save();
|
|
}
|
|
await ownership.assertHeld();
|
|
const [existing] = await db.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and(
|
|
eq(workspaceOperations.companyId, input.companyId), eq(workspaceOperations.heartbeatRunId, input.runId),
|
|
eq(workspaceOperations.phase, "workspace_recovery"), eq(workspaceOperations.status, "succeeded"),
|
|
sql`${workspaceOperations.metadata}->>'leaseId' = ${lease.id}`,
|
|
sql`${workspaceOperations.metadata}->>'snapshotId' = ${saved.snapshotId}`,
|
|
)).limit(1);
|
|
const operation = existing ?? (await db.insert(workspaceOperations).values({ companyId: input.companyId,
|
|
issueId: input.issueId, heartbeatRunId: input.runId, executionWorkspaceId: workspace.id,
|
|
phase: "workspace_recovery", status: "succeeded", finishedAt: new Date(), metadata: {
|
|
recoveryActionId: input.actionId, leaseId: lease.id, providerLeaseId: lease.providerLeaseId,
|
|
snapshotId: saved.snapshotId, generation: saved.generation, sha256: saved.sha256,
|
|
storage: "remote_workspace", epoch: source.remoteReference.epoch,
|
|
} }).returning({ id: workspaceOperations.id }))[0]!;
|
|
operationId = operation.id;
|
|
} else if (reference) {
|
|
const copied = await resumeNativeWorkspaceSync({ db, runId: input.runId, target, assertOwnership: ownership.assertHeld });
|
|
if (!copied) throw new SystemRecoveryFailure("snapshot_verification_failed", "service_required");
|
|
const after = await inspectWorkspaceRecoverySource(db, input);
|
|
const finalized = after.sources.find(item => item.lease.id === lease.id)?.reference;
|
|
if (finalized?.state !== "finalized" || !finalized.finalHostSha256)
|
|
throw new SystemRecoveryFailure("snapshot_verification_failed", "service_required");
|
|
await ownership.assertHeld();
|
|
const [existing] = await db.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and(
|
|
eq(workspaceOperations.companyId, input.companyId), eq(workspaceOperations.heartbeatRunId, input.runId),
|
|
eq(workspaceOperations.phase, "workspace_recovery"), eq(workspaceOperations.status, "succeeded"),
|
|
sql`${workspaceOperations.metadata}->>'leaseId' = ${lease.id}`)).limit(1);
|
|
const copy = existing ?? (await db.insert(workspaceOperations).values({ companyId: input.companyId, issueId: input.issueId,
|
|
heartbeatRunId: input.runId, executionWorkspaceId: lease.executionWorkspaceId, phase: "workspace_recovery",
|
|
status: "succeeded", finishedAt: new Date(), metadata: { recoveryActionId: input.actionId, leaseId: lease.id,
|
|
providerLeaseId: lease.providerLeaseId, descriptorSha256: finalized.descriptorSha256,
|
|
finalHostSha256: finalized.finalHostSha256 } }).returning({ id: workspaceOperations.id }))[0];
|
|
operationId = copy.id;
|
|
} else {
|
|
operationId = await preserveWorkspaceSnapshot({ ...input, leaseId: lease.id,
|
|
providerLeaseId: lease.providerLeaseId!, target, assertHeld: ownership.assertHeld });
|
|
}
|
|
await progress({ preservationOperationIds: [...new Set([...current!.preservationOperationIds, operationId])] });
|
|
clearInterval(keepalive); await renewal;
|
|
await ownership.assertHeld();
|
|
const receipt = await input.environmentRuntime.retryPendingSandboxTeardown({ environment: environment as unknown as Environment,
|
|
lease: cleanupLease as EnvironmentLease });
|
|
const settled = await settleStopOnlyCleanup(db, cleanupLease, { attemptId: ownership.token, receipt });
|
|
stopSettled = Boolean(settled && hasRemoteTerminationReceipt(settled));
|
|
if (!stopSettled) throw new SystemRecoveryFailure("agent_stop_unconfirmed", "recheck_evidence");
|
|
} finally {
|
|
clearInterval(keepalive); await renewal;
|
|
if (!stopSettled) {
|
|
// A lost resume/export reply retains the same stop-only obligation.
|
|
// The maintenance worker must never turn it into destructive cleanup.
|
|
try {
|
|
await ownership.assertHeld();
|
|
const settled = await input.environmentRuntime.retryPendingSandboxTeardown({ environment: environment as unknown as Environment,
|
|
lease: cleanupLease as EnvironmentLease }).then(receipt => settleStopOnlyCleanup(db, cleanupLease, { attemptId: ownership.token, receipt }), () => null);
|
|
if (!settled || !hasRemoteTerminationReceipt(settled))
|
|
await restoreSandboxStopAfterFailedRestart(db, cleanupLease, { attemptId: ownership.token });
|
|
} catch { /* The persisted stop-only obligation remains authoritative. */ }
|
|
}
|
|
}
|
|
}
|
|
await ownership.assertHeld();
|
|
if (sourceUnavailable) throw new SystemRecoveryFailure("source_unavailable", "use_alternative_strategy");
|
|
const next = { ...current!, revision: current!.revision + 1, phase: "reconcile_effects" as const,
|
|
state: "complete" as const, nextAttemptAt: null };
|
|
const activity = await db.transaction(async tx => {
|
|
await notifyNativeResultCommit(tx, input);
|
|
await notifyNativeWorkspaceFinalization(tx, input);
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionReplacement);
|
|
await notifyNativeStatusReassessment(tx, input);
|
|
await tx.insert(workspaceOperations).values({ companyId: input.companyId, issueId: input.issueId,
|
|
heartbeatRunId: input.runId, phase: "workspace_finalize", status: "succeeded", finishedAt: new Date(),
|
|
metadata: { owningService: "workspace_recovery", preservationOperationIds: next.preservationOperationIds } });
|
|
if (!await updateSystemRecovery(tx as unknown as Db, input, current!, next))
|
|
throw new Error("workspace_recovery_generation_changed");
|
|
if (inspected.action.evidence.nativeSourceRetention && !await settleRecoveredNativeSourceRetention(tx as unknown as Db, input, next.preservationOperationIds))
|
|
throw new SystemRecoveryFailure("effect_unresolved", "recheck_evidence");
|
|
// A settled restore hold returns its warm sandbox to reuse. A hold that
|
|
// fails any guard keeps its protection; the saved files still count.
|
|
await settleRecoveredWorkspaceRestoreHolds(tx as unknown as Db, input);
|
|
return persistActivity(tx as unknown as Db, { companyId: input.companyId, actorType: "system", actorId: "workspace-recovery",
|
|
action: "issue.workspace_recovered", entityType: "issue", entityId: input.issueId, runId: input.runId,
|
|
details: { recoveryActionId: input.actionId, summary: "Saved work recovered." } });
|
|
});
|
|
current = next;
|
|
try { publishActivity(activity.publication); } catch { logger.warn({ runId: input.runId }, "Recovery activity delivery deferred"); }
|
|
} catch (error) {
|
|
await ownership.assertHeld();
|
|
if (current!.state === "complete") {
|
|
logger.warn({ runId: input.runId }, "Preserved workspace activity delivery deferred");
|
|
return;
|
|
}
|
|
const classification = classifyNativeWorkspaceFailure(error);
|
|
const diskFull = !(error instanceof SystemRecoveryFailure) && !cacheReclaimUnconfirmed && isSandboxDiskFullSave(error);
|
|
const failure = error instanceof SystemRecoveryFailure ? error
|
|
: diskFull ? new SystemRecoveryFailure("export_interrupted", "user_action_required")
|
|
: classifyWorkspaceRestoreFailure(error) === "restore_invalid_metadata"
|
|
? new SystemRecoveryFailure("snapshot_verification_failed", "service_required")
|
|
: classification.code === "workspace_sync_out_unsafe_archive"
|
|
? new SystemRecoveryFailure("archive_unsafe", "service_required")
|
|
: classification.code === "workspace_sync_out_unrecoverable"
|
|
? new SystemRecoveryFailure("source_unavailable", "use_alternative_strategy")
|
|
: new SystemRecoveryFailure(current!.phase === "verify_execution" ? "provider_unavailable" : "export_interrupted", "retry_transient");
|
|
const retryAfterAt = (error as { retryAfterAt?: unknown })?.retryAfterAt;
|
|
// A generic transient code says only that the step can be retried. Keep
|
|
// why it failed in closed labels so a repeated deferral is diagnosable.
|
|
// A classified failure already names its subsystem.
|
|
const generic = !(error instanceof SystemRecoveryFailure) && failure.retryClass === "retry_transient"
|
|
&& (failure.code === "provider_unavailable" || failure.code === "export_interrupted");
|
|
const cause = diskFull ? { ...SANDBOX_DISK_FULL_RECOVERY_CAUSE } : generic ? sandboxProviderFailureLabel(error) : undefined;
|
|
const failed = failSystemRecovery(current!, failure, {
|
|
...(typeof retryAfterAt === "number" && Number.isFinite(retryAfterAt) ? { retryAfterAt } : {}),
|
|
...(cause ? { cause } : {}),
|
|
});
|
|
const recorded = await updateSystemRecovery(db, input, current!, failed);
|
|
// A permanent refusal of the recorded source repeats on every retry.
|
|
// Continue from the last save in this same pass, under this ownership.
|
|
if (recorded && failed.state === "exhausted" && isUnverifiableSourceFailure(failed.failure)) {
|
|
const fallback = await continueFromLastSavedWorkspace({ ...input, ownership }).catch(fallbackError => {
|
|
logger.warn({ err: fallbackError, runId: input.runId }, "Continuing from the last save was deferred; the hold remains");
|
|
return { kind: "ineligible" as const, reason: "fallback_failed" };
|
|
});
|
|
if (fallback.kind === "ineligible") await recordRetainedSourceFallbackDeferral(db, input, fallback.reason);
|
|
}
|
|
if (failure.retryClass === "recheck_evidence" && ["agent_stop_unconfirmed", "copyback_stop_unconfirmed"].includes(failure.code))
|
|
await recheckStoppedWorkspaceRecovery(db, input, false);
|
|
logger.warn({ err: error, runId: input.runId, failureCode: failure.code, failureCause: cause?.label, phase: current!.phase }, "Workspace preservation deferred; source retained");
|
|
} finally { clearInterval(observer); await observing; }
|
|
};
|
|
const owned = await withNativeWorkspaceFinalizationOwnership({ db, companyId: input.companyId, runId: input.runId,
|
|
retainedRepairActionId: input.actionId }, runAttempt);
|
|
if (!owned.acquired && owned.reason === "owner_unverified") {
|
|
await holdForUnverifiedCopybackOwner(db, input);
|
|
await recheckStoppedWorkspaceRecovery(db, input, false);
|
|
}
|
|
}
|
|
|
|
/**
|
|
* A copyback owner that a restart killed can never confirm its own stop, and
|
|
* Retry cannot release its receipt (RC-50). Rechecking that receipt is not a
|
|
* recovery attempt: the bounded recovery window still applies, but no attempt
|
|
* is spent. Record the exact receipt so the board can confirm the stop through
|
|
* `resumeNativeWorkspaceAfterOwnerStop`, and so Retry names this wait.
|
|
*/
|
|
async function holdForUnverifiedCopybackOwner(db: Db, input: Source) {
|
|
const current = await loadSystemRecovery(db, input);
|
|
if (!current || current.state === "complete" || current.state === "exhausted") return;
|
|
const next = failSystemRecovery(current, new SystemRecoveryFailure("copyback_stop_unconfirmed", "recheck_evidence"));
|
|
if (!await updateSystemRecovery(db, input, current, next)) return;
|
|
await recordCopybackOwnerHold(db, input, next.generationId);
|
|
}
|
|
|
|
/** Bookkeeping only: names the exact receipt for the board. Grants no attempt. */
|
|
async function recordCopybackOwnerHold(db: Db, input: Source, generationId: string) {
|
|
const [source] = await db.select({ owner: sql<unknown>`${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceFinalizationOwner'` })
|
|
.from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId)));
|
|
const owner = source?.owner;
|
|
if (!owner || typeof owner !== "object" || typeof (owner as { token?: unknown }).token !== "string") return;
|
|
const target = and(eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.id, input.actionId),
|
|
eq(issueRecoveryActions.sourceIssueId, input.issueId),
|
|
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,generationId}' = ${generationId}`,
|
|
sql`${issueRecoveryActions.evidence}#>'{workspaceOwnerRecovery,owner}' is distinct from ${JSON.stringify(owner)}::jsonb`);
|
|
await db.transaction(async tx => {
|
|
const [action] = await tx.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(target).for("update");
|
|
if (!action) return;
|
|
// Wake the ownership worker (on any server) before the hold it announces,
|
|
// so a quiet worker judges this owner and can wake the recheck.
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.ownershipInvariantReport,
|
|
JSON.stringify({ companyId: input.companyId, runId: input.runId, actionId: input.actionId }));
|
|
await tx.update(issueRecoveryActions).set({
|
|
// A confirmation names one receipt; a newer unverified owner needs its own.
|
|
evidence: sql`(${issueRecoveryActions.evidence} - 'workspaceOwnerStop') || jsonb_build_object('workspaceOwnerRecovery',
|
|
jsonb_build_object('state', 'operator_required', 'owner', ${JSON.stringify(owner)}::jsonb))`,
|
|
updatedAt: new Date(),
|
|
}).where(target);
|
|
});
|
|
}
|
|
|
|
const cursors = new WeakMap<Db, string>();
|
|
const workInFlight = new WeakMap<Db, Set<string>>();
|
|
const unfinishedCursors = new WeakMap<Db, string>();
|
|
|
|
/** Destruction is a later cleanup obligation, authorized only by a matching
|
|
* saved-file receipt and the original recorded disposition. */
|
|
export async function scheduleRecoveredWorkspaceCleanup(db: Db, run: typeof heartbeatRuns.$inferSelect) {
|
|
if (!await hasVerifiedWorkspaceRepair(db, run)) return;
|
|
const legacy = readNativeWorkspaceSyncReference(run.runnerProfileJson?.nativeWorkspaceSync);
|
|
const parsedRemote = remoteWorkspaceRunReferenceSchema.safeParse(run.runnerProfileJson?.remoteWorkspace);
|
|
const remote = parsedRemote.success && parsedRemote.data.state === "saved" ? parsedRemote.data : null;
|
|
const ref = legacy ?? remote;
|
|
const disposition = legacy?.resourceDisposition ?? (remote ? run.runnerProfileJson?.nativeResourceDisposition : undefined);
|
|
if (!ref || disposition !== "destroy") return;
|
|
const [lease] = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.id, ref.leaseId)));
|
|
if (!lease || !hasRemoteTerminationReceipt(lease)) return;
|
|
const [other] = await db.select({ id: environmentLeases.id }).from(environmentLeases).where(and(
|
|
ne(environmentLeases.id, lease.id), eq(environmentLeases.provider, lease.provider!), eq(environmentLeases.providerLeaseId, lease.providerLeaseId!),
|
|
)).limit(1);
|
|
if (other) return;
|
|
await writeEnvironmentCleanup(db, tx => tx.update(environmentLeases).set({ status: "pending_cleanup", cleanupStatus: "failed", leasePolicy: "ephemeral",
|
|
updatedAt: new Date(), metadata: sql`(${environmentLeases.metadata} - 'workspaceRestoreRecovery' - 'remoteExecutionTermination') ||
|
|
${JSON.stringify({ workspaceRecoveryCleanup: { runId: run.id,
|
|
// Keep the verified stop: successors wait on the stop, not on the destroy.
|
|
stopReceipt: lease.metadata?.remoteExecutionTermination, ...(legacy ? { descriptorSha256: legacy.descriptorSha256 }
|
|
: { snapshotId: remote!.snapshotId, epoch: remote!.epoch }) },
|
|
pendingCleanupInFlight: false, pendingCleanupRetryAfterMs: 0, pendingCleanupRetryAttempts: 0, pendingCleanupLeaseExpiresAtMs: 0 })}::jsonb`,
|
|
}).where(and(eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.id, ref.leaseId),
|
|
eq(environmentLeases.heartbeatRunId, run.id), eq(environmentLeases.providerLeaseId, ref.providerLeaseId),
|
|
eq(environmentLeases.status, "released"), eq(environmentLeases.cleanupStatus, "success"),
|
|
sql`${environmentLeases.metadata} = ${JSON.stringify(lease.metadata)}::jsonb`,
|
|
sql`${environmentLeases.metadata}->>'workspaceRecoveryCleanup' is null`, reusableSandboxNotKeptAfterReplacementSql())),
|
|
{ companyId: run.companyId, runId: run.id });
|
|
}
|
|
/** `dueOnly` narrows the obligations to exhausted holds whose re-arm or
|
|
* fallback is due now. Cleanup, interrupted preparation and saved input that
|
|
* waits stay with the obligation pass. */
|
|
export function retainedWorkspaceCandidates(db: Db, cursor?: string, unfinishedOnly = false, obligationsOnly = false, inFlightRunIds: string[] = [],
|
|
dueOnly = false) {
|
|
const cleanup = sql`(((${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceSync,resourceDisposition}' = 'destroy'
|
|
and ${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceSync,leaseId}' is not null)
|
|
or (${heartbeatRuns.runnerProfileJson}->>'nativeResourceDisposition' = 'destroy'
|
|
and ${heartbeatRuns.runnerProfileJson}#>>'{remoteWorkspace,schema}' = 'paperclip.remote-workspace-run.v1'
|
|
and ${heartbeatRuns.runnerProfileJson}#>>'{remoteWorkspace,state}' = 'saved')) and exists (
|
|
select 1 from ${environmentLeases} where ${environmentLeases.companyId} = ${heartbeatRuns.companyId}
|
|
and ${environmentLeases.heartbeatRunId} = ${heartbeatRuns.id}
|
|
and ${environmentLeases.id}::text = coalesce(${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceSync,leaseId}',
|
|
${heartbeatRuns.runnerProfileJson}#>>'{remoteWorkspace,leaseId}')
|
|
and ${environmentLeases.status} = 'released' and ${environmentLeases.cleanupStatus} = 'success'
|
|
and ${environmentLeases.metadata}->'workspaceRecoveryCleanup' is null))`;
|
|
const waitingInput = sql`exists (select 1 from ${agentWakeupRequests}
|
|
where ${agentWakeupRequests.companyId} = ${issueRecoveryActions.companyId}
|
|
and ${agentWakeupRequests.status} = 'deferred_issue_execution' and ${agentWakeupRequests.requestedByActorType} = 'user'
|
|
and ${agentWakeupRequests.payload}->>'issueId' = ${issueRecoveryActions.sourceIssueId}::text)`;
|
|
const interruptedPreparation = sql`(${heartbeatRuns.runtimeMode} = 'legacy'
|
|
and (${heartbeatRuns.status} = 'interrupted' or (${heartbeatRuns.status} = 'failed'
|
|
and ${heartbeatRuns.errorCode} = 'process_lost'
|
|
and ${heartbeatRuns.resultJson}#>>'{executionRecovery,kind}' = 'bootstrap'
|
|
and ${heartbeatRuns.resultJson}#>'{executionRecovery,providerWorkStarted}' = 'false'::jsonb))
|
|
and ${heartbeatRuns.runtimeModeResolvedAt} is null and ${heartbeatRuns.executionStage} = 'preparing'
|
|
and ${heartbeatRuns.runnerProfileJson}#>>'{adapterDispatch,adapterType}' = 'paperclip_runner'
|
|
and (${issueRecoveryActions.evidence}#>>'{automaticRecovery,replay}' = 'blocked'
|
|
or (${issueRecoveryActions.evidence}#>>'{automaticRecovery,replay}' = 'allowed'
|
|
and ${issueRecoveryActions.evidence}->>'bootstrapPreparationSettledAt' is not null
|
|
and coalesce(${issueRecoveryActions.evidence}->>'bootstrapPreparationContinuation', 'pending') in ('pending', 'deferred')))
|
|
and exists (select 1 from ${issues} where ${issues.companyId} = ${heartbeatRuns.companyId}
|
|
and ${issues.id} = ${issueRecoveryActions.sourceIssueId} and ${issues.assigneeAgentId} = ${heartbeatRuns.agentId}
|
|
and ${issues.status} in ('todo', 'in_progress', 'in_review', 'blocked')
|
|
and ${issues.executionRunId} is null and ${issues.checkoutRunId} is null
|
|
and (${issues.status} <> 'blocked' or
|
|
${issueRecoveryActions.evidence}#>>'{nativeBootstrapFailureBlock,statusVersion}' = ${issues.statusVersion}::text)))`;
|
|
const now = Date.now();
|
|
const dueHold = sql`(${retainedSourceFallbackDueCondition(now)} or ${stopProofRearmDueCondition(now)} or ${capturePolicyRearmDueCondition(now)})`;
|
|
return db.select({ action: issueRecoveryActions, run: heartbeatRuns }).from(issueRecoveryActions)
|
|
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.companyId, issueRecoveryActions.companyId), eq(heartbeatRuns.id,
|
|
sql`case when ${issueRecoveryActions.evidence}->>'runId' ~* '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'
|
|
then (${issueRecoveryActions.evidence}->>'runId')::uuid end`)))
|
|
.where(and(inArray(heartbeatRuns.status, terminal),
|
|
workspaceRecoverySourceEligible(true),
|
|
unfinishedOnly ? sql`(${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('scheduled', 'running', 'awaiting_evidence')
|
|
or (${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' = 'complete'
|
|
and ${issueRecoveryActions.evidence}->>'bootstrapPreparationSettledAt' is not null
|
|
and coalesce(${issueRecoveryActions.evidence}->>'bootstrapPreparationContinuation', 'pending') = 'pending'))` : undefined,
|
|
obligationsOnly ? sql`(${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('scheduled', 'running', 'awaiting_evidence')
|
|
or (${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' = 'complete' and (${cleanup} or ${interruptedPreparation})) or ${waitingInput}
|
|
or ${dueHold})` : undefined,
|
|
dueOnly ? dueHold : undefined,
|
|
sql`(${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('scheduled', 'running', 'awaiting_evidence', 'complete', 'exhausted')
|
|
and (${issueRecoveryActions.evidence}#>'{systemRecovery,nextAttemptAt}' = 'null'::jsonb
|
|
or ${issueRecoveryActions.evidence}#>'{systemRecovery,nextAttemptAt}' <= to_jsonb(${now}::bigint)))`,
|
|
sql`(${issueRecoveryActions.evidence}#>>'{workspaceRecovery,status}' is distinct from 'recovered' or
|
|
${cleanup} or ${waitingInput} or ${interruptedPreparation})`,
|
|
cursor ? gt(issueRecoveryActions.id, cursor) : undefined,
|
|
inFlightRunIds.length ? notInArray(heartbeatRuns.id, inFlightRunIds) : undefined,
|
|
)).orderBy(asc(issueRecoveryActions.id)).limit(5);
|
|
}
|
|
|
|
/**
|
|
* The worker's lane positions, kept in memory: a restart starts every lane
|
|
* from its first candidate, so no persisted state or migration is involved.
|
|
* `cursor`, `complete` and `restart` belong to the obligation pass; `dueCursor`
|
|
* to the due lane; `firstLane` names the lane that has priority in the next run.
|
|
*/
|
|
export type PassiveSweep = { cursor?: string; complete?: boolean; restart?: boolean; dueCursor?: string; firstLane?: number };
|
|
|
|
type CandidateRow = Awaited<ReturnType<typeof retainedWorkspaceCandidates>>[number];
|
|
type CandidateLane = { rows: CandidateRow[]; index: number; advance: (actionId: string) => void };
|
|
|
|
/** Bounded discovery shares the periodic recovery lifecycle. A repaired source
|
|
* still owes cleanup even without saved input or after task reconciliation.
|
|
*
|
|
* Candidates come from lanes, each an ordered page that a cursor advances:
|
|
* unfinished saves; for the worker, exhausted holds whose re-arm or fallback
|
|
* is due; and the obligation pass (cleanup, interrupted preparation, saved
|
|
* input that waits, and the same due holds). Each lane with work takes one
|
|
* free slot before any lane takes a second, and the worker rotates which lane
|
|
* goes first, so no kind of load (a save another server owns, a long local
|
|
* export, hundreds of waiting holds) starves another. */
|
|
export async function reconcileRetainedWorkspaces(db: Db, runtime: EnvironmentRuntimeService,
|
|
onRecovered: (run: typeof heartbeatRuns.$inferSelect) => Promise<void>, obligationsOnly = false,
|
|
onFinished?: (continueSweep: boolean) => void, passiveSweep?: PassiveSweep) {
|
|
await adoptHistoricalWorkspaceRecoveries(db);
|
|
let pending = workInFlight.get(db);
|
|
if (!pending) { pending = new Set(); workInFlight.set(db, pending); }
|
|
if (pending.size >= 2) return;
|
|
// Locally running exports already own their slot. Exclude them before
|
|
// pagination so a page containing only held work cannot prevent wrapping.
|
|
const inFlightRunIds = [...pending];
|
|
let unfinishedRows = await retainedWorkspaceCandidates(db, unfinishedCursors.get(db), true, false, inFlightRunIds);
|
|
if (!unfinishedRows.length) unfinishedRows = await retainedWorkspaceCandidates(db, undefined, true, false, inFlightRunIds);
|
|
const unfinished: CandidateLane = { rows: unfinishedRows, index: 0, advance: id => unfinishedCursors.set(db, id) };
|
|
let rows = await retainedWorkspaceCandidates(db, passiveSweep ? passiveSweep.cursor : cursors.get(db), false, obligationsOnly, inFlightRunIds);
|
|
if (passiveSweep && !rows.length && passiveSweep.restart) {
|
|
Object.assign(passiveSweep, { cursor: undefined, restart: false });
|
|
rows = await retainedWorkspaceCandidates(db, undefined, false, obligationsOnly, inFlightRunIds);
|
|
}
|
|
if (passiveSweep) passiveSweep.complete = !rows.length;
|
|
else if (!rows.length) rows = await retainedWorkspaceCandidates(db, undefined, false, obligationsOnly, inFlightRunIds);
|
|
const sweep: CandidateLane = { rows, index: 0,
|
|
advance: id => { if (passiveSweep) passiveSweep.cursor = id; else cursors.set(db, id); } };
|
|
const lanes = [unfinished];
|
|
if (passiveSweep) {
|
|
// The due lane wraps at its tail like the unfinished lane. A hold that
|
|
// becomes due by time alone (a lease expiry, a re-arm backoff) is reached
|
|
// even after the obligation pass completed with no committed change.
|
|
let dueRows = await retainedWorkspaceCandidates(db, passiveSweep.dueCursor, false, true, inFlightRunIds, true);
|
|
if (!dueRows.length && passiveSweep.dueCursor)
|
|
dueRows = await retainedWorkspaceCandidates(db, undefined, false, true, inFlightRunIds, true);
|
|
const due: CandidateLane = { rows: dueRows, index: 0, advance: id => { passiveSweep.dueCursor = id; } };
|
|
lanes.push(due, sweep);
|
|
// Priority rotates across all three lanes, so each lane is first in one
|
|
// run of three. With one free slot (a long local export holds the other),
|
|
// a save that keeps finding its run's lock busy cannot take it every run.
|
|
const first = (passiveSweep.firstLane ?? 0) % lanes.length;
|
|
lanes.push(...lanes.splice(0, first));
|
|
passiveSweep.firstLane = (first + 1) % lanes.length;
|
|
} else lanes.push(sweep);
|
|
const work: Promise<void>[] = [];
|
|
// Several actions of one run (an export hold and a later no-replay hold)
|
|
// run one after another in that run's job. Skipping the later one would
|
|
// starve it while the earlier one stays selectable (for example while saved
|
|
// input waits for the later one to settle).
|
|
const groups = new Map<string, { run: typeof heartbeatRuns.$inferSelect; actions: (typeof issueRecoveryActions.$inferSelect)[] }>();
|
|
const join = (lane: CandidateLane) => {
|
|
for (; lane.index < lane.rows.length; lane.index += 1) {
|
|
const { action, run } = lane.rows[lane.index]!;
|
|
const group = groups.get(run.id);
|
|
if (!group) return lane.rows[lane.index]!;
|
|
if (!group.actions.some(item => item.id === action.id)) group.actions.push(action);
|
|
lane.advance(action.id);
|
|
}
|
|
return undefined;
|
|
};
|
|
// Takes the lane's next run into a free slot. Actions of a run already taken
|
|
// (by any lane) join its job without a slot.
|
|
const take = (lane: CandidateLane) => {
|
|
const next = join(lane);
|
|
if (!next || pending.size >= 2) return false;
|
|
lane.index += 1;
|
|
lane.advance(next.action.id);
|
|
pending.add(next.run.id);
|
|
groups.set(next.run.id, { run: next.run, actions: [next.action] });
|
|
return true;
|
|
};
|
|
for (let taken = true; taken && pending.size < 2;) {
|
|
taken = false;
|
|
for (const lane of lanes) if (pending.size < 2 && take(lane)) taken = true;
|
|
}
|
|
for (const lane of lanes) join(lane);
|
|
const hasUnvisited = lanes.some(lane => lane.index < lane.rows.length);
|
|
for (const { run, actions } of groups.values()) {
|
|
const finish = beginIdleTrackedWork();
|
|
const physical = actions.some(action => ["scheduled", "running", "awaiting_evidence"].includes(readSystemRecovery(action.evidence)?.state ?? ""));
|
|
work.push((async () => { try {
|
|
for (const action of actions) await recoverRunAction(action, run);
|
|
} finally { pending.delete(run.id); finish(); onFinished?.(physical || hasUnvisited); } })());
|
|
}
|
|
return work;
|
|
|
|
async function recoverRunAction(action: typeof issueRecoveryActions.$inferSelect, run: typeof heartbeatRuns.$inferSelect) {
|
|
try {
|
|
await recoverRetainedWorkspace({ db, environmentRuntime: runtime, companyId: action.companyId, issueId: action.sourceIssueId, runId: run.id, actionId: action.id });
|
|
const [updated] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action.id));
|
|
const recovered = readWorkspaceRecovery(updated?.evidence ?? null)?.status === "recovered";
|
|
const exhausted = readSystemRecovery(updated?.evidence)?.state === "exhausted";
|
|
if (recovered || exhausted) {
|
|
const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
|
|
if (current) {
|
|
if (recovered) {
|
|
await resumePreservedResultFinalization(db, current.companyId, current.id);
|
|
await scheduleRecoveredWorkspaceCleanup(db, current);
|
|
}
|
|
// Only normal dispatch can consume existing authorized input. An
|
|
// exhausted export permits evaluation of an isolated continuation,
|
|
// never another export attempt or a replay of an old assignment.
|
|
await onRecovered(current);
|
|
}
|
|
}
|
|
} catch { logger.warn({ runId: run.id }, "Retained workspace remains pending"); }
|
|
}
|
|
}
|