Files
PaperClipAI/server/src/services/workspace-recovery.ts
T
DottaandClaude Opus 5.5 e9d7895648 fix(recovery): start an overdue saved-work recovery instead of looping on no pending path (#21478)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - When a run stops before its sandbox files are saved, Paperclip holds
the task and recovers the files first. A recovery queue does this work.
> - The queue runs two attempts at a time for the whole instance. A
remote attempt can take many minutes.
> - On staging, 29 recoveries waited with 0 attempts. The oldest waited
since 15:01Z. One task waited 3 h 33 min for its single attempt.
> - During that time the task said "Recovering saved work…", but the
assigned-work invariant saw no pending path. The sweep sent a
continuation wake every few minutes. The hold refused each wake, and the
activity log showed "could not start yet".
> - This pull request makes the queued recovery the task's pending path
for a bound. After the bound, the sweep starts the recovery itself
instead of sending a wake that the hold refuses.
> - The benefit: a held task gets its files recovered and continues
without a person, and the sweep stops its refused-wake loop.

## Linked Issues or Issue Description

**What happened?**
A scheduled task ran a legacy retry on a Daytona sandbox. Setup failed
with `setup_failed` ("remote_workspace_operation_timed_out: capture
(configured timeout 300000ms)"). Paperclip held the task for saved-work
recovery (`legacy_execution_requires_reconciliation`, resolved, replay
blocked, system recovery `scheduled`, 0 attempts). After 45 minutes the
recovery had not started. The task showed "Recovering saved work…" with
no Retry. The assigned-work sweep logged `issue.continued_automatically`
with outcome `not_started` and violation `no_pending_path` at 17:58,
18:02, 18:10 and 18:25 UTC.

**Expected behavior**
The recovery proceeds by itself and the task continues. If the recovery
cannot finish, the task shows a Retry that works. "Recovering…" never
shows with nothing behind it.

**Steps to reproduce**
1. A legacy run with a retained remote workspace fails during setup, and
its saved-work hold is adopted by system recovery.
2. The recovery queue does not reach the hold (it has two slots, and
they are busy).
3. The assigned-work sweep checks the task 10 or more minutes later. It
finds `no_pending_path` and sends a wake. The hold refuses the wake.
This repeats until the continuation cap.

The regression test
`server/src/services/saved-work-recovery-overdue.test.ts` reproduces
this state on real rows.

**Paperclip version or commit**
`master` at `c6301c92ff` (staging, 2026-11-13).

**Agent adapter(s) involved**
Not adapter-specific (core bug). Seen with a Daytona sandbox on the
legacy runtime.

## What Changed

- `assigned-work-invariant.ts`: new pending intent
`saved_work_recovery`. A saved-work recovery that the queue still owes
counts as the task's pending path while an attempt runs, while its next
attempt is scheduled, or for `SAVED_WORK_RECOVERY_START_BOUND_MS` (10
minutes) after it became due.
- `assigned-work-invariant.ts`: new violation
`saved_work_recovery_overdue`, used when that bound passes. It comes
before the `no_pending_path`, `retry_offered` and blocked checks, so the
sweep never sends a wake that the hold refuses.
- `assigned-work-sweep.ts`: the sweep loads the fact with the same
eligibility predicate the recovery worker uses. For the overdue
violation it calls `startSavedWorkRecovery` and records no continuation
activity.
- `workspace-recovery.ts`: `startOverdueWorkspaceRecovery` starts one
attempt outside the queue's two slots, with at most two such attempts
per process. It skips runs that are already in flight and uses the same
ownership lock and attempt bookkeeping as the queue. The queue skips
runs that this path started. `recoverRunAction` moved to module scope.
Its "Retained workspace remains pending" warning now logs the error.
- `heartbeat.ts` and `recovery/service.ts`: wire the dependency.
- The sweep returns the end of the bound as the task's re-check
deadline, so the overdue start does not depend on another notification.

## Verification

- New `server/src/services/saved-work-recovery-overdue.test.ts` (real
Postgres, recovery fixture with a mocked sandbox):
- Overdue hold: the sweep starts the recovery and does not call
`enqueueWakeup`. No continuation activity is written. The system
recovery reaches `complete`. The next sweep records a fresh
continuation, and `deliverReconciledExecutions` starts a successor run.
- Hold within the bound: verdict `ok / pending_intent /
saved_work_recovery`. No wake is sent and no attempt starts.
- A second start for the same run returns `in_flight`. Exactly one
attempt is recorded, and a completed hold returns `not_eligible`.
- Red on the old invariant and sweep, with `no_pending_path` and
`enqueue_refused`, the same as staging. Green with this change.
- `server/src/services/recovery/assigned-work-invariant.test.ts`: 54
pass, with new cases for the bound, a running or scheduled attempt, a
stale attempt, and a pending question.
- `server/src/__tests__/assigned-work-invariant.test.ts`: 31 pass.
- `server/src/services/workspace-recovery-passive-fairness.test.ts`:
passes.
- `tsc --noEmit -p server/tsconfig.json`: clean.

## Risks

- A sweep-started attempt can run beside the queue's two attempts. That
adds up to two more concurrent recovery attempts per process, each with
its own database connection while it holds the ownership lock. Each
attempt for a run still takes that run's advisory lock, so two servers
cannot run the same recovery at once.
- The 10-minute bound is new. A queue that is slow but working now gets
help after 10 minutes. It does not get the hours it took on staging.
- Out of scope, reported for follow-up: why the queue was so slow on
staging (two slots for the whole instance, attempts that take minutes),
and why the setup capture hit its 300 s timeout. Server logs were not
available from this session.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, browser reads of the staging API, local test
runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:26:26 -06:00

832 lines
60 KiB
TypeScript

import { classifyWorkspaceRestoreFailure } from "@paperclipai/adapter-utils/workspace-restore-merge";
import { createHash } from "node:crypto";
import { adapterExecutionControls } from "./adapter-execution-control.js";
import { isAcknowledgedNativeStop } from "./acknowledged-native-stop.js";
import { notifyNativeResultCommit } from "./native-result-commit-notifications.js";
import { notifyNativeWorkspaceFinalization } from "./native-workspace-finalization-notifications.js";
import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
import { beginIdleTrackedWork } from "./task-admission.js";
import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
import { resumePreservedResultFinalization } from "./preserved-result-finalization.js";
import { reclaimInactiveNativeProviderCache } from "./native-runtime/retained-provider-cache.js";
import { REMOTE_WORKSPACE_SANDBOX_DISK_FULL, sandboxProviderFailureLabel } from "./remote-workspace-transport.js";
import { and, asc, eq, gt, inArray, ne, notInArray, or, sql } from "drizzle-orm";
import { agentWakeupRequests, environmentLeases, executionWorkspaces, heartbeatRuns, issues, issueRecoveryActions, nativeRunFinalizations, workspaceOperations, type Db } from "@paperclipai/db";
import { remoteWorkspaceRunReferenceSchema, type Environment, type EnvironmentLease } from "@paperclipai/shared";
import { conflict } from "../errors.js";
import { logger } from "../middleware/logger.js";
import type { EnvironmentRuntimeService } from "./environment-runtime.js";
import { resolveEnvironmentExecutionTarget } from "./environment-execution-target.js";
import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
import { confirmSandboxRecoveryRestart, hasStopOnlyCleanup, prepareHistoricalSandboxStopVerification, prepareSandboxStopAndRetain, prepareStoppedSandboxForRecovery, restoreSandboxStopAfterFailedRestart, settleAbsentSandbox, settleStopOnlyCleanup } from "./sandbox-stop-and-retain.js";
import { requestSandboxAbsenceProof } from "./sandbox-absence-proof.js";
import { persistActivity, publishActivity } from "./activity-log.js";
import { hasRequiredWorkspaceRecovery } from "./workspace-restore-recovery-state.js";
import { hasVerifiedWorkspaceRepair, readWorkspaceRecovery, type WorkspaceRecovery } from "./workspace-recovery-receipt.js";
import { withNativeWorkspaceFinalizationOwnership } from "./native-runtime/native-workspace-finalization-ownership.js";
import { readNativeWorkspaceSyncReference, readNativeWorkspaceSyncLocalCwd, resumeNativeWorkspaceSync } from "./native-runtime/native-workspace-sync.js";
import { classifyNativeWorkspaceFailure } from "./native-runtime/native-workspace-failure.js";
import { handOffExhaustedSystemRecovery, adoptSystemRecovery, loadSystemRecovery, updateSystemRecovery } from "./system-recovery.js";
import { SYSTEM_RECOVERY_MAX_RETRIES, createSystemRecovery, failSystemRecovery, readSystemRecovery, reserveSystemRecoveryAttempt, SystemRecoveryFailure, type SystemRecoveryState } from "./system-recovery-state.js";
import { inspectWorkspaceRecoverySource, workspaceRecoverySourceEligible, workspaceSourceBindingFailure } from "./workspace-recovery-source.js";
import { preserveWorkspaceSnapshot } from "./workspace-recovery-snapshot.js";
import { adoptHistoricalWorkspaceRecoveries } from "./workspace-recovery-backfill.js";
import { canAdoptNativeSourceRetention, retainNativeRecoverySource, settleRecoveredNativeSourceRetention } from "./native-runtime/native-source-retention.js";
import { saveRemoteWorkspace } from "./remote-workspaces.js";
import { continueFromLastSavedWorkspace, isUnverifiableSourceFailure, recordRetainedSourceFallbackDeferral, settleExhaustedRetainedHold } from "./workspace-recovery-fallback.js";
import { capturePolicyRearmDueCondition, retainedSourceFallbackDueCondition, stopProofRearmDueCondition } from "./workspace-recovery-fallback-discovery.js";
import { rearmExhaustedCapturePolicy } from "./workspace-recovery-capture-policy.js";
import { rearmExhaustedStopProof } from "./workspace-recovery-stop-proof.js";
import { settleRecoveredWorkspaceRestoreHolds } from "./workspace-restore-hold-settlement.js";
import { reusableSandboxNotKeptAfterReplacementSql } from "./reusable-sandbox-kept-marker.js";
const exportCauses = ["native_workspace_sync_out_unrecoverable", "native_workspace_sync_out_retry_exhausted"];
const terminal = ["failed", "interrupted", "timed_out", "cancelled", "succeeded"];
const sandboxDiskFull = /No space left on device|Disk quota exceeded/i;
export const SANDBOX_DISK_FULL_RECOVERY_CAUSE = { label: "sandbox_disk_full", code: "ENOSPC", status: null } as const;
const isSandboxDiskFullSave = (error: unknown) => error instanceof Error && error.message === REMOTE_WORKSPACE_SANDBOX_DISK_FULL;
type Source = { companyId: string; issueId: string; runId: string; actionId: string };
type StopEvidenceSource = Omit<Source, "actionId"> & { actionId?: string };
async function recheckStoppedWorkspaceRecovery(db: Db, source: StopEvidenceSource, observe: boolean) {
return db.transaction(async tx => {
const [issue] = await tx.select({ id: issues.id }).from(issues).where(and(
eq(issues.id, source.issueId), eq(issues.companyId, source.companyId))).for("update");
const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, source.runId),
eq(heartbeatRuns.companyId, source.companyId))).for("update");
if (!issue || !run || run.runtimeMode !== "native" || run.nativeIssueId !== issue.id ||
!isAcknowledgedNativeStop(run) || adapterExecutionControls.has(run.id)) return 0;
const cancellation = run.resultJson?.executionCancellation as { state?: unknown; proof?: unknown; acknowledgedAt?: unknown } | undefined;
if (cancellation?.state !== "acknowledged" || cancellation.proof !== "provider_termination_receipt") return 0;
const [coordinator] = await tx.select({ phase: nativeRunFinalizations.phase, leaseOwner: nativeRunFinalizations.leaseOwner })
.from(nativeRunFinalizations).where(and(eq(nativeRunFinalizations.companyId, source.companyId),
eq(nativeRunFinalizations.runId, source.runId), eq(nativeRunFinalizations.issueId, source.issueId)));
if (!coordinator || coordinator.leaseOwner || !["terminal_failure", "applied"].includes(coordinator.phase)) return 0;
const leases = await tx.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, source.companyId),
eq(environmentLeases.heartbeatRunId, source.runId)));
if (!leases.length || leases.some(lease => !lease.provider || lease.provider === "local" || !hasRemoteTerminationReceipt(lease))) return 0;
const intent = run.resultJson!.nativeCancellation as Record<string, unknown>;
const fingerprint = createHash("sha256").update(JSON.stringify({ companyId: source.companyId, issueId: source.issueId,
runId: source.runId, intentAuditId: intent.intentAuditId, acknowledgementAuditId: intent.acknowledgementAuditId,
acknowledgedAt: cancellation.acknowledgedAt, leases: leases.sort((a, b) => a.id.localeCompare(b.id)).map(lease => {
const receipt = lease.metadata!.remoteExecutionTermination as Record<string, unknown>;
return { leaseId: lease.id, provider: lease.provider, providerLeaseId: lease.providerLeaseId,
state: receipt.state, confirmedAt: receipt.confirmedAt };
}) })).digest("hex");
const actions = await tx.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, source.companyId),
eq(issueRecoveryActions.sourceIssueId, source.issueId), or(inArray(issueRecoveryActions.status, ["active", "escalated"]), and(
eq(issueRecoveryActions.status, "resolved"), sql`${issueRecoveryActions.evidence}#>>'{automaticRecovery,policy}' = 'preserve_without_replay_v1'`,
sql`${issueRecoveryActions.evidence}#>>'{automaticRecovery,replay}' = 'blocked'`,
sql`${issueRecoveryActions.evidence}#>>'{automaticRecovery,runId}' = ${source.runId}`)),
sql`${issueRecoveryActions.evidence}->>'runId' = ${source.runId}`,
source.actionId ? eq(issueRecoveryActions.id, source.actionId) : undefined)).for("update");
let changed = 0;
for (const action of actions) {
const current = readSystemRecovery(action.evidence), workspace = readWorkspaceRecovery(action.evidence), now = Date.now();
if (!current || current.sourceRunId !== source.runId || current.phase !== "verify_execution" ||
!workspace || workspace.runId !== source.runId || ["complete", "exhausted"].includes(current.state) || current.retriesUsed >= SYSTEM_RECOVERY_MAX_RETRIES ||
(current.deadlineAt !== null && now >= current.deadlineAt)) continue;
const stopWait = current.failure?.retryClass === "recheck_evidence" &&
["agent_stop_unconfirmed", "copyback_stop_unconfirmed"].includes(current.failure.code);
if (current.state !== "running" && current.failure && !stopWait) continue;
const prior = action.evidence.workspaceStopEvidence as Record<string, unknown> | undefined;
const same = prior?.schema === "paperclip.workspace-stop-evidence.v1" && prior.sourceRunId === source.runId &&
prior.generationId === current.generationId && prior.fingerprint === fingerprint;
if ((same && prior!.consumed === true) || (!observe && (!same || prior!.consumed !== false))) continue;
const due = Math.max(now, current.providerCooldownAt ?? 0);
const advance = current.state === "awaiting_evidence" && stopWait && current.attempts > 0 &&
current.deadlineAt !== null && due < current.deadlineAt && current.nextAttemptAt !== null && current.nextAttemptAt > due;
const consumed = current.state !== "running";
if (!advance && same && prior!.consumed === consumed) continue;
await notifyDeliveryWork(tx, DELIVERY_QUEUES.workspaceRecovery);
if (advance && !await updateSystemRecovery(tx as unknown as Db, { ...source, actionId: action.id }, current,
{ ...current, revision: current.revision + 1, nextAttemptAt: due })) continue;
const marker = { schema: "paperclip.workspace-stop-evidence.v1", sourceRunId: source.runId,
generationId: current.generationId, fingerprint, consumed };
const recorded = await tx.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence} ||
${JSON.stringify({ workspaceStopEvidence: marker })}::jsonb`, updatedAt: new Date() }).where(and(
eq(issueRecoveryActions.companyId, source.companyId), eq(issueRecoveryActions.id, action.id),
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,generationId}' = ${current.generationId}`,
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,revision}' = ${String(current.revision + (advance ? 1 : 0))}`))
.returning({ id: issueRecoveryActions.id });
if (!recorded.length) throw new Error("workspace_stop_evidence_changed");
changed += 1;
}
return changed;
});
}
export function observeStoppedWorkspaceRecovery(db: Db, source: StopEvidenceSource) {
return recheckStoppedWorkspaceRecovery(db, source, true);
}
/**
* Ownership-reconciler settle for the RC-50 copyback-owner hold. The recorded
* owner was proven gone by the shared prover, so the waiting recheck is due
* now instead of after its backoff. It grants no attempt and settles nothing
* itself: the worker still proves the owner and its copyback process group
* before it takes ownership. Returns whether the recheck was woken.
*/
export async function wakeCopybackOwnerRecheck(db: Db, input: {
companyId: string; actionId: string; now: Date;
owner: { pid: number | null; processStartedAt: string | null; bootId: string | null; processSpace: string | null };
}): Promise<boolean> {
return db.transaction(async tx => {
const [action] = await tx.select().from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.id, input.actionId),
inArray(issueRecoveryActions.status, ["active", "escalated"]))).for("update");
const hold = action?.evidence?.workspaceOwnerRecovery as { state?: unknown; owner?: Record<string, unknown> } | undefined;
const owner = hold?.owner;
if (!action || hold?.state !== "operator_required" || !owner || owner.pid !== input.owner.pid
|| owner.processStartedAt !== input.owner.processStartedAt
|| (owner.controllerBootId ?? null) !== input.owner.bootId || (owner.hostHash ?? null) !== input.owner.processSpace) return false;
const current = readSystemRecovery(action.evidence);
const runId = action.evidence.runId;
const now = input.now.getTime();
if (!current || typeof runId !== "string" || current.sourceRunId !== runId || current.state !== "awaiting_evidence"
|| current.failure?.code !== "copyback_stop_unconfirmed"
|| (current.deadlineAt !== null && now >= current.deadlineAt)) return false;
const due = Math.max(now, current.providerCooldownAt ?? 0);
if (current.nextAttemptAt !== null && current.nextAttemptAt <= due) {
await notifyDeliveryWork(tx, DELIVERY_QUEUES.workspaceRecovery);
return true;
}
return updateSystemRecovery(tx as unknown as Db, { companyId: input.companyId, issueId: action.sourceIssueId,
runId, actionId: action.id }, current, { ...current, revision: current.revision + 1, nextAttemptAt: due });
});
}
/** Admission records intent only. No request handler runs physical recovery. */
export async function requestWorkspaceRecovery(input: Source & { db: Db; idempotencyKey: string; actorId: string }) {
return input.db.transaction(async tx => {
const [issue] = await tx.select().from(issues).where(and(eq(issues.id, input.issueId), eq(issues.companyId, input.companyId))).for("update");
const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId))).for("update");
const [action] = await tx.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.id, input.actionId),
eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, input.issueId))).for("update");
if (!issue || !run || !action || action.evidence.runId !== run.id
|| (run.nativeIssueId ?? run.issueId ?? run.contextSnapshot?.issueId) !== issue.id
|| !terminal.includes(run.status) || (!hasRequiredWorkspaceRecovery(run.resultJson) && !exportCauses.includes(action.cause))) {
throw conflict("The workspace recovery changed. Refresh the task before retrying.");
}
const prior = readWorkspaceRecovery(action.evidence);
if (prior?.requestIds.includes(input.idempotencyKey) || prior?.status === "recovered") return { actionId: action.id, status: prior.status };
const pending = prior && ["queued", "recovering"].includes(prior.status);
const state: WorkspaceRecovery = pending ? { ...prior!, requestIds: [...prior!.requestIds, input.idempotencyKey] }
: { schema: "paperclip.workspace-recovery.v1", runId: run.id, status: "queued", stage: "verifying_stop",
attempt: prior?.attempt ?? 0, attemptLimit: (prior?.attempt ?? 0) + 1,
requestIds: [...(prior?.requestIds ?? []), input.idempotencyKey] };
const previous = readSystemRecovery(action.evidence);
const active = previous && ["scheduled", "running", "awaiting_evidence"].includes(previous.state);
const system = previous?.state === "running" ? previous : active ? { ...previous, revision: previous.revision + 1,
nextAttemptAt: Math.max(Date.now(), previous.providerCooldownAt ?? 0) } : { ...createSystemRecovery(run.id),
revision: (previous?.revision ?? -1) + 1,
preservationOperationIds: previous?.preservationOperationIds ?? [],
providerCooldownAt: previous?.providerCooldownAt ?? null,
nextAttemptAt: previous?.providerCooldownAt ?? null };
await notifyDeliveryWork(tx, DELIVERY_QUEUES.workspaceRecovery);
await tx.update(issueRecoveryActions).set({ ownerType: "system", ownerAgentId: null, ownerUserId: null,
evidence: { ...action.evidence, workspaceRecovery: state, systemRecovery: system }, updatedAt: new Date() })
.where(eq(issueRecoveryActions.id, action.id));
await persistActivity(tx as unknown as Db, { companyId: input.companyId, actorType: "user", actorId: input.actorId,
action: "issue.workspace_recovery_requested", entityType: "issue", entityId: issue.id, runId: run.id,
details: { recoveryActionId: action.id, idempotencyKey: input.idempotencyKey,
...(previous && previous.generationId !== system.generationId ? { previousGeneration: previous } : {}) } });
return { actionId: action.id, status: state.status };
});
}
/** Physical repair remains serialized by the existing source ownership lock.
* Neither preservation nor cleanup creates an agent turn. */
export async function recoverRetainedWorkspace(input: Source & { db: Db; environmentRuntime: EnvironmentRuntimeService }) {
const { db } = input;
await adoptSystemRecovery(db, input);
let initial = await loadSystemRecovery(db, input);
// A hold whose files a sibling saved, or whose source is refused permanently
// (including holds saved by older releases), settles without a person.
// An adoptable native source keeps its retention path; only the sibling
// settlement applies to it.
if (initial?.state === "exhausted"
&& await settleExhaustedRetainedHold(input, initial, { fallback: !canAdoptNativeSourceRetention(initial) })) return;
// An exhausted wait for stop proof is re-armed, bounded, once the run is over.
if (initial?.state === "exhausted" && await rearmExhaustedStopProof(db, input, initial))
initial = await loadSystemRecovery(db, input);
// A save exhausted under an older capture policy is retried once under the current one.
if (initial?.state === "exhausted" && await rearmExhaustedCapturePolicy(db, input, initial))
initial = await loadSystemRecovery(db, input);
if (initial?.state === "exhausted" && !canAdoptNativeSourceRetention(initial)) {
// A budget spent before this hold existed still names its receipt, so the
// board confirmation in its handoff can release it (RC-50 wave databases).
if (initial.failure?.code === "copyback_stop_unconfirmed") await recordCopybackOwnerHold(db, input, initial.generationId);
await handOffExhaustedSystemRecovery(db, input, initial);
return;
}
if (!initial || initial.state === "complete" ||
(initial.nextAttemptAt !== null && initial.nextAttemptAt > Date.now())) return;
const runAttempt = async (ownership: { token: string; assertHeld(): Promise<void> }) => {
let current = await loadSystemRecovery(db, input);
if (!current) return;
if (current.state === "exhausted") {
const spent = current;
current = await retainNativeRecoverySource(db, input, current, ownership);
if (!current) {
// Not adoptable now. The board owns the spent budget until Retry or a
// later adoption returns it to the system.
await handOffExhaustedSystemRecovery(db, input, (await loadSystemRecovery(db, input)) ?? spent);
return;
}
}
const reserved = reserveSystemRecoveryAttempt(current);
if (!reserved) {
if (current.state !== "exhausted" && current.deadlineAt !== null && Date.now() >= current.deadlineAt)
await updateSystemRecovery(db, input, current, { ...current, revision: current.revision + 1,
state: "exhausted", nextAttemptAt: null });
return;
}
if (!await updateSystemRecovery(db, input, current, reserved)) return;
current = reserved;
const progress = async (patch: Partial<SystemRecoveryState>) => {
await ownership.assertHeld();
const next = { ...current!, ...patch, revision: current!.revision + 1 };
if (!await updateSystemRecovery(db, input, current!, next)) throw new Error("workspace_recovery_generation_changed");
current = next;
};
const observeActive = async () => {
await ownership.assertHeld();
await db.update(issueRecoveryActions).set({ evidence: sql`jsonb_set(${issueRecoveryActions.evidence},
'{systemRecoveryActiveAt}', ${JSON.stringify(Date.now())}::jsonb)` }).where(and(
eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.id, input.actionId),
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,attemptId}' = ${reserved.attemptId}`));
};
let observing = Promise.resolve();
let cacheReclaimUnconfirmed = false;
const observer = setInterval(() => { observing = observing.then(observeActive).catch(() => undefined); }, 15_000);
try {
await observeActive();
const inspected = await inspectWorkspaceRecoverySource(db, input);
let sourceUnavailable = false;
for (const source of inspected.sources) {
const { lease, environment, remoteCwd } = source;
if (hasRemoteTerminationReceipt(lease)
&& (lease.metadata?.remoteExecutionTermination as Record<string, unknown> | undefined)?.state === "destroyed") {
sourceUnavailable = true;
continue;
}
if (lease.provider === "daytona" && lease.providerLeaseId) {
const absence = await requestSandboxAbsenceProof({ companyId: input.companyId, issueId: input.issueId, runId: input.runId, leaseId: lease.id,
provider: "daytona", providerLeaseId: lease.providerLeaseId, ownerToken: ownership.token });
if (absence) {
await ownership.assertHeld();
if (!await settleAbsentSandbox(db, lease, absence)) throw new SystemRecoveryFailure("agent_stop_unconfirmed", "recheck_evidence");
sourceUnavailable = true;
continue;
}
}
let reference = source.preservationOnly ? null : source.reference;
if (reference) {
try { await readNativeWorkspaceSyncLocalCwd({ runId: input.runId, companyId: input.companyId,
workspaceId: reference.workspaceId, reference }); }
catch (error) {
// Missing local merge metadata permits a separate snapshot of the
// exact bound allocation. Corrupt metadata never weakens binding.
if ((error as NodeJS.ErrnoException)?.code === "ENOENT") reference = null;
else throw workspaceSourceBindingFailure("local_descriptor_invalid");
}
}
let stoppedLease = lease;
if (!hasRemoteTerminationReceipt(stoppedLease)) {
const pending = await db.transaction(async tx => {
const [fresh] = await tx.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, input.companyId),
eq(environmentLeases.id, lease.id))).for("update");
if (!fresh || fresh.metadata?.pendingCleanupInFlight === true) return null;
const prepared = hasStopOnlyCleanup(fresh) ? fresh : fresh.status === "released"
? await prepareHistoricalSandboxStopVerification(tx as unknown as Db, fresh)
: await prepareSandboxStopAndRetain(tx as unknown as Db, fresh);
if (!prepared) return null;
const [claimed] = await tx.update(environmentLeases).set({ metadata: sql`${environmentLeases.metadata} ||
${JSON.stringify({ pendingCleanupAttemptId: ownership.token, pendingCleanupInFlight: true,
pendingCleanupLeaseExpiresAtMs: Date.now() + 15 * 60_000 })}::jsonb` })
.where(eq(environmentLeases.id, lease.id)).returning();
return claimed;
});
if (!pending) throw new SystemRecoveryFailure("agent_stop_unconfirmed", "recheck_evidence");
// The cleanup service owns exact-allocation termination. No process ID
// or terminal run status is substituted for its provider receipt.
const settled = await input.environmentRuntime.retryPendingSandboxTeardown({
environment: environment as unknown as Environment, lease: pending as EnvironmentLease })
.then(receipt => settleStopOnlyCleanup(db, pending, { attemptId: ownership.token, receipt }),
async () => await restoreSandboxStopAfterFailedRestart(db, pending, { attemptId: ownership.token })
?? await settleStopOnlyCleanup(db, pending, { attemptId: ownership.token }));
if (!settled || !hasRemoteTerminationReceipt(settled))
throw new SystemRecoveryFailure("agent_stop_unconfirmed", "recheck_evidence");
// The provider confirmed the sandbox is gone: its files cannot be read.
if ((settled.metadata?.remoteExecutionTermination as Record<string, unknown> | undefined)?.state === "destroyed") {
sourceUnavailable = true;
continue;
}
stoppedLease = settled;
}
const target = await resolveEnvironmentExecutionTarget({ db, companyId: input.companyId, adapterType: "paperclip_runner",
environment, leaseId: lease.id, leaseMetadata: stoppedLease.metadata,
lease: stoppedLease as EnvironmentLease, environmentRuntime: input.environmentRuntime });
if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runner || target.remoteCwd !== remoteCwd)
throw workspaceSourceBindingFailure("execution_target_invalid");
await ownership.assertHeld();
let cleanupLease = await db.transaction(async tx => {
const [fresh] = await tx.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, input.companyId),
eq(environmentLeases.id, lease.id))).for("update");
if (!fresh || !hasRemoteTerminationReceipt(fresh) || fresh.providerLeaseId !== lease.providerLeaseId)
throw new SystemRecoveryFailure("agent_stop_unconfirmed", "recheck_evidence");
const pending = await prepareStoppedSandboxForRecovery(tx as unknown as Db, fresh);
if (!pending) throw workspaceSourceBindingFailure("recovery_stop_intent_invalid");
const [claimed] = await tx.update(environmentLeases).set({ metadata: sql`${environmentLeases.metadata} || ${JSON.stringify({
pendingCleanupAttemptId: ownership.token, pendingCleanupInFlight: true, pendingCleanupLeaseExpiresAtMs: Date.now() + 15 * 60_000,
})}::jsonb` }).where(eq(environmentLeases.id, fresh.id)).returning();
return claimed;
});
let renewal = Promise.resolve();
const keepalive = setInterval(() => { renewal = renewal.then(async () => {
await ownership.assertHeld();
await db.update(environmentLeases).set({ metadata: sql`jsonb_set(${environmentLeases.metadata}, '{pendingCleanupLeaseExpiresAtMs}', ${JSON.stringify(Date.now() + 15 * 60_000)}::jsonb)` })
.where(and(eq(environmentLeases.id, lease.id), sql`${environmentLeases.metadata}->>'pendingCleanupAttemptId' = ${ownership.token}`));
}).catch(() => undefined); }, 30_000);
let stopSettled = false;
try {
const opened = await input.environmentRuntime.resumeWorkspaceRecoveryLease({ source: input,
leaseId: cleanupLease.id, environmentId: environment.id, ownership }).catch(error => {
throw sandboxDiskFull.test((error as Error | undefined)?.message ?? "")
? new SystemRecoveryFailure("source_unavailable", "use_alternative_strategy") : error;
});
const confirmed = await confirmSandboxRecoveryRestart(db, cleanupLease, { attemptId: ownership.token });
if (!confirmed) throw new SystemRecoveryFailure("agent_stop_unconfirmed", "recheck_evidence");
cleanupLease = confirmed;
if (opened?.providerLeaseId !== lease.providerLeaseId) throw workspaceSourceBindingFailure("reopened_allocation_mismatch");
if (opened.metadata?.remoteCwd !== remoteCwd) throw workspaceSourceBindingFailure("reopened_cwd_mismatch");
await inspectWorkspaceRecoverySource(db, input);
await progress({ phase: "preserve_files" });
let operationId: string;
if (source.remoteReference && source.remoteWorkspace) {
const [workspace] = await db.select().from(executionWorkspaces).where(and(
eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, source.remoteWorkspace.id),
));
if (!workspace) throw workspaceSourceBindingFailure("workspace_reference_mismatch");
const save = () => saveRemoteWorkspace({ db, workspace, runId: input.runId,
lease: cleanupLease as EnvironmentLease, target, assertOwned: ownership.assertHeld });
let saved: Awaited<ReturnType<typeof save>>;
try { saved = await save(); }
catch (error) {
if (!isSandboxDiskFullSave(error)) throw error;
let reclaimed = false;
try {
const reclaim = await reclaimInactiveNativeProviderCache({ db, companyId: input.companyId,
issueId: input.issueId, runId: input.runId, workspaceId: workspace.id, target,
assertOwned: ownership.assertHeld });
reclaimed = reclaim.changed;
cacheReclaimUnconfirmed = !reclaim.confirmed;
} catch {
cacheReclaimUnconfirmed = true;
logger.warn({ runId: input.runId }, "Retained provider cache cleanup could not be confirmed");
}
if (!reclaimed) throw error;
await ownership.assertHeld();
saved = await save();
}
await ownership.assertHeld();
const [existing] = await db.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and(
eq(workspaceOperations.companyId, input.companyId), eq(workspaceOperations.heartbeatRunId, input.runId),
eq(workspaceOperations.phase, "workspace_recovery"), eq(workspaceOperations.status, "succeeded"),
sql`${workspaceOperations.metadata}->>'leaseId' = ${lease.id}`,
sql`${workspaceOperations.metadata}->>'snapshotId' = ${saved.snapshotId}`,
)).limit(1);
const operation = existing ?? (await db.insert(workspaceOperations).values({ companyId: input.companyId,
issueId: input.issueId, heartbeatRunId: input.runId, executionWorkspaceId: workspace.id,
phase: "workspace_recovery", status: "succeeded", finishedAt: new Date(), metadata: {
recoveryActionId: input.actionId, leaseId: lease.id, providerLeaseId: lease.providerLeaseId,
snapshotId: saved.snapshotId, generation: saved.generation, sha256: saved.sha256,
storage: "remote_workspace", epoch: source.remoteReference.epoch,
} }).returning({ id: workspaceOperations.id }))[0]!;
operationId = operation.id;
} else if (reference) {
const copied = await resumeNativeWorkspaceSync({ db, runId: input.runId, target, assertOwnership: ownership.assertHeld });
if (!copied) throw new SystemRecoveryFailure("snapshot_verification_failed", "service_required");
const after = await inspectWorkspaceRecoverySource(db, input);
const finalized = after.sources.find(item => item.lease.id === lease.id)?.reference;
if (finalized?.state !== "finalized" || !finalized.finalHostSha256)
throw new SystemRecoveryFailure("snapshot_verification_failed", "service_required");
await ownership.assertHeld();
const [existing] = await db.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and(
eq(workspaceOperations.companyId, input.companyId), eq(workspaceOperations.heartbeatRunId, input.runId),
eq(workspaceOperations.phase, "workspace_recovery"), eq(workspaceOperations.status, "succeeded"),
sql`${workspaceOperations.metadata}->>'leaseId' = ${lease.id}`)).limit(1);
const copy = existing ?? (await db.insert(workspaceOperations).values({ companyId: input.companyId, issueId: input.issueId,
heartbeatRunId: input.runId, executionWorkspaceId: lease.executionWorkspaceId, phase: "workspace_recovery",
status: "succeeded", finishedAt: new Date(), metadata: { recoveryActionId: input.actionId, leaseId: lease.id,
providerLeaseId: lease.providerLeaseId, descriptorSha256: finalized.descriptorSha256,
finalHostSha256: finalized.finalHostSha256 } }).returning({ id: workspaceOperations.id }))[0];
operationId = copy.id;
} else {
operationId = await preserveWorkspaceSnapshot({ ...input, leaseId: lease.id,
providerLeaseId: lease.providerLeaseId!, target, assertHeld: ownership.assertHeld });
}
await progress({ preservationOperationIds: [...new Set([...current!.preservationOperationIds, operationId])] });
clearInterval(keepalive); await renewal;
await ownership.assertHeld();
const receipt = await input.environmentRuntime.retryPendingSandboxTeardown({ environment: environment as unknown as Environment,
lease: cleanupLease as EnvironmentLease });
const settled = await settleStopOnlyCleanup(db, cleanupLease, { attemptId: ownership.token, receipt });
stopSettled = Boolean(settled && hasRemoteTerminationReceipt(settled));
if (!stopSettled) throw new SystemRecoveryFailure("agent_stop_unconfirmed", "recheck_evidence");
} finally {
clearInterval(keepalive); await renewal;
if (!stopSettled) {
// A lost resume/export reply retains the same stop-only obligation.
// The maintenance worker must never turn it into destructive cleanup.
try {
await ownership.assertHeld();
const settled = await input.environmentRuntime.retryPendingSandboxTeardown({ environment: environment as unknown as Environment,
lease: cleanupLease as EnvironmentLease }).then(receipt => settleStopOnlyCleanup(db, cleanupLease, { attemptId: ownership.token, receipt }), () => null);
if (!settled || !hasRemoteTerminationReceipt(settled))
await restoreSandboxStopAfterFailedRestart(db, cleanupLease, { attemptId: ownership.token });
} catch { /* The persisted stop-only obligation remains authoritative. */ }
}
}
}
await ownership.assertHeld();
if (sourceUnavailable) throw new SystemRecoveryFailure("source_unavailable", "use_alternative_strategy");
const next = { ...current!, revision: current!.revision + 1, phase: "reconcile_effects" as const,
state: "complete" as const, nextAttemptAt: null };
const activity = await db.transaction(async tx => {
await notifyNativeResultCommit(tx, input);
await notifyNativeWorkspaceFinalization(tx, input);
await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionReplacement);
await notifyNativeStatusReassessment(tx, input);
await tx.insert(workspaceOperations).values({ companyId: input.companyId, issueId: input.issueId,
heartbeatRunId: input.runId, phase: "workspace_finalize", status: "succeeded", finishedAt: new Date(),
metadata: { owningService: "workspace_recovery", preservationOperationIds: next.preservationOperationIds } });
if (!await updateSystemRecovery(tx as unknown as Db, input, current!, next))
throw new Error("workspace_recovery_generation_changed");
if (inspected.action.evidence.nativeSourceRetention && !await settleRecoveredNativeSourceRetention(tx as unknown as Db, input, next.preservationOperationIds))
throw new SystemRecoveryFailure("effect_unresolved", "recheck_evidence");
// A settled restore hold returns its warm sandbox to reuse. A hold that
// fails any guard keeps its protection; the saved files still count.
await settleRecoveredWorkspaceRestoreHolds(tx as unknown as Db, input);
return persistActivity(tx as unknown as Db, { companyId: input.companyId, actorType: "system", actorId: "workspace-recovery",
action: "issue.workspace_recovered", entityType: "issue", entityId: input.issueId, runId: input.runId,
details: { recoveryActionId: input.actionId, summary: "Saved work recovered." } });
});
current = next;
try { publishActivity(activity.publication); } catch { logger.warn({ runId: input.runId }, "Recovery activity delivery deferred"); }
} catch (error) {
await ownership.assertHeld();
if (current!.state === "complete") {
logger.warn({ runId: input.runId }, "Preserved workspace activity delivery deferred");
return;
}
const classification = classifyNativeWorkspaceFailure(error);
const diskFull = !(error instanceof SystemRecoveryFailure) && !cacheReclaimUnconfirmed && isSandboxDiskFullSave(error);
const failure = error instanceof SystemRecoveryFailure ? error
: diskFull ? new SystemRecoveryFailure("export_interrupted", "user_action_required")
: classifyWorkspaceRestoreFailure(error) === "restore_invalid_metadata"
? new SystemRecoveryFailure("snapshot_verification_failed", "service_required")
: classification.code === "workspace_sync_out_unsafe_archive"
? new SystemRecoveryFailure("archive_unsafe", "service_required")
: classification.code === "workspace_sync_out_unrecoverable"
? new SystemRecoveryFailure("source_unavailable", "use_alternative_strategy")
: new SystemRecoveryFailure(current!.phase === "verify_execution" ? "provider_unavailable" : "export_interrupted", "retry_transient");
const retryAfterAt = (error as { retryAfterAt?: unknown })?.retryAfterAt;
// A generic transient code says only that the step can be retried. Keep
// why it failed in closed labels so a repeated deferral is diagnosable.
// A classified failure already names its subsystem.
const generic = !(error instanceof SystemRecoveryFailure) && failure.retryClass === "retry_transient"
&& (failure.code === "provider_unavailable" || failure.code === "export_interrupted");
const cause = diskFull ? { ...SANDBOX_DISK_FULL_RECOVERY_CAUSE } : generic ? sandboxProviderFailureLabel(error) : undefined;
const failed = failSystemRecovery(current!, failure, {
...(typeof retryAfterAt === "number" && Number.isFinite(retryAfterAt) ? { retryAfterAt } : {}),
...(cause ? { cause } : {}),
});
const recorded = await updateSystemRecovery(db, input, current!, failed);
// A permanent refusal of the recorded source repeats on every retry.
// Continue from the last save in this same pass, under this ownership.
if (recorded && failed.state === "exhausted" && isUnverifiableSourceFailure(failed.failure)) {
const fallback = await continueFromLastSavedWorkspace({ ...input, ownership }).catch(fallbackError => {
logger.warn({ err: fallbackError, runId: input.runId }, "Continuing from the last save was deferred; the hold remains");
return { kind: "ineligible" as const, reason: "fallback_failed" };
});
if (fallback.kind === "ineligible") await recordRetainedSourceFallbackDeferral(db, input, fallback.reason);
}
if (failure.retryClass === "recheck_evidence" && ["agent_stop_unconfirmed", "copyback_stop_unconfirmed"].includes(failure.code))
await recheckStoppedWorkspaceRecovery(db, input, false);
logger.warn({ err: error, runId: input.runId, failureCode: failure.code, failureCause: cause?.label, phase: current!.phase }, "Workspace preservation deferred; source retained");
} finally { clearInterval(observer); await observing; }
};
const owned = await withNativeWorkspaceFinalizationOwnership({ db, companyId: input.companyId, runId: input.runId,
retainedRepairActionId: input.actionId }, runAttempt);
if (!owned.acquired && owned.reason === "owner_unverified") {
await holdForUnverifiedCopybackOwner(db, input);
await recheckStoppedWorkspaceRecovery(db, input, false);
}
}
/**
* A copyback owner that a restart killed can never confirm its own stop, and
* Retry cannot release its receipt (RC-50). Rechecking that receipt is not a
* recovery attempt: the bounded recovery window still applies, but no attempt
* is spent. Record the exact receipt so the board can confirm the stop through
* `resumeNativeWorkspaceAfterOwnerStop`, and so Retry names this wait.
*/
async function holdForUnverifiedCopybackOwner(db: Db, input: Source) {
const current = await loadSystemRecovery(db, input);
if (!current || current.state === "complete" || current.state === "exhausted") return;
const next = failSystemRecovery(current, new SystemRecoveryFailure("copyback_stop_unconfirmed", "recheck_evidence"));
if (!await updateSystemRecovery(db, input, current, next)) return;
await recordCopybackOwnerHold(db, input, next.generationId);
}
/** Bookkeeping only: names the exact receipt for the board. Grants no attempt. */
async function recordCopybackOwnerHold(db: Db, input: Source, generationId: string) {
const [source] = await db.select({ owner: sql<unknown>`${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceFinalizationOwner'` })
.from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId)));
const owner = source?.owner;
if (!owner || typeof owner !== "object" || typeof (owner as { token?: unknown }).token !== "string") return;
const target = and(eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.id, input.actionId),
eq(issueRecoveryActions.sourceIssueId, input.issueId),
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,generationId}' = ${generationId}`,
sql`${issueRecoveryActions.evidence}#>'{workspaceOwnerRecovery,owner}' is distinct from ${JSON.stringify(owner)}::jsonb`);
await db.transaction(async tx => {
const [action] = await tx.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(target).for("update");
if (!action) return;
// Wake the ownership worker (on any server) before the hold it announces,
// so a quiet worker judges this owner and can wake the recheck.
await notifyDeliveryWork(tx, DELIVERY_QUEUES.ownershipInvariantReport,
JSON.stringify({ companyId: input.companyId, runId: input.runId, actionId: input.actionId }));
await tx.update(issueRecoveryActions).set({
// A confirmation names one receipt; a newer unverified owner needs its own.
evidence: sql`(${issueRecoveryActions.evidence} - 'workspaceOwnerStop') || jsonb_build_object('workspaceOwnerRecovery',
jsonb_build_object('state', 'operator_required', 'owner', ${JSON.stringify(owner)}::jsonb))`,
updatedAt: new Date(),
}).where(target);
});
}
const cursors = new WeakMap<Db, string>();
const workInFlight = new WeakMap<Db, Set<string>>();
/** Overdue recoveries the assigned-work sweep started outside the queue's
* slots. The queue skips their runs; their count has its own small limit. */
const overdueInFlight = new WeakMap<Db, Set<string>>();
export const OVERDUE_WORKSPACE_RECOVERY_LIMIT = 2;
const unfinishedCursors = new WeakMap<Db, string>();
/** Destruction is a later cleanup obligation, authorized only by a matching
* saved-file receipt and the original recorded disposition. */
export async function scheduleRecoveredWorkspaceCleanup(db: Db, run: typeof heartbeatRuns.$inferSelect) {
if (!await hasVerifiedWorkspaceRepair(db, run)) return;
const legacy = readNativeWorkspaceSyncReference(run.runnerProfileJson?.nativeWorkspaceSync);
const parsedRemote = remoteWorkspaceRunReferenceSchema.safeParse(run.runnerProfileJson?.remoteWorkspace);
const remote = parsedRemote.success && parsedRemote.data.state === "saved" ? parsedRemote.data : null;
const ref = legacy ?? remote;
const disposition = legacy?.resourceDisposition ?? (remote ? run.runnerProfileJson?.nativeResourceDisposition : undefined);
if (!ref || disposition !== "destroy") return;
const [lease] = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.id, ref.leaseId)));
if (!lease || !hasRemoteTerminationReceipt(lease)) return;
const [other] = await db.select({ id: environmentLeases.id }).from(environmentLeases).where(and(
ne(environmentLeases.id, lease.id), eq(environmentLeases.provider, lease.provider!), eq(environmentLeases.providerLeaseId, lease.providerLeaseId!),
)).limit(1);
if (other) return;
await writeEnvironmentCleanup(db, tx => tx.update(environmentLeases).set({ status: "pending_cleanup", cleanupStatus: "failed", leasePolicy: "ephemeral",
updatedAt: new Date(), metadata: sql`(${environmentLeases.metadata} - 'workspaceRestoreRecovery' - 'remoteExecutionTermination') ||
${JSON.stringify({ workspaceRecoveryCleanup: { runId: run.id,
// Keep the verified stop: successors wait on the stop, not on the destroy.
stopReceipt: lease.metadata?.remoteExecutionTermination, ...(legacy ? { descriptorSha256: legacy.descriptorSha256 }
: { snapshotId: remote!.snapshotId, epoch: remote!.epoch }) },
pendingCleanupInFlight: false, pendingCleanupRetryAfterMs: 0, pendingCleanupRetryAttempts: 0, pendingCleanupLeaseExpiresAtMs: 0 })}::jsonb`,
}).where(and(eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.id, ref.leaseId),
eq(environmentLeases.heartbeatRunId, run.id), eq(environmentLeases.providerLeaseId, ref.providerLeaseId),
eq(environmentLeases.status, "released"), eq(environmentLeases.cleanupStatus, "success"),
sql`${environmentLeases.metadata} = ${JSON.stringify(lease.metadata)}::jsonb`,
sql`${environmentLeases.metadata}->>'workspaceRecoveryCleanup' is null`, reusableSandboxNotKeptAfterReplacementSql())),
{ companyId: run.companyId, runId: run.id });
}
/** `dueOnly` narrows the obligations to exhausted holds whose re-arm or
* fallback is due now. Cleanup, interrupted preparation and saved input that
* waits stay with the obligation pass. */
export function retainedWorkspaceCandidates(db: Db, cursor?: string, unfinishedOnly = false, obligationsOnly = false, inFlightRunIds: string[] = [],
dueOnly = false) {
const cleanup = sql`(((${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceSync,resourceDisposition}' = 'destroy'
and ${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceSync,leaseId}' is not null)
or (${heartbeatRuns.runnerProfileJson}->>'nativeResourceDisposition' = 'destroy'
and ${heartbeatRuns.runnerProfileJson}#>>'{remoteWorkspace,schema}' = 'paperclip.remote-workspace-run.v1'
and ${heartbeatRuns.runnerProfileJson}#>>'{remoteWorkspace,state}' = 'saved')) and exists (
select 1 from ${environmentLeases} where ${environmentLeases.companyId} = ${heartbeatRuns.companyId}
and ${environmentLeases.heartbeatRunId} = ${heartbeatRuns.id}
and ${environmentLeases.id}::text = coalesce(${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceSync,leaseId}',
${heartbeatRuns.runnerProfileJson}#>>'{remoteWorkspace,leaseId}')
and ${environmentLeases.status} = 'released' and ${environmentLeases.cleanupStatus} = 'success'
and ${environmentLeases.metadata}->'workspaceRecoveryCleanup' is null))`;
const waitingInput = sql`exists (select 1 from ${agentWakeupRequests}
where ${agentWakeupRequests.companyId} = ${issueRecoveryActions.companyId}
and ${agentWakeupRequests.status} = 'deferred_issue_execution' and ${agentWakeupRequests.requestedByActorType} = 'user'
and ${agentWakeupRequests.payload}->>'issueId' = ${issueRecoveryActions.sourceIssueId}::text)`;
const interruptedPreparation = sql`(${heartbeatRuns.runtimeMode} = 'legacy'
and (${heartbeatRuns.status} = 'interrupted' or (${heartbeatRuns.status} = 'failed'
and ${heartbeatRuns.errorCode} = 'process_lost'
and ${heartbeatRuns.resultJson}#>>'{executionRecovery,kind}' = 'bootstrap'
and ${heartbeatRuns.resultJson}#>'{executionRecovery,providerWorkStarted}' = 'false'::jsonb))
and ${heartbeatRuns.runtimeModeResolvedAt} is null and ${heartbeatRuns.executionStage} = 'preparing'
and ${heartbeatRuns.runnerProfileJson}#>>'{adapterDispatch,adapterType}' = 'paperclip_runner'
and (${issueRecoveryActions.evidence}#>>'{automaticRecovery,replay}' = 'blocked'
or (${issueRecoveryActions.evidence}#>>'{automaticRecovery,replay}' = 'allowed'
and ${issueRecoveryActions.evidence}->>'bootstrapPreparationSettledAt' is not null
and coalesce(${issueRecoveryActions.evidence}->>'bootstrapPreparationContinuation', 'pending') in ('pending', 'deferred')))
and exists (select 1 from ${issues} where ${issues.companyId} = ${heartbeatRuns.companyId}
and ${issues.id} = ${issueRecoveryActions.sourceIssueId} and ${issues.assigneeAgentId} = ${heartbeatRuns.agentId}
and ${issues.status} in ('todo', 'in_progress', 'in_review', 'blocked')
and ${issues.executionRunId} is null and ${issues.checkoutRunId} is null
and (${issues.status} <> 'blocked' or
${issueRecoveryActions.evidence}#>>'{nativeBootstrapFailureBlock,statusVersion}' = ${issues.statusVersion}::text)))`;
const now = Date.now();
const dueHold = sql`(${retainedSourceFallbackDueCondition(now)} or ${stopProofRearmDueCondition(now)} or ${capturePolicyRearmDueCondition(now)})`;
return db.select({ action: issueRecoveryActions, run: heartbeatRuns }).from(issueRecoveryActions)
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.companyId, issueRecoveryActions.companyId), eq(heartbeatRuns.id,
sql`case when ${issueRecoveryActions.evidence}->>'runId' ~* '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'
then (${issueRecoveryActions.evidence}->>'runId')::uuid end`)))
.where(and(inArray(heartbeatRuns.status, terminal),
workspaceRecoverySourceEligible(true),
unfinishedOnly ? sql`(${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('scheduled', 'running', 'awaiting_evidence')
or (${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' = 'complete'
and ${issueRecoveryActions.evidence}->>'bootstrapPreparationSettledAt' is not null
and coalesce(${issueRecoveryActions.evidence}->>'bootstrapPreparationContinuation', 'pending') = 'pending'))` : undefined,
obligationsOnly ? sql`(${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('scheduled', 'running', 'awaiting_evidence')
or (${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' = 'complete' and (${cleanup} or ${interruptedPreparation})) or ${waitingInput}
or ${dueHold})` : undefined,
dueOnly ? dueHold : undefined,
sql`(${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('scheduled', 'running', 'awaiting_evidence', 'complete', 'exhausted')
and (${issueRecoveryActions.evidence}#>'{systemRecovery,nextAttemptAt}' = 'null'::jsonb
or ${issueRecoveryActions.evidence}#>'{systemRecovery,nextAttemptAt}' <= to_jsonb(${now}::bigint)))`,
sql`(${issueRecoveryActions.evidence}#>>'{workspaceRecovery,status}' is distinct from 'recovered' or
${cleanup} or ${waitingInput} or ${interruptedPreparation})`,
cursor ? gt(issueRecoveryActions.id, cursor) : undefined,
inFlightRunIds.length ? notInArray(heartbeatRuns.id, inFlightRunIds) : undefined,
)).orderBy(asc(issueRecoveryActions.id)).limit(5);
}
/**
* The worker's lane positions, kept in memory: a restart starts every lane
* from its first candidate, so no persisted state or migration is involved.
* `cursor`, `complete` and `restart` belong to the obligation pass; `dueCursor`
* to the due lane; `firstLane` names the lane that has priority in the next run.
*/
export type PassiveSweep = { cursor?: string; complete?: boolean; restart?: boolean; dueCursor?: string; firstLane?: number };
type CandidateRow = Awaited<ReturnType<typeof retainedWorkspaceCandidates>>[number];
type CandidateLane = { rows: CandidateRow[]; index: number; advance: (actionId: string) => void };
/** Bounded discovery shares the periodic recovery lifecycle. A repaired source
* still owes cleanup even without saved input or after task reconciliation.
*
* Candidates come from lanes, each an ordered page that a cursor advances:
* unfinished saves; for the worker, exhausted holds whose re-arm or fallback
* is due; and the obligation pass (cleanup, interrupted preparation, saved
* input that waits, and the same due holds). Each lane with work takes one
* free slot before any lane takes a second, and the worker rotates which lane
* goes first, so no kind of load (a save another server owns, a long local
* export, hundreds of waiting holds) starves another. */
export async function reconcileRetainedWorkspaces(db: Db, runtime: EnvironmentRuntimeService,
onRecovered: (run: typeof heartbeatRuns.$inferSelect) => Promise<void>, obligationsOnly = false,
onFinished?: (continueSweep: boolean) => void, passiveSweep?: PassiveSweep) {
await adoptHistoricalWorkspaceRecoveries(db);
let pending = workInFlight.get(db);
if (!pending) { pending = new Set(); workInFlight.set(db, pending); }
if (pending.size >= 2) return;
// Locally running exports already own their slot. Exclude them before
// pagination so a page containing only held work cannot prevent wrapping.
const inFlightRunIds = [...pending, ...(overdueInFlight.get(db) ?? [])];
let unfinishedRows = await retainedWorkspaceCandidates(db, unfinishedCursors.get(db), true, false, inFlightRunIds);
if (!unfinishedRows.length) unfinishedRows = await retainedWorkspaceCandidates(db, undefined, true, false, inFlightRunIds);
const unfinished: CandidateLane = { rows: unfinishedRows, index: 0, advance: id => unfinishedCursors.set(db, id) };
let rows = await retainedWorkspaceCandidates(db, passiveSweep ? passiveSweep.cursor : cursors.get(db), false, obligationsOnly, inFlightRunIds);
if (passiveSweep && !rows.length && passiveSweep.restart) {
Object.assign(passiveSweep, { cursor: undefined, restart: false });
rows = await retainedWorkspaceCandidates(db, undefined, false, obligationsOnly, inFlightRunIds);
}
if (passiveSweep) passiveSweep.complete = !rows.length;
else if (!rows.length) rows = await retainedWorkspaceCandidates(db, undefined, false, obligationsOnly, inFlightRunIds);
const sweep: CandidateLane = { rows, index: 0,
advance: id => { if (passiveSweep) passiveSweep.cursor = id; else cursors.set(db, id); } };
const lanes = [unfinished];
if (passiveSweep) {
// The due lane wraps at its tail like the unfinished lane. A hold that
// becomes due by time alone (a lease expiry, a re-arm backoff) is reached
// even after the obligation pass completed with no committed change.
let dueRows = await retainedWorkspaceCandidates(db, passiveSweep.dueCursor, false, true, inFlightRunIds, true);
if (!dueRows.length && passiveSweep.dueCursor)
dueRows = await retainedWorkspaceCandidates(db, undefined, false, true, inFlightRunIds, true);
const due: CandidateLane = { rows: dueRows, index: 0, advance: id => { passiveSweep.dueCursor = id; } };
lanes.push(due, sweep);
// Priority rotates across all three lanes, so each lane is first in one
// run of three. With one free slot (a long local export holds the other),
// a save that keeps finding its run's lock busy cannot take it every run.
const first = (passiveSweep.firstLane ?? 0) % lanes.length;
lanes.push(...lanes.splice(0, first));
passiveSweep.firstLane = (first + 1) % lanes.length;
} else lanes.push(sweep);
const work: Promise<void>[] = [];
// Several actions of one run (an export hold and a later no-replay hold)
// run one after another in that run's job. Skipping the later one would
// starve it while the earlier one stays selectable (for example while saved
// input waits for the later one to settle).
const groups = new Map<string, { run: typeof heartbeatRuns.$inferSelect; actions: (typeof issueRecoveryActions.$inferSelect)[] }>();
const join = (lane: CandidateLane) => {
for (; lane.index < lane.rows.length; lane.index += 1) {
const { action, run } = lane.rows[lane.index]!;
const group = groups.get(run.id);
if (!group) return lane.rows[lane.index]!;
if (!group.actions.some(item => item.id === action.id)) group.actions.push(action);
lane.advance(action.id);
}
return undefined;
};
// Takes the lane's next run into a free slot. Actions of a run already taken
// (by any lane) join its job without a slot.
const take = (lane: CandidateLane) => {
const next = join(lane);
if (!next || pending.size >= 2) return false;
lane.index += 1;
lane.advance(next.action.id);
pending.add(next.run.id);
groups.set(next.run.id, { run: next.run, actions: [next.action] });
return true;
};
for (let taken = true; taken && pending.size < 2;) {
taken = false;
for (const lane of lanes) if (pending.size < 2 && take(lane)) taken = true;
}
for (const lane of lanes) join(lane);
const hasUnvisited = lanes.some(lane => lane.index < lane.rows.length);
for (const { run, actions } of groups.values()) {
const finish = beginIdleTrackedWork();
const physical = actions.some(action => ["scheduled", "running", "awaiting_evidence"].includes(readSystemRecovery(action.evidence)?.state ?? ""));
work.push((async () => { try {
for (const action of actions) await recoverRunAction(db, runtime, onRecovered, action, run);
} finally { pending.delete(run.id); finish(); onFinished?.(physical || hasUnvisited); } })());
}
return work;
}
/**
* Starts one saved-work recovery that the queue has not reached within its
* bound (the assigned-work sweep's repair). It runs outside the queue's two
* slots, so a queue held by slow or stuck attempts cannot keep a task waiting
* indefinitely, and it goes through the same ownership and attempt bookkeeping
* as the queue, so it never runs beside another attempt for the same run.
* Resolves once the attempt is started (or refused), not when it finishes.
*/
export async function startOverdueWorkspaceRecovery(db: Db, runtime: EnvironmentRuntimeService,
onRecovered: (run: typeof heartbeatRuns.$inferSelect) => Promise<void>, source: Source):
Promise<"started" | "in_flight" | "busy" | "not_eligible"> {
const overdue = overdueInFlight.get(db) ?? new Set<string>();
overdueInFlight.set(db, overdue);
if (overdue.has(source.runId) || workInFlight.get(db)?.has(source.runId)) return "in_flight";
if (overdue.size >= OVERDUE_WORKSPACE_RECOVERY_LIMIT) return "busy";
const [row] = await db.select({ action: issueRecoveryActions, run: heartbeatRuns }).from(issueRecoveryActions)
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.companyId, issueRecoveryActions.companyId), eq(heartbeatRuns.id, source.runId)))
.where(and(eq(issueRecoveryActions.companyId, source.companyId), eq(issueRecoveryActions.id, source.actionId),
eq(issueRecoveryActions.sourceIssueId, source.issueId), sql`${issueRecoveryActions.evidence}->>'runId' = ${source.runId}`,
inArray(heartbeatRuns.status, terminal), workspaceRecoverySourceEligible(),
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('scheduled', 'running', 'awaiting_evidence')`)).limit(1);
if (!row) return "not_eligible";
if (overdue.has(source.runId) || workInFlight.get(db)?.has(source.runId)) return "in_flight";
overdue.add(source.runId);
const finish = beginIdleTrackedWork();
void (async () => {
try { await recoverRunAction(db, runtime, onRecovered, row.action, row.run); }
finally { overdue.delete(source.runId); finish(); }
})();
return "started";
}
async function recoverRunAction(db: Db, runtime: EnvironmentRuntimeService,
onRecovered: (run: typeof heartbeatRuns.$inferSelect) => Promise<void>,
action: typeof issueRecoveryActions.$inferSelect, run: typeof heartbeatRuns.$inferSelect) {
try {
await recoverRetainedWorkspace({ db, environmentRuntime: runtime, companyId: action.companyId, issueId: action.sourceIssueId, runId: run.id, actionId: action.id });
const [updated] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action.id));
const recovered = readWorkspaceRecovery(updated?.evidence ?? null)?.status === "recovered";
const exhausted = readSystemRecovery(updated?.evidence)?.state === "exhausted";
if (recovered || exhausted) {
const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
if (current) {
if (recovered) {
await resumePreservedResultFinalization(db, current.companyId, current.id);
await scheduleRecoveredWorkspaceCleanup(db, current);
}
// Only normal dispatch can consume existing authorized input. An
// exhausted export permits evaluation of an isolated continuation,
// never another export attempt or a replay of an old assignment.
await onRecovered(current);
}
}
} catch (error) { logger.warn({ err: error, runId: run.id, actionId: action.id }, "Retained workspace remains pending"); }
}