Files
PaperClipAI/server/src/services/workspace-recovery.ts
T
DottaandClaude 1ac8b6a711 fix(workspaces): stop re-saving a retained sandbox whose disk is still full (#20758)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - After a native run on a remote sandbox, Paperclip saves the sandbox
workspace back to durable storage. When that save fails, saved-work
recovery saves the files again from the retained sandbox, with no
provider turn.
> - A staging census of the last 48 hours
(`doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`, class F)
found saves that failed because the sandbox disk was full.
> - #20678 already ends finalization at the first full-disk copy-back
and hands the work to saved-work recovery. Recovery then deletes
inactive tool caches that it can prove safe, and saves again.
> - When that save still found the disk full, recovery recorded a
transient `export_interrupted` failure with cause `provider_error`. It
then saved again on its automatic schedule (up to 5 retries in 30
minutes). Nothing frees space in a stopped, retained sandbox between
those attempts, so each attempt failed the same way. The task showed
"Recovering saved work…" and no Retry while this went on.
> - This pull request ends the recovery generation at the first such
failure with a named cause, `sandbox_disk_full`, and hands it to the
board with a Retry.
> - The benefit is that the task does not spend 30 minutes on saves that
cannot succeed, the user sees why, and one Retry saves the files and
starts one run.

## Linked Issues or Issue Description

Refs #20678, #20695 (named sandbox disk-full failures for copy-back and
restore)

**What happened?**
A native run on a remote sandbox finished, and its workspace copy-back
hit a full sandbox disk. Finalization handed the task to saved-work
recovery (#20678). Recovery's own save also hit the full disk. Recovery
classified this as `export_interrupted` / `retry_transient` with cause
`provider_error`, so it scheduled more automatic saves of the same
sandbox. Each one failed the same way. The task showed "Recovering saved
work…" with no Retry until the 30-minute window ran out.

**Expected behavior**
A save that still finds the disk full after the safe cache cleanup is
not repeated automatically. The recovery ends at once with the cause
`sandbox_disk_full`, and the board gets a Retry. The workspace keeps its
previous saved version, and the run's unsaved files stay in the retained
sandbox. One Retry saves the files once there is room and starts exactly
one run.

**Steps to reproduce**
1. Give a native run an accepted result whose copy-back failed
(`native_workspace_sync_out_retry_exhausted`).
2. Make every save from its retained sandbox fail with
`remote_workspace_sandbox_disk_full`, with no inactive tool cache to
delete.
3. Run saved-work recovery. Before this change, it records
`export_interrupted` with cause `provider_error` and schedules another
save.

## What Changed

- `workspace-recovery.ts`: a save that fails with
`remote_workspace_sandbox_disk_full` after a confirmed cache cleanup (or
with nothing to clean up) now fails the generation with
`export_interrupted`, retry class `user_action_required`, and cause `{
label: "sandbox_disk_full", code: "ENOSPC" }`. That class is not retried
automatically, and it is not a permanent-source class, so recovery never
falls back to the last save and never drops the run's unsaved files. The
next sweep hands the hold to the board.
- A cleanup whose outcome is unknown (lost acknowledgement, interrupted
journal) keeps the existing scheduled retry, because its journal resumes
on the next attempt and can still free space.
- `retained-provider-cache.ts`: `reclaimInactiveNativeProviderCache`
returns `{ changed, confirmed }` instead of a boolean, so the caller can
tell "nothing to delete" from "outcome unknown". It has one caller.
- `system-recovery-state.ts`: the board handoff text for this cause says
that the sandbox disk is full, that Paperclip deletes only inactive tool
caches it can prove safe, and that Retry saves the files again.
- New `workspace-recovery-disk-full.test.ts` covers the whole path: one
failed attempt, no further automatic saves, the board handoff text, the
previous saved version unchanged, the agent's files still in the
sandbox, no unfinished in-sandbox capture attempt, a Retry offered in
the task presentation and blocker, `POST /issues/:id/retry` accepted
(202), the files saved after Retry, and exactly one run started (also
with a replayed dispatch).
- `doc/execution-semantics.md`: documents the behavior.

## Verification

- Red on master: `cd server && npx vitest run
src/services/workspace-recovery-disk-full.test.ts` fails. The recovery
state is `scheduled` with cause `provider_error` instead of `exhausted`
with cause `sandbox_disk_full`.
- Green on this branch: the same command passes.
- Related suites pass: `workspace-recovery-retry-copyback.test.ts` (79
tests, including the 8 cache-cleanup resume tests that need the
scheduled retry after an unconfirmed cleanup),
`system-recovery-state.test.ts`,
`workspace-recovery-capture-policy.test.ts`,
`workspace-recovery-fallback.test.ts`,
`workspace-recovery-stop-proof.test.ts`,
`native-run-finalizer-telemetry.test.ts` (167 tests in all).
- `npx tsc --noEmit -p server/tsconfig.json` is clean.
- `pnpm verify`: exit 2 (broader verification required), because
`doc/execution-semantics.md` is a global CI input. The focused checks
above pass. Required CI supplies the broad coverage.

## Risks

- Low. The change touches only the saved-work recovery path when its
save fails with the exact full-disk message.
- Upgrade-safe: the failure code (`export_interrupted`), the retry class
(`user_action_required`) and the cause label all pass the earlier
release's recovery schema. An earlier release that reads this state
shows its generic "Retry to try again" text.
- A full-disk failure no longer gets automatic retries in its 30-minute
window. If space became free by itself in that window, the user must now
press Retry. Nothing in a stopped, retained sandbox frees space by
itself, so this is expected to be rare.
- No Runner (Rust) change, so no new sandbox image is needed.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, test execution) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-11 01:31:50 -06:00

787 lines
57 KiB
TypeScript

import { classifyWorkspaceRestoreFailure } from "@paperclipai/adapter-utils/workspace-restore-merge";
import { createHash } from "node:crypto";
import { adapterExecutionControls } from "./adapter-execution-control.js";
import { isAcknowledgedNativeStop } from "./acknowledged-native-stop.js";
import { notifyNativeResultCommit } from "./native-result-commit-notifications.js";
import { notifyNativeWorkspaceFinalization } from "./native-workspace-finalization-notifications.js";
import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
import { beginIdleTrackedWork } from "./task-admission.js";
import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
import { resumePreservedResultFinalization } from "./preserved-result-finalization.js";
import { reclaimInactiveNativeProviderCache } from "./native-runtime/retained-provider-cache.js";
import { REMOTE_WORKSPACE_SANDBOX_DISK_FULL, sandboxProviderFailureLabel } from "./remote-workspace-transport.js";
import { and, asc, eq, gt, inArray, ne, notInArray, or, sql } from "drizzle-orm";
import { agentWakeupRequests, environmentLeases, executionWorkspaces, heartbeatRuns, issues, issueRecoveryActions, nativeRunFinalizations, workspaceOperations, type Db } from "@paperclipai/db";
import { remoteWorkspaceRunReferenceSchema, type Environment, type EnvironmentLease } from "@paperclipai/shared";
import { conflict } from "../errors.js";
import { logger } from "../middleware/logger.js";
import type { EnvironmentRuntimeService } from "./environment-runtime.js";
import { resolveEnvironmentExecutionTarget } from "./environment-execution-target.js";
import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
import { confirmSandboxRecoveryRestart, hasStopOnlyCleanup, prepareHistoricalSandboxStopVerification, prepareSandboxStopAndRetain, prepareStoppedSandboxForRecovery, restoreSandboxStopAfterFailedRestart, settleAbsentSandbox, settleStopOnlyCleanup } from "./sandbox-stop-and-retain.js";
import { requestSandboxAbsenceProof } from "./sandbox-absence-proof.js";
import { persistActivity, publishActivity } from "./activity-log.js";
import { hasRequiredWorkspaceRecovery } from "./workspace-restore-recovery-state.js";
import { hasVerifiedWorkspaceRepair, readWorkspaceRecovery, type WorkspaceRecovery } from "./workspace-recovery-receipt.js";
import { withNativeWorkspaceFinalizationOwnership } from "./native-runtime/native-workspace-finalization-ownership.js";
import { readNativeWorkspaceSyncReference, readNativeWorkspaceSyncLocalCwd, resumeNativeWorkspaceSync } from "./native-runtime/native-workspace-sync.js";
import { classifyNativeWorkspaceFailure } from "./native-runtime/native-workspace-failure.js";
import { handOffExhaustedSystemRecovery, adoptSystemRecovery, loadSystemRecovery, updateSystemRecovery } from "./system-recovery.js";
import { SYSTEM_RECOVERY_MAX_RETRIES, createSystemRecovery, failSystemRecovery, readSystemRecovery, reserveSystemRecoveryAttempt, SystemRecoveryFailure, type SystemRecoveryState } from "./system-recovery-state.js";
import { inspectWorkspaceRecoverySource, workspaceRecoverySourceEligible, workspaceSourceBindingFailure } from "./workspace-recovery-source.js";
import { preserveWorkspaceSnapshot } from "./workspace-recovery-snapshot.js";
import { adoptHistoricalWorkspaceRecoveries } from "./workspace-recovery-backfill.js";
import { canAdoptNativeSourceRetention, retainNativeRecoverySource, settleRecoveredNativeSourceRetention } from "./native-runtime/native-source-retention.js";
import { saveRemoteWorkspace } from "./remote-workspaces.js";
import { continueFromLastSavedWorkspace, isUnverifiableSourceFailure, recordRetainedSourceFallbackDeferral, settleExhaustedRetainedHold } from "./workspace-recovery-fallback.js";
import { capturePolicyRearmDueCondition, retainedSourceFallbackDueCondition, stopProofRearmDueCondition } from "./workspace-recovery-fallback-discovery.js";
import { rearmExhaustedCapturePolicy } from "./workspace-recovery-capture-policy.js";
import { rearmExhaustedStopProof } from "./workspace-recovery-stop-proof.js";
import { settleRecoveredWorkspaceRestoreHolds } from "./workspace-restore-hold-settlement.js";
const exportCauses = ["native_workspace_sync_out_unrecoverable", "native_workspace_sync_out_retry_exhausted"];
const terminal = ["failed", "interrupted", "timed_out", "cancelled", "succeeded"];
const sandboxDiskFull = /No space left on device|Disk quota exceeded/i;
export const SANDBOX_DISK_FULL_RECOVERY_CAUSE = { label: "sandbox_disk_full", code: "ENOSPC", status: null } as const;
const isSandboxDiskFullSave = (error: unknown) => error instanceof Error && error.message === REMOTE_WORKSPACE_SANDBOX_DISK_FULL;
type Source = { companyId: string; issueId: string; runId: string; actionId: string };
type StopEvidenceSource = Omit<Source, "actionId"> & { actionId?: string };
async function recheckStoppedWorkspaceRecovery(db: Db, source: StopEvidenceSource, observe: boolean) {
return db.transaction(async tx => {
const [issue] = await tx.select({ id: issues.id }).from(issues).where(and(
eq(issues.id, source.issueId), eq(issues.companyId, source.companyId))).for("update");
const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, source.runId),
eq(heartbeatRuns.companyId, source.companyId))).for("update");
if (!issue || !run || run.runtimeMode !== "native" || run.nativeIssueId !== issue.id ||
!isAcknowledgedNativeStop(run) || adapterExecutionControls.has(run.id)) return 0;
const cancellation = run.resultJson?.executionCancellation as { state?: unknown; proof?: unknown; acknowledgedAt?: unknown } | undefined;
if (cancellation?.state !== "acknowledged" || cancellation.proof !== "provider_termination_receipt") return 0;
const [coordinator] = await tx.select({ phase: nativeRunFinalizations.phase, leaseOwner: nativeRunFinalizations.leaseOwner })
.from(nativeRunFinalizations).where(and(eq(nativeRunFinalizations.companyId, source.companyId),
eq(nativeRunFinalizations.runId, source.runId), eq(nativeRunFinalizations.issueId, source.issueId)));
if (!coordinator || coordinator.leaseOwner || !["terminal_failure", "applied"].includes(coordinator.phase)) return 0;
const leases = await tx.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, source.companyId),
eq(environmentLeases.heartbeatRunId, source.runId)));
if (!leases.length || leases.some(lease => !lease.provider || lease.provider === "local" || !hasRemoteTerminationReceipt(lease))) return 0;
const intent = run.resultJson!.nativeCancellation as Record<string, unknown>;
const fingerprint = createHash("sha256").update(JSON.stringify({ companyId: source.companyId, issueId: source.issueId,
runId: source.runId, intentAuditId: intent.intentAuditId, acknowledgementAuditId: intent.acknowledgementAuditId,
acknowledgedAt: cancellation.acknowledgedAt, leases: leases.sort((a, b) => a.id.localeCompare(b.id)).map(lease => {
const receipt = lease.metadata!.remoteExecutionTermination as Record<string, unknown>;
return { leaseId: lease.id, provider: lease.provider, providerLeaseId: lease.providerLeaseId,
state: receipt.state, confirmedAt: receipt.confirmedAt };
}) })).digest("hex");
const actions = await tx.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, source.companyId),
eq(issueRecoveryActions.sourceIssueId, source.issueId), or(inArray(issueRecoveryActions.status, ["active", "escalated"]), and(
eq(issueRecoveryActions.status, "resolved"), sql`${issueRecoveryActions.evidence}#>>'{automaticRecovery,policy}' = 'preserve_without_replay_v1'`,
sql`${issueRecoveryActions.evidence}#>>'{automaticRecovery,replay}' = 'blocked'`,
sql`${issueRecoveryActions.evidence}#>>'{automaticRecovery,runId}' = ${source.runId}`)),
sql`${issueRecoveryActions.evidence}->>'runId' = ${source.runId}`,
source.actionId ? eq(issueRecoveryActions.id, source.actionId) : undefined)).for("update");
let changed = 0;
for (const action of actions) {
const current = readSystemRecovery(action.evidence), workspace = readWorkspaceRecovery(action.evidence), now = Date.now();
if (!current || current.sourceRunId !== source.runId || current.phase !== "verify_execution" ||
!workspace || workspace.runId !== source.runId || ["complete", "exhausted"].includes(current.state) || current.retriesUsed >= SYSTEM_RECOVERY_MAX_RETRIES ||
(current.deadlineAt !== null && now >= current.deadlineAt)) continue;
const stopWait = current.failure?.retryClass === "recheck_evidence" &&
["agent_stop_unconfirmed", "copyback_stop_unconfirmed"].includes(current.failure.code);
if (current.state !== "running" && current.failure && !stopWait) continue;
const prior = action.evidence.workspaceStopEvidence as Record<string, unknown> | undefined;
const same = prior?.schema === "paperclip.workspace-stop-evidence.v1" && prior.sourceRunId === source.runId &&
prior.generationId === current.generationId && prior.fingerprint === fingerprint;
if ((same && prior!.consumed === true) || (!observe && (!same || prior!.consumed !== false))) continue;
const due = Math.max(now, current.providerCooldownAt ?? 0);
const advance = current.state === "awaiting_evidence" && stopWait && current.attempts > 0 &&
current.deadlineAt !== null && due < current.deadlineAt && current.nextAttemptAt !== null && current.nextAttemptAt > due;
const consumed = current.state !== "running";
if (!advance && same && prior!.consumed === consumed) continue;
await notifyDeliveryWork(tx, DELIVERY_QUEUES.workspaceRecovery);
if (advance && !await updateSystemRecovery(tx as unknown as Db, { ...source, actionId: action.id }, current,
{ ...current, revision: current.revision + 1, nextAttemptAt: due })) continue;
const marker = { schema: "paperclip.workspace-stop-evidence.v1", sourceRunId: source.runId,
generationId: current.generationId, fingerprint, consumed };
const recorded = await tx.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence} ||
${JSON.stringify({ workspaceStopEvidence: marker })}::jsonb`, updatedAt: new Date() }).where(and(
eq(issueRecoveryActions.companyId, source.companyId), eq(issueRecoveryActions.id, action.id),
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,generationId}' = ${current.generationId}`,
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,revision}' = ${String(current.revision + (advance ? 1 : 0))}`))
.returning({ id: issueRecoveryActions.id });
if (!recorded.length) throw new Error("workspace_stop_evidence_changed");
changed += 1;
}
return changed;
});
}
export function observeStoppedWorkspaceRecovery(db: Db, source: StopEvidenceSource) {
return recheckStoppedWorkspaceRecovery(db, source, true);
}
/**
* Ownership-reconciler settle for the RC-50 copyback-owner hold. The recorded
* owner was proven gone by the shared prover, so the waiting recheck is due
* now instead of after its backoff. It grants no attempt and settles nothing
* itself: the worker still proves the owner and its copyback process group
* before it takes ownership. Returns whether the recheck was woken.
*/
export async function wakeCopybackOwnerRecheck(db: Db, input: {
companyId: string; actionId: string; now: Date;
owner: { pid: number | null; processStartedAt: string | null; bootId: string | null; processSpace: string | null };
}): Promise<boolean> {
return db.transaction(async tx => {
const [action] = await tx.select().from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.id, input.actionId),
inArray(issueRecoveryActions.status, ["active", "escalated"]))).for("update");
const hold = action?.evidence?.workspaceOwnerRecovery as { state?: unknown; owner?: Record<string, unknown> } | undefined;
const owner = hold?.owner;
if (!action || hold?.state !== "operator_required" || !owner || owner.pid !== input.owner.pid
|| owner.processStartedAt !== input.owner.processStartedAt
|| (owner.controllerBootId ?? null) !== input.owner.bootId || (owner.hostHash ?? null) !== input.owner.processSpace) return false;
const current = readSystemRecovery(action.evidence);
const runId = action.evidence.runId;
const now = input.now.getTime();
if (!current || typeof runId !== "string" || current.sourceRunId !== runId || current.state !== "awaiting_evidence"
|| current.failure?.code !== "copyback_stop_unconfirmed"
|| (current.deadlineAt !== null && now >= current.deadlineAt)) return false;
const due = Math.max(now, current.providerCooldownAt ?? 0);
if (current.nextAttemptAt !== null && current.nextAttemptAt <= due) {
await notifyDeliveryWork(tx, DELIVERY_QUEUES.workspaceRecovery);
return true;
}
return updateSystemRecovery(tx as unknown as Db, { companyId: input.companyId, issueId: action.sourceIssueId,
runId, actionId: action.id }, current, { ...current, revision: current.revision + 1, nextAttemptAt: due });
});
}
/** Admission records intent only. No request handler runs physical recovery. */
export async function requestWorkspaceRecovery(input: Source & { db: Db; idempotencyKey: string; actorId: string }) {
return input.db.transaction(async tx => {
const [issue] = await tx.select().from(issues).where(and(eq(issues.id, input.issueId), eq(issues.companyId, input.companyId))).for("update");
const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId))).for("update");
const [action] = await tx.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.id, input.actionId),
eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, input.issueId))).for("update");
if (!issue || !run || !action || action.evidence.runId !== run.id
|| (run.nativeIssueId ?? run.issueId ?? run.contextSnapshot?.issueId) !== issue.id
|| !terminal.includes(run.status) || (!hasRequiredWorkspaceRecovery(run.resultJson) && !exportCauses.includes(action.cause))) {
throw conflict("The workspace recovery changed. Refresh the task before retrying.");
}
const prior = readWorkspaceRecovery(action.evidence);
if (prior?.requestIds.includes(input.idempotencyKey) || prior?.status === "recovered") return { actionId: action.id, status: prior.status };
const pending = prior && ["queued", "recovering"].includes(prior.status);
const state: WorkspaceRecovery = pending ? { ...prior!, requestIds: [...prior!.requestIds, input.idempotencyKey] }
: { schema: "paperclip.workspace-recovery.v1", runId: run.id, status: "queued", stage: "verifying_stop",
attempt: prior?.attempt ?? 0, attemptLimit: (prior?.attempt ?? 0) + 1,
requestIds: [...(prior?.requestIds ?? []), input.idempotencyKey] };
const previous = readSystemRecovery(action.evidence);
const active = previous && ["scheduled", "running", "awaiting_evidence"].includes(previous.state);
const system = previous?.state === "running" ? previous : active ? { ...previous, revision: previous.revision + 1,
nextAttemptAt: Math.max(Date.now(), previous.providerCooldownAt ?? 0) } : { ...createSystemRecovery(run.id),
revision: (previous?.revision ?? -1) + 1,
preservationOperationIds: previous?.preservationOperationIds ?? [],
providerCooldownAt: previous?.providerCooldownAt ?? null,
nextAttemptAt: previous?.providerCooldownAt ?? null };
await notifyDeliveryWork(tx, DELIVERY_QUEUES.workspaceRecovery);
await tx.update(issueRecoveryActions).set({ ownerType: "system", ownerAgentId: null, ownerUserId: null,
evidence: { ...action.evidence, workspaceRecovery: state, systemRecovery: system }, updatedAt: new Date() })
.where(eq(issueRecoveryActions.id, action.id));
await persistActivity(tx as unknown as Db, { companyId: input.companyId, actorType: "user", actorId: input.actorId,
action: "issue.workspace_recovery_requested", entityType: "issue", entityId: issue.id, runId: run.id,
details: { recoveryActionId: action.id, idempotencyKey: input.idempotencyKey,
...(previous && previous.generationId !== system.generationId ? { previousGeneration: previous } : {}) } });
return { actionId: action.id, status: state.status };
});
}
/** Physical repair remains serialized by the existing source ownership lock.
* Neither preservation nor cleanup creates an agent turn. */
export async function recoverRetainedWorkspace(input: Source & { db: Db; environmentRuntime: EnvironmentRuntimeService }) {
const { db } = input;
await adoptSystemRecovery(db, input);
let initial = await loadSystemRecovery(db, input);
// A hold whose files a sibling saved, or whose source is refused permanently
// (including holds saved by older releases), settles without a person.
// An adoptable native source keeps its retention path; only the sibling
// settlement applies to it.
if (initial?.state === "exhausted"
&& await settleExhaustedRetainedHold(input, initial, { fallback: !canAdoptNativeSourceRetention(initial) })) return;
// An exhausted wait for stop proof is re-armed, bounded, once the run is over.
if (initial?.state === "exhausted" && await rearmExhaustedStopProof(db, input, initial))
initial = await loadSystemRecovery(db, input);
// A save exhausted under an older capture policy is retried once under the current one.
if (initial?.state === "exhausted" && await rearmExhaustedCapturePolicy(db, input, initial))
initial = await loadSystemRecovery(db, input);
if (initial?.state === "exhausted" && !canAdoptNativeSourceRetention(initial)) {
// A budget spent before this hold existed still names its receipt, so the
// board confirmation in its handoff can release it (RC-50 wave databases).
if (initial.failure?.code === "copyback_stop_unconfirmed") await recordCopybackOwnerHold(db, input, initial.generationId);
await handOffExhaustedSystemRecovery(db, input, initial);
return;
}
if (!initial || initial.state === "complete" ||
(initial.nextAttemptAt !== null && initial.nextAttemptAt > Date.now())) return;
const runAttempt = async (ownership: { token: string; assertHeld(): Promise<void> }) => {
let current = await loadSystemRecovery(db, input);
if (!current) return;
if (current.state === "exhausted") {
const spent = current;
current = await retainNativeRecoverySource(db, input, current, ownership);
if (!current) {
// Not adoptable now. The board owns the spent budget until Retry or a
// later adoption returns it to the system.
await handOffExhaustedSystemRecovery(db, input, (await loadSystemRecovery(db, input)) ?? spent);
return;
}
}
const reserved = reserveSystemRecoveryAttempt(current);
if (!reserved) {
if (current.state !== "exhausted" && current.deadlineAt !== null && Date.now() >= current.deadlineAt)
await updateSystemRecovery(db, input, current, { ...current, revision: current.revision + 1,
state: "exhausted", nextAttemptAt: null });
return;
}
if (!await updateSystemRecovery(db, input, current, reserved)) return;
current = reserved;
const progress = async (patch: Partial<SystemRecoveryState>) => {
await ownership.assertHeld();
const next = { ...current!, ...patch, revision: current!.revision + 1 };
if (!await updateSystemRecovery(db, input, current!, next)) throw new Error("workspace_recovery_generation_changed");
current = next;
};
const observeActive = async () => {
await ownership.assertHeld();
await db.update(issueRecoveryActions).set({ evidence: sql`jsonb_set(${issueRecoveryActions.evidence},
'{systemRecoveryActiveAt}', ${JSON.stringify(Date.now())}::jsonb)` }).where(and(
eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.id, input.actionId),
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,attemptId}' = ${reserved.attemptId}`));
};
let observing = Promise.resolve();
let cacheReclaimUnconfirmed = false;
const observer = setInterval(() => { observing = observing.then(observeActive).catch(() => undefined); }, 15_000);
try {
await observeActive();
const inspected = await inspectWorkspaceRecoverySource(db, input);
let sourceUnavailable = false;
for (const source of inspected.sources) {
const { lease, environment, remoteCwd } = source;
if (hasRemoteTerminationReceipt(lease)
&& (lease.metadata?.remoteExecutionTermination as Record<string, unknown> | undefined)?.state === "destroyed") {
sourceUnavailable = true;
continue;
}
if (lease.provider === "daytona" && lease.providerLeaseId) {
const absence = await requestSandboxAbsenceProof({ companyId: input.companyId, issueId: input.issueId, runId: input.runId, leaseId: lease.id,
provider: "daytona", providerLeaseId: lease.providerLeaseId, ownerToken: ownership.token });
if (absence) {
await ownership.assertHeld();
if (!await settleAbsentSandbox(db, lease, absence)) throw new SystemRecoveryFailure("agent_stop_unconfirmed", "recheck_evidence");
sourceUnavailable = true;
continue;
}
}
let reference = source.preservationOnly ? null : source.reference;
if (reference) {
try { await readNativeWorkspaceSyncLocalCwd({ runId: input.runId, companyId: input.companyId,
workspaceId: reference.workspaceId, reference }); }
catch (error) {
// Missing local merge metadata permits a separate snapshot of the
// exact bound allocation. Corrupt metadata never weakens binding.
if ((error as NodeJS.ErrnoException)?.code === "ENOENT") reference = null;
else throw workspaceSourceBindingFailure("local_descriptor_invalid");
}
}
let stoppedLease = lease;
if (!hasRemoteTerminationReceipt(stoppedLease)) {
const pending = await db.transaction(async tx => {
const [fresh] = await tx.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, input.companyId),
eq(environmentLeases.id, lease.id))).for("update");
if (!fresh || fresh.metadata?.pendingCleanupInFlight === true) return null;
const prepared = hasStopOnlyCleanup(fresh) ? fresh : fresh.status === "released"
? await prepareHistoricalSandboxStopVerification(tx as unknown as Db, fresh)
: await prepareSandboxStopAndRetain(tx as unknown as Db, fresh);
if (!prepared) return null;
const [claimed] = await tx.update(environmentLeases).set({ metadata: sql`${environmentLeases.metadata} ||
${JSON.stringify({ pendingCleanupAttemptId: ownership.token, pendingCleanupInFlight: true,
pendingCleanupLeaseExpiresAtMs: Date.now() + 15 * 60_000 })}::jsonb` })
.where(eq(environmentLeases.id, lease.id)).returning();
return claimed;
});
if (!pending) throw new SystemRecoveryFailure("agent_stop_unconfirmed", "recheck_evidence");
// The cleanup service owns exact-allocation termination. No process ID
// or terminal run status is substituted for its provider receipt.
const settled = await input.environmentRuntime.retryPendingSandboxTeardown({
environment: environment as unknown as Environment, lease: pending as EnvironmentLease })
.then(receipt => settleStopOnlyCleanup(db, pending, { attemptId: ownership.token, receipt }),
async () => await restoreSandboxStopAfterFailedRestart(db, pending, { attemptId: ownership.token })
?? await settleStopOnlyCleanup(db, pending, { attemptId: ownership.token }));
if (!settled || !hasRemoteTerminationReceipt(settled))
throw new SystemRecoveryFailure("agent_stop_unconfirmed", "recheck_evidence");
stoppedLease = settled;
}
const target = await resolveEnvironmentExecutionTarget({ db, companyId: input.companyId, adapterType: "paperclip_runner",
environment, leaseId: lease.id, leaseMetadata: stoppedLease.metadata,
lease: stoppedLease as EnvironmentLease, environmentRuntime: input.environmentRuntime });
if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runner || target.remoteCwd !== remoteCwd)
throw workspaceSourceBindingFailure("execution_target_invalid");
await ownership.assertHeld();
let cleanupLease = await db.transaction(async tx => {
const [fresh] = await tx.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, input.companyId),
eq(environmentLeases.id, lease.id))).for("update");
if (!fresh || !hasRemoteTerminationReceipt(fresh) || fresh.providerLeaseId !== lease.providerLeaseId)
throw new SystemRecoveryFailure("agent_stop_unconfirmed", "recheck_evidence");
const pending = await prepareStoppedSandboxForRecovery(tx as unknown as Db, fresh);
if (!pending) throw workspaceSourceBindingFailure("recovery_stop_intent_invalid");
const [claimed] = await tx.update(environmentLeases).set({ metadata: sql`${environmentLeases.metadata} || ${JSON.stringify({
pendingCleanupAttemptId: ownership.token, pendingCleanupInFlight: true, pendingCleanupLeaseExpiresAtMs: Date.now() + 15 * 60_000,
})}::jsonb` }).where(eq(environmentLeases.id, fresh.id)).returning();
return claimed;
});
let renewal = Promise.resolve();
const keepalive = setInterval(() => { renewal = renewal.then(async () => {
await ownership.assertHeld();
await db.update(environmentLeases).set({ metadata: sql`jsonb_set(${environmentLeases.metadata}, '{pendingCleanupLeaseExpiresAtMs}', ${JSON.stringify(Date.now() + 15 * 60_000)}::jsonb)` })
.where(and(eq(environmentLeases.id, lease.id), sql`${environmentLeases.metadata}->>'pendingCleanupAttemptId' = ${ownership.token}`));
}).catch(() => undefined); }, 30_000);
let stopSettled = false;
try {
const opened = await input.environmentRuntime.resumeWorkspaceRecoveryLease({ source: input,
leaseId: cleanupLease.id, environmentId: environment.id, ownership }).catch(error => {
throw sandboxDiskFull.test((error as Error | undefined)?.message ?? "")
? new SystemRecoveryFailure("source_unavailable", "use_alternative_strategy") : error;
});
const confirmed = await confirmSandboxRecoveryRestart(db, cleanupLease, { attemptId: ownership.token });
if (!confirmed) throw new SystemRecoveryFailure("agent_stop_unconfirmed", "recheck_evidence");
cleanupLease = confirmed;
if (opened?.providerLeaseId !== lease.providerLeaseId) throw workspaceSourceBindingFailure("reopened_allocation_mismatch");
if (opened.metadata?.remoteCwd !== remoteCwd) throw workspaceSourceBindingFailure("reopened_cwd_mismatch");
await inspectWorkspaceRecoverySource(db, input);
await progress({ phase: "preserve_files" });
let operationId: string;
if (source.remoteReference && source.remoteWorkspace) {
const [workspace] = await db.select().from(executionWorkspaces).where(and(
eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, source.remoteWorkspace.id),
));
if (!workspace) throw workspaceSourceBindingFailure("workspace_reference_mismatch");
const save = () => saveRemoteWorkspace({ db, workspace, runId: input.runId,
lease: cleanupLease as EnvironmentLease, target, assertOwned: ownership.assertHeld });
let saved: Awaited<ReturnType<typeof save>>;
try { saved = await save(); }
catch (error) {
if (!isSandboxDiskFullSave(error)) throw error;
let reclaimed = false;
try {
const reclaim = await reclaimInactiveNativeProviderCache({ db, companyId: input.companyId,
issueId: input.issueId, runId: input.runId, workspaceId: workspace.id, target,
assertOwned: ownership.assertHeld });
reclaimed = reclaim.changed;
cacheReclaimUnconfirmed = !reclaim.confirmed;
} catch {
cacheReclaimUnconfirmed = true;
logger.warn({ runId: input.runId }, "Retained provider cache cleanup could not be confirmed");
}
if (!reclaimed) throw error;
await ownership.assertHeld();
saved = await save();
}
await ownership.assertHeld();
const [existing] = await db.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and(
eq(workspaceOperations.companyId, input.companyId), eq(workspaceOperations.heartbeatRunId, input.runId),
eq(workspaceOperations.phase, "workspace_recovery"), eq(workspaceOperations.status, "succeeded"),
sql`${workspaceOperations.metadata}->>'leaseId' = ${lease.id}`,
sql`${workspaceOperations.metadata}->>'snapshotId' = ${saved.snapshotId}`,
)).limit(1);
const operation = existing ?? (await db.insert(workspaceOperations).values({ companyId: input.companyId,
issueId: input.issueId, heartbeatRunId: input.runId, executionWorkspaceId: workspace.id,
phase: "workspace_recovery", status: "succeeded", finishedAt: new Date(), metadata: {
recoveryActionId: input.actionId, leaseId: lease.id, providerLeaseId: lease.providerLeaseId,
snapshotId: saved.snapshotId, generation: saved.generation, sha256: saved.sha256,
storage: "remote_workspace", epoch: source.remoteReference.epoch,
} }).returning({ id: workspaceOperations.id }))[0]!;
operationId = operation.id;
} else if (reference) {
const copied = await resumeNativeWorkspaceSync({ db, runId: input.runId, target, assertOwnership: ownership.assertHeld });
if (!copied) throw new SystemRecoveryFailure("snapshot_verification_failed", "service_required");
const after = await inspectWorkspaceRecoverySource(db, input);
const finalized = after.sources.find(item => item.lease.id === lease.id)?.reference;
if (finalized?.state !== "finalized" || !finalized.finalHostSha256)
throw new SystemRecoveryFailure("snapshot_verification_failed", "service_required");
await ownership.assertHeld();
const [existing] = await db.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and(
eq(workspaceOperations.companyId, input.companyId), eq(workspaceOperations.heartbeatRunId, input.runId),
eq(workspaceOperations.phase, "workspace_recovery"), eq(workspaceOperations.status, "succeeded"),
sql`${workspaceOperations.metadata}->>'leaseId' = ${lease.id}`)).limit(1);
const copy = existing ?? (await db.insert(workspaceOperations).values({ companyId: input.companyId, issueId: input.issueId,
heartbeatRunId: input.runId, executionWorkspaceId: lease.executionWorkspaceId, phase: "workspace_recovery",
status: "succeeded", finishedAt: new Date(), metadata: { recoveryActionId: input.actionId, leaseId: lease.id,
providerLeaseId: lease.providerLeaseId, descriptorSha256: finalized.descriptorSha256,
finalHostSha256: finalized.finalHostSha256 } }).returning({ id: workspaceOperations.id }))[0];
operationId = copy.id;
} else {
operationId = await preserveWorkspaceSnapshot({ ...input, leaseId: lease.id,
providerLeaseId: lease.providerLeaseId!, target, assertHeld: ownership.assertHeld });
}
await progress({ preservationOperationIds: [...new Set([...current!.preservationOperationIds, operationId])] });
clearInterval(keepalive); await renewal;
await ownership.assertHeld();
const receipt = await input.environmentRuntime.retryPendingSandboxTeardown({ environment: environment as unknown as Environment,
lease: cleanupLease as EnvironmentLease });
const settled = await settleStopOnlyCleanup(db, cleanupLease, { attemptId: ownership.token, receipt });
stopSettled = Boolean(settled && hasRemoteTerminationReceipt(settled));
if (!stopSettled) throw new SystemRecoveryFailure("agent_stop_unconfirmed", "recheck_evidence");
} finally {
clearInterval(keepalive); await renewal;
if (!stopSettled) {
// A lost resume/export reply retains the same stop-only obligation.
// The maintenance worker must never turn it into destructive cleanup.
try {
await ownership.assertHeld();
const settled = await input.environmentRuntime.retryPendingSandboxTeardown({ environment: environment as unknown as Environment,
lease: cleanupLease as EnvironmentLease }).then(receipt => settleStopOnlyCleanup(db, cleanupLease, { attemptId: ownership.token, receipt }), () => null);
if (!settled || !hasRemoteTerminationReceipt(settled))
await restoreSandboxStopAfterFailedRestart(db, cleanupLease, { attemptId: ownership.token });
} catch { /* The persisted stop-only obligation remains authoritative. */ }
}
}
}
await ownership.assertHeld();
if (sourceUnavailable) throw new SystemRecoveryFailure("source_unavailable", "use_alternative_strategy");
const next = { ...current!, revision: current!.revision + 1, phase: "reconcile_effects" as const,
state: "complete" as const, nextAttemptAt: null };
const activity = await db.transaction(async tx => {
await notifyNativeResultCommit(tx, input);
await notifyNativeWorkspaceFinalization(tx, input);
await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionReplacement);
await notifyNativeStatusReassessment(tx, input);
await tx.insert(workspaceOperations).values({ companyId: input.companyId, issueId: input.issueId,
heartbeatRunId: input.runId, phase: "workspace_finalize", status: "succeeded", finishedAt: new Date(),
metadata: { owningService: "workspace_recovery", preservationOperationIds: next.preservationOperationIds } });
if (!await updateSystemRecovery(tx as unknown as Db, input, current!, next))
throw new Error("workspace_recovery_generation_changed");
if (inspected.action.evidence.nativeSourceRetention && !await settleRecoveredNativeSourceRetention(tx as unknown as Db, input, next.preservationOperationIds))
throw new SystemRecoveryFailure("effect_unresolved", "recheck_evidence");
// A settled restore hold returns its warm sandbox to reuse. A hold that
// fails any guard keeps its protection; the saved files still count.
await settleRecoveredWorkspaceRestoreHolds(tx as unknown as Db, input);
return persistActivity(tx as unknown as Db, { companyId: input.companyId, actorType: "system", actorId: "workspace-recovery",
action: "issue.workspace_recovered", entityType: "issue", entityId: input.issueId, runId: input.runId,
details: { recoveryActionId: input.actionId, summary: "Saved work recovered." } });
});
current = next;
try { publishActivity(activity.publication); } catch { logger.warn({ runId: input.runId }, "Recovery activity delivery deferred"); }
} catch (error) {
await ownership.assertHeld();
if (current!.state === "complete") {
logger.warn({ runId: input.runId }, "Preserved workspace activity delivery deferred");
return;
}
const classification = classifyNativeWorkspaceFailure(error);
const diskFull = !(error instanceof SystemRecoveryFailure) && !cacheReclaimUnconfirmed && isSandboxDiskFullSave(error);
const failure = error instanceof SystemRecoveryFailure ? error
: diskFull ? new SystemRecoveryFailure("export_interrupted", "user_action_required")
: classifyWorkspaceRestoreFailure(error) === "restore_invalid_metadata"
? new SystemRecoveryFailure("snapshot_verification_failed", "service_required")
: classification.code === "workspace_sync_out_unsafe_archive"
? new SystemRecoveryFailure("archive_unsafe", "service_required")
: classification.code === "workspace_sync_out_unrecoverable"
? new SystemRecoveryFailure("source_unavailable", "use_alternative_strategy")
: new SystemRecoveryFailure(current!.phase === "verify_execution" ? "provider_unavailable" : "export_interrupted", "retry_transient");
const retryAfterAt = (error as { retryAfterAt?: unknown })?.retryAfterAt;
// A generic transient code says only that the step can be retried. Keep
// why it failed in closed labels so a repeated deferral is diagnosable.
// A classified failure already names its subsystem.
const generic = !(error instanceof SystemRecoveryFailure) && failure.retryClass === "retry_transient"
&& (failure.code === "provider_unavailable" || failure.code === "export_interrupted");
const cause = diskFull ? { ...SANDBOX_DISK_FULL_RECOVERY_CAUSE } : generic ? sandboxProviderFailureLabel(error) : undefined;
const failed = failSystemRecovery(current!, failure, {
...(typeof retryAfterAt === "number" && Number.isFinite(retryAfterAt) ? { retryAfterAt } : {}),
...(cause ? { cause } : {}),
});
const recorded = await updateSystemRecovery(db, input, current!, failed);
// A permanent refusal of the recorded source repeats on every retry.
// Continue from the last save in this same pass, under this ownership.
if (recorded && failed.state === "exhausted" && isUnverifiableSourceFailure(failed.failure)) {
const fallback = await continueFromLastSavedWorkspace({ ...input, ownership }).catch(fallbackError => {
logger.warn({ err: fallbackError, runId: input.runId }, "Continuing from the last save was deferred; the hold remains");
return { kind: "ineligible" as const, reason: "fallback_failed" };
});
if (fallback.kind === "ineligible") await recordRetainedSourceFallbackDeferral(db, input, fallback.reason);
}
if (failure.retryClass === "recheck_evidence" && ["agent_stop_unconfirmed", "copyback_stop_unconfirmed"].includes(failure.code))
await recheckStoppedWorkspaceRecovery(db, input, false);
logger.warn({ err: error, runId: input.runId, failureCode: failure.code, failureCause: cause?.label, phase: current!.phase }, "Workspace preservation deferred; source retained");
} finally { clearInterval(observer); await observing; }
};
const owned = await withNativeWorkspaceFinalizationOwnership({ db, companyId: input.companyId, runId: input.runId,
retainedRepairActionId: input.actionId }, runAttempt);
if (!owned.acquired && owned.reason === "owner_unverified") {
await holdForUnverifiedCopybackOwner(db, input);
await recheckStoppedWorkspaceRecovery(db, input, false);
}
}
/**
* A copyback owner that a restart killed can never confirm its own stop, and
* Retry cannot release its receipt (RC-50). Rechecking that receipt is not a
* recovery attempt: the bounded recovery window still applies, but no attempt
* is spent. Record the exact receipt so the board can confirm the stop through
* `resumeNativeWorkspaceAfterOwnerStop`, and so Retry names this wait.
*/
async function holdForUnverifiedCopybackOwner(db: Db, input: Source) {
const current = await loadSystemRecovery(db, input);
if (!current || current.state === "complete" || current.state === "exhausted") return;
const next = failSystemRecovery(current, new SystemRecoveryFailure("copyback_stop_unconfirmed", "recheck_evidence"));
if (!await updateSystemRecovery(db, input, current, next)) return;
await recordCopybackOwnerHold(db, input, next.generationId);
}
/** Bookkeeping only: names the exact receipt for the board. Grants no attempt. */
async function recordCopybackOwnerHold(db: Db, input: Source, generationId: string) {
const [source] = await db.select({ owner: sql<unknown>`${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceFinalizationOwner'` })
.from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId)));
const owner = source?.owner;
if (!owner || typeof owner !== "object" || typeof (owner as { token?: unknown }).token !== "string") return;
const target = and(eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.id, input.actionId),
eq(issueRecoveryActions.sourceIssueId, input.issueId),
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,generationId}' = ${generationId}`,
sql`${issueRecoveryActions.evidence}#>'{workspaceOwnerRecovery,owner}' is distinct from ${JSON.stringify(owner)}::jsonb`);
await db.transaction(async tx => {
const [action] = await tx.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(target).for("update");
if (!action) return;
// Wake the ownership worker (on any server) before the hold it announces,
// so a quiet worker judges this owner and can wake the recheck.
await notifyDeliveryWork(tx, DELIVERY_QUEUES.ownershipInvariantReport,
JSON.stringify({ companyId: input.companyId, runId: input.runId, actionId: input.actionId }));
await tx.update(issueRecoveryActions).set({
// A confirmation names one receipt; a newer unverified owner needs its own.
evidence: sql`(${issueRecoveryActions.evidence} - 'workspaceOwnerStop') || jsonb_build_object('workspaceOwnerRecovery',
jsonb_build_object('state', 'operator_required', 'owner', ${JSON.stringify(owner)}::jsonb))`,
updatedAt: new Date(),
}).where(target);
});
}
const cursors = new WeakMap<Db, string>();
const workInFlight = new WeakMap<Db, Set<string>>();
const unfinishedCursors = new WeakMap<Db, string>();
/** Destruction is a later cleanup obligation, authorized only by a matching
* saved-file receipt and the original recorded disposition. */
export async function scheduleRecoveredWorkspaceCleanup(db: Db, run: typeof heartbeatRuns.$inferSelect) {
if (!await hasVerifiedWorkspaceRepair(db, run)) return;
const legacy = readNativeWorkspaceSyncReference(run.runnerProfileJson?.nativeWorkspaceSync);
const parsedRemote = remoteWorkspaceRunReferenceSchema.safeParse(run.runnerProfileJson?.remoteWorkspace);
const remote = parsedRemote.success && parsedRemote.data.state === "saved" ? parsedRemote.data : null;
const ref = legacy ?? remote;
const disposition = legacy?.resourceDisposition ?? (remote ? run.runnerProfileJson?.nativeResourceDisposition : undefined);
if (!ref || disposition !== "destroy") return;
const [lease] = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.id, ref.leaseId)));
if (!lease || !hasRemoteTerminationReceipt(lease)) return;
const [other] = await db.select({ id: environmentLeases.id }).from(environmentLeases).where(and(
ne(environmentLeases.id, lease.id), eq(environmentLeases.provider, lease.provider!), eq(environmentLeases.providerLeaseId, lease.providerLeaseId!),
)).limit(1);
if (other) return;
await writeEnvironmentCleanup(db, tx => tx.update(environmentLeases).set({ status: "pending_cleanup", cleanupStatus: "failed", leasePolicy: "ephemeral",
updatedAt: new Date(), metadata: sql`(${environmentLeases.metadata} - 'workspaceRestoreRecovery' - 'remoteExecutionTermination') ||
${JSON.stringify({ workspaceRecoveryCleanup: { runId: run.id,
// Keep the verified stop: successors wait on the stop, not on the destroy.
stopReceipt: lease.metadata?.remoteExecutionTermination, ...(legacy ? { descriptorSha256: legacy.descriptorSha256 }
: { snapshotId: remote!.snapshotId, epoch: remote!.epoch }) },
pendingCleanupInFlight: false, pendingCleanupRetryAfterMs: 0, pendingCleanupRetryAttempts: 0, pendingCleanupLeaseExpiresAtMs: 0 })}::jsonb`,
}).where(and(eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.id, ref.leaseId),
eq(environmentLeases.heartbeatRunId, run.id), eq(environmentLeases.providerLeaseId, ref.providerLeaseId),
eq(environmentLeases.status, "released"), eq(environmentLeases.cleanupStatus, "success"),
sql`${environmentLeases.metadata} = ${JSON.stringify(lease.metadata)}::jsonb`,
sql`${environmentLeases.metadata}->>'workspaceRecoveryCleanup' is null`)), { companyId: run.companyId, runId: run.id });
}
/** `dueOnly` narrows the obligations to exhausted holds whose re-arm or
* fallback is due now. Cleanup, interrupted preparation and saved input that
* waits stay with the obligation pass. */
export function retainedWorkspaceCandidates(db: Db, cursor?: string, unfinishedOnly = false, obligationsOnly = false, inFlightRunIds: string[] = [],
dueOnly = false) {
const cleanup = sql`(((${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceSync,resourceDisposition}' = 'destroy'
and ${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceSync,leaseId}' is not null)
or (${heartbeatRuns.runnerProfileJson}->>'nativeResourceDisposition' = 'destroy'
and ${heartbeatRuns.runnerProfileJson}#>>'{remoteWorkspace,schema}' = 'paperclip.remote-workspace-run.v1'
and ${heartbeatRuns.runnerProfileJson}#>>'{remoteWorkspace,state}' = 'saved')) and exists (
select 1 from ${environmentLeases} where ${environmentLeases.companyId} = ${heartbeatRuns.companyId}
and ${environmentLeases.heartbeatRunId} = ${heartbeatRuns.id}
and ${environmentLeases.id}::text = coalesce(${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceSync,leaseId}',
${heartbeatRuns.runnerProfileJson}#>>'{remoteWorkspace,leaseId}')
and ${environmentLeases.status} = 'released' and ${environmentLeases.cleanupStatus} = 'success'
and ${environmentLeases.metadata}->'workspaceRecoveryCleanup' is null))`;
const waitingInput = sql`exists (select 1 from ${agentWakeupRequests}
where ${agentWakeupRequests.companyId} = ${issueRecoveryActions.companyId}
and ${agentWakeupRequests.status} = 'deferred_issue_execution' and ${agentWakeupRequests.requestedByActorType} = 'user'
and ${agentWakeupRequests.payload}->>'issueId' = ${issueRecoveryActions.sourceIssueId}::text)`;
const interruptedPreparation = sql`(${heartbeatRuns.runtimeMode} = 'legacy'
and (${heartbeatRuns.status} = 'interrupted' or (${heartbeatRuns.status} = 'failed'
and ${heartbeatRuns.errorCode} = 'process_lost'
and ${heartbeatRuns.resultJson}#>>'{executionRecovery,kind}' = 'bootstrap'
and ${heartbeatRuns.resultJson}#>'{executionRecovery,providerWorkStarted}' = 'false'::jsonb))
and ${heartbeatRuns.runtimeModeResolvedAt} is null and ${heartbeatRuns.executionStage} = 'preparing'
and ${heartbeatRuns.runnerProfileJson}#>>'{adapterDispatch,adapterType}' = 'paperclip_runner'
and (${issueRecoveryActions.evidence}#>>'{automaticRecovery,replay}' = 'blocked'
or (${issueRecoveryActions.evidence}#>>'{automaticRecovery,replay}' = 'allowed'
and ${issueRecoveryActions.evidence}->>'bootstrapPreparationSettledAt' is not null
and coalesce(${issueRecoveryActions.evidence}->>'bootstrapPreparationContinuation', 'pending') in ('pending', 'deferred')))
and exists (select 1 from ${issues} where ${issues.companyId} = ${heartbeatRuns.companyId}
and ${issues.id} = ${issueRecoveryActions.sourceIssueId} and ${issues.assigneeAgentId} = ${heartbeatRuns.agentId}
and ${issues.status} in ('todo', 'in_progress', 'in_review', 'blocked')
and ${issues.executionRunId} is null and ${issues.checkoutRunId} is null
and (${issues.status} <> 'blocked' or
${issueRecoveryActions.evidence}#>>'{nativeBootstrapFailureBlock,statusVersion}' = ${issues.statusVersion}::text)))`;
const now = Date.now();
const dueHold = sql`(${retainedSourceFallbackDueCondition(now)} or ${stopProofRearmDueCondition(now)} or ${capturePolicyRearmDueCondition(now)})`;
return db.select({ action: issueRecoveryActions, run: heartbeatRuns }).from(issueRecoveryActions)
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.companyId, issueRecoveryActions.companyId), eq(heartbeatRuns.id,
sql`case when ${issueRecoveryActions.evidence}->>'runId' ~* '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'
then (${issueRecoveryActions.evidence}->>'runId')::uuid end`)))
.where(and(inArray(heartbeatRuns.status, terminal),
workspaceRecoverySourceEligible(true),
unfinishedOnly ? sql`(${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('scheduled', 'running', 'awaiting_evidence')
or (${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' = 'complete'
and ${issueRecoveryActions.evidence}->>'bootstrapPreparationSettledAt' is not null
and coalesce(${issueRecoveryActions.evidence}->>'bootstrapPreparationContinuation', 'pending') = 'pending'))` : undefined,
obligationsOnly ? sql`(${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('scheduled', 'running', 'awaiting_evidence')
or (${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' = 'complete' and (${cleanup} or ${interruptedPreparation})) or ${waitingInput}
or ${dueHold})` : undefined,
dueOnly ? dueHold : undefined,
sql`(${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('scheduled', 'running', 'awaiting_evidence', 'complete', 'exhausted')
and (${issueRecoveryActions.evidence}#>'{systemRecovery,nextAttemptAt}' = 'null'::jsonb
or ${issueRecoveryActions.evidence}#>'{systemRecovery,nextAttemptAt}' <= to_jsonb(${now}::bigint)))`,
sql`(${issueRecoveryActions.evidence}#>>'{workspaceRecovery,status}' is distinct from 'recovered' or
${cleanup} or ${waitingInput} or ${interruptedPreparation})`,
cursor ? gt(issueRecoveryActions.id, cursor) : undefined,
inFlightRunIds.length ? notInArray(heartbeatRuns.id, inFlightRunIds) : undefined,
)).orderBy(asc(issueRecoveryActions.id)).limit(5);
}
/**
* The worker's lane positions, kept in memory: a restart starts every lane
* from its first candidate, so no persisted state or migration is involved.
* `cursor`, `complete` and `restart` belong to the obligation pass; `dueCursor`
* to the due lane; `firstLane` names the lane that has priority in the next run.
*/
export type PassiveSweep = { cursor?: string; complete?: boolean; restart?: boolean; dueCursor?: string; firstLane?: number };
type CandidateRow = Awaited<ReturnType<typeof retainedWorkspaceCandidates>>[number];
type CandidateLane = { rows: CandidateRow[]; index: number; advance: (actionId: string) => void };
/** Bounded discovery shares the periodic recovery lifecycle. A repaired source
* still owes cleanup even without saved input or after task reconciliation.
*
* Candidates come from lanes, each an ordered page that a cursor advances:
* unfinished saves; for the worker, exhausted holds whose re-arm or fallback
* is due; and the obligation pass (cleanup, interrupted preparation, saved
* input that waits, and the same due holds). Each lane with work takes one
* free slot before any lane takes a second, and the worker rotates which lane
* goes first, so no kind of load (a save another server owns, a long local
* export, hundreds of waiting holds) starves another. */
export async function reconcileRetainedWorkspaces(db: Db, runtime: EnvironmentRuntimeService,
onRecovered: (run: typeof heartbeatRuns.$inferSelect) => Promise<void>, obligationsOnly = false,
onFinished?: (continueSweep: boolean) => void, passiveSweep?: PassiveSweep) {
await adoptHistoricalWorkspaceRecoveries(db);
let pending = workInFlight.get(db);
if (!pending) { pending = new Set(); workInFlight.set(db, pending); }
if (pending.size >= 2) return;
// Locally running exports already own their slot. Exclude them before
// pagination so a page containing only held work cannot prevent wrapping.
const inFlightRunIds = [...pending];
let unfinishedRows = await retainedWorkspaceCandidates(db, unfinishedCursors.get(db), true, false, inFlightRunIds);
if (!unfinishedRows.length) unfinishedRows = await retainedWorkspaceCandidates(db, undefined, true, false, inFlightRunIds);
const unfinished: CandidateLane = { rows: unfinishedRows, index: 0, advance: id => unfinishedCursors.set(db, id) };
let rows = await retainedWorkspaceCandidates(db, passiveSweep ? passiveSweep.cursor : cursors.get(db), false, obligationsOnly, inFlightRunIds);
if (passiveSweep && !rows.length && passiveSweep.restart) {
Object.assign(passiveSweep, { cursor: undefined, restart: false });
rows = await retainedWorkspaceCandidates(db, undefined, false, obligationsOnly, inFlightRunIds);
}
if (passiveSweep) passiveSweep.complete = !rows.length;
else if (!rows.length) rows = await retainedWorkspaceCandidates(db, undefined, false, obligationsOnly, inFlightRunIds);
const sweep: CandidateLane = { rows, index: 0,
advance: id => { if (passiveSweep) passiveSweep.cursor = id; else cursors.set(db, id); } };
const lanes = [unfinished];
if (passiveSweep) {
// The due lane wraps at its tail like the unfinished lane. A hold that
// becomes due by time alone (a lease expiry, a re-arm backoff) is reached
// even after the obligation pass completed with no committed change.
let dueRows = await retainedWorkspaceCandidates(db, passiveSweep.dueCursor, false, true, inFlightRunIds, true);
if (!dueRows.length && passiveSweep.dueCursor)
dueRows = await retainedWorkspaceCandidates(db, undefined, false, true, inFlightRunIds, true);
const due: CandidateLane = { rows: dueRows, index: 0, advance: id => { passiveSweep.dueCursor = id; } };
lanes.push(due, sweep);
// Priority rotates across all three lanes, so each lane is first in one
// run of three. With one free slot (a long local export holds the other),
// a save that keeps finding its run's lock busy cannot take it every run.
const first = (passiveSweep.firstLane ?? 0) % lanes.length;
lanes.push(...lanes.splice(0, first));
passiveSweep.firstLane = (first + 1) % lanes.length;
} else lanes.push(sweep);
const work: Promise<void>[] = [];
// Several actions of one run (an export hold and a later no-replay hold)
// run one after another in that run's job. Skipping the later one would
// starve it while the earlier one stays selectable (for example while saved
// input waits for the later one to settle).
const groups = new Map<string, { run: typeof heartbeatRuns.$inferSelect; actions: (typeof issueRecoveryActions.$inferSelect)[] }>();
const join = (lane: CandidateLane) => {
for (; lane.index < lane.rows.length; lane.index += 1) {
const { action, run } = lane.rows[lane.index]!;
const group = groups.get(run.id);
if (!group) return lane.rows[lane.index]!;
if (!group.actions.some(item => item.id === action.id)) group.actions.push(action);
lane.advance(action.id);
}
return undefined;
};
// Takes the lane's next run into a free slot. Actions of a run already taken
// (by any lane) join its job without a slot.
const take = (lane: CandidateLane) => {
const next = join(lane);
if (!next || pending.size >= 2) return false;
lane.index += 1;
lane.advance(next.action.id);
pending.add(next.run.id);
groups.set(next.run.id, { run: next.run, actions: [next.action] });
return true;
};
for (let taken = true; taken && pending.size < 2;) {
taken = false;
for (const lane of lanes) if (pending.size < 2 && take(lane)) taken = true;
}
for (const lane of lanes) join(lane);
const hasUnvisited = lanes.some(lane => lane.index < lane.rows.length);
for (const { run, actions } of groups.values()) {
const finish = beginIdleTrackedWork();
const physical = actions.some(action => ["scheduled", "running", "awaiting_evidence"].includes(readSystemRecovery(action.evidence)?.state ?? ""));
work.push((async () => { try {
for (const action of actions) await recoverRunAction(action, run);
} finally { pending.delete(run.id); finish(); onFinished?.(physical || hasUnvisited); } })());
}
return work;
async function recoverRunAction(action: typeof issueRecoveryActions.$inferSelect, run: typeof heartbeatRuns.$inferSelect) {
try {
await recoverRetainedWorkspace({ db, environmentRuntime: runtime, companyId: action.companyId, issueId: action.sourceIssueId, runId: run.id, actionId: action.id });
const [updated] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action.id));
const recovered = readWorkspaceRecovery(updated?.evidence ?? null)?.status === "recovered";
const exhausted = readSystemRecovery(updated?.evidence)?.state === "exhausted";
if (recovered || exhausted) {
const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
if (current) {
if (recovered) {
await resumePreservedResultFinalization(db, current.companyId, current.id);
await scheduleRecoveredWorkspaceCleanup(db, current);
}
// Only normal dispatch can consume existing authorized input. An
// exhausted export permits evaluation of an isolated continuation,
// never another export attempt or a replay of an old assignment.
await onRecovered(current);
}
}
} catch { logger.warn({ runId: run.id }, "Retained workspace remains pending"); }
}
}