mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-21 02:43:25 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Restart recovery must continue a task after a server is killed. A person must never need to press Retry. > - When Paperclip cannot resume a cut-off run exactly, it records a "fresh continuation". For a remote workspace, the continuation abandons the old run's workspace epoch and restores the last save into a new root. It does not apply the old run's unsaved files. > - Workspace recovery can try to save the old run's files after that. The epoch is already abandoned, so the source is "unavailable". Recovery then wrote an active hold that asked a person to press Retry to continue in a new workspace, which the continuation had already arranged. > - That hold made the dispatch gate withhold every queued run of the task, and the task rested on a Retry banner. > - This pull request settles that hold into the recorded continuation at the moment workspace recovery would leave it exhausted. Dispatch and continuation delivery are unchanged. > - The benefit is that the queued continuation starts automatically after a hard kill, with no Retry banner. ## Linked Issues or Issue Description **What happened?** A restart-chaos release-swap wave hard-killed the old server. One task (of 6) then rested on a Retry banner for about 5 minutes. The other 5 tasks passed. The sequence for the failed task: 1. A run started on the incoming server. The wave's injected database delay held its process-identity write for 480 s, so its runner attach timed out. The run saved a same-run resume. 2. Restart recovery found a newer queued run for the task (a child-completion wake). It treated the same-run resume as superseded, and the run ended `failed` with `native_restart_recovery_blocked`. 3. The dispatch gate withheld the queued run (`execution_reconciliation_required`). In the same transaction, it recorded a fresh continuation with `remote_epoch` isolation, which abandoned the old run's remote workspace epoch. 4. 4 s later, workspace preservation for the old run found the abandoned epoch (`source_unavailable`, `use_alternative_strategy`). It left an active hold: "Paperclip cannot recover the unsaved files from the previous run ... Retry continues the task in a new workspace". The retained-source fallback also refused it later, with `workspace_authority_changed`, because the continuation owned the epoch. 5. The gate withheld both bounded retries because of that hold. The retries were spent, and the task showed a Retry banner. 6. A person pressed Retry. Retry started a run in a new workspace, which the recorded continuation had already decided to do. **Expected behavior** The queued continuation starts automatically. No Retry banner shows. **Steps to reproduce** 1. Run the restart-chaos K1 kill variant: 2 release swaps, and the old server is hard-killed 120 s after SIGTERM. 2. Use an OpenCode task on a remote sandbox whose run starts just before the second swap and that gets a child-completion wake while it runs. 3. Watch the task after the kill. Workspace recovery writes the unsaved-files hold, the gate cancels each queued run with `execution_reconciliation_required`, and the task rests on a Retry banner. `server/src/__tests__/heartbeat-isolated-continuation-hold.test.ts` reproduces the database state at step 5. **Paperclip version or commit** `master` at25b5cad7f9(wave) andc41145e72a(base of this branch). ## What Changed - `server/src/services/fresh-continuation-fallback.ts`: new `settleHoldIntoAbandonedEpochContinuation()`. It settles one workspace-recovery hold with the existing `settleLateHolds()` when all of these are true: - the hold is active or escalated, and its saved-work recovery is exhausted; - no person owns it, it has no human-gate evidence, and it is not an unsafe-archive hold; - a fresh-continuation record for the same run has `remote_epoch` isolation and delivery `pending`, `delivered`, `delegated` or `not_requested`; - durable proof shows the run's prepared epoch was abandoned (`isAbandonedPreparedRemoteWorkspace`: an abandoned-epoch receipt for this run, epoch and location, and an authority that moved past it). The hold becomes `resolved` / `restored` (`continued_fresh_from_last_saved`) and is coalesced into the continuation record. Its files are not applied. - `server/src/services/workspace-recovery.ts` calls it at two points: when an attempt records an exhausted failure, and when a recovery pass starts on an already exhausted hold (holds written before this change). - `server/src/services/recovery/human-gate-evidence.ts`: one shared list of human-gate evidence keys (`executionReconciliation`, `OWNER_GONE_ATTESTATION_KEY`, `explicitUserContinuation`), with a SQL helper and a TypeScript helper. Both treat a present key as a person's decision, whatever its value. `isHumanGate`, `completed-turn-cleanup-hold-settlement.ts` and `interrupted-preparation-hold-settlement.ts` use them. - New tests: `server/src/__tests__/heartbeat-isolated-continuation-hold.test.ts`. The dispatch gate, the execution-blocker predicate, the wake-queue lock release, and continuation delivery are unchanged from `master`. An earlier revision of this pull request changed them. That approach was removed because each review round found another dispatch case. `fresh_isolated_workspace` continuations are out of scope on purpose. They do not abandon a remote epoch, so they do not cause this failure. Only a run that carries the isolation marker is protected from the old local directory, so a late hold for such a run keeps blocking. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/heartbeat-isolated-continuation-hold.test.ts` gives 20 of 20 passing. - Recovery settles the late hold for each `remote_epoch` delivery state (`pending`, `delivered`, `delegated`, `not_requested`). The blocker is then empty, and the hold is coalesced into the continuation. - Assembled heartbeat test: after recovery settles the hold, the queued run is the only run that starts. The old run stays failed. With a hold that a person owns, the gate still withholds the queued run. - The hold stays for each of these cases: no continuation, a `remote_epoch` continuation while the run still owns its epoch, a run with no remote workspace, `fresh_isolated_workspace`, `writer_ended`, `invalidated`, `coalesced`, a hold that a person owns, an unsafe archive, each human-gate key, and a null-valued human-gate key. - Red/green: with the two `workspace-recovery.ts` calls disabled, 5 of 20 fail; without the abandoned-epoch check, the 2 epoch cases fail: the 4 settle cases and the assembled start test. - A focused regression run of the server test files for the blocker, fresh continuations, workspace recovery, both hold settlements, owner-gone attestation, and the run-dispatch and wake-queue adapters. - `npx tsc --noEmit -p server/tsconfig.json` passes. ## Risks - Only exhausted, system-owned workspace-recovery holds of a run whose remote epoch a recorded continuation replaced are settled. All other holds keep blocking. - After settlement, workspace recovery stops for that hold. Its files were already unreachable, because the epoch was abandoned. - A human-gate key with a null or false value now counts as a person's decision in TypeScript too. Before, TypeScript ignored it and SQL counted it. This makes the two sides agree and is more conservative. - No schema change, no migration, and no Runner change. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in Claude Code with tool use (shell, file edit, code search) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
838 lines
61 KiB
TypeScript
838 lines
61 KiB
TypeScript
import { classifyWorkspaceRestoreFailure } from "@paperclipai/adapter-utils/workspace-restore-merge";
|
|
import { createHash } from "node:crypto";
|
|
import { adapterExecutionControls } from "./adapter-execution-control.js";
|
|
import { isAcknowledgedNativeStop } from "./acknowledged-native-stop.js";
|
|
import { notifyNativeResultCommit } from "./native-result-commit-notifications.js";
|
|
import { notifyNativeWorkspaceFinalization } from "./native-workspace-finalization-notifications.js";
|
|
import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js";
|
|
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
|
import { beginIdleTrackedWork } from "./task-admission.js";
|
|
import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
|
import { resumePreservedResultFinalization } from "./preserved-result-finalization.js";
|
|
import { reclaimInactiveNativeProviderCache } from "./native-runtime/retained-provider-cache.js";
|
|
import { REMOTE_WORKSPACE_SANDBOX_DISK_FULL, sandboxProviderFailureLabel } from "./remote-workspace-transport.js";
|
|
import { and, asc, eq, gt, inArray, ne, notInArray, or, sql } from "drizzle-orm";
|
|
import { agentWakeupRequests, environmentLeases, executionWorkspaces, heartbeatRuns, issues, issueRecoveryActions, nativeRunFinalizations, workspaceOperations, type Db } from "@paperclipai/db";
|
|
import { remoteWorkspaceRunReferenceSchema, type Environment, type EnvironmentLease } from "@paperclipai/shared";
|
|
import { conflict } from "../errors.js";
|
|
import { logger } from "../middleware/logger.js";
|
|
import type { EnvironmentRuntimeService } from "./environment-runtime.js";
|
|
import { resolveEnvironmentExecutionTarget } from "./environment-execution-target.js";
|
|
import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
|
|
import { confirmSandboxRecoveryRestart, hasStopOnlyCleanup, prepareHistoricalSandboxStopVerification, prepareSandboxStopAndRetain, prepareStoppedSandboxForRecovery, restoreSandboxStopAfterFailedRestart, settleAbsentSandbox, settleStopOnlyCleanup } from "./sandbox-stop-and-retain.js";
|
|
import { requestSandboxAbsenceProof } from "./sandbox-absence-proof.js";
|
|
import { persistActivity, publishActivity } from "./activity-log.js";
|
|
import { hasRequiredWorkspaceRecovery, LEGACY_WORKSPACE_RECOVERY_SCHEMA } from "./workspace-restore-recovery-state.js";
|
|
import { TRANSIENT_WORKSPACE_PREPARATION_CODES } from "./failed-preparation-retry.js";
|
|
import { hasVerifiedWorkspaceRepair, readWorkspaceRecovery, type WorkspaceRecovery } from "./workspace-recovery-receipt.js";
|
|
import { withNativeWorkspaceFinalizationOwnership } from "./native-runtime/native-workspace-finalization-ownership.js";
|
|
import { readNativeWorkspaceSyncReference, readNativeWorkspaceSyncLocalCwd, resumeNativeWorkspaceSync } from "./native-runtime/native-workspace-sync.js";
|
|
import { classifyNativeWorkspaceFailure } from "./native-runtime/native-workspace-failure.js";
|
|
|
|
import { handOffExhaustedSystemRecovery, adoptSystemRecovery, loadSystemRecovery, updateSystemRecovery } from "./system-recovery.js";
|
|
import { SYSTEM_RECOVERY_MAX_RETRIES, createSystemRecovery, failSystemRecovery, readSystemRecovery, reserveSystemRecoveryAttempt, SystemRecoveryFailure, type SystemRecoveryState } from "./system-recovery-state.js";
|
|
import { inspectWorkspaceRecoverySource, workspaceRecoverySourceEligible, workspaceSourceBindingFailure } from "./workspace-recovery-source.js";
|
|
import { preserveWorkspaceSnapshot } from "./workspace-recovery-snapshot.js";
|
|
import { adoptHistoricalWorkspaceRecoveries } from "./workspace-recovery-backfill.js";
|
|
import { canAdoptNativeSourceRetention, retainNativeRecoverySource, settleRecoveredNativeSourceRetention } from "./native-runtime/native-source-retention.js";
|
|
import { saveRemoteWorkspace } from "./remote-workspaces.js";
|
|
import { continueFromLastSavedWorkspace, isUnverifiableSourceFailure, recordRetainedSourceFallbackDeferral, settleExhaustedRetainedHold } from "./workspace-recovery-fallback.js";
|
|
import { capturePolicyRearmDueCondition, retainedSourceFallbackDueCondition, stopProofRearmDueCondition } from "./workspace-recovery-fallback-discovery.js";
|
|
import { rearmExhaustedCapturePolicy } from "./workspace-recovery-capture-policy.js";
|
|
import { rearmExhaustedStopProof } from "./workspace-recovery-stop-proof.js";
|
|
import { settleRecoveredWorkspaceRestoreHolds } from "./workspace-restore-hold-settlement.js";
|
|
import { expediteFreshContinuationChecks, settleHoldIntoAbandonedEpochContinuation } from "./fresh-continuation-fallback.js";
|
|
import { reusableSandboxNotKeptAfterReplacementSql } from "./reusable-sandbox-kept-marker.js";
|
|
|
|
const exportCauses = ["native_workspace_sync_out_unrecoverable", "native_workspace_sync_out_retry_exhausted"];
|
|
const terminal = ["failed", "interrupted", "timed_out", "cancelled", "succeeded"];
|
|
const sandboxDiskFull = /No space left on device|Disk quota exceeded/i;
|
|
export const SANDBOX_DISK_FULL_RECOVERY_CAUSE = { label: "sandbox_disk_full", code: "ENOSPC", status: null } as const;
|
|
const isSandboxDiskFullSave = (error: unknown) => error instanceof Error && error.message === REMOTE_WORKSPACE_SANDBOX_DISK_FULL;
|
|
type Source = { companyId: string; issueId: string; runId: string; actionId: string };
|
|
|
|
type StopEvidenceSource = Omit<Source, "actionId"> & { actionId?: string };
|
|
|
|
async function recheckStoppedWorkspaceRecovery(db: Db, source: StopEvidenceSource, observe: boolean) {
|
|
return db.transaction(async tx => {
|
|
const [issue] = await tx.select({ id: issues.id }).from(issues).where(and(
|
|
eq(issues.id, source.issueId), eq(issues.companyId, source.companyId))).for("update");
|
|
const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, source.runId),
|
|
eq(heartbeatRuns.companyId, source.companyId))).for("update");
|
|
if (!issue || !run || run.runtimeMode !== "native" || run.nativeIssueId !== issue.id ||
|
|
!isAcknowledgedNativeStop(run) || adapterExecutionControls.has(run.id)) return 0;
|
|
const cancellation = run.resultJson?.executionCancellation as { state?: unknown; proof?: unknown; acknowledgedAt?: unknown } | undefined;
|
|
if (cancellation?.state !== "acknowledged" || cancellation.proof !== "provider_termination_receipt") return 0;
|
|
const [coordinator] = await tx.select({ phase: nativeRunFinalizations.phase, leaseOwner: nativeRunFinalizations.leaseOwner })
|
|
.from(nativeRunFinalizations).where(and(eq(nativeRunFinalizations.companyId, source.companyId),
|
|
eq(nativeRunFinalizations.runId, source.runId), eq(nativeRunFinalizations.issueId, source.issueId)));
|
|
if (!coordinator || coordinator.leaseOwner || !["terminal_failure", "applied"].includes(coordinator.phase)) return 0;
|
|
const leases = await tx.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, source.companyId),
|
|
eq(environmentLeases.heartbeatRunId, source.runId)));
|
|
if (!leases.length || leases.some(lease => !lease.provider || lease.provider === "local" || !hasRemoteTerminationReceipt(lease))) return 0;
|
|
const intent = run.resultJson!.nativeCancellation as Record<string, unknown>;
|
|
const fingerprint = createHash("sha256").update(JSON.stringify({ companyId: source.companyId, issueId: source.issueId,
|
|
runId: source.runId, intentAuditId: intent.intentAuditId, acknowledgementAuditId: intent.acknowledgementAuditId,
|
|
acknowledgedAt: cancellation.acknowledgedAt, leases: leases.sort((a, b) => a.id.localeCompare(b.id)).map(lease => {
|
|
const receipt = lease.metadata!.remoteExecutionTermination as Record<string, unknown>;
|
|
return { leaseId: lease.id, provider: lease.provider, providerLeaseId: lease.providerLeaseId,
|
|
state: receipt.state, confirmedAt: receipt.confirmedAt };
|
|
}) })).digest("hex");
|
|
const actions = await tx.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, source.companyId),
|
|
eq(issueRecoveryActions.sourceIssueId, source.issueId), or(inArray(issueRecoveryActions.status, ["active", "escalated"]), and(
|
|
eq(issueRecoveryActions.status, "resolved"), sql`${issueRecoveryActions.evidence}#>>'{automaticRecovery,policy}' = 'preserve_without_replay_v1'`,
|
|
sql`${issueRecoveryActions.evidence}#>>'{automaticRecovery,replay}' = 'blocked'`,
|
|
sql`${issueRecoveryActions.evidence}#>>'{automaticRecovery,runId}' = ${source.runId}`)),
|
|
sql`${issueRecoveryActions.evidence}->>'runId' = ${source.runId}`,
|
|
source.actionId ? eq(issueRecoveryActions.id, source.actionId) : undefined)).for("update");
|
|
let changed = 0;
|
|
for (const action of actions) {
|
|
const current = readSystemRecovery(action.evidence), workspace = readWorkspaceRecovery(action.evidence), now = Date.now();
|
|
if (!current || current.sourceRunId !== source.runId || current.phase !== "verify_execution" ||
|
|
!workspace || workspace.runId !== source.runId || ["complete", "exhausted"].includes(current.state) || current.retriesUsed >= SYSTEM_RECOVERY_MAX_RETRIES ||
|
|
(current.deadlineAt !== null && now >= current.deadlineAt)) continue;
|
|
const stopWait = current.failure?.retryClass === "recheck_evidence" &&
|
|
["agent_stop_unconfirmed", "copyback_stop_unconfirmed"].includes(current.failure.code);
|
|
if (current.state !== "running" && current.failure && !stopWait) continue;
|
|
const prior = action.evidence.workspaceStopEvidence as Record<string, unknown> | undefined;
|
|
const same = prior?.schema === "paperclip.workspace-stop-evidence.v1" && prior.sourceRunId === source.runId &&
|
|
prior.generationId === current.generationId && prior.fingerprint === fingerprint;
|
|
if ((same && prior!.consumed === true) || (!observe && (!same || prior!.consumed !== false))) continue;
|
|
const due = Math.max(now, current.providerCooldownAt ?? 0);
|
|
const advance = current.state === "awaiting_evidence" && stopWait && current.attempts > 0 &&
|
|
current.deadlineAt !== null && due < current.deadlineAt && current.nextAttemptAt !== null && current.nextAttemptAt > due;
|
|
const consumed = current.state !== "running";
|
|
if (!advance && same && prior!.consumed === consumed) continue;
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.workspaceRecovery);
|
|
if (advance && !await updateSystemRecovery(tx as unknown as Db, { ...source, actionId: action.id }, current,
|
|
{ ...current, revision: current.revision + 1, nextAttemptAt: due })) continue;
|
|
const marker = { schema: "paperclip.workspace-stop-evidence.v1", sourceRunId: source.runId,
|
|
generationId: current.generationId, fingerprint, consumed };
|
|
const recorded = await tx.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence} ||
|
|
${JSON.stringify({ workspaceStopEvidence: marker })}::jsonb`, updatedAt: new Date() }).where(and(
|
|
eq(issueRecoveryActions.companyId, source.companyId), eq(issueRecoveryActions.id, action.id),
|
|
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,generationId}' = ${current.generationId}`,
|
|
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,revision}' = ${String(current.revision + (advance ? 1 : 0))}`))
|
|
.returning({ id: issueRecoveryActions.id });
|
|
if (!recorded.length) throw new Error("workspace_stop_evidence_changed");
|
|
changed += 1;
|
|
}
|
|
return changed;
|
|
});
|
|
}
|
|
|
|
export function observeStoppedWorkspaceRecovery(db: Db, source: StopEvidenceSource) {
|
|
return recheckStoppedWorkspaceRecovery(db, source, true);
|
|
}
|
|
|
|
/**
|
|
* Ownership-reconciler settle for the RC-50 copyback-owner hold. The recorded
|
|
* owner was proven gone by the shared prover, so the waiting recheck is due
|
|
* now instead of after its backoff. It grants no attempt and settles nothing
|
|
* itself: the worker still proves the owner and its copyback process group
|
|
* before it takes ownership. Returns whether the recheck was woken.
|
|
*/
|
|
export async function wakeCopybackOwnerRecheck(db: Db, input: {
|
|
companyId: string; actionId: string; now: Date;
|
|
owner: { pid: number | null; processStartedAt: string | null; bootId: string | null; processSpace: string | null };
|
|
}): Promise<boolean> {
|
|
return db.transaction(async tx => {
|
|
const [action] = await tx.select().from(issueRecoveryActions).where(and(
|
|
eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.id, input.actionId),
|
|
inArray(issueRecoveryActions.status, ["active", "escalated"]))).for("update");
|
|
const hold = action?.evidence?.workspaceOwnerRecovery as { state?: unknown; owner?: Record<string, unknown> } | undefined;
|
|
const owner = hold?.owner;
|
|
if (!action || hold?.state !== "operator_required" || !owner || owner.pid !== input.owner.pid
|
|
|| owner.processStartedAt !== input.owner.processStartedAt
|
|
|| (owner.controllerBootId ?? null) !== input.owner.bootId || (owner.hostHash ?? null) !== input.owner.processSpace) return false;
|
|
const current = readSystemRecovery(action.evidence);
|
|
const runId = action.evidence.runId;
|
|
const now = input.now.getTime();
|
|
if (!current || typeof runId !== "string" || current.sourceRunId !== runId || current.state !== "awaiting_evidence"
|
|
|| current.failure?.code !== "copyback_stop_unconfirmed"
|
|
|| (current.deadlineAt !== null && now >= current.deadlineAt)) return false;
|
|
const due = Math.max(now, current.providerCooldownAt ?? 0);
|
|
if (current.nextAttemptAt !== null && current.nextAttemptAt <= due) {
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.workspaceRecovery);
|
|
return true;
|
|
}
|
|
return updateSystemRecovery(tx as unknown as Db, { companyId: input.companyId, issueId: action.sourceIssueId,
|
|
runId, actionId: action.id }, current, { ...current, revision: current.revision + 1, nextAttemptAt: due });
|
|
});
|
|
}
|
|
|
|
/** Admission records intent only. No request handler runs physical recovery. */
|
|
export async function requestWorkspaceRecovery(input: Source & { db: Db; idempotencyKey: string; actorId: string }) {
|
|
return input.db.transaction(async tx => {
|
|
const [issue] = await tx.select().from(issues).where(and(eq(issues.id, input.issueId), eq(issues.companyId, input.companyId))).for("update");
|
|
const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId))).for("update");
|
|
const [action] = await tx.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.id, input.actionId),
|
|
eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, input.issueId))).for("update");
|
|
if (!issue || !run || !action || action.evidence.runId !== run.id
|
|
|| (run.nativeIssueId ?? run.issueId ?? run.contextSnapshot?.issueId) !== issue.id
|
|
|| !terminal.includes(run.status) || (!hasRequiredWorkspaceRecovery(run.resultJson) && !exportCauses.includes(action.cause))) {
|
|
throw conflict("The workspace recovery changed. Refresh the task before retrying.");
|
|
}
|
|
const prior = readWorkspaceRecovery(action.evidence);
|
|
if (prior?.requestIds.includes(input.idempotencyKey) || prior?.status === "recovered") return { actionId: action.id, status: prior.status };
|
|
const pending = prior && ["queued", "recovering"].includes(prior.status);
|
|
const state: WorkspaceRecovery = pending ? { ...prior!, requestIds: [...prior!.requestIds, input.idempotencyKey] }
|
|
: { schema: "paperclip.workspace-recovery.v1", runId: run.id, status: "queued", stage: "verifying_stop",
|
|
attempt: prior?.attempt ?? 0, attemptLimit: (prior?.attempt ?? 0) + 1,
|
|
requestIds: [...(prior?.requestIds ?? []), input.idempotencyKey] };
|
|
const previous = readSystemRecovery(action.evidence);
|
|
const active = previous && ["scheduled", "running", "awaiting_evidence"].includes(previous.state);
|
|
const system = previous?.state === "running" ? previous : active ? { ...previous, revision: previous.revision + 1,
|
|
nextAttemptAt: Math.max(Date.now(), previous.providerCooldownAt ?? 0) } : { ...createSystemRecovery(run.id),
|
|
revision: (previous?.revision ?? -1) + 1,
|
|
preservationOperationIds: previous?.preservationOperationIds ?? [],
|
|
providerCooldownAt: previous?.providerCooldownAt ?? null,
|
|
nextAttemptAt: previous?.providerCooldownAt ?? null };
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.workspaceRecovery);
|
|
await tx.update(issueRecoveryActions).set({ ownerType: "system", ownerAgentId: null, ownerUserId: null,
|
|
evidence: { ...action.evidence, workspaceRecovery: state, systemRecovery: system }, updatedAt: new Date() })
|
|
.where(eq(issueRecoveryActions.id, action.id));
|
|
await persistActivity(tx as unknown as Db, { companyId: input.companyId, actorType: "user", actorId: input.actorId,
|
|
action: "issue.workspace_recovery_requested", entityType: "issue", entityId: issue.id, runId: run.id,
|
|
details: { recoveryActionId: action.id, idempotencyKey: input.idempotencyKey,
|
|
...(previous && previous.generationId !== system.generationId ? { previousGeneration: previous } : {}) } });
|
|
return { actionId: action.id, status: state.status };
|
|
});
|
|
}
|
|
|
|
/** Physical repair remains serialized by the existing source ownership lock.
|
|
* Neither preservation nor cleanup creates an agent turn. */
|
|
export async function recoverRetainedWorkspace(input: Source & { db: Db; environmentRuntime: EnvironmentRuntimeService }) {
|
|
const { db } = input;
|
|
await adoptSystemRecovery(db, input);
|
|
let initial = await loadSystemRecovery(db, input);
|
|
if (initial?.state === "exhausted" && await settleHoldIntoAbandonedEpochContinuation(db, input)) return;
|
|
// A hold whose files a sibling saved, or whose source is refused permanently
|
|
// (including holds saved by older releases), settles without a person.
|
|
// An adoptable native source keeps its retention path; only the sibling
|
|
// settlement applies to it.
|
|
if (initial?.state === "exhausted"
|
|
&& await settleExhaustedRetainedHold(input, initial, { fallback: !canAdoptNativeSourceRetention(initial) })) return;
|
|
// An exhausted wait for stop proof is re-armed, bounded, once the run is over.
|
|
if (initial?.state === "exhausted" && await rearmExhaustedStopProof(db, input, initial))
|
|
initial = await loadSystemRecovery(db, input);
|
|
// A save exhausted under an older capture policy is retried once under the current one.
|
|
if (initial?.state === "exhausted" && await rearmExhaustedCapturePolicy(db, input, initial))
|
|
initial = await loadSystemRecovery(db, input);
|
|
if (initial?.state === "exhausted" && !canAdoptNativeSourceRetention(initial)) {
|
|
// A budget spent before this hold existed still names its receipt, so the
|
|
// board confirmation in its handoff can release it (RC-50 wave databases).
|
|
if (initial.failure?.code === "copyback_stop_unconfirmed") await recordCopybackOwnerHold(db, input, initial.generationId);
|
|
await handOffExhaustedSystemRecovery(db, input, initial);
|
|
return;
|
|
}
|
|
if (!initial || initial.state === "complete" ||
|
|
(initial.nextAttemptAt !== null && initial.nextAttemptAt > Date.now())) return;
|
|
const runAttempt = async (ownership: { token: string; assertHeld(): Promise<void> }) => {
|
|
let current = await loadSystemRecovery(db, input);
|
|
if (!current) return;
|
|
if (current.state === "exhausted") {
|
|
const spent = current;
|
|
current = await retainNativeRecoverySource(db, input, current, ownership);
|
|
if (!current) {
|
|
// Not adoptable now. The board owns the spent budget until Retry or a
|
|
// later adoption returns it to the system.
|
|
await handOffExhaustedSystemRecovery(db, input, (await loadSystemRecovery(db, input)) ?? spent);
|
|
return;
|
|
}
|
|
}
|
|
const reserved = reserveSystemRecoveryAttempt(current);
|
|
if (!reserved) {
|
|
if (current.state !== "exhausted" && current.deadlineAt !== null && Date.now() >= current.deadlineAt)
|
|
await updateSystemRecovery(db, input, current, { ...current, revision: current.revision + 1,
|
|
state: "exhausted", nextAttemptAt: null });
|
|
return;
|
|
}
|
|
if (!await updateSystemRecovery(db, input, current, reserved)) return;
|
|
current = reserved;
|
|
const progress = async (patch: Partial<SystemRecoveryState>) => {
|
|
await ownership.assertHeld();
|
|
const next = { ...current!, ...patch, revision: current!.revision + 1 };
|
|
if (!await updateSystemRecovery(db, input, current!, next)) throw new Error("workspace_recovery_generation_changed");
|
|
current = next;
|
|
};
|
|
const observeActive = async () => {
|
|
await ownership.assertHeld();
|
|
await db.update(issueRecoveryActions).set({ evidence: sql`jsonb_set(${issueRecoveryActions.evidence},
|
|
'{systemRecoveryActiveAt}', ${JSON.stringify(Date.now())}::jsonb)` }).where(and(
|
|
eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.id, input.actionId),
|
|
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,attemptId}' = ${reserved.attemptId}`));
|
|
};
|
|
let observing = Promise.resolve();
|
|
let cacheReclaimUnconfirmed = false;
|
|
const observer = setInterval(() => { observing = observing.then(observeActive).catch(() => undefined); }, 15_000);
|
|
try {
|
|
await observeActive();
|
|
const inspected = await inspectWorkspaceRecoverySource(db, input);
|
|
let sourceUnavailable = false;
|
|
for (const source of inspected.sources) {
|
|
const { lease, environment, remoteCwd } = source;
|
|
if (hasRemoteTerminationReceipt(lease)
|
|
&& (lease.metadata?.remoteExecutionTermination as Record<string, unknown> | undefined)?.state === "destroyed") {
|
|
sourceUnavailable = true;
|
|
continue;
|
|
}
|
|
if (lease.provider === "daytona" && lease.providerLeaseId) {
|
|
const absence = await requestSandboxAbsenceProof({ companyId: input.companyId, issueId: input.issueId, runId: input.runId, leaseId: lease.id,
|
|
provider: "daytona", providerLeaseId: lease.providerLeaseId, ownerToken: ownership.token });
|
|
if (absence) {
|
|
await ownership.assertHeld();
|
|
if (!await settleAbsentSandbox(db, lease, absence)) throw new SystemRecoveryFailure("agent_stop_unconfirmed", "recheck_evidence");
|
|
sourceUnavailable = true;
|
|
continue;
|
|
}
|
|
}
|
|
let reference = source.preservationOnly ? null : source.reference;
|
|
if (reference) {
|
|
try { await readNativeWorkspaceSyncLocalCwd({ runId: input.runId, companyId: input.companyId,
|
|
workspaceId: reference.workspaceId, reference }); }
|
|
catch (error) {
|
|
// Missing local merge metadata permits a separate snapshot of the
|
|
// exact bound allocation. Corrupt metadata never weakens binding.
|
|
if ((error as NodeJS.ErrnoException)?.code === "ENOENT") reference = null;
|
|
else throw workspaceSourceBindingFailure("local_descriptor_invalid");
|
|
}
|
|
}
|
|
let stoppedLease = lease;
|
|
if (!hasRemoteTerminationReceipt(stoppedLease)) {
|
|
const pending = await db.transaction(async tx => {
|
|
const [fresh] = await tx.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, input.companyId),
|
|
eq(environmentLeases.id, lease.id))).for("update");
|
|
if (!fresh || fresh.metadata?.pendingCleanupInFlight === true) return null;
|
|
const prepared = hasStopOnlyCleanup(fresh) ? fresh : fresh.status === "released"
|
|
? await prepareHistoricalSandboxStopVerification(tx as unknown as Db, fresh)
|
|
: await prepareSandboxStopAndRetain(tx as unknown as Db, fresh);
|
|
if (!prepared) return null;
|
|
const [claimed] = await tx.update(environmentLeases).set({ metadata: sql`${environmentLeases.metadata} ||
|
|
${JSON.stringify({ pendingCleanupAttemptId: ownership.token, pendingCleanupInFlight: true,
|
|
pendingCleanupLeaseExpiresAtMs: Date.now() + 15 * 60_000 })}::jsonb` })
|
|
.where(eq(environmentLeases.id, lease.id)).returning();
|
|
return claimed;
|
|
});
|
|
if (!pending) throw new SystemRecoveryFailure("agent_stop_unconfirmed", "recheck_evidence");
|
|
// The cleanup service owns exact-allocation termination. No process ID
|
|
// or terminal run status is substituted for its provider receipt.
|
|
const settled = await input.environmentRuntime.retryPendingSandboxTeardown({
|
|
environment: environment as unknown as Environment, lease: pending as EnvironmentLease })
|
|
.then(receipt => settleStopOnlyCleanup(db, pending, { attemptId: ownership.token, receipt }),
|
|
async () => await restoreSandboxStopAfterFailedRestart(db, pending, { attemptId: ownership.token })
|
|
?? await settleStopOnlyCleanup(db, pending, { attemptId: ownership.token }));
|
|
if (!settled || !hasRemoteTerminationReceipt(settled))
|
|
throw new SystemRecoveryFailure("agent_stop_unconfirmed", "recheck_evidence");
|
|
// The provider confirmed the sandbox is gone: its files cannot be read.
|
|
if ((settled.metadata?.remoteExecutionTermination as Record<string, unknown> | undefined)?.state === "destroyed") {
|
|
sourceUnavailable = true;
|
|
continue;
|
|
}
|
|
stoppedLease = settled;
|
|
}
|
|
const target = await resolveEnvironmentExecutionTarget({ db, companyId: input.companyId, adapterType: "paperclip_runner",
|
|
environment, leaseId: lease.id, leaseMetadata: stoppedLease.metadata,
|
|
lease: stoppedLease as EnvironmentLease, environmentRuntime: input.environmentRuntime });
|
|
if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runner || target.remoteCwd !== remoteCwd)
|
|
throw workspaceSourceBindingFailure("execution_target_invalid");
|
|
await ownership.assertHeld();
|
|
let cleanupLease = await db.transaction(async tx => {
|
|
const [fresh] = await tx.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, input.companyId),
|
|
eq(environmentLeases.id, lease.id))).for("update");
|
|
if (!fresh || !hasRemoteTerminationReceipt(fresh) || fresh.providerLeaseId !== lease.providerLeaseId)
|
|
throw new SystemRecoveryFailure("agent_stop_unconfirmed", "recheck_evidence");
|
|
const pending = await prepareStoppedSandboxForRecovery(tx as unknown as Db, fresh);
|
|
if (!pending) throw workspaceSourceBindingFailure("recovery_stop_intent_invalid");
|
|
const [claimed] = await tx.update(environmentLeases).set({ metadata: sql`${environmentLeases.metadata} || ${JSON.stringify({
|
|
pendingCleanupAttemptId: ownership.token, pendingCleanupInFlight: true, pendingCleanupLeaseExpiresAtMs: Date.now() + 15 * 60_000,
|
|
})}::jsonb` }).where(eq(environmentLeases.id, fresh.id)).returning();
|
|
return claimed;
|
|
});
|
|
let renewal = Promise.resolve();
|
|
const keepalive = setInterval(() => { renewal = renewal.then(async () => {
|
|
await ownership.assertHeld();
|
|
await db.update(environmentLeases).set({ metadata: sql`jsonb_set(${environmentLeases.metadata}, '{pendingCleanupLeaseExpiresAtMs}', ${JSON.stringify(Date.now() + 15 * 60_000)}::jsonb)` })
|
|
.where(and(eq(environmentLeases.id, lease.id), sql`${environmentLeases.metadata}->>'pendingCleanupAttemptId' = ${ownership.token}`));
|
|
}).catch(() => undefined); }, 30_000);
|
|
let stopSettled = false;
|
|
try {
|
|
const opened = await input.environmentRuntime.resumeWorkspaceRecoveryLease({ source: input,
|
|
leaseId: cleanupLease.id, environmentId: environment.id, ownership }).catch(error => {
|
|
throw sandboxDiskFull.test((error as Error | undefined)?.message ?? "")
|
|
? new SystemRecoveryFailure("source_unavailable", "use_alternative_strategy") : error;
|
|
});
|
|
const confirmed = await confirmSandboxRecoveryRestart(db, cleanupLease, { attemptId: ownership.token });
|
|
if (!confirmed) throw new SystemRecoveryFailure("agent_stop_unconfirmed", "recheck_evidence");
|
|
cleanupLease = confirmed;
|
|
if (opened?.providerLeaseId !== lease.providerLeaseId) throw workspaceSourceBindingFailure("reopened_allocation_mismatch");
|
|
if (opened.metadata?.remoteCwd !== remoteCwd) throw workspaceSourceBindingFailure("reopened_cwd_mismatch");
|
|
await inspectWorkspaceRecoverySource(db, input);
|
|
await progress({ phase: "preserve_files" });
|
|
let operationId: string;
|
|
if (source.remoteReference && source.remoteWorkspace) {
|
|
const [workspace] = await db.select().from(executionWorkspaces).where(and(
|
|
eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, source.remoteWorkspace.id),
|
|
));
|
|
if (!workspace) throw workspaceSourceBindingFailure("workspace_reference_mismatch");
|
|
const save = () => saveRemoteWorkspace({ db, workspace, runId: input.runId,
|
|
lease: cleanupLease as EnvironmentLease, target, assertOwned: ownership.assertHeld });
|
|
let saved: Awaited<ReturnType<typeof save>>;
|
|
try { saved = await save(); }
|
|
catch (error) {
|
|
if (!isSandboxDiskFullSave(error)) throw error;
|
|
let reclaimed = false;
|
|
try {
|
|
const reclaim = await reclaimInactiveNativeProviderCache({ db, companyId: input.companyId,
|
|
issueId: input.issueId, runId: input.runId, workspaceId: workspace.id, target,
|
|
assertOwned: ownership.assertHeld });
|
|
reclaimed = reclaim.changed;
|
|
cacheReclaimUnconfirmed = !reclaim.confirmed;
|
|
} catch {
|
|
cacheReclaimUnconfirmed = true;
|
|
logger.warn({ runId: input.runId }, "Retained provider cache cleanup could not be confirmed");
|
|
}
|
|
if (!reclaimed) throw error;
|
|
await ownership.assertHeld();
|
|
saved = await save();
|
|
}
|
|
await ownership.assertHeld();
|
|
const [existing] = await db.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and(
|
|
eq(workspaceOperations.companyId, input.companyId), eq(workspaceOperations.heartbeatRunId, input.runId),
|
|
eq(workspaceOperations.phase, "workspace_recovery"), eq(workspaceOperations.status, "succeeded"),
|
|
sql`${workspaceOperations.metadata}->>'leaseId' = ${lease.id}`,
|
|
sql`${workspaceOperations.metadata}->>'snapshotId' = ${saved.snapshotId}`,
|
|
)).limit(1);
|
|
const operation = existing ?? (await db.insert(workspaceOperations).values({ companyId: input.companyId,
|
|
issueId: input.issueId, heartbeatRunId: input.runId, executionWorkspaceId: workspace.id,
|
|
phase: "workspace_recovery", status: "succeeded", finishedAt: new Date(), metadata: {
|
|
recoveryActionId: input.actionId, leaseId: lease.id, providerLeaseId: lease.providerLeaseId,
|
|
snapshotId: saved.snapshotId, generation: saved.generation, sha256: saved.sha256,
|
|
storage: "remote_workspace", epoch: source.remoteReference.epoch,
|
|
} }).returning({ id: workspaceOperations.id }))[0]!;
|
|
operationId = operation.id;
|
|
} else if (reference) {
|
|
const copied = await resumeNativeWorkspaceSync({ db, runId: input.runId, target, assertOwnership: ownership.assertHeld });
|
|
if (!copied) throw new SystemRecoveryFailure("snapshot_verification_failed", "service_required");
|
|
const after = await inspectWorkspaceRecoverySource(db, input);
|
|
const finalized = after.sources.find(item => item.lease.id === lease.id)?.reference;
|
|
if (finalized?.state !== "finalized" || !finalized.finalHostSha256)
|
|
throw new SystemRecoveryFailure("snapshot_verification_failed", "service_required");
|
|
await ownership.assertHeld();
|
|
const [existing] = await db.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and(
|
|
eq(workspaceOperations.companyId, input.companyId), eq(workspaceOperations.heartbeatRunId, input.runId),
|
|
eq(workspaceOperations.phase, "workspace_recovery"), eq(workspaceOperations.status, "succeeded"),
|
|
sql`${workspaceOperations.metadata}->>'leaseId' = ${lease.id}`)).limit(1);
|
|
const copy = existing ?? (await db.insert(workspaceOperations).values({ companyId: input.companyId, issueId: input.issueId,
|
|
heartbeatRunId: input.runId, executionWorkspaceId: lease.executionWorkspaceId, phase: "workspace_recovery",
|
|
status: "succeeded", finishedAt: new Date(), metadata: { recoveryActionId: input.actionId, leaseId: lease.id,
|
|
providerLeaseId: lease.providerLeaseId, descriptorSha256: finalized.descriptorSha256,
|
|
finalHostSha256: finalized.finalHostSha256 } }).returning({ id: workspaceOperations.id }))[0];
|
|
operationId = copy.id;
|
|
} else {
|
|
operationId = await preserveWorkspaceSnapshot({ ...input, leaseId: lease.id,
|
|
providerLeaseId: lease.providerLeaseId!, target, assertHeld: ownership.assertHeld });
|
|
}
|
|
await progress({ preservationOperationIds: [...new Set([...current!.preservationOperationIds, operationId])] });
|
|
clearInterval(keepalive); await renewal;
|
|
await ownership.assertHeld();
|
|
const receipt = await input.environmentRuntime.retryPendingSandboxTeardown({ environment: environment as unknown as Environment,
|
|
lease: cleanupLease as EnvironmentLease });
|
|
const settled = await settleStopOnlyCleanup(db, cleanupLease, { attemptId: ownership.token, receipt });
|
|
stopSettled = Boolean(settled && hasRemoteTerminationReceipt(settled));
|
|
if (!stopSettled) throw new SystemRecoveryFailure("agent_stop_unconfirmed", "recheck_evidence");
|
|
} finally {
|
|
clearInterval(keepalive); await renewal;
|
|
if (!stopSettled) {
|
|
// A lost resume/export reply retains the same stop-only obligation.
|
|
// The maintenance worker must never turn it into destructive cleanup.
|
|
try {
|
|
await ownership.assertHeld();
|
|
const settled = await input.environmentRuntime.retryPendingSandboxTeardown({ environment: environment as unknown as Environment,
|
|
lease: cleanupLease as EnvironmentLease }).then(receipt => settleStopOnlyCleanup(db, cleanupLease, { attemptId: ownership.token, receipt }), () => null);
|
|
if (!settled || !hasRemoteTerminationReceipt(settled))
|
|
await restoreSandboxStopAfterFailedRestart(db, cleanupLease, { attemptId: ownership.token });
|
|
} catch { /* The persisted stop-only obligation remains authoritative. */ }
|
|
}
|
|
}
|
|
}
|
|
await ownership.assertHeld();
|
|
if (sourceUnavailable) throw new SystemRecoveryFailure("source_unavailable", "use_alternative_strategy");
|
|
const next = { ...current!, revision: current!.revision + 1, phase: "reconcile_effects" as const,
|
|
state: "complete" as const, nextAttemptAt: null };
|
|
const activity = await db.transaction(async tx => {
|
|
await notifyNativeResultCommit(tx, input);
|
|
await notifyNativeWorkspaceFinalization(tx, input);
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionReplacement);
|
|
await notifyNativeStatusReassessment(tx, input);
|
|
await tx.insert(workspaceOperations).values({ companyId: input.companyId, issueId: input.issueId,
|
|
heartbeatRunId: input.runId, phase: "workspace_finalize", status: "succeeded", finishedAt: new Date(),
|
|
metadata: { owningService: "workspace_recovery", preservationOperationIds: next.preservationOperationIds } });
|
|
if (!await updateSystemRecovery(tx as unknown as Db, input, current!, next))
|
|
throw new Error("workspace_recovery_generation_changed");
|
|
if (inspected.action.evidence.nativeSourceRetention && !await settleRecoveredNativeSourceRetention(tx as unknown as Db, input, next.preservationOperationIds))
|
|
throw new SystemRecoveryFailure("effect_unresolved", "recheck_evidence");
|
|
// A settled restore hold returns its warm sandbox to reuse. A hold that
|
|
// fails any guard keeps its protection; the saved files still count.
|
|
await settleRecoveredWorkspaceRestoreHolds(tx as unknown as Db, input);
|
|
await expediteFreshContinuationChecks(tx as unknown as Db, input);
|
|
return persistActivity(tx as unknown as Db, { companyId: input.companyId, actorType: "system", actorId: "workspace-recovery",
|
|
action: "issue.workspace_recovered", entityType: "issue", entityId: input.issueId, runId: input.runId,
|
|
details: { recoveryActionId: input.actionId, summary: "Saved work recovered." } });
|
|
});
|
|
current = next;
|
|
try { publishActivity(activity.publication); } catch { logger.warn({ runId: input.runId }, "Recovery activity delivery deferred"); }
|
|
} catch (error) {
|
|
await ownership.assertHeld();
|
|
if (current!.state === "complete") {
|
|
logger.warn({ runId: input.runId }, "Preserved workspace activity delivery deferred");
|
|
return;
|
|
}
|
|
const classification = classifyNativeWorkspaceFailure(error);
|
|
const diskFull = !(error instanceof SystemRecoveryFailure) && !cacheReclaimUnconfirmed && isSandboxDiskFullSave(error);
|
|
const failure = error instanceof SystemRecoveryFailure ? error
|
|
: diskFull ? new SystemRecoveryFailure("export_interrupted", "user_action_required")
|
|
: classifyWorkspaceRestoreFailure(error) === "restore_invalid_metadata"
|
|
? new SystemRecoveryFailure("snapshot_verification_failed", "service_required")
|
|
: classification.code === "workspace_sync_out_unsafe_archive"
|
|
? new SystemRecoveryFailure("archive_unsafe", "service_required")
|
|
: classification.code === "workspace_sync_out_unrecoverable"
|
|
? new SystemRecoveryFailure("source_unavailable", "use_alternative_strategy")
|
|
: new SystemRecoveryFailure(current!.phase === "verify_execution" ? "provider_unavailable" : "export_interrupted", "retry_transient");
|
|
const retryAfterAt = (error as { retryAfterAt?: unknown })?.retryAfterAt;
|
|
// A generic transient code says only that the step can be retried. Keep
|
|
// why it failed in closed labels so a repeated deferral is diagnosable.
|
|
// A classified failure already names its subsystem.
|
|
const generic = !(error instanceof SystemRecoveryFailure) && failure.retryClass === "retry_transient"
|
|
&& (failure.code === "provider_unavailable" || failure.code === "export_interrupted");
|
|
const cause = diskFull ? { ...SANDBOX_DISK_FULL_RECOVERY_CAUSE } : generic ? sandboxProviderFailureLabel(error) : undefined;
|
|
const failed = failSystemRecovery(current!, failure, {
|
|
...(typeof retryAfterAt === "number" && Number.isFinite(retryAfterAt) ? { retryAfterAt } : {}),
|
|
...(cause ? { cause } : {}),
|
|
});
|
|
const recorded = await updateSystemRecovery(db, input, current!, failed);
|
|
if (recorded && failed.state === "exhausted" && await settleHoldIntoAbandonedEpochContinuation(db, input)) return;
|
|
// A permanent refusal of the recorded source repeats on every retry.
|
|
// Continue from the last save in this same pass, under this ownership.
|
|
if (recorded && failed.state === "exhausted" && isUnverifiableSourceFailure(failed.failure)) {
|
|
const fallback = await continueFromLastSavedWorkspace({ ...input, ownership }).catch(fallbackError => {
|
|
logger.warn({ err: fallbackError, runId: input.runId }, "Continuing from the last save was deferred; the hold remains");
|
|
return { kind: "ineligible" as const, reason: "fallback_failed" };
|
|
});
|
|
if (fallback.kind === "ineligible") await recordRetainedSourceFallbackDeferral(db, input, fallback.reason);
|
|
}
|
|
if (failure.retryClass === "recheck_evidence" && ["agent_stop_unconfirmed", "copyback_stop_unconfirmed"].includes(failure.code))
|
|
await recheckStoppedWorkspaceRecovery(db, input, false);
|
|
logger.warn({ err: error, runId: input.runId, failureCode: failure.code, failureCause: cause?.label, phase: current!.phase }, "Workspace preservation deferred; source retained");
|
|
} finally { clearInterval(observer); await observing; }
|
|
};
|
|
const owned = await withNativeWorkspaceFinalizationOwnership({ db, companyId: input.companyId, runId: input.runId,
|
|
retainedRepairActionId: input.actionId }, runAttempt);
|
|
if (!owned.acquired && owned.reason === "owner_unverified") {
|
|
await holdForUnverifiedCopybackOwner(db, input);
|
|
await recheckStoppedWorkspaceRecovery(db, input, false);
|
|
}
|
|
}
|
|
|
|
/**
|
|
* A copyback owner that a restart killed can never confirm its own stop, and
|
|
* Retry cannot release its receipt (RC-50). Rechecking that receipt is not a
|
|
* recovery attempt: the bounded recovery window still applies, but no attempt
|
|
* is spent. Record the exact receipt so the board can confirm the stop through
|
|
* `resumeNativeWorkspaceAfterOwnerStop`, and so Retry names this wait.
|
|
*/
|
|
async function holdForUnverifiedCopybackOwner(db: Db, input: Source) {
|
|
const current = await loadSystemRecovery(db, input);
|
|
if (!current || current.state === "complete" || current.state === "exhausted") return;
|
|
const next = failSystemRecovery(current, new SystemRecoveryFailure("copyback_stop_unconfirmed", "recheck_evidence"));
|
|
if (!await updateSystemRecovery(db, input, current, next)) return;
|
|
await recordCopybackOwnerHold(db, input, next.generationId);
|
|
}
|
|
|
|
/** Bookkeeping only: names the exact receipt for the board. Grants no attempt. */
|
|
async function recordCopybackOwnerHold(db: Db, input: Source, generationId: string) {
|
|
const [source] = await db.select({ owner: sql<unknown>`${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceFinalizationOwner'` })
|
|
.from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId)));
|
|
const owner = source?.owner;
|
|
if (!owner || typeof owner !== "object" || typeof (owner as { token?: unknown }).token !== "string") return;
|
|
const target = and(eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.id, input.actionId),
|
|
eq(issueRecoveryActions.sourceIssueId, input.issueId),
|
|
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,generationId}' = ${generationId}`,
|
|
sql`${issueRecoveryActions.evidence}#>'{workspaceOwnerRecovery,owner}' is distinct from ${JSON.stringify(owner)}::jsonb`);
|
|
await db.transaction(async tx => {
|
|
const [action] = await tx.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(target).for("update");
|
|
if (!action) return;
|
|
// Wake the ownership worker (on any server) before the hold it announces,
|
|
// so a quiet worker judges this owner and can wake the recheck.
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.ownershipInvariantReport,
|
|
JSON.stringify({ companyId: input.companyId, runId: input.runId, actionId: input.actionId }));
|
|
await tx.update(issueRecoveryActions).set({
|
|
// A confirmation names one receipt; a newer unverified owner needs its own.
|
|
evidence: sql`(${issueRecoveryActions.evidence} - 'workspaceOwnerStop') || jsonb_build_object('workspaceOwnerRecovery',
|
|
jsonb_build_object('state', 'operator_required', 'owner', ${JSON.stringify(owner)}::jsonb))`,
|
|
updatedAt: new Date(),
|
|
}).where(target);
|
|
});
|
|
}
|
|
|
|
const cursors = new WeakMap<Db, string>();
|
|
const workInFlight = new WeakMap<Db, Set<string>>();
|
|
/** Overdue recoveries the assigned-work sweep started outside the queue's
|
|
* slots. The queue skips their runs; their count has its own small limit. */
|
|
const overdueInFlight = new WeakMap<Db, Set<string>>();
|
|
export const OVERDUE_WORKSPACE_RECOVERY_LIMIT = 2;
|
|
const unfinishedCursors = new WeakMap<Db, string>();
|
|
|
|
/** Destruction is a later cleanup obligation, authorized only by a matching
|
|
* saved-file receipt and the original recorded disposition. */
|
|
export async function scheduleRecoveredWorkspaceCleanup(db: Db, run: typeof heartbeatRuns.$inferSelect) {
|
|
if (!await hasVerifiedWorkspaceRepair(db, run)) return;
|
|
const legacy = readNativeWorkspaceSyncReference(run.runnerProfileJson?.nativeWorkspaceSync);
|
|
const parsedRemote = remoteWorkspaceRunReferenceSchema.safeParse(run.runnerProfileJson?.remoteWorkspace);
|
|
const remote = parsedRemote.success && parsedRemote.data.state === "saved" ? parsedRemote.data : null;
|
|
const ref = legacy ?? remote;
|
|
const disposition = legacy?.resourceDisposition ?? (remote ? run.runnerProfileJson?.nativeResourceDisposition : undefined);
|
|
if (!ref || disposition !== "destroy") return;
|
|
const [lease] = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.id, ref.leaseId)));
|
|
if (!lease || !hasRemoteTerminationReceipt(lease)) return;
|
|
const [other] = await db.select({ id: environmentLeases.id }).from(environmentLeases).where(and(
|
|
ne(environmentLeases.id, lease.id), eq(environmentLeases.provider, lease.provider!), eq(environmentLeases.providerLeaseId, lease.providerLeaseId!),
|
|
)).limit(1);
|
|
if (other) return;
|
|
await writeEnvironmentCleanup(db, tx => tx.update(environmentLeases).set({ status: "pending_cleanup", cleanupStatus: "failed", leasePolicy: "ephemeral",
|
|
updatedAt: new Date(), metadata: sql`(${environmentLeases.metadata} - 'workspaceRestoreRecovery' - 'remoteExecutionTermination') ||
|
|
${JSON.stringify({ workspaceRecoveryCleanup: { runId: run.id,
|
|
// Keep the verified stop: successors wait on the stop, not on the destroy.
|
|
stopReceipt: lease.metadata?.remoteExecutionTermination, ...(legacy ? { descriptorSha256: legacy.descriptorSha256 }
|
|
: { snapshotId: remote!.snapshotId, epoch: remote!.epoch }) },
|
|
pendingCleanupInFlight: false, pendingCleanupRetryAfterMs: 0, pendingCleanupRetryAttempts: 0, pendingCleanupLeaseExpiresAtMs: 0 })}::jsonb`,
|
|
}).where(and(eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.id, ref.leaseId),
|
|
eq(environmentLeases.heartbeatRunId, run.id), eq(environmentLeases.providerLeaseId, ref.providerLeaseId),
|
|
eq(environmentLeases.status, "released"), eq(environmentLeases.cleanupStatus, "success"),
|
|
sql`${environmentLeases.metadata} = ${JSON.stringify(lease.metadata)}::jsonb`,
|
|
sql`${environmentLeases.metadata}->>'workspaceRecoveryCleanup' is null`, reusableSandboxNotKeptAfterReplacementSql())),
|
|
{ companyId: run.companyId, runId: run.id });
|
|
}
|
|
/** `dueOnly` narrows the obligations to exhausted holds whose re-arm or
|
|
* fallback is due now. Cleanup, interrupted preparation and saved input that
|
|
* waits stay with the obligation pass. */
|
|
export function retainedWorkspaceCandidates(db: Db, cursor?: string, unfinishedOnly = false, obligationsOnly = false, inFlightRunIds: string[] = [],
|
|
dueOnly = false) {
|
|
const cleanup = sql`(((${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceSync,resourceDisposition}' = 'destroy'
|
|
and ${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceSync,leaseId}' is not null)
|
|
or (${heartbeatRuns.runnerProfileJson}->>'nativeResourceDisposition' = 'destroy'
|
|
and ${heartbeatRuns.runnerProfileJson}#>>'{remoteWorkspace,schema}' = 'paperclip.remote-workspace-run.v1'
|
|
and ${heartbeatRuns.runnerProfileJson}#>>'{remoteWorkspace,state}' = 'saved')) and exists (
|
|
select 1 from ${environmentLeases} where ${environmentLeases.companyId} = ${heartbeatRuns.companyId}
|
|
and ${environmentLeases.heartbeatRunId} = ${heartbeatRuns.id}
|
|
and ${environmentLeases.id}::text = coalesce(${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceSync,leaseId}',
|
|
${heartbeatRuns.runnerProfileJson}#>>'{remoteWorkspace,leaseId}')
|
|
and ${environmentLeases.status} = 'released' and ${environmentLeases.cleanupStatus} = 'success'
|
|
and ${environmentLeases.metadata}->'workspaceRecoveryCleanup' is null))`;
|
|
const waitingInput = sql`exists (select 1 from ${agentWakeupRequests}
|
|
where ${agentWakeupRequests.companyId} = ${issueRecoveryActions.companyId}
|
|
and ${agentWakeupRequests.status} = 'deferred_issue_execution' and ${agentWakeupRequests.requestedByActorType} = 'user'
|
|
and ${agentWakeupRequests.payload}->>'issueId' = ${issueRecoveryActions.sourceIssueId}::text)`;
|
|
const interruptedPreparation = sql`(${heartbeatRuns.runtimeMode} = 'legacy'
|
|
and (${heartbeatRuns.status} = 'interrupted' or (${heartbeatRuns.status} = 'failed'
|
|
and (${heartbeatRuns.errorCode} = 'process_lost' or (${inArray(heartbeatRuns.errorCode, [...TRANSIENT_WORKSPACE_PREPARATION_CODES])}
|
|
and ${heartbeatRuns.resultJson}#>>'{workspaceRestoreRecovery,schema}' = ${LEGACY_WORKSPACE_RECOVERY_SCHEMA}))
|
|
and ${heartbeatRuns.resultJson}#>>'{executionRecovery,kind}' = 'bootstrap'
|
|
and ${heartbeatRuns.resultJson}#>'{executionRecovery,providerWorkStarted}' = 'false'::jsonb))
|
|
and ${heartbeatRuns.runtimeModeResolvedAt} is null and ${heartbeatRuns.executionStage} = 'preparing'
|
|
and ${heartbeatRuns.runnerProfileJson}#>>'{adapterDispatch,adapterType}' = 'paperclip_runner'
|
|
and (${issueRecoveryActions.evidence}#>>'{automaticRecovery,replay}' = 'blocked'
|
|
or (${issueRecoveryActions.evidence}#>>'{automaticRecovery,replay}' = 'allowed'
|
|
and ${issueRecoveryActions.evidence}->>'bootstrapPreparationSettledAt' is not null
|
|
and coalesce(${issueRecoveryActions.evidence}->>'bootstrapPreparationContinuation', 'pending') in ('pending', 'deferred')))
|
|
and exists (select 1 from ${issues} where ${issues.companyId} = ${heartbeatRuns.companyId}
|
|
and ${issues.id} = ${issueRecoveryActions.sourceIssueId} and ${issues.assigneeAgentId} = ${heartbeatRuns.agentId}
|
|
and ${issues.status} in ('todo', 'in_progress', 'in_review', 'blocked')
|
|
and ${issues.executionRunId} is null and ${issues.checkoutRunId} is null
|
|
and (${issues.status} <> 'blocked' or
|
|
${issueRecoveryActions.evidence}#>>'{nativeBootstrapFailureBlock,statusVersion}' = ${issues.statusVersion}::text)))`;
|
|
const now = Date.now();
|
|
const dueHold = sql`(${retainedSourceFallbackDueCondition(now)} or ${stopProofRearmDueCondition(now)} or ${capturePolicyRearmDueCondition(now)})`;
|
|
return db.select({ action: issueRecoveryActions, run: heartbeatRuns }).from(issueRecoveryActions)
|
|
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.companyId, issueRecoveryActions.companyId), eq(heartbeatRuns.id,
|
|
sql`case when ${issueRecoveryActions.evidence}->>'runId' ~* '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'
|
|
then (${issueRecoveryActions.evidence}->>'runId')::uuid end`)))
|
|
.where(and(inArray(heartbeatRuns.status, terminal),
|
|
workspaceRecoverySourceEligible(true),
|
|
unfinishedOnly ? sql`(${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('scheduled', 'running', 'awaiting_evidence')
|
|
or (${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' = 'complete'
|
|
and ${issueRecoveryActions.evidence}->>'bootstrapPreparationSettledAt' is not null
|
|
and coalesce(${issueRecoveryActions.evidence}->>'bootstrapPreparationContinuation', 'pending') = 'pending'))` : undefined,
|
|
obligationsOnly ? sql`(${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('scheduled', 'running', 'awaiting_evidence')
|
|
or (${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' = 'complete' and (${cleanup} or ${interruptedPreparation})) or ${waitingInput}
|
|
or ${dueHold})` : undefined,
|
|
dueOnly ? dueHold : undefined,
|
|
sql`(${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('scheduled', 'running', 'awaiting_evidence', 'complete', 'exhausted')
|
|
and (${issueRecoveryActions.evidence}#>'{systemRecovery,nextAttemptAt}' = 'null'::jsonb
|
|
or ${issueRecoveryActions.evidence}#>'{systemRecovery,nextAttemptAt}' <= to_jsonb(${now}::bigint)))`,
|
|
sql`(${issueRecoveryActions.evidence}#>>'{workspaceRecovery,status}' is distinct from 'recovered' or
|
|
${cleanup} or ${waitingInput} or ${interruptedPreparation})`,
|
|
cursor ? gt(issueRecoveryActions.id, cursor) : undefined,
|
|
inFlightRunIds.length ? notInArray(heartbeatRuns.id, inFlightRunIds) : undefined,
|
|
)).orderBy(asc(issueRecoveryActions.id)).limit(5);
|
|
}
|
|
|
|
/**
|
|
* The worker's lane positions, kept in memory: a restart starts every lane
|
|
* from its first candidate, so no persisted state or migration is involved.
|
|
* `cursor`, `complete` and `restart` belong to the obligation pass; `dueCursor`
|
|
* to the due lane; `firstLane` names the lane that has priority in the next run.
|
|
*/
|
|
export type PassiveSweep = { cursor?: string; complete?: boolean; restart?: boolean; dueCursor?: string; firstLane?: number };
|
|
|
|
type CandidateRow = Awaited<ReturnType<typeof retainedWorkspaceCandidates>>[number];
|
|
type CandidateLane = { rows: CandidateRow[]; index: number; advance: (actionId: string) => void };
|
|
|
|
/** Bounded discovery shares the periodic recovery lifecycle. A repaired source
|
|
* still owes cleanup even without saved input or after task reconciliation.
|
|
*
|
|
* Candidates come from lanes, each an ordered page that a cursor advances:
|
|
* unfinished saves; for the worker, exhausted holds whose re-arm or fallback
|
|
* is due; and the obligation pass (cleanup, interrupted preparation, saved
|
|
* input that waits, and the same due holds). Each lane with work takes one
|
|
* free slot before any lane takes a second, and the worker rotates which lane
|
|
* goes first, so no kind of load (a save another server owns, a long local
|
|
* export, hundreds of waiting holds) starves another. */
|
|
export async function reconcileRetainedWorkspaces(db: Db, runtime: EnvironmentRuntimeService,
|
|
onRecovered: (run: typeof heartbeatRuns.$inferSelect) => Promise<void>, obligationsOnly = false,
|
|
onFinished?: (continueSweep: boolean) => void, passiveSweep?: PassiveSweep) {
|
|
await adoptHistoricalWorkspaceRecoveries(db);
|
|
let pending = workInFlight.get(db);
|
|
if (!pending) { pending = new Set(); workInFlight.set(db, pending); }
|
|
if (pending.size >= 2) return;
|
|
// Locally running exports already own their slot. Exclude them before
|
|
// pagination so a page containing only held work cannot prevent wrapping.
|
|
const inFlightRunIds = [...pending, ...(overdueInFlight.get(db) ?? [])];
|
|
let unfinishedRows = await retainedWorkspaceCandidates(db, unfinishedCursors.get(db), true, false, inFlightRunIds);
|
|
if (!unfinishedRows.length) unfinishedRows = await retainedWorkspaceCandidates(db, undefined, true, false, inFlightRunIds);
|
|
const unfinished: CandidateLane = { rows: unfinishedRows, index: 0, advance: id => unfinishedCursors.set(db, id) };
|
|
let rows = await retainedWorkspaceCandidates(db, passiveSweep ? passiveSweep.cursor : cursors.get(db), false, obligationsOnly, inFlightRunIds);
|
|
if (passiveSweep && !rows.length && passiveSweep.restart) {
|
|
Object.assign(passiveSweep, { cursor: undefined, restart: false });
|
|
rows = await retainedWorkspaceCandidates(db, undefined, false, obligationsOnly, inFlightRunIds);
|
|
}
|
|
if (passiveSweep) passiveSweep.complete = !rows.length;
|
|
else if (!rows.length) rows = await retainedWorkspaceCandidates(db, undefined, false, obligationsOnly, inFlightRunIds);
|
|
const sweep: CandidateLane = { rows, index: 0,
|
|
advance: id => { if (passiveSweep) passiveSweep.cursor = id; else cursors.set(db, id); } };
|
|
const lanes = [unfinished];
|
|
if (passiveSweep) {
|
|
// The due lane wraps at its tail like the unfinished lane. A hold that
|
|
// becomes due by time alone (a lease expiry, a re-arm backoff) is reached
|
|
// even after the obligation pass completed with no committed change.
|
|
let dueRows = await retainedWorkspaceCandidates(db, passiveSweep.dueCursor, false, true, inFlightRunIds, true);
|
|
if (!dueRows.length && passiveSweep.dueCursor)
|
|
dueRows = await retainedWorkspaceCandidates(db, undefined, false, true, inFlightRunIds, true);
|
|
const due: CandidateLane = { rows: dueRows, index: 0, advance: id => { passiveSweep.dueCursor = id; } };
|
|
lanes.push(due, sweep);
|
|
// Priority rotates across all three lanes, so each lane is first in one
|
|
// run of three. With one free slot (a long local export holds the other),
|
|
// a save that keeps finding its run's lock busy cannot take it every run.
|
|
const first = (passiveSweep.firstLane ?? 0) % lanes.length;
|
|
lanes.push(...lanes.splice(0, first));
|
|
passiveSweep.firstLane = (first + 1) % lanes.length;
|
|
} else lanes.push(sweep);
|
|
const work: Promise<void>[] = [];
|
|
// Several actions of one run (an export hold and a later no-replay hold)
|
|
// run one after another in that run's job. Skipping the later one would
|
|
// starve it while the earlier one stays selectable (for example while saved
|
|
// input waits for the later one to settle).
|
|
const groups = new Map<string, { run: typeof heartbeatRuns.$inferSelect; actions: (typeof issueRecoveryActions.$inferSelect)[] }>();
|
|
const join = (lane: CandidateLane) => {
|
|
for (; lane.index < lane.rows.length; lane.index += 1) {
|
|
const { action, run } = lane.rows[lane.index]!;
|
|
const group = groups.get(run.id);
|
|
if (!group) return lane.rows[lane.index]!;
|
|
if (!group.actions.some(item => item.id === action.id)) group.actions.push(action);
|
|
lane.advance(action.id);
|
|
}
|
|
return undefined;
|
|
};
|
|
// Takes the lane's next run into a free slot. Actions of a run already taken
|
|
// (by any lane) join its job without a slot.
|
|
const take = (lane: CandidateLane) => {
|
|
const next = join(lane);
|
|
if (!next || pending.size >= 2) return false;
|
|
lane.index += 1;
|
|
lane.advance(next.action.id);
|
|
pending.add(next.run.id);
|
|
groups.set(next.run.id, { run: next.run, actions: [next.action] });
|
|
return true;
|
|
};
|
|
for (let taken = true; taken && pending.size < 2;) {
|
|
taken = false;
|
|
for (const lane of lanes) if (pending.size < 2 && take(lane)) taken = true;
|
|
}
|
|
for (const lane of lanes) join(lane);
|
|
const hasUnvisited = lanes.some(lane => lane.index < lane.rows.length);
|
|
for (const { run, actions } of groups.values()) {
|
|
const finish = beginIdleTrackedWork();
|
|
const physical = actions.some(action => ["scheduled", "running", "awaiting_evidence"].includes(readSystemRecovery(action.evidence)?.state ?? ""));
|
|
work.push((async () => { try {
|
|
for (const action of actions) await recoverRunAction(db, runtime, onRecovered, action, run);
|
|
} finally { pending.delete(run.id); finish(); onFinished?.(physical || hasUnvisited); } })());
|
|
}
|
|
return work;
|
|
}
|
|
|
|
/**
|
|
* Starts one saved-work recovery that the queue has not reached within its
|
|
* bound (the assigned-work sweep's repair). It runs outside the queue's two
|
|
* slots, so a queue held by slow or stuck attempts cannot keep a task waiting
|
|
* indefinitely, and it goes through the same ownership and attempt bookkeeping
|
|
* as the queue, so it never runs beside another attempt for the same run.
|
|
* Resolves once the attempt is started (or refused), not when it finishes.
|
|
*/
|
|
export async function startOverdueWorkspaceRecovery(db: Db, runtime: EnvironmentRuntimeService,
|
|
onRecovered: (run: typeof heartbeatRuns.$inferSelect) => Promise<void>, source: Source):
|
|
Promise<"started" | "in_flight" | "busy" | "not_eligible"> {
|
|
const overdue = overdueInFlight.get(db) ?? new Set<string>();
|
|
overdueInFlight.set(db, overdue);
|
|
if (overdue.has(source.runId) || workInFlight.get(db)?.has(source.runId)) return "in_flight";
|
|
if (overdue.size >= OVERDUE_WORKSPACE_RECOVERY_LIMIT) return "busy";
|
|
const [row] = await db.select({ action: issueRecoveryActions, run: heartbeatRuns }).from(issueRecoveryActions)
|
|
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.companyId, issueRecoveryActions.companyId), eq(heartbeatRuns.id, source.runId)))
|
|
.where(and(eq(issueRecoveryActions.companyId, source.companyId), eq(issueRecoveryActions.id, source.actionId),
|
|
eq(issueRecoveryActions.sourceIssueId, source.issueId), sql`${issueRecoveryActions.evidence}->>'runId' = ${source.runId}`,
|
|
inArray(heartbeatRuns.status, terminal), workspaceRecoverySourceEligible(),
|
|
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('scheduled', 'running', 'awaiting_evidence')`)).limit(1);
|
|
if (!row) return "not_eligible";
|
|
if (overdue.has(source.runId) || workInFlight.get(db)?.has(source.runId)) return "in_flight";
|
|
overdue.add(source.runId);
|
|
const finish = beginIdleTrackedWork();
|
|
void (async () => {
|
|
try { await recoverRunAction(db, runtime, onRecovered, row.action, row.run); }
|
|
finally { overdue.delete(source.runId); finish(); }
|
|
})();
|
|
return "started";
|
|
}
|
|
|
|
async function recoverRunAction(db: Db, runtime: EnvironmentRuntimeService,
|
|
onRecovered: (run: typeof heartbeatRuns.$inferSelect) => Promise<void>,
|
|
action: typeof issueRecoveryActions.$inferSelect, run: typeof heartbeatRuns.$inferSelect) {
|
|
try {
|
|
await recoverRetainedWorkspace({ db, environmentRuntime: runtime, companyId: action.companyId, issueId: action.sourceIssueId, runId: run.id, actionId: action.id });
|
|
const [updated] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action.id));
|
|
const recovered = readWorkspaceRecovery(updated?.evidence ?? null)?.status === "recovered";
|
|
const exhausted = readSystemRecovery(updated?.evidence)?.state === "exhausted";
|
|
if (recovered || exhausted) {
|
|
const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
|
|
if (current) {
|
|
if (recovered) {
|
|
await resumePreservedResultFinalization(db, current.companyId, current.id);
|
|
await scheduleRecoveredWorkspaceCleanup(db, current);
|
|
}
|
|
// Only normal dispatch can consume existing authorized input. An
|
|
// exhausted export permits evaluation of an isolated continuation,
|
|
// never another export attempt or a replay of an old assignment.
|
|
await onRecovered(current);
|
|
}
|
|
}
|
|
} catch (error) { logger.warn({ err: error, runId: run.id, actionId: action.id }, "Retained workspace remains pending"); }
|
|
}
|