mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
<!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before a remote run starts, workspace preparation attaches the verified saved work of earlier runs of the same task. > - Preparation failed with `remote_workspace_recovery_receipt_invalid` when any earlier run had a recovery receipt that it could not verify. > - A refusal cannot change the saved state, so every Retry failed in the same way, and the task stayed blocked. Recoveries must fall back automatically. > - This pull request skips an unverifiable source. It attaches nothing unverified and starts from the workspace's newest verified save. > - The benefit is that Retry and continuations start again, while strict receipt checks still decide what is attached, and no saved work is changed. ## Linked Issues or Issue Description Refs #20631 **What happened?** On staging, after the deploy that included #20631, a user pressed Retry on a task whose previous run failed with `native_controller_recovery_missing_evidence:runner_not_settled`. The new run failed in setup in under a minute with `setup_failed` / `remote_workspace_recovery_receipt_invalid` (`executionRecovery: bootstrap`, `providerWorkStarted: false`). The task stayed blocked. Another Retry would fail the same way. #20631 accepted one shape of this refusal (a source that saved only the snapshot its own restore loaded). This task had a different shape. **Expected behavior** A Retry starts. A source whose receipt cannot be verified is not attached, and its saved state stays intact for workspace recovery. **Steps to reproduce** 1. Let a remote native run fail before provider work and leave a recovery receipt or restore hold that does not verify (for example a receipt on an unsaved reference, a receipt from another epoch, or a saved reference without a receipt). 2. Press Retry on the task. 3. See the new run fail in setup with `remote_workspace_recovery_receipt_invalid`. **Paperclip version or commit** Staging deploy that includes `e3bc4ca798`. The gap remains on `master` at `a3b34a9d88`. ## What Changed - `attachPreservedRemoteWorkspaceSnapshots` no longer throws for an unverifiable source. Each of the five refusal shapes now skips the source and logs `remote_workspace_recovery_receipt_invalid` with a reason: `receipt_without_reference`, `reference_unreadable`, `receipt_on_unsaved_reference`, `receipt_unverified`, and `receipt_snapshot_missing` (was `remote_workspace_recovery_snapshot_missing`). - `retainRemoteRecoveryAttachments` no longer throws for a retained recovered copy that it cannot verify. It does not publish that copy again. The files stay in the workspace. - The writer check (`remote_workspace_recovery_writer_changed`), the workspace access check, and snapshot read and digest failures still refuse. - `doc/execution-semantics.md` describes the fallback rule. ## Verification - Red on unfixed code: `remote-workspace-recovery-receipt-fallback.test.ts` (moved out of `workspace-recovery-preparation.test.ts` to keep that file under 1,000 lines) "starts a Retry from the newest verified save when a predecessor has %s" (6 shapes: `receipt_without_reference`, `unreadable_reference_with_hold`, `receipt_on_prepared_reference`, `superseded_snapshot_with_hold`, `receipt_from_other_epoch`, `receipt_for_other_run`). All 6 failed with `remote_workspace_recovery_receipt_invalid`. The source is a native run that failed with `runner_not_settled`, and the writer is a Retry (`retryOfRunId`, `taskRetryRequestId`) that follows the real ancestry query. - Green: the same 6 tests pass. They also assert that nothing is attached, and that the source run row, its workspace operations, and its snapshot row are unchanged. - Updated safety tests: the canonical-attachment fault matrix now expects no attachment for receipt faults, and still expects a refusal for a missing or corrupt archive and for a wrong company or task. The retained-history test expects an unverifiable retained copy to be left out of the published list, while verified copies stay published. - `npx vitest run src/services/remote-workspace-recovery-receipt-fallback.test.ts src/services/workspace-recovery-preparation.test.ts`: 91 passed. - `npx tsc --noEmit -p server/tsconfig.json` passes. - `pnpm verify` returns exit 2, because a global doc input changed (`doc/execution-semantics.md`). Required CI supplies the broad coverage. - Not verified live on Daytona. I could not read the staging rows for this task: the browser had no free tab. The tests cover every refusal shape of this function, so the exact staging shape is one of them. ## Risks - Previous-release state: the rule reads the same rows as before (run reference, result hold, workspace operations, recovery actions, snapshots). Rows that earlier releases saved carry forward. No migration. A blocked task recovers on its next Retry or wake. - A Retry can now start without files from an unverifiable source. Those files are not lost: the source's snapshot, restore hold, receipts and recovery action stay unchanged, and workspace recovery still owns them. The skip is logged with a named reason. - A retained copy that fails verification is dropped from the published list. Its directory stays in the workspace and is no longer retired automatically. - A missing or corrupt snapshot archive still refuses. A permanently lost archive therefore still blocks attach. This PR does not change that. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
518 lines
39 KiB
TypeScript
518 lines
39 KiB
TypeScript
import { createHash, randomUUID } from "node:crypto";
|
|
import { createReadStream } from "node:fs";
|
|
import { mkdtemp, readdir, rm } from "node:fs/promises";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
import { and, eq, inArray, ne, or, sql } from "drizzle-orm";
|
|
import {
|
|
agentWakeupRequests, assets, environmentLeases, environments, executionWorkspaces, executionWorkspaceSnapshots, heartbeatRuns,
|
|
issueAttachments, issueRecoveryActions, issues, nativeRunFinalizations, statusDecisionEffects, toolInvocations, workspaceOperations, type Db,
|
|
} from "@paperclipai/db";
|
|
import { readRemoteWorkspaceAuthority, remoteWorkspaceRunReferenceSchema, type Environment, type EnvironmentLease, type ExecutionReconciliation } from "@paperclipai/shared";
|
|
import { createTarballFromDirectory } from "@paperclipai/adapter-utils/sandbox-managed-runtime";
|
|
import { logger } from "../middleware/logger.js";
|
|
import { assertWorkspaceFilePathAllowed } from "./workspace-file-resources.js";
|
|
import { getSecretProvider } from "../secrets/provider-registry.js";
|
|
import { getStorageService } from "../storage/index.js";
|
|
import { adapterExecutionControls } from "./adapter-execution-control.js";
|
|
import { persistActivity, publishActivity } from "./activity-log.js";
|
|
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
|
import type { EnvironmentRuntimeService } from "./environment-runtime.js";
|
|
import { resolveEnvironmentExecutionTarget } from "./environment-execution-target.js";
|
|
import { notifyRoutineExecutionStatus } from "./routine-execution-status-notifications.js";
|
|
import { notifyStatusCardIssueChange } from "./status-card-notifications.js";
|
|
import { hasRecoveredCleanupStopReceipt, hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
|
|
import { abandonUnverifiedRemoteWorkspaceEpoch } from "./remote-workspace-epoch-abandonment.js";
|
|
import { hasStopOnlyCleanup, prepareHistoricalSandboxStopVerification, prepareSandboxStopAndRetain, restoreSandboxStopAfterFailedRestart, settleStopOnlyCleanup } from "./sandbox-stop-and-retain.js";
|
|
import { holdOwnsBlockedStatus } from "./recovery-hold-blocked-status.js";
|
|
import { withNativeWorkspaceFinalizationOwnership } from "./native-runtime/native-workspace-finalization-ownership.js";
|
|
import { NATIVE_NO_REPLAY_HOLD_CAUSE, PERMANENT_SOURCE_RETRY_CLASSES, RETAINED_SOURCE_FALLBACK_CHECK_KEY, UNVERIFIABLE_SOURCE_CODES, isRetainedFileHold,
|
|
retainedHoldFallbackEligible } from "./workspace-recovery-fallback-discovery.js";
|
|
import { captureReadOnlyWorkspaceArchive, sandboxProviderFailureLabel } from "./remote-workspace-transport.js";
|
|
import { restoreRemoteWorkspaceArchive } from "./remote-workspace-snapshot-archive.js";
|
|
import { readSystemRecovery, type SystemRecoveryState } from "./system-recovery-state.js";
|
|
import { readWorkspaceRecovery, verifiedWorkspaceRepairReceipt } from "./workspace-recovery-receipt.js";
|
|
|
|
/**
|
|
* Automatic fallback for a retained-file hold whose recorded source can never
|
|
* be verified (for example a reusable sandbox chain that crosses tasks). The
|
|
* strict source check stays in force: unverified files are never applied. The
|
|
* task continues from its last committed workspace save instead, and the
|
|
* unverified files are attached to the task when they can still be read.
|
|
*/
|
|
export const RETAINED_SOURCE_FALLBACK_OUTCOME = "retained_source_unverifiable_continued_from_last_saved" as const;
|
|
export const RETAINED_SOURCE_FALLBACK_SCHEMA = "paperclip.retained-source-fallback.v1" as const;
|
|
export const SIBLING_WORKSPACE_RECOVERY_SCHEMA = "paperclip.sibling-workspace-recovery.v1" as const;
|
|
export const RETAINED_SOURCE_QUARANTINE_PHASE = "workspace_recovery_quarantine";
|
|
|
|
const terminal = ["failed", "interrupted", "timed_out", "cancelled", "succeeded"];
|
|
// A check deferred for a transient reason runs again after this delay, a bounded number of times.
|
|
const FALLBACK_RECHECK_MS = 5 * 60_000;
|
|
const FALLBACK_MAX_CHECKS = 12;
|
|
const outOfScopeReasons = ["not_unverifiable_source_hold", "not_remote_workspace_source"];
|
|
const transientReasons = ["writer_active", "workspace_operation_running", "execution_not_stopped", "side_effect_pending",
|
|
"allocation_in_use", "ownership_busy", "fallback_failed", "source_changed"];
|
|
const object = (value: unknown): Record<string, unknown> => value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
|
|
|
|
type Source = { companyId: string; issueId: string; runId: string; actionId: string };
|
|
type Ownership = { token: string; assertHeld(): Promise<void> };
|
|
type Quarantine = NonNullable<ExecutionReconciliation["retainedSourceFallback"]>["quarantine"];
|
|
export type RetainedSourceFallbackResult = { kind: "continued" | "settled" } | { kind: "ineligible"; reason: string };
|
|
|
|
export function isUnverifiableSourceFailure(failure: SystemRecoveryState["failure"] | null | undefined) {
|
|
return Boolean(failure && (UNVERIFIABLE_SOURCE_CODES as readonly string[]).includes(failure.code)
|
|
&& (PERMANENT_SOURCE_RETRY_CLASSES as readonly string[]).includes(failure.retryClass));
|
|
}
|
|
|
|
/** Text the agent receives with its continuation. It names no person who must act. */
|
|
export function retainedSourceFallbackNote(input: { runId: string; snapshotId: string | null; quarantine: Quarantine }) {
|
|
const from = input.snapshotId ? `the last saved version of the workspace (snapshot ${input.snapshotId})` : "the workspace's original source";
|
|
const files = input.quarantine.state === "captured"
|
|
? `They are attached to this task as ${input.quarantine.filename} (attachment ${input.quarantine.attachmentId}), without secrets, repository metadata or dependency directories. Inspect them and re-apply anything still needed deliberately.`
|
|
: "They could not be read, so they are not available.";
|
|
return `Paperclip could not verify the unsaved file changes of run ${input.runId}, so they were not applied. `
|
|
+ `This task continues from ${from}. ${files} Check which work from that run needs redoing. `
|
|
+ "Its completed external actions remain recorded and must not be repeated.";
|
|
}
|
|
|
|
async function uncertainExternalAction(db: Db, run: typeof heartbeatRuns.$inferSelect) {
|
|
for (const receipt of Object.values(object(run.resultJson?.apiToolReceipts)).map(object)) {
|
|
let result = object(receipt.result);
|
|
if (receipt.encryptedResult) {
|
|
try {
|
|
result = object(JSON.parse(await getSecretProvider("local_encrypted").resolveVersion({ material: object(receipt.encryptedResult), externalRef: null })));
|
|
} catch { return true; }
|
|
}
|
|
if (receipt.state !== "completed" || result.outcome === "unknown" || result.error === "api_outcome_unknown") return true;
|
|
// A definite client rejection had no effect. Anything else that is not ok is unknown.
|
|
const definiteRejection = result.ok === false && typeof result.status === "number" && result.status >= 400 && result.status < 500 && result.status !== 408;
|
|
if (result.ok !== true && !definiteRejection) return true;
|
|
}
|
|
// Semantic control-plane receipts are written in the same transaction as
|
|
// their effect, so they are never uncertain. Upstream tool calls can be.
|
|
const [invocation] = await db.select({ id: toolInvocations.id }).from(toolInvocations).where(and(
|
|
eq(toolInvocations.companyId, run.companyId), eq(toolInvocations.runId, run.id),
|
|
sql`${toolInvocations.riskLevel} is distinct from 'read'`,
|
|
inArray(toolInvocations.status, ["executing", "failed", "timed_out"]),
|
|
sql`${toolInvocations.startedAt} is not null`)).limit(1);
|
|
return Boolean(invocation);
|
|
}
|
|
|
|
/** Every fact the fallback depends on, read in the existing lock order:
|
|
* task, coordinator, run, recovery action, then workspace. */
|
|
async function inspectFallback(db: Db, input: Source, lock: boolean, requireStopped = lock) {
|
|
const one = async <T>(query: PromiseLike<T[]> & { for(mode: "update"): PromiseLike<T[]> }): Promise<T | null> =>
|
|
(await (lock ? query.for("update") : query))[0] ?? null;
|
|
const refuse = (reason: string) => ({ kind: "ineligible" as const, reason });
|
|
const issue = await one(db.select().from(issues).where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.issueId))).limit(1));
|
|
const coordinator = await one(db.select().from(nativeRunFinalizations).where(and(
|
|
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.runId, input.runId))).limit(1));
|
|
const run = await one(db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId))).limit(1));
|
|
const action = await one(db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, input.companyId),
|
|
eq(issueRecoveryActions.sourceIssueId, input.issueId), eq(issueRecoveryActions.id, input.actionId))).limit(1));
|
|
if (!issue || !run || !action) return refuse("source_changed");
|
|
if (object(action.evidence.retainedSourceFallback).schema === RETAINED_SOURCE_FALLBACK_SCHEMA) return { kind: "settled" as const };
|
|
const system = readSystemRecovery(action.evidence);
|
|
if (!isRetainedFileHold(action) || action.evidence.runId !== run.id
|
|
|| action.evidence.executionReconciliation || system?.state !== "exhausted" || system.sourceRunId !== run.id
|
|
|| !retainedHoldFallbackEligible(action.evidence, system)) return refuse("not_unverifiable_source_hold");
|
|
if (run.runtimeMode !== "native" || !terminal.includes(run.status) || !run.finishedAt
|
|
|| (run.nativeIssueId ?? run.issueId ?? run.contextSnapshot?.issueId) !== issue.id) return refuse("source_run_not_terminal");
|
|
// No second executor: the source run, its controller and its finalizer have stopped.
|
|
if (adapterExecutionControls.has(run.id) || (run.controllerLeaseExpiresAt && run.controllerLeaseExpiresAt > new Date())
|
|
|| coordinator?.leaseOwner || coordinator?.failureDetail?.successorRunId) return refuse("execution_not_stopped");
|
|
const parsed = remoteWorkspaceRunReferenceSchema.safeParse(run.runnerProfileJson?.remoteWorkspace);
|
|
if (!parsed.success || parsed.data.state !== "prepared") return refuse("not_remote_workspace_source");
|
|
const reference = parsed.data;
|
|
const workspace = await one(db.select().from(executionWorkspaces).where(and(
|
|
eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, reference.workspaceId))).limit(1));
|
|
let authority;
|
|
try { authority = readRemoteWorkspaceAuthority(workspace?.metadata); } catch { return refuse("workspace_authority_invalid"); }
|
|
// Only the exact epoch this run wrote is abandoned; a newer writer is never touched.
|
|
if (!workspace || !authority?.location || authority.writerRunId !== run.id || authority.epoch !== reference.epoch
|
|
|| authority.snapshotId !== reference.snapshotId || authority.location.leaseId !== reference.leaseId
|
|
|| authority.location.providerLeaseId !== reference.providerLeaseId || authority.location.remoteCwd !== reference.remoteCwd) {
|
|
return refuse("workspace_authority_changed");
|
|
}
|
|
if (reference.snapshotId) {
|
|
const [snapshot] = await db.select({ id: executionWorkspaceSnapshots.id }).from(executionWorkspaceSnapshots).where(and(
|
|
eq(executionWorkspaceSnapshots.companyId, input.companyId), eq(executionWorkspaceSnapshots.executionWorkspaceId, workspace.id),
|
|
eq(executionWorkspaceSnapshots.id, reference.snapshotId), eq(executionWorkspaceSnapshots.status, "committed"))).limit(1);
|
|
if (!snapshot) return refuse("last_save_missing");
|
|
}
|
|
// A live writer keeps its files. A queued turn of this task that never
|
|
// started cannot write: this hold gates its dispatch.
|
|
const [writer] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, input.companyId), ne(heartbeatRuns.id, run.id), inArray(heartbeatRuns.status, ["queued", "running"]),
|
|
sql`not coalesce(${heartbeatRuns.status} = 'queued' and ${heartbeatRuns.startedAt} is null
|
|
and ${heartbeatRuns.processPid} is null and ${heartbeatRuns.processGroupId} is null, false)`,
|
|
or(eq(heartbeatRuns.nativeIssueId, issue.id), eq(heartbeatRuns.issueId, issue.id),
|
|
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}`,
|
|
sql`${heartbeatRuns.contextSnapshot}->>'executionWorkspaceId' = ${workspace.id}`))).limit(1);
|
|
if (writer) return refuse("writer_active");
|
|
// No second executor: every allocation of the source run has a provider stop receipt.
|
|
if (requireStopped) {
|
|
const leases = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, input.companyId),
|
|
eq(environmentLeases.heartbeatRunId, run.id)));
|
|
if (!leases.some(lease => lease.id === reference.leaseId)
|
|
|| leases.some(lease => !hasRemoteTerminationReceipt(lease) && !hasRecoveredCleanupStopReceipt(lease))) return refuse("execution_not_stopped");
|
|
}
|
|
const [operation] = await db.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and(
|
|
eq(workspaceOperations.companyId, input.companyId), eq(workspaceOperations.executionWorkspaceId, workspace.id),
|
|
eq(workspaceOperations.status, "running"))).limit(1);
|
|
if (operation) return refuse("workspace_operation_running");
|
|
// Pending and uncertain effects keep the existing reconciliation path.
|
|
const [effect] = coordinator?.decisionId ? await db.select({ id: statusDecisionEffects.id }).from(statusDecisionEffects).where(and(
|
|
eq(statusDecisionEffects.companyId, input.companyId), eq(statusDecisionEffects.issueId, issue.id),
|
|
eq(statusDecisionEffects.decisionId, coordinator.decisionId), inArray(statusDecisionEffects.deliveryState, ["pending", "failed"]))).limit(1) : [];
|
|
if (effect) return refuse("side_effect_pending");
|
|
if (await uncertainExternalAction(db, run)) return refuse("external_action_uncertain");
|
|
return { kind: "eligible" as const, issue, coordinator, run, action, system, reference, workspace, authority };
|
|
}
|
|
|
|
/** The attachment follows the workspace file policy: secrets (such as `.env`
|
|
* and keys), repository metadata and dependency or build directories are omitted. */
|
|
async function omitDeniedWorkspaceFiles(root: string, relative = "") {
|
|
for (const entry of await readdir(path.join(root, relative), { withFileTypes: true })) {
|
|
const child = relative ? `${relative}/${entry.name}` : entry.name;
|
|
let allowed = true;
|
|
try { assertWorkspaceFilePathAllowed(child); } catch { allowed = false; }
|
|
if (!allowed) await rm(path.join(root, child), { recursive: true, force: true });
|
|
else if (entry.isDirectory() && !entry.isSymbolicLink()) await omitDeniedWorkspaceFiles(root, child);
|
|
}
|
|
}
|
|
|
|
function unreadable(reason: string): Quarantine {
|
|
return { state: "unreadable", reason: /^[a-z][a-z0-9_]{0,47}$/.test(reason) ? reason : "operation_failed" };
|
|
}
|
|
|
|
/** Reads the retained root without writing to the sandbox and attaches it to
|
|
* the task as an unverified archive. Never applied, never a recovery receipt.
|
|
* A root that cannot be read is recorded as such. Storage and database
|
|
* failures after a successful read throw, so the fallback retries later. */
|
|
async function quarantineRetainedRoot(db: Db, input: Source & { environmentRuntime: EnvironmentRuntimeService; ownership: Ownership },
|
|
source: Extract<Awaited<ReturnType<typeof inspectFallback>>, { kind: "eligible" }>): Promise<Quarantine> {
|
|
const [existing] = await db.select({ metadata: workspaceOperations.metadata }).from(workspaceOperations).where(and(
|
|
eq(workspaceOperations.companyId, input.companyId), eq(workspaceOperations.heartbeatRunId, input.runId),
|
|
eq(workspaceOperations.phase, RETAINED_SOURCE_QUARANTINE_PHASE), eq(workspaceOperations.status, "succeeded"),
|
|
sql`${workspaceOperations.metadata}->>'recoveryActionId' = ${input.actionId}`)).limit(1);
|
|
const saved = object(existing?.metadata?.quarantine);
|
|
if (saved.state === "captured") return saved as Quarantine;
|
|
const { reference } = source;
|
|
const [lease] = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, input.companyId),
|
|
eq(environmentLeases.id, reference.leaseId), eq(environmentLeases.heartbeatRunId, input.runId)));
|
|
if (!lease?.environmentId || lease.providerLeaseId !== reference.providerLeaseId) return unreadable("allocation_missing");
|
|
if (hasRemoteTerminationReceipt(lease) && object(lease.metadata?.remoteExecutionTermination).state === "destroyed")
|
|
return unreadable("sandbox_destroyed");
|
|
const [environment] = await db.select().from(environments).where(eq(environments.id, lease.environmentId));
|
|
if (!environment) return unreadable("environment_missing");
|
|
const scratch = await mkdtemp(path.join(os.tmpdir(), "paperclip-quarantine-"));
|
|
try {
|
|
const tarball = path.join(scratch, "unverified.tar");
|
|
try {
|
|
const target = await resolveEnvironmentExecutionTarget({ db, companyId: input.companyId, adapterType: "paperclip_runner",
|
|
environment, leaseId: lease.id, leaseMetadata: lease.metadata, lease: lease as EnvironmentLease, environmentRuntime: input.environmentRuntime });
|
|
if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runner) return unreadable("target_unavailable");
|
|
await input.ownership.assertHeld();
|
|
const captured = await captureReadOnlyWorkspaceArchive({ target, remoteCwd: reference.remoteCwd, operationId: randomUUID() });
|
|
const files = path.join(scratch, "files");
|
|
try {
|
|
await restoreRemoteWorkspaceArchive({ archivePath: captured.archivePath, targetDir: files, expectedManifestDigest: captured.manifestDigest });
|
|
} finally { await captured.cleanup(); }
|
|
await omitDeniedWorkspaceFiles(files);
|
|
await createTarballFromDirectory({ localDir: files, archivePath: tarball });
|
|
} catch (error) {
|
|
const message = error instanceof Error ? error.message : "";
|
|
const reason = /^remote_workspace_readonly_capture_failed: .*reason=([a-z_]+)/.exec(message)?.[1];
|
|
logger.warn({ err: error, runId: input.runId }, "Unverified retained files could not be read; continuing from the last save");
|
|
return unreadable(reason ?? sandboxProviderFailureLabel(error).label);
|
|
}
|
|
const hash = createHash("sha256");
|
|
let byteSize = 0;
|
|
for await (const chunk of createReadStream(tarball)) { hash.update(chunk as Buffer); byteSize += (chunk as Buffer).length; }
|
|
const sha256 = hash.digest("hex");
|
|
const filename = `unverified-files-run-${input.runId}.tar`;
|
|
await input.ownership.assertHeld();
|
|
const stored = await getStorageService().putFile({ companyId: input.companyId, namespace: "workspace-recovery-quarantine",
|
|
objectKey: `${input.companyId}/workspace-recovery-quarantine/${input.runId}/${sha256}.tar`, originalFilename: filename,
|
|
contentType: "application/x-tar", body: createReadStream(tarball), byteSize, sha256 });
|
|
await input.ownership.assertHeld();
|
|
return await db.transaction(async tx => {
|
|
const [asset] = await tx.insert(assets).values({ companyId: input.companyId, provider: stored.provider, objectKey: stored.objectKey,
|
|
contentType: "application/x-tar", byteSize, sha256, originalFilename: filename }).returning();
|
|
const [attachment] = await tx.insert(issueAttachments).values({ companyId: input.companyId, issueId: input.issueId,
|
|
assetId: asset!.id }).returning();
|
|
const quarantine: Quarantine = { state: "captured", attachmentId: attachment!.id, filename, sha256, byteSize };
|
|
await tx.insert(workspaceOperations).values({ companyId: input.companyId, issueId: input.issueId, heartbeatRunId: input.runId,
|
|
executionWorkspaceId: source.workspace.id, phase: RETAINED_SOURCE_QUARANTINE_PHASE, status: "succeeded", finishedAt: new Date(),
|
|
metadata: { schema: RETAINED_SOURCE_FALLBACK_SCHEMA, recoveryActionId: input.actionId, verified: false, applied: false,
|
|
leaseId: lease.id, providerLeaseId: lease.providerLeaseId, remoteCwd: reference.remoteCwd, epoch: reference.epoch, quarantine } });
|
|
return quarantine;
|
|
});
|
|
} finally { await rm(scratch, { recursive: true, force: true }).catch(() => undefined); }
|
|
}
|
|
|
|
async function sandboxHeldByAnotherLease(db: Db, lease: Pick<typeof environmentLeases.$inferSelect, "id" | "provider" | "providerLeaseId">) {
|
|
if (!lease.provider || !lease.providerLeaseId) return false;
|
|
const [other] = await db.select({ id: environmentLeases.id }).from(environmentLeases).where(and(
|
|
ne(environmentLeases.id, lease.id), eq(environmentLeases.provider, lease.provider), eq(environmentLeases.providerLeaseId, lease.providerLeaseId),
|
|
inArray(environmentLeases.status, ["active", "pending_cleanup"]))).limit(1);
|
|
return Boolean(other);
|
|
}
|
|
|
|
/** Obtains the provider stop receipt for the source allocation through the
|
|
* same stop-only cleanup as normal recovery. A sandbox that another lease holds
|
|
* is never stopped from here. */
|
|
async function confirmSourceAllocationStopped(db: Db, input: Source & { environmentRuntime: EnvironmentRuntimeService; ownership: Ownership },
|
|
leaseId: string): Promise<"stopped" | "in_use" | "unconfirmed"> {
|
|
const [lease] = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, input.companyId),
|
|
eq(environmentLeases.id, leaseId), eq(environmentLeases.heartbeatRunId, input.runId)));
|
|
if (!lease?.environmentId || !lease.provider || !lease.providerLeaseId) return "unconfirmed";
|
|
if (hasRemoteTerminationReceipt(lease) || hasRecoveredCleanupStopReceipt(lease)) return "stopped";
|
|
if (await sandboxHeldByAnotherLease(db, lease)) return "in_use";
|
|
const [environment] = await db.select().from(environments).where(eq(environments.id, lease.environmentId));
|
|
if (!environment) return "unconfirmed";
|
|
await input.ownership.assertHeld();
|
|
const pending = await db.transaction(async tx => {
|
|
const [fresh] = await tx.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, input.companyId),
|
|
eq(environmentLeases.id, lease.id))).for("update");
|
|
if (!fresh || fresh.metadata?.pendingCleanupInFlight === true) return null;
|
|
const prepared = hasStopOnlyCleanup(fresh) ? fresh : fresh.status === "released"
|
|
? await prepareHistoricalSandboxStopVerification(tx as unknown as Db, fresh)
|
|
: await prepareSandboxStopAndRetain(tx as unknown as Db, fresh);
|
|
if (!prepared) return null;
|
|
const [claimed] = await tx.update(environmentLeases).set({ metadata: sql`${environmentLeases.metadata} ||
|
|
${JSON.stringify({ pendingCleanupAttemptId: input.ownership.token, pendingCleanupInFlight: true,
|
|
pendingCleanupLeaseExpiresAtMs: Date.now() + 15 * 60_000 })}::jsonb` }).where(eq(environmentLeases.id, lease.id)).returning();
|
|
return claimed ?? null;
|
|
});
|
|
if (!pending) return "unconfirmed";
|
|
const attemptId = input.ownership.token;
|
|
const settled = await input.environmentRuntime.retryPendingSandboxTeardown({ environment: environment as unknown as Environment,
|
|
lease: pending as EnvironmentLease })
|
|
.then(receipt => settleStopOnlyCleanup(db, pending, { attemptId, receipt }),
|
|
async () => await restoreSandboxStopAfterFailedRestart(db, pending, { attemptId }) ?? await settleStopOnlyCleanup(db, pending, { attemptId }));
|
|
return settled && hasRemoteTerminationReceipt(settled) ? "stopped" : "unconfirmed";
|
|
}
|
|
|
|
/**
|
|
* Settles a retained-file hold whose source can never be verified. In one
|
|
* transaction it abandons the unverified workspace epoch, resolves the hold
|
|
* with a named outcome, and hands the task to its normal continuation from the
|
|
* last committed save. Waiting user input continues the task itself; otherwise
|
|
* the recovery delivery starts one continuation. Bounded and idempotent: a
|
|
* settled hold is never revisited, and a lost race leaves the hold unchanged.
|
|
*/
|
|
export async function continueFromLastSavedWorkspace(input: Source & {
|
|
db: Db; environmentRuntime: EnvironmentRuntimeService; ownership: Ownership;
|
|
}): Promise<RetainedSourceFallbackResult> {
|
|
const { db } = input;
|
|
const initial = await inspectFallback(db, input, false);
|
|
if (initial.kind !== "eligible") return initial;
|
|
// An unstopped source in a sandbox another lease holds waits; nothing is read or stopped.
|
|
const [sourceLease] = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, input.companyId),
|
|
eq(environmentLeases.id, initial.reference.leaseId), eq(environmentLeases.heartbeatRunId, input.runId)));
|
|
if (sourceLease && !hasRemoteTerminationReceipt(sourceLease) && !hasRecoveredCleanupStopReceipt(sourceLease)
|
|
&& await sandboxHeldByAnotherLease(db, sourceLease)) return { kind: "ineligible", reason: "allocation_in_use" };
|
|
// Read before stopping: a stopped sandbox cannot be read without resuming it.
|
|
const quarantine = await quarantineRetainedRoot(db, input, initial);
|
|
const stop = await confirmSourceAllocationStopped(db, input, initial.reference.leaseId);
|
|
if (stop !== "stopped") return { kind: "ineligible", reason: stop === "in_use" ? "allocation_in_use" : "execution_not_stopped" };
|
|
await input.ownership.assertHeld();
|
|
const outcome = await db.transaction(async tx => {
|
|
const current = await inspectFallback(tx as unknown as Db, input, true);
|
|
if (current.kind !== "eligible") return { result: current, publication: null };
|
|
if (current.system.generationId !== initial.system.generationId || current.system.revision !== initial.system.revision)
|
|
return { result: { kind: "ineligible" as const, reason: "source_changed" }, publication: null };
|
|
const { issue, run, action, reference } = current;
|
|
const abandoned = await abandonUnverifiedRemoteWorkspaceEpoch(tx, { companyId: input.companyId, workspaceId: reference.workspaceId,
|
|
runId: run.id, epoch: reference.epoch, snapshotId: reference.snapshotId, location: current.authority.location! });
|
|
const [waitingInput] = await tx.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.companyId, input.companyId), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
eq(agentWakeupRequests.requestedByActorType, "user"), sql`${agentWakeupRequests.payload}->>'issueId' = ${issue.id}`)).limit(1);
|
|
const fallback: NonNullable<ExecutionReconciliation["retainedSourceFallback"]> = { schema: RETAINED_SOURCE_FALLBACK_SCHEMA,
|
|
outcome: RETAINED_SOURCE_FALLBACK_OUTCOME, workspaceId: reference.workspaceId, abandonedEpoch: reference.epoch,
|
|
epoch: abandoned.authority.epoch, snapshotId: reference.snapshotId, quarantine };
|
|
const decision: ExecutionReconciliation = { runId: run.id, providerStopped: true, actionOutcome: "completed",
|
|
outcomeEvidence: retainedSourceFallbackNote({ runId: run.id, snapshotId: reference.snapshotId, quarantine }),
|
|
// The workspace repair that a restore failure requires before continuing:
|
|
// the unverified epoch is abandoned and the next writer restores the last
|
|
// committed save into a new root.
|
|
workspaceRepairEvidence: `Paperclip abandoned unverified workspace epoch ${reference.epoch} of run ${run.id}; `
|
|
+ `the continuation restores ${reference.snapshotId ? `committed snapshot ${reference.snapshotId}` : "the workspace source"} into a new root.`,
|
|
retainedSourceFallback: fallback };
|
|
const now = new Date();
|
|
// A Blocked status belongs to this hold only while its own receipt (or, for
|
|
// holds saved before receipts existed, a block set by this run's terminal
|
|
// failure that nothing changed since) still names it. Otherwise a newer
|
|
// decision owns the block: it is kept and no continuation is started.
|
|
const ownsBlock = holdOwnsBlockedStatus(action, issue, run);
|
|
const dispatchable = issue.status !== "blocked" || ownsBlock;
|
|
const userText = quarantine.state === "captured"
|
|
? "Paperclip could not verify the previous run's unsaved files, so the task continues from the last saved version. Those files are attached to the task, unapplied."
|
|
: "Paperclip could not verify the previous run's unsaved files, so the task continues from the last saved version. Those files could not be read.";
|
|
const evidence = { ...action.evidence, automaticRecovery: undefined, [RETAINED_SOURCE_FALLBACK_CHECK_KEY]: undefined,
|
|
retainedSourceFallback: { ...fallback, failureCode: current.system.failure?.code ?? null, recordedAt: now.toISOString() },
|
|
workspaceRecovery: { ...(readWorkspaceRecovery(action.evidence) ?? {}), message: userText, retryable: false },
|
|
executionReconciliation: { ...decision, actorId: "workspace-recovery", recordedAt: now.toISOString() },
|
|
// Saved user input continues the task through normal admission.
|
|
continuationDelivery: waitingInput || !dispatchable ? "not_requested" : "pending" };
|
|
await tx.update(issueRecoveryActions).set({ status: "resolved", outcome: "restored", resolvedAt: now, updatedAt: now,
|
|
ownerType: "system", ownerAgentId: null, ownerUserId: null, returnOwnerAgentId: action.returnOwnerAgentId ?? run.agentId,
|
|
resolutionNote: RETAINED_SOURCE_FALLBACK_OUTCOME, nextAction: userText, wakePolicy: null, monitorPolicy: null, evidence,
|
|
}).where(and(eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.id, action.id)));
|
|
// The abandoned source must never be replayed or replaced automatically.
|
|
await tx.update(nativeRunFinalizations).set({ failureDetail: sql`coalesce(${nativeRunFinalizations.failureDetail}, '{}'::jsonb)
|
|
|| ${JSON.stringify({ replacementDenied: RETAINED_SOURCE_FALLBACK_OUTCOME })}::jsonb` }).where(and(
|
|
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.runId, run.id)));
|
|
if (ownsBlock) {
|
|
await tx.update(issues).set({ status: "todo", unblockDescriptor: null, blockedTransitionAt: null,
|
|
blockedOwnerNotifiedAt: null, updatedAt: now }).where(eq(issues.id, issue.id));
|
|
}
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionContinuation);
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors);
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
|
|
await notifyStatusCardIssueChange(tx, input.companyId);
|
|
await notifyRoutineExecutionStatus(tx, issue.id, input.companyId);
|
|
const publication = await persistActivity(tx as unknown as Db, { companyId: input.companyId, actorType: "system",
|
|
actorId: "workspace-recovery", action: "issue.workspace_recovery_continued_from_last_saved", entityType: "issue",
|
|
entityId: issue.id, runId: run.id, details: { recoveryActionId: action.id, failureCode: current.system.failure?.code ?? null,
|
|
...fallback } });
|
|
return { result: { kind: "continued" as const }, publication };
|
|
});
|
|
if (outcome.publication) {
|
|
try { publishActivity(outcome.publication.publication); } catch { logger.warn({ runId: input.runId }, "Recovery activity delivery deferred"); }
|
|
}
|
|
return outcome.result;
|
|
}
|
|
|
|
/** Records why the fallback could not settle this generation yet. A transient
|
|
* reason is checked again after a delay, a bounded number of times; a
|
|
* permanent one waits for new input, Retry or a new recovery generation. */
|
|
export async function recordRetainedSourceFallbackDeferral(db: Db, input: Source, reason: string) {
|
|
const [row] = await db.select({ evidence: issueRecoveryActions.evidence }).from(issueRecoveryActions).where(and(
|
|
eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, input.issueId),
|
|
eq(issueRecoveryActions.id, input.actionId)));
|
|
const system = readSystemRecovery(row?.evidence);
|
|
// A hold outside the fallback's scope keeps its evidence unchanged.
|
|
if (!row || !system || row.evidence.retainedSourceFallback || outOfScopeReasons.includes(reason)) return;
|
|
const prior = object(row.evidence[RETAINED_SOURCE_FALLBACK_CHECK_KEY]);
|
|
const checks = prior.generationId === system.generationId && typeof prior.checks === "number" ? prior.checks + 1 : 1;
|
|
const transient = transientReasons.includes(reason) && checks < FALLBACK_MAX_CHECKS;
|
|
const marker = { generationId: system.generationId, reason, checks, checkedAt: Date.now(),
|
|
nextCheckAt: transient ? Date.now() + FALLBACK_RECHECK_MS : null };
|
|
await db.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence} ||
|
|
jsonb_build_object(${RETAINED_SOURCE_FALLBACK_CHECK_KEY}::text, ${JSON.stringify(marker)}::jsonb)` }).where(and(
|
|
eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.id, input.actionId),
|
|
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,generationId}' = ${system.generationId}`,
|
|
sql`${issueRecoveryActions.evidence}->'retainedSourceFallback' is null`));
|
|
}
|
|
|
|
/**
|
|
* The recovery worker's entry for an exhausted hold: settle it when a sibling
|
|
* saved its files, or continue from the last save when its source is refused
|
|
* permanently or its stop stayed unproven through every bounded re-arm. Runs under the workspace ownership lock, like any physical
|
|
* recovery attempt. Returns whether the hold is settled.
|
|
*/
|
|
export async function settleExhaustedRetainedHold(input: Source & { db: Db; environmentRuntime: EnvironmentRuntimeService },
|
|
state: SystemRecoveryState, options: { fallback: boolean } = { fallback: true }) {
|
|
const { db } = input;
|
|
const sibling = await settleSiblingRecoveredHold(db, input);
|
|
if (sibling === "settled") return true;
|
|
const [row] = options.fallback ? await db.select({ evidence: issueRecoveryActions.evidence }).from(issueRecoveryActions).where(and(
|
|
eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, input.issueId),
|
|
eq(issueRecoveryActions.id, input.actionId))) : [];
|
|
if (!options.fallback || !retainedHoldFallbackEligible(row?.evidence, state)) {
|
|
// A sibling marked recovered without a receipt that still verifies cannot
|
|
// settle this hold. Record it so discovery does not select it again.
|
|
if (sibling === "unverified") await recordRetainedSourceFallbackDeferral(db, input, "sibling_receipt_unverified");
|
|
return false;
|
|
}
|
|
let reason: string;
|
|
try {
|
|
const owned = await withNativeWorkspaceFinalizationOwnership({ db, companyId: input.companyId, runId: input.runId,
|
|
retainedRepairActionId: input.actionId }, ownership => continueFromLastSavedWorkspace({ ...input, ownership }));
|
|
if (owned.acquired && owned.value.kind !== "ineligible") return true;
|
|
reason = !owned.acquired ? "ownership_busy" : owned.value.kind === "ineligible" ? owned.value.reason : "fallback_failed";
|
|
} catch (error) {
|
|
logger.warn({ err: error, runId: input.runId }, "Continuing from the last save was deferred; the hold remains");
|
|
reason = "fallback_failed";
|
|
}
|
|
logger.info({ runId: input.runId, reason }, "Retained source fallback deferred");
|
|
await recordRetainedSourceFallbackDeferral(db, input, reason);
|
|
return false;
|
|
}
|
|
|
|
/**
|
|
* An exhausted hold for a run whose files another recovery action already
|
|
* saved (a verified receipt bound to that sibling) has nothing left to do.
|
|
* Settle it as recovered instead of handing a spent budget to a person.
|
|
*/
|
|
export async function settleSiblingRecoveredHold(db: Db, input: Source) {
|
|
const settled = await db.transaction(async tx => {
|
|
const [issue] = await tx.select().from(issues).where(and(eq(issues.companyId, input.companyId),
|
|
eq(issues.id, input.issueId))).for("update");
|
|
const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId),
|
|
eq(heartbeatRuns.id, input.runId))).for("update");
|
|
const [action] = await tx.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, input.companyId),
|
|
eq(issueRecoveryActions.sourceIssueId, input.issueId), eq(issueRecoveryActions.id, input.actionId))).for("update");
|
|
if (!issue || !run || !action || action.evidence.runId !== run.id) return null;
|
|
if (object(action.evidence.siblingWorkspaceRecovery).schema === SIBLING_WORKSPACE_RECOVERY_SCHEMA) return "already" as const;
|
|
// isRetainedFileHold also accepts export holds; only the no-replay cause
|
|
// may release the status here. An export hold leaves it to its result.
|
|
const noReplayHold = action.cause === NATIVE_NO_REPLAY_HOLD_CAUSE && isRetainedFileHold(action);
|
|
const system = readSystemRecovery(action.evidence);
|
|
// A no-replay hold is recorded resolved-but-blocked. It blocks the task
|
|
// until its run's files are saved, so a sibling's save settles it too.
|
|
if ((!["active", "escalated"].includes(action.status) && !noReplayHold)
|
|
|| system?.state !== "exhausted" || system.sourceRunId !== run.id) return null;
|
|
const receiptId = await verifiedWorkspaceRepairReceipt(tx as unknown as Db, run);
|
|
const [receipt] = receiptId ? await tx.select({ metadata: workspaceOperations.metadata }).from(workspaceOperations).where(and(
|
|
eq(workspaceOperations.companyId, input.companyId), eq(workspaceOperations.id, receiptId))) : [];
|
|
const siblingActionId = receipt?.metadata?.recoveryActionId;
|
|
if (!receiptId || typeof siblingActionId !== "string" || siblingActionId === action.id) {
|
|
const [recoveredSibling] = await tx.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and(
|
|
eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, input.issueId),
|
|
ne(issueRecoveryActions.id, action.id), sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`,
|
|
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' = 'complete'`,
|
|
sql`${issueRecoveryActions.evidence}#>>'{workspaceRecovery,status}' = 'recovered'`)).limit(1);
|
|
return recoveredSibling ? "unverified" as const : null;
|
|
}
|
|
const now = new Date();
|
|
const prior = readWorkspaceRecovery(action.evidence);
|
|
await tx.update(issueRecoveryActions).set({ status: "resolved", outcome: "restored", resolvedAt: now, updatedAt: now,
|
|
ownerType: "system", ownerAgentId: null, ownerUserId: null, nextAction: "Saved work recovered.",
|
|
resolutionNote: "workspace_recovered_by_sibling_action", wakePolicy: null, monitorPolicy: null,
|
|
// With the files saved, a no-replay hold no longer blocks the task.
|
|
// Settling starts no turn; the run's own result decides what continues.
|
|
evidence: { ...action.evidence, automaticRecovery: undefined,
|
|
workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: run.id, status: "recovered", stage: "saving_receipts",
|
|
attempt: prior?.attempt ?? system.attempts, attemptLimit: prior?.attemptLimit ?? 6, requestIds: prior?.requestIds ?? [], retryable: false },
|
|
siblingWorkspaceRecovery: { schema: SIBLING_WORKSPACE_RECOVERY_SCHEMA, receiptId, recoveryActionId: siblingActionId,
|
|
recordedAt: now.toISOString() } },
|
|
}).where(and(eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.id, action.id)));
|
|
// A no-replay hold that still owns the task's Blocked status releases it,
|
|
// so saved input and normal wakes continue the task. An export hold's
|
|
// status stays with its own result finalization. A newer decision's
|
|
// Blocked status is kept.
|
|
if (noReplayHold && holdOwnsBlockedStatus(action, issue, run)) {
|
|
await tx.update(issues).set({ status: "todo", unblockDescriptor: null, blockedTransitionAt: null,
|
|
blockedOwnerNotifiedAt: null, updatedAt: now }).where(eq(issues.id, issue.id));
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors);
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
|
|
await notifyStatusCardIssueChange(tx, input.companyId);
|
|
await notifyRoutineExecutionStatus(tx, issue.id, input.companyId);
|
|
}
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionContinuation);
|
|
return persistActivity(tx as unknown as Db, { companyId: input.companyId, actorType: "system", actorId: "workspace-recovery",
|
|
action: "issue.workspace_recovered", entityType: "issue", entityId: issue.id, runId: run.id,
|
|
details: { recoveryActionId: action.id, recoveredByRecoveryActionId: siblingActionId, receiptId,
|
|
summary: "Saved work recovered by another recovery of the same run." } });
|
|
});
|
|
if (settled === "unverified") return "unverified" as const;
|
|
if (settled && settled !== "already") {
|
|
try { publishActivity(settled.publication); } catch { logger.warn({ runId: input.runId }, "Recovery activity delivery deferred"); }
|
|
}
|
|
return settled ? "settled" as const : "none" as const;
|
|
}
|