mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 21:34:48 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Execution recovery continues tasks that a restart or a lost run left unfinished > - Recent recovery changes restored many held tasks at the same time, and each restored task got a queued run on its agent > - The agent's queue starts runs oldest first, so a person's Retry or chat message waited behind every older recovery run > - On staging, one agent had 131 queued runs, and person Retries waited more than 50 minutes > - This pull request paces how many recovery continuations enter an agent's queue, and it starts person-initiated work before system recovery work > - The benefit is that a burst of recovery cannot starve people, and every recovery run still runs ## Linked Issues or Issue Description Refs #21222, #21139, #21135 **What happened?** Staging deployed #21139 and #21135 at 22:53Z. Within one hour, the company had 201 queued runs and 78 running runs. 122 queued runs had the recovery wake reason `issue_recovery_action_restored`. Each one was on a different task. They all came from the reconciled-execution delivery pass (`source: execution.reconciled`), at a rate of 5 to 15 per minute from 22:57Z to 23:51Z. One agent had 131 queued runs. The oldest queued run had waited since 22:53Z. A person's Retry at 23:14Z was still queued 4 minutes later. Six person Retries on that agent, queued from 23:08Z to 23:14Z, were still queued at 00:04Z, more than 50 minutes behind recovery runs. At 00:04Z, that agent's newest start was a recovery run created at 22:57Z. In the last 400 runs, 25 succeeded. **Expected behavior** A person's Retry, chat message, or assignment starts before system recovery work on the same agent. Recovery work still drains at a steady rate. All recovery runs run eventually, and none is dropped. A task whose continuation waits shows that the task continues automatically. It does not show that the task is idle, and it does not ask for Retry. **Steps to reproduce** 1. Give an agent a concurrency limit of 1. 2. Queue many recovery continuations for the agent (`issue_recovery_action_restored`), for example after a server restart with many held tasks. 3. Click Retry on another task of the same agent. 4. See that the Retry stays queued until all older recovery runs start. **Paperclip version or commit** Staging build 2026.1106.0+2187.git.57aeda190 **Deployment mode** Hosted cloud (staging) **Agent adapter(s) involved** paperclip_runner (native and legacy runtimes). The problem is in the shared run queue, so it applies to all adapters. ## What Changed - New module `server/src/services/heartbeat/queued-run-dispatch-order.ts` decides the order in which an agent's queued runs start. Person-initiated runs start first: `task_retry`, external chat messages, and any wake that a user requested. Other work comes next, together with recovery runs for interruptions that ended in the last 30 minutes. Old-backlog recovery runs come last. Inside each class, the existing order applies: readiness, then priority, then age. Runs whose dependencies are not ready stay last. - Recovery runs get a bounded share. When the agent started 3 other runs after the oldest queued recovery run arrived and none of them was recovery work, the next slot goes to a recovery run. The share applies across the whole batch when one dispatch fills many slots. Thus people cannot starve recovery. - `startNextQueuedRunForAgent` in `server/src/services/heartbeat/queue.ts` uses the new order. It changes only the ranking. It does not change `claimQueuedRun`, the admission preflight, or the promoter. - `deliverReconciledExecutions` in `server/src/services/execution-recovery-resolution.ts` paces delivery. It leaves a continuation pending when the agent already has a queued recovery run, or when the agent's queue is at or above its concurrency limit. One pass delivers at most 5 continuations per company and at most 1 per agent. The pass reads pending continuations in pages and leaves out agents and companies that it already handled, so continuations for busy agents cannot hide continuations for other agents. A continuation that has waited 5 minutes or more is admitted even when ordinary work keeps the queue full, and it goes ahead of fresher continuations, longest wait first. Otherwise the most recent interruption goes first. The existing 15-second continuation schedule delivers the remaining items. This schedule runs only while pending continuations exist, so an idle server does not poll. - The assigned-work sweep (`server/src/services/recovery/assigned-work-sweep.ts`) counts a pending continuation as a pending wake for its task. Thus the invariant does not see a held-back task as stranded, and it does not enqueue a second wake. ## Verification - `pnpm exec vitest run src/services/heartbeat/queued-run-dispatch-order.test.ts src/__tests__/recovery-wake-herd.test.ts` (in `server/`): 17 tests pass. - Red/green: with the source changes reverted to master, all tests in `recovery-wake-herd.test.ts` fail. Each review fix also has a test that fails on the head before that fix. With this change, they pass. - `recovery-wake-herd.test.ts` covers these cases with embedded Postgres and the real heartbeat queue: - Six recovery wakes are queued, then a person's request. The first dispatch starts the person's run. All six recovery runs start later. - Four pending continuations for one agent: one pass delivers one. The others stay pending and are delivered on later passes. The continuation schedule stays armed while items are pending and returns null after all items are delivered. - A continuation stays pending while the agent's queue is at its concurrency limit. - The per-company cap applies, and the most recent interruption is delivered first. - A continuation that has waited past the bound is admitted while ordinary work keeps the queue full. - New interruptions arrive for the same agent on every 15-second pass, and the older continuation is still admitted within the bound. - Continuations for four busy agents (32 rows, more than one page) do not hide an idle agent's continuation. - A held-back continuation counts as a `queued_wake` pending intent for the assigned-work invariant, and the task shows `working` with `canRetry: false`. - `queued-run-dispatch-order.test.ts` simulates dispatch: a Retry starts ahead of 40 recovery wakes, and all 40 still run. With a person Retry arriving at every step, every window of 4 starts includes 1 recovery run. A fresh interruption's continuation is not put behind old-backlog recovery runs. When one dispatch fills 8 slots, the order is PPPRPPPR (P is a person run, R is a recovery run). - Related suites pass: fresh-continuation fallback, issue recovery actions, completed-turn cleanup hold, issue monitor scheduler, workspace recovery fallback, assigned-work invariant, execution control work, execution recovery resolution, and the heartbeat queue. - `node scripts/typecheck-layers.mjs` (in `server/`) passes. ## Risks - Behavior change: when a person's run waits behind recovery runs, it now starts first. Recovery runs start a little later, but they still get at least 1 of every 4 starts while people send work. - Delivery of continuations is slower during a burst: at most 1 per agent and 5 per company every 15 seconds. This is intentional. A continuation that waits shows as continuing automatically, and the invariant does not repair it twice. - Ranking adds up to three small indexed reads when an agent has more than one queued run. The reads run only when the queue has recovery runs or wake requests to classify. - No schema or migration changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs, and read-only staging API requests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
912 lines
44 KiB
TypeScript
912 lines
44 KiB
TypeScript
import { settleOperatorContainedNativeOwner } from "./native-runtime/native-failed-run-owner-release.js";
|
|
import { notifyRoutineExecutionStatus } from "./routine-execution-status-notifications.js";
|
|
import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js";
|
|
import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js";
|
|
import { notifyChatResponseAvailability } from "./chat-response-availability.js";
|
|
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
|
import { notifyExecutionRunWork } from "./execution-control-notifications.js";
|
|
import { hasWorkspaceRestoreFailure } from "@paperclipai/shared";
|
|
import { randomUUID } from "node:crypto";
|
|
import { canonicalExecutionUuid } from "./execution-identity.js";
|
|
import { conversationRecoveryActionPredicate, getConversationOwnershipBlocker } from "./conversation-continuation.js";
|
|
import { isInterruptedNativePreparationCandidate } from "./cancelled-native-startup.js";
|
|
import { persistActivity } from "./activity-log.js";
|
|
import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
|
|
import { logger } from "../middleware/logger.js";
|
|
import { and, asc, eq, exists, inArray, isNull, not, or, sql } from "drizzle-orm";
|
|
import {
|
|
agentWakeupRequests,
|
|
agents,
|
|
chatActions,
|
|
environmentLeases,
|
|
heartbeatRuns,
|
|
issueRecoveryActions,
|
|
issues,
|
|
nativeRunFinalizations,
|
|
type Db,
|
|
} from "@paperclipai/db";
|
|
import { conflict } from "../errors.js";
|
|
import { buildExecutionContinuation } from "./execution-continuation.js";
|
|
import {
|
|
EXECUTION_RECONCILIATION_CAUSES,
|
|
type ExecutionReconciliation,
|
|
} from "@paperclipai/shared";
|
|
import { parseIssueExecutionState } from "./issue-execution-policy.js";
|
|
import { isSupersededConversationRun } from "./agent-conversations.js";
|
|
import { hasRequiredWorkspaceRecovery } from "./workspace-restore-recovery-state.js";
|
|
import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
|
|
import { readSystemRecovery } from "./system-recovery-state.js";
|
|
import { provenGoneLegacyControllerBootId } from "./legacy-controller-lease.js";
|
|
import { continueFreshInTransaction, continueHeldExecutions, freshContinuationCheckDue, recordFreshContinuationCheck,
|
|
settleCoalescedFreshContinuationHolds } from "./fresh-continuation-fallback.js";
|
|
import { alias } from "drizzle-orm/pg-core";
|
|
import { RECOVERY_WAKE_REASONS } from "./heartbeat/queued-run-dispatch-order.js";
|
|
import { parseHeartbeatPolicy } from "./heartbeat/timer-policy.js";
|
|
|
|
export const RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS = 5;
|
|
export const RECOVERY_CONTINUATION_MAX_HOLD_MS = 5 * 60_000;
|
|
const RECOVERY_CONTINUATION_PAGES_PER_PASS = 8;
|
|
|
|
/** An operator records observed outcomes; this is not permission to blindly retry. */
|
|
/**
|
|
* The user-facing note for a settled recovery hold. A restart-recovery hold
|
|
* names its own cause: a workspace export retried after that hold does not
|
|
* mean the user must repair the workspace.
|
|
*/
|
|
export function executionRecoverySettlementNote(input: {
|
|
current: boolean; cause: string; workspaceRestoreFailed: boolean;
|
|
/** The hold's recorded trigger; a same-run retry is not a server restart. */
|
|
trigger?: unknown;
|
|
}): string {
|
|
if (!input.current) return "Recovery closed because the task's owner, execution, or status changed. No work was replayed.";
|
|
if (input.cause === "native_restart_recovery_blocked" && input.trigger === "same_run_retry") {
|
|
return "Automatic retry of this run stopped. Saved work is preserved; actions with unverified outcomes will not be repeated.";
|
|
}
|
|
if (input.cause === "native_restart_recovery_blocked") {
|
|
return "Automatic recovery stopped after a server restart because the previous execution could not be verified. Saved work is preserved; actions with unverified outcomes will not be repeated.";
|
|
}
|
|
return input.workspaceRestoreFailed
|
|
? "Workspace repair required. Verify safe staging or repair before continuing. Saved work and approval decisions remain in force."
|
|
: "Automatic recovery stopped. Recorded work is preserved; actions with unverified outcomes will not be repeated.";
|
|
}
|
|
|
|
type ExecutionReconciliationInput = {
|
|
db: Db;
|
|
companyId: string;
|
|
issueId: string;
|
|
agentId: string | null;
|
|
sourceRunId: unknown;
|
|
decision: ExecutionReconciliation | undefined;
|
|
/** Board-only source repair: never authorize a continuation of this run. */
|
|
workspaceRepairOnly?: boolean;
|
|
controllerContainment?: { actionId: string; actorId: string };
|
|
};
|
|
|
|
export async function validateExecutionReconciliation(input: ExecutionReconciliationInput) {
|
|
if (input.controllerContainment) {
|
|
// Validation and contained-owner settlement must roll back together even
|
|
// when a caller is not already inside the board route's transaction.
|
|
return input.db.transaction(tx => validateExecutionReconciliationInScope({ ...input, db: tx as unknown as Db }));
|
|
}
|
|
return validateExecutionReconciliationInScope(input);
|
|
}
|
|
|
|
async function validateExecutionReconciliationInScope(input: ExecutionReconciliationInput) {
|
|
const { db, companyId, issueId, agentId, decision } = input;
|
|
if (!decision || decision.runId !== input.sourceRunId || (!agentId && !input.workspaceRepairOnly)) {
|
|
throw conflict(
|
|
"Reconcile the recorded execution and its action outcomes before continuing this task.",
|
|
);
|
|
}
|
|
let operatorContained = false;
|
|
if (decision.controllerStopped !== undefined || decision.controllerAuthorityDigest !== undefined) {
|
|
if (!decision.controllerStopped || !decision.controllerAuthorityDigest || !input.controllerContainment
|
|
|| input.workspaceRepairOnly) throw conflict("Original-host controller containment requires its exact board recovery action.");
|
|
operatorContained = await settleOperatorContainedNativeOwner(db, {
|
|
companyId, issueId, runId: decision.runId, actionId: input.controllerContainment.actionId,
|
|
actorId: input.controllerContainment.actorId, authorityDigest: decision.controllerAuthorityDigest,
|
|
actionOutcome: decision.actionOutcome, outcomeEvidence: decision.outcomeEvidence,
|
|
});
|
|
if (!operatorContained) throw conflict("The recorded controller authority changed, is still leased, or its cleanup is unsettled. Refresh before recording containment.",
|
|
{ code: "native_controller_containment_stale" });
|
|
}
|
|
const [run] = await db
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.companyId, companyId),
|
|
eq(heartbeatRuns.id, decision.runId),
|
|
),
|
|
);
|
|
const [task] = await db
|
|
.select()
|
|
.from(issues)
|
|
.where(and(eq(issues.companyId, companyId), eq(issues.id, issueId)));
|
|
const review =
|
|
task?.status === "in_review"
|
|
? parseIssueExecutionState(task.executionState)
|
|
: null;
|
|
const isCurrentReviewer =
|
|
review?.status === "pending" &&
|
|
review.currentParticipant?.type === "agent" &&
|
|
review.currentParticipant.agentId === run?.agentId;
|
|
const retainedWorkspace = hasRequiredWorkspaceRecovery(run?.resultJson);
|
|
const repairOnly = input.workspaceRepairOnly === true && retainedWorkspace;
|
|
const terminalRepairSource = ["failed", "interrupted", "timed_out", "cancelled"].includes(run?.status ?? "") ||
|
|
(repairOnly && run?.runtimeMode === "legacy" && run.status === "succeeded");
|
|
if (
|
|
(input.workspaceRepairOnly && !retainedWorkspace) ||
|
|
!run ||
|
|
!task ||
|
|
(!repairOnly && (task.assigneeAgentId !== agentId || (run.agentId !== agentId && !isCurrentReviewer))) ||
|
|
(run.nativeIssueId ?? run.contextSnapshot?.issueId) !== issueId ||
|
|
!terminalRepairSource
|
|
) {
|
|
throw conflict(
|
|
"The recovery source or task owner changed. Inspect the current execution before continuing.",
|
|
);
|
|
}
|
|
if (hasWorkspaceRestoreFailure(run.resultJson) &&
|
|
(!decision.workspaceRepairEvidence || decision.workspaceRepairEvidence.trim().length < 20)) {
|
|
throw conflict("Verify safe workspace staging or repair and record workspaceRepairEvidence before continuing this run.");
|
|
}
|
|
let checkLocalProcesses = !operatorContained;
|
|
if (retainedWorkspace) {
|
|
const leases = await db.select().from(environmentLeases).where(and(
|
|
eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, run.id),
|
|
));
|
|
const recovery = run.resultJson!.workspaceRestoreRecovery as { leaseIds?: unknown };
|
|
const retainedIds = recovery.leaseIds;
|
|
// The terminal transaction records the exact source allocations. A release
|
|
// timestamp is written before stop dispatch, so only matching provider
|
|
// acknowledgements can establish that repair will not race active writes.
|
|
if (!Array.isArray(retainedIds) || !retainedIds.length ||
|
|
!retainedIds.every(id => typeof id === "string" && leases.some(lease =>
|
|
lease.id === id && hasRemoteTerminationReceipt(lease))) ||
|
|
leases.some(lease => lease.provider === "local"
|
|
? !lease.releasedAt || lease.status === "pending_cleanup" || lease.cleanupStatus === "failed"
|
|
: !hasRemoteTerminationReceipt(lease))) {
|
|
throw conflict("The retained workspace environment has not confirmed that it stopped. Wait for cleanup before recording workspace repair.");
|
|
}
|
|
checkLocalProcesses = !operatorContained && leases.some(lease => lease.provider === "local");
|
|
}
|
|
// A retained remote source's PIDs belong to its sandbox, not this server.
|
|
// Its exact lease receipts above replace host-local process probes.
|
|
for (const pid of checkLocalProcesses ? [
|
|
run.processPid,
|
|
run.processGroupId ? -run.processGroupId : null,
|
|
] : []) {
|
|
if (!pid) continue;
|
|
try {
|
|
process.kill(pid, 0);
|
|
} catch (error) {
|
|
if ((error as NodeJS.ErrnoException).code === "ESRCH") continue;
|
|
throw conflict(
|
|
"The previous provider's process ownership cannot be verified.",
|
|
);
|
|
}
|
|
throw conflict(
|
|
"The previous provider is still running. Stop it before continuing.",
|
|
);
|
|
}
|
|
const [coordinator] = await db
|
|
.select()
|
|
.from(nativeRunFinalizations)
|
|
.where(
|
|
and(
|
|
eq(nativeRunFinalizations.companyId, companyId),
|
|
eq(nativeRunFinalizations.runId, run.id),
|
|
),
|
|
);
|
|
if (coordinator?.leaseOwner || coordinator?.failureDetail?.successorRunId)
|
|
throw conflict(
|
|
"This execution still has a coordinator or a linked continuation. Inspect that run first.",
|
|
);
|
|
const leases = await db
|
|
.select({ id: environmentLeases.id })
|
|
.from(environmentLeases)
|
|
.where(
|
|
and(
|
|
eq(environmentLeases.companyId, companyId),
|
|
eq(environmentLeases.heartbeatRunId, run.id),
|
|
isNull(environmentLeases.releasedAt),
|
|
),
|
|
)
|
|
.limit(1);
|
|
if (leases.length)
|
|
throw conflict(
|
|
"The previous execution environment has not finished releasing its authority.",
|
|
);
|
|
if (!repairOnly) await buildExecutionContinuation({
|
|
db,
|
|
companyId,
|
|
issueId,
|
|
agentId: agentId!,
|
|
context: { previousRunId: run.id },
|
|
summary: null,
|
|
exposeLowTrustRaw: false,
|
|
});
|
|
return run;
|
|
}
|
|
|
|
/** Durable delivery marker lives on the existing source-scoped recovery action. */
|
|
export async function markExecutionReconciliation(
|
|
db: Db,
|
|
action: Pick<
|
|
typeof issueRecoveryActions.$inferSelect,
|
|
"companyId" | "id" | "evidence" | "sourceIssueId"
|
|
>,
|
|
decision: ExecutionReconciliation,
|
|
actorId: string,
|
|
deliveryOwner?: { kind: "chat_failed_run_retry"; actionId: string },
|
|
options?: { workspaceRepairOnly?: boolean; controllerContainmentOnly?: boolean },
|
|
) {
|
|
if (options?.controllerContainmentOnly && (!decision.controllerStopped || !decision.controllerAuthorityDigest || deliveryOwner))
|
|
throw conflict("Controller containment records no automatic continuation.");
|
|
if (options?.workspaceRepairOnly && (deliveryOwner || !hasRequiredWorkspaceRecovery(action.evidence))) {
|
|
throw conflict("Workspace repair must target the retained source without a continuation owner.");
|
|
}
|
|
if (deliveryOwner) {
|
|
const [retry] = await db
|
|
.select()
|
|
.from(chatActions)
|
|
.where(
|
|
and(
|
|
eq(chatActions.companyId, action.companyId),
|
|
eq(chatActions.id, deliveryOwner.actionId),
|
|
),
|
|
);
|
|
if (
|
|
deliveryOwner.kind !== "chat_failed_run_retry" ||
|
|
!retry ||
|
|
retry.kind !== "failed_run_retry" ||
|
|
!["issued", "processing", "processed"].includes(retry.status) ||
|
|
retry.payload.version !== 1 ||
|
|
retry.payload.failedRunId !== decision.runId ||
|
|
retry.payload.issueId !== action.sourceIssueId
|
|
) {
|
|
throw conflict("The authorized chat retry owner is no longer valid.");
|
|
}
|
|
}
|
|
await db.transaction(async tx => {
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionContinuation);
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionReplacement);
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionDisposition);
|
|
await tx
|
|
.update(nativeRunFinalizations)
|
|
.set({
|
|
failureDetail: sql`coalesce(${nativeRunFinalizations.failureDetail}, '{}'::jsonb) || ${JSON.stringify({ replacementDenied: "operator_reconciled" })}::jsonb`,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(nativeRunFinalizations.companyId, action.companyId),
|
|
eq(nativeRunFinalizations.runId, decision.runId),
|
|
),
|
|
);
|
|
const [source] = await tx
|
|
.update(issueRecoveryActions)
|
|
.set({
|
|
evidence: {
|
|
...action.evidence,
|
|
automaticRecovery: undefined,
|
|
executionReconciliation: {
|
|
...decision,
|
|
actorId,
|
|
recordedAt: new Date().toISOString(),
|
|
},
|
|
continuationDelivery: options?.workspaceRepairOnly || options?.controllerContainmentOnly ? "not_requested" : deliveryOwner ? "delegated" : "pending",
|
|
...(deliveryOwner ? { continuationDeliveryOwner: deliveryOwner } : {}),
|
|
},
|
|
})
|
|
.where(
|
|
and(
|
|
eq(issueRecoveryActions.companyId, action.companyId),
|
|
eq(issueRecoveryActions.id, action.id),
|
|
),
|
|
).returning({
|
|
cause: issueRecoveryActions.cause,
|
|
fingerprint: issueRecoveryActions.fingerprint,
|
|
kind: issueRecoveryActions.kind,
|
|
evidence: issueRecoveryActions.evidence,
|
|
});
|
|
if (options?.controllerContainmentOnly) {
|
|
// A Retry clicked before the operator reviewed the effects must not turn
|
|
// into an automatic replay after containment. Retain only that exact,
|
|
// unadmitted source receipt as skipped; newer/accepted intents are untouched.
|
|
await tx.update(agentWakeupRequests).set({ status: "skipped", finishedAt: new Date(), updatedAt: new Date(),
|
|
error: "controller_containment_requires_new_retry" }).where(and(
|
|
eq(agentWakeupRequests.companyId, action.companyId), isNull(agentWakeupRequests.runId),
|
|
inArray(agentWakeupRequests.status, ["queued", "deferred_issue_execution"]),
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${action.sourceIssueId}`,
|
|
sql`${agentWakeupRequests.payload}->'taskRetry'->>'schema' = 'paperclip.task-retry.v1'`,
|
|
sql`${agentWakeupRequests.payload}->'taskRetry'->>'sourceRunId' = ${decision.runId}`,
|
|
));
|
|
return;
|
|
}
|
|
if (
|
|
!source || options?.workspaceRepairOnly ||
|
|
source.cause !== "legacy_execution_requires_reconciliation" ||
|
|
source.fingerprint !== `legacy-execution:${decision.runId}` ||
|
|
["workspaceRestoreFailure", "workspaceRestoreRecovery", "systemRecovery"].some(key => key in source.evidence)
|
|
) return;
|
|
const now = new Date();
|
|
const siblings = await tx.update(issueRecoveryActions).set({
|
|
status: "resolved", outcome: "cancelled", resolvedAt: now, updatedAt: now,
|
|
wakePolicy: null, monitorPolicy: null,
|
|
resolutionNote: "The same source execution was reconciled through another recovery action.",
|
|
evidence: sql`(${issueRecoveryActions.evidence} - 'automaticRecovery') || ${JSON.stringify({
|
|
executionReconciliation: { ...decision, actorId, recordedAt: now.toISOString() },
|
|
continuationDelivery: "invalidated",
|
|
coalescedIntoRecoveryActionId: action.id,
|
|
})}::jsonb`,
|
|
}).where(and(
|
|
eq(issueRecoveryActions.companyId, action.companyId),
|
|
eq(issueRecoveryActions.sourceIssueId, action.sourceIssueId),
|
|
not(eq(issueRecoveryActions.id, action.id)),
|
|
eq(issueRecoveryActions.kind, source.kind),
|
|
eq(issueRecoveryActions.cause, source.cause),
|
|
eq(issueRecoveryActions.fingerprint, source.fingerprint),
|
|
sql`${issueRecoveryActions.evidence}->>'runId' = ${decision.runId}`,
|
|
sql`${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked'`,
|
|
sql`not (${issueRecoveryActions.evidence} ?| array['executionReconciliation', 'workspaceRestoreFailure', 'workspaceRestoreRecovery', 'systemRecovery'])`,
|
|
)).returning({ id: issueRecoveryActions.id });
|
|
if (siblings.length) await persistActivity(tx as unknown as Db, {
|
|
companyId: action.companyId, actorType: "user", actorId,
|
|
action: "issue.recovery_action_resolved", entityType: "issue", entityId: action.sourceIssueId,
|
|
details: { reason: "same_source_execution_reconciled", recoveryActionId: action.id,
|
|
coalescedRecoveryActionIds: siblings.map(sibling => sibling.id), runId: decision.runId },
|
|
});
|
|
});
|
|
}
|
|
|
|
export async function deliverReconciledExecutions(
|
|
db: Db,
|
|
wake: ReturnType<typeof import("./heartbeat.js").heartbeatService>["wakeup"],
|
|
) {
|
|
const recoveryReasons = [...RECOVERY_WAKE_REASONS];
|
|
const sourceRun = alias(heartbeatRuns, "continuation_source_run");
|
|
const seenActionIds = new Set<string>();
|
|
const seenAgentIds = new Set<string>();
|
|
const admittedByCompany = new Map<string, number>();
|
|
const fullCompanies = () => [...admittedByCompany].flatMap(([companyId, count]) =>
|
|
count >= RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS ? [companyId] : []);
|
|
const heldSince = sql`coalesce(${issueRecoveryActions.evidence}#>>'{freshContinuation,recordedAt}', ${issueRecoveryActions.resolvedAt}::text, ${issueRecoveryActions.updatedAt}::text)::timestamptz`;
|
|
const readPendingPage = async () => {
|
|
const companies = fullCompanies();
|
|
const heldBound = new Date(Date.now() - RECOVERY_CONTINUATION_MAX_HOLD_MS).toISOString();
|
|
const rows = await db
|
|
.select({ action: issueRecoveryActions })
|
|
.from(issueRecoveryActions)
|
|
.leftJoin(sourceRun, and(
|
|
eq(sourceRun.companyId, issueRecoveryActions.companyId),
|
|
sql`${sourceRun.id}::text = ${issueRecoveryActions.evidence}#>>'{executionReconciliation,runId}'`,
|
|
))
|
|
.where(
|
|
and(
|
|
eq(issueRecoveryActions.status, "resolved"),
|
|
sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`,
|
|
seenActionIds.size ? not(inArray(issueRecoveryActions.id, [...seenActionIds])) : undefined,
|
|
seenAgentIds.size ? sql`coalesce(${issueRecoveryActions.returnOwnerAgentId}::text, '') <> all(${sql.param([...seenAgentIds])}::text[])` : undefined,
|
|
companies.length ? not(inArray(issueRecoveryActions.companyId, companies)) : undefined,
|
|
sql`not exists (select 1 from ${heartbeatRuns} queued_recovery
|
|
where queued_recovery.company_id = ${issueRecoveryActions.companyId}
|
|
and queued_recovery.agent_id = ${issueRecoveryActions.returnOwnerAgentId}
|
|
and queued_recovery.status = 'queued'
|
|
and queued_recovery.context_snapshot->>'wakeReason' in (${sql.join(recoveryReasons.map(reason => sql`${reason}`), sql`, `)}))`,
|
|
),
|
|
)
|
|
.orderBy(
|
|
sql`${heldSince} <= ${heldBound}::timestamptz desc`,
|
|
sql`case when ${heldSince} <= ${heldBound}::timestamptz then ${heldSince} end asc nulls last`,
|
|
sql`${sourceRun.finishedAt} desc nulls last`,
|
|
asc(issueRecoveryActions.updatedAt),
|
|
)
|
|
.limit(25);
|
|
return rows.map(row => row.action);
|
|
};
|
|
const agentHasRoom = async (action: typeof issueRecoveryActions.$inferSelect, agentId: string) => {
|
|
if ((admittedByCompany.get(action.companyId) ?? 0) >= RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS) return false;
|
|
const [agent] = await db.select().from(agents).where(and(eq(agents.companyId, action.companyId), eq(agents.id, agentId)));
|
|
if (!agent) return true;
|
|
const [queued] = await db.select({ count: sql<number>`count(*)::int` }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, action.companyId), eq(heartbeatRuns.agentId, agentId), eq(heartbeatRuns.status, "queued")));
|
|
if (Number(queued?.count ?? 0) < parseHeartbeatPolicy(agent).maxConcurrentRuns) return true;
|
|
const recordedAt = Date.parse(String((action.evidence.freshContinuation as { recordedAt?: unknown } | undefined)?.recordedAt ?? ""));
|
|
const heldSince = Number.isFinite(recordedAt) ? recordedAt : (action.resolvedAt ?? action.updatedAt).getTime();
|
|
return Date.now() - heldSince >= RECOVERY_CONTINUATION_MAX_HOLD_MS;
|
|
};
|
|
for (let page = 0; page < RECOVERY_CONTINUATION_PAGES_PER_PASS; page += 1) {
|
|
const pending = await readPendingPage();
|
|
if (!pending.length) break;
|
|
for (const action of pending) {
|
|
seenActionIds.add(action.id);
|
|
if (action.returnOwnerAgentId) {
|
|
if (seenAgentIds.has(action.returnOwnerAgentId)) continue;
|
|
seenAgentIds.add(action.returnOwnerAgentId);
|
|
}
|
|
try {
|
|
const decision = action.evidence.executionReconciliation as
|
|
ExecutionReconciliation | undefined;
|
|
if (!decision || !action.returnOwnerAgentId) continue;
|
|
const pendingDecision = and(
|
|
eq(issueRecoveryActions.companyId, action.companyId),
|
|
eq(issueRecoveryActions.id, action.id),
|
|
eq(issueRecoveryActions.status, "resolved"),
|
|
sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`,
|
|
sql`${issueRecoveryActions.evidence}->'executionReconciliation' = ${JSON.stringify(decision)}::jsonb`,
|
|
);
|
|
const [task] = await db
|
|
.select()
|
|
.from(issues)
|
|
.where(
|
|
and(
|
|
eq(issues.companyId, action.companyId),
|
|
eq(issues.id, action.sourceIssueId),
|
|
),
|
|
);
|
|
if (
|
|
!task ||
|
|
task.assigneeAgentId !== action.returnOwnerAgentId ||
|
|
["done", "cancelled"].includes(task.status)
|
|
) {
|
|
await db
|
|
.update(issueRecoveryActions)
|
|
.set({
|
|
evidence: sql`${issueRecoveryActions.evidence} || '{"continuationDelivery":"invalidated"}'::jsonb`,
|
|
})
|
|
.where(pendingDecision);
|
|
await settleCoalescedFreshContinuationHolds(db, { companyId: action.companyId, primaryId: action.id, delivery: "invalidated" });
|
|
continue;
|
|
}
|
|
// A fresh continuation is the task's next run only while nothing else
|
|
// started one: a person's Retry or saved input that ran since it was
|
|
// recorded already continues the task, so it is never a second run.
|
|
const fresh = action.evidence.freshContinuation as { recordedAt?: unknown } | undefined;
|
|
if (fresh && typeof fresh.recordedAt === "string") {
|
|
const [newer] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, action.companyId), sql`${heartbeatRuns.id}::text <> ${decision.runId}`,
|
|
or(eq(heartbeatRuns.nativeIssueId, task.id), eq(heartbeatRuns.issueId, task.id),
|
|
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${task.id}::text`),
|
|
sql`${heartbeatRuns.createdAt} >= ${fresh.recordedAt}::timestamptz`)).limit(1);
|
|
if (newer) {
|
|
await db.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify(
|
|
{ continuationDelivery: "delivered", continuationRunId: newer.id, continuationSupersededByRun: true })}::jsonb` })
|
|
.where(pendingDecision);
|
|
await settleCoalescedFreshContinuationHolds(db, { companyId: action.companyId, primaryId: action.id,
|
|
delivery: "delivered", continuationRunId: newer.id });
|
|
continue;
|
|
}
|
|
}
|
|
if (!await agentHasRoom(action, action.returnOwnerAgentId)) continue;
|
|
admittedByCompany.set(action.companyId, (admittedByCompany.get(action.companyId) ?? 0) + 1);
|
|
const run = await wake(action.returnOwnerAgentId, {
|
|
source: "automation",
|
|
triggerDetail: "system",
|
|
reason: "issue_recovery_action_restored",
|
|
idempotencyKey: `execution-reconciliation:${action.id}`,
|
|
payload: { issueId: task.id, recoveryActionId: action.id },
|
|
requestedByActorType: "system",
|
|
requestedByActorId: "execution-recovery",
|
|
contextSnapshot: {
|
|
issueId: task.id,
|
|
taskId: task.id,
|
|
recoveryActionId: action.id,
|
|
previousRunId: decision.runId,
|
|
retryOfRunId: decision.runId,
|
|
forceFreshSession: true,
|
|
wakeReason: "issue_recovery_action_restored",
|
|
source: "execution.reconciled",
|
|
...((action.evidence.freshContinuation as { isolation?: unknown } | undefined)?.isolation === "fresh_isolated_workspace"
|
|
? { freshContinuationIsolation: action.id } : {}),
|
|
},
|
|
});
|
|
if (run)
|
|
await db.transaction(async (tx) => {
|
|
await tx
|
|
.update(heartbeatRuns)
|
|
.set({ retryOfRunId: decision.runId })
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.companyId, action.companyId),
|
|
eq(heartbeatRuns.id, run.id),
|
|
eq(heartbeatRuns.agentId, action.returnOwnerAgentId!),
|
|
sql`${heartbeatRuns.contextSnapshot}->>'recoveryActionId' = ${action.id}`,
|
|
sql`${heartbeatRuns.contextSnapshot}->>'previousRunId' = ${decision.runId}`,
|
|
),
|
|
);
|
|
await tx
|
|
.update(issueRecoveryActions)
|
|
.set({
|
|
evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify(
|
|
{
|
|
continuationDelivery: "delivered",
|
|
continuationRunId: run.id,
|
|
},
|
|
)}::jsonb`,
|
|
})
|
|
.where(pendingDecision);
|
|
await settleCoalescedFreshContinuationHolds(tx as unknown as Db, { companyId: action.companyId, primaryId: action.id,
|
|
delivery: "delivered", continuationRunId: run.id });
|
|
});
|
|
} catch {
|
|
logger.warn(
|
|
{ recoveryActionId: action.id },
|
|
"Reconciled execution continuation remains pending for retry",
|
|
);
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
const supersedingWorkspaceHold = alias(issueRecoveryActions, "superseding_workspace_hold");
|
|
|
|
function supersedingNativeWorkspaceHold(db: Db, input: {
|
|
companyId: string | typeof heartbeatRuns.companyId;
|
|
issueId: string | typeof issueRecoveryActions.sourceIssueId;
|
|
runId: string | typeof heartbeatRuns.id;
|
|
}) {
|
|
return db.select({ id: supersedingWorkspaceHold.id }).from(supersedingWorkspaceHold).where(and(
|
|
eq(supersedingWorkspaceHold.companyId, input.companyId),
|
|
eq(supersedingWorkspaceHold.sourceIssueId, input.issueId),
|
|
eq(supersedingWorkspaceHold.status, "resolved"),
|
|
eq(supersedingWorkspaceHold.cause, "native_continuation_requires_reconciliation"),
|
|
sql`${supersedingWorkspaceHold.evidence}->>'runId' = ${input.runId}::text`,
|
|
sql`${supersedingWorkspaceHold.evidence}->'automaticRecovery'->>'replay' = 'blocked'`,
|
|
sql`${supersedingWorkspaceHold.evidence}->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1'`,
|
|
));
|
|
}
|
|
|
|
export function unresolvedExecutionDispositions(db: Db, limit: number) {
|
|
return db
|
|
.select({ action: issueRecoveryActions })
|
|
.from(issueRecoveryActions)
|
|
.innerJoin(
|
|
heartbeatRuns,
|
|
and(
|
|
eq(heartbeatRuns.companyId, issueRecoveryActions.companyId),
|
|
eq(heartbeatRuns.id, canonicalExecutionUuid(sql`${issueRecoveryActions.evidence}->>'runId'`)),
|
|
or(
|
|
eq(heartbeatRuns.nativeIssueId, issueRecoveryActions.sourceIssueId),
|
|
and(isNull(heartbeatRuns.nativeIssueId), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueRecoveryActions.sourceIssueId}::text`),
|
|
),
|
|
),
|
|
)
|
|
.leftJoin(
|
|
nativeRunFinalizations,
|
|
and(
|
|
eq(nativeRunFinalizations.companyId, heartbeatRuns.companyId),
|
|
eq(nativeRunFinalizations.runId, heartbeatRuns.id),
|
|
),
|
|
)
|
|
.where(
|
|
and(
|
|
not(conversationRecoveryActionPredicate()!),
|
|
sql`not (${issueRecoveryActions.evidence} ? 'systemRecovery')`,
|
|
inArray(issueRecoveryActions.status, ["active", "escalated"]),
|
|
eq(issueRecoveryActions.kind, "active_run_watchdog"),
|
|
// A fresh continuation that backed off is checked again when due.
|
|
freshContinuationCheckDue(new Date()),
|
|
or(inArray(issueRecoveryActions.cause, [...EXECUTION_RECONCILIATION_CAUSES]), and(
|
|
eq(heartbeatRuns.runtimeMode, "native"), eq(heartbeatRuns.nativePhase, "terminal_failure"),
|
|
eq(nativeRunFinalizations.phase, "terminal_failure"), isNull(nativeRunFinalizations.nextAttemptAt),
|
|
sql`${issueRecoveryActions.wakePolicy}->>'kind' = 'resume_native_run'`,
|
|
sql`${issueRecoveryActions.wakePolicy}->>'runId' = ${heartbeatRuns.id}::text`,
|
|
exists(supersedingNativeWorkspaceHold(db, { companyId: heartbeatRuns.companyId,
|
|
issueId: issueRecoveryActions.sourceIssueId, runId: heartbeatRuns.id })),
|
|
)),
|
|
inArray(heartbeatRuns.status, [
|
|
"failed",
|
|
"timed_out",
|
|
"interrupted",
|
|
"cancelled",
|
|
]),
|
|
isNull(nativeRunFinalizations.leaseOwner),
|
|
isNull(nativeRunFinalizations.resultId),
|
|
or(
|
|
isNull(nativeRunFinalizations.runId),
|
|
eq(nativeRunFinalizations.phase, "terminal_failure"),
|
|
),
|
|
sql`coalesce(${nativeRunFinalizations.failureDetail}->>'successorRunId', '') = ''`,
|
|
sql`(${heartbeatRuns.runtimeMode} <> 'native' or coalesce(${nativeRunFinalizations.failureCode}, '') <> 'native_provider_terminal_failed'
|
|
or coalesce(${nativeRunFinalizations.failureDetail}->>'replacementDenied', '') <> '')`,
|
|
),
|
|
)
|
|
.limit(limit);
|
|
}
|
|
|
|
export function obsoleteExecutionConversationHold() {
|
|
return and(
|
|
conversationRecoveryActionPredicate(),
|
|
sql`not (${issueRecoveryActions.evidence} ? 'systemRecovery')`,
|
|
or(
|
|
inArray(issueRecoveryActions.status, ["active", "escalated"]),
|
|
sql`${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked'`,
|
|
),
|
|
);
|
|
}
|
|
|
|
/**
|
|
* Failed execution is a system responsibility, not a user questionnaire. After
|
|
* automatic recovery is ruled out, preserve evidence and stop without replay.
|
|
* This is NOT evidence that an external action succeeded or never happened.
|
|
* The resolved record retains a dispatch hold until actual evidence clears it.
|
|
*/
|
|
export async function settleUnrecoverableExecutions(
|
|
db: Db,
|
|
now = new Date(),
|
|
options: { failpoint?: (phase: "persisted") => void } = {},
|
|
) {
|
|
// Fold obsolete conversation holds without waking historical work on upgrade.
|
|
// Keep their evidence and record the policy change in the task's activity log.
|
|
const obsoleteConversationHold = obsoleteExecutionConversationHold();
|
|
await db.transaction(async tx => {
|
|
const foldable = await tx.select().from(issueRecoveryActions).where(obsoleteConversationHold)
|
|
.limit(25).for("update", { skipLocked: true });
|
|
for (const candidate of foldable) {
|
|
if (await getConversationOwnershipBlocker(tx as unknown as Db, candidate.companyId, candidate.sourceIssueId)) continue;
|
|
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: candidate.companyId, issueId: candidate.sourceIssueId });
|
|
const [action] = await tx.update(issueRecoveryActions).set({
|
|
status: "resolved",
|
|
outcome: "cancelled",
|
|
resolvedAt: now,
|
|
updatedAt: now,
|
|
nextAction: "Automatic attempts stopped. Send a new message to continue the conversation.",
|
|
resolutionNote: "Conversation continuation does not replay prior tool calls.",
|
|
wakePolicy: null,
|
|
monitorPolicy: null,
|
|
evidence: sql`case when ${issueRecoveryActions.evidence} ? 'automaticRecovery'
|
|
then jsonb_set(${issueRecoveryActions.evidence}, '{automaticRecovery,replay}', '"conversation_continuation"'::jsonb)
|
|
else ${issueRecoveryActions.evidence} end`,
|
|
}).where(and(obsoleteConversationHold, eq(issueRecoveryActions.id, candidate.id))).returning();
|
|
if (!action) continue;
|
|
await persistActivity(tx as unknown as Db, {
|
|
companyId: action.companyId,
|
|
actorType: "system",
|
|
actorId: "execution-recovery",
|
|
action: "issue.execution_recovery_settled",
|
|
entityType: "issue",
|
|
entityId: action.sourceIssueId,
|
|
details: { recoveryActionId: action.id, outcome: "cancelled", continuation: "conversation" },
|
|
});
|
|
}
|
|
});
|
|
// Filter eligibility before applying the batch limit. A queue of sessions
|
|
// awaiting replacement must not starve settled incidents behind it.
|
|
const candidates = await unresolvedExecutionDispositions(db, 25);
|
|
for (const { action: candidate } of candidates) {
|
|
const runId = candidate.evidence.runId;
|
|
if (typeof runId !== "string") continue;
|
|
const goneControllerBootId = await provenGoneLegacyControllerBootId(db, candidate.companyId, runId).catch(() => null);
|
|
try {
|
|
await db.transaction(async (tx) => {
|
|
await tx.execute(
|
|
sql`select set_config('statement_timeout', '15000', true), set_config('lock_timeout', '1000', true)`,
|
|
);
|
|
// Same issue -> coordinator -> run ordering as replacement/finalization.
|
|
const [task] = await tx
|
|
.select()
|
|
.from(issues)
|
|
.where(
|
|
and(
|
|
eq(issues.companyId, candidate.companyId),
|
|
eq(issues.id, candidate.sourceIssueId),
|
|
),
|
|
)
|
|
.for("update");
|
|
const [coordinator] = await tx
|
|
.select()
|
|
.from(nativeRunFinalizations)
|
|
.where(
|
|
and(
|
|
eq(nativeRunFinalizations.companyId, candidate.companyId),
|
|
eq(nativeRunFinalizations.runId, runId),
|
|
),
|
|
)
|
|
.for("update");
|
|
const [run] = await tx
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.companyId, candidate.companyId),
|
|
eq(heartbeatRuns.id, runId),
|
|
),
|
|
)
|
|
.for("update");
|
|
const [action] = await tx
|
|
.select()
|
|
.from(issueRecoveryActions)
|
|
.where(eq(issueRecoveryActions.id, candidate.id))
|
|
.for("update");
|
|
if (
|
|
!task ||
|
|
!run ||
|
|
!action ||
|
|
action.evidence.runId !== runId ||
|
|
!["active", "escalated"].includes(action.status) ||
|
|
(run.nativeIssueId ?? run.contextSnapshot?.issueId) !== task.id ||
|
|
!["failed", "timed_out", "interrupted", "cancelled"].includes(
|
|
run.status,
|
|
)
|
|
)
|
|
return;
|
|
if (!EXECUTION_RECONCILIATION_CAUSES.includes(action.cause as (typeof EXECUTION_RECONCILIATION_CAUSES)[number])) {
|
|
if (run.runtimeMode !== "native" || run.nativePhase !== "terminal_failure"
|
|
|| coordinator?.phase !== "terminal_failure" || coordinator.leaseOwner || coordinator.resultId
|
|
|| coordinator.nextAttemptAt || coordinator.failureDetail?.successorRunId
|
|
|| action.wakePolicy?.kind !== "resume_native_run" || action.wakePolicy.runId !== run.id) return;
|
|
const [hold] = await supersedingNativeWorkspaceHold(tx as unknown as Db, {
|
|
companyId: run.companyId, issueId: task.id, runId: run.id,
|
|
}).limit(1).for("update");
|
|
if (!hold) return;
|
|
await notifyExecutionRunWork(tx, { companyId: run.companyId });
|
|
const note = "Exact session recovery stopped. The recorded workspace recovery holds this execution without replaying its actions.";
|
|
await tx.update(issueRecoveryActions).set({ status: "cancelled", outcome: "cancelled",
|
|
resolvedAt: now, updatedAt: now, wakePolicy: null, monitorPolicy: null,
|
|
nextAction: note, resolutionNote: note,
|
|
evidence: { ...action.evidence, supersedingRecoveryActionId: hold.id },
|
|
}).where(eq(issueRecoveryActions.id, action.id));
|
|
await persistActivity(tx as unknown as Db, { companyId: run.companyId, actorType: "system",
|
|
actorId: "execution-recovery", action: "issue.execution_recovery_settled", entityType: "issue",
|
|
entityId: task.id, runId: run.id,
|
|
details: { recoveryActionId: action.id, outcome: "cancelled", supersedingRecoveryActionId: hold.id },
|
|
});
|
|
return;
|
|
}
|
|
// A legacy executor still holding its controller lease may be unwinding
|
|
// a Stop and can prove that no adapter started. Let it settle first,
|
|
// unless its recorded process is proven gone (a drained server): then
|
|
// expire that exact lease so every later preparation check sees it.
|
|
if (run.runtimeMode === "legacy" && run.controllerLeaseExpiresAt && run.controllerLeaseExpiresAt > now) {
|
|
if (goneControllerBootId === null || run.controllerBootId !== goneControllerBootId) return;
|
|
await tx.update(heartbeatRuns).set({ controllerLeaseExpiresAt: now }).where(and(
|
|
eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId),
|
|
eq(heartbeatRuns.controllerBootId, goneControllerBootId)));
|
|
run.controllerLeaseExpiresAt = now;
|
|
}
|
|
// Adoption transfers maintenance to the retained-workspace owner.
|
|
// This compatibility resolver must not close its hold or relabel it
|
|
// as a user repair merely because old return-owner metadata is absent.
|
|
if (readSystemRecovery(action.evidence)) return;
|
|
// Give durable native recovery its chance; never preempt a resume,
|
|
// replacement, result finalizer, or still-owned execution.
|
|
if (
|
|
coordinator?.leaseOwner ||
|
|
coordinator?.resultId ||
|
|
coordinator?.failureDetail?.successorRunId ||
|
|
(coordinator && coordinator.phase !== "terminal_failure") ||
|
|
(run.runtimeMode === "native" &&
|
|
coordinator?.failureCode === "native_provider_terminal_failed" &&
|
|
!coordinator.failureDetail?.replacementDenied)
|
|
)
|
|
return;
|
|
const current =
|
|
!isSupersededConversationRun(task, run) &&
|
|
action.returnOwnerAgentId !== null &&
|
|
task.assigneeAgentId === action.returnOwnerAgentId &&
|
|
!["done", "cancelled"].includes(task.status) &&
|
|
(!task.executionRunId || task.executionRunId === run.id) &&
|
|
(!task.checkoutRunId || task.checkoutRunId === run.id);
|
|
// Exact recovery is ruled out. Unless the cause is the person's, the
|
|
// task continues in a fresh run from its last saved work instead of
|
|
// waiting for a person.
|
|
// A preparation that never started provider work keeps its own replay
|
|
// lane, which takes over the settled hold; the held sweep follows it.
|
|
if (current && !isInterruptedNativePreparationCandidate(run)) {
|
|
const fresh = await continueFreshInTransaction(tx as unknown as Db, { companyId: task.companyId, runId: run.id,
|
|
trigger: "recovery_refused", now });
|
|
if (["continued", "already_continued", "successor_exists"].includes(fresh.kind)) return;
|
|
if (fresh.kind === "backoff" || fresh.kind === "exact_recovery_pending") {
|
|
await recordFreshContinuationCheck(tx as unknown as Db, { companyId: task.companyId, actionId: action.id }, fresh, now);
|
|
return;
|
|
}
|
|
}
|
|
const note = executionRecoverySettlementNote({ current, cause: action.cause,
|
|
workspaceRestoreFailed: hasWorkspaceRestoreFailure(run.resultJson), trigger: action.evidence.trigger });
|
|
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: task.companyId, issueId: task.id });
|
|
let nativeFailureBlock = action.evidence.nativeFailureBlock;
|
|
let nativeBootstrapFailureBlock = action.evidence.nativeBootstrapFailureBlock;
|
|
let legacyCancellationBlock = action.evidence.legacyCancellationBlock;
|
|
if (current) {
|
|
await notifyNativeStatusReassessment(tx, { companyId: task.companyId, issueId: task.id });
|
|
if (task.status !== "blocked") await notifyChatResponseAvailability(tx, { companyId: task.companyId, issueId: task.id });
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors);
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
|
|
await notifyRoutineExecutionStatus(tx, task.id, task.companyId);
|
|
const [projected] = await tx
|
|
.update(issues)
|
|
.set({
|
|
status: "blocked",
|
|
executionRunId: null,
|
|
checkoutRunId: null,
|
|
updatedAt: now,
|
|
})
|
|
.where(eq(issues.id, task.id)).returning();
|
|
// Only a transition owned by this failure grants a recovery receipt.
|
|
// An already-blocked task may have a separate human/dependency hold.
|
|
if (task.status !== "blocked" && run.runtimeMode === "native") {
|
|
nativeFailureBlock = { runId: run.id, statusVersion: projected!.statusVersion };
|
|
}
|
|
if (task.status !== "blocked" && run.runtimeMode === "legacy" && run.status === "cancelled") {
|
|
legacyCancellationBlock = { runId: run.id, statusVersion: projected!.statusVersion, previousStatus: task.status };
|
|
}
|
|
if (task.status !== "blocked" && isInterruptedNativePreparationCandidate(run)) {
|
|
nativeBootstrapFailureBlock = { runId: run.id, statusVersion: projected!.statusVersion, previousStatus: task.status };
|
|
}
|
|
}
|
|
await tx
|
|
.update(issueRecoveryActions)
|
|
.set({
|
|
status: "resolved",
|
|
outcome: current ? "blocked" : "cancelled",
|
|
resolvedAt: now,
|
|
updatedAt: now,
|
|
nextAction: note,
|
|
resolutionNote: note,
|
|
wakePolicy: null,
|
|
monitorPolicy: null,
|
|
evidence: {
|
|
...action.evidence,
|
|
...(nativeFailureBlock ? { nativeFailureBlock } : {}),
|
|
...(nativeBootstrapFailureBlock ? { nativeBootstrapFailureBlock } : {}),
|
|
...(legacyCancellationBlock ? { legacyCancellationBlock } : {}),
|
|
automaticRecovery: {
|
|
policy: "preserve_without_replay_v1",
|
|
runId: run.id,
|
|
replay: "blocked",
|
|
actionOutcome: "unknown",
|
|
recordedAt: now.toISOString(),
|
|
},
|
|
},
|
|
})
|
|
.where(eq(issueRecoveryActions.id, action.id));
|
|
await persistActivity(tx as unknown as Db, {
|
|
companyId: run.companyId,
|
|
actorType: "system",
|
|
actorId: "execution-recovery",
|
|
action: "issue.execution_recovery_settled",
|
|
entityType: "issue",
|
|
entityId: task.id,
|
|
runId: run.id,
|
|
details: {
|
|
recoveryActionId: action.id,
|
|
outcome: current ? "blocked" : "cancelled",
|
|
replay: "not_authorized",
|
|
},
|
|
});
|
|
await notifyExecutionRunWork(tx, { companyId: run.companyId });
|
|
await tx
|
|
.update(heartbeatRuns)
|
|
.set({ executionStatusDeliveryId: randomUUID() })
|
|
.where(eq(heartbeatRuns.id, run.id));
|
|
await appendHeartbeatRunEvent(tx as unknown as Db, {
|
|
companyId: run.companyId,
|
|
agentId: run.agentId,
|
|
runId: run.id,
|
|
eventType: "lifecycle",
|
|
stream: "system",
|
|
level: "warn",
|
|
message: note,
|
|
payload: {
|
|
recoveryActionId: action.id,
|
|
cause: action.cause,
|
|
automaticRecovery: "preserve_without_replay_v1",
|
|
replay: "blocked",
|
|
},
|
|
});
|
|
options.failpoint?.("persisted");
|
|
});
|
|
} catch (err) {
|
|
if (options.failpoint) throw err;
|
|
logger.warn(
|
|
{ err, recoveryActionId: candidate.id },
|
|
"Automatic recovery disposition remains pending",
|
|
);
|
|
}
|
|
}
|
|
// Tasks already held by settled records (including those older releases
|
|
// saved) continue from their saved work on this worker too.
|
|
await continueHeldExecutions(db, now);
|
|
}
|