mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 09:50:48 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run finishes but copying its files out of the sandbox fails, Paperclip holds the task and tries to recover the saved files. When recovery gives up, the task offers Retry. > - Some saved files can never be recovered: the sandbox that held them was destroyed, or Paperclip cannot prove which runs used a shared sandbox. Each Retry ran the same check again, failed again, and parked the Retry in a wait that never ends. > - The task told the person to press Retry again or to wait for a Paperclip update. That breaks the rule that recovery must work and that a Retry that is shown must work. > - This pull request makes Retry continue the task in a fresh session and a new workspace from the work that is recorded, when the saved files are provably unrecoverable or cannot be proven safe. The task says clearly that the unsaved files were not recovered. > - The benefit is that an offered Retry starts a run, and nobody waits for a fix that another click cannot bring. ## Linked Issues or Issue Description Refs #21315, #21295, #21267 **What happened?** On hosted staging, two blocked tasks offered Retry (`canRetry: true`). Each Retry was accepted, and within about 3 seconds it was parked in a wait. No run started. Every later Retry did the same. - Task A: the hold cause was `native_workspace_sync_out_retry_exhausted`. Recovery ended with `source_unavailable` (`use_alternative_strategy`). The sandbox that held the files had a signed absence proof and a `destroyed` termination receipt. Retry was parked with `saved_work_recovery_failed`: "Saving the agent's files failed. Retry to try again". - Task B: the same hold cause. Recovery ended with `allocation_ownership_ambiguous` (`service_required`), because the sandbox was a reusable sandbox that other runs also used. Each Retry opened a new recovery generation, ran the same check, and was parked with `saved_work_recovery_refused`: "Retry runs the same check again and succeeds only after Paperclip is updated." - In both tasks, the agent's turn had finished and its result was accepted. Its work was recorded outside the sandbox: a plan document and screenshot attachments for task A, and a pushed branch with an open pull request and an implementation document for task B. Root cause: admission had a branch for an accepted result whose files were not yet saved. When the recovery budget was spent, the branch always returned a `saved_work_recovery_*` wait. It did not check whether the files could ever be recovered, so the fresh-workspace path that admission already has was never reached. The automatic retained-source fallback did not help either, because it handles only the remote workspace v1 reference, and these runs use the native workspace sync v3 reference. Also, Retry opened a new recovery generation for an ownership refusal, which only repeated the same check. **Expected behavior** When the saved files are provably unrecoverable or cannot be proven safe, Retry continues the task in a fresh session and a new workspace from the recorded work. The old files are never applied and the source sandbox is not changed. No provider turn is replayed. The task says that the unsaved files could not be recovered. The Retry offer and admission agree. **Steps to reproduce** 1. Let a native run finish with an accepted result, and let its workspace copy-back fail (`native_workspace_sync_out_retry_exhausted`). 2. Let saved-work recovery end with `source_unavailable` (sandbox destroyed) or with `allocation_ownership_ambiguous` (shared reusable sandbox). 3. Press Retry. 4. Observe that the Retry receipt waits with `saved_work_recovery_failed` or `saved_work_recovery_refused`, and no run starts. Press Retry again and observe the same result. **Paperclip version or commit** Staging on `origin/master` after #21315 (`bbafa78fe2`). **Deployment mode** Hosted cloud (staging), remote Daytona sandbox runner. **Agent adapter(s) involved** `paperclip_runner` (native runtime runs). ## What Changed - `savedFilesUnrecoverable` and `unrecoverableSavedFilesFailure` (`system-recovery-state.ts`) are the one rule for saved files that no recovery can apply. The rule is true for a spent recovery whose failure is a permanent source refusal: `source_unavailable`, `allocation_ownership_ambiguous` or `historical_source_missing`. It is false for `source_binding_invalid`, because a new recovery generation can still verify that source. It is false for `archive_unsafe` and `snapshot_verification_failed`, because a fresh attempt still imports earlier saved snapshots and would fail before the agent starts. It is also false after a fresh attempt is recorded. - Admission (`explicit-native-continuation.ts`) no longer parks a Retry of an accepted result in `saved_work_recovery_*` when every live recovery of the run is unrecoverable. The Retry takes the existing fresh-workspace path (`fresh_isolated_workspace`). That path keeps all stop proofs, leaves the source sandbox and workspace untouched, and starts a fresh session. - When such a Retry starts, the task gets one system comment. The comment says that Paperclip could not recover the unsaved files, why, and that the task continues from its recorded work (comments, documents, attachments and pushed commits). The settled hold records the same text. - Retry (`task-retry.ts`) binds to an unrecoverable recovery generation. It does not open a new generation that repeats the same check. - The blocker text for unrecoverable files says that Retry continues the task in a new workspace from its recorded work. It no longer says that Retry succeeds only after an update. - `canAdoptNativeSourceRetention` and the permanent source-refusal lists moved into `system-recovery-state.ts`, so that admission, Retry and messages share them. The old modules re-export them. ## Verification - Red/green: `cd server && npx vitest run src/services/unrecoverable-saved-work-retry.test.ts`. The tests fail on `origin/master` sources (or on the earlier head for the snapshot case) and pass with this change. They are built from the two staging shapes (destroyed sandbox with `source_unavailable`, and a shared reusable sandbox with `allocation_ownership_ambiguous`): - offers a Retry that continues the task (blocker and task view agree, and the text names the fresh workspace) - starts a fresh run from the recorded work: the Retry receipt has no wait and a run, the strategy is `fresh_isolated_workspace`, the hold is settled with the fresh attempt, one system comment explains the lost files with no raw IDs, the source lease is unchanged, and no new recovery generation opens - keeps a Retry out of a fresh attempt that would import a succeeded snapshot whose archive is unsafe - `workspace-recovery-warm-handoff.test.ts` › "wakes physical preservation from HTTP Retry before passive history without replaying the provider" failed on an earlier head of this PR, which also treated `source_binding_invalid` as unrecoverable. It passes unchanged now. - Updated tests that pinned the old repeat-the-check behavior: `workspace-recovery-preparation-gates`, `task-retry`, `system-recovery-state`, `workspace-recovery-retry-copyback`. - Related suites pass: `system-recovery-state`, `workspace-recovery-preparation-gates`, `workspace-recovery-retry-copyback`, `workspace-recovery-warm-handoff`, `task-retry`, `saved-work-retry-eligibility`, `workspace-recovery-fallback`, `explicit-native-continuation`, `workspace-recovery-admission`, `task-interruption-query`, `native-workspace-sync-retention`. - `cd server && node scripts/typecheck-layers.mjs` passes (source and test layers). ## Risks - Medium-low. A Retry of an unrecoverable hold now starts a new agent turn in a new workspace instead of waiting. The turn starts only through the existing fresh-workspace admission, which keeps every stop and cleanup proof. - Unsaved files of the old run are not applied. That was already true; the change is that the task continues and says so. - `source_binding_invalid`, `snapshot_verification_failed` and `archive_unsafe` keep today's path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, browser for read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
984 lines
72 KiB
TypeScript
984 lines
72 KiB
TypeScript
import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js";
|
|
import { agentDirectoryPreparationWaitPredicate, isAgentDirectoryPreparationWait } from "./pre-dispatch-execution-evidence.js";
|
|
import { assertDurableChatWakeupRequest, assertDurableChatWakeupReceipt, type DurableChatWakeupRequest } from "./durable-chat-wakeup.js";
|
|
import { notifyExecutionRunWork } from "./execution-control-notifications.js";
|
|
import { canSelectFreshWorkspace, freshReadinessSchema, retainedIssueWorkspaceForRetry, type ContinuationReadiness } from "./fresh-workspace-continuation.js";
|
|
import { verifiedWorkspaceSnapshotReceipt } from "./workspace-recovery-snapshot.js";
|
|
import { readTaskRetryIntent, retireTaskRetriesContinuedBy } from "./task-retry.js";
|
|
import { isPreDispatchExecutionWaitVerified } from "./pre-dispatch-execution-wait.js";
|
|
import { readSystemRecovery, savedFilesUnrecoverable, unrecoverableSavedFilesFailure, unrecoverableSavedFilesNotice } from "./system-recovery-state.js";
|
|
import { hasRequiredWorkspaceRecovery } from "./workspace-restore-recovery-state.js";
|
|
import { hasVerifiedWorkspaceRepair } from "./workspace-recovery-receipt.js";
|
|
import { readMigratedLegacyWorkspaceSource } from "./legacy-workspace-recovery-migration.js";
|
|
import { readNativeWorkspaceSyncReference } from "./native-runtime/native-workspace-sync.js";
|
|
import { preserveNativeFailedWorkspace, preserveTaskRemoteWorkspaceWriter } from "./native-runtime/native-failed-workspace-recovery.js";
|
|
import { createHash } from "node:crypto";
|
|
import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
|
|
import { canContinueCancelledRun, userStopBoundaryAt } from "./run-cancellation.js";
|
|
import { readQueuedInteractionResponse } from "./queued-interaction-response.js";
|
|
import { isCancelledNativeStartup, isFailedNativePreparation, isFailedNativePreparationRetryAncestor } from "./cancelled-native-startup.js";
|
|
import { hasNativeLocalProcessStop, hasHistoricalSuspendedNativeSession, nativeGatedRunNeverLaunched, nativeRunNeverLaunched, recordedLocalExecutionStopped, recordUnidentifiedLocalRunnerStop } from "./native-local-process-stop.js";
|
|
import { completeTerminatedLocalNativeSessionCleanup, completeTerminatedRemoteNativeSessionCleanup, nativeSessionCleanupBlock } from "../vendor/paperclip-runner/index.js";
|
|
import { hasRecoveredCleanupStopReceipt, hasRemoteTerminationReceipt, remoteLeaseCleanupScope } from "./remote-execution-termination.js";
|
|
import { z } from "zod";
|
|
import { and, desc, eq, gt, inArray, isNull, ne, not, or, sql, type SQL } from "drizzle-orm";
|
|
import {
|
|
agents, agentWakeupRequests, approvals, issueApprovals, issueThreadInteractions,
|
|
environmentLeases, heartbeatRuns, issueComments, issueRecoveryActions,
|
|
issues, nativeRunFinalizations, nativeRunResults, type Db,
|
|
} from "@paperclipai/db";
|
|
import { executionBlockerPredicate, getExecutionBlocker, hasPendingRunCleanup, nativeExecutionRecoveryActionPredicate, restoreHoldReleasedForCurrentOwnerPredicate } from "./execution-blocker.js";
|
|
import { buildExecutionContinuation } from "./execution-continuation.js";
|
|
import { adapterExecutionControls } from "./adapter-execution-control.js";
|
|
import { persistActivity } from "./activity-log.js";
|
|
import { activeIssueInteractionCondition } from "./issue-question-context.js";
|
|
|
|
import { historicalAdapterType, isConversationAdapter } from "./conversation-continuation.js";
|
|
import { queuedCommentIdsFromWakePayload } from "./issue-queued-comment-queue.js";
|
|
import { firstTurnDeliveredCommentIds, historyReadDeliveredCommentIds } from "./queued-comment-delivery.js";
|
|
import { canResumePreservedResultFinalization } from "./preserved-result-finalization.js";
|
|
import { isServerReplacementInterruption } from "./server-replacement-interruption.js";
|
|
import { normalizeAgentNameKey } from "./agent-name-key.js";
|
|
import { liftInterruptionBlock } from "./interruption-block-release.js";
|
|
|
|
type Run = Pick<typeof heartbeatRuns.$inferSelect,
|
|
"id" | "companyId" | "agentId" | "status" | "startedAt" | "finishedAt" | "createdAt" |
|
|
"issueId" | "nativeIssueId" | "nativeSessionId" | "runnerInstanceId" | "runtimeMode" |
|
|
"runtimeModeResolvedAt" | "errorCode" | "error" | "resultJson" | "contextSnapshot" |
|
|
"runnerProfileJson" | "processPid" | "processGroupId" | "processStartedAt" |
|
|
"sessionIdAfter" | "controllerBootId" | "controllerLeaseExpiresAt" | "executionStage">;
|
|
const continuationContextKeys = ["issueId", "wakeCommentId", "wakeCommentIds", "source",
|
|
"connectionIntentResolution", "interactionKind", "interactionStatus", "mutation", "interactionId"];
|
|
const continuationContext = sql<Run["contextSnapshot"]>`case
|
|
when ${heartbeatRuns.contextSnapshot} is null then null else (
|
|
select coalesce(jsonb_object_agg(field.key, field.value), '{}'::jsonb)
|
|
from jsonb_each(case when jsonb_typeof(${heartbeatRuns.contextSnapshot}) = 'object'
|
|
then ${heartbeatRuns.contextSnapshot} else '{}'::jsonb end) as field(key, value)
|
|
where field.key in (${sql.join(continuationContextKeys.map(key => sql`${key}`), sql`, `)})
|
|
) end`;
|
|
const preflightProfile = sql<Run["runnerProfileJson"]>`case
|
|
when ${heartbeatRuns.runnerProfileJson} is null then null else (
|
|
select coalesce(jsonb_object_agg(field.key, case when field.key = 'adapterDispatch'
|
|
then jsonb_build_object('adapterType', field.value->'adapterType') else field.value end), '{}'::jsonb)
|
|
from jsonb_each(case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object'
|
|
then ${heartbeatRuns.runnerProfileJson} else '{}'::jsonb end) as field(key, value)
|
|
where field.key in ('adapterDispatch', 'nativeWorkspaceSync', 'remoteWorkspace')
|
|
) end`;
|
|
const continuationRunFields = {
|
|
id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId,
|
|
status: heartbeatRuns.status, startedAt: heartbeatRuns.startedAt, finishedAt: heartbeatRuns.finishedAt,
|
|
createdAt: heartbeatRuns.createdAt, issueId: heartbeatRuns.issueId, nativeIssueId: heartbeatRuns.nativeIssueId,
|
|
nativeSessionId: heartbeatRuns.nativeSessionId, runnerInstanceId: heartbeatRuns.runnerInstanceId,
|
|
runtimeMode: heartbeatRuns.runtimeMode, runtimeModeResolvedAt: heartbeatRuns.runtimeModeResolvedAt,
|
|
errorCode: heartbeatRuns.errorCode, error: heartbeatRuns.error, resultJson: heartbeatRuns.resultJson,
|
|
contextSnapshot: continuationContext, runnerProfileJson: heartbeatRuns.runnerProfileJson,
|
|
processPid: heartbeatRuns.processPid, processGroupId: heartbeatRuns.processGroupId,
|
|
processStartedAt: heartbeatRuns.processStartedAt, sessionIdAfter: heartbeatRuns.sessionIdAfter,
|
|
controllerBootId: heartbeatRuns.controllerBootId, controllerLeaseExpiresAt: heartbeatRuns.controllerLeaseExpiresAt,
|
|
executionStage: heartbeatRuns.executionStage,
|
|
};
|
|
const terminal = ["failed", "interrupted", "timed_out", "cancelled"];
|
|
const PREVIOUS_EXECUTION_UNVERIFIED = "Paperclip can't confirm that the previous attempt stopped before it started work, so Retry can't continue from it. Your message is saved.";
|
|
|
|
export function isUserCommentContinuationReason(reason: string | null | undefined): boolean {
|
|
return reason === "issue_commented" || reason === "issue_reopened_via_comment" || reason === "issue_assigned";
|
|
}
|
|
|
|
function processStopped(pid: number): boolean {
|
|
try { process.kill(pid, 0); return false; }
|
|
catch (error) { return (error as NodeJS.ErrnoException).code === "ESRCH"; }
|
|
}
|
|
|
|
/** Validate the whole saved queue, preserving order and original authors.
|
|
* Call again under the task lock before adopting IDs into a new run.
|
|
*/
|
|
export async function undeliveredLegacyUserCommentIds(
|
|
db: Db, companyId: string, issueId: string, agentId: string, commentIds: string[],
|
|
): Promise<string[]> {
|
|
if (!commentIds.length) return [];
|
|
const comments = await db.select({ id: issueComments.id }).from(issueComments).where(and(
|
|
eq(issueComments.companyId, companyId), eq(issueComments.issueId, issueId),
|
|
inArray(issueComments.id, commentIds), eq(issueComments.authorType, "user"),
|
|
isNull(issueComments.createdByRunId), isNull(issueComments.deletedAt),
|
|
sql`nullif(trim(${issueComments.body}), '') is not null`,
|
|
sql`nullif(trim(${issueComments.authorUserId}), '') is not null`,
|
|
));
|
|
const valid = new Set(comments.map(comment => comment.id));
|
|
const previous = await db.select({ context: heartbeatRuns.contextSnapshot }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.agentId, agentId),
|
|
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`,
|
|
// A queued turn cancelled before dispatch has not consumed input, whatever
|
|
// cancelled it (pause, stale assignment, or rejected admission). Reserved
|
|
// native identity/process metadata is not proof that its prompt was sent.
|
|
// Admission separately verifies process termination before a new turn.
|
|
sql`not (${heartbeatRuns.status} = 'cancelled' and ${heartbeatRuns.startedAt} is null)`,
|
|
// A claimed turn parked for a busy workspace stopped before provider
|
|
// dispatch; the host records that it sent no prompt.
|
|
sql`not coalesce(${heartbeatRuns.status} = 'cancelled' and ${heartbeatRuns.errorCode} = 'workspace_busy'
|
|
and ${heartbeatRuns.resultJson}#>>'{executionRecovery,kind}' = 'workspace_wait'
|
|
and ${heartbeatRuns.resultJson}#>'{executionRecovery,providerWorkStarted}' = 'false'::jsonb, false)`,
|
|
sql`not coalesce(${agentDirectoryPreparationWaitPredicate()}, false)`,
|
|
or(...commentIds.map(id => or(
|
|
sql`${heartbeatRuns.contextSnapshot}->>'wakeCommentId' = ${id}`,
|
|
sql`${heartbeatRuns.contextSnapshot}->'wakeCommentIds' @> ${JSON.stringify([id])}::jsonb`,
|
|
))),
|
|
));
|
|
for (const { context } of previous) {
|
|
if (typeof context?.wakeCommentId === "string") valid.delete(context.wakeCommentId);
|
|
if (Array.isArray(context?.wakeCommentIds)) {
|
|
for (const id of context.wakeCommentIds) if (typeof id === "string") valid.delete(id);
|
|
}
|
|
}
|
|
// An accepted first turn that took a queued message consumed it too.
|
|
const firstTurn = await firstTurnDeliveredCommentIds(db, { companyId, issueId, agentId, commentIds: [...valid] });
|
|
const historyRead = await historyReadDeliveredCommentIds(db, { companyId, issueId, agentId,
|
|
commentIds: [...valid].filter(id => !firstTurn.has(id)) });
|
|
return commentIds.filter(id => valid.has(id) && !firstTurn.has(id) && !historyRead.has(id));
|
|
}
|
|
|
|
function continuationDecisionHolds(companyId: string, wait: { issueId: SQL; actorId: SQL; taskRetrySourceRunId: SQL }) {
|
|
return sql<boolean>`(exists (select 1 from ${issueThreadInteractions}
|
|
where ${issueThreadInteractions.companyId} = ${companyId} and ${issueThreadInteractions.issueId} = ${wait.issueId}
|
|
and ${issueThreadInteractions.status} = 'pending'
|
|
and ${activeIssueInteractionCondition({ responsibleUser: wait.actorId })}
|
|
and (${wait.taskRetrySourceRunId} is null or (not (${issueThreadInteractions.kind} in ('ask_user_questions', 'connection_intent')
|
|
and not (${issueThreadInteractions.payload} ?| array['toolAction', 'secretProposal', 'connectionAuthorization'])
|
|
and coalesce(${issueThreadInteractions.payload} #>> '{target,type}', '') <> 'issue_document')
|
|
and not exists (select 1 from ${heartbeatRuns} admitted
|
|
where admitted.company_id = ${companyId} and admitted.id::text = ${wait.taskRetrySourceRunId}
|
|
and admitted.created_at > ${issueThreadInteractions.createdAt}))))
|
|
or exists (select 1 from ${issueApprovals} inner join ${approvals}
|
|
on ${approvals.id} = ${issueApprovals.approvalId} and ${approvals.companyId} = ${companyId}
|
|
where ${issueApprovals.companyId} = ${companyId} and ${issueApprovals.issueId} = ${wait.issueId}
|
|
and ${approvals.status} in ('pending', 'revision_requested')))`;
|
|
}
|
|
|
|
export async function hasPendingContinuationDecision(db: Db, input: {
|
|
companyId: string; issueId: string; actorId: string; taskRetrySourceRunId?: string | null;
|
|
}): Promise<boolean> {
|
|
const [row] = await db.execute<{ holds: boolean }>(sql`select ${continuationDecisionHolds(input.companyId, {
|
|
issueId: sql`${input.issueId}::uuid`, actorId: sql`${input.actorId}::text`,
|
|
taskRetrySourceRunId: sql`${input.taskRetrySourceRunId ?? null}::text`,
|
|
})} as holds`);
|
|
return row?.holds === true;
|
|
}
|
|
|
|
export async function pendingContinuationDecisionWaits(db: Db, companyId: string,
|
|
waits: Array<{ id: string; issueId: string; actorId: string; taskRetrySourceRunId: string | null }>): Promise<Set<string>> {
|
|
if (!waits.length) return new Set();
|
|
const rows = await db.execute<{ id: string }>(sql`select held.id from unnest(
|
|
${sql.param(waits.map(wait => wait.id))}::text[], ${sql.param(waits.map(wait => wait.issueId))}::uuid[],
|
|
${sql.param(waits.map(wait => wait.actorId))}::text[], ${sql.param(waits.map(wait => wait.taskRetrySourceRunId))}::text[])
|
|
as held(id, issue_id, actor_id, source_run_id)
|
|
where ${continuationDecisionHolds(companyId, { issueId: sql`held.issue_id`, actorId: sql`held.actor_id`,
|
|
taskRetrySourceRunId: sql`held.source_run_id` })}`);
|
|
return new Set(rows.map(row => row.id));
|
|
}
|
|
|
|
/** Called under the issue lock, in the transaction that creates the new turn.
|
|
* A user request authorizes a new conversation, not replay of the failed run.
|
|
* Unknown action outcomes and all prior records stay intact.
|
|
*/
|
|
export async function admitExplicitNativeContinuation(input: {
|
|
db: Db; companyId: string; issueId: string; agentId: string;
|
|
actorType: string | null | undefined; actorId: string | null | undefined;
|
|
reason: string | null; commentId: string | null; successorRunId: string;
|
|
failedRunId?: string | null;
|
|
taskRetryRequestId?: string;
|
|
queuedCommentInterruptId?: string;
|
|
/** Internal delivery of an unconsumed, user-authored saved queue entry. */
|
|
queuedCommentRequestId?: string;
|
|
durableChatRequest?: DurableChatWakeupRequest;
|
|
interactionPayload?: unknown;
|
|
dryRun?: boolean;
|
|
onBlocked?: (reason: string, message: string) => void;
|
|
}): Promise<{ previousRunId: string; commentId: string | null; failedRunId?: string; taskRetryRequestId?: string; queuedCommentInterruptId?: string; readiness?: ContinuationReadiness } | null> {
|
|
const { db, companyId, issueId, agentId, actorId, commentId } = input;
|
|
const blocked = (reason: string, message: string) => { input.onBlocked?.(reason, message); return null; };
|
|
let blocker: Awaited<ReturnType<typeof getExecutionBlocker>> = null;
|
|
const refused = (reason: string, message: string) => blocker ? blocked(reason, message) : null;
|
|
if (input.actorType !== "user" || !actorId) return null;
|
|
const retry = input.reason === "retry_failed_run" &&
|
|
z.string().guid().safeParse(input.failedRunId).success;
|
|
const decisionResponse = !retry && !input.taskRetryRequestId && !input.queuedCommentInterruptId &&
|
|
!input.queuedCommentRequestId && !commentId && input.reason === "issue_commented" && input.interactionPayload
|
|
? await readQueuedInteractionResponse(db, companyId, issueId, input.interactionPayload) : null;
|
|
const decided = decisionResponse && decisionResponse.comment.authorType === "user" &&
|
|
decisionResponse.comment.authorUserId === actorId && !decisionResponse.toolActionOutcome ? decisionResponse : null;
|
|
if (!retry && !input.taskRetryRequestId && !input.queuedCommentInterruptId && !decided && (!commentId || !z.string().guid().safeParse(commentId).success ||
|
|
!isUserCommentContinuationReason(input.reason))) return null;
|
|
const [task] = await db.select().from(issues).where(and(
|
|
eq(issues.companyId, companyId), eq(issues.id, issueId),
|
|
));
|
|
if (!task || task.assigneeAgentId !== agentId || task.status === "cancelled"
|
|
|| task.status === "done" && !input.queuedCommentInterruptId) return null;
|
|
const [retryWake] = input.taskRetryRequestId ? await db.select().from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.id, input.taskRetryRequestId),
|
|
eq(agentWakeupRequests.agentId, agentId), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
eq(agentWakeupRequests.requestedByActorType, "user"), eq(agentWakeupRequests.requestedByActorId, actorId),
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId}`,
|
|
)) : [];
|
|
const taskRetry = readTaskRetryIntent(retryWake?.payload);
|
|
if (input.taskRetryRequestId && (!taskRetry || taskRetry.generationId !== retryWake?.id || taskRetry.actorId !== actorId)) return null;
|
|
const [interruptQueue] = input.queuedCommentInterruptId ? await db.select().from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.id, input.queuedCommentInterruptId),
|
|
eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.agentId, agentId),
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId}`,
|
|
sql`${agentWakeupRequests.payload}->'queuedCommentInterrupt'->>'actorId' = ${actorId}`,
|
|
)) : [];
|
|
const response = interruptQueue
|
|
? await readQueuedInteractionResponse(db, companyId, issueId, interruptQueue.payload) : decided;
|
|
const queuedInterrupt = Boolean(interruptQueue && (response || (commentId &&
|
|
queuedCommentIdsFromWakePayload(interruptQueue.payload).includes(commentId))));
|
|
if (input.queuedCommentInterruptId && !queuedInterrupt) return null;
|
|
const [savedQueue] = input.queuedCommentRequestId ? await db.select().from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.id, input.queuedCommentRequestId),
|
|
eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.agentId, agentId),
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId}`,
|
|
)) : [];
|
|
const queuedRequest = Boolean(savedQueue && commentId &&
|
|
!savedQueue.idempotencyKey?.startsWith("chat-inbound:") &&
|
|
queuedCommentIdsFromWakePayload(savedQueue.payload).includes(commentId));
|
|
if (input.queuedCommentRequestId && !queuedRequest) return null;
|
|
const chatRequest = input.durableChatRequest;
|
|
let savedChatRequestId: string | undefined;
|
|
if (chatRequest) {
|
|
assertDurableChatWakeupRequest(chatRequest, { companyId, issueId, agentId, commentId,
|
|
requestedByActorType: input.actorType ?? undefined, requestedByActorId: actorId });
|
|
if (chatRequest.failedRunRetry || !chatRequest.resumeDeferred) return null;
|
|
const [receipt] = await db.select().from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.id, chatRequest.id), eq(agentWakeupRequests.companyId, companyId),
|
|
eq(agentWakeupRequests.agentId, agentId), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
isNull(agentWakeupRequests.runId), sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId}`,
|
|
));
|
|
if (!receipt || !commentId || !queuedCommentIdsFromWakePayload(receipt.payload).includes(commentId)) return null;
|
|
assertDurableChatWakeupReceipt(chatRequest, receipt);
|
|
savedChatRequestId = receipt.id;
|
|
}
|
|
let partiallyDeliveredQueue = false;
|
|
if (queuedRequest) {
|
|
const ids = queuedCommentIdsFromWakePayload(savedQueue!.payload);
|
|
const undelivered = await undeliveredLegacyUserCommentIds(db, companyId, issueId, agentId, ids);
|
|
// A partly consumed queue still owns its remaining input. Admission and
|
|
// adoption both recheck the durable queue and discard already delivered IDs.
|
|
if (undelivered.at(-1) !== commentId) return null;
|
|
partiallyDeliveredQueue = undelivered.length !== ids.length;
|
|
}
|
|
const [comment] = retry || taskRetry || response ? [] : await db.select().from(issueComments).where(and(
|
|
eq(issueComments.companyId, companyId), eq(issueComments.issueId, issueId),
|
|
eq(issueComments.id, commentId!), eq(issueComments.authorType, "user"),
|
|
queuedInterrupt ? undefined : eq(issueComments.authorUserId, actorId), isNull(issueComments.createdByRunId),
|
|
isNull(issueComments.deletedAt),
|
|
));
|
|
if (!retry && !taskRetry && !response && !comment?.body.trim()) return null;
|
|
const [agent] = await db.select().from(agents).where(and(eq(agents.companyId, companyId), eq(agents.id, agentId)));
|
|
if (!agent || (!isConversationAdapter(agent.adapterType) && agent.adapterType !== "paperclip_runner")) return null;
|
|
const agentIsNative = agent.adapterType === "paperclip_runner";
|
|
const interruptIntent = interruptQueue?.payload?.queuedCommentInterrupt as Record<string, unknown> | undefined;
|
|
const interruptRequestedAt = typeof interruptIntent?.requestedAt === "string" ? new Date(interruptIntent.requestedAt) : null;
|
|
const nativeQueuedSend = queuedInterrupt && agentIsNative;
|
|
if (nativeQueuedSend && (!interruptRequestedAt || !Number.isFinite(interruptRequestedAt.getTime())))
|
|
return blocked("send_intent_missing", "Send the saved message again to continue.");
|
|
const authorizedAt = nativeQueuedSend ? interruptRequestedAt!
|
|
: response?.comment.createdAt ?? comment?.createdAt ?? retryWake?.requestedAt ?? new Date();
|
|
if (!input.dryRun) await preserveTaskRemoteWorkspaceWriter(db, companyId, issueId);
|
|
blocker = await getExecutionBlocker(db, companyId, issueId);
|
|
const [boundary] = await db.select({ createdAt: issueComments.createdAt }).from(issues).innerJoin(issueComments, and(
|
|
eq(issueComments.companyId, issues.companyId), eq(issueComments.issueId, issues.id),
|
|
eq(issueComments.id, issues.conversationBoundaryCommentId),
|
|
)).where(and(eq(issues.companyId, companyId), eq(issues.id, issueId), sql`${issues.conversationAgentId} is not null`)).limit(1);
|
|
const actions = await db.select().from(issueRecoveryActions).where(and(
|
|
eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, issueId),
|
|
boundary ? or(gt(issueRecoveryActions.createdAt, boundary.createdAt), nativeExecutionRecoveryActionPredicate(),
|
|
blocker?.recoveryActionId ? eq(issueRecoveryActions.id, blocker.recoveryActionId) : undefined) : undefined,
|
|
not(restoreHoldReleasedForCurrentOwnerPredicate()),
|
|
or(executionBlockerPredicate(), and(
|
|
inArray(issueRecoveryActions.cause, ["native_workspace_sync_out_unrecoverable", "native_workspace_sync_out_retry_exhausted"]),
|
|
sql`${issueRecoveryActions.evidence}#>>'{workspaceRecovery,status}' = 'recovered'`,
|
|
sql`${issueRecoveryActions.evidence}->'executionReconciliation' is null`,
|
|
sql`${issueRecoveryActions.evidence}->'explicitUserContinuation' is null`,
|
|
sql`not exists (select 1 from ${heartbeatRuns} completed
|
|
join ${nativeRunFinalizations} finalized on finalized.run_id = completed.id and finalized.company_id = completed.company_id
|
|
join ${nativeRunResults} result on result.id = finalized.result_id and result.run_id = completed.id and result.company_id = completed.company_id
|
|
where completed.company_id = ${companyId} and result.issue_id = ${issueId}
|
|
and completed.id::text = coalesce(${issueRecoveryActions.evidence}->>'runId', ${issueRecoveryActions.evidence}->>'sourceRunId')
|
|
and completed.runtime_mode = 'native' and completed.status = 'succeeded' and finalized.phase = 'committed'
|
|
and result.schema_status = 'accepted' and result.result_json#>>'{terminal,runTerminalState}' = 'succeeded')`,
|
|
)),
|
|
)).for("update");
|
|
const sourceChecks: Array<{ id?: string; cause: string; evidence: Record<string, unknown> }> =
|
|
actions.map(action => ({ id: action.id, cause: action.cause, evidence: action.evidence }));
|
|
if (nativeQueuedSend) {
|
|
const [latest] = await db.select({ id: heartbeatRuns.id, status: heartbeatRuns.status, runtimeMode: heartbeatRuns.runtimeMode })
|
|
.from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.agentId, agentId),
|
|
eq(heartbeatRuns.runtimeMode, "native"),
|
|
or(eq(heartbeatRuns.nativeIssueId, issueId), eq(heartbeatRuns.issueId, issueId),
|
|
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`)))
|
|
.orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
|
|
if (latest?.status === "succeeded") {
|
|
const [coordinator] = await db.select().from(nativeRunFinalizations).where(and(
|
|
eq(nativeRunFinalizations.companyId, companyId), eq(nativeRunFinalizations.runId, latest.id),
|
|
)).for("update");
|
|
const [completed] = await db.select({ agentId: heartbeatRuns.agentId, status: heartbeatRuns.status,
|
|
runtimeMode: heartbeatRuns.runtimeMode, finishedAt: heartbeatRuns.finishedAt,
|
|
nativeIssueId: heartbeatRuns.nativeIssueId, issueId: heartbeatRuns.issueId, contextSnapshot: continuationContext,
|
|
}).from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, latest.id))).for("update");
|
|
if (!completed || completed.agentId !== agentId || completed.runtimeMode !== "native"
|
|
|| completed.status !== "succeeded" || !completed.finishedAt
|
|
|| (completed.nativeIssueId ?? completed.issueId ?? completed.contextSnapshot?.issueId) !== issueId
|
|
|| coordinator?.phase !== "committed" || coordinator.leaseOwner || adapterExecutionControls.has(latest.id))
|
|
return blocked("controller_settling", "Waiting for the previous run to finish recovery. Your message will start automatically.");
|
|
const [result] = coordinator.resultId ? await db.select({ id: nativeRunResults.id }).from(nativeRunResults).where(and(
|
|
eq(nativeRunResults.companyId, companyId), eq(nativeRunResults.issueId, issueId),
|
|
eq(nativeRunResults.runId, latest.id), eq(nativeRunResults.id, coordinator.resultId),
|
|
eq(nativeRunResults.schemaStatus, "accepted"),
|
|
sql`${nativeRunResults.resultJson}#>>'{terminal,runTerminalState}' = 'succeeded'`,
|
|
)).limit(1) : [];
|
|
if (!result || await hasPendingRunCleanup(db, companyId, latest.id))
|
|
return blocked("cleanup_pending", "Waiting for the previous run to finish cleanup. Your message is saved.");
|
|
}
|
|
if (latest?.runtimeMode === "native" && ["cancelled", "failed"].includes(latest.status) &&
|
|
!sourceChecks.some(source => (source.evidence.runId ?? source.evidence.sourceRunId) === latest.id)) {
|
|
sourceChecks.push({ cause: "native_continuation_requires_reconciliation", evidence: { runId: latest.id } });
|
|
}
|
|
}
|
|
if (taskRetry && !sourceChecks.some(source => (source.evidence.runId ?? source.evidence.sourceRunId) === taskRetry.sourceRunId)) {
|
|
const [coordinator] = await db.select().from(nativeRunFinalizations).where(and(
|
|
eq(nativeRunFinalizations.companyId, companyId), eq(nativeRunFinalizations.runId, taskRetry.sourceRunId),
|
|
)).for("update");
|
|
const [source] = await db.select(continuationRunFields).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, taskRetry.sourceRunId), eq(heartbeatRuns.agentId, agentId),
|
|
)).for("update");
|
|
if (!source || (source.nativeIssueId ?? source.issueId ?? source.contextSnapshot?.issueId) !== issueId
|
|
|| adapterExecutionControls.has(source.id) ||
|
|
!(await isFailedNativePreparation(db, source, coordinator) || await isPreDispatchExecutionWaitVerified(db, source))) return refused("previous_execution_unverified", PREVIOUS_EXECUTION_UNVERIFIED);
|
|
sourceChecks.push({ cause: "legacy_execution_requires_reconciliation", evidence: { runId: source.id } });
|
|
}
|
|
if (!sourceChecks.length && !nativeQueuedSend) return null;
|
|
if (blocker && blocker.recoveryActionId === null) return null;
|
|
if (await hasPendingContinuationDecision(db, { companyId, issueId, actorId,
|
|
taskRetrySourceRunId: taskRetry?.sourceRunId ?? null }))
|
|
return blocked("decision_pending", "A pending approval or question must be resolved before this message can start.");
|
|
if (!sourceChecks.length) return null;
|
|
|
|
const sources: Run[] = [];
|
|
let readiness: ContinuationReadiness | undefined;
|
|
const freshSources = new Map<string, string>();
|
|
const freshCandidates: Array<{ run: Run; intent: string; cause: string; evidence: Record<string, unknown>;
|
|
remote: boolean; leaseIds: string[] }> = [];
|
|
const stoppedSessions: Array<{ evidence: Record<string, unknown>; retire: () => boolean }> = [];
|
|
const cancelledStartupIds = new Set<string>();
|
|
for (const action of sourceChecks) {
|
|
const runId = action.evidence.runId ?? action.evidence.sourceRunId;
|
|
if (typeof runId !== "string") return blocked("source_missing", "The stopped run could not be identified. Your message is saved.");
|
|
// Text comparison keeps malformed historical evidence a hold, not a UUID cast error.
|
|
let [run] = await db.select({ ...continuationRunFields, runnerProfileJson: preflightProfile }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, companyId), sql`${heartbeatRuns.id}::text = ${runId}`,
|
|
));
|
|
if (!run || run.agentId !== agentId || !terminal.includes(run.status) ||
|
|
(run.nativeIssueId ?? run.issueId ?? run.contextSnapshot?.issueId) !== issueId ||
|
|
!run.finishedAt) return blocked("source_unavailable", "The previous execution has not finished or its owner changed. Your message is saved.");
|
|
if (!queuedInterrupt && !queuedRequest && authorizedAt <= userStopBoundaryAt(run)!) return blocked("message_predates_stop", "This message arrived before the previous run stopped. Send a new message to continue.");
|
|
if (adapterExecutionControls.has(run.id)) return blocked("execution_settling", "Waiting for the previous run to stop. Your message will start automatically.");
|
|
const [coordinator] = await db.select().from(nativeRunFinalizations).where(and(
|
|
eq(nativeRunFinalizations.companyId, companyId), eq(nativeRunFinalizations.runId, run.id),
|
|
)).for("update");
|
|
const [lockedRun] = await db.select(continuationRunFields).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, run.id),
|
|
)).for("update");
|
|
if (!lockedRun || lockedRun.status !== run.status || lockedRun.agentId !== run.agentId ||
|
|
lockedRun.finishedAt?.getTime() !== run.finishedAt.getTime()) return refused("execution_settling", "Waiting for the previous run to stop. Your message will start automatically.");
|
|
run = lockedRun;
|
|
// A user's message written after their Stop request is new direction even
|
|
// when the stopped native run settled later.
|
|
const stoppedAt = userStopBoundaryAt(run)!;
|
|
const failedPreparation = await isFailedNativePreparation(db, run, coordinator);
|
|
const unusedAdmission = run.status === "cancelled" && !run.startedAt &&
|
|
run.errorCode === "execution_reconciliation_required" &&
|
|
!run.processPid && !run.processGroupId && !run.nativeSessionId &&
|
|
(!taskRetry || await isPreDispatchExecutionWaitVerified(db, run));
|
|
const sourceRunId = run.id;
|
|
const strategyRequestId = input.taskRetryRequestId ?? input.queuedCommentRequestId ?? savedChatRequestId;
|
|
const strategyEvidence = strategyRequestId && !Object.hasOwn(action.evidence, "systemRecovery")
|
|
? actions.find(candidate => {
|
|
const recovery = readSystemRecovery(candidate.evidence);
|
|
return candidate.evidence.runId === sourceRunId && recovery?.sourceRunId === sourceRunId &&
|
|
(input.taskRetryRequestId ? recovery.intentId === input.taskRetryRequestId : true);
|
|
})?.evidence ?? action.evidence
|
|
: action.evidence;
|
|
const freshIntent = await canSelectFreshWorkspace({ db, companyId, issueId, agentId, actorId,
|
|
sourceRunId: run.id, evidence: strategyEvidence, taskRetryRequestId: input.taskRetryRequestId,
|
|
queuedRequestId: input.queuedCommentRequestId ?? savedChatRequestId });
|
|
const legacyUserTurn = failedPreparation || run.runtimeMode === "legacy" &&
|
|
action.cause === "legacy_execution_requires_reconciliation" &&
|
|
(isConversationAdapter(agent.adapterType) ||
|
|
(agent.adapterType === "paperclip_runner" && (freshIntent || await hasVerifiedWorkspaceRepair(db, run))));
|
|
// Saved input is a request for a new turn, never permission to undo an
|
|
// operator Stop or redeliver a message already consumed by this run.
|
|
if (queuedRequest && !queuedInterrupt && (run.contextSnapshot?.wakeCommentId === commentId ||
|
|
(Array.isArray(run.contextSnapshot?.wakeCommentIds) && run.contextSnapshot.wakeCommentIds.includes(commentId)))) return null;
|
|
if (legacyUserTurn && !failedPreparation) {
|
|
const historicalAdapter = await historicalAdapterType(db, run);
|
|
// A settings change never converts a known process/webhook execution into
|
|
// a conversation. Those adapters retain their reconciliation contract.
|
|
if (historicalAdapter && !isConversationAdapter(historicalAdapter)) {
|
|
if (!(taskRetry && run.id !== taskRetry.sourceRunId && freshIntent && agentIsNative &&
|
|
historicalAdapter === "paperclip_runner" && run.status === "failed"))
|
|
return refused("previous_execution_unverified", PREVIOUS_EXECUTION_UNVERIFIED);
|
|
if (run.controllerLeaseExpiresAt && run.controllerLeaseExpiresAt > new Date())
|
|
return refused("execution_settling", "Waiting for the previous run to stop. Your message will start automatically.");
|
|
if (await hasPendingRunCleanup(db, companyId, run.id))
|
|
return refused("cleanup_pending", "Waiting for the previous run to finish cleanup. Your message is saved.");
|
|
if (!await isFailedNativePreparationRetryAncestor(db, run, coordinator))
|
|
return refused("previous_execution_unverified", PREVIOUS_EXECUTION_UNVERIFIED);
|
|
}
|
|
if (!historicalAdapter && agent.adapterType === "paperclip_runner") return refused("previous_execution_unverified", PREVIOUS_EXECUTION_UNVERIFIED);
|
|
}
|
|
const interruptedByThisClick = queuedInterrupt &&
|
|
interruptIntent?.targetRunId === run.id && run.resultJson?.operatorInterrupted === true &&
|
|
run.resultJson?.interruptionSource === "issue_comment_interrupt" && run.resultJson?.interruptedIssueId === issueId &&
|
|
run.resultJson?.interruptedByActorType === "user" && run.resultJson?.interruptedByActorId === actorId &&
|
|
run.resultJson?.queuedCommentInterruptQueueId === input.queuedCommentInterruptId;
|
|
if (nativeQueuedSend && authorizedAt <= run.finishedAt! && !interruptedByThisClick)
|
|
return blocked("send_predates_stop", "Send the saved message again now that the previous run has stopped.");
|
|
if (coordinator && coordinator.phase !== "committed" && coordinator.resultId &&
|
|
readSystemRecovery(action.evidence)) {
|
|
const [completedResult] = await db.select({ id: nativeRunResults.id }).from(nativeRunResults).where(and(
|
|
eq(nativeRunResults.companyId, companyId), eq(nativeRunResults.issueId, issueId),
|
|
eq(nativeRunResults.runId, run.id), eq(nativeRunResults.id, coordinator.resultId),
|
|
eq(nativeRunResults.schemaStatus, "accepted"),
|
|
sql`${nativeRunResults.resultJson}#>>'{terminal,runTerminalState}' = 'succeeded'`,
|
|
)).limit(1);
|
|
const recovery = readSystemRecovery(action.evidence);
|
|
const snapshotSettled = completedResult && freshIntent && action.id && recovery?.state === "complete"
|
|
&& await verifiedWorkspaceSnapshotReceipt(db, { companyId, issueId, runId: run.id,
|
|
actionId: action.id, operationIds: recovery.preservationOperationIds });
|
|
const repairedTerminal = completedResult && coordinator.phase === "terminal_failure"
|
|
&& await hasVerifiedWorkspaceRepair(db, run);
|
|
// A Retry of this run whose files are saved applies the accepted result
|
|
// it already has, never a new agent turn. Saved files can be verified
|
|
// before their recovery attempt settles, so wait for that attempt too.
|
|
// The wait ends when preserved-result finalization coalesces this intent
|
|
// (resumeTaskRetry or the recovery sweep). Records that do not prove an
|
|
// applicable result (Stop, newer run, other owner) keep today's turn.
|
|
if (repairedTerminal && taskRetry?.sourceRunId === run.id && taskRetry.strategy === "evaluate"
|
|
&& await canResumePreservedResultFinalization(db, companyId, run.id,
|
|
["scheduled", "running", "awaiting_evidence"].includes(recovery?.state ?? "") ? "pending" : "ready"))
|
|
return blocked("finalizing_saved_result", "Paperclip is finishing the saved result.");
|
|
if (completedResult && !repairedTerminal && !(freshIntent && snapshotSettled)) {
|
|
// Nothing finishes the saved result while a stopped copyback owner is
|
|
// unverified. The run's live retained recovery records that hold; name
|
|
// its board action instead of an endless finalizing wait (RC-50).
|
|
const live = await db.select({
|
|
ownerHold: sql<boolean>`${issueRecoveryActions.evidence}#>>'{workspaceOwnerRecovery,state}' = 'operator_required'
|
|
and ${issueRecoveryActions.evidence}->'workspaceOwnerStop' is null`,
|
|
recoveryState: sql<string | null>`${issueRecoveryActions.evidence}#>>'{systemRecovery,state}'`,
|
|
retryClass: sql<string | null>`${issueRecoveryActions.evidence}#>>'{systemRecovery,failure,retryClass}'`,
|
|
evidence: issueRecoveryActions.evidence,
|
|
}).from(issueRecoveryActions).where(and(
|
|
eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, issueId),
|
|
inArray(issueRecoveryActions.status, ["active", "escalated"]),
|
|
sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`,
|
|
sql`${issueRecoveryActions.evidence} ? 'systemRecovery'`));
|
|
if (live.some(row => row.ownerHold))
|
|
return blocked("workspace_owner_stop_required", "Paperclip is waiting to confirm that the previous server stopped saving files. Your Retry is saved.");
|
|
// A spent recovery budget finishes nothing by itself. Another Retry
|
|
// starts one new recovery generation (task-retry), so name that action
|
|
// instead of an endless finalizing wait (RC-50b, s14).
|
|
if (!live.length || !live.every(row => row.recoveryState === "exhausted"))
|
|
return blocked("finalizing_saved_result", "Paperclip is finishing the saved result.");
|
|
if (!freshIntent || !live.every(row => savedFilesUnrecoverable(readSystemRecovery(row.evidence))))
|
|
return live.some(row => row.retryClass === "service_required")
|
|
? blocked("saved_work_recovery_refused", "Paperclip refused to recover the saved files because a safety check on them cannot pass. Your Retry is saved.")
|
|
: blocked("saved_work_recovery_failed", "Saving the agent's files failed. Retry to try again.");
|
|
}
|
|
}
|
|
// A repaired export no longer blocks execution. Its machine receipt still
|
|
// binds the saved user turn; finalized file state alone is insufficient.
|
|
if (["native_workspace_sync_out_unrecoverable", "native_workspace_sync_out_retry_exhausted"].includes(action.cause)
|
|
&& !freshIntent && !await hasVerifiedWorkspaceRepair(db, run)) return blocked("workspace_repair_required",
|
|
"Paperclip is recovering saved work before continuing. Your message is saved.");
|
|
if ((run.resultJson?.workspaceRestoreFailure === "restore_unsafe_archive" || hasRequiredWorkspaceRecovery(run.resultJson))
|
|
&& !freshIntent && !await hasVerifiedWorkspaceRepair(db, run)) {
|
|
return blocked("workspace_repair_required", "Paperclip is recovering saved work before continuing. Your message is saved.");
|
|
}
|
|
const cancelledStartup = await isCancelledNativeStartup(db, run, coordinator);
|
|
if (partiallyDeliveredQueue && run.runtimeMode !== "native" && !unusedAdmission && !cancelledStartup && !failedPreparation) return null;
|
|
if ((queuedInterrupt || queuedRequest) && !legacyUserTurn && !unusedAdmission &&
|
|
!(queuedRequest && run.runtimeMode === "native" &&
|
|
(run.status !== "cancelled" || authorizedAt > stoppedAt)) &&
|
|
!(queuedRequest && cancelledStartup && authorizedAt > run.finishedAt!) &&
|
|
!(nativeQueuedSend && (run.runtimeMode === "native" || cancelledStartup) &&
|
|
(authorizedAt > run.finishedAt! || interruptedByThisClick)) &&
|
|
!(queuedInterrupt && response?.source.requiresFreshSession && run.runtimeMode === "native")) return null;
|
|
// A saved native message sent after Stop is a new user direction. Older
|
|
// input cannot restart a cancelled run merely because it is still queued.
|
|
if (queuedRequest && !queuedInterrupt && run.status === "cancelled" && !unusedAdmission &&
|
|
!canContinueCancelledRun(run) && !(run.runtimeMode === "native" && authorizedAt > stoppedAt) &&
|
|
!(cancelledStartup && authorizedAt > run.finishedAt!)) return null;
|
|
if (retry && run.status === "cancelled" && !canContinueCancelledRun(run) && !cancelledStartup)
|
|
return blocked("cancelled_by_operator", "Inspect the stopped run and send a new message to continue.");
|
|
if (cancelledStartup) cancelledStartupIds.add(run.id);
|
|
if (run.runtimeMode !== "native" && !unusedAdmission && !legacyUserTurn && !cancelledStartup) return refused("previous_execution_unverified", PREVIOUS_EXECUTION_UNVERIFIED);
|
|
// A provider failure can finish the normal result/assessment commit path.
|
|
// Its accepted failed result is immutable history, not a live controller.
|
|
// Only a new user turn may pass this gate; the process/lease stop proofs
|
|
// below remain mandatory and no previous action is automatically replayed.
|
|
const [committedFailure] = coordinator?.phase === "committed" && coordinator.resultId && run.status === "failed"
|
|
? await db.select({ id: nativeRunResults.id }).from(nativeRunResults).where(and(
|
|
eq(nativeRunResults.companyId, companyId), eq(nativeRunResults.issueId, issueId),
|
|
eq(nativeRunResults.runId, run.id), eq(nativeRunResults.id, coordinator.resultId),
|
|
eq(nativeRunResults.schemaStatus, "accepted"),
|
|
sql`${nativeRunResults.resultJson}->'terminal'->>'runTerminalState' = 'failed'`,
|
|
)).limit(1)
|
|
: [];
|
|
// A terminal failure may retain a result accepted before checkpoint or
|
|
// cleanup failed. That immutable result is history, not an active commit.
|
|
// Keep it intact and require the same controller/process/lease stop proofs
|
|
// before admitting new user input; never apply or replay the old result.
|
|
const legacyReference = readNativeWorkspaceSyncReference(run.runnerProfileJson?.nativeWorkspaceSync);
|
|
const migratedSource = coordinator && !coordinator.resultId && !coordinator.leaseOwner
|
|
&& ["observed", "retryable_failure"].includes(coordinator.phase)
|
|
&& await hasVerifiedWorkspaceRepair(db, run)
|
|
? await readMigratedLegacyWorkspaceSource(db, { companyId, runId: run.id, reference: legacyReference }) : null;
|
|
// Offline migration fenced this exact unfinished turn and preserved its
|
|
// coordinator. A verified attachment permits new user input without
|
|
// inventing a terminal result or replaying the old session. All stop and
|
|
// ownership checks below still apply.
|
|
const preservedFailure = migratedSource?.issueId === issueId
|
|
&& migratedSource.unresolvedFinalization?.phase === coordinator?.phase;
|
|
if (!cancelledStartup && coordinator && (
|
|
(coordinator.phase !== "terminal_failure" && !committedFailure && !preservedFailure) || coordinator.leaseOwner ||
|
|
coordinator.failureDetail?.successorRunId)) return blocked("controller_settling",
|
|
run.status === "cancelled" && !coordinator.leaseOwner
|
|
? "The cancelled run still needs verified cleanup. Your message is saved. Inspect the run and its environment for details."
|
|
: "Waiting for the previous run to finish recovery. Your message will start automatically.");
|
|
const preserved = hasRequiredWorkspaceRecovery(run.resultJson) ? null
|
|
: await preserveNativeFailedWorkspace(db, run);
|
|
if (preserved && hasRequiredWorkspaceRecovery(preserved.resultJson)) return blocked("workspace_repair_required",
|
|
"Paperclip is recovering saved work before continuing. Your message is saved.");
|
|
const leases = await db.select()
|
|
.from(environmentLeases).where(and(
|
|
eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, run.id),
|
|
));
|
|
const remote = leases.some(lease => lease.provider !== "local");
|
|
if (remote) {
|
|
// Never interpret remote PIDs using the control-plane host's process table.
|
|
if (!leases.every(lease => hasRemoteTerminationReceipt(lease) || hasRecoveredCleanupStopReceipt(lease))) return blocked("remote_cleanup", "Waiting for the previous environment to stop. Your message will start automatically.");
|
|
if (run.runtimeMode === "native" && !input.dryRun && !leases.every(lease => completeTerminatedRemoteNativeSessionCleanup({
|
|
companyId, runId: run.id, remoteCleanupScope: remoteLeaseCleanupScope(lease)!,
|
|
}))) return refused("remote_cleanup", "Waiting for the previous environment to stop. Your message will start automatically.");
|
|
} else {
|
|
if (leases.some(lease => !lease.releasedAt || lease.cleanupStatus === "failed")) return blocked("local_cleanup", "Waiting for the previous environment to finish cleanup. Your message will start automatically.");
|
|
if (!unusedAdmission && !cancelledStartup && !failedPreparation) {
|
|
if (freshIntent && (run.processPid || run.processGroupId) && !await recordedLocalExecutionStopped(db, run))
|
|
return blocked("process_identity_missing", "Paperclip is verifying that the previous session stopped.");
|
|
// A missing process identity is not evidence that a provider exited.
|
|
if (!run.processPid && !run.processGroupId &&
|
|
!await hasNativeLocalProcessStop(db, companyId, run.id) &&
|
|
!await hasHistoricalSuspendedNativeSession(db, run) &&
|
|
!await nativeRunNeverLaunched(db, run) &&
|
|
!(run.runtimeMode === "native" &&
|
|
await recordUnidentifiedLocalRunnerStop(db, run, { record: !input.dryRun }))) return blocked("process_identity_missing", "The previous run has no verified stop record. Paperclip cannot start this message yet.");
|
|
if (run.processPid && !processStopped(run.processPid)) return blocked("process_running", "Waiting for the previous process to stop. Your message will start automatically.");
|
|
if (run.processGroupId && !processStopped(-run.processGroupId)) return blocked("process_running", "Waiting for the previous process to stop. Your message will start automatically.");
|
|
}
|
|
}
|
|
// The runner keeps a closing or quarantined provider session in process
|
|
// memory, keyed by this task. Its admission gate would refuse a successor
|
|
// at once, so hold the input here with the matching reason instead of
|
|
// dispatching a run that can only fail. A pending close settles by itself
|
|
// and its release wakes deferred execution; the held input then starts.
|
|
const retainedCleanup = !remote && run.runtimeMode === "native"
|
|
? nativeSessionCleanupBlock({ companyId, issueId }) : null;
|
|
if (retainedCleanup === "pending" && run.errorCode !== "native_session_cleanup_quarantined")
|
|
return blocked("local_cleanup", "Waiting for the previous provider session to finish cleanup. Your message will start automatically.");
|
|
if (!remote && run.runtimeMode === "native" &&
|
|
(run.errorCode === "native_session_cleanup_quarantined" || retainedCleanup === "operator_required")) {
|
|
// Never permission to resume the interrupted turn: new input and a
|
|
// native Retry both start a fresh provider conversation, and only after
|
|
// the old runner, provider and tools are proven stopped.
|
|
const { verifyStoppedNativeSessionForContinuation, provenStoppedOwnNativeSession, retireStoppedSessionCleanupGates } =
|
|
await import("./native-runtime/native-session-executor.js");
|
|
let stopped = await verifyStoppedNativeSessionForContinuation(db, run);
|
|
// The run's own session (s9107 t04): its saved state may be past the
|
|
// states the inventory proof reads, but its exact runner and every
|
|
// provider process it recorded are durably proven stopped.
|
|
const ownStop = stopped || retainedCleanup === "pending" ? null : await provenStoppedOwnNativeSession(db, run);
|
|
if (ownStop && run.runnerInstanceId) {
|
|
const runnerInstanceId = run.runnerInstanceId;
|
|
stopped = { evidence: { schema: "paperclip.stopped_own_native_session.v1", runId: run.id,
|
|
runnerInstanceId, ...ownStop },
|
|
retire: () => { try { return completeTerminatedLocalNativeSessionCleanup({ companyId, runId: run.id, runnerInstanceId }); }
|
|
catch { return false; } } };
|
|
}
|
|
// A source the gate refused before it launched (s9107 rerun t05: a
|
|
// follow-up run after its predecessor's failed close) owns no process.
|
|
// Once the task's gate is retired from its stopped owner, nothing holds
|
|
// the old session any more.
|
|
if (!stopped && retainedCleanup === "operator_required" && await nativeGatedRunNeverLaunched(db, run)
|
|
// A preview stays read-only; the committing call retires the gate.
|
|
&& await retireStoppedSessionCleanupGates(db, { companyId, issueId, preview: input.dryRun === true })) {
|
|
stopped = { evidence: { schema: "paperclip.unlaunched_gated_source.v1", runId: run.id },
|
|
retire: () => nativeSessionCleanupBlock({ companyId, issueId }) !== "operator_required" };
|
|
}
|
|
if (!stopped) return retry
|
|
? blocked("cleanup_quarantined", "Send a new message after the previous provider has stopped.")
|
|
: blocked("local_cleanup", "Waiting for the previous provider and its tools to stop. Your message is saved.");
|
|
stoppedSessions.push(stopped);
|
|
}
|
|
if (nativeQueuedSend && await hasPendingRunCleanup(db, companyId, run.id))
|
|
return blocked("cleanup_pending", "Waiting for the previous run to finish cleanup. Your message is saved.");
|
|
if (freshIntent && !await hasVerifiedWorkspaceRepair(db, run)) {
|
|
freshSources.set(run.id, freshIntent);
|
|
freshCandidates.push({ run, intent: freshIntent, cause: action.cause, evidence: strategyEvidence, remote,
|
|
leaseIds: leases.map(lease => lease.id) });
|
|
}
|
|
sources.push(run);
|
|
}
|
|
if (freshCandidates.length) {
|
|
// R2: keep the task's own workspace when its single prior writer is
|
|
// contained and the directory is safe to reuse. Otherwise start the fresh
|
|
// isolated attempt and leave every source directory untouched. Several
|
|
// holds can describe the same failed run; each hold must permit the reuse.
|
|
const sourceRunIds = new Set(freshCandidates.map(candidate => candidate.run.id));
|
|
const only = sourceRunIds.size === 1 && freshCandidates.every(candidate => !candidate.remote) ? freshCandidates[0]! : null;
|
|
let retained: { id: string; cwd: string | null } | null = null;
|
|
const gitProbes = new Map<string, Promise<boolean>>();
|
|
for (const candidate of only ? freshCandidates : []) {
|
|
const checked = await retainedIssueWorkspaceForRetry(db, { companyId, issueId, agentId,
|
|
successorRunId: input.successorRunId, cause: candidate.cause, source: candidate.run, evidence: candidate.evidence, gitProbes });
|
|
if (!checked || (retained && (retained.id !== checked.id || retained.cwd !== checked.cwd))) { retained = null; break; }
|
|
retained = checked;
|
|
}
|
|
const last = freshCandidates.at(-1)!;
|
|
readiness = retained && only ? { mode: "reuse_issue_workspace", retryIntentId: only.intent, sourceRunId: only.run.id,
|
|
workspaceId: retained.id, cwd: retained.cwd }
|
|
: { mode: "fresh_isolated_workspace", retryIntentId: last.intent, sourceRunId: last.run.id,
|
|
retainedSourceIds: [...new Set(freshCandidates.flatMap(candidate => candidate.leaseIds))] };
|
|
}
|
|
const nativeSources = sources.filter(run => run.runtimeMode === "native");
|
|
const executedSources = sources.filter(run => run.runtimeMode === "native" || run.errorCode !== "execution_reconciliation_required" || run.startedAt);
|
|
if (!executedSources.length || (retry && !sources.some(run => run.id === input.failedRunId)) ||
|
|
(taskRetry && !sources.some(run => run.id === taskRetry.sourceRunId))) return refused("previous_execution_unverified", PREVIOUS_EXECUTION_UNVERIFIED);
|
|
const [active] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, companyId),
|
|
or(eq(heartbeatRuns.nativeIssueId, issueId), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`),
|
|
inArray(heartbeatRuns.status, ["running", "queued", "scheduled_retry"]),
|
|
ne(heartbeatRuns.id, input.successorRunId),
|
|
)).limit(1);
|
|
if (active) return blocked("execution_active", "Waiting for the current run. Your message is saved.");
|
|
const previous = taskRetry ? sources.find(run => run.id === taskRetry.sourceRunId)!
|
|
: executedSources.sort((a, b) => b.createdAt.getTime() - a.createdAt.getTime())[0]!;
|
|
// Prove required task history is available before retiring any hold.
|
|
await buildExecutionContinuation({ db, companyId, issueId, agentId,
|
|
context: { previousRunId: previous.id, wakeCommentId: commentId },
|
|
...(nativeQueuedSend && task.status === "done" ? { prospectiveTaskStatus: "todo" as const } : {}),
|
|
summary: null, exposeLowTrustRaw: false });
|
|
const continuation = { previousRunId: previous.id, commentId, ...(retry ? { failedRunId: input.failedRunId! } : {}),
|
|
...(queuedInterrupt ? { queuedCommentInterruptId: input.queuedCommentInterruptId! } : {}),
|
|
...(taskRetry ? { taskRetryRequestId: input.taskRetryRequestId! } : {}), ...(readiness ? { readiness } : {}) };
|
|
if (input.dryRun) return continuation;
|
|
for (const stopped of stoppedSessions) {
|
|
if (!stopped.retire()) return blocked("local_cleanup", "The previous provider cleanup changed. Your message is saved.");
|
|
await appendHeartbeatRunEvent(db, {
|
|
companyId, runId: String(stopped.evidence.runId), agentId,
|
|
eventType: "native.stopped_conversation_verified", stream: "system", level: "info",
|
|
message: "The old runner and provider stopped. New user input will use a fresh session; prior action outcomes remain recorded.",
|
|
payload: { ...stopped.evidence, successorRunId: input.successorRunId,
|
|
...(input.taskRetryRequestId ? { taskRetryRequestId: input.taskRetryRequestId } : {}) },
|
|
});
|
|
}
|
|
const authorization = { ...continuation, actorId, commentId, ...(response ? { interactionId: response.source.interactionId } : {}), ...(retry ? { failedRunId: input.failedRunId } : {}),
|
|
...(comment ? { commentUpdatedAt: comment.updatedAt.toISOString(),
|
|
commentBodyHash: createHash("sha256").update(comment.body).digest("hex") } : {}),
|
|
...(queuedInterrupt ? { queuedCommentInterruptId: input.queuedCommentInterruptId } : {}),
|
|
...(queuedRequest ? { queuedCommentRequestId: input.queuedCommentRequestId } : {}), runId: input.successorRunId,
|
|
previousRunId: previous.id, recordedAt: new Date().toISOString() };
|
|
if (input.queuedCommentInterruptId) await db.update(agentWakeupRequests).set({
|
|
payload: sql`jsonb_set(${agentWakeupRequests.payload}, '{queuedCommentInterrupt,continuation}', ${JSON.stringify(authorization)}::jsonb)`,
|
|
}).where(and(eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.id, input.queuedCommentInterruptId),
|
|
eq(agentWakeupRequests.agentId, agentId), eq(agentWakeupRequests.status, "deferred_issue_execution")));
|
|
if (cancelledStartupIds.size) await db.transaction(async tx => {
|
|
await notifyExecutionRunWork(tx, { companyId, agentId });
|
|
for (const runId of cancelledStartupIds) {
|
|
await tx.update(nativeRunFinalizations).set({
|
|
phase: "terminal_failure", failureCode: "native_startup_cancelled", nextAttemptAt: null,
|
|
controlDeadlineAt: null, updatedAt: new Date(),
|
|
}).where(and(eq(nativeRunFinalizations.companyId, companyId), eq(nativeRunFinalizations.runId, runId)));
|
|
await tx.update(heartbeatRuns).set({
|
|
...(nativeSources.some(run => run.id === runId) ? { nativePhase: "terminal_failure", nativePhaseUpdatedAt: new Date() } : {}),
|
|
executionControlDeadlineAt: null,
|
|
}).where(and(eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, runId)));
|
|
}
|
|
});
|
|
if (nativeSources.length) await db.update(nativeRunFinalizations).set({
|
|
failureDetail: sql`coalesce(${nativeRunFinalizations.failureDetail}, '{}'::jsonb) || ${JSON.stringify({ replacementDenied: "explicit_user_continuation" })}::jsonb`,
|
|
updatedAt: new Date(),
|
|
}).where(and(eq(nativeRunFinalizations.companyId, companyId), inArray(nativeRunFinalizations.runId, nativeSources.map(run => run.id))));
|
|
const unrecoverable = actions.map(action => freshSources.has(String(action.evidence.runId))
|
|
? unrecoverableSavedFilesFailure(readSystemRecovery(action.evidence)) : null).find(failure => failure !== null) ?? null;
|
|
if (actions.length) await db.transaction(async tx => {
|
|
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId, issueId });
|
|
for (const action of actions) {
|
|
await tx.update(issueRecoveryActions).set({
|
|
status: "resolved", outcome: "cancelled", resolvedAt: new Date(), updatedAt: new Date(),
|
|
nextAction: unrecoverable ? unrecoverableSavedFilesNotice(unrecoverable)
|
|
: "The user started a fresh conversation turn. Prior action outcomes remain recorded.",
|
|
resolutionNote: "The user continued after the prior execution stopped. No action outcomes were inferred.",
|
|
wakePolicy: null, monitorPolicy: null,
|
|
evidence: { ...action.evidence, explicitUserContinuation: authorization,
|
|
...(freshSources.has(String(action.evidence.runId)) && readSystemRecovery(action.evidence) ? {
|
|
systemRecovery: { ...readSystemRecovery(action.evidence)!,
|
|
revision: readSystemRecovery(action.evidence)!.revision + 1, freshAttemptRunId: input.successorRunId },
|
|
} : {}),
|
|
...(action.evidence.automaticRecovery ? { automaticRecovery: {
|
|
...(action.evidence.automaticRecovery as Record<string, unknown>), replay: "explicit_user_continuation",
|
|
} } : {}),
|
|
},
|
|
}).where(eq(issueRecoveryActions.id, action.id));
|
|
}
|
|
if (unrecoverable) {
|
|
const { issueService } = await import("./issues.js");
|
|
await issueService(db).addComment(issueId, unrecoverableSavedFilesNotice(unrecoverable), {},
|
|
{ authorType: "system" }, tx);
|
|
}
|
|
});
|
|
await persistActivity(db, { companyId, actorType: "user", actorId,
|
|
action: "issue.execution_recovery_settled", entityType: "issue", entityId: issueId,
|
|
details: { continuation: retry ? "explicit_user_retry" : "explicit_user_message", ...authorization,
|
|
recoveryActionIds: actions.map(action => action.id), previousRunIds: sources.map(run => run.id) },
|
|
});
|
|
if (!input.taskRetryRequestId) await retireTaskRetriesContinuedBy(db, { companyId, issueId, agentId,
|
|
sourceRunIds: sources.map(run => run.id) });
|
|
if (readiness) await persistActivity(db, { companyId, actorType: "system", actorId: "task-recovery",
|
|
action: readiness.mode === "reuse_issue_workspace" ? "issue.retry_workspace_reused" : "issue.fresh_workspace_selected",
|
|
entityType: "issue", entityId: issueId, details: { ...readiness, runId: input.successorRunId } });
|
|
if (input.taskRetryRequestId) {
|
|
const payload = sql`jsonb_set(${agentWakeupRequests.payload} #- '{taskRetry,continuation}', '{taskRetry,strategy}',
|
|
${JSON.stringify(readiness?.mode === "fresh_isolated_workspace" ? "fresh_isolated_workspace" : "continue")}::jsonb)`;
|
|
// Preserve the action ledger as the authority when it exists. Preparation
|
|
// can fail before that ledger exists, so only that path needs a fallback.
|
|
const hasActionAuthorization = actions.some(action => (action.evidence.runId ?? action.evidence.sourceRunId) === previous.id);
|
|
await db.update(agentWakeupRequests).set({ payload: hasActionAuthorization ? payload
|
|
: sql`jsonb_set(${payload}, '{taskRetry,continuation}', ${JSON.stringify(authorization)}::jsonb)`,
|
|
}).where(and(eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.id, input.taskRetryRequestId)));
|
|
}
|
|
await liftInterruptionBlock(db, task, actions, sources, input.successorRunId);
|
|
return continuation;
|
|
}
|
|
|
|
type ContinuationRetryAuthority = {
|
|
continuation: { commentId: string } | { commentId: null; taskRetryRequestId: string };
|
|
actorId: string; sourceAuthorizationId: string; evidence: Record<string, unknown>;
|
|
/** The workspace a fresh-workspace Retry selected, carried to the successor. */
|
|
readiness?: CarriedRetryReadiness;
|
|
};
|
|
|
|
type CarriedRetryReadiness = z.infer<typeof freshReadinessSchema>;
|
|
|
|
async function readCommentRetryAuthority(db: Db, input: {
|
|
companyId: string; issueId: string; parentRunId: string; explicit: Record<string, unknown>;
|
|
}): Promise<ContinuationRetryAuthority | null> {
|
|
const { companyId, issueId, explicit } = input;
|
|
if (!z.string().guid().safeParse(explicit.commentId).success) return null;
|
|
const commentId = explicit.commentId as string;
|
|
const [comment] = await db.select().from(issueComments).where(and(
|
|
eq(issueComments.companyId, companyId), eq(issueComments.issueId, issueId),
|
|
eq(issueComments.id, commentId), eq(issueComments.authorType, "user"),
|
|
isNull(issueComments.createdByRunId), isNull(issueComments.deletedAt),
|
|
));
|
|
if (!comment?.authorUserId || !comment.body.trim()) return null;
|
|
const commentBodyHash = createHash("sha256").update(comment.body).digest("hex");
|
|
const receipts = await db.select().from(issueRecoveryActions).where(and(
|
|
eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, issueId),
|
|
eq(issueRecoveryActions.status, "resolved"),
|
|
sql`${issueRecoveryActions.evidence}->'explicitUserContinuation'->>'runId' = ${input.parentRunId}`,
|
|
));
|
|
const receipt = receipts.find(row => {
|
|
const auth = row.evidence.explicitUserContinuation as Record<string, unknown> | undefined;
|
|
return auth && auth.previousRunId === explicit.previousRunId && auth.commentId === commentId &&
|
|
auth.actorId === comment.authorUserId && !auth.failedRunId && !auth.queuedCommentInterruptId &&
|
|
auth.commentUpdatedAt === comment.updatedAt.toISOString() && auth.commentBodyHash === commentBodyHash;
|
|
});
|
|
return receipt ? { continuation: { commentId }, actorId: comment.authorUserId, sourceAuthorizationId: receipt.id,
|
|
evidence: { commentUpdatedAt: comment.updatedAt.toISOString(), commentBodyHash } } : null;
|
|
}
|
|
|
|
/** A task Retry admits one plain continuation in the saved workspace. When that
|
|
* run fails transiently, its bounded retry keeps the same actor and receipt;
|
|
* chat and queued-message Retries keep their own contracts.
|
|
*
|
|
* A fresh-workspace Retry (selected at admission, or when its preparation found
|
|
* the saved files unrestorable) is renewable only for a run that a graceful
|
|
* shutdown stopped before provider work: the person asked for that fresh
|
|
* attempt and it never ran. The successor carries the same workspace selection,
|
|
* so it prepares the same attempt rather than the workspace the Retry left. */
|
|
async function readTaskRetryRetryAuthority(db: Db, input: {
|
|
companyId: string; issueId: string; agentId: string; parentRunId: string; explicit: Record<string, unknown>;
|
|
workspaceSelectionRenewable: boolean;
|
|
}): Promise<ContinuationRetryAuthority | null> {
|
|
const { companyId, issueId, explicit } = input;
|
|
const requestId = z.string().guid().safeParse(explicit.taskRetryRequestId);
|
|
if (explicit.commentId !== null || !requestId.success || explicit.queuedCommentInterruptId) return null;
|
|
const admittedReadiness = explicit.readiness === undefined ? null : freshReadinessSchema.safeParse(explicit.readiness);
|
|
if (admittedReadiness && (!input.workspaceSelectionRenewable || !admittedReadiness.success
|
|
|| admittedReadiness.data.retryIntentId !== requestId.data
|
|
|| admittedReadiness.data.sourceRunId !== explicit.previousRunId)) return null;
|
|
const [wake] = await db.select().from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.id, requestId.data),
|
|
eq(agentWakeupRequests.agentId, input.agentId), eq(agentWakeupRequests.reason, "task_retry"),
|
|
eq(agentWakeupRequests.requestedByActorType, "user"), eq(agentWakeupRequests.status, "coalesced"),
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId}`,
|
|
));
|
|
const intent = readTaskRetryIntent(wake?.payload);
|
|
const freshAttempt = intent?.strategy === "fresh_isolated_workspace";
|
|
if (!wake?.requestedByActorId || !wake.runId || !intent || intent.generationId !== wake.id ||
|
|
intent.actorId !== wake.requestedByActorId ||
|
|
!(intent.strategy === "continue" || (freshAttempt && input.workspaceSelectionRenewable)) ||
|
|
intent.chatActionId || intent.queuedChatRequestId || intent.canonicalIntentId) return null;
|
|
// The admitted run's authority may be its action ledger, an earlier
|
|
// automatic-retry receipt, or the Retry receipt itself. Dispatch validation
|
|
// of the parent below proves which one applies.
|
|
const receipts = await db.select({ id: issueRecoveryActions.id, evidence: issueRecoveryActions.evidence })
|
|
.from(issueRecoveryActions).where(and(
|
|
eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, issueId),
|
|
eq(issueRecoveryActions.status, "resolved"),
|
|
sql`${issueRecoveryActions.evidence}->'explicitUserContinuation'->>'runId' = ${input.parentRunId}`,
|
|
sql`${issueRecoveryActions.evidence}->'explicitUserContinuation'->>'taskRetryRequestId' = ${wake.id}`,
|
|
));
|
|
let readiness: CarriedRetryReadiness | undefined = admittedReadiness?.success ? admittedReadiness.data : undefined;
|
|
if (freshAttempt && readiness?.mode !== "fresh_isolated_workspace") {
|
|
if (readiness) return null;
|
|
// An automatic retry of the fresh attempt carries the selection on its receipt.
|
|
const inherited = receipts.map(row => freshReadinessSchema.safeParse(
|
|
((row.evidence.explicitUserContinuation as Record<string, unknown> | undefined)?.automaticRetry as
|
|
Record<string, unknown> | undefined)?.readiness)).find(parsed => parsed.success)?.data;
|
|
if (inherited) {
|
|
if (inherited.mode !== "fresh_isolated_workspace" || inherited.retryIntentId !== wake.id
|
|
|| inherited.sourceRunId !== intent.sourceRunId) return null;
|
|
readiness = inherited;
|
|
} else {
|
|
// Preparation selected the fresh attempt after admission. Rebuild the
|
|
// selection it recorded: this Retry, its source and that source's leases.
|
|
if (intent.sourceRunId !== explicit.previousRunId) return null;
|
|
const leases = await db.select({ id: environmentLeases.id }).from(environmentLeases).where(and(
|
|
eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, intent.sourceRunId)));
|
|
readiness = { mode: "fresh_isolated_workspace", retryIntentId: wake.id, sourceRunId: intent.sourceRunId,
|
|
retainedSourceIds: leases.map(lease => lease.id) };
|
|
}
|
|
}
|
|
return { continuation: { commentId: null, taskRetryRequestId: wake.id }, actorId: wake.requestedByActorId,
|
|
sourceAuthorizationId: receipts[0]?.id ?? wake.id, evidence: {}, ...(readiness ? { readiness } : {}) };
|
|
}
|
|
|
|
/** A graceful server shutdown (an upgrade or restart) stopped this legacy run
|
|
* while it was still preparing: the bootstrap evidence proves that no provider
|
|
* work started. The interruption is not a failed attempt, so the user's
|
|
* original authorization still covers the work it admitted.
|
|
*/
|
|
function isShutdownInterruptedPreparation(run: typeof heartbeatRuns.$inferSelect): boolean {
|
|
const recovery = run.resultJson?.executionRecovery;
|
|
const evidence = recovery && typeof recovery === "object" && !Array.isArray(recovery)
|
|
? recovery as Record<string, unknown> : null;
|
|
return run.status === "interrupted" && isServerReplacementInterruption(run)
|
|
&& evidence?.kind === "bootstrap" && evidence.providerWorkStarted === false;
|
|
}
|
|
|
|
/** Re-admit a bounded automatic retry without lending it its parent's receipt.
|
|
* The caller holds the task lock and creates the successor in this transaction.
|
|
* Unchanged user messages and plain task Retries are renewable; Interrupt and
|
|
* failed-run retry intents keep their separate one-run contracts. The
|
|
* scheduler still owns retry and cleanup policy.
|
|
*
|
|
* A failed or timed-out parent must still hold the task's claim. A parent that
|
|
* a graceful shutdown stopped before provider work may also find the claim
|
|
* released: releases without this rule refused its continuation and released
|
|
* the claim, leaving the task in progress with no run. Admission then restores
|
|
* the claim to the parent, as the shutdown path leaves it, and every other
|
|
* check below still applies.
|
|
*/
|
|
export async function admitExplicitContinuationRetry(input: {
|
|
db: Db; companyId: string; issueId: string; agentId: string;
|
|
parentRunId: string; successorRunId: string; now: Date;
|
|
}): Promise<{ previousRunId: string; commentId: string | null; taskRetryRequestId?: string } | null> {
|
|
const { db, companyId, issueId, agentId } = input;
|
|
const [task] = await db.select().from(issues).where(and(
|
|
eq(issues.companyId, companyId), eq(issues.id, issueId),
|
|
)).for("update");
|
|
const [parent] = await db.select().from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, input.parentRunId),
|
|
eq(heartbeatRuns.agentId, agentId),
|
|
)).for("update");
|
|
const shutdownInterrupted = Boolean(parent && isShutdownInterruptedPreparation(parent));
|
|
const releasedClaim = Boolean(task && task.executionRunId === null && shutdownInterrupted);
|
|
if (!task || task.assigneeAgentId !== agentId || (task.executionRunId !== input.parentRunId && !releasedClaim) ||
|
|
["done", "cancelled"].includes(task.status) || !parent ||
|
|
(!["failed", "timed_out"].includes(parent.status) && !isAgentDirectoryPreparationWait(parent) && !shutdownInterrupted) ||
|
|
!parent.finishedAt ||
|
|
parent.runtimeMode !== "legacy" || parent.contextSnapshot?.issueId !== issueId ||
|
|
(parent.nativeIssueId !== null && parent.nativeIssueId !== issueId) ||
|
|
adapterExecutionControls.has(parent.id) ||
|
|
await getExecutionBlocker(db, companyId, issueId)) return null;
|
|
const context = parent.contextSnapshot;
|
|
const explicit = context.explicitUserContinuation as Record<string, unknown> | undefined;
|
|
if (!explicit || !z.string().guid().safeParse(explicit.previousRunId).success || explicit.failedRunId) return null;
|
|
const authority = explicit.commentId === null
|
|
? await readTaskRetryRetryAuthority(db, { companyId, issueId, agentId, parentRunId: parent.id, explicit,
|
|
workspaceSelectionRenewable: shutdownInterrupted })
|
|
: await readCommentRetryAuthority(db, { companyId, issueId, parentRunId: parent.id, explicit });
|
|
if (!authority) return null;
|
|
const [superseding] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, companyId),
|
|
or(eq(heartbeatRuns.nativeIssueId, issueId), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`),
|
|
ne(heartbeatRuns.id, parent.id), ne(heartbeatRuns.id, input.successorRunId),
|
|
or(eq(heartbeatRuns.retryOfRunId, parent.id),
|
|
// Keep PostgreSQL's timestamp precision: Date would round the parent
|
|
// down and mistake an older run in the same millisecond for a successor.
|
|
sql`${heartbeatRuns.createdAt} > (select source.created_at from heartbeat_runs source
|
|
where source.company_id = ${companyId} and source.id = ${parent.id})`,
|
|
inArray(heartbeatRuns.status, ["queued", "running", "scheduled_retry"])),
|
|
)).limit(1);
|
|
if (superseding) return null;
|
|
const [{ name: agentName } = { name: null }] = releasedClaim
|
|
? await db.select({ name: agents.name }).from(agents).where(and(eq(agents.companyId, companyId), eq(agents.id, agentId)))
|
|
: [];
|
|
const claim = (executionRunId: string | null) => db.update(issues).set(executionRunId
|
|
? { executionRunId, executionAgentNameKey: normalizeAgentNameKey(agentName), executionLockedAt: input.now }
|
|
: { executionRunId: null, executionAgentNameKey: null, executionLockedAt: null })
|
|
.where(and(eq(issues.companyId, companyId), eq(issues.id, issueId)));
|
|
if (releasedClaim) await claim(parent.id);
|
|
// Reuse dispatch's complete source/actor/receipt validation on the parent.
|
|
// It proves only the parent; the new receipt below authorizes the successor.
|
|
try {
|
|
await buildExecutionContinuation({ db, companyId, issueId, agentId, runId: parent.id,
|
|
context, summary: null, exposeLowTrustRaw: false });
|
|
} catch (error) {
|
|
if (error instanceof Error && ["continuation_user_authorization_missing",
|
|
"continuation_source_context_missing", "continuation_task_ownership_changed"].includes(error.message)) {
|
|
if (releasedClaim) await claim(null);
|
|
return null;
|
|
}
|
|
throw error;
|
|
}
|
|
const continuation = { previousRunId: parent.id, ...authority.continuation };
|
|
await db.insert(issueRecoveryActions).values({
|
|
companyId, sourceIssueId: issueId, kind: "active_run_watchdog",
|
|
cause: "explicit_user_continuation_retry", fingerprint: input.successorRunId,
|
|
status: "resolved", outcome: "retry_authorized", resolvedAt: input.now,
|
|
nextAction: authority.continuation.commentId
|
|
? "The bounded retry has its own authorization for the unchanged user message."
|
|
: "The bounded retry has its own authorization for the same task Retry.",
|
|
evidence: { explicitUserContinuation: {
|
|
...continuation, actorId: authority.actorId, runId: input.successorRunId, ...authority.evidence,
|
|
recordedAt: input.now.toISOString(), automaticRetry: {
|
|
sourceRunId: parent.id, sourceAuthorizationId: authority.sourceAuthorizationId,
|
|
...(authority.readiness ? { readiness: authority.readiness } : {}),
|
|
},
|
|
} },
|
|
});
|
|
return continuation;
|
|
}
|