Files
PaperClipAI/server/src/services/execution-blocker.ts
T
DottaandClaude Opus 5.5 f6cb3e43f4 fix(admission): admit new work independent of other runs' cleanup and accounting (#19944)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Budget admission decides if new agent work can start in a company
> - Five admission gates waited on the cleanup or accounting of other
runs
> - The restart-chaos campaign found that these gates froze companies
for up to 10 minutes, caused false budget pauses, denied in-run writes,
and dropped wakes, comments and Retries
> - The durable ownership plan (Decision 1) removes the company-wide
accounting gate. It does not narrow it again
> - This pull request removes the five gates. Unfinished accounting now
adds its bounded estimate to exposure
> - The benefit is that one run's unsettled accounting never stops the
other work in its company

## Linked Issues or Issue Description

Refs #19831 (plan, section D), #19838 (inventory, finding 3), #19927
(Phase 2 reconciler data), #19497, #19724, #19826.

**What happened?**
Admission refused new work in these cases, with no reached hard stop and
no pause:
- a terminal run awaited accounting (`waitingRunCount`);
- a native run recovered its accounting (`recoveringRunCount`);
- a budget-owned scope hold awaited reconciliation, after the hard stop
no longer applied;
- a budget input was older than 15 s (`budget_evaluation_work`);
- the predecessor run row of a task was missing (`hasPendingRunCleanup`
failed closed).

**Expected behavior**
Admission never waits on the cleanup or accounting of another run. Only
a reached hard stop, an explicit pause or an inactive company refuses
new work (invariant 3).

**Steps to reproduce**
Make a company with an active budget policy. Leave one failed run with
`cost_accounting_pending = true`. Request work for another agent. On
`master` admission refuses with "Company cannot start work while
completed runs await accounting."

## What Changed

- `budgets.ts`: admission refuses only for a reached hard stop or an
explicit pause. The code removes `observedBlocks`, the
`recoveringRunCount` check, the stale `budgetHeld` refusal and the 15 s
input-age gate. It also removes the unused
`inspectInvocationEligibility`. `reconcileScope` no longer returns
accounting holds.
- `budget-exposure.ts`: removes the `waiting` and `recovering`
aggregates and the own-task subquery. The admission query is smaller.
Unfinished accounting stays exposure: the larger of the held reservation
and the observed cost.
- `budget-evaluation-work.ts`: removes `oldestBudgetInputAt`, which
nothing reads now.
- `execution-blocker.ts`: `hasPendingRunCleanup` returns false for a
missing run row. A run row that is deleted removes its finalization
coordinator and nulls its lease references, so the row owns nothing. A
predecessor that owns a lease or a coordinator still holds the successor
of its own task.
- `recovery/ownership-invariant-report.ts`: the I3 check no longer
mirrors the removed gate. It reports a wake request or a run that was
refused with an accounting-hold reason. Any such report is a regression.
- `doc/SPEC-implementation.md`: adds the product rule "Admission is
independent of cleanup and accounting". `doc/execution-semantics.md` is
aligned.

## Verification

- New `server/src/__tests__/admission-independent-of-cleanup.test.ts`,
on embedded Postgres. It has one case per gate. All six cases fail on
`master` and pass here. A seventh case (an explicit pause still refuses)
passes on both. One case also asserts invariant 3 together with
exposure: two failed runs that each hold a 400-cent reservation fill a
1000-cent budget, and a hard stop still refuses.
- Existing tests that asserted the old gates now assert admission
(invariants 3 and 4): `closed-native-receipt-accounting`,
`budgets-service`, `budget-evaluation-work`,
`heartbeat-held-comment-wake`, `ownership-reconciliation-report`. In
`heartbeat-held-comment-wake`, each saved comment, Retry, assignment and
continuation is delivered exactly once while another run's accounting is
pending. Settling accounting later does not deliver it again.
- All six files pass locally, one file per process. The server package
typecheck passes.
- Speed: admission is on the turn path. This change only removes work
from it: two correlated aggregates, one own-task subquery and one
`budget_evaluation_work` read per policy. No wait or I/O is added before
`turn.start`.
- Not run here: the chaos waves (K13 storm, K6 crash loop, K4 long
holds). The coordinating session schedules them on the shared harness.

## Risks

- Behavior change: work starts while accounting for other runs settles.
Spend stays bounded by the reservation capacity check and the hard stop,
which read the ledger plus observed cost. A run that has no reservation
and no observed cost adds zero exposure until its charge commits. This
is the gap that Decision 1 accepts. Budget enforcement was already best
effort (SPEC 11.6).
- A budget-owned pause with no reached hard stop no longer refuses work
in budget admission. The paused company or agent is still refused by the
existing pause checks until the budget worker lifts the pause.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 21:46:39 -06:00

384 lines
27 KiB
TypeScript

import { isPreDispatchReviewWait, neverStartedDispatchCancellationPredicate } from "./pre-dispatch-review-wait.js";
import { canonicalExecutionUuid } from "./execution-identity.js";
import { preDispatchExecutionWaitRecoveryActionPredicate } from "./pre-dispatch-execution-wait.js";
import { and, desc, eq, gt, inArray, not, or, sql, type SQL } from "drizzle-orm";
import { conversationRecoveryActionPredicate, getConversationOwnershipBlocker } from "./conversation-continuation.js";
import { z } from "zod";
import { agentWakeupRequests, chatConversations, chatEndpoints, environmentLeases, heartbeatRunEvents, heartbeatRuns, issueComments, issues, issueRecoveryActions, nativeRunFinalizations, nativeRunResults, toolConnections, workspaceOperations, type Db } from "@paperclipai/db";
import { canContinueCancelledRun, readRunCancellation } from "./run-cancellation.js";
import { canRetryStoppedRun } from "./cancelled-native-startup.js";
import { queuedCommentIdsFromWakePayload } from "./issue-queued-comment-queue.js";
import { EXECUTION_RECONCILIATION_CAUSES, type ExecutionBlocker } from "@paperclipai/shared";
import { hasRequiredWorkspaceRecovery, LEGACY_WORKSPACE_RECOVERY_SCHEMA, nativeRunHasNoWorkspaceAccess } from "./workspace-restore-recovery-state.js";
import { unreconciledNativeExportPredicate } from "./native-runtime/native-workspace-reconstruction.js";
import { hasRecoveredCleanupStopReceipt, hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
import { readNativeWorkspaceSyncReference } from "./native-runtime/native-workspace-sync.js";
import { completedWorkspaceRecoveryCondition, hasVerifiedWorkspaceRepair, workspaceRecoveryDisplay } from "./workspace-recovery-receipt.js";
import { nativeManualAdmissionHoldsExecution } from "./native-runtime/native-runner-ownership.js";
const blockerContextKeys = ["issueId", "taskId", "source", "connectionIntentResolution",
"interactionKind", "interactionStatus", "mutation", "interactionId"] as const;
const blockerProfileKeys = ["adapterDispatch", "nativeExecutionInput", "nativeWorkspaceSync", "remoteWorkspace", "legacyWorkspaceRecoveryMigration", "finalizedWorkspaceRecoveryMigration"] as const;
const blockerContextColumn = sql<Record<string, unknown> | null>`case
when ${heartbeatRuns.contextSnapshot} is null then null else (
select coalesce(jsonb_object_agg(field.key, field.value), '{}'::jsonb)
from jsonb_each(case when jsonb_typeof(${heartbeatRuns.contextSnapshot}) = 'object'
then ${heartbeatRuns.contextSnapshot} else '{}'::jsonb end) as field(key, value)
where field.key in (${sql.join(blockerContextKeys.map(key => sql`${key}`), sql`, `)})
) end`;
const blockerProfileColumn = sql<Record<string, unknown> | null>`case
when ${heartbeatRuns.runnerProfileJson} is null then null else (
select coalesce(jsonb_object_agg(field.key, case field.key
when 'adapterDispatch' then jsonb_build_object('adapterType', field.value->'adapterType')
when 'nativeExecutionInput' then case when jsonb_typeof(field.value) = 'object' then (
select coalesce(jsonb_object_agg(input_field.key, input_field.value), '{}'::jsonb)
from jsonb_each(field.value) as input_field(key, value)
where input_field.key in ('schema', 'binding', 'provider', 'workspace')
) else field.value end
else field.value end), '{}'::jsonb)
from jsonb_each(case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object'
then ${heartbeatRuns.runnerProfileJson} else '{}'::jsonb end) as field(key, value)
where field.key in (${sql.join(blockerProfileKeys.map(key => sql`${key}`), sql`, `)})
) end`;
const blockerRunFields = {
id: heartbeatRuns.id,
companyId: heartbeatRuns.companyId,
agentId: heartbeatRuns.agentId,
status: heartbeatRuns.status,
startedAt: heartbeatRuns.startedAt,
finishedAt: heartbeatRuns.finishedAt,
error: heartbeatRuns.error,
errorCode: heartbeatRuns.errorCode,
issueId: heartbeatRuns.issueId,
nativeIssueId: heartbeatRuns.nativeIssueId,
nativeSessionId: heartbeatRuns.nativeSessionId,
runtimeMode: heartbeatRuns.runtimeMode,
runtimeModeResolvedAt: heartbeatRuns.runtimeModeResolvedAt,
driverKind: heartbeatRuns.driverKind,
processPid: heartbeatRuns.processPid,
processGroupId: heartbeatRuns.processGroupId,
processStartedAt: heartbeatRuns.processStartedAt,
sessionIdAfter: heartbeatRuns.sessionIdAfter,
controllerBootId: heartbeatRuns.controllerBootId,
controllerLeaseExpiresAt: heartbeatRuns.controllerLeaseExpiresAt,
executionStage: heartbeatRuns.executionStage,
resultJson: heartbeatRuns.resultJson,
contextSnapshot: blockerContextColumn,
runnerProfileJson: blockerProfileColumn,
};
export async function hasPendingRunCleanup(db: Db, companyId: string, runId: string): Promise<boolean> {
if (await nativeManualAdmissionHoldsExecution(db, { companyId, runId })) return true;
// Recovery saved the files and re-stopped this sandbox before queueing its
// destroy. That destroy is resource cleanup, not a live owner of the run.
const recoveredStopped = (await db.select().from(environmentLeases).where(and(
eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, runId),
sql`${environmentLeases.metadata}->'workspaceRecoveryCleanup' is not null`,
))).filter(hasRecoveredCleanupStopReceipt).map(lease => lease.id);
// A missing predecessor row owns nothing: deleting a run nulls its leases'
// run reference and removes its finalization coordinator. Failing closed on
// absence stranded the successor with no next action (invariants 3 and 7).
const [state] = await db.execute<{ pending: boolean }>(sql`SELECT EXISTS (SELECT 1 FROM ${heartbeatRuns}
WHERE ${heartbeatRuns.companyId} = ${companyId} AND ${heartbeatRuns.id} = ${runId} AND (
${heartbeatRuns.runnerProfileJson}->>'nativeWorkspaceFinalizationOwner' IS NOT NULL
OR EXISTS (SELECT 1 FROM ${environmentLeases}
WHERE ${environmentLeases.companyId} = ${companyId} AND ${environmentLeases.heartbeatRunId} = ${runId}
${recoveredStopped.length ? sql`AND ${environmentLeases.id}::text NOT IN (${sql.join(recoveredStopped.map(id => sql`${id}`), sql`, `)})` : sql``}
AND (( ${environmentLeases.releasedAt} IS NULL AND NOT COALESCE(
${environmentLeases.status} = 'retained' AND ${environmentLeases.leasePolicy} = 'reuse_by_environment'
AND ${environmentLeases.cleanupStatus} = 'success', false))
OR ${environmentLeases.status} IN ('active', 'pending_cleanup') OR ${environmentLeases.cleanupStatus} = 'failed'
OR COALESCE(${environmentLeases.metadata}->>'pendingCleanupInFlight', 'false') <> 'false'
OR (${heartbeatRuns.runtimeMode} = 'native' AND NOT COALESCE(
(${environmentLeases.status} IN ('released', 'expired', 'failed')
AND ${environmentLeases.releasedAt} IS NOT NULL AND (${environmentLeases.cleanupStatus} = 'success'
OR (${environmentLeases.cleanupStatus} IS NULL AND (
(${environmentLeases.provider} = 'local' AND ${environmentLeases.providerLeaseId} IS NULL)
OR (${environmentLeases.provider} = 'ssh' AND ${environmentLeases.metadata}->>'driver' = 'ssh'
AND ${environmentLeases.providerLeaseId} LIKE 'ssh://%')))))
OR (${environmentLeases.status} = 'retained' AND ${environmentLeases.leasePolicy} = 'reuse_by_environment'
AND ${environmentLeases.cleanupStatus} = 'success'), false))))))
OR EXISTS (SELECT 1 FROM ${nativeRunFinalizations}
WHERE ${nativeRunFinalizations.companyId} = ${companyId} AND ${nativeRunFinalizations.runId} = ${runId}
AND (${nativeRunFinalizations.leaseOwner} IS NOT NULL
OR ${nativeRunFinalizations.phase} NOT IN ('committed', 'applied', 'terminal_failure'))) AS pending`);
return state?.pending === true;
}
/** Resolved recovery bookkeeping can still carry an effective no-replay hold. */
export function executionBlockerPredicate() {
return and(
not(conversationRecoveryActionPredicate()!),
not(preDispatchExecutionWaitRecoveryActionPredicate()),
sql`not (${issueRecoveryActions.cause} = 'legacy_execution_requires_reconciliation' and exists (
select 1 from ${heartbeatRuns}
where ${heartbeatRuns.companyId} = ${issueRecoveryActions.companyId}
and ${heartbeatRuns.id} = ${canonicalExecutionUuid(sql`coalesce(${issueRecoveryActions.evidence}->>'runId', ${issueRecoveryActions.evidence}->>'sourceRunId')`)}
and ${issueRecoveryActions.sourceIssueId} = ${canonicalExecutionUuid(sql`${heartbeatRuns.contextSnapshot}->>'issueId'`)}
and (${heartbeatRuns.issueId} is null or ${heartbeatRuns.issueId} = ${issueRecoveryActions.sourceIssueId})
and ${neverStartedDispatchCancellationPredicate()}
and ${heartbeatRuns.errorCode} <> 'issue_continuation_waiting_on_review'
and coalesce(${issueRecoveryActions.evidence}->>'workspaceRestoreFailure', '') <> 'restore_unsafe_archive'
and coalesce(${issueRecoveryActions.evidence}->'workspaceRestoreRecovery'->>'schema', '') <> ${LEGACY_WORKSPACE_RECOVERY_SCHEMA}
))`,
or(unreconciledNativeExportPredicate(), and(
inArray(issueRecoveryActions.cause, [...EXECUTION_RECONCILIATION_CAUSES]),
or(inArray(issueRecoveryActions.status, ["active", "escalated"]),
sql`${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked'`),
)),
);
}
export function nativeExecutionRecoveryActionPredicate() {
return sql`exists (select 1 from ${heartbeatRuns}
where ${heartbeatRuns.companyId} = ${issueRecoveryActions.companyId}
and ${heartbeatRuns.id}::text = coalesce(${issueRecoveryActions.evidence}->>'runId', ${issueRecoveryActions.evidence}->>'sourceRunId')
and ${heartbeatRuns.runtimeMode} = 'native')`;
}
// Exact runner startup lifecycle events (event-v3 schema): they open no turn.
// Every other runner-sourced event counts as provider work, so a new, unlisted
// or turn-scoped event (for example workspace.diff.recorded or a session goal)
// keeps the hold (fail closed). Server-side records without a source event
// (spans, process identity, lifecycle) count only when they name turn or
// provider work.
export const PROVIDER_STARTUP_EVENT_TYPES = ["harness.starting", "harness.ready", "harness.diagnostic", "harness.exited",
"session.starting", "session.started", "session.resuming", "session.resumed", "session.reconciled", "session.updated",
"session.capabilities.updated", "session.closed", "session.failed", "workspace.ready", "runtime.phase.changed",
"runtime.activity.recorded", "provider.notice.recorded", "auth.credentials.updated", "run.attached", "run.detached",
"mcp_app.discovered", "mcp_app.initializing", "mcp_app.ready", "mcp_app.resource.resolved",
"mcp_app.host_context.changed", "mcp_app.teardown", "mcp_app.failed"] as const;
const SERVER_TURN_WORK_EVENT_PREFIXES = ["turn.", "item.", "tool.", "semantic_tool.", "run.result."] as const;
const SERVER_TURN_WORK_EVENT_TYPES = ["provider.event", "provider.rpc_result"] as const;
const anyPrefix = (eventType: SQL, prefixes: readonly string[]) =>
sql.join(prefixes.map(prefix => sql`starts_with(${eventType}, ${prefix})`), sql` or `);
const anyType = (eventType: SQL, types: readonly string[]) =>
sql`${eventType} in (${sql.join(types.map(type => sql`${type}`), sql`, `)})`;
/** SQL truth for an event row (alias columns) that may prove provider turn work. */
export function providerTurnWorkEventCondition(eventType: SQL, sourceEventId: SQL) {
return sql`((${sourceEventId} is not null and not ${anyType(eventType, PROVIDER_STARTUP_EVENT_TYPES)})
or ${anyType(eventType, SERVER_TURN_WORK_EVENT_TYPES)} or ${anyPrefix(eventType, SERVER_TURN_WORK_EVENT_PREFIXES)})`;
}
/**
* A Stop that the native startup owner fenced before it published a provider
* handle proves that no provider turn ran. The run's own durable record must
* agree: no turn, tool or provider event and no native result. `stopped` is
* the SQL alias of the stopped run.
*/
export function stoppedBeforeProviderTurnCondition() {
return sql`(stopped.status = 'cancelled' and stopped.runtime_mode = 'native'
and stopped.result_json->'nativeCancellation'->>'schema' = 'paperclip.native-cancellation.v1'
and stopped.result_json->'nativeCancellation'->>'runId' = stopped.id::text
and stopped.result_json->'nativeCancellation'->>'scope' = 'run'
and stopped.result_json->'nativeCancellation'->>'dispatchState' = 'acknowledged'
and stopped.result_json->'nativeCancellation'->'beforeProviderTurn' = 'true'::jsonb
and not exists (select 1 from ${heartbeatRunEvents} work
where work.company_id = stopped.company_id and work.run_id = stopped.id
and ${providerTurnWorkEventCondition(sql`work.event_type`, sql`work.source_event_id`)})
and not exists (select 1 from ${nativeRunResults} result
where result.company_id = stopped.company_id and result.run_id = stopped.id))`;
}
/**
* A file-preservation hold binds a newer task owner only until automatic
* recovery saved the stopped run's files. That owner's turn is a fresh
* assignment, not a replay: the stopped run's unknown outcomes stay with its
* own agent, which keeps the hold. A Stop fenced before any provider turn has
* no unknown outcome, so its hold releases for the same agent too. The proof is
* the durable completed-recovery receipt bound to this action, so later reuse
* of the workspace by the current owner (new epoch, writer or snapshot) cannot
* re-arm the hold.
*/
export function restoreHoldReleasedForCurrentOwnerPredicate() {
const sourceRunId = sql`coalesce(${issueRecoveryActions.evidence}->>'runId', ${issueRecoveryActions.evidence}->>'sourceRunId')`;
const operationIds = sql`case when jsonb_typeof(${issueRecoveryActions.evidence}#>'{systemRecovery,preservationOperationIds}') = 'array'
then ${issueRecoveryActions.evidence}#>'{systemRecovery,preservationOperationIds}' else '[]'::jsonb end`;
// JSON reads are NULL for other holds. Callers negate this predicate, so it
// must be strictly true or false: NOT NULL would hide unrelated blockers.
return sql`coalesce((${and(
sql`${issueRecoveryActions.evidence}->'workspaceRestoreRecovery'->>'schema' = ${LEGACY_WORKSPACE_RECOVERY_SCHEMA}`,
sql`coalesce(${issueRecoveryActions.evidence}->>'workspaceRestoreFailure', '') <> 'restore_unsafe_archive'`,
sql`exists (select 1 from ${heartbeatRuns} stopped
join ${issues} task on task.company_id = stopped.company_id and task.id = ${issueRecoveryActions.sourceIssueId}
where stopped.company_id = ${issueRecoveryActions.companyId}
and stopped.id::text = ${sourceRunId}
and stopped.status in ('failed', 'timed_out', 'interrupted', 'cancelled')
and stopped.finished_at is not null
and task.assignee_agent_id is not null
and ((task.assignee_agent_id <> stopped.agent_id
and not coalesce(task.status = 'in_review'
and task.execution_state->>'status' = 'pending'
and task.execution_state->'currentParticipant'->>'type' = 'agent'
and task.execution_state->'currentParticipant'->>'agentId' = stopped.agent_id::text, false))
or ${stoppedBeforeProviderTurnCondition()}))`,
completedWorkspaceRecoveryCondition(sourceRunId),
// Copyback put the files back into the task workspace that the new owner
// reuses. A snapshot-only save needs an explicit fresh-workspace request
// to attach it, so that hold stays until such a request or a repair.
sql`not exists (select 1 from jsonb_array_elements_text(${operationIds}) saved(id)
join ${workspaceOperations} op on op.id::text = saved.id
where op.company_id = ${issueRecoveryActions.companyId} and op.phase <> 'workspace_recovery')`,
)}), false)`;
}
export async function getExecutionBlocker(db: Db, companyId: string, issueId: string, options?: { conversationResetCommentId?: string | null }): Promise<ExecutionBlocker | null> {
const [conversation] = await db.select({ agentId: issues.conversationAgentId,
boundaryId: issues.conversationBoundaryCommentId, status: issues.status,
assigneeAgentId: issues.assigneeAgentId, statusVersion: issues.statusVersion,
executionRunId: issues.executionRunId, checkoutRunId: issues.checkoutRunId }).from(issues).where(and(
eq(issues.companyId, companyId), eq(issues.id, issueId),
)).limit(1);
// Resetting model context cannot repair or recover a workspace. This hold
// survives conversation boundaries until the existing repair/reconciliation path clears it.
const [restoreHold] = await db.select().from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, companyId),
eq(issueRecoveryActions.sourceIssueId, issueId),
executionBlockerPredicate(),
not(restoreHoldReleasedForCurrentOwnerPredicate()),
or(
unreconciledNativeExportPredicate(),
sql`${issueRecoveryActions.evidence}->>'workspaceRestoreFailure' = 'restore_unsafe_archive'`,
sql`${issueRecoveryActions.evidence}->'workspaceRestoreRecovery'->>'schema' = ${LEGACY_WORKSPACE_RECOVERY_SCHEMA}`,
),
)).orderBy(desc(issueRecoveryActions.updatedAt)).limit(1);
// A persisted user /new is an ordered context command, not a retry of uncertain work.
// The normal issue execution lock still serializes it behind any active turn.
if (!restoreHold && conversation?.agentId && options?.conversationResetCommentId) {
const [command] = await db.select().from(issueComments).where(and(
eq(issueComments.companyId, companyId), eq(issueComments.issueId, issueId),
eq(issueComments.id, options.conversationResetCommentId),
)).limit(1);
if (command?.authorUserId && !command.deletedAt && command.body.trim() === "/new") return null;
}
const [boundary] = conversation?.agentId && conversation.boundaryId
? await db.select({ createdAt: issueComments.createdAt }).from(issueComments).where(and(
eq(issueComments.companyId, companyId), eq(issueComments.issueId, issueId),
eq(issueComments.id, conversation.boundaryId),
)).limit(1) : [];
const ownership = await getConversationOwnershipBlocker(db, companyId, issueId);
if (ownership) return { ...ownership, recoveryActionId: null };
const [action] = restoreHold ? [restoreHold] : await db.select().from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, companyId),
eq(issueRecoveryActions.sourceIssueId, issueId),
executionBlockerPredicate(),
not(restoreHoldReleasedForCurrentOwnerPredicate()),
boundary ? or(gt(issueRecoveryActions.createdAt, boundary.createdAt), nativeExecutionRecoveryActionPredicate()) : undefined,
)).orderBy(desc(issueRecoveryActions.updatedAt), desc(issueRecoveryActions.id)).limit(1);
if (!action) return null;
const parsedRunId = z.string().guid().safeParse(action.evidence.runId ?? action.evidence.sourceRunId);
const runId = parsedRunId.success ? parsedRunId.data : null;
const [run] = runId ? await db.select(blockerRunFields).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, runId),
)).limit(1) : [];
const withoutWorkspace = Boolean(run?.driverKind === "grok_bot_mcp" && nativeRunHasNoWorkspaceAccess(run));
const workspaceRepairRequired = Boolean(restoreHold && !withoutWorkspace && (!run || !await hasVerifiedWorkspaceRepair(db, run)));
const workspaceRecovery = workspaceRecoveryDisplay(action.evidence, runId);
const reconstructableExport = ["native_workspace_sync_out_unrecoverable", "native_workspace_sync_out_retry_exhausted"].includes(action.cause);
const exportSource = reconstructableExport ? readNativeWorkspaceSyncReference(run?.runnerProfileJson?.nativeWorkspaceSync) : null;
const [exportLease] = exportSource ? await db.select().from(environmentLeases).where(and(
eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, run!.id), eq(environmentLeases.id, exportSource.leaseId),
eq(environmentLeases.providerLeaseId, exportSource.providerLeaseId),
)).limit(1) : [];
const canReconstructWorkspace = Boolean(exportLease && hasRemoteTerminationReceipt(exportLease)
&& (exportLease.metadata?.remoteExecutionTermination as { state?: string } | undefined)?.state === "destroyed"
&& !action.evidence.executionReconciliation && conversation && ["blocked", "in_progress"].includes(conversation.status)
&& !conversation.executionRunId && !conversation.checkoutRunId && conversation.assigneeAgentId === run?.agentId);
const reconstructionPending = Boolean((action.evidence.executionReconciliation as { workspaceReconstruction?: unknown } | undefined)?.workspaceReconstruction
&& action.evidence.continuationDelivery === "pending");
const queue = await db.select({ payload: agentWakeupRequests.payload }).from(agentWakeupRequests).where(and(
eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.status, "deferred_issue_execution"),
sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId}`,
));
const savedIds = [...new Set(queue.flatMap(entry => {
const context = entry.payload?._paperclipWakeContext as Record<string, unknown> | undefined;
return [...queuedCommentIdsFromWakePayload(entry.payload), entry.payload?.commentId, context?.wakeCommentId]
.filter((id): id is string => z.string().guid().safeParse(id).success);
}))];
const saved = savedIds.length ? await db.select({ id: issueComments.id }).from(issueComments).where(and(
eq(issueComments.companyId, companyId), eq(issueComments.issueId, issueId),
inArray(issueComments.id, savedIds), eq(issueComments.authorType, "user"), sql`${issueComments.deletedAt} is null`,
sql`not exists (select 1 from (
select consumed.context_snapshot->>'wakeCommentId' as comment_id,
consumed.context_snapshot->'wakeCommentIds' as comment_ids
from ${heartbeatRuns} consumed where consumed.company_id = ${companyId}
and consumed.context_snapshot->>'issueId' = ${issueId}
and not (consumed.status = 'cancelled' and consumed.started_at is null)
offset 0
) consumed where consumed.comment_id = ${issueComments.id}::text
or consumed.comment_ids @> jsonb_build_array(${issueComments.id}::text))`,
)) : [];
const cancellation = run ? readRunCancellation(run.resultJson) : null;
const eligibleContinuation = Boolean(!restoreHold && run && conversation?.assigneeAgentId === run.agentId &&
!["done", "cancelled"].includes(conversation.status) && canContinueCancelledRun(run));
// A failed run whose saved work is verifiably recovered no longer needs the
// hold to protect its files. Offer the same Retry that admission accepts, so
// an infrastructure failure (such as a restart) is not left to a new message.
// An intentional Stop keeps "send a new message" as its only continuation.
const recoveredFailure = Boolean(restoreHold && run?.status === "failed" && !workspaceRepairRequired &&
workspaceRecovery?.status === "recovered");
const canRetry = Boolean((!restoreHold || recoveredFailure) && run && conversation?.assigneeAgentId === run.agentId &&
!["done", "cancelled"].includes(conversation.status) && await canRetryStoppedRun(db, run));
const runError = run && canRetry && isPreDispatchReviewWait(run)
? "Waiting for review; this continuation never started."
: cancellation?.reason ?? (run?.status === "cancelled"
? "Execution was cancelled; its source was not recorded."
: run?.error);
const [chatBinding] = eligibleContinuation || canRetry ? await db.select({
state: chatConversations.state, endpointStatus: chatEndpoints.status,
connectionStatus: toolConnections.status, connectionEnabled: toolConnections.enabled,
}).from(chatConversations).leftJoin(chatEndpoints, and(
eq(chatEndpoints.companyId, companyId), eq(chatEndpoints.id, chatConversations.endpointId),
)).leftJoin(toolConnections, and(
eq(toolConnections.companyId, companyId), eq(toolConnections.id, chatEndpoints.connectionId),
)).where(and(
eq(chatConversations.companyId, companyId), eq(chatConversations.issueId, issueId),
)).limit(1) : [];
let nextAction = action.nextAction;
if (canRetry && !eligibleContinuation) nextAction += " Try again or send a new message to continue once the previous execution has stopped.";
if (chatBinding && !restoreHold) {
if (chatBinding.state === "endpoint_removed" || !chatBinding.endpointStatus || chatBinding.endpointStatus === "archived") {
nextAction = "This chat connection was removed. Inspect the stopped run and create a new task to continue the work.";
} else if (chatBinding.state === "unavailable" || !["active", "verifying"].includes(chatBinding.endpointStatus) ||
chatBinding.connectionStatus !== "active" || !chatBinding.connectionEnabled) {
nextAction = "This chat connection is unavailable. Restore access in Apps or create a new task to continue the work.";
} else {
nextAction = "Send a new chat message to continue this conversation.";
}
} else if (!restoreHold && run?.status === "cancelled" && !eligibleContinuation && !canRetry && action.cause === "legacy_execution_requires_reconciliation") {
nextAction += " Inspect the run before sending a new message to request continuation.";
}
if (withoutWorkspace && (restoreHold || run?.status === "cancelled") && !run?.resultJson?.externalWorkspaceInspection) {
nextAction = run?.driverKind === "grok_bot_mcp"
? "Inspect this Bot in Grok Bot, then clear the recovery hold and verify the connection in the agent's Runtime settings before assigning more work."
: "Inspect the stopped work in the external agent and its connection before requesting continuation.";
}
if (run?.resultJson?.externalWorkspaceInspection && !restoreHold) {
nextAction = "Verify the Grok connection, then send a new message to continue. The stopped work will not be replayed.";
}
return {
recoveryActionId: action.id,
runId,
// A stopped reviewer can differ from the task owner who receives the work back.
agentId: run?.agentId ?? null,
cause: action.cause,
nextAction: (workspaceRecovery?.status === "recovered" && canRetry && !chatBinding
? "Saved work recovered. Retry or send a message to continue." : workspaceRecovery?.message) ?? (reconstructionPending ? "Continuation from saved sources is queued and waiting for execution checks." : nextAction),
...(workspaceRecovery ? { workspaceRecovery } : {}),
runStatus: run?.status ?? null,
runError: runError?.slice(0, 1024) ?? null,
canContinue: eligibleContinuation && !chatBinding,
canRetry: canRetry && !chatBinding,
...(restoreHold ? { workspaceRepairRequired, canRecordWorkspaceRepair: workspaceRepairRequired && hasRequiredWorkspaceRecovery(action.evidence) } : {}),
savedMessageCount: saved.length,
...(canReconstructWorkspace ? { canReconstructWorkspace, workspaceRecoveryStatusVersion: conversation!.statusVersion } : {}),
...(reconstructionPending ? { workspaceReconstructionPending: true } : {}),
};
}