mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Tasks from a connected chat (for example GitHub pull request reviews) show Retry when a run fails > - A person's Retry on such a task first asks the chat retry owner to repeat the exact failed chat request. #21038 made a refusal at the click continue the task from saved work instead > - The chat owner can also defer its source check (a busy identity, endpoint, or agent row lock) and refuse it a few seconds later. Then the route has already answered 202, and the task Retry receipt was marked failed with no run > - The task stayed blocked, and its execution blocker told the person to send a message, while the interruption projection still offered Retry > - This pull request makes that late refusal take the same board continuation as a refusal at the click, and makes the blocker offer Retry for recovered saved work > - The benefit is that an accepted Retry always starts work, and the two projections agree ## Linked Issues or Issue Description No public issue exists. This is the bug description, in the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** On a hosted deploy, about 20 blocked tasks of one busy GitHub review agent (`paperclip_runner`, native remote runs) did not restart after Retry. `POST /api/issues/:id/retry` returned 202 with `accepted: true`, a `chatActionId`, and `presentation.state: retry_scheduled`. No run was created. The task Retry receipt changed to `failed` 5 to 10 seconds later, and the task stayed blocked. The activity record of each click (`chat.failed_run_retry_requested`) has `sourceAuthorizationPending: true`, so the chat owner deferred its source check at the click. No chat wake row was written, so the chat replay ended before admission. The last run failed with `native_continuation_requires_reconciliation` / `runner_remote_process_identity_unavailable`. Its saved work was recovered (`workspaceRecovery.status: recovered`, remote lease destroyed with a termination receipt). The projections disagreed: `interruption` had `canRetry: true`, but `executionBlocker` had `canRetry: false` and `nextAction: "Saved work recovered. Send a message to continue."`. Root cause: when the chat retry action settles `failed`, `releaseUnadmittedTaskRetryInput` in `server/src/services/chat-channels/task-retry-input.ts` marked the linked task Retry receipt `failed`. #21038 added a board-continuation fallback only for a refusal inside the click transaction. A refusal after a deferred source check (for example 8 lock-busy attempts on the agent row, about 7 seconds) had no fallback. Separately, `getExecutionBlocker` hid Retry for every chat-bound task (`canRetry && !chatBinding`), and `workspaceRecoveryDisplay` always said "Send a message to continue" for recovered work. **Expected behavior** A Retry that is accepted starts work. If the exact chat replay is refused after the click, the person's Retry continues the task from its recovered saved work. The execution blocker and the interruption both offer Retry, and neither tells the person to send a message to unblock a failed run. **Steps to reproduce** 1. Bind a task to a GitHub chat conversation. Make its latest native run fail with `native_continuation_requires_reconciliation` and recover its saved work. 2. Hold the chat identity advisory lock while you click Retry, so the source check is deferred (`sourceAuthorizationPending: true`). 3. Hold a row lock on the agent while the chat retry processes, so the deferred check fails 8 times. 4. The route returned 202. The chat action settles `failed`, the task Retry receipt changes to `failed`, and no run starts. **Paperclip version or commit** `master` ata8580160(the deployed revision). Reproduced onf30005440b. **Deployment mode** Hosted (cloud) instance with Daytona remote sandboxes and a GitHub chat connection. **Agent adapter(s) involved** `paperclip_runner` (native remote runs). The fix is in core Retry code and is not adapter-specific. ## What Changed - `server/src/services/task-retry.ts`: new `settleRefusedTaskChatRetry`. When a chat retry action settles `failed`, the linked task Retry receipt becomes an ordinary board continuation if all of these are true: it is a person's Retry (or an automatic Retry that the same person's click has joined), its source run is still the latest run and its saved work is recovered (system recovery `complete`, workspace recovery `recovered`), the clicking person still has task access, the chat wake has no run, and no saved or unadmitted chat input remains. Any other refusal still settles the receipt as failed, so the existing source-edited, revoked, and identity-expiry outcomes do not change. A paused agent does not fail the receipt: it stays deferred and starts when dispatch allows. The receipt drops `chatActionId`, stays `deferred_issue_execution`, notifies the deferred-receipt worker, and records `issue.retry_continued_from_saved_work`. Otherwise the receipt is failed as before. `reconcileTaskChatRetryReceipts` uses the same check, so a crash before settlement is also covered. Its failed write matches the chat binding it read, so a stale pass cannot fail a receipt that settlement already converted. - The task Retry intent now records `initiatedBy` for an automatic Retry, so an automatic Retry keeps the exact-replay requirement after the click too. - `server/src/services/chat-channels/task-retry-input.ts`: `releaseUnadmittedTaskRetryInput` calls the task owner's settlement instead of failing the receipt directly. - `server/src/services/execution-blocker.ts` and `workspace-recovery-receipt.ts`: for a failed or timed-out run whose saved work is recovered, the blocker offers Retry also on a chat-bound task. The text follows actual Retry eligibility: "Saved work recovered. Retry continues from it." when Retry is offered, "Saved work recovered." when it is not (for example cleanup quarantine), and "Send a message to continue" only for a stopped run. Only the task blocker decides Retry. The run-level projection (`execution-projection.ts`) has no task context, so it says "Saved work recovered." and never claims Retry. `execution-projection.ts` uses the same text. - `ui/src/lib/activity-format.ts`: labels for the new activity. - `doc/execution-semantics.md`: the late-refusal rule. - Tests: a new real-service case in `server/src/__tests__/chat-channels.integration.test.ts` that reproduces the hosted state (remote saved workspace, destroyed lease receipt, recovered recovery action, deferred source check, refused replay), for a person's Retry and for an automatic Retry. A gate case per continuation condition (unrecovered workspace and revoked access settle failed; a paused agent stays deferred and runs after resume), a stale-reconcile race case, and a joined-click case. Two existing text expectations are updated. ## Verification - Red before the fix: with only the source changes removed, the new test fails on the projection (`canRetry: false`, "Send a message to continue."). With only the Retry settlement removed, it fails with the receipt `status: "failed"` and no run. - Green after the fix: `npx vitest run src/__tests__/chat-channels.integration.test.ts -t "continues an accepted GitHub task Retry from recovered saved work"` passes 2/2. The person's Retry starts one queued run that resumes the preserved workspace, and the automatic Retry still settles as failed. - Focused suites pass: `task-retry`, `execution-projection`, `execution-projection-recovery`, `workspace-recovery-retry-copyback`, `chat-failed-run-retry-boundary`, `workspace-recovery-admission`, `remote-workspace-recovery-receipt-fallback` (372/372), and other recovery, blocker and continuation suites. - `npx tsc --noEmit -p server/tsconfig.json` passes. - Staging was read with GET requests only. No POST was sent. ## Risks - Behavior change: a person's Retry on a connected task that the chat owner refuses after the click now starts a board continuation. It is the same continuation #21038 already uses for a refusal at the click, with the clicking person's authority, and dispatch keeps every normal gate. - Receipts that already failed on staging stay failed. The next click on those tasks takes the fixed path. No migration. - The blocker now reports `canRetry: true` for chat-bound tasks only when the failed run's saved work is verifiably recovered. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
549 lines
26 KiB
TypeScript
549 lines
26 KiB
TypeScript
import { and, desc, eq, inArray, isNull, ne, sql } from "drizzle-orm";
|
|
import {
|
|
agents,
|
|
heartbeatRuns,
|
|
issues,
|
|
dotAgentBindings,
|
|
dotRunnerAssignments,
|
|
museAgentBindings,
|
|
museRunnerAssignments,
|
|
issueRecoveryActions,
|
|
issueThreadInteractions,
|
|
nativeRunFinalizations,
|
|
type Db,
|
|
} from "@paperclipai/db";
|
|
import type { ExecutionProjection } from "@paperclipai/shared";
|
|
import { EXECUTION_CONTROL_DEADLINE_MS } from "./execution-control-deadline.js";
|
|
import { executionFailureRetryCount } from "./execution-recovery-attempt.js";
|
|
import { retryIdempotentDatabaseOperation } from "../database-retry.js";
|
|
import { workspaceRecoveryDisplay } from "./workspace-recovery-receipt.js";
|
|
import { readTaskDrain } from "./task-admission.js";
|
|
import { getStartupRecoveryState, startupQueuedRunPassOutstanding } from "../startup-recovery-state.js";
|
|
import { hasPendingRunCleanup } from "./execution-blocker.js";
|
|
import { isAgentAwaitingSetup } from "../modules/agent-lifecycle/index.js";
|
|
import { NATIVE_RESPONSE_WAKE_WAITING_REASON } from "./native-runtime/status-arbiter.js";
|
|
const MAX_PREDECESSOR_CLEANUP_CHECKS = 16;
|
|
const text = (v: unknown) => (typeof v === "string" ? v : null);
|
|
const executionRunColumns = {
|
|
id: heartbeatRuns.id,
|
|
agentId: heartbeatRuns.agentId,
|
|
driverKind: heartbeatRuns.driverKind,
|
|
errorCode: heartbeatRuns.errorCode,
|
|
executionControlDeadlineAt: heartbeatRuns.executionControlDeadlineAt,
|
|
finishedAt: heartbeatRuns.finishedAt,
|
|
lastOutputAt: heartbeatRuns.lastOutputAt,
|
|
lastUsefulActionAt: heartbeatRuns.lastUsefulActionAt,
|
|
nativeIssueId: heartbeatRuns.nativeIssueId,
|
|
nextAction: heartbeatRuns.nextAction,
|
|
processPid: heartbeatRuns.processPid,
|
|
retryOfRunId: heartbeatRuns.retryOfRunId,
|
|
runtimeMode: heartbeatRuns.runtimeMode,
|
|
scheduledRetryAt: heartbeatRuns.scheduledRetryAt,
|
|
scheduledRetryAttempt: heartbeatRuns.scheduledRetryAttempt,
|
|
scheduledRetryReason: heartbeatRuns.scheduledRetryReason,
|
|
startedAt: heartbeatRuns.startedAt,
|
|
status: heartbeatRuns.status,
|
|
contextSnapshot: sql<Record<string, unknown>>`jsonb_build_object(
|
|
'issueId', ${heartbeatRuns.contextSnapshot}->'issueId',
|
|
'wakeReason', ${heartbeatRuns.contextSnapshot}->'wakeReason',
|
|
'failureRetriesBeforeAiConnectionWait', ${heartbeatRuns.contextSnapshot}->'failureRetriesBeforeAiConnectionWait',
|
|
'failureRetriesBeforeWorkspaceWait', ${heartbeatRuns.contextSnapshot}->'failureRetriesBeforeWorkspaceWait',
|
|
'executionRetryAccounting', ${heartbeatRuns.contextSnapshot}->'executionRetryAccounting')`,
|
|
resultJson: sql<Record<string, unknown>>`jsonb_build_object(
|
|
'finalizationPhase', ${heartbeatRuns.resultJson}->'finalizationPhase',
|
|
'finalizationReasonCode', ${heartbeatRuns.resultJson}->'finalizationReasonCode')`,
|
|
};
|
|
type Run = Pick<typeof heartbeatRuns.$inferSelect, keyof typeof executionRunColumns>;
|
|
type Coordinator = typeof nativeRunFinalizations.$inferSelect;
|
|
type WaitingExternalProvider = "muse" | "openai_dot";
|
|
type Recovery = Pick<
|
|
typeof issueRecoveryActions.$inferSelect,
|
|
"cause" | "nextAction"
|
|
> &
|
|
Partial<
|
|
Pick<typeof issueRecoveryActions.$inferSelect, "status" | "evidence">
|
|
>;
|
|
|
|
/** Batched reads; list consumers do not perform per-task polling. */
|
|
export async function executionProjectionsForRuns(
|
|
db: Db,
|
|
companyId: string,
|
|
runIds: string[],
|
|
now = new Date(),
|
|
options: { retryDatabaseReads?: boolean } = {},
|
|
) {
|
|
const projections = new Map<string, ExecutionProjection>();
|
|
if (!runIds.length) return projections;
|
|
// Opt in only with a pooled Db outside a transaction. Issue enrichment also
|
|
// calls this helper with a transaction, whose failed reads must not replay.
|
|
const read = <T>(query: () => Promise<T>) => options.retryDatabaseReads
|
|
? retryIdempotentDatabaseOperation(query)
|
|
: query();
|
|
const runs = await read(async () => db
|
|
.select(executionRunColumns)
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.companyId, companyId),
|
|
inArray(heartbeatRuns.id, runIds),
|
|
),
|
|
));
|
|
const coordinators = await read(async () => db
|
|
.select()
|
|
.from(nativeRunFinalizations)
|
|
.where(
|
|
and(
|
|
eq(nativeRunFinalizations.companyId, companyId),
|
|
inArray(nativeRunFinalizations.runId, runIds),
|
|
),
|
|
));
|
|
const issueIds = [
|
|
...new Set(
|
|
runs
|
|
.map((run) => run.nativeIssueId ?? text(run.contextSnapshot?.issueId))
|
|
.filter((id): id is string => !!id),
|
|
),
|
|
];
|
|
const pending = issueIds.length
|
|
? await read(async () => db
|
|
.select({
|
|
issueId: issueThreadInteractions.issueId,
|
|
kind: issueThreadInteractions.kind,
|
|
})
|
|
.from(issueThreadInteractions)
|
|
.where(
|
|
and(
|
|
eq(issueThreadInteractions.companyId, companyId),
|
|
inArray(issueThreadInteractions.issueId, issueIds),
|
|
eq(issueThreadInteractions.status, "pending"),
|
|
),
|
|
))
|
|
: [];
|
|
const recovery = issueIds.length
|
|
? await read(async () => db
|
|
.select({
|
|
issueId: issueRecoveryActions.sourceIssueId,
|
|
cause: issueRecoveryActions.cause,
|
|
nextAction: issueRecoveryActions.nextAction,
|
|
evidence: issueRecoveryActions.evidence,
|
|
status: issueRecoveryActions.status,
|
|
})
|
|
.from(issueRecoveryActions)
|
|
.where(
|
|
and(
|
|
eq(issueRecoveryActions.companyId, companyId),
|
|
inArray(issueRecoveryActions.sourceIssueId, issueIds),
|
|
inArray(issueRecoveryActions.status, [
|
|
"active",
|
|
"escalated",
|
|
"resolved",
|
|
]),
|
|
),
|
|
)
|
|
.orderBy(desc(issueRecoveryActions.updatedAt)))
|
|
: [];
|
|
// A live controller is not evidence that an external agent accepted work.
|
|
// Read each provider journal once for the batch, never once per task.
|
|
const waitingByRun = new Map<string, WaitingExternalProvider>();
|
|
const externalRunIds = (driverKind: string) => runs
|
|
.filter(run => run.status === "running" && run.runtimeMode === "native" && run.driverKind === driverKind)
|
|
.map(run => run.id);
|
|
const museRunIds = externalRunIds("muse_external");
|
|
if (museRunIds.length) {
|
|
const offered = await read(async () => db
|
|
.select({ runId: museRunnerAssignments.runId })
|
|
.from(museRunnerAssignments)
|
|
.innerJoin(museAgentBindings, and(
|
|
eq(museAgentBindings.id, museRunnerAssignments.bindingId),
|
|
eq(museAgentBindings.companyId, museRunnerAssignments.companyId),
|
|
eq(museAgentBindings.agentId, museRunnerAssignments.agentId),
|
|
eq(museAgentBindings.generation, museRunnerAssignments.bindingGeneration),
|
|
isNull(museAgentBindings.revokedAt),
|
|
))
|
|
.where(and(
|
|
eq(museRunnerAssignments.companyId, companyId),
|
|
inArray(museRunnerAssignments.runId, museRunIds),
|
|
inArray(museRunnerAssignments.status, ["offered", "claimed"]),
|
|
isNull(museRunnerAssignments.nativeAcceptedAt),
|
|
)));
|
|
for (const row of offered) waitingByRun.set(row.runId, "muse");
|
|
}
|
|
const dotRunIds = externalRunIds("openai_dot_mcp");
|
|
if (dotRunIds.length) {
|
|
const offered = await read(async () => db
|
|
.select({ runId: dotRunnerAssignments.runId })
|
|
.from(dotRunnerAssignments)
|
|
.innerJoin(dotAgentBindings, and(
|
|
eq(dotAgentBindings.id, dotRunnerAssignments.bindingId),
|
|
eq(dotAgentBindings.companyId, dotRunnerAssignments.companyId),
|
|
eq(dotAgentBindings.agentId, dotRunnerAssignments.agentId),
|
|
eq(dotAgentBindings.generation, dotRunnerAssignments.bindingGeneration),
|
|
isNull(dotAgentBindings.revokedAt),
|
|
))
|
|
.where(and(
|
|
eq(dotRunnerAssignments.companyId, companyId),
|
|
inArray(dotRunnerAssignments.runId, dotRunIds),
|
|
eq(dotRunnerAssignments.status, "offered"),
|
|
isNull(dotRunnerAssignments.acceptedAt),
|
|
)));
|
|
for (const row of offered) waitingByRun.set(row.runId, "openai_dot");
|
|
}
|
|
// A queued turn cannot start while its predecessor's cleanup is pending
|
|
// (for example a stopped run whose runner has not released yet). Name that
|
|
// wait instead of an unexplained "Queued". The predecessors are the ones
|
|
// admission checks: the task's execution owner, the run this one retries,
|
|
// and the agent's latest settled run on the same task, whose in-process
|
|
// executor the claim also waits for. Pending uses admission's own predicate.
|
|
const settlingPredecessorKeys = new Set<string>();
|
|
const queuedRuns = runs.filter(run => run.status === "queued");
|
|
const setupAgentIds = [...new Set(queuedRuns.filter(run => !run.startedAt).map(run => run.agentId))];
|
|
const setupAgents = setupAgentIds.length ? await read(async () => db
|
|
.select({ id: agents.id, lifecycleState: agents.lifecycleState,
|
|
lifecycleHolds: agents.lifecycleHolds,
|
|
retryRequired: sql<boolean>`coalesce(${agents.lifecycleOperation}->>'retryRequired', 'false') = 'true'` })
|
|
.from(agents).where(and(eq(agents.companyId, companyId), inArray(agents.id, setupAgentIds)))) : [];
|
|
const setupByAgent = new Map(setupAgents.map(agent => [agent.id, agent]));
|
|
const issueIdOf = (run: Run) => run.nativeIssueId ?? text(run.contextSnapshot?.issueId);
|
|
const queuedIssueIds = [...new Set(queuedRuns.map(issueIdOf).filter((id): id is string => !!id))];
|
|
if (queuedIssueIds.length) {
|
|
const issueKey = sql<string>`coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.contextSnapshot}->>'issueId')`;
|
|
// Two set-based reads for the whole batch, then one bounded, parallel
|
|
// admission check per distinct predecessor.
|
|
const [owners, latestSettled] = await Promise.all([
|
|
read(async () => db
|
|
.select({ id: issues.id, executionRunId: issues.executionRunId })
|
|
.from(issues)
|
|
.where(and(eq(issues.companyId, companyId), inArray(issues.id, queuedIssueIds)))),
|
|
read(async () => db
|
|
.selectDistinctOn([heartbeatRuns.agentId, issueKey], { id: heartbeatRuns.id, agentId: heartbeatRuns.agentId, issueId: issueKey })
|
|
.from(heartbeatRuns)
|
|
.where(and(
|
|
eq(heartbeatRuns.companyId, companyId),
|
|
inArray(heartbeatRuns.agentId, [...new Set(queuedRuns.map(run => run.agentId))]),
|
|
inArray(issueKey, queuedIssueIds),
|
|
inArray(heartbeatRuns.status, ["succeeded", "failed", "cancelled", "timed_out"]),
|
|
))
|
|
.orderBy(heartbeatRuns.agentId, issueKey, desc(heartbeatRuns.createdAt))),
|
|
]);
|
|
const ownerByIssue = new Map(owners.map(row => [row.id, row.executionRunId]));
|
|
const latestByAgentIssue = new Map(latestSettled.map(row => [`${row.agentId}:${row.issueId}`, row.id]));
|
|
const candidatesByRun = new Map<string, string[]>();
|
|
for (const run of queuedRuns) {
|
|
const issueId = issueIdOf(run);
|
|
if (!issueId) continue;
|
|
candidatesByRun.set(run.id, [...new Set([ownerByIssue.get(issueId), run.retryOfRunId,
|
|
latestByAgentIssue.get(`${run.agentId}:${issueId}`)].filter((id): id is string => !!id && id !== run.id))]);
|
|
}
|
|
const distinct = [...new Set([...candidatesByRun.values()].flat())];
|
|
// A label is not worth an unbounded fan-out on large list reads.
|
|
if (distinct.length <= MAX_PREDECESSOR_CLEANUP_CHECKS) {
|
|
const pendingIds = new Set((await Promise.all(distinct.map(async id =>
|
|
(await read(() => hasPendingRunCleanup(db, companyId, id))) ? id : null))).filter((id): id is string => !!id));
|
|
for (const [runId, candidates] of candidatesByRun)
|
|
if (candidates.some(id => pendingIds.has(id))) settlingPredecessorKeys.add(runId);
|
|
}
|
|
}
|
|
const coordinatorByRun = new Map(coordinators.map((row) => [row.runId, row]));
|
|
const responseWaitRuns = runs.filter(run => run.status === "succeeded"
|
|
&& run.resultJson?.finalizationReasonCode === NATIVE_RESPONSE_WAKE_WAITING_REASON && issueIdOf(run));
|
|
const responseWaitIssues = responseWaitRuns.length ? await read(async () => db
|
|
.select({ id: issues.id, status: issues.status, lastStatusDecisionId: issues.lastStatusDecisionId })
|
|
.from(issues)
|
|
.where(and(eq(issues.companyId, companyId), inArray(issues.id, [...new Set(responseWaitRuns.map(run => issueIdOf(run)!))])))) : [];
|
|
const responseWaitIssueById = new Map(responseWaitIssues.map(row => [row.id, row]));
|
|
const currentResponseWait = (run: Run) => {
|
|
const issue = responseWaitIssueById.get(issueIdOf(run) ?? "");
|
|
const decisionId = coordinatorByRun.get(run.id)?.decisionId;
|
|
return Boolean(issue && decisionId && issue.status === "in_progress" && issue.lastStatusDecisionId === decisionId);
|
|
};
|
|
for (const run of runs) {
|
|
const issueId = run.nativeIssueId ?? text(run.contextSnapshot?.issueId);
|
|
const matching = recovery.filter(
|
|
(row) =>
|
|
row.issueId === issueId &&
|
|
(row.status !== "resolved" || row.evidence.executionReconciliation || row.evidence.automaticRecovery) &&
|
|
(row.evidence.runId === run.id || row.evidence.sourceRunId === run.id),
|
|
);
|
|
const action =
|
|
matching.find((row) => row.status !== "resolved") ?? matching[0];
|
|
projections.set(
|
|
run.id,
|
|
projectExecution(
|
|
run,
|
|
coordinatorByRun.get(run.id),
|
|
pending.filter((row) => row.issueId === issueId),
|
|
action,
|
|
now,
|
|
waitingByRun.get(run.id),
|
|
settlingPredecessorKeys.has(run.id),
|
|
setupByAgent.get(run.agentId),
|
|
currentResponseWait(run),
|
|
),
|
|
);
|
|
}
|
|
return projections;
|
|
}
|
|
export async function executionProjectionForRun(
|
|
db: Db,
|
|
companyId: string,
|
|
runId: string,
|
|
now = new Date(),
|
|
): Promise<ExecutionProjection | null> {
|
|
return (
|
|
(await executionProjectionsForRuns(db, companyId, [runId], now)).get(
|
|
runId,
|
|
) ?? null
|
|
);
|
|
}
|
|
export function projectExecution(
|
|
run: Run,
|
|
coordinator: Coordinator | undefined,
|
|
pending: Array<{ kind: string }>,
|
|
recoveryAction: Recovery | undefined,
|
|
now = new Date(),
|
|
waitingForProvider?: WaitingExternalProvider,
|
|
waitingForPredecessor = false,
|
|
setupAgent?: { lifecycleState: string; lifecycleHolds: readonly string[]; retryRequired?: boolean },
|
|
responseWaitCurrent = false,
|
|
): ExecutionProjection {
|
|
const detail = coordinator?.failureDetail ?? {};
|
|
const successorRunId = text(detail.successorRunId);
|
|
const projection: ExecutionProjection = {
|
|
// A live run is always stoppable by the board (POST /heartbeat-runs/:id/cancel);
|
|
// a failed native run is stoppable only while its retry is still pending.
|
|
canStop: run.status === "running" || run.status === "queued" || (
|
|
run.runtimeMode === "native" && run.status === "failed" && run.driverKind !== "grok_bot_mcp" && !!run.nativeIssueId
|
|
&& coordinator?.runId === run.id && coordinator.issueId === run.nativeIssueId && coordinator.resultId === null
|
|
&& coordinator.phase === "retryable_failure"),
|
|
phase: "queued",
|
|
label: "Queued",
|
|
cause: run.errorCode ?? coordinator?.failureCode ?? null,
|
|
lastConfirmedActivityAt:
|
|
(
|
|
run.lastUsefulActionAt ??
|
|
run.lastOutputAt ??
|
|
run.startedAt
|
|
)?.toISOString() ?? null,
|
|
retryAt:
|
|
(coordinator?.nextAttemptAt ?? run.scheduledRetryAt)?.toISOString() ??
|
|
null,
|
|
attempt: coordinator?.attempt ?? executionFailureRetryCount(run) + 1,
|
|
maxAttempts: 3,
|
|
recoveryOwner: null,
|
|
nextAction: text(detail.nextAction) ?? run.nextAction,
|
|
permittedActions: ["inspect_run"],
|
|
predecessorRunId: run.retryOfRunId,
|
|
successorRunId,
|
|
};
|
|
const set = (phase: ExecutionProjection["phase"], label: string) =>
|
|
Object.assign(projection, { phase, label });
|
|
if (recoveryAction?.status === "resolved" && recoveryAction.evidence?.automaticRecovery) {
|
|
projection.cause = recoveryAction.cause;
|
|
projection.nextAction = workspaceRecoveryDisplay(recoveryAction.evidence, run.id, { status: run.status, retryOffered: false })?.message
|
|
?? recoveryAction.nextAction;
|
|
const continuation = recoveryAction.evidence.explicitUserContinuation as
|
|
{ previousRunId?: unknown; runId?: unknown } | undefined;
|
|
const explicitSuccessor = text(continuation?.runId);
|
|
if (continuation?.previousRunId === run.id && explicitSuccessor && explicitSuccessor !== run.id) {
|
|
// The admission transaction already recorded the user's successor. Keep
|
|
// the old failure diagnostic without making it hold the newer attempt.
|
|
projection.successorRunId = explicitSuccessor;
|
|
projection.nextAction = null;
|
|
return set("completed", "Continued in another run");
|
|
}
|
|
// Diagnostic projection only: no user decision or replay affordance.
|
|
return set("recovery_needed", "Stopped");
|
|
}
|
|
if (
|
|
recoveryAction?.status === "resolved" &&
|
|
recoveryAction.evidence?.executionReconciliation
|
|
) {
|
|
if (recoveryAction.evidence.continuationDelivery === "invalidated") {
|
|
projection.cause = "continuation_ownership_changed";
|
|
projection.nextAction =
|
|
"The task was closed or reassigned before delivery. Inspect the current task owner and status.";
|
|
return set("failed", "Continuation cancelled");
|
|
}
|
|
const reconciliation = recoveryAction.evidence.executionReconciliation;
|
|
if (recoveryAction.cause === "native_restart_recovery_blocked"
|
|
&& recoveryAction.evidence.reason === "controller_process_space_unverified"
|
|
&& recoveryAction.evidence.continuationDelivery === "not_requested"
|
|
&& reconciliation && typeof reconciliation === "object" && !Array.isArray(reconciliation)
|
|
&& "runId" in reconciliation && reconciliation.runId === run.id
|
|
&& "controllerStopped" in reconciliation && reconciliation.controllerStopped === true
|
|
&& "providerStopped" in reconciliation && reconciliation.providerStopped === true
|
|
&& "controllerAuthorityDigest" in reconciliation && typeof reconciliation.controllerAuthorityDigest === "string" && /^[0-9a-f]{64}$/.test(reconciliation.controllerAuthorityDigest)) {
|
|
const continuation = recoveryAction.evidence.explicitUserContinuation as
|
|
{ previousRunId?: unknown; runId?: unknown } | undefined;
|
|
const explicitSuccessor = text(continuation?.runId);
|
|
if (continuation?.previousRunId === run.id && explicitSuccessor && explicitSuccessor !== run.id) {
|
|
projection.successorRunId = explicitSuccessor;
|
|
projection.nextAction = null;
|
|
return set("completed", "Continued in another run");
|
|
}
|
|
projection.cause = null;
|
|
projection.nextAction = "The original execution was contained and its action outcomes reviewed. Use Retry to request another attempt.";
|
|
return set("failed", "Execution contained");
|
|
}
|
|
const deliveredRunId = text(recoveryAction.evidence.continuationRunId);
|
|
projection.successorRunId = deliveredRunId;
|
|
projection.cause = null;
|
|
projection.nextAction = deliveredRunId
|
|
? null
|
|
: "The previous execution is reconciled. Its continuation is queued for delivery.";
|
|
return set(
|
|
deliveredRunId ? "completed" : "queued",
|
|
deliveredRunId ? "Continued in another run" : "Continuation queued",
|
|
);
|
|
}
|
|
if (
|
|
(run.status === "running" &&
|
|
(run.executionControlDeadlineAt || coordinator?.controlDeadlineAt)) ||
|
|
[
|
|
"result_persisted",
|
|
"ready_for_assessment",
|
|
"arbitrating",
|
|
"workspace_pending",
|
|
].includes(coordinator?.phase ?? "")
|
|
)
|
|
return set("finishing", "Finishing");
|
|
if (successorRunId) return set("completed", "Continued in another run");
|
|
if (run.status === "scheduled_retry" && run.scheduledRetryReason === "ai_connection_busy") {
|
|
projection.nextAction = "Waiting for the AI subscription's current execution to finish; the scheduled check will revalidate access.";
|
|
return set("retry_scheduled", "Waiting for AI subscription");
|
|
}
|
|
if (run.status === "scheduled_retry" && run.scheduledRetryReason === "ai_connection_pool_wait"
|
|
&& run.contextSnapshot?.wakeReason === "provider_limit_retry") {
|
|
const resetAt = run.scheduledRetryAt ? new Date(run.scheduledRetryAt).toISOString() : "its reported reset";
|
|
projection.nextAction = `Waiting for the agent's provider usage limit to reset; the work starts at ${resetAt}. To continue sooner, restore the provider allowance or change the agent's AI connection, then start a manual run.`;
|
|
return set("retry_scheduled", "Waiting for usage limit reset");
|
|
}
|
|
if (run.status === "scheduled_retry" && run.scheduledRetryReason === "agent_directory_busy") {
|
|
projection.nextAction = "Waiting for another run of this agent to finish with its files; this task continues automatically.";
|
|
return set("retry_scheduled", "Waiting for agent files");
|
|
}
|
|
if (run.status === "scheduled_retry" && run.scheduledRetryReason === "workspace_busy") {
|
|
projection.nextAction = "Waiting for the live workspace holder to finish; the scheduled check will revalidate ownership.";
|
|
return set("retry_scheduled", "Waiting for workspace");
|
|
}
|
|
// Cleanup can fail before a finalization coordinator exists. The missing
|
|
// row must not turn a quarantined native session into an ordinary Retry.
|
|
const cleanupQuarantined = run.runtimeMode === "native" &&
|
|
["failed", "timed_out"].includes(run.status) &&
|
|
run.errorCode === "native_session_cleanup_quarantined";
|
|
if (!cleanupQuarantined && (
|
|
coordinator?.phase === "retryable_failure" ||
|
|
run.status === "scheduled_retry"
|
|
)) {
|
|
projection.recoveryOwner = "agent";
|
|
if (coordinator?.failureCode === "native_governed_wait_settlement_pending") {
|
|
projection.nextAction = "The work and waiting state are saved. Paperclip is confirming shutdown before continuing.";
|
|
return set("finishing", "Finishing shutdown");
|
|
}
|
|
return set(
|
|
projection.retryAt && new Date(projection.retryAt) > now
|
|
? "retry_scheduled"
|
|
: "reconnecting",
|
|
projection.retryAt && new Date(projection.retryAt) > now
|
|
? "Retry scheduled"
|
|
: "Reconnecting",
|
|
);
|
|
}
|
|
if (coordinator?.phase === "terminal_failure" || recoveryAction || cleanupQuarantined) {
|
|
if (
|
|
!cleanupQuarantined &&
|
|
coordinator?.failureCode === "native_provider_terminal_failed" &&
|
|
!detail.replacementDenied &&
|
|
run.finishedAt &&
|
|
now.getTime() - run.finishedAt.getTime() < EXECUTION_CONTROL_DEADLINE_MS
|
|
) {
|
|
projection.recoveryOwner = "agent";
|
|
projection.nextAction =
|
|
"Checking that the previous provider stopped and its action outcomes are known before continuing.";
|
|
return set("reconnecting", "Checking recovery");
|
|
}
|
|
projection.recoveryOwner = "board";
|
|
projection.cause =
|
|
recoveryAction?.cause ??
|
|
text(detail.replacementDenied) ??
|
|
projection.cause;
|
|
projection.nextAction = recoveryAction?.nextAction ?? projection.nextAction;
|
|
if (cleanupQuarantined && !projection.nextAction) {
|
|
projection.nextAction = "Verify the stopped session and its saved work before starting a new attempt.";
|
|
}
|
|
projection.permittedActions.push("inspect_recovery");
|
|
return set("recovery_needed", "Recovery needed");
|
|
}
|
|
if (run.status === "succeeded") {
|
|
if (pending.length)
|
|
return set(
|
|
pending.some((row) => row.kind === "connection_intent")
|
|
? "waiting_for_access"
|
|
: "waiting_for_answer",
|
|
pending.some((row) => row.kind === "connection_intent")
|
|
? "Waiting for access"
|
|
: "Waiting for answer",
|
|
);
|
|
if (responseWaitCurrent && run.resultJson?.finalizationPhase === "committed" &&
|
|
run.resultJson.finalizationReasonCode === NATIVE_RESPONSE_WAKE_WAITING_REASON) {
|
|
projection.nextAction = "The agent finished its turn and is waiting for a reply. Comment on the task or change its status to continue.";
|
|
return set("waiting_for_answer", "Waiting for a reply");
|
|
}
|
|
return set("completed", "Completed");
|
|
}
|
|
if (["failed", "cancelled", "timed_out", "interrupted"].includes(run.status))
|
|
return set("failed", run.status === "cancelled" ? "Cancelled" : "Failed");
|
|
if (run.status === "running") {
|
|
const leaseConfirmed =
|
|
coordinator?.phase === "observed" &&
|
|
coordinator.leaseExpiresAt &&
|
|
coordinator.leaseExpiresAt > now;
|
|
let processConfirmed = false;
|
|
if (run.runtimeMode === "legacy" && run.processPid) {
|
|
try {
|
|
process.kill(run.processPid, 0);
|
|
processConfirmed = true;
|
|
} catch {
|
|
/* No execution confirmation. */
|
|
}
|
|
}
|
|
if (leaseConfirmed && waitingForProvider) {
|
|
projection.nextAction = "The assignment is offered. Waiting for the external agent to accept it.";
|
|
return set("waiting_for_provider", `Waiting for ${waitingForProvider === "muse" ? "Muse" : "Dot"}`);
|
|
}
|
|
return leaseConfirmed || processConfirmed
|
|
? set("working", "Working")
|
|
: set("reconnecting", "Confirming execution");
|
|
}
|
|
if (run.status === "queued" && !run.startedAt && setupAgent
|
|
&& isAgentAwaitingSetup(setupAgent)) {
|
|
const needsRepair = setupAgent.retryRequired === true;
|
|
projection.cause = needsRepair ? "agent_setup_failed" : "agent_setup_pending";
|
|
projection.nextAction = needsRepair
|
|
? "Open the agent's Harness / Runtime settings to repair its configuration, then retry verification."
|
|
: "This turn starts automatically after the agent finishes setup and verification.";
|
|
return set("queued", needsRepair ? "Agent setup needs attention" : "Waiting for agent setup");
|
|
}
|
|
if (run.status === "queued" && !run.startedAt && readTaskDrain(now)) {
|
|
projection.cause = "task_drain";
|
|
projection.nextAction = "This turn will start automatically when maintenance finishes.";
|
|
return set("queued", "Waiting for maintenance");
|
|
}
|
|
// Process-local, like the drain: while this server starts, recovers, or
|
|
// shuts down, its full queued-run pass has not run (or no longer runs).
|
|
const serverPhase = getStartupRecoveryState().phase;
|
|
if (run.status === "queued" && startupQueuedRunPassOutstanding()) {
|
|
projection.cause = "server_restarting";
|
|
projection.nextAction = serverPhase === "draining"
|
|
? "The server is restarting for an update. This turn starts automatically on the replacement server."
|
|
: "The server restarted and is finishing recovery. This turn starts automatically when recovery completes.";
|
|
return set("queued", "Waiting for server restart");
|
|
}
|
|
if (run.status === "queued" && waitingForPredecessor) {
|
|
projection.cause = "previous_run_settling";
|
|
projection.nextAction = "The previous run on this task is still releasing its runner. This turn starts automatically when it settles.";
|
|
return set("queued", "Waiting for previous run to stop");
|
|
}
|
|
return projection;
|
|
}
|