mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 09:50:48 +01:00
## Thinking Path > - Paperclip lets people start and stop work by AI agents. > - Stop must end the current turn and preserve the task for a later message. > - A person can press Stop while the server still prepares the turn. > - The preparation owner can prove that no model work started. > - That path saved the proof but left the Stop receipt at `requested`. > - The page then reported an unverified Stop and a failed attempt. > - This PR completes the receipt after proof and shows the confirmed Stop without a failure notice. ## Linked Issues or Issue Description **What happened?** A person submits a task by mistake and presses Stop during preparation. The run ends before the adapter starts. The server records that no provider work started, but the Stop receipt stays at `requested`. The page waits for a receipt that will never change. It then reports that it cannot verify Stop. The task also shows a failed-attempt notice. **Expected behavior** Stop ends the current turn without a failure notice. A later message starts one new turn through normal admission. Stop does not restart the task by itself. **Steps to reproduce** Hold session preparation before adapter dispatch. Press Stop through the run API. Release preparation and wait for the owner to finish. Read the run and the task's interruption state. Send a new task comment. Related: #20644 covers saved Retry after a committed native cancellation. #21382 covers a provider failure after native Stop. This PR covers Stop before adapter dispatch and its page presentation. ## What Changed - Complete the Stop receipt in `settleCancelledLegacyPreparation`. Keep its existing checks for process, session, events, leases and saved files. - Require no saved `budgetDispatch.handedOffAt` marker under the run lock. This marker can exist before an adapter records its start. A stale controller must keep the recovery hold after this handoff. - Write and require the `preparation_settled` proof before clearing the failed-attempt notice. Both page queries use one bounded SQL projection. Unknown and foreign proof labels keep the notice. - Clear an obsolete automatic-recovery marker only for a user's operator Stop. Keep system cancellation delivery metadata so saved comments still reach admission. Cover both cases with persisted recovery records and the stale-comment journey. - Give the bounded SQL result an explicit column name. Keep its real-query tests for typed and bounded Stop evidence. - Give the Retry test an active company membership and remove it during cleanup. The test uses normal authorization. - Record an expected cancellation as information in the run log. - Add a regression through the real Stop, run-read and comment routes with a real test database. Only the adapter boundary is simulated. Check that preparation cannot confirm Stop early, Stop does not restart work, and one later message starts one turn. - Document preparation Stop settlement in the heartbeat module notes. ## Verification - Head: `372ad381f4b60258e17b6b05eda133ed875e7de3`. This revision includes master `20b0d4e5a5`. [Full CI attempt 2](https://github.com/paperclipai/paperclip/actions/runs/43910771322/attempts/2) completed successfully: all 52 executed jobs passed, including Build, Typecheck, all Runner lanes, all nine serialized suites, all twelve server shards and all eight browser shards. Greptile and Paperclip Review are both 5/5, and Superagent passed on this exact head. All three inline threads are resolved. The final commit inventory has 57 successful checks, one neutral review and three expected skips, with no pending or failing checks; Snyk also passed. - All 339 focused tests pass at this head across 11 suites. The Stop, saved-comment and fresh-continuation suites pass 132 tests. Eight other Stop contract suites pass 207 tests. - Red: the preparation regression receives `executionCancellation.state = requested` after the owner finishes. The presentation regressions report a failed attempt after a confirmed Stop. - Red: automatic recovery resolves the hold first, then preparation confirms Stop, but the run still shows a recovery notice. Green: the confirmed operator Stop clears the obsolete marker and projects `completed` / `Stopped`. - Red: the previous repair also changed system cancellations. Both stale-comment and Retry cases pass on the base and fail on the previous PR head. A direct regression also catches a preparation Stop receipt written for a system cancellation. Green: only a user's operator Stop gets this receipt; system cancellations keep their recovery delivery metadata. - Red: the real-query tests read PostgreSQL's implicit expression-column name after the expression changed. Green: the query has the explicit `result_json` alias, and all typed and bounded evidence assertions pass. - Red: a saved dispatch handoff marker was present before any adapter event. Both new cases returned an acknowledged Stop instead of keeping the hold. Green: the current locked row must have no marker, including a marker with JSON null. The receipt stays requested and the hold stays active. The tests use a stale owner snapshot. - The HTTP Stop/read/comment test uses real routes and PostgreSQL. It proves that pending preparation cannot confirm Stop, the owner confirms it after cleanup, Stop does not restart work, and a later message starts exactly one turn. Only the adapter boundary is simulated. - All source typecheck layers and the changed `test-tree-0` layer pass in low-memory mode. The first full typecheck attempt lost the compiler to the workspace memory limit while tests ran in parallel. The affected layers then passed alone. Other test layers passed on the previous head. The full current-head CI Typecheck + Release Registry job passed. `git diff --check` passes. - `pnpm verify` returned exit 2 and requested broader CI. No full local suite was started. - Environment: Linux x64, Node 25.9.0, fresh embedded PostgreSQL. This does not execute a live model or change staging. CI supplies Rust and full-suite coverage. - Previous-head CI found the SQL assertion and continuation regressions above. Its cancelled workspace lane lost its runner before a test result and was retried once. Previous-head results do not count as final-head passes. ## Risks - Only a requested, expected operator Stop with an acknowledged receipt gets neutral presentation. Unknown termination and file recovery remain visible. - The preparation owner writes acknowledgement only after all existing evidence checks pass. Terminal status alone is insufficient. A retained local lease test checks both refusal before release and confirmation after release. - The receipt does not grant replay authority. Budget, approval, dependency and cleanup gates still control later messages. - No schema or migration changes are required. ## Model Used OpenAI Codex, GPT-6, with reasoning and code tools. The exact served model build and context size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
614 lines
30 KiB
TypeScript
614 lines
30 KiB
TypeScript
import { and, desc, eq, inArray, isNull, ne, sql } from "drizzle-orm";
|
|
import {
|
|
agents,
|
|
heartbeatRuns,
|
|
issues,
|
|
dotAgentBindings,
|
|
dotRunnerAssignments,
|
|
museAgentBindings,
|
|
museRunnerAssignments,
|
|
issueRecoveryActions,
|
|
issueThreadInteractions,
|
|
nativeRunFinalizations,
|
|
type Db,
|
|
} from "@paperclipai/db";
|
|
import type { ExecutionProjection } from "@paperclipai/shared";
|
|
import { EXECUTION_CONTROL_DEADLINE_MS } from "./execution-control-deadline.js";
|
|
import { executionFailureRetryCount } from "./execution-recovery-attempt.js";
|
|
import { retryIdempotentDatabaseOperation } from "../database-retry.js";
|
|
import { workspaceRecoveryDisplay } from "./workspace-recovery-receipt.js";
|
|
import { readTaskDrain } from "./task-admission.js";
|
|
import { getStartupRecoveryState, startupQueuedRunPassOutstanding } from "../startup-recovery-state.js";
|
|
import { hasPendingRunCleanup } from "./execution-blocker.js";
|
|
import { isAgentAwaitingSetup } from "../modules/agent-lifecycle/index.js";
|
|
import { NATIVE_RESPONSE_WAKE_WAITING_REASON } from "./native-runtime/status-arbiter.js";
|
|
import { agentQueueLinesForRuns } from "./heartbeat/queued-run-line.js";
|
|
import { isAcknowledgedLegacyUserStop, runStopPresentationJson } from "./run-cancellation.js";
|
|
const MAX_PREDECESSOR_CLEANUP_CHECKS = 16;
|
|
const ANOTHER_ROUTINE_TASK = "another task in this routine";
|
|
const text = (v: unknown) => (typeof v === "string" ? v : null);
|
|
const executionRunColumns = {
|
|
id: heartbeatRuns.id,
|
|
agentId: heartbeatRuns.agentId,
|
|
driverKind: heartbeatRuns.driverKind,
|
|
errorCode: heartbeatRuns.errorCode,
|
|
executionControlDeadlineAt: heartbeatRuns.executionControlDeadlineAt,
|
|
finishedAt: heartbeatRuns.finishedAt,
|
|
lastOutputAt: heartbeatRuns.lastOutputAt,
|
|
lastUsefulActionAt: heartbeatRuns.lastUsefulActionAt,
|
|
nativeIssueId: heartbeatRuns.nativeIssueId,
|
|
nextAction: heartbeatRuns.nextAction,
|
|
processPid: heartbeatRuns.processPid,
|
|
retryOfRunId: heartbeatRuns.retryOfRunId,
|
|
runtimeMode: heartbeatRuns.runtimeMode,
|
|
scheduledRetryAt: heartbeatRuns.scheduledRetryAt,
|
|
scheduledRetryAttempt: heartbeatRuns.scheduledRetryAttempt,
|
|
scheduledRetryReason: heartbeatRuns.scheduledRetryReason,
|
|
startedAt: heartbeatRuns.startedAt,
|
|
status: heartbeatRuns.status,
|
|
contextSnapshot: sql<Record<string, unknown>>`jsonb_build_object(
|
|
'issueId', ${heartbeatRuns.contextSnapshot}->'issueId',
|
|
'wakeReason', ${heartbeatRuns.contextSnapshot}->'wakeReason',
|
|
'failureRetriesBeforeAiConnectionWait', ${heartbeatRuns.contextSnapshot}->'failureRetriesBeforeAiConnectionWait',
|
|
'failureRetriesBeforeWorkspaceWait', ${heartbeatRuns.contextSnapshot}->'failureRetriesBeforeWorkspaceWait',
|
|
'executionRetryAccounting', ${heartbeatRuns.contextSnapshot}->'executionRetryAccounting')`,
|
|
resultJson: sql<Record<string, unknown>>`${runStopPresentationJson(heartbeatRuns.resultJson)} || jsonb_build_object(
|
|
'finalizationPhase', ${heartbeatRuns.resultJson}->'finalizationPhase',
|
|
'finalizationReasonCode', ${heartbeatRuns.resultJson}->'finalizationReasonCode')`,
|
|
};
|
|
type Run = Pick<typeof heartbeatRuns.$inferSelect, keyof typeof executionRunColumns>;
|
|
type Coordinator = typeof nativeRunFinalizations.$inferSelect;
|
|
type WaitingExternalProvider = "muse" | "openai_dot";
|
|
type Recovery = Pick<
|
|
typeof issueRecoveryActions.$inferSelect,
|
|
"cause" | "nextAction"
|
|
> &
|
|
Partial<
|
|
Pick<typeof issueRecoveryActions.$inferSelect, "status" | "evidence">
|
|
>;
|
|
|
|
/** Batched reads; list consumers do not perform per-task polling. */
|
|
export async function executionProjectionsForRuns(
|
|
db: Db,
|
|
companyId: string,
|
|
runIds: string[],
|
|
now = new Date(),
|
|
options: { retryDatabaseReads?: boolean } = {},
|
|
) {
|
|
const projections = new Map<string, ExecutionProjection>();
|
|
if (!runIds.length) return projections;
|
|
// Opt in only with a pooled Db outside a transaction. Issue enrichment also
|
|
// calls this helper with a transaction, whose failed reads must not replay.
|
|
const read = <T>(query: () => Promise<T>) => options.retryDatabaseReads
|
|
? retryIdempotentDatabaseOperation(query)
|
|
: query();
|
|
const runs = await read(async () => db
|
|
.select(executionRunColumns)
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.companyId, companyId),
|
|
inArray(heartbeatRuns.id, runIds),
|
|
),
|
|
));
|
|
const coordinators = await read(async () => db
|
|
.select()
|
|
.from(nativeRunFinalizations)
|
|
.where(
|
|
and(
|
|
eq(nativeRunFinalizations.companyId, companyId),
|
|
inArray(nativeRunFinalizations.runId, runIds),
|
|
),
|
|
));
|
|
const issueIds = [
|
|
...new Set(
|
|
runs
|
|
.map((run) => run.nativeIssueId ?? text(run.contextSnapshot?.issueId))
|
|
.filter((id): id is string => !!id),
|
|
),
|
|
];
|
|
const pending = issueIds.length
|
|
? await read(async () => db
|
|
.select({
|
|
issueId: issueThreadInteractions.issueId,
|
|
kind: issueThreadInteractions.kind,
|
|
})
|
|
.from(issueThreadInteractions)
|
|
.where(
|
|
and(
|
|
eq(issueThreadInteractions.companyId, companyId),
|
|
inArray(issueThreadInteractions.issueId, issueIds),
|
|
eq(issueThreadInteractions.status, "pending"),
|
|
),
|
|
))
|
|
: [];
|
|
const recovery = issueIds.length
|
|
? await read(async () => db
|
|
.select({
|
|
issueId: issueRecoveryActions.sourceIssueId,
|
|
cause: issueRecoveryActions.cause,
|
|
nextAction: issueRecoveryActions.nextAction,
|
|
evidence: issueRecoveryActions.evidence,
|
|
status: issueRecoveryActions.status,
|
|
})
|
|
.from(issueRecoveryActions)
|
|
.where(
|
|
and(
|
|
eq(issueRecoveryActions.companyId, companyId),
|
|
inArray(issueRecoveryActions.sourceIssueId, issueIds),
|
|
inArray(issueRecoveryActions.status, [
|
|
"active",
|
|
"escalated",
|
|
"resolved",
|
|
]),
|
|
),
|
|
)
|
|
.orderBy(desc(issueRecoveryActions.updatedAt)))
|
|
: [];
|
|
// A live controller is not evidence that an external agent accepted work.
|
|
// Read each provider journal once for the batch, never once per task.
|
|
const waitingByRun = new Map<string, WaitingExternalProvider>();
|
|
const externalRunIds = (driverKind: string) => runs
|
|
.filter(run => run.status === "running" && run.runtimeMode === "native" && run.driverKind === driverKind)
|
|
.map(run => run.id);
|
|
const museRunIds = externalRunIds("muse_external");
|
|
if (museRunIds.length) {
|
|
const offered = await read(async () => db
|
|
.select({ runId: museRunnerAssignments.runId })
|
|
.from(museRunnerAssignments)
|
|
.innerJoin(museAgentBindings, and(
|
|
eq(museAgentBindings.id, museRunnerAssignments.bindingId),
|
|
eq(museAgentBindings.companyId, museRunnerAssignments.companyId),
|
|
eq(museAgentBindings.agentId, museRunnerAssignments.agentId),
|
|
eq(museAgentBindings.generation, museRunnerAssignments.bindingGeneration),
|
|
isNull(museAgentBindings.revokedAt),
|
|
))
|
|
.where(and(
|
|
eq(museRunnerAssignments.companyId, companyId),
|
|
inArray(museRunnerAssignments.runId, museRunIds),
|
|
inArray(museRunnerAssignments.status, ["offered", "claimed"]),
|
|
isNull(museRunnerAssignments.nativeAcceptedAt),
|
|
)));
|
|
for (const row of offered) waitingByRun.set(row.runId, "muse");
|
|
}
|
|
const dotRunIds = externalRunIds("openai_dot_mcp");
|
|
if (dotRunIds.length) {
|
|
const offered = await read(async () => db
|
|
.select({ runId: dotRunnerAssignments.runId })
|
|
.from(dotRunnerAssignments)
|
|
.innerJoin(dotAgentBindings, and(
|
|
eq(dotAgentBindings.id, dotRunnerAssignments.bindingId),
|
|
eq(dotAgentBindings.companyId, dotRunnerAssignments.companyId),
|
|
eq(dotAgentBindings.agentId, dotRunnerAssignments.agentId),
|
|
eq(dotAgentBindings.generation, dotRunnerAssignments.bindingGeneration),
|
|
isNull(dotAgentBindings.revokedAt),
|
|
))
|
|
.where(and(
|
|
eq(dotRunnerAssignments.companyId, companyId),
|
|
inArray(dotRunnerAssignments.runId, dotRunIds),
|
|
eq(dotRunnerAssignments.status, "offered"),
|
|
isNull(dotRunnerAssignments.acceptedAt),
|
|
)));
|
|
for (const row of offered) waitingByRun.set(row.runId, "openai_dot");
|
|
}
|
|
// A queued turn cannot start while its predecessor's cleanup is pending
|
|
// (for example a stopped run whose runner has not released yet). Name that
|
|
// wait instead of an unexplained "Queued". The predecessors are the ones
|
|
// admission checks: the task's execution owner, the run this one retries,
|
|
// and the agent's latest settled run on the same task, whose in-process
|
|
// executor the claim also waits for. Pending uses admission's own predicate.
|
|
const settlingPredecessorKeys = new Set<string>();
|
|
const queuedRuns = runs.filter(run => run.status === "queued");
|
|
const setupAgentIds = [...new Set(queuedRuns.filter(run => !run.startedAt).map(run => run.agentId))];
|
|
const setupAgents = setupAgentIds.length ? await read(async () => db
|
|
.select({ id: agents.id, lifecycleState: agents.lifecycleState,
|
|
lifecycleHolds: agents.lifecycleHolds,
|
|
retryRequired: sql<boolean>`coalesce(${agents.lifecycleOperation}->>'retryRequired', 'false') = 'true'` })
|
|
.from(agents).where(and(eq(agents.companyId, companyId), inArray(agents.id, setupAgentIds)))) : [];
|
|
const setupByAgent = new Map(setupAgents.map(agent => [agent.id, agent]));
|
|
// Only never-started runs wait on the agent's limit; their place in line is
|
|
// read once per full agent, not once per run.
|
|
const agentQueueByRun = queuedRuns.some(run => !run.startedAt)
|
|
? await read(() => agentQueueLinesForRuns(db, companyId, queuedRuns.filter(run => !run.startedAt), now))
|
|
: new Map<string, NonNullable<ExecutionProjection["agentQueue"]>>();
|
|
const issueIdOf = (run: Run) => run.nativeIssueId ?? text(run.contextSnapshot?.issueId);
|
|
const queuedIssueIds = [...new Set(queuedRuns.map(issueIdOf).filter((id): id is string => !!id))];
|
|
// A routine runs one execution at a time (issues_open_routine_execution_uq):
|
|
// a never-started run of a routine task waits while another open task of
|
|
// the same routine holds an execution. One read for the batch.
|
|
const routineHolderByIssue = new Map<string, string>();
|
|
const unstartedIssueIds = [...new Set(queuedRuns.filter(run => !run.startedAt).map(issueIdOf).filter((id): id is string => !!id))];
|
|
if (unstartedIssueIds.length) {
|
|
const holders = await read(async () => db.execute<{ issueId: string; holderIdentifier: string | null }>(sql`
|
|
-- A private holder is never named: this projection has no reader to check access for.
|
|
select waiting.id as "issueId", case when holder.visibility = 'open' then holder.identifier end as "holderIdentifier"
|
|
from ${issues} waiting
|
|
join ${issues} holder on holder.company_id = waiting.company_id
|
|
and holder.origin_kind = 'routine_execution'
|
|
and holder.origin_id = waiting.origin_id
|
|
and holder.origin_fingerprint is not distinct from waiting.origin_fingerprint
|
|
and holder.id <> waiting.id
|
|
and holder.hidden_at is null
|
|
and holder.execution_run_id is not null
|
|
and holder.status in ('backlog', 'todo', 'in_progress', 'in_review', 'blocked')
|
|
where waiting.company_id = ${companyId}
|
|
and waiting.origin_kind = 'routine_execution'
|
|
and waiting.origin_id is not null
|
|
and waiting.id in (${sql.join(unstartedIssueIds.map(id => sql`${id}::uuid`), sql`, `)})`));
|
|
for (const row of [...holders]) {
|
|
if (!routineHolderByIssue.has(row.issueId)) routineHolderByIssue.set(row.issueId, row.holderIdentifier ?? ANOTHER_ROUTINE_TASK);
|
|
}
|
|
}
|
|
if (queuedIssueIds.length) {
|
|
const issueKey = sql<string>`coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.contextSnapshot}->>'issueId')`;
|
|
// Two set-based reads for the whole batch, then one bounded, parallel
|
|
// admission check per distinct predecessor.
|
|
const [owners, latestSettled] = await Promise.all([
|
|
read(async () => db
|
|
.select({ id: issues.id, executionRunId: issues.executionRunId })
|
|
.from(issues)
|
|
.where(and(eq(issues.companyId, companyId), inArray(issues.id, queuedIssueIds)))),
|
|
read(async () => db
|
|
.selectDistinctOn([heartbeatRuns.agentId, issueKey], { id: heartbeatRuns.id, agentId: heartbeatRuns.agentId, issueId: issueKey })
|
|
.from(heartbeatRuns)
|
|
.where(and(
|
|
eq(heartbeatRuns.companyId, companyId),
|
|
inArray(heartbeatRuns.agentId, [...new Set(queuedRuns.map(run => run.agentId))]),
|
|
inArray(issueKey, queuedIssueIds),
|
|
inArray(heartbeatRuns.status, ["succeeded", "failed", "cancelled", "timed_out"]),
|
|
))
|
|
.orderBy(heartbeatRuns.agentId, issueKey, desc(heartbeatRuns.createdAt))),
|
|
]);
|
|
const ownerByIssue = new Map(owners.map(row => [row.id, row.executionRunId]));
|
|
const latestByAgentIssue = new Map(latestSettled.map(row => [`${row.agentId}:${row.issueId}`, row.id]));
|
|
const candidatesByRun = new Map<string, string[]>();
|
|
for (const run of queuedRuns) {
|
|
const issueId = issueIdOf(run);
|
|
if (!issueId) continue;
|
|
candidatesByRun.set(run.id, [...new Set([ownerByIssue.get(issueId), run.retryOfRunId,
|
|
latestByAgentIssue.get(`${run.agentId}:${issueId}`)].filter((id): id is string => !!id && id !== run.id))]);
|
|
}
|
|
const distinct = [...new Set([...candidatesByRun.values()].flat())];
|
|
// A label is not worth an unbounded fan-out on large list reads.
|
|
if (distinct.length <= MAX_PREDECESSOR_CLEANUP_CHECKS) {
|
|
const pendingIds = new Set((await Promise.all(distinct.map(async id =>
|
|
(await read(() => hasPendingRunCleanup(db, companyId, id))) ? id : null))).filter((id): id is string => !!id));
|
|
for (const [runId, candidates] of candidatesByRun)
|
|
if (candidates.some(id => pendingIds.has(id))) settlingPredecessorKeys.add(runId);
|
|
}
|
|
}
|
|
const coordinatorByRun = new Map(coordinators.map((row) => [row.runId, row]));
|
|
const responseWaitRuns = runs.filter(run => run.status === "succeeded"
|
|
&& run.resultJson?.finalizationReasonCode === NATIVE_RESPONSE_WAKE_WAITING_REASON && issueIdOf(run));
|
|
const responseWaitIssues = responseWaitRuns.length ? await read(async () => db
|
|
.select({ id: issues.id, status: issues.status, lastStatusDecisionId: issues.lastStatusDecisionId })
|
|
.from(issues)
|
|
.where(and(eq(issues.companyId, companyId), inArray(issues.id, [...new Set(responseWaitRuns.map(run => issueIdOf(run)!))])))) : [];
|
|
const responseWaitIssueById = new Map(responseWaitIssues.map(row => [row.id, row]));
|
|
const currentResponseWait = (run: Run) => {
|
|
const issue = responseWaitIssueById.get(issueIdOf(run) ?? "");
|
|
const decisionId = coordinatorByRun.get(run.id)?.decisionId;
|
|
return Boolean(issue && decisionId && issue.status === "in_progress" && issue.lastStatusDecisionId === decisionId);
|
|
};
|
|
for (const run of runs) {
|
|
const issueId = run.nativeIssueId ?? text(run.contextSnapshot?.issueId);
|
|
const matching = recovery.filter(
|
|
(row) =>
|
|
row.issueId === issueId &&
|
|
(row.status !== "resolved" || row.evidence.executionReconciliation || row.evidence.automaticRecovery) &&
|
|
(row.evidence.runId === run.id || row.evidence.sourceRunId === run.id),
|
|
);
|
|
const action =
|
|
matching.find((row) => row.status !== "resolved") ?? matching[0];
|
|
projections.set(
|
|
run.id,
|
|
projectExecution(
|
|
run,
|
|
coordinatorByRun.get(run.id),
|
|
pending.filter((row) => row.issueId === issueId),
|
|
action,
|
|
now,
|
|
waitingByRun.get(run.id),
|
|
settlingPredecessorKeys.has(run.id),
|
|
setupByAgent.get(run.agentId),
|
|
currentResponseWait(run),
|
|
agentQueueByRun.get(run.id),
|
|
run.status === "queued" && !run.startedAt ? routineHolderByIssue.get(issueIdOf(run) ?? "") : undefined,
|
|
),
|
|
);
|
|
}
|
|
return projections;
|
|
}
|
|
export async function executionProjectionForRun(
|
|
db: Db,
|
|
companyId: string,
|
|
runId: string,
|
|
now = new Date(),
|
|
): Promise<ExecutionProjection | null> {
|
|
return (
|
|
(await executionProjectionsForRuns(db, companyId, [runId], now)).get(
|
|
runId,
|
|
) ?? null
|
|
);
|
|
}
|
|
export function projectExecution(
|
|
run: Run,
|
|
coordinator: Coordinator | undefined,
|
|
pending: Array<{ kind: string }>,
|
|
recoveryAction: Recovery | undefined,
|
|
now = new Date(),
|
|
waitingForProvider?: WaitingExternalProvider,
|
|
waitingForPredecessor = false,
|
|
setupAgent?: { lifecycleState: string; lifecycleHolds: readonly string[]; retryRequired?: boolean },
|
|
responseWaitCurrent = false,
|
|
agentQueue?: ExecutionProjection["agentQueue"],
|
|
/** The open task of the same routine that holds the routine's one execution. */
|
|
routineHolder?: string,
|
|
): ExecutionProjection {
|
|
const detail = coordinator?.failureDetail ?? {};
|
|
const successorRunId = text(detail.successorRunId);
|
|
const projection: ExecutionProjection = {
|
|
// A live run is always stoppable by the board (POST /heartbeat-runs/:id/cancel);
|
|
// a failed native run is stoppable only while its retry is still pending.
|
|
canStop: run.status === "running" || run.status === "queued" || (
|
|
run.runtimeMode === "native" && run.status === "failed" && run.driverKind !== "grok_bot_mcp" && !!run.nativeIssueId
|
|
&& coordinator?.runId === run.id && coordinator.issueId === run.nativeIssueId && coordinator.resultId === null
|
|
&& coordinator.phase === "retryable_failure"),
|
|
phase: "queued",
|
|
label: "Queued",
|
|
cause: run.errorCode ?? coordinator?.failureCode ?? null,
|
|
lastConfirmedActivityAt:
|
|
(
|
|
run.lastUsefulActionAt ??
|
|
run.lastOutputAt ??
|
|
run.startedAt
|
|
)?.toISOString() ?? null,
|
|
retryAt:
|
|
(coordinator?.nextAttemptAt ?? run.scheduledRetryAt)?.toISOString() ??
|
|
null,
|
|
attempt: coordinator?.attempt ?? executionFailureRetryCount(run) + 1,
|
|
maxAttempts: 3,
|
|
recoveryOwner: null,
|
|
nextAction: text(detail.nextAction) ?? run.nextAction,
|
|
permittedActions: ["inspect_run"],
|
|
predecessorRunId: run.retryOfRunId,
|
|
successorRunId,
|
|
};
|
|
const set = (phase: ExecutionProjection["phase"], label: string) =>
|
|
Object.assign(projection, { phase, label });
|
|
if (recoveryAction?.status === "resolved" && recoveryAction.evidence?.automaticRecovery) {
|
|
projection.cause = recoveryAction.cause;
|
|
projection.nextAction = workspaceRecoveryDisplay(recoveryAction.evidence, run.id, { status: run.status, retryOffered: false })?.message
|
|
?? recoveryAction.nextAction;
|
|
const continuation = recoveryAction.evidence.explicitUserContinuation as
|
|
{ previousRunId?: unknown; runId?: unknown } | undefined;
|
|
const explicitSuccessor = text(continuation?.runId);
|
|
if (continuation?.previousRunId === run.id && explicitSuccessor && explicitSuccessor !== run.id) {
|
|
// The admission transaction already recorded the user's successor. Keep
|
|
// the old failure diagnostic without making it hold the newer attempt.
|
|
projection.successorRunId = explicitSuccessor;
|
|
projection.nextAction = null;
|
|
return set("completed", "Continued in another run");
|
|
}
|
|
// Diagnostic projection only: no user decision or replay affordance.
|
|
return set("recovery_needed", "Stopped");
|
|
}
|
|
if (
|
|
recoveryAction?.status === "resolved" &&
|
|
recoveryAction.evidence?.executionReconciliation
|
|
) {
|
|
if (recoveryAction.evidence.continuationDelivery === "invalidated") {
|
|
projection.cause = "continuation_ownership_changed";
|
|
projection.nextAction = "The task changed before this continuation started, so it was not started.";
|
|
return set("failed", "Continuation cancelled");
|
|
}
|
|
// A hold the fresh-continuation fallback settled together with another:
|
|
// the task continues through that record.
|
|
if (recoveryAction.evidence.continuationDelivery === "coalesced") {
|
|
projection.cause = null;
|
|
projection.nextAction = "Paperclip continues the task in a fresh run from its saved work.";
|
|
return set("queued", "Continuing automatically");
|
|
}
|
|
const reconciliation = recoveryAction.evidence.executionReconciliation;
|
|
if (recoveryAction.cause === "native_restart_recovery_blocked"
|
|
&& recoveryAction.evidence.reason === "controller_process_space_unverified"
|
|
&& recoveryAction.evidence.continuationDelivery === "not_requested"
|
|
&& reconciliation && typeof reconciliation === "object" && !Array.isArray(reconciliation)
|
|
&& "runId" in reconciliation && reconciliation.runId === run.id
|
|
&& "controllerStopped" in reconciliation && reconciliation.controllerStopped === true
|
|
&& "providerStopped" in reconciliation && reconciliation.providerStopped === true
|
|
&& "controllerAuthorityDigest" in reconciliation && typeof reconciliation.controllerAuthorityDigest === "string" && /^[0-9a-f]{64}$/.test(reconciliation.controllerAuthorityDigest)) {
|
|
const continuation = recoveryAction.evidence.explicitUserContinuation as
|
|
{ previousRunId?: unknown; runId?: unknown } | undefined;
|
|
const explicitSuccessor = text(continuation?.runId);
|
|
if (continuation?.previousRunId === run.id && explicitSuccessor && explicitSuccessor !== run.id) {
|
|
projection.successorRunId = explicitSuccessor;
|
|
projection.nextAction = null;
|
|
return set("completed", "Continued in another run");
|
|
}
|
|
projection.cause = null;
|
|
projection.nextAction = "The original execution was contained and its action outcomes reviewed. Use Retry to request another attempt.";
|
|
return set("failed", "Execution contained");
|
|
}
|
|
const deliveredRunId = text(recoveryAction.evidence.continuationRunId);
|
|
projection.successorRunId = deliveredRunId;
|
|
projection.cause = null;
|
|
projection.nextAction = deliveredRunId
|
|
? null
|
|
: recoveryAction.evidence.freshContinuationSettlement
|
|
? "Paperclip continues the task in a fresh run from its saved work."
|
|
: "The previous execution is reconciled. Its continuation is queued for delivery.";
|
|
return set(
|
|
deliveredRunId ? "completed" : "queued",
|
|
deliveredRunId ? "Continued in another run"
|
|
: recoveryAction.evidence.freshContinuationSettlement ? "Continuing automatically" : "Continuation queued",
|
|
);
|
|
}
|
|
if (
|
|
(run.status === "running" &&
|
|
(run.executionControlDeadlineAt || coordinator?.controlDeadlineAt)) ||
|
|
[
|
|
"result_persisted",
|
|
"ready_for_assessment",
|
|
"arbitrating",
|
|
"workspace_pending",
|
|
].includes(coordinator?.phase ?? "")
|
|
)
|
|
return set("finishing", "Finishing");
|
|
if (successorRunId) return set("completed", "Continued in another run");
|
|
if (run.status === "scheduled_retry" && run.scheduledRetryReason === "ai_connection_busy") {
|
|
projection.nextAction = "Waiting for the AI subscription's current execution to finish; the scheduled check will revalidate access.";
|
|
return set("retry_scheduled", "Waiting for AI subscription");
|
|
}
|
|
if (run.status === "scheduled_retry" && run.scheduledRetryReason === "ai_connection_pool_wait"
|
|
&& run.contextSnapshot?.wakeReason === "provider_limit_retry") {
|
|
const resetAt = run.scheduledRetryAt ? new Date(run.scheduledRetryAt).toISOString() : "its reported reset";
|
|
projection.nextAction = `Waiting for the agent's provider usage limit to reset; the work starts at ${resetAt}. To continue sooner, restore the provider allowance or change the agent's AI connection, then start a manual run.`;
|
|
return set("retry_scheduled", "Waiting for usage limit reset");
|
|
}
|
|
if (run.status === "scheduled_retry" && run.scheduledRetryReason === "agent_directory_busy") {
|
|
projection.nextAction = "Waiting for another run of this agent to finish with its files; this task continues automatically.";
|
|
return set("retry_scheduled", "Waiting for agent files");
|
|
}
|
|
if (run.status === "scheduled_retry" && run.scheduledRetryReason === "workspace_busy") {
|
|
projection.nextAction = "Waiting for the live workspace holder to finish; the scheduled check will revalidate ownership.";
|
|
return set("retry_scheduled", "Waiting for workspace");
|
|
}
|
|
// Cleanup can fail before a finalization coordinator exists. The missing
|
|
// row must not turn a quarantined native session into an ordinary Retry.
|
|
const cleanupQuarantined = run.runtimeMode === "native" &&
|
|
["failed", "timed_out"].includes(run.status) &&
|
|
run.errorCode === "native_session_cleanup_quarantined";
|
|
if (!cleanupQuarantined && (
|
|
coordinator?.phase === "retryable_failure" ||
|
|
run.status === "scheduled_retry"
|
|
)) {
|
|
projection.recoveryOwner = "agent";
|
|
if (coordinator?.failureCode === "native_governed_wait_settlement_pending") {
|
|
projection.nextAction = "The work and waiting state are saved. Paperclip is confirming shutdown before continuing.";
|
|
return set("finishing", "Finishing shutdown");
|
|
}
|
|
return set(
|
|
projection.retryAt && new Date(projection.retryAt) > now
|
|
? "retry_scheduled"
|
|
: "reconnecting",
|
|
projection.retryAt && new Date(projection.retryAt) > now
|
|
? "Retry scheduled"
|
|
: "Reconnecting",
|
|
);
|
|
}
|
|
if (coordinator?.phase === "terminal_failure" || recoveryAction || cleanupQuarantined) {
|
|
if (
|
|
!cleanupQuarantined &&
|
|
coordinator?.failureCode === "native_provider_terminal_failed" &&
|
|
!detail.replacementDenied &&
|
|
run.finishedAt &&
|
|
now.getTime() - run.finishedAt.getTime() < EXECUTION_CONTROL_DEADLINE_MS
|
|
) {
|
|
projection.recoveryOwner = "agent";
|
|
projection.nextAction =
|
|
"Checking that the previous provider stopped and its action outcomes are known before continuing.";
|
|
return set("reconnecting", "Checking recovery");
|
|
}
|
|
projection.recoveryOwner = "board";
|
|
projection.cause =
|
|
recoveryAction?.cause ??
|
|
text(detail.replacementDenied) ??
|
|
projection.cause;
|
|
projection.nextAction = recoveryAction?.nextAction ?? projection.nextAction;
|
|
if (cleanupQuarantined && !projection.nextAction) {
|
|
projection.nextAction = "Verify the stopped session and its saved work before starting a new attempt.";
|
|
}
|
|
projection.permittedActions.push("inspect_recovery");
|
|
return set("recovery_needed", "Recovery needed");
|
|
}
|
|
if (run.status === "succeeded") {
|
|
if (pending.length)
|
|
return set(
|
|
pending.some((row) => row.kind === "connection_intent")
|
|
? "waiting_for_access"
|
|
: "waiting_for_answer",
|
|
pending.some((row) => row.kind === "connection_intent")
|
|
? "Waiting for access"
|
|
: "Waiting for answer",
|
|
);
|
|
if (responseWaitCurrent && run.resultJson?.finalizationPhase === "committed" &&
|
|
run.resultJson.finalizationReasonCode === NATIVE_RESPONSE_WAKE_WAITING_REASON) {
|
|
projection.nextAction = "The agent finished its turn and is waiting for a reply. Comment on the task or change its status to continue.";
|
|
return set("waiting_for_answer", "Waiting for a reply");
|
|
}
|
|
return set("completed", "Completed");
|
|
}
|
|
if (isAcknowledgedLegacyUserStop(run)) {
|
|
projection.cause = null;
|
|
projection.nextAction = "Send a new message to continue from saved work.";
|
|
return set("completed", "Stopped");
|
|
}
|
|
if (["failed", "cancelled", "timed_out", "interrupted"].includes(run.status))
|
|
return set("failed", run.status === "cancelled" ? "Cancelled" : "Failed");
|
|
if (run.status === "running") {
|
|
const leaseConfirmed =
|
|
coordinator?.phase === "observed" &&
|
|
coordinator.leaseExpiresAt &&
|
|
coordinator.leaseExpiresAt > now;
|
|
let processConfirmed = false;
|
|
if (run.runtimeMode === "legacy" && run.processPid) {
|
|
try {
|
|
process.kill(run.processPid, 0);
|
|
processConfirmed = true;
|
|
} catch {
|
|
/* No execution confirmation. */
|
|
}
|
|
}
|
|
if (leaseConfirmed && waitingForProvider) {
|
|
projection.nextAction = "The assignment is offered. Waiting for the external agent to accept it.";
|
|
return set("waiting_for_provider", `Waiting for ${waitingForProvider === "muse" ? "Muse" : "Dot"}`);
|
|
}
|
|
return leaseConfirmed || processConfirmed
|
|
? set("working", "Working")
|
|
: set("reconnecting", "Confirming execution");
|
|
}
|
|
if (run.status === "queued" && !run.startedAt && setupAgent
|
|
&& isAgentAwaitingSetup(setupAgent)) {
|
|
const needsRepair = setupAgent.retryRequired === true;
|
|
projection.cause = needsRepair ? "agent_setup_failed" : "agent_setup_pending";
|
|
projection.nextAction = needsRepair
|
|
? "Open the agent's Harness / Runtime settings to repair its configuration, then retry verification."
|
|
: "This turn starts automatically after the agent finishes setup and verification.";
|
|
return set("queued", needsRepair ? "Agent setup needs attention" : "Waiting for agent setup");
|
|
}
|
|
if (run.status === "queued" && !run.startedAt && readTaskDrain(now)) {
|
|
projection.cause = "task_drain";
|
|
projection.nextAction = "This turn will start automatically when maintenance finishes.";
|
|
return set("queued", "Waiting for maintenance");
|
|
}
|
|
// Process-local, like the drain: while this server starts, recovers, or
|
|
// shuts down, its full queued-run pass has not run (or no longer runs).
|
|
const serverPhase = getStartupRecoveryState().phase;
|
|
if (run.status === "queued" && startupQueuedRunPassOutstanding()) {
|
|
projection.cause = "server_restarting";
|
|
projection.nextAction = serverPhase === "draining"
|
|
? "The server is restarting for an update. This turn starts automatically on the replacement server."
|
|
: "The server restarted and is finishing recovery. This turn starts automatically when recovery completes.";
|
|
return set("queued", "Waiting for server restart");
|
|
}
|
|
if (run.status === "queued" && waitingForPredecessor) {
|
|
projection.cause = "previous_run_settling";
|
|
projection.nextAction = "The previous run on this task is still releasing its runner. This turn starts automatically when it settles.";
|
|
return set("queued", "Waiting for previous run to stop");
|
|
}
|
|
// Checked before the agent's limit: a free slot would not start this run.
|
|
if (run.status === "queued" && !run.startedAt && routineHolder) {
|
|
projection.cause = "routine_execution_in_progress";
|
|
projection.nextAction = `This routine runs one task at a time and ${routineHolder === ANOTHER_ROUTINE_TASK ? "another of its tasks" : routineHolder} is still open with its run. This task starts automatically when that one finishes.`;
|
|
return set("queued", `Waiting for ${routineHolder} to finish`);
|
|
}
|
|
if (run.status === "queued" && !run.startedAt && agentQueue) {
|
|
projection.cause = "agent_at_capacity";
|
|
projection.agentQueue = agentQueue;
|
|
projection.nextAction = `${agentQueue.agentName} is running ${agentQueue.runningCount} of ${agentQueue.maxConcurrentRuns} tasks at once. This task is ${agentQueue.position} of ${agentQueue.queuedCount} in line and starts automatically when a slot frees.`;
|
|
return set("queued", `Waiting for ${agentQueue.agentName}`);
|
|
}
|
|
return projection;
|
|
}
|