mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 22:56:58 +01:00
fix(issues): let the board confirm an unidentified run stopped instead of an endless Retry wait (#20310)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before a new attempt can start on a task, Paperclip must prove the previous run's agent stopped, so two agents never work the same task at once > - A local native run can be interrupted after it asked to start the runner but before it recorded the runner's process. The runner may have saved state and started a provider, and provider processes can outlive it, so the existing check (#19694) rightly refuses to assume the run stopped > - Today that case has no way out: every Retry is accepted and then waits forever as `process_identity_missing` (storage chaos ST1, t02/t03) > - This pull request makes the wait explicit and actionable. The task explains why it waits, and a board operator who has checked can confirm the run stopped; the parked Retry then starts at the click > - A second shape from the Daytona storage baseline (t06) parked a Retry with no reason at all: a follow-up was interrupted by a graceful shutdown while it was still restoring its workspace. No provider ran, but the restore hold kept the same agent held, so the Retry never started > - The benefit is that no Retry stays accepted-and-waiting with nothing anyone can do ## Linked Issues or Issue Description Refs #19831. Storage chaos wave `w-20261109T145628-s9902` (ST1), tasks t02 and t03. **What happened (CHA-3, run `3c3f6241`):** - The run requested a runner launch (`native.process_start_requested`), and a K2 restart hit before its process identity was recorded. - After the restart, the resumed attempt failed on the storage outage. The task was blocked. - The board's Retry (202) was parked as `executionWait.reason = process_identity_missing`: "The previous run has no verified stop record." - `recordUnidentifiedLocalRunnerStop` cannot prove the stop because this run's runner saved control-plane state. Once that exists, a provider may have started (#19694), and a provider is not guaranteed to carry `--run-id` or `PAPERCLIP_RUN_ID` (the Codex provider env omits it on purpose). The shared owner prover needs a PID, which the run never recorded. - Nothing could ever clear the wait. **Daytona storage baseline `w-20261109T154810-s9911`, task t06:** - The run succeeded. The follow-up was cancelled (`continuation_task_ownership_changed`), and the next run was interrupted by a graceful shutdown (`server_shutdown_interrupted`) 3 s into `preparing`. It was restoring the saved remote workspace and had launched no provider. - Restore recovery saved the files (`workspaceRecovery.status: recovered`, system recovery `complete`). But the `legacy_execution_requires_reconciliation` restore hold releases only for a different owner, or for a native Stop fenced before any provider turn, so it kept the same agent held. - The UI Retry returned 202 and parked with `executionWait.reason: execution_recovery`, the generic wait, so the task showed a Retry that never started. Automatic proof needs the runner to record its identity before saving state and to group provider processes under it. That is follow-up runner work; this PR removes the endless wait now. ## What Changed - `server/src/services/native-local-process-stop.ts`: new `confirmUnidentifiedLocalRunnerStopped`. It accepts only a finished native run of the task with no recorded process and whose latest launch evidence is an unconfirmed launch on this host in a local environment. It still refuses while a runner with `--run-id <run>` is running. It records `native.local_process_stopped` with `basis: "operator_attested"` and the attesting user. That is the receipt the existing continuation gate already accepts. - `server/src/routes/issues.ts`: `POST /issues/:id/recovery-actions/confirm-run-stopped` (`runId`, `stopEvidence` of 20 to 12,000 characters). It is board-only and requires issue read plus `runtime:confirm_owner_stop`, like the other stop attestations. It logs `issue.run_stop_confirmed` with the evidence, promotes deferred intents for the issue, and resumes each parked task Retry, so work starts at the click. Refusals return 409 with a code. - `packages/shared`: `confirmRunStoppedSchema`, and the presentation user action `confirm_run_stopped`. - `server/src/services/task-interruption-presentation.ts`: a Retry parked as `process_identity_missing` for a native run that recorded no process now presents as waiting for the user (`canRetry: false`, `userAction: "confirm_run_stopped"`), with a message explaining why and who can act. A run with a recorded process keeps the existing verification wait. - UI: - `ExecutionBlockerNotice` shows the explanation and, for users who can confirm a stop, "Confirm it stopped" with a short evidence form. - It reuses `PreviousServerStopConfirmation`, now with optional `action`, explicit `runId` and `formId`, and a custom button label. - The Inbox names the action. - `server/src/services/execution-blocker.ts`: `interruptedPreparationBeforeProviderWorkCondition`. The restore hold also releases for the same agent when the stopped run was a system-interrupted preparation (legacy runtime never resolved, still `preparing`) with no process identity, no runner or provider events, no finalization or result, and every lease released with a confirmed stop. It mirrors the existing interrupted-preparation proof and still requires the completed workspace-recovery receipt. Auto-replay is unchanged; this only lets the user's Retry or message start. - Review fixes: - `unidentifiedLocalLaunchRefusal` holds the endpoint's launch, host and lease checks, and the presentation offers `confirm_run_stopped` only when it passes. - The attested stop wakes cost accounting in the same transaction. - The confirmation form is keyed by task and run. - The route is registered in OpenAPI. ## Verification Red/green (each new test fails against master and passes with this PR): - `server/src/__tests__/issue-confirm-run-stopped-routes.test.ts`: - The parked Retry presents `confirm_run_stopped` with the explanation. - Confirming records the operator-attested stop and the activity entry, satisfies the stop check, and resumes the parked Retry: its receipt is coalesced into a newly started run. - It refuses with 409 while a runner for the run is running. - Agents and viewers get 403. - `ui/src/components/ExecutionBlockerNotice.confirm-run-stopped.test.tsx`: the notice offers "Confirm it stopped" (and no dead Retry) and posts the run and evidence. A viewer sees the explanation without the action. - `server/src/__tests__/interrupted-preparation-restore-retry.test.ts` (t06 shape): - The restore hold no longer blocks. - A Retry is accepted and resumed into a started run, and the audit hold keeps `replay: blocked`. - The hold stays when a provider turn started, when a runner event was recorded, when an operator stopped the run, or when the sandbox has no stop receipt. - `issue-confirm-run-stopped-routes.test.ts`: - A case per refusal code checks the presentation keeps the generic wait and the endpoint returns 409. - An `alreadyStopped` case. - An accounting-wake assertion. - The UI test switches targets with stale evidence. Regression and static checks: - `ExecutionBlockerNotice`, `WorkspaceExportRecovery` and its control tests: 74 passed. - `tsc --noEmit` passes for `packages/shared`, `server` and `ui/src`. `pnpm check:token-gates` is clean. ## Risks - A board operator can now attest that an unidentified run stopped. The attestation is permission-gated, requires written evidence, is activity-logged, and is refused while a runner for the run is visibly running. A wrong attestation could start a second agent beside an orphaned provider, the same trust model as the existing previous-server attestations. - The confirmation component now accepts a run without a recovery action. Existing callers pass the action as before. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed
1 parent
ff2d463eed
commit
8fe524f796
18 files changed
+665
-16
No files matched your search
@@ -15,6 +15,7 @@ import { hasRecoveredCleanupStopReceipt, hasRemoteTerminationReceipt } from "./r
|
||||
import { readNativeWorkspaceSyncReference } from "./native-runtime/native-workspace-sync.js";
|
||||
import { completedWorkspaceRecoveryCondition, hasVerifiedWorkspaceRepair, workspaceRecoveryDisplay } from "./workspace-recovery-receipt.js";
|
||||
import { nativeManualAdmissionHoldsExecution } from "./native-runtime/native-runner-ownership.js";
|
||||
import { PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED } from "./native-local-process-stop.js";
|
||||
|
||||
const blockerContextKeys = ["issueId", "taskId", "source", "connectionIntentResolution",
|
||||
"interactionKind", "interactionStatus", "mutation", "interactionId"] as const;
|
||||
@@ -186,12 +187,57 @@ export function stoppedBeforeProviderTurnCondition() {
|
||||
where result.company_id = stopped.company_id and result.run_id = stopped.id))`;
|
||||
}
|
||||
|
||||
export function interruptedPreparationBeforeProviderWorkCondition() {
|
||||
return sql`(stopped.status = 'interrupted' and stopped.runtime_mode = 'legacy'
|
||||
and stopped.runtime_mode_resolved_at is null and stopped.execution_stage = 'preparing'
|
||||
and stopped.runner_profile_json#>>'{adapterDispatch,adapterType}' = 'paperclip_runner'
|
||||
and stopped.runner_profile_json->'nativeExecutionInput' is null
|
||||
and stopped.process_pid is null and stopped.process_group_id is null and stopped.process_started_at is null
|
||||
and stopped.session_id_after is null and stopped.native_issue_id is null and stopped.native_session_id is null
|
||||
and (stopped.controller_lease_expires_at is null or stopped.controller_lease_expires_at <= now())
|
||||
and (stopped.runner_profile_json->'remoteWorkspace' is null or exists (
|
||||
select 1 from jsonb_array_elements_text(case when jsonb_typeof(${issueRecoveryActions.evidence}#>'{systemRecovery,preservationOperationIds}') = 'array'
|
||||
then ${issueRecoveryActions.evidence}#>'{systemRecovery,preservationOperationIds}' else '[]'::jsonb end) saved(id)
|
||||
join ${workspaceOperations} op on op.id::text = saved.id
|
||||
where op.company_id = stopped.company_id and op.heartbeat_run_id = stopped.id and op.status = 'succeeded'
|
||||
and op.metadata->>'snapshotId' = stopped.runner_profile_json#>>'{remoteWorkspace,snapshotId}'))
|
||||
and coalesce(stopped.result_json#>'{executionRecovery,providerWorkStarted}', 'false'::jsonb) = 'false'::jsonb
|
||||
and case when stopped.result_json->'cancellation' is null then stopped.error_code = 'server_shutdown_interrupted'
|
||||
else stopped.result_json#>>'{cancellation,initiator,type}' = 'system'
|
||||
and coalesce(stopped.result_json#>>'{cancellation,source}', '') <> 'operator'
|
||||
and (stopped.result_json#>>'{cancellation,source}' = 'shutdown'
|
||||
or stopped.result_json#>'{cancellation,expected}' = 'false'::jsonb) end
|
||||
and not exists (select 1 from ${heartbeatRunEvents} work
|
||||
where work.company_id = stopped.company_id and work.run_id = stopped.id
|
||||
and (work.source_event_id is not null
|
||||
or work.event_type in ('adapter.invoke', ${PROCESS_START_REQUESTED}, ${PROCESS_IDENTITY_RECORDED})
|
||||
or ${providerTurnWorkEventCondition(sql`work.event_type`, sql`work.source_event_id`)}))
|
||||
and not exists (select 1 from ${nativeRunFinalizations} coordinator
|
||||
where coordinator.company_id = stopped.company_id and coordinator.run_id = stopped.id)
|
||||
and not exists (select 1 from ${nativeRunResults} result
|
||||
where result.company_id = stopped.company_id and result.run_id = stopped.id)
|
||||
and not exists (select 1 from ${environmentLeases} lease
|
||||
where lease.company_id = stopped.company_id and lease.heartbeat_run_id = stopped.id
|
||||
and not coalesce(lease.released_at is not null and lease.cleanup_status = 'success'
|
||||
and lease.status in ('released', 'expired', 'failed')
|
||||
and coalesce(lease.metadata->>'pendingCleanupInFlight', 'false') = 'false'
|
||||
and (lease.provider = 'local' or (
|
||||
lease.metadata#>>'{remoteExecutionTermination,schema}' = 'paperclip.remote-termination.v1'
|
||||
and lease.metadata#>>'{remoteExecutionTermination,companyId}' = lease.company_id::text
|
||||
and lease.metadata#>>'{remoteExecutionTermination,runId}' = stopped.id::text
|
||||
and lease.metadata#>>'{remoteExecutionTermination,leaseId}' = lease.id::text
|
||||
and lease.metadata#>>'{remoteExecutionTermination,provider}' = lease.provider
|
||||
and lease.metadata#>>'{remoteExecutionTermination,providerLeaseId}' = lease.provider_lease_id
|
||||
and lease.metadata#>>'{remoteExecutionTermination,state}' in ('stopped', 'destroyed'))), false)))`;
|
||||
}
|
||||
|
||||
/**
|
||||
* A file-preservation hold binds a newer task owner only until automatic
|
||||
* recovery saved the stopped run's files. That owner's turn is a fresh
|
||||
* assignment, not a replay: the stopped run's unknown outcomes stay with its
|
||||
* own agent, which keeps the hold. A Stop fenced before any provider turn has
|
||||
* no unknown outcome, so its hold releases for the same agent too. The proof is
|
||||
* own agent, which keeps the hold. A Stop fenced before any provider turn, or a
|
||||
* shutdown that interrupted preparation before any provider work, has no
|
||||
* unknown outcome, so its hold releases for the same agent too. The proof is
|
||||
* the durable completed-recovery receipt bound to this action, so later reuse
|
||||
* of the workspace by the current owner (new epoch, writer or snapshot) cannot
|
||||
* re-arm the hold.
|
||||
@@ -217,7 +263,7 @@ export function restoreHoldReleasedForCurrentOwnerPredicate() {
|
||||
and task.execution_state->>'status' = 'pending'
|
||||
and task.execution_state->'currentParticipant'->>'type' = 'agent'
|
||||
and task.execution_state->'currentParticipant'->>'agentId' = stopped.agent_id::text, false))
|
||||
or ${stoppedBeforeProviderTurnCondition()}))`,
|
||||
or ${stoppedBeforeProviderTurnCondition()} or ${interruptedPreparationBeforeProviderWorkCondition()}))`,
|
||||
completedWorkspaceRecoveryCondition(sourceRunId),
|
||||
// Copyback put the files back into the task workspace that the new owner
|
||||
// reuses. A snapshot-only save needs an explicit fresh-workspace request
|
||||
|
||||
@@ -143,6 +143,101 @@ export async function recordUnidentifiedLocalRunnerStop(db: Db, run: Pick<typeof
|
||||
return true;
|
||||
}
|
||||
|
||||
export class UnidentifiedRunnerStopRefusedError extends Error {
|
||||
constructor(readonly code: string, message: string) {
|
||||
super(message);
|
||||
this.name = "UnidentifiedRunnerStopRefusedError";
|
||||
}
|
||||
}
|
||||
|
||||
type UnidentifiedLaunchRun = Pick<typeof heartbeatRuns.$inferSelect,
|
||||
"id" | "companyId" | "runtimeMode" | "status" | "finishedAt" | "processPid" | "processGroupId">;
|
||||
type LaunchRefusal = { code: string; message: string } | "already_stopped" | null;
|
||||
|
||||
export async function unidentifiedLocalLaunchRefusals(db: Db, runs: UnidentifiedLaunchRun[]): Promise<Map<string, LaunchRefusal>> {
|
||||
const result = new Map<string, LaunchRefusal>();
|
||||
const candidates: UnidentifiedLaunchRun[] = [];
|
||||
for (const run of runs) {
|
||||
if (run.runtimeMode !== "native" || !run.finishedAt || !FINISHED_RUN_STATUSES.has(run.status))
|
||||
result.set(run.id, { code: "run_not_finished", message: "Only a finished run's stop can be confirmed." });
|
||||
else if (run.processPid || run.processGroupId)
|
||||
result.set(run.id, { code: "process_identity_recorded", message: "This run recorded its process. Paperclip verifies that process itself." });
|
||||
else candidates.push(run);
|
||||
}
|
||||
if (!candidates.length) return result;
|
||||
const ids = candidates.map(run => run.id);
|
||||
const companyIds = [...new Set(candidates.map(run => run.companyId))];
|
||||
const latest = await db.selectDistinctOn([heartbeatRunEvents.runId], { runId: heartbeatRunEvents.runId,
|
||||
eventType: heartbeatRunEvents.eventType, payload: heartbeatRunEvents.payload }).from(heartbeatRunEvents).where(and(
|
||||
inArray(heartbeatRunEvents.companyId, companyIds), inArray(heartbeatRunEvents.runId, ids),
|
||||
isNull(heartbeatRunEvents.sourceEventId),
|
||||
inArray(heartbeatRunEvents.eventType, [PROCESS_START_REQUESTED, PROCESS_IDENTITY_RECORDED, LOCAL_PROCESS_STOPPED]),
|
||||
)).orderBy(heartbeatRunEvents.runId, desc(heartbeatRunEvents.seq));
|
||||
const remote = new Set((await db.select({ runId: environmentLeases.heartbeatRunId }).from(environmentLeases).where(and(
|
||||
inArray(environmentLeases.companyId, companyIds), inArray(environmentLeases.heartbeatRunId, ids),
|
||||
sql`${environmentLeases.provider} is distinct from 'local'`,
|
||||
))).map(row => row.runId));
|
||||
const latestByRun = new Map(latest.map(row => [row.runId, row]));
|
||||
for (const run of candidates) {
|
||||
const event = latestByRun.get(run.id);
|
||||
result.set(run.id, event?.eventType === LOCAL_PROCESS_STOPPED ? "already_stopped"
|
||||
: event?.eventType !== PROCESS_START_REQUESTED
|
||||
? { code: "no_unidentified_launch", message: "This run has no unconfirmed runner launch to stop." }
|
||||
: record(event.payload).hostname !== os.hostname()
|
||||
? { code: "launch_on_other_host", message: "The runner was launched on another host. Confirm the stop there." }
|
||||
: remote.has(run.id)
|
||||
? { code: "remote_environment", message: "This run used a remote environment. Its environment cleanup confirms the stop." }
|
||||
: null);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
export async function unidentifiedLocalLaunchRefusal(db: Db, run: UnidentifiedLaunchRun): Promise<LaunchRefusal> {
|
||||
return (await unidentifiedLocalLaunchRefusals(db, [run])).get(run.id) ?? null;
|
||||
}
|
||||
|
||||
export async function confirmUnidentifiedLocalRunnerStopped(db: Db, input: {
|
||||
companyId: string; issueId: string; runId: string; actorId: string;
|
||||
}): Promise<{ runId: string; alreadyStopped: boolean }> {
|
||||
const refuse = (code: string, message: string) => { throw new UnidentifiedRunnerStopRefusedError(code, message); };
|
||||
const [run] = await db.select().from(heartbeatRuns)
|
||||
.where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId)));
|
||||
if (!run || ![run.issueId, run.nativeIssueId, record(run.contextSnapshot).issueId].includes(input.issueId))
|
||||
return refuse("run_not_found", "This run does not belong to the task.");
|
||||
const refusal = await unidentifiedLocalLaunchRefusal(db, run);
|
||||
if (refusal === "already_stopped") return { runId: run.id, alreadyStopped: true };
|
||||
if (refusal) return refuse(refusal.code, refusal.message);
|
||||
const { noLocalRunnerProcessForRun } = await import("./native-runtime/native-session-executor.js");
|
||||
if (!await noLocalRunnerProcessForRun(run.id))
|
||||
return refuse("runner_still_running", "A runner for this run is still running on this host. Stop it first.");
|
||||
await db.transaction(async tx => {
|
||||
const [locked] = await tx.select().from(heartbeatRuns)
|
||||
.where(and(eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.id, run.id))).for("update");
|
||||
const current = locked ? await unidentifiedLocalLaunchRefusal(tx as unknown as Db, locked) : null;
|
||||
if (!locked || current === "already_stopped" || current !== null || locked.finishedAt?.getTime() !== run.finishedAt?.getTime())
|
||||
return refuse("run_changed", "This run changed while its stop was being confirmed. Check it again.");
|
||||
await appendHeartbeatRunEvent(tx as unknown as Db, {
|
||||
companyId: run.companyId,
|
||||
runId: run.id,
|
||||
agentId: run.agentId,
|
||||
eventType: LOCAL_PROCESS_STOPPED,
|
||||
stream: "system",
|
||||
level: "info",
|
||||
message: "A board operator confirmed that this run's runner and its agent processes have stopped.",
|
||||
payload: { processPid: null, processGroupId: null, processStartedAt: null, basis: "operator_attested",
|
||||
attestedByUserId: input.actorId },
|
||||
});
|
||||
await notifyDeliveryWork(tx as unknown as Db, DELIVERY_QUEUES.accounting, run.companyId);
|
||||
});
|
||||
return { runId: run.id, alreadyStopped: false };
|
||||
}
|
||||
|
||||
const FINISHED_RUN_STATUSES = new Set(["failed", "cancelled", "timed_out", "succeeded", "interrupted"]);
|
||||
|
||||
function record(value: unknown): Record<string, unknown> {
|
||||
return value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
|
||||
}
|
||||
|
||||
/** A native run that an admission fence refused before it requested any
|
||||
* runner launch (a quarantined prior-session cleanup is checked before the
|
||||
* provider starts) never had a process: there is nothing to stop. Any launch
|
||||
|
||||
@@ -2224,7 +2224,7 @@ export async function unidentifiedLocalRunnerProvenAbsent(input: { runId: string
|
||||
|
||||
/** `paperclip-runnerd` is always launched with `--run-id <runId>`. A failed or
|
||||
* slow process listing proves nothing, so it keeps the caller blocked. */
|
||||
async function noLocalRunnerProcessForRun(runId: string): Promise<boolean> {
|
||||
export async function noLocalRunnerProcessForRun(runId: string): Promise<boolean> {
|
||||
const listing = await new Promise<string | null>(resolve => {
|
||||
execFile("ps", ["-axww", "-o", "command="], { encoding: "utf8", timeout: 5_000, maxBuffer: 64 * 1024 * 1024 },
|
||||
(error, stdout) => resolve(error ? null : stdout));
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { and, desc, eq, inArray, isNull, sql, type SQL } from "drizzle-orm";
|
||||
import { unidentifiedLocalLaunchRefusals } from "./native-local-process-stop.js";
|
||||
import { agents, agentWakeupRequests, approvals, chatActions, companies, heartbeatRuns, issueApprovals, issueRecoveryActions,
|
||||
issues, issueTreeHoldMembers, issueTreeHolds, nativeRunFinalizations, projects, type Db } from "@paperclipai/db";
|
||||
import { unionAll } from "drizzle-orm/pg-core";
|
||||
@@ -36,7 +37,7 @@ type ProjectionIssue = Pick<typeof issues.$inferSelect, "id" | "companyId" | "st
|
||||
type ProjectionRun = Pick<typeof heartbeatRuns.$inferSelect,
|
||||
"id" | "status" | "scheduledRetryAt" | "errorCode" | "livenessState" | "resultJson"> &
|
||||
Partial<Pick<typeof heartbeatRuns.$inferSelect, "agentId" | "error" | "companyId" | "nativeIssueId" | "runtimeMode" | "startedAt"
|
||||
| "scheduledRetryReason" | "createdAt" | "finishedAt">> & {
|
||||
| "scheduledRetryReason" | "createdAt" | "finishedAt" | "processPid" | "processGroupId">> & {
|
||||
/** Due time of the native coordinator's own same-run resume, if one is pending. */
|
||||
nativeResumeAt?: Date | null;
|
||||
/** When the native coordinator saved this run's terminal failure, if it did. */
|
||||
@@ -63,6 +64,7 @@ type ProjectionState = {
|
||||
approval: boolean;
|
||||
/** A board-owned workspace-owner fence on this run that no automatic proof has released. */
|
||||
ownerStopRequired?: boolean;
|
||||
unidentifiedLaunchConfirmable?: boolean;
|
||||
/** The latest wake for a task with no run yet, when admission refused it. */
|
||||
admissionWait?: { id: string; status: string; reason: string | null; agentId: string };
|
||||
retryDependencies?: RetryDependencies;
|
||||
@@ -136,6 +138,9 @@ const retryWaitMessages: Record<string, string> = {
|
||||
retry_source_changed: "Retry cannot continue because its saved message changed. Send a new message to continue from saved work.",
|
||||
};
|
||||
|
||||
const UNIDENTIFIED_RUNNER_STOP_MESSAGE = "Paperclip can't confirm that the previous run stopped. The server restarted after it started the agent but before it recorded the agent's process. "
|
||||
+ "If no agent is still working on this task, a board operator can confirm that it stopped, and your Retry starts right away. Your message is saved.";
|
||||
|
||||
type RetryDependencies = { count: number; blocked: boolean; cancelled: boolean; savingFilesOnly: boolean };
|
||||
|
||||
function dependencyRetryWaitMessage(dependencies: RetryDependencies): string {
|
||||
@@ -266,6 +271,10 @@ export function projectTaskInterruption(issue: ProjectionIssue, input: Projectio
|
||||
if (retryWait?.sourceRunId === run.id && !retryWait.executionWaitReason && !retryWait.preparationStatus && dependencies?.count)
|
||||
return { ...base, intentId: retryWait.id, state: "working",
|
||||
retryWait: { reason: "dependency_blocked", message: dependencyRetryWaitMessage(dependencies) } };
|
||||
if (retryWait?.sourceRunId === run.id && retryWait.executionWaitReason === "process_identity_missing"
|
||||
&& input.unidentifiedLaunchConfirmable)
|
||||
return { ...base, responsibility: "user", state: "waiting_for_user", canRetry: false, userAction: "confirm_run_stopped",
|
||||
retryWait: { reason: "process_identity_missing", message: UNIDENTIFIED_RUNNER_STOP_MESSAGE } };
|
||||
if (retryWait?.sourceRunId === run.id && retryWait.executionWaitReason) {
|
||||
const reason = Object.hasOwn(retryWaitMessages, retryWait.executionWaitReason) ? retryWait.executionWaitReason : "execution_recovery";
|
||||
// Keep the latest input receipt so message submission can settle independently.
|
||||
@@ -311,6 +320,7 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
|
||||
db.select({ issueId: requestedIssueId, id: heartbeatRuns.id, status: heartbeatRuns.status,
|
||||
createdAt: heartbeatRuns.createdAt, finishedAt: heartbeatRuns.finishedAt, agentId: heartbeatRuns.agentId, companyId: heartbeatRuns.companyId,
|
||||
nativeIssueId: heartbeatRuns.nativeIssueId, runtimeMode: heartbeatRuns.runtimeMode, startedAt: heartbeatRuns.startedAt,
|
||||
processPid: heartbeatRuns.processPid, processGroupId: heartbeatRuns.processGroupId,
|
||||
agentAdapter: agents.adapterType,
|
||||
error: sql<string | null>`case when ${heartbeatRuns.errorCode} = 'setup_failed'
|
||||
and ${heartbeatRuns.error} = 'Project is outside this low-trust boundary.'
|
||||
@@ -518,6 +528,12 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
|
||||
const projectMap = new Map(projectRows.map(project => [project.id, project]));
|
||||
const pauses = new Set(pauseRows.map(row => row.issueId)), approvalsSet = new Set(approvalRows.map(row => row.issueId));
|
||||
const admissionWaitMap = byIssue(admissionWaits);
|
||||
const launchWaits = [...retryWaitMap].flatMap(([issueId, wait]) => {
|
||||
const run = runMap.get(issueId);
|
||||
return run && wait.sourceRunId === run.id && wait.executionWaitReason === "process_identity_missing" ? [{ issueId, run }] : [];
|
||||
});
|
||||
const launchRefusals = await unidentifiedLocalLaunchRefusals(db, launchWaits.map(entry => entry.run));
|
||||
const confirmableLaunches = new Set(launchWaits.filter(entry => launchRefusals.get(entry.run.id) === null).map(entry => entry.issueId));
|
||||
const now = Date.now();
|
||||
for (const issue of selected) {
|
||||
const required = requirements.get(issue.id);
|
||||
@@ -532,6 +548,7 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
|
||||
agent: agentMap.get(issue.assigneeAgentId ?? ""), project: projectMap.get(issue.projectId ?? ""),
|
||||
company: companyRows[0], paused: pauses.has(issue.id), approval: approvalsSet.has(issue.id),
|
||||
ownerStopRequired: ownerStopRequired.has(`${issue.id}:${facts.run?.id}`),
|
||||
unidentifiedLaunchConfirmable: confirmableLaunches.has(issue.id),
|
||||
admissionWait: admissionWaitMap.get(issue.id),
|
||||
retryDependencies: retryDependencyMap.get(issue.id),
|
||||
}, now);
|
||||
|
||||
Reference in new issue
Block a user