Files
PaperClipAI/server/src/services/task-interruption-presentation.ts
T
DottaandClaude Opus 5.5 4a2eb920cf fix(storage): wait out a store outage on a run's critical path instead of failing it (#20299)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A cloud instance keeps run logs and evidence in object storage, so a
run touches the store as soon as it starts
> - In storage chaos wave `w-20261109T145628-s9902` (ST1: 75–176 s
windows where every S3 call returns 503), only 1 of 8 tasks finished. A
run starting or resuming during an outage failed within a second as
`adapter_failed`, generic recovery burned its attempts seconds apart,
the task was escalated to the board, and the user saw the raw S3 error
> - This pull request treats a transient storage error as an
infrastructure wait: the run is retried on a minute-scale schedule,
rapid recovery is suppressed, the board shows it as working, and the run
error says storage was unavailable
> - The benefit is that a store outage of several minutes delays work
instead of failing and stranding it

## Linked Issues or Issue Description

Refs #19831. Storage chaos wave `w-20261109T145628-s9902` (master
`610b54f551`, `--storage s3`, ST1).

**Where the run fails.** At run start or resume, `run-output` `begin`
opens the run log: `getRunLogStore().begin` → object run log store →
object byte journal `claim` → S3 conditional PUT. The journal retries
only 3 times within about 300 ms, so a 503 escapes and fails the run as
`adapter_failed`. The async stack ends in the SDK middleware with no app
frame. `log_store` stays null on the failed run, which pins the claim.

**What followed.**
- `failRun` released the issue into generic immediate recovery. Recovery
retried 1–2 s apart inside the outage (t01: 14:58:33, :35, :59,
14:59:01), then stopped and opened a board-owned
`stranded_assigned_issue` action. These are the open recovery actions on
CHA-1, CHA-4, CHA-7 and CHA-8; a later Retry hit the outage again and
escalated the same way.
- The run error was the raw S3 text ("Injected storage outage").

**Slow store (ST2, `w-20261109T151946-s9903`).** With about 9.7 s of
added latency per call, the same claim hit the journal's 15 s
per-operation deadline (`AbortSignal.timeout(15_000)`). The run failed
after about 36 s with `AbortError: Request aborted` and burned 3 runs
the same way; 4 of 8 tasks needed a Retry.

t02/t03 (Retry parked with `process_identity_missing`) are a separate
native-recovery class and are handled in a follow-up.


**Daytona storage chaos ST1 (`w-20261109T161202-s9912`, 1/8):** with S3
returning 503 for 72 to 154 s at a time:
- Workspace restore failed at once (`remote_workspace_restore_failed:
step=snapshot_download; storage=provider_error`), three times in 40 s,
then `setup_failed`. It left an active recovery action behind.
- Runs reattached after a restart died within 2 s on the run-log claim,
then became reconciliation-required.
- Setup failed on other storage reads.
- 6 of 7 failures needed a UI Retry.

So the rule here is one class rule: a transient store error on a run's
critical path waits, it never fails the run.

## What Changed

- New `server/src/storage/storage-unavailable.ts`:
- `isStorageUnavailableError` reads the recorded storage failure
diagnostics first, then the SDK error shape. Unavailable means
429/500/502/503/504, the S3 throttling and timeout names, network codes
(`ECONNRESET`, `ECONNREFUSED`, `ETIMEDOUT`, `EPIPE`, `EAI_AGAIN`,
`ENOTFOUND`), or an `AbortError` whose cause is a `TimeoutError` (a
per-operation deadline). 403, 404 and 412 are not outages, and neither
is an abort caused by the run's own Stop.
- It defines the `storage_unavailable` code, the
`storage_unavailable_wait` retry reason, the retry policy (60 s × 10
attempts, about 10 minutes) and the user-facing message.
- `server/src/services/heartbeat/run-completion.ts`:
- `failRun` classifies such a failure as `storage_unavailable` with the
plain message.
- Unless native recovery owns the run, it schedules the storage wait
with `scheduleBoundedRetryForRun`, skips the interaction-continuation
retry, and releases the issue with `suppressImmediateRecovery`.
  - `failRunSetup` uses the same policy for this code.
- `server/src/services/heartbeat/retries.ts`: `storage_unavailable`
joins the transient infrastructure codes. It gets its own retry budget
and policy, like `runner_revision_mismatch`.
- `server/src/services/task-interruption-presentation.ts`:
`storage_unavailable_wait` is a resource wait, so the board shows the
task as working with no Retry while it waits.

- **One shared outage wait for the run's critical path**
(`server/src/storage/storage-unavailable.ts`):
- `retryThroughStorageOutage` retries a transient store failure with
backoff (2 s doubling to 30 s) until that storage operation's outage has
lasted 5 minutes, then gives the error back. The outermost wait of an
operation records its first failure, and nested and parallel waits
inside it share that start; no other request's success extends it.
Shared probes and shared reads run under their own scope, so one run's
Stop never cancels work another run waits on. A stop signal ends the
wait, and non-outage errors are never retried.
- Inside `waitOutStorageOutages`, every replayable S3 request waits
instead of failing: get, head, delete, buffered and file puts, and
multipart create, file parts and complete. A consumed upload stream is
never resent, and requests outside the scope keep failing fast.
- The scope covers run execution (`heartbeat.ts` `executeRun`: setup,
workspace restore, run-log claim and the run itself), plus
`loadWorkspaceSnapshotArchive` and `uploadWorkspaceSnapshot`, which
copy-back and recovery workers use too.
- The run-log claim retries as a whole, because the journal bounds each
operation with a 15 s deadline.
- Past the budget, the run takes the bounded `storage_unavailable` wait
below.
- The classifier follows the error's `cause` chain, so a restore failure
wrapping an outage counts. `failRunSetup` checks storage before restore
failures, so that case takes the storage wait instead of two quick
retries and an escalation (the R6 leak).
- `object-byte-journal.ts`: when its deadline expires, the journal
rethrows the recorded storage error instead of a bare `TimeoutError`.

## Verification

Red/green (each new test fails against master and passes with this PR):
- `run-completion.test.ts` "waits a minute and suppresses rapid recovery
when a run fails because storage is unavailable (storage chaos ST1)". An
S3-shaped 503 records `storage_unavailable` with the plain message (no
raw S3 text), schedules `STORAGE_UNAVAILABLE_RETRY`, releases with
`suppressImmediateRecovery: true`, and skips the continuation retry.
"keeps an ordinary adapter failure on the normal recovery path" guards
the default.
- `task-interruption-query.test.ts` "does not present an overdue
storage_unavailable_wait wait as a failed attempt": working, `canRetry:
false`.
- `storage-unavailable.test.ts`: which errors count as an outage and
which do not, including recorded diagnostics, a deadline-aborted call
(ST2), and a Stop-aborted call that must not count.

Static checks:
- `tsc --noEmit` passes for `server`.

Live verification: Phase 3 will rerun ST1 (seed 9902) with the MinIO
fault proxy after merge.

- `server/src/__tests__/storage-outage-wait.test.ts` runs against a
local S3 that answers 503 for 3 s:
- Inside the scope, a snapshot `getObject` and a run-log conditional PUT
succeed once the store is back.
  - Outside it, a request fails fast.
  - A stop ends the wait.
- Unit cases cover the budget, non-outage errors, and a
`RemoteWorkspaceRestoreError` wrapping an outage.
- `run-completion.test.ts`: a setup failure from a restore that wrapped
an outage takes the storage wait (red before).
- `object-byte-journal.test.ts`: a real `claim` whose head PUT hits its
deadline is classified as a storage outage (red before).

## Risks

- A storage outage now holds a task for up to about 10 minutes of
automatic waits instead of escalating within seconds. A longer outage
still ends in the existing board recovery.
- A run whose failure is misclassified as storage-unavailable waits
instead of using fast recovery. The classifier accepts only server-side
or network failures from the storage SDK, never 4xx answers.
- No schema change.

- A run can now wait until a storage outage has lasted 5 minutes before
it fails over to the bounded wait. It shows as working throughout, and
Stop ends the wait.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 18:15:48 -06:00

625 lines
48 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import { STORAGE_UNAVAILABLE_WAIT_RETRY_REASON } from "../storage/storage-unavailable.js";
import { and, desc, eq, inArray, isNull, sql, type SQL } from "drizzle-orm";
import { unidentifiedLocalLaunchRefusals } from "./native-local-process-stop.js";
import { agents, agentWakeupRequests, approvals, chatActions, companies, heartbeatRuns, issueApprovals, issueRecoveryActions,
issues, issueTreeHoldMembers, issueTreeHolds, nativeRunFinalizations, projects, type Db } from "@paperclipai/db";
import { unionAll } from "drizzle-orm/pg-core";
import { HEARTBEAT_RUN_STATUSES, readProviderFailure, type TaskInterruptionPresentation } from "@paperclipai/shared";
import { captureTaskInterruptionFailure } from "../sentry.js";
import { isAiAuthenticationBlocked } from "./ai-auth-failure.js";
import { readSystemRecovery } from "./system-recovery-state.js";
import { PROJECT_ACCESS_DENIED } from "./project-access-denial.js";
import { authorizationService } from "./authorization.js";
import { taskWorkspaceProjectRequirements } from "./task-workspace-access.js";
import { isAcknowledgedNativeStop } from "./acknowledged-native-stop.js";
import { createRunSecretRedactionRegistry } from "./run-secret-redaction.js";
import { instanceSettingsService } from "./instance-settings.js";
import { redactCurrentUserText } from "../log-redaction.js";
import { logger } from "../middleware/logger.js";
import { canonicalExecutionUuid } from "./execution-identity.js";
import { chatActionDeadline } from "./chat-action-work.js";
import { isQueuedDependencyWait } from "./run-cancellation.js";
import { AI_CONNECTION_BUSY_RETRY_REASON, AI_CONNECTION_POOL_WAIT_RETRY_REASON, RUNNER_REVISION_WAIT_RETRY_REASON,
WORKSPACE_BUSY_RETRY_REASON } from "../modules/run-dispatch/index.js";
// These scheduled runs wait for a shared resource; none records a failed
// attempt. Their dispatcher re-checks the resource, so the run can stay
// scheduled past its retry time. Retry cannot free the resource either.
const resourceWaitRetryReasons = new Set<string>([WORKSPACE_BUSY_RETRY_REASON, AI_CONNECTION_BUSY_RETRY_REASON,
AI_CONNECTION_POOL_WAIT_RETRY_REASON, RUNNER_REVISION_WAIT_RETRY_REASON, STORAGE_UNAVAILABLE_WAIT_RETRY_REASON]);
// A due same-run resume dispatches within seconds; past this, offer Retry.
const NATIVE_RESUME_DISPATCH_GRACE_MS = 2 * 60_000;
// A queued-message Interrupt waits for the stopped run's cleanup before its
// successor starts. Past this, a hand-off that never started offers Retry.
const QUEUED_INTERRUPT_HANDOFF_GRACE_MS = 5 * 60_000;
type ProjectionIssue = Pick<typeof issues.$inferSelect, "id" | "companyId" | "status" | "assigneeAgentId" | "projectId">;
type ProjectionRun = Pick<typeof heartbeatRuns.$inferSelect,
"id" | "status" | "scheduledRetryAt" | "errorCode" | "livenessState" | "resultJson"> &
Partial<Pick<typeof heartbeatRuns.$inferSelect, "agentId" | "error" | "companyId" | "nativeIssueId" | "runtimeMode" | "startedAt"
| "scheduledRetryReason" | "createdAt" | "finishedAt" | "processPid" | "processGroupId">> & {
/** Due time of the native coordinator's own same-run resume, if one is pending. */
nativeResumeAt?: Date | null;
/** When the native coordinator saved this run's terminal failure, if it did. */
nativeFailedAt?: Date | null;
};
type ProjectionWait = { id: string; requestedAt: Date; sourceRunId?: string | null; executionWaitReason?: string | null;
preparationStatus?: string | null; preparationRetryAt?: number | string | null; preparationUpdatedAt?: Date | null;
/** When the user pressed Interrupt to deliver this queued message now. */
queuedInterruptRequestedAt?: string | null };
type ProjectionState = {
run?: ProjectionRun;
action?: typeof issueRecoveryActions.$inferSelect;
wait?: ProjectionWait;
retryWait?: ProjectionWait;
/** The pending queued message this run was interrupted for, if any. */
interruptWait?: ProjectionWait;
agent?: Pick<typeof agents.$inferSelect, "status" | "pauseReason">;
projectAccessAllowed?: boolean;
projectAccessProjectId?: string | null;
workspaceSelectionAvailable?: boolean;
project?: Pick<typeof projects.$inferSelect, "pausedAt" | "pauseReason">;
company?: Pick<typeof companies.$inferSelect, "status" | "pauseReason">;
paused: boolean;
approval: boolean;
/** A board-owned workspace-owner fence on this run that no automatic proof has released. */
ownerStopRequired?: boolean;
unidentifiedLaunchConfirmable?: boolean;
/** The latest wake for a task with no run yet, when admission refused it. */
admissionWait?: { id: string; status: string; reason: string | null; agentId: string };
retryDependencies?: RetryDependencies;
};
// A task read reports the change that made a task fail, not every later read of
// retained history. The delivery cache is process-local, so a restart or another
// replica would otherwise resend each old failure whenever a list shows it.
const TASK_INTERRUPTION_REPORT_WINDOW_MS = 60 * 60_000;
/** Latest saved fact that can move a task into the failed state, if known. The
* recovery row's update time covers recovery that newly fails on an old run:
* each saved recovery transition stamps it, while activity heartbeats do not.
* The native resume due time and terminal finalization cover a same-run resume
* that lapses or is exhausted on an old run. */
export function taskInterruptionChangedAt(input: Pick<ProjectionState, "run" | "action" | "wait" | "retryWait">): number | null {
const { run, wait, action } = input;
const retryWait = input.retryWait ?? wait;
const preparationFailedAt = retryWait?.preparationStatus === "failed" || retryWait?.preparationStatus === "cancelled"
? retryWait.preparationUpdatedAt : null;
const times: unknown[] = [run?.finishedAt ?? run?.createdAt, run?.scheduledRetryAt, run?.nativeResumeAt, run?.nativeFailedAt,
wait?.requestedAt, retryWait?.requestedAt, action?.updatedAt, preparationFailedAt].map(value => value?.getTime());
times.push(readSystemRecovery(action?.evidence)?.nextAttemptAt,
retryWait?.preparationRetryAt == null ? null : Number(retryWait.preparationRetryAt));
const known = times.filter((time): time is number => typeof time === "number" && Number.isFinite(time));
return known.length ? Math.max(...known) : null;
}
function boundedJsonString(value: SQL, limit: number) {
return sql<string | null>`case when jsonb_typeof(${value}) = 'string' then left(${value} #>> '{}', ${limit}) else null end`;
}
/** Projected reason for a budget.blocked refusal caused by unpriced usage. */
export const UNPRICED_USAGE_WAIT = "accounting.unpriced_usage";
/** Admission's unpriced-usage refusal text (budgets.ts). Only this fixed
* phrase is matched; the stored message itself is never displayed. */
const unpricedUsageWait = (reason: SQL, message: SQL) =>
sql<string | null>`case when ${reason} = 'budget.blocked' and ${message} like '%recorded usage has no reliable price%'
then ${UNPRICED_USAGE_WAIT} else ${reason} end`;
// Persisted wait messages can contain diagnostics. Ordinary task views expose
// only these stable categories, never arbitrary payloads or source identifiers.
const retryWaitMessages: Record<string, string> = {
workspace_repair_required: "Retry is waiting for Paperclip to recover the saved files. Your message is saved.",
remote_cleanup: "Retry is waiting for the previous environment to stop. Your message is saved.",
local_cleanup: "Retry is waiting for the previous execution to finish cleanup. Your message is saved.",
cleanup_pending: "Retry is waiting for the previous execution to finish cleanup. Your message is saved.",
execution_settling: "Retry is waiting for the previous execution to stop. Your message is saved.",
controller_settling: "Retry is waiting for the previous execution to finish recovery. Your message is saved.",
process_identity_missing: "Retry is waiting for Paperclip to verify that the previous execution stopped. Your message is saved.",
process_running: "Retry is waiting for the previous execution to stop. Your message is saved.",
execution_active: "Retry is waiting for the current run to finish. Your message is saved.",
decision_pending: "Retry is waiting for a pending approval or question to be resolved. Your message is saved.",
"budget.blocked": "Retry is waiting for the budget to allow new work. Check Costs for unrecorded or unpriced usage.",
// Admission records an unpriced-usage refusal as budget.blocked; the
// projection shows it as the accounting matter it is.
[UNPRICED_USAGE_WAIT]: "Retry is waiting because recorded usage has no price. Check Costs, or allow unpriced usage in the budget policy. Your message is saved.",
finalizing_saved_result: "Paperclip is finishing the saved result before Retry can continue.",
// Another click cannot release a copyback owner that stopped without reporting.
// Retry starts a new recovery generation; the agent does not repeat its work.
saved_work_recovery_failed: "Saving the agent's files failed. Retry to try again; the agent will not repeat its work.",
saved_work_recovery_refused: "Paperclip refused to recover the saved files because a safety check on them cannot pass. "
+ "Retry runs the same check again and succeeds only after Paperclip is updated. The agent's result is saved, and the agent will not repeat its work.",
workspace_owner_stop_required: "Retry is waiting until Paperclip confirms that the previous server stopped saving files. The agent's result and your Retry are saved.",
cleanup_quarantined: "Retry cannot continue this attempt. Send a new message after the previous execution has stopped.",
cancelled_by_operator: "This attempt was stopped. Send a new message to continue from saved work.",
// A new click replaces a Retry declined for its requester's access with a
// fresh attempt under the clicker's access, so Retry stays available.
retry_authority_denied: "Retry could not start with the requesting user's access. Press Retry to start it with yours.",
// An edited saved message can never be replayed; only new input continues.
retry_source_changed: "Retry cannot continue because its saved message changed. Send a new message to continue from saved work.",
};
const UNIDENTIFIED_RUNNER_STOP_MESSAGE = "Paperclip can't confirm that the previous run stopped. The server restarted after it started the agent but before it recorded the agent's process. "
+ "If no agent is still working on this task, a board operator can confirm that it stopped, and your Retry starts right away. Your message is saved.";
type RetryDependencies = { count: number; blocked: boolean; cancelled: boolean; savingFilesOnly: boolean };
function dependencyRetryWaitMessage(dependencies: RetryDependencies): string {
if (dependencies.cancelled) return "Retry is waiting on a cancelled task this one depends on. Remove or replace that dependency to continue. Your Retry is saved.";
if (dependencies.blocked) return "Retry is waiting on a blocked task this one depends on. Retry or unblock it; this task continues when it is done. Your Retry is saved.";
if (dependencies.savingFilesOnly) return "Retry is waiting for a finished task this one depends on to save its files. Your Retry is saved.";
return "Retry is waiting for the tasks this one depends on to finish. Your Retry is saved.";
}
// A task that never started can wait for a temporary admission hold. The
// assigned-task liveness sweep admits it once the hold clears; no Retry needed.
const admissionWaitMessages: Record<string, string> = {
"budget.blocked": "Waiting for the budget to allow new work. The task starts automatically when it does.",
[UNPRICED_USAGE_WAIT]: "Waiting because recorded usage has no price. Check Costs, or allow unpriced usage in the budget policy.",
};
/** Recognize old persisted failures without displaying arbitrary run error text. */
function deniedProjectId(issue: ProjectionIssue, run?: ProjectionRun): string | null | undefined {
const historical = run?.errorCode === "setup_failed" && run.error === "Project is outside this low-trust boundary.";
if (!run || run.status !== "failed" || run.agentId !== issue.assigneeAgentId
|| (run.errorCode !== PROJECT_ACCESS_DENIED && !historical)) return undefined;
const detail = run.resultJson?.projectAccessDenied;
return detail && typeof detail === "object" && "projectId" in detail && typeof detail.projectId === "string"
? detail.projectId : null;
}
/** Presentation never grants authority. Dispatch repeats all gates under its
* locks. The same projection is used in detail, lists and attention surfaces. */
export function projectTaskInterruption(issue: ProjectionIssue, input: ProjectionState, now = Date.now()): TaskInterruptionPresentation | null {
const { run, wait, action, agent, project, company } = input;
const retryWait = input.retryWait ?? wait;
if (!run) {
const admission = input.admissionWait;
if (!admission || admission.status !== "skipped" || !Object.hasOwn(admissionWaitMessages, admission.reason ?? "")
|| issue.status !== "todo" || admission.agentId !== issue.assigneeAgentId) return null;
const waiting: TaskInterruptionPresentation = { intentId: admission.id, responsibility: "system", state: "working",
nextRetryAt: null, canRetry: false, userAction: null, sourceRunIds: [], observedRunId: null };
// Budget pauses, ordinary pauses and approvals wait on a person; only an
// accounting hold settles by itself. Admission records all of them as budget.blocked.
const userAction = !issue.assigneeAgentId ? "assign_agent" : input.approval ? "approval"
: [agent?.pauseReason, project?.pauseReason, company?.pauseReason].some(reason => reason === "budget" || reason === "budget_exceeded")
? "budget" : input.paused || project?.pausedAt || agent?.status === "paused" || company?.status === "paused" ? "resume" : null;
if (userAction) return { ...waiting, responsibility: "user", state: "waiting_for_user", userAction };
return { ...waiting, retryWait: { reason: admission.reason!, message: admissionWaitMessages[admission.reason!] } };
}
const system = readSystemRecovery(action?.evidence);
const base: TaskInterruptionPresentation = { intentId: wait?.id ?? null, responsibility: "system",
state: "resolved", nextRetryAt: null, canRetry: false, userAction: null,
sourceRunIds: [run.id], observedRunId: run.id };
if (["done", "cancelled"].includes(issue.status) || (run.status === "succeeded" && !wait)) return base;
const providerFailure = run.status === "failed" || run.status === "scheduled_retry"
? readProviderFailure(run.resultJson?.providerFailure) : null;
if (providerFailure) base.providerFailure = providerFailure;
if (["failed", "scheduled_retry"].includes(run.status) && run.errorCode === "workspace_validation_failed"
&& typeof run.error === "string" && run.error.trim()) base.workspaceFailure = { message: run.error };
const userAction = !issue.assigneeAgentId ? "assign_agent" : input.approval || run.errorCode === "native_provider_approval_required" ? "approval"
: isAiAuthenticationBlocked(run) ? "reconnect"
: run.errorCode === "native_provider_model_rejected" ? "choose_model"
: [agent?.pauseReason, project?.pauseReason, company?.pauseReason].some(reason => reason === "budget" || reason === "budget_exceeded")
? "budget" : input.paused || project?.pausedAt || agent?.status === "paused" || company?.status === "paused" ? "resume" : null;
if (userAction) return { ...base, responsibility: "user", state: "waiting_for_user", userAction };
// The previous controller cannot confirm its own stop. Name the one board
// action instead of showing the fenced run as working (RC-47).
if (input.ownerStopRequired) return { ...base, responsibility: "user", state: "waiting_for_user", userAction: "confirm_owner_stop" };
// The task's existing dependency UI owns this wait. Keep actual recovery and
// Retry preparation failures authoritative; this does not change admission.
if (!action && !retryWait?.preparationStatus && isQueuedDependencyWait(run)) return base;
// Intentional Stop is not an unsuccessful attempt. This is presentation only;
// physical containment and pending cleanup keep their execution-side gates.
if (!wait && run.runtimeMode === "native" && run.companyId === issue.companyId && run.nativeIssueId === issue.id
&& isAcknowledgedNativeStop({ ...run, companyId: run.companyId, nativeIssueId: run.nativeIssueId })) return base;
// A queued-message Interrupt stops this run to deliver that exact message.
// The stop is the user's intent and the message starts next, so show the
// hand-off as working, not as a failed attempt.
// Newer queued input can follow it, so match the recorded queue, not the latest wait.
const interruptWait = input.interruptWait ?? wait;
const interruptAt = Date.parse(interruptWait?.queuedInterruptRequestedAt ?? "");
if (interruptWait && run.status === "cancelled" && run.resultJson?.queuedCommentInterruptQueueId === interruptWait.id
&& Number.isFinite(interruptAt) && now - interruptAt < QUEUED_INTERRUPT_HANDOFF_GRACE_MS
&& !(retryWait?.sourceRunId === run.id && retryWait.preparationStatus) && system?.state !== "exhausted")
return { ...base, intentId: interruptWait.id, state: "working" };
if (["running", "queued"].includes(run.status)) return { ...base, state: "working" };
const deniedProject = deniedProjectId(issue, run);
let projectAccess: TaskInterruptionPresentation["projectAccess"];
if (deniedProject !== undefined) {
// Permissions cannot repair a stale or unavailable workspace selection.
if (input.workspaceSelectionAvailable === false) return { ...base, state: "failed" };
const ready = input.projectAccessAllowed === true;
const projectId = input.projectAccessProjectId === undefined ? deniedProject : input.projectAccessProjectId;
projectAccess = { agentId: run.agentId!, projectId: projectId === issue.projectId ? projectId : null, ready };
if (!ready) return { ...base, responsibility: "user", state: "waiting_for_user",
userAction: "project_access", projectAccess };
}
const recoveryStep: TaskInterruptionPresentation["recoveryStep"] = !system ? undefined
: system.phase === "verify_execution" ? "checking_previous_run"
: system.phase === "preserve_files" ? "saving_previous_files" : "finishing_recovery";
const activeObservation = action?.evidence.systemRecoveryActiveAt;
const recoveryRunning = system?.state === "running" && typeof activeObservation === "number"
&& activeObservation > now - 45_000;
const recoveryScheduled = Boolean(system?.nextAttemptAt && system.nextAttemptAt > now);
const preparationDeadline = retryWait?.preparationRetryAt == null ? NaN : Number(retryWait.preparationRetryAt);
// A pending Retry preparation is already a Retry; never offer another one.
const preparationPending = retryWait?.sourceRunId === run.id && Boolean(retryWait.preparationStatus)
&& ["issued", "processing"].includes(retryWait.preparationStatus!) && Number.isFinite(preparationDeadline);
if (retryWait?.sourceRunId === run.id && retryWait.preparationStatus) {
const deadline = preparationDeadline;
const pending = preparationPending;
// Saved-work recovery gates a pending preparation, so it names the wait.
// Failed or cancelled preparation stays authoritative.
if (pending && !recoveryRunning && !recoveryScheduled)
return { ...base, state: "retry_scheduled", nextRetryAt: new Date(deadline).toISOString() };
if (!pending) return { ...base, state: "failed", canRetry: retryWait.preparationStatus !== "cancelled" };
}
if (run.status === "scheduled_retry" && resourceWaitRetryReasons.has(run.scheduledRetryReason ?? "") && !system)
return { ...base, state: "working" };
// The native coordinator resumes this same run by itself. A manual Retry
// would race it, so show the resume until it is due and briefly while its
// dispatch runs. A resume that never ran falls through to one Retry.
const nativeResumeAt = run.nativeResumeAt?.getTime();
const retryAt = system?.nextAttemptAt ?? nativeResumeAt ?? run.scheduledRetryAt?.getTime();
if (run.runtimeMode === "native" && run.errorCode === "native_governed_wait_settlement_pending") {
const settling = recoveryRunning || Boolean(retryAt && now - retryAt < NATIVE_RESUME_DISPATCH_GRACE_MS);
return { ...base, state: settling ? "working" : "failed", canRetry: false,
retryWait: { reason: "governed_wait_settlement", message: settling
? "Your work is saved. Paperclip is finishing shutdown."
: "Paperclip couldn’t confirm shutdown. Your work is saved; continuation is paused." } };
}
if (retryAt && retryAt > now) return { ...base, state: "retry_scheduled",
nextRetryAt: new Date(retryAt).toISOString(), canRetry: !preparationPending && !nativeResumeAt,
...(system?.nextAttemptAt && recoveryStep ? { recoveryStep } : {}) };
if (nativeResumeAt && !system && now - nativeResumeAt < NATIVE_RESUME_DISPATCH_GRACE_MS)
return { ...base, state: "working" };
if (recoveryRunning) return { ...base, state: "working", ...(recoveryStep ? { recoveryStep } : {}) };
const dependencies = input.retryDependencies;
if (retryWait?.sourceRunId === run.id && !retryWait.executionWaitReason && !retryWait.preparationStatus && dependencies?.count)
return { ...base, intentId: retryWait.id, state: "working",
retryWait: { reason: "dependency_blocked", message: dependencyRetryWaitMessage(dependencies) } };
if (retryWait?.sourceRunId === run.id && retryWait.executionWaitReason === "process_identity_missing"
&& input.unidentifiedLaunchConfirmable)
return { ...base, responsibility: "user", state: "waiting_for_user", canRetry: false, userAction: "confirm_run_stopped",
retryWait: { reason: "process_identity_missing", message: UNIDENTIFIED_RUNNER_STOP_MESSAGE } };
if (retryWait?.sourceRunId === run.id && retryWait.executionWaitReason) {
const reason = Object.hasOwn(retryWaitMessages, retryWait.executionWaitReason) ? retryWait.executionWaitReason : "execution_recovery";
// Keep the latest input receipt so message submission can settle independently.
return { ...base, state: "failed",
canRetry: !["cleanup_quarantined", "cancelled_by_operator", "retry_source_changed", "workspace_owner_stop_required"].includes(reason), retryWait: { reason,
message: retryWaitMessages[reason] ?? "Retry is waiting for execution checks. Your message is saved." } };
}
if (wait && wait.requestedAt.getTime() > now - 10_000 && system?.state !== "exhausted")
return { ...base, state: "working" };
return { ...base, state: "failed", canRetry: true, ...(projectAccess ? { projectAccess } : {}) };
}
/** A scheduled retry has no outcome yet, so its provider failure is read from
* the predecessor it retries: the countdown keeps its reason.
* Batch list reads rather than doing seven queries per row. Keep run logs and
* large result bodies out of this projection. */
export async function taskInterruptionPresentations(db: Db, companyId: string, input: ProjectionIssue[]) {
const selected = input.filter(issue => issue.companyId === companyId);
const result = new Map<string, TaskInterruptionPresentation | null>();
if (!selected.length) return result;
const ids = selected.map(issue => issue.id);
const requested = sql`unnest(${sql.param([...new Set(ids)])}::text[]) as requested_interruption_issues(requested_issue_id)`;
const requestedIssueId = sql<string>`"requested_interruption_issues"."requested_issue_id"`;
const requestedUuid = sql<string>`case when ${requestedIssueId} ~ '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'
then ${requestedIssueId}::uuid else null end`;
const legacyIssue = sql<string>`${heartbeatRuns.contextSnapshot}->>'issueId'`;
const candidate = (condition: ReturnType<typeof and>) => db.select({ id: heartbeatRuns.id, createdAt: heartbeatRuns.createdAt })
.from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), condition))
.orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
const candidates = unionAll(
candidate(eq(heartbeatRuns.nativeIssueId, requestedUuid)),
candidate(and(isNull(heartbeatRuns.nativeIssueId), eq(heartbeatRuns.issueId, requestedUuid))),
candidate(and(isNull(heartbeatRuns.nativeIssueId), isNull(heartbeatRuns.issueId), eq(legacyIssue, requestedIssueId))),
).as("interruption_run_candidates");
const latestRuns = db.select({ id: candidates.id }).from(candidates)
.orderBy(desc(candidates.createdAt), desc(candidates.id)).limit(1).as("latest_interruption_runs");
const wakeIssue = sql<string>`${agentWakeupRequests.payload}->>'issueId'`;
const isRetryWait = sql<boolean>`coalesce(jsonb_typeof(${agentWakeupRequests.payload}#>'{taskRetry,sourceRunId}') = 'string', false)`;
const agentIds = selected.flatMap(issue => issue.assigneeAgentId ? [issue.assigneeAgentId] : []);
const projectIds = selected.flatMap(issue => issue.projectId ? [issue.projectId] : []);
const startingIssueIds = selected.filter(issue => issue.status === "todo" && issue.assigneeAgentId).map(issue => issue.id);
const [runs, waits, agentRows, projectRows, pauseRows, approvalRows, companyRows, admissionWaits] = await Promise.all([
db.select({ issueId: requestedIssueId, id: heartbeatRuns.id, status: heartbeatRuns.status,
createdAt: heartbeatRuns.createdAt, finishedAt: heartbeatRuns.finishedAt, agentId: heartbeatRuns.agentId, companyId: heartbeatRuns.companyId,
nativeIssueId: heartbeatRuns.nativeIssueId, runtimeMode: heartbeatRuns.runtimeMode, startedAt: heartbeatRuns.startedAt,
processPid: heartbeatRuns.processPid, processGroupId: heartbeatRuns.processGroupId,
agentAdapter: agents.adapterType,
error: sql<string | null>`case when ${heartbeatRuns.errorCode} = 'setup_failed'
and ${heartbeatRuns.error} = 'Project is outside this low-trust boundary.'
or ${heartbeatRuns.errorCode} = 'workspace_validation_failed' then ${heartbeatRuns.error} else null end`,
errorCode: heartbeatRuns.errorCode, livenessState: heartbeatRuns.livenessState, scheduledRetryAt: heartbeatRuns.scheduledRetryAt,
scheduledRetryReason: heartbeatRuns.scheduledRetryReason,
// Candidate only: the coordinator also pushes nextAttemptAt forward while
// it defers an ineligible run. The scheduler's own admission check below
// decides whether this resume is real.
nativeResumeAt: sql<Date | null>`(select ${nativeRunFinalizations.nextAttemptAt} from ${nativeRunFinalizations}
where ${nativeRunFinalizations.companyId} = ${heartbeatRuns.companyId} and ${nativeRunFinalizations.runId} = ${heartbeatRuns.id}
and ${heartbeatRuns.runtimeMode} = 'native' and ${heartbeatRuns.status} = 'failed'
and ${nativeRunFinalizations.resultId} is null and ${nativeRunFinalizations.phase} = 'retryable_failure'
and ${heartbeatRuns.contextSnapshot}->'nativeQuestionCancellation' is null
and ${heartbeatRuns.runnerProfileJson}->'legacyWorkspaceRecoveryMigration' is null)`.mapWith(value => value == null ? null : new Date(value)),
nativeFailedAt: sql<Date | null>`(select ${nativeRunFinalizations.updatedAt} from ${nativeRunFinalizations}
where ${nativeRunFinalizations.companyId} = ${heartbeatRuns.companyId} and ${nativeRunFinalizations.runId} = ${heartbeatRuns.id}
and ${nativeRunFinalizations.phase} = 'terminal_failure')`.mapWith(value => value == null ? null : new Date(value)),
// The run whose resultJson supplied providerFailure. A scheduled retry
// shows its predecessor's reason, so redaction must use that run's registry.
providerFailureRunId: sql<string>`case when ${heartbeatRuns.resultJson}->'providerFailure' is null
and ${heartbeatRuns.status} = 'scheduled_retry' and ${heartbeatRuns.retryOfRunId} is not null
then ${heartbeatRuns.retryOfRunId} else ${heartbeatRuns.id} end`,
resultJson: sql<Record<string, unknown>>`jsonb_build_object('configurationIncomplete', ${heartbeatRuns.resultJson}->'configurationIncomplete',
'stopReason', ${boundedJsonString(sql`${heartbeatRuns.resultJson}->'stopReason'`, 80)},
'timeoutSource', ${boundedJsonString(sql`${heartbeatRuns.resultJson}->'timeoutSource'`, 80)},
'timeoutFired', coalesce(${heartbeatRuns.resultJson}->'timeoutFired' = 'true'::jsonb, false),
'projectAccessDenied', ${heartbeatRuns.resultJson}->'projectAccessDenied',
'cancelledByActorType', ${boundedJsonString(sql`${heartbeatRuns.resultJson}->'cancelledByActorType'`, 32)},
'cancelledByUserId', ${boundedJsonString(sql`${heartbeatRuns.resultJson}->'cancelledByUserId'`, 160)},
'queuedCommentInterruptQueueId', ${boundedJsonString(sql`${heartbeatRuns.resultJson}->'queuedCommentInterruptQueueId'`, 160)},
'nativeCancellation', jsonb_build_object(
'schema', ${boundedJsonString(sql`${heartbeatRuns.resultJson}->'nativeCancellation'->'schema'`, 64)},
'runId', ${boundedJsonString(sql`${heartbeatRuns.resultJson}->'nativeCancellation'->'runId'`, 160)},
'companyId', ${boundedJsonString(sql`${heartbeatRuns.resultJson}->'nativeCancellation'->'companyId'`, 160)},
'issueId', ${boundedJsonString(sql`${heartbeatRuns.resultJson}->'nativeCancellation'->'issueId'`, 160)},
'scope', ${boundedJsonString(sql`${heartbeatRuns.resultJson}->'nativeCancellation'->'scope'`, 32)},
'reasonCode', ${boundedJsonString(sql`${heartbeatRuns.resultJson}->'nativeCancellation'->'reasonCode'`, 64)},
'dispatchState', ${boundedJsonString(sql`${heartbeatRuns.resultJson}->'nativeCancellation'->'dispatchState'`, 32)},
'dispatched', coalesce(${heartbeatRuns.resultJson}->'nativeCancellation'->'dispatched' = 'true'::jsonb, false),
'intentAuditId', ${boundedJsonString(sql`${heartbeatRuns.resultJson}->'nativeCancellation'->'intentAuditId'`, 160)},
'acknowledgementAuditId', ${boundedJsonString(sql`${heartbeatRuns.resultJson}->'nativeCancellation'->'acknowledgementAuditId'`, 160)}),
'providerFailure', coalesce(${heartbeatRuns.resultJson}->'providerFailure', case when ${heartbeatRuns.status} = 'scheduled_retry'
then (select predecessor.result_json->'providerFailure' from ${heartbeatRuns} predecessor
where predecessor.company_id = ${heartbeatRuns.companyId} and predecessor.id = ${heartbeatRuns.retryOfRunId}) end))`,
}).from(requested).innerJoinLateral(latestRuns, sql`true`).innerJoin(heartbeatRuns, and(eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, latestRuns.id)))
.leftJoin(agents, and(eq(agents.companyId, heartbeatRuns.companyId), eq(agents.id, heartbeatRuns.agentId)))
.orderBy(requestedIssueId),
// Keep at most two candidates per task: latest Retry and latest other input.
// A new message must not hide an existing Retry's admission explanation.
db.selectDistinctOn([wakeIssue, isRetryWait], { issueId: wakeIssue, id: agentWakeupRequests.id, requestedAt: agentWakeupRequests.requestedAt,
sourceRunId: boundedJsonString(sql`${agentWakeupRequests.payload}#>'{taskRetry,sourceRunId}'`, 160),
executionWaitReason: boundedJsonString(sql`to_jsonb(${unpricedUsageWait(sql`${agentWakeupRequests.payload}#>>'{executionWait,reason}'`,
sql`${agentWakeupRequests.payload}#>>'{executionWait,message}'`)})`, 80),
preparationStatus: chatActions.status, preparationRetryAt: chatActionDeadline("failed_run_retry"),
preparationUpdatedAt: chatActions.updatedAt })
.from(agentWakeupRequests).leftJoin(chatActions, and(eq(chatActions.companyId, agentWakeupRequests.companyId),
eq(chatActions.id, canonicalExecutionUuid(sql`${agentWakeupRequests.payload}#>>'{taskRetry,chatActionId}'`)),
eq(chatActions.kind, "failed_run_retry"), sql`${chatActions.payload}->>'taskRetryIntentId' = ${agentWakeupRequests.id}::text`,
sql`${chatActions.payload}->>'issueId' = ${wakeIssue}`,
sql`${chatActions.payload}->>'agentId' = ${agentWakeupRequests.agentId}::text`,
sql`${chatActions.payload}->>'failedRunId' = ${agentWakeupRequests.payload}#>>'{taskRetry,sourceRunId}'`,
inArray(chatActions.status, ["issued", "processing", "failed", "cancelled"])))
.where(and(eq(agentWakeupRequests.companyId, companyId), inArray(wakeIssue, ids),
eq(agentWakeupRequests.status, "deferred_issue_execution"), eq(agentWakeupRequests.requestedByActorType, "user")))
.orderBy(wakeIssue, isRetryWait, desc(agentWakeupRequests.requestedAt), desc(agentWakeupRequests.id)),
agentIds.length ? db.select({ id: agents.id, status: agents.status, pauseReason: agents.pauseReason }).from(agents)
.where(and(eq(agents.companyId, companyId), inArray(agents.id, agentIds))) : [],
projectIds.length ? db.select({ id: projects.id, pausedAt: projects.pausedAt, pauseReason: projects.pauseReason }).from(projects)
.where(and(eq(projects.companyId, companyId), inArray(projects.id, projectIds))) : [],
db.select({ issueId: issueTreeHoldMembers.issueId }).from(issueTreeHoldMembers).innerJoin(issueTreeHolds, and(
eq(issueTreeHolds.companyId, issueTreeHoldMembers.companyId), eq(issueTreeHolds.id, issueTreeHoldMembers.holdId)))
.where(and(eq(issueTreeHoldMembers.companyId, companyId), inArray(issueTreeHoldMembers.issueId, ids),
eq(issueTreeHolds.status, "active"), eq(issueTreeHolds.mode, "pause"))),
db.select({ issueId: issueApprovals.issueId }).from(issueApprovals).innerJoin(approvals, and(
eq(approvals.companyId, issueApprovals.companyId), eq(approvals.id, issueApprovals.approvalId)))
.where(and(eq(issueApprovals.companyId, companyId), inArray(issueApprovals.issueId, ids),
inArray(approvals.status, ["pending", "revision_requested"]))),
db.select({ status: companies.status, pauseReason: companies.pauseReason }).from(companies).where(eq(companies.id, companyId)),
// Only consulted for tasks without a run: the latest wake says whether admission refused it.
startingIssueIds.length ? db.selectDistinctOn([wakeIssue], { issueId: wakeIssue, id: agentWakeupRequests.id,
status: agentWakeupRequests.status, reason: unpricedUsageWait(sql`${agentWakeupRequests.reason}`, sql`${agentWakeupRequests.error}`),
agentId: agentWakeupRequests.agentId })
.from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, companyId), inArray(wakeIssue, startingIssueIds)))
.orderBy(wakeIssue, desc(agentWakeupRequests.requestedAt), desc(agentWakeupRequests.id)) : [],
]);
// Advertise a pending resume only when the scheduler would admit it, using
// its own check, so the task never hides Retry behind a deferred candidate
// and never offers Retry beside a resume that will run. Rare rows only.
// Candidates are failed native runs inside an active coordinator retry, so
// there are few. Their rows load in three batched reads; admission itself
// runs a few bounded reads per candidate, four at a time.
const resumeCandidates = runs.filter(run => run.nativeResumeAt);
if (resumeCandidates.length) {
const { readNativeSessionResumptionAgent } = await import("./native-runtime/native-session-resumption-admission.js");
const candidateIds = resumeCandidates.map(run => run.id);
const [fullRuns, coordinators] = await Promise.all([
db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), inArray(heartbeatRuns.id, candidateIds))),
db.select().from(nativeRunFinalizations).where(and(eq(nativeRunFinalizations.companyId, companyId),
inArray(nativeRunFinalizations.runId, candidateIds))),
]);
const issueIds = [...new Set(fullRuns.flatMap(run => run.nativeIssueId ? [run.nativeIssueId] : []))];
const issueRows = issueIds.length ? await db.select().from(issues).where(and(eq(issues.companyId, companyId),
inArray(issues.id, issueIds))) : [];
const runById = new Map(fullRuns.map(run => [run.id, run]));
const coordinatorByRun = new Map(coordinators.map(row => [row.runId, row]));
const issueById = new Map(issueRows.map(row => [row.id, row]));
for (let start = 0; start < resumeCandidates.length; start += 4) {
await Promise.all(resumeCandidates.slice(start, start + 4).map(async candidate => {
const run = runById.get(candidate.id);
const issue = run?.nativeIssueId ? issueById.get(run.nativeIssueId) : undefined;
const coordinator = coordinatorByRun.get(candidate.id);
const admitted = run && issue && coordinator
&& await readNativeSessionResumptionAgent(db, { issue, run, coordinator }).catch(() => null);
if (!admitted) candidate.nativeResumeAt = null;
}));
}
}
// A queued-message Interrupt records its queue on the stopped run. Newer
// input can hide that wake from the latest-wait read, so load it by id.
const interruptQueueId = (run: typeof runs[number]) => {
const id = run.status === "cancelled" ? run.resultJson?.queuedCommentInterruptQueueId : null;
return typeof id === "string" && /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/.test(id) ? id : null;
};
const interruptQueueIds = runs.flatMap(run => interruptQueueId(run) ?? []);
const interruptWaits = interruptQueueIds.length ? await db.select({ issueId: wakeIssue, id: agentWakeupRequests.id,
requestedAt: agentWakeupRequests.requestedAt,
queuedInterruptRequestedAt: boundedJsonString(sql`${agentWakeupRequests.payload}#>'{queuedCommentInterrupt,requestedAt}'`, 40) })
.from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, companyId), inArray(agentWakeupRequests.id, interruptQueueIds),
eq(agentWakeupRequests.status, "deferred_issue_execution"), eq(agentWakeupRequests.requestedByActorType, "user"))) : [];
const interruptWaitById = new Map(interruptWaits.map(wait => [wait.id, wait]));
const runIds = runs.map(run => run.id);
const actions = runIds.length ? await db.selectDistinctOn([issueRecoveryActions.sourceIssueId]).from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, companyId), inArray(issueRecoveryActions.sourceIssueId, ids),
inArray(sql`${issueRecoveryActions.evidence}->>'runId'`, runIds),
sql`${issueRecoveryActions.evidence} ? 'systemRecovery'`,
)).orderBy(issueRecoveryActions.sourceIssueId, desc(issueRecoveryActions.createdAt)) : [];
const ownerStops = runIds.length ? await db.select({ issueId: issueRecoveryActions.sourceIssueId,
runId: sql<string>`${issueRecoveryActions.evidence}->>'runId'` }).from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, companyId), inArray(issueRecoveryActions.sourceIssueId, ids),
eq(issueRecoveryActions.cause, "native_workspace_finalization_owner_unverified"), eq(issueRecoveryActions.ownerType, "board"),
inArray(issueRecoveryActions.status, ["active", "escalated"]),
inArray(sql`${issueRecoveryActions.evidence}->>'runId'`, runIds),
sql`${issueRecoveryActions.evidence}->'workspaceOwnerStop' is null`,
sql`coalesce(${issueRecoveryActions.evidence}#>>'{workspaceOwnerRecovery,state}', '') <> 'resuming'`,
)) : [];
const ownerStopRequired = new Set(ownerStops.map(row => `${row.issueId}:${row.runId}`));
const byIssue = <T extends { issueId: string }>(rows: T[]) => new Map(rows.map(row => [row.issueId, row]));
const runMap = byIssue(runs), waitMap = new Map<string, typeof waits[number]>();
// A Retry belongs to the run it observed. New conversation work supersedes
// that authority, even before dispatch retires the old receipt. Retained
// ordinary input is independent and must still be shown as waiting.
const currentWaits = waits.filter(wait => !wait.sourceRunId || wait.sourceRunId === runMap.get(wait.issueId)?.id);
const retryWaitMap = byIssue(currentWaits.filter(wait => wait.sourceRunId));
const retryIssueIds = [...retryWaitMap.keys()];
const retryDependencyMap = new Map<string, RetryDependencies>();
if (retryIssueIds.length) {
const { issueService } = await import("./issues.js");
const readiness = await issueService(db).listDependencyReadiness(companyId, retryIssueIds);
const blockerIds = [...new Set([...readiness.values()].flatMap(row => row.unresolvedBlockerIssueIds))];
const statusRows = blockerIds.length ? await db.select({ id: issues.id, status: issues.status }).from(issues)
.where(and(eq(issues.companyId, companyId), inArray(issues.id, blockerIds))) : [];
const statusById = new Map(statusRows.map(row => [row.id, row.status]));
for (const [waitingIssueId, row] of readiness) {
const ids = [...new Set(row.unresolvedBlockerIssueIds)];
if (!ids.length) continue;
const statuses = ids.map(id => statusById.get(id));
const saving = new Set(row.pendingFinalizeBlockerIssueIds);
retryDependencyMap.set(waitingIssueId, { count: ids.length,
blocked: statuses.includes("blocked"), cancelled: statuses.includes("cancelled"),
savingFilesOnly: ids.every(id => saving.has(id)) });
}
}
for (const wait of currentWaits) {
const current = waitMap.get(wait.issueId);
if (!current || wait.requestedAt > current.requestedAt ||
(wait.requestedAt.getTime() === current.requestedAt.getTime() && wait.id > current.id)) waitMap.set(wait.issueId, wait);
}
// Recheck current authority, including standard trust, root-issue grants and
// project policy. Do not reuse the failed run's frozen policy for a new retry.
// Only denied tasks need these reads; deduplicate projects and batch per agent.
// A task's organizational project can differ from its retained source. Old
// failures did not record the denied project, and even typed failures can
// become obsolete after a workspace selection changes.
const deniedIssues = selected.filter(issue => issue.assigneeAgentId && !["done", "cancelled"].includes(issue.status)
&& deniedProjectId(issue, runMap.get(issue.id)) !== undefined);
const requirements = await taskWorkspaceProjectRequirements(db, companyId, deniedIssues.map(issue => issue.id));
const deniedByAgent = new Map<string, Set<string>>();
for (const issue of deniedIssues) {
const ids = deniedByAgent.get(issue.assigneeAgentId!) ?? new Set<string>();
for (const projectId of requirements.get(issue.id)?.projectIds ?? []) ids.add(projectId);
if (ids.size) deniedByAgent.set(issue.assigneeAgentId!, ids);
}
const currentAccess = new Map<string, boolean>();
const authorization = authorizationService(db);
for (const [agentId, ids] of deniedByAgent) {
const projectIds = [...ids];
const decisions = await authorization.decideReads({ type: "agent", agentId, companyId, source: "agent_key" },
projectIds.map(projectId => ({ type: "project" as const, companyId, projectId })));
projectIds.forEach((projectId, index) => currentAccess.set(`${agentId}:${projectId}`, decisions[index]?.allowed === true));
}
const actionMap = new Map(actions.map(action => [action.sourceIssueId, action]));
const interruptWaitFor = (issueId: string) => {
const run = runMap.get(issueId);
const wait = run ? interruptWaitById.get(interruptQueueId(run) ?? "") : undefined;
return wait?.issueId === issueId ? wait : undefined;
};
const agentMap = new Map(agentRows.map(agent => [agent.id, agent]));
const projectMap = new Map(projectRows.map(project => [project.id, project]));
const pauses = new Set(pauseRows.map(row => row.issueId)), approvalsSet = new Set(approvalRows.map(row => row.issueId));
const admissionWaitMap = byIssue(admissionWaits);
const launchWaits = [...retryWaitMap].flatMap(([issueId, wait]) => {
const run = runMap.get(issueId);
return run && wait.sourceRunId === run.id && wait.executionWaitReason === "process_identity_missing" ? [{ issueId, run }] : [];
});
const launchRefusals = await unidentifiedLocalLaunchRefusals(db, launchWaits.map(entry => entry.run));
const confirmableLaunches = new Set(launchWaits.filter(entry => launchRefusals.get(entry.run.id) === null).map(entry => entry.issueId));
const now = Date.now();
for (const issue of selected) {
const required = requirements.get(issue.id);
const denied = required?.projectIds.filter(projectId => currentAccess.get(`${issue.assigneeAgentId}:${projectId}`) !== true);
const facts = { run: runMap.get(issue.id), action: actionMap.get(issue.id), wait: waitMap.get(issue.id),
retryWait: retryWaitMap.get(issue.id) };
const presentation = projectTaskInterruption(issue, { ...facts,
interruptWait: interruptWaitFor(issue.id),
projectAccessAllowed: denied?.length === 0,
projectAccessProjectId: denied?.includes(issue.projectId ?? "") ? issue.projectId : null,
workspaceSelectionAvailable: required?.available ?? false,
agent: agentMap.get(issue.assigneeAgentId ?? ""), project: projectMap.get(issue.projectId ?? ""),
company: companyRows[0], paused: pauses.has(issue.id), approval: approvalsSet.has(issue.id),
ownerStopRequired: ownerStopRequired.has(`${issue.id}:${facts.run?.id}`),
unidentifiedLaunchConfirmable: confirmableLaunches.has(issue.id),
admissionWait: admissionWaitMap.get(issue.id),
retryDependencies: retryDependencyMap.get(issue.id),
}, now);
result.set(issue.id, presentation);
const run = facts.run;
const runStatus = HEARTBEAT_RUN_STATUSES.find(status => status === run?.status);
const changedAt = taskInterruptionChangedAt(facts);
if (presentation?.responsibility === "system" && presentation.state === "failed" && run && runStatus
&& (changedAt === null || changedAt > now - TASK_INTERRUPTION_REPORT_WINDOW_MS)) {
captureTaskInterruptionFailure({ companyId, taskId: issue.id, agentId: run.agentId,
runId: run.id, runStatus, intentId: presentation.intentId, runErrorCode: run.errorCode ?? null,
agentAdapter: run.agentAdapter ?? "unknown" });
}
}
// Redact each explanation with the registry of the run that recorded it: the
// workspace failure is the observed run's, while a scheduled retry's provider
// failure may be its predecessor's.
const disclosures = [...result.entries()].flatMap(([issueId, presentation]) =>
(presentation?.providerFailure || presentation?.workspaceFailure) && presentation.observedRunId
? [{ id: presentation.observedRunId, providerFailureRunId: runMap.get(issueId)?.providerFailureRunId ?? presentation.observedRunId,
providerFailure: presentation.providerFailure, workspaceFailure: presentation.workspaceFailure, presentation }] : []);
if (disclosures.length) {
try {
const [redacted, settings] = await Promise.all([
createRunSecretRedactionRegistry(db).redactForRuns(companyId, disclosures.flatMap(({ id, providerFailureRunId, providerFailure, workspaceFailure }) => [
{ id: providerFailureRunId, providerFailure, workspaceFailure: undefined },
{ id, providerFailure: undefined, workspaceFailure },
])),
instanceSettingsService(db).getGeneral(),
]);
const safe = disclosures.map((_, index) => ({ providerFailure: redacted[2 * index]?.providerFailure,
workspaceFailure: redacted[2 * index + 1]?.workspaceFailure }));
const safeMessage = (value: unknown) => typeof value === "string"
? [...redactCurrentUserText(value, { enabled: settings.censorUsernameInLogs })].slice(0, 4000).join("") : "";
disclosures.forEach(({ presentation }, index) => {
if (presentation.providerFailure) {
const value = safe[index]?.providerFailure;
const failure = readProviderFailure({ ...value, message: safeMessage(value?.message) });
if (failure) presentation.providerFailure = failure;
else delete presentation.providerFailure;
}
if (presentation.workspaceFailure) {
const message = safeMessage(safe[index]?.workspaceFailure?.message);
if (message.trim()) presentation.workspaceFailure = { message };
else delete presentation.workspaceFailure;
}
});
} catch {
for (const { presentation } of disclosures) {
delete presentation.providerFailure;
delete presentation.workspaceFailure;
}
logger.warn({ companyId, runCount: disclosures.length }, "Could not redact run failure display; explanations omitted");
}
}
return result;
}
export async function taskInterruptionPresentation(db: Db, issue: ProjectionIssue) {
return (await taskInterruptionPresentations(db, issue.companyId, [issue])).get(issue.id) ?? null;
}