mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 06:10:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A cloud instance keeps run logs and evidence in object storage, so a run touches the store as soon as it starts > - In storage chaos wave `w-20261109T145628-s9902` (ST1: 75–176 s windows where every S3 call returns 503), only 1 of 8 tasks finished. A run starting or resuming during an outage failed within a second as `adapter_failed`, generic recovery burned its attempts seconds apart, the task was escalated to the board, and the user saw the raw S3 error > - This pull request treats a transient storage error as an infrastructure wait: the run is retried on a minute-scale schedule, rapid recovery is suppressed, the board shows it as working, and the run error says storage was unavailable > - The benefit is that a store outage of several minutes delays work instead of failing and stranding it ## Linked Issues or Issue Description Refs #19831. Storage chaos wave `w-20261109T145628-s9902` (master `610b54f551`, `--storage s3`, ST1). **Where the run fails.** At run start or resume, `run-output` `begin` opens the run log: `getRunLogStore().begin` → object run log store → object byte journal `claim` → S3 conditional PUT. The journal retries only 3 times within about 300 ms, so a 503 escapes and fails the run as `adapter_failed`. The async stack ends in the SDK middleware with no app frame. `log_store` stays null on the failed run, which pins the claim. **What followed.** - `failRun` released the issue into generic immediate recovery. Recovery retried 1–2 s apart inside the outage (t01: 14:58:33, :35, :59, 14:59:01), then stopped and opened a board-owned `stranded_assigned_issue` action. These are the open recovery actions on CHA-1, CHA-4, CHA-7 and CHA-8; a later Retry hit the outage again and escalated the same way. - The run error was the raw S3 text ("Injected storage outage"). **Slow store (ST2, `w-20261109T151946-s9903`).** With about 9.7 s of added latency per call, the same claim hit the journal's 15 s per-operation deadline (`AbortSignal.timeout(15_000)`). The run failed after about 36 s with `AbortError: Request aborted` and burned 3 runs the same way; 4 of 8 tasks needed a Retry. t02/t03 (Retry parked with `process_identity_missing`) are a separate native-recovery class and are handled in a follow-up. **Daytona storage chaos ST1 (`w-20261109T161202-s9912`, 1/8):** with S3 returning 503 for 72 to 154 s at a time: - Workspace restore failed at once (`remote_workspace_restore_failed: step=snapshot_download; storage=provider_error`), three times in 40 s, then `setup_failed`. It left an active recovery action behind. - Runs reattached after a restart died within 2 s on the run-log claim, then became reconciliation-required. - Setup failed on other storage reads. - 6 of 7 failures needed a UI Retry. So the rule here is one class rule: a transient store error on a run's critical path waits, it never fails the run. ## What Changed - New `server/src/storage/storage-unavailable.ts`: - `isStorageUnavailableError` reads the recorded storage failure diagnostics first, then the SDK error shape. Unavailable means 429/500/502/503/504, the S3 throttling and timeout names, network codes (`ECONNRESET`, `ECONNREFUSED`, `ETIMEDOUT`, `EPIPE`, `EAI_AGAIN`, `ENOTFOUND`), or an `AbortError` whose cause is a `TimeoutError` (a per-operation deadline). 403, 404 and 412 are not outages, and neither is an abort caused by the run's own Stop. - It defines the `storage_unavailable` code, the `storage_unavailable_wait` retry reason, the retry policy (60 s × 10 attempts, about 10 minutes) and the user-facing message. - `server/src/services/heartbeat/run-completion.ts`: - `failRun` classifies such a failure as `storage_unavailable` with the plain message. - Unless native recovery owns the run, it schedules the storage wait with `scheduleBoundedRetryForRun`, skips the interaction-continuation retry, and releases the issue with `suppressImmediateRecovery`. - `failRunSetup` uses the same policy for this code. - `server/src/services/heartbeat/retries.ts`: `storage_unavailable` joins the transient infrastructure codes. It gets its own retry budget and policy, like `runner_revision_mismatch`. - `server/src/services/task-interruption-presentation.ts`: `storage_unavailable_wait` is a resource wait, so the board shows the task as working with no Retry while it waits. - **One shared outage wait for the run's critical path** (`server/src/storage/storage-unavailable.ts`): - `retryThroughStorageOutage` retries a transient store failure with backoff (2 s doubling to 30 s) until that storage operation's outage has lasted 5 minutes, then gives the error back. The outermost wait of an operation records its first failure, and nested and parallel waits inside it share that start; no other request's success extends it. Shared probes and shared reads run under their own scope, so one run's Stop never cancels work another run waits on. A stop signal ends the wait, and non-outage errors are never retried. - Inside `waitOutStorageOutages`, every replayable S3 request waits instead of failing: get, head, delete, buffered and file puts, and multipart create, file parts and complete. A consumed upload stream is never resent, and requests outside the scope keep failing fast. - The scope covers run execution (`heartbeat.ts` `executeRun`: setup, workspace restore, run-log claim and the run itself), plus `loadWorkspaceSnapshotArchive` and `uploadWorkspaceSnapshot`, which copy-back and recovery workers use too. - The run-log claim retries as a whole, because the journal bounds each operation with a 15 s deadline. - Past the budget, the run takes the bounded `storage_unavailable` wait below. - The classifier follows the error's `cause` chain, so a restore failure wrapping an outage counts. `failRunSetup` checks storage before restore failures, so that case takes the storage wait instead of two quick retries and an escalation (the R6 leak). - `object-byte-journal.ts`: when its deadline expires, the journal rethrows the recorded storage error instead of a bare `TimeoutError`. ## Verification Red/green (each new test fails against master and passes with this PR): - `run-completion.test.ts` "waits a minute and suppresses rapid recovery when a run fails because storage is unavailable (storage chaos ST1)". An S3-shaped 503 records `storage_unavailable` with the plain message (no raw S3 text), schedules `STORAGE_UNAVAILABLE_RETRY`, releases with `suppressImmediateRecovery: true`, and skips the continuation retry. "keeps an ordinary adapter failure on the normal recovery path" guards the default. - `task-interruption-query.test.ts` "does not present an overdue storage_unavailable_wait wait as a failed attempt": working, `canRetry: false`. - `storage-unavailable.test.ts`: which errors count as an outage and which do not, including recorded diagnostics, a deadline-aborted call (ST2), and a Stop-aborted call that must not count. Static checks: - `tsc --noEmit` passes for `server`. Live verification: Phase 3 will rerun ST1 (seed 9902) with the MinIO fault proxy after merge. - `server/src/__tests__/storage-outage-wait.test.ts` runs against a local S3 that answers 503 for 3 s: - Inside the scope, a snapshot `getObject` and a run-log conditional PUT succeed once the store is back. - Outside it, a request fails fast. - A stop ends the wait. - Unit cases cover the budget, non-outage errors, and a `RemoteWorkspaceRestoreError` wrapping an outage. - `run-completion.test.ts`: a setup failure from a restore that wrapped an outage takes the storage wait (red before). - `object-byte-journal.test.ts`: a real `claim` whose head PUT hits its deadline is classified as a storage outage (red before). ## Risks - A storage outage now holds a task for up to about 10 minutes of automatic waits instead of escalating within seconds. A longer outage still ends in the existing board recovery. - A run whose failure is misclassified as storage-unavailable waits instead of using fast recovery. The classifier accepts only server-side or network failures from the storage SDK, never 4xx answers. - No schema change. - A run can now wait until a storage outage has lasted 5 minutes before it fails over to the bounded wait. It shows as working throughout, and Stop ends the wait. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
625 lines
48 KiB
TypeScript
625 lines
48 KiB
TypeScript
import { STORAGE_UNAVAILABLE_WAIT_RETRY_REASON } from "../storage/storage-unavailable.js";
|
||
import { and, desc, eq, inArray, isNull, sql, type SQL } from "drizzle-orm";
|
||
import { unidentifiedLocalLaunchRefusals } from "./native-local-process-stop.js";
|
||
import { agents, agentWakeupRequests, approvals, chatActions, companies, heartbeatRuns, issueApprovals, issueRecoveryActions,
|
||
issues, issueTreeHoldMembers, issueTreeHolds, nativeRunFinalizations, projects, type Db } from "@paperclipai/db";
|
||
import { unionAll } from "drizzle-orm/pg-core";
|
||
import { HEARTBEAT_RUN_STATUSES, readProviderFailure, type TaskInterruptionPresentation } from "@paperclipai/shared";
|
||
import { captureTaskInterruptionFailure } from "../sentry.js";
|
||
import { isAiAuthenticationBlocked } from "./ai-auth-failure.js";
|
||
import { readSystemRecovery } from "./system-recovery-state.js";
|
||
import { PROJECT_ACCESS_DENIED } from "./project-access-denial.js";
|
||
import { authorizationService } from "./authorization.js";
|
||
import { taskWorkspaceProjectRequirements } from "./task-workspace-access.js";
|
||
import { isAcknowledgedNativeStop } from "./acknowledged-native-stop.js";
|
||
import { createRunSecretRedactionRegistry } from "./run-secret-redaction.js";
|
||
import { instanceSettingsService } from "./instance-settings.js";
|
||
import { redactCurrentUserText } from "../log-redaction.js";
|
||
import { logger } from "../middleware/logger.js";
|
||
import { canonicalExecutionUuid } from "./execution-identity.js";
|
||
import { chatActionDeadline } from "./chat-action-work.js";
|
||
import { isQueuedDependencyWait } from "./run-cancellation.js";
|
||
import { AI_CONNECTION_BUSY_RETRY_REASON, AI_CONNECTION_POOL_WAIT_RETRY_REASON, RUNNER_REVISION_WAIT_RETRY_REASON,
|
||
WORKSPACE_BUSY_RETRY_REASON } from "../modules/run-dispatch/index.js";
|
||
|
||
// These scheduled runs wait for a shared resource; none records a failed
|
||
// attempt. Their dispatcher re-checks the resource, so the run can stay
|
||
// scheduled past its retry time. Retry cannot free the resource either.
|
||
const resourceWaitRetryReasons = new Set<string>([WORKSPACE_BUSY_RETRY_REASON, AI_CONNECTION_BUSY_RETRY_REASON,
|
||
AI_CONNECTION_POOL_WAIT_RETRY_REASON, RUNNER_REVISION_WAIT_RETRY_REASON, STORAGE_UNAVAILABLE_WAIT_RETRY_REASON]);
|
||
|
||
// A due same-run resume dispatches within seconds; past this, offer Retry.
|
||
const NATIVE_RESUME_DISPATCH_GRACE_MS = 2 * 60_000;
|
||
// A queued-message Interrupt waits for the stopped run's cleanup before its
|
||
// successor starts. Past this, a hand-off that never started offers Retry.
|
||
const QUEUED_INTERRUPT_HANDOFF_GRACE_MS = 5 * 60_000;
|
||
|
||
type ProjectionIssue = Pick<typeof issues.$inferSelect, "id" | "companyId" | "status" | "assigneeAgentId" | "projectId">;
|
||
type ProjectionRun = Pick<typeof heartbeatRuns.$inferSelect,
|
||
"id" | "status" | "scheduledRetryAt" | "errorCode" | "livenessState" | "resultJson"> &
|
||
Partial<Pick<typeof heartbeatRuns.$inferSelect, "agentId" | "error" | "companyId" | "nativeIssueId" | "runtimeMode" | "startedAt"
|
||
| "scheduledRetryReason" | "createdAt" | "finishedAt" | "processPid" | "processGroupId">> & {
|
||
/** Due time of the native coordinator's own same-run resume, if one is pending. */
|
||
nativeResumeAt?: Date | null;
|
||
/** When the native coordinator saved this run's terminal failure, if it did. */
|
||
nativeFailedAt?: Date | null;
|
||
};
|
||
type ProjectionWait = { id: string; requestedAt: Date; sourceRunId?: string | null; executionWaitReason?: string | null;
|
||
preparationStatus?: string | null; preparationRetryAt?: number | string | null; preparationUpdatedAt?: Date | null;
|
||
/** When the user pressed Interrupt to deliver this queued message now. */
|
||
queuedInterruptRequestedAt?: string | null };
|
||
type ProjectionState = {
|
||
run?: ProjectionRun;
|
||
action?: typeof issueRecoveryActions.$inferSelect;
|
||
wait?: ProjectionWait;
|
||
retryWait?: ProjectionWait;
|
||
/** The pending queued message this run was interrupted for, if any. */
|
||
interruptWait?: ProjectionWait;
|
||
agent?: Pick<typeof agents.$inferSelect, "status" | "pauseReason">;
|
||
projectAccessAllowed?: boolean;
|
||
projectAccessProjectId?: string | null;
|
||
workspaceSelectionAvailable?: boolean;
|
||
project?: Pick<typeof projects.$inferSelect, "pausedAt" | "pauseReason">;
|
||
company?: Pick<typeof companies.$inferSelect, "status" | "pauseReason">;
|
||
paused: boolean;
|
||
approval: boolean;
|
||
/** A board-owned workspace-owner fence on this run that no automatic proof has released. */
|
||
ownerStopRequired?: boolean;
|
||
unidentifiedLaunchConfirmable?: boolean;
|
||
/** The latest wake for a task with no run yet, when admission refused it. */
|
||
admissionWait?: { id: string; status: string; reason: string | null; agentId: string };
|
||
retryDependencies?: RetryDependencies;
|
||
};
|
||
|
||
// A task read reports the change that made a task fail, not every later read of
|
||
// retained history. The delivery cache is process-local, so a restart or another
|
||
// replica would otherwise resend each old failure whenever a list shows it.
|
||
const TASK_INTERRUPTION_REPORT_WINDOW_MS = 60 * 60_000;
|
||
|
||
/** Latest saved fact that can move a task into the failed state, if known. The
|
||
* recovery row's update time covers recovery that newly fails on an old run:
|
||
* each saved recovery transition stamps it, while activity heartbeats do not.
|
||
* The native resume due time and terminal finalization cover a same-run resume
|
||
* that lapses or is exhausted on an old run. */
|
||
export function taskInterruptionChangedAt(input: Pick<ProjectionState, "run" | "action" | "wait" | "retryWait">): number | null {
|
||
const { run, wait, action } = input;
|
||
const retryWait = input.retryWait ?? wait;
|
||
const preparationFailedAt = retryWait?.preparationStatus === "failed" || retryWait?.preparationStatus === "cancelled"
|
||
? retryWait.preparationUpdatedAt : null;
|
||
const times: unknown[] = [run?.finishedAt ?? run?.createdAt, run?.scheduledRetryAt, run?.nativeResumeAt, run?.nativeFailedAt,
|
||
wait?.requestedAt, retryWait?.requestedAt, action?.updatedAt, preparationFailedAt].map(value => value?.getTime());
|
||
times.push(readSystemRecovery(action?.evidence)?.nextAttemptAt,
|
||
retryWait?.preparationRetryAt == null ? null : Number(retryWait.preparationRetryAt));
|
||
const known = times.filter((time): time is number => typeof time === "number" && Number.isFinite(time));
|
||
return known.length ? Math.max(...known) : null;
|
||
}
|
||
|
||
function boundedJsonString(value: SQL, limit: number) {
|
||
return sql<string | null>`case when jsonb_typeof(${value}) = 'string' then left(${value} #>> '{}', ${limit}) else null end`;
|
||
}
|
||
|
||
/** Projected reason for a budget.blocked refusal caused by unpriced usage. */
|
||
export const UNPRICED_USAGE_WAIT = "accounting.unpriced_usage";
|
||
/** Admission's unpriced-usage refusal text (budgets.ts). Only this fixed
|
||
* phrase is matched; the stored message itself is never displayed. */
|
||
const unpricedUsageWait = (reason: SQL, message: SQL) =>
|
||
sql<string | null>`case when ${reason} = 'budget.blocked' and ${message} like '%recorded usage has no reliable price%'
|
||
then ${UNPRICED_USAGE_WAIT} else ${reason} end`;
|
||
|
||
// Persisted wait messages can contain diagnostics. Ordinary task views expose
|
||
// only these stable categories, never arbitrary payloads or source identifiers.
|
||
const retryWaitMessages: Record<string, string> = {
|
||
workspace_repair_required: "Retry is waiting for Paperclip to recover the saved files. Your message is saved.",
|
||
remote_cleanup: "Retry is waiting for the previous environment to stop. Your message is saved.",
|
||
local_cleanup: "Retry is waiting for the previous execution to finish cleanup. Your message is saved.",
|
||
cleanup_pending: "Retry is waiting for the previous execution to finish cleanup. Your message is saved.",
|
||
execution_settling: "Retry is waiting for the previous execution to stop. Your message is saved.",
|
||
controller_settling: "Retry is waiting for the previous execution to finish recovery. Your message is saved.",
|
||
process_identity_missing: "Retry is waiting for Paperclip to verify that the previous execution stopped. Your message is saved.",
|
||
process_running: "Retry is waiting for the previous execution to stop. Your message is saved.",
|
||
execution_active: "Retry is waiting for the current run to finish. Your message is saved.",
|
||
decision_pending: "Retry is waiting for a pending approval or question to be resolved. Your message is saved.",
|
||
"budget.blocked": "Retry is waiting for the budget to allow new work. Check Costs for unrecorded or unpriced usage.",
|
||
// Admission records an unpriced-usage refusal as budget.blocked; the
|
||
// projection shows it as the accounting matter it is.
|
||
[UNPRICED_USAGE_WAIT]: "Retry is waiting because recorded usage has no price. Check Costs, or allow unpriced usage in the budget policy. Your message is saved.",
|
||
finalizing_saved_result: "Paperclip is finishing the saved result before Retry can continue.",
|
||
// Another click cannot release a copyback owner that stopped without reporting.
|
||
// Retry starts a new recovery generation; the agent does not repeat its work.
|
||
saved_work_recovery_failed: "Saving the agent's files failed. Retry to try again; the agent will not repeat its work.",
|
||
saved_work_recovery_refused: "Paperclip refused to recover the saved files because a safety check on them cannot pass. "
|
||
+ "Retry runs the same check again and succeeds only after Paperclip is updated. The agent's result is saved, and the agent will not repeat its work.",
|
||
workspace_owner_stop_required: "Retry is waiting until Paperclip confirms that the previous server stopped saving files. The agent's result and your Retry are saved.",
|
||
cleanup_quarantined: "Retry cannot continue this attempt. Send a new message after the previous execution has stopped.",
|
||
cancelled_by_operator: "This attempt was stopped. Send a new message to continue from saved work.",
|
||
// A new click replaces a Retry declined for its requester's access with a
|
||
// fresh attempt under the clicker's access, so Retry stays available.
|
||
retry_authority_denied: "Retry could not start with the requesting user's access. Press Retry to start it with yours.",
|
||
// An edited saved message can never be replayed; only new input continues.
|
||
retry_source_changed: "Retry cannot continue because its saved message changed. Send a new message to continue from saved work.",
|
||
};
|
||
|
||
const UNIDENTIFIED_RUNNER_STOP_MESSAGE = "Paperclip can't confirm that the previous run stopped. The server restarted after it started the agent but before it recorded the agent's process. "
|
||
+ "If no agent is still working on this task, a board operator can confirm that it stopped, and your Retry starts right away. Your message is saved.";
|
||
|
||
type RetryDependencies = { count: number; blocked: boolean; cancelled: boolean; savingFilesOnly: boolean };
|
||
|
||
function dependencyRetryWaitMessage(dependencies: RetryDependencies): string {
|
||
if (dependencies.cancelled) return "Retry is waiting on a cancelled task this one depends on. Remove or replace that dependency to continue. Your Retry is saved.";
|
||
if (dependencies.blocked) return "Retry is waiting on a blocked task this one depends on. Retry or unblock it; this task continues when it is done. Your Retry is saved.";
|
||
if (dependencies.savingFilesOnly) return "Retry is waiting for a finished task this one depends on to save its files. Your Retry is saved.";
|
||
return "Retry is waiting for the tasks this one depends on to finish. Your Retry is saved.";
|
||
}
|
||
|
||
// A task that never started can wait for a temporary admission hold. The
|
||
// assigned-task liveness sweep admits it once the hold clears; no Retry needed.
|
||
const admissionWaitMessages: Record<string, string> = {
|
||
"budget.blocked": "Waiting for the budget to allow new work. The task starts automatically when it does.",
|
||
[UNPRICED_USAGE_WAIT]: "Waiting because recorded usage has no price. Check Costs, or allow unpriced usage in the budget policy.",
|
||
};
|
||
|
||
/** Recognize old persisted failures without displaying arbitrary run error text. */
|
||
function deniedProjectId(issue: ProjectionIssue, run?: ProjectionRun): string | null | undefined {
|
||
const historical = run?.errorCode === "setup_failed" && run.error === "Project is outside this low-trust boundary.";
|
||
if (!run || run.status !== "failed" || run.agentId !== issue.assigneeAgentId
|
||
|| (run.errorCode !== PROJECT_ACCESS_DENIED && !historical)) return undefined;
|
||
const detail = run.resultJson?.projectAccessDenied;
|
||
return detail && typeof detail === "object" && "projectId" in detail && typeof detail.projectId === "string"
|
||
? detail.projectId : null;
|
||
}
|
||
|
||
/** Presentation never grants authority. Dispatch repeats all gates under its
|
||
* locks. The same projection is used in detail, lists and attention surfaces. */
|
||
export function projectTaskInterruption(issue: ProjectionIssue, input: ProjectionState, now = Date.now()): TaskInterruptionPresentation | null {
|
||
const { run, wait, action, agent, project, company } = input;
|
||
const retryWait = input.retryWait ?? wait;
|
||
if (!run) {
|
||
const admission = input.admissionWait;
|
||
if (!admission || admission.status !== "skipped" || !Object.hasOwn(admissionWaitMessages, admission.reason ?? "")
|
||
|| issue.status !== "todo" || admission.agentId !== issue.assigneeAgentId) return null;
|
||
const waiting: TaskInterruptionPresentation = { intentId: admission.id, responsibility: "system", state: "working",
|
||
nextRetryAt: null, canRetry: false, userAction: null, sourceRunIds: [], observedRunId: null };
|
||
// Budget pauses, ordinary pauses and approvals wait on a person; only an
|
||
// accounting hold settles by itself. Admission records all of them as budget.blocked.
|
||
const userAction = !issue.assigneeAgentId ? "assign_agent" : input.approval ? "approval"
|
||
: [agent?.pauseReason, project?.pauseReason, company?.pauseReason].some(reason => reason === "budget" || reason === "budget_exceeded")
|
||
? "budget" : input.paused || project?.pausedAt || agent?.status === "paused" || company?.status === "paused" ? "resume" : null;
|
||
if (userAction) return { ...waiting, responsibility: "user", state: "waiting_for_user", userAction };
|
||
return { ...waiting, retryWait: { reason: admission.reason!, message: admissionWaitMessages[admission.reason!] } };
|
||
}
|
||
const system = readSystemRecovery(action?.evidence);
|
||
const base: TaskInterruptionPresentation = { intentId: wait?.id ?? null, responsibility: "system",
|
||
state: "resolved", nextRetryAt: null, canRetry: false, userAction: null,
|
||
sourceRunIds: [run.id], observedRunId: run.id };
|
||
if (["done", "cancelled"].includes(issue.status) || (run.status === "succeeded" && !wait)) return base;
|
||
const providerFailure = run.status === "failed" || run.status === "scheduled_retry"
|
||
? readProviderFailure(run.resultJson?.providerFailure) : null;
|
||
if (providerFailure) base.providerFailure = providerFailure;
|
||
if (["failed", "scheduled_retry"].includes(run.status) && run.errorCode === "workspace_validation_failed"
|
||
&& typeof run.error === "string" && run.error.trim()) base.workspaceFailure = { message: run.error };
|
||
const userAction = !issue.assigneeAgentId ? "assign_agent" : input.approval || run.errorCode === "native_provider_approval_required" ? "approval"
|
||
: isAiAuthenticationBlocked(run) ? "reconnect"
|
||
: run.errorCode === "native_provider_model_rejected" ? "choose_model"
|
||
: [agent?.pauseReason, project?.pauseReason, company?.pauseReason].some(reason => reason === "budget" || reason === "budget_exceeded")
|
||
? "budget" : input.paused || project?.pausedAt || agent?.status === "paused" || company?.status === "paused" ? "resume" : null;
|
||
if (userAction) return { ...base, responsibility: "user", state: "waiting_for_user", userAction };
|
||
// The previous controller cannot confirm its own stop. Name the one board
|
||
// action instead of showing the fenced run as working (RC-47).
|
||
if (input.ownerStopRequired) return { ...base, responsibility: "user", state: "waiting_for_user", userAction: "confirm_owner_stop" };
|
||
// The task's existing dependency UI owns this wait. Keep actual recovery and
|
||
// Retry preparation failures authoritative; this does not change admission.
|
||
if (!action && !retryWait?.preparationStatus && isQueuedDependencyWait(run)) return base;
|
||
// Intentional Stop is not an unsuccessful attempt. This is presentation only;
|
||
// physical containment and pending cleanup keep their execution-side gates.
|
||
if (!wait && run.runtimeMode === "native" && run.companyId === issue.companyId && run.nativeIssueId === issue.id
|
||
&& isAcknowledgedNativeStop({ ...run, companyId: run.companyId, nativeIssueId: run.nativeIssueId })) return base;
|
||
// A queued-message Interrupt stops this run to deliver that exact message.
|
||
// The stop is the user's intent and the message starts next, so show the
|
||
// hand-off as working, not as a failed attempt.
|
||
// Newer queued input can follow it, so match the recorded queue, not the latest wait.
|
||
const interruptWait = input.interruptWait ?? wait;
|
||
const interruptAt = Date.parse(interruptWait?.queuedInterruptRequestedAt ?? "");
|
||
if (interruptWait && run.status === "cancelled" && run.resultJson?.queuedCommentInterruptQueueId === interruptWait.id
|
||
&& Number.isFinite(interruptAt) && now - interruptAt < QUEUED_INTERRUPT_HANDOFF_GRACE_MS
|
||
&& !(retryWait?.sourceRunId === run.id && retryWait.preparationStatus) && system?.state !== "exhausted")
|
||
return { ...base, intentId: interruptWait.id, state: "working" };
|
||
if (["running", "queued"].includes(run.status)) return { ...base, state: "working" };
|
||
const deniedProject = deniedProjectId(issue, run);
|
||
let projectAccess: TaskInterruptionPresentation["projectAccess"];
|
||
if (deniedProject !== undefined) {
|
||
// Permissions cannot repair a stale or unavailable workspace selection.
|
||
if (input.workspaceSelectionAvailable === false) return { ...base, state: "failed" };
|
||
const ready = input.projectAccessAllowed === true;
|
||
const projectId = input.projectAccessProjectId === undefined ? deniedProject : input.projectAccessProjectId;
|
||
projectAccess = { agentId: run.agentId!, projectId: projectId === issue.projectId ? projectId : null, ready };
|
||
if (!ready) return { ...base, responsibility: "user", state: "waiting_for_user",
|
||
userAction: "project_access", projectAccess };
|
||
}
|
||
const recoveryStep: TaskInterruptionPresentation["recoveryStep"] = !system ? undefined
|
||
: system.phase === "verify_execution" ? "checking_previous_run"
|
||
: system.phase === "preserve_files" ? "saving_previous_files" : "finishing_recovery";
|
||
const activeObservation = action?.evidence.systemRecoveryActiveAt;
|
||
const recoveryRunning = system?.state === "running" && typeof activeObservation === "number"
|
||
&& activeObservation > now - 45_000;
|
||
const recoveryScheduled = Boolean(system?.nextAttemptAt && system.nextAttemptAt > now);
|
||
const preparationDeadline = retryWait?.preparationRetryAt == null ? NaN : Number(retryWait.preparationRetryAt);
|
||
// A pending Retry preparation is already a Retry; never offer another one.
|
||
const preparationPending = retryWait?.sourceRunId === run.id && Boolean(retryWait.preparationStatus)
|
||
&& ["issued", "processing"].includes(retryWait.preparationStatus!) && Number.isFinite(preparationDeadline);
|
||
if (retryWait?.sourceRunId === run.id && retryWait.preparationStatus) {
|
||
const deadline = preparationDeadline;
|
||
const pending = preparationPending;
|
||
// Saved-work recovery gates a pending preparation, so it names the wait.
|
||
// Failed or cancelled preparation stays authoritative.
|
||
if (pending && !recoveryRunning && !recoveryScheduled)
|
||
return { ...base, state: "retry_scheduled", nextRetryAt: new Date(deadline).toISOString() };
|
||
if (!pending) return { ...base, state: "failed", canRetry: retryWait.preparationStatus !== "cancelled" };
|
||
}
|
||
if (run.status === "scheduled_retry" && resourceWaitRetryReasons.has(run.scheduledRetryReason ?? "") && !system)
|
||
return { ...base, state: "working" };
|
||
// The native coordinator resumes this same run by itself. A manual Retry
|
||
// would race it, so show the resume until it is due and briefly while its
|
||
// dispatch runs. A resume that never ran falls through to one Retry.
|
||
const nativeResumeAt = run.nativeResumeAt?.getTime();
|
||
const retryAt = system?.nextAttemptAt ?? nativeResumeAt ?? run.scheduledRetryAt?.getTime();
|
||
if (run.runtimeMode === "native" && run.errorCode === "native_governed_wait_settlement_pending") {
|
||
const settling = recoveryRunning || Boolean(retryAt && now - retryAt < NATIVE_RESUME_DISPATCH_GRACE_MS);
|
||
return { ...base, state: settling ? "working" : "failed", canRetry: false,
|
||
retryWait: { reason: "governed_wait_settlement", message: settling
|
||
? "Your work is saved. Paperclip is finishing shutdown."
|
||
: "Paperclip couldn’t confirm shutdown. Your work is saved; continuation is paused." } };
|
||
}
|
||
if (retryAt && retryAt > now) return { ...base, state: "retry_scheduled",
|
||
nextRetryAt: new Date(retryAt).toISOString(), canRetry: !preparationPending && !nativeResumeAt,
|
||
...(system?.nextAttemptAt && recoveryStep ? { recoveryStep } : {}) };
|
||
if (nativeResumeAt && !system && now - nativeResumeAt < NATIVE_RESUME_DISPATCH_GRACE_MS)
|
||
return { ...base, state: "working" };
|
||
if (recoveryRunning) return { ...base, state: "working", ...(recoveryStep ? { recoveryStep } : {}) };
|
||
const dependencies = input.retryDependencies;
|
||
if (retryWait?.sourceRunId === run.id && !retryWait.executionWaitReason && !retryWait.preparationStatus && dependencies?.count)
|
||
return { ...base, intentId: retryWait.id, state: "working",
|
||
retryWait: { reason: "dependency_blocked", message: dependencyRetryWaitMessage(dependencies) } };
|
||
if (retryWait?.sourceRunId === run.id && retryWait.executionWaitReason === "process_identity_missing"
|
||
&& input.unidentifiedLaunchConfirmable)
|
||
return { ...base, responsibility: "user", state: "waiting_for_user", canRetry: false, userAction: "confirm_run_stopped",
|
||
retryWait: { reason: "process_identity_missing", message: UNIDENTIFIED_RUNNER_STOP_MESSAGE } };
|
||
if (retryWait?.sourceRunId === run.id && retryWait.executionWaitReason) {
|
||
const reason = Object.hasOwn(retryWaitMessages, retryWait.executionWaitReason) ? retryWait.executionWaitReason : "execution_recovery";
|
||
// Keep the latest input receipt so message submission can settle independently.
|
||
return { ...base, state: "failed",
|
||
canRetry: !["cleanup_quarantined", "cancelled_by_operator", "retry_source_changed", "workspace_owner_stop_required"].includes(reason), retryWait: { reason,
|
||
message: retryWaitMessages[reason] ?? "Retry is waiting for execution checks. Your message is saved." } };
|
||
}
|
||
if (wait && wait.requestedAt.getTime() > now - 10_000 && system?.state !== "exhausted")
|
||
return { ...base, state: "working" };
|
||
return { ...base, state: "failed", canRetry: true, ...(projectAccess ? { projectAccess } : {}) };
|
||
}
|
||
|
||
/** A scheduled retry has no outcome yet, so its provider failure is read from
|
||
* the predecessor it retries: the countdown keeps its reason.
|
||
* Batch list reads rather than doing seven queries per row. Keep run logs and
|
||
* large result bodies out of this projection. */
|
||
export async function taskInterruptionPresentations(db: Db, companyId: string, input: ProjectionIssue[]) {
|
||
const selected = input.filter(issue => issue.companyId === companyId);
|
||
const result = new Map<string, TaskInterruptionPresentation | null>();
|
||
if (!selected.length) return result;
|
||
const ids = selected.map(issue => issue.id);
|
||
const requested = sql`unnest(${sql.param([...new Set(ids)])}::text[]) as requested_interruption_issues(requested_issue_id)`;
|
||
const requestedIssueId = sql<string>`"requested_interruption_issues"."requested_issue_id"`;
|
||
const requestedUuid = sql<string>`case when ${requestedIssueId} ~ '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'
|
||
then ${requestedIssueId}::uuid else null end`;
|
||
const legacyIssue = sql<string>`${heartbeatRuns.contextSnapshot}->>'issueId'`;
|
||
const candidate = (condition: ReturnType<typeof and>) => db.select({ id: heartbeatRuns.id, createdAt: heartbeatRuns.createdAt })
|
||
.from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), condition))
|
||
.orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
|
||
const candidates = unionAll(
|
||
candidate(eq(heartbeatRuns.nativeIssueId, requestedUuid)),
|
||
candidate(and(isNull(heartbeatRuns.nativeIssueId), eq(heartbeatRuns.issueId, requestedUuid))),
|
||
candidate(and(isNull(heartbeatRuns.nativeIssueId), isNull(heartbeatRuns.issueId), eq(legacyIssue, requestedIssueId))),
|
||
).as("interruption_run_candidates");
|
||
const latestRuns = db.select({ id: candidates.id }).from(candidates)
|
||
.orderBy(desc(candidates.createdAt), desc(candidates.id)).limit(1).as("latest_interruption_runs");
|
||
const wakeIssue = sql<string>`${agentWakeupRequests.payload}->>'issueId'`;
|
||
const isRetryWait = sql<boolean>`coalesce(jsonb_typeof(${agentWakeupRequests.payload}#>'{taskRetry,sourceRunId}') = 'string', false)`;
|
||
const agentIds = selected.flatMap(issue => issue.assigneeAgentId ? [issue.assigneeAgentId] : []);
|
||
const projectIds = selected.flatMap(issue => issue.projectId ? [issue.projectId] : []);
|
||
const startingIssueIds = selected.filter(issue => issue.status === "todo" && issue.assigneeAgentId).map(issue => issue.id);
|
||
const [runs, waits, agentRows, projectRows, pauseRows, approvalRows, companyRows, admissionWaits] = await Promise.all([
|
||
db.select({ issueId: requestedIssueId, id: heartbeatRuns.id, status: heartbeatRuns.status,
|
||
createdAt: heartbeatRuns.createdAt, finishedAt: heartbeatRuns.finishedAt, agentId: heartbeatRuns.agentId, companyId: heartbeatRuns.companyId,
|
||
nativeIssueId: heartbeatRuns.nativeIssueId, runtimeMode: heartbeatRuns.runtimeMode, startedAt: heartbeatRuns.startedAt,
|
||
processPid: heartbeatRuns.processPid, processGroupId: heartbeatRuns.processGroupId,
|
||
agentAdapter: agents.adapterType,
|
||
error: sql<string | null>`case when ${heartbeatRuns.errorCode} = 'setup_failed'
|
||
and ${heartbeatRuns.error} = 'Project is outside this low-trust boundary.'
|
||
or ${heartbeatRuns.errorCode} = 'workspace_validation_failed' then ${heartbeatRuns.error} else null end`,
|
||
errorCode: heartbeatRuns.errorCode, livenessState: heartbeatRuns.livenessState, scheduledRetryAt: heartbeatRuns.scheduledRetryAt,
|
||
scheduledRetryReason: heartbeatRuns.scheduledRetryReason,
|
||
// Candidate only: the coordinator also pushes nextAttemptAt forward while
|
||
// it defers an ineligible run. The scheduler's own admission check below
|
||
// decides whether this resume is real.
|
||
nativeResumeAt: sql<Date | null>`(select ${nativeRunFinalizations.nextAttemptAt} from ${nativeRunFinalizations}
|
||
where ${nativeRunFinalizations.companyId} = ${heartbeatRuns.companyId} and ${nativeRunFinalizations.runId} = ${heartbeatRuns.id}
|
||
and ${heartbeatRuns.runtimeMode} = 'native' and ${heartbeatRuns.status} = 'failed'
|
||
and ${nativeRunFinalizations.resultId} is null and ${nativeRunFinalizations.phase} = 'retryable_failure'
|
||
and ${heartbeatRuns.contextSnapshot}->'nativeQuestionCancellation' is null
|
||
and ${heartbeatRuns.runnerProfileJson}->'legacyWorkspaceRecoveryMigration' is null)`.mapWith(value => value == null ? null : new Date(value)),
|
||
nativeFailedAt: sql<Date | null>`(select ${nativeRunFinalizations.updatedAt} from ${nativeRunFinalizations}
|
||
where ${nativeRunFinalizations.companyId} = ${heartbeatRuns.companyId} and ${nativeRunFinalizations.runId} = ${heartbeatRuns.id}
|
||
and ${nativeRunFinalizations.phase} = 'terminal_failure')`.mapWith(value => value == null ? null : new Date(value)),
|
||
// The run whose resultJson supplied providerFailure. A scheduled retry
|
||
// shows its predecessor's reason, so redaction must use that run's registry.
|
||
providerFailureRunId: sql<string>`case when ${heartbeatRuns.resultJson}->'providerFailure' is null
|
||
and ${heartbeatRuns.status} = 'scheduled_retry' and ${heartbeatRuns.retryOfRunId} is not null
|
||
then ${heartbeatRuns.retryOfRunId} else ${heartbeatRuns.id} end`,
|
||
resultJson: sql<Record<string, unknown>>`jsonb_build_object('configurationIncomplete', ${heartbeatRuns.resultJson}->'configurationIncomplete',
|
||
'stopReason', ${boundedJsonString(sql`${heartbeatRuns.resultJson}->'stopReason'`, 80)},
|
||
'timeoutSource', ${boundedJsonString(sql`${heartbeatRuns.resultJson}->'timeoutSource'`, 80)},
|
||
'timeoutFired', coalesce(${heartbeatRuns.resultJson}->'timeoutFired' = 'true'::jsonb, false),
|
||
'projectAccessDenied', ${heartbeatRuns.resultJson}->'projectAccessDenied',
|
||
'cancelledByActorType', ${boundedJsonString(sql`${heartbeatRuns.resultJson}->'cancelledByActorType'`, 32)},
|
||
'cancelledByUserId', ${boundedJsonString(sql`${heartbeatRuns.resultJson}->'cancelledByUserId'`, 160)},
|
||
'queuedCommentInterruptQueueId', ${boundedJsonString(sql`${heartbeatRuns.resultJson}->'queuedCommentInterruptQueueId'`, 160)},
|
||
'nativeCancellation', jsonb_build_object(
|
||
'schema', ${boundedJsonString(sql`${heartbeatRuns.resultJson}->'nativeCancellation'->'schema'`, 64)},
|
||
'runId', ${boundedJsonString(sql`${heartbeatRuns.resultJson}->'nativeCancellation'->'runId'`, 160)},
|
||
'companyId', ${boundedJsonString(sql`${heartbeatRuns.resultJson}->'nativeCancellation'->'companyId'`, 160)},
|
||
'issueId', ${boundedJsonString(sql`${heartbeatRuns.resultJson}->'nativeCancellation'->'issueId'`, 160)},
|
||
'scope', ${boundedJsonString(sql`${heartbeatRuns.resultJson}->'nativeCancellation'->'scope'`, 32)},
|
||
'reasonCode', ${boundedJsonString(sql`${heartbeatRuns.resultJson}->'nativeCancellation'->'reasonCode'`, 64)},
|
||
'dispatchState', ${boundedJsonString(sql`${heartbeatRuns.resultJson}->'nativeCancellation'->'dispatchState'`, 32)},
|
||
'dispatched', coalesce(${heartbeatRuns.resultJson}->'nativeCancellation'->'dispatched' = 'true'::jsonb, false),
|
||
'intentAuditId', ${boundedJsonString(sql`${heartbeatRuns.resultJson}->'nativeCancellation'->'intentAuditId'`, 160)},
|
||
'acknowledgementAuditId', ${boundedJsonString(sql`${heartbeatRuns.resultJson}->'nativeCancellation'->'acknowledgementAuditId'`, 160)}),
|
||
'providerFailure', coalesce(${heartbeatRuns.resultJson}->'providerFailure', case when ${heartbeatRuns.status} = 'scheduled_retry'
|
||
then (select predecessor.result_json->'providerFailure' from ${heartbeatRuns} predecessor
|
||
where predecessor.company_id = ${heartbeatRuns.companyId} and predecessor.id = ${heartbeatRuns.retryOfRunId}) end))`,
|
||
}).from(requested).innerJoinLateral(latestRuns, sql`true`).innerJoin(heartbeatRuns, and(eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, latestRuns.id)))
|
||
.leftJoin(agents, and(eq(agents.companyId, heartbeatRuns.companyId), eq(agents.id, heartbeatRuns.agentId)))
|
||
.orderBy(requestedIssueId),
|
||
// Keep at most two candidates per task: latest Retry and latest other input.
|
||
// A new message must not hide an existing Retry's admission explanation.
|
||
db.selectDistinctOn([wakeIssue, isRetryWait], { issueId: wakeIssue, id: agentWakeupRequests.id, requestedAt: agentWakeupRequests.requestedAt,
|
||
sourceRunId: boundedJsonString(sql`${agentWakeupRequests.payload}#>'{taskRetry,sourceRunId}'`, 160),
|
||
executionWaitReason: boundedJsonString(sql`to_jsonb(${unpricedUsageWait(sql`${agentWakeupRequests.payload}#>>'{executionWait,reason}'`,
|
||
sql`${agentWakeupRequests.payload}#>>'{executionWait,message}'`)})`, 80),
|
||
preparationStatus: chatActions.status, preparationRetryAt: chatActionDeadline("failed_run_retry"),
|
||
preparationUpdatedAt: chatActions.updatedAt })
|
||
.from(agentWakeupRequests).leftJoin(chatActions, and(eq(chatActions.companyId, agentWakeupRequests.companyId),
|
||
eq(chatActions.id, canonicalExecutionUuid(sql`${agentWakeupRequests.payload}#>>'{taskRetry,chatActionId}'`)),
|
||
eq(chatActions.kind, "failed_run_retry"), sql`${chatActions.payload}->>'taskRetryIntentId' = ${agentWakeupRequests.id}::text`,
|
||
sql`${chatActions.payload}->>'issueId' = ${wakeIssue}`,
|
||
sql`${chatActions.payload}->>'agentId' = ${agentWakeupRequests.agentId}::text`,
|
||
sql`${chatActions.payload}->>'failedRunId' = ${agentWakeupRequests.payload}#>>'{taskRetry,sourceRunId}'`,
|
||
inArray(chatActions.status, ["issued", "processing", "failed", "cancelled"])))
|
||
.where(and(eq(agentWakeupRequests.companyId, companyId), inArray(wakeIssue, ids),
|
||
eq(agentWakeupRequests.status, "deferred_issue_execution"), eq(agentWakeupRequests.requestedByActorType, "user")))
|
||
.orderBy(wakeIssue, isRetryWait, desc(agentWakeupRequests.requestedAt), desc(agentWakeupRequests.id)),
|
||
agentIds.length ? db.select({ id: agents.id, status: agents.status, pauseReason: agents.pauseReason }).from(agents)
|
||
.where(and(eq(agents.companyId, companyId), inArray(agents.id, agentIds))) : [],
|
||
projectIds.length ? db.select({ id: projects.id, pausedAt: projects.pausedAt, pauseReason: projects.pauseReason }).from(projects)
|
||
.where(and(eq(projects.companyId, companyId), inArray(projects.id, projectIds))) : [],
|
||
db.select({ issueId: issueTreeHoldMembers.issueId }).from(issueTreeHoldMembers).innerJoin(issueTreeHolds, and(
|
||
eq(issueTreeHolds.companyId, issueTreeHoldMembers.companyId), eq(issueTreeHolds.id, issueTreeHoldMembers.holdId)))
|
||
.where(and(eq(issueTreeHoldMembers.companyId, companyId), inArray(issueTreeHoldMembers.issueId, ids),
|
||
eq(issueTreeHolds.status, "active"), eq(issueTreeHolds.mode, "pause"))),
|
||
db.select({ issueId: issueApprovals.issueId }).from(issueApprovals).innerJoin(approvals, and(
|
||
eq(approvals.companyId, issueApprovals.companyId), eq(approvals.id, issueApprovals.approvalId)))
|
||
.where(and(eq(issueApprovals.companyId, companyId), inArray(issueApprovals.issueId, ids),
|
||
inArray(approvals.status, ["pending", "revision_requested"]))),
|
||
db.select({ status: companies.status, pauseReason: companies.pauseReason }).from(companies).where(eq(companies.id, companyId)),
|
||
// Only consulted for tasks without a run: the latest wake says whether admission refused it.
|
||
startingIssueIds.length ? db.selectDistinctOn([wakeIssue], { issueId: wakeIssue, id: agentWakeupRequests.id,
|
||
status: agentWakeupRequests.status, reason: unpricedUsageWait(sql`${agentWakeupRequests.reason}`, sql`${agentWakeupRequests.error}`),
|
||
agentId: agentWakeupRequests.agentId })
|
||
.from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, companyId), inArray(wakeIssue, startingIssueIds)))
|
||
.orderBy(wakeIssue, desc(agentWakeupRequests.requestedAt), desc(agentWakeupRequests.id)) : [],
|
||
]);
|
||
// Advertise a pending resume only when the scheduler would admit it, using
|
||
// its own check, so the task never hides Retry behind a deferred candidate
|
||
// and never offers Retry beside a resume that will run. Rare rows only.
|
||
// Candidates are failed native runs inside an active coordinator retry, so
|
||
// there are few. Their rows load in three batched reads; admission itself
|
||
// runs a few bounded reads per candidate, four at a time.
|
||
const resumeCandidates = runs.filter(run => run.nativeResumeAt);
|
||
if (resumeCandidates.length) {
|
||
const { readNativeSessionResumptionAgent } = await import("./native-runtime/native-session-resumption-admission.js");
|
||
const candidateIds = resumeCandidates.map(run => run.id);
|
||
const [fullRuns, coordinators] = await Promise.all([
|
||
db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), inArray(heartbeatRuns.id, candidateIds))),
|
||
db.select().from(nativeRunFinalizations).where(and(eq(nativeRunFinalizations.companyId, companyId),
|
||
inArray(nativeRunFinalizations.runId, candidateIds))),
|
||
]);
|
||
const issueIds = [...new Set(fullRuns.flatMap(run => run.nativeIssueId ? [run.nativeIssueId] : []))];
|
||
const issueRows = issueIds.length ? await db.select().from(issues).where(and(eq(issues.companyId, companyId),
|
||
inArray(issues.id, issueIds))) : [];
|
||
const runById = new Map(fullRuns.map(run => [run.id, run]));
|
||
const coordinatorByRun = new Map(coordinators.map(row => [row.runId, row]));
|
||
const issueById = new Map(issueRows.map(row => [row.id, row]));
|
||
for (let start = 0; start < resumeCandidates.length; start += 4) {
|
||
await Promise.all(resumeCandidates.slice(start, start + 4).map(async candidate => {
|
||
const run = runById.get(candidate.id);
|
||
const issue = run?.nativeIssueId ? issueById.get(run.nativeIssueId) : undefined;
|
||
const coordinator = coordinatorByRun.get(candidate.id);
|
||
const admitted = run && issue && coordinator
|
||
&& await readNativeSessionResumptionAgent(db, { issue, run, coordinator }).catch(() => null);
|
||
if (!admitted) candidate.nativeResumeAt = null;
|
||
}));
|
||
}
|
||
}
|
||
// A queued-message Interrupt records its queue on the stopped run. Newer
|
||
// input can hide that wake from the latest-wait read, so load it by id.
|
||
const interruptQueueId = (run: typeof runs[number]) => {
|
||
const id = run.status === "cancelled" ? run.resultJson?.queuedCommentInterruptQueueId : null;
|
||
return typeof id === "string" && /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/.test(id) ? id : null;
|
||
};
|
||
const interruptQueueIds = runs.flatMap(run => interruptQueueId(run) ?? []);
|
||
const interruptWaits = interruptQueueIds.length ? await db.select({ issueId: wakeIssue, id: agentWakeupRequests.id,
|
||
requestedAt: agentWakeupRequests.requestedAt,
|
||
queuedInterruptRequestedAt: boundedJsonString(sql`${agentWakeupRequests.payload}#>'{queuedCommentInterrupt,requestedAt}'`, 40) })
|
||
.from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, companyId), inArray(agentWakeupRequests.id, interruptQueueIds),
|
||
eq(agentWakeupRequests.status, "deferred_issue_execution"), eq(agentWakeupRequests.requestedByActorType, "user"))) : [];
|
||
const interruptWaitById = new Map(interruptWaits.map(wait => [wait.id, wait]));
|
||
const runIds = runs.map(run => run.id);
|
||
const actions = runIds.length ? await db.selectDistinctOn([issueRecoveryActions.sourceIssueId]).from(issueRecoveryActions).where(and(
|
||
eq(issueRecoveryActions.companyId, companyId), inArray(issueRecoveryActions.sourceIssueId, ids),
|
||
inArray(sql`${issueRecoveryActions.evidence}->>'runId'`, runIds),
|
||
sql`${issueRecoveryActions.evidence} ? 'systemRecovery'`,
|
||
)).orderBy(issueRecoveryActions.sourceIssueId, desc(issueRecoveryActions.createdAt)) : [];
|
||
const ownerStops = runIds.length ? await db.select({ issueId: issueRecoveryActions.sourceIssueId,
|
||
runId: sql<string>`${issueRecoveryActions.evidence}->>'runId'` }).from(issueRecoveryActions).where(and(
|
||
eq(issueRecoveryActions.companyId, companyId), inArray(issueRecoveryActions.sourceIssueId, ids),
|
||
eq(issueRecoveryActions.cause, "native_workspace_finalization_owner_unverified"), eq(issueRecoveryActions.ownerType, "board"),
|
||
inArray(issueRecoveryActions.status, ["active", "escalated"]),
|
||
inArray(sql`${issueRecoveryActions.evidence}->>'runId'`, runIds),
|
||
sql`${issueRecoveryActions.evidence}->'workspaceOwnerStop' is null`,
|
||
sql`coalesce(${issueRecoveryActions.evidence}#>>'{workspaceOwnerRecovery,state}', '') <> 'resuming'`,
|
||
)) : [];
|
||
const ownerStopRequired = new Set(ownerStops.map(row => `${row.issueId}:${row.runId}`));
|
||
const byIssue = <T extends { issueId: string }>(rows: T[]) => new Map(rows.map(row => [row.issueId, row]));
|
||
const runMap = byIssue(runs), waitMap = new Map<string, typeof waits[number]>();
|
||
// A Retry belongs to the run it observed. New conversation work supersedes
|
||
// that authority, even before dispatch retires the old receipt. Retained
|
||
// ordinary input is independent and must still be shown as waiting.
|
||
const currentWaits = waits.filter(wait => !wait.sourceRunId || wait.sourceRunId === runMap.get(wait.issueId)?.id);
|
||
const retryWaitMap = byIssue(currentWaits.filter(wait => wait.sourceRunId));
|
||
const retryIssueIds = [...retryWaitMap.keys()];
|
||
const retryDependencyMap = new Map<string, RetryDependencies>();
|
||
if (retryIssueIds.length) {
|
||
const { issueService } = await import("./issues.js");
|
||
const readiness = await issueService(db).listDependencyReadiness(companyId, retryIssueIds);
|
||
const blockerIds = [...new Set([...readiness.values()].flatMap(row => row.unresolvedBlockerIssueIds))];
|
||
const statusRows = blockerIds.length ? await db.select({ id: issues.id, status: issues.status }).from(issues)
|
||
.where(and(eq(issues.companyId, companyId), inArray(issues.id, blockerIds))) : [];
|
||
const statusById = new Map(statusRows.map(row => [row.id, row.status]));
|
||
for (const [waitingIssueId, row] of readiness) {
|
||
const ids = [...new Set(row.unresolvedBlockerIssueIds)];
|
||
if (!ids.length) continue;
|
||
const statuses = ids.map(id => statusById.get(id));
|
||
const saving = new Set(row.pendingFinalizeBlockerIssueIds);
|
||
retryDependencyMap.set(waitingIssueId, { count: ids.length,
|
||
blocked: statuses.includes("blocked"), cancelled: statuses.includes("cancelled"),
|
||
savingFilesOnly: ids.every(id => saving.has(id)) });
|
||
}
|
||
}
|
||
for (const wait of currentWaits) {
|
||
const current = waitMap.get(wait.issueId);
|
||
if (!current || wait.requestedAt > current.requestedAt ||
|
||
(wait.requestedAt.getTime() === current.requestedAt.getTime() && wait.id > current.id)) waitMap.set(wait.issueId, wait);
|
||
}
|
||
// Recheck current authority, including standard trust, root-issue grants and
|
||
// project policy. Do not reuse the failed run's frozen policy for a new retry.
|
||
// Only denied tasks need these reads; deduplicate projects and batch per agent.
|
||
// A task's organizational project can differ from its retained source. Old
|
||
// failures did not record the denied project, and even typed failures can
|
||
// become obsolete after a workspace selection changes.
|
||
const deniedIssues = selected.filter(issue => issue.assigneeAgentId && !["done", "cancelled"].includes(issue.status)
|
||
&& deniedProjectId(issue, runMap.get(issue.id)) !== undefined);
|
||
const requirements = await taskWorkspaceProjectRequirements(db, companyId, deniedIssues.map(issue => issue.id));
|
||
const deniedByAgent = new Map<string, Set<string>>();
|
||
for (const issue of deniedIssues) {
|
||
const ids = deniedByAgent.get(issue.assigneeAgentId!) ?? new Set<string>();
|
||
for (const projectId of requirements.get(issue.id)?.projectIds ?? []) ids.add(projectId);
|
||
if (ids.size) deniedByAgent.set(issue.assigneeAgentId!, ids);
|
||
}
|
||
const currentAccess = new Map<string, boolean>();
|
||
const authorization = authorizationService(db);
|
||
for (const [agentId, ids] of deniedByAgent) {
|
||
const projectIds = [...ids];
|
||
const decisions = await authorization.decideReads({ type: "agent", agentId, companyId, source: "agent_key" },
|
||
projectIds.map(projectId => ({ type: "project" as const, companyId, projectId })));
|
||
projectIds.forEach((projectId, index) => currentAccess.set(`${agentId}:${projectId}`, decisions[index]?.allowed === true));
|
||
}
|
||
const actionMap = new Map(actions.map(action => [action.sourceIssueId, action]));
|
||
const interruptWaitFor = (issueId: string) => {
|
||
const run = runMap.get(issueId);
|
||
const wait = run ? interruptWaitById.get(interruptQueueId(run) ?? "") : undefined;
|
||
return wait?.issueId === issueId ? wait : undefined;
|
||
};
|
||
const agentMap = new Map(agentRows.map(agent => [agent.id, agent]));
|
||
const projectMap = new Map(projectRows.map(project => [project.id, project]));
|
||
const pauses = new Set(pauseRows.map(row => row.issueId)), approvalsSet = new Set(approvalRows.map(row => row.issueId));
|
||
const admissionWaitMap = byIssue(admissionWaits);
|
||
const launchWaits = [...retryWaitMap].flatMap(([issueId, wait]) => {
|
||
const run = runMap.get(issueId);
|
||
return run && wait.sourceRunId === run.id && wait.executionWaitReason === "process_identity_missing" ? [{ issueId, run }] : [];
|
||
});
|
||
const launchRefusals = await unidentifiedLocalLaunchRefusals(db, launchWaits.map(entry => entry.run));
|
||
const confirmableLaunches = new Set(launchWaits.filter(entry => launchRefusals.get(entry.run.id) === null).map(entry => entry.issueId));
|
||
const now = Date.now();
|
||
for (const issue of selected) {
|
||
const required = requirements.get(issue.id);
|
||
const denied = required?.projectIds.filter(projectId => currentAccess.get(`${issue.assigneeAgentId}:${projectId}`) !== true);
|
||
const facts = { run: runMap.get(issue.id), action: actionMap.get(issue.id), wait: waitMap.get(issue.id),
|
||
retryWait: retryWaitMap.get(issue.id) };
|
||
const presentation = projectTaskInterruption(issue, { ...facts,
|
||
interruptWait: interruptWaitFor(issue.id),
|
||
projectAccessAllowed: denied?.length === 0,
|
||
projectAccessProjectId: denied?.includes(issue.projectId ?? "") ? issue.projectId : null,
|
||
workspaceSelectionAvailable: required?.available ?? false,
|
||
agent: agentMap.get(issue.assigneeAgentId ?? ""), project: projectMap.get(issue.projectId ?? ""),
|
||
company: companyRows[0], paused: pauses.has(issue.id), approval: approvalsSet.has(issue.id),
|
||
ownerStopRequired: ownerStopRequired.has(`${issue.id}:${facts.run?.id}`),
|
||
unidentifiedLaunchConfirmable: confirmableLaunches.has(issue.id),
|
||
admissionWait: admissionWaitMap.get(issue.id),
|
||
retryDependencies: retryDependencyMap.get(issue.id),
|
||
}, now);
|
||
result.set(issue.id, presentation);
|
||
const run = facts.run;
|
||
const runStatus = HEARTBEAT_RUN_STATUSES.find(status => status === run?.status);
|
||
const changedAt = taskInterruptionChangedAt(facts);
|
||
if (presentation?.responsibility === "system" && presentation.state === "failed" && run && runStatus
|
||
&& (changedAt === null || changedAt > now - TASK_INTERRUPTION_REPORT_WINDOW_MS)) {
|
||
captureTaskInterruptionFailure({ companyId, taskId: issue.id, agentId: run.agentId,
|
||
runId: run.id, runStatus, intentId: presentation.intentId, runErrorCode: run.errorCode ?? null,
|
||
agentAdapter: run.agentAdapter ?? "unknown" });
|
||
}
|
||
}
|
||
// Redact each explanation with the registry of the run that recorded it: the
|
||
// workspace failure is the observed run's, while a scheduled retry's provider
|
||
// failure may be its predecessor's.
|
||
const disclosures = [...result.entries()].flatMap(([issueId, presentation]) =>
|
||
(presentation?.providerFailure || presentation?.workspaceFailure) && presentation.observedRunId
|
||
? [{ id: presentation.observedRunId, providerFailureRunId: runMap.get(issueId)?.providerFailureRunId ?? presentation.observedRunId,
|
||
providerFailure: presentation.providerFailure, workspaceFailure: presentation.workspaceFailure, presentation }] : []);
|
||
if (disclosures.length) {
|
||
try {
|
||
const [redacted, settings] = await Promise.all([
|
||
createRunSecretRedactionRegistry(db).redactForRuns(companyId, disclosures.flatMap(({ id, providerFailureRunId, providerFailure, workspaceFailure }) => [
|
||
{ id: providerFailureRunId, providerFailure, workspaceFailure: undefined },
|
||
{ id, providerFailure: undefined, workspaceFailure },
|
||
])),
|
||
instanceSettingsService(db).getGeneral(),
|
||
]);
|
||
const safe = disclosures.map((_, index) => ({ providerFailure: redacted[2 * index]?.providerFailure,
|
||
workspaceFailure: redacted[2 * index + 1]?.workspaceFailure }));
|
||
const safeMessage = (value: unknown) => typeof value === "string"
|
||
? [...redactCurrentUserText(value, { enabled: settings.censorUsernameInLogs })].slice(0, 4000).join("") : "";
|
||
disclosures.forEach(({ presentation }, index) => {
|
||
if (presentation.providerFailure) {
|
||
const value = safe[index]?.providerFailure;
|
||
const failure = readProviderFailure({ ...value, message: safeMessage(value?.message) });
|
||
if (failure) presentation.providerFailure = failure;
|
||
else delete presentation.providerFailure;
|
||
}
|
||
if (presentation.workspaceFailure) {
|
||
const message = safeMessage(safe[index]?.workspaceFailure?.message);
|
||
if (message.trim()) presentation.workspaceFailure = { message };
|
||
else delete presentation.workspaceFailure;
|
||
}
|
||
});
|
||
} catch {
|
||
for (const { presentation } of disclosures) {
|
||
delete presentation.providerFailure;
|
||
delete presentation.workspaceFailure;
|
||
}
|
||
logger.warn({ companyId, runCount: disclosures.length }, "Could not redact run failure display; explanations omitted");
|
||
}
|
||
}
|
||
return result;
|
||
}
|
||
|
||
export async function taskInterruptionPresentation(db: Db, issue: ProjectionIssue) {
|
||
return (await taskInterruptionPresentations(db, issue.companyId, [issue])).get(issue.id) ?? null;
|
||
}
|