mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 11:54:35 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run stops before it finishes, automatic recovery continues the task, through the fresh-continuation fallback, in a new run that starts from the saved work > - A follow-up run (for example one started by a new comment or by finished subtasks) can fail before any provider work and leave a saved-work hold. The system then queues its own retry, and the dispatch gate cancels that retry at claim with `execution_reconciliation_required` because the hold is still in place > - The fallback takes over the hold when that happens, but it continued only "recent interruptions". That test required the failed run to be a recovery run itself or to fail with a restart code. A first follow-up run that failed with `adapter_failed` passed neither test, even though the system had already decided to retry it. So the fallback answered "a person must press Retry". The backlog pass used the same test, and the assigned-work sweep never listed the task. The task stayed blocked with no run, no pending path and no Retry button > - This pull request treats a system retry refused at dispatch as proof that the system continues the task. It also stops cancelled-at-dispatch attempts from counting against the hourly budget, lets the sweep continue a blocked task whose latest run was refused at dispatch, and keeps recovery notices from asking for a Retry that the task does not offer > - The benefit is that such a task continues by itself in exactly one fresh run from the saved work, whether the saved work was recovered before or after the refusal, and a person's own failed Retry keeps today's behaviour ## Linked Issues or Issue Description Refs #21394, #21348, #21363, #21478 **What happened?** A task's follow-up run, a legacy run that was not a recovery run, failed before any provider work with `adapter_failed`. Its files could not be restored, so it left a saved-work hold (`legacy_execution_requires_reconciliation`, `workspaceRestoreFailure: restore_failed`). The system queued a retry of that run. The dispatch gate cancelled the retry at claim with `execution_reconciliation_required`, and the "older run hold" path handed the hold to the fresh-continuation fallback (`run-dispatch/adapters/postgres.ts` `settleOlderRunHold`). The fallback found the refused run and asked `interruptionEligibility` whether the interruption was recent. That predicate required `continuesAutomaticallySql(source)` (a recovery run, or a restart, deploy or bootstrap code), so it returned `person_owned` / `retry_needed`. The stranded reconciler then blocked the task with "Press Retry", but the saved-work hold reports that Retry is not available. This happened twice in a local restart-chaos wave. In one case the saved work was recovered two minutes before the claim. In the other it was recovered thirty seconds after the claim. **Expected behavior** A task never rests blocked with no live run, no pending path and no Retry. While the saved-work recovery runs, that recovery is the pending path. When it finishes, one continuation starts from the saved work through the fallback (isolation, the shared hourly cap). A run that was cancelled at dispatch before it started does not use up the automatic budget. A person's own failed Retry is still left for that person. **Steps to reproduce** 1. Fail a follow-up run (wake reason `issue_children_completed`) before provider work with `adapter_failed` and `workspaceRestoreFailure: restore_failed`. 2. Queue the system's retry of that run (`retryOfRunId`, wake reason `issue_continuation_needed`) and claim it. The dispatch gate cancels it with `execution_reconciliation_required`. 3. Before this change, no continuation is recorded, the stranded reconciler blocks the task, and nothing ever continues it. **Paperclip version or commit** master at2756be2f32## What Changed - `fresh-continuation-fallback.ts`, `interruptionEligibility`: the interruption counts as one the system continues when the failed run's cause is system-owned and a later run that the dispatch gate refused before it started is a system recovery run (`recoveryRunSql`, which excludes a person's Retry). A failed run that was itself a person's Retry (`task_retry` or `explicitUserContinuation`) never qualifies this way. The 24-hour recency bound is unchanged. The claim-gate hand-off, the backlog pass and its deadline query all use this one predicate. - `recovery/continuation-ledger.ts`, `recovery/task-automatic-attempts.ts`: one shared rule, `startedOrPendingAutomaticRetry`. An automatic retry that the dispatch gate cancelled before it started (`cancelled`, no `startedAt`, `execution_reconciliation_required`) does not count toward the hourly cap in either ledger. - `recovery/assigned-work-sweep.ts`: - When the latest run was refused at dispatch, the sweep continues from the run it was refused for through the fallback. Before, it gave up. - A blocked task with an active hold and a recent refused-at-dispatch latest run is now listed as a suspect. The check hands it to the fallback before it answers `left_to_hold_owner`. A pending recovery becomes `waiting_on_recovery` with its recheck time. - Holds that wait for a person's configuration or confirmation are still excluded. - Review follow-up (`recovery/task-automatic-attempts.ts`, `recovery/assigned-work-sweep.ts`): - The task ledger counts a sweep's fresh-continuation record (`trigger: assigned_work`) itself. It no longer depends on a companion activity entry, and an activity entry marked `continuedFromSavedWork` is not counted a second time. Every sweep continuation counts once, whatever path recorded it. - When the sweep claims a continuation, it marks its claim entry `continuedFromSavedWork` inside the transaction that records the fresh continuation. A restart between the two writes can no longer leave one attempt counted twice. - The sweep's continuation behind a refused run spends the per-minute repair budget (`ASSIGNED_WORK_REPAIRS_PER_MINUTE`). The excess is deferred with a `retryAt`. - `recovery/stranded-notice.ts`, `recovery/service.ts`: when the task's execution blocker offers no Retry, the escalation notice and the recovery action's next action leave out every Retry instruction. They say that Paperclip is holding the task until its saved work is settled. - No change to the dispatch gate (`heartbeat/queue.ts`, the restore-hold predicate in `run-dispatch/adapters/postgres.ts`, or `execution-blocker.ts`). The recorded continuation settles the hold, so the delivered run is not refused again. ## Verification Red before the change, green after (embedded Postgres, real services: `terminalizeLegacyExecution`, the dispatch adapter's `cancelStaleQueuedRun`, `reconcileStrandedAssignedIssues`, `deliverReconciledExecutions`, `continueHeldExecutions`, `recoveryService().assignedWork`): - `server/src/services/fresh-continuation-fallback.test.ts` - "continues a follow-up … when the saved work was recovered before the system retry was refused at claim": red (`continuationDelivery: "invalidated"`, no continuation). Green: the continuation is recorded at the claim, and the stranded reconciler does not block the task. The delivered run is queued and passes the claim gate (`not_stale`). - "… recovered after the system retry was refused at claim": red (no pending check). Green: the claim records `exact_recovery_pending` / `saved_work_recovery_running`. When the recovery completes, the backlog pass continues the task, and the delivered run passes the claim gate. - "continues a follow-up … with its files in place, after its system retry was refused at claim": red, then green. This is the control without a restore failure. - "still leaves a person's failed Retry of a follow-up for that person …": stays `person_owned` / `retry_needed`, before and after. - "counts only automatic retries that started, or still may, against the task's hourly budget": new. - `server/src/__tests__/failed-continuation-continues-again.test.ts`, "the sweep continues a blocked task whose system retry was refused at claim behind a recovered saved-work hold": red (the task was not a suspect). Green: `repaired`, a pending continuation in the hold, the task back in `todo`, and no wake enqueued. The next check reports a pending intent. - Same file, "a continuation the sweep records behind a refused run counts once toward the task's hourly cap": red (5 counted after 5 attempts plus the new continuation), green (6, and the transient-retry cap check refuses the next attempt). - Same file, "continuations behind refused runs spend the sweep's per-minute repair budget": 20 tasks. Red (20 continued in one pass), green (10 continue, 10 are deferred with a `retryAt`, and the 10 continue in the next pass). - Same file, "a sweep continuation interrupted right after it was recorded still counts once": the continuation commits, then the post-commit activity write fails (a simulated server stop). Red: 6 counted for 5 real attempts. Green: 5 counted, and the next automatic retry is still allowed. - `server/src/services/recovery/stranded-notice.test.ts`, "never tells a person to press Retry when the task offers no Retry": new. Local runs on this branch, one file at a time: - `fresh-continuation-fallback.test.ts`: 77 passed. The suite's `afterAll` teardown hit the 30 s hook timeout on a heavily loaded machine. - `failed-continuation-continues-again.test.ts`: 32 passed - `assigned-work-invariant.test.ts`: 31 passed - `stranded-notice.test.ts`: 33 passed - CI on the first head (ce4431410b): every required check passed - `server` `tsc --noEmit`: clean - `pnpm verify --explain` selects most of the repository (2666 files), so required CI supplies the broad coverage. ## Risks - More tasks continue automatically. Each continuation goes through the existing fallback. The 24-hour recency bound, the per-task hourly cap (6), isolation of the next run, and every person-owned gate (person-owned failure codes, a person's Stop, review, a person-owned hold, a person's failed Retry) are unchanged. - The hourly cap now ignores automatic retries that never started. A task whose retries are refused at dispatch again and again is still bounded: the refusal hands the hold to the fallback, and the fallback records one continuation per source run. - Known separate gap, not changed here: after an interrupted preparation's holds are resolved (`settleInterruptedNativeBootstrap`), `scheduleBoundedRetry` can report `exhausted` or `blocked`, and nothing follows that up. The assigned-work sweep may now pick such a task up through the refused-at-dispatch path, but this pull request does not prove that. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, local test runs) ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
968 lines
67 KiB
TypeScript
968 lines
67 KiB
TypeScript
import { and, asc, desc, eq, gt, inArray, isNull, or, sql, type SQL } from "drizzle-orm";
|
|
import {
|
|
agentWakeupRequests, agents, approvals, companies, documentRevisions, documents, executionWorkspaces, heartbeatRuns, issueApprovals,
|
|
issueComments, issueRecoveryActions, issueThreadInteractions, issues, nativeRunFinalizations, toolInvocations, type Db,
|
|
} from "@paperclipai/db";
|
|
import {
|
|
EXECUTION_RECONCILIATION_CAUSES, readRemoteWorkspaceAuthority, remoteWorkspaceRunReferenceSchema, type ExecutionReconciliation,
|
|
} from "@paperclipai/shared";
|
|
import { logger } from "../middleware/logger.js";
|
|
import { logActivity } from "./activity-log.js";
|
|
import { taskHistoryReadsForRun } from "./queued-comment-delivery.js";
|
|
import { undeliveredLegacyUserCommentIds } from "./explicit-native-continuation.js";
|
|
import { queuedCommentIdsFromWakePayload } from "./issue-queued-comment-queue.js";
|
|
import { FRESH_CONTINUATION_SUPERSEDED_KEY } from "../agent-run-cancellation.js";
|
|
import { isAcknowledgedNativeReassignmentStop, isAcknowledgedNativeStop } from "./acknowledged-native-stop.js";
|
|
import { adapterExecutionControls } from "./adapter-execution-control.js";
|
|
import { isInterruptedNativePreparationCandidate } from "./cancelled-native-startup.js";
|
|
import { notifyDeferredWakesForIssue } from "./deferred-execution-notifications.js";
|
|
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
|
import { issueService } from "./issues.js";
|
|
import { holdOwnsBlockedStatus } from "./recovery-hold-blocked-status.js";
|
|
import { proveRunProcessGone } from "./run-process-owner.js";
|
|
import { hasPendingRunCleanup } from "./execution-blocker.js";
|
|
import { abandonUnverifiedRemoteWorkspaceEpoch } from "./remote-workspace-epoch-abandonment.js";
|
|
import { notifyRoutineExecutionStatus } from "./routine-execution-status-notifications.js";
|
|
import { notifyStatusCardIssueChange } from "./status-card-notifications.js";
|
|
import { readRunCancellation } from "./run-cancellation.js";
|
|
import { readSystemRecovery } from "./system-recovery-state.js";
|
|
import { RESTART_CLASS_FAILURE_CODES, continuationCauseIsSystemOwned, continuationCauseIsSystemOwnedSql, continuesAutomaticallySql, isPersonOwnedFailureCode, recoveryRunSql } from "./continuation-cause.js";
|
|
import { sharedContinuationAttempts } from "./recovery/continuation-ledger.js";
|
|
|
|
export { AUTOMATIC_CONTINUATION_WAKE_REASONS, PERSON_OWNED_FAILURE_CODES, isPersonOwnedFailureCode, isRecoveryRun } from "./continuation-cause.js";
|
|
|
|
/**
|
|
* Fresh-continuation fallback: exact resumption of an unfinished run is an
|
|
* optimization, never a requirement.
|
|
*
|
|
* When a run ended without finishing its work and nothing exact follows it
|
|
* (no resumed session, no replacement, no scheduled retry, no saved input),
|
|
* `ensureFreshContinuation` starts the task again in a fresh session from its
|
|
* last verified saved workspace and its conversation. It needs no person and
|
|
* no proof that the old executor stopped:
|
|
*
|
|
* - The old run keeps its own fencing. Its remote workspace epoch is abandoned
|
|
* when it still owns it, so its writer can never publish again and the next
|
|
* run restores the last committed save into a new root on a new allocation.
|
|
* The unverified files stay in the abandoned root (recorded on the
|
|
* workspace), unapplied.
|
|
* - The opening context of the continuation names the cut-off run and every
|
|
* action whose outcome is unknown, so the agent checks before repeating a
|
|
* side effect.
|
|
*
|
|
* Eligibility is by classification, not by an allowlist: every failure is
|
|
* eligible except the causes a person must resolve (`PERSON_OWNED_FAILURE_CODES`
|
|
* and the task-level waits in `personOwnedTaskWait`). A failure code added
|
|
* later falls back without a change here.
|
|
*
|
|
* Idempotent per source run: the task row lock serializes callers on every
|
|
* server, and the recorded `freshContinuation.sourceRunId` (or any newer run of
|
|
* the task) makes later calls a no-op. Bounded per task, not per failure: at
|
|
* most `FRESH_CONTINUATIONS_PER_HOUR` continuations in a rolling hour, then it
|
|
* backs off and is checked again; it never charges the task's failure budget.
|
|
*/
|
|
|
|
export const FRESH_CONTINUATION_SCHEMA = "paperclip.fresh-continuation.v1" as const;
|
|
export const FRESH_CONTINUATION_ACTOR = "fresh-continuation" as const;
|
|
/** Cause of the record written when the run left no hold of its own. Not an execution blocker. */
|
|
export const FRESH_CONTINUATION_CAUSE = "fresh_continuation" as const;
|
|
/** Evidence key for the last check that did not continue (and when to check again). */
|
|
export const FRESH_CONTINUATION_CHECK_KEY = "freshContinuationCheck" as const;
|
|
/** The restart lane's bound, reused: continuations per task per rolling hour. */
|
|
// Literal values (an import would be undefined under the heartbeat module's
|
|
// import cycle); a test pins them to SERVER_RESTART_RETRY_MAX_ATTEMPTS/_WINDOW_MS.
|
|
export const FRESH_CONTINUATIONS_PER_HOUR = 6;
|
|
export const FRESH_CONTINUATION_WINDOW_MS = 60 * 60_000;
|
|
const PERSON_RECHECK_MS = 60 * 60_000;
|
|
/** Consecutive pending checks (about 12 minutes) before the fallback stops waiting. */
|
|
/** Worst case from the first pending check to the fresh continuation. */
|
|
export const FRESH_CONTINUATION_PENDING_BOUND_MS = 10 * 60_000;
|
|
/** Longer cap while the exact-recovery owner shows progress (a new event or a renewed lease). */
|
|
export const FRESH_CONTINUATION_PROGRESS_BOUND_MS = 15 * 60_000;
|
|
/** Saved-work recovery that is running and advancing keeps its own budget. */
|
|
export const FRESH_CONTINUATION_SAVED_WORK_BOUND_MS = 30 * 60_000;
|
|
export const NATIVE_INTERRUPTION_NO_PROGRESS_DEADLINE_MS = FRESH_CONTINUATION_PROGRESS_BOUND_MS;
|
|
const EXACT_RECOVERY_RECHECK_MS = 60_000;
|
|
|
|
const UNFINISHED_STATUSES = ["failed", "timed_out", "interrupted", "cancelled"];
|
|
const LIVE_RUN_STATUSES = ["queued", "running", "scheduled_retry"];
|
|
const EXACT_RECOVERY_STATES = ["scheduled", "running", "awaiting_evidence"];
|
|
|
|
/**
|
|
* How the continuation is kept apart from the old executor:
|
|
* - `remote_epoch`: the old run's remote workspace epoch is fenced (abandoned, or
|
|
* already taken from it), so it can never publish; the next preparation
|
|
* restores the last committed save into a new root.
|
|
* - `writer_ended`: the old local writer is proven gone (no process, no
|
|
* unsettled allocation), so the task workspace is safe to reuse.
|
|
* - `fresh_isolated_workspace`: the old local writer is not proven gone, so the
|
|
* continuation gets a new isolated workspace and never writes the old
|
|
* directory. Nothing waits for that proof.
|
|
*/
|
|
export type FreshContinuationIsolation = "remote_epoch" | "writer_ended" | "fresh_isolated_workspace";
|
|
|
|
export type FreshContinuationTrigger = "recovery_refused" | "held_recovery" | "continuation_wake" | "assigned_work";
|
|
|
|
export type FreshContinuationResult =
|
|
| { kind: "continued"; recoveryActionId: string; delivery: "recorded" | "caller_owns" | "saved_input"; isolation: FreshContinuationIsolation }
|
|
| { kind: "already_continued"; recoveryActionId: string }
|
|
| { kind: "successor_exists" }
|
|
| { kind: "dependency_wait" }
|
|
| { kind: "person_owned"; reason: string }
|
|
| { kind: "backoff"; retryAt: Date }
|
|
| { kind: "exact_recovery_pending"; reason: string; retryAt: Date; progressAt?: number; savedWorkSignature?: string }
|
|
| { kind: "not_applicable"; reason: string };
|
|
|
|
type Input = {
|
|
companyId: string;
|
|
runId: string;
|
|
trigger: FreshContinuationTrigger;
|
|
now?: Date;
|
|
/**
|
|
* `record`: record one durable continuation that the execution-continuation
|
|
* worker delivers. `caller_owns`: the caller is admitting a wake for this
|
|
* task in the same transaction, and that wake is the continuation.
|
|
*/
|
|
delivery?: "record" | "caller_owns";
|
|
};
|
|
|
|
const object = (value: unknown): Record<string, unknown> =>
|
|
value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
|
|
|
|
function holdRunId(evidence: Record<string, unknown>): string | null {
|
|
const value = evidence.runId ?? evidence.sourceRunId;
|
|
return typeof value === "string" ? value : null;
|
|
}
|
|
|
|
/** Task-level waits that belong to a person. Read under the task lock. */
|
|
async function personOwnedTaskWait(tx: Db, input: { companyId: string; issue: typeof issues.$inferSelect;
|
|
run: typeof heartbeatRuns.$inferSelect }): Promise<string | null> {
|
|
const { companyId, issue, run } = input;
|
|
const [approval] = await tx.select({ id: approvals.id }).from(issueApprovals).innerJoin(approvals, and(
|
|
eq(approvals.id, issueApprovals.approvalId), eq(approvals.companyId, companyId),
|
|
)).where(and(eq(issueApprovals.companyId, companyId), eq(issueApprovals.issueId, issue.id),
|
|
inArray(approvals.status, ["pending", "revision_requested"]))).limit(1);
|
|
if (approval) return "approval_pending";
|
|
const [question] = await tx.select({ id: issueThreadInteractions.id }).from(issueThreadInteractions).where(and(
|
|
eq(issueThreadInteractions.companyId, companyId), eq(issueThreadInteractions.issueId, issue.id),
|
|
eq(issueThreadInteractions.status, "pending"))).limit(1);
|
|
if (question) return "question_pending";
|
|
const [agent] = await tx.select({ status: agents.status, lifecycleState: agents.lifecycleState }).from(agents)
|
|
.where(and(eq(agents.companyId, companyId), eq(agents.id, run.agentId)));
|
|
if (!agent || ["paused", "terminated", "pending_approval"].includes(agent.status) || agent.lifecycleState !== "ready")
|
|
return "agent_not_available";
|
|
const [company] = await tx.select({ status: companies.status }).from(companies).where(eq(companies.id, companyId));
|
|
if (company && company.status !== "active") return "company_paused";
|
|
return null;
|
|
}
|
|
|
|
/** Actions of the source run that ended with an unknown outcome. */
|
|
async function unverifiedActions(tx: Db, run: typeof heartbeatRuns.$inferSelect): Promise<string[]> {
|
|
const names = new Set<string>();
|
|
for (const receipt of Object.values(object(object(run.resultJson).apiToolReceipts)).map(object)) {
|
|
if (receipt.state !== "completed" || object(receipt.result).outcome === "unknown")
|
|
names.add(typeof receipt.toolName === "string" ? receipt.toolName : "a Paperclip API call");
|
|
}
|
|
const rows = await tx.select({ toolName: toolInvocations.toolName }).from(toolInvocations).where(and(
|
|
eq(toolInvocations.companyId, run.companyId), eq(toolInvocations.runId, run.id),
|
|
sql`${toolInvocations.riskLevel} is distinct from 'read'`,
|
|
inArray(toolInvocations.status, ["executing", "failed", "timed_out"]))).limit(20);
|
|
for (const row of rows) names.add(row.toolName);
|
|
return [...names].slice(0, 20);
|
|
}
|
|
|
|
export type FreshContinuationCreatedObjects = { subtasks: string[]; comments: string[]; documents: string[]; readFollowUps?: string[];
|
|
editedFollowUps?: string[] };
|
|
|
|
async function taskHistoryReadsOfRun(tx: Db, run: Pick<typeof heartbeatRuns.$inferSelect, "id" | "companyId">) {
|
|
const reads = await taskHistoryReadsForRun(tx, { companyId: run.companyId, runId: run.id });
|
|
return new Map(reads.map(read => [read.commentId, read]));
|
|
}
|
|
|
|
const quoted = (value: string, max = 80) => `"${value.replace(/\s+/g, " ").trim().slice(0, max)}${value.length > max ? "..." : ""}"`;
|
|
|
|
async function createdObjects(tx: Db, run: typeof heartbeatRuns.$inferSelect): Promise<FreshContinuationCreatedObjects> {
|
|
const [children, comments, revisions] = await Promise.all([
|
|
tx.select({ identifier: issues.identifier, title: issues.title }).from(issues).where(and(
|
|
eq(issues.companyId, run.companyId), eq(issues.originRunId, run.id), isNull(issues.hiddenAt)))
|
|
.orderBy(asc(issues.createdAt)).limit(20),
|
|
tx.select({ body: issueComments.body }).from(issueComments).where(and(
|
|
eq(issueComments.companyId, run.companyId), eq(issueComments.createdByRunId, run.id), isNull(issueComments.deletedAt)))
|
|
.orderBy(asc(issueComments.createdAt)).limit(10),
|
|
tx.selectDistinct({ title: documents.title }).from(documentRevisions)
|
|
.innerJoin(documents, eq(documents.id, documentRevisions.documentId))
|
|
.where(and(eq(documentRevisions.companyId, run.companyId), eq(documentRevisions.createdByRunId, run.id))).limit(10),
|
|
]);
|
|
const reads = await taskHistoryReadsOfRun(tx, run);
|
|
const readComments = reads.size ? await tx.select({ id: issueComments.id, body: issueComments.body, updatedAt: issueComments.updatedAt })
|
|
.from(issueComments).where(and(
|
|
eq(issueComments.companyId, run.companyId), inArray(issueComments.id, [...reads.keys()]),
|
|
eq(issueComments.authorType, "user"), isNull(issueComments.deletedAt))).limit(10) : [];
|
|
const unchanged = (comment: (typeof readComments)[number]) => comment.updatedAt.getTime() <= reads.get(comment.id)!.revision.getTime();
|
|
const edited = readComments.filter(comment => !unchanged(comment));
|
|
return {
|
|
readFollowUps: readComments.filter(unchanged)
|
|
.map(comment => `${quoted(comment.body)} (read at ${reads.get(comment.id)!.readAt.toISOString().slice(11, 16)} UTC)`),
|
|
...(edited.length ? { editedFollowUps: edited.map(comment => quoted(comment.body)) } : {}),
|
|
subtasks: children.map(child => `${child.identifier ?? "subtask"} ${quoted(child.title)}`),
|
|
comments: comments.map(comment => quoted(comment.body)),
|
|
documents: revisions.map(revision => quoted(revision.title ?? "untitled document")),
|
|
};
|
|
}
|
|
|
|
/** The opening context of the continuation. It names no person who must act. */
|
|
export function freshContinuationNote(input: { runId: string; unverified: string[]; workspace: "restored_last_save" | "kept";
|
|
created?: FreshContinuationCreatedObjects }) {
|
|
const actions = input.unverified.length
|
|
? `These actions of that run have unverified outcomes: ${input.unverified.join(", ")}. For each one, first check whether it took effect.`
|
|
: "No action of that run is recorded with an unknown outcome; check the task's recent history before repeating any side effect.";
|
|
const files = input.workspace === "restored_last_save"
|
|
? "Its unsaved file changes could not be verified, so this run starts from the last saved version of the workspace; those changes were not applied."
|
|
: "This run starts from the task's saved workspace.";
|
|
// How to proceed. Nobody has to restore the previous attempt: the agent
|
|
// checks, redoes what is missing, and asks only through the normal approval.
|
|
const guidance = "Completed actions with recorded results are still not repeated; this applies only to the actions above, whose outcome is unknown. "
|
|
+ "If an action did not take effect, or repeating it is safe, do it again and continue the task. "
|
|
+ "If repeating it would be a governed or irreversible action, request it through the normal approval step. "
|
|
+ "Nothing about the previous attempt needs to be restored before you continue.";
|
|
const created = input.created;
|
|
const read = created?.readFollowUps?.length
|
|
? ` That run had already read these messages from a person: ${created.readFollowUps.join(", ")}. Answer each once; what it did after reading is below.`
|
|
: "";
|
|
const edited = created?.editedFollowUps?.length
|
|
? ` A person edited these messages after that run read them, so it never saw their current text; treat them as new: ${created.editedFollowUps.join(", ")}.`
|
|
: "";
|
|
const existing = created &&(created.subtasks.length || created.comments.length || created.documents.length)
|
|
? ` That run already did these, and they exist now: ${[
|
|
created.subtasks.length ? `created subtasks ${created.subtasks.join(", ")}` : null,
|
|
created.comments.length ? `posted comments ${created.comments.join(", ")}` : null,
|
|
created.documents.length ? `saved documents ${created.documents.join(", ")}` : null,
|
|
].filter(Boolean).join("; ")}. Do not create or post them again; read them and continue from where they leave off.`
|
|
: "";
|
|
return `The previous attempt (run ${input.runId}) was cut off before it finished. ${files}${read}${edited}${existing} ${actions} ${guidance}`;
|
|
}
|
|
|
|
const USER_TEXT = "The previous run was cut off. Paperclip continues the task in a fresh run from its last saved work.";
|
|
|
|
/** Abandons the source run's remote workspace epoch when it still owns it. */
|
|
async function fenceSourceWorkspace(tx: Db, run: typeof heartbeatRuns.$inferSelect) {
|
|
const parsed = remoteWorkspaceRunReferenceSchema.safeParse(object(run.runnerProfileJson).remoteWorkspace);
|
|
if (!parsed.success || parsed.data.state !== "prepared") return null;
|
|
const reference = parsed.data;
|
|
const [workspace] = await tx.select({ metadata: executionWorkspaces.metadata }).from(executionWorkspaces).where(and(
|
|
eq(executionWorkspaces.companyId, run.companyId), eq(executionWorkspaces.id, reference.workspaceId)));
|
|
let authority;
|
|
try { authority = readRemoteWorkspaceAuthority(workspace?.metadata); } catch { return null; }
|
|
if (!authority?.location || authority.writerRunId !== run.id || authority.epoch !== reference.epoch) return null;
|
|
try {
|
|
const abandoned = await tx.transaction(savepoint => abandonUnverifiedRemoteWorkspaceEpoch(savepoint, {
|
|
companyId: run.companyId, workspaceId: reference.workspaceId, runId: run.id, epoch: authority.epoch,
|
|
snapshotId: authority.snapshotId, location: authority.location! }));
|
|
return { workspaceId: reference.workspaceId, abandonedEpoch: authority.epoch, epoch: abandoned.authority.epoch,
|
|
snapshotId: authority.snapshotId,
|
|
quarantine: { state: "retained_unapplied" as const, remoteCwd: authority.location.remoteCwd,
|
|
providerLeaseId: authority.location.providerLeaseId } };
|
|
} catch (error) {
|
|
if (error instanceof Error && error.message === "remote_workspace_authority_changed") return null;
|
|
throw error;
|
|
}
|
|
}
|
|
|
|
/**
|
|
* The one distinction this module turns on: a real human gate versus system
|
|
* uncertainty.
|
|
*
|
|
* A human gate is a decision a person actually made or still owes: a pending
|
|
* approval or question (`personOwnedTaskWait`), a hold a user owns, or an
|
|
* operator decision recorded on the hold (an owner-gone attestation, an
|
|
* explicit continuation, a reconciliation). The fallback never retires or
|
|
* bypasses one.
|
|
*
|
|
* Everything else is system uncertainty, however it is labelled: a
|
|
* no-replay record (`automaticRecovery.replay: blocked`, "outcome unknown after
|
|
* an interruption"), a hold whose owner type is `board` because nobody else was
|
|
* assigned, or a controller whose process space cannot be verified. Owner type
|
|
* alone never proves that a person must act. For these the fallback starts a
|
|
* fresh run, never a replay: the interrupted turn is not re-sent, no tool call
|
|
* or write is re-executed, and any governed action goes through its approval
|
|
* gate again.
|
|
*/
|
|
export function isHumanGate(hold: Pick<typeof issueRecoveryActions.$inferSelect, "ownerUserId" | "evidence">): boolean {
|
|
return Boolean(hold.ownerUserId || hold.evidence.ownerGoneAttestation || hold.evidence.explicitUserContinuation
|
|
|| hold.evidence.executionReconciliation);
|
|
}
|
|
|
|
const isRestoreHold = (hold: typeof issueRecoveryActions.$inferSelect) =>
|
|
hold.evidence.workspaceRestoreFailure !== undefined || hold.evidence.workspaceRestoreRecovery !== undefined;
|
|
const blocksExecution = (hold: typeof issueRecoveryActions.$inferSelect) =>
|
|
["active", "escalated"].includes(hold.status) || object(hold.evidence.automaticRecovery).replay === "blocked";
|
|
|
|
/** A reconciliation hold that blocks the task and that this decision settles. */
|
|
function isOwnedReconciliationHold(hold: typeof issueRecoveryActions.$inferSelect) {
|
|
if (hold.kind === "stranded_assigned_issue") return false;
|
|
if (!(EXECUTION_RECONCILIATION_CAUSES as readonly string[]).includes(hold.cause) || !blocksExecution(hold)) return false;
|
|
if (isHumanGate(hold)) return false;
|
|
// A restore hold is owned only once its saved-work recovery completed.
|
|
return !isRestoreHold(hold) || readSystemRecovery(hold.evidence)?.state === "complete";
|
|
}
|
|
|
|
/**
|
|
* The old local writer's state. `alive`: its process is running on this host,
|
|
* so it is still executing and is settled by its own owner (stop, reaper)
|
|
* first. `ended`: no process this host can see and no allocation or workspace
|
|
* finalization left unsettled. `unproven`: anything else (another host, an
|
|
* unsettled allocation); the continuation is isolated from it, never waits.
|
|
*/
|
|
async function localWriterState(tx: Db, run: typeof heartbeatRuns.$inferSelect): Promise<"alive" | "ended" | "unproven"> {
|
|
// The shared owner proof compares the recorded process identity (host, PID
|
|
// space, start time), so a recycled PID is never mistaken for the old writer.
|
|
const proof = await proveRunProcessGone(run, "fresh_continuation");
|
|
if (proof?.verdict === "alive") return "alive";
|
|
if (proof?.verdict === "unprovable" || await hasPendingRunCleanup(tx, run.companyId, run.id)) return "unproven";
|
|
return "ended";
|
|
}
|
|
|
|
/** The workspace selection a delivered continuation carries (read only when its context names it). */
|
|
export async function readFreshContinuationReadiness(db: Db, run: Pick<typeof heartbeatRuns.$inferSelect,
|
|
"id" | "companyId" | "contextSnapshot">, issueId: string) {
|
|
const recordId = object(run.contextSnapshot).freshContinuationIsolation;
|
|
if (typeof recordId !== "string" || !/^[0-9a-f-]{36}$/i.test(recordId)) return null;
|
|
const [record] = await db.select({ evidence: issueRecoveryActions.evidence }).from(issueRecoveryActions).where(and(
|
|
eq(issueRecoveryActions.companyId, run.companyId), eq(issueRecoveryActions.sourceIssueId, issueId),
|
|
sql`${issueRecoveryActions.id}::text = ${recordId}`,
|
|
sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,isolation}' = 'fresh_isolated_workspace'`)).limit(1);
|
|
const sourceRunId = object(record?.evidence.freshContinuation).sourceRunId;
|
|
if (typeof sourceRunId !== "string") return null;
|
|
return { mode: "fresh_isolated_workspace" as const, retryIntentId: recordId, sourceRunId, retainedSourceIds: [] as string[] };
|
|
}
|
|
|
|
/**
|
|
* True when a wake continues a cut-off run in a new isolated workspace. The
|
|
* fallback cleared the task's workspace binding on purpose, and the run
|
|
* prepares a new workspace itself, so admission must not refuse the wake for
|
|
* lacking that binding (it would refuse it forever).
|
|
*/
|
|
export async function isFreshIsolatedContinuationWake(db: Db, input: { companyId: string; issueId: string;
|
|
contextSnapshot: Record<string, unknown> | null | undefined }) {
|
|
if (typeof object(input.contextSnapshot).freshContinuationIsolation !== "string") return false;
|
|
return (await readFreshContinuationReadiness(db, { id: "", companyId: input.companyId,
|
|
contextSnapshot: input.contextSnapshot ?? {} }, input.issueId)) !== null;
|
|
}
|
|
|
|
/**
|
|
* True when the fallback replaced this run and its continuation reuses none of
|
|
* the run's workspace: a new isolated workspace or an abandoned remote epoch.
|
|
* Holds that only protect the old executor's resources then no longer stand
|
|
* between the task and its next turn.
|
|
*/
|
|
export async function freshContinuationIsolatesRun(db: Db, input: { companyId: string; runId: string }): Promise<false | {
|
|
/** The isolated continuation is recorded but not yet delivered as a run. */
|
|
continuationPending: boolean;
|
|
}> {
|
|
const [record] = await db.select({ isolation: sql<string>`${issueRecoveryActions.evidence}#>>'{freshContinuation,isolation}'`,
|
|
delivery: sql<string | null>`${issueRecoveryActions.evidence}->>'continuationDelivery'` }).from(issueRecoveryActions).where(and(
|
|
eq(issueRecoveryActions.companyId, input.companyId),
|
|
sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,sourceRunId}' = ${input.runId}`,
|
|
sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,isolation}' in ('fresh_isolated_workspace', 'remote_epoch')`)).limit(1);
|
|
if (!record) return false;
|
|
// Only the recorded continuation carries the isolation marker that prepares
|
|
// the new workspace, so while it is pending it must be the task's next run.
|
|
return { continuationPending: record.isolation === "fresh_isolated_workspace" && record.delivery === "pending" };
|
|
}
|
|
|
|
export async function ensureFreshContinuation(db: Db, input: Input): Promise<FreshContinuationResult> {
|
|
return db.transaction(tx => continueFreshInTransaction(tx as unknown as Db, input));
|
|
}
|
|
|
|
/**
|
|
* The fallback inside the caller's transaction. Locks in the shared order
|
|
* (task, coordinator, run, recovery actions, workspace); rows the caller
|
|
* already locked are locked again harmlessly.
|
|
*/
|
|
export async function continueFreshInTransaction(tx: Db, input: Input): Promise<FreshContinuationResult> {
|
|
const now = input.now && Number.isFinite(input.now.getTime()) ? input.now : new Date();
|
|
const { companyId } = input;
|
|
const [peek] = await tx.select({ issueId: heartbeatRuns.issueId, nativeIssueId: heartbeatRuns.nativeIssueId,
|
|
contextSnapshot: heartbeatRuns.contextSnapshot }).from(heartbeatRuns)
|
|
.where(and(eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, input.runId)));
|
|
if (!peek) return { kind: "not_applicable", reason: "run_missing" };
|
|
const contextIssueId = object(peek.contextSnapshot).issueId;
|
|
const issueId = peek.nativeIssueId ?? peek.issueId ?? (typeof contextIssueId === "string" ? contextIssueId : null);
|
|
if (!issueId) return { kind: "not_applicable", reason: "run_has_no_task" };
|
|
const [issue] = await tx.select().from(issues).where(and(eq(issues.companyId, companyId), eq(issues.id, issueId))).for("update");
|
|
const [coordinator] = await tx.select().from(nativeRunFinalizations).where(and(
|
|
eq(nativeRunFinalizations.companyId, companyId), eq(nativeRunFinalizations.runId, input.runId))).for("update");
|
|
const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId),
|
|
eq(heartbeatRuns.id, input.runId))).for("update");
|
|
if (!issue || !run) return { kind: "not_applicable", reason: "run_missing" };
|
|
if (!UNFINISHED_STATUSES.includes(run.status) || !run.finishedAt) return { kind: "not_applicable", reason: "run_not_unfinished" };
|
|
if (!issue.assigneeAgentId || issue.assigneeAgentId !== run.agentId || issue.assigneeUserId
|
|
|| ["done", "cancelled"].includes(issue.status)) return { kind: "not_applicable", reason: "task_changed" };
|
|
|
|
const holds = await tx.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, companyId),
|
|
eq(issueRecoveryActions.sourceIssueId, issue.id),
|
|
or(sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`, sql`${issueRecoveryActions.evidence}->>'sourceRunId' = ${run.id}`)))
|
|
.orderBy(asc(issueRecoveryActions.createdAt), asc(issueRecoveryActions.id)).for("update");
|
|
const prior = holds.find(hold => object(hold.evidence.freshContinuation).sourceRunId === run.id);
|
|
if (prior) {
|
|
await settleLateHolds(tx, { companyId, issue, run, prior, holds: holds.filter(isOwnedReconciliationHold), now });
|
|
return { kind: "already_continued", recoveryActionId: prior.id };
|
|
}
|
|
// The holds this decision owns: reconciliation holds that still block the
|
|
// task. Workspace restore/repair holds stay with workspace recovery, and
|
|
// resolved history is never rewritten.
|
|
const ownedHolds = holds.filter(isOwnedReconciliationHold);
|
|
|
|
const [refused] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, companyId), sql`${heartbeatRuns.id} <> ${run.id}`,
|
|
or(eq(heartbeatRuns.nativeIssueId, issue.id), eq(heartbeatRuns.issueId, issue.id),
|
|
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}`),
|
|
sql`(${heartbeatRuns.createdAt}, ${heartbeatRuns.id}) > (${run.createdAt.toISOString()}::timestamptz, ${run.id}::uuid)`,
|
|
eq(heartbeatRuns.status, "cancelled"), isNull(heartbeatRuns.startedAt), eq(heartbeatRuns.errorCode, "execution_reconciliation_required")))
|
|
.orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
|
|
let personAsked = false;
|
|
if (refused) {
|
|
if (!holds.length) return { kind: "successor_exists" };
|
|
const rule = interruptionEligibility({ now, companyId: sql`${companyId}`, issueId: sql`${issue.id}`, sourceRunId: sql`${run.id}` });
|
|
const [decision] = await tx.execute<{ recent: boolean; person: boolean }>(sql`select ${rule.recent} as recent, ${rule.personAsked} as person`);
|
|
personAsked = decision?.person === true;
|
|
if (!personAsked && decision?.recent !== true) return { kind: "person_owned", reason: "retry_needed" };
|
|
}
|
|
|
|
// Every wait is bounded in time: about 10 minutes from the first pending
|
|
// check, or 15 while the exact-recovery owner shows progress (a new run
|
|
// event, output or lease renewal since the last check). Then the fallback
|
|
// stops waiting and continues in isolation (writes revoked, new root).
|
|
const pendingMarker = holds.map(hold => object(hold.evidence[FRESH_CONTINUATION_CHECK_KEY]))
|
|
.filter(check => check.kind === "exact_recovery_pending" && typeof check.firstPendingAt === "number")
|
|
.sort((a, b) => Number(a.firstPendingAt) - Number(b.firstPendingAt))[0];
|
|
// Saved-work recovery shows progress when its generation advances (revision
|
|
// or attempt) or its hold is updated; the run and coordinator when they
|
|
// emit output, renew a lease or update.
|
|
const savedWork = holds.map(hold => readSystemRecovery(hold.evidence)).filter(recovery => recovery !== null);
|
|
const progressAt = Math.max(run.updatedAt?.getTime() ?? 0, run.lastOutputAt?.getTime() ?? 0,
|
|
coordinator?.updatedAt?.getTime() ?? 0, coordinator?.leaseExpiresAt?.getTime() ?? 0,
|
|
...holds.filter(hold => readSystemRecovery(hold.evidence)).map(hold => hold.updatedAt.getTime()));
|
|
const savedWorkSignature = savedWork.map(recovery => `${recovery!.generationId}:${recovery!.revision}:${recovery!.attempts}:${recovery!.state}`)
|
|
.sort().join("|");
|
|
const sourceAgeMs = now.getTime() - run.finishedAt.getTime();
|
|
const waitedMs = pendingMarker ? now.getTime() - Number(pendingMarker.firstPendingAt)
|
|
: sourceAgeMs <= BACKLOG_CONTINUATION_MAX_AGE_MS ? sourceAgeMs : 0;
|
|
const progressing = Boolean(pendingMarker && ((typeof pendingMarker.progressAt === "number" && progressAt > pendingMarker.progressAt)
|
|
|| (typeof pendingMarker.savedWorkSignature === "string" && savedWorkSignature !== pendingMarker.savedWorkSignature)));
|
|
// Saved-work recovery that is running and advancing keeps its own 30-minute
|
|
// budget: it may still rescue unsaved files. Anything idle gets the normal bound.
|
|
const savedWorkAdvancing = progressing && savedWork.some(recovery => recovery!.state === "running");
|
|
const boundedAfter = (waited: number) => savedWorkAdvancing ? waited >= FRESH_CONTINUATION_SAVED_WORK_BOUND_MS
|
|
: waited >= FRESH_CONTINUATION_PROGRESS_BOUND_MS || (waited >= FRESH_CONTINUATION_PENDING_BOUND_MS && !progressing)
|
|
|| (personAsked && !savedWork.some(recovery => EXACT_RECOVERY_STATES.includes(recovery!.state)));
|
|
const waitBounded = boundedAfter(waitedMs);
|
|
const markerBounded = boundedAfter(pendingMarker ? waitedMs : 0);
|
|
// Rechecks back off 1, 2, 3, 4 minutes, so the bound is reached on time.
|
|
const step = Math.min(4, (typeof pendingMarker?.checks === "number" ? pendingMarker.checks : 0) + 1) * EXACT_RECOVERY_RECHECK_MS;
|
|
let waitSkipped = false;
|
|
const pastBound = () => markerBounded || waitSkipped;
|
|
const pending = (reason: string, retryAt: Date): FreshContinuationResult | null => waitBounded ? (waitSkipped = true, null)
|
|
: !pendingMarker && sourceAgeMs > BACKLOG_CONTINUATION_MAX_AGE_MS ? { kind: "person_owned", reason: "retry_needed" }
|
|
: { kind: "exact_recovery_pending", reason, progressAt, savedWorkSignature,
|
|
retryAt: new Date(Math.min(retryAt.getTime(), now.getTime() + step)) };
|
|
|
|
// (a) Something already follows this run: a newer run of the task, a linked
|
|
// replacement, or an operator's continuation.
|
|
const [newer] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, companyId), sql`${heartbeatRuns.id} <> ${run.id}`,
|
|
or(eq(heartbeatRuns.nativeIssueId, issue.id), eq(heartbeatRuns.issueId, issue.id),
|
|
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}`),
|
|
sql`not coalesce(${heartbeatRuns.status} = 'cancelled' and ${heartbeatRuns.startedAt} is null
|
|
and ${heartbeatRuns.errorCode} = 'execution_reconciliation_required', false)`,
|
|
or(inArray(heartbeatRuns.status, LIVE_RUN_STATUSES),
|
|
sql`(${heartbeatRuns.createdAt}, ${heartbeatRuns.id}) > (${run.createdAt.toISOString()}::timestamptz, ${run.id}::uuid)`))).limit(1);
|
|
if (newer || typeof object(coordinator?.failureDetail).successorRunId === "string"
|
|
|| holds.some(hold => hold.evidence.executionReconciliation && ["pending", "delivered", "delegated"]
|
|
.includes(String(hold.evidence.continuationDelivery)))) return { kind: "successor_exists" };
|
|
|
|
// Exact recovery that is still under way keeps its chance, for a bounded
|
|
// number of checks; each owner below also bounds itself.
|
|
const recheck = new Date(now.getTime() + EXACT_RECOVERY_RECHECK_MS);
|
|
const waits: Array<FreshContinuationResult | null> = [];
|
|
if (adapterExecutionControls.has(run.id)) waits.push(pending("execution_settling", recheck));
|
|
if (coordinator?.leaseOwner || (coordinator?.resultId && coordinator.phase !== "committed")
|
|
|| (coordinator?.nextAttemptAt && coordinator.nextAttemptAt > now))
|
|
waits.push(pending("native_finalizer_active",
|
|
coordinator?.nextAttemptAt && coordinator.nextAttemptAt > now ? coordinator.nextAttemptAt : recheck));
|
|
if (run.controllerLeaseExpiresAt && run.controllerLeaseExpiresAt > now)
|
|
waits.push(pending("controller_lease_live", run.controllerLeaseExpiresAt));
|
|
// A preparation that never started provider work has its own bounded retry
|
|
// lane (the ownership pass). It gets the first chance; the fallback follows.
|
|
if (isInterruptedNativePreparationCandidate(run) && !holds.some(hold => hold.evidence[FRESH_CONTINUATION_CHECK_KEY]))
|
|
waits.push(pending("interrupted_preparation_lane", recheck));
|
|
for (const hold of holds) {
|
|
const recovery = readSystemRecovery(hold.evidence);
|
|
if (recovery && EXACT_RECOVERY_STATES.includes(recovery.state))
|
|
waits.push(pending("saved_work_recovery_running",
|
|
new Date(Math.max(recheck.getTime(), recovery.nextAttemptAt ?? 0))));
|
|
if (!isOwnedReconciliationHold(hold) && isRestoreHold(hold) && blocksExecution(hold) && recovery?.state !== "complete")
|
|
waits.push(pending("workspace_recovery_owns_restore", recheck));
|
|
const retainedCheck = object(hold.evidence.retainedSourceFallbackCheck);
|
|
if (recovery?.state === "exhausted" && typeof retainedCheck.nextCheckAt === "number" && retainedCheck.nextCheckAt > now.getTime())
|
|
waits.push(pending("retained_source_fallback_pending", new Date(retainedCheck.nextCheckAt)));
|
|
}
|
|
const firstWait = waits.find(wait => wait !== null);
|
|
if (firstWait) return firstWait;
|
|
// Past the bound, a hold that workspace recovery still owns is settled too:
|
|
// the continuation never uses that workspace (isolation below).
|
|
const owned = pastBound() ? holds.filter(hold => isOwnedReconciliationHold(hold)
|
|
|| (isRestoreHold(hold) && blocksExecution(hold) && !isHumanGate(hold))) : ownedHolds;
|
|
|
|
// (b) Causes that are the person's.
|
|
if (isPersonOwnedFailureCode(run.errorCode) || (run.errorCode && !continuationCauseIsSystemOwned(run)))
|
|
return { kind: "person_owned", reason: run.errorCode! };
|
|
// A Stop a person (or an agent acting on the task) asked for is never undone.
|
|
const cancellation = readRunCancellation(run.resultJson);
|
|
if ((cancellation && ["user", "agent"].includes(cancellation.initiator.type))
|
|
|| object(run.resultJson).operatorInterrupted === true
|
|
|| (run.status === "cancelled" && (isAcknowledgedNativeStop(run) || isAcknowledgedNativeReassignmentStop(run)
|
|
// A cancellation that recorded neither its cause nor a system initiator may
|
|
// be a person's Stop saved by an older release; it is never undone.
|
|
|| (!cancellation && !run.errorCode))))
|
|
return { kind: "person_owned", reason: "stopped_on_request" };
|
|
// A task in review waits for its reviewer.
|
|
if (issue.status === "in_review") return { kind: "person_owned", reason: "in_review" };
|
|
// A recorded operator reconciliation that already delivered a continuation
|
|
// was handled above (successor_exists); any other human gate stays.
|
|
if (holds.some(hold => blocksExecution(hold) && isHumanGate(hold) && !hold.evidence.executionReconciliation)
|
|
|| holds.some(hold => hold.ownerUserId || hold.evidence.ownerGoneAttestation || hold.evidence.explicitUserContinuation))
|
|
return { kind: "person_owned", reason: "operator_decision_recorded" };
|
|
const taskWait = await personOwnedTaskWait(tx, { companyId, issue, run });
|
|
if (taskWait) return { kind: "person_owned", reason: taskWait };
|
|
let ownsBlock = false;
|
|
if (issue.status === "blocked") {
|
|
const receipts = holds.filter(hold => hold.evidence.nativeFailureBlock !== undefined || hold.evidence.legacyCancellationBlock !== undefined
|
|
|| hold.evidence.nativeBootstrapFailureBlock !== undefined);
|
|
ownsBlock = (receipts.length ? receipts : holds).some(hold => holdOwnsBlockedStatus(hold, issue, run))
|
|
|| receipts.some(hold => [hold.evidence.legacyCancellationBlock, hold.evidence.nativeBootstrapFailureBlock]
|
|
.some(block => object(block).runId === run.id && object(block).statusVersion === Number(issue.statusVersion)))
|
|
// A run that left no hold: its terminal failure set the block and nothing changed it since.
|
|
|| (!holds.length && issue.lastStatusDecisionId === null && issue.blockedTransitionAt !== null
|
|
&& issue.blockedTransitionAt.getTime() >= run.finishedAt.getTime() - 1_000);
|
|
if (!ownsBlock) return { kind: "person_owned", reason: "blocked_by_another_decision" };
|
|
const readiness = await issueService(tx).listDependencyReadiness(companyId, [issue.id], tx);
|
|
if (readiness.get(issue.id)?.isDependencyReady === false) return { kind: "dependency_wait" };
|
|
}
|
|
|
|
// (c) Bounded per task: the restart lane's hourly cap, then back off.
|
|
const attempts = await sharedContinuationAttempts(tx, companyId, issue.id, now);
|
|
if (attempts.length >= FRESH_CONTINUATIONS_PER_HOUR) {
|
|
return { kind: "backoff", retryAt: new Date(attempts[0]!.getTime() + FRESH_CONTINUATION_WINDOW_MS) };
|
|
}
|
|
|
|
// Saved user input (a message or a Retry the person pressed) continues the
|
|
// task through normal admission once nothing blocks it; a second, automatic
|
|
// continuation would run after it.
|
|
const savedWakes = await tx.select({ id: agentWakeupRequests.id, payload: agentWakeupRequests.payload }).from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
eq(agentWakeupRequests.requestedByActorType, "user"), sql`${agentWakeupRequests.payload}->>'issueId' = ${issue.id}`)).limit(5);
|
|
// Saved input continues the task only while it still holds something the
|
|
// agent has not received: a message the cut-off run already took into its
|
|
// turn is settled as delivered when promoted, and then nothing would run.
|
|
let savedInput: { id: string } | undefined;
|
|
for (const wake of savedWakes) {
|
|
const commentIds = queuedCommentIdsFromWakePayload(wake.payload);
|
|
if (refused && !commentIds.length && object(wake.payload).taskRetry) continue;
|
|
if (!commentIds.length || (await undeliveredLegacyUserCommentIds(tx, companyId, issue.id, run.agentId, commentIds)).length) {
|
|
savedInput = wake;
|
|
break;
|
|
}
|
|
}
|
|
let delivery: "recorded" | "caller_owns" | "saved_input" = input.delivery === "caller_owns" ? "caller_owns"
|
|
: savedInput ? "saved_input" : "recorded";
|
|
|
|
// (d) Continue: fence the old workspace epoch, settle every hold of the run,
|
|
// release the Blocked status this failure owns, and record one continuation.
|
|
const remoteSource = remoteWorkspaceRunReferenceSchema.safeParse(object(run.runnerProfileJson).remoteWorkspace).success;
|
|
const writer = remoteSource ? null : await localWriterState(tx, run);
|
|
// A local process still running on this host is executing: its own owner
|
|
// (the reaper or a Stop) settles it first, then the fallback continues.
|
|
const aliveWait = writer === "alive" ? pending("previous_process_running", new Date(now.getTime() + step)) : null;
|
|
if (aliveWait) return aliveWait;
|
|
const fenced = await fenceSourceWorkspace(tx, run);
|
|
// Never claim a stop that was not proven: record what isolates the successor.
|
|
const isolation: FreshContinuationIsolation = fenced || remoteSource ? "remote_epoch"
|
|
: writer === "ended" && !pastBound() && !refused ? "writer_ended" : "fresh_isolated_workspace";
|
|
if (isolation === "fresh_isolated_workspace" && delivery === "saved_input") delivery = "recorded";
|
|
const unverified = await unverifiedActions(tx, run);
|
|
const created = await createdObjects(tx, run);
|
|
const note = freshContinuationNote({ runId: run.id, unverified, created,
|
|
workspace: isolation === "writer_ended" ? "kept" : "restored_last_save" });
|
|
const decision: ExecutionReconciliation & { providerStopProof: "writer_ended" | "isolated" } = { runId: run.id, providerStopped: true,
|
|
// Only a proven end is a stop. `isolated` means exactly: the old run's agent
|
|
// token is refused for writes, and the continuation runs in a new workspace
|
|
// root (a fenced remote epoch or a new local workspace). Nothing more.
|
|
providerStopProof: isolation === "writer_ended" ? "writer_ended" : "isolated",
|
|
actionOutcome: "mixed", outcomeEvidence: note,
|
|
workspaceRepairEvidence: fenced
|
|
? `Paperclip abandoned unverified workspace epoch ${fenced.abandonedEpoch} of run ${run.id}; the continuation restores `
|
|
+ `${fenced.snapshotId ? `committed snapshot ${fenced.snapshotId}` : "the workspace source"} into a new root.`
|
|
: isolation === "fresh_isolated_workspace"
|
|
? `The previous writer of run ${run.id} is not proven stopped, so the continuation starts in a new isolated workspace; no unverified files are applied.`
|
|
: `The continuation of run ${run.id} starts from the task's last saved workspace; no unverified files are applied.` };
|
|
const freshContinuation = { schema: FRESH_CONTINUATION_SCHEMA, sourceRunId: run.id, trigger: input.trigger,
|
|
failureCode: run.errorCode ?? null, delivery, recordedAt: now.toISOString(), isolation,
|
|
stopProof: isolation === "writer_ended" ? "writer_ended" : fenced ? "unproven_epoch_abandoned" : "unproven_isolated",
|
|
unverifiedActions: unverified, createdObjects: created, ...(fenced ? { workspace: fenced } : {}) };
|
|
const reconciliation = { ...decision, actorId: FRESH_CONTINUATION_ACTOR, recordedAt: now.toISOString() };
|
|
// Who, what, why and when. A fresh run is not a replay: the interrupted turn
|
|
// is not re-sent, no tool call or write is re-executed, and any governed
|
|
// action the new run wants goes through its approval gate again.
|
|
const settlement = { schema: FRESH_CONTINUATION_SCHEMA, actorType: "system", actorId: FRESH_CONTINUATION_ACTOR,
|
|
decision: "continue_in_fresh_run", replay: "none", sourceRunId: run.id, failureCode: run.errorCode ?? null,
|
|
reason: "exact_resumption_unavailable", isolation, unverifiedActions: unverified, recordedAt: now.toISOString(),
|
|
...(owned.some(isRestoreHold) ? { heldFiles: "kept_not_applied" } : {}),
|
|
...(refused ? { refusedAtDispatchRunId: refused.id, requestedBy: personAsked ? "person_retry" : "system" } : {}) };
|
|
if (refused && delivery === "recorded") {
|
|
await tx.update(agentWakeupRequests).set({ status: "coalesced", finishedAt: now, updatedAt: now,
|
|
payload: sql`coalesce(${agentWakeupRequests.payload}, '{}'::jsonb) || ${JSON.stringify({ coalescedIntoFreshContinuationOf: run.id })}::jsonb` })
|
|
.where(and(eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
eq(agentWakeupRequests.reason, "task_retry"), eq(agentWakeupRequests.requestedByActorType, "user"), isNull(agentWakeupRequests.runId),
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${issue.id}`));
|
|
}
|
|
const ownerHold =owned.find(hold => issue.status === "blocked" && holdOwnsBlockedStatus(hold, issue, run)) ?? owned.at(-1);
|
|
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionContinuation);
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionReplacement);
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors);
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
|
|
await notifyStatusCardIssueChange(tx, companyId);
|
|
await notifyRoutineExecutionStatus(tx, issue.id, companyId);
|
|
if (delivery === "saved_input") await notifyDeferredWakesForIssue(tx, { companyId, issueId: issue.id });
|
|
|
|
let primaryId: string;
|
|
const continuationDelivery = delivery === "recorded" ? "pending" : "not_requested";
|
|
if (ownerHold) {
|
|
primaryId = ownerHold.id;
|
|
for (const hold of owned) {
|
|
const primary = hold.id === ownerHold.id;
|
|
// The earlier no-replay decision stays on the hold as history, and the
|
|
// fallback is recorded beside it as its own attributed system decision.
|
|
const { automaticRecovery, [FRESH_CONTINUATION_CHECK_KEY]: _check, ...evidence } = hold.evidence;
|
|
await tx.update(issueRecoveryActions).set({ status: "resolved", outcome: "restored", resolvedAt: hold.resolvedAt ?? now,
|
|
updatedAt: now, ownerType: "system", ownerAgentId: null, ownerUserId: null, wakePolicy: null, monitorPolicy: null,
|
|
returnOwnerAgentId: primary ? run.agentId : hold.returnOwnerAgentId, nextAction: USER_TEXT,
|
|
resolutionNote: "continued_fresh_from_last_saved",
|
|
evidence: { ...evidence, executionReconciliation: reconciliation, freshContinuationSettlement: settlement,
|
|
...(automaticRecovery !== undefined ? { automaticRecoveryHistory: automaticRecovery } : {}),
|
|
...(primary ? { freshContinuation, continuationDelivery }
|
|
: { continuationDelivery: "coalesced", coalescedIntoRecoveryActionId: ownerHold.id }) },
|
|
}).where(and(eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.id, hold.id)));
|
|
}
|
|
} else {
|
|
const [created] = await tx.insert(issueRecoveryActions).values({ companyId, sourceIssueId: issue.id,
|
|
kind: "active_run_watchdog", status: "resolved", outcome: "restored", ownerType: "system",
|
|
returnOwnerAgentId: run.agentId, cause: FRESH_CONTINUATION_CAUSE, fingerprint: `fresh-continuation:${run.id}`,
|
|
nextAction: USER_TEXT, resolutionNote: "continued_fresh_from_last_saved", resolvedAt: now, attemptCount: 1,
|
|
evidence: { runId: run.id, originalFailureCode: run.errorCode ?? null, executionReconciliation: reconciliation,
|
|
freshContinuationSettlement: settlement, freshContinuation, continuationDelivery } }).returning({ id: issueRecoveryActions.id });
|
|
primaryId = created!.id;
|
|
}
|
|
// The replaced run loses its write authority: its agent may still be alive
|
|
// in an unreachable sandbox, and only the continuation may act on the task.
|
|
await tx.update(heartbeatRuns).set({ resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb)
|
|
|| ${JSON.stringify({ [FRESH_CONTINUATION_SUPERSEDED_KEY]: now.toISOString() })}::jsonb` }).where(and(
|
|
eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, run.id)));
|
|
// The source is never replayed or replaced exactly after this decision.
|
|
if (coordinator) {
|
|
await tx.update(nativeRunFinalizations).set({ failureDetail: sql`coalesce(${nativeRunFinalizations.failureDetail}, '{}'::jsonb)
|
|
|| ${JSON.stringify({ replacementDenied: "fresh_continuation" })}::jsonb` }).where(and(
|
|
eq(nativeRunFinalizations.companyId, companyId), eq(nativeRunFinalizations.runId, run.id)));
|
|
}
|
|
// The isolation is a property of the task's next run, whatever path starts
|
|
// it (a system continuation, saved input, a person's Retry, a sweep wake):
|
|
// the task no longer points at the directory the old writer may still use,
|
|
// so every next run prepares a new workspace. The old one is left in place.
|
|
if (isolation === "fresh_isolated_workspace" && issue.executionWorkspaceId) {
|
|
await tx.update(issues).set({ executionWorkspaceId: null, workspaceBindingRevision: sql`${issues.workspaceBindingRevision} + 1`,
|
|
updatedAt: now }).where(and(eq(issues.companyId, companyId), eq(issues.id, issue.id)));
|
|
}
|
|
if (issue.status === "blocked") {
|
|
await tx.update(issues).set({ status: "todo", unblockDescriptor: null, blockedTransitionAt: null,
|
|
blockedOwnerNotifiedAt: null, updatedAt: now }).where(and(eq(issues.companyId, companyId), eq(issues.id, issue.id)));
|
|
await logActivity(tx, { companyId, actorType: "system", actorId: FRESH_CONTINUATION_ACTOR, action: "issue.updated",
|
|
entityType: "issue", entityId: issue.id, agentId: run.agentId, runId: run.id,
|
|
details: { status: "todo", previousStatus: "blocked", reason: "fresh_continuation", recoveryActionId: primaryId,
|
|
_previous: { status: "blocked" } } });
|
|
}
|
|
await logActivity(tx, { companyId, actorType: "system", actorId: FRESH_CONTINUATION_ACTOR,
|
|
action: "issue.continued_from_saved_work", entityType: "issue", entityId: issue.id, agentId: run.agentId, runId: run.id,
|
|
details: { recoveryActionId: primaryId, sourceRunId: run.id, failureCode: run.errorCode ?? null, trigger: input.trigger,
|
|
delivery, workspaceEpochAbandoned: Boolean(fenced), unverifiedActionCount: unverified.length } });
|
|
logger.info({ runId: run.id, issueId: issue.id, failureCode: run.errorCode, trigger: input.trigger, delivery },
|
|
"Unfinished run continues in a fresh run from saved work");
|
|
return { kind: "continued", recoveryActionId: primaryId, delivery, isolation };
|
|
}
|
|
|
|
async function settleLateHolds(tx: Db, input: { companyId: string; issue: typeof issues.$inferSelect; run: typeof heartbeatRuns.$inferSelect;
|
|
prior: typeof issueRecoveryActions.$inferSelect; holds: Array<typeof issueRecoveryActions.$inferSelect>; now: Date }) {
|
|
const { companyId, issue, run, prior, holds, now } = input;
|
|
if (!holds.length) return;
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionContinuation);
|
|
await notifyStatusCardIssueChange(tx, companyId);
|
|
await notifyRoutineExecutionStatus(tx, issue.id, companyId);
|
|
const ownsBlock = issue.status === "blocked" && holds.some(hold => holdOwnsBlockedStatus(hold, issue, run));
|
|
const delivery = String(prior.evidence.continuationDelivery);
|
|
const followed = ["delivered", "invalidated"].includes(delivery) ? delivery : "coalesced";
|
|
for (const hold of holds) {
|
|
const { automaticRecovery, [FRESH_CONTINUATION_CHECK_KEY]: _check, ...evidence } = hold.evidence;
|
|
await tx.update(issueRecoveryActions).set({ status: "resolved", outcome: "restored", resolvedAt: now, updatedAt: now,
|
|
ownerType: "system", ownerAgentId: null, ownerUserId: null, wakePolicy: null, monitorPolicy: null, nextAction: USER_TEXT,
|
|
resolutionNote: "continued_fresh_from_last_saved",
|
|
evidence: { ...evidence, ...(automaticRecovery !== undefined ? { automaticRecoveryHistory: automaticRecovery } : {}),
|
|
continuationDelivery: followed, coalescedIntoRecoveryActionId: prior.id,
|
|
...(typeof prior.evidence.continuationRunId === "string" ? { continuationRunId: prior.evidence.continuationRunId } : {}) },
|
|
}).where(and(eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.id, hold.id)));
|
|
}
|
|
if (ownsBlock) {
|
|
await tx.update(issues).set({ status: "todo", unblockDescriptor: null, blockedTransitionAt: null,
|
|
blockedOwnerNotifiedAt: null, updatedAt: now }).where(and(eq(issues.companyId, companyId), eq(issues.id, issue.id)));
|
|
await logActivity(tx, { companyId, actorType: "system", actorId: FRESH_CONTINUATION_ACTOR, action: "issue.updated",
|
|
entityType: "issue", entityId: issue.id, agentId: run.agentId, runId: run.id,
|
|
details: { status: "todo", previousStatus: "blocked", reason: "fresh_continuation", recoveryActionId: prior.id,
|
|
_previous: { status: "blocked" } } });
|
|
}
|
|
}
|
|
|
|
/** Records why a hold was not continued yet, and when to look again. */
|
|
export async function recordFreshContinuationCheck(db: Db, input: { companyId: string; actionId: string },
|
|
result: FreshContinuationResult, now = new Date()) {
|
|
if (result.kind === "continued" || result.kind === "already_continued") return;
|
|
const nextCheckAt = result.kind === "backoff" || result.kind === "exact_recovery_pending" ? result.retryAt.getTime()
|
|
: result.kind === "person_owned" || result.kind === "dependency_wait" ? now.getTime() + PERSON_RECHECK_MS : null;
|
|
const [row] = await db.select({ evidence: issueRecoveryActions.evidence }).from(issueRecoveryActions).where(and(
|
|
eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.id, input.actionId)));
|
|
const prior = object(row?.evidence[FRESH_CONTINUATION_CHECK_KEY]);
|
|
const same = prior.kind === result.kind;
|
|
const checks = same && typeof prior.checks === "number" ? prior.checks + 1 : 1;
|
|
const firstPendingAt = result.kind === "exact_recovery_pending"
|
|
? (same && typeof prior.firstPendingAt === "number" ? prior.firstPendingAt : now.getTime()) : null;
|
|
const marker = { kind: result.kind, reason: "reason" in result ? result.reason : null, checks, checkedAt: now.getTime(), nextCheckAt,
|
|
...(firstPendingAt !== null ? { firstPendingAt,
|
|
progressAt: result.kind === "exact_recovery_pending" ? result.progressAt ?? null : null,
|
|
savedWorkSignature: result.kind === "exact_recovery_pending" ? result.savedWorkSignature ?? null : null } : {}) };
|
|
await db.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence}
|
|
|| jsonb_build_object(${FRESH_CONTINUATION_CHECK_KEY}::text, ${JSON.stringify(marker)}::jsonb)` }).where(and(
|
|
eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.id, input.actionId)));
|
|
}
|
|
|
|
/** Not yet checked, or due for another check. */
|
|
export function freshContinuationCheckDue(now: Date) {
|
|
return sql`(${issueRecoveryActions.evidence}->${FRESH_CONTINUATION_CHECK_KEY} is null
|
|
or coalesce((${issueRecoveryActions.evidence}#>>'{freshContinuationCheck,nextCheckAt}')::bigint, 9223372036854775807) <= ${now.getTime()})`;
|
|
}
|
|
|
|
/**
|
|
* Holds that today keep a task waiting with no automatic continuation, including
|
|
* every shape older releases saved: settled-but-blocked no-replay holds, and
|
|
* saved-work recoveries that finished or gave up. Exact recovery still under
|
|
* way is excluded (it is checked again by its owner and here afterwards).
|
|
*/
|
|
export function heldExecutionsAwaitingFreshContinuation(db: Db, now: Date, limit: number) {
|
|
return db.select({ id: issueRecoveryActions.id, companyId: issueRecoveryActions.companyId,
|
|
runId: sql<string>`coalesce(${issueRecoveryActions.evidence}->>'runId', ${issueRecoveryActions.evidence}->>'sourceRunId')` })
|
|
.from(issueRecoveryActions).innerJoin(issues, and(eq(issues.id, issueRecoveryActions.sourceIssueId),
|
|
eq(issues.companyId, issueRecoveryActions.companyId)))
|
|
.where(and(heldExecutionWhere(), freshContinuationCheckDue(now), recentInterruptionSourceSql(now)))
|
|
// Newest interruption first: the most recent work is the work someone is waiting for.
|
|
.orderBy(desc(sql`(select source.finished_at from ${heartbeatRuns} source where source.company_id = ${issueRecoveryActions.companyId}
|
|
and source.id::text = coalesce(${issueRecoveryActions.evidence}->>'runId', ${issueRecoveryActions.evidence}->>'sourceRunId'))`),
|
|
asc(issueRecoveryActions.id))
|
|
.limit(limit);
|
|
}
|
|
|
|
/**
|
|
* The backlog pass continues a held record only when it is a recent
|
|
* interruption: its source run ended within BACKLOG_CONTINUATION_MAX_AGE_MS
|
|
* and ended with a restart, deploy, lease, process-loss or recovery-refusal
|
|
* code. A configuration or provider failure, or anything older, stays exactly
|
|
* as it is for a person's Retry: re-running last week's failures is not
|
|
* "an interruption is invisible". Fresh interruptions reach the fallback
|
|
* through restart recovery and the disposition pass, not through this list.
|
|
*/
|
|
export const BACKLOG_CONTINUATION_MAX_AGE_MS = 24 * 60 * 60_000;
|
|
export const BACKLOG_CONTINUATION_SOURCE_CODES = RESTART_CLASS_FAILURE_CODES;
|
|
/** At most this many backlog continuations per server per rolling minute, and per pass. */
|
|
export const BACKLOG_CONTINUATIONS_PER_MINUTE = 10;
|
|
const BACKLOG_CONTINUATIONS_PER_PASS = 5;
|
|
|
|
/** Holds the disposition pass (unresolvedExecutionDispositions) handles itself, whatever their age. */
|
|
function liveDispositionHoldSql() {
|
|
return and(inArray(issueRecoveryActions.status, ["active", "escalated"]), eq(issueRecoveryActions.kind, "active_run_watchdog"))!;
|
|
}
|
|
|
|
const cancelledAtDispatch = (alias: "newer" | "held") => sql.raw(`coalesce(${alias}.status = 'cancelled' and ${alias}.started_at is null
|
|
and ${alias}.error_code = 'execution_reconciliation_required', false)`);
|
|
const runOfTask = (alias: "newer" | "held", issueId: SQL) => sql`(${sql.raw(alias)}.native_issue_id::text = ${issueId}::text
|
|
or ${sql.raw(alias)}.issue_id::text = ${issueId}::text or ${sql.raw(alias)}.context_snapshot->>'issueId' = ${issueId}::text)`;
|
|
|
|
function interruptionEligibility(input: { now: Date; companyId: SQL; issueId: SQL; sourceRunId: SQL }) {
|
|
const since = sql`${new Date(input.now.getTime() - BACKLOG_CONTINUATION_MAX_AGE_MS).toISOString()}::timestamptz`;
|
|
const source = (condition: SQL) => sql`exists (select 1 from ${heartbeatRuns} source where source.company_id = ${input.companyId}
|
|
and source.id::text = ${input.sourceRunId}::text and ${condition})`;
|
|
const refusedLater = sql`held.company_id = source.company_id and held.id <> source.id and ${runOfTask("held", input.issueId)}
|
|
and (held.created_at, held.id) > (source.created_at, source.id) and ${cancelledAtDispatch("held")}`;
|
|
const personAsked = source(sql`(exists (select 1 from ${agentWakeupRequests} retry where retry.company_id = source.company_id
|
|
and retry.status = 'deferred_issue_execution' and retry.reason = 'task_retry' and retry.requested_by_actor_type = 'user'
|
|
and retry.run_id is null and retry.payload->>'issueId' = ${input.issueId}::text and retry.requested_at > source.finished_at)
|
|
or exists (select 1 from ${heartbeatRuns} held where ${refusedLater}
|
|
and held.context_snapshot->>'wakeReason' = 'task_retry' and held.finished_at > ${since}))`);
|
|
const systemRetriedSource = sql`(${continuationCauseIsSystemOwnedSql("source")}
|
|
and coalesce(source.context_snapshot->>'wakeReason', '') <> 'task_retry' and not coalesce(source.context_snapshot ? 'explicitUserContinuation', false)
|
|
and exists (select 1 from ${heartbeatRuns} held where ${refusedLater} and ${recoveryRunSql("held")}))`;
|
|
const recent = source(sql`(${continuesAutomaticallySql("source")} or ${systemRetriedSource})
|
|
and greatest(source.finished_at, (select max(held.finished_at) from ${heartbeatRuns} held where ${refusedLater})) > ${since}`);
|
|
return { recent, personAsked };
|
|
}
|
|
|
|
/** The backlog pass's eligibility: the one predicate its listing and its deadline query share. */
|
|
function recentInterruptionSourceSql(now: Date) {
|
|
const { recent, personAsked } = interruptionEligibility({ now, companyId: sql`${issueRecoveryActions.companyId}`,
|
|
issueId: sql`${issueRecoveryActions.sourceIssueId}`,
|
|
sourceRunId: sql`coalesce(${issueRecoveryActions.evidence}->>'runId', ${issueRecoveryActions.evidence}->>'sourceRunId')` });
|
|
return sql`(${recent} or ${personAsked})`;
|
|
}
|
|
|
|
/** Held records the backlog pass leaves for a person (older, or not an interruption). */
|
|
async function countHeldExecutionsLeftAlone(db: Db, now: Date): Promise<number> {
|
|
const [row] = await db.select({ count: sql<number>`count(*)::int` }).from(issueRecoveryActions)
|
|
.innerJoin(issues, and(eq(issues.id, issueRecoveryActions.sourceIssueId), eq(issues.companyId, issueRecoveryActions.companyId)))
|
|
.where(and(heldExecutionWhere(), sql`not ${recentInterruptionSourceSql(now)}`));
|
|
return Number(row?.count ?? 0);
|
|
}
|
|
|
|
/** Per database client: one per server process in production, one per test database in tests. */
|
|
const backlogContinuationTimes = new WeakMap<object, number[]>();
|
|
export function resetBacklogContinuationPacingForTests(db?: Db) {
|
|
if (db) backlogContinuationTimes.delete(db);
|
|
lastLeftAloneLogAt = 0;
|
|
}
|
|
let lastLeftAloneLogAt = 0;
|
|
const LEFT_ALONE_LOG_INTERVAL_MS = 10 * 60_000;
|
|
|
|
/** Held reconciliation records of a task's latest unfinished run (joined to its task). */
|
|
function heldExecutionWhere(options: { includeActive?: boolean } = {}) {
|
|
return and(
|
|
inArray(issueRecoveryActions.cause, [...EXECUTION_RECONCILIATION_CAUSES]),
|
|
sql`${issueRecoveryActions.evidence}->'executionReconciliation' is null`,
|
|
sql`${issueRecoveryActions.evidence}->'freshContinuation' is null`,
|
|
sql`coalesce(${issueRecoveryActions.evidence}->>'runId', ${issueRecoveryActions.evidence}->>'sourceRunId') is not null`,
|
|
or(
|
|
sql`${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked'`,
|
|
and(inArray(issueRecoveryActions.status, ["active", "escalated"]),
|
|
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('complete', 'exhausted')`),
|
|
options.includeActive ? and(inArray(issueRecoveryActions.status, ["active", "escalated"]),
|
|
eq(issueRecoveryActions.kind, "active_run_watchdog")) : undefined,
|
|
// Saved-work recovery still under way: checked so its bound is reached
|
|
// (exact recovery keeps its first chance inside the fallback).
|
|
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('scheduled', 'running', 'awaiting_evidence')`,
|
|
),
|
|
isNull(issueRecoveryActions.ownerUserId),
|
|
sql`${issues.status} not in ('done', 'cancelled')`,
|
|
sql`${issues.assigneeAgentId} is not null`,
|
|
// Only the task's latest run, unfinished and still its assignee's: other
|
|
sql`exists (select 1 from ${heartbeatRuns} source where source.company_id = ${issueRecoveryActions.companyId}
|
|
and source.id::text = coalesce(${issueRecoveryActions.evidence}->>'runId', ${issueRecoveryActions.evidence}->>'sourceRunId')
|
|
and source.status in ('failed', 'timed_out', 'interrupted', 'cancelled') and source.finished_at is not null
|
|
and source.agent_id = ${issues.assigneeAgentId}
|
|
and not exists (select 1 from ${heartbeatRuns} newer where newer.company_id = source.company_id and newer.id <> source.id
|
|
and (newer.native_issue_id = ${issues.id} or newer.issue_id = ${issues.id} or newer.context_snapshot->>'issueId' = ${issues.id}::text)
|
|
and not ${cancelledAtDispatch("newer")}
|
|
and (newer.status in ('queued', 'running', 'scheduled_retry') or (newer.created_at, newer.id) > (source.created_at, source.id))))`,
|
|
)!;
|
|
}
|
|
|
|
/** The earliest time a deferred check becomes due again, if any. */
|
|
export async function nextFreshContinuationCheckAt(db: Db, now = new Date()): Promise<number | null> {
|
|
// The same records the held-run query and the disposition pass would select;
|
|
// a deadline on a record nothing would act on never schedules a scan.
|
|
// That includes active holds that backed off: the disposition pass rechecks
|
|
// them when due, and nothing else would wake it then.
|
|
const [held] = await db.select({ at: sql<string | null>`min((${issueRecoveryActions.evidence}#>>'{freshContinuationCheck,nextCheckAt}')::bigint)` })
|
|
.from(issueRecoveryActions).innerJoin(issues, and(eq(issues.id, issueRecoveryActions.sourceIssueId),
|
|
eq(issues.companyId, issueRecoveryActions.companyId)))
|
|
.where(and(heldExecutionWhere({ includeActive: true }),
|
|
sql`${issueRecoveryActions.evidence}#>>'{freshContinuationCheck,nextCheckAt}' is not null`,
|
|
// A record the backlog pass leaves alone schedules nothing; active holds
|
|
// stay with the disposition pass, which rechecks them when due.
|
|
or(liveDispositionHoldSql(), recentInterruptionSourceSql(now))));
|
|
return held?.at ? Number(held.at) : null;
|
|
}
|
|
|
|
/**
|
|
* Continues tasks held by recovery records, including records saved by older
|
|
* releases, so tasks already waiting continue after deploy without a Retry.
|
|
* Runs on the execution-disposition worker. Bounded per pass.
|
|
*/
|
|
export async function continueHeldExecutions(db: Db, now = new Date(), limit = BACKLOG_CONTINUATIONS_PER_PASS) {
|
|
if (now.getTime() - lastLeftAloneLogAt >= LEFT_ALONE_LOG_INTERVAL_MS) {
|
|
lastLeftAloneLogAt = now.getTime();
|
|
const leftAlone = await countHeldExecutionsLeftAlone(db, now).catch(() => null);
|
|
if (leftAlone) logger.info({ leftAlone, maxAgeHours: BACKLOG_CONTINUATION_MAX_AGE_MS / 3_600_000 },
|
|
"Held tasks left for a person: older than the backlog window or not an interruption");
|
|
}
|
|
// Per-server pacing: a backlog saved by older releases must not start a herd of runs.
|
|
let times = backlogContinuationTimes.get(db);
|
|
if (!times) backlogContinuationTimes.set(db, times = []);
|
|
while (times.length && times[0]! <= now.getTime() - 60_000) times.shift();
|
|
const room = Math.min(limit, BACKLOG_CONTINUATIONS_PER_MINUTE - times.length);
|
|
if (room <= 0) return { checked: 0, continued: 0 };
|
|
const candidates = await heldExecutionsAwaitingFreshContinuation(db, now, room);
|
|
let continued = 0;
|
|
const seen = new Set<string>();
|
|
for (const candidate of candidates) {
|
|
if (!candidate.runId || seen.has(candidate.runId)) continue;
|
|
seen.add(candidate.runId);
|
|
times.push(now.getTime());
|
|
try {
|
|
const result = await ensureFreshContinuation(db, { companyId: candidate.companyId, runId: candidate.runId,
|
|
trigger: "held_recovery", now });
|
|
if (result.kind === "continued") continued += 1;
|
|
else await recordFreshContinuationCheck(db, { companyId: candidate.companyId, actionId: candidate.id }, result, now);
|
|
} catch (error) {
|
|
logger.warn({ err: error, recoveryActionId: candidate.id }, "Fresh continuation of a held task deferred");
|
|
await recordFreshContinuationCheck(db, { companyId: candidate.companyId, actionId: candidate.id },
|
|
{ kind: "exact_recovery_pending", reason: "fallback_failed", retryAt: new Date(now.getTime() + EXACT_RECOVERY_RECHECK_MS) }, now)
|
|
.catch(() => undefined);
|
|
}
|
|
}
|
|
return { checked: seen.size, continued };
|
|
}
|
|
|
|
/** Holds settled together with a primary record follow its final delivery
|
|
* state, so no coalesced hold outlives the record it points at. */
|
|
export async function settleCoalescedFreshContinuationHolds(db: Db, input: { companyId: string; primaryId: string;
|
|
delivery: "delivered" | "invalidated"; continuationRunId?: string | null }) {
|
|
await db.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify({
|
|
continuationDelivery: input.delivery, ...(input.continuationRunId ? { continuationRunId: input.continuationRunId } : {}) })}::jsonb` })
|
|
.where(and(eq(issueRecoveryActions.companyId, input.companyId),
|
|
sql`${issueRecoveryActions.evidence}->>'coalescedIntoRecoveryActionId' = ${input.primaryId}`,
|
|
sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'coalesced'`));
|
|
}
|
|
|
|
export async function hasPendingFreshContinuation(db: Db, companyId: string, issueId: string) {
|
|
const [row] = await db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and(
|
|
eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, issueId),
|
|
eq(issueRecoveryActions.status, "resolved"),
|
|
sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`,
|
|
sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,schema}' = ${FRESH_CONTINUATION_SCHEMA}`)).limit(1);
|
|
if (!row) return false;
|
|
await notifyDeliveryWork(db, DELIVERY_QUEUES.executionContinuation);
|
|
return true;
|
|
}
|
|
|
|
/** Newest-first list of fresh continuations of one task (diagnostics and tests). */
|
|
export async function listFreshContinuations(db: Db, input: { companyId: string; issueId: string }) {
|
|
return db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, input.companyId),
|
|
eq(issueRecoveryActions.sourceIssueId, input.issueId),
|
|
sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,schema}' = ${FRESH_CONTINUATION_SCHEMA}`,
|
|
gt(sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,recordedAt}'`, "")))
|
|
.orderBy(desc(issueRecoveryActions.updatedAt));
|
|
}
|