From 3f030971dc0ca86e398370582b82eacf84265a59 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 07:13:49 -0600 Subject: [PATCH] fix(recovery): deliver a recorded continuation as the person's Retry run, keeping its isolation (#21378) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When Paperclip cannot resume a cut-off run exactly, the fresh-continuation fallback records a continuation for the task. A delivery pass later starts it as a run. Staging shows this pending state can last 30 to 110 minutes for backlog items. > - The fallback also decides how the next run is isolated from the old writer. For `fresh_isolated_workspace`, the next run must prepare a new workspace. > - A person's Retry pressed while the continuation was recorded but not yet delivered started its own run at once. That run did not carry the recorded isolation, so it could use the shared directory next to a writer that is not proven stopped. It also did not carry the continuation's recovery context. > - This pull request makes wake admission deliver the recorded continuation as the person's run, under the task lock. It also folds a saved Retry into a delivered continuation for every record kind. > - The benefit is one run per interruption, whatever arrives first, and the recorded isolation decision is never lost. ## Linked Issues or Issue Description Refs #21348 **What happened?** A person's Retry (`requestTaskRetry` + `resumeTaskRetry`), pressed while a fresh continuation was recorded but not delivered, was admitted at once as an ordinary `task_retry` run. That run had no `freshContinuationIsolation` and no `recoveryActionId`. On a projectless task the run used the shared directory that the fallback had decided to isolate from. The later delivery pass marked the continuation as superseded by that run. The reproduction showed exactly one run, so the reported "two runs" did not happen in that order. The isolation was lost for all three record kinds tested. For `writer_ended` and native records, a Retry saved before delivery was not folded into the delivered continuation, because the fold only covered `fresh_isolated_workspace`. **Expected behavior** With a recorded, undelivered continuation, the next run admitted for the task is that continuation. It has the recorded isolation and the person's Retry is recorded as answered by it. **Steps to reproduce** 1. Record a fresh continuation (`ensureFreshContinuation`, delivery `recorded`) for a projectless task whose legacy writer is on another host. 2. Call `requestTaskRetry` and then `resumeTaskRetry`. 3. Before this change: one `task_retry` run without `freshContinuationIsolation` or `recoveryActionId`, while the record is still `pending`. ## What Changed - `fresh-continuation-retry-fold.ts`: - `undeliveredContinuation` reads the task's pending fresh-continuation record under the task lock (`for update`). It returns nothing when a run of the task was created after the record. - `undeliveredContinuationContext` gives the run the continuation's `recoveryActionId`, `retryOfRunId`, `forceFreshSession` and, for `fresh_isolated_workspace`, `freshContinuationIsolation`. - `deliverContinuationAsRun` marks the record `delivered` to the new run in the same transaction, settles its coalesced holds, and folds other waiting Retries. - `foldWaitingRetriesIntoRecordedContinuation` folds a waiting Retry into any delivered fresh continuation, not only an isolated one. - `heartbeat/queue.ts` (wake admission, issue transaction): a non-system, non-delivery wake with no execution blocker takes the pending continuation. The run is inserted as before, then delivered as the continuation. The fold after a delivery wake now applies to every fresh-continuation kind. No change to ranking or to the claim loop. - `execution-recovery-resolution.ts`: the delivery pass folds a saved Retry for every fresh-continuation kind. The superseded-run fold keys on `recoveryActionId`. - The person's run keeps its `task_retry` wake reason and the user's wake receipt, so the existing #21349 rules rank it as person work. - Limits: the person-triggered delivery does not spend an automatic attempt. It adds no `issue.continued_automatically` entry and no scheduled transient retry, and it is not gated by the hourly cap or by delivery-pass pacing. The record itself was already counted when it was recorded. The agent's concurrency limit still applies at claim time. ## Verification New file `server/src/services/fresh-continuation-retry-delivery.test.ts`, with 12 tests. All cases cover `fresh_isolated_workspace`, `writer_ended` and native (`remote_epoch`) records, through the real `requestTaskRetry` + `resumeTaskRetry` path: - A Retry during pending delivery gives exactly one run. That run is the continuation, carries the recorded isolation (the workspace readiness reads `fresh_isolated_workspace`), coalesces the Retry, and ranks as person work. Later delivery passes add nothing. - Delivery first, then a saved Retry: one run, and the Retry is coalesced. A Retry on the delivered run joins it through the busy path. - A Retry and two delivery passes race (concurrent transactions): one run. - After the continuation's own failure, a Retry gets its own run. - The delivery pass is held by pacing (backlog source, agent queue full), and the hourly cap and the shared attempt ledger are exhausted. The person's Retry still starts the run. Red before the fix: 8 of 12 failed. Green after: 12 of 12. Local runs, one file at a time: - `fresh-continuation-fallback.test.ts`: 71 of 72. The failure is "runs a person's Retry pressed before delivery…", a deadlock in `afterEach` truncate. It passes alone, and it fails the same way on unmodified master under this machine's load. - `__tests__/recovery-wake-herd.test.ts`: 29 passed - `__tests__/execution-control-work.test.ts`: 27 passed - `__tests__/issue-recovery-actions.test.ts`: 112 passed - `workspace-recovery-fallback.test.ts`: 19 passed - `heartbeat/queue.test.ts`: 101 passed - `task-retry.test.ts`: 117 passed; `legacy-preparation-retry-hold.test.ts`: 5 passed - `server` `tsc --noEmit`: clean ## Risks - A person's message, chat input or manual wake that arrives while a continuation is pending also becomes that continuation, with a fresh session and the recorded isolation. Before, such a wake ran without isolation and superseded the record. This is the same rule for every admission path, and it closes the shared-directory gap there too. - A wake that takes the context and is then deferred by a later gate can keep the isolation marker in its saved context. If the record is delivered meanwhile, that later run prepares a new isolated workspace. That is the safe direction. - No schema change and no migration. Nothing is added before turn start for healthy runs: the new read runs only at wake admission. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), Claude Code agent with tool use (shell, file edit, test runs). Model ID `claude-opus-5-5`. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (no documented behavior or command changes) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 --- .../services/execution-recovery-resolution.ts | 4 +- .../fresh-continuation-retry-delivery.test.ts | 203 ++++++++++++++++++ .../services/fresh-continuation-retry-fold.ts | 62 +++++- server/src/services/heartbeat/queue.ts | 16 +- 4 files changed, 277 insertions(+), 8 deletions(-) create mode 100644 server/src/services/fresh-continuation-retry-delivery.test.ts diff --git a/server/src/services/execution-recovery-resolution.ts b/server/src/services/execution-recovery-resolution.ts index d7cf5b54fe0..2b1a7098c17 100644 --- a/server/src/services/execution-recovery-resolution.ts +++ b/server/src/services/execution-recovery-resolution.ts @@ -534,7 +534,7 @@ export async function deliverReconciledExecutions( .where(pendingDecision); await settleCoalescedFreshContinuationHolds(db, { companyId: action.companyId, primaryId: action.id, delivery: "delivered", continuationRunId: newer.id }); - if (newer.contextSnapshot?.freshContinuationIsolation === action.id) + if (newer.contextSnapshot?.recoveryActionId === action.id) await db.transaction(async tx => { await tx.execute(sql`select id from issues where company_id = ${action.companyId} and id = ${task.id} for update`); await foldWaitingRetriesIntoContinuation(tx as unknown as Db, { companyId: action.companyId, issueId: task.id, runId: newer.id, @@ -600,7 +600,7 @@ export async function deliverReconciledExecutions( .where(pendingDecision); await settleCoalescedFreshContinuationHolds(tx as unknown as Db, { companyId: action.companyId, primaryId: action.id, delivery: "delivered", continuationRunId: run.id }); - if ((action.evidence.freshContinuation as { isolation?: unknown } | undefined)?.isolation === "fresh_isolated_workspace") + if (fresh) await foldWaitingRetriesIntoContinuation(tx as unknown as Db, { companyId: action.companyId, issueId: task.id, runId: run.id, sourceRunId: decision.runId }); }); diff --git a/server/src/services/fresh-continuation-retry-delivery.test.ts b/server/src/services/fresh-continuation-retry-delivery.test.ts new file mode 100644 index 00000000000..af078c0c878 --- /dev/null +++ b/server/src/services/fresh-continuation-retry-delivery.test.ts @@ -0,0 +1,203 @@ +import { randomUUID } from "node:crypto"; +import { and, eq, sql } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest"; +import { + activityLog, agentWakeupRequests, agents, authUsers, companies, createDb, heartbeatRuns, issueRecoveryActions, issues, + nativeRunFinalizations, +} from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js"; +import { deliverReconciledExecutions } from "./execution-recovery-resolution.js"; +import { FRESH_CONTINUATIONS_PER_HOUR, FRESH_CONTINUATION_SCHEMA, ensureFreshContinuation, + resetBacklogContinuationPacingForTests } from "./fresh-continuation-fallback.js"; +import { readFreshWorkspaceReadiness } from "./fresh-workspace-continuation.js"; +import { heartbeatService } from "./heartbeat.js"; +import { queuedRunDispatchClass } from "./heartbeat/queued-run-dispatch-order.js"; +import { ASSIGNED_WORK_CONTINUED_ACTION } from "./recovery/task-automatic-attempts.js"; +import { CONTINUATION_CAP } from "./recovery/assigned-work-invariant.js"; +import { requestTaskRetry } from "./task-retry.js"; + +const support = await getEmbeddedPostgresTestSupport(); +const suite = support.supported ? describe : describe.skip; + +const KINDS = ["fresh_isolated_workspace", "writer_ended", "native"] as const; +type Kind = (typeof KINDS)[number]; +const RECORDED_ISOLATION: Record = { + fresh_isolated_workspace: "fresh_isolated_workspace", writer_ended: "writer_ended", native: "remote_epoch" }; + +suite("a person's Retry while a continuation waits for delivery", () => { + let db!: ReturnType; + let temporary: Awaited> | null = null; + let companyId = "", agentId = "", userId = ""; + + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("paperclip-retry-delivery-"); + db = createDb(temporary.connectionString); + companyId = randomUUID(); agentId = randomUUID(); userId = randomUUID(); + await db.insert(authUsers).values({ id: userId, name: "Board", email: `${userId}@example.test`, + emailVerified: true, createdAt: new Date(), updatedAt: new Date() }); + await db.insert(companies).values({ id: companyId, name: "Paperclip", issuePrefix: "FRD", + requireBoardApprovalForNewAgents: false, defaultResponsibleUserId: userId }); + await db.insert(agents).values({ id: agentId, companyId, name: "Coder", role: "engineer", status: "idle", + adapterType: "paperclip_runner", adapterConfig: {}, runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } }, permissions: {} }); + }, 60_000); + + afterEach(async () => { + resetBacklogContinuationPacingForTests(db); + await db.execute(sql`truncate table issue_recovery_actions, native_run_finalizations, activity_log, + agent_wakeup_requests, heartbeat_runs, issues cascade`); + }); + + afterAll(async () => { await db?.$client.end(); await temporary?.cleanup(); }); + + async function pendingContinuation(kind: Kind, endedAgoMs = 60_000) { + const issueId = randomUUID(), runId = randomUUID(); + const finishedAt = new Date(Date.now() - endedAgoMs); + const native = kind === "native"; + await db.insert(issues).values({ id: issueId, companyId, title: "Ship the release notes", status: "blocked", + assigneeAgentId: agentId, statusVersion: 4, blockedTransitionAt: finishedAt }); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "assignment", issueId, + nativeIssueId: native ? issueId : null, runtimeMode: native ? "native" : "legacy", + status: "failed", errorCode: native ? "native_session_interrupted" : kind === "writer_ended" ? "process_lost" : "controller_lease_lost", + error: "cut off", startedAt: new Date(finishedAt.getTime() - 10 * 60_000), finishedAt, + createdAt: new Date(finishedAt.getTime() - 11 * 60_000), responsibleUserId: userId, + contextSnapshot: { issueId, wakeReason: "issue_assigned" }, + ...(native ? { nativePhase: "terminal_failure", runnerProfileJson: { remoteWorkspace: { schema: "paperclip.remote-workspace-run.v1", + workspaceId: randomUUID(), leaseId: randomUUID(), providerLeaseId: "sandbox-1", remoteCwd: "/home/daytona/work", epoch: 3, + state: "prepared", snapshotId: null } } } + : { runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } } }), + ...(kind === "fresh_isolated_workspace" + ? { processPid: 4242, processHostHash: "replaced-server", processStartedAt: new Date(Date.now() - 600_000) } : {}) }); + if (native) { + await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "terminal_failure", + recoveryState: "blocked", failureCode: "native_continuation_requires_reconciliation", leaseOwner: null }); + } + const [action] = await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: issueId, kind: "active_run_watchdog", + status: "active", ownerType: "board", returnOwnerAgentId: agentId, + cause: native ? "native_continuation_requires_reconciliation" : "legacy_execution_requires_reconciliation", + fingerprint: `${native ? "native-continuation" : "legacy-execution"}:${runId}`, nextAction: "Held.", maxAttempts: 3, + createdAt: finishedAt, evidence: { runId, originalFailureCode: "cut_off", nativeFailureBlock: { runId, statusVersion: 4 } } }) + .returning(); + await db.update(issues).set({ projectId: null, projectWorkspaceId: null, executionWorkspaceId: null }).where(eq(issues.id, issueId)); + expect(await ensureFreshContinuation(db, { companyId, runId, trigger: "recovery_refused" })) + .toMatchObject({ kind: "continued", delivery: "recorded", isolation: RECORDED_ISOLATION[kind] }); + return { issueId, runId, actionId: action!.id, isolation: RECORDED_ISOLATION[kind] }; + } + + async function occupySlot() { + const [other] = await db.insert(issues).values({ companyId, title: "Other work", status: "in_progress", assigneeAgentId: agentId }).returning(); + await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: other!.id, status: "running", contextSnapshot: { issueId: other!.id } }); + } + const taskRuns = (f: { issueId: string; runId: string }) => db.select().from(heartbeatRuns).where(and( + sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${f.issueId}`, sql`${heartbeatRuns.id} <> ${f.runId}`)); + const record = async (actionId: string) => (await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, actionId)))[0]!; + const wake = async (id: string) => (await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, id)))[0]!; + const deliveryPass = () => deliverReconciledExecutions(db, heartbeatService(db).wakeup); + async function deliver() { await deliveryPass(); await deliveryPass(); } + async function requestRetry(issueId: string, observedRunId: string) { + const receipt = await requestTaskRetry({ db, companyId, issueId, actorId: userId, observedRunId, + idempotencyKey: randomUUID(), actorSource: "local_implicit" }); + expect(receipt).toMatchObject({ accepted: true }); + return receipt.requestId!; + } + async function pressRetry(f: { issueId: string; runId: string }) { + const requestId = await requestRetry(f.issueId, f.runId); + await heartbeatService(db).resumeTaskRetry(companyId, requestId); + return requestId; + } + + async function expectSingleContinuationRun(f: Awaited>, requestId: string) { + const runs = await taskRuns(f); + expect(runs).toHaveLength(1); + const run = runs[0]!; + expect(run.contextSnapshot).toMatchObject({ recoveryActionId: f.actionId, previousRunId: f.runId, forceFreshSession: true }); + expect(run.retryOfRunId).toBe(f.runId); + if (f.isolation === "fresh_isolated_workspace") { + expect(run.contextSnapshot).toMatchObject({ freshContinuationIsolation: f.actionId }); + expect(await readFreshWorkspaceReadiness(db, run)).toMatchObject({ mode: "fresh_isolated_workspace", retryIntentId: f.actionId }); + } else expect(run.contextSnapshot?.freshContinuationIsolation).toBeUndefined(); + expect(await wake(requestId)).toMatchObject({ status: "coalesced", runId: run.id }); + expect((await record(f.actionId)).evidence).toMatchObject({ continuationDelivery: "delivered", continuationRunId: run.id }); + return run; + } + + it.each(KINDS)("runs a Retry pressed before delivery as the recorded continuation itself, ranked as person work (%s)", async kind => { + const f = await pendingContinuation(kind); + await occupySlot(); + const requestId = await pressRetry(f); + const run = await expectSingleContinuationRun(f, requestId); + expect(run.contextSnapshot).toMatchObject({ wakeReason: "task_retry", taskRetryRequestId: requestId }); + const [runWake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, run.wakeupRequestId!)); + expect(runWake!.requestedByActorType).toBe("user"); + expect(queuedRunDispatchClass({ id: run.id, createdAt: run.createdAt, wakeReason: "task_retry", requestedByUser: true, + sourceEndedAt: null, readinessRank: 1, priorityRank: 0 }, new Date())).toBe("person"); + await deliver(); + expect((await taskRuns(f)).map(row => row.id)).toEqual([run.id]); + }, 60_000); + + it.each(KINDS)("answers a Retry saved before delivery with the delivered continuation, never a second run (%s)", async kind => { + const f = await pendingContinuation(kind); + await occupySlot(); + const requestId = await requestRetry(f.issueId, f.runId); + await deliver(); + await heartbeatService(db).resumeTaskRetry(companyId, requestId); + const runs = await taskRuns(f); + expect(runs).toHaveLength(1); + expect(runs[0]!.contextSnapshot).toMatchObject({ recoveryActionId: f.actionId, wakeReason: "issue_recovery_action_restored" }); + expect(await wake(requestId)).toMatchObject({ status: "coalesced", runId: runs[0]!.id }); + const joined = await requestRetry(f.issueId, runs[0]!.id); + expect(await wake(joined)).toMatchObject({ status: "coalesced", runId: runs[0]!.id }); + expect(await taskRuns(f)).toHaveLength(1); + }, 60_000); + + it.each(KINDS)("starts exactly one run when a Retry and the delivery pass race (%s)", async kind => { + const f = await pendingContinuation(kind); + await occupySlot(); + const requestId = await requestRetry(f.issueId, f.runId); + await Promise.all([heartbeatService(db).resumeTaskRetry(companyId, requestId), deliveryPass(), deliveryPass()]); + await deliver(); + await heartbeatService(db).resumeTaskRetry(companyId, requestId); + const runs = await taskRuns(f); + expect(runs).toHaveLength(1); + expect(runs[0]!.contextSnapshot).toMatchObject({ recoveryActionId: f.actionId }); + if (f.isolation === "fresh_isolated_workspace") expect(runs[0]!.contextSnapshot).toMatchObject({ freshContinuationIsolation: f.actionId }); + expect(await wake(requestId)).toMatchObject({ status: "coalesced", runId: runs[0]!.id }); + expect((await record(f.actionId)).evidence).toMatchObject({ continuationDelivery: "delivered", continuationRunId: runs[0]!.id }); + }, 60_000); + + it.each(["fresh_isolated_workspace", "writer_ended"] as const)("gives a Retry of the continuation's own failure its own run (%s)", async kind => { + const f = await pendingContinuation(kind); + await occupySlot(); + await deliver(); + const [continuation] = await taskRuns(f); + await db.update(heartbeatRuns).set({ status: "failed", errorCode: "adapter_failed", startedAt: new Date(), finishedAt: new Date() }) + .where(eq(heartbeatRuns.id, continuation!.id)); + const requestId = await pressRetry({ issueId: f.issueId, runId: continuation!.id }); + const retryWake = await wake(requestId); + expect(retryWake.runId, JSON.stringify(retryWake)).toBeTruthy(); + expect(retryWake.runId).not.toBe(continuation!.id); + const [own] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, retryWake.runId!)); + expect(own!.contextSnapshot).toMatchObject({ wakeReason: "task_retry", previousRunId: continuation!.id }); + expect(own!.contextSnapshot?.recoveryActionId).toBeUndefined(); + expect((await record(f.actionId)).evidence).toMatchObject({ continuationRunId: continuation!.id }); + }, 60_000); + + it("starts the person's run while the delivery is held by pacing and the task's automatic attempts are spent", async () => { + const f = await pendingContinuation("fresh_isolated_workspace", 45 * 60_000); + const [other] = await db.insert(issues).values({ companyId, title: "Queued work", status: "todo", assigneeAgentId: agentId }).returning(); + await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: other!.id, status: "queued", contextSnapshot: { issueId: other!.id } }); + const now = new Date(); + for (let index = 0; index < FRESH_CONTINUATIONS_PER_HOUR; index += 1) + await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: f.issueId, kind: "active_run_watchdog", status: "resolved", + ownerType: "system", cause: "fresh_continuation", fingerprint: `spent:${index}:${f.issueId}`, nextAction: "Continued.", + evidence: { freshContinuation: { schema: FRESH_CONTINUATION_SCHEMA, recordedAt: new Date(now.getTime() - 60_000).toISOString() }, + continuationDelivery: "delivered" } }); + for (let index = 0; index < CONTINUATION_CAP; index += 1) + await db.insert(activityLog).values({ companyId, actorType: "system", actorId: "assigned-work", action: ASSIGNED_WORK_CONTINUED_ACTION, + entityType: "issue", entityId: f.issueId }); + await deliver(); + expect(await taskRuns(f)).toHaveLength(0); + expect((await record(f.actionId)).evidence.continuationDelivery).toBe("pending"); + const requestId = await pressRetry(f); + await expectSingleContinuationRun(f, requestId); + }, 60_000); +}); diff --git a/server/src/services/fresh-continuation-retry-fold.ts b/server/src/services/fresh-continuation-retry-fold.ts index f09509b581e..3544c9736de 100644 --- a/server/src/services/fresh-continuation-retry-fold.ts +++ b/server/src/services/fresh-continuation-retry-fold.ts @@ -1,6 +1,7 @@ -import { and, eq, inArray, isNull, sql } from "drizzle-orm"; -import { agentWakeupRequests, heartbeatRuns, type Db } from "@paperclipai/db"; +import { and, desc, eq, inArray, isNull, or, sql } from "drizzle-orm"; +import { agentWakeupRequests, heartbeatRuns, issueRecoveryActions, type Db } from "@paperclipai/db"; import { queuedCommentIdsFromWakePayload } from "./issue-queued-comment-queue.js"; +import { FRESH_CONTINUATION_SCHEMA, settleCoalescedFreshContinuationHolds } from "./fresh-continuation-fallback.js"; const LIVE_CONTINUATION_STATUSES = ["queued", "running"]; @@ -29,3 +30,60 @@ export async function foldWaitingRetriesIntoContinuation(tx: Db, input: { .where(and(eq(agentWakeupRequests.companyId, input.companyId), inArray(agentWakeupRequests.id, retries), eq(agentWakeupRequests.status, "deferred_issue_execution"))); } + +export async function foldWaitingRetriesIntoRecordedContinuation(tx: Db, input: { + companyId: string; issueId: string; runId: string; recoveryActionId: string | null; sourceRunId: string | null; +}) { + if (!input.recoveryActionId || !input.sourceRunId) return; + const [record] = await tx.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, input.issueId), + sql`${issueRecoveryActions.id}::text = ${input.recoveryActionId}`, + sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,schema}' = ${FRESH_CONTINUATION_SCHEMA}`, + sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,sourceRunId}' = ${input.sourceRunId}`)).limit(1); + if (record) await foldWaitingRetriesIntoContinuation(tx, input); +} + +export type UndeliveredContinuation = { recoveryActionId: string; sourceRunId: string; isolation: string | null }; + +/** Caller holds the task row lock. The task's recorded continuation that no run took yet: a run admitted now is that continuation. */ +export async function undeliveredContinuation(tx: Db, input: { companyId: string; issueId: string }): Promise { + const [record] = await tx.select({ id: issueRecoveryActions.id, evidence: issueRecoveryActions.evidence }).from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, input.issueId), + eq(issueRecoveryActions.status, "resolved"), + sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`, + sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,schema}' = ${FRESH_CONTINUATION_SCHEMA}`)) + .orderBy(desc(sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,recordedAt}'`)).limit(1).for("update"); + if (!record) return null; + const fresh = record.evidence.freshContinuation as { recordedAt?: unknown; isolation?: unknown } | undefined; + const sourceRunId = (record.evidence.executionReconciliation as { runId?: unknown } | undefined)?.runId; + if (typeof sourceRunId !== "string" || typeof fresh?.recordedAt !== "string") return null; + const [intervening] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, input.companyId), sql`${heartbeatRuns.id}::text <> ${sourceRunId}`, + or(eq(heartbeatRuns.nativeIssueId, input.issueId), eq(heartbeatRuns.issueId, input.issueId), + sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${input.issueId}::text`), + sql`${heartbeatRuns.createdAt} >= ${fresh.recordedAt}::timestamptz`)).limit(1); + if (intervening) return null; + return { recoveryActionId: record.id, sourceRunId, isolation: typeof fresh.isolation === "string" ? fresh.isolation : null }; +} + +export function undeliveredContinuationContext(continuation: UndeliveredContinuation, context: Record) { + return { recoveryActionId: continuation.recoveryActionId, retryOfRunId: continuation.sourceRunId, forceFreshSession: true, + ...(typeof context.previousRunId === "string" ? {} : { previousRunId: continuation.sourceRunId }), + ...(continuation.isolation === "fresh_isolated_workspace" ? { freshContinuationIsolation: continuation.recoveryActionId } : {}) }; +} + +/** In the admitting transaction: the new run is the recorded continuation, so the delivery pass finds nothing left to start. */ +export async function deliverContinuationAsRun(tx: Db, input: { companyId: string; issueId: string; runId: string; + continuation: UndeliveredContinuation }) { + const { continuation } = input; + await tx.update(heartbeatRuns).set({ retryOfRunId: continuation.sourceRunId }) + .where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId))); + await tx.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify({ + continuationDelivery: "delivered", continuationRunId: input.runId, continuationDeliveredByPersonRun: true })}::jsonb` }) + .where(and(eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.id, continuation.recoveryActionId), + sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`)); + await settleCoalescedFreshContinuationHolds(tx, { companyId: input.companyId, primaryId: continuation.recoveryActionId, + delivery: "delivered", continuationRunId: input.runId }); + await foldWaitingRetriesIntoContinuation(tx, { companyId: input.companyId, issueId: input.issueId, runId: input.runId, + sourceRunId: continuation.sourceRunId }); +} diff --git a/server/src/services/heartbeat/queue.ts b/server/src/services/heartbeat/queue.ts index 3e25cddcb02..40d2af26f65 100644 --- a/server/src/services/heartbeat/queue.ts +++ b/server/src/services/heartbeat/queue.ts @@ -4,7 +4,8 @@ import { verifyBlockerStallWake } from "../recovery/blocker-stall-wakeups.js"; import { CONTEXT_COMPACTION_WAKE_REASON, ContextCompactionError, isContextCompactionRun, readContextCompactionRequest } from "../context-compaction.js"; import { isProcessRunAdmissionClosed } from "./run-admission.js"; import { continueFreshInTransaction, hasPendingFreshContinuation, isFreshIsolatedContinuationWake, settleCoalescedFreshContinuationHolds } from "../fresh-continuation-fallback.js"; -import { foldWaitingRetriesIntoContinuation } from "../fresh-continuation-retry-fold.js"; +import { deliverContinuationAsRun, foldWaitingRetriesIntoContinuation, foldWaitingRetriesIntoRecordedContinuation, undeliveredContinuation, + undeliveredContinuationContext } from "../fresh-continuation-retry-fold.js"; import { refreshRunStartYieldIfStale } from "../run-start-handover.js"; import { deliveredQueuedCommentIds } from "../queued-comment-delivery.js"; import { hasAutomaticInteractionDeliveryHold, INTERACTION_DELIVERY_DENIED, recheckQueuedRunInteractions, withInteractionDeliveryOrigin } from "../agent-interaction-delivery.js"; @@ -4028,6 +4029,10 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende }); return { kind: "skipped" as const }; } + const continuationToDeliver = !systemWake && !executionReconciliationWake && !executionBlocker + ? await undeliveredContinuation(tx as unknown as Db, { companyId: issue.companyId, issueId: issue.id }) : null; + if (continuationToDeliver) + Object.assign(enrichedContextSnapshot, undeliveredContinuationContext(continuationToDeliver, enrichedContextSnapshot)); if ((executionBlocker || opts.queuedCommentInterruptId) && !ownsReconstructionContinuation && !(await admitExplicitNativeContinuation({ db: tx as unknown as Db, companyId: issue.companyId, issueId: issue.id, agentId, actorType: taskRetryActorId ? "user" : opts.requestedByActorType, actorId: taskRetryActorId ?? opts.requestedByActorId, @@ -5083,9 +5088,12 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende eq(issueRecoveryActions.id, String(enrichedContextSnapshot.recoveryActionId)), sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`)); } - if (readNonEmptyString(enrichedContextSnapshot.freshContinuationIsolation)) - await foldWaitingRetriesIntoContinuation(tx as unknown as Db, { companyId: issue.companyId, issueId: issue.id, runId: newRun.id, - sourceRunId: readNonEmptyString(enrichedContextSnapshot.previousRunId) }); + if (continuationToDeliver) + await deliverContinuationAsRun(tx as unknown as Db, { companyId: issue.companyId, issueId: issue.id, runId: newRun.id, + continuation: continuationToDeliver }); + else if (reconciledSourceRunId) + await foldWaitingRetriesIntoRecordedContinuation(tx as unknown as Db, { companyId: issue.companyId, issueId: issue.id, + runId: newRun.id, recoveryActionId: readNonEmptyString(enrichedContextSnapshot.recoveryActionId), sourceRunId: reconciledSourceRunId }); await tx .update(agentWakeupRequests)