From 62d673cc3fa59ae81234c0b3044beeded389a11b Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 12 Nov 2026 23:07:17 -0600 Subject: [PATCH] fix(retry): start a held Retry once a stopped preparation is proven stopped (#21295) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When an agent run fails, a person can press Retry. Admission starts a new turn from the recorded work only after it proves that the previous execution stopped. > - For a legacy runner run that failed while it prepared its sandbox, the proof reads the run's events and its environment lease. > - After the automatic retry budget is spent, execution recovery settles a no-replay hold and appends its own lifecycle event to the run. The proof accepted only the error and measured events, so this platform event made the proof fail. > - The proof also asked for a workspace repair receipt for every run with a lease, even when the run never recorded a remote workspace and its sandbox was destroyed. No repair could ever exist, so the proof could never pass. > - Admission then refused the Retry without a named reason. The receipt kept the generic `execution_recovery` wait ("Retry is waiting for execution checks"), and every later Retry joined the same held receipt. > - This pull request accepts the recovery hold's own settlement event when it is bound to a settled hold for the same run, and asks for a workspace repair only when the run holds prepared files. > - The benefit is that Retry starts work at once when the previous preparation is proven stopped, and it continues from the recorded work. ## Linked Issues or Issue Description **What happened?** A board user pressed Retry on a blocked task. The Retry was accepted, but no run started for more than 13 hours. The task said "Retry is waiting for execution checks. Your message is saved." A second Retry the next day attached to the same held request and also did nothing. The execution blocker said `legacy_execution_requires_reconciliation` with `canRetry: true` and "A person can use Retry on this task ... to continue once the previous execution has stopped." The previous run was a legacy `paperclip_runner` run that failed with `setup_failed` while preparing (`executionRecovery.kind = bootstrap`, `providerWorkStarted = false`). It had no process identity, no controller lease, and its Daytona sandbox lease had a `destroyed` termination receipt. Execution recovery had settled a no-replay hold on it and appended the "Automatic recovery stopped" lifecycle event. **Expected behavior** When the previous execution is proven stopped, a person's Retry starts a new turn promptly and continues from the recorded work. The task view, the execution blocker and admission agree that Retry works. **Steps to reproduce** 1. Let a legacy `paperclip_runner` run fail while it prepares a remote sandbox (`setup_failed`, bootstrap evidence with `providerWorkStarted = false`) on its third automatic attempt. 2. Let the sandbox lease be released with a `destroyed` termination receipt. 3. Let execution recovery settle the `legacy_execution_requires_reconciliation` hold. It appends the "Automatic recovery stopped" lifecycle event to the run. 4. Press Retry on the task. 5. Observe that the Retry receipt keeps `executionWait.reason = execution_recovery` and no run starts. Press Retry again and observe that it attaches to the same held receipt. **Paperclip version or commit** Staging Core `53608bd1d6`. Fixed on top of `origin/master` `c4f639ed1b`. **Deployment mode** Hosted cloud (staging), Daytona remote sandbox. **Agent adapter(s) involved** `paperclip_runner` (legacy runtime). Related: #21267 (a Retry held on a resolved decision, `decision_pending`). This pull request does not change its files. ## What Changed - `readFailedPreparationSettlement` (`failed-preparation-settlement.ts`) accepts one trailing host lifecycle event: the no-replay settlement event that execution recovery appends. The event must have the exact payload shape and must name a resolved `legacy_execution_requires_reconciliation` hold whose evidence binds the same run. Any other event still fails the proof. - `failedNativePreparation` (`cancelled-native-startup.ts`) asks for a verified workspace repair only when the run holds prepared files: a recorded remote workspace, a workspace restore failure, or required workspace recovery. A run with no recorded remote workspace and a stopped sandbox has nothing to repair. The lease termination receipt, the bootstrap evidence and the settled-event proof are still required. - New regression test `legacy-preparation-retry-hold.test.ts` uses the exact shape of the staging run. ## Verification - Red/green: `cd server && npx vitest run src/services/legacy-preparation-retry-hold.test.ts`. - "starts the person's Retry once the stopped preparation's sandbox is destroyed" fails on `origin/master` (the receipt keeps `executionWait.reason = execution_recovery`, no run) and passes with this change (one queued successor with `previousRunId` set to the failed run and the Retry request id). - The Retry stays held when the proof does not hold: a settlement event bound to another run's hold, a provider execution event, a lease without a termination receipt, and a recorded remote workspace without a repair receipt. - Related suites pass (299 tests): `task-retry`, `saved-bootstrap-continuation`, `interrupted-preparation-hold-settle`, `workspace-recovery-preparation`, `workspace-recovery-preparation-gates`, `heartbeat-session-preparation`. - `npx tsc --noEmit -p server/tsconfig.json` passes. ## Risks - Low. The change only widens two proofs for a run that never started provider work: the platform's own settlement event is accepted only when it binds a settled hold for the same run, and the repair receipt is skipped only when the run recorded no files to repair. - Retry still starts a new turn. It never replays the stopped run. - Not in scope: two other tasks show the same generic wait with a different cause on the saved-work side. Their blocker says `canRetry: false` while the task view offers Retry. One has saved-work recovery that failed with `source_unavailable`. The other shows recovered work without a verified repair receipt. They need a separate fix. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, browser for read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- .../src/services/cancelled-native-startup.ts | 7 +- .../services/failed-preparation-settlement.ts | 24 +++- .../legacy-preparation-retry-hold.test.ts | 126 ++++++++++++++++++ 3 files changed, 152 insertions(+), 5 deletions(-) create mode 100644 server/src/services/legacy-preparation-retry-hold.test.ts diff --git a/server/src/services/cancelled-native-startup.ts b/server/src/services/cancelled-native-startup.ts index d2a40388ef5..38843d08951 100644 --- a/server/src/services/cancelled-native-startup.ts +++ b/server/src/services/cancelled-native-startup.ts @@ -68,7 +68,12 @@ async function failedNativePreparation(db: Db, run: Run, coordinator: Coordinato return typeof issueId === "string" && leases.length > 0 && leases.every(lease => lease.issueId === issueId && lease.metadata?.pendingCleanupInFlight !== true && hasRemoteTerminationReceipt(lease)) && await hasSettledFailedPreparationEvidence(db, run, historicalRetryAncestor) - && await hasVerifiedWorkspaceRepair(db, run); + && (!holdsPreparedFiles(run) || await hasVerifiedWorkspaceRepair(db, run)); +} + +function holdsPreparedFiles(run: Run) { + return run.runnerProfileJson?.remoteWorkspace != null || run.resultJson?.workspaceRestoreFailure != null + || hasRequiredWorkspaceRecovery(run.resultJson); } async function hasSettledFailedPreparationEvidence(db: Db, run: Run, historicalRetryAncestor = false): Promise { diff --git a/server/src/services/failed-preparation-settlement.ts b/server/src/services/failed-preparation-settlement.ts index 3fb3844b73a..bcb4a55118f 100644 --- a/server/src/services/failed-preparation-settlement.ts +++ b/server/src/services/failed-preparation-settlement.ts @@ -1,5 +1,5 @@ import { and, asc, eq, sql } from "drizzle-orm"; -import { heartbeatRunEvents, heartbeatRuns, type Db } from "@paperclipai/db"; +import { heartbeatRunEvents, heartbeatRuns, issueRecoveryActions, type Db } from "@paperclipai/db"; import { isUuidLike } from "@paperclipai/shared"; type Source = Pick; @@ -10,7 +10,7 @@ export async function readFailedPreparationSettlement(db: Db, source: Source, al if (typeof source.error !== "string" || !source.error.length) return null; const events = await db.select().from(heartbeatRunEvents).where(and( eq(heartbeatRunEvents.companyId, source.companyId), eq(heartbeatRunEvents.runId, source.id), - )).orderBy(asc(heartbeatRunEvents.seq)).limit(allowRunStarted ? 6 : 5); + )).orderBy(asc(heartbeatRunEvents.seq)).limit(allowRunStarted ? 7 : 6); const hostEvent = (event: typeof heartbeatRunEvents.$inferSelect) => event.agentId === source.agentId && event.stream === "system" && event.sourceInstanceId === null && event.sourceEventId === null && event.sourceSeq === null && event.sourcePayloadSha256 === null && event.protocolSchemaVersion === null; @@ -35,8 +35,9 @@ export async function readFailedPreparationSettlement(db: Db, source: Source, al if (!hostEvent(started) || started.seq !== 1 + offset || started.level !== "info" || started.payload !== null) return null; offset++; } - const [error, measured] = events.slice(offset); - if (events.length !== 2 + offset || !error || !measured || !hostEvent(error) || !hostEvent(measured) + const [error, measured, settledHold] = events.slice(offset); + if (settledHold && !(hostEvent(settledHold) && await isRecoveryHoldSettlement(db, source, settledHold, 3 + offset))) return null; + if (events.length !== (settledHold ? 3 : 2) + offset || !error || !measured || !hostEvent(error) || !hostEvent(measured) || error.seq !== 1 + offset || error.eventType !== "error" || error.level !== "error" || error.message !== source.error || error.payload !== null || measured.seq !== 2 + offset || measured.eventType !== "lifecycle" || measured.level !== "info" || measured.message !== "task.run.measured" @@ -73,3 +74,18 @@ async function matchesScheduledRetryPromotion(db: Db, source: Source, event: typ && event.payload.scheduledRetryReason === run.scheduledRetryReason && Object.keys(event.payload).every(key => ["scheduledRetryAt", "scheduledRetryAttempt", "scheduledRetryReason"].includes(key))); } + +async function isRecoveryHoldSettlement(db: Db, source: Source, event: typeof heartbeatRunEvents.$inferSelect, seq: number) { + const payload = event.payload; + if (event.seq !== seq || event.eventType !== "lifecycle" || event.level !== "warn" || !payload + || Object.keys(payload).some(key => !["recoveryActionId", "cause", "automaticRecovery", "replay"].includes(key)) + || payload.cause !== "legacy_execution_requires_reconciliation" || payload.automaticRecovery !== "preserve_without_replay_v1" + || payload.replay !== "blocked" || typeof payload.recoveryActionId !== "string" || !isUuidLike(payload.recoveryActionId)) return false; + const [hold] = await db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.companyId, source.companyId), eq(issueRecoveryActions.id, payload.recoveryActionId), + eq(issueRecoveryActions.cause, "legacy_execution_requires_reconciliation"), eq(issueRecoveryActions.status, "resolved"), + sql`${issueRecoveryActions.evidence}->>'runId' = ${source.id}`, + sql`${issueRecoveryActions.evidence}#>>'{automaticRecovery,replay}' = 'blocked'`, + )).limit(1); + return Boolean(hold); +} diff --git a/server/src/services/legacy-preparation-retry-hold.test.ts b/server/src/services/legacy-preparation-retry-hold.test.ts new file mode 100644 index 00000000000..4b7092950b6 --- /dev/null +++ b/server/src/services/legacy-preparation-retry-hold.test.ts @@ -0,0 +1,126 @@ +import { randomUUID } from "node:crypto"; +import { and, eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { agents, agentWakeupRequests, companies, createDb, environmentLeases, executionWorkspaces, + heartbeatRunEvents, heartbeatRuns, issueRecoveryActions, issues } from "@paperclipai/db"; +import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js"; +import { requestTaskRetry } from "./task-retry.js"; +import { heartbeatService } from "./heartbeat.js"; +import { getExecutionBlocker } from "./execution-blocker.js"; + +describe("Retry of a legacy preparation that automatic recovery stopped", () => { + let postgres: Awaited>; + let db: ReturnType; + beforeAll(async () => { + postgres = await startEmbeddedPostgresTestDatabase("legacy-preparation-retry-"); + db = createDb(postgres.connectionString); + }, 30_000); + afterAll(async () => { await db?.$client.end(); await postgres?.cleanup(); }); + + async function seed() { + const companyId = randomUUID(), issueId = randomUUID(), agentId = randomUUID(); + const parentId = randomUUID(), sourceId = randomUUID(), actionId = randomUUID(); + const workspaceId = randomUUID(), leaseId = randomUUID(), providerLeaseId = randomUUID(); + const actorId = "board", now = Date.now(), error = "Worker process exited (code=null, signal=SIGKILL)"; + const controllerProcess = { pid: 3, bootId: randomUUID(), hostHash: "f".repeat(32), processStartedAt: new Date(now - 3_600_000).toISOString() }; + await db.insert(companies).values({ id: companyId, name: "Legacy preparation", issuePrefix: `L${companyId.slice(0, 6)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Worker", adapterType: "paperclip_runner", status: "idle", + runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } } }); + await db.insert(issues).values({ id: issueId, companyId, title: "Repair a failed run", status: "blocked", assigneeAgentId: agentId }); + await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, sourceIssueId: issueId, + mode: "isolated_workspace", strategyType: "git_worktree", name: "Task workspace", providerType: "cloud_sandbox" }); + await db.update(issues).set({ executionWorkspaceId: workspaceId }).where(eq(issues.id, issueId)); + await db.insert(heartbeatRuns).values({ id: parentId, companyId, agentId, issueId, status: "failed", + runtimeMode: "legacy", errorCode: "environment_provider_unavailable", executionStage: "preparing", + createdAt: new Date(now - 600_000), startedAt: new Date(now - 590_000), finishedAt: new Date(now - 400_000), + contextSnapshot: { issueId }, runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } }, + resultJson: { stopReason: "adapter_failed", executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } }); + const scheduledRetryAt = new Date(now - 300_000); + const [source] = await db.insert(heartbeatRuns).values({ id: sourceId, companyId, agentId, issueId, status: "failed", + runtimeMode: "legacy", errorCode: "setup_failed", error, executionStage: "preparing", invocationSource: "automation", + createdAt: new Date(now - 360_000), startedAt: new Date(now - 250_000), finishedAt: new Date(now - 200_000), + retryOfRunId: parentId, scheduledRetryAt, scheduledRetryAttempt: 2, scheduledRetryReason: "transient_failure", + controllerBootId: controllerProcess.bootId, controllerProcess, + usageJson: { ledgerScope: { issueId, projectId: null, billingCode: null } }, + contextSnapshot: { issueId, taskId: issueId, retryOfRunId: parentId, retryReason: "transient_failure", + scheduledRetryAt: scheduledRetryAt.toISOString(), scheduledRetryAttempt: 2, wakeReason: "transient_failure_retry" }, + runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" }, + chatControlRecoveryAdmission: { phase: "required", runId: sourceId, agentId, issueId, version: 1, companyId } }, + resultJson: { stopReason: "adapter_failed", failureCause: "JsonRpcCallError", timeoutFired: false, timeoutSource: "config", + executionRecovery: { kind: "bootstrap", providerWorkStarted: false }, timeoutConfigured: false, effectiveTimeoutSec: 0 }, + }).returning(); + const event = { companyId, agentId, runId: sourceId, stream: "system" }; + await db.insert(heartbeatRunEvents).values([ + { ...event, seq: 1, eventType: "lifecycle", level: "info", createdAt: new Date(scheduledRetryAt.getTime() + 200), + message: "Scheduled retry became due and was promoted to the queued run pool", + payload: { scheduledRetryAt: scheduledRetryAt.toISOString(), scheduledRetryReason: "transient_failure", scheduledRetryAttempt: 2 } }, + { ...event, seq: 2, eventType: "error", level: "error", message: error }, + { ...event, seq: 3, eventType: "lifecycle", level: "info", message: "task.run.measured", + payload: { phase: "task.run.measured", outcome: "failed", durationMs: 141_150, durableSettlement: true } }, + { ...event, seq: 4, eventType: "lifecycle", level: "warn", + message: "Automatic recovery stopped. Recorded work is preserved; actions with unverified outcomes will not be repeated.", + payload: { cause: "legacy_execution_requires_reconciliation", replay: "blocked", recoveryActionId: actionId, + automaticRecovery: "preserve_without_replay_v1" } }, + ]); + const releasedAt = new Date(now - 150_000); + await db.insert(environmentLeases).values({ id: leaseId, companyId, issueId, heartbeatRunId: sourceId, + executionWorkspaceId: workspaceId, provider: "daytona", providerLeaseId, status: "expired", + leasePolicy: "reuse_by_environment", releasedAt, cleanupStatus: "success", failureReason: "pending_cleanup_retry", + metadata: { pendingCleanupInFlight: false, remoteExecutionTermination: { schema: "paperclip.remote-termination.v1", + companyId, runId: sourceId, leaseId, provider: "daytona", providerLeaseId, state: "destroyed", + confirmedAt: releasedAt.toISOString() } } }); + await db.insert(issueRecoveryActions).values({ id: actionId, companyId, sourceIssueId: issueId, + kind: "active_run_watchdog", cause: "legacy_execution_requires_reconciliation", fingerprint: sourceId, + status: "resolved", outcome: "blocked", + nextAction: "Automatic recovery stopped. Recorded work is preserved; actions with unverified outcomes will not be repeated.", + evidence: { runId: sourceId, automaticRecovery: { replay: "blocked", actionOutcome: "unknown" } } }); + await db.insert(heartbeatRuns).values({ companyId, agentId, status: "running" }); + return { companyId, issueId, agentId, actorId, source, actionId, leaseId }; + } + + it("starts the person's Retry once the stopped preparation's sandbox is destroyed", async () => { + const f = await seed(); + const blocker = await getExecutionBlocker(db, f.companyId, f.issueId); + expect(blocker).toMatchObject({ cause: "legacy_execution_requires_reconciliation", canRetry: true }); + const receipt = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issueId, actorId: f.actorId, + observedRunId: f.source.id, idempotencyKey: randomUUID(), actorSource: "local_implicit" }); + expect(receipt).toMatchObject({ accepted: true }); + const heartbeat = heartbeatService(db); + await heartbeat.resumeTaskRetry(f.companyId, receipt.requestId!); + const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId!)); + expect(wake.payload?.executionWait).toBeUndefined(); + expect(wake).toMatchObject({ status: "coalesced", runId: expect.any(String) }); + const [successor] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, wake.runId!)); + expect(successor.status).toBe("queued"); + expect(successor.contextSnapshot).toMatchObject({ previousRunId: f.source.id, taskRetryRequestId: receipt.requestId }); + }); + + it.each(["foreign_hold", "provider_event", "unstopped_lease", "unsaved_remote_workspace"] as const)( + "keeps the Retry held when the stopped preparation is not proven: %s", async gate => { + const f = await seed(); + if (gate === "foreign_hold") { + const [other] = await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: f.issueId, + kind: "active_run_watchdog", cause: "legacy_execution_requires_reconciliation", fingerprint: randomUUID(), + status: "resolved", outcome: "blocked", nextAction: "Other run.", + evidence: { runId: randomUUID(), automaticRecovery: { replay: "blocked", actionOutcome: "unknown" } } }).returning(); + await db.update(heartbeatRunEvents).set({ payload: { cause: "legacy_execution_requires_reconciliation", replay: "blocked", + recoveryActionId: other.id, automaticRecovery: "preserve_without_replay_v1" } }) + .where(and(eq(heartbeatRunEvents.runId, f.source.id), eq(heartbeatRunEvents.seq, 4))); + } + if (gate === "provider_event") await db.insert(heartbeatRunEvents).values({ companyId: f.companyId, agentId: f.agentId, + runId: f.source.id, seq: 5, eventType: "tool.execution.started", stream: "system" }); + if (gate === "unstopped_lease") await db.update(environmentLeases).set({ metadata: { pendingCleanupInFlight: false } }) + .where(eq(environmentLeases.id, f.leaseId)); + if (gate === "unsaved_remote_workspace") await db.update(heartbeatRuns).set({ runnerProfileJson: { + ...f.source.runnerProfileJson, remoteWorkspace: { schema: "paperclip.remote-workspace-run.v1", workspaceId: randomUUID(), + leaseId: f.leaseId, providerLeaseId: randomUUID(), remoteCwd: "/fixture", epoch: 1, state: "prepared", snapshotId: null } } }) + .where(eq(heartbeatRuns.id, f.source.id)); + const receipt = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issueId, actorId: f.actorId, + observedRunId: f.source.id, idempotencyKey: randomUUID(), actorSource: "local_implicit" }); + await heartbeatService(db).resumeTaskRetry(f.companyId, receipt.requestId!); + const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId!)); + expect(wake).toMatchObject({ status: "deferred_issue_execution", runId: null }); + expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), + eq(heartbeatRuns.status, "queued")))).toHaveLength(0); + }); +});