fix(retry): start a held Retry once a stopped preparation is proven stopped (#21295)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When an agent run fails, a person can press Retry. Admission starts
a new turn from the recorded work only after it proves that the previous
execution stopped.
> - For a legacy runner run that failed while it prepared its sandbox,
the proof reads the run's events and its environment lease.
> - After the automatic retry budget is spent, execution recovery
settles a no-replay hold and appends its own lifecycle event to the run.
The proof accepted only the error and measured events, so this platform
event made the proof fail.
> - The proof also asked for a workspace repair receipt for every run
with a lease, even when the run never recorded a remote workspace and
its sandbox was destroyed. No repair could ever exist, so the proof
could never pass.
> - Admission then refused the Retry without a named reason. The receipt
kept the generic `execution_recovery` wait ("Retry is waiting for
execution checks"), and every later Retry joined the same held receipt.
> - This pull request accepts the recovery hold's own settlement event
when it is bound to a settled hold for the same run, and asks for a
workspace repair only when the run holds prepared files.
> - The benefit is that Retry starts work at once when the previous
preparation is proven stopped, and it continues from the recorded work.

## Linked Issues or Issue Description

**What happened?**

A board user pressed Retry on a blocked task. The Retry was accepted,
but no run started for more than 13 hours. The task said "Retry is
waiting for execution checks. Your message is saved." A second Retry the
next day attached to the same held request and also did nothing. The
execution blocker said `legacy_execution_requires_reconciliation` with
`canRetry: true` and "A person can use Retry on this task ... to
continue once the previous execution has stopped." The previous run was
a legacy `paperclip_runner` run that failed with `setup_failed` while
preparing (`executionRecovery.kind = bootstrap`, `providerWorkStarted =
false`). It had no process identity, no controller lease, and its
Daytona sandbox lease had a `destroyed` termination receipt. Execution
recovery had settled a no-replay hold on it and appended the "Automatic
recovery stopped" lifecycle event.

**Expected behavior**

When the previous execution is proven stopped, a person's Retry starts a
new turn promptly and continues from the recorded work. The task view,
the execution blocker and admission agree that Retry works.

**Steps to reproduce**

1. Let a legacy `paperclip_runner` run fail while it prepares a remote
sandbox (`setup_failed`, bootstrap evidence with `providerWorkStarted =
false`) on its third automatic attempt.
2. Let the sandbox lease be released with a `destroyed` termination
receipt.
3. Let execution recovery settle the
`legacy_execution_requires_reconciliation` hold. It appends the
"Automatic recovery stopped" lifecycle event to the run.
4. Press Retry on the task.
5. Observe that the Retry receipt keeps `executionWait.reason =
execution_recovery` and no run starts. Press Retry again and observe
that it attaches to the same held receipt.

**Paperclip version or commit**

Staging Core `53608bd1d6`. Fixed on top of `origin/master` `c4f639ed1b`.

**Deployment mode**

Hosted cloud (staging), Daytona remote sandbox.

**Agent adapter(s) involved**

`paperclip_runner` (legacy runtime).

Related: #21267 (a Retry held on a resolved decision,
`decision_pending`). This pull request does not change its files.

## What Changed

- `readFailedPreparationSettlement` (`failed-preparation-settlement.ts`)
accepts one trailing host lifecycle event: the no-replay settlement
event that execution recovery appends. The event must have the exact
payload shape and must name a resolved
`legacy_execution_requires_reconciliation` hold whose evidence binds the
same run. Any other event still fails the proof.
- `failedNativePreparation` (`cancelled-native-startup.ts`) asks for a
verified workspace repair only when the run holds prepared files: a
recorded remote workspace, a workspace restore failure, or required
workspace recovery. A run with no recorded remote workspace and a
stopped sandbox has nothing to repair. The lease termination receipt,
the bootstrap evidence and the settled-event proof are still required.
- New regression test `legacy-preparation-retry-hold.test.ts` uses the
exact shape of the staging run.

## Verification

- Red/green: `cd server && npx vitest run
src/services/legacy-preparation-retry-hold.test.ts`.
- "starts the person's Retry once the stopped preparation's sandbox is
destroyed" fails on `origin/master` (the receipt keeps
`executionWait.reason = execution_recovery`, no run) and passes with
this change (one queued successor with `previousRunId` set to the failed
run and the Retry request id).
- The Retry stays held when the proof does not hold: a settlement event
bound to another run's hold, a provider execution event, a lease without
a termination receipt, and a recorded remote workspace without a repair
receipt.
- Related suites pass (299 tests): `task-retry`,
`saved-bootstrap-continuation`, `interrupted-preparation-hold-settle`,
`workspace-recovery-preparation`,
`workspace-recovery-preparation-gates`, `heartbeat-session-preparation`.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low. The change only widens two proofs for a run that never started
provider work: the platform's own settlement event is accepted only when
it binds a settled hold for the same run, and the repair receipt is
skipped only when the run recorded no files to repair.
- Retry still starts a new turn. It never replays the stopped run.
- Not in scope: two other tasks show the same generic wait with a
different cause on the saved-work side. Their blocker says `canRetry:
false` while the task view offers Retry. One has saved-work recovery
that failed with `source_unavailable`. The other shows recovered work
without a verified repair receipt. They need a separate fix.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser for read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-12 23:07:17 -06:00
1 parent db3d7223e0
commit 62d673cc3f
3 files changed
+152 -5

No files matched your search

@@ -68,7 +68,12 @@ async function failedNativePreparation(db: Db, run: Run, coordinator: Coordinato
return typeof issueId === "string" && leases.length > 0 && leases.every(lease => lease.issueId === issueId
&& lease.metadata?.pendingCleanupInFlight !== true && hasRemoteTerminationReceipt(lease))
&& await hasSettledFailedPreparationEvidence(db, run, historicalRetryAncestor)
&& await hasVerifiedWorkspaceRepair(db, run);
&& (!holdsPreparedFiles(run) || await hasVerifiedWorkspaceRepair(db, run));
}
function holdsPreparedFiles(run: Run) {
return run.runnerProfileJson?.remoteWorkspace != null || run.resultJson?.workspaceRestoreFailure != null
|| hasRequiredWorkspaceRecovery(run.resultJson);
}
async function hasSettledFailedPreparationEvidence(db: Db, run: Run, historicalRetryAncestor = false): Promise<boolean> {
@@ -1,5 +1,5 @@
import { and, asc, eq, sql } from "drizzle-orm";
import { heartbeatRunEvents, heartbeatRuns, type Db } from "@paperclipai/db";
import { heartbeatRunEvents, heartbeatRuns, issueRecoveryActions, type Db } from "@paperclipai/db";
import { isUuidLike } from "@paperclipai/shared";
type Source = Pick<typeof heartbeatRuns.$inferSelect, "id" | "companyId" | "agentId" | "error">;
@@ -10,7 +10,7 @@ export async function readFailedPreparationSettlement(db: Db, source: Source, al
if (typeof source.error !== "string" || !source.error.length) return null;
const events = await db.select().from(heartbeatRunEvents).where(and(
eq(heartbeatRunEvents.companyId, source.companyId), eq(heartbeatRunEvents.runId, source.id),
)).orderBy(asc(heartbeatRunEvents.seq)).limit(allowRunStarted ? 6 : 5);
)).orderBy(asc(heartbeatRunEvents.seq)).limit(allowRunStarted ? 7 : 6);
const hostEvent = (event: typeof heartbeatRunEvents.$inferSelect) => event.agentId === source.agentId
&& event.stream === "system" && event.sourceInstanceId === null && event.sourceEventId === null
&& event.sourceSeq === null && event.sourcePayloadSha256 === null && event.protocolSchemaVersion === null;
@@ -35,8 +35,9 @@ export async function readFailedPreparationSettlement(db: Db, source: Source, al
if (!hostEvent(started) || started.seq !== 1 + offset || started.level !== "info" || started.payload !== null) return null;
offset++;
}
const [error, measured] = events.slice(offset);
if (events.length !== 2 + offset || !error || !measured || !hostEvent(error) || !hostEvent(measured)
const [error, measured, settledHold] = events.slice(offset);
if (settledHold && !(hostEvent(settledHold) && await isRecoveryHoldSettlement(db, source, settledHold, 3 + offset))) return null;
if (events.length !== (settledHold ? 3 : 2) + offset || !error || !measured || !hostEvent(error) || !hostEvent(measured)
|| error.seq !== 1 + offset || error.eventType !== "error" || error.level !== "error"
|| error.message !== source.error || error.payload !== null
|| measured.seq !== 2 + offset || measured.eventType !== "lifecycle" || measured.level !== "info" || measured.message !== "task.run.measured"
@@ -73,3 +74,18 @@ async function matchesScheduledRetryPromotion(db: Db, source: Source, event: typ
&& event.payload.scheduledRetryReason === run.scheduledRetryReason
&& Object.keys(event.payload).every(key => ["scheduledRetryAt", "scheduledRetryAttempt", "scheduledRetryReason"].includes(key)));
}
async function isRecoveryHoldSettlement(db: Db, source: Source, event: typeof heartbeatRunEvents.$inferSelect, seq: number) {
const payload = event.payload;
if (event.seq !== seq || event.eventType !== "lifecycle" || event.level !== "warn" || !payload
|| Object.keys(payload).some(key => !["recoveryActionId", "cause", "automaticRecovery", "replay"].includes(key))
|| payload.cause !== "legacy_execution_requires_reconciliation" || payload.automaticRecovery !== "preserve_without_replay_v1"
|| payload.replay !== "blocked" || typeof payload.recoveryActionId !== "string" || !isUuidLike(payload.recoveryActionId)) return false;
const [hold] = await db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, source.companyId), eq(issueRecoveryActions.id, payload.recoveryActionId),
eq(issueRecoveryActions.cause, "legacy_execution_requires_reconciliation"), eq(issueRecoveryActions.status, "resolved"),
sql`${issueRecoveryActions.evidence}->>'runId' = ${source.id}`,
sql`${issueRecoveryActions.evidence}#>>'{automaticRecovery,replay}' = 'blocked'`,
)).limit(1);
return Boolean(hold);
}
@@ -0,0 +1,126 @@
import { randomUUID } from "node:crypto";
import { and, eq } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { agents, agentWakeupRequests, companies, createDb, environmentLeases, executionWorkspaces,
heartbeatRunEvents, heartbeatRuns, issueRecoveryActions, issues } from "@paperclipai/db";
import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
import { requestTaskRetry } from "./task-retry.js";
import { heartbeatService } from "./heartbeat.js";
import { getExecutionBlocker } from "./execution-blocker.js";
describe("Retry of a legacy preparation that automatic recovery stopped", () => {
let postgres: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
beforeAll(async () => {
postgres = await startEmbeddedPostgresTestDatabase("legacy-preparation-retry-");
db = createDb(postgres.connectionString);
}, 30_000);
afterAll(async () => { await db?.$client.end(); await postgres?.cleanup(); });
async function seed() {
const companyId = randomUUID(), issueId = randomUUID(), agentId = randomUUID();
const parentId = randomUUID(), sourceId = randomUUID(), actionId = randomUUID();
const workspaceId = randomUUID(), leaseId = randomUUID(), providerLeaseId = randomUUID();
const actorId = "board", now = Date.now(), error = "Worker process exited (code=null, signal=SIGKILL)";
const controllerProcess = { pid: 3, bootId: randomUUID(), hostHash: "f".repeat(32), processStartedAt: new Date(now - 3_600_000).toISOString() };
await db.insert(companies).values({ id: companyId, name: "Legacy preparation", issuePrefix: `L${companyId.slice(0, 6)}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Worker", adapterType: "paperclip_runner", status: "idle",
runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } } });
await db.insert(issues).values({ id: issueId, companyId, title: "Repair a failed run", status: "blocked", assigneeAgentId: agentId });
await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, sourceIssueId: issueId,
mode: "isolated_workspace", strategyType: "git_worktree", name: "Task workspace", providerType: "cloud_sandbox" });
await db.update(issues).set({ executionWorkspaceId: workspaceId }).where(eq(issues.id, issueId));
await db.insert(heartbeatRuns).values({ id: parentId, companyId, agentId, issueId, status: "failed",
runtimeMode: "legacy", errorCode: "environment_provider_unavailable", executionStage: "preparing",
createdAt: new Date(now - 600_000), startedAt: new Date(now - 590_000), finishedAt: new Date(now - 400_000),
contextSnapshot: { issueId }, runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } },
resultJson: { stopReason: "adapter_failed", executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } });
const scheduledRetryAt = new Date(now - 300_000);
const [source] = await db.insert(heartbeatRuns).values({ id: sourceId, companyId, agentId, issueId, status: "failed",
runtimeMode: "legacy", errorCode: "setup_failed", error, executionStage: "preparing", invocationSource: "automation",
createdAt: new Date(now - 360_000), startedAt: new Date(now - 250_000), finishedAt: new Date(now - 200_000),
retryOfRunId: parentId, scheduledRetryAt, scheduledRetryAttempt: 2, scheduledRetryReason: "transient_failure",
controllerBootId: controllerProcess.bootId, controllerProcess,
usageJson: { ledgerScope: { issueId, projectId: null, billingCode: null } },
contextSnapshot: { issueId, taskId: issueId, retryOfRunId: parentId, retryReason: "transient_failure",
scheduledRetryAt: scheduledRetryAt.toISOString(), scheduledRetryAttempt: 2, wakeReason: "transient_failure_retry" },
runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" },
chatControlRecoveryAdmission: { phase: "required", runId: sourceId, agentId, issueId, version: 1, companyId } },
resultJson: { stopReason: "adapter_failed", failureCause: "JsonRpcCallError", timeoutFired: false, timeoutSource: "config",
executionRecovery: { kind: "bootstrap", providerWorkStarted: false }, timeoutConfigured: false, effectiveTimeoutSec: 0 },
}).returning();
const event = { companyId, agentId, runId: sourceId, stream: "system" };
await db.insert(heartbeatRunEvents).values([
{ ...event, seq: 1, eventType: "lifecycle", level: "info", createdAt: new Date(scheduledRetryAt.getTime() + 200),
message: "Scheduled retry became due and was promoted to the queued run pool",
payload: { scheduledRetryAt: scheduledRetryAt.toISOString(), scheduledRetryReason: "transient_failure", scheduledRetryAttempt: 2 } },
{ ...event, seq: 2, eventType: "error", level: "error", message: error },
{ ...event, seq: 3, eventType: "lifecycle", level: "info", message: "task.run.measured",
payload: { phase: "task.run.measured", outcome: "failed", durationMs: 141_150, durableSettlement: true } },
{ ...event, seq: 4, eventType: "lifecycle", level: "warn",
message: "Automatic recovery stopped. Recorded work is preserved; actions with unverified outcomes will not be repeated.",
payload: { cause: "legacy_execution_requires_reconciliation", replay: "blocked", recoveryActionId: actionId,
automaticRecovery: "preserve_without_replay_v1" } },
]);
const releasedAt = new Date(now - 150_000);
await db.insert(environmentLeases).values({ id: leaseId, companyId, issueId, heartbeatRunId: sourceId,
executionWorkspaceId: workspaceId, provider: "daytona", providerLeaseId, status: "expired",
leasePolicy: "reuse_by_environment", releasedAt, cleanupStatus: "success", failureReason: "pending_cleanup_retry",
metadata: { pendingCleanupInFlight: false, remoteExecutionTermination: { schema: "paperclip.remote-termination.v1",
companyId, runId: sourceId, leaseId, provider: "daytona", providerLeaseId, state: "destroyed",
confirmedAt: releasedAt.toISOString() } } });
await db.insert(issueRecoveryActions).values({ id: actionId, companyId, sourceIssueId: issueId,
kind: "active_run_watchdog", cause: "legacy_execution_requires_reconciliation", fingerprint: sourceId,
status: "resolved", outcome: "blocked",
nextAction: "Automatic recovery stopped. Recorded work is preserved; actions with unverified outcomes will not be repeated.",
evidence: { runId: sourceId, automaticRecovery: { replay: "blocked", actionOutcome: "unknown" } } });
await db.insert(heartbeatRuns).values({ companyId, agentId, status: "running" });
return { companyId, issueId, agentId, actorId, source, actionId, leaseId };
}
it("starts the person's Retry once the stopped preparation's sandbox is destroyed", async () => {
const f = await seed();
const blocker = await getExecutionBlocker(db, f.companyId, f.issueId);
expect(blocker).toMatchObject({ cause: "legacy_execution_requires_reconciliation", canRetry: true });
const receipt = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issueId, actorId: f.actorId,
observedRunId: f.source.id, idempotencyKey: randomUUID(), actorSource: "local_implicit" });
expect(receipt).toMatchObject({ accepted: true });
const heartbeat = heartbeatService(db);
await heartbeat.resumeTaskRetry(f.companyId, receipt.requestId!);
const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId!));
expect(wake.payload?.executionWait).toBeUndefined();
expect(wake).toMatchObject({ status: "coalesced", runId: expect.any(String) });
const [successor] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, wake.runId!));
expect(successor.status).toBe("queued");
expect(successor.contextSnapshot).toMatchObject({ previousRunId: f.source.id, taskRetryRequestId: receipt.requestId });
});
it.each(["foreign_hold", "provider_event", "unstopped_lease", "unsaved_remote_workspace"] as const)(
"keeps the Retry held when the stopped preparation is not proven: %s", async gate => {
const f = await seed();
if (gate === "foreign_hold") {
const [other] = await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: f.issueId,
kind: "active_run_watchdog", cause: "legacy_execution_requires_reconciliation", fingerprint: randomUUID(),
status: "resolved", outcome: "blocked", nextAction: "Other run.",
evidence: { runId: randomUUID(), automaticRecovery: { replay: "blocked", actionOutcome: "unknown" } } }).returning();
await db.update(heartbeatRunEvents).set({ payload: { cause: "legacy_execution_requires_reconciliation", replay: "blocked",
recoveryActionId: other.id, automaticRecovery: "preserve_without_replay_v1" } })
.where(and(eq(heartbeatRunEvents.runId, f.source.id), eq(heartbeatRunEvents.seq, 4)));
}
if (gate === "provider_event") await db.insert(heartbeatRunEvents).values({ companyId: f.companyId, agentId: f.agentId,
runId: f.source.id, seq: 5, eventType: "tool.execution.started", stream: "system" });
if (gate === "unstopped_lease") await db.update(environmentLeases).set({ metadata: { pendingCleanupInFlight: false } })
.where(eq(environmentLeases.id, f.leaseId));
if (gate === "unsaved_remote_workspace") await db.update(heartbeatRuns).set({ runnerProfileJson: {
...f.source.runnerProfileJson, remoteWorkspace: { schema: "paperclip.remote-workspace-run.v1", workspaceId: randomUUID(),
leaseId: f.leaseId, providerLeaseId: randomUUID(), remoteCwd: "/fixture", epoch: 1, state: "prepared", snapshotId: null } } })
.where(eq(heartbeatRuns.id, f.source.id));
const receipt = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issueId, actorId: f.actorId,
observedRunId: f.source.id, idempotencyKey: randomUUID(), actorSource: "local_implicit" });
await heartbeatService(db).resumeTaskRetry(f.companyId, receipt.requestId!);
const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId!));
expect(wake).toMatchObject({ status: "deferred_issue_execution", runId: null });
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
eq(heartbeatRuns.status, "queued")))).toHaveLength(0);
});
});