fix(recovery): continue twice-interrupted tasks and waiting follow-ups after a release swap (#21250)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is cut off by a restart or a release swap, the
fresh-continuation fallback continues the task in a new run.
> - The continuation goes through normal admission, or through a
person's saved follow-up when one is waiting.
> - In release-swap waves, three gaps left tasks stopped with no run:
admission refused an isolated continuation of a task with no project, a
person's saved follow-up behind the replaced run was never promoted, and
the fallback relied on a saved follow-up that the cut-off run had
already read.
> - This pull request closes those three gaps.
> - The benefit is that a twice-interrupted task, and a task with a
waiting follow-up, continue by themselves.

## Linked Issues or Issue Description

Refs #21135, #21222

**What happened?**
1. A task with no project was interrupted twice. The second continuation
used a fresh isolated workspace, which clears the task's workspace
binding. The admission preflight still required that binding for an
isolated task with no project, so it refused the continuation as
`workspace_worktree_requires_project` about every 15 s. Each refusal
re-blocked the task and added a comment (329 to 428 skipped wakes per
wave). The runtime itself supports this case: it prepares a new
workspace (`create_new`).
2. The fallback continued a task with delivery `saved_input` (a person's
deferred follow-up is the next turn). The stranded-queue pass refused to
promote that follow-up, because the replaced native run's finalization
was `terminal_failure` (restart recovery ended it), not `committed`. The
release path would also have re-blocked the task with a reconciliation
incident. The follow-up never ran.
3. The fallback chose `saved_input` because a deferred user wake
existed, but the cut-off run had already taken that message into its
turn. Promotion settled the message as delivered, and no run followed.
The task showed "working".

**Expected behavior**
An interrupted task continues by itself in each case, with no person
action and no repeated comments.

**Steps to reproduce**
1. Run a release-swap wave with projectless tasks in isolated mode and
person follow-ups during runs.
2. Interrupt the same task twice, or interrupt a run that has a deferred
follow-up behind it.
3. See the task stay blocked or "working" with no run.

## What Changed

- Admission (`server/src/services/execution-workspace-policy.ts`,
`server/src/services/heartbeat/queue.ts`,
`server/src/services/fresh-continuation-fallback.ts`):
`isUnrunnableWorktreeCombo` accepts `freshIsolatedContinuation`. The
admission preflight sets it from the new
`isFreshIsolatedContinuationWake`, which returns at once (no query)
unless the wake context carries `freshContinuationIsolation`. This does
not touch the dispatch ranking.
- Follow-up promotion
(`server/src/modules/wake-queue/adapters/postgres.ts`): the
stranded-queue pass and the release path skip the native finalization
gate and the reconciliation incident for a run that the fallback
superseded (`agentRunSupersededByFreshContinuation`).
- Saved-input choice
(`server/src/services/fresh-continuation-fallback.ts`): a deferred user
wake counts as saved input only when it has no comment ids, or still has
a message that the agent has not received
(`undeliveredLegacyUserCommentIds`, called, not changed). Otherwise the
fallback records its own continuation.

- Review round 1: (a) whenever the old writer is not proven stopped
(`fresh_isolated_workspace`), the fallback records its own isolated
continuation first, also when the task had a binding; saved input
follows it on the new workspace. (b) Release skips the manual-admission
hold and the pending-cleanup check for a run the fallback replaced with
an isolated continuation (`fresh_isolated_workspace` or `remote_epoch`,
via `freshContinuationIsolatesRun`); other replacements still wait.
- Review round 2 (the isolated continuation must run before the waiting
follow-up): while the fallback's `fresh_isolated_workspace` continuation
is still pending delivery, release leaves the person's follow-up
deferred. Otherwise the follow-up would become the task's next run
without the isolation marker, and delivery would count it as the
continuation and skip the isolated run. The continuation's own release
promotes the follow-up behind it. `freshContinuationIsolatesRun` now
also reports `continuationPending`.

## Verification

Red-to-green tests (each failed before its fix and passes after it):
- `server/src/__tests__/execution-workspace-policy.test.ts`: admission
keeps a fresh isolated continuation of a projectless task runnable.
- `server/src/modules/wake-queue/adapters/postgres.test.ts`
"restart-ended native run": saved input behind a run the fallback
replaced is promoted.
- `server/src/services/fresh-continuation-fallback.test.ts` "relies on
saved input only while it still holds an undelivered message" (taken:
true gives `recorded`, taken: false gives `saved_input`).
- `postgres.test.ts` "promotes saved input behind an exhausted restart
hold ... (released, continuation pending)": the follow-up stays deferred
while the isolated continuation is pending, then is promoted when the
continuation's run is released. It was red before the fix (`expected
'promoted' to be 'released'`).
- Local, round 2: `cd server && npx vitest run
src/modules/wake-queue/adapters/postgres.test.ts
src/services/fresh-continuation-fallback.test.ts`: 2 files, 177 tests
passed.

Local command, on this branch based on current master (includes #21222
and #21234):
- `cd server && npx vitest run
src/services/fresh-continuation-fallback.test.ts
src/__tests__/execution-workspace-policy.test.ts
src/modules/wake-queue/adapters/postgres.test.ts`: 3 files, 221 tests
passed (re-run after rebasing on #21234).

Local release-swap cells:
- L1 (6 tasks, 3 swaps): 5 fresh isolated continuations delivered,
including a second one on the same task; 0
`workspace_worktree_requires_project` rows (before: 329 to 428 skipped
wakes per wave).
- R2 (6 tasks, 3 swaps): a task interrupted three times had its second
and third continuations delivered. The same cell found gap 3 (a
follow-up already read by the cut-off run), which the last commit fixes.
- Hour-long release-swap wave (w-20261112T233630-s7201) on the branch
that holds these fixes and the related recovery fixes: 11/11 tasks
finished hands-off (master: 9/11). The lost-continuation and
follow-up-promotion fixes were visibly working in it.

Review-round tests: "moves the task off the old writer's workspace and
runs the isolated continuation before saved input" (changed expectation:
`recorded` instead of `saved_input`; red on the previous head);
"promotes saved input behind an exhausted restart hold once a %s
continuation replaced the run" (isolated: promoted, red before;
`writer_ended`: held). 3 files, 224 tests passed.

Conflict note: the last commit was resolved onto current master. It no
longer imports `taskHistoryReadsForRun`, which belongs to a separate
opening-note change.

## Risks

- An isolated projectless continuation now passes admission. It runs in
a new workspace that the run prepares, so it never shares a workspace
with the cut-off run.
- A superseded run's follow-up is promoted without its native
finalization committed. The superseded run is fenced (its writes are
refused), so there is still one writer.
- No migration. No schema change. The normal admission path adds no
query.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-12 21:10:47 -06:00
1 parent a298ca7fbc
commit 6cb0dd0c6b
7 files changed
+200 -16

No files matched your search

@@ -2,9 +2,10 @@ import { resolveExternalTestDatabaseUrl } from "../../../__tests__/helpers/exter
import { DELIVERY_QUEUES } from "../../../services/delivery-work-notifications.js";
import { instanceSettingsService } from "../../../services/instance-settings.js";
import { randomBytes, randomUUID } from "node:crypto";
import { and, eq } from "drizzle-orm";
import { and, eq, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import type { Db } from "@paperclipai/db";
import { createSystemRecovery } from "../../../services/system-recovery-state.js";
import {
activityLog,
agentWakeupRequests,
@@ -474,6 +475,85 @@ describeEmbeddedPostgres("wake-queue postgres adapter", () => {
expect((await release({ companyId, runId, now: new Date(), suppressImmediateRecovery: true, resumeStrandedQueue: true })).outcome.kind).toBe("promoted");
});
it.each([true, false])("promotes saved input behind a restart-ended native run only once the fresh continuation replaced it (replaced: %s)", async replaced => {
const companyId = await seedCompany(), agentId = await seedAgent({ companyId });
const issueId = await seedIssue({ companyId, status: "in_progress", assigneeAgentId: agentId });
const runId = await seedRun({ companyId, agentId, status: "failed", contextSnapshot: { issueId } });
await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: issueId, finishedAt: new Date(),
errorCode: "native_restart_recovery_blocked",
resultJson: replaced ? { freshContinuationSupersededAt: new Date().toISOString() } : {},
}).where(eq(heartbeatRuns.id, runId));
// Restart recovery ended the run: its finalization stays terminal_failure, never committed.
await db.insert(nativeRunFinalizations).values({ companyId, issueId, runId, phase: "terminal_failure",
recoveryState: "blocked", failureCode: "native_restart_recovery_blocked" });
const [comment] = await db.insert(issueComments).values({ companyId, issueId, authorType: "user", authorUserId: "responsible-user", body: "Follow-up during the run" }).returning();
const wakeId = await seedDeferredWake({ companyId, agentId, issueId, requestedByActorId: "responsible-user",
payload: { _paperclipWakeContext: { issueId, wakeReason: "issue_commented", wakeCommentIds: [comment.id] } },
});
const release = createReleaseIssueExecution({ issueLock: createPostgresWakeQueueAdapter(db, stubDeps),
recovery: { escalateStrandedAssignedIssue: async () => undefined, escalateStrandedRecoveryIssueInPlace: async () => undefined },
});
const result = await release({ companyId, runId, now: new Date(), suppressImmediateRecovery: true, resumeStrandedQueue: true });
if (replaced) {
expect(result.outcome.kind).toBe("promoted");
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wakeId)))[0].status).not.toBe("deferred_issue_execution");
// No reconciliation incident re-blocks the task the fallback released.
expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0].status).not.toBe("blocked");
} else {
expect(result.outcome.kind).toBe("released");
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wakeId)))[0].status).toBe("deferred_issue_execution");
}
});
it.each([
["fresh_isolated_workspace", "promoted", "not_requested"], ["remote_epoch", "promoted", "not_requested"],
["writer_ended", "released", "not_requested"],
// The isolated continuation is still to be delivered: it runs first and binds the new workspace;
// promoting the follow-up now would make it the task's next run, without the isolation.
["fresh_isolated_workspace", "released", "pending"],
] as const)("promotes saved input behind an exhausted restart hold once a %s continuation replaced the run (%s, continuation %s)", async (isolation, expected, continuationDelivery) => {
// Restart recovery exhausted its retries (systemRecovery.state exhausted, no stop receipt), then the
// fallback replaced the run. An isolated continuation reuses none of the old executor's workspace,
// so the manual-admission hold no longer stands between the task and the person's follow-up.
const companyId = await seedCompany(), agentId = await seedAgent({ companyId });
const issueId = await seedIssue({ companyId, status: "in_progress", assigneeAgentId: agentId });
const runId = await seedRun({ companyId, agentId, status: "failed", contextSnapshot: { issueId } });
await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: issueId, finishedAt: new Date(),
errorCode: "native_restart_recovery_blocked", resultJson: { freshContinuationSupersededAt: new Date().toISOString() },
}).where(eq(heartbeatRuns.id, runId));
await db.insert(nativeRunFinalizations).values({ companyId, issueId, runId, phase: "terminal_failure",
recoveryState: "blocked", failureCode: "native_restart_recovery_blocked",
failureDetail: { systemRecovery: { ...createSystemRecovery(runId), state: "exhausted" }, replacementDenied: "fresh_continuation" } });
await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: issueId, kind: "active_run_watchdog",
status: "resolved", outcome: "restored", ownerType: "system", returnOwnerAgentId: agentId,
cause: "fresh_continuation", fingerprint: `fresh-continuation:${runId}`, nextAction: "Continuing.", resolvedAt: new Date(),
evidence: { runId, freshContinuation: { sourceRunId: runId, isolation,
delivery: continuationDelivery === "pending" ? "recorded" : "saved_input" }, continuationDelivery } });
const [comment] = await db.insert(issueComments).values({ companyId, issueId, authorType: "user", authorUserId: "responsible-user", body: "Follow-up during the run" }).returning();
const wakeId = await seedDeferredWake({ companyId, agentId, issueId, requestedByActorId: "responsible-user",
payload: { _paperclipWakeContext: { issueId, wakeReason: "issue_commented", wakeCommentIds: [comment.id] } },
});
const release = createReleaseIssueExecution({ issueLock: createPostgresWakeQueueAdapter(db, stubDeps),
recovery: { escalateStrandedAssignedIssue: async () => undefined, escalateStrandedRecoveryIssueInPlace: async () => undefined },
});
const result = await release({ companyId, runId, now: new Date(), suppressImmediateRecovery: true, resumeStrandedQueue: true });
expect(result.outcome.kind).toBe(expected);
const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wakeId));
if (expected === "promoted") expect(wake.status).not.toBe("deferred_issue_execution");
else expect(wake.status).toBe("deferred_issue_execution");
if (continuationDelivery !== "pending") return;
// The isolated continuation is delivered and runs; its release promotes the follow-up behind it.
const continuationRunId = await seedRun({ companyId, agentId, status: "succeeded", contextSnapshot: { issueId } });
await db.update(heartbeatRuns).set({ finishedAt: new Date() }).where(eq(heartbeatRuns.id, continuationRunId));
await db.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence}
|| ${JSON.stringify({ continuationDelivery: "delivered", continuationRunId })}::jsonb` })
.where(eq(issueRecoveryActions.sourceIssueId, issueId));
expect((await release({ companyId, runId: continuationRunId, now: new Date(), suppressImmediateRecovery: true })).outcome.kind)
.toBe("promoted");
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wakeId)))[0].status)
.not.toBe("deferred_issue_execution");
});
it.each([
"completed", "multiple_comments", "repeated_reference", "parent_reference", "mixed_issue_references", "mixed_foreign_references", "mixed_unknown_references", "no_comments", "missing_comment",
"human_comment", "other_run_comment", "foreign_comment", "other_issue_comment", "deleted_comment",