mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
fix(recovery): continue twice-interrupted tasks and waiting follow-ups after a release swap (#21250)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run is cut off by a restart or a release swap, the fresh-continuation fallback continues the task in a new run. > - The continuation goes through normal admission, or through a person's saved follow-up when one is waiting. > - In release-swap waves, three gaps left tasks stopped with no run: admission refused an isolated continuation of a task with no project, a person's saved follow-up behind the replaced run was never promoted, and the fallback relied on a saved follow-up that the cut-off run had already read. > - This pull request closes those three gaps. > - The benefit is that a twice-interrupted task, and a task with a waiting follow-up, continue by themselves. ## Linked Issues or Issue Description Refs #21135, #21222 **What happened?** 1. A task with no project was interrupted twice. The second continuation used a fresh isolated workspace, which clears the task's workspace binding. The admission preflight still required that binding for an isolated task with no project, so it refused the continuation as `workspace_worktree_requires_project` about every 15 s. Each refusal re-blocked the task and added a comment (329 to 428 skipped wakes per wave). The runtime itself supports this case: it prepares a new workspace (`create_new`). 2. The fallback continued a task with delivery `saved_input` (a person's deferred follow-up is the next turn). The stranded-queue pass refused to promote that follow-up, because the replaced native run's finalization was `terminal_failure` (restart recovery ended it), not `committed`. The release path would also have re-blocked the task with a reconciliation incident. The follow-up never ran. 3. The fallback chose `saved_input` because a deferred user wake existed, but the cut-off run had already taken that message into its turn. Promotion settled the message as delivered, and no run followed. The task showed "working". **Expected behavior** An interrupted task continues by itself in each case, with no person action and no repeated comments. **Steps to reproduce** 1. Run a release-swap wave with projectless tasks in isolated mode and person follow-ups during runs. 2. Interrupt the same task twice, or interrupt a run that has a deferred follow-up behind it. 3. See the task stay blocked or "working" with no run. ## What Changed - Admission (`server/src/services/execution-workspace-policy.ts`, `server/src/services/heartbeat/queue.ts`, `server/src/services/fresh-continuation-fallback.ts`): `isUnrunnableWorktreeCombo` accepts `freshIsolatedContinuation`. The admission preflight sets it from the new `isFreshIsolatedContinuationWake`, which returns at once (no query) unless the wake context carries `freshContinuationIsolation`. This does not touch the dispatch ranking. - Follow-up promotion (`server/src/modules/wake-queue/adapters/postgres.ts`): the stranded-queue pass and the release path skip the native finalization gate and the reconciliation incident for a run that the fallback superseded (`agentRunSupersededByFreshContinuation`). - Saved-input choice (`server/src/services/fresh-continuation-fallback.ts`): a deferred user wake counts as saved input only when it has no comment ids, or still has a message that the agent has not received (`undeliveredLegacyUserCommentIds`, called, not changed). Otherwise the fallback records its own continuation. - Review round 1: (a) whenever the old writer is not proven stopped (`fresh_isolated_workspace`), the fallback records its own isolated continuation first, also when the task had a binding; saved input follows it on the new workspace. (b) Release skips the manual-admission hold and the pending-cleanup check for a run the fallback replaced with an isolated continuation (`fresh_isolated_workspace` or `remote_epoch`, via `freshContinuationIsolatesRun`); other replacements still wait. - Review round 2 (the isolated continuation must run before the waiting follow-up): while the fallback's `fresh_isolated_workspace` continuation is still pending delivery, release leaves the person's follow-up deferred. Otherwise the follow-up would become the task's next run without the isolation marker, and delivery would count it as the continuation and skip the isolated run. The continuation's own release promotes the follow-up behind it. `freshContinuationIsolatesRun` now also reports `continuationPending`. ## Verification Red-to-green tests (each failed before its fix and passes after it): - `server/src/__tests__/execution-workspace-policy.test.ts`: admission keeps a fresh isolated continuation of a projectless task runnable. - `server/src/modules/wake-queue/adapters/postgres.test.ts` "restart-ended native run": saved input behind a run the fallback replaced is promoted. - `server/src/services/fresh-continuation-fallback.test.ts` "relies on saved input only while it still holds an undelivered message" (taken: true gives `recorded`, taken: false gives `saved_input`). - `postgres.test.ts` "promotes saved input behind an exhausted restart hold ... (released, continuation pending)": the follow-up stays deferred while the isolated continuation is pending, then is promoted when the continuation's run is released. It was red before the fix (`expected 'promoted' to be 'released'`). - Local, round 2: `cd server && npx vitest run src/modules/wake-queue/adapters/postgres.test.ts src/services/fresh-continuation-fallback.test.ts`: 2 files, 177 tests passed. Local command, on this branch based on current master (includes #21222 and #21234): - `cd server && npx vitest run src/services/fresh-continuation-fallback.test.ts src/__tests__/execution-workspace-policy.test.ts src/modules/wake-queue/adapters/postgres.test.ts`: 3 files, 221 tests passed (re-run after rebasing on #21234). Local release-swap cells: - L1 (6 tasks, 3 swaps): 5 fresh isolated continuations delivered, including a second one on the same task; 0 `workspace_worktree_requires_project` rows (before: 329 to 428 skipped wakes per wave). - R2 (6 tasks, 3 swaps): a task interrupted three times had its second and third continuations delivered. The same cell found gap 3 (a follow-up already read by the cut-off run), which the last commit fixes. - Hour-long release-swap wave (w-20261112T233630-s7201) on the branch that holds these fixes and the related recovery fixes: 11/11 tasks finished hands-off (master: 9/11). The lost-continuation and follow-up-promotion fixes were visibly working in it. Review-round tests: "moves the task off the old writer's workspace and runs the isolated continuation before saved input" (changed expectation: `recorded` instead of `saved_input`; red on the previous head); "promotes saved input behind an exhausted restart hold once a %s continuation replaced the run" (isolated: promoted, red before; `writer_ended`: held). 3 files, 224 tests passed. Conflict note: the last commit was resolved onto current master. It no longer imports `taskHistoryReadsForRun`, which belongs to a separate opening-note change. ## Risks - An isolated projectless continuation now passes admission. It runs in a new workspace that the run prepares, so it never shares a workspace with the cut-off run. - A superseded run's follow-up is promoted without its native finalization committed. The superseded run is fenced (its writes are refused), so there is still one writer. - No migration. No schema change. The normal admission path adds no query. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (no documented behavior or command changes) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
a298ca7fbc
commit
6cb0dd0c6b
7 files changed
+200
-16
No files matched your search
@@ -154,6 +154,14 @@ describe("execution workspace policy helpers", () => {
|
||||
).toEqual({ type: "git_worktree", existingBranch: "PAP-14754-run-redaction" });
|
||||
});
|
||||
|
||||
it("admits a fresh isolated continuation of a projectless task whose binding the fallback cleared", () => {
|
||||
const issue = { projectId: null, projectWorkspaceId: null, executionWorkspaceId: null,
|
||||
executionWorkspacePreference: "reuse_existing" };
|
||||
expect(isUnrunnableWorktreeCombo({ issue, resolvedMode: "isolated_workspace", resolvedStrategy: "git_worktree" })).toBe(true);
|
||||
expect(isUnrunnableWorktreeCombo({ issue, resolvedMode: "isolated_workspace", resolvedStrategy: "git_worktree",
|
||||
freshIsolatedContinuation: true })).toBe(false);
|
||||
});
|
||||
|
||||
it("centralizes unrunnable isolated worktree detection", () => {
|
||||
expect(
|
||||
isUnrunnableWorktreeCombo({
|
||||
|
||||
@@ -2,9 +2,10 @@ import { resolveExternalTestDatabaseUrl } from "../../../__tests__/helpers/exter
|
||||
import { DELIVERY_QUEUES } from "../../../services/delivery-work-notifications.js";
|
||||
import { instanceSettingsService } from "../../../services/instance-settings.js";
|
||||
import { randomBytes, randomUUID } from "node:crypto";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { createSystemRecovery } from "../../../services/system-recovery-state.js";
|
||||
import {
|
||||
activityLog,
|
||||
agentWakeupRequests,
|
||||
@@ -474,6 +475,85 @@ describeEmbeddedPostgres("wake-queue postgres adapter", () => {
|
||||
expect((await release({ companyId, runId, now: new Date(), suppressImmediateRecovery: true, resumeStrandedQueue: true })).outcome.kind).toBe("promoted");
|
||||
});
|
||||
|
||||
it.each([true, false])("promotes saved input behind a restart-ended native run only once the fresh continuation replaced it (replaced: %s)", async replaced => {
|
||||
const companyId = await seedCompany(), agentId = await seedAgent({ companyId });
|
||||
const issueId = await seedIssue({ companyId, status: "in_progress", assigneeAgentId: agentId });
|
||||
const runId = await seedRun({ companyId, agentId, status: "failed", contextSnapshot: { issueId } });
|
||||
await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: issueId, finishedAt: new Date(),
|
||||
errorCode: "native_restart_recovery_blocked",
|
||||
resultJson: replaced ? { freshContinuationSupersededAt: new Date().toISOString() } : {},
|
||||
}).where(eq(heartbeatRuns.id, runId));
|
||||
// Restart recovery ended the run: its finalization stays terminal_failure, never committed.
|
||||
await db.insert(nativeRunFinalizations).values({ companyId, issueId, runId, phase: "terminal_failure",
|
||||
recoveryState: "blocked", failureCode: "native_restart_recovery_blocked" });
|
||||
const [comment] = await db.insert(issueComments).values({ companyId, issueId, authorType: "user", authorUserId: "responsible-user", body: "Follow-up during the run" }).returning();
|
||||
const wakeId = await seedDeferredWake({ companyId, agentId, issueId, requestedByActorId: "responsible-user",
|
||||
payload: { _paperclipWakeContext: { issueId, wakeReason: "issue_commented", wakeCommentIds: [comment.id] } },
|
||||
});
|
||||
const release = createReleaseIssueExecution({ issueLock: createPostgresWakeQueueAdapter(db, stubDeps),
|
||||
recovery: { escalateStrandedAssignedIssue: async () => undefined, escalateStrandedRecoveryIssueInPlace: async () => undefined },
|
||||
});
|
||||
const result = await release({ companyId, runId, now: new Date(), suppressImmediateRecovery: true, resumeStrandedQueue: true });
|
||||
if (replaced) {
|
||||
expect(result.outcome.kind).toBe("promoted");
|
||||
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wakeId)))[0].status).not.toBe("deferred_issue_execution");
|
||||
// No reconciliation incident re-blocks the task the fallback released.
|
||||
expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0].status).not.toBe("blocked");
|
||||
} else {
|
||||
expect(result.outcome.kind).toBe("released");
|
||||
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wakeId)))[0].status).toBe("deferred_issue_execution");
|
||||
}
|
||||
});
|
||||
|
||||
it.each([
|
||||
["fresh_isolated_workspace", "promoted", "not_requested"], ["remote_epoch", "promoted", "not_requested"],
|
||||
["writer_ended", "released", "not_requested"],
|
||||
// The isolated continuation is still to be delivered: it runs first and binds the new workspace;
|
||||
// promoting the follow-up now would make it the task's next run, without the isolation.
|
||||
["fresh_isolated_workspace", "released", "pending"],
|
||||
] as const)("promotes saved input behind an exhausted restart hold once a %s continuation replaced the run (%s, continuation %s)", async (isolation, expected, continuationDelivery) => {
|
||||
// Restart recovery exhausted its retries (systemRecovery.state exhausted, no stop receipt), then the
|
||||
// fallback replaced the run. An isolated continuation reuses none of the old executor's workspace,
|
||||
// so the manual-admission hold no longer stands between the task and the person's follow-up.
|
||||
const companyId = await seedCompany(), agentId = await seedAgent({ companyId });
|
||||
const issueId = await seedIssue({ companyId, status: "in_progress", assigneeAgentId: agentId });
|
||||
const runId = await seedRun({ companyId, agentId, status: "failed", contextSnapshot: { issueId } });
|
||||
await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: issueId, finishedAt: new Date(),
|
||||
errorCode: "native_restart_recovery_blocked", resultJson: { freshContinuationSupersededAt: new Date().toISOString() },
|
||||
}).where(eq(heartbeatRuns.id, runId));
|
||||
await db.insert(nativeRunFinalizations).values({ companyId, issueId, runId, phase: "terminal_failure",
|
||||
recoveryState: "blocked", failureCode: "native_restart_recovery_blocked",
|
||||
failureDetail: { systemRecovery: { ...createSystemRecovery(runId), state: "exhausted" }, replacementDenied: "fresh_continuation" } });
|
||||
await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: issueId, kind: "active_run_watchdog",
|
||||
status: "resolved", outcome: "restored", ownerType: "system", returnOwnerAgentId: agentId,
|
||||
cause: "fresh_continuation", fingerprint: `fresh-continuation:${runId}`, nextAction: "Continuing.", resolvedAt: new Date(),
|
||||
evidence: { runId, freshContinuation: { sourceRunId: runId, isolation,
|
||||
delivery: continuationDelivery === "pending" ? "recorded" : "saved_input" }, continuationDelivery } });
|
||||
const [comment] = await db.insert(issueComments).values({ companyId, issueId, authorType: "user", authorUserId: "responsible-user", body: "Follow-up during the run" }).returning();
|
||||
const wakeId = await seedDeferredWake({ companyId, agentId, issueId, requestedByActorId: "responsible-user",
|
||||
payload: { _paperclipWakeContext: { issueId, wakeReason: "issue_commented", wakeCommentIds: [comment.id] } },
|
||||
});
|
||||
const release = createReleaseIssueExecution({ issueLock: createPostgresWakeQueueAdapter(db, stubDeps),
|
||||
recovery: { escalateStrandedAssignedIssue: async () => undefined, escalateStrandedRecoveryIssueInPlace: async () => undefined },
|
||||
});
|
||||
const result = await release({ companyId, runId, now: new Date(), suppressImmediateRecovery: true, resumeStrandedQueue: true });
|
||||
expect(result.outcome.kind).toBe(expected);
|
||||
const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wakeId));
|
||||
if (expected === "promoted") expect(wake.status).not.toBe("deferred_issue_execution");
|
||||
else expect(wake.status).toBe("deferred_issue_execution");
|
||||
if (continuationDelivery !== "pending") return;
|
||||
// The isolated continuation is delivered and runs; its release promotes the follow-up behind it.
|
||||
const continuationRunId = await seedRun({ companyId, agentId, status: "succeeded", contextSnapshot: { issueId } });
|
||||
await db.update(heartbeatRuns).set({ finishedAt: new Date() }).where(eq(heartbeatRuns.id, continuationRunId));
|
||||
await db.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence}
|
||||
|| ${JSON.stringify({ continuationDelivery: "delivered", continuationRunId })}::jsonb` })
|
||||
.where(eq(issueRecoveryActions.sourceIssueId, issueId));
|
||||
expect((await release({ companyId, runId: continuationRunId, now: new Date(), suppressImmediateRecovery: true })).outcome.kind)
|
||||
.toBe("promoted");
|
||||
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wakeId)))[0].status)
|
||||
.not.toBe("deferred_issue_execution");
|
||||
});
|
||||
|
||||
it.each([
|
||||
"completed", "multiple_comments", "repeated_reference", "parent_reference", "mixed_issue_references", "mixed_foreign_references", "mixed_unknown_references", "no_comments", "missing_comment",
|
||||
"human_comment", "other_run_comment", "foreign_comment", "other_issue_comment", "deleted_comment",
|
||||
|
||||
@@ -18,6 +18,8 @@ import { nativeManualAdmissionHoldsExecution } from "../../../services/native-ru
|
||||
import { and, asc, eq, inArray, isNull, notInArray, or, sql } from "drizzle-orm";
|
||||
import { readReleaseRun, type ReleaseRun } from "./release-run-read.js";
|
||||
import { isGovernancePauseStop } from "../../../services/governance-pause-stop.js";
|
||||
import { agentRunSupersededByFreshContinuation } from "../../../agent-run-cancellation.js";
|
||||
import { freshContinuationIsolatesRun } from "../../../services/fresh-continuation-fallback.js";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { deliveredQueuedCommentIds, EXTERNAL_CHAT_MESSAGE_WAKE_MUTATION, EXTERNAL_CHAT_MESSAGE_WAKE_REASON } from "../../../services/queued-comment-delivery.js";
|
||||
import { extractIssueReferenceIdentifiers, isUuidLike } from "@paperclipai/shared";
|
||||
@@ -1284,7 +1286,10 @@ export function createPostgresWakeQueueAdapter(db: Db, deps: WakeQueuePostgresAd
|
||||
`,
|
||||
);
|
||||
|
||||
if (input.resumeStrandedQueue && run.runtimeMode === "native") {
|
||||
// A run the fresh-continuation fallback replaced is settled for good: its
|
||||
// finalization may stay terminal_failure (restart recovery ended it), and
|
||||
// the saved input queued behind it is now the task's next turn.
|
||||
if (input.resumeStrandedQueue && run.runtimeMode === "native" && !agentRunSupersededByFreshContinuation(run)) {
|
||||
const [finalized] = contextIssueId && run.nativeIssueId === contextIssueId
|
||||
? await tx.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations).where(and(
|
||||
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.runId, run.id),
|
||||
@@ -1293,10 +1298,16 @@ export function createPostgresWakeQueueAdapter(db: Db, deps: WakeQueuePostgresAd
|
||||
: [];
|
||||
if (!finalized) return { outcome: { kind: "released" }, postCommitEffects: [], run: runSnapshot };
|
||||
}
|
||||
if (await nativeManualAdmissionHoldsExecution(tx, { companyId: input.companyId, runId: run.id })) {
|
||||
// These two holds protect the old executor's workspace and resources. A
|
||||
// continuation in a new isolated workspace (or an abandoned remote epoch)
|
||||
// reuses none of them, so they no longer hold the task's next turn; any
|
||||
// other replacement still waits for them.
|
||||
const isolatedReplacement = agentRunSupersededByFreshContinuation(run)
|
||||
? await freshContinuationIsolatesRun(tx, { companyId: input.companyId, runId: run.id }) : false;
|
||||
if (!isolatedReplacement && await nativeManualAdmissionHoldsExecution(tx, { companyId: input.companyId, runId: run.id })) {
|
||||
return { outcome: { kind: "released" }, postCommitEffects: [], run: runSnapshot };
|
||||
}
|
||||
if (input.resumeStrandedQueue && await hasPendingRunCleanup(tx, input.companyId, run.id)) {
|
||||
if (input.resumeStrandedQueue && !isolatedReplacement && await hasPendingRunCleanup(tx, input.companyId, run.id)) {
|
||||
return { outcome: { kind: "released" }, postCommitEffects: [], run: runSnapshot };
|
||||
}
|
||||
|
||||
@@ -1448,13 +1459,25 @@ export function createPostgresWakeQueueAdapter(db: Db, deps: WakeQueuePostgresAd
|
||||
&& issueRow.status === "blocked" && issueRow.assigneeAgentId === run.agentId) {
|
||||
return { outcome: { kind: "released" }, postCommitEffects: [], run: runSnapshot };
|
||||
}
|
||||
if (!handoffAssigneeAgentId && await recordNativeTerminalRecoveryIfNeeded(tx, run, issueRow, input.now)) {
|
||||
// The fallback already decided this run's continuation; a reconciliation
|
||||
// incident would re-block the task it just released.
|
||||
if (!handoffAssigneeAgentId && !agentRunSupersededByFreshContinuation(run)
|
||||
&& await recordNativeTerminalRecoveryIfNeeded(tx, run, issueRow, input.now)) {
|
||||
return { outcome: { kind: "released" }, postCommitEffects: [], run: runSnapshot };
|
||||
}
|
||||
|
||||
// A release must leave deferred messages intact while execution is held.
|
||||
// The finishing conversation may still own its lease until finally cleanup.
|
||||
// Allow only bounded retry planning in that case; admission stays gated.
|
||||
// The fallback's isolated continuation runs before any saved input: it
|
||||
// alone carries the marker that prepares the new workspace and binds it.
|
||||
// Promoting the follow-up now would make it the task's next run without
|
||||
// that isolation (and the delivery worker would then skip the
|
||||
// continuation). The follow-up stays deferred and is promoted behind
|
||||
// the continuation when that run is released.
|
||||
if (isolatedReplacement && isolatedReplacement.continuationPending) {
|
||||
return { outcome: { kind: "released" }, postCommitEffects: [], run: runSnapshot };
|
||||
}
|
||||
const executionBlocker = await getExecutionBlocker(tx, issueRow.companyId, issueRow.id);
|
||||
const recoveryOnly = Boolean(executionBlocker &&
|
||||
executionBlocker.cause === "execution_owner_active" && executionBlocker.runId === run.id &&
|
||||
|
||||
@@ -94,7 +94,11 @@ export function isUnrunnableWorktreeCombo(input: {
|
||||
resolvedStrategy: string | null | undefined;
|
||||
reusableExecutionWorkspaceAvailable?: boolean | null;
|
||||
hasResolvablePriorSessionWorkspace?: boolean | null;
|
||||
/** The wake continues a cut-off run in a new isolated workspace that the run
|
||||
* prepares itself (`create_new`); it never needs the task's old binding. */
|
||||
freshIsolatedContinuation?: boolean | null;
|
||||
}): boolean {
|
||||
if (input.freshIsolatedContinuation === true) return false;
|
||||
if (input.resolvedMode !== "isolated_workspace" && input.resolvedMode !== "operator_branch") return false;
|
||||
if (input.resolvedStrategy !== "git_worktree") return false;
|
||||
if (input.issue.projectId || input.issue.projectWorkspaceId) return false;
|
||||
|
||||
@@ -3,7 +3,7 @@ import { and, eq, sql } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import {
|
||||
activityLog, agentWakeupRequests, agents, approvals, authUsers, companies, createDb, environmentLeases, environments, executionWorkspaces, heartbeatRuns, issueApprovals, issueRecoveryActions, issueThreadInteractions,
|
||||
issues, nativeRunFinalizations, toolInvocations,
|
||||
issueComments, issues, nativeRunFinalizations, toolInvocations,
|
||||
} from "@paperclipai/db";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
|
||||
import { deliverReconciledExecutions, settleUnrecoverableExecutions } from "./execution-recovery-resolution.js";
|
||||
@@ -803,6 +803,22 @@ suite("fresh-continuation fallback", () => {
|
||||
expect(decision.outcomeEvidence).not.toMatch(adminText);
|
||||
}, 60_000);
|
||||
|
||||
it.each([true, false])("relies on saved input only while it still holds an undelivered message (already taken by the cut-off run: %s)", async taken => {
|
||||
// R2 cell t003: the cut-off run had taken the follow-up into its turn; the fallback chose saved_input,
|
||||
// promotion settled the message as delivered, and no run followed.
|
||||
const f = await unfinishedRun({ runtime: "native", errorCode: "native_restart_recovery_blocked" });
|
||||
const [comment] = await db.insert(issueComments).values({ companyId, issueId: f.issueId, authorType: "user",
|
||||
authorUserId: userId, body: "Additional requirement: include FOLLOWUP-1" } as typeof issueComments.$inferInsert).returning();
|
||||
if (taken) await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, wakeCommentIds: [comment!.id] } })
|
||||
.where(eq(heartbeatRuns.id, f.runId));
|
||||
await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "automation", triggerDetail: "system",
|
||||
reason: "issue_execution_deferred", status: "deferred_issue_execution", requestedByActorType: "user", requestedByActorId: userId,
|
||||
payload: { issueId: f.issueId, _paperclipWakeContext: { issueId: f.issueId, wakeReason: "issue_commented", wakeCommentIds: [comment!.id] } },
|
||||
} as typeof agentWakeupRequests.$inferInsert);
|
||||
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
|
||||
.toMatchObject({ kind: "continued", delivery: taken ? "recorded" : "saved_input" });
|
||||
}, 60_000);
|
||||
|
||||
it.each([
|
||||
{ title: "a pending approval", setup: "approval" as const, errorCode: "native_continuation_requires_reconciliation" },
|
||||
{ title: "a credential the person has not shared", setup: "none" as const, errorCode: "configuration_incomplete" },
|
||||
@@ -881,18 +897,20 @@ suite("fresh-continuation fallback", () => {
|
||||
}, 60_000);
|
||||
|
||||
// The isolation decision is a property of the task's next run, whatever path starts it.
|
||||
it("moves the task off the old writer's workspace for every next run, saved input included", async () => {
|
||||
it("moves the task off the old writer's workspace and runs the isolated continuation before saved input", async () => {
|
||||
const f = await unfinishedRun({ runtime: "legacy", errorCode: "controller_lease_lost", error: "Legacy controller lease lost" });
|
||||
const [workspace] = await db.insert(executionWorkspaces).values({ companyId, sourceIssueId: f.issueId, mode: "isolated",
|
||||
strategyType: "git_worktree", name: "Task workspace", cwd: "/tmp/old-writer" }).returning();
|
||||
await db.update(issues).set({ executionWorkspaceId: workspace!.id, workspaceBindingRevision: 2 }).where(eq(issues.id, f.issueId));
|
||||
await db.update(heartbeatRuns).set({ processPid: 4242, processHostHash: "replaced-server", processStartedAt: new Date(Date.now() - 600_000),
|
||||
contextSnapshot: { issueId: f.issueId, executionWorkspaceId: workspace!.id } }).where(eq(heartbeatRuns.id, f.runId));
|
||||
// A person's saved message is waiting: it continues the task itself.
|
||||
// A person's saved message is waiting. It carries no isolation marker, so it cannot prepare the
|
||||
// new workspace (a projectless task would be refused at admission once the binding is cleared):
|
||||
// the isolated continuation runs first and the saved message follows it.
|
||||
await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "on_demand", status: "deferred_issue_execution",
|
||||
requestedByActorType: "user", requestedByActorId: userId, payload: { issueId: f.issueId } });
|
||||
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
|
||||
.toMatchObject({ kind: "continued", isolation: "fresh_isolated_workspace", delivery: "saved_input" });
|
||||
.toMatchObject({ kind: "continued", isolation: "fresh_isolated_workspace", delivery: "recorded" });
|
||||
// No delivery path can reach the old directory: the task is no longer bound to it.
|
||||
expect(await issue(f.issueId)).toMatchObject({ executionWorkspaceId: null, workspaceBindingRevision: 3 });
|
||||
const [kept] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace!.id));
|
||||
|
||||
@@ -8,6 +8,8 @@ import {
|
||||
} from "@paperclipai/shared";
|
||||
import { logger } from "../middleware/logger.js";
|
||||
import { logActivity } from "./activity-log.js";
|
||||
import { undeliveredLegacyUserCommentIds } from "./explicit-native-continuation.js";
|
||||
import { queuedCommentIdsFromWakePayload } from "./issue-queued-comment-queue.js";
|
||||
import { FRESH_CONTINUATION_SUPERSEDED_KEY } from "../agent-run-cancellation.js";
|
||||
import { isAcknowledgedNativeReassignmentStop, isAcknowledgedNativeStop } from "./acknowledged-native-stop.js";
|
||||
import { adapterExecutionControls } from "./adapter-execution-control.js";
|
||||
@@ -305,6 +307,40 @@ export async function readFreshContinuationReadiness(db: Db, run: Pick<typeof he
|
||||
return { mode: "fresh_isolated_workspace" as const, retryIntentId: recordId, sourceRunId, retainedSourceIds: [] as string[] };
|
||||
}
|
||||
|
||||
/**
|
||||
* True when a wake continues a cut-off run in a new isolated workspace. The
|
||||
* fallback cleared the task's workspace binding on purpose, and the run
|
||||
* prepares a new workspace itself, so admission must not refuse the wake for
|
||||
* lacking that binding (it would refuse it forever).
|
||||
*/
|
||||
export async function isFreshIsolatedContinuationWake(db: Db, input: { companyId: string; issueId: string;
|
||||
contextSnapshot: Record<string, unknown> | null | undefined }) {
|
||||
if (typeof object(input.contextSnapshot).freshContinuationIsolation !== "string") return false;
|
||||
return (await readFreshContinuationReadiness(db, { id: "", companyId: input.companyId,
|
||||
contextSnapshot: input.contextSnapshot ?? {} }, input.issueId)) !== null;
|
||||
}
|
||||
|
||||
/**
|
||||
* True when the fallback replaced this run and its continuation reuses none of
|
||||
* the run's workspace: a new isolated workspace or an abandoned remote epoch.
|
||||
* Holds that only protect the old executor's resources then no longer stand
|
||||
* between the task and its next turn.
|
||||
*/
|
||||
export async function freshContinuationIsolatesRun(db: Db, input: { companyId: string; runId: string }): Promise<false | {
|
||||
/** The isolated continuation is recorded but not yet delivered as a run. */
|
||||
continuationPending: boolean;
|
||||
}> {
|
||||
const [record] = await db.select({ isolation: sql<string>`${issueRecoveryActions.evidence}#>>'{freshContinuation,isolation}'`,
|
||||
delivery: sql<string | null>`${issueRecoveryActions.evidence}->>'continuationDelivery'` }).from(issueRecoveryActions).where(and(
|
||||
eq(issueRecoveryActions.companyId, input.companyId),
|
||||
sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,sourceRunId}' = ${input.runId}`,
|
||||
sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,isolation}' in ('fresh_isolated_workspace', 'remote_epoch')`)).limit(1);
|
||||
if (!record) return false;
|
||||
// Only the recorded continuation carries the isolation marker that prepares
|
||||
// the new workspace, so while it is pending it must be the task's next run.
|
||||
return { continuationPending: record.isolation === "fresh_isolated_workspace" && record.delivery === "pending" };
|
||||
}
|
||||
|
||||
export async function ensureFreshContinuation(db: Db, input: Input): Promise<FreshContinuationResult> {
|
||||
return db.transaction(tx => continueFreshInTransaction(tx as unknown as Db, input));
|
||||
}
|
||||
@@ -474,9 +510,20 @@ export async function continueFreshInTransaction(tx: Db, input: Input): Promise<
|
||||
// Saved user input (a message or a Retry the person pressed) continues the
|
||||
// task through normal admission once nothing blocks it; a second, automatic
|
||||
// continuation would run after it.
|
||||
const [savedInput] = await tx.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
|
||||
const savedWakes = await tx.select({ id: agentWakeupRequests.id, payload: agentWakeupRequests.payload }).from(agentWakeupRequests).where(and(
|
||||
eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
||||
eq(agentWakeupRequests.requestedByActorType, "user"), sql`${agentWakeupRequests.payload}->>'issueId' = ${issue.id}`)).limit(1);
|
||||
eq(agentWakeupRequests.requestedByActorType, "user"), sql`${agentWakeupRequests.payload}->>'issueId' = ${issue.id}`)).limit(5);
|
||||
// Saved input continues the task only while it still holds something the
|
||||
// agent has not received: a message the cut-off run already took into its
|
||||
// turn is settled as delivered when promoted, and then nothing would run.
|
||||
let savedInput: { id: string } | undefined;
|
||||
for (const wake of savedWakes) {
|
||||
const commentIds = queuedCommentIdsFromWakePayload(wake.payload);
|
||||
if (!commentIds.length || (await undeliveredLegacyUserCommentIds(tx, companyId, issue.id, run.agentId, commentIds)).length) {
|
||||
savedInput = wake;
|
||||
break;
|
||||
}
|
||||
}
|
||||
let delivery: "recorded" | "caller_owns" | "saved_input" = input.delivery === "caller_owns" ? "caller_owns"
|
||||
: savedInput ? "saved_input" : "recorded";
|
||||
|
||||
@@ -492,10 +539,12 @@ export async function continueFreshInTransaction(tx: Db, input: Input): Promise<
|
||||
// Never claim a stop that was not proven: record what isolates the successor.
|
||||
const isolation: FreshContinuationIsolation = fenced || remoteSource ? "remote_epoch"
|
||||
: writer === "ended" && !waitBounded ? "writer_ended" : "fresh_isolated_workspace";
|
||||
// With no task workspace binding to move, only a run that carries the
|
||||
// isolation in its context can avoid the shared directory: the recorded
|
||||
// continuation runs first, and saved input follows it.
|
||||
if (isolation === "fresh_isolated_workspace" && !issue.executionWorkspaceId && delivery === "saved_input") delivery = "recorded";
|
||||
// Only a run that carries the isolation in its context prepares the new
|
||||
// workspace (create_new). Saved input carries no such marker: on a task with
|
||||
// no project it would be refused at admission once the binding is cleared,
|
||||
// and otherwise could resolve the old directory. So the recorded
|
||||
// continuation runs first, binds the new workspace, and saved input follows.
|
||||
if (isolation === "fresh_isolated_workspace" && delivery === "saved_input") delivery = "recorded";
|
||||
const unverified = await unverifiedActions(tx, run);
|
||||
const note = freshContinuationNote({ runId: run.id, unverified,
|
||||
workspace: isolation === "writer_ended" ? "kept" : "restored_last_save" });
|
||||
|
||||
@@ -3,7 +3,7 @@ import { connectionIntentService } from "../connection-intents.js";
|
||||
import { verifyBlockerStallWake } from "../recovery/blocker-stall-wakeups.js";
|
||||
import { CONTEXT_COMPACTION_WAKE_REASON, ContextCompactionError, isContextCompactionRun, readContextCompactionRequest } from "../context-compaction.js";
|
||||
import { isProcessRunAdmissionClosed } from "./run-admission.js";
|
||||
import { continueFreshInTransaction, hasPendingFreshContinuation, settleCoalescedFreshContinuationHolds } from "../fresh-continuation-fallback.js";
|
||||
import { continueFreshInTransaction, hasPendingFreshContinuation, isFreshIsolatedContinuationWake, settleCoalescedFreshContinuationHolds } from "../fresh-continuation-fallback.js";
|
||||
import { refreshRunStartYieldIfStale } from "../run-start-handover.js";
|
||||
import { deliveredQueuedCommentIds } from "../queued-comment-delivery.js";
|
||||
import { hasAutomaticInteractionDeliveryHold, INTERACTION_DELIVERY_DENIED, recheckQueuedRunInteractions, withInteractionDeliveryOrigin } from "../agent-interaction-delivery.js";
|
||||
@@ -4278,6 +4278,8 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
|
||||
reusableExecutionWorkspaceAvailable:
|
||||
reuseRequest.existingExecutionWorkspaceAvailable,
|
||||
hasResolvablePriorSessionWorkspace,
|
||||
freshIsolatedContinuation: await isFreshIsolatedContinuationWake(tx as unknown as Db, {
|
||||
companyId: issue.companyId, issueId: issue.id, contextSnapshot: enrichedContextSnapshot }),
|
||||
})
|
||||
) {
|
||||
if (caseRequest) waitForCaseAdmission(WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE, WORKSPACE_WORKTREE_REQUIRES_PROJECT_MESSAGE);
|
||||
|
||||
Reference in new issue
Block a user