mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
fix(recovery): continue a pre-turn interrupt after recovery, not after the destroy (#19292)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A board user can interrupt a native run with a comment. The next turn should start soon after the run stops > - When the stop leaves sandbox files unsaved, recovery saves them, re-stops the sandbox, and then queues the destroy for an ephemeral lease > - #19224 kept the verified stop on that lease as `workspaceRecoveryCleanup.stopReceipt`, and taught the continuation gates to accept it > - One gate was missed: `acknowledgedNativeStopExecutionHasStopped`. An interrupt that lands before the provider turn opens leaves no gating hold, so the saved comment goes through this gate, not through the explicit continuation gate > - That gate still wanted the raw `remoteExecutionTermination` receipt, so the comment waited about 5 minutes for the destroy retry > - This pull request lets that gate accept the carried-forward stop receipt too > - The benefit is that an interrupt during startup continues right after recovery, as #19224 intended for every lease policy ## Linked Issues or Issue Description Refs #19224 Refs #19206 **What happened?** On hosted staging, an `interrupt: true` comment reached a native run of an ephemeral-lease agent while the run was still in startup (the provider pack upload). The run stopped before any provider turn. Recovery saved its files at once. The follow-up turn started about 6 minutes after the interrupt. The stopped run's lease was released by `pending_cleanup_retry`, and the follow-up started 2 seconds later. Agents whose interrupt landed after the provider turn opened continued in 30 to 50 seconds on the same build. **Expected behavior** The saved comment starts a new turn right after recovery saves the files and the sandbox is stopped. The destroy of a stopped sandbox is cleanup and must not hold the next turn. **Steps to reproduce** 1. Use an agent whose sandbox lease is ephemeral (destroy after each turn). 2. Post an `interrupt: true` comment while its native run is still starting, before the provider turn opens. 3. Let recovery save the files. See the follow-up start only after the destroy retry releases the lease. **Paperclip version or commit** Hosted staging at `e37ab23d` (includes #19206 and #19224). ## What Changed - `server/src/services/acknowledged-native-stop.ts`: `acknowledgedNativeStopExecutionHasStopped` accepts a remote lease that has `hasRemoteTerminationReceipt` or `hasRecoveredCleanupStopReceipt`. Before, it accepted only the first one. - `server/src/services/workspace-recovery.test.ts`: new regression for a pre-turn interrupt on an ephemeral lease. Recovery completes, the destroy is queued, and the saved comment is handed to `resumeRemoteStopComments` twice. With a verified carried stop, exactly one turn starts while the lease is still `pending_cleanup`. Without the stop receipt, the comment stays deferred and no turn starts. - `doc/execution-semantics.md`: one sentence that extends the #19224 paragraph to the pre-turn Stop case. ## Verification - Red before the fix: `cd server && npx vitest run src/services/workspace-recovery.test.ts -t "starts the interrupt redirect once after recovery"` failed the `verified-stop` case with `expected [] to have a length of 1 but got +0`. The `no-stop-receipt` case passed. - Green after the fix: the same command passes both cases. - `npx vitest run src/services/workspace-recovery.test.ts`: 382 passed. - Related suites pass: `acknowledged-native-stop`, `explicit-native-continuation`, `heartbeat/run-control`, `heartbeat/recovery`, `heartbeat/native-stop-evidence-read`, `native-session-executor`, `heartbeat-observed-workspace-preservation`, `native-startup-cleanup-recovery`, `execution-control-work`. Two `run-control` cases timed out at 30 s under parallel load and passed when run alone. - `pnpm --filter @paperclipai/server typecheck` passes. - Not done: a live re-run on staging. This change is not deployed. ## Risks - Low risk. `hasRecoveredCleanupStopReceipt` is the same check #19224 added to the other gates. It is true only for a lease whose recovery cleanup belongs to its own run, which was released, and whose carried stop receipt matches the company, run, lease, provider, and provider lease id. A lease without that receipt still blocks. - The no-replay hold is unchanged. The successor still goes through ordinary admission with all task gates. The test calls the resume twice and sees one turn only. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, browser reads of staging). Extended reasoning was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
b400f76647
commit
b05d3db2b3
3 files changed
+67
-4
No files matched your search
@@ -2221,7 +2221,10 @@ The lease carries the verified stop forward as
|
||||
`workspaceRecoveryCleanup.stopReceipt`. The saved message, Retry and the next
|
||||
remote workspace writer wait for that stop, not for the destroy. So they continue
|
||||
right after recovery, instead of after the cleanup retry. A lease without a
|
||||
matching stop receipt still blocks them.
|
||||
matching stop receipt still blocks them. The same holds when the Stop or
|
||||
interrupt landed before the provider turn opened: the saved message then goes
|
||||
through ordinary admission once the acknowledged Stop has a verified stop, and
|
||||
that check also accepts the carried-forward receipt.
|
||||
|
||||
The deadline is a backstop, not the normal failure path. When native task
|
||||
preparation fails after a fresh controller claim and before any provider turn,
|
||||
|
||||
@@ -3,7 +3,7 @@ import { environmentLeases, heartbeatRunEvents, heartbeatRuns, type Db } from "@
|
||||
import { parseObject } from "../adapters/utils.js";
|
||||
import { resolveProviderApiPlacement } from "./native-runtime/provider-api-placement.js";
|
||||
import { hasNativeLocalProcessStop } from "./native-local-process-stop.js";
|
||||
import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
|
||||
import { hasRecoveredCleanupStopReceipt, hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
|
||||
import { validatePrpEvent } from "../vendor/paperclip-runner/index.js";
|
||||
import { nativeSha256 } from "./native-runtime/canonical.js";
|
||||
import { isConfirmedCursorCloudTerminalObservation } from "./native-runtime/cursor-cloud-runtime.js";
|
||||
@@ -96,8 +96,10 @@ export async function acknowledgedNativeStopExecutionHasStopped(db: Db, run: Pic
|
||||
const leases = await db.select().from(environmentLeases).where(and(
|
||||
eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id),
|
||||
));
|
||||
// Provider PIDs are meaningful only on their own host.
|
||||
if (leases.some(lease => lease.provider !== "local")) return leases.every(hasRemoteTerminationReceipt);
|
||||
// Provider PIDs are meaningful only on their own host. A recovered sandbox
|
||||
// queued for destroy keeps its verified stop; the destroy is only cleanup.
|
||||
if (leases.some(lease => lease.provider !== "local"))
|
||||
return leases.every(lease => hasRemoteTerminationReceipt(lease) || hasRecoveredCleanupStopReceipt(lease));
|
||||
if (leases.some(lease => !lease.releasedAt || lease.cleanupStatus === "failed")) return false;
|
||||
if (!run.processPid && !run.processGroupId) return hasNativeLocalProcessStop(db, run.companyId, run.id);
|
||||
const absent = (pid: number) => {
|
||||
|
||||
@@ -3783,6 +3783,64 @@ describe("automatic retained workspace recovery", () => {
|
||||
}
|
||||
}, 30_000);
|
||||
|
||||
it.each(["verified-stop", "no-stop-receipt"] as const)
|
||||
("starts the interrupt redirect once after recovery when the interrupt landed before the provider turn, without waiting for the destroy (%s)", async kind => {
|
||||
// An interrupt comment while an ephemeral-lease run was still starting: the
|
||||
// startup owner fenced the handle before any provider turn, its files were
|
||||
// recovered, and the re-stopped sandbox was queued for destroy. The queued
|
||||
// follow-up was cancelled before start and its comment saved for recovery.
|
||||
const f = await seed("native", true, "destroy", false, true);
|
||||
const agentId = (await run(f.runId)).agentId;
|
||||
await stopBeforeSessionOpen(f, { errorCode: "operator_interrupted" });
|
||||
const stopped = await run(f.runId);
|
||||
await db.update(heartbeatRuns).set({ resultJson: { ...stopped.resultJson, operatorInterrupted: true,
|
||||
interruptionSource: "issue_comment_interrupt", interruptedIssueId: f.issueId, interruptedByActorType: "user",
|
||||
interruptedByActorId: "operator", cancelledByActorType: "user", cancelledByUserId: "operator" } })
|
||||
.where(eq(heartbeatRuns.id, f.runId));
|
||||
const [comment] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId,
|
||||
authorType: "user", authorUserId: "operator", body: "Use the other approach instead." }).returning();
|
||||
const context = { issueId: f.issueId, wakeReason: "issue_commented", wakeCommentId: comment.id, wakeCommentIds: [comment.id] };
|
||||
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId, runtimeMode: "legacy", status: "cancelled",
|
||||
invocationSource: "automation", errorCode: "execution_reconciliation_required", finishedAt: new Date(), contextSnapshot: context });
|
||||
const [saved] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId,
|
||||
source: "automation", reason: "issue_commented", status: "deferred_issue_execution",
|
||||
requestedByActorType: "user", requestedByActorId: "operator",
|
||||
payload: { issueId: f.issueId, commentId: comment.id, _paperclipWakeContext: context,
|
||||
executionWait: { reason: "execution_recovery", message: "Waiting for execution recovery. Your message is saved." } } }).returning();
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
expect(readSystemRecovery((await action(f.actionId)).evidence)).toMatchObject({ state: "complete" });
|
||||
await scheduleRecoveredWorkspaceCleanup(db, await run(f.runId));
|
||||
const [queued] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
|
||||
expect(queued).toMatchObject({ status: "pending_cleanup", metadata: { workspaceRecoveryCleanup: { runId: f.runId } } });
|
||||
if (kind === "no-stop-receipt") {
|
||||
const cleanup = queued.metadata!.workspaceRecoveryCleanup as Record<string, unknown>;
|
||||
await db.update(environmentLeases).set({ metadata: { ...queued.metadata,
|
||||
workspaceRecoveryCleanup: { ...cleanup, stopReceipt: undefined } } }).where(eq(environmentLeases.id, f.lease.id));
|
||||
}
|
||||
// No provider turn ran, so the no-replay hold does not gate the task.
|
||||
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
||||
// Recovery hands the saved comment to ordinary admission (onRecovered), twice
|
||||
// to prove a repeated sweep adds no second executor.
|
||||
for (let pass = 0; pass < 2; pass += 1) await heartbeatService(db).resumeRemoteStopComments(await run(f.runId));
|
||||
const turns = await pendingTurns(f);
|
||||
const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, saved.id));
|
||||
if (kind === "verified-stop") {
|
||||
// The destroy is still pending: the follow-up waits on the stop, not on it.
|
||||
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0])
|
||||
.toMatchObject({ status: "pending_cleanup" });
|
||||
expect(turns).toHaveLength(1);
|
||||
expect(turns[0]).toMatchObject({ agentId, contextSnapshot: { wakeCommentId: comment.id } });
|
||||
expect(wake.status).not.toBe("deferred_issue_execution");
|
||||
} else {
|
||||
// Without a verified stop, the saved comment keeps waiting.
|
||||
expect(turns).toEqual([]);
|
||||
expect(wake.status).toBe("deferred_issue_execution");
|
||||
}
|
||||
// The no-replay audit row is untouched.
|
||||
expect(await action(f.actionId)).toMatchObject({ status: "resolved", outcome: "blocked",
|
||||
evidence: { automaticRecovery: { replay: "blocked" } } });
|
||||
}, 30_000);
|
||||
|
||||
it("keeps a snapshot-only recovery held for a new assignee because the task workspace lacks the saved files", async () => {
|
||||
const f = await seed("native", false);
|
||||
await db.update(heartbeatRuns).set({ status: "cancelled", errorCode: "issue_reassigned" }).where(eq(heartbeatRuns.id, f.runId));
|
||||
|
||||
Reference in new issue
Block a user