fix: settle interrupted preparation with a retained cancellation fence (#15022)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A cancelled preparation must preserve history while allowing a new
user turn.
> - Older preparation can retain its cancellation fence but omit the
unwind marker.
> - The controller from that older boot is gone, its lease expired, and
no provider started.
> - Use the same narrow preparation proof for this retained-fence state.
> - The benefit is working Retry and saved-message recovery after
interrupted startup.

## Linked Issues or Issue Description

Refs #15020 and #15015. Related #13293 addresses post-launch recovery.

**What happened?**

A preparation interrupted before native selection retained
`startupCancellation.beforeNativeSelection: true` but no
preparation-settled marker. Its owner expired across restart and it had
no environment leases. The missing-receipt compatibility rule did not
recognize the retained fence, leaving Retry absent and saved messages
deferred.

**Expected behavior**

Admit one new user turn when the preparation evidence agrees, the old
controller expired, and cleanup is complete. Preserve previous results
and do not replay the cancelled input.

**Steps to reproduce**

1. Cancel Paperclip Runner preparation before runtime selection.
2. Retain the cancellation fence without an unwind marker or environment
leases.
3. Restart after its old controller lease expires.
4. Send a new message, select Retry, or allow the saved-message worker
to reconsider new input.
5. Confirm one successor receives only undelivered input. Repeat with
live ownership, invocation evidence, or pending cleanup and confirm
execution stays held.

**Paperclip version or commit**

Reproduced on `94f6f3eb4`.

**Deployment mode**

Authenticated private self-hosted server, built from source.

## What Changed

- Accept a retained before-selection cancellation fence in the expired
historical preparation proof.
- Retain runtime, stage, adapter, ownership expiry, process, event, and
cleanup requirements.
- Extend Retry, fresh-message, partial-queue, concurrent-worker, and
contradictory-evidence tests to both receipt states.
- Document recovery when the preparation-settled marker was not
retained.

## Verification

- Red: five exact-state regressions fail on the parent commit.
- Green: all 203 continuation tests pass, including the five new
regressions and 13 additional negative evidence cases. Process recovery
and queued-comment routes add 413 passing tests.
- A read-only candidate service check against the retained live run
returns Retry eligibility without changing any task state.
- An unrelated containment assertion failed once in CI. All 85 tests in
that route suite pass locally and the CI shard passes on rerun.
- Workspace `pnpm -r typecheck` and `pnpm build` pass.
- Final head `48240de3d`: all 54 CI checks pass, two optional Storybook
checks skip, and Greptile is 5/5 with zero unresolved threads. The
complete local monolithic suite is covered by sharded CI; the earlier
local run was stopped after a workspace case failed, and that case
passed in isolation.
- After merge, deploy the exact merged commit and verify an actual agent
reply through the task composer.

## Risks

A cancellation receipt alone must not certify that provider execution
stopped. This path still requires unresolved preparation, no native
identity or coordinator, an expired controller from another boot, no
launch or provider evidence, and completed environment cleanup.
Current-boot preparation remains held until it settles. No schema or
dependency change.

## Model Used

OpenAI Codex, an agent based on GPT-6. The exact runtime model ID and
context window are not exposed in this session. Used reasoning,
repository tools, code execution, and browser inspection.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-10-03 00:04:21 -05:00
1 parent 94f6f3eb47
commit 215586d127
3 files changed
+24 -16

No files matched your search

+4 -2
View File
@@ -1156,10 +1156,12 @@ controller, lease, or result). It also checks for contradictory launch/process
evidence and verifies local cleanup or exact remote termination receipts. The
preparer must have finished or its startup lease must have expired. A missing
PID alone does not establish this proof.
For older interrupted preparation rows without a cancellation receipt, the
For older interrupted preparation rows without a cancellation receipt, or with
a retained startup fence but no preparation-settled marker, the
immutable Paperclip Runner adapter claim, unresolved runtime, and preparing stage
must agree. The old controller must belong to another server boot and its lease
must have expired. No native identity, coordinator, result, adapter invocation,
must have expired. Only absent results or the retained before-selection
cancellation fence qualify. No native identity, coordinator, adapter invocation,
provider event, or process-launch evidence may exist. Environment cleanup still
requires the same receipts. This historical proof permits explicit Retry or a
newer saved user message; it does not replay the cancelled input.
@@ -28,14 +28,15 @@ export async function isCancelledNativeStartup(db: Db, run: Run, coordinator: Co
if (run.status !== "cancelled" || !run.finishedAt || run.processPid || run.processGroupId ||
run.processStartedAt || run.sessionIdAfter) return false;
const cancellation = run.resultJson?.startupCancellation as Record<string, unknown> | undefined;
// Older builds did not retain the cancellation fence. Their immutable
// Older builds could omit the cancellation fence or its unwind marker. Their immutable
// native-adapter claim and unresolved preparation stage still prove that
// provider dispatch did not begin. Require an expired owner from another
// server boot; neither a missing PID nor mutable agent settings is proof.
const historicalBeforeSelection = run.runtimeMode === "legacy" && !run.runtimeModeResolvedAt &&
run.executionStage === "preparing" && !run.nativeIssueId && !run.nativeSessionId && !coordinator &&
claimedAdapterType(run) === "paperclip_runner" && run.errorCode === "operator_interrupted" &&
run.resultJson === null && Boolean(run.controllerBootId && run.controllerBootId !== legacyControllerBootId &&
(run.resultJson === null || cancellation?.beforeNativeSelection === true) &&
Boolean(run.controllerBootId && run.controllerBootId !== legacyControllerBootId &&
run.controllerLeaseExpiresAt && run.controllerLeaseExpiresAt <= new Date());
const beforeSelection = historicalBeforeSelection || run.runtimeMode === "legacy" && !run.runtimeModeResolvedAt &&
!run.nativeSessionId && !coordinator && claimedAdapterType(run) === "paperclip_runner" &&
@@ -598,21 +598,24 @@ const support = await getEmbeddedPostgresTestSupport();
expect(await admit(f)).toMatchObject({ previousRunId: f.sourceRunId });
});
async function seedHistoricalCancelledPreparation() {
async function seedHistoricalCancelledPreparation(retainedReceipt = false) {
const f = await seedCancelledStartup();
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
await db.delete(environmentLeases).where(eq(environmentLeases.heartbeatRunId, f.sourceRunId));
await db.update(heartbeatRuns).set({ runtimeMode: "legacy", runtimeModeResolvedAt: null,
nativeIssueId: null, nativeSessionId: null, executionStage: "preparing", errorCode: "operator_interrupted",
runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } }, resultJson: null,
runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } },
resultJson: retainedReceipt ? { operatorInterrupted: true, stopReason: "cancelled",
startupCancellation: { beforeNativeSelection: true, requestedAt: "2026-09-11T10:00:00Z" } } : null,
}).where(eq(heartbeatRuns.id, f.sourceRunId));
await db.update(issueRecoveryActions).set({ cause: "legacy_execution_requires_reconciliation" })
.where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
return f;
}
it.each(["message", "retry"])("recovers historical native preparation through an explicit %s", async kind => {
const f = await seedHistoricalCancelledPreparation();
it.each([false, true].flatMap(receipt => ["message", "retry"].map(kind => ({ receipt, kind }))))(
"recovers historical native preparation through an explicit $kind (receipt: $receipt)", async ({ receipt, kind }) => {
const f = await seedHistoricalCancelledPreparation(receipt);
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ canRetry: true });
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
const successor = await heartbeatService(db).wakeup(f.agentId, {
@@ -625,11 +628,13 @@ const support = await getEmbeddedPostgresTestSupport();
previousRunId: f.sourceRunId, explicitUserContinuation: { commentId: kind === "message" ? f.commentId : null } } });
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId));
expect(source).toMatchObject({ status: "cancelled", runtimeMode: "legacy", resultJson: null });
expect(source).toMatchObject({ status: "cancelled", runtimeMode: "legacy",
resultJson: receipt ? { startupCancellation: { beforeNativeSelection: true } } : null });
});
it.each(["fresh", "earlier_delivered", "last_delivered"])("resumes saved input after historical native preparation expires across restart exactly once (%s)", async kind => {
const f = await seedHistoricalCancelledPreparation();
it.each([false, true].flatMap(receipt => ["fresh", "earlier_delivered", "last_delivered"].map(kind => ({ receipt, kind }))))(
"resumes saved input after historical native preparation expires across restart exactly once ($kind, receipt: $receipt)", async ({ receipt, kind }) => {
const f = await seedHistoricalCancelledPreparation(receipt);
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() + 60_000) })
.where(eq(heartbeatRuns.id, f.sourceRunId));
@@ -658,14 +663,14 @@ const support = await getEmbeddedPostgresTestSupport();
expect(successors).toHaveLength(1);
expect(successors[0].contextSnapshot).toMatchObject({ wakeCommentIds: [f.commentId],
explicitUserContinuation: { commentId: f.commentId }, forceFreshSession: true });
const [receipt] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, waiting.id));
expect(receipt).toMatchObject({ status: "coalesced", runId: successors[0].id });
const [queueReceipt] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, waiting.id));
expect(queueReceipt).toMatchObject({ status: "coalesced", runId: successors[0].id });
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
});
it.each(["resolved", "dispatching", "adapter", "missing_boot", "current_boot", "missing_lease", "live_lease", "receipt", "launch", "invoked", "coordinator", "cleanup", "remote"])(
"holds historical native preparation with contradictory or incomplete %s evidence", async kind => {
const f = await seedHistoricalCancelledPreparation();
it.each([false, true].flatMap(receipt => ["resolved", "dispatching", "adapter", "missing_boot", "current_boot", "missing_lease", "live_lease", "receipt", "launch", "invoked", "coordinator", "cleanup", "remote"].map(kind => ({ receipt, kind }))))(
"holds historical native preparation with contradictory or incomplete $kind evidence (receipt: $receipt)", async ({ receipt, kind }) => {
const f = await seedHistoricalCancelledPreparation(receipt);
const patch = kind === "resolved" ? { runtimeModeResolvedAt: new Date() }
: kind === "dispatching" ? { executionStage: "dispatching" }
: kind === "adapter" ? { runnerProfileJson: { adapterDispatch: { adapterType: "process" } } }