mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 11:54:35 +01:00
fix: settle interrupted preparation with a retained cancellation fence (#15022)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A cancelled preparation must preserve history while allowing a new user turn. > - Older preparation can retain its cancellation fence but omit the unwind marker. > - The controller from that older boot is gone, its lease expired, and no provider started. > - Use the same narrow preparation proof for this retained-fence state. > - The benefit is working Retry and saved-message recovery after interrupted startup. ## Linked Issues or Issue Description Refs #15020 and #15015. Related #13293 addresses post-launch recovery. **What happened?** A preparation interrupted before native selection retained `startupCancellation.beforeNativeSelection: true` but no preparation-settled marker. Its owner expired across restart and it had no environment leases. The missing-receipt compatibility rule did not recognize the retained fence, leaving Retry absent and saved messages deferred. **Expected behavior** Admit one new user turn when the preparation evidence agrees, the old controller expired, and cleanup is complete. Preserve previous results and do not replay the cancelled input. **Steps to reproduce** 1. Cancel Paperclip Runner preparation before runtime selection. 2. Retain the cancellation fence without an unwind marker or environment leases. 3. Restart after its old controller lease expires. 4. Send a new message, select Retry, or allow the saved-message worker to reconsider new input. 5. Confirm one successor receives only undelivered input. Repeat with live ownership, invocation evidence, or pending cleanup and confirm execution stays held. **Paperclip version or commit** Reproduced on `94f6f3eb4`. **Deployment mode** Authenticated private self-hosted server, built from source. ## What Changed - Accept a retained before-selection cancellation fence in the expired historical preparation proof. - Retain runtime, stage, adapter, ownership expiry, process, event, and cleanup requirements. - Extend Retry, fresh-message, partial-queue, concurrent-worker, and contradictory-evidence tests to both receipt states. - Document recovery when the preparation-settled marker was not retained. ## Verification - Red: five exact-state regressions fail on the parent commit. - Green: all 203 continuation tests pass, including the five new regressions and 13 additional negative evidence cases. Process recovery and queued-comment routes add 413 passing tests. - A read-only candidate service check against the retained live run returns Retry eligibility without changing any task state. - An unrelated containment assertion failed once in CI. All 85 tests in that route suite pass locally and the CI shard passes on rerun. - Workspace `pnpm -r typecheck` and `pnpm build` pass. - Final head `48240de3d`: all 54 CI checks pass, two optional Storybook checks skip, and Greptile is 5/5 with zero unresolved threads. The complete local monolithic suite is covered by sharded CI; the earlier local run was stopped after a workspace case failed, and that case passed in isolation. - After merge, deploy the exact merged commit and verify an actual agent reply through the task composer. ## Risks A cancellation receipt alone must not certify that provider execution stopped. This path still requires unresolved preparation, no native identity or coordinator, an expired controller from another boot, no launch or provider evidence, and completed environment cleanup. Current-boot preparation remains held until it settles. No schema or dependency change. ## Model Used OpenAI Codex, an agent based on GPT-6. The exact runtime model ID and context window are not exposed in this session. Used reasoning, repository tools, code execution, and browser inspection. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
94f6f3eb47
commit
215586d127
3 files changed
+24
-16
No files matched your search
@@ -1156,10 +1156,12 @@ controller, lease, or result). It also checks for contradictory launch/process
|
||||
evidence and verifies local cleanup or exact remote termination receipts. The
|
||||
preparer must have finished or its startup lease must have expired. A missing
|
||||
PID alone does not establish this proof.
|
||||
For older interrupted preparation rows without a cancellation receipt, the
|
||||
For older interrupted preparation rows without a cancellation receipt, or with
|
||||
a retained startup fence but no preparation-settled marker, the
|
||||
immutable Paperclip Runner adapter claim, unresolved runtime, and preparing stage
|
||||
must agree. The old controller must belong to another server boot and its lease
|
||||
must have expired. No native identity, coordinator, result, adapter invocation,
|
||||
must have expired. Only absent results or the retained before-selection
|
||||
cancellation fence qualify. No native identity, coordinator, adapter invocation,
|
||||
provider event, or process-launch evidence may exist. Environment cleanup still
|
||||
requires the same receipts. This historical proof permits explicit Retry or a
|
||||
newer saved user message; it does not replay the cancelled input.
|
||||
|
||||
@@ -28,14 +28,15 @@ export async function isCancelledNativeStartup(db: Db, run: Run, coordinator: Co
|
||||
if (run.status !== "cancelled" || !run.finishedAt || run.processPid || run.processGroupId ||
|
||||
run.processStartedAt || run.sessionIdAfter) return false;
|
||||
const cancellation = run.resultJson?.startupCancellation as Record<string, unknown> | undefined;
|
||||
// Older builds did not retain the cancellation fence. Their immutable
|
||||
// Older builds could omit the cancellation fence or its unwind marker. Their immutable
|
||||
// native-adapter claim and unresolved preparation stage still prove that
|
||||
// provider dispatch did not begin. Require an expired owner from another
|
||||
// server boot; neither a missing PID nor mutable agent settings is proof.
|
||||
const historicalBeforeSelection = run.runtimeMode === "legacy" && !run.runtimeModeResolvedAt &&
|
||||
run.executionStage === "preparing" && !run.nativeIssueId && !run.nativeSessionId && !coordinator &&
|
||||
claimedAdapterType(run) === "paperclip_runner" && run.errorCode === "operator_interrupted" &&
|
||||
run.resultJson === null && Boolean(run.controllerBootId && run.controllerBootId !== legacyControllerBootId &&
|
||||
(run.resultJson === null || cancellation?.beforeNativeSelection === true) &&
|
||||
Boolean(run.controllerBootId && run.controllerBootId !== legacyControllerBootId &&
|
||||
run.controllerLeaseExpiresAt && run.controllerLeaseExpiresAt <= new Date());
|
||||
const beforeSelection = historicalBeforeSelection || run.runtimeMode === "legacy" && !run.runtimeModeResolvedAt &&
|
||||
!run.nativeSessionId && !coordinator && claimedAdapterType(run) === "paperclip_runner" &&
|
||||
|
||||
@@ -598,21 +598,24 @@ const support = await getEmbeddedPostgresTestSupport();
|
||||
expect(await admit(f)).toMatchObject({ previousRunId: f.sourceRunId });
|
||||
});
|
||||
|
||||
async function seedHistoricalCancelledPreparation() {
|
||||
async function seedHistoricalCancelledPreparation(retainedReceipt = false) {
|
||||
const f = await seedCancelledStartup();
|
||||
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
||||
await db.delete(environmentLeases).where(eq(environmentLeases.heartbeatRunId, f.sourceRunId));
|
||||
await db.update(heartbeatRuns).set({ runtimeMode: "legacy", runtimeModeResolvedAt: null,
|
||||
nativeIssueId: null, nativeSessionId: null, executionStage: "preparing", errorCode: "operator_interrupted",
|
||||
runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } }, resultJson: null,
|
||||
runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } },
|
||||
resultJson: retainedReceipt ? { operatorInterrupted: true, stopReason: "cancelled",
|
||||
startupCancellation: { beforeNativeSelection: true, requestedAt: "2026-09-11T10:00:00Z" } } : null,
|
||||
}).where(eq(heartbeatRuns.id, f.sourceRunId));
|
||||
await db.update(issueRecoveryActions).set({ cause: "legacy_execution_requires_reconciliation" })
|
||||
.where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
||||
return f;
|
||||
}
|
||||
|
||||
it.each(["message", "retry"])("recovers historical native preparation through an explicit %s", async kind => {
|
||||
const f = await seedHistoricalCancelledPreparation();
|
||||
it.each([false, true].flatMap(receipt => ["message", "retry"].map(kind => ({ receipt, kind }))))(
|
||||
"recovers historical native preparation through an explicit $kind (receipt: $receipt)", async ({ receipt, kind }) => {
|
||||
const f = await seedHistoricalCancelledPreparation(receipt);
|
||||
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ canRetry: true });
|
||||
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
||||
const successor = await heartbeatService(db).wakeup(f.agentId, {
|
||||
@@ -625,11 +628,13 @@ const support = await getEmbeddedPostgresTestSupport();
|
||||
previousRunId: f.sourceRunId, explicitUserContinuation: { commentId: kind === "message" ? f.commentId : null } } });
|
||||
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
||||
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId));
|
||||
expect(source).toMatchObject({ status: "cancelled", runtimeMode: "legacy", resultJson: null });
|
||||
expect(source).toMatchObject({ status: "cancelled", runtimeMode: "legacy",
|
||||
resultJson: receipt ? { startupCancellation: { beforeNativeSelection: true } } : null });
|
||||
});
|
||||
|
||||
it.each(["fresh", "earlier_delivered", "last_delivered"])("resumes saved input after historical native preparation expires across restart exactly once (%s)", async kind => {
|
||||
const f = await seedHistoricalCancelledPreparation();
|
||||
it.each([false, true].flatMap(receipt => ["fresh", "earlier_delivered", "last_delivered"].map(kind => ({ receipt, kind }))))(
|
||||
"resumes saved input after historical native preparation expires across restart exactly once ($kind, receipt: $receipt)", async ({ receipt, kind }) => {
|
||||
const f = await seedHistoricalCancelledPreparation(receipt);
|
||||
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
||||
await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() + 60_000) })
|
||||
.where(eq(heartbeatRuns.id, f.sourceRunId));
|
||||
@@ -658,14 +663,14 @@ const support = await getEmbeddedPostgresTestSupport();
|
||||
expect(successors).toHaveLength(1);
|
||||
expect(successors[0].contextSnapshot).toMatchObject({ wakeCommentIds: [f.commentId],
|
||||
explicitUserContinuation: { commentId: f.commentId }, forceFreshSession: true });
|
||||
const [receipt] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, waiting.id));
|
||||
expect(receipt).toMatchObject({ status: "coalesced", runId: successors[0].id });
|
||||
const [queueReceipt] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, waiting.id));
|
||||
expect(queueReceipt).toMatchObject({ status: "coalesced", runId: successors[0].id });
|
||||
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
||||
});
|
||||
|
||||
it.each(["resolved", "dispatching", "adapter", "missing_boot", "current_boot", "missing_lease", "live_lease", "receipt", "launch", "invoked", "coordinator", "cleanup", "remote"])(
|
||||
"holds historical native preparation with contradictory or incomplete %s evidence", async kind => {
|
||||
const f = await seedHistoricalCancelledPreparation();
|
||||
it.each([false, true].flatMap(receipt => ["resolved", "dispatching", "adapter", "missing_boot", "current_boot", "missing_lease", "live_lease", "receipt", "launch", "invoked", "coordinator", "cleanup", "remote"].map(kind => ({ receipt, kind }))))(
|
||||
"holds historical native preparation with contradictory or incomplete $kind evidence (receipt: $receipt)", async ({ receipt, kind }) => {
|
||||
const f = await seedHistoricalCancelledPreparation(receipt);
|
||||
const patch = kind === "resolved" ? { runtimeModeResolvedAt: new Date() }
|
||||
: kind === "dispatching" ? { executionStage: "dispatching" }
|
||||
: kind === "adapter" ? { runnerProfileJson: { adapterDispatch: { adapterType: "process" } } }
|
||||
|
||||
Reference in new issue
Block a user