mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 05:11:49 +01:00
fix(native): let a resume claimed after a restart write the run's existing log (#20427)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a native run is interrupted, Paperclip can persist a same-run resume: a later attempt continues the same run from its exact checkpoint > - If the server restarts before that resume is dispatched, the new server process claims it. The resume then reopens the run's existing log, and only the current controller may write it > - The resume claim took the lease owner but left the dead process's controller identity on the lease. The log writer check compares the boot id, so it refused, and the exact checkpoint resume failed as `run_log_writer_authority_lost` > - This pull request stamps the claiming process's identity on the resume claim, as restart recovery's own claim already does > - The benefit is that a resume persisted before a restart continues after it, instead of failing and needing a manual Retry ## Linked Issues or Issue Description Refs #19831. Storage chaos wave `w-20261110T003423-s9903` (local ST2, seed 9903), task t01 (codex C1, K1). The wave went 7/8; t01 was the one task that needed a Retry. - Run 279772b3 was SIGKILLed during its first turn. The restarted server resumed it, and the Codex snapshot failed ("cannot safely recover an ambiguous Codex turn start"). A same-run resume was persisted. - A second SIGKILL hit (00:35:37), and the next process dispatched the scheduled resume at 00:35:56. - The resume's run-log claim failed in `createRunLogWriterAuthority`: the coordinator lease still carried the killed process's `controllerBootId`. The exact checkpoint resume was refused (`run_log_writer_authority_lost`), and the run ended `adapter_failed`, needing a Retry. The wave had S3 latency injected, but the failure does not depend on it. The new test reproduces it on master with no storage faults. ## What Changed - `server/src/services/native-runtime/native-finalization-reconciler.ts`: `dispatchNativeSessionResumptions` writes the claiming process's controller identity (boot id, pid, host hash, process start) along with its lease owner and expiry. This matches the restart-recovery claim, so ownership proofs and the run-log writer check see the process that actually owns the resume. - `server/src/__tests__/native-session-resumption.test.ts`: a regression test for this case. ## Verification - `native-session-resumption.test.ts`: - A coordinator persisted for resume by a previous process (stale boot id), on a run with an existing object-store log, is dispatched. - The run-log writer check accepts the new claim, and the lease carries this process's identity. - The test fails on master with `run_log_writer_authority_lost`. - Regression: the native resumption, binding, recovery-deadline, restart-recovery, scheduled-resume, owner-release, runner-ownership and recovery suites pass locally. `tsc --noEmit` passes for `server`. ## Risks - The resume claim now also refreshes the controller pid, host hash and process start on the coordinator row. These describe the process that owns the lease, exactly as after a restart-recovery claim. A claim by the same process writes the same values as before. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
232480d18f
commit
1ca3e8ea2a
2 files changed
+25
No files matched your search
@@ -1,4 +1,5 @@
|
||||
import { readFile } from "node:fs/promises";
|
||||
import { createRunLogWriterAuthority } from "../services/heartbeat/run-log-authority.js";
|
||||
import { enableRunFailureReporting } from "./helpers/run-failure-reporting.js";
|
||||
import { EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db";
|
||||
import { spawn } from "node:child_process";
|
||||
@@ -402,6 +403,24 @@ describe("P6-25 pre-result native session recovery", () => {
|
||||
await expect(db.select().from(nativeRunFinalizations)).resolves.toHaveLength(8);
|
||||
});
|
||||
|
||||
it("lets a resume claimed by a restarted process write the run's existing log (storage chaos ST2 t01)", async () => {
|
||||
await db.update(nativeRunFinalizations).set({
|
||||
phase: "retryable_failure", leaseOwner: null, leaseExpiresAt: null, nextAttemptAt: new Date(0),
|
||||
controllerBootId: "previous-process-boot",
|
||||
}).where(eq(nativeRunFinalizations.runId, runId));
|
||||
await db.update(heartbeatRuns).set({ status: "failed", logStore: "s3_segments", logRef: `${companyId}/run-log` })
|
||||
.where(eq(heartbeatRuns.id, runId));
|
||||
const dispatched: Array<{ runId: string; leaseOwner: string; attempt: number }> = [];
|
||||
await dispatchNativeSessionResumptions({ db, runnerInstanceId: "heartbeat-restarted", runIds: [runId],
|
||||
dispatch: (claim) => dispatched.push(claim) });
|
||||
expect(dispatched).toHaveLength(1);
|
||||
const [resumed] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
||||
await expect(createRunLogWriterAuthority(db, resumed!, dispatched[0]!.leaseOwner)()).resolves.toBeUndefined();
|
||||
const [owner] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId));
|
||||
expect(owner).toMatchObject({ controllerPid: process.pid, leaseOwner: dispatched[0]!.leaseOwner });
|
||||
expect(owner!.controllerBootId).not.toBe("previous-process-boot");
|
||||
});
|
||||
|
||||
it("uses provider-neutral checkpoint-free bootstrap retry only when durable evidence proves no session existed", async () => {
|
||||
await expect(claimNativeSessionResumptions({
|
||||
db,
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { notifyLiveRunRecovery } from "../live-run-recovery-notifications.js";
|
||||
import { currentControllerProcessStamp } from "../controller-process-identity.js";
|
||||
import { NATIVE_SESSION_RESUMPTION_ISSUE_STATUSES } from "./native-session-resumption-statuses.js";
|
||||
import { notifyExternalAgentSettlement } from "../../modules/external-agents/index.js";
|
||||
import { pendingNativeRunProjectionCondition } from "./native-run-projection.js";
|
||||
@@ -312,6 +313,11 @@ export async function claimNativeSessionResumptions(input: {
|
||||
phase: "observed",
|
||||
leaseOwner,
|
||||
leaseExpiresAt,
|
||||
...(() => {
|
||||
const stamp = currentControllerProcessStamp();
|
||||
return { controllerBootId: stamp.bootId, controllerPid: stamp.pid, controllerHostHash: stamp.hostHash,
|
||||
controllerProcessStartedAt: stamp.processStartedAt ? new Date(stamp.processStartedAt) : null };
|
||||
})(),
|
||||
failureCode: null,
|
||||
failureDetail: null,
|
||||
nextAttemptAt: null,
|
||||
|
||||
Reference in new issue
Block a user