mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 05:11:49 +01:00
fix(recovery): bootstrap a remote follow-up that a restart cut before its Runner launched (#20642)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a control-plane restart, restart recovery classifies each claimed native run: reattach a live Runner, resume from a checkpoint, or bootstrap again when durable records prove no provider work > - A follow-up run on a continued session gets a rebound admission checkpoint. It names the new run, but it keeps the previous turn's provider session ID > - Recovery read that copied provider session as this run's provider work. A follow-up killed after `native.process_start_requested` but before its remote Runner launched was classified `reattach_remote_runner` > - The reattach then had no Runner to find, failed with `native_continuation_requires_reconciliation`, and the task needed Retry > - This pull request stops the copied session from counting as this run's work, and keeps the existing durable proofs as the authority > - The benefit is that these restarts bootstrap again by themselves and do not repeat provider work ## Linked Issues or Issue Description Refs #20561, #20555 **What happened?** Two Daytona restart-chaos tasks needed a Retry click: - Wave `w-20261110T173243-s9202` t02 (claude C1, K2): the follow-up run was killed at `stage.asset.home`, after `native.process_start_requested`. Recovery chose `reattach_remote_runner`. Verification failed with `native.remote_runner_recovery_failed {stage: runner_state, reason: state_read_failed}` (`runner_remote_state_unavailable: exit=1`): `runner-state.json` did not exist because the Runner never launched. The storage fault window (ST7) had ended 30 s before, so it was not the cause. - Wave `w-20261110T174930-s9205` t04 (codex C1, RR1): the scheduled retry of a `process_lost` run was killed at `session.checkpoint.restore`, after `native.process_start_requested`. Recovery chose `reattach_remote_runner`. Verification failed with `native_remote_recovery_lease_mismatch: stage=runner_verification field=identity_run_id`: the local Runner identity belonged to the previous run, because this run's Runner never started. **Expected behavior** A run whose durable records prove that no Runner received a provider turn bootstraps again without a person. **Steps to reproduce** 1. Start a remote follow-up run on a continued session (for example, a comment after a completed turn). 2. Kill the server after `native.process_start_requested` and before the remote Runner launches. 3. Observe `reattach_remote_runner`, a failed verification and a Retry banner. **Paperclip version or commit** master close to `6d6b2d0e` (waves); still reproduced by the new tests on master `90dd864bff`. ## What Changed - `native-restart-recovery.ts`, no controller checkpoint for the run (`remote_runner_never_received_commands`): a follow-up with `sessionIdBefore` qualifies when it carries only its continued-session checkpoint, and a predecessor's settled Runner identity (`carriesInheritedRunnerIdentity`) no longer disqualifies it. A recorded process identity, a Runner-sent event, a provider event or a pending request still keeps the run held. - `native-restart-recovery.ts`, controller checkpoint exists: the existing Postgres proof (`nativeRunnerCheckpointRemoteBootstrapProof`) now also runs when the run carries only a rebound checkpoint whose provider session a settled earlier run of the same session saved (`carriesSettledPredecessorSession`). The copied provider session then does not count as this run's provider evidence. The controller checkpoint stays the authority: any turn command or turn event keeps the run held. - `doc/execution-semantics.md`: describe the rule. ## Verification - `server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts` (real Postgres): - "restarts a follow-up whose controller never saved a checkpoint (s9205 t04)" and the variant "with its predecessor's remote runner identity": red on master (`reattach_remote_runner`), green now. - "restarts a follow-up whose saved controller checkpoint proves no provider turn (s9202 t02)": red on master, green now. - Held cases stay held: own recorded Runner identity, a carried active turn, a provider session that no settled earlier run saved, a `turn.start` command, a projected `turn.started` event. - Whole file: 171 passed. - `npx tsc --noEmit -p server` passes. - The executor's existing `bootstrap_incomplete` path handles both shapes: a scoped root that belongs to the previous run goes through the existing prior-owner verification, and a root of this run restored from the controller checkpoint goes through `runnerdStateProvesIncompleteBootstrap` / `runnerdStateProvesNoRemoteProviderTurn`. ## Risks - A run with any recorded Runner process, Runner-sent event, provider event, turn command or pending request keeps today's reattach or hold. - A bootstrap claim charges one provider attempt, as today. The bounded limit (3) still applies. - No persisted shape changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
DottaandClaude authored and GitHub committed
1 parent
6892ed3a98
commit
24661683db
3 files changed
+131
-10
No files matched your search
@@ -4198,6 +4198,14 @@ its normalized current run ID does not prove that the current controller attache
|
||||
Fenced remote Runner recovery requires a matching database controller checkpoint
|
||||
or a valid staged attachment for the target run before selecting exact-checkpoint
|
||||
resume. Local Runner recovery keeps its existing local controller checks.
|
||||
A follow-up run that a restart interrupted before its remote Runner launched
|
||||
carries only that copied checkpoint. Restart recovery does not treat the copied
|
||||
provider session as this run's provider work. When the run saved no controller
|
||||
checkpoint, or its controller checkpoint records no provider turn, and no Runner
|
||||
process identity or Runner-sent event exists for the run, recovery bootstraps
|
||||
the run again instead of attempting to reattach a Runner that never started. A
|
||||
copied session counts only when a settled earlier run of the same session saved
|
||||
that exact provider session.
|
||||
If the first attachment finds the retained runner gone, or replacement cannot
|
||||
archive its remote state before the new turn starts, ordinary failure settlement
|
||||
requests one normal Retry under the
|
||||
|
||||
@@ -25,7 +25,7 @@ import {
|
||||
recordNativeLocalProcessStopInTransaction,
|
||||
} from "../native-local-process-stop.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { and, desc, eq, gt, inArray, isNotNull, isNull, lte, ne, or, sql } from "drizzle-orm";
|
||||
import { and, desc, eq, gt, inArray, isNotNull, isNull, lt, lte, ne, or, sql } from "drizzle-orm";
|
||||
import { alias } from "drizzle-orm/pg-core";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import {
|
||||
@@ -821,6 +821,24 @@ async function hasNativeRunnerOriginatedEvidence(
|
||||
return event !== undefined;
|
||||
}
|
||||
|
||||
/** A settled earlier run of the same native session saved this exact provider
|
||||
* session. Admission rebinds that checkpoint to a follow-up run, so the
|
||||
* follow-up's copy of the provider session is carried, not its own work. */
|
||||
async function carriesSettledPredecessorSession(
|
||||
tx: Pick<Db, "select">,
|
||||
run: { companyId: string; id: string; nativeSessionId: string | null; createdAt: Date },
|
||||
providerSessionId: unknown,
|
||||
): Promise<boolean> {
|
||||
if (typeof providerSessionId !== "string" || !providerSessionId || !run.nativeSessionId) return false;
|
||||
const [predecessor] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.nativeSessionId, run.nativeSessionId),
|
||||
ne(heartbeatRuns.id, run.id), lt(heartbeatRuns.createdAt, run.createdAt), isNotNull(heartbeatRuns.finishedAt),
|
||||
inArray(heartbeatRuns.status, ["succeeded", "failed", "cancelled", "timed_out", "interrupted"]),
|
||||
sql`${heartbeatRuns.runnerProfileJson}#>>'{sessionCheckpoint,providerSessionId}' = ${providerSessionId}`,
|
||||
)).limit(1);
|
||||
return predecessor !== undefined;
|
||||
}
|
||||
|
||||
async function hasNativeRunEvent(
|
||||
tx: Pick<Db, "select">, run: { companyId: string; id: string }, eventType: string,
|
||||
): Promise<boolean> {
|
||||
@@ -1516,18 +1534,29 @@ export async function claimNativeRestartRecoveries(input: {
|
||||
// provider session: session-open events may already be projected here,
|
||||
// but no session snapshot or turn exists. Those events alone do not make
|
||||
// the checkpoint proof ambiguous; any turn or provider event does.
|
||||
const providerEvidenceIsSessionOpenOnly = !hasCheckpointProviderIdentity
|
||||
// A follow-up on a continued session carries only its rebound admission
|
||||
// checkpoint: the previous turn's provider session, with no cursor,
|
||||
// turn, result or request of this run. When a settled earlier run of the
|
||||
// same session saved that exact provider session, the carried session is
|
||||
// not this run's provider work, so it does not make the controller
|
||||
// checkpoint proof ambiguous (Daytona waves s9202, s9205).
|
||||
const reboundSessionOnly = continuedSessionOnly && Boolean(durableAuthority) && remoteWorkspace !== null
|
||||
&& hasCheckpointProviderIdentity && !unclaimedBootstrap && !pluginTask && stagedAttachment === null
|
||||
&& await carriesSettledPredecessorSession(tx, row.run, checkpointRecord.providerSessionId);
|
||||
const providerEvidenceIsSessionOpenOnly = (!hasCheckpointProviderIdentity || reboundSessionOnly)
|
||||
&& providerEvents.every(event => SESSION_OPEN_PROVIDER_EVENT_TYPES.has(event.eventType));
|
||||
const remoteBootstrapProof = !unclaimedBootstrap && remoteWorkspace !== null && !pluginTask
|
||||
&& Boolean(durableAuthority) && stagedAttachment === null && !hasCheckpoint && providerEvidenceIsSessionOpenOnly
|
||||
&& Boolean(durableAuthority) && stagedAttachment === null && (!hasCheckpoint || reboundSessionOnly)
|
||||
&& providerEvidenceIsSessionOpenOnly
|
||||
&& Boolean(row.run.runnerInstanceId && row.run.nativeSessionId)
|
||||
? await nativeRunnerCheckpointRemoteBootstrapProof(tx, {
|
||||
companyId: row.run.companyId, issueId: row.coordinator.issueId, runId: row.run.id,
|
||||
}, { runnerInstanceId: row.run.runnerInstanceId!, normalizedSessionId: row.run.nativeSessionId! })
|
||||
: null;
|
||||
// Projected session-open events need the stronger session-opened proof.
|
||||
const ownProviderEvidence = reboundSessionOnly ? providerEvents.length > 0 : hasProviderEvidence;
|
||||
const remoteBootstrapProvenIncomplete = remoteBootstrapProof !== null
|
||||
&& (!hasProviderEvidence || remoteBootstrapProof === "session_opened_without_turn");
|
||||
&& (!ownProviderEvidence || remoteBootstrapProof === "session_opened_without_turn");
|
||||
|
||||
const remoteRunnerNeverReceivedCommands = !unclaimedBootstrap && !remoteRunnerNeverLaunched
|
||||
&& !remoteRunnerExitedBeforeProviderWork && !remoteBootstrapProvenIncomplete
|
||||
@@ -1538,8 +1567,10 @@ export async function claimNativeRestartRecoveries(input: {
|
||||
&& row.coordinator.attempt >= 1 && row.coordinator.controllerGeneration >= 1
|
||||
&& Boolean(row.coordinator.controllerBootId && row.coordinator.controllerPid && row.coordinator.controllerProcessStartedAt)
|
||||
&& !row.coordinator.failureDetail?.successorRunId
|
||||
&& !row.run.processPid && !row.run.processGroupId && !row.run.processStartedAt
|
||||
&& !row.run.sessionIdBefore && !row.run.sessionIdAfter
|
||||
// A follow-up may carry its predecessor's settled runner identity and
|
||||
// session; neither is this run's runner or provider work.
|
||||
&& (inheritedRunnerIdentity || (!row.run.processPid && !row.run.processGroupId && !row.run.processStartedAt))
|
||||
&& (!row.run.sessionIdBefore || continuedSessionOnly) && !row.run.sessionIdAfter
|
||||
&& Boolean(row.run.nativeSessionId && row.run.runnerInstanceId)
|
||||
&& !(await hasNativeRunnerOriginatedEvidence(tx, row.run))
|
||||
&& !(await hasNativeRunEvent(tx, row.run, PROCESS_IDENTITY_RECORDED));
|
||||
@@ -1565,7 +1596,7 @@ export async function claimNativeRestartRecoveries(input: {
|
||||
knownProviderProcessIdentityAmbiguous:
|
||||
providerProcesses.ambiguousLivePids.length > 0,
|
||||
hasCheckpoint: hasCheckpoint && !remoteRunnerNeverReceivedCommands && !(unclaimedBootstrap && inheritedCheckpointOnly)
|
||||
&& !remoteRunnerNeverLaunched,
|
||||
&& !remoteRunnerNeverLaunched && !(remoteBootstrapProvenIncomplete && reboundSessionOnly),
|
||||
stagedAttachmentVerified: stagedAttachment !== null,
|
||||
checkpointIdentityMatches:
|
||||
checkpointIdentityMatches &&
|
||||
|
||||
+85
-3
@@ -2588,18 +2588,20 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
|
||||
attempt?: number;
|
||||
mutate?: (core: DurablePrpControlPlane) => void;
|
||||
checkpoint?: boolean;
|
||||
/** False: the controller saved its checkpoint, but no remote runner was launched yet. */
|
||||
launched?: boolean;
|
||||
} = {}) {
|
||||
const fixture = await seedRun(`REMOTE-CLAIMED-${label}`);
|
||||
const { runId, issueId, native } = fixture;
|
||||
const remote = remotePreparedWorkspace();
|
||||
await fixture.db.update(heartbeatRuns).set({
|
||||
runnerProfileJson: { ...fixture.preparedProfile, nativeWorkspaceSync: remote },
|
||||
processPid: 443, processStartedAt: new Date(Date.now() - 30_000),
|
||||
...(options.launched === false ? {} : { processPid: 443, processStartedAt: new Date(Date.now() - 30_000) }),
|
||||
}).where(eq(heartbeatRuns.id, runId));
|
||||
await fixture.db.insert(heartbeatRunEvents).values([
|
||||
{ companyId, runId, agentId, seq: 900, stream: "system", level: "info", eventType: "native.process_start_requested" },
|
||||
{ companyId, runId, agentId, seq: 901, stream: "system", level: "info", eventType: "native.process_identity_recorded",
|
||||
payload: { targetKind: "remote", processPid: 443 } },
|
||||
...(options.launched === false ? [] : [{ companyId, runId, agentId, seq: 901, stream: "system", level: "info",
|
||||
eventType: "native.process_identity_recorded", payload: { targetKind: "remote", processPid: 443 } }]),
|
||||
]);
|
||||
const bootId = randomUUID();
|
||||
const leaseOwner = `${bootId}:1:${randomUUID()}`;
|
||||
@@ -2926,6 +2928,86 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
|
||||
expect(coordinator.recoveryHistory.at(-1)).toMatchObject({ reason: "remote_runner_never_received_commands" });
|
||||
});
|
||||
|
||||
// Daytona waves s9202 t02 (B1) and s9205 t04 (B2): a follow-up run on a
|
||||
// continued session was killed after native.process_start_requested but
|
||||
// before its remote runner launched. Admission rebinds the previous turn's
|
||||
// checkpoint to the new run: its identity names this run and it keeps the
|
||||
// provider session ID, but it records no cursor, turn, result or request.
|
||||
// That carried session is not this run's provider work.
|
||||
async function reboundFollowUpCheckpoint(fixture: Awaited<ReturnType<typeof seedRun>>, extra: Record<string, unknown> = {},
|
||||
predecessor = true) {
|
||||
const [seeded] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
|
||||
// The settled previous turn of the same session saved the provider session.
|
||||
if (predecessor) await fixture.db.insert(heartbeatRuns).values({ companyId, agentId, status: "succeeded", runtimeMode: "native",
|
||||
nativeIssueId: fixture.issueId, nativeSessionId: seeded.nativeSessionId, runnerInstanceId: seeded.runnerInstanceId,
|
||||
createdAt: new Date(seeded.createdAt.getTime() - 120_000), finishedAt: new Date(seeded.createdAt.getTime() - 10_000),
|
||||
contextSnapshot: { issueId: fixture.issueId }, runnerProfileJson: { sessionCheckpoint: { providerSessionId: "inherited-thread" } } });
|
||||
await fixture.db.update(heartbeatRuns).set({ sessionIdBefore: "inherited-thread", runnerProfileJson: {
|
||||
...seeded.runnerProfileJson, sessionCheckpoint: {
|
||||
driverKind: "codex_app_server", backendKind: "runner", sessionId: "inherited-thread", cursor: null,
|
||||
identity: { companyId, issueId: fixture.issueId, runId: fixture.runId, agentId, sessionId: fixture.native.normalizedSessionId },
|
||||
providerSessionId: "inherited-thread", activeTurnId: null, terminal: null, terminalTurns: [], semanticResult: null,
|
||||
pendingRuntimeRequests: [], lineage: [], ...extra,
|
||||
} } }).where(eq(heartbeatRuns.id, fixture.runId));
|
||||
}
|
||||
|
||||
it.each([
|
||||
{ label: "restarts", mode: "unlaunched" },
|
||||
{ label: "restarts with its predecessor's remote runner identity", mode: "inherited_process" },
|
||||
{ label: "keeps its own recorded runner held", mode: "identity_recorded" },
|
||||
{ label: "keeps a carried active turn held", mode: "active_turn" },
|
||||
] as const)("$label a follow-up whose controller never saved a checkpoint (s9205 t04)", async ({ mode }) => {
|
||||
const fixture = await seedUnlaunchedRemoteClaim(`FOLLOW-UP-UNCOMMANDED-${mode}`,
|
||||
{ event: { eventType: "native.process_start_requested" } });
|
||||
await reboundFollowUpCheckpoint(fixture, mode === "active_turn" ? { activeTurnId: "turn-in-flight" } : {});
|
||||
if (mode === "inherited_process") {
|
||||
const processStartedAt = new Date(Date.now() - 120_000);
|
||||
const [current] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
|
||||
await fixture.db.update(heartbeatRuns).set({ processPid: 596, processGroupId: null, processStartedAt, processHostHash: "remote" })
|
||||
.where(eq(heartbeatRuns.id, fixture.runId));
|
||||
await fixture.db.insert(heartbeatRuns).values({ companyId, agentId, status: "succeeded", runtimeMode: "native",
|
||||
nativeIssueId: fixture.issueId, nativeSessionId: current.nativeSessionId, runnerInstanceId: current.runnerInstanceId,
|
||||
createdAt: new Date(current.createdAt.getTime() - 60_000), finishedAt: new Date(current.createdAt.getTime() - 5_000),
|
||||
processPid: 596, processGroupId: null, processStartedAt, processHostHash: "remote",
|
||||
contextSnapshot: { issueId: fixture.issueId } });
|
||||
}
|
||||
if (mode === "identity_recorded") await fixture.db.insert(heartbeatRunEvents).values({ companyId, runId: fixture.runId, agentId,
|
||||
seq: 901, stream: "system", level: "info", eventType: "native.process_identity_recorded" });
|
||||
const [claim] = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
|
||||
restartKind: "hard", runIds: [fixture.runId] });
|
||||
if (mode === "identity_recorded" || mode === "active_turn") {
|
||||
expect(claim).not.toMatchObject({ kind: "bootstrap_incomplete" });
|
||||
return;
|
||||
}
|
||||
expect(claim).toMatchObject({ kind: "bootstrap_incomplete", runId: fixture.runId, controllerGeneration: 2 });
|
||||
const [coordinator] = await fixture.db.select().from(nativeRunFinalizations)
|
||||
.where(eq(nativeRunFinalizations.runId, fixture.runId));
|
||||
expect(coordinator.recoveryHistory.at(-1)).toMatchObject({ reason: "remote_runner_never_received_commands" });
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ label: "restarts", mode: "bootstrap" },
|
||||
{ label: "keeps a provider session no settled earlier run saved held in", mode: "no_predecessor" },
|
||||
{ label: "keeps a turn command held", mode: "turn_command" },
|
||||
{ label: "keeps a projected provider turn event held", mode: "turn_event" },
|
||||
] as const)("$label a follow-up whose saved controller checkpoint proves no provider turn (s9202 t02)", async ({ mode }) => {
|
||||
const fixture = await seedClaimedRemoteBootstrap(`FOLLOW-UP-CLAIMED-${mode}`, {
|
||||
launched: false,
|
||||
mutate: mode === "turn_command" ? core => { openSession(core);
|
||||
core.queueCommand("turn.start", { prompt: "Prior work must stay held." }); } : undefined,
|
||||
});
|
||||
await reboundFollowUpCheckpoint(fixture, {}, mode !== "no_predecessor");
|
||||
if (mode === "turn_event") await fixture.db.insert(heartbeatRunEvents).values({ companyId, runId: fixture.runId, agentId,
|
||||
seq: 902, stream: "system", level: "info", eventType: "turn.started" });
|
||||
const [claim] = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
|
||||
restartKind: "hard", runIds: [fixture.runId] });
|
||||
if (mode !== "bootstrap") {
|
||||
expect(claim).not.toMatchObject({ kind: "bootstrap_incomplete" });
|
||||
return;
|
||||
}
|
||||
expect(claim).toMatchObject({ kind: "bootstrap_incomplete", runId: fixture.runId, providerAttempt: 1, controllerGeneration: 2 });
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ label: "restarts", providerEvent: false, process: "none" },
|
||||
{ label: "restarts with its predecessor's runner identity", providerEvent: false, process: "inherited" },
|
||||
|
||||
Reference in new issue
Block a user