feat(runner): relaunch a released task as the next run after a restart (#22343)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - An environment plugin can keep a warm task for an agent session.
After each turn, Core releases the task's Runner, and the next run of
the session resumes it
> - #22288 added `resumeIdentityVersion: 1`: for a provider that stops a
released Runner, Core rotates the session to the next run and the
provider starts the Runner again directly as that run
> - Core takes that path only while it still holds the released session
in memory. After a Core restart, it recovers the released task through
the older hand-off: it resumes the task as the previous run, then
switches the Runner's authority to the new run
> - So a provider that advertises the capability still has to keep the
hand-off for every follow-up after a Core restart, and cannot drop it
> - This pull request makes restart recovery take the same path when the
provider advertises the capability
> - The benefit is that a provider with the capability gets one start
shape for a released task, before and after a Core restart, and can
retire the hand-off

## Linked Issues or Issue Description

Follows #22288.

**What existing behavior does this improve?**

The first run of a warm plugin session after a Core restart, when its
task was released and its provider advertises `resumeIdentityVersion:
1`.

**Subsystem affected**

The native session executor and the plugin Runner task owner.

**Current behavior**

After a restart, Core finds the released source on disk
(`inspectRemoteWarmSessionRecovery`), creates a backend that adopts the
task, and calls `recoverSession`. The task owner calls `resume` without
`runner`, the provider starts the Runner as the previous run, and Core
hands it to the new run.

**Proposed behavior**

When the provider advertises the capability, Core recorded the build the
task was released on, the process configuration is unchanged, and the
session is not moving to another task, Core treats the recovery like a
cached released session. It revalidates the recovery, loads the warm
checkpoint, and opens a new backend on the same task with
`relaunchReleased`. A new task owner takes over the submitted task
without adopting a Runner process, prepares the claimant run with the
released source as `priorIdentity`, and calls `resume` with `runner`.
Every other recovery keeps the hand-off.

**Reason and benefit**

A provider that advertises the capability no longer needs the hand-off
for its own tasks, after a restart or not.

## What Changed

- Executor: a remote warm recovery with a released source, a recorded
build, a reusable process configuration and no session move, on a
provider that advertises the capability, revalidates the recovery and
becomes a relaunch: `relaunchReleased` with the released source and
build, the warm checkpoint, and no `remoteWarmRecovery` for the backend.
The startup context ends as for a cached relaunch.
- Executor: the relaunch's task owner can now come from the new backend
(`onPluginTaskOwner`), so the unavailable-reuse fallback and the
rotation revert work without a retained owner. Without an owner, no
relaunch can have started, so the revert applies.
- Backend: a relaunch neither stops a resumed task that has no durable
binding nor resolves a `resumeFrom` task.
- Task owner: an owner created with `relaunch` and the capability takes
over an already submitted task, as an adopting owner does, but adopts no
process: its prepare sends the released source as `priorIdentity`, and
its launcher calls `resume` with `runner`.
- `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md`: Core prepares the claimant
run before the resume, also after a restart, and a provider must accept
that prepare without state kept in memory.

## Verification

- `server`: `npx vitest run
src/services/native-runtime/plugin-runner-release-resume.integration.test.ts`:
new `adopted restart relaunch` cases for Codex and Claude, with real
runnerd processes. Core restarts while the task is released; the next
run relaunches the task as itself, with no claim, one inspection and no
second submit. The existing `adopted relaunch` case now relaunches both
follow-ups, since the idle close also recovers through this path.
- `server`: `plugin-runner-task.test.ts`: a new owner relaunches without
submitting or adopting, and prepares with the released source; without
the capability it prepares a new task.
- `server`: `plugin-runner-release-resume.integration.test.ts` and
`plugin-runner-eviction.integration.test.ts`: 61 passed.
`native-session-executor.test.ts`: 1535 passed, 7 skipped.
`plugin-runner-task.test.ts`: 118 passed.
- Typecheck: `server` passes.
- Red-green: with the executor lane turned off, both `adopted restart
relaunch` cases fail. Without the task owner change, the new-owner
relaunch test fails.

## Risks

- Only providers that advertise `resumeIdentityVersion: 1` see a change.
Recovery without a recorded build, with a changed process configuration,
or for a session moving to another task keeps the hand-off.
- The provider sees a `prepare` for the claimant run on a released task
right after a Core restart. A provider that tracked unclaimed released
tasks only in memory must handle that prepare (the doc now says so).
- A Core restart in the middle of this path is still not qualified, as
for the cached relaunch.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code with tool use
(code search, editing, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
authored and GitHub committed 2026-11-19 11:20:48 -08:00
1 parent 46dde54f52
commit 806440d12c
5 files changed
+85 -25

No files matched your search

+7
View File
@@ -174,6 +174,13 @@ event acknowledged and no command in progress. Core then sends one ordinary
and `runner` must return the original result and must not start a second
Runner. Without `runner` on `resume`, keep the claimant handover above.
Core prepares the claimant run before this `resume`: its `prepare` has
`priorIdentity` set to the released source, and the released task is not yet
claimed. Core also takes this path when it recovers a released task after a
Core restart, if it recorded the build the task was released on and the
session's process configuration has not changed. A provider must then accept
that `prepare` without state it kept in memory before the restart.
While a released task can still be resumed, its `status` must report
`executionStopped: false`, even when its Runner process is stopped. Core
replaces a reused task whose `status` reports a stop, and then cannot resume
@@ -12232,7 +12232,7 @@ async function executePaperclipNativeSessionAttempt(
);
let existingWarmSession: NativeSession | undefined;
let reusingReleasedRunner = false;
let relaunchReleased: { owner: ReturnType<typeof createPluginRunnerTaskOwner>; source: DurableRecoveryIdentity;
let relaunchReleased: { owner?: ReturnType<typeof createPluginRunnerTaskOwner>; source: DurableRecoveryIdentity;
runner: { version: string; digest: string }; published?: true } | undefined;
let managedCredentialSession: NativeSession | undefined;
let githubAccess: NativeGitHubAccess | undefined;
@@ -12560,24 +12560,44 @@ async function executePaperclipNativeSessionAttempt(
pluginTaskDescription,
retainedPluginTaskOwner: existingWarmSession ? pluginTaskSessionOwners.get(existingWarmSession) : relaunchReleased?.owner,
relaunchReleased: relaunchReleased ? { source: relaunchReleased.source, runner: relaunchReleased.runner } : undefined,
onPluginTaskOwner: owner => { if (relaunchReleased && !relaunchReleased.owner) relaunchReleased.owner = owner; },
beginRetainedRunAttachment: existingWarmSession && reusingReleasedRunner ? beginRetainedRunAttachment(existingWarmSession) : undefined,
durableEnvironmentLeaseId: durableRunnerBinding?.environmentLeaseId,
remoteWarmRecovery: remoteWarmRecovery ? {
...remoteWarmRecovery,
remoteWarmRecovery: remoteWarmRecovery ? (recovery => ({
...recovery,
revalidate: async () => {
if (remoteWarmRecoveryPending) await remoteWarmRecovery.revalidate(leaseOwner);
if (remoteWarmRecoveryPending) await recovery.revalidate(leaseOwner);
},
} : undefined,
}))(remoteWarmRecovery) : undefined,
trace,
toolTrace,
});
const releasedRecovery = remoteWarmRecovery;
if (releasedRecovery?.releasedSource && releasedRecovery.releasedSourceRunner && releasedRecovery.reuseProcess
&& !releasedRecovery.portableSourceTaskId && pluginTaskDescription?.resumeIdentityVersion === 1 && warmConfigDigest !== null) {
const checkpoint = loadWarmNativeCheckpoint(input.execution, warmConfigDigest, "remote");
if (checkpoint) {
if (executionContext) await executionContext.wait(() => releasedRecovery.revalidate(leaseOwner));
else await releasedRecovery.revalidate(leaseOwner);
relaunchReleased = { source: releasedRecovery.releasedSource, runner: releasedRecovery.releasedSourceRunner };
persistedWarmSession = checkpoint;
remoteWarmRecovery = undefined;
remoteWarmRecoveryPending = false;
await emitStartupReuse(input.onEvent, "runner", { state: "fresh", reason: "replacement" });
executionContext?.assertOpen();
await emitStartupReuse(input.onEvent, "provider_session", { state: "reused", reason: "verified_reuse" });
trace.setBudgetStart("warm");
if (executionContext) await finishStartupContextForFallback();
}
}
let runnerdBackend =
input.useRunnerd && input.backend === undefined
? await timeController("backend_setup", () => executionContext ? executionContext.wait(createExecutionBackend) : createExecutionBackend())
: null;
if (remoteWarmRecovery) {
if (executionContext) await executionContext.wait(() => remoteWarmRecovery.revalidate(leaseOwner));
else await remoteWarmRecovery.revalidate(leaseOwner);
const recovery = remoteWarmRecovery;
if (executionContext) await executionContext.wait(() => recovery.revalidate(leaseOwner));
else await recovery.revalidate(leaseOwner);
const recovered = await runnerdBackend?.recoverSession?.(remoteWarmRecovery.checkpoint, {
signal: executionContext?.signal ?? new AbortController().signal,
});
@@ -13109,7 +13129,7 @@ async function executePaperclipNativeSessionAttempt(
try {
result = await trace.run(runnerSessionStartupScope, runExecution);
} catch (error) {
if (relaunchReleased && !relaunchReleased.published && !relaunchReleased.owner.relaunchMayHaveStarted()) {
if (relaunchReleased && !relaunchReleased.published && !relaunchReleased.owner?.relaunchMayHaveStarted()) {
revertUnattachedRelaunchRotation(scopedRunnerdStateRoot(input.execution), relaunchReleased.source);
}
if (!await continueAfterUnavailableReuse(error)) throw error;
@@ -16400,6 +16420,7 @@ export async function createRunnerdBackend(input: {
remoteWarmRecovery?: Omit<RemoteWarmSessionRecovery, "revalidate"> & { revalidate: () => Promise<void> };
retainedPluginTaskOwner?: ReturnType<typeof createPluginRunnerTaskOwner>;
relaunchReleased?: { source: DurableRecoveryIdentity; runner: { version: string; digest: string } };
onPluginTaskOwner?: (owner: ReturnType<typeof createPluginRunnerTaskOwner>) => void;
beginRetainedRunAttachment?: () => void;
pluginTaskDescription?: PluginTaskDescription;
onSpawn?: (meta: {
@@ -16874,7 +16895,8 @@ async function createRunnerdBackendWithinSessionClaim(
}
let priorDescription = input.pluginTaskDescription?.taskId === remoteTarget.sandboxLeaseAcquisition?.providerLeaseId
? input.pluginTaskDescription : undefined;
if (remoteTarget.sandboxLeaseAcquisition?.outcome === "resumed" && !durableBinding && !input.retainedPluginTaskOwner && !input.remoteWarmRecovery) {
if (remoteTarget.sandboxLeaseAcquisition?.outcome === "resumed" && !durableBinding && !input.retainedPluginTaskOwner && !input.remoteWarmRecovery
&& !input.relaunchReleased) {
// A new logical session (including fresh retries) cannot inherit the old process authority.
await pluginTask({kind:"stop"});
const stopped=environmentTaskResultSchema.parse(await pluginTask({kind:"status"}));
@@ -16901,7 +16923,7 @@ async function createRunnerdBackendWithinSessionClaim(
turnId: input.remoteWarmRecovery.identity.turnId, itemId: input.remoteWarmRecovery.identity.itemId,
} : undefined;
const resumeFrom = !retainedOwner && !adopting && prior && prior.environmentLeaseId !== remoteTarget.leaseId
const resumeFrom = !retainedOwner && !adopting && !input.relaunchReleased && prior && prior.environmentLeaseId !== remoteTarget.leaseId
? await resolvePluginTaskResume(input.db, { companyId: input.execution.binding.companyId,
leaseId: remoteTarget.leaseId!, priorIdentity: prior }) : undefined;
const credentials = await preparePluginRunnerCredentials({
@@ -16986,6 +17008,7 @@ async function createRunnerdBackendWithinSessionClaim(
if (input.execution.session.lifecyclePolicy.mode === "warm" && (!credentials.accountIdentity || description.warmTaskVersion !== 1)) throw new Error("Plugin warm task capability or account unavailable");
const owner = retainedOwner ?? createPluginRunnerTaskOwner(ownerInput);
if (retainedOwner) retainedOwner.rebind(ownerInput);
input.onPluginTaskOwner?.(owner);
return { provider, description, prior, adoptionIdentity, adopting, owner };
}
if (pluginTask && remoteTarget?.transport === "sandbox" && input.execution.schema !== "paperclip.native-execution-input.v6" && input.execution.schema !== "paperclip.native-execution-input.v8") {
@@ -68,6 +68,7 @@ describeEmbeddedPostgres("plugin Runner release/resume through the Core executio
nativeIt.for([...(["codex", "claude"] as const).flatMap(harness => (["cached", "adopted", "adopted restart"] as const).map(mode => [harness, mode] as const)), ["codex", "adopted cancellation"] as const, ["codex", "adopted legacy cancellation"] as const,
["codex", "cached cancellation"] as const, ["codex", "cached held claim"] as const, ["claude", "cached held claim"] as const,
["codex", "cached relaunch"] as const, ["claude", "cached relaunch"] as const, ["codex", "adopted relaunch"] as const,
["codex", "adopted restart relaunch"] as const, ["claude", "adopted restart relaunch"] as const,
["codex", "cached managed GitHub"] as const, ["codex", "adopted managed GitHub"] as const, ["codex", "adopted restart managed GitHub"] as const])("completes %s runs through a retained %s process", { timeout: 60_000 }, async ([harness, mode], context) => {
if (harness === "claude" && !existsSync(fakeAcpxSidecar)) context.skip();
const thisCase: { teardown?: () => Promise<void> } = {};
@@ -263,7 +264,7 @@ describeEmbeddedPostgres("plugin Runner release/resume through the Core executio
return { kind: "resume", taskId, outcome: "resumed", runner: artifact, archivedState };
}
if (operation.kind === "resume") {
expect(relaunch && !adopted).toBe(false);
expect(relaunch).toBe(false);
const retainedProviderCount = await providerStarts();
const reply = await host.request({ op: "claim", id: `claim_${sha(operation.bootstrapTicket)}`, parkId: parkId(operation.identity), bootstrapTicket: operation.bootstrapTicket });
if (reply.status !== "claimed") throw new Error("Fixture native claim refused");
@@ -386,19 +387,21 @@ describeEmbeddedPostgres("plugin Runner release/resume through the Core executio
expect(releases[1]!.environmentLeaseId).toBe(leaseA); expect(submissions).toBe(1);
if (relaunch) {
expect(handle!.child.pid).not.toBe(pid);
const relaunched = adopted ? releases[1]! : releases[0]!;
expect(relaunches).toEqual([{ source: relaunched, runId: adopted ? runC : runB, archivedRunId: relaunched.runId }]);
const relaunched = adopted ? [[releases[0]!, runB], [releases[1]!, runC!]] as const : [[releases[0]!, runB]] as const;
expect(relaunches).toEqual(relaunched.map(([source, next]) => ({ source, runId: next, archivedRunId: source.runId })));
expect((await state()).runId).toBe(releases.at(-1)!.runId);
expect(JSON.parse(await readFile(join(runnerRoot, `archived-${relaunched.runId}.json`), "utf8"))).toMatchObject(relaunched);
for (const [source] of relaunched) {
expect(JSON.parse(await readFile(join(runnerRoot, `archived-${source.runId}.json`), "utf8"))).toMatchObject(source);
}
} else expect(handle!.child.pid).toBe(pid);
expect([...new Set(preparedBrokers.map(prepared => prepared.runId))]).toEqual(releases.map(release => release.runId));
const methods = (await readFile(calls, "utf8")).trim().split(/\r?\n/);
if (harness === "codex") {
expect(methods.filter(method => method === "process-start")).toHaveLength(relaunch ? 2 : 1);
expect(methods.filter(method => method === "process-start")).toHaveLength(1 + relaunches.length);
expect(methods.filter(method => method === "turn/start")).toHaveLength(adopted ? 3 : 2);
} else {
const journal = methods.map(line => JSON.parse(line));
expect(journal.filter(entry => entry.processStart)).toHaveLength(adopted || relaunch ? 4 : 3);
expect(journal.filter(entry => entry.processStart)).toHaveLength((adopted ? 4 : 3) + relaunches.length);
if (relaunch) {
const attach = journal.findIndex(entry => entry.command === "run.attach" && entry.params?.runId === runB);
const turn = journal.findIndex((entry, index) => index > attach && entry.command === "turn.start");
@@ -813,15 +813,17 @@ describe("plugin Runner relaunch as the claimant run", () => {
const spec = launchSpec();
return { ...spec, nativeRunner: { ...spec.nativeRunner!, identity: next, lifecyclePolicy: { mode: "warm", idleTimeoutMs: 60_000 } } };
};
async function relaunchOwner(resume: (operation: any) => unknown, options: { resumeIdentityVersion?: 1; onResumed?: () => Promise<void> } = { resumeIdentityVersion: 1 }) {
async function relaunchOwner(resume: (operation: any) => unknown,
options: { resumeIdentityVersion?: 1; onResumed?: () => Promise<void>; fresh?: true } = { resumeIdentityVersion: 1 }) {
const { task } = provider();
const perform = task.getMockImplementation()!;
task.mockImplementation(async operation => operation.kind === "resume" ? resume(operation) : perform(operation));
const close = vi.fn(async () => {});
ingress.connect.mockReset().mockReturnValue({ ready: Promise.resolve(), quiesce: vi.fn(), close });
const owner = createPluginRunnerTaskOwner({ ...base, task, adoptIdentity: source, authorityRunId: source.runId });
owner.rebind({ ...base, task, authorityRunId: next.runId, resumeRelaunchedRunnerVersion: 1,
resumeIdentityVersion: options.resumeIdentityVersion, relaunch: { source, runner: parked }, onResumed: options.onResumed });
const relaunchInput = { ...base, task, authorityRunId: next.runId, resumeRelaunchedRunnerVersion: 1 as const,
resumeIdentityVersion: options.resumeIdentityVersion, relaunch: { source, runner: parked }, onResumed: options.onResumed };
const owner = createPluginRunnerTaskOwner(options.fresh ? relaunchInput : { ...base, task, adoptIdentity: source, authorityRunId: source.runId });
if (!options.fresh) owner.rebind(relaunchInput);
await prepare(owner, { ...requirements, identity: next });
const authority = { acceptRelaunchedRunner: vi.fn() };
const registration = await owner.registration(authority as never, next.runId, undefined, next);
@@ -855,6 +857,30 @@ describe("plugin Runner relaunch as the claimant run", () => {
} finally { await registration.release("detach"); }
});
it("relaunches through a new owner, as after a Core restart, without submitting or adopting", async () => {
const { owner, task, authority, registration } = await relaunchOwner(
() => ({ kind: "resume", taskId: "task", outcome: "resumed", runner: build, archivedState: settledCut }),
{ resumeIdentityVersion: 1, fresh: true });
expect(task.mock.calls.find(([operation]) => operation.kind === "prepare")?.[0]).toMatchObject({ priorIdentity: source });
expect(owner.driver.relaunchReleasedRunner).toEqual(source);
expect(owner.driver.adoptExistingRunner).toBeUndefined();
owner.launcher(relaunchSpec());
registration.activate();
try {
await registration.ready();
expect(task.mock.calls.filter(([operation]) => operation.kind === "resume").map(([operation]) => operation.runner))
.toEqual([{ runId: next.runId, turnId: next.turnId, itemId: next.itemId }]);
expect(task.mock.calls.some(([operation]) => operation.kind === "submit")).toBe(false);
expect(authority.acceptRelaunchedRunner).toHaveBeenCalledExactlyOnceWith("synthetic-ticket", build, parked);
} finally { await registration.release("detach"); }
});
it("treats a new owner as a new task when the provider does not advertise relaunch", async () => {
const { owner, task } = await relaunchOwner(() => { throw new Error("unexpected resume"); }, { fresh: true });
expect(owner.driver.relaunchReleasedRunner).toBeUndefined();
expect(task.mock.calls.find(([operation]) => operation.kind === "prepare")?.[0]).not.toHaveProperty("priorIdentity");
});
it("retries a lost reply with the same ticket and identity", async () => {
let attempts = 0;
const { task, registration, owner } = await relaunchOwner(() => {
@@ -76,12 +76,13 @@ export function createPluginRunnerTaskOwner(input: {
let relaunchProgress: "none" | "sent" | "refused" = "none";
let activeOutbound: ReturnType<typeof connectRunnerPrpIngress> | null = null;
let preparedLaunchId: string | undefined;
let preparedIdentity: DurableRecoveryIdentity | undefined = input.adoptIdentity;
let launched = !!input.adoptIdentity;
let launchIdentity = input.adoptIdentity;
const relaunchedTask = !input.adoptIdentity && input.resumeIdentityVersion === 1 ? input.relaunch?.source : undefined;
let preparedIdentity: DurableRecoveryIdentity | undefined = input.adoptIdentity ?? relaunchedTask;
let launched = !!(input.adoptIdentity ?? relaunchedTask);
let launchIdentity = input.adoptIdentity ?? relaunchedTask;
let stopRequested = false;
let released = false;
let submitted = !!input.adoptIdentity;
let submitted = launched;
let detached = false;
let adoptionVerified = false;
let markReady!: () => void;
@@ -102,7 +103,7 @@ export function createPluginRunnerTaskOwner(input: {
if (input.adoptIdentity) markReady();
const configuredEnvironmentFields = (launchesRunner: boolean) =>
input.configuredEnvironmentVersion === 1 && launchesRunner ? { configuredEnvironment: { ...input.configuredEnvironment } } : {};
let configuredEnvironmentDelivered: boolean | undefined = input.adoptIdentity ? undefined : false;
let configuredEnvironmentDelivered: boolean | undefined = launched ? undefined : false;
const call = async (operation: PluginEnvironmentTaskOperation) => {
const phase = ({ prepare: "execution.plugin_prepare", check_project_auth: "execution.plugin_project_auth",
submit: "execution.plugin_submit", connection: "execution.plugin_connection" } as const)[operation.kind as "prepare" | "check_project_auth" | "submit" | "connection"];