Files
PaperClipAI/packages/plugins/sdk/tests/environment-task-release-resume.test.ts
T
Nicky LeachandPaperclip a924d2e2c7 Persist portable Runner ownership through controller recovery (#21066)
## Thinking Path

> - Paperclip delegates remote Runner execution to environment plugins.
> - A plugin can move a session to another task allocation.
> - The Core controller can lose an ownership reply or restart before
the Runner starts.
> - The source and target need a durable record of the same request.
> - This change binds restart and attachment to that request and the
original unlaunched task lease.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Recovery of plugin Runner sessions on replacement compute.

**Current behavior**
Core cannot replay a session ownership request from saved lease state
during startup.

**Proposed behavior**
Record the operation on both leases and check the matched ownership
receipt before attaching. Preserve the original pre-admission lease
across controller restart.

Managed environment configuration is in
[#21065](https://github.com/paperclipai/paperclip/pull/21065). Related
warm sandbox cleanup is in
[#20065](https://github.com/paperclipai/paperclip/pull/20065). This
layer preserves the same-sandbox fallback from
[#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace
placement follows in
[#20743](https://github.com/paperclipai/paperclip/pull/20743).

## What Changed

- Add the optional `sessionHandoffVersion` capability and the ownership
task operation.
- Save matching ownership claims and fence source checkpoint access.
Replay a pending request on the original target.
- Restore a portable session using the durable claim and the current
credentials for the same account. Start a fresh conversation when the
provider reports an unavailable checkpoint, or when Core's persisted
compatibility decision selects a different session after transfer. The
latter requires an unlaunched target; subsequent recovery must match the
fresh task's saved identity. Both paths retain the workspace transfer
and source fencing.
- Reuse the original unlaunched task lease after hard or graceful
controller restart. After a move has attached, validate the surviving
Runner through the saved claim rather than requiring its logical lease
ID to equal the new physical lease.
- Keep older retained leases without a saved account identity on the
existing resume path.
- Apply the existing supported MCP-only compatibility rule to warm
recovery and predecessor accounting. Persist the capability discovered
at admission and reuse it after restart without repeating discovery.
- Keep predecessor-retirement fallback scoped to the same physical
sandbox. A refusal after a portable move preserves the session instead
of requesting a replacement that the plugin driver cannot enforce.

## Verification

- On `1eb527be27`, server typecheck (including Runner build and SDK
dependencies) passed.
- Independent final verification passed 168 tests across portable source
recovery, session resume, controller recovery, and task authority; two
existing tests were skipped. The full portable source integration suite
passed 64 tests.
- The new integration follows confirmed ownership through an
incompatible instruction change, persisted fresh admission, target-only
prepare/auth/submit without the source checkpoint, and reattachment
after controller replacement. It also rejects unpersisted session
selection, ambiguous target leases, and incorrect saved identities.
- Before this final change, independent verification passed 1,667 tests
across executor, session resume, recovery usage, remote warm recovery,
runtime selection, and runtime execution; the environment lifecycle
suite passed 208 tests.
- Existing regressions cover previous-release account metadata,
supported and unsupported MCP changes, provider/instruction
incompatibility, and admission capability reuse without another
discovery call.
- On `51d7eec373`, all four S3 multipart throttle tests passed
independently. CI exposed an existing timing-sensitive test twice; its
20 ms overlap assumption now uses synchronized retries and an event-loop
yield. Production upload code is unchanged. Temporary concurrency caps
of one and eight both fail the retained exact-four assertion.
- Exact SDK source-pin checks passed for both stack heads. The upper
#20743 code patch remains unchanged; its documentation preserves both
layers' behavior.

## Risks

- The handoff protocol is opt-in. Existing plugin leases without
portable ownership metadata keep the previous release's resume path.
Live attachment still requires end-to-end qualification. An unverified
claim blocks startup. Git workspaces prepared on a portable allocation
also need the workspace placement layer.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 10:43:48 -08:00

131 lines
7.5 KiB
TypeScript

import { createInterface } from "node:readline";
import { PassThrough } from "node:stream";
import { describe, expect, it } from "vitest";
import { definePlugin } from "../src/define-plugin.js";
import {
environmentTaskOperationSchema,
environmentTaskResultSchema,
parseEnvironmentTaskResult,
type PluginEnvironmentTaskOperation,
type PluginEnvironmentTaskParams,
type PluginEnvironmentTaskResult,
} from "../src/environment-tasks.js";
import { createRequest, isJsonRpcResponse, parseMessage, serializeMessage, type JsonRpcResponse } from "../src/protocol.js";
import { startWorkerRpcHost } from "../src/worker-rpc-host.js";
const identity = { runnerInstanceId: "runner", environmentLeaseId: "lease", runId: "source-run",
normalizedSessionId: "session", turnId: "turn", itemId: "item" };
const release = { kind: "release", identity } as const;
const resume = { kind: "resume", identity, bootstrapTicket: "fixture-ticket" } as const;
const ownership = {
kind: "ensure_ownership", operationId: "9df4526f-36c5-4fba-9cf6-3c4fd317f531", identity,
sourceTaskId: "source-task", sourceGeneration: "source-generation", targetGeneration: "target-generation", policy: "reclaim_if_needed",
} as const;
const description = { kind: "description", taskId: "task", runner: { version: "fixture", digest: `sha256:${"a".repeat(64)}` } };
async function invoke(operation: unknown, handler: (params: PluginEnvironmentTaskParams) => Promise<PluginEnvironmentTaskResult>) {
const stdin = new PassThrough(), stdout = new PassThrough(), lines = createInterface({ input: stdout });
const worker = startWorkerRpcHost({ stdin, stdout, plugin: definePlugin({ async setup() {}, onEnvironmentTask: handler }) });
try {
const response = new Promise<JsonRpcResponse>(resolve => lines.on("line", line => {
const message = parseMessage(line);
if (isJsonRpcResponse(message)) resolve(message);
}));
stdin.write(serializeMessage(createRequest("environmentTask", { companyId: "company", taskId: "task",
runId: "claimant-run", lease: { providerLeaseId: "task" }, operation }, "lifecycle")));
return await response;
} finally { worker.stop(); lines.close(); stdin.destroy(); stdout.destroy(); }
}
describe("release and resume task contract", () => {
it("keeps the capability optional and independent of warm and source recovery support", () => {
for (const capabilities of [{}, { releaseResumeVersion: 1 }, { warmTaskVersion: 1 },
{ codexSourceRecoveryVersion: 1 }, { acpxSourceRecoveryVersion: 1 }]) {
expect(environmentTaskResultSchema.parse({ ...description, ...capabilities })).toEqual({ ...description, ...capabilities });
}
expect(environmentTaskResultSchema.safeParse({ ...description, releaseResumeVersion: 2 }).success).toBe(false);
});
it.each([release, resume])("requires the exact durable identity for $kind", operation => {
expect(environmentTaskOperationSchema.parse(operation)).toEqual(operation);
for (const field of Object.keys(identity)) {
expect(environmentTaskOperationSchema.safeParse({ ...operation, identity: { ...identity, [field]: undefined } }).success).toBe(false);
}
expect(environmentTaskOperationSchema.safeParse({ ...operation, identity: { ...identity, targetRunId: "other" } }).success).toBe(false);
expect(environmentTaskOperationSchema.safeParse({ ...operation, cancelActiveWork: true }).success).toBe(false);
});
it.each([undefined, "", "\0", "x".repeat(65_537)])("rejects a missing or invalid resume ticket", bootstrapTicket => {
expect(environmentTaskOperationSchema.safeParse({ ...resume, bootstrapTicket }).success).toBe(false);
});
it("matches each operation to its own closed result and task", () => {
const released = { kind: "released", taskId: "task" };
expect(parseEnvironmentTaskResult(release, "task", released)).toEqual(released);
for (const outcome of ["resumed", "unavailable"]) {
const result = { kind: "resume", taskId: "task", outcome };
expect(parseEnvironmentTaskResult(resume, "task", result)).toEqual(result);
expect(() => parseEnvironmentTaskResult(release, "task", result)).toThrow();
}
for (const operation of [release, resume]) {
expect(() => parseEnvironmentTaskResult(operation, "task", { kind: "accepted", taskId: "task" })).toThrow();
const result = operation.kind === "release" ? released : { kind: "resume", taskId: "task", outcome: "resumed" };
expect(() => parseEnvironmentTaskResult(operation, "other-task", result)).toThrow();
}
expect(() => parseEnvironmentTaskResult(resume, "task", { kind: "resume", taskId: "task", outcome: "unknown" })).toThrow();
expect(environmentTaskResultSchema.safeParse({ ...released, retained: true }).success).toBe(false);
expect(environmentTaskResultSchema.safeParse({ kind: "resume", taskId: "task", outcome: "resumed", bootstrapTicket: "fixture-ticket" }).success).toBe(false);
});
it.each([release, resume])("dispatches $kind through the serialized worker RPC", async operation => {
const result: PluginEnvironmentTaskResult = operation.kind === "release" ? { kind: "released", taskId: "task" }
: { kind: "resume", taskId: "task", outcome: "resumed" };
const response = await invoke(operation, async params => {
expect(params.operation).toEqual(operation);
expect(params.runId).toBe("claimant-run");
return result;
});
expect(response).toMatchObject({ result });
});
it("passes known absence through RPC without treating it as a successful reservation", async () => {
expect(await invoke(resume, async () => ({ kind: "resume", taskId: "task", outcome: "unavailable" })))
.toMatchObject({ result: { kind: "resume", taskId: "task", outcome: "unavailable" } });
});
it("binds a declarative ownership response to the exact worker request", async () => {
for (const status of ["pending", "ready"] as const) {
const result = { kind: "ownership", status, taskId: "task", operationId: ownership.operationId,
targetGeneration: ownership.targetGeneration } as const;
expect(await invoke(ownership, async params => {
expect(params.operation).toEqual(ownership);
return result;
})).toMatchObject({ result });
for (const changes of [
{ taskId: "another-task" },
{ operationId: "1f083ad0-a357-4f14-ad46-e9df981845b5" },
{ targetGeneration: "stale-generation" },
]) {
expect(await invoke(ownership, async () => ({ ...result, ...changes }))).toHaveProperty("error");
}
}
});
it.each(["source mismatch", "claim conflict", "authentication failed", "transport timeout"])("keeps %s as an RPC error", async message => {
expect(await invoke(resume, async () => { throw new Error(message); })).toHaveProperty("error");
});
it("rejects cross-task and wrong-kind replies at the worker boundary", async () => {
for (const result of [{ kind: "resume", taskId: "other", outcome: "resumed" }, { kind: "accepted", taskId: "task" }] as PluginEnvironmentTaskResult[]) {
expect(await invoke(resume, async () => result)).toHaveProperty("error");
}
});
it("preserves legacy describe and stop replies through RPC", async () => {
expect(await invoke({ kind: "describe" }, async () => description as PluginEnvironmentTaskResult)).toMatchObject({ result: description });
const operation: PluginEnvironmentTaskOperation = { kind: "stop" };
expect(await invoke(operation, async () => ({ kind: "accepted", taskId: "task" })))
.toMatchObject({ result: { kind: "accepted", taskId: "task" } });
});
});