mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 18:26:40 +01:00
## Thinking Path > - Paperclip delegates remote Runner execution to environment plugins. > - A plugin can move a session to another task allocation. > - The Core controller can lose an ownership reply or restart before the Runner starts. > - The source and target need a durable record of the same request. > - This change binds restart and attachment to that request and the original unlaunched task lease. ## Linked Issues or Issue Description **What existing behavior does this improve?** Recovery of plugin Runner sessions on replacement compute. **Current behavior** Core cannot replay a session ownership request from saved lease state during startup. **Proposed behavior** Record the operation on both leases and check the matched ownership receipt before attaching. Preserve the original pre-admission lease across controller restart. Managed environment configuration is in [#21065](https://github.com/paperclipai/paperclip/pull/21065). Related warm sandbox cleanup is in [#20065](https://github.com/paperclipai/paperclip/pull/20065). This layer preserves the same-sandbox fallback from [#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace placement follows in [#20743](https://github.com/paperclipai/paperclip/pull/20743). ## What Changed - Add the optional `sessionHandoffVersion` capability and the ownership task operation. - Save matching ownership claims and fence source checkpoint access. Replay a pending request on the original target. - Restore a portable session using the durable claim and the current credentials for the same account. Start a fresh conversation when the provider reports an unavailable checkpoint, or when Core's persisted compatibility decision selects a different session after transfer. The latter requires an unlaunched target; subsequent recovery must match the fresh task's saved identity. Both paths retain the workspace transfer and source fencing. - Reuse the original unlaunched task lease after hard or graceful controller restart. After a move has attached, validate the surviving Runner through the saved claim rather than requiring its logical lease ID to equal the new physical lease. - Keep older retained leases without a saved account identity on the existing resume path. - Apply the existing supported MCP-only compatibility rule to warm recovery and predecessor accounting. Persist the capability discovered at admission and reuse it after restart without repeating discovery. - Keep predecessor-retirement fallback scoped to the same physical sandbox. A refusal after a portable move preserves the session instead of requesting a replacement that the plugin driver cannot enforce. ## Verification - On `1eb527be27`, server typecheck (including Runner build and SDK dependencies) passed. - Independent final verification passed 168 tests across portable source recovery, session resume, controller recovery, and task authority; two existing tests were skipped. The full portable source integration suite passed 64 tests. - The new integration follows confirmed ownership through an incompatible instruction change, persisted fresh admission, target-only prepare/auth/submit without the source checkpoint, and reattachment after controller replacement. It also rejects unpersisted session selection, ambiguous target leases, and incorrect saved identities. - Before this final change, independent verification passed 1,667 tests across executor, session resume, recovery usage, remote warm recovery, runtime selection, and runtime execution; the environment lifecycle suite passed 208 tests. - Existing regressions cover previous-release account metadata, supported and unsupported MCP changes, provider/instruction incompatibility, and admission capability reuse without another discovery call. - On `51d7eec373`, all four S3 multipart throttle tests passed independently. CI exposed an existing timing-sensitive test twice; its 20 ms overlap assumption now uses synchronized retries and an event-loop yield. Production upload code is unchanged. Temporary concurrency caps of one and eight both fail the retained exact-four assertion. - Exact SDK source-pin checks passed for both stack heads. The upper #20743 code patch remains unchanged; its documentation preserves both layers' behavior. ## Risks - The handoff protocol is opt-in. Existing plugin leases without portable ownership metadata keep the previous release's resume path. Live attachment still requires end-to-end qualification. An unverified claim blocks startup. Git workspaces prepared on a portable allocation also need the workspace placement layer. ## Model Used - OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and independent review. ## Checklist - [x] Thinking path and model are included. - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] The issue is described using the enhancement template. - [x] Local checks passed and new behavior has tests. - [x] All Paperclip CI gates are green. - [x] Greptile is 5/5 with no open findings. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
131 lines
7.5 KiB
TypeScript
131 lines
7.5 KiB
TypeScript
import { createInterface } from "node:readline";
|
|
import { PassThrough } from "node:stream";
|
|
import { describe, expect, it } from "vitest";
|
|
import { definePlugin } from "../src/define-plugin.js";
|
|
import {
|
|
environmentTaskOperationSchema,
|
|
environmentTaskResultSchema,
|
|
parseEnvironmentTaskResult,
|
|
type PluginEnvironmentTaskOperation,
|
|
type PluginEnvironmentTaskParams,
|
|
type PluginEnvironmentTaskResult,
|
|
} from "../src/environment-tasks.js";
|
|
import { createRequest, isJsonRpcResponse, parseMessage, serializeMessage, type JsonRpcResponse } from "../src/protocol.js";
|
|
import { startWorkerRpcHost } from "../src/worker-rpc-host.js";
|
|
|
|
const identity = { runnerInstanceId: "runner", environmentLeaseId: "lease", runId: "source-run",
|
|
normalizedSessionId: "session", turnId: "turn", itemId: "item" };
|
|
const release = { kind: "release", identity } as const;
|
|
const resume = { kind: "resume", identity, bootstrapTicket: "fixture-ticket" } as const;
|
|
const ownership = {
|
|
kind: "ensure_ownership", operationId: "9df4526f-36c5-4fba-9cf6-3c4fd317f531", identity,
|
|
sourceTaskId: "source-task", sourceGeneration: "source-generation", targetGeneration: "target-generation", policy: "reclaim_if_needed",
|
|
} as const;
|
|
const description = { kind: "description", taskId: "task", runner: { version: "fixture", digest: `sha256:${"a".repeat(64)}` } };
|
|
|
|
async function invoke(operation: unknown, handler: (params: PluginEnvironmentTaskParams) => Promise<PluginEnvironmentTaskResult>) {
|
|
const stdin = new PassThrough(), stdout = new PassThrough(), lines = createInterface({ input: stdout });
|
|
const worker = startWorkerRpcHost({ stdin, stdout, plugin: definePlugin({ async setup() {}, onEnvironmentTask: handler }) });
|
|
try {
|
|
const response = new Promise<JsonRpcResponse>(resolve => lines.on("line", line => {
|
|
const message = parseMessage(line);
|
|
if (isJsonRpcResponse(message)) resolve(message);
|
|
}));
|
|
stdin.write(serializeMessage(createRequest("environmentTask", { companyId: "company", taskId: "task",
|
|
runId: "claimant-run", lease: { providerLeaseId: "task" }, operation }, "lifecycle")));
|
|
return await response;
|
|
} finally { worker.stop(); lines.close(); stdin.destroy(); stdout.destroy(); }
|
|
}
|
|
|
|
describe("release and resume task contract", () => {
|
|
it("keeps the capability optional and independent of warm and source recovery support", () => {
|
|
for (const capabilities of [{}, { releaseResumeVersion: 1 }, { warmTaskVersion: 1 },
|
|
{ codexSourceRecoveryVersion: 1 }, { acpxSourceRecoveryVersion: 1 }]) {
|
|
expect(environmentTaskResultSchema.parse({ ...description, ...capabilities })).toEqual({ ...description, ...capabilities });
|
|
}
|
|
expect(environmentTaskResultSchema.safeParse({ ...description, releaseResumeVersion: 2 }).success).toBe(false);
|
|
});
|
|
|
|
it.each([release, resume])("requires the exact durable identity for $kind", operation => {
|
|
expect(environmentTaskOperationSchema.parse(operation)).toEqual(operation);
|
|
for (const field of Object.keys(identity)) {
|
|
expect(environmentTaskOperationSchema.safeParse({ ...operation, identity: { ...identity, [field]: undefined } }).success).toBe(false);
|
|
}
|
|
expect(environmentTaskOperationSchema.safeParse({ ...operation, identity: { ...identity, targetRunId: "other" } }).success).toBe(false);
|
|
expect(environmentTaskOperationSchema.safeParse({ ...operation, cancelActiveWork: true }).success).toBe(false);
|
|
});
|
|
|
|
it.each([undefined, "", "\0", "x".repeat(65_537)])("rejects a missing or invalid resume ticket", bootstrapTicket => {
|
|
expect(environmentTaskOperationSchema.safeParse({ ...resume, bootstrapTicket }).success).toBe(false);
|
|
});
|
|
|
|
it("matches each operation to its own closed result and task", () => {
|
|
const released = { kind: "released", taskId: "task" };
|
|
expect(parseEnvironmentTaskResult(release, "task", released)).toEqual(released);
|
|
for (const outcome of ["resumed", "unavailable"]) {
|
|
const result = { kind: "resume", taskId: "task", outcome };
|
|
expect(parseEnvironmentTaskResult(resume, "task", result)).toEqual(result);
|
|
expect(() => parseEnvironmentTaskResult(release, "task", result)).toThrow();
|
|
}
|
|
for (const operation of [release, resume]) {
|
|
expect(() => parseEnvironmentTaskResult(operation, "task", { kind: "accepted", taskId: "task" })).toThrow();
|
|
const result = operation.kind === "release" ? released : { kind: "resume", taskId: "task", outcome: "resumed" };
|
|
expect(() => parseEnvironmentTaskResult(operation, "other-task", result)).toThrow();
|
|
}
|
|
expect(() => parseEnvironmentTaskResult(resume, "task", { kind: "resume", taskId: "task", outcome: "unknown" })).toThrow();
|
|
expect(environmentTaskResultSchema.safeParse({ ...released, retained: true }).success).toBe(false);
|
|
expect(environmentTaskResultSchema.safeParse({ kind: "resume", taskId: "task", outcome: "resumed", bootstrapTicket: "fixture-ticket" }).success).toBe(false);
|
|
});
|
|
|
|
it.each([release, resume])("dispatches $kind through the serialized worker RPC", async operation => {
|
|
const result: PluginEnvironmentTaskResult = operation.kind === "release" ? { kind: "released", taskId: "task" }
|
|
: { kind: "resume", taskId: "task", outcome: "resumed" };
|
|
const response = await invoke(operation, async params => {
|
|
expect(params.operation).toEqual(operation);
|
|
expect(params.runId).toBe("claimant-run");
|
|
return result;
|
|
});
|
|
expect(response).toMatchObject({ result });
|
|
});
|
|
|
|
it("passes known absence through RPC without treating it as a successful reservation", async () => {
|
|
expect(await invoke(resume, async () => ({ kind: "resume", taskId: "task", outcome: "unavailable" })))
|
|
.toMatchObject({ result: { kind: "resume", taskId: "task", outcome: "unavailable" } });
|
|
});
|
|
|
|
it("binds a declarative ownership response to the exact worker request", async () => {
|
|
for (const status of ["pending", "ready"] as const) {
|
|
const result = { kind: "ownership", status, taskId: "task", operationId: ownership.operationId,
|
|
targetGeneration: ownership.targetGeneration } as const;
|
|
expect(await invoke(ownership, async params => {
|
|
expect(params.operation).toEqual(ownership);
|
|
return result;
|
|
})).toMatchObject({ result });
|
|
for (const changes of [
|
|
{ taskId: "another-task" },
|
|
{ operationId: "1f083ad0-a357-4f14-ad46-e9df981845b5" },
|
|
{ targetGeneration: "stale-generation" },
|
|
]) {
|
|
expect(await invoke(ownership, async () => ({ ...result, ...changes }))).toHaveProperty("error");
|
|
}
|
|
}
|
|
});
|
|
|
|
it.each(["source mismatch", "claim conflict", "authentication failed", "transport timeout"])("keeps %s as an RPC error", async message => {
|
|
expect(await invoke(resume, async () => { throw new Error(message); })).toHaveProperty("error");
|
|
});
|
|
|
|
it("rejects cross-task and wrong-kind replies at the worker boundary", async () => {
|
|
for (const result of [{ kind: "resume", taskId: "other", outcome: "resumed" }, { kind: "accepted", taskId: "task" }] as PluginEnvironmentTaskResult[]) {
|
|
expect(await invoke(resume, async () => result)).toHaveProperty("error");
|
|
}
|
|
});
|
|
|
|
it("preserves legacy describe and stop replies through RPC", async () => {
|
|
expect(await invoke({ kind: "describe" }, async () => description as PluginEnvironmentTaskResult)).toMatchObject({ result: description });
|
|
const operation: PluginEnvironmentTaskOperation = { kind: "stop" };
|
|
expect(await invoke(operation, async () => ({ kind: "accepted", taskId: "task" })))
|
|
.toMatchObject({ result: { kind: "accepted", taskId: "task" } });
|
|
});
|
|
});
|