mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-04 02:32:07 +01:00
feat(runner): define bounded startup lifetime contracts (#18490)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native sessions and plugin workers have separate startup lifetimes. > - A caller timeout does not prove that a worker or an accepted host write has stopped. > - Shared startup code needs an exact deadline and an actual settlement contract. > - This PR adds those contracts and tests them before any production caller uses them. > - The dependent integration in #18405 will use the contracts with verified cleanup. > - This split lets reviewers assess the lifetime rules separately from runtime activation. ## Linked Issues or Issue Description Refs #18405. This PR extracts the foundation from that change. The integration will depend on this PR. **Problem or motivation** A bounded startup must distinguish the caller's wait from actual worker completion. A late worker reply or an accepted host request can outlive the original RPC promise. A shorter timeout must not discard that responsibility or authorize a replacement run. **Proposed solution** Add one Core execution context, optional Runner startup deadlines, and worker settlement and retirement APIs. Test the contracts with real child-process IPC. Keep production context creation, transport projection, capability negotiation, durable cleanup, and Stop/recovery integration in #18405. **Alternatives considered** One combined PR mixed the lifetime contract with its activation and cleanup policy. Activating a shorter deadline before the complete transport and cleanup path is unsafe. Splitting only the SDK declarations would not provide a useful behavioral review boundary. **Roadmap alignment** This supports the existing plugin, sandbox execution, and bounded recovery capabilities. It adds no new product workflow, endpoint, migration, provider or runtime dependency. ## What Changed - Add `NativeExecutionContext`. It retains the selected input, binds one transport target, rejects work after cancellation or expiry, and tracks actual worker settlement. Expiry alone does not complete pending work. - Add worker startup groups. A caller timeout retains the group's responsibility. Retirement revokes obsolete invocation scopes, waits for accepted host handlers, and retires only the verified group's bookkeeping. Each cleanup wait is bounded; an unfinished handler remains pending for a later retry. - Add an optional Runner startup signal and absolute deadline for session open and retained attachment. Clean up abort listeners after synchronous failures. Existing callers keep their default timeout path. - Expose the remaining confirmed controller-lease budget without a new query or renewal timer. - Add optional SDK context and preparation-retirement fields. No production bridge sends or advertises them in this slice. - Keep the required lease mock update with the new `remainingMs()` contract so this foundation typechecks independently. Production activation files, including the native executor and environment task bridge, match the current master base. No production caller constructs `NativeExecutionContext` or passes the new Runner startup option. This PR adds no durable cleanup producer and claims **no standalone TTFT improvement**. The dependent integration must enable the context, transport and verified cleanup together. ## Verification Initial split verification on `8ebc8a2f9c6c02e90f920356138b3742a4067b28`, based on `9a3f395737e7a92faf99971459979061d423531a`: - From `server`: `../node_modules/.bin/vitest run src/services/native-runtime/native-execution-context.test.ts src/services/native-runtime/native-execution-context-worker.test.ts src/services/native-runtime/native-execution-lease.test.ts` — **17 passed**. Includes real child IPC, caller timeout with late reply, worker death, accepted host-handler draining, obsolete cleanup invocation rejection, unrelated work preservation, context scope, expiry, and confirmed lease budget. - From `packages/paperclip-runner`: `../../node_modules/.bin/vitest run src/native-session-runtime.test.ts` — **188 passed**. Includes existing callers without the new option and new synchronous-failure/expired-deadline cases for fresh and retained sessions. - Direct TypeScript checks for `packages/plugins/sdk/tsconfig.json` and `packages/paperclip-runner/tsconfig.json` — passed. Built only their shared/eval TypeScript prerequisites. - From `server`: `../node_modules/.bin/tsc --noEmit` — passed. - Checked the partition against the reviewed combined candidate. The twelve selected files match exactly. The thirteenth file carries only the required test-mock update. The production executor, bridge and target wiring remain equal to the base. - `git diff --check`, added-line secret/private-data scan, and added-code-comment scan — passed. No generated files or run artifacts are committed. - The first old-base split passed its focused tests but failed server typechecking because the unchanged executor test mock lacked `remainingMs()`. The original failure is retained. Moving that one-line mock update into this foundation resolved the split dependency; the final current-master checks above pass. - Full local CI suites, Rust builds, deployments and provider calls were not run, as requested. GitHub CI will provide full regression coverage. Current-head GitHub CI, standard review and Apex are complete; exact acceptance is recorded below. Review follow-up `d7635f4d280dab757a3666703737bcda955c9361` fixes three reproduced lifetime edge cases. A retiring startup group rejects new calls and shares one retirement promise; failed bounded retirement remains retryable. Transport identity uses a fixed field tuple rather than object-property order. Finishing a context rejects pending lookup waits without aborting an already-started backend or pretending the worker completed. All three new regressions failed before the fix; the updated foundation selection passes **20 tests**, including actual IPC and failed-retirement retry. Server typechecking and added-line checks pass. These replace the earlier 17-case selection, rather than adding 20 unique cases to it. The original failures and passing reruns are retained. Apex follow-up `cb8860bab5ab9d6357a352e11f276be0d0cdb26d` keeps removed worker handles reachable while they own startup calls, accepted host handlers or retirement work. Drain/retirement includes active and retained handles; completion callbacks remove settled handles. Stop and replacement remove only the exact handle they observed. This adds no database/network operation, polling loop or generic shutdown wait. The SDK, Runner, context and transport bridge are unchanged. Seven additional IPC regressions failed on the prior manager and pass after the fix, covering removal/replacement, Stop-all, host-handler settlement, bounded retirement failure/retry, pruning, and replacement ownership. The updated foundation selection passes **27 tests**, with server typechecking passing; these supersede the earlier 20-case selection. A separate real-PostgreSQL/child-IPC characterization uses the integration's actual SDK host, storage registry and run-control path with the exact corrected manager bytes. Before the fix, cleanup released run ownership while an accepted host write was blocked, then the late write restored obsolete metadata. After the fix, ownership and cleanup remain pending until the write completes; cleanup then removes the stale metadata before releasing ownership. The original failure, corrected result and source snapshots are preserved. Final head `cb8860bab5ab9d6357a352e11f276be0d0cdb26d` passes GitHub CI (53 successful checks, 2 expected skips, Snyk success), standard Greptile **5/5** and fresh Apex **5/5**. All review threads are resolved and the PR is conflict-free. ## Risks The shared Runner helper now observes synchronous operation failures through its asynchronous cleanup path. The affected runtime file's 188 tests pass, but full regression coverage still depends on CI. The new lifetime APIs require honest settlement signals. A future caller must not treat a deadline, worker crash or Stop intent as verified retirement. Production activation remains dependent on the complete integration and its cleanup tests in #18405. This foundation introduces no shorter production startup budget or new cleanup policy by itself. The context's transport projection is process-local. It must not be persisted or used as a general authorization cache. No production caller supplies it in this PR. There is no database migration, new dependency, or deployment requirement for merging this foundation. ## Model Used OpenAI Codex, based on GPT-6, assisted with repository inspection, the split, tests and command execution. The exact serving model ID and context-window size were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal issue identifier - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes — this description documents the inactive contracts; operational activation documentation remains with #18405 - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green on `cb8860bab5` - [x] Current-head standard Greptile and Apex are 5/5 with no unresolved findings - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Nicky LeachandPaperclip authored and GitHub committed
1 parent
412834a860
commit
f8a888a072
13 files changed
+762
-16
No files matched your search
@@ -0,0 +1,318 @@
|
||||
import { mkdtemp, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import type { PaperclipPluginManifestV1 } from "@paperclipai/shared";
|
||||
import { createHostClientHandlers } from "@paperclipai/plugin-sdk";
|
||||
import { createPluginWorkerManager } from "../plugin-worker-manager.js";
|
||||
import type { NativeExecutionInput } from "../../vendor/paperclip-runner/index.js";
|
||||
import { NativeExecutionContext } from "./native-execution-context.js";
|
||||
|
||||
const execution = { binding: { companyId: "company", agentId: "agent", runId: "run" } } as NativeExecutionInput;
|
||||
const manifest: PaperclipPluginManifestV1 = {
|
||||
id: "test.execution-context", apiVersion: 1, version: "1.0.0", displayName: "Execution context fixture",
|
||||
description: "Execution context fixture", author: "Paperclip", categories: ["automation"], capabilities: [],
|
||||
entrypoints: { worker: "worker.cjs" },
|
||||
};
|
||||
const params = {
|
||||
companyId: "company", agentId: "agent", runId: "run", environmentId: "environment", issueId: "issue",
|
||||
driverKey: "tasks", taskId: "task", config: {}, projectIds: [],
|
||||
lease: { providerLeaseId: "task" }, operation: { kind: "describe" as const },
|
||||
};
|
||||
|
||||
let directory: string;
|
||||
let entrypointPath: string;
|
||||
beforeAll(async () => {
|
||||
directory = await mkdtemp(path.join(tmpdir(), "paperclip-startup-worker-"));
|
||||
entrypointPath = path.join(directory, "worker.cjs");
|
||||
await writeFile(entrypointPath, `
|
||||
const readline = require("node:readline");
|
||||
const held = [];
|
||||
const invocations = [];
|
||||
const send = (id, result) => process.stdout.write(JSON.stringify({jsonrpc:"2.0",id,result})+"\\n");
|
||||
readline.createInterface({input:process.stdin,crlfDelay:Infinity}).on("line", line => {
|
||||
const message = JSON.parse(line);
|
||||
if (message.method === "initialize") send(message.id, {ok:true,supportedMethods:["environmentTask","configChanged","health"]});
|
||||
else if (message.method === "environmentTask" || message.method === "environmentDestroyLease") {
|
||||
held.push(message.id);
|
||||
invocations.push(message.paperclipInvocation?.id);
|
||||
if (message.params.config.hostWrite) process.stdout.write(JSON.stringify({jsonrpc:"2.0",id:"host-"+message.id,method:"companies.get",params:{companyId:"company"},paperclipInvocationId:message.paperclipInvocation.id})+"\\n");
|
||||
if (message.params.config.replyEarly) { held.splice(held.indexOf(message.id),1); send(message.id,{kind:"description",taskId:"task"}); }
|
||||
}
|
||||
else if (message.method === "health") send(message.id, {status:"ok",details:{pending:held.length}});
|
||||
else if (message.method === "configChanged") {
|
||||
if (message.params.config.exit) process.exit(1);
|
||||
if (message.params.config.hostAfter) process.stdout.write(JSON.stringify({jsonrpc:"2.0",id:"late-host",method:"companies.get",params:{companyId:"company"},paperclipInvocationId:invocations.at(-1)})+"\\n");
|
||||
if (message.params.config.release) for (const id of held.splice(0)) send(id, {kind:"description",taskId:"task"});
|
||||
send(message.id, {});
|
||||
} else if (message.method === "shutdown") {send(message.id, {});setImmediate(()=>process.exit(0));}
|
||||
});
|
||||
`, { mode: 0o600 });
|
||||
});
|
||||
afterAll(async () => { if (directory) await rm(directory, { recursive: true, force: true }); });
|
||||
|
||||
async function startWorker(hostHandlers: Record<string, unknown> = {}) {
|
||||
const manager = createPluginWorkerManager();
|
||||
await manager.startWorker(manifest.id, { entrypointPath, manifest, config: {},
|
||||
instanceInfo: { instanceId: "instance", hostVersion: "1.0.0" }, apiVersion: 1, hostHandlers: hostHandlers as never, autoRestart: false });
|
||||
return manager;
|
||||
}
|
||||
|
||||
describe("Core startup responsibility across real worker IPC", () => {
|
||||
it.each(["stop", "replace", "stopAll"])("keeps stopped startup work reachable through %s until verified retirement", async mode => {
|
||||
let release!: () => void, entered!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; }), started = new Promise<void>(resolve => { entered = resolve; });
|
||||
const manager = await startWorker({ "companies.get": async () => { entered(); await gate; return {}; } });
|
||||
const original = manager.getWorker(manifest.id)!, settled = vi.fn();
|
||||
try {
|
||||
const request = manager.call(manifest.id, "environmentTask", { ...params, config: { hostWrite: true } }, 50, undefined, settled, "stopped-context");
|
||||
const rejected = expect(request).rejects.toThrow("timed out");
|
||||
await started; await rejected;
|
||||
if (mode === "stop") await manager.stopWorker(manifest.id);
|
||||
else if (mode === "replace") await original.stop();
|
||||
else await manager.stopAll();
|
||||
if (mode !== "stopAll") await manager.startWorker(manifest.id, { entrypointPath, manifest, config: {},
|
||||
instanceInfo: { instanceId: "replacement", hostVersion: "1.0.0" }, apiVersion: 1, hostHandlers: {} });
|
||||
let drained = false, retired = false;
|
||||
const draining = manager.drainStartup!("stopped-context").then(() => { drained = true; });
|
||||
const retiring = manager.retireStartup!("stopped-context").then(() => { retired = true; });
|
||||
await new Promise<void>(resolve => setImmediate(resolve));
|
||||
expect(drained).toBe(false);
|
||||
expect(retired).toBe(false);
|
||||
expect(settled).not.toHaveBeenCalled();
|
||||
release(); await Promise.all([draining, retiring]);
|
||||
expect(settled).toHaveBeenCalledOnce();
|
||||
const oldDrain = vi.spyOn(original, "drainStartup");
|
||||
await manager.drainStartup!("stopped-context");
|
||||
expect(oldDrain).not.toHaveBeenCalled();
|
||||
} finally { release(); await manager.stopAll(); }
|
||||
});
|
||||
|
||||
it("prunes a stopped handle after its last accepted startup host call settles", async () => {
|
||||
let release!: () => void, entered!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; }), started = new Promise<void>(resolve => { entered = resolve; });
|
||||
const manager = await startWorker({ "companies.get": async () => { entered(); await gate; return {}; } });
|
||||
const original = manager.getWorker(manifest.id)!, settled = vi.fn();
|
||||
try {
|
||||
const request = manager.call(manifest.id, "environmentTask", { ...params, config: { hostWrite: true, replyEarly: true } }, 5_000, undefined, settled, "host-only-context");
|
||||
await started; await request;
|
||||
expect(settled).toHaveBeenCalledOnce();
|
||||
await manager.stopWorker(manifest.id);
|
||||
let drained = false;
|
||||
const draining = manager.drainStartup!("host-only-context").then(() => { drained = true; });
|
||||
await new Promise<void>(resolve => setImmediate(resolve));
|
||||
expect(drained).toBe(false);
|
||||
release(); await draining;
|
||||
const oldDrain = vi.spyOn(original, "drainStartup");
|
||||
await manager.drainStartup!("host-only-context");
|
||||
expect(oldDrain).not.toHaveBeenCalled();
|
||||
} finally { release(); await manager.stopAll(); }
|
||||
});
|
||||
|
||||
it("keeps a stopped handle owned after bounded retirement fails and permits verified retry", async () => {
|
||||
let release!: () => void, entered!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; }), started = new Promise<void>(resolve => { entered = resolve; });
|
||||
const manager = await startWorker({ "companies.get": async () => { entered(); await gate; return {}; } }), settled = vi.fn();
|
||||
try {
|
||||
const request = manager.call(manifest.id, "environmentTask", { ...params, config: { hostWrite: true } }, 50, undefined, settled, "stopped-retry");
|
||||
const rejected = expect(request).rejects.toThrow("timed out");
|
||||
await started; await rejected;
|
||||
await manager.stopWorker(manifest.id);
|
||||
vi.useFakeTimers();
|
||||
const failed = manager.retireStartup!("stopped-retry").catch(error => error.message);
|
||||
await vi.advanceTimersByTimeAsync(15_000);
|
||||
expect(await failed).toBe("Task cleanup is still pending");
|
||||
expect(settled).not.toHaveBeenCalled();
|
||||
vi.useRealTimers();
|
||||
let retired = false;
|
||||
const retry = manager.retireStartup!("stopped-retry").then(() => { retired = true; });
|
||||
await new Promise<void>(resolve => setImmediate(resolve));
|
||||
expect(retired).toBe(false);
|
||||
release(); await retry;
|
||||
expect(settled).toHaveBeenCalledOnce();
|
||||
} finally { vi.useRealTimers(); release(); await manager.stopAll(); }
|
||||
});
|
||||
|
||||
it.each(["stop", "stopAll"])("preserves a replacement registered while %s is finishing", async mode => {
|
||||
const manager = await startWorker(), original = manager.getWorker(manifest.id)!;
|
||||
let release!: () => void, entered!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; }), stopped = new Promise<void>(resolve => { entered = resolve; });
|
||||
const stop = original.stop.bind(original);
|
||||
let replacement: Awaited<ReturnType<typeof manager.startWorker>> | undefined;
|
||||
vi.spyOn(original, "stop").mockImplementation(async () => { await stop(); entered(); await gate; });
|
||||
try {
|
||||
const stopping = mode === "stop" ? manager.stopWorker(manifest.id) : manager.stopAll();
|
||||
await stopped;
|
||||
replacement = await manager.startWorker(manifest.id, { entrypointPath, manifest, config: {},
|
||||
instanceInfo: { instanceId: "replacement", hostVersion: "1.0.0" }, apiVersion: 1, hostHandlers: {} });
|
||||
release(); await stopping;
|
||||
expect(manager.getWorker(manifest.id)).toBe(replacement);
|
||||
expect(await manager.call(manifest.id, "health", {})).toMatchObject({ status: "ok" });
|
||||
} finally { release(); await replacement?.stop(); await manager.stopAll(); }
|
||||
});
|
||||
|
||||
it("refuses new calls while the same startup group retires and joins concurrent retirement", async () => {
|
||||
let release!: () => void, entered!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; }), started = new Promise<void>(resolve => { entered = resolve; });
|
||||
const write = vi.fn(async () => { entered(); await gate; return {}; });
|
||||
const handlers = createHostClientHandlers({ pluginId: manifest.id, capabilities: ["companies.read"], services: { companies: { get: write } } as never });
|
||||
const manager = await startWorker(handlers), firstSettled = vi.fn(), secondSettled = vi.fn();
|
||||
try {
|
||||
const request = manager.call(manifest.id, "environmentTask", { ...params, config: { hostWrite: true } }, 5_000, undefined, firstSettled, "retiring-context");
|
||||
void request.catch(() => undefined);
|
||||
await started;
|
||||
let retired = false;
|
||||
const retiring = manager.retireStartup!("retiring-context").then(() => { retired = true; });
|
||||
const joined = manager.retireStartup!("retiring-context");
|
||||
await expect(manager.call(manifest.id, "environmentTask", { ...params, config: { hostWrite: true } }, 100, undefined, secondSettled, "retiring-context"))
|
||||
.rejects.toThrow("retiring");
|
||||
expect(write).toHaveBeenCalledOnce();
|
||||
expect(firstSettled).not.toHaveBeenCalled();
|
||||
expect(secondSettled).toHaveBeenCalledOnce();
|
||||
expect(retired).toBe(false);
|
||||
expect(await manager.call(manifest.id, "health", {})).toMatchObject({ status: "ok" });
|
||||
release(); await Promise.all([retiring, joined]);
|
||||
expect(firstSettled).toHaveBeenCalledOnce();
|
||||
await manager.call(manifest.id, "configChanged", { companyId: "company", config: { release: true } });
|
||||
await request;
|
||||
expect(firstSettled).toHaveBeenCalledOnce();
|
||||
} finally { release(); await manager.stopAll(); }
|
||||
});
|
||||
|
||||
it("revokes a cleanup invocation created after the initial drain and waits for its accepted host handler", async () => {
|
||||
let release!: () => void, entered!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; }), started = new Promise<void>(resolve => { entered = resolve; });
|
||||
const write = vi.fn(async () => { entered(); await gate; return {}; });
|
||||
const handlers = createHostClientHandlers({ pluginId: manifest.id, capabilities: ["companies.read"], services: { companies: { get: write } } as never });
|
||||
const manager = await startWorker(handlers), settled = vi.fn();
|
||||
try {
|
||||
await manager.drainStartup!("cleanup-context");
|
||||
const request = manager.call(manifest.id, "environmentDestroyLease", { companyId: "company", environmentId: "environment", issueId: "issue", driverKey: "tasks",
|
||||
config: { hostWrite: true }, providerLeaseId: "task", leaseMetadata: { runnerTaskStartupCleanup: { contextId: "cleanup-context", requiresPreparationStop: true } } },
|
||||
5_000, undefined, settled);
|
||||
await started;
|
||||
let retired = false;
|
||||
const retiring = manager.retireStartup!("cleanup-context").then(() => { retired = true; });
|
||||
await manager.call(manifest.id, "configChanged", { companyId: "company", config: { hostAfter: true } });
|
||||
expect(write).toHaveBeenCalledOnce();
|
||||
expect(retired).toBe(false);
|
||||
release(); await retiring;
|
||||
expect(settled).toHaveBeenCalledOnce();
|
||||
await manager.call(manifest.id, "configChanged", { companyId: "company", config: { release: true } });
|
||||
await request;
|
||||
expect(settled).toHaveBeenCalledOnce();
|
||||
expect(write).toHaveBeenCalledOnce();
|
||||
} finally { release(); await manager.stopAll(); }
|
||||
});
|
||||
it("bounds the cleanup wait without discarding an accepted host handler", async () => {
|
||||
let release!: () => void, entered!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; }), started = new Promise<void>(resolve => { entered = resolve; });
|
||||
const manager = await startWorker({ "companies.get": async () => { entered(); await gate; return {}; } }), settled = vi.fn();
|
||||
try {
|
||||
const request = manager.call(manifest.id, "environmentTask", { ...params, config: { hostWrite: true } }, 100, undefined, settled, "bounded-context");
|
||||
const rejected = expect(request).rejects.toThrow("timed out");
|
||||
await started; await rejected;
|
||||
vi.useFakeTimers();
|
||||
const waiting = manager.retireStartup!("bounded-context");
|
||||
const pending = expect(waiting).rejects.toThrow("cleanup is still pending");
|
||||
await vi.advanceTimersByTimeAsync(15_000); await pending;
|
||||
expect(settled).not.toHaveBeenCalled();
|
||||
vi.useRealTimers();
|
||||
const retried = vi.fn();
|
||||
const retry = manager.call(manifest.id, "environmentTask", params, 5_000, undefined, retried, "bounded-context");
|
||||
void retry.catch(() => undefined);
|
||||
expect(await manager.call(manifest.id, "health", {})).toMatchObject({ details: { pending: 2 } });
|
||||
expect(retried).not.toHaveBeenCalled();
|
||||
release();
|
||||
await manager.drainStartup!("bounded-context");
|
||||
await manager.retireStartup!("bounded-context");
|
||||
expect(settled).toHaveBeenCalledOnce();
|
||||
expect(retried).toHaveBeenCalledOnce();
|
||||
await manager.call(manifest.id, "configChanged", { companyId: "company", config: { release: true } });
|
||||
await retry;
|
||||
} finally { vi.useRealTimers(); release(); await manager.stopAll(); }
|
||||
});
|
||||
it.each(["startup", "cleanup"])("drains accepted %s host writes before retiring only verified bookkeeping", async mode => {
|
||||
let release!: () => void, entered!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; }), started = new Promise<void>(resolve => { entered = resolve; });
|
||||
const manager = await startWorker({ "companies.get": async () => { entered(); await gate; return {}; } });
|
||||
const verified = vi.fn(), unrelated = vi.fn();
|
||||
try {
|
||||
const request = mode === "startup"
|
||||
? manager.call(manifest.id, "environmentTask", { ...params, config: { hostWrite: true } }, 100, undefined, verified, "verified-context")
|
||||
: manager.call(manifest.id, "environmentDestroyLease", { companyId: "company", environmentId: "environment", issueId: "issue", driverKey: "tasks",
|
||||
config: { hostWrite: true }, providerLeaseId: "task", leaseMetadata: { runnerTaskStartupCleanup: { contextId: "verified-context", requiresPreparationStop: true } } },
|
||||
100, undefined, verified);
|
||||
const rejected = expect(request).rejects.toThrow("timed out");
|
||||
await started;
|
||||
await rejected;
|
||||
await expect(manager.call(manifest.id, "environmentTask", params, 10, undefined, unrelated, "other-context")).rejects.toThrow("timed out");
|
||||
let drained = false;
|
||||
const drain = manager.drainStartup!("verified-context").then(() => { drained = true; });
|
||||
await manager.call(manifest.id, "health", {});
|
||||
expect(drained).toBe(false);
|
||||
expect(verified).not.toHaveBeenCalled();
|
||||
release(); await drain;
|
||||
expect(verified).not.toHaveBeenCalled();
|
||||
await manager.retireStartup!("verified-context");
|
||||
expect(verified).toHaveBeenCalledOnce();
|
||||
expect(unrelated).not.toHaveBeenCalled();
|
||||
await manager.call(manifest.id, "configChanged", { companyId: "company", config: { release: true } });
|
||||
expect(verified).toHaveBeenCalledOnce();
|
||||
expect(unrelated).toHaveBeenCalledOnce();
|
||||
} finally { release(); await manager.stopAll(); }
|
||||
});
|
||||
|
||||
it("can retire a verified startup after worker death without declaring unrelated work settled", async () => {
|
||||
const manager = await startWorker(), verified = vi.fn(), unrelated = vi.fn();
|
||||
try {
|
||||
await expect(manager.call(manifest.id, "environmentTask", params, 10, undefined, verified, "verified-context")).rejects.toThrow("timed out");
|
||||
await expect(manager.call(manifest.id, "environmentTask", params, 10, undefined, unrelated, "other-context")).rejects.toThrow("timed out");
|
||||
await expect(manager.call(manifest.id, "configChanged", { companyId: "company", config: { exit: true } })).rejects.toThrow("exited");
|
||||
expect(verified).not.toHaveBeenCalled();
|
||||
await manager.drainStartup!("verified-context");
|
||||
await manager.retireStartup!("verified-context");
|
||||
expect(verified).toHaveBeenCalledOnce();
|
||||
expect(unrelated).not.toHaveBeenCalled();
|
||||
} finally { await manager.stopAll(); }
|
||||
});
|
||||
it("keeps a timed-out caller pending until the actual late worker reply", async () => {
|
||||
const manager = await startWorker();
|
||||
const context = new NativeExecutionContext(execution, 5_000);
|
||||
const settled = vi.fn(context.trackWorker());
|
||||
try {
|
||||
await expect(manager.call(manifest.id, "environmentTask", params, 10, undefined, settled)).rejects.toThrow("timed out");
|
||||
expect(context.pending).toBe(true);
|
||||
expect(settled).not.toHaveBeenCalled();
|
||||
context.abort(new Error("Stop requested"));
|
||||
let drained = false;
|
||||
const drain = context.drain().then(result => { drained = true; return result; });
|
||||
expect(await manager.call(manifest.id, "health", {})).toMatchObject({ details: { pending: 1 } });
|
||||
expect(drained).toBe(false);
|
||||
await manager.call(manifest.id, "configChanged", { companyId: "company", config: { release: true } });
|
||||
expect(await drain).toBe(true);
|
||||
expect(context.pending).toBe(false);
|
||||
expect(settled).toHaveBeenCalledOnce();
|
||||
expect(() => context.assertOpen()).toThrow("Stop requested");
|
||||
} finally { context.finish(); await manager.stopAll(); }
|
||||
});
|
||||
|
||||
it("does not treat the startup deadline as a worker completion receipt", async () => {
|
||||
const manager = await startWorker();
|
||||
const context = new NativeExecutionContext(execution, 100);
|
||||
const settled = vi.fn(context.trackWorker());
|
||||
try {
|
||||
await expect(manager.call(manifest.id, "environmentTask", params, 10, undefined, settled)).rejects.toThrow("timed out");
|
||||
if (!context.signal.aborted) await new Promise<void>(resolve => context.signal.addEventListener("abort", () => resolve(), { once: true }));
|
||||
expect(await context.drain()).toBe(false);
|
||||
expect(context.pending).toBe(true);
|
||||
expect(settled).not.toHaveBeenCalled();
|
||||
expect(await manager.call(manifest.id, "health", {})).toMatchObject({ details: { pending: 1 } });
|
||||
await manager.call(manifest.id, "configChanged", { companyId: "company", config: { release: true } });
|
||||
expect(settled).toHaveBeenCalledOnce();
|
||||
expect(context.pending).toBe(false);
|
||||
expect(await context.drain()).toBe(true);
|
||||
expect(() => context.assertOpen()).toThrow("startup_expired");
|
||||
} finally { context.finish(); await manager.stopAll(); }
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,106 @@
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import type { NativeExecutionInput } from "../../vendor/paperclip-runner/index.js";
|
||||
import { NativeExecutionContext } from "./native-execution-context.js";
|
||||
|
||||
const execution = { binding: { companyId: "company", agentId: "agent", runId: "run" } } as NativeExecutionInput;
|
||||
const target = { pluginId: "plugin", leaseId: "lease", taskId: "task", environmentId: "environment" };
|
||||
|
||||
afterEach(() => vi.useRealTimers());
|
||||
|
||||
describe("Core execution startup context", () => {
|
||||
it("retains the selected input and resolves one exact transport target per context", async () => {
|
||||
const context = new NativeExecutionContext(execution, 1_000);
|
||||
const resolve = vi.fn(async () => ({ instanceSecrets: { binding: "synthetic" }, configuration: {} }));
|
||||
try {
|
||||
expect(context.execution).toBe(execution);
|
||||
context.assertScope(execution.binding);
|
||||
expect(() => context.assertScope({ ...execution.binding, runId: "other" })).toThrow("scope_mismatch");
|
||||
const first = await context.projection(target, resolve);
|
||||
expect(await context.projection(target, resolve)).toEqual(first);
|
||||
expect(resolve).toHaveBeenCalledOnce();
|
||||
await expect(context.projection({ ...target, taskId: "other" }, resolve)).rejects.toThrow("target_changed");
|
||||
context.finish();
|
||||
await expect(context.projection(target, resolve)).rejects.toThrow("startup_completed");
|
||||
} finally { context.finish(); }
|
||||
});
|
||||
|
||||
it("reuses an identical transport target regardless of property insertion order", async () => {
|
||||
const context = new NativeExecutionContext(execution, 1_000);
|
||||
const resolve = vi.fn(async () => ({ instanceSecrets: {}, configuration: {} }));
|
||||
try {
|
||||
const first = await context.projection(target, resolve);
|
||||
const reordered = { environmentId: target.environmentId, taskId: target.taskId,
|
||||
leaseId: target.leaseId, pluginId: target.pluginId };
|
||||
expect(await context.projection(reordered, resolve)).toEqual(first);
|
||||
expect(resolve).toHaveBeenCalledOnce();
|
||||
} finally { context.finish(); }
|
||||
});
|
||||
|
||||
it("finishes a pending lookup without aborting a successful backend or completing worker responsibility", async () => {
|
||||
vi.useFakeTimers();
|
||||
const context = new NativeExecutionContext(execution, 1_000), settled = context.trackWorker();
|
||||
const remove = vi.spyOn(context.signal, "removeEventListener");
|
||||
let outcome = "pending";
|
||||
const waiting = context.projection(target, () => new Promise<never>(() => {}))
|
||||
.then(() => { outcome = "resolved"; }, error => { outcome = error.message; });
|
||||
try {
|
||||
await vi.advanceTimersByTimeAsync(0);
|
||||
context.finish();
|
||||
await vi.advanceTimersByTimeAsync(0);
|
||||
expect(outcome).toBe("native_execution_startup_completed");
|
||||
expect(context.signal.aborted).toBe(false);
|
||||
expect(context.pending).toBe(true);
|
||||
expect(remove).toHaveBeenCalledWith("abort", expect.any(Function));
|
||||
settled();
|
||||
expect(await context.drain()).toBe(true);
|
||||
} finally { settled(); context.abort(new Error("test cleanup")); context.finish(); await waiting; }
|
||||
});
|
||||
|
||||
it("keeps pending worker responsibility after cancellation until actual settlement", async () => {
|
||||
vi.useFakeTimers();
|
||||
const context = new NativeExecutionContext(execution, 1_000), settled = context.trackWorker();
|
||||
let drained = false;
|
||||
const drain = context.drain().then(result => { drained = true; return result; });
|
||||
try {
|
||||
context.abort(new Error("Stop requested"));
|
||||
expect(() => context.assertOpen()).toThrow("Stop requested");
|
||||
await vi.advanceTimersByTimeAsync(100);
|
||||
expect(drained).toBe(false);
|
||||
expect(context.pending).toBe(true);
|
||||
settled();
|
||||
expect(await drain).toBe(true);
|
||||
expect(context.pending).toBe(false);
|
||||
} finally { settled(); context.finish(); }
|
||||
});
|
||||
|
||||
it("reports an unsettled worker at the deadline without treating expiry as completion", async () => {
|
||||
vi.useFakeTimers();
|
||||
const context = new NativeExecutionContext(execution, 1_000), settled = context.trackWorker();
|
||||
const drain = context.drain();
|
||||
try {
|
||||
await vi.advanceTimersByTimeAsync(1_000);
|
||||
expect(await drain).toBe(false);
|
||||
expect(context.pending).toBe(true);
|
||||
expect(() => context.assertOpen()).toThrow("startup_expired");
|
||||
settled();
|
||||
expect(await context.drain()).toBe(true);
|
||||
} finally { settled(); context.finish(); }
|
||||
});
|
||||
|
||||
it("rejects a delayed transport lookup before it can send a worker request", async () => {
|
||||
vi.useFakeTimers();
|
||||
const context = new NativeExecutionContext(execution, 100);
|
||||
let resolve!: (value: { instanceSecrets: {}; configuration: {} }) => void;
|
||||
const transport = new Promise<{ instanceSecrets: {}; configuration: {} }>(complete => { resolve = complete; });
|
||||
const send = vi.fn();
|
||||
const operation = context.projection(target, () => transport).then(send);
|
||||
const rejected = expect(operation).rejects.toThrow("startup_expired");
|
||||
await vi.advanceTimersByTimeAsync(100);
|
||||
await rejected;
|
||||
resolve({ instanceSecrets: {}, configuration: {} });
|
||||
await Promise.resolve();
|
||||
expect(send).not.toHaveBeenCalled();
|
||||
expect(context.pending).toBe(false);
|
||||
context.finish();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,110 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import type { PluginEnvironmentTaskExecutionContext } from "@paperclipai/plugin-sdk";
|
||||
import type { NativeExecutionInput } from "../../vendor/paperclip-runner/index.js";
|
||||
|
||||
export class NativeExecutionContext {
|
||||
readonly id = randomUUID();
|
||||
readonly deadlineAtMs: number;
|
||||
readonly signal: AbortSignal;
|
||||
readonly execution: NativeExecutionInput;
|
||||
#abort = new AbortController();
|
||||
#timer: ReturnType<typeof setTimeout>;
|
||||
#calls = new Set<object>();
|
||||
#waiters = new Set<() => void>();
|
||||
#finishWaiters = new Set<() => void>();
|
||||
#transport?: Promise<PluginEnvironmentTaskExecutionContext["transport"]>;
|
||||
#target?: string;
|
||||
#finished = false;
|
||||
pluginSupported = false;
|
||||
|
||||
constructor(execution: NativeExecutionInput, timeoutMs: number) {
|
||||
this.execution = execution;
|
||||
this.deadlineAtMs = Date.now() + timeoutMs;
|
||||
this.signal = this.#abort.signal;
|
||||
this.#timer = setTimeout(() => this.abort(new Error("native_execution_startup_expired")), timeoutMs);
|
||||
this.#timer.unref?.();
|
||||
}
|
||||
|
||||
get active(): boolean { return !this.#finished; }
|
||||
get pending(): boolean { return this.#calls.size > 0; }
|
||||
|
||||
assertOpen(): void {
|
||||
if (this.#finished) throw new Error("native_execution_startup_completed");
|
||||
if (Date.now() >= this.deadlineAtMs) this.abort(new Error("native_execution_startup_expired"));
|
||||
this.signal.throwIfAborted();
|
||||
}
|
||||
|
||||
assertScope(scope: { companyId: string; agentId: string; runId: string }): void {
|
||||
const binding = this.execution.binding;
|
||||
if (scope.companyId !== binding.companyId || scope.agentId !== binding.agentId || scope.runId !== binding.runId) {
|
||||
throw new Error("native_execution_context_scope_mismatch");
|
||||
}
|
||||
this.assertOpen();
|
||||
}
|
||||
|
||||
async wait<T>(operation: () => Promise<T>): Promise<T> {
|
||||
this.assertOpen();
|
||||
let remove = () => {};
|
||||
try {
|
||||
const value = await Promise.race([
|
||||
Promise.resolve().then(() => { this.assertOpen(); return operation(); }),
|
||||
new Promise<never>((_resolve, reject) => {
|
||||
const abort = () => reject(this.signal.reason);
|
||||
const finish = () => reject(new Error("native_execution_startup_completed"));
|
||||
this.#finishWaiters.add(finish);
|
||||
this.signal.addEventListener("abort", abort, { once: true });
|
||||
remove = () => {
|
||||
this.signal.removeEventListener("abort", abort);
|
||||
this.#finishWaiters.delete(finish);
|
||||
};
|
||||
}),
|
||||
]);
|
||||
this.assertOpen();
|
||||
return value;
|
||||
} finally { remove(); }
|
||||
}
|
||||
|
||||
async projection(target: { pluginId: string; leaseId: string; taskId: string; environmentId: string },
|
||||
resolve: () => Promise<PluginEnvironmentTaskExecutionContext["transport"]>): Promise<PluginEnvironmentTaskExecutionContext> {
|
||||
this.assertOpen();
|
||||
const key = JSON.stringify([target.pluginId, target.leaseId, target.taskId, target.environmentId]);
|
||||
if (this.#target !== undefined && this.#target !== key) throw new Error("native_execution_context_target_changed");
|
||||
this.#target = key;
|
||||
this.#transport ??= resolve();
|
||||
const transport = await this.wait(() => this.#transport!);
|
||||
this.assertOpen();
|
||||
return { id: this.id, deadlineAtMs: this.deadlineAtMs, transport };
|
||||
}
|
||||
|
||||
trackWorker(): () => void {
|
||||
this.assertOpen();
|
||||
const call = {};
|
||||
this.#calls.add(call);
|
||||
return () => {
|
||||
this.#calls.delete(call);
|
||||
if (!this.#calls.size) for (const waiter of this.#waiters) waiter();
|
||||
};
|
||||
}
|
||||
|
||||
async drain(): Promise<boolean> {
|
||||
if (!this.pending) return true;
|
||||
let timer: ReturnType<typeof setTimeout> | undefined, complete!: () => void;
|
||||
try {
|
||||
await Promise.race([
|
||||
new Promise<void>(resolve => { complete = resolve; this.#waiters.add(complete); }),
|
||||
new Promise<void>(resolve => { timer = setTimeout(resolve, Math.max(0, this.deadlineAtMs - Date.now())); }),
|
||||
]);
|
||||
return !this.pending;
|
||||
} finally { clearTimeout(timer); this.#waiters.delete(complete); }
|
||||
}
|
||||
|
||||
abort(reason: Error): void { this.#abort.abort(reason); }
|
||||
|
||||
finish(): void {
|
||||
clearTimeout(this.#timer);
|
||||
this.#finished = true;
|
||||
this.#transport = undefined;
|
||||
for (const finish of this.#finishWaiters) finish();
|
||||
this.#finishWaiters.clear();
|
||||
}
|
||||
}
|
||||
@@ -13,6 +13,20 @@ function fixture() {
|
||||
}
|
||||
afterEach(() => vi.useRealTimers());
|
||||
describe("native execution renewal ownership", () => {
|
||||
it("reports only the remaining confirmed ownership budget", async () => {
|
||||
vi.useFakeTimers();
|
||||
const { input, returning } = fixture();
|
||||
let complete!: (value: { runId: string }[]) => void;
|
||||
returning.mockImplementationOnce(() => new Promise(resolve => { complete = resolve; }));
|
||||
const guard = startNativeSessionExecutionLeaseRenewal({ ...input, confirmedDeadline: performance.now() + 45_000 });
|
||||
expect(guard.remainingMs()).toBe(45_000);
|
||||
await vi.advanceTimersByTimeAsync(40_000);
|
||||
expect(guard.remainingMs()).toBe(5_000);
|
||||
complete([{ runId: "run" }]);
|
||||
await vi.advanceTimersByTimeAsync(0);
|
||||
expect(guard.remainingMs()).toBe(40_000);
|
||||
await guard.stop();
|
||||
});
|
||||
it("keeps a successfully renewed lease through repeated slow connection acquisition", async () => {
|
||||
vi.useFakeTimers();
|
||||
const { input, update, transaction } = fixture();
|
||||
|
||||
@@ -107,9 +107,10 @@ type RenewalInput = {
|
||||
db: Db; runId: string; companyId: string; issueId: string;
|
||||
leaseOwner: string; attempt: number; controller: NativeControllerIdentity;
|
||||
remote?: boolean;
|
||||
confirmedDeadline?: number;
|
||||
onLost: () => Promise<unknown>;
|
||||
};
|
||||
export interface NativeExecutionLeaseRenewal { stop(): Promise<void>; }
|
||||
export interface NativeExecutionLeaseRenewal { stop(): Promise<void>; remainingMs(): number; }
|
||||
|
||||
// Preparation and execution share one renewal timer. Each caller releases its
|
||||
// own reference; the final release drains any SQL already in flight.
|
||||
@@ -117,6 +118,7 @@ const renewals = new WeakMap<Db, Map<string, {
|
||||
input: RenewalInput; references: number; timer: ReturnType<typeof setInterval>;
|
||||
pending: Promise<void>; busy: boolean; lost: Error | null;
|
||||
deadline: ReturnType<typeof setTimeout>; loss: Promise<unknown>;
|
||||
confirmedDeadline: number;
|
||||
}>>();
|
||||
|
||||
export function startNativeSessionExecutionLeaseRenewal(input: RenewalInput): NativeExecutionLeaseRenewal {
|
||||
@@ -154,6 +156,7 @@ export function startNativeSessionExecutionLeaseRenewal(input: RenewalInput): Na
|
||||
// or recreate a timer after the final caller has stopped.
|
||||
if (state!.lost || state!.references === 0) return;
|
||||
clearTimeout(state!.deadline);
|
||||
state!.confirmedDeadline = confirmedDeadline;
|
||||
state!.deadline = deadline(confirmedDeadline - performance.now());
|
||||
})
|
||||
.catch((error: unknown) => {
|
||||
@@ -167,13 +170,14 @@ export function startNativeSessionExecutionLeaseRenewal(input: RenewalInput): Na
|
||||
};
|
||||
const timer = setInterval(renew, input.remote ? REMOTE_CONTROLLER_LEASE_MS / 3 : NATIVE_SESSION_EXECUTION_LEASE_RENEW_INTERVAL_MS);
|
||||
timer.unref?.();
|
||||
const confirmedDeadline = input.confirmedDeadline ?? performance.now() + ttlMs;
|
||||
state = { input, references: 1, timer, pending: Promise.resolve(), busy: false, lost: null,
|
||||
deadline: deadline(ttlMs), loss: Promise.resolve() };
|
||||
deadline: deadline(confirmedDeadline - performance.now()), confirmedDeadline, loss: Promise.resolve() };
|
||||
owners.set(input.leaseOwner, state);
|
||||
}
|
||||
const owned = state;
|
||||
let stopped = false;
|
||||
return { stop: async () => {
|
||||
return { remainingMs: () => owned.lost ? 0 : Math.max(0, owned.confirmedDeadline - performance.now()), stop: async () => {
|
||||
if (!stopped) {
|
||||
stopped = true;
|
||||
if (--owned.references === 0) {
|
||||
|
||||
@@ -11060,7 +11060,7 @@ describe("native session bounded recovery", () => {
|
||||
return query;
|
||||
});
|
||||
const stop = vi.fn(async () => undefined);
|
||||
const renew = vi.spyOn(nativeLeaseRenewal, "startNativeSessionExecutionLeaseRenewal").mockReturnValue({ stop });
|
||||
const renew = vi.spyOn(nativeLeaseRenewal, "startNativeSessionExecutionLeaseRenewal").mockReturnValue({ stop, remainingMs: () => 60_000 });
|
||||
const releasePreparation = vi.fn(async () => {
|
||||
expect(renew).toHaveBeenCalledOnce();
|
||||
expect(stop).not.toHaveBeenCalled();
|
||||
|
||||
Reference in new issue
Block a user