Files
PaperClipAI/server/src/__tests__/fixtures/plugin-worker-login-pty.cjs
T
DottaandPaperclip 250deab910 fix(runner): keep healthy native sessions alive (#13261)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native sessions keep a provider process and its work alive across
control-plane operations.
> - A hidden 15-minute turn deadline stopped work even when the agent
timeout was zero.
> - A one-hour runner lifetime and fixed connection lease added two more
limits.
> - Recovery also rejected goal commands because it reconstructed their
startup summary with the wrong protocol version.
> - This pull request removes implicit duration limits and renews
authenticated leases in the harness.
> - Healthy sessions can continue without model action or a user
interface change.

## Linked Issues or Issue Description

Refs #13092 and #12845. Related: #13163 covers sandbox recovery after
app restarts; this change covers session duration and lease renewal.

**What happened?**

A native Codex session stopped after 15 minutes while a tool was still
running. The agent had `timeoutSec: 0`. Recovery then rejected a
`session.goal.get` startup command with `invalid provider startup
ownership fence`.

**Expected behavior**

An unlimited session keeps working while its provider and authenticated
controller remain healthy. Lease maintenance is transparent. Explicit
timeouts, cancellation and revoked authority still take effect.

**Steps to reproduce**

Start a native session with `timeoutSec: 0` and run a tool beyond 15
minutes. Before this fix, the runtime cancels the turn. A recovery
startup that uses a goal command also exposes the protocol-version
mismatch.

## What Changed

- Honor the agent turn timeout. Zero disables the timer. Long explicit
durations use timer chunks to avoid Node timer overflow.
- Default native runner lifetime to unlimited. Keep bounded startup,
reconnect and control-operation deadlines.
- Renew leases over the authenticated connection. Persist renewal before
the reply. Validate identity, epoch and expiry. Handle duplicate
requests and a lost reply on reconnect.
- Freeze renewal during warm ownership transitions and terminal
handling.
- Validate persisted goal startup commands with protocol v2.
- Add duration, renewal, ownership, recovery and real-process regression
tests. Update runner protocol and recovery docs.
- Add no UI components or controls. Renewal requires no model output or
user action.

## Verification

Current head: `348e369c35c5da8bb8be378f4b35dcf6f40882e7`. [CI
run](https://github.com/paperclipai/paperclip/actions/runs/34649767113).

- All 32 checks pass on this head. The two Storybook checks are skipped
as expected. CI includes full build, typecheck, runner verification,
browser suites, server tests and the canary package dry run.
- Greptile reports 5/5 on this head. All review threads are resolved,
and the security scan passes.
- Passed `pnpm -r typecheck` and `pnpm build` locally.
- Passed 219 native-runtime and controller tests, including fake-clock
tests for three weeks of renewal and 30-day explicit timeouts. Six
denial tests confirm that renewal cannot extend expired, revoked or
mismatched authority.
- Passed 337 executor, cancellation and restart-recovery tests, plus 278
Rust runner-core library tests.
- Passed a real runner with a silent fake Codex provider across its
original lease expiry. Runner PID, provider PID, thread and active turn
stayed unchanged. Warm-attach recovery tests also pass.
- Passed all 83 plugin-worker tests and 159 of 161 workspace-runtime
tests locally. The two remaining assertions passed with a canonical
macOS temporary directory, as did the changed runtime fixture. The full
affected server shard passes in CI.
- An unchanged GitHub callback-ordering test failed once in CI, passed
locally in isolation, and passed its one test-shard retry. The final CI
summary is successful.
- The full local `pnpm test:run` sweep was interrupted after dependency
setup failures and load-related timeouts. Identified failing suites
passed in isolated reruns after the dependency repair. The complete test
matrix passed remotely in CI.

## Risks

- Deploy the controller and runner together to enable renewal. Older
peers keep their existing bounded lease behavior.
- Unlimited runtime permits long resource use until completion, explicit
cancellation, configured timeout or loss of valid authority.
- Lease renewal changes authenticated protocol handling. Regression
tests cover stale, revoked and mismatched authority, lost replies and
warm handoff behavior.
- Simulated multi-week tests and a real lease-boundary test do not
constitute a weeks-long production soak.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, code execution
and TypeScript/Rust test tools. The exact backend revision and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-11 17:09:43 -05:00

309 lines
12 KiB
JavaScript

// Test worker fixture for the host-owned setup-token login pseudo-terminal route
// gate. The fixture drives the manager route state machine through
// the four typed methods (open, input, stop, close) and the output and exit
// notifications.
//
// The manager allowlists `command` to the fixed `CLAUDE_SETUP_TOKEN_COMMAND`. The
// test encodes a JSON directive in the forwarded `providerLeaseId`, so one fixture
// serves every route-gate case:
// - `mode`: "normal" | "malformed-open" | "no-open-reply" | "duplicate-open-reply" |
// "exit-before-open-reply"
// - `workerSessionId`: the worker session id the open reply returns (default "ws-1")
// - `outputs`: an array of `{ chunk, sid?, crossRoute?, omitHostRouteId? }`. The
// fixture emits each as an output notification after the open reply. `sid`
// defaults to the real worker session id; a test sets a wrong `sid` to prove
// the host drops a mismatched notification. `crossRoute: true` sends this
// route's own worker session id under a DIFFERENT, already-open route's host
// route identifier (any other entry currently registered), so a test proves
// the host drops a swapped `(hostRouteId, workerSessionId)` pair instead of
// misdelivering it. `omitHostRouteId: true` sends the notification with no
// `hostRouteId` field at all, so a test proves the host warns about a plugin
// build old enough to omit the field, instead of silently dropping it.
// - `emitOnInput`: when true, wait for the first input before sending scripted
// notifications, so legacy routing tests know the worker session is bound.
// - `exitCode`: when set, the fixture emits an exit notification after the outputs.
// - `omitHostRouteIdOnExit`: when true, the main `exitCode` exit notification
// carries no `hostRouteId` field, so a test proves the host still resolves
// a legacy worker's exit by the worker session id.
// - `extraExits`: an array of `{ exitCode, sid? }`. The fixture emits each as a
// further exit notification, after the main `exitCode` exit. `sid` defaults
// to the real worker session id; a test sets a wrong `sid` to script a worker
// that sends a valid exit, then a mismatched exit, before the bind.
// - `outputsAfterExit`: an array of `{ chunk, sid? }`, same shape as `outputs`.
// The fixture emits these after the exit notification, so a test proves the
// host drops output that arrives behind a queued exit.
// - `sequence`: an array of `{ type: "output", chunk, sid? }` or
// `{ type: "exit", exitCode, sid? }` entries. When set, the fixture emits
// exactly this sequence, in order, instead of the fixed
// outputs/exitCode/extraExits/outputsAfterExit composition. A test uses
// this to script an arrival order the fixed composition cannot express,
// for example a mismatched exit that arrives before a valid one.
// - `closeMode`: "ack" | "bad-ack" | "no-ack" (default "ack"). It controls the
// close reply, so a test proves the host retires the worker on an unconfirmed
// close.
// - `batchWithOpenReply`: when true, the fixture writes the open reply and the
// scripted outputs and exit in one stdout write. The host then reads the open
// reply and the notifications in one batch, so a test proves the host queues
// and replays a record that arrives before the route binds.
// - `mode: "exit-before-open-reply"`: the fixture emits the scripted outputs,
// then exits with no open reply, so a test proves the host clears the
// pre-bind queue on a worker exit during the open window.
const readline = require("node:readline");
function send(message) {
process.stdout.write(`${JSON.stringify(message)}\n`);
}
// Find some other host route identifier the fixture already registered, for
// a `crossRoute` output entry. Returns null when no other route is open,
// which a well-formed test never triggers.
function pickOtherHostRouteId(ownHostRouteId) {
for (const key of routes.keys()) {
if (key !== ownHostRouteId) return key;
}
return null;
}
// Serialize one array of `{ chunk, sid?, crossRoute?, omitHostRouteId? }`
// entries as newline-delimited output notification lines. A test sets a
// wrong `sid` on one entry to force a mismatch. `crossRoute: true` stamps
// some OTHER already-open route's host route identifier on this route's own
// worker session id, so a test proves the host drops a swapped pair. Every
// other line echoes this route's own host route identifier, so the host can
// route each chunk to its own route when the worker holds more than one.
// `omitHostRouteId: true` sends the notification with no `hostRouteId` field
// at all, so a test proves the host warns instead of silently dropping it.
function outputLines(entries, hostRouteId, workerSessionId) {
let lines = "";
for (const entry of entries) {
const effectiveHostRouteId = entry.crossRoute
? pickOtherHostRouteId(hostRouteId) ?? hostRouteId
: entry.hostRouteId ?? hostRouteId;
const params = {
workerSessionId: entry.sid ?? workerSessionId,
chunk: entry.chunk,
};
if (!entry.omitHostRouteId) params.hostRouteId = effectiveHostRouteId;
lines += `${JSON.stringify({
jsonrpc: "2.0",
method: "loginPty.output",
params,
})}\n`;
}
return lines;
}
// Serialize one exit notification line for the given host route and worker
// session id. `omit: true` sends the notification with no `hostRouteId`
// field at all, matching a plugin build old enough to predate it.
function exitLine(hostRouteId, workerSessionId, exitCode, omit) {
const params = { workerSessionId, exitCode };
if (!omit) params.hostRouteId = hostRouteId;
return `${JSON.stringify({
jsonrpc: "2.0",
method: "loginPty.exit",
params,
})}\n`;
}
// Serialize the scripted output and exit notifications as newline-delimited
// lines. The `sequence` directive, when set, emits exactly that order. The
// fixed composition otherwise emits, in order: the pre-exit outputs, then
// the main exit, then every extra exit, then the post-exit outputs. The
// batch mode writes these together with the open reply in one stdout write.
function scriptedOutputLines(directive, hostRouteId, workerSessionId) {
if (Array.isArray(directive.sequence)) {
let lines = "";
for (const entry of directive.sequence) {
if (entry.type === "output") {
lines += outputLines(
[{ chunk: entry.chunk, sid: entry.sid, hostRouteId: entry.hostRouteId, crossRoute: entry.crossRoute }],
hostRouteId,
workerSessionId,
);
} else if (entry.type === "exit") {
lines += exitLine(entry.hostRouteId ?? hostRouteId, entry.sid ?? workerSessionId, entry.exitCode);
}
}
return lines;
}
const outputs = Array.isArray(directive.outputs) ? directive.outputs : [];
const outputsAfterExit = Array.isArray(directive.outputsAfterExit)
? directive.outputsAfterExit
: [];
const extraExits = Array.isArray(directive.extraExits) ? directive.extraExits : [];
let lines = outputLines(outputs, hostRouteId, workerSessionId);
if (typeof directive.exitCode === "number") {
lines += exitLine(hostRouteId, workerSessionId, directive.exitCode, directive.omitHostRouteIdOnExit);
}
for (const exit of extraExits) {
lines += exitLine(exit.hostRouteId ?? hostRouteId, exit.sid ?? workerSessionId, exit.exitCode);
}
lines += outputLines(outputsAfterExit, hostRouteId, workerSessionId);
return lines;
}
// The registered terminals, keyed by the host route id. Each entry records the
// bound worker session id and the close directive.
const routes = new Map();
function parseDirective(raw) {
try {
const parsed = JSON.parse(raw);
return parsed && typeof parsed === "object" ? parsed : {};
} catch {
return {};
}
}
const rl = readline.createInterface({ input: process.stdin, crlfDelay: Infinity });
rl.on("line", (line) => {
if (!line.trim()) return;
const message = JSON.parse(line);
const method = message && typeof message.method === "string" ? message.method : null;
const params = message.params ?? {};
if (method === "initialize") {
send({
jsonrpc: "2.0",
id: message.id,
result: {
ok: true,
supportedMethods: [
"loginPtyOpen",
"loginPtyInput",
"loginPtyStop",
"loginPtyClose",
],
},
});
return;
}
if (method === "loginPtyOpen") {
const directive = parseDirective(params.providerLeaseId);
const mode = directive.mode ?? "normal";
const workerSessionId = directive.workerSessionId ?? "ws-1";
const closeMode = directive.closeMode ?? "ack";
routes.set(params.hostRouteId, {
workerSessionId,
closeMode,
pendingOutput: directive.emitOnInput === true
? scriptedOutputLines(directive, params.hostRouteId, workerSessionId)
: null,
});
if (mode === "no-open-reply") {
// Never reply, so the host open call times out.
return;
}
if (mode === "exit-before-open-reply") {
// Emit the scripted pre-bind outputs, then exit with no open reply. The
// route never binds, so a test proves the worker-exit path clears the
// pre-bind queue.
process.stdout.write(scriptedOutputLines(directive, params.hostRouteId, workerSessionId));
process.exit(1);
return;
}
// A malformed reply carries no worker session id, so the host cannot bind
// and terminalizes the route.
const isMalformedOpen = mode === "malformed-open";
const openReplyLine = `${JSON.stringify({
jsonrpc: "2.0",
id: message.id,
result: isMalformedOpen ? {} : { workerSessionId },
})}\n`;
if (directive.batchWithOpenReply === true) {
// Write the open reply and the scripted outputs and exit in one stdout
// write. The host reads them in one batch, so an output or an exit
// notification arrives before the route binds — even a malformed reply
// that never binds. The host must queue and, on a bind, replay it; on a
// malformed reply, it must clear the queue instead.
process.stdout.write(
openReplyLine + scriptedOutputLines(directive, params.hostRouteId, workerSessionId),
);
return;
}
process.stdout.write(openReplyLine);
if (mode === "duplicate-open-reply") {
// Send a second open reply for the same request id. The host drops it.
process.stdout.write(openReplyLine);
}
if (isMalformedOpen) {
// No scripted output follows a non-batched malformed reply.
return;
}
if (directive.emitOnInput === true) return;
// Emit the scripted output and the exit after the open reply, so the host
// binds the route first.
setImmediate(() => {
process.stdout.write(scriptedOutputLines(directive, params.hostRouteId, workerSessionId));
});
return;
}
if (method === "loginPtyInput") {
// Echo the input back as one output notification for the bound session, so a
// test proves the input reaches the worker and the output routes back.
for (const [hostRouteId, entry] of routes.entries()) {
if (entry.workerSessionId === params.workerSessionId) {
if (entry.pendingOutput !== null) {
process.stdout.write(entry.pendingOutput);
entry.pendingOutput = null;
continue;
}
send({
jsonrpc: "2.0",
method: "loginPty.output",
params: {
hostRouteId,
workerSessionId: entry.workerSessionId,
chunk: `echo:${params.data}`,
},
});
}
}
send({ jsonrpc: "2.0", id: message.id, result: null });
return;
}
if (method === "loginPtyStop") {
send({ jsonrpc: "2.0", id: message.id, result: null });
return;
}
if (method === "loginPtyClose") {
const entry = routes.get(params.hostRouteId);
routes.delete(params.hostRouteId);
const closeMode = entry ? entry.closeMode : "ack";
if (closeMode === "no-ack") {
// Never reply, so the host close call times out and the host retires us.
return;
}
if (closeMode === "bad-ack") {
send({ jsonrpc: "2.0", id: message.id, result: { hostRouteId: "mismatched-route" } });
return;
}
send({ jsonrpc: "2.0", id: message.id, result: { hostRouteId: params.hostRouteId } });
return;
}
if (method === "shutdown") {
send({ jsonrpc: "2.0", id: message.id, result: {} });
setImmediate(() => process.exit(0));
return;
}
send({
jsonrpc: "2.0",
id: message.id,
error: { code: -32601, message: `Unhandled method: ${method}` },
});
});