From 76a796e964493a5d969597ddec14b5370a2bd41e Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 29 Oct 2026 10:56:24 -0500 Subject: [PATCH] test(runner): cover ACPX execution across controller replacement (#18161) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner keeps provider execution separate from the main server. > - A server replacement must reconnect to the original execution. > - ACPX recovery also needs the original provider session to remain alive. > - We need to distinguish a controller outage from a provider crash or an explicit close. > - This pull request adds five process-level regression tests for those boundaries. > - The tests prevent duplicate launches and preserve the scope of the evidence. ## Linked Issues or Issue Description Refs #17097, #17439 and #17455. These changes add coverage for the existing recovery contract. They do not add another recovery implementation. **What existing behavior does this improve?** Native ACPX execution across authenticated PRP controller replacement. **Current behavior** Runner has recovery coverage, but this ACPX sequence needed a focused test: lose the controller, restore its checkpoint, adopt the same process, and close the old controller. A provider that has already closed can return `ACPX request snapshot requires the surviving provider`. That message alone does not identify why it closed. **Proposed behavior** Run five credential-free regressions with the production Rust Runner and a scripted ACPX peer. Preserve active work and offline completion. Check native transport adoption and old-controller cleanup. Keep provider-crash and explicit-close controls separate from controller recovery. **Reason and benefit** Catch a future change that kills or replaces a live execution during controller replacement. Keep a matching error control from being mistaken for proof of a hosted root cause. ## What Changed - Add five process-level ACPX controller recovery tests. - Restore controller state from an external checkpoint after deleting local controller storage. - Check that native transport adoption does not launch a replacement Runner. - Build the debug Runner before the tests. Pass only the test bootstrap credential to its process. - Verify scripted-peer shutdown through its lifetime sockets instead of signaling a saved provider PID. - Document the command, dependencies, optional prebuilt binary/profile overrides and verification limits in the Runner README. - Give the setup hook 30 seconds of margin above the bounded Cargo build. ## Verification - Focused test: `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/control-plane/acpx-controller-recovery.test.ts --maxWorkers=1` — 5 passed. - Adjacent regression selection: `src/control-plane/`, `src/live/runnerd-controller-outage.test.ts`, and `src/live/acpx-request-recovery.test.ts` — 5 files and 195 tests passed. - Protocol manifest and package ownership checks passed. The standalone forbidden-import checker reports existing imports in unchanged files; this PR changes none of those files or the checker. - `pnpm -r typecheck` and `pnpm build` passed. The full `pnpm test:run` attempt is retained. Its first database cases failed because the local macOS PostgreSQL package lacked required dylib aliases. Direct native probes identified the missing libraries. Local dependency aliases restored initialization, and one unchanged failing warm-session database case then passed. The original full-suite attempt was stopped after that diagnosis. Its failures remain retained; full-suite completion is not claimed. - Current-head CI repair: browser shard 5 failed after the task reload because the result was still behind the conversation loading veil. The saved snapshots show this state. The UI has a 15-second history reveal limit, but the result assertion had a 5-second wait. The test now returns to the task tab and waits for its existing history-content readiness before it checks the result. `PAPERCLIP_E2E_PORT=38741 pnpm exec playwright test --config tests/e2e/playwright.config.ts tests/e2e/connection-reviews.spec.ts --grep 'connection review: (approve|decline|always|failure),'` passed all four journeys in 2.9 minutes against a fresh local server and database. Master independently received the same readiness fix in #17923. This branch merged that exact base version, so the browser file is not part of the final PR diff. The five ACPX tests passed again after that merge. No production UI code changed in this PR. The original failed CI report is retained. - Retained CI failure: the first Linux run rejected the scripted peer's automatically assigned ports because they can be below the protocol minimum of 49152. The fixture now binds explicit random ports in the permitted range, with bounded collision retries. No production validation was relaxed. The fixed peer also passed a network-isolated Linux arm64 container check; that check used the actual generated peer and verified permitted ports and clean EOF shutdown. - Review follow-up: both native-transport reads require the original active turn ID with status `inProgress`. Adoption always runs for a selected prebuilt binary. The crash control accepts only explicit provider-loss diagnostics, not any snapshot-error wrapper. Focused tests cover these changes. - Final review follow-up: the setup hook has a 150-second limit around the 120-second Cargo build. The README documents both optional overrides. All five focused tests passed in default mode (17.72 seconds) and with the prebuilt binary plus profile overrides (10.90 seconds). The fixture version is explicitly passed through `--runner-version`; it is not a check against Cargo package metadata. The crash control verifies the original PID receipt and released lifetime fence ports before it accepts only known provider-loss diagnostics. - CI completed successfully for `d31647fb95f38b7d9d56bc8c13a9a8c0b391d247`, including all eight browser shards. Final head `949a70596f07042cf7b39a5a3ccb9a9445b6e855` also passed [all CI gates](https://github.com/paperclipai/paperclip/actions/runs/40904592159), including all eight browser shards, and received fresh 5/5 Greptile and Paperclip reviews. All six review threads are resolved. - Separate manual provider evidence used real Claude Opus 5.5, ACPX 0.13.1, Claude ACP 0.73.0 and Claude SDK 0.3.286 on macOS arm64. Source `929e840595c8933ca485ac7a2e99413e1e11ca64` survived 10-second and 300-second controller outages. Original source `4f8104dec6a0bb37009080daa8984a80f58b7a04` survived a 10-second outage. The five-minute case retained the original processes and session, made useful progress, reattached at 303.793 seconds from disconnect, and completed once. These runs preceded this branch rebase and do not qualify every change in the rebased server. - Retained manual failures: the first historical attempt failed installation admission because scratch dependency symlinks escaped their root. Correcting that setup allowed the next attempt to start. The five-minute harness initially checked the final message before asynchronous projection finished. Saved final-item and terminal events passed an independent evidence check. The original failed result remains retained. ## Risks - This PR adds tests and documentation only. It changes no production protocol, recovery logic, database schema or browser behavior. - The automated tests use a scripted ACPX peer. They do not establish real Claude, PostgreSQL, Daytona or Cloud rollout acceptance. - The tests require Cargo and a POSIX host. Their waits are bounded. They stop only their own Runner and scripted peer. - The earlier hosted provider loss remains unexplained. The separate real-provider experiments did not reproduce it. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use and code execution. The host does not expose a more specific serving model ID or context-window size. Anthropic Claude Opus 5.5 (`claude-opus-5-5`) performed the separate synthetic verification workloads. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- packages/paperclip-runner/README.md | 27 ++ .../acpx-controller-recovery.test.ts | 284 ++++++++++++++++++ 2 files changed, 311 insertions(+) create mode 100644 packages/paperclip-runner/src/control-plane/acpx-controller-recovery.test.ts diff --git a/packages/paperclip-runner/README.md b/packages/paperclip-runner/README.md index 6988bc56e3d..93c6bd1e03b 100644 --- a/packages/paperclip-runner/README.md +++ b/packages/paperclip-runner/README.md @@ -957,3 +957,30 @@ the host restarts that resource with a new socket. Do not remove its lock marker while daemon work might still be running. The socket and its parent must be private, canonical paths owned by the same account. Local stdio assignment requires Unix; other platforms retain the HTTP path. + +## ACPX controller replacement regression tests + +`src/control-plane/acpx-controller-recovery.test.ts` runs the production Rust +Runner and authenticated PRP controller with a scripted ACPX peer. It covers +active-turn reconnection, checkpoint restoration after deleting controller +storage, native transport adoption and old-controller finalization, completion +while disconnected, a provider crash, and an explicitly closed provider. +It builds the debug Runner before execution and requires Cargo and a POSIX host. +No model credentials, provider requests, application server or database are used. + +```sh +pnpm --filter @paperclipai/paperclip-runner exec vitest run \ + src/control-plane/acpx-controller-recovery.test.ts --maxWorkers=1 +``` + +Set `PAPERCLIP_ACPX_REPRO_BINARY` to an existing Runner executable to skip the +Cargo build. All five tests still run. Set `PAPERCLIP_ACPX_REPRO_PROFILES` to an +alternate `acpx-profiles.json` when that executable uses a different profile +manifest. Both overrides are optional. + +The failure controls distinguish a lost provider from a controller outage; +matching a diagnostic after explicit close does not identify the cause of a +hosted failure. Real Claude/ACPX acceptance, server/Postgres recovery, and +Daytona deployment continuity need their own evidence. See the +[remote continuity guide](../../doc/testing-native-remote-continuity.md) and the +[Daytona Product E2E suite](../../tests/runner-e2e/README.md). diff --git a/packages/paperclip-runner/src/control-plane/acpx-controller-recovery.test.ts b/packages/paperclip-runner/src/control-plane/acpx-controller-recovery.test.ts new file mode 100644 index 00000000000..446322a0ec9 --- /dev/null +++ b/packages/paperclip-runner/src/control-plane/acpx-controller-recovery.test.ts @@ -0,0 +1,284 @@ +import { execFileSync, spawn } from "node:child_process"; +import { once } from "node:events"; +import { createHash } from "node:crypto"; +import { createServer as createNetServer } from "node:net"; +import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import { beforeAll, expect, it, vi } from "vitest"; +import { DurablePrpControlPlane, decodeDurablePrpCheckpoint, type DurablePrpCheckpoint } from "./durable-prp-control-plane.js"; +import { GENERATED_ACPX_SIDECAR_PROTOCOL_VERSION } from "../drivers/acpx/generated-sidecar-contract.js"; +import { createCapabilityRunnerdCodexTransport } from "../live/runnerd-codex-transport.js"; + +const packageRoot = fileURLToPath(new URL("../..", import.meta.url)); +const binary = process.env.PAPERCLIP_ACPX_REPRO_BINARY ?? resolve(packageRoot, "runner/target/debug/paperclip-runnerd"); +const profiles = JSON.parse(readFileSync(process.env.PAPERCLIP_ACPX_REPRO_PROFILES ?? resolve(packageRoot, "acpx-profiles.json"), "utf8")); +const profile = profiles.profiles.claude; +const digest = (path: string) => "sha256:" + createHash("sha256").update(readFileSync(path)).digest("hex"); +const identity = { runnerInstanceId: "runner-acpx-recovery", environmentLeaseId: "lease-acpx-recovery", + runId: "run-1", normalizedSessionId: "session-1", turnId: "durable-turn-1", itemId: "item-1" }; +const expectedRunnerVersion = "0.3.0"; +let expectedRunnerDigest: string; + +beforeAll(() => { + if (!process.env.PAPERCLIP_ACPX_REPRO_BINARY) { + execFileSync("cargo", ["build", "--locked", "--manifest-path", resolve(packageRoot, "runner/Cargo.toml"), + "-p", "paperclip-runner-core", "--bin", "paperclip-runnerd"], { timeout: 120_000, stdio: "pipe" }); + } + expectedRunnerDigest = digest(binary); +}, 150_000); + +function sidecarSource(root: string) { + return [ + "const fs = require('node:fs'); const net = require('node:net'); const rl = require('node:readline'); const {randomInt}=require('node:crypto');", + "const root = " + JSON.stringify(root) + ";", + "const profileDigest = " + JSON.stringify(profile.commandDigest) + ";", + "const protocolVersion = " + GENERATED_ACPX_SIDECAR_PROTOCOL_VERSION + ";", + "const send = value => process.stdout.write(JSON.stringify(value) + '\\n');", + "let identity, runId, turnId = null, sequence = 0, lastAction = '';", + "const event = (eventType, payload) => send({protocolVersion, sequence:++sequence, eventType, runId, turnId, payload});", + "(async () => {", + " const lease = async () => {for(let attempt=0;attempt<64;attempt++){const s=net.createServer();try{await new Promise((resolve,reject)=>{s.once('error',reject);s.listen(randomInt(49152,65536),'127.0.0.1',resolve);});return s;}catch(error){if(error.code!=='EADDRINUSE')throw error;}}throw new Error('No lifetime fence port available');};", + " const fences = await Promise.all([0,0,0].map(lease));", + " const ports = fences.map(s=>s.address().port); fences[2].close();", + " fs.appendFileSync(root+'/launches.jsonl', JSON.stringify({pid:process.pid, ports})+'\\n');", + " const reply = (r,result) => send({protocolVersion,id:r.id,ok:true,result});", + " const lines=rl.createInterface({input:process.stdin}); lines.on('close',()=>process.exit(0)); lines.on('line', line => {", + " const r=JSON.parse(line), p=r.params || {};", + " switch(r.command) {", + " case 'initialize': reply(r,{protocolVersion,sidecarPid:process.pid,profile:{agent:'claude'},capabilities:{persistentSessions:true,exactModelVerification:true,permissions:'runner_policy',semanticTools:'runner_bridge',structuredInput:'paperclip.question_set.v1'}});break;", + " case 'session.open': identity={kind:'acpx',normalizedSessionId:p.normalizedSessionId,acpxRecordId:'record-1',backendSessionId:'backend-1',agentSessionId:'agent-1',profileDigest,workspaceDigest:'sha256:'+'2'.repeat(64),requestedModel:p.model,effectiveModel:p.model,permissionMode:p.permissionMode,mode:p.mode,piThinkingLevel:p.piThinkingLevel,claudeEffort:p.claudeEffort,grokEffort:p.grokEffort,grokDisableWebSearch:p.grokDisableWebSearch,providerLifetimeFenceCandidates:ports};reply(r,{sidecarPid:process.pid,identity,status:{},turnControls:{steering:false,queuedFollowUp:false}});break;", + " case 'run.attach': runId=p.runId; reply(r,{runId,catalogRevision:p.catalogRevision});break;", + " case 'session.goal.get': reply(r,{schema:'paperclip.session_goal.snapshot.v1',goal:null,workingNow:false,sessionGoals:{availability:'unsupported',actions:[],autonomousUpdates:false,persistentAcrossResume:false,maxObjectiveChars:4000,tokenBudgetControl:false,usageReporting:false}});break;", + " case 'turn.start': turnId=p.turnId;reply(r,{turnId,turnControls:{steering:false,queuedFollowUp:false}});event('runtime.event',{type:'text_delta',text:'started'});break;", + " case 'session.snapshot': reply(r,{identity,runId,turnId,pendingRuntimeRequests:[]});break;", + " case 'turn.cancel': reply(r,{cancelled:true});event('runtime.turn_terminal',{status:'interrupted'});turnId=null;break;", + " case 'session.suspend': reply(r,{identity});break;", + " case 'session.close': reply(r,{closed:true});break;", + " default: reply(r,{});", + " }", + " });", + " setInterval(()=>{if(!turnId)return;let action;try{action=fs.readFileSync(root+'/action','utf8');}catch{return;}if(action===lastAction)return;lastAction=action;if(action==='progress'){event('runtime.event',{type:'text_delta',text:'work-during-outage'});fs.writeFileSync(root+'/progress-receipt',String(process.pid));}if(action==='complete'){event('runtime.turn_terminal',{status:'completed'});turnId=null;fs.writeFileSync(root+'/complete-receipt',String(process.pid));}if(action==='exit'){fs.writeFileSync(root+'/exit-receipt',String(process.pid));process.exit(9);}},10);", + "})().catch(e=>{process.stderr.write(String(e));process.exit(1);});", + ].join("\n"); +} + +async function fixture() { + const root = mkdtempSync(resolve(tmpdir(), "acpx-controller-recovery-")); + const stateDirectory = resolve(root, "control-plane"); + const script = resolve(root, "sidecar.cjs"); + writeFileSync(script, sidecarSource(root), { mode: 0o700 }); + const runtime = resolve(root, "runtime"); + const descriptor = { kind: "acpx", provider: "acpx", driver: "acpx_runtime", providerVersion: profiles.acpxVersion, + agent: "claude", model: "scripted-model", acpxVersion: profiles.acpxVersion, + agentServerPackage: profile.agentServerPackage, agentServerVersion: profile.agentServerVersion, + agentRuntimePackage: profile.agentRuntimePackage, agentRuntimeVersion: profile.agentRuntimeVersion, + commandDigest: profile.commandDigest, sidecarCommand: process.execPath, sidecarArgs: [script], + runtimeDirectory: runtime, normalizedSessionId: identity.normalizedSessionId, runId: identity.runId, + cwd: root, instructions: "Do the supplied synthetic task.", permissionMode: "approve-reads", permissionModePinned: true, + runtimeContext: null }; + let checkpoint!: DurablePrpCheckpoint; + const create = () => { + const controller = new DurablePrpControlPlane({ stateDirectory, identity, expectedRunnerVersion, expectedRunnerDigest }); + controller.setCheckpointPersistence(async saved => { checkpoint = saved; }); + return controller; + }; + let controller = create(); + await controller.start(); + const port = Number(new URL(controller.connectUrl).port); + const ticket = controller.issueBootstrapTicket(); + await controller.flushCheckpoint(); + let diagnostics = ""; + const args = ["--connect-url",controller.connectUrl,"--state-dir",resolve(root,"runner"), + "--runner-id",identity.runnerInstanceId,"--environment-lease-id",identity.environmentLeaseId, + "--run-id",identity.runId,"--session-id",identity.normalizedSessionId,"--turn-id",identity.turnId,"--item-id",identity.itemId, + "--runner-version",expectedRunnerVersion,"--runner-digest",expectedRunnerDigest,"--max-runtime-ms","0", + "--reconnect-grace-ms","15000","--reconnect-delay-ms","25","--acpx-launch-authority-digest","sha256:"+"d".repeat(64), + "--acpx-sidecar-command",process.execPath,"--acpx-sidecar-command-sha256",digest(process.execPath), + "--acpx-sidecar-script",script,"--acpx-sidecar-script-sha256",digest(script)]; + const startedAt = new Date().toISOString(); + const child = spawn(binary,args,{env:{PATH:process.env.PATH,PAPERCLIP_RUNNER_BOOTSTRAP_TICKET:ticket},stdio:["ignore","ignore","pipe"],detached:true}); + const exit = once(child,"exit"); + child.stderr!.on("data",bytes=>{diagnostics=(diagnostics+String(bytes)).slice(-4096);}); + let commandNumber=0; + async function command(type:string,payload:Record={}) { + const id="command-"+(++commandNumber); + controller.queueCommand(type,payload,id,true); + await vi.waitFor(()=>{ + expect(child.exitCode,"runner exit; "+diagnostics).toBeNull(); + expect(child.signalCode,"runner signal; "+diagnostics).toBeNull(); + expect(["completed","failed"]).toContain(controller.getCommand(id)?.status); + },{timeout:8000}); + return controller.getCommand(id)!; + } + const ready = async () => { + expect(await command("run.prepare",{provider:descriptor})).toMatchObject({status:"completed"}); + const opened = await command("session.open"); + expect(opened, JSON.stringify(opened)).toMatchObject({status:"completed"}); + }; + const stopped = async () => { + await controller.stop(); await controller.drainPendingConnectionProcessing(); await controller.flushCheckpoint(); + }; + const restoreCheckpoint = () => { + rmSync(stateDirectory,{recursive:true,force:true}); + decodeDurablePrpCheckpoint(checkpoint.records).restoreSessionDirectory(root); + }; + return {root,child,exit,command,ready,startedAt, + controller:()=>controller, + stopController:stopped, + restoreCheckpoint, + adoptController:async(next:DurablePrpControlPlane)=>{ + controller=next;await controller.start(port); + return {connectUrl:controller.connectUrl,release:()=>controller.stop()}; + }, + replaceController:async()=>{ + restoreCheckpoint(); + controller=create();await controller.start(port); + }, + launches:()=>readFileSync(resolve(root,"launches.jsonl"),"utf8").trim().split("\n").map(line=>JSON.parse(line)), + cleanup:async()=>{ + await controller.stop(); + if(child.exitCode===null && child.signalCode===null){try{process.kill(-child.pid!,"SIGKILL");}catch{}} + await exit; + try { + for (const line of readFileSync(resolve(root,"launches.jsonl"),"utf8").trim().split("\n")) { + const launch = JSON.parse(line) as { pid: number; ports: number[] }; + await vi.waitFor(async () => { + for (const port of launch.ports.slice(0, 2)) await new Promise((resolveFree, reject) => { + const probe = createNetServer(); + probe.once("error", reject); + probe.listen(port, "127.0.0.1", () => probe.close(error => error ? reject(error) : resolveFree())); + }); + }, { timeout: 3_000 }); + } + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; + } + rmSync(root,{recursive:true,force:true}); + }, + }; +} + +it("keeps the original native ACPX process and active turn through controller replacement",async()=>{ + const run=await fixture(); + try{ + await run.ready(); + expect(await run.command("turn.start",{turnId:"provider-turn-1",text:"Keep working"})).toMatchObject({status:"completed"}); + const pid=run.launches()[0].pid; + await run.stopController(); + writeFileSync(resolve(run.root,"action"),"progress"); + await vi.waitFor(()=>expect(readFileSync(resolve(run.root,"progress-receipt"),"utf8")).toBe(String(pid)),{timeout:3000}); + expect(()=>process.kill(pid,0)).not.toThrow(); + await run.replaceController(); + const result=await run.command("session.snapshot",{includePendingRuntimeRequests:true}); + expect(result).toMatchObject({status:"completed",result:{result:{runtimeRequestsLive:true,activeProviderTurnId:"provider-turn-1"}}}); + expect(run.launches()).toHaveLength(1); + expect(()=>process.kill(pid,0)).not.toThrow(); + writeFileSync(resolve(run.root,"action"),"complete"); + await vi.waitFor(()=>expect(run.controller().store.state.committedEvents.some(event=>event.eventType==="turn.completed")).toBe(true),{timeout:5000}); + }finally{await run.cleanup();} +},20000); + +it("adopts the live ACPX execution through the production native transport",async()=>{ + const run=await fixture(); + let adopted:ReturnType|undefined; + try{ + await run.ready(); + expect(await run.command("turn.start",{turnId:"provider-turn-1",text:"Retain this exact execution"})).toMatchObject({status:"completed"}); + const pid=run.launches()[0].pid; + await run.stopController();run.restoreCheckpoint(); + const launcher=vi.fn(()=>{throw new Error("duplicate runner launch");}); + const options:Parameters[0]={ + provider:"acpx",acpxAgent:"claude",stateDirectory:run.root, + runnerBinary:binary,runnerArtifact:{version:expectedRunnerVersion,digest:expectedRunnerDigest},prpIdentity:identity, + resumeProviderSession:{driverSessionId:"record-1",providerSessionId:"agent-1"}, + runnerProcessLauncher:launcher,controlPlaneRegistration:run.adoptController, + runnerReconnectGraceMs:10000, + adoptExistingRunner:{pid:run.child.pid!,processGroupId:run.child.pid!,startedAt:run.startedAt, + isAlive:()=>{try{process.kill(run.child.pid!,0);return true;}catch{return false;}}, + signal:signal=>{try{process.kill(-run.child.pid!,signal);return true;}catch{return false;}}}, + }; + adopted=createCapabilityRunnerdCodexTransport(options); + adopted.transport.setServerRequestHandler(async()=>({success:true,contentItems:[]})); + expect(await adopted.transport.request("thread/read",{})).toMatchObject({thread:{id:"record-1",sessionId:"agent-1",turns:[{id:"provider-turn-1",status:"inProgress"}]}}); + expect(adopted.evidence().runnerPid).toBe(run.child.pid); + expect(run.launches()).toHaveLength(1); + expect(()=>process.kill(pid,0)).not.toThrow(); + expect(launcher).not.toHaveBeenCalled(); + await adopted.detachControllerForRestart(); + await adopted.transport.close("old controller finalizer"); + expect(()=>process.kill(pid,0)).not.toThrow(); + adopted=createCapabilityRunnerdCodexTransport(options); + adopted.transport.setServerRequestHandler(async()=>({success:true,contentItems:[]})); + expect(await adopted.transport.request("thread/read",{})).toMatchObject({thread:{id:"record-1",sessionId:"agent-1",turns:[{id:"provider-turn-1",status:"inProgress"}]}}); + expect(adopted.evidence().runnerPid).toBe(run.child.pid); + expect(run.launches()).toHaveLength(1); + expect(launcher).not.toHaveBeenCalled(); + }finally{ + await adopted?.transport.close().catch(()=>undefined); + await run.cleanup(); + } +},25000); + +it("recovers a turn that completed while its controller was offline",async()=>{ + const run=await fixture(); + try{ + await run.ready(); + expect(await run.command("turn.start",{turnId:"provider-turn-1",text:"Finish while disconnected"})).toMatchObject({status:"completed"}); + const pid=run.launches()[0].pid; + await run.stopController(); + writeFileSync(resolve(run.root,"action"),"complete"); + await vi.waitFor(()=>expect(readFileSync(resolve(run.root,"complete-receipt"),"utf8")).toBe(String(pid)),{timeout:3000}); + await run.replaceController(); + await vi.waitFor(()=>expect(run.controller().store.state.committedEvents.some(event=>event.eventType==="turn.completed")).toBe(true),{timeout:5000}); + expect(await run.command("session.snapshot",{includePendingRuntimeRequests:true})).toMatchObject({status:"completed",result:{result:{runtimeRequestsLive:true,activeProviderTurnId:null}}}); + expect(run.launches()).toHaveLength(1); + expect(()=>process.kill(pid,0)).not.toThrow(); + }finally{await run.cleanup();} +},20000); + +it("distinguishes a provider-process crash from controller replacement",async()=>{ + const run=await fixture(); + try{ + await run.ready(); + expect(await run.command("turn.start",{turnId:"provider-turn-1",text:"Keep working"})).toMatchObject({status:"completed"}); + const pid=run.launches()[0].pid; + await run.stopController(); + writeFileSync(resolve(run.root,"action"),"exit"); + await vi.waitFor(()=>expect(readFileSync(resolve(run.root,"exit-receipt"),"utf8")).toBe(String(pid)),{timeout:3000}); + await vi.waitFor(async()=>{ + for(const port of run.launches()[0].ports.slice(0,2)) await new Promise((resolveFree,reject)=>{ + const probe=createNetServer();probe.once("error",reject); + probe.listen(port,"127.0.0.1",()=>probe.close(error=>error?reject(error):resolveFree())); + }); + },{timeout:3000}); + await run.replaceController(); + const result=await run.command("session.snapshot",{includePendingRuntimeRequests:true}); + expect(result,JSON.stringify(result)).toMatchObject({status:"failed",result:{result:{code:"command_execution_failed"}}}); + const message = (result.result as {result:{message:string}}).result.message; + const lossReasons = [ + "ACPX request snapshot requires a live provider", + "ACPX sidecar request transport failed at session.snapshot: command serialization failed: Broken pipe (os error 32)", + "ACPX sidecar request transport failed at session.snapshot: failed to write process command: Broken pipe (os error 32)", + "ACPX sidecar exited at session.snapshot: exitCode=Some(9) signal=None", + "ACPX sidecar closed stdout at session.snapshot", + ]; + expect(lossReasons.map(reason => `ACPX live request snapshot failed: ${reason}`)).toContain(message); + expect(run.launches()).toHaveLength(1); + }finally{await run.cleanup();} +},20000); + +it("requires a surviving provider when requesting a live snapshot after explicit close",async()=>{ + const run=await fixture(); + try{ + await run.ready(); + expect(await run.command("session.close")).toMatchObject({status:"completed"}); + expect(run.child.exitCode).toBeNull(); + await run.stopController(); + await run.replaceController(); + const result=await run.command("session.snapshot",{includePendingRuntimeRequests:true}); + expect(result).toMatchObject({status:"failed",result:{result:{message:"ACPX request snapshot requires the surviving provider"}}}); + expect(run.launches()).toHaveLength(1); + }finally{await run.cleanup();} +},20000);