Files
PaperClipAI/tests/runner-e2e/chat-catalog-domains/runtime-provider-cleanup.ts
T
f4337bb5fb test(evals): land the chat catalog eval harness on master (#19614)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature evals measure whether agents complete real chat
requests through the product
> - The chat catalog eval harness and its repairs live on the
`chat-eval/master-candidate` branch, not on master
> - Today's work made 139 of 242 chat evals pass in verified single-case
live runs. Each product fix landed on master as its own pull request
> - The eval-side changes (setups, graders, fault seams, evidence
capture) still exist only on the candidate
> - This pull request lands the harness on master. Master stays the
source of truth for product behavior
> - The benefit is one tree: evals run against master commits, and
product fixes reach the evals as soon as they merge

## Linked Issues or Issue Description

Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs
#20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409

**What existing behavior does this improve?**
The chat catalog eval harness and the product it evaluates live on a
long-running branch outside master. Eval fixes do not reach master, so
master-based eval runs use old setups and graders.

**Subsystem affected**
`tests/runner-e2e` chat catalog harness, plus a small set of product
changes that exist only on the candidate.

**Proposed change**
Land the harness on master. Every product fix from this eval work
already merged to master as its own pull request. Conflicts keep
master's product behavior.

## What Changed

- Most of the diff (808 of 851 files) is under `tests/runner-e2e/`:
chat-catalog domains, scenarios, descriptors, the TestOps manifest,
private evidence tooling, and their tests.
- Graders now check saved state, setup literals and rendered or clicked
links. Many keyword and regex checks on free-form replies are removed.
Each changed grader has a bumped `oracleVersion` and calibration tests
(a valid reply, a reworded valid reply, and broken variants that must
fail).
- The shared link helper `hasLink` no longer treats a list link (for
example `/COM/issues?view=mine`) as a link to every task without a URL
key.
- Setup fixes build real state through supported product routes: routine
history, agent grants through the board grant route, budget holds with a
real cap, and a storage-layer archive fault for
`inbox-search.archive-partial-failure`.
- Browser boundary journeys (sign-in, chat-disabled, onboarding) now
write the standard pass evidence files.
- Retargeted cases, approved by the maintainer:
`agent-chat.chat-disabled` (Conference Room gate),
`agents.restore-instructions` (no saved instruction history),
`access.manage-membership` (member permission edits stay board-only).
- The private evidence trusted files (`manual-private-handoff.ts`,
`manual-private-retention.ts`, `manual-private-report-assets.ts`) are
byte-identical to master.
- Product files outside `tests/` (43 files, +1,328 −49) carry
candidate-only changes that master does not yet have. Most are tests.
The main code changes are in `request-human-input.ts`,
`heartbeat-run-summary.ts`, `native-completion-feedback.ts`,
`native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and
`IssueChatLongThreadPerf.tsx`.

## Verification

- `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts
chat-catalog-consistency.test.ts unit-launcher.test.mjs
--testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests
pass.
- `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no change.
- `tsc` for `server` and `tests/runner-e2e`: clean.
- Live single-case verification: 139 of 242 cases passed on paid runs,
and a person read the saved evidence for each. These are single-case
runs, not one full-cohort run.

## Risks

- Large diff. Most of it is test-only, but the 43 non-test files change
product code paths. Review those files first.
- Some graders are now less strict on reply wording. A person must read
replies for meaning on those cases.
- 42 cases need disposable outside accounts (Notion, Slack, GitHub,
Daytona and others) and stay blocked. `access.invite-teammate` stays
failing until local agent isolation lands, because the runner's durable
state keeps a live invite token.
- The paid workflow compares the trusted private-evidence files with
master. They are unchanged, so this risk is low.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution, in Claude Code. Subagents used the same model.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 08:02:11 -06:00

134 lines
15 KiB
TypeScript

import { execFile } from "node:child_process";
import { promisify } from "node:util";
import { createHash } from "node:crypto";
import { readFile, realpath } from "node:fs/promises";
import path from "node:path";
import { fileURLToPath } from "node:url";
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
import { collectRunEvents } from "../run-observations.js";
import { readProcessStartedAt } from "../../../server/src/services/hot-restart.js";
import { verifyOwnedNativeRunner, ownedNativeExecutablePath } from "./runtime-native-fault.js";
interface ProviderIdentity { pid: number; parentPid: number; uid: number; startedAt: string; executable: string; digest: string; appServer: boolean; openCodeServe: boolean; proxyScript?: string; proxyDigest?: string }
interface SuspendedOwner { identity: ProviderIdentity; runId: string; timer: ReturnType<typeof setTimeout> }
const suspended = new WeakMap<ChatCatalogContext, SuspendedOwner>();
const retainedProvider = new WeakMap<ChatCatalogContext, ProviderIdentity>();
export function matchesOwnedProvider(input: { identity: ProviderIdentity; expected: { providerPid: number; providerStartedAt: string; runnerPid: number; providerParentPid?: number; uid: number; runStartedAt: string; kind?: "codex" | "opencode"; executable?: string; digest?: string; proxyScript?: string; proxyDigest?: string } }) {
const { identity: p, expected: e } = input;
return Number.isSafeInteger(p.pid) && p.pid > 1 && p.pid !== process.pid && p.pid === e.providerPid && p.pid !== e.runnerPid && p.parentPid === (e.providerParentPid ?? e.runnerPid) && p.uid === e.uid
&& p.startedAt === e.providerStartedAt && Date.parse(p.startedAt) + 1_000 >= Date.parse(e.runStartedAt)
&& /^[a-f0-9]{64}$/.test(p.digest)
&& (e.kind === "opencode" && !e.proxyScript
// Direct OpenCode server: the runner spawns the pinned `opencode serve` itself.
? p.openCodeServe && !p.proxyScript && !p.appServer
: e.kind === "opencode"
? !!e.executable && !!e.proxyScript && /^[a-f0-9]{64}$/.test(e.digest ?? "") && /^[a-f0-9]{64}$/.test(e.proxyDigest ?? "")
&& p.executable === e.executable && p.digest === e.digest && p.proxyScript === e.proxyScript && p.proxyDigest === e.proxyDigest && !p.appServer
: p.appServer === true && path.basename(p.executable) === "codex");
}
async function providerIdentity(pid: number): Promise<ProviderIdentity> {
try {
const ps = async (field: string) => (await promisify(execFile)("/bin/ps", ["-p", String(pid), "-o", `${field}=`], { timeout: 5_000, maxBuffer: 65_536 })).stdout.trim();
const [uid, parent, executable, command, startedAt] = await Promise.all([ps("uid"), ps("ppid"), ps("comm"), ps("args"), readProcessStartedAt(pid)]);
// A pinned binary may be executed from an inherited descriptor (/proc/self/fd/N); its real
// executable is then only reachable through the kernel's /proc/<pid>/exe link.
// An in-memory (memfd) binary has no resolvable path, but the kernel link still reads its bytes.
const canonical = await realpath(ownedNativeExecutablePath(pid, executable)).catch(() => realpath(`/proc/${pid}/exe`)).catch(() => `/proc/${pid}/exe`);
const args = command.split(/\s+/);
// node may carry runtime flags before the script; the absolute proxy script is the last argument.
const script = args.at(-1)!, flags = args.slice(1, -1);
const proxy = args.length >= 2 && path.isAbsolute(script) && path.basename(script) === "opencode-app-server-proxy.cjs" && flags.every(arg => /^--[a-z][a-z0-9-]*(?:=[^\s]*)?$/.test(arg)) ? await realpath(script) : undefined;
const proxyDigest = proxy ? createHash("sha256").update(await readFile(proxy)).digest("hex") : undefined;
if (!startedAt) throw new Error("missing start identity");
return { pid, parentPid: Number(parent), uid: Number(uid), startedAt, executable: canonical,
digest: createHash("sha256").update(await readFile(canonical)).digest("hex"), appServer: /(?:^|\s)app-server(?=\s|$)/.test(command),
// `opencode serve` runs either as its own binary or as node running the opencode launcher script.
openCodeServe: args.includes("serve") && (/^opencode/.test(path.basename(canonical)) || /^\/(?:proc\/self|dev)\/fd\/\d+$/.test(args[0] ?? "")
|| args.slice(1, args.indexOf("serve")).some(arg => path.isAbsolute(arg) && /^opencode(?:\.[cm]?js)?$/.test(path.basename(arg)))), proxyScript: proxy, proxyDigest };
} catch { throw new Error("Provider cleanup fixture could not qualify the exact owned process; private diagnostics withheld"); }
}
/** The single qualifying provider among the verified runner's descendants (depth <= 4). The
* launch path may wrap the provider, so it need not be a direct child. Records a closed summary of
* what it saw (depth, executable basename, qualification flags) without command lines. */
async function soleProviderDescendant(ctx: ChatCatalogContext, runnerPid: number, kind: "codex" | "opencode"): Promise<{ pid: number; parentPid: number } | undefined> {
const rows = (await promisify(execFile)("/bin/ps", ["-A", "-o", "pid=,ppid="], { timeout: 5_000, maxBuffer: 1_048_576 })).stdout
.split("\n").map(line => line.trim().split(/\s+/).map(Number)).filter(([pid, parent]) => Number.isSafeInteger(pid) && Number.isSafeInteger(parent));
const seen: Array<{ depth: number; command: string | null; argCount: number | null; scripts: string[]; hasServe: boolean; executable: string | null; proxy: boolean; appServer: boolean; openCodeServe: boolean }> = [];
const providers: Array<{ pid: number; parentPid: number }> = [];
let frontier = [runnerPid];
for (let depth = 1; depth <= 4 && frontier.length; depth += 1) {
const next = rows.filter(([, parent]) => frontier.includes(parent!)).map(([pid, parent]) => ({ pid: pid!, parent: parent! }));
for (const { pid, parent } of next) {
const identity = await providerIdentity(pid).catch(() => undefined);
const args = await promisify(execFile)("/bin/ps", ["-p", String(pid), "-o", "args="], { timeout: 5_000, maxBuffer: 65_536 }).then(out => out.stdout.trim().split(/\s+/), () => null);
const command = await promisify(execFile)("/bin/ps", ["-p", String(pid), "-o", "comm="], { timeout: 5_000, maxBuffer: 65_536 }).then(out => path.basename(out.stdout.trim()) || null, () => null);
seen.push({ depth, command, argCount: args?.length ?? null, scripts: (args ?? []).filter(arg => path.isAbsolute(arg)).map(arg => path.basename(arg)).slice(0, 8), hasServe: !!args?.includes("serve"), executable: identity ? path.basename(identity.executable) : null, proxy: !!identity?.proxyScript, appServer: !!identity?.appServer, openCodeServe: !!identity?.openCodeServe });
if (identity && (kind === "opencode" ? !!identity.proxyScript || identity.openCodeServe : identity.appServer)) providers.push({ pid, parentPid: parent });
}
frontier = next.map(row => row.pid);
}
await ctx.input.evidence("runtime-cleanup-provider-search.json", { kind, descendants: seen, qualifying: providers.length });
return providers.length === 1 ? providers[0] : undefined;
}
/** Upper bound on the hold, only so an aborted attempt never leaves a stopped process behind. The
* case releases the hold itself, deterministically, once the follow-up message is durably accepted. */
const MAXIMUM_SUSPENSION_MS = 120_000;
/** This does not signal any group or control ACK timing. Actual public Stop must immediately follow. */
export async function suspendOwnedProviderForCleanup(ctx: ChatCatalogContext, binding: { runId: string; issueId: string; agentId: string }) {
if (ctx.scenario.id !== "recovery.post-stop-message" || suspended.has(ctx)) throw new Error("Bounded provider cleanup suspension is exclusive to the selected post-stop scenario");
const { run, identity: runner } = await verifyOwnedNativeRunner(ctx, binding);
const checkpoint = run.runnerProfileJson?.sessionCheckpoint as { process?: { providerPid?: number; providerProcessStartedAt?: string } } | undefined;
const provider = run.runnerProfileJson?.nativeExecutionInput as { provider?: { kind?: string } } | undefined;
const kind = provider?.provider?.kind;
const events = await collectRunEvents<{ seq?: number; payload?: { prpEvent?: { runId?: string; normalizedSessionId?: string; sourceKind?: string; sourceInstanceId?: string; eventType?: string; payload?: { processId?: number; processStartedAt?: string | null; providerSessionId?: string; driverSessionId?: string } } } }>((afterSeq, limit) => ctx.input.api.get(`/api/heartbeat-runs/${run.id}/events?afterSeq=${afterSeq}&limit=${limit}`));
const sessionReceipts = events.map(row => row.payload?.prpEvent).filter(e => e?.runId === run.id && e.sourceKind === "runner" && e.sourceInstanceId === run.runnerInstanceId && e.normalizedSessionId === run.nativeSessionId
&& ["session.started", "session.resumed"].includes(e.eventType ?? "") && !!(e.payload?.providerSessionId ?? e.payload?.driverSessionId));
if (kind !== "codex" && kind !== "opencode") throw new Error("Actual public native session lacks a supported local provider kind");
if (!sessionReceipts.length) throw new Error("Provider suspension lacks the exact durable native session receipt");
// The runnerd transport records the provider in the checkpoint. When the session runs inside the
// verified runnerd itself, the provider is its direct child: the one qualifying child is chosen,
// and every ownership check below still applies to it.
const descendant = checkpoint?.process?.providerPid ? undefined : await soleProviderDescendant(ctx, runner.pid, kind);
const pid = checkpoint?.process?.providerPid ?? descendant?.pid;
const started = checkpoint?.process?.providerPid ? checkpoint.process.providerProcessStartedAt : pid ? await readProcessStartedAt(pid) : undefined;
if (!Number.isSafeInteger(pid) || !pid || !started) throw new Error("Actual public native session lacks a qualified local provider PID/start identity");
const viaProxy = kind === "opencode" && !!(await providerIdentity(pid)).proxyScript;
const executable = viaProxy ? await realpath(process.execPath) : undefined;
const proxyScript = viaProxy ? await realpath(fileURLToPath(new URL("../../../packages/paperclip-runner/dist/cli/opencode-app-server-proxy.cjs", import.meta.url))) : undefined;
const digest = executable ? createHash("sha256").update(await readFile(executable)).digest("hex") : undefined;
const proxyDigest = proxyScript ? createHash("sha256").update(await readFile(proxyScript)).digest("hex") : undefined;
if (checkpoint?.process?.providerPid && !sessionReceipts.some(e => e?.payload?.processId === pid)) throw new Error("Provider suspension lacks the exact durable native session/PID receipt");
const expected: Parameters<typeof matchesOwnedProvider>[0]["expected"] = { providerPid: pid, providerStartedAt: started, runnerPid: runner.pid, ...(descendant ? { providerParentPid: descendant.parentPid } : {}), uid: runner.uid, runStartedAt: String(run.startedAt), kind, executable, digest, proxyScript, proxyDigest };
const first = await providerIdentity(pid), second = await providerIdentity(pid);
if (!matchesOwnedProvider({ identity: first, expected }) || !matchesOwnedProvider({ identity: second, expected }) || JSON.stringify(first) !== JSON.stringify(second)) throw new Error("Provider cleanup suspension rejected changed or unowned process identity");
process.kill(pid, "SIGSTOP"); retainedProvider.set(ctx, second);
const timer = setTimeout(() => { void releaseOwnedCleanupProvider(ctx).catch(() => undefined); }, MAXIMUM_SUSPENSION_MS); timer.unref(); suspended.set(ctx, { identity: second, runId: run.id, timer });
await ctx.input.evidence("runtime-cleanup-owned-provider.json", { runId: run.id, companyId: run.companyId, issueId: run.nativeIssueId, agentId: run.agentId, providerPid: pid, providerStartedAt: started,
providerKind: kind, cleanupBoundary: kind === "opencode" ? viaProxy ? "owned-protocol-proxy-retirement" : "owned-opencode-serve-retirement" : "owned-app-server-retirement", sdkServerExitRequired: false, sameUid: true, directOwnedRunnerParent: true, proxyScriptSha256: proxyDigest, executableSha256: second.digest, durableSessionReceipt: true, identityReadTwice: true, signal: "SIGSTOP", groupSignalled: false, maximumSuspensionMs: MAXIMUM_SUSPENSION_MS, releasedBy: "follow-up-message-accepted" });
return { providerPid: pid, providerStartedAt: started, runId: run.id };
}
export async function releaseOwnedCleanupProvider(ctx: ChatCatalogContext) {
const owned = suspended.get(ctx); if (!owned) return; clearTimeout(owned.timer);
try { process.kill(owned.identity.pid, 0); } catch (error) { if ((error as NodeJS.ErrnoException).code === "ESRCH") { suspended.delete(ctx); return; } throw new Error("Owned provider cleanup liveness could not be verified"); }
const fresh = await providerIdentity(owned.identity.pid);
if (fresh.pid !== owned.identity.pid || fresh.uid !== owned.identity.uid || fresh.startedAt !== owned.identity.startedAt || fresh.executable !== owned.identity.executable || fresh.digest !== owned.identity.digest || fresh.proxyScript !== owned.identity.proxyScript || fresh.proxyDigest !== owned.identity.proxyDigest)
throw new Error("Provider cleanup refuses signalling a reused or changed PID");
process.kill(fresh.pid, "SIGCONT"); suspended.delete(ctx);
}
/** True only while this fixture still holds the exact owned provider stopped: same PID, start
* identity, executable digest, and a kernel state of stopped (T). The old provider's cleanup cannot
* finish while this holds, so anything observed under it happened during that cleanup. */
export async function ownedCleanupProviderIsHeld(ctx: ChatCatalogContext) {
const owned = suspended.get(ctx); if (!owned) return false;
try { process.kill(owned.identity.pid, 0); } catch { return false; }
const state = await promisify(execFile)("/bin/ps", ["-p", String(owned.identity.pid), "-o", "stat="], { timeout: 5_000, maxBuffer: 4_096 }).then(out => out.stdout.trim(), () => "");
const fresh = await providerIdentity(owned.identity.pid).catch(() => undefined);
return state.startsWith("T") && !!fresh && fresh.startedAt === owned.identity.startedAt && fresh.uid === owned.identity.uid && fresh.executable === owned.identity.executable && fresh.digest === owned.identity.digest;
}
export async function ownedCleanupProviderHasExited(ctx: ChatCatalogContext) {
const identity = retainedProvider.get(ctx); if (!identity) throw new Error("No retained exact provider identity for cleanup verification");
try { process.kill(identity.pid, 0); } catch (error) { if ((error as NodeJS.ErrnoException).code === "ESRCH") return true; return false; }
const startedAt = await readProcessStartedAt(identity.pid);
return !!startedAt && startedAt !== identity.startedAt;
}