mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat-feature evals measure whether agents complete real chat requests through the product > - The chat catalog eval harness and its repairs live on the `chat-eval/master-candidate` branch, not on master > - Today's work made 139 of 242 chat evals pass in verified single-case live runs. Each product fix landed on master as its own pull request > - The eval-side changes (setups, graders, fault seams, evidence capture) still exist only on the candidate > - This pull request lands the harness on master. Master stays the source of truth for product behavior > - The benefit is one tree: evals run against master commits, and product fixes reach the evals as soon as they merge ## Linked Issues or Issue Description Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs #20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409 **What existing behavior does this improve?** The chat catalog eval harness and the product it evaluates live on a long-running branch outside master. Eval fixes do not reach master, so master-based eval runs use old setups and graders. **Subsystem affected** `tests/runner-e2e` chat catalog harness, plus a small set of product changes that exist only on the candidate. **Proposed change** Land the harness on master. Every product fix from this eval work already merged to master as its own pull request. Conflicts keep master's product behavior. ## What Changed - Most of the diff (808 of 851 files) is under `tests/runner-e2e/`: chat-catalog domains, scenarios, descriptors, the TestOps manifest, private evidence tooling, and their tests. - Graders now check saved state, setup literals and rendered or clicked links. Many keyword and regex checks on free-form replies are removed. Each changed grader has a bumped `oracleVersion` and calibration tests (a valid reply, a reworded valid reply, and broken variants that must fail). - The shared link helper `hasLink` no longer treats a list link (for example `/COM/issues?view=mine`) as a link to every task without a URL key. - Setup fixes build real state through supported product routes: routine history, agent grants through the board grant route, budget holds with a real cap, and a storage-layer archive fault for `inbox-search.archive-partial-failure`. - Browser boundary journeys (sign-in, chat-disabled, onboarding) now write the standard pass evidence files. - Retargeted cases, approved by the maintainer: `agent-chat.chat-disabled` (Conference Room gate), `agents.restore-instructions` (no saved instruction history), `access.manage-membership` (member permission edits stay board-only). - The private evidence trusted files (`manual-private-handoff.ts`, `manual-private-retention.ts`, `manual-private-report-assets.ts`) are byte-identical to master. - Product files outside `tests/` (43 files, +1,328 −49) carry candidate-only changes that master does not yet have. Most are tests. The main code changes are in `request-human-input.ts`, `heartbeat-run-summary.ts`, `native-completion-feedback.ts`, `native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and `IssueChatLongThreadPerf.tsx`. ## Verification - `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts chat-catalog-consistency.test.ts unit-launcher.test.mjs --testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests pass. - `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: no change. - `tsc` for `server` and `tests/runner-e2e`: clean. - Live single-case verification: 139 of 242 cases passed on paid runs, and a person read the saved evidence for each. These are single-case runs, not one full-cohort run. ## Risks - Large diff. Most of it is test-only, but the 43 non-test files change product code paths. Review those files first. - Some graders are now less strict on reply wording. A person must read replies for meaning on those cases. - 42 cases need disposable outside accounts (Notion, Slack, GitHub, Daytona and others) and stay blocked. `access.invite-teammate` stays failing until local agent isolation lands, because the runner's durable state keeps a live invite token. - The paid workflow compares the trusted private-evidence files with master. They are unchanged, so this risk is low. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution, in Claude Code. Subagents used the same model. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing>
134 lines
15 KiB
TypeScript
134 lines
15 KiB
TypeScript
import { execFile } from "node:child_process";
|
|
import { promisify } from "node:util";
|
|
import { createHash } from "node:crypto";
|
|
import { readFile, realpath } from "node:fs/promises";
|
|
import path from "node:path";
|
|
import { fileURLToPath } from "node:url";
|
|
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
|
|
import { collectRunEvents } from "../run-observations.js";
|
|
import { readProcessStartedAt } from "../../../server/src/services/hot-restart.js";
|
|
import { verifyOwnedNativeRunner, ownedNativeExecutablePath } from "./runtime-native-fault.js";
|
|
interface ProviderIdentity { pid: number; parentPid: number; uid: number; startedAt: string; executable: string; digest: string; appServer: boolean; openCodeServe: boolean; proxyScript?: string; proxyDigest?: string }
|
|
interface SuspendedOwner { identity: ProviderIdentity; runId: string; timer: ReturnType<typeof setTimeout> }
|
|
const suspended = new WeakMap<ChatCatalogContext, SuspendedOwner>();
|
|
const retainedProvider = new WeakMap<ChatCatalogContext, ProviderIdentity>();
|
|
export function matchesOwnedProvider(input: { identity: ProviderIdentity; expected: { providerPid: number; providerStartedAt: string; runnerPid: number; providerParentPid?: number; uid: number; runStartedAt: string; kind?: "codex" | "opencode"; executable?: string; digest?: string; proxyScript?: string; proxyDigest?: string } }) {
|
|
const { identity: p, expected: e } = input;
|
|
return Number.isSafeInteger(p.pid) && p.pid > 1 && p.pid !== process.pid && p.pid === e.providerPid && p.pid !== e.runnerPid && p.parentPid === (e.providerParentPid ?? e.runnerPid) && p.uid === e.uid
|
|
&& p.startedAt === e.providerStartedAt && Date.parse(p.startedAt) + 1_000 >= Date.parse(e.runStartedAt)
|
|
&& /^[a-f0-9]{64}$/.test(p.digest)
|
|
&& (e.kind === "opencode" && !e.proxyScript
|
|
// Direct OpenCode server: the runner spawns the pinned `opencode serve` itself.
|
|
? p.openCodeServe && !p.proxyScript && !p.appServer
|
|
: e.kind === "opencode"
|
|
? !!e.executable && !!e.proxyScript && /^[a-f0-9]{64}$/.test(e.digest ?? "") && /^[a-f0-9]{64}$/.test(e.proxyDigest ?? "")
|
|
&& p.executable === e.executable && p.digest === e.digest && p.proxyScript === e.proxyScript && p.proxyDigest === e.proxyDigest && !p.appServer
|
|
: p.appServer === true && path.basename(p.executable) === "codex");
|
|
}
|
|
async function providerIdentity(pid: number): Promise<ProviderIdentity> {
|
|
try {
|
|
const ps = async (field: string) => (await promisify(execFile)("/bin/ps", ["-p", String(pid), "-o", `${field}=`], { timeout: 5_000, maxBuffer: 65_536 })).stdout.trim();
|
|
const [uid, parent, executable, command, startedAt] = await Promise.all([ps("uid"), ps("ppid"), ps("comm"), ps("args"), readProcessStartedAt(pid)]);
|
|
// A pinned binary may be executed from an inherited descriptor (/proc/self/fd/N); its real
|
|
// executable is then only reachable through the kernel's /proc/<pid>/exe link.
|
|
// An in-memory (memfd) binary has no resolvable path, but the kernel link still reads its bytes.
|
|
const canonical = await realpath(ownedNativeExecutablePath(pid, executable)).catch(() => realpath(`/proc/${pid}/exe`)).catch(() => `/proc/${pid}/exe`);
|
|
const args = command.split(/\s+/);
|
|
// node may carry runtime flags before the script; the absolute proxy script is the last argument.
|
|
const script = args.at(-1)!, flags = args.slice(1, -1);
|
|
const proxy = args.length >= 2 && path.isAbsolute(script) && path.basename(script) === "opencode-app-server-proxy.cjs" && flags.every(arg => /^--[a-z][a-z0-9-]*(?:=[^\s]*)?$/.test(arg)) ? await realpath(script) : undefined;
|
|
const proxyDigest = proxy ? createHash("sha256").update(await readFile(proxy)).digest("hex") : undefined;
|
|
if (!startedAt) throw new Error("missing start identity");
|
|
return { pid, parentPid: Number(parent), uid: Number(uid), startedAt, executable: canonical,
|
|
digest: createHash("sha256").update(await readFile(canonical)).digest("hex"), appServer: /(?:^|\s)app-server(?=\s|$)/.test(command),
|
|
// `opencode serve` runs either as its own binary or as node running the opencode launcher script.
|
|
openCodeServe: args.includes("serve") && (/^opencode/.test(path.basename(canonical)) || /^\/(?:proc\/self|dev)\/fd\/\d+$/.test(args[0] ?? "")
|
|
|| args.slice(1, args.indexOf("serve")).some(arg => path.isAbsolute(arg) && /^opencode(?:\.[cm]?js)?$/.test(path.basename(arg)))), proxyScript: proxy, proxyDigest };
|
|
} catch { throw new Error("Provider cleanup fixture could not qualify the exact owned process; private diagnostics withheld"); }
|
|
}
|
|
/** The single qualifying provider among the verified runner's descendants (depth <= 4). The
|
|
* launch path may wrap the provider, so it need not be a direct child. Records a closed summary of
|
|
* what it saw (depth, executable basename, qualification flags) without command lines. */
|
|
async function soleProviderDescendant(ctx: ChatCatalogContext, runnerPid: number, kind: "codex" | "opencode"): Promise<{ pid: number; parentPid: number } | undefined> {
|
|
const rows = (await promisify(execFile)("/bin/ps", ["-A", "-o", "pid=,ppid="], { timeout: 5_000, maxBuffer: 1_048_576 })).stdout
|
|
.split("\n").map(line => line.trim().split(/\s+/).map(Number)).filter(([pid, parent]) => Number.isSafeInteger(pid) && Number.isSafeInteger(parent));
|
|
const seen: Array<{ depth: number; command: string | null; argCount: number | null; scripts: string[]; hasServe: boolean; executable: string | null; proxy: boolean; appServer: boolean; openCodeServe: boolean }> = [];
|
|
const providers: Array<{ pid: number; parentPid: number }> = [];
|
|
let frontier = [runnerPid];
|
|
for (let depth = 1; depth <= 4 && frontier.length; depth += 1) {
|
|
const next = rows.filter(([, parent]) => frontier.includes(parent!)).map(([pid, parent]) => ({ pid: pid!, parent: parent! }));
|
|
for (const { pid, parent } of next) {
|
|
const identity = await providerIdentity(pid).catch(() => undefined);
|
|
const args = await promisify(execFile)("/bin/ps", ["-p", String(pid), "-o", "args="], { timeout: 5_000, maxBuffer: 65_536 }).then(out => out.stdout.trim().split(/\s+/), () => null);
|
|
const command = await promisify(execFile)("/bin/ps", ["-p", String(pid), "-o", "comm="], { timeout: 5_000, maxBuffer: 65_536 }).then(out => path.basename(out.stdout.trim()) || null, () => null);
|
|
seen.push({ depth, command, argCount: args?.length ?? null, scripts: (args ?? []).filter(arg => path.isAbsolute(arg)).map(arg => path.basename(arg)).slice(0, 8), hasServe: !!args?.includes("serve"), executable: identity ? path.basename(identity.executable) : null, proxy: !!identity?.proxyScript, appServer: !!identity?.appServer, openCodeServe: !!identity?.openCodeServe });
|
|
if (identity && (kind === "opencode" ? !!identity.proxyScript || identity.openCodeServe : identity.appServer)) providers.push({ pid, parentPid: parent });
|
|
}
|
|
frontier = next.map(row => row.pid);
|
|
}
|
|
await ctx.input.evidence("runtime-cleanup-provider-search.json", { kind, descendants: seen, qualifying: providers.length });
|
|
return providers.length === 1 ? providers[0] : undefined;
|
|
}
|
|
/** Upper bound on the hold, only so an aborted attempt never leaves a stopped process behind. The
|
|
* case releases the hold itself, deterministically, once the follow-up message is durably accepted. */
|
|
const MAXIMUM_SUSPENSION_MS = 120_000;
|
|
/** This does not signal any group or control ACK timing. Actual public Stop must immediately follow. */
|
|
export async function suspendOwnedProviderForCleanup(ctx: ChatCatalogContext, binding: { runId: string; issueId: string; agentId: string }) {
|
|
if (ctx.scenario.id !== "recovery.post-stop-message" || suspended.has(ctx)) throw new Error("Bounded provider cleanup suspension is exclusive to the selected post-stop scenario");
|
|
const { run, identity: runner } = await verifyOwnedNativeRunner(ctx, binding);
|
|
const checkpoint = run.runnerProfileJson?.sessionCheckpoint as { process?: { providerPid?: number; providerProcessStartedAt?: string } } | undefined;
|
|
const provider = run.runnerProfileJson?.nativeExecutionInput as { provider?: { kind?: string } } | undefined;
|
|
const kind = provider?.provider?.kind;
|
|
const events = await collectRunEvents<{ seq?: number; payload?: { prpEvent?: { runId?: string; normalizedSessionId?: string; sourceKind?: string; sourceInstanceId?: string; eventType?: string; payload?: { processId?: number; processStartedAt?: string | null; providerSessionId?: string; driverSessionId?: string } } } }>((afterSeq, limit) => ctx.input.api.get(`/api/heartbeat-runs/${run.id}/events?afterSeq=${afterSeq}&limit=${limit}`));
|
|
const sessionReceipts = events.map(row => row.payload?.prpEvent).filter(e => e?.runId === run.id && e.sourceKind === "runner" && e.sourceInstanceId === run.runnerInstanceId && e.normalizedSessionId === run.nativeSessionId
|
|
&& ["session.started", "session.resumed"].includes(e.eventType ?? "") && !!(e.payload?.providerSessionId ?? e.payload?.driverSessionId));
|
|
if (kind !== "codex" && kind !== "opencode") throw new Error("Actual public native session lacks a supported local provider kind");
|
|
if (!sessionReceipts.length) throw new Error("Provider suspension lacks the exact durable native session receipt");
|
|
// The runnerd transport records the provider in the checkpoint. When the session runs inside the
|
|
// verified runnerd itself, the provider is its direct child: the one qualifying child is chosen,
|
|
// and every ownership check below still applies to it.
|
|
const descendant = checkpoint?.process?.providerPid ? undefined : await soleProviderDescendant(ctx, runner.pid, kind);
|
|
const pid = checkpoint?.process?.providerPid ?? descendant?.pid;
|
|
const started = checkpoint?.process?.providerPid ? checkpoint.process.providerProcessStartedAt : pid ? await readProcessStartedAt(pid) : undefined;
|
|
if (!Number.isSafeInteger(pid) || !pid || !started) throw new Error("Actual public native session lacks a qualified local provider PID/start identity");
|
|
const viaProxy = kind === "opencode" && !!(await providerIdentity(pid)).proxyScript;
|
|
const executable = viaProxy ? await realpath(process.execPath) : undefined;
|
|
const proxyScript = viaProxy ? await realpath(fileURLToPath(new URL("../../../packages/paperclip-runner/dist/cli/opencode-app-server-proxy.cjs", import.meta.url))) : undefined;
|
|
const digest = executable ? createHash("sha256").update(await readFile(executable)).digest("hex") : undefined;
|
|
const proxyDigest = proxyScript ? createHash("sha256").update(await readFile(proxyScript)).digest("hex") : undefined;
|
|
if (checkpoint?.process?.providerPid && !sessionReceipts.some(e => e?.payload?.processId === pid)) throw new Error("Provider suspension lacks the exact durable native session/PID receipt");
|
|
const expected: Parameters<typeof matchesOwnedProvider>[0]["expected"] = { providerPid: pid, providerStartedAt: started, runnerPid: runner.pid, ...(descendant ? { providerParentPid: descendant.parentPid } : {}), uid: runner.uid, runStartedAt: String(run.startedAt), kind, executable, digest, proxyScript, proxyDigest };
|
|
const first = await providerIdentity(pid), second = await providerIdentity(pid);
|
|
if (!matchesOwnedProvider({ identity: first, expected }) || !matchesOwnedProvider({ identity: second, expected }) || JSON.stringify(first) !== JSON.stringify(second)) throw new Error("Provider cleanup suspension rejected changed or unowned process identity");
|
|
process.kill(pid, "SIGSTOP"); retainedProvider.set(ctx, second);
|
|
const timer = setTimeout(() => { void releaseOwnedCleanupProvider(ctx).catch(() => undefined); }, MAXIMUM_SUSPENSION_MS); timer.unref(); suspended.set(ctx, { identity: second, runId: run.id, timer });
|
|
await ctx.input.evidence("runtime-cleanup-owned-provider.json", { runId: run.id, companyId: run.companyId, issueId: run.nativeIssueId, agentId: run.agentId, providerPid: pid, providerStartedAt: started,
|
|
providerKind: kind, cleanupBoundary: kind === "opencode" ? viaProxy ? "owned-protocol-proxy-retirement" : "owned-opencode-serve-retirement" : "owned-app-server-retirement", sdkServerExitRequired: false, sameUid: true, directOwnedRunnerParent: true, proxyScriptSha256: proxyDigest, executableSha256: second.digest, durableSessionReceipt: true, identityReadTwice: true, signal: "SIGSTOP", groupSignalled: false, maximumSuspensionMs: MAXIMUM_SUSPENSION_MS, releasedBy: "follow-up-message-accepted" });
|
|
return { providerPid: pid, providerStartedAt: started, runId: run.id };
|
|
}
|
|
export async function releaseOwnedCleanupProvider(ctx: ChatCatalogContext) {
|
|
const owned = suspended.get(ctx); if (!owned) return; clearTimeout(owned.timer);
|
|
try { process.kill(owned.identity.pid, 0); } catch (error) { if ((error as NodeJS.ErrnoException).code === "ESRCH") { suspended.delete(ctx); return; } throw new Error("Owned provider cleanup liveness could not be verified"); }
|
|
const fresh = await providerIdentity(owned.identity.pid);
|
|
if (fresh.pid !== owned.identity.pid || fresh.uid !== owned.identity.uid || fresh.startedAt !== owned.identity.startedAt || fresh.executable !== owned.identity.executable || fresh.digest !== owned.identity.digest || fresh.proxyScript !== owned.identity.proxyScript || fresh.proxyDigest !== owned.identity.proxyDigest)
|
|
throw new Error("Provider cleanup refuses signalling a reused or changed PID");
|
|
process.kill(fresh.pid, "SIGCONT"); suspended.delete(ctx);
|
|
}
|
|
|
|
/** True only while this fixture still holds the exact owned provider stopped: same PID, start
|
|
* identity, executable digest, and a kernel state of stopped (T). The old provider's cleanup cannot
|
|
* finish while this holds, so anything observed under it happened during that cleanup. */
|
|
export async function ownedCleanupProviderIsHeld(ctx: ChatCatalogContext) {
|
|
const owned = suspended.get(ctx); if (!owned) return false;
|
|
try { process.kill(owned.identity.pid, 0); } catch { return false; }
|
|
const state = await promisify(execFile)("/bin/ps", ["-p", String(owned.identity.pid), "-o", "stat="], { timeout: 5_000, maxBuffer: 4_096 }).then(out => out.stdout.trim(), () => "");
|
|
const fresh = await providerIdentity(owned.identity.pid).catch(() => undefined);
|
|
return state.startsWith("T") && !!fresh && fresh.startedAt === owned.identity.startedAt && fresh.uid === owned.identity.uid && fresh.executable === owned.identity.executable && fresh.digest === owned.identity.digest;
|
|
}
|
|
export async function ownedCleanupProviderHasExited(ctx: ChatCatalogContext) {
|
|
const identity = retainedProvider.get(ctx); if (!identity) throw new Error("No retained exact provider identity for cleanup verification");
|
|
try { process.kill(identity.pid, 0); } catch (error) { if ((error as NodeJS.ErrnoException).code === "ESRCH") return true; return false; }
|
|
const startedAt = await readProcessStartedAt(identity.pid);
|
|
return !!startedAt && startedAt !== identity.startedAt;
|
|
}
|