Files
PaperClipAI/tests/runner-e2e/chat-catalog-domains/runtime-busy-worker.ts
T
f4337bb5fb test(evals): land the chat catalog eval harness on master (#19614)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature evals measure whether agents complete real chat
requests through the product
> - The chat catalog eval harness and its repairs live on the
`chat-eval/master-candidate` branch, not on master
> - Today's work made 139 of 242 chat evals pass in verified single-case
live runs. Each product fix landed on master as its own pull request
> - The eval-side changes (setups, graders, fault seams, evidence
capture) still exist only on the candidate
> - This pull request lands the harness on master. Master stays the
source of truth for product behavior
> - The benefit is one tree: evals run against master commits, and
product fixes reach the evals as soon as they merge

## Linked Issues or Issue Description

Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs
#20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409

**What existing behavior does this improve?**
The chat catalog eval harness and the product it evaluates live on a
long-running branch outside master. Eval fixes do not reach master, so
master-based eval runs use old setups and graders.

**Subsystem affected**
`tests/runner-e2e` chat catalog harness, plus a small set of product
changes that exist only on the candidate.

**Proposed change**
Land the harness on master. Every product fix from this eval work
already merged to master as its own pull request. Conflicts keep
master's product behavior.

## What Changed

- Most of the diff (808 of 851 files) is under `tests/runner-e2e/`:
chat-catalog domains, scenarios, descriptors, the TestOps manifest,
private evidence tooling, and their tests.
- Graders now check saved state, setup literals and rendered or clicked
links. Many keyword and regex checks on free-form replies are removed.
Each changed grader has a bumped `oracleVersion` and calibration tests
(a valid reply, a reworded valid reply, and broken variants that must
fail).
- The shared link helper `hasLink` no longer treats a list link (for
example `/COM/issues?view=mine`) as a link to every task without a URL
key.
- Setup fixes build real state through supported product routes: routine
history, agent grants through the board grant route, budget holds with a
real cap, and a storage-layer archive fault for
`inbox-search.archive-partial-failure`.
- Browser boundary journeys (sign-in, chat-disabled, onboarding) now
write the standard pass evidence files.
- Retargeted cases, approved by the maintainer:
`agent-chat.chat-disabled` (Conference Room gate),
`agents.restore-instructions` (no saved instruction history),
`access.manage-membership` (member permission edits stay board-only).
- The private evidence trusted files (`manual-private-handoff.ts`,
`manual-private-retention.ts`, `manual-private-report-assets.ts`) are
byte-identical to master.
- Product files outside `tests/` (43 files, +1,328 −49) carry
candidate-only changes that master does not yet have. Most are tests.
The main code changes are in `request-human-input.ts`,
`heartbeat-run-summary.ts`, `native-completion-feedback.ts`,
`native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and
`IssueChatLongThreadPerf.tsx`.

## Verification

- `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts
chat-catalog-consistency.test.ts unit-launcher.test.mjs
--testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests
pass.
- `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no change.
- `tsc` for `server` and `tests/runner-e2e`: clean.
- Live single-case verification: 139 of 242 cases passed on paid runs,
and a person read the saved evidence for each. These are single-case
runs, not one full-cohort run.

## Risks

- Large diff. Most of it is test-only, but the 43 non-test files change
product code paths. Review those files first.
- Some graders are now less strict on reply wording. A person must read
replies for meaning on those cases.
- 42 cases need disposable outside accounts (Notion, Slack, GitHub,
Daytona and others) and stay blocked. `access.invite-teammate` stays
failing until local agent isolation lands, because the runner's durable
state keeps a live invite token.
- The paid workflow compares the trusted private-evidence files with
master. They are unchanged, so this risk is low.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution, in Claude Code. Subagents used the same model.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 08:02:11 -06:00

90 lines
7.9 KiB
TypeScript

import path from "node:path";
import { createHash } from "node:crypto";
import { mkdir, readFile, realpath, writeFile } from "node:fs/promises";
import { resolveDefaultAgentWorkspaceDir } from "../../../server/src/home-paths.js";
import { prepareChatBrief } from "../chat-stories.js";
import { pollUntil } from "../api.js";
import { runtimeWorker, type RuntimeAgent } from "./runtime-support.js";
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
import type { Entity } from "./internal-support.js";
export interface RuntimeBriefGate { gate: string; ready: string; scriptPath: string; command: string; taskBound: boolean;
gateName?: string; script?: string; boundTaskId?: string; boundWorkspaceId?: string }
export function taskBoundRuntimeGate(nonce: string, timeoutMs = 300_000): RuntimeBriefGate {
if (!Number.isFinite(timeoutMs) || timeoutMs <= 0 || timeoutMs > 300_000) throw new Error("Task brief wait must remain bounded");
const gateName = `runtime-brief-${createHash("sha256").update(nonce).digest("hex").slice(0, 24)}.txt`;
const script = `const fs=require("node:fs");const gate=${JSON.stringify(gateName)};fs.writeFileSync(gate+".waiting","waiting",{flag:"wx"});const end=Date.now()+${timeoutMs};const timer=setInterval(()=>{if(fs.existsSync(gate)){console.log(fs.readFileSync(gate,"utf8"));clearInterval(timer)}else if(Date.now()>end){clearInterval(timer);process.exitCode=1}},100);`;
const quote = (value: string) => "'" + value.replace(/'/g, "'\\''") + "'";
return { gate: "", ready: "", scriptPath: "", command: `node -e ${quote(script)}`, taskBound: true, gateName, script };
}
/** Bind only the actual assigned task's persisted workspace, never AGENT_HOME. */
export async function readRuntimeTaskGate(ctx: ChatCatalogContext, gate: RuntimeBriefGate, taskId: string, workerId: string) {
if (!gate.taskBound || !gate.gateName) throw new Error("Expected task-bound brief listener");
const root = process.env.PAPERCLIP_RUNNER_E2E_TEMP_ROOT;
if (!root) throw new Error("Task gate requires the owned attempt root");
const [task, view] = await Promise.all([
ctx.input.api.get<{ id: string; companyId: string; assigneeAgentId: string | null; executionWorkspaceId: string | null }>(`/api/issues/${taskId}`),
ctx.input.api.get<{ workspace: { id: string; companyId: string; cwd: string } | null }>(`/api/issues/${taskId}/workspace`),
]);
const mismatched = [task.id !== taskId && "task", task.companyId !== ctx.input.fixtures.company.id && "company",
task.assigneeAgentId !== workerId && (task.assigneeAgentId ? "assignee" : "assignee-cleared"),
!view.workspace ? "workspace-unbound" : view.workspace.companyId !== task.companyId && "workspace-company",
!!view.workspace && view.workspace.id !== task.executionWorkspaceId && (task.executionWorkspaceId ? "workspace-id" : "task-workspace-field-missing")].filter(Boolean);
if (mismatched.length) throw new Error(`Task gate does not match the actual company, assignee and workspace binding (${mismatched.join(", ")})`);
const workspace = view.workspace!;
const cwd = await realpath(workspace.cwd), ownedRoot = await realpath(root);
if (!cwd.startsWith(`${ownedRoot}${path.sep}`)) throw new Error("Task gate workspace escapes the owned attempt root");
const filename = path.join(cwd, gate.gateName);
if (gate.boundTaskId && (gate.boundTaskId !== taskId || gate.boundWorkspaceId !== workspace.id || gate.gate !== filename))
throw new Error("Task gate binding changed after admission");
gate.boundTaskId = taskId; gate.boundWorkspaceId = workspace.id; gate.gate = filename; gate.ready = `${filename}.waiting`;
try {
const ready = await realpath(gate.ready);
if (path.dirname(ready) !== cwd) throw new Error("Task gate readiness path escapes its binding");
return await readFile(ready, "utf8");
} catch (error) {
if ((error as NodeJS.ErrnoException).code === "ENOENT") return "";
throw error;
}
}
export async function prepareRuntimeWorkerGate(ctx: ChatCatalogContext, options: { worker?: RuntimeAgent; taskBound?: boolean } = {}) {
if (ctx.input.execution.environment.id !== "local") throw new Error("Owned file-gate fixture requires a real local environment");
const root = process.env.PAPERCLIP_RUNNER_E2E_TEMP_ROOT; if (!root) throw new Error("Busy fixture requires its isolated attempt root");
const worker = options.worker ?? await runtimeWorker(ctx);
const isolated = await realpath(root);
const previousRuns = await ctx.allRuns();
let gate: RuntimeBriefGate;
if (options.taskBound) gate = taskBoundRuntimeGate(`${ctx.input.nonce}-${previousRuns.length}`);
else {
const candidate = path.resolve(resolveDefaultAgentWorkspaceDir(worker.id));
if (!candidate.startsWith(`${isolated}${path.sep}`)) throw new Error("Busy fixture escaped its isolated company home");
await mkdir(candidate, { recursive: true });
const home = await realpath(candidate);
if (!home.startsWith(`${isolated}${path.sep}`)) throw new Error("Busy fixture resolved outside its isolated company home");
const original = await prepareChatBrief(home, `runtime-${ctx.input.nonce}-${previousRuns.length}`, 300_000);
gate = { ...original, command: `node ${JSON.stringify(original.scriptPath)}`, taskBound: false };
}
const company = `/api/companies/${ctx.input.fixtures.company.id}`;
const fundedAgents = [...new Set(options.taskBound ? [worker.id, ctx.input.fixtures.agent.id] : [worker.id])];
for (const scope of [{ scopeType: "company", scopeId: ctx.input.fixtures.company.id }, ...fundedAgents.map(scopeId => ({ scopeType: "agent", scopeId }))])
await ctx.input.api.post(`${company}/budgets/policies`, { ...scope, amount: 500, hardStopEnabled: true, isActive: true, unpricedUsagePolicy: "allow" });
return { worker, gate, previousRuns, company };
}
export async function busyRuntimeWorker(ctx: ChatCatalogContext, options: { worker?: RuntimeAgent; observationDeadlineAt?: number } = {}) {
const { worker, gate, previousRuns, company } = await prepareRuntimeWorkerGate(ctx, { ...options, taskBound: true });
// Each busy worker owns its own task. Without allowDuplicate the product's recent-open-title
// deduplication returns the first worker's identically titled "Website task" (HTTP 200).
const task = await ctx.input.api.post<Entity>(`${company}/issues`, { title: "Website task", status: "todo", assigneeAgentId: worker.id, allowDuplicate: true,
description: `The website brief is being supplied by the human. Run ${gate.command} once and wait for its real output. Do not invent a brief, replace the script, or mark work complete before it returns. Once the brief arrives, save it as a task document and finish normally.` });
if (task.assigneeAgentId !== worker.id || previousRuns.some(run => run.contextSnapshot?.issueId === task.id))
throw new Error("Busy worker fixture did not create its own task for this worker");
const running = await pollUntil({ label: "Real Morgan process waiting for its human brief", deadlineAt: options.observationDeadlineAt ?? Date.now() + 150_000,
load: async () => ({ runs: (await ctx.allRuns()).filter(run => !previousRuns.some(prior => prior.id === run.id)), waiting: await readRuntimeTaskGate(ctx, gate, task.id, worker.id) }),
accept: value => (options.observationDeadlineAt === undefined || Date.now() < options.observationDeadlineAt) && value.waiting === "waiting" && value.runs.length === 1 && value.runs[0]?.agentId === worker.id && value.runs[0].status === "running",
reject: value => value.runs.some(run => ["failed", "cancelled", "timed_out", "succeeded"].includes(run.status)) ? "Busy fixture ended before its required active boundary" : undefined });
return { worker, task, gate, workerRunId: running.runs[0]!.id, workerWasReallyWaiting: true };
}
export async function releaseRuntimeWorker(state: Record<string, unknown>) {
const gate = state.gate as { gate?: string } | undefined;
if (gate?.gate) await writeFile(gate.gate, "The human ended this disposable fixture. Preserve any saved work and finish normally.", "utf8");
}