mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-15 01:52:55 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat-feature evals measure whether agents complete real chat requests through the product > - The chat catalog eval harness and its repairs live on the `chat-eval/master-candidate` branch, not on master > - Today's work made 139 of 242 chat evals pass in verified single-case live runs. Each product fix landed on master as its own pull request > - The eval-side changes (setups, graders, fault seams, evidence capture) still exist only on the candidate > - This pull request lands the harness on master. Master stays the source of truth for product behavior > - The benefit is one tree: evals run against master commits, and product fixes reach the evals as soon as they merge ## Linked Issues or Issue Description Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs #20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409 **What existing behavior does this improve?** The chat catalog eval harness and the product it evaluates live on a long-running branch outside master. Eval fixes do not reach master, so master-based eval runs use old setups and graders. **Subsystem affected** `tests/runner-e2e` chat catalog harness, plus a small set of product changes that exist only on the candidate. **Proposed change** Land the harness on master. Every product fix from this eval work already merged to master as its own pull request. Conflicts keep master's product behavior. ## What Changed - Most of the diff (808 of 851 files) is under `tests/runner-e2e/`: chat-catalog domains, scenarios, descriptors, the TestOps manifest, private evidence tooling, and their tests. - Graders now check saved state, setup literals and rendered or clicked links. Many keyword and regex checks on free-form replies are removed. Each changed grader has a bumped `oracleVersion` and calibration tests (a valid reply, a reworded valid reply, and broken variants that must fail). - The shared link helper `hasLink` no longer treats a list link (for example `/COM/issues?view=mine`) as a link to every task without a URL key. - Setup fixes build real state through supported product routes: routine history, agent grants through the board grant route, budget holds with a real cap, and a storage-layer archive fault for `inbox-search.archive-partial-failure`. - Browser boundary journeys (sign-in, chat-disabled, onboarding) now write the standard pass evidence files. - Retargeted cases, approved by the maintainer: `agent-chat.chat-disabled` (Conference Room gate), `agents.restore-instructions` (no saved instruction history), `access.manage-membership` (member permission edits stay board-only). - The private evidence trusted files (`manual-private-handoff.ts`, `manual-private-retention.ts`, `manual-private-report-assets.ts`) are byte-identical to master. - Product files outside `tests/` (43 files, +1,328 −49) carry candidate-only changes that master does not yet have. Most are tests. The main code changes are in `request-human-input.ts`, `heartbeat-run-summary.ts`, `native-completion-feedback.ts`, `native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and `IssueChatLongThreadPerf.tsx`. ## Verification - `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts chat-catalog-consistency.test.ts unit-launcher.test.mjs --testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests pass. - `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: no change. - `tsc` for `server` and `tests/runner-e2e`: clean. - Live single-case verification: 139 of 242 cases passed on paid runs, and a person read the saved evidence for each. These are single-case runs, not one full-cohort run. ## Risks - Large diff. Most of it is test-only, but the 43 non-test files change product code paths. Review those files first. - Some graders are now less strict on reply wording. A person must read replies for meaning on those cases. - 42 cases need disposable outside accounts (Notion, Slack, GitHub, Daytona and others) and stay blocked. `access.invite-teammate` stays failing until local agent isolation lands, because the runner's durable state keeps a live invite token. - The paid workflow compares the trusted private-evidence files with master. They are unchanged, so this risk is low. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution, in Claude Code. Subagents used the same model. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing>
90 lines
7.9 KiB
TypeScript
90 lines
7.9 KiB
TypeScript
import path from "node:path";
|
|
import { createHash } from "node:crypto";
|
|
import { mkdir, readFile, realpath, writeFile } from "node:fs/promises";
|
|
import { resolveDefaultAgentWorkspaceDir } from "../../../server/src/home-paths.js";
|
|
import { prepareChatBrief } from "../chat-stories.js";
|
|
import { pollUntil } from "../api.js";
|
|
import { runtimeWorker, type RuntimeAgent } from "./runtime-support.js";
|
|
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
|
|
import type { Entity } from "./internal-support.js";
|
|
export interface RuntimeBriefGate { gate: string; ready: string; scriptPath: string; command: string; taskBound: boolean;
|
|
gateName?: string; script?: string; boundTaskId?: string; boundWorkspaceId?: string }
|
|
export function taskBoundRuntimeGate(nonce: string, timeoutMs = 300_000): RuntimeBriefGate {
|
|
if (!Number.isFinite(timeoutMs) || timeoutMs <= 0 || timeoutMs > 300_000) throw new Error("Task brief wait must remain bounded");
|
|
const gateName = `runtime-brief-${createHash("sha256").update(nonce).digest("hex").slice(0, 24)}.txt`;
|
|
const script = `const fs=require("node:fs");const gate=${JSON.stringify(gateName)};fs.writeFileSync(gate+".waiting","waiting",{flag:"wx"});const end=Date.now()+${timeoutMs};const timer=setInterval(()=>{if(fs.existsSync(gate)){console.log(fs.readFileSync(gate,"utf8"));clearInterval(timer)}else if(Date.now()>end){clearInterval(timer);process.exitCode=1}},100);`;
|
|
const quote = (value: string) => "'" + value.replace(/'/g, "'\\''") + "'";
|
|
return { gate: "", ready: "", scriptPath: "", command: `node -e ${quote(script)}`, taskBound: true, gateName, script };
|
|
}
|
|
/** Bind only the actual assigned task's persisted workspace, never AGENT_HOME. */
|
|
export async function readRuntimeTaskGate(ctx: ChatCatalogContext, gate: RuntimeBriefGate, taskId: string, workerId: string) {
|
|
if (!gate.taskBound || !gate.gateName) throw new Error("Expected task-bound brief listener");
|
|
const root = process.env.PAPERCLIP_RUNNER_E2E_TEMP_ROOT;
|
|
if (!root) throw new Error("Task gate requires the owned attempt root");
|
|
const [task, view] = await Promise.all([
|
|
ctx.input.api.get<{ id: string; companyId: string; assigneeAgentId: string | null; executionWorkspaceId: string | null }>(`/api/issues/${taskId}`),
|
|
ctx.input.api.get<{ workspace: { id: string; companyId: string; cwd: string } | null }>(`/api/issues/${taskId}/workspace`),
|
|
]);
|
|
const mismatched = [task.id !== taskId && "task", task.companyId !== ctx.input.fixtures.company.id && "company",
|
|
task.assigneeAgentId !== workerId && (task.assigneeAgentId ? "assignee" : "assignee-cleared"),
|
|
!view.workspace ? "workspace-unbound" : view.workspace.companyId !== task.companyId && "workspace-company",
|
|
!!view.workspace && view.workspace.id !== task.executionWorkspaceId && (task.executionWorkspaceId ? "workspace-id" : "task-workspace-field-missing")].filter(Boolean);
|
|
if (mismatched.length) throw new Error(`Task gate does not match the actual company, assignee and workspace binding (${mismatched.join(", ")})`);
|
|
const workspace = view.workspace!;
|
|
const cwd = await realpath(workspace.cwd), ownedRoot = await realpath(root);
|
|
if (!cwd.startsWith(`${ownedRoot}${path.sep}`)) throw new Error("Task gate workspace escapes the owned attempt root");
|
|
const filename = path.join(cwd, gate.gateName);
|
|
if (gate.boundTaskId && (gate.boundTaskId !== taskId || gate.boundWorkspaceId !== workspace.id || gate.gate !== filename))
|
|
throw new Error("Task gate binding changed after admission");
|
|
gate.boundTaskId = taskId; gate.boundWorkspaceId = workspace.id; gate.gate = filename; gate.ready = `${filename}.waiting`;
|
|
try {
|
|
const ready = await realpath(gate.ready);
|
|
if (path.dirname(ready) !== cwd) throw new Error("Task gate readiness path escapes its binding");
|
|
return await readFile(ready, "utf8");
|
|
} catch (error) {
|
|
if ((error as NodeJS.ErrnoException).code === "ENOENT") return "";
|
|
throw error;
|
|
}
|
|
}
|
|
export async function prepareRuntimeWorkerGate(ctx: ChatCatalogContext, options: { worker?: RuntimeAgent; taskBound?: boolean } = {}) {
|
|
if (ctx.input.execution.environment.id !== "local") throw new Error("Owned file-gate fixture requires a real local environment");
|
|
const root = process.env.PAPERCLIP_RUNNER_E2E_TEMP_ROOT; if (!root) throw new Error("Busy fixture requires its isolated attempt root");
|
|
const worker = options.worker ?? await runtimeWorker(ctx);
|
|
const isolated = await realpath(root);
|
|
const previousRuns = await ctx.allRuns();
|
|
let gate: RuntimeBriefGate;
|
|
if (options.taskBound) gate = taskBoundRuntimeGate(`${ctx.input.nonce}-${previousRuns.length}`);
|
|
else {
|
|
const candidate = path.resolve(resolveDefaultAgentWorkspaceDir(worker.id));
|
|
if (!candidate.startsWith(`${isolated}${path.sep}`)) throw new Error("Busy fixture escaped its isolated company home");
|
|
await mkdir(candidate, { recursive: true });
|
|
const home = await realpath(candidate);
|
|
if (!home.startsWith(`${isolated}${path.sep}`)) throw new Error("Busy fixture resolved outside its isolated company home");
|
|
const original = await prepareChatBrief(home, `runtime-${ctx.input.nonce}-${previousRuns.length}`, 300_000);
|
|
gate = { ...original, command: `node ${JSON.stringify(original.scriptPath)}`, taskBound: false };
|
|
}
|
|
const company = `/api/companies/${ctx.input.fixtures.company.id}`;
|
|
const fundedAgents = [...new Set(options.taskBound ? [worker.id, ctx.input.fixtures.agent.id] : [worker.id])];
|
|
for (const scope of [{ scopeType: "company", scopeId: ctx.input.fixtures.company.id }, ...fundedAgents.map(scopeId => ({ scopeType: "agent", scopeId }))])
|
|
await ctx.input.api.post(`${company}/budgets/policies`, { ...scope, amount: 500, hardStopEnabled: true, isActive: true, unpricedUsagePolicy: "allow" });
|
|
return { worker, gate, previousRuns, company };
|
|
}
|
|
export async function busyRuntimeWorker(ctx: ChatCatalogContext, options: { worker?: RuntimeAgent; observationDeadlineAt?: number } = {}) {
|
|
const { worker, gate, previousRuns, company } = await prepareRuntimeWorkerGate(ctx, { ...options, taskBound: true });
|
|
// Each busy worker owns its own task. Without allowDuplicate the product's recent-open-title
|
|
// deduplication returns the first worker's identically titled "Website task" (HTTP 200).
|
|
const task = await ctx.input.api.post<Entity>(`${company}/issues`, { title: "Website task", status: "todo", assigneeAgentId: worker.id, allowDuplicate: true,
|
|
description: `The website brief is being supplied by the human. Run ${gate.command} once and wait for its real output. Do not invent a brief, replace the script, or mark work complete before it returns. Once the brief arrives, save it as a task document and finish normally.` });
|
|
if (task.assigneeAgentId !== worker.id || previousRuns.some(run => run.contextSnapshot?.issueId === task.id))
|
|
throw new Error("Busy worker fixture did not create its own task for this worker");
|
|
const running = await pollUntil({ label: "Real Morgan process waiting for its human brief", deadlineAt: options.observationDeadlineAt ?? Date.now() + 150_000,
|
|
load: async () => ({ runs: (await ctx.allRuns()).filter(run => !previousRuns.some(prior => prior.id === run.id)), waiting: await readRuntimeTaskGate(ctx, gate, task.id, worker.id) }),
|
|
accept: value => (options.observationDeadlineAt === undefined || Date.now() < options.observationDeadlineAt) && value.waiting === "waiting" && value.runs.length === 1 && value.runs[0]?.agentId === worker.id && value.runs[0].status === "running",
|
|
reject: value => value.runs.some(run => ["failed", "cancelled", "timed_out", "succeeded"].includes(run.status)) ? "Busy fixture ended before its required active boundary" : undefined });
|
|
return { worker, task, gate, workerRunId: running.runs[0]!.id, workerWasReallyWaiting: true };
|
|
}
|
|
export async function releaseRuntimeWorker(state: Record<string, unknown>) {
|
|
const gate = state.gate as { gate?: string } | undefined;
|
|
if (gate?.gate) await writeFile(gate.gate, "The human ended this disposable fixture. Preserve any saved work and finish normally.", "utf8");
|
|
}
|