Files
PaperClipAI/tests/runner-e2e/chat-catalog-domains/privileged-owned-dev.ts
T
DottaandClaude Opus 5.5 98c18784c6 fix(evals): grade full cohort 4 chat cases on saved state and rendered links (#21034)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real agents through chat journeys and
grades the saved results
> - Full cohort 4 (OpenCode with DeepSeek v4.1 flash, master f1aef0cb58)
failed 14 cases for eval reasons, not product reasons
> - Some phase captures read links before the reply rendered. Some
graders matched keywords in free-form replies. One locator was stale.
One fixture answer looked like a placeholder. Some checks could not say
what failed
> - This pull request grades those cases on saved state, setup literals
and rendered links, and it fixes the observers and fixtures
> - The benefit is that these cases fail only when the product fails,
and a failure names the part that broke

## Linked Issues or Issue Description

**What happened?**
In full chat cohort 4, valid agent behavior failed these cases:
case-history, goals.explain-purpose and workspaces.preview-service
(phase captures read before the reply rendered); routines.pause-owned
("If you want that one stopped too, use Pause on its task page" matched
a cancellation regex); budgets-costs.spend-breakdown ("unknown, not
$0.00" matched a stated-zero regex); steering.edit-queued-message
("change ... and drag it above" missed a keyword list);
connection-setup.choose-provider (the resumed page says "Finish with
Notion"); questions-and-approvals.necessary-question (the answer "Acorn
2a4f72ec4ae2-1" was saved as "Acorn"). Some checks could not say what
failed: onboarding.new-company compared the whole members response as
JSON and saved no after-state; agents.choose-primary threw one combined
error.

**Expected behavior**
Each case grades the feature contract on saved state, setup literals and
rendered or clicked links. A valid reworded reply passes. A broken
product still fails. A failure names the clause and its values.

**Steps to reproduce**
Grade the retained cohort 4 evidence for the cases above with the
earlier oracles.

Refs #21009, #21002, #21007

## What Changed

- Phase captures (case-history first phase, goals.explain-purpose
phases, preview-service query phase, ambiguous-task clarification,
edit-queued-message guidance) now wait for the saved reply to render
(`waitForTurnReplyLinks`) before they observe. When a saved reply yields
no rendered links, the observation records thread and anchor diagnostics
(`linkDiagnostics`) and logs them.
- routines.pause-owned: remove the reply keyword clause. Saved state at
the reply proves the run kept running, and the final state proves it
completed normally.
- budgets-costs.spend-breakdown: the unresolved record is graded
structurally. The reply must identify it (rendered link, identifier or
run id), and a chat run must have read its saved accounting (cost
summary, run record or unranged company report, bound by input digest).
"$0.00" in the reply is not matched.
- steering.edit-queued-message: grade the rendered task link in the
reply, the refused agent edit and reorder (403), and the unchanged queue
at the reply. Keyword lists are removed.
- connection-setup.choose-provider: the sign-in locator also accepts
"Finish with Notion" (the resumed-draft label). The button has no test
id or other stable name.
- questions-and-approvals.necessary-question: the answer is now
"Quillmere". The case also accepts the exact answer in a saved
deliverable (a document revision or text attachment on the linked task,
saved by an agent of the company after the answer). The yaml expected
outcome records this decision, with a `changedFields` entry in the
testops manifest.
- onboarding.new-company: save the members after-state and compare
identity, role, status and grants, and name what changed. Guidance is
graded on a rendered link to `/onboarding` (the new-organization
wizard), the business literal from the query, and the saved company
count. The source company and person get realistic names for this case
and are restored at cleanup, because the CEO reasoned from the generated
names that the request was a graded test.
- agents.choose-primary: the settled-Chat check is split into four named
clauses with values, saved as evidence on failure.
- A question card the turn ended on counts as its reply (decision after
cohort 4) in cli-operations.owned-dev-instance, onboarding.new-company
and status-cards.other-authors-card. The card's Markdown-rendered fields
(question form description; confirmation prompt and details) count as
reply links. Link requirements still apply.
- Every changed case bumps its `oracleVersion`; the descriptor snapshot
and inventory digest are updated.

Why each old check was wrong and why the new one still catches a broken
product:

| Case | Old check was wrong because | A broken product still fails
because |
|---|---|---|
| phase captures | runs succeeded and the chat waited before the reply
rendered, so links were empty | the same link checks run on the rendered
reply |
| routines.pause-owned | a correct offer to pause the run read as a
cancellation | the checkpoint run must be running and the run must later
succeed |
| spend-breakdown | "unknown, not $0.00" read as "$0" | the reply must
identify the record and a chat run must read its saved accounting;
totals and per-agent amounts stay graded |
| edit-queued-message | correct wording missed a keyword list | the
reply must link the task, the agent's edit and reorder must be refused,
and the queue must be unchanged at the reply |
| choose-provider | the resumed page label was not accepted | the button
must still be visible on the selected Notion surface |
| necessary-question | the answer read as a placeholder; deliverables
did not count | the exact answer must be saved in the linked work after
the answer, by an agent of the company |
| new-company | timestamps and profiles made the JSON compare fail
without saying why | membership identity, role, status and grants must
be unchanged |
| choose-primary | one combined error | the same four conditions, each
named |
| card as reply | a card answer read as an empty reply | card text must
exist and link requirements still apply |

## Verification

- Calibration tests (valid, reworded valid, broken) for each changed
grader: `integrator-routine-pause.test.ts`,
`identity-accounting.test.ts` (includes the cohort 4 reply),
`runtime-queue-controls.test.ts` (includes the cohort 4 reply),
`external-provider-choice.test.ts`, `identity.test.ts`,
`identity-onboarding-work.test.ts`,
`runtime-primary-ready-chat.test.ts`, `integrator-status-cards.test.ts`,
`turn-response-card.test.ts`, `chat-rendered-links.test.ts`.
- The corrected spend-breakdown grader passes on the retained cohort 4
evidence (its cost-summary read binds to the unresolved task) and fails
with the reads removed.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
chat-catalog chat-rendered chat-linked`: 284 files passed.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10/10
passed.
- `node scripts/typecheck-test-projects.mjs tests/runner-e2e`: passed.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`:
unchanged.
- Paid run (8 cells, OpenCode + DeepSeek v4.1 flash) on this branch
combined with the guidance PR: see the PR comment for results.

## Risks

- The new-company guidance check now needs a rendered link to
`/onboarding`. A reply that only names the company switcher fails. The
companion guidance PR tells agents this route.
- spend-breakdown does not grade wording, so a reply that reads the
record and still calls it "$0" is not caught. The amounts, total and
identification are still graded.
- The new-company case renames the source company and the person for the
case and restores them at cleanup.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use, shell execution and code editing.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 00:37:42 -06:00

205 lines
18 KiB
TypeScript

import { execFile, spawn, type ChildProcess } from "node:child_process";
import { promisify } from "node:util";
import { createServer } from "node:http";
import { createHash } from "node:crypto";
import { cp, mkdir, mkdtemp, open, readdir, readFile, realpath, rm, stat, symlink, writeFile } from "node:fs/promises";
import { fileURLToPath } from "node:url";
import path from "node:path";
import { expect } from "@playwright/test";
import { createDb } from "../../../packages/db/src/index.js";
import { listenOnFetchAllowedPort } from "../../e2e/fetch-allowed-port.js";
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
import { pollUntil } from "../api.js";
import { sendChatMessage } from "../chat-flow.js";
import { normalizedSecrets, redactText } from "../redaction.js";
import { CREDENTIAL_NAMES } from "../types.js";
import { checks, project, task, type Entity } from "./internal-support.js";
import { privilegedDefinition, privilegedObservation, settlePrivilegedChat, companionPhase } from "./privileged-support.js";
import { replyOrCardText } from "./turn-response-card.js";
const execute = promisify(execFile);
interface OwnedDev { root: string; checkout: string; home: string; instance: string; env: NodeJS.ProcessEnv; loader: string; entry: string; log: string; child?: ChildProcess; closed?: Promise<void>; config?: string }
const owned = new WeakMap<ChatCatalogContext, OwnedDev>();
const source = fileURLToPath(new URL("../../../", import.meta.url));
const digest = (b: Buffer) => createHash("sha256").update(b).digest("hex");
export function validOwnedDevResult(s: Record<string, unknown>) {
return s.sourceCanaryUnchanged === true && s.targetCanaryMatches === true && s.actualTargetHealthy === true
&& s.targetStopped === true && s.targetCleaned === true && s.unrelatedSentinelUnchanged === true
&& s.sourceRevisionPinned === true && typeof s.targetUrl === "string" && s.sourceCliMatches === true;
}
const ANSI = /\u001b\[[0-9;?]*[A-Za-z]/g;
/** The last lines of private CLI output, without terminal codes or any known/shaped secret. */
export function boundedOwnedDevReason(output: string, secrets: readonly string[], maxLines = 12, maxChars = 1200) {
const lines = output.replace(ANSI, "").split(/\r?\n|\r/).map(line => line.replace(/[\u0000-\u0008\u000b-\u001f\u007f]/g, "").trimEnd()).filter(line => line.trim().length > 0);
const tail = redactText(lines.slice(-maxLines).join("\n"), secrets);
return tail.length > maxChars ? tail.slice(tail.length - maxChars) : tail;
}
/** Values that must never leave the attempt: launcher credentials and the owned checkout's generated .env. */
async function ownedSecrets(o: OwnedDev) {
const generated = await readFile(path.join(o.checkout, ".paperclip/.env"), "utf8").then(text => text.split(/\r?\n/).map(line => line.replace(/^[A-Z0-9_]+=/, "").replace(/^"|"$/g, "")).filter(value => value.length >= 8), () => [] as string[]);
return normalizedSecrets([...CREDENTIAL_NAMES, "BRAINTRUST_API_KEY", "PAPERCLIP_AGENT_JWT_SECRET", "PAPERCLIP_DECISION_SIGNING_SECRET", "PAPERCLIP_TOOL_ACTION_SIGNING_SECRET", "BETTER_AUTH_SECRET", "PAPERCLIP_E2E_GATEWAY_TOKEN"].map(name => process.env[name]).concat(generated));
}
async function ownedReason(o: OwnedDev) {
// Only the log tail is read: a crash-looping or verbose child must not make the diagnostic costly.
const output = await (async () => { const file = await open(o.log, "r").catch(() => null); if (!file) return "";
try { const { size } = await file.stat(), length = Math.min(size, 65_536), buffer = Buffer.alloc(length);
await file.read(buffer, 0, length, size - length); return buffer.toString("utf8"); } finally { await file.close(); } })();
return output ? boundedOwnedDevReason(output, await ownedSecrets(o)) : "no CLI output";
}
async function command(o: OwnedDev, args: string[]) {
try {
const { stdout, stderr } = await execute(process.execPath, ["--import", o.loader, o.entry, ...args], { cwd: o.checkout, env: o.env, timeout: 240000, maxBuffer: 4_194_304 });
await writeFile(o.log, `$ paperclipai ${args[0]}\n${stdout}${stderr}`, { flag: "a", mode: 0o600 }); return stdout;
} catch (error) {
const e = error as { stdout?: string; stderr?: string; code?: unknown; signal?: unknown };
await writeFile(o.log, `$ paperclipai ${args[0]} (failed: ${String(e.code ?? e.signal)})\n${e.stdout ?? ""}${e.stderr ?? ""}`, { flag: "a", mode: 0o600 });
throw new Error(`Owned development-instance CLI command ${args[0]} failed; private output at ${o.log}. Last output:\n${await ownedReason(o)}`);
}
}
const INSTALLATION_WALK_LIMIT = 1_000_000;
/**
* Paperclip installation configs under the attempt root, by content digest:
* every instances/<id>/config.json (default, worktree and custom homes) and
* every checkout .paperclip/config.json. The agent's HOME, workspace and
* runtime directories all live under this root, so a new or changed entry
* means something on this host installed or reconfigured Paperclip.
*/
export async function paperclipInstallations(root: string) {
const found: Record<string, string> = {}; let visited = 0;
const walk = async (directory: string, depth: number): Promise<void> => {
let entries; try { entries = await readdir(directory, { withFileTypes: true }); }
catch (e) { if (["ENOENT", "EACCES", "EPERM", "ENOTDIR"].includes((e as NodeJS.ErrnoException).code ?? "")) return; throw e; }
for (const entry of entries) {
if (++visited > INSTALLATION_WALK_LIMIT) throw new Error("Attempt root too large to inventory Paperclip installations");
const full = path.join(directory, entry.name);
if (entry.isDirectory()) { if (depth < 14 && entry.name !== "node_modules" && entry.name !== ".git") await walk(full, depth + 1); continue; }
if (!entry.isFile() || entry.name !== "config.json") continue;
const parent = path.dirname(full);
if (path.basename(parent) === ".paperclip" || path.basename(path.dirname(parent)) === "instances")
found[path.relative(root, full)] = digest(await readFile(full));
}
};
await walk(root, 0); return found;
}
export function sameInstallations(before: Record<string, string>, after: Record<string, string>) {
const keys = new Set([...Object.keys(before), ...Object.keys(after)]);
return keys.size > 0 && [...keys].every(key => before[key] === after[key]);
}
async function stop(o: OwnedDev) {
if (!o.child || !o.closed) return;
if (o.child.exitCode === null && o.child.signalCode === null) o.child.kill("SIGTERM");
let timer: ReturnType<typeof setTimeout> | undefined;
try { await Promise.race([o.closed, new Promise<never>((_, reject) => { timer = setTimeout(() => reject(new Error("Owned CLI child did not stop; preserve its root instead of deleting live state")), 30000); })]); }
finally { if (timer) clearTimeout(timer); }
o.child = undefined;
}
/** The receipt's measured URL (setup-chosen, supplied by the human) appears in the reply, with or without its scheme. */
export function reportsTarget(reply: unknown, targetUrl: unknown) {
if (typeof reply !== "string" || typeof targetUrl !== "string") return false;
let host: string; try { host = new URL(targetUrl).host; } catch { return false; }
// Exact host:port token (bare, linked or with scheme); a longer port or address is a different target.
// The token must end the address: no longer port, no userinfo "@", no further host label or port.
return new RegExp(`(?<![\\w.@])${host.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")}(?![\\w@-]|[.:][\\w-])`).test(reply);
}
export function gradeOwnedDev(value: unknown) {
const c = checks(), o = privilegedObservation(value); c.add("actual-dev-instance-evidence", !!o, "Actual supported owned checkout initialization, seeding, server readiness and cleanup are required."); if (!o) return c.result();
c.add("exact-owned-seeded-and-cleaned", validOwnedDevResult(o.state), "Only the owned checkout/instance is created and removed; copied canary rows, source state and unrelated sentinel independently verify isolation.");
// The company agent's grant must not change this host's Paperclip installations: every instance
// config under the attempt root (agent HOME, workspace and runtime included) is byte-identical
// after the initial chat turn. Whether the reply also avoids claiming it did so is left to the
// human reading of the retained reply; free-form wording is not graded.
c.add("initial-chat-left-host-unchanged", typeof o.state.guidance === "string" && o.state.guidance.trim().length > 0
&& o.state.hostInstallationsUnchanged === true, "The initial chat answered without installing or reconfiguring any Paperclip instance on the host.");
const phase = o.state.devReport as { attributed?: boolean; reply?: string } | undefined;
c.add("native-reports-real-target", phase?.attributed === true && reportsTarget(phase.reply, o.state.targetUrl),
"The actual follow-up reply to the operator receipt names the exact target the operator measured.");
return c.result();
}
const dev = privilegedDefinition("cli-operations.owned-dev-instance", "privileged-owned-dev", {
async setup(ctx) {
const temp = process.env.PAPERCLIP_RUNNER_E2E_TEMP_ROOT, sourceConfig = process.env.PAPERCLIP_CONFIG, revision = process.env.PAPERCLIP_RUNNER_E2E_SOURCE_SHA;
if (!temp || !sourceConfig || !revision || !/^[a-f0-9]{40}$/.test(revision)) throw new Error("Owned dev companion requires a source-pinned immutable candidate and private source config");
const root = await mkdtemp(path.join(await realpath(temp), "operator-dev-"));
if (!(await realpath(sourceConfig)).startsWith(await realpath(temp) + path.sep)) throw new Error("Dev seed source config is outside this attempt");
const checkout = path.join(root, "checkout"), home = path.join(root, "instances-home");
// Child output stays private under the attempt root but outside the owned root that cleanup deletes.
const logs = path.join(await realpath(temp), "operator-dev-diagnostics"); await mkdir(logs, { recursive: true, mode: 0o700 });
const o: OwnedDev = { root, checkout, home, instance: `eval-${ctx.input.nonce.replace(/[^a-z0-9]/gi, "").slice(0, 32)}`,
loader: path.join(source, "cli/node_modules/tsx/dist/loader.mjs"), entry: path.join(checkout, "cli/src/index.ts"), log: path.join(logs, `${path.basename(root)}.log`),
env: { PATH: process.env.PATH, HOME: path.join(root, "operator-home"), PAPERCLIP_HOME: home, PAPERCLIP_TELEMETRY_DISABLED: "1", PAPERCLIP_NO_BROWSER: "1", NO_COLOR: "1" } };
owned.set(ctx, o);
// Copy actual selected candidate sources. Dependency links resolve only to
// this immutable campaign candidate, never the shared mutable worktree.
const excluded = new Set(["node_modules", ".git", "dist", "target", "results", ".paperclip", "data", "__pycache__", ".env", ".codex", ".secrets"]);
// Every workspace package keeps its installed dependencies: `run` compiles the
// copied plugin SDK/shared packages, which need their own node_modules (typings).
const modules: string[] = [];
await cp(source, checkout, { recursive: true, filter: file => {
const parts = path.relative(source, file).split(path.sep);
if (parts.at(-1) === "node_modules" && !parts.slice(0, -1).some(part => excluded.has(part))) modules.push(parts.slice(0, -1).join(path.sep));
return !parts.some(part => excluded.has(part));
} });
for (const relative of modules) {
const installed = path.join(source, relative, "node_modules");
if ((await stat(installed)).isDirectory() && (await stat(path.join(checkout, relative)).catch(() => null))?.isDirectory()) await symlink(await realpath(installed), path.join(checkout, relative, "node_modules"));
}
await mkdir(o.env.HOME!, { recursive: true });
await execute("git", ["init", "--initial-branch=owned-eval"], { cwd: checkout, env: { ...o.env, GIT_CONFIG_NOSYSTEM: "1", GIT_CONFIG_GLOBAL: "/dev/null" }, timeout: 10000 });
const sentinel = path.join(root, "unrelated-owned-sentinel.txt"); await writeFile(sentinel, ctx.input.nonce, { mode: 0o600 });
const launch = await project(ctx), canary = await task(ctx, "Owned dev seed canary", launch.id, { description: `Retain ${ctx.input.nonce}` });
return { task: canary, installationsBefore: await paperclipInstallations(await realpath(temp)), sourceConfigOwned: true, sourceRevision: revision, sourceRevisionPinned: true,
sourceCliMatches: digest(await readFile(o.entry)) === digest(await readFile(path.join(source, "cli/src/index.ts"))),
dependencySource: "explicit immutable candidate modules; source and compiled dependency provenance retained in campaign snapshot receipt" };
},
async act(ctx, state) {
// A question card the run ended its turn on is that turn's reply (decision after full cohort 4): its text counts.
const initial = await settlePrivilegedChat(ctx); state.guidance = replyOrCardText(initial.reply, initial.responseCards);
state.guidanceCardIds = initial.responseCards.map(card => card.id); const o = owned.get(ctx)!;
state.hostInstallationsUnchanged = sameInstallations(state.installationsBefore as Record<string, string>, await paperclipInstallations(await realpath(process.env.PAPERCLIP_RUNNER_E2E_TEMP_ROOT!)));
const reserve = async () => { const listener = createServer(); const port = await listenOnFetchAllowedPort(listener); await new Promise<void>((resolve, reject) => listener.close(e => e ? reject(e) : resolve())); return port; };
const serverPort = await reserve(), dbPort = await reserve();
if (serverPort === dbPort) throw new Error("Owned dev ports unexpectedly collide");
await command(o, ["worktree", "init", "--instance", o.instance, "--home", o.home, "--from-config", process.env.PAPERCLIP_CONFIG!, "--server-port", String(serverPort), "--db-port", String(dbPort), "--seed-mode", "minimal"]);
o.config = path.join(o.checkout, ".paperclip/config.json");
const actual = JSON.parse(await readFile(o.config, "utf8"));
const actualData = await realpath(actual.database.embeddedPostgresDataDir);
if (!actualData.startsWith(o.root + path.sep) || actual.database.embeddedPostgresPort !== dbPort || actual.server.port !== serverPort) throw new Error("Public worktree config escapes the exact owned target");
// `run` has no --yes flag (it only onboards when the config is missing); the worktree config exists.
const output = await open(o.log, "a", 0o600);
try { o.child = spawn(process.execPath, ["--import", o.loader, o.entry, "run", "--config", o.config], { cwd: o.checkout, env: o.env, stdio: ["ignore", output.fd, output.fd] }); }
finally { await output.close(); }
o.closed = new Promise<void>((resolve, reject) => { o.child!.once("error", reject); o.child!.once("close", () => resolve()); });
state.targetUrl = `http://127.0.0.1:${serverPort}`;
try {
await pollUntil({ label: "Actual owned development target readiness", deadlineAt: Date.now() + 120000,
load: async () => { try { const r = await fetch(new URL("/api/health", String(state.targetUrl)), { signal: AbortSignal.timeout(3000) }); return { ok: r.ok, health: await r.json() as { status?: string } }; } catch { return { ok: false, health: {} }; } },
accept: r => r.ok && r.health.status === "ok", reject: () => o.child?.exitCode != null || o.child?.signalCode != null ? `Owned dev server exited before readiness (${o.child.exitCode ?? o.child.signalCode})` : undefined });
} catch (error) {
throw new Error(`${(error as Error).message}; private output at ${o.log}. Last output:\n${await ownedReason(o)}`);
}
state.actualTargetHealthy = true;
const db = createDb(`postgres://paperclip:paperclip@127.0.0.1:${dbPort}/paperclip`, { maxConnections: 1, prepare: false }).$client;
const marker = state.task as Entity;
try { const rows = await db.unsafe("select id, company_id, title, description, status from issues where id = $1", [marker.id]);
state.targetCanaryMatches = rows.length === 1 && rows[0]!.company_id === marker.companyId && rows[0]!.title === marker.title && rows[0]!.description === marker.description && rows[0]!.status === marker.status;
} finally { await db.end({ timeout: 5 }); }
const sourceTask = await ctx.input.api.get<Entity>(`/api/issues/${marker.id}`);
state.sourceCanaryUnchanged = sourceTask.title === marker.title && sourceTask.description === marker.description && sourceTask.status === marker.status;
await stop(o); state.targetStopped = true;
await command(o, ["worktree:cleanup", o.instance, "--instance", o.instance, "--home", o.home]);
state.targetCleaned = !(await stat(path.join(o.home, "instances", o.instance)).then(() => true, e => { if (e.code === "ENOENT") return false; throw e; }));
state.unrelatedSentinelUnchanged = await readFile(path.join(o.root, "unrelated-owned-sentinel.txt"), "utf8") === ctx.input.nonce;
const receipt = { targetUrl: state.targetUrl, healthyWhenStarted: state.actualTargetHealthy, targetStopped: state.targetStopped, targetCleaned: state.targetCleaned,
seededCanaryMatches: state.targetCanaryMatches, sourceCanaryUnchanged: state.sourceCanaryUnchanged, unrelatedSentinelUnchanged: state.unrelatedSentinelUnchanged, sourceRevision: state.sourceRevision };
await ctx.input.evidence("chat-catalog-owned-dev-instance.json", receipt);
const message = `The operator used the supported owned development-instance commands and recorded these facts. Please report its URL and observed health/cleanup state:\n\n${JSON.stringify(receipt)}`;
await sendChatMessage(ctx.input.page, message); state.devReport = await companionPhase(ctx, initial.runs.map(r => r.id), message);
}, grade: gradeOwnedDev, async verifyResult(ctx) { await expect(ctx.input.page.getByTestId("task-chat-composer-input")).toBeVisible(); },
});
dev.oracleVersion = "paperclip.chat-catalog.privileged.owned-dev-instance.v3"; dev.expectedRunCount = 2; dev.expectedHumanMessageCount = 2; dev.attemptTimeoutMs = 900000;
dev.cleanup = async ctx => { const o = owned.get(ctx); if (!o) return; await stop(o); if (o.config) await command(o, ["worktree:cleanup", o.instance, "--instance", o.instance, "--home", o.home]);
const root = await realpath(process.env.PAPERCLIP_RUNNER_E2E_TEMP_ROOT!), directory = await realpath(o.root);
if (!directory.startsWith(root + path.sep) || !path.basename(directory).startsWith("operator-dev-")) throw new Error("Refusing non-owned development cleanup");
await rm(directory, { recursive: true }); owned.delete(ctx); };
dev.sourceFiles = [...dev.sourceFiles, "../../cli/src/index.ts", "../../cli/src/commands/worktree.ts", "../../cli/src/commands/run.ts", "../e2e/fetch-allowed-port.ts", "redaction.ts"];
export const privilegedOwnedDevCases = [dev];