mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-14 08:53:19 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The chat feature catalog runs real native agents to prove the chat and task-steering promises. > - Two steering cases failed in setup before they could measure the product. > - `steering.approval-delivery` waited for a worker run to stay running while its question was pending. The product parks a native turn as soon as a durable question is saved, so that state never occurs. > - `steering.steer-active` waited for the queue to say steering is "available". The OpenCode harness reports that it cannot steer, so the queue says "unsupported". > - This pull request gates each case on the real product state and grades the behavior that state promises. > - The benefit is that both cases now measure their user promise on the evaluated profile, without faked runs or faked steering. ## Linked Issues or Issue Description **What happened** Both cases failed during setup on `runner-opencode`. `steering.approval-delivery` stopped with "The actual question fixture did not reach its running pending-input boundary": the worker's `request_human_input` call parked its turn (run succeeded, `reportedWorkDisposition: yielded`, continuation `response_wake` for the question). `steering.steer-active` timed out waiting for `steeringDisposition: "available"`; the OpenCode driver declares `steering: false`, so the queue reports `"unsupported"` and the page shows Interrupt instead of Steer. **Expected behavior** `steering.approval-delivery` should treat the saved yield and the pending question as the pending-input state, queue ordinary comments and the bound answer behind a real active worker turn, and prove one continuation receives the answer. `steering.steer-active` should grade same-turn steering when the harness can steer, and grade the honest fallback when it cannot: the message stays visibly queued, nothing is recorded as steered, and one normal turn consumes it. **Steps to reproduce** 1. Dispatch `chat-features.runner-opencode.local.steering.approval-delivery` on master. 2. See the setup stop after the worker run succeeds with a yielded result. 3. Dispatch `chat-features.runner-opencode.local.steering.steer-active` on master. 4. See setup time out while `GET /api/issues/:id/queued-comments` reports `steeringDisposition: "unsupported"`. ## What Changed - `runtime-approval-queue.ts`: gate the question on the parked turn (`pendingInputYield`: succeeded, yielded, `response_wake` keyed to the question). An ordinary human message then wakes a real second Morgan turn that waits at the task gate. Ordinary comments and the bound answer queue behind that turn. The run count goes from 3 to 4. - `runtime-approval-queue.ts`: remove the reply keyword checks (`acornNamingDirection` and the "I have answered" regex). Grade exactly one saved answer path instead: the human response control, or an attributed canonical response. - `runtime-active-steering.ts`: accept either definitive capability. Unsupported branch: no enabled Steer control, a visible Interrupt control, a steer attempt with the person's session refused with `409 steering_unsupported`, no `issue.queued_comment_steered` activity, the exact message still queued, and one continuation that consumes it and registers a rendered blue `website.html`. - `runtime-active-steering.ts`: remove the reply keyword checks and the `/blue/` match on the delivery document. The rendered page proves the color. - Update calibration tests in both directions, the workspace setup test, and the descriptors (`oracleVersion` bumps, `expectedRunCount` 4 for approval-delivery). The scenario YAML is unchanged. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains/runtime-active-steering.test.ts tests/runner-e2e/chat-catalog-domains/runtime-approval-queue.test.ts tests/runner-e2e/chat-catalog-domains/runtime-approval-workspace.test.ts tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts`: 87 passed. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in the changed files. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged. - The live proof is a paid run of `chat-features.runner-opencode.local.steering.approval-delivery` and `chat-features.runner-opencode.local.steering.steer-active`. ## Risks - approval-delivery now depends on a human message waking the worker while its question is pending. The product does this (`supersedeOnUserComment: false` keeps the card open, and comments wake the assignee unless the task is done or cancelled). If the worker does not run the gate on that turn, setup fails with a named label. - steer-active on OpenCode now measures the honest unsupported fallback, not same-turn delivery. A steering-capable profile still runs the same-turn branch. - Low risk to the product: only eval code changes. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
61 lines
4.9 KiB
TypeScript
61 lines
4.9 KiB
TypeScript
import { afterEach, expect, it, vi } from "vitest";
|
|
import { mkdir, mkdtemp, realpath, rm, writeFile } from "node:fs/promises";
|
|
import { tmpdir } from "node:os";
|
|
import path from "node:path";
|
|
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
|
|
const prepare = vi.hoisted(() => vi.fn());
|
|
const poll = vi.hoisted(() => vi.fn());
|
|
vi.mock("./runtime-busy-worker.js", async original => ({ ...await original<object>(), prepareRuntimeWorkerGate: prepare }));
|
|
vi.mock("../api.js", async original => ({ ...await original<object>(), pollUntil: poll }));
|
|
import { readRuntimeTaskGate, taskBoundRuntimeGate } from "./runtime-busy-worker.js";
|
|
import { runtimeApprovalQueueCases } from "./runtime-approval-queue.js";
|
|
const roots: string[] = [];
|
|
afterEach(async () => { vi.restoreAllMocks(); vi.unstubAllEnvs(); poll.mockReset(); for (const root of roots.splice(0)) await rm(root, { recursive: true, force: true }); });
|
|
const parkedResult = { nativeResult: { reportedWorkDisposition: "yielded", evidence: [{ ref: "interaction:question" }],
|
|
continuation: { kind: "response_wake", idempotencyKey: "interaction-response:question" } } };
|
|
async function fixture(run: { status: string; resultJson: unknown } = { status: "succeeded", resultJson: parkedResult }) {
|
|
const root = await mkdtemp(path.join(tmpdir(), "approval-owned-task-")); roots.push(root); vi.stubEnv("PAPERCLIP_RUNNER_E2E_TEMP_ROOT", root);
|
|
const cwd = path.join(root, "actual-isolated-task"); await mkdir(cwd);
|
|
const gate = taskBoundRuntimeGate("approval-owned");
|
|
await writeFile(path.join(cwd, `${gate.gateName}.waiting`), "waiting");
|
|
const task = { id: "task", companyId: "company", assigneeAgentId: "morgan", executionWorkspaceId: "workspace" };
|
|
const workspace = { id: "workspace", companyId: "company", cwd };
|
|
const post = vi.fn(async (_url: string, _body: unknown) => task);
|
|
prepare.mockImplementation(async (_ctx, options) => { expect(options).toEqual({ taskBound: true }); return { gate, worker: { id: "morgan" } }; });
|
|
const get = async (url: string) => url.endsWith("/workspace") ? { workspace }
|
|
: url.endsWith("/interactions") ? [{ id: "question", kind: "ask_user_questions", status: "pending", issueId: "task", createdByAgentId: "morgan", sourceRunId: "run" }]
|
|
: url === "/api/heartbeat-runs/run" ? { id: "run", ...run } : task;
|
|
const ctx = { input: { nonce: "approval-owned", fixtures: { company: { id: "company" } }, api: { post, get } }, allRuns: async () => [{ id: "run", agentId: "morgan", status: run.status }] } as unknown as ChatCatalogContext;
|
|
return { ctx, root, cwd, task, workspace, gate, post };
|
|
}
|
|
it("approval setup gates the question on the product's parked pending-input turn and scripts a cwd-local bounded wait", async () => {
|
|
const f = await fixture();
|
|
poll.mockImplementationOnce(async options => { const value = await options.load(); expect(options.accept(value)).toBe(true); expect(options.reject(value)).toBeUndefined(); throw new Error("verified pending-input boundary"); });
|
|
await expect(runtimeApprovalQueueCases[0]!.setup(f.ctx)).rejects.toThrow("verified pending-input boundary");
|
|
expect(f.post.mock.calls[0]![1]).toMatchObject({ assigneeAgentId: "morgan", status: "todo" });
|
|
const body = f.post.mock.calls[0]![1] as { description: string };
|
|
expect(body.description).toContain(`run ${f.gate.command} once`);
|
|
expect(body.description).not.toContain(f.root);
|
|
});
|
|
it.each([
|
|
["a question turn that is still running", { status: "running", resultJson: null }, false],
|
|
["a turn that completed without parking on the question", { status: "succeeded", resultJson: { nativeResult: { ...parkedResult.nativeResult, reportedWorkDisposition: "completed" } } }, true],
|
|
])("approval setup does not accept %s", async (_label, run, rejected) => {
|
|
const f = await fixture(run);
|
|
poll.mockImplementationOnce(async options => { const value = await options.load(); expect(options.accept(value)).toBe(false); expect(Boolean(options.reject(value))).toBe(rejected); throw new Error("checked"); });
|
|
await expect(runtimeApprovalQueueCases[0]!.setup(f.ctx)).rejects.toThrow("checked");
|
|
});
|
|
it("the active-turn gate binds the actual assigned task workspace", async () => {
|
|
const f = await fixture();
|
|
await expect(readRuntimeTaskGate(f.ctx, f.gate, "task", "morgan")).resolves.toBe("waiting");
|
|
expect(f.gate.boundTaskId).toBe("task"); expect(f.gate.boundWorkspaceId).toBe("workspace");
|
|
expect(f.gate.ready).toBe(path.join(await realpath(f.cwd), `${f.gate.gateName}.waiting`));
|
|
});
|
|
it.each(["company", "assignee", "workspace"])("the active-turn gate rejects foreign %s readiness", async invalid => {
|
|
const f = await fixture();
|
|
if (invalid === "company") f.workspace.companyId = "foreign";
|
|
if (invalid === "assignee") f.task.assigneeAgentId = "other";
|
|
if (invalid === "workspace") f.task.executionWorkspaceId = "other";
|
|
await expect(readRuntimeTaskGate(f.ctx, f.gate, "task", "morgan")).rejects.toThrow(/actual company, assignee and workspace/);
|
|
});
|