mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
fix(evals): admit the owner-routed routine follow-up run and brief only the requested change (#21198)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The chat eval catalog checks that agents in chat do the right thing on saved product state. > - In `routines.manage-other-owner`, the CEO cannot change Morgan's routine. The valid outcome is to route the change to Morgan in one linked task. > - That valid path uses three runs: the CEO's turn, Morgan's run on the task, and the CEO's automatic completion follow-up on the chat. The case allowed only two runs, so the harness stopped valid attempts with "exceeded its declared run bound". > - In one of those attempts, the CEO also told Morgan to use UTC. Nobody asked for UTC. The routine was on America/Chicago. The oracle correctly failed that change. > - This pull request admits the third run only in its declared role, and it tells agents to brief only the change the person asked for. > - The benefit is that the eval accepts the valid delegation path and still fails extra work and unrequested setting changes. ## Linked Issues or Issue Description **What happened?** Runner Full-Stack E2E runs 43298807862 and 43300040122 stopped `routines.manage-other-owner` with "Scenario exceeded its declared run bound". Both had three runs: the CEO turn, Morgan's run on the routed task, and the CEO's `chat_task_completed` follow-up. In 43298807862 the CEO's brief also told Morgan to switch the routine to UTC, and `same-owned-definition-and-triggers` failed. **Expected behavior** The delegation path with exactly one routed owner task passes with three runs. The refusal path stays at two runs. An agent that relays a change keeps every setting the person did not ask to change. **Steps to reproduce** Run `chat-features.runner-opencode.local.routines.manage-other-owner` and let the CEO route the change to Morgan. **Paperclip version or commit** master at7747487ee7. ## What Changed - `tests/runner-e2e/chat-catalog-domains/integrator-routines.ts`: the case bound is now 3 runs (minimum 1). A new `declared-run-roles` check allows at most 2 runs, plus at most one run that is the CEO's completion follow-up (`chatCompletionFollowUp` from `run-roles.ts`) on the source chat for the single owner task with a bound Morgan run. The observation now records the source chat id. Oracle version is now `manage-other-owner.v7`. - `tests/runner-e2e/chat-catalog-descriptors.json`: the static descriptor matches the new bound and oracle version. - `tests/runner-e2e/chat-catalog-domains/integrator-routines.test.ts`: calibration tests. The delegation path with 3 runs passes. A 4th run fails. A second follow-up fails. 3 runs without a delegated task fail. A follow-up for a different task fails. The 2-run refusal path still passes. - `server/src/services/agent-conversations.ts`: one sentence in the chat directive task guidance: "When you relay or delegate a change, state only what the person asked to change, and keep every other setting, such as a time zone, owner or schedule detail, as it is." - `server/src/__tests__/agent-conversations.test.ts`: red/green test that the directive and the built chat task prompt carry that sentence. - No scenario yaml change, so the TestOps manifest does not change. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog tests/runner-e2e/chat-catalog-domains/integrator`: 283 files, 3850 tests pass. - `pnpm test:e2e:runner:typecheck`: pass. - `node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10/10 pass. - `server`: `agent-conversations.test.ts` and `board-access-denial.test.ts` pass (109 tests); `codex-local-execute.test.ts` passes. The new directive test failed before the directive change and passes after it. - `node tests/runner-e2e/source-sdk-public-inputs.mjs792f8fb308--check`: `changed: false`. - `pnpm verify` exited 2 (descriptor JSON is a global CI input), so required CI supplies broad coverage. - Paid single-case run: Runner Full-Stack E2E 43313272302 on a throwaway branch at this commit. Result is reported in a comment. ## Risks - Low risk. The eval change only admits a third run in one declared role. Extra or unbound runs still fail. - The directive gains one sentence. It can make an agent more conservative when it relays a change, which is the intent. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
549816e6a0
commit
7f1cdb842d
5 files changed
+35
-8
No files matched your search
@@ -1011,6 +1011,12 @@ describe("chat prompt policy", () => {
|
||||
const prompt = buildPaperclipTaskMarkdown({ issue: { id: "chat", identifier: null, title: "Chat", workMode: "ask", conversationAgentId: "agent" } });
|
||||
expect(prompt).toContain("blockedByIssueIds with PATCH /api/issues/{id}");
|
||||
});
|
||||
it("tells chat to brief or delegate only the requested change and keep every other setting", () => {
|
||||
const line = "When you relay or delegate a change, state only what the person asked to change, and keep every other setting, such as a time zone, owner or schedule detail, as it is.";
|
||||
expect(AGENT_CHAT_DIRECTIVE).toContain(line);
|
||||
const prompt = buildPaperclipTaskMarkdown({ issue: { id: "chat", identifier: null, title: "Chat", workMode: "ask", conversationAgentId: "agent" } });
|
||||
expect(prompt).toContain(line);
|
||||
});
|
||||
it("tells chat to link the profile of an agent it acted on without turning the link into a wake mention", () => {
|
||||
expect(AGENT_CHAT_DIRECTIVE).toContain("link its profile at /<prefix>/agents/<urlKey or id>");
|
||||
expect(AGENT_CHAT_DIRECTIVE).toContain("use agent:// only when you intend to notify that agent");
|
||||
|
||||
@@ -116,7 +116,7 @@ When a user answers a pending confirmation in chat, read the latest cards and us
|
||||
|
||||
Before handing off work, inspect available projects and repositories. Every task you create from this chat must belong to a suitable project. Reuse an appropriate existing project; otherwise use create_project. Consider all relevant available repositories and pass repositoryIds for one or multiple repositories when the work spans them. For existing GitHub repositories you can access that are absent from the catalog, pass their HTTPS repositoryUrls; this registers them with the project without creating remote GitHub repositories. You may combine known IDs and URLs and attach multiple repositories. The direct HTTP equivalent is POST /api/companies/{companyId}/projects with name, repositoryIds and/or repositoryUrls arrays, and an idempotencyKey. Include all selected repositories in that creation; do not combine these arrays with workspace. Never invent repository IDs or substitute inaccessible repositories. Ask when the choice is materially ambiguous or required access is missing. Non-code projects may need no repository.
|
||||
|
||||
Create ordinary assigned tasks; this conversation is never a parent. Choose the shape that fits the work: one task for one coherent outcome, a top-level task with child tasks under it when one owner coordinates several parts, or separate top-level tasks for unrelated outcomes. With create_task, create the top-level task first, then pass its ID as parentTaskId for each child. Nest only one level: a child of a child is placed beside it as a sibling. Use blockers, not nesting, for ordering. Give each task a clear outcome, context, acceptance criteria, and project. Default the assignee to the agent in this conversation (yourself). Assign a different agent or person only when the user explicitly specifies that owner. A task requested "for me" does not by itself mean assigning it to the user. Do not choose a different agent based on perceived suitability. Use create_task with initialPlan to copy the relevant plan into the new task before execution starts. If using the HTTP API directly, POST /api/companies/{companyId}/issues with projectId, assigneeAgentId, status: "todo", initialPlan containing the relevant plan Markdown, and an idempotencyKey; omit parentId for a top-level task, or set it to a top-level task you created from this conversation for a child. Putting a plan in description does not create the task's plan document. Verify the new task's plan document before claiming the handoff is complete. Preserve the original plan here. Carry forward only the remaining execution steps, not completed planning, approval, project creation, or task creation steps. Include the source conversation ID, approved plan revision ID, and accepted interaction ID so the worker can verify the recorded approval. State the approved scope and what is already done; do not claim the new plan document has its own approval. The worker should execute that authorized scope, and ask again only if scope changes or another applicable gate requires it. When splitting work, include the relevant part of the plan in each task. Create and link each task before claiming it exists.
|
||||
Create ordinary assigned tasks; this conversation is never a parent. Choose the shape that fits the work: one task for one coherent outcome, a top-level task with child tasks under it when one owner coordinates several parts, or separate top-level tasks for unrelated outcomes. With create_task, create the top-level task first, then pass its ID as parentTaskId for each child. Nest only one level: a child of a child is placed beside it as a sibling. Use blockers, not nesting, for ordering. Give each task a clear outcome, context, acceptance criteria, and project. When you relay or delegate a change, state only what the person asked to change, and keep every other setting, such as a time zone, owner or schedule detail, as it is. Default the assignee to the agent in this conversation (yourself). Assign a different agent or person only when the user explicitly specifies that owner. A task requested "for me" does not by itself mean assigning it to the user. Do not choose a different agent based on perceived suitability. Use create_task with initialPlan to copy the relevant plan into the new task before execution starts. If using the HTTP API directly, POST /api/companies/{companyId}/issues with projectId, assigneeAgentId, status: "todo", initialPlan containing the relevant plan Markdown, and an idempotencyKey; omit parentId for a top-level task, or set it to a top-level task you created from this conversation for a child. Putting a plan in description does not create the task's plan document. Verify the new task's plan document before claiming the handoff is complete. Preserve the original plan here. Carry forward only the remaining execution steps, not completed planning, approval, project creation, or task creation steps. Include the source conversation ID, approved plan revision ID, and accepted interaction ID so the worker can verify the recorded approval. State the approved scope and what is already done; do not claim the new plan document has its own approval. The worker should execute that authorized scope, and ask again only if scope changes or another applicable gate requires it. When splitting work, include the relevant part of the plan in each task. Create and link each task before claiming it exists.
|
||||
|
||||
When the person says an existing task must not start until another task is done or approved, save that order as a blocker: add the prerequisite to the waiting task's blockedByIssueIds with PATCH /api/issues/{id}, keeping its existing blockers. A comment or a promise to hold the work does not stop it; the saved blocker keeps the waiting task from starting and wakes its assignee once the prerequisite is done. Link both tasks in your reply.
|
||||
|
||||
|
||||
@@ -4678,8 +4678,8 @@
|
||||
"../../server/src/services/cron.ts",
|
||||
"chat-scenarios/routines.yaml"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.integrator.manage-other-owner.v6",
|
||||
"expectedRunCount": 2,
|
||||
"oracleVersion": "paperclip.chat-catalog.integrator.manage-other-owner.v7",
|
||||
"expectedRunCount": 3,
|
||||
"minimumRunCount": 1,
|
||||
"expectedHumanMessageCount": 1,
|
||||
"turnTimeoutMs": 120000,
|
||||
|
||||
@@ -8,7 +8,7 @@ function valid(id: string) {
|
||||
return { scenarioId: id, origin: "https://fixture.test", companyId: "company", leadId: "ceo", state: { project: { id: "project" }, worker: { id: "morgan" }, routine: structuredClone(routine), originalRevisions: [structuredClone(revision)], setupActivityIds: ["setup-created"] },
|
||||
tasks: [], projects: [], goals: [], comments: [], documents: [], revisions: [], humanQueries: [], activity: (managed ? [{ id: "setup-created", action: "routine.created", entityType: "routine", entityId: "routine", actorType: "user", actorId: "local-board", agentId: null }] : []) as Array<Record<string, unknown>>, links: managed ? ["/ACORN/routines/routine"] : [] as string[], runs: [{ id: "run", agentId: "ceo", status: "succeeded" }] as Array<Record<string, unknown>>,
|
||||
reply: managed ? "I cannot manage Morgan's routine because only its owner may change the schedule. Morgan or a human operator can make this change." : "Saved one weekday routine for 9 in the morning America/Chicago. Each occurrence creates a task in Launch; future summaries have not run.",
|
||||
routines: [routine], histories: { routine: [] }, routineRevisions: { routine: [revision] }, browserTimezone: "America/Chicago" };
|
||||
routines: [routine], histories: { routine: [] }, routineRevisions: { routine: [revision] }, browserTimezone: "America/Chicago", sourceId: "source" };
|
||||
}
|
||||
describe("routine definition and management independent controls", () => {
|
||||
for (const definition of integratorRoutinesCases) it(`${definition.scenarioId}: saved public outcome and missing evidence`, () => {
|
||||
@@ -63,6 +63,24 @@ describe("owner-routed routine change", () => {
|
||||
outside.activity.push({ id: "w", action: "routine.trigger_updated", entityType: "routine", entityId: "routine", agentId: "morgan", runId: "other-run" });
|
||||
expect(gradeIntegratorRoutines(outside).passed).toBe(false);
|
||||
});
|
||||
|
||||
const followUp = (id: string, taskId = "delegated") => ({ id, agentId: "ceo", status: "succeeded",
|
||||
contextSnapshot: { issueId: "source", wakeReason: "chat_task_completed", paperclipWake: { chatCompletionUpdates: [{ id: taskId }] } } });
|
||||
const roles = (o: ReturnType<typeof valid>) => gradeIntegratorRoutines(o).checks.find(check => check.id === "declared-run-roles")?.passed;
|
||||
it("admits a third run only as the CEO's one completion follow-up for the single routed owner task", () => {
|
||||
const relayed = delegated(); relayed.runs.push(followUp("follow-up"));
|
||||
expect(gradeIntegratorRoutines(relayed).passed).toBe(true);
|
||||
const fourth = delegated(); fourth.runs.push(followUp("follow-up"), { id: "extra", agentId: "ceo", status: "succeeded", contextSnapshot: { issueId: "source" } });
|
||||
expect(roles(fourth)).toBe(false);
|
||||
const repeated = delegated(); repeated.runs.push(followUp("follow-up"), followUp("again"));
|
||||
expect(roles(repeated)).toBe(false);
|
||||
const undelegated = valid("routines.manage-other-owner"); undelegated.runs.push({ id: "second", agentId: "ceo", status: "succeeded" }, followUp("follow-up"));
|
||||
expect(roles(undelegated)).toBe(false);
|
||||
const otherTask = delegated(); otherTask.runs.push(followUp("follow-up", "elsewhere"));
|
||||
expect(roles(otherTask)).toBe(false);
|
||||
const refused = valid("routines.manage-other-owner"); refused.runs.push({ id: "second", agentId: "ceo", status: "succeeded" });
|
||||
expect(roles(refused)).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
// Full cohort 7 on 9e910ff9ae: after a 403 the CEO changed Morgan's routine through the local board API, so the
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
import type { RoutineDetail, RoutineListItem, RoutineRevision, RoutineRunSummary } from "../../../packages/shared/src/types/routine.js";
|
||||
import type { ChatCatalogContext, ChatCatalogCase } from "../chat-catalog-contract.js";
|
||||
import { checks, hasLink, project, observation, type Entity, type InternalObservation } from "./internal-support.js";
|
||||
import { runIssueId } from "./run-roles.js";
|
||||
import { chatCompletionFollowUp, runIssueId } from "./run-roles.js";
|
||||
import { parseCron } from "../../../server/src/services/cron.js";
|
||||
import { fixtureWorker } from "./internal-definition.js";
|
||||
import { integratorDefinition, groundedStatement, inspectResult, observeInternal } from "./integrator-support.js";
|
||||
interface RoutinesObservation extends InternalObservation {
|
||||
routines: RoutineDetail[]; histories: Record<string, RoutineRunSummary[]>; routineRevisions: Record<string, RoutineRevision[]>;
|
||||
browserTimezone: string;
|
||||
browserTimezone: string; sourceId?: string;
|
||||
}
|
||||
const root = (ctx: ChatCatalogContext) => `/api/companies/${ctx.input.fixtures.company.id}`;
|
||||
async function observeRoutines(ctx: ChatCatalogContext, state: Record<string, unknown>): Promise<RoutinesObservation> {
|
||||
@@ -16,7 +16,7 @@ async function observeRoutines(ctx: ChatCatalogContext, state: Record<string, un
|
||||
const routines = await Promise.all(listed.map(row => api.get<RoutineDetail>(`/api/routines/${row.id}`)));
|
||||
const histories = Object.fromEntries(await Promise.all(routines.map(async row => [row.id, await api.get<RoutineRunSummary[]>(`/api/routines/${row.id}/runs`)])));
|
||||
const routineRevisions = Object.fromEntries(await Promise.all(routines.map(async row => [row.id, await api.get<RoutineRevision[]>(`/api/routines/${row.id}/revisions`)])));
|
||||
return { ...base, routines, histories, routineRevisions, browserTimezone: await ctx.input.page.evaluate(() => Intl.DateTimeFormat().resolvedOptions().timeZone) };
|
||||
return { ...base, routines, histories, routineRevisions, browserTimezone: await ctx.input.page.evaluate(() => Intl.DateTimeFormat().resolvedOptions().timeZone), sourceId: ctx.issue()?.id };
|
||||
}
|
||||
function routineObservation(value: unknown): RoutinesObservation | undefined {
|
||||
const o = value as RoutinesObservation;
|
||||
@@ -81,6 +81,9 @@ export function gradeIntegratorRoutines(value: unknown) {
|
||||
c.add("ownership-boundary-explained", route.valid && (route.delegated.length === 1 ? hasLink(o, "issues", route.delegated[0]!)
|
||||
: o.links.some(href => linksTo(o, href, "routines", [before?.id]) || linksTo(o, href, "agents", [worker?.id, worker?.urlKey]))),
|
||||
"The change is routed structurally: one task assigned to owner Morgan, created by the CEO and linked in the reply, with at most one Morgan run bound to it; or a reply link to the routine or Morgan's page where an authorized person acts. The reply's wording is left to a human reader.");
|
||||
const followUps = route.delegated.length === 1 && route.ownerRun && o.sourceId ? o.runs.filter(run => chatCompletionFollowUp(run, { leadId: o.leadId, sourceId: o.sourceId!, taskId: route.delegated[0]!.id })) : [];
|
||||
c.add("declared-run-roles", followUps.length <= 1 && o.runs.length <= 2 + followUps.length,
|
||||
"At most two runs, plus only the CEO's one completion follow-up on the source chat after the single owner task it routed to Morgan finished.");
|
||||
const setupActivityIds = Array.isArray(o.state.setupActivityIds) ? new Set(o.state.setupActivityIds as string[]) : undefined;
|
||||
const ownerRunWrite = (agentId: unknown, runId: unknown) => agentId === worker?.id && !!route.ownerRun && runId === route.ownerRun.id;
|
||||
const turnWrites = o.activity.filter(event => routineWrite(event, before?.id) && !setupActivityIds?.has(event.id) && !ownerRunWrite(event.agentId, (event as Activity).runId));
|
||||
@@ -107,6 +110,6 @@ export const integratorRoutinesCases: readonly ChatCatalogCase[] = [
|
||||
const routine = await ctx.input.api.get<RoutineDetail>(`/api/routines/${created.id}`), originalRevisions = await ctx.input.api.get<RoutineRevision[]>(`/api/routines/${created.id}/revisions`);
|
||||
const setupActivityIds = (await ctx.input.api.getRunObservation<Entity[]>(`${root(ctx)}/activity?limit=1000`)).map(row => row.id);
|
||||
return { project: p, worker, routine, originalRevisions, setupActivityIds };
|
||||
}, { oracleVersion: "paperclip.chat-catalog.integrator.manage-other-owner.v6", expectedRunCount: 2, minimumRunCount: 1, attemptTimeoutMs: 480_000,
|
||||
}, { oracleVersion: "paperclip.chat-catalog.integrator.manage-other-owner.v7", expectedRunCount: 3, minimumRunCount: 1, attemptTimeoutMs: 480_000,
|
||||
sourceFiles: ["chat-catalog-domains/integrator-support.ts", "chat-catalog-domains/internal-support.ts", "chat-catalog-domains/integrator-routines.ts", "chat-catalog-domains/run-roles.ts", "../../server/src/services/cron.ts", "chat-scenarios/routines.yaml"] }),
|
||||
];
|
||||
Reference in new issue
Block a user