Files
PaperClipAI/server/src/services
DottaandClaude Opus 5.5 7f1cdb842d fix(evals): admit the owner-routed routine follow-up run and brief only the requested change (#21198)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The chat eval catalog checks that agents in chat do the right thing
on saved product state.
> - In `routines.manage-other-owner`, the CEO cannot change Morgan's
routine. The valid outcome is to route the change to Morgan in one
linked task.
> - That valid path uses three runs: the CEO's turn, Morgan's run on the
task, and the CEO's automatic completion follow-up on the chat. The case
allowed only two runs, so the harness stopped valid attempts with
"exceeded its declared run bound".
> - In one of those attempts, the CEO also told Morgan to use UTC.
Nobody asked for UTC. The routine was on America/Chicago. The oracle
correctly failed that change.
> - This pull request admits the third run only in its declared role,
and it tells agents to brief only the change the person asked for.
> - The benefit is that the eval accepts the valid delegation path and
still fails extra work and unrequested setting changes.

## Linked Issues or Issue Description

**What happened?**

Runner Full-Stack E2E runs 43298807862 and 43300040122 stopped
`routines.manage-other-owner` with "Scenario exceeded its declared run
bound". Both had three runs: the CEO turn, Morgan's run on the routed
task, and the CEO's `chat_task_completed` follow-up. In 43298807862 the
CEO's brief also told Morgan to switch the routine to UTC, and
`same-owned-definition-and-triggers` failed.

**Expected behavior**

The delegation path with exactly one routed owner task passes with three
runs. The refusal path stays at two runs. An agent that relays a change
keeps every setting the person did not ask to change.

**Steps to reproduce**

Run `chat-features.runner-opencode.local.routines.manage-other-owner`
and let the CEO route the change to Morgan.

**Paperclip version or commit**

master at 7747487ee7.

## What Changed

- `tests/runner-e2e/chat-catalog-domains/integrator-routines.ts`: the
case bound is now 3 runs (minimum 1). A new `declared-run-roles` check
allows at most 2 runs, plus at most one run that is the CEO's completion
follow-up (`chatCompletionFollowUp` from `run-roles.ts`) on the source
chat for the single owner task with a bound Morgan run. The observation
now records the source chat id. Oracle version is now
`manage-other-owner.v7`.
- `tests/runner-e2e/chat-catalog-descriptors.json`: the static
descriptor matches the new bound and oracle version.
- `tests/runner-e2e/chat-catalog-domains/integrator-routines.test.ts`:
calibration tests. The delegation path with 3 runs passes. A 4th run
fails. A second follow-up fails. 3 runs without a delegated task fail. A
follow-up for a different task fails. The 2-run refusal path still
passes.
- `server/src/services/agent-conversations.ts`: one sentence in the chat
directive task guidance: "When you relay or delegate a change, state
only what the person asked to change, and keep every other setting, such
as a time zone, owner or schedule detail, as it is."
- `server/src/__tests__/agent-conversations.test.ts`: red/green test
that the directive and the built chat task prompt carry that sentence.
- No scenario yaml change, so the TestOps manifest does not change.

## Verification

- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog
tests/runner-e2e/chat-catalog-domains/integrator`: 283 files, 3850 tests
pass.
- `pnpm test:e2e:runner:typecheck`: pass.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10/10
pass.
- `server`: `agent-conversations.test.ts` and
`board-access-denial.test.ts` pass (109 tests);
`codex-local-execute.test.ts` passes. The new directive test failed
before the directive change and passes after it.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs
792f8fb308 --check`: `changed: false`.
- `pnpm verify` exited 2 (descriptor JSON is a global CI input), so
required CI supplies broad coverage.
- Paid single-case run: Runner Full-Stack E2E 43313272302 on a throwaway
branch at this commit. Result is reported in a comment.

## Risks

- Low risk. The eval change only admits a third run in one declared
role. Extra or unbound runs still fail.
- The directive gains one sentence. It can make an agent more
conservative when it relays a change, which is the intent.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:10:47 -06:00
..
…
…
…
…
…