[codex] Add heartbeat policy eval coverage (#9087)

## Thinking Path

> - Paperclip is the open source control plane people use to manage AI
agents for work.
> - The relevant subsystem is the agent heartbeat policy surface: the
Paperclip skill, default onboarding AGENTS.md, new-agent runtime
defaults, and promptfoo eval coverage for agent behavior.
> - A broad recovery PR collected several unrelated local-mainline
changes, which made review too large and mixed policy/eval updates with
server execution and UI work.
> - This PR extracts only the heartbeat policy and prompt-eval slice so
reviewers can assess the behavior contract independently.
> - The eval additions cover scoped wake handling, idle no-op behavior,
dependency-blocked comment triage, final disposition, budget hard stops,
and Phase 5 memory/control-surface policy expectations.
> - The benefit is a narrower review surface plus deterministic
follow-up guidance for server/shared tests that should back these
prompt-level checks.

## Linked Issues or Issue Description

Refs #8866

No public issue was filed for this split. This is a focused extraction
from the closed broad recovery PR so heartbeat policy and eval coverage
can be reviewed separately from execution behavior, work-product feature
work, plugin hardening, pipeline health, and unrelated UI polish.

## What Changed

- Added promptfoo release-gate cases for scoped wake payload handling,
idle exits, dependency-blocked comment triage, final disposition, and
budget hard-stop behavior.
- Added Phase 5 memory/control-surface prompt eval cases for provider
binding precedence, provenance/audit fields, hook cost/trust handling,
and auditable board command surfaces.
- Documented how these prompt evals map to deterministic server/shared
follow-up coverage.
- Updated agent policy guidance so operator-facing engineering outputs
such as PRs, branches, commits, previews, and runtime services get
matching work products.
- Defaulted new agent runtime config to skip timer heartbeats when there
is no actionable work, with focused test coverage.

## Verification

- `cd evals/promptfoo && npx promptfoo@latest validate -c
promptfooconfig.yaml` passes.
- `/srv/paperclip/home/paperclipai/paperclip/node_modules/.bin/vitest
run ui/src/lib/new-agent-runtime-config.test.ts` passes in an isolated
worktree after `pnpm install --ignore-scripts --frozen-lockfile` created
workspace links.
- A live promptfoo eval was not run because `OPENROUTER_API_KEY`,
`OPENAI_API_KEY`, and `ANTHROPIC_API_KEY` were unset in the workspace.

## Risks

Low-to-medium risk. The runtime default reduces timer-driven empty
heartbeats for newly created agents, so the main behavioral risk is
missing an edge case where timer wakes were expected despite no
actionable work. The promptfoo additions are deterministic assertion
coverage and documentation-only until a live eval is run with provider
credentials.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI GPT-5-based Codex coding agent in the Paperclip local Codex
adapter environment; exact hosted model ID and context window were not
exposed to the agent runtime. Tool use included shell, git, promptfoo
validation, Vitest, and the GitHub connector/CLI.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-07-06 16:31:14 -05:00
1 parent 903886bc79
commit dfc256a543
8 files changed
+252 -3

No files matched your search

@@ -6,6 +6,7 @@ You are an agent at Paperclip company.
- Keep the work moving until it is done. If you need QA to review it, ask them. If you need your boss to review it, ask them.
- Leave durable progress in task comments, documents, or work products, then update the issue to a clear final disposition before you exit.
- When your work produces a user-inspectable deliverable file, follow the Paperclip skill's "Generated Artifacts and Work Products" workflow before final disposition. Use `skills/paperclip/scripts/paperclip-upload-artifact.sh` when working in this repo, create/update an artifact work product when the file is the deliverable, and link the uploaded attachment in the final comment. Do not rely on local filesystem paths as the only access path. If an important file intentionally remains workspace-only, create/update a work product with `metadata.resourceRef.kind: "workspace_file"` and a workspace-relative path, then name that work product and path in the final comment. Treat browse/search as a fallback for recovering workspace files, not the preferred deliverable path.
- When your work produces or updates an operator-facing engineering output, create/update the matching work product: `pull_request` for opened PRs, `preview_url` for published previews, `runtime_service` for managed preview/dev services, `commit` for notable pushed commits, and `branch` when the branch itself is the handoff. A comment is not a substitute for the work product access path.
- Comments, documents, screenshots, work products, and `Remaining` bullets are evidence, not valid liveness paths by themselves.
- Final disposition checklist: mark `done` when complete and verified; use `in_review` only with a real reviewer, approval, interaction, or monitor path; use `blocked` only with first-class blockers or a named unblock owner/action; create delegated follow-up issues with blockers when another agent owns the next step; keep `in_progress` only when a live continuation path exists.
- Use child issues for parallel or long delegated work instead of polling agents, sessions, or processes.