## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Heartbeat prepares, dispatches, and completes each agent run. > - After the runtime-selection extraction, the main service still had 7,295 lines. > - Provider dispatch and workspace finalization formed one large block with shared cleanup rules. > - This pull request moves that block into a typed runtime-execution module. > - The service now has 6,417 lines. Readers can follow admission and outer cleanup without reading the provider execution details. ## Linked Issues or Issue Description Follow-up to #15746. I searched open heartbeat PRs and issues. I found no competing runtime-execution extraction. **What existing behavior does this improve?** The organization and test coverage of native and legacy heartbeat execution. **Current behavior** The heartbeat service contains a 912-line block for provider dispatch, usage capture, workspace finalization, and instruction cleanup. **Proposed behavior** Move that block to `server/src/services/heartbeat/runtime-execution.ts`. Keep the atomic dispatch gate, admission, terminal run status, and outer lease cleanup in the service. Preserve run behavior. **Reason and benefit** This gives runtime execution a named boundary and direct tests. It removes 878 lines from the main service in one bounded change. **Breaking changes** None intended. There are no schema, API, or policy changes. ## What Changed - Add `executeHeartbeatRuntime` for prepared native and legacy execution, usage capture, workspace finalization, and gateway/instruction cleanup. - Return an explicit dispatch result. Update outer ownership state through immediate callbacks, including before exceptions. Read instruction state through live getters. - Move the two internal recovery error classes with their existing behavior. The caller imports the same classes for `instanceof` checks. - Add 15 database-backed execution tests. Keep the existing heartbeat regression suites. - Repair the native Stop acceptance fixture: installed Codex now takes precedence over `PATH`. A test-only loader pins the fake provider. An isolated home and dummy key keep readiness offline. Add four loader tests. - Document the module boundary and fixture setup. ## Verification - Passed: eight focused heartbeat suites (483 distinct tests, including 15 new execution tests). The boundary suite passed again after adding the failed-restore retention case from review. - Passed: `node --test tests/e2e/fixtures/composer-stop-codex-loader.test.mjs` (four tests). - Passed: `pnpm -r typecheck` and `pnpm build`. - Passed: AST comparison of all 912 moved lines after the explicit boundary substitutions. SQL template text, recovery class bodies, public exports, and the rest of the executor match the baseline. - Passed: browser Stop journeys for legacy process and native Paperclip Runner (two tests). The native fixture log proves provider `turn/start` before Stop and `turn/interrupt` after Stop. - Passed: all three ACP continuation browser journeys (same-session queued input, no replay of an interrupted write, and a new direction after composer Stop). Five relevant browser journeys passed in total. - Passed: the full CI test groups, including all server, chat, workspace, serialized-server, runner, and browser shards. - Local full-suite limitation: `pnpm test:run` first hit the macOS shared-memory segment limit while starting embedded PostgreSQL. After releasing only this task’s unused acceptance-server segments, the retry progressed without reported failures. I stopped that duplicate run after the full CI test groups passed. - Passed: all 56 PR checks/statuses on `8694c116e8d43ed8ac11bc4c58623ea3af34a675`. Greptile is 5/5 with no unresolved review threads. ## Risks - Cleanup depends on state written before an error. The module reports dispatch, retention, and recovery ownership immediately. Tests cover early returns and exceptional cleanup. - Instruction callbacks can replace saved metadata during execution. Live getters preserve those changes. - Native acceptance uses a real server, database, browser, and Rust runner with a deterministic provider. It does not verify a live provider service. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and repository/tool access. The exact serving model ID and context-window limit were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
12 KiB
Composer Stop and task controls
The empty composer shows Stop while this task has a live execution and the viewer can manage task controls. Stop creates the same manual pause hold as Pause work / Pause subtree in the task menu. A parent pause includes its descendants. Cancellation remains a separate menu action.
Text (after trimming) or attachments switch the button back to Send. Uploading and failed attachments retain their existing restrictions. Keyboard submission never invokes Stop. Queued-message editing and structured interactions keep their existing actions, and text entered while stopping remains in the draft.
Pause dispatches without a preview dialog or reason. The button stays pending while affected run state is checked; native cancellation must be acknowledged. A saved hold with unconfirmed termination produces an error rather than a success claim. The cancellation dialog requires a valid preview and excludes terminal tasks. Resume/restore retain the optional wake-agents checkbox.
Resume and execution recovery
Resume releases a hold. Waking agents is optional and only applies to tasks in
todo, in_progress, or in_review; parked and terminal tasks stay untouched.
The current execution-recovery policy requires verified outcomes before a stopped
provider can restart. If any affected task still needs that reconciliation,
Resume with wake enabled returns an inline error and preserves the pause. The
operator can release the pause without waking agents, then use the existing
execution-reconciliation flow after reviewing the stopped run. Resume does not
claim that unknown provider actions completed or were never performed.
A completed release remains successful if a best-effort wake fails. Its response
includes optional wakeFailures, the page reports them inline, and remaining
eligible tasks still receive their wake requests. No new endpoint is introduced.
The deterministic E2E fixtures prove interruption, then record their known lack
of external effects through the existing reconciliation API before continuing.
Embedded ACP also supports verified continuation of an interrupted local session.
The adapter must acknowledge cancellation and prove a preserved session with
settled read-only work. Stop waits for provider cleanup. Forced local termination uses the actual
child-process handle captured at spawn, including on Windows. An unavailable
handle does not authorize a signal or replay. Unknown actions remain
blocked, and task detail shows the reason even after recovery bookkeeping resolves.
A run-level Stop leaves the task unpaused; a subsequent comment can continue the
same session with the earlier queued messages. Composer Stop still creates a
pause hold. New board messages require Resume first. Both comment creation and
updates that include a comment return 409 while an effective task or ancestor
pause hold is active. Interrupted agents may still report their results. Neither path permits a fresh-session fallback
when the interrupted checkpoint cannot be restored.
The credential-free ACP regression journey uses an actual ACP child process:
pnpm exec playwright test --config tests/e2e/playwright.config.ts tests/e2e/acp-stop-continuation.spec.ts
It covers the queued-follow-up sequence (queue a second request, stop, then send “go”), same-session delivery of both messages, and an unfinished write that stops mutating its file but retains a visible execution blocker. Unit and integration tests additionally cover pre-start Stop, unavailable/changed sessions, rotating scratch directories, cancellation acknowledgment, a provider that hangs during cleanup after returning cancellation, deferred-wake adoption, and company-scoped blocker lookup. Hosted-provider behavior is a separate smoke test.
The browser tests also require the continued provider to complete the task through the agent API. Restoring a session must refresh its run identity, API credential, and scratch environment. The same conversation must not reuse the stopped run's credential. A regression test checks distinct run IDs and token hashes across the restart without logging the credentials themselves.
Historical behavior, superseded by the composer takeover: on 2026-09-09, all three ACP browser journeys passed. A manual browser walk-through also queued a request, used composer Stop, sent “go” while paused, and selected Resume work. The pause stayed in place during the conversation reply. Resume restored the same provider session, answered the pending request once, and moved the task to Done through the current run's authenticated API call. These tests use a deterministic ACP child process; they do not call Drive or another external app. The manual unfinished-write check also confirmed that file size stayed unchanged for five seconds after Interrupt. Sending “go” displayed the reconciliation reason and did not start another provider prompt.
A separate live Claude ACP smoke test interrupted a Bash tool writing only to a disposable local file: cancellation settled in 1,167 ms, output remained unchanged for five seconds, and no matching tool process remained. The shell action correctly did not receive automatic replay permission. A second live Claude check interrupted a response with no tools, restored the exact same provider session, and received the requested follow-up answer. These are local-provider observations, not a latency guarantee or proof for every provider and remote sandbox.
Quiet task feedback
The visible task/subtree does not produce duplicate state toasts. Its live
notifications are suppressed while foregrounded, including descendant runs;
unrelated and background work retains notifications. Tree-control results use
inline state, and failures stay in the composer, page, or confirmation dialog.
The amber composer takeover contains “Subtree is paused.” (or “Task is paused.”),
a short instruction to resume before sending, and Resume. It replaces input
controls in both task interfaces, cannot be dismissed, and preserves text and
attachment drafts. An inherited hold links to the ancestor task. Pending resume
keeps the takeover visible; failed resume leaves the task paused. Expected cancellation uses a muted gray disclosure with optional details.
This is recorded as a product rule in DESIGN.md.
The follow-up passed 213 focused tests, both isolated runner E2E journeys (including no-toast assertions), UI typecheck/build, token gates, and Storybook build. Paused, expanded cancellation, Stop without toasts, mobile, and light stories were inspected in the browser.
Storybook
Run from the worktree:
pnpm --filter @paperclipai/ui exec storybook dev -p 6016 -c storybook/.storybook --no-open
Open http://localhost:6016/?path=/story/composer-execution-controls--running-empty.
The Composer / Execution controls stories compose the production composer and
menu/dialog controls together. They cover text switching, attachment-only,
idle, stopping, paused, errors, cancellation preview/loading, and mobile/light
presentations. The Storybook state transitions simulate requests; runner
verification belongs to the isolated browser suite below.
Automated verification
pnpm exec vitest run --project @paperclipai/ui ui/src/components/task-chat/TaskChatComposer.test.tsx ui/src/components/TaskChatThread.test.tsx ui/src/pages/IssueDetail.test.tsx ui/src/lib/wait-for-stopped-runs.test.ts
pnpm exec vitest run --project @paperclipai/server server/src/__tests__/issue-tree-control-routes.test.ts
pnpm check:token-gates
pnpm build-storybook
pnpm -r typecheck
pnpm test:run
pnpm build
The component/page tests cover whitespace, attachments and upload restrictions, permissions, keyboard submission, queue edits, duplicate clicks, draft preservation, shared pause requests, compact cancellation, and visible errors. Stop verification tests include continued execution, native acknowledgment, network failures, and hung status requests. Route tests cover pause dispatch, authorization, cancellation, and opt-in resume wakeups with terminal/company exclusions.
Isolated browser acceptance
Legacy process coverage needs no provider credentials:
pnpm exec playwright test --config tests/e2e/playwright-composer-stop.config.ts
To include native execution, build the real runner and deterministic Codex protocol fixture from this checkout, then point the suite at their absolute paths:
cargo build --manifest-path packages/paperclip-runner/runner/Cargo.toml --bin paperclip-runnerd --bin fake-codex-app-server
PAPERCLIP_STOP_FAKE_CODEX="$PWD/packages/paperclip-runner/runner/target/debug/fake-codex-app-server" \
PAPERCLIP_RUNNER_BINARY="$PWD/packages/paperclip-runner/runner/target/debug/paperclip-runnerd" \
pnpm exec playwright test --config tests/e2e/playwright-composer-stop.config.ts
The suite boots a disposable local-trusted instance on port 3199 (override with
PAPERCLIP_E2E_PORT). It never attaches to an existing server. Native coverage
is explicitly skipped without the fixture; it must not use a logged-in provider
as a fallback. The acceptance server pins its Codex command resolver to the fixture
and uses an empty credential home, because an installed Codex dependency takes
precedence over PATH. Test companies are archived during cleanup.
For each runner, the journey starts a parent, child, and unrelated task, plus a terminal child. It sends while running and verifies the durable queue, clicks Stop, verifies interruption and the persisted hold, and observes three ten-second scheduler intervals without continuation. It resumes with wakeups, pauses from the menu, dismisses and then confirms cancellation, and verifies terminal-task exclusions and unrelated execution.
Timing attachments distinguish click-to-request from request-to-observed-stop.
Native proof requires provider turn/start before the click, turn/interrupt
after it, and durable cancellation acknowledgment. Legacy proof checks the
actual parent and child PIDs have exited with a one-second configured grace
period. HTTP success alone is insufficient.
For a release with real providers, repeat while an actual long-running tool is active, confirm the tool's own process/output stops, and verify a queued message is consumed on continuation. The deterministic provider exercises the native transport and cancellation protocol, not external provider behavior or every tool's process cleanup. Preserve these limits in any test report.
Recorded acceptance run (2026-09-09)
Both isolated browser journeys passed. Legacy Stop dispatched after 214 ms and observed both runs stopped 155 ms after dispatch; native dispatched after 212 ms and observed stop 320 ms later. These are single local observations including browser automation overhead, not latency guarantees. Native used the real runnerd and the repository's deterministic Codex protocol fixture. Real hosted provider/tool cleanup remains a release acceptance check.
Repository typecheck, production build, Storybook build, token gates, targeted
UI tests, and tree-control route tests passed. The broad UI run passed 5,491
tests with one five-second timeout in an unchanged IssuesList test; rerunning
that file passed all 46 tests. The repository test run encountered 17 failures
in unchanged suites, all reproduced in the original checkout:
server/src/__tests__/workspace-runtime.test.ts: 2 failures.server/src/services/workspace-runtime-exposure.test.ts: 7 failures.server/src/__tests__/execution-workspace-runtime-control-conflict.test.ts: 4 failures.server/src/__tests__/workspace-instance-cleanup.test.ts: 1 failure.server/src/__tests__/company-skills.test.ts: 2 failures.server/src/__tests__/worktree-seed-server-spawn.test.ts: 1 failure.
These baseline failures prevent a green repository-wide test result. The broad run was stopped after more than 30 minutes in its serial server lane once these failures were independently reproduced. Later full-suite groups did not run. The full UI suite and the feature's server route suite were run separately as described above.
The Tasks / Composer / Paused task takeover stories use the production composer
and cover task/subtree holds, saved drafts, resume progress/failure, and light/mobile layouts.