mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-09 06:15:21 +02:00
DottaandPaperclip
263316609e
## Thinking Path > - Paperclip is the open source control plane people use to manage AI agents and their work > - The server coordinates agent heartbeats and preserves eligible live runs during a hot restart > - Shutdown previously waited for all heartbeat scheduler work before capturing the hot-restart snapshot > - A deployment heartbeat can itself be in that scheduler set while waiting for the restart, creating a circular wait > - The missing snapshot prevents startup from classifying and adopting the still-running agent process > - This pull request captures the snapshot first and skips scheduler/drain waits only for an eligible hot restart > - The benefit is a single SIGTERM can restart the server without losing eligible live agent runs ## Linked Issues or Issue Description - **Preflight:** Searched open and closed PRs for the hot-restart shutdown deadlock; no duplicate found. Reproduced on `master` and confirmed this is core Paperclip behavior. - **What happened:** During a hot restart initiated by a running deployment heartbeat, the SIGTERM handler waited for `heartbeatSchedulerInFlight` before calling `prepareHotRestartShutdown()`. The heartbeat was itself in that set and waited for restart completion, so shutdown never wrote the adoption snapshot. - **Expected behavior:** An eligible hot restart captures its snapshot before waiting for scheduler work, preserves live child processes, and exits after one SIGTERM. - **Steps to reproduce:** 1. Start a heartbeat that remains active while requesting a hot restart. 2. Send SIGTERM to the server process. 3. Observe shutdown waiting on the active scheduler task and startup finding an intent without a shutdown snapshot. - **Paperclip commit:** `992389480a243b97bda214227e0767eb8c3672af` - **Deployment/install:** Self-hosted server built from source. - **Adapter:** Not adapter-specific; reproduced with a Codex heartbeat. - **Database/access:** Embedded PGlite; agent bearer context. - **Environment:** Node `v22.22.2` on `Linux 6.17.0-1015-aws aarch64 GNU/Linux`. - **Privacy:** No secrets, private logs, user paths, or internal issue references are included. ## What Changed - Add a focused shutdown coordinator that prepares hot-restart state before waiting for heartbeat scheduler idleness. - Skip scheduler-idle and graceful-drain waits only when the hot-restart service returns `skipDrain: true`. - Preserve normal graceful shutdown behavior when no eligible intent exists or preparation fails. - Add regression coverage for pending scheduler work, normal shutdown, and preparation failure. ## Verification - `pnpm exec vitest run server/src/shutdown.test.ts` — 3 passed. - `pnpm --filter @paperclipai/server typecheck` — passed. - `pnpm exec vitest run server/src/__tests__/heartbeat-process-recovery.test.ts -t 'hot-restart'` — 3 passed, 88 skipped. - `git diff --check origin/master...HEAD` — passed. ## Risks - Low-to-moderate risk: shutdown ordering changes, but only the explicitly eligible hot-restart path bypasses scheduler-idle and run-drain waits. - Normal shutdown and hot-restart preparation failures retain the existing graceful behavior. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. This is a focused bug fix and does not duplicate planned roadmap work. ## Model Used - OpenAI Codex coding agent; exact runtime model ID and context-window size are not exposed to the agent. Tool use, shell execution, repository editing, and test execution were enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (no documentation change required for this internal shutdown-order fix) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>