mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-06 21:05:21 +02:00
Devin Foley
8b83d69e3c
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The heartbeat service dispatches agent runs, and issues in a project can share one project workspace (one working tree on disk). > - Two runs can execute in the same shared workspace at the same time. Each run mutates the same uncommitted files and branches, and the runs corrupt each other's state. > - Multi-agent projects hit this as soon as two issues in one project become active together, so the platform needs to serialize shared-workspace execution instead of relying on luck. > - This pull request adds a pre-dispatch gate: a run whose issue targets a busy shared workspace is deferred with a bounded scheduled retry instead of dispatched. > - The benefit is that concurrent issue runs in one project take turns in the shared working tree, while isolated-workspace runs and unrelated workspaces stay fully parallel. ## Linked Issues or Issue Description Fixes #10645 ## What Changed - `server/src/services/heartbeat.ts`: - New pre-dispatch gate in the run executor. Before adapter dispatch, when the run's issue has a `projectWorkspaceId` and the effective execution workspace mode is `shared_workspace`, the executor looks for a holder: another `running` run whose context issue shares the same project workspace. The gate covers every run shape that reaches adapter dispatch with issue context — assignee execution runs, comment/mention interaction wakes, and review-participant runs. - When a holder exists, the run throws `WorkspaceBusyDeferral` instead of dispatching. The outer catch recognizes the deferral: it cancels the run with `errorCode: "workspace_busy"` (contention is not a failure), cancels its wakeup, schedules a retry through the existing `scheduleBoundedRetryForRun` primitive (`workspace_busy` reason, 60–120 s jittered delay), and returns the agent to idle. The issue execution lock transfers to the scheduled retry run, so the issue keeps an active execution path and stranded-issue recovery does not fire. - An adapter never dispatches alongside a live holder: deferral has no attempt ceiling, so a deferred run keeps rescheduling until the workspace frees. Deadlock safety comes from holder liveness, not a counter — a holder silent past `ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS` (recovery's own "suspicious silence" bar, 1 h) stops counting as a holder, so a zombie run can only delay work, never park it forever, and recovery's silent-run escalation is already reaping it in parallel. If no retry can be scheduled (agent paused, issue reassigned), the deferral releases the issue execution lock so the issue does not strand. - Holder detection honors isolation: when the isolated-workspaces experiment is enabled, holders whose issue settings select `isolated_workspace` / `operator_branch` (or the legacy `isolated` alias) are not counted, because they never touch the shared tree. A NULL or `agent_default` mode counts as a holder — over-serializing is the safe direction. - Non-assignee deferrals survive replay: the deferral stamps `workspaceBusyDeferredWhileAssignee` into the run context (inherited by the scheduled retry), and both the retry promotion gate and the claim-time staleness check exempt a non-assignee `workspace_busy` retry from the reassignment cancellation — for such a retry an assignee mismatch is the expected state, not a reassignment race. An assignee run's retry keeps the full protection: if the issue is reassigned while the retry pends, it still cancels with `issue_reassigned`. - `server/src/__tests__/heartbeat-workspace-busy.test.ts` (new): embedded-Postgres coverage of the full lifecycle plus unit coverage of the delay window. ## Verification - `cd server && pnpm vitest run src/__tests__/heartbeat-workspace-busy.test.ts` — 10 tests: - a run whose issue targets a busy shared workspace is cancelled with `workspace_busy`, its adapter never executes, a `scheduled_retry` run exists with the 60–120 s window, the issue execution lock points at the retry run, the holder run is untouched, and the agent returns to idle; - after the holder finishes, `promoteDueScheduledRetries` + `resumeQueuedRuns` execute the retry run to success; - a non-assignee comment-mention wake defers, does not touch the issue execution lock, and its retry promotes, survives the claim-time staleness check, and executes despite the assignee mismatch; - an assignee retry is still cancelled with `issue_reassigned` when the issue is reassigned while the retry pends; - a holder issue with `executionWorkspaceSettings.mode = "isolated_workspace"` does not cause deferral; - a running run in a different project workspace does not cause deferral; - a holder silent past the staleness threshold does not cause deferral (the run executes); - a retry with ten prior deferrals still defers again — never dispatches — while the holder is live; - delay jitter stays inside the base-to-base-plus-jitter window and clamps out-of-range random sources. - `cd server && pnpm vitest run src/__tests__/heartbeat-` — full heartbeat suite sweep. - `cd server && pnpm run typecheck`. ## Risks - Behavioral shift: shared-workspace runs that used to start immediately now wait for the workspace to free. Against a long-running live holder the wait is unbounded by design — the alternative is dispatching into a held working tree, which is the corruption this PR removes. Every deferral is visible in the run timeline (lifecycle event with the holder run, issue, and attempt number), and the wake is parked, never dropped. - A zombie holder (a `running` row whose process died) delays contending runs by up to the 1 h staleness threshold before it stops counting. Recovery's silent-run escalation targets the same run on the same clock, so this window matches what the system already tolerates for silent active runs. - The holder check and the dispatch are not atomic; two runs that pass the gate in the same instant can still race. The gate closes the common window (a second run waking while the first is mid-execution); the pre-existing sync-conflict handling remains the backstop for the rare simultaneous start. - No schema change, no API change, no new configuration. ## Model Used Claude Fable 5 (`claude-fable-5`, Anthropic) via Claude Code — extended thinking, tool use, full repository access; implementation, tests, and verification runs. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge