Files
PaperClipAI/server
DottaandPaperclip c96c751f39 fix(heartbeat): claim task ownership with queued runs (#13973)
## Thinking Path

> - Paperclip manages agents and their tasks.
> - The scheduler can start several runs for one agent.
> - Each task still needs one execution owner.
> - Task creation and recovery can queue assignments for the same task.
> - The scheduler previously marked both runs running before starting
either executor.
> - This change claims the run and task ownership in one transaction.
> - Tasks with different IDs retain concurrent execution.

## Linked Issues or Issue Description

Refs #13882. Related queue work: #11830 and #13965; those address
different admission and cancellation paths.

**What happened?**

The Grok subscription Product campaign exposed two assignment runs for
one task. Task creation and periodic recovery started runs within 200
ms. Both acquired Daytona sandboxes. One failed before provider work
with `paperclip_runner_attachment_staging_not_authorized` because the
other owned the task. Fixture cleanup then found an active session. The
original failed result is retained in [campaign
36071063537](https://github.com/paperclipai/paperclip/actions/runs/36071063537).

**Expected behavior**

Only the task execution owner may start provider setup. Competing queued
work must wait. Different tasks may use the agent's available slots.

**Steps to reproduce**

1. Set an agent's concurrent run limit to two.
2. Queue task-creation and recovery assignment runs for the same task.
3. Resume the queue while holding adapter execution open.
4. Before this fix, both runs become running. Only one has the task
execution lock.

**Paperclip version or commit**

Reproduced on master `8781f06a8` and in the Grok campaign at `4196a4cd`.

## What Changed

- Use the same company-scoped task ownership gate for assignments,
direct comments, and queued comments.
- Leave competing work queued while another live run owns the task.
Transfer a terminal pointer only after the tracked executor and durable
environment leases/finalization settle, including cleanup on another
controller.
- Commit the running state and task execution owner together.
- Preserve the dedicated review path and the native replacement checkout
guard.
- Add sixteen database regressions for assignment/comment orderings,
unrelated concurrent tasks, and cross-controller cleanup fences.

## Verification

- Live subscription Product E2E planning passed on its first attempt in
both Daytona (351,197 ms) and local execution (268,883 ms), with 6/6
matchers and cleanup passing in each, on combined source
`1b0551bb7c8de3c54f4bee64dbe2c88328b3645e`
([campaign](https://github.com/paperclipai/paperclip/actions/runs/36080870743)).
The local screenshots and persisted state show the same plan revised,
the first approval rejected, the revised approval accepted, and one
completion. The full campaign remains unqualified because of a separate
startup retry and EC2 Spot interruption.
- The new same-task regression failed before the fix: two running rows
instead of one.
- All seven focused concurrency cases passed. Three comment cases failed
before the shared gate was added. Five cross-controller cleanup cases
failed before the durable gate was added. A warm-retention regression
also failed before its successful release receipt was admitted;
missing/failed receipts and a different retention policy stay blocked.
- `node ../node_modules/vitest/vitest.mjs run
src/__tests__/heartbeat-stale-queue-invalidation.test.ts` from `server`:
48 passed. Native cleanup admission adds one passing test; task-drain
release adds two. Total focused checks: 51 passed.
- `git diff --check`: passed.
- Current-head
[CI](https://github.com/paperclipai/paperclip/actions/runs/36078495577)
passes repository typecheck, tests, Rust checks, build, and browser
suites. The unrelated repository-form browser shard passed one bounded
rerun without assertion or code changes; its original navigation failure
remains retained. No local Docker or Rust build was used.
- The failed test's exact two sandboxes were checked: one was already
absent; the other was deleted after its company, environment, and run
labels matched.

## Risks

The claim transaction adds a task row lock. It follows task-before-run
lock order. A competing run stays queued until the current owner
releases the task. The change does not alter attachment authorization,
provider permissions, schemas, or public APIs. Review is 5/5 with all
threads resolved on `23c0d13b6`. All CI checks pass on that exact head.
Live Grok requalification will use the combined runner and scheduler
source.

## Model Used

OpenAI GPT-6 through Codex, with repository tools and code execution.
The exact serving identifier and context-window size are not exposed in
this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

### Live regression verification

The Daytona plan/revise/accept lifecycle in [Grok campaign
36080870743](https://github.com/paperclipai/paperclip/actions/runs/36080870743)
passed on its first attempt at combined source
`1b0551bb7c8de3c54f4bee64dbe2c88328b3645e`, in 351,197 ms, with all six
outcome matchers and cleanup passing. The earlier competing-run failure
remains retained. This is one live planning result; the complete Grok
matrix and repeated qualification are separate gates, and this campaign
also has separately retained infrastructure failures.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-25 10:06:05 -05:00
..