Files
PaperClipAI/doc
DottaandPaperclip 6364fa7327 fix(heartbeat): wait for retry predecessor cleanup (#18989)
## Thinking Path

> - Paperclip manages agent work and preserves its task ownership.
> - Heartbeat admission controls when a queued run can start.
> - A failed run can still own workspace saving and environment cleanup.
> - Automatic recovery can transfer the task execution lock to its
queued retry before that cleanup finishes.
> - The old queue check then sees the retry itself and misses the
predecessor's remaining work.
> - This pull request checks the exact retry predecessor before claiming
the queued run.
> - Existing cleanup notifications can then resume that same run through
normal admission.

## Linked Issues or Issue Description

Refs #18898 and #18691. The first repair covers canonical connected-task
retries with a native source. The second covers live remote workspace
writers. This change covers ordinary automatic continuation when the
predecessor failed before native runtime selection. The search found no
open repair for that path.

**What happened?**

A preparation failure saved a terminal legacy run row while its remote
workspace save and environment cleanup were still active. Generic
continuation created one queued successor and moved the task execution
lock to it. The queue checked that current lock but did not check the
successor's predecessor. The new run started and failed at the existing
unfinished-writer guard.

**Expected behavior**

Keep that same successor queued until its exact predecessor finishes
execution and cleanup. Check company, agent and task identity before
using retry provenance as a wait condition. Preserve Stop, reassignment,
failed cleanup and independent recovery gates.

**Steps to reproduce**

1. Save a failed legacy preparation run with an active environment
lease.
2. Queue its ordinary automatic continuation and bind the task execution
lock to the successor.
3. Claim the queued run before the predecessor finishes cleanup.
4. The old code starts the successor. The fixed code retains it in the
queue.
5. Commit successful cleanup through the existing cleanup owner. The
ordinary dispatcher starts that same run once.

**Paperclip version or commit**

The causal regression fails on master
`86814c6550f52e99ede150a883bdc773cbb540e3`.

**Deployment mode**

Source checkout on macOS arm64 with disposable PostgreSQL, the assembled
heartbeat service and the production work coordinator. The adapter
boundary is controlled. No hosted provider replay or deployment is
claimed.

## What Changed

- Check the current task execution owner and the retry's exact
predecessor under the existing task claim lock.
- Use the saved native task binding, ordinary task binding and
historical context fallback in that order. Ignore unrelated retry
provenance.
- Reuse the existing live-executor and pending-cleanup checks. Reuse
committed cleanup notifications and the existing dispatcher.
- Add real PostgreSQL claim controls and assembled cleanup-event
acceptance. Keep the original source failure, saved files and current
task ownership intact.
- Document the queued predecessor-cleanup rule.

## Verification

- RED: the unchanged queue claimed the successor and wrote a new
controller lease while its legacy predecessor retained an active remote
environment lease. The regression expected the successor to remain
queued.
- GREEN: the same regression passes after the change. The complete queue
suite passes all 71 tests.
- The assembled heartbeat suite passes all six tests. The added cases
cover one continuation after committed cleanup, saved-file reuse, final
task lock release, board Stop, reassignment and failed cleanup.
- Scope controls cover a deleted source after scan, another company,
another task, another agent, canonical native task binding and the
existing current-owner hold. A failed cleanup and a still-live local
executor retain the wait.
- Node 24.19.0 and pnpm 9.15.4. Module boundaries and `git diff --check`
pass.
- Initial fixture attempts failed on database constraints and observer
timing. The final fixtures use valid immutable bindings, the board Stop
attribution and a completed real dispatcher call after reading the exact
committed lease state. Production ownership guards were retained
throughout.
- Full low-memory repository typecheck passed at `5cddb49322`. The sync
to master `04a24f4e06` adds only the independent retained-allocation
test fixture from #18984; production code is unchanged.
- Full `pnpm build` passes on current head
`c7c2bcf08ffd05be5ce714683ae1dcfe6f5b49f1`.
- The post-sync run executes all 80 behavior assertions successfully
across the queue, assembled cleanup and updated upstream retention
suites. Its queue `afterAll` then exceeds the existing 30-second
database shutdown hook. That failed suite result is retained. The
controlled post-build queue rerun passes all 71 tests with normal
teardown. The earlier shutdown cause remains undetermined. Its owned
queue-test PostgreSQL process has exited. The separate local default
`pnpm test:run` was intentionally stopped after full unfiltered CI
passed on the exact head. Four local chat partitions completed before
that stop; this is not a local full-suite pass. The verified owned
process group and its PostgreSQL fixture have exited. Logs and retained
fixture data remain available. The timeout has not been increased.
- Full unfiltered CI passes on current head
`c7c2bcf08ffd05be5ce714683ae1dcfe6f5b49f1`: all 12 server shards, all
six internal chat partitions, all nine serialized server shards, the
workspace groups, native Runner lanes, typecheck, build and browser E2E.
Both required [ci /
verify](https://github.com/paperclipai/paperclip/actions/runs/41866478617/job/126711606613)
and [ci /
e2e](https://github.com/paperclipai/paperclip/actions/runs/41866478617/job/126711263146)
pass. The file selector is skipped, so this is the full CI test matrix.
The head has 55 successful checks and three expected skipped checks. The
ready-for-review contributor trust and security scan also pass.
- Current-head Greptile is 5/5, its check is successful, and no inline
review threads remain. The ordinary review check also passes. No merge,
deployment or original task retry was performed.

## Risks

- A predecessor with unfinished cleanup keeps its successor queued. Its
existing cleanup owner remains responsible for settlement.
- A task claim reads at most one additional predecessor row. A matching
source also uses the existing ownership and cleanup queries. The change
adds no timer, schema, credential, provider or model policy.
- A committed cleanup signal only requests normal admission. Current
ownership and independent execution gates still apply.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, code editing, tool use and
PostgreSQL verification. The session does not expose a more specific
serving model ID or context-window limit.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass — focused suites pass; the
separate local full-suite attempt is explicitly incomplete above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 11:16:54 -06:00
..
…
…
…
…
…