Commit Graph
2 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 7e3caa7bac fix(tree-control): resume work a pause gate refused after release (#19421)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The board can pause an issue tree. A pause stops work. Resume or
release must continue that work once.
> - PR #19394 made release re-dispatch a run that a pause Stop
cancelled.
> - That rule covers only a run that an operator Stop ended and that the
hold snapshot named.
> - A scheduled retry that comes due during a pause is not stopped. The
dispatch gate cancels it with `issue_paused`. The release did not see it
as owed work, so the task stalled with no run.
> - This pull request makes every gate that refuses work for a pause
record the covering hold. Release then re-dispatches that work exactly
once.
> - The benefit is that a paused task with a parked retry or a queued
run continues after release, with no manual wake.

## Linked Issues or Issue Description

Refs #19394

**What happened?**
A child task had a `workspace_busy` scheduled retry. The board paused
the parent tree. The retry came due during the pause. The
scheduled-retry gate cancelled it with `issue_paused` ("Scheduled retry
suppressed because the issue is held by an active subtree pause hold").
The board released the hold. Four minutes later the child was still
`in_progress` with no run, no execution blocker, and no new activity.

**Expected behavior**
Release (or `mode: "resume"`) re-dispatches the refused work exactly
once. No reconciliation and no user message are necessary.
Stranded-issue recovery does not add a second run.

**Steps to reproduce**
1. Give an assigned child task a parked `workspace_busy` scheduled
retry.
2. Pause the parent tree.
3. Let the retry come due while the pause is active. The gate cancels it
with `issue_paused`.
4. Release the pause. No run starts for the child.

**Paperclip version or commit**
master at 4d49a31286 (includes #19394).

**Agent adapter(s) involved**
Not adapter-specific (core bug).

## What Changed

- `issue-tree-control.ts`: add `ISSUE_PAUSED_ERROR_CODE` and
`pauseSuppressionResult()`. A gate records `treeHoldSuppression: {
holdId, rootIssueId }` on a run that it refused for a pause.
- `findOwedPauseResume` now treats the latest run as owed when it is an
`issue_paused` cancellation under a pause hold. It uses the recorded
hold. For an unmarked run (a scheduled retry cancelled before this
change), it uses a snapshot pause that was active at cancellation. The
operator Stop rule does not change.
- Scheduled-retry gate (`run-dispatch/adapters/postgres.ts`,
`cancelSuppressedRetryInTx`): locks the pause row `FOR SHARE` and
records the covering hold. A concurrent release either sees the refusal,
or the retry sees no pause and runs normally.
- Queued-run claim gate (`heartbeat/queue.ts`): cancels with
`issue_paused` and the hold marker, after the agent start lock is
released. Before, the cancel re-entered that lock from inside it, and
the agent's queue stalled until the 30-second stale-lock timeout. After
the refusal is saved, the gate re-checks the hold and resumes the task
itself if release already finished (`resumeIfPauseAlreadyReleased`).
- Release (`issue-tree-resume.ts`, route): revisits the snapshot members
and every task with a run refused for that hold, also when `wakeAgents`
is set. This covers tasks created under the tree after the snapshot and
tasks moved out before release.
- `packages/db`: partial index
`heartbeat_runs_tree_hold_suppression_idx` (migration 0436, `CREATE
INDEX CONCURRENTLY`, registered as a top-level migration) for that
lookup.
- Docs: `doc/execution-semantics.md` (gate-refused work, race handling,
non-assignee and pre-fix scope) and `doc/DATABASE.md` (index).

## Verification

- Red before the fix, green after: `pnpm --filter @paperclipai/server
exec vitest run src/__tests__/issue-tree-pause-resume-run.test.ts` (21
tests).
- "resumes a parked retry the pause suppressed exactly once after
release / resume / legacy": before, `expected [] to have a length of 1`
(no resumed run). After, one `issue_tree_resumed` run with
`interruptedRunId` = the suppressed retry, no execution blocker, and the
task finished. `reconcileStrandedAssignedIssues` added no run during the
hold or after release.
- "resumes a queued run the claim gate refused during the hold exactly
once": before, the test timed out at 15 s (start-lock re-entry). After,
the run is cancelled with `issue_paused` and the hold marker, and
release re-dispatches it once.
- "resumes a refused retry on a task moved out of the paused tree before
release" and "... on a task outside the snapshot when release also wakes
members": review findings, one resume each.
- "leaves a due retry parked when the pause is released while the gate
refuses it": the test waits until the gate blocks on an uncommitted
release. The retry is not cancelled and runs once. Without the `FOR
SHARE` lock the gate does not wait and the test fails.
- "resumes work a gate refused after the release that should have
resumed it had finished": the late-refusal re-check resumes once, and a
repeated check does not dispatch again.
- "does not resume a paused-code cancellation that no pause hold
covered": negative case.
- Related suites pass locally: `src/__tests__/issue-tree*`,
`src/modules/run-dispatch`, `src/services/heartbeat/queue.test.ts`,
`heartbeat-dependency-scheduling`, `heartbeat-workspace-busy`, and
`packages/db` `client.test.ts`.
- I did not run a live staging check here. Staging APIs are not
reachable from this environment.

## Risks

- Migration 0436 adds a partial index on `heartbeat_runs`. It is built
concurrently and covers only cancelled runs with the marker, so it is
small.
- The scheduled-retry gate now takes a `FOR SHARE` row lock on the pause
hold before it cancels. Release updates that row, so the two serialize
briefly. Release does not lock issues or runs, so the lock order cannot
deadlock with the gate's issue-then-run locks.
- The queued-run claim gate now records `issue_paused` in place of the
generic `cancelled`. Stranded recovery already classifies `issue_paused`
as non-retryable. The pause resume owns that work, so recovery does not
add a second run.
- Scope limits (documented): a non-assignee run refused by a pause is
not replayed, the same as non-assignee wakes that admission drops during
a hold. Among pre-fix rows, only scheduled retries carry `issue_paused`,
so only those fall back to the snapshot. A pre-fix row whose hold is
already released does not self-recover; the staging task that showed
this bug needs one manual wake, or a new pause and release.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 03:03:15 -06:00
DottaandClaude Opus 5.5 5c185db77c fix(tree-control): resume work a subtree pause interrupted (#19394)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Task tree control lets the board pause a task tree and then resume
it.
> - A pause stops the runs that are active in the tree. The release must
then continue the work that was pending.
> - A child task whose run was active at pause time did not start again
after release. The child became blocked and asked for a user message.
> - A pause is an operator hold, not a failure. It must not strand work,
and it must not require reconciliation for a run that never started.
> - This pull request re-dispatches interrupted work one time on
release, and it settles a legacy Stop that never entered the adapter.
> - The benefit is that pause and resume keep a task tree moving without
manual recovery, and real safety gates stay in force.

## Linked Issues or Issue Description

**What happened?**

A parent task delegated two children. One child's run was active (in
host workspace preparation, before the adapter started) when the board
paused the parent's tree. The pause cancelled that run. After the board
released the hold, no new run started for that child. The child became
`blocked` with the execution blocker cause
`legacy_execution_requires_reconciliation`. Later the next action became
"Send a message to continue." A sibling child that had no active run at
pause time resumed normally.

**Expected behavior**

A run that a subtree pause cancels is an operator hold. On release, the
pending work starts again one time, automatically. It does not go
through execution reconciliation when the run never entered its adapter.
It does not wait for a user message.

**Steps to reproduce**

1. Create a parent task and a child task that is assigned to an agent.
2. Wake the child. While its run is in host preparation (or queued),
call `POST /api/issues/:parentId/tree-holds` with `{ "mode": "pause" }`.
3. Call `POST /api/issues/:parentId/tree-holds/:holdId/release` without
`wakeAgents`.
4. Before this change, no new run starts, and `getExecutionBlocker`
reports `legacy_execution_requires_reconciliation` for the child.

**Root cause**

- (a) The pause records each interrupted run in the hold member snapshot
(`activeRunId`). Nothing used that record on release.
- (b) Release woke tasks only when `metadata.wakeAgents === true`.
Without it, release relied on stranded-issue recovery. That recovery
stands down when the latest run was cancelled by an operator
(`isOperatorCancelledRun`). So the interrupted child had no owner.
- (c) The pause Stop of a legacy run in preparation is written directly
as `cancelled`. Legacy terminalization classifies a cancelled run with
no bootstrap evidence as an unknown outcome. It records
`legacy_execution_requires_reconciliation`, although the gated dispatch
never entered the adapter.
- Native runs: the verified native Stop does not create the
reconciliation hold, but gap (b) applies to native runs too. The release
change is runtime-independent.

## What Changed

- `issueTreeControlService.findOwedPauseResume` decides if a task is
owed one resume. The task's latest run must be an operator Stop that a
pause hold member snapshot names, recorded while that hold was active.
The task must still be executable by the same agent, and no pause may
still cover it. The obligation belongs to the run, so overlapping pauses
keep it until the last covering release.
- New `services/issue-tree-resume.ts` re-dispatches owed work with wake
reason `issue_tree_resumed` (`holdId` and `interruptedRunId` in the
context). A compare-and-set claim (`treeHoldResume`) on the interrupted
run makes the resume happen at most once. A failed wake returns the
claim.
- Pause release and `mode: "resume"` call it for every member of the
released hold, also without `wakeAgents`. When `wakeAgents` is true, the
existing loop wakes every executable member.
- A task that still has an execution blocker is not replayed. The
release writes `issue.tree_resume_wake_skipped`, the blocker stays
visible, and the obligation stays unclaimed.
- `releaseHold` updates only an `active` hold. A repeated release gets
`409`.
- `settleCancelledLegacyPreparation`: when the executor that owned a
cancelled legacy run proves that dispatch never entered the adapter, it
records bootstrap evidence and resolves that run's
`legacy-execution:<runId>` hold as `false_positive`. It also corrects an
automatic no-replay disposition and restores a `blocked` status only
when that disposition made the change. Then it re-dispatches owed work,
for a release that happened while the Stop was unwinding. It requires a
requested Stop with no existing recovery receipt, and no process,
session, native finalization, unreleased lease, or execution-start
event.
- `settleUnrecoverableExecutions` waits while a legacy executor still
holds its controller lease, and records a `legacyCancellationBlock`
receipt when it blocks a task for a cancelled legacy run.
- Docs: `doc/execution-semantics.md` and `doc/composer-stop.md` describe
the resume rule.

## Verification

- New regression
`server/src/__tests__/issue-tree-pause-resume-run.test.ts` uses embedded
Postgres, the real heartbeat, and the real tree-control routes:
- A run paused during preparation: nothing runs during the hold, there
is no blocker, and release creates exactly one `issue_tree_resumed` run
that succeeds. A second release returns `409` and creates no run.
- Release before the stopped preparation finishes unwinding (with the
periodic disposition run in between): exactly one resumed run after the
executor settles.
- An automatic no-replay disposition that ran first: the blocker clears,
the status returns to `in_progress`, and release resumes once.
- Overlapping child and parent pauses: no resume while the parent pause
remains, then exactly one resume after it is released.
- A queued run cancelled by the pause: release creates exactly one
resumed run.
  - A task reassigned to a user during the hold: no resume.
- A mid-turn legacy Stop with unknown outcome: no replay, the
reconciliation blocker remains, and `issue.tree_resume_wake_skipped` is
logged.
- Red before each fix: the first 4 cases failed on master; the 3 review
cases failed on the first PR commit. All 7 pass now.
- Also passed locally: `issue-tree-control-routes`,
`issue-tree-control-service`, `heartbeat-workspace-busy`,
`heartbeat-undispatched-execution-wait`, `execution-control-work`,
`legacy-workspace-restore-recovery`, `legacy-continuation-authority`,
`task-retry`, `legacy-execution-recovery`, `agent-conversations`,
`heartbeat-retry-scheduling`, `satisfied-connection-intents`.
- Not verified here: a live staging pause/resume. Re-check on staging
after deploy.

## Risks

- Behavior change: release now starts the work that the pause
interrupted, also without `wakeAgents`. Before, that work stayed idle.
Other idle tasks still wake only with `wakeAgents`.
- The settlement trusts the executor's adapter-entry flag. The flag is
set only inside the atomic dispatch gate, which refuses a cancelled run.
The settlement also requires no process, session, lease, or execution
event.
- A native run whose Stop has not finalized at release time is not a
target. Stranded recovery and explicit wake still apply.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 22:08:01 -06:00