Enforce durable external-wait liveness (#9373)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The heartbeat/recovery subsystem decides whether an agent run has a
durable continuation path after the process stops.
> - External waits need stricter semantics than local background
watchers: a killed local process is not durable, while a first-class
blocker/monitor/scheduled wake is.
> - Without that distinction, recovery can repeatedly treat
adapter-failed continuations as live work and obscure the real reason a
task stopped.
> - This pull request adds explicit durable external-wait liveness
handling and documents the expected execution semantics.
> - It also improves operator-visible recovery evidence so invalid
external-wait paths explain why they were rejected.
> - The benefit is clearer recovery behavior, fewer duplicate
continuation recoveries, and a safer contract for monitor-backed
external waits.

## Linked Issues or Issue Description

- Refs #5978
- Related PRs: #4988, #7495, #8502

## What Changed

- Added durable external-wait liveness classification so
local/background watchers are not accepted as durable live paths after
the owning process exits.
- Preserved first-class blocker/monitor/scheduled wake paths as valid
external-wait continuations.
- Added backend regression coverage for killed watcher failure,
monitor-backed durable wait resumption, normal completion, blocker
behavior, and no duplicate recovery.
- Added adapter utility coverage for terminal cleanup behavior used by
local process adapters.
- Surfaced invalid external-wait recovery evidence in the recovery
action card and run ledger.
- Updated execution semantics documentation and the V1 implementation
contract.

## Verification

- `pnpm check:token-gates` passed.
- `pnpm -r typecheck` passed.
- `node scripts/run-vitest-stable.mjs --mode general --group
general-server` equivalent lane passed in CI-clean env: 238 files, 2164
tests passed, 1 skipped.
- `node scripts/run-vitest-stable.mjs --mode general --group
general-workspaces-a` passed in fully Paperclip-env-clean env: UI 305
files / 2430 tests; CLI 43 files / 230 tests.
- `node scripts/run-vitest-stable.mjs --mode general --group
general-workspaces-b` passed in fully Paperclip-env-clean env:
shared/db/adapters/plugin packages all green.
- `node scripts/run-vitest-stable.mjs --mode serialized` passed in fully
Paperclip-env-clean env: 107 serialized server suites green, including
84/84 heartbeat-process-recovery tests.
- `pnpm build` passed in fully Paperclip-env-clean env.

Notes: running `pnpm test:run` directly inside the Paperclip heartbeat
environment exposed local harness env contamination in existing tests
(`PAPERCLIP_CONFIG`, `PAPERCLIP_DB_BACKUP_DIR`, and
`PAPERCLIP_WORKTREE_START_POINT`). Re-running the same lanes with
inherited `PAPERCLIP_*` and port env removed produced the CI-equivalent
green results above.

## Risks

- Medium behavioral risk: this changes recovery classification for
stopped local external-wait processes, so adapters relying on unmanaged
background watchers must use blockers, monitors, scheduled wakes, or
explicit durable handoff instead.
- Low UI risk: recovery-card copy changes are covered by component tests
and Storybook screenshot QA.
- No database migration is included.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex, GPT-5-based coding agent, tool-enabled terminal/code
execution. Exact context-window metadata was not exposed in the runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

doc/execution-semantics.md

@@ -258,6 +258,29 @@ The valid action-path primitives are:
 - a first-class blocker chain whose unresolved leaf issues are themselves healthy
 - an open explicit recovery action that names the owner and action needed to restore liveness
 
+### Durable external waits and heartbeat finalization
+
+An external wait counts as a live or waiting path only when the next move survives the current heartbeat and is represented in Paperclip's durable control-plane state. Valid external-wait shapes are:
+
+- a one-shot issue monitor or other persisted scheduled wake that names the responsible assignee, next check time, and bounded timeout/attempt policy
+- a first-class blocker or `blocked` disposition that names the external owner and concrete action required to unblock the issue
+- a delegated child issue with a responsible owner and its own healthy action path, plus a blocker edge when the source issue must wait for that child; `parentId` alone is not a dependency
+
+An unmanaged local process is not a durable action path. Shell jobs started with `&`, `nohup`, local polling loops, detached PTY sessions, adapter child processes, or similar background watchers do not keep an issue live unless Paperclip persists them as a run or pairs a managed runtime service with a monitor, scheduled wake, blocker, or delegated issue that owns the next check. A PID, session id, log file, comment, or promise to check later is evidence only. The process may be killed when the adapter invocation or heartbeat exits and cannot be assumed observable or recoverable by another worker.
+
+Before a heartbeat finalizes, its issue disposition must therefore be evaluated from durable Paperclip state, not from processes still visible only to that heartbeat. An agent-owned issue may remain `in_progress` after the heartbeat only when another valid action-path primitive already exists. If the only claimed continuation is a local/background watcher, finalization treats the issue as having no live path even when the process has not yet been observed exiting.
+
+If useful deliverable work can continue without the external result, the agent should continue that work or delegate it rather than parking the issue. Use `blocked` only for a real dependency that prevents productive progress. Use a monitor when the assignee owns a bounded future check, and use delegated child work when another owner can make progress independently.
+
+Recovery from an invalid external wait is bounded and idempotent:
+
+1. Record bounded evidence that the completed heartbeat left no durable action path, including the terminal run and any reported local watcher metadata without treating that metadata as liveness.
+2. Queue at most one normal-model continuation for the same source state and recovery fingerprint so the assignee can inspect the external result, replace the watcher with a durable wait, continue productive work, or choose a valid disposition.
+3. If that continuation also exits without creating a durable path, do not queue another equivalent continuation. Move the issue to `blocked` only when a real external dependency can be named; otherwise open or update an explicit recovery action with a named owner and concrete repair/escalation action.
+4. New durable source activity may produce a new recovery fingerprint, but unchanged killed/local-watcher evidence must not create an infinite wake/recovery loop.
+
+This rule is intentionally conservative: local watcher evidence can help the recovery owner decide what happened, but only persisted control-plane state can prove that the work will move again.
+
 ### Comment and document activity wake sources
 
 Issue-thread comments and document-scoped comments have different wake semantics.
@@ -383,7 +406,7 @@ A healthy active-work state means at least one of these is true:
 - there is an active one-shot monitor that will wake the assignee for a future check
 - there is an open explicit recovery action for the lost execution path
 
-An agent-owned `in_progress` issue is stalled when it has no active run, no queued continuation, and no explicit recovery surface. A still-running but silent process is not automatically stalled; it is handled by the active-run watchdog contract.
+An agent-owned `in_progress` issue is stalled when it has no active run, no queued continuation, no persisted monitor, and no explicit recovery surface. An unmanaged local/background watcher does not satisfy any of those conditions. A Paperclip-tracked run that is still running but silent is not automatically stalled; it is handled by the active-run watchdog contract.
 
 ### `in_review`
 
@@ -478,6 +501,8 @@ Recovery rule:
 
 This is an active-work continuity recovery.
 
+The same bounded rule applies when the previous heartbeat reported waiting on a local/background watcher and that watcher was killed, disappeared, or was never represented by a durable Paperclip primitive. Paperclip queues at most one continuation for the same recovery fingerprint. If the continuation also leaves only local watcher evidence, Paperclip must surface a real blocker or explicit recovery action instead of repeating continuation recovery. A new monitor, scheduled wake, healthy delegated blocker issue, or other durable source mutation resolves that recovery fingerprint normally.
+
 #### Deliberate wait is not a lost run
 
 A continuation that the staleness gate cancelled with `issue_continuation_waiting_on_review` is a *deliberate park*, not a disappeared execution path. The latest run reported that the issue is waiting for review/approval (for example, an umbrella issue whose work was just decomposed into sub-tasks). Treating that park as a stranded run would retry it, then escalate it to `blocked` with a recovery action and an operator-facing failure notice — even though nothing failed and there is nothing for a human to do.