Files
PaperClipAI/server/src/__tests__
DottaandClaude Opus 5.5 9668e73b4b fix(native): continue a turn automatically when reattachment loses its runner (#19399)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs in managed sandboxes recover through reattachment: after
a server restart, and for a same-run retry when a turn ends without a
semantic result
> - Reattachment fails closed when the original runner or sandbox is
gone (`native_remote_recovery_lease_mismatch`,
`runner_remote_process_identity_unavailable`)
> - Saved-work recovery then saves the files, but the task stays blocked
with "Saved work recovered. Send a message to continue." The recovered
hold offers no Retry
> - A restart, deploy or maintenance window must not need a user message
to continue work
> - This pull request requests the task Retry automatically, once, when
it is safe, and offers Retry on the recovered hold when it is not
> - The benefit is that interrupted work continues after infrastructure
events, and a user who must act gets a Retry that works on first use

## Linked Issues or Issue Description

**What happened?**
A native task in a managed sandbox was interrupted. In one case a server
restart claimed a queued user-comment turn for reattachment. In another
case a turn completed without a semantic result, and the same-run retry
claimed the run for reattachment with no restart. In both cases the
original runner or sandbox lease was gone. The run failed with
`native_remote_recovery_lease_mismatch` or
`runner_remote_process_identity_unavailable`. Recovery saved the work.
The task stayed `blocked` with "Saved work recovered. Send a message to
continue." The execution blocker had `canRetry: false`. A manual `POST
/api/issues/:id/retry` started a run that succeeded.

**Expected behavior**
After an infrastructure interruption, the task continues automatically
when the interrupted turn's outcome is known. Otherwise the task offers
a Retry that works on first use.

**Steps to reproduce**
1. Start a native task in a managed sandbox from a user comment.
2. Lose the sandbox lease or the remote runner process (for example,
restart the server during a maintenance window).
3. Let reattachment fail and saved-work recovery complete.
4. See that the task waits for a new message and offers no Retry on the
execution blocker.

**Root cause**
Reattachment (`reattach_remote_runner`) correctly refuses to replace a
missing runner or allocation. Nothing then continued the user's request.
The automatic continuation rule only allows a new user turn after
reconciliation. `getExecutionBlocker` hid Retry whenever a workspace
restore hold existed, also after the files were verifiably recovered.
The same-run retry path also re-enters the restart-recovery owner, so
its event said "after server restart" with no restart.

## What Changed

- New `server/src/services/heartbeat/restart-interrupted-turn.ts`: after
a reattachment claim executes, if the run failed because its runner or
allocation was lost, request the same task Retry that `POST
/issues/:id/retry` creates. It runs once per run (idempotency key
`restart-interrupted-turn:<runId>`), under the authority of the user who
sent the message, assigned the task, or clicked Retry. It requires that
the run recorded no provider turn or that its last provider turn is
settled. System wakes, in-flight turns, and a Retry that this hook
already requested do not continue automatically.
- `heartbeat/recovery.ts`: call the hook after each reattachment claim.
Label same-run retry reattachment as "Reattaching to the existing remote
runner to resume this run", and add `trigger: "same_run_retry"` to the
event payload.
- `execution-blocker.ts`: a failed run whose saved work is verifiably
recovered now offers Retry (`canRetry: true`) with "Saved work
recovered. Retry or send a message to continue." An intentional Stop
keeps "send a message" only.
- `task-retry.ts`: optional `initiatedBy: "restart_recovery"` records
the activity as actor `system`/`restart-recovery` with
`onBehalfOfUserId`.
- `doc/execution-semantics.md`: document the rule.

## Verification

- New regressions in `server/src/services/workspace-recovery.test.ts`
use the real recovery, Retry and dispatch services on embedded Postgres:
- `continues a user turn once ... after $title and recovery saved the
work` (3 cases: restart lease loss before the turn, same-run retry with
the runner gone after a settled turn, same-run retry with the lease gone
after a settled turn). Each asserts one queued successor bound to the
Retry and the user's authority after recovery, idempotent hook calls,
system activity attribution, and no second automatic attempt for the
automatic Retry's own failure.
- `does not continue automatically when $title, and offers a Retry that
works on first use` (in-flight turn, system wake, unrelated error). Each
asserts no successor, `canRetry: true` on the blocker and the
interruption presentation, and one successor after one Retry click.
- Red before the fix: all 6 tests fail (no successor, blocker `canRetry:
false`). Green after the fix.
- `npx vitest run src/services/workspace-recovery.test.ts
src/services/task-retry.test.ts
src/services/task-interruption-presentation.test.ts
src/services/execution-blocker-read.test.ts
src/services/heartbeat/recovery.test.ts
src/services/explicit-native-continuation.test.ts
src/services/quarantined-task-retry.test.ts
src/__tests__/workspace-restore-repair-route.test.ts
src/services/native-runtime/native-failed-workspace-recovery.test.ts`:
1098 passed.
- Not verified on staging. After the next deploy, check a task
interrupted by a restart: it should continue without a message, or show
Retry.

## Risks

- The automatic Retry uses the original user's authority. Admission
rechecks that user's current permission, stop proof, saved-work
recovery, approvals, pauses, budget and ownership. A refused request
leaves the visible Retry.
- A settled turn that ended without a result is continued with a fresh
Retry turn from saved work. A turn whose outcome is unknown (in flight)
is not.
- The hook runs after the claim's execution returns. If the server stops
between the failure and the request, the task shows Retry instead.
- Not in scope: same-run disposition recovery still cannot start a new
runner in the existing sandbox after the original remote runner exits.
This change continues the task with one fresh Retry turn instead. If
that Retry turn also ends without a result and loses its runner, the
task shows Retry.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #19355

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 22:29:46 -06:00
..
…

Server Tests

Server tests that need a real PostgreSQL process must use ./helpers/embedded-postgres.ts instead of constructing embedded-postgres directly.

The shared helper creates a throwaway data directory and a reserved-safe loopback port for each test database. This protects the live Paperclip control-plane Postgres from server vitest runs; see PAP-2033 for the incident that introduced this guard.