Files
PaperClipAI/server/src/services
Devin FoleyandPaperclip 8464a8c9ae fix(native): settle a bootstrap retry that lost its task lock (#19307)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run that fails before any provider work gets a bounded
same-run bootstrap retry. Restart recovery decides whether that retry
can run.
> - The retry needs the task's issue lock. A user comment or another
wake can move that lock to a new run before the retry is due.
> - Restart recovery then returned `issue_execution_lock_changed`
without a write. The same check ran on every recovery pass and never
ended. The coordinator stayed `retryable_failure`.
> - The orphaned-lease sweep waits for a terminal coordinator before it
preserves the workspace. So the sandbox lease stayed active, and every
later run for the task was cancelled as `workspace_busy`.
> - This pull request gives a lock-lost bootstrap retry the same bounded
admission that a superseded remote retry already has.
> - The benefit is that the retry settles, the board gets a
reconciliation action, and the sandbox lease is released instead of
blocking the task.

## Linked Issues or Issue Description

No public issue exists. Bug description:

**What happened?**

On a hosted instance, a native agent run failed at startup because the
provider rejected `session.open` with a rate limit. Paperclip scheduled
a same-run bootstrap retry. Seconds later, a new run for the task took
its issue lock, most likely woken by a user comment. From then on, every
restart recovery pass for the old run returned `blocked` /
`issue_execution_lock_changed` and wrote nothing. The run recorded about
1,000 recovery events over 12 hours. Its coordinator stayed
`retryable_failure` with `recoveryMode: bootstrap_retry`. The scoped
orphaned-lease sweep found a pending workspace export for the lease. It
could not preserve the workspace, because
`preserveNativeFailedWorkspace` requires `terminal_failure`. So the
lease stayed `active`, and 16 later runs for the task were cancelled as
`workspace_busy`. Three more runs on the same instance were stuck in the
same state. A board Stop on the failed run moved the coordinator to
`terminal_failure`, and the sweep then released the lease within about a
minute.

**Expected behavior**

A bootstrap retry that cannot get its task lock back settles within a
bounded number of recovery passes. The coordinator reaches
`terminal_failure`, the board gets a reconciliation action, and the
existing orphaned-lease sweep can preserve the workspace and release the
sandbox lease.

**Steps to reproduce**

1. Fail a native remote run before provider work, so its coordinator is
`retryable_failure` with `recoveryMode: bootstrap_retry`.
2. Move the task's `executionRunId` to a newer run for the same task, or
clear it.
3. Run `claimNativeRestartRecoveries` for the failed run.
4. Before this change, it returns `blocked` with no `retryAt`, and the
coordinator stays `retryable_failure` on every pass. The new tests in
`native-remote-retry-admission.test.ts` reproduce this.

**Paperclip version or commit**

Observed on a hosted deployment. Reproduced in tests on `master` at
`10a7d039c8e`.

Related: #19190 (a `session.open` model rejection no longer schedules a
doomed bootstrap retry), #19131 (preserve superseded observed
workspaces) and #19281 (release the leases of a cancelled legacy run).
None of them changes this admission path.

## What Changed

- **Restart recovery
(`server/src/services/native-runtime/native-restart-recovery.ts`)**: a
new condition, `lockLostBootstrapRetry`, matches a failed run whose
coordinator is `retryable_failure` with `recoveryMode: bootstrap_retry`
and whose issue lock is not on the run.
- A newer run for the task: the existing superseded check now covers
this case too. It defers with `native_retry_source_superseded` as
permanent, so admission is exhausted at once.
- No newer run (the lock was released): recovery defers with
`issue_execution_lock_changed` through the existing `deferRemoteRetry`.
That path persists a deadline, backs off and exhausts after the system
recovery budget.
- When admission is exhausted, the existing hold path writes
`terminal_failure` / `native_restart_recovery_blocked`, keeps the
provider evidence and creates the board reconciliation action. This PR
adds no new terminal path.
- A bootstrap retry that still holds its own lock is unchanged. The
exact-checkpoint remote retry path is unchanged.
- **Docs (`doc/execution-semantics.md`)**: the bootstrap retry section
now states the lock-lost rule.

## Verification

- New test `settles a %s bootstrap retry that lost its task lock instead
of re-checking forever` in
`server/src/services/native-runtime/native-remote-retry-admission.test.ts`,
for the `superseded` and `released` cases. The fixture is a genuine
pre-provider failure: a `session.open` rejection with no session
checkpoint, no runner checkpoint row and no provider evidence.
- On `master` without the fix, both cases fail: recovery returns
`blocked` without a deadline, and the coordinator stays
`retryable_failure`.
- With the fix, `superseded` settles on the first pass, and `released`
settles after `SYSTEM_RECOVERY_MAX_RETRIES` deferred passes. Each
deferred pass persists its `nextAttemptAt`. Both end in
`terminal_failure` with one board recovery action, and later passes open
no transaction.
- `npx vitest run
src/services/native-runtime/native-remote-retry-admission.test.ts` (from
`server/`): 92 passed.
- Related suites, each run on its own with exit code 0:
`native-ingress-bootstrap-retry`, `native-startup-retry`,
`native-scheduled-resume-lock`, `native-restart-recovery-isolation`,
`native-runner-restart-recovery`, `native-runner-restart-preparation`,
`native-runner-restart-diskless`, `native-warm-transition-restart`,
`native-controller-recovery`, `native-failed-workspace-recovery`,
`native-session-resumption`, `native-startup-cleanup-recovery`,
`heartbeat-observed-workspace-preservation`, `heartbeat/recovery`,
`runtime-selection`, `remote-workspace-migration` and
`native-session-executor` (1,149 tests). Two other suites each had one
local failure that this change does not cause:
- `heartbeat-process-recovery`: "redacts opaque environment-bound
credentials from Sentry diagnostics" fails the same way on this machine
without the change.
- `chat-channels.integration`: "settles a shared-endpoint
accepted-response backlog without retrying its own locks" failed once
under full-suite load. It passes alone, with and without the change.
- `pnpm run typecheck` in `server/`: passed.

## Risks

- Low to medium. The change only affects failed bootstrap retries whose
lock is not on the run. Before this change, such a retry could never
run, so no working path is removed.
- A released lock that comes back to the same run during the backoff
still resumes normally, because the condition reads the current lock on
each pass.
- When the retry settles, the task gets a board reconciliation action.
Before this change, the task was silently stuck behind `workspace_busy`.
- No migration and no API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, code search, test runs), extended thinking, and a read-only
investigation subagent for the root cause.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-05 11:45:54 -08:00
..
…
…