Files
PaperClipAI/server/src/services/native-runtime
DottaandClaude Opus 5.5 84908d176b fix(recovery): name the failed check in native_remote_recovery_lease_mismatch (#19839)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Remote native runs execute in a sandbox, and recovery can reattach a
run to its original sandbox runner
> - Reattachment checks the saved lease, workspace, and runner identity
in five places. Each failed check throws the same bare
`native_remote_recovery_lease_mismatch`
> - On staging, 34 OpenCode runs on 31 tasks failed with this bare error
in one evening. The run records could not show which check failed, so
the cause could not be confirmed from the run alone
> - This pull request makes every producer name the failed step and
field with closed values
> - The benefit is that an operator and the next investigation can see
which reattachment check failed, without new logging or identifiers in
the error

## Linked Issues or Issue Description

Refs #19399, Refs #19733, Refs #19755, Refs #19779

**What happened?**
Remote same-run retries failed with
`native_remote_recovery_lease_mismatch` and the task was blocked for
reconciliation. The error is thrown by environment acquire, environment
resume, workspace preparation, workspace sync, and remote runner
verification. All of them used the same bare string. The run's `error`
field and its run events did not show which check failed.

**Steps to reproduce**
1. Start reattachment for a remote native run whose saved lease,
workspace reference, or runner identity no longer matches.
2. Read the run's `error`.
3. It shows only `native_remote_recovery_lease_mismatch`.

**Expected behavior**
The error names the step and the field that differed, with no
identifiers, and existing consumers of the code keep working.

## What Changed

- New
`server/src/services/native-runtime/remote-recovery-lease-mismatch.ts`:
the code constant, a closed `stage` and `field` vocabulary, and
`remoteRecoveryLeaseMismatch(stage, field)`. It builds
`native_remote_recovery_lease_mismatch: stage=<stage> field=<field>`.
- `environment-run-orchestrator.ts` (stages `environment_acquire` and
`environment_resume`), `heartbeat/workspace-preparation.ts`
(`workspace_preparation`), `native-workspace-sync.ts`
(`workspace_sync`), and `verifyRemoteRunnerReattachment` in
`native-session-executor.ts` (`runner_verification`) now throw the
detailed form. Each check keeps the same condition and order. Only the
message changes.
- `restart-interrupted-turn.ts` uses the shared constant. It already
matches the code exactly or as the `code: ` prefix, so automatic
continuation after a reattachment loss is unchanged. No other consumer
matches the code.
- `doc/execution-semantics.md` documents the detail format.

## Previous-release state

This change adds no persisted state and changes no saved identity. Runs
that a previous release failed keep the bare
`native_remote_recovery_lease_mismatch` in `error`. Runs failed by this
release store the detailed form. The only consumer (`isReattachmentLoss`
in `restart-interrupted-turn.ts`) accepts both forms, and
`workspace-recovery-retry-copyback.test.ts` covers both.

## Verification

- Exact-detail assertions in the producer tests:
- `server/src/__tests__/environment-run-orchestrator.test.ts`: every
scope, status, expiry, provider, lease-ID, cwd, plugin-task, and resume
case asserts its stage and field. Against the previous producer, 21 of
these cases fail (the message was the bare code). With this change, all
57 tests pass.
- `native-session-executor.test.ts`: a replaced sandbox reports
`runner_verification`/`provider_lease_id`. A missing durable runner
identity reports `identity_run_id` before any sandbox read.
- `native-workspace-sync.test.ts`,
`heartbeat/workspace-preparation.test.ts`,
`native-controller-recovery.integration.test.ts` assert their exact
detail.
- End to end on embedded PostgreSQL:
`heartbeat-native-remote-failure-retention.test.ts` records
`native_remote_recovery_lease_mismatch: stage=environment_resume
field=lease` on the run when the provider no longer resumes the lease,
and it still requests the user's Retry once.
`workspace-recovery-retry-copyback.test.ts` adds a detailed-form row
that still continues once.
- The affected server test files pass locally. `pnpm --filter
@paperclipai/server typecheck` passes.

## Risks

- Low risk. Only the error text changes. Conditions, ordering, and
fail-closed behavior are unchanged.
- Logs and run errors show a longer message. The detail has closed
values only, so it adds no identifiers or secrets.
- A first version of this PR also changed remote retry selection. Review
showed that a plain resume in a new sandbox hits the remote workspace's
unsaved-work hold, and that a `destroy` disposition does not prove the
old sandbox is gone. That change is removed. It needs a separate design.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, run in Claude
Code with tool use (shell, file editing, read-only browser access to
staging).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 09:27:14 -06:00
..
…