mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
DottaandClaude Opus 5.5
757a20d2f1
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote run starts `paperclip-runnerd` in a sandbox. The controller
reads and controls it through the environment plugin
(`environmentExecute`).
> - #20940 made a stalled supervision probe "unknown" while the Runner
stays connected. Other callers still turn the same plugin stall into a
terminal failure.
> - During restart reattachment, the saved-state read, the
identity-marker read and the process check share a 30-second budget. One
plugin RPC that stalls for 40 seconds uses all of it, so a short stall
fails a reattachment to a healthy Runner.
> - At launch, a stalled launch call or identity read fails the run with
`runner_process_termination_unconfirmed` or
`runner_remote_process_identity_unavailable`, although the runner may be
healthy.
> - This pull request treats a plugin RPC timeout as "unknown" and
retries the read-only call within a bounded stall budget. A stalled
launch call is not repeated: the controller reads the launch nonce's
identity marker instead.
> - The benefit is that a short plugin stall during recovery or launch
no longer fails a healthy run. Every positive answer from the sandbox
keeps its meaning, and the bound still fails closed.
## Linked Issues or Issue Description
Refs #20940, #20909
**What happened?**
A plugin `environmentExecute` RPC stall of more than 40 seconds makes a
remote recovery read fail. Plugin RPCs time out at the requested timeout
plus a 30-second overhead, so a 10-second probe can take 40 seconds. The
reattachment read budget is 30 seconds. One stalled read therefore
failed reattachment with `probe_transport_failed` or `probe_timeout`. At
launch, a stalled launch call rejected the run with exit unconfirmed. A
stalled identity read (20-second wait, 2-second reads that can take 32
seconds) failed the run with
`runner_remote_process_identity_unavailable`.
**Expected behavior**
A plugin RPC timeout is a stall of the plugin, not an answer from the
sandbox. Read-only recovery reads retry within a bound. A launch call
that stalls is not repeated. The launch nonce's marker decides whether
the one runner was launched. At the bound, the run fails closed and
Retry works.
**Steps to reproduce**
1. Start a remote (Daytona) run.
2. Restart the controller so that the run reattaches, or start a new
run.
3. While the controller reads the sandbox, stop the plugin worker for 45
seconds.
4. The reattachment or launch fails.
**Paperclip version or commit**
master at 48d23d1530.
## What Changed
- `remote-runner-recovery-probe.ts`: adds `isPluginRpcTimeout` (an
error, or its cause chain, with `PLUGIN_RPC_ERROR_CODES.TIMEOUT`). It
also adds a per-recovery `RemoteRunnerPluginStall` with a 3-minute
budget (`REMOTE_RUNNER_PLUGIN_STALL_BUDGET_MS`). The caller's recovery
deadline caps it. After the first plugin RPC timeout in a recovery, its
read-only probes use the stall deadline instead of the 30-second
deadline. Signals are never retried. Other transport failures keep the
30-second budget.
- `verifyRemoteRunnerReattachment` and `verifyRemoteRunnerOrSettledExit`
share one stall budget across the saved-state read, the identity-marker
read, the process check and the post-exit settled-state read
(`readRemoteRunnerState`).
- `waitForRemoteRunnerProcessIdentity`: a plugin RPC timeout extends the
20-second identity wait to the stall budget.
- Launch: a launch call that fails with a plugin RPC timeout is not
repeated. The controller goes on to the identity-marker wait for the
same nonce. A found marker adopts the one launched runner. A missing
marker uses the existing nonce cleanup, and the result stays unconfirmed
when the cleanup fails.
- `doc/execution-semantics.md` describes these rules.
## Verification
- Red on master: 7 new tests in
`server/src/services/native-runtime/native-session-executor.test.ts`
failed. Each one stalls a plugin RPC for 40 seconds with a
`JsonRpcCallError` `TIMEOUT`:
- reattachment through a stall in the saved-state, identity-marker and
process-check reads (the reattachment failed with the stall error);
- the bound test (it settled before the stall budget);
- the post-exit settled-state read (it failed with the stall error);
- launch-call and identity-read stalls (they failed with the stall error
or `runner_remote_process_identity_unavailable`).
- Green: all 7 pass, and a control test proves that a non-stall
transport error keeps the 30-second bound.
`native-session-executor.test.ts` and
`remote-runner-recovery-probe.test.ts` pass (1400 tests). `tsc --noEmit`
passes for `server`.
- The tests also assert that one launch call happens, no cleanup or
signal runs on the success path, and the bound fails closed without exit
proof.
## Risks
- During a real plugin stall, recovery and launch can now wait up to 3
minutes instead of about 30 seconds before they fail closed. The longer
wait applies only after a plugin RPC timeout. A sandbox that answers, or
a non-timeout transport error, keeps the old timing.
- No wait is added before `turn.start` on the normal path.
- State saved by an older version is not changed.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, code search). Extended reasoning was
on.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>