mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
DottaandClaude Opus 5.5
48d23d1530
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote run starts `paperclip-runnerd` in a sandbox. The controller
monitors the process with a health probe that goes through the
environment plugin (`environmentExecute`).
> - A stall of the Daytona plugin worker makes that probe RPC time out
after 40 seconds. The monitor treated the timeout as "process
supervision failed".
> - The transport then failed closed with
`runner_process_termination_unconfirmed`. The run ended
`notification_transport_failed`, `recoverable: false`, and the close
path suspended a healthy Runner. The task rested blocked with Retry.
> - Nothing was wrong with the sandbox, the Runner or the provider. The
Runner held its own authenticated connection to the controller during
the full stall.
> - This pull request makes a failed or timed-out probe "unknown". While
the Runner stays connected on its own channel, the monitor keeps the
same runner and turn and probes again, for a bounded time.
> - The benefit is that a short plugin stall no longer kills a healthy
run. The fail-closed cases stay the same.
## Linked Issues or Issue Description
Refs #20909 (chaos cell CR3, which reproduces this failure)
**What happened?**
A 45-second stall of the Daytona plugin worker killed a healthy run.
Chaos cell CR3 reproduced it 2 of 2 times (OpenCode and Claude,
Daytona). A staging run had the same failure with no deploy: a plugin
`environmentExecute` RPC stalled for more than 40 seconds. The run
failed with "Notification transport failed closed:
runner_process_termination_unconfirmed: runnerd process supervision
failed: RPC call \"environmentExecute\" timed out after 40000ms". The
close path then sent `runner.suspend`, and the healthy Runner suspended
and exited. The task rested blocked with Retry.
**Expected behavior**
A probe that times out or fails in transport is not proof that the
process stopped. While the Runner stays connected on its own channel,
the run continues. If supervision cannot come back within a bound, the
run rests with a working Retry. A probe that positively reports the
process gone still starts the existing recovery.
**Steps to reproduce**
1. Start a remote (Daytona) run with a T1 workload.
2. Send SIGSTOP to the server's Daytona plugin worker for 45 seconds,
then SIGCONT.
3. The run fails `notification_transport_failed` about 40 seconds into
the stall.
**Paperclip version or commit**
master at 7cd5558e41.
## What Changed
- `createRemoteRunnerProcessLauncher`
(`server/src/services/native-runtime/native-session-executor.ts`): when
the health probe throws (RPC timeout or transport error) and the Runner
holds an authenticated connection, the monitor waits one poll interval
and probes again. It does not reject the process completion. The unknown
period has a bound: the remote runner reconnect grace (7 minutes). When
the Runner is not connected, the monitor keeps the old behavior: it
rejects after the 30-second probe budget, without exit proof.
- Probe results that positively answer (exit 45 "process gone", or
identity and generation failures) keep their exact current handling.
- `RunnerProcessLaunchSpec`
(`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`)
has a new optional `isRunnerConnected` callback. `spawnRunner` passes it
to the process owner and keeps it on restart.
- The runnerd transport and the external runnerd driver supply
`isRunnerConnected` from the control plane's count of authenticated
Runner connections.
- `doc/execution-semantics.md` describes the unknown-probe rule and its
bound.
## Verification
- Red, on master: `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "plugin
RPC timeouts until"` (in `server/`). The completion rejected after the
first stalled probe with `{ stage: "process_verification", reason:
"probe_transport_failed" }`. The transport turns that into
`runner_process_termination_unconfirmed`, which is the staging error.
- Red, on master: `npx vitest run
src/live/runnerd-codex-transport.test.ts -t "live evidence of the
runner"` (in `packages/paperclip-runner/`). The launch spec had no
`isRunnerConnected` (`expected undefined to be true`).
- Green: both tests pass. New tests also prove:
- three 40-second plugin RPC timeouts with the Runner connected leave
the same runner supervised, and a later exit-45 probe still confirms
exit;
- with the Runner disconnected, the same timeout rejects without exit
proof (`probe_transport_failed`, `isExitConfirmed() === false`);
- with the Runner connected and probes that never answer, supervision
rejects without exit proof after the 7-minute bound.
- `server/src/services/native-runtime/native-session-executor.test.ts`,
`remote-runner-recovery-probe.test.ts`,
`packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts`,
`src/control-plane/durable-prp-control-plane.test.ts` and
`src/drivers/external` pass locally.
- `tsc --noEmit` passes for `server` and `packages/paperclip-runner`.
- Chaos: CR3 results are in a follow-up comment on this PR.
## Risks
- While the Runner stays connected and the plugin cannot answer, the
controller waits up to 7 minutes before it fails closed. Before this
change it failed after about 40 seconds. During that time the Runner's
own connection is the liveness evidence, and the controller never
reports an exit.
- When the Runner is not connected, the timing is the same as before.
Sandbox loss still fails closed after the 30-second probe budget.
- The new spec field is optional. A process owner that does not read it
keeps the old behavior. State saved by an older version is not changed.
- No wait is added before `turn.start`.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, code search). Extended reasoning was
on.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>