Commit Graph
2 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 0c9962a9bc fix(plugins): replace a retired worker with a fresh process (#19245)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Sandbox providers such as Daytona run as plugin workers that the
host manages in `server/src/services/plugin-worker-manager.ts`
> - The host retires a worker when it cannot prove that a login
pseudo-terminal or a duplex channel route is gone (fail closed)
> - Retirement used `killProcess()`, which marks a deliberate stop, so
the handle stayed `stopped` and no restart followed
> - Every later call then failed with `Cannot call "<method>" — worker
for "<id>" is stopped` until the server process restarted
> - This pull request makes a retirement replace the worker with a fresh
process through the existing backoff timer, with a bound on repeated
retirements
> - The benefit is that one unconfirmed close no longer makes a sandbox
provider unavailable for the whole server

## Linked Issues or Issue Description

Refs #19150

**What happened?**
A Daytona plugin worker was retired after an unacknowledged duplex
close. After that, every run that used the Daytona environment failed
setup with `Cannot call "environmentExecute" — worker for <plugin> is
stopped`. 15 runs failed in about 12 minutes. The plugin stayed
unavailable until the server restarted. #19150 removed the specific
trigger. It keeps fail-closed retirement for close replies that the host
cannot confirm, so the same outage can still occur.

**Expected behavior**
The retired process, its routes, and its tombstones are never reused.
The plugin comes back on a fresh process after a short, bounded backoff.
A deliberate stop (plugin disable, upgrade, shutdown) stays stopped.

**Steps to reproduce**
1. Start a plugin worker and open a duplex channel whose close
acknowledgement is wrong or missing.
2. Close the channel. The host retires the worker.
3. Call any worker method. Before this change, the call fails with `is
stopped` forever.

## What Changed

- Add `retireWorkerProcess()`. It kills the process but does not mark a
deliberate stop. The three fail-closed retirement paths use it instead
of `killProcess()`.
- In the exit handler, a retired exit settles every route as before,
then counts the retirement and schedules a fresh process through the
existing backoff timer (`crashed` → `backoff` → `running`). It emits the
existing `crash` event with `willRestart`.
- A deliberate stop wins. If `stop()` runs before the exit, the exit
ends `stopped`. If `stop()` runs during backoff, it cancels the restart.
- Bound repeated retirements: at most 5 retirements in the 10-minute
crash window. A successful restart does not reset this count (it does
reset the crash count), so a worker that starts cleanly and is retired
again cannot loop forever. After the bound, the worker stays `crashed`.
- `scheduleRestart()` / `computeBackoffMs()` take the attempt number, so
the crash path and the retirement path each use their own counter.
- Add an optional `crashRecovery` start option (`minBackoffMs`,
`maxConsecutiveRetirements`) so tests can use short waits.
- Update the Daytona worker close test: a stalled close now shows
`backoff`, then a fresh `running` process with a new pid.

## Verification

- Red before the fix: the three new tests in
`server/src/__tests__/plugin-worker-manager-duplex.test.ts` (`plugin
worker replacement after a fail-closed retirement`) failed with
`expected 'stopped' to be 'running'` / `'backoff'`.
- Green after the fix. The tests prove:
- the retired route and a sibling route on the same worker both settle
with `{ exitCode: null }`;
- the fresh process has a new pid, a write to a retired route does not
revive it, and a new route on the fresh process echoes data;
  - `stop()` during backoff stays `stopped` with no process;
- with `maxConsecutiveRetirements: 2`, the third retirement leaves the
worker `crashed` and `call()` rejects.
- The real Daytona plugin worker test
(`plugin-worker-manager-daytona-close.test.ts`, `stalled-close`) now
shows the replacement with the actual provider bundle. The closed lease
still rejects on the fresh process.
- `cd server && npx vitest run src/__tests__/plugin
src/__tests__/managed-plugin src/services/plugin` → 48 files, 649 tests
pass. The new tests passed 5 runs in a row.
- `npx tsc --noEmit -p server` shows no errors in the changed files.
- Not run: a live staging run. This PR does not deploy.

## Risks

- A plugin that keeps causing retirements now restarts up to 5 times in
10 minutes (1s, 2s, 4s, 8s, 16s backoff) instead of staying dead. After
that, it stays `crashed`, as a crash loop does now.
- A retirement now emits `plugin.worker.crashed`, so host-service
cleanup runs for the plugin, the same as for a crash. A restart emits
`plugin.worker.restarted`. A retirement also counts in `totalCrashes`,
so idle sleep treats the worker as it treats a crashed worker.
- A call made during backoff fails with `is backoff` rather than `is
stopped`. The outage is bounded to the backoff, not to a server restart.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 08:37:16 -06:00
DottaandPaperclip 9e5039f6c8 fix(daytona): retain duplex close ownership after lease cleanup (#19150)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Remote agents use provider plugins to manage sandbox leases and
duplex channels.
> - The host verifies the exact route and session when a bound channel
closes.
> - Daytona lease teardown removed that binding before a later host
close.
> - The incomplete reply made the host retire the shared worker. A later
command then failed during setup.
> - This pull request retains bounded close receipts and joins the
original cleanup.
> - Later work can use the same worker while invalid or unconfirmed
closes still fail closed.

## Linked Issues or Issue Description

**What happened?**

A normal lease teardown can close a duplex channel before the host
closes its bound route. Daytona then replies with only the route ID. The
host requires the original session ID too. It logs `duplex channel close
not acknowledged; retiring worker`, stops the worker, and rejects a
later `environmentExecute` call as stopped.

**Expected behavior**

A later close must identify the original route and session and join its
existing cleanup. It must not revive the channel or restart the worker.
Unknown ownership and an unconfirmed close must retain the existing
failure behavior.

**Steps to reproduce**

1. Acquire two leases through one Daytona provider worker.
2. Open a duplex channel on the first lease.
3. Release that lease, then close the bound channel through the host.
4. Execute a command on the second lease.

**Paperclip version or commit**

The real host/worker regression fails on
`69dea878c0b920b335594debb0286443ba9f6c35`. This branch includes master
`3a72efebe5b050e7594bb309e71a0d5f3072db00`.

**Deployment mode**

Managed Daytona sandbox execution in Cloud. The regression uses a local
real host and SDK worker. Only the external Daytona SDK and PTY are
simulated.

Related work: Refs #19004, Refs #10429, Refs #18065, Refs #19050. These
cover task Retry, managed worker startup, drain release, and stale
provider sessions.

## What Changed

- Retain at most 4096 retired route/session bindings with their original
close promises.
- Remove live channel authority before cleanup starts. Concurrent and
later closes join the same promise and return the stored binding.
- Reject reuse of a retained route ID. Keep unknown and evicted routes
unable to provide a session acknowledgement.
- Add a real child-process host/SDK-worker/Daytona regression with only
the external Daytona SDK and PTY simulated.
- Cover release, cancellation, stop, destroy, resume, a stalled close,
concurrent closes, failed cleanup, forged IDs and bounded eviction.
- Document close receipt scope and its separate provider-termination
checks.

## Verification

- RED before the repair: the actual host and SDK worker reproduced the
retirement log, SIGKILL, stopped state, and later `environmentExecute`
rejection. The direct provider cases also failed because the session ID
was missing.
- Combined-branch provider suite: 364 passed.
- Combined-branch host manager, duplex, managed readiness and composed
suites: 142 passed. The final test-only manifest-loading change passed
all 6 composed cases again.
- Full repository typecheck (`node scripts/typecheck-low-memory.mjs`),
focused server types and the standalone Daytona build passed. The
provider build uses pinned SDK `0.203.0` and TypeScript `5.9.3`.
- The composed cases check the same worker/PID, one later command,
preserved saved content, revoked channel writes/stops, and continued
rejection of commands on closed leases. A stalled close still retires
the worker and dispatches no later command.
- Reproduce with `pnpm --dir server exec vitest run --config
vitest.config.ts
src/__tests__/plugin-worker-manager-daytona-close.test.ts` from a
checkout with the SDK build dependencies prepared. Run the provider
suite with its own Vitest config.
- An initial local typecheck lacked the excluded providers' SDK links
after an install with scripts disabled. The standard root postinstall
linker restored those local links, and full typechecking then passed.
The fixture loads its actual built manifest. Failed setup output is
retained; no type configuration was relaxed.
- The first two composed attempts stopped during fixture setup. They are
not the behavioral RED above. Their failed outputs are retained.
- The first broad local typecheck stopped at an IPC permission error
during preflight. It passed after rerunning with allowed local IPC and
the standard SDK links.
- [Full repository
CI](https://github.com/paperclipai/paperclip/actions/runs/41945022881)
passed all 48 jobs on this head. Selection fell back to the complete
task plan, so the run was unfiltered. The complete check inventory has
54 successes and two expected Storybook skips; the separate Snyk status
passed. Fresh Greptile review on
`daebbeb5005b9d9bd59d32ea30094279048b9213` is [5/5 with no actionable
comments](https://github.com/paperclipai/paperclip/pull/19150#issuecomment-6646621914).
Independent production and fixture review found no actionable findings.
Hosted acceptance of this repair has not run.

## Risks

- A receipt removed at the 4096-entry bound cannot confirm an old bound
close. The existing host guard remains effective in that case.
- Close acknowledgements identify channels. They do not certify remote
process termination or clear unknown external work. Provider termination
receipts and existing recovery gates remain in force.
- Hosted evidence confirms the unacknowledged-close retirement class.
The historical log does not distinguish a missing session ID from a
timeout or another close error. The regression proves and repairs the
missing-binding path; it also preserves timeout retirement.
- The provider is outside the main pnpm workspace. A new Cloud image
must compile its changed source. The unchanged plugin version alone does
not prove activation.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, source editing and tool
execution. The exact model ID and context window are not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 19:15:50 -06:00