Files
PaperClipAI/server
Devin Foley 627eef7cbd fix(plugins): retry errored plugins at boot instead of leaving them dead (#12054)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugins extend the server with sandbox providers, tools, and jobs; a
loader activates them at boot
> - When activation fails, the loader marks the plugin `error` and skips
it on every later boot
> - Activation failures are often environmental — missing package
dependencies, a stale build output, a module that moved under a pull —
and the fix lands on disk without any write to the plugin row
> - The plugin therefore stays dead forever, and every feature behind it
(sandbox destroys, cleanup sweeps, probes) silently stops working until
an operator flips the row by hand
> - This pull request makes `loadAll` retry errored plugins once per
boot: flip to `ready`, attempt activation, and re-record the error if
the attempt fails
> - The benefit is that a plugin recovers on the next boot after its
environment is fixed, with no manual database or lifecycle intervention

## Linked Issues or Issue Description

**What happened?**

Several sandbox-provider plugins sat in `error` status for weeks after a
transient activation failure (a module resolution error from an older
checkout state). The boot loader only loads plugins in `ready` status,
so it never retried them. Environments backed by those providers lost
sandbox destroys, cleanup sweeps, and probes with no visible signal
other than the stale `last_error`.

**Expected behavior**

A plugin whose activation failure has been fixed on disk recovers on the
next server boot. A plugin that still fails stays in `error` with a
fresh error message.

**Steps to reproduce**

1. Install a plugin whose worker cannot start (for example, delete one
of its dependencies), then boot the server. The plugin lands in `error`
status.
2. Restore the dependency.
3. Restart the server. Before this change, the plugin stays in `error`
forever. After this change, the boot retries it and the plugin
activates.

## What Changed

- `server/src/services/plugin-loader.ts`: `loadAll` also fetches plugins
in `error` status, flips each to `ready`, and activates it with the
normal batch. The flip runs before activation because the `error` status
only legally transitions to `ready` or `uninstalled`; a retry that
failed while still in `error` could not re-mark itself. A failed flip
logs a warning and never aborts the boot load. The stale comment at the
`markError` site now describes the retry.
- `server/src/__tests__/plugin-loader-error-retry.test.ts`: covers the
flip-then-retry flow, the failed-flip isolation, and the empty case.

## Verification

- `npx vitest run server/src/__tests__/plugin-loader-error-retry.test.ts
server/src/__tests__/bundled-plugins.test.ts
server/src/__tests__/plugin-lifecycle-restart.test.ts
server/src/__tests__/cloud-image-bundled-plugins.test.ts`
- Manual: mark an installed plugin's status to `error`, restart the
server, and observe the loader log line `retrying plugins that failed
activation on a previous boot` followed by a successful activation (or a
fresh `last_error` if the plugin is genuinely broken).

## Risks

- A genuinely broken plugin now costs one bounded activation attempt per
boot (the attempts run in parallel with the ready batch under
`Promise.allSettled`). It cannot crash-loop within a running process,
and it returns to `error` with a fresh message.
- The flip clears `last_error` before the attempt. If the process dies
between the flip and the activation, the row is `ready` with no error
text; the next boot simply loads it as a ready plugin.
- Operators who relied on `error` as a manual "keep this off" latch
should use the `disabled` status, which this change does not touch.

## Model Used

- Claude (Anthropic) — Fable 5, model id `claude-fable-5`, extended
thinking, agentic tool use via Claude Code CLI.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-08-23 16:52:42 -07:00
..