Files
PaperClipAI/server
DottaandClaude Opus 5.5 a665ed6a1b fix(recovery): wait out a sandbox provider outage with no Retry (#22139)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task that runs in a sandbox environment first gets a lease from
the environment provider (for example Daytona)
> - When the provider refuses the lease with a transient error
(`environment_provider_unavailable`), Paperclip retries twice, after 30
s and 60 s
> - After those two retries the task rests on "Retry when ready". A
provider outage longer than about 90 s therefore needs a person to click
Retry
> - Recovery must just work, and the target for runs that need Retry is
0%. The fix must not multiply automatic runs into a crash loop
> - This pull request adds a provider-outage wait after the fast
retries. The task stays system-owned, one waiter per environment checks
the provider on a capped backoff, and the other waiters continue by
themselves once the provider gives out a lease again. A person gets
Retry only after a 30-minute ceiling
> - The benefit is that a provider outage of up to 30 minutes delays
work instead of stranding it, without extra load on the provider

## Linked Issues or Issue Description

Refs #22087 (it classifies Daytona connection errors as transient, so
they reach this path).

**What happened?**
In a restart-chaos wave on 2026-11-17, the Daytona API host names did
not resolve from 19:40:28Z to 19:52:17Z (182 `ENOTFOUND` errors for the
API and proxy hosts). Tasks that started in that window failed with
`environment_provider_unavailable`. The two bounded retries (30 s, then
60 s) ran inside the outage, so these tasks rested on "Retry when ready"
and needed a person to continue them.

**Expected behavior**
Paperclip continues the task by itself when the provider is reachable
again. The task shows that it waits for the provider. It shows no Retry.
Only a long outage hands the task to a person, and then Retry works.

**Steps to reproduce**
1. Make lease acquisition fail with a transient provider error (HTTP 502
or a connection errno) for longer than about 90 s.
2. Assign a task to an agent that uses that environment.
3. After the second automatic retry fails, the task is blocked with a
Retry button, although the provider becomes available again later.

## What Changed

- `server/src/services/environment-provider-wait.ts` (new): the
`environment_provider_wait` retry lane. The delays are 1 min, 2 min,
then 5 min. The lane stops at 30 minutes after the wait started, with at
most 12 attempts. It also builds the task text "Waiting for
"<environment name>" to come back. Paperclip continues this task
automatically when it does."
- `scheduleBoundedRetryForRun` (`heartbeat/retries.ts`): when a run
fails with `environment_provider_unavailable` after the fast retries are
spent, it schedules the next attempt in the wait lane. The retry context
records the environment and the time the wait started. The time is kept
across attempts. The lane is uncharged
(`execution-recovery-attempt.ts`), so it does not spend the failure
budget. Other lanes drop the wait context.
- The environment orchestrator puts the environment id on
`EnvironmentProviderUnavailableError`, and the setup-failure path saves
it in `resultJson.environmentProvider`.
- Run dispatch (`run-dispatch/adapters/environment-provider-wait.ts`,
new): a due wait stays parked while an earlier waiter for the same
environment is still in line or still acquiring its lease, and no lease
was acquired in that environment since this waiter last failed. Parked
waiters check again every 30 s. This uses database reads only, and it
adds no work to the run start path. When a lease is acquired, the parked
waiters are promoted in arrival order. The automatic-run brake then
meters their starts as before.
- The probe must be able to start. A queued earlier waiter blocks later
waiters for only 60 s after promotion, and a running one only while it
is still acquiring its lease (at most 10 minutes). If the probe's agent
is busy, the next runnable waiter becomes the probe. A waiter past the
30-minute ceiling is never parked, so it makes its last attempt and the
exhaustion path runs.
- The notify-and-update mechanics of parking are in one helper,
`parkScheduledRetryInTx`
(`run-dispatch/adapters/scheduled-retry-park.ts`). The workspace-busy
wait and the provider wait use it and only decide whether to park and
until when.
- UI: `IssueScheduledRetryCard` shows nothing for
`environment_provider_wait` (as for `workspace_busy`). `IssueProperties`
shows "Sandbox: Waiting for the provider" instead of the scheduled-retry
picker. `IssueMonitorBanner` shows a steady "Waiting for the sandbox
provider" with no countdown and no Check now.
- Task presentation: a task in the wait lane is "working", system-owned,
with no Retry, and shows the wait text. After the ceiling the existing
path applies: the task is blocked with the system-owned notice and a
Retry that works. The notice and the provider-failure text now say that
Paperclip retried for 30 minutes.
- The per-task transient cap, the automatic-run brake and the bounded
fast retries are unchanged.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/heartbeat-process-recovery.test.ts -t "transient sandbox
provider refusal"` (embedded Postgres):
- New: "resumes ten tasks by itself when a five-minute provider outage
ends, one probe at a time and within the automatic-run brake". Ten tasks
in one company share one environment. The provider refuses leases for 5
minutes of simulated time (the test shifts all stored timestamps). All
ten tasks complete with no Retry. No task is blocked and no task offers
Retry while it waits. At most one waiter per tick is refused by the
provider. While the brake is open, at most one automatic start happens
per tick. All ten finish within 5 minutes after the outage ends. Red on
master: tasks become `blocked` after the fast retries. Green with this
change.
- New: "lets one waiter per environment probe the provider and releases
the others once a lease is acquired". The second waiter stays parked
while the first one is queued or running without a lease. It is promoted
after a lease is acquired. Red without the park step, green with it.
- Updated: "waits for the provider without a Retry, and offers a working
Retry only after the 30-minute ceiling". After the fast retries the task
is working with the wait text and no Retry. The backoff grows from 1 to
2 minutes. After the ceiling the task is blocked, Retry is offered, and
Retry runs the task.
- Updated: "rests a provider deadline that never clears on a working
Retry, never on a board recovery action". It now reaches the ceiling
first.
- New: "lets the next waiter probe when the first waiter cannot start
because its agent is busy" (red before, green after).
- New: "hands a parked task to a person with a working Retry at the
30-minute ceiling even while the leader stays queued" (red before, green
after).
- `src/services/environment-provider-wait.test.ts`: backoff, ceiling and
text.
- UI: `IssueScheduledRetryCard.test.tsx` ("offers no Retry while the
task waits for the sandbox provider to come back") and
`IssueMonitorBanner.test.tsx` ("shows a steady provider wait with no
countdown"). `pnpm check:token-gates` is clean.
- Also run locally: `heartbeat/retries.test.ts`,
`heartbeat/run-completion.test.ts` (all 163 tests pass; the file-level
`afterAll` teardown timed out on this loaded machine),
`task-interruption-query.test.ts`, `stranded-notice.test.ts`,
`execution-recovery-attempt.test.ts`,
`environment-provider-unavailable-error.test.ts`,
`task-interruption-presentation.test.ts`, the automatic-run brake tests,
and `src/modules/run-dispatch`.
- Server `tsc --noEmit` passes.

## Risks

- **Automatic-run volume:** while a provider is down, each environment
has one waiter that calls the provider on the backoff (1, 2, then 5
minutes). The other waiters only read the database every 30 s. Before
this change the volume was zero after the second retry, because the task
waited for a person. The new volume per environment is at most about 8
provider calls in 30 minutes. It does not grow with the number of tasks.
The automatic-run brake still opens after 5 consecutive provider
refusals, and it limits the probe to one start per minute.
- After the outage, waiting tasks start again as the brake allows (a
probe, then a release ramp of 2, 4, 8 and more starts per minute). They
do not all start at once.
- A run that was in the old fast lane when this deploys continues in the
new lane when its second retry fails. A run that is already blocked
keeps its Retry.
- No migration. The new fields are optional JSON in `resultJson` and
`contextSnapshot`.

## Model Used

- Claude Opus 5.5 (provider Anthropic, model ID `claude-opus-5-5`), with
tool use and code execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 01:24:13 -06:00
..