mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
DottaandClaude Opus 5.5
aa94939988
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) use a detached
`paperclip-runnerd`. After a server restart, recovery reattaches to that
Runner on the run's own sandbox lease
> - On staging, a deploy detached the controller and left the Runner
alive. The restart's reattachment then found the Runner process gone
with a provider turn in flight. The run failed with
`runner_remote_process_identity_unavailable` and the task waited for a
person
> - The run log said `startup reuse: environment (fresh) created`, so
the sandbox looked re-created. It was not. The reattachment resumed the
exact retained lease. The receipt reported the lease's original creation
> - `requestInterruptedNativeTurnRetry` refused every unsettled turn
(`provider_turn_unsettled`), and refused every system wake unless the
journal proved that no turn was issued. So this run got no automatic
continuation
> - This pull request continues such a run once in a fresh provider
session when the refused reattachment proved the Runner process gone. It
records that the turn's outcome is unknown and never replays the turn.
It also makes the environment receipt tell the truth
> - The benefit is that a restart that kills a Runner in the middle of a
turn no longer needs a Retry click
## Linked Issues or Issue Description
Refs #20853, #20923, #21023, #20278, #20627, #20828
**What happened?**
A staging deploy landed in the middle of a remote native Codex run on
Daytona. Shutdown detached the controller and logged "durable runner
left alive". The new server reattached through the run's original lease
(same lease row, same provider sandbox id, same directory). Its process
check then found the Runner gone: `process_verification` /
`process_not_running`, `runnerLifecycle: ready`, `runnerSettled: true`,
last diagnostic "WebSocket peer closed the connection", empty stderr.
The provider checkpoint still had an active turn. The run failed with
`runner_remote_process_identity_unavailable` /
`native_continuation_requires_reconciliation`, and the task went to
`blocked`.
**Root cause**
- `server/src/services/heartbeat/restart-interrupted-turn.ts`
(`requestInterruptedNativeTurnRetry`): an unsettled last turn always
returned `provider_turn_unsettled`, also when the refusal proved the
Runner process gone. A system wake (here `transient_failure_retry`)
continued only with `providerTurnIssued: false`.
- The sandbox was not re-created. The environment and lease listings
show one lease for the run, never replaced, and the final workspace
snapshot used the same provider sandbox. `environmentStartupReuse` read
`sandboxLeaseAcquisition: created` from the lease's first acquisition,
so the reattachment's receipt said `fresh / created`. The 46 s acquire
was the provider resume of that exact lease. The evidence available to
GET requests cannot say what stopped the Runner process (the resume
result is not kept). This pull request does not change that path.
- #21023 named this run as not covered. #20853 resumes a Runner only at
a settled, idle `ready` cut. Here the provider checkpoint has an active
turn, so #20853 stays fail-closed, as it should.
**Expected behavior**
The run continues once by itself in a fresh provider session over the
saved work. The turn whose outcome is unknown is recorded and is not
replayed.
## What Changed
- `restart-interrupted-turn.ts`: read the last
`native.remote_runner_recovery_failed` refusal once. When it proves the
Runner process gone, an unsettled last turn no longer blocks the
automatic Retry. Before it requests the Retry, the server records
`native.interrupted_turn_outcome_unknown` once (turn id, last turn
event, `replayed: false`, closed `runnerExit` fields). The Retry is the
existing one: saved-work recovery and stop proof first, then a fresh
provider session (`forceFreshSession: true`), once per chain.
- The same file: a system wake whose run is an automatic retry of owed
work (`retryOfRunId` set) also continues under the run's responsible
user once the refusal proved the Runner gone. Other system wakes (for
example a monitor) keep their policy.
- `packages/adapter-utils/src/startup-reuse.ts` and
`native-session-executor.ts`: a `reattach_remote_runner` recovery
reports its `environment` receipt as `reused / verified_reuse`. That
path only resumes the run's exact retained lease and never acquires one.
- Docs: `doc/execution-semantics.md` and `doc/run-log-events.md`.
Previous-release story: no persisted shape changes. The rule reads the
closed `remoteRunnerRecovery` fields that the previous release already
writes; the new test uses the staging refusal payload as written. A run
that a previous release already failed this way continues on the next
completion or restart hook call only; runs already settled stay as they
are and keep their Retry. The new event is additive.
## Verification
- Red on `origin/master` (51d77648bc), with only the new tests: `cd
server && npx vitest run
src/services/runner-lost-mid-turn-continuation.test.ts` fails 4 of 6.
The three continuation cases return `{ requested: false, reason:
"provider_turn_unsettled" }`, and the monitor case returns that reason
instead of `not_user_request`.
- Green with the fix: 6 of 6. The cases cover a user's assignment
through the restart claim, the staging shape (system
`transient_failure_retry` with `retryOfRunId`), and a same-run retry.
Each one checks one recorded unknown-outcome event, an idempotent second
call, saved-work recovery first, one queued successor with
`forceFreshSession: true`, and one Retry intent. Negative cases: a
refusal that did not prove the process gone, no refusal, and a monitor
wake. Each keeps the visible Retry and records nothing.
- `cd packages/adapter-utils && npx vitest run
src/startup-reuse.test.ts`: the new receipt case fails before (`fresh /
created`) and passes after.
- Focused suites pass: `workspace-recovery-retry-copyback.test.ts`,
`governed-wait-runner-loss.test.ts`,
`heartbeat-native-remote-failure-retention.test.ts`,
`native-runner-restart-recovery.integration.test.ts` (302 passed, 14
skipped), and `native-session-executor.test.ts -t "startup
reuse|restart|reattach"` (149 passed).
- `pnpm --filter @paperclipai/server typecheck` and `pnpm --filter
@paperclipai/adapter-utils typecheck` pass.
- `pnpm verify` exits 2 because `doc/execution-semantics.md` is a global
CI input. CI supplies that coverage.
- Not run: a live Daytona restart. The change is in the server's
recovery decision; the tests use the fixture's sandbox confinement.
## Risks
- Medium. A run whose Runner died in the middle of a turn now continues
by itself. The turn's outcome is unknown, so tool work from that turn
can be partly done. The successor is a fresh session over the saved
workspace; it does not replay the turn. Saved-work recovery stops and
keeps the sandbox before the successor starts, so the old provider
cannot still run.
- Without positive proof that the Runner process is gone, an in-flight
turn keeps the visible Retry, as before.
- An automatic retry of owed work now continues after a lost Runner.
Monitor and other system wakes without `retryOfRunId` do not. The bound
is still one automatic Retry per chain.
- The environment receipt changes only for the reattachment path. No
execution decision reads it.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test runs, read-only staging API reads).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>