mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 04:21:24 +01:00
497ccba8f1f3a4caf03887f5ee1d155c650bd6bf
374
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
497ccba8f1 |
fix(native): replace a reattached Runner that left its recovery read unanswered (#21272)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native Codex runs execute in a remote Runner that the server controls over the durable PRP channel > - When a run is interrupted, a same-run retry reattaches the live Runner and reads its session (`session.snapshot`) before it continues > - On staging, the reattached Runner authenticated and restored its session, and then did not answer the read > - The recovery kept that Runner, so each later attempt adopted the same Runner and failed the same way until the attempt budget ended the run > - This pull request makes the next attempt replace that Runner with a fresh Runner over its saved journal, and makes a failed Codex history read say what failed > - The benefit is that an interrupted Codex run continues without a manual Retry, and its provider turn is not sent again ## Linked Issues or Issue Description Refs #21161, Refs #21053, Refs #20853 **What happened?** A Codex agent on Daytona ran recovery runs (wake `issue_recovery_action_restored`). In one hour, 11 of about 35 finished runs failed. All failed the same way: - `native_session_recovery_failed: HarnessCapabilityUnavailableError: read is unavailable: Error: runnerd exited while waiting for session.snapshot` (8 runs) - `native_session_recovery_failed: HarnessReconciliationError: codex_history_read_failed: thread/turns/list requires supported paginated history; Error: provider_transport_failed: PRP command session.snapshot timed out (runner connected; pending ahead 0; last runner event seq 338 76s ago)` (3 runs) The run log of each run shows one sequence: 1. Attempt 1 fails with `runnerd did not report the provider turn identity`. The close leaves the Runner alive and unsettled for the same-run reattachment. 2. Attempt 2 adopts that Runner (same PID). The Runner authenticates. The log shows `runnerd restored its durable PRP session and provider thread`. Then the driver's recovery read (`thread/read` or `thread/turns/list`, which both send `session.snapshot`) gets no answer. The Runner still answers WebSocket pings, but it sends no command result and almost no events (`pending ahead 0`, last event 76 s to 391 s earlier). The recovery treats this as transient and detaches. The Runner stays alive. 3. Attempt 3 adopts the same Runner again and fails the same way. In some runs a protocol fault (`thread_binding_mismatch`) closes the transport during the read, so the message is `runnerd exited while waiting for session.snapshot`. 4. The provider attempt budget ends the run. #21161 did not help. Its wait extension applies only while a queued answer is being committed. Here the Runner sent no answer at all. The `paginated history` text was also wrong. `codex-history.ts` called every failed history read unsupported, also a transport timeout. **Expected behavior** A Runner that left a recovery read unanswered is not adopted again. The next attempt starts a fresh Runner from the saved journal, resumes the provider session from the saved checkpoint, and does not send the provider turn again. A failed history read reports the real failure. A Codex build that refuses paginated history falls back to one complete thread read. **Steps to reproduce** 1. Start a Codex run in a remote (Daytona) Runner. 2. Let the first attempt fail at turn start, so that the close keeps the Runner alive. 3. Let the reattached Runner stop answering commands after it authenticates (for example, stop the Runner process after its first answer). 4. See `PRP command session.snapshot timed out (runner connected; pending ahead 0; ...)` on attempt 2 and the same failure on attempt 3, with the same Runner PID. **Paperclip version or commit** Staging Core `57aeda190b` (contains #21161 and #21120). Fixed on `master` after `53608bd1d6`. **Deployment mode** Hosted (staging), remote Daytona sandbox Runner. **Agent adapter(s) involved** Codex (native Runner, `paperclip-runnerd`, `codex_app_server`). ## What Changed - `nativeRecoveryCommandUnanswered` (executor) finds a `native_session_recovery_failed` whose Runner did not answer a recovery command (`PRP command <type> timed out` or `runnerd exited while waiting for <type>`). Identity, authority, integrity and incompatibility refusals do not match. - When such a failure ends a `reattach_remote_runner` attempt on a sandbox target, the failure detail records `adoptedRunnerUnanswered: true`. - `claimNativeRestartRecoveries` copies it to the next claim as `replaceUnansweredRunner: true`, in the same way as `exitedBeforeSessionOpen`. - `verifyRemoteRunnerOrSettledExit` with `replaceUnansweredRunner` keeps all identity checks. When the old Runner is still alive, it returns no adopted process, and the attempt launches a fresh Runner from the saved journal. The existing launch fence stops the old `paperclip-runnerd` on that state folder and its whole lineage, and fails the launch closed when it cannot prove them gone. A Runner that already exited mid-turn keeps the existing fail-closed path, because its provider or tool processes can survive where the fence cannot find them. The task then continues in a fresh session from the saved work. The attempt budget is unchanged. - `codex-history.ts`: a history read that fails without a provider answer is now `codex_history_read_failed: <method> read failed; <error>`. A provider refusal of the method (`-32601` or `-32600`) on the first page falls back to one `thread/read` with `includeTurns: true`. That fallback fails closed when the thread differs, or when turns, items or turn identities are missing. A refusal on a later page still fails closed. - `doc/execution-semantics.md` describes both rules. The launch-fence code comment now names the replacement case. ### Changed existing test `codex-history.test.ts` "does not turn an interrupted page or unsupported method into idle history" used a plain `Error("method not found")` on page 2 and expected the `supported paginated history` text. A plain error is now a failed read, not an unsupported method. The test now uses a real provider refusal (`CodexRpcError` `-32601`) on page 2 and still expects a fail-closed rejection. A new test covers the failed-read message. ## Verification - Red before the fix, green after: - `server/src/services/native-runtime/native-session-executor.test.ts` › "a same-run reattachment whose adopted Runner never answered its recovery read": the replacement and staging-message tests failed on the old code (the Runner was adopted again, the three staging messages were not recognized). Two further tests ("never resumes beside an exited unanswered Runner's lineage") failed on the first revision of this PR, which relaunched beside an exited mid-turn Runner, and pass now. All 15 pass. - `native-runner-restart-recovery.integration.test.ts` › "asks the same-run reattachment to replace a Runner that left its recovery read unanswered" (embedded Postgres): the claim had no `replaceUnansweredRunner` on the old code. Now passes. - `codex-history.test.ts`: 7 new tests failed on the old code. All 16 pass. - Real `paperclip-runnerd` and fake Codex: `runnerd-codex-transport-remote-retention.test.ts` › "continues the same provider thread from a fresh Runner over the journal of a retained Runner that stopped answering, without replaying its turn". It runs the staging sequence: turn started, close keeps the stopped Runner, a second controller adopts it, the Runner stops answering and the controller detaches, the Runner and provider are killed (the launch fence), and a fresh Runner over the same journal returns the same thread. The fake Codex call log shows one `thread/start` and one `turn/start`. - Focused suites: `packages/paperclip-runner` `src/drivers/codex/` plus the remote-retention, adoption and cold-restore transport suites (442 tests). Two suites timed out under parallel load and pass alone. Server: `native-session-executor`, `native-restart-recovery`, `native-runner-restart-recovery.integration`, `runner-lost-mid-turn-continuation` (1714 passed, 21 skipped). - `pnpm typecheck` in `server` (all layers) and `tsc` for `paperclip-runner` source and tests pass. - Not run locally: the container-gated remote Runner suites (`PAPERCLIP_TEST_RUNNER_CONTAINER`) and the full suite. CI covers the full suite. ## Risks - Medium. The replacement relaunches a Runner over a journal that is not settled, but only while the old Runner is alive. Safety comes from three facts: the launch fence stops the old Runner and its lineage with proof or fails the launch; the Runner restores its own durable journal (as local `resume_dead_runner` does); and the real-runnerd test shows that the turn is not sent again. An exited Runner is never relaunched beside its possible survivors. - If the previous close left a `runner.suspend` pending, a fresh Runner can execute it and exit. The attempt then fails as it does today. The usual detach path after an unanswered read sends no suspend. - The replacement uses the existing attempt budget. It does not add attempts. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser reads of staging run events and logs), extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
04b50e6007 |
fix(native-runtime): a timed-out step records its real cause instead of leaving the run unrecoverable (#21262)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs start a Runner and a provider session, then send the first turn > - When a startup step fails, the native session executor redacts the error, classifies it and records the failure > - A timed-out step fails with a `DOMException` (`TimeoutError` or `AbortError`). Its `message` property has only a getter > - The redaction assigned to `error.message`. The assignment threw a `TypeError`, so the classifier did not run and the real cause was lost > - The run then fell into a restart-recovery chain that ended in "requires reconciliation" wording for an operator > - This pull request makes the redaction safe for errors with a read-only message > - The benefit is that a timed-out step records its real cause and settles through the normal failure path ## Linked Issues or Issue Description **What happened?** In a local chaos cell with 8 opencode tasks on Daytona sandboxes, four first runs failed before their first turn. A step before the first turn ran under a five-minute `AbortSignal.timeout`. The step took longer than five minutes. The run log shows `TypeError: Cannot set property message of which has only a getter` at the startup failure handler, then `native_finalization_missing`. The run never recorded the timeout. Recovery then restarted the bootstrap, saw "runnerd exited while waiting for run.prepare", and ended with `runner_state_identity_mismatch: bootstrap_not_proven_incomplete` and `native_continuation_requires_reconciliation`. **Expected behavior** A failed startup step records its own cause. The failure handler classifies it and settles the run. It does not throw a second error that hides the first one. **Steps to reproduce** Make any step before the first turn reject with `AbortSignal.timeout(n).reason`. Before this change, `executePaperclipNativeSession` rejects with the `TypeError` above and the coordinator row keeps no failure detail. The new integration test does this with the embedded database. ## What Changed - Add `redactNativeFailureError` in `server/src/services/native-runtime/native-failure-redaction.ts`. It redacts the message and stack in place when it can. When the message is read-only, it returns a new `Error` with the redacted message, the same `name` and the redacted stack. - Use it in the startup failure handler of `executePaperclipNativeSession` in place of the direct assignment. - Add unit tests for a real `TimeoutError`, a read-only `AbortError` that holds a secret, and an ordinary typed error. - Add an integration test: a step before the first turn rejects with a real `TimeoutError`. The run settles through the classifier. `failureDetail.message` is "The operation was aborted due to timeout". The run error code is not `native_continuation_requires_reconciliation`. ## Verification - Red on master: `native-failure-redaction.test.ts` failed 2 of 3 with `TypeError: Cannot set property message of which has only a getter`. The new integration test in `native-startup-retry.integration.test.ts` failed: the call rejected with that `TypeError` and not with the `TimeoutError`. - Green with this change: - `pnpm --filter @paperclipai/server exec vitest run --config vitest.config.ts src/services/native-runtime/native-failure-redaction.test.ts src/services/native-runtime/native-startup-retry.integration.test.ts` - `src/services/native-runtime/native-session-executor.test.ts`: 1431 passed, 7 skipped. - `pnpm verify` for the affected files. ## Risks - Low risk. The change only alters behavior when the error message cannot be assigned. Before, that case threw a `TypeError` and skipped the failure handler. - The replacement error keeps the `name` but is a plain `Error`, so `instanceof DOMException` is false after redaction. No code in the handler checks for `DOMException`. - After this PR, a capture timeout before the first turn still ends blocked. The run records the real cause, but it shows generic wording and waits for a person to press Retry. The reason: a generic timeout has no typed cause, so the classifier records it as `native_session_interrupted` and treats it as ambiguous state. Follow-up PRs give the capture step a typed cause and per-step bounds, and retry a failure that provably happened before any prompt automatically. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a924d2e2c7 |
Persist portable Runner ownership through controller recovery (#21066)
## Thinking Path > - Paperclip delegates remote Runner execution to environment plugins. > - A plugin can move a session to another task allocation. > - The Core controller can lose an ownership reply or restart before the Runner starts. > - The source and target need a durable record of the same request. > - This change binds restart and attachment to that request and the original unlaunched task lease. ## Linked Issues or Issue Description **What existing behavior does this improve?** Recovery of plugin Runner sessions on replacement compute. **Current behavior** Core cannot replay a session ownership request from saved lease state during startup. **Proposed behavior** Record the operation on both leases and check the matched ownership receipt before attaching. Preserve the original pre-admission lease across controller restart. Managed environment configuration is in [#21065](https://github.com/paperclipai/paperclip/pull/21065). Related warm sandbox cleanup is in [#20065](https://github.com/paperclipai/paperclip/pull/20065). This layer preserves the same-sandbox fallback from [#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace placement follows in [#20743](https://github.com/paperclipai/paperclip/pull/20743). ## What Changed - Add the optional `sessionHandoffVersion` capability and the ownership task operation. - Save matching ownership claims and fence source checkpoint access. Replay a pending request on the original target. - Restore a portable session using the durable claim and the current credentials for the same account. Start a fresh conversation when the provider reports an unavailable checkpoint, or when Core's persisted compatibility decision selects a different session after transfer. The latter requires an unlaunched target; subsequent recovery must match the fresh task's saved identity. Both paths retain the workspace transfer and source fencing. - Reuse the original unlaunched task lease after hard or graceful controller restart. After a move has attached, validate the surviving Runner through the saved claim rather than requiring its logical lease ID to equal the new physical lease. - Keep older retained leases without a saved account identity on the existing resume path. - Apply the existing supported MCP-only compatibility rule to warm recovery and predecessor accounting. Persist the capability discovered at admission and reuse it after restart without repeating discovery. - Keep predecessor-retirement fallback scoped to the same physical sandbox. A refusal after a portable move preserves the session instead of requesting a replacement that the plugin driver cannot enforce. ## Verification - On `1eb527be27`, server typecheck (including Runner build and SDK dependencies) passed. - Independent final verification passed 168 tests across portable source recovery, session resume, controller recovery, and task authority; two existing tests were skipped. The full portable source integration suite passed 64 tests. - The new integration follows confirmed ownership through an incompatible instruction change, persisted fresh admission, target-only prepare/auth/submit without the source checkpoint, and reattachment after controller replacement. It also rejects unpersisted session selection, ambiguous target leases, and incorrect saved identities. - Before this final change, independent verification passed 1,667 tests across executor, session resume, recovery usage, remote warm recovery, runtime selection, and runtime execution; the environment lifecycle suite passed 208 tests. - Existing regressions cover previous-release account metadata, supported and unsupported MCP changes, provider/instruction incompatibility, and admission capability reuse without another discovery call. - On `51d7eec373`, all four S3 multipart throttle tests passed independently. CI exposed an existing timing-sensitive test twice; its 20 ms overlap assumption now uses synchronized retries and an event-loop yield. Production upload code is unchanged. Temporary concurrency caps of one and eight both fail the retained exact-four assertion. - Exact SDK source-pin checks passed for both stack heads. The upper #20743 code patch remains unchanged; its documentation preserves both layers' behavior. ## Risks - The handoff protocol is opt-in. Existing plugin leases without portable ownership metadata keep the previous release's resume path. Live attachment still requires end-to-end qualification. An unverified claim blocks startup. Git workspaces prepared on a portable allocation also need the workspace placement layer. ## Model Used - OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and independent review. ## Checklist - [x] Thinking path and model are included. - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] The issue is described using the enhancement template. - [x] Local checks passed and new behavior has tests. - [x] All Paperclip CI gates are green. - [x] Greptile is 5/5 with no open findings. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
3fb35fcf13 |
fix(native): restart an interrupted bootstrap whose runner connected before the deploy (#21120)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner (`paperclip-runnerd`) in a
Daytona sandbox. A deploy stops the server and a new server recovers
each run.
> - When the server stops during a run's bootstrap, recovery restarts
that bootstrap on the same run (`bootstrap_incomplete`). The claim is
only granted when the controller checkpoint in Postgres proves that no
provider turn was sent.
> - Host preparation then reads the sandbox Runner state. A Runner that
already authenticated is `ready`, not `suspended`. The restart is
allowed only with a stop proof for that Runner.
> - The stop proof was taken in two places only: from a restored local
controller root, or for a Runner that never connected. A new server
after a deploy can have neither, so the run failed with
`runner_harness_state_mismatch: resumed_not_suspended` and needed a
manual Retry.
> - This pull request lets host preparation take the same Postgres
checkpoint proof that granted the claim, stop that exact Runner, and
restart the bootstrap.
> - The benefit is that a deploy during a bootstrap no longer fails the
run. No provider turn can be replayed, because the proof shows that none
was sent.
## Linked Issues or Issue Description
**What happened?**
On staging, three Codex runs on Daytona failed with
`native_session_interrupted` and `runner_ingress_unavailable: runnerd
process failed: runner_harness_state_mismatch: resumed_not_suspended`
(12:48Z twice and 13:19Z on 2026-11-12). In each run the Runner
authenticated 1 to 4 seconds before a deploy stopped the server
("controller authority detached for restart; durable runner left
alive"). The new server chose "Restarting an incomplete native runner
bootstrap on the same heartbeat run". Its host preparation found the
sandbox Runner `ready`, had no stop proof, and failed the run. The run
log has no "Stopped the interrupted runner" line, so the earlier stop
step did not run on the new server.
**Expected behavior**
The run restarts its bootstrap on the same sandbox and continues without
a manual Retry. A provider turn that may have run is never replayed.
**Steps to reproduce**
1. Start a Codex run on a Daytona sandbox.
2. Deploy the server after the Runner authenticates and before the first
turn starts.
3. Let the new server recover the run with a `bootstrap_incomplete`
claim on a server that has no local controller root for the run.
4. Host preparation fails with `runner_harness_state_mismatch:
resumed_not_suspended`.
**Paperclip version or commit**
Staging at
|
||
|
|
d5bd38468d |
fix(runner): stop a leftover Runner on the same state folder before a relaunch (#21079)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner (`paperclip-runnerd`) in a
Daytona sandbox. The Runner keeps its saved state in one folder per
session and listens on one port in the sandbox.
> - During an automatic upgrade, the old server can launch a Runner a
few seconds before it stops. That Runner stays alive in the sandbox and
keeps the state folder and the port.
> - The new server then restarts the run on the same sandbox. Each new
Runner fails at once with "Address already in use"
(`runner_ingress_bind_conflict`). Each launch also overwrites the
process marker that pointed at the old Runner, so the next check cannot
find the old Runner to stop it. The run restarted about 12 times in 19
minutes and then stopped, and the task stayed blocked.
> - Two Runner processes must never use the same saved state at the same
time.
> - This pull request makes each Runner launch stop any live Runner that
still names the same state folder, before the new Runner starts.
> - The benefit is that a restart after an upgrade gets the port and
continues the run, instead of looping on the bind conflict.
## Linked Issues or Issue Description
**What happened?**
On staging, a Runner launched by the old server just before an upgrade
kept the Runner port. Every relaunch by the new server exited within
seconds with "runner_ingress_bind_conflict: failed to bind listener:
Address already in use (os error 98)". The run looped for about 19
minutes and then failed with `native_runner_process_exited`. The task
was blocked and needed a person.
**Expected behavior**
The relaunch stops the leftover Runner of the same session, gets the
port, and continues the run.
**Steps to reproduce**
1. Launch a Runner for a run in a sandbox.
2. Stop the server before the Runner is adopted (an upgrade). The Runner
stays alive.
3. Let the new server restart the run's bootstrap. The new Runner cannot
bind the port, and the next restart cannot see the old Runner because
the marker now names the dead new process.
## What Changed
- `native-session-executor.ts`: the detached Runner launch script now
runs a short fence before it writes the process marker.
- It finds processes whose argv[0] is `paperclip-runnerd`, that are not
a `reap-lineage` command, and whose `--state-dir` is followed by this
exact folder. Runners on another folder, a folder named by a different
flag, and other programs are not touched.
- It records the stale Runner's lineage by PID **and start time**: its
descendants, and members of a process group or session that a recorded
process leads while that leader is alive (provider tools start their own
sessions). It signals a PID only while its start time still matches, so
a reused PID is never signalled, and it never signals a whole process
group.
- TERM, up to 5 s wait, then KILL, up to 2 s wait. Then it runs
`<runnerd> reap-lineage --state-dir … --runner-pid …` for the stopped
Runner, which stops the tools its lineage journal recorded that have
already left the process tree.
- It fails closed: a survivor (`runner_stale_state_holder_alive`) or a
reap that leaves a recorded process alive or gives no report
(`runner_stale_lineage_unproven`) makes the launch exit 75 **before**
the marker is written, so the existing marker is kept. The reason goes
to the process stderr log only when that path is a plain file (same rule
as the Runner stderr log). The whole stop is bounded at about 14 s,
under the 20 s marker wait.
- Two reap refusals let the launch continue with a
`runner_stale_lineage_partial` warning: `journal_unavailable` (the
Runner kept no journal; its visible lineage is already gone) and
`unattributed_orphan` (reported only after every recorded process is
gone, for orphans the reaper can neither claim nor signal) **only when
every orphan it names is proven to have another owner**: this launch's
own detached shell, a `paperclip-runnerd` serving a different
`--state-dir`, or that Runner's stderr trimmer. Any other orphan could
be an old tool that escaped between samples, so the launch fails closed.
- When `/proc` is not readable, or `grep` is missing, the fence does
nothing (the old behavior).
- `native-session-executor.test.ts`: Linux tests (a stale Runner whose
tool runs in its own session, plus four look-alikes that must survive;
fail-closed with the marker kept, with a plain and a symlinked log; reap
refusals for orphans owned elsewhere, an unknown orphan, and no journal;
a normal launch starts no extra node process) and a cross-platform check
that the embedded script parses and has no shell-special characters.
**When a launch happens vs. a reattach.** The launch is reached only
after the server chose not to adopt a Runner. A same-run reattachment
(`reattach_remote_runner`) first verifies the marker's Runner: if it is
alive it is adopted and nothing launches; a transient verification
failure (for example the read timeout that #21053 keeps the Runner alive
for) throws and never reaches a launch; only a Runner proven not running
falls through to a fresh launch. Warm recovery likewise throws instead
of launching when revalidation fails. Every launch overwrites the
marker, so after a launch no reattachment could adopt a leftover Runner
anyway, and its port would make the new Runner fail with
`runner_ingress_bind_conflict`. Stopping it therefore never removes a
Runner that the same run could still re-adopt.
## Verification
- Linux (Debian bookworm container, node 24): the Linux tests pass on
this head. The same tests against the previous head (
|
||
|
|
fb8b61bc3e |
fix(native): restart a bootstrap whose closing server staged files but never launched its Runner (#21134)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent turns run in a remote Runner (`paperclip-runnerd`) in a Daytona sandbox. A release swap stops the old server and a new server recovers each run. > - When the old server stops during a run's bootstrap, the new server restarts that bootstrap on the same run (`bootstrap_incomplete`). The claim is granted only when the Postgres controller checkpoint proves that no provider turn was sent. > - The old server can stop after it staged launch files (Codex home, config) into the sandbox session folder, but before it launched the Runner. Its detach fence stops it there. > - The new server then finds no Runner state in the sandbox. A fresh bootstrap must create the session folder itself, and the folder already exists, so the run failed with `runner_harness_state_mismatch: resumed_unavailable`. The failure is an ambiguous session, so the task waits for a person. > - This pull request lets host preparation prove that no Runner was ever launched in that folder, move the folder aside, and restart the bootstrap. > - The benefit is that a release swap during this part of a bootstrap no longer blocks the task. No provider work can be lost or replayed, because no Runner ever ran. ## Linked Issues or Issue Description Refs #21120 (same release-swap class: the Runner was launched and connected before the old server stopped; that pull request stops it in host preparation. This pull request covers the case where no Runner was launched.) Refs #21079 (launch-time fence for a stale Runner on the same state folder.) **What happened?** On hosted staging, during a release swap on 2026-11-12, a Codex run on Daytona failed with `native_session_interrupted` and `runner_ingress_unavailable: runnerd process failed: runner_harness_state_mismatch: resumed_unavailable`. The old server was in `harness_state.launch_material` when its shutdown drain detached the run. It recorded no Runner process identity. The new server chose "Restarting an incomplete native runner bootstrap on the same heartbeat run". Its host preparation found the sandbox resumed and no Runner state. Then the claim of the session folder failed, because the old server had already created that folder. The run needed a person. **Expected behavior** The run restarts its bootstrap on the same sandbox and continues without a person. A provider turn is never replayed and two Runners never share the run. **Steps to reproduce** 1. Start a Codex run on a Daytona sandbox. 2. Stop the server after the bootstrap stages launch files into the sandbox session folder and before it launches the Runner. 3. Let a new server recover the run with a `bootstrap_incomplete` claim. The new server has no local controller root for the run. 4. Host preparation fails with `runner_harness_state_mismatch: resumed_unavailable`. **Paperclip version or commit** Staging release `704b3f4d63`. The code is the same on master at `ef2b53b675`. **Deployment mode** Hosted staging, remote sandbox execution on Daytona. **Agent adapter(s) involved** Paperclip Runner with Codex (`codex_app_server`). ## What Changed - `native-session-executor.ts`: under a same-run `bootstrap_incomplete` claim with no failover backup, when host preparation finds no Runner state, a new check asks the sandbox in one command whether the session folder exists and holds no Runner trace. A Runner trace is the live `runner/runner-state.json`, the `runner/runner-process.identity` launch marker (the launch script writes it before the Runner starts), an `authority-epochs` archive, or any other `runner-state.json` or launch marker anywhere under the folder. If the folder holds no Runner trace, no Runner was launched there, so host preparation records the restart proof (`absent_confirmed`). It then uses the existing path: move the folder aside (it is kept for inspection), claim a new folder, and bootstrap again. If the folder is missing, the existing fresh-claim path runs unchanged. If any Runner trace exists, the run stays fail-closed as before. An archived session is therefore never discarded. - `native-session-executor.test.ts`: new `interrupted_bootstrap_unlaunched` cases. Resumed and created sandboxes restart the bootstrap. A launch marker, an archived authority epoch, Runner state elsewhere in the folder, or a failover backup keeps the run fail-closed with `resumed_unavailable`. ## Verification - Red: with the source change reverted, `pnpm exec vitest run server/src/services/native-runtime/native-session-executor.test.ts -t "bootstraps only an untouched provider session" --maxWorkers=2` fails on both `interrupted_bootstrap_unlaunched` restart cases with `runner_harness_state_mismatch: resumed_unavailable`, which is the staging error. - Red: without the archived-state guard, the `_archived` and `_nested_state` cases fail (the folder is moved aside). With the guard they pass. - Green: the same command passes (64 tests). - `pnpm exec vitest run server/src/services/native-runtime/native-session-executor.test.ts --maxWorkers=2`: full file, 1422 passed before the guard commit. The focused 64 pass on the final revision. - `npx tsc --noEmit -p server/tsconfig.json`: passes. - `git merge-tree` against #21120's head (`0d18f18928`) merges without conflict. The new check runs only when no Runner state exists, so #21120's new call site (it runs only when a Runner state exists) gets the same results as before. ## Risks - Low. The new path runs only in host preparation, under a `bootstrap_incomplete` claim for the same run id, with no failover backup. That claim is granted only after the previous controller's lease has ended. With no Runner state or launch marker anywhere in the folder, no Runner and no provider session existed there. The old folder is moved aside, not deleted. - A controller that is still launching at the same moment would have to hold a lease that the claim already took over. The detach fence stops such a launch, and #21079 adds a launch-time fence for a leftover Runner on the same state folder. - No added wait before a normal turn starts: the check runs only on this failure path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, browser for read-only run evidence, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
67403baaa4 |
fix(native): an agent termination Stop with no turn result stays a cancellation (#21093)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a person terminates an agent, Paperclip stops the agent's
running native turn
> - Two paths can record that Stop: the terminate route ("Cancelled
because the agent was terminated ...") and the lifecycle worker ("Agent
lifecycle stop requested"). Both are system-initiated and expected
> - When the stopped turn ends without a semantic result, the native
executor keeps the run cancelled only for a person's Stop, a
reassignment, or a governance pause. A termination Stop fell through, so
the run ended `failed` with `native_session_interrupted`
> - Whether the run ended cancelled or failed depended on which path won
the race
> - This pull request treats an acknowledged termination Stop like a
person's Stop in the executor and in the lease-release terminalization
> - The benefit is that a terminated agent's interrupted run ends
cancelled, whichever path records the Stop first
## Linked Issues or Issue Description
Refs #20186 (the same treatment for a governance pause Stop)
**What happened?**
A person chose Terminate for an agent while its native turn ran. The
lifecycle Stop recorded an acknowledged native cancellation (initiator
system). The turn ended without a semantic result. The executor then
wrote the run as `failed` with `native_finalization_missing: session
returned no semantic result`.
**Expected behavior**
The run ends `cancelled`. A termination is an intentional Stop, not a
failure to retry.
**Steps to reproduce**
Terminate an agent while its native turn runs, so that the Stop
interrupts the turn before it records a result. Before this change, the
run became `failed` when the lifecycle worker recorded the Stop first.
## What Changed
- `governance-pause-stop.ts`: adds `AGENT_TERMINATED_STOP_REASON`,
`COMPANY_ARCHIVED_STOP_REASON` and `isAgentLifecycleStopRecord`. The
predicate accepts a system-initiated, expected control-plane
cancellation whose reason is a lifecycle, termination or company-archive
Stop.
- `acknowledged-native-stop.ts`: adds
`hasAcknowledgedNativeLifecycleStopIntent`, which requires that record
and an acknowledged, dispatched native Stop receipt.
- `native-session-executor.ts`: the no-semantic-result branch accepts
that intent, so it settles the coordinator as `native_retry_cancelled`
and raises `NativeCancellationPendingRecoveryError`, as for a person's
Stop.
- `heartbeat/run-lifecycle.ts`: lease-release terminalization keeps such
a run `cancelled`.
- The terminate route and the lifecycle driver use the shared reason
constants. The reason text does not change.
## Verification
- `native-session-executor.test.ts`: a new case for each reason
(lifecycle Stop, terminate route, company archive). Each was red before
the change and is green now. The earlier control case now uses a system
Stop that is neither a pause nor a termination, and it still takes the
retry path. Full file: 1414 passed.
- `heartbeat-run-lease-release-terminalization.test.ts`: a new case for
a lifecycle Stop when teardown wins the finalizer race. It was red
before the change (`interrupted`) and is green now.
- Product evidence: a paid chat-eval run of `agents.retire-agent` on a
branch that combines this change with three related fixes. The result is
in a PR comment.
## Risks
- A termination or company-archive Stop that interrupts a turn no longer
schedules an exact resume of that turn. The agent is leaving service, so
no resume can run.
- Other system Stops keep their current behavior.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
3eca7a6112 |
fix(runner): log which check refuses a saved provider-pack admission on reattach (#21085)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - OpenCode turns run in a remote Runner. When a server restarts, the new server reconnects to the Runner and reuses the provider pack admission that the run saved earlier. > - On staging, a run reconnected cleanly after an upgrade but then failed with `native_provider_artifacts_admission_unverified`. Its saved admission state stayed "pending". > - The reuse step refuses a saved admission for several reasons. It returns only false and logs nothing, so we cannot tell which check refused it. > - This pull request adds diagnostics only. It names the refused check in the run log and the server log. It also logs when the admission stays unverified for a session that needs it. > - The benefit is that the next upgrade audit shows the real cause, so the next fix can target it. ## Linked Issues or Issue Description **What happened?** After an automatic staging upgrade, a run reconnected to its live Runner without a new process. The next checkpoint failed with `native_provider_artifacts_admission_unverified`, which used one of the run's 3 attempts. The run log had no line that explained why the saved admission was not reused. **Expected behavior** When a saved admission is not reused, the run log and the server log name the check that refused it. **Steps to reproduce** 1. Start an OpenCode run on a remote sandbox. 2. Restart the server, so the run reconnects to its Runner. 3. If the saved admission is refused, the run fails later with `native_provider_artifacts_admission_unverified` and no reason in the logs. ## What Changed - `native-provider-pack-admission.ts`: `rememberRecoveredNativeProviderPack` accepts an optional `onRefused` callback. It is called with one reason: `source_state_pending`, `source_state_unknown`, `source_state_missing`, `receipt_missing`, `runner_instance_differs`, `environment_lease_differs`, `receipt_invalid`, or a session identity reason. The result of every check stays the same. An error in the callback is ignored. - `native-session-executor.ts`: the reattach path logs the reason to the run log and the server log. When `recordNativeProviderPackAdmission` returns false for a session that requires admission, it logs that too. - `native-provider-pack-admission.test.ts`: each refusal names its reason, and a reused admission reports nothing. ## Verification - `npx vitest run src/services/native-runtime/native-provider-pack-admission.test.ts` passes 46/46. The new reason tests fail on the old code because `onRefused` is never called. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts` passes 1412/1412. - `npx tsc --noEmit -p server/tsconfig.json` passes. ## Risks - Low risk. This change adds diagnostics only, and admission decisions do not change. The new log lines contain only a fixed reason code and run ids. They do not contain secrets or file paths. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use (shell, code search, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5820d43197 |
fix(runner): relaunch a settled OpenCode Runner with its admitted artifact paths (#21076)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - OpenCode agent turns run in a remote Runner in a Daytona sandbox.
The Runner saves its state, so a retry of the same run can start a new
Runner from that saved state.
> - The first launch of a run starts OpenCode from the admitted provider
files under one folder in the sandbox. The new Runner replays that first
launch and checks that its own launch profile names the same files.
> - When a retry reconnects to a Runner that already stopped, Paperclip
starts a new Runner. On that path, Paperclip did not restore the
admitted provider files and gave the new Runner the provider pack folder
instead.
> - The new Runner then refused the replayed launch ("does not match the
runner-owned qualified profile"). That used the last retry, and the task
stayed blocked.
> - This pull request restores the admitted provider files and gives the
new Runner the same folder as the first launch.
> - The benefit is that a retry after a server upgrade continues the
same run instead of blocking the task.
## Linked Issues or Issue Description
**What happened?**
On staging, after an automatic upgrade, a run used all 3 attempts. The
last attempt started a new Runner from its saved state. It failed in the
drain step with "OpenCode run.prepare launch does not match the
runner-owned qualified profile". The task was blocked and needed a
person.
**Expected behavior**
The new Runner gets the same OpenCode launch files as the first launch
of the run, and the run continues.
**Steps to reproduce**
1. Start an OpenCode run on a remote sandbox with durable provider
artifacts.
2. Let the server restart, so the retry reconnects with
`reattach_remote_runner`.
3. The Runner is found settled, so a new Runner starts from the saved
state. Its launch profile points at the provider pack folder, not at the
admitted artifact folder.
## What Changed
- `native-session-executor.ts`: when a reconnect relaunches a settled
Runner and the run has admitted provider artifacts, restore those
artifacts and prepare the same artifact folder as the first launch.
Other reconnects (live adoption) do not change. Normal turn starts do
not change.
- `native-session-executor.test.ts`: the existing settled-relaunch case
now also covers a run with admitted provider artifacts. It expects the
restore and the admitted folder in the launch profile.
## Verification
- Red: without the fix, the new test case fails
(`restoreNativeProviderArtifacts` is not called, and the launch uses the
provider pack folder).
- Green: `npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/remote-opencode-artifacts.test.ts` passes
1417/1417.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
## Risks
- Low risk. The change applies only when a reconnect must relaunch a
settled OpenCode Runner and the run has admitted provider artifacts.
- That relaunch now downloads the admitted artifacts from storage, as a
cold continuation already does. This adds time only on that recovery
path, not on a normal turn start.
- The artifact folder check (same account and same digest) still
applies, so a different sandbox account still fails closed.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, code search, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
559551e977 |
fix(native): wait for a busy agent directory while retiring a predecessor (#20991)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run on a reused warm sandbox first retires the previous run's warm session, and that retirement collects the old session's agent files > - On a server without a persistent volume, agent-file collection takes a per-agent Postgres lock, so runs of one agent collect one at a time > - #20863 sends a run whose predecessor cannot be retired to a fresh sandbox. It classified every retirement error as unprovable, including a busy lock held by another run of the same agent > - So ordinary lock contention failed the run, dropped a healthy sandbox and rested the task > - This pull request makes retirement wait for a busy directory lock in the same run, and keeps the fresh-sandbox fallback for retirements that stay unprovable after the lock is obtained > - The benefit is that busy agents no longer fail follow-up runs on their warm sandboxes ## Linked Issues or Issue Description **What happened?** A no-fault restart-chaos wave with 8 concurrent runs of one Claude agent on Daytona, with a volume-free server, failed one monitor-due follow-up after 6 seconds. The error was `adapter_failed`, cause `native_predecessor_retirement_unproven`: "The previous run's session on this sandbox could not be retired (Agent directory operation is already active)". The run also logged `INSTRUCTION_COLLECTION_UNAVAILABLE`. The stack shows the error came from `claimWarmNativeInstructionCopy` during instruction preparation. The previous session had closed; only its agent-file collection met the per-agent lock that another run held. **Expected behavior** A busy agent-directory lock is transient. The run waits for it and then retires the predecessor on the same sandbox. Only a retirement that stays unprovable after the lock is obtained takes the fresh-sandbox fallback. **Steps to reproduce** 1. Run several warm native tasks of one agent at once on Daytona with a volume-free server. 2. Re-wake one task while another run of the agent collects its files. 3. The follow-up fails with `native_predecessor_retirement_unproven` and its sandbox is dropped. Refs #20863, #20829, #20957. ## What Changed - `retireWarmPredecessor` (native executor): a busy agent-directory lock (`AgentDirectoryOperationBusyError`) is waited out in the same run, polling every second for up to five minutes (`warmRetirementDirectoryWait`). The usual holder is another run's short collection, so retirement then finishes on the same sandbox with the copy this run adopted. The run is not deferred: deferring after adoption would release the adopted copy and could skip saving the predecessor's edits. - Past that budget the run fails with its own error, `native_predecessor_retirement_directory_busy`, not as an unprovable retirement. Before failing it files the adopted copy's retirement receipt, so the copy is kept as `pending_collection` and run cleanup does not mark it `unavailable`. The owner is not marked unretired and stays registered for its sandbox, so a later run retires it there. - An idle-timeout close that meets the lock retries after the agent-directory retry delay instead of leaving the session registered with no retry. - Closing idle sessions for environment delete (#20957) reports such an owner as `busy`, not `failed`. ## Verification - RED on master, then GREEN: - `native-session-executor.test.ts -t "directory lock"`: two runs of one agent contend on the lock while a follow-up retires the predecessor. On master the claim rejects with `NativePredecessorRetirementUnprovenError` (`Agent directory operation is already active`). Now a short-lived lock is waited out and the predecessor is retired and collected on the same sandbox; a lock that outlasts the budget fails with `native_predecessor_retirement_directory_busy`, environment delete reports `busy: 1`, a fresh-sandbox attempt does not forget the owner, and once the lock is free a later run retires it. - `native-session-executor.test.ts -t "idle close whose file collection"`: on master the idle close never retries; now it retries and closes. - Existing suites pass: native executor, agent-directory busy heartbeat, native pre-dispatch retry, native controller recovery and plugin runner eviction. Server `tsc --noEmit` passes. ## Risks - Under contention a follow-up can wait up to five minutes before its first turn. The normal path adds no wait. - A lock held for more than five minutes fails the run visibly instead of dropping the sandbox. - An idle owner whose collection stays busy keeps retrying at the 5 to 15 second agent-directory delay until the lock frees, a controller restart, or the next admission. - No persisted state changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with tool use (shell, file edits, test runs) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
814fe5d20f |
fix(runner): issue the remote bootstrap ticket at dispatch and retry unauthenticated starts (#21073)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a remote sandbox start `runnerd` in the sandbox, and Paperclip connects to it through provider ingress and authenticates it with a one-use bootstrap ticket > - The ticket lives for 60 seconds, but Paperclip issued it before the remote launcher staged its files in the sandbox > - Under load, staging took 60 to 81 seconds, so `runnerd` presented an expired ticket and every connection was closed. The run failed with `runner_ingress_auth_failed`, and the task stopped behind a manual Retry > - This pull request issues a fresh ticket right before dispatch, retries these failures automatically when the controller checkpoint proves no runner ever authenticated, and adds spans that attribute the slow startup window > - The benefit is that a slow but healthy start authenticates, a failed start recovers on its own, and the remaining startup delay becomes measurable ## Linked Issues or Issue Description **What happened?** On a hosted instance, 9 runs of one Codex agent on Daytona failed in about 70 minutes with `native_session_interrupted` and the message `runner_ingress_auth_failed: Runner ingress did not complete PRP authentication before the deadline.` In the same window only 6 runs succeeded company-wide. The run log showed Paperclip reconnecting for 60 seconds while `runnerd` reported that its peer closed each WebSocket. In 294 remote runs with timing spans, the time between the bootstrap ticket issue (end of `runner.transport.connect`) and the process dispatch (start of `runner.process.launch`) decides the outcome without exception: - 60 s or more: 23 of 23 runs failed authentication. - Under 60 s: 271 of 271 runs authenticated, in 0 to 1 s. The tasks did not recover by themselves, because the failure was classified as ambiguous state. **Expected behavior** A runner that starts slowly but correctly authenticates. If ingress authentication still fails before any runner authenticated, Paperclip retries the start automatically. The time between the start request and the runner connection is visible in run spans. **Steps to reproduce** 1. Run a Codex agent on a Daytona sandbox environment that uses provider ingress. 2. Make remote staging (`runner.runtime.stage`, inside the remote launcher) take 60 seconds or more. Many concurrent runs of the same agent do this on staging. 3. See that `runner.prp.authenticate` fails after 60 seconds and the run ends with `runner_ingress_auth_failed`. 4. The regression test `runnerd-bootstrap-ticket-refresh.test.ts` reproduces this locally with a 75-second staging delay. **Paperclip version or commit** `4e519bd4` (deployed), reproduced on `origin/master` at `4228bcf89c`. **Deployment mode** Hosted (cloud), controller on Railway, sandboxes on Daytona. **Agent adapter(s) involved** `paperclip_runner` with the Codex app-server provider (`gpt-6.1-sol`), remote sandbox target, provider-ingress transport. ## What Changed - `RunnerProcessLaunchSpec` and `spawnRunner` accept `refreshBootstrapTicket`. A restart does not inherit it. - The runnerd Codex transport passes a refresh that issues a new bootstrap ticket and flushes the controller checkpoint. It does not do this for a warm-transition launch. - The remote sandbox launcher calls the refresh after `ensureArtifact` and immediately before the launch RPC, and puts the new ticket in the launch environment. - `inspectUnauthenticatedBootstrapCheckpoint` proves from a controller checkpoint that no runner authenticated: no connections, no leases, no used tickets, no events, and only pending commands. - `nativeSessionFailureSourceCode` classifies the outbound ingress startup failures as `runner_ingress_startup_failed`. `nativeProviderRecoveryEvidence` treats it as pre-session (`bootstrap_retry`) only when that proof holds for the exact run identity and there is no provider session or provider event. Restart recovery and failed-run retry accept the new code with the same rules as similar startup codes. - Remote startup now records `runner.control_plane.open`, `runner.launch.prepare`, `runner.checkpoint.register` and `stage.asset.codex_skills.verify`. The 6 to 11 minute gap between `native.process_start_requested` and `runner.transport.connect` had no spans before. ## Verification - Red/green: `packages/paperclip-runner/src/live/runnerd-bootstrap-ticket-refresh.test.ts` starts a real `runnerd` with a 75-second staging delay. Without the fix it fails with `runner never connected`. With the fix it opens the session. - Red/green: `native-session-executor.test.ts` "launches runnerd with a bootstrap ticket issued after remote staging". Without the fix the launch RPC gets the stale ticket. - Red/green: `native-controller-recovery.integration.test.ts` "retries a fresh remote start whose launched runner never authenticated through provider ingress" (embedded Postgres). Without the fix the failure is `native_session_interrupted` and `ambiguous_state`. - `durable-prp-control-plane.test.ts`: the new checkpoint proof, its negative cases, and the restart rule for the refresh. - `native-run-trace.test.ts`: remote startup spans are recorded without duplicate launcher spans. - Focused suites pass: runner package 5 files and 235 tests; server 9 files and 1,607 tests, including `native-session-executor`, `native-runner-execution-driver`, `remote-launch-assets`, `native-restart-recovery`, `runner-prp-outbound` and the recovery integration tests. - `tsc --noEmit` passes for `packages/paperclip-runner` and `server`. ## Risks - A launch now issues two bootstrap tickets. The first one is never sent to `runnerd` and expires in 60 seconds. Bootstrap counters already allow more than one issued credential. - The new retry path needs a checkpoint proof that no runner authenticated, so a run that did provider work cannot be retried this way. The existing attempt budget still bounds the retries. - This change does not remove the 6 to 11 minute controller-side delay before `runner.transport.connect`. The new spans make it attributable for a follow-up change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, browser, code search) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f30005440b |
fix(recovery): reconnect a live remote Runner with its admitted skills instead of re-downloading them (#21064)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner sandbox. A server upgrade
detaches each running turn, and the new server must reconnect to its
Runner (restart recovery).
> - A reconnect runs the full turn preparation before it contacts the
Runner. On a new server, skill preparation downloads every saved skill
version again, under one lock for the whole company.
> - After an upgrade, many reconnects wait behind each other's
downloads. The Runner stops answering after about 2 minutes, so a late
reconnect loses the turn.
> - This pull request lets a reconnect to the same run's live remote
Runner use the exact saved skill versions, without downloading them and
without the lock.
> - The benefit is that skill preparation no longer adds to the
reconnect delay, and it does not grow with the number of reconnects.
Normal turn starts do not change.
## Linked Issues or Issue Description
**What happened?**
After an automatic staging upgrade, runs that the new server reached
late failed. The measured time from detach to reconnect was 114–152 s
for the failed runs. Turn preparation took 30–130 s, and skill
preparation alone took 46 s on one run.
**Expected behavior**
A reconnect to a live Runner reaches it quickly. Its speed must not
depend on how many other runs prepare skills at the same time.
**Steps to reproduce**
1. Run a remote Runner turn for each of many tasks in one company.
2. Upgrade the server while those turns run, so the new server starts
with an empty skill cache.
3. See that each reconnect waits in
`heartbeat.prepare.configuration_skills` for the company skill cache
lock and downloads the skills again.
**Paperclip version or commit**
master at
|
||
|
|
fd9985b571 |
fix(native): say what a Runner command waited behind when it times out (#21056)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on remote sandboxes talk to a Runner through durable PRP commands. The controller waits for each command's result > - After a server replacement, the new controller reattached to live Runners. For runs it reached later than about 115 s after detach, `session.snapshot` got no answer within 30 s > - The failure said only `provider_transport_failed: PRP command session.snapshot timed out`. The run log could not tell whether the Runner was disconnected, whether the command waited behind older commands, or when the Runner last sent an event > - This pull request adds that evidence to the timeout message. It names types, counts and durations only, never payloads > - The benefit is that the next upgrade's failures explain the stall, so the runner-side cause can be fixed ## Linked Issues or Issue Description Refs #21053, #21052, #21039 **What happened?** At the 07:10Z staging swap, 9 remote native runs that the new controller reached 114–152 s after detach failed. Each recovery read ended with `PRP command session.snapshot timed out`. The transport had authenticated, and in some runs the same Runner process was verified. The Runner journal recorded no diagnostic other than "WebSocket peer closed the connection". Runs reached within 39–116 s succeeded. **What did you expect to happen?** A command timeout should say what the command waited behind, so the cause can be found from the run log. **Steps to reproduce** 1. Start a remote native run and replace the server while a provider turn is active. 2. Let the new controller reach the Runner more than about two minutes after detach. 3. Read the run log. The failure says only "timed out". ## What Changed - `packages/paperclip-runner/src/live/runnerd-codex-transport.ts`: `waitForRunnerCommand` accepts an optional `describeStall`. When a command times out, the message ends with a bounded description, for example `(runner connected; pending ahead 1 [semantic_tool.result] oldest 188s; last runner event seq 152 181s ago)`. The description covers whether a Runner is connected or how long it has been disconnected, the pending commands ahead of this one (types and count, up to 6 types, and the age of the oldest), and the last committed Runner event's sequence and age. A failing or empty description keeps the plain message. - The transport passes this description for every command it waits on. The last event's age comes from the PRP payload's `emittedAt`. - `server/src/services/native-runtime/native-session-executor.ts`: the startup `session.open` timeout classifier accepts the optional description, so a startup timeout stays `provider_session_open_timeout` and keeps its bounded same-run retry. A regression case covers each startup code with the suffix. - Tests: the exact message for the staging shape, the plain message when there is nothing to describe or the description throws, and a disconnected Runner with no events. ## Verification - `cd packages/paperclip-runner && npx vitest run src/live/runnerd-codex-transport.test.ts -t "a Runner command that gets no answer"` passes 3 of 3. These tests fail on `origin/master`, which has no `describeRunnerCommandStall` export and no description in the message. - The full `runnerd-codex-transport.test.ts` file passes 307 of 308. The one failure (`stops a lost runnerd's provider and escaped tool job from its lineage journal`) is a process-timing test that passes alone and does not touch this code. - `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` passes. - Message matchers: the only matcher anchored on the end of the timeout text was the startup `session.open` classifier. It now accepts the suffix. `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "classifies a startup session.open timeout"` passes. Without the classifier change it fails, because the suffix turns the timeout into `native_session_interrupted`. ## Risks - Low. Only the text of a timeout message changes, by appending text. No control flow, persisted state or Runner behavior changes. - The description holds command types, counts and durations only. It holds no payloads, identifiers or provider text. - No wait is added before a turn start. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic) via Claude Code, with tool use (shell, file edits, test runs, read-only staging API reads). Model ID as attributed in the commit trailer: Claude Fable 5.2. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
aa94939988 |
fix(native): continue a run whose remote Runner died mid-turn after a restart, without replaying the turn (#21039)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) use a detached
`paperclip-runnerd`. After a server restart, recovery reattaches to that
Runner on the run's own sandbox lease
> - On staging, a deploy detached the controller and left the Runner
alive. The restart's reattachment then found the Runner process gone
with a provider turn in flight. The run failed with
`runner_remote_process_identity_unavailable` and the task waited for a
person
> - The run log said `startup reuse: environment (fresh) created`, so
the sandbox looked re-created. It was not. The reattachment resumed the
exact retained lease. The receipt reported the lease's original creation
> - `requestInterruptedNativeTurnRetry` refused every unsettled turn
(`provider_turn_unsettled`), and refused every system wake unless the
journal proved that no turn was issued. So this run got no automatic
continuation
> - This pull request continues such a run once in a fresh provider
session when the refused reattachment proved the Runner process gone. It
records that the turn's outcome is unknown and never replays the turn.
It also makes the environment receipt tell the truth
> - The benefit is that a restart that kills a Runner in the middle of a
turn no longer needs a Retry click
## Linked Issues or Issue Description
Refs #20853, #20923, #21023, #20278, #20627, #20828
**What happened?**
A staging deploy landed in the middle of a remote native Codex run on
Daytona. Shutdown detached the controller and logged "durable runner
left alive". The new server reattached through the run's original lease
(same lease row, same provider sandbox id, same directory). Its process
check then found the Runner gone: `process_verification` /
`process_not_running`, `runnerLifecycle: ready`, `runnerSettled: true`,
last diagnostic "WebSocket peer closed the connection", empty stderr.
The provider checkpoint still had an active turn. The run failed with
`runner_remote_process_identity_unavailable` /
`native_continuation_requires_reconciliation`, and the task went to
`blocked`.
**Root cause**
- `server/src/services/heartbeat/restart-interrupted-turn.ts`
(`requestInterruptedNativeTurnRetry`): an unsettled last turn always
returned `provider_turn_unsettled`, also when the refusal proved the
Runner process gone. A system wake (here `transient_failure_retry`)
continued only with `providerTurnIssued: false`.
- The sandbox was not re-created. The environment and lease listings
show one lease for the run, never replaced, and the final workspace
snapshot used the same provider sandbox. `environmentStartupReuse` read
`sandboxLeaseAcquisition: created` from the lease's first acquisition,
so the reattachment's receipt said `fresh / created`. The 46 s acquire
was the provider resume of that exact lease. The evidence available to
GET requests cannot say what stopped the Runner process (the resume
result is not kept). This pull request does not change that path.
- #21023 named this run as not covered. #20853 resumes a Runner only at
a settled, idle `ready` cut. Here the provider checkpoint has an active
turn, so #20853 stays fail-closed, as it should.
**Expected behavior**
The run continues once by itself in a fresh provider session over the
saved work. The turn whose outcome is unknown is recorded and is not
replayed.
## What Changed
- `restart-interrupted-turn.ts`: read the last
`native.remote_runner_recovery_failed` refusal once. When it proves the
Runner process gone, an unsettled last turn no longer blocks the
automatic Retry. Before it requests the Retry, the server records
`native.interrupted_turn_outcome_unknown` once (turn id, last turn
event, `replayed: false`, closed `runnerExit` fields). The Retry is the
existing one: saved-work recovery and stop proof first, then a fresh
provider session (`forceFreshSession: true`), once per chain.
- The same file: a system wake whose run is an automatic retry of owed
work (`retryOfRunId` set) also continues under the run's responsible
user once the refusal proved the Runner gone. Other system wakes (for
example a monitor) keep their policy.
- `packages/adapter-utils/src/startup-reuse.ts` and
`native-session-executor.ts`: a `reattach_remote_runner` recovery
reports its `environment` receipt as `reused / verified_reuse`. That
path only resumes the run's exact retained lease and never acquires one.
- Docs: `doc/execution-semantics.md` and `doc/run-log-events.md`.
Previous-release story: no persisted shape changes. The rule reads the
closed `remoteRunnerRecovery` fields that the previous release already
writes; the new test uses the staging refusal payload as written. A run
that a previous release already failed this way continues on the next
completion or restart hook call only; runs already settled stay as they
are and keep their Retry. The new event is additive.
## Verification
- Red on `origin/master` (
|
||
|
|
d68088465c | Reuse retained Runner descriptions during startup (#21004) | ||
|
|
776e900b48 |
fix(native): retire a finished run's idle Runner that still holds the sandbox Runner port before launching (#20995)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs execute through a Runner (`paperclip-runnerd`) inside a remote sandbox. Daytona reaches every Runner through one fixed preview port (43127), so a sandbox serves one listening Runner at a time. > - A warm sandbox is reused by the next run of the same task. That run often opens a new session, so it neither adopts the previous run's warm Runner nor retires it. Retirement of another session's warm owner works only through the in-memory warm registry of the server that launched it. > - After a server restart (or on another server), that registry is empty. The previous run's idle warm Runner keeps the port, and every new launch fails with `runner_ingress_bind_conflict` until the bounded retry runs out. > - This pull request finds such a holder on the sandbox before launch and proves from durable records whether it may be stopped. A finished run's idle Runner is stopped through its verified process identity. Any other holder sends the run to the fresh-sandbox fallback from #20863 and is never stopped. > - The benefit is that a follow-up run after a restart starts on its reused sandbox instead of colliding with its predecessor and needing Retry. ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T230537-s6402` (Claude through ACPX, Daytona warm sandboxes, `--ephemeral-server`, K1/K2/K15 with `--k15-fresh-home`, seed 6402, master, no injected fault) needed Retry on 4 of 6 tasks. Each failing run's first Runner failed with `runner_ingress_bind_conflict: failed to bind listener: Address already in use`. The harness's sandbox captures name the process holding port 43127 at the time. In every case it was the previous, already `succeeded` run of the same task, started under the server generation before the restart: | Task | Failing run | Port 43127 held by | Holder run status | | --- | --- | --- | --- | | t01 | b4769dfd | 395b202d (Runner started 23:11:30) | succeeded 23:12:19 | | t05 | b6d466fd | 0969ad46 (23:11:29) | succeeded 23:12:18 | | t02 | 4097fc1f | 527a1659 (23:11:28) | succeeded 23:12:17 | | t03 | 60d821af | 1c06bf75 (23:11:28) | succeeded 23:12:25 | **Root cause** Warm owners are tracked only in the process-local `warmNativeSessions` map in `server/src/services/native-runtime/native-session-executor.ts`. `claimWarmNativeInstructionCopy`, `reserveWarmNativeInstructionDirectory` and the same-session admission path retire a predecessor only when it is in that map. A restarted server has an empty map. The follow-up run uses a new session, so `inspectRemoteWarmSessionRecovery` (same session only) does not adopt it either. The launch then meets the old Runner on the fixed port. `stopStaleRemoteRunnerBeforeBootstrap` stops only the run's own Runner, so the same-run retries collide again. **Expected behavior** A finished predecessor's idle warm Runner is retired before the next Runner launches on the same sandbox, wherever it was launched. A holder that cannot be proven safe to stop is never stopped, and the run moves to a fresh sandbox. ## What Changed - New `server/src/services/native-runtime/sandbox-runner-port-holders.ts`: `retireSandboxRunnerPortHolders` lists every other session's Runner on the sandbox (session roots with a `runner-process.identity`; this run's own root and moved-aside roots are skipped). It checks each one with the existing verified process check (`REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT`: pid, runner id in the command line and, when recorded, the boot generation). Then, for a live holder: - Registered in this server's warm registry: it is retired through `retireWarmPredecessor`. - Its company-scoped run row is `succeeded` or `cancelled`, for the same agent, and its Runner state is idle (empty outbox, no pending terminal delivery, and both saved fences `pendingProviderCleanup` and `warmTransition` clear): it is stopped through the verified identity (`TERM`, then `KILL`, each with a bounded wait). An activity log entry `native.runner_port_holder_retired` is written. - When the inventory command itself cannot run, the holder state is unproven, so the run takes the fresh-sandbox fallback. - Anything else (a live run, a failed or interrupted run that may still be resumed, an unknown owner, work in the outbox, an unverifiable or changed identity marker, or a stop that is not confirmed): `NativePredecessorRetirementUnprovenError`. Before any process start, the existing #20863 path marks the lease replace-before-reuse and re-queues the run in the uncharged lane on a fresh sandbox. - `executePaperclipNativeSessionAttempt` runs this check before `native.process_start_requested`, and only for a resumed (reused) sandbox launched through a shell runner without a warm session to reuse and without a reattach recovery. A created or replacement sandbox cannot hold another session's Runner, so it adds no command there. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/sandbox-runner-port-holders.integration.test.ts` (embedded Postgres, real processes): two Runners contend for one real port. The second bind first fails with `EADDRINUSE`, the same failure as staging. After `retireSandboxRunnerPortHolders` the finished run's holder has exited and the bind succeeds, with the activity entry recorded. The commands run through a confined runner that refuses any path outside the sandbox workspace base. A holder owned by a live run, by a finished run with work in its outbox, by a cancelled run with pending provider cleanup, or by no recorded run is never stopped: the call raises `NativePredecessorRetirementUnprovenError` (`runner_port_holder_owner_not_settled` / `runner_port_holder_unknown_owner`) and the port stays held. This run's own session, moved-aside roots and exited holders are ignored. - `native-warm-checkpoint-ownership.integration.test.ts`: its fixture sandbox runner now answers the inventory command with an empty list (that sandbox has no other Runner); all 9 cells pass. - `native-session-executor.test.ts`: 1406 of 1406 pass. Server `tsc --noEmit` passes. - Chaos results are in a comment below. ## Risks - A finished run's idle warm Runner is stopped earlier than its idle timeout. It held no unsaved work (its run is `succeeded` or `cancelled` and its outbox is empty), and SIGTERM is the same stop as its idle close. Only that warm reuse is lost. - Failed and interrupted holders are not stopped, because their run may still reattach. Such a run goes to a fresh sandbox instead. - One extra sandbox command before launch on a reused sandbox for a cold launch, no wait and no file changes. A warm reuse and a reattach skip it. - Not changed: a bind conflict from a holder that is not a Runner (no identity file) still uses the existing same-run bootstrap budget. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edit, test runs). Default context window, extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #20863, #20953, #20972, #20992. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
85bb942aad |
fix(native): prove an unconnected bootstrap from the sandbox's own Runner state on a volume-free server (#20992)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs execute through a Runner (`paperclip-runnerd`) in a remote sandbox. When a Runner exits before `session.open`, the run restarts its bootstrap on the same run. > - #20972 lets host preparation restart that bootstrap when the Runner wrote `runner-state.json` but no provider state. It needs a proof that this run's own interrupted Runner is stopped or gone. > - That proof is recorded only when the server still has the run's local Runner controller root. A volume-free server, or a root that held no identity, skips it. The live chaos wave after #20972 still failed with `runner_harness_state_mismatch: resumed_unavailable`. > - This pull request lets host preparation establish the same proof from durable sandbox evidence when no local proof exists. > - The benefit is that the restart also works on volume-free servers, which is how Cloud runs. ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261112T002811-s6402` ran #20972 (Claude through ACPX, Daytona warm, `--ephemeral-server`, K1/K2/K15 with `--k15-fresh-home`, seed 6402, `--runner-bind-fault`). 5 of 6 tasks needed Retry, with 6 `resumed_unavailable` failures. Run `fbb1666d` in t01: the sandbox held `runner-state.json` for this run with `lifecycle: connecting`, the same Runner instance and session, `lastConnectionProtocolVersion: null`, and no provider state. That is exactly the state #20972 restarts, but its proof was absent. **Root cause** `recordRemoteBootstrapRestartProof` is called only inside the `existsSync(scoped)` block of the restart path in `server/src/services/native-runtime/native-session-executor.ts`, after the local controller identity is read. With no local root (volume-free, a fresh home, or an emptied root), `provenOwnBootstrapRestart` stays false, so `inspectOwnUnpreparedBootstrap` from #20972 returns nothing and the resumed branch throws `resumed_unavailable`. **Expected behavior** A run with a `bootstrap_incomplete` recovery claim whose sandbox Runner state proves it never connected restarts its bootstrap without Retry. ## What Changed - `inspectOwnUnpreparedBootstrap` now runs for any `bootstrap_incomplete` claim of this run. When no proof was recorded, it records one only if the sandbox Runner state proves it never connected (`remoteRunnerStateNeverConnected`: `lastConnectionProtocolVersion: null`, `lastControllerCommandSeq: 0`, `reconnectCount: 0`, empty outbox and processed commands, no pending terminal delivery). The exact Runner process must also be stopped or proven absent by the existing `stopStaleRemoteRunnerBeforeBootstrap`, which uses the Runner state's own identity. All earlier conditions from #20972 still apply: same run, Runner instance and session, a lifecycle other than `suspended`, no failover backup and no archived rotation snapshot. ## Verification - Red on master (#20972 merged): `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-session-executor.test.ts -t "bootstraps only an untouched provider session"`. The new `interrupted_bootstrap_unprepared_unrecorded` cells failed: resumed lease `Error: runner_harness_state_mismatch: resumed_unavailable`, created lease no restart. - Green: those cells pass. `_connected` (a Runner state that connected, no local proof) and `_unproven` (the stop is not proven) keep the refusal. The full file passes (1409 of 1409). Server `tsc --noEmit` passes. - Chaos, same cell with `--runner-bind-fault`. Wave `w-20261112T002811-s6402` (#20972 only): 1 of 6 passed, 6 `resumed_unavailable`. Wave `w-20261112T010800-s6402` (this change): 3 of 6 passed, 2 `resumed_unavailable`. In this wave, run `0a080e83` restarted its bootstrap three times on the same sandbox: three `.bootstrap-restart-*` roots were moved aside. The baseline wave on master before #20972 (`w-20261111T230537-s6402`, no injected fault) passed 2 of 6. ## Risks - The new path runs only for a `bootstrap_incomplete` claim of this run when the full harness check fails, so the normal path is unchanged and adds no wait before `turn.start`. - A Runner state that ever connected, received a command or holds outbox data keeps today's refusal. The moved-aside root is kept for inspection. - Not fixed here, and seen in both waves: every Runner on a sandbox listens on one fixed port (43127 in all captures). A live Runner of another run or session on the same sandbox therefore makes each relaunch fail with `runner_ingress_bind_conflict`, until the bounded retry is exhausted (`native_session_retry_exhausted`). A later attempt can still end with `resumed_unavailable` or `resumed_not_suspended`. That needs its own fix in predecessor retirement or port selection. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edit, test runs). Default context window, extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #20972, #20928, #20953, #20863. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
45170b0cac | Reuse Runner plugin capability checks (#20987) | ||
|
|
66344b76ae |
fix(native): restart a bootstrap whose Runner exited before writing provider state on a resumed sandbox (#20972)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs execute through a Runner (`paperclip-runnerd`) inside a remote sandbox. A warm Daytona sandbox is reused by later runs. > - When a Runner exits before `session.open`, the run does a bounded same-run bootstrap retry. Durable evidence proves that no provider session and no provider event exist. > - The retry fails when the Runner wrote `runner-state.json` but exited before it wrote its provider state. Host preparation then calls the sandbox state "unavailable" and refuses with `runner_harness_state_mismatch: resumed_unavailable`. The task needs Retry. > - This pull request reads the Runner state on its own when the full check fails, but only for a proven bootstrap restart of the same run. A non-suspended Runner state of this run is then the run's own interrupted bootstrap, and the existing restart path moves it aside and claims a new session root. > - The benefit is that a Runner that exits before `session.open` on a reused sandbox recovers without a person. Saved state, a verified backup and every other mismatch keep their refusal. ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T230537-s6402` (Claude through ACPX, Daytona warm sandboxes, `--ephemeral-server`, K1/K2/K15 with `--k15-fresh-home`, seed 6402) needed Retry on 4 of 6 tasks, with the same signature after each restart kind. The first attempt of each run reused the warm sandbox. Its Runner failed with `runner_ingress_bind_conflict: failed to bind listener: Address already in use` and exited with `lifecycle=connecting`. Recovery logged "Restarting an incomplete native runner bootstrap on the same heartbeat run". The second attempt reused the environment and failed with `runner_harness_state_mismatch: resumed_unavailable`. The earlier wave `w-20261111T212429-s6402` had one run with the same error code. **Root cause** `inspectRemoteHarnessState` in `server/src/services/native-runtime/native-session-executor.ts` checks the persistence directories, `runner-state.json` and the provider state file in one shell test. A Runner that exits at bind time wrote `runner-state.json` but no provider state, so the one test fails and the result is `incompleteReason: "unavailable"` before the Runner state is read. The restart path (`restartableBootstrapState`) accepts only `not_suspended` and `provider_unopened`, so it is skipped. `claimUntouchedRemoteSession` then cannot create the session root that the first attempt created, and the resumed branch throws `resumed_unavailable`. On a created lease the same state fell through without a restart. **Expected behavior** A proven, interrupted bootstrap of the same run restarts on the same sandbox. The run finishes without Retry. **Steps to reproduce** Run the s6402 cell above, or add `--runner-bind-fault` (this pull request) to force the Runner exit on every run's first launch. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts`: when the full harness check fails during host preparation, `inspectOwnUnpreparedBootstrap` reads `runner-state.json` alone. It returns `not_suspended` for this run only when all of these are true: this run has a proven bootstrap restart (`bootstrap_incomplete` recovery plus the stop proof), no failover backup exists, no archived rotation snapshot is in use, and the Runner state names this run, this Runner instance and this session with a lifecycle other than `suspended`. Otherwise the result stays `unavailable`, as before. The existing restart path then moves the session root aside (it is kept, not deleted) and claims a new one. - Chaos coverage: `--runner-bind-fault` in `tests/restart-chaos` gives every server of the wave `PAPERCLIP_TEST_RUNNER_BIND_FAULT=first_launch`. With `NODE_ENV=test` only, the remote Runner launcher starts a short-lived listener on the Runner's listen port before a Runner's first launch on that server, so the Runner exits with a real `runner_ingress_bind_conflict`. Documented in `tests/restart-chaos/README.md`. ## Verification - Red on master: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-session-executor.test.ts -t "bootstraps only an untouched provider session"` failed 2 of the new cells. Resumed lease: `Error: runner_harness_state_mismatch: resumed_unavailable` (the staging error). Created lease: no restart happened (`expected [] to have a length of 1 but got +0`). - Green: the same cells pass. The negative cells keep `resumed_unavailable`: no restart proof, a Runner state of another run, a `suspended` Runner state with missing provider state, and a failover backup present. - `native-session-executor.test.ts`: 1406 of 1406 pass. A new test proves the bind fault holds the listen port (a second bind gets `EADDRINUSE`) and fires once per Runner instance. - `tests/restart-chaos`: `runner-bind-fault.test.ts`, `question-handoff.test.ts`, `shape-cells.test.ts` pass. Server and harness `tsc --noEmit` pass. - Chaos wave: see the comment below for the before/after counts. ## Risks - The new read runs only for a proven bootstrap restart of the same run with a failed full check, so the normal path has no extra command and no wait before `turn.start`. - A partly saved session (`suspended` Runner state without provider state), a Runner state of another run and any failover backup keep today's refusal. The moved-aside root is kept for inspection. - A Runner that started under an older server and is restarted by this version takes the same path, because it reads only files that older Runners already write. - The bind fault is test instrumentation. It is inert unless `NODE_ENV=test` and the variable are both set. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edit, test runs). Default context window, extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Related: #20928 (restart a Runner that exited before session.open), #20874, #20863, #20953. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
38501b1170 |
Add turn-scoped attachment downloads to native Runner (#20956)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native agents open input images with a file reader. > - Plugin Runner tasks use an authenticated Runner connection. > - Runner needs a way to fetch an authorized file from the tenant app. > - The tenant grants access to the exact attachment for the active run. > - Runner downloads and verifies it before it returns a local path. ## Linked Issues or Issue Description Refs #19740. Builds on the attachment authorization and command transport added in #20939. ## What Changed - Add the PRP v3 `runner.attachment.download` command for native workspace providers. - Issue short-lived grants for exact authorized wake and historical attachments. Recheck the active run, source, size, and SHA-256 at the tenant endpoint. - Download current-wake inputs before provider dispatch. Download historical files when the authorized chat reader asks for them. Preserve SSH and sandbox command staging as a fallback. - Run HTTPS transfers off the Runner control loop. Report completion with one PRP event per transfer. Drop pending results on cancellation and clear files at turn settlement, authority rotation, and restart. - Grant restricted Codex sessions read-only access to the attachment directory. ## Verification - 69 targeted server attachment tests passed. They cover wake and historical downloads, authorization failures, and command fallback. - 22 focused historical attachment tests passed, including grant redemption under a held database lock. - Six focused Rust attachment tests passed, including a source-pinned Codex sandbox check. The sandbox could read the staged file but could not modify it or read adjacent Runner state. - The real Runner transport test confirmed one durable command and one completion event for a transfer. Restart-continuation tests confirmed fresh attachment paths appear before dispatch. - Server and Runner TypeScript checks, Rust formatting, and PRP generated schema checks passed. - Required GitHub CI passed on `7a33752bd7`. [CI run](https://github.com/paperclipai/paperclip/actions/runs/43122967837) - Greptile scored the same revision 5/5 with no unresolved review threads. [Review](https://github.com/paperclipai/paperclip/pull/20956#issuecomment-6808582134) ## Risks - Saved provider and Runner identities keep their existing formats. A previous Runner falls back to command staging where the target supports it. A restart continuation refreshes temporary attachment paths before dispatch. - Plugin Runner tasks require a compatible Runner ROM and outbound HTTPS access to the configured tenant origin. - Live agent-computer qualification remains a release check. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, shell tools, and local test execution. Exact variant and context window are unavailable in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #123` / `Closes #123` / `Refs #123` or (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
c45f54ee74 |
fix(native): stop or release unclosable idle warm Runners on environment delete, and say why a close failed (#20957)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep a warm Runner in a Daytona sandbox between turns of
a task, so the next turn starts fast
> - Each idle warm Runner has an in-memory owner in the controller.
Deleting an environment closes these owners before it destroys the
sandboxes
> - In every warm chaos wave today, some owners never closed. The server
logged only `{"failed":2}`. Environment delete returned 409 forever, and
all sandboxes of the wave stayed `started` until their 60-minute TTL
> - On staging, that is leaked sandboxes (cost) and owners that block
the work after them
> - This pull request logs and reports the exact close error, stops an
unsettled idle Runner when its environment is deleted, and releases an
owner whose sandbox is already destroyed
> - The benefit is that environment delete finishes or names the exact
run and error, and every warm chaos wave now grades a clean shutdown
## Linked Issues or Issue Description
**What happened?**
After warm Daytona chaos waves (W2, W2Q, W2C cells, `--faults K0`) on
Codex, Claude and OpenCode, the server logged `idle native session
cleanup failed before controller shutdown {"idleSessions":{"failed":2}}`
with no reason. `DELETE
/api/environments/:id?destroyReusableSandboxLeases=true` returned 409
"warm native runner did not shut down cleanly" on every retry. Eight
sandboxes per wave stayed `started`.
**Root cause.**
`closeIdleWarmNativeSessions`
(`server/src/services/native-runtime/native-session-executor.ts`)
dropped the close error (`catch {}`) and counted it. The owners that
failed are the W2Q owners. A W2Q task parks a question and is answered
by a later run on a fresh sandbox. With the reason logged, a wave shows
two causes for the same owner:
1. The owner's close cannot settle (`native_session_close_unrecoverable:
runner did not durably suspend before checkpoint`, the governed-wait
shape from #20844). Only a successor reset could force a stop
(`discardUnsettledSession`). An idle sweep or environment delete had no
forced stop.
2. The owner's sandbox lease was already destroyed by the
pending-cleanup sweep after the successor moved to a fresh sandbox
(#20863). Every close then fails with `Daytona sandbox lease <id> is no
longer active.`, because the stop RPC has no sandbox to reach. The owner
stays registered for ever.
The Runner close memoizes its rejection, so every later attempt fails at
once with the same error. This is not the "controller sequence must be
contiguous" exit (#20946): no close failure in these waves names a
sequence mismatch.
**Expected behavior**
An idle warm owner with no turn in flight never blocks environment
delete for ever. Every failed close names its run and its error.
**Steps to reproduce**
1. Run `tests/restart-chaos/main.ts --env daytona --daytona-warm
--adapters claude --workloads W2,W2Q,W2C --faults K0 --concurrency 8
--tasks 8`.
2. Read `<wave>/sandbox-sweep.json`: `environmentDeleteBlocked` is set
and the sandboxes are `started`.
## What Changed
- `closeIdleWarmNativeSessions` now returns `failures: [{ runId,
issueId, environmentId, providerLeaseId, reason }]`. The reason is the
close error with its code, its settlement record and its cause. It also
logs `idle warm native sessions did not close` with that list. The
idle-timeout close logs its error too.
- `closeWarmNativeSessionsForEnvironment` takes `forceStopUnsettled` and
`db`. The environment delete route sets both. When the graceful close
fails, Core retires the owner as #20844 does: one confirmed Launcher
`stop`, the credential snapshot, `complete`, then instruction
collection. A failed forced stop is added to the reason.
- When retirement still fails and the owner's sandbox lease has a
`destroyed` termination receipt, Core releases the owner. It does not
contact the sandbox. It logs `warm native session's sandbox was already
destroyed; released its owner` with the close error.
- The environment delete 409 now names each run and its reasons, and
`details.nativeRunnerSessions.failures` lists them.
- Idle timeout, controller restart and tenant sleep do not force a stop.
Their conversation can still be resumed, so they only log the reason.
- Chaos harness: new check `warm-shutdown.ts`, "no leaked sandboxes /
clean warm shutdown". Every warm wave (`--daytona-warm` or `--lifecycle
warm`) runs it. It fails the wave on a blocked delete, on a sandbox left
running after cleanup, or on a warm session that did not close at
environment delete or controller shutdown. It writes
`<wave>/warm-shutdown.json` and a section in `SUMMARY.md`.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` and the chaos README.
## Verification
- RED on master: `native-session-executor.test.ts -t "stops an idle warm
owner whose close cannot settle"` fails: `expected { closed: 0, busy: 0,
failed: 1 } to match object { closed: 0, failed: 1, failures: [...] }`
(a bare count, no reason). `environment-routes.test.ts -t "names the
warm runner session"` fails: the 409 does not name the run. On the first
fix commit, `-t "sandbox lease is gone"` fails: the lease state is never
read, and the owner stays.
- GREEN: those tests pass. The executor suite, environment routes, run
control, recovery, heartbeat process recovery, instance settings, the
idle-sleep, eviction, release-resume, restart-recovery and
warm-checkpoint integration suites pass (11 files, 2471 tests).
`tests/restart-chaos` unit test `warm-shutdown` passes. Server and
harness `tsc --noEmit` pass.
- Daytona warm waves (Claude, W2/W2Q/W2C, K0, 8 tasks). The runner image
`sha256:e04d9ce4…` matches
|
||
|
|
e01ba16bfd |
fix(native): continue the interrupted turn after a relaunch that failed before its runner existed (#20944)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native agent runs survive a server restart: recovery verifies the old runner stopped, then resumes the same provider session and its interrupted turn > - Recovery is a chain of steps. When one step fails once, the next attempt must still be allowed to finish the work > - Here the first recovery step was correct, its runner spawn failed once (EAGAIN), and the next attempt refused to continue the interrupted turn. The run died and needed a manual Retry > - This pull request lets the scheduled same-run resume prove that the failed launch left nothing behind, and then continue the turn > - The benefit is that a single failed spawn during recovery no longer turns a recoverable lost turn into a terminal failure ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T190113-s5303` (local, codex, A1, fault K1 with `--fail-recovery-step runnerd-start:1`) on master `683fd1e9a5`. Run timeline: 1. K1 killed the server mid-turn. Recovery recorded `native.local_process_stopped` for the dead runner and chose `resume_dead_runner`. 2. That attempt wrote `native.process_start_requested`, then the spawn failed with `spawn paperclip-runnerd EAGAIN`. The executor persisted a scheduled same-run resume (`recoveryMode: exact_checkpoint_resume`, `providerSessionEstablished: true`, `checkpointExists: true`, `recoverable: true`). 3. The scheduled attempt resumed the session. The runner reported `provider_turn_lost_on_restore` (recoverable). The run ended `native_provider_terminal_failed` with "Automatic recovery stopped", and the task needed Retry. The same seed with K1 and no injected step passes. Root cause: in `executePaperclipNativeSession` (`native-session-executor.ts`), `resumeInterruptedTurn` is `restartRecovery.kind === "resume_dead_runner" || scheduledResumeContinuation`. The second attempt is not a restart claim, so it depends on `scheduledResumeContinuationAllowed` (`scheduled-resume-continuation.ts`). That decision refused with `local_stop_unverified`: `readNativeLocalProcessStop` reads the newest launch event, and the newest one was the failed `process_start_requested` with no process identity and no stop receipt. The checkpoint evidence was correct; the stop evidence was the gap. The refusal was not logged. **Expected behavior** When a launch fails before its runner exists, a scheduled same-run resume proves on this host that no runner for the run exists and that the launch wrote no runner state, records that stop, and continues the interrupted turn. A live runner for the run still blocks it. **Steps to reproduce** `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex --workloads A1 --faults K1 --fail-recovery-step runnerd-start:1 --fault-after-seconds 10 --concurrency 1 --tasks 1 --seed 5303` ## What Changed - `scheduled-resume-continuation.ts`: when the decision refuses only for `local_stop_unverified` on a local target, `scheduledResumeContinuationAllowed` calls the existing `recordUnidentifiedLocalRunnerStop` (the same proof explicit continuations use). It requires that the newest event is this host's launch request, that no process carries `--run-id <runId>`, and that no runner state for the run changed after the launch request. On success it records `native.local_process_stopped` (`basis: unidentified_runner_absent`) in the claim transaction and decides again. All other checks (exact checkpoint, recoverable, session established, checkpoint exists, no replacement, admitted attempt) still apply first. - `native-session-executor.ts`: logs the refusal reason when a scheduled `exact_checkpoint_resume` will not continue the interrupted turn. - `native-scheduled-resume-continuation.integration.test.ts`: two cases. A failed relaunch after a verified stop now continues the turn (`resumeInterruptedTurn: true`, attempt 2). A failed relaunch while a process with `--run-id <runId>` is alive still refuses. The fixture now advances `nextEventSeq` after inserting events directly. - `tests/restart-chaos/README.md`: records what the existing `runnerd-start:1` cell found and what now passes. ## Verification - RED on master: "continues the lost turn after a relaunch that failed before its runner existed" fails with `expected [ false ] to deeply equal [ true ]` (the runtime gets `resumeInterruptedTurn: false`, which is the path that produced `provider_turn_lost_on_restore` → `native_provider_terminal_failed`). - GREEN: `scheduled-resume-continuation`, `native-scheduled-resume-continuation.integration`, `native-exact-resume-failure`, `native-restart-recovery`, `native-runner-restart-recovery.integration`, `native-session-executor`, `native-session-resumption`, `heartbeat-native-restart-suspension` pass (1797 tests). `pnpm --filter @paperclipai/server typecheck` passes. - Chaos rerun of seed 5303 with `runnerd-start:1`: result is added in a comment on this PR. ## Risks - The proof is host-local: it only applies on the host that wrote the launch request, the same as explicit continuations. Remote targets and restart claims are unchanged. - Old saved state: a run already stuck in a scheduled resume after a failed spawn now continues on its next attempt instead of failing. That is the intended recovery; the checkpoint and attempt limits still bound it. - Sibling recovery steps (`teardown`, `directory-lock`) are Daytona-only. The saved waves that armed them (`s5311`, `s5312`) never reached the step, so this PR does not change or prove them. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d9d5958d09 |
fix(runner): reject an out-of-order controller command instead of exiting (#20946)
## Thinking Path - Paperclip's controller and the sandbox Runner exchange numbered commands; the Runner refuses a command whose number is not exactly one past the last it processed. - Staging's most common "Runner gone" failure was a follow-up controller whose journal was behind the Runner. #20928 removed the cause on the server (a warm follow-up now launches from the journal its retired predecessor saved) and added a defence that reconciles a journal behind only by the predecessor's settled close. - The Runner still treated an out-of-order command as fatal: `durable/state.rs` returned an error and the process exited. A controller with a stale journal from any other cause (state saved by an older release, a mid-run deploy) therefore lost the Runner process and the run could only fail as "Runner gone" (`runner_remote_process_identity_unavailable`). - This pull request makes the Runner refuse such a command with a precise result instead of exiting, and makes the server rest that run with a precise code and a Retry whose next run reconciles the journal (the #20928 defence) before it opens the session. ## Linked Issues or Issue Description - Refs #20928 (the server-side root-cause fix and the journal reconciliation this relies on) - Refs #20907 (makes runnerd exit text visible in the run log) - Duplicate search: searched PRs for "controller sequence contiguous", "controller_sequence_mismatch" and "out-of-order command"; none changes the Runner's handling. ## What Changed - `runner-core/src/durable/state.rs`: a new command whose `controllerSeq` is not `lastControllerCommandSeq + 1` (behind or a gap) now yields `CommandDisposition::Reject` with `status: "rejected"` and `{ code: "controller_sequence_mismatch", message, expectedControllerSeq, receivedControllerSeq }`. The command is not executed or journaled and the last sequence is unchanged; the process keeps running. Replay of an already-processed command and the compacted-history rejection are unchanged. - `native-session-executor.ts`: a failure carrying `controller_sequence_mismatch` is classified as `native_controller_journal_behind_runner`, a permanent (terminal) failure with a Retry action that says no provider work began and that Retry reconciles the journal from the saved checkpoint or stops with a precise error. - Tests: - Rust `tests/durable_recovery.rs`: out-of-order (behind and gap) commands are rejected with the precise code, the journal is unchanged, and the next contiguous command executes. The existing `command_gaps_and_identifier_reuse_fail_closed` unit test now asserts the rejection instead of an error. - Real runnerd `src/live/runnerd-stale-controller-journal.test.ts`: a follow-up controller with a journal missing the predecessor's close now gets `"code":"controller_sequence_mismatch"`, the Runner process stays alive and its durable sequence is unchanged; with the current journal it resumes. - Server: the classification and terminal disposition of the new code. ## Verification - Red (master Runner): the Rust test panics on `expect("an out-of-order command is not fatal")`; the real-runnerd test receives `runnerd exited unexpectedly with code 1: paperclip-runnerd: controller sequence must be contiguous: expected 6, received 4`. - Green: `cargo test -p paperclip-runner-core --test durable_recovery` (4 passed); `cargo test -p paperclip-runner-core` lib suite (the only other failure, `task_memory::tests::monitor_bounds_notices`, is a timing test that passes alone); `npx vitest run src/live/runnerd-stale-controller-journal.test.ts` with a runnerd built from this branch (2 passed); server `native-session-executor.test.ts` focused tests (71 passed). - `pnpm --filter @paperclipai/paperclip-runner check:acpx-profiles`: profiles match the release manifest. No ACPX profile, attestation, pin or pack contract schema changed (`rules/acpx-profile-identity.mdc`): the change is Runner source behaviour with an existing command-result shape, so no profile version or `commandDigest` changes. ## Risks - A controller that is *ahead* of the Runner (a gap) is now rejected instead of fatal. Neither case executes the command; the Runner journal is unchanged, so this is strictly safer than exiting. - The rested run needs a Retry. Before this change the same run failed after its retries as "Runner gone" with no precise cause. - Runner source changes the Daytona image content ID; a new image is needed for chaos waves on this revision. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), 1M-token context, tool use (shell, file edits, tests) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6a43890326 |
Fix image attachments in remote native workspaces (#20939)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native runs must receive the files that users attach to their messages. > - Remote wake preparation returned file metadata without readable bytes. > - The historical chat reader also rejected every remote target. > - Command-backed sandboxes and SSH targets already provide a bound command transport. > - This pull request stages verified files through that transport and clears them after the turn. > - Agents can then open attached images with their normal image reader. ## Linked Issues or Issue Description Refs #19740. This change covers SSH and command-backed sandbox execution, including Daytona. Plugin Runner tasks without a command transport still need a separate versioned file-transfer contract. This PR does not close that broader issue. The open attachment-recovery PR #20778 concerns provider-session attachment journals, not input files. **What happened?** A user uploaded an image and sent it to an agent. The remote native run received its filename and size, but no readable file. Wake preparation reported `remote_workspace_staging_unsupported`. The historical read fallback also refused remote execution. **Expected behavior** Copy the exact admitted attachment into the bound remote workspace. Let the provider read the image. Preserve company, task, source-comment, size, and hash checks. Clear temporary file contents when the turn ends. **Steps to reproduce** 1. Select a native agent on a command-backed sandbox or Linux SSH environment. 2. Attach a PNG to its current task or in-app chat message. 3. Ask the agent to inspect it. 4. Check that wake preparation supplies a readable workspace-relative image path. ## What Changed - Add a remote staging owner with bounded commands, stdin transfer, hash verification, opaque paths, and safe image extensions. - Confine Linux filesystem operations through directory descriptors. Refuse symlinks and replacement files during cleanup. - Bind current wake files and authorized historical external-chat reads to the selected lease transport. - Join active writes during cancellation. Retain failed cleanup work until scope close. - Tell providers to open image files with an image-capable reader. - Handle stdin stream errors in the shared SSH transport. Failed uploads return failure instead of raising an unhandled EPIPE or reporting success. - Add database, transport, cancellation, integrity, cleanup, executor-wiring, and Linux filesystem regressions. Document support and limits. ## Verification - Prepared an isolated source checkout with `node scripts/agent-preflight.mjs --skip-rust-debug`. - Focused suites: `pnpm exec vitest run server/src/services/native-runtime/remote-attachment-staging.test.ts server/src/services/native-runtime/native-runner-file-handoff.test.ts server/src/services/native-runtime/chat-attachment-read.test.ts` — 74 passed, 4 Linux-only tests skipped. - Executor checks: `pnpm exec vitest run server/src/services/native-runtime/native-session-executor.test.ts -t 'attachment staging to the remote|stages from the authenticated|verifies frozen staged admission'` — 7 passed. - Server typecheck: `pnpm --filter @paperclipai/server typecheck`, followed by the server layered check after the final changes. - `git diff --check` passed. The local secret, PII, and private-reference diff scan found no matches. - `pnpm verify` returned exit 2 because the documentation change requires broad CI. No full local suite is claimed. - The four Linux filesystem tests were skipped locally because this macOS host has no available Docker daemon. Linux CI executed all 17 remote-staging tests successfully, including those four confinement tests: [server shard 10](https://github.com/paperclipai/paperclip/actions/runs/43102035096/job/130796646266). - SSH regression: the real pipe test reproduced two unhandled EPIPE errors before the fix, plus false success when the peer exited with code zero. The fixed SSH tests and existing argv tests pass (12 tests). The remote-staging transport suite also passes (13 tests; 4 Linux tests require CI). - `pnpm --filter @paperclipai/adapter-utils typecheck` passed after the SSH change. - Final revision `1c5a9288b8d5f267ddeac84638ff6075dbbbf8ec`: all applicable CI jobs passed, including typecheck, build, general and serialized tests, Runner checks, canary dry run, and all eight browser shards. [CI run](https://github.com/paperclipai/paperclip/actions/runs/43102035096). Linux CI also passed all 6 native SSH tests: [server shard 5](https://github.com/paperclipai/paperclip/actions/runs/43102035096/job/130796646349). - Greptile reviewed that final revision at 5/5 with no remaining findings. The review thread is resolved. GitHub reports the PR mergeable with a clean merge state. - No live model turn or deployed release is claimed. ## Risks - Remote staging requires Linux, Node, and a command transport bound to the selected lease. Plugin Runner tasks without that transport retain their unavailable result. - Each file uses bounded independent commands. Slow transports can delay preparation or cleanup. File bytes stay out of command arguments and provider-facing error messages. - An abrupt controller or remote-host failure can bypass cleanup. Ephemeral sandbox disposal remains the backstop. Retained files are never blindly overwritten or deleted. - Previous-release provider sessions and checkpoints retain their format and identity. New turns receive newly staged input files. Existing restart-admission checks still apply. No migration, runner-image change, or account-credential transfer is required. ## Model Used OpenAI `gpt-6.1-sol` through the Codex harness in T3 Code, with xhigh reasoning, repository inspection, code execution, and tests. No subagents were used. The context-window size is not exposed by this runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
757a20d2f1 |
fix(runner): keep remote recovery and launch through a short plugin RPC stall (#20945)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote run starts `paperclip-runnerd` in a sandbox. The controller
reads and controls it through the environment plugin
(`environmentExecute`).
> - #20940 made a stalled supervision probe "unknown" while the Runner
stays connected. Other callers still turn the same plugin stall into a
terminal failure.
> - During restart reattachment, the saved-state read, the
identity-marker read and the process check share a 30-second budget. One
plugin RPC that stalls for 40 seconds uses all of it, so a short stall
fails a reattachment to a healthy Runner.
> - At launch, a stalled launch call or identity read fails the run with
`runner_process_termination_unconfirmed` or
`runner_remote_process_identity_unavailable`, although the runner may be
healthy.
> - This pull request treats a plugin RPC timeout as "unknown" and
retries the read-only call within a bounded stall budget. A stalled
launch call is not repeated: the controller reads the launch nonce's
identity marker instead.
> - The benefit is that a short plugin stall during recovery or launch
no longer fails a healthy run. Every positive answer from the sandbox
keeps its meaning, and the bound still fails closed.
## Linked Issues or Issue Description
Refs #20940, #20909
**What happened?**
A plugin `environmentExecute` RPC stall of more than 40 seconds makes a
remote recovery read fail. Plugin RPCs time out at the requested timeout
plus a 30-second overhead, so a 10-second probe can take 40 seconds. The
reattachment read budget is 30 seconds. One stalled read therefore
failed reattachment with `probe_transport_failed` or `probe_timeout`. At
launch, a stalled launch call rejected the run with exit unconfirmed. A
stalled identity read (20-second wait, 2-second reads that can take 32
seconds) failed the run with
`runner_remote_process_identity_unavailable`.
**Expected behavior**
A plugin RPC timeout is a stall of the plugin, not an answer from the
sandbox. Read-only recovery reads retry within a bound. A launch call
that stalls is not repeated. The launch nonce's marker decides whether
the one runner was launched. At the bound, the run fails closed and
Retry works.
**Steps to reproduce**
1. Start a remote (Daytona) run.
2. Restart the controller so that the run reattaches, or start a new
run.
3. While the controller reads the sandbox, stop the plugin worker for 45
seconds.
4. The reattachment or launch fails.
**Paperclip version or commit**
master at
|
||
|
|
48d23d1530 |
fix(runner): keep a connected remote Runner through a stalled plugin supervision probe (#20940)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote run starts `paperclip-runnerd` in a sandbox. The controller
monitors the process with a health probe that goes through the
environment plugin (`environmentExecute`).
> - A stall of the Daytona plugin worker makes that probe RPC time out
after 40 seconds. The monitor treated the timeout as "process
supervision failed".
> - The transport then failed closed with
`runner_process_termination_unconfirmed`. The run ended
`notification_transport_failed`, `recoverable: false`, and the close
path suspended a healthy Runner. The task rested blocked with Retry.
> - Nothing was wrong with the sandbox, the Runner or the provider. The
Runner held its own authenticated connection to the controller during
the full stall.
> - This pull request makes a failed or timed-out probe "unknown". While
the Runner stays connected on its own channel, the monitor keeps the
same runner and turn and probes again, for a bounded time.
> - The benefit is that a short plugin stall no longer kills a healthy
run. The fail-closed cases stay the same.
## Linked Issues or Issue Description
Refs #20909 (chaos cell CR3, which reproduces this failure)
**What happened?**
A 45-second stall of the Daytona plugin worker killed a healthy run.
Chaos cell CR3 reproduced it 2 of 2 times (OpenCode and Claude,
Daytona). A staging run had the same failure with no deploy: a plugin
`environmentExecute` RPC stalled for more than 40 seconds. The run
failed with "Notification transport failed closed:
runner_process_termination_unconfirmed: runnerd process supervision
failed: RPC call \"environmentExecute\" timed out after 40000ms". The
close path then sent `runner.suspend`, and the healthy Runner suspended
and exited. The task rested blocked with Retry.
**Expected behavior**
A probe that times out or fails in transport is not proof that the
process stopped. While the Runner stays connected on its own channel,
the run continues. If supervision cannot come back within a bound, the
run rests with a working Retry. A probe that positively reports the
process gone still starts the existing recovery.
**Steps to reproduce**
1. Start a remote (Daytona) run with a T1 workload.
2. Send SIGSTOP to the server's Daytona plugin worker for 45 seconds,
then SIGCONT.
3. The run fails `notification_transport_failed` about 40 seconds into
the stall.
**Paperclip version or commit**
master at
|
||
|
|
884a9e19ea |
fix(native): launch a warm follow-up from its predecessor's saved journal, and restart a Runner that exited before session.open (#20928)
## Thinking Path - Paperclip runs agent work through a controller (server) and a runner (`paperclip-runnerd`) that exchange numbered commands; the runner refuses any command whose number is not exactly one past the last it processed. - Staging's most common failure is "Runner gone" (`runner_remote_process_identity_unavailable`, `native.remote_runner_recovery_failed` with `process_not_running`, `runnerLifecycle: ready`, `runnerSettled: false`). Chaos wave `w-20261111T192658-s5201` (Daytona, warm, Codex, W2/W2Q/W2C re-wake cells) captured it live: the new runner wrote `controller sequence must be contiguous: expected 13, received 11` (and `expected 5, received 3`) and exited 0.15 s after authenticating. - In all three captured failures the follow-up run started 1–2 s before the predecessor's warm session was closed (`transport_close_requested` for the predecessor arrives after the follow-up's `run started`). The close is issued by the follow-up's own admission. - Admission first materializes the controller projection for the session root. With an idle retained warm owner the root only holds the external-persistence marker, so `restoreWarmRunnerdController` copies that owner's saved checkpoint into the root. Later in the same admission, `executePaperclipNativeSessionWithinScope` finds the owner cannot be reused (configuration, process or GitHub access changed) and retires it. Retirement sends `runner.drain` and `runner.suspend`; both are saved to the owner's checkpoint before they are sent, but the root still holds the projection taken before the close. - The follow-up then opens its controller from that projection. Its journal is two commands behind the runner's durable `lastControllerCommandSeq`, so its first command repeats sequence 11; the runner treats this as fatal and exits, and every later attempt only reattaches to a process that is gone. - So the saved controller journal was persisted correctly; it was not selected: the projection is never refreshed after the retirement that extends it. ## Linked Issues or Issue Description - Refs #20874 (the runner-exit proof this extends from a rejected `session.open` to a Runner that exited before answering it) - Refs #20907 (makes the runnerd exit text, including `controller sequence must be contiguous`, visible in the run log; independent of this fix) - Refs #20853 (related: resumes a remote Runner that died at a settled `ready` cut; this PR covers the unsettled `ready` state left by an exit before `session.open`) - Duplicate search: searched open and closed PRs for "controller sequence contiguous", "runner_remote_process_identity_unavailable" and "Runner gone"; none fixes the stale controller projection. ## What Changed - `server/src/services/native-runtime/warm-controller-journal.ts` (new): - `refreshRetiredWarmOwnerProjection` reloads the retired owner's canonical checkpoint into the session root after retirement, only when the root still projects that owner and the saved journal extends the projected one. A diverged journal holds admission with `native_runner_checkpoint_projection_held:retired_owner_journal_diverged`; a projection changed underneath holds with `...:journal_changed`. - `reconcileControllerJournalWithRunner` is the defence for state already saved in the bad shape (old-saved/new-resumed, mid-run deploys). When the remote runner state shows its journal ahead of the controller projection and the runner's last command is a completed `runner.suspend`, it reconciles only if every extra sequence is a completed `command_close_drain_*`/`command_close_suspend_*` in the runner's processed-command journal and the predecessor's canonical checkpoint holds exactly those commands. Anything else refuses before launch with `native_controller_recovery_missing_evidence:controller_journal_behind_runner` instead of guessing. A runner whose last command is not a completed suspend is left to the existing live-adoption checks. - Defence in depth for the server side of the same failure (`native_session_recovery_failed: runnerd exited while waiting for session.open`, then `runner_remote_process_identity_unavailable` on every same-run retry): a Runner proven to have exited while this attempt's opening command (`session.open`, or the recovery probe) was still unanswered is a failed fresh start, not a live Runner to reattach. - `runnerd-codex-transport.ts`: `runnerExitConfirmed()` (#20874's proof) now also holds for a resumed session whose provider identity was preloaded, when the attempt's opening command is still pending at the proven exit. - The Codex driver's recovery result, `HarnessDriverBackend` and `native-session-runtime` carry that proof on the `native_session_recovery_failed` error. - The executor records `runnerExitedBeforeSessionOpen` on the settled failure; restart recovery puts `exitedBeforeSessionOpen` on the `reattach_remote_runner` claim; `verifyRemoteRunnerOrSettledExit` and `verifyStagedRemoteRunner` then start a new Runner when the process is gone and the saved Runner state has no pending terminal delivery, provider cleanup or warm transition (any lifecycle other than an active turn). Attempt budget and identity checks are unchanged. - `native-session-executor.ts`: calls the refresh right after the configuration-changed retirement of a warm owner, and the reconciliation in `restoreRemoteSuspendedRunnerCheckpoint` with the runner state it already reads (no added remote read, no wait before `turn.start`). - Tests: - `native-session-executor.test.ts`: new `changed-external-checkpoint` scenario of "rotates an owned warm codex remote runner before checking changed configuration" with a real `DurablePrpControlPlane`: the retired owner's close saves drain/suspend, and the journal the follow-up launches from must end with them. - `warm-controller-journal.test.ts`: refresh, divergence hold, untouched marker/other run, reconcile of a settled close, refusal when the canonical journal cannot prove the close, refusal for any other extra command, and no action for a live runner. - `packages/paperclip-runner/src/live/runnerd-stale-controller-journal.test.ts`: real `paperclip-runnerd` + fake Codex. A follow-up controller on the predecessor's suspended runner with a journal that lacks the close commands reproduces `controller sequence must be contiguous` and loses the runner; with the current journal it resumes. ## Verification - Red for the added part: `verifyRemoteRunnerOrSettledExit` with the staging state (process gone, lifecycle `ready`, unsettled outbox) rejects with `runner_remote_process_identity_unavailable` without the change; with `exitedBeforeSessionOpen` it resolves to a fresh start, and stays fail-closed for a pending cleanup, terminal delivery, active turn or exhausted budget. Without the propagation, the backend recovery result lacks `runnerExitConfirmed`; the real-runnerd stale-journal test asserts the transport's exit proof after the contiguity exit (`expected false to be true` on the old transport). - Red on master (executor unchanged, new scenario): `AssertionError: expected [] to deeply equal [ 'runner.drain', 'runner.suspend' ]` — the follow-up launched from the projection taken before the predecessor's close. - Green: `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "rotates an owned warm codex remote runner before checking changed configuration"` (14 passed), `npx vitest run src/services/native-runtime/warm-controller-journal.test.ts` (7 passed). - Real runner: `cd packages/paperclip-runner && npx vitest run src/live/runnerd-stale-controller-journal.test.ts` (stale journal → `controller sequence must be contiguous`; current journal → resumes). - `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter @paperclipai/paperclip-runner typecheck:typescript`; `pnpm verify` selected broader verification (a new runner test file is outside the resolved graphs), so broad coverage comes from required CI. ## Risks - The refresh replaces the root projection only with a checkpoint whose identity matches and whose journal extends the projection; otherwise it holds admission with an existing `native_runner_checkpoint_projection_held:*` code, which is safer than launching a runner that will exit. - The defence refuses launch when a runner is ahead by anything other than a provable settled close. Before this change that launch always ended in a runner exit and a failed run, so the refusal only changes the error code and the moment of failure. - This PR does not change the runner. The runner still exits on an out-of-order new command; a follow-up runner PR will turn that into a rejected command result. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), 1M-token context, tool use (shell, file edits, tests) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
80b2369f26 |
fix(native): keep remote runnerd exit evidence and report it on failed recovery (#20907)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on Daytona start a detached `paperclip-runnerd` in the sandbox, and the controller adopts it after a restart > - On staging, recovery fails with `runner_remote_process_identity_unavailable` when the Runner is gone, but nobody can see why it exited > - The launch script sends runnerd's stdout and stderr to `/dev/null`, so only a panic or a graceful error reaches `runnerd.stderr.log`, and the recovery event does not read that file > - This pull request keeps runnerd's stderr in a bounded file in its private diagnostics directory, and adds the exit reason to the `native.remote_runner_recovery_failed` event > - The benefit is that the reason a Runner exited shows in the run log on staging without SSH, which the class R investigation needs ## Linked Issues or Issue Description **What happened?** Staging runs fail with `runner_remote_process_identity_unavailable` after the remote Runner exits (`native.remote_runner_recovery_failed` with `stage: process_verification`, `reason: process_not_running`). The event does not say why the Runner exited. **What did you expect?** The run log names the Runner's exit reason: its durable `recoverableFailure` and diagnostics, and what it wrote to stderr. **Steps to reproduce** Restart-chaos re-wake and stall cells reproduce the refusal. Each failed run log shows only `runnerLifecycle` and `runnerSettled`. Refs #20893 ## What Changed - `REMOTE_RUNNER_CHILD_LAUNCH_SCRIPT` sends runnerd's stderr to `diagnostics/runnerd.process-stderr.log` (mode 0600, in the 0700 directory) instead of `/dev/null`. - At launch, a file above 1 MiB is rotated once to `.1`. - While runnerd runs, a detached trimmer is a disk backstop. Past 8 MiB, it keeps the last 1 MiB. It checks every 15 s and ends when runnerd ends. It is double-forked with its own argv, so it is not a child of runnerd and does not carry runnerd's arguments. runnerd's own exit reason goes to `runnerd.stderr.log`, which is never trimmed. - A symlink or non-file at that path keeps the old `/dev/null` behavior. - Stdout is unchanged (`/dev/null`). Runnerd's PID is still the launch shell's PID, so the identity marker and fingerprint are unchanged. - `remote-runner-recovery-diagnostics.ts` has a read-only `REMOTE_RUNNER_EXIT_EVIDENCE_SCRIPT`. It prints a 2 KiB tail of `runnerd.stderr.log` and of `runnerd.process-stderr.log`. It reads the session diagnostics directory first, then the shared runtime-root directory used by external Runners (Dot, Muse, grok_bot). It never reads through a symlink. Each file keeps its own share of the 4 KiB budget. - When recovery's process verification finds `process_not_running`, the diagnostic adds these fields: - `runnerRecoverableFailure`: a code-shaped string from `runner-state.json`; - `runnerDiagnostics`: the last 4 entries, redacted, 500 characters each; - `runnerExitEvidence`: the redacted file tails, 4 KiB at most. `native.remote_runner_recovery_failed` carries them. - The live process monitor's exit report now puts the durable state summary first, then the file tails. Before, it used one or the other. - `run-failure-diagnostics.ts` is unchanged. It still copies only the closed labels, so no free text reaches failure reports. ## Verification - `pnpm exec vitest run server/src/services/native-runtime/remote-runner-launch-stderr.test.ts`: the launch writer keeps stderr in the file, writes the identity marker with the child's PID, appends across relaunches, rotates an oversized file, trims a growing file while the child runs, and never writes through a symlink. - `pnpm exec vitest run server/src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts`: the evidence script output is bounded and refuses symlinks. Redaction and bounds hold, and the new fields survive the event payload. - `pnpm exec vitest run server/src/services/native-runtime/native-session-executor.test.ts` passes (1374 tests). One expectation was updated for the combined exit report. - `npx tsc --noEmit -p server/tsconfig.json` is clean. - Older runner images: the script is server-side. It adds only `wc`, `tr`, `tail`, `cat`, `mv`, `sleep` and `kill`. The script already needed `node`, `date`, `chmod`, `mv` and `printf`, so any image that launched runnerd before has these tools. The runner content ID is unchanged, so the current Daytona image runs it unmodified. The restart-chaos waves run on that image. ## Risks - Low. A sandbox disk that is already full could make the stderr redirect fail. `exec ... 2>>file` then fails, and the launch reports `runner_remote_process_identity_unavailable` the same way a launch that cannot write its identity marker does. The marker is written first, in the same directory tree. - Stderr content stays in the sandbox. Only a redacted 4 KiB tail goes into the instance's own run log (run-log data path; not Telemetry and not Observability). - No wait is added before `turn.start`. The evidence read runs only on the recovery failure path, with its own 10 s bound. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
683fd1e9a5 |
test(restart-chaos): cells for staging's no-restart failures (re-wake, Nth recovery step, long turn, K15 without volume) (#20893)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos/`) tests that agent work survives control-plane faults > - In one 8-hour staging window, 20 of 25 failed runs happened with no restart at all, so the harness did not see them > - The harness had blind spots: one short task per agent, no wake right after a successful run, no fault in a later recovery step, short turns, and a redeploy that always keeps its instance home > - This pull request adds cells for those shapes, so the harness reproduces these failure classes on master > - The benefit is that fixes for these classes can be proven red and green with the harness, without staging ## Linked Issues or Issue Description **What happened?** On staging, an agent ran 10–13 tasks at once on reused sandboxes. 30–60 s after a long run succeeded, an ordinary wake (a comment or an issue monitor) started the next run of the same task. That run or its recovery failed before any provider work, with errors such as `runner_remote_process_identity_unavailable`, `Controller journal drain timed out` and `journal_tool_input_unanswered`. Other failures happened in a later recovery step (a held agent-directory lock, a missing runtime asset after a deploy without a volume). **Expected behavior** The chaos harness can reproduce each of these shapes on demand, and grades them: no failed attempt, every task `done`, and no task left blocked without a working Retry. **Steps to reproduce** See the commands in the README sections "Re-wake after success", "Fail the Nth recovery step", "Long turn" and "Redeploy without a volume, with overlap". Related: #20844 (session fault SR1). W2C is the configuration change after a successful run; SR1 is the same change while a question is parked. This PR does not add or change `session-faults.ts`. ## What Changed - New workloads W2, W2Q, W2C (`rewake.ts`): phase 1 ends with the task left open; the harness, as the board, wakes the same task the seeded 0–60 s after its run settled. W2 uses a comment or a board-scheduled issue monitor (by nonce). W2Q parks on an unanswered `request_human_input` question and is woken by a comment before the answer. W2C changes the agent's `adapterConfig.idleTimeoutMs` before the wake. Oracle: phase 1 succeeded, the wake was served by a run that succeeded, `no_failed_attempt`, `done`, markers. - `--faults K0`: cells with no fault, for workloads whose shape is the fault. - `--fail-recovery-step <step>[:<nth>]` (`recovery-step-faults.ts`): when the wave's first restart fault starts its replacement, the harness arms one crash point in a new fail mode; the nth hit throws the error of the real step, once. Steps: `runnerd-start` (`spawn paperclip-runnerd EAGAIN`), `teardown` (provider destroy did not finish), `directory-lock` (`Agent directory operation is already active`). Crossed tasks get `recovery_step_failed_once` and `no_failed_attempt`. - Crash points (`packages/shared/src/crash-points.ts`): fail mode in the control file (`<token> <name>:<nth> fail [pid=<n>]`), with an optional call-site error; `pid=` binds the line to one server process. The harness arms the step for the replacement only and disarms it once its receipt lands. New point `agent_directory.operation_lock.before_acquire`. The teardown point moved into its `try`, so fail mode behaves like a failed provider destroy; kill mode is unchanged. All points stay disarmed under `NODE_ENV=production`. The runner package's own crash-point copy is not changed (runner image content ID unchanged). - `--long-turn-minutes <n>` (`long-turn.ts`, 15–60, opt-in): every task's first turn runs rounds of real tool work for n minutes before its workload. Checks `long_turn_rounds` and `long_turn_run_observed`. - K15 `--k15-fresh-home` (Daytona) and `--k15-overlap-seconds <min>-<max>` (`redeploy.ts`, `supervisor.ts`, `edge-proxy.ts`): each deploy generation gets a new, empty instance home and no volume variables; the next generation boots beside the old one behind an edge proxy on the wave's port, takes traffic once ready, and the old one gets SIGTERM after the overlap. - `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pins for the changed shared crash-point files. - README: one section per new cell, with what it covers. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts` (48 files, 581 tests after rebasing on master, includes the new `shape-cells.test.ts`). - `pnpm exec vitest run packages/shared/src/crash-points.test.ts server/src/__tests__/crash-point-call-sites.test.ts` (fail mode, call sites). - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`. - Chaos waves on master `1d67e309e0` server code (Daytona image `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:99fc2671fc320816407f63cffd3acbd27471621febcd26494fe25bb5d7c5dd77`, content ID `1836c515…`), all RED unless noted: | Cell | Wave | Result on master | Staging class | | --- | --- | --- | --- | | W2/W2Q/W2C, 10 tasks on one Codex agent, `--daytona-warm`, K0 | `w-20261111T163206-s5201` | 9/10 fail. 3 tasks (W2 t02, W2C t03, t06): first fault `native_runner_process_exited`, same-run reattach, `native.remote_runner_recovery_failed` with `stage: process_verification`, `reason: process_not_running`, `runnerLifecycle: ready`, `runnerSettled: false`, terminal `runner_remote_process_identity_unavailable`, "Workspace repair required". 3 tasks (t04, t08, t10): `native_restart_recovery_blocked` (`native_retry_source_superseded`) while its successor ran (R4 `concurrent_runs`). 3 W2Q tasks: the answer's continuation was cancelled `issue_not_in_progress` (status `blocked`), and the task needed Retry. 2 W2C tasks then looped: 60+ succeeded runs in 20 minutes on an `in_progress` task. | D′/T terminal step reproduced with no restart, no governed wait and no PRP timeout (same error code and refusal fields). C+ retry outcome (ownership ambiguous) reproduced as `native_restart_recovery_blocked`. Not reproduced: `Controller journal drain timed out`, `did not durably suspend`, `journal_tool_input_unanswered`. | | Same, local `--lifecycle warm` | `w-20261111T160103-s5101` | 4/10 fail. 3/3 W2Q: continuation cancelled `issue_not_in_progress`, Retry needed. 1 W2C: the same `in_progress` run loop (39+ runs). | The remote-runner classes need a remote runner; locally they do not show. | | `--fail-recovery-step runnerd-start:1`, K1 at 10 s, local A1 | `w-20261111T162432-s5303` | Fail. After K1 the first runnerd start of recovery failed (`spawn paperclip-runnerd EAGAIN`); the run ended `native_provider_terminal_failed` ("previous turn is no longer active", "Automatic recovery stopped") and needed Retry. Control, same seed and K1 without the step fault (`w-20261111T171114-s5303`): pass. | A later recovery step that fails once ends the run instead of trying again (the A-chain shape). | | `runnerd-start:2`, local | `w-20261111T170909-s5304` | Step never reached: recovery started the runner once and succeeded. Grade fails `recovery_step_failed_once` (tested nothing). | — | | `teardown:1`, K1, Daytona A1 | `w-20261111T172048-s5312` | Pass. The teardown failed once and the sweep settled it. | Not reproduced. | | `directory-lock:1`, K2, Daytona A1 | `w-20261111T171247-s5311` | Step never reached by A1 after K2. | Not reproduced; A1 recovery does not collect remote agent files. | | K15 `--k15-fresh-home --k15-overlap-seconds 30-150`, `--storage s3`, Codex and OpenCode, T1/C1 | `w-20261111T173618-s5322` | 1/4 fail (OpenCode T1): `native_continuation_requires_reconciliation` with "Agent directory belongs to a different execution environment"; the Retry runs failed `adapter_failed` "Configured instruction entry is missing"; the task rests `blocked` on a board recovery action, and two sandboxes were left (R6). One Codex T1 task finished before its fault (`fault_crossed_tool`). | A-deploy: a controller-local file missing after a deploy without a volume. Here the missing file is the agent's instruction bundle, not a runtime-context asset; whether staging keeps instruction bundles outside the instance home is not verified. | | `--long-turn-minutes 15`, K1 at 600 s, local A1 | `w-20261111T175505-s5401` | Task pass (10/10 rounds, 992 s span, run 1019 s, K1 crossed mid-turn). The wave exited 2 only because I switched branches in the worktree during the wave. | Shape check only. | ## Risks - Product code changes are test instrumentation only: three `crashPoint` call sites take a fail-mode error, and one new call site was added. Disarmed (always in production) each call is one comparison. Moving the teardown point into its `try` does not change kill mode: the process dies at the same instant. - The new cells are opt-in. Default waves plan the same tasks as before for the same seed. - The W2 cells depend on model behaviour (phase 1 leaves the task open). The oracle records the task status at the wake. - `--k15-fresh-home` drops everything under the instance home. Whether staging keeps agent instruction bundles outside the instance home is not verified (see the K15 result). ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with tool use (shell, file edits, test and chaos runs) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8eabb13ebd |
fix(recovery): retry a system-woken run once when it lost its runner before any turn (#20873)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a remote sandbox can lose its Runner process before
its same-run retry reattaches. The run then fails with
`runner_remote_process_identity_unavailable`
> - After saved-work recovery, the server requests one Retry by itself,
but only for user wakes. System wakes (monitors, automatic assignments)
rest on the visible Retry
> - On staging, a monitor wake's run lost its Runner before it started
any provider turn. Nothing it did could be repeated, but the task stayed
blocked until a person clicked Retry
> - This pull request lets that one case continue automatically, once,
under the run's responsible user
> - The benefit is that a system-woken task whose run never reached a
provider turn continues without a person
## Linked Issues or Issue Description
**What happened?**
On staging, a monitor wake (`issue_monitor_due`) started a remote native
run. Its `session.snapshot` command timed out before any turn. The
same-run retry found the Runner process gone (`process_not_running`,
lifecycle `ready`) and the run failed with
`runner_remote_process_identity_unavailable`. The run had no turn event
and no tool event. Saved-work recovery recovered the files, but
`requestInterruptedNativeTurnRetry` refused with `not_user_request`, so
the task stayed blocked on a manual Retry.
**What did you expect to happen?**
When the run proves that it did no provider turn, the server requests
the Retry once by itself, as it does for user wakes.
**Steps to reproduce**
1. A system wake starts a remote native run.
2. The run fails before any turn event, and the same-run retry finds the
Runner gone.
3. Saved-work recovery completes. No Retry is requested; the task waits
for a person.
Refs #20872, #20853
## What Changed
The rule that changed, exactly: in `requestInterruptedNativeTurnRetry`
(`server/src/services/heartbeat/restart-interrupted-turn.ts`), a run
whose wake was not requested by a user used to stop with
`not_user_request` in all cases. Now it requests the task Retry only
when all of these are true:
- the run failed with one of the existing reattachment-loss codes
(unchanged gate);
- a positive durable record proves that no turn was ever issued: the
run's last `native.remote_runner_recovery_failed` event carries
`remoteRunnerRecovery` with `stage: process_verification`, `reason:
process_not_running` and `providerTurnIssued: false`;
- the run also recorded no turn event and no tool event;
- the run has a `responsibleUserId`; the Retry runs under that user;
- the run is not itself a `task_retry` run;
- the task is still open and assigned to the same agent (unchanged
gate).
An absence of run events is never enough: a turn can reach the provider
before its events are stored (Greptile, first review).
- New `withRemoteRunnerTurnProof` (`native-session-executor.ts`): when a
same-run reattachment is refused because the sandbox proved the Runner
process gone (exit 45, not an unreachable sandbox), the executor reads
the run's controller journal. The controller records each turn command
there before it sends it. Only when
`runnerdStateProvesNoRemoteProviderTurn` holds (just
`run.prepare`/`session.open` and maintenance commands such as
`session.snapshot`, and only session-open events) does it add
`providerTurnIssued: false` to the diagnostic. This is the same proof
that restart recovery already uses for `bootstrap_incomplete`.
`RemoteRunnerRecoveryDiagnostic` keeps the field only as `false`.
- Which staging shapes this covers: a run whose journal holds no turn
command. Whether c14bab19 (the `session.snapshot` timeout) qualifies
depends on its controller journal, which staging does not expose; if its
journal held a `turn.start`, it stays on the manual Retry floor.
32376b3a (`turn.submitted`, then a `turn.start` timeout) always stays on
the floor.
- Everything else about system wakes is unchanged: without the journal
proof, with any turn or tool event, or without a responsible user, a
system wake still rests on the visible Retry.
- Bounded: the Retry uses the existing idempotency key
`restart-interrupted-turn:<runId>`, so it is requested at most once per
run. Its successor is a user-attributed `task_retry` run, so if that run
also loses its Runner, the existing `automatic_retry_used` rule leaves
it on the visible Retry. No loop is possible.
- Attribution: `requestTaskRetry` records `issue.retry_requested` with
actor `restart-recovery` on behalf of the run's responsible user.
Admission keeps every gate (saved-work recovery, stop proof, approvals,
pause, budget, ownership).
- Tests in
`server/src/services/workspace-recovery-retry-copyback.test.ts`: a
positive case (monitor wake, Runner proven gone, journal proof),
including the second-loss `automatic_retry_used` check. Negative cases:
no run events but no journal proof (the Greptile shape), a settled turn,
a tool event, no responsible user.
- New
`server/src/services/native-runtime/remote-runner-turn-proof.test.ts`:
the proof is recorded only for a journal of session commands; not for a
`turn.start` command, a committed turn event, an unreadable journal, or
an unreachable sandbox (`probe_transport_failed`).
- `doc/execution-semantics.md`: describes the rule.
## Verification
- `cd server && pnpm exec vitest run
src/services/workspace-recovery-retry-copyback.test.ts`: 83 passed.
- Red on master (product hunk removed): the positive case fails with
`expected { requested: false, … } to match object { requested: true }`.
- Red on the first head (
|
||
|
|
566a97b882 |
fix(native): an unretirable predecessor sends the next attempt to a fresh sandbox (#20863)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a reused sandbox keep the previous run's warm Runner, and a new run must retire that owner before it starts > - #20844 retires the owner when Paperclip can prove it stopped. Some owners can never be proven stopped: the controller journal drain outlives the close grace, the Runner never confirms suspension, or a parked question leaves a tool input unanswered > - Then the new run failed before any provider work, and the task stayed blocked with no automatic continuation > - The general answer is to leave the old Runner on its sandbox and start the next attempt on a fresh sandbox restored from the saved workspace > - This pull request adds that fallback for every path that retires a predecessor, and settles a parked question's tool input once its Runner is gone > - The benefit is that an unprovable predecessor no longer fails a task that did no work ## Linked Issues or Issue Description **What happened?** An automatic wake started a new run 30 to 60 seconds after a long successful run on the same task, on the same reused sandbox. The new run had to retire the previous run's warm session first. The retirement failed and the run failed before any provider work, with one of: - `Controller journal drain timed out; preserve the session before replacing its writer.` (during instruction preparation, before runtime selection, or in the executor); - `provider_transport_failed: runner did not durably suspend before checkpoint` (during instruction preparation); - `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=journal_tool_input_unanswered` (the previous run had parked on a question). The tasks rested `blocked` with a reconciliation hold or with nothing scheduled. One bounded retry reused the same sandbox and ended `execution_finalization_deadline_exceeded`, then its recovery refused with `allocation_ownership_ambiguous`. **Expected behavior** A run that did no provider work does not fail the task because its predecessor cannot be retired. The next attempt drops the reused sandbox and starts on a fresh one restored from the saved workspace. Unsaved work is never dropped: when the workspace or the session backup is not durably saved, the sandbox is kept and today's behavior applies. **Steps to reproduce** 1. Complete a warm native run on a Daytona sandbox. 2. Wake the task again so that the next run must retire the warm owner (configuration change, new session, changed agent files). 3. Make the owner's close fail (journal drain timeout, unconfirmed suspension). 4. The new run fails before provider work and the task is not continued. Refs #20844, #18373. ## What Changed - `native-runtime/predecessor-retirement.ts`: a typed pre-provider error `NativePredecessorRetirementUnprovenError` (`native_predecessor_retirement_unproven`) and the `replaceBeforeReuse` lease marker. - Native executor: every path that retires a predecessor for a new run (instruction-copy claim, instruction-directory reservation, session reset, configuration change) throws the typed error when the close fails. The owner stays registered for its own sandbox, so no run on that sandbox launches over it. A run on a different sandbox forgets that owner instead of replaying its cached rejected close. The executor does not settle this error as a provider failure. - A prior-owner fence that refuses a terminal owner on a resumed sandbox (`prior_owner_terminal_state_indeterminate`) also throws the typed error. - Retirement of a finished remote owner accepts an unanswered tool input when the owner's committed result is a governed-wait park (`response_wake` on a task interaction) and its Runner's sandbox lease ended elsewhere. - Heartbeat (`heartbeat/predecessor-retirement-fallback.ts`): on the typed error with no execution-start evidence and no result, mark this run's reusable lease on that sandbox `replaceBeforeReuse` and log `environment.lease_replacement_requested`. After the run ends, `failRun` hands the prepared workspace on only through the shared unchanged-files proof (`proveRemoteWorkspaceUnchanged`; a provisioning command or any runner command refuses it). With the proof it records `predecessorRetirementFallback`, closes the run's coordinator without a result and re-queues the run in the uncharged `interrupted_preparation` lane (native and legacy runtime modes); a run that answered a task interaction keeps its own wake in that uncharged lane, so the retry still carries the answer. Without the proof the workspace stays dirty, heartbeat runs today's failed-workspace preservation and no retry is scheduled. - Environment runtime: a marked lease is replaced instead of resumed, under the same rule as a failed resume (`reusableSandboxReplacementIsSafe`: run ended, clean workspace with a committed save, verified harness backup). The old lease goes to `pending_cleanup` for the sweep. Otherwise the sandbox is resumed as before. - Chaos: session fault SR1U, the SR1 variant where the retained owner cannot be retired. The server gets a test-only fault (`PAPERCLIP_TEST_WARM_RETIREMENT_FAULT=unconfirmed`, armed only when `NODE_ENV=test`). The oracle allows only the fallback failure and requires the succeeding follow-up on a different sandbox. - Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md`, chaos README. ## Verification - RED on master before this change (with #20844), then GREEN: - `native-session-executor.test.ts -t "predecessor cannot be retired"`: base fails with `Error: Controller journal drain timed out…` instead of the typed error. - `native-session-executor.test.ts -t "unretired predecessor"` (instruction preparation): same base failure. - `native-recovered-owner-followup.test.ts -t "unacknowledged"`: base fails with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=journal_tool_input_unanswered`. - New contract tests: `heartbeat-native-predispatch-retry.test.ts` (claimed native run: lease marked, activity logged, uncharged `interrupted_preparation` retry, no recovery action, task not blocked, retry succeeds) and `environment-runtime-reusable-resume-replacement.test.ts` (a marked lease is replaced without a resume; with unsaved work or no verified backup it resumes as before), `remote-workspace-unchanged-release.test.ts` (an untouched workspace is handed on; a setup-written one stays dirty) and `retries.test.ts` (an answered interaction re-queued in the uncharged lane stays a resolved-interaction continuation with no failure retry spent). - Existing suites pass: the native executor, recovered-owner follow-up, instruction working copy, run completion, run failure cause and resume replacement test files. `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts question-handoff scheduler config` passes. Server and chaos harness `tsc --noEmit` pass. - Not run: a Daytona SR1U wave. It needs a runner image that matches this checkout. ## Risks - A sandbox marked for replacement is resumed when it cannot be replaced safely. That run can meet the same unretired owner again, fail again in the bounded lane, and then rest with Retry, as today. - The replacement session restores the latest verified harness backup. When the old Runner's final checkpoint was never saved, a same-session continuation resumes from that backup. - An unanswered tool input is accepted only for a committed governed-wait park whose Runner's sandbox lease ended on another sandbox. - Previous-release story: the marker and the run record are additive JSON. An older server ignores the marker and resumes the sandbox as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with tool use (shell, file edits, test runs) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a52f622be8 |
fix(native): retry a remote provider startup failure once its runner exit is proven (#20874)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) start a Runner in the
sandbox, and the Runner opens the provider session
> - When the provider rejects `session.open` at startup, no provider
work began
> - #20808 ends every such remote rejection at once with Retry, because
restart recovery could not prove that the failed runner was gone. A
fault that clears by itself therefore needs one human click
> - The Runner transport already settles the failed runner on close and
observes its exit. That is the missing proof
> - This pull request records that proof on the failure and lets the
same-run bootstrap retry use it. It stops a startup error that repeats,
so an image fault that a retry cannot fix does not loop
> - The benefit is that a transient remote startup failure recovers
without a person, and a permanent one still rests at once with a working
Retry that names the error
## Linked Issues or Issue Description
Refs #20808
Refs #20832
Refs #20820
**What happened?**
A remote (Daytona) native run whose provider rejected `session.open` at
startup ended as `native_provider_startup_rejected` with Retry. This
happened even when the Runner had settled and exited, and no provider
session, event or checkpoint existed. A person had to click Retry for a
fault that the durable record proves did no provider work.
**Expected behavior**
With a proven runner exit and no provider work, the run retries its
bootstrap on the same run automatically. A startup error that repeats on
the next attempt stops at once with Retry and names that error. Failures
saved without the proof keep today's behaviour.
**Steps to reproduce**
1. `cd server && npx vitest run
src/services/native-runtime/native-startup-rejected-retry.integration.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "remote startup rejection|session.open rejection"`
2. On master, 3 cases fail: the proven remote rejection is
`terminal_failure`, the repeat does not name its error, and restart
recovery holds the proven failure on
`ambiguous_remote_recovery_evidence`.
**Paperclip version or commit**
master at
|
||
|
|
e6b538efd2 |
fix(native): retire a superseded warm owner instead of failing the new run (#20844)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep a warm Runner between turns of the same task, so
the next run can start fast
> - When the next run needs a new provider session (for example after an
agent setting or skill changes), it first retires the previous run's
warm Runner
> - That retirement uses the previous owner's close. When the close
cannot prove checkpoint suspension, the new run failed before any
provider work, with the previous owner's error
`provider_transport_failed: runner did not durably suspend before
checkpoint`
> - A governed-wait park (a saved question or review request) always
leaves such an owner: its question callback stays unsettled by design. A
failed Launcher stop RPC leaves one too
> - Nothing retried the run, and the task rested as "recovery needed"
with no provider work done
> - This pull request lets the new run retire the old owner when that
owner is contained, and forces the containment when the close left it
unproven
> - The benefit is that a fresh run never fails on its predecessor's
close, and the chaos harness now covers this class
## Linked Issues or Issue Description
**What happened?**
A new run on a task whose previous run left a warm Runner retained
failed within two minutes, before any provider turn. The error was
`provider_transport_failed: runner did not durably suspend before
checkpoint`. The run row recorded `executionRecovery: { kind:
"bootstrap", providerWorkStarted: false }`. Two runs of the same agent
on two tasks failed this way 47 seconds apart, after a configuration
change made both start new provider sessions. One predecessor had parked
on a `request_human_input` question; its callback never settled.
**Expected behavior**
A run that has done no provider work does not fail because its
predecessor's Runner could not prove suspension. The new session never
resumes the old checkpoint. Core must only prove that the old owner is
contained, and then start.
**Steps to reproduce**
1. Run a warm native task that parks on a saved question (chaos workload
Q1).
2. After the asking run settles, change the agent's adapter
configuration, so the next run cannot resume the saved session.
3. Answer the question. The follow-up run retires the parked owner and
fails with the error above.
Related: #18373 added the stopped-owner reset path that this change
extends.
## What Changed
- `closeWarmNativeSession` (server native executor): during retirement
for a different normalized session, accept an unrecoverable close when
the old owner is contained. A plugin task owner must show its
`executionStopped: true` receipt and a completed handoff. Without a task
owner, the Runner's own durable suspension
(`settlement.runnerSuspended`) is the proof. Unsettled Core tool
callbacks of the discarded conversation no longer block the reset.
- With a plugin task owner, any close error qualifies once the Launcher
receipt exists. This includes a failed `stop` that
`releaseRunnerProcessOwnership` rethrows, and "Controller journal drain
timed out": the discarded session's journal lives in its own state root.
Owners without a task keep the narrower rule. `--matrix` now rejects
session faults.
- `createPluginRunnerTaskOwner`: add `retireSupersededTask()`. When
close left the receipt unproven, the new run sends one confirmed `stop`
(not resent after a confirmed stop), takes the credential snapshot, and
calls `complete`. It never reads the old state.
- Same-session continuation, a handoff that still fails, other close
errors and failed instruction collection still block replacement.
- Chaos harness: add session fault SR1
(`tests/restart-chaos/session-faults.ts`). With Q1 and `--lifecycle
warm` (or `--daytona-warm`), the board changes
`adapterConfig.idleTimeoutMs` after the asking run settles and before it
answers. `--tasks 2 --concurrency 2` is the two-concurrent-runs variant.
Q1's oracle applies, plus `session_rotated_before_answer` and
`follow_up_new_session`.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` and the chaos README
describe the new boundary and cell.
## Verification
- RED on master: `native-session-executor.test.ts -t "permanently
rejected plugin close during reset"` fails the `pending-semantic` and
`stop-unconfirmed-at-close` cases with
`NativeSessionCloseUnrecoverableError: provider_transport_failed: runner
did not durably suspend before checkpoint`. GREEN with this change: all
identity-transition cases pass (49).
- Chaos SR1 local, master server code (`--env local --lifecycle warm
--workloads Q1 --faults SR1`, seeds 301 with 2 tasks and 303 with 1
task): all tasks pass, and the follow-up started a new provider session.
The local parked owner closes cleanly, so the local cell does not
reproduce the failure. It needs a Daytona task owner.
- Chaos SR1 local with this change (seed 302, 2 concurrent tasks on one
agent): 2/2 pass, `follow_up_new_session` true for both.
- Not run: the Daytona SR1 variant. No runner image matches this
checkout's runner content ID, and this change does not touch runner
inputs.
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
question-handoff scheduler config` passes. Server and harness `tsc
--noEmit` pass.
## Risks
- The reset no longer waits for unsettled Core tool callbacks of the
discarded session. Their run is already terminal and their provider is
stopped, so no result can reach a provider, and the new session never
replays them. Core-side tool executions keep their own receipts under
the old run.
- A local owner without a task needs the Runner's durable suspension. A
local owner that did not suspend still blocks replacement, as before.
- Previous-release story: this changes no persisted state. Warm owners
live only in controller memory, so a deploy drops them as before.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with
tool use (shell, file edits, test runs) in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
c8ae02e5ce |
fix(native): start a fresh session when an accepted run's provider state was not saved (#20811)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run in a per-turn remote sandbox saves a provider state backup after its result is accepted. The sandbox is then released, so the backup is the only copy of the provider session. > - When that backup fails to save (for example, #20772: storage throttling), the run keeps its accepted result. Nothing records that its checkpoint can no longer be resumed. > - The next run on the session rebinds that checkpoint and asks the Runner to resume. The Runner has no state and refuses (#20757). With the normal policy, the run opens a replacement session after that failed attempt. With a governed-wait or unfinished-goal checkpoint, the run fails with `native_provider_state_incompatible`, and only Retry or the next message starts fresh. > - The server already knew the state was gone when the backup failed. The failed resume is wasted work. > - This pull request marks the accepted run's checkpoint as not resumable at the moment the backup fails. The next run then starts a fresh provider session directly. > - The benefit is that a lost backup no longer costs a failed run or a failed resume attempt before the task continues. ## Linked Issues or Issue Description Refs #20757 (the missing-state refusal that the next run reaches today). Refs #20772 (the throttled backup upload that loses the backup). Refs #20809 (a related fix for governed-wait follow-ups that reach the missing state; this PR stops most of them from reaching it). **What happened?** 1. A native run in a per-turn sandbox finished, and its result was accepted. 2. `harness_state.backup.persist` failed. The run recorded `native.post_result_failure`. Its run row kept a normal `sessionCheckpoint`. 3. The sandbox was released. 4. The next run on the same session selected that run as its resume source, and `rebindNativeSessionCheckpoint` returned the checkpoint. 5. The Runner found no saved state and refused with `PRP provider resume state is unavailable`. The run then opened a replacement session after the failed attempt, or (governed-wait or unfinished-goal checkpoint) failed with `native_provider_state_incompatible`. Retry or the next message then started fresh. **Expected behavior** The next run starts a fresh provider session at once, because the server already knows the saved state is not there. The accepted result and the task outcome stay as they are. **Steps to reproduce** 1. Give a run row an accepted result and a normal `sessionCheckpoint`. 2. Make its close checkpoint fail in a per-turn sandbox (`reusableLeaseConfigured: false`). 3. Select the next run's resume source with `findNativeSessionResumeRun` and call `rebindNativeSessionCheckpoint`. 4. On `master` it returns the checkpoint, so the next run tries to resume state that does not exist. **Paperclip version or commit** `master` at `556b249897`. ## What Changed - `server/src/services/native-runtime/native-provider-state-unsaved.ts` (new): `markAcceptedRunProviderStateUnsaved` adds `nativeProviderStateUnsaved: { code: "native_provider_state_incompatible", reason: "harness_backup_not_saved" }` to the run's `runnerProfileJson`. One conditional `UPDATE` writes it only when the run's coordinator holds a result with `schema_status = 'accepted'`. `clearRunProviderStateUnsaved` removes it. - `server/src/services/native-runtime/native-session-executor.ts`: the remote close checkpoint (`checkpointRemoteRunner`) now wraps the existing body. When the checkpoint throws in a sandbox without a reusable lease, it calls the marker, writes one run-log line, and throws the original error. A marker or log failure cannot replace that error. If a later checkpoint of the same run saves, it clears the mark. - `server/src/services/native-runtime/native-session-resume.ts`: `rebindNativeSessionCheckpoint` returns `null` for a source with the mark, as it already does for `errorCode = native_provider_state_incompatible`. `findNativeSessionResumeRun` projects the new key. - `doc/execution-semantics.md`: one additive paragraph. - Tests in `native-session-resume.test.ts` and `native-session-executor.test.ts`. Guard conditions. The mark is written only when all of these are true: - The close checkpoint threw before the durable backup was stored (after the upload's own bounded retries). A later local publication, stamp or cleanup failure does not mark the run. - The target is a remote `sandbox` with `reusableLeaseConfigured !== true`. A retained sandbox or an SSH host keeps the live state, so its resume path is unchanged. - The run holds an accepted result. A failure before acceptance changes nothing. Saved state from the previous release: run rows without the key keep today's resume path. An older server that reads a row with the key ignores it, and so it behaves as it does today. No schema change. User-visible: the next run's transcript shows "Paperclip session started" instead of "resumed", with a new session id. The source run's log shows `provider state was not saved after the accepted result; the next run starts a fresh provider session`. The fresh session gets the full task context, as for other fresh starts. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-session-resume.test.ts -t "not resumable only after|provider state was not saved"`: 9 pass (6 pure, 3 embedded Postgres). - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "marks the provider state unsaved|uses the bundled Linux image authority for remote opencode-new-durable"`: 8 pass. They include a durable store failure (marked) and a local publication failure after the store (not marked). - Red on `master` (source files reverted, tests kept): the 3 "starts the next run in a fresh provider session directly" cases fail (`expected {…} to be null`), the DB "accepted" case fails the same way, and the executor per-turn case fails (`mark` called 0 times). - Guard removal checks: - Sandbox guard replaced with `true`: the "retained sandbox" case fails (`mark` called once). - `!checkpointBackupStored` removed: the "publish-failed" case fails. - `schema_status = 'accepted'` removed from the marker: the "rejected" case fails (`expected true to be false`). - Negative cases that pass: no result, rejected result, retained sandbox, records without the key or with another code (all three recovery policies), and a later save that clears the mark. - `cd server && npx tsc --noEmit -p .`: clean. - Not related to this change and failing on `master` in this worktree too: `runtime-selection.test.ts` connector skill cases (missing `.claude/skills/*/SKILL.md`) and `native-session-resume.test.ts` "archives a damaged prior epoch" (`ENOENT … authority-epochs`). Required CI is the authority for those suites. ## Risks - Low. The change only skips a resume that cannot succeed. The fresh session uses the same workspace and full task context, like a Retry after `native_provider_state_incompatible`. - If the backup failed but the provider state were still somewhere recoverable, the next run would lose provider context. For a per-turn sandbox without a reusable lease, the restore path uses only the backup, so this does not apply. Retained sandboxes and SSH hosts are excluded. - One extra conditional `UPDATE` runs only on the failure path. The normal path before `turn.start` has no new waits or work. - `doc/execution-semantics.md` is also edited by #20809. The paragraphs are at different places; whichever merges second rebases. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
1b39cf1794 |
fix(native): retry a contended provider-pack admission write (#20850)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native OpenCode run on a remote sandbox must save a provider-pack admission receipt on its run row after the provider session opens. Artifact retention refuses to continue without it. > - The admission write locks the coordinator row (`native_run_finalizations`) and then the run row, each with a 1000 ms `lock_timeout`. The runner's durable checkpoint writes lock the same coordinator row for the whole checkpoint transaction. > - On a loaded database one checkpoint transaction can outlast one lock wait. The admission write then fails with `55P03`, the state stays `pending`, and artifact retention fails the run as `native_provider_artifacts_admission_unverified` before the first turn. The task rests blocked. > - #20682 fixed the same symptom for a run-event append on the run row. This is the coordinator-row half. > - This pull request retries a contended admission write a bounded number of times, each in a new transaction that re-proves the claim. It also names the database cause in the run log and adds a chaos option that makes every fresh sandbox take the pack upload and admission path. > - The benefit is that a healthy fresh sandbox is no longer failed and blocked by ordinary lock contention. ## Linked Issues or Issue Description Refs #20682 (same symptom, run-row lock; this is the coordinator-row lock). **What happened?** On a hosted staging instance, an OpenCode run on a fresh Daytona sandbox found no matching preinstalled provider pack (`provider_pack.select` reason `cached_packs_unverified`). It uploaded and verified the pack (61 s), started the runner, and the runner authenticated to the durable control plane. 3.4 s later the run log shows `native provider-pack admission receipt could not be saved`. `nativeProviderPackAdmissionState.state` stayed `pending`. The run failed with `native_provider_artifacts_admission_unverified` before any provider turn. The task rests `blocked` with no Retry. **Steps to reproduce** 1. Start a native run that waits for its claim (`waitForClaim: true`, as a fresh run with artifact storage does). 2. Hold its `native_run_finalizations` row `FOR UPDATE` in another transaction for 1.6 s, as a checkpoint write does. 3. Call `recordNativeProviderPackAdmission`. It fails with `canceling statement due to lock timeout` (`55P03`). **Expected behavior** The admission waits out the checkpoint write within a bound, re-proves the claim, and saves the receipt. ## What Changed - `recordNativeProviderPackAdmission` retries a write that waits for its claim when it fails with `55P03`, `40P01` or `40001`, up to 5 attempts with 100–400 ms back-off. Each attempt is a new transaction with the same checks (claim, run, epoch, receipt). A stale claim still returns `false`. Other errors still throw. A write that does not wait for its claim keeps its `NOWAIT` refusal. - New `nativeProviderPackAdmissionFailureCode` reads the SQLSTATE from the error or its cause. - The executor's run-log line now ends with `(cause=<SQLSTATE or native_* code>)`, so a lost admission is diagnosable. It carries no paths or secrets. - Restart chaos: `--daytona-pack-mismatch <image@sha256:…>` starts sandboxes from another image while the controller keeps `--daytona-image`'s runnerd and pack. Every fresh sandbox then takes the upload and admission path, which the matched-image check made unreachable. Config validation and README updated. - `doc/execution-semantics.md` describes the bounded retry. ## Verification - RED on `origin/master` (`1d67e309e0`): `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-provider-pack-admission.test.ts -t "checkpoint write holds"` fails with `canceling statement due to lock timeout` (`55P03`), the staging error. - GREEN: the same file, 39/39 pass, including the existing fence tests (`cannot publish while another transaction holds the coordinator claim`, revoked-claim recheck) and the new bound test (`gives up after the bounded retry window while the claim stays held`, under 15 s). - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/scheduler.test.ts`: 34/34 pass, including the new option test. - `pnpm --filter @paperclipai/server exec tsc --noEmit` and `tsc --noEmit -p tests/restart-chaos/tsconfig.json` pass. - Not run: a Daytona chaos wave with `--daytona-pack-mismatch`. It needs a runner image built at this revision. The staging cause is inferred from the timing and the code path; the original error was swallowed, which is why this PR adds the cause to the run log. ## Risks - Low risk. The normal path has no contention and makes one attempt. Under contention the write waits at most about 11 s before the first provider turn. Without this change that case failed the run. - A retry cannot widen authority: every attempt re-checks the coordinator claim, lease expiry, run status and admission epoch. - When admission still fails after the retries, the run fails as before. This PR does not change that recovery path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
381b862135 |
fix(native): let a finish through after one stale-result refusal; rebuild a missing instruction bundle (#20833)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native Codex run ends its task with a Paperclip terminal call
(`paperclip_finish` or `paperclip_block`)
> - The Runner refuses a terminal call that the agent decided before it
saw earlier tool results (`terminal_call_before_results`). This protects
against stale calls held across a restart.
> - On staging, Codex cancelled a nested exec cell. Codex never sent
`item/completed` for one Paperclip tool call in that cell. The Runner
then refused every later finish and block in the turn, and the turn
ended with no semantic result.
> - The same-run recovery then resumed the run on a new controller
container. The run's admitted instruction bundle is stored on the
controller's local disk, so it was missing there, and runnerd failed
with ENOENT.
> - This pull request makes a refusal name a returned call once, and
rebuilds a missing admitted instruction bundle before the Runner backend
starts.
> - The agent-directory lock part of the same failure chain ("Agent
directory operation is already active") is fixed separately by #20829.
> - The benefit is that the agent can finish after one refusal, and a
resumed run does not fail because a deploy moved it to a host without
the bundle.
## Linked Issues or Issue Description
Refs #20829 (the agent-directory lock wait and dead-owner release; this
PR does not change that lock).
**What happened?**
Staging Codex runs on Daytona sandboxes failed after about 30 minutes of
work:
1. Early in the turn, the agent called `paperclip_search_assigned_tools`
from a nested exec cell. Codex cancelled that cell. The controller
returned the result, but Codex never sent `item/completed`, so the
Runner never committed `tool.execution.completed` for it.
2. At the end, the agent called `paperclip_finish` four times and
`paperclip_block` once. The Runner refused each one with
`terminal_call_before_results` ("Those calls have now returned"). The
old call stayed "returned but unseen" for the rest of the turn.
3. The turn completed with no semantic result
(`native_finalization_missing`). Same-run recovery scheduled another
attempt.
4. On the new controller container, the attempt failed with
`runner_ingress_unavailable: runnerd process failed: ENOENT ... scandir
'/paperclip/instances/default/runtime-context-assets/bundles/<sha>'`
(one run) or `ENOENT ... lstat` of the same kind of path (a run during a
rolling deploy). In both, `<sha>` is the run's admitted instruction
bundle. The asset root is controller-local, and the new containers had
no volume.
5. The last attempt failed with "Agent directory operation is already
active" (fixed by #20829).
**Expected behavior**
The agent finishes after one refusal. A resumed run rebuilds its exact
admitted instruction bundle on the new host.
**Steps to reproduce**
1. Run a Codex task that calls a Paperclip tool, then
`paperclip_finish`.
2. Drop Codex's `item/completed` for the first tool call (new chaos
fault `--faults PV2 --provider-fault-message tool_completed`).
3. On master, every `paperclip_finish` is refused and the run ends with
`native_finalization_missing`.
4. Separately, delete `runtime-context-assets/bundles/<instruction
digest>` and resume a run that pinned it: runnerd fails with ENOENT.
**Paperclip version or commit**
master at
|
||
|
|
fa3cd6f330 |
fix(native): end provider startup rejections with Retry instead of an ownership hold (#20808)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner starts Codex for native runs. On a remote
sandbox such as Daytona, the sandbox is the OS boundary.
> - A Codex planning run on Daytona failed in `session.open`. Codex's
read-only sandbox helper could not set up loopback in the container
(`bwrap: loopback: Failed RTM_NEWADDR`), and Codex loads `AGENTS.md`
through that helper.
> - The server treated that definite provider rejection as an
interrupted transport. Its same-run bootstrap retry went through remote
restart recovery and stopped on `ambiguous_remote_recovery_evidence`.
The task showed "Recovering saved work…" and no Retry, although no
provider work was done.
> - This pull request gives such startup rejections a named, permanent
outcome with a working Retry, and pins ask and standard runs to the
external sandbox policy.
> - The benefit is that a provider startup refusal ends as a clear
failure with Retry, never as an ownership hold.
## Linked Issues or Issue Description
**What happened?**
A Codex planning run on a Daytona sandbox failed in `session.open` with
`error creating thread: failed to load AGENTS.md instructions ... bwrap:
loopback: Failed RTM_NEWADDR: Operation not permitted`. The server
recorded `native_session_interrupted` with recovery mode
`bootstrap_retry`. Thirty seconds later, remote restart recovery blocked
the run with `native_restart_recovery_blocked` /
`ambiguous_remote_recovery_evidence`. No server restart happened. All
default-mode Codex runs on the same image succeeded that day; only the
planning run failed.
**Expected behavior**
A provider startup failure with no provider work must end as a clear
failure with a working Retry, or get a bounded automatic retry that also
ends with Retry. It must never become an ambiguous-ownership hold.
**Steps to reproduce**
1. `cd server && npx vitest run
src/services/native-runtime/native-startup-rejected-retry.integration.test.ts`.
On master the deterministic case is `retryable_failure` /
`native_session_interrupted`.
2. `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "provider
itself rejected"`. On master the new code is not produced.
**Paperclip version or commit**
master at
|
||
|
|
c44a422607 |
fix(native): retry a follow-up whose own server stopped serving before its remote runner (#20786)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime runs each agent task through a Runner. On a graceful shutdown (SIGTERM), the server closes its runtime API listener first, then drains runs. > - A run that is still preparing can reach the runtime API lookup after the listener closed. It then fails with `local_runtime_api_listener_unavailable` before it reaches its Runner. > - For a remote (sandbox) Runner, the executor already settles this failure as a same-run bootstrap retry, so the next server restarts it with no manual Retry. > - A follow-up run is different. Admission rebinds the settled predecessor's provider session into the follow-up's run as its checkpoint. The recovery evidence saw "a checkpoint exists" and called the failure `ambiguous_state`. That blocked the task and the user had to click Retry. > - This pull request recognizes that carried checkpoint. When this server's own listener closed, the run has no Runner checkpoint and no provider event of its own, and a settled predecessor saved that exact provider session, the failure is a same-run bootstrap retry. > - The benefit is that a graceful restart during a follow-up's preparation no longer needs a manual Retry. ## Linked Issues or Issue Description **What happened?** In a Daytona restart-chaos wave (codex, remote sandbox Runner), a follow-up run was created. About 2 s later a graceful SIGTERM drain began. The draining server kept preparing the run. `resolveLocalRuntimeApiUrl()` in `createRunnerdBackendWithinSessionClaim` threw `LocalRuntimeApiListenerClosedError`. The run failed with `local_runtime_api_listener_unavailable`. The task went to `blocked` with a board-owned recovery action, and it needed a manual Retry. The manual Retry worked. **Expected behavior** No provider work started for the run. The next server must pick it up automatically. **Steps to reproduce** 1. Finish a remote native run for a task, so that its provider session is saved. 2. Comment on the task to start a follow-up run. The follow-up carries the saved provider session as its admission checkpoint. 3. Send SIGTERM while the follow-up is in preparation, before it reaches its Runner. 4. The follow-up fails with `ambiguous_state` and the task is blocked. **Paperclip version or commit** `master` at `62b7733161` (wave) and `eb75c9bb85` (this branch base). Refs #20514, #20721 ## What Changed - `native-session-executor.ts` (`nativeProviderRecoveryEvidence`): add one narrow case before the generic checks. When all of these are true, return `bootstrap_retry` (with `checkpointExists: true` and `providerSessionEstablished: false`): - the source failure is `local_runtime_api_listener_unavailable` (this server stopped serving); - the target is remote, the run has the current preparation fencing version, and the run has no Runner controller checkpoint; - the run has no provider event; - the run checkpoint has no cursor, turn, terminal, result, or pending request of its own; - a settled earlier run of the same native session saved that exact provider session. - `native-restart-recovery.ts`: export the existing `carriesSettledPredecessorSession` proof, so that the executor uses the same proof that restart recovery uses for rebound sessions. - `native-controller-recovery.integration.test.ts`: add a regression test. It also shows that the verdict stays `ambiguous_state` for a different failure code, for an unknown provider session, for a checkpoint with its own turn, and for a run with a provider event. ## Verification - Red on master: `cd server && npx vitest run src/services/native-runtime/native-controller-recovery.integration.test.ts -t "own server stopped serving"` returned `recoveryMode: "ambiguous_state", providerSessionEstablished: true`. - Green with the fix: the same file passes (25 tests). - `npx vitest run src/services/native-runtime/native-startup-retry.integration.test.ts src/services/native-runtime/native-runner-restart-preparation.integration.test.ts`: 43 passed (the existing closed-listener bootstrap-retry and ambiguous cases are unchanged). - `cd server && pnpm typecheck`: passed. - Not run: a live restart-chaos wave. ## Risks - Low risk. The new case applies only to the closed-listener failure on a remote target. It needs durable proof that the run has no Runner checkpoint and no provider event, and that the provider session belongs to a settled predecessor. All other cases keep their earlier verdict. - If the retry later finds different evidence, the existing restart recovery and resumption checks re-prove it and fail closed. The task then keeps a working Retry. - Upgrade safety: no persisted shape changes. The proof reads only rows that older servers already write (`sessionCheckpoint`, `nativePreparationFencingVersion`, Runner checkpoints, run events). A run that an older server prepared without the fencing version keeps its earlier verdict. - No waits are added before `turn.start`. The new queries run only on the failure path. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
4d141cdf64 | Carry plugin task refusals and batch project reads (#20762) | ||
|
|
f25fee3532 |
fix(native): hash runner artifacts and run archive tar off the event loop (#20722)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process prepares native runs, renews every controller lease on timers, and serves every request > - On a run path, the server hashed the 31 MB runner binary with `readFileSync` (more than once per run), hashed provider packs and harness backups synchronously, and ran `tar` with `execFileSync` (timeouts up to 60 s, one with no timeout) > - On a slow disk, each of these calls freezes the whole process. A 103 s stall on a hosted instance lapsed every lease. The last event before that stall was immediately before a synchronous read and hash of the runner binary (see #20694 and #20709) > - This pull request moves those reads, hashes and archive commands off the event loop. It computes the runner binary digest once per boot > - The benefit is that a slow disk delays only the run that waits for it. Turns also start faster, because the runner binary is no longer re-read for every run ## Linked Issues or Issue Description Refs #20694, #20709 **What happened?** On a hosted instance, the server process stopped for 103 s without a restart. Every controller lease lapsed, and two tasks needed a manual Retry. The last run event before the gap was immediately before `matchesControllerRunnerArtifact`, which does `createHash("sha256").update(readFileSync(controllerRunnerBinary))`. Four workspace restores were running at the same time, and file I/O was very slow (a 43 s archive hash, 20 s snapshot decrypts). The trigger is not proven. But each synchronous call below blocks the event loop for as long as the disk is slow. **Expected behavior** File hashing and archive commands on a run path must not block the event loop. A slow disk must delay only the run that waits for it. **Steps to reproduce** 1. Make a remote runner binary slow to read (the test uses a FIFO whose writer waits), or make `tar` slow (the test puts a `tar` wrapper that sleeps first on `PATH`). 2. Call `resolveRemotePaperclipRunnerBinary`, `stageRemoteRunnerDirectory`, or `syncRemoteRunnerDirectoryOut` while a 10 ms timer runs. 3. Before this change, the timer stops for the whole slow step (814 ms, 1,764 ms and 3,379 ms). After this change, it keeps firing. ## What Changed - `server/src/services/native-runtime/off-loop-file-work.ts` (new): - `sha256FileStreamed` - `cachedFileSha256`: a streamed digest, reused while the file's device, inode, size, mtime and ctime do not change - `binaryTargetFromHeader`: reads 64 KiB; reads the rest of the file only when the header cannot answer, so the result equals `runnerBinaryTarget(readFileSync(path))` - `sha256DirectoryTreeStreamed` - `execFileBuffered`: `execFile` with stdin input, `maxBuffer`, `timeout` and `env`. It rejects on a non-zero exit, a timeout or a `maxBuffer` overflow, like `execFileSync`. - `native-session-executor.ts`: - `matchesControllerRunnerArtifact` uses `cachedFileSha256`. Each boot reads the binary once, not once or twice for every run. - The remote Claude daemon target is read from the binary header. - The bundled remote runner binary is checked by `verifiedBundledRemoteRunnerBinary`: the same checks and errors, with a cached digest. - Provider-pack upload verification uses the new `verifyRemoteProviderPackManifest`. It runs the same checks, in the same order, with the same errors. One generator drives both the sync and the streamed readers. - Harness backup build and verify on run paths use streamed digests (`buildNativeHarnessBackupManifestStreamed`, `verifyNativeHarnessBackupStreamed`). The digest format does not change. - `stageRemoteRunnerDirectory` (3 calls), `assertSafeRemoteCheckpointArchive` (3 calls) and the checkpoint extract use `execFileBuffered`, with the same arguments, timeouts and buffer limits. `stageRemoteRunnerFile` uses `readFile`. - `packages/paperclip-runner/src/live/runner-binary.ts`: the default daemon lookup (`resolvePackagedRunnerBinary`, used by the local and remote resolvers) classifies the daemon with `runnerBinaryFileTarget`. It reads the 64 KiB header, not the whole file. - `packages/paperclip-runner/src/live/bundled-remote-provider-pack.ts`: the release-manifest checks now live in one helper, `bundledRemoteRunnerRelease`. The sync `bundledRemoteRunnerBinary` and the server's streamed check both use it. - `native-codex-runner.ts`: - `resolveRemotePaperclipRunnerBinary` reads only the header. - The local `runnerDigest` uses `cachedFileSha256`. - New `verifiedBundledRemoteRunnerBinary`. **Ordering and atomicity.** I checked each converted call for code that relied on the sync call being atomic with respect to other runs: - **Checkpoint copy-back.** It extracts into a private `mkdtemp` directory. The replace sequence (`existsSync`, `rename`, `rename`, `rm`) stays synchronous, so no other run can see a half-installed checkpoint. This function already awaited a remote command before the extract, so this change adds no new interleaving with the target. - **Harness backup verify then restore.** The verify was sync, but the restore after it was already async. The window between them existed before. One owner serializes the turns of a session, so no other turn publishes a backup for that session while a turn verifies it. - **Staging and provider-pack hashing.** These read build-owned, immutable trees, or the private staging copies of a run. ## Verification - Red/green: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/off-loop-file-work.test.ts` - With `native-session-executor.ts` and `native-codex-runner.ts` from `origin/master`: 3 failed. The longest timer gaps were 814 ms (runner binary target read), 1,764 ms (directory staging tar) and 3,379 ms (checkpoint list, size check and extract). - With this change: 6 passed. The gaps stay under 400 ms. The same file also checks the digest cache, the header fallback, and the `execFileSync`-like errors (exit code, timeout kill, `ERR_CHILD_PROCESS_STDIO_MAXBUFFER`). - `native-session-executor.test.ts`: - The streamed provider-pack verifier returns the same manifest as the sync reader, and rejects a changed asset tree with the same error. - The streamed harness backup manifest equals the sync one. - The streamed verify agrees with the sync verify, for a valid backup and for a corrupt one. - Run together: `native-session-executor`, `off-loop-file-work`, `native-codex-runner`, `setup-readiness`, `antigravity-connection-probe` (1,653 passed). `run-completion.test.ts` passed all 153 tests. Its `afterAll` embedded-Postgres cleanup timed out locally, which is not related to this change; CI runs it. - `pnpm --filter @paperclipai/server exec tsc --noEmit` passes. ## Risks - **Turn start.** This change adds no wait before `turn.start`; it removes work. The first run of a boot streams the runner binary once. Later runs reuse the digest after one `stat`. - **Cache staleness.** The cache keys on device, inode, size, mtime and ctime. An in-place rewrite changes mtime and ctime, and a replaced file changes its inode, so a changed binary is hashed again. - **stderr.** `execFileSync` wrote a child's stderr to the server's stderr. `execFileBuffered` captures it. Tar warnings no longer appear in server logs; errors and exit codes are unchanged. - **Persisted state.** No persisted shape changes. Old saved state and new code (and the reverse) read the same manifests and digests. - **Not converted in this PR:** small metadata syscalls (`existsSync`, `lstatSync`, `readdirSync`, small JSON reads and writes, renames) in the native state machines. Some of these rely on running synchronously. Also not converted: the sync harness backup verify inside the sync restart-authority checks, and the sync stamp verify in `native-harness-backup-storage.ts`. Both are listed for a follow-up. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use, shell, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (code comments carry the contract) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
94c7c83ce8 |
fix(runner): refuse a moved-root same-session checkpoint by name (#20723)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a remote sandbox resume a task's provider session from a saved checkpoint > - #20712 made a checkpoint that names an abandoned remote workspace root decline recovery, so a replacement session opens in the new root > - A checkpoint that requires the same session (an unfinished goal, `same_session_only`) cannot be replaced. It then failed with the generic `native_session_recovery_failed`, which recovery treats as a retryable interruption > - A retry cannot move a saved session to another root, so the failure repeats. The upgrade-safe rule requires a permanent, named code for this case > - Also, the working-directory errors said "Codex" on Claude and other ACPX runs, because one app-server driver serves all of them > - This pull request reports the moved root as provider state that is incompatible, and makes the errors name the agent generically > - The benefit is a named, permanent refusal with a working board Retry, and error text that is correct for every agent ## Linked Issues or Issue Description Refs #20712 **What happened?** After #20712, a `same_session_only` checkpoint with a moved remote root ended with `native_session_recovery_failed: persisted session working directory is not the assigned remote workspace`. That is not a named permanent code, so recovery could retry a run that cannot succeed. On ACPX Claude runs, path errors read "Remote Codex working directory ...". **Expected behavior** The run ends with the permanent code `native_provider_state_incompatible`. Automatic retries stop. A board Retry continues the task in a fresh provider session in the same workspace. Errors do not name Codex on other agents. **Steps to reproduce** 1. Save a native checkpoint with `providerRecoveryPolicy: same_session_only` under one remote workspace root. 2. Assign the run a different sibling root (the epoch was abandoned). 3. Resume the run. ## What Changed - `codex-app-server-driver-impl.ts`: the moved-root decline from #20712 now carries `failureClass: provider_state_incompatible`. The runtime already maps that class: - checkpoint allows replacement (`allow_replacement_after_resume_failure`): a replacement session opens in the assigned root, as before; - checkpoint requires the same session: `NativeProviderStateIncompatibleError`, code `native_provider_state_incompatible`. - The server already treats `native_provider_state_incompatible` as a permanent failure (no automatic retry, board action text "Retry the task to continue in a fresh provider session with the same workspace"). `rebindNativeSessionCheckpoint` returns no checkpoint after that code, so a board Retry starts a fresh session in the same workspace. No server change is needed. - A moved-root checkpoint with an active turn or a pending runtime request now reports `provider_side_effect_pending`. The runtime never replaces it under any policy, and the run ends with the permanent code `native_provider_side_effect_pending` for an operator to reconcile. Before, it threw the working-directory refusal, which recovery recorded as the retryable `native_session_interrupted`. - `harness-driver.ts`: the `PROVIDER_STATE_INCOMPATIBLE` doc now names the moved-workspace case. - `codex-boundaries.ts`: working-directory errors say "Agent working directory", "Remote agent working directory" and "Provider thread response" instead of "Codex". Tests updated to the new text. - Tests: - `native-session-runtime.test.ts`: the real app-server backend with `remote_runner` authority and a moved root, for each recovery policy. Same-session policies end with `native_provider_state_incompatible`, the replacement policy opens a replacement in the assigned root, an in-flight turn ends with `native_provider_side_effect_pending` under every policy, and no transport starts. - `native-backend-factory.test.ts`: the decline carries the failure class, and an ACPX Claude refusal reads "Remote agent working directory does not match the assigned workspace". Previous-release story: no persisted shape changes. Saved checkpoints are classified at recovery time. A saved same-session checkpoint under an old root now ends with the named permanent code instead of a generic one. ## Verification - Red: without the failure classes, the same-session and in-flight runtime cases fail (generic `native_session_recovery_failed`). Before the rename, the ACPX Claude message test fails. - Green: `cd packages/paperclip-runner && npx vitest run src/backends src/drivers/codex src/live/runner-prepared-launch-workspace.test.ts src/native-session-runtime.test.ts` passes (784 tests). - `npx tsc --noEmit -p packages/paperclip-runner` passes. - Existing server coverage for the code: `native-session-resume.test.ts` (no rebind after `native_provider_state_incompatible`) and `native-session-executor.test.ts` (source-code mapping). ## Risks - Low. Error message text changes. A search found no code that matches on the old text outside the updated tests. - Same-session checkpoints with a moved root now stop for a board Retry instead of a generic interruption. That is the intended behavior: a retry cannot succeed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
08ff0828d6 |
fix(native): offer Retry at once when the Runner refuses a turn start on a closed session (#20729)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native ACPX run (for example Claude) starts each turn with `turn.start`. When a start is cut off, the Runner reads the sidecar's dispatch journal (#20675). > - If the prompt never left, the Runner keeps the session reopenable, and the same-run resume sends the prompt once. If the prompt may have left, the Runner closes the session, because a resend could duplicate it. > - A closed session cannot reopen for the same run. Every same-run resume got `ACPX session is not open`. The server treated that as a generic interruption and spent three attempts (about 95 s) before the board offered Retry. > - This pull request gives that refusal a closed class in the Runner and a permanent, named failure code in the server. > - The benefit is that the board offers a working Retry on the first refusal, while every recoverable case keeps recovering by itself. ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T011704-s9107`, task t05 (local, claude, workload C1, fault TM2). A freeze cut off a turn start, and the Runner saved the session `closed`. Three same-run resumes each failed with `PRP command turn.start failed: … "message":"ACPX session is not open"`. Recovery stopped with `native_session_retry_exhausted` at 01:25:14, and the board then offered Retry. The Retry run succeeded. **Expected behavior** When the Runner proves the session is closed for this run, the run stops at the first refusal with a named failure, and the board offers a Retry that works. A start that the Runner proved never reached the provider still recovers by itself. **Steps to reproduce** 1. `cd server && npx vitest run src/services/native-runtime/native-closed-session-retry.integration.test.ts` 2. On master, "offers Retry on the first refusal, and Retry starts a new run" fails: the coordinator is `retryable_failure` / `native_session_interrupted` (a same-run retry in 30 s). Refs #20675, #20720 ## What Changed - Runner (`durable/mod.rs`, `acpx_provider_backend.rs`): new closed class `provider_session_closed`. The Runner sets it on these refusals: - a turn start that saves the session `closed` (`ACPX turn start failed closed: …`), but not the rejected-checkpoint case, which stays `prepared` and reopens; - `ACPX session is not open` when the saved lifecycle is `closed` for this run; - the saved turn-end-restart refusal for this run (asserted in `acpx_turn_end_restart`). A session that waits for provider-exit proof can reopen after a restart. Its refusal gets no class. - Server (`native-session-executor.ts`): a `turn.start` failure with that class maps to the new permanent code `native_provider_session_closed`. The server reads the class from the authenticated command result, and from the command-failure text if the error was rewrapped. `nativeSessionFailureDisposition` treats the code as permanent, so the first refusal ends the run as `terminal_failure` with a board owner and a plain next action. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` states the class and the older-Runner behavior. - Tests: - `native-closed-session-retry.integration.test.ts` (new, embedded Postgres) drives `executePaperclipNativeSession` with the Runner refusal. On the first refusal, the board shows `canRetry: true`. `requestTaskRetry` is accepted, and `resumeTaskRetry` starts exactly one new run with a fresh provider session. A start proved undispatched stays `retryable_failure`. - Unit tests in `native-session-executor.test.ts`: both closed refusals, a rewrapped error, an undispatched start, an unclassified older-Runner refusal, and a class on another command. - Rust `acpx_cut_turn_start`: a start cut off before dispatch has no class. A start cut off after dispatch, and the later refusal, carry `provider_session_closed`. ## Verification - Red on master, green here: step 1 above. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts`: 1358/1358 pass. - `cargo test -p paperclip-runner-core --test acpx_park_preparation` (26 pass), `--lib acpx` (94 pass), `--test acpx_provider_turns --test acpx_provider_session --test acpx_warm_attachment` (all pass), `cargo fmt --check` clean. - `npx tsc --noEmit -p server/tsconfig.json`: clean. - Not run: a chaos wave. ## Risks - This changes the Rust Runner. Daytona needs a new runner image before remote runs get the class. Until then, an older Runner sends the refusal without the class, and the server keeps today's bounded same-run retries. That path is unchanged and ends with a working Retry. The server does not trust the text alone, because an older Runner sends the same text for a session that can still reopen. - The class is set only where the Runner's saved state proves that no later start of this run can reopen the session. A false class would cost one automatic resume and show Retry earlier. It never replays provider work. - No persisted shape changes. The class is an optional field of a failed command result. Old saved state resumes unchanged. Nothing new runs before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, run in Claude Code with tool use (shell, file edit) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
200c6e40e2 |
fix(native): restart a bootstrap whose Runner suspended before the provider opened (#20691)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When a native run's first provider start fails before any session opens, Paperclip retries the bootstrap on the same run. Durable controller evidence proves that no provider session or provider event exists. > - A remote sandbox keeps the Runner state inside the workspace. A replacement sandbox gets that state back when the saved workspace is restored. > - When the provider refused `session.open` (for example an Anthropic 429), the Runner still suspended cleanly. Its state says `suspended`, but the provider state has no session identity. > - The same-run restart handled a Runner that never suspended. It did not handle a Runner that suspended before the provider opened a session. It failed with `runner_harness_state_mismatch: provider_identity_incomplete (provider_not_suspended)`, and the task showed "Recovery needed" without Retry. > - This pull request lets that proven restart set the unopened session aside and start the provider again. > - The benefit is that a rate-limited provider start continues by itself after the limit resets. ## Linked Issues or Issue Description Refs #20627 and #20683 (other restore paths). This PR changes only the same-run bootstrap restart. **What happened?** Two ACPX Claude tasks on remote Daytona sandboxes, on staging: 1. The first attempt sent `session.open`. ACPX rejected it with `classification=provider_rate_limit` (Anthropic 429). The Runner exited with `lifecycle=suspended`. 2. Paperclip recorded `recoveryMode: bootstrap_retry`, `providerSessionEstablished: false`, `providerEventsExist: false`, and a retry after the limit reset. 3. One hour later the retry ran as `bootstrap_incomplete` on the same run, in a new sandbox with the saved workspace restored. 4. Host preparation inspected the restored Runner state. The Runner was `suspended`, but the provider had no session identity. The inspection threw `provider_identity_incomplete (provider_not_suspended)` before the bootstrap restart branch ran. 5. The run failed. The task stopped at "Recovery needed" with only inspect actions. **Expected behavior** The restart sets the unopened session aside and opens a new provider session, as it already does for a Runner that never suspended. **Steps to reproduce** 1. Restart a remote run as `bootstrap_incomplete` with a proven bootstrap restart. 2. Give the sandbox a Runner state for this run with `lifecycle: suspended` and a provider state with no session identity. 3. Prepare the host. On `master` it throws `runner_harness_state_mismatch: provider_identity_incomplete`. The new `interrupted_bootstrap_unopened` cases of `bootstraps only an untouched provider session in a $outcome workspace lease` run these steps for a created and a resumed lease. **Paperclip version or commit** `master` at `7435abcc63`. ## What Changed - During host preparation, a Runner state that is suspended with no provider session identity returns the new incomplete reason `provider_unopened`, but only when all of these hold: - the run is restarting as `bootstrap_incomplete`; - the state belongs to this same run; - the controller already recorded the bootstrap restart proof (its durable journal shows no provider turn, and the old Runner is stopped or absent). - the provider state is the one saved before any session opened (`remoteProviderStateNeverOpened`): `prepared`, no session or identity, no turn in progress, no unconfirmed exit. Only ACPX and Codex-family states qualify. - The created-lease and resumed-lease restart branches share one check (`restartableBootstrapState`) and accept `provider_unopened` exactly like `not_suspended`. They move the old session aside and claim a new, untouched session root. - Every other case still throws `provider_identity_incomplete`: another run's state, a restart without the proof, a normal resume, and close checkpointing. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-session-executor.test.ts -t "bootstraps only an untouched provider session"`: 47 passed. - Red on `master`: the `created` and `resumed` `interrupted_bootstrap_unopened` cases fail with `runner_harness_state_mismatch: provider_identity_incomplete (identity_binding)`. Green with this change. - Fail-closed cases stay red-proof: `interrupted_bootstrap_unopened_unproven` (no restart proof) and `interrupted_bootstrap_unopened_other_run` (another run's state) still throw. - Full `native-session-executor.test.ts` and `pnpm --filter @paperclipai/server typecheck` pass. ## Risks - Low risk. The new branch needs the same proof that already authorizes a fresh bootstrap for a Runner that never suspended. The proof comes from the controller's durable journal, not from the sandbox. - The normal path is unchanged and has no added wait. - Not proven live: I did not confirm on staging that the controller recorded the restart proof for these runs. If it did not, the run still fails closed as before. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
a46b9ac724 |
fix(native): wait out an ACP usage limit's reported reset, then Retry the task (#20692)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs (Paperclip Runner with an ACP harness, such as Claude)
can stop because the provider account runs out of its usage allowance
> - The ACP harness reports this as a "terminal limit failure".
Paperclip blocked the task behind a board hold that called the session
state ambiguous, and the only way forward was a manual Retry
> - The provider also says when the limit resets, for example "resets
Nov 12, 6pm (UTC)" for a weekly limit. Paperclip read only clock-only
resets ("resets 2am"), so the agent-level wait fell back to one hour.
Each later wake for the agent then hit the limit again and blocked
another task
> - This pull request reads calendar resets, says on the hold when the
limit resets, and requests the task's Retry itself at that time when the
reset is reported and bounded
> - The benefit is that a usage limit no longer needs a person to click
Retry, and the agent's other tasks wait instead of failing one by one
## Linked Issues or Issue Description
**What happened?**
A native ACP run failed with `native_provider_usage_limit` ("ACP agent
reported a terminal limit failure. You've hit your weekly limit · resets
Nov 12, 6pm (UTC)"). The task went to `blocked` with a board-owned hold.
The hold said "Inspect the original provider failure and explicitly
resolve the ambiguous session state". Nothing resumed the task after the
reset. The agent-level `provider_limit_wait` recorded "no reset time was
reported, so Paperclip waits one hour", so every hour another wake
dispatched, hit the limit, and blocked another task. In one 48-hour
window on a staging instance, this blocked 7 tasks for one agent.
**What did you expect?**
When the provider reports a bounded reset, the task waits and continues
after the reset, like the existing `provider_limit_wait` path. When the
reset is unknown, the task fails with a clear message and a working
Retry.
**Steps to reproduce**
1. Run a native ACP agent until its provider reports a weekly or session
limit with a reset time.
2. Look at the task: it is blocked with an "ambiguous session state"
hold.
3. Wait past the reported reset: the task stays blocked until a person
clicks Retry.
Refs #20666 (other recovery work in the same staging census; not changed
here)
## What Changed
- `provider-quota-reset.ts`: parse calendar resets such as "resets Nov
12, 6pm (UTC)", "resets Jul 17th at 4:30 PM (America/Chicago)" and
"Resets on August 2, 9am (UTC)". A past date rolls into next year. An
impossible date (Nov 31) is refused. Clock-only parsing is unchanged.
This also fixes the agent-level `provider_limit_wait` reset and the
legacy `provider_quota` reset for these messages.
- New `native-runtime/provider-limit-resume.ts`:
- `readNativeUsageLimitReset` reads the reset only from the run's own
durable events. The last turn event must be a provider-observed
`turn.failed` with a `rate_limit` or `quota_exhausted` provider failure,
and the reset must be at most eight days out.
- `requestProviderLimitResumeRetry` asks for the task's Retry
(`requestTaskRetry`, `initiatedBy: "provider_limit_reset"`) on behalf of
the user who requested the run, or the run's accountable user for
automatic wakes. It is idempotent per run and records its outcome on the
hold. At most three automatic Retries follow one another without a
successful run.
- `registerProviderLimitResumeWork` is a scoped deadline owner (shared
work scheduler, no polling). Each active usage-limit hold is due at its
reset. Startup rebuilds it from the holds, and the executor notifies it
in the hold's transaction.
- `native-session-executor.ts`: for a terminal usage limit, the hold and
run failure detail say when the limit resets and whether Paperclip
retries then, instead of "ambiguous session state". The evidence records
`providerLimitResetAt`.
- `task-retry.ts`: accept `initiatedBy: "provider_limit_reset"`. Its
activity is recorded as system actor `provider-limit-reset` on behalf of
the user, like `restart_recovery`.
- `doc/execution-semantics.md`: document the wait and its limits.
Previous-release story: holds saved by the previous release carry
forward. At startup the owner reads each active
`native_provider_usage_limit` hold. It rewrites the hold's message with
the reset when the reset is known and still ahead, and it requests the
Retry at the reset. Holds whose reset is unknown, more than a day in the
past, or that have no accountable user keep their Retry and are recorded
as not requested, so they are not read again.
## Verification
- `npx vitest run server/src/services/provider-quota-reset.test.ts` — 4
tests. Three fail on `origin/master` code (calendar resets return
`null`), all pass with this change.
- `npx vitest run
server/src/services/native-runtime/native-provider-capacity.integration.test.ts`
— 5 tests on embedded Postgres:
- the existing usage-terminal test (both replay modes) still blocks with
a Retry, and no automatic Retry is requested without a reported reset;
- "waits for a reported usage-limit reset and then requests the task's
Retry once": the hold says when the limit resets, an old-release hold is
rewritten, nothing starts before the reset, one Retry receipt is created
for the accountable user at the reset, and a second pass does nothing.
Fails on `origin/master` executor code;
- "starts the Retry from the hold's own deadline": a real delivery
coordinator rebuilds the deadline from the hold and calls Retry dispatch
once, after the reset;
- "leaves a visible Retry when no user is accountable for the run".
Fails on `origin/master` executor code.
- Focused regression suites pass (642 tests): `task-retry`,
`quarantined-task-retry`, `heartbeat-native-restart-suspension`,
`native-startup-retry.integration`, `provider-failure-classification`,
`heartbeat-provider-limit-wait`, `deferred-execution-receipt-work`,
`chat-run-publications`, `native-runner-restart-recovery.integration`,
`native-remote-retry-admission`.
- `pnpm --filter @paperclipai/server exec tsc --noEmit` passes.
- `pnpm verify` selected a broad run (exit 2, because the docs file is a
global CI input). CI supplies the broad coverage.
## Risks
- The automatic Retry uses the accountable user's authority for
automatic (system) wakes. It is the same Retry the hold already offered
that user, and dispatch still checks Retry authority, stop proof, saved
work and action reconciliation. It is never broader than a click.
- A provider that keeps reporting resets that do not restore capacity
gets at most three automatic Retries in a row; then the Retry stays
visible.
- The deadline owner reads all active usage-limit holds at startup
(limit 500 rows). The number of such holds is small.
- Not changed: the agent-level wait still holds new wakes for at most
one day, as before.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, read-only staging API reads). Extended
reasoning enabled.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
|
a1ec05a1a4 |
fix(native): read settled harness state again after a dropped sandbox connection (#20690)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner keeps one provider conversation across the per-turn runs of a task. After each turn, the controller reads the settled Runner and provider state out of the sandbox and saves it as the harness backup. > - A per-turn sandbox is released after the turn. The harness backup is then the only copy of the provider session. > - On staging, one dropped sandbox connection (`write ECONNRESET`) during that read lost the backup. The run itself succeeded, because its result was already accepted. > - The next run of the task could not resume the conversation: `PRP provider resume state is unavailable`. When the task was waiting for a reply, the run stopped at "Recovery needed" with no Retry. A person had to send a message to start a new conversation. > - This pull request reads the settled state again when the sandbox connection drops during that read. > - The benefit is that one network reset no longer loses a task's conversation. ## Linked Issues or Issue Description Refs #20627 (warm backup restore when the source sandbox is idle) and #20683 (relaunch of a settled runner). This PR changes a different step: the backup capture at the end of a turn. **What happened?** A per-turn OpenCode task ran on a remote Daytona sandbox with reuse disabled. 1. Run A accepted its result. The Runner suspended. 2. The controller read the settled state out of the sandbox. The read failed with `remote checkpoint failed: write ECONNRESET`. `harness_state.backup.persist` and `session.checkpoint.persist` failed. 3. Run A still finished as `succeeded`, because the result was durable. The sandbox was released. No harness backup of turn A existed. 4. Run B started in a new sandbox. It failed in 0.2 s with `native_session_recovery_failed: Error: PRP provider resume state is unavailable`. When the prior turn yielded for a reply, the checkpoint policy does not allow a replacement session, so the task showed "Recovery needed" with only inspect actions. On staging this happened on 4 tasks in 48 hours. In all 3 predecessor runs that I could inspect, the backup capture failed. 2 of them show the `ECONNRESET` in the run log. No deploy happened between run A and run B in one case, so a deploy is not the cause. **Expected behavior** A transient connection drop while reading settled state does not lose the provider conversation. The read is repeated, and the backup is saved. **Steps to reproduce** 1. Run the first durable turn of a remote OpenCode session. 2. Make the sandbox `syncOut` of the backup directories fail once with `ECONNRESET`. 3. Call the settled checkpoint. On `master` it throws `write ECONNRESET`, and no backup is saved. The new test `uses the bundled Linux image authority for remote opencode-new-durable-reset without controller overrides` runs these steps. **Paperclip version or commit** `master` at `7435abcc63`. ## What Changed - The settled-state capture (inspect plus `syncOut` of each persistence directory into a pending directory) is now one step that can repeat. The step writes nothing durable, so it is safe to repeat. - A dropped, reset or refused sandbox connection repeats the step, at most 3 attempts, with 1 s and 2 s delays. Each repeat clears the pending directory and inspects the live state again, so the copy always matches the inspected identity. - Any other failure fails at once, as before: a state mismatch, a missing directory, an unsafe archive, or a TLS failure. - The durable backup upload and the local publish run only after a complete capture. They did not change. - `isTransientRemoteCheckpointReadFailure` classifies the error with the existing `classifyRemoteConnectionError`, plus a message match for SDK errors that lose their `code`. - The first completeness inspection and its visibility-window re-reads use the same bounded retry (`retryTransientRemoteCheckpointRead`). A dropped provider-state read keeps its `provider_state_unreadable` refusal for every other caller, but the checkpoint retry looks through it to the dropped read. Undici `fetch failed` and `no response returned` also count as drops. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-session-executor.test.ts`: 1343 passed. - Red on `master`: the new `opencode-new-durable-reset` case fails with `Error: write ECONNRESET`. Green with this change. - The `isTransientRemoteCheckpointReadFailure` test checks that state mismatches, unsafe archives and TLS failures do not repeat. - `pnpm --filter @paperclipai/server typecheck` passes. ## Risks - Low risk. The normal path has no added wait. A repeat adds at most 3 s, and only after a connection failure. - A repeated read inspects the live state again. If the state changed between attempts, the existing identity checks still refuse it. - A connection that stays down for all 3 attempts fails the same way as before. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
2eede8b318 |
fix(native): give a relaunched settled runner its admitted provider pack (#20683)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - After a server restart or a same-run retry, a remote native run reattaches to its runner in the sandbox (`reattach_remote_runner`). > - When the controller saved the runner artifact, the executor prepares that run as an adoption. It skips provider-pack selection, because a live adopted runner keeps its launched provider. > - The reattach probe can instead find the runner settled and exited (`verifyRemoteRunnerOrSettledExit`). Then nothing is adopted, and the transport relaunches the runner from its durable state. That relaunch must start the provider again, so it needs the provider-pack launch inputs. > - The executor did not give them. OpenCode failed with "OpenCode executable is missing from the provider pack". ACPX failed with "ACPX launch profile omitted its provider-pack authority or verified artifact digests". Both ended the run. > - This pull request selects the saved (admitted) provider-pack authority on that settled-exit relaunch and verifies it as an exact installed identity, as an ordinary reattach does. > - The benefit is that a restart or a same-run retry can resume an exited runner with the same provider pack, instead of failing. ## Linked Issues or Issue Description **What happened?** On a hosted staging instance, two OpenCode runs and one ACPX (Claude) run were interrupted by a deploy (SIGTERM) or ended a turn without a result. The server scheduled a same-run retry that reattached to the existing remote runner. The runner had exited cleanly (`runner_remote_process_exited lifecycle=suspended`). The retry then failed within three seconds with `runner_remote_provider_artifact_incompatible`. **Steps to reproduce** 1. Prepare a remote OpenCode or ACPX run with a persisted runner artifact (`controllerPersistence.runnerArtifact`). 2. Reattach with `restartRecovery.kind = "reattach_remote_runner"`, where the runner process is not running and its state is `suspended` and settled. 3. Observe that the transport options have no `adoptExistingRunner`, no `providerPackAuthorityDigest` and no `opencodeCommand` / `acpxSidecarSha256`. **Expected behavior** The relaunch gets the admitted provider pack's launch inputs, or it fails with a clear pack-identity error. ## What Changed - `createRunnerdBackend` (native session executor): when a persisted-runner reattachment finds no live runner to adopt, select the saved provider-pack authority (`reattach: true`), so `expectedProviderPackManifest` is the admitted pack. - `prepareRemoteRunnerAssets` no longer returns early for that relaunch. It verifies the installed pack as an exact identity and sets the active pack root, the same path as a reattach without a persisted artifact. - A live adopted runner is unchanged: it still skips pack selection. - The test `uses the bundled Linux image authority for remote opencode|claude-x64` now also covers the settled-exit relaunch for OpenCode and ACPX. ## Verification - RED on `origin/master` (`c444da7d05`): the extended test fails for `opencode` and `claude-x64` (the relaunch options have no provider-pack inputs). - GREEN with this change: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-session-executor.test.ts -t "uses the bundled Linux image authority"` (14 passed). - `pnpm --filter @paperclipai/server typecheck` passes. - `pnpm verify`: see the result below. ## Risks - Upgrade safety: the relaunch reads the saved authority, so a newer controller does not select a newer pack for an older session. It requires an exact installed identity. If the sandbox no longer has that pack, the relaunch fails with a pack-identity error, and the existing Retry path applies, as before. - No change to a live adoption, to fresh runs or to the normal path before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
cec903c6f0 |
fix(native): retire a finished remote owner whose gone Runner left a delivery unacknowledged (#20677)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native follow-up run on a remote sandbox first proves that the previous per-turn owner finished. Then it retires that owner's controller journal and starts a new Runner. > - The proof requires a settled journal. A command that the Runner did not acknowledge holds the owner. > - On staging, a per-turn owner committed its result and its Daytona sandbox was destroyed after the turn (`paperclip_runner_destroy_after_turn`, cleanup `success`). Its journal still held a `run.attach` or a `semantic_tool.result` that the Runner never acknowledged. > - The next run failed with `prior_owner_terminal_state_indeterminate; retirement_refused=journal_commands_unsettled(run.attach.pending)` (or `semantic_tool.result.pending`). A Retry or a new message gets the same journal, so it fails the same way. The task stays blocked. > - Retirement already excuses the Runner's own lifecycle commands when the prior sandbox lease ended on another sandbox. This pull request applies the same rule to deliveries that only that gone Runner could receive. > - The benefit is that these follow-ups start by themselves. Commands that carry user intent still hold the owner. ## Linked Issues or Issue Description **What happened?** A census of staging failures (48 h, Nov 9–10) found 6 failed runs with `prior_owner_terminal_state_indeterminate` and `retirement_refused=journal_commands_unsettled(run.attach.pending | semantic_tool.result.pending)`. The last one was at 18:56 UTC on Nov 10. In two examples that I checked: - The previous run succeeded. Its result was committed and its coordinator lease was released. - Its lease policy was `ephemeral`. The lease is `expired`, with `failureReason=paperclip_runner_destroy_after_turn` and `cleanupStatus=success`. - The follow-up got a new sandbox (a different provider lease) and refused in about 11 s. It started no Runner. The task became `blocked`, and the next action was "Send a message to continue". **Expected behavior** When the record proves that the previous Runner is gone, an undelivered attachment, tool answer or read cannot run. The follow-up retires the finished owner and starts a new Runner. Retirement replays no command. **Steps to reproduce** 1. Finish a per-turn remote native run whose controller journal keeps a `run.attach` or `semantic_tool.result` command in `pending`. 2. Let the sandbox lease of that run end (`expired` or `released`) on its own sandbox. 3. Start a follow-up run that gets a new sandbox. 4. Observe `retirement_refused=journal_commands_unsettled(run.attach.pending)`. **Paperclip version or commit** Staging builds of Nov 10. The refusal still reproduces on `master` at `48c6e996da` (the new tests fail there). ## What Changed - `native-session-executor.ts` (`readSettledRunnerdProjection`): when the prior run's sandbox leases ended on another sandbox (`remoteRunnerGone`, the existing proof), unfinished `run.attach`, `semantic_tool.result`, `session.snapshot` and `session.goal.get` commands no longer hold the retirement. These join the lifecycle commands that the rule already excused. - In the same case, a journaled `semantic_tool.result` answers its `semantic_tool.input`. The controller journals the answer after the tool ran. Only its delivery was outstanding. - No other proof changes. The prior run must still be terminal, with a committed result, a released coordinator lease, no recovery state, a stopped or fenced controller, a delivered `run.terminal`, and no live work after it. Quarantine copies must pass the same checks. - `doc/execution-semantics.md`: describe the rule. ## Verification - `server/src/services/native-runtime/native-recovered-owner-followup.test.ts` (real Postgres): - New: "retires a finished remote owner with an unacknowledged pending-attach / pending-tool-result when its sandbox lease ended on another sandbox". It fails on `master` with the exact staging errors `journal_commands_unsettled(run.attach.pending)` and `journal_commands_unsettled(semantic_tool.result.pending)`. It passes with this change, and the log shows the retirement. - Held cases stay held: a lease that is still retained, a lease on this run's sandbox, no lease record, and a tool input with no journaled answer (`journal_tool_input_unanswered`). - New: "keeps a stopped local owner held by an unacknowledged run.attach". A local Runner is not gone by record. - Whole file: 95 passed. - Neighbouring suites pass: `explicit-native-continuation.test.ts` and `native-controller-recovery.integration.test.ts` (460 tests). - `npx tsc --noEmit -p server/tsconfig.json` passes. - `pnpm verify` exits 2: a global CI input changed (`doc/execution-semantics.md`). Required CI supplies the broad coverage. - Live proof: not done. The redeploy chaos scenario (K15) is not on `master` yet. Tests only. ## Risks - The rule applies only when every sandbox lease of the prior run is `released` or `expired`, has a release time, and names a different provider lease than this run's. A reusable sandbox that this run resumed keeps the existing hold. - A command that carries user intent (`turn.start`, `turn.steer`, `request.resolve`, `interaction.receipt`, `external_provider.operation`, `session.goal.set`/`clear`, `session.close`, `run.cancel` and others) still holds the owner. - Retirement archives the old journal under `continuity-breaks` and replays nothing, as before. - No persisted shape changes. Old saved journals are read by the same rule. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
c14f2fe45d |
fix(native): read the scheduled-resume evidence the resume reconciler claimed (#20656)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - #20645 lets a local same-run scheduled resume send the one same-conversation continuation for a turn lost on restore. It applies only when durable records prove three facts: the previous runner stopped, the attempt was admitted within the budget, and it is the same checkpointed conversation. > - It read the third proof from the coordinator's saved `failure_detail` while the phase was `retryable_failure`. > - In production, the resume reconciler (`claimNativeSessionResumptions`) claims the run first. It moves the coordinator to `observed` and clears `failure_detail`. So the executor never saw that evidence, and #20645 refused every real scheduled resume. > - The restart-chaos L7 wave showed this live. Codex B5 TM2 hit the exact path (`runner.drain` timeout, scheduled resume, `provider_turn_lost_on_restore`), sent no continuation, and needed Retry. > - This pull request makes the reconciler record that evidence durably when it claims the run, and makes the executor read it from there. > - The benefit is that #20645 works on the real path. The proof rule is unchanged. ## Linked Issues or Issue Description Refs #20645 **What happened?** Wave `w-20261110T212307-s9107` (L7, seed 9107, local, master plus #20645), task codex B5 TM2: - `native_session_recovery_failed: runner_local_connect_failed: PRP command runner.drain timed out`; - `retryable_failure` with `recoveryMode: "exact_checkpoint_resume"`, `recoverable`, `providerSessionEstablished` and `checkpointExists` all set; - `native.local_process_stopped`; - attempt 2 started; - `turn.failed` with `provider_turn_lost_on_restore` for the same turn ID that the checkpoint was running. No continuation was sent. The run failed with `native_provider_terminal_failed` and was preserved with `uncertain_provider_action`. **Expected behavior** The proven scheduled resume sends the single continuation. **Steps to reproduce** The new integration test below. The live path is timing-dependent (the L7 command in #20645). ## What Changed - `native-finalization-reconciler.ts`: when it claims a same-run resume, the reconciler appends one `same_run_resume` entry to `recovery_history`. The append is bounded like other history writes, and it runs in the same update that clears the failure. The entry copies `recoveryMode`, `recoverable`, `providerSessionEstablished`, `checkpointExists`, `successorRunId` and `replacementDenied` from the saved failure, with the controller boot and provider attempt. - `scheduled-resume-continuation.ts`: the evidence for the third proof now comes from either of two durable records: - the saved failure, when the executor claims a `retryable_failure` coordinator directly (unchanged); or - a reconciler claim: an `observed` coordinator whose last history entry is a `same_run_resume` entry naming the executor's own controller boot. The two other proofs are unchanged: the stop receipt, and an admitted attempt from 1 to 3. - `native-session-executor.ts`: passes the coordinator's last history entry and the controller boot to the decision. - `doc/execution-semantics.md`: describes the recorded evidence. Readers of `recovery_history` that look at the last entry do so only for restart claims, Stop recovery and shutdown suspension. Restart recovery's previous-claim lookup skips non-claim dispositions, so a `same_run_resume` entry does not change those paths. **Still refused** (tested): - every #20645 case; - the last history entry is not the resume claim, or there is no history; - the resume claim was recorded by another boot; - the claimed evidence is a bootstrap retry, has no checkpoint, or records a replacement; - a reconciler-claimed resume without a stop receipt. ## Verification - `native-scheduled-resume-continuation.integration.test.ts` now also runs through the real reconciler claim (`claimNativeSessionResumptions`) before `executePaperclipNativeSession`. On master the reconciler-claimed allowed case fails (`resumeInterruptedTurn` is false). With this change it is true, and the claim admits attempt 2. A reconciler-claimed resume without a stop receipt still passes false. - `scheduled-resume-continuation.test.ts`: decision cases for the reconciler-claim entry. - `native-session-resumption.test.ts` passes. 54 tests pass across the three files. - `cd server && npx tsc --noEmit` passes. - Live: not yet. The failing wave above is the reproduction. A rerun is timing-dependent. ## Risks - One more `recovery_history` entry per scheduled resume, inside the existing 20-entry bound. - The proof rule itself is unchanged. This only reads the same evidence from the record where it now survives. - No schema change. A run claimed by an earlier build has no `same_run_resume` entry, so it keeps the existing failure: refused, with Retry offered as today. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). Extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
805531cc71 |
fix(native): never leave a cleanup-gated task without Retry or an owner (#20639)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs close their provider session at the end of each attempt.
When close cannot prove that the runner stopped, the runner keeps a
cleanup gate (`native_session_cleanup_quarantined`) in server memory
> - #20572 hid Retry for every gated run and said "Paperclip finishes
the cleanup automatically". Only a remote environment release settles
the gate. A local gate that needs an operator never settles, and a
server restart loses it
> - A restart-chaos wave on master found two tasks that rested blocked
with no Retry, no scheduled retry and no recovery owner. A person could
not continue them
> - This pull request makes the run's own recovery retire the gate when
the host proves the old processes stopped. It also offers Retry whenever
nothing settles the gate, and it makes that Retry work
> - The benefit is that a cleanup-gated task either continues by itself
or shows a Retry that starts. It never rests without both
## Linked Issues or Issue Description
Refs #20572, #20578
**What happened?**
Restart-chaos wave `w-20261110T185144-s9107` (local, claude, fault TM2:
the server is paused with SIGSTOP for about 33 s). Tasks t04 and t05
completed their provider turn. The runner checkpoint owner lapsed while
the server was paused (`native_runner_checkpoint_owner_lost`). The
session close then failed with `native_session_close_unrecoverable:
runner did not durably suspend before checkpoint`. The provider had
drained and no tool call was pending. The runner kept an
operator-required cleanup gate. Thirty seconds later the run's own
same-run recovery verified that the runner process and group had stopped
(`native.local_process_stopped`). The gate still refused the recovery,
and the run ended `failed` with `native_session_cleanup_quarantined`.
The task showed `canRetry: false` and a cleanup wait until the deadline.
Nothing owned that wait.
**Expected behavior**
When the old runner and provider processes are proven stopped, the run's
own recovery continues. When nothing settles the gate, the task offers a
Retry that starts.
**Steps to reproduce**
1. Run restart-chaos L7: `--env local --adapters codex,claude
--workloads A1,B5,C1 --faults TM1,TM2,TM3 --time-age-hours 1
--concurrency 3 --tasks 6 --seed 9107`.
2. Observe a claude TM2 task rest `blocked` with `canRetry=false`,
`retryWait=true` and no scheduled retry.
**Paperclip version or commit**
master at
|
||
|
|
37c50f70c2 |
fix(native): continue a turn lost on restore in a proven same-run resume (#20645)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run keeps its provider conversation when its local runner dies. When the restored runner finds that the active turn was lost, it reports `provider_turn_lost_on_restore` and does not invent a result. > - After a control-plane restart, the admitted restart attempt may send one continuation in that same conversation (#15297). The agent then reconciles unfinished work. The task is not resent. > - A runner can also be lost without a server restart, for example when `runner.drain` times out. The run then resumes itself after the retry delay from its exact checkpoint. That path never allowed the continuation, because the executor passed `resumeInterruptedTurn` only for a `resume_dead_runner` restart claim. > - So the lost turn failed with `native_provider_terminal_failed`, automatic replacement was refused (`uncertain_provider_action`), and the user had to click Retry. > - This pull request allows the same single continuation for that resume, but only when durable records prove the same three facts that a restart claim proves. > - The benefit is that a runner lost mid-turn continues in the same conversation with no Retry. No case that is refused today becomes allowed without that proof. ## Linked Issues or Issue Description Refs #15297, #20625 **What happened?** Restart-chaos wave L7 (seed 9107, local), task codex B5 TM2, run on master after #20625. A TM3 restart resumed the run (attempt 1). During the TM2 freeze the TS runtime lost its runner: `native_session_recovery_failed: runner_local_connect_failed: PRP command runner.drain timed out`. The executor saved `retryable_failure` with `recoveryMode: "exact_checkpoint_resume"`. After the delay, recovery verified the runner process stopped (`native.local_process_stopped`). The same run then resumed as attempt 2 (`native.process_start_requested`, `providerAttempt: 2`). The restored runner reported `turn.failed` with `provider_turn_lost_on_restore` (`recoverable: true`). No continuation was sent. The run failed with `native_provider_terminal_failed`, recovery stopped with `uncertain_provider_action` / `preserve_without_replay_v1`, and the task needed Retry. **Expected behavior** The run sends the one same-conversation continuation, exactly as it does after a restart claim, and finishes without Retry. **Steps to reproduce** `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex,claude --workloads A1,B5,C1 --faults TM1,TM2,TM3 --time-age-hours 1 --concurrency 3 --tasks 6 --seed 9107` (timing-dependent: the drain timeout must land during an active turn). ## What Changed - New `server/src/services/native-runtime/scheduled-resume-continuation.ts`. It decides whether a same-run scheduled resume may continue a lost turn. Each of the three proofs comes from a durable record: 1. **The previous runner is stopped.** The latest process record of the run in `heartbeat_run_events` (`native.process_start_requested`, `native.process_identity_recorded`, `native.local_process_stopped`) must be `native.local_process_stopped`. Its `processPid` must equal its `processGroupId`, and neither the pid nor the group may be alive now. This is the existing `readNativeLocalProcessStop` check, read in the claim transaction before this attempt records its own `native.process_start_requested`. 2. **The attempt was admitted under the bounded budget.** The executor's claim moves `native_run_finalizations` from phase `retryable_failure` to `observed`. It charges a provider attempt, and the claim refuses any attempt above 3 (`native_session_retry_exhausted`). The decision also requires `1 <= attempt <= NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT`. 3. **It is the same provider conversation.** The saved `failure_detail` must show `recoveryMode: "exact_checkpoint_resume"`, `recoverable: true`, `providerSessionEstablished: true` and `checkpointExists: true`, with no `successorRunId` or `replacementDenied`. At run time the existing runtime guards still apply: the backend must recover the checkpointed session (`recovered`, not a continuity-break replacement), and the lost turn must equal the checkpoint's `activeTurnId` and the snapshot's interrupted turn. The provider kinds stay codex, acpx and opencode, with no compaction, goal control, held result or pending runtime request. - `native-session-executor.ts`: computes that decision in the coordinator claim. It passes `resumeInterruptedTurn` when either the restart claim is `resume_dead_runner` (unchanged) or the decision allows it. - `doc/execution-semantics.md`: describes the rule and the refused cases. **Cases that stay refused** (each has a test in `scheduled-resume-continuation.test.ts`): - a restart-recovery claim (it keeps its own existing rule); - a remote execution target; - a coordinator that is not `retryable_failure`, including a fresh run; - `recoveryMode` `bootstrap_retry` or `ambiguous_state`, or no saved failure detail; - a failure that is not recoverable; - no established provider session; - no checkpoint; - a recorded replacement (`successorRunId` or `replacementDenied`); - an attempt outside 1..3; - no stop receipt, a stop receipt superseded by a newer launch, a stopped process that is still alive, or a run that never had a process. The runtime-side refusals are unchanged and already covered by `native-session-runtime.test.ts`: real provider errors, historical error-null failures, and no restart permission. ## Verification - `cd server && npx vitest run src/services/native-runtime/scheduled-resume-continuation.test.ts`: 22 tests, covering the decision table and the stop receipt in embedded Postgres. - `cd server && npx tsc --noEmit` passes. - Live: not proven. An L7 wave on this branch (seed 9107) did not reproduce the drain-timeout path, so this pull request has unit and database tests only. In that wave the same cell (codex B5 TM2) needed Retry for a different reason: TM2 aged its restart-recovery claim lease before dispatch, and budget admission refused with `Native recovery no longer owns provider dispatch`. This pull request does not change that path. Four other tasks passed. Claude C1 TM2 rested blocked without a Retry button, which is a separate cleanup-quarantine issue. ## Risks - This allows one more path to send the same-conversation continuation. It is limited to local runs. It needs the same proofs as a restart claim, and it uses the attempt the resume already charged. It does not change the generation cap or `restartResumeKeepsAttempt`. - A rolling deploy reads `failure_detail` and stop events written by earlier releases. Those fields and events already have this shape, so an older saved failure is judged by the same rule. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). Extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |