Files
PaperClipAI/doc
DottaandClaude Opus 5.5 fc7e3a6503 fix(recovery): never replace a native session after a deterministic refusal (#19733)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs keep a saved provider session, and same-run
recovery resumes it after a failure
> - When resume fails and the checkpoint allows it, recovery archives
the saved continuity state and opens a fresh replacement session
> - Recovery used that path for every resume failure, including
deterministic refusals that left the saved state unchanged and repeat on
every retry
> - In five staging tasks a refusal (a protected launch argument, or a
predecessor tool call with no result) escalated into a replacement. The
remote archive move failed part way, the next attempt found no runner
state, and each task was blocked for workspace repair
> - This pull request stops before replacement after a closed-class
refusal, keeps the saved state, and hands the run to an operator with a
named reason
> - The benefit is that a deterministic refusal can no longer damage
saved state or open a replacement over an unsettled external action

## Linked Issues or Issue Description

Refs #19659, Refs #19516, Refs #19334

**What happened?**
Five Codex tasks on Daytona sandboxes failed on staging. The first
attempt failed with a deterministic refusal:
- `run.attach cannot change protected launch arguments` (four tasks,
fixed separately in #19659), or
- `native_runner_source_semantic_reconciliation_required` with a
predecessor `paperclip_call_assigned_tool` call still pending (one
task).

The same-run retry then opened a replacement session.
`archiveContinuityState` failed with
`runner_continuity_break_archive_failed` and no detail. The third
attempt failed with `runner_remote_state_unavailable: exit=1`, and the
task was blocked with "Workspace repair required".

**Steps to reproduce**
1. Save a native session whose checkpoint policy is
`allow_replacement_after_resume_failure`.
2. Make the resume fail with a refusal that does not change state (an
attach refusal, or a pending tool call).
3. Same-run recovery opens a replacement session and archives the saved
continuity state.

**Expected behavior**
A deterministic refusal stops the run for an operator. The saved state
stays intact. A run with a pending side-effecting call never reaches
replacement. Transport drops keep the existing retry behavior.

**Root cause**
`executeNativeSession` checked only the checkpoint's replacement policy.
The closed failure class (#19334) only had an effect when replacement
was forbidden. Attach refusals and pending tool calls had no class at
all, and the source-reconciliation refusal was a plain `Error`.

## What Changed

- Runner (`provider_backend.rs`, `durable/mod.rs`): `run.attach`
refusals that would change the durable Codex profile or a protected
launch argument carry the closed classification
`provider_attach_refused`. An attach blocked by
`durable_pending_tool_calls` carries `provider_side_effect_pending`.
These refusals already leave the saved state unchanged.
- Contracts (`harness-driver.ts`): new closed classes
`PROVIDER_ATTACH_REFUSED` and `PROVIDER_SIDE_EFFECT_PENDING`. A typed
`NativeRunnerSourceReconciliationRequiredError` replaces the plain
`Error` at the five `native_runner_source_*_reconciliation_required`
throw sites. `runnerRecoveryFailureClass` reads the authenticated
command-result classification or the typed error, never message text.
- Runtime (`native-session-runtime.ts`): after either class, recovery
never opens a replacement session, whatever the checkpoint policy.
- An attach refusal ends with the existing
`NativeProviderStateIncompatibleError`, which maps to the existing
operator path `native_provider_state_incompatible`.
- A pending side effect ends with the new
`NativeProviderSideEffectPendingError`.
- Server (`native-session-executor.ts`, `native-session-resume.ts`,
shared `EXECUTION_RECONCILIATION_CAUSES`):
`native_provider_side_effect_pending` is a permanent failure and an
execution-reconciliation cause, so the board gets a reconciliation
(record whether the action ran) instead of a generic Retry. The
reconciled continuation does not rebind the saved session, which still
refuses attachment. It starts a fresh provider session in the same
workspace. The saved session and workspace are unchanged.
- Archive (`archiveContinuityState`): a failed remote move now records
the bounded (512 characters), redacted exit status and output in
`runner_continuity_break_archive_failed: exit=… timedOut=… <output>`.
The remote move already ran before the local move on `master`, so a
failed remote move leaves the local state in place.
`restart-interrupted-turn.ts` accepts the detailed message for its
reattachment-loss retry.
- Docs: `doc/execution-semantics.md` and
`packages/paperclip-runner/docs/durable-recovery.md` describe the new
rule.

## Verification

Red before the fix, green after (each test first run against unchanged
source from `origin/master` 644ee66f5d):

- `native-session-runtime.test.ts`, test "never replaces a session after
a deterministic, non-destructive recovery refusal". Red: replacement was
attempted (`TypeError: Invalid value used as weak map key` from the stub
replacement). Green: typed error, and `openReplacementSession`,
`openSession` and `openRun` are not called.
- `harness-driver-backend.test.ts`, test "keeps the runner's closed
class of a failed recovery". Red: the typed error did not exist. Green:
both new classes are read from the command result and from the typed
error. The same message text without the type stays unclassified.
- `tests/codex_provider.rs` (real durable executor and fake Codex
app-server):
-
`attach_with_a_pending_tool_call_is_classified_as_a_pending_side_effect`.
Red: `left: None, right: Some("provider_side_effect_pending")`.
-
`saved_codex_thread_attaches_after_a_release_changes_its_feature_toggles`
(protected sandbox change). Red: `left: None, right:
Some("provider_attach_refused")`.
- `native-session-executor.test.ts`:
- Test "stops same-run retries for an operator while a saved
side-effecting call has no result". Red: `expected
'native_session_interrupted' to be
'native_provider_side_effect_pending'`.
- Test "retains the local controller checkpoint until remote continuity
archiving succeeds: failed/timeout". Red: the message had no detail.
Green: the message has the exit status and output, a token in the output
is redacted, and the local runner and control-plane files are intact.

Safety and regression checks:
- New test "still replaces a session after an unclassified resume
failure when the checkpoint allows it". It passes before and after: a
transport drop still reaches replacement.
- Existing test "types a runner-proven incompatible session when
same-session recovery is required" still passes.
- `native-session-resume.test.ts`: a source stopped with
`native_provider_side_effect_pending` is not rebound.
- `run-completion.test.ts` "replays saved user intent through normal
Retry …" passes with the detailed archive message.
- Runner Vitest: `native-session-runtime.test.ts` and
`harness-driver-backend.test.ts` (266 passed);
`runnerd-codex-transport-warm-attachment.test.ts` and
`codex-app-server-driver.recovery.test.ts` (74 passed).
- Rust: `--lib` (560 passed, plus one ACPX timing test that failed under
load and passed when rerun alone), `codex_provider` (118 passed),
`native_provider_backend` (45 passed), `durable_recovery` (3 passed).

Not run: a live staging recovery at this revision. The local server
typecheck could not resolve the unbuilt runner package, so required CI
provides typecheck.

## Risks

- A run that a replacement session would have continued after an attach
refusal now stops for an operator. The refusal repeats on every retry,
so the replacement was the only way past it. It also destroyed saved
state, which is what blocked the staging tasks. A board Retry still
continues the task in a fresh session in the same workspace. This is the
existing `native_provider_state_incompatible` behavior.
- A new failure code `native_provider_side_effect_pending` appears in
run records and recovery actions. It joins
`EXECUTION_RECONCILIATION_CAUSES`, so the UI shows the reconciliation
flow, not Retry.
- An attach refusal used to reach an automatic replacement; it now needs
one board Retry. This is intended: the replacement archived saved state
and failed part way on staging.
- The pending-side-effect class means "no result delivered to the
runner". The controller replays a recorded tool outcome only after
attach. A call whose outcome the server already recorded therefore also
stops for an operator. That is conservative; the reconciliation shows
the recorded outcome.
- Only a pending tool call classifies an attach blocker. Other blockers
(active turn, ambiguous turn start, active goal) keep their current
behavior.
- The archive error message is longer. The one exact-match consumer now
accepts the `code: detail` form.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser reads of staging).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 23:25:54 -06:00
..
…
…
…
…
…