Files
PaperClipAI/server
DottaandClaude cec903c6f0 fix(native): retire a finished remote owner whose gone Runner left a delivery unacknowledged (#20677)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native follow-up run on a remote sandbox first proves that the
previous per-turn owner finished. Then it retires that owner's
controller journal and starts a new Runner.
> - The proof requires a settled journal. A command that the Runner did
not acknowledge holds the owner.
> - On staging, a per-turn owner committed its result and its Daytona
sandbox was destroyed after the turn
(`paperclip_runner_destroy_after_turn`, cleanup `success`). Its journal
still held a `run.attach` or a `semantic_tool.result` that the Runner
never acknowledged.
> - The next run failed with `prior_owner_terminal_state_indeterminate;
retirement_refused=journal_commands_unsettled(run.attach.pending)` (or
`semantic_tool.result.pending`). A Retry or a new message gets the same
journal, so it fails the same way. The task stays blocked.
> - Retirement already excuses the Runner's own lifecycle commands when
the prior sandbox lease ended on another sandbox. This pull request
applies the same rule to deliveries that only that gone Runner could
receive.
> - The benefit is that these follow-ups start by themselves. Commands
that carry user intent still hold the owner.

## Linked Issues or Issue Description

**What happened?**
A census of staging failures (48 h, Nov 9–10) found 6 failed runs with
`prior_owner_terminal_state_indeterminate` and
`retirement_refused=journal_commands_unsettled(run.attach.pending |
semantic_tool.result.pending)`. The last one was at 18:56 UTC on Nov 10.
In two examples that I checked:
- The previous run succeeded. Its result was committed and its
coordinator lease was released.
- Its lease policy was `ephemeral`. The lease is `expired`, with
`failureReason=paperclip_runner_destroy_after_turn` and
`cleanupStatus=success`.
- The follow-up got a new sandbox (a different provider lease) and
refused in about 11 s. It started no Runner. The task became `blocked`,
and the next action was "Send a message to continue".

**Expected behavior**
When the record proves that the previous Runner is gone, an undelivered
attachment, tool answer or read cannot run. The follow-up retires the
finished owner and starts a new Runner. Retirement replays no command.

**Steps to reproduce**
1. Finish a per-turn remote native run whose controller journal keeps a
`run.attach` or `semantic_tool.result` command in `pending`.
2. Let the sandbox lease of that run end (`expired` or `released`) on
its own sandbox.
3. Start a follow-up run that gets a new sandbox.
4. Observe
`retirement_refused=journal_commands_unsettled(run.attach.pending)`.

**Paperclip version or commit**
Staging builds of Nov 10. The refusal still reproduces on `master` at
`48c6e996da` (the new tests fail there).

## What Changed

- `native-session-executor.ts` (`readSettledRunnerdProjection`): when
the prior run's sandbox leases ended on another sandbox
(`remoteRunnerGone`, the existing proof), unfinished `run.attach`,
`semantic_tool.result`, `session.snapshot` and `session.goal.get`
commands no longer hold the retirement. These join the lifecycle
commands that the rule already excused.
- In the same case, a journaled `semantic_tool.result` answers its
`semantic_tool.input`. The controller journals the answer after the tool
ran. Only its delivery was outstanding.
- No other proof changes. The prior run must still be terminal, with a
committed result, a released coordinator lease, no recovery state, a
stopped or fenced controller, a delivered `run.terminal`, and no live
work after it. Quarantine copies must pass the same checks.
- `doc/execution-semantics.md`: describe the rule.

## Verification

-
`server/src/services/native-runtime/native-recovered-owner-followup.test.ts`
(real Postgres):
- New: "retires a finished remote owner with an unacknowledged
pending-attach / pending-tool-result when its sandbox lease ended on
another sandbox". It fails on `master` with the exact staging errors
`journal_commands_unsettled(run.attach.pending)` and
`journal_commands_unsettled(semantic_tool.result.pending)`. It passes
with this change, and the log shows the retirement.
- Held cases stay held: a lease that is still retained, a lease on this
run's sandbox, no lease record, and a tool input with no journaled
answer (`journal_tool_input_unanswered`).
- New: "keeps a stopped local owner held by an unacknowledged
run.attach". A local Runner is not gone by record.
  - Whole file: 95 passed.
- Neighbouring suites pass: `explicit-native-continuation.test.ts` and
`native-controller-recovery.integration.test.ts` (460 tests).
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- `pnpm verify` exits 2: a global CI input changed
(`doc/execution-semantics.md`). Required CI supplies the broad coverage.
- Live proof: not done. The redeploy chaos scenario (K15) is not on
`master` yet. Tests only.

## Risks

- The rule applies only when every sandbox lease of the prior run is
`released` or `expired`, has a release time, and names a different
provider lease than this run's. A reusable sandbox that this run resumed
keeps the existing hold.
- A command that carries user intent (`turn.start`, `turn.steer`,
`request.resolve`, `interaction.receipt`, `external_provider.operation`,
`session.goal.set`/`clear`, `session.close`, `run.cancel` and others)
still holds the owner.
- Retirement archives the old journal under `continuity-breaks` and
replays nothing, as before.
- No persisted shape changes. Old saved journals are read by the same
rule.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 18:49:26 -06:00
..