Commit Graph
476 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 2178c3df88 fix(native): keep a slow but live Runner on retry instead of replacing it (#21462)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a recovery read to a remote Runner goes unanswered, the next
attempt of the same run carries `replaceUnansweredRunner`
> - That attempt verified the Runner was alive and then refused to adopt
it anyway. On a resumed sandbox lease the reuse check then failed with
`resumed_not_suspended` and left the Runner running. Otherwise a second
Runner was launched, and its launch fence killed the busy Runner, which
lost the provider turn.
> - After a late reconnect a Runner is often slow, not dead: it is still
replaying the events it saved with no controller attached
> - This pull request makes the attempt check progress first. It adopts
a live Runner that is making progress and replaces only a Runner that
made no progress.
> - The benefit is that a slow but working Runner keeps its turn, and
the retry no longer causes the failure it was meant to fix.

## Linked Issues or Issue Description

**What happened?**

Hosted staging, release swaps on 2026-11-13. The retry after a
first-attempt timeout failed with these codes (counts from the traced
sample of 136 failures):

- `runner_ingress_unavailable: runnerd process failed:
runner_harness_state_mismatch: resumed_not_suspended` (part of 40
`ingress` failures)
- `runner_remote_process_identity_unavailable` (25): the Runner was gone
by the time the retry checked, because the launch fence or a previous
retry had killed it
- "restored the conversation after process loss, but…" /
`provider_turn_lost_on_restore` (10): the replacement Runner restored
from the old state and the provider turn was gone

Runs that crossed a swap with a reconnect gap under 150 s: 0 of 9
failed. Over 420 s: 14 of 15 failed. The process ID of the Runner
changed inside the same sandbox in the failed runs, which shows the
replacement.

Code path: `nativeRecoveryCommandUnanswered` grants an extra attempt
with `replaceUnansweredRunner: true`. `verifyRemoteRunnerOrSettledExit`
verifies the Runner is alive and then returns `undefined` because of
that flag, so `adoptedProcess` is `undefined`.

**Expected behavior**

A live Runner that is making progress is adopted. A Runner that made no
progress is stopped with proof and then restored from its own journal.

**Steps to reproduce**

1. Reattach a remote run whose previous attempt left a recovery read
unanswered (`replaceUnansweredRunner: true`), with the Runner alive and
its saved event sequence past the controller's last acknowledgement.
2. On master the attempt launches a second Runner, or fails with
`resumed_not_suspended` on a resumed sandbox lease.
3. The new tests in `native-session-executor.test.ts` and
`native-runner-restart-recovery.integration.test.ts` do this.

**Paperclip version or commit**

`master` at 426e8a35dc.

## What Changed

- `native-session-executor.ts` (`verifyRemoteRunnerOrSettledExit`): with
`replaceUnansweredRunner`, the attempt adopts the live Runner, launching
nothing and signalling nothing, when:
- the Runner's saved event sequence is past the controller's last
acknowledgement, or
- its saved state (event and command sequences, generation, lifecycle)
changes within a bounded wait of about 10 s.
- The baseline is the first successful state read; every successful read
is also compared against the controller's acknowledgement. A Runner
whose state cannot be read at all during the wait is adopted, not
replaced: missing evidence is not proof of a wedge. A Runner whose
identity changes during the wait fails closed.
- A settled exit during the wait keeps the attempt allowance that
admitted the replacement, so it also restores on the extra fourth
attempt.
- Only a Runner that made no progress is replaced. The relaunch's
existing launch fence stops it and its process lineage with proof, and
the new Runner restores its own journal. A resumed sandbox lease no
longer refuses that relaunch with `resumed_not_suspended`.
- A Runner that exits during the wait follows the existing exited-Runner
rules: a settled suspension restores, and a mid-turn exit stays
fail-closed.
- The wait runs only on this recovery path, never on a normal run start.
- `doc/execution-semantics.md`: describes adopt versus replace.
- The claim flag `replaceUnansweredRunner` is set in claim logic that
another session owns. This PR does not change it; it changes only what
the executor does with it.

## Verification

Red on master, green on this branch:

- `native-session-executor.test.ts`
- `adopts a live Runner that emitted events past the controller's last
acknowledgement`
- `adopts a live Runner whose saved state advances within the bounded
wait`
- `hands a live Runner that made no progress to the relaunch, whose
launch fence stops it, without signalling it here`
- `restores from the journal of a Runner that suspended and exited
during the wait`
- `stays fail-closed for a Runner that exited mid-turn during the wait`
- `native-runner-restart-recovery.integration.test.ts`: `reattaches a
live %s Runner whose recovery read went unanswered on a resumed sandbox
lease` (progressing, wedged)
- Review fixes, `native-runner-restart-recovery.integration.test.ts`
(the first four were red before the fix):
- `adopts a live Runner whose first state read failed and whose later
reads advance`
- `adopts a live Runner whose first readable state is already past the
controller's acknowledgement`
- `does not replace a live Runner whose state could not be read at all
during the wait`
- `restores from the journal of a Runner that suspended and exited
during the wait on the admitted fourth attempt`
- `fails closed when the Runner's identity changes during the wait`
(pins existing behavior)
- `native-remote-retry-admission.test.ts`: unchanged, still passes.

Local runs on this branch: `tsc --noEmit` is clean for `server` on this
branch (master c6b497d831, which contains #21442).
`native-runner-restart-recovery.integration.test.ts`, the
adopt-or-replace block: 12 of 12 pass. The full
`native-session-executor.test.ts` and
`native-remote-retry-admission.test.ts` runs did not finish on this
shared machine (load 20 to 96); CI ran them on this head and they
passed.

Conflict resolved: master added a relaunch branch for a Runner that
exited for a fresh bootstrap (#21374) at the same place. Both branches
are kept; the cheaper check for an unanswered Runner that made no
progress comes first.

Daytona results that exist. They ran on a combined branch with this PR,
#21450, #21442 (merged) and #21362 (merged), so they cannot be credited
to this PR alone:

- G5 baseline (master d1d71a7755; 480 s delay injected on the first
identity write per run): one Claude run reproduced the staging chain:
authentication 419 ms, identity +423.5 s, attempt 1 `native session
provider recovery timed out after 900000ms`, then the retry failed in 6
s with `runner_harness_state_mismatch: resumed_not_suspended`.
- G5c (same cell, combined branch 588c6ba267): 6 of 6 tasks finished
hands-off. The chain did not recur. All runs that crossed swap 2 were
adopted on the same Runner process ID.
- Caveat: the adopt-or-replace retry path that this PR changes was never
exercised in a cell. In G5c no first attempt failed, because all
identity writes started together 4 to 5 s after the claim. G5c also ran
before the phase timing fix in #21450.

## Risks

- Not proven on Daytona: the adopt-or-replace path itself (see the
caveat above).
- Without #21442 (merged) an adopted Runner can still time out on its
first command while it replays a backlog. This PR stops the replacement;
#21442 (merged) makes the adopted Runner answer.
- A Runner whose state cannot be read during the whole wait is adopted.
If it is in fact wedged, the adopted attempt times out again and the
attempt limit bounds it.
- A wedged Runner that still emits a few events counts as progressing
and is adopted. The existing attempt limit bounds this.
- The progress check adds a wait of up to about 10 s on this recovery
path only. A normal run start has no added wait.
- The claim construction (`native-restart-recovery.ts`) is owned by
another session and is unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 18:56:11 -06:00
DottaandPaperclip 8f105e743a fix(accounting): recover closed legacy runs without inventing usage (#21493)
Preserve exact physical process closure independently from incomplete cost reporting. Recover a missing receipt through the existing authorized accounting Retry without replaying provider work.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-13 14:37:13 -06:00
d018a9d1cd fix(queue): retain chat bindings on resolved card continuations (#20825)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A task can receive a card response while its agent runs.
> - The queue saves this response for the next turn.
> - Chat authorization requires the saved inbound message binding.
> - Queue promotion removed this binding when a prior turn received the
message.
> - This pull request retains live bindings for resolved card responses.
> - The next turn can check current chat access and continue the
approved work.

## Linked Issues or Issue Description

**What happened?**

An accepted plan card queued behind a chat run could start with a chat
source and no message IDs. Reviewed chat authorization then failed with
context_mismatch before provider work started.

**Expected behavior**

A resolved card must retain its live chat binding through queue
promotion. The new run must check current access. A consumed
message-only wake must not start another turn.

**Steps to reproduce**

1. Start a chat run with an inbound message.
2. Queue a card response with that message binding.
3. Complete the first run and release its queue.
4. Inspect the promoted context and check chat authorization.

Related work: #20689 cancels consumed message-only wakes. #19156
protects a successor checkout. This change retains the binding for an
independent card response.

## What Changed

- Retain delivered message bindings for resolved chat card responses
during queue promotion.
- Keep deleted, foreign-task, and self-authored message filters and
current chat authorization.
- Add a persisted queue regression and Slack/Discord
promotion-to-authorization checks, including access revocation.
- Document the queue behavior.

## Verification

- Before the fix, the persisted queue test failed because the promoted
context had no wakeCommentIds. Three existing tests passed.
- The first focused run passed 108 tests. The added Slack and Discord
checks passed after promotion and refused revoked access.
- Final head a38bcb9bc5: 110/110 focused
tests pass. Low-memory server typecheck passes. pnpm verify exits 2 and
requires broad hosted CI. CI typecheck, build, all
E2E/general/serialized tests and Runner checks pass except the Runner
support suite and its aggregate gate. The support suite fails on
inherited stale SDK lock pins (source-sdk-public-type,
source-sdk-evidence and source-sdk-public-inputs); existing PR #20623
owns that repair and has passing CI. No retry is appropriate for this
deterministic failure. Greptile reviewed this exact head at 5/5 with no
actionable findings; there are no inline or unresolved review threads.
- The hosted failed task independently resumed. No source task replay or
deployment was performed. Fixed-release hosted provider acceptance
remains unverified.
- Preflight installed dependencies but could not build Rust binaries
because Cargo is absent. The first server typecheck ended without a
diagnostic; the low-memory retry passed.

## Risks

An overly broad exception could deliver consumed input twice. The
exception applies only to resolved card responses with a chat source.
Message-only wakes still cancel. Message existence, task scope, author
filters, execution ownership, and current chat access still apply.

## Model Used

OpenAI GPT-6 Codex agent with reasoning, repository tools, and code
execution. The runtime did not expose a more specific model build ID.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket id
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:12:02 -06:00
DottaandPaperclip 6ad4529c5a fix(native): preserve pack proof across interrupted reattachment (#21246)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server can connect again to a live Runner after a restart.
> - Each new controller claim changes provider-pack admission to
pending.
> - The claim kept the previous verified proof only in local memory.
> - If recovery failed before its callback, the next claim lost that
proof.
> - This pull request saves a small reference to the verified proof with
the claim.
> - The next claim can check the same receipt without weakening
admission checks.

## Linked Issues or Issue Description

**What happened?**
A restart recovery attempt timed out before its admission callback. The
next attempt refused the saved pack with `source_state_pending`. It then
failed with `native_provider_artifacts_admission_unverified`.

**Expected behavior**
The next attempt can reuse the previous verified proof for the same
receipt, session, Runner, and lease. Current admission must remain
pending until its normal callback succeeds.

**Steps to reproduce**
1. Save verified provider-pack admission for a run.
2. Claim that run again. Commit the pending admission state.
3. Interrupt recovery before the session admission callback.
4. Claim the same run again and reconnect to the same Runner.
5. Observe that the old code refuses the saved receipt.

Related work: #20682 and #20850 address lock waits. #21085 adds refusal
logs. #21161 addresses command waits after restart. This change
addresses lost admission proof across interrupted claims. The original
transport timeout remains a separate investigation.

## What Changed

- The admission module prepares the claim patch and its recovery source.
- The claim saves a versioned reference with the source run, receipt
digest, and previous verified state. It adds no copy of provider files
or credentials.
- A later pending claim can use that reference only if the receipt is
unchanged. Normal company, agent, session, Runner, and lease checks
still apply.
- Missing proof, unknown state, a changed receipt, or a different Runner
stays refused. The callback clears the reference when it records its
decision.
- The executor uses this helper under its existing claim lock. Existing
artifact-retention and ownership checks stay in place.
- The execution guide states the previous-release behavior. Old verified
records seed the reference. Old pending records without a reference
remain refused. Older readers ignore the field and keep their existing
refusal behavior. No migration is needed.

## Verification

- The new interrupted-claim regression failed before the fix: admission
returned false instead of true.
- Focused admission, PostgreSQL fencing, and executor lease tests passed
after the rebase: 61 tests on commit c8123cc35d.
- The database case saves two interrupted claims, reads their persisted
state, checks a different Runner is refused, and publishes verified
admission for the original Runner.
- Negative cases cover missing proof, unknown state, a changed receipt,
a different source run, a different Runner, and a different lease.
Existing company and claim fences also pass.
- `pnpm verify` returned exit 2 because the execution guide requires
broader CI. Required CI must cover the final commit.
- Local server typecheck was killed with SIGKILL, including its
single-thread retry. This is an environment gap. Required CI must
provide typecheck and build results.
- Repository preflight installed dependencies and built shared packages.
It could not finish Runner preparation because cargo is absent.
- A live OpenCode restart workflow and recovery of the original task are
not verified. No task replay or deployment was performed.

## Risks

- The added profile field carries historical evidence. It does not mark
the current claim verified. Its use is limited to an unchanged receipt
with the normal identity checks.
- Existing pending or unknown incidents without this evidence cannot
gain proof from this fix. They keep their normal recovery path.
- No provider attempt limit, approval gate, workspace hold, or company
boundary changes.

## Model Used

OpenAI GPT-6, through Codex, with code search, tool use, and local test
execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 12:56:57 -06:00
DottaandPaperclip 08be8be414 fix(recovery): report exact retained sandbox resumes as resumed (#21259)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A controller restart must keep the same remote execution owner.
> - A retained sandbox lease stores the result of its first acquisition.
> - That result can say replacement after the same allocation resumes.
> - The Runner then rejects a valid resume as a changed sandbox.
> - This pull request reports the verified operation as resumed.
> - The saved acquisition history and identity checks stay intact.

## Linked Issues or Issue Description

Refs #18934. That change makes the retained sandbox ready before
preparation. This change corrects the acquisition receipt after that
resume.

**What happened?**
A remote native run resumed the exact retained sandbox after a
controller restart. The lease still carried a historical replacement
receipt. Runner verification rejected it with
`native_remote_recovery_lease_mismatch`, stage `runner_verification`,
field `replacement_sandbox`.

**What did you expect to happen?**
Report the verified exact-allocation operation as resumed. Preserve the
original saved history. Reject a changed, absent or expired allocation.

**Steps to reproduce**
Save an active lease with historical replacement metadata. Resume that
exact provider allocation through `acquireForRun`. Resolve its execution
target. Before the fix, the target still reports replacement and Runner
verification refuses it.

**Deployment mode**
Native Runner on a remote sandbox. The reported provider was Daytona.

## What Changed

- Return a separate operation receipt after the exact retained
allocation passes the existing resume checks.
- Pass the receipt through heartbeat preparation and apply it only to
the execution target.
- Keep the saved lease metadata and workspace unchanged.
- Test historical created, replacement and resumed receipts. Test
ordinary and remote-authority realization through the real execution
target resolver. Check every saved metadata write.
- Document the difference between saved acquisition history and the
current resume operation.

## Verification

- Head: `74d03e8ddadd1274891bea99a10455795c500557`.
- Red: three historical receipt cases fail without the resume receipt.
- Review regression red: realization writes resumed over the historical
replacement receipt. The saved metadata loses the previous allocation
and cause.
- Green: all 69 orchestrator tests pass. Ordinary and remote-authority
realization preserve history in every metadata write and return resumed
in the target.
- Green: 135 focused orchestrator and remote controller adoption cases
pass. Identity and changed-allocation refusals remain effective.
- Server typecheck passes, including its TypeScript build dependencies
and test layers.
- `git diff --check` passes.
- `pnpm verify` exits 2 because the execution-semantics document needs
broad CI. Required CI supplies that coverage.
- Linux x64, Node 25.9.0. Provider boundaries are simulated.
- The first post-merge attempt could not collect tests because the
compiled plugin SDK lacked the new taskLease schema. Shared and plugin
SDK builds passed, then the unchanged selection passed.
- Full preflight stopped because Cargo is absent. Dependency
installation completed. The later server TypeScript build and typecheck
passed.
- Live retained Runner reattachment is not verified. This workspace has
no provider credentials. No rollout or original-task recovery by this
patch is claimed.
- Required current-head CI and automated review remain pending.

## Risks

A wrong receipt could hide an actual replacement. The projection occurs
only after the existing company, task, run, agent, workspace, provider
allocation, expiry and directory checks pass. The provider must resume
the exact allocation. Runner process and identity verification still
runs. No replacement is acquired and no saved acquisition history is
changed. The heartbeat change only carries the typed receipt between the
owning orchestrator stages. Callers without a receipt keep their
existing behavior.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository inspection, code
execution and tool use. The exact backend snapshot and context window
are not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 12:55:55 -06:00
DottaandClaude Opus 5.5 5026093d0f feat(presentation): offer Move it forward now while a continuation is still waiting to start (#21385)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is interrupted, Paperclip records a fresh continuation
and delivers it later as a new run. Since #21353 the task reads as
continuing (`working`, no Retry) while it waits
> - #21373 added "Move it forward now" (`retryAction: "move_forward"`).
It is offered only after the delivered continuation run waited 15
minutes in the agent's queue
> - Before delivery (`continuationDelivery: "pending"`) the task showed
no button. On hosted staging such continuations waited 30 to 110 minutes
behind a deep queue, and the person had no control
> - #21378 and #21379 made the click safe in that state: a
`move_forward` request with a recorded, undelivered continuation is
accepted and delivers that continuation as the one run, with its
recorded isolation
> - This pull request also offers "Move it forward now" while the
continuation waits for delivery, once it has waited 15 minutes since it
was recorded
> - The benefit is one action, with the same label and copy, for both
waiting states. The click never starts a second run

## Linked Issues or Issue Description

Refs #21353, #21373, #21378, #21379

**What happened?**

A task has a fresh continuation that is recorded but not delivered
(`continuationDelivery: "pending"`). It waits behind a deep agent queue
for 15 minutes or more. The task shows "continuing" with no action.

**Expected behavior**

After 15 minutes the task still reads as continuing (`working`,
responsibility `system`, never `failed`). It also offers "Move it
forward now". The click delivers the recorded continuation as the one
run.

**Steps to reproduce**

1. Record a fresh continuation for a task while its agent is busy. Do
not run the delivery pass.
2. Wait 15 minutes.
3. Read the task presentation. It shows `canRetry: false` and no
`retryAction`.

## What Changed

- `server/src/services/task-interruption-presentation.ts`: the
fresh-continuation read also loads the record status, schema,
`recordedAt` and reconciled run. A `continuing` state can now carry
`pendingSince` when the primary record is `resolved`,
`continuationDelivery` is `pending`, the schema is the
fresh-continuation schema, and the reconciled run is the observed run.
These are the same facts that `undeliveredContinuation` checks at click
time.
- The projection offers `canRetry: true, retryAction: "move_forward"`
(state `working`) when `pendingSince` is 15 minutes or older, no person
Retry for that run already waits, and the run's agent is the current
assignee. Under 15 minutes, nothing changes.
- No button when the system gave up delivery (`invalidated`; the normal
Retry rules apply), when the delivery is `not_requested`, when a person
already asked, or for a held restore (the existing execution-blocker
post-filter removes the offer).
- New integration test
`server/src/services/fresh-continuation-move-forward-offer.test.ts`. It
uses the real click path for the three recorded kinds
(`fresh_isolated_workspace`, `writer_ended`, native `remote_epoch`):
presentation, `requestTaskRetry` with the shown `observedRunId` and
`retryAction`, `resumeTaskRetry`, then the delivery pass.
- Updated guards in `task-interruption-automatic-path.test.ts` and
`recovery-wake-herd.test.ts` (#21353, #21373). Their assertions that a
pending continuation shows no Retry at 30, 45, 110 and 240 minutes now
expect the move-forward offer. The `working` / `system` / never-failed
assertions stay. New cases cover 14 minutes, `not_requested`, a waiting
person Retry and reassignment.
- `doc/execution-semantics.md`: describes the new offer and its races.
- No shared type, validator, UI or API client change. The action and its
UI copy ("Waiting for the agent to finish other work. Paperclip
continues this task by itself." / "Move it forward now") already exist
and do not depend on the state. No runner-e2e SDK public-input pin
covers the changed files.

## Verification

- Red: with the presentation change reverted, 16 tests fail (`expected {
intentId: null, …(7) } to match object { responsibility: 'system', …
retryAction: 'move_forward' }`).
- Green: `npx vitest run
src/services/fresh-continuation-move-forward-offer.test.ts`: 14 of 14
pass. For each kind:
- The click gives exactly one run. It is the recorded continuation
(`recoveryActionId`, `previousRunId`, `retryOfRunId`,
`forceFreshSession`). The fresh isolated workspace is kept
(`freshContinuationIsolation`, `readFreshWorkspaceReadiness`). The
record reads `delivered` with that run. Two later delivery passes and a
repeated resume create no second run, and the offer then ends.
  - The click and two concurrent delivery passes give one run.
- Delivered between the offer and the click: the click is refused
(`accepted: false`, no wake) and the only run is the delivered
continuation.
- The system gave up the delivery between the offer and the click:
`alreadyMovedOn: true`, no run.
- The task moved to another run between the offer and the click:
refused, the only run is that other run.
  - Under 15 minutes: no button.
- Guards: `task-interruption-automatic-path.test.ts` (37/37),
`recovery-wake-herd.test.ts`, `task-interruption-presentation.test.ts`,
`task-interruption-query.test.ts`,
`fresh-continuation-retry-delivery.test.ts`,
`fresh-continuation-fallback.test.ts`, `task-retry.test.ts`: all pass.
Also the saved-work, workspace-recovery, native restart and
closed-session retry suites that read the presentation.
- `npx tsc --noEmit -p .` in `server/`: no errors.

## Risks

Low risk. The change is presentation only. It adds four selected JSON
fields to a read that already exists and one conditional module import.
The click path (`task-retry.ts`, `fresh-continuation-retry-fold.ts`) is
unchanged and already holds the task row lock. A record without
`recordedAt` or with an older schema gets no button, as before. People
now see a button on tasks that showed none after 15 minutes. The button
can only deliver the recorded continuation or do nothing.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:24:12 -06:00
DottaandClaude Opus 5.5 5ba26600ef fix(native-runtime): a remote Runner reattachment continues a turn the Runner can recover (#21362)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime runs agent turns through a Runner, often in a
remote sandbox, and must recover runs after a restart without help from
a person
> - When a Runner process is lost, a new Runner restores the
conversation. If the provider turn is gone, the Runner sends
`turn.failed` with `provider_turn_lost_on_restore` and `recoverable:
true`. The Runner sends this only after it proves that the old process
ended
> - The runtime can continue such a turn in the same conversation, but
only when the server sets `resumeInterruptedTurn`. The server set it
only for a local `resume_dead_runner` claim or a scheduled resume, and a
scheduled remote resume is refused. So on a remote Runner reattachment
the runtime did not continue the turn, and the run ended
> - This pull request sets `resumeInterruptedTurn` for a
`reattach_remote_runner` claim too
> - The benefit is that a remote run continues the lost turn in the same
conversation immediately, instead of failing and then restarting 3 to 6
minutes later

## Linked Issues or Issue Description

**What happened?**
On hosted staging (build `d1d71a7755`, read-only trace), 10 runs in 3
hours ended with `native_provider_terminal_failed` and the message "The
runner restored the conversation after process loss, but the previous
turn is no longer active." The Runner marked each of these events
`recoverable: true`. Example: run 646e2efb. A new Runner process started
at 08:32:04. `session.resumed` came at 08:34:49. `turn.failed
provider_turn_lost_on_restore` (recoverable) came at 08:34:51. The run
failed at 08:35:21. Example: run 11fe2cdc. `session.resumed` came at
09:23:24. `turn.failed` came at 09:23:25. The run failed at 09:23:52. In
each case, a separate recovery path continued the task only 3 to 6
minutes later.

**Expected behavior**
When the Runner marks a lost turn as recoverable after a restore on a
remote reattachment, the run continues the turn in the same
conversation.

**Steps to reproduce**
1. Start a native run on a remote sandbox Runner.
2. Lose the Runner process while a turn is active, so that a new Runner
reattaches through a `reattach_remote_runner` claim.
3. The new Runner restores the conversation and reports
`provider_turn_lost_on_restore` with `recoverable: true`.
4. See that the run ends with `native_provider_terminal_failed` and does
not continue the turn.

**Agent adapter(s) involved**
Not adapter-specific (core bug). Seen with Codex on a Daytona remote
sandbox.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`:
`resumeInterruptedTurn` is now also true for a `reattach_remote_runner`
restart-recovery claim. The runtime still continues the turn only when
the Runner itself sends `turn.failed provider_turn_lost_on_restore` with
`recoverable: true` for the checkpointed active turn.
- `doc/execution-semantics.md`: one paragraph describes this behaviour,
and says that a lost turn that still ends the run keeps
`native_provider_terminal_failed`.
- Tests:
- `native-scheduled-resume-continuation.integration.test.ts`: "continues
a lost turn on a same-run remote Runner reattachment" (2 cases: a plain
reattachment and a `replaceUnansweredRunner` reattachment).
- New `native-recoverable-lost-turn.integration.test.ts`: a guard
through `executePaperclipNativeSession`. The runtime saves a
failed-terminal checkpoint and then raises the lost turn (recoverable
and not recoverable). The run ends as `native_provider_terminal_failed`,
and the hold cause stays one that the replacement lane continues.
- An earlier revision of this PR also mapped a recoverable lost turn to
`native_session_interrupted`. Review showed that this does not make the
run retryable, and the hold it writes is not picked up by any automatic
continuation. That hunk is removed.

## Verification

- Red without the product change: the two integration cases fail with
`passed` = `[false]` (expected `[true]`).
- Green with the change:
- `cd server && npx vitest run
src/services/native-runtime/native-scheduled-resume-continuation.integration.test.ts
src/services/native-runtime/native-recoverable-lost-turn.integration.test.ts`
— 14 passed.
- `cd server && npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`
— passed (with the scheduled-resume file: 224 passed, 14 skipped,
pre-existing skips).
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts` — 1469
passed, 7 skipped.
  - `cd server && npx tsc --noEmit` — exit 0.
- Staging evidence (hosted staging, build `d1d71a7755`, read-only
trace): 10 runs in 3 hours ended with `native_provider_terminal_failed`
although the Runner marked the lost turn `recoverable: true`. Run
646e2efb: new Runner pid 08:32:04, `session.resumed` 08:34:49,
`turn.failed provider_turn_lost_on_restore` (recoverable) 08:34:51, run
failed 08:35:21. Run 11fe2cdc: `session.resumed` 09:23:24, `turn.failed`
09:23:25, run failed 09:23:52. The task continued only 3 to 6 minutes
later through a separate recovery path.

## Risks

- This change has not run live on staging or on a Daytona sandbox yet.
The evidence is the staging trace and the local tests.
- This PR does not introduce or widen the gap where a recoverable lost
turn still ends the run. On master, that run already ends as
`native_provider_terminal_failed`, and the replacement lane continues it
after its decision (3 to 6 minutes on staging). This PR removes the
run-ending case whenever the Runner reports the turn recoverable for the
checkpointed active turn on a remote reattachment. Where the run still
ends (for example, when the continuation itself is lost), the behaviour
is unchanged, and the guard test
`native-recoverable-lost-turn.integration.test.ts` proves that it stays
`native_provider_terminal_failed`.
- The remaining case is tracked with the owner of the recovery path:
recovery evidence will accept `provider_turn_lost_on_restore` /
`recoverable` directly, after their ownership-deadline PR #21364. An
earlier hunk here tried to shorten that case. An integration test
through the real path showed that it would create a board-owned
ambiguous hold that no automatic pass continues, so the hunk is removed.
- This change does not fix why the turn is lost. In staging, a retry
replaced a live Runner. A separate fix for that is in progress.
- The runtime continues a turn only when the Runner sends the
recoverable lost-turn signal. The Runner sends it only after it proves
that the old process ended. So no provider turn runs twice.
- No Runner image change. No schema change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, in Claude
Code with tool use (shell, file edit, code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 07:09:20 -06:00
DottaandClaude Opus 5.5 d9b42b50fd feat(presentation): let a person move a waiting continuation forward without starting a second run (#21373)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is interrupted, Paperclip records a fresh continuation
and continues the task by itself. Since #21353 the task reads as
continuing (`working`, no Retry) for as long as that continuation waits
> - That is truthful, but a person has no control while it waits. On
hosted staging one agent had 21 running and 209 queued runs, and
continuations waited 30 to 110 minutes. Pacing (#21234, #21354) admits a
waiting continuation into the agent's queue within minutes, so most of
that wait is in the queue, behind other work
> - #21353 removed Retry from this state because a Retry started its own
run, which raced the continuation and could give the task two runs
> - A Retry that observes the queued continuation run cannot start a
run: admission sees a queued run and joins the Retry to it (`coalesced`,
`runId` = that run). Under #21234's ranking, a queued run that a person
joined starts ahead of system recovery and ordinary work
> - This pull request keeps the continuing state and, once the queued
continuation has waited 15 minutes, also offers Retry labelled "Move it
forward now"
> - The benefit is that a person can move a long-waiting task to the
front of its agent's queue, and the click can never create a second run

## Linked Issues or Issue Description

Refs #21353, #21348, #21234, #21354, #21264, #21320, #21200

**What happened?**

On hosted staging, tasks with a recorded continuation waited 30 to 110
minutes behind a deep agent queue. The task showed "continuing" with no
action. The person could only wait.

**Expected behavior**

The task still never shows "failed". After a long wait the person can
move the waiting continuation forward. The click must not start a second
run.

## Which states a Retry joins (established before building)

I read #21348's fold (`foldWaitingRetriesIntoContinuation`) and its
tests, and drove the real click path (`requestTaskRetry` followed by
`resumeTaskRetry`, as the route does) in a probe against each record
state:

| Continuation state | What a Retry does today | Offer Retry? |
|---|---|---|
| Recorded, waiting for delivery (`continuationDelivery: "pending"`, no
run yet) | The Retry is admitted at once as its own `task_retry` run,
for every isolation (`fresh_isolated_workspace`, `writer_ended`,
native). The delivery pass then marks the record delivered (superseded).
#21348's fold never runs: it folds only a Retry that is still
`deferred_issue_execution` when the continuation run is created, and
only for `fresh_isolated_workspace`. Nothing keeps the Retry deferred,
so the click races the delivery pass. | **No** (unchanged) |
| Delivered, continuation run `queued` | `requestTaskRetry` sees the
latest run is busy and records the Retry as `coalesced` into that run in
the same transaction. No run is created; `resumeTaskRetry` acts only on
deferred Retries. The run is now ranked as a person's (`joinedByUser` in
`queue.ts`). | **Yes, after 15 minutes** |
| Continuation run `running` | Same join, but the run already started,
so there is nothing to move forward. | No |
| Continuation ended | A Retry gets its own run (the existing failed-run
path). | Existing behaviour |

So the Retry offered here is joined by the existing busy-run admission
in `requestTaskRetry`, not by #21348's fold. Its source run is the
queued continuation itself (the run the presentation names as
`observedRunId`), not the run the continuation was created for.

## What Changed

- `server/src/services/task-interruption-presentation.ts`: for a task
whose latest run is `queued`, created 15 minutes ago or more, and named
as `continuationRunId` by a delivered fresh-continuation record, the
projection returns `state: "working"`, `canRetry: true`, `retryAction:
"move_forward"`, responsibility `system`, `observedRunId` = that queued
run. It does not offer this when a person's wake already joined the run,
when the run's agent is not the assignee, or when a person-owned wait
applies (approval, reconnect, budget, pause and the others are checked
first). Two small reads run only for queued runs older than 15 minutes.
The existing restore-hold check that withdraws `canRetry` also withdraws
`retryAction`.
- `packages/shared/src/types/execution-projection.ts`: new optional
`retryAction?: "move_forward"` on `TaskInterruptionPresentation`. It
only changes the label; `canRetry` still decides whether Retry is
offered.
- `ui/src/components/ExecutionBlockerNotice.tsx`: a `working`
presentation with `retryAction: "move_forward"` shows "Waiting for the
agent to finish other work. Paperclip continues this task by itself."
and a "Move it forward now" button that calls the same task Retry.
Token-only classes; `pnpm check:token-gates` is clean.
- One join-only action, `move_forward`, carried from the offer to
admission (contracts kept in sync):
- `packages/shared/src/validators/issue.ts`: `taskRetrySchema` accepts
`retryAction: "move_forward"`. The route already passes the body to
`requestTaskRetry`.
- `server/src/services/task-retry.ts` (`admitTaskRetry`): right after
the stale check, still under the task row lock and before any other
Retry path, a `move_forward` Retry whose observed run is no longer
`queued` or `running` returns `{ accepted: false, alreadyMovedOn: true,
requestId: null }` and writes nothing. This block was cleared with the
owner of Retry admission. The owner will later extend it so that
"continuation recorded, not yet delivered" also counts as actionable,
after their pending-delivery change lands (about two lines on their
side). Until then this presentation never offers the button in that
state, so the early return cannot misfire from our UI.
- `ui/src/api/issues.ts` and `ui/src/hooks/useTaskRetry.ts`: the click
sends `retryAction: "move_forward"` when the offer has it.
`alreadyMovedOn` shows "The task already moved on, so nothing new was
started. It now shows its current state."
- What a click does at each timing: continuation `queued` → joined to
that run (moves it forward); `running` → joined (nothing more to do);
ended → "already moved on", nothing started.
- A joined click cannot become a run later. The busy path writes it as
`coalesced` with `runId` set to the continuation and `finishedAt` set
(`task-retry.ts:456-461`). Promotion after a run ends
(`heartbeat/run-control.ts:944` `releaseIssueExecutionAndPromote`;
`modules/wake-queue/adapters/postgres.ts:89` `DEFERRED_WAKE_STATUS` and
the selects near `:1358`), `resumeTaskRetry` (`run-control.ts:833-839`)
and `resumeExecutionWaitComments` (`:876`) pick up only
`deferred_issue_execution` wakes.
- `ui/src/pages/Inbox.tsx`: the failed-run row does not show its Retry
for this state. That row sends the failed run as the observed run, which
is no longer the latest run, so its Retry would be refused as stale.
- `doc/execution-semantics.md`: one paragraph on the offer, when it
ends, and why there is none before delivery.

## Verification

- Red (before the product change, new tests only), in `server/`: `pnpm
exec vitest run src/__tests__/task-interruption-automatic-path.test.ts
src/__tests__/recovery-wake-herd.test.ts`: 2 failed, 58 passed. Both
failures are the missing offer (`canRetry: false`, no `retryAction`) for
a continuation queued 15+ minutes.
- Green after merging master 8e41247b65, in `server/`: `pnpm exec vitest
run src/__tests__/recovery-wake-herd.test.ts
src/__tests__/task-interruption-automatic-path.test.ts
src/services/task-retry.test.ts
src/services/task-interruption-presentation.test.ts
src/__tests__/task-interruption-query.test.ts`: 5 files, 279 tests pass.
UI: `useTaskRetry.test.tsx` and `ExecutionBlockerNotice.test.tsx`: 35
tests pass.
- New tests:
- (b, integration, `recovery-wake-herd.test.ts`) A continuation
delivered by the real delivery pass waits 16 minutes behind older
ordinary work. The task offers "move forward". The click goes through
the real `requestTaskRetry` and `resumeTaskRetry`. Result: the task has
exactly one run after the source run, the continuation; the Retry wake
is `coalesced` with `runId` = the continuation; the offer disappears;
the next delivery pass adds nothing; the real dispatcher
(`resumeQueuedRuns`) starts the continuation ahead of the older work;
after it ends, still only that one run.
- (Greptile P1, red then green) "starts nothing when the continuation
ended between showing Move it forward now and the click": the
continuation ends after the offer is read. Before the admission block
the click was accepted (`accepted: true`, a new deferred Retry that
becomes a turn). Now it returns `alreadyMovedOn`, the task has one run,
and there is no `task_retry` wake.
- "never turns a joined move-forward click into a new run after the
continuation it joined ends": the click joins the queued continuation.
The run is then cancelled through the real `cancelRun` release. Two
rounds of `resumeExecutionWaitComments`, `resumeTaskRetry`, the delivery
pass and `resumeQueuedRuns` follow. The joined wake stays `coalesced` on
the ended run. No run comes from the click, and the task's later runs
equal those of an identical task with no click (the system's own
`issue_assignment_recovery`).
- UI hook (`useTaskRetry.test.tsx`): the click sends `retryAction:
"move_forward"`, and the `alreadyMovedOn` outcome shows the plain
message.
- (a) Under 15 minutes: no Retry. Control in the herd file: without a
click the older work starts first.
- (b, presentation) Queued 15, 45 and 110 minutes: `working`, `canRetry:
true`, `retryAction: "move_forward"`.
- (c) No Retry while the continuation waits for delivery (16, 45, 110
minutes), when it is running, when a person already joined it, for a
queued run that is not a recorded continuation, and a paused agent still
asks the person to resume.
- UI (`ExecutionBlockerNotice.test.tsx`): plain wording with one "Move
it forward now" button that calls Retry; no button for a read-only user;
nothing shown for a plain `working` state.
- (d) The #21200, #21264, #21320 and #21353 guard tests pass unchanged,
including "follows the new run once a recorded continuation is
delivered" (a just-queued continuation shows no Retry).
- `node scripts/typecheck-layers.mjs` in `server/`, `tsc --noEmit` in
`packages/shared`, `pnpm run typecheck` in `ui/`: clean.
- Not related to this change:
`src/services/fresh-continuation-fallback.test.ts` has one failure
("starts the continuation while the unreachable old lease is active", a
deadlock in the `afterEach` truncate) that also fails on master
18fb6ee571 without this change.

## Risks

- Moving a waiting continuation forward lets it start before other
queued work of the same agent. That is the purpose, and it is the
ranking #21234 already gives any run a person joined.
- No Retry is offered while a continuation is still waiting for
delivery. The delivery pacing admits a waiting continuation into the
queue within minutes, and a fresh interruption at once (#21354), so the
offer appears after the continuation reaches the queue. For the recovery
owner (not changed here): offering Retry before delivery would need the
Retry admission to hold or deliver the pending continuation itself under
the task lock, instead of starting its own run beside it.
- Admission changes only for a request that carries `retryAction:
"move_forward"`. Every other Retry is unchanged. No schema, migration or
dispatch change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 07:03:23 -06:00
DottaandClaude Opus 5.5 eb54831d28 fix(native): relaunch a remote Runner that exited for a fresh bootstrap (#21374)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs often use a remote Runner in a sandbox. After a restart,
a deploy or a same-run retry, the server reattaches to that Runner or
starts a new Runner over its saved state
> - A Runner that no controller reaches before its connection lease
expires stops its provider, saves its state as `recoverable_failure`
with `lease_expired_requires_bootstrap`, and exits. The Runner protocol
says that only a new process with a fresh bootstrap ticket can
reconnect, and that the new process continues from the same cursor
> - Reattachment started a new Runner over saved state only for a
settled `suspended` exit (or an exit before `session.open`). So it
refused this exit, and the run ended with
`runner_remote_process_identity_unavailable`
(`native_continuation_requires_reconciliation`)
> - This pull request lets reattachment launch a Runner over the exact
saved state of a Runner that exited to ask for a fresh bootstrap
> - The benefit is that the run continues by itself after a long
controller gap, with no Retry and no manual sandbox repair

## Linked Issues or Issue Description

Refs #21329, #21362, #20853

**What happened?**
On hosted staging (build `2b69cdfcd6`, read-only trace), Codex runs on
remote Daytona Runners ended with
`native_continuation_requires_reconciliation` and the error
`runner_remote_process_identity_unavailable`, about 5 per hour, after
#21329 was deployed. All five sample runs (c6fabe9a, 31d54b02, 995b63df,
dd595934, 50c6e6e6) ended the same way. The last
`native.remote_runner_recovery_failed` event recorded
`process_verification` / `process_not_running`, `runnerLifecycle:
recoverable_failure`, `runnerRecoverableFailure:
lease_expired_requires_bootstrap`, and the Runner diagnostic "connection
lease expired; a fresh bootstrap is required". Since 10:30 UTC, 8 of the
15 runs that ended with this error had this exact exit.

Timeline of run c6fabe9a:
- 10:46 to 10:54: `runner.session.startup` took 476 s in a loaded
sandbox.
- 10:54:02: the controller sent `turn.start`. At 10:54:47 the command
timed out: "PRP command turn.start timed out (runner connected; pending
ahead 0; last runner event seq 2 91s ago)". The run resumed in the same
run and the turn then ran (tools completed from 11:14).
- 11:39: a graceful server shutdown suspended the controller. At 11:43
the next controller started the reattachment, but it did not reconnect
to the Runner until the server restarted again at 11:58. In that gap the
Runner's lease expired and the Runner exited.
- 11:58:53: the reattachment proved the process gone and refused the
`recoverable_failure` state. The run failed at 11:59.

Why `turn.start` timed out while the Runner was connected: the Runner
answers `turn.start` only after Codex answers its `turn/start` JSON-RPC
request. While it waits, the Runner buffers Codex notifications and
commits no events, so the controller sees a quiet connected Runner with
nothing pending ahead. The controller's bound for the `turn.start`
command result is a fixed 30 s. #21329 changed only the later wait for
`turn/started`. In a sandbox that needed 8 minutes to start, Codex
answered later than 30 s. That timeout is recoverable, and the run did
recover from it. This pull request does not change it (see Risks). The
failure that ended the runs is the refused reattachment.

**Expected behavior**
When a remote Runner exits because its lease expired (or its reconnect
grace or deadline ended) and its saved state is intact, reattachment
starts a new Runner from that state and the run continues.

**Steps to reproduce**
1. Start a native Codex run on a remote sandbox Runner.
2. Keep every controller from reaching the Runner for longer than its
connection lease (on staging: a deploy that stalled the reattachment for
about 15 minutes).
3. The Runner saves `recoverable_failure` /
`lease_expired_requires_bootstrap` and exits.
4. The next reattachment fails with
`runner_remote_process_identity_unavailable`, and the run ends.

**Agent adapter(s) involved**
`paperclip_runner` with Codex on a Daytona remote sandbox. The rule is
not provider-specific.

## What Changed

- `remote-runner-recovery-diagnostics.ts`: new
`exitedRunnerRequestedFreshBootstrap` (diagnostic) and
`runnerStateRequestsFreshBootstrap` (saved state). They accept lifecycle
`recoverable_failure` with `lease_expired_requires_bootstrap`,
`transport_reconnect_grace_exceeded` or
`transport_reconnect_deadline_exceeded`. These are the three codes the
Rust Runner records after `shutdown_preserving_cleanup`. The state check
also requires no pending terminal delivery and no warm transition.
- `native-session-executor.ts` (`verifyRemoteRunnerOrSettledExit`): when
the exact Runner process is proven gone and its diagnostic names one of
these exits, the server reads the saved state again. If the state has
the same six identity fields and still names that exit, the reattachment
launches a Runner from it instead of failing. The outbox does not need
to be empty and a turn may be active: the new Runner replays from the
same cursor. The attempt bound is the one the reattachment already
admits (3, or 4 for an unanswered-Runner replacement claim). Other exits
are unchanged.
- `native-session-executor.ts` (host preparation in
`createRunnerdBackend`): a resumed sandbox for a reattachment that did
not adopt a process re-reads the Runner state. When it still has the
same six identity fields and names the fresh-bootstrap exit, preparation
keeps it in place, as for an adopted live Runner. It no longer fails
with `runner_harness_state_mismatch: resumed_not_suspended` (found by
Greptile).
- `doc/execution-semantics.md`: one paragraph describes the rule and the
staging evidence.

## Verification

- Red on `origin/master`: the 5 new "launches a new Runner over the
saved state after ..." cases failed (the reattachment threw
`runner_remote_process_identity_unavailable`). The 7 new fail-closed
cases passed.
- Green: `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "remote
controller restart adoption"` passes 118 tests.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
1712 passed, 21 skipped, 1 failed. The failure ("verifies a live warm
owner before refreshing run authority", warm-session supervision, not
touched here) passes when run alone. It is a timing failure under
full-file load.
- Launch path: "keeps the saved mid-turn state of a remote Runner that
exited for lease_expired_requires_bootstrap when the reattachment
prepares its sandbox" runs `createRunnerdBackend` and
`prepareExternalRunnerState` with a mid-turn Codex checkpoint. Red
before the host-preparation change (`resumed_not_suspended`), green
after. An unknown recoverable failure stays fail-closed.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts`
on the final revision: 1494 passed, 7 skipped.
- `cd server && node scripts/typecheck-layers.mjs` (includes test
programs): passes.
- Fail-closed cases covered: an unknown recoverable failure, a warm
transition, a pending terminal delivery, a state that changed after the
exit, a different Runner instance, attempt 4 on an ordinary claim, and
attempt 5 on a replacement claim.
- Staging read (read-only): the 5 sample runs and 69 runs with this
error in the last 9 hours were classified from their
`native.remote_runner_recovery_failed` events.

## Risks

- Medium. Reattachment now starts a new Runner over a saved state that
can be mid-turn. The Runner itself stops its provider before it saves
this state. The new Runner restores the session. It continues the turn,
or it reports the lost turn as recoverable only after it proves that the
old provider process ended. A recoverable lost turn on a remote
reattachment ends the run as `native_provider_terminal_failed` until
#21362 merges; the replacement lane then continues the task. That is
still better than the current reconciliation hold.
- The attempt bound does not grow. A Runner that exits this way again on
a later reattachment is bounded by the same provider attempt budget.
- This does not change the 30 s `turn.start` command bound or the deploy
gaps that let the lease expire. Those are separate follow-ups. 57 older
runs with this error had a different exit (`ready`, not settled). This
pull request keeps that case fail-closed; #20853 covers the
settled-`ready` part.
- No schema, API, UI or Runner image change.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, in Claude
Code with tool use (shell, file edit, code execution, read-only browser
reads of staging run data).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:53:38 -06:00
DottaandClaude Opus 5.5 0986c3eb3a fix(native): let a same-run reattachment take over its own retained remote Runner cleanup (#21328)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) use a detached
`paperclip-runnerd`. After a recoverable failure, the server resumes the
same run and reattaches to that Runner
> - #20923 keeps an unreachable remote Runner alive when the failed
attempt closes, so that the same-run reattachment can adopt it
> - The same close also leaves the session in the in-process cleanup
quarantine. The quarantine refuses every new session of the task, and
that includes the reattachment that the Runner was kept for
> - Close retries cannot settle it. After the first close fails, the
Runner transport returns the same rejected close for every retry
> - On staging, this was the largest failure class: 14 Codex runs in one
hour failed with `native_session_cleanup_quarantined`, while only 5 runs
succeeded. Each task then needed Retry
> - This pull request lets the same run's reattachment to that Runner,
in the same remote sandbox, take over its own retained cleanup
> - The benefit is that the task continues on its own, with no spent
attempt, no failed run and no Retry

## Linked Issues or Issue Description

Refs #20923, #21031

**What happened?**
A remote native run (Codex on Daytona) failed a turn with a recoverable
error, for example `runnerd did not report the provider turn identity`
or `native_finalization_missing: session returned no semantic result`.
The close then logged `runner did not prove durable suspension before
checkpoint` and `remote runner left alive unsettled for same-run
reattachment`. Each later same-run reattachment was refused at
admission:
- When the close error was retryable, admission waited out the grace and
failed with `native_session_cleanup_quarantined: prior session cleanup
remains incomplete`. That spent a same-run attempt. After the attempt
budget, the run failed. All close retries replayed the same failed close
within milliseconds (`runner transport close requested: native session
quarantined cleanup recovery`), so they could never settle the
quarantine.
- When the close error was `native_session_close_unrecoverable`
(operator-required), the next attempt failed at once with
`native_session_cleanup_quarantined: the previous session is still
shutting down; Paperclip finishes the cleanup automatically and the
saved work is kept.` That code is a permanent failure.

"Paperclip finishes the cleanup automatically" did not happen in the
process. Only a server restart (a deploy) removed the in-memory
quarantine, and by then the run had failed and the task waited for
Retry. Staging restarts every 20 to 45 minutes, so attempts often end
mid-cleanup and meet this gate.

**Expected behavior**
The same run's reattachment adopts the Runner that its earlier attempt
kept for it. It does not wait for a close that cannot settle, and it
does not spend an attempt. The task continues without Retry. Other runs
of the task and other sandboxes stay gated.

**Steps to reproduce**
1. Start a remote native run, and let its turn fail with a recoverable
transport error while the Runner cannot prove suspension.
2. The close keeps the Runner alive and quarantines the session.
3. The server schedules the same-run reattachment 30 s later.
4. The reattachment fails with `native_session_cleanup_quarantined`, and
the task needs Retry.

**Paperclip version or commit**
Deployed Core `c730872974` on staging. Reproduced on `master` at
`49f0030f06`.

**Deployment mode**
Hosted cloud (staging), Daytona remote sandboxes.

**Agent adapter(s) involved**
`paperclip_runner` native runtime with the Codex app server backend.

## What Changed

- `packages/paperclip-runner/src/native-session-runtime.ts`:
`executeNativeSession` takes a new option `retainedRunnerReattachment`.
With a remote cleanup scope, admission retires quarantine entries that
belong to the same run in the same remote scope. Removal stops later
close retries. A close attempt that is still in flight stays an
admission owner until it settles, so the old and new controllers do not
overlap. Other runs, other sandboxes and local backends keep the gate.
- `server/src/services/native-runtime/native-session-executor.ts`: sets
`retainedRunnerReattachment` only for a `reattach_remote_runner`
recovery of the same run.
- `doc/execution-semantics.md`: records the takeover.
- Tests:
- `packages/paperclip-runner/src/native-session-runtime.test.ts`: four
new cases. An operator-required close with a retained Runner. A
retryable close whose retries replay one failure. A close that a deploy
interrupts while it is in flight (the reattachment waits for it, then
takes over). Controls: another run, a run without the takeover option
and a local backend stay refused.
- `server/src/services/native-runtime/native-session-executor.test.ts`:
asserts that the option is set for a same-run remote reattachment and is
not set for a fresh sandbox run.

## Verification

- Red on `master` (runtime source restored): 3 of the 4 new runtime
cases fail with exactly the two staging errors (`the previous session is
still shutting down…` and `prior session cleanup remains incomplete`).
The control case passes.
- Red on `master` (executor source restored): the 3 executor assertions
fail.
- Green with this change:
- `cd packages/paperclip-runner && npx vitest run
src/native-session-runtime.test.ts` (294 passed)
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "exact
Stop cleanup|fresh sandbox|retirement"` (46 passed)
- `cd server && npx vitest run
src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts
src/services/quarantined-task-retry.test.ts
src/__tests__/completed-turn-cleanup-hold-settle.test.ts` (61 passed)
- `npx tsc --noEmit` in `packages/paperclip-runner` and `server` passes.
- Staging evidence: runs `7b7ded99` and `27c487ef` show the sequence
above in their events and logs.

## Risks

- The takeover applies only when the server recovers the same run with
`reattach_remote_runner` in the same remote cleanup scope. This is the
case that #20923 keeps the Runner alive for. After a deploy the
quarantine is already gone and the same reattachment runs today, so the
takeover gives the in-process retry the same behavior as a restart.
- A close attempt that is in flight is never overlapped. The
reattachment waits for it within the existing admission grace.
- The reattachment still verifies Runner ownership and its controller
generation before it adopts the Runner. If the Runner is gone, the
existing relaunch or reconciliation path handles it.
- No persisted shape changes. Saved state from the previous release is
not affected because the quarantine lives only in process memory.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 03:15:40 -06:00
DottaandClaude Opus 5.5 7e7452e88f fix(native): keep a reattached Runner alive when its provider recovery deadline ends (#21321)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs on remote sandbox Runners. A same-run reattachment
adopts a live Runner after a failed attempt or a server deploy.
> - On staging, a remote Codex Runner is often gone when the next
attempt reattaches (`process_verification` / `process_not_running`). The
task then continues in a fresh session and loses its context.
> - The deploy does not stop the Runner. The logs show `controller
authority detached for restart; durable runner left alive`, and the new
controller adopts the same Runner after the deploy.
> - Our own cleanup stops it. The adopted Runner answers its recovery
reads slowly while its in-flight turn runs. The 900 s provider recovery
deadline then aborts the recovery, and the abort closes the transport
with `turn.stop` and `runner.suspend` and signals the process.
> - This pull request makes the recovery deadline detach from the
Runner, as a recovery read timeout already does. The server marks the
failure as an unanswered adopted Runner, so the next attempt replaces
the Runner from its saved journal and resumes the same provider thread.
> - The benefit is that a deadline no longer kills a live Runner and no
longer forces a fresh session. The task keeps its session context.

## Linked Issues or Issue Description

Refs #21314, #21272, #21161, #21023

**What happened?**
Codex runs on Daytona remote Runners failed often on staging. In three
sampled chains (attempt 2 of each), the same-run reattachment adopted
the live Runner, authenticated in under 1 s, and then waited the full
deadline: `native session provider recovery timed out after 900000ms`
(909 s, 930 s and 915 s). The deadline cleanup sent `turn.stop` and
`runner.suspend` (`provider turn stop timed out before runner
suspension`). The next attempt found the Runner gone 25–180 s later
(`process_not_running`, `runnerLifecycle: ready`, empty stderr) and
continued in a fresh session. During the 900 s wait, the in-flight turn
was still running Paperclip tools (`tool.execute` spans at 04:43 in a
recovery that started at 04:29).

**Expected behavior**
A remote Runner survives a recovery deadline and a control-plane deploy.
The next attempt continues the same provider session. A Stop still stops
the Runner.

**Steps to reproduce**
1. Start a Codex run on a remote Runner and let a turn start.
2. Fail the attempt so that its close keeps the Runner (for example, a
turn start that outlives its bound).
3. Let the same-run reattachment adopt the Runner, and make the Runner
stop answering recovery commands.
4. Let the provider recovery deadline expire.
5. See that the Runner process is gone and the next attempt reports
`process_not_running`.

**Paperclip version or commit**
master at b555087f0f (after #21314).

**Deployment mode**
Hosted staging (cloud), remote Daytona sandbox Runners, deploys every
20–45 minutes.

**Agent adapter(s) involved**
Native Codex (`paperclip_runner`, Codex app-server through
`paperclip-runnerd`).

## What Changed

- Add `NativeProviderRecoveryDeadlineError` (same message as before) and
use it for the provider recovery deadline in the native session runtime.
- In the Codex driver, a recovery aborted by this deadline detaches the
controller (no `turn.stop`, no suspend, no signal). A Stop and every
other abort still close and stop the Runner.
- On the server, `nativeRecoveryCommandUnanswered` recognizes the
deadline. The failure records `adoptedRunnerUnanswered`, so the next
claim carries `replaceUnansweredRunner`. That attempt fences the old
Runner lineage, launches a fresh Runner from the saved journal and
resumes the same provider thread (the #21272 path). It does not adopt
the slow Runner again for another 900 s.
- Update `doc/execution-semantics.md`.

## Verification

- Real `paperclip-runnerd` + fake Codex
(`runnerd-codex-transport-remote-retention.test.ts`): a retained Runner
is adopted, authenticates, then stops answering. A deadline abort leaves
it alive with no pending `turn.stop`/`runner.suspend`/`runner.shutdown`.
A Stop abort still stops it.
- Red/green: without the fix, the real-runnerd deadline test fails
(`expected 'gone' to be 'alive'`), the driver unit test fails (detach
called 0 times), and the server classification test fails. With the fix,
all pass.
- Measurements on real runnerd: the deadline cleanup took 10.0 s and
killed the Runner before. Now it returns in 0 ms and the Runner stays
alive. The replacement over the kept Runner's journal resumes the same
provider thread in about 2.2 s with one `thread/start` and one
`turn/start` (no replay).
- `npx vitest run
src/live/runnerd-codex-transport-remote-retention.test.ts
src/drivers/codex/codex-app-server-driver.recovery.test.ts
src/native-session-runtime.test.ts` (368 passed).
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "never
answered its recovery read"` (passed).
- `tsc --noEmit` for `packages/paperclip-runner` and `server` (clean).

## Risks

- A deadline now leaves a live Runner behind. The next attempt replaces
it through the launch fence, which fails closed when it cannot prove the
old lineage gone. This is the same risk as the existing recovery read
timeout detach.
- When the provider attempt budget is already spent, the kept Runner is
not replaced by this run. This also matches the existing timeout detach.
- Not in this PR: the upstream trigger. The first attempt abandons a
slow turn start (`runnerd did not report the provider turn identity` or
`turn.start timed out`) while Codex later runs that turn. Two sampled
chains then stopped the Runner on
`protocol_failure:tool_binding_mismatch` / `thread_binding_mismatch`
after a deploy reattachment. These need a separate fix.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:30:22 -06:00
DottaandClaude Opus 5.5 d9a4bb23fe fix(runner): wait for a slow Codex turn start while it reports progress (#21329)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Codex agents run through the Paperclip runner, and the server-side
transport starts each provider turn
> - After Codex accepts a turn, the transport waits a fixed 30 s for
that turn's `turn/started` before it reports the turn identity
> - Codex starts its MCP servers before it starts the turn, so a healthy
cold start in a loaded sandbox often takes longer than 30 s
> - The transport then failed a live turn with `runnerd did not report
the provider turn identity`, and the abandoned turn's late notifications
broke the reattachment (`did not name its execution owner`,
`runner_harness_state_mismatch: resumed_not_suspended`)
> - This pull request sizes the start window to the measured cold start
and extends it while the provider keeps reporting startup progress
> - The benefit is that slow but healthy Codex starts succeed on the
first attempt, and a dead start still fails at a bound

## Linked Issues or Issue Description

**What happened?**

On staging, Codex runs on Daytona failed at turn start with
`native_session_interrupted: runnerd did not report the provider turn
identity`. The run log shows the same sequence each time. The runner
accepts the turn. Codex then reports startup progress (a bubblewrap
config warning, `account/updated`, and `mcpServer/startupStatus/updated`
failures for an MCP server that is not logged in) for about 30 s. The
transport gives up 30 s after acceptance. A few seconds later, Codex
sends the turn's `turn/started`, which now arrives at a closed turn
(`protocol_failure:turn_binding_mismatch`). The same-run reattachment
then hits that orphaned turn. It fails with `Provider authoritative
notification item/started did not name its execution owner.` or
`runner_harness_state_mismatch: resumed_not_suspended`.

Of 198 recent runs in the staging company, 52 contain this turn identity
failure. In 68 successful starts, submit to `turn/started` took p50 25
s, p75 37 s, p90 50 s, p95 54 s and max 76 s. The fixed 30 s wait sits
near the median.

**Expected behavior**

A slow but healthy Codex start succeeds. A start that makes no progress,
whose Runner exits, or that the runner fails still fails at a bound and
recovers through the normal same-run resume.

**Steps to reproduce**

1. Start a Codex turn whose provider accepts the turn but sends
`turn/started` after the 30 s default window (on staging: several MCP
servers and a loaded sandbox).
2. Let the provider report startup notices during the wait, and let the
runner not prove the accepted turn live in time (on staging no turn was
adopted).
3. See `runnerd did not report the provider turn identity`, then the
reattachment conflict.

The new tests reproduce this with the fake Codex app server
(`--turn-started-delay-ms` with `--turn-started-progress-ms`) and a
runner snapshot that cannot prove liveness.

**Paperclip version or commit**

Staging Core `c730872974` (`2026.1106.0+2253`). Base of this PR:
`origin/master`.

**Deployment mode**

Authenticated, public, managed cloud (staging). Remote runner on
Daytona.

**Agent adapter(s) involved**

`paperclip_runner` with the Codex provider (`gpt-6.1-sol`).

## What Changed

- `runnerd-codex-transport.ts`: the default wait for an accepted turn's
`turn/started` is now 90 s (measured max 76 s), not 30 s. An explicit
`turnStartTimeoutMs` keeps its old bound.
- The wait restarts its window each time the runner commits a provider
notice, or a runner diagnostic that names the accepted turn. The total
wait stops at 180 s (option `turnStartProgressCeilingMs`, so tests can
lower it). A provider relaunch after the accepted `turn.start` does not
extend it.
- The adoption check (runner snapshot proves the accepted turn live) now
runs before each window end, not only before the first one. A Runner
exit and a runner failure of the pending turn still fail at once.
- `fake-codex-app-server`: new `--turn-started-progress-ms` flag sends
`mcpServer/startupStatus/updated` failures while it delays
`turn/started`.
- `doc/execution-semantics.md`: documents the start window, the progress
rule and the staging measurements.

## Verification

- Red on `origin/master` transport (with the new fake flag): `keeps
waiting for an accepted Codex turn while its startup keeps reporting
progress` failed after 4.6 s, and `gives a silent Codex cold start the
measured default start window` failed after 30.9 s. Both failed with
`runnerd did not report the provider turn identity`.
- Green: `npx vitest run
src/live/runnerd-codex-transport-deadlines.test.ts` passes 9/9. This
includes a ceiling test: provider notices every 1 s and no
`turn/started` fail at an 8 s test ceiling, and the test goes red with
the ceiling cap removed. It also includes the existing adoption,
no-adoption-within-bound and runner-failure tests, and a new test that a
silent start still fails within its bound.
- `npx vitest run` on the adoption, cut-turn-start, stall,
turn-start-timeout-retention, remote-retention, restart-continuation and
controller-outage transport suites passes 48/48.
- `npx vitest run src/live/runnerd-codex-transport.test.ts
src/live/live-session.test.ts src/live/turn-stream.test.ts`: 391/392
pass. The one failure (`stops a lost runnerd's provider and escaped tool
job from its lineage journal (journal=true)`, 1 s, process-lineage
timing under full-suite load) passes when run alone, both on this branch
and on `origin/master`.
- `tsc --noEmit` for `packages/paperclip-runner` and `cargo fmt --check`
pass.

## Risks

- A silent dead start (no Runner exit, no provider output, no runner
failure) now fails after 90 s, not 30 s. The Runner exit and
runner-failure paths still fail at once, so this applies only to a
provider that stops without a trace.
- A provider that keeps sending notices but never starts the turn waits
at most 180 s.
- No schema, API or UI change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, browser reads of staging run data).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

Refs #21272, #21120

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:21:37 -06:00
Nicky LeachandClaude Fable 5.2 314b893678 fix(native-runtime): deliver assigned gateway tools to every native harness through the PRP relay (#21274)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An agent can have app connections. Paperclip gives the agent the
tools of those apps through an MCP gateway, with its approval and audit
rules.
> - Native harnesses reach the gateway in two ways today. In the relay,
the server keeps the gateway token and offers the tools over the
authenticated Runner channel. In the direct way, the server gives the
gateway URL and a bearer token to the harness.
> - Which way a harness uses comes from a list of special cases that
grew one harness at a time. Local Codex and all ACPX agents use the
direct way. The others use the relay.
> - The direct way puts a run-scoped token in a provider process. A
process that the agent starts can read it. A provider process that
serves more than one run can hold a revoked token.
> - This pull request replaces the list with one rule. Each native
harness gets its assigned gateway tools through the relay.
> - The benefit is one behavior for all native harnesses, and no gateway
token in a provider process.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

Refs #13942 (the relay), #17149 (ACPX accepts relayed tools) and #19912
(the design for turn-scoped ACPX MCP authority).

**What existing behavior does this improve?**

How a native harness reaches the tools of its assigned MCP gateway.

**Subsystem affected**

Native runtime in the server (`native-session-executor.ts`,
`native-session-resume.ts`). One Runner test file. One Rust test.

**Current behavior**

- `relayAssignedMcp` is true for plugin tasks, Cursor Cloud, OpenCode,
Codex on a remote target, and Dot.
- For local Codex and for each ACPX agent (Claude, Codex, Cursor, Pi,
Kimi, Grok, Copilot, Antigravity), Core puts
`PAPERCLIP_NATIVE_MCP_NAME`, `PAPERCLIP_NATIVE_MCP_URL` and
`PAPERCLIP_NATIVE_MCP_TOKEN` in the Runner environment. The ACPX host
gives them to the provider as an HTTP MCP server. The local Codex home
gets them in `config.toml`.
- The token is run-scoped. Core revokes it when the run ends.

**Proposed behavior**

- A native run that has an assigned gateway uses the relay. The
condition names no provider.
- Core builds the gateway tools on the control plane, offers them with
the other Runner tools, dispatches each call over PRP, and authorizes
each call again through the gateway.
- Core removes the three `PAPERCLIP_NATIVE_MCP_*` variables from each
Runner environment.
- The tool contract fingerprint has the relay marker for each target,
not only for remote targets.

**Reason and benefit**

- All native harnesses have the same behavior.
- No provider process holds a gateway token.
- A sandbox does not need to reach the gateway URL of the host.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`
- The list of special cases is gone. Core validates the binding and
builds the relayed tools when a binding is present.
- Core does not list the gateway for a review run, Grok Bot or Muse. The
tool authority never offered app tools to those. A gateway error can
therefore not fail those runs.
- OpenCode keeps its `on_demand` catalog. That option depends on the
provider, not on the relay.
  - Core removes the three variables from each Runner environment.
- `server/src/services/native-runtime/native-session-resume.ts`: the
relay marker in the tool contract fingerprint is unconditional.
- `doc/execution-semantics.md`: the text about the tool contract
fingerprint now states the rule, the tool names and the cutover.
- App tool schemas. New module
`packages/paperclip-runner/src/drivers/tool-input-schema.ts` compiles
each schema by itself, in its declared dialect (draft-07, 2019-09 or
2020-12), with no shared `$id` registry. Core uses it when it builds the
relayed tools (`assigned-mcp-tools.ts`). Core does not offer a tool
whose schema has an unsupported dialect or does not compile, and writes
a log entry with the tool name and the reason. The run does not fail.
The ACPX tool bridge and the OpenCode MCP bridge use the same function.
The tools of Paperclip itself stay strict.
- Large arguments. The ACPX sidecar sends the arguments of a tool call
to runnerd with no display limit. The limit is the tool-call admission
limit of runnerd, 256 KiB. A larger call fails with a tool error that
gives the limit. The sidecar does not dispatch it.
- Tests in `native-session-executor.test.ts`,
`native-session-resume.test.ts`, `runtime-context-materializer.test.ts`
and `acpx_provider_backend.rs`.

Not changed in this pull request: the direct code path. It is still in
the Runner, but it receives no binding. #21311 deletes it.

## Behavior that each harness sees

| Harness | Before | After |
|---|---|---|
| Codex on a remote target, OpenCode, Cursor Cloud, Dot, plugin tasks |
Relay | No change |
| Local Codex | Direct: a `paperclip-assigned` server in `config.toml` |
Relay. App tools are dynamic tools named `app_<name>_<digest>` |
| ACPX agents | Direct: a second HTTP MCP server | Relay. App tools are
on the `paperclip` bridge as `mcp__paperclip__app_<name>_<digest>` |
| ACPX with a preconfigured home | No gateway | No change |
| Grok Bot, Muse | Got the variables and ignored them | Do not get the
variables |

- A relayed tool name has a maximum of 64 characters. With the
`mcp__paperclip__` prefix it has a maximum of 80.
- Permission behavior keeps the same intent. For Claude in the
`approve-reads` and `approve-paperclip` modes, app tools ask for
approval, as before. For the other ACPX agents the permission decision
does not use the server name.
- In a planning run or an ask run, the relay offers only tools with read
risk. The direct gateway offered all tools. This is stricter.
- When the catalog is too large, each harness gets the two compact tools
`paperclip_search_assigned_tools` and `paperclip_call_assigned_tool`.
- The four helper tools for resources and prompts
(`paperclip_list_resources`, `paperclip_read_resource`,
`paperclip_list_prompts`, `paperclip_get_prompt`) are not in the relay.
No harness loses them: a run token permits only `tools/list` and
`tools/call` (`run-preparation.ts`), so these four tools already fail
with `gateway_token_action_denied` on the direct path.

## State saved by the previous release

- **Local sessions of each provider start again one time.** The tool
contract fingerprint changes for local targets. The next run sees a
different fingerprint, starts a new provider session with the full task
context in the same workspace, and keeps the old saved state. This is
the existing rotation for a tool contract change. Codex needs it,
because Codex cannot add tools to a thread that it resumes. For the
local providers that already used the relay, the restart is not
necessary. It is accepted so that the rule names no provider.
- **Remote sessions continue.** Their fingerprint does not change. For
ACPX, the attach of the next run replaces the saved tool set.
- **A kept Codex home.** Paperclip writes `config.toml` again at each
start and recovery. The old gateway server and its token are removed.
- **A deploy lands during a run, and the Runner survives.** The
controller attaches to the same Runner for the same run. The old
provider keeps its direct gateway connection. The controller detach
keeps the gateway token of that run, and Paperclip revokes it when the
run ends.
- **A deploy lands during a run, and the Runner is dead.** The Runner
starts again for the same run with its saved tool set and no direct
gateway. The assigned app tools are not available for the remainder of
that one run. The next run gets them through the relay. This applies to
each harness that moves to the relay. It has no test.
- Paperclip refuses no saved state and discards none.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-session-resume.test.ts`: `1508 passed
| 7 skipped`, 1 failed. The failed test is "archives a damaged prior
epoch before completing guarded same-task replacement with real runnerd"
(`ENOENT … authority-epochs`). It fails in the same way on `master` at
53608bd1d6 and with the product change removed, on macOS.
- `cd server && npx tsc --noEmit -p tsconfig.json`: no errors.
- `cd packages/paperclip-runner && npx vitest run
src/drivers/runtime-context-materializer.test.ts`: `18 passed`. `npx tsc
--noEmit -p tsconfig.test.json`: no errors.
- `cargo fmt --manifest-path runner/Cargo.toml --all -- --check`: no
changes. `cargo test -p paperclip-runner-core --lib
acpx_provider_backend`: `50 passed`.

- `npx vitest run src/drivers/tool-input-schema.test.ts
src/drivers/runner-tool-bridge.schemas.test.ts
src/drivers/runner-tool-bridge.test.ts
src/cli/acpx-runtime-sidecar.test.ts src/drivers/opencode` in
`packages/paperclip-runner`: `21 passed (21)` files, `377 passed` tests.
`pnpm run typecheck:typescript`: no errors.
- `cargo test -p paperclip-runner-core --test acpx_event_payload --test
tool_input_schema_fixtures`: 10 and 3 passed.
- Schema tests use one shared fixture
(`tool-input-schema-fixtures.json`) for the TypeScript compiler, the
tool bridge and the Rust `begin_call`: a draft-07 schema, a second
schema with the same `$id`, 2019-09, 2020-12, draft-04, an unknown
dialect, a `$ref` with no target, and an invalid keyword. Before the fix
the bridge failed with `no schema with key or ref
"http://json-schema.org/draft-07/schema#"`.
- A tool schema whose `$id` is the URI of a meta-schema cannot damage
the shared compiler. Before this fix, `removeSchema` deleted that
meta-schema, and later schemas failed to compile for each run in the
process. A test with 7 reserved ids failed before the fix and passes
now.
- Core limits the compile work of each run start. A schema over 512 KiB
is not compiled, and one listing compiles a maximum of 768 KiB of
schemas. These are the limits of the Runner. A tool over a limit is
served only through the search and call tools. Core keeps the outcome
for each schema digest (1,024 entries) and yields to the event loop
before each compile. Four tests failed before this change.
- A sidecar test sends a tool call with 200 KiB of arguments and
receives them byte for byte, refuses a call with 300 KiB with the named
error, and keeps the replacement of a lone surrogate.

Tests, each seen to fail without the product change:

- "keeps assigned MCP credentials on the control plane for %s" now runs
for remote Codex, local Codex, OpenCode and ACPX Claude, Pi, Kimi,
Copilot and Cursor. It checks that Core builds the relay, offers the
tool, dispatches a call, and gives no `PAPERCLIP_NATIVE_MCP_*` variable
to the Runner. Without the change, the five new harness cases fail.
- "relays the assigned gateway when a deploy lands mid-run…" runs for
Claude and Pi.
- The fingerprint tests keep the old local fingerprint as a retained
value and check that the new one is different. The remote fingerprints
for Codex and OpenCode do not change.
- "drops a retained direct gateway server when the relayed launch has no
binding": a Codex home with the old `config.toml` entry loses that entry
at the next start and at a recovery of the same run.
-
`next_run_attach_offers_relayed_gateway_tools_to_a_direct_gateway_checkpoint`
runs for Claude and Pi.

Not run: a live run with a real provider and a real app connection for
each harness. Not run: `pnpm verify` and the full suite. CI supplies
that coverage.

## Risks

- **An ACPX app call has a limit of 256 KiB for its arguments.** The
direct gateway accepted a request body of 10 MB. Codex and OpenCode
permit 480 KiB. To get nearer to 10 MB, runnerd needs a larger admission
limit for tool calls or a protocol that sends arguments in parts. That
is not in this pull request.
- **A tool with a draft-04 or draft-06 schema is not offered.** Ajv 8
does not support draft-04, and the TypeScript and Rust validators must
agree. On the direct path the provider got such a tool.
- **Core compiles app schemas on its event loop.** Before this pull
request Core did not compile them. With the limits above, one valid
schema near 512 KiB can hold the loop for about 0.65 s, and one listing
for about 1.2 s in total (measured on a developer machine). An app that
changes its schemas at each listing gets no help from the cache. A
worker thread with a timeout would remove this; it is not in this pull
request.
- **Tool names change for local Codex and all ACPX agents.** A saved
agent instruction that names a gateway tool directly does not match the
new name.
- **Each local native session starts again one time after the deploy.**
The provider conversation of that session does not continue. The task
context and the workspace stay.
- **A run that is in flight at deploy, with a dead Runner, has no app
tools until it ends.**
- **Names of 80 characters are not tested live** with Kimi, Copilot,
Grok, Antigravity and Codex through ACPX. Pi already shortens a name
longer than 64 characters.
- **Managed Claude and AWS AgentCore now receive relayed app tools.**
This is not tested live.
- **A gateway token expires after 1 hour.** The relay keeps the token on
the controller for the whole run, as OpenCode and remote Codex do today.
A longer run loses its app tools on both paths. This pull request does
not change that.
- **The gateway URL must have the path `/mcp/gateways/<id>`.** With a
path prefix in the API base URL, a run with an assigned gateway fails
with `assigned native MCP gateway path is invalid`. This was true for
the relayed harnesses before. It is now true for all.
- No test covers the rule that Core does not list the gateway for Grok
Bot and Muse.

## Follow-up

The direct path is now unreachable. #21311 deletes it. That pull request
is in the same stack, above the pull requests that change the same ACPX
host files.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change, reviewed the diff and ran the verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), investigated each harness
and wrote the change and the tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 00:16:58 -08:00
DottaandClaude Opus 5.5 e73fc71331 fix(environments): continue on a new sandbox when an idle runner sandbox never starts (#21323)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task that runs on a reusable cloud sandbox resumes the same
sandbox on the next wake
> - On staging, Daytona left two idle sandboxes in the `archiving` state
for more than three hours, so no resume call could succeed
> - The native runner leases had no verified harness backup, so the run
never replaced the sandbox, even though the task files were saved
> - Each retry spent six minutes on the same stuck sandbox, the bounded
retries ran out, and the task waited for a person to press Retry
> - This pull request continues on a new sandbox when the provider has
proven it cannot start the old one and the workspace is saved, and it
keeps the old sandbox
> - The benefit is that the task recovers by itself and a sandbox that
may hold unsaved runner state is never destroyed

## Linked Issues or Issue Description

Refs #20626, #21113

**What happened?**
Eight Codie runs on staging failed in one hour with
`reusable_sandbox_resume_failed`: `Reusable sandbox lease "<id>" could
not be resumed (cause=transport_timeout); the lease was preserved and no
replacement was created.` They came in two clusters (05:59 to 06:00Z and
06:32 to 06:38Z). Each run took about six minutes, so the 330-second
idle resume budget from #21113 was applied. All runs were already
`transient_failure_retry` wakes.

Investigation of one instance (lease `c23d6120…`, run `d16c272e…`):
- The lease was `released` at 02:37Z with a confirmed stop-and-retain.
No live run held it, so the lease was idle and the extended budget
applied.
- A read-only Daytona API read at 06:5xZ showed the sandbox in state
`archiving` (desired state `archived`), last updated 03:38Z, one hour
after the stop (the auto-archive interval is 60 minutes). The second
sandbox (`e0e6cfe5…`) showed the same state since 03:35Z. Daytona does
not start a sandbox while it archives, so every resume call timed out.
More time or a new HTTP connection could not help.
- The task workspace was clean and had a committed saved snapshot.
- The agent uses `paperclip_runner`. The last run on the sandbox failed
while it saved the runner session, so the lease had no
`nativeHarnessBackup` stamp. `reusableSandboxReplacementIsSafe` requires
a verified harness backup for a runner lease, so the run never replaced
the sandbox. It failed strictly, the bounded retries (2) ran out, and
the task waited for a manual Retry.

**Expected behavior**
When an idle sandbox never starts through the whole provider start
deadline and the task files are saved, the run continues on a new
sandbox automatically. A sandbox that may hold unsaved runner state is
not destroyed.

**Steps to reproduce**
1. Run a `paperclip_runner` task on a reusable Daytona sandbox. Let the
last run end without a harness backup (for example, the runner exits
while it saves the session). The workspace is clean with a committed
snapshot.
2. Let Daytona start to archive the stopped sandbox and get stuck in
`archiving`.
3. Wake the task. Every `environmentResumeLease` call times out, the run
fails after about 330 seconds with `reusable_sandbox_resume_failed`, and
the retries fail the same way.

**Paperclip version or commit**
Staging Core `0c04ff76eb` (current `master` has the same code path).

**Deployment mode**
Hosted staging (cloud), Daytona sandbox provider plugin, reusable leases
on.

**Agent adapter(s) involved**
`paperclip_runner` (native runner) on a Daytona sandbox environment.

**Root cause**
`server/src/services/reusable-sandbox-resume-replacement.ts`: a runner
lease without a verified harness backup was never replaceable, for any
failure. The only other path was to retry the same sandbox, which cannot
work while the provider is stuck. Replacement also always retired the
old lease to `pending_cleanup`, so allowing it without a backup would
have destroyed the only copy of the runner session state.

## What Changed

- `reusable-sandbox-resume-replacement.ts`:
`reusableSandboxReplacementIsSafe` becomes
`reusableSandboxReplacementPlan`. It returns `retire` (same rules as
before), `keep`, or `null`. It returns `keep` for a runner lease without
a verified harness backup only when the caller passes
`providerCannotStart`. All other conditions still apply: idle lease,
same workspace, clean workspace, committed snapshot, this run is the
writer, and no access refusal.
- Add `keepReplacedReusableSandboxLease`. After the replacement lease is
recorded, one guarded write marks the old lease with
`reusableSandboxKeptAfterReplacement`. The lease keeps its status and
its sandbox. Nothing destroys it.
- `environment-runtime.ts`: set `providerCannotStart` only when the idle
start deadline was extended and the run then used all of it on timeouts.
Lease selection skips a lease with the keep marker, so later runs do not
spend time on it and obsolete-lease cleanup does not destroy it.
- The replacement records `replacement` / `resume_failed` with no
harness backup. The native executor already starts a fresh session in a
new sandbox in this case (`freshReplacement`), and workspace preparation
restores the committed snapshot.
- The keep plan needs every resume call to time out. A rate limit, a
reset, or any other failure in between keeps the strict behavior.
- `reusable-sandbox-kept-marker.ts`: the marker key and a SQL guard.
Every automatic teardown path skips a marked lease: retained-orphan
listing and settle, `destroyReusableSandboxLease` (obsolete cleanup,
expiry, run destroy), `destroyClaimedSandboxLease` (issue and workspace
teardown), recovered-workspace cleanup, and abandoned-file release. Only
an operator-consented environment delete still destroys it.
- `doc/execution-semantics.md`: document the rule.

## Verification

- Red before the fix: the new staging-shape test in
`server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts`
(released runner lease, no harness backup, clean committed snapshot,
every resume times out) failed with the staging error `could not be
resumed (cause=transport_timeout); the lease was preserved and no
replacement was created.`
- Review round: "keeps an idle sandbox with no harness backup when some
resume calls failed for another reason", "never destroys a sandbox kept
after replacement through the run or workspace teardown paths", and the
retained-orphan case "keeps a warm sandbox while a replacement kept it"
each failed before the fix and pass after it.
`environment-retained-orphan-sandbox.test.ts`,
`abandoned-workspace-files-hold.test.ts`,
`workspace-recovery-preparation-gates.test.ts`,
`workspace-recovery-admission.test.ts`: all pass.
- Green after the fix: 22/22 pass in the replacement suite. The run gets
a new sandbox within 330 seconds, makes no destroy call, the old lease
stays `released` with the keep marker, and the next run resumes the new
sandbox only. With unsaved work or no saved snapshot, the run still
fails strictly and does not mark the old lease. Earlier tests (a refusal
without a backup, a live owner, a dirty workspace with a timeout) are
unchanged.
- `pnpm exec vitest run src/__tests__/environment-runtime.test.ts
src/services/environment-runtime-reusable-lease-probe.test.ts
src/__tests__/reusable-leases-default.test.ts`: 215 pass.
- `tsc -p server/tsconfig.json --noEmit` passes.
- Not run: a live replacement on staging. It is verified after deploy on
the next wake of the affected tasks.

## Risks

- A runner task that hits this path starts a fresh runner session. The
files come from the saved snapshot, and the task, comments, and run
history are kept. Before this change, the task stopped and waited for a
person.
- The kept sandbox stays with the provider (archived storage) until
someone removes it. This is the cost of never destroying unsaved runner
state.
- The rule needs the full extended start deadline (about 330 seconds) of
timeouts, so a slow but healthy start still resumes the same sandbox.
- No schema change. Old leases follow the same rule.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:06:25 -06:00
DottaandClaude Opus 5.5 86f9555695 fix(native): give a Runner whose first unanswered read came on the last attempt one replacement attempt (#21322)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs execute in a remote Runner, and a same-run retry
reattaches that Runner after an interruption
> - #21272 made the next attempt replace a Runner that left its recovery
read unanswered, instead of adopting it again
> - On staging, three runs after #21272 went live still failed: their
first unanswered read came on the third and last attempt, so no next
attempt existed
> - This pull request gives such a run one more attempt, used only for
that replacement
> - The benefit is that the replacement from #21272 can run when a
restart used up the middle attempt

## Linked Issues or Issue Description

Refs #21272, Refs #21161

**What happened?**

After #21272 (497ccba8f1) went live on staging at 05:34Z, three Codex
runs still failed with `native_session_recovery_failed:
HarnessCapabilityUnavailableError: read is unavailable: Error:
provider_transport_failed: PRP command session.snapshot timed out
(runner connected; pending ahead 0; last runner event seq … 55–84s
ago)`. The run events of all three show the same sequence:

1. Attempt 1 fails at 06:24 with `runnerd did not report the provider
turn identity`.
2. Attempt 2 starts at about 06:25. The 06:24–06:25 server restart ends
it with `local_runtime_api_listener_unavailable`. This attempt sends no
recovery read.
3. Attempt 3 adopts the same Runner. Its first recovery read gets no
answer. Attempt 3 is the last of three, so the run ends.

`adoptedRunnerUnanswered` was recorded on attempt 3, but no next attempt
read it. The replacement from #21272 never ran.

**Expected behavior**

A Runner whose first unanswered read comes on the last attempt is still
replaced once by a fresh Runner from the saved journal. The provider
turn is not sent again. The run does not end only because a restart used
the middle attempt.

**Steps to reproduce**

1. Start a Codex run in a remote Daytona Runner. Let attempt 1 fail at
turn start, so the Runner stays alive.
2. Restart the server during attempt 2, so that attempt fails without a
recovery read.
3. Let the Runner leave its recovery read unanswered on attempt 3.
4. See the run end with `session.snapshot timed out` and no replacement.

**Paperclip version or commit**

Staging with #21272 (497ccba8f1). Fixed on `master` after `bbafa78fe2`.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`).

## What Changed

- `nativeUnansweredRunnerReplacementDisposition` (executor): when
attempt 3 fails because its adopted live Runner left a recovery read
unanswered, and that attempt was not itself a replacement, the failure
stays `retryable_failure` (`native_session_interrupted`, retry in 30 s)
instead of `native_session_retry_exhausted`.
- `claimNativeRestartRecoveries`: a failed remote retry at attempt 3
with `adoptedRunnerUnanswered` is not treated as an exhausted budget.
The claim is `reattach_remote_runner` at provider attempt 4 with
`replaceUnansweredRunner`.
- `nativeAdmittedProviderAttemptLimit` (executor admission gate): a
`reattach_remote_runner` claim with `replaceUnansweredRunner` is
admitted up to provider attempt 4. Every other claim keeps the limit of
3.
- `verifyRemoteRunnerOrSettledExit`: the replacement of a live Runner is
allowed up to attempt 4. An exited Runner keeps the fail-closed path.
- Bound: the extra attempt is a replacement (`replaceUnansweredRunner`),
so it never gets another one. Attempt 4 is exhausted by the existing
rule.
- `doc/execution-semantics.md` describes the extra attempt.

## Verification

- End to end (embedded Postgres):
`native-remote-retry-admission.test.ts` › "executes the one replacement
attempt after the last attempt's first unanswered recovery read, and no
further one". It seeds attempt 3 with the staging failure, claims it,
and executes the claim through `executePaperclipNativeSession`. The
provider is reached once, the run then ends at attempt 4, and no further
claim is made. With the old `nextAttempt > 3` gate this test fails
before the provider is reached.
- Red on `master`, green with the fix (7 tests, including "admits
provider attempt 4 at execution only for the replacement claim"):
- `native-runner-restart-recovery.integration.test.ts` › "gives a Runner
whose first unanswered read came on the last attempt one replacement
attempt: true" (embedded Postgres; the coordinator is seeded at attempt
3 with the staging message). On `master` no reattach claim was made.
- `native-session-executor.test.ts` › "bounds the replacement after …"
(4 cases: the last attempt's first unanswered read is retryable; a
replacement that also went unanswered, another failure, and attempt 4
stay terminal) and "replaces the live Runner on the extra replacement
attempt".
- Focused suites: `native-session-executor`, `native-restart-recovery`,
`native-runner-restart-recovery.integration`,
`runner-lost-mid-turn-continuation`: 1737 passed, 21 skipped.
- `pnpm typecheck` in `server` passes.
- Not run locally: the container-gated remote Runner suites and the full
suite (CI covers it). Why the remote Runner stops answering is a
separate investigation and is not changed here.

## Risks

- Low to medium. This adds at most one provider attempt per run, and
only after an unanswered recovery read on a live adopted Runner. The
attempt replaces the Runner through the launch fence, which proves the
old Runner and its lineage gone or fails closed. No provider turn is
sent again.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code
with tool use (shell, file edits, browser reads of staging run events),
extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 01:27:35 -06:00
DottaandClaude Opus 5.5 b555087f0f fix(native): continue again when a deploy interrupts an automatic continuation (#21314)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs execute in a remote Runner (Daytona) that the
server controls; a deploy restarts the server while runs are in flight
> - When the restart's reattachment proves the Runner process gone
mid-turn, #21039 makes the server request one automatic Retry that
continues in a fresh session from the saved work, without replaying the
turn
> - Staging deploys every 20 to 45 minutes, and Codex runs often take
longer, so that automatic continuation is often interrupted by the next
deploy too
> - The hook treated that second loss as "automatic Retry already used"
and left the task blocked for a person
(`native_continuation_requires_reconciliation`)
> - This pull request gives an automatic continuation one more
fresh-session continuation each time a server restart or deploy took its
controller, and a deploy spends no automatic budget
> - The benefit is that a task interrupted by repeated deploys continues
by itself, and no provider turn that may have run is sent again

## Linked Issues or Issue Description

Refs #21039, Refs #21052, Refs #21272, Refs #21023, Refs #21120, Refs
#21161

**What happened?**

On staging, the largest failure class for a Codex agent on Daytona was
`runner_remote_process_identity_unavailable` with error code
`native_continuation_requires_reconciliation`: 18 runs in 3 hours. The
error name is misleading. It is the shared code for every refused remote
reattachment (`REMOTE_RUNNER_RECOVERY_ERROR_CODE`). In every sampled run
the refusal was `process_verification` / `process_not_running`: the
Runner process was gone, its stderr log was empty, and the last turn
event was `turn.submitted` or `turn.accepted` (unsettled). It was not a
missing identity file.

Sampled 12 such runs. In 9 the #21039 hook worked: it recorded
`native.interrupted_turn_outcome_unknown` and requested the
fresh-session Retry. The tasks that stayed blocked had one shape:

1. A run lost its Runner after a deploy. The hook requested the
automatic Retry (key `task-retry:restart-interrupted-turn:<run>`).
2. The Retry run started a fresh session. The next deploy came during
its turn (a graceful `SIGTERM` shutdown transition, then "Recovering the
existing native runner process after server restart").
3. The same-run reattachment found that Runner gone too
(`process_not_running`).
4. The hook returned `automatic_retry_used`, because the run's Retry
intent came from the hook. The task stayed blocked.

Examples: a `task_retry` run created 40 s after its predecessor failed
(the hook's Retry), interrupted by the 05:17 deploy, failed at 05:18,
task blocked. A second chain (assignment, then the hook's Retry) failed
the same way at 05:12.

**Expected behavior**

A deploy is not the task's failure. When a server restart or deploy took
the controller of an automatic continuation and its Runner is proven
gone, the server continues the task again in a fresh session from the
saved work. It never replays the turn. A loss without restart evidence
keeps the existing one-time rule.

**Steps to reproduce**

1. Start a long Codex run in a remote Daytona Runner.
2. Deploy the server during its provider turn so that the reattachment
finds the Runner gone. The server requests the automatic Retry.
3. Deploy again during the Retry run's turn so that its Runner is also
lost.
4. See the Retry run fail with
`runner_remote_process_identity_unavailable` and the task stay blocked
with no new run.

**Paperclip version or commit**

Staging Core `25e9a9c2e9`. Fixed on `master` after `731b52453c`.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`, `codex_app_server`).

## What Changed

- `restart-interrupted-turn.ts`: `lostControllerToRestart` finds restart
evidence on the run: a `native.recovery.transition` with `restartKind:
"graceful"` (shutdown), or a restart claim (`reattach_remote_runner`,
`reattach_existing_runner`, `resume_dead_runner`,
`bootstrap_incomplete`) without the `same_run_retry` trigger.
- Both `automatic_retry_used` refusals (a person's Retry that the hook
requested, and a system retry of owed work) now continue when the run
has restart evidence. A deploy interruption spends no budget, so a task
that only deploys interrupt never runs out. Each continuation needs a
new deploy during its run, so deploys bound it. A loss without restart
evidence keeps the one-time rule. All other gates are unchanged: proven
Runner exit, unknown-turn record, the issue still assigned and open,
idempotency per run, and the Retry's admission gates.
- `doc/execution-semantics.md` describes the rule.

## Verification

- Red before the fix, green after:
`server/src/services/runner-lost-mid-turn-continuation.test.ts` › "an
automatic continuation that the next deploy interrupted". The test seeds
the staging sequence: a lost run, the hook's Retry run with a
`turn.submitted` turn, the shutdown and restart transitions with staging
payloads, then a same-run reattachment that proves the Runner gone. On
the old code the two continuation tests return `automatic_retry_used`.
With the fix they request one fresh-session Retry, record the unknown
turn outcome once, recover the saved work and queue the successor with
`forceFreshSession: true`.
- A chain of 12 deploy continuations, each with a system retry that
inherits its Retry receipt, still continues (red on the first revision
of this PR, which had a budget of 8). Losses without deploy evidence
stay refused (with and without earlier deploy continuations).
- `npx vitest run
src/services/runner-lost-mid-turn-continuation.test.ts` 23/23 pass.
`governed-wait-runner-loss.test.ts` and
`workspace-recovery-retry-copyback.test.ts` 103/103 pass.
- `pnpm --filter @paperclipai/server typecheck` passes.
- `pnpm verify` exits 2 (a global doc input changed), so required CI
supplies the broad coverage.

## Risks

- Low to medium. A task whose Runner is lost on every deploy now
continues after each deploy instead of once. Each continuation is a full
run, so it costs tokens. The deploy count bounds it, because each
continuation needs a new restart during its run.
- Not fixed here: why the reattached Runner is gone after a deploy. In
the sampled runs the reattachment after the restart hung in `native
session provider recovery timed out after 900000ms`, and the next
attempt found the process gone with empty stderr. That needs its own
investigation.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, browser read-only API GETs on staging).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 00:41:32 -06:00
DottaandClaude Opus 5.5 4dc5492c2d fix(recovery): retry a failed Retry run's sandbox resume failure when the sweep reaches it first (#21124)
<!-- Write all pull request text in Simplified Technical English
(ASD-STE100): short sentences, one instruction per sentence, simple
approved vocabulary, and the active voice. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task that runs on a reusable cloud sandbox resumes the same
sandbox on its next run
> - When the resume fails before provider work
(`reusable_sandbox_resume_failed`), the run gets a bounded automatic
retry
> - On a hosted staging server, 8 of 14 such failures in 10 hours did
not get that retry. Each task ended `blocked` and needed a person to
press Retry
> - In all 8, the stranded-task sweep blocked the task between the
failed write and the run's own retry scheduling. The block cleared the
claim that the retry needed
> - This pull request makes the sweep give such a run to the same
bounded retry scheduler, not escalate it
> - The benefit is that a transient sandbox resume failure no longer
costs the task, also when settlement is slow or the server stops during
an upgrade

## Linked Issues or Issue Description

Refs #21108, #21113, #21078

**What happened?**
A person pressed Retry on a task. The run tried to resume the task's
saved Daytona sandbox before provider work. The resume failed
(`cause=transport_timeout`, or `provider_unavailable status=503`), and
the run was written `failed` with `reusable_sandbox_resume_failed`. Then
the stranded-task sweep moved the task to `blocked` and cleared its
claim. This happened 1 to 43 seconds after the failed write. The run's
executor scheduled its bounded retry 2 to 81 seconds after the failed
write, and the gate refused it ("The automatic retry could not
revalidate the original user continuation."). Read-only audit of 14
failures (all legacy runtime, `executionRecovery: bootstrap,
providerWorkStarted: false`): 6 got their retry, and the 5 most recent
ones recovered. 8 were refused this way and wait for a person. In one
example the run was written failed at 14:23:31, the task was blocked at
14:24:14, and the retry was refused at 14:24:52. The server was being
replaced at that time.

**Expected behavior**
A transient preparation failure before provider work continues with its
bounded retry. The sweep must not take the claim away from that retry.
When the retry budget is spent, or the continuation is refused, the task
escalates once, as before.

**Steps to reproduce**
1. Press Retry on a task whose reusable sandbox resume fails before
provider work.
2. Let the run's executor be slow to settle the failure (more than the
remaining controller lease, at most 60 seconds), or stop its server
after the failed write.
3. Let the stranded-task sweep run on another server. The task becomes
`blocked`, and no retry is scheduled.

**Root cause**
`classifyContinuationFailure` lists `reusable_sandbox_resume_failed` and
the other transient preparation codes in
`NON_RETRYABLE_CONTINUATION_ERROR_CODES`. The sweep assumes that the
executor already scheduled the dedicated bounded retry, so it escalates
at once and never asks the scheduler. #21108 added a skip while the
executor still settles the run. That skip uses a live local execution or
an unexpired controller lease. The lease stops renewing when the run is
written terminal (renewal requires `status = 'running'`). So the skip
covers at most about 60 seconds after the last renewal, and it covers
nothing when the executor's server stops during settlement.

**Paperclip version or commit**
Staging at `a65644d1e1`. Current `master` has the same code path.

**Deployment mode**
Hosted (cloud), two servers alive during each upgrade.

**Agent adapter(s) involved**
`codex_local` on a Daytona sandbox environment. The change is
adapter-independent.

## What Changed

- `server/src/services/heartbeat/retries.ts`: add
`failedPreparationOwnsBoundedRetry(run)`. It is the persisted form of
the run-completion decision: a failed legacy run with no native session,
bootstrap evidence that provider work did not start, and a transient
preparation code. A restore failure counts only when its owner marked it
retryable. Codes that an accepted interaction retries itself are
excluded.
- `server/src/services/recovery/service.ts`: when the latest run is such
a run, the sweep does not escalate it as non-retryable. It goes to
`scheduleRecoveryRetry`, which is the run's own bounded scheduler. The
scheduler reuses a successor that already exists, so the executor and
the sweep create one successor. The task escalates when the scheduler
reports exhaustion or a refused continuation. When the scheduler
declines the retry for another reason (for example a permanent refusal),
the task escalates once as a non-retryable failure, as before. The
#21108 settling skip still runs first.
- `server/src/services/heartbeat.ts`: pass the predicate to the recovery
service.
- `doc/execution-semantics.md`: document the rule.

## Verification

- New tests in `server/src/services/task-retry.test.ts`, in the staging
shape: a Retry-admitted run failed with
`reusable_sandbox_resume_failed`, bootstrap evidence, and an expired
controller lease from another server.
- "retries a failed Retry run's transient preparation failure when the
sweep reaches it first" (`slow_executor`, `executor_gone`). Red before
the fix: the task became `blocked`, as on staging. Green after the fix:
the task stays `in_progress` with one `scheduled_retry` successor
(`transient_failure`, attempt 1), and dispatch accepts it. For
`slow_executor`, the executor's own scheduling then reuses the same
successor, and there is still one successor.
- "still escalates a failed Retry run's transient preparation failure
once its bounded retries are spent": the task becomes `blocked` with no
successor.
- "escalates a failed Retry run's transient preparation failure once
when the scheduler declines it permanently" (review finding): red before
the follow-up commit (the task stayed `in_progress` with no run), green
after (the task is `blocked` once, and a second sweep adds no notice).
- `pnpm exec vitest run src/services/task-retry.test.ts`: 114/114 pass.
- `pnpm exec vitest run src/__tests__/heartbeat-retry-scheduling.test.ts
src/__tests__/heartbeat-process-recovery.test.ts` (with task-retry): 631
of 633 passed. The 2 failures were 15-second timeouts under parallel
load in native Board-wait tests that this change does not touch. They
passed when run alone.
- `tsc -p server/tsconfig.json --noEmit` passes.
- Not run: a live staging reproduction. The staging evidence is
read-only (run events, task activity, run records).

## Risks

- The sweep can now schedule the bounded retry before the executor does.
The scheduler locks the task and the run and reuses an existing
successor, so there is never a second successor. All continuation gate
checks still apply.
- The sweep acts only after the #21108 settling skip lapses, so a
healthy run is not affected. No wait is added before a normal turn
start.
- No schema change. Rows from older releases follow the same rule
because the predicate reads only persisted fields.
- Out of scope: the provider-side causes (Daytona 503 at 14:21Z, resume
calls that time out). #21113 handles the in-run wait for idle transport
timeouts.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, read-only API reads, Vitest on embedded PostgreSQL) in
Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:20:55 -06:00
Nicky LeachandClaude Opus 5.5 afe86e9ded fix(runtime): record a plugin task's stop receipt when its stop is proven (#21237)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A plugin environment runs each Paperclip Runner turn as a provider
task, and a follow-up run can continue the same Claude/ACPX conversation
from the saved checkpoint
> - Core authorizes that continuation only when the predecessor run's
remote execution has stopped (`remoteExecutionHasStopped`), so two
Runners never write one checkpoint
> - The predecessor's own settlement already proves the stop: the
provider reports `executionStopped`, and Core then completes the task
> - On a per-turn (ephemeral) lease, Core records the stop receipt only
when it releases the lease, and that release can be held for a long time
(for example by a pending workspace export)
> - In that window, an ordinary follow-up fails with `adapter_failed:
Plugin Runner retained session is not authorized`, even though the stop
is already proven
> - This pull request records the stop receipt when the stop is proven,
and the stop check accepts it before the release
> - The benefit is that follow-ups continue as soon as the predecessor
is proven stopped, and the release gates keep their meaning

## Linked Issues or Issue Description

Fixes #21211

Refs #21214 and #21223. Those cover the creation fence for same-run
recovery, which still requires a released lease.

Production evidence for this change: in both observed failures, the
predecessor was a per-turn plugin run. Its `task.settle` completed with
all provider calls successful. Core then did not release its lease for
30 minutes or more, and no release call was made in that time. The
follow-up failed in checkpoint recovery (`recoverPluginRunnerCheckpoint`
→ `resolvePluginTaskResume`) within seconds of admission.

## What Changed

- `server/src/services/environment-task-runtime.ts`
- After a recorded provider stop, `complete` writes the bound stop
receipt (`remoteTerminationReceipt(..., { state: "stopped" })`) on the
current lease for every lease policy. Before, it did this only for
`reuse_by_environment` leases. Reusable tasks still confirm every owner
and still move retained owners to released.
- New work on a lease (`submit`, `copilot_metadata`, first portable
resume) removes the stop receipt in the same write that clears
`runnerTaskExecutionStopped`. A replacement physical task does not copy
the old receipt.
- `server/src/services/remote-execution-termination.ts`
- New `hasBoundRemoteStopReceipt` checks only the receipt binding
(company, run, lease, provider and physical task).
`hasRemoteTerminationReceipt` uses it and keeps its released-only
semantics. Resource cleanup gates do not change.
- `remoteExecutionHasStopped` and `stoppedRemoteCleanupScopes` accept a
lease that is released with its receipt, or that is still `active` (not
released) with a bound stop receipt. A lease whose release started and
did not settle does not count. After release starts, the release owns
the receipt.
  - New `remoteExecutionHasReleased` gives the release-based answer.
- `server/src/services/environment-run-orchestrator.ts`: same-run plugin
recovery uses `remoteExecutionHasReleased`. The same-run resume check
after it requires released receipts, so its behavior does not change.
- `resolvePluginTaskResume` keeps its checks and their order. Its
refusal now names the first failing check: `Plugin Runner retained
session is not authorized (reason: <code>)`, for example
`predecessor_not_stopped`, `scope_mismatch` or `identity_mismatch`. The
old message stays as the prefix, and no ids go into it. The
stale-availability recheck in `environment-task-runtime.ts` names its
refusal the same way. Production telemetry for #21211 could not tell
which check fired. The next occurrence will show it.
- `doc/execution-semantics.md` and
`doc/testing-native-remote-continuity.md` describe the stop receipt and
when it is recorded.

Caller audit for the changed meaning of `remoteExecutionHasStopped`:

| Caller | Needs | Decision |
| --- | --- | --- |
| `agent-instruction-working-copies.ts` | stop | Accepts. Directory
cleanup still requires a released "destroyed" receipt. |
| `environment-run-orchestrator.ts` | release | Moved to
`remoteExecutionHasReleased`. |
| `plugin-task-resume.ts` | stop | Accepts. This is the fix. |
| `native-runtime/native-runner-ownership.ts` | stop | Accepts.
`hasPendingRunCleanup` still counts the active lease as pending. |
| `native-runtime/preserved-native-result-finalization.ts` | stop |
Accepts. Workspace recovery has its own gate. |
| `heartbeat/case-handoff-lifecycle.ts` | stop | Accepts.
`hasPendingRunCleanup` still holds active leases. |
| `heartbeat/run-control.ts` (3 sites) | stop | Accepts. Stop
acknowledgement and quarantined ownership retirement. Holds and blockers
still apply on the normal path. |
| `heartbeat/runtime-execution.ts` | stop | Accepts. Release errors
still fail it. |

## Verification

From `server/`, after `node scripts/agent-preflight.mjs`:

- `npx vitest run
src/__tests__/environment-task-authority.integration.test.ts`: 10/10
pass. The 2 new tests fail without the fix.
- The plugin task and environment runtime suites
(`environment-task-authority.integration`,
`plugin-runner-task.integration`, `plugin-warm-tasks.integration`,
`environment-task-runtime`, `plugin-task-source-recovery.integration`,
`environment-runtime`): 6 files, 366 pass.
- The audited caller suites (`agent-instruction-working-copies`, case
handoff legacy and native, `environment-run-orchestrator`,
`heartbeat-case-handoff`, `explicit-native-continuation`, `run-control`,
`native-remote-retry-admission`, `preserved-native-response-wait`): 9
files, 806 pass.
- `pnpm --filter @paperclipai/server typecheck`: pass.

The new integration tests cover:

- `complete` writes a bound stop receipt on a still-active ephemeral
lease. `remoteExecutionHasStopped` is true and
`hasRemoteTerminationReceipt` is still false.
- `resolvePluginTaskResume` (inspect and launch) authorizes a
follow-up's new task, and the real host `read_state` checkpoint read
succeeds, while the predecessor's release is held.
- Refusals: no provider stop, a receipt for another provider task, an
unstopped sibling lease, and a release in progress.
- New work on the lease removes the receipt. Release overwrites the
receipt with its own receipt. Warm owners keep their behavior.

I did not add an executor-level test. `native-session-executor.test.ts`
mocks `resolvePluginTaskResume`, so the integration test runs the real
checkpoint read path instead. I did not run the full suite locally. CI
runs it.

## Risks

- `remoteExecutionHasStopped` now returns true earlier for a run whose
leases are still held. Each caller is audited above. Callers that need
released resources use the release-based predicates.
- The stop receipt can now be on an `active` lease. The SQL readers of
`remoteExecutionTermination` (cost accounting, execution blockers,
idle-work proof, orphan settlement) also require a released status.
`pinFailedObservedWorkspace` treats an active ephemeral lease with a
receipt as not pinnable. That path applies only to the remote-workspace
flow and falls back to `preserveNativeFailedWorkspace`.
- Leases that settled before this change have no early receipt. They
continue to wait for their release.
- Nothing compares the refusal message exactly. Matchers use substrings,
and run failure classification does not use the message.
- A follow-up still fails closed if the predecessor's stop was never
proven. This change does not add a recoverable wait.
- The reason the release was held in production is not addressed here.
- No migration and no API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, sub-agents, Honeycomb trace queries) and extended
thinking. One sub-agent wrote the change. A second session reviewed it
before this pull request.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:20:45 -08:00
DottaandClaude Opus 5.5 497ccba8f1 fix(native): replace a reattached Runner that left its recovery read unanswered (#21272)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs execute in a remote Runner that the server
controls over the durable PRP channel
> - When a run is interrupted, a same-run retry reattaches the live
Runner and reads its session (`session.snapshot`) before it continues
> - On staging, the reattached Runner authenticated and restored its
session, and then did not answer the read
> - The recovery kept that Runner, so each later attempt adopted the
same Runner and failed the same way until the attempt budget ended the
run
> - This pull request makes the next attempt replace that Runner with a
fresh Runner over its saved journal, and makes a failed Codex history
read say what failed
> - The benefit is that an interrupted Codex run continues without a
manual Retry, and its provider turn is not sent again

## Linked Issues or Issue Description

Refs #21161, Refs #21053, Refs #20853

**What happened?**

A Codex agent on Daytona ran recovery runs (wake
`issue_recovery_action_restored`). In one hour, 11 of about 35 finished
runs failed. All failed the same way:

- `native_session_recovery_failed: HarnessCapabilityUnavailableError:
read is unavailable: Error: runnerd exited while waiting for
session.snapshot` (8 runs)
- `native_session_recovery_failed: HarnessReconciliationError:
codex_history_read_failed: thread/turns/list requires supported
paginated history; Error: provider_transport_failed: PRP command
session.snapshot timed out (runner connected; pending ahead 0; last
runner event seq 338 76s ago)` (3 runs)

The run log of each run shows one sequence:

1. Attempt 1 fails with `runnerd did not report the provider turn
identity`. The close leaves the Runner alive and unsettled for the
same-run reattachment.
2. Attempt 2 adopts that Runner (same PID). The Runner authenticates.
The log shows `runnerd restored its durable PRP session and provider
thread`. Then the driver's recovery read (`thread/read` or
`thread/turns/list`, which both send `session.snapshot`) gets no answer.
The Runner still answers WebSocket pings, but it sends no command result
and almost no events (`pending ahead 0`, last event 76 s to 391 s
earlier). The recovery treats this as transient and detaches. The Runner
stays alive.
3. Attempt 3 adopts the same Runner again and fails the same way. In
some runs a protocol fault (`thread_binding_mismatch`) closes the
transport during the read, so the message is `runnerd exited while
waiting for session.snapshot`.
4. The provider attempt budget ends the run.

#21161 did not help. Its wait extension applies only while a queued
answer is being committed. Here the Runner sent no answer at all.

The `paginated history` text was also wrong. `codex-history.ts` called
every failed history read unsupported, also a transport timeout.

**Expected behavior**

A Runner that left a recovery read unanswered is not adopted again. The
next attempt starts a fresh Runner from the saved journal, resumes the
provider session from the saved checkpoint, and does not send the
provider turn again. A failed history read reports the real failure. A
Codex build that refuses paginated history falls back to one complete
thread read.

**Steps to reproduce**

1. Start a Codex run in a remote (Daytona) Runner.
2. Let the first attempt fail at turn start, so that the close keeps the
Runner alive.
3. Let the reattached Runner stop answering commands after it
authenticates (for example, stop the Runner process after its first
answer).
4. See `PRP command session.snapshot timed out (runner connected;
pending ahead 0; ...)` on attempt 2 and the same failure on attempt 3,
with the same Runner PID.

**Paperclip version or commit**

Staging Core `57aeda190b` (contains #21161 and #21120). Fixed on
`master` after `53608bd1d6`.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`, `codex_app_server`).

## What Changed

- `nativeRecoveryCommandUnanswered` (executor) finds a
`native_session_recovery_failed` whose Runner did not answer a recovery
command (`PRP command <type> timed out` or `runnerd exited while waiting
for <type>`). Identity, authority, integrity and incompatibility
refusals do not match.
- When such a failure ends a `reattach_remote_runner` attempt on a
sandbox target, the failure detail records `adoptedRunnerUnanswered:
true`.
- `claimNativeRestartRecoveries` copies it to the next claim as
`replaceUnansweredRunner: true`, in the same way as
`exitedBeforeSessionOpen`.
- `verifyRemoteRunnerOrSettledExit` with `replaceUnansweredRunner` keeps
all identity checks. When the old Runner is still alive, it returns no
adopted process, and the attempt launches a fresh Runner from the saved
journal. The existing launch fence stops the old `paperclip-runnerd` on
that state folder and its whole lineage, and fails the launch closed
when it cannot prove them gone. A Runner that already exited mid-turn
keeps the existing fail-closed path, because its provider or tool
processes can survive where the fence cannot find them. The task then
continues in a fresh session from the saved work. The attempt budget is
unchanged.
- `codex-history.ts`: a history read that fails without a provider
answer is now `codex_history_read_failed: <method> read failed;
<error>`. A provider refusal of the method (`-32601` or `-32600`) on the
first page falls back to one `thread/read` with `includeTurns: true`.
That fallback fails closed when the thread differs, or when turns, items
or turn identities are missing. A refusal on a later page still fails
closed.
- `doc/execution-semantics.md` describes both rules. The launch-fence
code comment now names the replacement case.

### Changed existing test

`codex-history.test.ts` "does not turn an interrupted page or
unsupported method into idle history" used a plain `Error("method not
found")` on page 2 and expected the `supported paginated history` text.
A plain error is now a failed read, not an unsupported method. The test
now uses a real provider refusal (`CodexRpcError` `-32601`) on page 2
and still expects a fail-closed rejection. A new test covers the
failed-read message.

## Verification

- Red before the fix, green after:
- `server/src/services/native-runtime/native-session-executor.test.ts` ›
"a same-run reattachment whose adopted Runner never answered its
recovery read": the replacement and staging-message tests failed on the
old code (the Runner was adopted again, the three staging messages were
not recognized). Two further tests ("never resumes beside an exited
unanswered Runner's lineage") failed on the first revision of this PR,
which relaunched beside an exited mid-turn Runner, and pass now. All 15
pass.
- `native-runner-restart-recovery.integration.test.ts` › "asks the
same-run reattachment to replace a Runner that left its recovery read
unanswered" (embedded Postgres): the claim had no
`replaceUnansweredRunner` on the old code. Now passes.
- `codex-history.test.ts`: 7 new tests failed on the old code. All 16
pass.
- Real `paperclip-runnerd` and fake Codex:
`runnerd-codex-transport-remote-retention.test.ts` › "continues the same
provider thread from a fresh Runner over the journal of a retained
Runner that stopped answering, without replaying its turn". It runs the
staging sequence: turn started, close keeps the stopped Runner, a second
controller adopts it, the Runner stops answering and the controller
detaches, the Runner and provider are killed (the launch fence), and a
fresh Runner over the same journal returns the same thread. The fake
Codex call log shows one `thread/start` and one `turn/start`.
- Focused suites: `packages/paperclip-runner` `src/drivers/codex/` plus
the remote-retention, adoption and cold-restore transport suites (442
tests). Two suites timed out under parallel load and pass alone. Server:
`native-session-executor`, `native-restart-recovery`,
`native-runner-restart-recovery.integration`,
`runner-lost-mid-turn-continuation` (1714 passed, 21 skipped).
- `pnpm typecheck` in `server` (all layers) and `tsc` for
`paperclip-runner` source and tests pass.
- Not run locally: the container-gated remote Runner suites
(`PAPERCLIP_TEST_RUNNER_CONTAINER`) and the full suite. CI covers the
full suite.

## Risks

- Medium. The replacement relaunches a Runner over a journal that is not
settled, but only while the old Runner is alive. Safety comes from three
facts: the launch fence stops the old Runner and its lineage with proof
or fails the launch; the Runner restores its own durable journal (as
local `resume_dead_runner` does); and the real-runnerd test shows that
the turn is not sent again. An exited Runner is never relaunched beside
its possible survivors.
- If the previous close left a `runner.suspend` pending, a fresh Runner
can execute it and exit. The attempt then fails as it does today. The
usual detach path after an unanswered read sends no suspend.
- The replacement uses the existing attempt budget. It does not add
attempts.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code
with tool use (shell, file edits, browser reads of staging run events
and logs), extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:06:24 -06:00
DottaandClaude Opus 5.5 334b2ae066 fix(recovery): continue a task blocked behind its workspace's ended writer once the hold releases (#21221)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Remote task workspaces have one writer run at a time, and an ended
writer keeps the workspace until its files are saved or set aside
> - When a task's next runs fail setup on that hold ("The workspace
still has an unfinished writer or save operation."), stranded-work
recovery blocks the task with a board escalation
> - The retained-file recovery from #21136 later released such a hold on
staging (it continued from the last save), but nothing settled the board
escalation, so the task stayed Blocked until a person pressed Retry
> - Recovery must work without a person: the block belonged to the hold,
and the hold is gone
> - This pull request adds a pass to the active recovery-action sweep
that settles that escalation once the workspace's writer hold is free
> - The benefit is that such tasks continue by themselves with one plain
notice and one run, with no Retry

## Linked Issues or Issue Description

Refs #21136
Refs #21143

**What happened?**
On staging, an ended run held its task's own remote workspace after its
superseded workspace export failed. The task's queued runs failed setup
with the unfinished-writer error, and stranded-work recovery blocked the
task with a board-owned escalation. After #21136 was deployed, its
retained-file recovery recorded the hold 77 seconds after server start
and, 19 minutes later, continued the task from its last save and
released the workspace. The task still stayed Blocked behind the board
escalation, and no run was queued. A board user had to press Retry.

**What did you expect to happen?**
Once the hold that caused the setup failures is released, the task
returns to `todo` and its assignee continues it, with no Retry.

**Steps to reproduce**
1. Give a task a remote workspace whose writer is an ended run with
unsaved files.
2. Let the task's next run fail setup with the unfinished-writer error,
and let stranded-work recovery block the task.
3. Release the hold (save the files, or continue from the last save).
4. Run the active recovery-action sweep. Before this change the task
stays Blocked with an active board escalation.

**Why the merged fix did not unblock the task**
The retained-file fallback only releases a Blocked status that its own
hold set (`holdOwnsBlockedStatus`). Here stranded-work recovery set the
block with a different, board-owned escalation, and a late status
decision 1.4 seconds later kept that block. So the fallback correctly
did not take the status, and no code settled the board escalation after
the hold was gone.

## What Changed

- Add `continueTasksAfterWorkspaceWriterReleased`
(`server/src/services/recovery/released-writer-escalation.ts`). It
selects board-owned `stranded_assigned_issue` escalations whose latest
run failed setup with the unfinished-writer error before any provider
work, by the current assignee. In one transaction it checks that the
block belongs to the escalation, that no newer run exists for the task,
that dependencies allow work, and that the workspace's writer hold is
now free. Then it resolves the escalation with a named outcome, returns
the task to `todo`, records one plain activity notice, and records a
pending wake.
- A later status decision that only kept the Blocked status it found
(from `blocked` to `blocked`) does not own the block. A decision that
chose the block still keeps the task blocked.
- Run the pass in `reconcileActiveRecoveryActions`, next to the legacy
rehome pass, and include its next wake retry time in the pass deadline.
- Generalize the legacy rehome wake delivery to
`deliverPendingRecoveryWakes`, keyed by the evidence key, so both passes
share one crash-safe, backed-off wake delivery. The rehome wake keeps
its fresh session. The new wake resumes normally.
- Keep the company selected by the scheduled sweep while a
released-writer wake is owed (`recoveryActionSweepCondition`).
- Document the rule in `doc/execution-semantics.md`.

## Verification

- `cd server && npx vitest run
src/__tests__/released-writer-escalation.test.ts
src/__tests__/legacy-workspace-rehome.test.ts
src/__tests__/issue-blocker-stall-wakeups.test.ts
src/__tests__/issue-blocker-stall-waiter-statement.test.ts
--maxWorkers=2`: all pass.
- Red/green: with the sweep hook removed, "continues the task once the
hold is released" fails (the task stays `blocked`). With the hook, it
passes.
- The new test seeds the staging row shape: a native writer run ended
with `native_finalization_superseded` still writing the task's own dirty
workspace, three legacy setup failures with the unfinished-writer error
and `{ kind: "bootstrap", providerWorkStarted: false }`, a board
escalation with `latestRunErrorCode: "setup_failed"`, the block 4.4
seconds after the last failure, and a later `blocked` to `blocked`
status decision.
- The tests also cover: no change while the writer still holds the
workspace, one wake only, a pending wake after a crash, a refused wake
that backs off, a later decision that chose the block, a dependency that
forbids work, a newer run, and another setup error.
- `cd server && npx tsc --noEmit -p tsconfig.json`: passes.

## Risks

- Low risk. The pass only acts on board escalations whose evidence is
exactly the unfinished-writer setup failure, and only when workspace
reservation would now admit a new writer. It never changes the
workspace, never applies a result, and never releases a writer fence.
- The task gets one normal wake through the issue-state guard (`todo`,
same assignee), so it cannot start twice.
- The pass adds one bounded query per sweep of a company with active
recovery actions.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, read-only browser API reads of staging) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 17:58:06 -06:00
DottaandClaude Opus 5.5 ef6d347f79 fix(runner): keep reattached runners answering after a restart (#21161)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs execute in a remote Runner that the server controls over
the durable PRP channel
> - After a server restart, the next server reattaches to the live
Runner and sends recovery barriers (`runner.drain`, `session.snapshot`)
> - These barriers timed out while the Runner stayed connected, and each
same-run retry failed the same way
> - After three attempts the run stopped with
`native_session_retry_exhausted` and needed a manual Retry
> - This pull request finds the two causes and fixes both in the
controller and the transport
> - The benefit is that restart recovery finishes on its own, without a
manual Retry

## Linked Issues or Issue Description

**What happened?**

After a server restart, a run reattached to its live remote Runner. The
Runner authenticated. Then a recovery command got no recorded answer for
30 s or more, although the Runner stayed connected. Errors on staging:

- `PRP command session.snapshot timed out (runner connected; pending
ahead 0; last runner event seq 869 55s ago)`
- `PRP command runner.drain timed out (runner connected; pending ahead 1
[semantic_tool.result] oldest 170s ...)`
- `runnerd exited while waiting for runner.drain`

Each reattach failed the same way. After three attempts the run ended
with "Workspace repair required" (`native_session_retry_exhausted`).

Root causes, proven with the real Runner and the fake Codex harness:

1. The Runner answered in about 40 ms. At reconnect it also sends the
event backlog that it collected while the server was down. The
controller handles inbound frames of a connection in order, and it
commits each event (accounting write and checkpoint flush) before the
next frame. So the answer waited behind the whole backlog. With 400
events at 100 ms each, the answer was handled after 44 s, and the 30 s
barrier had already failed. The `semantic_tool.result` case is the same:
its answer waited behind the backlog, and the next command is sent only
after that answer is handled. The Runner code was not the cause.
2. When a Runner did not answer during close, the close path queued
`runner.drain` and `runner.suspend`, kept the Runner alive for
reattachment, and saved them in the journal. The next attempt delivered
that old `runner.suspend` to the reattached Runner, and the Runner
suspended and exited.

**Expected behavior**

A reattach to a live Runner completes recovery. A Runner answer that is
queued behind earlier frames is not a timeout. A reattach never delivers
a terminal command from an earlier attempt.

**Steps to reproduce**

1. Start a Codex run in a remote Runner with a turn that streams output.
2. Restart the server. Let the Runner collect a few hundred events while
it is disconnected.
3. Let the next server reattach while event commits are slow, as on a
loaded database.
4. See `session.snapshot timed out (runner connected; pending ahead 0)`.
5. Alternatively, let a close path leave `runner.drain` and
`runner.suspend` pending, then reattach. See `runnerd exited while
waiting for runner.drain`.

**Paperclip version or commit**

`master` at 3fb35fcf13.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`).

## What Changed

- `DurablePrpControlPlane` records each Runner `command_result` frame
when it arrives, before the ordered frame chain handles it.
`queuedCommandResultProgressAt(commandId)` returns the latest time that
the chain made progress while that answer is queued.
- `waitForRunnerCommand` keeps waiting past its deadline while the
answer to the command, or to a pending command ahead of it, is queued
and the chain advanced in the last 30 s. When the chain stops for 30 s,
the barrier fails as before. A wait that starts after its deadline still
rejects a late ACK. Frame order does not change.
- When a barrier completes after such an extension, it credits the time
past the deadline (plus the 5 s post-stop grace) to the admission clock
(`RunnerStallLedger.credit`). Later barriers that share a cold admission
deadline (Pi, OpenCode, ACPX) keep their time.
`#waitForProviderIdentityWork` checks the ready state once before it
rejects an expired deadline.
- `DurablePrpControlPlane.withdrawUndeliveredTerminalCommands()` runs
only when the transport adopts a live Runner. The controller sends only
the first pending command, so no command after it reached the Runner,
and a live, serving Runner cannot have executed a suspend. The method
removes every pending `runner.suspend` and `runner.shutdown` from that
pending suffix. It keeps every other pending command, for example a
`semantic_tool.result` that a still-running tool callback queued after
the close, and renumbers the kept commands so the Runner's command
sequence stays contiguous. Pending drains stay, because they have no
effect.
- A generated command id now skips an id that a renumbered command
already holds.
- `doc/execution-semantics.md` describes both rules.

### Changed existing test

`runnerd-codex-transport-remote-retention.test.ts` asserted that the
reattaching controller delivers the failed close's `runner.suspend`, so
the adopted Runner suspends and exits, and a later attempt relaunches it
(#20923). On staging that path failed the attempt with `runnerd exited
while waiting for runner.drain` (run 3e28c2b0) and did not recover. This
PR requires that a reattach does not inherit that suspend. The test now
asserts that the adopted Runner stays alive, answers `thread/read`,
keeps a `ready` journal, gets no suspend, and no duplicate Runner is
launched. The retention part of the test (the close keeps the frozen
Runner alive) is unchanged.

## Verification

- Red before the fix, green after (real `paperclip-runnerd` and fake
Codex):
- `... past an event backlog ahead of its recovery snapshot result`
failed with the staging error `session.snapshot timed out (runner
connected; pending ahead 0; ...)`. Debug timing showed the Runner
answered in 40 ms and the controller handled the answer 44 s later.
- `... past a runner.suspend left by an earlier attempt's close path`
failed with the staging error `runnerd exited while waiting for
runner.drain`.
- `... past a runner.suspend with a tool result queued after it` failed
the same way with tail-only withdrawal (the Greptile case), and passes
now. The tool result is delivered once and all sequence numbers stay
contiguous.
- Unit tests for `waitForRunnerCommand`: it waits while a queued answer
advances, it times out when progress stops, and a later barrier that
shares the admission deadline still completes after an extended one.
- `... past a tool result its predecessor queued after the restart`
guards delivery of a `semantic_tool.result` queued between controller
generations.
- The existing `shares Pi's absolute cold deadline across recovery
barriers and rejects a late ACK` passes unchanged.
- `cd packages/paperclip-runner && pnpm exec vitest run
src/live/runnerd-codex-transport.test.ts
src/live/runnerd-codex-transport-remote-retention.test.ts
src/live/runnerd-codex-transport-reattach-carried-commands.test.ts`
passes (320 tests).
- Related suites pass: adoption, carried suspend, stale controller
journal, stall, deadlines,
`src/backends/harness-driver-backend.test.ts`, and all
`src/control-plane` tests (330 tests).
- `tsc --noEmit -p tsconfig.test.json` passes in
`packages/paperclip-runner`.
- `pnpm verify` asked for broader coverage (exit 2). Required CI
supplies it.

## Risks

- A barrier can now wait longer than its deadline when the Runner has
answered and the controller still commits a large backlog. The wait ends
30 s after progress stops. This trades a slower recovery for no failed
run.
- The withdrawal applies only to a live adopted Runner. If such a Runner
had somehow journaled a withdrawn suspend, the next command gets a
`controller_sequence_mismatch` rejection, and the Runner executes
nothing.
- Renumbering changes the `controllerSeq` of pending commands that never
reached the Runner. Their command ids do not change.
- Not in this PR: a fallback that stops a connected Runner that never
answers and resumes it from its journal. With these fixes, the observed
staging failures no longer reach that state.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool
use (shell, file editing, browser for read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 12:50:43 -06:00
DottaandClaude Opus 5.5 90ae3c4337 fix(native): process a superseded export's late stop receipt and stop its sandbox when none arrives (#21143)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run can be retired as superseded while its workspace export
fails. Its files stay unsaved, and the run stays the writer of the
task's workspace.
> - #21136 records those files for the automatic workspace recovery.
That record needs the sandbox's stop-and-retain receipt.
> - The native run projection worker checked for that receipt for one
hour after the run ended. After that, it dropped the run. Nothing woke
it when the receipt arrived later.
> - If the receipt came late or never, the run kept the task's workspace
fenced. Every queued run on the task waited and then failed setup.
Nobody owned the problem.
> - This pull request makes the worker process a late receipt, stop and
keep the sandbox itself when no receipt arrives, and never drop the run
silently.
> - The benefit is that the task's queue moves again by itself, and two
writers never use one workspace.

## Linked Issues or Issue Description

Refs #21136 (review thread on
`native-superseded-workspace-export.ts:84`)

**What happened?**
`preserveSupersededWorkspaceExport` returned `recheckAt: null` when the
sandbox stop receipt had not arrived within `finishedAt + 1h`. The
projection worker then deleted its entry. A later stop receipt
(`settleStopOnlyCleanup`) woke only the workspace recovery queue, not
the projection queue. `preserveNativeFailedWorkspace` requires
`hasConfirmedSandboxStopAndRetain`, so no hold was recorded. The run
stayed the dirty writer, and queued runs kept failing setup.

**Expected behavior**
A late receipt is processed at once. When no receipt arrives, Paperclip
stops and keeps the sandbox itself and waits for the provider's
confirmation. The workspace stays held while the sandbox may still run.
When the provider confirms that the sandbox is gone, the task continues
from its last save. When the stop stays unconfirmed for hours, the task
shows one plain notice, and the check continues.

**Steps to reproduce**
1. Retire a native run as superseded after its workspace export fails
(the #21136 shape).
2. Let the run's sandbox stop not settle (for example, the server that
owned the release is replaced).
3. Wait more than one hour after the run ended.
4. See that the projection worker drops the run, and that a later stop
receipt records nothing. The workspace stays held.

## What Changed

- `sandbox-stop-and-retain.ts`: a confirmed stop-only receipt now also
notifies `DELIVERY_QUEUES.nativeRunProjection` (scoped to the company
and run). New `prepareUnconfirmedSourceSandboxStop` puts a
stop-and-retain intent on the source lease from its current status. It
skips a destroyed sandbox, a stop already in progress, and a cleanup in
flight.
- `native-superseded-workspace-export.ts`:
- Within one hour of the run ending, the worker checks every minute
(unchanged).
- After that hour, it requests the stop-and-retain itself. The existing
pending-cleanup sweep dispatches the stop and retries it until the
provider confirms. The worker checks again every 15 minutes, and the
receipt wakes it at once. The workspace stays held.
  - A sandbox that another lease holds is never stopped.
- When the provider confirms that the sandbox was destroyed, the files
cannot be saved. In one transaction (task, run, then workspace locks),
the worker sets aside only that run's workspace epoch, wakes queued-run
dispatch, and records `issue.previous_run_files_lost` on the task.
- When the stop is still unconfirmed six hours after the run ended (or
the sandbox is held by another lease), the task gets one
`issue.previous_run_files_waiting` activity entry. The worker keeps
checking. It never drops the run.
- A stop-only dispatch now settles when the provider confirms that the
exact sandbox is already gone. Before, the runtime accepted only a
`stopped` reply and threw on `destroyed`, so a gone sandbox kept the
stop pending forever.
- `sandbox-stop-and-retain.ts`: new `acceptedStopOnlyReceipt` accepts
`stopped`, or `destroyed` for a stop-and-retain intent (a native
workspace export resume still needs `stopped`). `settleStopOnlyCleanup`
stores a `destroyed` receipt as `released` / `success` /
`workspace_source_unavailable`, removes the stop intent, never writes a
retain receipt, and wakes the projection and workspace recovery queues.
- `environment-runtime.ts`: the plugin `environmentStopLease` and
built-in `stopLease` checks use `acceptedStopOnlyReceipt`.
- `workspace-recovery.ts`: when its own stop settles as `destroyed`, the
source is unavailable (the existing alternative-strategy path) instead
of a binding failure.
- Daytona plugin: stop of a sandbox that Daytona reports as not found
returns `destroyed` (destroy already did this) instead of throwing. boxd
already returns `destroyed`. Other sandbox plugins have no stop-only
method, so nothing changes for them.
- New shared `SANDBOX_ALLOCATION_HOLDER_STATUSES` (`active`, `retained`,
`pending_cleanup`). The stop-only cleanup ownership check and the
worker's "held by another lease" guard both use it, so a retained warm
sandbox is never stopped and the two checks cannot drift.
- `ui/src/lib/activity-format.ts`: plain labels for the two task
activity entries. They do not ask a person to fix anything.
- `doc/execution-semantics.md`: document the rule.
- Not changed: the shared owner prover still refuses to treat a stopped
instance on another host as proof that its child processes stopped.

## Verification

- `server/src/services/superseded-workspace-export-recovery.test.ts` (4
new tests, red/green):
- "checks again every minute while the run's own release may still stop
its sandbox": the recheck branch. Nothing is recorded or stopped, and
the workspace keeps its writer.
- "stops and keeps the sandbox itself when no stop receipt arrived, and
records the files once the late receipt lands": after the hour, the
lease gets a stop-only intent. A second pass changes nothing. The late
receipt wakes the projection queue. The next check records the hold, and
the system recovery frees the workspace. Fails on master (`expected {
…(19) } to match object { status: 'pending_cleanup' }`).
- "names the unconfirmed stop on the task once, and keeps checking, when
the provider never confirms it": the expiry branch. One notice, the
check continues, no hold, and the workspace stays held. Fails on master.
- "continues from the last save when the provider confirms the sandbox
is gone": epoch 1 is set aside, the workspace is free, the task says so,
and the result is not applied. Fails on master.
- Review-round tests (all 5 fail on the first head `a208bfa58f`, pass
now):
- `environment-runtime.test.ts` "settles a plugin stop whose provider
reports the sandbox gone (boxd, Daytona not found) through the cleanup
sweep": provider dispatch through `releaseRunLeases` and
`sweepPendingCleanupLeases`. The lease settles `released` with a
`destroyed` receipt and no retain receipt.
- `environment-runtime.test.ts` "settles a built-in stop whose provider
reports the sandbox gone, without claiming retained files": the built-in
`stopLease` path. No destroy call.
- `superseded-workspace-export-recovery.test.ts` "continues from the
last save when Paperclip's own stop finds the sandbox already gone": the
worker's stop request gets a `destroyed` reply, the projection queue
wakes, and the workspace is freed at epoch 2.
- `superseded-workspace-export-recovery.test.ts` "never stops a sandbox
a retained warm lease still holds, says so once and keeps checking": the
source lease is unchanged, a recheck is returned, and `sandbox_in_use`
is recorded.
- Daytona `plugin.test.ts` "explicit stop confirms a sandbox Daytona no
longer has as destroyed".
- `environment-runtime.test.ts` (207) and Daytona `plugin.test.ts` (421)
pass.
- Related suites pass: `native-run-projection-work`,
`retained-native-session-cleanup-work`, `workspace-recovery-stop-proof`,
`native-failed-workspace-recovery`, `legacy-workspace-restore-recovery`,
`environment-retained-orphan-sandbox`,
`workspace-recovery-retry-copyback`, `workspace-recovery-fallback` (398
tests), and `ui/src/lib/activity-format.test.ts`.
- `tsc --noEmit` for `server`.

## Risks

- Low to medium risk. The new behavior acts only on an ended superseded
run that failed in its workspace export, still holds a dirty workspace,
and has no retained-file hold.
- The worker now stops an ended run's sandbox one hour after the run
ended if no stop receipt arrived. The stop uses the existing stop-only
contract and keeps the sandbox. A sandbox held by another lease is never
stopped.
- A destroyed sandbox's unsaved files are not recoverable. The task
continues from its last save and says so.
- If a provider never confirms a stop, the workspace stays held (by
design: no second writer). The task shows one notice, and Paperclip
keeps retrying.
- Shared cleanup code now treats a confirmed `destroyed` stop-only reply
as settled. Every caller was checked: the cleanup sweep and run release
count it as settled; the workspace recovery treats the source as
unavailable; the fallback treats it as stopped (nothing can run there).
No caller reads it as retained files, because no retain receipt is
written. A native workspace export resume still requires `stopped`.
- Each extra delivery notification per settled stop-only cleanup is
cheap. The projection worker's scoped load finds nothing for runs that
do not match.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 11:54:47 -06:00
DottaandClaude Opus 5.5 dfc510b887 fix(server): hand new run starts to the replacement server during a deploy overlap (#21127)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Cloud replaces the server process on each release. The old
server keeps serving until the new one passes health checks and takes
the traffic.
> - During that overlap the old server keeps starting new runs right up
to SIGTERM, although the new server can already start them.
> - SIGTERM then cuts off exactly those runs while they still prepare.
In one audited swap the old server started 30 runs after the new server
booted, and 23 of them ended `process_lost`.
> - Admission must never pause, so the fix must hand over who starts new
runs, not stop admission.
> - This pull request makes the new server record when it starts runs.
The old server then leaves new run starts to it, and starts them again
if the new server stops or dies.
> - The benefit is that a deploy no longer starts runs on a server that
is about to stop, and the company always has one server that starts
runs.

## Linked Issues or Issue Description

Refs #20852 (preserves preparing and launching turns that a deploy
interrupts; this PR reduces how many such turns the old server starts).
Related: #21046, #21107 (shared controller-instance owner proof), #20986
(queued-run dispatch signal).

**What existing behavior does this improve?**

The start of new runs while two servers overlap during a Cloud deploy.

**Subsystem affected**

Server run admission (`server/src/services/heartbeat/queue.ts`), the
controller instance table, and startup and shutdown in
`server/src/index.ts`.

**Current behavior**

The old server starts queued runs until it receives SIGTERM. Runs that
it starts in the last minutes of the overlap are interrupted while they
prepare. Nothing tells the old server that its replacement already
starts runs: `controller_instances.epoch` is written at process start,
and that happens minutes before the new server can start runs.

**Proposed behavior**

The new server records `dispatching_at` and a short
`dispatch_lease_expires_at` on its `controller_instances` row right
after its startup queued-run pass. The old server reads this through an
in-memory flag at run admission and leaves new run starts to the new
server. Runs it already started continue and are handed over at SIGTERM
as today. If the new server stops, drains or dies, the old server starts
runs again within seconds.

**Reason and benefit**

Fewer runs are cut off by each swap. There is never a moment without a
server that starts runs.

**Breaking changes**

None. The migration adds two nullable columns. Releases without this
code ignore the columns, and their rows never carry the marker, so a
mixed-version overlap behaves as today.

## What Changed

- `controller_instances`: new nullable columns `dispatching_at` and
`dispatch_lease_expires_at` (migration
`0471_controller_instances_dispatch_marker`, two `ADD COLUMN`
statements, no lock beyond the catalog update).
- New `server/src/services/run-start-handover.ts`. It is one worker on
the shared delivery coordinator (`DELIVERY_QUEUES.runStartHandover`).
Its renewal and recheck are coordinator deadlines, and a drain change or
expiry wakes it.
- Replacement side: `begin()` records the intent and writes the marker
(it creates the boot's row if the row is missing). If the server drains
at that moment, the worker writes the marker when the drain ends. The
worker then publishes a `run-start-handover` hint and renews the 20 s
lease every 5 s. It renews only while an older boot that is not stopped
or fenced may still serve, and for at most 30 min. All marker writes run
in one serialized chain. `release()` at shutdown lands after any renewal
already in flight. A clear that fails is retried, and shutdown waits for
the release. While the server cannot dispatch (drain, quiesce, admission
closed), the worker clears the lease.
- Old-server side: `isYielding()` is an in-memory read. The hint
refreshes it, and so does one read that a dispatch pass waits for (at
most 1 s) when the last read is more than 2 s old. A read that gets no
answer within 1 s never keeps the server yielding. The server admits,
and the yield also ends when the replacement's last seen lease runs out
on the old server's own clock. Reads are a single flight on the
dedicated connection. While it yields, the server rechecks every 3 s,
calls `onResume` when the marker is gone, and sends a coalesced
queued-run-dispatch hint for skipped starts.
- It yields only to a newer epoch with a live marker. It never yields to
an older boot, to itself, or to a row without the marker, and it does
not yield without a row of its own.
- **Database invariant:** every statement of the hand-over runs in a
small transaction on the dedicated one-connection controller-lease pool
(`controllerLeaseDb`) with a 1 s statement timeout. That covers reads,
marker writes, the `run-start-handover` notification (commit-bound, in
the marker write's own transaction) and the skipped-start hint. None of
it touches the shared pool or registers tracked database work. The pool
is shared only with legacy controller-lease renewals, and never with the
liveness pool that native lease renewals use. A hand-over statement
waits at most one legacy batch (10 s), inside the 15 s between a marker
renewal and the marker's expiry. It delays a legacy renewal (60 s lease,
renewed every 10 s) by at most 1 s. I did not piggyback on the
controller-instance lease renewal, because that renewal runs only while
the server holds work and only every 30 s.
- **Import audit of `run-start-handover.ts`:**
  - `drizzle-orm` `sql`: builds statements, no I/O.
- `controllerLeaseDb`: the dedicated connection, used by every statement
here.
- `notifyDatabaseWorkOnCommit` (new in `work-signals.ts`): runs
`pg_notify` only on the transaction it is given and registers no intent,
so the coordinator's `reconcileDatabaseWork` has nothing of ours to
check.
  - `logger`: no I/O to the database.
  - `controller-process-identity`: in-memory boot id and host hash.
- `delivery-work-coordinator`: type only. The worker is registered for
lifecycle and deadlines; its `run` awaits only the serialized marker
chain, and reads are not awaited.
  - `DELIVERY_QUEUES`: constants.
  - `task-admission` drain subscriptions: in-memory events.
- `heartbeat.ts`: run admission is also closed while the server yields.
A `runStartHandover` service option (default: the process's registered
hand-over) lets one process host two assembled services in tests.
`queue.ts`: `startNextQueuedRunForAgent` and
`resumeQueuedRunsAfterDrain` first call `refreshRunAdmission()`. That
call returns `null` (nothing to await) unless the hand-over read is
stale.
- `index.ts`: creates the hand-over only when the instance lease is on,
the deployment notification channel exists, and the scheduler is
enabled. It calls `begin()` after the RunDispatch startup pass and wakes
that queue. It releases the hand-over when run admission closes at
shutdown, and stops it before the lease stops.
- `deployment-notifications.ts`: keeps the LISTEN connection open while
a hand-over involves this server, so skipped starts and lease changes
reach the other server at once.
- `doc/execution-semantics.md`: new section "Run starts during a deploy
overlap". It covers the marker, who yields, failure handling, what
counts as starting a new run, and what a post-deploy audit should show.

**Marker contract for other components (for example the deployer):** a
replacement starts runs when its `controller_instances` row has an
`epoch` greater than the old boot's, and `dispatching_at is not null and
dispatch_lease_expires_at > now() and stopped_at is null and fenced_at
is null`. `dispatching_at` stays set after the lease lapses, as the time
the boot began starting runs.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-run-start-handover.test.ts`: assembled-app test
with two `heartbeatService`s, each with its own pool, listener,
coordinator and boot. A wake that the old server accepts stays queued
there. The replacement's real queued-run dispatch starts it, and claims
are observed per pool. After a replacement release, the old server
starts the run itself. Red: with the yield removed from
`isRunAdmissionClosed`, the old server claims the run.
- `cd server && npx vitest run
src/__tests__/run-start-handover.test.ts`: 19 tests pass against
embedded PostgreSQL. Each test uses two servers, each with its own pool,
boot id, LISTEN connection and delivery coordinator. The tests cover:
- the old server is ready and the new server starts; the new server sets
the marker; the old server yields; the queued run is claimed only by the
new server, and a skipped start wakes the new server's dispatch
- the old server learns from its own dispatch path without a
notification: a read older than 2 s is refreshed before the start
- the new server dies after it sets the marker: the old server resumes
and runs its queued-run pass
  - the new server shuts down: the old server resumes at once
- the new server drains: it releases the marker and takes it back
afterwards
- a replacement that starts up during a drain takes over when the drain
ends
  - a shutdown release during an in-flight renewal stays released
- a clear that fails at shutdown (statement timeout behind a row lock)
is retried by a later release
- an older server that outlives the lease TTL keeps yielding while the
replacement renews
- a yield that the old server cannot confirm within 1 s (its only
connection is busy) does not stop starts (red/green)
- a yield ends when the replacement's lease runs out, even while every
read waits for a connection (red/green)
- renewals blocked past the TTL hand starts back once, then the
replacement takes them again (bounded flapping)
- a clear that fails when the replacement starts draining is retried
well inside the lease (red/green)
- with both servers' shared pools blocked past the TTL, the marker is
written, renewed and released, and the old server yields and resumes,
all before the block ends (red when hand-over statements use the shared
pool)
  - an older-release row without the marker changes nothing
- a server never yields to an older boot or to itself, and does not
yield without a row of its own
  - a lone server writes once, never renews, and admits from memory
  - without a hand-over (self-hosted) there is nothing to await
- Red: on `origin/master` the test file cannot run, because the module
and the columns do not exist. The old server has no signal and keeps
starting the handed-over run.
- `cd server && npx vitest run
src/__tests__/deployment-notifications.test.ts
src/services/heartbeat/dispatch-work.test.ts
src/__tests__/delivery-work-coordinator.test.ts
src/__tests__/controller-instance-lease.test.ts src/shutdown.test.ts`:
all pass.
- `cd packages/db && pnpm check:migrations`: safety and compatibility
checks pass. `cd server && npx tsc --noEmit -p .`: clean.
- `pnpm verify` asks for broad verification because
`doc/execution-semantics.md` is a global CI input. Required CI supplies
that coverage.

## Risks

- Run starts move from the old server to the new server during the
overlap. If the new server cannot dispatch, the old server resumes on
the lease clear (about 3 s) or the lease lapse (up to about 23 s after a
crash). If a hand-over read fails, the server keeps admitting.
- If the old server was not listening for notifications, it can start
runs for up to about 2 s after the marker (the age of its last read). On
a dispatching server, the pre-dispatch read adds one small bounded query
at most every 2 s.
- An older boot that has been idle for more than 30 min counts as gone.
The new server then does not renew, and the old server admits again as
today.
- The migration adds nullable columns only. The deployer can apply it
while the previous release serves.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 11:09:53 -06:00
DottaandClaude Opus 5.5 f2e449f8f6 fix(retry): continue an accepted Retry past chat input held because it predates the stop (#21129)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person's Retry on a connected task (for example a GitHub pull
request review) must start work. #21080 made a Retry whose chat replay
is refused after the click continue from the recovered saved work
> - After #21080 was deployed, accepted Retries on a busy GitHub review
agent still did not start
> - Those tasks had a saved chat request that arrived while the failed
run was still running. Admission holds such a request as
`message_predates_stop`, so it can never start alone
> - #21080 refused to continue while any saved chat input existed, so
this held input blocked the continuation. For a second group, the latest
run was an admission that the recovery hold cancelled before dispatch,
and the recovered saved work belongs to the earlier failed run
> - This pull request lets the continuation adopt input that was held
because it predates the stop, and lets the recovered run that held a
cancelled admission count as the saved work
> - The benefit is that an accepted Retry starts work on these tasks too

## Linked Issues or Issue Description

Follow-up to #21080. No public issue exists. This is the bug
description, in the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

After #21080 was deployed, `POST /api/issues/:id/retry` returned 202 for
blocked tasks of a GitHub review agent (`paperclip_runner`), but no run
started. Each Retry receipt changed to `failed` about 5 seconds later
(on the 14:48 Retries, 5 seconds for one task and about 75 seconds for
two others), with no run and no `issue.retry_continued_from_saved_work`
activity. Every affected task had a saved chat request (a newer GitHub
review request) with an `issue.execution_waiting` record of reason
`message_predates_stop`. The request arrived while the failed run was
running, so admission holds it permanently. The interruption projection
kept pointing at that held request. In a second group of tasks, the
latest run was cancelled before dispatch with
`execution_reconciliation_required`, and the recovered saved work
belonged to the earlier failed run that held it.

Root cause: in `continueRefusedTaskChatRetry`
(`server/src/services/task-retry.ts`), any saved chat input from
`selectTaskRetryInput` stopped the continuation, so the receipt settled
`failed`. The held input can never start by itself, so nothing ran. The
recovered-work check also looked only at the latest run, so it missed
the recovered run that held a cancelled admission.

**Expected behavior**

An accepted Retry starts work. Saved input that admission holds because
it predates the stop joins the continuation, and the task continues from
the recovered saved work.

**Steps to reproduce**

1. Bind a task to a GitHub chat conversation. Let a native run fail with
`native_continuation_requires_reconciliation`, and recover its saved
work.
2. Save a chat request while that run is still running, so admission
holds it with `message_predates_stop`.
3. Click Retry while the chat identity lock is busy, so the source check
is deferred and then refused.
4. The route returns 202. The Retry receipt changes to `failed`, and no
run starts.

**Paperclip version or commit**

a65644d1e1 (the deployed #21080). Reproduced on ef2b53b675.

**Deployment mode**

Hosted (cloud) instance with Daytona remote sandboxes and a GitHub chat
connection.

**Agent adapter(s) involved**

`paperclip_runner` (native runs, and legacy admissions cancelled before
dispatch). The fix is in core Retry code and is not adapter-specific.

## What Changed

- `server/src/services/task-retry.ts` (`continueRefusedTaskChatRetry`):
- Saved chat input whose source comment was written no later than the
failed run's stop boundary (`userStopBoundaryAt`: the recorded user Stop
request, otherwise `finishedAt`) no longer stops the continuation. This
is the same boundary and timestamp that admission uses. Admission holds
that input as `message_predates_stop`. The continuation coalesces its
receipts into the Retry receipt (`coalescedIntoWakeupRequestId`), so the
task stops waiting on it. The receipt ids are recorded in the activity
details.
- Saved input created after the stop, and unadmitted chat input, still
settle the Retry as failed, as before.
- When the latest run is an admission cancelled before dispatch
(`execution_reconciliation_required`, never started), the recovered-work
check also accepts the run named by that run's
`executionWait.recoveryActionId`.
- `doc/execution-semantics.md`: both rules.
- Tests in `server/src/__tests__/chat-channels.integration.test.ts`: two
new variants of the #21080 regression test.
`person_with_input_before_stop` saves a chat request before the failed
run stops. `person_after_held_admission` adds a legacy admission that
the recovery hold cancelled. Both start one run that resumes the
recovered workspace (`resume_preserved_workspace`). The held-admission
fixture binds the task to the recovered workspace, as on the hosted
tasks. Refusal edges `input_after_stop` and `input_inside_user_stop`
settle the Retry failed with no run and leave the input deferred.

## Verification

- Red before the fix: both new variants fail. The Retry receipt is
`failed` (or stays held), and no run starts.
- Green after the fix: both variants pass, and the held input receipt is
coalesced into the Retry receipt.
- `npx vitest run src/__tests__/chat-channels.integration.test.ts -t
"[Rr]etry"` with `src/services/task-retry.test.ts` passes 369/369 on the
final commit.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- Staging was read with GET requests only.

## Risks

- Behavior change: input that arrived during a failed run no longer
starts its own turn through the chat retry. It is adopted into the
saved-work continuation, and the comment stays in the task thread.
Before this change, that input could never start at all.
- Input that arrived after the stop keeps its own admission path.
- No migration. Receipts that already failed stay failed. The next click
takes the new path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:34:36 -06:00
DottaandClaude Opus 5.5 df89950eb6 fix(native): save or set aside a superseded run's unsaved workspace so its task queue moves (#21136)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run saves its accepted result. Then a finalizer exports the
run's workspace files and applies the result to the task.
> - When a newer run already decided the task, the finalizer retires the
older result as superseded. Since #21106, this also ends the run.
> - If that retirement interrupts a failed workspace export, the run's
files stay unsaved. The run also stays the writer of the task's
workspace. Nothing records those files for recovery, so nothing ever
releases the workspace.
> - Every run queued on the task then waits behind a save that never
settles. After one hour, each queued run fails setup instead.
> - This pull request records those files for the existing automatic
workspace recovery. Recovery saves them from the retained sandbox, or,
when every bounded attempt fails, continues the task from its last save
with the files attached.
> - The benefit is that the task's queue moves again by itself, and the
run's unsaved work is saved or kept findable. The superseded result is
never applied.

## Linked Issues or Issue Description

Refs #21106

**What happened?**
On a hosted staging server, a native run accepted its result at
11:31:08Z. Its workspace export failed once with
`remote_workspace_archive_fs_eacces`. The finalizer then retired it as
`native_finalization_superseded`, because another run had already
decided the task. #21106 ended that run at 14:59:17Z (`failed`). The
task still could not move. The next queued run was cancelled with
`workspace_busy` ("ended, but its workspace save has not settled"). Its
retry re-armed every minute. Four more runs stayed queued. The task
showed no recovery action.

**Expected behavior**
The ended run's unsaved files are saved from its retained sandbox, and
the workspace is released. If they cannot be saved within the bounded
retries, the task continues from its last committed save, and the files
are attached to the task. The queued runs proceed in both cases. The
superseded result is never applied.

**Steps to reproduce**
1. Accept a native result for a run that writes a remote workspace.
2. Let the workspace export fail once, while a newer run has already
committed the task's decision.
3. See that the finalizer is `terminal_failure` /
`native_finalization_superseded` and the run is `failed`.
4. See that the workspace still names the run as its dirty writer, and
that no retained-file recovery action exists.
5. Queue another run on the task. It defers with `workspace_busy`, and
after one hour it fails setup with "The workspace still has an
unfinished writer or save operation."

**Root cause**
`recordRetryableFailure` (`native-run-finalizer.ts`) retires a
superseded finalizer without preserving the export.
`preserveNativeFailedWorkspace` records a retained-file hold for an
accepted result only when the failure code is
`native_workspace_sync_out_retry_exhausted`. A
`native_finalization_superseded` coordinator never qualifies.
`readRemoteWorkspaceWriterHold` keeps an ended dirty writer as the
holder until its save settles. Without a hold, no worker ever settles
it.

**Paperclip version or commit**
Staging Core with #21106 (`81d2e4d8d9`). Master `ef2b53b675` has the
same behavior.

## What Changed

- `native-superseded-workspace-export.ts` (new): identifies a superseded
finalizer that failed in the workspace export. It selects ended runs
that still hold their workspace with unsaved files and no retained-file
hold. It records the hold through `preserveNativeFailedWorkspace`. It
checks again only while a just-ended run's sandbox stop receipt is still
arriving (at most one hour).
- `native-failed-workspace-recovery.ts`: `preserveNativeFailedWorkspace`
treats a superseded workspace export like an exhausted copy-back, with
the same proofs (run ended, no live controller, accepted result,
confirmed sandbox stop-and-retain, source still bound to the workspace).
It marks the hold `supersededWorkspaceExport`.
- `native-run-projection-work.ts`: the native run projection worker also
selects these runs. It loads them at startup and wakes when a finalizer
changes. Rows written by older releases are therefore settled after
deploy without a person.
- `workspace-recovery-fallback-discovery.ts` and
`workspace-recovery-fallback.ts`: a marked hold whose bounded re-export
is spent (at most 5 retries in 30 minutes) is eligible for the existing
continue-from-last-save fallback. That fallback sets aside only this
run's workspace epoch and keeps its location on the workspace record. It
attaches the readable files to the task unapplied. The task message says
Paperclip "could not save" the previous run's unsaved files.
- `preserved-result-finalization.ts`: a superseded result is never
resumed for application, even after its files are saved.
- `doc/execution-semantics.md`: document the rule.
- No schema change. Nothing changes on the path that starts a run or a
turn.

## Verification

- `server/src/services/superseded-workspace-export-recovery.test.ts`
(new). The first two tests fail on master and pass on this branch:
- "records the files for automatic recovery, saves them from the
retained sandbox and frees the workspace". It uses the staging row
shape. The projection worker records the hold, and a second pass changes
nothing. The workspace waits for that save. The system recovery saves
`saved.txt` into a committed snapshot. The workspace becomes free. The
finalizer, the result and the task's decision do not change, and the
result is not resumed. On master the worker selects nothing (`expected 0
to be 1`).
- "continues from the last save with the files attached when every
re-export fails". After an exhausted re-export budget, the hold is
resolved `restored`. The files are attached (`quarantine.state =
captured`). The workspace moves to epoch 2 with no writer, and the
abandoned epoch keeps its lease location. The message has no request for
a person to fix anything.
- "leaves a superseded result alone when it never failed in the
workspace export".
- Related suites pass: `native-run-projection-work`,
`native-finalization-recovery`, `workspace-recovery-fallback`,
`native-failed-workspace-recovery`, `workspace-recovery-retry-copyback`.
- `tsc --noEmit` for `server`.

## Risks

- Low to medium risk. The change acts only on runs whose finalizer is
already terminal and superseded, that failed in the workspace export,
and that still own a dirty workspace. All existing preservation proofs
still apply.
- The saved files of a superseded run become the workspace's next saved
version. The run's result (the task decision) is still never applied.
- When the export can never succeed, the task continues from its last
save. Files that cannot be read are not attached. The workspace record
keeps the set-aside epoch's location. Saving the readable part of a
partly unreadable workspace is not in scope.
- Runs that already failed setup before this deploys are not restarted
by this change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution, read-only API inspection) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:20:54 -06:00
DottaandClaude Opus 5.5 ef2b53b675 fix(environments): keep resuming an idle sandbox whose resume timed out (#21113)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task that runs on a reusable cloud sandbox resumes the same
sandbox on the next wake
> - When the resume call times out, the run failed after 90 seconds with
`reusable_sandbox_resume_failed (cause=transport_timeout)`
> - The provider keeps starting the sandbox after the host call times
out, so the sandbox is usually ready a little later
> - The run gave up after one call, and the task then waited behind a
Retry button
> - This pull request keeps the same run retrying an idle sandbox whose
resume only timed out, up to the provider's start deadline
> - The benefit is that recovery works automatically and a sandbox with
unsaved work is never destroyed

## Linked Issues or Issue Description

Refs #20626, #20828, #21020

**What happened?**
Three Codex runs on Daytona failed before provider dispatch with:
`Reusable sandbox lease "<id>" could not be resumed
(cause=transport_timeout); the lease was preserved and no replacement
was created.` Each run had `executionRecovery: { kind: "bootstrap",
providerWorkStarted: false }`. Each run took 100 to 122 seconds: one
resume call used the whole 90-second budget, so no in-run retry was
possible. The preserved leases were idle. Their last runs had ended
about two hours before, with a confirmed stop-and-retain. The workspaces
were not safe to replace, so the run failed strictly. The automatic
bootstrap retry was then refused ("The automatic retry could not
revalidate the original user continuation."), and all three tasks ended
`blocked` behind a Retry.

**Expected behavior**
A resume that only timed out, before any provider work, on a sandbox
that no live run holds, continues in the same run. The run keeps the
same sandbox and does not create a replacement or destroy the sandbox.

**Steps to reproduce**
1. Let a task's reusable Daytona sandbox stop and stay idle for some
hours (it auto-archives after 60 minutes).
2. Retry the task. The workspace has unsaved work, so replacement is not
safe.
3. The `environmentResumeLease` call takes longer than 90 seconds. The
host call times out (`RPC call "environmentResumeLease" timed out after
90000ms`), the 90-second budget is spent, and the run fails with
`reusable_sandbox_resume_failed`.

**Paperclip version or commit**
Staging at `849f109f` (current `master` has the same code path).

**Deployment mode**
Hosted staging (cloud), Daytona sandbox provider plugin, reusable leases
on.

**Agent adapter(s) involved**
`codex_local` (Codex) on a Daytona sandbox environment. The change is
adapter-independent.

**Root cause**
`server/src/services/environment-runtime.ts`, reusable resume loop: the
first `environmentResumeLease` call got the whole 90-second budget as
its RPC timeout. When that call timed out, the loop saw no time left and
threw. The 90-second budget assumes that the bounded bootstrap retry
resumes the same sandbox later. That retry is not always admitted. A
timeout is different from a refusal: the provider is still starting the
sandbox.

## What Changed

- `reusable-sandbox-resume-replacement.ts`: add
`reusableSandboxLeaseIsIdle` (lease is `released` or `retained` and no
other live run holds it). `reusableSandboxReplacementIsSafe` now uses
it, with the same rules as before. Add
`REUSABLE_SANDBOX_IDLE_RESUME_BUDGET_MS` (330 seconds, the Daytona start
deadline plus the call margin).
- `environment-runtime.ts`: when the 90-second budget ends on a
`transport_timeout` and the preserved lease is idle, extend the deadline
one time, up to the provider start deadline (at most 330 seconds from
the first attempt). The run keeps retrying the same sandbox with
backoff. Each call still has at most 90 seconds. Refusals, a live owner,
and other failures keep the current behavior. The run never replaces or
destroys the sandbox because of this extension.
- `doc/execution-semantics.md`: document the idle-timeout extension.

## Verification

- Red before the fix, with the new tests in
`server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts`
(staging shape: released lease, finished owner run, workspace with
unsaved work, 300-second provider deadline, host RPC timeout): 2 tests
failed with the staging error `could not be resumed
(cause=transport_timeout); the lease was preserved and no replacement
was created.`
- Green after the fix: 16/16 pass. The timed-out idle sandbox resumes in
the same run with no create or destroy call. If every call times out,
the run stops at or before 330 seconds, makes more than one call, and
keeps the lease `released`. A sandbox that a live run still holds gets
one call and fails at 90 seconds, as before.
- `pnpm exec vitest run src/__tests__/environment-runtime.test.ts
src/services/environment-runtime-reusable-lease-probe.test.ts
src/services/remote-workspace-transport.test.ts`: 378 pass.
- `tsc -p server/tsconfig.json --noEmit` passes.
- `pnpm verify` exits 2 because a doc is a global CI input. Required CI
supplies the broad coverage.
- Not run: a live Daytona resume on staging. No later run attempted the
three preserved leases, so it is inferred, not observed, that the
sandboxes were ready after the host call timed out.

## Risks

- A run can now wait up to 330 seconds in preparation before its first
turn, but only when the resume timed out on an idle sandbox. Before this
change, that run failed and the task waited for a person.
- A provider that is down (not slow) now takes up to 330 seconds to fail
instead of 90 seconds. The failure is the same
`reusable_sandbox_resume_failed`, and the lease is still preserved.
- No persisted fields change. Leases saved by older releases follow the
same rule.
- The refused bootstrap retry admission for task Retry continuations is
a separate gap. This pull request does not change it.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 08:28:32 -06:00
DottaandClaude Opus 5.5 81d2e4d8d9 fix(native): end a run whose finalizer a newer run superseded, so it stops holding its task (#21106)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run saves its accepted result, then a finalizer exports the
workspace and applies the result to the task.
> - When a newer run already committed the task's decision, the
finalizer retires the older result as `native_finalization_superseded`.
> - That retirement set the finalizer terminal, but it did not end the
run. The run stayed `running` and held the task and every run queued
behind it, with no way out except a person.
> - A stuck run must never hold a task and its queue forever.
> - This pull request ends the run when its finalizer is retired as
superseded, and it ends runs that an older release left in that state.
> - The benefit is that the task continues with its queued runs by
itself.

## Linked Issues or Issue Description

Refs #21046

**What happened?**
On a hosted staging server, a native run accepted its result at
11:31:08Z. The workspace export then failed
(`remote_workspace_archive_fs_eacces`). The task's last decision came
from another run that was created 42 seconds later but ran earlier. The
finalizer judged this run superseded: it set the finalizer to
`terminal_failure` with `native_finalization_superseded`. The run stayed
`running` with "Recovery needed". Over one hour later it still held the
task, and five queued runs waited behind it. A server upgrade at 12:02Z
did not change it.

**Expected behavior**
A run whose finalizer is retired ends. The task keeps the newer run's
decision, the retired result is never applied, and the queued runs
proceed.

**Steps to reproduce**
1. Accept a native result for a run.
2. Commit a status decision on the same task from a run that was created
later.
3. Let the older run's finalization fail once, so the finalizer records
the failure.
4. See that the finalizer is `terminal_failure` /
`native_finalization_superseded` and the run is still `running`.

**Root cause**
`recordRetryableFailure`
(`server/src/services/native-runtime/native-run-finalizer.ts`) projects
a terminal run status only when `exhausted && !supersededByNewerRun`. A
superseded retirement therefore never ended the run. No other path ends
a run whose finalizer is already `terminal_failure`: the reconciler
skips `terminal_failure` rows.

**Paperclip version or commit**
Staging Core `3eca7a6112`. The behavior is the same on master
`482063aeca`.

## What Changed

- `native-run-finalizer.ts`: a superseded retirement now ends the run
with the same status mapping as an exhausted finalization (workspace
failure: `failed`; otherwise the provider's terminal state). It does not
change the task, does not apply the result, and does not keep the
workspace export lease. The run's error says in plain words that a newer
run already decided the task.
- `native-run-finalizer.ts`: add `endSupersededNativeFinalizationRuns`.
It ends a native run that is still `running` while its finalizer is
`terminal_failure` / `native_finalization_superseded`. This covers rows
that an older release wrote. It locks in the finalizer's order
(coordinator, task, run), and re-checks the row under the lock.
- `native-run-projection-work.ts`: the registered native run projection
worker (it loads at startup, pages through all candidates, and wakes on
finalization changes) also selects such runs and ends them with
`endSupersededNativeFinalizationRuns`.
- `doc/execution-semantics.md`: document the rule.
- No schema change. Nothing changes on the path that starts a run or a
turn.

## Verification

- `server/src/__tests__/native-finalization-recovery.test.ts`, red on
master and green on this branch:
- "retires an older failed finalizer when a newer run already committed
the issue": the retired run now ends (`succeeded`, `finishedAt` set).
Master: the run stays `running`.
- "ends a run an earlier release left running after retiring its
finalizer as superseded": the shape of the staging row (workspace export
failed, superseded finalizer, run `running`). The repair ends it as
`failed` with the plain message. The task, its decision and the
finalizer stay unchanged, and no status decision is written for the run.
A second pass changes nothing.
- `server/src/__tests__/native-run-projection-work.test.ts`, "ends, at
startup, a run an earlier release left running after retiring its
finalizer as superseded": the registered worker ends the staging-shaped
row at startup and settles its claimed wake. Red without the worker
change.
- Related suites pass: `native-finalization-recovery`,
`native-run-projection-work`, `native-finalization-reconciler-timing`,
`native-reconciliation-timing`, `native-workspace-finalizer`,
`native-workspace-finalizer-failure`, `native-run-finalizer-telemetry`.
- `tsc --noEmit` for `server`.

## Risks

- Low risk. The change only ends runs whose finalizer is already
terminal and superseded. It never applies a result and never changes the
task.
- A superseded run whose workspace export failed now releases its
sandbox lease through the normal settlement instead of waiting forever.
Its result was not applied, because a newer run decided the task.
- Not changed here: the superseded check compares run creation times. A
run created earlier but executed later is still judged superseded. This
PR makes that verdict end the run; it does not change the verdict.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution, read-only API inspection) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 08:07:18 -06:00
DottaandClaude Opus 5.5 0a2f618cae fix(native): leave a run another live server is preparing alone, and re-adopt a run a shutdown handed off (#21107)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An upgrade replaces the server. The old and the new server serve
together for a few minutes, then the old server gets SIGTERM and hands
its native runs to the next controller.
> - #21046 taught the legacy reaper to leave a run with its owner while
the owner's server process is proven alive. Native runs have a similar
gap.
> - Runtime selection writes a native run's coordinator with no
controller and no lease, and stops the run lease. The executor claims
the coordinator about 20 seconds later. In that window the other
server's sweep sees no owner and marks the run "ownership could not be
verified".
> - When the old server then hands the run off before any Runner
checkpoint exists, no sweep looks at it again. Only the next server boot
does, 15 to 40 minutes later. The task and its queued runs wait.
> - This pull request leaves such a run with its live preparing server,
and lets the serving server re-adopt a handed-off run through the
ordinary restart claim.
> - The benefit is that a run started during an upgrade continues within
one sweep, with no false "blocked" mark and no wait for another upgrade.

## Linked Issues or Issue Description

Refs #21046

**What happened?**
On a hosted staging server, an upgrade started a new server at
12:19:57Z. The old server started a native run and selected its runtime
at 12:20:32Z. At 12:20:35Z the run logged "Native execution ownership
could not be verified; automatic recovery is blocked"
(`observed_owner_unverified`). The old server's executor claimed the run
at 12:20:51Z. The old server got SIGTERM and suspended the run at
12:20:54Z (`awaiting_runner_reattach`). After that the run showed
"Confirming execution" for 29 minutes, and two queued runs waited behind
it. At 12:49:45Z the next upgrade's startup recovery restarted its
bootstrap, and the run succeeded at 12:58Z.

**Expected behavior**
- A server does not mark a run unverified, or recover it, while another
live server is preparing it.
- A run that a graceful shutdown hands to the next controller is
re-adopted by the serving server at once, not at the next boot.

**Steps to reproduce**
1. Seed a native run whose coordinator is `observed` with no controller
and no lease, and whose `controllerProcess` names another server with a
live controller instance lease.
2. Run the recovery sweep on this server. See that the run gets
`native_execution_ownership_unverified`.
3. Claim the coordinator for the other server, then write what its
graceful shutdown writes (`restart_suspended`,
`awaiting_runner_reattach`, lapsed lease) and stop its instance lease.
Give the run a predecessor's Runner PID.
4. Run the sweep again. See that nothing claims the run.

**Root cause**
- `recoverActiveExecutions`
(`server/src/services/heartbeat/recovery.ts`) treats an `observed`
coordinator without a live controller lease as an owner it cannot
verify. Between runtime selection and the claim, the only owner on
record is the run's `controllerProcess`, and the sweep did not check it.
- `listExpiredNativeRunnerAuthorities` (`native-runner-checkpoint.ts`)
selects runs for the restart claim only when they have a controller
checkpoint, or no process identity at all. A run handed off just after
its claim has no checkpoint, and a follow-up carries its predecessor's
Runner PID. The sweep then skips `awaiting_runner_reattach` runs, so
only a server boot looked at them again.

**Paperclip version or commit**
Staging Core `814fe5d20f` (old server) and `3eca7a6112` (new server).
The behavior is the same on master `849f109f62`.

## What Changed

- `recovery.ts`: before it lists expired native runs, the sweep checks
runs in the runtime-selection window (coordinator `observed`, no
controller, no lease, not tracked here). When the run's
`controllerProcess` names another server process that the shared owner
prover proves alive, the sweep leaves the run with that server: it does
not recover it and does not mark it unverified. It re-checks within 30
seconds. After 5 minutes without a claim
(`LEGACY_LIVE_OWNER_SETTLE_GRACE_MS`), or once that server is gone, the
existing rules apply.
- `native-owner-stamps.ts`: add `preparingControllerOwnerStamp` for that
proof.
- `native-runner-checkpoint.ts`: `listExpiredNativeRunnerAuthorities`
also selects a run that a graceful shutdown handed off and whose lease
lapsed: recovery state `awaiting_runner_reattach` or
`awaiting_evidence`, with a `restart_suspended` history entry that no
restart claim followed. It stays selected through ownership waits (the
departing server can still be alive at the first look). `recovery.ts`
schedules a 30-second re-check for an ownership wait that has no retry
of its own. The ordinary restart claim then proves the takeover and
picks the disposition, exactly as the startup pass of a new server does.
A row that an older release already marked unverified is re-adopted too:
the claim clears the mark.
- `doc/execution-semantics.md`: document both rules.
- No schema change. Nothing changes on the path that starts a run or a
turn: the new proof runs only in the recovery sweep, and only for runs
in the window.

## Verification

- `server/src/__tests__/heartbeat-native-restart-suspension.test.ts`,
block "a run the old server started during an upgrade overlap". Red on
master, green on this branch:
- "leaves the run with its live preparing server, then re-adopts it
after that server hands it off": two concurrent sweeps leave the run
`running` with no error and do not touch the coordinator. After the old
server claims, hands the run off and stops, one sweep of the new server
re-adopts it and reaches provider dispatch. Master: the run is marked
`native_execution_ownership_unverified`, and after the handoff nothing
claims it.
- "keeps a handed-off run eligible while the departing server is still
alive, and re-adopts it once it exits": the first sweep waits
(`awaiting_evidence`), the sweep after the departing process exits
re-adopts the run. Red with a filter that matches only the last history
entry.
- "re-adopts a handed-off run that an older release already marked
ownership-unverified": the stuck staging shape. Master: nothing claims
it.
- Related suites pass: `heartbeat-native-restart-suspension` (all 109),
`services/heartbeat/recovery.test.ts` (all 103).
- `tsc --noEmit` for `server`.

## Risks

- A run that another live server prepares, but never claims, now stays
`running` for up to 5 minutes before the existing rules apply. That is
the same bound #21046 uses.
- The guard trusts the shared owner prover. It reads "alive" only from a
live controller instance lease or a live PID in the same PID space.
- More handed-off runs reach the restart claim earlier. The claim keeps
all its fencing and evidence checks; this change only makes the serving
server ask the same question a new boot asks.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution, read-only API inspection) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 08:06:57 -06:00
DottaandClaude Opus 5.5 a65644d1e1 fix(retry): continue an accepted connected Retry from recovered saved work when its chat replay is refused later (#21080)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Tasks from a connected chat (for example GitHub pull request
reviews) show Retry when a run fails
> - A person's Retry on such a task first asks the chat retry owner to
repeat the exact failed chat request. #21038 made a refusal at the click
continue the task from saved work instead
> - The chat owner can also defer its source check (a busy identity,
endpoint, or agent row lock) and refuse it a few seconds later. Then the
route has already answered 202, and the task Retry receipt was marked
failed with no run
> - The task stayed blocked, and its execution blocker told the person
to send a message, while the interruption projection still offered Retry
> - This pull request makes that late refusal take the same board
continuation as a refusal at the click, and makes the blocker offer
Retry for recovered saved work
> - The benefit is that an accepted Retry always starts work, and the
two projections agree

## Linked Issues or Issue Description

No public issue exists. This is the bug description, in the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

On a hosted deploy, about 20 blocked tasks of one busy GitHub review
agent (`paperclip_runner`, native remote runs) did not restart after
Retry. `POST /api/issues/:id/retry` returned 202 with `accepted: true`,
a `chatActionId`, and `presentation.state: retry_scheduled`. No run was
created. The task Retry receipt changed to `failed` 5 to 10 seconds
later, and the task stayed blocked. The activity record of each click
(`chat.failed_run_retry_requested`) has `sourceAuthorizationPending:
true`, so the chat owner deferred its source check at the click. No chat
wake row was written, so the chat replay ended before admission. The
last run failed with `native_continuation_requires_reconciliation` /
`runner_remote_process_identity_unavailable`. Its saved work was
recovered (`workspaceRecovery.status: recovered`, remote lease destroyed
with a termination receipt). The projections disagreed: `interruption`
had `canRetry: true`, but `executionBlocker` had `canRetry: false` and
`nextAction: "Saved work recovered. Send a message to continue."`.

Root cause: when the chat retry action settles `failed`,
`releaseUnadmittedTaskRetryInput` in
`server/src/services/chat-channels/task-retry-input.ts` marked the
linked task Retry receipt `failed`. #21038 added a board-continuation
fallback only for a refusal inside the click transaction. A refusal
after a deferred source check (for example 8 lock-busy attempts on the
agent row, about 7 seconds) had no fallback. Separately,
`getExecutionBlocker` hid Retry for every chat-bound task (`canRetry &&
!chatBinding`), and `workspaceRecoveryDisplay` always said "Send a
message to continue" for recovered work.

**Expected behavior**

A Retry that is accepted starts work. If the exact chat replay is
refused after the click, the person's Retry continues the task from its
recovered saved work. The execution blocker and the interruption both
offer Retry, and neither tells the person to send a message to unblock a
failed run.

**Steps to reproduce**

1. Bind a task to a GitHub chat conversation. Make its latest native run
fail with `native_continuation_requires_reconciliation` and recover its
saved work.
2. Hold the chat identity advisory lock while you click Retry, so the
source check is deferred (`sourceAuthorizationPending: true`).
3. Hold a row lock on the agent while the chat retry processes, so the
deferred check fails 8 times.
4. The route returned 202. The chat action settles `failed`, the task
Retry receipt changes to `failed`, and no run starts.

**Paperclip version or commit**

`master` at a8580160 (the deployed revision). Reproduced on f30005440b.

**Deployment mode**

Hosted (cloud) instance with Daytona remote sandboxes and a GitHub chat
connection.

**Agent adapter(s) involved**

`paperclip_runner` (native remote runs). The fix is in core Retry code
and is not adapter-specific.

## What Changed

- `server/src/services/task-retry.ts`: new `settleRefusedTaskChatRetry`.
When a chat retry action settles `failed`, the linked task Retry receipt
becomes an ordinary board continuation if all of these are true: it is a
person's Retry (or an automatic Retry that the same person's click has
joined), its source run is still the latest run and its saved work is
recovered (system recovery `complete`, workspace recovery `recovered`),
the clicking person still has task access, the chat wake has no run, and
no saved or unadmitted chat input remains. Any other refusal still
settles the receipt as failed, so the existing source-edited, revoked,
and identity-expiry outcomes do not change. A paused agent does not fail
the receipt: it stays deferred and starts when dispatch allows. The
receipt drops `chatActionId`, stays `deferred_issue_execution`, notifies
the deferred-receipt worker, and records
`issue.retry_continued_from_saved_work`. Otherwise the receipt is failed
as before. `reconcileTaskChatRetryReceipts` uses the same check, so a
crash before settlement is also covered. Its failed write matches the
chat binding it read, so a stale pass cannot fail a receipt that
settlement already converted.
- The task Retry intent now records `initiatedBy` for an automatic
Retry, so an automatic Retry keeps the exact-replay requirement after
the click too.
- `server/src/services/chat-channels/task-retry-input.ts`:
`releaseUnadmittedTaskRetryInput` calls the task owner's settlement
instead of failing the receipt directly.
- `server/src/services/execution-blocker.ts` and
`workspace-recovery-receipt.ts`: for a failed or timed-out run whose
saved work is recovered, the blocker offers Retry also on a chat-bound
task. The text follows actual Retry eligibility: "Saved work recovered.
Retry continues from it." when Retry is offered, "Saved work recovered."
when it is not (for example cleanup quarantine), and "Send a message to
continue" only for a stopped run. Only the task blocker decides Retry.
The run-level projection (`execution-projection.ts`) has no task
context, so it says "Saved work recovered." and never claims Retry.
`execution-projection.ts` uses the same text.
- `ui/src/lib/activity-format.ts`: labels for the new activity.
- `doc/execution-semantics.md`: the late-refusal rule.
- Tests: a new real-service case in
`server/src/__tests__/chat-channels.integration.test.ts` that reproduces
the hosted state (remote saved workspace, destroyed lease receipt,
recovered recovery action, deferred source check, refused replay), for a
person's Retry and for an automatic Retry. A gate case per continuation
condition (unrecovered workspace and revoked access settle failed; a
paused agent stays deferred and runs after resume), a stale-reconcile
race case, and a joined-click case. Two existing text expectations are
updated.

## Verification

- Red before the fix: with only the source changes removed, the new test
fails on the projection (`canRetry: false`, "Send a message to
continue."). With only the Retry settlement removed, it fails with the
receipt `status: "failed"` and no run.
- Green after the fix: `npx vitest run
src/__tests__/chat-channels.integration.test.ts -t "continues an
accepted GitHub task Retry from recovered saved work"` passes 2/2. The
person's Retry starts one queued run that resumes the preserved
workspace, and the automatic Retry still settles as failed.
- Focused suites pass: `task-retry`, `execution-projection`,
`execution-projection-recovery`, `workspace-recovery-retry-copyback`,
`chat-failed-run-retry-boundary`, `workspace-recovery-admission`,
`remote-workspace-recovery-receipt-fallback` (372/372), and other
recovery, blocker and continuation suites.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- Staging was read with GET requests only. No POST was sent.

## Risks

- Behavior change: a person's Retry on a connected task that the chat
owner refuses after the click now starts a board continuation. It is the
same continuation #21038 already uses for a refusal at the click, with
the clicking person's authority, and dispatch keeps every normal gate.
- Receipts that already failed on staging stay failed. The next click on
those tasks takes the fixed path. No migration.
- The blocker now reports `canRetry: true` for chat-bound tasks only
when the failed run's saved work is verifiably recovered.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 07:57:42 -06:00
DottaandClaude Opus 5.5 cc6728e8de fix(recovery): continue upgrade-stopped Retry runs that the sweep refused or skipped (#21108)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Hosted servers are upgraded often. Each upgrade stops the old server
gracefully, and a run that is still preparing ends `interrupted`
(`server_shutdown_interrupted`) with bootstrap evidence that provider
work did not start.
> - #21078 let a run that a person started with Retry continue
automatically after such a stop. After it deployed, a read-only audit of
a staging server found four groups of Retry-started tasks that still did
not continue: some were refused on every sweep, some were never looked
at, and one Retry that a person pressed failed and was not retried.
> - Product rule: an upgrade never leaves a task that needs a manual
Retry, and a Retry a person presses must work. A refusal that cannot
succeed later must not repeat forever.
> - This pull request fixes each cause found, keeps every authorization
check of the continuation gate, and ends the refusal loop with one plain
escalation.
> - The benefit is that Retry-started tasks continue through upgrades,
including tasks earlier releases left waiting, with no manual step.

## Linked Issues or Issue Description

Refs #21078, #21046, #21019, #14987, #19384

**What happened?** On a hosted staging server after #21078 deployed
(read-only audit, run events and task activity):
1. A Retry-started task was refused every ~2 minutes ("The automatic
retry could not revalidate the original user continuation."). While
preparing, its Retry had switched to a fresh isolated workspace because
the saved files could not be restored (activity
`issue.fresh_workspace_selected` by the system, 40 minutes after the
Retry). That rewrote the Retry request's strategy from `continue` to
`fresh_isolated_workspace`, and `readTaskRetryRetryAuthority` accepts
only `continue`.
2. Two tasks whose Retry replaced a native session had no recovery event
for hours. The stranded-task sweep skipped them silently.
3. A task whose Retry selected a fresh workspace at admission
(`explicitUserContinuation.readiness`) was refused on purpose
("fresh-workspace Retries keep their own contracts"), yet that contract
had no path for a shutdown before provider work. The task stayed in
progress after every upgrade.
4. A person pressed Retry. The run failed before provider work with
`reusable_sandbox_resume_failed`, a transient failure that is already in
the bounded bootstrap retry lane. One second later the stranded-task
sweep escalated the task to `blocked`, which cleared the claim. Two
seconds after that, the run's own failure path scheduled its retry and
was refused because the claim no longer named the run.

**Expected behavior:** each of these continues automatically once, under
the person's original Retry. A refusal that cannot succeed later
escalates the task once with a working Retry.

**Root causes:**
- Cases 1 and 3: the continuation gate refused every fresh-workspace
Retry, whether the fresh workspace was chosen at admission or later in
preparation. That includes one that a shutdown stopped before it did
anything.
- Case 2: the sweep's generic skips run before the continuation. Two of
them can match only because of state that the replaced session left. An
unfinished session goal (`agent_task_sessions.goal_status`) is skipped
as "goal recovery owns it". A native finalization row is counted as a
live execution path by `hasActiveExecutionPath`. The person's Retry had
already started a fresh session, so neither owns the task. The audit
could not read the database, so which of the two applied on staging is
not proven; the tests cover both.
- Case 4: a race. The sweep does not wait while a terminal run's own
executor is still settling it, so it escalates before that executor
schedules the run's bounded retry.
- The refusal loop: after a refusal the sweep left the task
`in_progress` and tried again on the next pass.

**Paperclip version or commit:** master `482063aeca`.

## What Changed

- `explicit-native-continuation.ts`: `readTaskRetryRetryAuthority`
renews a fresh-workspace Retry **only** when the parent run was stopped
by a graceful shutdown before provider work
(`isShutdownInterruptedPreparation`). The admitted `readiness` must name
the same Retry and source. A strategy that preparation switched to fresh
is rebuilt exactly as preparation records it: this Retry, its source,
and the source's leases. An automatic retry of the fresh attempt
inherits the selection from its own receipt. The successor's receipt
records the selection under `automaticRetry.readiness`. The successor's
`explicitUserContinuation` keeps its plain shape.
- `fresh-workspace-continuation.ts`: `readFreshWorkspaceReadiness` reads
that receipt for an automatic continuation, so the successor prepares
the same fresh attempt. A remote fresh workspace is keyed by the Retry,
so it is reused rather than created again. `freshReadinessSchema` is
exported and reused.
- `recovery/service.ts`:
- **Shutdown-stopped Retry preparations go first:** at the start of the
stranded sweep, every `in_progress` task whose latest run is a legacy
`server_shutdown_interrupted` preparation (`providerWorkStarted: false`)
carrying `explicitUserContinuation` is offered to the scheduler. It is
skipped when the agent is not invokable, the task has a live execution
path, a board-owned recovery action or a pause hold, or the run is still
settling. The unfinished-goal skip does not apply here, because the
person's Retry started a fresh session.
- `hasActiveExecutionPath` ignores a native finalization that a person's
Retry already replaced (`failureDetail.replacementDenied =
explicit_user_continuation` on a terminal run).
- The sweep skips a task while its latest terminal run is still being
settled: a live execution in this process, or another server's unexpired
controller lease. It checks again when that lease expires.
- A refused continuation (`continuation_user_authorization_missing`)
escalates the task once to `blocked` with the plain notice "Retry needed
… Press Retry to continue." Before, the task stayed in progress and was
refused again on the next pass.
- `heartbeat.ts`: `scheduleRecoveryRetry` reports a refused continuation
to the sweep.
- `doc/execution-semantics.md`: describes these rules.

**Failure classes admitted for a Retry-started run (case 4):** no new
class. The run's own bounded retry lanes are unchanged.
`reusable_sandbox_resume_failed`, `remote_workspace_restore_failed`,
`plugin_worker_unavailable`, `environment_provider_unavailable`,
`repository_catalog_unavailable`, runner revision mismatch and the other
transient bootstrap failures were already retried before provider work.
The fix removes the race that took the claim away from that retry.

**Refusals that remain by design:**
- The task is reassigned, `done` or `cancelled`.
- Another run holds the claim, or newer or live work exists.
- The Retry actor, generation or run changed, or it is a chat or
queued-message Retry.
- The message was edited or deleted.
- The receipt is missing or revoked.
- The run had another interruption cause, or provider work started.
- A fresh-workspace Retry whose run *failed* (not a shutdown) keeps its
one-run contract.
- A readiness that names another Retry or another source.

Each of these now escalates the task once instead of refusing on every
sweep.

## Verification

- `server/src/services/task-retry.test.ts`, new tests. Red on
`origin/master` sources and green on this branch (checked by restoring
the master sources):
- "continues a fresh-workspace Retry that a shutdown stopped before
provider work":
- Runs twice: once with the fresh workspace selected at admission, once
with it selected at preparation.
- Two concurrent sweeps create one successor, which gets the claim, and
dispatch accepts it.
- `readFreshWorkspaceReadiness(successor)` returns the same fresh
attempt.
- A second upgrade stops the successor, and the next successor carries
the same attempt.
- "continues a shutdown-stopped Retry whose replaced session left state
behind": two cases, an unfinished session goal and a native finalization
that the Retry replaced.
- "escalates a refused shutdown continuation once instead of refusing it
on every sweep": two sweeps. The task ends `blocked` with one "Press
Retry to continue" notice and one refusal event.
- "leaves a failed Retry run to its own executor while that executor
still settles it":
- Production shape: `reusable_sandbox_resume_failed` and a live
controller lease from the server that ran the run.
    - The sweep leaves the task and its claim alone.
    - The run's bounded retry is then scheduled and dispatch accepts it.
- "keeps a fresh-workspace Retry's own contract when it does not match":
three refusal cases (other Retry, other source, failed rather than
stopped). Green on both.
- Existing suites pass:
- `task-retry`, `explicit-native-continuation`,
`heartbeat-retry-scheduling`, `heartbeat-process-recovery`,
`interrupted-preparation-hold-settle`,
`heartbeat-issue-liveness-escalation`, `issue-recovery-actions`,
`recovery-deadline-acceptance`, `legacy-continuation-authority`,
`workspace-recovery-admission`, `native-workspace-export-retry`,
`heartbeat-paused-assignee-new-task`, `issue-tree-pause-resume-run`,
`heartbeat-stale-queue-invalidation`. 1,619 tests passed, with one
failure that a change in this branch fixed; rerun of the affected
suites: 560/560.
- The one failure came from the first version of the settling check,
which also waited for this process's own lease after a shutdown drain.
The check now uses this process's live executions and other processes'
leases only.
- `tsc --noEmit` for `server`.
- Required CI covers the full suite.

**After deploy, the next audit should show:**
- Tasks stopped by a shutdown while their Retry was preparing get one
`scheduled_retry` successor on the first sweep. This includes tasks
whose Retry selected a fresh workspace and tasks whose Retry replaced a
native session.
- They show no further "could not revalidate" events.
- A Retry-started run that fails transiently before provider work gets
its bounded retry, and the task is not escalated in between.
- A task whose continuation is truly refused is `blocked` once with
"Retry needed".

## Risks

- **Behavior change:** the sweep now offers shutdown-stopped Retry or
message preparations to the scheduler before its generic skips. All
authorization checks still run at admission, under the task and run row
locks. The scheduler reuses an existing successor, so two servers create
one continuation. The tests run two sweeps at once.
- **Settling wait:** a task whose latest terminal run still has another
server's live controller lease waits until that lease expires (at most
90 s) before the sweep acts. Normal turn starts are unchanged, and
nothing is added before a turn starts.
- **Escalation on refusal:** a refused continuation now moves the task
to `blocked` with a Retry notice, where before it stayed `in_progress`
with no path. Pressing Retry is a new, separately admitted request.
- **Rows from older releases:** no schema change and no backfill. A
shutdown-stopped fresh-workspace Retry is continued once. An older
successor receipt without `automaticRetry.readiness` behaves as before.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, read-only API reads, Vitest on embedded PostgreSQL) in
Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 07:41:25 -06:00
DottaandClaude Opus 5.5 533f135780 fix(native): send a re-wake whose crashed predecessor's warm Runner cannot be adopted to a fresh sandbox instead of failing it (#21001)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a warm remote sandbox leaves its Runner `ready` in
the sandbox. The next run of the task adopts that Runner live, after it
proves that the previous owner finished
> - When the server is killed after a run parked on a question, the warm
owner lived only in the killed server's memory. The Runner in the
sandbox was never suspended
> - A re-wake on the replacement server resumed that sandbox and tried
live adoption. A proof failed, and the new run failed with
`runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate` and needed a Retry, although
it did no provider work
> - #20863 already sends such a run to a fresh sandbox when the
predecessor's Runner cannot be retired, but the live-adoption path of a
resumed sandbox (`remote-warm-session-recovery.ts`) was not connected to
it, and the failed run's inherited Runner identity blocked the
"workspace unchanged" proof that the fallback needs
> - This pull request connects the two: a refusal of live adoption after
the predecessor is durably terminal with an accepted result goes to the
#20863 fresh-sandbox path, and an inherited Runner identity no longer
counts as a process of the follow-up
> - The benefit is that a re-wake seconds after a crash restart
continues without a Retry

## Linked Issues or Issue Description

Refs #20863, #20844, #20928, #20979, #20953, #20972

**What happened?**
Restart-chaos wave `w-20261112T012841-s7000` (master `4b24352712`,
volume-free mode, Daytona warm, Codex, 10 tasks, K1 and K15 with a fresh
home and overlap), task t08 (W2Q):
- Run `58f57bf1` asked its question and succeeded at 01:29:56. Its warm
owner stayed in the server's memory.
- K1 killed the server at 01:30:06. The replacement was ready at
01:30:15.
- A board comment woke the task at 01:30:17. Run `8f812df6` started at
01:30:19 and resumed the same sandbox (`run.startup.reuse`
`verified_reuse`).
- The first live-adoption inspection passed. The second check
(`revalidate`, in backend setup, before the Runner attach) refused at
01:30:21 (stack: `remote-warm-session-recovery.ts:217` from
`revalidate`). The run failed `native_session_interrupted` with
`INSTRUCTION_COLLECTION_UNAVAILABLE`.
- The failed run kept its full save because the unchanged-files proof
refused with `process_recorded`: the run carried its predecessor's
Runner process identity. Retry (`6968dc0f`) worked.

**Expected behavior**
The predecessor finished with an accepted result before the kill, and
the new run did no provider work. The task must continue without a
Retry.

**Steps to reproduce**
1. Finish a warm remote run that parks on a question.
2. Kill the server (SIGKILL) and start a replacement.
3. Wake the task within seconds. The new run resumes the warm sandbox.
4. Any live-adoption proof fails, and the run fails terminally with
`prior_owner_terminal_state_indeterminate`.

**Which proof failed**
The old refusal had no reason. The wave evidence proves that it was
`revalidate`, after a passed inspection, and not the SIGTERM of the next
K15 fault (that server logged "shutdown signal received" only at
01:31:09). `revalidate` checks the predecessor's coordinator row, the
current run's coordinator row and lease, and the task. The evidence
cannot tell which one changed in the first seconds after the restart.
This pull request names the failed proof
(`warm_recovery_refused=<reason>`), so the next occurrence identifies
it. The fix covers the class: every refusal that concerns the
predecessor or the sandbox.

## What Changed

- `server/src/services/native-runtime/remote-warm-session-recovery.ts`:
- Each refusal names its proof from a closed list:
`identity_incomplete`, `target_not_resumed_sandbox`, `owner_unsettled`,
`execution_mismatch`, `checkpoint_unsettled`, `journal_unsettled`,
`controller_not_stopped`, `prior_lease_mismatch`, `owner_changed`,
`current_lease_changed`, `current_run_not_admitted`,
`current_owner_changed`.
- After the predecessor's run is `succeeded` and its coordinator is
`committed` with a result and no lease, a refusal about the predecessor,
its journal or the sandbox throws
`NativePredecessorRetirementUnprovenError` (predecessor run, sandbox).
The existing #20863 path then marks the sandbox `replaceBeforeReuse` and
re-queues the run in the uncharged `interrupted_preparation` lane. The
fresh sandbox is used only when the workspace save is clean and the
harness backup is verified (unchanged #20863 safety rules); otherwise
the sandbox is resumed as before.
- A refusal because the new run is no longer admitted (stopped,
reassigned, paused, company inactive) or no longer owned by this
controller stays an ordinary failure, so a fallback cannot become a path
around Stop or reassignment.
- `server/src/services/remote-workspace-unchanged-proof.ts`: a run whose
recorded Runner process identity is exactly a settled predecessor's
identity of the same session (`carriesInheritedRunnerIdentity`, from
#20979), with no `native.process_identity_recorded` event of its own, is
no longer refused with `process_recorded`. A provider session of its own
(`sessionIdAfter`) and every other check still refuse.
- `doc/execution-semantics.md`: document both rules.

## Verification

- Red on `origin/master` (`8818402e7a`):
- `cd server && npx vitest run
src/services/native-runtime/native-controller-recovery.integration.test.ts
-t "crash restart"`: `AssertionError: expected Error:
runner_state_identity_mismatch: pr… to be an instance of
NativePredecessorRetirementUnprovenError`.
- `npx vitest run
src/services/remote-workspace-unchanged-release.test.ts -t "follow-up
that carries"`: `expected false to be true` for the inherited-identity
case; the own-identity and own-record cases passed (still refused).
- Green with the fix: `native-controller-recovery.integration`,
`remote-workspace-unchanged-release`, `native-recovered-owner-followup`
and `heartbeat-native-predispatch-retry` pass (149 tests). Also run:
`native-session-executor.test`, `explicit-native-continuation.test`,
`task-workspace-binding.test`,
`native-runner-restart-recovery.integration.test` (see the PR comment
for the result). `tsc --noEmit` for the server passes.
- The new integration case is the staging shape: a predecessor that
parked on a question (unanswered `semantic_tool.input` before its
terminal), a replacement controller, a resumed sandbox. It checks the
journal refusal, a predecessor row taken by another worker between
inspection and revalidation, a changed current lease, and that a
cancelled current run is not sent to the fallback.
- Chaos: the before/after rerun of seed 7000 is in a PR comment.

## Risks

- The fallback starts the next attempt on a fresh sandbox instead of
reusing the warm Runner. That costs one cold start, only after live
adoption is refused.
- The unchanged-files proof accepts an inherited Runner identity. The
identity must match a settled predecessor of the same session exactly
(PID, group and start time), and the run must have no own identity
record, no execution event, no runner checkpoint, no workspace operation
and no result. These are the same facts #20979 uses for restart
recovery.
- Error text of this refusal gains `; warm_recovery_refused=<reason>`.
Callers match the `runner_state_identity_mismatch: ` prefix only.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:08:03 -06:00
DottaandClaude Opus 5.5 c4927e1ec2 fix(recovery): continue a Retry-started run that a server shutdown stopped while preparing (#21078)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An upgrade stops the old server gracefully. A legacy run that is
still preparing ends `interrupted` with `server_shutdown_interrupted`.
The new server must continue it automatically (#21019, #21046).
> - A run that a person started with Retry (or a message) carries
`explicitUserContinuation`. Its automatic retries need their own receipt
from `admitExplicitContinuationRetry` (#14987, #19384).
> - That admission refused every shutdown-interrupted Retry run. The
task then stayed in progress with no run, and the stranded-task sweep
refused it again about every 2.5 minutes, forever. A person had to press
Retry.
> - An upgrade is routine. It must never leave a task that needs a
manual Retry.
> - This pull request lets the person's original authorization cover a
continuation after a shutdown that stopped the run before provider work
started. All other checks of the gate stay.
> - The benefit is that Retry-started tasks continue through an upgrade,
and tasks that earlier releases left waiting continue on their own after
deploy.

## Linked Issues or Issue Description

Refs #21019, #21046, #14987, #19384

**What happened?**
On a hosted staging server, each automatic upgrade stopped legacy runs
that were still preparing (`executionRecovery {kind: bootstrap,
providerWorkStarted: false}`). When the run came from a person's Retry
(wake reason `task_retry`), the continuation was refused with "The
automatic retry could not revalidate the original user continuation."
(`continuation_user_authorization_missing`). The task stayed
`in_progress` with no run and no next retry. Runs from older upgrades
were refused again by each sweep. A run from the same upgrade that did
not come from a Retry got its continuation in the
`interrupted_preparation` lane.

**Expected behavior**
A Retry-started run that a graceful shutdown stopped before provider
work continues automatically, once, under the person's original Retry.
Tasks that an earlier release already refused continue after deploy with
no manual step.

**Root cause**
Three conditions refused it (`server/src/services/heartbeat/retries.ts`,
`server/src/services/explicit-native-continuation.ts`):
1. The scheduler admits an explicit continuation only for the
`transient_failure`, `server_restart`, preparation-wait and
agent-directory lanes. A shutdown-interrupted preparation with unspent
failure budget uses `interrupted_preparation`, so it was refused before
admission ran.
2. `admitExplicitContinuationRetry` accepted only a `failed` or
`timed_out` parent. These parents are `interrupted`.
3. It required `task.executionRunId` to be the parent. The refusal
released the claim (`releaseIssueExecutionAndPromote`), so every later
sweep was refused by this check too.

**Paperclip version or commit**
master `f30005440b`.

## What Changed

- `retries.ts`: an explicit continuation may also use the
`interrupted_preparation` lane. Lane choice is unchanged (see below).
- `explicit-native-continuation.ts`: `admitExplicitContinuationRetry`
also accepts a parent that is `interrupted` by a graceful shutdown
(`isServerReplacementInterruption`, so `server_shutdown_interrupted`)
with bootstrap evidence `providerWorkStarted: false`
(`isShutdownInterruptedPreparation`).
- For such a parent only, a released claim (`executionRunId` null) is
accepted. Admission restores the claim to the parent under the task
lock, as the shutdown path leaves it, so the scheduler passes the claim
to the successor and dispatch accepts it. If the parent validation then
refuses, admission releases the claim again in the same transaction. A
claim that another run holds still refuses.
- `doc/execution-semantics.md`: describe the rule and the conditions
that still refuse. Also apply the sentence that #19384 left pending (a
plain task Retry is renewable).
- No schema change, no backfill. Nothing changes on the path that starts
a turn.

**Which lane a graceful-shutdown legacy preparation uses (after
#21046).** `scheduleBoundedRetryForRun` sets `interruptedPreparation`
for an `interrupted` run with the bootstrap evidence.
`usesServerRestartLane(run, true)` returns false while the failure
budget is unspent, so the run uses `interrupted_preparation`: it does
not spend the failure budget, the first continuation is due at once, and
it is bounded by `INTERRUPTED_PREPARATION_RETRY_MAX_ATTEMPTS`. With a
spent failure budget it uses `server_restart`. This is the intended
design (#21046 comment on `usesServerRestartLane`), so no lane change.
The `server_restart` lane is for `process_lost` replaced preparations
and for runs past dispatch.

**Conditions that still refuse** (each has a test or an existing test):
- The task has another assignee, or is `done` or `cancelled`.
- Another run holds the task's claim.
- Newer work on the task (a run created after the parent, a retry child,
or a live run).
- The Retry request changed: other actor, not admitted, other
generation, bound to another run, a chat Retry, a fresh-workspace Retry.
A message continuation with an edited or deleted message.
- A missing or revoked receipt (dispatch validation of the parent
fails).
- An execution blocker on the task, or a live adapter controller for the
parent.
- An `interrupted` parent with another cause (for example an operator
interrupt), or with no bootstrap evidence, or with `providerWorkStarted:
true`. A parent that failed after provider work keeps the existing rules
(`failed`/`timed_out` only).
- Native runtime parents (unchanged, see below).

**Native runtime.** A native `paperclip_runner` run is not interrupted
by a graceful shutdown: `drainRunningRunsForShutdown` suspends it and
the next server reattaches it (`restart_suspended`). It does not reach
this gate. Native replacement paths (`native-safe-replacement`,
`native-provider-capacity-retry`) remove `explicitUserContinuation` from
the successor and use their own contract. So the native path does not
have this refusal, and this pull request does not change it.

## Verification

- `server/src/services/task-retry.test.ts`, new tests. Red on
`origin/master` (no successor), green on this branch:
- "continues a Retry-admitted run that a graceful shutdown stopped while
it was preparing": production shape. A legacy run admitted by a task
Retry, `preparing`, owned by this server, with bootstrap evidence.
`drainRunningRunsForShutdown("SIGTERM")` interrupts it with
`server_shutdown_interrupted` and schedules one
`interrupted_preparation` successor (`failureRetries: 0`,
`interruptedPreparations: 1`) with its own receipt. The claim passes to
the successor and dispatch accepts it. A concurrent stranded sweep and a
second schedule call reuse the same successor.
- "continues a task an earlier release left waiting after refusing its
shutdown continuation" (a direct Retry run, and an automatic retry of a
Retry run): the row and task as an older release left them (claim
released). Two concurrent stranded sweeps schedule exactly one
successor; the claim passes to it; dispatch accepts it.
- "does not continue a shutdown-interrupted Retry run without its
original authority" for: provider work started, no bootstrap proof,
another interruption cause, claim held by another run, reassigned,
superseded, other Retry actor, cancelled task. No successor, no receipt,
and the claim is left as it was.
- I confirmed that each source change is needed: without the lane change
both main tests fail; without the released-claim rule the older-release
test fails.
- Suites: `task-retry`, `explicit-native-continuation`,
`legacy-explicit-continuation-recovery`, `quarantined-task-retry`
(587/587); `heartbeat-process-recovery`, `heartbeat-retry-scheduling`
(520/520).
- `tsc --noEmit` for `server`.
- `pnpm verify` selects a broad run (exit 2) because `retries.ts` is
central; required CI covers that.

**After deploy, the next audit should show:** a Retry-started run that a
shutdown stops while preparing gets a `scheduled_retry` successor with
`scheduledRetryReason: interrupted_preparation` (or `server_restart`
with a spent failure budget) and no
`continuation_user_authorization_missing` event. The tasks that earlier
upgrades left waiting get one such successor on the first stranded sweep
of the new server, with the task claim on the successor.

## Risks

- Behavior change: a shutdown-interrupted preparation of a Retry or
message turn now continues automatically, within the existing uncharged
and bounded lanes. The run did no provider work, so there is nothing to
reconcile.
- Rows saved by older releases: a refused row whose claim was released
is continued once. If another run took the claim, or newer work exists,
it stays refused.
- Two servers at once: admission runs in the scheduler transaction under
the task and run row locks, and the scheduler reuses an existing
successor (`retryOfRunId`). The tests run two sweeps concurrently and
get one successor. An old server that still refuses cannot clear the
claim after the new server moved it to the successor (its release
matches only the stopped run).
- The claim restore writes `executionRunId` for a terminal run inside
the scheduling transaction. If admission refuses after the restore, it
releases the claim again before the transaction commits.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, Vitest on embedded PostgreSQL) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:02:56 -06:00
DottaandClaude Opus 5.5 f30005440b fix(recovery): reconnect a live remote Runner with its admitted skills instead of re-downloading them (#21064)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner sandbox. A server upgrade
detaches each running turn, and the new server must reconnect to its
Runner (restart recovery).
> - A reconnect runs the full turn preparation before it contacts the
Runner. On a new server, skill preparation downloads every saved skill
version again, under one lock for the whole company.
> - After an upgrade, many reconnects wait behind each other's
downloads. The Runner stops answering after about 2 minutes, so a late
reconnect loses the turn.
> - This pull request lets a reconnect to the same run's live remote
Runner use the exact saved skill versions, without downloading them and
without the lock.
> - The benefit is that skill preparation no longer adds to the
reconnect delay, and it does not grow with the number of reconnects.
Normal turn starts do not change.

## Linked Issues or Issue Description

**What happened?**
After an automatic staging upgrade, runs that the new server reached
late failed. The measured time from detach to reconnect was 114–152 s
for the failed runs. Turn preparation took 30–130 s, and skill
preparation alone took 46 s on one run.

**Expected behavior**
A reconnect to a live Runner reaches it quickly. Its speed must not
depend on how many other runs prepare skills at the same time.

**Steps to reproduce**
1. Run a remote Runner turn for each of many tasks in one company.
2. Upgrade the server while those turns run, so the new server starts
with an empty skill cache.
3. See that each reconnect waits in
`heartbeat.prepare.configuration_skills` for the company skill cache
lock and downloads the skills again.

**Paperclip version or commit**
master at 4228bcf89c

**Additional context**
Root cause: `withRuntimeSkillEntries` holds the company-wide skill cache
lock (a Postgres advisory lock) while it materializes each saved skill
version. A new server has an empty skill cache, so each reconnect
downloads its skills again, one run at a time. The Runner already has
these files, and the reconnect path does not read them.

## What Changed

- `company-skills.ts`: new `admittedVersionsOnly` option. It still
checks that each saved version exists and that the agent can read the
skill. It does not materialize the files or take the company skill cache
lock.
- `heartbeat/live-runner-adoption.ts`: `adoptsLiveRemoteRunner` is true
only for a `reattach_remote_runner` claim for this run, with the claim's
own lease owner, on a native run.
- `heartbeat/execution-configuration.ts`: uses the option only when all
of these are true: the reconnect gate is true, the run has its saved
native input with a runtime context, and the run has its saved skill
versions. In that case it times the phase as
`configuration_skills_adopted` and adds one lifecycle run event
(`skillPreparation: "reused_admitted"`).
- `heartbeat.ts`: passes the gate to execution configuration.
- `native-runtime/runtime-context.ts` and `native-session-executor.ts`:
on the same reconnect to a remote Runner filesystem, the Runner
controller describes the Runner's skills by their admitted digests. It
rehydrates the pinned skill files (which takes the same lock) only if it
must launch a Runner process and stage them. All other starts rehydrate
as before.
- `doc/execution-semantics.md`: describes the new behavior.

Not in this PR (by design): a separate "dispatch-only" reconnect path,
and a new startup scheduling lane. The native executor needs about 40
values from preparation (runner environment, MCP token, managed AI
credentials, the reattached execution target, run output). Connector and
instruction preparation also feed the session fingerprint. That cut is
not mechanical, so this PR removes the largest measured single cost
only.

Other `heartbeat.prepare.*` costs that stay on the reconnect path (use
post-upgrade timings to choose the next cut):
- `configuration_secrets`, `configuration_connector_assignments`,
`configuration_connection_instructions`,
`configuration_connector_skills`/`_delivery`,
`configuration_agent_identity`, `configuration_ai_runtime`
- `environment_selection`, `workspace` (retained lease resume and
realization), `session`, `task_context`, `run_identity`
- `github_environment`, `github_launchers`, `run_output`,
`runtime_services`, `instructions`, `runtime_selection` (including
`runtime_selection.skill_assets` and `runtime_selection.workspace_sync`)
- The wait from detach to "run started" (queue and controller-preparing
waits) is outside `heartbeat.prepare.*`.

## Verification

- New test `reconnects to a live remote Runner with its admitted skills
without re-downloading them or waiting on the cache`
(`server/src/__tests__/heartbeat-execution-configuration.test.ts`):
- It admits a run, deletes its skill cache (like a new server), and
holds the company cache lock.
- The reconnect preparation finishes while the lock is held. It returns
the same key, runtime name and version. The session fingerprint is
identical. The cache is not created again.
- Normal preparation of the same saved selection still waits for the
lock and materializes the files ("normal start unchanged").
- Red on master source: `A live Runner reattach must not queue behind
skill downloads: expected null not to be null`. Green with this change.
- New test in `managed-skill-content.test.ts`: with the cache lock held
and an empty cache, `prepareRuntimeContextSource` for a reconnect
returns at once with the admitted context. Its staging step and a normal
start still wait for the lock and then rehydrate the same digests.
- New `server/src/services/heartbeat/live-runner-adoption.test.ts`: the
gate is false for a normal start, a scheduled resume, a dead-Runner
resume, an incomplete bootstrap, another run's claim, a different lease
owner, and a legacy run.
- `npx vitest run
src/__tests__/heartbeat-execution-configuration.test.ts
src/__tests__/skill-version-cache.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/services/heartbeat/live-runner-adoption.test.ts`: 289 passed, 14
skipped. After the controller change: restart-recovery, diskless
restart, runtime skill preservation, managed skill content, session
executor and execution configuration suites: 1686 passed, 14 skipped.
- `tsc --noEmit -p server/tsconfig.json`: pass.
- `pnpm verify` asks for broader coverage (exit 2). Required CI supplies
it.

After deploy, a reconnect run shows
`heartbeat.prepare.configuration_skills_adopted` and the
`reused_admitted` event, and no `skills.cache.acquire` or
`skills.version.materialize` phases.

## Risks

- Low. The option applies only to a same-run reconnect of a retained
remote Runner that has a saved runtime context and saved skill versions.
Every other start keeps the current path.
- Access revocation and deleted versions still fail the reconnect, as
before. A failure to download the files on this server no longer fails a
live reconnect, because the Runner already holds them.
- Nothing new is saved, so rows from older releases, two live servers,
and a deploy during a reconnect behave as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (code search, editing, shell, tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 03:58:01 -06:00
DottaandClaude Fable 5.2 8e956da1cf fix(native): keep a live reattached Runner when its restart recovery read times out (#21053)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on remote sandboxes (Daytona) use a detached Runner. A
server replacement detaches the controller and leaves the Runner alive.
The new server reattaches to it and continues the same turn
> - On staging, about half of the in-flight runs at each upgrade lost
their turn. The new controller reattached to the live Runner (same
process verified), its recovery read timed out (`PRP command
session.snapshot timed out`), and the failure cleanup killed the Runner
> - The next same-run reattachment then found `process_not_running`.
#21039 continues these runs in a fresh session, but the in-flight turn
is lost and its outcome is unknown
> - The cause of the kill is in the Codex driver's `recoverSession`: its
cleanup closed the transport without retention. For a reattached Runner
that means SIGKILL after the close deadline. #21023 already keeps the
Runner for a failed open session, but not for a failed recovery
> - This pull request keeps the live Runner when a recovery fails or is
aborted on its way to it. Identity refusals and protocol integrity
faults still stop it
> - The benefit is that a slow reattachment no longer destroys a turn
that was still running

## Linked Issues or Issue Description

Refs #21023, #20923, #21039, #21052

**What happened?**
Staging replaced its server at about 07:10Z with 17 remote native runs
in flight. For each, the old server logged "Server shutdown suspended
native controller ownership without cancelling provider work". The new
server reattached. Example run: the Runner process (same pid and start
time) was verified at 07:12:37 and the transport authenticated at
07:12:32. At 07:13:28 the recovery failed with
`native_session_recovery_failed: ... PRP command session.snapshot timed
out`, and a same-run resume was saved. At 07:14:12 the same-run
reattachment found `process_verification` / `process_not_running`, with
the Runner journal still `ready` and no exit record from the Runner
itself. The run failed with `runner_remote_process_identity_unavailable`
/ `native_continuation_requires_reconciliation`. 90 of 93 such failures
in two hours of upgrades had this sequence.

**What did you expect to happen?**
A recovery that times out against a live Runner is a transient failure.
The same-run reattachment should find the Runner alive and adopt it
again. Killing the Runner is only for a deliberate stop, an identity
refusal, or an integrity fault.

**Root cause**

`packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts`
`recoverSession`: the `catch` block and the abort handler call
`bootstrapCancellation(...).close()`, which calls `transport.close()`
without `retainUnsettledRemoteRunner`. In `runnerd-codex-transport.ts`
`#closeOnce`, an unresponsive adopted Runner that is not retained gets
`SIGKILL` after the close deadline. The Codex driver serves remote
Codex, OpenCode and ACPX runs, which matches the failures seen on
staging for all three driver kinds.

## What Changed

- `codex-app-server-driver-impl.ts`: `recoverSession` classifies a
thrown recovery failure. A transient one (`provider_transport_failed`, a
`PRP command ... timed out`, or the transport closing while waiting) now
detaches this controller with the transport's restart detach
(`detachControllerForRestart`). That path queues no `turn.stop` and no
`runner.suspend` and sends no signal, the same as a server restart. The
Runner and its in-flight turn stay for the same run's next reattachment.
An abort (a Stop or the recovery deadline), any message that names a
mismatch, a different session, identity, authority, integrity or
incompatibility, any unclassified failure, any returned identity
refusal, and a protocol integrity fault still close and stop the Runner.
`openSession` is unchanged.
- `codex-app-server-driver.recovery.test.ts`: the staging message
(`codex_history_read_failed: ... provider_transport_failed: PRP command
session.snapshot timed out`), and a plain snapshot timeout now detach
and never close. An abort of a hung read still closes and stops the
Runner. A thrown different-session refusal,
`native_adopted_provider_identity_mismatch`, an identity refusal that
also mentions a timeout, and an unclassified failure still close without
retention. The existing integrity-fault cases check that the close does
not retain the Runner.
- `doc/execution-semantics.md`: describes the rule.

## Verification

- Red on `origin/master` with only the new tests: `cd
packages/paperclip-runner && npx vitest run
src/drivers/codex/codex-app-server-driver.recovery.test.ts` fails 2 of
69 (the two detach cases; master calls `close()` and stops the Runner).
- Green with the fix: 69 of 69.
- `npx vitest run src/drivers/codex src/native-session-runtime.test.ts
src/backends src/live/runnerd-codex-transport-remote-retention.test.ts
--maxWorkers=4` passed except two timing tests (Codex command
resolution, background cleanup containment) that pass alone and do not
touch this code.
- `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` passes.
- `detachControllerForRestart` is the existing restart path. It stops
the pump, disconnects the Runner, and drains the controller journal. It
queues no command and never signals the process.
- Not run: a live Daytona restart. The next post-upgrade audit should
show fewer `runner_remote_process_identity_unavailable` failures after a
`session.snapshot timed out` resume. Instead, the run log should show
"controller authority detached for restart; durable runner left alive"
and then a successful same-run reattachment.

## Risks

- Medium. A Runner that the recovery could not reach now stays alive,
with its turn, for the same-run reattachment. This is the state a server
restart already leaves. The same-run reattachment adopts it, or a Stop
cleanup or saved-work recovery stops it.
- If the Runner stays unresponsive across every same-run attempt, the
run no longer gets the `process_not_running` proof that #21039 uses for
its automatic fresh-session continuation. The run then keeps its visible
Retry. The staging data shows that reattachments reaching the Runner
sooner succeed, so a later attempt is expected to succeed. This is not
proven against a live sandbox.
- An abort of the recovery (a Stop or the recovery deadline) is
unchanged: it closes and stops the Runner.
`plugin-runner-release-resume.integration.test.ts` (14 of 14) covers a
Stop during an adopted resume.
- No turn is replayed and no second Runner is launched. Adopting an
existing Runner never launches a new one.
- Previous-release story: no persisted state changes. The decision is
made in the process that runs the recovery, so a deploy during a run
changes nothing.
- No wait is added before a normal turn start. Only the recovery failure
path changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic) via Claude Code, with tool use (shell, file edits,
test runs, read-only staging API reads). Model ID as attributed in the
commit trailer: Claude Fable 5.2.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 02:59:46 -06:00
DottaandClaude Opus 5.5 424e2c6488 fix(recovery): keep live owners' runs through an upgrade and continue replaced preparations in the restart lane (#21046)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An upgrade replaces the server. The old and the new server serve
together for several minutes. A legacy run proves that its owner is
alive by renewing a 60-second lease on its `heartbeat_runs` row.
> - #21019 added an uncharged `server_restart` retry lane for runs that
a restart or replacement ended. On the first upgrade that shipped it, no
run used that lane. The new server also failed many runs as
`process_lost` while the old server that owned them was still serving.
> - Each of these runs then spent one failure retry. A run that was
already the last automatic retry was held for manual reconciliation, so
the task needed a person.
> - An upgrade is routine. It must not cost a task its work or a retry,
and it must not leave a task waiting for a manual Retry.
> - This pull request sends a run that a replacement stopped during
preparation to the `server_restart` lane. It also stops the reaper from
failing a run while the server process that owns it is proven alive.
> - The benefit is that a task continues by itself through an upgrade,
also when the database is slow during the swap.

## Linked Issues or Issue Description

Refs #21019, #21042

**What happened?**
A hosted staging server was replaced by an automatic upgrade. The new
server started at 06:18:46Z, became ready at 06:26:17Z, and the old
server received SIGTERM at 06:27:34Z. Between 06:23Z and 06:26Z the new
server failed 44 runs as `process_lost` ("Process lost -- server may
have restarted; retrying once",
`processLossDiagnostic.controllerReplaced: true`). The old server still
owned these runs and was still serving. Every reaped run was a Runner
run in `executionStage: preparing` with no process recorded. Each
successor was scheduled with `scheduledRetryReason: transient_failure`
and `executionRetryAccounting.failureRetries: 1`. No run on the server
has `retryReason: server_restart` or
`executionRetryAccounting.serverRestarts`.

The run evidence also shows why the leases lapsed. The old server's
lease renewals waited 19 to 50 seconds on the database
(`controllerLeaseLoss.renewalPendingMs`) while its timers fired on time
(`deadlineLateMs` 0 to 17 ms). #21042 fixes that renewal stall. This
pull request fixes how the reaper and the retry scheduler treat these
runs.

**Expected behavior**
- A server never fails a run as lost while the server process that owns
it is alive and serving.
- A run that a replacement really ended continues once, automatically,
without spending the failure budget.

**Steps to reproduce**
1. Seed a legacy `paperclip_runner` run in `executionStage: preparing`,
owned by another server boot, with a lapsed run lease.
2. Run `reapOrphanedRuns()` on the new server.
3. See that the successor is `transient_failure` with `failureRetries:
1`. When the chain already spent its failure budget, see that no
successor is created and the run is held for reconciliation.
4. Give the owner a live controller instance lease (the old server is
still serving) and run `reapOrphanedRuns()` again: the run is failed as
`process_lost` anyway.

**Root cause**
- `usesServerRestartLane` (`server/src/services/heartbeat/retries.ts`)
left every run with the bootstrap evidence `providerWorkStarted: false`
to the `interrupted_preparation` lane. But `scheduleBoundedRetryForRun`
gives that lane only to an `interrupted` run (or a native preparation).
The reaper writes the same evidence on a `failed`/`process_lost` run
that it stops during preparation. That run fell through to the charged
`transient_failure` lane. The #21019 tests used runs in `dispatching`,
so they did not catch it.
- `legacyExecutionNeedsReconciliation` held every run whose failure
budget was spent before it checked the bootstrap evidence. A run that a
replacement stopped before dispatch did no provider work, but it was
held for reconciliation, and the task waited for a person.
- The orphan reaper (`recoverActiveExecutions`,
`server/src/services/heartbeat/recovery.ts`) treated a lapsed run lease
as proof that the owner was gone. It checked the shared owner prover
only for "gone", never for "alive". During the swap the old server's
renewals were slow, so its leases lapsed while it was alive.

**Paperclip version or commit**
Staging Core `51d77648bc` (first build with #21019). The behavior is the
same on master `148d2b629c`.

## What Changed

- `retries.ts`: `usesServerRestartLane(run, preparationLaneTakesRun?)`
keeps the `interrupted_preparation` exception only when that lane really
takes the run. `scheduleBoundedRetryForRun` passes its own verdict.
Without one, only an `interrupted` run qualifies. A `process_lost` run
that a replacement stopped during preparation now continues in the
`server_restart` lane.
- `legacy-execution-recovery.ts`: a replaced preparation
(`isReplacedPreparation`) does not need reconciliation, also when its
failure budget is spent. The `server_restart` bound limits it.
- `recovery.ts` (orphan reaper):
- When a run's lease lapsed but the shared owner prover proves its owner
process alive (for example, the owner's controller instance lease is
live), the reaper leaves the run with that owner. The owner stops and
settles the run itself (`controller_lease_lost`, uncharged
`interrupted_preparation` lane). The reaper recovers the run once the
owner is gone or stopped. It also recovers the run when the owner has
not settled it within `LEGACY_LIVE_OWNER_SETTLE_GRACE_MS` (5 minutes)
after the lease lapsed.
- A setup that a replacement stopped before dispatch uses the
`server_restart` bound for its retry decision, whatever the adapter.
- `legacy-controller-lease.ts`: add `proveLegacyController(run)`, which
returns the shared prover's verdict. `legacyControllerProvenGone` uses
it, and the reaper computes the proof once per run.
- `server-replacement-interruption.ts`: add
`isReplacedPreparation(run)`: bootstrap evidence `providerWorkStarted:
false`, a recorded controller process stamp, and a server-replacement
interruption. A row saved by an older release without the stamp keeps
the charged lane (the existing documented rule).
- `legacy-execution-recovery.ts` (`settleInterruptedNativeBootstrap`): a
reconciliation hold that an older release saved on a replaced
preparation is settled once its owner is proven gone, even when the
failure budget is spent. The run continues in the `server_restart` lane.
- `legacy-controller-lease.ts`, `recovery/ownership-registry.ts`: the
`legacy_interrupted_preparation_hold` kind judges a finished run's hold
by its recorded controller process
(`finishedLegacyControllerOwnerStamp`). Revocation by the orphan reaper
swaps in a random boot id, so before this change a reaped preparation's
hold had owner "none" and was never settled.
- `recovery/service.ts`, `heartbeat.ts`: the stranded-task sweep reports
exhaustion from the scheduler's own `retry_exhausted` result, so a spent
`interrupted_preparation` lane blocks the task even when the synchronous
budget check reads an unspent restart bound.
- `doc/execution-semantics.md`: document these rules.
- No schema change. Nothing changes on the path that starts a run or a
turn (no new waits and no file work).

## Verification

- Red on `origin/master`, green on this branch
(`server/src/__tests__/heartbeat-process-recovery.test.ts`, block "runs
ended by a server restart or replacement"):
- "continues a first attempt / a run that was already the last automatic
retry that a replaced server left preparing, in the restart lane". This
is the production shape: a `paperclip_runner` run in `preparing`, two
servers sweep at the same time. Master: `transient_failure` with
`failureRetries: 1`, and no successor for the spent chain. Now: one
`server_restart` continuation, due at once, with the failure count
unchanged.
- "leaves a preparing/dispatching run whose lease lapsed with its owner,
then recovers it once the owner stops". The owner's controller instance
lease is live and two servers sweep. Master: `process_lost` at once.
Now: the run stays `running` with its owner. After the owner stops, it
becomes `process_lost` and a `server_restart` continuation is scheduled.
- "recovers the run when its live owner has not settled it within the
grace".
- `interrupted-preparation-hold-settle.test.ts`: "releases an older
release's hold on a preparation a server replacement stopped with its
failure budget spent" for an unrevoked and a reaper-revoked lease (red
without the settlement and owner-stamp changes) and "blocks the task
when a replaced preparation's lane is spent even though the restart
bound is not" (red without the sweep change).
- Related suites: heartbeat process recovery, retry scheduling, orphaned
active lease sweep, recovery deadline (bootstrap and ordinary
bootstrap), legacy continuation authority, undispatched execution wait,
agent status recovery, legacy workspace restore recovery, legacy
execution recovery, legacy controller lease, server replacement
interruption, heartbeat recovery, provider failure classification,
reconciliation run read, and wake-queue policy.
- `tsc --noEmit` for `server`.

## Risks

- A run whose owner is alive but no longer drives it now stays `running`
for up to 5 minutes after its lease lapsed, and then the reaper recovers
it. The owner's own sweep still recovers a run that the owner itself
lost the handle for (this boot is never deferred).
- The guard trusts the shared owner prover. That prover reads "alive"
only from a live or grace controller instance lease, or from a live PID
in the same PID space. A server whose instance lease cannot renew for 60
seconds stops itself, so a stuck owner cannot hold runs forever.
- More runs continue in the uncharged lanes. The `server_restart` bound
(6 in 60 minutes) still stops a loop, and the stop is shown on the task.
- Rows saved by older releases: a `process_lost` row with the bootstrap
evidence and an old-boot owner now continues in the `server_restart`
lane instead of being held. These runs did no provider work.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution, read-only log and API inspection) in
Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 02:18:39 -06:00
DottaandClaude Fable 5.2 fd5ee7e385 fix(native): continue a routine assignment or chat message whose remote Runner an upgrade lost mid-turn (#21052)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on remote sandboxes (Daytona) use a detached Runner.
When the server is replaced during an upgrade, the new server reattaches
to that Runner
> - On staging, upgrades every 15–40 minutes lose the in-flight Runner.
The same-run reattachment then finds `process_verification` /
`process_not_running`, the run fails with
`runner_remote_process_identity_unavailable`, and the task is blocked
> - #21039 made these runs continue once in a fresh session (the turn is
recorded as having an unknown outcome and is not replayed). It covers
user wakes and automatic retries (`retryOfRunId`) only
> - A routine's system assignment (`issue_assigned`, requested by the
system) and an external chat message wake have no `retryOfRunId`. They
got `not_user_request` and stayed blocked for a person
> - This pull request treats those two wakes as owed task work, so they
continue the same way once the Runner is proven gone
> - The benefit is that an upgrade no longer leaves these tasks waiting
for a Retry click

## Linked Issues or Issue Description

Refs #21039, #21023, #20923

**What happened?**
Staging replaced its server at about 07:10Z. 17 remote native runs were
in flight. Each new controller reached its Runner 114–152 s after
detach. Recovery failed with "PRP command session.snapshot timed out".
The same-run reattachment then found the Runner process gone, and each
run failed with `native_continuation_requires_reconciliation`. #21039
continued 14 of the 17 automatically. The other 3 were a routine's
assignment (two runs) and an external chat message. They have no
`retryOfRunId`, so `requestInterruptedNativeTurnRetry` returned
`not_user_request`. Those tasks are still blocked with no retry.

**What did you expect to happen?**
The Runner is proven gone, so nothing can still run the turn. The run
should continue once in a fresh session, the same as an automatic retry
of owed work.

**Steps to reproduce**
1. Start a remote native run from a routine assignment, or from an
external chat message.
2. Replace the server while the provider turn is active, so that the
reattachment finds `process_not_running`.
3. The run fails and the task stays blocked. No automatic Retry is
requested.

## What Changed

- `server/src/services/heartbeat/restart-interrupted-turn.ts`: after the
Runner is proven gone, a system wake also continues when it is a system
assignment (`issue_assigned`), or when it is a comment or external chat
message wake whose message is still on the task (not deleted, not
written by a run). It runs under the run's responsible user, at most
once per chain of runs, like an automatic retry already does. A linked
person's external chat message continues under that person when the
message is still on the task. Monitors and other system wakes are
unchanged.
- The same file and `server/src/app.ts`: the hook now passes the chat
service's Retry callback (`prepareFailedChatRunRetry`, registered at app
startup next to the budget-resume callback). A task in a connected
conversation whose endpoint restricts external execution then continues
through the chat Retry path. Before, `requestTaskRetry` refused it
(`retry_refused`) and the task stayed blocked.
- `server/src/services/runner-lost-mid-turn-continuation.test.ts`:
production-shape cases for a system assignment, an unlinked external
chat sender, and a linked person's chat message. Negative cases: a
deleted message, a message written by a run, a deleted linked message,
and a system assignment whose Runner was not proven gone.
- `doc/execution-semantics.md`: describes the rule.

## Verification

- Red on `origin/master` with only the new tests: `cd server && npx
vitest run src/services/runner-lost-mid-turn-continuation.test.ts` fails
4 of 16. The three new continuation cases return `{ requested: false }`.
The deleted linked-message case returns `not_user_request` instead of
`message_unavailable`.
- Green with the fix: 16 of 16. Two more cases bind the task to a real
restricted Slack endpoint and conversation. With the chat callback
registered, the run continues and the Retry carries the chat action (18
of 18). Without the callback, the result is `retry_refused`, which was
the behavior before for every connected conversation.
- Related suites pass: `runner-lost-mid-turn-continuation.test.ts`,
`heartbeat-native-remote-failure-retention.test.ts`,
`workspace-recovery-retry-copyback.test.ts`,
`governed-wait-runner-loss.test.ts` (116 tests).
- `cd server && npx tsc --noEmit` passes.
- Production evidence was read from staging run events through the
read-only API: 3 of 17 losses at the 07:10Z swap took this path. After
this change the next post-upgrade audit should show an automatic
`task_retry` for every `native_continuation_requires_reconciliation` run
whose refusal is `process_not_running`, including routine assignments
and chat messages.

## Risks

- Low. The change applies only after the Runner process is proven gone
(`process_verification` / `process_not_running`), so no Runner can still
run the turn. The turn is not replayed. The fresh session continues over
the saved work, as in #21039.
- The task can contain partial tool work from the lost turn. This is the
same accepted risk as #21039.
- Previous-release story: no persisted shape changes. The rule reads the
wake's reason and comment id, which earlier releases already write. A
run an older release already settled keeps its visible Retry.
- This does not stop the Runner loss itself. That is a separate fix.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic) via Claude Code, with tool use (shell, file edits,
test runs, read-only staging API reads). Model ID as attributed in the
commit trailer: Claude Fable 5.2.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 02:11:28 -06:00
DottaandClaude Opus 5.5 3c058c37e6 fix(retry): continue a connected task from saved work when its exact chat request cannot be repeated (#21038)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Tasks that come from a connected chat (for example GitHub pull
request reviews) show a Retry button when a run fails
> - The task projection offers Retry, but the Retry route sends a
connected task to the chat retry owner, which only replays the exact
failed chat request
> - When that exact request cannot be replayed (the attempt already
saved a result, its output was published, the latest run did not start
from a chat message, or the requester's access changed), the route
returns 409 and tells the person to check the connection by hand
> - A Retry button that is shown must work, and recovery must not ask
the person to inspect anything
> - This pull request makes a person's Retry continue the task from its
saved work when the exact chat replay is refused, the same way a new
message on the task would
> - The benefit is that the projection and the route agree, and blocked
connected tasks become retryable after deploy with no migration

## Linked Issues or Issue Description

No public issue exists. This is the bug description, in the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

On a hosted deploy, five blocked tasks that came from a connected GitHub
conversation showed Retry (`interruption.canRetry: true`). Each click on
Retry returned 409 `chat_failed_run_retry_not_authorized`. The message
was "The exact failed chat request can no longer be retried safely.
Check its current connection, access, and recovery state." or "This
attempt already recorded a result. Send a new message on the task to
continue from the saved work." The latest runs were legacy sandbox setup
failures, native runs that needed reconciliation, a run that started
from an issue monitor instead of a chat message, and a native run that
already saved its result. Root cause: `admitTaskRetry` in
`server/src/services/task-retry.ts` called the chat retry owner for
every connected task and passed its exact-replay refusal straight to the
person. The projection cannot predict that refusal.

**Expected behavior**

A shown Retry works. If the exact chat request cannot be repeated, Retry
continues the task from its saved work, the same way a new message on
the task would. Users do not get operator instructions.

**Steps to reproduce**

1. Bind a task to a restricted chat conversation.
2. Make its latest run fail so that the chat retry owner refuses the
exact replay. For example, use a run whose source is not a chat message,
or a native run with an accepted result.
3. Click Retry (`POST /api/issues/:id/retry`).
4. The route returns 409 `chat_failed_run_retry_not_authorized`.

**Paperclip version or commit**

`master` at 5a7378af5 (the deployed revision), reproduced on 51d77648bc.

**Deployment mode**

Hosted (cloud) instance with Daytona remote sandboxes and a GitHub chat
connection.

**Agent adapter(s) involved**

Not adapter-specific (core bug). Seen with `paperclip_runner` legacy and
native runs.

## What Changed

- `server/src/services/task-retry.ts`: a person's Retry runs the exact
chat replay inside a savepoint. On a
`chat_failed_run_retry_not_authorized` refusal, the savepoint rolls back
and the same Retry receipt stays an ordinary board continuation. The
clicking person is the responsible user, the run starts a fresh session
(`forceFreshSession`) from saved work, and no chat action, chat
authority, or output replay is created. Retry that Paperclip requests by
itself (`initiatedBy: restart_recovery | provider_limit_reset`) keeps
the exact-replay requirement. Other errors (403, lock contention,
database errors) still propagate.
- `server/src/services/chat-channels/failed-run-retry.ts`: plain refusal
wording. Runs saved by an earlier release with the old diagnostic are
still recognized by the pre-provider check.
- Tests: new red/green cases in `server/src/services/task-retry.test.ts`
(four refusal reasons on a chat task whose latest run came from an issue
monitor, plus automatic Retry still refused). Updated two real-service
route cases in `server/src/__tests__/chat-channels.integration.test.ts`
(accepted native result on an inherited GitHub task, and an unknown chat
source) from expecting 409 to expecting a board continuation without a
chat replay.
- A person's Retry on that person's own saved board turn (`source:
issue.comment.reopen`) that cannot be certified for replay also
continues from saved work, without replaying the saved message.
Automatic Retry still gets 409 `saved_board_message_not_replayable`.
- A person's Retry when no chat retry owner is wired also continues from
saved work. Automatic Retry still gets 409.
- The fallback is not used while the conversation holds saved or
unadmitted chat input for the task. That input keeps its own admission
path and the existing refusal.
- Docs: `doc/execution-semantics.md` and `doc/connections/GITHUB.md`.

### 409 paths in the task Retry route after this change

| Path | Who sees it | Result |
| --- | --- | --- |
| `chat_failed_run_retry_not_authorized` from the chat retry owner |
Person | Continues from saved work (board continuation) |
| `saved_board_message_not_replayable` | Person | Continues from saved
work |
| No chat retry owner wired | Person | Continues from saved work |
| Any of the three above while saved or unadmitted chat input is pending
| Person | 409. The pending input continues through its own admission
and delivery. |
| Any of the three above | Automatic Retry (restart recovery, provider
limit reset) | 409, recorded by the hook. No button. |
| `task_retry_busy` (chat authorization lock held) | Person | 409 with
`retryable: true` after 10 internal attempts. A new click works. |
| Idempotency key reused for a different task or user | Client error |
409. The UI makes a new key for each click. |

## Verification

- Red before the fix: `npx vitest run src/services/task-retry.test.ts -t
"connected task"` failed 4/4 with `Error: refused` (409). The board-turn
and missing-owner cases failed with "This saved message cannot be
retried safely..." and "Retry is unavailable for this connected
conversation." `src/__tests__/chat-channels.integration.test.ts -t
"continues an inherited GitHub task"` failed with `expected 202
"Accepted", got 409 "Conflict"`.
- Green after the fix: `src/services/task-retry.test.ts`,
`src/__tests__/durable-chat-wakeup.test.ts`,
`src/services/budget-resume-continuation.test.ts`: 160/160 passed.
- Full `src/__tests__/chat-channels.integration.test.ts`: 1949/1951
passed. The 2 local failures (`binds Slack tools...`, `materializes
direct external-chat finals...`) read `.claude/skills/slack/SKILL.md`
from the parent checkout of a nested local worktree (ENOENT). They do
not touch Retry and depend on the local checkout layout.
- `npx tsc --noEmit -p server/tsconfig.json`: passed.
- `pnpm verify` exited 2 (a doc file is a global CI input), so required
CI supplies broad coverage.
- Staging was inspected with GET requests only. No POST was sent to
staging.

## Risks

- Behavior change: a person's Retry on a connected task can now start a
board continuation where it was refused before. It uses the same
authority as a new message from that person, and dispatch keeps all
normal gates (task access, executor containment, recovery holds,
unadmitted chat input).
- The continuation does not publish the earlier result again and does
not reuse the chat requester's authority.
- No migration. Existing blocked tasks become retryable at the next
click.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 01:27:54 -06:00
DottaandClaude Opus 5.5 aa94939988 fix(native): continue a run whose remote Runner died mid-turn after a restart, without replaying the turn (#21039)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) use a detached
`paperclip-runnerd`. After a server restart, recovery reattaches to that
Runner on the run's own sandbox lease
> - On staging, a deploy detached the controller and left the Runner
alive. The restart's reattachment then found the Runner process gone
with a provider turn in flight. The run failed with
`runner_remote_process_identity_unavailable` and the task waited for a
person
> - The run log said `startup reuse: environment (fresh) created`, so
the sandbox looked re-created. It was not. The reattachment resumed the
exact retained lease. The receipt reported the lease's original creation
> - `requestInterruptedNativeTurnRetry` refused every unsettled turn
(`provider_turn_unsettled`), and refused every system wake unless the
journal proved that no turn was issued. So this run got no automatic
continuation
> - This pull request continues such a run once in a fresh provider
session when the refused reattachment proved the Runner process gone. It
records that the turn's outcome is unknown and never replays the turn.
It also makes the environment receipt tell the truth
> - The benefit is that a restart that kills a Runner in the middle of a
turn no longer needs a Retry click

## Linked Issues or Issue Description

Refs #20853, #20923, #21023, #20278, #20627, #20828

**What happened?**
A staging deploy landed in the middle of a remote native Codex run on
Daytona. Shutdown detached the controller and logged "durable runner
left alive". The new server reattached through the run's original lease
(same lease row, same provider sandbox id, same directory). Its process
check then found the Runner gone: `process_verification` /
`process_not_running`, `runnerLifecycle: ready`, `runnerSettled: true`,
last diagnostic "WebSocket peer closed the connection", empty stderr.
The provider checkpoint still had an active turn. The run failed with
`runner_remote_process_identity_unavailable` /
`native_continuation_requires_reconciliation`, and the task went to
`blocked`.

**Root cause**
- `server/src/services/heartbeat/restart-interrupted-turn.ts`
(`requestInterruptedNativeTurnRetry`): an unsettled last turn always
returned `provider_turn_unsettled`, also when the refusal proved the
Runner process gone. A system wake (here `transient_failure_retry`)
continued only with `providerTurnIssued: false`.
- The sandbox was not re-created. The environment and lease listings
show one lease for the run, never replaced, and the final workspace
snapshot used the same provider sandbox. `environmentStartupReuse` read
`sandboxLeaseAcquisition: created` from the lease's first acquisition,
so the reattachment's receipt said `fresh / created`. The 46 s acquire
was the provider resume of that exact lease. The evidence available to
GET requests cannot say what stopped the Runner process (the resume
result is not kept). This pull request does not change that path.
- #21023 named this run as not covered. #20853 resumes a Runner only at
a settled, idle `ready` cut. Here the provider checkpoint has an active
turn, so #20853 stays fail-closed, as it should.

**Expected behavior**
The run continues once by itself in a fresh provider session over the
saved work. The turn whose outcome is unknown is recorded and is not
replayed.

## What Changed

- `restart-interrupted-turn.ts`: read the last
`native.remote_runner_recovery_failed` refusal once. When it proves the
Runner process gone, an unsettled last turn no longer blocks the
automatic Retry. Before it requests the Retry, the server records
`native.interrupted_turn_outcome_unknown` once (turn id, last turn
event, `replayed: false`, closed `runnerExit` fields). The Retry is the
existing one: saved-work recovery and stop proof first, then a fresh
provider session (`forceFreshSession: true`), once per chain.
- The same file: a system wake whose run is an automatic retry of owed
work (`retryOfRunId` set) also continues under the run's responsible
user once the refusal proved the Runner gone. Other system wakes (for
example a monitor) keep their policy.
- `packages/adapter-utils/src/startup-reuse.ts` and
`native-session-executor.ts`: a `reattach_remote_runner` recovery
reports its `environment` receipt as `reused / verified_reuse`. That
path only resumes the run's exact retained lease and never acquires one.
- Docs: `doc/execution-semantics.md` and `doc/run-log-events.md`.

Previous-release story: no persisted shape changes. The rule reads the
closed `remoteRunnerRecovery` fields that the previous release already
writes; the new test uses the staging refusal payload as written. A run
that a previous release already failed this way continues on the next
completion or restart hook call only; runs already settled stay as they
are and keep their Retry. The new event is additive.

## Verification

- Red on `origin/master` (51d77648bc), with only the new tests: `cd
server && npx vitest run
src/services/runner-lost-mid-turn-continuation.test.ts` fails 4 of 6.
The three continuation cases return `{ requested: false, reason:
"provider_turn_unsettled" }`, and the monitor case returns that reason
instead of `not_user_request`.
- Green with the fix: 6 of 6. The cases cover a user's assignment
through the restart claim, the staging shape (system
`transient_failure_retry` with `retryOfRunId`), and a same-run retry.
Each one checks one recorded unknown-outcome event, an idempotent second
call, saved-work recovery first, one queued successor with
`forceFreshSession: true`, and one Retry intent. Negative cases: a
refusal that did not prove the process gone, no refusal, and a monitor
wake. Each keeps the visible Retry and records nothing.
- `cd packages/adapter-utils && npx vitest run
src/startup-reuse.test.ts`: the new receipt case fails before (`fresh /
created`) and passes after.
- Focused suites pass: `workspace-recovery-retry-copyback.test.ts`,
`governed-wait-runner-loss.test.ts`,
`heartbeat-native-remote-failure-retention.test.ts`,
`native-runner-restart-recovery.integration.test.ts` (302 passed, 14
skipped), and `native-session-executor.test.ts -t "startup
reuse|restart|reattach"` (149 passed).
- `pnpm --filter @paperclipai/server typecheck` and `pnpm --filter
@paperclipai/adapter-utils typecheck` pass.
- `pnpm verify` exits 2 because `doc/execution-semantics.md` is a global
CI input. CI supplies that coverage.
- Not run: a live Daytona restart. The change is in the server's
recovery decision; the tests use the fixture's sandbox confinement.

## Risks

- Medium. A run whose Runner died in the middle of a turn now continues
by itself. The turn's outcome is unknown, so tool work from that turn
can be partly done. The successor is a fresh session over the saved
workspace; it does not replay the turn. Saved-work recovery stops and
keeps the sandbox before the successor starts, so the old provider
cannot still run.
- Without positive proof that the Runner process is gone, an in-flight
turn keeps the visible Retry, as before.
- An automatic retry of owed work now continues after a lost Runner.
Monitor and other system wakes without `retryOfRunId` do not. The bound
is still one automatic Retry per chain.
- The environment receipt changes only for the reattachment path. No
execution decision reads it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test runs, read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 00:13:25 -06:00
DottaandClaude Opus 5.5 51d77648bc fix(native): keep a remote Runner alive when a failed turn start or event stream closes the run (#21023)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) start a detached
`paperclip-runnerd`. After a recoverable failure, the server resumes the
same run and reattaches to that Runner
> - #20923 stopped a failed close from force-stopping an unreachable
remote Runner. It asks for retention only when the session was not
cancelled
> - The runtime cancels the session itself when a turn start or the
event stream fails ("Native session event consumption failed."). That
cleanup sets the same flag as a user Stop
> - So after a failed turn start, the close still sent `SIGKILL` to the
Runner, and the same-run reattachment failed with
`runner_remote_process_identity_unavailable`. The task waited for Retry
> - This pull request separates failure cleanup from Stop. Failure
cleanup still revokes the turn's output, but the close keeps the Runner
for the same-run reattachment
> - The benefit is that these runs continue without Retry. Stop and
protocol integrity faults still stop the Runner

## Linked Issues or Issue Description

Refs #20923, #20853

**What happened?**
After the staging redeploy, five remote native runs failed with error
`runner_remote_process_identity_unavailable` and error code
`native_continuation_requires_reconciliation`. In four of them, the
first attempt started after the deploy. Its `turn/start` failed with
`runnerd did not report the provider turn identity`. The run log then
showed `runner did not prove durable suspension before checkpoint` and
`runner shutdown failed: Error: Durable recovery runner timed out.` The
diagnostic `remote runner left alive unsettled for same-run
reattachment` from #20923 did not appear. One second later the Runner
exited (`runner_remote_process_exited lifecycle=ready`). The same-run
reattachment 20–40 s later found `process_verification` /
`process_not_running` with `runnerSettled: false`.

**What did you expect to happen?**
The failed close keeps the Runner alive, as #20923 intended. The
same-run reattachment adopts it, and the run continues without Retry.

**Steps to reproduce**
1. Start a remote native Codex run.
2. Make `turn/start` fail (for example the Runner does not report the
provider turn identity in time), or let the event stream fail or time
out.
3. The runtime cancels the session as cleanup. `HarnessNativeSession`
treats that as a Stop, so `close` asks for `retainUnsettledRemoteRunner:
false`, and the transport kills the Runner.

**Root cause**
`packages/paperclip-runner/src/native-session-runtime.ts`
`attemptOptionalSessionCancellation` calls `session.cancel`.
`HarnessNativeSession.cancel`
(`packages/paperclip-runner/src/backends/harness-driver-backend.ts`)
sets `#explicitlyCancelled`, and `close` used
`retainUnsettledRemoteRunner: !this.#explicitlyCancelled`.

The fifth run is different. A server shutdown detached its controller
and left the Runner alive. Then the sandbox environment was acquired
fresh (45.9 s) before the reattachment, and the Runner was gone in the
middle of a turn. This pull request does not change that path. See
Risks.

## What Changed

- `NativeSession.cancel` accepts an optional `failureCleanup` field.
- `attemptOptionalSessionCancellation` passes `failureCleanup: true`,
except for a `NativeSessionProtocolIntegrityError`.
- `HarnessNativeSession` tracks a Stop separately from output
revocation. `cancel` with `failureCleanup` revokes the turn's output
only. `cancel` without it, and `revokeTurnPublication`, record a Stop.
`close` asks for retention when no Stop was recorded and no protocol
integrity fault occurred.
- Tests: three runtime cases with the real Codex backend (turn start
fails, event stream times out, Stop then failure) and a new backend
case.
- Docs: `doc/execution-semantics.md` describes the rule.

## Verification

- Red, on `master` with only the new tests: `cd
packages/paperclip-runner && npx vitest run
src/native-session-runtime.test.ts -t "remote Runner retention after a
failed run"` fails 2 of 3 (`expected false to be true`): the close asked
to kill the Runner after the turn start failed and after the event
stream timed out. The Stop case passed.
- Green with the fix: the same command passes 3 of 3.
`src/backends/harness-driver-backend.test.ts -t "retain an unsettled"`
passes 3 of 3, including the failure-cleanup case. That case also checks
that the turn's result stays unpublished.
- `npx vitest run src/native-session-runtime.test.ts src/backends
src/drivers/codex
src/live/runnerd-codex-transport-remote-retention.test.ts
--maxWorkers=4`: 31 files, 814 tests pass.
`codex-protocol-integrity.test.ts` confirms that an integrity fault
still stops the Runner.
- `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` is
clean.
- `pnpm verify` exits 2 because `doc/execution-semantics.md` is a global
CI input. CI supplies that coverage.
- Not run: a live Daytona run. The change is in the controller library
that runs on the server, so it needs no new Runner image.

## Risks

- Low to medium. After a failed turn start or event stream, an
unreachable remote Runner now stays alive instead of being killed. This
is the state #20923 already accepted for other failed closes. The
session stays quarantined, and the same-run reattachment or a Stop
cleanup owns the Runner.
- Stop is unchanged. A server Stop calls `cancel` without
`failureCleanup`, so the close still stops the Runner, also when the
stopped run later fails. A governed wait uses `revokeTurnPublication`,
which also records a Stop.
- A protocol integrity fault needs an operator and is not resumed on the
same run, so it still stops the Runner.
- Previous-release story: this change adds and changes no persisted
state. A Runner that an older release already killed keeps its current
named refusal and Retry. A deploy during a run changes nothing, because
the decision is made in the closing process.
- Not covered: a Runner that dies with its sandbox while the controller
is detached for a restart, in the middle of a turn. Its journal is a
settled `ready` cut with an active turn. #20853 resumes only idle
`ready` cuts. This needs a separate fix.
- No wait is added before `turn.start`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test runs, read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 23:21:05 -06:00
DottaandClaude Opus 5.5 156453fe7c fix(recovery): continue runs that a server restart or replacement ended, without spending the failure budget (#21019)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A deploy replaces the server. The old server stops its running
legacy runs at SIGTERM (`server_shutdown_interrupted`). The new server
finds runs whose owner died without that drain (`process_lost`, "server
may have restarted").
> - Each of these runs got its follow-up from the bounded transient
retry. That retry spends the failure budget (2 attempts) and waits 30 to
60 seconds.
> - So the first deploy spent one failure retry. A run that was already
the last automatic retry got no follow-up at all. The task then rested
on a manual Retry.
> - A deploy is routine. An interruption by the server is not a failed
attempt, and it must never leave a task waiting for a person.
> - This pull request moves these runs to their own `server_restart`
retry lane. The lane never spends the failure budget, starts the first
continuation at once, and has its own bound.
> - The benefit is that a task continues by itself through several
deploys a day, and a real loop still stops and shows why.

## Linked Issues or Issue Description

**What happened?**
A zero-downtime deploy replaced a hosted server. In the 11 minutes
around the swap, 11 legacy runs ended with `server_shutdown_interrupted`
and 7 ended with `process_lost`. At least 7 tasks were left with no
follow-up run and no scheduled retry, so each needed a manual Retry. Two
of the stranded runs were already automatic retries.

**Expected behavior**
Each interrupted run continues once, automatically, on the surviving
server, with its original wake.

**Paperclip version or commit**
Core commit 7d9973405c. The behavior is
the same on master bf18b348c5.

**Steps to reproduce**
1. Start a legacy run with a conversation adapter (for example
`codex_local`) as the second automatic retry of its chain (failure
budget spent).
2. Send SIGTERM to the server (`drainRunningRunsForShutdown`), or let a
new server boot find the run without its owner (`reapOrphanedRuns`).
3. See that no successor run is created. The scheduler logs "Bounded
retry exhausted".

**Root cause**
- `enqueueProcessLossRetry`
(`server/src/services/heartbeat/recovery.ts`) calls
`scheduleBoundedRetryForRun`. That function put these runs in the
`transient_failure` lane, with `maxAttempts = 2` counted from
`executionRetryAccounting.failureRetries`
(`server/src/services/heartbeat/retries.ts`). A first attempt got a
charged retry 30 to 60 seconds later. A run with 2 spent failure retries
got `retry_exhausted`.
- The orphan reaper (`recoverActiveExecutions`) skipped the retry when
`failedRunRetryBudgetSpent(run)` was true. It also skipped a
conversation run that had no local child process recorded (for example a
run that was dispatching), because `shouldRetry` needed a local child, a
preparing stage, or a monitor wake.
- The stranded-task sweeper then saw the spent budget and moved the task
to `blocked`.

## What Changed

- `execution-recovery-attempt.ts`: add the `server_restart` retry lane.
It is an uncharged lane. Its counter (`serverRestarts`,
`serverRestartWindowStartedAt`) is saved in `executionRetryAccounting`,
so it is visible on each run. The bound is 6 continuations in one
60-minute window. A later interruption after the window starts a new
window.
- New `server-replacement-interruption.ts`:
`isServerReplacementInterruption(run)` names a legacy
`server_shutdown_interrupted` run, or a `process_lost` run whose owner
was another server boot. It reads the new
`processLossDiagnostic.controllerReplaced` verdict. For rows saved by an
older release, it reads the owner boot id or `runPredatesObserver`.
- `retries.ts`: `scheduleBoundedRetryForRun` uses the `server_restart`
lane for these runs (`usesServerRestartLane`). The first continuation is
due at once; later ones in the window back off like the
interrupted-preparation lane. `failedRunRetryBudgetSpent` reports the
restart bound for these runs, so the sweeper escalates only when that
bound trips. An interrupted preparation keeps its own lane while its
chain has failure budget left.
- `recovery.ts` (orphan reaper): record `controllerReplaced`. A
conversation run that a restart ended is eligible for continuation even
without a local child process, and it checks the restart bound, not the
failure budget.
- `wake-queue/adapters/postgres.ts`: carry the restart counter to
successors that the wake queue creates.
- No schema change. No change to how runs start (no new waits before a
turn).

Behavior that did not change:
- At most one successor per run: every path still uses the predecessor
row lock and the `retry_of_run_id` check in
`scheduleBoundedRetryForRun`. Two servers alive at once reuse the same
successor.
- A non-conversation adapter that reached dispatch stays held for
reconciliation, because a replay can repeat its actions.
- Native runtime runs keep their own recovery.
- A server that began shutdown already starts no queued run
(`closeProcessRunAdmission` in `server/src/index.ts`, checked in
`heartbeat/queue.ts`). Its queued and newly scheduled runs stay in the
database for the new server.

## Verification

- Red on `origin/master`, green on this branch
(`server/src/__tests__/heartbeat-process-recovery.test.ts`, block "runs
ended by a server restart or replacement"):
- Graceful shutdown, first attempt: master scheduled a charged
`transient_failure` retry. Now one immediate `server_restart`
continuation.
- Graceful shutdown, already the last retry: master scheduled nothing.
Now one continuation; the failure count stays 2.
- Owner lost at boot (`process_lost`), first attempt and
already-retried: same results, with two servers running the boot sweep
at the same time. Only one successor.
- Old server exited after it committed the interruption and before it
scheduled the continuation: the new server's sweep continues the run.
- Run saved by an older release (no `controllerReplaced`, spent failure
budget): the sweep continues it instead of moving the task to `blocked`.
- The restart bound trips after 6 in one window, records `retryReason:
server_restart` on the run, and the sweeper escalates. The same count in
a closed window continues.
- A run that failed by itself (`adapter_failed`) with a spent failure
budget still gets no retry.
- Each continuation's execution continuation reports the original wake
(`trigger.retriedReason = issue_monitor_due`).
- Unit tests:
`server/src/services/server-replacement-interruption.test.ts`.
- Six earlier tests asserted escalation after "three deploy restarts"
spent the failure budget. They now spend the restart bound instead, so
they still prove escalation when the bound trips.
- Existing test `server/src/services/heartbeat/queue.test.ts` covers
"starts no run once shutdown closed run admission".
- `pnpm verify`: see the PR checks.

## Risks

- A run that a restart ends now continues more often (up to 6 times per
hour per chain, not 2 in total). A run that itself crashes the server is
still stopped by the bound, and the stop is shown on the task.
- Tasks that an older release already moved to `blocked` stay blocked.
Tasks that are still `in_progress` or `todo` with a stranded run
continue on the next sweep.
- A `process_lost` row from an older release whose owner was the same
boot that later restarted is read as a restart. That case is a lost
in-memory handle, which is also not a failure of the agent's work.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:56:49 -06:00
DottaandClaude Opus 5.5 5cd5adfccc fix(native): link a person-owned agent blocker to the agent's open question whatever owner kind the model reports (#20969)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #20931 lets a person's answer continue a task that its agent blocked
while it waited for that answer
> - That link is recorded only when the blocker's unblock descriptor is
bound to the board
> - Codex reports such a wait as `paperclip_block` with `owner: {"kind":
"user", "name": "Board"}` and scope `task_wide`. The evidence classifier
treats only `kind: "board"` (not a PRP owner kind) as board-owned, so
the arbiter binds this person-owned blocker to the reporting agent, and
no link is recorded
> - So every Codex W2Q task still loses the answer
(`issue_not_in_progress`) and needs Retry
> - This pull request classifies a `user` owner as a person
(board-bound) and records the link from the reported owner kind, not
from the spelling of the owner
> - The benefit is that the person's answer reaches the agent whatever
owner shape the model uses

## Linked Issues or Issue Description

Refs #20931

**What happened?**
Restart-chaos wave w-20261111T230717-s5201 on master 9c56e546 (Daytona
warm, Codex, no fault): all 3 W2Q tasks (t01, t05, t09) failed. Each
continuation after the answer (runs 8864fa72, 814b2cd0, 757c1c8f) was
cancelled with `issue_not_in_progress (current status: blocked)`. Each
agent blocked with `owner: {"kind": "user", "name": "Board"}`, scope
`task_wide` (reason codes `missing_required_answer`,
`awaiting_codename_answer`). The decision was `task_wide_blocker_bound`
with the unblock owner `{agentId}`, so `questionsAwaitedByAgentBlock`
recorded no `awaitingInteractionIds`.

**What did you expect to happen?**
The answer continues the task, as it does for a board-bound blocker.

**Steps to reproduce**
1. A Codex agent asks a question; the person comments before answering.
2. The agent calls `paperclip_block` with a `user` owner and scope
`task_wide`.
3. The person answers. The continuation is cancelled with
`issue_not_in_progress`.

## What Changed

- `evidence-classifier.ts`: new `nativeBlockerOwnerKind` and an optional
`ownerKind` on the blocker assessment (`agent`, `user`, `system`,
`external`, legacy `board`, or `unspecified`). A `user` owner is now
board-owned, like `board`: a person resolves it, so it is not bound to
the reporting agent.
- `status-arbiter.ts`: both agent-reported blocker decisions
(`task_wide_blocker_bound`, `current_track_blocker_waiting`) carry
`blockerOwnerKind`.
- `blocked-question-answer.ts` and the status committer:
`questionsAwaitedByAgentBlock` records the agent's own pending,
undismissed questions when the reported owner is one a person resolves:
`user`, `external`, legacy `board`, or no recognized kind. `agent` and
`system` owners and dependency blocks
(`durable_dependency_blocker_bound`) stay excluded. A decision without
`blockerOwnerKind` keeps the previous board-bound rule. All other guards
are unchanged: asked by this agent, answered by a person, decision still
current, `statusVersion` match, guarded checkout.
- `doc/execution-semantics.md` describes the rule.

## Verification

- `server/src/__tests__/heartbeat-blocked-question-answer.test.ts` now
builds the block decision with the real `classifyNativeEvidence` and
`arbitrateNativeStatus` from the exact Codex report shape, commits it
with `commitNativeStatusDecision`, and delivers the answer through
`heartbeat.wakeup`.
- Red on master (product hunks removed, test kept): `user` task-wide
fails because the unblock owner is `{ agentId }` instead of `board`;
`external` task-wide fails with `status: cancelled, errorCode:
issue_not_in_progress`.
- Green: 10/10. `user` task-wide, `external` task-wide and `user`
current-track are delivered. `agent` and `system` owners still cancel
with `issue_not_in_progress`, as do the earlier negative cases.
- `cd server && pnpm exec vitest run
src/services/native-runtime/status-arbiter.test.ts
src/services/native-runtime/evidence-classifier.test.ts
src/services/native-runtime/historical-task-questions.test.ts
src/modules/run-dispatch`: 257 passed. `recovery-observability`,
`heartbeat-issue-liveness-escalation`, `native-completion-feedback`,
`paperclip-control-plane-port`: all passed.
- `cd server && pnpm exec tsc --noEmit -p tsconfig.json`: passes.
- Chaos: see the comment below.

## Risks

- A task-wide blocker that names a person (`user`) no longer binds its
unblock descriptor to the reporting agent and no longer wakes that agent
about its own blocker. The board owns it, as for a current-track
blocker.
- An `external` owner can now link to the agent's open question. If that
blocker was about something else, the answer resumes the agent and it
reports the blocker again.
- Upgrade: `ownerKind` and `blockerOwnerKind` are optional. A decision
built by an older release keeps the previous rule. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in an agent harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 18:50:00 -06:00
DottaandClaude Opus 5.5 d1e4109df1 fix(runner): keep a connected adopted Runner through a failed liveness probe (#20965)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - After a controller restart, the controller adopts the remote Runner
that kept running in the sandbox. A monitor then checks the Runner's
liveness every 250 ms through the environment plugin.
> - #20940 and #20945 made a stalled plugin probe "unknown" for the
launched Runner's monitor, reattachment and launch. The adopted Runner's
monitor still failed the run on the first probe that failed without an
answer.
> - A short plugin stall after adoption therefore failed a healthy run
with `native_adopted_runner_identity_unverifiable`.
> - This pull request uses the evidence that is available: while the
adopted Runner holds its authenticated connection to the controller, a
liveness check that fails without an answer does not fail the run, for
at most the reconnect grace.
> - The benefit is that a healthy adopted Runner survives a short plugin
stall. A check that reports the process gone still fails the run at
once.

## Linked Issues or Issue Description

Refs #20940, #20945

**What happened?**
`#watchAdoptedRunner` in
`packages/paperclip-runner/src/live/runnerd-codex-transport.ts` called
`failTransport("native_adopted_runner_identity_unverifiable: runner
liveness could not be revalidated")` on the first `isAlive()` rejection.
For a remote adopted Runner, `isAlive()` is a plugin
`environmentExecute` RPC. A plugin stall makes it reject, even while the
Runner holds its authenticated connection and works normally.

**Expected behavior**
A rejection is not an answer. While the Runner holds its authenticated
connection, the monitor keeps checking. The bound is the reconnect
grace. A check that returns `false` (the process is gone) still fails
the run at once.

**Steps to reproduce**
1. Adopt a remote Runner after a controller restart.
2. Stall the plugin worker, so that the liveness probe rejects.
3. Within 250 ms the run fails with
`native_adopted_runner_identity_unverifiable`.

**Paperclip version or commit**
master at ef755cecc6.

## What Changed

- `#watchAdoptedRunner`: on an `isAlive()` rejection, the monitor
records when verification was first lost. It does not fail the transport
while the control plane has an authenticated Runner connection and less
than `runnerReconnectGraceMs` has passed. A successful check resets the
clock.
- Unchanged: an `isAlive()` result of `false` fails the run at once. A
rejection with no Runner connection fails it as before. A transport with
no reconnect grace (local adoption) fails it as before. The close path
still uses `isAlive()` directly.
- `doc/execution-semantics.md` describes the rule.

## Verification

- Red on master: `npx vitest run
src/live/runnerd-codex-transport-adoption.test.ts -t "liveness probe
rejects while"` (in `packages/paperclip-runner/`). It failed with
`native_adopted_runner_identity_unverifiable: runner liveness could not
be revalidated`.
- Green: the test passes. The real adopted runnerd keeps its connection,
`isAlive()` rejects for 1.5 seconds (6 monitor ticks), the run still
answers `thread/read`, no signal is sent, and the runner is alive.
- `runnerd-codex-transport-adoption.test.ts` passes (24 tests),
including "retains authenticated adopted-runner ownership when a close
probe rejects". `tsc --noEmit` passes for `packages/paperclip-runner`.

## Risks

- A remote adopted Runner whose liveness cannot be verified, but which
stays connected, keeps running for up to 7 minutes before the run fails
closed. Its own authenticated connection is the evidence during that
time, and no exit is reported.
- The change touches `packages/paperclip-runner/src`, so it changes the
Daytona runner image content ID.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, code search). Extended reasoning was
on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 17:49:13 -06:00
DottaandClaude Opus 5.5 827bd21017 fix(recovery): retry lost-lease preparations instead of holding them for reconciliation (#20936)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Run recovery restarts work that stops for infrastructure reasons,
and keeps saved files safe
> - A legacy Runner run can lose its controller lease while it prepares
the workspace, before any provider work starts (`controller_lease_lost`
with `executionRecovery: { kind: "bootstrap", providerWorkStarted: false
}`)
> - Such a run has no unsaved provider state, but recovery still blocked
the task: either a saved-files hold that ends in `source_unavailable`,
or an immediate "No live execution path" block after one failed
continuation retry
> - Recovery must work without a person, and a shown Retry must work
> - This pull request lets lost-lease preparations use the existing
interrupted-preparation settlement, and lets the bounded stranded-task
sweep retry a continuation that failed before provider work
> - The benefit is that these tasks retry automatically, and tasks that
an older release already blocked recover after deploy

## Linked Issues or Issue Description

Refs #20924 (fixes the lease renewal starvation that caused the lost
leases; this PR fixes what recovery did afterwards).

**What happened?**
A staging burst of lost controller leases stopped many legacy Runner
runs while they prepared the workspace. No provider work started. Most
of the affected tasks then became blocked. One shape kept a saved-files
hold (`legacy_execution_requires_reconciliation`). Its file recovery
ended `exhausted` with `source_unavailable`, because the workspace had
already retired the prepared epoch back to the last save. The hold never
settled. The other shape was a continuation retry that also lost its
lease. The release path blocked the task at once with "No live execution
path", although `controller_lease_lost` is a transient cause with a
3-attempt budget in the stranded-task sweep.

**Expected behavior**
A run that fails before provider work starts retries automatically,
within a bounded budget. It never needs reconciliation. If the prepared
workspace is gone, the task continues from the last committed save.

**Steps to reproduce**
1. Start a legacy Runner run with a remote workspace.
2. Make the controller lose its lease while the run prepares (before
provider dispatch).
3. Let restore recovery record a saved-files hold, and let the workspace
retire the prepared epoch.
4. See that the hold stays `resolved/blocked` with `source_unavailable`,
and the task stays blocked.

**Paperclip version or commit**
master at c1edb9ec02.

**Agent adapter(s) involved**
Not adapter-specific (core bug). Seen with `paperclip_runner` (Daytona
sandbox).

## What Changed

- `server/src/services/cancelled-native-startup.ts`:
`isInterruptedNativePreparationCandidate` accepts a failed
`controller_lease_lost` run with the bootstrap certificate
(`providerWorkStarted: false`), the same as a certified `process_lost`
run. The error label alone still grants nothing. This lets the existing
`legacy_interrupted_preparation_hold` settlement, terminal-time proof,
status receipt, and `interrupted_preparation` retry lane apply to these
runs.
- `server/src/modules/wake-queue/*`: the release snapshot carries
`retryableBeforeProviderWork` (bootstrap certificate plus a
`transient_infra` continuation classification). When an automatic
continuation retry failed for such a cause, release returns `released`
instead of blocking. The stranded-task sweep then retries with its
existing backoff and 3-attempt cap, and blocks only after that budget is
spent. Non-retryable causes, uninvokable agents, and failures after
provider work still block at once.
- `doc/execution-semantics.md`: documents both rules.
- No migration. Holds saved by the running release are settled by the
existing ownership reconciler after deploy.

## Verification

- Red then green, in the staging shape (failed `controller_lease_lost`,
`renewal_deadline` lease-loss evidence, `restore_failed` with
`workspaceRestoreRecovery`, exhausted `verify_execution` with
`source_unavailable` / `use_alternative_strategy`, retired prepared
epoch):
- `src/__tests__/interrupted-preparation-hold-settle.test.ts`: before
the fix, both new cases fail (`settled: 0`). After the fix, the hold
settles as `false_positive`. With the status receipt, the task returns
to `in_progress` and one `interrupted_preparation` successor is queued.
For a hold saved by an older release without a receipt, the execution
gate clears and one Retry starts a run. A lost lease without the
no-provider-work certificate stays held.
- `src/modules/wake-queue/domain/policy.test.ts` and
`application/use-cases.test.ts`: before the policy change, the two new
release cases fail (the task is blocked). After it, a pre-provider
lease-loss retry failure is released to the sweep, and other failures
still block.
-
`src/modules/wake-queue/adapters/retryable-before-provider-work.test.ts`:
classification of the release snapshot fact.
- `pnpm verify` and server `tsc --noEmit` pass locally.

## Risks

- Lost-lease preparations now use the `interrupted_preparation` retry
lane. That lane is uncharged but bounded at 12 retries, and it does not
revive a chain whose failure budget is spent.
- A released continuation waits for the stranded-task sweep (30 s
reconstruction, plus backoff) instead of a direct requeue. The sweep's
budget bounds the chain.
- Settlement still needs every existing proof: the controller is gone or
settled, leases are released with stop receipts, there are no execution
events or native coordinator, and the prepared epoch is retired or the
files are repaired.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, in Claude Code with tool
use (shell, file edits, browser reads of staging with GET requests
only).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 17:30:32 -06:00
DottaandClaude Opus 5.5 9c56e5465c fix(native): keep the agent's own unanswered question open after a turn ends without a report (#20934)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent can ask the person a question and wait for the answer.
Native task status comes from the server's status arbiter, not from
model prose
> - When a native turn ends without `paperclip_finish`, the Runner sends
a default report: `done`, objective satisfied, with the agent's final
message as the only evidence
> - A later comment from the person moves an open question into history
(#15392), so the question no longer gates completion
> - Together these close a task as `done` while the agent is still
waiting for its own answer and has produced nothing. No Retry appears
and nothing flags it
> - This pull request keeps such a task `in_review` on the agent's own
unanswered question when the only completion evidence is the agent's
final message
> - The benefit is that a "waiting for your answer" turn can no longer
become a silent false completion

## Linked Issues or Issue Description

Refs #15392

**What happened?**
Restart-chaos wave on master 7e8901aa60, OpenCode task t06 (W2Q), no
fault injected. Run fec9530f asks a question (interaction c2d3b3f2); the
task goes `in_review`. A board comment says "include NOTE-238ef87a in
the brief; do not write the brief yet; I will answer shortly". Run
f6cbefa8 replies "I will not write the brief until the board's answer
arrives. Ending turn to await the codename answer." and ends its turn
without `paperclip_finish`. The Runner's default report
(`provider:opencode:agent-message` as the only evidence,
`reportedWorkDisposition: done`) is accepted, and the low-risk claim
policy sets the task `done`. The question is still unanswered and the
`brief` document does not exist.

**What did you expect to happen?**
The task keeps waiting for the answer. The answer continues the task,
and the agent writes the brief.

**Steps to reproduce**
1. An agent asks a question on its task.
2. The person posts a comment before answering.
3. The agent ends its turn without a completion report, saying it waits
for the answer.
4. The task is `done`; the question stays pending.

## What Changed

- New `server/src/services/native-runtime/own-question-wait.ts`:
- `reportsOnlyProviderMessage(result)`: true when every evidence
reference in the report is `provider:<provider>:agent-message` and the
report has no artifact and no verification. This is the shape of the
Runner's default report for a turn that ended without
`paperclip_finish`.
- `pendingOwnQuestionId(db, …)`: the agent's own pending, undismissed
ordinary question (`ask_user_questions`, not a governed tool, secret or
connection request) on the task.
- `finalizeNativeRun` (`native-run-finalizer.ts`): when the proposed
decision is `done`, the task is not an Agent Chat conversation, the
report cites only the agent's final message, and the agent's own
question is still pending, the decision becomes `in_review` with reason
`own_question_unanswered` and releases the checkout. The question stays
pending, so the person's answer continues the task through the existing
answer path.
- Dismissal (`POST /issues/:id/interactions/:interactionId/dismiss`,
`dismissInteraction`): new `enqueueOwnQuestionDismissalWake` runs inside
the dismissal transaction, only when the card moves from pending to
dismissed. When the task is `in_review` on an applied
`own_question_unanswered` decision of the assignee's run, the dismissed
card is the assignee's own question, and no other pending, undismissed
question of that agent remains, it saves a durable wake intent
(`issue_question_dismissed`, idempotency key per question) in the
existing native status wake outbox (`enqueueWake`, now exported). The
outbox dispatcher delivers and retries it, so the agent continues
without the answer. Dismissal of any other request keeps the no-wake
rule.
- `assignmentDenial` (`agent-interaction-delivery.ts`): an
`issue_question_dismissed` wake counts as a person's wake when its
`dismissedInteractionId` names the recipient's own question on that task
with a `dismissedByUserId`, so a recipient that refuses agent
assignments still receives it.
- `doc/execution-semantics.md` describes the exception to the
historical-question rule.

## Verification

- New case in
`server/src/services/native-runtime/historical-task-questions.test.ts`
("keeps the agent's own unanswered question open after an unreported
turn"), run through the real `PaperclipControlPlanePort.completeRun` and
`finalizeNativeRun` with a low-risk `agent_claim_policy` contract, as on
staging:
- Red on master (finalizer hunk removed, test kept): `unreported` fails,
`expected 'done' to be 'in_review'`.
- Green: `unreported` rests `in_review` with `own_question_unanswered`
and the question still pending. `unreported-dismissed`,
`unreported-answered` and `verified` (work-product evidence) still rest
`done`, so the #15392 behavior is unchanged.
- Dismissal after finalization (same test): the release check is null
while a second own question is open and returns the agent once both are
dismissed. A repeated dismissal saves no second intent; the saved intent
is a queued `issue_question_dismissed` outbox row. Delivery test
`delivers a dismissal wake only for a person's dismissal of the
recipient's question` (30 passed). Route test `saves the dismissal wake
intent in the dismissal transaction only when the card is dismissed`
(147 passed).
- `cd server && pnpm exec vitest run
src/services/native-runtime/historical-task-questions.test.ts
src/services/native-runtime/native-run-finalizer`: all passed.
- `cd server && pnpm exec vitest run
src/services/native-runtime/paperclip-control-plane-port.test.ts
src/services/native-runtime/status-arbiter.test.ts`: 119 passed.
- `cd server && pnpm exec tsc --noEmit -p tsconfig.json`: passes.
- Chaos, Daytona OpenCode, seed 6201, W2Q K0, 4 tasks (wave
w-20261111T211421-s6201, run from master 7e8901aa60 plus this change
because the pinned runner image must match runner sources): no false
completion. t02 and t04 PASS (in_review -> in_progress -> done after the
answer). t01 failed `R3:retry_needed` through the blocked path fixed
separately in #20931 (not in this build). t03 timed out because its
rewake OpenCode turn stayed active for 20 minutes without ending
(provider turn never finished; this change acts only at finalization,
which never ran). The unreported-done path did not occur in this sample,
so the red/green proof for this change is the integration test. Before:
wave w-20261111T200718-s6201 t06 closed done with the question
unanswered.

## Risks

- Low. Only a `done` decision whose report cites nothing but the agent's
final message changes, and only while the same agent's own ordinary
question is pending and undismissed. Explicit completion reports with
their own evidence keep the current rule. A person can still dismiss the
question to release it.
- An agent that really finished without `paperclip_finish` while its own
old question is open now rests `in_review` instead of `done`. The open
question is visible on the task; answering or dismissing it settles the
task.
- No schema change. A result saved by an older release is finalized by
the same rule because it reads only the saved report and interaction
rows.
- Related observation (not changed here): in 5 Claude/OpenCode W2/W2C
cells the agent wrote "the task remains in_progress as instructed" yet
the task became `done` (Claude t06, wave s6101). The run's
`paperclip_finish` input itself sent `reportedWorkDisposition: "done"`,
`objectiveSatisfied: true`, with the deferred phase listed as
`blocksCompletion: false`. The server did not map a yielded disposition
to done. That is model behavior on a contradictory report, a different
root from this change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in an agent harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 16:50:28 -06:00
DottaandClaude Opus 5.5 6cfb1f3d87 fix(environments): resume a warm sandbox that its finished predecessor is still releasing (#20953)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm Paperclip Runner session keeps its Daytona sandbox between
runs. The next run of the same task resumes that sandbox through a
`reuse_by_environment` lease.
> - Acquisition only resumes a lease that is `released`, `retained`, or
held by the same run. A finished run releases its lease only after it
saves its instruction and workspace files, which takes several seconds.
> - A board wake that arrives in that window does not see the warm
lease. It creates a fresh sandbox. The session's durable backup cannot
be restored there, because its source lease is still held. The run fails
with `native_session_recovery_failed: runner_harness_state_mismatch:
backup_without_reusable_lease` and the task needs Retry, which starts a
new provider session.
> - This pull request makes acquisition wait, for a short bounded time,
for a lease that a recently finished predecessor is still releasing, and
then resume it.
> - The benefit is that the follow-up continues the same warm session in
the same sandbox, with no Retry, no restore and no lost provider
context.

## Linked Issues or Issue Description

**What happened?**
Restart-chaos wave `w-20261111T205259-s6301` (Codex, Daytona warm,
volume-free server, `--ephemeral-server` from #20930), task t01. The
board re-wake run `e7129ec7` started at 20:57:44. Its predecessor
`5cf8c306` reported `run.terminal` at 20:57:38 but saved its instruction
copy until 20:57:51, so its warm lease was still `active`. At 20:57:53
the new run recorded `startup reuse: environment (fresh)`. It then
failed with `runner_harness_state_mismatch:
backup_without_reusable_lease` (error code
`native_session_interrupted`). Recording the `workspace_finalize`
operation then failed with a Postgres deadlock, and the run settled
`native_finalization_missing`. Retry run `79aa18ec` worked with a fresh
provider session. The K1 restart in that wave was at 20:55:32 and is not
the cause: six warm runs after it reused the sandbox normally.

**Root cause**
`reusableLeaseCanBeResumed` in
`server/src/services/environment-runtime.ts` excludes an `active` lease
held by another run. That rule is correct for a running predecessor. For
a predecessor that has already ended, the `active` state is a short
release tail. The new run creates a sandbox, and
`retiredSourceAuthorizesNativeBackupRestore`
(`native-retired-backup-source.ts`) correctly refuses to restore the
backup from a source lease that is not yet `released` or `retained`
(`idleWarmSourceRetiresBackup`). The executor's last branch
(`native-session-executor.ts`, host preparation, created allocation)
then throws `backup_without_reusable_lease`.

**How often**
Seen in 2 of 361 retained wave result directories on this machine
(`s6301` today, `s5201` earlier). It needs a board wake within a few
seconds of the predecessor's terminal event, so it is timing dependent.

## What Changed

- `server/src/services/environment-runtime.ts`: add
`waitForSettlingReusableLease`. Before the plugin and built-in sandbox
paths select a reusable lease, acquisition polls (250 ms) for up to 20 s
while a matching `reuse_by_environment` lease is `active` under a
different run that is `succeeded`, `failed`, `cancelled` or `timed_out`
and finished in the last two minutes. It then continues with the normal
selection, which resumes the released lease.
- A running predecessor causes no wait (unchanged: the new run gets a
new sandbox). A predecessor that ended more than two minutes ago and
never released causes no wait.
- `server/src/__tests__/environment-runtime.test.ts`: two red/green
cases (succeeded and failed predecessor release the lease 400 ms after
the successor starts acquiring) and one case for an old unreleased lease
(no wait, new sandbox, old lease untouched).
- `doc/execution-semantics.md`: describe the wait next to the existing
warm-reuse rules.

## Verification

- Red on master: `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/environment-runtime.test.ts -t predecessor` fails both
cases with `expected 'warm-2' to be 'warm-1'`: the successor created a
second sandbox instead of resuming the first.
- Green: the same command passes (3 tests).
`environment-runtime.test.ts`, `plugin-warm-tasks.integration.test.ts`
and `environment-run-orchestrator.test.ts` pass (288 tests). Server `tsc
--noEmit` passes.
- Not verified in a chaos wave. The window is a few seconds wide and
appeared in 2 of 361 waves, so one wave cannot show the change.

## Risks

- A successor can start up to 20 s later when its predecessor's release
is slow. Creating a sandbox and restoring a backup takes longer, and
failing needs a Retry. The two-minute bound keeps a stuck,
never-released lease from delaying every later run.
- If a caller held a lock that the predecessor's release needs, the wait
ends at 20 s and acquisition behaves as before.
- Compared with #20863 (fresh-sandbox fallback) and #20811 (unsaved
provider state marker): this change does not alter either. It reduces
how often a warm continuation reaches the fresh-sandbox branch. The
backup restore proof is unchanged, so an unproven backup still fails
closed with a working Retry.
- The `workspace_finalize` deadlock in the same run is a separate defect
and is not fixed here.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, test runs). Default context window,
extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 16:40:58 -06:00
DottaandClaude Opus 5.5 757a20d2f1 fix(runner): keep remote recovery and launch through a short plugin RPC stall (#20945)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote run starts `paperclip-runnerd` in a sandbox. The controller
reads and controls it through the environment plugin
(`environmentExecute`).
> - #20940 made a stalled supervision probe "unknown" while the Runner
stays connected. Other callers still turn the same plugin stall into a
terminal failure.
> - During restart reattachment, the saved-state read, the
identity-marker read and the process check share a 30-second budget. One
plugin RPC that stalls for 40 seconds uses all of it, so a short stall
fails a reattachment to a healthy Runner.
> - At launch, a stalled launch call or identity read fails the run with
`runner_process_termination_unconfirmed` or
`runner_remote_process_identity_unavailable`, although the runner may be
healthy.
> - This pull request treats a plugin RPC timeout as "unknown" and
retries the read-only call within a bounded stall budget. A stalled
launch call is not repeated: the controller reads the launch nonce's
identity marker instead.
> - The benefit is that a short plugin stall during recovery or launch
no longer fails a healthy run. Every positive answer from the sandbox
keeps its meaning, and the bound still fails closed.

## Linked Issues or Issue Description

Refs #20940, #20909

**What happened?**
A plugin `environmentExecute` RPC stall of more than 40 seconds makes a
remote recovery read fail. Plugin RPCs time out at the requested timeout
plus a 30-second overhead, so a 10-second probe can take 40 seconds. The
reattachment read budget is 30 seconds. One stalled read therefore
failed reattachment with `probe_transport_failed` or `probe_timeout`. At
launch, a stalled launch call rejected the run with exit unconfirmed. A
stalled identity read (20-second wait, 2-second reads that can take 32
seconds) failed the run with
`runner_remote_process_identity_unavailable`.

**Expected behavior**
A plugin RPC timeout is a stall of the plugin, not an answer from the
sandbox. Read-only recovery reads retry within a bound. A launch call
that stalls is not repeated. The launch nonce's marker decides whether
the one runner was launched. At the bound, the run fails closed and
Retry works.

**Steps to reproduce**
1. Start a remote (Daytona) run.
2. Restart the controller so that the run reattaches, or start a new
run.
3. While the controller reads the sandbox, stop the plugin worker for 45
seconds.
4. The reattachment or launch fails.

**Paperclip version or commit**
master at 48d23d1530.

## What Changed

- `remote-runner-recovery-probe.ts`: adds `isPluginRpcTimeout` (an
error, or its cause chain, with `PLUGIN_RPC_ERROR_CODES.TIMEOUT`). It
also adds a per-recovery `RemoteRunnerPluginStall` with a 3-minute
budget (`REMOTE_RUNNER_PLUGIN_STALL_BUDGET_MS`). The caller's recovery
deadline caps it. After the first plugin RPC timeout in a recovery, its
read-only probes use the stall deadline instead of the 30-second
deadline. Signals are never retried. Other transport failures keep the
30-second budget.
- `verifyRemoteRunnerReattachment` and `verifyRemoteRunnerOrSettledExit`
share one stall budget across the saved-state read, the identity-marker
read, the process check and the post-exit settled-state read
(`readRemoteRunnerState`).
- `waitForRemoteRunnerProcessIdentity`: a plugin RPC timeout extends the
20-second identity wait to the stall budget.
- Launch: a launch call that fails with a plugin RPC timeout is not
repeated. The controller goes on to the identity-marker wait for the
same nonce. A found marker adopts the one launched runner. A missing
marker uses the existing nonce cleanup, and the result stays unconfirmed
when the cleanup fails.
- `doc/execution-semantics.md` describes these rules.

## Verification

- Red on master: 7 new tests in
`server/src/services/native-runtime/native-session-executor.test.ts`
failed. Each one stalls a plugin RPC for 40 seconds with a
`JsonRpcCallError` `TIMEOUT`:
- reattachment through a stall in the saved-state, identity-marker and
process-check reads (the reattachment failed with the stall error);
  - the bound test (it settled before the stall budget);
  - the post-exit settled-state read (it failed with the stall error);
- launch-call and identity-read stalls (they failed with the stall error
or `runner_remote_process_identity_unavailable`).
- Green: all 7 pass, and a control test proves that a non-stall
transport error keeps the 30-second bound.
`native-session-executor.test.ts` and
`remote-runner-recovery-probe.test.ts` pass (1400 tests). `tsc --noEmit`
passes for `server`.
- The tests also assert that one launch call happens, no cleanup or
signal runs on the success path, and the bound fails closed without exit
proof.

## Risks

- During a real plugin stall, recovery and launch can now wait up to 3
minutes instead of about 30 seconds before they fail closed. The longer
wait applies only after a plugin RPC timeout. A sandbox that answers, or
a non-timeout transport error, keeps the old timing.
- No wait is added before `turn.start` on the normal path.
- State saved by an older version is not changed.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, code search). Extended reasoning was
on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 16:04:13 -06:00
DottaandClaude Opus 5.5 228b0169be fix(dispatch): let the answer to the agent's own question continue the task it blocked on (#20931)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent can ask the person a question (`request_human_input`) and
wait for the answer
> - A later comment from the person moves an open question into history
(#15392). The agent still needs the answer, so it reports a blocker and
the task becomes Blocked
> - The person then answers. Dispatch cancels the answer's continuation
with `issue_not_in_progress`, because only a dependency-blocked task may
take an answer while Blocked
> - The task rests Blocked with a failed run and a Retry. The answer is
lost until a person clicks Retry
> - This pull request lets the answer continue the task when the same
agent blocked it to wait for that answer, proven from durable records
> - The benefit is that the person's answer always reaches the agent
that asked, without a Retry

## Linked Issues or Issue Description

Refs #15392, Refs #20872

**What happened?**
Restart-chaos waves with no fault injected fail every W2Q task on every
provider (Claude, OpenCode, Codex) with `R3:retry_needed`. For Claude
task t03 on master 7e8901aa60: run 5aeb9244 asks a question and the task
goes `in_review`. A board comment ("I will answer shortly; do not write
the brief yet") wakes run d728886f. That comment makes the question
historical, so `paperclip_finish` with a `response_wake` wait is
rejected ("without a pending wait condition"). The agent then calls
`paperclip_block`; the status decision `current_track_blocker_waiting`
sets the task Blocked. The board answers. Continuation run adbda5de is
cancelled: "resolved-interaction continuation issue is no longer
in_progress (current status: blocked)". The task rests blocked/failed
with Retry.

**What did you expect to happen?**
The answer continues the task. The agent writes the brief with the
answer and finishes.

**Steps to reproduce**
1. An agent asks a question on its task.
2. The person posts a comment before answering.
3. The agent reports a blocker while it waits for the answer.
4. The person answers the question.
5. The answer's continuation run is cancelled with
`issue_not_in_progress`.

## What Changed

- Status committer (`status-decision-committer.ts`): when it applies an
agent-reported, board-owned blocker (`current_track_blocker_waiting` or
`task_wide_blocker_bound`), it records the agent's own pending,
undismissed ordinary questions on the task as `awaitingInteractionIds`
in the decision JSON, in the same transaction under the issue lock
(`questionsAwaitedByAgentBlock`).
- New `agentBlockingQuestionState`
(`server/src/services/blocked-question-answer.ts`). It reads only
server-owned rows. It returns the verified `lastStatusDecisionId` and
`statusVersion` when: the answered interaction was created by this agent
on this task and resolved by a person; the task is Blocked and still
assigned to the agent; the task's `lastStatusDecisionId` is an applied
Blocked decision with one of the two reason codes, made by a run of the
same agent, whose `awaitingInteractionIds` contains this interaction;
and the task's `statusVersion` still equals that decision's
`projectedStatusVersion` (nothing changed the status since).
- `decideQueuedRunStaleness`
(`server/src/modules/run-dispatch/domain/policy.ts`): the Blocked
exemption for an answer continuation also accepts this fact
(`isAgentBlockingQuestionAnswer`). The postgres adapter loads it under
the issue lock.
- `heartbeat.ts` resolved-interaction checkout: for that answer,
`blocked` is an accepted expected status, and the verified state is
passed to `issues.checkout` as `blockedState`.
- `issues.checkout`: with `blockedState`, the guarded update accepts a
Blocked row only when `lastStatusDecisionId` and `statusVersion` still
match, so a person who re-blocks the task in between keeps the new
block. When the expected statuses include `blocked`, a checkout from
Blocked also clears `unblockDescriptor`, `blockedTransitionAt` and
`blockedOwnerNotifiedAt`, as an ordinary status change from Blocked
does.
- `doc/execution-semantics.md` describes the rule.

## Verification

- New integration test
`server/src/__tests__/heartbeat-blocked-question-answer.test.ts`
(embedded Postgres, real `commitNativeStatusDecision`): the agent's
blocker decision makes the task Blocked, the person answers, the answer
is delivered through `heartbeat.wakeup` with the same context shape as
staging.
- Red on master (product hunks removed, test kept): `delivers the answer
when the agent blocked the task to wait for it` fails with `status:
cancelled, errorCode: issue_not_in_progress`, the same code as the chaos
run.
- Green: 5/5 pass. The answer run succeeds, the adapter runs, and the
blocker fields are cleared. Negative cases still cancel with
`issue_not_in_progress`: a person re-blocked the task after the agent,
the question did not exist when the agent blocked, and the task is done.
A checkout with a stale verified state conflicts and keeps the person's
new blocker.
- `policy.test.ts`: new case for the fact (continue when set, cancel
when unset, reassignment, backlog/done/cancelled).
- `cd server && pnpm exec vitest run src/modules/run-dispatch
src/__tests__/heartbeat-dependency-scheduling.test.ts`: 208 passed.
- `cd server && pnpm exec vitest run
src/__tests__/issues-service.test.ts
src/__tests__/heartbeat-stale-queue-invalidation.test.ts
src/services/explicit-native-continuation.test.ts`: 656 passed.
- `cd server && pnpm exec tsc --noEmit -p tsconfig.json`: passes.
- Chaos, local, seed 5101, `--workloads W2Q --faults K0 --adapters
codex,claude --lifecycle warm`, on 722303e4a2 (first head): wave
w-20261111T204405-s5101, 6/6 PASS (3 Codex, 3 Claude). Every task went
in_review -> blocked -> in_progress -> done without Retry. Before: the
same local seed on master (w-20261111T190352-s5101) failed 3/3 Codex W2Q
cells with `R3:retry_needed`, and Daytona Claude wave
w-20261111T200553-s6101 failed 2/3 W2Q cells the same way.

## Risks

- Low to medium. Only a resolved interaction continuation on a Blocked
task changes behavior, and only when the durable linkage holds. If a
board-owned blocker about something else and the agent's own question
are open together, the answer resumes the agent; it then reports its
blocker again. The blocker report carries no interaction reference, so
the server cannot tell them apart; cancelling the answer instead loses
it. A person's Blocked status, a block from before the question,
dependency blocks, reassignment, done and cancelled keep the old cancel.
- State saved by an older release: no schema change and no migration. A
Blocked decision saved before this change has no
`awaitingInteractionIds`, so its answer keeps the old behavior (cancel,
Retry works).
- The checkout change only applies when a caller passes `blocked` as an
expected status.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in an agent harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 15:54:33 -06:00
DottaandClaude Fable 5.2 b01959cb15 fix(recovery): release a handed-off run's saved-files hold for the person it handed the task to (#20941)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An agent can hand its own task to a person for review. With #20884,
a native run that does this keeps running and finishes its turn.
> - A restart can land between that hand-off and the end of the same
turn. Restart recovery then settles the run as failed and keeps its
unsaved sandbox files, so no work is lost.
> - Automatic workspace recovery saves those files. But the
workspace-repair hold then stayed on the person's task, because the hold
only released for a newer agent owner.
> - This pull request releases that hold for the person who received the
task through the stopped run's own recorded hand-off, once the files are
saved.
> - The benefit is that the person's task rests in review with no stale
"Saved work recovered" hold, and the agent's files are still saved
first.

## Linked Issues or Issue Description

Follow-up to #20884 (merged). It was reviewed there as a known limit:
"Restart blocks the person's task". It uses the
`resultJson.humanSelfHandoff` record that #20884 added. Related: #19184,
which released the same hold for a newer agent owner.

**What happened?**

A native run handed its own task to a person, and a restart landed
before the run finished its turn. Restart recovery settled the run and
retained its files. After automatic recovery saved them, the task still
had a workspace-repair hold ("Saved work recovered. Send a message to
continue."). The person had nothing to do with that run's work, and
nothing released the hold.

**Expected behavior**

The hold protects the workspace until the files are saved. After that,
the task rests with the person and has no execution blocker.

**Steps to reproduce**

1. A native run records a hand-off to a person, and the task is now
`in_review` with that person.
2. A restart settles the run as failed, with its prepared remote
workspace retained.
3. Automatic workspace recovery completes.
4. Before this fix, `getExecutionBlocker` still returns the run's
workspace-repair hold.

## What Changed

- `server/src/services/execution-blocker.ts`:
`restoreHoldReleasedForCurrentOwnerPredicate` now also releases a
recovered file-preservation hold in one more case. The task must be
assigned to a person, and the stopped run's
`resultJson.humanSelfHandoff` must name that task and that person. The
existing proof is still required: the completed-recovery receipt bound
to the action, and a copyback rather than a snapshot-only save. Until
the files are saved, the hold stays.
- `server/src/services/workspace-recovery-admission.test.ts`: two
embedded-Postgres tests.
- The hold blocks the person's task before recovery and releases after
it.
  - A hand-off recorded for a different person keeps the hold.
- `doc/execution-semantics.md`: one sentence on the restart case.

## Verification

- Red: with the `execution-blocker.ts` change removed, `releases a
handed-off run's recovered files hold for the person it handed the task
to` fails. `getExecutionBlocker` still returns the hold after
`recoverRetainedWorkspace`.
- Green: `cd server && npx vitest run
src/services/workspace-recovery-admission.test.ts
src/__tests__/interrupted-preparation-hold-settle.test.ts
src/services/explicit-native-continuation.test.ts
src/services/native-runtime/native-human-self-handoff.test.ts`. All 617
tests pass. The server typecheck passes.
- Chaos: the H2 workload from #20884 covers the hand-off with a K1
restart after it. Locally the runs had no unsaved files, so no hold
appeared. This PR's case needs a remote sandbox with unsaved files. **No
Daytona wave was run.**

## Risks

- The release applies only when the stopped run's own record names the
current person and task, and only after the completed-recovery receipt.
Other holds, snapshot-only saves and unsafe archives are unchanged.
- No schema change. A run without the record behaves as before.
- This branch is rebased onto master after #20884 merged. The only
conflict was an adjacent import line.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, through Claude Code
with tool use and code execution. The session does not expose the
context window or reasoning mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 15:52:35 -06:00
DottaandClaude Opus 5.5 48d23d1530 fix(runner): keep a connected remote Runner through a stalled plugin supervision probe (#20940)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote run starts `paperclip-runnerd` in a sandbox. The controller
monitors the process with a health probe that goes through the
environment plugin (`environmentExecute`).
> - A stall of the Daytona plugin worker makes that probe RPC time out
after 40 seconds. The monitor treated the timeout as "process
supervision failed".
> - The transport then failed closed with
`runner_process_termination_unconfirmed`. The run ended
`notification_transport_failed`, `recoverable: false`, and the close
path suspended a healthy Runner. The task rested blocked with Retry.
> - Nothing was wrong with the sandbox, the Runner or the provider. The
Runner held its own authenticated connection to the controller during
the full stall.
> - This pull request makes a failed or timed-out probe "unknown". While
the Runner stays connected on its own channel, the monitor keeps the
same runner and turn and probes again, for a bounded time.
> - The benefit is that a short plugin stall no longer kills a healthy
run. The fail-closed cases stay the same.

## Linked Issues or Issue Description

Refs #20909 (chaos cell CR3, which reproduces this failure)

**What happened?**
A 45-second stall of the Daytona plugin worker killed a healthy run.
Chaos cell CR3 reproduced it 2 of 2 times (OpenCode and Claude,
Daytona). A staging run had the same failure with no deploy: a plugin
`environmentExecute` RPC stalled for more than 40 seconds. The run
failed with "Notification transport failed closed:
runner_process_termination_unconfirmed: runnerd process supervision
failed: RPC call \"environmentExecute\" timed out after 40000ms". The
close path then sent `runner.suspend`, and the healthy Runner suspended
and exited. The task rested blocked with Retry.

**Expected behavior**
A probe that times out or fails in transport is not proof that the
process stopped. While the Runner stays connected on its own channel,
the run continues. If supervision cannot come back within a bound, the
run rests with a working Retry. A probe that positively reports the
process gone still starts the existing recovery.

**Steps to reproduce**
1. Start a remote (Daytona) run with a T1 workload.
2. Send SIGSTOP to the server's Daytona plugin worker for 45 seconds,
then SIGCONT.
3. The run fails `notification_transport_failed` about 40 seconds into
the stall.

**Paperclip version or commit**
master at 7cd5558e41.

## What Changed

- `createRemoteRunnerProcessLauncher`
(`server/src/services/native-runtime/native-session-executor.ts`): when
the health probe throws (RPC timeout or transport error) and the Runner
holds an authenticated connection, the monitor waits one poll interval
and probes again. It does not reject the process completion. The unknown
period has a bound: the remote runner reconnect grace (7 minutes). When
the Runner is not connected, the monitor keeps the old behavior: it
rejects after the 30-second probe budget, without exit proof.
- Probe results that positively answer (exit 45 "process gone", or
identity and generation failures) keep their exact current handling.
- `RunnerProcessLaunchSpec`
(`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`)
has a new optional `isRunnerConnected` callback. `spawnRunner` passes it
to the process owner and keeps it on restart.
- The runnerd transport and the external runnerd driver supply
`isRunnerConnected` from the control plane's count of authenticated
Runner connections.
- `doc/execution-semantics.md` describes the unknown-probe rule and its
bound.

## Verification

- Red, on master: `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "plugin
RPC timeouts until"` (in `server/`). The completion rejected after the
first stalled probe with `{ stage: "process_verification", reason:
"probe_transport_failed" }`. The transport turns that into
`runner_process_termination_unconfirmed`, which is the staging error.
- Red, on master: `npx vitest run
src/live/runnerd-codex-transport.test.ts -t "live evidence of the
runner"` (in `packages/paperclip-runner/`). The launch spec had no
`isRunnerConnected` (`expected undefined to be true`).
- Green: both tests pass. New tests also prove:
- three 40-second plugin RPC timeouts with the Runner connected leave
the same runner supervised, and a later exit-45 probe still confirms
exit;
- with the Runner disconnected, the same timeout rejects without exit
proof (`probe_transport_failed`, `isExitConfirmed() === false`);
- with the Runner connected and probes that never answer, supervision
rejects without exit proof after the 7-minute bound.
- `server/src/services/native-runtime/native-session-executor.test.ts`,
`remote-runner-recovery-probe.test.ts`,
`packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts`,
`src/control-plane/durable-prp-control-plane.test.ts` and
`src/drivers/external` pass locally.
- `tsc --noEmit` passes for `server` and `packages/paperclip-runner`.
- Chaos: CR3 results are in a follow-up comment on this PR.

## Risks

- While the Runner stays connected and the plugin cannot answer, the
controller waits up to 7 minutes before it fails closed. Before this
change it failed after about 40 seconds. During that time the Runner's
own connection is the liveness evidence, and the controller never
reports an exit.
- When the Runner is not connected, the timing is the same as before.
Sandbox loss still fails closed after the 30-second probe budget.
- The new spec field is optional. A process owner that does not read it
keeps the old behavior. State saved by an older version is not changed.
- No wait is added before `turn.start`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, code search). Extended reasoning was
on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 15:45:25 -06:00
DottaandClaude Fable 5.2 b2779054d4 fix(issues): let a native run hand its own task to a person without cancelling itself (#20884)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A task has one assignee. An agent often finishes its work by handing
its own task to a person for review.
> - The task update route stops the outgoing run before it saves a
reassignment.
> - Since #16082, a legacy run that hands its own task to a person is
exempt, so it finishes its turn. Native runs were not exempt.
> - On staging, a native run (ACPX Claude, 11.5 minutes of work) sent
`PATCH /api/issues/{id}` to hand its task to a person. The route
cancelled that run with `issue_reassigned`. The agent saw
`api_transport_failure`. The task moved to the person, but it kept a
stale execution blocker ("Saved work recovered. Send a message to
continue.").
> - The native runtime assumes a running run still owns its task. So
widening the exemption alone is not enough: the run then fails after its
result is accepted, and its finalizer can act on the person's task.
> - This pull request applies the exemption to native runs and makes the
native terminal fence, restart recovery and finalizer handle a run that
gave its task away.
> - The benefit is that an agent can hand its task to a person and
finish normally, with no blocker on the person's task.

## Linked Issues or Issue Description

Refs #16082 (legacy exemption). Related: #20541 (agent-to-agent self
handoff) and #18293 (agent key handoff). Those PRs cover other actors
and targets. This PR covers native runs that hand their task to a
person.

**What happened?**

A native agent run handed its own task to a person (`assigneeAgentId:
null`, `assigneeUserId`, status `in_review`) through its signed run
token. The route cancelled the run that sent the request
(`issue_reassigned`, "Cancelled before issue reassignment"). The
cancelled native run then went through workspace recovery. The person's
task kept an execution blocker with `restore_failed`.

**Expected behavior**

The run that hands its own task to a person keeps running and finishes
its turn. The task rests `in_review` with the person. No execution
blocker or recovery action stays on the task. The outgoing run cannot
change the task after the hand-off.

**Steps to reproduce**

1. Run a native agent on its assigned task.
2. From that run, send `PATCH /api/issues/{id}` with `assigneeAgentId:
null`, a person's `assigneeUserId`, and status `in_review`.
3. Before this fix, the route cancels the run with `issue_reassigned`.

**Why the exemption was legacy-only**

#16082 limited the exemption to the run shape of its incident and kept a
verified stop for every other case. The new H2 chaos workload showed why
native needed more than the route change. With only the route widened:

1. The run's terminal fence `assertCurrentWakeCommentsRead` requires the
run to still own the task. It threw
`native_current_wake_comments_binding_changed` after the result was
accepted. The run then failed with `native_session_lease_lost`, and its
finalizer recorded `workspace_finalization_failed` as an agent-owned
recovery on the person's task.
2. A restart after the hand-off left the run `running`, because restart
recovery refused it with `issue_execution_lock_changed`.
3. The native finalizer only superseded a `done` decision. A
`needs_review` report moved the person's task back to `in_progress` and
queued a continuation for the outgoing agent.
4. The REST guard against the outgoing run's later mutations
(`issue_handed_off`) applied to legacy runs only.

## What Changed

- `server/src/routes/issues.ts`: the signed current-run human
self-handoff exemption now applies to native runs. All other checks
stay: exact run, agent, company and task, `agent_jwt` source, human
recipient. For a native run the route first records the hand-off on the
run (`resultJson.humanSelfHandoff`, new `native-human-self-handoff.ts`).
Board reassignment, agent keys, and handoffs to an agent or to nobody
still stop the outgoing run.
- `current-wake-comments.ts`: the terminal fence accepts a run whose
recorded hand-off matches the person who now owns the task. Without the
record (for example a board reassignment), or after a later
reassignment, it still refuses.
- `native-restart-recovery.ts`: after a restart, a handed-off run is not
resumed. It passes the execution-lock check and is then settled by the
existing ownership gate (`task_ownership_or_status_changed`) without
changing the person's task, instead of staying `running`.
- `status-arbiter.ts` and `native-run-finalizer.ts`:
`supersedeHandedOffDecision` keeps the person's task status and strips
retries, continuations, owner wakes, recoveries, blockers and reviewer
bindings, also from decisions that already preserve the status. Two
effects stay: a governed request's interaction, and the
workspace-finalization repair, so saved work is not lost. Review runs
are not affected.
- `server/src/run-task-ownership.ts` (renamed from
`legacy-run-task-ownership.ts`): `assertHandedOffRunTaskOwnership`
rejects later task mutations from the outgoing legacy or native run, in
the route and under the update transaction's locks.
- `tests/restart-chaos/`: new opt-in workload `H2`. The agent saves a
brief, then hands the task to its creator in review as its last step.
The oracle requires the task to rest `in_review` with the creator, every
run to settle without `issue_reassigned`, readable non-empty run
history, and no execution blocker or active recovery action.
- Docs: `doc/execution-semantics.md`, task feature map and agent access
map, chaos README.

## Verification

- Red on `origin/master` (same tests, master source): the route cancels
the native run (500, and 409
`runner_goal_reassignment_stop_unconfirmed`); the finalizer moves the
person's task to `in_progress` with a continuation for the outgoing
agent; a late `done` from the outgoing native run closes the person's
task.
- Red chaos wave on master's server (`--server-from` a master checkout,
`--adapters claude --workloads H2 --faults K1 --matrix`): 0/2 pass. Both
runs ended `cancelled` / `issue_reassigned`
(`T:handoff_did_not_cancel_run`).
- Green focused tests on this branch:
- `native-human-self-handoff.test.ts` (embedded Postgres): hand-off
through the real issue service; the run stays `running`; `done`,
`needs_review` and failed-turn reports leave the task `in_review` with
the person, with `getExecutionBlocker` null and no recovery action; the
terminal fence accepts only the recorded hand-off; a late change from
the outgoing run fails with `issue_handed_off`; pure policy cases.
- `issue-agent-mutation-ownership-routes.test.ts`: native and legacy
self-handoffs persist without `cancelRun`, and native records the
hand-off before the update. Board, agent key, other run, task, agent and
company still stop the run.
- Native runtime suite (restart recovery, wake comments, arbiter,
stale-owner completion), `issues-service`, `low-trust-red-team-routes`:
pass. Server and restart-chaos typecheck pass. `node
scripts/check-maps.mjs` passes.
- Green chaos waves on this branch (local, `--adapters claude
--workloads H2`):
- `--matrix --faults K1 --seed 4459`: 2/2 pass (K0 baseline and a K1
crossing).
- `--tasks 3 --faults K1 --fault-after-seconds 60` and
`--fault-after-seconds 100`: 3/3 pass each. In the first, all three
restarts landed after the hand-off while the runs were live; the runs
settled and the tasks stayed with the person.
- The final waves ran on the pre-rebase head `8a2a3b6a74`. The rebase
onto master only merged the chaos workload lists (`types.ts`,
`config.ts`, `journeys.ts`). No Daytona wave was run.

## Risks

- The outgoing native turn keeps running after the person owns the task.
Its native task tools already require the run to own the task, so they
refuse (`paperclip_runner_tool_binding_not_authorized`). Its REST
mutations are rejected with `issue_handed_off`.
- A restart after the hand-off settles the run as failed
(`native_restart_recovery_blocked`) instead of resuming it. The person's
task is not changed and needs no Retry.
- The finalizer rule covers any native run whose task belongs to a
person at finalization, not only a self-handoff. Before this change such
a run could still move the person's task and queue its own continuation.
- Saved state: `resultJson.humanSelfHandoff` is a new optional run key.
An older server ignores it, so a run handed off on the new server and
finished on an old one fails its fence as today. No schema change.
- No API shape change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, through Claude Code
with tool use and code execution. The session does not expose the
context window or reasoning mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 15:30:59 -06:00