Commit Graph
278 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 01c1e708e9 fix(runner): stop a turn that keeps resending a refused tool call and continue fresh (#21734)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs call Paperclip tools, and
`RepeatedToolFailureGuard` refuses an exact repeat of a call that
already failed validation twice
> - The refusals are not counted and never end the turn, so a model
whose host drops an argument can resend the same refused call until the
run's wall clock
> - In a recorded restart-chaos wave, an opencode/DeepSeek agent sent
the same `PATCH /api/issues/{id}` through `call_api` 431 times in 40
minutes without `body`; 427 were refused. The run held an agent slot and
delayed other tasks
> - The system knows the next step (stop the turn and continue on a
fresh session), so the harness must own it, with no Retry banner
> - This pull request counts refusals, stops the provider turn after a
bounded number with the classified cause `agent_tool_call_loop`, and
queues an automatic fresh-session continuation that tells the model to
use a dedicated tool
> - It also records the declared argument names that each opencode tool
call delivered, so we can tell whether the bridge or the model drops
`body`
> - The benefit is that a looping agent frees its slot in seconds
instead of 40 minutes, and the task continues by itself

## Linked Issues or Issue Description

**What happened?**
An opencode agent on DeepSeek created two subtasks and then tried to set
their descriptions with `call_api` `PATCH /api/issues/{id}`. Every call
arrived with `operationId`, `pathParams`, `query` and `responseText`,
but without `body`. The guard refused each repeat. The model said "I
need to include body" more than 20 times and kept sending the same call:
431 calls, 427 refused, over 40 minutes. The run stayed active and its
agent slot delayed other queued work.

**Expected behavior**
After a small, bounded number of refusals, Paperclip stops the provider
turn and continues the task automatically on a fresh session, with a
hint to use a dedicated Paperclip tool. The board shows no Retry banner
because the system owns this recovery.

**Steps to reproduce**
1. Run a native agent whose model resends one tool call with a missing
required argument.
2. The guard refuses the third and later identical calls.
3. On master, the refusals never end the turn.

**Paperclip version or commit**
`master` at 6400698a8e.

## What Changed

- `repeated-tool-failure-guard.ts`: counts refusals per exact call and
per run. At 6 refusals of one call, or 20 refusals in the run, it throws
`ToolCallLoopError` (`code: "agent_tool_call_loop"`, tool, counts and
argument names, never values). The message tells the model the turn
stops and gives a short hint.
- `paperclip-runner-tool-authority.ts`:
`scheduleToolCallLoopContinuation` writes one idempotent native
status-wake outbox row (key `agent_tool_call_loop:<runId>`) with
`forceFreshSession: true` and the hint as the wake instruction. The
outbox dispatcher admits it with `allowRunCoalescing: false`, so it
never merges into a deferred comment wake or the active run. An assignee
run uses `issue_continuation_needed`. A completion-review run uses
`native_completion_review` with its review assignment, which the
dispatcher revalidates, so the fresh reviewer session keeps the review.
- `native-session-executor.ts`: the session authority epoch queues that
continuation when the guard raises the loop error. The governed-wait
lookup finds the wake on the next tool event and ends the provider turn
as `yielded` with a `same_agent` continuation, the same way the
connection-tools refresh ends a turn. If the wake cannot be queued, the
refusal still reaches the model and the error is logged.
- `status-decision-committer.ts`: a yielded continuation with that key
binds to the queued wake instead of creating a second one.
- `status-arbiter.ts` and `native-run-finalizer.ts`: a pending review
run that ended through this continuation skips the
`native_review_unresolved` resume recovery, because the queued review
wake owns the next reviewer run.
- `mcp-bridge.ts` and `opencode-proxy-finish-diagnostics.ts`: the
opencode tool-delivery diagnostic records `argumentNames` (only names
declared in the tool's input schema, sorted, at most 1,024 characters of
JSON), `argumentNamesTruncated` and `undeclaredArgumentCount`. Values
and undeclared names are never recorded. The forwarder's closed key
allowlist accepts the new fields and validates them, so a record stays
under its 2,048-character limit.

### Threshold choice

- **6 refusals of one exact call.** The guard already runs a call twice
and then refuses with an explicit correction. Six more refusals (eight
identical attempts in total) give a model that reads errors many chances
to change its arguments. In the recorded run the model never changed
them in 427 refusals, so six refusals end that loop within seconds.
- **20 refusals in the run.** A model can alternate between a few
refused variants and keep each one under 6. Refusals happen only for
exact repeats of a certain validation failure, so a healthy run almost
never reaches 20.

### Previous-release story

The guard counters live in memory for one run's tool authority, as
before. A deploy or restart during a run starts the counters again at
zero on the new process; that only delays the stop. The continuation is
an ordinary wake row. If a server on the previous release finalizes a
run that a new server parked, it does not know the key and queues its
own `same_agent` continuation too; the issue execution lock lets one run
and the other coalesces.

## Verification

- Red then green, `repeated-tool-failure-guard.test.ts`: "after N
refusals the guard raises a turn-stopping loop error" and "stops the
turn once refusals across different calls reach the run limit". Both
fail on master.
- Red then green, `native-session-executor.test.ts`: "a turn whose model
repeats a refused call N times ends with agent_tool_call_loop and
schedules a continuation". With the epoch handling disabled it fails (13
calls instead of 8, nothing parked). Also "keeps the refused call's
error when the loop continuation cannot be queued".
- `paperclip-runner-tool-authority.test.ts`: "queues one fresh-session
continuation outbox for the assignee / a completion reviewer stopped by
a tool-call loop" (real Postgres; idempotent).
- `native-tool-call-loop-continuation.test.ts`: with a comment wake
already deferred behind the looping run, the continuation keeps its own
keyed deferred wake, the comment wake is unchanged, and after release a
run starts with `forceFreshSession` and the loop cause.
- `status-arbiter.test.ts`: a pending reviewer stopped by a loop
releases its checkout without a resume recovery.
- `native-status-arbiter-corpus.test.ts`: the existing wake-binding test
now also covers the `agent_tool_call_loop` key.
- `mcp-bridge-delivery.test.ts` and
`opencode-proxy-finish-diagnostics.test.ts`: argument names are
recorded, values and undeclared names are not, invalid records are
dropped, and a boundary test with 32 names of 64 characters stays
forwardable with `argumentNamesTruncated: true`.
- Server: the six touched test files pass (1755 tests). Runner:
`src/drivers/opencode` and `src/cli` pass (569 tests). `tsc --noEmit`
passes for `server` and `packages/paperclip-runner`.
- `pnpm verify` exits 2 (runner package changes need broad coverage), so
required CI supplies the broad run.

## Risks

- A run that hits the limit ends its turn early. The limits are far
above what a model that reads its errors needs, and the task continues
automatically on a fresh session.
- The fresh session loses the in-turn conversation. The task, its
comments, documents and workspace stay, and the continuation names the
cause and the hint.
- The diagnostic now carries declared argument names. These are schema
names, not values, and the forwarder validates the shape before it
forwards a record.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #21360

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 01:52:50 -06:00
DottaandClaude Opus 5.5 958eda6f16 fix(native): wait for a live outgoing server before resuming its warm session in a deploy overlap (#21697)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A deploy swaps servers with an overlap. The outgoing server still
serves while the incoming server starts new runs.
> - A warm agent session stays with the server that ran its last turn.
That server closes the session about 1 s after it leaves new run starts
to the replacement (#21627).
> - A follow-up that started within about 1 s of its predecessor's
finish read the session as suspended, while the outgoing server was
still closing it. The follow-up did not wait. It started a second Runner
with the wrong OpenCode launch files, and the Runner refused the launch.
> - The same-run retry then went to the outgoing server, which had
already left run starts. The run failed, and a blocked parent showed a
Retry during its scheduled resume.
> - This pull request makes the follow-up wait for the live outgoing
server, rebuilds the admitted launch files on the new Runner, keeps
same-run retries on the replacement, and shows the scheduled resume for
a blocked parent.
> - The benefit is that a follow-up during a deploy overlap finishes
without a failed run or a Retry banner.

## Linked Issues or Issue Description

Refs #21627

**What happened?**
A zero-downtime release-swap test (Gate B, wave B1 on master 890dcbdd66)
failed 2 of 8 cross-server hand-offs. In both, the follow-up started 1.0
s and 0.6 s after its predecessor finished on the outgoing server. The
other 6 hand-offs started 34 to 217 s later and passed.
- The incoming server took the resume path that skips
`inspectRemoteWarmSessionRecovery`, because the local authority state
read `suspended`. The live-owner wait from #21627 lives only in that
inspection, so it never ran (0 waits recorded).
- The new home had no harness backup, so the run had no provider
artifact receipt. `prepareRemoteRunner` returned early on
`legacyProviderContinuity`, and the launch profile named the provider
pack folder. Runner core refused it: "OpenCode run.prepare launch does
not match the runner-owned qualified profile".
- The same-run retry was claimed by the outgoing server after it left
run starts. Its predecessor retirement reported `not_suspended`. Both
runs failed with `native_continuation_requires_reconciliation`, and
recovery started fresh runs.
- For a parent already blocked on its subtasks, the task showed "failed"
with Retry for 31 s while the same-run resume was scheduled.

**Expected behavior**
The follow-up waits while the outgoing server may still hold the
session. A new Runner gets the same OpenCode launch files as the
session's first launch. Only the replacement claims same-run retries. A
blocked parent shows the scheduled resume and no Retry.

**Steps to reproduce**
1. Run the restart-chaos release-swap preset with fresh homes and a warm
OpenCode staging mix.
2. Let a run finish on the outgoing server after the traffic switch,
with a follow-up comment already queued.
3. See the follow-up start on the incoming server within about 1 s and
fail on `run.attach` with the qualified-profile mismatch.

**Paperclip version or commit**
master 890dcbdd66.

**Deployment mode**
Hosted deploy overlap (release swap) with Daytona sandboxes and S3
storage.

## What Changed

- `remote-warm-session-recovery.ts`: new `waitForLiveWarmSessionOwner`.
It reads the finished run's finalization owner. While that server is
alive inside the warm window, the follow-up defers with the existing
`native_controller_recovery_waiting_for_owner` wait. A harness backup
that holds the suspended state and was completed after the turn proves
that the owner finished its close, so the follow-up continues at once.
- `native-session-executor.ts`: calls that wait before the `suspended`
check, so every warm follow-up on another server's session passes
through it. The backup check runs only when the owner is alive.
- `native-session-executor.ts`: when a resume has a durable session but
no receipt from a harness backup or the admission, it reads the provider
artifact receipt recorded on the session's last run. The new Runner then
gets the admitted launch root, as a cold continuation already does.
- `heartbeat/recovery.ts` and `heartbeat.ts`: a server that yields run
starts to a replacement does not claim same-run retries, from the
recovery sweep or from its own resume timer. It re-checks every 3 s, so
it takes the retries back when the replacement stops.
- `task-interruption-presentation.ts`: a blocked task whose remote run
waits for its `exact_checkpoint_resume` keeps the resume time, so it
shows `retry_scheduled` with no Retry.
- `doc/execution-semantics.md`: documents these rules.

## Verification

Red on master, green with this change:
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "close
window waits for its live lease"`: "a follow-up that starts within the
outgoing server's close window waits for its live lease instead of
relaunching the Runner". Master proceeds and fails with
`runner_state_identity_mismatch`.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "remote
opencode-continuation-prior-receipt"`: a resume relaunch without an
adopted process uses the original OpenCode file paths. Master launches
from the provider pack folder and does not retain the receipt.
- `cd server && npx vitest run src/services/heartbeat/recovery.test.ts
-t "leaves run starts"`: "an outgoing server does not claim a same-run
retry once it leaves run starts". Master claims it.
- `cd server && npx vitest run
src/__tests__/task-interruption-query.test.ts -t "blocked parent whose
run awaits"`: "a blocked parent whose run awaits its exact same-run
resume shows retry_scheduled with no Retry". Master shows `failed` with
Retry.

Also new: "keeps a follow-up waiting for a live outgoing server until
that server has checkpointed the session after the turn" in
`native-controller-recovery.integration.test.ts` (the backup proof and
the owner exit end the wait).

Focused suites pass: native-session-executor,
remote-warm-session-recovery, native-controller-recovery (integration),
heartbeat recovery, task-interruption-query,
task-interruption-presentation, native-recovered-owner-followup,
heartbeat-native-predispatch-retry, run-start-handover. `npx tsc
--noEmit -p server/tsconfig.json` passes.

Not run here: the restart-chaos release-swap wave. The maintainer runs
it after merge.

## Risks

- A follow-up on a session that a live outgoing server ran can now wait
until that server exits or the warm window ends. This applies only when
no harness backup proves that the server finished its close after the
turn. Before, such a follow-up failed and needed recovery.
- The backup proof hashes the restored backup. It runs only when the
previous owner is alive in another process (a deploy overlap), not on a
normal turn.
- A resume without a backup receipt now restores the admitted artifacts
from storage, as a cold continuation does. A session whose last run
recorded no receipt keeps the legacy pack-folder launch.
- A yielding server leaves same-run retries to the replacement. If the
replacement stops, the yield ends with its lease, and the older server
claims them again within seconds.
- Previous-release state: no persisted field is added. Receipts,
finalization owners and `failureDetail.recoveryMode` are already saved
by the previous release.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 19:50:01 -06:00
Nicky LeachandClaude Fable 5.2 0ab73d4e30 feat(native-runtime): ask the Runner for provider-started turns on native ACPX Claude runs (#21456)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core runs Claude Code through Paperclip Runner. Claude can start a
shell command in the background and end its turn. When the command
finishes, Claude Code starts a cycle of its own to continue.
> - Today the run ends when the turn ends. The command stops, and the
run fails with `native_finalization_missing`.
> - The Runner can keep Claude while a background shell is running and
give the cycle of Claude to the controller as a turn that the provider
started. Core can wait for that turn. The transport can ask for it
(#21607), and a controller restart during it is safe (#21612).
> - No run does this yet, because Core does not ask the Runner for it.
> - This pull request makes the native runtime of Core ask, for ACPX
Claude. It is the only part that changes what a production run does.
> - The benefit is the goal of the whole work: a run with a background
command that is still running continues when the command finishes, and
does not fail with `native_finalization_missing`.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request turned the behavior on and had grown too large to
review. It is now three stacked pull requests:

1. #21607 (merged): the transport can ask the Runner for
provider-started turns. No caller asks.
2. #21612: the recovery after a controller restart during a wait or
during a provider-started turn.
3. **This pull request:** Core sets the option for native ACPX Claude
runs.

This is part 3. Its base branch is `fix/provider-turn-restart-recovery`.
Refs #19912 (design).

**What existing behavior does this improve?**

What a native ACPX Claude run does after a turn that ends with no result
while a background command is still running.

**Subsystem affected**

Core, the native runtime:
`server/src/services/native-runtime/native-session-executor.ts` and
`native-provider-turn-wait.ts`. ACPX Claude only.

**Current behavior**

- The run ends at the end of the turn. The Runner restarts Claude, the
background command stops, and the run fails with
`native_finalization_missing`.

**Proposed behavior**

- The native runtime sets `providerStartedTurns: true` in the transport
options for ACPX Claude, at the two places that build those options. One
function, `nativeProviderStartedTurns`, decides this and also decides if
the consumer gets the wait options, so the two cannot be different.
- A turn that ends with no result while a background command is still
running now waits, for a maximum of 10 minutes, for the turn that Claude
starts by itself. That turn is consumed as the next turn of the same
run, and its result settles the run.
- Other providers and other ACPX agents do the same as today.
- Limit of the Runner (see #21421): after a follow-up turn, the Runner
waits again only while a background command is still running. A command
that finishes while Claude reports on an earlier command gets no
follow-up.

**Reason and benefit**

- The opt-in is in the last part, so each earlier part can merge with no
change for a production run.

## What Changed

- `server/src/services/native-runtime/native-provider-turn-wait.ts` (5
lines added, 1 removed): `nativeProviderStartedTurns`.
- `server/src/services/native-runtime/native-session-executor.ts` (4
added, 2 removed): the option at the two places that build the transport
options, and the same function for the wait options.
- `server/src/services/native-runtime/native-session-executor.test.ts`
(4 added, 2 removed): the two native paths set the option for ACPX
Claude and not for other providers.

## State saved by the previous release

- No persisted state changes in this part.
- **A Runner that does not know the field** ignores it, and the run does
the same as today.
- **A deploy during a run:** a turn that started before the deploy has
no opt-in and ends as today. The next turn has the opt-in.
- **Rollback of Core:** the previous Core does not set the option, so
the Runner does not wait.

## Verification

The items below were run on the tree that was verified before the split
into three parts (commit `1b15ded1c4` has that tree, byte for byte). The
head of this branch (`23e2c13ad3`) is that commit on the new head of
#21612; its three server files are the same.

- `pnpm typecheck` and `node
scripts/verify-runner-vendor-dependencies.mjs` in `server`; `npx tsc
--noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json`,
`pnpm run typecheck:typescript` and `pnpm run build:typescript` in
`packages/paperclip-runner`: all passed.
- `server/src/services/native-runtime/native-session-executor.test.ts`
and `native-runner-restart-recovery.integration.test.ts`: 1,732 passed,
21 skipped.
- The end-to-end behavior is tested in parts 1 and 2 with the real
`paperclip-runnerd`, where the test sets the option in the way that this
part does.
- Live run with a real Claude account, not in CI, at the adapter level
of the Runner (see #21316): the cycle of Claude was given as a turn with
its own usage, and each Paperclip tool call was admitted.

Not run: a live run through Core, `paperclip-runnerd` and the real
sidecar with a real Claude account (that path needs an API key, not a
local sign-in); a remote sandbox; the full suite. CI supplies the broad
coverage.

## Risks

- **This part turns the behavior on for each native ACPX Claude run.** A
run with no background command is not changed: the Runner waits only
when a shell is still running at the end of a turn that has no result.
- **This part must not merge before #21612.** Without #21612, a
controller restart during a wait or during a provider-started turn fails
the run.
- **A run can now stay open up to 10 minutes longer** when the
background command does not finish or Claude does not start a turn. The
agent computer stays active for that time.
- **A waiting run can show as "suspicious" after 5 minutes of silence**
in the run watchdog. The watchdog only labels a native run; it does not
stop it.
- **Follow-ups that the Runner cannot wait for are lost,** as the limit
above says.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 13:10:20 -08:00
DottaandClaude Opus 5.5 890dcbdd66 fix(native): wait for a live warm session owner during a deploy overlap (#21627)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A deploy swaps servers with an overlap: the outgoing server still
serves while the incoming server starts new runs
> - A warm agent session stays open in the server process that ran the
last turn, after that run finished
> - The incoming server took over a warm session while the outgoing
server was alive and still held it, because the finished run's
finalization lease ends at commit
> - The takeover then failed, the task went blocked with a Retry, and in
one case a retry loop ran 27 times on the same sandbox
> - This pull request makes the incoming server wait while the server
that holds the warm session is alive, makes the outgoing server close
idle warm sessions as soon as it leaves new runs to the replacement, and
stops the retry loop when the promised fresh sandbox does not come
> - The benefit is that a follow-up during a deploy finishes without a
person, and no Runner ever gets two controllers

## Linked Issues or Issue Description

**What happened?**
A zero-downtime release-swap test (Gate B, wave B1, master aaec67b891)
found two failures with one cause.
Task 1: run A finished on the outgoing server at 15:13:56. Its warm
session stayed idle in that server. The incoming server claimed the
follow-up at 15:13:57. `inspectRemoteWarmSessionRecovery` asked
`evaluateNativeControllerTakeover`, which allowed the takeover with
`remote_controller_lease_released_or_expired`. That lease covers the
finished run, not the warm session. The outgoing server was alive until
15:18:18. `runner.prp.authenticate` failed after 62.3 s, the run failed
with `remote_warm_session_unproven`, the failure was saved as ambiguous
and owned by the board, and the task stayed blocked with a Retry.
Task 2: the same shape on a later swap started 27 automatic runs in 13
minutes. Fourteen of them said "The next attempt starts on a fresh
sandbox", but each one resumed the same sandbox and got the same refusal
(`journal_unsettled`).

**Expected behavior**
A follow-up claimed during an overlap waits for the warm session owner
to let go, then continues with no Retry banner and no second controller.
If the outgoing server dies with the session open, the takeover
continues after its ownership ends. A message that promises a fresh
sandbox is true, or the retries stop after a bounded number with a true
state.

**Steps to reproduce**
1. Run the restart-chaos release-swap preset with a warm (opencode or
codex) staging mix.
2. Let a run finish on the outgoing server after the traffic switch,
with a follow-up comment already queued.
3. See the follow-up fail with `remote_warm_session_unproven` and the
task go blocked.

**Paperclip version or commit**
master aaec67b891 and 45531b719e.

**Deployment mode**
Hosted deploy overlap (release swap) with Daytona sandboxes.

## What Changed

- `remote-warm-session-recovery.ts`: before live adoption, the run
proves that the server process that ran the finished turn is gone. It
uses the shared owner proof (`proveOwnerGone`): a PID check in the same
process space, or the `controller_instances` lease in another process
space. While that server is alive, the run defers with the existing
`native_controller_recovery_waiting_for_owner` wait and checks again
after 5 s. The wait ends when the server exits or is fenced, or when the
finished turn's warm window (idle timeout plus two minutes) ends.
- `native-session-executor.ts` and `run-start-handover.ts`: a server
that leaves new run starts to a replacement closes its idle warm
sessions after 1 s, through the existing idle-timeout close. A session
that becomes idle after that closes after 1 s too. This is the same
close that shutdown already does, done earlier.
- `predecessor-retirement-fallback.ts`: when a retry meets an unretired
owner on the same sandbox that its source already asked to replace, it
does not ask again. The run fails without another automatic retry, and
the task shows a Retry.
- `predecessor-retirement.ts`: the error no longer says "The next
attempt starts on a fresh sandbox", because the replacement is only a
request.
- `doc/execution-semantics.md`: documents the owner wait, the hand-off
at the swap, and the retry stop.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-controller-recovery.integration.test.ts`
- New: "waits while the outgoing server that ran the finished turn still
holds its warm session, then takes it over once that server exits" (red
on master: master returned a takeover while the owner was alive).
- New: "waits for a live outgoing server in another process space until
its controller lease ends" (red on master).
- New: "stops waiting for a live outgoing server once the finished
turn's warm window has closed" (bound check).
- `cd server && npx vitest run
src/__tests__/heartbeat-native-predispatch-retry.test.ts`
- New: "stops retrying with a truthful failure when the next attempt met
the same unretired sandbox instead of a fresh one" (red on master:
master scheduled another retry on the same sandbox).
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t
"replacement server starts"`
- New: "closes a warm session promptly when a replacement server starts
this server's runs (before/after the turn ends)" (both red on master).
- Focused suites: native-session-executor, remote-warm-session-recovery,
native-controller-recovery, native-recovered-owner-followup,
native-runner-restart-recovery, heartbeat-native-predispatch-retry,
environment-runtime-reusable-resume-replacement,
task-interruption-presentation, sandbox-runner-port-holders.
- `npx tsc --noEmit -p server/tsconfig.json` passes. `pnpm verify` exits
2 (a doc is a global CI input), so required CI supplies broad coverage.
- Not run here: the restart-chaos release-swap wave. The maintainer runs
it after merge.

## Risks

- A follow-up can now wait up to the outgoing server's remaining overlap
when that server cannot close its idle session. Before, it failed and
needed a Retry. The wait is bounded by the owner's exit or the warm
window.
- The outgoing server closes idle warm sessions earlier during an
overlap. The next turn then resumes from a suspended Runner instead of a
live one, as it already does after a shutdown.
- A retry chain that meets the same unretired sandbox twice now stops
one attempt earlier than the shared cap. The task shows a Retry instead
of looping.
- Rows without a recorded process space keep the existing one-release
local proof, so they behave as before outside the same host.
- Previous-release state: this change adds no persisted field.
Finalization rows saved by the previous release already record the
controller boot, PID, process space and start time, so they carry
forward. A follow-up already deferred by the previous release resumes
through the same wait code.
- Runtime cost: the owner proof adds one `ps` start-time read (same
process space) or one `controller_instances` row read (other process
space) per inspection. It runs once per attempt, at most every 5 s while
a live owner holds the session.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 11:59:17 -06:00
DottaandClaude Opus 5.5 aaec67b891 fix(native): close idle warm sessions in parallel across agents at shutdown (#21597)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - At shutdown, the server closes its idle warm Runner sessions before
it prepares the hot-restart snapshot
> - It closed them one at a time, and each full close (checkpoint, then
two stopped-copy collections) takes 3-5 s
> - In a zero-downtime swap test, five closes used the whole 20 s
pre-drain budget. Hot-restart preparation timed out, a checkpoint was
cut mid-way, and shutdown logged "hot-restart shutdown preparation
failed"
> - The same test logged 34 "Delivery reconciliation failed" errors, all
from directory cleanup retries that found the agent directory lock busy
> - This pull request closes idle warm sessions in parallel across
agents, one at a time within an agent, and treats a busy directory lock
as a quiet retry
> - The benefit is that shutdown preparation fits its budget with
several warm sessions, and expected lock contention no longer looks like
a failure

## Linked Issues or Issue Description

**What happened?**
`closeIdleWarmNativeSessions` awaited each warm session's full close in
turn. In swap 2 of a release-swap test, five idle sessions closed at
22:28:42, :46, :51, :56 and :59 (one "transport_close_requested"
diagnostic each). The `hot_restart_prepare` step then timed out at 19.9
s. Separately, `recoverDirectoryRelease` threw
`AgentDirectoryOperationBusyError` when another close held the agent
directory lock. The delivery coordinator logged each one at ERROR level.

I first checked if these sessions should have taken the cheap
released-session detach. They could not: `entry.released` is set only
after a successful `session.release()` through a release/resume-capable
plugin Runner task. The Runner state files from all sandboxes in the
test recorded only `run.prepare`, `session.open`, `turn.start`,
`runner.drain` and `runner.suspend` commands, and no `runner.release`.
The Daytona daemon logs show only `/process/execute` and
`/process/session` calls, and no Runner-task API. So no session in that
test was ever released, and a full close was the only correct shutdown
for them.

**Expected behavior**
Idle warm sessions of different agents close at the same time. Two
sessions of the same agent never close together, because they collide on
the agent directory advisory lock. A cleanup retry that finds that lock
busy stays pending until its claimed retry time and logs no error.

**Steps to reproduce**
1. Keep five or more idle warm sessions on remote sandboxes.
2. Send SIGTERM.
3. The closes run one after another and `hot_restart_prepare` times out.

**Paperclip version or commit**
`master` at bbed32450c.

## What Changed

- `closeIdleWarmNativeSessions` groups sessions by company and agent. Up
to four agents close at once, and each agent's sessions close one at a
time. The per-session close logic is unchanged.
- Idle-sleep retirement (`prepareIdleWarmNativeSessionsForSleep`) keeps
a concurrency of one, so it still checks drain authority before each
retirement.
- `recoverDirectoryRelease` catches `AgentDirectoryOperationBusyError`,
keeps the deferred retry time, and returns the pending entry instead of
throwing.

## Verification

- Red, then green (each failed against `master` source and passes with
the fix):
- `closes idle warm sessions of different agents in parallel and one at
a time within an agent` in
`server/src/services/native-runtime/native-session-executor.test.ts`.
Before the fix, the second agent's close was never started while the
first agent's close was pending.
- `keeps a cleanup pending without an error while another server holds
the agent directory lock` in
`server/src/__tests__/agent-directory-working-copies.test.ts`. It holds
the real Postgres advisory lock from a second connection. Before the
fix, the coordinator received an `AggregateError` with
`AgentDirectoryOperationBusyError`.
- `native-session-executor.test.ts` (1502 passed) and
`agent-directory-working-copies.test.ts` pass in full.
- `pnpm typecheck` in `server` passes.

## Risks

- Up to four warm closes now run at once, so shutdown makes more Runner
and database calls at the same time. The cap keeps this small.
- The order of entries in `failures` can change when several closes
fail.
- A busy release now waits for its deferred retry silently. A lock that
stays busy for a long time shows as a stuck pending cleanup, not as
repeated errors.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 04:15:19 -06:00
DottaandClaude Opus 5.5 1f388ed385 Start follow-up runs normally when a warm sandbox's window ends during its idle close (#21504)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm sandbox keeps an agent's Runner alive between runs, so the
next run on the task starts fast
> - When a run ends, the server closes the idle warm Runner in-process.
A separate SQL worker stops the sandbox when its warm window ends
(#21410, merged today)
> - Both steps start at the same moment. The sandbox stop wins, so the
idle close fails. The failed owner stays registered, and the next run
throws `native_instruction_materialization_handoff_unavailable` before
any provider work
> - Recovery must just work: a follow-up run must never fail because a
predecessor's warm copy is gone
> - This pull request lets the idle close finish before the sandbox
stops, prepares instructions fresh when the warm copy cannot be handed
off, and releases an owner whose sandbox the host already gave up
> - The benefit is that automatic follow-up runs start normally after a
warm window ends

## Linked Issues or Issue Description

Refs #21472 (base of this stacked PR), #21410 (introduced the
warm-window stop).

**What happened?**
In a Daytona two-server chaos wave (`w-20261113T200824-s7106`), three
automatic follow-up runs failed before any provider work with
`native_instruction_materialization_handoff_unavailable`: run
`5f68e194`, run `d6bd5a97` and run `d1198e40`. None of them was at a
release swap. In each case, the predecessor's idle close had failed
first with `provider_state_unreadable` / "Daytona sandbox lease … is no
longer active". The cause: the warm-window stop from #21410 stopped the
sandbox at the same moment that the idle close started. For the first
run's predecessor, the close was requested at 20:30:34 and the sandbox
got SIGTERM at 20:30:37.8.

**What did you expect?**
The follow-up run prepares its instructions fresh and continues. The
warm-window stop does not interrupt the host's own idle close.

**Steps to reproduce**
1. Run a warm Daytona task until it ends, so its warm window starts.
2. Let the warm window end while the idle timer closes the Runner.
3. Start a follow-up run on the same task. It fails with
`native_instruction_materialization_handoff_unavailable`.

Two of the three tasks then ended stranded. That downstream part (the
claim-gate cancellation with `execution_reconciliation_required` and the
stranded notice) is owned and fixed by another session. It is **not** in
this PR.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`
`claimWarmNativeInstructionCopy`: after it retires a stale warm owner
whose copy cannot be adopted, it returns `null`. The caller then
prepares instructions fresh. Before, it threw
`native_instruction_materialization_handoff_unavailable`.
- `server/src/services/warm-sandbox-expiry.ts`: the warm-window stop now
waits `WARM_SANDBOX_IDLE_CLOSE_MARGIN_MS` (2 minutes) past the window.
The in-process idle close (close grace plus the 30 s proof read)
finishes first.
- `native-session-executor.ts` idle timer: when an idle close fails, it
reads the lease rows for that sandbox. If every row is no longer
`active` or `retained` (the host gave the sandbox up), it releases the
in-process owner. The next run then prepares fresh.
- `doc/execution-semantics.md`: one paragraph about the stop margin and
fresh preparation.

## Verification

Red → green (each test fails on the base `35ba76d0ef` and passes on this
branch):

- `native-session-executor.test.ts` › "hands off collection before
probing or retiring a warm owner: unavailable" — red: throws
`handoff_unavailable`; green: returns `null` (fresh preparation).
- `native-session-executor.test.ts` › "forgets a warm owner whose idle
close failed only once the host gave up its sandbox: pending_cleanup /
released / retained" — red on all three; green: the owner is released
for `pending_cleanup` and `released`, and kept (retired by the
successor) for `retained`.
- `warm-sandbox-expiry.test.ts` › "leaves the host's idle close of the
warm runner time to finish before stopping its sandbox" — red: the
sandbox stops 3 s after the window; green: it stops only after the
margin. Two existing tests ("keeps a sandbox still inside its warm
window, and wakes when the window ends", "reconciles a warm sandbox left
running across a restart") now expect the deadline with the margin.

Local runs on this branch:

- `server`: `tsc --noEmit` clean.
- `vitest run src/__tests__/warm-sandbox-expiry.test.ts
src/services/heartbeat/instruction-working-copy.test.ts`: 24/24 passed.
- `vitest run
src/services/native-runtime/native-session-executor.test.ts -t "hands
off collection before probing or retiring a warm owner|forgets a warm
owner whose idle close failed"`: 10/10 passed.

Not run: a Daytona wave with this change.

## Risks

- Warm sandboxes now stop up to 2 minutes later than the configured
window. This is a small cost increase per warm sandbox. I chose a fixed
margin over an in-process "is a close running?" check because the stop
is pure SQL. The margin works when two servers run at the same time (the
server that runs the stop may not be the server that owns the Runner),
and it needs no busy-loop or cross-server signal.
- The margin does not cover a re-armed idle timer or a close that takes
longer than 2 minutes. Fresh preparation (fix 1) and the owner release
(fix 3) cover those cases.
- Not run on Daytona. The evidence is from the wave above and the unit
tests.
- In the wave, a `restore_failed` copy-back onto the just-restarted
sandbox was also seen. I did not find its root cause. It is not changed
here.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, test runs) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:47:42 -06:00
DottaandClaude Opus 5.5 2178c3df88 fix(native): keep a slow but live Runner on retry instead of replacing it (#21462)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a recovery read to a remote Runner goes unanswered, the next
attempt of the same run carries `replaceUnansweredRunner`
> - That attempt verified the Runner was alive and then refused to adopt
it anyway. On a resumed sandbox lease the reuse check then failed with
`resumed_not_suspended` and left the Runner running. Otherwise a second
Runner was launched, and its launch fence killed the busy Runner, which
lost the provider turn.
> - After a late reconnect a Runner is often slow, not dead: it is still
replaying the events it saved with no controller attached
> - This pull request makes the attempt check progress first. It adopts
a live Runner that is making progress and replaces only a Runner that
made no progress.
> - The benefit is that a slow but working Runner keeps its turn, and
the retry no longer causes the failure it was meant to fix.

## Linked Issues or Issue Description

**What happened?**

Hosted staging, release swaps on 2026-11-13. The retry after a
first-attempt timeout failed with these codes (counts from the traced
sample of 136 failures):

- `runner_ingress_unavailable: runnerd process failed:
runner_harness_state_mismatch: resumed_not_suspended` (part of 40
`ingress` failures)
- `runner_remote_process_identity_unavailable` (25): the Runner was gone
by the time the retry checked, because the launch fence or a previous
retry had killed it
- "restored the conversation after process loss, but…" /
`provider_turn_lost_on_restore` (10): the replacement Runner restored
from the old state and the provider turn was gone

Runs that crossed a swap with a reconnect gap under 150 s: 0 of 9
failed. Over 420 s: 14 of 15 failed. The process ID of the Runner
changed inside the same sandbox in the failed runs, which shows the
replacement.

Code path: `nativeRecoveryCommandUnanswered` grants an extra attempt
with `replaceUnansweredRunner: true`. `verifyRemoteRunnerOrSettledExit`
verifies the Runner is alive and then returns `undefined` because of
that flag, so `adoptedProcess` is `undefined`.

**Expected behavior**

A live Runner that is making progress is adopted. A Runner that made no
progress is stopped with proof and then restored from its own journal.

**Steps to reproduce**

1. Reattach a remote run whose previous attempt left a recovery read
unanswered (`replaceUnansweredRunner: true`), with the Runner alive and
its saved event sequence past the controller's last acknowledgement.
2. On master the attempt launches a second Runner, or fails with
`resumed_not_suspended` on a resumed sandbox lease.
3. The new tests in `native-session-executor.test.ts` and
`native-runner-restart-recovery.integration.test.ts` do this.

**Paperclip version or commit**

`master` at 426e8a35dc.

## What Changed

- `native-session-executor.ts` (`verifyRemoteRunnerOrSettledExit`): with
`replaceUnansweredRunner`, the attempt adopts the live Runner, launching
nothing and signalling nothing, when:
- the Runner's saved event sequence is past the controller's last
acknowledgement, or
- its saved state (event and command sequences, generation, lifecycle)
changes within a bounded wait of about 10 s.
- The baseline is the first successful state read; every successful read
is also compared against the controller's acknowledgement. A Runner
whose state cannot be read at all during the wait is adopted, not
replaced: missing evidence is not proof of a wedge. A Runner whose
identity changes during the wait fails closed.
- A settled exit during the wait keeps the attempt allowance that
admitted the replacement, so it also restores on the extra fourth
attempt.
- Only a Runner that made no progress is replaced. The relaunch's
existing launch fence stops it and its process lineage with proof, and
the new Runner restores its own journal. A resumed sandbox lease no
longer refuses that relaunch with `resumed_not_suspended`.
- A Runner that exits during the wait follows the existing exited-Runner
rules: a settled suspension restores, and a mid-turn exit stays
fail-closed.
- The wait runs only on this recovery path, never on a normal run start.
- `doc/execution-semantics.md`: describes adopt versus replace.
- The claim flag `replaceUnansweredRunner` is set in claim logic that
another session owns. This PR does not change it; it changes only what
the executor does with it.

## Verification

Red on master, green on this branch:

- `native-session-executor.test.ts`
- `adopts a live Runner that emitted events past the controller's last
acknowledgement`
- `adopts a live Runner whose saved state advances within the bounded
wait`
- `hands a live Runner that made no progress to the relaunch, whose
launch fence stops it, without signalling it here`
- `restores from the journal of a Runner that suspended and exited
during the wait`
- `stays fail-closed for a Runner that exited mid-turn during the wait`
- `native-runner-restart-recovery.integration.test.ts`: `reattaches a
live %s Runner whose recovery read went unanswered on a resumed sandbox
lease` (progressing, wedged)
- Review fixes, `native-runner-restart-recovery.integration.test.ts`
(the first four were red before the fix):
- `adopts a live Runner whose first state read failed and whose later
reads advance`
- `adopts a live Runner whose first readable state is already past the
controller's acknowledgement`
- `does not replace a live Runner whose state could not be read at all
during the wait`
- `restores from the journal of a Runner that suspended and exited
during the wait on the admitted fourth attempt`
- `fails closed when the Runner's identity changes during the wait`
(pins existing behavior)
- `native-remote-retry-admission.test.ts`: unchanged, still passes.

Local runs on this branch: `tsc --noEmit` is clean for `server` on this
branch (master c6b497d831, which contains #21442).
`native-runner-restart-recovery.integration.test.ts`, the
adopt-or-replace block: 12 of 12 pass. The full
`native-session-executor.test.ts` and
`native-remote-retry-admission.test.ts` runs did not finish on this
shared machine (load 20 to 96); CI ran them on this head and they
passed.

Conflict resolved: master added a relaunch branch for a Runner that
exited for a fresh bootstrap (#21374) at the same place. Both branches
are kept; the cheaper check for an unanswered Runner that made no
progress comes first.

Daytona results that exist. They ran on a combined branch with this PR,
#21450, #21442 (merged) and #21362 (merged), so they cannot be credited
to this PR alone:

- G5 baseline (master d1d71a7755; 480 s delay injected on the first
identity write per run): one Claude run reproduced the staging chain:
authentication 419 ms, identity +423.5 s, attempt 1 `native session
provider recovery timed out after 900000ms`, then the retry failed in 6
s with `runner_harness_state_mismatch: resumed_not_suspended`.
- G5c (same cell, combined branch 588c6ba267): 6 of 6 tasks finished
hands-off. The chain did not recur. All runs that crossed swap 2 were
adopted on the same Runner process ID.
- Caveat: the adopt-or-replace retry path that this PR changes was never
exercised in a cell. In G5c no first attempt failed, because all
identity writes started together 4 to 5 s after the claim. G5c also ran
before the phase timing fix in #21450.

## Risks

- Not proven on Daytona: the adopt-or-replace path itself (see the
caveat above).
- Without #21442 (merged) an adopted Runner can still time out on its
first command while it replays a backlog. This PR stops the replacement;
#21442 (merged) makes the adopted Runner answer.
- A Runner whose state cannot be read during the whole wait is adopted.
If it is in fact wedged, the adopted attempt times out again and the
attempt limit bounds it.
- A wedged Runner that still emits a few events counts as progressing
and is adopted. The existing attempt limit bounds this.
- The progress check adds a wait of up to about 10 s on this recovery
path only. A normal run start has no added wait.
- The claim construction (`native-restart-recovery.ts`) is owned by
another session and is unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 18:56:11 -06:00
DottaandClaude Opus 5.5 47a2c82d35 fix(heartbeat): reach the previous server's live runs first after a release swap (#21450)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - During a release swap, the outgoing server stops and the incoming
server takes over the live runs on remote Runners
> - The incoming server reached those runs late: it waited up to 60 s
for the outgoing server's controller lease to lapse, re-verified each
run's whole log from object storage, and deferred a reattach when the
agent directory was busy. Queued starts competed with the reattaches for
the same resources.
> - On hosted staging, the later a live run was reached, the more likely
it was to fail. Runs reached in under 150 s never failed; runs reached
after 420 s almost always failed.
> - This pull request hands live runs over at the start of shutdown,
makes a reattach skip work that the same run already did, and holds
queued starts while handed-over runs are being reached, with a 180 s
cap.
> - The benefit is that live runs are reached within seconds of a swap
instead of minutes, so they continue instead of failing.

## Linked Issues or Issue Description

**What happened?**

Hosted staging, release swaps on 2026-11-13 (06:05Z to 09:45Z). For each
run that crossed a swap we measured the gap from the last Runner output
to the moment the new server recorded the Runner's process identity:

- gap under 150 s: 0 of 9 runs failed
- gap over 420 s: 14 of 15 runs failed
- median gap: 180 s for runs that resumed, 470 s for runs that failed

In one traced run the incoming server waited on
`remote_controller_lease_live` three times (09:14:21, 09:15:18,
09:16:17), claimed at 09:16:59, and spent 85 s in `task.prepare`. Of
that, `heartbeat.prepare.run_output` took 58.7 s, because it re-read the
run's whole log from object storage. A control run spent 23 s in
`task.prepare`. Queued starts used the same preparation path, the same 8
object-I/O slots and the same database pool, and nothing gave a reattach
priority. The first-attempt failures were `PRP command session.snapshot
timed out` and `codex_history_read_failed`. The retry then failed with
`runner_harness_state_mismatch: resumed_not_suspended`,
`runner_remote_process_identity_unavailable` or
`provider_turn_lost_on_restore`.

**Expected behavior**

Every live run is reached within about 30 s of the swap. Queued work
starts after that, or after a bounded wait.

**Steps to reproduce**

1. Run a two-server release swap with live remote runs and queued runs.
2. Observe `native.recovery.transition` "waiting" events with
`remote_controller_lease_live`, then the claim, then a long
`heartbeat.prepare.run_output` span.
3. The new tests in `reattach-phase.test.ts` and
`heartbeat-native-early-hand-off.test.ts` model this.

**Paperclip version or commit**

`master` at 426e8a35dc.

## What Changed

- Early hand-off at shutdown (`heartbeat/recovery.ts`,
`server/src/index.ts`): in deploy-overlap mode, the outgoing server
releases the remote controller lease of its live runs at SIGTERM
(`recoveryState` `awaiting_runner_reattach`). A run another server
already took over is left alone. The incoming server no longer waits for
the lease to lapse.
- Reattach fast path:
- `heartbeat/run-output.ts`, `native-runtime/native-run-log-export.ts`:
a reattach starts without waiting for the existing log to be
re-verified.
- `heartbeat/agent-directory-busy.ts`, `heartbeat.ts`: a reattach that
meets a busy agent directory waits in process (at most 30 s) instead of
being deferred to a full re-preparation. An ordinary start is still
deferred at once, with no added wait.
- Reattach phase lifecycle (`heartbeat/reattach-phase.ts`): the watch is
a worker on the shared delivery coordinator, queue `reattach-phase`.
- Trigger: one startup listing of a previous server's live runs; then
the hand-over notification (`notifyReattachHandOver`, sent in the
hand-over transaction); the earliest lease deadline it watches (a killed
owner); and a 2 s re-check only while handed-over runs are still
pending, which ends with the phase (cap 180 s). The bounded re-check
exists because a run can also end on another path that sends no
notification to this queue.
- Source of truth: `native_run_finalizations` and `heartbeat_runs` rows.
- Quiet condition: nothing pending and no watched lease, so no deadline
and no SQL. It stops with the coordinator at shutdown.
- Reached hook for every driver (`native-session-executor.ts`): a
reattached run counts as reached when the native session runtime hands
the executor its recovered or opened session (`onSession`). Every driver
reaches that point only after the Runner answered this server's attach
and recovery commands, including an adopted `RunnerdExternalDriver`
Runner, which records no new process identity. The identity hook stays
as an earlier signal.
- Claim wake: when the reattach-phase worker finds runs that just became
claimable (hand-over notification or a killed owner's lease deadline),
it starts this server's restart-recovery claim for them at once. A
validation cell showed the claim otherwise waited about 21.5 s after the
hand-over.
- `recovery/assigned-work-sweep.ts`: the `deferred_reattach_phase`
return now sits before every action path of `act()`, except starting an
overdue saved-work recovery (it resumes an existing run).
- The in-transaction admission re-check of a queued claim
(`confirmRunAdmissionStillOpen`) also honours the reattach phase. A
phase that begins during a claim returns the run to the queue through
the existing rollback.
- A reattach whose log re-verification runs in the background awaits it
before the log is finalized (`settleDeferredLogRecovery`, both
finalization sites in `run-completion.ts`).
- Reattach phase (`heartbeat/reattach-phase.ts`,
`heartbeat/run-admission.ts`): `isReattachPhaseActive()` is true while
runs that a previous controller handed over (lease released or observed
expired, or claimed by this server's restart recovery) are claimable but
not yet reached. A run is reached when its process identity is recorded.
Authentication is logged as a mark. The cap (180 s,
`PAPERCLIP_REATTACH_PHASE_CAP_MS`) counts from the moment the runs
became claimable. Later batches join or re-arm the phase. Runs reached
after 150 s are counted and logged.
- Gates on the phase:
- `claimQueuedRun`, start-next, resume-queued and dispatch in
`heartbeat/queue.ts`, `run-lifecycle.ts` and
`native-runner-execution-driver.ts` leave queued runs queued.
- `execution-recovery-resolution.ts`: one line, the first line of
`deliverReconciledExecutions`.
- `recovery/assigned-work-sweep.ts`: one line, outcome
`deferred_reattach_phase`.
- `heartbeat/recovery.ts`: a one-line claim hook
(`noteReattachRunsClaimed`) after restart recovery claims runs.
  - The owner of those three files agreed to these one-line gates.
- `tests/restart-chaos/daytona-image.ts`:
`PAPERCLIP_CHAOS_TEST_ACCEPT_IMAGE_CONTENT_ID` lets a local wave run a
controller-only branch against an existing Daytona image. It accepts a
content-ID mismatch only for the exact content ID of the named image,
and it prints a TEST OVERRIDE warning. This is harness-only.

## Verification

Red on master, green on this branch:

- `heartbeat/reattach-phase.test.ts` (27 tests). Five of the six
two-server scenarios were red on the earlier phase timing:
- `does not hold queued starts while the previous owner is alive and not
handing over`
- `closes the gate when the previous owner hands its runs over, and
counts reach latency from the hand-over`
- `runs the cap from the hand-over, so a wedged run ends the phase
without the overlap eating it`
- `begins when a killed owner's lease is observed expired, without any
hand-over`
- `re-arms for a later batch, including runs the previous owner started
after discovery`
- `closes the gate at once when this server claims a run between two
lease reads, dated from the hand-over`
- plus `claimQueuedRun leaves a queued run queued and unstarted, in the
admission gate's shape`, `is the same gate the assigned-work sweep and
reconciled continuation delivery read`, `ends at the cap, counted from
when the runs became claimable, so one wedged Runner cannot hold run
starts`
- `heartbeat-native-early-hand-off.test.ts`: `releases the remote
controller lease of %i live runs at once` (1, 20, 40 runs; 40 runs in 56
ms), `leaves a run another server already took over alone`, `holds the
incoming server's queued starts only from the hand-over, not while the
outgoing server still drives its runs`
- `heartbeat/run-output.test.ts`: `lets a reattach start before its
existing log is re-verified`
- `heartbeat/agent-directory-busy.test.ts`: `defers an ordinary start at
once, with no wait added to the run-start path`, `lets a reattach wait
its turn in process instead of a deferred full re-preparation`, `still
defers a reattach once its bounded wait runs out`
- `tests/restart-chaos/daytona-image.test.ts`: `accepts a mismatch only
for the exact image content ID named by the test override`
- Review fixes (red before the fix, green after):
`reattach-phase.test.ts` `starts the claim of handed-over runs within a
second of the hand-over notification`; `assigned-work-invariant.test.ts`
`takes no action while handed-over live runs are being reached, and acts
once the phase ends`; `heartbeat-native-early-hand-off.test.ts` `never
lets the outgoing server claim back a run it handed off, while the
incoming server may take it over`; `reattach-phase.test.ts` `retries a
failed first listing instead of going quiet, and still finds the
previous owner's runs`, `keeps a hand-over notification whose listing
failed until a listing succeeds`, `moves the cap earlier when a claimed
run turns out to have been handed over before its claim`;
`queue.test.ts` `returns the claim to the queue when a reattach phase
begins while the claim is being checked`; `run-completion.test.ts`
`finalizes the run log in completeRun|failRun only after a reattach's
deferred log recovery settles`. `native-session-executor.test.ts` `ends
the reattach phase once a reattached %s session is driven by this server
again, without the cap or manual settling` over codex_app_server,
opencode_server, acpx_runtime, muse_external, grok_bot_mcp and
openai_dot_mcp (6 of 6 red without the hook; the test drives the
executor with a mocked session runtime, not each real driver). Lifecycle
tests for the new worker: `does no database work and keeps no deadline
on an instance with nothing to reattach`, `waits for the hand-over
notification or the lease deadline instead of polling while the previous
owner drives its runs`, `begins at a killed owner's lease deadline,
without any hand-over or notification`, `stops with the delivery
coordinator at shutdown and ignores a later hand-over`.

Local runs on this branch: `tsc --noEmit` is clean for `server`. Passing
on this branch: `reattach-phase.test.ts`,
`heartbeat-native-early-hand-off.test.ts`, `run-output.test.ts` (53
tests together), `agent-directory-busy.test.ts`,
`heartbeat-native-restart-suspension.test.ts` (115 of 115, alone), the
new cases in `queue.test.ts` and `run-completion.test.ts`. Under load 30
to 96 on this shared machine, the full `queue.test.ts` and
`recovery-wake-herd.test.ts` runs stalled in their embedded-Postgres
setup; CI supplies them.

Daytona results that exist (local servers with Daytona sandboxes,
release-swap cells). They ran on a combined branch with this PR, #21462,
#21442 and #21362 (merged), so they cannot be credited to this PR alone:

- G5 baseline (master d1d71a7755): SIGTERM to PRP authentication 21.5 to
38 s at swap 1.
- G5c (combined branch 588c6ba267): SIGTERM to authentication 6.2 to 7.7
s at swap 1. All runs that crossed swap 2 were adopted on the same
Runner process. The staging chain (`provider recovery timed out after
900000ms`, then `resumed_not_suspended`) did not recur.
- Caveat: G5c ran before the phase timing fix in this PR. In G5c the
phase opened at process start, its 180 s cap ran out during the overlap,
and swap 2 ended `cap_reached` with 0 runs reached, 1 s before the lease
claims. The timing fix (the phase begins when the runs become claimable)
has not run on Daytona.
- G6c (12 live runs and 20 queued, two graceful swaps, no injection,
combined branch 588c6ba267): 0 failed runs at either swap; 11 of 11
survivors at both.
- Swap 1: hand-off of 11 runs in 291 ms; claims 0.3 to 2.9 s after
hand-off; authentication 7.6 to 12.5 s and identity 8.3 to 13.3 s after
SIGTERM. The 30 s target was met.
- Swap 2: hand-off took 8,562 ms and `hot_restart_prepare` timed out at
19 s, so the outgoing server drained gracefully. The incoming server did
not claim on the hand-over; it claimed only when the outgoing server
exited, 33 s after SIGTERM. Authentication was 46 to 54 s and identity
47 to 57 s after SIGTERM. The target was missed. Why the claims waited
for the exit is not explained.
- The host load was 49 to 70 from other work. 7 of 32 tasks timed out in
the queue or budget after the swaps; we suspect the overload.

## Risks

- Product trade-off: while handed-over live runs are being reached,
queued starts wait, up to the 180 s cap. On an instance with no
handed-over runs, nothing waits.
- Not proven: two-server behavior with the phase timing fix and the
notification-driven watch (no cell has run them); warm standby becoming
active; runs that the outgoing server starts after discovery.
- Early hand-off at SIGTERM orphans a server-side exec tool that is in
flight. With #21442 the next controller settles it as interrupted with
an unknown outcome; it is never run again.
- Swap 2 in G6c: the incoming server claimed only at the outgoing
server's exit. That is unexplained.
- Cross-session ownership: the gates in
`execution-recovery-resolution.ts` and
`recovery/assigned-work-sweep.ts`, and the claim hook in
`heartbeat/recovery.ts`, are in files that the restart-recovery work
owns. They are one-line gates agreed with that owner. That owner said
the gate did not need to be inside the claim transaction; that was
before the phase could begin during a claim. After review, the
in-transaction re-check in `heartbeat/queue.ts` also reads the phase,
through the existing rollback. #21427 (open) also edits
`execution-recovery-resolution.ts`, `heartbeat/queue.ts` and
`recovery/assigned-work-sweep.ts`; whichever merges second needs a
rebase. `native-runtime/native-restart-recovery.ts` (claim logic, owned
by the restart-recovery work): a 3-line guard in
`claimNativeRestartRecoveries` returns a run this process handed off at
shutdown as `blocked` / `handed_off_to_next_controller` before takeover
is evaluated, so the outgoing server never claims back a run it handed
off. #21478 (merged) added a sweep path that starts an overdue
saved-work recovery; it is deliberately not gated by the reattach phase,
because it resumes an existing run (reattach-like work), which the phase
makes room for rather than competes with.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 18:55:31 -06:00
Nicky LeachandClaude Opus 5.5 6d8ebc4a85 fix(secrets): mask secret-bound environment values in run output (#21341)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can bind a company secret to an environment variable for an
agent, an environment, a project, or a routine
> - Core resolves these values for each run and gives them to the agent
process or to the Runner
> - Core stores the run output: the run log, the run excerpts, adapter
events, and native (Runner protocol) transcript events
> - The run-output redactor did not know the resolved secret values.
Only failure diagnostics and the `meta.env` keys of `adapter.invoke`
were masked
> - So when an agent ran `env` or `echo $STRIPE_KEY`, Core stored the
secret in plaintext, and every person who can view the run could read it
> - This pull request gives the resolved values of secret-bound keys to
the run-output redactor and to the native control-plane port
> - The benefit is that an accidental print of a secret does not put the
secret into stored run output, for local and native runs

## Linked Issues or Issue Description

**What happened?**
An agent printed a secret-bound environment variable during a run (for
example with `env` or `echo $STRIPE_KEY`). Core stored the value in
plaintext in the run log, in `stdoutExcerpt` and `stderrExcerpt`, and in
the transcript events. Anyone who can view the run can read the value.

**Expected behavior**
Core masks the value of each secret-bound environment variable before it
stores run output. Core does not mask plain (non-secret) configured
values.

**Steps to reproduce**
1. Create a company secret and bind it to an agent environment variable
with `secret_ref`.
2. Configure the agent so that its command prints the variable.
3. Run the agent and open the run log. The value shows in plaintext.

**Related**
Refs #21330. That PR keeps configured secrets out of OpenCode's own
storage and model requests inside the Runner. This PR is the Core side:
it masks secrets in the run output that Core stores, for every run type.
The two changes are independent.

## What Changed

- `secretEnvironmentValues(env, secretKeys)` in
`agent-identity-redaction.ts` selects the raw values of keys in
`secretKeys` (the keys that came from `secret_ref` or `user_secret_ref`
bindings). `secretOutputForms` derives the forms to mask: the full
value, the value without outer whitespace, the JSON-escaped form, and
each line of a multi-line value (PEM armor lines are not added). It
skips a form that has fewer than 8 characters after trimming. This is
the same 8-character limit that diagnostic redaction uses for unknown
values.
- The output redactor keeps these secrets in a separate set, beside the
existing agent private-key values. The private-key values are masked
everywhere, as before. Environment secrets are masked only in output
text. They are not masked in protocol fields: `schema`, `type`, `kind`,
`status`, `state`, `outcome`, `stream`, `channel`, `role`, `level`, ids,
refs, codes, revisions, versions, digests, timestamps, and names that
end in these words (for example `eventType`, `turnTerminalState`,
`reportedWorkDisposition`). So a secret that equals a protocol word (for
example `completed`) cannot change an event type or a terminal state.
- The streaming match now scans left to right. At each position it tries
the longest value first. If a longer value could still start at that
position, it holds the text until the next chunk, also when a shorter
value already matches there. So the tail of a longer secret cannot leak
when a shorter secret is its prefix.
- `prepareHeartbeatExecutionConfiguration` adds the resolved secrets to
the run's output redactor immediately after it resolves them. This one
redactor already masks the local adapter log stream, the excerpts, live
log events, adapter events, `adapter.invoke` metadata, the adapter
result, and the canonical native log export.
- For a native run, `executeHeartbeatRuntime` stores the delivered
secrets (only values that have a form to mask) encrypted under a new
context key, `paperclipOutputSecretRedactions`, in one transaction. It
then adds all stored values of the run to the redactor. A run that
reconnects to a live Runner after a `"latest"` secret rotation then
masks the value that the Runner holds and the new value. This store
reuses the encryption and fingerprint logic of the run redaction
registry, but it is separate: API read redaction does not apply its
values, so a secret cannot change ids, event names or terminal states in
API responses. API reads hide the new key like the existing registry
key.
- `executeHeartbeatRuntime` gives the native executor the run's secrets
and the observability credentials (`observabilitySecretValues`).
- The native executor gives the secrets to its failure-diagnostic
redactor and to `PaperclipControlPlanePort` (`secrets`). The port masks
them in each PRP event before it persists the event, with the existing
delta-aware handling for values that are split across `item.delta`
events. This covers native runs on the local Runner and on plugin-hosted
agent computers.
- After a controller restart, a new control-plane port cannot know the
text that the old port held back, because that text is never stored. On
the first delta of each stream it has not seen, the port masks leading
text that matches the tail of a secret (4 or more characters). It skips
text that can start a secret, and it keeps checking the stream while a
delta falls entirely inside such a tail.
- The native executor classifies failures from the original error. It
masks only the agent private key in place, as before. It masks
environment secrets in a separate redactor that it uses only for text
that it writes or logs (`failureDetail`, failure messages, stderr
diagnostics). Core masks the rethrown error before it stores it.
- `coalesceIntoActiveExecutionRun` (wake queue) writes the merged
context with `withRunRedactionHistory`, which takes
`paperclipSecretRedactions` and `paperclipOutputSecretRedactions` from
the locked current row. Before, a coalesced wake built from an older
context could delete either key. This also protects the existing
registry. This is a change outside the secrets module, in the one place
that replaces a running run's context concurrently.
- When an adapter or native run fails, the heartbeat keeps the original
error for retry decisions and rethrows it. The copies that it saves in
`workspace_finalize` metadata (`errorMessage`) and in a failed session
goal action go through the run's output redactor first.
- `docs/deploy/secrets.md` describes the mask and its limits.

## Verification

- `server/src/__tests__/run-output-secret-env-redaction.test.ts` (new,
embedded Postgres): a real `process` adapter run with a `secret_ref`
bound to `"latest"`. The agent prints the secret split across two
writes, in JSON, and on stderr. The stored log, run row, and run events
do not contain the value. The plain value and a 5-character secret are
not masked. The test then rotates the secret and runs again. The second
run masks the rotated value, and a SHA-256 digest in the output proves
the run received the rotated value. Red-green: it fails without the new
line in `execution-configuration.ts`.
-
`server/src/services/native-runtime/paperclip-control-plane-port.test.ts`:
the port masks a secret in a persisted tool event and in a value split
across two `item.delta` events; plain and short values stay (red-green:
fails without the port change). A new test binds secrets equal to
`interrupted`, `completed`, `succeeded` and `satisfied`, then appends a
`turn.interrupted` event, checkpoints, and completes the run. The event
type, payload status, terminal states and criterion status stay
unchanged and the result is saved (red-green: fails when protocol fields
are masked).
- `server/src/services/heartbeat/runtime-execution.test.ts`: a value
stored by an earlier attempt, the current value, and an observability
header value all reach native dispatch and the heartbeat redactor. Short
values are not stored. The shared run redaction registry stays empty,
and an API read hides the new key (red-green: fails without the store
read).
- `server/src/services/native-runtime/native-session-executor.test.ts`:
a native failure that contains the secret is masked in failure writes,
logs and the rethrown error, and the executor gives the secrets to the
control-plane port.
- `server/src/services/agent-identity-redaction.test.ts` (new): value
selection, the minimum length, multi-line and JSON-escaped forms, nested
values, every split point of a streamed value, overlapping secrets
(`abcdefgh` and `abcdefgh1234567890`) split at the shorter value,
protocol fields in events, terminal states and deltas, and secrets added
by a later attempt.
- Restart, retry and coalescing (each fails without its fix): a port
test restarts between two deltas of a secret and stores `echo ` and
`***REDACTED*** done`; an executor test binds `timed out` and keeps
`provider_session_open_timeout`; a wake-queue adapter test keeps both
redaction keys when a coalesced wake replaces the context.
- `server/src/services/heartbeat/run-output.test.ts`: the local output
path masks the log, excerpt, live event, adapter event, and adapter
metadata.
- `pnpm --filter @paperclipai/server typecheck` passes. The changed test
files and the other users of the redactor (`agent-identity`, `run-log`,
`heartbeat-case-handoff`, `native-run-log-export.integration`) pass in
full locally: 1986 passed, 7 skipped; the 6 run-redaction-registry test
files also pass (231 tests) (macOS, embedded Postgres). `pnpm verify`
exits 2 (broader verification required) because `docs/deploy/secrets.md`
is a global CI input, so required CI supplies broad coverage.

## Risks

- This is a guard against accidental leaks. It is not a security
boundary. An agent can change a value before it prints it (for example,
encode it in base64 or print one character at a time), and Core cannot
detect that.
- Core does not mask a secret value or line that has fewer than 8
characters. This prevents a short value from erasing ordinary output. A
secret that short is not masked in run output.
- Core does not mask environment secrets in protocol fields (see What
Changed). If a secret appears in such a field, it stays. These fields
hold protocol words and generated identifiers, not agent output.
- If a value is common text of 8 or more characters, Core masks that
text wherever it occurs in that run's output text. This only affects
runs that bind that secret.
- If a key is secret-bound in one source and a later source sets the
same key to a plain value, `secretKeys` still includes the key, so Core
masks the plain value. This is the existing meaning of `secretKeys`.
- For native runs, Core now stores the delivered secrets encrypted in
`contextSnapshot` under `paperclipOutputSecretRedactions`, with the same
local encryption as the existing run redaction registry. It adds one
transaction per native run that has secrets. API reads do not apply
these values and hide the key.
- After a controller restart in the middle of a secret, a tail shorter
than 4 characters can still be stored. A new stream that starts with the
tail of a secret (4 or more characters) is masked at its start even
without a restart. This is a small amount of extra masking.
- Core masks values at write time. Run output that Core stored before
this change is not changed.
- No schema change and no migration. The new fields are optional and
in-process only.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 16:14:41 -08:00
Nicky LeachandClaude Opus 5.5 5df1ada9c1 feat(plugins): send configured environment on plugin task resume (#21352)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #21335 lets a plugin task provider that restarts the Runner on
`resume` opt in (`resumeCredentialVersion: 1`) to receive the claimant
run's harness credential
> - A provider restarting the Runner also needs the agent's configured
environment, which Core sends on `resume` only during a session handoff
> - Without it, a Runner a provider moves to a new computer starts
without the agent's configured environment variables
> - This pull request widens that opt-in to both launch inputs, under a
name that says so, before any provider adopts the old name
> - The benefit is that a provider can restart a parked Runner on resume
with the same credential and environment a `submit` would carry

## Linked Issues or Issue Description

Refs #21289
Follows: #21335

**What existing behavior does this improve?**
The opt-in resume launch inputs added in #21335.

**Subsystem affected**
Plugin SDK environment task schema (`describe` capability), the plugin
Runner task owner's `resume`, and the plugin environment task docs.

**Current behavior**
With `resumeCredentialVersion: 1`, `resume` carries `credential`.
`configuredEnvironment` is sent on `resume` only for a session handoff.

**Proposed behavior**
The capability is renamed `resumeLaunchInputsVersion: 1`. With it,
`resume` carries `credential` as before and also `configuredEnvironment`
(same builder as `submit`) when the provider advertises
`configuredEnvironmentVersion: 1`. Session handoff is unchanged.

**Reason and benefit**
A provider that restarts the Runner on resume gets both launch inputs it
needs.

**Breaking changes**
The capability name changes from `resumeCredentialVersion` to
`resumeLaunchInputsVersion`. No provider advertises the old name yet (it
merged in #21335 and has no adopter), so nothing changes behavior.

## What Changed

- Rename `resumeCredentialVersion` → `resumeLaunchInputsVersion` in the
SDK `describe` result schema, executor plumbing, owner input, docs and
tests.
- With the capability, `resume` also adds `configuredEnvironment` when
the provider advertises `configuredEnvironmentVersion: 1`. A normal
resume does not mark the variables as delivered, since a live Runner
keeps the environment it started with.
- Docs: the "Release and resume" section describes both fields; the
"Configured environment" section points to it.
- The runner-e2e SDK public-input pin is updated.

## Verification

- SDK vitest `tests/environment-task-release-resume.test.ts` and
`tests/environment-tasks.test.ts`: 68 passed.
- Server vitest `plugin-runner-task.test.ts` and
`environment-task-runtime.test.ts`: 153 passed, including a normal
resume with the capability (credential sent, variables added only with
`configuredEnvironmentVersion: 1`, delivered flag unset) and redaction
of both fields from metadata writes and sanitized errors.
- `native-session-executor.test.ts -t "uses a task-only"`: 5 passed.
- runner-e2e `source-sdk-evidence.test.ts` and
`source-sdk-public-type.test.ts`: 46 passed.
- `pnpm typecheck` in `server/` (builds shared and plugin-sdk) passed.
- Not run locally: the full suite and `pnpm verify`; required CI covers
them.

## Risks

- A provider restarting the Runner gets the claimant run's current
variables, which can differ from the ones the parked Runner started
with. That is intended.
- Rollback: revert this pull request. No migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs), implemented by a sub-agent and reviewed
before opening.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 13:46:37 -08:00
Devin FoleyandPaperclip 12aa08721c feat(ssh): use the agent AI account in new environments (#21248)
Use the agent-selected AI account for new SSH environments. Preserve
remote-machine authentication for existing environments and expose it as
an advanced option. Keep managed credentials private while preserving
supported conversation state and credential refreshes across turns.

Validated with focused route, adapter, lifecycle, and real-process tests;
final CI passed on d634944df1.
Greptile Apex reviewed that revision at 5/5 with no unresolved threads.
External neutral security reports remain documented in the pull request.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-13 13:20:13 -08:00
DottaandPaperclip 6ad4529c5a fix(native): preserve pack proof across interrupted reattachment (#21246)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server can connect again to a live Runner after a restart.
> - Each new controller claim changes provider-pack admission to
pending.
> - The claim kept the previous verified proof only in local memory.
> - If recovery failed before its callback, the next claim lost that
proof.
> - This pull request saves a small reference to the verified proof with
the claim.
> - The next claim can check the same receipt without weakening
admission checks.

## Linked Issues or Issue Description

**What happened?**
A restart recovery attempt timed out before its admission callback. The
next attempt refused the saved pack with `source_state_pending`. It then
failed with `native_provider_artifacts_admission_unverified`.

**Expected behavior**
The next attempt can reuse the previous verified proof for the same
receipt, session, Runner, and lease. Current admission must remain
pending until its normal callback succeeds.

**Steps to reproduce**
1. Save verified provider-pack admission for a run.
2. Claim that run again. Commit the pending admission state.
3. Interrupt recovery before the session admission callback.
4. Claim the same run again and reconnect to the same Runner.
5. Observe that the old code refuses the saved receipt.

Related work: #20682 and #20850 address lock waits. #21085 adds refusal
logs. #21161 addresses command waits after restart. This change
addresses lost admission proof across interrupted claims. The original
transport timeout remains a separate investigation.

## What Changed

- The admission module prepares the claim patch and its recovery source.
- The claim saves a versioned reference with the source run, receipt
digest, and previous verified state. It adds no copy of provider files
or credentials.
- A later pending claim can use that reference only if the receipt is
unchanged. Normal company, agent, session, Runner, and lease checks
still apply.
- Missing proof, unknown state, a changed receipt, or a different Runner
stays refused. The callback clears the reference when it records its
decision.
- The executor uses this helper under its existing claim lock. Existing
artifact-retention and ownership checks stay in place.
- The execution guide states the previous-release behavior. Old verified
records seed the reference. Old pending records without a reference
remain refused. Older readers ignore the field and keep their existing
refusal behavior. No migration is needed.

## Verification

- The new interrupted-claim regression failed before the fix: admission
returned false instead of true.
- Focused admission, PostgreSQL fencing, and executor lease tests passed
after the rebase: 61 tests on commit c8123cc35d.
- The database case saves two interrupted claims, reads their persisted
state, checks a different Runner is refused, and publishes verified
admission for the original Runner.
- Negative cases cover missing proof, unknown state, a changed receipt,
a different source run, a different Runner, and a different lease.
Existing company and claim fences also pass.
- `pnpm verify` returned exit 2 because the execution guide requires
broader CI. Required CI must cover the final commit.
- Local server typecheck was killed with SIGKILL, including its
single-thread retry. This is an environment gap. Required CI must
provide typecheck and build results.
- Repository preflight installed dependencies and built shared packages.
It could not finish Runner preparation because cargo is absent.
- A live OpenCode restart workflow and recovery of the original task are
not verified. No task replay or deployment was performed.

## Risks

- The added profile field carries historical evidence. It does not mark
the current claim verified. Its use is limited to an unchanged receipt
with the normal identity checks.
- Existing pending or unknown incidents without this evidence cannot
gain proof from this fix. They keep their normal recovery path.
- No provider attempt limit, approval gate, workspace hold, or company
boundary changes.

## Model Used

OpenAI GPT-6, through Codex, with code search, tool use, and local test
execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 12:56:57 -06:00
DottaandClaude Opus 5.5 89e6575418 test(native-runtime): cover released session tool authority and add a catalog memory benchmark (#21440)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs get their tools from a session tool authority,
which holds the run's assigned app (MCP) tool catalog
> - #21415 releases a finished session's authority, so the server heap
no longer grows with each finished session
> - Two behaviors of that fix had no test: a released session fails
closed, and the release when a follow-up run retires its warm
predecessor and then fails before it admits its own authority
> - The memory saving was measured once, by hand, with no script in the
repository
> - This pull request adds the two tests and a benchmark script. It
changes no product code
> - The benefit is that a later change cannot silently bring back the
leak or let a released session reach its tools

## Linked Issues or Issue Description

Refs #21415 (the fix these tests cover). Refs #21430 (closed duplicate;
a reviewer comment there named the follow-up failure path).

**What is missing?**

#21415 tests that finished sessions return the authority map to its
starting size. No test checks that a released session refuses late tool
calls. No test checks the follow-up path, where a follow-up run retires
the previous warm session (so the release at forget time is skipped
while the follow-up owns the scope) and then fails before it admits its
own authority.

**Expected behavior**

A late tool call or completion feedback for a released session fails
with `native_session_tool_authority_unavailable`. The next run of the
same scope gets a new authority with only its own assignments. When a
follow-up fails after it retires its predecessor, the authority map is
back at its starting size after that run settles.

**Paperclip version or commit**

master `edb6945ff3` (after #21415, `85f4a91e70`).

## What Changed

- Test "refuses a released session's late tool call and completion
feedback and admits the next run with only its own assignments".
- Test "releases a retired warm predecessor's tool authority when the
follow-up fails before admitting its own". A follow-up of the same warm
scope with changed configuration retires the predecessor. Then its
assigned tool listing fails. The test asserts that the predecessor was
closed, that the follow-up failed with that error, and that
`sessionToolAuthorityCount()` is back at its starting value.
- `server/scripts/benchmark-assigned-mcp-tools-memory.ts`: retained
bytes of one assigned tool catalog per session, and schema copies per
`definitions()` call.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "released
native session tool authority|retired warm predecessor"`: 2 passed.
- Red check: with the release at settle removed from
`native-session-executor.ts`, both tests fail (`expected 1 to be +0`,
and the late tool call is not refused).
- Retained bytes per finished per-turn session were measured through
`executePaperclipNativeSession` with the real `createAssignedMcpTools`
(80 tools, 2 KB schemas, 100 sessions, forced GC, `heapUsed` delta). On
`85f4a91e70^` (before #21415): 604,566 bytes. On `85f4a91e70`: 19,794
bytes (test-harness noise), and the authority map stayed at its starting
size.
- `cd server && node --expose-gc --import tsx
scripts/benchmark-assigned-mcp-tools-memory.ts --tools 80 --schema-bytes
2048 --sessions 200 --turns 20` reports about 546 to 567 KB retained per
assigned tool catalog. Each `definitions()` call makes 80 new schema
copies (about 296 KB), and none are kept.
- Full `native-session-executor.test.ts` locally, with a load average
near 78 from other work on the machine: 1,483 passed and 5 failed. All 5
failures were timeouts in tests this change does not touch (the slowest
took 163 s). CI gives the clean full-suite result.

## Risks

- Low risk. Tests and one benchmark script only. No product code
changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:17:25 -06:00
DottaandClaude Opus 5.5 85f4a91e70 fix(native): release a finished session's tool authority so the server heap does not grow per run (#21415)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime starts each agent run in a Runner session. The
server keeps a tool authority for each session scope, so the Runner can
call Paperclip tools and assigned app tools.
> - The server stored that authority in a map that lives as long as the
process. Only an aborted start removed an entry.
> - Each finished session kept its authority and the run's assigned
app-tool catalog. A large app catalog (hundreds of tools with schemas)
is several megabytes per run.
> - Under a busy queue with many short runs, the server heap grew until
the process ran out of memory. Each crash stopped every run that was in
progress.
> - This pull request releases a scope's authority when nothing can use
it again.
> - The benefit is that the server heap stays flat however many runs
finish, so the server does not crash and stop in-progress work.

## Linked Issues or Issue Description

**What happened?**
A hosted server ran out of JavaScript heap ("Ineffective mark-compacts
near heap limit") about 50 minutes after it started, and again after the
next start. The last GC line showed about 3.3 GB still live after a full
mark-compact. Each crash stopped every run in progress. The server
started about 300 native runs per hour, and most of them failed quickly.
Each run start loaded a large assigned app-tool catalog (the server
logged the "over the compile budget" warning with hundreds of tool ids
for each start).

**Expected behavior**
Memory that a run uses is released when the run's session ends. Heap
size does not grow with the number of finished runs.

**Steps to reproduce**
1. Run `executePaperclipNativeSession` with `useRunnerd: true` and a
`per_turn` session, for many distinct sessions, each with an assigned
MCP catalog.
2. Let each run succeed or fail.
3. Read `sessionToolAuthorityEpochs.size` and the post-GC heap. Before
this change, both grow by one entry and about 3 MB for each finished
run.

**Paperclip version or commit**
`master` (and the hosted build that crashed).

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`: new
`releaseIdleSessionToolAuthority(scope)`. It removes and revokes the
scope's tool-authority epoch when no execution, authority
initialization, settle, warm owner, or warm close holds the scope. It is
called:
- at the end of `executePaperclipNativeSession` (after the scope
reservation and the settle mark are released),
  - after a warm session's idle-timeout close,
- after a warm session's close in `closeIdleWarmNativeSessions` (restart
and sleep sweeps).
- A warm session that stays retained keeps its authority, so the next
run in that session still drains the previous run's accounting events as
before.
- `forgetWarmNativeSession` also calls it, so a warm session that a new
session identity supersedes (`retireSupersededWarmNativeSessions`)
releases its authority too (review finding).
- New `sessionToolAuthorityCount()` export, for the regression tests.

## Verification

Red/green regression test in
`server/src/services/native-runtime/native-session-executor.test.ts`:
"releases each finished session's tool authority and assigned tool
catalog (succeeded|failed)". It runs five runnerd `per_turn` sessions
with distinct session ids and an assigned MCP catalog, then checks that
the authority map is back at its starting size.
- Red (fix disabled): `expected 10 to be 5` (one entry stays for each
finished run).
- Green (fix): both cases pass.

Second red/green test, "releases a superseded warm session's tool
authority and keeps only the retained session's": two warm runnerd runs,
where the second has a new session identity and retires the first. Red
(no release in `forgetWarmNativeSession`): `expected 2 to be 1`. Green:
one entry for the retained warm session, and zero after the warm sweep
closes it.

Heap measurement, same harness, 200 runs with distinct sessions, each
with a 300-tool assigned catalog, provider failing fast, mock call
history cleared after each run, post-GC `heapUsed` delta:

| Finished runs | Entries before | Heap delta before | Entries after |
Heap delta after |
| --- | --- | --- | --- | --- |
| 50 | 50 | 150.5 MB | 0 | 16.2 MB |
| 100 | 100 | 300.6 MB | 0 | 16.8 MB |
| 150 | 150 | 452.8 MB | 0 | 17.3 MB |
| 200 | 200 | 605.3 MB | 0 | 18.0 MB |

Before: about 3.0 MB retained for each finished run, linear, never
released. After: flat.

Commands:
- `cd server && pnpm exec vitest run
src/services/native-runtime/native-session-executor.test.ts -t "releases
each finished session"`
- `cd server && pnpm exec vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-session-resume.test.ts`
- `cd server && pnpm exec tsc --noEmit`

## Risks

- A late Runner callback for a session that already ended now gets "tool
authority unavailable" instead of reaching the old, revoked authority. A
finished session has no Runner left, so no supported path makes that
call. A warm session that stays retained keeps its authority.
- A same-run restart recovery in the same process starts with no
previous epoch for an ended session, so it does not drain a previous
run's accounting events. That only applies when the earlier session
already ended.
- No schema change. No migration.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs, heap measurement).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 11:00:09 -06:00
DottaandClaude Opus 5.5 eb54831d28 fix(native): relaunch a remote Runner that exited for a fresh bootstrap (#21374)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs often use a remote Runner in a sandbox. After a restart,
a deploy or a same-run retry, the server reattaches to that Runner or
starts a new Runner over its saved state
> - A Runner that no controller reaches before its connection lease
expires stops its provider, saves its state as `recoverable_failure`
with `lease_expired_requires_bootstrap`, and exits. The Runner protocol
says that only a new process with a fresh bootstrap ticket can
reconnect, and that the new process continues from the same cursor
> - Reattachment started a new Runner over saved state only for a
settled `suspended` exit (or an exit before `session.open`). So it
refused this exit, and the run ended with
`runner_remote_process_identity_unavailable`
(`native_continuation_requires_reconciliation`)
> - This pull request lets reattachment launch a Runner over the exact
saved state of a Runner that exited to ask for a fresh bootstrap
> - The benefit is that the run continues by itself after a long
controller gap, with no Retry and no manual sandbox repair

## Linked Issues or Issue Description

Refs #21329, #21362, #20853

**What happened?**
On hosted staging (build `2b69cdfcd6`, read-only trace), Codex runs on
remote Daytona Runners ended with
`native_continuation_requires_reconciliation` and the error
`runner_remote_process_identity_unavailable`, about 5 per hour, after
#21329 was deployed. All five sample runs (c6fabe9a, 31d54b02, 995b63df,
dd595934, 50c6e6e6) ended the same way. The last
`native.remote_runner_recovery_failed` event recorded
`process_verification` / `process_not_running`, `runnerLifecycle:
recoverable_failure`, `runnerRecoverableFailure:
lease_expired_requires_bootstrap`, and the Runner diagnostic "connection
lease expired; a fresh bootstrap is required". Since 10:30 UTC, 8 of the
15 runs that ended with this error had this exact exit.

Timeline of run c6fabe9a:
- 10:46 to 10:54: `runner.session.startup` took 476 s in a loaded
sandbox.
- 10:54:02: the controller sent `turn.start`. At 10:54:47 the command
timed out: "PRP command turn.start timed out (runner connected; pending
ahead 0; last runner event seq 2 91s ago)". The run resumed in the same
run and the turn then ran (tools completed from 11:14).
- 11:39: a graceful server shutdown suspended the controller. At 11:43
the next controller started the reattachment, but it did not reconnect
to the Runner until the server restarted again at 11:58. In that gap the
Runner's lease expired and the Runner exited.
- 11:58:53: the reattachment proved the process gone and refused the
`recoverable_failure` state. The run failed at 11:59.

Why `turn.start` timed out while the Runner was connected: the Runner
answers `turn.start` only after Codex answers its `turn/start` JSON-RPC
request. While it waits, the Runner buffers Codex notifications and
commits no events, so the controller sees a quiet connected Runner with
nothing pending ahead. The controller's bound for the `turn.start`
command result is a fixed 30 s. #21329 changed only the later wait for
`turn/started`. In a sandbox that needed 8 minutes to start, Codex
answered later than 30 s. That timeout is recoverable, and the run did
recover from it. This pull request does not change it (see Risks). The
failure that ended the runs is the refused reattachment.

**Expected behavior**
When a remote Runner exits because its lease expired (or its reconnect
grace or deadline ended) and its saved state is intact, reattachment
starts a new Runner from that state and the run continues.

**Steps to reproduce**
1. Start a native Codex run on a remote sandbox Runner.
2. Keep every controller from reaching the Runner for longer than its
connection lease (on staging: a deploy that stalled the reattachment for
about 15 minutes).
3. The Runner saves `recoverable_failure` /
`lease_expired_requires_bootstrap` and exits.
4. The next reattachment fails with
`runner_remote_process_identity_unavailable`, and the run ends.

**Agent adapter(s) involved**
`paperclip_runner` with Codex on a Daytona remote sandbox. The rule is
not provider-specific.

## What Changed

- `remote-runner-recovery-diagnostics.ts`: new
`exitedRunnerRequestedFreshBootstrap` (diagnostic) and
`runnerStateRequestsFreshBootstrap` (saved state). They accept lifecycle
`recoverable_failure` with `lease_expired_requires_bootstrap`,
`transport_reconnect_grace_exceeded` or
`transport_reconnect_deadline_exceeded`. These are the three codes the
Rust Runner records after `shutdown_preserving_cleanup`. The state check
also requires no pending terminal delivery and no warm transition.
- `native-session-executor.ts` (`verifyRemoteRunnerOrSettledExit`): when
the exact Runner process is proven gone and its diagnostic names one of
these exits, the server reads the saved state again. If the state has
the same six identity fields and still names that exit, the reattachment
launches a Runner from it instead of failing. The outbox does not need
to be empty and a turn may be active: the new Runner replays from the
same cursor. The attempt bound is the one the reattachment already
admits (3, or 4 for an unanswered-Runner replacement claim). Other exits
are unchanged.
- `native-session-executor.ts` (host preparation in
`createRunnerdBackend`): a resumed sandbox for a reattachment that did
not adopt a process re-reads the Runner state. When it still has the
same six identity fields and names the fresh-bootstrap exit, preparation
keeps it in place, as for an adopted live Runner. It no longer fails
with `runner_harness_state_mismatch: resumed_not_suspended` (found by
Greptile).
- `doc/execution-semantics.md`: one paragraph describes the rule and the
staging evidence.

## Verification

- Red on `origin/master`: the 5 new "launches a new Runner over the
saved state after ..." cases failed (the reattachment threw
`runner_remote_process_identity_unavailable`). The 7 new fail-closed
cases passed.
- Green: `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "remote
controller restart adoption"` passes 118 tests.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
1712 passed, 21 skipped, 1 failed. The failure ("verifies a live warm
owner before refreshing run authority", warm-session supervision, not
touched here) passes when run alone. It is a timing failure under
full-file load.
- Launch path: "keeps the saved mid-turn state of a remote Runner that
exited for lease_expired_requires_bootstrap when the reattachment
prepares its sandbox" runs `createRunnerdBackend` and
`prepareExternalRunnerState` with a mid-turn Codex checkpoint. Red
before the host-preparation change (`resumed_not_suspended`), green
after. An unknown recoverable failure stays fail-closed.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts`
on the final revision: 1494 passed, 7 skipped.
- `cd server && node scripts/typecheck-layers.mjs` (includes test
programs): passes.
- Fail-closed cases covered: an unknown recoverable failure, a warm
transition, a pending terminal delivery, a state that changed after the
exit, a different Runner instance, attempt 4 on an ordinary claim, and
attempt 5 on a replacement claim.
- Staging read (read-only): the 5 sample runs and 69 runs with this
error in the last 9 hours were classified from their
`native.remote_runner_recovery_failed` events.

## Risks

- Medium. Reattachment now starts a new Runner over a saved state that
can be mid-turn. The Runner itself stops its provider before it saves
this state. The new Runner restores the session. It continues the turn,
or it reports the lost turn as recoverable only after it proves that the
old provider process ended. A recoverable lost turn on a remote
reattachment ends the run as `native_provider_terminal_failed` until
#21362 merges; the replacement lane then continues the task. That is
still better than the current reconciliation hold.
- The attempt bound does not grow. A Runner that exits this way again on
a later reattachment is bounded by the same provider attempt budget.
- This does not change the 30 s `turn.start` command bound or the deploy
gaps that let the lease expire. Those are separate follow-ups. 57 older
runs with this error had a different exit (`ready`, not settled). This
pull request keeps that case fail-closed; #20853 covers the
settled-`ready` part.
- No schema, API, UI or Runner image change.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, in Claude
Code with tool use (shell, file edit, code execution, read-only browser
reads of staging run data).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:53:38 -06:00
DottaandClaude Opus 5.5 0986c3eb3a fix(native): let a same-run reattachment take over its own retained remote Runner cleanup (#21328)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) use a detached
`paperclip-runnerd`. After a recoverable failure, the server resumes the
same run and reattaches to that Runner
> - #20923 keeps an unreachable remote Runner alive when the failed
attempt closes, so that the same-run reattachment can adopt it
> - The same close also leaves the session in the in-process cleanup
quarantine. The quarantine refuses every new session of the task, and
that includes the reattachment that the Runner was kept for
> - Close retries cannot settle it. After the first close fails, the
Runner transport returns the same rejected close for every retry
> - On staging, this was the largest failure class: 14 Codex runs in one
hour failed with `native_session_cleanup_quarantined`, while only 5 runs
succeeded. Each task then needed Retry
> - This pull request lets the same run's reattachment to that Runner,
in the same remote sandbox, take over its own retained cleanup
> - The benefit is that the task continues on its own, with no spent
attempt, no failed run and no Retry

## Linked Issues or Issue Description

Refs #20923, #21031

**What happened?**
A remote native run (Codex on Daytona) failed a turn with a recoverable
error, for example `runnerd did not report the provider turn identity`
or `native_finalization_missing: session returned no semantic result`.
The close then logged `runner did not prove durable suspension before
checkpoint` and `remote runner left alive unsettled for same-run
reattachment`. Each later same-run reattachment was refused at
admission:
- When the close error was retryable, admission waited out the grace and
failed with `native_session_cleanup_quarantined: prior session cleanup
remains incomplete`. That spent a same-run attempt. After the attempt
budget, the run failed. All close retries replayed the same failed close
within milliseconds (`runner transport close requested: native session
quarantined cleanup recovery`), so they could never settle the
quarantine.
- When the close error was `native_session_close_unrecoverable`
(operator-required), the next attempt failed at once with
`native_session_cleanup_quarantined: the previous session is still
shutting down; Paperclip finishes the cleanup automatically and the
saved work is kept.` That code is a permanent failure.

"Paperclip finishes the cleanup automatically" did not happen in the
process. Only a server restart (a deploy) removed the in-memory
quarantine, and by then the run had failed and the task waited for
Retry. Staging restarts every 20 to 45 minutes, so attempts often end
mid-cleanup and meet this gate.

**Expected behavior**
The same run's reattachment adopts the Runner that its earlier attempt
kept for it. It does not wait for a close that cannot settle, and it
does not spend an attempt. The task continues without Retry. Other runs
of the task and other sandboxes stay gated.

**Steps to reproduce**
1. Start a remote native run, and let its turn fail with a recoverable
transport error while the Runner cannot prove suspension.
2. The close keeps the Runner alive and quarantines the session.
3. The server schedules the same-run reattachment 30 s later.
4. The reattachment fails with `native_session_cleanup_quarantined`, and
the task needs Retry.

**Paperclip version or commit**
Deployed Core `c730872974` on staging. Reproduced on `master` at
`49f0030f06`.

**Deployment mode**
Hosted cloud (staging), Daytona remote sandboxes.

**Agent adapter(s) involved**
`paperclip_runner` native runtime with the Codex app server backend.

## What Changed

- `packages/paperclip-runner/src/native-session-runtime.ts`:
`executeNativeSession` takes a new option `retainedRunnerReattachment`.
With a remote cleanup scope, admission retires quarantine entries that
belong to the same run in the same remote scope. Removal stops later
close retries. A close attempt that is still in flight stays an
admission owner until it settles, so the old and new controllers do not
overlap. Other runs, other sandboxes and local backends keep the gate.
- `server/src/services/native-runtime/native-session-executor.ts`: sets
`retainedRunnerReattachment` only for a `reattach_remote_runner`
recovery of the same run.
- `doc/execution-semantics.md`: records the takeover.
- Tests:
- `packages/paperclip-runner/src/native-session-runtime.test.ts`: four
new cases. An operator-required close with a retained Runner. A
retryable close whose retries replay one failure. A close that a deploy
interrupts while it is in flight (the reattachment waits for it, then
takes over). Controls: another run, a run without the takeover option
and a local backend stay refused.
- `server/src/services/native-runtime/native-session-executor.test.ts`:
asserts that the option is set for a same-run remote reattachment and is
not set for a fresh sandbox run.

## Verification

- Red on `master` (runtime source restored): 3 of the 4 new runtime
cases fail with exactly the two staging errors (`the previous session is
still shutting down…` and `prior session cleanup remains incomplete`).
The control case passes.
- Red on `master` (executor source restored): the 3 executor assertions
fail.
- Green with this change:
- `cd packages/paperclip-runner && npx vitest run
src/native-session-runtime.test.ts` (294 passed)
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "exact
Stop cleanup|fresh sandbox|retirement"` (46 passed)
- `cd server && npx vitest run
src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts
src/services/quarantined-task-retry.test.ts
src/__tests__/completed-turn-cleanup-hold-settle.test.ts` (61 passed)
- `npx tsc --noEmit` in `packages/paperclip-runner` and `server` passes.
- Staging evidence: runs `7b7ded99` and `27c487ef` show the sequence
above in their events and logs.

## Risks

- The takeover applies only when the server recovers the same run with
`reattach_remote_runner` in the same remote cleanup scope. This is the
case that #20923 keeps the Runner alive for. After a deploy the
quarantine is already gone and the same reattachment runs today, so the
takeover gives the in-process retry the same behavior as a restart.
- A close attempt that is in flight is never overlapped. The
reattachment waits for it within the existing admission grace.
- The reattachment still verifies Runner ownership and its controller
generation before it adopts the Runner. If the Runner is gone, the
existing relaunch or reconciliation path handles it.
- No persisted shape changes. Saved state from the previous release is
not affected because the quarantine lives only in process memory.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 03:15:40 -06:00
Nicky LeachandClaude Opus 5.5 41998d551c feat(plugins): send the claimant's credential on plugin task resume (#21335)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugin environment drivers run agent tasks on remote computers, and
a warm follow-up run resumes a parked Runner through the `resume` task
operation
> - `resume` carries only the durable identity and a bootstrap ticket,
which is enough while the parked Runner process is still alive
> - A provider sometimes has to start the Runner again to resume the
session: after it moved the parked session to a new sandbox, or after
the computer rebooted
> - A Claude Runner cannot start without its credential, and today only
the portable handoff path (`handoffCredential`) delivers one
> - This pull request lets a provider opt in to receiving the claimant
run's harness credential on `resume`
> - The benefit is that a provider can continue a parked session on a
new computer without Core orchestrating a handoff

## Linked Issues or Issue Description

Refs #21289
Related: #21305

**What existing behavior does this improve?**
Warm resume for plugin task providers that restart the Runner to resume
a session.

**Subsystem affected**
Plugin SDK environment task schema, the server's plugin Runner task
owner, and the plugin environment task docs.

**Current behavior**
`submit` can carry the harness credential (`credential`), but `resume`
cannot. A provider that must start the Runner again on resume has no
credential for a Claude task.

**Proposed behavior**
A provider that returns `resumeCredentialVersion: 1` from `describe`
receives `credential` on `resume` whenever Core would send it on
`submit`, resolved for the claimant run. Other providers see no change.

**Reason and benefit**
Providers can restart a Claude Runner on resume without the Core-run
handoff protocol.

**Breaking changes**
None. The field is opt-in: the SDK worker parses operations strictly, so
Core sends it only to providers that advertise the capability.

## What Changed

- `resume` gains an optional `credential`, with the same schema and
validation as `submit`'s.
- `describe` gains an optional `resumeCredentialVersion: 1` capability.
- The plugin Runner task owner adds the credential on `resume` when the
provider advertised the capability and Core resolved a credential, using
the same resolution as `submit` (`preparePluginRunnerCredentials`).
After a warm rebind, the claimant run's credential is sent.
- `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md` documents the field and its
handling rules.
- The runner-e2e SDK public-input pin is updated.

`handoffCredential` on the session handoff path is unchanged.

## Verification

- SDK vitest `tests/environment-task-release-resume.test.ts` and
`tests/environment-tasks.test.ts` (68 passed): `resume` accepts and
rejects the same 11 credential cases as `submit`; the capability is
optional and rejects other versions; a resume credential reaches the
plugin through the serialized worker RPC.
- Server vitest `plugin-runner-task.test.ts` and
`environment-task-runtime.test.ts` (151 passed): credential sent with
the capability and a resolved credential; omitted without a credential
or without the capability; the rebound claimant's credential is sent;
the credential appears in no metadata write and not in the sanitized
error, even when the worker error echoes it.
- `native-session-executor.test.ts -t "uses a task-only"` (5 passed):
the executor forwards the capability to the owner.
- runner-e2e `source-sdk-evidence.test.ts` and
`source-sdk-public-type.test.ts` (46 passed).
- `tsc --noEmit` for `packages/plugins/sdk` and `server`.
- Not run locally: the full suite and `pnpm verify`; required CI covers
them.

## Risks

- No server-side provider check on resume, unlike `submit`: the lease
has no reliable harness at resume time. Protection relies on the owner,
whose rebind refuses a changed harness, account or credential identity.
- The credential is the claimant run's current one, which can be newer
than the one that started the parked Runner. That is intended.
- A provider must advertise the capability only after its SDK pin
includes this schema; roll Core out first.
- Rollback: revert this pull request. No migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs), implemented by a sub-agent and reviewed
before opening.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:40:46 +00:00
DottaandClaude Opus 5.5 7e7452e88f fix(native): keep a reattached Runner alive when its provider recovery deadline ends (#21321)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs on remote sandbox Runners. A same-run reattachment
adopts a live Runner after a failed attempt or a server deploy.
> - On staging, a remote Codex Runner is often gone when the next
attempt reattaches (`process_verification` / `process_not_running`). The
task then continues in a fresh session and loses its context.
> - The deploy does not stop the Runner. The logs show `controller
authority detached for restart; durable runner left alive`, and the new
controller adopts the same Runner after the deploy.
> - Our own cleanup stops it. The adopted Runner answers its recovery
reads slowly while its in-flight turn runs. The 900 s provider recovery
deadline then aborts the recovery, and the abort closes the transport
with `turn.stop` and `runner.suspend` and signals the process.
> - This pull request makes the recovery deadline detach from the
Runner, as a recovery read timeout already does. The server marks the
failure as an unanswered adopted Runner, so the next attempt replaces
the Runner from its saved journal and resumes the same provider thread.
> - The benefit is that a deadline no longer kills a live Runner and no
longer forces a fresh session. The task keeps its session context.

## Linked Issues or Issue Description

Refs #21314, #21272, #21161, #21023

**What happened?**
Codex runs on Daytona remote Runners failed often on staging. In three
sampled chains (attempt 2 of each), the same-run reattachment adopted
the live Runner, authenticated in under 1 s, and then waited the full
deadline: `native session provider recovery timed out after 900000ms`
(909 s, 930 s and 915 s). The deadline cleanup sent `turn.stop` and
`runner.suspend` (`provider turn stop timed out before runner
suspension`). The next attempt found the Runner gone 25–180 s later
(`process_not_running`, `runnerLifecycle: ready`, empty stderr) and
continued in a fresh session. During the 900 s wait, the in-flight turn
was still running Paperclip tools (`tool.execute` spans at 04:43 in a
recovery that started at 04:29).

**Expected behavior**
A remote Runner survives a recovery deadline and a control-plane deploy.
The next attempt continues the same provider session. A Stop still stops
the Runner.

**Steps to reproduce**
1. Start a Codex run on a remote Runner and let a turn start.
2. Fail the attempt so that its close keeps the Runner (for example, a
turn start that outlives its bound).
3. Let the same-run reattachment adopt the Runner, and make the Runner
stop answering recovery commands.
4. Let the provider recovery deadline expire.
5. See that the Runner process is gone and the next attempt reports
`process_not_running`.

**Paperclip version or commit**
master at b555087f0f (after #21314).

**Deployment mode**
Hosted staging (cloud), remote Daytona sandbox Runners, deploys every
20–45 minutes.

**Agent adapter(s) involved**
Native Codex (`paperclip_runner`, Codex app-server through
`paperclip-runnerd`).

## What Changed

- Add `NativeProviderRecoveryDeadlineError` (same message as before) and
use it for the provider recovery deadline in the native session runtime.
- In the Codex driver, a recovery aborted by this deadline detaches the
controller (no `turn.stop`, no suspend, no signal). A Stop and every
other abort still close and stop the Runner.
- On the server, `nativeRecoveryCommandUnanswered` recognizes the
deadline. The failure records `adoptedRunnerUnanswered`, so the next
claim carries `replaceUnansweredRunner`. That attempt fences the old
Runner lineage, launches a fresh Runner from the saved journal and
resumes the same provider thread (the #21272 path). It does not adopt
the slow Runner again for another 900 s.
- Update `doc/execution-semantics.md`.

## Verification

- Real `paperclip-runnerd` + fake Codex
(`runnerd-codex-transport-remote-retention.test.ts`): a retained Runner
is adopted, authenticates, then stops answering. A deadline abort leaves
it alive with no pending `turn.stop`/`runner.suspend`/`runner.shutdown`.
A Stop abort still stops it.
- Red/green: without the fix, the real-runnerd deadline test fails
(`expected 'gone' to be 'alive'`), the driver unit test fails (detach
called 0 times), and the server classification test fails. With the fix,
all pass.
- Measurements on real runnerd: the deadline cleanup took 10.0 s and
killed the Runner before. Now it returns in 0 ms and the Runner stays
alive. The replacement over the kept Runner's journal resumes the same
provider thread in about 2.2 s with one `thread/start` and one
`turn/start` (no replay).
- `npx vitest run
src/live/runnerd-codex-transport-remote-retention.test.ts
src/drivers/codex/codex-app-server-driver.recovery.test.ts
src/native-session-runtime.test.ts` (368 passed).
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "never
answered its recovery read"` (passed).
- `tsc --noEmit` for `packages/paperclip-runner` and `server` (clean).

## Risks

- A deadline now leaves a live Runner behind. The next attempt replaces
it through the launch fence, which fails closed when it cannot prove the
old lineage gone. This is the same risk as the existing recovery read
timeout detach.
- When the provider attempt budget is already spent, the kept Runner is
not replaced by this run. This also matches the existing timeout detach.
- Not in this PR: the upstream trigger. The first attempt abandons a
slow turn start (`runnerd did not report the provider turn identity` or
`turn.start timed out`) while Codex later runs that turn. Two sampled
chains then stopped the Runner on
`protocol_failure:tool_binding_mismatch` / `thread_binding_mismatch`
after a deploy reattachment. These need a separate fix.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:30:22 -06:00
Nicky LeachandClaude Fable 5.2 314b893678 fix(native-runtime): deliver assigned gateway tools to every native harness through the PRP relay (#21274)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An agent can have app connections. Paperclip gives the agent the
tools of those apps through an MCP gateway, with its approval and audit
rules.
> - Native harnesses reach the gateway in two ways today. In the relay,
the server keeps the gateway token and offers the tools over the
authenticated Runner channel. In the direct way, the server gives the
gateway URL and a bearer token to the harness.
> - Which way a harness uses comes from a list of special cases that
grew one harness at a time. Local Codex and all ACPX agents use the
direct way. The others use the relay.
> - The direct way puts a run-scoped token in a provider process. A
process that the agent starts can read it. A provider process that
serves more than one run can hold a revoked token.
> - This pull request replaces the list with one rule. Each native
harness gets its assigned gateway tools through the relay.
> - The benefit is one behavior for all native harnesses, and no gateway
token in a provider process.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

Refs #13942 (the relay), #17149 (ACPX accepts relayed tools) and #19912
(the design for turn-scoped ACPX MCP authority).

**What existing behavior does this improve?**

How a native harness reaches the tools of its assigned MCP gateway.

**Subsystem affected**

Native runtime in the server (`native-session-executor.ts`,
`native-session-resume.ts`). One Runner test file. One Rust test.

**Current behavior**

- `relayAssignedMcp` is true for plugin tasks, Cursor Cloud, OpenCode,
Codex on a remote target, and Dot.
- For local Codex and for each ACPX agent (Claude, Codex, Cursor, Pi,
Kimi, Grok, Copilot, Antigravity), Core puts
`PAPERCLIP_NATIVE_MCP_NAME`, `PAPERCLIP_NATIVE_MCP_URL` and
`PAPERCLIP_NATIVE_MCP_TOKEN` in the Runner environment. The ACPX host
gives them to the provider as an HTTP MCP server. The local Codex home
gets them in `config.toml`.
- The token is run-scoped. Core revokes it when the run ends.

**Proposed behavior**

- A native run that has an assigned gateway uses the relay. The
condition names no provider.
- Core builds the gateway tools on the control plane, offers them with
the other Runner tools, dispatches each call over PRP, and authorizes
each call again through the gateway.
- Core removes the three `PAPERCLIP_NATIVE_MCP_*` variables from each
Runner environment.
- The tool contract fingerprint has the relay marker for each target,
not only for remote targets.

**Reason and benefit**

- All native harnesses have the same behavior.
- No provider process holds a gateway token.
- A sandbox does not need to reach the gateway URL of the host.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`
- The list of special cases is gone. Core validates the binding and
builds the relayed tools when a binding is present.
- Core does not list the gateway for a review run, Grok Bot or Muse. The
tool authority never offered app tools to those. A gateway error can
therefore not fail those runs.
- OpenCode keeps its `on_demand` catalog. That option depends on the
provider, not on the relay.
  - Core removes the three variables from each Runner environment.
- `server/src/services/native-runtime/native-session-resume.ts`: the
relay marker in the tool contract fingerprint is unconditional.
- `doc/execution-semantics.md`: the text about the tool contract
fingerprint now states the rule, the tool names and the cutover.
- App tool schemas. New module
`packages/paperclip-runner/src/drivers/tool-input-schema.ts` compiles
each schema by itself, in its declared dialect (draft-07, 2019-09 or
2020-12), with no shared `$id` registry. Core uses it when it builds the
relayed tools (`assigned-mcp-tools.ts`). Core does not offer a tool
whose schema has an unsupported dialect or does not compile, and writes
a log entry with the tool name and the reason. The run does not fail.
The ACPX tool bridge and the OpenCode MCP bridge use the same function.
The tools of Paperclip itself stay strict.
- Large arguments. The ACPX sidecar sends the arguments of a tool call
to runnerd with no display limit. The limit is the tool-call admission
limit of runnerd, 256 KiB. A larger call fails with a tool error that
gives the limit. The sidecar does not dispatch it.
- Tests in `native-session-executor.test.ts`,
`native-session-resume.test.ts`, `runtime-context-materializer.test.ts`
and `acpx_provider_backend.rs`.

Not changed in this pull request: the direct code path. It is still in
the Runner, but it receives no binding. #21311 deletes it.

## Behavior that each harness sees

| Harness | Before | After |
|---|---|---|
| Codex on a remote target, OpenCode, Cursor Cloud, Dot, plugin tasks |
Relay | No change |
| Local Codex | Direct: a `paperclip-assigned` server in `config.toml` |
Relay. App tools are dynamic tools named `app_<name>_<digest>` |
| ACPX agents | Direct: a second HTTP MCP server | Relay. App tools are
on the `paperclip` bridge as `mcp__paperclip__app_<name>_<digest>` |
| ACPX with a preconfigured home | No gateway | No change |
| Grok Bot, Muse | Got the variables and ignored them | Do not get the
variables |

- A relayed tool name has a maximum of 64 characters. With the
`mcp__paperclip__` prefix it has a maximum of 80.
- Permission behavior keeps the same intent. For Claude in the
`approve-reads` and `approve-paperclip` modes, app tools ask for
approval, as before. For the other ACPX agents the permission decision
does not use the server name.
- In a planning run or an ask run, the relay offers only tools with read
risk. The direct gateway offered all tools. This is stricter.
- When the catalog is too large, each harness gets the two compact tools
`paperclip_search_assigned_tools` and `paperclip_call_assigned_tool`.
- The four helper tools for resources and prompts
(`paperclip_list_resources`, `paperclip_read_resource`,
`paperclip_list_prompts`, `paperclip_get_prompt`) are not in the relay.
No harness loses them: a run token permits only `tools/list` and
`tools/call` (`run-preparation.ts`), so these four tools already fail
with `gateway_token_action_denied` on the direct path.

## State saved by the previous release

- **Local sessions of each provider start again one time.** The tool
contract fingerprint changes for local targets. The next run sees a
different fingerprint, starts a new provider session with the full task
context in the same workspace, and keeps the old saved state. This is
the existing rotation for a tool contract change. Codex needs it,
because Codex cannot add tools to a thread that it resumes. For the
local providers that already used the relay, the restart is not
necessary. It is accepted so that the rule names no provider.
- **Remote sessions continue.** Their fingerprint does not change. For
ACPX, the attach of the next run replaces the saved tool set.
- **A kept Codex home.** Paperclip writes `config.toml` again at each
start and recovery. The old gateway server and its token are removed.
- **A deploy lands during a run, and the Runner survives.** The
controller attaches to the same Runner for the same run. The old
provider keeps its direct gateway connection. The controller detach
keeps the gateway token of that run, and Paperclip revokes it when the
run ends.
- **A deploy lands during a run, and the Runner is dead.** The Runner
starts again for the same run with its saved tool set and no direct
gateway. The assigned app tools are not available for the remainder of
that one run. The next run gets them through the relay. This applies to
each harness that moves to the relay. It has no test.
- Paperclip refuses no saved state and discards none.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-session-resume.test.ts`: `1508 passed
| 7 skipped`, 1 failed. The failed test is "archives a damaged prior
epoch before completing guarded same-task replacement with real runnerd"
(`ENOENT … authority-epochs`). It fails in the same way on `master` at
53608bd1d6 and with the product change removed, on macOS.
- `cd server && npx tsc --noEmit -p tsconfig.json`: no errors.
- `cd packages/paperclip-runner && npx vitest run
src/drivers/runtime-context-materializer.test.ts`: `18 passed`. `npx tsc
--noEmit -p tsconfig.test.json`: no errors.
- `cargo fmt --manifest-path runner/Cargo.toml --all -- --check`: no
changes. `cargo test -p paperclip-runner-core --lib
acpx_provider_backend`: `50 passed`.

- `npx vitest run src/drivers/tool-input-schema.test.ts
src/drivers/runner-tool-bridge.schemas.test.ts
src/drivers/runner-tool-bridge.test.ts
src/cli/acpx-runtime-sidecar.test.ts src/drivers/opencode` in
`packages/paperclip-runner`: `21 passed (21)` files, `377 passed` tests.
`pnpm run typecheck:typescript`: no errors.
- `cargo test -p paperclip-runner-core --test acpx_event_payload --test
tool_input_schema_fixtures`: 10 and 3 passed.
- Schema tests use one shared fixture
(`tool-input-schema-fixtures.json`) for the TypeScript compiler, the
tool bridge and the Rust `begin_call`: a draft-07 schema, a second
schema with the same `$id`, 2019-09, 2020-12, draft-04, an unknown
dialect, a `$ref` with no target, and an invalid keyword. Before the fix
the bridge failed with `no schema with key or ref
"http://json-schema.org/draft-07/schema#"`.
- A tool schema whose `$id` is the URI of a meta-schema cannot damage
the shared compiler. Before this fix, `removeSchema` deleted that
meta-schema, and later schemas failed to compile for each run in the
process. A test with 7 reserved ids failed before the fix and passes
now.
- Core limits the compile work of each run start. A schema over 512 KiB
is not compiled, and one listing compiles a maximum of 768 KiB of
schemas. These are the limits of the Runner. A tool over a limit is
served only through the search and call tools. Core keeps the outcome
for each schema digest (1,024 entries) and yields to the event loop
before each compile. Four tests failed before this change.
- A sidecar test sends a tool call with 200 KiB of arguments and
receives them byte for byte, refuses a call with 300 KiB with the named
error, and keeps the replacement of a lone surrogate.

Tests, each seen to fail without the product change:

- "keeps assigned MCP credentials on the control plane for %s" now runs
for remote Codex, local Codex, OpenCode and ACPX Claude, Pi, Kimi,
Copilot and Cursor. It checks that Core builds the relay, offers the
tool, dispatches a call, and gives no `PAPERCLIP_NATIVE_MCP_*` variable
to the Runner. Without the change, the five new harness cases fail.
- "relays the assigned gateway when a deploy lands mid-run…" runs for
Claude and Pi.
- The fingerprint tests keep the old local fingerprint as a retained
value and check that the new one is different. The remote fingerprints
for Codex and OpenCode do not change.
- "drops a retained direct gateway server when the relayed launch has no
binding": a Codex home with the old `config.toml` entry loses that entry
at the next start and at a recovery of the same run.
-
`next_run_attach_offers_relayed_gateway_tools_to_a_direct_gateway_checkpoint`
runs for Claude and Pi.

Not run: a live run with a real provider and a real app connection for
each harness. Not run: `pnpm verify` and the full suite. CI supplies
that coverage.

## Risks

- **An ACPX app call has a limit of 256 KiB for its arguments.** The
direct gateway accepted a request body of 10 MB. Codex and OpenCode
permit 480 KiB. To get nearer to 10 MB, runnerd needs a larger admission
limit for tool calls or a protocol that sends arguments in parts. That
is not in this pull request.
- **A tool with a draft-04 or draft-06 schema is not offered.** Ajv 8
does not support draft-04, and the TypeScript and Rust validators must
agree. On the direct path the provider got such a tool.
- **Core compiles app schemas on its event loop.** Before this pull
request Core did not compile them. With the limits above, one valid
schema near 512 KiB can hold the loop for about 0.65 s, and one listing
for about 1.2 s in total (measured on a developer machine). An app that
changes its schemas at each listing gets no help from the cache. A
worker thread with a timeout would remove this; it is not in this pull
request.
- **Tool names change for local Codex and all ACPX agents.** A saved
agent instruction that names a gateway tool directly does not match the
new name.
- **Each local native session starts again one time after the deploy.**
The provider conversation of that session does not continue. The task
context and the workspace stay.
- **A run that is in flight at deploy, with a dead Runner, has no app
tools until it ends.**
- **Names of 80 characters are not tested live** with Kimi, Copilot,
Grok, Antigravity and Codex through ACPX. Pi already shortens a name
longer than 64 characters.
- **Managed Claude and AWS AgentCore now receive relayed app tools.**
This is not tested live.
- **A gateway token expires after 1 hour.** The relay keeps the token on
the controller for the whole run, as OpenCode and remote Codex do today.
A longer run loses its app tools on both paths. This pull request does
not change that.
- **The gateway URL must have the path `/mcp/gateways/<id>`.** With a
path prefix in the API base URL, a run with an assigned gateway fails
with `assigned native MCP gateway path is invalid`. This was true for
the relayed harnesses before. It is now true for all.
- No test covers the rule that Core does not list the gateway for Grok
Bot and Muse.

## Follow-up

The direct path is now unreachable. #21311 deletes it. That pull request
is in the same stack, above the pull requests that change the same ACPX
host files.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change, reviewed the diff and ran the verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), investigated each harness
and wrote the change and the tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 00:16:58 -08:00
Nicky LeachandClaude Opus 5.5 2483cb46c2 feat(plugins): deliver configured environment to plugin Runner tasks (#21280)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent gets environment variables from its execution environment,
its own configuration, its project, and its routine. Secrets are
resolved for each run.
> - A plugin environment can run the agent as a native Runner task on
its own host, through the environment task contract.
> - That contract has no field for the agent's variables. Runs on a
plugin environment get only the variables that the plugin sets itself.
> - This pull request adds a versioned contract field for the values,
sends them on a session handoff, lets Codex commands inherit the
variables in an external sandbox, and lets a plugin protect the
variables that it sets itself.
> - The benefit is that plain and secret variables work on plugin
environments in the same way as on local environments.

## Linked Issues or Issue Description

Refs #21233. That pull request by @lucktastic finds the same gap and
proposes the same capability and field names
(`configuredEnvironmentVersion: 1`, `configuredEnvironment`). This pull
request uses the same names, so providers implement one contract. It
differs in these places:

- Codex commands get the variables through an external-sandbox denylist,
not through key names sent at `prepare`. A retained Runner does not need
a new preparation when a key is added or removed.
- A session handoff `resume` gets the values, because it starts a new
Runner on another computer. Value changes on the same computer need no
new field: Core's process reuse key includes the values, so Core starts
a new task.
- A driver can declare `protectedEnvironmentKeys`. Paperclip rejects or
warns when one is saved.
- The run log names skipped variables only for an older provider.

Thank you to @lucktastic for the diagnosis and the contract shape.

**What happened?**
An agent has environment variables, plain and secret. Its runs on a
plugin task environment do not get them. `PAPERCLIP_CONFIGURED_ENV_KEYS`
in the run lists only the variables that the plugin sets.

**Expected behavior**
Each run gets the agent's configured variables.

**Steps to reproduce**
1. Set a plain and a secret environment variable on an agent.
2. Assign the agent to a plugin environment that provides Runner tasks.
3. Start a run. Read the names in `PAPERCLIP_CONFIGURED_ENV_KEYS` in a
command that the agent runs.
4. The agent's names are not in the list.

## What Changed

- **SDK contract** (`packages/plugins/sdk/src/environment-tasks.ts`):
- A provider advertises `configuredEnvironmentVersion: 1` on `describe`.
- `submit` gets `configuredEnvironment`. `resume` gets it only for a
session handoff.
- Names, count, and sizes have the same limits as the Runner projection.
- **Server**:
- `createRunnerdBackendWithinSessionClaim` gives the run's configured
projection to the plugin task owner. The projection has no provider
credentials and no generated runtime keys.
- `createPluginRunnerTaskOwner` sends the fields only to a provider with
the capability. An older provider gets no new field, and the run log
names the variables that were not set. The log never prints values.
- **Codex** (`codex-security-config.ts`):
- An externally sandboxed launch uses `inherit="all"`,
`ignore_default_excludes=true`, and an `exclude` list of Codex's own
credentials, instead of an `include_only` list fixed at preparation
time.
- A test reads the Runner's Codex environment keys, and fails if a new
key is not classified.
  - Host launches keep the bounded allowlist.
- **runnerd** (`codex_provider.rs`): `runtime_notification_codex_args`
still grants the notification files to read-only profiles when no
allowlist is present. Plan mode in an external sandbox needs these
grants.
- **Protected keys**:
- An environment driver can declare `protectedEnvironmentKeys` in its
manifest.
- Paperclip rejects a protected key on an agent or environment that uses
the driver. The 422 has code `protected_environment_variable`.
- On a project, a routine, or another agent, Paperclip saves the key.
The response has a `configurationWarnings` notice that names the
environments. The project and routine pages show the notice as a toast.
The agent pages already show configuration warnings.
- **Docs**: `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md` describes the
provider obligations.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refresh the reviewed
SDK source pin.

## Verification

- `packages/plugins/sdk`: `vitest run tests/environment-tasks.test.ts`
(38 pass). Accepts bounded values. Rejects bad names, NUL bytes, too
many variables, and oversized values.
- `server`: `vitest run
src/services/native-runtime/plugin-runner-task.test.ts` (95 pass).
Values go only to a provider with the capability, never in `prepare`.
`resume` sends values only for a handoff.
- `server`: `vitest run src/services/protected-environment-keys.test.ts
src/__tests__/project-routes-env.test.ts` (10 pass). Rejection, notices,
and the project route response.
- `server`: all route suites that import the project, routine,
environment, or agent routes (45 files, 1689 pass).
- `packages/shared`: `vitest run src/validators/plugin.test.ts` (64
pass).
- `packages/paperclip-runner`: `vitest run src/drivers/codex
src/configured-environment.test.ts
src/live/runner-provider-launch.test.ts` (444 pass).
- `packages/paperclip-runner/runner`: `cargo test -p
paperclip-runner-core --lib runtime_notification` (9 pass, 2 ignored as
before).
- `pnpm typecheck` in `packages/shared`, `packages/plugins/sdk`,
`packages/paperclip-runner`, `server`, and `ui`.
- Not run locally: `pnpm verify --full` and browser suites. Required CI
covers the final revision.

## Risks

- **Existing providers:** low risk. Core sends no new field to a
provider without the capability. Their runs behave as before, with one
added log line when variables were skipped.
- **Codex in an external sandbox:** commands now see every variable in
the Runner environment except Codex's own credentials. The execution
environment builds that environment from an allowlist. If the Runner
gives Codex a new credential, the drift test fails until the key is
classified.
- **Saving env:** new rejections happen only when an installed driver
declares `protectedEnvironmentKeys`. No driver declares them today.
- **Providers:** they must keep the values transient and never save or
log them. The doc lists these obligations.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:11:23 +00:00
DottaandClaude Opus 5.5 86f9555695 fix(native): give a Runner whose first unanswered read came on the last attempt one replacement attempt (#21322)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs execute in a remote Runner, and a same-run retry
reattaches that Runner after an interruption
> - #21272 made the next attempt replace a Runner that left its recovery
read unanswered, instead of adopting it again
> - On staging, three runs after #21272 went live still failed: their
first unanswered read came on the third and last attempt, so no next
attempt existed
> - This pull request gives such a run one more attempt, used only for
that replacement
> - The benefit is that the replacement from #21272 can run when a
restart used up the middle attempt

## Linked Issues or Issue Description

Refs #21272, Refs #21161

**What happened?**

After #21272 (497ccba8f1) went live on staging at 05:34Z, three Codex
runs still failed with `native_session_recovery_failed:
HarnessCapabilityUnavailableError: read is unavailable: Error:
provider_transport_failed: PRP command session.snapshot timed out
(runner connected; pending ahead 0; last runner event seq … 55–84s
ago)`. The run events of all three show the same sequence:

1. Attempt 1 fails at 06:24 with `runnerd did not report the provider
turn identity`.
2. Attempt 2 starts at about 06:25. The 06:24–06:25 server restart ends
it with `local_runtime_api_listener_unavailable`. This attempt sends no
recovery read.
3. Attempt 3 adopts the same Runner. Its first recovery read gets no
answer. Attempt 3 is the last of three, so the run ends.

`adoptedRunnerUnanswered` was recorded on attempt 3, but no next attempt
read it. The replacement from #21272 never ran.

**Expected behavior**

A Runner whose first unanswered read comes on the last attempt is still
replaced once by a fresh Runner from the saved journal. The provider
turn is not sent again. The run does not end only because a restart used
the middle attempt.

**Steps to reproduce**

1. Start a Codex run in a remote Daytona Runner. Let attempt 1 fail at
turn start, so the Runner stays alive.
2. Restart the server during attempt 2, so that attempt fails without a
recovery read.
3. Let the Runner leave its recovery read unanswered on attempt 3.
4. See the run end with `session.snapshot timed out` and no replacement.

**Paperclip version or commit**

Staging with #21272 (497ccba8f1). Fixed on `master` after `bbafa78fe2`.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`).

## What Changed

- `nativeUnansweredRunnerReplacementDisposition` (executor): when
attempt 3 fails because its adopted live Runner left a recovery read
unanswered, and that attempt was not itself a replacement, the failure
stays `retryable_failure` (`native_session_interrupted`, retry in 30 s)
instead of `native_session_retry_exhausted`.
- `claimNativeRestartRecoveries`: a failed remote retry at attempt 3
with `adoptedRunnerUnanswered` is not treated as an exhausted budget.
The claim is `reattach_remote_runner` at provider attempt 4 with
`replaceUnansweredRunner`.
- `nativeAdmittedProviderAttemptLimit` (executor admission gate): a
`reattach_remote_runner` claim with `replaceUnansweredRunner` is
admitted up to provider attempt 4. Every other claim keeps the limit of
3.
- `verifyRemoteRunnerOrSettledExit`: the replacement of a live Runner is
allowed up to attempt 4. An exited Runner keeps the fail-closed path.
- Bound: the extra attempt is a replacement (`replaceUnansweredRunner`),
so it never gets another one. Attempt 4 is exhausted by the existing
rule.
- `doc/execution-semantics.md` describes the extra attempt.

## Verification

- End to end (embedded Postgres):
`native-remote-retry-admission.test.ts` › "executes the one replacement
attempt after the last attempt's first unanswered recovery read, and no
further one". It seeds attempt 3 with the staging failure, claims it,
and executes the claim through `executePaperclipNativeSession`. The
provider is reached once, the run then ends at attempt 4, and no further
claim is made. With the old `nextAttempt > 3` gate this test fails
before the provider is reached.
- Red on `master`, green with the fix (7 tests, including "admits
provider attempt 4 at execution only for the replacement claim"):
- `native-runner-restart-recovery.integration.test.ts` › "gives a Runner
whose first unanswered read came on the last attempt one replacement
attempt: true" (embedded Postgres; the coordinator is seeded at attempt
3 with the staging message). On `master` no reattach claim was made.
- `native-session-executor.test.ts` › "bounds the replacement after …"
(4 cases: the last attempt's first unanswered read is retryable; a
replacement that also went unanswered, another failure, and attempt 4
stay terminal) and "replaces the live Runner on the extra replacement
attempt".
- Focused suites: `native-session-executor`, `native-restart-recovery`,
`native-runner-restart-recovery.integration`,
`runner-lost-mid-turn-continuation`: 1737 passed, 21 skipped.
- `pnpm typecheck` in `server` passes.
- Not run locally: the container-gated remote Runner suites and the full
suite (CI covers it). Why the remote Runner stops answering is a
separate investigation and is not changed here.

## Risks

- Low to medium. This adds at most one provider attempt per run, and
only after an unanswered recovery read on a live adopted Runner. The
attempt replaces the Runner through the launch fence, which proves the
old Runner and its lineage gone or fails closed. No provider turn is
sent again.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code
with tool use (shell, file edits, browser reads of staging run events),
extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 01:27:35 -06:00
DottaandClaude Opus 5.5 8a0f95849d perf(native-runtime): start the first turn without re-copying launch files, and never fail a run on a slow copy (#21287)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents that use OpenCode in a remote sandbox must have a durable,
verified copy of the provider launch files before their first turn (the
rule from #19622)
> - Today each new run copies these files out of the sandbox, uploads
them and verifies them again, even when the company already has a ready
artifact with the same files
> - The whole copy runs under one five-minute clock. When eight runs
start together on small sandboxes, the copies compete, and slow copies
fail the run before its first turn
> - This makes cold starts slow (p50 273 s from run start to turn
submitted) and makes half of the first runs fail in a burst
> - This pull request reuses a ready artifact with the same digests,
shares one in-flight copy per company and manifest, and bounds each
network step instead of the whole copy
> - The benefit is that first turns start in about one minute on a cold
burst and in about 20 s when the artifact is already ready, and a slow
copy no longer fails the run

## Linked Issues or Issue Description

Refs #19622

**What happened?**
When 8 OpenCode runs started together on 1 vCPU Daytona sandboxes, 4 of
8 first runs failed before their first turn. Each run copied, uploaded
and verified the same provider launch files. The single 300 s clock over
the whole copy expired. Run start to turn submitted was p50 273 s and
p95 314 s.

**Expected behavior**
A run starts its first turn without a new copy when a ready, verified
artifact with the same files exists. A slow copy does not fail the run.

**Steps to reproduce**
1. Use a company with remote OpenCode agents on Daytona sandboxes with
durable storage.
2. Start 8 runs at the same time on new sandboxes.
3. See the time before the first turn and the failed runs.

**Agent adapter(s) involved**
OpenCode (native runtime, remote sandbox).

## What Changed

- `retainReadyNativeProviderArtifacts` (`native-provider-artifacts.ts`):
reuse a ready, verified company artifact with the same paths and SHA-256
digests through a lookup only. No file is copied, uploaded or verified
again. The lookup (`readyNativeProviderArtifactCandidates`) matches
target, paths and digests in Postgres with jsonb containment on the
company's ready rows (existing partial index), newest first, limit 4. It
does not load or hash other retained versions. No migration.
- `shareNativeProviderArtifactCapture`
(`native-provider-artifact-capture.ts`): one in-flight capture per
company and manifest digest per server. Other runs wait for it, then
reuse its result.
- The single 300 s clock over the whole capture is replaced by a bound
for each network step (upload, verifying download): max(120 s, size at
256 KiB/s). A timeout has the typed cause
`native_provider_artifacts_transfer_timeout`. The capture makes up to 3
attempts from the local copy.
- Stop now interrupts the capture and the wait for a shared capture
(`ActiveNativeSession.admissionAbort`).
- `doc/opencode-cold-recovery.md` describes the new behavior.
- Tests: new `native-provider-artifact-gate.test.ts` (7 tests), 5 new
tests in `native-harness-backup-storage.integration.test.ts`, and new
executor variants `opencode-new-durable-slow-capture` and
`opencode-new-durable-ready` in `native-session-executor.test.ts`.

## Verification

Local (this branch, on master 9b613562b7):

- `npx vitest run
src/services/native-runtime/native-provider-artifact-gate.test.ts
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-harness-backup-storage.integration.test.ts`:
3 files, 1511 passed, 7 skipped.
- `npx tsc --noEmit -p server`: pass.
- Lookup scale: the integration test "loads only the ready artifact
whose launch files match, however many other versions the company keeps"
adds 200 ready non-matching versions. It returns only the matching
artifact. Without the SQL filter it fails (4 rows returned).
- Red/green: with master's `native-session-executor.ts` and the new
tests, `opencode-new-durable-slow-capture` fails with `TimeoutError: The
operation was aborted due to timeout`, and `opencode-new-durable-ready`
fails because the ready artifact is not looked up. With this change,
both pass.

Daytona (same commit content, cherry-picked onto the baseline commit
84a1ee3e11 with #21262; image `sha256:9461cc75…`; 1 vCPU sandboxes;
OpenCode):

| Wave | Shape | First runs OK | Run start → turn submitted | Other |
|---|---|---|---|---|
| Baseline `w-20261113T011027-s1702` | 8 runs started together | 4/8 (4
failed) | p50 273 s / p95 314 s | "only a getter" TypeError ×8,
`bootstrap_not_proven_incomplete` ×8 |
| Fix `w-20261113T031045-s1702` | same seed and shape | 8/8 | p50 66.0 s
/ p95 85.7 s | 8/8 tasks hands-off, 0 Retry banners; first agent event
p50 72.9 s / p95 90.0 s; exactly 1 upload, 7 runs waited on it (~65 s
shared capture); all failure codes 0 |
| Fix `w-20261113T034033-s1702` | 16 tasks, cap 8 | 16/16 | p50 19.1 s /
p95 24.3 s | second batch (artifact already ready, cold sandboxes) gate
time 24, 25, 26, 27, 31, 31, 45, 119 ms; first agent event p50 25.9 s |

## Risks

- The rule from #19622 still holds: a durable, verified artifact exists
before the first turn. Reuse only accepts a ready, verified artifact of
the same company with the same paths and digests.
- The first run per company and manifest per server still pays the full
transfer.
- Two servers can still upload duplicate artifacts for the same company
and manifest. Sharing is per server.
- No test drives Stop through the executor during a capture. Signal
handling is unit-tested only.
- A capture that fails all 3 attempts still ends blocked with a working
Retry. A follow-up PR (pre-turn failures continue automatically) removes
that.
- The documented five-minute deadline per capture is replaced by bounds
per network step. A very large artifact on a slow link can take longer
than five minutes in total, but each step still has a bound.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edit, test runs). Default context
window; extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:48:55 -06:00
DottaandClaude Opus 5.5 0c04ff76eb fix(native-runtime): a follow-up that arrives as a warm session closes continues the task (#21279)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native agent session on a remote sandbox stays warm after a run,
and an idle timeout closes it after about 5 minutes
> - When a person's follow-up comment arrived just as that idle close
began, the new run failed with `native_session_supervisor_busy`
> - The idle close of the finished owner could also fail its final
checkpoint with `native_runner_checkpoint_owner_lost`, because the task
already pointed at the queued follow-up run
> - Then the retry was held for workspace repair and the task stayed
blocked; the environment could not be deleted (409) because its warm
runner could not be stopped
> - This pull request makes the follow-up wait for the idle close that
is in flight, and lets the finished owner save its final checkpoint
while its successor still waits in the queue
> - The benefit is that a follow-up that lands at the warm session's
idle close continues the task with no person involved

## Linked Issues or Issue Description

Refs #18011 (the checkpoint ownership fence this change extends).

**What happened?**

A follow-up comment arrived 295–340 s after the previous run ended,
while the warm session's idle close was in progress. The new run failed
with `native_session_supervisor_busy`. In another case, the idle close
failed its final checkpoint with `native_runner_checkpoint_owner_lost`,
the next run failed
`native_instruction_materialization_handoff_unavailable`, and the retry
was cancelled with `execution_reconciliation_required`. The task stayed
blocked. A hosted instance showed the same class as "previous run's
session on this sandbox could not be retired".

**Expected behavior**

The follow-up run waits for the idle close (inside the existing bounded
release wait) and then starts. The finished owner saves its final
checkpoint and closes. The task completes hands-off.

**Steps to reproduce**

1. Run an opencode agent on a Daytona sandbox with a warm session (idle
timeout 300 s).
2. Let a run finish.
3. Post a comment on the task 298–306 s after the run ended.

**Paperclip version or commit**

master `444dc18d98`.

**Deployment mode**

Local server with Daytona remote sandboxes (restart-chaos harness, light
lane). The same class was seen on a hosted staging instance.

## What Changed

- `claimWarmNativeInstructionCopy` now also waits when an idle close is
in flight for the session scope (`closingWarmNativeSessions`). The wait
uses the same deadline as the existing release wait
(`warmSessionReleaseWait`), so it adds no new wait and nothing before
`turn.start` on the normal path.
- `validateNativeRunnerCheckpointOwner` accepts the finished owner's
final checkpoint (accepted result, no new work) when
`issues.executionRunId` points at a successor run that is `queued` or
`scheduled_retry` for the same company and task and has no
`native_run_finalizations` row.
- Tests: a server test for a follow-up that arrives while the idle
timeout closes the warm session; four new scenarios (`queued_successor`,
`scheduled_retry_successor` × `runner.suspend`/`turn.stop`) and four new
negatives (`queued_wrong_issue`, `queued_new_work` × both commands) in
the early-retirement checkpoint matrix.
- Restart-chaos harness:
`RESTART_CHAOS_MIX_FOLLOW_UP_AFTER_END_MS=<min>-<max>` and
`RESTART_CHAOS_MIX_FOLLOW_UP_COUNT` post the staging-mix follow-up a set
delay after the task's last run ended, so a cell can aim the follow-up
at the warm idle close. The restart-chaos README describes both
variables. A failed post is retried with the same `clientRequestId`, the
grade check `after_end_follow_ups_served` needs an agent reply after
each post and the ledger marker, and the task budget includes the
planned waits.
- Harness: the wave database cleanup
(`tests/runner-e2e/controller-database.ts`) no longer waits forever when
a group signal (for example `timeout` or Ctrl-C) already stopped the
embedded Postgres. Before, `embedded-postgres` `stop()` waited for an
exit event that had already fired, and an interrupted wave hung after
the Daytona R6 sweep.

## Verification

- Fix 1, red without the product hunk: `native-session-executor.test.ts
-t "idle timeout is closing"` fails with `promise rejected "Error:
native_session_supervisor_busy"`. Green with the fix: 1 passed.
- Fix 2, red without the product hunk:
`native-runner-restart-recovery.integration.test.ts` fails 4 of 208
(`queued_successor` and `scheduled_retry_successor` ×
`runner.suspend`/`turn.stop`) with
`native_runner_checkpoint_owner_lost`. Green with the fix: 208 passed,
14 skipped. The two new negative cases pass in both states.
- Harness hang fix, red without the fix: the new
`controller-database.test.ts` case ("finishes cleanup when the database
already stopped because a group signal reached it") times out on cleanup
(`expected 'hung' to be 'cleaned'`). Green with the fix: 5 passed, 1
skipped.
- After the rebase onto master `7154030ef6`, both server test files
together: 1639 passed, 21 skipped.
- Restart-chaos harness tests (`staging-mix`, `staging-mix-retry`,
`observer`): 75 passed, including a test for the after-end retry, the
`after_end_follow_ups_served` grade check and the added budget. `tsc
--noEmit` passes for `server`, `tests/runner-e2e` and
`tests/restart-chaos`.
- `pnpm verify` exits 2 because `tests/restart-chaos/observer.ts` is a
global CI input. Required CI supplies the broad coverage.
- Daytona cell on this branch (4 opencode tasks, warm sandboxes, 1 vCPU,
follow-up comment posted 300.8–308.2 s after each task's last run ended,
image built for master `444dc18d98`): 4/4 tasks done hands-off, 8/8 runs
succeeded, 0 Retry banners, 0 `owner_lost` / `supervisor_busy` / "could
not be retired" / `execution_reconciliation_required` lines in the
server log, environment delete clean, warm-shutdown grade pass, 0
sandboxes left. The harness exited by itself after 1076 s.
- Same cell before the fix (master, 8 tasks): 1/8 tasks blocked
(`native_session_supervisor_busy` →
`native_runner_checkpoint_owner_lost` →
`execution_reconciliation_required`). With fix 1 only: 1/8 still blocked
(`owner_lost` on the idle close). With both fixes: 8/8 done and 0 failed
runs (cut by the 20-minute cap before cleanup), then 4/4 above.
- The cell grade `T:completion_once` fails on all 4 tasks by design: the
timed follow-up comment arrives after the task is done, so the agent
posts a second completion comment.

## Risks

- Fix 1 can delay a follow-up by up to the existing warm release wait
(30 s) while an idle close finishes. Before this change, that run
failed. If the close fails or the wait expires, the claim returns null
and the run uses a fresh instruction copy, as before.
- Fix 2 widens the checkpoint ownership fence from #18011. It does not
reopen what #18011 closed:
- The exemption applies only to the finished owner's final save: the
checkpoint must hold an accepted result and no new work, so no provider
turn can be granted through it.
- The successor must still be unclaimed (`queued` or `scheduled_retry`)
and must have no native coordinator (`native_run_finalizations`) row. No
controller has started it, so nothing can have consumed or depended on
the predecessor's checkpoint. Once the successor is claimed, the
existing preparing/claimed rules from #18011 apply unchanged.
  - The successor must belong to the same company and the same task.
- Cases that must still reject, with tests in the same matrix:
- `queued_wrong_issue`: a queued successor for a different task is
rejected with `native_runner_checkpoint_owner_lost` and the stored
checkpoint does not change.
- `queued_new_work`: a `turn.start` (new work) while the successor is
queued is rejected with `native_runner_checkpoint_owner_lost` and the
stored checkpoint does not change.
- All earlier rejections stay: unclaimed native coordinator,
dispatching, expired or foreign preparation, foreign native preparation,
wrong task, new work, unaccepted result, replaced controller.
- Not changed: an owner whose close fails still blocks environment
delete (409). That is a separate product decision.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edit, test runs). Earlier commits on
this branch were written by the same model in an earlier session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:42:50 -06:00
DottaandClaude Opus 5.5 497ccba8f1 fix(native): replace a reattached Runner that left its recovery read unanswered (#21272)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs execute in a remote Runner that the server
controls over the durable PRP channel
> - When a run is interrupted, a same-run retry reattaches the live
Runner and reads its session (`session.snapshot`) before it continues
> - On staging, the reattached Runner authenticated and restored its
session, and then did not answer the read
> - The recovery kept that Runner, so each later attempt adopted the
same Runner and failed the same way until the attempt budget ended the
run
> - This pull request makes the next attempt replace that Runner with a
fresh Runner over its saved journal, and makes a failed Codex history
read say what failed
> - The benefit is that an interrupted Codex run continues without a
manual Retry, and its provider turn is not sent again

## Linked Issues or Issue Description

Refs #21161, Refs #21053, Refs #20853

**What happened?**

A Codex agent on Daytona ran recovery runs (wake
`issue_recovery_action_restored`). In one hour, 11 of about 35 finished
runs failed. All failed the same way:

- `native_session_recovery_failed: HarnessCapabilityUnavailableError:
read is unavailable: Error: runnerd exited while waiting for
session.snapshot` (8 runs)
- `native_session_recovery_failed: HarnessReconciliationError:
codex_history_read_failed: thread/turns/list requires supported
paginated history; Error: provider_transport_failed: PRP command
session.snapshot timed out (runner connected; pending ahead 0; last
runner event seq 338 76s ago)` (3 runs)

The run log of each run shows one sequence:

1. Attempt 1 fails with `runnerd did not report the provider turn
identity`. The close leaves the Runner alive and unsettled for the
same-run reattachment.
2. Attempt 2 adopts that Runner (same PID). The Runner authenticates.
The log shows `runnerd restored its durable PRP session and provider
thread`. Then the driver's recovery read (`thread/read` or
`thread/turns/list`, which both send `session.snapshot`) gets no answer.
The Runner still answers WebSocket pings, but it sends no command result
and almost no events (`pending ahead 0`, last event 76 s to 391 s
earlier). The recovery treats this as transient and detaches. The Runner
stays alive.
3. Attempt 3 adopts the same Runner again and fails the same way. In
some runs a protocol fault (`thread_binding_mismatch`) closes the
transport during the read, so the message is `runnerd exited while
waiting for session.snapshot`.
4. The provider attempt budget ends the run.

#21161 did not help. Its wait extension applies only while a queued
answer is being committed. Here the Runner sent no answer at all.

The `paginated history` text was also wrong. `codex-history.ts` called
every failed history read unsupported, also a transport timeout.

**Expected behavior**

A Runner that left a recovery read unanswered is not adopted again. The
next attempt starts a fresh Runner from the saved journal, resumes the
provider session from the saved checkpoint, and does not send the
provider turn again. A failed history read reports the real failure. A
Codex build that refuses paginated history falls back to one complete
thread read.

**Steps to reproduce**

1. Start a Codex run in a remote (Daytona) Runner.
2. Let the first attempt fail at turn start, so that the close keeps the
Runner alive.
3. Let the reattached Runner stop answering commands after it
authenticates (for example, stop the Runner process after its first
answer).
4. See `PRP command session.snapshot timed out (runner connected;
pending ahead 0; ...)` on attempt 2 and the same failure on attempt 3,
with the same Runner PID.

**Paperclip version or commit**

Staging Core `57aeda190b` (contains #21161 and #21120). Fixed on
`master` after `53608bd1d6`.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`, `codex_app_server`).

## What Changed

- `nativeRecoveryCommandUnanswered` (executor) finds a
`native_session_recovery_failed` whose Runner did not answer a recovery
command (`PRP command <type> timed out` or `runnerd exited while waiting
for <type>`). Identity, authority, integrity and incompatibility
refusals do not match.
- When such a failure ends a `reattach_remote_runner` attempt on a
sandbox target, the failure detail records `adoptedRunnerUnanswered:
true`.
- `claimNativeRestartRecoveries` copies it to the next claim as
`replaceUnansweredRunner: true`, in the same way as
`exitedBeforeSessionOpen`.
- `verifyRemoteRunnerOrSettledExit` with `replaceUnansweredRunner` keeps
all identity checks. When the old Runner is still alive, it returns no
adopted process, and the attempt launches a fresh Runner from the saved
journal. The existing launch fence stops the old `paperclip-runnerd` on
that state folder and its whole lineage, and fails the launch closed
when it cannot prove them gone. A Runner that already exited mid-turn
keeps the existing fail-closed path, because its provider or tool
processes can survive where the fence cannot find them. The task then
continues in a fresh session from the saved work. The attempt budget is
unchanged.
- `codex-history.ts`: a history read that fails without a provider
answer is now `codex_history_read_failed: <method> read failed;
<error>`. A provider refusal of the method (`-32601` or `-32600`) on the
first page falls back to one `thread/read` with `includeTurns: true`.
That fallback fails closed when the thread differs, or when turns, items
or turn identities are missing. A refusal on a later page still fails
closed.
- `doc/execution-semantics.md` describes both rules. The launch-fence
code comment now names the replacement case.

### Changed existing test

`codex-history.test.ts` "does not turn an interrupted page or
unsupported method into idle history" used a plain `Error("method not
found")` on page 2 and expected the `supported paginated history` text.
A plain error is now a failed read, not an unsupported method. The test
now uses a real provider refusal (`CodexRpcError` `-32601`) on page 2
and still expects a fail-closed rejection. A new test covers the
failed-read message.

## Verification

- Red before the fix, green after:
- `server/src/services/native-runtime/native-session-executor.test.ts` ›
"a same-run reattachment whose adopted Runner never answered its
recovery read": the replacement and staging-message tests failed on the
old code (the Runner was adopted again, the three staging messages were
not recognized). Two further tests ("never resumes beside an exited
unanswered Runner's lineage") failed on the first revision of this PR,
which relaunched beside an exited mid-turn Runner, and pass now. All 15
pass.
- `native-runner-restart-recovery.integration.test.ts` › "asks the
same-run reattachment to replace a Runner that left its recovery read
unanswered" (embedded Postgres): the claim had no
`replaceUnansweredRunner` on the old code. Now passes.
- `codex-history.test.ts`: 7 new tests failed on the old code. All 16
pass.
- Real `paperclip-runnerd` and fake Codex:
`runnerd-codex-transport-remote-retention.test.ts` › "continues the same
provider thread from a fresh Runner over the journal of a retained
Runner that stopped answering, without replaying its turn". It runs the
staging sequence: turn started, close keeps the stopped Runner, a second
controller adopts it, the Runner stops answering and the controller
detaches, the Runner and provider are killed (the launch fence), and a
fresh Runner over the same journal returns the same thread. The fake
Codex call log shows one `thread/start` and one `turn/start`.
- Focused suites: `packages/paperclip-runner` `src/drivers/codex/` plus
the remote-retention, adoption and cold-restore transport suites (442
tests). Two suites timed out under parallel load and pass alone. Server:
`native-session-executor`, `native-restart-recovery`,
`native-runner-restart-recovery.integration`,
`runner-lost-mid-turn-continuation` (1714 passed, 21 skipped).
- `pnpm typecheck` in `server` (all layers) and `tsc` for
`paperclip-runner` source and tests pass.
- Not run locally: the container-gated remote Runner suites
(`PAPERCLIP_TEST_RUNNER_CONTAINER`) and the full suite. CI covers the
full suite.

## Risks

- Medium. The replacement relaunches a Runner over a journal that is not
settled, but only while the old Runner is alive. Safety comes from three
facts: the launch fence stops the old Runner and its lineage with proof
or fails the launch; the Runner restores its own durable journal (as
local `resume_dead_runner` does); and the real-runnerd test shows that
the turn is not sent again. An exited Runner is never relaunched beside
its possible survivors.
- If the previous close left a `runner.suspend` pending, a fresh Runner
can execute it and exit. The attempt then fails as it does today. The
usual detach path after an unanswered read sends no suspend.
- The replacement uses the existing attempt budget. It does not add
attempts.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code
with tool use (shell, file edits, browser reads of staging run events
and logs), extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:06:24 -06:00
Nicky LeachandPaperclip a924d2e2c7 Persist portable Runner ownership through controller recovery (#21066)
## Thinking Path

> - Paperclip delegates remote Runner execution to environment plugins.
> - A plugin can move a session to another task allocation.
> - The Core controller can lose an ownership reply or restart before
the Runner starts.
> - The source and target need a durable record of the same request.
> - This change binds restart and attachment to that request and the
original unlaunched task lease.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Recovery of plugin Runner sessions on replacement compute.

**Current behavior**
Core cannot replay a session ownership request from saved lease state
during startup.

**Proposed behavior**
Record the operation on both leases and check the matched ownership
receipt before attaching. Preserve the original pre-admission lease
across controller restart.

Managed environment configuration is in
[#21065](https://github.com/paperclipai/paperclip/pull/21065). Related
warm sandbox cleanup is in
[#20065](https://github.com/paperclipai/paperclip/pull/20065). This
layer preserves the same-sandbox fallback from
[#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace
placement follows in
[#20743](https://github.com/paperclipai/paperclip/pull/20743).

## What Changed

- Add the optional `sessionHandoffVersion` capability and the ownership
task operation.
- Save matching ownership claims and fence source checkpoint access.
Replay a pending request on the original target.
- Restore a portable session using the durable claim and the current
credentials for the same account. Start a fresh conversation when the
provider reports an unavailable checkpoint, or when Core's persisted
compatibility decision selects a different session after transfer. The
latter requires an unlaunched target; subsequent recovery must match the
fresh task's saved identity. Both paths retain the workspace transfer
and source fencing.
- Reuse the original unlaunched task lease after hard or graceful
controller restart. After a move has attached, validate the surviving
Runner through the saved claim rather than requiring its logical lease
ID to equal the new physical lease.
- Keep older retained leases without a saved account identity on the
existing resume path.
- Apply the existing supported MCP-only compatibility rule to warm
recovery and predecessor accounting. Persist the capability discovered
at admission and reuse it after restart without repeating discovery.
- Keep predecessor-retirement fallback scoped to the same physical
sandbox. A refusal after a portable move preserves the session instead
of requesting a replacement that the plugin driver cannot enforce.

## Verification

- On `1eb527be27`, server typecheck (including Runner build and SDK
dependencies) passed.
- Independent final verification passed 168 tests across portable source
recovery, session resume, controller recovery, and task authority; two
existing tests were skipped. The full portable source integration suite
passed 64 tests.
- The new integration follows confirmed ownership through an
incompatible instruction change, persisted fresh admission, target-only
prepare/auth/submit without the source checkpoint, and reattachment
after controller replacement. It also rejects unpersisted session
selection, ambiguous target leases, and incorrect saved identities.
- Before this final change, independent verification passed 1,667 tests
across executor, session resume, recovery usage, remote warm recovery,
runtime selection, and runtime execution; the environment lifecycle
suite passed 208 tests.
- Existing regressions cover previous-release account metadata,
supported and unsupported MCP changes, provider/instruction
incompatibility, and admission capability reuse without another
discovery call.
- On `51d7eec373`, all four S3 multipart throttle tests passed
independently. CI exposed an existing timing-sensitive test twice; its
20 ms overlap assumption now uses synchronized retries and an event-loop
yield. Production upload code is unchanged. Temporary concurrency caps
of one and eight both fail the retained exact-four assertion.
- Exact SDK source-pin checks passed for both stack heads. The upper
#20743 code patch remains unchanged; its documentation preserves both
layers' behavior.

## Risks

- The handoff protocol is opt-in. Existing plugin leases without
portable ownership metadata keep the previous release's resume path.
Live attachment still requires end-to-end qualification. An unverified
claim blocks startup. Git workspaces prepared on a portable allocation
also need the workspace placement layer.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 10:43:48 -08:00
DottaandClaude Opus 5.5 3fb35fcf13 fix(native): restart an interrupted bootstrap whose runner connected before the deploy (#21120)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner (`paperclip-runnerd`) in a
Daytona sandbox. A deploy stops the server and a new server recovers
each run.
> - When the server stops during a run's bootstrap, recovery restarts
that bootstrap on the same run (`bootstrap_incomplete`). The claim is
only granted when the controller checkpoint in Postgres proves that no
provider turn was sent.
> - Host preparation then reads the sandbox Runner state. A Runner that
already authenticated is `ready`, not `suspended`. The restart is
allowed only with a stop proof for that Runner.
> - The stop proof was taken in two places only: from a restored local
controller root, or for a Runner that never connected. A new server
after a deploy can have neither, so the run failed with
`runner_harness_state_mismatch: resumed_not_suspended` and needed a
manual Retry.
> - This pull request lets host preparation take the same Postgres
checkpoint proof that granted the claim, stop that exact Runner, and
restart the bootstrap.
> - The benefit is that a deploy during a bootstrap no longer fails the
run. No provider turn can be replayed, because the proof shows that none
was sent.

## Linked Issues or Issue Description

**What happened?**
On staging, three Codex runs on Daytona failed with
`native_session_interrupted` and `runner_ingress_unavailable: runnerd
process failed: runner_harness_state_mismatch: resumed_not_suspended`
(12:48Z twice and 13:19Z on 2026-11-12). In each run the Runner
authenticated 1 to 4 seconds before a deploy stopped the server
("controller authority detached for restart; durable runner left
alive"). The new server chose "Restarting an incomplete native runner
bootstrap on the same heartbeat run". Its host preparation found the
sandbox Runner `ready`, had no stop proof, and failed the run. The run
log has no "Stopped the interrupted runner" line, so the earlier stop
step did not run on the new server.

**Expected behavior**
The run restarts its bootstrap on the same sandbox and continues without
a manual Retry. A provider turn that may have run is never replayed.

**Steps to reproduce**
1. Start a Codex run on a Daytona sandbox.
2. Deploy the server after the Runner authenticates and before the first
turn starts.
3. Let the new server recover the run with a `bootstrap_incomplete`
claim on a server that has no local controller root for the run.
4. Host preparation fails with `runner_harness_state_mismatch:
resumed_not_suspended`.

**Paperclip version or commit**
Staging at 1d153f62b6. The code is the same on master at 704b3f4d63.

**Deployment mode**
Hosted staging (paperclip.staging.paperclip.app), remote sandbox
execution on Daytona.

**Agent adapter(s) involved**
Paperclip Runner with Codex (`codex_app_server`).

## What Changed

- `native-runner-checkpoint.ts`:
`nativeRunnerCheckpointRemoteBootstrapEvidence` returns the bootstrap
proof together with the runner instance that the checkpoint names.
`nativeRunnerCheckpointRemoteBootstrapProof` now wraps it and behaves as
before.
- `native-session-executor.ts`:
- State migration runs before this server registers its own controller
for the run. Under a `bootstrap_incomplete` claim for a remote sandbox,
it now reads the interrupted controller's checkpoint proof there
(`captureClaimTimeRemoteBootstrapProof`). A later read is not used,
because the new controller rewrites the checkpoint with its own
bootstrap commands.
- Host preparation: when this run's sandbox Runner is not suspended, it
runs the same-run restart proof. The proof accepts a Runner that never
connected (as before), or the claim-time proof for the exact runner
instance in the sandbox state. It then stops that exact Runner by its
identity marker and records the stop proof. A backup, a different run
id, another runner instance, or a failed stop still fails closed.
- `native-session-executor.test.ts`: new
`interrupted_bootstrap_connected_checkpoint_proven` case. The Runner is
`ready` and connected, and no stop proof exists yet. It fails before the
fix with `resumed_not_suspended` and passes after it.
`interrupted_bootstrap_unproven` still fails closed.
- `native-bootstrap-claim-time-proof.integration.test.ts` (embedded
Postgres, no mocks of the proof): an interrupted controller that had
connected registers its checkpoint, the claim-time proof is read, and
then a replacement controller registers its own checkpoint. A live read
after that returns `bootstrap_incomplete` even when the interrupted
controller had sent `turn.start`. The claim-time proof stays false for
that case and true for the bootstrap-only case, and it refuses another
runner instance.

## Verification

- Red: with the source change reverted, `pnpm exec vitest run
server/src/services/native-runtime/native-session-executor.test.ts -t
"bootstraps only an untouched provider session"` fails on the new case.
- Green: the same command passes (65 tests after merging master).
- `pnpm exec vitest run
server/src/services/native-runtime/native-bootstrap-claim-time-proof.integration.test.ts`:
2 passed.
- `pnpm exec vitest run
server/src/services/native-runtime/native-session-executor.test.ts
server/src/services/native-runtime/native-restart-recovery.test.ts`:
1474 passed.
- `npx tsc --noEmit -p server/tsconfig.json`: passes.
- Merged `origin/master` (including #21134) without conflicts and reran
the tests above.

## Risks

- Low. The new path runs only under a `bootstrap_incomplete` claim for
the same run id, with no failover backup, and only after the Postgres
controller checkpoint proves that no provider turn was sent. It stops a
Runner only when the identity marker and process fingerprint match,
which is the same check the reattach path uses.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, browser, code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #21134 (the never-launched branch of the same helper), #21079
(stops a leftover Runner at launch time; this change acts earlier, in
host preparation).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 11:38:06 -06:00
DottaandClaude Opus 5.5 d5bd38468d fix(runner): stop a leftover Runner on the same state folder before a relaunch (#21079)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner (`paperclip-runnerd`) in a
Daytona sandbox. The Runner keeps its saved state in one folder per
session and listens on one port in the sandbox.
> - During an automatic upgrade, the old server can launch a Runner a
few seconds before it stops. That Runner stays alive in the sandbox and
keeps the state folder and the port.
> - The new server then restarts the run on the same sandbox. Each new
Runner fails at once with "Address already in use"
(`runner_ingress_bind_conflict`). Each launch also overwrites the
process marker that pointed at the old Runner, so the next check cannot
find the old Runner to stop it. The run restarted about 12 times in 19
minutes and then stopped, and the task stayed blocked.
> - Two Runner processes must never use the same saved state at the same
time.
> - This pull request makes each Runner launch stop any live Runner that
still names the same state folder, before the new Runner starts.
> - The benefit is that a restart after an upgrade gets the port and
continues the run, instead of looping on the bind conflict.

## Linked Issues or Issue Description

**What happened?**
On staging, a Runner launched by the old server just before an upgrade
kept the Runner port. Every relaunch by the new server exited within
seconds with "runner_ingress_bind_conflict: failed to bind listener:
Address already in use (os error 98)". The run looped for about 19
minutes and then failed with `native_runner_process_exited`. The task
was blocked and needed a person.

**Expected behavior**
The relaunch stops the leftover Runner of the same session, gets the
port, and continues the run.

**Steps to reproduce**
1. Launch a Runner for a run in a sandbox.
2. Stop the server before the Runner is adopted (an upgrade). The Runner
stays alive.
3. Let the new server restart the run's bootstrap. The new Runner cannot
bind the port, and the next restart cannot see the old Runner because
the marker now names the dead new process.

## What Changed

- `native-session-executor.ts`: the detached Runner launch script now
runs a short fence before it writes the process marker.
- It finds processes whose argv[0] is `paperclip-runnerd`, that are not
a `reap-lineage` command, and whose `--state-dir` is followed by this
exact folder. Runners on another folder, a folder named by a different
flag, and other programs are not touched.
- It records the stale Runner's lineage by PID **and start time**: its
descendants, and members of a process group or session that a recorded
process leads while that leader is alive (provider tools start their own
sessions). It signals a PID only while its start time still matches, so
a reused PID is never signalled, and it never signals a whole process
group.
- TERM, up to 5 s wait, then KILL, up to 2 s wait. Then it runs
`<runnerd> reap-lineage --state-dir … --runner-pid …` for the stopped
Runner, which stops the tools its lineage journal recorded that have
already left the process tree.
- It fails closed: a survivor (`runner_stale_state_holder_alive`) or a
reap that leaves a recorded process alive or gives no report
(`runner_stale_lineage_unproven`) makes the launch exit 75 **before**
the marker is written, so the existing marker is kept. The reason goes
to the process stderr log only when that path is a plain file (same rule
as the Runner stderr log). The whole stop is bounded at about 14 s,
under the 20 s marker wait.
- Two reap refusals let the launch continue with a
`runner_stale_lineage_partial` warning: `journal_unavailable` (the
Runner kept no journal; its visible lineage is already gone) and
`unattributed_orphan` (reported only after every recorded process is
gone, for orphans the reaper can neither claim nor signal) **only when
every orphan it names is proven to have another owner**: this launch's
own detached shell, a `paperclip-runnerd` serving a different
`--state-dir`, or that Runner's stderr trimmer. Any other orphan could
be an old tool that escaped between samples, so the launch fails closed.
- When `/proc` is not readable, or `grep` is missing, the fence does
nothing (the old behavior).
- `native-session-executor.test.ts`: Linux tests (a stale Runner whose
tool runs in its own session, plus four look-alikes that must survive;
fail-closed with the marker kept, with a plain and a symlinked log; reap
refusals for orphans owned elsewhere, an unknown orphan, and no journal;
a normal launch starts no extra node process) and a cross-platform check
that the embedded script parses and has no shell-special characters.

**When a launch happens vs. a reattach.** The launch is reached only
after the server chose not to adopt a Runner. A same-run reattachment
(`reattach_remote_runner`) first verifies the marker's Runner: if it is
alive it is adopted and nothing launches; a transient verification
failure (for example the read timeout that #21053 keeps the Runner alive
for) throws and never reaches a launch; only a Runner proven not running
falls through to a fresh launch. Warm recovery likewise throws instead
of launching when revalidation fails. Every launch overwrites the
marker, so after a launch no reattachment could adopt a leftover Runner
anyway, and its port would make the new Runner fail with
`runner_ingress_bind_conflict`. Stopping it therefore never removes a
Runner that the same run could still re-adopt.

## Verification

- Linux (Debian bookworm container, node 24): the Linux tests pass on
this head. The same tests against the previous head (8df6a9099e) fail:
it killed a `paperclip-runnerd reap-lineage --state-dir <same folder>`
command, and it launched although the stale lineage was not proven gone.
On master there is no fence, so the stale Runner stays alive.
- Boot time: a normal launch took 19–35 ms for the whole launch script
in the container with the fence; node is not started by the fence
(asserted by a test). With a stale holder, the stop and launch took
about 160 ms.
- macOS: `npx vitest run
src/services/native-runtime/native-session-executor.test.ts` passes
(1412 passed, 4 skipped: the Linux-only tests).
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- A launch now stops an older Runner on the same state folder. See "When
a launch happens vs. a reattach" above: this only happens when that
Runner is already impossible to adopt and would block the new one.
- A normal launch pays one `grep` over `/proc` plus reading argv[0] of
the processes that name the folder (its own subshell). No node start, no
extra round trip, no wait.
- If `paperclip-runnerd reap-lineage` cannot verify the stale lineage,
the launch fails closed instead of starting a second Runner next to old
tools.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, code search, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:34:22 -06:00
DottaandClaude Opus 5.5 fb8b61bc3e fix(native): restart a bootstrap whose closing server staged files but never launched its Runner (#21134)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner (`paperclip-runnerd`) in a
Daytona sandbox. A release swap stops the old server and a new server
recovers each run.
> - When the old server stops during a run's bootstrap, the new server
restarts that bootstrap on the same run (`bootstrap_incomplete`). The
claim is granted only when the Postgres controller checkpoint proves
that no provider turn was sent.
> - The old server can stop after it staged launch files (Codex home,
config) into the sandbox session folder, but before it launched the
Runner. Its detach fence stops it there.
> - The new server then finds no Runner state in the sandbox. A fresh
bootstrap must create the session folder itself, and the folder already
exists, so the run failed with `runner_harness_state_mismatch:
resumed_unavailable`. The failure is an ambiguous session, so the task
waits for a person.
> - This pull request lets host preparation prove that no Runner was
ever launched in that folder, move the folder aside, and restart the
bootstrap.
> - The benefit is that a release swap during this part of a bootstrap
no longer blocks the task. No provider work can be lost or replayed,
because no Runner ever ran.

## Linked Issues or Issue Description

Refs #21120 (same release-swap class: the Runner was launched and
connected before the old server stopped; that pull request stops it in
host preparation. This pull request covers the case where no Runner was
launched.)
Refs #21079 (launch-time fence for a stale Runner on the same state
folder.)

**What happened?**
On hosted staging, during a release swap on 2026-11-12, a Codex run on
Daytona failed with `native_session_interrupted` and
`runner_ingress_unavailable: runnerd process failed:
runner_harness_state_mismatch: resumed_unavailable`. The old server was
in `harness_state.launch_material` when its shutdown drain detached the
run. It recorded no Runner process identity. The new server chose
"Restarting an incomplete native runner bootstrap on the same heartbeat
run". Its host preparation found the sandbox resumed and no Runner
state. Then the claim of the session folder failed, because the old
server had already created that folder. The run needed a person.

**Expected behavior**
The run restarts its bootstrap on the same sandbox and continues without
a person. A provider turn is never replayed and two Runners never share
the run.

**Steps to reproduce**
1. Start a Codex run on a Daytona sandbox.
2. Stop the server after the bootstrap stages launch files into the
sandbox session folder and before it launches the Runner.
3. Let a new server recover the run with a `bootstrap_incomplete` claim.
The new server has no local controller root for the run.
4. Host preparation fails with `runner_harness_state_mismatch:
resumed_unavailable`.

**Paperclip version or commit**
Staging release `704b3f4d63`. The code is the same on master at
`ef2b53b675`.

**Deployment mode**
Hosted staging, remote sandbox execution on Daytona.

**Agent adapter(s) involved**
Paperclip Runner with Codex (`codex_app_server`).

## What Changed

- `native-session-executor.ts`: under a same-run `bootstrap_incomplete`
claim with no failover backup, when host preparation finds no Runner
state, a new check asks the sandbox in one command whether the session
folder exists and holds no Runner trace. A Runner trace is the live
`runner/runner-state.json`, the `runner/runner-process.identity` launch
marker (the launch script writes it before the Runner starts), an
`authority-epochs` archive, or any other `runner-state.json` or launch
marker anywhere under the folder. If the folder holds no Runner trace,
no Runner was launched there, so host preparation records the restart
proof (`absent_confirmed`). It then uses the existing path: move the
folder aside (it is kept for inspection), claim a new folder, and
bootstrap again. If the folder is missing, the existing fresh-claim path
runs unchanged. If any Runner trace exists, the run stays fail-closed as
before. An archived session is therefore never discarded.
- `native-session-executor.test.ts`: new
`interrupted_bootstrap_unlaunched` cases. Resumed and created sandboxes
restart the bootstrap. A launch marker, an archived authority epoch,
Runner state elsewhere in the folder, or a failover backup keeps the run
fail-closed with `resumed_unavailable`.

## Verification

- Red: with the source change reverted, `pnpm exec vitest run
server/src/services/native-runtime/native-session-executor.test.ts -t
"bootstraps only an untouched provider session" --maxWorkers=2` fails on
both `interrupted_bootstrap_unlaunched` restart cases with
`runner_harness_state_mismatch: resumed_unavailable`, which is the
staging error.
- Red: without the archived-state guard, the `_archived` and
`_nested_state` cases fail (the folder is moved aside). With the guard
they pass.
- Green: the same command passes (64 tests).
- `pnpm exec vitest run
server/src/services/native-runtime/native-session-executor.test.ts
--maxWorkers=2`: full file, 1422 passed before the guard commit. The
focused 64 pass on the final revision.
- `npx tsc --noEmit -p server/tsconfig.json`: passes.
- `git merge-tree` against #21120's head (`0d18f18928`) merges without
conflict. The new check runs only when no Runner state exists, so
#21120's new call site (it runs only when a Runner state exists) gets
the same results as before.

## Risks

- Low. The new path runs only in host preparation, under a
`bootstrap_incomplete` claim for the same run id, with no failover
backup. That claim is granted only after the previous controller's lease
has ended. With no Runner state or launch marker anywhere in the folder,
no Runner and no provider session existed there. The old folder is moved
aside, not deleted.
- A controller that is still launching at the same moment would have to
hold a lease that the claim already took over. The detach fence stops
such a launch, and #21079 adds a launch-time fence for a leftover Runner
on the same state folder.
- No added wait before a normal turn starts: the check runs only on this
failure path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, browser for read-only run evidence, code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:06:55 -06:00
DottaandClaude Opus 5.5 67403baaa4 fix(native): an agent termination Stop with no turn result stays a cancellation (#21093)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a person terminates an agent, Paperclip stops the agent's
running native turn
> - Two paths can record that Stop: the terminate route ("Cancelled
because the agent was terminated ...") and the lifecycle worker ("Agent
lifecycle stop requested"). Both are system-initiated and expected
> - When the stopped turn ends without a semantic result, the native
executor keeps the run cancelled only for a person's Stop, a
reassignment, or a governance pause. A termination Stop fell through, so
the run ended `failed` with `native_session_interrupted`
> - Whether the run ended cancelled or failed depended on which path won
the race
> - This pull request treats an acknowledged termination Stop like a
person's Stop in the executor and in the lease-release terminalization
> - The benefit is that a terminated agent's interrupted run ends
cancelled, whichever path records the Stop first

## Linked Issues or Issue Description

Refs #20186 (the same treatment for a governance pause Stop)

**What happened?**
A person chose Terminate for an agent while its native turn ran. The
lifecycle Stop recorded an acknowledged native cancellation (initiator
system). The turn ended without a semantic result. The executor then
wrote the run as `failed` with `native_finalization_missing: session
returned no semantic result`.

**Expected behavior**
The run ends `cancelled`. A termination is an intentional Stop, not a
failure to retry.

**Steps to reproduce**
Terminate an agent while its native turn runs, so that the Stop
interrupts the turn before it records a result. Before this change, the
run became `failed` when the lifecycle worker recorded the Stop first.

## What Changed

- `governance-pause-stop.ts`: adds `AGENT_TERMINATED_STOP_REASON`,
`COMPANY_ARCHIVED_STOP_REASON` and `isAgentLifecycleStopRecord`. The
predicate accepts a system-initiated, expected control-plane
cancellation whose reason is a lifecycle, termination or company-archive
Stop.
- `acknowledged-native-stop.ts`: adds
`hasAcknowledgedNativeLifecycleStopIntent`, which requires that record
and an acknowledged, dispatched native Stop receipt.
- `native-session-executor.ts`: the no-semantic-result branch accepts
that intent, so it settles the coordinator as `native_retry_cancelled`
and raises `NativeCancellationPendingRecoveryError`, as for a person's
Stop.
- `heartbeat/run-lifecycle.ts`: lease-release terminalization keeps such
a run `cancelled`.
- The terminate route and the lifecycle driver use the shared reason
constants. The reason text does not change.

## Verification

- `native-session-executor.test.ts`: a new case for each reason
(lifecycle Stop, terminate route, company archive). Each was red before
the change and is green now. The earlier control case now uses a system
Stop that is neither a pause nor a termination, and it still takes the
retry path. Full file: 1414 passed.
- `heartbeat-run-lease-release-terminalization.test.ts`: a new case for
a lifecycle Stop when teardown wins the finalizer race. It was red
before the change (`interrupted`) and is green now.
- Product evidence: a paid chat-eval run of `agents.retire-agent` on a
branch that combines this change with three related fixes. The result is
in a PR comment.

## Risks

- A termination or company-archive Stop that interrupts a turn no longer
schedules an exact resume of that turn. The agent is leaving service, so
no resume can run.
- Other system Stops keep their current behavior.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 06:19:21 -06:00
DottaandClaude Opus 5.5 5820d43197 fix(runner): relaunch a settled OpenCode Runner with its admitted artifact paths (#21076)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - OpenCode agent turns run in a remote Runner in a Daytona sandbox.
The Runner saves its state, so a retry of the same run can start a new
Runner from that saved state.
> - The first launch of a run starts OpenCode from the admitted provider
files under one folder in the sandbox. The new Runner replays that first
launch and checks that its own launch profile names the same files.
> - When a retry reconnects to a Runner that already stopped, Paperclip
starts a new Runner. On that path, Paperclip did not restore the
admitted provider files and gave the new Runner the provider pack folder
instead.
> - The new Runner then refused the replayed launch ("does not match the
runner-owned qualified profile"). That used the last retry, and the task
stayed blocked.
> - This pull request restores the admitted provider files and gives the
new Runner the same folder as the first launch.
> - The benefit is that a retry after a server upgrade continues the
same run instead of blocking the task.

## Linked Issues or Issue Description

**What happened?**
On staging, after an automatic upgrade, a run used all 3 attempts. The
last attempt started a new Runner from its saved state. It failed in the
drain step with "OpenCode run.prepare launch does not match the
runner-owned qualified profile". The task was blocked and needed a
person.

**Expected behavior**
The new Runner gets the same OpenCode launch files as the first launch
of the run, and the run continues.

**Steps to reproduce**
1. Start an OpenCode run on a remote sandbox with durable provider
artifacts.
2. Let the server restart, so the retry reconnects with
`reattach_remote_runner`.
3. The Runner is found settled, so a new Runner starts from the saved
state. Its launch profile points at the provider pack folder, not at the
admitted artifact folder.

## What Changed

- `native-session-executor.ts`: when a reconnect relaunches a settled
Runner and the run has admitted provider artifacts, restore those
artifacts and prepare the same artifact folder as the first launch.
Other reconnects (live adoption) do not change. Normal turn starts do
not change.
- `native-session-executor.test.ts`: the existing settled-relaunch case
now also covers a run with admitted provider artifacts. It expects the
restore and the admitted folder in the launch profile.

## Verification

- Red: without the fix, the new test case fails
(`restoreNativeProviderArtifacts` is not called, and the launch uses the
provider pack folder).
- Green: `npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/remote-opencode-artifacts.test.ts` passes
1417/1417.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low risk. The change applies only when a reconnect must relaunch a
settled OpenCode Runner and the run has admitted provider artifacts.
- That relaunch now downloads the admitted artifacts from storage, as a
cold continuation already does. This adds time only on that recovery
path, not on a normal turn start.
- The artifact folder check (same account and same digest) still
applies, so a different sandbox account still fails closed.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, code search, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:24:17 -06:00
DottaandClaude Opus 5.5 559551e977 fix(native): wait for a busy agent directory while retiring a predecessor (#20991)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a reused warm sandbox first retires the previous
run's warm session, and that retirement collects the old session's agent
files
> - On a server without a persistent volume, agent-file collection takes
a per-agent Postgres lock, so runs of one agent collect one at a time
> - #20863 sends a run whose predecessor cannot be retired to a fresh
sandbox. It classified every retirement error as unprovable, including a
busy lock held by another run of the same agent
> - So ordinary lock contention failed the run, dropped a healthy
sandbox and rested the task
> - This pull request makes retirement wait for a busy directory lock in
the same run, and keeps the fresh-sandbox fallback for retirements that
stay unprovable after the lock is obtained
> - The benefit is that busy agents no longer fail follow-up runs on
their warm sandboxes

## Linked Issues or Issue Description

**What happened?**
A no-fault restart-chaos wave with 8 concurrent runs of one Claude agent
on Daytona, with a volume-free server, failed one monitor-due follow-up
after 6 seconds. The error was `adapter_failed`, cause
`native_predecessor_retirement_unproven`: "The previous run's session on
this sandbox could not be retired (Agent directory operation is already
active)". The run also logged `INSTRUCTION_COLLECTION_UNAVAILABLE`. The
stack shows the error came from `claimWarmNativeInstructionCopy` during
instruction preparation. The previous session had closed; only its
agent-file collection met the per-agent lock that another run held.

**Expected behavior**
A busy agent-directory lock is transient. The run waits for it and then
retires the predecessor on the same sandbox. Only a retirement that
stays unprovable after the lock is obtained takes the fresh-sandbox
fallback.

**Steps to reproduce**
1. Run several warm native tasks of one agent at once on Daytona with a
volume-free server.
2. Re-wake one task while another run of the agent collects its files.
3. The follow-up fails with `native_predecessor_retirement_unproven` and
its sandbox is dropped.

Refs #20863, #20829, #20957.

## What Changed

- `retireWarmPredecessor` (native executor): a busy agent-directory lock
(`AgentDirectoryOperationBusyError`) is waited out in the same run,
polling every second for up to five minutes
(`warmRetirementDirectoryWait`). The usual holder is another run's short
collection, so retirement then finishes on the same sandbox with the
copy this run adopted. The run is not deferred: deferring after adoption
would release the adopted copy and could skip saving the predecessor's
edits.
- Past that budget the run fails with its own error,
`native_predecessor_retirement_directory_busy`, not as an unprovable
retirement. Before failing it files the adopted copy's retirement
receipt, so the copy is kept as `pending_collection` and run cleanup
does not mark it `unavailable`. The owner is not marked unretired and
stays registered for its sandbox, so a later run retires it there.
- An idle-timeout close that meets the lock retries after the
agent-directory retry delay instead of leaving the session registered
with no retry.
- Closing idle sessions for environment delete (#20957) reports such an
owner as `busy`, not `failed`.

## Verification

- RED on master, then GREEN:
- `native-session-executor.test.ts -t "directory lock"`: two runs of one
agent contend on the lock while a follow-up retires the predecessor. On
master the claim rejects with `NativePredecessorRetirementUnprovenError`
(`Agent directory operation is already active`). Now a short-lived lock
is waited out and the predecessor is retired and collected on the same
sandbox; a lock that outlasts the budget fails with
`native_predecessor_retirement_directory_busy`, environment delete
reports `busy: 1`, a fresh-sandbox attempt does not forget the owner,
and once the lock is free a later run retires it.
- `native-session-executor.test.ts -t "idle close whose file
collection"`: on master the idle close never retries; now it retries and
closes.
- Existing suites pass: native executor, agent-directory busy heartbeat,
native pre-dispatch retry, native controller recovery and plugin runner
eviction. Server `tsc --noEmit` passes.

## Risks

- Under contention a follow-up can wait up to five minutes before its
first turn. The normal path adds no wait.
- A lock held for more than five minutes fails the run visibly instead
of dropping the sandbox.
- An idle owner whose collection stays busy keeps retrying at the 5 to
15 second agent-directory delay until the lock frees, a controller
restart, or the next admission.
- No persisted state changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with
tool use (shell, file edits, test runs) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:08:15 -06:00
DottaandClaude Opus 5.5 814fe5d20f fix(runner): issue the remote bootstrap ticket at dispatch and retry unauthenticated starts (#21073)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox start `runnerd` in the sandbox, and
Paperclip connects to it through provider ingress and authenticates it
with a one-use bootstrap ticket
> - The ticket lives for 60 seconds, but Paperclip issued it before the
remote launcher staged its files in the sandbox
> - Under load, staging took 60 to 81 seconds, so `runnerd` presented an
expired ticket and every connection was closed. The run failed with
`runner_ingress_auth_failed`, and the task stopped behind a manual Retry
> - This pull request issues a fresh ticket right before dispatch,
retries these failures automatically when the controller checkpoint
proves no runner ever authenticated, and adds spans that attribute the
slow startup window
> - The benefit is that a slow but healthy start authenticates, a failed
start recovers on its own, and the remaining startup delay becomes
measurable

## Linked Issues or Issue Description

**What happened?**

On a hosted instance, 9 runs of one Codex agent on Daytona failed in
about 70 minutes with `native_session_interrupted` and the message
`runner_ingress_auth_failed: Runner ingress did not complete PRP
authentication before the deadline.` In the same window only 6 runs
succeeded company-wide. The run log showed Paperclip reconnecting for 60
seconds while `runnerd` reported that its peer closed each WebSocket. In
294 remote runs with timing spans, the time between the bootstrap ticket
issue (end of `runner.transport.connect`) and the process dispatch
(start of `runner.process.launch`) decides the outcome without
exception:

- 60 s or more: 23 of 23 runs failed authentication.
- Under 60 s: 271 of 271 runs authenticated, in 0 to 1 s.

The tasks did not recover by themselves, because the failure was
classified as ambiguous state.

**Expected behavior**

A runner that starts slowly but correctly authenticates. If ingress
authentication still fails before any runner authenticated, Paperclip
retries the start automatically. The time between the start request and
the runner connection is visible in run spans.

**Steps to reproduce**

1. Run a Codex agent on a Daytona sandbox environment that uses provider
ingress.
2. Make remote staging (`runner.runtime.stage`, inside the remote
launcher) take 60 seconds or more. Many concurrent runs of the same
agent do this on staging.
3. See that `runner.prp.authenticate` fails after 60 seconds and the run
ends with `runner_ingress_auth_failed`.
4. The regression test `runnerd-bootstrap-ticket-refresh.test.ts`
reproduces this locally with a 75-second staging delay.

**Paperclip version or commit**

`4e519bd4` (deployed), reproduced on `origin/master` at `4228bcf89c`.

**Deployment mode**

Hosted (cloud), controller on Railway, sandboxes on Daytona.

**Agent adapter(s) involved**

`paperclip_runner` with the Codex app-server provider (`gpt-6.1-sol`),
remote sandbox target, provider-ingress transport.

## What Changed

- `RunnerProcessLaunchSpec` and `spawnRunner` accept
`refreshBootstrapTicket`. A restart does not inherit it.
- The runnerd Codex transport passes a refresh that issues a new
bootstrap ticket and flushes the controller checkpoint. It does not do
this for a warm-transition launch.
- The remote sandbox launcher calls the refresh after `ensureArtifact`
and immediately before the launch RPC, and puts the new ticket in the
launch environment.
- `inspectUnauthenticatedBootstrapCheckpoint` proves from a controller
checkpoint that no runner authenticated: no connections, no leases, no
used tickets, no events, and only pending commands.
- `nativeSessionFailureSourceCode` classifies the outbound ingress
startup failures as `runner_ingress_startup_failed`.
`nativeProviderRecoveryEvidence` treats it as pre-session
(`bootstrap_retry`) only when that proof holds for the exact run
identity and there is no provider session or provider event. Restart
recovery and failed-run retry accept the new code with the same rules as
similar startup codes.
- Remote startup now records `runner.control_plane.open`,
`runner.launch.prepare`, `runner.checkpoint.register` and
`stage.asset.codex_skills.verify`. The 6 to 11 minute gap between
`native.process_start_requested` and `runner.transport.connect` had no
spans before.

## Verification

- Red/green:
`packages/paperclip-runner/src/live/runnerd-bootstrap-ticket-refresh.test.ts`
starts a real `runnerd` with a 75-second staging delay. Without the fix
it fails with `runner never connected`. With the fix it opens the
session.
- Red/green: `native-session-executor.test.ts` "launches runnerd with a
bootstrap ticket issued after remote staging". Without the fix the
launch RPC gets the stale ticket.
- Red/green: `native-controller-recovery.integration.test.ts` "retries a
fresh remote start whose launched runner never authenticated through
provider ingress" (embedded Postgres). Without the fix the failure is
`native_session_interrupted` and `ambiguous_state`.
- `durable-prp-control-plane.test.ts`: the new checkpoint proof, its
negative cases, and the restart rule for the refresh.
- `native-run-trace.test.ts`: remote startup spans are recorded without
duplicate launcher spans.
- Focused suites pass: runner package 5 files and 235 tests; server 9
files and 1,607 tests, including `native-session-executor`,
`native-runner-execution-driver`, `remote-launch-assets`,
`native-restart-recovery`, `runner-prp-outbound` and the recovery
integration tests.
- `tsc --noEmit` passes for `packages/paperclip-runner` and `server`.

## Risks

- A launch now issues two bootstrap tickets. The first one is never sent
to `runnerd` and expires in 60 seconds. Bootstrap counters already allow
more than one issued credential.
- The new retry path needs a checkpoint proof that no runner
authenticated, so a run that did provider work cannot be retried this
way. The existing attempt budget still bounds the retries.
- This change does not remove the 6 to 11 minute controller-side delay
before `runner.transport.connect`. The new spans make it attributable
for a follow-up change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, browser, code search) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:02:30 -06:00
DottaandClaude Fable 5.2 fd9985b571 fix(native): say what a Runner command waited behind when it times out (#21056)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on remote sandboxes talk to a Runner through durable PRP
commands. The controller waits for each command's result
> - After a server replacement, the new controller reattached to live
Runners. For runs it reached later than about 115 s after detach,
`session.snapshot` got no answer within 30 s
> - The failure said only `provider_transport_failed: PRP command
session.snapshot timed out`. The run log could not tell whether the
Runner was disconnected, whether the command waited behind older
commands, or when the Runner last sent an event
> - This pull request adds that evidence to the timeout message. It
names types, counts and durations only, never payloads
> - The benefit is that the next upgrade's failures explain the stall,
so the runner-side cause can be fixed

## Linked Issues or Issue Description

Refs #21053, #21052, #21039

**What happened?**
At the 07:10Z staging swap, 9 remote native runs that the new controller
reached 114–152 s after detach failed. Each recovery read ended with
`PRP command session.snapshot timed out`. The transport had
authenticated, and in some runs the same Runner process was verified.
The Runner journal recorded no diagnostic other than "WebSocket peer
closed the connection". Runs reached within 39–116 s succeeded.

**What did you expect to happen?**
A command timeout should say what the command waited behind, so the
cause can be found from the run log.

**Steps to reproduce**
1. Start a remote native run and replace the server while a provider
turn is active.
2. Let the new controller reach the Runner more than about two minutes
after detach.
3. Read the run log. The failure says only "timed out".

## What Changed

- `packages/paperclip-runner/src/live/runnerd-codex-transport.ts`:
`waitForRunnerCommand` accepts an optional `describeStall`. When a
command times out, the message ends with a bounded description, for
example `(runner connected; pending ahead 1 [semantic_tool.result]
oldest 188s; last runner event seq 152 181s ago)`. The description
covers whether a Runner is connected or how long it has been
disconnected, the pending commands ahead of this one (types and count,
up to 6 types, and the age of the oldest), and the last committed Runner
event's sequence and age. A failing or empty description keeps the plain
message.
- The transport passes this description for every command it waits on.
The last event's age comes from the PRP payload's `emittedAt`.
- `server/src/services/native-runtime/native-session-executor.ts`: the
startup `session.open` timeout classifier accepts the optional
description, so a startup timeout stays `provider_session_open_timeout`
and keeps its bounded same-run retry. A regression case covers each
startup code with the suffix.
- Tests: the exact message for the staging shape, the plain message when
there is nothing to describe or the description throws, and a
disconnected Runner with no events.

## Verification

- `cd packages/paperclip-runner && npx vitest run
src/live/runnerd-codex-transport.test.ts -t "a Runner command that gets
no answer"` passes 3 of 3. These tests fail on `origin/master`, which
has no `describeRunnerCommandStall` export and no description in the
message.
- The full `runnerd-codex-transport.test.ts` file passes 307 of 308. The
one failure (`stops a lost runnerd's provider and escaped tool job from
its lineage journal`) is a process-timing test that passes alone and
does not touch this code.
- `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` passes.
- Message matchers: the only matcher anchored on the end of the timeout
text was the startup `session.open` classifier. It now accepts the
suffix. `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t
"classifies a startup session.open timeout"` passes. Without the
classifier change it fails, because the suffix turns the timeout into
`native_session_interrupted`.

## Risks

- Low. Only the text of a timeout message changes, by appending text. No
control flow, persisted state or Runner behavior changes.
- The description holds command types, counts and durations only. It
holds no payloads, identifiers or provider text.
- No wait is added before a turn start.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic) via Claude Code, with tool use (shell, file edits,
test runs, read-only staging API reads). Model ID as attributed in the
commit trailer: Claude Fable 5.2.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 03:00:32 -06:00
Nicky Leach d68088465c Reuse retained Runner descriptions during startup (#21004) 2026-11-11 21:20:08 -08:00
DottaandClaude Opus 5.5 85bb942aad fix(native): prove an unconnected bootstrap from the sandbox's own Runner state on a volume-free server (#20992)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs execute through a Runner (`paperclip-runnerd`) in a
remote sandbox. When a Runner exits before `session.open`, the run
restarts its bootstrap on the same run.
> - #20972 lets host preparation restart that bootstrap when the Runner
wrote `runner-state.json` but no provider state. It needs a proof that
this run's own interrupted Runner is stopped or gone.
> - That proof is recorded only when the server still has the run's
local Runner controller root. A volume-free server, or a root that held
no identity, skips it. The live chaos wave after #20972 still failed
with `runner_harness_state_mismatch: resumed_unavailable`.
> - This pull request lets host preparation establish the same proof
from durable sandbox evidence when no local proof exists.
> - The benefit is that the restart also works on volume-free servers,
which is how Cloud runs.

## Linked Issues or Issue Description

**What happened?**
Restart-chaos wave `w-20261112T002811-s6402` ran #20972 (Claude through
ACPX, Daytona warm, `--ephemeral-server`, K1/K2/K15 with
`--k15-fresh-home`, seed 6402, `--runner-bind-fault`). 5 of 6 tasks
needed Retry, with 6 `resumed_unavailable` failures. Run `fbb1666d` in
t01: the sandbox held `runner-state.json` for this run with `lifecycle:
connecting`, the same Runner instance and session,
`lastConnectionProtocolVersion: null`, and no provider state. That is
exactly the state #20972 restarts, but its proof was absent.

**Root cause**
`recordRemoteBootstrapRestartProof` is called only inside the
`existsSync(scoped)` block of the restart path in
`server/src/services/native-runtime/native-session-executor.ts`, after
the local controller identity is read. With no local root (volume-free,
a fresh home, or an emptied root), `provenOwnBootstrapRestart` stays
false, so `inspectOwnUnpreparedBootstrap` from #20972 returns nothing
and the resumed branch throws `resumed_unavailable`.

**Expected behavior**
A run with a `bootstrap_incomplete` recovery claim whose sandbox Runner
state proves it never connected restarts its bootstrap without Retry.

## What Changed

- `inspectOwnUnpreparedBootstrap` now runs for any
`bootstrap_incomplete` claim of this run. When no proof was recorded, it
records one only if the sandbox Runner state proves it never connected
(`remoteRunnerStateNeverConnected`: `lastConnectionProtocolVersion:
null`, `lastControllerCommandSeq: 0`, `reconnectCount: 0`, empty outbox
and processed commands, no pending terminal delivery). The exact Runner
process must also be stopped or proven absent by the existing
`stopStaleRemoteRunnerBeforeBootstrap`, which uses the Runner state's
own identity. All earlier conditions from #20972 still apply: same run,
Runner instance and session, a lifecycle other than `suspended`, no
failover backup and no archived rotation snapshot.

## Verification

- Red on master (#20972 merged): `pnpm --filter @paperclipai/server exec
vitest run src/services/native-runtime/native-session-executor.test.ts
-t "bootstraps only an untouched provider session"`. The new
`interrupted_bootstrap_unprepared_unrecorded` cells failed: resumed
lease `Error: runner_harness_state_mismatch: resumed_unavailable`,
created lease no restart.
- Green: those cells pass. `_connected` (a Runner state that connected,
no local proof) and `_unproven` (the stop is not proven) keep the
refusal. The full file passes (1409 of 1409). Server `tsc --noEmit`
passes.
- Chaos, same cell with `--runner-bind-fault`. Wave
`w-20261112T002811-s6402` (#20972 only): 1 of 6 passed, 6
`resumed_unavailable`. Wave `w-20261112T010800-s6402` (this change): 3
of 6 passed, 2 `resumed_unavailable`. In this wave, run `0a080e83`
restarted its bootstrap three times on the same sandbox: three
`.bootstrap-restart-*` roots were moved aside. The baseline wave on
master before #20972 (`w-20261111T230537-s6402`, no injected fault)
passed 2 of 6.

## Risks

- The new path runs only for a `bootstrap_incomplete` claim of this run
when the full harness check fails, so the normal path is unchanged and
adds no wait before `turn.start`.
- A Runner state that ever connected, received a command or holds outbox
data keeps today's refusal. The moved-aside root is kept for inspection.
- Not fixed here, and seen in both waves: every Runner on a sandbox
listens on one fixed port (43127 in all captures). A live Runner of
another run or session on the same sandbox therefore makes each relaunch
fail with `runner_ingress_bind_conflict`, until the bounded retry is
exhausted (`native_session_retry_exhausted`). A later attempt can still
end with `resumed_unavailable` or `resumed_not_suspended`. That needs
its own fix in predecessor retirement or port selection.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, test runs). Default context window,
extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #20972, #20928, #20953, #20863.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 20:07:02 -06:00
DottaandClaude Opus 5.5 66344b76ae fix(native): restart a bootstrap whose Runner exited before writing provider state on a resumed sandbox (#20972)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs execute through a Runner (`paperclip-runnerd`) inside a
remote sandbox. A warm Daytona sandbox is reused by later runs.
> - When a Runner exits before `session.open`, the run does a bounded
same-run bootstrap retry. Durable evidence proves that no provider
session and no provider event exist.
> - The retry fails when the Runner wrote `runner-state.json` but exited
before it wrote its provider state. Host preparation then calls the
sandbox state "unavailable" and refuses with
`runner_harness_state_mismatch: resumed_unavailable`. The task needs
Retry.
> - This pull request reads the Runner state on its own when the full
check fails, but only for a proven bootstrap restart of the same run. A
non-suspended Runner state of this run is then the run's own interrupted
bootstrap, and the existing restart path moves it aside and claims a new
session root.
> - The benefit is that a Runner that exits before `session.open` on a
reused sandbox recovers without a person. Saved state, a verified backup
and every other mismatch keep their refusal.

## Linked Issues or Issue Description

**What happened?**
Restart-chaos wave `w-20261111T230537-s6402` (Claude through ACPX,
Daytona warm sandboxes, `--ephemeral-server`, K1/K2/K15 with
`--k15-fresh-home`, seed 6402) needed Retry on 4 of 6 tasks, with the
same signature after each restart kind. The first attempt of each run
reused the warm sandbox. Its Runner failed with
`runner_ingress_bind_conflict: failed to bind listener: Address already
in use` and exited with `lifecycle=connecting`. Recovery logged
"Restarting an incomplete native runner bootstrap on the same heartbeat
run". The second attempt reused the environment and failed with
`runner_harness_state_mismatch: resumed_unavailable`. The earlier wave
`w-20261111T212429-s6402` had one run with the same error code.

**Root cause**
`inspectRemoteHarnessState` in
`server/src/services/native-runtime/native-session-executor.ts` checks
the persistence directories, `runner-state.json` and the provider state
file in one shell test. A Runner that exits at bind time wrote
`runner-state.json` but no provider state, so the one test fails and the
result is `incompleteReason: "unavailable"` before the Runner state is
read. The restart path (`restartableBootstrapState`) accepts only
`not_suspended` and `provider_unopened`, so it is skipped.
`claimUntouchedRemoteSession` then cannot create the session root that
the first attempt created, and the resumed branch throws
`resumed_unavailable`. On a created lease the same state fell through
without a restart.

**Expected behavior**
A proven, interrupted bootstrap of the same run restarts on the same
sandbox. The run finishes without Retry.

**Steps to reproduce**
Run the s6402 cell above, or add `--runner-bind-fault` (this pull
request) to force the Runner exit on every run's first launch.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`: when
the full harness check fails during host preparation,
`inspectOwnUnpreparedBootstrap` reads `runner-state.json` alone. It
returns `not_suspended` for this run only when all of these are true:
this run has a proven bootstrap restart (`bootstrap_incomplete` recovery
plus the stop proof), no failover backup exists, no archived rotation
snapshot is in use, and the Runner state names this run, this Runner
instance and this session with a lifecycle other than `suspended`.
Otherwise the result stays `unavailable`, as before. The existing
restart path then moves the session root aside (it is kept, not deleted)
and claims a new one.
- Chaos coverage: `--runner-bind-fault` in `tests/restart-chaos` gives
every server of the wave
`PAPERCLIP_TEST_RUNNER_BIND_FAULT=first_launch`. With `NODE_ENV=test`
only, the remote Runner launcher starts a short-lived listener on the
Runner's listen port before a Runner's first launch on that server, so
the Runner exits with a real `runner_ingress_bind_conflict`. Documented
in `tests/restart-chaos/README.md`.

## Verification

- Red on master: `pnpm --filter @paperclipai/server exec vitest run
src/services/native-runtime/native-session-executor.test.ts -t
"bootstraps only an untouched provider session"` failed 2 of the new
cells. Resumed lease: `Error: runner_harness_state_mismatch:
resumed_unavailable` (the staging error). Created lease: no restart
happened (`expected [] to have a length of 1 but got +0`).
- Green: the same cells pass. The negative cells keep
`resumed_unavailable`: no restart proof, a Runner state of another run,
a `suspended` Runner state with missing provider state, and a failover
backup present.
- `native-session-executor.test.ts`: 1406 of 1406 pass. A new test
proves the bind fault holds the listen port (a second bind gets
`EADDRINUSE`) and fires once per Runner instance.
- `tests/restart-chaos`: `runner-bind-fault.test.ts`,
`question-handoff.test.ts`, `shape-cells.test.ts` pass. Server and
harness `tsc --noEmit` pass.
- Chaos wave: see the comment below for the before/after counts.

## Risks

- The new read runs only for a proven bootstrap restart of the same run
with a failed full check, so the normal path has no extra command and no
wait before `turn.start`.
- A partly saved session (`suspended` Runner state without provider
state), a Runner state of another run and any failover backup keep
today's refusal. The moved-aside root is kept for inspection.
- A Runner that started under an older server and is restarted by this
version takes the same path, because it reads only files that older
Runners already write.
- The bind fault is test instrumentation. It is inert unless
`NODE_ENV=test` and the variable are both set.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, test runs). Default context window,
extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #20928 (restart a Runner that exited before session.open),
#20874, #20863, #20953.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 18:36:33 -06:00
DottaandClaude Opus 5.5 c45f54ee74 fix(native): stop or release unclosable idle warm Runners on environment delete, and say why a close failed (#20957)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep a warm Runner in a Daytona sandbox between turns of
a task, so the next turn starts fast
> - Each idle warm Runner has an in-memory owner in the controller.
Deleting an environment closes these owners before it destroys the
sandboxes
> - In every warm chaos wave today, some owners never closed. The server
logged only `{"failed":2}`. Environment delete returned 409 forever, and
all sandboxes of the wave stayed `started` until their 60-minute TTL
> - On staging, that is leaked sandboxes (cost) and owners that block
the work after them
> - This pull request logs and reports the exact close error, stops an
unsettled idle Runner when its environment is deleted, and releases an
owner whose sandbox is already destroyed
> - The benefit is that environment delete finishes or names the exact
run and error, and every warm chaos wave now grades a clean shutdown

## Linked Issues or Issue Description

**What happened?**
After warm Daytona chaos waves (W2, W2Q, W2C cells, `--faults K0`) on
Codex, Claude and OpenCode, the server logged `idle native session
cleanup failed before controller shutdown {"idleSessions":{"failed":2}}`
with no reason. `DELETE
/api/environments/:id?destroyReusableSandboxLeases=true` returned 409
"warm native runner did not shut down cleanly" on every retry. Eight
sandboxes per wave stayed `started`.

**Root cause.**
`closeIdleWarmNativeSessions`
(`server/src/services/native-runtime/native-session-executor.ts`)
dropped the close error (`catch {}`) and counted it. The owners that
failed are the W2Q owners. A W2Q task parks a question and is answered
by a later run on a fresh sandbox. With the reason logged, a wave shows
two causes for the same owner:

1. The owner's close cannot settle (`native_session_close_unrecoverable:
runner did not durably suspend before checkpoint`, the governed-wait
shape from #20844). Only a successor reset could force a stop
(`discardUnsettledSession`). An idle sweep or environment delete had no
forced stop.
2. The owner's sandbox lease was already destroyed by the
pending-cleanup sweep after the successor moved to a fresh sandbox
(#20863). Every close then fails with `Daytona sandbox lease <id> is no
longer active.`, because the stop RPC has no sandbox to reach. The owner
stays registered for ever.

The Runner close memoizes its rejection, so every later attempt fails at
once with the same error. This is not the "controller sequence must be
contiguous" exit (#20946): no close failure in these waves names a
sequence mismatch.

**Expected behavior**
An idle warm owner with no turn in flight never blocks environment
delete for ever. Every failed close names its run and its error.

**Steps to reproduce**
1. Run `tests/restart-chaos/main.ts --env daytona --daytona-warm
--adapters claude --workloads W2,W2Q,W2C --faults K0 --concurrency 8
--tasks 8`.
2. Read `<wave>/sandbox-sweep.json`: `environmentDeleteBlocked` is set
and the sandboxes are `started`.

## What Changed

- `closeIdleWarmNativeSessions` now returns `failures: [{ runId,
issueId, environmentId, providerLeaseId, reason }]`. The reason is the
close error with its code, its settlement record and its cause. It also
logs `idle warm native sessions did not close` with that list. The
idle-timeout close logs its error too.
- `closeWarmNativeSessionsForEnvironment` takes `forceStopUnsettled` and
`db`. The environment delete route sets both. When the graceful close
fails, Core retires the owner as #20844 does: one confirmed Launcher
`stop`, the credential snapshot, `complete`, then instruction
collection. A failed forced stop is added to the reason.
- When retirement still fails and the owner's sandbox lease has a
`destroyed` termination receipt, Core releases the owner. It does not
contact the sandbox. It logs `warm native session's sandbox was already
destroyed; released its owner` with the close error.
- The environment delete 409 now names each run and its reasons, and
`details.nativeRunnerSessions.failures` lists them.
- Idle timeout, controller restart and tenant sleep do not force a stop.
Their conversation can still be resumed, so they only log the reason.
- Chaos harness: new check `warm-shutdown.ts`, "no leaked sandboxes /
clean warm shutdown". Every warm wave (`--daytona-warm` or `--lifecycle
warm`) runs it. It fails the wave on a blocked delete, on a sandbox left
running after cleanup, or on a warm session that did not close at
environment delete or controller shutdown. It writes
`<wave>/warm-shutdown.json` and a section in `SUMMARY.md`.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` and the chaos README.

## Verification

- RED on master: `native-session-executor.test.ts -t "stops an idle warm
owner whose close cannot settle"` fails: `expected { closed: 0, busy: 0,
failed: 1 } to match object { closed: 0, failed: 1, failures: [...] }`
(a bare count, no reason). `environment-routes.test.ts -t "names the
warm runner session"` fails: the 409 does not name the run. On the first
fix commit, `-t "sandbox lease is gone"` fails: the lease state is never
read, and the owner stays.
- GREEN: those tests pass. The executor suite, environment routes, run
control, recovery, heartbeat process recovery, instance settings, the
idle-sleep, eviction, release-resume, restart-recovery and
warm-checkpoint integration suites pass (11 files, 2471 tests).
`tests/restart-chaos` unit test `warm-shutdown` passes. Server and
harness `tsc --noEmit` pass.
- Daytona warm waves (Claude, W2/W2Q/W2C, K0, 8 tasks). The runner image
`sha256:e04d9ce4…` matches 7e8901aa60, not current master, so each wave
ran this change cherry-picked onto 7e8901aa60:
- `w-20261111T210049-s6111` (first commit only): the close reason is now
logged: `close failed: Daytona sandbox lease 192606aa… is no longer
active.` The delete stayed 409 with that reason. The new check failed
the wave correctly.
- `w-20261111T222826-s6112` (all commits): two W2Q owners were released
with `sandbox was already destroyed`. The environment was deleted. 0
sandboxes left running. The warm-shutdown check passed. No 409 from a
warm session.
- Not verified: a wave where the graceful close fails and the forced
Launcher stop succeeds on a live sandbox (that path is covered by the
unit test). The wave's task failures (`T:first_run_succeeded`,
`R3:retry_needed`) are other open defects. They are also on master.

## Risks

- Environment delete now stops an idle Runner whose close did not
settle. The delete already has the operator's consent to destroy those
sandboxes. The workspace-export and restore-source guards on sandbox
destruction are unchanged. Instruction files are collected after the
confirmed stop, as in #20844.
- An owner is released without contact only when a lease for the same
company, environment and sandbox has a valid `destroyed` termination
receipt. A `stopped` receipt is not enough, because a stopped sandbox
can be resumed.
- `closeIdleWarmNativeSessions` results have a new `failures` field.
Callers that read only the counts are unchanged.
- No persisted state changes. Warm owners live only in controller
memory, so older saved state and mid-run deploys are not affected.
- The new harness check can fail a warm restart-fault wave that leaves a
warm session unclosed at controller shutdown. That is a real leaked
Runner.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with
tool use (shell, file edits, test runs, chaos waves) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 17:12:21 -06:00
DottaandClaude Opus 5.5 d9d5958d09 fix(runner): reject an out-of-order controller command instead of exiting (#20946)
## Thinking Path

- Paperclip's controller and the sandbox Runner exchange numbered
commands; the Runner refuses a command whose number is not exactly one
past the last it processed.
- Staging's most common "Runner gone" failure was a follow-up controller
whose journal was behind the Runner. #20928 removed the cause on the
server (a warm follow-up now launches from the journal its retired
predecessor saved) and added a defence that reconciles a journal behind
only by the predecessor's settled close.
- The Runner still treated an out-of-order command as fatal:
`durable/state.rs` returned an error and the process exited. A
controller with a stale journal from any other cause (state saved by an
older release, a mid-run deploy) therefore lost the Runner process and
the run could only fail as "Runner gone"
(`runner_remote_process_identity_unavailable`).
- This pull request makes the Runner refuse such a command with a
precise result instead of exiting, and makes the server rest that run
with a precise code and a Retry whose next run reconciles the journal
(the #20928 defence) before it opens the session.

## Linked Issues or Issue Description

- Refs #20928 (the server-side root-cause fix and the journal
reconciliation this relies on)
- Refs #20907 (makes runnerd exit text visible in the run log)
- Duplicate search: searched PRs for "controller sequence contiguous",
"controller_sequence_mismatch" and "out-of-order command"; none changes
the Runner's handling.

## What Changed

- `runner-core/src/durable/state.rs`: a new command whose
`controllerSeq` is not `lastControllerCommandSeq + 1` (behind or a gap)
now yields `CommandDisposition::Reject` with `status: "rejected"` and `{
code: "controller_sequence_mismatch", message, expectedControllerSeq,
receivedControllerSeq }`. The command is not executed or journaled and
the last sequence is unchanged; the process keeps running. Replay of an
already-processed command and the compacted-history rejection are
unchanged.
- `native-session-executor.ts`: a failure carrying
`controller_sequence_mismatch` is classified as
`native_controller_journal_behind_runner`, a permanent (terminal)
failure with a Retry action that says no provider work began and that
Retry reconciles the journal from the saved checkpoint or stops with a
precise error.
- Tests:
- Rust `tests/durable_recovery.rs`: out-of-order (behind and gap)
commands are rejected with the precise code, the journal is unchanged,
and the next contiguous command executes. The existing
`command_gaps_and_identifier_reuse_fail_closed` unit test now asserts
the rejection instead of an error.
- Real runnerd `src/live/runnerd-stale-controller-journal.test.ts`: a
follow-up controller with a journal missing the predecessor's close now
gets `"code":"controller_sequence_mismatch"`, the Runner process stays
alive and its durable sequence is unchanged; with the current journal it
resumes.
  - Server: the classification and terminal disposition of the new code.

## Verification

- Red (master Runner): the Rust test panics on `expect("an out-of-order
command is not fatal")`; the real-runnerd test receives `runnerd exited
unexpectedly with code 1: paperclip-runnerd: controller sequence must be
contiguous: expected 6, received 4`.
- Green: `cargo test -p paperclip-runner-core --test durable_recovery`
(4 passed); `cargo test -p paperclip-runner-core` lib suite (the only
other failure, `task_memory::tests::monitor_bounds_notices`, is a timing
test that passes alone); `npx vitest run
src/live/runnerd-stale-controller-journal.test.ts` with a runnerd built
from this branch (2 passed); server `native-session-executor.test.ts`
focused tests (71 passed).
- `pnpm --filter @paperclipai/paperclip-runner check:acpx-profiles`:
profiles match the release manifest. No ACPX profile, attestation, pin
or pack contract schema changed (`rules/acpx-profile-identity.mdc`): the
change is Runner source behaviour with an existing command-result shape,
so no profile version or `commandDigest` changes.

## Risks

- A controller that is *ahead* of the Runner (a gap) is now rejected
instead of fatal. Neither case executes the command; the Runner journal
is unchanged, so this is strictly safer than exiting.
- The rested run needs a Retry. Before this change the same run failed
after its retries as "Runner gone" with no precise cause.
- Runner source changes the Daytona image content ID; a new image is
needed for chaos waves on this revision.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), 1M-token context, tool
use (shell, file edits, tests) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 16:14:31 -06:00
Devin FoleyandPaperclip 6a43890326 Fix image attachments in remote native workspaces (#20939)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Native runs must receive the files that users attach to their
messages.
> - Remote wake preparation returned file metadata without readable
bytes.
> - The historical chat reader also rejected every remote target.
> - Command-backed sandboxes and SSH targets already provide a bound
command transport.
> - This pull request stages verified files through that transport and
clears them after the turn.
> - Agents can then open attached images with their normal image reader.

## Linked Issues or Issue Description

Refs #19740. This change covers SSH and command-backed sandbox
execution, including Daytona. Plugin Runner tasks without a command
transport still need a separate versioned file-transfer contract. This
PR does not close that broader issue. The open attachment-recovery PR
#20778 concerns provider-session attachment journals, not input files.

**What happened?**

A user uploaded an image and sent it to an agent. The remote native run
received its filename and size, but no readable file. Wake preparation
reported `remote_workspace_staging_unsupported`. The historical read
fallback also refused remote execution.

**Expected behavior**

Copy the exact admitted attachment into the bound remote workspace. Let
the provider read the image. Preserve company, task, source-comment,
size, and hash checks. Clear temporary file contents when the turn ends.

**Steps to reproduce**

1. Select a native agent on a command-backed sandbox or Linux SSH
environment.
2. Attach a PNG to its current task or in-app chat message.
3. Ask the agent to inspect it.
4. Check that wake preparation supplies a readable workspace-relative
image path.

## What Changed

- Add a remote staging owner with bounded commands, stdin transfer, hash
verification, opaque paths, and safe image extensions.
- Confine Linux filesystem operations through directory descriptors.
Refuse symlinks and replacement files during cleanup.
- Bind current wake files and authorized historical external-chat reads
to the selected lease transport.
- Join active writes during cancellation. Retain failed cleanup work
until scope close.
- Tell providers to open image files with an image-capable reader.
- Handle stdin stream errors in the shared SSH transport. Failed uploads
return failure instead of raising an unhandled EPIPE or reporting
success.
- Add database, transport, cancellation, integrity, cleanup,
executor-wiring, and Linux filesystem regressions. Document support and
limits.

## Verification

- Prepared an isolated source checkout with `node
scripts/agent-preflight.mjs --skip-rust-debug`.
- Focused suites: `pnpm exec vitest run
server/src/services/native-runtime/remote-attachment-staging.test.ts
server/src/services/native-runtime/native-runner-file-handoff.test.ts
server/src/services/native-runtime/chat-attachment-read.test.ts` — 74
passed, 4 Linux-only tests skipped.
- Executor checks: `pnpm exec vitest run
server/src/services/native-runtime/native-session-executor.test.ts -t
'attachment staging to the remote|stages from the authenticated|verifies
frozen staged admission'` — 7 passed.
- Server typecheck: `pnpm --filter @paperclipai/server typecheck`,
followed by the server layered check after the final changes.
- `git diff --check` passed. The local secret, PII, and
private-reference diff scan found no matches.
- `pnpm verify` returned exit 2 because the documentation change
requires broad CI. No full local suite is claimed.
- The four Linux filesystem tests were skipped locally because this
macOS host has no available Docker daemon. Linux CI executed all 17
remote-staging tests successfully, including those four confinement
tests: [server shard
10](https://github.com/paperclipai/paperclip/actions/runs/43102035096/job/130796646266).
- SSH regression: the real pipe test reproduced two unhandled EPIPE
errors before the fix, plus false success when the peer exited with code
zero. The fixed SSH tests and existing argv tests pass (12 tests). The
remote-staging transport suite also passes (13 tests; 4 Linux tests
require CI).
- `pnpm --filter @paperclipai/adapter-utils typecheck` passed after the
SSH change.
- Final revision `1c5a9288b8d5f267ddeac84638ff6075dbbbf8ec`: all
applicable CI jobs passed, including typecheck, build, general and
serialized tests, Runner checks, canary dry run, and all eight browser
shards. [CI
run](https://github.com/paperclipai/paperclip/actions/runs/43102035096).
Linux CI also passed all 6 native SSH tests: [server shard
5](https://github.com/paperclipai/paperclip/actions/runs/43102035096/job/130796646349).
- Greptile reviewed that final revision at 5/5 with no remaining
findings. The review thread is resolved. GitHub reports the PR mergeable
with a clean merge state.
- No live model turn or deployed release is claimed.

## Risks

- Remote staging requires Linux, Node, and a command transport bound to
the selected lease. Plugin Runner tasks without that transport retain
their unavailable result.
- Each file uses bounded independent commands. Slow transports can delay
preparation or cleanup. File bytes stay out of command arguments and
provider-facing error messages.
- An abrupt controller or remote-host failure can bypass cleanup.
Ephemeral sandbox disposal remains the backstop. Retained files are
never blindly overwritten or deleted.
- Previous-release provider sessions and checkpoints retain their format
and identity. New turns receive newly staged input files. Existing
restart-admission checks still apply. No migration, runner-image change,
or account-credential transfer is required.

## Model Used

OpenAI `gpt-6.1-sol` through the Codex harness in T3 Code, with xhigh
reasoning, repository inspection, code execution, and tests. No
subagents were used. The context-window size is not exposed by this
runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-11 14:12:09 -08:00
DottaandClaude Opus 5.5 757a20d2f1 fix(runner): keep remote recovery and launch through a short plugin RPC stall (#20945)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote run starts `paperclip-runnerd` in a sandbox. The controller
reads and controls it through the environment plugin
(`environmentExecute`).
> - #20940 made a stalled supervision probe "unknown" while the Runner
stays connected. Other callers still turn the same plugin stall into a
terminal failure.
> - During restart reattachment, the saved-state read, the
identity-marker read and the process check share a 30-second budget. One
plugin RPC that stalls for 40 seconds uses all of it, so a short stall
fails a reattachment to a healthy Runner.
> - At launch, a stalled launch call or identity read fails the run with
`runner_process_termination_unconfirmed` or
`runner_remote_process_identity_unavailable`, although the runner may be
healthy.
> - This pull request treats a plugin RPC timeout as "unknown" and
retries the read-only call within a bounded stall budget. A stalled
launch call is not repeated: the controller reads the launch nonce's
identity marker instead.
> - The benefit is that a short plugin stall during recovery or launch
no longer fails a healthy run. Every positive answer from the sandbox
keeps its meaning, and the bound still fails closed.

## Linked Issues or Issue Description

Refs #20940, #20909

**What happened?**
A plugin `environmentExecute` RPC stall of more than 40 seconds makes a
remote recovery read fail. Plugin RPCs time out at the requested timeout
plus a 30-second overhead, so a 10-second probe can take 40 seconds. The
reattachment read budget is 30 seconds. One stalled read therefore
failed reattachment with `probe_transport_failed` or `probe_timeout`. At
launch, a stalled launch call rejected the run with exit unconfirmed. A
stalled identity read (20-second wait, 2-second reads that can take 32
seconds) failed the run with
`runner_remote_process_identity_unavailable`.

**Expected behavior**
A plugin RPC timeout is a stall of the plugin, not an answer from the
sandbox. Read-only recovery reads retry within a bound. A launch call
that stalls is not repeated. The launch nonce's marker decides whether
the one runner was launched. At the bound, the run fails closed and
Retry works.

**Steps to reproduce**
1. Start a remote (Daytona) run.
2. Restart the controller so that the run reattaches, or start a new
run.
3. While the controller reads the sandbox, stop the plugin worker for 45
seconds.
4. The reattachment or launch fails.

**Paperclip version or commit**
master at 48d23d1530.

## What Changed

- `remote-runner-recovery-probe.ts`: adds `isPluginRpcTimeout` (an
error, or its cause chain, with `PLUGIN_RPC_ERROR_CODES.TIMEOUT`). It
also adds a per-recovery `RemoteRunnerPluginStall` with a 3-minute
budget (`REMOTE_RUNNER_PLUGIN_STALL_BUDGET_MS`). The caller's recovery
deadline caps it. After the first plugin RPC timeout in a recovery, its
read-only probes use the stall deadline instead of the 30-second
deadline. Signals are never retried. Other transport failures keep the
30-second budget.
- `verifyRemoteRunnerReattachment` and `verifyRemoteRunnerOrSettledExit`
share one stall budget across the saved-state read, the identity-marker
read, the process check and the post-exit settled-state read
(`readRemoteRunnerState`).
- `waitForRemoteRunnerProcessIdentity`: a plugin RPC timeout extends the
20-second identity wait to the stall budget.
- Launch: a launch call that fails with a plugin RPC timeout is not
repeated. The controller goes on to the identity-marker wait for the
same nonce. A found marker adopts the one launched runner. A missing
marker uses the existing nonce cleanup, and the result stays unconfirmed
when the cleanup fails.
- `doc/execution-semantics.md` describes these rules.

## Verification

- Red on master: 7 new tests in
`server/src/services/native-runtime/native-session-executor.test.ts`
failed. Each one stalls a plugin RPC for 40 seconds with a
`JsonRpcCallError` `TIMEOUT`:
- reattachment through a stall in the saved-state, identity-marker and
process-check reads (the reattachment failed with the stall error);
  - the bound test (it settled before the stall budget);
  - the post-exit settled-state read (it failed with the stall error);
- launch-call and identity-read stalls (they failed with the stall error
or `runner_remote_process_identity_unavailable`).
- Green: all 7 pass, and a control test proves that a non-stall
transport error keeps the 30-second bound.
`native-session-executor.test.ts` and
`remote-runner-recovery-probe.test.ts` pass (1400 tests). `tsc --noEmit`
passes for `server`.
- The tests also assert that one launch call happens, no cleanup or
signal runs on the success path, and the bound fails closed without exit
proof.

## Risks

- During a real plugin stall, recovery and launch can now wait up to 3
minutes instead of about 30 seconds before they fail closed. The longer
wait applies only after a plugin RPC timeout. A sandbox that answers, or
a non-timeout transport error, keeps the old timing.
- No wait is added before `turn.start` on the normal path.
- State saved by an older version is not changed.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, code search). Extended reasoning was
on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 16:04:13 -06:00
DottaandClaude Opus 5.5 48d23d1530 fix(runner): keep a connected remote Runner through a stalled plugin supervision probe (#20940)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote run starts `paperclip-runnerd` in a sandbox. The controller
monitors the process with a health probe that goes through the
environment plugin (`environmentExecute`).
> - A stall of the Daytona plugin worker makes that probe RPC time out
after 40 seconds. The monitor treated the timeout as "process
supervision failed".
> - The transport then failed closed with
`runner_process_termination_unconfirmed`. The run ended
`notification_transport_failed`, `recoverable: false`, and the close
path suspended a healthy Runner. The task rested blocked with Retry.
> - Nothing was wrong with the sandbox, the Runner or the provider. The
Runner held its own authenticated connection to the controller during
the full stall.
> - This pull request makes a failed or timed-out probe "unknown". While
the Runner stays connected on its own channel, the monitor keeps the
same runner and turn and probes again, for a bounded time.
> - The benefit is that a short plugin stall no longer kills a healthy
run. The fail-closed cases stay the same.

## Linked Issues or Issue Description

Refs #20909 (chaos cell CR3, which reproduces this failure)

**What happened?**
A 45-second stall of the Daytona plugin worker killed a healthy run.
Chaos cell CR3 reproduced it 2 of 2 times (OpenCode and Claude,
Daytona). A staging run had the same failure with no deploy: a plugin
`environmentExecute` RPC stalled for more than 40 seconds. The run
failed with "Notification transport failed closed:
runner_process_termination_unconfirmed: runnerd process supervision
failed: RPC call \"environmentExecute\" timed out after 40000ms". The
close path then sent `runner.suspend`, and the healthy Runner suspended
and exited. The task rested blocked with Retry.

**Expected behavior**
A probe that times out or fails in transport is not proof that the
process stopped. While the Runner stays connected on its own channel,
the run continues. If supervision cannot come back within a bound, the
run rests with a working Retry. A probe that positively reports the
process gone still starts the existing recovery.

**Steps to reproduce**
1. Start a remote (Daytona) run with a T1 workload.
2. Send SIGSTOP to the server's Daytona plugin worker for 45 seconds,
then SIGCONT.
3. The run fails `notification_transport_failed` about 40 seconds into
the stall.

**Paperclip version or commit**
master at 7cd5558e41.

## What Changed

- `createRemoteRunnerProcessLauncher`
(`server/src/services/native-runtime/native-session-executor.ts`): when
the health probe throws (RPC timeout or transport error) and the Runner
holds an authenticated connection, the monitor waits one poll interval
and probes again. It does not reject the process completion. The unknown
period has a bound: the remote runner reconnect grace (7 minutes). When
the Runner is not connected, the monitor keeps the old behavior: it
rejects after the 30-second probe budget, without exit proof.
- Probe results that positively answer (exit 45 "process gone", or
identity and generation failures) keep their exact current handling.
- `RunnerProcessLaunchSpec`
(`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`)
has a new optional `isRunnerConnected` callback. `spawnRunner` passes it
to the process owner and keeps it on restart.
- The runnerd transport and the external runnerd driver supply
`isRunnerConnected` from the control plane's count of authenticated
Runner connections.
- `doc/execution-semantics.md` describes the unknown-probe rule and its
bound.

## Verification

- Red, on master: `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "plugin
RPC timeouts until"` (in `server/`). The completion rejected after the
first stalled probe with `{ stage: "process_verification", reason:
"probe_transport_failed" }`. The transport turns that into
`runner_process_termination_unconfirmed`, which is the staging error.
- Red, on master: `npx vitest run
src/live/runnerd-codex-transport.test.ts -t "live evidence of the
runner"` (in `packages/paperclip-runner/`). The launch spec had no
`isRunnerConnected` (`expected undefined to be true`).
- Green: both tests pass. New tests also prove:
- three 40-second plugin RPC timeouts with the Runner connected leave
the same runner supervised, and a later exit-45 probe still confirms
exit;
- with the Runner disconnected, the same timeout rejects without exit
proof (`probe_transport_failed`, `isExitConfirmed() === false`);
- with the Runner connected and probes that never answer, supervision
rejects without exit proof after the 7-minute bound.
- `server/src/services/native-runtime/native-session-executor.test.ts`,
`remote-runner-recovery-probe.test.ts`,
`packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts`,
`src/control-plane/durable-prp-control-plane.test.ts` and
`src/drivers/external` pass locally.
- `tsc --noEmit` passes for `server` and `packages/paperclip-runner`.
- Chaos: CR3 results are in a follow-up comment on this PR.

## Risks

- While the Runner stays connected and the plugin cannot answer, the
controller waits up to 7 minutes before it fails closed. Before this
change it failed after about 40 seconds. During that time the Runner's
own connection is the liveness evidence, and the controller never
reports an exit.
- When the Runner is not connected, the timing is the same as before.
Sandbox loss still fails closed after the 30-second probe budget.
- The new spec field is optional. A process owner that does not read it
keeps the old behavior. State saved by an older version is not changed.
- No wait is added before `turn.start`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, code search). Extended reasoning was
on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 15:45:25 -06:00
DottaandClaude Opus 5.5 884a9e19ea fix(native): launch a warm follow-up from its predecessor's saved journal, and restart a Runner that exited before session.open (#20928)
## Thinking Path

- Paperclip runs agent work through a controller (server) and a runner
(`paperclip-runnerd`) that exchange numbered commands; the runner
refuses any command whose number is not exactly one past the last it
processed.
- Staging's most common failure is "Runner gone"
(`runner_remote_process_identity_unavailable`,
`native.remote_runner_recovery_failed` with `process_not_running`,
`runnerLifecycle: ready`, `runnerSettled: false`). Chaos wave
`w-20261111T192658-s5201` (Daytona, warm, Codex, W2/W2Q/W2C re-wake
cells) captured it live: the new runner wrote `controller sequence must
be contiguous: expected 13, received 11` (and `expected 5, received 3`)
and exited 0.15 s after authenticating.
- In all three captured failures the follow-up run started 1–2 s before
the predecessor's warm session was closed (`transport_close_requested`
for the predecessor arrives after the follow-up's `run started`). The
close is issued by the follow-up's own admission.
- Admission first materializes the controller projection for the session
root. With an idle retained warm owner the root only holds the
external-persistence marker, so `restoreWarmRunnerdController` copies
that owner's saved checkpoint into the root. Later in the same
admission, `executePaperclipNativeSessionWithinScope` finds the owner
cannot be reused (configuration, process or GitHub access changed) and
retires it. Retirement sends `runner.drain` and `runner.suspend`; both
are saved to the owner's checkpoint before they are sent, but the root
still holds the projection taken before the close.
- The follow-up then opens its controller from that projection. Its
journal is two commands behind the runner's durable
`lastControllerCommandSeq`, so its first command repeats sequence 11;
the runner treats this as fatal and exits, and every later attempt only
reattaches to a process that is gone.
- So the saved controller journal was persisted correctly; it was not
selected: the projection is never refreshed after the retirement that
extends it.

## Linked Issues or Issue Description

- Refs #20874 (the runner-exit proof this extends from a rejected
`session.open` to a Runner that exited before answering it)
- Refs #20907 (makes the runnerd exit text, including `controller
sequence must be contiguous`, visible in the run log; independent of
this fix)
- Refs #20853 (related: resumes a remote Runner that died at a settled
`ready` cut; this PR covers the unsettled `ready` state left by an exit
before `session.open`)
- Duplicate search: searched open and closed PRs for "controller
sequence contiguous", "runner_remote_process_identity_unavailable" and
"Runner gone"; none fixes the stale controller projection.

## What Changed

- `server/src/services/native-runtime/warm-controller-journal.ts` (new):
- `refreshRetiredWarmOwnerProjection` reloads the retired owner's
canonical checkpoint into the session root after retirement, only when
the root still projects that owner and the saved journal extends the
projected one. A diverged journal holds admission with
`native_runner_checkpoint_projection_held:retired_owner_journal_diverged`;
a projection changed underneath holds with `...:journal_changed`.
- `reconcileControllerJournalWithRunner` is the defence for state
already saved in the bad shape (old-saved/new-resumed, mid-run deploys).
When the remote runner state shows its journal ahead of the controller
projection and the runner's last command is a completed
`runner.suspend`, it reconciles only if every extra sequence is a
completed `command_close_drain_*`/`command_close_suspend_*` in the
runner's processed-command journal and the predecessor's canonical
checkpoint holds exactly those commands. Anything else refuses before
launch with
`native_controller_recovery_missing_evidence:controller_journal_behind_runner`
instead of guessing. A runner whose last command is not a completed
suspend is left to the existing live-adoption checks.
- Defence in depth for the server side of the same failure
(`native_session_recovery_failed: runnerd exited while waiting for
session.open`, then `runner_remote_process_identity_unavailable` on
every same-run retry): a Runner proven to have exited while this
attempt's opening command (`session.open`, or the recovery probe) was
still unanswered is a failed fresh start, not a live Runner to reattach.
- `runnerd-codex-transport.ts`: `runnerExitConfirmed()` (#20874's proof)
now also holds for a resumed session whose provider identity was
preloaded, when the attempt's opening command is still pending at the
proven exit.
- The Codex driver's recovery result, `HarnessDriverBackend` and
`native-session-runtime` carry that proof on the
`native_session_recovery_failed` error.
- The executor records `runnerExitedBeforeSessionOpen` on the settled
failure; restart recovery puts `exitedBeforeSessionOpen` on the
`reattach_remote_runner` claim; `verifyRemoteRunnerOrSettledExit` and
`verifyStagedRemoteRunner` then start a new Runner when the process is
gone and the saved Runner state has no pending terminal delivery,
provider cleanup or warm transition (any lifecycle other than an active
turn). Attempt budget and identity checks are unchanged.
- `native-session-executor.ts`: calls the refresh right after the
configuration-changed retirement of a warm owner, and the reconciliation
in `restoreRemoteSuspendedRunnerCheckpoint` with the runner state it
already reads (no added remote read, no wait before `turn.start`).
- Tests:
- `native-session-executor.test.ts`: new `changed-external-checkpoint`
scenario of "rotates an owned warm codex remote runner before checking
changed configuration" with a real `DurablePrpControlPlane`: the retired
owner's close saves drain/suspend, and the journal the follow-up
launches from must end with them.
- `warm-controller-journal.test.ts`: refresh, divergence hold, untouched
marker/other run, reconcile of a settled close, refusal when the
canonical journal cannot prove the close, refusal for any other extra
command, and no action for a live runner.
-
`packages/paperclip-runner/src/live/runnerd-stale-controller-journal.test.ts`:
real `paperclip-runnerd` + fake Codex. A follow-up controller on the
predecessor's suspended runner with a journal that lacks the close
commands reproduces `controller sequence must be contiguous` and loses
the runner; with the current journal it resumes.

## Verification

- Red for the added part: `verifyRemoteRunnerOrSettledExit` with the
staging state (process gone, lifecycle `ready`, unsettled outbox)
rejects with `runner_remote_process_identity_unavailable` without the
change; with `exitedBeforeSessionOpen` it resolves to a fresh start, and
stays fail-closed for a pending cleanup, terminal delivery, active turn
or exhausted budget. Without the propagation, the backend recovery
result lacks `runnerExitConfirmed`; the real-runnerd stale-journal test
asserts the transport's exit proof after the contiguity exit (`expected
false to be true` on the old transport).

- Red on master (executor unchanged, new scenario): `AssertionError:
expected [] to deeply equal [ 'runner.drain', 'runner.suspend' ]` — the
follow-up launched from the projection taken before the predecessor's
close.
- Green: `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "rotates
an owned warm codex remote runner before checking changed
configuration"` (14 passed), `npx vitest run
src/services/native-runtime/warm-controller-journal.test.ts` (7 passed).
- Real runner: `cd packages/paperclip-runner && npx vitest run
src/live/runnerd-stale-controller-journal.test.ts` (stale journal →
`controller sequence must be contiguous`; current journal → resumes).
- `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter
@paperclipai/paperclip-runner typecheck:typescript`; `pnpm verify`
selected broader verification (a new runner test file is outside the
resolved graphs), so broad coverage comes from required CI.

## Risks

- The refresh replaces the root projection only with a checkpoint whose
identity matches and whose journal extends the projection; otherwise it
holds admission with an existing
`native_runner_checkpoint_projection_held:*` code, which is safer than
launching a runner that will exit.
- The defence refuses launch when a runner is ahead by anything other
than a provable settled close. Before this change that launch always
ended in a runner exit and a failed run, so the refusal only changes the
error code and the moment of failure.
- This PR does not change the runner. The runner still exits on an
out-of-order new command; a follow-up runner PR will turn that into a
rejected command result.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), 1M-token context, tool
use (shell, file edits, tests) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 15:41:03 -06:00
DottaandClaude 80b2369f26 fix(native): keep remote runnerd exit evidence and report it on failed recovery (#20907)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on Daytona start a detached `paperclip-runnerd` in the
sandbox, and the controller adopts it after a restart
> - On staging, recovery fails with
`runner_remote_process_identity_unavailable` when the Runner is gone,
but nobody can see why it exited
> - The launch script sends runnerd's stdout and stderr to `/dev/null`,
so only a panic or a graceful error reaches `runnerd.stderr.log`, and
the recovery event does not read that file
> - This pull request keeps runnerd's stderr in a bounded file in its
private diagnostics directory, and adds the exit reason to the
`native.remote_runner_recovery_failed` event
> - The benefit is that the reason a Runner exited shows in the run log
on staging without SSH, which the class R investigation needs

## Linked Issues or Issue Description

**What happened?**
Staging runs fail with `runner_remote_process_identity_unavailable`
after the remote Runner exits (`native.remote_runner_recovery_failed`
with `stage: process_verification`, `reason: process_not_running`). The
event does not say why the Runner exited.

**What did you expect?**
The run log names the Runner's exit reason: its durable
`recoverableFailure` and diagnostics, and what it wrote to stderr.

**Steps to reproduce**
Restart-chaos re-wake and stall cells reproduce the refusal. Each failed
run log shows only `runnerLifecycle` and `runnerSettled`.

Refs #20893

## What Changed

- `REMOTE_RUNNER_CHILD_LAUNCH_SCRIPT` sends runnerd's stderr to
`diagnostics/runnerd.process-stderr.log` (mode 0600, in the 0700
directory) instead of `/dev/null`.
  - At launch, a file above 1 MiB is rotated once to `.1`.
- While runnerd runs, a detached trimmer is a disk backstop. Past 8 MiB,
it keeps the last 1 MiB. It checks every 15 s and ends when runnerd
ends. It is double-forked with its own argv, so it is not a child of
runnerd and does not carry runnerd's arguments. runnerd's own exit
reason goes to `runnerd.stderr.log`, which is never trimmed.
- A symlink or non-file at that path keeps the old `/dev/null` behavior.
- Stdout is unchanged (`/dev/null`). Runnerd's PID is still the launch
shell's PID, so the identity marker and fingerprint are unchanged.
- `remote-runner-recovery-diagnostics.ts` has a read-only
`REMOTE_RUNNER_EXIT_EVIDENCE_SCRIPT`. It prints a 2 KiB tail of
`runnerd.stderr.log` and of `runnerd.process-stderr.log`. It reads the
session diagnostics directory first, then the shared runtime-root
directory used by external Runners (Dot, Muse, grok_bot). It never reads
through a symlink. Each file keeps its own share of the 4 KiB budget.
- When recovery's process verification finds `process_not_running`, the
diagnostic adds these fields:
- `runnerRecoverableFailure`: a code-shaped string from
`runner-state.json`;
- `runnerDiagnostics`: the last 4 entries, redacted, 500 characters
each;
  - `runnerExitEvidence`: the redacted file tails, 4 KiB at most.
  
  `native.remote_runner_recovery_failed` carries them.
- The live process monitor's exit report now puts the durable state
summary first, then the file tails. Before, it used one or the other.
- `run-failure-diagnostics.ts` is unchanged. It still copies only the
closed labels, so no free text reaches failure reports.

## Verification

- `pnpm exec vitest run
server/src/services/native-runtime/remote-runner-launch-stderr.test.ts`:
the launch writer keeps stderr in the file, writes the identity marker
with the child's PID, appends across relaunches, rotates an oversized
file, trims a growing file while the child runs, and never writes
through a symlink.
- `pnpm exec vitest run
server/src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts`:
the evidence script output is bounded and refuses symlinks. Redaction
and bounds hold, and the new fields survive the event payload.
- `pnpm exec vitest run
server/src/services/native-runtime/native-session-executor.test.ts`
passes (1374 tests). One expectation was updated for the combined exit
report.
- `npx tsc --noEmit -p server/tsconfig.json` is clean.
- Older runner images: the script is server-side. It adds only `wc`,
`tr`, `tail`, `cat`, `mv`, `sleep` and `kill`. The script already needed
`node`, `date`, `chmod`, `mv` and `printf`, so any image that launched
runnerd before has these tools. The runner content ID is unchanged, so
the current Daytona image runs it unmodified. The restart-chaos waves
run on that image.

## Risks

- Low. A sandbox disk that is already full could make the stderr
redirect fail. `exec ... 2>>file` then fails, and the launch reports
`runner_remote_process_identity_unavailable` the same way a launch that
cannot write its identity marker does. The marker is written first, in
the same directory tree.
- Stderr content stays in the sandbox. Only a redacted 4 KiB tail goes
into the instance's own run log (run-log data path; not Telemetry and
not Observability).
- No wait is added before `turn.start`. The evidence read runs only on
the recovery failure path, with its own 10 s bound.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-11 15:31:04 -06:00
DottaandClaude Opus 5.5 566a97b882 fix(native): an unretirable predecessor sends the next attempt to a fresh sandbox (#20863)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a reused sandbox keep the previous run's warm Runner,
and a new run must retire that owner before it starts
> - #20844 retires the owner when Paperclip can prove it stopped. Some
owners can never be proven stopped: the controller journal drain
outlives the close grace, the Runner never confirms suspension, or a
parked question leaves a tool input unanswered
> - Then the new run failed before any provider work, and the task
stayed blocked with no automatic continuation
> - The general answer is to leave the old Runner on its sandbox and
start the next attempt on a fresh sandbox restored from the saved
workspace
> - This pull request adds that fallback for every path that retires a
predecessor, and settles a parked question's tool input once its Runner
is gone
> - The benefit is that an unprovable predecessor no longer fails a task
that did no work

## Linked Issues or Issue Description

**What happened?**
An automatic wake started a new run 30 to 60 seconds after a long
successful run on the same task, on the same reused sandbox. The new run
had to retire the previous run's warm session first. The retirement
failed and the run failed before any provider work, with one of:
- `Controller journal drain timed out; preserve the session before
replacing its writer.` (during instruction preparation, before runtime
selection, or in the executor);
- `provider_transport_failed: runner did not durably suspend before
checkpoint` (during instruction preparation);
- `runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate;
retirement_refused=journal_tool_input_unanswered` (the previous run had
parked on a question).
The tasks rested `blocked` with a reconciliation hold or with nothing
scheduled. One bounded retry reused the same sandbox and ended
`execution_finalization_deadline_exceeded`, then its recovery refused
with `allocation_ownership_ambiguous`.

**Expected behavior**
A run that did no provider work does not fail the task because its
predecessor cannot be retired. The next attempt drops the reused sandbox
and starts on a fresh one restored from the saved workspace. Unsaved
work is never dropped: when the workspace or the session backup is not
durably saved, the sandbox is kept and today's behavior applies.

**Steps to reproduce**
1. Complete a warm native run on a Daytona sandbox.
2. Wake the task again so that the next run must retire the warm owner
(configuration change, new session, changed agent files).
3. Make the owner's close fail (journal drain timeout, unconfirmed
suspension).
4. The new run fails before provider work and the task is not continued.

Refs #20844, #18373.

## What Changed

- `native-runtime/predecessor-retirement.ts`: a typed pre-provider error
`NativePredecessorRetirementUnprovenError`
(`native_predecessor_retirement_unproven`) and the `replaceBeforeReuse`
lease marker.
- Native executor: every path that retires a predecessor for a new run
(instruction-copy claim, instruction-directory reservation, session
reset, configuration change) throws the typed error when the close
fails. The owner stays registered for its own sandbox, so no run on that
sandbox launches over it. A run on a different sandbox forgets that
owner instead of replaying its cached rejected close. The executor does
not settle this error as a provider failure.
- A prior-owner fence that refuses a terminal owner on a resumed sandbox
(`prior_owner_terminal_state_indeterminate`) also throws the typed
error.
- Retirement of a finished remote owner accepts an unanswered tool input
when the owner's committed result is a governed-wait park
(`response_wake` on a task interaction) and its Runner's sandbox lease
ended elsewhere.
- Heartbeat (`heartbeat/predecessor-retirement-fallback.ts`): on the
typed error with no execution-start evidence and no result, mark this
run's reusable lease on that sandbox `replaceBeforeReuse` and log
`environment.lease_replacement_requested`. After the run ends, `failRun`
hands the prepared workspace on only through the shared unchanged-files
proof (`proveRemoteWorkspaceUnchanged`; a provisioning command or any
runner command refuses it). With the proof it records
`predecessorRetirementFallback`, closes the run's coordinator without a
result and re-queues the run in the uncharged `interrupted_preparation`
lane (native and legacy runtime modes); a run that answered a task
interaction keeps its own wake in that uncharged lane, so the retry
still carries the answer. Without the proof the workspace stays dirty,
heartbeat runs today's failed-workspace preservation and no retry is
scheduled.
- Environment runtime: a marked lease is replaced instead of resumed,
under the same rule as a failed resume
(`reusableSandboxReplacementIsSafe`: run ended, clean workspace with a
committed save, verified harness backup). The old lease goes to
`pending_cleanup` for the sweep. Otherwise the sandbox is resumed as
before.
- Chaos: session fault SR1U, the SR1 variant where the retained owner
cannot be retired. The server gets a test-only fault
(`PAPERCLIP_TEST_WARM_RETIREMENT_FAULT=unconfirmed`, armed only when
`NODE_ENV=test`). The oracle allows only the fallback failure and
requires the succeeding follow-up on a different sandbox.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md`, chaos README.

## Verification

- RED on master before this change (with #20844), then GREEN:
- `native-session-executor.test.ts -t "predecessor cannot be retired"`:
base fails with `Error: Controller journal drain timed out…` instead of
the typed error.
- `native-session-executor.test.ts -t "unretired predecessor"`
(instruction preparation): same base failure.
- `native-recovered-owner-followup.test.ts -t "unacknowledged"`: base
fails with `runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate;
retirement_refused=journal_tool_input_unanswered`.
- New contract tests: `heartbeat-native-predispatch-retry.test.ts`
(claimed native run: lease marked, activity logged, uncharged
`interrupted_preparation` retry, no recovery action, task not blocked,
retry succeeds) and
`environment-runtime-reusable-resume-replacement.test.ts` (a marked
lease is replaced without a resume; with unsaved work or no verified
backup it resumes as before),
`remote-workspace-unchanged-release.test.ts` (an untouched workspace is
handed on; a setup-written one stays dirty) and `retries.test.ts` (an
answered interaction re-queued in the uncharged lane stays a
resolved-interaction continuation with no failure retry spent).
- Existing suites pass: the native executor, recovered-owner follow-up,
instruction working copy, run completion, run failure cause and resume
replacement test files. `pnpm exec vitest run --config
tests/restart-chaos/vitest.config.ts question-handoff scheduler config`
passes. Server and chaos harness `tsc --noEmit` pass.
- Not run: a Daytona SR1U wave. It needs a runner image that matches
this checkout.

## Risks

- A sandbox marked for replacement is resumed when it cannot be replaced
safely. That run can meet the same unretired owner again, fail again in
the bounded lane, and then rest with Retry, as today.
- The replacement session restores the latest verified harness backup.
When the old Runner's final checkpoint was never saved, a same-session
continuation resumes from that backup.
- An unanswered tool input is accepted only for a committed
governed-wait park whose Runner's sandbox lease ended on another
sandbox.
- Previous-release story: the marker and the run record are additive
JSON. An older server ignores the marker and resumes the sandbox as
before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with
tool use (shell, file edits, test runs) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 12:10:24 -06:00
DottaandClaude Opus 5.5 a52f622be8 fix(native): retry a remote provider startup failure once its runner exit is proven (#20874)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) start a Runner in the
sandbox, and the Runner opens the provider session
> - When the provider rejects `session.open` at startup, no provider
work began
> - #20808 ends every such remote rejection at once with Retry, because
restart recovery could not prove that the failed runner was gone. A
fault that clears by itself therefore needs one human click
> - The Runner transport already settles the failed runner on close and
observes its exit. That is the missing proof
> - This pull request records that proof on the failure and lets the
same-run bootstrap retry use it. It stops a startup error that repeats,
so an image fault that a retry cannot fix does not loop
> - The benefit is that a transient remote startup failure recovers
without a person, and a permanent one still rests at once with a working
Retry that names the error

## Linked Issues or Issue Description

Refs #20808
Refs #20832
Refs #20820

**What happened?**
A remote (Daytona) native run whose provider rejected `session.open` at
startup ended as `native_provider_startup_rejected` with Retry. This
happened even when the Runner had settled and exited, and no provider
session, event or checkpoint existed. A person had to click Retry for a
fault that the durable record proves did no provider work.

**Expected behavior**
With a proven runner exit and no provider work, the run retries its
bootstrap on the same run automatically. A startup error that repeats on
the next attempt stops at once with Retry and names that error. Failures
saved without the proof keep today's behaviour.

**Steps to reproduce**
1. `cd server && npx vitest run
src/services/native-runtime/native-startup-rejected-retry.integration.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "remote startup rejection|session.open rejection"`
2. On master, 3 cases fail: the proven remote rejection is
`terminal_failure`, the repeat does not name its error, and restart
recovery holds the proven failure on
`ambiguous_remote_recovery_evidence`.

**Paperclip version or commit**
master at 3ded7bbd91.

## What Changed

- Runner transport (`runnerd-codex-transport.ts`): new
`runnerExitConfirmed()`. It is true only after a completed close settled
the runner (provider stopped, runner suspended) and the runner process
exit was observed.
- Codex driver (`codex-app-server-driver-impl.ts`): after it closes the
transport for a failed session open, it sets `runnerExitConfirmed` on
the error when the transport proves the exit.
- Executor (`native-session-executor.ts`):
- A remote `provider_session_open_rejected` with a startup
`providerFailure` keeps the same-run `bootstrap_retry` only with that
proof. The saved failure records `runnerExitConfirmed: true`.
- Each failed attempt's recovery history entry records a startup failure
signature: provider, category and message, with numbers ignored. When
the next attempt fails with the same signature, it stops as
`native_provider_startup_rejected`. Its next action names the startup
error and keeps "Retry starts a new run".
- Deterministic rejections (`requires_action` authentication, permission
or invalid request) stop at once as before.
  - Local targets are unchanged.
- Restart recovery (`native-restart-recovery.ts`):
`remoteRunnerExitedBeforeProviderWork` also accepts a settled
`provider_session_open_rejected` with `runnerExitConfirmed: true`. All
its other checks still apply: no provider session, no provider event, no
checkpoint, no runner-sent event, no successor.
- Docs: `doc/execution-semantics.md` and
`packages/paperclip-runner/docs/durable-recovery.md`.

## Verification

- Red on master, green here:
- `native-startup-rejected-retry.integration.test.ts`: proven remote
rejection → `retryable_failure` with `runnerExitConfirmed`; unproven →
terminal; repeat → terminal, the next action names `bwrap: loopback`,
and Retry is offered.
- `native-runner-restart-recovery.integration.test.ts`: proven failure →
`bootstrap_incomplete` (`remote_runner_exited_before_provider_work`); an
older release's failure without the proof stays held.
- `codex-app-server-driver.lifecycle.test.ts`: the driver records the
proof only when the transport reports it after close.
- `native-session-executor.test.ts`: unit cases for the rule, the
`cause` chain, the repeat signature and deterministic rejections.
- Local runs: the 3 server files above pass (1380 tests); restart
recovery `-t remote` 75 pass; driver lifecycle 25 pass.
- `tsc --noEmit` clean for `server` and `packages/paperclip-runner`.
- Not run: a Daytona chaos wave. There is no in-sandbox startup fault
seam yet (see Risks).

## Risks

- The proof depends on the transport's close settling the runner after a
failed `session.open`. The Rust runner keeps serving after a failed
`session.open` (lifecycle `Continue`) and accepts `runner.suspend`, so
close should settle. If it does not settle, there is no proof and the
run ends at once with Retry, which is today's behaviour. It is never
less safe than today.
- Retries stay bounded: the startup retry budget (3 attempts) and the
repeat rule both apply. A repeated identical error stops after one
automatic retry.
- Saved state: failures saved by older releases have no
`runnerExitConfirmed` and keep the old rule. History entries without a
signature never match a repeat. A mid-run deploy is safe: an old
executor never writes the proof.
- Not covered by chaos: a remote "Codex fails at startup N times" fault
needs a hook inside the sandbox Runner or Codex. DF1's seam is
server-side only, and no open chaos PR adds a seam inside the sandbox.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic) Opus 5.5, model ID `claude-opus-5-5`, run in Claude
Code with tool use (shell, file edit) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 11:55:15 -06:00
DottaandClaude Opus 5.5 e6b538efd2 fix(native): retire a superseded warm owner instead of failing the new run (#20844)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep a warm Runner between turns of the same task, so
the next run can start fast
> - When the next run needs a new provider session (for example after an
agent setting or skill changes), it first retires the previous run's
warm Runner
> - That retirement uses the previous owner's close. When the close
cannot prove checkpoint suspension, the new run failed before any
provider work, with the previous owner's error
`provider_transport_failed: runner did not durably suspend before
checkpoint`
> - A governed-wait park (a saved question or review request) always
leaves such an owner: its question callback stays unsettled by design. A
failed Launcher stop RPC leaves one too
> - Nothing retried the run, and the task rested as "recovery needed"
with no provider work done
> - This pull request lets the new run retire the old owner when that
owner is contained, and forces the containment when the close left it
unproven
> - The benefit is that a fresh run never fails on its predecessor's
close, and the chaos harness now covers this class

## Linked Issues or Issue Description

**What happened?**
A new run on a task whose previous run left a warm Runner retained
failed within two minutes, before any provider turn. The error was
`provider_transport_failed: runner did not durably suspend before
checkpoint`. The run row recorded `executionRecovery: { kind:
"bootstrap", providerWorkStarted: false }`. Two runs of the same agent
on two tasks failed this way 47 seconds apart, after a configuration
change made both start new provider sessions. One predecessor had parked
on a `request_human_input` question; its callback never settled.

**Expected behavior**
A run that has done no provider work does not fail because its
predecessor's Runner could not prove suspension. The new session never
resumes the old checkpoint. Core must only prove that the old owner is
contained, and then start.

**Steps to reproduce**
1. Run a warm native task that parks on a saved question (chaos workload
Q1).
2. After the asking run settles, change the agent's adapter
configuration, so the next run cannot resume the saved session.
3. Answer the question. The follow-up run retires the parked owner and
fails with the error above.

Related: #18373 added the stopped-owner reset path that this change
extends.

## What Changed

- `closeWarmNativeSession` (server native executor): during retirement
for a different normalized session, accept an unrecoverable close when
the old owner is contained. A plugin task owner must show its
`executionStopped: true` receipt and a completed handoff. Without a task
owner, the Runner's own durable suspension
(`settlement.runnerSuspended`) is the proof. Unsettled Core tool
callbacks of the discarded conversation no longer block the reset.
- With a plugin task owner, any close error qualifies once the Launcher
receipt exists. This includes a failed `stop` that
`releaseRunnerProcessOwnership` rethrows, and "Controller journal drain
timed out": the discarded session's journal lives in its own state root.
Owners without a task keep the narrower rule. `--matrix` now rejects
session faults.
- `createPluginRunnerTaskOwner`: add `retireSupersededTask()`. When
close left the receipt unproven, the new run sends one confirmed `stop`
(not resent after a confirmed stop), takes the credential snapshot, and
calls `complete`. It never reads the old state.
- Same-session continuation, a handoff that still fails, other close
errors and failed instruction collection still block replacement.
- Chaos harness: add session fault SR1
(`tests/restart-chaos/session-faults.ts`). With Q1 and `--lifecycle
warm` (or `--daytona-warm`), the board changes
`adapterConfig.idleTimeoutMs` after the asking run settles and before it
answers. `--tasks 2 --concurrency 2` is the two-concurrent-runs variant.
Q1's oracle applies, plus `session_rotated_before_answer` and
`follow_up_new_session`.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` and the chaos README
describe the new boundary and cell.

## Verification

- RED on master: `native-session-executor.test.ts -t "permanently
rejected plugin close during reset"` fails the `pending-semantic` and
`stop-unconfirmed-at-close` cases with
`NativeSessionCloseUnrecoverableError: provider_transport_failed: runner
did not durably suspend before checkpoint`. GREEN with this change: all
identity-transition cases pass (49).
- Chaos SR1 local, master server code (`--env local --lifecycle warm
--workloads Q1 --faults SR1`, seeds 301 with 2 tasks and 303 with 1
task): all tasks pass, and the follow-up started a new provider session.
The local parked owner closes cleanly, so the local cell does not
reproduce the failure. It needs a Daytona task owner.
- Chaos SR1 local with this change (seed 302, 2 concurrent tasks on one
agent): 2/2 pass, `follow_up_new_session` true for both.
- Not run: the Daytona SR1 variant. No runner image matches this
checkout's runner content ID, and this change does not touch runner
inputs.
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
question-handoff scheduler config` passes. Server and harness `tsc
--noEmit` pass.

## Risks

- The reset no longer waits for unsettled Core tool callbacks of the
discarded session. Their run is already terminal and their provider is
stopped, so no result can reach a provider, and the new session never
replays them. Core-side tool executions keep their own receipts under
the old run.
- A local owner without a task needs the Runner's durable suspension. A
local owner that did not suspend still blocks replacement, as before.
- Previous-release story: this changes no persisted state. Warm owners
live only in controller memory, so a deploy drops them as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with
tool use (shell, file edits, test runs) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:53:50 -06:00
DottaandClaude c8ae02e5ce fix(native): start a fresh session when an accepted run's provider state was not saved (#20811)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run in a per-turn remote sandbox saves a provider state
backup after its result is accepted. The sandbox is then released, so
the backup is the only copy of the provider session.
> - When that backup fails to save (for example, #20772: storage
throttling), the run keeps its accepted result. Nothing records that its
checkpoint can no longer be resumed.
> - The next run on the session rebinds that checkpoint and asks the
Runner to resume. The Runner has no state and refuses (#20757). With the
normal policy, the run opens a replacement session after that failed
attempt. With a governed-wait or unfinished-goal checkpoint, the run
fails with `native_provider_state_incompatible`, and only Retry or the
next message starts fresh.
> - The server already knew the state was gone when the backup failed.
The failed resume is wasted work.
> - This pull request marks the accepted run's checkpoint as not
resumable at the moment the backup fails. The next run then starts a
fresh provider session directly.
> - The benefit is that a lost backup no longer costs a failed run or a
failed resume attempt before the task continues.

## Linked Issues or Issue Description

Refs #20757 (the missing-state refusal that the next run reaches today).
Refs #20772 (the throttled backup upload that loses the backup). Refs
#20809 (a related fix for governed-wait follow-ups that reach the
missing state; this PR stops most of them from reaching it).

**What happened?**

1. A native run in a per-turn sandbox finished, and its result was
accepted.
2. `harness_state.backup.persist` failed. The run recorded
`native.post_result_failure`. Its run row kept a normal
`sessionCheckpoint`.
3. The sandbox was released.
4. The next run on the same session selected that run as its resume
source, and `rebindNativeSessionCheckpoint` returned the checkpoint.
5. The Runner found no saved state and refused with `PRP provider resume
state is unavailable`. The run then opened a replacement session after
the failed attempt, or (governed-wait or unfinished-goal checkpoint)
failed with `native_provider_state_incompatible`. Retry or the next
message then started fresh.

**Expected behavior**

The next run starts a fresh provider session at once, because the server
already knows the saved state is not there. The accepted result and the
task outcome stay as they are.

**Steps to reproduce**

1. Give a run row an accepted result and a normal `sessionCheckpoint`.
2. Make its close checkpoint fail in a per-turn sandbox
(`reusableLeaseConfigured: false`).
3. Select the next run's resume source with `findNativeSessionResumeRun`
and call `rebindNativeSessionCheckpoint`.
4. On `master` it returns the checkpoint, so the next run tries to
resume state that does not exist.

**Paperclip version or commit**

`master` at `556b249897`.

## What Changed

- `server/src/services/native-runtime/native-provider-state-unsaved.ts`
(new): `markAcceptedRunProviderStateUnsaved` adds
`nativeProviderStateUnsaved: { code:
"native_provider_state_incompatible", reason: "harness_backup_not_saved"
}` to the run's `runnerProfileJson`. One conditional `UPDATE` writes it
only when the run's coordinator holds a result with `schema_status =
'accepted'`. `clearRunProviderStateUnsaved` removes it.
- `server/src/services/native-runtime/native-session-executor.ts`: the
remote close checkpoint (`checkpointRemoteRunner`) now wraps the
existing body. When the checkpoint throws in a sandbox without a
reusable lease, it calls the marker, writes one run-log line, and throws
the original error. A marker or log failure cannot replace that error.
If a later checkpoint of the same run saves, it clears the mark.
- `server/src/services/native-runtime/native-session-resume.ts`:
`rebindNativeSessionCheckpoint` returns `null` for a source with the
mark, as it already does for `errorCode =
native_provider_state_incompatible`. `findNativeSessionResumeRun`
projects the new key.
- `doc/execution-semantics.md`: one additive paragraph.
- Tests in `native-session-resume.test.ts` and
`native-session-executor.test.ts`.

Guard conditions. The mark is written only when all of these are true:

- The close checkpoint threw before the durable backup was stored (after
the upload's own bounded retries). A later local publication, stamp or
cleanup failure does not mark the run.
- The target is a remote `sandbox` with `reusableLeaseConfigured !==
true`. A retained sandbox or an SSH host keeps the live state, so its
resume path is unchanged.
- The run holds an accepted result. A failure before acceptance changes
nothing.

Saved state from the previous release: run rows without the key keep
today's resume path. An older server that reads a row with the key
ignores it, and so it behaves as it does today. No schema change.

User-visible: the next run's transcript shows "Paperclip session
started" instead of "resumed", with a new session id. The source run's
log shows `provider state was not saved after the accepted result; the
next run starts a fresh provider session`. The fresh session gets the
full task context, as for other fresh starts.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-session-resume.test.ts -t "not
resumable only after|provider state was not saved"`: 9 pass (6 pure, 3
embedded Postgres).
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "marks
the provider state unsaved|uses the bundled Linux image authority for
remote opencode-new-durable"`: 8 pass. They include a durable store
failure (marked) and a local publication failure after the store (not
marked).
- Red on `master` (source files reverted, tests kept): the 3 "starts the
next run in a fresh provider session directly" cases fail (`expected {…}
to be null`), the DB "accepted" case fails the same way, and the
executor per-turn case fails (`mark` called 0 times).
- Guard removal checks:
- Sandbox guard replaced with `true`: the "retained sandbox" case fails
(`mark` called once).
  - `!checkpointBackupStored` removed: the "publish-failed" case fails.
- `schema_status = 'accepted'` removed from the marker: the "rejected"
case fails (`expected true to be false`).
- Negative cases that pass: no result, rejected result, retained
sandbox, records without the key or with another code (all three
recovery policies), and a later save that clears the mark.
- `cd server && npx tsc --noEmit -p .`: clean.
- Not related to this change and failing on `master` in this worktree
too: `runtime-selection.test.ts` connector skill cases (missing
`.claude/skills/*/SKILL.md`) and `native-session-resume.test.ts`
"archives a damaged prior epoch" (`ENOENT … authority-epochs`). Required
CI is the authority for those suites.

## Risks

- Low. The change only skips a resume that cannot succeed. The fresh
session uses the same workspace and full task context, like a Retry
after `native_provider_state_incompatible`.
- If the backup failed but the provider state were still somewhere
recoverable, the next run would lose provider context. For a per-turn
sandbox without a reusable lease, the restore path uses only the backup,
so this does not apply. Retained sandboxes and SSH hosts are excluded.
- One extra conditional `UPDATE` runs only on the failure path. The
normal path before `turn.start` has no new waits or work.
- `doc/execution-semantics.md` is also edited by #20809. The paragraphs
are at different places; whichever merges second rebases.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code, with
tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-11 10:36:00 -06:00
Nicky LeachandClaude Opus 5.5 ff8ec41e6a feat(runner): upgrade ACPX to 0.22.0 and move patched behavior into Paperclip (#20770)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Runner and the shared `adapter-utils` engine run ACP
agents through the embedded ACPX library
> - Paperclip pins ACPX 0.13.1 and keeps a large patch
(`patches/acpx@0.13.1.patch`) on it
> - ACPX 0.22.0 adds host hooks (`onAcpMessage`, `assertActive`,
`processLifecycle`, prompt-response `_meta`, `spawnAgent`-friendly
launch) that make much of that patch unnecessary
> - A large patch is expensive to port and hides behavior that belongs
to Paperclip
> - This pull request upgrades to ACPX 0.22.0 and moves most patched
behavior into Runner and `adapter-utils`
> - The benefit is a patch that is 484 lines instead of 1,334, with the
moved behavior covered by Paperclip tests

## Linked Issues or Issue Description

Refs #20675, #14430, #15075, #17942, #9980

**Proposed enhancement**
Upgrade the embedded ACPX library from 0.13.1 to 0.22.0. Keep only the
patch behavior that stock 0.22 cannot express.

**Motivation**
The 0.13.1 patch held behavior that Paperclip owns, for example usage
receipts, Copilot policy, model checks and the agent environment. Each
ACPX upgrade had to port all of it. ACPX 0.22 gives hooks that let
Paperclip do this work itself.

**Current behavior**
Runner and `adapter-utils` depend on 1,334 lines of patched ACPX 0.13.1
code.

**Expected behavior**
Runner and `adapter-utils` use ACPX 0.22.0. The patch keeps only:
extension methods, Cursor subagent routing, event `_meta` and the
message boundary chunk, the protocol guard hook, `spawnAgent`, and
`requestExtension`.

## What Changed

- ACPX 0.13.1 → 0.22.0 in Runner, `adapter-utils`, the server, the
Daytona runner images and the qualified profiles (`acpxVersion`).
`@agentclientprotocol/sdk` moves to 1.9.0.
- New `patches/acpx@0.22.0.patch` (484 lines). The 0.12.0 and 0.13.1
patches are removed.
- Delivery receipts: Runner confirms a permission reply when
`onAcpMessage` sees the exact outbound reply. Question and extension
replies count as delivered when Runner hands them back to ACPX. New
`interaction-delivery.ts`.
- Copilot policy: the existing policy function runs from `onAcpMessage`.
A violation stops the agent, cancels later permission requests and fails
the next prompt through `assertActive`.
- Model selection: stock 0.22 validation, plus a sidecar gate
(`createAcpxEffectiveModelGate`) that refuses a prompt if the agent
reports another model. Grok binds its model when the session opens.
Antigravity adds its concrete model to the model list ACPX reads.
- Typed session failures: Runner and `adapter-utils` read them from the
turn result `_meta` and fail the turn. They advertise the capability
themselves.
- Pi and Cursor usage receipts: a collector (`prompt-receipts.ts`) binds
them to the turn's own `session/prompt` reply.
- Goal capability: Runner reads it from the `initialize` reply. It keeps
it under `paperclip.goalCapability` in the saved agent capabilities
`_meta` and restores it when a saved session loads.
- Agent environment: Runner and `adapter-utils` set the agent
environment in `spawnAgent`. ACP terminals keep their current
environment. `adapter-utils` blocks inheritance of `XAI_API_KEY` and
refuses to start if the server has an `ACPX_AUTH_*` variable.
- `adapter-utils` spawn hooks (`spawnCwd`, stderr, `onAgentSpawn`): a
new `stockAcpRuntimeOptions` maps them to stock `spawnAgent` and
`processLifecycle`.
- Windows: ACPX 0.22 rejects raw command strings on Windows.
`adapter-utils` now passes an argv array there.
- Grok rules: a Runner guard (`grok-session-rules.ts`) writes
`session/new` `_meta.rules`.
- Turn-dispatch journal (#20675): ACPX 0.22 ignores errors from
`onAcpMessage`. The adapter keeps a failed `dispatched` write and throws
it from the turn's `assertActive`. ACPX calls `assertActive` before the
prompt write, so the write still stops. The real-ACPX test now checks
this route. `docs/durable-recovery.md` is updated.
- Plan events: the sidecar reads 0.22 `status` events tagged `plan`.
- CI: the backup-checkpoint lane now installs through the shared
`setup-ci-dependencies` action, like PR CI, so it resolves a stale
`pnpm-lock.yaml` in place. It runs on any root `package.json` change,
and pull requests never commit the lockfile, so its own frozen install
failed on dependency changes.
- Tests that checked removed patch internals are removed or rewritten:
the response-delivery, Cursor-usage and Pi-receipt package contracts,
and the goal-persistence packaging test. New unit tests cover the moved
behavior.

## Verification

Local runs on macOS (darwin-arm64), Node 26, on the rebased head:

- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/drivers/acpx src/cli`: 2,000 passed.
- `tsc -p packages/paperclip-runner/tsconfig.json` and
`tsconfig.test.json`: no errors, except the existing
`@paperclipai/paperclip-eval-kernel` build gap in `workflow-harness.ts`.
- Runner package contracts (`node --test
test/acpx-*-package-contract.test.mjs
test/acpx-sidecar-contract.test.mjs`), `scripts/build-inputs.test.mjs`,
`scripts/contracts/contract-fingerprints.test.mjs`,
`scripts/acpx-patch-packaging.test.mjs`: all pass. `node
scripts/generate-acpx-sidecar-contract.mjs --check` passes.
- `cargo test -p paperclip-runner-core --test acpx_park_preparation
--test acpx_park_readiness --test acpx_warm_attachment --test
native_provider_backend`: 86 passed. `cargo fmt --check` passes.
- `adapter-utils`: `tsc` clean. `vitest run src/acpx-engine
src/mcp-isolation.integration.test.ts`: 457 passed (one Gemini flag test
failed once under load and passes when run alone).
- Adapters: `claude-local` and `codex-local` `acp.quota.test.ts` (23 and
39 passed), `grok-local` (149 passed).
- Server: `scripts/prepare-runner-vendor.test.mjs`,
`native-session-executor.test.ts`, `quarantined-task-retry.test.ts`
(1,416 passed).
- `cargo test -p paperclip-runner-core --lib -- previous_acpx_release
acpx_upgrade unrestorable_saved_sessions`: 3 passed. `node --test
test/acpx-session-carry-forward-package-contract.test.mjs`: passes.
- `patch -p1 --forward` on a clean `acpx-0.22.0.tgz` gives output
identical to the port.
- Under full-suite load, some Runner live and recovery tests time out.
Each one passes when run alone. Full regression is left to CI.

Not run: live provider runs (they need provider credentials), the opt-in
Kimi live peer test, the Gemini filesystem smoke test, and Windows.

## Risks

- Previous-release story: sessions saved by the previous release (ACPX
0.13.1) carry forward. ACPX 0.22 loads the session records 0.13.1 wrote.
Runner moves a saved descriptor that names 0.13.1
(`CARRIED_ACPX_RELEASES`) to 0.22.0, and the sidecar keeps 0.13.1 as the
ACPX value in the session identity, so the saved session key and
identity record still match. The existing launch-profile rebind still
decides which sessions resume: a settled session, or the same run's
interrupted turn through lost-turn recovery. A session saved by any
other ACPX release is still refused as `provider_state_incompatible`.
`rules/acpx-profile-identity.mdc` and `docs/durable-recovery.md` now
describe this.
- Proof: `acpx-session-carry-forward-package-contract.test.mjs` resumes
a session record written by the real ACPX 0.13.1 with its Paperclip
patch. 0.22 loads the same provider session (no `session/new`), runs a
turn and keeps the earlier turn. Rust tests cover a settled 0.13.1
session that the next run attaches to, and a turn interrupted by the
upgrade. A recovery-identity test checks that the session key and
identity record match across the versions.
- `pnpm-lock.yaml` is not committed. CI resolves it for each run, and
the lockfile refresh workflow updates it after merge.
- Behavior change on the server: `adapter-utils` refuses to create an
ACPX runtime if the server environment has an `ACPX_AUTH_*` variable.
Before, ACPX ignored such variables. Stock 0.22 would use them to
authenticate agents with server credentials.
- Delivery confirmation is now Runner-owned. A permission reply is
confirmed by the exact outbound message. Question and extension replies
are confirmed when Runner hands them to ACPX, not when the pipe accepts
them.
- Users can select only models that the agent advertises. Stock 0.22
rejects other model IDs (Claude is exempt).
- ACP terminals keep the sidecar or server environment, as before. This
existing exposure is not changed by this pull request.
- The `adapter-utils` Gemini version probe now runs with the server
environment plus the run variables, not the run variables alone.
- The pack contract digest and profile `acpxVersion` change, so Daytona
needs a new runner image.
- Grok launch flags (`--model`, `--reasoning-effort`) were checked on
1.0.13 but session open needs a login, so Grok keeps the existing
`_meta` path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool
use (shell, file editing, subagents). Extended reasoning was used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 07:28:00 -08:00
DottaandClaude Opus 5.5 fa3cd6f330 fix(native): end provider startup rejections with Retry instead of an ownership hold (#20808)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner starts Codex for native runs. On a remote
sandbox such as Daytona, the sandbox is the OS boundary.
> - A Codex planning run on Daytona failed in `session.open`. Codex's
read-only sandbox helper could not set up loopback in the container
(`bwrap: loopback: Failed RTM_NEWADDR`), and Codex loads `AGENTS.md`
through that helper.
> - The server treated that definite provider rejection as an
interrupted transport. Its same-run bootstrap retry went through remote
restart recovery and stopped on `ambiguous_remote_recovery_evidence`.
The task showed "Recovering saved work…" and no Retry, although no
provider work was done.
> - This pull request gives such startup rejections a named, permanent
outcome with a working Retry, and pins ask and standard runs to the
external sandbox policy.
> - The benefit is that a provider startup refusal ends as a clear
failure with Retry, never as an ownership hold.

## Linked Issues or Issue Description

**What happened?**
A Codex planning run on a Daytona sandbox failed in `session.open` with
`error creating thread: failed to load AGENTS.md instructions ... bwrap:
loopback: Failed RTM_NEWADDR: Operation not permitted`. The server
recorded `native_session_interrupted` with recovery mode
`bootstrap_retry`. Thirty seconds later, remote restart recovery blocked
the run with `native_restart_recovery_blocked` /
`ambiguous_remote_recovery_evidence`. No server restart happened. All
default-mode Codex runs on the same image succeeded that day; only the
planning run failed.

**Expected behavior**
A provider startup failure with no provider work must end as a clear
failure with a working Retry, or get a bounded automatic retry that also
ends with Retry. It must never become an ambiguous-ownership hold.

**Steps to reproduce**
1. `cd server && npx vitest run
src/services/native-runtime/native-startup-rejected-retry.integration.test.ts`.
On master the deterministic case is `retryable_failure` /
`native_session_interrupted`.
2. `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "provider
itself rejected"`. On master the new code is not produced.

**Paperclip version or commit**
master at 84fc67317e.

## What Changed

- Server (`native-session-executor.ts`): new permanent code
`native_provider_startup_rejected`.
- A definite `session.open` failure maps to it when its startup
`providerFailure` is `requires_action` for `authentication`,
`permission` or `invalid_request`.
- On a remote execution target (`nativeRemoteStartupRejectionCode`), any
definite `session.open` failure with a startup `providerFailure` maps to
it. A remote same-run bootstrap after the runner launched cannot prove
ownership and stops on `ambiguous_remote_recovery_evidence`.
- The run ends as `terminal_failure` with a board owner and a plain next
action. The board offers Retry, which starts one new run with a fresh
provider session.
- On a local target, unknown, transient, rate-limit, quota and model
startup failures keep the bounded same-run retries and end as
`native_session_retry_exhausted` with Retry. That code is unchanged
because the chat-channel failed-run retry depends on it.
- Runner: planning runs keep Codex's
`paperclip-runner-workspace-read-only` profile, also in an external
sandbox. Codex 0.160.0's plan collaboration mode is a prompt ("Plan Mode
(Conversational)") and does not block writes or commands, so it cannot
replace that enforcement.
- Ask mode: an ask task builds `executionMode: "default"`. Its launch
has `readOnly: false`, the external sandbox profile, and no
collaboration mode. Ask mode restricts only Paperclip mutation tools,
not the Codex sandbox.
- Tests:
- `native-startup-rejected-retry.integration.test.ts` (new, embedded
Postgres, Codex provider): a deterministic rejection ends at once with
`canRetry: true`, and Retry starts exactly one new run. Unknown and
transient local failures stay `retryable_failure`. A spent retry budget
ends `terminal_failure` with `canRetry: true`.
  - Unit tests for the deterministic, local-retry and remote rules.
- `runner-provider-launch.test.ts`: ask, standard and planning launches
in an external sandbox.
- The Rust external-sandbox test now fails if a non-planning turn sends
a collaboration mode (`--forbid-plan-collaboration` switch on the fake
app server).
- Docs: `packages/paperclip-runner/docs/codex-driver.md`,
`packages/paperclip-runner/docs/durable-recovery.md`,
`doc/execution-semantics.md`.

## Verification

- Red on master, green here: the commands in "Steps to reproduce".
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts`: 1364 pass.
- `npx vitest run src/live/runner-provider-launch.test.ts`
(packages/paperclip-runner): 22 pass.
- `cargo test -p paperclip-runner-core --test codex_provider
external_sandbox`: pass. `cargo fmt --check`: clean.
- `npx tsc --noEmit -p server/tsconfig.json`: clean.
- `pnpm verify` returned exit 2 (a global CI input changed). Required CI
supplies the broad coverage.

## Risks

- Codex planning on Daytona still fails at thread creation. With this
change it ends at once with the provider message and Retry instead of an
ownership hold. A follow-up must make Codex's read-only sandbox work in
that container, or add a verified Paperclip read-only guard. This PR
does not relax read-only enforcement.
- A remote startup failure that could clear by itself now needs a Retry
click instead of an automatic same-run retry. Before this change, that
retry ended in an ownership hold, so no working automatic path is lost.
- A false `native_provider_startup_rejected` costs automatic retries and
shows Retry earlier. It never replays provider work, because
`session.open` precedes every prompt.
- Saved state: no persisted shape or launch argument changes. A run that
an older release held on `ambiguous_remote_recovery_evidence` after the
same rejection already offers Retry once its saved workspace is
recovered. No migration is needed.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic) Opus 5.5, model ID `claude-opus-5-5`, run in Claude
Code with tool use (shell, file edit, browser reads) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 07:48:18 -06:00
Nicky Leach 4d141cdf64 Carry plugin task refusals and batch project reads (#20762) 2026-11-11 00:49:03 -08:00