Commit Graph
422 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 d1ca0cc218 fix(native): stop retrying a run that filled its sandbox disk (#22239)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs execute in a managed sandbox with a fixed disk
size
> - An agent can fill that disk. The runner then exits with "No space
left on device". The disk reserve restarts the runner, but the turn in
progress can still be lost, and the run fails
> - The run failure did not carry the full-disk fact. The run was
classified as a generic provider or runner failure. Automatic recovery
then started the same work on a fresh sandbox of the same size, which
filled the disk again. After three attempts the task stopped in
"Recovery needed" with no useful explanation
> - This pull request names the cause `sandbox_disk_full` when the
runner reported a full disk during the run, and stops automatic retries
for that cause
> - The benefit is that Paperclip no longer spends sandboxes and time on
retries that cannot succeed, and the task tells the user what to do in
plain language

## Linked Issues or Issue Description

**What happened?**
A native run in a managed sandbox filled the 10 GiB disk. The runner
exited with "No space left on device (os error 28)". The runner was
restored, but the in-flight turn was lost and the run failed as
`native_provider_terminal_failed`. Automatic recovery repeated the same
work on a fresh sandbox of the same size three times. Each attempt
filled the disk again. The task ended in "Recovery needed" with an
unrelated cause.

**Expected behavior**
When a run ends because its sandbox disk is full, the run failure names
that cause. Automatic recovery does not repeat the same work on a
sandbox of the same size. The task shows a plain-language explanation
and a working Retry.

**Steps to reproduce**
1. Run a native agent task in a managed sandbox whose work installs more
than the free disk space.
2. Wait until the runner exits with "No space left on device".
3. See the run fail as a generic provider failure, and see automatic
retries start fresh sandboxes that fail the same way.

**Relevant logs or output**
`paperclip-runner` exited with "No space left on device (os error 28)".
The run then failed with `provider_turn_lost_on_restore`, classified as
`native_provider_terminal_failed`.

Refs #22068 (disk reserve that restarts the runner). Refs #22235
(continues a turn that a runner restore lost, once, in the same
sandbox). This PR covers the case where the run still ends.

## What Changed

- Server: each run's diagnostic lease now records when a runner
diagnostic reports a full disk (`ENOSPC`, `EDQUOT`, "No space left on
device", "Disk quota exceeded"). The runner's exit detail already
reaches the server as a diagnostic, so no runner change is needed. The
fact belongs to that run's diagnostic epoch only. A later run on the
same warm session, or a diagnostic after the epoch retires, does not see
it.
- Server: native failure classification returns `sandbox_disk_full` when
either (a) the run's lease saw a full-disk report, or (b) the error
chain carries the runner's existing `sandbox_disk_full` launch code (a
relaunch that cannot stage on the full disk). A fixed list of failures
keeps its own meaning: permission, approval, usage limit, model,
runtime, integrity, cleanup quarantine, ownership, saved-state and
admission failures.
- Server: `sandbox_disk_full` is a permanent native failure. The run
does not take another same-run attempt. Native safe replacement does not
pick it up, because it is not one of its failure codes. The task becomes
blocked with a board-owned recovery action and a plain-language next
action.
- Server: the bounded retry scheduler (`scheduleBoundedRetryForRun`)
refuses `sandbox_disk_full`. It does not schedule a `transient_failure`
retry on a fresh sandbox of the same size.
- UI: the task chat shows "Disk full" and a plain explanation instead of
the raw failure code.
- Retry stays available. `sandbox_disk_full` is not a reconciliation
cause, so the board's Retry starts a new run when the user chooses.

## Verification

- Red before the fix (captured on the unfixed code):
- `heartbeat-retry-scheduling.test.ts` "does not repeat a run that
filled its sandbox disk on a fresh sandbox of the same size": expected
`not_scheduled`, received `scheduled`.
- `native-sandbox-disk-full.test.ts` "names the full disk and stops
instead of retrying the same work": expected `sandbox_disk_full`,
received `native_provider_terminal_failed`.
- `native-sandbox-disk-full.test.ts` "names the full disk when the
runner could not be relaunched on it": expected `sandbox_disk_full`,
received `native_session_interrupted`.
- Green after the fix:
- `cd server && npx vitest run
src/services/native-runtime/native-sandbox-disk-full.test.ts
src/__tests__/heartbeat-retry-scheduling.test.ts
src/services/native-runtime/remote-runner-disk-reserve.test.ts` (90
passed).
- `cd ui && npx vitest run src/components/TaskChatThread.test.tsx` (426
passed).
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-session-diagnostics.test.ts`.
- Safety cases in the tests:
- A runner that exited for another reason still gets its bounded retry.
- A turn lost on restore without a full-disk report keeps
`native_provider_terminal_failed`.
- Integrity and approval failures keep their codes when a full disk was
reported.
- A full-disk report belongs only to its own run's diagnostic epoch; run
B on the same session does not see run A's report.
- After a full-disk report, other runner and provider loss codes
(session.open timeout, initialize timeout, protocol error, process
status failure) also become `sandbox_disk_full`.
- Worst-case automatic run starts per hour across a company: before,
each disk-full task started up to 3 automatic runs, each on a fresh
sandbox (3 × the number of affected tasks). After, a disk-full failure
starts 0 automatic runs. Only a person's Retry starts a new run. Real
transient failures keep their existing bounded lanes, so their worst
case is unchanged.
- Not run locally: the full server suite, the full typecheck and build.
Required CI covers them.
- Not verified live on staging yet. A live check needs a run that fills
its sandbox disk after this change deploys.

## Risks

- A run whose runner reported a full disk, and which later fails with a
runner or provider loss for an unrelated reason, is now labeled
`sandbox_disk_full` and is not retried automatically. The board's Retry
still works. This is the safer error: a full disk is the most likely
cause, and an automatic retry would repeat the same disk use.
- The full-disk report lives on the run's in-memory diagnostic lease. A
server restart between the report and the failure loses it, and the run
falls back to the previous classification. That is the previous
behavior, not a new failure.
- Low risk otherwise. No schema or API change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 23:19:12 +00:00
Devin FoleyandPaperclip 8285a81280 fix(native): retry a fresh launch whose provider pack could not be verified (#22106)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a remote native run starts its provider, the controller
selects a provider pack and verifies it in the sandbox
> - The verification command retries once. If both reads fail without a
result (a timeout, a signal, or a transport error such as HTTP 502), the
run fails with `runner_remote_provider_verification_unavailable`
> - That code holds the run for a person in every case. The hold exists
for reattachment, which does not spend a new provider attempt and could
loop
> - On a fresh launch the code is also not on the list of failures that
prove no provider session started. So the run is classified as
"ambiguous state", and the task waits behind a Board-owned Retry
> - This pull request retries a fresh launch's unverified pack on the
same run, within the provider attempt budget, and keeps the hold for
every other attempt
> - The benefit is that a short sandbox outage during startup no longer
stops a chat or task until a person clicks Retry

## Linked Issues or Issue Description

**What happened?**
A chat message started a fresh native run on a new sandbox. The
preinstalled provider pack did not match, so the controller uploaded the
pack. Both reads that verified the uploaded pack failed with "Request
failed with status code 502". No provider process started. The run
failed with `runner_remote_provider_verification_unavailable`, the
recovery mode was `ambiguous_state`, and the chat showed a Retry button.
A manual Retry then succeeded.

**Steps to reproduce**
1. Start a fresh remote native run.
2. Make both provider-pack verification reads fail without a result, for
example with a 502 from the sandbox command API.

**Expected behavior**
The run retries its startup on the same run, with backoff, because no
provider session started.

**Actual behavior**
The run is held for a person with "Inspect the original provider failure
and explicitly resolve the ambiguous session state".

## What Changed

- `native-session-executor.ts`:
- `nativeProviderRecoveryEvidence` treats
`runner_remote_provider_verification_unavailable` as definitely
pre-session when the attempt is a fresh bootstrap (the caller passes
`bootstrap`). As for every pre-session code, this applies only when the
run has no checkpoint and no provider event.
- `nativeSessionFailureDisposition` takes an optional `freshBootstrap`
flag. With it, `runner_remote_provider_verification_unavailable` is
retryable instead of permanent, so the existing same-run startup retry
and its three-attempt budget apply. The executor passes the same
fresh-bootstrap condition it already uses for `bootstrap`.
- Reattachment, warm session recovery and retained transitions do not
pass `bootstrap`, so they keep the hold.
- `doc/native-workspace-finalization-recovery.md`: describe the
fresh-launch retry.
- Tests: the disposition retries attempts 1 and 2 of a fresh launch and
stops at attempt 3, and the existing reattachment hold test is
unchanged. An integration test shows `bootstrap_retry` for a fresh
launch with no checkpoint, and `ambiguous_state` without `bootstrap`.

## Verification

- `npx vitest run` on the new cases: pass. Both fail without the source
change.
- `npx vitest run` on the native session executor, controller recovery,
remote provider pack, online source, status arbiter corpus, continuation
and startup/retry suites: 2203 pass, 8 skipped.
- `npx tsc --noEmit` passes for the server.

## Risks

- Low. Only a fresh launch with no checkpoint and no provider event
changes. A deterministic verification outage now costs up to two more
startup attempts, with backoff, before the run is held.
- A completed verification that fails
(`runner_remote_provider_artifact_incompatible`) is unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-18 10:40:31 -08:00
DottaandClaude Opus 5.5 73dc33c620 fix(runner): keep a disk reserve so a full sandbox disk does not strand the runner (#22068)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents in a Paperclip-managed sandbox run under the Paperclip runner
(`runnerd`), which keeps durable state on the sandbox disk
> - An agent can fill that disk with its own work. In one case the agent
installed a Rust toolchain and a second checkout with dependencies on a
10 GiB sandbox
> - `runnerd` then could not write its state and exited. Each relaunch
tried to upload its launch files to the full disk and failed, 83 times
in 7 minutes
> - The run ended as a generic provider failure that asked a person to
inspect it, and Retry repeated the same failure
> - This pull request keeps a small disk reserve that belongs to
Paperclip. Recovery releases it to relaunch the runner, and stops at
once with a named code when nothing is left to free
> - The benefit is that the agent's turn can continue and see the full
disk in its own tool output, instead of the run dying with it

## Linked Issues or Issue Description

**What happened?**
An agent in a managed Daytona sandbox filled the 10 GiB disk. `runnerd`
exited with `failed to persist bounded diagnostic: No space left on
device (os error 28)`. Runner process recovery then retried for the full
420000 ms reconnect grace. Every attempt failed with
`.paperclip-upload-<id>.tar.gz: create: ... no space left on device`.
The run failed as `native_provider_terminal_failed`, and the recovery
asked a board user to verify that the provider stopped.

**Steps to reproduce**
1. Run an agent in a managed sandbox and have it fill the disk (for
example a Rust toolchain plus a full dependency install).
2. `runnerd` exits when it next writes its state.
3. Watch recovery retry the relaunch upload until the grace expires.

**Expected behavior**
Paperclip relaunches its runner on a disk the agent filled, so the turn
continues and the agent sees the full disk in its tools. When the runner
truly cannot start, recovery says so at once with a named cause instead
of retrying for 7 minutes.

**Environment**
Managed Daytona sandbox (fleet snapshot, 4 CPU, 8 GiB memory, 10 GiB
disk), `paperclip_runner` native runtime.

## What Changed

- New `server/src/services/native-runtime/remote-runner-disk-reserve.ts`
owns the reserve: hold/release scripts, the full-disk classifier, and
`createRemoteRunnerDiskReserve`. A release waits for an in-flight hold,
and no hold starts after a release.
- Both remote runner launchers (agent runs and Dot) get a reserve path
in managed sandboxes only:
`<workspace>/.paperclip-runtime/paperclip-runner/disk-reserve`.
Workspace saves and Git already exclude `.paperclip-runtime`. SSH hosts
get no reserve.
- After the first launch's identity is confirmed, the launcher holds 256
MiB in the background, only while 2 GiB more is free. It is off the
startup path.
- A relaunch releases the reserve before it stages. A launch that hits a
full disk while staging releases the reserve and stages once more.
- Release tries twice and reports `released`, `not_held` or `failed`.
Only `not_held` ends the launch with `sandbox_disk_full`. A release that
cannot run rethrows the original error, so recovery still retries.
- Command-based staging (no `syncIn`) keeps a full disk recognizable
with an `ENOSPC` code instead of a bare
`runner_remote_*_staging_failed`.
- `SANDBOX_DISK_FULL_FAILURE_CODE` and `isSandboxDiskFullFailure` live
in the runner package. Runner process recovery checks the error code and
stops at once instead of retrying for the whole grace. Other staging
failures still retry.

## Verification

- Red before the fix, green after:
-
`server/src/services/native-runtime/remote-runner-disk-reserve.test.ts`:
release-and-restage, release before relaunch, `sandbox_disk_full` when
release frees too little, no retry for a failure that is not a full
disk, hold/release scripts in a real shell, no hold below the free-space
floor. Before the fix, the launch rejected with the exact upload error
from the incident.
- `packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts`
"fails at once when a replacement cannot be staged because the sandbox
disk is full": before the fix it was still retrying after 4 s; after, it
fails with `sandbox_disk_full` after one replacement.
- Kept: "retries a replacement whose staging failed before any process
launched" still passes, so a raw transient staging failure still
retries.
- Updated: "confirms no replacement process exists when staging fails
before the launch RPC" now expects `sandbox_disk_full` with the upload
error as cause. Exit is still confirmed and no launch RPC runs.
- Real sandbox from the same fleet snapshot (Daytona, 10 GiB): hold
created a 256 MiB reserve. The disk was filled to 100%. A 5 MB write
then failed. Release returned `true`, and a 50 MB write then succeeded.
A second release returned `false`. Hold on the now 98% full disk did not
create a reserve.
- `pnpm --filter @paperclipai/server exec tsc --noEmit` passes. `pnpm
verify` result is in a comment below.
- Review round: tests for a lost release RPC, hold/release ordering,
command-based staging on a full disk, the hold after a dispatched first
launch (not on relaunch), and no reserve on SSH. Each launcher-level
test failed with its fix removed.
- Focused suites on the review revision: transport 335/335, launcher
130/130 (filtered), launch assets 35/35, reserve 14/14. Server and
runner typecheck pass.
- Not run: a full product run in which an agent fills the disk and the
relaunched `runnerd` resumes the turn end to end.

## Risks

- Each managed sandbox holds 256 MiB more disk while it has more than 2
GiB free. That is 2.5% of a 10 GiB sandbox.
- A relaunch for any reason releases the reserve for the rest of that
run. The next run's launch takes it again.
- If the agent keeps writing after the relaunch, the disk can fill
again. The second time, nothing is left to release, and the run fails at
once with `sandbox_disk_full` instead of after 7 minutes.
- The run's error code is still `native_provider_terminal_failed`. The
message now names `sandbox_disk_full`. A dedicated failure code and
recovery text are a follow-up.
- `runnerd` loses its own error message when it cannot persist its
diagnostic. This PR does not change the Rust binary.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 15:45:20 -06:00
Nicky Leach bc0a236b95 feat(runner): accept a resume-compatible build for a relaunched released Runner (#21976) 2026-11-17 06:15:01 -08:00
DottaandClaude Opus 5.5 22d3f9a03c fix(native): finish a failed warm session's cleanup by itself and continue the task (#21977)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runner runs keep a provider session per task. A follow-up run
can reuse the warm session of the run before it, in a remote sandbox.
> - When a warm follow-up fails, the previous session is closed. The
close saves a backup of the provider session first.
> - The close looked up the backup owner through the task's current tool
authority. The failed follow-up had already released that authority.
Every close attempt failed with `native_harness_storage_owner_missing`,
so the close could never finish.
> - After nine failed attempts, the task's admission gate became
`operator_required`. The gate retires only for a stopped local process
or for the exact run that owns the session. A remote sandbox session
owned by an earlier run never matched. A queued continuation and a Retry
stayed queued forever, and the task showed a Retry button that could not
work.
> - Separately, a remote transport failure could never produce the local
stop proof that the safe-replacement lane waits for. Nothing continued
the task until an unrelated wake arrived 15 minutes later.
> - This pull request makes the close save its backup under its own
owner, retires a cleanup gate when the task's sandbox is confirmed
terminated, and refuses an unprovable remote transport failure at once
so the task continues in a fresh isolated run.
> - The benefit is that recovery finishes by itself, and a Retry that is
shown starts a run.

## Linked Issues or Issue Description

**What happened?**
In a restart-chaos wave, a warm follow-up run on a remote sandbox failed
with `provider_transport_failed` (`PRP command run.attach timed out`).
The previous run's session close then failed on every attempt with
`native_harness_storage_owner_missing`. The cleanup gate became
`operator_required`. The continuation run was created 15 minutes later
and then stayed queued until the wave ended ("queued run kept queued
while the task's previous session finishes cleanup"). The task showed a
Retry banner, but a Retry would wait behind the same gate.

**Expected behavior**
A failed session close settles or is retired by itself with evidence.
The next run of the task starts in seconds. A Retry that is shown starts
a run.

**Steps to reproduce**
1. Run a native task on a remote sandbox with a warm session lifecycle.
2. Make the warm follow-up run fail during startup, after it has reused
the previous session.
3. Watch the previous session's close: every attempt fails with
`native_harness_storage_owner_missing`, and the queued next run of the
task never starts.

**Paperclip version or commit**
master at b2b2c1e60b (wave) and 7685b7a4fc (this branch base).

**Deployment mode**
Hosted server with remote Daytona sandboxes (restart-chaos harness).

## What Changed

- `native-session-executor.ts`: the remote checkpoint resolves its
backup owner from the current scope authority, and falls back to the
backend's own authority epoch when the scope has none. The
controller-persistence `authorityRunId` closure uses the same fallback.
A newer run that took over the session still wins.
- `paperclip-runner` `native-session-runtime.ts`: new
`nativeSessionCleanupRemoteScopes` lists the remote resources whose
quarantined cleanup holds a task. New
`completeTerminatedRemoteNativeSessionCleanupForTask` retires settled
quarantines of the task in one terminated remote resource, whichever
run's session they hold. An in-flight close keeps its entry.
- `remote-execution-termination.ts`: new `terminatedRemoteCleanupScope`
confirms that the company's newest lease on the remote resource is
released with its termination receipt and that no lease on it is still
held.
- `native-cleanup-admission-hold.ts`: queue admission now retires a held
cleanup when its sandbox is confirmed terminated. It logs the retirement
and records a `native.session_cleanup_gate_retired` run event with the
termination receipt. The local stopped-process path is unchanged.
- `native-safe-replacement.ts`: a transport failure on a remote or
non-Codex session can never produce the local stop proof. It is now
refused at once with `provider_failure_meaning_unverified`. The existing
disposition pass then continues the task in a fresh isolated run from
saved work. A local Codex run keeps the existing wait for stop proof.

## Verification

- Red/green, each verified red without its fix:
- `server/src/services/native-runtime/native-session-executor.test.ts` —
"saves a closing session's backup under its own owner after a failed
follow-up released the scope authority". Red:
`native_harness_storage_owner_missing` (the incident error).
- `server/src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts`
(embedded Postgres) — "starts a queued run once the previous session's
sandbox is confirmed terminated, whichever run released it", plus "keeps
the run queued while a different sandbox of the task is terminated".
Red: the queued run stays queued.
- `server/src/services/native-runtime/native-safe-replacement.test.ts`
(embedded Postgres) — "continues a transport failure whose stop can
never be proven locally without waiting for a person" for a remote
sandbox (refused and continued fresh) and a local Codex run (unchanged).
Red: no denial and no hold.
- `pnpm verify` and server and runner typechecks pass locally.

## Risks

- The admission gate now trusts a provider termination receipt on the
newest lease of the same remote resource, not only one bound to the
session's own run. A reused sandbox with a newer held lease is not
retired.
- A remote transport failure no longer waits for a stop proof that
cannot arrive. The fresh continuation is isolated (write authority
revoked, new root) and stays under the existing per-task and per-minute
continuation brakes. Nothing is added before `turn.start`.
- The run.attach timeout itself is not fixed here. The controller held
the attach frame behind a checkpoint flush to the follow-up's writer.
The run's own diagnostics were lost because they were written to the
predecessor's run log. This PR makes the failure recover by itself; the
attach stall needs a separate fix with better diagnostics.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with
tool use and code execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 02:25:02 -06:00
DottaandClaude Opus 5.5 784bc06276 perf(runner): reuse the sandbox's identical runnerd instead of uploading it on every cold boot (#21949)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Remote agent runs start `paperclip-runnerd` inside a sandbox, such
as a Daytona sandbox
> - Agent boot time is a measured product constraint, and cold boot on
Daytona grew from about 11 s to 25–75 s in restart-chaos waves
> - The span that grew is `runner.artifact.prepare`: the controller
uploaded its 31 MB `paperclip-runnerd` to every new sandbox
> - The controller uploads this file whenever
`PAPERCLIP_RUNNER_REMOTE_BINARY_PATH` is set, even when the sandbox
image already has the same bytes
> - The upload time depends on the sandbox file-upload throughput. That
throughput changed from about 1 s to 5–48 s for the same 31 MB, on the
same image and with no runner change
> - This pull request makes the controller use the image's
`paperclip-runnerd` when its SHA-256 equals the configured artifact
> - The benefit is a cold boot that does not depend on sandbox upload
throughput: about 9–11 s instead of 33–45 s in the same hour

## Linked Issues or Issue Description

No issue exists. This describes the bug, following
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

Cold `agent_boot` on Daytona became 2–7 times slower. In restart-chaos
waves on master `c5218f5ec4`, the cold boot p50 was about 11 s for
claude, about 10 s for codex and 15–25 s for OpenCode. On master
`3452639a81` it was 25 s, then 55 s, then 70–78 s for claude and codex
over five waves. OpenCode reached 238–265 s. The run spans show that the
extra time for every adapter is in `runner.artifact.prepare` (codex
example: 42.7 s, before it was 1.2 s). In this span the controller
uploads its own 31 MB `paperclip-runnerd` to the new sandbox.
`PAPERCLIP_RUNNER_REMOTE_BINARY_PATH` is set, so
`mayUsePreinstalledRunnerArtifact` returned `false` and the controller
never used the identical `paperclip-runnerd` in the sandbox image. The
upload time follows the sandbox upload throughput, which changes over
time. None of the 17 commits between the two masters changed this path.

**Expected behavior**

When the sandbox already has a `paperclip-runnerd` with the same SHA-256
as the configured artifact, the controller links it and does not upload
it. Cold boot does not depend on upload throughput.

**Steps to reproduce**

1. Set `PAPERCLIP_RUNNER_REMOTE_BINARY_PATH` to the `paperclip-runnerd`
from the Daytona runner image.
2. Start a cold agent run in a new Daytona sandbox from that image.
3. Look at the run spans. `runner.artifact.prepare` holds a full upload
of the runner (5–48 s in our waves), and there is no
`runner.artifact.discover` span.

**Paperclip version or commit**

`3452639a81` (also current master `6205b24d54`). Image
`ghcr.io/paperclipai/paperclip-daytona-runner@sha256:710deab7375a…`.

**Agent adapter(s) involved**

All remote adapters: claude, codex and OpenCode.

## What Changed

- `mayUsePreinstalledRunnerArtifact` accepts a preinstalled
`paperclip-runnerd` when an artifact is configured and the controller
has that artifact on disk.
- In that case the existing SHA-256 check
(`runner_remote_preinstalled_artifact_mismatch`) is mandatory. A
different, missing, or unreadable image copy still falls back to the
upload.
- When an artifact is configured, a failed discovery probe counts as "no
preinstalled runnerd", so the controller uploads the configured file as
before.
- Tests: a unit case for the gate, and three cases in the runner wiring
test: identical configured artifact (no `syncIn`), mismatched configured
artifact (upload), and failed discovery with a configured artifact
(upload).
- `docs/deploy/environment-variables.md` describes the new behavior.

## Verification

- Red/green: `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t
"server-owned replacement artifact|contract-compatible
runnerd|configured artifact the controller holds"`. Without the fix 3
tests fail. With the fix 27 pass.
- `pnpm verify` passes.
- Restart-chaos waves on Daytona, image `710deab7375a`, 6 tasks (2
claude, 2 codex, 2 opencode), A1/K0, run back to back in one hour:

| Wave | Fix | claude cold boot p50 | codex | opencode |
`runner.artifact.prepare` |
| --- | --- | --- | --- | --- | --- |
| w-20261117T005138 | no | 38.4 s | 33.2 s | 36.0 s | not recorded |
| w-20261117T005627 | yes | 9.5 s | 8.8 s | 10.6 s | 0.5–0.6 s, all runs
use the image copy |
| w-20261117T005956 | no | 44.1 s | 13.1 s (max 47.1 s) | 45.5 s |
4.9–37.8 s upload |

All 18 tasks passed.

## Risks

- Low risk. The controller uses the image copy only when its SHA-256
equals the configured artifact, so the runner bytes do not change. Every
other case keeps the current upload.
- One extra `command -v` probe and one `sha256sum` probe run when an
artifact is configured. Each takes about 0.1 s.
- Saved run state is not changed. A run saved by the previous release
resumes with the same runner identity, because the bytes are the same.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edit) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 19:56:57 -06:00
Devin FoleyandPaperclip cc9ff75cb8 fix(runtime): keep managed GitHub access across a controller restart (#21928)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs give agents managed GitHub access through `gh` and `git`
launchers on the provider's `PATH`
> - A provider process can keep running while the server restarts, and
the next controller adopts it
> - The detaching controller deleted the launcher directory, and the
adopting controller staged launchers in a new random directory
> - The running provider still names the old directory, so every later
`gh` or `git` call ran without credentials
> - This pull request keeps the launcher directory stable for one
provider session and task, and re-arms it after the restart
> - The benefit is that a long run keeps its GitHub identity across a
server deploy or restart

## Linked Issues or Issue Description

No public issue exists. This describes the bug in-PR, with the bug
template labels.

**What happened?**
An agent run on a Daytona sandbox used `gh` with managed GitHub access.
The server restarted during the run (a rolling deploy), and the
controller re-attached to the live provider process. After that, every
`gh` call failed with `You are not logged into any GitHub hosts`. `git
credential fill` returned no token. The agent reported that its token
had "expired" and used the unauthenticated public API for the rest of
the run.

**Steps to reproduce**
1. Start a native run with managed GitHub access on a remote sandbox.
2. Run a `gh` command in the run. It succeeds.
3. Restart the server while the provider process is still running. The
new controller re-attaches to the process.
4. Run a `gh` command again in the same run. It fails with "not logged
in", and no `Paperclip: GitHub ...` diagnostic appears.

**Expected behavior**
After the new controller adopts the provider, `gh` and `git` get
credentials from the new controller, as they did before the restart.

**Root cause**
- `createNativeGitHubAccess` staged the launchers in
`native-session-<random UUID>` and put that path in the provider's
`PATH`, `BASH_ENV`, and `GH_CONFIG_DIR`.
- On a restart detach, `githubAccess.stop()` ran `rm -rf` on that
directory (`detachReleasedWarmNativeSession`, and the run's final
`github_stop` while `nativeRunsDetachingForRestart` is set).
- The adopting controller made a new random directory. The provider
environment cannot change after spawn, so the provider never saw it.
- The shims were gone. `gh` resolved to the real binary with an empty
`GH_CONFIG_DIR`, so no Paperclip diagnostic appeared.

Refs #15354. That PR keeps ACPX sessions across launcher rotation in the
legacy adapter path. This PR covers the native runtime path.

## What Changed

- `createNativeGitHubAccess` takes an optional `launcherKey`. The
launcher directory is `native-github-<sha256(key)>`, so the same
provider session and task always get the same directory.
- The native session executor passes a key built from
`nativeSessionScopeKey` and the issue id.
- `prepareGitHubOperationLaunchers` takes an optional `broker`. It then
stages broker-file launchers and an owner-only `broker.json`. The
launchers already read `broker.json` on each operation in this mode
(`githubLauncherSource({ brokerFile: true })`). A restaged directory
therefore gives the running provider the new endpoint.
- Native access passes `broker` only for remote sandbox targets. A
sandbox provider is the one that outlives a controller restart. Local
launchers live in the shared host temp directory, so they keep the
capability in the provider environment only, as before.
- `stop({ preserveLaunchers: true })` keeps the launcher directory. A
restart detach of an active run and of a released warm session use this
option. A normal stop still removes the directory.
- A final stop does not remove a directory that a newer access in the
same process has already restaged.
- The callback bridge runtime stays in its own per-access directory. A
new bridge never shares a queue directory with an old bridge.
- `syncRemoteTextFiles` takes a `private` flag. It sets `umask 077` for
the whole staging script, so the upload copy, the partial file, and the
final file are never readable by group or other users. Local launcher
files are written to a temporary file and renamed, so a running launcher
never reads a partly written file.

## Verification

- New `native-github-access` test on a sandbox target: a live parent
process gets credentials, the access stops with `preserveLaunchers`, and
a new access with the same key re-arms the directory. The same parent
PID then gets the new controller's credentials, with empty stderr.
`broker.json` has mode `600`. The final stop removes the directory.
- New previous-release cutover test: a provider launched with the old
random-directory layout keeps working without managed GitHub after its
controller removes the directory. The new controller does not resolve
credentials for it. A provider that the new controller launches gets
credentials.
- New local regression: local launcher files do not contain the broker
capability, and there is no `broker.json`.
- New local restart test: a local provider adopted after a restart gets
a Paperclip diagnostic and no credentials. The new controller resolves
nothing for the old endpoint.
- New `sandbox-callback-bridge` test: staging a private file runs with
umask `077` and leaves the file at mode `600`.
- New `native-github-access` test: when a superseded access stops, its
successor's launchers stay usable. A different key gives a different
directory.
- New `native-session-executor` test: a run detached for restart while
active stops its GitHub access with `{ preserveLaunchers: true }`, and
creates it with a `launcherKey`. The existing overlapping-failure test
now asserts `{ preserveLaunchers: false }`.
- The new tests fail on the old behavior. I checked this by forcing
`brokerFile: false`, forcing `preserveLaunchers = false`, removing the
`umask 077` line, and writing `broker.json` for local targets.
- Passed: `native-github-access` (26), `native-session-executor` +
`github-operation-credentials` + `heartbeat-github-launchers` +
`native-stop-accounting` + `heartbeat-project-env` (1640 passed, 7
skipped), `native-runner-restart-recovery.integration` and
`heartbeat-native-restart-suspension`.
- Passed: adapter-utils `execution-target`, `github-launcher`,
`github-launcher-environment`, `sandbox-callback-bridge`; runner
`acpx/github-launcher`, `acpx/environment`,
`codex/codex-security-config`.
- `pnpm --filter @paperclipai/adapter-utils typecheck` and the server
`pnpm typecheck` pass.
- Not run locally: `native-warm-checkpoint-ownership.integration` needs
the compiled `paperclip-runnerd` binary. CI builds it.

## Previous-release story

- No persisted run, session, checkpoint, or database state changes. Only
the in-sandbox launcher layout changes.
- A provider launched by the previous release has a random
`native-session-*` launcher directory in its environment. The previous
release's controller removes that directory at the cutover restart. The
new controller cannot find it, so it does not re-arm it.
- That provider carries forward without managed GitHub until its next
spawn, which is the current behavior. The run is not refused and is not
retried. `gh` and `git` run without managed credentials, as they do
today after a restart. The cutover test covers this.
- A provider that the new release launches gets the stable directory and
survives later restarts.

## Risks

- On sandbox targets, the provider can now read the bridge capability in
`broker.json`, as well as in its environment. The file holds the same
per-access transport token that the environment already holds, not a run
token or a GitHub credential. The file and its staging copies are mode
`600`, and the broker still checks the active run and grants on each
request. Local targets write no `broker.json`.
- Local providers that outlive a restart still lose managed GitHub. This
is not a regression: master deletes the launcher directory at the
detach, so the old local provider also runs without credentials there.
With this PR, the local launchers stay in place and print a `Paperclip:
GitHub ...` diagnostic instead of a silent "not logged in". Re-arming
local providers needs a broker channel that other local runs under the
same user cannot read. That is out of scope here, because `broker.json`
in the shared temp directory was a security finding on this PR.
- On sandbox targets, `git` and `gh` children no longer inherit
`PAPERCLIP_API_URL`, `PAPERCLIP_API_KEY`, or
`PAPERCLIP_GITHUB_BRIDGE_TOKEN`. Broker-file mode removes them, as it
already did in its existing tests.
- After a restart detach, a launcher directory stays in the workspace
runtime directory. If no controller adopts the provider, that directory
stays until a later access with the same key stops, or until the
workspace is removed. It contains no GitHub credential.
- Between the detach and the adoption, `gh` gets a transport error from
the old endpoint. It then prints `broker_transport_unavailable` and
continues without credentials, as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs). Standard context window.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-16 12:13:04 -08:00
DottaandClaude Opus 5.5 d149e30323 fix(native): stop a finished warm owner's retained Runner when suspend is refused, and release a suspended Runner whose process left (#21853)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm agent session keeps its Runner alive in a sandbox between
turns. The server closes an idle session after 5 minutes, and closes it
again for an environment delete, idle sleep, or a restart.
> - In a zero-downtime release-swap test, two warm closes failed and did
not recover.
> - In the first case, the Runner of a finished run refused
`runner.suspend`. The server kept the Runner alive and the session
registered. An environment delete returned 409 thirty-five times, and
only an outside sandbox destroy released the session. A live Runner and
a paid sandbox were held, and idle sleep was blocked.
> - In the second case, the Runner durably suspended and exited, but its
process check returned exit 44 (`process_identity_changed`) on every
attempt. Exit 44 also covers "a live process without `--runner-id`", so
no proof could release the owner.
> - Also, the bounded idle retries from #21782 did not run for the first
case.
> - This pull request stops the retained Runner of a finished owner with
the verified port-holder retirement, releases the owner only when a
fresh check reports `process_not_running`, splits exit 44, accepts a
durably suspended Runner with no live process that carries its runner
id, and keeps the idle retries alive after a failed sweep.
> - The benefit is that a finished warm session can no longer hold a
live Runner and a sandbox until someone destroys the sandbox.

## Linked Issues or Issue Description

Refs #21782
Refs #21835

**What happened?**
Release-swap gate wave, two warm-session close failures:

1. A finished run's idle close failed with
`native_session_close_unrecoverable`: `runner.suspend` failed,
`runnerRetained: true`, Runner lifecycle `ready`. Only close attempt 1
was logged. An environment delete with `forceStopUnsettled` could not
discard the session, because `closeWarmNativeSession` discards only with
a Launcher task owner or a suspended Runner. The delete got 409
thirty-five times. The session was released only after an outside
sandbox destroy.
2. A Runner's `runner-state.json` showed lifecycle `suspended`, with
drain and suspend `completed`. The process check exited 44 on every
attempt, and the Runner process was gone later while the sandbox stayed
started.

**Expected behavior**
A finished owner whose Runner refused suspend is stopped and released,
and a failed stop keeps it held. A durably suspended, settled Runner
with no live process that carries its runner id releases its owner. A
bare `process_identity_changed` never releases an owner.

**Steps to reproduce**
1. Keep a warm sandbox session idle after its run succeeds.
2. Make its close fail with `NativeSessionCloseUnrecoverableError` with
`runnerRetained: true`.
3. Call `closeWarmNativeSessionsForEnvironment({ forceStopUnsettled:
true })`. On master the result is `failed: 1`, every time.

**Why the retries did not run**
- The transport memoizes `close()`. A second close returns the same
rejected promise and does no sandbox work. The 27 environment-delete
failures have identical timings (`totalMs: 6055`), and the other
Runner's daemon log shows no sandbox commands during its three retries.
So a retry can only succeed through a fresh proof, not through the
close.
- `closeIdleWarmNativeSessions` (environment delete, idle sleep,
restart) clears the entry's idle timer, which is also the pending retry
timer, and never re-arms it after a failed close. The test "keeps
retrying a failed idle warm close after a failed environment sweep
cleared its retry timer" is red on master for this reason.
- The log shows one more `close()` request on the same transport in the
same second as attempt 1, and no attempt 2. Every path that closes an
idle owner from outside the timer clears that timer. The log does not
show which caller it was, so this part is not proven.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`:
- A warm entry keeps its sandbox runner and remote directory when the
run used a sandbox that the server reaches directly (no Launcher task).
  - `warmCloseFailureProvesOwnerGone` adds two proofs:
- The close failed with `NativeSessionCloseUnrecoverableError`,
`runnerRetained: true` and not suspended. The server stops the session's
own Runner with `retireSandboxRunnerPortHolders` (owner run must be
`succeeded` or `cancelled`, same agent, idle state; TERM, then KILL). It
releases the owner without a checkpoint only if a fresh process check
then exits 45 (`process_not_running`). A stop that cannot be confirmed
keeps the owner.
- The close failed with the new `process_runner_id_missing`. A fresh
sandbox inspection must show lifecycle `suspended`, settled state (empty
outbox, no pending terminal delivery, provider cleanup, or warm
transition), matching run, session and runner ids, and no process that
carries `--runner-id <id>`.
- Both the idle-timer failure path and `closeIdleWarmNativeSessions`
(environment delete with `forceStopUnsettled`, idle sleep, restart) use
these proofs.
- The first Runner-check or refused-suspend close failure for a sandbox
owner logs one evidence line: stage, reason, Runner lifecycle, suspend
command status, proof window, and an inspection of the sandbox (durable
lifecycle, settled, identity match, marker pid, and pid/state of each
`paperclip-runnerd` or runner-id process). It logs no nonce and no
command line.
- A sweep that fails to close an owner whose idle close already failed
re-arms the bounded idle retry.
-
`server/src/services/native-runtime/remote-runner-recovery-diagnostics.ts`:
`REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT` exits 48
(`process_runner_id_missing`) for a live process whose command line
lacks the runner id. Exit 44 now means only an identity file mismatch.
- `server/src/services/native-runtime/sandbox-runner-port-holders.ts`:
`retireSandboxRunnerPortHolders` takes `only: { sessionDirectory, runId
}` to retire one session's Runner. New `sandboxSessionRunnerExited` and
`inspectSandboxSessionRunner` (a read-only `node` script).

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts
src/services/native-runtime/sandbox-runner-port-holders.test.ts
src/services/native-runtime/sandbox-runner-port-holders.integration.test.ts
src/services/runner-lost-mid-turn-continuation.test.ts
src/services/native-runtime/native-warm-checkpoint-ownership.integration.test.ts`
- Red on master (master sources with these tests): "stops the retained
Runner of a finished warm owner that refused suspend ..." returns
`failed: 1`. "keeps retrying a failed idle warm close after a failed
environment sweep ..." sees 2 closes, not 3. "releases at controller
restart a warm owner whose Runner durably suspended ..." returns
`failed`.
- Negative tests: a Runner that survives TERM and KILL stays `failed: 1`
with no exit check. A run that is still `running` is never signalled. A
live process that still carries the runner id, and a bare
`process_identity_changed`, keep the owner.
- Script tests: the process check exits 48 for a live process without
`--runner-id`, 0 with it, and 44 for a changed marker. The inspection
script reports state, marker pid and runner processes, never the nonce
or a command line, and fails closed without `/proc` or through a
symlink. The port-holder integration test stops only the named session's
finished Runner (a real local process) and then proves it exited.
- `pnpm verify` exited 2 (opaque setup dependency), so focused checks
and required CI give the coverage. `npx tsc --noEmit -p server`: no
errors.

## Risks

- A finished owner's retained Runner is now stopped when its suspend was
refused. The session is released without a new checkpoint. The last
saved checkpoint stays the recovery source, which is the same as an
environment delete today.
- The stop uses the same rules as a successor's port-holder retirement:
finished owner run, same agent, idle Runner state, and a verified
process identity. Any doubt keeps the owner.
- The stop can take up to about 20 s (TERM and KILL waits) inside a
sweep, while the entry stays fenced.
- The first close failure of a sandbox owner runs one read-only
inspection command (5 s limit).
- Previous-release story: no persisted state or launch identity changes.
The scripts are sent with each command, so a server always runs its own
exit-code table. Exit 44 from the previous release maps to
`process_identity_changed`, which still never releases an owner.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code
with tool use (shell, file edit) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 20:54:47 -06:00
DottaandClaude Opus 5.5 03309f1650 fix(native): release an idle warm owner whose sandbox Runner port a later run took over (#21835)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm agent session keeps its Runner alive in a sandbox between
turns. The server closes idle sessions, and closes them again before it
exits.
> - A sandbox has one Runner port. Before a new run launches its Runner
on a reused sandbox, it retires every other Runner on that port.
> - In a zero-downtime release-swap test, a run finished on the outgoing
server and stayed warm. 77 ms later, a follow-up run started on the
incoming server. It reused the same sandbox, started a fresh provider
session, retired the old Runner, and launched its own Runner on the same
port.
> - The outgoing server's idle close could not suspend the old Runner.
Its checkpoint was correctly refused with
`native_runner_checkpoint_owner_lost`. The idle timer retried 3 times
and then kept the owner registered. That phantom owner blocked idle
sleep for 4.8 minutes, and the shutdown close failed again.
> - The existing proofs that a warm owner is gone (#21782) do not count
"a later run launched its own Runner on this sandbox".
> - This pull request adds that proof. When it holds, the server forgets
the owner without a checkpoint. The checkpoint fence does not change.
> - The benefit is that a superseded warm owner no longer blocks idle
sleep or fails a clean shutdown.

## Linked Issues or Issue Description

Refs #21627
Refs #21782

**What happened?**
An idle warm owner finished on the outgoing server. A follow-up run on
the incoming server reused the same sandbox and launched its own Runner
on the sandbox's single Runner port. The owner's idle close failed with
`native_runner_checkpoint_owner_lost` 4 times. The owner stayed
registered, held the idle-work flag, and blocked idle sleep. At SIGTERM
the restart sweep logged "idle warm native sessions did not close" with
"runner did not durably suspend before checkpoint". No task was
affected.

**Expected behavior**
A failed idle close releases its owner when a later run proves that the
owner's Runner is gone. The checkpoint stays refused while the successor
is live.

**Steps to reproduce**
1. Finish a warm run on a sandbox and keep it idle.
2. Start a successor run for the same task on another server. Give it a
lease on the same sandbox, its own `native.process_identity_recorded`
event and its own `session.started` event.
3. Call `closeIdleWarmNativeSessionsForRestart()`. On master the result
is `closed: 0, failed: 1`, and `readNativeSessionIdleWork()` stays
`present`.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`: new
`warmRunnerPortTakenBySuccessor`, used by
`warmCloseFailureProvesOwnerGone` after the existing sandbox-termination
proof. It holds only when another run has all of these:
- an `environment_leases` row with the same environment and
`providerLeaseId` as the owner;
- its own `native.process_identity_recorded` event (`source_event_id is
null`, so an inherited identity does not count);
- a `session.started` event from its own Runner instance
(`source_instance_id` equals the run's `runner_instance_id`);
  - both events created after the owner run's `finishedAt`.
- When the proof holds, the idle-timer path and the controller-restart
path log "warm native session's Runner or sandbox is gone; released its
owner" and forget the owner without a checkpoint. Both paths already
call `warmCloseFailureProvesOwnerGone`.
- `validateNativeRunnerCheckpointOwner` does not change. A checkpoint
still cannot land under a live successor.
- Cost: the proof runs only after a failed close. It does one
primary-key read of the owner run and one query on
`environment_leases.provider_lease_id` (indexed), joined to
`heartbeat_runs` by primary key, with `heartbeat_run_events` read
through `(company_id, run_id)`.
- New real-database test file
`server/src/services/native-runtime/native-warm-successor-runner-release.integration.test.ts`.
- `native-session-executor.test.ts`: one environment-delete test counts
the extra proof read when the first proof does not hold.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-warm-successor-runner-release.integration.test.ts`:
6 passed.
- "releases an idle warm owner whose checkpoint is refused because a
successor on another server launched its own Runner on the same sandbox:
own_runner": the close is refused with
`native_runner_checkpoint_owner_lost`, the result is `closed: 1, failed:
0`, the idle-work flag is `none`, the owner has no checkpoint row, and
`assertNativeRunnerCheckpointOwner` still throws
`native_runner_checkpoint_owner_lost`.
- These stay failed and keep the owner registered: the successor is on
another sandbox; the successor has only
`native.process_start_requested`; the successor has only inherited
events (`source_event_id` set); the successor's events come before the
owner finished.
- "releases a superseded idle warm owner from its idle timer without
writing a checkpoint": the idle timer releases the owner after the first
refused close.
- Red on master (test file only, fix reverted): `own_runner` fails with
`closed: 0, failed: 1`; the idle-timer test fails with the idle-work
flag still `present`. The 4 negative cases pass on master and with the
fix.
- `cd server && npx vitest run
src/services/native-runtime/native-warm-checkpoint-ownership.integration.test.ts`:
9 passed.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts`: all
passed.
- `cd server && npx tsc --noEmit -p .`: no errors.
- `pnpm verify`: see the comment below for the result.
- Not run: a restart-chaos wave. The release-swap timing is reproduced
in the database test, not on a live sandbox.

## Risks

- A superseded owner is now released without a checkpoint. Its old
provider conversation cannot be resumed. The successor had already
started a different conversation and holds the task, so no one would
resume the old one.
- The proof needs the successor's own Runner identity and own session
start after the owner finished, on the same sandbox lease. A successor
that reused the owner's Runner, only requested a start, or recorded its
events earlier does not count. In those cases the close fails as before.
- The proof adds two small indexed reads after a failed close only. The
normal close path does not change.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code
with tool use (shell, file edit) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 17:49:01 -06:00
DottaandClaude Opus 5.5 c6cdd57f7a perf(native-runtime): reuse a same-content image provider pack across builds (#21784)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote agent run needs the provider pack (Node, OpenCode and the
Runner bridge) inside its sandbox before the provider can start
> - The server reuses a pack that is installed in the sandbox image only
when its full manifest equals the server's own manifest
> - The manifest includes `runnerSourceRevision`, which is the commit
that built the server, so an image pack fails this check on every later
server build, also when every file in the pack is the same
> - The run then uploads the whole pack again, which cost about 15 s per
task in a recent load test
> - This pull request accepts an image pack whose sealed content equals
the expected pack and that differs only in build provenance, and caches
verified archive hashes in the server process
> - The benefit is that a sandbox image stays usable across server
releases that do not change the pack, and the server does less hashing
on its main thread

## Linked Issues or Issue Description

Refs #21781 (merged), which names the failed check on the run spans.

**What existing behavior does this improve?**
Selection of the remote provider pack for a new sandbox
(`provider_pack.select`).

**Subsystem affected**
Server: native runtime, remote provider pack verification and upload.

**Current behavior**
`provider_pack.verify_preinstalled` compares the complete manifest of
the image pack with the server's manifest. Any difference fails,
including `runnerSourceRevision` and `runnerInputsDigest`, which only
record where the pack was built. The run then uploads the full pack. The
server also hashes the same archive or pack directory again for every
task.

**Proposed behavior**
When the run is not bound to a saved pack and the operator did not
select a custom pack, the remote verification accepts an image pack when
(1) its own digest seals its payload, and (2) its schema and payload
equal the expected pack after the two provenance fields are removed.
Every artifact, the `dist` tree, the candidate trees, the package
versions, the Node version, the target platform and the OpenCode version
are still checked in the sandbox, against the same values. The server
checks the reported manifest again and selects it through the pack
authority, as the compatibility path already does. An upload still
happens for different content, a corrupted or partial pack, or no pack.
The server also remembers a verified transport archive (by path, inode,
size and times), so it does not hash it again for the next task.

**Reason and benefit**
In the harness (fake sandbox, 150 ms per command, 300 ms + 10 MiB/s per
transfer, 24 MiB pack), the cross-build case goes from 4.1 s and 24 MiB
uploaded to 1.1 s and 0 bytes. Other cases do not change.

**Breaking changes**
None. A saved or reattached session, and an operator-selected custom
pack, still require the exact pack identity.

## What Changed

- `remote-provider-pack-verification.ts`: new `acceptProvenance` option.
The remote script accepts a sealed pack with the same content and prints
its manifest. The server checks that manifest (`sameContentManifest`)
and returns it. A forged or different report fails with
`installed_manifest_invalid`.
- `native-session-executor.ts` (`prepareRemoteProviderPackOnTarget`):
staged and preinstalled verification use the option when
`acceptProvenanceDifferences` is set. The function returns the selected
manifest. The run caller sets the option only through
`providerPackProvenanceReuseAllowed` (not retained, no explicit custom
pack, selectable authority), and selects a different manifest through
the pack authority.
- `provider-pack-layers.ts`: `verifyArchive` skips the hash for an
archive it already verified with the same file identity.
- `remote-provider-pack.ts`: `providerPackProvenanceReuseAllowed` gate,
with tests for each case.
- `provider-pack-selection-harness.test.ts`: new harness. Uploads go
through the workspace confinement check of the Daytona sync guard. It
runs the real remote verification in local processes, with modelled
sandbox latency and transfer speed, for five cases, before and after.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/provider-pack-selection-harness.test.ts
src/services/native-runtime/provider-pack-archive.test.ts
src/services/native-runtime/remote-provider-pack-verification.test.ts
src/services/native-runtime/remote-provider-pack.test.ts
src/services/native-runtime/retained-provider-pack-source.test.ts
src/__tests__/kimi-connection-probe.test.ts` passes.
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-artifact-preparation.test.ts
src/services/native-runtime/remote-provider-pack-authority.test.ts`
passes (1541 passed, 7 skipped).
- Red/green: with the source changes reverted, the harness case
`same_content_other_build` and the archive cache test fail. With the
changes, they pass.
- Harness output (load average about 9 on a shared machine; modelled
time = wall time + modelled sandbox latency):

| Case | Before | After |
| --- | --- | --- |
| Exact image pack | preinstalled, 1.1 s, 0 MiB | preinstalled, 1.0 s, 0
MiB |
| Same content, other build | uploaded, 4.1 s, 24 MiB | preinstalled,
1.1 s, 0 MiB |
| Different content | uploaded, 4.1 s, 24 MiB | uploaded, 4.0 s, 24 MiB
|
| Corrupted image pack | uploaded, 4.0 s, 24 MiB | uploaded, 4.1 s, 24
MiB |
| No image pack | uploaded, 3.7 s, 24 MiB | uploaded, 3.8 s, 24 MiB |

- The pack step runs no database statements.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- This changes which provider pack the server accepts. It accepts a
different manifest digest only when all content fields are equal and
every file is hashed in the sandbox, and only for a run that is not
bound to a saved pack. The selected manifest goes through the existing
pack authority, so launch profiles and receipts bind the installed
pack's own identity.
- The archive hash cache trusts each archive file's identity (inode,
size, times) on the server. The sandbox still checks the uploaded bundle
digest and verifies every uploaded file, so a changed local file still
fails closed.
- This does not help when the image pack has other content (for example
a different contract digest). See the hand-off notes for the image
build.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 12:07:56 -06:00
DottaandClaude Opus 5.5 df9914dbd9 fix(native): let the outgoing server checkpoint its idle warm session while the incoming follow-up waits for it (#21791)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A deploy swaps servers with an overlap. A warm agent session stays
with the outgoing server that ran its last turn.
> - Since #21774, a follow-up that the incoming server claims during the
overlap waits (`native_controller_recovery_waiting_for_owner`) and
checks again every 5 s.
> - About 1 s after the traffic switch (#21627), the outgoing server
tries to save and close its idle warm session. The checkpoint fence
refuses that save with `native_runner_checkpoint_owner_lost`. The task's
current run is the parked follow-up. That run has its own finalization
row under another controller boot, and the fence does not accept that
kind of successor.
> - The close retry from #21782 fails the same way each time. So the
follow-up waits until the outgoing server exits, or until the
predecessor's finish time plus the idle window plus 2 min. In the CHA-5
case that was about 225 s, and the worst case is about 425 s. Speed is a
requirement.
> - This pull request records which owner a parked follow-up waits on,
and lets the fence accept the outgoing owner's final save when the
follow-up is parked on exactly that owner and has not started.
> - The benefit is that the outgoing server saves and closes the session
on its first attempt, and the follow-up continues at its next 5 s check.

## Linked Issues or Issue Description

Refs #21774, #21782, #21697, #21627, #21279

**What happened?**

During a release-swap overlap, a warm follow-up was claimed by the
incoming server while the outgoing server still owned the idle warm
session. The follow-up waited correctly. The outgoing server's idle
close failed with `native_runner_checkpoint_owner_lost` on every
attempt. The wait ended only when the outgoing server exited, about 225
s after the follow-up was created.

**Expected behavior**

The outgoing server saves and closes the session while the follow-up
waits. The follow-up sees the saved session and continues within one 5 s
check.

**Steps to reproduce**

1. Run a warm native session on a remote sandbox and let a run finish on
server A.
2. Start server B with an overlap and switch traffic to it.
3. Post a follow-up comment so that B claims the follow-up while A is
still alive.
4. A's idle close fails with `native_runner_checkpoint_owner_lost`, and
the follow-up waits until A exits.

**Paperclip version or commit**

master `d14a3b88b1`.

## What Changed

- `NativeControllerRecoveryWaitingError` carries the owner that the
follow-up waits on: the predecessor run and its controller boot.
`waitForLiveWarmSessionOwner` and the live-owner wait in
`inspectRemoteWarmSessionRecovery` set it.
- `deferNativeControllerRecovery` stores that owner in the finalization
`failureDetail.waitingForOwner`. The executor passes it through.
- `validateNativeRunnerCheckpointOwner` accepts one more successor kind:
a parked follow-up waiting on this exact owner. All of these must be
true, or the fence still refuses:
- The checkpoint holds an accepted result and has no new work (unchanged
rule).
- The follow-up finalization is `retryable_failure` with
`native_controller_recovery_waiting_for_owner`, `attempt = 0`, no
result, and no lease owner (no claim is running it).
  - `waitingForOwner` names this owner's run and controller boot.
- The follow-up run is a running native run for the same company and
task, on the same native session as the owner's run.
- The follow-up has no process fields, no own process-start events, and
no checkpoint rows.
- The fence locks the follow-up's finalization row (`for share`), so a
resumption claim and the save cannot interleave.
- Tests in `native-controller-recovery.integration.test.ts` (real
Postgres):
- The outgoing server saves and closes its idle warm session while the
incoming follow-up waits, and the follow-up resumes within one recheck.
- Negative cases: the save is still refused when the follow-up has its
own process, its own process-start event, its own checkpoint, or a live
resumption claim, or when it waits on a different owner boot or run.

## Verification

- Red on master: with the `native-runner-checkpoint.ts` change removed,
the scenario test fails with `native_runner_checkpoint_owner_lost` at
the outgoing server's first close attempt. The follow-up keeps waiting.
Green with the change.
- The negative cases pass with and without the change.
- Measured in the test scenario (warm idle window 600 s):
- Before: the follow-up waits until the outgoing server exits, or about
720 s (finish + 600 s idle window + 120 s margin).
- After: the outgoing save and close succeed on the first attempt. The
follow-up is claimable and its wait resolves at `retryAt`, 4998 ms after
it parked.
- `vitest run` of `native-controller-recovery.integration.test.ts`,
`native-runner-restart-recovery.integration.test.ts` and
`native-harness-backup-storage.integration.test.ts`: 349 passed, 14
skipped.
- `vitest run` of `native-session-executor.test.ts`,
`remote-warm-session-recovery.test.ts`,
`native-runner-checkpoint-save.test.ts`,
`native-runner-checkpoint-retirement.test.ts` and
`native-runner-checkpoint-authority.integration.test.ts`: 1566 passed, 7
skipped.
- `tsc --noEmit` for `server` passes.
- `pnpm verify` exits 2 (opaque setup dependency). Required CI supplies
the broad coverage.
- Not run: a release-swap wave. The fix is not yet measured on a live
overlap.

## Risks

- The fence is wider. It accepts only a final save with an accepted
result and no new work, and only while the current run is parked on this
exact owner and has not started. A follow-up that has a claim, a
process, process-start events or checkpoint rows still causes a refusal.
- A follow-up that a server parked before this change has no
`waitingForOwner`. The fence refuses its owner's save as before. At its
next recheck, the follow-up parks again with the owner recorded.
- While a resumption claim runs the follow-up's recheck (a short window
every 5 s), the save is refused. The close retry from #21782 then tries
again.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 11:02:31 -06:00
DottaandClaude Opus 5.5 cec8c2b3f4 perf(recovery): name the run or task in run writers' recovery hints instead of the company (#21765)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server runs issue recovery queues on one Node main thread. The
`stranded-assignments` and `assigned-work-invariant` queues check
assigned tasks that have no live run. One stranded check costs about 19
database statements.
> - #21552 scoped run hints to the run's task graph. #21729 made the
5-minute backstop re-check only changed tasks. A hint with only a
company id still makes the queue re-check every open task of the
company, 25 per pass.
> - Under load on staging (7 native runs), the stranded check ran about
6.7 times per second, and its statement bundle was more than a third of
all statements. An activity fixture shows the cause: ordinary run
writers send a company-only hint once per run, and one company hint
costs about 850 checks. At 1,000 agents this work never stops.
> - This pull request makes those writers name the run (or the task)
they already know. A run that has no task now resolves to no tasks
instead of the whole company. The changed-tasks backstop no longer
counts an agent's own run heartbeat as a change to every task of that
agent.
> - The benefit: with 7 runs a minute plus one taskless run, stranded
checks drop from 18.4 to 1.0 per second and stranded-check statements
from 344 to 7 per second. Real company-wide changes (pause lifted,
budget, company status, tree holds) keep the full company pass.

## Linked Issues or Issue Description

Refs #21729, #21552, #21676, #21724

**What existing behavior does this improve?**

The issue recovery task queues (`stranded-assignments`,
`assigned-work-invariant`, `resolved-dependencies`, `stale-issue-locks`)
and the writers that wake them.

**Subsystem affected**

Server: issue recovery background work and the run lifecycle writers
that publish recovery hints.

**Current behavior**

Many run lifecycle writers know the run but publish a company-only hint,
for example `notifyExecutionRunWork(tx, { companyId })` in the native
finalizer and projection, `settleNativeTerminalWake`, the claimed-wake
settlement, and an environment lease release of a run without a task. A
queued run without a task publishes a company hint. A graph hint for a
run without a task resolves to "whole company". A comment sends a
company hint to `assigned-work-invariant` and `resolved-dependencies`.
Each of these re-checks all open tasks of the company. Also, the
changed-tasks backstop treats any change to the assignee agent's row as
a change to all of that agent's tasks, and every run start and finish
writes that row.

**Proposed behavior**

A writer that knows the run sends the run's graph hint. A writer that
knows the task sends the task hint. A run without a task affects no task
queue entry. An agent row change that is only the agent's own run
heartbeat does not mark its other tasks changed. Company-wide facts keep
the company hint.

**Reason and benefit**

The work is now proportional to activity, not to (activity x history).
See Verification.

## What Changed

- `server/src/services/issue-recovery-notifications.ts`:
- `notifyIssueRecovery` with a task sends the task hint to all four task
queues (before: only to `stranded-assignments` and `stale-issue-locks`;
`assigned-work-invariant` and `resolved-dependencies` got the company
hint). Only comments (`issue-comment-writes.ts`) call it with a task.
- `resolveIssueGraph` returns `[]` when a run-only graph has no task. A
missing task seed still returns `null` (whole company).
- `server/src/services/recovery/work.ts`: a load for an empty graph
returns no rows. The `assigned-work-invariant` queue emits the same
`check` debug event as `stranded-assignments` (with
`PAPERCLIP_DEBUG_WORK_SIGNALS=assigned-work-invariant`), with the load
reason.
- `server/src/services/recovery/changed-tasks.ts`: the assignee-agent
part of the changed-tasks condition skips an agent whose last write was
its run heartbeat (`last_heartbeat_at = updated_at`). A resume, pause or
configuration write (no heartbeat stamp) still counts.
- `server/src/services/heartbeat/queue.ts`: a queued run without a task
gets its id before the insert and publishes a run hint, not a company
hint. The daily-cap comment release publishes the run's graph hint.
- `server/src/services/environments.ts`, `work-intent-outbox.ts`,
`chat-inbound-wake-notice-notifications.ts`: a lease release passes its
run id. `promoteDeferredWorkIntents` scopes `lock_released` with only a
run to that run's graph. `notifyChatInboundWakeNotices` accepts an
optional `runId` for its settlement hint.
- Native and recovery run writers now pass the run id they already have
to `notifyExecutionRunWork`, `writeExecutionControl` or
`notifyExternalConversationSettlement`: `native-run-finalizer.ts` (3),
`native-run-projection.ts` (2), `native-session-executor.ts` (3),
`wake-request-claimed-settlement.ts`, `native-superseded-retry.ts`,
`native-safe-replacement.ts`, `native-restart-recovery.ts` (2),
`native-failed-workspace-recovery.ts`,
`native-recovery-preparation-failure.ts`,
`execution-control-reconciliation.ts`,
`execution-recovery-resolution.ts` (2). These are the native equivalents
of the legacy run paths in `heartbeat/queue.ts`, which already pass the
run id.
- Tests:
- `server/src/__tests__/issue-recovery-activity-hints.test.ts` (new):
staging-shaped company (780 in-review tasks with succeeded native runs,
40 in progress, 30 to-do, 50 agents) on embedded PostgreSQL, the real
recovery, dispatch and retry-cancellation workers, a fake clock, and
real process-adapter runs. Each run is woken by a user comment through
`heartbeat.wakeup`, starts, calls back to post an agent comment and move
the task to `in_review`, and finishes. One (or five) taskless runs a
minute run beside them. It counts checks by queue and reason, task-queue
signals by caller, and statements by owner. It asserts the workload
(runs started and succeeded, agent comments, run events, zero agent
write failures, at most 10% failed runs), then no company check and no
company-only task-queue signal. In CI it runs the 7-runs-a-minute
workload for 4 minutes (about 1 minute of real time). With
`RECOVERY_ACTIVITY_REPORT=<file>` it runs both workloads at full length,
attributes statements by stack, and writes the numbers below to that
file.
- `server/src/__tests__/issue-recovery-deadlines.test.ts`: pickup tests
for each narrowed writer (see Verification).

### Publishers of company-only (or `*`) hints to the recovery task
queues

"Per" is how often the writer fires on a busy instance. "Known" says
whether the run or task is known at the call site.

Narrowed in this pull request:

| Call site | Event | Per | Known | Now |
| --- | --- | --- | --- | --- |
| `heartbeat/queue.ts:5396` `enqueueWakeup` | queue a run without a task
(timer, manual, chat without a task) | each taskless run | run (id
generated) | run graph (no tasks) |
| `environments.ts:1780` `releaseLease` ->
`notifyChatInboundWakeNotices` | environment lease released | each run
without a task | run | run graph |
| `environments.ts:1792` `releaseLease` ->
`promoteDeferredWorkIntents(lock_released)` | lease released | each run
without a task | run | run graph |
| `environments.ts:1934` `releaseLeasesForRun` | leases of a run
released | each run without a task | run | run graph |
| `issue-comment-writes.ts:12` (`assigned-work-invariant`,
`resolved-dependencies` only) | comment | each comment | task | task |
| `native-runtime/native-run-projection.ts:54`
`settleNativeTerminalWake` | native run's wake claim settled | each
native run finish | run | run graph |
| `native-runtime/native-run-projection.ts:177` | native run status
projected | each native run finish | run | run graph |
| `native-runtime/native-run-finalizer.ts:408`, `:629`, `:2080` | native
finalization terminal / failure / projection | each native run finish |
run | run graph |
| `native-runtime/native-session-executor.ts:13018`, `:13085`, `:13284`
| native execution control deadline, cancelled retry, pre-result
settlement | each native run end | run | run graph |
| `recovery/wake-request-claimed-settlement.ts:67` | claimed wake
settled after the run | each run whose claim is reconciled | run | run
graph |
| `heartbeat/queue.ts:590` | daily-cap comment key release | each run
cancelled at the daily cap | run | run graph |
| `native-runtime/native-superseded-retry.ts:42`,
`native-safe-replacement.ts:613`, `native-restart-recovery.ts:1033`,
`:1935`, `native-failed-workspace-recovery.ts:164`,
`native-recovery-preparation-failure.ts:30`,
`execution-control-reconciliation.ts:270`,
`execution-recovery-resolution.ts:830`, `:965` | native recovery
terminal writes | each recovered run | run | run graph |
| `resolveIssueGraph` for any run-only hint of a run without a task (for
example `setWakeupStatus`, `run-lifecycle.ts:1570`) | run lifecycle |
each taskless run event | run | no tasks |

Left with the company hint (company-wide fact, or the graph is not
known):

| Call site | Event | Per | Why kept |
| --- | --- | --- | --- |
| `budgets.ts:371`, `:386`, `:429`, `:437`, `:454` | budget policy
changed, scope paused or resumed | operator or budget event |
eligibility of every task can change |
| `work-intent-outbox.ts:79` with `pause_lifted`, `tree_hold_released`,
`scheduling_resumed`, `reconciler_backstop`, and `accounting_settled`
with an active hard stop | hold release | operator event;
`accounting_settled` once per run only with a hard-stop policy | can
admit any task; guarded by "keeps the company pass for hold releases
that can admit any task" |
| `companies.ts:77`, `:350`, `:353`, `:361`, `:512`, `:522` | company
created, paused, resumed, archived | operator | company-wide |
| `agent-lifecycle.ts:65`, `:72`, `agent-lifecycle-driver.ts:29`,
`agent-configuration.ts:130`, `:159`, `agent-deletion.ts:61` | agent
paused, resumed, configured, deleted | operator | can change many tasks,
and other agents' review paths |
| `access.ts:570`, `:752`, `:912`, `:926`, `:960`, `:975`, `:1175`,
`board-claim.ts:129`, `:142`, `middleware/auth.ts:706` | membership and
authority changes | operator, first sign-in | company authority |
| `chat-channels/endpoints.ts:461`, `identity-links.ts:323`,
`resource-reconciliation.ts:290`, `slack/setup-services.ts:50`,
`chat-github-management.ts:182`, `:352`, `chat-work-notifications.ts:50`
| chat authority changes | operator | company authority |
| `instance-settings.ts:666`, `:669`, `:680` (`*`) | instance setting
changed | operator | instance-wide |
| `issue-tree-control.ts:809`, `:889`, `:1236`, `:1339` | tree hold
created or released | operator | many tasks |
| `company-deletion.ts:83`, `company-portability.ts:6560` | company
delete, import | rare | many tasks |
| `issues.ts:8015` (blockers replaced), `:11165` (task created),
`:12685`, `:12699` (task deleted), `:12991`, `:14618` | task graph
changes | per task create or blocker edit | graph moves; #21552 kept
these |
| `documents.ts:615`, `:673`, `:757`, `:808`, `:860` | document delete
and restore | rare | #21552 kept these |
| `task-watchdogs.ts:854`, `:1841` | watchdog changes | rare | #21552
kept these |
| `system-recovery.ts:17`, `:70`, `task-retry.ts:294`,
`workspace-recovery-capture-policy.ts:59`,
`workspace-recovery-stop-proof.ts:122` | system and workspace recovery |
per recovery | not shown to stay in one graph |
| `agent-conversations.ts:607`,
`chat-channels/telegram/attachment-failure.ts:83`,
`native-runtime/chat-attachment-reuse.ts:1754`,
`native-runner-file-handoff.ts:1317` | conversation and attachment paths
| per failure | not shown to stay in one graph |
| `wake-queue/adapters/postgres.ts:376`, `:410`, `:555`, `:1169`,
`routes/issues.ts:14552`, `issues.ts:12070`, `:12171`,
`heartbeat/run-control.ts:1128`, `:1770`, `projects.ts:100`, `:1051`,
`environments.ts:1617` (`notifyChatInboundWakeNotices` without a task) |
saved notice and wake changes | per manual wake or project change |
company admission state; not reached by run activity in the fixture |
| `heartbeat/run-state.ts:1746`,
`native-harness-backup-expiration.ts:356`, `:526`,
`explicit-native-continuation.ts:705` | session goal clear, backup
expiry, explicit continuation | rare | no run id at the call site |
| `fast-responses.ts:201`, `:559`, `:672`, `:1053`,
`chat-github-response-comments.ts:62`, `:229`, `:440`,
`chat-github-reviews.ts:1063` (`notifyChatResponseAvailability` with a
delivery only) | chat response availability | per chat response | no
task at the call site; not measured here (chat paths are not in the
fixture) |

Per-run-event path: appending run events, saving native checkpoints,
renewing leases and streaming deltas publish no recovery task-queue
hint. `heartbeat-run-events.ts` reaches only live-run recovery queues
(`native.process_*`, `turn.*` events), chat progress, response
presentation, and `notifyNativeStatusReassessment` for control-plane
evidence events only. Checkpoints and lease renewals reach
`notifyLiveRunRecovery`, which does not include the task queues. A run's
start and finish publish its graph hint from several writers (about 20
task-queue signals per legacy run in the fixture, for example
`notifyChatRunMilestones` in `run-identity.ts` activation,
`claimQueuedRun`, `run-completion.ts`, `setWakeupStatus`). They coalesce
into about 3 checks of the run's own task per run. That is not one hint,
but it is bounded by the run's graph. This pull request does not change
it.

## Verification

- Activity fixture (`issue-recovery-activity-hints.test.ts`), checks per
second by queue and reason, and statements per second. Each workload
minute is exactly 60 s of fake-clock time, and the rates use the elapsed
fake-clock time. "Before" is this branch's test against the
`origin/master` service files. Both versions did the same work (7
runs/min: 150 runs started, 143 to 145 succeeded, 133 to 135 agent
comments, about 1,280 run events; 50 runs/min: 629 to 630 started, 620
to 622 succeeded, about 590 agent comments, about 5,550 run events). The
test now asserts this workload and zero agent write failures before it
accepts the counts.

  | | master | this branch |
  | --- | --- | --- |
| 7 runs/min + 1 taskless run/min (600 s), stranded checks/s | 18.30
company + 0.07 hint | 0.91 hint + 0.06 backstop, 0 company |
  | 7 runs/min, statements/s in stranded checks | 343.9 | 7.2 |
  | 7 runs/min, all statements/s | 577.5 | 220.4 |
  | 7 runs/min, company-only task-queue signals | 710 | 0 |
| 50 runs/min + 5 taskless runs/min (360 s), stranded checks/s | 16.86
company + 0.31 hint | 1.05 hint, 0 company |
  | 50 runs/min, statements/s in stranded checks | 314.0 | 7.4 |
  | 50 runs/min, all statements/s | 1,717.6 | 1,387.5 |
  | 50 runs/min, company-only task-queue signals | 2,864 | 0 |

"Statements in stranded checks" counts statements whose stack includes
`reconcileStrandedAssignedIssues`. The other statements are the runs
themselves. Recovery load queries (graph lookups and hint loads) are
12.5 to 17.2/s at 7 runs and 69 to 72/s at 50 runs in both versions.
`stale-issue-locks` saved deadlines check about 1 to 1.6 tasks/s in both
versions.
- Without the taskless runs, master already showed the backstop degrade
under activity: with 7 runs a minute, 2.46 backstop checks/s, because
every run writes its agent's row and marks all of that agent's tasks
changed. With this branch: 0.27/s.
- New tests in `issue-recovery-deadlines.test.ts`, red on
`origin/master` service files and green here:
- "re-checks a run's own tasks for its settlement, lease and dispatch
writers, and nothing for a run without a task": for `notifyRunDispatch`,
`notifyExecutionRunWork`, `notifyExternalConversationSettlement`,
`notifyChatInboundWakeNotices` and
`promoteDeferredWorkIntents(lock_released)` with a run id, the run's
graph (task, parent, dependent) is checked within one pass and none of
300 backlog tasks is. For a run without a task, no task is checked. A
comment checks only its task in `stranded-assignments` and
`assigned-work-invariant`.
- "counts an agent's own run heartbeat as no change to its other tasks,
and a resume as a change".
- "resolves a run without a task to no tasks and a missing task to the
whole company".
- Unchanged and passing: the #21729 pickup tests in
`issue-recovery-idle-backstop.test.ts` (finished run, user comment,
raised budget, resumed agent within a minute; missed hint within the
backstop bound; outside change within the hourly pass), the #21552 tests
in `issue-recovery-deadlines.test.ts` (including "keeps the company pass
for hold releases that can admit any task" and "falls back to the
company pass when a run affects more tasks than a hint names").
- Regression, embedded PostgreSQL: `issue-recovery-deadlines`,
`issue-recovery-work-debug-attribution`, `issue-recovery-signal-fanout`,
`issue-recovery-work-lifecycle` (42 tests);
`native-run-projection-work`, `native-recovery-notice-work`,
`muse-terminal-settlement`, `heartbeat-held-comment-wake`,
`superseded-workspace-export-recovery`,
`native-recovery-preparation-failure`, `upgrade-readiness-post-result`,
`chat-wake-conversation-batch`, `environment-service`,
`retained-native-session-cleanup-work`, `slack-conversation-lifecycle`,
`workspace-recovery-dependency-finalize`,
`native-failed-run-owner-release`, `chat-run-milestone-work` (462
tests).
- `tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- **Recovery time bounds. The owner must see this.** No hint-driven
pickup becomes slower for a task in the run's graph. Two missed-hint
bounds change:
- An agent change that sends no hint and is followed by a run heartbeat
of the same agent inside the 5-minute window (for example a resume whose
hint was lost, after which the agent ran another task) is now found by
the hourly full pass, not the 5-minute changed-tasks pass. A normal
resume sends `pause_lifted` and is checked at once (tested).
- A task outside the run's graph whose recovery depended on the end of a
taskless run, or on the native finish writers above, is now found by its
own hint, the 5-minute changed-tasks pass (its own rows), or the hourly
full pass. Before, any run finish re-checked the whole company. I found
no recovery decision that waits on another run of the agent: concurrency
waits are queued runs that dispatch resumes.
- A run-only hint for a run row that does not exist resolves to no tasks
(it was the whole company). A run row is not deleted while its tasks are
open, except by company deletion, which sends the company hint.
- Native finish writers now send run-scoped hints to the live-run
recovery queues too (`notifyLiveRunRecovery`,
`notifyLiveResourceRecovery`, attention retention). The legacy run paths
already send these with the run id.
- The fixture uses process-adapter runs, not native runs. The native
writers are narrowed by code reading. Their effect on the hosted staging
instance is not verified.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 10:47:34 -06:00
DottaandClaude Opus 5.5 3ee6049f71 perf(native-runtime): name why a preinstalled provider pack was not reused (#21781)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote agent run needs the provider pack (Node, OpenCode and the
Runner bridge) inside its sandbox before the provider can start
> - The server reuses a pack that is already installed in the sandbox
image, or it uploads its own pack
> - In a recent load test every task uploaded the pack, and this step
was about 43% of the time to first token
> - The compatibility check and the preinstalled-pack verification
failed, but the run records did not say which check failed
> - This pull request records the failed check on the run's performance
spans
> - The benefit is that the next run names the reason, so the image or
the check can be fixed without guesswork

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The run performance spans for provider pack selection. Today a failed
`provider_pack.compatibility` or `provider_pack.verify_preinstalled`
span only says `outcome: failed`. The code catches the error and drops
it.

**Subsystem affected**
Server: native runtime, remote provider pack selection
(`server/src/services/native-runtime`).

**Current behavior**
`provider_pack.select` records `reason: cached_packs_unverified` when it
uploads. The compatibility check throws one generic message for about
ten different conditions, and the caller swallows it. The remote
verification script throws a generic `manifest mismatch` without the
fields that differ.

**Proposed behavior**
Each failed check has a short, fixed name (for example
`installation_unprotected`, `contract_digest`, `artifact:nodeCommand`,
`manifest_mismatch:runnerSourceRevision`). The trace records this name
as the `check` attribute on the failed span. `provider_pack.select`
records the last preinstalled rejection as `check` when it uploads. The
compatibility span records `target: image` or `target: workspace`.

**Reason and benefit**
Upload of the provider pack costs about 15 s per task on the test
instance. To remove it, we must know which check rejects the image pack.
The attribute holds only fixed check names and manifest field names,
never paths or values.

**Breaking changes**
None. Error messages keep their existing text. The upload decision does
not change.

## What Changed

- `remote-provider-pack-verification.ts`: the compatibility check names
the failed condition (`failureCheck`). The remote verification script
prints a fixed `paperclip-pack-check=<name>` line, and for a manifest
mismatch it names the differing manifest fields. Missing files now give
a named check instead of a raw stack trace.
- `native-run-trace.ts`: a failed `measure` records the error's
`failureCheck` as the safe `check` attribute.
- `remote-provider-pack.ts`: `provider_pack.select` records the last
preinstalled rejection (or `no_preinstalled_pack`) when it uploads.
- `native-session-executor.ts`: the `provider_pack.compatibility` span
records `target`, the preinstalled-pack selection passes its rejection
callback, and a failed link has the `link_failed` check. No other lines
change.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-run-trace.test.ts
src/services/native-runtime/remote-provider-pack.test.ts
src/services/native-runtime/remote-provider-pack-verification.test.ts
src/__tests__/redaction.test.ts` passes (131 tests).
- Red/green: with the source changes reverted and the new tests kept, 12
tests fail. With the changes, all pass.
- The verification test runs the real remote script in a Node process: a
provenance-only difference gives
`manifest_mismatch:runnerSourceRevision`, a wrong artifact gives
`artifact_digest:nodeCommand`, and a missing manifest gives
`manifest_unreadable`.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low risk. The change adds diagnostics only. The selection order, the
checks themselves and the upload path do not change.
- The `check` attribute is filtered by the existing safe-attribute list
and a strict character pattern, so it cannot carry paths or secrets.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 10:46:33 -06:00
DottaandClaude Opus 5.5 d14a3b88b1 fix(native): release an idle warm session whose Runner is proven gone, and retry a failed idle close (#21782)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm agent session keeps its Runner alive in a sandbox between
turns. The server closes idle sessions, and closes them again before it
exits.
> - When an idle close failed, the idle-timer path kept the session
registered and never tried again, unless the sandbox lease was already
inactive.
> - The controller-restart path tried once more, but it accepted no
proof that the session was already gone. It checked a destroyed sandbox
only for a forced environment delete.
> - In a release-swap test, one idle close failed with
`runner_remote_process_identity_unavailable`. The session stayed
registered until shutdown, the shutdown close failed again, and the
sandbox was held until the host stopped it.
> - This pull request treats a failed close as complete when the Runner
is proven gone or the sandbox is stopped or destroyed, retries a failed
idle close a bounded number of times, and logs which Runner check
failed.
> - The benefit is that a dead warm session is released, and the next
failure names its cause.

## Linked Issues or Issue Description

Refs #21627

**What happened?**
In a zero-downtime release-swap test, a warm session's idle close failed
4 s after the server left new run starts. The failure was
`runner_remote_process_identity_unavailable`. The idle-timer path kept
the session registered with no retry. At SIGTERM,
`closeIdleWarmNativeSessionsForRestart` tried again and failed the same
way. The log showed "idle warm native sessions did not close" and the
restart-chaos harness counted a warm-shutdown failure. The log did not
say which Runner check failed, so the cause of the unreadable identity
is not known.

**Expected behavior**
A close whose Runner is proven gone, or whose sandbox is stopped or
destroyed, releases the session. A failed idle close is tried again a
bounded number of times. The failure text names the Runner check stage
and reason.

**Steps to reproduce**
1. Keep a warm remote session idle.
2. Make its close fail with a `RemoteRunnerRecoveryError` (for example,
`process_verification/process_not_running`).
3. Call `closeIdleWarmNativeSessionsForRestart()`. On master the result
is `failed: 1`.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`:
- New `warmCloseFailureProvesOwnerGone`. It uses existing proofs only:
the close error's Runner diagnostic is
`process_verification/process_not_running`, or a lease row for the
sandbox has a termination receipt (`hasRemoteTerminationReceipt`) with
state `destroyed`, or the warm run's own lease row has a receipt with
state `stopped`. `pending_cleanup` and a lease without a receipt prove
nothing. A lease read error counts as no proof.
- `closeIdleWarmNativeSessions` (controller restart, environment delete,
idle sleep) uses this proof for every failed close, not only for a
forced environment delete. A released Runner detach failure is
unchanged.
- The idle-timer path uses the same proof. Its earlier status-only rule
from #21504 (any lease status other than `active` or `retained`) is
removed, because `pending_cleanup` is written before the provider stop.
Without proof, it re-arms the timer and tries again, every 5 s, at most
3 times (`WARM_IDLE_CLOSE_RETRY_MS`, `WARM_IDLE_CLOSE_MAX_RETRIES`). The
log line records the attempt number.
- `describeWarmCloseFailure` appends the Runner check as
`[stage/reason/lifecycle]` when the error carries one.
- `server/src/services/native-runtime/native-session-executor.test.ts`:
three tests.
- "an idle warm session whose close fails with
runner_remote_process_identity_unavailable and whose Runner is proven
gone is released at controller restart"
- "names the Runner check in a warm close failure and keeps an unproven
owner registered at controller restart"
  - "retries a failed idle warm close a bounded number of times"
- The #21504 idle-path test is now "forgets a warm owner whose idle
close failed only once its sandbox has a stop or destroy receipt". A
stop or destroy receipt releases the owner. `pending_cleanup`,
`released` without a receipt, and `retained` keep it.
- "reads the warm owner's own lease rows at controller restart" (a stop
receipt for this run releases the owner; `pending_cleanup`, no receipt,
or another run's receipt do not)

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts`: 1519
passed, 7 skipped.
- Red on master (with only the two exported constants added so the file
compiles): all three new tests fail. The first fails with `"failed": 1`.
- `npx tsc --noEmit -p server`: no errors.

## Risks

- A session is released without a checkpoint only when its Runner is
proven gone or its sandbox is stopped or destroyed. The last saved
checkpoint stays the recovery source, which is the same as the forced
environment delete path today.
- Environment delete and idle sleep now also accept the proof. Before,
they accepted only a destroyed receipt during a forced delete.
`pending_cleanup` is written before the provider stop, so it never
counts as proof here.
- The idle retry adds at most 3 more close attempts per failed session,
5 s apart.
- A failed idle close whose sandbox stop is still pending now keeps its
owner until a receipt lands or a successor retires it. Before, the
status alone released it.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code
with tool use (shell, file edit) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 10:09:32 -06:00
DottaandClaude Opus 5.5 01c1e708e9 fix(runner): stop a turn that keeps resending a refused tool call and continue fresh (#21734)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs call Paperclip tools, and
`RepeatedToolFailureGuard` refuses an exact repeat of a call that
already failed validation twice
> - The refusals are not counted and never end the turn, so a model
whose host drops an argument can resend the same refused call until the
run's wall clock
> - In a recorded restart-chaos wave, an opencode/DeepSeek agent sent
the same `PATCH /api/issues/{id}` through `call_api` 431 times in 40
minutes without `body`; 427 were refused. The run held an agent slot and
delayed other tasks
> - The system knows the next step (stop the turn and continue on a
fresh session), so the harness must own it, with no Retry banner
> - This pull request counts refusals, stops the provider turn after a
bounded number with the classified cause `agent_tool_call_loop`, and
queues an automatic fresh-session continuation that tells the model to
use a dedicated tool
> - It also records the declared argument names that each opencode tool
call delivered, so we can tell whether the bridge or the model drops
`body`
> - The benefit is that a looping agent frees its slot in seconds
instead of 40 minutes, and the task continues by itself

## Linked Issues or Issue Description

**What happened?**
An opencode agent on DeepSeek created two subtasks and then tried to set
their descriptions with `call_api` `PATCH /api/issues/{id}`. Every call
arrived with `operationId`, `pathParams`, `query` and `responseText`,
but without `body`. The guard refused each repeat. The model said "I
need to include body" more than 20 times and kept sending the same call:
431 calls, 427 refused, over 40 minutes. The run stayed active and its
agent slot delayed other queued work.

**Expected behavior**
After a small, bounded number of refusals, Paperclip stops the provider
turn and continues the task automatically on a fresh session, with a
hint to use a dedicated Paperclip tool. The board shows no Retry banner
because the system owns this recovery.

**Steps to reproduce**
1. Run a native agent whose model resends one tool call with a missing
required argument.
2. The guard refuses the third and later identical calls.
3. On master, the refusals never end the turn.

**Paperclip version or commit**
`master` at 6400698a8e.

## What Changed

- `repeated-tool-failure-guard.ts`: counts refusals per exact call and
per run. At 6 refusals of one call, or 20 refusals in the run, it throws
`ToolCallLoopError` (`code: "agent_tool_call_loop"`, tool, counts and
argument names, never values). The message tells the model the turn
stops and gives a short hint.
- `paperclip-runner-tool-authority.ts`:
`scheduleToolCallLoopContinuation` writes one idempotent native
status-wake outbox row (key `agent_tool_call_loop:<runId>`) with
`forceFreshSession: true` and the hint as the wake instruction. The
outbox dispatcher admits it with `allowRunCoalescing: false`, so it
never merges into a deferred comment wake or the active run. An assignee
run uses `issue_continuation_needed`. A completion-review run uses
`native_completion_review` with its review assignment, which the
dispatcher revalidates, so the fresh reviewer session keeps the review.
- `native-session-executor.ts`: the session authority epoch queues that
continuation when the guard raises the loop error. The governed-wait
lookup finds the wake on the next tool event and ends the provider turn
as `yielded` with a `same_agent` continuation, the same way the
connection-tools refresh ends a turn. If the wake cannot be queued, the
refusal still reaches the model and the error is logged.
- `status-decision-committer.ts`: a yielded continuation with that key
binds to the queued wake instead of creating a second one.
- `status-arbiter.ts` and `native-run-finalizer.ts`: a pending review
run that ended through this continuation skips the
`native_review_unresolved` resume recovery, because the queued review
wake owns the next reviewer run.
- `mcp-bridge.ts` and `opencode-proxy-finish-diagnostics.ts`: the
opencode tool-delivery diagnostic records `argumentNames` (only names
declared in the tool's input schema, sorted, at most 1,024 characters of
JSON), `argumentNamesTruncated` and `undeclaredArgumentCount`. Values
and undeclared names are never recorded. The forwarder's closed key
allowlist accepts the new fields and validates them, so a record stays
under its 2,048-character limit.

### Threshold choice

- **6 refusals of one exact call.** The guard already runs a call twice
and then refuses with an explicit correction. Six more refusals (eight
identical attempts in total) give a model that reads errors many chances
to change its arguments. In the recorded run the model never changed
them in 427 refusals, so six refusals end that loop within seconds.
- **20 refusals in the run.** A model can alternate between a few
refused variants and keep each one under 6. Refusals happen only for
exact repeats of a certain validation failure, so a healthy run almost
never reaches 20.

### Previous-release story

The guard counters live in memory for one run's tool authority, as
before. A deploy or restart during a run starts the counters again at
zero on the new process; that only delays the stop. The continuation is
an ordinary wake row. If a server on the previous release finalizes a
run that a new server parked, it does not know the key and queues its
own `same_agent` continuation too; the issue execution lock lets one run
and the other coalesces.

## Verification

- Red then green, `repeated-tool-failure-guard.test.ts`: "after N
refusals the guard raises a turn-stopping loop error" and "stops the
turn once refusals across different calls reach the run limit". Both
fail on master.
- Red then green, `native-session-executor.test.ts`: "a turn whose model
repeats a refused call N times ends with agent_tool_call_loop and
schedules a continuation". With the epoch handling disabled it fails (13
calls instead of 8, nothing parked). Also "keeps the refused call's
error when the loop continuation cannot be queued".
- `paperclip-runner-tool-authority.test.ts`: "queues one fresh-session
continuation outbox for the assignee / a completion reviewer stopped by
a tool-call loop" (real Postgres; idempotent).
- `native-tool-call-loop-continuation.test.ts`: with a comment wake
already deferred behind the looping run, the continuation keeps its own
keyed deferred wake, the comment wake is unchanged, and after release a
run starts with `forceFreshSession` and the loop cause.
- `status-arbiter.test.ts`: a pending reviewer stopped by a loop
releases its checkout without a resume recovery.
- `native-status-arbiter-corpus.test.ts`: the existing wake-binding test
now also covers the `agent_tool_call_loop` key.
- `mcp-bridge-delivery.test.ts` and
`opencode-proxy-finish-diagnostics.test.ts`: argument names are
recorded, values and undeclared names are not, invalid records are
dropped, and a boundary test with 32 names of 64 characters stays
forwardable with `argumentNamesTruncated: true`.
- Server: the six touched test files pass (1755 tests). Runner:
`src/drivers/opencode` and `src/cli` pass (569 tests). `tsc --noEmit`
passes for `server` and `packages/paperclip-runner`.
- `pnpm verify` exits 2 (runner package changes need broad coverage), so
required CI supplies the broad run.

## Risks

- A run that hits the limit ends its turn early. The limits are far
above what a model that reads its errors needs, and the task continues
automatically on a fresh session.
- The fresh session loses the in-turn conversation. The task, its
comments, documents and workspace stay, and the continuation names the
cause and the hint.
- The diagnostic now carries declared argument names. These are schema
names, not values, and the forwarder validates the shape before it
forwards a record.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #21360

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 01:52:50 -06:00
Nicky LeachandClaude Opus 5.5 a78be8b601 feat(native): send launch preparation and run digests to plugin task prepare (#21717)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Environment plugins can run a native Runner as a plugin task, and a
warm session keeps that Runner between turns
> - On each warm turn Core sends the plugin a `prepare` operation with
the full launch requirements for the new run
> - The requirements' `requirementsDigest` covers the run identity, and
the run identity changes on every turn
> - So a plugin cannot tell that only the run moved, and it must redo
its whole preparation every warm turn
> - This pull request gives the plugin two digests: one that stays the
same while its preparation stays valid, and one that names the run
> - The benefit is that a plugin can transfer only the run identity when
nothing it reconciles changed

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The `prepare` operation that Core sends to an environment plugin for a
plugin Runner task, on cold starts and on warm attaches.

**Subsystem affected**
Native runtime, plugin Runner tasks, the Runner launch contract, and the
plugin SDK environment task contract.

**Current behavior**
A warm attach sends the full requirements. The only digest available,
`requirementsDigest`, includes `runId`, `turnId` and `itemId`, so it
changes on every turn even when the launch is otherwise the same.

**Proposed behavior**
Core computes two digests for every `prepare`:
- `preparationDigest`: the launch requirements an environment reconciles
when it prepares a Runner:
  - the runner, lease and session identity;
  - provider, model and settings;
  - the prompt and instruction text;
- the runtime context's instruction and skill bundles, working copy and
connection instructions.

It leaves out the `runId`, `turnId` and `itemId`, and the runtime
context's MCP assignment and binding (with `aggregateDigest`, which
hashes the assignment). MCP reaches the Runner through Core's tool
authority on every run attachment, never through files the environment
publishes. Core already continues a warm session across an MCP-only
change (`nativeRuntimeContextsMatch` with tool refresh). The binding is
also named per run (`native-mcp:<runId>`). So the digest stays the same
across warm turns of one Runner, lease and session, including when the
agent's MCP access changes.
- `runDigest`: `preparationDigest` plus everything it leaves out:
`runId`, `turnId`, `itemId`, and the MCP assignment and binding. It is
unique to one run's launch.

A plugin that advertises `launchDigestVersion: 1` in its task
description gets both fields on `prepare`.

**Reason and benefit**
A plugin can compare `preparationDigest` with its saved preparation and
transfer only the run identity when they match, instead of a full
preparation.

**Breaking changes**
None. Plugins without `launchDigestVersion` get no new fields.

## What Changed

- `packages/paperclip-runner/src/live/runner-provider-launch.ts`: add
`runnerLaunchPreparationDigest` and `runnerLaunchRunDigest`. The
preparation digest drops the MCP assignment and binding; the run digest
takes the full requirements and includes them. Both use the same
canonical JSON as `runnerLaunchRequirementsDigest`, with their own
schema tags so the three digests never collide. Export them from the
package and the server's vendored runner shim.
- `packages/plugins/sdk/src/environment-tasks.ts`: add optional
`preparationDigest` and `runDigest` to the `prepare` operation, and
optional `launchDigestVersion: 1` to the task description. The
capability gates both digests.
- `server/src/services/native-runtime/plugin-runner-task.ts` and
`native-session-executor.ts`: send both digests on `prepare` when the
description advertises `launchDigestVersion: 1`.
- `server/src/services/environment-task-runtime.ts`: recompute both
digests from the requirements, and refuse a `prepare` whose digests are
missing in part or do not match.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refresh the SDK
public source pin.
- Tests:
- the preparation digest is stable across runs and changes when the
runner, lease, session, instructions, model, approval policy or settings
change;
- with a runtime context, an MCP-only change (the run's binding, no
binding, a changed access set) keeps the preparation digest and changes
the run digest;
- a change to the instruction bundle, instruction entry, a skill bundle,
the skill set, the working copy or the connection instructions changes
the preparation digest;
  - the run digest changes per run;
  - Core sends the digests only when advertised;
- the runtime refuses forged or partial digests before it calls the
plugin.

## Verification

- `npx vitest run src/live/runner-launch-preparation.test.ts` in
`packages/paperclip-runner`: 26 passed.
- `npx vitest run src/services/native-runtime/plugin-runner-task.test.ts
src/__tests__/environment-task-runtime.test.ts` in `server`: 160 passed.
- `npx vitest run source-sdk-public-inputs.test.mjs
source-sdk-evidence.test.ts source-sdk-public-type.test.ts` in
`tests/runner-e2e`: 59 passed.
- `pnpm typecheck` in `server` and in `packages/paperclip-runner`:
clean.
- Not run locally: the full suite. CI covers it.

## Risks

Low risk. The digests are new optional fields behind a plugin capability
version. Core computes them and checks them again before it calls the
plugin, so a plugin can rely on them. They prove only that the launch
requirements are unchanged. A plugin that skips work on a match must
still check its own state, such as files it published into the sandbox.
If the runtime context carries values that change per run, the
preparation digest changes too, so a plugin does its full preparation.
That is safe, but the plugin saves nothing. The MCP binding is the one
such value Core sets per run today, and MCP is excluded entirely. A
plugin that does publish MCP configuration into its sandbox must not
rely on this digest for it. Values outside the requirements are not
covered at all: the account identity and credentials, the GitHub broker,
project IDs, and the plugin's own runtime revision. A plugin must keep
checking those itself. A match lets a plugin skip publishing its launch
configuration, not computing the descriptor: `prepare` must still return
a launch whose `requirementsDigest` matches the current requirements,
which carry the current MCP assignment and binding.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, code search).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:07:54 -08:00
DottaandClaude Opus 5.5 adc3356ad3 fix(native): retry a coordinator claim that loses a deadlock instead of failing the run (#21703)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native (Paperclip Runner) run claims its
`native_run_finalizations` coordinator before it starts any provider
work. The claim is one transaction. It locks the coordinator row, then
the run row.
> - A writer that holds the run row and then waits for the coordinator
row closes a lock cycle. PostgreSQL then aborts the claim with a
deadlock (`40P01`), because the claim waited first.
> - The aborted claim rolls back whole and starts no provider work. But
the raw database error failed the run before its first turn.
> - The failure path then preserved the prepared workspace. That stamped
the unclaimed coordinator `native_continuation_requires_reconciliation`,
and `failRun` used it as the run's error code. The task waited for
reconciliation.
> - A transient database fault must never end a run like this. The
target rate of runs that need Retry is 0%.
> - This pull request retries the claim transaction on transient lock
contention with the shared bounded helper.
> - The benefit is that a fresh run that loses a lock race at its claim
continues to its provider turn by itself.

## Linked Issues or Issue Description

**What happened?**
On a hosted QA stack, a fresh native assignment run (not a retry) failed
after about 50 s. The run status was `failed`. The error code was
`native_continuation_requires_reconciliation`. The error text was
`Database query failed (40P01)`. The stderr was empty. The task needed
reconciliation.

**Steps to reproduce**
1. Prepare a native run with an unclaimed coordinator (`observed`,
attempt 0), as runtime selection leaves it.
2. In another transaction, lock the run row.
3. Start the native session. Its claim locks the coordinator and waits
for the run row.
4. In the other transaction, lock the coordinator row.
5. PostgreSQL aborts the claim with `40P01`, and
`executePaperclipNativeSession` rejects with that error.

**Expected behavior**
The claim retries after the competing transaction commits. The run
reaches its provider turn.

**Paperclip version or commit**
`master` at 958eda6f16.

## What Changed

- `native-session-executor.ts`: the coordinator claim transaction runs
inside `retryTransientLockContention` (`40P01`, `40001`, `55P03`; 5
attempts, 100 ms × attempt backoff). Each attempt re-reads and re-proves
the coordinator, the run, cancellation and lineage, so a retry admits
nothing that the first attempt would refuse.
- A startup context that a rolled-back attempt opened is finished, and
the startup owner's previous context is restored, before the next
attempt.
- New PostgreSQL test
`native-coordinator-claim-deadlock.integration.test.ts`: `retries a
claim that lost a deadlock and reaches the provider`.

## Verification

- RED on `master` (958eda6f16): the new test fails.
`executePaperclipNativeSession` rejects with a `40P01` cause, and the
provider is not called.
- GREEN with this change: `pnpm --filter @paperclipai/server exec vitest
run
src/services/native-runtime/native-coordinator-claim-deadlock.integration.test.ts`
passes. The provider is called once and the coordinator is at attempt 1.
- `tsc --noEmit -p server/tsconfig.json` passes.
- `pnpm verify` for the affected files: see the PR comments.

## Risks

- Low risk. The normal path has no contention and does not retry.
- Under contention, the claim waits at most about 1 s more (backoff
100–400 ms) before provider work. This is before `turn.start`, so time
to first token grows only in the contended case.
- The lock order (coordinator, then run) is unchanged. It matches
admission and the runner checkpoint owner lock. The writer that took the
run row first on the QA stack is not identified, because no server logs
or SQL were available. This change makes the claim survive any such
writer.
- Out of scope: a transient fault in other pre-dispatch statements of a
run with a prepared remote workspace. There, workspace preservation
still stamps the pre-claim code before `failRun`. The pre-claim
fresh-continuation fallback then continues the task.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, embedded PostgreSQL tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #20561 (pre-dispatch transient retry), #20682 (provider-pack
admission lock wait).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:23:33 -06:00
DottaandClaude Opus 5.5 958eda6f16 fix(native): wait for a live outgoing server before resuming its warm session in a deploy overlap (#21697)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A deploy swaps servers with an overlap. The outgoing server still
serves while the incoming server starts new runs.
> - A warm agent session stays with the server that ran its last turn.
That server closes the session about 1 s after it leaves new run starts
to the replacement (#21627).
> - A follow-up that started within about 1 s of its predecessor's
finish read the session as suspended, while the outgoing server was
still closing it. The follow-up did not wait. It started a second Runner
with the wrong OpenCode launch files, and the Runner refused the launch.
> - The same-run retry then went to the outgoing server, which had
already left run starts. The run failed, and a blocked parent showed a
Retry during its scheduled resume.
> - This pull request makes the follow-up wait for the live outgoing
server, rebuilds the admitted launch files on the new Runner, keeps
same-run retries on the replacement, and shows the scheduled resume for
a blocked parent.
> - The benefit is that a follow-up during a deploy overlap finishes
without a failed run or a Retry banner.

## Linked Issues or Issue Description

Refs #21627

**What happened?**
A zero-downtime release-swap test (Gate B, wave B1 on master 890dcbdd66)
failed 2 of 8 cross-server hand-offs. In both, the follow-up started 1.0
s and 0.6 s after its predecessor finished on the outgoing server. The
other 6 hand-offs started 34 to 217 s later and passed.
- The incoming server took the resume path that skips
`inspectRemoteWarmSessionRecovery`, because the local authority state
read `suspended`. The live-owner wait from #21627 lives only in that
inspection, so it never ran (0 waits recorded).
- The new home had no harness backup, so the run had no provider
artifact receipt. `prepareRemoteRunner` returned early on
`legacyProviderContinuity`, and the launch profile named the provider
pack folder. Runner core refused it: "OpenCode run.prepare launch does
not match the runner-owned qualified profile".
- The same-run retry was claimed by the outgoing server after it left
run starts. Its predecessor retirement reported `not_suspended`. Both
runs failed with `native_continuation_requires_reconciliation`, and
recovery started fresh runs.
- For a parent already blocked on its subtasks, the task showed "failed"
with Retry for 31 s while the same-run resume was scheduled.

**Expected behavior**
The follow-up waits while the outgoing server may still hold the
session. A new Runner gets the same OpenCode launch files as the
session's first launch. Only the replacement claims same-run retries. A
blocked parent shows the scheduled resume and no Retry.

**Steps to reproduce**
1. Run the restart-chaos release-swap preset with fresh homes and a warm
OpenCode staging mix.
2. Let a run finish on the outgoing server after the traffic switch,
with a follow-up comment already queued.
3. See the follow-up start on the incoming server within about 1 s and
fail on `run.attach` with the qualified-profile mismatch.

**Paperclip version or commit**
master 890dcbdd66.

**Deployment mode**
Hosted deploy overlap (release swap) with Daytona sandboxes and S3
storage.

## What Changed

- `remote-warm-session-recovery.ts`: new `waitForLiveWarmSessionOwner`.
It reads the finished run's finalization owner. While that server is
alive inside the warm window, the follow-up defers with the existing
`native_controller_recovery_waiting_for_owner` wait. A harness backup
that holds the suspended state and was completed after the turn proves
that the owner finished its close, so the follow-up continues at once.
- `native-session-executor.ts`: calls that wait before the `suspended`
check, so every warm follow-up on another server's session passes
through it. The backup check runs only when the owner is alive.
- `native-session-executor.ts`: when a resume has a durable session but
no receipt from a harness backup or the admission, it reads the provider
artifact receipt recorded on the session's last run. The new Runner then
gets the admitted launch root, as a cold continuation already does.
- `heartbeat/recovery.ts` and `heartbeat.ts`: a server that yields run
starts to a replacement does not claim same-run retries, from the
recovery sweep or from its own resume timer. It re-checks every 3 s, so
it takes the retries back when the replacement stops.
- `task-interruption-presentation.ts`: a blocked task whose remote run
waits for its `exact_checkpoint_resume` keeps the resume time, so it
shows `retry_scheduled` with no Retry.
- `doc/execution-semantics.md`: documents these rules.

## Verification

Red on master, green with this change:
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "close
window waits for its live lease"`: "a follow-up that starts within the
outgoing server's close window waits for its live lease instead of
relaunching the Runner". Master proceeds and fails with
`runner_state_identity_mismatch`.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "remote
opencode-continuation-prior-receipt"`: a resume relaunch without an
adopted process uses the original OpenCode file paths. Master launches
from the provider pack folder and does not retain the receipt.
- `cd server && npx vitest run src/services/heartbeat/recovery.test.ts
-t "leaves run starts"`: "an outgoing server does not claim a same-run
retry once it leaves run starts". Master claims it.
- `cd server && npx vitest run
src/__tests__/task-interruption-query.test.ts -t "blocked parent whose
run awaits"`: "a blocked parent whose run awaits its exact same-run
resume shows retry_scheduled with no Retry". Master shows `failed` with
Retry.

Also new: "keeps a follow-up waiting for a live outgoing server until
that server has checkpointed the session after the turn" in
`native-controller-recovery.integration.test.ts` (the backup proof and
the owner exit end the wait).

Focused suites pass: native-session-executor,
remote-warm-session-recovery, native-controller-recovery (integration),
heartbeat recovery, task-interruption-query,
task-interruption-presentation, native-recovered-owner-followup,
heartbeat-native-predispatch-retry, run-start-handover. `npx tsc
--noEmit -p server/tsconfig.json` passes.

Not run here: the restart-chaos release-swap wave. The maintainer runs
it after merge.

## Risks

- A follow-up on a session that a live outgoing server ran can now wait
until that server exits or the warm window ends. This applies only when
no harness backup proves that the server finished its close after the
turn. Before, such a follow-up failed and needed recovery.
- The backup proof hashes the restored backup. It runs only when the
previous owner is alive in another process (a deploy overlap), not on a
normal turn.
- A resume without a backup receipt now restores the admitted artifacts
from storage, as a cold continuation does. A session whose last run
recorded no receipt keeps the legacy pack-folder launch.
- A yielding server leaves same-run retries to the replacement. If the
replacement stops, the yield ends with its lease, and the older server
claims them again within seconds.
- Previous-release state: no persisted field is added. Receipts,
finalization owners and `failureDetail.recoveryMode` are already saved
by the previous release.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 19:50:01 -06:00
Nicky LeachandClaude Fable 5.2 0ab73d4e30 feat(native-runtime): ask the Runner for provider-started turns on native ACPX Claude runs (#21456)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core runs Claude Code through Paperclip Runner. Claude can start a
shell command in the background and end its turn. When the command
finishes, Claude Code starts a cycle of its own to continue.
> - Today the run ends when the turn ends. The command stops, and the
run fails with `native_finalization_missing`.
> - The Runner can keep Claude while a background shell is running and
give the cycle of Claude to the controller as a turn that the provider
started. Core can wait for that turn. The transport can ask for it
(#21607), and a controller restart during it is safe (#21612).
> - No run does this yet, because Core does not ask the Runner for it.
> - This pull request makes the native runtime of Core ask, for ACPX
Claude. It is the only part that changes what a production run does.
> - The benefit is the goal of the whole work: a run with a background
command that is still running continues when the command finishes, and
does not fail with `native_finalization_missing`.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request turned the behavior on and had grown too large to
review. It is now three stacked pull requests:

1. #21607 (merged): the transport can ask the Runner for
provider-started turns. No caller asks.
2. #21612: the recovery after a controller restart during a wait or
during a provider-started turn.
3. **This pull request:** Core sets the option for native ACPX Claude
runs.

This is part 3. Its base branch is `fix/provider-turn-restart-recovery`.
Refs #19912 (design).

**What existing behavior does this improve?**

What a native ACPX Claude run does after a turn that ends with no result
while a background command is still running.

**Subsystem affected**

Core, the native runtime:
`server/src/services/native-runtime/native-session-executor.ts` and
`native-provider-turn-wait.ts`. ACPX Claude only.

**Current behavior**

- The run ends at the end of the turn. The Runner restarts Claude, the
background command stops, and the run fails with
`native_finalization_missing`.

**Proposed behavior**

- The native runtime sets `providerStartedTurns: true` in the transport
options for ACPX Claude, at the two places that build those options. One
function, `nativeProviderStartedTurns`, decides this and also decides if
the consumer gets the wait options, so the two cannot be different.
- A turn that ends with no result while a background command is still
running now waits, for a maximum of 10 minutes, for the turn that Claude
starts by itself. That turn is consumed as the next turn of the same
run, and its result settles the run.
- Other providers and other ACPX agents do the same as today.
- Limit of the Runner (see #21421): after a follow-up turn, the Runner
waits again only while a background command is still running. A command
that finishes while Claude reports on an earlier command gets no
follow-up.

**Reason and benefit**

- The opt-in is in the last part, so each earlier part can merge with no
change for a production run.

## What Changed

- `server/src/services/native-runtime/native-provider-turn-wait.ts` (5
lines added, 1 removed): `nativeProviderStartedTurns`.
- `server/src/services/native-runtime/native-session-executor.ts` (4
added, 2 removed): the option at the two places that build the transport
options, and the same function for the wait options.
- `server/src/services/native-runtime/native-session-executor.test.ts`
(4 added, 2 removed): the two native paths set the option for ACPX
Claude and not for other providers.

## State saved by the previous release

- No persisted state changes in this part.
- **A Runner that does not know the field** ignores it, and the run does
the same as today.
- **A deploy during a run:** a turn that started before the deploy has
no opt-in and ends as today. The next turn has the opt-in.
- **Rollback of Core:** the previous Core does not set the option, so
the Runner does not wait.

## Verification

The items below were run on the tree that was verified before the split
into three parts (commit `1b15ded1c4` has that tree, byte for byte). The
head of this branch (`23e2c13ad3`) is that commit on the new head of
#21612; its three server files are the same.

- `pnpm typecheck` and `node
scripts/verify-runner-vendor-dependencies.mjs` in `server`; `npx tsc
--noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json`,
`pnpm run typecheck:typescript` and `pnpm run build:typescript` in
`packages/paperclip-runner`: all passed.
- `server/src/services/native-runtime/native-session-executor.test.ts`
and `native-runner-restart-recovery.integration.test.ts`: 1,732 passed,
21 skipped.
- The end-to-end behavior is tested in parts 1 and 2 with the real
`paperclip-runnerd`, where the test sets the option in the way that this
part does.
- Live run with a real Claude account, not in CI, at the adapter level
of the Runner (see #21316): the cycle of Claude was given as a turn with
its own usage, and each Paperclip tool call was admitted.

Not run: a live run through Core, `paperclip-runnerd` and the real
sidecar with a real Claude account (that path needs an API key, not a
local sign-in); a remote sandbox; the full suite. CI supplies the broad
coverage.

## Risks

- **This part turns the behavior on for each native ACPX Claude run.** A
run with no background command is not changed: the Runner waits only
when a shell is still running at the end of a turn that has no result.
- **This part must not merge before #21612.** Without #21612, a
controller restart during a wait or during a provider-started turn fails
the run.
- **A run can now stay open up to 10 minutes longer** when the
background command does not finish or Claude does not start a turn. The
agent computer stays active for that time.
- **A waiting run can show as "suspicious" after 5 minutes of silence**
in the run watchdog. The watchdog only labels a native run; it does not
stop it.
- **Follow-ups that the Runner cannot wait for are lost,** as the limit
above says.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 13:10:20 -08:00
DottaandClaude Opus 5.5 890dcbdd66 fix(native): wait for a live warm session owner during a deploy overlap (#21627)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A deploy swaps servers with an overlap: the outgoing server still
serves while the incoming server starts new runs
> - A warm agent session stays open in the server process that ran the
last turn, after that run finished
> - The incoming server took over a warm session while the outgoing
server was alive and still held it, because the finished run's
finalization lease ends at commit
> - The takeover then failed, the task went blocked with a Retry, and in
one case a retry loop ran 27 times on the same sandbox
> - This pull request makes the incoming server wait while the server
that holds the warm session is alive, makes the outgoing server close
idle warm sessions as soon as it leaves new runs to the replacement, and
stops the retry loop when the promised fresh sandbox does not come
> - The benefit is that a follow-up during a deploy finishes without a
person, and no Runner ever gets two controllers

## Linked Issues or Issue Description

**What happened?**
A zero-downtime release-swap test (Gate B, wave B1, master aaec67b891)
found two failures with one cause.
Task 1: run A finished on the outgoing server at 15:13:56. Its warm
session stayed idle in that server. The incoming server claimed the
follow-up at 15:13:57. `inspectRemoteWarmSessionRecovery` asked
`evaluateNativeControllerTakeover`, which allowed the takeover with
`remote_controller_lease_released_or_expired`. That lease covers the
finished run, not the warm session. The outgoing server was alive until
15:18:18. `runner.prp.authenticate` failed after 62.3 s, the run failed
with `remote_warm_session_unproven`, the failure was saved as ambiguous
and owned by the board, and the task stayed blocked with a Retry.
Task 2: the same shape on a later swap started 27 automatic runs in 13
minutes. Fourteen of them said "The next attempt starts on a fresh
sandbox", but each one resumed the same sandbox and got the same refusal
(`journal_unsettled`).

**Expected behavior**
A follow-up claimed during an overlap waits for the warm session owner
to let go, then continues with no Retry banner and no second controller.
If the outgoing server dies with the session open, the takeover
continues after its ownership ends. A message that promises a fresh
sandbox is true, or the retries stop after a bounded number with a true
state.

**Steps to reproduce**
1. Run the restart-chaos release-swap preset with a warm (opencode or
codex) staging mix.
2. Let a run finish on the outgoing server after the traffic switch,
with a follow-up comment already queued.
3. See the follow-up fail with `remote_warm_session_unproven` and the
task go blocked.

**Paperclip version or commit**
master aaec67b891 and 45531b719e.

**Deployment mode**
Hosted deploy overlap (release swap) with Daytona sandboxes.

## What Changed

- `remote-warm-session-recovery.ts`: before live adoption, the run
proves that the server process that ran the finished turn is gone. It
uses the shared owner proof (`proveOwnerGone`): a PID check in the same
process space, or the `controller_instances` lease in another process
space. While that server is alive, the run defers with the existing
`native_controller_recovery_waiting_for_owner` wait and checks again
after 5 s. The wait ends when the server exits or is fenced, or when the
finished turn's warm window (idle timeout plus two minutes) ends.
- `native-session-executor.ts` and `run-start-handover.ts`: a server
that leaves new run starts to a replacement closes its idle warm
sessions after 1 s, through the existing idle-timeout close. A session
that becomes idle after that closes after 1 s too. This is the same
close that shutdown already does, done earlier.
- `predecessor-retirement-fallback.ts`: when a retry meets an unretired
owner on the same sandbox that its source already asked to replace, it
does not ask again. The run fails without another automatic retry, and
the task shows a Retry.
- `predecessor-retirement.ts`: the error no longer says "The next
attempt starts on a fresh sandbox", because the replacement is only a
request.
- `doc/execution-semantics.md`: documents the owner wait, the hand-off
at the swap, and the retry stop.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-controller-recovery.integration.test.ts`
- New: "waits while the outgoing server that ran the finished turn still
holds its warm session, then takes it over once that server exits" (red
on master: master returned a takeover while the owner was alive).
- New: "waits for a live outgoing server in another process space until
its controller lease ends" (red on master).
- New: "stops waiting for a live outgoing server once the finished
turn's warm window has closed" (bound check).
- `cd server && npx vitest run
src/__tests__/heartbeat-native-predispatch-retry.test.ts`
- New: "stops retrying with a truthful failure when the next attempt met
the same unretired sandbox instead of a fresh one" (red on master:
master scheduled another retry on the same sandbox).
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t
"replacement server starts"`
- New: "closes a warm session promptly when a replacement server starts
this server's runs (before/after the turn ends)" (both red on master).
- Focused suites: native-session-executor, remote-warm-session-recovery,
native-controller-recovery, native-recovered-owner-followup,
native-runner-restart-recovery, heartbeat-native-predispatch-retry,
environment-runtime-reusable-resume-replacement,
task-interruption-presentation, sandbox-runner-port-holders.
- `npx tsc --noEmit -p server/tsconfig.json` passes. `pnpm verify` exits
2 (a doc is a global CI input), so required CI supplies broad coverage.
- Not run here: the restart-chaos release-swap wave. The maintainer runs
it after merge.

## Risks

- A follow-up can now wait up to the outgoing server's remaining overlap
when that server cannot close its idle session. Before, it failed and
needed a Retry. The wait is bounded by the owner's exit or the warm
window.
- The outgoing server closes idle warm sessions earlier during an
overlap. The next turn then resumes from a suspended Runner instead of a
live one, as it already does after a shutdown.
- A retry chain that meets the same unretired sandbox twice now stops
one attempt earlier than the shared cap. The task shows a Retry instead
of looping.
- Rows without a recorded process space keep the existing one-release
local proof, so they behave as before outside the same host.
- Previous-release state: this change adds no persisted field.
Finalization rows saved by the previous release already record the
controller boot, PID, process space and start time, so they carry
forward. A follow-up already deferred by the previous release resumes
through the same wait code.
- Runtime cost: the owner proof adds one `ps` start-time read (same
process space) or one `controller_instances` row read (other process
space) per inspection. It runs once per attempt, at most every 5 s while
a live owner holds the session.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 11:59:17 -06:00
DottaandClaude Opus 5.5 aaec67b891 fix(native): close idle warm sessions in parallel across agents at shutdown (#21597)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - At shutdown, the server closes its idle warm Runner sessions before
it prepares the hot-restart snapshot
> - It closed them one at a time, and each full close (checkpoint, then
two stopped-copy collections) takes 3-5 s
> - In a zero-downtime swap test, five closes used the whole 20 s
pre-drain budget. Hot-restart preparation timed out, a checkpoint was
cut mid-way, and shutdown logged "hot-restart shutdown preparation
failed"
> - The same test logged 34 "Delivery reconciliation failed" errors, all
from directory cleanup retries that found the agent directory lock busy
> - This pull request closes idle warm sessions in parallel across
agents, one at a time within an agent, and treats a busy directory lock
as a quiet retry
> - The benefit is that shutdown preparation fits its budget with
several warm sessions, and expected lock contention no longer looks like
a failure

## Linked Issues or Issue Description

**What happened?**
`closeIdleWarmNativeSessions` awaited each warm session's full close in
turn. In swap 2 of a release-swap test, five idle sessions closed at
22:28:42, :46, :51, :56 and :59 (one "transport_close_requested"
diagnostic each). The `hot_restart_prepare` step then timed out at 19.9
s. Separately, `recoverDirectoryRelease` threw
`AgentDirectoryOperationBusyError` when another close held the agent
directory lock. The delivery coordinator logged each one at ERROR level.

I first checked if these sessions should have taken the cheap
released-session detach. They could not: `entry.released` is set only
after a successful `session.release()` through a release/resume-capable
plugin Runner task. The Runner state files from all sandboxes in the
test recorded only `run.prepare`, `session.open`, `turn.start`,
`runner.drain` and `runner.suspend` commands, and no `runner.release`.
The Daytona daemon logs show only `/process/execute` and
`/process/session` calls, and no Runner-task API. So no session in that
test was ever released, and a full close was the only correct shutdown
for them.

**Expected behavior**
Idle warm sessions of different agents close at the same time. Two
sessions of the same agent never close together, because they collide on
the agent directory advisory lock. A cleanup retry that finds that lock
busy stays pending until its claimed retry time and logs no error.

**Steps to reproduce**
1. Keep five or more idle warm sessions on remote sandboxes.
2. Send SIGTERM.
3. The closes run one after another and `hot_restart_prepare` times out.

**Paperclip version or commit**
`master` at bbed32450c.

## What Changed

- `closeIdleWarmNativeSessions` groups sessions by company and agent. Up
to four agents close at once, and each agent's sessions close one at a
time. The per-session close logic is unchanged.
- Idle-sleep retirement (`prepareIdleWarmNativeSessionsForSleep`) keeps
a concurrency of one, so it still checks drain authority before each
retirement.
- `recoverDirectoryRelease` catches `AgentDirectoryOperationBusyError`,
keeps the deferred retry time, and returns the pending entry instead of
throwing.

## Verification

- Red, then green (each failed against `master` source and passes with
the fix):
- `closes idle warm sessions of different agents in parallel and one at
a time within an agent` in
`server/src/services/native-runtime/native-session-executor.test.ts`.
Before the fix, the second agent's close was never started while the
first agent's close was pending.
- `keeps a cleanup pending without an error while another server holds
the agent directory lock` in
`server/src/__tests__/agent-directory-working-copies.test.ts`. It holds
the real Postgres advisory lock from a second connection. Before the
fix, the coordinator received an `AggregateError` with
`AgentDirectoryOperationBusyError`.
- `native-session-executor.test.ts` (1502 passed) and
`agent-directory-working-copies.test.ts` pass in full.
- `pnpm typecheck` in `server` passes.

## Risks

- Up to four warm closes now run at once, so shutdown makes more Runner
and database calls at the same time. The cap keeps this small.
- The order of entries in `failures` can change when several closes
fail.
- A busy release now waits for its deferred retry silently. A lock that
stays busy for a long time shows as a stuck pending cleanup, not as
repeated errors.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 04:15:19 -06:00
Nicky LeachandClaude Opus 5.5 e977c3b922 feat(native-runtime): end the wait for the provider's own turn when a person comments (#21455)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core runs Claude Code through Paperclip Runner. Claude can start a
shell command in the background and end its turn. When the command
finishes, Claude Code starts a cycle of its own to continue.
> - Today the run ends when the turn ends. The command stops, and the
run fails with `native_finalization_missing`.
> - The Runner part of this work is complete in #21316 and the pull
requests below it: when a controller sends `providerStartedTurns` on
`turn.start`, the Runner keeps Claude while a background shell is
running, and gives the cycle of Claude to the controller as a turn that
the provider started.
> - Part 2 gave the consumer a wait with a limit and two inputs: a hook
that ends the wait early, and a start time.
> - This pull request gives both from the server for a native ACPX
Claude run. A new comment from a person ends the wait. The start time
comes from an event that is already saved, so a restart of Core does not
make the wait longer.
> - The benefit is that a person who comments does not wait up to 10
minutes for the agent to see it.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request is one of four parts on the Core side. They are
stacked on #21316:

1. The driver admits a turn that the provider started.
2. The consumer waits for that turn, for a limited time.
3. The server ends the wait when a person comments, and keeps the clock
across a restart.
4. The transport sends the opt-in. Only this part changes what a
production run does.

This is part 3. Its base branch is `feat/core-provider-turn-consumer`.
Refs #19912 (design).

**What existing behavior does this improve?**

Nothing visible yet. What Core gives to the consumer of a native ACPX
Claude run.

**Subsystem affected**

Server: `server/src/services/native-runtime`.

**Current behavior**

- A comment from a person during a run is queued as a deferred wake. The
next run delivers it. This does not change.
- Core has no wait for a provider-started turn.

**Proposed behavior**

- For a native ACPX Claude run, Core gives `executeNativeSession` the
option `providerTurnWait`.
- One delivery worker does the person-comment check for all runs that
wait. The server registers it at startup with the shared delivery work
coordinator, as it registers the cost accounting worker. The worker
listens to the existing notification for deferred wakes. On each notice
it runs one query for each run that waits: a comment from a person on
this issue, made after the start of the run, that a deferred wake of
this agent names. A comment that the query finds ends the wait of that
run. The run then takes the path of today, and the next wake delivers
the comment as today.
- If a database read fails, that run stays owed a check, and the
coordinator runs the worker again after 5 seconds, as for each other
delivery worker. A read that fails for one run does not stop the check
of the other runs. When no run is owed a check, the worker has no
deadline and does no query.
- The start time of the wait is the time of the first saved
`turn.completed` event of the run that has `awaitingProviderTurn`. With
no such event it is empty.
- Each query has the company id in its condition.

**Reason and benefit**

- The comment window starts at the start of the run, not at the start of
the wait. A comment made during the prompt turn must not wait for the
limit.

## What Changed

- `server/src/services/native-runtime/native-provider-turn-wait.ts`
(new, 109 lines) and 4 lines in `native-session-executor.ts`.
- `server/src/index.ts` (4 lines): the registration of the worker, its
stop at server close, and its place in the list of delivery workers that
must be ready. `delivery-work-notifications.ts` (1 line): the queue key
of the worker. The key needs no database change, because each topic uses
the one shared channel.
- `server/src/vendor/paperclip-runner/index.ts`: one type export.
- Tests: `native-provider-turn-wait.test.ts` (new, 252 lines) and one
case in `native-usage-accounting.test.ts` (25 lines).

## State saved by the previous release

- No new persisted state and no schema change. The start time is read
from `heartbeat_run_events`.
- **A run that the previous release started:** it has no waiting
terminal, so the start time is empty.
- **Core restarts during a wait:** the new process reads the start time,
so the limit is the original one. The consumer then continues the wait
(#21454), and the worker checks the comments again from the start of the
run, so a comment that was made before the restart still ends the wait.
- **Rollback of Core during a wait:** the previous Core stops at the
waiting terminal and finalizes the run as `native_finalization_missing`,
as today.

## Verification

At the head of this branch:

- `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p
tsconfig.test.json` and `pnpm run build:typescript` in
`packages/paperclip-runner`; `node
scripts/verify-runner-vendor-dependencies.mjs` and `pnpm typecheck` in
`server`; `node --test
scripts/__tests__/runner-release-crate-cache.test.mjs`: all passed.
- `native-provider-turn-wait.test.ts`, 10 tests on embedded Postgres
with the real coordinator: a comment from a person in a deferred wake
ends the wait; a comment from an agent, a comment before the run, and a
comment of a different company do not; the start time comes from the
first waiting terminal, and a run of the previous release has none.
- The 6 new tests: a read that fails one time is tried again after 5
seconds and finds a comment that is already saved, with no new notice; a
notice during a read that fails gives a new check; a read that fails for
one run does not hold back a second run, and each run ends only for its
own comment; a run that starts to wait during a check is checked; a wait
that is aborted is removed, nothing checks it again and no deadline
stays; with no run owed a check there is no deadline and no query in one
hour of test time.
- The test helper now registers the wake intent before it writes the
deferred wake, in the same transaction. This is the order of each
production writer (`modules/wake-queue/adapters/postgres.ts` and
`services/heartbeat/queue.ts`). Production code did not change for this.
- Mutation checks: a read that fails clears the owed check (3 tests
fail); an aborted wait is not removed (the abort test fails).
- Also run, unchanged and green: `delivery-work-coordinator.test.ts`,
`budget-enforcement-worker.test.ts`,
`server-startup-feedback-export.test.ts`, `startup-timeline.test.ts`.
The 7 files together: 1,699 passed, 7 skipped.
- `native-usage-accounting.test.ts`: a prompt turn and the
provider-started turn after it are each counted one time, with the
events as the executor receives them. The cost of the provider-started
turn is the session total in the cumulative channel, and it is not added
two times. The accounting code is not changed.

Not run: a run through the executor with a database; the full suite. CI
supplies the broad coverage.

## Risks

- **One more query at each start of a native ACPX Claude run,** on
`heartbeat_run_events` by company and run.
- **The worker is registered in each server process, and it is idle with
no wait.** A notice with no run that waits does no query. No wait starts
before part 4.
- **After an abort during a retry, the 5 second retry of the coordinator
runs one more time.** It finds nothing owed and does no query. The
coordinator has no call to cancel the retry of one worker.
- **A process that does not register the worker does no comment check.**
The server registers it with no condition at startup, and native runs
execute only in that process. Only a unit test can be in this state.
- **A comment ends the wait but does not interrupt a turn that is
running.** That is the rule of part 2.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests and cut this slice from the full change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 22:55:11 -08:00
DottaandClaude Opus 5.5 1f388ed385 Start follow-up runs normally when a warm sandbox's window ends during its idle close (#21504)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm sandbox keeps an agent's Runner alive between runs, so the
next run on the task starts fast
> - When a run ends, the server closes the idle warm Runner in-process.
A separate SQL worker stops the sandbox when its warm window ends
(#21410, merged today)
> - Both steps start at the same moment. The sandbox stop wins, so the
idle close fails. The failed owner stays registered, and the next run
throws `native_instruction_materialization_handoff_unavailable` before
any provider work
> - Recovery must just work: a follow-up run must never fail because a
predecessor's warm copy is gone
> - This pull request lets the idle close finish before the sandbox
stops, prepares instructions fresh when the warm copy cannot be handed
off, and releases an owner whose sandbox the host already gave up
> - The benefit is that automatic follow-up runs start normally after a
warm window ends

## Linked Issues or Issue Description

Refs #21472 (base of this stacked PR), #21410 (introduced the
warm-window stop).

**What happened?**
In a Daytona two-server chaos wave (`w-20261113T200824-s7106`), three
automatic follow-up runs failed before any provider work with
`native_instruction_materialization_handoff_unavailable`: run
`5f68e194`, run `d6bd5a97` and run `d1198e40`. None of them was at a
release swap. In each case, the predecessor's idle close had failed
first with `provider_state_unreadable` / "Daytona sandbox lease … is no
longer active". The cause: the warm-window stop from #21410 stopped the
sandbox at the same moment that the idle close started. For the first
run's predecessor, the close was requested at 20:30:34 and the sandbox
got SIGTERM at 20:30:37.8.

**What did you expect?**
The follow-up run prepares its instructions fresh and continues. The
warm-window stop does not interrupt the host's own idle close.

**Steps to reproduce**
1. Run a warm Daytona task until it ends, so its warm window starts.
2. Let the warm window end while the idle timer closes the Runner.
3. Start a follow-up run on the same task. It fails with
`native_instruction_materialization_handoff_unavailable`.

Two of the three tasks then ended stranded. That downstream part (the
claim-gate cancellation with `execution_reconciliation_required` and the
stranded notice) is owned and fixed by another session. It is **not** in
this PR.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`
`claimWarmNativeInstructionCopy`: after it retires a stale warm owner
whose copy cannot be adopted, it returns `null`. The caller then
prepares instructions fresh. Before, it threw
`native_instruction_materialization_handoff_unavailable`.
- `server/src/services/warm-sandbox-expiry.ts`: the warm-window stop now
waits `WARM_SANDBOX_IDLE_CLOSE_MARGIN_MS` (2 minutes) past the window.
The in-process idle close (close grace plus the 30 s proof read)
finishes first.
- `native-session-executor.ts` idle timer: when an idle close fails, it
reads the lease rows for that sandbox. If every row is no longer
`active` or `retained` (the host gave the sandbox up), it releases the
in-process owner. The next run then prepares fresh.
- `doc/execution-semantics.md`: one paragraph about the stop margin and
fresh preparation.

## Verification

Red → green (each test fails on the base `35ba76d0ef` and passes on this
branch):

- `native-session-executor.test.ts` › "hands off collection before
probing or retiring a warm owner: unavailable" — red: throws
`handoff_unavailable`; green: returns `null` (fresh preparation).
- `native-session-executor.test.ts` › "forgets a warm owner whose idle
close failed only once the host gave up its sandbox: pending_cleanup /
released / retained" — red on all three; green: the owner is released
for `pending_cleanup` and `released`, and kept (retired by the
successor) for `retained`.
- `warm-sandbox-expiry.test.ts` › "leaves the host's idle close of the
warm runner time to finish before stopping its sandbox" — red: the
sandbox stops 3 s after the window; green: it stops only after the
margin. Two existing tests ("keeps a sandbox still inside its warm
window, and wakes when the window ends", "reconciles a warm sandbox left
running across a restart") now expect the deadline with the margin.

Local runs on this branch:

- `server`: `tsc --noEmit` clean.
- `vitest run src/__tests__/warm-sandbox-expiry.test.ts
src/services/heartbeat/instruction-working-copy.test.ts`: 24/24 passed.
- `vitest run
src/services/native-runtime/native-session-executor.test.ts -t "hands
off collection before probing or retiring a warm owner|forgets a warm
owner whose idle close failed"`: 10/10 passed.

Not run: a Daytona wave with this change.

## Risks

- Warm sandboxes now stop up to 2 minutes later than the configured
window. This is a small cost increase per warm sandbox. I chose a fixed
margin over an in-process "is a close running?" check because the stop
is pure SQL. The margin works when two servers run at the same time (the
server that runs the stop may not be the server that owns the Runner),
and it needs no busy-loop or cross-server signal.
- The margin does not cover a re-armed idle timer or a close that takes
longer than 2 minutes. Fresh preparation (fix 1) and the owner release
(fix 3) cover those cases.
- Not run on Daytona. The evidence is from the wave above and the unit
tests.
- In the wave, a `restore_failed` copy-back onto the just-restarted
sandbox was also seen. I did not find its root cause. It is not changed
here.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, test runs) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:47:42 -06:00
DottaandClaude Opus 5.5 2178c3df88 fix(native): keep a slow but live Runner on retry instead of replacing it (#21462)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a recovery read to a remote Runner goes unanswered, the next
attempt of the same run carries `replaceUnansweredRunner`
> - That attempt verified the Runner was alive and then refused to adopt
it anyway. On a resumed sandbox lease the reuse check then failed with
`resumed_not_suspended` and left the Runner running. Otherwise a second
Runner was launched, and its launch fence killed the busy Runner, which
lost the provider turn.
> - After a late reconnect a Runner is often slow, not dead: it is still
replaying the events it saved with no controller attached
> - This pull request makes the attempt check progress first. It adopts
a live Runner that is making progress and replaces only a Runner that
made no progress.
> - The benefit is that a slow but working Runner keeps its turn, and
the retry no longer causes the failure it was meant to fix.

## Linked Issues or Issue Description

**What happened?**

Hosted staging, release swaps on 2026-11-13. The retry after a
first-attempt timeout failed with these codes (counts from the traced
sample of 136 failures):

- `runner_ingress_unavailable: runnerd process failed:
runner_harness_state_mismatch: resumed_not_suspended` (part of 40
`ingress` failures)
- `runner_remote_process_identity_unavailable` (25): the Runner was gone
by the time the retry checked, because the launch fence or a previous
retry had killed it
- "restored the conversation after process loss, but…" /
`provider_turn_lost_on_restore` (10): the replacement Runner restored
from the old state and the provider turn was gone

Runs that crossed a swap with a reconnect gap under 150 s: 0 of 9
failed. Over 420 s: 14 of 15 failed. The process ID of the Runner
changed inside the same sandbox in the failed runs, which shows the
replacement.

Code path: `nativeRecoveryCommandUnanswered` grants an extra attempt
with `replaceUnansweredRunner: true`. `verifyRemoteRunnerOrSettledExit`
verifies the Runner is alive and then returns `undefined` because of
that flag, so `adoptedProcess` is `undefined`.

**Expected behavior**

A live Runner that is making progress is adopted. A Runner that made no
progress is stopped with proof and then restored from its own journal.

**Steps to reproduce**

1. Reattach a remote run whose previous attempt left a recovery read
unanswered (`replaceUnansweredRunner: true`), with the Runner alive and
its saved event sequence past the controller's last acknowledgement.
2. On master the attempt launches a second Runner, or fails with
`resumed_not_suspended` on a resumed sandbox lease.
3. The new tests in `native-session-executor.test.ts` and
`native-runner-restart-recovery.integration.test.ts` do this.

**Paperclip version or commit**

`master` at 426e8a35dc.

## What Changed

- `native-session-executor.ts` (`verifyRemoteRunnerOrSettledExit`): with
`replaceUnansweredRunner`, the attempt adopts the live Runner, launching
nothing and signalling nothing, when:
- the Runner's saved event sequence is past the controller's last
acknowledgement, or
- its saved state (event and command sequences, generation, lifecycle)
changes within a bounded wait of about 10 s.
- The baseline is the first successful state read; every successful read
is also compared against the controller's acknowledgement. A Runner
whose state cannot be read at all during the wait is adopted, not
replaced: missing evidence is not proof of a wedge. A Runner whose
identity changes during the wait fails closed.
- A settled exit during the wait keeps the attempt allowance that
admitted the replacement, so it also restores on the extra fourth
attempt.
- Only a Runner that made no progress is replaced. The relaunch's
existing launch fence stops it and its process lineage with proof, and
the new Runner restores its own journal. A resumed sandbox lease no
longer refuses that relaunch with `resumed_not_suspended`.
- A Runner that exits during the wait follows the existing exited-Runner
rules: a settled suspension restores, and a mid-turn exit stays
fail-closed.
- The wait runs only on this recovery path, never on a normal run start.
- `doc/execution-semantics.md`: describes adopt versus replace.
- The claim flag `replaceUnansweredRunner` is set in claim logic that
another session owns. This PR does not change it; it changes only what
the executor does with it.

## Verification

Red on master, green on this branch:

- `native-session-executor.test.ts`
- `adopts a live Runner that emitted events past the controller's last
acknowledgement`
- `adopts a live Runner whose saved state advances within the bounded
wait`
- `hands a live Runner that made no progress to the relaunch, whose
launch fence stops it, without signalling it here`
- `restores from the journal of a Runner that suspended and exited
during the wait`
- `stays fail-closed for a Runner that exited mid-turn during the wait`
- `native-runner-restart-recovery.integration.test.ts`: `reattaches a
live %s Runner whose recovery read went unanswered on a resumed sandbox
lease` (progressing, wedged)
- Review fixes, `native-runner-restart-recovery.integration.test.ts`
(the first four were red before the fix):
- `adopts a live Runner whose first state read failed and whose later
reads advance`
- `adopts a live Runner whose first readable state is already past the
controller's acknowledgement`
- `does not replace a live Runner whose state could not be read at all
during the wait`
- `restores from the journal of a Runner that suspended and exited
during the wait on the admitted fourth attempt`
- `fails closed when the Runner's identity changes during the wait`
(pins existing behavior)
- `native-remote-retry-admission.test.ts`: unchanged, still passes.

Local runs on this branch: `tsc --noEmit` is clean for `server` on this
branch (master c6b497d831, which contains #21442).
`native-runner-restart-recovery.integration.test.ts`, the
adopt-or-replace block: 12 of 12 pass. The full
`native-session-executor.test.ts` and
`native-remote-retry-admission.test.ts` runs did not finish on this
shared machine (load 20 to 96); CI ran them on this head and they
passed.

Conflict resolved: master added a relaunch branch for a Runner that
exited for a fresh bootstrap (#21374) at the same place. Both branches
are kept; the cheaper check for an unanswered Runner that made no
progress comes first.

Daytona results that exist. They ran on a combined branch with this PR,
#21450, #21442 (merged) and #21362 (merged), so they cannot be credited
to this PR alone:

- G5 baseline (master d1d71a7755; 480 s delay injected on the first
identity write per run): one Claude run reproduced the staging chain:
authentication 419 ms, identity +423.5 s, attempt 1 `native session
provider recovery timed out after 900000ms`, then the retry failed in 6
s with `runner_harness_state_mismatch: resumed_not_suspended`.
- G5c (same cell, combined branch 588c6ba267): 6 of 6 tasks finished
hands-off. The chain did not recur. All runs that crossed swap 2 were
adopted on the same Runner process ID.
- Caveat: the adopt-or-replace retry path that this PR changes was never
exercised in a cell. In G5c no first attempt failed, because all
identity writes started together 4 to 5 s after the claim. G5c also ran
before the phase timing fix in #21450.

## Risks

- Not proven on Daytona: the adopt-or-replace path itself (see the
caveat above).
- Without #21442 (merged) an adopted Runner can still time out on its
first command while it replays a backlog. This PR stops the replacement;
#21442 (merged) makes the adopted Runner answer.
- A Runner whose state cannot be read during the whole wait is adopted.
If it is in fact wedged, the adopted attempt times out again and the
attempt limit bounds it.
- A wedged Runner that still emits a few events counts as progressing
and is adopted. The existing attempt limit bounds this.
- The progress check adds a wait of up to about 10 s on this recovery
path only. A normal run start has no added wait.
- The claim construction (`native-restart-recovery.ts`) is owned by
another session and is unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 18:56:11 -06:00
DottaandClaude Opus 5.5 47a2c82d35 fix(heartbeat): reach the previous server's live runs first after a release swap (#21450)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - During a release swap, the outgoing server stops and the incoming
server takes over the live runs on remote Runners
> - The incoming server reached those runs late: it waited up to 60 s
for the outgoing server's controller lease to lapse, re-verified each
run's whole log from object storage, and deferred a reattach when the
agent directory was busy. Queued starts competed with the reattaches for
the same resources.
> - On hosted staging, the later a live run was reached, the more likely
it was to fail. Runs reached in under 150 s never failed; runs reached
after 420 s almost always failed.
> - This pull request hands live runs over at the start of shutdown,
makes a reattach skip work that the same run already did, and holds
queued starts while handed-over runs are being reached, with a 180 s
cap.
> - The benefit is that live runs are reached within seconds of a swap
instead of minutes, so they continue instead of failing.

## Linked Issues or Issue Description

**What happened?**

Hosted staging, release swaps on 2026-11-13 (06:05Z to 09:45Z). For each
run that crossed a swap we measured the gap from the last Runner output
to the moment the new server recorded the Runner's process identity:

- gap under 150 s: 0 of 9 runs failed
- gap over 420 s: 14 of 15 runs failed
- median gap: 180 s for runs that resumed, 470 s for runs that failed

In one traced run the incoming server waited on
`remote_controller_lease_live` three times (09:14:21, 09:15:18,
09:16:17), claimed at 09:16:59, and spent 85 s in `task.prepare`. Of
that, `heartbeat.prepare.run_output` took 58.7 s, because it re-read the
run's whole log from object storage. A control run spent 23 s in
`task.prepare`. Queued starts used the same preparation path, the same 8
object-I/O slots and the same database pool, and nothing gave a reattach
priority. The first-attempt failures were `PRP command session.snapshot
timed out` and `codex_history_read_failed`. The retry then failed with
`runner_harness_state_mismatch: resumed_not_suspended`,
`runner_remote_process_identity_unavailable` or
`provider_turn_lost_on_restore`.

**Expected behavior**

Every live run is reached within about 30 s of the swap. Queued work
starts after that, or after a bounded wait.

**Steps to reproduce**

1. Run a two-server release swap with live remote runs and queued runs.
2. Observe `native.recovery.transition` "waiting" events with
`remote_controller_lease_live`, then the claim, then a long
`heartbeat.prepare.run_output` span.
3. The new tests in `reattach-phase.test.ts` and
`heartbeat-native-early-hand-off.test.ts` model this.

**Paperclip version or commit**

`master` at 426e8a35dc.

## What Changed

- Early hand-off at shutdown (`heartbeat/recovery.ts`,
`server/src/index.ts`): in deploy-overlap mode, the outgoing server
releases the remote controller lease of its live runs at SIGTERM
(`recoveryState` `awaiting_runner_reattach`). A run another server
already took over is left alone. The incoming server no longer waits for
the lease to lapse.
- Reattach fast path:
- `heartbeat/run-output.ts`, `native-runtime/native-run-log-export.ts`:
a reattach starts without waiting for the existing log to be
re-verified.
- `heartbeat/agent-directory-busy.ts`, `heartbeat.ts`: a reattach that
meets a busy agent directory waits in process (at most 30 s) instead of
being deferred to a full re-preparation. An ordinary start is still
deferred at once, with no added wait.
- Reattach phase lifecycle (`heartbeat/reattach-phase.ts`): the watch is
a worker on the shared delivery coordinator, queue `reattach-phase`.
- Trigger: one startup listing of a previous server's live runs; then
the hand-over notification (`notifyReattachHandOver`, sent in the
hand-over transaction); the earliest lease deadline it watches (a killed
owner); and a 2 s re-check only while handed-over runs are still
pending, which ends with the phase (cap 180 s). The bounded re-check
exists because a run can also end on another path that sends no
notification to this queue.
- Source of truth: `native_run_finalizations` and `heartbeat_runs` rows.
- Quiet condition: nothing pending and no watched lease, so no deadline
and no SQL. It stops with the coordinator at shutdown.
- Reached hook for every driver (`native-session-executor.ts`): a
reattached run counts as reached when the native session runtime hands
the executor its recovered or opened session (`onSession`). Every driver
reaches that point only after the Runner answered this server's attach
and recovery commands, including an adopted `RunnerdExternalDriver`
Runner, which records no new process identity. The identity hook stays
as an earlier signal.
- Claim wake: when the reattach-phase worker finds runs that just became
claimable (hand-over notification or a killed owner's lease deadline),
it starts this server's restart-recovery claim for them at once. A
validation cell showed the claim otherwise waited about 21.5 s after the
hand-over.
- `recovery/assigned-work-sweep.ts`: the `deferred_reattach_phase`
return now sits before every action path of `act()`, except starting an
overdue saved-work recovery (it resumes an existing run).
- The in-transaction admission re-check of a queued claim
(`confirmRunAdmissionStillOpen`) also honours the reattach phase. A
phase that begins during a claim returns the run to the queue through
the existing rollback.
- A reattach whose log re-verification runs in the background awaits it
before the log is finalized (`settleDeferredLogRecovery`, both
finalization sites in `run-completion.ts`).
- Reattach phase (`heartbeat/reattach-phase.ts`,
`heartbeat/run-admission.ts`): `isReattachPhaseActive()` is true while
runs that a previous controller handed over (lease released or observed
expired, or claimed by this server's restart recovery) are claimable but
not yet reached. A run is reached when its process identity is recorded.
Authentication is logged as a mark. The cap (180 s,
`PAPERCLIP_REATTACH_PHASE_CAP_MS`) counts from the moment the runs
became claimable. Later batches join or re-arm the phase. Runs reached
after 150 s are counted and logged.
- Gates on the phase:
- `claimQueuedRun`, start-next, resume-queued and dispatch in
`heartbeat/queue.ts`, `run-lifecycle.ts` and
`native-runner-execution-driver.ts` leave queued runs queued.
- `execution-recovery-resolution.ts`: one line, the first line of
`deliverReconciledExecutions`.
- `recovery/assigned-work-sweep.ts`: one line, outcome
`deferred_reattach_phase`.
- `heartbeat/recovery.ts`: a one-line claim hook
(`noteReattachRunsClaimed`) after restart recovery claims runs.
  - The owner of those three files agreed to these one-line gates.
- `tests/restart-chaos/daytona-image.ts`:
`PAPERCLIP_CHAOS_TEST_ACCEPT_IMAGE_CONTENT_ID` lets a local wave run a
controller-only branch against an existing Daytona image. It accepts a
content-ID mismatch only for the exact content ID of the named image,
and it prints a TEST OVERRIDE warning. This is harness-only.

## Verification

Red on master, green on this branch:

- `heartbeat/reattach-phase.test.ts` (27 tests). Five of the six
two-server scenarios were red on the earlier phase timing:
- `does not hold queued starts while the previous owner is alive and not
handing over`
- `closes the gate when the previous owner hands its runs over, and
counts reach latency from the hand-over`
- `runs the cap from the hand-over, so a wedged run ends the phase
without the overlap eating it`
- `begins when a killed owner's lease is observed expired, without any
hand-over`
- `re-arms for a later batch, including runs the previous owner started
after discovery`
- `closes the gate at once when this server claims a run between two
lease reads, dated from the hand-over`
- plus `claimQueuedRun leaves a queued run queued and unstarted, in the
admission gate's shape`, `is the same gate the assigned-work sweep and
reconciled continuation delivery read`, `ends at the cap, counted from
when the runs became claimable, so one wedged Runner cannot hold run
starts`
- `heartbeat-native-early-hand-off.test.ts`: `releases the remote
controller lease of %i live runs at once` (1, 20, 40 runs; 40 runs in 56
ms), `leaves a run another server already took over alone`, `holds the
incoming server's queued starts only from the hand-over, not while the
outgoing server still drives its runs`
- `heartbeat/run-output.test.ts`: `lets a reattach start before its
existing log is re-verified`
- `heartbeat/agent-directory-busy.test.ts`: `defers an ordinary start at
once, with no wait added to the run-start path`, `lets a reattach wait
its turn in process instead of a deferred full re-preparation`, `still
defers a reattach once its bounded wait runs out`
- `tests/restart-chaos/daytona-image.test.ts`: `accepts a mismatch only
for the exact image content ID named by the test override`
- Review fixes (red before the fix, green after):
`reattach-phase.test.ts` `starts the claim of handed-over runs within a
second of the hand-over notification`; `assigned-work-invariant.test.ts`
`takes no action while handed-over live runs are being reached, and acts
once the phase ends`; `heartbeat-native-early-hand-off.test.ts` `never
lets the outgoing server claim back a run it handed off, while the
incoming server may take it over`; `reattach-phase.test.ts` `retries a
failed first listing instead of going quiet, and still finds the
previous owner's runs`, `keeps a hand-over notification whose listing
failed until a listing succeeds`, `moves the cap earlier when a claimed
run turns out to have been handed over before its claim`;
`queue.test.ts` `returns the claim to the queue when a reattach phase
begins while the claim is being checked`; `run-completion.test.ts`
`finalizes the run log in completeRun|failRun only after a reattach's
deferred log recovery settles`. `native-session-executor.test.ts` `ends
the reattach phase once a reattached %s session is driven by this server
again, without the cap or manual settling` over codex_app_server,
opencode_server, acpx_runtime, muse_external, grok_bot_mcp and
openai_dot_mcp (6 of 6 red without the hook; the test drives the
executor with a mocked session runtime, not each real driver). Lifecycle
tests for the new worker: `does no database work and keeps no deadline
on an instance with nothing to reattach`, `waits for the hand-over
notification or the lease deadline instead of polling while the previous
owner drives its runs`, `begins at a killed owner's lease deadline,
without any hand-over or notification`, `stops with the delivery
coordinator at shutdown and ignores a later hand-over`.

Local runs on this branch: `tsc --noEmit` is clean for `server`. Passing
on this branch: `reattach-phase.test.ts`,
`heartbeat-native-early-hand-off.test.ts`, `run-output.test.ts` (53
tests together), `agent-directory-busy.test.ts`,
`heartbeat-native-restart-suspension.test.ts` (115 of 115, alone), the
new cases in `queue.test.ts` and `run-completion.test.ts`. Under load 30
to 96 on this shared machine, the full `queue.test.ts` and
`recovery-wake-herd.test.ts` runs stalled in their embedded-Postgres
setup; CI supplies them.

Daytona results that exist (local servers with Daytona sandboxes,
release-swap cells). They ran on a combined branch with this PR, #21462,
#21442 and #21362 (merged), so they cannot be credited to this PR alone:

- G5 baseline (master d1d71a7755): SIGTERM to PRP authentication 21.5 to
38 s at swap 1.
- G5c (combined branch 588c6ba267): SIGTERM to authentication 6.2 to 7.7
s at swap 1. All runs that crossed swap 2 were adopted on the same
Runner process. The staging chain (`provider recovery timed out after
900000ms`, then `resumed_not_suspended`) did not recur.
- Caveat: G5c ran before the phase timing fix in this PR. In G5c the
phase opened at process start, its 180 s cap ran out during the overlap,
and swap 2 ended `cap_reached` with 0 runs reached, 1 s before the lease
claims. The timing fix (the phase begins when the runs become claimable)
has not run on Daytona.
- G6c (12 live runs and 20 queued, two graceful swaps, no injection,
combined branch 588c6ba267): 0 failed runs at either swap; 11 of 11
survivors at both.
- Swap 1: hand-off of 11 runs in 291 ms; claims 0.3 to 2.9 s after
hand-off; authentication 7.6 to 12.5 s and identity 8.3 to 13.3 s after
SIGTERM. The 30 s target was met.
- Swap 2: hand-off took 8,562 ms and `hot_restart_prepare` timed out at
19 s, so the outgoing server drained gracefully. The incoming server did
not claim on the hand-over; it claimed only when the outgoing server
exited, 33 s after SIGTERM. Authentication was 46 to 54 s and identity
47 to 57 s after SIGTERM. The target was missed. Why the claims waited
for the exit is not explained.
- The host load was 49 to 70 from other work. 7 of 32 tasks timed out in
the queue or budget after the swaps; we suspect the overload.

## Risks

- Product trade-off: while handed-over live runs are being reached,
queued starts wait, up to the 180 s cap. On an instance with no
handed-over runs, nothing waits.
- Not proven: two-server behavior with the phase timing fix and the
notification-driven watch (no cell has run them); warm standby becoming
active; runs that the outgoing server starts after discovery.
- Early hand-off at SIGTERM orphans a server-side exec tool that is in
flight. With #21442 the next controller settles it as interrupted with
an unknown outcome; it is never run again.
- Swap 2 in G6c: the incoming server claimed only at the outgoing
server's exit. That is unexplained.
- Cross-session ownership: the gates in
`execution-recovery-resolution.ts` and
`recovery/assigned-work-sweep.ts`, and the claim hook in
`heartbeat/recovery.ts`, are in files that the restart-recovery work
owns. They are one-line gates agreed with that owner. That owner said
the gate did not need to be inside the claim transaction; that was
before the phase could begin during a claim. After review, the
in-transaction re-check in `heartbeat/queue.ts` also reads the phase,
through the existing rollback. #21427 (open) also edits
`execution-recovery-resolution.ts`, `heartbeat/queue.ts` and
`recovery/assigned-work-sweep.ts`; whichever merges second needs a
rebase. `native-runtime/native-restart-recovery.ts` (claim logic, owned
by the restart-recovery work): a 3-line guard in
`claimNativeRestartRecoveries` returns a run this process handed off at
shutdown as `blocked` / `handed_off_to_next_controller` before takeover
is evaluated, so the outgoing server never claims back a run it handed
off. #21478 (merged) added a sweep path that starts an overdue
saved-work recovery; it is deliberately not gated by the reattach phase,
because it resumes an existing run (reattach-like work), which the phase
makes room for rather than competes with.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 18:55:31 -06:00
Nicky LeachandClaude Opus 5.5 6d8ebc4a85 fix(secrets): mask secret-bound environment values in run output (#21341)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can bind a company secret to an environment variable for an
agent, an environment, a project, or a routine
> - Core resolves these values for each run and gives them to the agent
process or to the Runner
> - Core stores the run output: the run log, the run excerpts, adapter
events, and native (Runner protocol) transcript events
> - The run-output redactor did not know the resolved secret values.
Only failure diagnostics and the `meta.env` keys of `adapter.invoke`
were masked
> - So when an agent ran `env` or `echo $STRIPE_KEY`, Core stored the
secret in plaintext, and every person who can view the run could read it
> - This pull request gives the resolved values of secret-bound keys to
the run-output redactor and to the native control-plane port
> - The benefit is that an accidental print of a secret does not put the
secret into stored run output, for local and native runs

## Linked Issues or Issue Description

**What happened?**
An agent printed a secret-bound environment variable during a run (for
example with `env` or `echo $STRIPE_KEY`). Core stored the value in
plaintext in the run log, in `stdoutExcerpt` and `stderrExcerpt`, and in
the transcript events. Anyone who can view the run can read the value.

**Expected behavior**
Core masks the value of each secret-bound environment variable before it
stores run output. Core does not mask plain (non-secret) configured
values.

**Steps to reproduce**
1. Create a company secret and bind it to an agent environment variable
with `secret_ref`.
2. Configure the agent so that its command prints the variable.
3. Run the agent and open the run log. The value shows in plaintext.

**Related**
Refs #21330. That PR keeps configured secrets out of OpenCode's own
storage and model requests inside the Runner. This PR is the Core side:
it masks secrets in the run output that Core stores, for every run type.
The two changes are independent.

## What Changed

- `secretEnvironmentValues(env, secretKeys)` in
`agent-identity-redaction.ts` selects the raw values of keys in
`secretKeys` (the keys that came from `secret_ref` or `user_secret_ref`
bindings). `secretOutputForms` derives the forms to mask: the full
value, the value without outer whitespace, the JSON-escaped form, and
each line of a multi-line value (PEM armor lines are not added). It
skips a form that has fewer than 8 characters after trimming. This is
the same 8-character limit that diagnostic redaction uses for unknown
values.
- The output redactor keeps these secrets in a separate set, beside the
existing agent private-key values. The private-key values are masked
everywhere, as before. Environment secrets are masked only in output
text. They are not masked in protocol fields: `schema`, `type`, `kind`,
`status`, `state`, `outcome`, `stream`, `channel`, `role`, `level`, ids,
refs, codes, revisions, versions, digests, timestamps, and names that
end in these words (for example `eventType`, `turnTerminalState`,
`reportedWorkDisposition`). So a secret that equals a protocol word (for
example `completed`) cannot change an event type or a terminal state.
- The streaming match now scans left to right. At each position it tries
the longest value first. If a longer value could still start at that
position, it holds the text until the next chunk, also when a shorter
value already matches there. So the tail of a longer secret cannot leak
when a shorter secret is its prefix.
- `prepareHeartbeatExecutionConfiguration` adds the resolved secrets to
the run's output redactor immediately after it resolves them. This one
redactor already masks the local adapter log stream, the excerpts, live
log events, adapter events, `adapter.invoke` metadata, the adapter
result, and the canonical native log export.
- For a native run, `executeHeartbeatRuntime` stores the delivered
secrets (only values that have a form to mask) encrypted under a new
context key, `paperclipOutputSecretRedactions`, in one transaction. It
then adds all stored values of the run to the redactor. A run that
reconnects to a live Runner after a `"latest"` secret rotation then
masks the value that the Runner holds and the new value. This store
reuses the encryption and fingerprint logic of the run redaction
registry, but it is separate: API read redaction does not apply its
values, so a secret cannot change ids, event names or terminal states in
API responses. API reads hide the new key like the existing registry
key.
- `executeHeartbeatRuntime` gives the native executor the run's secrets
and the observability credentials (`observabilitySecretValues`).
- The native executor gives the secrets to its failure-diagnostic
redactor and to `PaperclipControlPlanePort` (`secrets`). The port masks
them in each PRP event before it persists the event, with the existing
delta-aware handling for values that are split across `item.delta`
events. This covers native runs on the local Runner and on plugin-hosted
agent computers.
- After a controller restart, a new control-plane port cannot know the
text that the old port held back, because that text is never stored. On
the first delta of each stream it has not seen, the port masks leading
text that matches the tail of a secret (4 or more characters). It skips
text that can start a secret, and it keeps checking the stream while a
delta falls entirely inside such a tail.
- The native executor classifies failures from the original error. It
masks only the agent private key in place, as before. It masks
environment secrets in a separate redactor that it uses only for text
that it writes or logs (`failureDetail`, failure messages, stderr
diagnostics). Core masks the rethrown error before it stores it.
- `coalesceIntoActiveExecutionRun` (wake queue) writes the merged
context with `withRunRedactionHistory`, which takes
`paperclipSecretRedactions` and `paperclipOutputSecretRedactions` from
the locked current row. Before, a coalesced wake built from an older
context could delete either key. This also protects the existing
registry. This is a change outside the secrets module, in the one place
that replaces a running run's context concurrently.
- When an adapter or native run fails, the heartbeat keeps the original
error for retry decisions and rethrows it. The copies that it saves in
`workspace_finalize` metadata (`errorMessage`) and in a failed session
goal action go through the run's output redactor first.
- `docs/deploy/secrets.md` describes the mask and its limits.

## Verification

- `server/src/__tests__/run-output-secret-env-redaction.test.ts` (new,
embedded Postgres): a real `process` adapter run with a `secret_ref`
bound to `"latest"`. The agent prints the secret split across two
writes, in JSON, and on stderr. The stored log, run row, and run events
do not contain the value. The plain value and a 5-character secret are
not masked. The test then rotates the secret and runs again. The second
run masks the rotated value, and a SHA-256 digest in the output proves
the run received the rotated value. Red-green: it fails without the new
line in `execution-configuration.ts`.
-
`server/src/services/native-runtime/paperclip-control-plane-port.test.ts`:
the port masks a secret in a persisted tool event and in a value split
across two `item.delta` events; plain and short values stay (red-green:
fails without the port change). A new test binds secrets equal to
`interrupted`, `completed`, `succeeded` and `satisfied`, then appends a
`turn.interrupted` event, checkpoints, and completes the run. The event
type, payload status, terminal states and criterion status stay
unchanged and the result is saved (red-green: fails when protocol fields
are masked).
- `server/src/services/heartbeat/runtime-execution.test.ts`: a value
stored by an earlier attempt, the current value, and an observability
header value all reach native dispatch and the heartbeat redactor. Short
values are not stored. The shared run redaction registry stays empty,
and an API read hides the new key (red-green: fails without the store
read).
- `server/src/services/native-runtime/native-session-executor.test.ts`:
a native failure that contains the secret is masked in failure writes,
logs and the rethrown error, and the executor gives the secrets to the
control-plane port.
- `server/src/services/agent-identity-redaction.test.ts` (new): value
selection, the minimum length, multi-line and JSON-escaped forms, nested
values, every split point of a streamed value, overlapping secrets
(`abcdefgh` and `abcdefgh1234567890`) split at the shorter value,
protocol fields in events, terminal states and deltas, and secrets added
by a later attempt.
- Restart, retry and coalescing (each fails without its fix): a port
test restarts between two deltas of a secret and stores `echo ` and
`***REDACTED*** done`; an executor test binds `timed out` and keeps
`provider_session_open_timeout`; a wake-queue adapter test keeps both
redaction keys when a coalesced wake replaces the context.
- `server/src/services/heartbeat/run-output.test.ts`: the local output
path masks the log, excerpt, live event, adapter event, and adapter
metadata.
- `pnpm --filter @paperclipai/server typecheck` passes. The changed test
files and the other users of the redactor (`agent-identity`, `run-log`,
`heartbeat-case-handoff`, `native-run-log-export.integration`) pass in
full locally: 1986 passed, 7 skipped; the 6 run-redaction-registry test
files also pass (231 tests) (macOS, embedded Postgres). `pnpm verify`
exits 2 (broader verification required) because `docs/deploy/secrets.md`
is a global CI input, so required CI supplies broad coverage.

## Risks

- This is a guard against accidental leaks. It is not a security
boundary. An agent can change a value before it prints it (for example,
encode it in base64 or print one character at a time), and Core cannot
detect that.
- Core does not mask a secret value or line that has fewer than 8
characters. This prevents a short value from erasing ordinary output. A
secret that short is not masked in run output.
- Core does not mask environment secrets in protocol fields (see What
Changed). If a secret appears in such a field, it stays. These fields
hold protocol words and generated identifiers, not agent output.
- If a value is common text of 8 or more characters, Core masks that
text wherever it occurs in that run's output text. This only affects
runs that bind that secret.
- If a key is secret-bound in one source and a later source sets the
same key to a plain value, `secretKeys` still includes the key, so Core
masks the plain value. This is the existing meaning of `secretKeys`.
- For native runs, Core now stores the delivered secrets encrypted in
`contextSnapshot` under `paperclipOutputSecretRedactions`, with the same
local encryption as the existing run redaction registry. It adds one
transaction per native run that has secrets. API reads do not apply
these values and hide the key.
- After a controller restart in the middle of a secret, a tail shorter
than 4 characters can still be stored. A new stream that starts with the
tail of a secret (4 or more characters) is masked at its start even
without a restart. This is a small amount of extra masking.
- Core masks values at write time. Run output that Core stored before
this change is not changed.
- No schema change and no migration. The new fields are optional and
in-process only.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 16:14:41 -08:00
defed64af9 fix(native-runtime): retry a native failure write after a brief lock timeout (#21494)
An interrupted native turn records its same-run resume in one transaction
under a 1 s lock_timeout. When another transaction briefly held a row that
the write needs (for example the agent row that the new recovery action
references), the insert failed with 55P03 and the whole write rolled back.
The run kept its dead lease, and the task needed reconciliation instead of
the recorded automatic resume.

Retry the whole failure write on transient lock contention with the
existing bounded helper. A rolled-back attempt has no effect, so a fresh
attempt is safe. The regression test holds the agent row until one attempt
is cancelled by its lock timeout, so only a retry can complete the write.

Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:02:04 -08:00
Nicky LeachandClaude Opus 5.5 5df1ada9c1 feat(plugins): send configured environment on plugin task resume (#21352)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #21335 lets a plugin task provider that restarts the Runner on
`resume` opt in (`resumeCredentialVersion: 1`) to receive the claimant
run's harness credential
> - A provider restarting the Runner also needs the agent's configured
environment, which Core sends on `resume` only during a session handoff
> - Without it, a Runner a provider moves to a new computer starts
without the agent's configured environment variables
> - This pull request widens that opt-in to both launch inputs, under a
name that says so, before any provider adopts the old name
> - The benefit is that a provider can restart a parked Runner on resume
with the same credential and environment a `submit` would carry

## Linked Issues or Issue Description

Refs #21289
Follows: #21335

**What existing behavior does this improve?**
The opt-in resume launch inputs added in #21335.

**Subsystem affected**
Plugin SDK environment task schema (`describe` capability), the plugin
Runner task owner's `resume`, and the plugin environment task docs.

**Current behavior**
With `resumeCredentialVersion: 1`, `resume` carries `credential`.
`configuredEnvironment` is sent on `resume` only for a session handoff.

**Proposed behavior**
The capability is renamed `resumeLaunchInputsVersion: 1`. With it,
`resume` carries `credential` as before and also `configuredEnvironment`
(same builder as `submit`) when the provider advertises
`configuredEnvironmentVersion: 1`. Session handoff is unchanged.

**Reason and benefit**
A provider that restarts the Runner on resume gets both launch inputs it
needs.

**Breaking changes**
The capability name changes from `resumeCredentialVersion` to
`resumeLaunchInputsVersion`. No provider advertises the old name yet (it
merged in #21335 and has no adopter), so nothing changes behavior.

## What Changed

- Rename `resumeCredentialVersion` → `resumeLaunchInputsVersion` in the
SDK `describe` result schema, executor plumbing, owner input, docs and
tests.
- With the capability, `resume` also adds `configuredEnvironment` when
the provider advertises `configuredEnvironmentVersion: 1`. A normal
resume does not mark the variables as delivered, since a live Runner
keeps the environment it started with.
- Docs: the "Release and resume" section describes both fields; the
"Configured environment" section points to it.
- The runner-e2e SDK public-input pin is updated.

## Verification

- SDK vitest `tests/environment-task-release-resume.test.ts` and
`tests/environment-tasks.test.ts`: 68 passed.
- Server vitest `plugin-runner-task.test.ts` and
`environment-task-runtime.test.ts`: 153 passed, including a normal
resume with the capability (credential sent, variables added only with
`configuredEnvironmentVersion: 1`, delivered flag unset) and redaction
of both fields from metadata writes and sanitized errors.
- `native-session-executor.test.ts -t "uses a task-only"`: 5 passed.
- runner-e2e `source-sdk-evidence.test.ts` and
`source-sdk-public-type.test.ts`: 46 passed.
- `pnpm typecheck` in `server/` (builds shared and plugin-sdk) passed.
- Not run locally: the full suite and `pnpm verify`; required CI covers
them.

## Risks

- A provider restarting the Runner gets the claimant run's current
variables, which can differ from the ones the parked Runner started
with. That is intended.
- Rollback: revert this pull request. No migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs), implemented by a sub-agent and reviewed
before opening.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 13:46:37 -08:00
4cf04d64cb fix(native): retry a transient endpoint failure after a same-run bootstrap restart (#21488)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime starts a Runner in a remote sandbox for each
agent run. A bounded same-run retry (at most 3 startup attempts) handles
transient startup failures.
> - A server restart can stop a run during its first Runner bootstrap.
Restart recovery then claims `bootstrap_incomplete`. It stops and
quarantines the unfinished bootstrap and launches a fresh bootstrap on
the same run.
> - The executor builds its "unclaimed bootstrap" proof only when
`restartRecovery` is not set. Thus the fresh bootstrap of a
`bootstrap_incomplete` restart can never prove that a transient failure
is safe to retry.
> - A transient HTTP 502 from the Runner endpoint request in that fresh
bootstrap was then classified as `ambiguous_state`. The run stopped and
needed manual board recovery.
> - This pull request gives the unclaimed-bootstrap proof to a
`bootstrap_incomplete` restart as well as to a first attempt.
> - The benefit is that a transient endpoint failure after a restart
uses the bounded automatic retry and does not need a person to recover
the run.

## Linked Issues or Issue Description

Refs #20414 (related: it fixes executor admission of a
`bootstrap_incomplete` restart at state-root migration. It does not
change the retry evidence that this PR changes.)

**What happened?**
A server restart interrupted the first Runner bootstrap of a remote
native run. Restart recovery claimed `bootstrap_incomplete` and launched
a fresh bootstrap on the same run. The fresh bootstrap registered its
controller checkpoint. Then `runner.ingress.acquire` failed with HTTP
502, before any Runner connection or provider event. The run ended with
`nativePhase: terminal_failure`, `recoveryOwner: board`, attempt 1 of 3,
and the next action "resolve the ambiguous session state". The error was
`Runner connection endpoint could not be acquired (HTTP 502).`
(`runner_ingress_acquisition_failed`).

**What did you expect to happen?**
The transient 502 occurred before the Runner connected and before any
provider work. It should use the bounded same-run startup retry, as it
does on a first attempt.

**Steps to reproduce**
1. Start a native run on a remote sandbox target.
2. Restart the server during the first Runner bootstrap, so that restart
recovery claims `bootstrap_incomplete`.
3. Make the next `runner.ingress.acquire` call return HTTP 502.
4. See that the run stops as ambiguous state and does not schedule a
startup retry.

## What Changed

- Add `attemptLaunchesFreshBootstrap()` to `native-bootstrap-proof.ts`.
It returns true for a first attempt and for a `bootstrap_incomplete`
restart. It returns false for `reattach_existing_runner`,
`reattach_remote_runner` and `resume_dead_runner`, because these
continue an existing Runner or provider session.
- In `native-session-executor.ts`, use this helper in place of
`!input.restartRecovery` when the executor builds the
unclaimed-bootstrap evidence for a startup failure.
- Add `native-bootstrap-proof.test.ts` for the helper.
- Add a case to `native-startup-retry.integration.test.ts`. It runs the
real executor under a `bootstrap_incomplete` restart claim, and the
fresh bootstrap gets a typed HTTP 502 from the Runner endpoint before
the Runner connects.

## Verification

- `native-bootstrap-proof.test.ts`: the test "accepts a same-run restart
of an incomplete bootstrap" failed before the fix and passes after the
fix (5/5 tests pass, vitest 5.1.0).
- `native-startup-retry.integration.test.ts`, case "transient 502 after
a bootstrap_incomplete restart": the run must settle as
`retryable_failure` with `recoveryMode: "bootstrap_retry"`, attempt 1
and a set `nextAttemptAt`. Without the executor change, the executor
builds no bootstrap evidence and the run settles as `ambiguous_state`.
CI runs this suite; I could not run it locally.
- The other guards for a retry did not change. The checkpoint must be
unclaimed, there must be no provider events and no session checkpoint,
and the 3-attempt startup budget applies. If the earlier attempt opened
a provider session, provider events exist and the run still fails closed
as ambiguous.
- I could not run the full typecheck and the real-Runner integration
suites locally (no Rust toolchain in the sandbox). CI must run them.

## Risks

- Low risk. The change only affects a `bootstrap_incomplete` restart
whose fresh bootstrap fails during startup. Such a failure now gets the
same retry decision as a first attempt. All the other evidence checks
still apply, so a run with provider activity still stops for manual
recovery.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, in Claude Code with
tool use (shell, file edit, and code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 13:37:24 -08:00
Devin FoleyandPaperclip 12aa08721c feat(ssh): use the agent AI account in new environments (#21248)
Use the agent-selected AI account for new SSH environments. Preserve
remote-machine authentication for existing environments and expose it as
an advanced option. Keep managed credentials private while preserving
supported conversation state and credential refreshes across turns.

Validated with focused route, adapter, lifecycle, and real-process tests;
final CI passed on d634944df1.
Greptile Apex reviewed that revision at 5/5 with no unresolved threads.
External neutral security reports remain documented in the pull request.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-13 13:20:13 -08:00
DottaandClaude Opus 5.5 11f4a76743 perf(native-runtime): share one frozen assigned tool catalog across sessions (#21482)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process controls every native agent session. Each session
gets the app tools (MCP tools) that are assigned to its agent.
> - Each live or warm session kept two private copies of its assigned
tool catalog: the gateway descriptors and a canonical copy of every
input schema.
> - With 400 tools of about 5.7 KB each, that was about 7.6 MB of heap
for each session. With 800 tools, it was about 15 MB. Warm sessions stay
for 5 minutes after the last turn, so idle agents also hold this memory.
> - At 1,000 concurrent agents, this is 7 to 15 GB of heap, and the
server cannot get there.
> - This pull request keeps one deeply frozen catalog for all sessions
of the same company that get the same admitted tools and schemas.
> - The benefit is that each additional session on the same catalog adds
about 40 to 80 KB of heap instead of 7.6 to 15 MB, and session start is
faster.

## Linked Issues or Issue Description

Refs #21415 (releases a finished session's tool authority). Refs #21440
(adds the catalog memory benchmark that this change extends).

**What existing behavior does this improve?**
`createAssignedMcpTools` in
`server/src/services/native-runtime/assigned-mcp-tools.ts`. Each call
kept every gateway descriptor (with its provider metadata) and a
separate canonical clone of each input schema. Each `definitions()` call
also made a new `structuredClone` of every schema.

**Subsystem affected**
Native runtime: assigned app tools (MCP) for native agent sessions.

**Current behavior**
Retained heap grows by one full catalog for each session. Measured with
the real `createAssignedMcpTools` (5.7 KB schemas): 400 tools is 7.58 MB
per session, and 800 tools is 15.1 MB per session, for 1, 10, and 100
sessions. Each `definitions()` call takes 14 to 41 ms because it clones
every schema.

**Proposed behavior**
Sessions of the same company that get identical admitted tools share one
frozen catalog. The total for 100 sessions is about one catalog: 4.2 MB
at 400 tools and 8.1 MB at 800 tools. `definitions()` takes 0.01 ms.

**Reason and benefit**
The server must hold 1,000 concurrent agents in one process. The catalog
was the largest per-session heap cost. Revocation and authorization do
not change: every call is still authorized again by the gateway, and
every new session still reads a fresh listing.

**Breaking changes**
None in the API or data. The direct definitions that `definitions()`
returns are now frozen. A consumer that writes to a returned definition
or schema gets a `TypeError` and does not change other sessions. No
production consumer writes to them (see Risks).

## What Changed

- `server/src/services/native-runtime/assigned-mcp-tools.ts`:
- A session keeps only a small frozen record for each tool (gateway
name, display name, description, risk), plus the frozen schema and the
frozen definition. It does not keep the gateway descriptors.
- New per-process index of shared catalogs. The key is the SHA-256 of an
identity string: `[companyId, [[assignedName, gatewayName, displayName,
description, risk, sha256(JSON of the input schema)] for each listed
tool, in listing order]]`. On a hit, the code also compares the full
identity string, so a hash collision cannot give a wrong catalog.
- The identity is computed from the fresh listing that the gateway
returns for this session. The gateway applies all per-agent, per-run,
per-profile, and policy filtering, and the approval description suffix,
before it returns the listing. Thus two sessions share only if they were
given exactly the same tools, names, descriptions, risks, and schemas in
the same company. Without `companyId`, nothing is shared.
- Schemas, definitions, tool records, and the definition arrays are
deeply frozen. Search results still return cloned copies.
- Sessions that start at the same time with the same catalog wait for
one build and share it.
- The index holds each catalog through a `WeakRef`, so it never keeps a
catalog alive. The index has at most 512 entries and 256 MiB (accounted
schema and description bytes). The oldest entries are removed first. A
catalog that is larger than the byte limit is not indexed.
- New `close()` on the assigned tools. When the last session that holds
a catalog closes, the index entry is removed. A `FinalizationRegistry`
also removes entries whose catalog was collected.
- `definitions()` returns the shared frozen definitions for the
session's work mode. It no longer clones every schema on each call.
-
`server/src/services/native-runtime/paperclip-runner-tool-authority.ts`:
`close()` also closes the assigned tools. The session executor already
closes the authority when it revokes a session's tool authority.
- `server/src/services/native-runtime/native-session-executor.ts`: one
line. It passes the run binding's `companyId` to
`createAssignedMcpTools`.
- `server/scripts/benchmark-assigned-mcp-tools-memory.ts`: passes
`companyId`, adds `--companies`, measures session start and
`definitions()` time, and returns fresh copies of fixed rows (like
database reads).
- Tests in `assigned-mcp-tools.test.ts` and
`paperclip-runner-tool-authority.test.ts`.

## Verification

Red/green tests in
`server/src/services/native-runtime/assigned-mcp-tools.test.ts` ("shared
assigned tool catalogs"):

- Retained heap: 20 sessions of one company on a 60-tool, 4 KB-schema
catalog. The heap for the other 19 sessions must be less than two
catalogs. Red on master: `expected 14025976 to be less than 3788112`.
Green with this change.
- Shared deeply frozen schemas, also for sessions that start together.
Red on master (`Object.is` equality fails).
- Mutation isolation: writes to a returned definition, schema property,
and `required` array, and a delete, throw `TypeError`. Changes to the
gateway's source rows and to a search result do not change any session.
Red on master (no throw).
- Release and bounds: the entry stays while one session holds it and
goes when the last session closes. `close()` is idempotent. A new
session after release builds a new catalog. 517 companies keep the index
at 512 entries or fewer. Red on master (no `close` or stats).
- Guard tests: two companies with identical catalogs do not share.
Sessions without a company do not share. Fewer tools, an edited schema,
an edited description, or a different risk do not share. A revoked or
edited tool is served from fresh discovery. On master these pass,
because master never shares. To prove they test the key, I removed
`companyId` and the schema digest from the identity: three guard tests
failed, and they pass with the full key.
- An existing test changed: "does not project gateway secrets..." now
expects a write to a returned schema to throw `TypeError` (before, the
write changed only a private clone). The isolation check after the write
is the same.
- `paperclip-runner-tool-authority.test.ts`: `authority.close()` closes
the assigned tools.

Heap (post-GC `heapUsed` delta, `node --expose-gc`, 5.7 KB schemas, one
company). Command: `cd server && node --expose-gc --import tsx
scripts/benchmark-assigned-mcp-tools-memory.ts --tools <N>
--schema-bytes 5700 --sessions <S> --turns 5`.

| Tools | Sessions | Total before | Per session before | Total after |
Per session after |
| --- | --- | --- | --- | --- | --- |
| 400 | 1 | 7.60 MB | 7.60 MB | 3.96 MB | 3.96 MB |
| 400 | 10 | 75.9 MB | 7.59 MB | 4.04 MB | 0.40 MB |
| 400 | 100 | 757.9 MB | 7.58 MB | 4.18 MB | 0.042 MB |
| 800 | 1 | 15.2 MB | 15.2 MB | 7.91 MB | 7.91 MB |
| 800 | 10 | 151.4 MB | 15.1 MB | 7.98 MB | 0.80 MB |
| 800 | 100 | 1,513 MB | 15.1 MB | 8.13 MB | 0.081 MB |

The first session also uses about half the heap of before, because the
session no longer keeps the gateway descriptors. When all sessions are
dropped without `close()`, the catalog is collected (0 bytes held after
release).

Session start, interleaved A/B in one process (old and new
`createAssignedMcpTools` alternate, 40 starts each, median of the last
35, includes a fresh copy of the listing; the machine had a load average
of 25 to 65):

| Tools | Listing copy only | Before | After |
| --- | --- | --- | --- |
| 400 | 22.6 ms | 68.0 ms | 37.6 ms |
| 800 | 30.5 ms | 72.5 ms | 41.2 ms |

A start that reuses a catalog does no schema clone and no admission. It
still serializes and hashes each schema to compute the key. The first
start of a new catalog does the same admission work as before (the cold
first start was about 600 to 750 ms in both versions on this busy
machine).

`definitions()` median: 14 to 41 ms before, 0.01 ms after.

Other checks:

- `cd server && npx vitest run
src/services/native-runtime/assigned-mcp-tools.test.ts
src/services/native-runtime/paperclip-runner-tool-authority.test.ts
src/services/native-runtime/runner-api.integration.test.ts
src/services/native-runtime/runner-tool-results-prp.integration.test.ts
src/__tests__/deployment-connections.test.ts`: 204 passed.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t
"assigned|tool authority|released|MCP"`: 31 passed.
- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t "assigned"`: 10
passed.
- `cd server && npx tsc --noEmit`: pass.
- `pnpm check:module-boundaries`: pass.
- `pnpm verify` selects a broad plan (the benchmark script is a broad
input), so CI supplies the full suite.

## Risks

- Frozen definitions: the direct definitions now go to the runner
drivers as frozen objects. I checked the consumers.
`PaperclipRunnerToolAuthority.definitions()` filters and spreads them.
The Codex, Cursor Cloud, OpenCode, ACPX, and runnerd drivers read them,
spread them, clone them (`structuredClone` in ACPX and the OpenCode MCP
bridge), or serialize them. Their types are `readonly
Readonly<Record<string, unknown>>[]`. The first-party semantic tool
definitions are already shared module constants that go through the same
path without clones. If a consumer that I did not find writes to a
definition, it gets a `TypeError` instead of a silent change. Catalogs
over 256 tools or over the listing budget use the two on-demand tools
and are not affected.
- Authorization: the shared data is only the result of a listing that
the gateway already authorized for that session, keyed on the full
content plus the company. Execution still goes through the gateway with
the session's own token on every call. Search still intersects with a
fresh listing. A new session still reads a fresh listing, so a revoked
or edited tool takes effect for new sessions and for existing sessions
at the same points as before.
- Several server processes: the index is a per-process memo of the
listing content. Each process builds its own catalog from its own
listing. Nothing is stored or shared outside the process.
- The index is bounded (512 entries, 256 MiB) and holds catalogs only
weakly. Index eviction only stops future sharing. It never frees memory
that a session uses.
- No schema change. No migration.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs, heap and timing benchmarks).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:36:30 -06:00
aa4a06653c fix(native): name the recorded turn failure when a native run fails (#21235)
Failed native runs now report the runner's recorded turn-failure code instead of the generic unclassified fallback, and the Sentry run-failure fingerprint includes runner:<code>.

Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 11:22:55 -08:00
DottaandPaperclip 6ad4529c5a fix(native): preserve pack proof across interrupted reattachment (#21246)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server can connect again to a live Runner after a restart.
> - Each new controller claim changes provider-pack admission to
pending.
> - The claim kept the previous verified proof only in local memory.
> - If recovery failed before its callback, the next claim lost that
proof.
> - This pull request saves a small reference to the verified proof with
the claim.
> - The next claim can check the same receipt without weakening
admission checks.

## Linked Issues or Issue Description

**What happened?**
A restart recovery attempt timed out before its admission callback. The
next attempt refused the saved pack with `source_state_pending`. It then
failed with `native_provider_artifacts_admission_unverified`.

**Expected behavior**
The next attempt can reuse the previous verified proof for the same
receipt, session, Runner, and lease. Current admission must remain
pending until its normal callback succeeds.

**Steps to reproduce**
1. Save verified provider-pack admission for a run.
2. Claim that run again. Commit the pending admission state.
3. Interrupt recovery before the session admission callback.
4. Claim the same run again and reconnect to the same Runner.
5. Observe that the old code refuses the saved receipt.

Related work: #20682 and #20850 address lock waits. #21085 adds refusal
logs. #21161 addresses command waits after restart. This change
addresses lost admission proof across interrupted claims. The original
transport timeout remains a separate investigation.

## What Changed

- The admission module prepares the claim patch and its recovery source.
- The claim saves a versioned reference with the source run, receipt
digest, and previous verified state. It adds no copy of provider files
or credentials.
- A later pending claim can use that reference only if the receipt is
unchanged. Normal company, agent, session, Runner, and lease checks
still apply.
- Missing proof, unknown state, a changed receipt, or a different Runner
stays refused. The callback clears the reference when it records its
decision.
- The executor uses this helper under its existing claim lock. Existing
artifact-retention and ownership checks stay in place.
- The execution guide states the previous-release behavior. Old verified
records seed the reference. Old pending records without a reference
remain refused. Older readers ignore the field and keep their existing
refusal behavior. No migration is needed.

## Verification

- The new interrupted-claim regression failed before the fix: admission
returned false instead of true.
- Focused admission, PostgreSQL fencing, and executor lease tests passed
after the rebase: 61 tests on commit c8123cc35d.
- The database case saves two interrupted claims, reads their persisted
state, checks a different Runner is refused, and publishes verified
admission for the original Runner.
- Negative cases cover missing proof, unknown state, a changed receipt,
a different source run, a different Runner, and a different lease.
Existing company and claim fences also pass.
- `pnpm verify` returned exit 2 because the execution guide requires
broader CI. Required CI must cover the final commit.
- Local server typecheck was killed with SIGKILL, including its
single-thread retry. This is an environment gap. Required CI must
provide typecheck and build results.
- Repository preflight installed dependencies and built shared packages.
It could not finish Runner preparation because cargo is absent.
- A live OpenCode restart workflow and recovery of the original task are
not verified. No task replay or deployment was performed.

## Risks

- The added profile field carries historical evidence. It does not mark
the current claim verified. Its use is limited to an unchanged receipt
with the normal identity checks.
- Existing pending or unknown incidents without this evidence cannot
gain proof from this fix. They keep their normal recovery path.
- No provider attempt limit, approval gate, workspace hold, or company
boundary changes.

## Model Used

OpenAI GPT-6, through Codex, with code search, tool use, and local test
execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 12:56:57 -06:00
DottaandClaude Opus 5.5 85f4a91e70 fix(native): release a finished session's tool authority so the server heap does not grow per run (#21415)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime starts each agent run in a Runner session. The
server keeps a tool authority for each session scope, so the Runner can
call Paperclip tools and assigned app tools.
> - The server stored that authority in a map that lives as long as the
process. Only an aborted start removed an entry.
> - Each finished session kept its authority and the run's assigned
app-tool catalog. A large app catalog (hundreds of tools with schemas)
is several megabytes per run.
> - Under a busy queue with many short runs, the server heap grew until
the process ran out of memory. Each crash stopped every run that was in
progress.
> - This pull request releases a scope's authority when nothing can use
it again.
> - The benefit is that the server heap stays flat however many runs
finish, so the server does not crash and stop in-progress work.

## Linked Issues or Issue Description

**What happened?**
A hosted server ran out of JavaScript heap ("Ineffective mark-compacts
near heap limit") about 50 minutes after it started, and again after the
next start. The last GC line showed about 3.3 GB still live after a full
mark-compact. Each crash stopped every run in progress. The server
started about 300 native runs per hour, and most of them failed quickly.
Each run start loaded a large assigned app-tool catalog (the server
logged the "over the compile budget" warning with hundreds of tool ids
for each start).

**Expected behavior**
Memory that a run uses is released when the run's session ends. Heap
size does not grow with the number of finished runs.

**Steps to reproduce**
1. Run `executePaperclipNativeSession` with `useRunnerd: true` and a
`per_turn` session, for many distinct sessions, each with an assigned
MCP catalog.
2. Let each run succeed or fail.
3. Read `sessionToolAuthorityEpochs.size` and the post-GC heap. Before
this change, both grow by one entry and about 3 MB for each finished
run.

**Paperclip version or commit**
`master` (and the hosted build that crashed).

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`: new
`releaseIdleSessionToolAuthority(scope)`. It removes and revokes the
scope's tool-authority epoch when no execution, authority
initialization, settle, warm owner, or warm close holds the scope. It is
called:
- at the end of `executePaperclipNativeSession` (after the scope
reservation and the settle mark are released),
  - after a warm session's idle-timeout close,
- after a warm session's close in `closeIdleWarmNativeSessions` (restart
and sleep sweeps).
- A warm session that stays retained keeps its authority, so the next
run in that session still drains the previous run's accounting events as
before.
- `forgetWarmNativeSession` also calls it, so a warm session that a new
session identity supersedes (`retireSupersededWarmNativeSessions`)
releases its authority too (review finding).
- New `sessionToolAuthorityCount()` export, for the regression tests.

## Verification

Red/green regression test in
`server/src/services/native-runtime/native-session-executor.test.ts`:
"releases each finished session's tool authority and assigned tool
catalog (succeeded|failed)". It runs five runnerd `per_turn` sessions
with distinct session ids and an assigned MCP catalog, then checks that
the authority map is back at its starting size.
- Red (fix disabled): `expected 10 to be 5` (one entry stays for each
finished run).
- Green (fix): both cases pass.

Second red/green test, "releases a superseded warm session's tool
authority and keeps only the retained session's": two warm runnerd runs,
where the second has a new session identity and retires the first. Red
(no release in `forgetWarmNativeSession`): `expected 2 to be 1`. Green:
one entry for the retained warm session, and zero after the warm sweep
closes it.

Heap measurement, same harness, 200 runs with distinct sessions, each
with a 300-tool assigned catalog, provider failing fast, mock call
history cleared after each run, post-GC `heapUsed` delta:

| Finished runs | Entries before | Heap delta before | Entries after |
Heap delta after |
| --- | --- | --- | --- | --- |
| 50 | 50 | 150.5 MB | 0 | 16.2 MB |
| 100 | 100 | 300.6 MB | 0 | 16.8 MB |
| 150 | 150 | 452.8 MB | 0 | 17.3 MB |
| 200 | 200 | 605.3 MB | 0 | 18.0 MB |

Before: about 3.0 MB retained for each finished run, linear, never
released. After: flat.

Commands:
- `cd server && pnpm exec vitest run
src/services/native-runtime/native-session-executor.test.ts -t "releases
each finished session"`
- `cd server && pnpm exec vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-session-resume.test.ts`
- `cd server && pnpm exec tsc --noEmit`

## Risks

- A late Runner callback for a session that already ended now gets "tool
authority unavailable" instead of reaching the old, revoked authority. A
finished session has no Runner left, so no supported path makes that
call. A warm session that stays retained keeps its authority.
- A same-run restart recovery in the same process starts with no
previous epoch for an ended session, so it does not drain a previous
run's accounting events. That only applies when the earlier session
already ended.
- No schema change. No migration.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs, heap measurement).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 11:00:09 -06:00
DottaandClaude Opus 5.5 5ba26600ef fix(native-runtime): a remote Runner reattachment continues a turn the Runner can recover (#21362)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime runs agent turns through a Runner, often in a
remote sandbox, and must recover runs after a restart without help from
a person
> - When a Runner process is lost, a new Runner restores the
conversation. If the provider turn is gone, the Runner sends
`turn.failed` with `provider_turn_lost_on_restore` and `recoverable:
true`. The Runner sends this only after it proves that the old process
ended
> - The runtime can continue such a turn in the same conversation, but
only when the server sets `resumeInterruptedTurn`. The server set it
only for a local `resume_dead_runner` claim or a scheduled resume, and a
scheduled remote resume is refused. So on a remote Runner reattachment
the runtime did not continue the turn, and the run ended
> - This pull request sets `resumeInterruptedTurn` for a
`reattach_remote_runner` claim too
> - The benefit is that a remote run continues the lost turn in the same
conversation immediately, instead of failing and then restarting 3 to 6
minutes later

## Linked Issues or Issue Description

**What happened?**
On hosted staging (build `d1d71a7755`, read-only trace), 10 runs in 3
hours ended with `native_provider_terminal_failed` and the message "The
runner restored the conversation after process loss, but the previous
turn is no longer active." The Runner marked each of these events
`recoverable: true`. Example: run 646e2efb. A new Runner process started
at 08:32:04. `session.resumed` came at 08:34:49. `turn.failed
provider_turn_lost_on_restore` (recoverable) came at 08:34:51. The run
failed at 08:35:21. Example: run 11fe2cdc. `session.resumed` came at
09:23:24. `turn.failed` came at 09:23:25. The run failed at 09:23:52. In
each case, a separate recovery path continued the task only 3 to 6
minutes later.

**Expected behavior**
When the Runner marks a lost turn as recoverable after a restore on a
remote reattachment, the run continues the turn in the same
conversation.

**Steps to reproduce**
1. Start a native run on a remote sandbox Runner.
2. Lose the Runner process while a turn is active, so that a new Runner
reattaches through a `reattach_remote_runner` claim.
3. The new Runner restores the conversation and reports
`provider_turn_lost_on_restore` with `recoverable: true`.
4. See that the run ends with `native_provider_terminal_failed` and does
not continue the turn.

**Agent adapter(s) involved**
Not adapter-specific (core bug). Seen with Codex on a Daytona remote
sandbox.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`:
`resumeInterruptedTurn` is now also true for a `reattach_remote_runner`
restart-recovery claim. The runtime still continues the turn only when
the Runner itself sends `turn.failed provider_turn_lost_on_restore` with
`recoverable: true` for the checkpointed active turn.
- `doc/execution-semantics.md`: one paragraph describes this behaviour,
and says that a lost turn that still ends the run keeps
`native_provider_terminal_failed`.
- Tests:
- `native-scheduled-resume-continuation.integration.test.ts`: "continues
a lost turn on a same-run remote Runner reattachment" (2 cases: a plain
reattachment and a `replaceUnansweredRunner` reattachment).
- New `native-recoverable-lost-turn.integration.test.ts`: a guard
through `executePaperclipNativeSession`. The runtime saves a
failed-terminal checkpoint and then raises the lost turn (recoverable
and not recoverable). The run ends as `native_provider_terminal_failed`,
and the hold cause stays one that the replacement lane continues.
- An earlier revision of this PR also mapped a recoverable lost turn to
`native_session_interrupted`. Review showed that this does not make the
run retryable, and the hold it writes is not picked up by any automatic
continuation. That hunk is removed.

## Verification

- Red without the product change: the two integration cases fail with
`passed` = `[false]` (expected `[true]`).
- Green with the change:
- `cd server && npx vitest run
src/services/native-runtime/native-scheduled-resume-continuation.integration.test.ts
src/services/native-runtime/native-recoverable-lost-turn.integration.test.ts`
— 14 passed.
- `cd server && npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`
— passed (with the scheduled-resume file: 224 passed, 14 skipped,
pre-existing skips).
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts` — 1469
passed, 7 skipped.
  - `cd server && npx tsc --noEmit` — exit 0.
- Staging evidence (hosted staging, build `d1d71a7755`, read-only
trace): 10 runs in 3 hours ended with `native_provider_terminal_failed`
although the Runner marked the lost turn `recoverable: true`. Run
646e2efb: new Runner pid 08:32:04, `session.resumed` 08:34:49,
`turn.failed provider_turn_lost_on_restore` (recoverable) 08:34:51, run
failed 08:35:21. Run 11fe2cdc: `session.resumed` 09:23:24, `turn.failed`
09:23:25, run failed 09:23:52. The task continued only 3 to 6 minutes
later through a separate recovery path.

## Risks

- This change has not run live on staging or on a Daytona sandbox yet.
The evidence is the staging trace and the local tests.
- This PR does not introduce or widen the gap where a recoverable lost
turn still ends the run. On master, that run already ends as
`native_provider_terminal_failed`, and the replacement lane continues it
after its decision (3 to 6 minutes on staging). This PR removes the
run-ending case whenever the Runner reports the turn recoverable for the
checkpointed active turn on a remote reattachment. Where the run still
ends (for example, when the continuation itself is lost), the behaviour
is unchanged, and the guard test
`native-recoverable-lost-turn.integration.test.ts` proves that it stays
`native_provider_terminal_failed`.
- The remaining case is tracked with the owner of the recovery path:
recovery evidence will accept `provider_turn_lost_on_restore` /
`recoverable` directly, after their ownership-deadline PR #21364. An
earlier hunk here tried to shorten that case. An integration test
through the real path showed that it would create a board-owned
ambiguous hold that no automatic pass continues, so the hunk is removed.
- This change does not fix why the turn is lost. In staging, a retry
replaced a live Runner. A separate fix for that is in progress.
- The runtime continues a turn only when the Runner sends the
recoverable lost-turn signal. The Runner sends it only after it proves
that the old process ended. So no provider turn runs twice.
- No Runner image change. No schema change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, in Claude
Code with tool use (shell, file edit, code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 07:09:20 -06:00
DottaandClaude Opus 5.5 eb54831d28 fix(native): relaunch a remote Runner that exited for a fresh bootstrap (#21374)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs often use a remote Runner in a sandbox. After a restart,
a deploy or a same-run retry, the server reattaches to that Runner or
starts a new Runner over its saved state
> - A Runner that no controller reaches before its connection lease
expires stops its provider, saves its state as `recoverable_failure`
with `lease_expired_requires_bootstrap`, and exits. The Runner protocol
says that only a new process with a fresh bootstrap ticket can
reconnect, and that the new process continues from the same cursor
> - Reattachment started a new Runner over saved state only for a
settled `suspended` exit (or an exit before `session.open`). So it
refused this exit, and the run ended with
`runner_remote_process_identity_unavailable`
(`native_continuation_requires_reconciliation`)
> - This pull request lets reattachment launch a Runner over the exact
saved state of a Runner that exited to ask for a fresh bootstrap
> - The benefit is that the run continues by itself after a long
controller gap, with no Retry and no manual sandbox repair

## Linked Issues or Issue Description

Refs #21329, #21362, #20853

**What happened?**
On hosted staging (build `2b69cdfcd6`, read-only trace), Codex runs on
remote Daytona Runners ended with
`native_continuation_requires_reconciliation` and the error
`runner_remote_process_identity_unavailable`, about 5 per hour, after
#21329 was deployed. All five sample runs (c6fabe9a, 31d54b02, 995b63df,
dd595934, 50c6e6e6) ended the same way. The last
`native.remote_runner_recovery_failed` event recorded
`process_verification` / `process_not_running`, `runnerLifecycle:
recoverable_failure`, `runnerRecoverableFailure:
lease_expired_requires_bootstrap`, and the Runner diagnostic "connection
lease expired; a fresh bootstrap is required". Since 10:30 UTC, 8 of the
15 runs that ended with this error had this exact exit.

Timeline of run c6fabe9a:
- 10:46 to 10:54: `runner.session.startup` took 476 s in a loaded
sandbox.
- 10:54:02: the controller sent `turn.start`. At 10:54:47 the command
timed out: "PRP command turn.start timed out (runner connected; pending
ahead 0; last runner event seq 2 91s ago)". The run resumed in the same
run and the turn then ran (tools completed from 11:14).
- 11:39: a graceful server shutdown suspended the controller. At 11:43
the next controller started the reattachment, but it did not reconnect
to the Runner until the server restarted again at 11:58. In that gap the
Runner's lease expired and the Runner exited.
- 11:58:53: the reattachment proved the process gone and refused the
`recoverable_failure` state. The run failed at 11:59.

Why `turn.start` timed out while the Runner was connected: the Runner
answers `turn.start` only after Codex answers its `turn/start` JSON-RPC
request. While it waits, the Runner buffers Codex notifications and
commits no events, so the controller sees a quiet connected Runner with
nothing pending ahead. The controller's bound for the `turn.start`
command result is a fixed 30 s. #21329 changed only the later wait for
`turn/started`. In a sandbox that needed 8 minutes to start, Codex
answered later than 30 s. That timeout is recoverable, and the run did
recover from it. This pull request does not change it (see Risks). The
failure that ended the runs is the refused reattachment.

**Expected behavior**
When a remote Runner exits because its lease expired (or its reconnect
grace or deadline ended) and its saved state is intact, reattachment
starts a new Runner from that state and the run continues.

**Steps to reproduce**
1. Start a native Codex run on a remote sandbox Runner.
2. Keep every controller from reaching the Runner for longer than its
connection lease (on staging: a deploy that stalled the reattachment for
about 15 minutes).
3. The Runner saves `recoverable_failure` /
`lease_expired_requires_bootstrap` and exits.
4. The next reattachment fails with
`runner_remote_process_identity_unavailable`, and the run ends.

**Agent adapter(s) involved**
`paperclip_runner` with Codex on a Daytona remote sandbox. The rule is
not provider-specific.

## What Changed

- `remote-runner-recovery-diagnostics.ts`: new
`exitedRunnerRequestedFreshBootstrap` (diagnostic) and
`runnerStateRequestsFreshBootstrap` (saved state). They accept lifecycle
`recoverable_failure` with `lease_expired_requires_bootstrap`,
`transport_reconnect_grace_exceeded` or
`transport_reconnect_deadline_exceeded`. These are the three codes the
Rust Runner records after `shutdown_preserving_cleanup`. The state check
also requires no pending terminal delivery and no warm transition.
- `native-session-executor.ts` (`verifyRemoteRunnerOrSettledExit`): when
the exact Runner process is proven gone and its diagnostic names one of
these exits, the server reads the saved state again. If the state has
the same six identity fields and still names that exit, the reattachment
launches a Runner from it instead of failing. The outbox does not need
to be empty and a turn may be active: the new Runner replays from the
same cursor. The attempt bound is the one the reattachment already
admits (3, or 4 for an unanswered-Runner replacement claim). Other exits
are unchanged.
- `native-session-executor.ts` (host preparation in
`createRunnerdBackend`): a resumed sandbox for a reattachment that did
not adopt a process re-reads the Runner state. When it still has the
same six identity fields and names the fresh-bootstrap exit, preparation
keeps it in place, as for an adopted live Runner. It no longer fails
with `runner_harness_state_mismatch: resumed_not_suspended` (found by
Greptile).
- `doc/execution-semantics.md`: one paragraph describes the rule and the
staging evidence.

## Verification

- Red on `origin/master`: the 5 new "launches a new Runner over the
saved state after ..." cases failed (the reattachment threw
`runner_remote_process_identity_unavailable`). The 7 new fail-closed
cases passed.
- Green: `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "remote
controller restart adoption"` passes 118 tests.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
1712 passed, 21 skipped, 1 failed. The failure ("verifies a live warm
owner before refreshing run authority", warm-session supervision, not
touched here) passes when run alone. It is a timing failure under
full-file load.
- Launch path: "keeps the saved mid-turn state of a remote Runner that
exited for lease_expired_requires_bootstrap when the reattachment
prepares its sandbox" runs `createRunnerdBackend` and
`prepareExternalRunnerState` with a mid-turn Codex checkpoint. Red
before the host-preparation change (`resumed_not_suspended`), green
after. An unknown recoverable failure stays fail-closed.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts`
on the final revision: 1494 passed, 7 skipped.
- `cd server && node scripts/typecheck-layers.mjs` (includes test
programs): passes.
- Fail-closed cases covered: an unknown recoverable failure, a warm
transition, a pending terminal delivery, a state that changed after the
exit, a different Runner instance, attempt 4 on an ordinary claim, and
attempt 5 on a replacement claim.
- Staging read (read-only): the 5 sample runs and 69 runs with this
error in the last 9 hours were classified from their
`native.remote_runner_recovery_failed` events.

## Risks

- Medium. Reattachment now starts a new Runner over a saved state that
can be mid-turn. The Runner itself stops its provider before it saves
this state. The new Runner restores the session. It continues the turn,
or it reports the lost turn as recoverable only after it proves that the
old provider process ended. A recoverable lost turn on a remote
reattachment ends the run as `native_provider_terminal_failed` until
#21362 merges; the replacement lane then continues the task. That is
still better than the current reconciliation hold.
- The attempt bound does not grow. A Runner that exits this way again on
a later reattachment is bounded by the same provider attempt budget.
- This does not change the 30 s `turn.start` command bound or the deploy
gaps that let the lease expire. Those are separate follow-ups. 57 older
runs with this error had a different exit (`ready`, not settled). This
pull request keeps that case fail-closed; #20853 covers the
settled-`ready` part.
- No schema, API, UI or Runner image change.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, in Claude
Code with tool use (shell, file edit, code execution, read-only browser
reads of staging run data).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:53:38 -06:00
DottaandClaude Opus 5.5 a090a6a932 fix(runner): stop a model from resending the same rejected tool call (#21360)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents act through Paperclip tools such as `call_api`, and the tool
layer refuses an invalid call before it sends anything
> - Some model hosts drop optional tool arguments. A `PATCH` call then
arrives without `body` every time, even when the model says it added it
> - The refusal already names the missing argument, but nothing stops
the model from resending the same call. In one recorded run the agent
sent the same rejected call 12 times in 80 seconds and never finished
the task
> - This pull request makes the tool layer recognise a repeated
identical validation failure, correct it once with the exact arguments
that arrived, and then stop running that exact call for the rest of the
run
> - The benefit is that a dropped argument ends in a clear correction or
a reported blocker, not a long loop that burns time and tokens

## Linked Issues or Issue Description

**What happened?**
An agent created a task and then tried to update it with `call_api`
`PATCH /api/issues/{id}`. Every call arrived with `operationId`,
`pathParams`, `query` and `responseText`, but no `body`. The tool layer
refused each call with "requires body" and showed the shape to resend.
The model's reasoning said that the body was missing, but each new call
arrived without `body` again. The agent sent the same call 12 times and
was still running when the test ended.

**Expected behavior**
After the same call fails twice in the same way, the agent should get
one clear correction that says what arrived and what is missing.
Paperclip should then refuse to run that exact call again, so the agent
changes the call or reports the blocker.

**Paperclip version or commit**
`master` at ff971149c2.

**Steps to reproduce**
1. Run an agent on a model host that drops optional tool arguments.
2. Ask the agent to create a task and then update it.
3. See the same body-less `PATCH` call refused again and again until the
run times out.

## What Changed

- Add `RepeatedToolFailureGuard`
(`server/src/services/native-runtime/repeated-tool-failure-guard.ts`).
It keys each failure by tool name and a hash of the canonical arguments.
- Count only failures that must repeat for identical arguments:
`400`/`422` refusals and schema errors. Other errors, such as `503`,
pass through unchanged and the call still runs.
- On the second identical failure, keep the original error text and add
the names of the arguments that arrived, a note that a missing argument
did not arrive, and that Paperclip will not run this exact call again.
- On the third and later identical calls, do not run the tool. Return
the original error with an instruction to change the arguments or stop
and report the blocker.
- Any change to the arguments runs normally. A success clears the record
for that call.
- Run identical calls that arrive in parallel one after another, so each
sees the failures before it. Different calls never wait on each other.
- Wire the guard into the per-run `SessionAuthorityEpoch.execute`,
inside the tool trace, so blocked attempts are still recorded. State is
in memory, per run, and capped at 64 entries.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/repeated-tool-failure-guard.test.ts
src/services/native-runtime/native-tool-trace.test.ts`: 11 passed (6 in
the new file).
- Red/green: with the guard removed from
`SessionAuthorityEpoch.execute`, the recorded-run test fails (the second
failure has no correction and every call runs). With the guard in place,
it passes.
- The test replays the recorded body-less `PATCH` through the real
`executeRunnerApi` validation. It checks that the tool runs only twice
for five identical calls and that no HTTP request is sent.
- Parallel identical calls run at most twice, and a different call is
not held back while an identical one runs.
- Negative controls: changed arguments run again, a `503` keeps running,
and different error messages are not merged.
- `cd server && npx tsc --noEmit`: clean.

## Risks

- Low risk. The guard acts only on identical arguments with an identical
`400`/`422` or schema error, which cannot succeed on resend.
- A refusal that depends on state that changes during the run (for
example, a validation that later passes) would be blocked after two
tries with the same arguments. The agent can still send changed
arguments.
- Latency: the guard adds one SHA-256 of the canonical arguments per
tool call (about 2 µs for a 4 KB argument). It adds no wait and nothing
before turn start. Only identical calls that arrive in parallel wait for
each other.
- No persisted state changes. After a restart, the record starts empty.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:00:26 -06:00
DottaandClaude Opus 5.5 43aabb7080 fix(runner): keep configured secret values out of OpenCode output, storage and model requests (#21330)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can configure a secret that Paperclip delivers to an
agent's commands as an environment variable
> - The value must stay out of what the agent runtime stores and what it
sends to the model
> - With the OpenCode runtime, a command that printed the variable put
the value in OpenCode's tool output. OpenCode stored that output in its
session database and sent it to the model. Paperclip's own run log
already redacted it
> - Paperclip did not tell the runner which configured values are
secrets, so the runner could not protect them without also hiding
ordinary settings
> - This pull request sends secret names (never values) to the runner,
filters exact secret values before OpenCode captures any tool output,
redacts them from model requests, and tells the agent how to check a
secret without printing it
> - The benefit is that a configured secret stays out of model context
and out of runtime storage, even when an agent prints it

## Linked Issues or Issue Description

**What happened?**
The `chat-features.runner-opencode.local.secrets.personal-secret` eval
failed because the persisted-state scan found a configured secret value
in OpenCode's session database. The person entered the value through the
normal secret entry form. Nobody pasted it in chat. The agent ran a
shell command that printed the environment variable it was given.
Paperclip's run log redacted the value. OpenCode stored the raw output
in its database and included it in the next model request.

**Expected behavior**
A configured secret must never reach model context or runtime
persistence. Ordinary configured values must not be redacted.

**Steps to reproduce**
Configure a secret environment variable for an OpenCode agent. Have the
agent run a command that prints the variable. Then search the OpenCode
session data and the model requests for the value.

## What Changed

- **Secret names cross to the runner.**
`configuredEnvironmentProjection` takes the controller's secret key
names and adds `PAPERCLIP_SECRET_ENV_KEYS`. This marker holds only the
names of configured entries that came from secrets. It travels with the
configured projection across the TypeScript and Rust runner boundaries
(`configured-environment.ts`, `configured_environment.rs`). The server
passes `secretKeys` in `runtime-execution.ts`.
- **Command output is filtered before OpenCode reads it.** For a launch
that has configured secrets, the runner sets OpenCode's `shell` to a
small POSIX wrapper (`drivers/opencode/output-custody.ts`). The wrapper
runs the shell OpenCode would use anyway. It sends stdout and stderr
through an `awk` filter that replaces exact secret values. OpenCode
never receives the raw bytes, so its streamed previews, its database and
its model requests stay clean. The wrapper keeps the exit status. It
does not wait for background processes. The values come from the
environment that OpenCode already holds and are never written to disk.
- **Other tool results are rewritten before they are saved.** A local
OpenCode plugin rewrites completed non-shell results (file reads,
searches, MCP results) in `tool.execute.after`. OpenCode runs that hook
before it stores or returns the result. The wrapper refuses to run any
command until the plugin has started for the current launch (per-launch
nonce). So no command can write a secret to a file that a later read
returns without filtering.
- **Model requests are redacted as a second layer.** The OpenCode
provider proxy removes exact secret values from the string values of
each parsed model request. Field names and other literals are never
changed. A quick byte scan for each value, raw and JSON-escaped, sends
the original bytes unchanged when no value is present.
- **The agent gets guidance.** Native task prompts list the configured
secret names. They say to check a secret with `[ -n "$VAR" ]` and never
to print, echo, log or write a secret value.
- **No added startup delay.** The wrapper is content-addressed under the
runtime directory and reused across launches, so it is not rewritten on
every launch. OpenCode's plugin-dependency install is marked as already
done, so loading the plugin does not add wait time to the first tool
call.
- **Only exact values are redacted.** Command output is filtered line by
line, so each line of a multi-line value that has at least 4 characters
is redacted too. If a configured secret has no line of 4 or more
characters, the launch is refused with
`opencode_output_custody_unavailable` instead of running unprotected.

## Verification

-
`packages/paperclip-runner/src/drivers/opencode/output-custody.test.ts`
(10 tests): the wrapper filters stdout, stderr and a partial last line;
keeps exit codes (7, and 143 for SIGTERM); filters each line of a
multi-line value; refuses to run before plugin readiness or with an old
nonce; returns at once when a background process keeps running; leaves
ordinary values unchanged. The plugin rewrites nested results.
Model-request redaction covers raw and JSON-escaped values and never
changes field names or literals (a secret equal to `true` leaves
`"stream": true` intact). Secrets the filter cannot match are refused.
No value is written to disk.
- `opencode-server-driver.test.ts`: one saved session launches three
times. First with no secret marker (older controller), then with the
marker, then with no marker again (rollback). The first launch is
unchanged, the second applies custody, and the third launch still
starts. This test fails without the driver change.
- `configured-environment.test.ts`: the marker names only secret
entries, crosses the runner spawn and OpenCode boundaries, reads as "no
secrets" when absent, and is refused when malformed. Rust:
`secret_name_marker_crosses_only_with_its_projection`.
- `server/src/__tests__/configured-secret-guidance.test.ts`: guidance
content, and no guidance when there are no secrets.
- Real OpenCode 1.18.34 qualification (`PAPERCLIP_OPENCODE_QUALIFY=1
node --test scripts/qualify-opencode-runtime.test.mjs`). A fake model
asks for a command that prints the secret and writes it to a file, then
a read of that file. Unprotected baseline: the value is in the session
database and in model requests (red). With custody: it is in neither,
and the exit status is kept (green).
- **Latency** (local, macOS, real OpenCode, 8 runs each): boot 629 ms vs
697 ms baseline; first model request after the prompt 678 ms vs 698 ms
(no regression); first command 139 ms vs 191 ms; each later command +9
ms (36 vs 27 ms). Proxy cost per request when no secret is present (the
normal case): 33 µs at 50 KiB, 164 µs at 300 KiB, 0.55 ms at 1 MiB, 2.3
ms at 4 MiB. When a secret is present: 91 µs, 0.48 ms, 1.6 ms and 6.8
ms. The scan runs only for launches that have configured secrets, and it
adds nothing to OpenCode's own startup.
- A narrow paid run of
`chat-features.runner-opencode.local.secrets.personal-secret` on this
branch is linked in a comment.

## Risks

- **Previous-release story.** The marker is new and optional. A runner
that receives no marker (older controller) launches exactly as before.
An older runner drops the marker at its environment boundary. The change
saves no new run or session state. OpenCode configuration is regenerated
on every launch, so sessions saved before this change resume under
custody (covered by the cutover test). Warm-process reuse keys include
the marker, so a warm process from before the deploy is not reused for a
run that has secrets.
- **Commands go through the wrapper.** Output arrives line by line.
Output written after the shell exits is dropped, which matches how
OpenCode stops reading at exit. If `awk`, `mkfifo` or `rm` is missing,
the launch is refused with `opencode_output_custody_unavailable` and
does not run unprotected.
- **Scope.** Only exact values are redacted. An agent that encodes a
value (for example base64) is out of scope. The model-request layer
applies when Paperclip routes the model through its proxy (OpenRouter
and Paperclip connections).

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 03:23:56 -06:00
DottaandClaude Opus 5.5 0986c3eb3a fix(native): let a same-run reattachment take over its own retained remote Runner cleanup (#21328)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) use a detached
`paperclip-runnerd`. After a recoverable failure, the server resumes the
same run and reattaches to that Runner
> - #20923 keeps an unreachable remote Runner alive when the failed
attempt closes, so that the same-run reattachment can adopt it
> - The same close also leaves the session in the in-process cleanup
quarantine. The quarantine refuses every new session of the task, and
that includes the reattachment that the Runner was kept for
> - Close retries cannot settle it. After the first close fails, the
Runner transport returns the same rejected close for every retry
> - On staging, this was the largest failure class: 14 Codex runs in one
hour failed with `native_session_cleanup_quarantined`, while only 5 runs
succeeded. Each task then needed Retry
> - This pull request lets the same run's reattachment to that Runner,
in the same remote sandbox, take over its own retained cleanup
> - The benefit is that the task continues on its own, with no spent
attempt, no failed run and no Retry

## Linked Issues or Issue Description

Refs #20923, #21031

**What happened?**
A remote native run (Codex on Daytona) failed a turn with a recoverable
error, for example `runnerd did not report the provider turn identity`
or `native_finalization_missing: session returned no semantic result`.
The close then logged `runner did not prove durable suspension before
checkpoint` and `remote runner left alive unsettled for same-run
reattachment`. Each later same-run reattachment was refused at
admission:
- When the close error was retryable, admission waited out the grace and
failed with `native_session_cleanup_quarantined: prior session cleanup
remains incomplete`. That spent a same-run attempt. After the attempt
budget, the run failed. All close retries replayed the same failed close
within milliseconds (`runner transport close requested: native session
quarantined cleanup recovery`), so they could never settle the
quarantine.
- When the close error was `native_session_close_unrecoverable`
(operator-required), the next attempt failed at once with
`native_session_cleanup_quarantined: the previous session is still
shutting down; Paperclip finishes the cleanup automatically and the
saved work is kept.` That code is a permanent failure.

"Paperclip finishes the cleanup automatically" did not happen in the
process. Only a server restart (a deploy) removed the in-memory
quarantine, and by then the run had failed and the task waited for
Retry. Staging restarts every 20 to 45 minutes, so attempts often end
mid-cleanup and meet this gate.

**Expected behavior**
The same run's reattachment adopts the Runner that its earlier attempt
kept for it. It does not wait for a close that cannot settle, and it
does not spend an attempt. The task continues without Retry. Other runs
of the task and other sandboxes stay gated.

**Steps to reproduce**
1. Start a remote native run, and let its turn fail with a recoverable
transport error while the Runner cannot prove suspension.
2. The close keeps the Runner alive and quarantines the session.
3. The server schedules the same-run reattachment 30 s later.
4. The reattachment fails with `native_session_cleanup_quarantined`, and
the task needs Retry.

**Paperclip version or commit**
Deployed Core `c730872974` on staging. Reproduced on `master` at
`49f0030f06`.

**Deployment mode**
Hosted cloud (staging), Daytona remote sandboxes.

**Agent adapter(s) involved**
`paperclip_runner` native runtime with the Codex app server backend.

## What Changed

- `packages/paperclip-runner/src/native-session-runtime.ts`:
`executeNativeSession` takes a new option `retainedRunnerReattachment`.
With a remote cleanup scope, admission retires quarantine entries that
belong to the same run in the same remote scope. Removal stops later
close retries. A close attempt that is still in flight stays an
admission owner until it settles, so the old and new controllers do not
overlap. Other runs, other sandboxes and local backends keep the gate.
- `server/src/services/native-runtime/native-session-executor.ts`: sets
`retainedRunnerReattachment` only for a `reattach_remote_runner`
recovery of the same run.
- `doc/execution-semantics.md`: records the takeover.
- Tests:
- `packages/paperclip-runner/src/native-session-runtime.test.ts`: four
new cases. An operator-required close with a retained Runner. A
retryable close whose retries replay one failure. A close that a deploy
interrupts while it is in flight (the reattachment waits for it, then
takes over). Controls: another run, a run without the takeover option
and a local backend stay refused.
- `server/src/services/native-runtime/native-session-executor.test.ts`:
asserts that the option is set for a same-run remote reattachment and is
not set for a fresh sandbox run.

## Verification

- Red on `master` (runtime source restored): 3 of the 4 new runtime
cases fail with exactly the two staging errors (`the previous session is
still shutting down…` and `prior session cleanup remains incomplete`).
The control case passes.
- Red on `master` (executor source restored): the 3 executor assertions
fail.
- Green with this change:
- `cd packages/paperclip-runner && npx vitest run
src/native-session-runtime.test.ts` (294 passed)
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "exact
Stop cleanup|fresh sandbox|retirement"` (46 passed)
- `cd server && npx vitest run
src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts
src/services/quarantined-task-retry.test.ts
src/__tests__/completed-turn-cleanup-hold-settle.test.ts` (61 passed)
- `npx tsc --noEmit` in `packages/paperclip-runner` and `server` passes.
- Staging evidence: runs `7b7ded99` and `27c487ef` show the sequence
above in their events and logs.

## Risks

- The takeover applies only when the server recovers the same run with
`reattach_remote_runner` in the same remote cleanup scope. This is the
case that #20923 keeps the Runner alive for. After a deploy the
quarantine is already gone and the same reattachment runs today, so the
takeover gives the in-process retry the same behavior as a restart.
- A close attempt that is in flight is never overlapped. The
reattachment waits for it within the existing admission grace.
- The reattachment still verifies Runner ownership and its controller
generation before it adopts the Runner. If the Runner is gone, the
existing relaunch or reconciliation path handles it.
- No persisted shape changes. Saved state from the previous release is
not affected because the quarantine lives only in process memory.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 03:15:40 -06:00
DottaandClaude Opus 5.5 22c9300dac fix(recovery): a run that fails before the agent is asked anything continues by itself on a fresh computer (#21297)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a remote sandbox opens a provider session and then
waits at a launch gate before the first prompt goes to the provider
> - On remote OpenCode sandboxes the gate saves the provider launch
files; when that save fails, no prompt was ever sent
> - Today that failure ends as `native_session_interrupted` with
ambiguous recovery evidence, so the task goes blocked and the user must
press Retry
> - Nothing was sent to the provider, so nothing can be duplicated: the
system can safely continue by itself
> - This pull request types that failure, proves it from the Runner turn
ledger at the failure point, and continues the task on a fresh sandbox
in the existing interrupted-preparation lane
> - The benefit is that a setup failure before the first prompt no
longer needs a human, and the user sees plain words instead of internal
codes

## Linked Issues or Issue Description

Refs #21287
Refs #21304

**What happened?**
On a remote OpenCode sandbox, the provider launch-files gate failed
before the first prompt (for example
`native_provider_artifacts_transfer_timeout`). The run ended blocked
with a Retry banner. The run error showed internal codes and raw byte
counts to the user.

**Expected behavior**
A failure that happens before any prompt reaches the provider continues
automatically on a fresh sandbox. The task shows as working while the
continuation is pending. The visible text is plain and has no internal
codes or raw numbers.

**Steps to reproduce**
1. Use an OpenCode agent on a Daytona sandbox environment with
controller persistence.
2. Arm the crash point `native.provider_artifacts.before_capture` in
fail mode (or let the launch-file upload time out).
3. Start a task. The first run fails at the gate before any prompt.
4. On master the task goes blocked with Retry. With this change a
successor starts on a new sandbox and the task completes.

**Agent adapter(s) involved**
OpenCode (native runtime, remote sandbox).

## What Changed

- New typed cause `native_failed_before_provider_turn`
(`server/src/services/native-runtime/failed-before-provider-turn.ts`).
The executor raises it only when the launch gate fails, the run was not
cancelled, and the Runner turn ledger proves that no turn command was
issued. The proof is taken at the failure point, before the session
closes. In all other cases the original error is rethrown and today's
path stays.
- The executor quarantines the failed run's session root, so the
successor starts in a new root.
- `heartbeat/predecessor-retirement-fallback.ts` accepts the new cause:
it records the proof on `runnerProfileJson.failedBeforeProviderTurn`,
marks the run's reusable sandbox leases for replacement, and schedules
the successor in the existing `interrupted_preparation` lane (same cap
of 12 and the same backoff).
- `remote-workspace-unchanged-proof.ts` accepts the runner process,
runner command and pre-turn events only when that proof record is
present. It still refuses any turn evidence, an unsettled save by
another run on the same workspace, and setup writes. When the proof
refuses, the run's epoch is abandoned unapplied: the files are kept and
recorded, and the next run restores the last committed save into a fresh
root.
- When the proof refuses and the epoch is abandoned, the replacement
mark stays. The next acquisition replaces the sandbox. #21304's
retention hold makes `retireReplacedReusableSandboxLease` park the old
lease as `retained` instead of `pending_cleanup`, so the task continues
on a fresh sandbox and the sandbox that holds the abandoned files is
kept. When the proof holds (only the Runner's own session folder was
written), the old sandbox is replaced and cleaned up as usual.
- `task-interruption-presentation.ts` shows a pending
`interrupted_preparation` retry as `working` with no Retry until it is
due (plus the dispatch grace).
- The visible run error is now plain: "Setting up the computer took too
long, so the task is continuing on a fresh one." (for a setup timeout)
or "The computer could not be set up before the agent started, so the
task is continuing on a fresh one." The technical cause code stays in
`runnerProfileJson.failedBeforeProviderTurn.technicalCause`, the server
log and the run log.
- New crash point `native.provider_artifacts.before_capture` (fail mode
throws the typed transfer timeout at the gate).
`tests/runner-e2e/source-sdk-public-inputs.json` is refreshed for the
changed registry file.
- `tests/restart-chaos/main.ts`: `RESTART_CHAOS_EXTERNAL_CRASH_CONTROL`
lets an outside driver arm fail-mode crash points without a restart
fault.

## Verification

Local (targeted):
- `server/src/__tests__/heartbeat-native-predispatch-retry.test.ts`: 8/8
pass. New cases: fresh remote sandbox and fresh plugin-task sandbox
continuation (now also asserts the plain visible text with no
identifier-looking tokens, and the technical cause in the proof record),
the ambiguous hold when the ledger cannot prove no turn, and backoff
plus the lane cap.
- `server/src/services/remote-workspace-unchanged-release.test.ts`:
27/27 pass. New cases: hand-on when the ledger proves no turn; refusal
on a recorded turn, an unsettled predecessor save, and setup writes. Two
more cases follow the marked lease through the next acquisition's
replacement decision (`readReplaceBeforeReuseMarker` +
`reusableSandboxReplacementPlan` + `retireReplacedReusableSandboxLease`)
and then through `sweepPendingCleanupLeases`. "replaces the marked
sandbox when the failed run provably left its files unchanged": the old
lease goes to `pending_cleanup` and the sweep destroys it. "continues on
a fresh sandbox and parks the one that holds abandoned files, which
cleanup leaves alone": the old lease is `retained` with the
abandoned-files reason and is held by the receipt, the sweep does not
destroy it, and the file is still there.
- Red without the product change (product files reset to the base, tests
kept): 8 failed, 25 passed of 33. All 8 failures are the new cases. The
visible-text assertion alone was red on the previous commit: the error
read "The computer could not be prepared before the agent started
(native_provider_artifacts_transfer_timeout: the upload of 536870912
bytes did not finish within 120 s). The task continues on a fresh
computer."
- `task-interruption-presentation.test.ts`,
`native-provider-artifact-gate.test.ts`,
`crash-point-call-sites.test.ts`: 79/79 pass.
`packages/shared/src/crash-points.test.ts`: 16/16.
`tests/restart-chaos/crash-points.test.ts`: 12/12.
- `tsc --noEmit` clean for `server`, `packages/shared` and
`tests/restart-chaos`.

Daytona (local servers, Daytona sandboxes, image `sha256:787c3e9c…`,
these commits rebased onto a298ca7fbc, gate failures forced with the
crash point):
- F1, wave `w-20261113T042929-s2301`: 4 OpenCode tasks, one forced
failure on each first run. 4/4 failed runs recorded cause
`native_failed_before_provider_turn`. Each successor (retry reason
`interrupted_preparation`) was created 0.0–0.1 s after the failure and
started 1–3 s later, each on a NEW sandbox. Successor first agent event
at 32.8–56.6 s. 4/4 tasks done hands-off. 0 Retry banners. 0 blocked. No
operator wording. No reconciliation codes.
- F3, wave `w-20261113T043934-s2303`: 1 task, 3 consecutive forced
failures. Retries started 7 s, 35 s and 57 s after creation (backoff).
The 4th run succeeded. 0 Retry banners while pending. Not proven: that
retries 2 and 3 in F3 got fresh sandboxes (proven in F1 only).
- F2 (what bootstrap writes before the first turn): the whole sandbox
workspace was sampled every 5 s on all 4 failed runs, right after the
workspace was ready and at the gate failure. Bootstrap wrote nothing to
project files. All 14 new paths are under
`<workspace>/.paperclip-runtime/paperclip-runner/sessions/<session-sha>/`
(runner state, identity and lineage, `runtime-context.json`, the
instructions copy `AGENTS.md`, OpenCode config, data and log). The only
deletion was the staged agent-files `AGENTS.md` (or a
`.paperclip-upload-<uuid>` temp file). The successor never reuses them:
new session root, new workspace generation, restored from the snapshot.
3 of the files are byte-identical when recreated; the rest are
run-specific and recreated by the successor's own bootstrap. The project
workspace is not a git checkout there. Raw manifests are kept with the
PR author.
- The Daytona runs used the error text before the plain-wording commit.
The wording change is covered by the unit test above.

## Risks

- Scope: this applies only where the gate exists: remote OpenCode
sandboxes with controller persistence. Local runs cannot hit it.
- The proof is the Runner's in-memory turn ledger, read at the failure
point. In unit tests it is mocked. If a turn may have been sent, today's
ambiguous hold stays (tested).
- At the lane cap (12) the run stops with a working Retry instead of
waiting longer. A later change can move this onto the shared hourly
limit.
- A direct "is the workspace dirty" check by the Runner is not added,
because it needs a new Runner command. The unchanged proof instead
relies on the recorded evidence and refuses on any turn evidence,
unsettled saves or setup writes.
- Whether a replacement mark gives a fresh sandbox depends on
`reusableSandboxReplacementPlan`. When it refuses, the sandbox is
resumed, but the session root and workspace root are still fresh.
- Refused case (abandoned files): the task continues on a fresh sandbox,
in a fresh session root and a fresh workspace root. The old files stay
on the parked sandbox, recorded in the abandoned-epoch receipt and
unapplied. That sandbox is kept under #21304's hold until the files are
recovered, discarded, or the retention period ends; this PR does not
change that hold.
- The changes in `heartbeat/` and `remote-workspace-unchanged-proof.ts`
were cleared with their owner under these conditions: evidence-scoped,
latest committed epoch only, files kept when unproven, negatives tested.
- `server/src/services/fresh-continuation-fallback.ts` is deliberately
untouched.
- Upgrade safety: runs saved by an older build have no proof record, so
the unchanged proof keeps its previous behavior for them.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with
tool use (shell, file edits, test runs). Earlier commits in this branch
came from Claude sessions in the same fix loop.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:48:51 -06:00
Nicky LeachandClaude Opus 5.5 41998d551c feat(plugins): send the claimant's credential on plugin task resume (#21335)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugin environment drivers run agent tasks on remote computers, and
a warm follow-up run resumes a parked Runner through the `resume` task
operation
> - `resume` carries only the durable identity and a bootstrap ticket,
which is enough while the parked Runner process is still alive
> - A provider sometimes has to start the Runner again to resume the
session: after it moved the parked session to a new sandbox, or after
the computer rebooted
> - A Claude Runner cannot start without its credential, and today only
the portable handoff path (`handoffCredential`) delivers one
> - This pull request lets a provider opt in to receiving the claimant
run's harness credential on `resume`
> - The benefit is that a provider can continue a parked session on a
new computer without Core orchestrating a handoff

## Linked Issues or Issue Description

Refs #21289
Related: #21305

**What existing behavior does this improve?**
Warm resume for plugin task providers that restart the Runner to resume
a session.

**Subsystem affected**
Plugin SDK environment task schema, the server's plugin Runner task
owner, and the plugin environment task docs.

**Current behavior**
`submit` can carry the harness credential (`credential`), but `resume`
cannot. A provider that must start the Runner again on resume has no
credential for a Claude task.

**Proposed behavior**
A provider that returns `resumeCredentialVersion: 1` from `describe`
receives `credential` on `resume` whenever Core would send it on
`submit`, resolved for the claimant run. Other providers see no change.

**Reason and benefit**
Providers can restart a Claude Runner on resume without the Core-run
handoff protocol.

**Breaking changes**
None. The field is opt-in: the SDK worker parses operations strictly, so
Core sends it only to providers that advertise the capability.

## What Changed

- `resume` gains an optional `credential`, with the same schema and
validation as `submit`'s.
- `describe` gains an optional `resumeCredentialVersion: 1` capability.
- The plugin Runner task owner adds the credential on `resume` when the
provider advertised the capability and Core resolved a credential, using
the same resolution as `submit` (`preparePluginRunnerCredentials`).
After a warm rebind, the claimant run's credential is sent.
- `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md` documents the field and its
handling rules.
- The runner-e2e SDK public-input pin is updated.

`handoffCredential` on the session handoff path is unchanged.

## Verification

- SDK vitest `tests/environment-task-release-resume.test.ts` and
`tests/environment-tasks.test.ts` (68 passed): `resume` accepts and
rejects the same 11 credential cases as `submit`; the capability is
optional and rejects other versions; a resume credential reaches the
plugin through the serialized worker RPC.
- Server vitest `plugin-runner-task.test.ts` and
`environment-task-runtime.test.ts` (151 passed): credential sent with
the capability and a resolved credential; omitted without a credential
or without the capability; the rebound claimant's credential is sent;
the credential appears in no metadata write and not in the sanitized
error, even when the worker error echoes it.
- `native-session-executor.test.ts -t "uses a task-only"` (5 passed):
the executor forwards the capability to the owner.
- runner-e2e `source-sdk-evidence.test.ts` and
`source-sdk-public-type.test.ts` (46 passed).
- `tsc --noEmit` for `packages/plugins/sdk` and `server`.
- Not run locally: the full suite and `pnpm verify`; required CI covers
them.

## Risks

- No server-side provider check on resume, unlike `submit`: the lease
has no reliable harness at resume time. Protection relies on the owner,
whose rebind refuses a changed harness, account or credential identity.
- The credential is the claimant run's current one, which can be newer
than the one that started the parked Runner. That is intended.
- A provider must advertise the capability only after its SDK pin
includes this schema; roll Core out first.
- Rollback: revert this pull request. No migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs), implemented by a sub-agent and reviewed
before opening.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:40:46 +00:00
DottaandClaude Opus 5.5 7e7452e88f fix(native): keep a reattached Runner alive when its provider recovery deadline ends (#21321)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs on remote sandbox Runners. A same-run reattachment
adopts a live Runner after a failed attempt or a server deploy.
> - On staging, a remote Codex Runner is often gone when the next
attempt reattaches (`process_verification` / `process_not_running`). The
task then continues in a fresh session and loses its context.
> - The deploy does not stop the Runner. The logs show `controller
authority detached for restart; durable runner left alive`, and the new
controller adopts the same Runner after the deploy.
> - Our own cleanup stops it. The adopted Runner answers its recovery
reads slowly while its in-flight turn runs. The 900 s provider recovery
deadline then aborts the recovery, and the abort closes the transport
with `turn.stop` and `runner.suspend` and signals the process.
> - This pull request makes the recovery deadline detach from the
Runner, as a recovery read timeout already does. The server marks the
failure as an unanswered adopted Runner, so the next attempt replaces
the Runner from its saved journal and resumes the same provider thread.
> - The benefit is that a deadline no longer kills a live Runner and no
longer forces a fresh session. The task keeps its session context.

## Linked Issues or Issue Description

Refs #21314, #21272, #21161, #21023

**What happened?**
Codex runs on Daytona remote Runners failed often on staging. In three
sampled chains (attempt 2 of each), the same-run reattachment adopted
the live Runner, authenticated in under 1 s, and then waited the full
deadline: `native session provider recovery timed out after 900000ms`
(909 s, 930 s and 915 s). The deadline cleanup sent `turn.stop` and
`runner.suspend` (`provider turn stop timed out before runner
suspension`). The next attempt found the Runner gone 25–180 s later
(`process_not_running`, `runnerLifecycle: ready`, empty stderr) and
continued in a fresh session. During the 900 s wait, the in-flight turn
was still running Paperclip tools (`tool.execute` spans at 04:43 in a
recovery that started at 04:29).

**Expected behavior**
A remote Runner survives a recovery deadline and a control-plane deploy.
The next attempt continues the same provider session. A Stop still stops
the Runner.

**Steps to reproduce**
1. Start a Codex run on a remote Runner and let a turn start.
2. Fail the attempt so that its close keeps the Runner (for example, a
turn start that outlives its bound).
3. Let the same-run reattachment adopt the Runner, and make the Runner
stop answering recovery commands.
4. Let the provider recovery deadline expire.
5. See that the Runner process is gone and the next attempt reports
`process_not_running`.

**Paperclip version or commit**
master at b555087f0f (after #21314).

**Deployment mode**
Hosted staging (cloud), remote Daytona sandbox Runners, deploys every
20–45 minutes.

**Agent adapter(s) involved**
Native Codex (`paperclip_runner`, Codex app-server through
`paperclip-runnerd`).

## What Changed

- Add `NativeProviderRecoveryDeadlineError` (same message as before) and
use it for the provider recovery deadline in the native session runtime.
- In the Codex driver, a recovery aborted by this deadline detaches the
controller (no `turn.stop`, no suspend, no signal). A Stop and every
other abort still close and stop the Runner.
- On the server, `nativeRecoveryCommandUnanswered` recognizes the
deadline. The failure records `adoptedRunnerUnanswered`, so the next
claim carries `replaceUnansweredRunner`. That attempt fences the old
Runner lineage, launches a fresh Runner from the saved journal and
resumes the same provider thread (the #21272 path). It does not adopt
the slow Runner again for another 900 s.
- Update `doc/execution-semantics.md`.

## Verification

- Real `paperclip-runnerd` + fake Codex
(`runnerd-codex-transport-remote-retention.test.ts`): a retained Runner
is adopted, authenticates, then stops answering. A deadline abort leaves
it alive with no pending `turn.stop`/`runner.suspend`/`runner.shutdown`.
A Stop abort still stops it.
- Red/green: without the fix, the real-runnerd deadline test fails
(`expected 'gone' to be 'alive'`), the driver unit test fails (detach
called 0 times), and the server classification test fails. With the fix,
all pass.
- Measurements on real runnerd: the deadline cleanup took 10.0 s and
killed the Runner before. Now it returns in 0 ms and the Runner stays
alive. The replacement over the kept Runner's journal resumes the same
provider thread in about 2.2 s with one `thread/start` and one
`turn/start` (no replay).
- `npx vitest run
src/live/runnerd-codex-transport-remote-retention.test.ts
src/drivers/codex/codex-app-server-driver.recovery.test.ts
src/native-session-runtime.test.ts` (368 passed).
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "never
answered its recovery read"` (passed).
- `tsc --noEmit` for `packages/paperclip-runner` and `server` (clean).

## Risks

- A deadline now leaves a live Runner behind. The next attempt replaces
it through the launch fence, which fails closed when it cannot prove the
old lineage gone. This is the same risk as the existing recovery read
timeout detach.
- When the provider attempt budget is already spent, the kept Runner is
not replaced by this run. This also matches the existing timeout detach.
- Not in this PR: the upstream trigger. The first attempt abandons a
slow turn start (`runnerd did not report the provider turn identity` or
`turn.start timed out`) while Codex later runs that turn. Two sampled
chains then stopped the Runner on
`protocol_failure:tool_binding_mismatch` / `thread_binding_mismatch`
after a deploy reattachment. These need a separate fix.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:30:22 -06:00
Nicky LeachandClaude Fable 5.2 314b893678 fix(native-runtime): deliver assigned gateway tools to every native harness through the PRP relay (#21274)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An agent can have app connections. Paperclip gives the agent the
tools of those apps through an MCP gateway, with its approval and audit
rules.
> - Native harnesses reach the gateway in two ways today. In the relay,
the server keeps the gateway token and offers the tools over the
authenticated Runner channel. In the direct way, the server gives the
gateway URL and a bearer token to the harness.
> - Which way a harness uses comes from a list of special cases that
grew one harness at a time. Local Codex and all ACPX agents use the
direct way. The others use the relay.
> - The direct way puts a run-scoped token in a provider process. A
process that the agent starts can read it. A provider process that
serves more than one run can hold a revoked token.
> - This pull request replaces the list with one rule. Each native
harness gets its assigned gateway tools through the relay.
> - The benefit is one behavior for all native harnesses, and no gateway
token in a provider process.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

Refs #13942 (the relay), #17149 (ACPX accepts relayed tools) and #19912
(the design for turn-scoped ACPX MCP authority).

**What existing behavior does this improve?**

How a native harness reaches the tools of its assigned MCP gateway.

**Subsystem affected**

Native runtime in the server (`native-session-executor.ts`,
`native-session-resume.ts`). One Runner test file. One Rust test.

**Current behavior**

- `relayAssignedMcp` is true for plugin tasks, Cursor Cloud, OpenCode,
Codex on a remote target, and Dot.
- For local Codex and for each ACPX agent (Claude, Codex, Cursor, Pi,
Kimi, Grok, Copilot, Antigravity), Core puts
`PAPERCLIP_NATIVE_MCP_NAME`, `PAPERCLIP_NATIVE_MCP_URL` and
`PAPERCLIP_NATIVE_MCP_TOKEN` in the Runner environment. The ACPX host
gives them to the provider as an HTTP MCP server. The local Codex home
gets them in `config.toml`.
- The token is run-scoped. Core revokes it when the run ends.

**Proposed behavior**

- A native run that has an assigned gateway uses the relay. The
condition names no provider.
- Core builds the gateway tools on the control plane, offers them with
the other Runner tools, dispatches each call over PRP, and authorizes
each call again through the gateway.
- Core removes the three `PAPERCLIP_NATIVE_MCP_*` variables from each
Runner environment.
- The tool contract fingerprint has the relay marker for each target,
not only for remote targets.

**Reason and benefit**

- All native harnesses have the same behavior.
- No provider process holds a gateway token.
- A sandbox does not need to reach the gateway URL of the host.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`
- The list of special cases is gone. Core validates the binding and
builds the relayed tools when a binding is present.
- Core does not list the gateway for a review run, Grok Bot or Muse. The
tool authority never offered app tools to those. A gateway error can
therefore not fail those runs.
- OpenCode keeps its `on_demand` catalog. That option depends on the
provider, not on the relay.
  - Core removes the three variables from each Runner environment.
- `server/src/services/native-runtime/native-session-resume.ts`: the
relay marker in the tool contract fingerprint is unconditional.
- `doc/execution-semantics.md`: the text about the tool contract
fingerprint now states the rule, the tool names and the cutover.
- App tool schemas. New module
`packages/paperclip-runner/src/drivers/tool-input-schema.ts` compiles
each schema by itself, in its declared dialect (draft-07, 2019-09 or
2020-12), with no shared `$id` registry. Core uses it when it builds the
relayed tools (`assigned-mcp-tools.ts`). Core does not offer a tool
whose schema has an unsupported dialect or does not compile, and writes
a log entry with the tool name and the reason. The run does not fail.
The ACPX tool bridge and the OpenCode MCP bridge use the same function.
The tools of Paperclip itself stay strict.
- Large arguments. The ACPX sidecar sends the arguments of a tool call
to runnerd with no display limit. The limit is the tool-call admission
limit of runnerd, 256 KiB. A larger call fails with a tool error that
gives the limit. The sidecar does not dispatch it.
- Tests in `native-session-executor.test.ts`,
`native-session-resume.test.ts`, `runtime-context-materializer.test.ts`
and `acpx_provider_backend.rs`.

Not changed in this pull request: the direct code path. It is still in
the Runner, but it receives no binding. #21311 deletes it.

## Behavior that each harness sees

| Harness | Before | After |
|---|---|---|
| Codex on a remote target, OpenCode, Cursor Cloud, Dot, plugin tasks |
Relay | No change |
| Local Codex | Direct: a `paperclip-assigned` server in `config.toml` |
Relay. App tools are dynamic tools named `app_<name>_<digest>` |
| ACPX agents | Direct: a second HTTP MCP server | Relay. App tools are
on the `paperclip` bridge as `mcp__paperclip__app_<name>_<digest>` |
| ACPX with a preconfigured home | No gateway | No change |
| Grok Bot, Muse | Got the variables and ignored them | Do not get the
variables |

- A relayed tool name has a maximum of 64 characters. With the
`mcp__paperclip__` prefix it has a maximum of 80.
- Permission behavior keeps the same intent. For Claude in the
`approve-reads` and `approve-paperclip` modes, app tools ask for
approval, as before. For the other ACPX agents the permission decision
does not use the server name.
- In a planning run or an ask run, the relay offers only tools with read
risk. The direct gateway offered all tools. This is stricter.
- When the catalog is too large, each harness gets the two compact tools
`paperclip_search_assigned_tools` and `paperclip_call_assigned_tool`.
- The four helper tools for resources and prompts
(`paperclip_list_resources`, `paperclip_read_resource`,
`paperclip_list_prompts`, `paperclip_get_prompt`) are not in the relay.
No harness loses them: a run token permits only `tools/list` and
`tools/call` (`run-preparation.ts`), so these four tools already fail
with `gateway_token_action_denied` on the direct path.

## State saved by the previous release

- **Local sessions of each provider start again one time.** The tool
contract fingerprint changes for local targets. The next run sees a
different fingerprint, starts a new provider session with the full task
context in the same workspace, and keeps the old saved state. This is
the existing rotation for a tool contract change. Codex needs it,
because Codex cannot add tools to a thread that it resumes. For the
local providers that already used the relay, the restart is not
necessary. It is accepted so that the rule names no provider.
- **Remote sessions continue.** Their fingerprint does not change. For
ACPX, the attach of the next run replaces the saved tool set.
- **A kept Codex home.** Paperclip writes `config.toml` again at each
start and recovery. The old gateway server and its token are removed.
- **A deploy lands during a run, and the Runner survives.** The
controller attaches to the same Runner for the same run. The old
provider keeps its direct gateway connection. The controller detach
keeps the gateway token of that run, and Paperclip revokes it when the
run ends.
- **A deploy lands during a run, and the Runner is dead.** The Runner
starts again for the same run with its saved tool set and no direct
gateway. The assigned app tools are not available for the remainder of
that one run. The next run gets them through the relay. This applies to
each harness that moves to the relay. It has no test.
- Paperclip refuses no saved state and discards none.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-session-resume.test.ts`: `1508 passed
| 7 skipped`, 1 failed. The failed test is "archives a damaged prior
epoch before completing guarded same-task replacement with real runnerd"
(`ENOENT … authority-epochs`). It fails in the same way on `master` at
53608bd1d6 and with the product change removed, on macOS.
- `cd server && npx tsc --noEmit -p tsconfig.json`: no errors.
- `cd packages/paperclip-runner && npx vitest run
src/drivers/runtime-context-materializer.test.ts`: `18 passed`. `npx tsc
--noEmit -p tsconfig.test.json`: no errors.
- `cargo fmt --manifest-path runner/Cargo.toml --all -- --check`: no
changes. `cargo test -p paperclip-runner-core --lib
acpx_provider_backend`: `50 passed`.

- `npx vitest run src/drivers/tool-input-schema.test.ts
src/drivers/runner-tool-bridge.schemas.test.ts
src/drivers/runner-tool-bridge.test.ts
src/cli/acpx-runtime-sidecar.test.ts src/drivers/opencode` in
`packages/paperclip-runner`: `21 passed (21)` files, `377 passed` tests.
`pnpm run typecheck:typescript`: no errors.
- `cargo test -p paperclip-runner-core --test acpx_event_payload --test
tool_input_schema_fixtures`: 10 and 3 passed.
- Schema tests use one shared fixture
(`tool-input-schema-fixtures.json`) for the TypeScript compiler, the
tool bridge and the Rust `begin_call`: a draft-07 schema, a second
schema with the same `$id`, 2019-09, 2020-12, draft-04, an unknown
dialect, a `$ref` with no target, and an invalid keyword. Before the fix
the bridge failed with `no schema with key or ref
"http://json-schema.org/draft-07/schema#"`.
- A tool schema whose `$id` is the URI of a meta-schema cannot damage
the shared compiler. Before this fix, `removeSchema` deleted that
meta-schema, and later schemas failed to compile for each run in the
process. A test with 7 reserved ids failed before the fix and passes
now.
- Core limits the compile work of each run start. A schema over 512 KiB
is not compiled, and one listing compiles a maximum of 768 KiB of
schemas. These are the limits of the Runner. A tool over a limit is
served only through the search and call tools. Core keeps the outcome
for each schema digest (1,024 entries) and yields to the event loop
before each compile. Four tests failed before this change.
- A sidecar test sends a tool call with 200 KiB of arguments and
receives them byte for byte, refuses a call with 300 KiB with the named
error, and keeps the replacement of a lone surrogate.

Tests, each seen to fail without the product change:

- "keeps assigned MCP credentials on the control plane for %s" now runs
for remote Codex, local Codex, OpenCode and ACPX Claude, Pi, Kimi,
Copilot and Cursor. It checks that Core builds the relay, offers the
tool, dispatches a call, and gives no `PAPERCLIP_NATIVE_MCP_*` variable
to the Runner. Without the change, the five new harness cases fail.
- "relays the assigned gateway when a deploy lands mid-run…" runs for
Claude and Pi.
- The fingerprint tests keep the old local fingerprint as a retained
value and check that the new one is different. The remote fingerprints
for Codex and OpenCode do not change.
- "drops a retained direct gateway server when the relayed launch has no
binding": a Codex home with the old `config.toml` entry loses that entry
at the next start and at a recovery of the same run.
-
`next_run_attach_offers_relayed_gateway_tools_to_a_direct_gateway_checkpoint`
runs for Claude and Pi.

Not run: a live run with a real provider and a real app connection for
each harness. Not run: `pnpm verify` and the full suite. CI supplies
that coverage.

## Risks

- **An ACPX app call has a limit of 256 KiB for its arguments.** The
direct gateway accepted a request body of 10 MB. Codex and OpenCode
permit 480 KiB. To get nearer to 10 MB, runnerd needs a larger admission
limit for tool calls or a protocol that sends arguments in parts. That
is not in this pull request.
- **A tool with a draft-04 or draft-06 schema is not offered.** Ajv 8
does not support draft-04, and the TypeScript and Rust validators must
agree. On the direct path the provider got such a tool.
- **Core compiles app schemas on its event loop.** Before this pull
request Core did not compile them. With the limits above, one valid
schema near 512 KiB can hold the loop for about 0.65 s, and one listing
for about 1.2 s in total (measured on a developer machine). An app that
changes its schemas at each listing gets no help from the cache. A
worker thread with a timeout would remove this; it is not in this pull
request.
- **Tool names change for local Codex and all ACPX agents.** A saved
agent instruction that names a gateway tool directly does not match the
new name.
- **Each local native session starts again one time after the deploy.**
The provider conversation of that session does not continue. The task
context and the workspace stay.
- **A run that is in flight at deploy, with a dead Runner, has no app
tools until it ends.**
- **Names of 80 characters are not tested live** with Kimi, Copilot,
Grok, Antigravity and Codex through ACPX. Pi already shortens a name
longer than 64 characters.
- **Managed Claude and AWS AgentCore now receive relayed app tools.**
This is not tested live.
- **A gateway token expires after 1 hour.** The relay keeps the token on
the controller for the whole run, as OpenCode and remote Codex do today.
A longer run loses its app tools on both paths. This pull request does
not change that.
- **The gateway URL must have the path `/mcp/gateways/<id>`.** With a
path prefix in the API base URL, a run with an assigned gateway fails
with `assigned native MCP gateway path is invalid`. This was true for
the relayed harnesses before. It is now true for all.
- No test covers the rule that Core does not list the gateway for Grok
Bot and Muse.

## Follow-up

The direct path is now unreachable. #21311 deletes it. That pull request
is in the same stack, above the pull requests that change the same ACPX
host files.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change, reviewed the diff and ran the verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), investigated each harness
and wrote the change and the tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 00:16:58 -08:00
Nicky LeachandClaude Opus 5.5 2483cb46c2 feat(plugins): deliver configured environment to plugin Runner tasks (#21280)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent gets environment variables from its execution environment,
its own configuration, its project, and its routine. Secrets are
resolved for each run.
> - A plugin environment can run the agent as a native Runner task on
its own host, through the environment task contract.
> - That contract has no field for the agent's variables. Runs on a
plugin environment get only the variables that the plugin sets itself.
> - This pull request adds a versioned contract field for the values,
sends them on a session handoff, lets Codex commands inherit the
variables in an external sandbox, and lets a plugin protect the
variables that it sets itself.
> - The benefit is that plain and secret variables work on plugin
environments in the same way as on local environments.

## Linked Issues or Issue Description

Refs #21233. That pull request by @lucktastic finds the same gap and
proposes the same capability and field names
(`configuredEnvironmentVersion: 1`, `configuredEnvironment`). This pull
request uses the same names, so providers implement one contract. It
differs in these places:

- Codex commands get the variables through an external-sandbox denylist,
not through key names sent at `prepare`. A retained Runner does not need
a new preparation when a key is added or removed.
- A session handoff `resume` gets the values, because it starts a new
Runner on another computer. Value changes on the same computer need no
new field: Core's process reuse key includes the values, so Core starts
a new task.
- A driver can declare `protectedEnvironmentKeys`. Paperclip rejects or
warns when one is saved.
- The run log names skipped variables only for an older provider.

Thank you to @lucktastic for the diagnosis and the contract shape.

**What happened?**
An agent has environment variables, plain and secret. Its runs on a
plugin task environment do not get them. `PAPERCLIP_CONFIGURED_ENV_KEYS`
in the run lists only the variables that the plugin sets.

**Expected behavior**
Each run gets the agent's configured variables.

**Steps to reproduce**
1. Set a plain and a secret environment variable on an agent.
2. Assign the agent to a plugin environment that provides Runner tasks.
3. Start a run. Read the names in `PAPERCLIP_CONFIGURED_ENV_KEYS` in a
command that the agent runs.
4. The agent's names are not in the list.

## What Changed

- **SDK contract** (`packages/plugins/sdk/src/environment-tasks.ts`):
- A provider advertises `configuredEnvironmentVersion: 1` on `describe`.
- `submit` gets `configuredEnvironment`. `resume` gets it only for a
session handoff.
- Names, count, and sizes have the same limits as the Runner projection.
- **Server**:
- `createRunnerdBackendWithinSessionClaim` gives the run's configured
projection to the plugin task owner. The projection has no provider
credentials and no generated runtime keys.
- `createPluginRunnerTaskOwner` sends the fields only to a provider with
the capability. An older provider gets no new field, and the run log
names the variables that were not set. The log never prints values.
- **Codex** (`codex-security-config.ts`):
- An externally sandboxed launch uses `inherit="all"`,
`ignore_default_excludes=true`, and an `exclude` list of Codex's own
credentials, instead of an `include_only` list fixed at preparation
time.
- A test reads the Runner's Codex environment keys, and fails if a new
key is not classified.
  - Host launches keep the bounded allowlist.
- **runnerd** (`codex_provider.rs`): `runtime_notification_codex_args`
still grants the notification files to read-only profiles when no
allowlist is present. Plan mode in an external sandbox needs these
grants.
- **Protected keys**:
- An environment driver can declare `protectedEnvironmentKeys` in its
manifest.
- Paperclip rejects a protected key on an agent or environment that uses
the driver. The 422 has code `protected_environment_variable`.
- On a project, a routine, or another agent, Paperclip saves the key.
The response has a `configurationWarnings` notice that names the
environments. The project and routine pages show the notice as a toast.
The agent pages already show configuration warnings.
- **Docs**: `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md` describes the
provider obligations.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refresh the reviewed
SDK source pin.

## Verification

- `packages/plugins/sdk`: `vitest run tests/environment-tasks.test.ts`
(38 pass). Accepts bounded values. Rejects bad names, NUL bytes, too
many variables, and oversized values.
- `server`: `vitest run
src/services/native-runtime/plugin-runner-task.test.ts` (95 pass).
Values go only to a provider with the capability, never in `prepare`.
`resume` sends values only for a handoff.
- `server`: `vitest run src/services/protected-environment-keys.test.ts
src/__tests__/project-routes-env.test.ts` (10 pass). Rejection, notices,
and the project route response.
- `server`: all route suites that import the project, routine,
environment, or agent routes (45 files, 1689 pass).
- `packages/shared`: `vitest run src/validators/plugin.test.ts` (64
pass).
- `packages/paperclip-runner`: `vitest run src/drivers/codex
src/configured-environment.test.ts
src/live/runner-provider-launch.test.ts` (444 pass).
- `packages/paperclip-runner/runner`: `cargo test -p
paperclip-runner-core --lib runtime_notification` (9 pass, 2 ignored as
before).
- `pnpm typecheck` in `packages/shared`, `packages/plugins/sdk`,
`packages/paperclip-runner`, `server`, and `ui`.
- Not run locally: `pnpm verify --full` and browser suites. Required CI
covers the final revision.

## Risks

- **Existing providers:** low risk. Core sends no new field to a
provider without the capability. Their runs behave as before, with one
added log line when variables were skipped.
- **Codex in an external sandbox:** commands now see every variable in
the Runner environment except Codex's own credentials. The execution
environment builds that environment from an allowlist. If the Runner
gives Codex a new credential, the drift test fails until the key is
classified.
- **Saving env:** new rejections happen only when an installed driver
declares `protectedEnvironmentKeys`. No driver declares them today.
- **Providers:** they must keep the values transient and never save or
log them. The doc lists these obligations.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:11:23 +00:00
DottaandClaude Opus 5.5 86f9555695 fix(native): give a Runner whose first unanswered read came on the last attempt one replacement attempt (#21322)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs execute in a remote Runner, and a same-run retry
reattaches that Runner after an interruption
> - #21272 made the next attempt replace a Runner that left its recovery
read unanswered, instead of adopting it again
> - On staging, three runs after #21272 went live still failed: their
first unanswered read came on the third and last attempt, so no next
attempt existed
> - This pull request gives such a run one more attempt, used only for
that replacement
> - The benefit is that the replacement from #21272 can run when a
restart used up the middle attempt

## Linked Issues or Issue Description

Refs #21272, Refs #21161

**What happened?**

After #21272 (497ccba8f1) went live on staging at 05:34Z, three Codex
runs still failed with `native_session_recovery_failed:
HarnessCapabilityUnavailableError: read is unavailable: Error:
provider_transport_failed: PRP command session.snapshot timed out
(runner connected; pending ahead 0; last runner event seq … 55–84s
ago)`. The run events of all three show the same sequence:

1. Attempt 1 fails at 06:24 with `runnerd did not report the provider
turn identity`.
2. Attempt 2 starts at about 06:25. The 06:24–06:25 server restart ends
it with `local_runtime_api_listener_unavailable`. This attempt sends no
recovery read.
3. Attempt 3 adopts the same Runner. Its first recovery read gets no
answer. Attempt 3 is the last of three, so the run ends.

`adoptedRunnerUnanswered` was recorded on attempt 3, but no next attempt
read it. The replacement from #21272 never ran.

**Expected behavior**

A Runner whose first unanswered read comes on the last attempt is still
replaced once by a fresh Runner from the saved journal. The provider
turn is not sent again. The run does not end only because a restart used
the middle attempt.

**Steps to reproduce**

1. Start a Codex run in a remote Daytona Runner. Let attempt 1 fail at
turn start, so the Runner stays alive.
2. Restart the server during attempt 2, so that attempt fails without a
recovery read.
3. Let the Runner leave its recovery read unanswered on attempt 3.
4. See the run end with `session.snapshot timed out` and no replacement.

**Paperclip version or commit**

Staging with #21272 (497ccba8f1). Fixed on `master` after `bbafa78fe2`.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`).

## What Changed

- `nativeUnansweredRunnerReplacementDisposition` (executor): when
attempt 3 fails because its adopted live Runner left a recovery read
unanswered, and that attempt was not itself a replacement, the failure
stays `retryable_failure` (`native_session_interrupted`, retry in 30 s)
instead of `native_session_retry_exhausted`.
- `claimNativeRestartRecoveries`: a failed remote retry at attempt 3
with `adoptedRunnerUnanswered` is not treated as an exhausted budget.
The claim is `reattach_remote_runner` at provider attempt 4 with
`replaceUnansweredRunner`.
- `nativeAdmittedProviderAttemptLimit` (executor admission gate): a
`reattach_remote_runner` claim with `replaceUnansweredRunner` is
admitted up to provider attempt 4. Every other claim keeps the limit of
3.
- `verifyRemoteRunnerOrSettledExit`: the replacement of a live Runner is
allowed up to attempt 4. An exited Runner keeps the fail-closed path.
- Bound: the extra attempt is a replacement (`replaceUnansweredRunner`),
so it never gets another one. Attempt 4 is exhausted by the existing
rule.
- `doc/execution-semantics.md` describes the extra attempt.

## Verification

- End to end (embedded Postgres):
`native-remote-retry-admission.test.ts` › "executes the one replacement
attempt after the last attempt's first unanswered recovery read, and no
further one". It seeds attempt 3 with the staging failure, claims it,
and executes the claim through `executePaperclipNativeSession`. The
provider is reached once, the run then ends at attempt 4, and no further
claim is made. With the old `nextAttempt > 3` gate this test fails
before the provider is reached.
- Red on `master`, green with the fix (7 tests, including "admits
provider attempt 4 at execution only for the replacement claim"):
- `native-runner-restart-recovery.integration.test.ts` › "gives a Runner
whose first unanswered read came on the last attempt one replacement
attempt: true" (embedded Postgres; the coordinator is seeded at attempt
3 with the staging message). On `master` no reattach claim was made.
- `native-session-executor.test.ts` › "bounds the replacement after …"
(4 cases: the last attempt's first unanswered read is retryable; a
replacement that also went unanswered, another failure, and attempt 4
stay terminal) and "replaces the live Runner on the extra replacement
attempt".
- Focused suites: `native-session-executor`, `native-restart-recovery`,
`native-runner-restart-recovery.integration`,
`runner-lost-mid-turn-continuation`: 1737 passed, 21 skipped.
- `pnpm typecheck` in `server` passes.
- Not run locally: the container-gated remote Runner suites and the full
suite (CI covers it). Why the remote Runner stops answering is a
separate investigation and is not changed here.

## Risks

- Low to medium. This adds at most one provider attempt per run, and
only after an unanswered recovery read on a live adopted Runner. The
attempt replaces the Runner through the launch fence, which proves the
old Runner and its lineage gone or fails closed. No provider turn is
sent again.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code
with tool use (shell, file edits, browser reads of staging run events),
extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 01:27:35 -06:00
DottaandClaude Opus 5.5 bcadbde449 feat(runner): record the OpenRouter host that served each OpenCode model call (#21298)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - OpenCode agents can use models through OpenRouter, and the Runner
proxies those model calls
> - OpenRouter routes one model across many hosts, and these hosts use
different quantizations (fp4, mxfp4, fp8)
> - Chat evaluations with identical prompts behave very differently
between runs, but the run log does not record which host served a call
> - The run usage shows `provider: "deepseek"` and `biller:
"openrouter"`, and `providerRequestId` is always `null`
> - This pull request records the serving host and the OpenRouter
generation ID for each model call in the run log
> - The benefit is that we can compare evaluation cohorts by host, and
the cost event names a real OpenRouter generation

## Linked Issues or Issue Description

**Subsystem affected**
Runner OpenCode provider proxy, runnerd usage normalization, and native
run usage accounting.

**Problem or motivation**
OpenRouter's default routing sends a request to one of many hosts. Each
host can use a different quantization. Chat evaluation results change a
lot between runs with identical prompts. Paperclip does not record which
host served a call, so we cannot compare results by host.
`usageJson.providerRequestId` is always `null` for OpenCode runs.

**Proposed solution**
Every OpenRouter completion response has a top-level `provider` field
and an `id` (the generation ID). Each streamed chunk has them too. The
Runner proxy reads them from the first chunk after it forwards the
bytes. The Runner attaches them to the usage of that call. The values go
to the run log, and the run's first generation ID goes to
`usageJson.providerRequestId`.

**Alternatives considered**
A new PRP event type such as `provider.upstream.served` would need a new
event enum value. An older control plane rejects an unknown event type.
A `harness.diagnostic` event shows in the task chat as a diagnostic. The
usage payload already flows to every consumer, and older consumers
ignore extra fields. Thus this change uses the usage payload.

**Additional context**
Run-log change only (AGENTS.md section 7). The data stays in
`heartbeat_run_events` and `heartbeat_runs.usageJson`. It does not touch
`packages/shared/src/telemetry/` or OpenTelemetry.

## What Changed

- New `createUpstreamServedTap`
(`packages/paperclip-runner/src/drivers/opencode/upstream-served.ts`).
It is a pass-through stream. It pushes each chunk downstream first, then
it reads `provider` and `id` from the first SSE `data:` event (or from a
JSON body at its end). It scans at most 64 KiB of a stream and 1 MiB of
a JSON body. It skips encoded bodies and unknown content types. Any
parse problem or recorder error stops the scan and leaves the response
unchanged.
- `startOpenCodeProviderProxy` uses the tap only for 2xx responses. The
OpenCode session keeps the most recent served call. Each
`item.completed` usage payload and `session.usage()` now carry
`upstreamProvider` and `providerRequestId`. The usage dedupe fingerprint
does not include these fields, so the number of usage events does not
change.
- The OpenCode app-server proxy adds the two fields to
`thread/tokenUsage/updated`. Rust `usage.reported` now fills its
existing `providerRequestId` field and adds an optional
`upstreamProvider`. The TypeScript runnerd transport keeps both fields
when it rehydrates the usage notification. `usage.schema.json` documents
`upstreamProvider` (the protocol manifest and schema bundle are
regenerated).
- Server native turn accounting keeps the first generation ID the run
reports. The usage checkpoint and the native adapter result carry it as
`providerRequestId`, so `usageJson.providerRequestId` and the final cost
event name it.
- `doc/run-log-events.md` documents the fields in a new "OpenRouter
serving host" section.

## Verification

- Red/green: `opencode-server-driver.test.ts` "passes OpenRouter's
stream through unchanged and records the host that served it on the
turn's usage". The test sends a streamed fixture with a comment line, a
host chunk, a malformed chunk, and `[DONE]` through the proxy. It checks
that the bytes are identical and that the turn's usage names `Novita`
and the generation ID. The test fails without the proxy wiring.
- `upstream-served.test.ts`: streamed fixture, chunk split inside a
multibyte character, non-streamed JSON, malformed chunks, a truncated
JSON body, an encoded body, the 64 KiB limit, and a recorder that
throws.
- `opencode-proxy-usage.test.ts`, `runnerd-codex-transport.test.ts`
(rehydration), Rust
`provider_events::tests::carries_the_serving_host_and_generation_on_proxy_usage`,
and server `native-usage-accounting.test.ts` (first generation ID stays
stable across turns).
- Local: OpenCode driver, OpenCode proxy, and runnerd transport suites
(31 files, 596 tests) pass. `cargo test -p paperclip-runner-core` passes
(one timing test failed once under load and passed alone). Runner and
server `tsc --noEmit` pass. Protocol manifest, type, and coverage checks
pass.
- Latency: a local benchmark proxied a 400-chunk SSE stream 200 times
with the tap and 200 times without it. The median time to first byte was
292 µs with the tap and 259 µs without it (+33 µs, which is the extra
stream stage). The total stream time changed from 491.4 ms to 492.5 ms.
The tap spends about 5.5 µs of CPU to inspect the first chunk, after it
has forwarded that chunk. OpenRouter time to first token is hundreds of
milliseconds.
- Paid evidence: one single-case run of
`chat-features.runner-opencode.local.skills.fork-read-only` on this
branch. The result is in a comment on this pull request.

## Risks

- Low risk. All new fields are optional and additive. An older control
plane ignores the extra usage fields. `usage.schema.json` is not bound
by the event schema, and no validator enforces it on `usage.reported`.
- Saved state from the previous release: this change adds no persisted
run, session, checkpoint, or launch state. A saved session resumes
unchanged. A run that is in flight during a deploy reports usage without
the new fields until its Runner is new. `usageJson.providerRequestId`
then stays `null`, as before.
- `providerRequestId` on the run and its cost event is the run's first
generation ID, not each call's ID. The value is stable, so a late usage
receipt still matches the posted charge. Per-call IDs and hosts are in
the run log.
- If OpenCode starts a new model call before it reports the usage of the
previous call, a usage event can name the newer call. OpenCode reports a
message's usage when its step ends, so this is not expected.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code as an agent with tool use (shell, file edits, test runs). Extended
reasoning was on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 00:15:53 -06:00
DottaandClaude Opus 5.5 8a0f95849d perf(native-runtime): start the first turn without re-copying launch files, and never fail a run on a slow copy (#21287)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents that use OpenCode in a remote sandbox must have a durable,
verified copy of the provider launch files before their first turn (the
rule from #19622)
> - Today each new run copies these files out of the sandbox, uploads
them and verifies them again, even when the company already has a ready
artifact with the same files
> - The whole copy runs under one five-minute clock. When eight runs
start together on small sandboxes, the copies compete, and slow copies
fail the run before its first turn
> - This makes cold starts slow (p50 273 s from run start to turn
submitted) and makes half of the first runs fail in a burst
> - This pull request reuses a ready artifact with the same digests,
shares one in-flight copy per company and manifest, and bounds each
network step instead of the whole copy
> - The benefit is that first turns start in about one minute on a cold
burst and in about 20 s when the artifact is already ready, and a slow
copy no longer fails the run

## Linked Issues or Issue Description

Refs #19622

**What happened?**
When 8 OpenCode runs started together on 1 vCPU Daytona sandboxes, 4 of
8 first runs failed before their first turn. Each run copied, uploaded
and verified the same provider launch files. The single 300 s clock over
the whole copy expired. Run start to turn submitted was p50 273 s and
p95 314 s.

**Expected behavior**
A run starts its first turn without a new copy when a ready, verified
artifact with the same files exists. A slow copy does not fail the run.

**Steps to reproduce**
1. Use a company with remote OpenCode agents on Daytona sandboxes with
durable storage.
2. Start 8 runs at the same time on new sandboxes.
3. See the time before the first turn and the failed runs.

**Agent adapter(s) involved**
OpenCode (native runtime, remote sandbox).

## What Changed

- `retainReadyNativeProviderArtifacts` (`native-provider-artifacts.ts`):
reuse a ready, verified company artifact with the same paths and SHA-256
digests through a lookup only. No file is copied, uploaded or verified
again. The lookup (`readyNativeProviderArtifactCandidates`) matches
target, paths and digests in Postgres with jsonb containment on the
company's ready rows (existing partial index), newest first, limit 4. It
does not load or hash other retained versions. No migration.
- `shareNativeProviderArtifactCapture`
(`native-provider-artifact-capture.ts`): one in-flight capture per
company and manifest digest per server. Other runs wait for it, then
reuse its result.
- The single 300 s clock over the whole capture is replaced by a bound
for each network step (upload, verifying download): max(120 s, size at
256 KiB/s). A timeout has the typed cause
`native_provider_artifacts_transfer_timeout`. The capture makes up to 3
attempts from the local copy.
- Stop now interrupts the capture and the wait for a shared capture
(`ActiveNativeSession.admissionAbort`).
- `doc/opencode-cold-recovery.md` describes the new behavior.
- Tests: new `native-provider-artifact-gate.test.ts` (7 tests), 5 new
tests in `native-harness-backup-storage.integration.test.ts`, and new
executor variants `opencode-new-durable-slow-capture` and
`opencode-new-durable-ready` in `native-session-executor.test.ts`.

## Verification

Local (this branch, on master 9b613562b7):

- `npx vitest run
src/services/native-runtime/native-provider-artifact-gate.test.ts
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-harness-backup-storage.integration.test.ts`:
3 files, 1511 passed, 7 skipped.
- `npx tsc --noEmit -p server`: pass.
- Lookup scale: the integration test "loads only the ready artifact
whose launch files match, however many other versions the company keeps"
adds 200 ready non-matching versions. It returns only the matching
artifact. Without the SQL filter it fails (4 rows returned).
- Red/green: with master's `native-session-executor.ts` and the new
tests, `opencode-new-durable-slow-capture` fails with `TimeoutError: The
operation was aborted due to timeout`, and `opencode-new-durable-ready`
fails because the ready artifact is not looked up. With this change,
both pass.

Daytona (same commit content, cherry-picked onto the baseline commit
84a1ee3e11 with #21262; image `sha256:9461cc75…`; 1 vCPU sandboxes;
OpenCode):

| Wave | Shape | First runs OK | Run start → turn submitted | Other |
|---|---|---|---|---|
| Baseline `w-20261113T011027-s1702` | 8 runs started together | 4/8 (4
failed) | p50 273 s / p95 314 s | "only a getter" TypeError ×8,
`bootstrap_not_proven_incomplete` ×8 |
| Fix `w-20261113T031045-s1702` | same seed and shape | 8/8 | p50 66.0 s
/ p95 85.7 s | 8/8 tasks hands-off, 0 Retry banners; first agent event
p50 72.9 s / p95 90.0 s; exactly 1 upload, 7 runs waited on it (~65 s
shared capture); all failure codes 0 |
| Fix `w-20261113T034033-s1702` | 16 tasks, cap 8 | 16/16 | p50 19.1 s /
p95 24.3 s | second batch (artifact already ready, cold sandboxes) gate
time 24, 25, 26, 27, 31, 31, 45, 119 ms; first agent event p50 25.9 s |

## Risks

- The rule from #19622 still holds: a durable, verified artifact exists
before the first turn. Reuse only accepts a ready, verified artifact of
the same company with the same paths and digests.
- The first run per company and manifest per server still pays the full
transfer.
- Two servers can still upload duplicate artifacts for the same company
and manifest. Sharing is per server.
- No test drives Stop through the executor during a capture. Signal
handling is unit-tested only.
- A capture that fails all 3 attempts still ends blocked with a working
Retry. A follow-up PR (pre-turn failures continue automatically) removes
that.
- The documented five-minute deadline per capture is replaced by bounds
per network step. A very large artifact on a slow link can take longer
than five minutes in total, but each step still has a bound.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edit, test runs). Default context
window; extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:48:55 -06:00
DottaandClaude Opus 5.5 0c04ff76eb fix(native-runtime): a follow-up that arrives as a warm session closes continues the task (#21279)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native agent session on a remote sandbox stays warm after a run,
and an idle timeout closes it after about 5 minutes
> - When a person's follow-up comment arrived just as that idle close
began, the new run failed with `native_session_supervisor_busy`
> - The idle close of the finished owner could also fail its final
checkpoint with `native_runner_checkpoint_owner_lost`, because the task
already pointed at the queued follow-up run
> - Then the retry was held for workspace repair and the task stayed
blocked; the environment could not be deleted (409) because its warm
runner could not be stopped
> - This pull request makes the follow-up wait for the idle close that
is in flight, and lets the finished owner save its final checkpoint
while its successor still waits in the queue
> - The benefit is that a follow-up that lands at the warm session's
idle close continues the task with no person involved

## Linked Issues or Issue Description

Refs #18011 (the checkpoint ownership fence this change extends).

**What happened?**

A follow-up comment arrived 295–340 s after the previous run ended,
while the warm session's idle close was in progress. The new run failed
with `native_session_supervisor_busy`. In another case, the idle close
failed its final checkpoint with `native_runner_checkpoint_owner_lost`,
the next run failed
`native_instruction_materialization_handoff_unavailable`, and the retry
was cancelled with `execution_reconciliation_required`. The task stayed
blocked. A hosted instance showed the same class as "previous run's
session on this sandbox could not be retired".

**Expected behavior**

The follow-up run waits for the idle close (inside the existing bounded
release wait) and then starts. The finished owner saves its final
checkpoint and closes. The task completes hands-off.

**Steps to reproduce**

1. Run an opencode agent on a Daytona sandbox with a warm session (idle
timeout 300 s).
2. Let a run finish.
3. Post a comment on the task 298–306 s after the run ended.

**Paperclip version or commit**

master `444dc18d98`.

**Deployment mode**

Local server with Daytona remote sandboxes (restart-chaos harness, light
lane). The same class was seen on a hosted staging instance.

## What Changed

- `claimWarmNativeInstructionCopy` now also waits when an idle close is
in flight for the session scope (`closingWarmNativeSessions`). The wait
uses the same deadline as the existing release wait
(`warmSessionReleaseWait`), so it adds no new wait and nothing before
`turn.start` on the normal path.
- `validateNativeRunnerCheckpointOwner` accepts the finished owner's
final checkpoint (accepted result, no new work) when
`issues.executionRunId` points at a successor run that is `queued` or
`scheduled_retry` for the same company and task and has no
`native_run_finalizations` row.
- Tests: a server test for a follow-up that arrives while the idle
timeout closes the warm session; four new scenarios (`queued_successor`,
`scheduled_retry_successor` × `runner.suspend`/`turn.stop`) and four new
negatives (`queued_wrong_issue`, `queued_new_work` × both commands) in
the early-retirement checkpoint matrix.
- Restart-chaos harness:
`RESTART_CHAOS_MIX_FOLLOW_UP_AFTER_END_MS=<min>-<max>` and
`RESTART_CHAOS_MIX_FOLLOW_UP_COUNT` post the staging-mix follow-up a set
delay after the task's last run ended, so a cell can aim the follow-up
at the warm idle close. The restart-chaos README describes both
variables. A failed post is retried with the same `clientRequestId`, the
grade check `after_end_follow_ups_served` needs an agent reply after
each post and the ledger marker, and the task budget includes the
planned waits.
- Harness: the wave database cleanup
(`tests/runner-e2e/controller-database.ts`) no longer waits forever when
a group signal (for example `timeout` or Ctrl-C) already stopped the
embedded Postgres. Before, `embedded-postgres` `stop()` waited for an
exit event that had already fired, and an interrupted wave hung after
the Daytona R6 sweep.

## Verification

- Fix 1, red without the product hunk: `native-session-executor.test.ts
-t "idle timeout is closing"` fails with `promise rejected "Error:
native_session_supervisor_busy"`. Green with the fix: 1 passed.
- Fix 2, red without the product hunk:
`native-runner-restart-recovery.integration.test.ts` fails 4 of 208
(`queued_successor` and `scheduled_retry_successor` ×
`runner.suspend`/`turn.stop`) with
`native_runner_checkpoint_owner_lost`. Green with the fix: 208 passed,
14 skipped. The two new negative cases pass in both states.
- Harness hang fix, red without the fix: the new
`controller-database.test.ts` case ("finishes cleanup when the database
already stopped because a group signal reached it") times out on cleanup
(`expected 'hung' to be 'cleaned'`). Green with the fix: 5 passed, 1
skipped.
- After the rebase onto master `7154030ef6`, both server test files
together: 1639 passed, 21 skipped.
- Restart-chaos harness tests (`staging-mix`, `staging-mix-retry`,
`observer`): 75 passed, including a test for the after-end retry, the
`after_end_follow_ups_served` grade check and the added budget. `tsc
--noEmit` passes for `server`, `tests/runner-e2e` and
`tests/restart-chaos`.
- `pnpm verify` exits 2 because `tests/restart-chaos/observer.ts` is a
global CI input. Required CI supplies the broad coverage.
- Daytona cell on this branch (4 opencode tasks, warm sandboxes, 1 vCPU,
follow-up comment posted 300.8–308.2 s after each task's last run ended,
image built for master `444dc18d98`): 4/4 tasks done hands-off, 8/8 runs
succeeded, 0 Retry banners, 0 `owner_lost` / `supervisor_busy` / "could
not be retired" / `execution_reconciliation_required` lines in the
server log, environment delete clean, warm-shutdown grade pass, 0
sandboxes left. The harness exited by itself after 1076 s.
- Same cell before the fix (master, 8 tasks): 1/8 tasks blocked
(`native_session_supervisor_busy` →
`native_runner_checkpoint_owner_lost` →
`execution_reconciliation_required`). With fix 1 only: 1/8 still blocked
(`owner_lost` on the idle close). With both fixes: 8/8 done and 0 failed
runs (cut by the 20-minute cap before cleanup), then 4/4 above.
- The cell grade `T:completion_once` fails on all 4 tasks by design: the
timed follow-up comment arrives after the task is done, so the agent
posts a second completion comment.

## Risks

- Fix 1 can delay a follow-up by up to the existing warm release wait
(30 s) while an idle close finishes. Before this change, that run
failed. If the close fails or the wait expires, the claim returns null
and the run uses a fresh instruction copy, as before.
- Fix 2 widens the checkpoint ownership fence from #18011. It does not
reopen what #18011 closed:
- The exemption applies only to the finished owner's final save: the
checkpoint must hold an accepted result and no new work, so no provider
turn can be granted through it.
- The successor must still be unclaimed (`queued` or `scheduled_retry`)
and must have no native coordinator (`native_run_finalizations`) row. No
controller has started it, so nothing can have consumed or depended on
the predecessor's checkpoint. Once the successor is claimed, the
existing preparing/claimed rules from #18011 apply unchanged.
  - The successor must belong to the same company and the same task.
- Cases that must still reject, with tests in the same matrix:
- `queued_wrong_issue`: a queued successor for a different task is
rejected with `native_runner_checkpoint_owner_lost` and the stored
checkpoint does not change.
- `queued_new_work`: a `turn.start` (new work) while the successor is
queued is rejected with `native_runner_checkpoint_owner_lost` and the
stored checkpoint does not change.
- All earlier rejections stay: unclaimed native coordinator,
dispatching, expired or foreign preparation, foreign native preparation,
wrong task, new work, unaccepted result, replaced controller.
- Not changed: an owner whose close fails still blocks environment
delete (409). That is a separate product decision.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edit, test runs). Earlier commits on
this branch were written by the same model in an earlier session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:42:50 -06:00
DottaandClaude Opus 5.5 497ccba8f1 fix(native): replace a reattached Runner that left its recovery read unanswered (#21272)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs execute in a remote Runner that the server
controls over the durable PRP channel
> - When a run is interrupted, a same-run retry reattaches the live
Runner and reads its session (`session.snapshot`) before it continues
> - On staging, the reattached Runner authenticated and restored its
session, and then did not answer the read
> - The recovery kept that Runner, so each later attempt adopted the
same Runner and failed the same way until the attempt budget ended the
run
> - This pull request makes the next attempt replace that Runner with a
fresh Runner over its saved journal, and makes a failed Codex history
read say what failed
> - The benefit is that an interrupted Codex run continues without a
manual Retry, and its provider turn is not sent again

## Linked Issues or Issue Description

Refs #21161, Refs #21053, Refs #20853

**What happened?**

A Codex agent on Daytona ran recovery runs (wake
`issue_recovery_action_restored`). In one hour, 11 of about 35 finished
runs failed. All failed the same way:

- `native_session_recovery_failed: HarnessCapabilityUnavailableError:
read is unavailable: Error: runnerd exited while waiting for
session.snapshot` (8 runs)
- `native_session_recovery_failed: HarnessReconciliationError:
codex_history_read_failed: thread/turns/list requires supported
paginated history; Error: provider_transport_failed: PRP command
session.snapshot timed out (runner connected; pending ahead 0; last
runner event seq 338 76s ago)` (3 runs)

The run log of each run shows one sequence:

1. Attempt 1 fails with `runnerd did not report the provider turn
identity`. The close leaves the Runner alive and unsettled for the
same-run reattachment.
2. Attempt 2 adopts that Runner (same PID). The Runner authenticates.
The log shows `runnerd restored its durable PRP session and provider
thread`. Then the driver's recovery read (`thread/read` or
`thread/turns/list`, which both send `session.snapshot`) gets no answer.
The Runner still answers WebSocket pings, but it sends no command result
and almost no events (`pending ahead 0`, last event 76 s to 391 s
earlier). The recovery treats this as transient and detaches. The Runner
stays alive.
3. Attempt 3 adopts the same Runner again and fails the same way. In
some runs a protocol fault (`thread_binding_mismatch`) closes the
transport during the read, so the message is `runnerd exited while
waiting for session.snapshot`.
4. The provider attempt budget ends the run.

#21161 did not help. Its wait extension applies only while a queued
answer is being committed. Here the Runner sent no answer at all.

The `paginated history` text was also wrong. `codex-history.ts` called
every failed history read unsupported, also a transport timeout.

**Expected behavior**

A Runner that left a recovery read unanswered is not adopted again. The
next attempt starts a fresh Runner from the saved journal, resumes the
provider session from the saved checkpoint, and does not send the
provider turn again. A failed history read reports the real failure. A
Codex build that refuses paginated history falls back to one complete
thread read.

**Steps to reproduce**

1. Start a Codex run in a remote (Daytona) Runner.
2. Let the first attempt fail at turn start, so that the close keeps the
Runner alive.
3. Let the reattached Runner stop answering commands after it
authenticates (for example, stop the Runner process after its first
answer).
4. See `PRP command session.snapshot timed out (runner connected;
pending ahead 0; ...)` on attempt 2 and the same failure on attempt 3,
with the same Runner PID.

**Paperclip version or commit**

Staging Core `57aeda190b` (contains #21161 and #21120). Fixed on
`master` after `53608bd1d6`.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`, `codex_app_server`).

## What Changed

- `nativeRecoveryCommandUnanswered` (executor) finds a
`native_session_recovery_failed` whose Runner did not answer a recovery
command (`PRP command <type> timed out` or `runnerd exited while waiting
for <type>`). Identity, authority, integrity and incompatibility
refusals do not match.
- When such a failure ends a `reattach_remote_runner` attempt on a
sandbox target, the failure detail records `adoptedRunnerUnanswered:
true`.
- `claimNativeRestartRecoveries` copies it to the next claim as
`replaceUnansweredRunner: true`, in the same way as
`exitedBeforeSessionOpen`.
- `verifyRemoteRunnerOrSettledExit` with `replaceUnansweredRunner` keeps
all identity checks. When the old Runner is still alive, it returns no
adopted process, and the attempt launches a fresh Runner from the saved
journal. The existing launch fence stops the old `paperclip-runnerd` on
that state folder and its whole lineage, and fails the launch closed
when it cannot prove them gone. A Runner that already exited mid-turn
keeps the existing fail-closed path, because its provider or tool
processes can survive where the fence cannot find them. The task then
continues in a fresh session from the saved work. The attempt budget is
unchanged.
- `codex-history.ts`: a history read that fails without a provider
answer is now `codex_history_read_failed: <method> read failed;
<error>`. A provider refusal of the method (`-32601` or `-32600`) on the
first page falls back to one `thread/read` with `includeTurns: true`.
That fallback fails closed when the thread differs, or when turns, items
or turn identities are missing. A refusal on a later page still fails
closed.
- `doc/execution-semantics.md` describes both rules. The launch-fence
code comment now names the replacement case.

### Changed existing test

`codex-history.test.ts` "does not turn an interrupted page or
unsupported method into idle history" used a plain `Error("method not
found")` on page 2 and expected the `supported paginated history` text.
A plain error is now a failed read, not an unsupported method. The test
now uses a real provider refusal (`CodexRpcError` `-32601`) on page 2
and still expects a fail-closed rejection. A new test covers the
failed-read message.

## Verification

- Red before the fix, green after:
- `server/src/services/native-runtime/native-session-executor.test.ts` ›
"a same-run reattachment whose adopted Runner never answered its
recovery read": the replacement and staging-message tests failed on the
old code (the Runner was adopted again, the three staging messages were
not recognized). Two further tests ("never resumes beside an exited
unanswered Runner's lineage") failed on the first revision of this PR,
which relaunched beside an exited mid-turn Runner, and pass now. All 15
pass.
- `native-runner-restart-recovery.integration.test.ts` › "asks the
same-run reattachment to replace a Runner that left its recovery read
unanswered" (embedded Postgres): the claim had no
`replaceUnansweredRunner` on the old code. Now passes.
- `codex-history.test.ts`: 7 new tests failed on the old code. All 16
pass.
- Real `paperclip-runnerd` and fake Codex:
`runnerd-codex-transport-remote-retention.test.ts` › "continues the same
provider thread from a fresh Runner over the journal of a retained
Runner that stopped answering, without replaying its turn". It runs the
staging sequence: turn started, close keeps the stopped Runner, a second
controller adopts it, the Runner stops answering and the controller
detaches, the Runner and provider are killed (the launch fence), and a
fresh Runner over the same journal returns the same thread. The fake
Codex call log shows one `thread/start` and one `turn/start`.
- Focused suites: `packages/paperclip-runner` `src/drivers/codex/` plus
the remote-retention, adoption and cold-restore transport suites (442
tests). Two suites timed out under parallel load and pass alone. Server:
`native-session-executor`, `native-restart-recovery`,
`native-runner-restart-recovery.integration`,
`runner-lost-mid-turn-continuation` (1714 passed, 21 skipped).
- `pnpm typecheck` in `server` (all layers) and `tsc` for
`paperclip-runner` source and tests pass.
- Not run locally: the container-gated remote Runner suites
(`PAPERCLIP_TEST_RUNNER_CONTAINER`) and the full suite. CI covers the
full suite.

## Risks

- Medium. The replacement relaunches a Runner over a journal that is not
settled, but only while the old Runner is alive. Safety comes from three
facts: the launch fence stops the old Runner and its lineage with proof
or fails the launch; the Runner restores its own durable journal (as
local `resume_dead_runner` does); and the real-runnerd test shows that
the turn is not sent again. An exited Runner is never relaunched beside
its possible survivors.
- If the previous close left a `runner.suspend` pending, a fresh Runner
can execute it and exit. The attempt then fails as it does today. The
usual detach path after an unanswered read sends no suspend.
- The replacement uses the existing attempt budget. It does not add
attempts.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code
with tool use (shell, file edits, browser reads of staging run events
and logs), extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:06:24 -06:00
DottaandClaude Opus 5.5 04b50e6007 fix(native-runtime): a timed-out step records its real cause instead of leaving the run unrecoverable (#21262)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs start a Runner and a provider session, then send the
first turn
> - When a startup step fails, the native session executor redacts the
error, classifies it and records the failure
> - A timed-out step fails with a `DOMException` (`TimeoutError` or
`AbortError`). Its `message` property has only a getter
> - The redaction assigned to `error.message`. The assignment threw a
`TypeError`, so the classifier did not run and the real cause was lost
> - The run then fell into a restart-recovery chain that ended in
"requires reconciliation" wording for an operator
> - This pull request makes the redaction safe for errors with a
read-only message
> - The benefit is that a timed-out step records its real cause and
settles through the normal failure path

## Linked Issues or Issue Description

**What happened?**

In a local chaos cell with 8 opencode tasks on Daytona sandboxes, four
first runs failed before their first turn. A step before the first turn
ran under a five-minute `AbortSignal.timeout`. The step took longer than
five minutes. The run log shows `TypeError: Cannot set property message
of which has only a getter` at the startup failure handler, then
`native_finalization_missing`. The run never recorded the timeout.
Recovery then restarted the bootstrap, saw "runnerd exited while waiting
for run.prepare", and ended with `runner_state_identity_mismatch:
bootstrap_not_proven_incomplete` and
`native_continuation_requires_reconciliation`.

**Expected behavior**

A failed startup step records its own cause. The failure handler
classifies it and settles the run. It does not throw a second error that
hides the first one.

**Steps to reproduce**

Make any step before the first turn reject with
`AbortSignal.timeout(n).reason`. Before this change,
`executePaperclipNativeSession` rejects with the `TypeError` above and
the coordinator row keeps no failure detail. The new integration test
does this with the embedded database.

## What Changed

- Add `redactNativeFailureError` in
`server/src/services/native-runtime/native-failure-redaction.ts`. It
redacts the message and stack in place when it can. When the message is
read-only, it returns a new `Error` with the redacted message, the same
`name` and the redacted stack.
- Use it in the startup failure handler of
`executePaperclipNativeSession` in place of the direct assignment.
- Add unit tests for a real `TimeoutError`, a read-only `AbortError`
that holds a secret, and an ordinary typed error.
- Add an integration test: a step before the first turn rejects with a
real `TimeoutError`. The run settles through the classifier.
`failureDetail.message` is "The operation was aborted due to timeout".
The run error code is not `native_continuation_requires_reconciliation`.

## Verification

- Red on master: `native-failure-redaction.test.ts` failed 2 of 3 with
`TypeError: Cannot set property message of which has only a getter`. The
new integration test in `native-startup-retry.integration.test.ts`
failed: the call rejected with that `TypeError` and not with the
`TimeoutError`.
- Green with this change:
- `pnpm --filter @paperclipai/server exec vitest run --config
vitest.config.ts
src/services/native-runtime/native-failure-redaction.test.ts
src/services/native-runtime/native-startup-retry.integration.test.ts`
- `src/services/native-runtime/native-session-executor.test.ts`: 1431
passed, 7 skipped.
  - `pnpm verify` for the affected files.

## Risks

- Low risk. The change only alters behavior when the error message
cannot be assigned. Before, that case threw a `TypeError` and skipped
the failure handler.
- The replacement error keeps the `name` but is a plain `Error`, so
`instanceof DOMException` is false after redaction. No code in the
handler checks for `DOMException`.
- After this PR, a capture timeout before the first turn still ends
blocked. The run records the real cause, but it shows generic wording
and waits for a person to press Retry. The reason: a generic timeout has
no typed cause, so the classifier records it as
`native_session_interrupted` and treats it as ambiguous state. Follow-up
PRs give the capture step a typed cause and per-step bounds, and retry a
failure that provably happened before any prompt automatically.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs). Extended reasoning
was on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:01:04 -06:00