Commit Graph
4 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 4612fa9fb4 fix(native): restart a drained Daytona run whose runner opened its session (#19711)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on Daytona start a `paperclip-runnerd` process in a
sandbox. A graceful shutdown (every rolling deploy) suspends live native
runs "without cancelling provider work". The next boot then decides how
each run continues.
> - #19583, #19616 and #19684 let restart recovery start a remote
bootstrap again when durable evidence proves that no provider work
exists.
> - A chaos replay found another window. The drain suspended a remote
run just after its sandbox runner opened the provider session, before
the controller sent a turn and before a session snapshot existed.
> - Recovery accepted only an unfinished `session.open` in the
controller checkpoint, so it blocked the task as ambiguous. A Retry
recovered it, but automatic continuation was lost.
> - A related replay showed that the same-run bootstrap reuses the
sandbox. The interrupted runner can still hold the runner ingress port
there (`runner_ingress_bind_conflict`).
> - This pull request proves the "session opened, no turn" case and
stops the interrupted runner before relaunch.
> - The benefit is that a deploy during Daytona startup continues
automatically, without a port conflict.

## Linked Issues or Issue Description

**What happened?**

A chaos replay (ACPX Claude on Daytona) sent SIGTERM while a remote run
was starting. The drain recorded `restart_suspended`. The dying
controller still launched the runner and finished `session.open`. On the
next boot, restart recovery returned
`ambiguous_remote_recovery_evidence` with disposition `blocked`.

The decrypted controller checkpoint of that run showed:
- `run.prepare` completed and `session.open` completed (provider session
opened).
- Committed events `session.started`, `session.capabilities.updated` and
`session.goal.snapshot` (`ackedSourceSeq: 3`).
- No turn command. No run-profile session snapshot and no projected
provider events.

On the next run in that replay, a same-run bootstrap relaunched the
runner in the same sandbox (one lease for all attempts). It failed with
`runner_ingress_bind_conflict: failed to bind listener: Address already
in use`, because the interrupted runner from 51 s earlier was still
listening there. The bounded same-run retry recovered it on its last
attempt.

**Expected behavior**

When the durable controller record proves that no turn was ever sent,
recovery starts the bootstrap again. It first stops the interrupted
runner in the reused sandbox.

**Steps to reproduce**

1. Start a native run on a Daytona environment.
2. Send SIGTERM to the control plane while the runner opens its provider
session.
3. Restart the control plane after the controller lease expires.
4. See that restart recovery blocks the task with
`ambiguous_remote_recovery_evidence`.

Root cause: in
`server/src/services/native-runtime/native-restart-recovery.ts`, the
remote checkpoint proof (`remoteBootstrapProvenIncomplete`, #19583)
required an unfinished `session.open` with no committed events, and no
provider events at all. A completed `session.open` with only
session-open events had no rule, so it fell through to
`ambiguous_remote_recovery_evidence`.

Refs #19583, #19616, #19684, #19524

## What Changed

- `native-bootstrap-proof.ts`: new `runnerdStateHasNoProviderTurn`
proof. The state must hold only `run.prepare` and `session.open`
(pending or completed) and no warm, staged or adopted state. Every
committed event must be a session-open event, with contiguous sequence
numbers from 1, so no event is hidden. Only the controller issues turn
commands, and it writes each one to Postgres before sending it.
- `native-runner-checkpoint.ts`:
`nativeRunnerCheckpointRemoteBootstrapProof` replaces
`nativeRunnerCheckpointProvesIncompleteBootstrap`. It also reads a
bounded number of `event:N` parts and returns which proof holds. It
still rebuilds the state in memory.
- `native-restart-recovery.ts`: a remote run can now claim
`bootstrap_incomplete` with reason `remote_session_opened_without_turn`.
Projected provider events may only be session-open events
(`harness.ready`, `session.started`, `session.resumed`,
`session.updated`), and only together with the stronger session-opened
proof. A turn or provider event still blocks.
- `native-session-executor.ts`:
- For a remote sandbox target, the executor accepts the same proof on
the restored record.
- Before the same-run bootstrap relaunches,
`stopStaleRemoteRunnerBeforeBootstrap` stops the interrupted runner in
the reused sandbox. It uses the verified identity marker and process
check from reattachment, which signals only a process whose identity
matches this run: SIGTERM, a wait of up to 10 s, then SIGKILL.
- The stop is best effort. If no prior runner can be verified, nothing
is stopped, and the bounded same-run retry still handles any conflict.
- `doc/run-log-events.md`: document the reason and the stop.

## Verification

- Red, then green: `cd server && npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "drained remote"`. Before the fix, the opened-session case (with a
projected `session.started`) returned `blocked` /
`ambiguous_remote_recovery_evidence`.
- These keep the hold: a `turn.start` command, a committed
`turn.started`, an acknowledged event that is missing from the record,
and a projected `turn.started`. The executor test restores the
checkpoint and checks that the local proof is false and the remote proof
is true.
- Regression: `native-runner-restart-recovery.integration`,
`native-session-executor`, `native-restart-recovery`,
`native-bootstrap-retry.integration` and `heartbeat/recovery`: 1460
passed, 14 skipped.
- Stale runner stop: controlled sandbox-runner tests
(`native-session-executor.test.ts`, "stale runner stop before a same-run
remote bootstrap") run the real reattachment identity and process
scripts against a fake sandbox runner:
  - a verified runner that exits on SIGTERM gets SIGTERM only;
  - a runner that ignores SIGTERM gets SIGTERM, then SIGKILL;
  - a runner with another identity gets no signal;
  - no live prior runner means no signal.
- Live Daytona proof (wave `w-20261107T021837-s29`, ACPX Claude,
`--faults K2 --triggers time,result`, 4 tasks, concurrency 1). The image
was built from current master
(`paperclip-daytona-runner@sha256:71f711ab…`), and its content id
matches this branch.
- All 4 tasks crossed a graceful restart, and all passed R1. Two of the
K2s fired on the `result` trigger, during a running remote turn.
- The K2 during a turn reattached to the live remote runner
(`reattach_remote_runner`, generation 2), as designed.
- Task t01 failed R6 with an orphan Daytona sandbox that no lease
references. The graceful restart came during sandbox creation. This is a
separate signature, outside this change.
- The timed faults did not land in the exact "session opened, no turn"
window. The integration tests cover that window with the evidence from
the saved replay database.

## Risks

- The proof relies on two facts: only the controller issues turn
commands, and the controller writes each command before sending it
(`#sendJson` waits for the checkpoint flush). A turn always leaves a
`turn.start` command in the record.
- The provider session opened in the old sandbox runner is not resumed.
The new bootstrap opens a fresh session. No turn ran in the old one, so
no work is lost.
- The runner stop sends signals inside the run's own sandbox. It signals
only after the existing reattachment identity and process checks pass.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, GitHub CLI) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 20:58:32 -06:00
DottaandClaude Opus 5.5 803cb3bfe3 fix(native): re-bootstrap an OpenCode runner drained before its first turn (#19704)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs start a local `paperclip-runnerd`. After a restart,
recovery may start the bootstrap again when the durable state root
proves that no provider work started.
> - #19665 added a proof for a graceful drain that lands after
`session.open` started a fresh session and before the first turn. It
requires every recorded provider process to have exited.
> - OpenCode records provider startup ownership twice during
`session.open`. The first record comes before launch, with `processId:
null` and `processGroupId: null`.
> - The proof treated `processId: null` as a process it could not prove
gone. An OpenCode run drained in this window still failed with
`runner_state_identity_mismatch: bootstrap_not_proven_incomplete`.
> - This pull request treats a null process id as "no process yet". The
benefit is that a graceful restart during OpenCode bootstrap continues
automatically, as it already does for ACPX.

## Linked Issues or Issue Description

Refs #19665, #19524

**What happened?**
A restart chaos wave sent SIGTERM to the control plane about 11 s after
an OpenCode run started. The drain suspended the run. At the next boot,
recovery verified that the local runner had stopped and chose
`bootstrap_incomplete`. The executor then failed the run with
`adapter_failed` and `runner_state_identity_mismatch:
bootstrap_not_proven_incomplete`. The kept state root had:
- `run.prepare` and `session.open` completed, with a freshly started
session (`status: "started"`);
- only `session.started`, `session.capabilities.updated` and
`session.goal.snapshot` committed;
- two `harness.diagnostic` events of type `provider_startup_ownership`
in the runner outbox, the first with null process ids and the second
with the server pid and process group;
- no turn command, and the OpenCode server process had exited.

**Steps to reproduce**
1. Start an OpenCode task on a local runner.
2. Send SIGTERM to the control plane after `session.open` completes and
before the first turn.
3. Let the control plane restart.

**Expected behavior**
Recovery starts the bootstrap again, and the task continues.

**Actual behavior**
The run fails. The task needs a manual Retry.

## What Changed

- `native-bootstrap-proof.ts`: the process-id collector skips
`processId`, `pid` and `processGroupId` values that are `null`. A null
value records a launch that has not produced a process yet. Any other
non-number value still fails the proof, and every numeric id must still
be proven gone.
- Tests: a red test with the OpenCode startup-ownership shape, plus
counter-tests for a live OpenCode server process, a non-numeric process
id, and a started turn.

## Verification

- Red before the fix: "allows a fresh bootstrap after a drain once
OpenCode opened its session before the first turn" failed with `expected
false to be true`. Green after the fix: the `native incomplete-bootstrap
evidence` block passes 36/36.
- I ran the proof read-only against kept real roots:
  - the OpenCode drained root now returns `true`;
- an OpenCode root from the same wave where a turn ran (108 events)
still returns `false`;
  - the three roots from #19524 and #19665 still return `true`.
- `native-runner-restart-recovery.integration.test.ts` and
`native-restart-recovery.test.ts`: 134 passed. `tsc --noEmit -p
server/tsconfig.json` passes.
- Live replay: see the PR comment.

## Risks

- Low. The change only skips null ids. Numeric ids still need a proven
exit. Any turn, provider output, resumed session, or remote runner still
fails closed.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits, test execution). Default context
window. Extended reasoning was available.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 19:59:34 -06:00
DottaandClaude Opus 5.5 e89be6ee83 fix(native): re-bootstrap a runner that a graceful drain suspended during bootstrap (#19665)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs start a local `paperclip-runnerd` process. The control
plane keeps a durable state root for each session.
> - After a restart, recovery gives a run with no provider evidence the
`bootstrap_incomplete` disposition. The executor quarantines the old
root and starts the bootstrap again, but only when the root proves that
no provider work started.
> - #19524 made that proof accept a root killed after `run.prepare` and
before `session.open` (a hard kill).
> - A graceful restart (SIGTERM) leaves different shapes. The drain
sends `runner.drain` and `runner.suspend`, and the kill can land just
after `session.open` failed or just after it started a fresh session.
The proof refused both shapes. The run failed with
`runner_state_identity_mismatch: bootstrap_not_proven_incomplete`,
although no turn was ever sent.
> - This pull request extends the proof to both drain shapes and keeps
provider work fail-closed.
> - The benefit is that a graceful restart during native bootstrap
continues automatically.

## Linked Issues or Issue Description

Refs #19524

**What happened?**
Restart chaos waves (ACPX Claude, K2 graceful restart about 10 s after
the run started) failed with `adapter_failed` and
`runner_state_identity_mismatch: bootstrap_not_proven_incomplete`.
Startup recovery chose `bootstrap_incomplete`, and the executor proof
refused the state root. Two real roots were kept:

1. `session.open` failed (the ACPX sidecar timed out), then
`runner.drain` completed and `runner.suspend` completed. There was 1
connection, no committed or acknowledged event, an empty runner outbox,
and `nextSourceSeq: 1`.
2. `session.open` completed with a freshly started session (`status:
"started"`). The committed events were only `session.started`,
`session.capabilities.updated` and `session.goal.snapshot`, and the
runner outbox held one `harness.diagnostic` (the sidecar process). No
turn command existed, and the sidecar process had exited. The database
had not yet recorded the session events, so the classifier saw no
provider evidence.

**Steps to reproduce**
1. Start an ACPX Claude task on a local runner.
2. Send SIGTERM to the control plane about 10 s after the run starts,
before the first turn.
3. Let the control plane restart.

**Expected behavior**
Recovery starts the bootstrap again, and the task continues without a
user action.

**Actual behavior**
The run fails and the task stops.

## What Changed

- `native-bootstrap-proof.ts`, pre-session proof: a `session.open` that
is pending or `failed` is now accepted, as are the runner lifecycle
commands `runner.drain`, `runner.suspend` and `runner.shutdown` in any
state. The controller journal and the runner journal must both agree.
Any committed or acknowledged event, any turn command, a completed
`session.open`, and any queued runner event still fail closed.
- New `runnerdStateIsOpenedSessionBeforeTurn`. It accepts a completed
`session.open` only when all of these are true:
  - the session was freshly started (`status: "started"`, not resumed);
- the commands are only `run.prepare`, `session.open` and runner
maintenance commands;
- every committed event and every event in the runner outbox is a
session setup or runner lifecycle event;
  - `ackedSourceSeq` equals the count of committed events, with no gap;
  - the runner journal is present;
  - every recorded provider process ID has exited on this host.
- `native-session-executor.ts`: the executor applies the opened-session
proof only for a local execution target. Remote runners stay fail-closed
for that shape, because their process IDs belong to another host. The
remote checkpoint classifier still reads only commands 1 and 2.

## Verification

- Red before the fix: "allows a fresh bootstrap after a drain suspended
a runner whose session.open failed" and "allows a fresh bootstrap after
a drain that landed between session.open and the first turn" both failed
with `expected false to be true`. Green after the fix: the `native
incomplete-bootstrap evidence` block passes 32/32.
- Fail-closed counter-tests:
  - completed `session.open`;
  - committed event, acknowledged event, or queued runner event;
  - turn command, `turn.stop`, or a turn processed by the runner;
  - `turn.started` or `provider.event`;
  - resumed session;
  - compacted events;
  - no recorded provider process;
  - missing runner journal;
  - live provider process;
  - remote runner.
- I ran the new proof read-only against the three real kept roots: the
K1 root from #19524, the K2 `session.open` failed root and the K2
opened-session root. All three return `true` for a local runner. The
opened-session root returns `false` when the processes are on another
host.
- `native-session-executor.test.ts` and
`native-restart-recovery.test.ts` filtered to bootstrap, restart, remote
and drain: 333 passed.
`native-runner-restart-recovery.integration.test.ts` and
`native-bootstrap-retry.integration.test.ts`: 106 passed. `tsc --noEmit
-p server/tsconfig.json` passes.
- Live restart chaos replay: see the PR comment.

## Risks

- After a `session.open` failure or timeout, the provider sidecar might
have created an empty session. No turn command was sent, so no provider
work can repeat.
- An opened session before any turn is discarded, and a new one is
started. The proof requires a fresh `started` session, so no earlier
conversation is lost. Resumed sessions stay fail-closed.
- The opened-session proof is local only. It requires every recorded
provider process to be gone, so a live provider process blocks it.
- Out of scope: the Retry offer when the proof fails legitimately
(RC-05/06).

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits, test execution). Default context
window. Extended reasoning was available.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 17:55:47 -06:00
DottaandClaude Opus 5.5 d728be8fbf fix(native): re-bootstrap a Daytona runner restarted before session.open (#19583)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on Daytona start a `paperclip-runnerd` process in a
sandbox. The local control plane keeps the runner's controller state in
an encrypted Postgres checkpoint.
> - When the control plane restarts, restart recovery classifies each
live native run and decides to reattach, resume, bootstrap again, or
hold the task.
> - A restart chaos wave killed the control plane twice while a Daytona
runner was connecting. The run was claimed and the sandbox runner was
launched, but `session.open` never completed. No provider evidence
existed.
> - Recovery waited correctly for the old 60 s controller lease to
expire. Then the classifier had no rule for a claimed remote bootstrap.
It returned `ambiguous_remote_recovery_evidence` and blocked the task as
"Workspace repair required".
> - The controller writes each command to the Postgres checkpoint before
it sends it. After the lease expires, that checkpoint is the complete
record of what reached the sandbox runner.
> - This pull request lets recovery prove from that checkpoint that no
provider work exists, and then start the same-run bootstrap again.
> - The benefit is that a restart during Daytona runner startup
continues automatically instead of stopping the task.

## Linked Issues or Issue Description

**What happened?**

A restart during Daytona remote runner startup (native ACPX Claude, two
SIGKILLs about 20 s apart) left the task blocked. The run history showed
`remote_controller_lease_live` twice, then
`ambiguous_remote_recovery_evidence` with disposition `blocked`. The
task showed "Workspace repair required"
(`native_restart_recovery_blocked`) and did not continue.

**Expected behavior**

When the durable controller record proves that the runner received only
bootstrap commands, recovery continues the run with a fresh bootstrap.
Any evidence of provider work keeps the hold.

**Steps to reproduce**

1. Start a native run on a Daytona environment.
2. Kill the control plane after the sandbox runner connects and before
`session.open` completes.
3. Restart the control plane and wait for the old controller lease to
expire.
4. See that restart recovery blocks the task with
`ambiguous_remote_recovery_evidence`.

Root cause:
`server/src/services/native-runtime/native-restart-recovery.ts`
`classifyNativeRunnerRecoveryEvidence`, remote branch. It admitted only
an unclaimed bootstrap (`remote_execution_never_claimed`, #16349) or a
run with a provider checkpoint. A claimed bootstrap with no provider
evidence fell through to `ambiguous_remote_recovery_evidence`.

The decrypted checkpoint of the failed run held `connectionCount: 1`,
`run.prepare` delivered once and pending, `session.open` pending and
never delivered, no committed events, and `ackedSourceSeq: 0`.

Refs #16349, #19524

## What Changed

- `native-bootstrap-proof.ts`: move the incomplete-bootstrap predicates
from the executor into a small pure module. Restart recovery and the
executor now use the same proof. No behavior change.
- `native-runner-checkpoint.ts`: add
`nativeRunnerCheckpointProvesIncompleteBootstrap`. It reads only a
checkpoint with the bootstrap parts (header, cursor, `command:1`,
`command:2`), checks the run, runner instance, and session identity, and
applies the shared proof. A remote runner journal is not on this host,
so the proof uses the controller record alone.
- `native-restart-recovery.ts`: for a classic sandbox run whose owner
lease has expired, with a durable checkpoint, no session checkpoint, no
provider evidence, and no staged attachment, claim
`bootstrap_incomplete` with the reason
`remote_bootstrap_proven_incomplete` when the checkpoint proof holds. At
three provider attempts, block with
`execution_recovery_budget_exhausted`.
- `native-session-executor.ts`: add
`restoreRemoteBootstrapCheckpointForRecovery`. For a
`bootstrap_incomplete` recovery on a sandbox target, it restores the
Postgres checkpoint over the local external-state marker. The existing
incomplete-bootstrap proof then checks the restored state before it
quarantines the root and bootstraps again.
- `doc/run-log-events.md`: document the new disposition reason.

## Verification

- Red, then green: `cd server && npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "remote bootstrap"`. Before the fix, the proven case and the budget
case returned `blocked` / `ambiguous_remote_recovery_evidence`.
- New cases: the proven claimed bootstrap gets `bootstrap_incomplete`
(attempt kept, generation +1, run still `running`, no recovery action).
The hold stays for a queued `turn.start`, an acknowledged event, adopted
state, no checkpoint, and a spent budget. The executor helper restores
the checkpoint only for a bootstrap recovery on a classic sandbox and
only over a marker for the same run.
- Regression set: `native-runner-restart-recovery.integration`,
`native-session-executor`, `native-restart-recovery`,
`native-bootstrap-retry.integration`, `heartbeat/recovery`: 1383 passed,
14 skipped.
- Live Daytona replay of the failing chaos task (double SIGKILL during
remote runner startup), 3 runs: REPLAY_RESULTS

## Risks

- A fresh bootstrap acquires a new sandbox. The old sandbox keeps its
lease until run settlement releases all run leases. The old runner gets
no turn command, because only the controller issues one and the new
controller never connects to it.
- If `session.open` reached the old runner, the old provider session can
exist without a turn. No provider work or tool call can happen in it.
- Any checkpoint shape other than the bootstrap-only shape keeps the
previous fail-closed hold.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, GitHub CLI) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 14:43:22 -06:00