Commit Graph
5 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 57aeda190b test(runtime): stop one slow plugin Runner eviction case from failing the cases after it (#21199)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The plugin Runner eviction integration test proves that a retained
Runner continues after the host evicts it
> - The test failed on many unrelated pull requests in the "Verify
Paperclip Runner (vitest native server)" lane
> - One slow case caused other cases to fail, and the PR retry could not
run
> - This pull request isolates each case from a case that timed out, and
gives the retry a fresh issue
> - The benefit is that one slow case no longer fails the cases after it

## Linked Issues or Issue Description

**What happened?**

`server/src/services/native-runtime/plugin-runner-eviction.integration.test.ts`
failed in CI on at least 5 unrelated branches in 48 hours, most recently
on #21149. The errors were `Test timed out in 60000ms` and `Timed out
waiting for automatic release A`. Each retry then failed at once with
`duplicate key value violates unique constraint
"issues_identifier_idx"`. The failed cases logged a Runner in
`connecting` with no wire frames.

Root cause: every case changed process-wide state. It set the default
TLS trust and `https.globalAgent` for its own certificate, and it
restored them in its own `finally` block. Vitest continues to the next
case when a case times out, but the timed-out function continues to run.
Its late `finally` then removed the TLS trust of the next case and
closed the warm sessions of the next case with
`closeIdleWarmNativeSessionsForRestart()`. The next case could not open
its Runner connection, waited for the 60s startup deadline, and also
timed out. That case then did the same to the case after it. In CI run
43266976387, one timeout caused 3 more failures. Locally, one timeout
caused 5 more failures.

Two more problems made this worse:
- The issue identifier came only from the case name. A retry of the same
case therefore failed on the unique identifier, so the PR retry never
ran the test again.
- Detached cases used a 5s idle timeout. The product limits a release to
`min(idleTimeoutMs, 30s)`, so these cases had 5s for the drain and the
park. Under load, one park took more than 4.8s, and the release expired.

**Expected behavior**
A case that times out fails alone. Later cases and the PR retry run
normally.

**Steps to reproduce**
1. Make one case slower than 60s. For example, add a 62s sleep after
release A in `codex cached retired`.
2. Run `npx vitest run
src/services/native-runtime/plugin-runner-eviction.integration.test.ts
-t "codex with a (cached|detached) Core owner"` from `server/`.
3. Before this change, 3 of 4 cases fail: the slow case, and 2 cases
that it disturbs. With `--retry=1`, the slow case also fails on the
duplicate identifier.
4. After this change, only the slow case fails. With `--retry=1`, all 4
cases pass.

**Paperclip version or commit**
master at ef6d347f79, rebased on the current origin/master

**Deployment mode**
CI and local test only (embedded Postgres, real `paperclip-runnerd`)

**Agent adapter(s) involved**
paperclip_runner (codex and claude ACPX harnesses through the plugin
task driver)

## What Changed

- Create one fixture certificate in `beforeAll`. Set the TLS trust and
`https.globalAgent` once for the file, and restore them in `afterAll`. A
case no longer changes process-wide trust.
- Move the per-case teardown (close idle warm sessions, close the host
controls, close the TLS proxy) into an `afterEach`. It is registered
before the database fixture, so it runs before the database clients
close, also when a case times out.
- Add a random suffix to the issue identifier of each attempt, so the PR
retry can run the case again.
- Increase the detached idle timeout from 5s to 10s, and wait 200ms past
it as before. This gives the release the time that it needs under load.
- Probe the Runner port on `0.0.0.0`, the address that the Runner binds.

No product code changed.

## Verification

All loops ran on a shared, loaded macOS host (load average 15 to 20 on
18 cores), plus 2 CPU busy loops. Each iteration ran the 12 cases that
match `with a (after inspection|missing state|reused lease) Core owner`.

- Before: 20 iterations, 2 failed (10%). In iteration 10, `claude reused
lease retired` timed out at 60s, and 5 later cases failed on
`readNativeSessionIdleWork() === "none"` because of the case that timed
out. In iteration 17, `claude after inspection retired` failed with
`Timed out waiting for automatic release A`.
- After: see the loop result comment on this PR.
- Injection test (one case sleeps 62s): before, 3 of 4 cases failed, and
with `--retry=1` the retry failed on the duplicate identifier. After,
only the slow case failed, and with `--retry=1` all 4 passed.
- Full file after the change: 40 of 40 passed in 320s locally (335s
before, because the file now creates one certificate instead of 40).
- `tsc --noEmit -p server/tsconfig.json` reports no errors in the
changed file.

## Risks

Low risk. Only the test changed. The detached cases take about 5s more
each (12 cases, about 60s in all), which is partly offset by creating
one certificate per file. A case that hangs inside Core execution can
still leave a busy warm session in module state. This change does not
prevent that. It removes the causes of the cascade that this PR found.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with
tool use (shell, file edits) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:18:01 -06:00
Nicky LeachandClaude Opus 5.5 2d93413085 perf(runner): prepare the next Claude provider generation at park (#20145)
## Thinking Path

> - Paperclip runs agents through Paperclip Runner, and a host can park
an idle Runner between runs and resume it for the next run on the same
task.
> - For Claude (ACPX), a follow-up run on a parked Runner pays about
1.5–2.3 s at `run.attach` before the provider can start the turn. The
attachment closes the Claude sidecar and spawns a new one: initialize,
`session.open` with `session/load`, `run.attach`, then a dormant verify.
> - The respawn is required. The tool-bridge MCP endpoint lasts for the
whole session and can't tell which turn a late callback came from, so a
new turn must not run in a process that ran an earlier one.
> - Nothing requires the respawn to happen at the *start* of the next
run. A turn that ends 2 s later is a better experience than one that
starts 2 s later.
> - This pull request does the respawn while parking. The next run's
`run.attach` then only rebinds the run id on a fresh, never-used
generation.
> - Follow-up Claude runs on a parked Runner reach the provider about
1.5–2 s sooner, and the late-callback guarantees are unchanged.

## Linked Issues or Issue Description

No public issue exists. The issue is described inline.

### What happened?

On a hosted deployment with parked Runners, Claude follow-up runs
reached the provider about 1.8 s later than Codex follow-ups.
`runner.session.resume` medians were 5.2 s for Claude and 3.4 s for
Codex. The extra time is the ACPX respawn inside `run.attach`. Codex
keeps its process in place.

### Expected behavior

The respawn happens while the Runner parks, so the next run starts on a
prepared generation.

### Steps to reproduce

1. Run a Claude (ACPX) task on a host that parks Runners between runs.
2. Let the run finish and the Runner park.
3. Send a follow-up. `run.attach` closes the provider and starts,
initializes, opens and attaches a new sidecar before `turn.start`. The
Claude CLI itself starts at the first prompt on either path.

### Paperclip version or commit

`master` at e194e193.

### Deployment mode

Hosted, with parked Runner tasks.

## What Changed

- **`durable/runner.rs`:**
- New `CommandExecutor::prepare_for_park(deadline)` hook, a no-op by
default. The runner passes a deadline 20 s after the park request; the
host waits 30 s for the reply.
- The Park host-control arm calls it after `host_park_ready()`,
re-proves readiness, and only then replies `parked`.
- A failure records a diagnostic and replies `conflict`, the existing
park-failure path.
- **ACPX, park side** (`acpx_park_preparation.rs`,
`acpx_provider_generation.rs`):
- For Claude, with the session open, nothing pending and nothing queued:
reap the old generation, using the same suspend-and-reap sequence a
second turn uses.
- Start the replacement with the current run attached, and prove it
dormant.
  - Roll the tool receipt epochs and the turn-identity ledger over.
- Record a digest of the session configuration it was opened with,
ignoring the MCP binding id and the aggregate digest.
- Every sidecar request during preparation is capped at the park
deadline. Running out of time fails like any other preparation failure.
- **ACPX, attach side:**
- A prepared, never-used generation whose digest matches the next run is
rebound: the sidecar gets `run.attach` with the new run id, the reducer
and tool bridges are reset, and the snapshot is checked for the new run
id.
- Anything else takes today's cold path: a different tool set, skill,
prompt or launch; a buffered callback; or any rebind error.
- **ACPX, turn start:** the provider restarts when its generation has
run a turn, instead of when the run has settled turns.
`has_settled_turns` is removed.
- **ACPX, polling:** polling skips a prepared generation until it is
rebound or starts a turn, so nothing it emits is attributed to the
parked run.
- **Sidecar** (`acpx-runtime-sidecar.ts`): rejects `run.attach` once a
turn has started in that process.
- **Fake sidecar:** reports live predecessors at process start, and adds
the faults `reject-open`, `late-tool-after-suspend` and `pre-turn-tool`.

## Verification

- `cargo test -p paperclip-runner-core`: 1029 passed (debug).
  - New `tests/acpx_park_preparation.rs` (10 tests). These prove:
- park reaps the old pid before spawning the new one, and acks only
after the dormant verify;
- a late callback from the old generation after park is rejected, and so
is a pre-turn call from the new generation;
- a compatible attach keeps the pid, and the sidecar and snapshot carry
the new run id;
- changed instructions, tools, skills or runtime context take the cold
path;
    - a used generation is never rebound;
- a prepare failure produces a park conflict and a recoverable state;
    - Retire works after a prepared park;
- a replacement whose `session.open` stalls fails at the park deadline,
leaves the session suspended, and the next run attaches and completes a
turn.
- `cargo fmt --check` and `cargo check --workspace --locked`: clean.
- `vitest` `acpx-runtime-sidecar.test.ts` and
`acpx-sidecar-input.test.ts`: 126 passed. The new re-attach guard test
fails if the guard is removed.
- `vitest` `plugin-runner-release-resume.integration.test.ts` and
`plugin-runner-eviction.integration.test.ts` with the branch's Runner:
48 passed. Both now expect the provider process to be replaced at park
instead of at attach.
- Fake-sidecar timing (release build, 3 runs): `run.attach` took 230–260
ms cold and 40–80 ms after a prepared park.
- Real Claude (claude-agent-acp 0.73.0, release build on a hosted test
VM, 3 runs each way plus one with a 180 s cgroup freeze between park and
attach):

  | | before | this PR |
  |---|---|---|
  | `run.attach` | 1691 ms mean | 212 ms mean |
| `run.attach` start → follow-up `turn.started` | 2565 ms mean | 1065 ms
mean |
  | `turn.start` → first provider output | 2261 ms mean | 2232 ms mean |
  | park preparation added to park | — | about 1450 ms |

Session recall, an authorized tool call after the rebind, and the
absence of any tool call or event from the old run all passed in every
run, including after the freeze. The prepared generation holds only the
sidecar: the Claude CLI starts at the follow-up prompt on both paths, so
a parked Runner also holds one node process instead of the whole Claude
tree.
- Hit rate: on a hosted test deployment, all 15 consecutive Claude runs
across two tasks carried identical content-addressed instruction and
skill bundle paths, so the prepared digest would have matched.
- Not run: the hosted park, claim and attach loop with real Claude, a
native MCP binding or skills with real Claude, `cargo test --release`,
smoke tests.
- Pre-existing flake:
`unconfirmed_suspension_state_becomes_recoverable_only_after_cleanup_and_suspend`
fails intermittently on clean `master` when the `hosted acpx durable::`
subset runs in parallel. It passed in both full runs.

## Risks

- **Park takes about 1.5 s longer** before it replies with real Claude.
Preparation is capped at 20 s; the host's park wait is 30 s.
- **Configuration changes:** a change between park and the next run
(registered asset paths, tools, prompt) takes the cold path. That's
safe, but saves nothing.
- **Stale environment:** a prepared process was opened with the previous
run's environment and disk state. If credential or configuration files
change on disk while the Runner is parked, the prepared process won't
see them, whereas the cold path would.
- **Failed prepare:** a park that fails after the reap leaves no live
provider. The next turn or `run.attach` starts Claude cold, as today.
- **Scope:** only ACPX Claude prepares. Other providers park exactly as
before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, git, GitHub CLI) and subagents.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 21:38:57 -08:00
Nicky Leach 35cbd15175 fix(native): recover a stopped Runner whose journal stayed ready at a settled cut (#19907) 2026-11-07 16:01:47 -08:00
Nicky Leach 94718ce168 fix(native): continue on a replacement when a retained Runner retires before claim (#19897) 2026-11-07 14:51:32 -08:00
Nicky LeachandPaperclip beb3e55072 Continue a released Runner after confirmed graceful eviction (#19760)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core can release a completed Runner while its host retains the
session.
> - A host can later retire that Runner under its retention policy.
> - Graceful retirement saves a suspended session before physical
shutdown.
> - A cached Core owner still expects the old physical task and rejects
a new allocation.
> - This pull request retires that dead local controller after verified
stop.
> - The benefit is ordinary session continuation for Codex and qualified
Claude.

## Linked Issues or Issue Description

Refs #19611, #19683, #19196, #19733 and #19157.

After a host gracefully evicts a released task, the next run can receive
a new allocation. A cached controller rejects it with
`native_session_released_owner_scope_mismatch`. The detached-controller
path can already recover the suspended session.

## What Changed

- Require the exact source, old task and current account/workspace scope
before releasing a cached controller.
- Use the existing physical-stop receipt checks. Detach only the local
controller, then use the same suspended-state admission as the detached
path.
- Preserve the current run's instruction reservation. Retain a failed
detach for cleanup.
- Add actual Core, PostgreSQL and native Runner tests for Codex and
qualified Claude through ACPX. Route them once through the native CI
lane.

## Verification

- The original real-entry Codex case failed at the cached-task scope
check after native retirement, confirmed process exit and a recorded
stop receipt.
- The current head's native CI lane passed all 16 eviction cases for
Codex and qualified Claude. These check cached and detached owners,
current instruction preparation, reused-allocation inspection, missing
stop/state, foreign identity and non-suspended state. They verify the
same provider session, two ordinary turns, one accepted source result
and no replacement submit on refusal.
- Local checks passed the composition matrix, 12 targeted executor
controls, 43 CI lane-selection checks and server TypeScript. All eight
refusal cases now assert the expected error and whether the checkpoint
request or host read was reached. This addresses the first standard
review's test finding.
- Commands: `pnpm exec vitest run
src/services/native-runtime/plugin-runner-eviction.integration.test.ts`,
focused `native-session-executor.test.ts` controls, `node
../node_modules/typescript/bin/tsc --noEmit -p tsconfig.json`, and `node
--test scripts/__tests__/run-vitest-stable-shard.test.mjs`.
- [Full
CI](https://github.com/paperclipai/paperclip/actions/runs/42349579107)
passed on head `9a25709ef43365b1f1849413b935785838eca19b`. [Standard
review](https://github.com/paperclipai/paperclip/pull/19760#issuecomment-6703629255)
is 5/5 on this head, with no unresolved findings. Whitespace and
added-line secret/PII checks passed.

## Risks

- Controlled provider processes and an authenticated local host fixture
model the Launcher edge. This does not qualify live vendor transcript
storage, mounts or a deployment.
- This covers graceful eviction before allocation or before stopped-task
inspection during `describe`. Loss after that inspection or an ambiguous
claim remains held for reconciliation. No unavailable reply proves
physical stop.
- Missing or invalid suspended state still refuses continuation. The
deterministic refusal rule from #19733 remains unchanged. Abrupt death
without a suspended checkpoint stays with the separate #19157 work.
- No new capability, mode, schema, dependency, deployment or activation
is added. The existing draft #19157 and dependent branches remain
unchanged.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), high reasoning, through Codex with
repository tools and code execution. The runtime does not expose a
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 23:21:45 -08:00